2024-05-09T08:29:35.408 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-09T08:29:35.408 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_teuthology@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} --name teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693 -- /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693/orig.config.yaml 2024-05-09T08:29:35.468 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T08:29:35.563 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7698693' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: be85c202969d25af6e130678e7d73a20e9c4ca11 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 2334 sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc sleep_before_teardown: 0 subset: 359/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: be85c202969d25af6e130678e7d73a20e9c4ca11 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2Xk2SCnW0szSLk+R34gSsGHeRM+w0CU1VV/2WcV7on2+GaHjZuz/erqdV1m+HjPZkkdn+9Ozr4qDXQVh3lusg= smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLkdC13GzgQAr5Umlo+FDlXJUKL6j8XSP006IfOOaX5XWWxF4T0m5VsnOwft+N60zHD/VlyxqK4zrP8LXLQG1i4= smithi142.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLNc+Ez3VQuCDBR+B4wFK9ht8Hj8hDFr9v14/8wpOBMKDnAzkj3ne3qs7GSQDXGqci4Zv41xUZaY1No3hpTQvJk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-08_21:24:04 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-09T08:29:35.563 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa; will attempt to use it 2024-05-09T08:29:35.564 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks 2024-05-09T08:29:35.564 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-09T08:29:35.565 INFO:teuthology.task.internal:Checking packages... 2024-05-09T08:29:35.590 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc' 2024-05-09T08:29:35.590 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-09T08:29:35.590 INFO:teuthology.packaging:ref: None 2024-05-09T08:29:35.591 INFO:teuthology.packaging:tag: None 2024-05-09T08:29:35.591 INFO:teuthology.packaging:branch: squid 2024-05-09T08:29:35.591 INFO:teuthology.packaging:sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:29:35.591 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-05-09T08:29:35.913 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2529.gbf77c77e 2024-05-09T08:29:35.914 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-09T08:29:35.921 INFO:teuthology.task.internal:no buildpackages task found 2024-05-09T08:29:35.922 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-09T08:29:36.012 INFO:teuthology.task.internal:Saving configuration 2024-05-09T08:29:36.034 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-09T08:29:36.078 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T08:29:36.101 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 08:24:33.159354', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2Xk2SCnW0szSLk+R34gSsGHeRM+w0CU1VV/2WcV7on2+GaHjZuz/erqdV1m+HjPZkkdn+9Ozr4qDXQVh3lusg='} 2024-05-09T08:29:36.124 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 08:24:33.157396', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLkdC13GzgQAr5Umlo+FDlXJUKL6j8XSP006IfOOaX5XWWxF4T0m5VsnOwft+N60zHD/VlyxqK4zrP8LXLQG1i4='} 2024-05-09T08:29:36.146 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 08:24:33.160990', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLNc+Ez3VQuCDBR+B4wFK9ht8Hj8hDFr9v14/8wpOBMKDnAzkj3ne3qs7GSQDXGqci4Zv41xUZaY1No3hpTQvJk='} 2024-05-09T08:29:36.146 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-09T08:29:36.194 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-09T08:29:36.194 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-09T08:29:36.194 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-09T08:29:36.194 INFO:teuthology.run_tasks:Running task console_log... 2024-05-09T08:29:36.345 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f241b128160>, signals=[15]) 2024-05-09T08:29:36.346 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-09T08:29:36.352 INFO:teuthology.task.internal:Opening connections... 2024-05-09T08:29:36.353 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-05-09T08:29:36.354 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:29:36.432 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2024-05-09T08:29:36.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:29:36.508 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi142.front.sepia.ceph.com 2024-05-09T08:29:36.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:29:36.582 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-09T08:29:36.589 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-05-09T08:29:36.607 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-05-09T08:29:36.607 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-05-09T08:29:36.663 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-05-09T08:29:36.663 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-05-09T08:29:36.663 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-05-09T08:29:36.664 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-05-09T08:29:36.664 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-05-09T08:29:36.664 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-05-09T08:29:36.664 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T08:29:36.664 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-05-09T08:29:36.664 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-05-09T08:29:36.664 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T08:29:36.664 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-05-09T08:29:36.664 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T08:29:36.664 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T08:29:36.664 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T08:29:36.665 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-05-09T08:29:36.686 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2024-05-09T08:29:36.703 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2024-05-09T08:29:36.704 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-05-09T08:29:36.760 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2024-05-09T08:29:36.760 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="9" 2024-05-09T08:29:36.761 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2024-05-09T08:29:36.761 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2024-05-09T08:29:36.761 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="9" 2024-05-09T08:29:36.761 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el9" 2024-05-09T08:29:36.761 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T08:29:36.761 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2024-05-09T08:29:36.761 INFO:teuthology.orchestra.run.smithi105.stdout:LOGO="fedora-logo-icon" 2024-05-09T08:29:36.761 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T08:29:36.761 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2024-05-09T08:29:36.761 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T08:29:36.761 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T08:29:36.762 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T08:29:36.762 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-05-09T08:29:36.785 DEBUG:teuthology.orchestra.run.smithi142:> uname -m 2024-05-09T08:29:36.807 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2024-05-09T08:29:36.807 DEBUG:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2024-05-09T08:29:36.864 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Stream" 2024-05-09T08:29:36.865 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="9" 2024-05-09T08:29:36.865 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos" 2024-05-09T08:29:36.865 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora" 2024-05-09T08:29:36.865 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="9" 2024-05-09T08:29:36.865 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el9" 2024-05-09T08:29:36.865 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T08:29:36.865 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2024-05-09T08:29:36.865 INFO:teuthology.orchestra.run.smithi142.stdout:LOGO="fedora-logo-icon" 2024-05-09T08:29:36.865 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T08:29:36.865 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://centos.org/" 2024-05-09T08:29:36.865 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T08:29:36.865 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T08:29:36.865 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T08:29:36.866 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2024-05-09T08:29:36.885 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-09T08:29:36.896 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-09T08:29:37.064 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-09T08:29:37.065 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-05-09T08:29:37.068 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2024-05-09T08:29:37.070 DEBUG:teuthology.orchestra.run.smithi142:> test '!' -e /home/ubuntu/cephtest 2024-05-09T08:29:37.086 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-09T08:29:37.148 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-09T08:29:37.148 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-05-09T08:29:37.151 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2024-05-09T08:29:37.153 DEBUG:teuthology.orchestra.run.smithi142:> test -z $(ls -A /var/lib/ceph) 2024-05-09T08:29:37.438 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-09T08:29:37.512 INFO:teuthology.run_tasks:Running task kernel... 2024-05-09T08:29:37.561 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-05-09T08:29:37.561 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-05-09T08:29:37.561 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-05-09T08:29:37.561 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T08:29:37.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:29:37.578 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-05-09T08:29:37.635 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-437.el9.x86_64 2024-05-09T08:29:37.636 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-437.el9.x86_64 2024-05-09T08:29:37.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-05-09T08:29:39.893 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.1 MB 00:00 2024-05-09T08:29:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 8.8 MB/s | 19 MB 00:02 2024-05-09T08:29:47.981 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 6.2 MB 00:01 2024-05-09T08:29:49.806 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-05-09T08:29:52.933 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 21 MB 00:02 2024-05-09T08:29:58.100 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-05-09T08:30:00.164 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T08:30:00.164 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T08:30:00.261 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-05-09T08:30:00.273 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-05-09T08:30:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-05-09T08:30:00.273 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-05-09T08:30:00.273 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-05-09T08:30:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T08:30:00.274 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-05-09T08:30:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T08:30:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T08:30:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T08:30:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:30:00.274 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-05-09T08:30:00.274 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-05-09T08:30:00.274 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-05-09T08:30:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:30:00.275 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 78 M 2024-05-09T08:30:00.275 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 123 M 2024-05-09T08:30:00.275 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-05-09T08:30:01.634 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 391 kB/s | 484 kB 00:01 2024-05-09T08:30:02.142 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 8.8 MB/s | 15 MB 00:01 2024-05-09T08:30:02.951 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 13 MB/s | 34 MB 00:02 2024-05-09T08:30:03.910 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-core-5.14.0-444.el9.x86_6 12 MB/s | 28 MB 00:02 2024-05-09T08:30:03.910 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:30:03.910 INFO:teuthology.orchestra.run.smithi098.stdout:Total 21 MB/s | 78 MB 00:03 2024-05-09T08:30:04.588 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-05-09T08:30:04.931 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-05-09T08:30:04.931 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-05-09T08:30:05.876 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-05-09T08:30:05.876 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-05-09T08:30:08.274 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-05-09T08:30:08.896 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T08:30:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:30:11.628 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:30:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:30:16.481 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:30:16.554 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:30:21.276 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:30:51.186 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:30:51.224 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:30:51.558 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:30:51.558 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T08:30:51.559 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:30:51.559 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:30:52.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:30:52.316 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:30:52.316 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-05-09T08:30:52.316 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T08:30:52.316 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T08:30:52.316 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T08:30:52.317 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T08:30:52.317 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:30:52.317 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-05-09T08:30:52.687 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T08:30:52.766 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T08:30:52.768 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T08:30:52.768 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T08:30:52.768 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T08:30:52.768 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T08:30:52.768 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-05-09T08:30:52.768 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T08:30:52.768 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T08:30:52.818 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T08:30:52.818 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T08:30:52.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-05-09T08:30:52.849 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-05-09T08:30:52.849 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-05-09T08:30:52.849 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-05-09T08:30:52.850 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 68.el9 2024-05-09T08:30:52.850 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-05-09T08:30:52.850 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T08:30:52.850 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-05-09T08:30:52.850 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8264254 2024-05-09T08:30:52.850 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-05-09T08:30:52.850 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T08:30:52.850 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T08:30:52.850 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T08:30:52.850 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T08:30:52.850 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-05-09T08:30:52.850 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-05-09T08:30:52.850 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T08:30:52.850 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-05-09T08:30:52.851 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-05-09T08:30:52.851 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:30:52.851 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T08:30:52.851 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T08:30:52.851 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T08:30:52.851 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-05-09T08:30:52.851 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:30:52.851 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T08:30:52.852 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T08:30:52.852 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-444.el9.x86_64 2024-05-09T08:30:52.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T08:30:53.535 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-05-09T08:30:54.907 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T08:30:54.935 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-05-09T08:30:54.937 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-05-09T08:30:54.952 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.P6huWsuKqU 2024-05-09T08:30:54.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.P6huWsuKqU 2024-05-09T08:30:55.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.P6huWsuKqU 2024-05-09T08:30:55.160 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.P6huWsuKqU is 6KB 2024-05-09T08:30:55.208 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.P6huWsuKqU 2024-05-09T08:30:55.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T08:30:55.283 INFO:teuthology.orchestra.run.smithi098.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T08:30:55.284 INFO:teuthology.orchestra.run.smithi098.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T08:30:55.284 INFO:teuthology.orchestra.run.smithi098.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T08:30:55.284 INFO:teuthology.orchestra.run.smithi098.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T08:30:55.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T08:30:55.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-05-09T08:31:25.421 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-09T08:31:25.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-05-09T08:31:25.422 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:32:25.425 DEBUG:teuthology.orchestra.remote:timed out 2024-05-09T08:32:34.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-05-09T08:32:34.428 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:32:34.742 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-05-09T08:32:35.187 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-05-09T08:32:35.188 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-444.el9.x86_64"... 2024-05-09T08:32:35.188 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-05-09T08:32:35.203 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-444.el9.x86_64 2024-05-09T08:32:35.203 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T08:32:35.203 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T08:32:35.203 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T08:32:36.204 DEBUG:teuthology.parallel:result is None 2024-05-09T08:32:36.205 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-09T08:32:36.212 INFO:teuthology.task.internal:Creating test directory... 2024-05-09T08:32:36.213 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T08:32:36.215 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T08:32:36.218 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T08:32:36.235 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-09T08:32:36.309 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-09T08:32:36.438 INFO:teuthology.task.internal:Creating archive directory... 2024-05-09T08:32:36.439 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T08:32:36.441 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T08:32:36.443 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T08:32:36.557 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-09T08:32:36.565 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-09T08:32:36.565 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T08:32:36.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:32:36.581 DEBUG:teuthology.orchestra.run.smithi142:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T08:32:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:32:36.598 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T08:32:36.600 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T08:32:36.625 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T08:32:36.680 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:32:36.694 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:32:36.695 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:32:36.698 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:32:36.714 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:32:36.717 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:32:36.719 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-09T08:32:36.728 INFO:teuthology.task.internal:Configuring sudo... 2024-05-09T08:32:36.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T08:32:36.731 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T08:32:36.733 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T08:32:36.760 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-09T08:32:36.786 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-09T08:32:36.786 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T08:32:36.796 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T08:32:36.800 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T08:32:36.816 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:32:36.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:32:36.935 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:32:37.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:32:37.057 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:32:37.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T08:32:37.138 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:32:37.182 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:32:37.236 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:32:37.320 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:32:37.359 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-09T08:32:37.359 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T08:32:37.424 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:32:37.469 DEBUG:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:32:37.515 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:32:37.599 DEBUG:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:32:37.640 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-05-09T08:32:37.640 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T08:32:37.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-05-09T08:32:37.708 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2024-05-09T08:32:37.710 DEBUG:teuthology.orchestra.run.smithi142:> sudo service rsyslog restart 2024-05-09T08:32:37.741 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T08:32:37.747 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T08:32:37.790 INFO:teuthology.orchestra.run.smithi142.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T08:32:38.198 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-09T08:32:38.206 INFO:teuthology.task.internal:Starting timer... 2024-05-09T08:32:38.206 INFO:teuthology.run_tasks:Running task pcp... 2024-05-09T08:32:38.286 INFO:teuthology.run_tasks:Running task selinux... 2024-05-09T08:32:38.371 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-09T08:32:38.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-05-09T08:32:38.443 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-05-09T08:32:38.445 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2024-05-09T08:32:38.506 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: 2024-05-09T08:32:38.508 DEBUG:teuthology.orchestra.run.smithi142:> sudo service auditd rotate 2024-05-09T08:32:38.572 INFO:teuthology.orchestra.run.smithi142.stdout:Rotating logs: 2024-05-09T08:32:38.574 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-09T08:32:38.574 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-05-09T08:32:38.598 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-05-09T08:32:38.599 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2024-05-09T08:32:38.623 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2024-05-09T08:32:38.623 DEBUG:teuthology.orchestra.run.smithi142:> /usr/sbin/getenforce 2024-05-09T08:32:38.654 INFO:teuthology.orchestra.run.smithi142.stdout:Permissive 2024-05-09T08:32:38.654 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi105.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi142.front.sepia.ceph.com': 'permissive'} 2024-05-09T08:32:38.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T08:32:38.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:32:38.679 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T08:32:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:32:38.705 DEBUG:teuthology.orchestra.run.smithi142:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T08:32:38.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:32:38.731 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-09T08:32:38.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-05-09T08:32:38.757 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2024-05-09T08:32:38.784 DEBUG:teuthology.orchestra.run.smithi142:> sudo /usr/sbin/setenforce permissive 2024-05-09T08:32:38.815 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-09T08:32:38.830 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-09T08:32:38.836 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-09T08:32:38.985 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-09T08:32:39.003 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-09T08:32:39.028 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi105.front.sepia.ceph.com,smithi142.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-09T08:36:59.673 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi105.front.sepia.ceph.com'), Remote(name='ubuntu@smithi142.front.sepia.ceph.com')] 2024-05-09T08:36:59.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-05-09T08:36:59.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:36:59.754 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-05-09T08:36:59.830 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-05-09T08:36:59.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-05-09T08:36:59.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:36:59.926 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-05-09T08:37:00.005 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-05-09T08:37:00.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-05-09T08:37:00.006 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:37:00.120 DEBUG:teuthology.orchestra.run.smithi142:> true 2024-05-09T08:37:00.193 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-05-09T08:37:00.193 INFO:teuthology.run_tasks:Running task clock... 2024-05-09T08:37:00.203 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-09T08:37:00.204 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T08:37:00.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T08:37:00.206 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T08:37:00.207 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T08:37:00.209 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T08:37:00.209 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T08:37:00.235 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T08:37:00.238 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T08:37:00.249 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T08:37:00.252 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T08:37:00.260 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T08:37:00.273 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T08:37:00.274 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-05-09T08:37:00.277 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2024-05-09T08:37:00.286 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-05-09T08:37:00.287 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2024-05-09T08:37:00.295 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: ntpd: command not found 2024-05-09T08:37:00.300 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T08:37:00.301 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T08:37:00.305 INFO:teuthology.orchestra.run.smithi142.stdout:506 Cannot talk to daemon 2024-05-09T08:37:00.313 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T08:37:00.314 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T08:37:00.318 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T08:37:00.332 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T08:37:00.366 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-05-09T08:37:00.369 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T08:37:00.369 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-05-09T08:37:00.369 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:37:00.370 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:37:00.370 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:37:00.370 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:37:00.370 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-05-09T08:37:00.372 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T08:37:00.373 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-05-09T08:37:00.373 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:37:00.373 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:37:00.373 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:37:00.373 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:37:00.384 INFO:teuthology.orchestra.run.smithi142.stderr:bash: line 1: ntpq: command not found 2024-05-09T08:37:00.387 INFO:teuthology.orchestra.run.smithi142.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T08:37:00.387 INFO:teuthology.orchestra.run.smithi142.stdout:=============================================================================== 2024-05-09T08:37:00.388 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:37:00.388 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:37:00.388 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:37:00.388 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:37:00.388 INFO:teuthology.run_tasks:Running task install... 2024-05-09T08:37:00.398 DEBUG:teuthology.task.install:project ceph 2024-05-09T08:37:00.398 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc'}} 2024-05-09T08:37:00.398 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc'} 2024-05-09T08:37:00.398 INFO:teuthology.task.install:Using flavor: default 2024-05-09T08:37:00.412 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-09T08:37:00.412 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-09T08:37:00.413 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'tag': None, 'wait_for_package': False} 2024-05-09T08:37:00.414 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:37:00.417 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'tag': None, 'wait_for_package': False} 2024-05-09T08:37:00.417 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:37:00.419 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'tag': None, 'wait_for_package': False} 2024-05-09T08:37:00.419 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:37:00.587 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/ 2024-05-09T08:37:00.587 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2515.g96805dcf 2024-05-09T08:37:00.590 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/ 2024-05-09T08:37:00.591 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2515.g96805dcf 2024-05-09T08:37:00.594 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/ 2024-05-09T08:37:00.595 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2515.g96805dcf 2024-05-09T08:37:00.697 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T08:37:00.698 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:37:00.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T08:37:00.704 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T08:37:00.704 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-09T08:37:00.704 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T08:37:00.724 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T08:37:00.724 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T08:37:00.729 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T08:37:00.729 DEBUG:teuthology.orchestra.run.smithi105:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T08:37:00.760 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T08:37:00.760 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-05-09T08:37:00.760 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T08:37:00.784 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T08:37:00.784 DEBUG:teuthology.orchestra.run.smithi142:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T08:37:00.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T08:37:00.793 DEBUG:teuthology.orchestra.run.smithi105:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T08:37:00.849 DEBUG:teuthology.orchestra.run.smithi142:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T08:37:00.866 DEBUG:teuthology.orchestra.run.smithi105:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T08:37:00.867 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T08:37:00.923 DEBUG:teuthology.orchestra.run.smithi142:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T08:37:00.929 INFO:teuthology.orchestra.run.smithi105.stdout:check_obsoletes = 1 2024-05-09T08:37:00.930 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum clean all 2024-05-09T08:37:00.932 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-05-09T08:37:00.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-05-09T08:37:00.986 INFO:teuthology.orchestra.run.smithi142.stdout:check_obsoletes = 1 2024-05-09T08:37:00.988 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum clean all 2024-05-09T08:37:01.153 INFO:teuthology.orchestra.run.smithi105.stdout:54 files removed 2024-05-09T08:37:01.163 INFO:teuthology.orchestra.run.smithi098.stdout:54 files removed 2024-05-09T08:37:01.180 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T08:37:01.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T08:37:01.215 INFO:teuthology.orchestra.run.smithi142.stdout:54 files removed 2024-05-09T08:37:01.242 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T08:37:01.834 INFO:teuthology.orchestra.run.smithi105.stdout:ceph packages for x86_64 211 kB/s | 83 kB 00:00 2024-05-09T08:37:01.854 INFO:teuthology.orchestra.run.smithi142.stdout:ceph packages for x86_64 237 kB/s | 83 kB 00:00 2024-05-09T08:37:01.874 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 206 kB/s | 83 kB 00:00 2024-05-09T08:37:02.238 INFO:teuthology.orchestra.run.smithi142.stdout:ceph noarch packages 34 kB/s | 11 kB 00:00 2024-05-09T08:37:02.243 INFO:teuthology.orchestra.run.smithi105.stdout:ceph noarch packages 31 kB/s | 11 kB 00:00 2024-05-09T08:37:02.308 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 30 kB/s | 11 kB 00:00 2024-05-09T08:37:02.660 INFO:teuthology.orchestra.run.smithi105.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-05-09T08:37:02.697 INFO:teuthology.orchestra.run.smithi142.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2024-05-09T08:37:02.734 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-05-09T08:37:03.618 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 8.1 MB 00:00 2024-05-09T08:37:04.660 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.1 MB 00:01 2024-05-09T08:37:04.738 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.1 MB 00:02 2024-05-09T08:37:06.428 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-05-09T08:37:08.077 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 19 MB 00:03 2024-05-09T08:37:09.198 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 19 MB 00:03 2024-05-09T08:37:11.279 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 6.2 MB 00:01 2024-05-09T08:37:12.796 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 16 kB 00:00 2024-05-09T08:37:13.286 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.2 MB 00:01 2024-05-09T08:37:13.390 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 6.2 MB 00:00 2024-05-09T08:37:14.714 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-09T08:37:14.828 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 16 kB 00:00 2024-05-09T08:37:14.999 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-05-09T08:37:16.522 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-09T08:37:16.969 INFO:teuthology.orchestra.run.smithi142.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-09T08:37:19.931 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-05-09T08:37:21.614 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-05-09T08:37:22.167 INFO:teuthology.orchestra.run.smithi142.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-09T08:37:22.224 INFO:teuthology.orchestra.run.smithi105.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:37:22.225 INFO:teuthology.orchestra.run.smithi105.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:37:22.236 INFO:teuthology.orchestra.run.smithi105.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T08:37:22.236 INFO:teuthology.orchestra.run.smithi105.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T08:37:22.237 INFO:teuthology.orchestra.run.smithi105.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T08:37:22.238 INFO:teuthology.orchestra.run.smithi105.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T08:37:22.240 INFO:teuthology.orchestra.run.smithi105.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T08:37:22.242 INFO:teuthology.orchestra.run.smithi105.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T08:37:22.243 INFO:teuthology.orchestra.run.smithi105.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T08:37:22.305 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-05-09T08:37:22.320 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-05-09T08:37:22.320 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-05-09T08:37:22.320 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-05-09T08:37:22.320 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-05-09T08:37:22.320 INFO:teuthology.orchestra.run.smithi105.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T08:37:22.320 INFO:teuthology.orchestra.run.smithi105.stdout: ceph x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 6.6 k 2024-05-09T08:37:22.320 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.4 M 2024-05-09T08:37:22.320 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 889 k 2024-05-09T08:37:22.321 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 145 k 2024-05-09T08:37:22.321 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 1.5 M 2024-05-09T08:37:22.321 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 142 k 2024-05-09T08:37:22.321 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 3.5 M 2024-05-09T08:37:22.321 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 7.4 M 2024-05-09T08:37:22.321 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 49 k 2024-05-09T08:37:22.321 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 10 M 2024-05-09T08:37:22.321 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 48 M 2024-05-09T08:37:22.321 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 264 k 2024-05-09T08:37:22.321 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 763 k 2024-05-09T08:37:22.321 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 222 k 2024-05-09T08:37:22.321 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-shell x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 27 k 2024-05-09T08:37:22.321 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-top noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 18 k 2024-05-09T08:37:22.322 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T08:37:22.322 INFO:teuthology.orchestra.run.smithi105.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T08:37:22.322 INFO:teuthology.orchestra.run.smithi105.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T08:37:22.322 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 32 k 2024-05-09T08:37:22.322 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 746 k 2024-05-09T08:37:22.322 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 127 k 2024-05-09T08:37:22.322 INFO:teuthology.orchestra.run.smithi105.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T08:37:22.322 INFO:teuthology.orchestra.run.smithi105.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T08:37:22.322 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T08:37:22.322 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T08:37:22.322 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T08:37:22.322 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T08:37:22.322 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 162 k 2024-05-09T08:37:22.323 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 321 k 2024-05-09T08:37:22.323 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 298 k 2024-05-09T08:37:22.323 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 99 k 2024-05-09T08:37:22.323 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 85 k 2024-05-09T08:37:22.323 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:37:22.323 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 171 k 2024-05-09T08:37:22.323 INFO:teuthology.orchestra.run.smithi105.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T08:37:22.323 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrading: 2024-05-09T08:37:22.323 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T08:37:22.323 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T08:37:22.323 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T08:37:22.323 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T08:37:22.323 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.4 M 2024-05-09T08:37:22.323 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:37:22.324 INFO:teuthology.orchestra.run.smithi105.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T08:37:22.324 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T08:37:22.324 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-05-09T08:37:22.324 INFO:teuthology.orchestra.run.smithi105.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T08:37:22.324 INFO:teuthology.orchestra.run.smithi105.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T08:37:22.324 INFO:teuthology.orchestra.run.smithi105.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T08:37:22.324 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 21 M 2024-05-09T08:37:22.324 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 32 k 2024-05-09T08:37:22.324 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 2.4 M 2024-05-09T08:37:22.324 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 249 k 2024-05-09T08:37:22.324 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 4.7 M 2024-05-09T08:37:22.324 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 17 M 2024-05-09T08:37:22.324 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 16 k 2024-05-09T08:37:22.324 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 25 k 2024-05-09T08:37:22.325 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T08:37:22.325 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T08:37:22.325 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T08:37:22.325 INFO:teuthology.orchestra.run.smithi105.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T08:37:22.325 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T08:37:22.325 INFO:teuthology.orchestra.run.smithi105.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T08:37:22.325 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T08:37:22.325 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T08:37:22.325 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 163 k 2024-05-09T08:37:22.325 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T08:37:22.325 INFO:teuthology.orchestra.run.smithi105.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T08:37:22.325 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T08:37:22.325 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T08:37:22.325 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T08:37:22.326 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 500 k 2024-05-09T08:37:22.326 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T08:37:22.326 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.3 M 2024-05-09T08:37:22.326 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T08:37:22.326 INFO:teuthology.orchestra.run.smithi105.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T08:37:22.326 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T08:37:22.326 INFO:teuthology.orchestra.run.smithi105.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T08:37:22.326 INFO:teuthology.orchestra.run.smithi105.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T08:37:22.326 INFO:teuthology.orchestra.run.smithi105.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T08:37:22.326 INFO:teuthology.orchestra.run.smithi105.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T08:37:22.326 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T08:37:22.326 INFO:teuthology.orchestra.run.smithi105.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T08:37:22.326 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T08:37:22.326 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 45 k 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 135 k 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T08:37:22.327 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T08:37:22.328 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T08:37:22.328 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T08:37:22.328 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T08:37:22.328 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T08:37:22.328 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T08:37:22.328 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T08:37:22.328 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T08:37:22.328 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T08:37:22.328 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T08:37:22.328 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T08:37:22.328 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T08:37:22.328 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T08:37:22.328 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T08:37:22.328 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T08:37:22.329 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T08:37:22.329 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T08:37:22.329 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T08:37:22.329 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T08:37:22.329 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T08:37:22.329 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T08:37:22.329 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T08:37:22.329 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T08:37:22.329 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T08:37:22.329 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T08:37:22.329 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T08:37:22.329 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T08:37:22.329 INFO:teuthology.orchestra.run.smithi105.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T08:37:22.329 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T08:37:22.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T08:37:22.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T08:37:22.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T08:37:22.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T08:37:22.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T08:37:22.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T08:37:22.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T08:37:22.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T08:37:22.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T08:37:22.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T08:37:22.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T08:37:22.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T08:37:22.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T08:37:22.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T08:37:22.331 INFO:teuthology.orchestra.run.smithi105.stdout:Installing weak dependencies: 2024-05-09T08:37:22.332 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T08:37:22.332 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:37:22.332 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-05-09T08:37:22.332 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-05-09T08:37:22.332 INFO:teuthology.orchestra.run.smithi105.stdout:Install 148 Packages 2024-05-09T08:37:22.332 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrade 8 Packages 2024-05-09T08:37:22.332 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:37:22.332 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 225 M 2024-05-09T08:37:22.333 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-05-09T08:37:22.974 INFO:teuthology.orchestra.run.smithi105.stdout:(1/156): ceph-19.0.0-2515.g96805dcf.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-05-09T08:37:23.700 INFO:teuthology.orchestra.run.smithi105.stdout:(2/156): ceph-fuse-19.0.0-2515.g96805dcf.el9.x8 1.2 MB/s | 889 kB 00:00 2024-05-09T08:37:23.850 INFO:teuthology.orchestra.run.smithi105.stdout:(3/156): ceph-immutable-object-cache-19.0.0-251 968 kB/s | 145 kB 00:00 2024-05-09T08:37:23.974 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:37:23.974 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:37:23.985 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T08:37:23.986 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T08:37:23.987 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T08:37:23.988 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T08:37:23.989 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T08:37:23.992 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T08:37:23.993 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T08:37:24.054 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-05-09T08:37:24.075 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-05-09T08:37:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-05-09T08:37:24.076 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-05-09T08:37:24.076 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-05-09T08:37:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T08:37:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 6.6 k 2024-05-09T08:37:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.4 M 2024-05-09T08:37:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 889 k 2024-05-09T08:37:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 145 k 2024-05-09T08:37:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 1.5 M 2024-05-09T08:37:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 142 k 2024-05-09T08:37:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 3.5 M 2024-05-09T08:37:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 7.4 M 2024-05-09T08:37:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 49 k 2024-05-09T08:37:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 10 M 2024-05-09T08:37:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 48 M 2024-05-09T08:37:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 264 k 2024-05-09T08:37:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 763 k 2024-05-09T08:37:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 222 k 2024-05-09T08:37:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 27 k 2024-05-09T08:37:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 18 k 2024-05-09T08:37:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T08:37:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T08:37:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T08:37:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 32 k 2024-05-09T08:37:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 746 k 2024-05-09T08:37:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 127 k 2024-05-09T08:37:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T08:37:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T08:37:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T08:37:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T08:37:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T08:37:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T08:37:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 162 k 2024-05-09T08:37:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 321 k 2024-05-09T08:37:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 298 k 2024-05-09T08:37:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 99 k 2024-05-09T08:37:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 85 k 2024-05-09T08:37:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:37:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 171 k 2024-05-09T08:37:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T08:37:24.080 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-05-09T08:37:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T08:37:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T08:37:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T08:37:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T08:37:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.4 M 2024-05-09T08:37:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:37:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T08:37:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T08:37:24.080 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-05-09T08:37:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T08:37:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T08:37:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T08:37:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 21 M 2024-05-09T08:37:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 32 k 2024-05-09T08:37:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 2.4 M 2024-05-09T08:37:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 249 k 2024-05-09T08:37:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 4.7 M 2024-05-09T08:37:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 17 M 2024-05-09T08:37:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 16 k 2024-05-09T08:37:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 25 k 2024-05-09T08:37:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T08:37:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T08:37:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T08:37:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T08:37:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T08:37:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T08:37:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T08:37:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T08:37:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 163 k 2024-05-09T08:37:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T08:37:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T08:37:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T08:37:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T08:37:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T08:37:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 500 k 2024-05-09T08:37:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T08:37:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.3 M 2024-05-09T08:37:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T08:37:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T08:37:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T08:37:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T08:37:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T08:37:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T08:37:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T08:37:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T08:37:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T08:37:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T08:37:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T08:37:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T08:37:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T08:37:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T08:37:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T08:37:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T08:37:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T08:37:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 45 k 2024-05-09T08:37:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 135 k 2024-05-09T08:37:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T08:37:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T08:37:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T08:37:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T08:37:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T08:37:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T08:37:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T08:37:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T08:37:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T08:37:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T08:37:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T08:37:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T08:37:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T08:37:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T08:37:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T08:37:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T08:37:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T08:37:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T08:37:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T08:37:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T08:37:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T08:37:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T08:37:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T08:37:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T08:37:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T08:37:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T08:37:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T08:37:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T08:37:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T08:37:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T08:37:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T08:37:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T08:37:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T08:37:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T08:37:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T08:37:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T08:37:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T08:37:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T08:37:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T08:37:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T08:37:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T08:37:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T08:37:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T08:37:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T08:37:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T08:37:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T08:37:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T08:37:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T08:37:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T08:37:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T08:37:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T08:37:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T08:37:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T08:37:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T08:37:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T08:37:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T08:37:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T08:37:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T08:37:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T08:37:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T08:37:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T08:37:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T08:37:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T08:37:24.089 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-05-09T08:37:24.089 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T08:37:24.089 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:37:24.089 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-05-09T08:37:24.089 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-05-09T08:37:24.089 INFO:teuthology.orchestra.run.smithi098.stdout:Install 148 Packages 2024-05-09T08:37:24.089 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 8 Packages 2024-05-09T08:37:24.089 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:37:24.090 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 225 M 2024-05-09T08:37:24.090 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-05-09T08:37:24.468 INFO:teuthology.orchestra.run.smithi142.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:37:24.469 INFO:teuthology.orchestra.run.smithi142.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:37:24.481 INFO:teuthology.orchestra.run.smithi142.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T08:37:24.482 INFO:teuthology.orchestra.run.smithi142.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T08:37:24.482 INFO:teuthology.orchestra.run.smithi142.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T08:37:24.482 INFO:teuthology.orchestra.run.smithi142.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T08:37:24.484 INFO:teuthology.orchestra.run.smithi142.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T08:37:24.486 INFO:teuthology.orchestra.run.smithi142.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T08:37:24.487 INFO:teuthology.orchestra.run.smithi142.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T08:37:24.547 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-05-09T08:37:24.561 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================= 2024-05-09T08:37:24.561 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-05-09T08:37:24.561 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================= 2024-05-09T08:37:24.561 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-05-09T08:37:24.561 INFO:teuthology.orchestra.run.smithi142.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T08:37:24.561 INFO:teuthology.orchestra.run.smithi142.stdout: ceph x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 6.6 k 2024-05-09T08:37:24.561 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.4 M 2024-05-09T08:37:24.561 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 889 k 2024-05-09T08:37:24.561 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 145 k 2024-05-09T08:37:24.561 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 1.5 M 2024-05-09T08:37:24.561 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 142 k 2024-05-09T08:37:24.562 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 3.5 M 2024-05-09T08:37:24.562 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 7.4 M 2024-05-09T08:37:24.562 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 49 k 2024-05-09T08:37:24.562 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 10 M 2024-05-09T08:37:24.562 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 48 M 2024-05-09T08:37:24.562 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-volume noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 264 k 2024-05-09T08:37:24.562 INFO:teuthology.orchestra.run.smithi142.stdout: cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 763 k 2024-05-09T08:37:24.562 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 222 k 2024-05-09T08:37:24.562 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-shell x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 27 k 2024-05-09T08:37:24.562 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-top noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 18 k 2024-05-09T08:37:24.562 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T08:37:24.562 INFO:teuthology.orchestra.run.smithi142.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T08:37:24.563 INFO:teuthology.orchestra.run.smithi142.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T08:37:24.563 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 32 k 2024-05-09T08:37:24.563 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 746 k 2024-05-09T08:37:24.563 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 127 k 2024-05-09T08:37:24.563 INFO:teuthology.orchestra.run.smithi142.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T08:37:24.563 INFO:teuthology.orchestra.run.smithi142.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T08:37:24.563 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T08:37:24.563 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T08:37:24.563 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T08:37:24.563 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T08:37:24.563 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 162 k 2024-05-09T08:37:24.563 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 321 k 2024-05-09T08:37:24.563 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 298 k 2024-05-09T08:37:24.563 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 99 k 2024-05-09T08:37:24.564 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 85 k 2024-05-09T08:37:24.564 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:37:24.564 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 171 k 2024-05-09T08:37:24.564 INFO:teuthology.orchestra.run.smithi142.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T08:37:24.564 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrading: 2024-05-09T08:37:24.564 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T08:37:24.564 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T08:37:24.564 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T08:37:24.564 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T08:37:24.564 INFO:teuthology.orchestra.run.smithi142.stdout: librados2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.4 M 2024-05-09T08:37:24.564 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:37:24.564 INFO:teuthology.orchestra.run.smithi142.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T08:37:24.564 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T08:37:24.564 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-05-09T08:37:24.565 INFO:teuthology.orchestra.run.smithi142.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T08:37:24.565 INFO:teuthology.orchestra.run.smithi142.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T08:37:24.565 INFO:teuthology.orchestra.run.smithi142.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T08:37:24.565 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 21 M 2024-05-09T08:37:24.565 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 32 k 2024-05-09T08:37:24.565 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 2.4 M 2024-05-09T08:37:24.565 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 249 k 2024-05-09T08:37:24.565 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 4.7 M 2024-05-09T08:37:24.565 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 17 M 2024-05-09T08:37:24.565 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 16 k 2024-05-09T08:37:24.565 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 25 k 2024-05-09T08:37:24.565 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T08:37:24.565 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T08:37:24.565 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T08:37:24.566 INFO:teuthology.orchestra.run.smithi142.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T08:37:24.566 INFO:teuthology.orchestra.run.smithi142.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T08:37:24.566 INFO:teuthology.orchestra.run.smithi142.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T08:37:24.566 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T08:37:24.566 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T08:37:24.566 INFO:teuthology.orchestra.run.smithi142.stdout: libcephsqlite x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 163 k 2024-05-09T08:37:24.566 INFO:teuthology.orchestra.run.smithi142.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T08:37:24.566 INFO:teuthology.orchestra.run.smithi142.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T08:37:24.566 INFO:teuthology.orchestra.run.smithi142.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T08:37:24.566 INFO:teuthology.orchestra.run.smithi142.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T08:37:24.566 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T08:37:24.566 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 500 k 2024-05-09T08:37:24.566 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T08:37:24.566 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.3 M 2024-05-09T08:37:24.567 INFO:teuthology.orchestra.run.smithi142.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T08:37:24.567 INFO:teuthology.orchestra.run.smithi142.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T08:37:24.567 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T08:37:24.567 INFO:teuthology.orchestra.run.smithi142.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T08:37:24.567 INFO:teuthology.orchestra.run.smithi142.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T08:37:24.567 INFO:teuthology.orchestra.run.smithi142.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T08:37:24.567 INFO:teuthology.orchestra.run.smithi142.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T08:37:24.567 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T08:37:24.567 INFO:teuthology.orchestra.run.smithi142.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T08:37:24.567 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T08:37:24.567 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T08:37:24.567 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T08:37:24.567 INFO:teuthology.orchestra.run.smithi142.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T08:37:24.567 INFO:teuthology.orchestra.run.smithi142.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T08:37:24.568 INFO:teuthology.orchestra.run.smithi142.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T08:37:24.568 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T08:37:24.568 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T08:37:24.568 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 45 k 2024-05-09T08:37:24.568 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 135 k 2024-05-09T08:37:24.568 INFO:teuthology.orchestra.run.smithi142.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T08:37:24.568 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T08:37:24.568 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T08:37:24.568 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T08:37:24.568 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T08:37:24.568 INFO:teuthology.orchestra.run.smithi142.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T08:37:24.568 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T08:37:24.568 INFO:teuthology.orchestra.run.smithi142.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T08:37:24.569 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T08:37:24.569 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T08:37:24.569 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T08:37:24.569 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T08:37:24.569 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T08:37:24.569 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T08:37:24.569 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T08:37:24.569 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T08:37:24.569 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T08:37:24.569 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T08:37:24.569 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T08:37:24.569 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T08:37:24.569 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T08:37:24.569 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T08:37:24.570 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T08:37:24.570 INFO:teuthology.orchestra.run.smithi142.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T08:37:24.570 INFO:teuthology.orchestra.run.smithi142.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T08:37:24.570 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T08:37:24.570 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T08:37:24.570 INFO:teuthology.orchestra.run.smithi142.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T08:37:24.570 INFO:teuthology.orchestra.run.smithi142.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T08:37:24.570 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T08:37:24.570 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T08:37:24.570 INFO:teuthology.orchestra.run.smithi142.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T08:37:24.570 INFO:teuthology.orchestra.run.smithi142.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T08:37:24.570 INFO:teuthology.orchestra.run.smithi142.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T08:37:24.570 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T08:37:24.570 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T08:37:24.571 INFO:teuthology.orchestra.run.smithi142.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T08:37:24.572 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T08:37:24.572 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T08:37:24.572 INFO:teuthology.orchestra.run.smithi142.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T08:37:24.572 INFO:teuthology.orchestra.run.smithi142.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T08:37:24.572 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T08:37:24.572 INFO:teuthology.orchestra.run.smithi142.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T08:37:24.572 INFO:teuthology.orchestra.run.smithi142.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T08:37:24.572 INFO:teuthology.orchestra.run.smithi142.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T08:37:24.572 INFO:teuthology.orchestra.run.smithi142.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T08:37:24.572 INFO:teuthology.orchestra.run.smithi142.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T08:37:24.572 INFO:teuthology.orchestra.run.smithi142.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T08:37:24.572 INFO:teuthology.orchestra.run.smithi142.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T08:37:24.572 INFO:teuthology.orchestra.run.smithi142.stdout:Installing weak dependencies: 2024-05-09T08:37:24.572 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T08:37:24.573 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:37:24.573 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-05-09T08:37:24.573 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================= 2024-05-09T08:37:24.573 INFO:teuthology.orchestra.run.smithi142.stdout:Install 148 Packages 2024-05-09T08:37:24.573 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrade 8 Packages 2024-05-09T08:37:24.573 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:37:24.573 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 225 M 2024-05-09T08:37:24.573 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-05-09T08:37:24.867 INFO:teuthology.orchestra.run.smithi105.stdout:(4/156): ceph-mds-19.0.0-2515.g96805dcf.el9.x86 2.3 MB/s | 2.4 MB 00:01 2024-05-09T08:37:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:(1/156): ceph-19.0.0-2515.g96805dcf.el9.x86_64. 13 kB/s | 6.6 kB 00:00 2024-05-09T08:37:25.176 INFO:teuthology.orchestra.run.smithi142.stdout:(1/156): ceph-19.0.0-2515.g96805dcf.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-05-09T08:37:25.684 INFO:teuthology.orchestra.run.smithi105.stdout:(5/156): ceph-mgr-19.0.0-2515.g96805dcf.el9.x86 1.8 MB/s | 1.5 MB 00:00 2024-05-09T08:37:26.041 INFO:teuthology.orchestra.run.smithi098.stdout:(2/156): ceph-fuse-19.0.0-2515.g96805dcf.el9.x8 997 kB/s | 889 kB 00:00 2024-05-09T08:37:26.224 INFO:teuthology.orchestra.run.smithi098.stdout:(3/156): ceph-immutable-object-cache-19.0.0-251 791 kB/s | 145 kB 00:00 2024-05-09T08:37:26.368 INFO:teuthology.orchestra.run.smithi142.stdout:(2/156): ceph-fuse-19.0.0-2515.g96805dcf.el9.x8 747 kB/s | 889 kB 00:01 2024-05-09T08:37:26.559 INFO:teuthology.orchestra.run.smithi105.stdout:(6/156): ceph-base-19.0.0-2515.g96805dcf.el9.x8 1.4 MB/s | 5.4 MB 00:03 2024-05-09T08:37:26.577 INFO:teuthology.orchestra.run.smithi142.stdout:(3/156): ceph-immutable-object-cache-19.0.0-251 697 kB/s | 145 kB 00:00 2024-05-09T08:37:27.609 INFO:teuthology.orchestra.run.smithi105.stdout:(7/156): ceph-common-19.0.0-2515.g96805dcf.el9. 4.3 MB/s | 21 MB 00:04 2024-05-09T08:37:27.676 INFO:teuthology.orchestra.run.smithi105.stdout:(8/156): ceph-mon-19.0.0-2515.g96805dcf.el9.x86 2.4 MB/s | 4.7 MB 00:01 2024-05-09T08:37:27.768 INFO:teuthology.orchestra.run.smithi105.stdout:(9/156): ceph-selinux-19.0.0-2515.g96805dcf.el9 274 kB/s | 25 kB 00:00 2024-05-09T08:37:28.135 INFO:teuthology.orchestra.run.smithi142.stdout:(4/156): ceph-mds-19.0.0-2515.g96805dcf.el9.x86 1.5 MB/s | 2.4 MB 00:01 2024-05-09T08:37:28.141 INFO:teuthology.orchestra.run.smithi098.stdout:(4/156): ceph-mds-19.0.0-2515.g96805dcf.el9.x86 1.2 MB/s | 2.4 MB 00:01 2024-05-09T08:37:28.502 INFO:teuthology.orchestra.run.smithi142.stdout:(5/156): ceph-base-19.0.0-2515.g96805dcf.el9.x8 1.5 MB/s | 5.4 MB 00:03 2024-05-09T08:37:29.127 INFO:teuthology.orchestra.run.smithi142.stdout:(6/156): ceph-mgr-19.0.0-2515.g96805dcf.el9.x86 1.5 MB/s | 1.5 MB 00:00 2024-05-09T08:37:29.166 INFO:teuthology.orchestra.run.smithi098.stdout:(5/156): ceph-mgr-19.0.0-2515.g96805dcf.el9.x86 1.5 MB/s | 1.5 MB 00:01 2024-05-09T08:37:29.760 INFO:teuthology.orchestra.run.smithi105.stdout:(10/156): ceph-radosgw-19.0.0-2515.g96805dcf.el 4.8 MB/s | 10 MB 00:02 2024-05-09T08:37:29.868 INFO:teuthology.orchestra.run.smithi105.stdout:(11/156): cephfs-mirror-19.0.0-2515.g96805dcf.e 2.0 MB/s | 222 kB 00:00 2024-05-09T08:37:29.960 INFO:teuthology.orchestra.run.smithi105.stdout:(12/156): cephfs-shell-19.0.0-2515.g96805dcf.el 295 kB/s | 27 kB 00:00 2024-05-09T08:37:30.052 INFO:teuthology.orchestra.run.smithi105.stdout:(13/156): libcephfs-devel-19.0.0-2515.g96805dcf 346 kB/s | 32 kB 00:00 2024-05-09T08:37:30.252 INFO:teuthology.orchestra.run.smithi142.stdout:(7/156): ceph-mon-19.0.0-2515.g96805dcf.el9.x86 2.7 MB/s | 4.7 MB 00:01 2024-05-09T08:37:30.303 INFO:teuthology.orchestra.run.smithi105.stdout:(14/156): libcephfs2-19.0.0-2515.g96805dcf.el9. 2.9 MB/s | 746 kB 00:00 2024-05-09T08:37:30.394 INFO:teuthology.orchestra.run.smithi105.stdout:(15/156): libcephsqlite-19.0.0-2515.g96805dcf.e 1.7 MB/s | 163 kB 00:00 2024-05-09T08:37:30.478 INFO:teuthology.orchestra.run.smithi105.stdout:(16/156): librados-devel-19.0.0-2515.g96805dcf. 1.5 MB/s | 127 kB 00:00 2024-05-09T08:37:30.653 INFO:teuthology.orchestra.run.smithi105.stdout:(17/156): libradosstriper1-19.0.0-2515.g96805dc 2.8 MB/s | 500 kB 00:00 2024-05-09T08:37:31.100 INFO:teuthology.orchestra.run.smithi098.stdout:(6/156): ceph-base-19.0.0-2515.g96805dcf.el9.x8 848 kB/s | 5.4 MB 00:06 2024-05-09T08:37:31.983 INFO:teuthology.orchestra.run.smithi098.stdout:(7/156): ceph-mon-19.0.0-2515.g96805dcf.el9.x86 1.7 MB/s | 4.7 MB 00:02 2024-05-09T08:37:32.062 INFO:teuthology.orchestra.run.smithi105.stdout:(18/156): librgw2-19.0.0-2515.g96805dcf.el9.x86 3.8 MB/s | 5.3 MB 00:01 2024-05-09T08:37:32.154 INFO:teuthology.orchestra.run.smithi105.stdout:(19/156): python3-ceph-argparse-19.0.0-2515.g96 493 kB/s | 45 kB 00:00 2024-05-09T08:37:32.246 INFO:teuthology.orchestra.run.smithi105.stdout:(20/156): python3-ceph-common-19.0.0-2515.g9680 1.4 MB/s | 135 kB 00:00 2024-05-09T08:37:32.346 INFO:teuthology.orchestra.run.smithi105.stdout:(21/156): python3-cephfs-19.0.0-2515.g96805dcf. 1.6 MB/s | 162 kB 00:00 2024-05-09T08:37:33.021 INFO:teuthology.orchestra.run.smithi105.stdout:(22/156): python3-rados-19.0.0-2515.g96805dcf.e 476 kB/s | 321 kB 00:00 2024-05-09T08:37:33.196 INFO:teuthology.orchestra.run.smithi105.stdout:(23/156): python3-rbd-19.0.0-2515.g96805dcf.el9 1.7 MB/s | 298 kB 00:00 2024-05-09T08:37:33.297 INFO:teuthology.orchestra.run.smithi105.stdout:(24/156): python3-rgw-19.0.0-2515.g96805dcf.el9 993 kB/s | 99 kB 00:00 2024-05-09T08:37:33.405 INFO:teuthology.orchestra.run.smithi105.stdout:(25/156): rbd-fuse-19.0.0-2515.g96805dcf.el9.x8 789 kB/s | 85 kB 00:00 2024-05-09T08:37:34.397 INFO:teuthology.orchestra.run.smithi105.stdout:(26/156): rbd-mirror-19.0.0-2515.g96805dcf.el9. 3.1 MB/s | 3.1 MB 00:00 2024-05-09T08:37:34.481 INFO:teuthology.orchestra.run.smithi105.stdout:(27/156): rbd-nbd-19.0.0-2515.g96805dcf.el9.x86 2.0 MB/s | 171 kB 00:00 2024-05-09T08:37:34.565 INFO:teuthology.orchestra.run.smithi105.stdout:(28/156): ceph-grafana-dashboards-19.0.0-2515.g 381 kB/s | 32 kB 00:00 2024-05-09T08:37:34.648 INFO:teuthology.orchestra.run.smithi105.stdout:(29/156): ceph-mgr-cephadm-19.0.0-2515.g96805dc 1.7 MB/s | 142 kB 00:00 2024-05-09T08:37:34.694 INFO:teuthology.orchestra.run.smithi142.stdout:(8/156): ceph-radosgw-19.0.0-2515.g96805dcf.el9 2.3 MB/s | 10 MB 00:04 2024-05-09T08:37:34.794 INFO:teuthology.orchestra.run.smithi142.stdout:(9/156): ceph-selinux-19.0.0-2515.g96805dcf.el9 252 kB/s | 25 kB 00:00 2024-05-09T08:37:35.690 INFO:teuthology.orchestra.run.smithi105.stdout:(30/156): ceph-mgr-dashboard-19.0.0-2515.g96805 3.4 MB/s | 3.5 MB 00:01 2024-05-09T08:37:36.032 INFO:teuthology.orchestra.run.smithi105.stdout:(31/156): ceph-osd-19.0.0-2515.g96805dcf.el9.x8 1.8 MB/s | 17 MB 00:09 2024-05-09T08:37:36.232 INFO:teuthology.orchestra.run.smithi105.stdout:(32/156): ceph-mgr-modules-core-19.0.0-2515.g96 1.2 MB/s | 249 kB 00:00 2024-05-09T08:37:36.349 INFO:teuthology.orchestra.run.smithi105.stdout:(33/156): ceph-mgr-rook-19.0.0-2515.g96805dcf.e 423 kB/s | 49 kB 00:00 2024-05-09T08:37:36.457 INFO:teuthology.orchestra.run.smithi105.stdout:(34/156): ceph-prometheus-alerts-19.0.0-2515.g9 152 kB/s | 16 kB 00:00 2024-05-09T08:37:36.691 INFO:teuthology.orchestra.run.smithi105.stdout:(35/156): ceph-volume-19.0.0-2515.g96805dcf.el9 1.1 MB/s | 264 kB 00:00 2024-05-09T08:37:36.850 INFO:teuthology.orchestra.run.smithi098.stdout:(8/156): ceph-common-19.0.0-2515.g96805dcf.el9. 1.7 MB/s | 21 MB 00:12 2024-05-09T08:37:36.925 INFO:teuthology.orchestra.run.smithi098.stdout:(9/156): ceph-selinux-19.0.0-2515.g96805dcf.el9 337 kB/s | 25 kB 00:00 2024-05-09T08:37:37.100 INFO:teuthology.orchestra.run.smithi105.stdout:(36/156): cephadm-19.0.0-2515.g96805dcf.el9.noa 1.8 MB/s | 763 kB 00:00 2024-05-09T08:37:37.209 INFO:teuthology.orchestra.run.smithi105.stdout:(37/156): cephfs-top-19.0.0-2515.g96805dcf.el9. 168 kB/s | 18 kB 00:00 2024-05-09T08:37:37.259 INFO:teuthology.orchestra.run.smithi098.stdout:(10/156): ceph-radosgw-19.0.0-2515.g96805dcf.el 1.9 MB/s | 10 MB 00:05 2024-05-09T08:37:37.409 INFO:teuthology.orchestra.run.smithi098.stdout:(11/156): cephfs-mirror-19.0.0-2515.g96805dcf.e 1.4 MB/s | 222 kB 00:00 2024-05-09T08:37:37.493 INFO:teuthology.orchestra.run.smithi098.stdout:(12/156): cephfs-shell-19.0.0-2515.g96805dcf.el 324 kB/s | 27 kB 00:00 2024-05-09T08:37:37.576 INFO:teuthology.orchestra.run.smithi098.stdout:(13/156): libcephfs-devel-19.0.0-2515.g96805dcf 381 kB/s | 32 kB 00:00 2024-05-09T08:37:37.659 INFO:teuthology.orchestra.run.smithi105.stdout:(38/156): ceph-mgr-diskprediction-local-19.0.0- 3.8 MB/s | 7.4 MB 00:01 2024-05-09T08:37:37.876 INFO:teuthology.orchestra.run.smithi098.stdout:(14/156): libcephfs2-19.0.0-2515.g96805dcf.el9. 2.4 MB/s | 746 kB 00:00 2024-05-09T08:37:37.943 INFO:teuthology.orchestra.run.smithi105.stdout:(39/156): fuse-2.9.9-15.el9.x86_64.rpm 109 kB/s | 80 kB 00:00 2024-05-09T08:37:38.010 INFO:teuthology.orchestra.run.smithi098.stdout:(15/156): libcephsqlite-19.0.0-2515.g96805dcf.e 1.2 MB/s | 163 kB 00:00 2024-05-09T08:37:38.027 INFO:teuthology.orchestra.run.smithi142.stdout:(10/156): ceph-osd-19.0.0-2515.g96805dcf.el9.x8 1.9 MB/s | 17 MB 00:08 2024-05-09T08:37:38.086 INFO:teuthology.orchestra.run.smithi098.stdout:(16/156): librados-devel-19.0.0-2515.g96805dcf. 1.7 MB/s | 127 kB 00:00 2024-05-09T08:37:38.194 INFO:teuthology.orchestra.run.smithi142.stdout:(11/156): cephfs-mirror-19.0.0-2515.g96805dcf.e 1.3 MB/s | 222 kB 00:00 2024-05-09T08:37:38.303 INFO:teuthology.orchestra.run.smithi142.stdout:(12/156): cephfs-shell-19.0.0-2515.g96805dcf.el 250 kB/s | 27 kB 00:00 2024-05-09T08:37:38.319 INFO:teuthology.orchestra.run.smithi098.stdout:(17/156): libradosstriper1-19.0.0-2515.g96805dc 2.1 MB/s | 500 kB 00:00 2024-05-09T08:37:38.395 INFO:teuthology.orchestra.run.smithi142.stdout:(13/156): libcephfs-devel-19.0.0-2515.g96805dcf 347 kB/s | 32 kB 00:00 2024-05-09T08:37:38.635 INFO:teuthology.orchestra.run.smithi105.stdout:(40/156): libgfortran-11.4.1-3.el9.x86_64.rpm 824 kB/s | 803 kB 00:00 2024-05-09T08:37:38.735 INFO:teuthology.orchestra.run.smithi105.stdout:(41/156): libquadmath-11.4.1-3.el9.x86_64.rpm 247 kB/s | 195 kB 00:00 2024-05-09T08:37:38.795 INFO:teuthology.orchestra.run.smithi142.stdout:(14/156): libcephfs2-19.0.0-2515.g96805dcf.el9. 1.8 MB/s | 746 kB 00:00 2024-05-09T08:37:38.954 INFO:teuthology.orchestra.run.smithi142.stdout:(15/156): libcephsqlite-19.0.0-2515.g96805dcf.e 1.0 MB/s | 163 kB 00:00 2024-05-09T08:37:39.054 INFO:teuthology.orchestra.run.smithi142.stdout:(16/156): librados-devel-19.0.0-2515.g96805dcf. 1.2 MB/s | 127 kB 00:00 2024-05-09T08:37:39.379 INFO:teuthology.orchestra.run.smithi142.stdout:(17/156): libradosstriper1-19.0.0-2515.g96805dc 1.5 MB/s | 500 kB 00:00 2024-05-09T08:37:39.468 INFO:teuthology.orchestra.run.smithi105.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 304 kB/s | 253 kB 00:00 2024-05-09T08:37:39.710 INFO:teuthology.orchestra.run.smithi105.stdout:(43/156): python3-cryptography-36.0.1-4.el9.x86 1.3 MB/s | 1.2 MB 00:00 2024-05-09T08:37:40.211 INFO:teuthology.orchestra.run.smithi105.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 144 kB/s | 106 kB 00:00 2024-05-09T08:37:40.524 INFO:teuthology.orchestra.run.smithi105.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 172 kB/s | 135 kB 00:00 2024-05-09T08:37:40.553 INFO:teuthology.orchestra.run.smithi098.stdout:(18/156): librgw2-19.0.0-2515.g96805dcf.el9.x86 2.4 MB/s | 5.3 MB 00:02 2024-05-09T08:37:40.644 INFO:teuthology.orchestra.run.smithi098.stdout:(19/156): python3-ceph-argparse-19.0.0-2515.g96 494 kB/s | 45 kB 00:00 2024-05-09T08:37:40.745 INFO:teuthology.orchestra.run.smithi098.stdout:(20/156): python3-ceph-common-19.0.0-2515.g9680 1.3 MB/s | 135 kB 00:00 2024-05-09T08:37:40.846 INFO:teuthology.orchestra.run.smithi098.stdout:(21/156): python3-cephfs-19.0.0-2515.g96805dcf. 1.6 MB/s | 162 kB 00:00 2024-05-09T08:37:40.953 INFO:teuthology.orchestra.run.smithi105.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 168 kB/s | 125 kB 00:00 2024-05-09T08:37:41.262 INFO:teuthology.orchestra.run.smithi105.stdout:(47/156): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 689 kB 00:00 2024-05-09T08:37:41.304 INFO:teuthology.orchestra.run.smithi105.stdout:(48/156): python3-urllib3-1.26.5-5.el9.noarch.r 276 kB/s | 215 kB 00:00 2024-05-09T08:37:41.412 INFO:teuthology.orchestra.run.smithi105.stdout:(49/156): bison-3.7.4-5.el9.x86_64.rpm 6.1 MB/s | 943 kB 00:00 2024-05-09T08:37:41.429 INFO:teuthology.orchestra.run.smithi098.stdout:(22/156): ceph-osd-19.0.0-2515.g96805dcf.el9.x8 1.6 MB/s | 17 MB 00:10 2024-05-09T08:37:41.462 INFO:teuthology.orchestra.run.smithi105.stdout:(50/156): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-05-09T08:37:41.463 INFO:teuthology.orchestra.run.smithi098.stdout:(23/156): python3-rados-19.0.0-2515.g96805dcf.e 521 kB/s | 321 kB 00:00 2024-05-09T08:37:41.488 INFO:teuthology.orchestra.run.smithi105.stdout:(51/156): boost-program-options-1.75.0-8.el9.x8 582 kB/s | 107 kB 00:00 2024-05-09T08:37:41.521 INFO:teuthology.orchestra.run.smithi105.stdout:(52/156): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-05-09T08:37:41.538 INFO:teuthology.orchestra.run.smithi105.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 651 kB/s | 33 kB 00:00 2024-05-09T08:37:41.554 INFO:teuthology.orchestra.run.smithi098.stdout:(24/156): python3-rgw-19.0.0-2515.g96805dcf.el9 1.1 MB/s | 99 kB 00:00 2024-05-09T08:37:41.588 INFO:teuthology.orchestra.run.smithi105.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 364 kB/s | 18 kB 00:00 2024-05-09T08:37:41.639 INFO:teuthology.orchestra.run.smithi105.stdout:(55/156): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-05-09T08:37:41.655 INFO:teuthology.orchestra.run.smithi098.stdout:(25/156): rbd-fuse-19.0.0-2515.g96805dcf.el9.x8 853 kB/s | 85 kB 00:00 2024-05-09T08:37:41.681 INFO:teuthology.orchestra.run.smithi098.stdout:(26/156): python3-rbd-19.0.0-2515.g96805dcf.el9 1.2 MB/s | 298 kB 00:00 2024-05-09T08:37:41.731 INFO:teuthology.orchestra.run.smithi105.stdout:(56/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-05-09T08:37:41.789 INFO:teuthology.orchestra.run.smithi105.stdout:(57/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 778 kB/s | 45 kB 00:00 2024-05-09T08:37:41.848 INFO:teuthology.orchestra.run.smithi105.stdout:(58/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.2 MB/s | 3.0 MB 00:00 2024-05-09T08:37:41.864 INFO:teuthology.orchestra.run.smithi098.stdout:(27/156): rbd-nbd-19.0.0-2515.g96805dcf.el9.x86 935 kB/s | 171 kB 00:00 2024-05-09T08:37:41.940 INFO:teuthology.orchestra.run.smithi105.stdout:(59/156): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 582 kB 00:00 2024-05-09T08:37:41.956 INFO:teuthology.orchestra.run.smithi098.stdout:(28/156): ceph-grafana-dashboards-19.0.0-2515.g 347 kB/s | 32 kB 00:00 2024-05-09T08:37:41.999 INFO:teuthology.orchestra.run.smithi105.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-05-09T08:37:42.024 INFO:teuthology.orchestra.run.smithi105.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 305 kB/s | 25 kB 00:00 2024-05-09T08:37:42.207 INFO:teuthology.orchestra.run.smithi105.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 227 kB/s | 47 kB 00:00 2024-05-09T08:37:42.214 INFO:teuthology.orchestra.run.smithi098.stdout:(29/156): ceph-mgr-cephadm-19.0.0-2515.g96805dc 549 kB/s | 142 kB 00:00 2024-05-09T08:37:42.233 INFO:teuthology.orchestra.run.smithi105.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-05-09T08:37:42.283 INFO:teuthology.orchestra.run.smithi105.stdout:(64/156): openblas-0.3.21-2.el9.x86_64.rpm 702 kB/s | 35 kB 00:00 2024-05-09T08:37:42.302 INFO:teuthology.orchestra.run.smithi105.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-05-09T08:37:42.609 INFO:teuthology.orchestra.run.smithi105.stdout:(66/156): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-05-09T08:37:42.743 INFO:teuthology.orchestra.run.smithi105.stdout:(67/156): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-05-09T08:37:42.843 INFO:teuthology.orchestra.run.smithi105.stdout:(68/156): postgresql-contrib-13.14-1.el9.x86_64 8.9 MB/s | 911 kB 00:00 2024-05-09T08:37:42.901 INFO:teuthology.orchestra.run.smithi105.stdout:(69/156): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-05-09T08:37:43.027 INFO:teuthology.orchestra.run.smithi105.stdout:(70/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 6.1 MB/s | 4.4 MB 00:00 2024-05-09T08:37:43.169 INFO:teuthology.orchestra.run.smithi105.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-05-09T08:37:43.244 INFO:teuthology.orchestra.run.smithi105.stdout:(72/156): postgresql-server-13.14-1.el9.x86_64. 17 MB/s | 5.8 MB 00:00 2024-05-09T08:37:43.269 INFO:teuthology.orchestra.run.smithi105.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 873 kB/s | 87 kB 00:00 2024-05-09T08:37:43.328 INFO:teuthology.orchestra.run.smithi105.stdout:(74/156): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-05-09T08:37:43.395 INFO:teuthology.orchestra.run.smithi105.stdout:(75/156): python3-jinja2-2.11.3-5.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-05-09T08:37:43.445 INFO:teuthology.orchestra.run.smithi105.stdout:(76/156): python3-jsonpatch-1.21-16.el9.noarch. 514 kB/s | 26 kB 00:00 2024-05-09T08:37:43.496 INFO:teuthology.orchestra.run.smithi105.stdout:(77/156): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-05-09T08:37:43.579 INFO:teuthology.orchestra.run.smithi105.stdout:(78/156): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-05-09T08:37:43.604 INFO:teuthology.orchestra.run.smithi105.stdout:(79/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-05-09T08:37:43.629 INFO:teuthology.orchestra.run.smithi105.stdout:(80/156): python3-markupsafe-1.1.1-12.el9.x86_6 693 kB/s | 35 kB 00:00 2024-05-09T08:37:43.689 INFO:teuthology.orchestra.run.smithi105.stdout:(81/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-05-09T08:37:43.747 INFO:teuthology.orchestra.run.smithi105.stdout:(82/156): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-05-09T08:37:43.806 INFO:teuthology.orchestra.run.smithi105.stdout:(83/156): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-05-09T08:37:43.839 INFO:teuthology.orchestra.run.smithi098.stdout:(30/156): rbd-mirror-19.0.0-2515.g96805dcf.el9. 1.4 MB/s | 3.1 MB 00:02 2024-05-09T08:37:43.856 INFO:teuthology.orchestra.run.smithi105.stdout:(84/156): python3-prettytable-0.7.2-27.el9.noar 847 kB/s | 42 kB 00:00 2024-05-09T08:37:43.914 INFO:teuthology.orchestra.run.smithi105.stdout:(85/156): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-05-09T08:37:43.990 INFO:teuthology.orchestra.run.smithi105.stdout:(86/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-05-09T08:37:44.082 INFO:teuthology.orchestra.run.smithi105.stdout:(87/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-05-09T08:37:44.115 INFO:teuthology.orchestra.run.smithi105.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 2.2 MB/s | 279 kB 00:00 2024-05-09T08:37:44.141 INFO:teuthology.orchestra.run.smithi105.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 871 kB/s | 51 kB 00:00 2024-05-09T08:37:44.174 INFO:teuthology.orchestra.run.smithi105.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 919 kB/s | 54 kB 00:00 2024-05-09T08:37:44.233 INFO:teuthology.orchestra.run.smithi105.stdout:(91/156): python3-toml-0.10.2-6.el9.noarch.rpm 714 kB/s | 42 kB 00:00 2024-05-09T08:37:44.300 INFO:teuthology.orchestra.run.smithi105.stdout:(92/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 662 kB/s | 44 kB 00:00 2024-05-09T08:37:44.329 INFO:teuthology.orchestra.run.smithi142.stdout:(18/156): librgw2-19.0.0-2515.g96805dcf.el9.x86 1.1 MB/s | 5.3 MB 00:04 2024-05-09T08:37:44.359 INFO:teuthology.orchestra.run.smithi105.stdout:(93/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-05-09T08:37:44.407 INFO:teuthology.orchestra.run.smithi098.stdout:(31/156): ceph-mgr-dashboard-19.0.0-2515.g96805 1.6 MB/s | 3.5 MB 00:02 2024-05-09T08:37:44.417 INFO:teuthology.orchestra.run.smithi105.stdout:(94/156): qatlib-service-24.02.0-1.el9.x86_64.r 616 kB/s | 36 kB 00:00 2024-05-09T08:37:44.454 INFO:teuthology.orchestra.run.smithi142.stdout:(19/156): python3-ceph-argparse-19.0.0-2515.g96 362 kB/s | 45 kB 00:00 2024-05-09T08:37:44.476 INFO:teuthology.orchestra.run.smithi105.stdout:(95/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 813 kB/s | 47 kB 00:00 2024-05-09T08:37:44.547 INFO:teuthology.orchestra.run.smithi105.stdout:(96/156): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-05-09T08:37:44.602 INFO:teuthology.orchestra.run.smithi105.stdout:(97/156): uuid-1.6.2-55.el9.x86_64.rpm 990 kB/s | 58 kB 00:00 2024-05-09T08:37:44.694 INFO:teuthology.orchestra.run.smithi105.stdout:(98/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.0 MB/s | 94 kB 00:00 2024-05-09T08:37:44.738 INFO:teuthology.orchestra.run.smithi142.stdout:(20/156): python3-ceph-common-19.0.0-2515.g9680 477 kB/s | 135 kB 00:00 2024-05-09T08:37:44.752 INFO:teuthology.orchestra.run.smithi105.stdout:(99/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-05-09T08:37:44.811 INFO:teuthology.orchestra.run.smithi105.stdout:(100/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 767 kB/s | 45 kB 00:00 2024-05-09T08:37:44.869 INFO:teuthology.orchestra.run.smithi105.stdout:(101/156): lua-devel-5.4.4-4.el9.x86_64.rpm 384 kB/s | 22 kB 00:00 2024-05-09T08:37:44.898 INFO:teuthology.orchestra.run.smithi098.stdout:(32/156): ceph-mgr-modules-core-19.0.0-2515.g96 506 kB/s | 249 kB 00:00 2024-05-09T08:37:44.999 INFO:teuthology.orchestra.run.smithi098.stdout:(33/156): ceph-mgr-rook-19.0.0-2515.g96805dcf.e 494 kB/s | 49 kB 00:00 2024-05-09T08:37:45.061 INFO:teuthology.orchestra.run.smithi105.stdout:(102/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-05-09T08:37:45.071 INFO:teuthology.orchestra.run.smithi142.stdout:(21/156): python3-cephfs-19.0.0-2515.g96805dcf. 486 kB/s | 162 kB 00:00 2024-05-09T08:37:45.099 INFO:teuthology.orchestra.run.smithi098.stdout:(34/156): ceph-prometheus-alerts-19.0.0-2515.g9 165 kB/s | 16 kB 00:00 2024-05-09T08:37:45.120 INFO:teuthology.orchestra.run.smithi105.stdout:(103/156): protobuf-compiler-3.14.0-13.el9.x86_ 3.4 MB/s | 863 kB 00:00 2024-05-09T08:37:45.420 INFO:teuthology.orchestra.run.smithi105.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 1.5 MB/s | 548 kB 00:00 2024-05-09T08:37:45.432 INFO:teuthology.orchestra.run.smithi098.stdout:(35/156): ceph-volume-19.0.0-2515.g96805dcf.el9 791 kB/s | 264 kB 00:00 2024-05-09T08:37:45.445 INFO:teuthology.orchestra.run.smithi105.stdout:(105/156): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-05-09T08:37:45.479 INFO:teuthology.orchestra.run.smithi105.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 332 kB/s | 19 kB 00:00 2024-05-09T08:37:45.538 INFO:teuthology.orchestra.run.smithi105.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 443 kB/s | 26 kB 00:00 2024-05-09T08:37:45.596 INFO:teuthology.orchestra.run.smithi105.stdout:(108/156): liboath-2.6.7-2.el9.x86_64.rpm 835 kB/s | 49 kB 00:00 2024-05-09T08:37:45.621 INFO:teuthology.orchestra.run.smithi142.stdout:(22/156): python3-rados-19.0.0-2515.g96805dcf.e 584 kB/s | 321 kB 00:00 2024-05-09T08:37:45.647 INFO:teuthology.orchestra.run.smithi105.stdout:(109/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-05-09T08:37:45.705 INFO:teuthology.orchestra.run.smithi105.stdout:(110/156): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-05-09T08:37:45.839 INFO:teuthology.orchestra.run.smithi105.stdout:(111/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.2 MB/s | 839 kB 00:00 2024-05-09T08:37:45.914 INFO:teuthology.orchestra.run.smithi105.stdout:(112/156): libarrow-9.0.0-11.el9.x86_64.rpm 9.4 MB/s | 4.4 MB 00:00 2024-05-09T08:37:45.973 INFO:teuthology.orchestra.run.smithi105.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-05-09T08:37:45.998 INFO:teuthology.orchestra.run.smithi105.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 521 kB/s | 43 kB 00:00 2024-05-09T08:37:46.032 INFO:teuthology.orchestra.run.smithi105.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 550 kB/s | 32 kB 00:00 2024-05-09T08:37:46.049 INFO:teuthology.orchestra.run.smithi105.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-05-09T08:37:46.091 INFO:teuthology.orchestra.run.smithi105.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-05-09T08:37:46.124 INFO:teuthology.orchestra.run.smithi105.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-05-09T08:37:46.150 INFO:teuthology.orchestra.run.smithi105.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-05-09T08:37:46.170 INFO:teuthology.orchestra.run.smithi105.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 835 kB/s | 35 kB 00:00 2024-05-09T08:37:46.209 INFO:teuthology.orchestra.run.smithi105.stdout:(121/156): python3-google-auth-2.29.0-1.el9.noa 3.4 MB/s | 206 kB 00:00 2024-05-09T08:37:46.267 INFO:teuthology.orchestra.run.smithi105.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-05-09T08:37:46.283 INFO:teuthology.orchestra.run.smithi098.stdout:(36/156): cephadm-19.0.0-2515.g96805dcf.el9.noa 897 kB/s | 763 kB 00:00 2024-05-09T08:37:46.309 INFO:teuthology.orchestra.run.smithi105.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 257 kB/s | 11 kB 00:00 2024-05-09T08:37:46.355 INFO:teuthology.orchestra.run.smithi142.stdout:(23/156): python3-rbd-19.0.0-2515.g96805dcf.el9 407 kB/s | 298 kB 00:00 2024-05-09T08:37:46.359 INFO:teuthology.orchestra.run.smithi105.stdout:(124/156): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-05-09T08:37:46.377 INFO:teuthology.orchestra.run.smithi105.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 264 kB/s | 18 kB 00:00 2024-05-09T08:37:46.383 INFO:teuthology.orchestra.run.smithi098.stdout:(37/156): cephfs-top-19.0.0-2515.g96805dcf.el9. 182 kB/s | 18 kB 00:00 2024-05-09T08:37:46.418 INFO:teuthology.orchestra.run.smithi105.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 396 kB/s | 23 kB 00:00 2024-05-09T08:37:46.444 INFO:teuthology.orchestra.run.smithi105.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 290 kB/s | 19 kB 00:00 2024-05-09T08:37:46.477 INFO:teuthology.orchestra.run.smithi105.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 337 kB/s | 20 kB 00:00 2024-05-09T08:37:46.492 INFO:teuthology.orchestra.run.smithi098.stdout:(38/156): fuse-2.9.9-15.el9.x86_64.rpm 743 kB/s | 80 kB 00:00 2024-05-09T08:37:46.536 INFO:teuthology.orchestra.run.smithi105.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 794 kB/s | 46 kB 00:00 2024-05-09T08:37:46.595 INFO:teuthology.orchestra.run.smithi105.stdout:(130/156): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-05-09T08:37:46.600 INFO:teuthology.orchestra.run.smithi098.stdout:(39/156): libgfortran-11.4.1-3.el9.x86_64.rpm 7.2 MB/s | 803 kB 00:00 2024-05-09T08:37:46.620 INFO:teuthology.orchestra.run.smithi105.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 946 kB/s | 79 kB 00:00 2024-05-09T08:37:46.638 INFO:teuthology.orchestra.run.smithi142.stdout:(24/156): python3-rgw-19.0.0-2515.g96805dcf.el9 351 kB/s | 99 kB 00:00 2024-05-09T08:37:46.650 INFO:teuthology.orchestra.run.smithi098.stdout:(40/156): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-05-09T08:37:46.701 INFO:teuthology.orchestra.run.smithi098.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-05-09T08:37:46.703 INFO:teuthology.orchestra.run.smithi105.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 531 kB/s | 58 kB 00:00 2024-05-09T08:37:46.729 INFO:teuthology.orchestra.run.smithi105.stdout:(133/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-05-09T08:37:46.762 INFO:teuthology.orchestra.run.smithi105.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-05-09T08:37:46.777 INFO:teuthology.orchestra.run.smithi098.stdout:(42/156): python3-cryptography-36.0.1-4.el9.x86 16 MB/s | 1.2 MB 00:00 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi105.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-05-09T08:37:46.822 INFO:teuthology.orchestra.run.smithi105.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-05-09T08:37:46.827 INFO:teuthology.orchestra.run.smithi098.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-05-09T08:37:46.839 INFO:teuthology.orchestra.run.smithi142.stdout:(25/156): rbd-fuse-19.0.0-2515.g96805dcf.el9.x8 426 kB/s | 85 kB 00:00 2024-05-09T08:37:46.846 INFO:teuthology.orchestra.run.smithi105.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 525 kB/s | 31 kB 00:00 2024-05-09T08:37:46.877 INFO:teuthology.orchestra.run.smithi098.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-05-09T08:37:46.880 INFO:teuthology.orchestra.run.smithi105.stdout:(138/156): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-05-09T08:37:46.906 INFO:teuthology.orchestra.run.smithi105.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-05-09T08:37:46.927 INFO:teuthology.orchestra.run.smithi098.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-05-09T08:37:46.939 INFO:teuthology.orchestra.run.smithi105.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 609 kB/s | 36 kB 00:00 2024-05-09T08:37:46.964 INFO:teuthology.orchestra.run.smithi105.stdout:(141/156): python3-typing-extensions-4.4.0-2.el 874 kB/s | 51 kB 00:00 2024-05-09T08:37:46.977 INFO:teuthology.orchestra.run.smithi098.stdout:(46/156): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-05-09T08:37:46.998 INFO:teuthology.orchestra.run.smithi105.stdout:(142/156): python3-webob-1.8.7-6.el9.noarch.rpm 3.9 MB/s | 230 kB 00:00 2024-05-09T08:37:47.023 INFO:teuthology.orchestra.run.smithi105.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-05-09T08:37:47.057 INFO:teuthology.orchestra.run.smithi105.stdout:(144/156): python3-werkzeug-2.0.3-3.el9.1.noarc 7.1 MB/s | 427 kB 00:00 2024-05-09T08:37:47.082 INFO:teuthology.orchestra.run.smithi105.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 341 kB/s | 20 kB 00:00 2024-05-09T08:37:47.116 INFO:teuthology.orchestra.run.smithi105.stdout:(146/156): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-05-09T08:37:47.275 INFO:teuthology.orchestra.run.smithi105.stdout:(147/156): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-05-09T08:37:47.400 INFO:teuthology.orchestra.run.smithi142.stdout:(26/156): ceph-common-19.0.0-2515.g96805dcf.el9 952 kB/s | 21 MB 00:22 2024-05-09T08:37:47.636 INFO:teuthology.orchestra.run.smithi098.stdout:(47/156): automake-1.16.2-8.el9.noarch.rpm 1.0 MB/s | 689 kB 00:00 2024-05-09T08:37:47.698 INFO:teuthology.orchestra.run.smithi142.stdout:(27/156): rbd-nbd-19.0.0-2515.g96805dcf.el9.x86 576 kB/s | 171 kB 00:00 2024-05-09T08:37:47.806 INFO:teuthology.orchestra.run.smithi142.stdout:(28/156): ceph-grafana-dashboards-19.0.0-2515.g 293 kB/s | 32 kB 00:00 2024-05-09T08:37:47.845 INFO:teuthology.orchestra.run.smithi098.stdout:(48/156): bison-3.7.4-5.el9.x86_64.rpm 4.4 MB/s | 943 kB 00:00 2024-05-09T08:37:47.962 INFO:teuthology.orchestra.run.smithi098.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 916 kB/s | 107 kB 00:00 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi142.stdout:(29/156): ceph-mgr-cephadm-19.0.0-2515.g96805dc 812 kB/s | 142 kB 00:00 2024-05-09T08:37:48.063 INFO:teuthology.orchestra.run.smithi098.stdout:(50/156): elfutils-libelf-devel-0.190-2.el9.x86 215 kB/s | 21 kB 00:00 2024-05-09T08:37:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:(51/156): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-05-09T08:37:48.272 INFO:teuthology.orchestra.run.smithi098.stdout:(52/156): flexiblas-3.0.4-8.el9.x86_64.rpm 327 kB/s | 33 kB 00:00 2024-05-09T08:37:48.588 INFO:teuthology.orchestra.run.smithi098.stdout:(53/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.4 MB/s | 3.0 MB 00:00 2024-05-09T08:37:48.697 INFO:teuthology.orchestra.run.smithi098.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 168 kB/s | 18 kB 00:00 2024-05-09T08:37:48.797 INFO:teuthology.orchestra.run.smithi098.stdout:(55/156): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-05-09T08:37:48.906 INFO:teuthology.orchestra.run.smithi098.stdout:(56/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-05-09T08:37:49.014 INFO:teuthology.orchestra.run.smithi098.stdout:(57/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-05-09T08:37:49.132 INFO:teuthology.orchestra.run.smithi098.stdout:(58/156): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-05-09T08:37:49.249 INFO:teuthology.orchestra.run.smithi098.stdout:(59/156): ceph-mgr-diskprediction-local-19.0.0- 1.4 MB/s | 7.4 MB 00:05 2024-05-09T08:37:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:(60/156): libtool-2.4.6-45.el9.x86_64.rpm 4.0 MB/s | 582 kB 00:00 2024-05-09T08:37:49.367 INFO:teuthology.orchestra.run.smithi105.stdout:(148/156): librados2-19.0.0-2515.g96805dcf.el9. 1.5 MB/s | 3.4 MB 00:02 2024-05-09T08:37:49.374 INFO:teuthology.orchestra.run.smithi098.stdout:(61/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 474 kB/s | 47 kB 00:00 2024-05-09T08:37:49.483 INFO:teuthology.orchestra.run.smithi098.stdout:(62/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-05-09T08:37:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:(63/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 95 kB/s | 25 kB 00:00 2024-05-09T08:37:49.592 INFO:teuthology.orchestra.run.smithi098.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-05-09T08:37:49.709 INFO:teuthology.orchestra.run.smithi098.stdout:(65/156): openblas-0.3.21-2.el9.x86_64.rpm 183 kB/s | 35 kB 00:00 2024-05-09T08:37:50.467 INFO:teuthology.orchestra.run.smithi105.stdout:(149/156): NetworkManager-1.47.5-1.el9.x86_64.r 2.1 MB/s | 2.3 MB 00:01 2024-05-09T08:37:50.592 INFO:teuthology.orchestra.run.smithi105.stdout:(150/156): librbd1-19.0.0-2515.g96805dcf.el9.x8 968 kB/s | 3.1 MB 00:03 2024-05-09T08:37:50.684 INFO:teuthology.orchestra.run.smithi098.stdout:(66/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 4.5 MB/s | 4.4 MB 00:00 2024-05-09T08:37:51.115 INFO:teuthology.orchestra.run.smithi142.stdout:(30/156): rbd-mirror-19.0.0-2515.g96805dcf.el9. 740 kB/s | 3.1 MB 00:04 2024-05-09T08:37:51.165 INFO:teuthology.orchestra.run.smithi142.stdout:(31/156): ceph-mgr-dashboard-19.0.0-2515.g96805 1.1 MB/s | 3.5 MB 00:03 2024-05-09T08:37:51.242 INFO:teuthology.orchestra.run.smithi105.stdout:(151/156): ceph-test-19.0.0-2515.g96805dcf.el9. 2.1 MB/s | 48 MB 00:23 2024-05-09T08:37:51.441 INFO:teuthology.orchestra.run.smithi142.stdout:(32/156): ceph-mgr-modules-core-19.0.0-2515.g96 905 kB/s | 249 kB 00:00 2024-05-09T08:37:51.549 INFO:teuthology.orchestra.run.smithi142.stdout:(33/156): ceph-mgr-rook-19.0.0-2515.g96805dcf.e 456 kB/s | 49 kB 00:00 2024-05-09T08:37:51.657 INFO:teuthology.orchestra.run.smithi142.stdout:(34/156): ceph-prometheus-alerts-19.0.0-2515.g9 152 kB/s | 16 kB 00:00 2024-05-09T08:37:51.692 INFO:teuthology.orchestra.run.smithi105.stdout:(152/156): NetworkManager-team-1.47.5-1.el9.x86 38 kB/s | 42 kB 00:01 2024-05-09T08:37:51.916 INFO:teuthology.orchestra.run.smithi142.stdout:(35/156): ceph-volume-19.0.0-2515.g96805dcf.el9 1.0 MB/s | 264 kB 00:00 2024-05-09T08:37:51.951 INFO:teuthology.orchestra.run.smithi105.stdout:(153/156): NetworkManager-libnm-1.47.5-1.el9.x8 1.2 MB/s | 1.8 MB 00:01 2024-05-09T08:37:52.101 INFO:teuthology.orchestra.run.smithi105.stdout:(154/156): NetworkManager-tui-1.47.5-1.el9.x86_ 299 kB/s | 248 kB 00:00 2024-05-09T08:37:52.550 INFO:teuthology.orchestra.run.smithi142.stdout:(36/156): cephadm-19.0.0-2515.g96805dcf.el9.noa 1.2 MB/s | 763 kB 00:00 2024-05-09T08:37:52.683 INFO:teuthology.orchestra.run.smithi142.stdout:(37/156): cephfs-top-19.0.0-2515.g96805dcf.el9. 137 kB/s | 18 kB 00:00 2024-05-09T08:37:52.742 INFO:teuthology.orchestra.run.smithi142.stdout:(38/156): fuse-2.9.9-15.el9.x86_64.rpm 1.4 MB/s | 80 kB 00:00 2024-05-09T08:37:52.743 INFO:teuthology.orchestra.run.smithi105.stdout:(155/156): openssl-3.2.1-1.el9.x86_64.rpm 1.3 MB/s | 1.3 MB 00:01 2024-05-09T08:37:52.834 INFO:teuthology.orchestra.run.smithi098.stdout:(67/156): postgresql-13.14-1.el9.x86_64.rpm 756 kB/s | 1.6 MB 00:02 2024-05-09T08:37:53.027 INFO:teuthology.orchestra.run.smithi105.stdout:(156/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.3 MB/s | 2.4 MB 00:01 2024-05-09T08:37:53.033 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:37:53.033 INFO:teuthology.orchestra.run.smithi105.stdout:Total 7.3 MB/s | 225 MB 00:30 2024-05-09T08:37:53.067 INFO:teuthology.orchestra.run.smithi142.stdout:(39/156): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-05-09T08:37:53.142 INFO:teuthology.orchestra.run.smithi142.stdout:(40/156): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-05-09T08:37:53.218 INFO:teuthology.orchestra.run.smithi142.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-05-09T08:37:53.326 INFO:teuthology.orchestra.run.smithi142.stdout:(42/156): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-05-09T08:37:53.393 INFO:teuthology.orchestra.run.smithi142.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-05-09T08:37:53.460 INFO:teuthology.orchestra.run.smithi142.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-05-09T08:37:53.527 INFO:teuthology.orchestra.run.smithi142.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-05-09T08:37:53.603 INFO:teuthology.orchestra.run.smithi142.stdout:(46/156): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-05-09T08:37:53.618 INFO:teuthology.orchestra.run.smithi098.stdout:(68/156): openblas-openmp-0.3.21-2.el9.x86_64.r 1.2 MB/s | 4.7 MB 00:04 2024-05-09T08:37:53.869 INFO:teuthology.orchestra.run.smithi142.stdout:(47/156): automake-1.16.2-8.el9.noarch.rpm 2.5 MB/s | 689 kB 00:00 2024-05-09T08:37:53.876 INFO:teuthology.orchestra.run.smithi098.stdout:(69/156): postgresql-private-libs-13.14-1.el9.x 516 kB/s | 133 kB 00:00 2024-05-09T08:37:54.111 INFO:teuthology.orchestra.run.smithi142.stdout:(48/156): bison-3.7.4-5.el9.x86_64.rpm 3.8 MB/s | 943 kB 00:00 2024-05-09T08:37:54.118 INFO:teuthology.orchestra.run.smithi098.stdout:(70/156): postgresql-contrib-13.14-1.el9.x86_64 710 kB/s | 911 kB 00:01 2024-05-09T08:37:54.162 INFO:teuthology.orchestra.run.smithi142.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-05-09T08:37:54.189 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-05-09T08:37:54.212 INFO:teuthology.orchestra.run.smithi142.stdout:(50/156): elfutils-libelf-devel-0.190-2.el9.x86 431 kB/s | 21 kB 00:00 2024-05-09T08:37:54.296 INFO:teuthology.orchestra.run.smithi142.stdout:(51/156): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 313 kB 00:00 2024-05-09T08:37:54.339 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-05-09T08:37:54.340 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-05-09T08:37:54.354 INFO:teuthology.orchestra.run.smithi142.stdout:(52/156): flexiblas-3.0.4-8.el9.x86_64.rpm 561 kB/s | 33 kB 00:00 2024-05-09T08:37:54.929 INFO:teuthology.orchestra.run.smithi142.stdout:(53/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.2 MB/s | 3.0 MB 00:00 2024-05-09T08:37:54.980 INFO:teuthology.orchestra.run.smithi142.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-05-09T08:37:55.030 INFO:teuthology.orchestra.run.smithi142.stdout:(55/156): libaio-devel-0.3.111-13.el9.x86_64.rp 224 kB/s | 11 kB 00:00 2024-05-09T08:37:55.089 INFO:teuthology.orchestra.run.smithi142.stdout:(56/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-05-09T08:37:55.139 INFO:teuthology.orchestra.run.smithi142.stdout:(57/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 909 kB/s | 45 kB 00:00 2024-05-09T08:37:55.281 INFO:teuthology.orchestra.run.smithi142.stdout:(58/156): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 662 kB 00:00 2024-05-09T08:37:55.464 INFO:teuthology.orchestra.run.smithi142.stdout:(59/156): libtool-2.4.6-45.el9.x86_64.rpm 3.1 MB/s | 582 kB 00:00 2024-05-09T08:37:55.531 INFO:teuthology.orchestra.run.smithi142.stdout:(60/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 382 kB/s | 25 kB 00:00 2024-05-09T08:37:55.552 INFO:teuthology.orchestra.run.smithi098.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 721 kB/s | 1.0 MB 00:01 2024-05-09T08:37:55.590 INFO:teuthology.orchestra.run.smithi142.stdout:(61/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 815 kB/s | 47 kB 00:00 2024-05-09T08:37:55.674 INFO:teuthology.orchestra.run.smithi142.stdout:(62/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-05-09T08:37:55.732 INFO:teuthology.orchestra.run.smithi142.stdout:(63/156): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-05-09T08:37:55.744 INFO:teuthology.orchestra.run.smithi098.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 456 kB/s | 87 kB 00:00 2024-05-09T08:37:55.791 INFO:teuthology.orchestra.run.smithi142.stdout:(64/156): openblas-0.3.21-2.el9.x86_64.rpm 603 kB/s | 35 kB 00:00 2024-05-09T08:37:56.206 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-05-09T08:37:56.206 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-05-09T08:37:56.499 INFO:teuthology.orchestra.run.smithi142.stdout:(65/156): openblas-openmp-0.3.21-2.el9.x86_64.r 6.7 MB/s | 4.7 MB 00:00 2024-05-09T08:37:56.991 INFO:teuthology.orchestra.run.smithi142.stdout:(66/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 9.0 MB/s | 4.4 MB 00:00 2024-05-09T08:37:57.175 INFO:teuthology.orchestra.run.smithi142.stdout:(67/156): postgresql-13.14-1.el9.x86_64.rpm 8.7 MB/s | 1.6 MB 00:00 2024-05-09T08:37:57.292 INFO:teuthology.orchestra.run.smithi142.stdout:(68/156): postgresql-contrib-13.14-1.el9.x86_64 7.6 MB/s | 911 kB 00:00 2024-05-09T08:37:57.350 INFO:teuthology.orchestra.run.smithi142.stdout:(69/156): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-05-09T08:37:57.834 INFO:teuthology.orchestra.run.smithi142.stdout:(70/156): postgresql-server-13.14-1.el9.x86_64. 12 MB/s | 5.8 MB 00:00 2024-05-09T08:37:57.959 INFO:teuthology.orchestra.run.smithi142.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-05-09T08:37:58.018 INFO:teuthology.orchestra.run.smithi142.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-05-09T08:37:58.235 INFO:teuthology.orchestra.run.smithi142.stdout:(73/156): ceph-mgr-diskprediction-local-19.0.0- 1.0 MB/s | 7.4 MB 00:07 2024-05-09T08:37:58.477 INFO:teuthology.orchestra.run.smithi142.stdout:(74/156): python3-devel-3.9.18-3.el9.x86_64.rpm 1.0 MB/s | 244 kB 00:00 2024-05-09T08:37:58.560 INFO:teuthology.orchestra.run.smithi142.stdout:(75/156): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-05-09T08:37:58.636 INFO:teuthology.orchestra.run.smithi142.stdout:(76/156): python3-babel-2.9.1-2.el9.noarch.rpm 9.7 MB/s | 6.0 MB 00:00 2024-05-09T08:37:58.677 INFO:teuthology.orchestra.run.smithi142.stdout:(77/156): python3-jsonpointer-2.0-4.el9.noarch. 462 kB/s | 19 kB 00:00 2024-05-09T08:37:58.703 INFO:teuthology.orchestra.run.smithi142.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 180 kB/s | 26 kB 00:00 2024-05-09T08:37:58.737 INFO:teuthology.orchestra.run.smithi142.stdout:(79/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-05-09T08:37:58.787 INFO:teuthology.orchestra.run.smithi142.stdout:(80/156): python3-markupsafe-1.1.1-12.el9.x86_6 416 kB/s | 35 kB 00:00 2024-05-09T08:37:58.904 INFO:teuthology.orchestra.run.smithi142.stdout:(81/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-05-09T08:37:59.021 INFO:teuthology.orchestra.run.smithi142.stdout:(82/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 222 kB 00:00 2024-05-09T08:37:59.088 INFO:teuthology.orchestra.run.smithi142.stdout:(83/156): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-05-09T08:37:59.155 INFO:teuthology.orchestra.run.smithi142.stdout:(84/156): python3-prettytable-0.7.2-27.el9.noar 634 kB/s | 42 kB 00:00 2024-05-09T08:37:59.222 INFO:teuthology.orchestra.run.smithi142.stdout:(85/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-05-09T08:37:59.288 INFO:teuthology.orchestra.run.smithi142.stdout:(86/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-05-09T08:37:59.314 INFO:teuthology.orchestra.run.smithi142.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 1.7 MB/s | 269 kB 00:00 2024-05-09T08:37:59.348 INFO:teuthology.orchestra.run.smithi142.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-05-09T08:37:59.373 INFO:teuthology.orchestra.run.smithi142.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 872 kB/s | 51 kB 00:00 2024-05-09T08:37:59.406 INFO:teuthology.orchestra.run.smithi142.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 921 kB/s | 54 kB 00:00 2024-05-09T08:37:59.465 INFO:teuthology.orchestra.run.smithi142.stdout:(91/156): python3-toml-0.10.2-6.el9.noarch.rpm 715 kB/s | 42 kB 00:00 2024-05-09T08:37:59.515 INFO:teuthology.orchestra.run.smithi142.stdout:(92/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 886 kB/s | 44 kB 00:00 2024-05-09T08:37:59.574 INFO:teuthology.orchestra.run.smithi142.stdout:(93/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-05-09T08:37:59.633 INFO:teuthology.orchestra.run.smithi142.stdout:(94/156): qatlib-service-24.02.0-1.el9.x86_64.r 615 kB/s | 36 kB 00:00 2024-05-09T08:37:59.682 INFO:teuthology.orchestra.run.smithi142.stdout:(95/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 952 kB/s | 47 kB 00:00 2024-05-09T08:37:59.741 INFO:teuthology.orchestra.run.smithi142.stdout:(96/156): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-05-09T08:37:59.800 INFO:teuthology.orchestra.run.smithi142.stdout:(97/156): uuid-1.6.2-55.el9.x86_64.rpm 996 kB/s | 58 kB 00:00 2024-05-09T08:37:59.858 INFO:teuthology.orchestra.run.smithi142.stdout:(98/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-05-09T08:37:59.909 INFO:teuthology.orchestra.run.smithi142.stdout:(99/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-05-09T08:37:59.968 INFO:teuthology.orchestra.run.smithi142.stdout:(100/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 768 kB/s | 45 kB 00:00 2024-05-09T08:38:00.018 INFO:teuthology.orchestra.run.smithi142.stdout:(101/156): lua-devel-5.4.4-4.el9.x86_64.rpm 447 kB/s | 22 kB 00:00 2024-05-09T08:38:00.118 INFO:teuthology.orchestra.run.smithi142.stdout:(102/156): protobuf-compiler-3.14.0-13.el9.x86_ 8.4 MB/s | 863 kB 00:00 2024-05-09T08:38:00.594 INFO:teuthology.orchestra.run.smithi098.stdout:(73/156): ceph-test-19.0.0-2515.g96805dcf.el9.x 2.0 MB/s | 48 MB 00:23 2024-05-09T08:38:00.735 INFO:teuthology.orchestra.run.smithi142.stdout:(103/156): abseil-cpp-20211102.0-3.el9.x86_64.r 889 kB/s | 548 kB 00:00 2024-05-09T08:38:00.861 INFO:teuthology.orchestra.run.smithi142.stdout:(104/156): gperftools-libs-2.9.1-2.el9.x86_64.r 2.4 MB/s | 309 kB 00:00 2024-05-09T08:38:00.936 INFO:teuthology.orchestra.run.smithi142.stdout:(105/156): grpc-data-1.46.7-10.el9.noarch.rpm 260 kB/s | 19 kB 00:00 2024-05-09T08:38:01.253 INFO:teuthology.orchestra.run.smithi142.stdout:(106/156): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-05-09T08:38:01.261 INFO:teuthology.orchestra.run.smithi098.stdout:(74/156): python3-devel-3.9.18-3.el9.x86_64.rpm 366 kB/s | 244 kB 00:00 2024-05-09T08:38:01.328 INFO:teuthology.orchestra.run.smithi142.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 348 kB/s | 26 kB 00:00 2024-05-09T08:38:01.412 INFO:teuthology.orchestra.run.smithi142.stdout:(108/156): liboath-2.6.7-2.el9.x86_64.rpm 581 kB/s | 49 kB 00:00 2024-05-09T08:38:01.461 INFO:teuthology.orchestra.run.smithi098.stdout:(75/156): python3-jinja2-2.11.3-5.el9.noarch.rp 1.2 MB/s | 248 kB 00:00 2024-05-09T08:38:01.487 INFO:teuthology.orchestra.run.smithi142.stdout:(109/156): libunwind-1.6.2-1.el9.x86_64.rpm 900 kB/s | 67 kB 00:00 2024-05-09T08:38:01.562 INFO:teuthology.orchestra.run.smithi142.stdout:(110/156): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-05-09T08:38:01.570 INFO:teuthology.orchestra.run.smithi098.stdout:(76/156): python3-jsonpatch-1.21-16.el9.noarch. 236 kB/s | 26 kB 00:00 2024-05-09T08:38:01.646 INFO:teuthology.orchestra.run.smithi142.stdout:(111/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.9 MB/s | 839 kB 00:00 2024-05-09T08:38:01.678 INFO:teuthology.orchestra.run.smithi098.stdout:(77/156): python3-jsonpointer-2.0-4.el9.noarch. 177 kB/s | 19 kB 00:00 2024-05-09T08:38:01.795 INFO:teuthology.orchestra.run.smithi098.stdout:(78/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-05-09T08:38:01.904 INFO:teuthology.orchestra.run.smithi098.stdout:(79/156): python3-markupsafe-1.1.1-12.el9.x86_6 320 kB/s | 35 kB 00:00 2024-05-09T08:38:02.013 INFO:teuthology.orchestra.run.smithi098.stdout:(80/156): postgresql-server-13.14-1.el9.x86_64. 727 kB/s | 5.8 MB 00:08 2024-05-09T08:38:02.088 INFO:teuthology.orchestra.run.smithi142.stdout:(112/156): ceph-test-19.0.0-2515.g96805dcf.el9. 1.8 MB/s | 48 MB 00:27 2024-05-09T08:38:02.121 INFO:teuthology.orchestra.run.smithi142.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 1.1 MB/s | 548 kB 00:00 2024-05-09T08:38:02.205 INFO:teuthology.orchestra.run.smithi142.stdout:(114/156): python3-cachetools-4.2.4-1.el9.noarc 387 kB/s | 32 kB 00:00 2024-05-09T08:38:02.276 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-05-09T08:38:02.280 INFO:teuthology.orchestra.run.smithi142.stdout:(115/156): python3-certifi-2021.10.8-2.el9.noar 200 kB/s | 15 kB 00:00 2024-05-09T08:38:02.355 INFO:teuthology.orchestra.run.smithi142.stdout:(116/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-05-09T08:38:02.356 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T08:38:02.388 INFO:teuthology.orchestra.run.smithi098.stdout:(81/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 668 kB/s | 250 kB 00:00 2024-05-09T08:38:02.389 INFO:teuthology.orchestra.run.smithi142.stdout:(117/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 144 kB/s | 43 kB 00:00 2024-05-09T08:38:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T08:38:02.447 INFO:teuthology.orchestra.run.smithi142.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-05-09T08:38:02.448 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T08:38:02.482 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T08:38:02.507 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T08:38:02.531 INFO:teuthology.orchestra.run.smithi142.stdout:(119/156): python3-colorama-0.4.4-12.el9.noarch 423 kB/s | 35 kB 00:00 2024-05-09T08:38:02.550 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T08:38:02.631 INFO:teuthology.orchestra.run.smithi142.stdout:(120/156): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-05-09T08:38:02.790 INFO:teuthology.orchestra.run.smithi142.stdout:(121/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.8 MB/s | 16 MB 00:03 2024-05-09T08:38:02.796 INFO:teuthology.orchestra.run.smithi098.stdout:(82/156): python3-oauthlib-3.1.1-5.el9.noarch.r 544 kB/s | 222 kB 00:00 2024-05-09T08:38:02.815 INFO:teuthology.orchestra.run.smithi142.stdout:(122/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 711 kB/s | 303 kB 00:00 2024-05-09T08:38:02.866 INFO:teuthology.orchestra.run.smithi142.stdout:(123/156): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-05-09T08:38:02.872 INFO:teuthology.orchestra.run.smithi098.stdout:(83/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.3 MB/s | 5.1 MB 00:00 2024-05-09T08:38:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T08:38:02.908 INFO:teuthology.orchestra.run.smithi142.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-05-09T08:38:02.958 INFO:teuthology.orchestra.run.smithi142.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 273 kB/s | 18 kB 00:00 2024-05-09T08:38:02.975 INFO:teuthology.orchestra.run.smithi142.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 346 kB/s | 23 kB 00:00 2024-05-09T08:38:02.983 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:38:02.997 INFO:teuthology.orchestra.run.smithi098.stdout:(84/156): python3-prettytable-0.7.2-27.el9.noar 338 kB/s | 42 kB 00:00 2024-05-09T08:38:03.017 INFO:teuthology.orchestra.run.smithi142.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 332 kB/s | 19 kB 00:00 2024-05-09T08:38:03.022 INFO:teuthology.orchestra.run.smithi098.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 342 kB/s | 77 kB 00:00 2024-05-09T08:38:03.033 INFO:teuthology.orchestra.run.smithi142.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-05-09T08:38:03.088 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:38:03.100 INFO:teuthology.orchestra.run.smithi142.stdout:(129/156): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-05-09T08:38:03.110 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:38:03.126 INFO:teuthology.orchestra.run.smithi142.stdout:(130/156): python3-logutils-0.3.5-21.el9.noarch 504 kB/s | 46 kB 00:00 2024-05-09T08:38:03.168 INFO:teuthology.orchestra.run.smithi142.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 1.1 MB/s | 79 kB 00:00 2024-05-09T08:38:03.184 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:38:03.193 INFO:teuthology.orchestra.run.smithi142.stdout:(132/156): python3-grpcio-tools-1.46.7-10.el9.x 359 kB/s | 144 kB 00:00 2024-05-09T08:38:03.214 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:38:03.218 INFO:teuthology.orchestra.run.smithi142.stdout:(133/156): python3-natsort-7.1.1-5.el9.noarch.r 626 kB/s | 58 kB 00:00 2024-05-09T08:38:03.244 INFO:teuthology.orchestra.run.smithi142.stdout:(134/156): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-05-09T08:38:03.269 INFO:teuthology.orchestra.run.smithi142.stdout:(135/156): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-05-09T08:38:03.303 INFO:teuthology.orchestra.run.smithi142.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-05-09T08:38:03.306 INFO:teuthology.orchestra.run.smithi098.stdout:(86/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 560 kB/s | 159 kB 00:00 2024-05-09T08:38:03.328 INFO:teuthology.orchestra.run.smithi142.stdout:(137/156): python3-pyperclip-1.8.2-2.el9.noarch 336 kB/s | 28 kB 00:00 2024-05-09T08:38:03.353 INFO:teuthology.orchestra.run.smithi142.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-05-09T08:38:03.367 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T08:38:03.403 INFO:teuthology.orchestra.run.smithi142.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 787 kB/s | 59 kB 00:00 2024-05-09T08:38:03.423 INFO:teuthology.orchestra.run.smithi098.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 633 kB/s | 269 kB 00:00 2024-05-09T08:38:03.427 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T08:38:03.428 INFO:teuthology.orchestra.run.smithi142.stdout:(140/156): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-05-09T08:38:03.454 INFO:teuthology.orchestra.run.smithi142.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-05-09T08:38:03.455 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:38:03.479 INFO:teuthology.orchestra.run.smithi142.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-05-09T08:38:03.529 INFO:teuthology.orchestra.run.smithi142.stdout:(143/156): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-05-09T08:38:03.532 INFO:teuthology.orchestra.run.smithi098.stdout:(88/156): python3-pytz-2021.1-5.el9.noarch.rpm 471 kB/s | 51 kB 00:00 2024-05-09T08:38:03.555 INFO:teuthology.orchestra.run.smithi142.stdout:(144/156): python3-websocket-client-1.2.3-2.el9 892 kB/s | 90 kB 00:00 2024-05-09T08:38:03.559 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:38:03.581 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:38:03.588 INFO:teuthology.orchestra.run.smithi142.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-05-09T08:38:03.615 INFO:teuthology.orchestra.run.smithi142.stdout:(146/156): python3-zc-lockfile-2.0-10.el9.noarc 238 kB/s | 20 kB 00:00 2024-05-09T08:38:03.632 INFO:teuthology.orchestra.run.smithi098.stdout:(89/156): python3-requests-oauthlib-1.3.0-12.el 537 kB/s | 54 kB 00:00 2024-05-09T08:38:03.644 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:38:03.666 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:38:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:38:03.705 INFO:teuthology.orchestra.run.smithi142.stdout:(147/156): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-05-09T08:38:03.722 INFO:teuthology.orchestra.run.smithi142.stdout:(148/156): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-05-09T08:38:03.774 INFO:teuthology.orchestra.run.smithi098.stdout:(90/156): python3-pyasn1-modules-0.4.8-6.el9.no 596 kB/s | 279 kB 00:00 2024-05-09T08:38:03.873 INFO:teuthology.orchestra.run.smithi142.stdout:(149/156): NetworkManager-1.47.5-1.el9.x86_64.r 15 MB/s | 2.3 MB 00:00 2024-05-09T08:38:03.875 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:38:03.883 INFO:teuthology.orchestra.run.smithi098.stdout:(91/156): python3-toml-0.10.2-6.el9.noarch.rpm 384 kB/s | 42 kB 00:00 2024-05-09T08:38:03.998 INFO:teuthology.orchestra.run.smithi142.stdout:(150/156): NetworkManager-libnm-1.47.5-1.el9.x8 15 MB/s | 1.8 MB 00:00 2024-05-09T08:38:04.000 INFO:teuthology.orchestra.run.smithi098.stdout:(92/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 379 kB/s | 44 kB 00:00 2024-05-09T08:38:04.003 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:38:04.056 INFO:teuthology.orchestra.run.smithi142.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 715 kB/s | 42 kB 00:00 2024-05-09T08:38:04.132 INFO:teuthology.orchestra.run.smithi142.stdout:(152/156): NetworkManager-tui-1.47.5-1.el9.x86_ 3.2 MB/s | 248 kB 00:00 2024-05-09T08:38:04.146 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:38:04.207 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:38:04.249 INFO:teuthology.orchestra.run.smithi142.stdout:(153/156): openssl-3.2.1-1.el9.x86_64.rpm 12 MB/s | 1.3 MB 00:00 2024-05-09T08:38:04.324 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:38:04.342 INFO:teuthology.orchestra.run.smithi098.stdout:(93/156): qatlib-24.02.0-1.el9.x86_64.rpm 646 kB/s | 221 kB 00:00 2024-05-09T08:38:04.399 INFO:teuthology.orchestra.run.smithi142.stdout:(154/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-05-09T08:38:04.400 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:38:04.467 INFO:teuthology.orchestra.run.smithi098.stdout:(94/156): qatlib-service-24.02.0-1.el9.x86_64.r 287 kB/s | 36 kB 00:00 2024-05-09T08:38:04.584 INFO:teuthology.orchestra.run.smithi098.stdout:(95/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 407 kB/s | 47 kB 00:00 2024-05-09T08:38:04.589 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:38:04.660 INFO:teuthology.orchestra.run.smithi098.stdout:(96/156): python3-babel-2.9.1-2.el9.noarch.rpm 685 kB/s | 6.0 MB 00:08 2024-05-09T08:38:04.962 INFO:teuthology.orchestra.run.smithi098.stdout:(97/156): uuid-1.6.2-55.el9.x86_64.rpm 534 kB/s | 58 kB 00:00 2024-05-09T08:38:04.963 INFO:teuthology.orchestra.run.smithi098.stdout:(98/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 511 kB/s | 94 kB 00:00 2024-05-09T08:38:04.963 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T08:38:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T08:38:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-09T08:38:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-09T08:38:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-09T08:38:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-09T08:38:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-09T08:38:05.015 INFO:teuthology.orchestra.run.smithi142.stdout:(155/156): librbd1-19.0.0-2515.g96805dcf.el9.x8 2.4 MB/s | 3.1 MB 00:01 2024-05-09T08:38:05.040 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-09T08:38:05.053 INFO:teuthology.orchestra.run.smithi098.stdout:(99/156): socat-1.7.4.1-5.el9.x86_64.rpm 651 kB/s | 305 kB 00:00 2024-05-09T08:38:05.069 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-09T08:38:05.137 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-09T08:38:05.144 INFO:teuthology.orchestra.run.smithi098.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 332 kB/s | 64 kB 00:00 2024-05-09T08:38:05.169 INFO:teuthology.orchestra.run.smithi098.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 383 kB/s | 45 kB 00:00 2024-05-09T08:38:05.191 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-09T08:38:05.221 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-09T08:38:05.265 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-09T08:38:05.278 INFO:teuthology.orchestra.run.smithi098.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-05-09T08:38:05.341 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-09T08:38:05.384 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 33/164 2024-05-09T08:38:05.403 INFO:teuthology.orchestra.run.smithi098.stdout:(103/156): abseil-cpp-20211102.0-3.el9.x86_64.r 4.3 MB/s | 548 kB 00:00 2024-05-09T08:38:05.419 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:38:05.445 INFO:teuthology.orchestra.run.smithi098.stdout:(104/156): gperftools-libs-2.9.1-2.el9.x86_64.r 7.3 MB/s | 309 kB 00:00 2024-05-09T08:38:05.448 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 35/164 2024-05-09T08:38:05.479 INFO:teuthology.orchestra.run.smithi098.stdout:(105/156): grpc-data-1.46.7-10.el9.noarch.rpm 586 kB/s | 19 kB 00:00 2024-05-09T08:38:05.518 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-09T08:38:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-09T08:38:05.596 INFO:teuthology.orchestra.run.smithi098.stdout:(106/156): libarrow-9.0.0-11.el9.x86_64.rpm 38 MB/s | 4.4 MB 00:00 2024-05-09T08:38:05.621 INFO:teuthology.orchestra.run.smithi098.stdout:(107/156): protobuf-compiler-3.14.0-13.el9.x86_ 1.9 MB/s | 863 kB 00:00 2024-05-09T08:38:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-09T08:38:05.646 INFO:teuthology.orchestra.run.smithi098.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 518 kB/s | 26 kB 00:00 2024-05-09T08:38:05.680 INFO:teuthology.orchestra.run.smithi098.stdout:(109/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-05-09T08:38:05.706 INFO:teuthology.orchestra.run.smithi098.stdout:(110/156): liboath-2.6.7-2.el9.x86_64.rpm 578 kB/s | 49 kB 00:00 2024-05-09T08:38:05.747 INFO:teuthology.orchestra.run.smithi098.stdout:(111/156): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-05-09T08:38:05.773 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-09T08:38:05.800 INFO:teuthology.orchestra.run.smithi098.stdout:(112/156): python3-asyncssh-2.13.2-5.el9.noarch 10 MB/s | 548 kB 00:00 2024-05-09T08:38:05.823 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-09T08:38:05.831 INFO:teuthology.orchestra.run.smithi098.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-05-09T08:38:05.857 INFO:teuthology.orchestra.run.smithi098.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 777 kB/s | 43 kB 00:00 2024-05-09T08:38:05.882 INFO:teuthology.orchestra.run.smithi098.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 640 kB/s | 32 kB 00:00 2024-05-09T08:38:05.907 INFO:teuthology.orchestra.run.smithi098.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-05-09T08:38:05.933 INFO:teuthology.orchestra.run.smithi098.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-05-09T08:38:05.958 INFO:teuthology.orchestra.run.smithi098.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-05-09T08:38:05.991 INFO:teuthology.orchestra.run.smithi098.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-05-09T08:38:06.017 INFO:teuthology.orchestra.run.smithi098.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 601 kB/s | 35 kB 00:00 2024-05-09T08:38:06.042 INFO:teuthology.orchestra.run.smithi098.stdout:(121/156): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-05-09T08:38:06.092 INFO:teuthology.orchestra.run.smithi098.stdout:(122/156): python3-grpcio-1.46.7-10.el9.x86_64. 27 MB/s | 2.0 MB 00:00 2024-05-09T08:38:06.118 INFO:teuthology.orchestra.run.smithi098.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-05-09T08:38:06.135 INFO:teuthology.orchestra.run.smithi098.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 255 kB/s | 11 kB 00:00 2024-05-09T08:38:06.152 INFO:teuthology.orchestra.run.smithi098.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 522 kB/s | 18 kB 00:00 2024-05-09T08:38:06.177 INFO:teuthology.orchestra.run.smithi098.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 548 kB/s | 23 kB 00:00 2024-05-09T08:38:06.202 INFO:teuthology.orchestra.run.smithi098.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-05-09T08:38:06.228 INFO:teuthology.orchestra.run.smithi098.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-05-09T08:38:06.257 INFO:teuthology.orchestra.run.smithi142.stdout:(156/156): librados2-19.0.0-2515.g96805dcf.el9. 1.3 MB/s | 3.4 MB 00:02 2024-05-09T08:38:06.261 INFO:teuthology.orchestra.run.smithi098.stdout:(129/156): python3-kubernetes-26.1.0-2.el9.noar 17 MB/s | 1.0 MB 00:00 2024-05-09T08:38:06.264 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:38:06.264 INFO:teuthology.orchestra.run.smithi142.stdout:Total 5.4 MB/s | 225 MB 00:41 2024-05-09T08:38:06.287 INFO:teuthology.orchestra.run.smithi098.stdout:(130/156): python3-logutils-0.3.5-21.el9.noarch 789 kB/s | 46 kB 00:00 2024-05-09T08:38:06.312 INFO:teuthology.orchestra.run.smithi098.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-05-09T08:38:06.337 INFO:teuthology.orchestra.run.smithi098.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-05-09T08:38:06.354 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-09T08:38:06.362 INFO:teuthology.orchestra.run.smithi098.stdout:(133/156): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-05-09T08:38:06.388 INFO:teuthology.orchestra.run.smithi098.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-05-09T08:38:06.413 INFO:teuthology.orchestra.run.smithi098.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-05-09T08:38:06.438 INFO:teuthology.orchestra.run.smithi098.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-05-09T08:38:06.444 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-09T08:38:06.464 INFO:teuthology.orchestra.run.smithi098.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-05-09T08:38:06.473 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-09T08:38:06.489 INFO:teuthology.orchestra.run.smithi098.stdout:(138/156): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-05-09T08:38:06.514 INFO:teuthology.orchestra.run.smithi098.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-05-09T08:38:06.515 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-09T08:38:06.540 INFO:teuthology.orchestra.run.smithi098.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 711 kB/s | 36 kB 00:00 2024-05-09T08:38:06.554 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 45/164 2024-05-09T08:38:06.582 INFO:teuthology.orchestra.run.smithi098.stdout:(141/156): python3-typing-extensions-4.4.0-2.el 761 kB/s | 51 kB 00:00 2024-05-09T08:38:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-09T08:38:06.607 INFO:teuthology.orchestra.run.smithi098.stdout:(142/156): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-05-09T08:38:06.624 INFO:teuthology.orchestra.run.smithi098.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-05-09T08:38:06.649 INFO:teuthology.orchestra.run.smithi098.stdout:(144/156): python3-werkzeug-2.0.3-3.el9.1.noarc 10 MB/s | 427 kB 00:00 2024-05-09T08:38:06.674 INFO:teuthology.orchestra.run.smithi098.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-05-09T08:38:06.700 INFO:teuthology.orchestra.run.smithi098.stdout:(146/156): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-05-09T08:38:06.741 INFO:teuthology.orchestra.run.smithi098.stdout:(147/156): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-05-09T08:38:06.749 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-09T08:38:06.798 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-09T08:38:06.854 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-09T08:38:06.927 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T08:38:06.957 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T08:38:06.993 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T08:38:07.040 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T08:38:07.076 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T08:38:07.119 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T08:38:07.162 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T08:38:07.297 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T08:38:07.350 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T08:38:07.380 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T08:38:07.422 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T08:38:07.450 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-05-09T08:38:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T08:38:07.547 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T08:38:07.587 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T08:38:07.601 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-05-09T08:38:07.601 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-05-09T08:38:07.613 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T08:38:07.637 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T08:38:07.712 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T08:38:07.839 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T08:38:07.878 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T08:38:07.906 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T08:38:07.940 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T08:38:07.979 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T08:38:08.011 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T08:38:08.049 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T08:38:08.084 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T08:38:08.117 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T08:38:08.178 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T08:38:08.210 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T08:38:08.297 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T08:38:08.348 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T08:38:08.379 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T08:38:08.426 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T08:38:08.525 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T08:38:08.573 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T08:38:08.900 INFO:teuthology.orchestra.run.smithi098.stdout:(148/156): librados2-19.0.0-2515.g96805dcf.el9. 1.5 MB/s | 3.4 MB 00:02 2024-05-09T08:38:09.101 INFO:teuthology.orchestra.run.smithi098.stdout:(149/156): NetworkManager-1.47.5-1.el9.x86_64.r 12 MB/s | 2.3 MB 00:00 2024-05-09T08:38:09.184 INFO:teuthology.orchestra.run.smithi098.stdout:(150/156): NetworkManager-libnm-1.47.5-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-05-09T08:38:09.226 INFO:teuthology.orchestra.run.smithi098.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-05-09T08:38:09.276 INFO:teuthology.orchestra.run.smithi098.stdout:(152/156): NetworkManager-tui-1.47.5-1.el9.x86_ 4.9 MB/s | 248 kB 00:00 2024-05-09T08:38:09.299 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T08:38:09.353 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T08:38:09.360 INFO:teuthology.orchestra.run.smithi098.stdout:(153/156): openssl-3.2.1-1.el9.x86_64.rpm 16 MB/s | 1.3 MB 00:00 2024-05-09T08:38:09.386 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T08:38:09.397 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-05-09T08:38:09.398 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-05-09T08:38:09.469 INFO:teuthology.orchestra.run.smithi098.stdout:(154/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 22 MB/s | 2.4 MB 00:00 2024-05-09T08:38:09.486 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T08:38:09.519 INFO:teuthology.orchestra.run.smithi098.stdout:(155/156): librbd1-19.0.0-2515.g96805dcf.el9.x8 1.1 MB/s | 3.1 MB 00:02 2024-05-09T08:38:09.545 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T08:38:09.633 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T08:38:09.680 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T08:38:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T08:38:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T08:38:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T08:38:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T08:38:10.719 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T08:38:11.955 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T08:38:12.032 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T08:38:12.060 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T08:38:12.093 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T08:38:12.402 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T08:38:12.447 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:38:12.537 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:38:12.568 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 102/164 2024-05-09T08:38:12.605 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T08:38:13.076 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T08:38:13.102 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:38:13.157 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:38:13.187 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 106/164 2024-05-09T08:38:15.461 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:38:15.464 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-05-09T08:38:15.500 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:38:15.545 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T08:38:15.583 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T08:38:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T08:38:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T08:38:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T08:38:15.747 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T08:38:16.036 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:38:16.070 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:38:16.111 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T08:38:16.154 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:38:16.154 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T08:38:16.154 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T08:38:16.154 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:38:16.172 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:38:16.205 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:38:16.275 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:38:16.298 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:38:16.389 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:38:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:38:16.566 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T08:38:16.616 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T08:38:16.644 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:38:16.747 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:38:16.769 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:38:16.832 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:38:16.854 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:38:16.878 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:38:17.079 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:38:17.209 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:38:17.351 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:38:17.411 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:38:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:38:17.596 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:38:17.793 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:38:17.900 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T08:38:17.995 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T08:38:18.026 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-09T08:38:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-09T08:38:18.124 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-09T08:38:18.160 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-09T08:38:18.203 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-09T08:38:18.235 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-09T08:38:18.264 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-09T08:38:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-09T08:38:18.385 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-09T08:38:18.416 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-09T08:38:18.451 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-09T08:38:18.526 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-09T08:38:18.562 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 33/164 2024-05-09T08:38:18.605 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:38:18.662 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 35/164 2024-05-09T08:38:18.736 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-09T08:38:18.783 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-09T08:38:18.864 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-09T08:38:18.995 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-09T08:38:19.050 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-09T08:38:19.567 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-09T08:38:19.646 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-09T08:38:19.676 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-09T08:38:19.710 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-09T08:38:19.748 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 45/164 2024-05-09T08:38:19.789 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-09T08:38:19.945 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-09T08:38:20.001 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-09T08:38:20.056 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-09T08:38:20.136 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T08:38:20.168 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T08:38:20.204 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T08:38:20.250 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T08:38:20.287 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T08:38:20.329 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T08:38:20.372 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T08:38:20.507 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T08:38:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T08:38:20.599 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T08:38:20.640 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T08:38:20.734 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T08:38:20.766 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T08:38:20.805 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T08:38:20.831 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T08:38:20.856 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T08:38:20.928 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T08:38:21.055 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T08:38:21.096 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T08:38:21.124 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T08:38:21.158 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T08:38:21.198 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T08:38:21.229 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T08:38:21.259 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T08:38:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T08:38:21.326 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T08:38:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T08:38:21.402 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T08:38:21.488 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T08:38:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T08:38:21.564 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T08:38:21.610 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T08:38:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T08:38:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T08:38:22.478 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T08:38:22.520 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T08:38:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T08:38:22.668 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T08:38:22.720 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T08:38:22.807 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T08:38:22.847 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T08:38:23.024 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T08:38:23.054 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T08:38:23.131 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T08:38:23.771 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T08:38:23.893 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T08:38:25.107 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T08:38:25.189 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T08:38:25.217 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T08:38:25.250 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T08:38:25.562 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T08:38:25.605 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:38:25.679 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:38:25.709 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 102/164 2024-05-09T08:38:25.745 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T08:38:25.860 INFO:teuthology.orchestra.run.smithi098.stdout:(156/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 749 kB/s | 16 MB 00:22 2024-05-09T08:38:25.862 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:38:25.863 INFO:teuthology.orchestra.run.smithi098.stdout:Total 3.6 MB/s | 225 MB 01:01 2024-05-09T08:38:26.215 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T08:38:26.241 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:38:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:38:26.342 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 106/164 2024-05-09T08:38:27.013 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-05-09T08:38:27.162 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-05-09T08:38:27.162 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-05-09T08:38:27.631 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:38:27.631 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /sys 2024-05-09T08:38:27.631 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /proc 2024-05-09T08:38:27.631 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /mnt 2024-05-09T08:38:27.632 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /var/tmp 2024-05-09T08:38:27.632 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /home 2024-05-09T08:38:27.632 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /root 2024-05-09T08:38:27.632 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /tmp 2024-05-09T08:38:27.632 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:38:27.803 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:38:27.818 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:38:27.818 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:27.818 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T08:38:27.818 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:38:28.627 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:38:28.664 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:38:29.200 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:38:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:38:29.314 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:38:29.314 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:29.315 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T08:38:29.315 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:38:29.315 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:38:29.315 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:38:29.353 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-05-09T08:38:29.354 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-05-09T08:38:29.450 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:38:29.540 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:38:29.562 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:38:29.562 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:29.562 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T08:38:29.562 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:38:29.562 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:38:29.562 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:38:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:38:29.615 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T08:38:29.615 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T08:38:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:38:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:38:30.024 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:38:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:38:30.046 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:30.046 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T08:38:30.046 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:38:30.046 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:38:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:38:30.154 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T08:38:30.211 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T08:38:30.266 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T08:38:30.452 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T08:38:30.499 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T08:38:30.569 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T08:38:30.657 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T08:38:30.789 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T08:38:30.842 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T08:38:30.882 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T08:38:30.909 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T08:38:31.299 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:38:31.325 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:38:31.356 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 126/164 2024-05-09T08:38:31.418 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 127/164 2024-05-09T08:38:31.447 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:38:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:38:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:38:33.481 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:38:33.516 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:38:33.617 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:38:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 131/164 2024-05-09T08:38:33.818 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:38:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:38:33.842 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:33.842 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T08:38:33.842 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:38:33.842 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:38:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:38:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:38:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:38:33.993 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 134/164 2024-05-09T08:38:34.042 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 135/164 2024-05-09T08:38:34.934 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T08:38:34.963 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:38:34.985 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:38:34.985 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:34.985 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T08:38:34.985 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:38:34.985 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:38:34.986 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:38:35.039 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:38:35.060 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:38:35.061 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:35.061 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T08:38:35.061 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:38:35.106 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:38:35.127 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:38:35.127 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:35.127 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T08:38:35.128 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:38:35.128 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:38:35.128 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:38:35.432 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:38:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-05-09T08:38:35.454 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:38:35.454 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:35.454 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T08:38:35.454 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:38:35.454 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:38:35.454 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:38:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T08:38:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T08:38:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T08:38:35.698 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T08:38:35.732 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T08:38:35.782 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T08:38:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T08:38:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:38:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:38:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:38:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:38:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:38:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T08:38:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T08:38:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:38:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:38:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:38:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:38:36.957 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:38:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:38:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:38:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:38:37.438 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:38:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:38:37.637 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:38:37.709 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:38:37.928 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:38:38.042 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T08:38:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T08:38:38.177 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-09T08:38:38.216 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-09T08:38:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-09T08:38:38.324 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-09T08:38:38.367 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-09T08:38:38.408 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-09T08:38:38.436 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-09T08:38:38.505 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-09T08:38:38.557 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-09T08:38:38.589 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-09T08:38:38.632 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-09T08:38:38.708 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-09T08:38:38.743 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 33/164 2024-05-09T08:38:38.787 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:38:38.815 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 35/164 2024-05-09T08:38:38.885 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-09T08:38:38.930 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-09T08:38:39.004 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-09T08:38:39.148 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-09T08:38:39.199 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-09T08:38:39.723 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-09T08:38:39.820 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-09T08:38:39.849 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-09T08:38:39.901 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-09T08:38:39.939 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 45/164 2024-05-09T08:38:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 141/164 2024-05-09T08:38:39.981 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-09T08:38:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 142/164 2024-05-09T08:38:40.034 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 143/164 2024-05-09T08:38:40.066 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T08:38:40.093 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T08:38:40.135 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-09T08:38:40.200 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-09T08:38:40.263 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-09T08:38:40.281 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 146/164 2024-05-09T08:38:40.331 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T08:38:40.337 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T08:38:40.361 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 148/164 2024-05-09T08:38:40.367 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T08:38:40.391 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T08:38:40.404 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T08:38:40.437 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T08:38:40.450 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T08:38:40.487 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T08:38:40.530 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T08:38:40.572 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T08:38:40.707 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T08:38:40.744 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T08:38:40.775 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T08:38:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T08:38:40.910 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T08:38:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T08:38:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T08:38:41.024 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T08:38:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T08:38:41.053 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:38:41.054 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:38:41.054 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /sys 2024-05-09T08:38:41.054 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /proc 2024-05-09T08:38:41.054 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /mnt 2024-05-09T08:38:41.054 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /var/tmp 2024-05-09T08:38:41.054 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /home 2024-05-09T08:38:41.054 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /root 2024-05-09T08:38:41.054 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /tmp 2024-05-09T08:38:41.054 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:38:41.094 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:38:41.114 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:38:41.134 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T08:38:41.230 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:38:41.246 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:38:41.246 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:41.246 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T08:38:41.246 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:38:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T08:38:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T08:38:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T08:38:41.343 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T08:38:41.382 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T08:38:41.414 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T08:38:41.444 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T08:38:41.478 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T08:38:41.511 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T08:38:41.553 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T08:38:41.588 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T08:38:41.675 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T08:38:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T08:38:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T08:38:41.813 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T08:38:41.911 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T08:38:41.945 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 152/164 2024-05-09T08:38:41.959 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T08:38:42.032 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T08:38:42.072 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T08:38:42.157 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T08:38:42.209 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T08:38:42.236 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T08:38:42.236 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:38:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:38:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T08:38:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:38:42.413 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:38:42.437 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T08:38:42.442 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:38:42.443 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:38:42.623 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:38:42.623 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:38:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T08:38:42.715 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:38:42.724 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T08:38:42.741 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:38:42.757 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T08:38:42.766 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:38:42.767 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:42.767 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T08:38:42.767 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:38:42.767 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:38:42.767 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:38:42.779 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:38:42.856 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T08:38:42.916 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T08:38:42.993 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:38:43.003 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T08:38:43.015 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:38:43.016 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:43.016 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T08:38:43.016 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:38:43.016 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:38:43.016 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:38:43.034 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T08:38:43.211 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T08:38:43.241 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T08:38:43.319 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T08:38:43.461 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:38:43.483 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:38:43.483 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:43.483 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T08:38:43.484 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:38:43.484 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:38:43.484 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:38:43.582 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T08:38:43.630 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T08:38:43.686 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T08:38:43.879 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T08:38:43.928 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T08:38:43.972 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T08:38:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T08:38:44.089 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T08:38:44.094 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T08:38:44.235 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T08:38:44.278 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T08:38:44.310 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T08:38:44.337 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T08:38:44.777 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:38:44.802 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:38:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 126/164 2024-05-09T08:38:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 127/164 2024-05-09T08:38:44.959 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:38:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T08:38:45.420 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T08:38:45.450 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T08:38:45.483 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T08:38:45.802 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T08:38:45.847 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:38:45.929 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:38:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 102/164 2024-05-09T08:38:45.995 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:38:45.996 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T08:38:46.044 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:38:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T08:38:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:38:46.579 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:38:46.601 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 106/164 2024-05-09T08:38:47.007 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:38:47.036 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:38:47.138 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:38:47.298 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 131/164 2024-05-09T08:38:47.330 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:38:47.353 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:38:47.353 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:47.353 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T08:38:47.353 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:38:47.353 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:38:47.353 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:38:47.397 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:38:47.408 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:38:47.506 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 134/164 2024-05-09T08:38:47.554 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 135/164 2024-05-09T08:38:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:38:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 1/164 2024-05-09T08:38:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2/164 2024-05-09T08:38:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 3/164 2024-05-09T08:38:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 4/164 2024-05-09T08:38:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2515.g96805 5/164 2024-05-09T08:38:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 6/164 2024-05-09T08:38:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 7/164 2024-05-09T08:38:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:38:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:38:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:38:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 11/164 2024-05-09T08:38:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 12/164 2024-05-09T08:38:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:38:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 14/164 2024-05-09T08:38:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 15/164 2024-05-09T08:38:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 16/164 2024-05-09T08:38:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 17/164 2024-05-09T08:38:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 18/164 2024-05-09T08:38:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 19/164 2024-05-09T08:38:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 20/164 2024-05-09T08:38:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 21/164 2024-05-09T08:38:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 22/164 2024-05-09T08:38:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 23/164 2024-05-09T08:38:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 24/164 2024-05-09T08:38:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 25/164 2024-05-09T08:38:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 26/164 2024-05-09T08:38:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 27/164 2024-05-09T08:38:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 28/164 2024-05-09T08:38:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 29/164 2024-05-09T08:38:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 30/164 2024-05-09T08:38:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 31/164 2024-05-09T08:38:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 32/164 2024-05-09T08:38:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 33/164 2024-05-09T08:38:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:38:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 35/164 2024-05-09T08:38:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 36/164 2024-05-09T08:38:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 37/164 2024-05-09T08:38:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 38/164 2024-05-09T08:38:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 39/164 2024-05-09T08:38:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T08:38:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T08:38:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T08:38:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T08:38:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T08:38:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T08:38:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T08:38:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T08:38:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T08:38:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T08:38:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T08:38:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T08:38:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T08:38:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T08:38:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T08:38:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T08:38:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T08:38:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T08:38:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T08:38:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T08:38:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T08:38:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T08:38:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T08:38:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T08:38:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T08:38:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T08:38:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T08:38:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T08:38:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T08:38:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T08:38:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T08:38:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T08:38:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T08:38:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T08:38:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T08:38:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T08:38:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T08:38:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T08:38:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T08:38:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T08:38:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T08:38:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T08:38:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T08:38:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T08:38:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T08:38:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T08:38:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T08:38:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T08:38:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T08:38:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T08:38:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T08:38:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T08:38:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T08:38:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T08:38:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T08:38:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T08:38:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T08:38:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T08:38:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T08:38:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T08:38:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T08:38:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T08:38:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T08:38:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T08:38:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T08:38:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T08:38:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T08:38:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T08:38:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T08:38:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T08:38:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T08:38:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T08:38:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T08:38:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T08:38:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T08:38:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T08:38:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T08:38:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T08:38:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T08:38:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T08:38:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T08:38:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T08:38:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T08:38:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T08:38:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T08:38:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T08:38:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T08:38:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T08:38:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T08:38:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T08:38:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T08:38:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T08:38:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T08:38:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T08:38:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T08:38:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T08:38:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T08:38:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T08:38:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T08:38:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T08:38:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T08:38:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T08:38:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T08:38:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T08:38:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T08:38:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T08:38:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T08:38:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T08:38:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T08:38:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 149/164 2024-05-09T08:38:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T08:38:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 151/164 2024-05-09T08:38:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T08:38:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T08:38:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T08:38:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T08:38:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T08:38:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T08:38:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T08:38:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T08:38:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T08:38:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T08:38:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T08:38:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T08:38:48.452 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T08:38:48.491 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:38:48.514 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:38:48.515 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:48.515 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T08:38:48.515 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:38:48.515 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:38:48.515 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:38:48.558 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:38:48.580 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:38:48.580 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:48.580 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T08:38:48.580 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:38:48.625 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:38:48.648 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:38:48.648 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:48.648 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T08:38:48.649 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:38:48.649 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:38:48.649 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:38:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:38:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:38:48.959 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:38:48.982 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:38:48.982 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:38:48.983 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T08:38:48.983 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:38:48.983 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:38:48.983 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:38:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:38:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:38:49.627 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:38:49.627 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T08:38:49.627 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T08:38:49.627 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:38:49.671 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:38:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:38:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:38:50.881 INFO:teuthology.orchestra.run.smithi105.stdout:Upgraded: 2024-05-09T08:38:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T08:38:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T08:38:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T08:38:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T08:38:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T08:38:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T08:38:50.881 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-05-09T08:38:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T08:38:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T08:38:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T08:38:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T08:38:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:38:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:38:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:38:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:38:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:38:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:38:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:38:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:38:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:38:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:38:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T08:38:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T08:38:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T08:38:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T08:38:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T08:38:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T08:38:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T08:38:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T08:38:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T08:38:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T08:38:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T08:38:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T08:38:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T08:38:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T08:38:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T08:38:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T08:38:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T08:38:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T08:38:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T08:38:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T08:38:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T08:38:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T08:38:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T08:38:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T08:38:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T08:38:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T08:38:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T08:38:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T08:38:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T08:38:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T08:38:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T08:38:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T08:38:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T08:38:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T08:38:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T08:38:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T08:38:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T08:38:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T08:38:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T08:38:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T08:38:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T08:38:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T08:38:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T08:38:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T08:38:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T08:38:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T08:38:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T08:38:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T08:38:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T08:38:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T08:38:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T08:38:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T08:38:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T08:38:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T08:38:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T08:38:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T08:38:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T08:38:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T08:38:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T08:38:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T08:38:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T08:38:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T08:38:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T08:38:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T08:38:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T08:38:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:38:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T08:38:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T08:38:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T08:38:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T08:38:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T08:38:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T08:38:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T08:38:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-09T08:38:50.892 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-05-09T08:38:51.285 DEBUG:teuthology.parallel:result is None 2024-05-09T08:38:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 141/164 2024-05-09T08:38:53.546 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 142/164 2024-05-09T08:38:53.587 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 143/164 2024-05-09T08:38:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T08:38:53.646 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T08:38:53.830 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 146/164 2024-05-09T08:38:53.876 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T08:38:53.906 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 148/164 2024-05-09T08:38:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T08:38:53.973 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T08:38:54.583 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:38:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:38:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:38:55.459 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 152/164 2024-05-09T08:38:55.533 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T08:38:55.574 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T08:38:55.654 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T08:38:55.687 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T08:38:55.712 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T08:38:55.713 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:38:55.797 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:38:55.821 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T08:38:55.821 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:38:55.906 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:38:55.931 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T08:38:55.936 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:38:55.936 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:38:56.124 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:38:56.124 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:38:56.217 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:38:56.279 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:39:01.674 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:39:01.674 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 1/164 2024-05-09T08:39:01.674 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2/164 2024-05-09T08:39:01.674 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 3/164 2024-05-09T08:39:01.674 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 4/164 2024-05-09T08:39:01.674 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2515.g96805 5/164 2024-05-09T08:39:01.674 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 6/164 2024-05-09T08:39:01.674 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 7/164 2024-05-09T08:39:01.674 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:39:01.675 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:39:01.675 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:39:01.675 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 11/164 2024-05-09T08:39:01.675 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 12/164 2024-05-09T08:39:01.675 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:39:01.675 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 14/164 2024-05-09T08:39:01.675 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 15/164 2024-05-09T08:39:01.675 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 16/164 2024-05-09T08:39:01.675 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 17/164 2024-05-09T08:39:01.675 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 18/164 2024-05-09T08:39:01.675 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 19/164 2024-05-09T08:39:01.675 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 20/164 2024-05-09T08:39:01.675 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 21/164 2024-05-09T08:39:01.675 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 22/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 23/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 24/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 25/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 26/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 27/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 28/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 29/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 30/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 31/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 32/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 33/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 35/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 36/164 2024-05-09T08:39:01.676 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 37/164 2024-05-09T08:39:01.677 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 38/164 2024-05-09T08:39:01.677 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 39/164 2024-05-09T08:39:01.677 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T08:39:01.677 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T08:39:01.677 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T08:39:01.677 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T08:39:01.677 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T08:39:01.677 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T08:39:01.677 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T08:39:01.677 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T08:39:01.677 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T08:39:01.677 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T08:39:01.677 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T08:39:01.677 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T08:39:01.678 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T08:39:01.678 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T08:39:01.678 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T08:39:01.678 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T08:39:01.678 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T08:39:01.678 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T08:39:01.678 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T08:39:01.678 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T08:39:01.678 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T08:39:01.678 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T08:39:01.678 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T08:39:01.678 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T08:39:01.678 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T08:39:01.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T08:39:01.680 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T08:39:01.680 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T08:39:01.680 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T08:39:01.680 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T08:39:01.680 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T08:39:01.680 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T08:39:01.680 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T08:39:01.680 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T08:39:01.680 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T08:39:01.680 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T08:39:01.680 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T08:39:01.681 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T08:39:01.681 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T08:39:01.681 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T08:39:01.681 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T08:39:01.681 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T08:39:01.681 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T08:39:01.681 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T08:39:01.681 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T08:39:01.681 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T08:39:01.681 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T08:39:01.681 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T08:39:01.681 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T08:39:01.681 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T08:39:01.681 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T08:39:01.682 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T08:39:01.682 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T08:39:01.682 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T08:39:01.682 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T08:39:01.682 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T08:39:01.682 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T08:39:01.682 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T08:39:01.682 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T08:39:01.682 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T08:39:01.682 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T08:39:01.682 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T08:39:01.682 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T08:39:01.682 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T08:39:01.682 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T08:39:01.683 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T08:39:01.683 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T08:39:01.683 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T08:39:01.683 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T08:39:01.683 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T08:39:01.683 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T08:39:01.683 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T08:39:01.683 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T08:39:01.683 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T08:39:01.683 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T08:39:01.683 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T08:39:01.683 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T08:39:01.683 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T08:39:01.683 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T08:39:01.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T08:39:01.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T08:39:01.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T08:39:01.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T08:39:01.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T08:39:01.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T08:39:01.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T08:39:01.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T08:39:01.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T08:39:01.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T08:39:01.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T08:39:01.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T08:39:01.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T08:39:01.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T08:39:01.685 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T08:39:01.685 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T08:39:01.685 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 149/164 2024-05-09T08:39:01.685 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T08:39:01.685 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 151/164 2024-05-09T08:39:01.685 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T08:39:01.685 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T08:39:01.685 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T08:39:01.685 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T08:39:01.685 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T08:39:01.685 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T08:39:01.686 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T08:39:01.686 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T08:39:01.686 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T08:39:01.686 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T08:39:01.686 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T08:39:01.686 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T08:39:01.848 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:39:01.848 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-05-09T08:39:01.848 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-05-09T08:39:01.848 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-05-09T08:39:01.848 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-05-09T08:39:01.848 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-05-09T08:39:01.848 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-05-09T08:39:01.848 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-05-09T08:39:01.849 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:39:02.013 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:39:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:39:02.029 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:39:02.029 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T08:39:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:39:03.533 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:39:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:39:03.560 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:39:03.560 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T08:39:03.560 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:39:03.560 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:39:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:39:03.783 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:39:03.807 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:39:03.807 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:39:03.807 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T08:39:03.807 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:39:03.807 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:39:03.807 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:39:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:39:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:39:04.275 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:39:04.275 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T08:39:04.275 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:39:04.275 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:39:04.275 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:39:04.301 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:39:04.301 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:39:04.301 INFO:teuthology.orchestra.run.smithi142.stdout:Upgraded: 2024-05-09T08:39:04.301 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T08:39:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T08:39:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T08:39:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T08:39:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T08:39:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T08:39:04.302 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-05-09T08:39:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T08:39:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T08:39:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T08:39:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T08:39:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T08:39:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T08:39:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T08:39:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T08:39:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T08:39:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T08:39:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T08:39:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T08:39:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T08:39:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T08:39:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T08:39:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T08:39:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T08:39:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T08:39:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T08:39:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T08:39:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T08:39:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T08:39:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T08:39:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T08:39:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T08:39:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T08:39:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T08:39:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T08:39:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T08:39:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T08:39:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T08:39:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T08:39:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T08:39:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T08:39:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T08:39:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T08:39:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T08:39:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T08:39:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T08:39:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T08:39:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T08:39:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T08:39:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T08:39:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T08:39:04.313 INFO:teuthology.orchestra.run.smithi142.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T08:39:04.313 INFO:teuthology.orchestra.run.smithi142.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T08:39:04.313 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:39:04.313 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-05-09T08:39:04.373 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T08:39:04.430 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T08:39:04.485 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T08:39:04.672 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T08:39:04.727 DEBUG:teuthology.parallel:result is None 2024-05-09T08:39:04.727 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T08:39:04.804 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T08:39:04.893 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T08:39:05.032 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T08:39:05.094 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T08:39:05.135 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T08:39:05.162 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T08:39:05.529 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:39:05.561 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:39:05.593 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 126/164 2024-05-09T08:39:05.655 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 127/164 2024-05-09T08:39:05.684 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:39:06.731 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:39:06.761 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:39:07.735 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:39:07.761 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:39:07.862 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:39:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 131/164 2024-05-09T08:39:08.064 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:39:08.088 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:39:08.088 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:39:08.088 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T08:39:08.088 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:39:08.088 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:39:08.088 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:39:08.139 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:39:08.151 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:39:08.248 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 134/164 2024-05-09T08:39:08.303 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 135/164 2024-05-09T08:39:09.208 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T08:39:09.234 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:39:09.257 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:39:09.257 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:39:09.257 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T08:39:09.257 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:39:09.257 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:39:09.257 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:39:09.301 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:39:09.324 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:39:09.324 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:39:09.324 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T08:39:09.324 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:39:09.377 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:39:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:39:09.400 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:39:09.401 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T08:39:09.401 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:39:09.401 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:39:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:39:09.703 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:39:09.726 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:39:09.726 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:39:09.726 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T08:39:09.726 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:39:09.726 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:39:09.726 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:39:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 141/164 2024-05-09T08:39:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 142/164 2024-05-09T08:39:14.355 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 143/164 2024-05-09T08:39:14.388 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T08:39:14.415 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T08:39:14.613 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 146/164 2024-05-09T08:39:14.662 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T08:39:14.692 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 148/164 2024-05-09T08:39:14.721 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T08:39:14.768 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T08:39:15.372 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:39:15.416 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:39:15.435 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:39:16.262 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 152/164 2024-05-09T08:39:16.346 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T08:39:16.392 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T08:39:16.496 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T08:39:16.533 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T08:39:16.559 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T08:39:16.559 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:39:16.644 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:39:16.668 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T08:39:16.668 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:39:16.753 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:39:16.778 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T08:39:16.783 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:39:16.783 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:39:16.963 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:39:16.964 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:39:17.048 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:39:17.113 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:39:22.820 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:39:22.820 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 1/164 2024-05-09T08:39:22.820 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2/164 2024-05-09T08:39:22.820 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 3/164 2024-05-09T08:39:22.820 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 4/164 2024-05-09T08:39:22.821 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2515.g96805 5/164 2024-05-09T08:39:22.821 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 6/164 2024-05-09T08:39:22.821 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 7/164 2024-05-09T08:39:22.821 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:39:22.821 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:39:22.821 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:39:22.821 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 11/164 2024-05-09T08:39:22.821 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 12/164 2024-05-09T08:39:22.821 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:39:22.821 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 14/164 2024-05-09T08:39:22.821 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 15/164 2024-05-09T08:39:22.821 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 16/164 2024-05-09T08:39:22.821 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 17/164 2024-05-09T08:39:22.822 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 18/164 2024-05-09T08:39:22.822 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 19/164 2024-05-09T08:39:22.822 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 20/164 2024-05-09T08:39:22.822 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 21/164 2024-05-09T08:39:22.822 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 22/164 2024-05-09T08:39:22.822 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 23/164 2024-05-09T08:39:22.822 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 24/164 2024-05-09T08:39:22.822 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 25/164 2024-05-09T08:39:22.822 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 26/164 2024-05-09T08:39:22.822 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 27/164 2024-05-09T08:39:22.822 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 28/164 2024-05-09T08:39:22.835 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 29/164 2024-05-09T08:39:22.835 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 30/164 2024-05-09T08:39:22.835 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 31/164 2024-05-09T08:39:22.835 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 32/164 2024-05-09T08:39:22.835 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 33/164 2024-05-09T08:39:22.835 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:39:22.835 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 35/164 2024-05-09T08:39:22.835 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 36/164 2024-05-09T08:39:22.835 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 37/164 2024-05-09T08:39:22.836 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 38/164 2024-05-09T08:39:22.836 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 39/164 2024-05-09T08:39:22.836 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T08:39:22.836 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T08:39:22.836 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T08:39:22.836 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T08:39:22.836 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T08:39:22.836 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T08:39:22.836 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T08:39:22.836 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T08:39:22.836 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T08:39:22.836 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T08:39:22.836 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T08:39:22.837 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T08:39:22.837 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T08:39:22.837 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T08:39:22.837 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T08:39:22.837 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T08:39:22.837 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T08:39:22.837 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T08:39:22.837 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T08:39:22.837 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T08:39:22.837 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T08:39:22.837 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T08:39:22.837 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T08:39:22.837 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T08:39:22.838 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T08:39:22.838 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T08:39:22.838 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T08:39:22.838 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T08:39:22.838 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T08:39:22.838 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T08:39:22.838 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T08:39:22.838 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T08:39:22.838 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T08:39:22.838 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T08:39:22.838 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T08:39:22.838 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T08:39:22.838 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T08:39:22.839 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T08:39:22.839 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T08:39:22.839 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T08:39:22.839 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T08:39:22.839 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T08:39:22.839 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T08:39:22.839 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T08:39:22.839 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T08:39:22.839 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T08:39:22.839 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T08:39:22.839 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T08:39:22.839 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T08:39:22.839 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T08:39:22.840 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T08:39:22.840 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T08:39:22.840 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T08:39:22.840 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T08:39:22.840 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T08:39:22.840 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T08:39:22.840 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T08:39:22.840 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T08:39:22.840 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T08:39:22.840 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T08:39:22.840 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T08:39:22.840 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T08:39:22.840 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T08:39:22.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T08:39:22.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T08:39:22.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T08:39:22.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T08:39:22.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T08:39:22.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T08:39:22.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T08:39:22.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T08:39:22.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T08:39:22.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T08:39:22.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T08:39:22.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T08:39:22.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T08:39:22.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T08:39:22.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T08:39:22.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T08:39:22.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T08:39:22.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T08:39:22.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T08:39:22.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T08:39:22.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T08:39:22.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T08:39:22.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T08:39:22.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T08:39:22.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T08:39:22.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T08:39:22.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T08:39:22.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T08:39:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T08:39:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T08:39:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T08:39:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T08:39:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T08:39:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T08:39:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T08:39:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T08:39:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T08:39:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T08:39:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T08:39:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T08:39:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T08:39:22.843 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T08:39:22.844 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T08:39:22.844 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T08:39:22.844 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T08:39:22.844 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T08:39:22.844 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 149/164 2024-05-09T08:39:22.844 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T08:39:22.844 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 151/164 2024-05-09T08:39:22.844 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T08:39:22.844 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T08:39:22.844 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T08:39:22.844 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T08:39:22.844 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T08:39:22.844 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T08:39:22.844 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T08:39:22.845 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T08:39:22.845 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T08:39:22.845 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T08:39:22.845 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T08:39:22.845 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T08:39:25.473 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:39:25.473 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:39:25.473 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-05-09T08:39:25.473 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T08:39:25.474 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T08:39:25.474 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T08:39:25.474 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T08:39:25.474 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.474 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.474 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T08:39:25.474 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T08:39:25.474 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-05-09T08:39:25.474 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T08:39:25.474 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T08:39:25.474 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T08:39:25.474 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T08:39:25.475 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.475 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.475 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.475 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.475 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:25.475 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.475 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.475 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.475 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:25.475 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:25.475 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:25.475 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:25.475 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:39:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T08:39:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T08:39:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T08:39:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T08:39:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T08:39:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T08:39:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T08:39:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T08:39:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T08:39:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T08:39:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T08:39:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T08:39:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T08:39:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T08:39:25.478 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T08:39:25.478 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T08:39:25.478 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.478 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.478 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T08:39:25.478 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.478 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T08:39:25.478 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T08:39:25.478 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T08:39:25.478 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T08:39:25.478 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T08:39:25.478 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T08:39:25.479 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T08:39:25.479 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T08:39:25.479 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T08:39:25.479 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T08:39:25.479 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T08:39:25.479 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T08:39:25.479 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T08:39:25.479 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T08:39:25.479 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T08:39:25.479 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T08:39:25.479 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T08:39:25.479 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T08:39:25.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T08:39:25.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T08:39:25.481 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T08:39:25.482 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T08:39:25.482 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T08:39:25.482 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T08:39:25.482 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T08:39:25.482 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T08:39:25.482 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T08:39:25.482 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T08:39:25.482 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T08:39:25.482 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T08:39:25.482 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T08:39:25.482 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T08:39:25.482 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T08:39:25.482 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T08:39:25.483 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T08:39:25.483 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T08:39:25.483 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T08:39:25.483 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T08:39:25.483 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.483 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.483 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T08:39:25.483 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T08:39:25.483 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T08:39:25.483 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.483 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T08:39:25.483 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T08:39:25.483 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:39:25.484 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T08:39:25.485 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T08:39:25.485 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T08:39:25.485 INFO:teuthology.orchestra.run.smithi098.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T08:39:25.485 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T08:39:25.485 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T08:39:25.485 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T08:39:25.485 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:39:25.485 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-05-09T08:39:25.917 DEBUG:teuthology.parallel:result is None 2024-05-09T08:39:25.918 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:39:26.140 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T08:39:26.162 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-2515.g96805dcf.el9 2024-05-09T08:39:26.163 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2515.g96805dcf.el9 2024-05-09T08:39:26.163 INFO:teuthology.task.install:The correct ceph version 19.0.0-2515.g96805dcf is installed. 2024-05-09T08:39:26.165 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:39:26.308 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T08:39:26.334 INFO:teuthology.orchestra.run.smithi105.stdout:19.0.0-2515.g96805dcf.el9 2024-05-09T08:39:26.334 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2515.g96805dcf.el9 2024-05-09T08:39:26.334 INFO:teuthology.task.install:The correct ceph version 19.0.0-2515.g96805dcf is installed. 2024-05-09T08:39:26.336 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:39:26.553 DEBUG:teuthology.orchestra.run.smithi142:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T08:39:26.580 INFO:teuthology.orchestra.run.smithi142.stdout:19.0.0-2515.g96805dcf.el9 2024-05-09T08:39:26.580 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2515.g96805dcf.el9 2024-05-09T08:39:26.580 INFO:teuthology.task.install:The correct ceph version 19.0.0-2515.g96805dcf is installed. 2024-05-09T08:39:26.582 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-09T08:39:26.582 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:39:26.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T08:39:26.613 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-09T08:39:26.613 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T08:39:26.644 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-05-09T08:39:26.644 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T08:39:26.674 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-09T08:39:26.674 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:39:26.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T08:39:26.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T08:39:26.770 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-09T08:39:26.770 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T08:39:26.798 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T08:39:26.866 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-05-09T08:39:26.866 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T08:39:26.894 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T08:39:26.960 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-09T08:39:26.960 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:39:26.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T08:39:26.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T08:39:27.060 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-09T08:39:27.060 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T08:39:27.085 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T08:39:27.149 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-05-09T08:39:27.149 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T08:39:27.176 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T08:39:27.242 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-09T08:39:27.243 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:39:27.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T08:39:27.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T08:39:27.342 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-09T08:39:27.342 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T08:39:27.373 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T08:39:27.439 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-05-09T08:39:27.439 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T08:39:27.468 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T08:39:27.537 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-09T08:39:27.710 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-05-09T08:39:27.710 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:39:27.711 INFO:tasks.cephadm:Cluster fsid is a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:39:27.711 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-09T08:39:27.711 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.98', 'mon.b': '172.21.15.105', 'mon.c': '172.21.15.142'} 2024-05-09T08:39:27.712 INFO:tasks.cephadm:First mon is mon.a on smithi098 2024-05-09T08:39:27.712 INFO:tasks.cephadm:First mgr is x 2024-05-09T08:39:27.712 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-09T08:39:27.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-05-09T08:39:27.743 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname $(hostname -s) 2024-05-09T08:39:27.768 DEBUG:teuthology.orchestra.run.smithi142:> sudo hostname $(hostname -s) 2024-05-09T08:39:27.796 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-09T08:39:27.797 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:39:27.939 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55097/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2515-g96805dcf', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2515.g96805dcf'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-07 19:00:25.721799', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-09T08:39:28.077 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:39:28.079 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm 2024-05-09T08:39:28.079 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-09T08:39:29.593 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r--. 1 ubuntu ubuntu 782039 May 9 08:39 /home/ubuntu/cephtest/cephadm 2024-05-09T08:39:29.593 DEBUG:teuthology.orchestra.run.smithi105:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-09T08:39:30.713 INFO:teuthology.orchestra.run.smithi105.stdout:-rw-r--r--. 1 ubuntu ubuntu 782039 May 9 08:39 /home/ubuntu/cephtest/cephadm 2024-05-09T08:39:30.714 DEBUG:teuthology.orchestra.run.smithi142:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-09T08:39:32.004 INFO:teuthology.orchestra.run.smithi142.stdout:-rw-r--r--. 1 ubuntu ubuntu 782039 May 9 08:39 /home/ubuntu/cephtest/cephadm 2024-05-09T08:39:32.004 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-09T08:39:32.024 DEBUG:teuthology.orchestra.run.smithi105:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-09T08:39:32.042 DEBUG:teuthology.orchestra.run.smithi142:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-09T08:39:32.075 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc on all hosts... 2024-05-09T08:39:32.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc pull 2024-05-09T08:39:32.078 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc pull 2024-05-09T08:39:32.084 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc pull 2024-05-09T08:39:33.647 INFO:teuthology.orchestra.run.smithi105.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:39:33.694 INFO:teuthology.orchestra.run.smithi142.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:39:33.726 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:40:01.078 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-05-09T08:40:01.079 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T08:40:01.079 INFO:teuthology.orchestra.run.smithi105.stdout: "image_id": "d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662", 2024-05-09T08:40:01.079 INFO:teuthology.orchestra.run.smithi105.stdout: "repo_digests": [ 2024-05-09T08:40:01.079 INFO:teuthology.orchestra.run.smithi105.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac" 2024-05-09T08:40:01.079 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-05-09T08:40:01.079 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-05-09T08:40:06.990 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:40:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T08:40:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662", 2024-05-09T08:40:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-05-09T08:40:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac" 2024-05-09T08:40:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-05-09T08:40:06.991 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:40:08.186 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-05-09T08:40:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T08:40:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: "image_id": "d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662", 2024-05-09T08:40:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: "repo_digests": [ 2024-05-09T08:40:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac" 2024-05-09T08:40:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-05-09T08:40:08.186 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-05-09T08:40:08.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-05-09T08:40:08.244 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph 2024-05-09T08:40:08.276 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /etc/ceph 2024-05-09T08:40:08.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-05-09T08:40:08.332 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /etc/ceph 2024-05-09T08:40:08.359 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 777 /etc/ceph 2024-05-09T08:40:08.383 INFO:tasks.cephadm:Writing seed config... 2024-05-09T08:40:08.384 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-09T08:40:08.384 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-09T08:40:08.384 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-09T08:40:08.384 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-09T08:40:08.384 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-09T08:40:08.384 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-09T08:40:08.385 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-05-09T08:40:08.386 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-05-09T08:40:08.387 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-09T08:40:08.387 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-09T08:40:08.387 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-05-09T08:40:08.387 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-09T08:40:08.387 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-09T08:40:08.387 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-09T08:40:08.387 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-09T08:40:08.387 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-09T08:40:08.387 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-09T08:40:08.387 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-09T08:40:08.387 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-05-09T08:40:08.388 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:40:08.388 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-09T08:40:08.406 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a57d9712-0ddf-11ef-bc98-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-09T08:40:08.407 DEBUG:teuthology.orchestra.run.smithi098:mon.a> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.a.service 2024-05-09T08:40:08.450 DEBUG:teuthology.orchestra.run.smithi098:mgr.x> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mgr.x.service 2024-05-09T08:40:08.498 INFO:tasks.cephadm:Bootstrapping... 2024-05-09T08:40:08.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc -v bootstrap --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.98 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:40:08.693 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:40:08.693 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', '-v', 'bootstrap', '--fsid', 'a57d9712-0ddf-11ef-bc98-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.98', '--skip-admin-label'] 2024-05-09T08:40:08.694 INFO:teuthology.orchestra.run.smithi098.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-09T08:40:08.694 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying podman|docker is present... 2024-05-09T08:40:08.718 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-09T08:40:08.719 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying lvm2 is present... 2024-05-09T08:40:08.719 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying time synchronization is in place... 2024-05-09T08:40:08.726 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-09T08:40:08.726 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-09T08:40:08.732 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-09T08:40:08.733 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-05-09T08:40:08.739 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-05-09T08:40:08.745 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-05-09T08:40:08.745 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-05-09T08:40:08.745 INFO:teuthology.orchestra.run.smithi098.stdout:Repeating the final host check... 2024-05-09T08:40:08.770 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-09T08:40:08.770 INFO:teuthology.orchestra.run.smithi098.stdout:podman (/bin/podman) version 4.9.4 is present 2024-05-09T08:40:08.770 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl is present 2024-05-09T08:40:08.770 INFO:teuthology.orchestra.run.smithi098.stdout:lvcreate is present 2024-05-09T08:40:08.776 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-09T08:40:08.776 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-09T08:40:08.782 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-09T08:40:08.782 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-05-09T08:40:08.789 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-05-09T08:40:08.797 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-05-09T08:40:08.797 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-05-09T08:40:08.797 INFO:teuthology.orchestra.run.smithi098.stdout:Host looks OK 2024-05-09T08:40:08.797 INFO:teuthology.orchestra.run.smithi098.stdout:Cluster fsid: a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:40:08.798 INFO:teuthology.orchestra.run.smithi098.stdout:Acquiring lock 140588669739120 on /run/cephadm/a57d9712-0ddf-11ef-bc98-c7b262605968.lock 2024-05-09T08:40:08.798 INFO:teuthology.orchestra.run.smithi098.stdout:Lock 140588669739120 acquired on /run/cephadm/a57d9712-0ddf-11ef-bc98-c7b262605968.lock 2024-05-09T08:40:08.798 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 3300 ... 2024-05-09T08:40:08.798 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 6789 ... 2024-05-09T08:40:08.798 INFO:teuthology.orchestra.run.smithi098.stdout:Base mon IP(s) is [172.21.15.98:3300, 172.21.15.98:6789], mon addrv is [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-05-09T08:40:08.800 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.98 metric 100 2024-05-09T08:40:08.800 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.98 metric 100 2024-05-09T08:40:08.803 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-09T08:40:08.803 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-05-09T08:40:08.803 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1791sec hoplimit 64 pref medium 2024-05-09T08:40:08.806 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-09T08:40:08.806 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-09T08:40:08.806 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-09T08:40:08.806 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-09T08:40:08.806 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-05-09T08:40:08.806 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-09T08:40:08.807 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-05-09T08:40:08.807 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-05-09T08:40:08.807 INFO:teuthology.orchestra.run.smithi098.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-09T08:40:08.807 INFO:teuthology.orchestra.run.smithi098.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-09T08:40:08.807 INFO:teuthology.orchestra.run.smithi098.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:40:09.301 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 2024-05-09T08:40:09.302 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:40:09.302 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Getting image source signatures 2024-05-09T08:40:09.302 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:9852312f87792557815c3e860a5c4f8cbf2b59bbbde9b5f86ab07b438f905b2d 2024-05-09T08:40:09.302 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:bf5b95bd68251da39c6cd2eb96c6ba4138bb37a62bdc39287908b25e92047d7e 2024-05-09T08:40:09.302 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying config sha256:d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 2024-05-09T08:40:09.302 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-09T08:40:10.199 INFO:teuthology.orchestra.run.smithi098.stdout:ceph: stdout ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:40:10.200 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph version: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:40:10.200 INFO:teuthology.orchestra.run.smithi098.stdout:Extracting ceph user uid/gid from container image... 2024-05-09T08:40:11.259 INFO:teuthology.orchestra.run.smithi098.stdout:stat: stdout 167 167 2024-05-09T08:40:11.259 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial keys... 2024-05-09T08:40:12.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQDrizxm4q0DJBAAkc2nomBnwIJt7F8uJ3P3NA== 2024-05-09T08:40:13.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQDsizxmwIt5IhAAVdT5fH1eAYiusxVIlyiPpA== 2024-05-09T08:40:14.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQDtizxmWxK/GRAAagPeCee+6vYqJfMU54xp8A== 2024-05-09T08:40:14.129 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial monmap... 2024-05-09T08:40:15.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-09T08:40:15.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-09T08:40:15.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:40:15.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-09T08:40:15.092 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool for a [v2:172.21.15.98:3300,v1:172.21.15.98:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-09T08:40:15.092 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = quincy 2024-05-09T08:40:15.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: set fsid to a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:40:15.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-09T08:40:15.092 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:40:15.093 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mon... 2024-05-09T08:40:16.081 INFO:teuthology.orchestra.run.smithi098.stdout:create mon.a on 2024-05-09T08:40:16.284 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-09T08:40:16.427 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-09T08:40:16.600 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a57d9712-0ddf-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-a57d9712-0ddf-11ef-bc98-c7b262605968.target. 2024-05-09T08:40:16.600 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a57d9712-0ddf-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-a57d9712-0ddf-11ef-bc98-c7b262605968.target. 2024-05-09T08:40:16.848 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.a 2024-05-09T08:40:16.848 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.a.service: Unit ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.a.service not loaded. 2024-05-09T08:40:17.009 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a57d9712-0ddf-11ef-bc98-c7b262605968.target.wants/ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.a.service → /etc/systemd/system/ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@.service. 2024-05-09T08:40:17.490 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T08:40:17.490 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-05-09T08:40:17.497 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T08:40:17.497 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-05-09T08:40:17.497 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-05-09T08:40:17.498 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-05-09T08:40:17.498 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon to start... 2024-05-09T08:40:17.498 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon... 2024-05-09T08:40:17.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:17 smithi098 systemd[1]: Started Ceph mon.a for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:40:18.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:18 smithi098 ceph-mon[29718]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-09T08:40:18.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:18 smithi098 ceph-mon[29718]: monmap epoch 1 2024-05-09T08:40:18.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:18 smithi098 ceph-mon[29718]: fsid a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:40:18.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:18 smithi098 ceph-mon[29718]: last_changed 2024-05-09T08:40:14.468055+0000 2024-05-09T08:40:18.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:18 smithi098 ceph-mon[29718]: created 2024-05-09T08:40:14.468055+0000 2024-05-09T08:40:18.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:18 smithi098 ceph-mon[29718]: min_mon_release 19 (squid) 2024-05-09T08:40:18.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:18 smithi098 ceph-mon[29718]: election_strategy: 1 2024-05-09T08:40:18.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:18 smithi098 ceph-mon[29718]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-05-09T08:40:18.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:18 smithi098 ceph-mon[29718]: fsmap 2024-05-09T08:40:18.618 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:18 smithi098 ceph-mon[29718]: osdmap e1: 0 total, 0 up, 0 in 2024-05-09T08:40:18.618 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:18 smithi098 ceph-mon[29718]: mgrmap e1: no daemons active 2024-05-09T08:40:18.618 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:18 smithi098 ceph-mon[29718]: from='client.? 172.21.15.98:0/2756958663' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-09T08:40:18.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout cluster: 2024-05-09T08:40:18.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout id: a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:40:18.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-09T08:40:18.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-05-09T08:40:18.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout services: 2024-05-09T08:40:18.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.57125s) 2024-05-09T08:40:18.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-09T08:40:18.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-09T08:40:18.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-05-09T08:40:18.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout data: 2024-05-09T08:40:18.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-09T08:40:18.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-09T08:40:18.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-09T08:40:18.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pgs: 2024-05-09T08:40:18.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-05-09T08:40:18.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.946+0000 7f9e9615b700 1 Processor -- start 2024-05-09T08:40:18.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.946+0000 7f9e9615b700 1 -- start start 2024-05-09T08:40:18.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.946+0000 7f9e9615b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9007de70 0x7f9e9007c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:18.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.946+0000 7f9e9615b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9007c8f0 con 0x7f9e9007de70 2024-05-09T08:40:18.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.946+0000 7f9e8f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9007de70 0x7f9e9007c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:18.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.946+0000 7f9e8f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9007de70 0x7f9e9007c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35960/0 (socket says 172.21.15.98:35960) 2024-05-09T08:40:18.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.946+0000 7f9e8f7fe700 1 -- 172.21.15.98:0/474370343 learned_addr learned my addr 172.21.15.98:0/474370343 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.947+0000 7f9e8f7fe700 1 -- 172.21.15.98:0/474370343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9007ca70 con 0x7f9e9007de70 2024-05-09T08:40:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.947+0000 7f9e8f7fe700 1 --2- 172.21.15.98:0/474370343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9007de70 0x7f9e9007c320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9e800098a0 tx=0x7f9e80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11863db342610404 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.947+0000 7f9e8e7fc700 1 -- 172.21.15.98:0/474370343 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e80004160 con 0x7f9e9007de70 2024-05-09T08:40:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.947+0000 7f9e8e7fc700 1 -- 172.21.15.98:0/474370343 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e80004300 con 0x7f9e9007de70 2024-05-09T08:40:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.947+0000 7f9e8e7fc700 1 -- 172.21.15.98:0/474370343 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e800036a0 con 0x7f9e9007de70 2024-05-09T08:40:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.948+0000 7f9e9615b700 1 -- 172.21.15.98:0/474370343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9007de70 msgr2=0x7f9e9007c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.948+0000 7f9e9615b700 1 --2- 172.21.15.98:0/474370343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9007de70 0x7f9e9007c320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9e800098a0 tx=0x7f9e80009bb0 comp rx=0 tx=0).stop 2024-05-09T08:40:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.948+0000 7f9e9615b700 1 -- 172.21.15.98:0/474370343 shutdown_connections 2024-05-09T08:40:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.948+0000 7f9e9615b700 1 --2- 172.21.15.98:0/474370343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9007de70 0x7f9e9007c320 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.948+0000 7f9e9615b700 1 -- 172.21.15.98:0/474370343 >> 172.21.15.98:0/474370343 conn(0x7f9e901033a0 msgr2=0x7f9e901057c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.948+0000 7f9e9615b700 1 -- 172.21.15.98:0/474370343 shutdown_connections 2024-05-09T08:40:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.948+0000 7f9e9615b700 1 -- 172.21.15.98:0/474370343 wait complete. 2024-05-09T08:40:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.948+0000 7f9e9615b700 1 Processor -- start 2024-05-09T08:40:18.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.948+0000 7f9e9615b700 1 -- start start 2024-05-09T08:40:18.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.949+0000 7f9e9615b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9007de70 0x7f9e9019d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:18.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.949+0000 7f9e9615b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9007cfa0 con 0x7f9e9007de70 2024-05-09T08:40:18.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.949+0000 7f9e8f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9007de70 0x7f9e9019d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:18.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.949+0000 7f9e8f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9007de70 0x7f9e9019d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35972/0 (socket says 172.21.15.98:35972) 2024-05-09T08:40:18.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.949+0000 7f9e8f7fe700 1 -- 172.21.15.98:0/2756958663 learned_addr learned my addr 172.21.15.98:0/2756958663 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:18.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.949+0000 7f9e8f7fe700 1 -- 172.21.15.98:0/2756958663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9019dae0 con 0x7f9e9007de70 2024-05-09T08:40:18.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.949+0000 7f9e8f7fe700 1 --2- 172.21.15.98:0/2756958663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9007de70 0x7f9e9019d5a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9e80009fb0 tx=0x7f9e80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:18.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.949+0000 7f9e8cff9700 1 -- 172.21.15.98:0/2756958663 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e80003de0 con 0x7f9e9007de70 2024-05-09T08:40:18.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.949+0000 7f9e8cff9700 1 -- 172.21.15.98:0/2756958663 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e80011630 con 0x7f9e9007de70 2024-05-09T08:40:18.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.949+0000 7f9e9615b700 1 -- 172.21.15.98:0/2756958663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e9019dd70 con 0x7f9e9007de70 2024-05-09T08:40:18.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.949+0000 7f9e8cff9700 1 -- 172.21.15.98:0/2756958663 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e80011930 con 0x7f9e9007de70 2024-05-09T08:40:18.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.949+0000 7f9e9615b700 1 -- 172.21.15.98:0/2756958663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e901a0a60 con 0x7f9e9007de70 2024-05-09T08:40:18.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.950+0000 7f9e8cff9700 1 -- 172.21.15.98:0/2756958663 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9e80011ad0 con 0x7f9e9007de70 2024-05-09T08:40:18.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.950+0000 7f9e8cff9700 1 -- 172.21.15.98:0/2756958663 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9e8001b450 con 0x7f9e9007de70 2024-05-09T08:40:18.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.950+0000 7f9e9615b700 1 -- 172.21.15.98:0/2756958663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e90063980 con 0x7f9e9007de70 2024-05-09T08:40:18.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:17.953+0000 7f9e8cff9700 1 -- 172.21.15.98:0/2756958663 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9e80033020 con 0x7f9e9007de70 2024-05-09T08:40:18.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:18.022+0000 7f9e9615b700 1 -- 172.21.15.98:0/2756958663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f9e901a0f40 con 0x7f9e9007de70 2024-05-09T08:40:18.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:18.023+0000 7f9e8cff9700 1 -- 172.21.15.98:0/2756958663 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7f9e80004260 con 0x7f9e9007de70 2024-05-09T08:40:18.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:18.024+0000 7f9e9615b700 1 -- 172.21.15.98:0/2756958663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9007de70 msgr2=0x7f9e9019d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:18.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:18.024+0000 7f9e9615b700 1 --2- 172.21.15.98:0/2756958663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9007de70 0x7f9e9019d5a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9e80009fb0 tx=0x7f9e80004750 comp rx=0 tx=0).stop 2024-05-09T08:40:18.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:18.024+0000 7f9e9615b700 1 -- 172.21.15.98:0/2756958663 shutdown_connections 2024-05-09T08:40:18.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:18.024+0000 7f9e9615b700 1 --2- 172.21.15.98:0/2756958663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9007de70 0x7f9e9019d5a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:18.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:18.024+0000 7f9e9615b700 1 -- 172.21.15.98:0/2756958663 >> 172.21.15.98:0/2756958663 conn(0x7f9e901033a0 msgr2=0x7f9e901040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:18.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:18.024+0000 7f9e9615b700 1 -- 172.21.15.98:0/2756958663 shutdown_connections 2024-05-09T08:40:18.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:18.024+0000 7f9e9615b700 1 -- 172.21.15.98:0/2756958663 wait complete. 2024-05-09T08:40:18.629 INFO:teuthology.orchestra.run.smithi098.stdout:mon is available 2024-05-09T08:40:18.629 INFO:teuthology.orchestra.run.smithi098.stdout:Assimilating anything we can from ceph.conf... 2024-05-09T08:40:19.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:19 smithi098 ceph-mon[29718]: from='client.? 172.21.15.98:0/1290538390' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-09T08:40:19.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:19 smithi098 ceph-mon[29718]: from='client.? 172.21.15.98:0/1290538390' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-09T08:40:19.699 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-05-09T08:40:19.699 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-05-09T08:40:19.699 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:40:19.699 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-05-09T08:40:19.699 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-09T08:40:19.699 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-09T08:40:19.699 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-09T08:40:19.699 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-09T08:40:19.699 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-05-09T08:40:19.700 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-09T08:40:19.700 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-09T08:40:19.700 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-05-09T08:40:19.700 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-05-09T08:40:19.700 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-09T08:40:19.700 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-09T08:40:19.700 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-09T08:40:19.700 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-09T08:40:19.700 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-09T08:40:19.700 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.011+0000 7fc2e8ab2700 1 Processor -- start 2024-05-09T08:40:19.700 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.011+0000 7fc2e8ab2700 1 -- start start 2024-05-09T08:40:19.700 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.011+0000 7fc2e8ab2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2e4107e20 0x7fc2e4108240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:19.700 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.011+0000 7fc2e8ab2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e4108810 con 0x7fc2e4107e20 2024-05-09T08:40:19.700 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.012+0000 7fc2e259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2e4107e20 0x7fc2e4108240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:19.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.012+0000 7fc2e259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2e4107e20 0x7fc2e4108240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35988/0 (socket says 172.21.15.98:35988) 2024-05-09T08:40:19.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.012+0000 7fc2e259c700 1 -- 172.21.15.98:0/4069841951 learned_addr learned my addr 172.21.15.98:0/4069841951 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:19.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.012+0000 7fc2e259c700 1 -- 172.21.15.98:0/4069841951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e4108fd0 con 0x7fc2e4107e20 2024-05-09T08:40:19.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.012+0000 7fc2e259c700 1 --2- 172.21.15.98:0/4069841951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2e4107e20 0x7fc2e4108240 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc2d40098a0 tx=0x7fc2d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3bf8535968e140 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:19.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.012+0000 7fc2e1d9b700 1 -- 172.21.15.98:0/4069841951 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2d4004160 con 0x7fc2e4107e20 2024-05-09T08:40:19.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.012+0000 7fc2e1d9b700 1 -- 172.21.15.98:0/4069841951 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d4004300 con 0x7fc2e4107e20 2024-05-09T08:40:19.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.012+0000 7fc2e1d9b700 1 -- 172.21.15.98:0/4069841951 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2d40036a0 con 0x7fc2e4107e20 2024-05-09T08:40:19.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.013+0000 7fc2e8ab2700 1 -- 172.21.15.98:0/4069841951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2e4107e20 msgr2=0x7fc2e4108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:19.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.013+0000 7fc2e8ab2700 1 --2- 172.21.15.98:0/4069841951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2e4107e20 0x7fc2e4108240 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc2d40098a0 tx=0x7fc2d4009bb0 comp rx=0 tx=0).stop 2024-05-09T08:40:19.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.013+0000 7fc2e8ab2700 1 -- 172.21.15.98:0/4069841951 shutdown_connections 2024-05-09T08:40:19.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.013+0000 7fc2e8ab2700 1 --2- 172.21.15.98:0/4069841951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2e4107e20 0x7fc2e4108240 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:19.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.013+0000 7fc2e8ab2700 1 -- 172.21.15.98:0/4069841951 >> 172.21.15.98:0/4069841951 conn(0x7fc2e4103400 msgr2=0x7fc2e4105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:19.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.013+0000 7fc2e8ab2700 1 -- 172.21.15.98:0/4069841951 shutdown_connections 2024-05-09T08:40:19.701 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.013+0000 7fc2e8ab2700 1 -- 172.21.15.98:0/4069841951 wait complete. 2024-05-09T08:40:19.702 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.014+0000 7fc2e8ab2700 1 Processor -- start 2024-05-09T08:40:19.702 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.014+0000 7fc2e8ab2700 1 -- start start 2024-05-09T08:40:19.702 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.014+0000 7fc2e8ab2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2e4107e20 0x7fc2e419b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:19.702 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.014+0000 7fc2e8ab2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e4109500 con 0x7fc2e4107e20 2024-05-09T08:40:19.702 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.014+0000 7fc2e259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2e4107e20 0x7fc2e419b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:19.702 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.014+0000 7fc2e259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2e4107e20 0x7fc2e419b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36002/0 (socket says 172.21.15.98:36002) 2024-05-09T08:40:19.702 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.014+0000 7fc2e259c700 1 -- 172.21.15.98:0/1290538390 learned_addr learned my addr 172.21.15.98:0/1290538390 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:19.702 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.014+0000 7fc2e259c700 1 -- 172.21.15.98:0/1290538390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e419b980 con 0x7fc2e4107e20 2024-05-09T08:40:19.702 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.015+0000 7fc2e259c700 1 --2- 172.21.15.98:0/1290538390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2e4107e20 0x7fc2e419b440 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4000c00 tx=0x7fc2d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:19.702 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.015+0000 7fc2db7fe700 1 -- 172.21.15.98:0/1290538390 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2d4003db0 con 0x7fc2e4107e20 2024-05-09T08:40:19.702 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.015+0000 7fc2db7fe700 1 -- 172.21.15.98:0/1290538390 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d4011630 con 0x7fc2e4107e20 2024-05-09T08:40:19.702 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.015+0000 7fc2e8ab2700 1 -- 172.21.15.98:0/1290538390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2e419bc10 con 0x7fc2e4107e20 2024-05-09T08:40:19.702 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.015+0000 7fc2db7fe700 1 -- 172.21.15.98:0/1290538390 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2d4011950 con 0x7fc2e4107e20 2024-05-09T08:40:19.702 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.015+0000 7fc2e8ab2700 1 -- 172.21.15.98:0/1290538390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2e419e900 con 0x7fc2e4107e20 2024-05-09T08:40:19.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.015+0000 7fc2db7fe700 1 -- 172.21.15.98:0/1290538390 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc2d401a460 con 0x7fc2e4107e20 2024-05-09T08:40:19.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.015+0000 7fc2db7fe700 1 -- 172.21.15.98:0/1290538390 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc2d401b780 con 0x7fc2e4107e20 2024-05-09T08:40:19.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.016+0000 7fc2e8ab2700 1 -- 172.21.15.98:0/1290538390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2e4063980 con 0x7fc2e4107e20 2024-05-09T08:40:19.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.018+0000 7fc2db7fe700 1 -- 172.21.15.98:0/1290538390 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc2d4031020 con 0x7fc2e4107e20 2024-05-09T08:40:19.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.085+0000 7fc2e8ab2700 1 -- 172.21.15.98:0/1290538390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fc2e419c0f0 con 0x7fc2e4107e20 2024-05-09T08:40:19.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.088+0000 7fc2db7fe700 1 -- 172.21.15.98:0/1290538390 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7fc2e419c0f0 con 0x7fc2e4107e20 2024-05-09T08:40:19.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.089+0000 7fc2db7fe700 1 -- 172.21.15.98:0/1290538390 <== mon.0 v2:172.21.15.98:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc2d401a8e0 con 0x7fc2e4107e20 2024-05-09T08:40:19.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.089+0000 7fc2e8ab2700 1 -- 172.21.15.98:0/1290538390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2e4107e20 msgr2=0x7fc2e419b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:19.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.089+0000 7fc2e8ab2700 1 --2- 172.21.15.98:0/1290538390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2e4107e20 0x7fc2e419b440 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4000c00 tx=0x7fc2d4004750 comp rx=0 tx=0).stop 2024-05-09T08:40:19.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.089+0000 7fc2e8ab2700 1 -- 172.21.15.98:0/1290538390 shutdown_connections 2024-05-09T08:40:19.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.089+0000 7fc2e8ab2700 1 --2- 172.21.15.98:0/1290538390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2e4107e20 0x7fc2e419b440 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:19.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.089+0000 7fc2e8ab2700 1 -- 172.21.15.98:0/1290538390 >> 172.21.15.98:0/1290538390 conn(0x7fc2e4103400 msgr2=0x7fc2e41920e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:19.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.089+0000 7fc2e8ab2700 1 -- 172.21.15.98:0/1290538390 shutdown_connections 2024-05-09T08:40:19.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:19.089+0000 7fc2e8ab2700 1 -- 172.21.15.98:0/1290538390 wait complete. 2024-05-09T08:40:19.704 INFO:teuthology.orchestra.run.smithi098.stdout:Generating new minimal ceph.conf... 2024-05-09T08:40:20.713 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:20 smithi098 ceph-mon[29718]: from='client.? 172.21.15.98:0/3830644141' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:40:20.720 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.091+0000 7f9f80b36700 1 Processor -- start 2024-05-09T08:40:20.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.091+0000 7f9f80b36700 1 -- start start 2024-05-09T08:40:20.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.092+0000 7f9f80b36700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f7c1060d0 0x7f9f7c1064f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:20.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.092+0000 7f9f80b36700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7c106ac0 con 0x7f9f7c1060d0 2024-05-09T08:40:20.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.092+0000 7f9f7a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f7c1060d0 0x7f9f7c1064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:20.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.092+0000 7f9f7a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f7c1060d0 0x7f9f7c1064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36004/0 (socket says 172.21.15.98:36004) 2024-05-09T08:40:20.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.092+0000 7f9f7a59c700 1 -- 172.21.15.98:0/3454445864 learned_addr learned my addr 172.21.15.98:0/3454445864 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:20.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.092+0000 7f9f7a59c700 1 -- 172.21.15.98:0/3454445864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7c109060 con 0x7f9f7c1060d0 2024-05-09T08:40:20.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.092+0000 7f9f7a59c700 1 --2- 172.21.15.98:0/3454445864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f7c1060d0 0x7f9f7c1064f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c0098a0 tx=0x7f9f6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52fa619f1c55f0ac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:20.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.092+0000 7f9f7959a700 1 -- 172.21.15.98:0/3454445864 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f6c004160 con 0x7f9f7c1060d0 2024-05-09T08:40:20.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.093+0000 7f9f7959a700 1 -- 172.21.15.98:0/3454445864 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9f6c004300 con 0x7f9f7c1060d0 2024-05-09T08:40:20.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.093+0000 7f9f7959a700 1 -- 172.21.15.98:0/3454445864 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f6c011b50 con 0x7f9f7c1060d0 2024-05-09T08:40:20.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.093+0000 7f9f80b36700 1 -- 172.21.15.98:0/3454445864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f7c1060d0 msgr2=0x7f9f7c1064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:20.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.093+0000 7f9f80b36700 1 --2- 172.21.15.98:0/3454445864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f7c1060d0 0x7f9f7c1064f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c0098a0 tx=0x7f9f6c009bb0 comp rx=0 tx=0).stop 2024-05-09T08:40:20.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.093+0000 7f9f80b36700 1 -- 172.21.15.98:0/3454445864 shutdown_connections 2024-05-09T08:40:20.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.093+0000 7f9f80b36700 1 --2- 172.21.15.98:0/3454445864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f7c1060d0 0x7f9f7c1064f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:20.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.093+0000 7f9f80b36700 1 -- 172.21.15.98:0/3454445864 >> 172.21.15.98:0/3454445864 conn(0x7f9f7c101a20 msgr2=0x7f9f7c103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:20.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.093+0000 7f9f80b36700 1 -- 172.21.15.98:0/3454445864 shutdown_connections 2024-05-09T08:40:20.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.093+0000 7f9f80b36700 1 -- 172.21.15.98:0/3454445864 wait complete. 2024-05-09T08:40:20.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.094+0000 7f9f80b36700 1 Processor -- start 2024-05-09T08:40:20.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.094+0000 7f9f80b36700 1 -- start start 2024-05-09T08:40:20.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.094+0000 7f9f80b36700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f7c1060d0 0x7f9f7c19bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:20.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.094+0000 7f9f80b36700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7c10c410 con 0x7f9f7c1060d0 2024-05-09T08:40:20.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.094+0000 7f9f7a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f7c1060d0 0x7f9f7c19bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:20.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.094+0000 7f9f7a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f7c1060d0 0x7f9f7c19bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36006/0 (socket says 172.21.15.98:36006) 2024-05-09T08:40:20.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.094+0000 7f9f7a59c700 1 -- 172.21.15.98:0/3830644141 learned_addr learned my addr 172.21.15.98:0/3830644141 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:20.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.094+0000 7f9f7a59c700 1 -- 172.21.15.98:0/3830644141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7c19c1f0 con 0x7f9f7c1060d0 2024-05-09T08:40:20.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.095+0000 7f9f7a59c700 1 --2- 172.21.15.98:0/3830644141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f7c1060d0 0x7f9f7c19bcb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c011670 tx=0x7f9f6c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:20.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.095+0000 7f9f6b7fe700 1 -- 172.21.15.98:0/3830644141 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f6c0039c0 con 0x7f9f7c1060d0 2024-05-09T08:40:20.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.095+0000 7f9f6b7fe700 1 -- 172.21.15.98:0/3830644141 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9f6c011e50 con 0x7f9f7c1060d0 2024-05-09T08:40:20.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.095+0000 7f9f80b36700 1 -- 172.21.15.98:0/3830644141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f7c19c480 con 0x7f9f7c1060d0 2024-05-09T08:40:20.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.095+0000 7f9f6b7fe700 1 -- 172.21.15.98:0/3830644141 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f6c01a5c0 con 0x7f9f7c1060d0 2024-05-09T08:40:20.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.095+0000 7f9f80b36700 1 -- 172.21.15.98:0/3830644141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f7c19f170 con 0x7f9f7c1060d0 2024-05-09T08:40:20.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.095+0000 7f9f6b7fe700 1 -- 172.21.15.98:0/3830644141 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9f6c01ac00 con 0x7f9f7c1060d0 2024-05-09T08:40:20.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.095+0000 7f9f6b7fe700 1 -- 172.21.15.98:0/3830644141 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9f6c019700 con 0x7f9f7c1060d0 2024-05-09T08:40:20.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.096+0000 7f9f80b36700 1 -- 172.21.15.98:0/3830644141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f7c063980 con 0x7f9f7c1060d0 2024-05-09T08:40:20.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.098+0000 7f9f6b7fe700 1 -- 172.21.15.98:0/3830644141 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9f6c003b60 con 0x7f9f7c1060d0 2024-05-09T08:40:20.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.164+0000 7f9f80b36700 1 -- 172.21.15.98:0/3830644141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f9f7c19f340 con 0x7f9f7c1060d0 2024-05-09T08:40:20.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.165+0000 7f9f6b7fe700 1 -- 172.21.15.98:0/3830644141 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f9f6c0199a0 con 0x7f9f7c1060d0 2024-05-09T08:40:20.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.166+0000 7f9f80b36700 1 -- 172.21.15.98:0/3830644141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f7c1060d0 msgr2=0x7f9f7c19bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:20.725 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.166+0000 7f9f80b36700 1 --2- 172.21.15.98:0/3830644141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f7c1060d0 0x7f9f7c19bcb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c011670 tx=0x7f9f6c011b60 comp rx=0 tx=0).stop 2024-05-09T08:40:20.725 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.166+0000 7f9f80b36700 1 -- 172.21.15.98:0/3830644141 shutdown_connections 2024-05-09T08:40:20.725 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.166+0000 7f9f80b36700 1 --2- 172.21.15.98:0/3830644141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f7c1060d0 0x7f9f7c19bcb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:20.725 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.166+0000 7f9f80b36700 1 -- 172.21.15.98:0/3830644141 >> 172.21.15.98:0/3830644141 conn(0x7f9f7c101a20 msgr2=0x7f9f7c102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:20.725 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.166+0000 7f9f80b36700 1 -- 172.21.15.98:0/3830644141 shutdown_connections 2024-05-09T08:40:20.725 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:20.166+0000 7f9f80b36700 1 -- 172.21.15.98:0/3830644141 wait complete. 2024-05-09T08:40:20.725 INFO:teuthology.orchestra.run.smithi098.stdout:Restarting the monitor... 2024-05-09T08:40:21.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:20 smithi098 systemd[1]: Stopping Ceph mon.a for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:40:21.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:21 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a[29714]: 2024-05-09T08:40:21.001+0000 7f6c0548a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:40:21.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:21 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a[29714]: 2024-05-09T08:40:21.001+0000 7f6c0548a700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-09T08:40:21.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:21 smithi098 podman[29962]: 2024-05-09 08:40:21.145413822 +0000 UTC m=+0.264578756 container died 1a774360cfa65db5171bac158cf40d80436981d3273fe98ba17290a65a248d8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-09T08:40:21.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:21 smithi098 podman[29962]: 2024-05-09 08:40:21.501094839 +0000 UTC m=+0.620259764 container cleanup 1a774360cfa65db5171bac158cf40d80436981d3273fe98ba17290a65a248d8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc) 2024-05-09T08:40:21.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:21 smithi098 bash[29962]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a 2024-05-09T08:40:22.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:21 smithi098 podman[29976]: 2024-05-09 08:40:21.785705762 +0000 UTC m=+0.631951649 container remove 1a774360cfa65db5171bac158cf40d80436981d3273fe98ba17290a65a248d8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a, ceph=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:40:22.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:21 smithi098 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-09T08:40:22.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:21 smithi098 systemd[1]: Stopped Ceph mon.a for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:40:22.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:21 smithi098 systemd[1]: Starting Ceph mon.a for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:40:22.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 podman[30073]: 2024-05-09 08:40:22.136701763 +0000 UTC m=+0.098094141 container create 3ed83f274f2c63abef2cc7f6b6cf066f924ff174464da0af85b24ae57aedf276 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-96805dc, org.label-schema.license=GPLv2) 2024-05-09T08:40:22.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 podman[30073]: 2024-05-09 08:40:22.057036313 +0000 UTC m=+0.018428684 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:40:22.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 podman[30073]: 2024-05-09 08:40:22.322323974 +0000 UTC m=+0.283716349 container init 3ed83f274f2c63abef2cc7f6b6cf066f924ff174464da0af85b24ae57aedf276 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.build-date=20240507) 2024-05-09T08:40:22.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 podman[30073]: 2024-05-09 08:40:22.325426436 +0000 UTC m=+0.286818806 container start 3ed83f274f2c63abef2cc7f6b6cf066f924ff174464da0af85b24ae57aedf276 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True) 2024-05-09T08:40:22.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: set uid:gid to 167:167 (ceph:ceph) 2024-05-09T08:40:22.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev), process ceph-mon, pid 2 2024-05-09T08:40:22.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: pidfile_write: ignore empty --pid-file 2024-05-09T08:40:22.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: load: jerasure load: lrc 2024-05-09T08:40:22.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: RocksDB version: 7.9.2 2024-05-09T08:40:22.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Git sha 0 2024-05-09T08:40:22.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Compile date 2024-05-07 19:19:25 2024-05-09T08:40:22.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: DB SUMMARY 2024-05-09T08:40:22.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: DB Session ID: CNN2OM9VCNAY7F62OZ9B 2024-05-09T08:40:22.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: CURRENT file: CURRENT 2024-05-09T08:40:22.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: IDENTITY file: IDENTITY 2024-05-09T08:40:22.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-09T08:40:22.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-09T08:40:22.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105318 ; 2024-05-09T08:40:22.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.error_if_exists: 0 2024-05-09T08:40:22.398 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.create_if_missing: 0 2024-05-09T08:40:22.398 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.paranoid_checks: 1 2024-05-09T08:40:22.398 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-09T08:40:22.398 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-09T08:40:22.398 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-09T08:40:22.398 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.env: 0x557f20751c60 2024-05-09T08:40:22.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.fs: PosixFileSystem 2024-05-09T08:40:22.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.info_log: 0x557f214a7ac0 2024-05-09T08:40:22.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_file_opening_threads: 16 2024-05-09T08:40:22.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.statistics: (nil) 2024-05-09T08:40:22.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.use_fsync: 0 2024-05-09T08:40:22.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_log_file_size: 0 2024-05-09T08:40:22.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-09T08:40:22.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-09T08:40:22.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.keep_log_file_num: 1000 2024-05-09T08:40:22.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.recycle_log_file_num: 0 2024-05-09T08:40:22.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.allow_fallocate: 1 2024-05-09T08:40:22.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.allow_mmap_reads: 0 2024-05-09T08:40:22.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.allow_mmap_writes: 0 2024-05-09T08:40:22.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.use_direct_reads: 0 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.create_missing_column_families: 0 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.db_log_dir: 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.wal_dir: 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.advise_random_on_open: 1 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.db_write_buffer_size: 0 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.write_buffer_manager: 0x557f2147a1e0 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-09T08:40:22.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.rate_limiter: (nil) 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.wal_recovery_mode: 2 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.enable_thread_tracking: 0 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.enable_pipelined_write: 0 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.unordered_write: 0 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.row_cache: None 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.wal_filter: None 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-09T08:40:22.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.allow_ingest_behind: 0 2024-05-09T08:40:22.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.two_write_queues: 0 2024-05-09T08:40:22.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.manual_wal_flush: 0 2024-05-09T08:40:22.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.wal_compression: 0 2024-05-09T08:40:22.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.atomic_flush: 0 2024-05-09T08:40:22.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-09T08:40:22.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-09T08:40:22.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-09T08:40:22.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.log_readahead_size: 0 2024-05-09T08:40:22.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-09T08:40:22.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.best_efforts_recovery: 0 2024-05-09T08:40:22.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-09T08:40:22.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-09T08:40:22.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.allow_data_in_errors: 0 2024-05-09T08:40:22.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.db_host_id: __hostname__ 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_background_jobs: 2 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_background_compactions: -1 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_subcompactions: 1 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_total_wal_size: 0 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_open_files: -1 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.bytes_per_sync: 0 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-09T08:40:22.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compaction_readahead_size: 0 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_background_flushes: -1 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Compression algorithms supported: 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: kZSTD supported: 0 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: kXpressCompression supported: 0 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: kLZ4HCCompression supported: 1 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: kZlibCompression supported: 1 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: kSnappyCompression supported: 1 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: kLZ4Compression supported: 1 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: kBZip2Compression supported: 0 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-09T08:40:22.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-09T08:40:22.405 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-09T08:40:22.405 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.merge_operator: 2024-05-09T08:40:22.405 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compaction_filter: None 2024-05-09T08:40:22.405 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compaction_filter_factory: None 2024-05-09T08:40:22.405 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.sst_partitioner_factory: None 2024-05-09T08:40:22.405 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-09T08:40:22.405 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-09T08:40:22.405 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557f214a7bc0) 2024-05-09T08:40:22.405 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-05-09T08:40:22.405 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-09T08:40:22.405 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-09T08:40:22.405 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-05-09T08:40:22.405 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_type: 0 2024-05-09T08:40:22.405 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_index_type: 0 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_shortening: 1 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: checksum: 4 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: no_block_cache: 0 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache: 0x557f214ab090 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_options: 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: capacity : 536870912 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_shard_bits : 4 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: strict_capacity_limit : 0 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_compressed: (nil) 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: persistent_cache: (nil) 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size: 4096 2024-05-09T08:40:22.406 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size_deviation: 10 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_restart_interval: 16 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_block_restart_interval: 1 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: metadata_block_size: 4096 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: partition_filters: 0 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: use_delta_encoding: 1 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: filter_policy: bloomfilter 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: whole_key_filtering: 1 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: verify_compression: 0 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: format_version: 5 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_index_compression: 1 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_align: 0 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: max_auto_readahead_size: 262144 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: prepopulate_block_cache: 0 2024-05-09T08:40:22.407 INFO:journalctl@ceph.mon.a.smithi098.stdout: initial_auto_readahead_size: 8192 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.write_buffer_size: 33554432 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_write_buffer_number: 2 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compression: NoCompression 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.bottommost_compression: Disabled 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.prefix_extractor: nullptr 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.num_levels: 7 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-09T08:40:22.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compression_opts.level: 32767 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compression_opts.strategy: 0 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compression_opts.enabled: false 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-09T08:40:22.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-09T08:40:22.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-09T08:40:22.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-09T08:40:22.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.target_file_size_base: 67108864 2024-05-09T08:40:22.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-09T08:40:22.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-09T08:40:22.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-09T08:40:22.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.arena_block_size: 1048576 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.disable_auto_compactions: 0 2024-05-09T08:40:22.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.table_properties_collectors: 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.inplace_update_support: 0 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-09T08:40:22.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.bloom_locality: 0 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.max_successive_merges: 0 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.paranoid_file_checks: 0 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.force_consistency_checks: 1 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.report_bg_io_stats: 0 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.ttl: 2592000 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.enable_blob_files: false 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.min_blob_size: 0 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.blob_file_size: 268435456 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-09T08:40:22.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-09T08:40:22.414 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-09T08:40:22.414 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-09T08:40:22.414 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-09T08:40:22.414 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.blob_file_starting_level: 0 2024-05-09T08:40:22.414 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-09T08:40:22.414 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-09T08:40:22.414 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-09T08:40:22.414 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6d4b66b7-77a9-40ef-a00b-04d3309dba8d 2024-05-09T08:40:22.414 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715244022355829, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-09T08:40:22.414 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-09T08:40:22.414 INFO:teuthology.orchestra.run.smithi098.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-09T08:40:22.415 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715244022356772, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100361, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98281, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14306, "raw_average_key_size": 49, "raw_value_size": 90527, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715244022, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6d4b66b7-77a9-40ef-a00b-04d3309dba8d", "db_session_id": "CNN2OM9VCNAY7F62OZ9B", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-09T08:40:22.415 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715244022356853, "job": 1, "event": "recovery_finished"} 2024-05-09T08:40:22.415 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-09T08:40:22.415 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-09T08:40:22.415 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557f2159c000 2024-05-09T08:40:22.415 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: DB pointer 0x557f21586000 2024-05-09T08:40:22.415 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-09T08:40:22.415 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-09T08:40:22.415 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** DB Stats ** 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: L0 2/0 99.84 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: Sum 2/0 99.84 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T08:40:22.416 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 165.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative compaction: 0.00 GB write, 32.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T08:40:22.417 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval compaction: 0.00 GB write, 32.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache BinnedLRUCache@0x557f214ab090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: starting mon.a rank 0 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: mon.a@-1(???) e1 preinit fsid a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: mon.a@-1(???).mds e1 new map 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: mon.a@-1(???).mds e1 print_map 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout: e1 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout: legacy client fscid: -1 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-05-09T08:40:22.418 INFO:journalctl@ceph.mon.a.smithi098.stdout: No filesystems configured 2024-05-09T08:40:22.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-09T08:40:22.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:40:22.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:40:22.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:40:22.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-09T08:40:22.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 bash[30073]: 3ed83f274f2c63abef2cc7f6b6cf066f924ff174464da0af85b24ae57aedf276 2024-05-09T08:40:22.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 systemd[1]: Started Ceph mon.a for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:40:22.690 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-09T08:40:22.690 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: monmap epoch 1 2024-05-09T08:40:22.690 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: fsid a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:40:22.691 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: last_changed 2024-05-09T08:40:14.468055+0000 2024-05-09T08:40:22.691 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: created 2024-05-09T08:40:14.468055+0000 2024-05-09T08:40:22.691 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: min_mon_release 19 (squid) 2024-05-09T08:40:22.691 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: election_strategy: 1 2024-05-09T08:40:22.691 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-05-09T08:40:22.691 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: fsmap 2024-05-09T08:40:22.691 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: osdmap e1: 0 total, 0 up, 0 in 2024-05-09T08:40:22.691 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:22 smithi098 ceph-mon[30092]: mgrmap e1: no daemons active 2024-05-09T08:40:23.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.796+0000 7f58313cc700 1 Processor -- start 2024-05-09T08:40:23.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.796+0000 7f58313cc700 1 -- start start 2024-05-09T08:40:23.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.796+0000 7f58313cc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f582c108840 0x7f582c10ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:23.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.796+0000 7f58313cc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f582c07bed0 con 0x7f582c108840 2024-05-09T08:40:23.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.796+0000 7f582affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f582c108840 0x7f582c10ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:23.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.796+0000 7f582affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f582c108840 0x7f582c10ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36014/0 (socket says 172.21.15.98:36014) 2024-05-09T08:40:23.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.796+0000 7f582affd700 1 -- 172.21.15.98:0/700904180 learned_addr learned my addr 172.21.15.98:0/700904180 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:23.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.797+0000 7f582affd700 1 -- 172.21.15.98:0/700904180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f582c10b170 con 0x7f582c108840 2024-05-09T08:40:23.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.797+0000 7f582affd700 1 --2- 172.21.15.98:0/700904180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f582c108840 0x7f582c10ac30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f581c0098a0 tx=0x7f581c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48f891404a591c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.797+0000 7f5829ffb700 1 -- 172.21.15.98:0/700904180 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f581c004160 con 0x7f582c108840 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.797+0000 7f5829ffb700 1 -- 172.21.15.98:0/700904180 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f581c004300 con 0x7f582c108840 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.797+0000 7f5829ffb700 1 -- 172.21.15.98:0/700904180 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f581c011b50 con 0x7f582c108840 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.798+0000 7f58313cc700 1 -- 172.21.15.98:0/700904180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f582c108840 msgr2=0x7f582c10ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.798+0000 7f58313cc700 1 --2- 172.21.15.98:0/700904180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f582c108840 0x7f582c10ac30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f581c0098a0 tx=0x7f581c009bb0 comp rx=0 tx=0).stop 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.798+0000 7f58313cc700 1 -- 172.21.15.98:0/700904180 shutdown_connections 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.798+0000 7f58313cc700 1 --2- 172.21.15.98:0/700904180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f582c108840 0x7f582c10ac30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.798+0000 7f58313cc700 1 -- 172.21.15.98:0/700904180 >> 172.21.15.98:0/700904180 conn(0x7f582c102230 msgr2=0x7f582c1046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.798+0000 7f58313cc700 1 -- 172.21.15.98:0/700904180 shutdown_connections 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.798+0000 7f58313cc700 1 -- 172.21.15.98:0/700904180 wait complete. 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.798+0000 7f58313cc700 1 Processor -- start 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.798+0000 7f58313cc700 1 -- start start 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.799+0000 7f58313cc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f582c108840 0x7f582c19bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.799+0000 7f58313cc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f582c10c1a0 con 0x7f582c108840 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.799+0000 7f582affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f582c108840 0x7f582c19bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.799+0000 7f582affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f582c108840 0x7f582c19bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36024/0 (socket says 172.21.15.98:36024) 2024-05-09T08:40:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.799+0000 7f582affd700 1 -- 172.21.15.98:0/2674765875 learned_addr learned my addr 172.21.15.98:0/2674765875 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.799+0000 7f582affd700 1 -- 172.21.15.98:0/2674765875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f582c19c1c0 con 0x7f582c108840 2024-05-09T08:40:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.799+0000 7f582affd700 1 --2- 172.21.15.98:0/2674765875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f582c108840 0x7f582c19bc80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f581c011670 tx=0x7f581c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.799+0000 7f5813fff700 1 -- 172.21.15.98:0/2674765875 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f581c0039c0 con 0x7f582c108840 2024-05-09T08:40:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.799+0000 7f58313cc700 1 -- 172.21.15.98:0/2674765875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f582c19c450 con 0x7f582c108840 2024-05-09T08:40:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.799+0000 7f5813fff700 1 -- 172.21.15.98:0/2674765875 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f581c011e50 con 0x7f582c108840 2024-05-09T08:40:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.799+0000 7f58313cc700 1 -- 172.21.15.98:0/2674765875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f582c19f140 con 0x7f582c108840 2024-05-09T08:40:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.799+0000 7f5813fff700 1 -- 172.21.15.98:0/2674765875 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f581c004300 con 0x7f582c108840 2024-05-09T08:40:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.800+0000 7f5813fff700 1 -- 172.21.15.98:0/2674765875 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f581c01a7c0 con 0x7f582c108840 2024-05-09T08:40:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.800+0000 7f5813fff700 1 -- 172.21.15.98:0/2674765875 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f581c019700 con 0x7f582c108840 2024-05-09T08:40:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.800+0000 7f58313cc700 1 -- 172.21.15.98:0/2674765875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f582c063980 con 0x7f582c108840 2024-05-09T08:40:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.803+0000 7f5813fff700 1 -- 172.21.15.98:0/2674765875 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f581c003b60 con 0x7f582c108840 2024-05-09T08:40:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.869+0000 7f58313cc700 1 -- 172.21.15.98:0/2674765875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f582c19f310 con 0x7f582c108840 2024-05-09T08:40:23.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.873+0000 7f5813fff700 1 -- 172.21.15.98:0/2674765875 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f581c032030 con 0x7f582c108840 2024-05-09T08:40:23.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.874+0000 7f58313cc700 1 -- 172.21.15.98:0/2674765875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f582c108840 msgr2=0x7f582c19bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:23.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.874+0000 7f58313cc700 1 --2- 172.21.15.98:0/2674765875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f582c108840 0x7f582c19bc80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f581c011670 tx=0x7f581c011b60 comp rx=0 tx=0).stop 2024-05-09T08:40:23.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.874+0000 7f58313cc700 1 -- 172.21.15.98:0/2674765875 shutdown_connections 2024-05-09T08:40:23.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.874+0000 7f58313cc700 1 --2- 172.21.15.98:0/2674765875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f582c108840 0x7f582c19bc80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:23.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.874+0000 7f58313cc700 1 -- 172.21.15.98:0/2674765875 >> 172.21.15.98:0/2674765875 conn(0x7f582c102230 msgr2=0x7f582c102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:23.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.874+0000 7f58313cc700 1 -- 172.21.15.98:0/2674765875 shutdown_connections 2024-05-09T08:40:23.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:22.874+0000 7f58313cc700 1 -- 172.21.15.98:0/2674765875 wait complete. 2024-05-09T08:40:23.387 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-09T08:40:23.410 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:40:23.410 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mgr... 2024-05-09T08:40:23.410 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-09T08:40:23.411 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-09T08:40:23.683 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mgr.x 2024-05-09T08:40:23.683 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mgr.x.service: Unit ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mgr.x.service not loaded. 2024-05-09T08:40:23.831 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a57d9712-0ddf-11ef-bc98-c7b262605968.target.wants/ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@.service. 2024-05-09T08:40:24.176 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:23 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2674765875' entity='client.admin' 2024-05-09T08:40:24.354 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T08:40:24.354 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-05-09T08:40:24.361 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T08:40:24.361 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-05-09T08:40:24.361 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-05-09T08:40:24.361 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-05-09T08:40:24.370 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T08:40:24.370 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-05-09T08:40:24.377 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T08:40:24.377 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-05-09T08:40:24.378 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-05-09T08:40:24.378 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-09T08:40:24.378 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr to start... 2024-05-09T08:40:24.378 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr... 2024-05-09T08:40:25.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:24 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/826560479' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-09T08:40:25.088 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:25 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:25.047+0000 7f2a331c2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:40:25.488 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:25 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:25.134+0000 7f2a331c2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:40:25.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-05-09T08:40:25.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:40:25.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "a57d9712-0ddf-11ef-bc98-c7b262605968", 2024-05-09T08:40:25.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-05-09T08:40:25.493 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-09T08:40:25.494 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-09T08:40:25.495 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:25.495 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-09T08:40:25.495 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:40:25.495 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-09T08:40:25.495 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-09T08:40:25.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-09T08:40:25.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-09T08:40:25.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-09T08:40:25.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-09T08:40:25.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:25.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-09T08:40:25.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-09T08:40:25.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-09T08:40:25.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-09T08:40:25.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-09T08:40:25.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-09T08:40:25.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-09T08:40:25.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-05-09T08:40:25.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-05-09T08:40:17.456508+0000", 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.845+0000 7fc01154a700 1 Processor -- start 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.845+0000 7fc01154a700 1 -- start start 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.846+0000 7fc01154a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c1088c0 0x7fc00c10acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.846+0000 7fc01154a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c07be90 con 0x7fc00c1088c0 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.846+0000 7fc00affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c1088c0 0x7fc00c10acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.846+0000 7fc00affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c1088c0 0x7fc00c10acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36038/0 (socket says 172.21.15.98:36038) 2024-05-09T08:40:25.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.846+0000 7fc00affd700 1 -- 172.21.15.98:0/4214311029 learned_addr learned my addr 172.21.15.98:0/4214311029 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.846+0000 7fc00affd700 1 -- 172.21.15.98:0/4214311029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00c10b1f0 con 0x7fc00c1088c0 2024-05-09T08:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.846+0000 7fc00affd700 1 --2- 172.21.15.98:0/4214311029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c1088c0 0x7fc00c10acb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0098a0 tx=0x7fbffc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83fe50fc4af6afc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.847+0000 7fc009ffb700 1 -- 172.21.15.98:0/4214311029 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbffc004160 con 0x7fc00c1088c0 2024-05-09T08:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.847+0000 7fc009ffb700 1 -- 172.21.15.98:0/4214311029 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbffc004300 con 0x7fc00c1088c0 2024-05-09T08:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.847+0000 7fc009ffb700 1 -- 172.21.15.98:0/4214311029 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbffc011b50 con 0x7fc00c1088c0 2024-05-09T08:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.847+0000 7fc01154a700 1 -- 172.21.15.98:0/4214311029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c1088c0 msgr2=0x7fc00c10acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.847+0000 7fc01154a700 1 --2- 172.21.15.98:0/4214311029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c1088c0 0x7fc00c10acb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0098a0 tx=0x7fbffc009bb0 comp rx=0 tx=0).stop 2024-05-09T08:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.847+0000 7fc01154a700 1 -- 172.21.15.98:0/4214311029 shutdown_connections 2024-05-09T08:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.847+0000 7fc01154a700 1 --2- 172.21.15.98:0/4214311029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c1088c0 0x7fc00c10acb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.847+0000 7fc01154a700 1 -- 172.21.15.98:0/4214311029 >> 172.21.15.98:0/4214311029 conn(0x7fc00c1022b0 msgr2=0x7fc00c1046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.847+0000 7fc01154a700 1 -- 172.21.15.98:0/4214311029 shutdown_connections 2024-05-09T08:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.847+0000 7fc01154a700 1 -- 172.21.15.98:0/4214311029 wait complete. 2024-05-09T08:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.848+0000 7fc01154a700 1 Processor -- start 2024-05-09T08:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.848+0000 7fc01154a700 1 -- start start 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.848+0000 7fc01154a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c1088c0 0x7fc00c19bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.848+0000 7fc01154a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c10c210 con 0x7fc00c1088c0 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.848+0000 7fc00affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c1088c0 0x7fc00c19bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.849+0000 7fc00affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c1088c0 0x7fc00c19bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36040/0 (socket says 172.21.15.98:36040) 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.849+0000 7fc00affd700 1 -- 172.21.15.98:0/826560479 learned_addr learned my addr 172.21.15.98:0/826560479 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.849+0000 7fc00affd700 1 -- 172.21.15.98:0/826560479 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00c19c200 con 0x7fc00c1088c0 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.849+0000 7fc00affd700 1 --2- 172.21.15.98:0/826560479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c1088c0 0x7fc00c19bcc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbffc011b60 tx=0x7fbffc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.849+0000 7fbff3fff700 1 -- 172.21.15.98:0/826560479 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbffc003b20 con 0x7fc00c1088c0 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.849+0000 7fc01154a700 1 -- 172.21.15.98:0/826560479 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc00c19c490 con 0x7fc00c1088c0 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.849+0000 7fbff3fff700 1 -- 172.21.15.98:0/826560479 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbffc003cc0 con 0x7fc00c1088c0 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.849+0000 7fbff3fff700 1 -- 172.21.15.98:0/826560479 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbffc024840 con 0x7fc00c1088c0 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.849+0000 7fc01154a700 1 -- 172.21.15.98:0/826560479 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc00c19f180 con 0x7fc00c1088c0 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.849+0000 7fbff3fff700 1 -- 172.21.15.98:0/826560479 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbffc022020 con 0x7fc00c1088c0 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.850+0000 7fbff3fff700 1 -- 172.21.15.98:0/826560479 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbffc011e00 con 0x7fc00c1088c0 2024-05-09T08:40:25.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.850+0000 7fc01154a700 1 -- 172.21.15.98:0/826560479 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc00c063980 con 0x7fc00c1088c0 2024-05-09T08:40:25.508 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.854+0000 7fbff3fff700 1 -- 172.21.15.98:0/826560479 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fbffc043dd0 con 0x7fc00c1088c0 2024-05-09T08:40:25.508 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.924+0000 7fc01154a700 1 -- 172.21.15.98:0/826560479 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc00c19f350 con 0x7fc00c1088c0 2024-05-09T08:40:25.508 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.925+0000 7fbff3fff700 1 -- 172.21.15.98:0/826560479 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fbffc032020 con 0x7fc00c1088c0 2024-05-09T08:40:25.508 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.926+0000 7fc01154a700 1 -- 172.21.15.98:0/826560479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c1088c0 msgr2=0x7fc00c19bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:25.508 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.926+0000 7fc01154a700 1 --2- 172.21.15.98:0/826560479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c1088c0 0x7fc00c19bcc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbffc011b60 tx=0x7fbffc011c40 comp rx=0 tx=0).stop 2024-05-09T08:40:25.508 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.926+0000 7fc01154a700 1 -- 172.21.15.98:0/826560479 shutdown_connections 2024-05-09T08:40:25.508 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.926+0000 7fc01154a700 1 --2- 172.21.15.98:0/826560479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c1088c0 0x7fc00c19bcc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:25.508 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.926+0000 7fc01154a700 1 -- 172.21.15.98:0/826560479 >> 172.21.15.98:0/826560479 conn(0x7fc00c1022b0 msgr2=0x7fc00c102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:25.508 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.926+0000 7fc01154a700 1 -- 172.21.15.98:0/826560479 shutdown_connections 2024-05-09T08:40:25.508 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:24.926+0000 7fc01154a700 1 -- 172.21.15.98:0/826560479 wait complete. 2024-05-09T08:40:25.508 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (1/15)... 2024-05-09T08:40:26.693 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:26 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:26.345+0000 7f2a331c2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:40:26.693 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:26 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:26.518+0000 7f2a331c2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:40:26.693 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:26 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:26.694+0000 7f2a331c2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:40:27.274 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:26 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:26.982+0000 7f2a331c2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:40:27.274 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:27 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:27.075+0000 7f2a331c2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:40:27.274 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:27 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:27.275+0000 7f2a331c2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:40:27.587 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:27 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:27.466+0000 7f2a331c2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:40:28.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:28 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3774978134' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-09T08:40:28.337 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:28 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:28.105+0000 7f2a331c2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:40:28.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-05-09T08:40:28.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:40:28.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "a57d9712-0ddf-11ef-bc98-c7b262605968", 2024-05-09T08:40:28.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-05-09T08:40:28.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-09T08:40:28.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-09T08:40:28.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-09T08:40:28.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:28.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-09T08:40:28.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-09T08:40:28.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-05-09T08:40:28.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:40:28.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-09T08:40:28.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-09T08:40:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-09T08:40:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-09T08:40:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-09T08:40:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-09T08:40:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-09T08:40:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-09T08:40:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-09T08:40:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-09T08:40:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-09T08:40:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-09T08:40:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-09T08:40:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:40:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-09T08:40:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-09T08:40:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-09T08:40:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-09T08:40:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-09T08:40:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-09T08:40:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-09T08:40:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-09T08:40:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-05-09T08:40:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:40:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:40:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-09T08:40:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:40:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-05-09T08:40:17.456508+0000", 2024-05-09T08:40:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:40:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-09T08:40:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:40:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.918+0000 7f2a5afdd700 1 Processor -- start 2024-05-09T08:40:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.918+0000 7f2a5afdd700 1 -- start start 2024-05-09T08:40:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.918+0000 7f2a5afdd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a541093b0 0x7f2a541097d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.918+0000 7f2a5afdd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a54109da0 con 0x7f2a541093b0 2024-05-09T08:40:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.918+0000 7f2a58d79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a541093b0 0x7f2a541097d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.918+0000 7f2a58d79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a541093b0 0x7f2a541097d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49016/0 (socket says 172.21.15.98:49016) 2024-05-09T08:40:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.918+0000 7f2a58d79700 1 -- 172.21.15.98:0/2927773054 learned_addr learned my addr 172.21.15.98:0/2927773054 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.918+0000 7f2a58d79700 1 -- 172.21.15.98:0/2927773054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a54109f20 con 0x7f2a541093b0 2024-05-09T08:40:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.919+0000 7f2a58d79700 1 --2- 172.21.15.98:0/2927773054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a541093b0 0x7f2a541097d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2a480098a0 tx=0x7f2a48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780ac060338ec0c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.919+0000 7f2a537fe700 1 -- 172.21.15.98:0/2927773054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a48004160 con 0x7f2a541093b0 2024-05-09T08:40:28.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.919+0000 7f2a537fe700 1 -- 172.21.15.98:0/2927773054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2a48004300 con 0x7f2a541093b0 2024-05-09T08:40:28.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.919+0000 7f2a537fe700 1 -- 172.21.15.98:0/2927773054 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a48011b50 con 0x7f2a541093b0 2024-05-09T08:40:28.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.919+0000 7f2a5afdd700 1 -- 172.21.15.98:0/2927773054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a541093b0 msgr2=0x7f2a541097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.919+0000 7f2a5afdd700 1 --2- 172.21.15.98:0/2927773054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a541093b0 0x7f2a541097d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2a480098a0 tx=0x7f2a48009bb0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.920+0000 7f2a5afdd700 1 -- 172.21.15.98:0/2927773054 shutdown_connections 2024-05-09T08:40:28.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.920+0000 7f2a5afdd700 1 --2- 172.21.15.98:0/2927773054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a541093b0 0x7f2a541097d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.920+0000 7f2a5afdd700 1 -- 172.21.15.98:0/2927773054 >> 172.21.15.98:0/2927773054 conn(0x7f2a541019f0 msgr2=0x7f2a54103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.920+0000 7f2a5afdd700 1 -- 172.21.15.98:0/2927773054 shutdown_connections 2024-05-09T08:40:28.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.920+0000 7f2a5afdd700 1 -- 172.21.15.98:0/2927773054 wait complete. 2024-05-09T08:40:28.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.920+0000 7f2a5afdd700 1 Processor -- start 2024-05-09T08:40:28.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.920+0000 7f2a5afdd700 1 -- start start 2024-05-09T08:40:28.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.920+0000 7f2a5afdd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a541093b0 0x7f2a541089d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.920+0000 7f2a5afdd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a54106c10 con 0x7f2a541093b0 2024-05-09T08:40:28.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.920+0000 7f2a58d79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a541093b0 0x7f2a541089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.920+0000 7f2a58d79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a541093b0 0x7f2a541089d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49020/0 (socket says 172.21.15.98:49020) 2024-05-09T08:40:28.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.920+0000 7f2a58d79700 1 -- 172.21.15.98:0/3774978134 learned_addr learned my addr 172.21.15.98:0/3774978134 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:28.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.921+0000 7f2a58d79700 1 -- 172.21.15.98:0/3774978134 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a54108f10 con 0x7f2a541093b0 2024-05-09T08:40:28.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.921+0000 7f2a58d79700 1 --2- 172.21.15.98:0/3774978134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a541093b0 0x7f2a541089d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2a48011670 tx=0x7f2a48011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.921+0000 7f2a51ffb700 1 -- 172.21.15.98:0/3774978134 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a480039c0 con 0x7f2a541093b0 2024-05-09T08:40:28.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.921+0000 7f2a5afdd700 1 -- 172.21.15.98:0/3774978134 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a54107100 con 0x7f2a541093b0 2024-05-09T08:40:28.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.921+0000 7f2a51ffb700 1 -- 172.21.15.98:0/3774978134 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2a48011e50 con 0x7f2a541093b0 2024-05-09T08:40:28.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.921+0000 7f2a51ffb700 1 -- 172.21.15.98:0/3774978134 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a4801a5c0 con 0x7f2a541093b0 2024-05-09T08:40:28.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.921+0000 7f2a5afdd700 1 -- 172.21.15.98:0/3774978134 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a54107640 con 0x7f2a541093b0 2024-05-09T08:40:28.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.921+0000 7f2a51ffb700 1 -- 172.21.15.98:0/3774978134 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2a4801ac50 con 0x7f2a541093b0 2024-05-09T08:40:28.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.922+0000 7f2a51ffb700 1 -- 172.21.15.98:0/3774978134 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2a48019700 con 0x7f2a541093b0 2024-05-09T08:40:28.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.922+0000 7f2a5afdd700 1 -- 172.21.15.98:0/3774978134 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a54063980 con 0x7f2a541093b0 2024-05-09T08:40:28.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:27.925+0000 7f2a51ffb700 1 -- 172.21.15.98:0/3774978134 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2a48032080 con 0x7f2a541093b0 2024-05-09T08:40:28.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:28.017+0000 7f2a5afdd700 1 -- 172.21.15.98:0/3774978134 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2a54107bd0 con 0x7f2a541093b0 2024-05-09T08:40:28.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:28.018+0000 7f2a51ffb700 1 -- 172.21.15.98:0/3774978134 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2a48003b60 con 0x7f2a541093b0 2024-05-09T08:40:28.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:28.019+0000 7f2a5afdd700 1 -- 172.21.15.98:0/3774978134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a541093b0 msgr2=0x7f2a541089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:28.019+0000 7f2a5afdd700 1 --2- 172.21.15.98:0/3774978134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a541093b0 0x7f2a541089d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2a48011670 tx=0x7f2a48011b60 comp rx=0 tx=0).stop 2024-05-09T08:40:28.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:28.019+0000 7f2a5afdd700 1 -- 172.21.15.98:0/3774978134 shutdown_connections 2024-05-09T08:40:28.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:28.019+0000 7f2a5afdd700 1 --2- 172.21.15.98:0/3774978134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a541093b0 0x7f2a541089d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:28.019+0000 7f2a5afdd700 1 -- 172.21.15.98:0/3774978134 >> 172.21.15.98:0/3774978134 conn(0x7f2a541019f0 msgr2=0x7f2a541024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:28.019+0000 7f2a5afdd700 1 -- 172.21.15.98:0/3774978134 shutdown_connections 2024-05-09T08:40:28.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:28.019+0000 7f2a5afdd700 1 -- 172.21.15.98:0/3774978134 wait complete. 2024-05-09T08:40:28.676 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (2/15)... 2024-05-09T08:40:28.829 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:28 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:28.531+0000 7f2a331c2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:40:28.830 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:28 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:28.629+0000 7f2a331c2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:40:28.830 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:28 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:28.830+0000 7f2a331c2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:40:29.668 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:29 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:29.342+0000 7f2a331c2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:40:29.668 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:29 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:29.433+0000 7f2a331c2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:40:30.007 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:29 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:29.669+0000 7f2a331c2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:40:30.007 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:29 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:29.800+0000 7f2a331c2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:40:30.008 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:29 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:29.893+0000 7f2a331c2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:40:30.008 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:30 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:30.008+0000 7f2a331c2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:40:30.587 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:30 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:30.310+0000 7f2a331c2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:40:31.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:30 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:30.912+0000 7f2a331c2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:40:31.587 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:31 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:31.126+0000 7f2a331c2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:40:31.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:31 smithi098 ceph-mon[30092]: Activating manager daemon x 2024-05-09T08:40:31.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:31 smithi098 ceph-mon[30092]: mgrmap e2: x(active, starting, since 0.00308788s) 2024-05-09T08:40:31.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:31 smithi098 ceph-mon[30092]: from='mgr.14100 172.21.15.98:0/320818652' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:40:31.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:31 smithi098 ceph-mon[30092]: from='mgr.14100 172.21.15.98:0/320818652' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:40:31.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:31 smithi098 ceph-mon[30092]: from='mgr.14100 172.21.15.98:0/320818652' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:40:31.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:31 smithi098 ceph-mon[30092]: from='mgr.14100 172.21.15.98:0/320818652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:40:31.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:31 smithi098 ceph-mon[30092]: from='mgr.14100 172.21.15.98:0/320818652' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:40:31.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:31 smithi098 ceph-mon[30092]: Manager daemon x is now available 2024-05-09T08:40:31.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:31 smithi098 ceph-mon[30092]: from='mgr.14100 172.21.15.98:0/320818652' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:40:31.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:31 smithi098 ceph-mon[30092]: from='mgr.14100 172.21.15.98:0/320818652' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:40:31.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:31 smithi098 ceph-mon[30092]: from='mgr.14100 172.21.15.98:0/320818652' entity='mgr.x' 2024-05-09T08:40:31.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:31 smithi098 ceph-mon[30092]: from='mgr.14100 172.21.15.98:0/320818652' entity='mgr.x' 2024-05-09T08:40:31.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:31 smithi098 ceph-mon[30092]: from='mgr.14100 172.21.15.98:0/320818652' entity='mgr.x' 2024-05-09T08:40:33.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:33 smithi098 ceph-mon[30092]: mgrmap e3: x(active, since 1.0069s) 2024-05-09T08:40:34.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:34 smithi098 ceph-mon[30092]: mgrmap e4: x(active, since 2s) 2024-05-09T08:40:35.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:35 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1921806051' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-09T08:40:40.416 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-05-09T08:40:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:40:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "a57d9712-0ddf-11ef-bc98-c7b262605968", 2024-05-09T08:40:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-05-09T08:40:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-09T08:40:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-09T08:40:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-09T08:40:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-09T08:40:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-09T08:40:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-05-09T08:40:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:40:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-09T08:40:40.417 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-05-09T08:40:40.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:40:40.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-05-09T08:40:40.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-09T08:40:40.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:40:40.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-09T08:40:40.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-09T08:40:40.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:40.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-09T08:40:40.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:40:40.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-09T08:40:40.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-09T08:40:40.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-09T08:40:40.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-09T08:40:40.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-09T08:40:40.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-09T08:40:40.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:40.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-09T08:40:40.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-09T08:40:40.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-09T08:40:40.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-09T08:40:40.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-09T08:40:40.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-05-09T08:40:17.456508+0000", 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.147+0000 7fef91a39700 1 Processor -- start 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.147+0000 7fef91a39700 1 -- start start 2024-05-09T08:40:40.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.147+0000 7fef91a39700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef8c108840 0x7fef8c10ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:40.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.147+0000 7fef91a39700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c07bed0 con 0x7fef8c108840 2024-05-09T08:40:40.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.147+0000 7fef8affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef8c108840 0x7fef8c10ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:40.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.147+0000 7fef8affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef8c108840 0x7fef8c10ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49106/0 (socket says 172.21.15.98:49106) 2024-05-09T08:40:40.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.147+0000 7fef8affd700 1 -- 172.21.15.98:0/287626544 learned_addr learned my addr 172.21.15.98:0/287626544 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:40.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.147+0000 7fef8affd700 1 -- 172.21.15.98:0/287626544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8c10b170 con 0x7fef8c108840 2024-05-09T08:40:40.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.147+0000 7fef8affd700 1 --2- 172.21.15.98:0/287626544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef8c108840 0x7fef8c10ac30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0098a0 tx=0x7fef7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=273125317a40928 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:40.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.148+0000 7fef89ffb700 1 -- 172.21.15.98:0/287626544 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef7c004160 con 0x7fef8c108840 2024-05-09T08:40:40.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.148+0000 7fef89ffb700 1 -- 172.21.15.98:0/287626544 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fef7c004300 con 0x7fef8c108840 2024-05-09T08:40:40.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.148+0000 7fef89ffb700 1 -- 172.21.15.98:0/287626544 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef7c011b50 con 0x7fef8c108840 2024-05-09T08:40:40.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.148+0000 7fef91a39700 1 -- 172.21.15.98:0/287626544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef8c108840 msgr2=0x7fef8c10ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:40.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.148+0000 7fef91a39700 1 --2- 172.21.15.98:0/287626544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef8c108840 0x7fef8c10ac30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0098a0 tx=0x7fef7c009bb0 comp rx=0 tx=0).stop 2024-05-09T08:40:40.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.148+0000 7fef91a39700 1 -- 172.21.15.98:0/287626544 shutdown_connections 2024-05-09T08:40:40.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.148+0000 7fef91a39700 1 --2- 172.21.15.98:0/287626544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef8c108840 0x7fef8c10ac30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:40.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.148+0000 7fef91a39700 1 -- 172.21.15.98:0/287626544 >> 172.21.15.98:0/287626544 conn(0x7fef8c102270 msgr2=0x7fef8c1046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.148+0000 7fef91a39700 1 -- 172.21.15.98:0/287626544 shutdown_connections 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.148+0000 7fef91a39700 1 -- 172.21.15.98:0/287626544 wait complete. 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.149+0000 7fef91a39700 1 Processor -- start 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.149+0000 7fef91a39700 1 -- start start 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.149+0000 7fef91a39700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef8c108840 0x7fef8c19bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.149+0000 7fef91a39700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c10c1a0 con 0x7fef8c108840 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.149+0000 7fef8affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef8c108840 0x7fef8c19bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.149+0000 7fef8affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef8c108840 0x7fef8c19bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49110/0 (socket says 172.21.15.98:49110) 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.149+0000 7fef8affd700 1 -- 172.21.15.98:0/1921806051 learned_addr learned my addr 172.21.15.98:0/1921806051 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.150+0000 7fef8affd700 1 -- 172.21.15.98:0/1921806051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8c19c1c0 con 0x7fef8c108840 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.150+0000 7fef8affd700 1 --2- 172.21.15.98:0/1921806051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef8c108840 0x7fef8c19bc80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fef7c011b60 tx=0x7fef7c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.150+0000 7fef90a37700 1 -- 172.21.15.98:0/1921806051 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef7c003b20 con 0x7fef8c108840 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.150+0000 7fef90a37700 1 -- 172.21.15.98:0/1921806051 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fef7c003cc0 con 0x7fef8c108840 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.150+0000 7fef91a39700 1 -- 172.21.15.98:0/1921806051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef8c19c450 con 0x7fef8c108840 2024-05-09T08:40:40.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.150+0000 7fef90a37700 1 -- 172.21.15.98:0/1921806051 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef7c024840 con 0x7fef8c108840 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.150+0000 7fef91a39700 1 -- 172.21.15.98:0/1921806051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef8c19f140 con 0x7fef8c108840 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.151+0000 7fef90a37700 1 -- 172.21.15.98:0/1921806051 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fef7c022020 con 0x7fef8c108840 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.151+0000 7fef90a37700 1 --2- 172.21.15.98:0/1921806051 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7fef740390c0 0x7fef7403b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.151+0000 7fef90a37700 1 -- 172.21.15.98:0/1921806051 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fef7c011e00 con 0x7fef8c108840 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.151+0000 7fef8a7fc700 1 --2- 172.21.15.98:0/1921806051 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7fef740390c0 0x7fef7403b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.151+0000 7fef91a39700 1 -- 172.21.15.98:0/1921806051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef8c063980 con 0x7fef8c108840 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.151+0000 7fef8a7fc700 1 --2- 172.21.15.98:0/1921806051 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7fef740390c0 0x7fef7403b580 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fef80006fd0 tx=0x7fef80006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.156+0000 7fef90a37700 1 -- 172.21.15.98:0/1921806051 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fef7c0172b0 con 0x7fef8c108840 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.434+0000 7fef91a39700 1 -- 172.21.15.98:0/1921806051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fef8c19f310 con 0x7fef8c108840 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.434+0000 7fef90a37700 1 -- 172.21.15.98:0/1921806051 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fef7c00b7e0 con 0x7fef8c108840 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.436+0000 7fef91a39700 1 -- 172.21.15.98:0/1921806051 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7fef740390c0 msgr2=0x7fef7403b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.436+0000 7fef91a39700 1 --2- 172.21.15.98:0/1921806051 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7fef740390c0 0x7fef7403b580 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fef80006fd0 tx=0x7fef80006e40 comp rx=0 tx=0).stop 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.436+0000 7fef91a39700 1 -- 172.21.15.98:0/1921806051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef8c108840 msgr2=0x7fef8c19bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.436+0000 7fef91a39700 1 --2- 172.21.15.98:0/1921806051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef8c108840 0x7fef8c19bc80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fef7c011b60 tx=0x7fef7c011c40 comp rx=0 tx=0).stop 2024-05-09T08:40:40.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.436+0000 7fef91a39700 1 -- 172.21.15.98:0/1921806051 shutdown_connections 2024-05-09T08:40:40.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.436+0000 7fef91a39700 1 --2- 172.21.15.98:0/1921806051 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7fef740390c0 0x7fef7403b580 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:40.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.436+0000 7fef91a39700 1 --2- 172.21.15.98:0/1921806051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef8c108840 0x7fef8c19bc80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:40.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.436+0000 7fef91a39700 1 -- 172.21.15.98:0/1921806051 >> 172.21.15.98:0/1921806051 conn(0x7fef8c102270 msgr2=0x7fef8c102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:40.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.437+0000 7fef91a39700 1 -- 172.21.15.98:0/1921806051 shutdown_connections 2024-05-09T08:40:40.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:34.437+0000 7fef91a39700 1 -- 172.21.15.98:0/1921806051 wait complete. 2024-05-09T08:40:40.433 INFO:teuthology.orchestra.run.smithi098.stdout:mgr is available 2024-05-09T08:40:42.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:42 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1585110852' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-09T08:40:43.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-05-09T08:40:43.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-05-09T08:40:43.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:40:43.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-09T08:40:43.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.389+0000 7f1b73cdd700 1 Processor -- start 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.389+0000 7f1b73cdd700 1 -- start start 2024-05-09T08:40:43.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.389+0000 7f1b73cdd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6c106d10 0x7f1b6c107130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.389+0000 7f1b73cdd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6c107700 con 0x7f1b6c106d10 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.390+0000 7f1b71a79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6c106d10 0x7f1b6c107130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.390+0000 7f1b71a79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6c106d10 0x7f1b6c107130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36106/0 (socket says 172.21.15.98:36106) 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.390+0000 7f1b71a79700 1 -- 172.21.15.98:0/1316177201 learned_addr learned my addr 172.21.15.98:0/1316177201 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.390+0000 7f1b71a79700 1 -- 172.21.15.98:0/1316177201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6c107f10 con 0x7f1b6c106d10 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.390+0000 7f1b71a79700 1 --2- 172.21.15.98:0/1316177201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6c106d10 0x7f1b6c107130 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1b680098a0 tx=0x7f1b68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd1788efb41dac6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.390+0000 7f1b70a77700 1 -- 172.21.15.98:0/1316177201 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b68004160 con 0x7f1b6c106d10 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.390+0000 7f1b70a77700 1 -- 172.21.15.98:0/1316177201 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1b68004300 con 0x7f1b6c106d10 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.390+0000 7f1b70a77700 1 -- 172.21.15.98:0/1316177201 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b68011b50 con 0x7f1b6c106d10 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.391+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1316177201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6c106d10 msgr2=0x7f1b6c107130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.391+0000 7f1b73cdd700 1 --2- 172.21.15.98:0/1316177201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6c106d10 0x7f1b6c107130 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1b680098a0 tx=0x7f1b68009bb0 comp rx=0 tx=0).stop 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.391+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1316177201 shutdown_connections 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.391+0000 7f1b73cdd700 1 --2- 172.21.15.98:0/1316177201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6c106d10 0x7f1b6c107130 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.391+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1316177201 >> 172.21.15.98:0/1316177201 conn(0x7f1b6c1022b0 msgr2=0x7f1b6c1046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.391+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1316177201 shutdown_connections 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.391+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1316177201 wait complete. 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.392+0000 7f1b73cdd700 1 Processor -- start 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.392+0000 7f1b73cdd700 1 -- start start 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.392+0000 7f1b73cdd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6c106d10 0x7f1b6c19bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.392+0000 7f1b73cdd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6c10b2c0 con 0x7f1b6c106d10 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.392+0000 7f1b71a79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6c106d10 0x7f1b6c19bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.392+0000 7f1b71a79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6c106d10 0x7f1b6c19bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36110/0 (socket says 172.21.15.98:36110) 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.392+0000 7f1b71a79700 1 -- 172.21.15.98:0/1585110852 learned_addr learned my addr 172.21.15.98:0/1585110852 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.392+0000 7f1b71a79700 1 -- 172.21.15.98:0/1585110852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6c19c2d0 con 0x7f1b6c106d10 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.392+0000 7f1b71a79700 1 --2- 172.21.15.98:0/1585110852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6c106d10 0x7f1b6c19bd90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1b68000c00 tx=0x7f1b68011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.393+0000 7f1b62ffd700 1 -- 172.21.15.98:0/1585110852 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b68003b20 con 0x7f1b6c106d10 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.393+0000 7f1b62ffd700 1 -- 172.21.15.98:0/1585110852 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1b68003cc0 con 0x7f1b6c106d10 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.393+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1585110852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b6c19c560 con 0x7f1b6c106d10 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.393+0000 7f1b62ffd700 1 -- 172.21.15.98:0/1585110852 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b68024910 con 0x7f1b6c106d10 2024-05-09T08:40:43.162 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.393+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1585110852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b6c19f210 con 0x7f1b6c106d10 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.393+0000 7f1b62ffd700 1 -- 172.21.15.98:0/1585110852 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f1b68022020 con 0x7f1b6c106d10 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.394+0000 7f1b62ffd700 1 --2- 172.21.15.98:0/1585110852 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f1b58039110 0x7f1b5803b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.394+0000 7f1b62ffd700 1 -- 172.21.15.98:0/1585110852 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1b68011e00 con 0x7f1b6c106d10 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.394+0000 7f1b71278700 1 --2- 172.21.15.98:0/1585110852 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f1b58039110 0x7f1b5803b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.394+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1585110852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b6c063980 con 0x7f1b6c106d10 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.394+0000 7f1b71278700 1 --2- 172.21.15.98:0/1585110852 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f1b58039110 0x7f1b5803b5d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c006fd0 tx=0x7f1b5c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.399+0000 7f1b62ffd700 1 -- 172.21.15.98:0/1585110852 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1b6800b3e0 con 0x7f1b6c106d10 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.595+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1585110852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f1b6c02de00 con 0x7f1b6c106d10 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.596+0000 7f1b62ffd700 1 -- 172.21.15.98:0/1585110852 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f1b6802cda0 con 0x7f1b6c106d10 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.599+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1585110852 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f1b58039110 msgr2=0x7f1b5803b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.599+0000 7f1b73cdd700 1 --2- 172.21.15.98:0/1585110852 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f1b58039110 0x7f1b5803b5d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c006fd0 tx=0x7f1b5c006e40 comp rx=0 tx=0).stop 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.599+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1585110852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6c106d10 msgr2=0x7f1b6c19bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.599+0000 7f1b73cdd700 1 --2- 172.21.15.98:0/1585110852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6c106d10 0x7f1b6c19bd90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1b68000c00 tx=0x7f1b68011c10 comp rx=0 tx=0).stop 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.599+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1585110852 shutdown_connections 2024-05-09T08:40:43.163 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.599+0000 7f1b73cdd700 1 --2- 172.21.15.98:0/1585110852 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f1b58039110 0x7f1b5803b5d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:43.164 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.599+0000 7f1b73cdd700 1 --2- 172.21.15.98:0/1585110852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b6c106d10 0x7f1b6c19bd90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:43.164 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.599+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1585110852 >> 172.21.15.98:0/1585110852 conn(0x7f1b6c1022b0 msgr2=0x7f1b6c1030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:43.164 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.599+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1585110852 shutdown_connections 2024-05-09T08:40:43.164 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:42.599+0000 7f1b73cdd700 1 -- 172.21.15.98:0/1585110852 wait complete. 2024-05-09T08:40:43.164 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling cephadm module... 2024-05-09T08:40:44.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:43 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1245728039' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-09T08:40:44.088 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:43 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: ignoring --setuser ceph since I am not root 2024-05-09T08:40:44.088 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:43 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: ignoring --setgroup ceph since I am not root 2024-05-09T08:40:44.088 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:44 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:44.076+0000 7f5225504200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:40:44.465 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:44 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:44.261+0000 7f5225504200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:40:44.466 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:44 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:44.362+0000 7f5225504200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:40:44.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.573+0000 7f9f88a82700 1 Processor -- start 2024-05-09T08:40:44.475 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.573+0000 7f9f88a82700 1 -- start start 2024-05-09T08:40:44.475 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.573+0000 7f9f88a82700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f84106080 0x7f9f841064a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:44.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.573+0000 7f9f88a82700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9f84106a70 con 0x7f9f84106080 2024-05-09T08:40:44.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.574+0000 7f9f8259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f84106080 0x7f9f841064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:44.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.574+0000 7f9f8259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f84106080 0x7f9f841064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36126/0 (socket says 172.21.15.98:36126) 2024-05-09T08:40:44.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.574+0000 7f9f8259c700 1 -- 172.21.15.98:0/2887031176 learned_addr learned my addr 172.21.15.98:0/2887031176 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:44.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.574+0000 7f9f8259c700 1 -- 172.21.15.98:0/2887031176 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f84106df0 con 0x7f9f84106080 2024-05-09T08:40:44.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.574+0000 7f9f8259c700 1 --2- 172.21.15.98:0/2887031176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f84106080 0x7f9f841064a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9f780098a0 tx=0x7f9f78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972d8188fc72f337 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:44.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.574+0000 7f9f8159a700 1 -- 172.21.15.98:0/2887031176 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f78004160 con 0x7f9f84106080 2024-05-09T08:40:44.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.575+0000 7f9f8159a700 1 -- 172.21.15.98:0/2887031176 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9f78004300 con 0x7f9f84106080 2024-05-09T08:40:44.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.575+0000 7f9f8159a700 1 -- 172.21.15.98:0/2887031176 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f78011b50 con 0x7f9f84106080 2024-05-09T08:40:44.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.575+0000 7f9f88a82700 1 -- 172.21.15.98:0/2887031176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f84106080 msgr2=0x7f9f841064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:44.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.575+0000 7f9f88a82700 1 --2- 172.21.15.98:0/2887031176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f84106080 0x7f9f841064a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9f780098a0 tx=0x7f9f78009bb0 comp rx=0 tx=0).stop 2024-05-09T08:40:44.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.575+0000 7f9f88a82700 1 -- 172.21.15.98:0/2887031176 shutdown_connections 2024-05-09T08:40:44.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.575+0000 7f9f88a82700 1 --2- 172.21.15.98:0/2887031176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f84106080 0x7f9f841064a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:44.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.575+0000 7f9f88a82700 1 -- 172.21.15.98:0/2887031176 >> 172.21.15.98:0/2887031176 conn(0x7f9f84101a00 msgr2=0x7f9f84103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.575+0000 7f9f88a82700 1 -- 172.21.15.98:0/2887031176 shutdown_connections 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.575+0000 7f9f88a82700 1 -- 172.21.15.98:0/2887031176 wait complete. 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.576+0000 7f9f88a82700 1 Processor -- start 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.576+0000 7f9f88a82700 1 -- start start 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.576+0000 7f9f88a82700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f84106080 0x7f9f8419deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.576+0000 7f9f88a82700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9f841081b0 con 0x7f9f84106080 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.576+0000 7f9f8259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f84106080 0x7f9f8419deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.576+0000 7f9f8259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f84106080 0x7f9f8419deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36130/0 (socket says 172.21.15.98:36130) 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.576+0000 7f9f8259c700 1 -- 172.21.15.98:0/1245728039 learned_addr learned my addr 172.21.15.98:0/1245728039 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.577+0000 7f9f8259c700 1 -- 172.21.15.98:0/1245728039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8419e3f0 con 0x7f9f84106080 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.577+0000 7f9f8259c700 1 --2- 172.21.15.98:0/1245728039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f84106080 0x7f9f8419deb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9f78011670 tx=0x7f9f78011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.577+0000 7f9f737fe700 1 -- 172.21.15.98:0/1245728039 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f780039c0 con 0x7f9f84106080 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.577+0000 7f9f88a82700 1 -- 172.21.15.98:0/1245728039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f8419e680 con 0x7f9f84106080 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.577+0000 7f9f737fe700 1 -- 172.21.15.98:0/1245728039 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9f78011e50 con 0x7f9f84106080 2024-05-09T08:40:44.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.577+0000 7f9f737fe700 1 -- 172.21.15.98:0/1245728039 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f7801a5c0 con 0x7f9f84106080 2024-05-09T08:40:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.577+0000 7f9f88a82700 1 -- 172.21.15.98:0/1245728039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f841a1370 con 0x7f9f84106080 2024-05-09T08:40:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.578+0000 7f9f737fe700 1 -- 172.21.15.98:0/1245728039 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f9f7801a8c0 con 0x7f9f84106080 2024-05-09T08:40:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.578+0000 7f9f737fe700 1 --2- 172.21.15.98:0/1245728039 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f9f6c039070 0x7f9f6c03b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.578+0000 7f9f737fe700 1 -- 172.21.15.98:0/1245728039 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9f7804d2a0 con 0x7f9f84106080 2024-05-09T08:40:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.578+0000 7f9f81d9b700 1 --2- 172.21.15.98:0/1245728039 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f9f6c039070 0x7f9f6c03b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.578+0000 7f9f88a82700 1 -- 172.21.15.98:0/1245728039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f84063980 con 0x7f9f84106080 2024-05-09T08:40:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.579+0000 7f9f81d9b700 1 --2- 172.21.15.98:0/1245728039 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f9f6c039070 0x7f9f6c03b530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9f74006fd0 tx=0x7f9f74006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.588+0000 7f9f737fe700 1 -- 172.21.15.98:0/1245728039 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9f7802cc00 con 0x7f9f84106080 2024-05-09T08:40:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.850+0000 7f9f88a82700 1 -- 172.21.15.98:0/1245728039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f9f84197410 con 0x7f9f84106080 2024-05-09T08:40:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.901+0000 7f9f737fe700 1 -- 172.21.15.98:0/1245728039 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f9f84197410 con 0x7f9f84106080 2024-05-09T08:40:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.903+0000 7f9f737fe700 1 -- 172.21.15.98:0/1245728039 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f9f7802d910 con 0x7f9f84106080 2024-05-09T08:40:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.904+0000 7f9f88a82700 1 -- 172.21.15.98:0/1245728039 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f9f6c039070 msgr2=0x7f9f6c03b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.904+0000 7f9f88a82700 1 --2- 172.21.15.98:0/1245728039 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f9f6c039070 0x7f9f6c03b530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9f74006fd0 tx=0x7f9f74006e40 comp rx=0 tx=0).stop 2024-05-09T08:40:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.904+0000 7f9f88a82700 1 -- 172.21.15.98:0/1245728039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f84106080 msgr2=0x7f9f8419deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:44.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.904+0000 7f9f88a82700 1 --2- 172.21.15.98:0/1245728039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f84106080 0x7f9f8419deb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9f78011670 tx=0x7f9f78011b60 comp rx=0 tx=0).stop 2024-05-09T08:40:44.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.904+0000 7f9f88a82700 1 -- 172.21.15.98:0/1245728039 shutdown_connections 2024-05-09T08:40:44.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.904+0000 7f9f88a82700 1 --2- 172.21.15.98:0/1245728039 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f9f6c039070 0x7f9f6c03b530 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:44.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.904+0000 7f9f88a82700 1 --2- 172.21.15.98:0/1245728039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f84106080 0x7f9f8419deb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:44.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.904+0000 7f9f88a82700 1 -- 172.21.15.98:0/1245728039 >> 172.21.15.98:0/1245728039 conn(0x7f9f84101a00 msgr2=0x7f9f841944d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:44.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.904+0000 7f9f88a82700 1 -- 172.21.15.98:0/1245728039 shutdown_connections 2024-05-09T08:40:44.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:43.904+0000 7f9f88a82700 1 -- 172.21.15.98:0/1245728039 wait complete. 2024-05-09T08:40:44.725 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:44 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:44.678+0000 7f5225504200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:40:45.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:44 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:44.767+0000 7f5225504200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:40:45.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:44 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1245728039' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-09T08:40:45.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:44 smithi098 ceph-mon[30092]: mgrmap e5: x(active, since 12s) 2024-05-09T08:40:46.162 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:45 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3492762355' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-09T08:40:46.163 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:45 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:45.990+0000 7f5225504200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:40:46.163 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:46 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:46.163+0000 7f5225504200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:40:46.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:40:46.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-09T08:40:46.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-09T08:40:46.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-09T08:40:46.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-09T08:40:46.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:40:46.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.930+0000 7f3d71103700 1 Processor -- start 2024-05-09T08:40:46.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.930+0000 7f3d71103700 1 -- start start 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.930+0000 7f3d71103700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d6c108840 0x7f3d6c10ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.930+0000 7f3d71103700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6c07beb0 con 0x7f3d6c108840 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.931+0000 7f3d6ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d6c108840 0x7f3d6c10ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.931+0000 7f3d6ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d6c108840 0x7f3d6c10ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36164/0 (socket says 172.21.15.98:36164) 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.931+0000 7f3d6ad9d700 1 -- 172.21.15.98:0/1129637234 learned_addr learned my addr 172.21.15.98:0/1129637234 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.931+0000 7f3d6ad9d700 1 -- 172.21.15.98:0/1129637234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6c10b170 con 0x7f3d6c108840 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.931+0000 7f3d6ad9d700 1 --2- 172.21.15.98:0/1129637234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d6c108840 0x7f3d6c10ac30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c0098a0 tx=0x7f3d5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f965dacf366d4070 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.931+0000 7f3d69d9b700 1 -- 172.21.15.98:0/1129637234 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d5c004160 con 0x7f3d6c108840 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.931+0000 7f3d69d9b700 1 -- 172.21.15.98:0/1129637234 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3d5c004300 con 0x7f3d6c108840 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.931+0000 7f3d69d9b700 1 -- 172.21.15.98:0/1129637234 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d5c011b50 con 0x7f3d6c108840 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.932+0000 7f3d71103700 1 -- 172.21.15.98:0/1129637234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d6c108840 msgr2=0x7f3d6c10ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.932+0000 7f3d71103700 1 --2- 172.21.15.98:0/1129637234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d6c108840 0x7f3d6c10ac30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c0098a0 tx=0x7f3d5c009bb0 comp rx=0 tx=0).stop 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.932+0000 7f3d71103700 1 -- 172.21.15.98:0/1129637234 shutdown_connections 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.932+0000 7f3d71103700 1 --2- 172.21.15.98:0/1129637234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d6c108840 0x7f3d6c10ac30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.932+0000 7f3d71103700 1 -- 172.21.15.98:0/1129637234 >> 172.21.15.98:0/1129637234 conn(0x7f3d6c102210 msgr2=0x7f3d6c104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.932+0000 7f3d71103700 1 -- 172.21.15.98:0/1129637234 shutdown_connections 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.932+0000 7f3d71103700 1 -- 172.21.15.98:0/1129637234 wait complete. 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.933+0000 7f3d71103700 1 Processor -- start 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.933+0000 7f3d71103700 1 -- start start 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.933+0000 7f3d71103700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d6c108840 0x7f3d6c19bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.933+0000 7f3d71103700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6c10c120 con 0x7f3d6c108840 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.933+0000 7f3d6ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d6c108840 0x7f3d6c19bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.933+0000 7f3d6ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d6c108840 0x7f3d6c19bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36172/0 (socket says 172.21.15.98:36172) 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.933+0000 7f3d6ad9d700 1 -- 172.21.15.98:0/3492762355 learned_addr learned my addr 172.21.15.98:0/3492762355 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.934+0000 7f3d6ad9d700 1 -- 172.21.15.98:0/3492762355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6c19c260 con 0x7f3d6c108840 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.934+0000 7f3d6ad9d700 1 --2- 172.21.15.98:0/3492762355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d6c108840 0x7f3d6c19bd20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c011b60 tx=0x7f3d5c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.934+0000 7f3d53fff700 1 -- 172.21.15.98:0/3492762355 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d5c003b20 con 0x7f3d6c108840 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.934+0000 7f3d71103700 1 -- 172.21.15.98:0/3492762355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d6c19c550 con 0x7f3d6c108840 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.934+0000 7f3d53fff700 1 -- 172.21.15.98:0/3492762355 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3d5c003cc0 con 0x7f3d6c108840 2024-05-09T08:40:46.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.934+0000 7f3d53fff700 1 -- 172.21.15.98:0/3492762355 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d5c024840 con 0x7f3d6c108840 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.934+0000 7f3d71103700 1 -- 172.21.15.98:0/3492762355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d6c19f1d0 con 0x7f3d6c108840 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.935+0000 7f3d53fff700 1 -- 172.21.15.98:0/3492762355 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f3d5c022020 con 0x7f3d6c108840 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.935+0000 7f3d71103700 1 -- 172.21.15.98:0/3492762355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d6c063980 con 0x7f3d6c108840 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.935+0000 7f3d53fff700 1 --2- 172.21.15.98:0/3492762355 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f3d540390a0 0x7f3d5403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.935+0000 7f3d53fff700 1 -- 172.21.15.98:0/3492762355 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3d5c011e00 con 0x7f3d6c108840 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.935+0000 7f3d6a59c700 1 -- 172.21.15.98:0/3492762355 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f3d540390a0 msgr2=0x7f3d5403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1313924889 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.935+0000 7f3d6a59c700 1 --2- 172.21.15.98:0/3492762355 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f3d540390a0 0x7f3d5403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:44.940+0000 7f3d53fff700 1 -- 172.21.15.98:0/3492762355 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3d5c04cc20 con 0x7f3d6c108840 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:45.136+0000 7f3d6a59c700 1 -- 172.21.15.98:0/3492762355 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f3d540390a0 msgr2=0x7f3d5403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1313924889 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:45.136+0000 7f3d6a59c700 1 --2- 172.21.15.98:0/3492762355 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f3d540390a0 0x7f3d5403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:45.232+0000 7f3d71103700 1 -- 172.21.15.98:0/3492762355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f3d6c19f980 con 0x7f3d6c108840 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:45.232+0000 7f3d53fff700 1 -- 172.21.15.98:0/3492762355 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f3d5c00bc70 con 0x7f3d6c108840 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:45.234+0000 7f3d71103700 1 -- 172.21.15.98:0/3492762355 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f3d540390a0 msgr2=0x7f3d5403b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:45.234+0000 7f3d71103700 1 --2- 172.21.15.98:0/3492762355 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f3d540390a0 0x7f3d5403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:46.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:45.234+0000 7f3d71103700 1 -- 172.21.15.98:0/3492762355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d6c108840 msgr2=0x7f3d6c19bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:46.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:45.234+0000 7f3d71103700 1 --2- 172.21.15.98:0/3492762355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d6c108840 0x7f3d6c19bd20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c011b60 tx=0x7f3d5c011c40 comp rx=0 tx=0).stop 2024-05-09T08:40:46.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:45.234+0000 7f3d71103700 1 -- 172.21.15.98:0/3492762355 shutdown_connections 2024-05-09T08:40:46.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:45.234+0000 7f3d71103700 1 --2- 172.21.15.98:0/3492762355 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f3d540390a0 0x7f3d5403b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:46.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:45.234+0000 7f3d71103700 1 --2- 172.21.15.98:0/3492762355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d6c108840 0x7f3d6c19bd20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:46.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:45.234+0000 7f3d71103700 1 -- 172.21.15.98:0/3492762355 >> 172.21.15.98:0/3492762355 conn(0x7f3d6c102210 msgr2=0x7f3d6c102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:46.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:45.234+0000 7f3d71103700 1 -- 172.21.15.98:0/3492762355 shutdown_connections 2024-05-09T08:40:46.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:45.234+0000 7f3d71103700 1 -- 172.21.15.98:0/3492762355 wait complete. 2024-05-09T08:40:46.194 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-05-09T08:40:46.194 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 5... 2024-05-09T08:40:46.412 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:46 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:46.339+0000 7f5225504200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:40:46.732 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:46 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:46.640+0000 7f5225504200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:40:46.732 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:46 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:46.733+0000 7f5225504200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:40:47.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:46 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:46.936+0000 7f5225504200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:40:47.587 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:47 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:47.132+0000 7f5225504200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:40:48.002 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:47 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:47.738+0000 7f5225504200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:40:48.257 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:48 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:48.160+0000 7f5225504200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:40:48.257 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:48 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:48.257+0000 7f5225504200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:40:48.587 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:48 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:48.458+0000 7f5225504200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:40:49.318 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:48 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:48.980+0000 7f5225504200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:40:49.318 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:49 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:49.073+0000 7f5225504200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:40:49.587 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:49 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:49.318+0000 7f5225504200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:40:49.587 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:49 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:49.454+0000 7f5225504200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:40:49.588 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:49 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:49.552+0000 7f5225504200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:40:49.970 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:49 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:49.670+0000 7f5225504200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:40:49.971 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:49 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:49.971+0000 7f5225504200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:40:50.837 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:50 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:50.573+0000 7f5225504200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:40:50.837 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:50 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:40:50.790+0000 7f5225504200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:40:51.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: Active manager daemon x restarted 2024-05-09T08:40:51.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: Activating manager daemon x 2024-05-09T08:40:51.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: osdmap e2: 0 total, 0 up, 0 in 2024-05-09T08:40:51.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: mgrmap e6: x(active, starting, since 0.003346s) 2024-05-09T08:40:51.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:40:51.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:40:51.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:40:51.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:40:51.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:40:51.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: Manager daemon x is now available 2024-05-09T08:40:51.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' 2024-05-09T08:40:51.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' 2024-05-09T08:40:51.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:40:51.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:40:51.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:40:51.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:50 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:40:52.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:51 smithi098 ceph-mon[30092]: Found migration_current of "None". Setting to last migration. 2024-05-09T08:40:52.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:51 smithi098 ceph-mon[30092]: mgrmap e7: x(active, since 1.00536s) 2024-05-09T08:40:52.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:40:52.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-09T08:40:52.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-09T08:40:52.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:40:52.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.595+0000 7f08ecc64700 1 Processor -- start 2024-05-09T08:40:52.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.595+0000 7f08ecc64700 1 -- start start 2024-05-09T08:40:52.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.596+0000 7f08ecc64700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8106cc0 0x7f08e81070e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:52.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.596+0000 7f08ecc64700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08e81076b0 con 0x7f08e8106cc0 2024-05-09T08:40:52.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.596+0000 7f08e659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8106cc0 0x7f08e81070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:52.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.596+0000 7f08e659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8106cc0 0x7f08e81070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49734/0 (socket says 172.21.15.98:49734) 2024-05-09T08:40:52.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.596+0000 7f08e659c700 1 -- 172.21.15.98:0/3507531267 learned_addr learned my addr 172.21.15.98:0/3507531267 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.596+0000 7f08e659c700 1 -- 172.21.15.98:0/3507531267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e8107ec0 con 0x7f08e8106cc0 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.596+0000 7f08e659c700 1 --2- 172.21.15.98:0/3507531267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8106cc0 0x7f08e81070e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f08d800bb40 tx=0x7f08d800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4300825220d6ed07 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.596+0000 7f08e559a700 1 -- 172.21.15.98:0/3507531267 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08d800dbe0 con 0x7f08e8106cc0 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.596+0000 7f08e559a700 1 -- 172.21.15.98:0/3507531267 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f08d80044f0 con 0x7f08e8106cc0 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.597+0000 7f08e559a700 1 -- 172.21.15.98:0/3507531267 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08d8010470 con 0x7f08e8106cc0 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.597+0000 7f08ecc64700 1 -- 172.21.15.98:0/3507531267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8106cc0 msgr2=0x7f08e81070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.597+0000 7f08ecc64700 1 --2- 172.21.15.98:0/3507531267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8106cc0 0x7f08e81070e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f08d800bb40 tx=0x7f08d800be50 comp rx=0 tx=0).stop 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.597+0000 7f08ecc64700 1 -- 172.21.15.98:0/3507531267 shutdown_connections 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.597+0000 7f08ecc64700 1 --2- 172.21.15.98:0/3507531267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8106cc0 0x7f08e81070e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.597+0000 7f08ecc64700 1 -- 172.21.15.98:0/3507531267 >> 172.21.15.98:0/3507531267 conn(0x7f08e8102240 msgr2=0x7f08e81046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.597+0000 7f08ecc64700 1 -- 172.21.15.98:0/3507531267 shutdown_connections 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.597+0000 7f08ecc64700 1 -- 172.21.15.98:0/3507531267 wait complete. 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.598+0000 7f08ecc64700 1 Processor -- start 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.598+0000 7f08ecc64700 1 -- start start 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.598+0000 7f08ecc64700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8106cc0 0x7f08e8199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:52.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.598+0000 7f08ecc64700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08e807aff0 con 0x7f08e8106cc0 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.598+0000 7f08e659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8106cc0 0x7f08e8199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.598+0000 7f08e659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8106cc0 0x7f08e8199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49736/0 (socket says 172.21.15.98:49736) 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.598+0000 7f08e659c700 1 -- 172.21.15.98:0/2275886603 learned_addr learned my addr 172.21.15.98:0/2275886603 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.599+0000 7f08e659c700 1 -- 172.21.15.98:0/2275886603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e819a0b0 con 0x7f08e8106cc0 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.599+0000 7f08e659c700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8106cc0 0x7f08e8199b70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f08d800b340 tx=0x7f08d800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.599+0000 7f08d77fe700 1 -- 172.21.15.98:0/2275886603 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08d8027070 con 0x7f08e8106cc0 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.599+0000 7f08d77fe700 1 -- 172.21.15.98:0/2275886603 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f08d8010b40 con 0x7f08e8106cc0 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.599+0000 7f08ecc64700 1 -- 172.21.15.98:0/2275886603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08e819a340 con 0x7f08e8106cc0 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.599+0000 7f08d77fe700 1 -- 172.21.15.98:0/2275886603 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08d8022b00 con 0x7f08e8106cc0 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.599+0000 7f08ecc64700 1 -- 172.21.15.98:0/2275886603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08e819d030 con 0x7f08e8106cc0 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.600+0000 7f08d77fe700 1 -- 172.21.15.98:0/2275886603 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f08d802a070 con 0x7f08e8106cc0 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.600+0000 7f08d77fe700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f08d00390a0 0x7f08d003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.600+0000 7f08d77fe700 1 -- 172.21.15.98:0/2275886603 --> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f08d003bc70 con 0x7f08d00390a0 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.600+0000 7f08d77fe700 1 -- 172.21.15.98:0/2275886603 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f08d801f070 con 0x7f08e8106cc0 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.600+0000 7f08e5d9b700 1 -- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f08d00390a0 msgr2=0x7f08d003b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1313924889 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.600+0000 7f08e5d9b700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f08d00390a0 0x7f08d003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:40:52.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.801+0000 7f08e5d9b700 1 -- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f08d00390a0 msgr2=0x7f08d003b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1313924889 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:46.801+0000 7f08e5d9b700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f08d00390a0 0x7f08d003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:47.202+0000 7f08e5d9b700 1 -- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f08d00390a0 msgr2=0x7f08d003b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1313924889 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:47.202+0000 7f08e5d9b700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f08d00390a0 0x7f08d003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:48.003+0000 7f08e5d9b700 1 -- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f08d00390a0 msgr2=0x7f08d003b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1313924889 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:48.003+0000 7f08e5d9b700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f08d00390a0 0x7f08d003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:49.604+0000 7f08e5d9b700 1 -- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f08d00390a0 msgr2=0x7f08d003b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1313924889 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:49.604+0000 7f08e5d9b700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f08d00390a0 0x7f08d003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:50.794+0000 7f08d77fe700 1 -- 172.21.15.98:0/2275886603 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7f08d802ca00 con 0x7f08e8106cc0 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:50.794+0000 7f08d77fe700 1 -- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f08d00390a0 msgr2=0x7f08d003b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:50.794+0000 7f08d77fe700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f08d00390a0 0x7f08d003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.795+0000 7f08d77fe700 1 -- 172.21.15.98:0/2275886603 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f08d80510c0 con 0x7f08e8106cc0 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.795+0000 7f08d77fe700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f08d003c9d0 0x7f08d003edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.795+0000 7f08d77fe700 1 -- 172.21.15.98:0/2275886603 --> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f08d003bc70 con 0x7f08d003c9d0 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.795+0000 7f08e5d9b700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f08d003c9d0 0x7f08d003edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.796+0000 7f08e5d9b700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f08d003c9d0 0x7f08d003edc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f08dc003a10 tx=0x7f08dc0092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:52.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.797+0000 7f08d77fe700 1 -- 172.21.15.98:0/2275886603 <== mgr.14118 v2:172.21.15.98:6800/498605085 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f08d003bc70 con 0x7f08d003c9d0 2024-05-09T08:40:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.803+0000 7f08ecc64700 1 -- 172.21.15.98:0/2275886603 --> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f08e8069450 con 0x7f08d003c9d0 2024-05-09T08:40:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.803+0000 7f08d77fe700 1 -- 172.21.15.98:0/2275886603 <== mgr.14118 v2:172.21.15.98:6800/498605085 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f08e8069450 con 0x7f08d003c9d0 2024-05-09T08:40:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.803+0000 7f08ecc64700 1 -- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f08d003c9d0 msgr2=0x7f08d003edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.803+0000 7f08ecc64700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f08d003c9d0 0x7f08d003edc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f08dc003a10 tx=0x7f08dc0092b0 comp rx=0 tx=0).stop 2024-05-09T08:40:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.803+0000 7f08ecc64700 1 -- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8106cc0 msgr2=0x7f08e8199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.803+0000 7f08ecc64700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8106cc0 0x7f08e8199b70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f08d800b340 tx=0x7f08d800dd90 comp rx=0 tx=0).stop 2024-05-09T08:40:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.803+0000 7f08ecc64700 1 -- 172.21.15.98:0/2275886603 shutdown_connections 2024-05-09T08:40:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.803+0000 7f08ecc64700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f08d003c9d0 0x7f08d003edc0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.804+0000 7f08ecc64700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e8106cc0 0x7f08e8199b70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.804+0000 7f08ecc64700 1 --2- 172.21.15.98:0/2275886603 >> [v2:172.21.15.98:6800/1313924889,v1:172.21.15.98:6801/1313924889] conn(0x7f08d00390a0 0x7f08d003b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.804+0000 7f08ecc64700 1 -- 172.21.15.98:0/2275886603 >> 172.21.15.98:0/2275886603 conn(0x7f08e8102240 msgr2=0x7f08e8190690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.804+0000 7f08ecc64700 1 -- 172.21.15.98:0/2275886603 shutdown_connections 2024-05-09T08:40:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:51.804+0000 7f08ecc64700 1 -- 172.21.15.98:0/2275886603 wait complete. 2024-05-09T08:40:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 5 is available 2024-05-09T08:40:52.379 INFO:teuthology.orchestra.run.smithi098.stdout:Setting orchestrator backend to cephadm... 2024-05-09T08:40:53.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:52 smithi098 ceph-mon[30092]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-09T08:40:53.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:52 smithi098 ceph-mon[30092]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-09T08:40:53.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:52 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' 2024-05-09T08:40:53.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:52 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' 2024-05-09T08:40:53.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:52 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:40:53.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.777+0000 7f6cb3c72700 1 Processor -- start 2024-05-09T08:40:53.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.778+0000 7f6cb3c72700 1 -- start start 2024-05-09T08:40:53.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.778+0000 7f6cb3c72700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cac107dc0 0x7f6cac1081e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:53.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.778+0000 7f6cb3c72700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac1087b0 con 0x7f6cac107dc0 2024-05-09T08:40:53.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.778+0000 7f6cb1a0e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cac107dc0 0x7f6cac1081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:53.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.778+0000 7f6cb1a0e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cac107dc0 0x7f6cac1081e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49800/0 (socket says 172.21.15.98:49800) 2024-05-09T08:40:53.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.778+0000 7f6cb1a0e700 1 -- 172.21.15.98:0/1844648192 learned_addr learned my addr 172.21.15.98:0/1844648192 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:53.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.778+0000 7f6cb1a0e700 1 -- 172.21.15.98:0/1844648192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cac108fc0 con 0x7f6cac107dc0 2024-05-09T08:40:53.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.778+0000 7f6cb1a0e700 1 --2- 172.21.15.98:0/1844648192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cac107dc0 0x7f6cac1081e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009e80 tx=0x7f6ca800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba54fff55baa56e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:53.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.778+0000 7f6cb0a0c700 1 -- 172.21.15.98:0/1844648192 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ca8017070 con 0x7f6cac107dc0 2024-05-09T08:40:53.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.779+0000 7f6cb0a0c700 1 -- 172.21.15.98:0/1844648192 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6ca800b8c0 con 0x7f6cac107dc0 2024-05-09T08:40:53.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.779+0000 7f6cb0a0c700 1 -- 172.21.15.98:0/1844648192 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ca80045c0 con 0x7f6cac107dc0 2024-05-09T08:40:53.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.779+0000 7f6cb3c72700 1 -- 172.21.15.98:0/1844648192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cac107dc0 msgr2=0x7f6cac1081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:53.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.779+0000 7f6cb3c72700 1 --2- 172.21.15.98:0/1844648192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cac107dc0 0x7f6cac1081e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009e80 tx=0x7f6ca800b330 comp rx=0 tx=0).stop 2024-05-09T08:40:53.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.779+0000 7f6cb3c72700 1 -- 172.21.15.98:0/1844648192 shutdown_connections 2024-05-09T08:40:53.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.779+0000 7f6cb3c72700 1 --2- 172.21.15.98:0/1844648192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cac107dc0 0x7f6cac1081e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:53.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.779+0000 7f6cb3c72700 1 -- 172.21.15.98:0/1844648192 >> 172.21.15.98:0/1844648192 conn(0x7f6cac1033a0 msgr2=0x7f6cac1057c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:53.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.779+0000 7f6cb3c72700 1 -- 172.21.15.98:0/1844648192 shutdown_connections 2024-05-09T08:40:53.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.779+0000 7f6cb3c72700 1 -- 172.21.15.98:0/1844648192 wait complete. 2024-05-09T08:40:53.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.780+0000 7f6cb3c72700 1 Processor -- start 2024-05-09T08:40:53.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.780+0000 7f6cb3c72700 1 -- start start 2024-05-09T08:40:53.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.780+0000 7f6cb3c72700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cac107dc0 0x7f6cac19bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:53.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.780+0000 7f6cb3c72700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac10c370 con 0x7f6cac107dc0 2024-05-09T08:40:53.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.780+0000 7f6cb1a0e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cac107dc0 0x7f6cac19bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:53.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.780+0000 7f6cb1a0e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cac107dc0 0x7f6cac19bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49812/0 (socket says 172.21.15.98:49812) 2024-05-09T08:40:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.780+0000 7f6cb1a0e700 1 -- 172.21.15.98:0/2402101523 learned_addr learned my addr 172.21.15.98:0/2402101523 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.780+0000 7f6cb1a0e700 1 -- 172.21.15.98:0/2402101523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cac19c180 con 0x7f6cac107dc0 2024-05-09T08:40:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.781+0000 7f6cb1a0e700 1 --2- 172.21.15.98:0/2402101523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cac107dc0 0x7f6cac19bc40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009530 tx=0x7f6ca800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.781+0000 7f6ca2ffd700 1 -- 172.21.15.98:0/2402101523 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ca8020020 con 0x7f6cac107dc0 2024-05-09T08:40:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.781+0000 7f6cb3c72700 1 -- 172.21.15.98:0/2402101523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cac19c410 con 0x7f6cac107dc0 2024-05-09T08:40:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.781+0000 7f6ca2ffd700 1 -- 172.21.15.98:0/2402101523 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6ca8003b50 con 0x7f6cac107dc0 2024-05-09T08:40:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.781+0000 7f6ca2ffd700 1 -- 172.21.15.98:0/2402101523 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ca8017040 con 0x7f6cac107dc0 2024-05-09T08:40:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.781+0000 7f6cb3c72700 1 -- 172.21.15.98:0/2402101523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cac19f100 con 0x7f6cac107dc0 2024-05-09T08:40:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.782+0000 7f6ca2ffd700 1 -- 172.21.15.98:0/2402101523 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f6ca8023020 con 0x7f6cac107dc0 2024-05-09T08:40:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.782+0000 7f6cb3c72700 1 -- 172.21.15.98:0/2402101523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cac063980 con 0x7f6cac107dc0 2024-05-09T08:40:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.782+0000 7f6ca2ffd700 1 --2- 172.21.15.98:0/2402101523 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f6c98038ff0 0x7f6c9803b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.782+0000 7f6ca2ffd700 1 -- 172.21.15.98:0/2402101523 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6ca800f050 con 0x7f6cac107dc0 2024-05-09T08:40:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.782+0000 7f6cb120d700 1 --2- 172.21.15.98:0/2402101523 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f6c98038ff0 0x7f6c9803b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.782+0000 7f6cb120d700 1 --2- 172.21.15.98:0/2402101523 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f6c98038ff0 0x7f6c9803b4b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c006fd0 tx=0x7f6c9c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:53.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:52.787+0000 7f6ca2ffd700 1 -- 172.21.15.98:0/2402101523 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6ca802a020 con 0x7f6cac107dc0 2024-05-09T08:40:53.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:53.002+0000 7f6cb3c72700 1 -- 172.21.15.98:0/2402101523 --> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f6cac104480 con 0x7f6c98038ff0 2024-05-09T08:40:53.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:53.010+0000 7f6ca2ffd700 1 -- 172.21.15.98:0/2402101523 <== mgr.14118 v2:172.21.15.98:6800/498605085 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6cac104480 con 0x7f6c98038ff0 2024-05-09T08:40:53.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:53.012+0000 7f6cb3c72700 1 -- 172.21.15.98:0/2402101523 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f6c98038ff0 msgr2=0x7f6c9803b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:53.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:53.012+0000 7f6cb3c72700 1 --2- 172.21.15.98:0/2402101523 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f6c98038ff0 0x7f6c9803b4b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c006fd0 tx=0x7f6c9c006e40 comp rx=0 tx=0).stop 2024-05-09T08:40:53.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:53.012+0000 7f6cb3c72700 1 -- 172.21.15.98:0/2402101523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cac107dc0 msgr2=0x7f6cac19bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:53.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:53.012+0000 7f6cb3c72700 1 --2- 172.21.15.98:0/2402101523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cac107dc0 0x7f6cac19bc40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009530 tx=0x7f6ca800bd50 comp rx=0 tx=0).stop 2024-05-09T08:40:53.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:53.012+0000 7f6cb3c72700 1 -- 172.21.15.98:0/2402101523 shutdown_connections 2024-05-09T08:40:53.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:53.012+0000 7f6cb3c72700 1 --2- 172.21.15.98:0/2402101523 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f6c98038ff0 0x7f6c9803b4b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:53.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:53.012+0000 7f6cb3c72700 1 --2- 172.21.15.98:0/2402101523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cac107dc0 0x7f6cac19bc40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:53.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:53.012+0000 7f6cb3c72700 1 -- 172.21.15.98:0/2402101523 >> 172.21.15.98:0/2402101523 conn(0x7f6cac1033a0 msgr2=0x7f6cac103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:53.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:53.012+0000 7f6cb3c72700 1 -- 172.21.15.98:0/2402101523 shutdown_connections 2024-05-09T08:40:53.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:53.012+0000 7f6cb3c72700 1 -- 172.21.15.98:0/2402101523 wait complete. 2024-05-09T08:40:53.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:53 smithi098 ceph-mon[30092]: [09/May/2024:08:40:52] ENGINE Bus STARTING 2024-05-09T08:40:53.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:53 smithi098 ceph-mon[30092]: [09/May/2024:08:40:52] ENGINE Serving on https://172.21.15.98:7150 2024-05-09T08:40:53.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:53 smithi098 ceph-mon[30092]: [09/May/2024:08:40:52] ENGINE Serving on http://172.21.15.98:8765 2024-05-09T08:40:53.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:53 smithi098 ceph-mon[30092]: [09/May/2024:08:40:52] ENGINE Bus STARTED 2024-05-09T08:40:53.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:53 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' 2024-05-09T08:40:53.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:53 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:40:53.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:53 smithi098 ceph-mon[30092]: mgrmap e8: x(active, since 2s) 2024-05-09T08:40:54.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-09T08:40:54.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.049+0000 7fc70dd84700 1 Processor -- start 2024-05-09T08:40:54.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.049+0000 7fc70dd84700 1 -- start start 2024-05-09T08:40:54.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.050+0000 7fc70dd84700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7081088d0 0x7fc70810acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:54.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.050+0000 7fc70dd84700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc70807be90 con 0x7fc7081088d0 2024-05-09T08:40:54.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.050+0000 7fc7077fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7081088d0 0x7fc70810acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:54.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.050+0000 7fc7077fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7081088d0 0x7fc70810acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49820/0 (socket says 172.21.15.98:49820) 2024-05-09T08:40:54.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.050+0000 7fc7077fe700 1 -- 172.21.15.98:0/896290003 learned_addr learned my addr 172.21.15.98:0/896290003 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:54.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.050+0000 7fc7077fe700 1 -- 172.21.15.98:0/896290003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70810b200 con 0x7fc7081088d0 2024-05-09T08:40:54.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.050+0000 7fc7077fe700 1 --2- 172.21.15.98:0/896290003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7081088d0 0x7fc70810acc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc6f80098a0 tx=0x7fc6f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f068df88a8b73fb1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:54.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.050+0000 7fc7067fc700 1 -- 172.21.15.98:0/896290003 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6f8004160 con 0x7fc7081088d0 2024-05-09T08:40:54.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.050+0000 7fc7067fc700 1 -- 172.21.15.98:0/896290003 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc6f8004300 con 0x7fc7081088d0 2024-05-09T08:40:54.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.050+0000 7fc7067fc700 1 -- 172.21.15.98:0/896290003 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6f8011b50 con 0x7fc7081088d0 2024-05-09T08:40:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.051+0000 7fc70dd84700 1 -- 172.21.15.98:0/896290003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7081088d0 msgr2=0x7fc70810acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.051+0000 7fc70dd84700 1 --2- 172.21.15.98:0/896290003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7081088d0 0x7fc70810acc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc6f80098a0 tx=0x7fc6f8009bb0 comp rx=0 tx=0).stop 2024-05-09T08:40:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.051+0000 7fc70dd84700 1 -- 172.21.15.98:0/896290003 shutdown_connections 2024-05-09T08:40:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.051+0000 7fc70dd84700 1 --2- 172.21.15.98:0/896290003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7081088d0 0x7fc70810acc0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.051+0000 7fc70dd84700 1 -- 172.21.15.98:0/896290003 >> 172.21.15.98:0/896290003 conn(0x7fc7081022c0 msgr2=0x7fc708104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.051+0000 7fc70dd84700 1 -- 172.21.15.98:0/896290003 shutdown_connections 2024-05-09T08:40:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.051+0000 7fc70dd84700 1 -- 172.21.15.98:0/896290003 wait complete. 2024-05-09T08:40:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.052+0000 7fc70dd84700 1 Processor -- start 2024-05-09T08:40:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.052+0000 7fc70dd84700 1 -- start start 2024-05-09T08:40:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.052+0000 7fc70dd84700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7081088d0 0x7fc70819bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.052+0000 7fc70dd84700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc70810c220 con 0x7fc7081088d0 2024-05-09T08:40:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.052+0000 7fc7077fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7081088d0 0x7fc70819bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.052+0000 7fc7077fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7081088d0 0x7fc70819bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49834/0 (socket says 172.21.15.98:49834) 2024-05-09T08:40:54.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.052+0000 7fc7077fe700 1 -- 172.21.15.98:0/2020672768 learned_addr learned my addr 172.21.15.98:0/2020672768 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:54.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.053+0000 7fc7077fe700 1 -- 172.21.15.98:0/2020672768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70819c240 con 0x7fc7081088d0 2024-05-09T08:40:54.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.053+0000 7fc7077fe700 1 --2- 172.21.15.98:0/2020672768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7081088d0 0x7fc70819bd00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009fb0 tx=0x7fc6f8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:54.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.053+0000 7fc704ff9700 1 -- 172.21.15.98:0/2020672768 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6f80038b0 con 0x7fc7081088d0 2024-05-09T08:40:54.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.053+0000 7fc70dd84700 1 -- 172.21.15.98:0/2020672768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc70819c4d0 con 0x7fc7081088d0 2024-05-09T08:40:54.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.053+0000 7fc704ff9700 1 -- 172.21.15.98:0/2020672768 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc6f8011e50 con 0x7fc7081088d0 2024-05-09T08:40:54.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.053+0000 7fc704ff9700 1 -- 172.21.15.98:0/2020672768 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6f8019420 con 0x7fc7081088d0 2024-05-09T08:40:54.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.053+0000 7fc70dd84700 1 -- 172.21.15.98:0/2020672768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc70819f1b0 con 0x7fc7081088d0 2024-05-09T08:40:54.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.054+0000 7fc704ff9700 1 -- 172.21.15.98:0/2020672768 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fc6f8019720 con 0x7fc7081088d0 2024-05-09T08:40:54.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.054+0000 7fc70dd84700 1 -- 172.21.15.98:0/2020672768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc708069450 con 0x7fc7081088d0 2024-05-09T08:40:54.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.054+0000 7fc704ff9700 1 --2- 172.21.15.98:0/2020672768 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fc6f00390c0 0x7fc6f003b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:54.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.054+0000 7fc704ff9700 1 -- 172.21.15.98:0/2020672768 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc6f804d360 con 0x7fc7081088d0 2024-05-09T08:40:54.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.054+0000 7fc706ffd700 1 --2- 172.21.15.98:0/2020672768 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fc6f00390c0 0x7fc6f003b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:54.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.055+0000 7fc706ffd700 1 --2- 172.21.15.98:0/2020672768 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fc6f00390c0 0x7fc6f003b580 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc006fd0 tx=0x7fc6fc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:54.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.059+0000 7fc704ff9700 1 -- 172.21.15.98:0/2020672768 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc6f8023c90 con 0x7fc7081088d0 2024-05-09T08:40:54.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.256+0000 7fc70dd84700 1 -- 172.21.15.98:0/2020672768 --> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fc7081034c0 con 0x7fc6f00390c0 2024-05-09T08:40:54.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.257+0000 7fc704ff9700 1 -- 172.21.15.98:0/2020672768 <== mgr.14118 v2:172.21.15.98:6800/498605085 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fc7081034c0 con 0x7fc6f00390c0 2024-05-09T08:40:54.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.259+0000 7fc70dd84700 1 -- 172.21.15.98:0/2020672768 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fc6f00390c0 msgr2=0x7fc6f003b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:54.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.260+0000 7fc70dd84700 1 --2- 172.21.15.98:0/2020672768 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fc6f00390c0 0x7fc6f003b580 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc006fd0 tx=0x7fc6fc006e40 comp rx=0 tx=0).stop 2024-05-09T08:40:54.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.260+0000 7fc70dd84700 1 -- 172.21.15.98:0/2020672768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7081088d0 msgr2=0x7fc70819bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:54.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.260+0000 7fc70dd84700 1 --2- 172.21.15.98:0/2020672768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7081088d0 0x7fc70819bd00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009fb0 tx=0x7fc6f8011bf0 comp rx=0 tx=0).stop 2024-05-09T08:40:54.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.260+0000 7fc70dd84700 1 -- 172.21.15.98:0/2020672768 shutdown_connections 2024-05-09T08:40:54.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.260+0000 7fc70dd84700 1 --2- 172.21.15.98:0/2020672768 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fc6f00390c0 0x7fc6f003b580 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:54.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.260+0000 7fc70dd84700 1 --2- 172.21.15.98:0/2020672768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7081088d0 0x7fc70819bd00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:54.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.260+0000 7fc70dd84700 1 -- 172.21.15.98:0/2020672768 >> 172.21.15.98:0/2020672768 conn(0x7fc7081022c0 msgr2=0x7fc708102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:54.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.260+0000 7fc70dd84700 1 -- 172.21.15.98:0/2020672768 shutdown_connections 2024-05-09T08:40:54.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:54.260+0000 7fc70dd84700 1 -- 172.21.15.98:0/2020672768 wait complete. 2024-05-09T08:40:54.826 INFO:teuthology.orchestra.run.smithi098.stdout:Generating ssh key... 2024-05-09T08:40:55.066 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:54 smithi098 ceph-mon[30092]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:40:55.732 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: Generating public/private rsa key pair. 2024-05-09T08:40:55.732 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: Your identification has been saved in /tmp/tmpwyet67ax/key. 2024-05-09T08:40:55.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: Your public key has been saved in /tmp/tmpwyet67ax/key.pub. 2024-05-09T08:40:55.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: The key fingerprint is: 2024-05-09T08:40:55.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: SHA256:UO0qsS0vc3sEm4mM0DrxavUpo1HJl4LsJw+hVzZlyfY ceph-a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:40:55.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: The key's randomart image is: 2024-05-09T08:40:55.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: +---[RSA 3072]----+ 2024-05-09T08:40:55.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: | .. | 2024-05-09T08:40:55.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: | . o . | 2024-05-09T08:40:55.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: | . B . | 2024-05-09T08:40:55.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: | .oo.=.+. . | 2024-05-09T08:40:55.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: | +=Ooo=E* | 2024-05-09T08:40:55.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: | oo=++* * . | 2024-05-09T08:40:55.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: |. *+.. = . | 2024-05-09T08:40:55.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: | .o*o = o . | 2024-05-09T08:40:55.734 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: | ....o +.o | 2024-05-09T08:40:55.734 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:40:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: +----[SHA256]-----+ 2024-05-09T08:40:56.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:55 smithi098 ceph-mon[30092]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:40:56.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:55 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' 2024-05-09T08:40:56.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:55 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' 2024-05-09T08:40:56.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.275+0000 7f704f500700 1 Processor -- start 2024-05-09T08:40:56.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.275+0000 7f704f500700 1 -- start start 2024-05-09T08:40:56.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.275+0000 7f704f500700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048108850 0x7f704810ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:56.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.275+0000 7f704f500700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f704807bed0 con 0x7f7048108850 2024-05-09T08:40:56.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.276+0000 7f704d29c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048108850 0x7f704810ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:56.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.276+0000 7f704d29c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048108850 0x7f704810ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49848/0 (socket says 172.21.15.98:49848) 2024-05-09T08:40:56.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.276+0000 7f704d29c700 1 -- 172.21.15.98:0/3479295751 learned_addr learned my addr 172.21.15.98:0/3479295751 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:56.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.276+0000 7f704d29c700 1 -- 172.21.15.98:0/3479295751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f704810b180 con 0x7f7048108850 2024-05-09T08:40:56.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.276+0000 7f704d29c700 1 --2- 172.21.15.98:0/3479295751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048108850 0x7f704810ac40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f70440098a0 tx=0x7f7044009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed454ad30b6baef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:56.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.276+0000 7f703ffff700 1 -- 172.21.15.98:0/3479295751 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7044004160 con 0x7f7048108850 2024-05-09T08:40:56.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.276+0000 7f703ffff700 1 -- 172.21.15.98:0/3479295751 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7044004300 con 0x7f7048108850 2024-05-09T08:40:56.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.276+0000 7f703ffff700 1 -- 172.21.15.98:0/3479295751 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7044011b50 con 0x7f7048108850 2024-05-09T08:40:56.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.277+0000 7f704f500700 1 -- 172.21.15.98:0/3479295751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048108850 msgr2=0x7f704810ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:56.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.277+0000 7f704f500700 1 --2- 172.21.15.98:0/3479295751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048108850 0x7f704810ac40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f70440098a0 tx=0x7f7044009bb0 comp rx=0 tx=0).stop 2024-05-09T08:40:56.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.277+0000 7f704f500700 1 -- 172.21.15.98:0/3479295751 shutdown_connections 2024-05-09T08:40:56.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.277+0000 7f704f500700 1 --2- 172.21.15.98:0/3479295751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048108850 0x7f704810ac40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:56.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.277+0000 7f704f500700 1 -- 172.21.15.98:0/3479295751 >> 172.21.15.98:0/3479295751 conn(0x7f7048102280 msgr2=0x7f70481046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:56.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.277+0000 7f704f500700 1 -- 172.21.15.98:0/3479295751 shutdown_connections 2024-05-09T08:40:56.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.277+0000 7f704f500700 1 -- 172.21.15.98:0/3479295751 wait complete. 2024-05-09T08:40:56.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.278+0000 7f704f500700 1 Processor -- start 2024-05-09T08:40:56.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.278+0000 7f704f500700 1 -- start start 2024-05-09T08:40:56.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.278+0000 7f704f500700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048108850 0x7f70481080d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:56.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.278+0000 7f704f500700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f704810c1b0 con 0x7f7048108850 2024-05-09T08:40:56.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.278+0000 7f704d29c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048108850 0x7f70481080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:56.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.278+0000 7f704d29c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048108850 0x7f70481080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49850/0 (socket says 172.21.15.98:49850) 2024-05-09T08:40:56.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.278+0000 7f704d29c700 1 -- 172.21.15.98:0/885908234 learned_addr learned my addr 172.21.15.98:0/885908234 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:56.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.278+0000 7f704d29c700 1 -- 172.21.15.98:0/885908234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7048108610 con 0x7f7048108850 2024-05-09T08:40:56.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.278+0000 7f704d29c700 1 --2- 172.21.15.98:0/885908234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048108850 0x7f70481080d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7044011670 tx=0x7f7044011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:56.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.278+0000 7f703e7fc700 1 -- 172.21.15.98:0/885908234 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70440039c0 con 0x7f7048108850 2024-05-09T08:40:56.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.278+0000 7f703e7fc700 1 -- 172.21.15.98:0/885908234 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7044011e50 con 0x7f7048108850 2024-05-09T08:40:56.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.278+0000 7f704f500700 1 -- 172.21.15.98:0/885908234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7048106800 con 0x7f7048108850 2024-05-09T08:40:56.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.278+0000 7f703e7fc700 1 -- 172.21.15.98:0/885908234 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f704401a5c0 con 0x7f7048108850 2024-05-09T08:40:56.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.279+0000 7f704f500700 1 -- 172.21.15.98:0/885908234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7048106d40 con 0x7f7048108850 2024-05-09T08:40:56.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.279+0000 7f703e7fc700 1 -- 172.21.15.98:0/885908234 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f7044024430 con 0x7f7048108850 2024-05-09T08:40:56.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.279+0000 7f704f500700 1 -- 172.21.15.98:0/885908234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7048063980 con 0x7f7048108850 2024-05-09T08:40:56.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.280+0000 7f703e7fc700 1 --2- 172.21.15.98:0/885908234 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f7034039110 0x7f703403b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:56.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.280+0000 7f703e7fc700 1 -- 172.21.15.98:0/885908234 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f704404d4a0 con 0x7f7048108850 2024-05-09T08:40:56.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.280+0000 7f704ca9b700 1 --2- 172.21.15.98:0/885908234 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f7034039110 0x7f703403b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:56.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.280+0000 7f704ca9b700 1 --2- 172.21.15.98:0/885908234 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f7034039110 0x7f703403b5d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7038006fd0 tx=0x7f7038006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:56.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.285+0000 7f703e7fc700 1 -- 172.21.15.98:0/885908234 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f704402d910 con 0x7f7048108850 2024-05-09T08:40:56.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.482+0000 7f704f500700 1 -- 172.21.15.98:0/885908234 --> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f7048103400 con 0x7f7034039110 2024-05-09T08:40:56.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.719+0000 7f703e7fc700 1 -- 172.21.15.98:0/885908234 <== mgr.14118 v2:172.21.15.98:6800/498605085 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7048103400 con 0x7f7034039110 2024-05-09T08:40:56.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.721+0000 7f704f500700 1 -- 172.21.15.98:0/885908234 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f7034039110 msgr2=0x7f703403b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:56.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.721+0000 7f704f500700 1 --2- 172.21.15.98:0/885908234 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f7034039110 0x7f703403b5d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7038006fd0 tx=0x7f7038006e40 comp rx=0 tx=0).stop 2024-05-09T08:40:56.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.721+0000 7f704f500700 1 -- 172.21.15.98:0/885908234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048108850 msgr2=0x7f70481080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:56.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.721+0000 7f704f500700 1 --2- 172.21.15.98:0/885908234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048108850 0x7f70481080d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7044011670 tx=0x7f7044011b60 comp rx=0 tx=0).stop 2024-05-09T08:40:56.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.722+0000 7f704f500700 1 -- 172.21.15.98:0/885908234 shutdown_connections 2024-05-09T08:40:56.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.722+0000 7f704f500700 1 --2- 172.21.15.98:0/885908234 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f7034039110 0x7f703403b5d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:56.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.722+0000 7f704f500700 1 --2- 172.21.15.98:0/885908234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048108850 0x7f70481080d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:56.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.722+0000 7f704f500700 1 -- 172.21.15.98:0/885908234 >> 172.21.15.98:0/885908234 conn(0x7f7048102280 msgr2=0x7f7048102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:56.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.722+0000 7f704f500700 1 -- 172.21.15.98:0/885908234 shutdown_connections 2024-05-09T08:40:56.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:55.722+0000 7f704f500700 1 -- 172.21.15.98:0/885908234 wait complete. 2024-05-09T08:40:56.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:56 smithi098 ceph-mon[30092]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:40:56.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:56 smithi098 ceph-mon[30092]: Generating ssh key... 2024-05-09T08:40:57.545 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM08QhTU9RCj64TMHPr8xMaD2Ggju/QS/0ZtCwyPuKPgrejfryn302vt6awYcYMVgjAQpkBkaRlPqfogVnozATLQM1oWPIg80beCjWShdY6SefSde+xGAGZouZa0SVmT4acx4cZN8o9NGFwr2pmNh+Io6hiUuiGyCq7svM9Oq+rTDBjA3zz8Qi216KQ6cu2gv49/J4OfiHbvRVfAAU5c1RGuc1fBiOefAaN0zOgmAUIKkyDBf1EfQ3/apAfpD0Rp5wfMRCTAm8iLbrAfwONyttgKsBHgiCJ82HCuoR96doEfEmBwpAqvka0fdkNHETqKjbAnQ0JcpIZX4uHmtUFurEVw3Oz0eU0Wgx53KdxYVwtRCCxKrE/etAPD7cMo2tqDObDhK1UqxOKLJ6Iuju60/jjy4OIh2w7LwtfhY1DOHwRbd4ly9c9YZJSW+tx9UTp8F0dpJTt1ibS3Y3c+QpPynSNqTDOClBAlPlRoi3xJqaCZ5yHTTwpBBYnrZpfVm2I5s= ceph-a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:40:57.545 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.731+0000 7fc108329700 1 Processor -- start 2024-05-09T08:40:57.545 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.731+0000 7fc108329700 1 -- start start 2024-05-09T08:40:57.545 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.731+0000 7fc108329700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc10007de20 0x7fc10007c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:57.545 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.731+0000 7fc108329700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc10007c860 con 0x7fc10007de20 2024-05-09T08:40:57.545 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.731+0000 7fc1060c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc10007de20 0x7fc10007c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:57.546 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.731+0000 7fc1060c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc10007de20 0x7fc10007c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32910/0 (socket says 172.21.15.98:32910) 2024-05-09T08:40:57.546 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.731+0000 7fc1060c5700 1 -- 172.21.15.98:0/124206852 learned_addr learned my addr 172.21.15.98:0/124206852 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:57.546 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.732+0000 7fc1060c5700 1 -- 172.21.15.98:0/124206852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10007c9e0 con 0x7fc10007de20 2024-05-09T08:40:57.546 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.732+0000 7fc1060c5700 1 --2- 172.21.15.98:0/124206852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc10007de20 0x7fc10007c320 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc0098a0 tx=0x7fc0fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf07a09e1f932dc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:57.546 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.732+0000 7fc1050c3700 1 -- 172.21.15.98:0/124206852 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0fc004160 con 0x7fc10007de20 2024-05-09T08:40:57.546 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.732+0000 7fc1050c3700 1 -- 172.21.15.98:0/124206852 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc0fc004300 con 0x7fc10007de20 2024-05-09T08:40:57.546 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.732+0000 7fc1050c3700 1 -- 172.21.15.98:0/124206852 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0fc011b50 con 0x7fc10007de20 2024-05-09T08:40:57.546 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.733+0000 7fc108329700 1 -- 172.21.15.98:0/124206852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc10007de20 msgr2=0x7fc10007c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:57.546 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.733+0000 7fc108329700 1 --2- 172.21.15.98:0/124206852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc10007de20 0x7fc10007c320 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc0098a0 tx=0x7fc0fc009bb0 comp rx=0 tx=0).stop 2024-05-09T08:40:57.546 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.733+0000 7fc108329700 1 -- 172.21.15.98:0/124206852 shutdown_connections 2024-05-09T08:40:57.546 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.733+0000 7fc108329700 1 --2- 172.21.15.98:0/124206852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc10007de20 0x7fc10007c320 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:57.546 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.733+0000 7fc108329700 1 -- 172.21.15.98:0/124206852 >> 172.21.15.98:0/124206852 conn(0x7fc100103440 msgr2=0x7fc100105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:57.546 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.733+0000 7fc108329700 1 -- 172.21.15.98:0/124206852 shutdown_connections 2024-05-09T08:40:57.547 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.733+0000 7fc108329700 1 -- 172.21.15.98:0/124206852 wait complete. 2024-05-09T08:40:57.547 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.733+0000 7fc108329700 1 Processor -- start 2024-05-09T08:40:57.547 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.733+0000 7fc108329700 1 -- start start 2024-05-09T08:40:57.547 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.733+0000 7fc108329700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc10007de20 0x7fc10019df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:57.547 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.733+0000 7fc108329700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc10010e430 con 0x7fc10007de20 2024-05-09T08:40:57.547 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.734+0000 7fc1060c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc10007de20 0x7fc10019df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:57.547 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.734+0000 7fc1060c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc10007de20 0x7fc10019df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32920/0 (socket says 172.21.15.98:32920) 2024-05-09T08:40:57.547 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.734+0000 7fc1060c5700 1 -- 172.21.15.98:0/1864526023 learned_addr learned my addr 172.21.15.98:0/1864526023 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:40:57.547 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.734+0000 7fc1060c5700 1 -- 172.21.15.98:0/1864526023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10019e450 con 0x7fc10007de20 2024-05-09T08:40:57.547 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.734+0000 7fc1060c5700 1 --2- 172.21.15.98:0/1864526023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc10007de20 0x7fc10019df10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc011b60 tx=0x7fc0fc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:57.547 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.734+0000 7fc0f77fe700 1 -- 172.21.15.98:0/1864526023 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0fc003970 con 0x7fc10007de20 2024-05-09T08:40:57.547 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.734+0000 7fc0f77fe700 1 -- 172.21.15.98:0/1864526023 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc0fc003b10 con 0x7fc10007de20 2024-05-09T08:40:57.548 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.734+0000 7fc0f77fe700 1 -- 172.21.15.98:0/1864526023 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0fc024610 con 0x7fc10007de20 2024-05-09T08:40:57.548 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.734+0000 7fc108329700 1 -- 172.21.15.98:0/1864526023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc10019e6e0 con 0x7fc10007de20 2024-05-09T08:40:57.548 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.734+0000 7fc108329700 1 -- 172.21.15.98:0/1864526023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1001a13d0 con 0x7fc10007de20 2024-05-09T08:40:57.548 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.735+0000 7fc0f77fe700 1 -- 172.21.15.98:0/1864526023 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fc0fc0248c0 con 0x7fc10007de20 2024-05-09T08:40:57.548 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.735+0000 7fc108329700 1 -- 172.21.15.98:0/1864526023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc100063980 con 0x7fc10007de20 2024-05-09T08:40:57.548 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.735+0000 7fc0f77fe700 1 --2- 172.21.15.98:0/1864526023 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fc0ec039160 0x7fc0ec03b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:57.548 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.735+0000 7fc0f77fe700 1 -- 172.21.15.98:0/1864526023 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc0fc011e00 con 0x7fc10007de20 2024-05-09T08:40:57.548 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.735+0000 7fc1058c4700 1 --2- 172.21.15.98:0/1864526023 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fc0ec039160 0x7fc0ec03b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:57.548 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.736+0000 7fc1058c4700 1 --2- 172.21.15.98:0/1864526023 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fc0ec039160 0x7fc0ec03b620 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0006fd0 tx=0x7fc0f0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:57.548 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.740+0000 7fc0f77fe700 1 -- 172.21.15.98:0/1864526023 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc0fc00ba00 con 0x7fc10007de20 2024-05-09T08:40:57.548 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.936+0000 7fc108329700 1 -- 172.21.15.98:0/1864526023 --> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fc100104640 con 0x7fc0ec039160 2024-05-09T08:40:57.548 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.937+0000 7fc0f77fe700 1 -- 172.21.15.98:0/1864526023 <== mgr.14118 v2:172.21.15.98:6800/498605085 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fc100104640 con 0x7fc0ec039160 2024-05-09T08:40:57.548 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.939+0000 7fc108329700 1 -- 172.21.15.98:0/1864526023 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fc0ec039160 msgr2=0x7fc0ec03b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:57.549 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.939+0000 7fc108329700 1 --2- 172.21.15.98:0/1864526023 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fc0ec039160 0x7fc0ec03b620 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0006fd0 tx=0x7fc0f0006e40 comp rx=0 tx=0).stop 2024-05-09T08:40:57.549 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.939+0000 7fc108329700 1 -- 172.21.15.98:0/1864526023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc10007de20 msgr2=0x7fc10019df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:57.549 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.939+0000 7fc108329700 1 --2- 172.21.15.98:0/1864526023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc10007de20 0x7fc10019df10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc011b60 tx=0x7fc0fc011c40 comp rx=0 tx=0).stop 2024-05-09T08:40:57.549 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.939+0000 7fc108329700 1 -- 172.21.15.98:0/1864526023 shutdown_connections 2024-05-09T08:40:57.549 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.939+0000 7fc108329700 1 --2- 172.21.15.98:0/1864526023 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fc0ec039160 0x7fc0ec03b620 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:57.549 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.939+0000 7fc108329700 1 --2- 172.21.15.98:0/1864526023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc10007de20 0x7fc10019df10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:57.549 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.939+0000 7fc108329700 1 -- 172.21.15.98:0/1864526023 >> 172.21.15.98:0/1864526023 conn(0x7fc100103440 msgr2=0x7fc100103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:57.549 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.939+0000 7fc108329700 1 -- 172.21.15.98:0/1864526023 shutdown_connections 2024-05-09T08:40:57.549 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:56.939+0000 7fc108329700 1 -- 172.21.15.98:0/1864526023 wait complete. 2024-05-09T08:40:57.549 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-09T08:40:57.549 INFO:teuthology.orchestra.run.smithi098.stdout:Adding key to root@localhost authorized_keys... 2024-05-09T08:40:57.549 INFO:teuthology.orchestra.run.smithi098.stdout:Adding host smithi098... 2024-05-09T08:40:58.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:57 smithi098 ceph-mon[30092]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:40:59.024 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:59 smithi098 ceph-mon[30092]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:40:59.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:40:59 smithi098 ceph-mon[30092]: Deploying cephadm binary to smithi098 2024-05-09T08:41:02.736 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Added host 'smithi098' with addr '172.21.15.98' 2024-05-09T08:41:02.736 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.972+0000 7f350ae1a700 1 Processor -- start 2024-05-09T08:41:02.736 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.972+0000 7f350ae1a700 1 -- start start 2024-05-09T08:41:02.736 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.972+0000 7f350ae1a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3504108950 0x7f350410ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.736 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.972+0000 7f350ae1a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f350407bdf0 con 0x7f3504108950 2024-05-09T08:41:02.736 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.972+0000 7f3508bb6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3504108950 0x7f350410ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.736 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.972+0000 7f3508bb6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3504108950 0x7f350410ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32936/0 (socket says 172.21.15.98:32936) 2024-05-09T08:41:02.736 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.972+0000 7f3508bb6700 1 -- 172.21.15.98:0/597572380 learned_addr learned my addr 172.21.15.98:0/597572380 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:02.736 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.972+0000 7f3508bb6700 1 -- 172.21.15.98:0/597572380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350410b280 con 0x7f3504108950 2024-05-09T08:41:02.736 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.972+0000 7f3508bb6700 1 --2- 172.21.15.98:0/597572380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3504108950 0x7f350410ad40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f34f80098a0 tx=0x7f34f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d0848bd81d6faa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.737 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.973+0000 7f35037fe700 1 -- 172.21.15.98:0/597572380 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34f8004160 con 0x7f3504108950 2024-05-09T08:41:02.737 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.973+0000 7f35037fe700 1 -- 172.21.15.98:0/597572380 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f34f8004300 con 0x7f3504108950 2024-05-09T08:41:02.737 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.973+0000 7f35037fe700 1 -- 172.21.15.98:0/597572380 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34f8011b50 con 0x7f3504108950 2024-05-09T08:41:02.737 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.973+0000 7f350ae1a700 1 -- 172.21.15.98:0/597572380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3504108950 msgr2=0x7f350410ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.737 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.973+0000 7f350ae1a700 1 --2- 172.21.15.98:0/597572380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3504108950 0x7f350410ad40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f34f80098a0 tx=0x7f34f8009bb0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.737 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.973+0000 7f350ae1a700 1 -- 172.21.15.98:0/597572380 shutdown_connections 2024-05-09T08:41:02.737 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.973+0000 7f350ae1a700 1 --2- 172.21.15.98:0/597572380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3504108950 0x7f350410ad40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.737 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.973+0000 7f350ae1a700 1 -- 172.21.15.98:0/597572380 >> 172.21.15.98:0/597572380 conn(0x7f35041022a0 msgr2=0x7f35041046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.737 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.973+0000 7f350ae1a700 1 -- 172.21.15.98:0/597572380 shutdown_connections 2024-05-09T08:41:02.737 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.973+0000 7f350ae1a700 1 -- 172.21.15.98:0/597572380 wait complete. 2024-05-09T08:41:02.737 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.974+0000 7f350ae1a700 1 Processor -- start 2024-05-09T08:41:02.737 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.974+0000 7f350ae1a700 1 -- start start 2024-05-09T08:41:02.737 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.974+0000 7f350ae1a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3504108950 0x7f350419bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.738 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.974+0000 7f350ae1a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f350410c1e0 con 0x7f3504108950 2024-05-09T08:41:02.738 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.974+0000 7f3508bb6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3504108950 0x7f350419bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.738 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.974+0000 7f3508bb6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3504108950 0x7f350419bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32946/0 (socket says 172.21.15.98:32946) 2024-05-09T08:41:02.738 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.974+0000 7f3508bb6700 1 -- 172.21.15.98:0/410429005 learned_addr learned my addr 172.21.15.98:0/410429005 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:02.738 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.974+0000 7f3508bb6700 1 -- 172.21.15.98:0/410429005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350419c1e0 con 0x7f3504108950 2024-05-09T08:41:02.738 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.975+0000 7f3508bb6700 1 --2- 172.21.15.98:0/410429005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3504108950 0x7f350419bca0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f34f8011670 tx=0x7f34f8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.738 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.975+0000 7f3501ffb700 1 -- 172.21.15.98:0/410429005 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34f80039c0 con 0x7f3504108950 2024-05-09T08:41:02.738 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.975+0000 7f3501ffb700 1 -- 172.21.15.98:0/410429005 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f34f8011e50 con 0x7f3504108950 2024-05-09T08:41:02.738 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.975+0000 7f3501ffb700 1 -- 172.21.15.98:0/410429005 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34f801a5c0 con 0x7f3504108950 2024-05-09T08:41:02.738 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.975+0000 7f350ae1a700 1 -- 172.21.15.98:0/410429005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f350419c470 con 0x7f3504108950 2024-05-09T08:41:02.738 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.975+0000 7f350ae1a700 1 -- 172.21.15.98:0/410429005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f350419f160 con 0x7f3504108950 2024-05-09T08:41:02.738 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.976+0000 7f3501ffb700 1 -- 172.21.15.98:0/410429005 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f34f8024430 con 0x7f3504108950 2024-05-09T08:41:02.738 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.976+0000 7f3501ffb700 1 --2- 172.21.15.98:0/410429005 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f34f40390c0 0x7f34f403b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.738 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.976+0000 7f350ae1a700 1 -- 172.21.15.98:0/410429005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3504063980 con 0x7f3504108950 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.976+0000 7f3501ffb700 1 -- 172.21.15.98:0/410429005 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f34f804d480 con 0x7f3504108950 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.976+0000 7f3503fff700 1 --2- 172.21.15.98:0/410429005 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f34f40390c0 0x7f34f403b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.977+0000 7f3503fff700 1 --2- 172.21.15.98:0/410429005 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f34f40390c0 0x7f34f403b580 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f34f0006fd0 tx=0x7f34f0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:57.981+0000 7f3501ffb700 1 -- 172.21.15.98:0/410429005 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f34f80151c0 con 0x7f3504108950 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:40:58.179+0000 7f350ae1a700 1 -- 172.21.15.98:0/410429005 --> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}) -- 0x7f3504103400 con 0x7f34f40390c0 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:02.235+0000 7f3501ffb700 1 -- 172.21.15.98:0/410429005 <== mgr.14118 v2:172.21.15.98:6800/498605085 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f3504103400 con 0x7f34f40390c0 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:02.237+0000 7f350ae1a700 1 -- 172.21.15.98:0/410429005 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f34f40390c0 msgr2=0x7f34f403b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:02.237+0000 7f350ae1a700 1 --2- 172.21.15.98:0/410429005 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f34f40390c0 0x7f34f403b580 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f34f0006fd0 tx=0x7f34f0006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:02.237+0000 7f350ae1a700 1 -- 172.21.15.98:0/410429005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3504108950 msgr2=0x7f350419bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:02.237+0000 7f350ae1a700 1 --2- 172.21.15.98:0/410429005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3504108950 0x7f350419bca0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f34f8011670 tx=0x7f34f8011b60 comp rx=0 tx=0).stop 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:02.238+0000 7f350ae1a700 1 -- 172.21.15.98:0/410429005 shutdown_connections 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:02.238+0000 7f350ae1a700 1 --2- 172.21.15.98:0/410429005 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f34f40390c0 0x7f34f403b580 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:02.238+0000 7f350ae1a700 1 --2- 172.21.15.98:0/410429005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3504108950 0x7f350419bca0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:02.238+0000 7f350ae1a700 1 -- 172.21.15.98:0/410429005 >> 172.21.15.98:0/410429005 conn(0x7f35041022a0 msgr2=0x7f3504102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.739 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:02.238+0000 7f350ae1a700 1 -- 172.21.15.98:0/410429005 shutdown_connections 2024-05-09T08:41:02.740 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:02.238+0000 7f350ae1a700 1 -- 172.21.15.98:0/410429005 wait complete. 2024-05-09T08:41:02.740 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mon service... 2024-05-09T08:41:03.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:03 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' 2024-05-09T08:41:03.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:03 smithi098 ceph-mon[30092]: Added host smithi098 2024-05-09T08:41:03.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:03 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:41:05.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:04 smithi098 ceph-mon[30092]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:41:05.128 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:04 smithi098 ceph-mon[30092]: Saving service mon spec with placement count:5 2024-05-09T08:41:05.128 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:04 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' 2024-05-09T08:41:05.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-09T08:41:05.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.448+0000 7f08ce36b700 1 Processor -- start 2024-05-09T08:41:05.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.449+0000 7f08ce36b700 1 -- start start 2024-05-09T08:41:05.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.449+0000 7f08ce36b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08c8108290 0x7f08c81086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.449+0000 7f08ce36b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08c8108c80 con 0x7f08c8108290 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.449+0000 7f08c7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08c8108290 0x7f08c81086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.449+0000 7f08c7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08c8108290 0x7f08c81086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32954/0 (socket says 172.21.15.98:32954) 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.449+0000 7f08c7fff700 1 -- 172.21.15.98:0/3964027831 learned_addr learned my addr 172.21.15.98:0/3964027831 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.449+0000 7f08c7fff700 1 -- 172.21.15.98:0/3964027831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c8106ef0 con 0x7f08c8108290 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.449+0000 7f08c7fff700 1 --2- 172.21.15.98:0/3964027831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08c8108290 0x7f08c81086b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f08b800bb40 tx=0x7f08b800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6447ecb1cbd0b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.450+0000 7f08c6ffd700 1 -- 172.21.15.98:0/3964027831 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08b800dbe0 con 0x7f08c8108290 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.450+0000 7f08c6ffd700 1 -- 172.21.15.98:0/3964027831 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f08b80044f0 con 0x7f08c8108290 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.450+0000 7f08c6ffd700 1 -- 172.21.15.98:0/3964027831 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08b8010470 con 0x7f08c8108290 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.450+0000 7f08ce36b700 1 -- 172.21.15.98:0/3964027831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08c8108290 msgr2=0x7f08c81086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.450+0000 7f08ce36b700 1 --2- 172.21.15.98:0/3964027831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08c8108290 0x7f08c81086b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f08b800bb40 tx=0x7f08b800be50 comp rx=0 tx=0).stop 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.450+0000 7f08ce36b700 1 -- 172.21.15.98:0/3964027831 shutdown_connections 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.450+0000 7f08ce36b700 1 --2- 172.21.15.98:0/3964027831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08c8108290 0x7f08c81086b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.450+0000 7f08ce36b700 1 -- 172.21.15.98:0/3964027831 >> 172.21.15.98:0/3964027831 conn(0x7f08c8101a20 msgr2=0x7f08c8103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:05.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.450+0000 7f08ce36b700 1 -- 172.21.15.98:0/3964027831 shutdown_connections 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.450+0000 7f08ce36b700 1 -- 172.21.15.98:0/3964027831 wait complete. 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.451+0000 7f08ce36b700 1 Processor -- start 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.451+0000 7f08ce36b700 1 -- start start 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.451+0000 7f08ce36b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08c8108290 0x7f08c8199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.451+0000 7f08ce36b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08c8105de0 con 0x7f08c8108290 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.451+0000 7f08c7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08c8108290 0x7f08c8199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.451+0000 7f08c7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08c8108290 0x7f08c8199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32958/0 (socket says 172.21.15.98:32958) 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.451+0000 7f08c7fff700 1 -- 172.21.15.98:0/1473997565 learned_addr learned my addr 172.21.15.98:0/1473997565 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.452+0000 7f08c7fff700 1 -- 172.21.15.98:0/1473997565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c8199fc0 con 0x7f08c8108290 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.452+0000 7f08c7fff700 1 --2- 172.21.15.98:0/1473997565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08c8108290 0x7f08c8199a80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f08b800b340 tx=0x7f08b80040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.452+0000 7f08c57fa700 1 -- 172.21.15.98:0/1473997565 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08b8027070 con 0x7f08c8108290 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.452+0000 7f08c57fa700 1 -- 172.21.15.98:0/1473997565 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f08b8010be0 con 0x7f08c8108290 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.452+0000 7f08ce36b700 1 -- 172.21.15.98:0/1473997565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08c819a250 con 0x7f08c8108290 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.452+0000 7f08c57fa700 1 -- 172.21.15.98:0/1473997565 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08b8022c20 con 0x7f08c8108290 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.452+0000 7f08ce36b700 1 -- 172.21.15.98:0/1473997565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c819cf40 con 0x7f08c8108290 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.453+0000 7f08c57fa700 1 -- 172.21.15.98:0/1473997565 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f08b802a070 con 0x7f08c8108290 2024-05-09T08:41:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.453+0000 7f08ce36b700 1 -- 172.21.15.98:0/1473997565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08c8063980 con 0x7f08c8108290 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.453+0000 7f08c57fa700 1 --2- 172.21.15.98:0/1473997565 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f08b00390c0 0x7f08b003b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.453+0000 7f08c57fa700 1 -- 172.21.15.98:0/1473997565 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f08b801f070 con 0x7f08c8108290 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.453+0000 7f08c77fe700 1 --2- 172.21.15.98:0/1473997565 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f08b00390c0 0x7f08b003b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.453+0000 7f08c77fe700 1 --2- 172.21.15.98:0/1473997565 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f08b00390c0 0x7f08b003b580 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f08bc006fd0 tx=0x7f08bc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.458+0000 7f08c57fa700 1 -- 172.21.15.98:0/1473997565 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f08b8015070 con 0x7f08c8108290 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.682+0000 7f08ce36b700 1 -- 172.21.15.98:0/1473997565 --> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f08c8062760 con 0x7f08b00390c0 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.686+0000 7f08c57fa700 1 -- 172.21.15.98:0/1473997565 <== mgr.14118 v2:172.21.15.98:6800/498605085 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f08c8062760 con 0x7f08b00390c0 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.689+0000 7f08ce36b700 1 -- 172.21.15.98:0/1473997565 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f08b00390c0 msgr2=0x7f08b003b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.689+0000 7f08ce36b700 1 --2- 172.21.15.98:0/1473997565 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f08b00390c0 0x7f08b003b580 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f08bc006fd0 tx=0x7f08bc006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.689+0000 7f08ce36b700 1 -- 172.21.15.98:0/1473997565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08c8108290 msgr2=0x7f08c8199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.689+0000 7f08ce36b700 1 --2- 172.21.15.98:0/1473997565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08c8108290 0x7f08c8199a80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f08b800b340 tx=0x7f08b80040b0 comp rx=0 tx=0).stop 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.689+0000 7f08ce36b700 1 -- 172.21.15.98:0/1473997565 shutdown_connections 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.689+0000 7f08ce36b700 1 --2- 172.21.15.98:0/1473997565 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f08b00390c0 0x7f08b003b580 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.689+0000 7f08ce36b700 1 --2- 172.21.15.98:0/1473997565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08c8108290 0x7f08c8199a80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.689+0000 7f08ce36b700 1 -- 172.21.15.98:0/1473997565 >> 172.21.15.98:0/1473997565 conn(0x7f08c8101a20 msgr2=0x7f08c81900c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:05.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.689+0000 7f08ce36b700 1 -- 172.21.15.98:0/1473997565 shutdown_connections 2024-05-09T08:41:05.198 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:03.689+0000 7f08ce36b700 1 -- 172.21.15.98:0/1473997565 wait complete. 2024-05-09T08:41:05.198 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mgr service... 2024-05-09T08:41:06.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:05 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' 2024-05-09T08:41:06.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:05 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' 2024-05-09T08:41:06.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-09T08:41:06.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.612+0000 7f5bcf757700 1 Processor -- start 2024-05-09T08:41:06.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.612+0000 7f5bcf757700 1 -- start start 2024-05-09T08:41:06.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.612+0000 7f5bcf757700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bc81071b0 0x7f5bc81075d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:06.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.612+0000 7f5bcf757700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc8107ba0 con 0x7f5bc81071b0 2024-05-09T08:41:06.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.612+0000 7f5bcd4f3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bc81071b0 0x7f5bc81075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:06.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.612+0000 7f5bcd4f3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bc81071b0 0x7f5bc81075d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40588/0 (socket says 172.21.15.98:40588) 2024-05-09T08:41:06.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.612+0000 7f5bcd4f3700 1 -- 172.21.15.98:0/339980780 learned_addr learned my addr 172.21.15.98:0/339980780 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:06.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.613+0000 7f5bcd4f3700 1 -- 172.21.15.98:0/339980780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc8107ed0 con 0x7f5bc81071b0 2024-05-09T08:41:06.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.613+0000 7f5bcd4f3700 1 --2- 172.21.15.98:0/339980780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bc81071b0 0x7f5bc81075d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40098a0 tx=0x7f5bc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33156abc39d007b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:06.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.613+0000 7f5bbffff700 1 -- 172.21.15.98:0/339980780 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bc4004160 con 0x7f5bc81071b0 2024-05-09T08:41:06.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.613+0000 7f5bbffff700 1 -- 172.21.15.98:0/339980780 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc4004300 con 0x7f5bc81071b0 2024-05-09T08:41:06.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.613+0000 7f5bbffff700 1 -- 172.21.15.98:0/339980780 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bc4011b20 con 0x7f5bc81071b0 2024-05-09T08:41:06.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.614+0000 7f5bcf757700 1 -- 172.21.15.98:0/339980780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bc81071b0 msgr2=0x7f5bc81075d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:06.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.614+0000 7f5bcf757700 1 --2- 172.21.15.98:0/339980780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bc81071b0 0x7f5bc81075d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40098a0 tx=0x7f5bc4009bb0 comp rx=0 tx=0).stop 2024-05-09T08:41:06.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.614+0000 7f5bcf757700 1 -- 172.21.15.98:0/339980780 shutdown_connections 2024-05-09T08:41:06.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.614+0000 7f5bcf757700 1 --2- 172.21.15.98:0/339980780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bc81071b0 0x7f5bc81075d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:06.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.614+0000 7f5bcf757700 1 -- 172.21.15.98:0/339980780 >> 172.21.15.98:0/339980780 conn(0x7f5bc81019f0 msgr2=0x7f5bc8103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:06.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.614+0000 7f5bcf757700 1 -- 172.21.15.98:0/339980780 shutdown_connections 2024-05-09T08:41:06.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.614+0000 7f5bcf757700 1 -- 172.21.15.98:0/339980780 wait complete. 2024-05-09T08:41:06.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.614+0000 7f5bcf757700 1 Processor -- start 2024-05-09T08:41:06.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.615+0000 7f5bcf757700 1 -- start start 2024-05-09T08:41:06.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.615+0000 7f5bcf757700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bc81071b0 0x7f5bc8106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:06.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.615+0000 7f5bcf757700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc810b280 con 0x7f5bc81071b0 2024-05-09T08:41:06.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.615+0000 7f5bcd4f3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bc81071b0 0x7f5bc8106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.615+0000 7f5bcd4f3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bc81071b0 0x7f5bc8106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40602/0 (socket says 172.21.15.98:40602) 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.615+0000 7f5bcd4f3700 1 -- 172.21.15.98:0/2117610714 learned_addr learned my addr 172.21.15.98:0/2117610714 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.615+0000 7f5bcd4f3700 1 -- 172.21.15.98:0/2117610714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc819e970 con 0x7f5bc81071b0 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.615+0000 7f5bcd4f3700 1 --2- 172.21.15.98:0/2117610714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bc81071b0 0x7f5bc8106790 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40117e0 tx=0x7f5bc4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.615+0000 7f5bbe7fc700 1 -- 172.21.15.98:0/2117610714 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bc4003ba0 con 0x7f5bc81071b0 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.615+0000 7f5bbe7fc700 1 -- 172.21.15.98:0/2117610714 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc401a910 con 0x7f5bc81071b0 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.615+0000 7f5bcf757700 1 -- 172.21.15.98:0/2117610714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc819eb40 con 0x7f5bc81071b0 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.615+0000 7f5bbe7fc700 1 -- 172.21.15.98:0/2117610714 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bc40248f0 con 0x7f5bc81071b0 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.616+0000 7f5bcf757700 1 -- 172.21.15.98:0/2117610714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc819eee0 con 0x7f5bc81071b0 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.616+0000 7f5bbe7fc700 1 -- 172.21.15.98:0/2117610714 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f5bc4022020 con 0x7f5bc81071b0 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.617+0000 7f5bcf757700 1 -- 172.21.15.98:0/2117610714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bc8063980 con 0x7f5bc81071b0 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.617+0000 7f5bbe7fc700 1 --2- 172.21.15.98:0/2117610714 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f5bb4039110 0x7f5bb403b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.617+0000 7f5bbe7fc700 1 -- 172.21.15.98:0/2117610714 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5bc404d8d0 con 0x7f5bc81071b0 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.617+0000 7f5bcccf2700 1 --2- 172.21.15.98:0/2117610714 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f5bb4039110 0x7f5bb403b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:06.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.617+0000 7f5bcccf2700 1 --2- 172.21.15.98:0/2117610714 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f5bb4039110 0x7f5bb403b5d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8006fd0 tx=0x7f5bb8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:06.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.622+0000 7f5bbe7fc700 1 -- 172.21.15.98:0/2117610714 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5bc402cdf0 con 0x7f5bc81071b0 2024-05-09T08:41:06.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.877+0000 7f5bcf757700 1 -- 172.21.15.98:0/2117610714 --> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f5bc8102aa0 con 0x7f5bb4039110 2024-05-09T08:41:06.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.882+0000 7f5bbe7fc700 1 -- 172.21.15.98:0/2117610714 <== mgr.14118 v2:172.21.15.98:6800/498605085 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5bc8102aa0 con 0x7f5bb4039110 2024-05-09T08:41:06.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.885+0000 7f5bcf757700 1 -- 172.21.15.98:0/2117610714 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f5bb4039110 msgr2=0x7f5bb403b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:06.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.885+0000 7f5bcf757700 1 --2- 172.21.15.98:0/2117610714 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f5bb4039110 0x7f5bb403b5d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8006fd0 tx=0x7f5bb8006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:06.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.885+0000 7f5bcf757700 1 -- 172.21.15.98:0/2117610714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bc81071b0 msgr2=0x7f5bc8106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:06.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.885+0000 7f5bcf757700 1 --2- 172.21.15.98:0/2117610714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bc81071b0 0x7f5bc8106790 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40117e0 tx=0x7f5bc4011ba0 comp rx=0 tx=0).stop 2024-05-09T08:41:06.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.885+0000 7f5bcf757700 1 -- 172.21.15.98:0/2117610714 shutdown_connections 2024-05-09T08:41:06.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.885+0000 7f5bcf757700 1 --2- 172.21.15.98:0/2117610714 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f5bb4039110 0x7f5bb403b5d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:06.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.885+0000 7f5bcf757700 1 --2- 172.21.15.98:0/2117610714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bc81071b0 0x7f5bc8106790 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:06.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.885+0000 7f5bcf757700 1 -- 172.21.15.98:0/2117610714 >> 172.21.15.98:0/2117610714 conn(0x7f5bc81019f0 msgr2=0x7f5bc8102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:06.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.885+0000 7f5bcf757700 1 -- 172.21.15.98:0/2117610714 shutdown_connections 2024-05-09T08:41:06.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:05.885+0000 7f5bcf757700 1 -- 172.21.15.98:0/2117610714 wait complete. 2024-05-09T08:41:07.047 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:06 smithi098 ceph-mon[30092]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:41:07.047 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:06 smithi098 ceph-mon[30092]: Saving service mgr spec with placement count:2 2024-05-09T08:41:07.047 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:06 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' 2024-05-09T08:41:08.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.356+0000 7f11add75700 1 Processor -- start 2024-05-09T08:41:08.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.356+0000 7f11add75700 1 -- start start 2024-05-09T08:41:08.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.357+0000 7f11add75700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8108860 0x7f11a810ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:08.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.357+0000 7f11add75700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11a807be90 con 0x7f11a8108860 2024-05-09T08:41:08.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.357+0000 7f11a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8108860 0x7f11a810ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:08.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.357+0000 7f11a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8108860 0x7f11a810ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40604/0 (socket says 172.21.15.98:40604) 2024-05-09T08:41:08.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.357+0000 7f11a77fe700 1 -- 172.21.15.98:0/1170878326 learned_addr learned my addr 172.21.15.98:0/1170878326 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:08.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.357+0000 7f11a77fe700 1 -- 172.21.15.98:0/1170878326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a810b190 con 0x7f11a8108860 2024-05-09T08:41:08.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.357+0000 7f11a77fe700 1 --2- 172.21.15.98:0/1170878326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8108860 0x7f11a810ac50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f11900098a0 tx=0x7f1190009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=276e945ffbe4af7f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:08.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.358+0000 7f11a67fc700 1 -- 172.21.15.98:0/1170878326 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1190004160 con 0x7f11a8108860 2024-05-09T08:41:08.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.358+0000 7f11a67fc700 1 -- 172.21.15.98:0/1170878326 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1190004300 con 0x7f11a8108860 2024-05-09T08:41:08.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.358+0000 7f11a67fc700 1 -- 172.21.15.98:0/1170878326 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1190011b20 con 0x7f11a8108860 2024-05-09T08:41:08.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.358+0000 7f11add75700 1 -- 172.21.15.98:0/1170878326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8108860 msgr2=0x7f11a810ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:08.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.358+0000 7f11add75700 1 --2- 172.21.15.98:0/1170878326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8108860 0x7f11a810ac50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f11900098a0 tx=0x7f1190009bb0 comp rx=0 tx=0).stop 2024-05-09T08:41:08.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.358+0000 7f11add75700 1 -- 172.21.15.98:0/1170878326 shutdown_connections 2024-05-09T08:41:08.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.358+0000 7f11add75700 1 --2- 172.21.15.98:0/1170878326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8108860 0x7f11a810ac50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:08.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.358+0000 7f11add75700 1 -- 172.21.15.98:0/1170878326 >> 172.21.15.98:0/1170878326 conn(0x7f11a8102210 msgr2=0x7f11a8104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:08.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.358+0000 7f11add75700 1 -- 172.21.15.98:0/1170878326 shutdown_connections 2024-05-09T08:41:08.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.358+0000 7f11add75700 1 -- 172.21.15.98:0/1170878326 wait complete. 2024-05-09T08:41:08.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.359+0000 7f11add75700 1 Processor -- start 2024-05-09T08:41:08.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.359+0000 7f11add75700 1 -- start start 2024-05-09T08:41:08.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.359+0000 7f11add75700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8108860 0x7f11a819bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:08.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.359+0000 7f11add75700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11a810c1f0 con 0x7f11a8108860 2024-05-09T08:41:08.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.360+0000 7f11a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8108860 0x7f11a819bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:08.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.360+0000 7f11a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8108860 0x7f11a819bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40614/0 (socket says 172.21.15.98:40614) 2024-05-09T08:41:08.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.360+0000 7f11a77fe700 1 -- 172.21.15.98:0/2126834606 learned_addr learned my addr 172.21.15.98:0/2126834606 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:08.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.360+0000 7f11a77fe700 1 -- 172.21.15.98:0/2126834606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a819c250 con 0x7f11a8108860 2024-05-09T08:41:08.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.360+0000 7f11a77fe700 1 --2- 172.21.15.98:0/2126834606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8108860 0x7f11a819bd10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f11900117e0 tx=0x7f1190011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:08.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.360+0000 7f11a4ff9700 1 -- 172.21.15.98:0/2126834606 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1190003ba0 con 0x7f11a8108860 2024-05-09T08:41:08.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.360+0000 7f11a4ff9700 1 -- 172.21.15.98:0/2126834606 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f119001a910 con 0x7f11a8108860 2024-05-09T08:41:08.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.360+0000 7f11a4ff9700 1 -- 172.21.15.98:0/2126834606 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11900248f0 con 0x7f11a8108860 2024-05-09T08:41:08.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.360+0000 7f11add75700 1 -- 172.21.15.98:0/2126834606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a819c4e0 con 0x7f11a8108860 2024-05-09T08:41:08.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.360+0000 7f11add75700 1 -- 172.21.15.98:0/2126834606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a819c940 con 0x7f11a8108860 2024-05-09T08:41:08.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.361+0000 7f11a4ff9700 1 -- 172.21.15.98:0/2126834606 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f1190022020 con 0x7f11a8108860 2024-05-09T08:41:08.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.361+0000 7f11a4ff9700 1 --2- 172.21.15.98:0/2126834606 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f1194039050 0x7f119403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:08.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.361+0000 7f11a4ff9700 1 -- 172.21.15.98:0/2126834606 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f119001a4f0 con 0x7f11a8108860 2024-05-09T08:41:08.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.361+0000 7f11add75700 1 -- 172.21.15.98:0/2126834606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11a8063980 con 0x7f11a8108860 2024-05-09T08:41:08.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.362+0000 7f11a6ffd700 1 --2- 172.21.15.98:0/2126834606 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f1194039050 0x7f119403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:08.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.362+0000 7f11a6ffd700 1 --2- 172.21.15.98:0/2126834606 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f1194039050 0x7f119403b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1198006fd0 tx=0x7f1198006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:08.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.367+0000 7f11a4ff9700 1 -- 172.21.15.98:0/2126834606 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f119002cdf0 con 0x7f11a8108860 2024-05-09T08:41:08.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.571+0000 7f11add75700 1 -- 172.21.15.98:0/2126834606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f11a819f660 con 0x7f11a8108860 2024-05-09T08:41:08.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.574+0000 7f11a4ff9700 1 -- 172.21.15.98:0/2126834606 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f1190020c10 con 0x7f11a8108860 2024-05-09T08:41:08.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.577+0000 7f11add75700 1 -- 172.21.15.98:0/2126834606 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f1194039050 msgr2=0x7f119403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:08.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.577+0000 7f11add75700 1 --2- 172.21.15.98:0/2126834606 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f1194039050 0x7f119403b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1198006fd0 tx=0x7f1198006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:08.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.577+0000 7f11add75700 1 -- 172.21.15.98:0/2126834606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8108860 msgr2=0x7f11a819bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:08.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.577+0000 7f11add75700 1 --2- 172.21.15.98:0/2126834606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8108860 0x7f11a819bd10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f11900117e0 tx=0x7f1190011ba0 comp rx=0 tx=0).stop 2024-05-09T08:41:08.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.577+0000 7f11add75700 1 -- 172.21.15.98:0/2126834606 shutdown_connections 2024-05-09T08:41:08.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.577+0000 7f11add75700 1 --2- 172.21.15.98:0/2126834606 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f1194039050 0x7f119403b510 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:08.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.577+0000 7f11add75700 1 --2- 172.21.15.98:0/2126834606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8108860 0x7f11a819bd10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:08.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.577+0000 7f11add75700 1 -- 172.21.15.98:0/2126834606 >> 172.21.15.98:0/2126834606 conn(0x7f11a8102210 msgr2=0x7f11a8102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:08.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.577+0000 7f11add75700 1 -- 172.21.15.98:0/2126834606 shutdown_connections 2024-05-09T08:41:08.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:07.577+0000 7f11add75700 1 -- 172.21.15.98:0/2126834606 wait complete. 2024-05-09T08:41:08.299 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:08 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' 2024-05-09T08:41:08.300 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:08 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2126834606' entity='client.admin' 2024-05-09T08:41:09.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:09 smithi098 ceph-mon[30092]: from='mgr.14118 172.21.15.98:0/2478690691' entity='mgr.x' 2024-05-09T08:41:09.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:09 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1324656023' entity='client.admin' 2024-05-09T08:41:09.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.631+0000 7f1046d18700 1 Processor -- start 2024-05-09T08:41:09.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.631+0000 7f1046d18700 1 -- start start 2024-05-09T08:41:09.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.631+0000 7f1046d18700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1040106cd0 0x7f10401070f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:09.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.631+0000 7f1046d18700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10401076c0 con 0x7f1040106cd0 2024-05-09T08:41:09.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.631+0000 7f1044ab4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1040106cd0 0x7f10401070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:09.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.631+0000 7f1044ab4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1040106cd0 0x7f10401070f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40618/0 (socket says 172.21.15.98:40618) 2024-05-09T08:41:09.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.631+0000 7f1044ab4700 1 -- 172.21.15.98:0/221394075 learned_addr learned my addr 172.21.15.98:0/221394075 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:09.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.631+0000 7f1044ab4700 1 -- 172.21.15.98:0/221394075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1040107ed0 con 0x7f1040106cd0 2024-05-09T08:41:09.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.631+0000 7f1044ab4700 1 --2- 172.21.15.98:0/221394075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1040106cd0 0x7f10401070f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f10340098a0 tx=0x7f1034009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f9c6ce84ef42f70 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:09.522 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.632+0000 7f103f7fe700 1 -- 172.21.15.98:0/221394075 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1034004160 con 0x7f1040106cd0 2024-05-09T08:41:09.522 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.632+0000 7f103f7fe700 1 -- 172.21.15.98:0/221394075 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1034004300 con 0x7f1040106cd0 2024-05-09T08:41:09.522 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.632+0000 7f103f7fe700 1 -- 172.21.15.98:0/221394075 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1034011b20 con 0x7f1040106cd0 2024-05-09T08:41:09.522 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.632+0000 7f1046d18700 1 -- 172.21.15.98:0/221394075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1040106cd0 msgr2=0x7f10401070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:09.522 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.632+0000 7f1046d18700 1 --2- 172.21.15.98:0/221394075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1040106cd0 0x7f10401070f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f10340098a0 tx=0x7f1034009bb0 comp rx=0 tx=0).stop 2024-05-09T08:41:09.522 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.632+0000 7f1046d18700 1 -- 172.21.15.98:0/221394075 shutdown_connections 2024-05-09T08:41:09.522 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.632+0000 7f1046d18700 1 --2- 172.21.15.98:0/221394075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1040106cd0 0x7f10401070f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:09.522 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.632+0000 7f1046d18700 1 -- 172.21.15.98:0/221394075 >> 172.21.15.98:0/221394075 conn(0x7f1040102230 msgr2=0x7f10401046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:09.522 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.632+0000 7f1046d18700 1 -- 172.21.15.98:0/221394075 shutdown_connections 2024-05-09T08:41:09.522 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.632+0000 7f1046d18700 1 -- 172.21.15.98:0/221394075 wait complete. 2024-05-09T08:41:09.522 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.633+0000 7f1046d18700 1 Processor -- start 2024-05-09T08:41:09.522 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.633+0000 7f1046d18700 1 -- start start 2024-05-09T08:41:09.522 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.633+0000 7f1046d18700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1040106cd0 0x7f104007ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.633+0000 7f1046d18700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f104010b280 con 0x7f1040106cd0 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.633+0000 7f1044ab4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1040106cd0 0x7f104007ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.633+0000 7f1044ab4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1040106cd0 0x7f104007ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40624/0 (socket says 172.21.15.98:40624) 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.633+0000 7f1044ab4700 1 -- 172.21.15.98:0/1324656023 learned_addr learned my addr 172.21.15.98:0/1324656023 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.634+0000 7f1044ab4700 1 -- 172.21.15.98:0/1324656023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104019c790 con 0x7f1040106cd0 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.634+0000 7f1044ab4700 1 --2- 172.21.15.98:0/1324656023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1040106cd0 0x7f104007ba70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f10340117e0 tx=0x7f1034011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.634+0000 7f103dffb700 1 -- 172.21.15.98:0/1324656023 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1034003bf0 con 0x7f1040106cd0 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.634+0000 7f1046d18700 1 -- 172.21.15.98:0/1324656023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10401a0da0 con 0x7f1040106cd0 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.634+0000 7f103dffb700 1 -- 172.21.15.98:0/1324656023 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f103401a910 con 0x7f1040106cd0 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.634+0000 7f103dffb700 1 -- 172.21.15.98:0/1324656023 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10340249a0 con 0x7f1040106cd0 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.634+0000 7f1046d18700 1 -- 172.21.15.98:0/1324656023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10401a1140 con 0x7f1040106cd0 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.635+0000 7f103dffb700 1 -- 172.21.15.98:0/1324656023 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f1034022020 con 0x7f1040106cd0 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.635+0000 7f1046d18700 1 -- 172.21.15.98:0/1324656023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1040063980 con 0x7f1040106cd0 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.636+0000 7f103dffb700 1 --2- 172.21.15.98:0/1324656023 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f10300390c0 0x7f103003b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.636+0000 7f103dffb700 1 -- 172.21.15.98:0/1324656023 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f103404d350 con 0x7f1040106cd0 2024-05-09T08:41:09.523 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.636+0000 7f103ffff700 1 --2- 172.21.15.98:0/1324656023 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f10300390c0 0x7f103003b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.636+0000 7f103ffff700 1 --2- 172.21.15.98:0/1324656023 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f10300390c0 0x7f103003b580 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f102c006fd0 tx=0x7f102c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.641+0000 7f103dffb700 1 -- 172.21.15.98:0/1324656023 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1034011e20 con 0x7f1040106cd0 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.849+0000 7f1046d18700 1 -- 172.21.15.98:0/1324656023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f10401a1550 con 0x7f1040106cd0 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.851+0000 7f103dffb700 1 -- 172.21.15.98:0/1324656023 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f1034024c00 con 0x7f1040106cd0 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.854+0000 7f1046d18700 1 -- 172.21.15.98:0/1324656023 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f10300390c0 msgr2=0x7f103003b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.854+0000 7f1046d18700 1 --2- 172.21.15.98:0/1324656023 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f10300390c0 0x7f103003b580 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f102c006fd0 tx=0x7f102c006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.854+0000 7f1046d18700 1 -- 172.21.15.98:0/1324656023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1040106cd0 msgr2=0x7f104007ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.854+0000 7f1046d18700 1 --2- 172.21.15.98:0/1324656023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1040106cd0 0x7f104007ba70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f10340117e0 tx=0x7f1034011c10 comp rx=0 tx=0).stop 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.854+0000 7f1046d18700 1 -- 172.21.15.98:0/1324656023 shutdown_connections 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.854+0000 7f1046d18700 1 --2- 172.21.15.98:0/1324656023 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f10300390c0 0x7f103003b580 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.854+0000 7f1046d18700 1 --2- 172.21.15.98:0/1324656023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1040106cd0 0x7f104007ba70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.854+0000 7f1046d18700 1 -- 172.21.15.98:0/1324656023 >> 172.21.15.98:0/1324656023 conn(0x7f1040102230 msgr2=0x7f1040102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.854+0000 7f1046d18700 1 -- 172.21.15.98:0/1324656023 shutdown_connections 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:08.854+0000 7f1046d18700 1 -- 172.21.15.98:0/1324656023 wait complete. 2024-05-09T08:41:09.524 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling the dashboard module... 2024-05-09T08:41:10.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:10 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4049659433' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-09T08:41:10.338 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:10 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: ignoring --setuser ceph since I am not root 2024-05-09T08:41:10.338 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:10 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: ignoring --setgroup ceph since I am not root 2024-05-09T08:41:10.587 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:10 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:10.423+0000 7f4efe2b2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:41:10.906 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:10 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:10.608+0000 7f4efe2b2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:41:10.906 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:10 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:10.712+0000 7f4efe2b2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:41:10.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.910+0000 7fe799a2d700 1 Processor -- start 2024-05-09T08:41:10.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.910+0000 7fe799a2d700 1 -- start start 2024-05-09T08:41:10.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.910+0000 7fe799a2d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe794106d00 0x7fe794107120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:10.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.911+0000 7fe799a2d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe7941076f0 con 0x7fe794106d00 2024-05-09T08:41:10.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.911+0000 7fe792ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe794106d00 0x7fe794107120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:10.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.911+0000 7fe792ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe794106d00 0x7fe794107120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40626/0 (socket says 172.21.15.98:40626) 2024-05-09T08:41:10.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.911+0000 7fe792ffd700 1 -- 172.21.15.98:0/851219881 learned_addr learned my addr 172.21.15.98:0/851219881 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:10.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.911+0000 7fe792ffd700 1 -- 172.21.15.98:0/851219881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe794107f00 con 0x7fe794106d00 2024-05-09T08:41:10.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.911+0000 7fe792ffd700 1 --2- 172.21.15.98:0/851219881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe794106d00 0x7fe794107120 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe7840098a0 tx=0x7fe784009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cddc7c47555fd9f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:10.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.911+0000 7fe791ffb700 1 -- 172.21.15.98:0/851219881 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe784004160 con 0x7fe794106d00 2024-05-09T08:41:10.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.911+0000 7fe791ffb700 1 -- 172.21.15.98:0/851219881 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe784004300 con 0x7fe794106d00 2024-05-09T08:41:10.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.911+0000 7fe791ffb700 1 -- 172.21.15.98:0/851219881 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe784011b20 con 0x7fe794106d00 2024-05-09T08:41:10.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.912+0000 7fe799a2d700 1 -- 172.21.15.98:0/851219881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe794106d00 msgr2=0x7fe794107120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:10.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.912+0000 7fe799a2d700 1 --2- 172.21.15.98:0/851219881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe794106d00 0x7fe794107120 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe7840098a0 tx=0x7fe784009bb0 comp rx=0 tx=0).stop 2024-05-09T08:41:10.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.912+0000 7fe799a2d700 1 -- 172.21.15.98:0/851219881 shutdown_connections 2024-05-09T08:41:10.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.912+0000 7fe799a2d700 1 --2- 172.21.15.98:0/851219881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe794106d00 0x7fe794107120 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:10.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.912+0000 7fe799a2d700 1 -- 172.21.15.98:0/851219881 >> 172.21.15.98:0/851219881 conn(0x7fe7941022a0 msgr2=0x7fe7941046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:10.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.912+0000 7fe799a2d700 1 -- 172.21.15.98:0/851219881 shutdown_connections 2024-05-09T08:41:10.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.912+0000 7fe799a2d700 1 -- 172.21.15.98:0/851219881 wait complete. 2024-05-09T08:41:10.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.912+0000 7fe799a2d700 1 Processor -- start 2024-05-09T08:41:10.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.913+0000 7fe799a2d700 1 -- start start 2024-05-09T08:41:10.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.913+0000 7fe799a2d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe794106d00 0x7fe79419bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:10.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.913+0000 7fe799a2d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe79410b2b0 con 0x7fe794106d00 2024-05-09T08:41:10.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.913+0000 7fe792ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe794106d00 0x7fe79419bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:10.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.913+0000 7fe792ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe794106d00 0x7fe79419bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40642/0 (socket says 172.21.15.98:40642) 2024-05-09T08:41:10.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.913+0000 7fe792ffd700 1 -- 172.21.15.98:0/4049659433 learned_addr learned my addr 172.21.15.98:0/4049659433 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:10.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.913+0000 7fe792ffd700 1 -- 172.21.15.98:0/4049659433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79419c240 con 0x7fe794106d00 2024-05-09T08:41:10.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.913+0000 7fe792ffd700 1 --2- 172.21.15.98:0/4049659433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe794106d00 0x7fe79419bd00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe7840117e0 tx=0x7fe784011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:10.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.914+0000 7fe798a2b700 1 -- 172.21.15.98:0/4049659433 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe784003bf0 con 0x7fe794106d00 2024-05-09T08:41:10.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.914+0000 7fe798a2b700 1 -- 172.21.15.98:0/4049659433 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe78401a910 con 0x7fe794106d00 2024-05-09T08:41:10.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.914+0000 7fe798a2b700 1 -- 172.21.15.98:0/4049659433 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7840249a0 con 0x7fe794106d00 2024-05-09T08:41:10.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.914+0000 7fe799a2d700 1 -- 172.21.15.98:0/4049659433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe79419c4d0 con 0x7fe794106d00 2024-05-09T08:41:10.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.914+0000 7fe799a2d700 1 -- 172.21.15.98:0/4049659433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe79419c930 con 0x7fe794106d00 2024-05-09T08:41:10.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.914+0000 7fe798a2b700 1 -- 172.21.15.98:0/4049659433 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fe784022020 con 0x7fe794106d00 2024-05-09T08:41:10.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.915+0000 7fe799a2d700 1 -- 172.21.15.98:0/4049659433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe794063980 con 0x7fe794106d00 2024-05-09T08:41:10.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.915+0000 7fe798a2b700 1 --2- 172.21.15.98:0/4049659433 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fe77c039110 0x7fe77c03b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:10.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.915+0000 7fe798a2b700 1 -- 172.21.15.98:0/4049659433 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe78404d310 con 0x7fe794106d00 2024-05-09T08:41:10.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.915+0000 7fe7927fc700 1 --2- 172.21.15.98:0/4049659433 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fe77c039110 0x7fe77c03b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:10.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.915+0000 7fe7927fc700 1 --2- 172.21.15.98:0/4049659433 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fe77c039110 0x7fe77c03b5d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe788006fd0 tx=0x7fe788006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:10.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:09.920+0000 7fe798a2b700 1 -- 172.21.15.98:0/4049659433 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe784011e20 con 0x7fe794106d00 2024-05-09T08:41:10.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:10.175+0000 7fe799a2d700 1 -- 172.21.15.98:0/4049659433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fe794062f30 con 0x7fe794106d00 2024-05-09T08:41:10.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:10.245+0000 7fe798a2b700 1 -- 172.21.15.98:0/4049659433 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fe784019ca0 con 0x7fe794106d00 2024-05-09T08:41:10.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:10.246+0000 7fe798a2b700 1 -- 172.21.15.98:0/4049659433 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fe78402a2a0 con 0x7fe794106d00 2024-05-09T08:41:10.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:10.247+0000 7fe799a2d700 1 -- 172.21.15.98:0/4049659433 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fe77c039110 msgr2=0x7fe77c03b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:10.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:10.247+0000 7fe799a2d700 1 --2- 172.21.15.98:0/4049659433 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fe77c039110 0x7fe77c03b5d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe788006fd0 tx=0x7fe788006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:10.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:10.247+0000 7fe799a2d700 1 -- 172.21.15.98:0/4049659433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe794106d00 msgr2=0x7fe79419bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:10.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:10.247+0000 7fe799a2d700 1 --2- 172.21.15.98:0/4049659433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe794106d00 0x7fe79419bd00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe7840117e0 tx=0x7fe784011c10 comp rx=0 tx=0).stop 2024-05-09T08:41:10.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:10.247+0000 7fe799a2d700 1 -- 172.21.15.98:0/4049659433 shutdown_connections 2024-05-09T08:41:10.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:10.247+0000 7fe799a2d700 1 --2- 172.21.15.98:0/4049659433 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7fe77c039110 0x7fe77c03b5d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:10.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:10.247+0000 7fe799a2d700 1 --2- 172.21.15.98:0/4049659433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe794106d00 0x7fe79419bd00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:10.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:10.247+0000 7fe799a2d700 1 -- 172.21.15.98:0/4049659433 >> 172.21.15.98:0/4049659433 conn(0x7fe7941022a0 msgr2=0x7fe794103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:10.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:10.248+0000 7fe799a2d700 1 -- 172.21.15.98:0/4049659433 shutdown_connections 2024-05-09T08:41:10.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:10.248+0000 7fe799a2d700 1 -- 172.21.15.98:0/4049659433 wait complete. 2024-05-09T08:41:11.244 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:11 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:11.030+0000 7f4efe2b2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:41:11.244 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:11 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:11.117+0000 7f4efe2b2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:41:11.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:11 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4049659433' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-09T08:41:11.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:11 smithi098 ceph-mon[30092]: mgrmap e9: x(active, since 19s) 2024-05-09T08:41:12.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:12 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/303075422' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-09T08:41:12.700 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:12 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:12.339+0000 7f4efe2b2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:41:12.700 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:12 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:12.523+0000 7f4efe2b2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:41:12.700 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:12 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:12.700+0000 7f4efe2b2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:41:12.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:41:12.751 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-09T08:41:12.751 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-09T08:41:12.751 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-09T08:41:12.751 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-09T08:41:12.751 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:41:12.751 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.822+0000 7f812864c700 1 Processor -- start 2024-05-09T08:41:12.751 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.822+0000 7f812864c700 1 -- start start 2024-05-09T08:41:12.751 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.823+0000 7f812864c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120108870 0x7f812010ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:12.751 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.823+0000 7f812864c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f812007be90 con 0x7f8120108870 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.823+0000 7f81263e8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120108870 0x7f812010ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.823+0000 7f81263e8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120108870 0x7f812010ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40658/0 (socket says 172.21.15.98:40658) 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.823+0000 7f81263e8700 1 -- 172.21.15.98:0/3018440754 learned_addr learned my addr 172.21.15.98:0/3018440754 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.823+0000 7f81263e8700 1 -- 172.21.15.98:0/3018440754 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f812010b1a0 con 0x7f8120108870 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.823+0000 7f81263e8700 1 --2- 172.21.15.98:0/3018440754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120108870 0x7f812010ac60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f811c0098a0 tx=0x7f811c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ed700696f87d2e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.824+0000 7f81253e6700 1 -- 172.21.15.98:0/3018440754 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f811c004160 con 0x7f8120108870 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.824+0000 7f81253e6700 1 -- 172.21.15.98:0/3018440754 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f811c004300 con 0x7f8120108870 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.824+0000 7f81253e6700 1 -- 172.21.15.98:0/3018440754 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f811c011b20 con 0x7f8120108870 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.824+0000 7f812864c700 1 -- 172.21.15.98:0/3018440754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120108870 msgr2=0x7f812010ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.824+0000 7f812864c700 1 --2- 172.21.15.98:0/3018440754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120108870 0x7f812010ac60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f811c0098a0 tx=0x7f811c009bb0 comp rx=0 tx=0).stop 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.824+0000 7f812864c700 1 -- 172.21.15.98:0/3018440754 shutdown_connections 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.824+0000 7f812864c700 1 --2- 172.21.15.98:0/3018440754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120108870 0x7f812010ac60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.824+0000 7f812864c700 1 -- 172.21.15.98:0/3018440754 >> 172.21.15.98:0/3018440754 conn(0x7f8120102260 msgr2=0x7f81201046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.824+0000 7f812864c700 1 -- 172.21.15.98:0/3018440754 shutdown_connections 2024-05-09T08:41:12.752 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.824+0000 7f812864c700 1 -- 172.21.15.98:0/3018440754 wait complete. 2024-05-09T08:41:12.753 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.825+0000 7f812864c700 1 Processor -- start 2024-05-09T08:41:12.753 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.825+0000 7f812864c700 1 -- start start 2024-05-09T08:41:12.753 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.825+0000 7f812864c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120108870 0x7f812019de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:12.753 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.825+0000 7f812864c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f812010c200 con 0x7f8120108870 2024-05-09T08:41:12.753 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.825+0000 7f81263e8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120108870 0x7f812019de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:12.753 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.825+0000 7f81263e8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120108870 0x7f812019de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40674/0 (socket says 172.21.15.98:40674) 2024-05-09T08:41:12.753 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.825+0000 7f81263e8700 1 -- 172.21.15.98:0/303075422 learned_addr learned my addr 172.21.15.98:0/303075422 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:12.753 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.826+0000 7f81263e8700 1 -- 172.21.15.98:0/303075422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f812019e370 con 0x7f8120108870 2024-05-09T08:41:12.753 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.826+0000 7f81263e8700 1 --2- 172.21.15.98:0/303075422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120108870 0x7f812019de30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f811c0117e0 tx=0x7f811c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:12.753 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.826+0000 7f81177fe700 1 -- 172.21.15.98:0/303075422 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f811c003bf0 con 0x7f8120108870 2024-05-09T08:41:12.753 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.826+0000 7f81177fe700 1 -- 172.21.15.98:0/303075422 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f811c01a910 con 0x7f8120108870 2024-05-09T08:41:12.753 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.826+0000 7f812864c700 1 -- 172.21.15.98:0/303075422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f812019e600 con 0x7f8120108870 2024-05-09T08:41:12.753 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.826+0000 7f81177fe700 1 -- 172.21.15.98:0/303075422 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f811c024970 con 0x7f8120108870 2024-05-09T08:41:12.753 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.826+0000 7f812864c700 1 -- 172.21.15.98:0/303075422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f812019ea60 con 0x7f8120108870 2024-05-09T08:41:12.754 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.827+0000 7f81177fe700 1 -- 172.21.15.98:0/303075422 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f811c022020 con 0x7f8120108870 2024-05-09T08:41:12.754 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.827+0000 7f81177fe700 1 --2- 172.21.15.98:0/303075422 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f810c039110 0x7f810c03b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:12.754 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.827+0000 7f81177fe700 1 -- 172.21.15.98:0/303075422 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f811c04d430 con 0x7f8120108870 2024-05-09T08:41:12.754 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.827+0000 7f8125be7700 1 -- 172.21.15.98:0/303075422 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f810c039110 msgr2=0x7f810c03b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/498605085 2024-05-09T08:41:12.754 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.827+0000 7f8125be7700 1 --2- 172.21.15.98:0/303075422 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f810c039110 0x7f810c03b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:41:12.754 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.827+0000 7f812864c700 1 -- 172.21.15.98:0/303075422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8120063980 con 0x7f8120108870 2024-05-09T08:41:12.754 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:11.832+0000 7f81177fe700 1 -- 172.21.15.98:0/303075422 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f811c011e20 con 0x7f8120108870 2024-05-09T08:41:12.754 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:12.028+0000 7f8125be7700 1 -- 172.21.15.98:0/303075422 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f810c039110 msgr2=0x7f810c03b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/498605085 2024-05-09T08:41:12.754 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:12.028+0000 7f8125be7700 1 --2- 172.21.15.98:0/303075422 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f810c039110 0x7f810c03b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:41:12.754 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:12.089+0000 7f812864c700 1 -- 172.21.15.98:0/303075422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f81201a1390 con 0x7f8120108870 2024-05-09T08:41:12.754 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:12.090+0000 7f81177fe700 1 -- 172.21.15.98:0/303075422 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f811c02c350 con 0x7f8120108870 2024-05-09T08:41:12.754 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:12.092+0000 7f812864c700 1 -- 172.21.15.98:0/303075422 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f810c039110 msgr2=0x7f810c03b5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:41:12.755 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:12.092+0000 7f812864c700 1 --2- 172.21.15.98:0/303075422 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f810c039110 0x7f810c03b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:12.755 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:12.092+0000 7f812864c700 1 -- 172.21.15.98:0/303075422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120108870 msgr2=0x7f812019de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:12.755 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:12.092+0000 7f812864c700 1 --2- 172.21.15.98:0/303075422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120108870 0x7f812019de30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f811c0117e0 tx=0x7f811c011c10 comp rx=0 tx=0).stop 2024-05-09T08:41:12.755 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:12.092+0000 7f812864c700 1 -- 172.21.15.98:0/303075422 shutdown_connections 2024-05-09T08:41:12.755 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:12.092+0000 7f812864c700 1 --2- 172.21.15.98:0/303075422 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f810c039110 0x7f810c03b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:12.755 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:12.092+0000 7f812864c700 1 --2- 172.21.15.98:0/303075422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120108870 0x7f812019de30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:12.755 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:12.092+0000 7f812864c700 1 -- 172.21.15.98:0/303075422 >> 172.21.15.98:0/303075422 conn(0x7f8120102260 msgr2=0x7f8120106690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:12.755 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:12.092+0000 7f812864c700 1 -- 172.21.15.98:0/303075422 shutdown_connections 2024-05-09T08:41:12.755 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:12.092+0000 7f812864c700 1 -- 172.21.15.98:0/303075422 wait complete. 2024-05-09T08:41:12.755 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-05-09T08:41:12.755 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 9... 2024-05-09T08:41:13.314 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:13 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:13.009+0000 7f4efe2b2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:41:13.314 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:13 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:13.106+0000 7f4efe2b2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:41:13.314 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:13 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:13.315+0000 7f4efe2b2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:41:13.587 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:13 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:13.510+0000 7f4efe2b2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:41:14.517 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:14 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:14.103+0000 7f4efe2b2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:41:14.813 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:14 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:14.518+0000 7f4efe2b2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:41:14.813 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:14 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:14.614+0000 7f4efe2b2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:41:14.813 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:14 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:14.814+0000 7f4efe2b2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:41:15.587 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:15 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:15.327+0000 7f4efe2b2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:41:15.587 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:15 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:15.417+0000 7f4efe2b2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:41:15.993 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:15 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:15.656+0000 7f4efe2b2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:41:15.993 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:15 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:15.787+0000 7f4efe2b2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:41:15.993 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:15 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:15.880+0000 7f4efe2b2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:41:15.993 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:15 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:15.993+0000 7f4efe2b2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:41:16.587 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:16 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:16.290+0000 7f4efe2b2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:41:17.147 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:16 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:16.882+0000 7f4efe2b2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:41:17.148 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:41:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:41:17.096+0000 7f4efe2b2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:41:17.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:17 smithi098 ceph-mon[30092]: Active manager daemon x restarted 2024-05-09T08:41:17.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:17 smithi098 ceph-mon[30092]: Activating manager daemon x 2024-05-09T08:41:17.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:17 smithi098 ceph-mon[30092]: osdmap e3: 0 total, 0 up, 0 in 2024-05-09T08:41:17.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:17 smithi098 ceph-mon[30092]: mgrmap e10: x(active, starting, since 0.00322402s) 2024-05-09T08:41:17.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:41:17.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:41:17.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:41:17.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:41:17.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:41:17.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:17 smithi098 ceph-mon[30092]: Manager daemon x is now available 2024-05-09T08:41:17.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:41:17.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:41:18.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:41:18.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:18 smithi098 ceph-mon[30092]: mgrmap e11: x(active, since 1.00581s) 2024-05-09T08:41:19.054 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:41:19.054 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-09T08:41:19.054 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-09T08:41:19.054 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:41:19.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.157+0000 7f2da08c6700 1 Processor -- start 2024-05-09T08:41:19.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.157+0000 7f2da08c6700 1 -- start start 2024-05-09T08:41:19.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.158+0000 7f2da08c6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d9c1088a0 0x7f2d9c10ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:19.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.158+0000 7f2da08c6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c07be70 con 0x7f2d9c1088a0 2024-05-09T08:41:19.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.158+0000 7f2d9a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d9c1088a0 0x7f2d9c10ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:19.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.158+0000 7f2d9a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d9c1088a0 0x7f2d9c10ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40688/0 (socket says 172.21.15.98:40688) 2024-05-09T08:41:19.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.158+0000 7f2d9a59c700 1 -- 172.21.15.98:0/1298429531 learned_addr learned my addr 172.21.15.98:0/1298429531 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:19.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.159+0000 7f2d9a59c700 1 -- 172.21.15.98:0/1298429531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9c10b1d0 con 0x7f2d9c1088a0 2024-05-09T08:41:19.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.159+0000 7f2d9a59c700 1 --2- 172.21.15.98:0/1298429531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d9c1088a0 0x7f2d9c10ac90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0098a0 tx=0x7f2d8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe95fdfad856b2d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:19.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.159+0000 7f2d9959a700 1 -- 172.21.15.98:0/1298429531 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d8c004160 con 0x7f2d9c1088a0 2024-05-09T08:41:19.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.159+0000 7f2d9959a700 1 -- 172.21.15.98:0/1298429531 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8c004300 con 0x7f2d9c1088a0 2024-05-09T08:41:19.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.159+0000 7f2d9959a700 1 -- 172.21.15.98:0/1298429531 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d8c011b20 con 0x7f2d9c1088a0 2024-05-09T08:41:19.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.160+0000 7f2da08c6700 1 -- 172.21.15.98:0/1298429531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d9c1088a0 msgr2=0x7f2d9c10ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:19.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.160+0000 7f2da08c6700 1 --2- 172.21.15.98:0/1298429531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d9c1088a0 0x7f2d9c10ac90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0098a0 tx=0x7f2d8c009bb0 comp rx=0 tx=0).stop 2024-05-09T08:41:19.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.160+0000 7f2da08c6700 1 -- 172.21.15.98:0/1298429531 shutdown_connections 2024-05-09T08:41:19.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.160+0000 7f2da08c6700 1 --2- 172.21.15.98:0/1298429531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d9c1088a0 0x7f2d9c10ac90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:19.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.160+0000 7f2da08c6700 1 -- 172.21.15.98:0/1298429531 >> 172.21.15.98:0/1298429531 conn(0x7f2d9c102250 msgr2=0x7f2d9c1046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:19.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.160+0000 7f2da08c6700 1 -- 172.21.15.98:0/1298429531 shutdown_connections 2024-05-09T08:41:19.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.160+0000 7f2da08c6700 1 -- 172.21.15.98:0/1298429531 wait complete. 2024-05-09T08:41:19.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.161+0000 7f2da08c6700 1 Processor -- start 2024-05-09T08:41:19.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.161+0000 7f2da08c6700 1 -- start start 2024-05-09T08:41:19.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.161+0000 7f2da08c6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d9c1088a0 0x7f2d9c108120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:19.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.161+0000 7f2da08c6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c10c1e0 con 0x7f2d9c1088a0 2024-05-09T08:41:19.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.161+0000 7f2d9a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d9c1088a0 0x7f2d9c108120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:19.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.161+0000 7f2d9a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d9c1088a0 0x7f2d9c108120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40702/0 (socket says 172.21.15.98:40702) 2024-05-09T08:41:19.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.161+0000 7f2d9a59c700 1 -- 172.21.15.98:0/3404124406 learned_addr learned my addr 172.21.15.98:0/3404124406 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:19.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.161+0000 7f2d9a59c700 1 -- 172.21.15.98:0/3404124406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9c108660 con 0x7f2d9c1088a0 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.161+0000 7f2d9a59c700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d9c1088a0 0x7f2d9c108120 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0117e0 tx=0x7f2d8c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.162+0000 7f2d8b7fe700 1 -- 172.21.15.98:0/3404124406 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d8c003bf0 con 0x7f2d9c1088a0 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.162+0000 7f2da08c6700 1 -- 172.21.15.98:0/3404124406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d9c106850 con 0x7f2d9c1088a0 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.162+0000 7f2d8b7fe700 1 -- 172.21.15.98:0/3404124406 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8c01a910 con 0x7f2d9c1088a0 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.162+0000 7f2da08c6700 1 -- 172.21.15.98:0/3404124406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d9c106d10 con 0x7f2d9c1088a0 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.162+0000 7f2d8b7fe700 1 -- 172.21.15.98:0/3404124406 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d8c024930 con 0x7f2d9c1088a0 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.163+0000 7f2d8b7fe700 1 -- 172.21.15.98:0/3404124406 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f2d8c022020 con 0x7f2d9c1088a0 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.163+0000 7f2d8b7fe700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f2d84039110 0x7f2d8403b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.163+0000 7f2d8b7fe700 1 -- 172.21.15.98:0/3404124406 --> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2d8403bce0 con 0x7f2d84039110 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.163+0000 7f2d8b7fe700 1 -- 172.21.15.98:0/3404124406 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2d8c04d650 con 0x7f2d9c1088a0 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.163+0000 7f2d99d9b700 1 -- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f2d84039110 msgr2=0x7f2d8403b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/498605085 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.163+0000 7f2d99d9b700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f2d84039110 0x7f2d8403b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.364+0000 7f2d99d9b700 1 -- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f2d84039110 msgr2=0x7f2d8403b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/498605085 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.364+0000 7f2d99d9b700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f2d84039110 0x7f2d8403b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:41:19.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.765+0000 7f2d99d9b700 1 -- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f2d84039110 msgr2=0x7f2d8403b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/498605085 2024-05-09T08:41:19.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:13.765+0000 7f2d99d9b700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f2d84039110 0x7f2d8403b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T08:41:19.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:14.566+0000 7f2d99d9b700 1 -- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f2d84039110 msgr2=0x7f2d8403b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/498605085 2024-05-09T08:41:19.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:14.566+0000 7f2d99d9b700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f2d84039110 0x7f2d8403b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T08:41:19.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:16.167+0000 7f2d99d9b700 1 -- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f2d84039110 msgr2=0x7f2d8403b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/498605085 2024-05-09T08:41:19.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:16.167+0000 7f2d99d9b700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f2d84039110 0x7f2d8403b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-09T08:41:19.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:17.099+0000 7f2d8b7fe700 1 -- 172.21.15.98:0/3404124406 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f2d8c024bb0 con 0x7f2d9c1088a0 2024-05-09T08:41:19.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:17.099+0000 7f2d8b7fe700 1 -- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f2d84039110 msgr2=0x7f2d8403b5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:41:19.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:17.099+0000 7f2d8b7fe700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f2d84039110 0x7f2d8403b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:19.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.100+0000 7f2d8b7fe700 1 -- 172.21.15.98:0/3404124406 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f2d8c04f7e0 con 0x7f2d9c1088a0 2024-05-09T08:41:19.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.101+0000 7f2d8b7fe700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2d8403cbd0 0x7f2d8403efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:19.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.101+0000 7f2d8b7fe700 1 -- 172.21.15.98:0/3404124406 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2d8403bce0 con 0x7f2d8403cbd0 2024-05-09T08:41:19.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.101+0000 7f2d99d9b700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2d8403cbd0 0x7f2d8403efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:19.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.102+0000 7f2d99d9b700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2d8403cbd0 0x7f2d8403efc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2d90003a10 tx=0x7f2d900092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.102+0000 7f2d8b7fe700 1 -- 172.21.15.98:0/3404124406 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f2d8403bce0 con 0x7f2d8403cbd0 2024-05-09T08:41:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.109+0000 7f2da08c6700 1 -- 172.21.15.98:0/3404124406 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f2d9c069450 con 0x7f2d8403cbd0 2024-05-09T08:41:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.109+0000 7f2d8b7fe700 1 -- 172.21.15.98:0/3404124406 <== mgr.14150 v2:172.21.15.98:6800/1228297308 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f2d9c069450 con 0x7f2d8403cbd0 2024-05-09T08:41:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.109+0000 7f2da08c6700 1 -- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2d8403cbd0 msgr2=0x7f2d8403efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.110+0000 7f2da08c6700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2d8403cbd0 0x7f2d8403efc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2d90003a10 tx=0x7f2d900092b0 comp rx=0 tx=0).stop 2024-05-09T08:41:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.110+0000 7f2da08c6700 1 -- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d9c1088a0 msgr2=0x7f2d9c108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.110+0000 7f2da08c6700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d9c1088a0 0x7f2d9c108120 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0117e0 tx=0x7f2d8c011c10 comp rx=0 tx=0).stop 2024-05-09T08:41:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.110+0000 7f2da08c6700 1 -- 172.21.15.98:0/3404124406 shutdown_connections 2024-05-09T08:41:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.110+0000 7f2da08c6700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2d8403cbd0 0x7f2d8403efc0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.110+0000 7f2da08c6700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d9c1088a0 0x7f2d9c108120 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.110+0000 7f2da08c6700 1 --2- 172.21.15.98:0/3404124406 >> [v2:172.21.15.98:6800/498605085,v1:172.21.15.98:6801/498605085] conn(0x7f2d84039110 0x7f2d8403b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.110+0000 7f2da08c6700 1 -- 172.21.15.98:0/3404124406 >> 172.21.15.98:0/3404124406 conn(0x7f2d9c102250 msgr2=0x7f2d9c102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.110+0000 7f2da08c6700 1 -- 172.21.15.98:0/3404124406 shutdown_connections 2024-05-09T08:41:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:18.110+0000 7f2da08c6700 1 -- 172.21.15.98:0/3404124406 wait complete. 2024-05-09T08:41:19.060 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 9 is available 2024-05-09T08:41:19.060 INFO:teuthology.orchestra.run.smithi098.stdout:Generating a dashboard self-signed certificate... 2024-05-09T08:41:19.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:19 smithi098 ceph-mon[30092]: [09/May/2024:08:41:17] ENGINE Bus STARTING 2024-05-09T08:41:19.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:19 smithi098 ceph-mon[30092]: [09/May/2024:08:41:17] ENGINE Serving on http://172.21.15.98:8765 2024-05-09T08:41:19.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:19 smithi098 ceph-mon[30092]: [09/May/2024:08:41:17] ENGINE Serving on https://172.21.15.98:7150 2024-05-09T08:41:19.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:19 smithi098 ceph-mon[30092]: [09/May/2024:08:41:17] ENGINE Bus STARTED 2024-05-09T08:41:19.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:19 smithi098 ceph-mon[30092]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-09T08:41:19.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:19 smithi098 ceph-mon[30092]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-09T08:41:19.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:19 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:20.344 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-09T08:41:20.344 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.480+0000 7f5a2f900700 1 Processor -- start 2024-05-09T08:41:20.344 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.480+0000 7f5a2f900700 1 -- start start 2024-05-09T08:41:20.345 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.480+0000 7f5a2f900700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28108860 0x7f5a2810ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:20.345 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.480+0000 7f5a2f900700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2807be90 con 0x7f5a28108860 2024-05-09T08:41:20.345 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.481+0000 7f5a2d69c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28108860 0x7f5a2810ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:20.345 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.481+0000 7f5a2d69c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28108860 0x7f5a2810ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43186/0 (socket says 172.21.15.98:43186) 2024-05-09T08:41:20.345 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.481+0000 7f5a2d69c700 1 -- 172.21.15.98:0/638918360 learned_addr learned my addr 172.21.15.98:0/638918360 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:20.346 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.481+0000 7f5a2d69c700 1 -- 172.21.15.98:0/638918360 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a2810b190 con 0x7f5a28108860 2024-05-09T08:41:20.346 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.481+0000 7f5a2d69c700 1 --2- 172.21.15.98:0/638918360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28108860 0x7f5a2810ac50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5a240098a0 tx=0x7f5a24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16613d673a880f07 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:20.346 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.481+0000 7f5a1bfff700 1 -- 172.21.15.98:0/638918360 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a24004160 con 0x7f5a28108860 2024-05-09T08:41:20.346 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.481+0000 7f5a1bfff700 1 -- 172.21.15.98:0/638918360 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a24004300 con 0x7f5a28108860 2024-05-09T08:41:20.346 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.481+0000 7f5a1bfff700 1 -- 172.21.15.98:0/638918360 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a24011b20 con 0x7f5a28108860 2024-05-09T08:41:20.346 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.482+0000 7f5a2f900700 1 -- 172.21.15.98:0/638918360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28108860 msgr2=0x7f5a2810ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:20.347 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.482+0000 7f5a2f900700 1 --2- 172.21.15.98:0/638918360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28108860 0x7f5a2810ac50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5a240098a0 tx=0x7f5a24009bb0 comp rx=0 tx=0).stop 2024-05-09T08:41:20.347 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.482+0000 7f5a2f900700 1 -- 172.21.15.98:0/638918360 shutdown_connections 2024-05-09T08:41:20.347 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.482+0000 7f5a2f900700 1 --2- 172.21.15.98:0/638918360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28108860 0x7f5a2810ac50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:20.347 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.482+0000 7f5a2f900700 1 -- 172.21.15.98:0/638918360 >> 172.21.15.98:0/638918360 conn(0x7f5a28102210 msgr2=0x7f5a28104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:20.347 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.482+0000 7f5a2f900700 1 -- 172.21.15.98:0/638918360 shutdown_connections 2024-05-09T08:41:20.347 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.482+0000 7f5a2f900700 1 -- 172.21.15.98:0/638918360 wait complete. 2024-05-09T08:41:20.347 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.483+0000 7f5a2f900700 1 Processor -- start 2024-05-09T08:41:20.347 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.483+0000 7f5a2f900700 1 -- start start 2024-05-09T08:41:20.347 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.483+0000 7f5a2f900700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28108860 0x7f5a281080e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:20.347 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.483+0000 7f5a2f900700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2810c1f0 con 0x7f5a28108860 2024-05-09T08:41:20.347 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.483+0000 7f5a2d69c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28108860 0x7f5a281080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.483+0000 7f5a2d69c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28108860 0x7f5a281080e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43198/0 (socket says 172.21.15.98:43198) 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.483+0000 7f5a2d69c700 1 -- 172.21.15.98:0/1564815916 learned_addr learned my addr 172.21.15.98:0/1564815916 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.483+0000 7f5a2d69c700 1 -- 172.21.15.98:0/1564815916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a28108620 con 0x7f5a28108860 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.484+0000 7f5a2d69c700 1 --2- 172.21.15.98:0/1564815916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28108860 0x7f5a281080e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5a240117e0 tx=0x7f5a24011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.484+0000 7f5a1a7fc700 1 -- 172.21.15.98:0/1564815916 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a24003bf0 con 0x7f5a28108860 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.484+0000 7f5a2f900700 1 -- 172.21.15.98:0/1564815916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a28106810 con 0x7f5a28108860 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.484+0000 7f5a1a7fc700 1 -- 172.21.15.98:0/1564815916 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a2401a910 con 0x7f5a28108860 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.484+0000 7f5a1a7fc700 1 -- 172.21.15.98:0/1564815916 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a24024970 con 0x7f5a28108860 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.484+0000 7f5a2f900700 1 -- 172.21.15.98:0/1564815916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a28106cd0 con 0x7f5a28108860 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.485+0000 7f5a2f900700 1 -- 172.21.15.98:0/1564815916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a28063980 con 0x7f5a28108860 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.485+0000 7f5a1a7fc700 1 -- 172.21.15.98:0/1564815916 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f5a24022020 con 0x7f5a28108860 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.485+0000 7f5a1a7fc700 1 --2- 172.21.15.98:0/1564815916 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5a14038f40 0x7f5a1403b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.485+0000 7f5a1a7fc700 1 -- 172.21.15.98:0/1564815916 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5a2404d330 con 0x7f5a28108860 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.485+0000 7f5a2ce9b700 1 --2- 172.21.15.98:0/1564815916 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5a14038f40 0x7f5a1403b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:20.348 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.486+0000 7f5a2ce9b700 1 --2- 172.21.15.98:0/1564815916 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5a14038f40 0x7f5a1403b400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c006fd0 tx=0x7f5a1c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.490+0000 7f5a1a7fc700 1 -- 172.21.15.98:0/1564815916 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5a24011e20 con 0x7f5a28108860 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.510+0000 7f5a1a7fc700 1 -- 172.21.15.98:0/1564815916 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f5a2402d910 con 0x7f5a28108860 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.700+0000 7f5a2f900700 1 -- 172.21.15.98:0/1564815916 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f5a280008d0 con 0x7f5a14038f40 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.766+0000 7f5a1a7fc700 1 -- 172.21.15.98:0/1564815916 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f5a280008d0 con 0x7f5a14038f40 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.768+0000 7f5a2f900700 1 -- 172.21.15.98:0/1564815916 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5a14038f40 msgr2=0x7f5a1403b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.768+0000 7f5a2f900700 1 --2- 172.21.15.98:0/1564815916 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5a14038f40 0x7f5a1403b400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c006fd0 tx=0x7f5a1c006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.768+0000 7f5a2f900700 1 -- 172.21.15.98:0/1564815916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28108860 msgr2=0x7f5a281080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.768+0000 7f5a2f900700 1 --2- 172.21.15.98:0/1564815916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28108860 0x7f5a281080e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5a240117e0 tx=0x7f5a24011c10 comp rx=0 tx=0).stop 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.768+0000 7f5a2f900700 1 -- 172.21.15.98:0/1564815916 shutdown_connections 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.768+0000 7f5a2f900700 1 --2- 172.21.15.98:0/1564815916 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5a14038f40 0x7f5a1403b400 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.768+0000 7f5a2f900700 1 --2- 172.21.15.98:0/1564815916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28108860 0x7f5a281080e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.768+0000 7f5a2f900700 1 -- 172.21.15.98:0/1564815916 >> 172.21.15.98:0/1564815916 conn(0x7f5a28102210 msgr2=0x7f5a28102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.768+0000 7f5a2f900700 1 -- 172.21.15.98:0/1564815916 shutdown_connections 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:19.769+0000 7f5a2f900700 1 -- 172.21.15.98:0/1564815916 wait complete. 2024-05-09T08:41:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial admin user... 2024-05-09T08:41:20.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:20 smithi098 ceph-mon[30092]: mgrmap e12: x(active, since 2s) 2024-05-09T08:41:20.590 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:20 smithi098 ceph-mon[30092]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:41:20.590 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:20 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:20.590 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:20 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:20.590 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:20 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:21.846 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$J8i9HppwCzjxcj5AEdvPCeJ6OBfJO6Q1eAsRfGjJSYWzXhwftGP9y", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1715244081, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-09T08:41:21.846 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.787+0000 7f880f4c9700 1 Processor -- start 2024-05-09T08:41:21.846 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.787+0000 7f880f4c9700 1 -- start start 2024-05-09T08:41:21.846 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.787+0000 7f880f4c9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8808108260 0x7f8808108680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:21.846 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.787+0000 7f880f4c9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8808108c50 con 0x7f8808108260 2024-05-09T08:41:21.846 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.787+0000 7f880d265700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8808108260 0x7f8808108680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.787+0000 7f880d265700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8808108260 0x7f8808108680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43208/0 (socket says 172.21.15.98:43208) 2024-05-09T08:41:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.787+0000 7f880d265700 1 -- 172.21.15.98:0/1904518476 learned_addr learned my addr 172.21.15.98:0/1904518476 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.788+0000 7f880d265700 1 -- 172.21.15.98:0/1904518476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8808106ec0 con 0x7f8808108260 2024-05-09T08:41:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.788+0000 7f880d265700 1 --2- 172.21.15.98:0/1904518476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8808108260 0x7f8808108680 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f88040098a0 tx=0x7f8804009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a937c54c742e37d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.788+0000 7f87fffff700 1 -- 172.21.15.98:0/1904518476 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8804004160 con 0x7f8808108260 2024-05-09T08:41:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.788+0000 7f87fffff700 1 -- 172.21.15.98:0/1904518476 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8804004300 con 0x7f8808108260 2024-05-09T08:41:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.788+0000 7f87fffff700 1 -- 172.21.15.98:0/1904518476 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8804011b20 con 0x7f8808108260 2024-05-09T08:41:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.789+0000 7f880f4c9700 1 -- 172.21.15.98:0/1904518476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8808108260 msgr2=0x7f8808108680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.789+0000 7f880f4c9700 1 --2- 172.21.15.98:0/1904518476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8808108260 0x7f8808108680 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f88040098a0 tx=0x7f8804009bb0 comp rx=0 tx=0).stop 2024-05-09T08:41:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.789+0000 7f880f4c9700 1 -- 172.21.15.98:0/1904518476 shutdown_connections 2024-05-09T08:41:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.789+0000 7f880f4c9700 1 --2- 172.21.15.98:0/1904518476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8808108260 0x7f8808108680 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.789+0000 7f880f4c9700 1 -- 172.21.15.98:0/1904518476 >> 172.21.15.98:0/1904518476 conn(0x7f88081019f0 msgr2=0x7f8808103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.789+0000 7f880f4c9700 1 -- 172.21.15.98:0/1904518476 shutdown_connections 2024-05-09T08:41:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.789+0000 7f880f4c9700 1 -- 172.21.15.98:0/1904518476 wait complete. 2024-05-09T08:41:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.790+0000 7f880f4c9700 1 Processor -- start 2024-05-09T08:41:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.790+0000 7f880f4c9700 1 -- start start 2024-05-09T08:41:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.790+0000 7f880f4c9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8808108260 0x7f880819bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.790+0000 7f880f4c9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f880810b280 con 0x7f8808108260 2024-05-09T08:41:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.790+0000 7f880d265700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8808108260 0x7f880819bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.790+0000 7f880d265700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8808108260 0x7f880819bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43222/0 (socket says 172.21.15.98:43222) 2024-05-09T08:41:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.790+0000 7f880d265700 1 -- 172.21.15.98:0/275566041 learned_addr learned my addr 172.21.15.98:0/275566041 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.790+0000 7f880d265700 1 -- 172.21.15.98:0/275566041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880819c130 con 0x7f8808108260 2024-05-09T08:41:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.790+0000 7f880d265700 1 --2- 172.21.15.98:0/275566041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8808108260 0x7f880819bbf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f88040117e0 tx=0x7f8804011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.791+0000 7f87fe7fc700 1 -- 172.21.15.98:0/275566041 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8804003bf0 con 0x7f8808108260 2024-05-09T08:41:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.791+0000 7f87fe7fc700 1 -- 172.21.15.98:0/275566041 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f880401a910 con 0x7f8808108260 2024-05-09T08:41:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.791+0000 7f880f4c9700 1 -- 172.21.15.98:0/275566041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f880819c3c0 con 0x7f8808108260 2024-05-09T08:41:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.791+0000 7f87fe7fc700 1 -- 172.21.15.98:0/275566041 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8804024970 con 0x7f8808108260 2024-05-09T08:41:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.791+0000 7f880f4c9700 1 -- 172.21.15.98:0/275566041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f880819c820 con 0x7f8808108260 2024-05-09T08:41:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.792+0000 7f87fe7fc700 1 -- 172.21.15.98:0/275566041 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f8804022020 con 0x7f8808108260 2024-05-09T08:41:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.792+0000 7f880f4c9700 1 -- 172.21.15.98:0/275566041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8808063980 con 0x7f8808108260 2024-05-09T08:41:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.792+0000 7f87fe7fc700 1 --2- 172.21.15.98:0/275566041 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f87f4039170 0x7f87f403b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.792+0000 7f87fe7fc700 1 -- 172.21.15.98:0/275566041 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f880404d4c0 con 0x7f8808108260 2024-05-09T08:41:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.792+0000 7f880ca64700 1 --2- 172.21.15.98:0/275566041 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f87f4039170 0x7f87f403b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.793+0000 7f880ca64700 1 --2- 172.21.15.98:0/275566041 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f87f4039170 0x7f87f403b630 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f87f8006fd0 tx=0x7f87f8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:20.797+0000 7f87fe7fc700 1 -- 172.21.15.98:0/275566041 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8804011e20 con 0x7f8808108260 2024-05-09T08:41:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:21.017+0000 7f880f4c9700 1 -- 172.21.15.98:0/275566041 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f88081956b0 con 0x7f87f4039170 2024-05-09T08:41:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:21.240+0000 7f87fe7fc700 1 -- 172.21.15.98:0/275566041 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f88081956b0 con 0x7f87f4039170 2024-05-09T08:41:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:21.242+0000 7f880f4c9700 1 -- 172.21.15.98:0/275566041 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f87f4039170 msgr2=0x7f87f403b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:21.242+0000 7f880f4c9700 1 --2- 172.21.15.98:0/275566041 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f87f4039170 0x7f87f403b630 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f87f8006fd0 tx=0x7f87f8006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:21.242+0000 7f880f4c9700 1 -- 172.21.15.98:0/275566041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8808108260 msgr2=0x7f880819bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:21.242+0000 7f880f4c9700 1 --2- 172.21.15.98:0/275566041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8808108260 0x7f880819bbf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f88040117e0 tx=0x7f8804011c10 comp rx=0 tx=0).stop 2024-05-09T08:41:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:21.242+0000 7f880f4c9700 1 -- 172.21.15.98:0/275566041 shutdown_connections 2024-05-09T08:41:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:21.242+0000 7f880f4c9700 1 --2- 172.21.15.98:0/275566041 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f87f4039170 0x7f87f403b630 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:21.242+0000 7f880f4c9700 1 --2- 172.21.15.98:0/275566041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8808108260 0x7f880819bbf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:21.242+0000 7f880f4c9700 1 -- 172.21.15.98:0/275566041 >> 172.21.15.98:0/275566041 conn(0x7f88081019f0 msgr2=0x7f88081927f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:21.242+0000 7f880f4c9700 1 -- 172.21.15.98:0/275566041 shutdown_connections 2024-05-09T08:41:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:21.242+0000 7f880f4c9700 1 -- 172.21.15.98:0/275566041 wait complete. 2024-05-09T08:41:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:Fetching dashboard port number... 2024-05-09T08:41:22.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:22 smithi098 ceph-mon[30092]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:41:22.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:23.364 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 8443 2024-05-09T08:41:23.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.254+0000 7ff2aed3e700 1 Processor -- start 2024-05-09T08:41:23.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.254+0000 7ff2aed3e700 1 -- start start 2024-05-09T08:41:23.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.255+0000 7ff2aed3e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a8108860 0x7ff2a810ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:23.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.255+0000 7ff2aed3e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a807be90 con 0x7ff2a8108860 2024-05-09T08:41:23.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.255+0000 7ff2acada700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a8108860 0x7ff2a810ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:23.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.255+0000 7ff2acada700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a8108860 0x7ff2a810ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43230/0 (socket says 172.21.15.98:43230) 2024-05-09T08:41:23.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.255+0000 7ff2acada700 1 -- 172.21.15.98:0/1874917436 learned_addr learned my addr 172.21.15.98:0/1874917436 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:23.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.255+0000 7ff2acada700 1 -- 172.21.15.98:0/1874917436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a810b190 con 0x7ff2a8108860 2024-05-09T08:41:23.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.255+0000 7ff2acada700 1 --2- 172.21.15.98:0/1874917436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a8108860 0x7ff2a810ac50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff2a40098a0 tx=0x7ff2a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c6b2ebc300e26c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:23.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.255+0000 7ff29f7fe700 1 -- 172.21.15.98:0/1874917436 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a4004160 con 0x7ff2a8108860 2024-05-09T08:41:23.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.256+0000 7ff29f7fe700 1 -- 172.21.15.98:0/1874917436 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2a4004300 con 0x7ff2a8108860 2024-05-09T08:41:23.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.256+0000 7ff29f7fe700 1 -- 172.21.15.98:0/1874917436 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a4011b20 con 0x7ff2a8108860 2024-05-09T08:41:23.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.256+0000 7ff2aed3e700 1 -- 172.21.15.98:0/1874917436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a8108860 msgr2=0x7ff2a810ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:23.365 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.256+0000 7ff2aed3e700 1 --2- 172.21.15.98:0/1874917436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a8108860 0x7ff2a810ac50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff2a40098a0 tx=0x7ff2a4009bb0 comp rx=0 tx=0).stop 2024-05-09T08:41:23.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.256+0000 7ff2aed3e700 1 -- 172.21.15.98:0/1874917436 shutdown_connections 2024-05-09T08:41:23.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.256+0000 7ff2aed3e700 1 --2- 172.21.15.98:0/1874917436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a8108860 0x7ff2a810ac50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:23.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.256+0000 7ff2aed3e700 1 -- 172.21.15.98:0/1874917436 >> 172.21.15.98:0/1874917436 conn(0x7ff2a8102210 msgr2=0x7ff2a8104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:23.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.257+0000 7ff2aed3e700 1 -- 172.21.15.98:0/1874917436 shutdown_connections 2024-05-09T08:41:23.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.257+0000 7ff2aed3e700 1 -- 172.21.15.98:0/1874917436 wait complete. 2024-05-09T08:41:23.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.257+0000 7ff2aed3e700 1 Processor -- start 2024-05-09T08:41:23.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.257+0000 7ff2aed3e700 1 -- start start 2024-05-09T08:41:23.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.258+0000 7ff2aed3e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a8108860 0x7ff2a819bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:23.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.258+0000 7ff2aed3e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a810c1f0 con 0x7ff2a8108860 2024-05-09T08:41:23.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.258+0000 7ff2acada700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a8108860 0x7ff2a819bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:23.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.258+0000 7ff2acada700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a8108860 0x7ff2a819bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43238/0 (socket says 172.21.15.98:43238) 2024-05-09T08:41:23.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.258+0000 7ff2acada700 1 -- 172.21.15.98:0/2112663833 learned_addr learned my addr 172.21.15.98:0/2112663833 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:23.366 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.258+0000 7ff2acada700 1 -- 172.21.15.98:0/2112663833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a819c250 con 0x7ff2a8108860 2024-05-09T08:41:23.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.258+0000 7ff2acada700 1 --2- 172.21.15.98:0/2112663833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a8108860 0x7ff2a819bd10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff2a40117e0 tx=0x7ff2a4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:23.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.258+0000 7ff29dffb700 1 -- 172.21.15.98:0/2112663833 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a4003bf0 con 0x7ff2a8108860 2024-05-09T08:41:23.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.258+0000 7ff29dffb700 1 -- 172.21.15.98:0/2112663833 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2a401a910 con 0x7ff2a8108860 2024-05-09T08:41:23.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.258+0000 7ff2aed3e700 1 -- 172.21.15.98:0/2112663833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2a819c4e0 con 0x7ff2a8108860 2024-05-09T08:41:23.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.258+0000 7ff29dffb700 1 -- 172.21.15.98:0/2112663833 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a4024970 con 0x7ff2a8108860 2024-05-09T08:41:23.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.259+0000 7ff2aed3e700 1 -- 172.21.15.98:0/2112663833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2a819c940 con 0x7ff2a8108860 2024-05-09T08:41:23.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.259+0000 7ff29dffb700 1 -- 172.21.15.98:0/2112663833 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7ff2a4022020 con 0x7ff2a8108860 2024-05-09T08:41:23.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.260+0000 7ff29dffb700 1 --2- 172.21.15.98:0/2112663833 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff290039050 0x7ff29003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:23.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.260+0000 7ff29dffb700 1 -- 172.21.15.98:0/2112663833 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff2a404d4e0 con 0x7ff2a8108860 2024-05-09T08:41:23.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.260+0000 7ff29ffff700 1 --2- 172.21.15.98:0/2112663833 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff290039050 0x7ff29003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:23.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.260+0000 7ff2aed3e700 1 -- 172.21.15.98:0/2112663833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2a8063980 con 0x7ff2a8108860 2024-05-09T08:41:23.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.260+0000 7ff29ffff700 1 --2- 172.21.15.98:0/2112663833 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff290039050 0x7ff29003b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff298006fd0 tx=0x7ff298006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:23.367 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.265+0000 7ff29dffb700 1 -- 172.21.15.98:0/2112663833 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff2a4011e20 con 0x7ff2a8108860 2024-05-09T08:41:23.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.478+0000 7ff2aed3e700 1 -- 172.21.15.98:0/2112663833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7ff2a819f1c0 con 0x7ff2a8108860 2024-05-09T08:41:23.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.479+0000 7ff29dffb700 1 -- 172.21.15.98:0/2112663833 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7ff2a402c350 con 0x7ff2a8108860 2024-05-09T08:41:23.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.481+0000 7ff2aed3e700 1 -- 172.21.15.98:0/2112663833 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff290039050 msgr2=0x7ff29003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:23.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.481+0000 7ff2aed3e700 1 --2- 172.21.15.98:0/2112663833 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff290039050 0x7ff29003b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff298006fd0 tx=0x7ff298006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:23.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.481+0000 7ff2aed3e700 1 -- 172.21.15.98:0/2112663833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a8108860 msgr2=0x7ff2a819bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:23.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.481+0000 7ff2aed3e700 1 --2- 172.21.15.98:0/2112663833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a8108860 0x7ff2a819bd10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff2a40117e0 tx=0x7ff2a4011c10 comp rx=0 tx=0).stop 2024-05-09T08:41:23.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.481+0000 7ff2aed3e700 1 -- 172.21.15.98:0/2112663833 shutdown_connections 2024-05-09T08:41:23.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.481+0000 7ff2aed3e700 1 --2- 172.21.15.98:0/2112663833 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff290039050 0x7ff29003b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:23.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.481+0000 7ff2aed3e700 1 --2- 172.21.15.98:0/2112663833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a8108860 0x7ff2a819bd10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:23.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.481+0000 7ff2aed3e700 1 -- 172.21.15.98:0/2112663833 >> 172.21.15.98:0/2112663833 conn(0x7ff2a8102210 msgr2=0x7ff2a8102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:23.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.481+0000 7ff2aed3e700 1 -- 172.21.15.98:0/2112663833 shutdown_connections 2024-05-09T08:41:23.368 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:22.481+0000 7ff2aed3e700 1 -- 172.21.15.98:0/2112663833 wait complete. 2024-05-09T08:41:23.372 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T08:41:23.372 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-05-09T08:41:23.381 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T08:41:23.381 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-05-09T08:41:23.381 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-05-09T08:41:23.381 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-09T08:41:23.383 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph Dashboard is now available at: 2024-05-09T08:41:23.383 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:41:23.383 INFO:teuthology.orchestra.run.smithi098.stdout: URL: https://smithi098.front.sepia.ceph.com:8443/ 2024-05-09T08:41:23.383 INFO:teuthology.orchestra.run.smithi098.stdout: User: admin 2024-05-09T08:41:23.383 INFO:teuthology.orchestra.run.smithi098.stdout: Password: tkctwgemis 2024-05-09T08:41:23.383 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:41:23.384 INFO:teuthology.orchestra.run.smithi098.stdout:Saving cluster configuration to /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config directory 2024-05-09T08:41:23.384 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling autotune for osd_memory_target 2024-05-09T08:41:23.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:23 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2112663833' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-09T08:41:24.518 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:24 smithi098 ceph-mon[30092]: mgrmap e13: x(active, since 6s) 2024-05-09T08:41:24.638 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.796+0000 7fcce322f700 1 Processor -- start 2024-05-09T08:41:24.638 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.796+0000 7fcce322f700 1 -- start start 2024-05-09T08:41:24.638 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.796+0000 7fcce322f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccdc108820 0x7fccdc10ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:24.638 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.796+0000 7fcce322f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fccdc07bed0 con 0x7fccdc108820 2024-05-09T08:41:24.638 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.797+0000 7fcce0fcb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccdc108820 0x7fccdc10ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:24.639 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.797+0000 7fcce0fcb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccdc108820 0x7fccdc10ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43244/0 (socket says 172.21.15.98:43244) 2024-05-09T08:41:24.639 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.797+0000 7fcce0fcb700 1 -- 172.21.15.98:0/874336363 learned_addr learned my addr 172.21.15.98:0/874336363 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:24.639 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.797+0000 7fcce0fcb700 1 -- 172.21.15.98:0/874336363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccdc10b150 con 0x7fccdc108820 2024-05-09T08:41:24.639 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.797+0000 7fcce0fcb700 1 --2- 172.21.15.98:0/874336363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccdc108820 0x7fccdc10ac10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fccd00098a0 tx=0x7fccd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20f9bc8f53ead88 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:24.639 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.797+0000 7fccdb7fe700 1 -- 172.21.15.98:0/874336363 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccd0004160 con 0x7fccdc108820 2024-05-09T08:41:24.639 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.797+0000 7fccdb7fe700 1 -- 172.21.15.98:0/874336363 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccd0004300 con 0x7fccdc108820 2024-05-09T08:41:24.639 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.797+0000 7fccdb7fe700 1 -- 172.21.15.98:0/874336363 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccd0011b20 con 0x7fccdc108820 2024-05-09T08:41:24.639 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.798+0000 7fcce322f700 1 -- 172.21.15.98:0/874336363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccdc108820 msgr2=0x7fccdc10ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:24.639 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.798+0000 7fcce322f700 1 --2- 172.21.15.98:0/874336363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccdc108820 0x7fccdc10ac10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fccd00098a0 tx=0x7fccd0009bb0 comp rx=0 tx=0).stop 2024-05-09T08:41:24.639 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.798+0000 7fcce322f700 1 -- 172.21.15.98:0/874336363 shutdown_connections 2024-05-09T08:41:24.639 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.798+0000 7fcce322f700 1 --2- 172.21.15.98:0/874336363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccdc108820 0x7fccdc10ac10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:24.639 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.798+0000 7fcce322f700 1 -- 172.21.15.98:0/874336363 >> 172.21.15.98:0/874336363 conn(0x7fccdc102210 msgr2=0x7fccdc104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:24.639 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.798+0000 7fcce322f700 1 -- 172.21.15.98:0/874336363 shutdown_connections 2024-05-09T08:41:24.639 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.798+0000 7fcce322f700 1 -- 172.21.15.98:0/874336363 wait complete. 2024-05-09T08:41:24.640 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.799+0000 7fcce322f700 1 Processor -- start 2024-05-09T08:41:24.640 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.799+0000 7fcce322f700 1 -- start start 2024-05-09T08:41:24.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.799+0000 7fcce322f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccdc108820 0x7fccdc19bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:24.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.799+0000 7fcce322f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fccdc10c1c0 con 0x7fccdc108820 2024-05-09T08:41:24.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.799+0000 7fcce0fcb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccdc108820 0x7fccdc19bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:24.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.799+0000 7fcce0fcb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccdc108820 0x7fccdc19bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43260/0 (socket says 172.21.15.98:43260) 2024-05-09T08:41:24.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.799+0000 7fcce0fcb700 1 -- 172.21.15.98:0/963316807 learned_addr learned my addr 172.21.15.98:0/963316807 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:24.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.799+0000 7fcce0fcb700 1 -- 172.21.15.98:0/963316807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccdc19c260 con 0x7fccdc108820 2024-05-09T08:41:24.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.799+0000 7fcce0fcb700 1 --2- 172.21.15.98:0/963316807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccdc108820 0x7fccdc19bd20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fccd00117e0 tx=0x7fccd0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:24.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.799+0000 7fccd9ffb700 1 -- 172.21.15.98:0/963316807 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccd0003ba0 con 0x7fccdc108820 2024-05-09T08:41:24.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.799+0000 7fccd9ffb700 1 -- 172.21.15.98:0/963316807 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccd001a910 con 0x7fccdc108820 2024-05-09T08:41:24.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.799+0000 7fcce322f700 1 -- 172.21.15.98:0/963316807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccdc19c4f0 con 0x7fccdc108820 2024-05-09T08:41:24.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.799+0000 7fccd9ffb700 1 -- 172.21.15.98:0/963316807 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccd00248f0 con 0x7fccdc108820 2024-05-09T08:41:24.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.800+0000 7fcce322f700 1 -- 172.21.15.98:0/963316807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccdc19c950 con 0x7fccdc108820 2024-05-09T08:41:24.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.800+0000 7fccd9ffb700 1 -- 172.21.15.98:0/963316807 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fccd0022020 con 0x7fccdc108820 2024-05-09T08:41:24.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.801+0000 7fcce322f700 1 -- 172.21.15.98:0/963316807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccdc063980 con 0x7fccdc108820 2024-05-09T08:41:24.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.801+0000 7fccd9ffb700 1 --2- 172.21.15.98:0/963316807 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcccc0391a0 0x7fcccc03b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:24.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.801+0000 7fccd9ffb700 1 -- 172.21.15.98:0/963316807 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fccd004d890 con 0x7fccdc108820 2024-05-09T08:41:24.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.801+0000 7fccdbfff700 1 --2- 172.21.15.98:0/963316807 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcccc0391a0 0x7fcccc03b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:24.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.801+0000 7fccdbfff700 1 --2- 172.21.15.98:0/963316807 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcccc0391a0 0x7fcccc03b660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fccc8006fd0 tx=0x7fccc8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:24.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:23.806+0000 7fccd9ffb700 1 -- 172.21.15.98:0/963316807 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fccd001ab30 con 0x7fccdc108820 2024-05-09T08:41:24.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:24.000+0000 7fcce322f700 1 -- 172.21.15.98:0/963316807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fccdc19f3b0 con 0x7fccdc108820 2024-05-09T08:41:24.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:24.000+0000 7fccd9ffb700 1 -- 172.21.15.98:0/963316807 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fccd000b5a0 con 0x7fccdc108820 2024-05-09T08:41:24.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:24.002+0000 7fcce322f700 1 -- 172.21.15.98:0/963316807 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcccc0391a0 msgr2=0x7fcccc03b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:24.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:24.002+0000 7fcce322f700 1 --2- 172.21.15.98:0/963316807 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcccc0391a0 0x7fcccc03b660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fccc8006fd0 tx=0x7fccc8006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:24.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:24.002+0000 7fcce322f700 1 -- 172.21.15.98:0/963316807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccdc108820 msgr2=0x7fccdc19bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:24.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:24.002+0000 7fcce322f700 1 --2- 172.21.15.98:0/963316807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccdc108820 0x7fccdc19bd20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fccd00117e0 tx=0x7fccd0011ba0 comp rx=0 tx=0).stop 2024-05-09T08:41:24.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:24.003+0000 7fcce322f700 1 -- 172.21.15.98:0/963316807 shutdown_connections 2024-05-09T08:41:24.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:24.003+0000 7fcce322f700 1 --2- 172.21.15.98:0/963316807 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcccc0391a0 0x7fcccc03b660 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:24.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:24.003+0000 7fcce322f700 1 --2- 172.21.15.98:0/963316807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccdc108820 0x7fccdc19bd20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:24.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:24.003+0000 7fcce322f700 1 -- 172.21.15.98:0/963316807 >> 172.21.15.98:0/963316807 conn(0x7fccdc102210 msgr2=0x7fccdc102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:24.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:24.003+0000 7fcce322f700 1 -- 172.21.15.98:0/963316807 shutdown_connections 2024-05-09T08:41:24.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:24.003+0000 7fcce322f700 1 -- 172.21.15.98:0/963316807 wait complete. 2024-05-09T08:41:25.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.068+0000 7f03df8c4700 1 Processor -- start 2024-05-09T08:41:25.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.069+0000 7f03df8c4700 1 -- start start 2024-05-09T08:41:25.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.069+0000 7f03df8c4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03d81066c0 0x7f03d8106ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:25.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.069+0000 7f03df8c4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03d807be50 con 0x7f03d81066c0 2024-05-09T08:41:25.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.069+0000 7f03dd660700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03d81066c0 0x7f03d8106ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:25.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.069+0000 7f03dd660700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03d81066c0 0x7f03d8106ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43274/0 (socket says 172.21.15.98:43274) 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.069+0000 7f03dd660700 1 -- 172.21.15.98:0/3689972065 learned_addr learned my addr 172.21.15.98:0/3689972065 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.069+0000 7f03dd660700 1 -- 172.21.15.98:0/3689972065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d8107020 con 0x7f03d81066c0 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.069+0000 7f03dd660700 1 --2- 172.21.15.98:0/3689972065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03d81066c0 0x7f03d8106ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f03cc0098a0 tx=0x7f03cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d984f715eb06818 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.070+0000 7f03cbfff700 1 -- 172.21.15.98:0/3689972065 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03cc004160 con 0x7f03d81066c0 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.070+0000 7f03cbfff700 1 -- 172.21.15.98:0/3689972065 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03cc004300 con 0x7f03d81066c0 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.070+0000 7f03cbfff700 1 -- 172.21.15.98:0/3689972065 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03cc011b20 con 0x7f03d81066c0 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.070+0000 7f03df8c4700 1 -- 172.21.15.98:0/3689972065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03d81066c0 msgr2=0x7f03d8106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.070+0000 7f03df8c4700 1 --2- 172.21.15.98:0/3689972065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03d81066c0 0x7f03d8106ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f03cc0098a0 tx=0x7f03cc009bb0 comp rx=0 tx=0).stop 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.070+0000 7f03df8c4700 1 -- 172.21.15.98:0/3689972065 shutdown_connections 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.070+0000 7f03df8c4700 1 --2- 172.21.15.98:0/3689972065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03d81066c0 0x7f03d8106ae0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.070+0000 7f03df8c4700 1 -- 172.21.15.98:0/3689972065 >> 172.21.15.98:0/3689972065 conn(0x7f03d8102280 msgr2=0x7f03d81046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.070+0000 7f03df8c4700 1 -- 172.21.15.98:0/3689972065 shutdown_connections 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.070+0000 7f03df8c4700 1 -- 172.21.15.98:0/3689972065 wait complete. 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.071+0000 7f03df8c4700 1 Processor -- start 2024-05-09T08:41:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.071+0000 7f03df8c4700 1 -- start start 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.071+0000 7f03df8c4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03d81066c0 0x7f03d819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.071+0000 7f03df8c4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03d81080f0 con 0x7f03d81066c0 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.071+0000 7f03dd660700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03d81066c0 0x7f03d819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.071+0000 7f03dd660700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03d81066c0 0x7f03d819de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43290/0 (socket says 172.21.15.98:43290) 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.071+0000 7f03dd660700 1 -- 172.21.15.98:0/1449156879 learned_addr learned my addr 172.21.15.98:0/1449156879 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.071+0000 7f03dd660700 1 -- 172.21.15.98:0/1449156879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d819e380 con 0x7f03d81066c0 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.072+0000 7f03dd660700 1 --2- 172.21.15.98:0/1449156879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03d81066c0 0x7f03d819de40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f03cc0117e0 tx=0x7f03cc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.072+0000 7f03ca7fc700 1 -- 172.21.15.98:0/1449156879 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03cc003bf0 con 0x7f03d81066c0 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.072+0000 7f03ca7fc700 1 -- 172.21.15.98:0/1449156879 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03cc01a910 con 0x7f03d81066c0 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.072+0000 7f03df8c4700 1 -- 172.21.15.98:0/1449156879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03d819e610 con 0x7f03d81066c0 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.072+0000 7f03ca7fc700 1 -- 172.21.15.98:0/1449156879 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03cc024970 con 0x7f03d81066c0 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.072+0000 7f03df8c4700 1 -- 172.21.15.98:0/1449156879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03d819ea70 con 0x7f03d81066c0 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.073+0000 7f03ca7fc700 1 -- 172.21.15.98:0/1449156879 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f03cc022020 con 0x7f03d81066c0 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.073+0000 7f03df8c4700 1 -- 172.21.15.98:0/1449156879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03d8063980 con 0x7f03d81066c0 2024-05-09T08:41:25.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.073+0000 7f03ca7fc700 1 --2- 172.21.15.98:0/1449156879 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f03c40391b0 0x7f03c403b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.073+0000 7f03ca7fc700 1 -- 172.21.15.98:0/1449156879 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f03cc04d5b0 con 0x7f03d81066c0 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.073+0000 7f03dce5f700 1 --2- 172.21.15.98:0/1449156879 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f03c40391b0 0x7f03c403b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.073+0000 7f03dce5f700 1 --2- 172.21.15.98:0/1449156879 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f03c40391b0 0x7f03c403b670 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f03d4006fd0 tx=0x7f03d4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.078+0000 7f03ca7fc700 1 -- 172.21.15.98:0/1449156879 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f03cc011e20 con 0x7f03d81066c0 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.386+0000 7f03df8c4700 1 -- 172.21.15.98:0/1449156879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f03d8062f30 con 0x7f03d81066c0 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.388+0000 7f03ca7fc700 1 -- 172.21.15.98:0/1449156879 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f03cc00b3e0 con 0x7f03d81066c0 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.390+0000 7f03df8c4700 1 -- 172.21.15.98:0/1449156879 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f03c40391b0 msgr2=0x7f03c403b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.390+0000 7f03df8c4700 1 --2- 172.21.15.98:0/1449156879 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f03c40391b0 0x7f03c403b670 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f03d4006fd0 tx=0x7f03d4006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.390+0000 7f03df8c4700 1 -- 172.21.15.98:0/1449156879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03d81066c0 msgr2=0x7f03d819de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.390+0000 7f03df8c4700 1 --2- 172.21.15.98:0/1449156879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03d81066c0 0x7f03d819de40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f03cc0117e0 tx=0x7f03cc011c10 comp rx=0 tx=0).stop 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.391+0000 7f03df8c4700 1 -- 172.21.15.98:0/1449156879 shutdown_connections 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.391+0000 7f03df8c4700 1 --2- 172.21.15.98:0/1449156879 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f03c40391b0 0x7f03c403b670 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.391+0000 7f03df8c4700 1 --2- 172.21.15.98:0/1449156879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03d81066c0 0x7f03d819de40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.391+0000 7f03df8c4700 1 -- 172.21.15.98:0/1449156879 >> 172.21.15.98:0/1449156879 conn(0x7f03d8102280 msgr2=0x7f03d8194ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:25.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.391+0000 7f03df8c4700 1 -- 172.21.15.98:0/1449156879 shutdown_connections 2024-05-09T08:41:25.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-05-09T08:41:25.391+0000 7f03df8c4700 1 -- 172.21.15.98:0/1449156879 wait complete. 2024-05-09T08:41:25.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-09T08:41:25.991 INFO:teuthology.orchestra.run.smithi098.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-09T08:41:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:41:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:41:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:41:25.991 INFO:teuthology.orchestra.run.smithi098.stdout:Or, if you are only running a single cluster on this host: 2024-05-09T08:41:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:41:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-09T08:41:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:41:25.991 INFO:teuthology.orchestra.run.smithi098.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-09T08:41:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:41:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: ceph telemetry on 2024-05-09T08:41:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:41:25.992 INFO:teuthology.orchestra.run.smithi098.stdout:For more information see: 2024-05-09T08:41:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:41:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-09T08:41:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:41:25.992 INFO:teuthology.orchestra.run.smithi098.stdout:Bootstrap complete. 2024-05-09T08:41:26.026 INFO:tasks.cephadm:Fetching config... 2024-05-09T08:41:26.026 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:41:26.027 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-09T08:41:26.043 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-09T08:41:26.043 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:41:26.043 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-09T08:41:26.099 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-09T08:41:26.099 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:41:26.099 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-09T08:41:26.165 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-09T08:41:26.165 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:41:26.165 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-09T08:41:26.221 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-09T08:41:26.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM08QhTU9RCj64TMHPr8xMaD2Ggju/QS/0ZtCwyPuKPgrejfryn302vt6awYcYMVgjAQpkBkaRlPqfogVnozATLQM1oWPIg80beCjWShdY6SefSde+xGAGZouZa0SVmT4acx4cZN8o9NGFwr2pmNh+Io6hiUuiGyCq7svM9Oq+rTDBjA3zz8Qi216KQ6cu2gv49/J4OfiHbvRVfAAU5c1RGuc1fBiOefAaN0zOgmAUIKkyDBf1EfQ3/apAfpD0Rp5wfMRCTAm8iLbrAfwONyttgKsBHgiCJ82HCuoR96doEfEmBwpAqvka0fdkNHETqKjbAnQ0JcpIZX4uHmtUFurEVw3Oz0eU0Wgx53KdxYVwtRCCxKrE/etAPD7cMo2tqDObDhK1UqxOKLJ6Iuju60/jjy4OIh2w7LwtfhY1DOHwRbd4ly9c9YZJSW+tx9UTp8F0dpJTt1ibS3Y3c+QpPynSNqTDOClBAlPlRoi3xJqaCZ5yHTTwpBBYnrZpfVm2I5s= ceph-a57d9712-0ddf-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-09T08:41:26.296 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM08QhTU9RCj64TMHPr8xMaD2Ggju/QS/0ZtCwyPuKPgrejfryn302vt6awYcYMVgjAQpkBkaRlPqfogVnozATLQM1oWPIg80beCjWShdY6SefSde+xGAGZouZa0SVmT4acx4cZN8o9NGFwr2pmNh+Io6hiUuiGyCq7svM9Oq+rTDBjA3zz8Qi216KQ6cu2gv49/J4OfiHbvRVfAAU5c1RGuc1fBiOefAaN0zOgmAUIKkyDBf1EfQ3/apAfpD0Rp5wfMRCTAm8iLbrAfwONyttgKsBHgiCJ82HCuoR96doEfEmBwpAqvka0fdkNHETqKjbAnQ0JcpIZX4uHmtUFurEVw3Oz0eU0Wgx53KdxYVwtRCCxKrE/etAPD7cMo2tqDObDhK1UqxOKLJ6Iuju60/jjy4OIh2w7LwtfhY1DOHwRbd4ly9c9YZJSW+tx9UTp8F0dpJTt1ibS3Y3c+QpPynSNqTDOClBAlPlRoi3xJqaCZ5yHTTwpBBYnrZpfVm2I5s= ceph-a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:41:26.307 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM08QhTU9RCj64TMHPr8xMaD2Ggju/QS/0ZtCwyPuKPgrejfryn302vt6awYcYMVgjAQpkBkaRlPqfogVnozATLQM1oWPIg80beCjWShdY6SefSde+xGAGZouZa0SVmT4acx4cZN8o9NGFwr2pmNh+Io6hiUuiGyCq7svM9Oq+rTDBjA3zz8Qi216KQ6cu2gv49/J4OfiHbvRVfAAU5c1RGuc1fBiOefAaN0zOgmAUIKkyDBf1EfQ3/apAfpD0Rp5wfMRCTAm8iLbrAfwONyttgKsBHgiCJ82HCuoR96doEfEmBwpAqvka0fdkNHETqKjbAnQ0JcpIZX4uHmtUFurEVw3Oz0eU0Wgx53KdxYVwtRCCxKrE/etAPD7cMo2tqDObDhK1UqxOKLJ6Iuju60/jjy4OIh2w7LwtfhY1DOHwRbd4ly9c9YZJSW+tx9UTp8F0dpJTt1ibS3Y3c+QpPynSNqTDOClBAlPlRoi3xJqaCZ5yHTTwpBBYnrZpfVm2I5s= ceph-a57d9712-0ddf-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-09T08:41:26.346 INFO:teuthology.orchestra.run.smithi105.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM08QhTU9RCj64TMHPr8xMaD2Ggju/QS/0ZtCwyPuKPgrejfryn302vt6awYcYMVgjAQpkBkaRlPqfogVnozATLQM1oWPIg80beCjWShdY6SefSde+xGAGZouZa0SVmT4acx4cZN8o9NGFwr2pmNh+Io6hiUuiGyCq7svM9Oq+rTDBjA3zz8Qi216KQ6cu2gv49/J4OfiHbvRVfAAU5c1RGuc1fBiOefAaN0zOgmAUIKkyDBf1EfQ3/apAfpD0Rp5wfMRCTAm8iLbrAfwONyttgKsBHgiCJ82HCuoR96doEfEmBwpAqvka0fdkNHETqKjbAnQ0JcpIZX4uHmtUFurEVw3Oz0eU0Wgx53KdxYVwtRCCxKrE/etAPD7cMo2tqDObDhK1UqxOKLJ6Iuju60/jjy4OIh2w7LwtfhY1DOHwRbd4ly9c9YZJSW+tx9UTp8F0dpJTt1ibS3Y3c+QpPynSNqTDOClBAlPlRoi3xJqaCZ5yHTTwpBBYnrZpfVm2I5s= ceph-a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:41:26.358 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM08QhTU9RCj64TMHPr8xMaD2Ggju/QS/0ZtCwyPuKPgrejfryn302vt6awYcYMVgjAQpkBkaRlPqfogVnozATLQM1oWPIg80beCjWShdY6SefSde+xGAGZouZa0SVmT4acx4cZN8o9NGFwr2pmNh+Io6hiUuiGyCq7svM9Oq+rTDBjA3zz8Qi216KQ6cu2gv49/J4OfiHbvRVfAAU5c1RGuc1fBiOefAaN0zOgmAUIKkyDBf1EfQ3/apAfpD0Rp5wfMRCTAm8iLbrAfwONyttgKsBHgiCJ82HCuoR96doEfEmBwpAqvka0fdkNHETqKjbAnQ0JcpIZX4uHmtUFurEVw3Oz0eU0Wgx53KdxYVwtRCCxKrE/etAPD7cMo2tqDObDhK1UqxOKLJ6Iuju60/jjy4OIh2w7LwtfhY1DOHwRbd4ly9c9YZJSW+tx9UTp8F0dpJTt1ibS3Y3c+QpPynSNqTDOClBAlPlRoi3xJqaCZ5yHTTwpBBYnrZpfVm2I5s= ceph-a57d9712-0ddf-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-09T08:41:26.397 INFO:teuthology.orchestra.run.smithi142.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM08QhTU9RCj64TMHPr8xMaD2Ggju/QS/0ZtCwyPuKPgrejfryn302vt6awYcYMVgjAQpkBkaRlPqfogVnozATLQM1oWPIg80beCjWShdY6SefSde+xGAGZouZa0SVmT4acx4cZN8o9NGFwr2pmNh+Io6hiUuiGyCq7svM9Oq+rTDBjA3zz8Qi216KQ6cu2gv49/J4OfiHbvRVfAAU5c1RGuc1fBiOefAaN0zOgmAUIKkyDBf1EfQ3/apAfpD0Rp5wfMRCTAm8iLbrAfwONyttgKsBHgiCJ82HCuoR96doEfEmBwpAqvka0fdkNHETqKjbAnQ0JcpIZX4uHmtUFurEVw3Oz0eU0Wgx53KdxYVwtRCCxKrE/etAPD7cMo2tqDObDhK1UqxOKLJ6Iuju60/jjy4OIh2w7LwtfhY1DOHwRbd4ly9c9YZJSW+tx9UTp8F0dpJTt1ibS3Y3c+QpPynSNqTDOClBAlPlRoi3xJqaCZ5yHTTwpBBYnrZpfVm2I5s= ceph-a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:41:26.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-09T08:41:26.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:26 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1449156879' entity='client.admin' 2024-05-09T08:41:26.661 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:41:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.023+0000 7ff697fff700 1 -- 172.21.15.98:0/2144925825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6980762d0 msgr2=0x7ff698074730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.023+0000 7ff697fff700 1 --2- 172.21.15.98:0/2144925825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6980762d0 0x7ff698074730 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff688009910 tx=0x7ff688009c20 comp rx=0 tx=0).stop 2024-05-09T08:41:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.023+0000 7ff697fff700 1 -- 172.21.15.98:0/2144925825 shutdown_connections 2024-05-09T08:41:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.023+0000 7ff697fff700 1 --2- 172.21.15.98:0/2144925825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6980762d0 0x7ff698074730 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.023+0000 7ff697fff700 1 -- 172.21.15.98:0/2144925825 >> 172.21.15.98:0/2144925825 conn(0x7ff6980fd160 msgr2=0x7ff6980ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.023+0000 7ff697fff700 1 -- 172.21.15.98:0/2144925825 shutdown_connections 2024-05-09T08:41:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.023+0000 7ff697fff700 1 -- 172.21.15.98:0/2144925825 wait complete. 2024-05-09T08:41:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.024+0000 7ff697fff700 1 Processor -- start 2024-05-09T08:41:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.024+0000 7ff697fff700 1 -- start start 2024-05-09T08:41:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.024+0000 7ff697fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6980762d0 0x7ff698199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.024+0000 7ff697fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff698109e60 con 0x7ff6980762d0 2024-05-09T08:41:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.024+0000 7ff696ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6980762d0 0x7ff698199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.025+0000 7ff696ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6980762d0 0x7ff698199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60114/0 (socket says 172.21.15.98:60114) 2024-05-09T08:41:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.025+0000 7ff696ffd700 1 -- 172.21.15.98:0/1084689583 learned_addr learned my addr 172.21.15.98:0/1084689583 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.025+0000 7ff696ffd700 1 -- 172.21.15.98:0/1084689583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6981999b0 con 0x7ff6980762d0 2024-05-09T08:41:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.025+0000 7ff696ffd700 1 --2- 172.21.15.98:0/1084689583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6980762d0 0x7ff698199470 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff688005950 tx=0x7ff688005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.025+0000 7ff69c802700 1 -- 172.21.15.98:0/1084689583 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff688026070 con 0x7ff6980762d0 2024-05-09T08:41:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.025+0000 7ff69c802700 1 -- 172.21.15.98:0/1084689583 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff68800bc30 con 0x7ff6980762d0 2024-05-09T08:41:29.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.025+0000 7ff697fff700 1 -- 172.21.15.98:0/1084689583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff698199c40 con 0x7ff6980762d0 2024-05-09T08:41:29.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.025+0000 7ff69c802700 1 -- 172.21.15.98:0/1084689583 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff688003b30 con 0x7ff6980762d0 2024-05-09T08:41:29.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.025+0000 7ff697fff700 1 -- 172.21.15.98:0/1084689583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff69819c930 con 0x7ff6980762d0 2024-05-09T08:41:29.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.026+0000 7ff69c802700 1 -- 172.21.15.98:0/1084689583 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7ff688029070 con 0x7ff6980762d0 2024-05-09T08:41:29.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.026+0000 7ff697fff700 1 -- 172.21.15.98:0/1084689583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff69810a5e0 con 0x7ff6980762d0 2024-05-09T08:41:29.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.026+0000 7ff69c802700 1 --2- 172.21.15.98:0/1084689583 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6800390f0 0x7ff68003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:29.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.026+0000 7ff69c802700 1 -- 172.21.15.98:0/1084689583 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff68801e070 con 0x7ff6980762d0 2024-05-09T08:41:29.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.026+0000 7ff6967fc700 1 --2- 172.21.15.98:0/1084689583 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6800390f0 0x7ff68003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:29.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.027+0000 7ff6967fc700 1 --2- 172.21.15.98:0/1084689583 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6800390f0 0x7ff68003b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff68c006fd0 tx=0x7ff68c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:29.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.031+0000 7ff69c802700 1 -- 172.21.15.98:0/1084689583 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff688013070 con 0x7ff6980762d0 2024-05-09T08:41:29.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:29.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:29.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:41:29.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:29.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:41:29.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.233+0000 7ff697fff700 1 -- 172.21.15.98:0/1084689583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7ff69802dc90 con 0x7ff6980762d0 2024-05-09T08:41:29.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.235+0000 7ff69c802700 1 -- 172.21.15.98:0/1084689583 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7ff688030070 con 0x7ff6980762d0 2024-05-09T08:41:29.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.238+0000 7ff697fff700 1 -- 172.21.15.98:0/1084689583 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6800390f0 msgr2=0x7ff68003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:29.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.238+0000 7ff697fff700 1 --2- 172.21.15.98:0/1084689583 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6800390f0 0x7ff68003b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff68c006fd0 tx=0x7ff68c006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:29.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.238+0000 7ff697fff700 1 -- 172.21.15.98:0/1084689583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6980762d0 msgr2=0x7ff698199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:29.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.238+0000 7ff697fff700 1 --2- 172.21.15.98:0/1084689583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6980762d0 0x7ff698199470 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff688005950 tx=0x7ff688005030 comp rx=0 tx=0).stop 2024-05-09T08:41:29.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.238+0000 7ff697fff700 1 -- 172.21.15.98:0/1084689583 shutdown_connections 2024-05-09T08:41:29.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.238+0000 7ff697fff700 1 --2- 172.21.15.98:0/1084689583 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6800390f0 0x7ff68003b5b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.238+0000 7ff697fff700 1 --2- 172.21.15.98:0/1084689583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6980762d0 0x7ff698199470 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.238+0000 7ff697fff700 1 -- 172.21.15.98:0/1084689583 >> 172.21.15.98:0/1084689583 conn(0x7ff6980fd160 msgr2=0x7ff6980fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:29.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.238+0000 7ff697fff700 1 -- 172.21.15.98:0/1084689583 shutdown_connections 2024-05-09T08:41:29.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:29.238+0000 7ff697fff700 1 -- 172.21.15.98:0/1084689583 wait complete. 2024-05-09T08:41:29.931 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-09T08:41:29.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-09T08:41:30.484 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:30 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1084689583' entity='client.admin' 2024-05-09T08:41:30.484 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:41:30.484 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:41:32.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.120+0000 7fd9c518d700 1 -- 172.21.15.98:0/1836741951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9c0103490 msgr2=0x7fd9c01038b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:32.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.120+0000 7fd9c518d700 1 --2- 172.21.15.98:0/1836741951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9c0103490 0x7fd9c01038b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009910 tx=0x7fd9a8009c20 comp rx=0 tx=0).stop 2024-05-09T08:41:32.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.120+0000 7fd9c518d700 1 -- 172.21.15.98:0/1836741951 shutdown_connections 2024-05-09T08:41:32.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.120+0000 7fd9c518d700 1 --2- 172.21.15.98:0/1836741951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9c0103490 0x7fd9c01038b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:32.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.120+0000 7fd9c518d700 1 -- 172.21.15.98:0/1836741951 >> 172.21.15.98:0/1836741951 conn(0x7fd9c00fd370 msgr2=0x7fd9c00ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:32.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.120+0000 7fd9c518d700 1 -- 172.21.15.98:0/1836741951 shutdown_connections 2024-05-09T08:41:32.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.120+0000 7fd9c518d700 1 -- 172.21.15.98:0/1836741951 wait complete. 2024-05-09T08:41:32.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.121+0000 7fd9c518d700 1 Processor -- start 2024-05-09T08:41:32.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.121+0000 7fd9c518d700 1 -- start start 2024-05-09T08:41:32.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.121+0000 7fd9c518d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9c0103490 0x7fd9c01975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:32.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.121+0000 7fd9c518d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c0107f50 con 0x7fd9c0103490 2024-05-09T08:41:32.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.121+0000 7fd9bed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9c0103490 0x7fd9c01975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:32.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.121+0000 7fd9bed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9c0103490 0x7fd9c01975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60138/0 (socket says 172.21.15.98:60138) 2024-05-09T08:41:32.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.121+0000 7fd9bed9d700 1 -- 172.21.15.98:0/2974805334 learned_addr learned my addr 172.21.15.98:0/2974805334 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:32.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.122+0000 7fd9bed9d700 1 -- 172.21.15.98:0/2974805334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c0197b10 con 0x7fd9c0103490 2024-05-09T08:41:32.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.122+0000 7fd9bed9d700 1 --2- 172.21.15.98:0/2974805334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9c0103490 0x7fd9c01975d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8006010 tx=0x7fd9a8005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:32.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.122+0000 7fd9b7fff700 1 -- 172.21.15.98:0/2974805334 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9a8027070 con 0x7fd9c0103490 2024-05-09T08:41:32.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.122+0000 7fd9b7fff700 1 -- 172.21.15.98:0/2974805334 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9a800be40 con 0x7fd9c0103490 2024-05-09T08:41:32.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.122+0000 7fd9b7fff700 1 -- 172.21.15.98:0/2974805334 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9a800f460 con 0x7fd9c0103490 2024-05-09T08:41:32.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.122+0000 7fd9c518d700 1 -- 172.21.15.98:0/2974805334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c0197da0 con 0x7fd9c0103490 2024-05-09T08:41:32.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.122+0000 7fd9c518d700 1 -- 172.21.15.98:0/2974805334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c019aa90 con 0x7fd9c0103490 2024-05-09T08:41:32.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.123+0000 7fd9b7fff700 1 -- 172.21.15.98:0/2974805334 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fd9a802a070 con 0x7fd9c0103490 2024-05-09T08:41:32.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.123+0000 7fd9c518d700 1 -- 172.21.15.98:0/2974805334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9c01086d0 con 0x7fd9c0103490 2024-05-09T08:41:32.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.123+0000 7fd9b7fff700 1 --2- 172.21.15.98:0/2974805334 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd9ac0390f0 0x7fd9ac03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:32.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.123+0000 7fd9b7fff700 1 -- 172.21.15.98:0/2974805334 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd9a801e030 con 0x7fd9c0103490 2024-05-09T08:41:32.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.123+0000 7fd9be59c700 1 --2- 172.21.15.98:0/2974805334 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd9ac0390f0 0x7fd9ac03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:32.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.124+0000 7fd9be59c700 1 --2- 172.21.15.98:0/2974805334 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd9ac0390f0 0x7fd9ac03b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0006fd0 tx=0x7fd9b0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:32.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.128+0000 7fd9b7fff700 1 -- 172.21.15.98:0/2974805334 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd9a8013070 con 0x7fd9c0103490 2024-05-09T08:41:32.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.327+0000 7fd9c518d700 1 -- 172.21.15.98:0/2974805334 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fd9c0062760 con 0x7fd9ac0390f0 2024-05-09T08:41:32.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.331+0000 7fd9b7fff700 1 -- 172.21.15.98:0/2974805334 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd9c0062760 con 0x7fd9ac0390f0 2024-05-09T08:41:32.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.333+0000 7fd9c518d700 1 -- 172.21.15.98:0/2974805334 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd9ac0390f0 msgr2=0x7fd9ac03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:32.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.333+0000 7fd9c518d700 1 --2- 172.21.15.98:0/2974805334 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd9ac0390f0 0x7fd9ac03b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0006fd0 tx=0x7fd9b0006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:32.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.333+0000 7fd9c518d700 1 -- 172.21.15.98:0/2974805334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9c0103490 msgr2=0x7fd9c01975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:32.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.333+0000 7fd9c518d700 1 --2- 172.21.15.98:0/2974805334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9c0103490 0x7fd9c01975d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8006010 tx=0x7fd9a8005050 comp rx=0 tx=0).stop 2024-05-09T08:41:32.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.333+0000 7fd9c518d700 1 -- 172.21.15.98:0/2974805334 shutdown_connections 2024-05-09T08:41:32.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.333+0000 7fd9c518d700 1 --2- 172.21.15.98:0/2974805334 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd9ac0390f0 0x7fd9ac03b5b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:32.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.333+0000 7fd9c518d700 1 --2- 172.21.15.98:0/2974805334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9c0103490 0x7fd9c01975d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:32.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.333+0000 7fd9c518d700 1 -- 172.21.15.98:0/2974805334 >> 172.21.15.98:0/2974805334 conn(0x7fd9c00fd370 msgr2=0x7fd9c00fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:32.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.333+0000 7fd9c518d700 1 -- 172.21.15.98:0/2974805334 shutdown_connections 2024-05-09T08:41:32.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:32.333+0000 7fd9c518d700 1 -- 172.21.15.98:0/2974805334 wait complete. 2024-05-09T08:41:33.097 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi105 2024-05-09T08:41:33.097 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-09T08:41:33.097 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/etc/ceph/ceph.conf 2024-05-09T08:41:33.116 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-09T08:41:33.116 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:41:33.174 INFO:tasks.cephadm:Adding host smithi105 to orchestrator... 2024-05-09T08:41:33.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch host add smithi105 2024-05-09T08:41:33.412 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:41:33.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:33 smithi098 ceph-mon[30092]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:41:33.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:33 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:33 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:41:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:33 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:41:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:33 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:41:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:33 smithi098 ceph-mon[30092]: Updating smithi098:/etc/ceph/ceph.conf 2024-05-09T08:41:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:33 smithi098 ceph-mon[30092]: Updating smithi098:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:41:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:33 smithi098 ceph-mon[30092]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:41:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:33 smithi098 ceph-mon[30092]: Updating smithi098:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:41:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:33 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:33 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:33 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:34.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.958+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3750075417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8100e70 msgr2=0x7f3bf8101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.958+0000 7f3bfcc6b700 1 --2- 172.21.15.98:0/3750075417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8100e70 0x7f3bf8101250 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3be0009910 tx=0x7f3be0009c20 comp rx=0 tx=0).stop 2024-05-09T08:41:34.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.958+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3750075417 shutdown_connections 2024-05-09T08:41:34.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.958+0000 7f3bfcc6b700 1 --2- 172.21.15.98:0/3750075417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8100e70 0x7f3bf8101250 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.958+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3750075417 >> 172.21.15.98:0/3750075417 conn(0x7f3bf80fc720 msgr2=0x7f3bf80feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.958+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3750075417 shutdown_connections 2024-05-09T08:41:34.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.958+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3750075417 wait complete. 2024-05-09T08:41:34.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.959+0000 7f3bfcc6b700 1 Processor -- start 2024-05-09T08:41:34.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.959+0000 7f3bfcc6b700 1 -- start start 2024-05-09T08:41:34.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.959+0000 7f3bfcc6b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8100e70 0x7f3bf81986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.959+0000 7f3bfcc6b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf810b240 con 0x7f3bf8100e70 2024-05-09T08:41:34.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.959+0000 7f3bf659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8100e70 0x7f3bf81986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.960+0000 7f3bf659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8100e70 0x7f3bf81986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60158/0 (socket says 172.21.15.98:60158) 2024-05-09T08:41:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.960+0000 7f3bf659c700 1 -- 172.21.15.98:0/3496690567 learned_addr learned my addr 172.21.15.98:0/3496690567 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.960+0000 7f3bf659c700 1 -- 172.21.15.98:0/3496690567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf8198c20 con 0x7f3bf8100e70 2024-05-09T08:41:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.960+0000 7f3bf659c700 1 --2- 172.21.15.98:0/3496690567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8100e70 0x7f3bf81986e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3be0005850 tx=0x7f3be0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.960+0000 7f3bef7fe700 1 -- 172.21.15.98:0/3496690567 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3be0026070 con 0x7f3bf8100e70 2024-05-09T08:41:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.960+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3496690567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bf8198f10 con 0x7f3bf8100e70 2024-05-09T08:41:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.960+0000 7f3bef7fe700 1 -- 172.21.15.98:0/3496690567 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3be000bbd0 con 0x7f3bf8100e70 2024-05-09T08:41:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.960+0000 7f3bef7fe700 1 -- 172.21.15.98:0/3496690567 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3be0003970 con 0x7f3bf8100e70 2024-05-09T08:41:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.960+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3496690567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bf81993d0 con 0x7f3bf8100e70 2024-05-09T08:41:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.961+0000 7f3bef7fe700 1 -- 172.21.15.98:0/3496690567 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f3be0029070 con 0x7f3bf8100e70 2024-05-09T08:41:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.961+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3496690567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bf8102350 con 0x7f3bf8100e70 2024-05-09T08:41:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.961+0000 7f3bef7fe700 1 --2- 172.21.15.98:0/3496690567 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3be40390f0 0x7f3be403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.961+0000 7f3bef7fe700 1 -- 172.21.15.98:0/3496690567 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3be001e070 con 0x7f3bf8100e70 2024-05-09T08:41:34.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.961+0000 7f3bf5d9b700 1 --2- 172.21.15.98:0/3496690567 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3be40390f0 0x7f3be403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.962+0000 7f3bf5d9b700 1 --2- 172.21.15.98:0/3496690567 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3be40390f0 0x7f3be403b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3be8006fd0 tx=0x7f3be8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:34.966+0000 7f3bef7fe700 1 -- 172.21.15.98:0/3496690567 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3be0013070 con 0x7f3bf8100e70 2024-05-09T08:41:35.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:35.165+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3496690567 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}) -- 0x7f3bf806a5c0 con 0x7f3be40390f0 2024-05-09T08:41:35.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:35 smithi098 ceph-mon[30092]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:41:36.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:36 smithi098 ceph-mon[30092]: Deploying cephadm binary to smithi105 2024-05-09T08:41:38.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:38 smithi098 ceph-mon[30092]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:41:39.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:39.268+0000 7f3bef7fe700 1 -- 172.21.15.98:0/3496690567 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f3bf806a5c0 con 0x7f3be40390f0 2024-05-09T08:41:39.267 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi105' with addr '172.21.15.105' 2024-05-09T08:41:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:39.270+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3496690567 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3be40390f0 msgr2=0x7f3be403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:39.271+0000 7f3bfcc6b700 1 --2- 172.21.15.98:0/3496690567 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3be40390f0 0x7f3be403b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3be8006fd0 tx=0x7f3be8006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:39.271+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3496690567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8100e70 msgr2=0x7f3bf81986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:39.271+0000 7f3bfcc6b700 1 --2- 172.21.15.98:0/3496690567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8100e70 0x7f3bf81986e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3be0005850 tx=0x7f3be0005030 comp rx=0 tx=0).stop 2024-05-09T08:41:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:39.271+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3496690567 shutdown_connections 2024-05-09T08:41:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:39.271+0000 7f3bfcc6b700 1 --2- 172.21.15.98:0/3496690567 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3be40390f0 0x7f3be403b5b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:39.271+0000 7f3bfcc6b700 1 --2- 172.21.15.98:0/3496690567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bf8100e70 0x7f3bf81986e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:39.271+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3496690567 >> 172.21.15.98:0/3496690567 conn(0x7f3bf80fc720 msgr2=0x7f3bf8102840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:39.271+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3496690567 shutdown_connections 2024-05-09T08:41:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:39.271+0000 7f3bfcc6b700 1 -- 172.21.15.98:0/3496690567 wait complete. 2024-05-09T08:41:39.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-09T08:41:40.143 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:41:40.577 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:40 smithi098 ceph-mon[30092]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:41:40.577 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:40 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:40.577 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:40 smithi098 ceph-mon[30092]: Added host smithi105 2024-05-09T08:41:40.577 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:40 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:41:40.577 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:40 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:41.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:41.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:41 smithi098 ceph-mon[30092]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:41:41.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.737+0000 7f167518a700 1 -- 172.21.15.98:0/2757325170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670102f30 msgr2=0x7f1670103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:41.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.737+0000 7f167518a700 1 --2- 172.21.15.98:0/2757325170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670102f30 0x7f1670103310 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1660009910 tx=0x7f1660009c20 comp rx=0 tx=0).stop 2024-05-09T08:41:41.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.737+0000 7f167518a700 1 -- 172.21.15.98:0/2757325170 shutdown_connections 2024-05-09T08:41:41.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.737+0000 7f167518a700 1 --2- 172.21.15.98:0/2757325170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670102f30 0x7f1670103310 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:41.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.737+0000 7f167518a700 1 -- 172.21.15.98:0/2757325170 >> 172.21.15.98:0/2757325170 conn(0x7f16700fc780 msgr2=0x7f16700feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:41.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.737+0000 7f167518a700 1 -- 172.21.15.98:0/2757325170 shutdown_connections 2024-05-09T08:41:41.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.737+0000 7f167518a700 1 -- 172.21.15.98:0/2757325170 wait complete. 2024-05-09T08:41:41.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.738+0000 7f167518a700 1 Processor -- start 2024-05-09T08:41:41.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.738+0000 7f167518a700 1 -- start start 2024-05-09T08:41:41.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.738+0000 7f167518a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670102f30 0x7f167019d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:41.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.738+0000 7f167518a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f167010c400 con 0x7f1670102f30 2024-05-09T08:41:41.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.739+0000 7f166ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670102f30 0x7f167019d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:41.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.739+0000 7f166ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670102f30 0x7f167019d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38630/0 (socket says 172.21.15.98:38630) 2024-05-09T08:41:41.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.739+0000 7f166ed9d700 1 -- 172.21.15.98:0/397472409 learned_addr learned my addr 172.21.15.98:0/397472409 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:41.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.739+0000 7f166ed9d700 1 -- 172.21.15.98:0/397472409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167019db50 con 0x7f1670102f30 2024-05-09T08:41:41.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.739+0000 7f166ed9d700 1 --2- 172.21.15.98:0/397472409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670102f30 0x7f167019d610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1660006010 tx=0x7f1660005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:41.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.739+0000 7f1657fff700 1 -- 172.21.15.98:0/397472409 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1660026070 con 0x7f1670102f30 2024-05-09T08:41:41.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.739+0000 7f1657fff700 1 -- 172.21.15.98:0/397472409 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f166000bc30 con 0x7f1670102f30 2024-05-09T08:41:41.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.739+0000 7f167518a700 1 -- 172.21.15.98:0/397472409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167019dde0 con 0x7f1670102f30 2024-05-09T08:41:41.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.739+0000 7f1657fff700 1 -- 172.21.15.98:0/397472409 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1660003ab0 con 0x7f1670102f30 2024-05-09T08:41:41.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.739+0000 7f167518a700 1 -- 172.21.15.98:0/397472409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16701976e0 con 0x7f1670102f30 2024-05-09T08:41:41.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.740+0000 7f1657fff700 1 -- 172.21.15.98:0/397472409 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f1660029070 con 0x7f1670102f30 2024-05-09T08:41:41.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.740+0000 7f1657fff700 1 --2- 172.21.15.98:0/397472409 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1658039150 0x7f165803b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:41.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.740+0000 7f167518a700 1 -- 172.21.15.98:0/397472409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f167006aa70 con 0x7f1670102f30 2024-05-09T08:41:41.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.740+0000 7f1657fff700 1 -- 172.21.15.98:0/397472409 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f166001e070 con 0x7f1670102f30 2024-05-09T08:41:41.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.740+0000 7f166e59c700 1 --2- 172.21.15.98:0/397472409 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1658039150 0x7f165803b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:41.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.741+0000 7f166e59c700 1 --2- 172.21.15.98:0/397472409 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1658039150 0x7f165803b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1664006fd0 tx=0x7f1664006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:41.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.745+0000 7f1657fff700 1 -- 172.21.15.98:0/397472409 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1660013070 con 0x7f1670102f30 2024-05-09T08:41:41.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.949+0000 7f167518a700 1 -- 172.21.15.98:0/397472409 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1670198c10 con 0x7f1658039150 2024-05-09T08:41:41.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.950+0000 7f1657fff700 1 -- 172.21.15.98:0/397472409 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f1670198c10 con 0x7f1658039150 2024-05-09T08:41:41.950 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:41:41.950 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}] 2024-05-09T08:41:41.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.953+0000 7f167518a700 1 -- 172.21.15.98:0/397472409 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1658039150 msgr2=0x7f165803b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:41.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.953+0000 7f167518a700 1 --2- 172.21.15.98:0/397472409 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1658039150 0x7f165803b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1664006fd0 tx=0x7f1664006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:41.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.953+0000 7f167518a700 1 -- 172.21.15.98:0/397472409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670102f30 msgr2=0x7f167019d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:41.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.953+0000 7f167518a700 1 --2- 172.21.15.98:0/397472409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670102f30 0x7f167019d610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1660006010 tx=0x7f1660005050 comp rx=0 tx=0).stop 2024-05-09T08:41:41.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.953+0000 7f167518a700 1 -- 172.21.15.98:0/397472409 shutdown_connections 2024-05-09T08:41:41.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.953+0000 7f167518a700 1 --2- 172.21.15.98:0/397472409 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1658039150 0x7f165803b610 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:41.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.953+0000 7f167518a700 1 --2- 172.21.15.98:0/397472409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670102f30 0x7f167019d610 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:41.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.953+0000 7f167518a700 1 -- 172.21.15.98:0/397472409 >> 172.21.15.98:0/397472409 conn(0x7f16700fc780 msgr2=0x7f1670103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:41.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.953+0000 7f167518a700 1 -- 172.21.15.98:0/397472409 shutdown_connections 2024-05-09T08:41:41.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:41.953+0000 7f167518a700 1 -- 172.21.15.98:0/397472409 wait complete. 2024-05-09T08:41:42.583 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi142 2024-05-09T08:41:42.583 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-05-09T08:41:42.583 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/etc/ceph/ceph.conf 2024-05-09T08:41:42.601 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-05-09T08:41:42.601 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:41:42.659 INFO:tasks.cephadm:Adding host smithi142 to orchestrator... 2024-05-09T08:41:42.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch host add smithi142 2024-05-09T08:41:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:41:43.109 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:42 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:43.109 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:42 smithi098 ceph-mon[30092]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:41:44.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:43 smithi098 ceph-mon[30092]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:41:44.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.332+0000 7efddf2c0700 1 -- 172.21.15.98:0/316769220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100ed0 msgr2=0x7efdd81012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:44.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.332+0000 7efddf2c0700 1 --2- 172.21.15.98:0/316769220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100ed0 0x7efdd81012b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efdd4009910 tx=0x7efdd4009c20 comp rx=0 tx=0).stop 2024-05-09T08:41:44.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.332+0000 7efddf2c0700 1 -- 172.21.15.98:0/316769220 shutdown_connections 2024-05-09T08:41:44.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.332+0000 7efddf2c0700 1 --2- 172.21.15.98:0/316769220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100ed0 0x7efdd81012b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:44.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.332+0000 7efddf2c0700 1 -- 172.21.15.98:0/316769220 >> 172.21.15.98:0/316769220 conn(0x7efdd8076780 msgr2=0x7efdd8076b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:44.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.332+0000 7efddf2c0700 1 -- 172.21.15.98:0/316769220 shutdown_connections 2024-05-09T08:41:44.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.332+0000 7efddf2c0700 1 -- 172.21.15.98:0/316769220 wait complete. 2024-05-09T08:41:44.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.333+0000 7efddf2c0700 1 Processor -- start 2024-05-09T08:41:44.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.333+0000 7efddf2c0700 1 -- start start 2024-05-09T08:41:44.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.333+0000 7efddf2c0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100ed0 0x7efdd819c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:44.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.333+0000 7efddf2c0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdd810b250 con 0x7efdd8100ed0 2024-05-09T08:41:44.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.333+0000 7efddd05c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100ed0 0x7efdd819c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:44.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.333+0000 7efddd05c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100ed0 0x7efdd819c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38640/0 (socket says 172.21.15.98:38640) 2024-05-09T08:41:44.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.333+0000 7efddd05c700 1 -- 172.21.15.98:0/2112084840 learned_addr learned my addr 172.21.15.98:0/2112084840 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:44.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.333+0000 7efddd05c700 1 -- 172.21.15.98:0/2112084840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd819c9a0 con 0x7efdd8100ed0 2024-05-09T08:41:44.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.334+0000 7efddd05c700 1 --2- 172.21.15.98:0/2112084840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100ed0 0x7efdd819c460 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efdd4005850 tx=0x7efdd4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:44.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.334+0000 7efdce7fc700 1 -- 172.21.15.98:0/2112084840 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdd4026070 con 0x7efdd8100ed0 2024-05-09T08:41:44.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.334+0000 7efdce7fc700 1 -- 172.21.15.98:0/2112084840 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdd400bbd0 con 0x7efdd8100ed0 2024-05-09T08:41:44.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.334+0000 7efddf2c0700 1 -- 172.21.15.98:0/2112084840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd819cc90 con 0x7efdd8100ed0 2024-05-09T08:41:44.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.334+0000 7efdce7fc700 1 -- 172.21.15.98:0/2112084840 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdd4003970 con 0x7efdd8100ed0 2024-05-09T08:41:44.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.334+0000 7efddf2c0700 1 -- 172.21.15.98:0/2112084840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd8196620 con 0x7efdd8100ed0 2024-05-09T08:41:44.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.335+0000 7efdce7fc700 1 -- 172.21.15.98:0/2112084840 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7efdd4029070 con 0x7efdd8100ed0 2024-05-09T08:41:44.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.335+0000 7efddf2c0700 1 -- 172.21.15.98:0/2112084840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdd8102360 con 0x7efdd8100ed0 2024-05-09T08:41:44.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.335+0000 7efdce7fc700 1 --2- 172.21.15.98:0/2112084840 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7efdc4039150 0x7efdc403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:44.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.335+0000 7efdce7fc700 1 -- 172.21.15.98:0/2112084840 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efdd401e070 con 0x7efdd8100ed0 2024-05-09T08:41:44.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.335+0000 7efddc85b700 1 --2- 172.21.15.98:0/2112084840 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7efdc4039150 0x7efdc403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:44.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.336+0000 7efddc85b700 1 --2- 172.21.15.98:0/2112084840 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7efdc4039150 0x7efdc403b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efdc8006fd0 tx=0x7efdc8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:44.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.340+0000 7efdce7fc700 1 -- 172.21.15.98:0/2112084840 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efdd4051070 con 0x7efdd8100ed0 2024-05-09T08:41:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:44.541+0000 7efddf2c0700 1 -- 172.21.15.98:0/2112084840 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi142", "target": ["mon-mgr", ""]}) -- 0x7efdd8197b50 con 0x7efdc4039150 2024-05-09T08:41:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:46 smithi098 ceph-mon[30092]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi142", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:41:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:41:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:41:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:41:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:46 smithi098 ceph-mon[30092]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-09T08:41:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:46 smithi098 ceph-mon[30092]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:41:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:46 smithi098 ceph-mon[30092]: Deploying cephadm binary to smithi142 2024-05-09T08:41:47.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:47 smithi098 ceph-mon[30092]: Updating smithi105:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:41:47.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:47 smithi098 ceph-mon[30092]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:41:48.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:48 smithi098 ceph-mon[30092]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:41:48.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:48.779+0000 7efdce7fc700 1 -- 172.21.15.98:0/2112084840 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7efdd8197b50 con 0x7efdc4039150 2024-05-09T08:41:48.778 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi142' with addr '172.21.15.142' 2024-05-09T08:41:48.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:48.781+0000 7efddf2c0700 1 -- 172.21.15.98:0/2112084840 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7efdc4039150 msgr2=0x7efdc403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:48.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:48.781+0000 7efddf2c0700 1 --2- 172.21.15.98:0/2112084840 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7efdc4039150 0x7efdc403b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efdc8006fd0 tx=0x7efdc8006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:48.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:48.781+0000 7efddf2c0700 1 -- 172.21.15.98:0/2112084840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100ed0 msgr2=0x7efdd819c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:48.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:48.781+0000 7efddf2c0700 1 --2- 172.21.15.98:0/2112084840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100ed0 0x7efdd819c460 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efdd4005850 tx=0x7efdd4005030 comp rx=0 tx=0).stop 2024-05-09T08:41:48.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:48.781+0000 7efddf2c0700 1 -- 172.21.15.98:0/2112084840 shutdown_connections 2024-05-09T08:41:48.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:48.781+0000 7efddf2c0700 1 --2- 172.21.15.98:0/2112084840 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7efdc4039150 0x7efdc403b610 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:48.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:48.781+0000 7efddf2c0700 1 --2- 172.21.15.98:0/2112084840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100ed0 0x7efdd819c460 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:48.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:48.781+0000 7efddf2c0700 1 -- 172.21.15.98:0/2112084840 >> 172.21.15.98:0/2112084840 conn(0x7efdd8076780 msgr2=0x7efdd8103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:48.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:48.782+0000 7efddf2c0700 1 -- 172.21.15.98:0/2112084840 shutdown_connections 2024-05-09T08:41:48.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:48.782+0000 7efddf2c0700 1 -- 172.21.15.98:0/2112084840 wait complete. 2024-05-09T08:41:49.362 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:49 smithi098 ceph-mon[30092]: Updating smithi105:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:41:49.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:49 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:49.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:49 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:49.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:49 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:49.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:49 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:49.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:49 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:41:49.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-09T08:41:49.607 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:41:50.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:50 smithi098 ceph-mon[30092]: Added host smithi142 2024-05-09T08:41:50.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:50 smithi098 ceph-mon[30092]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:41:50.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:50.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:50.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:50.999+0000 7f63cd4ec700 1 -- 172.21.15.98:0/2719535266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8100e70 msgr2=0x7f63c8101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:50.999+0000 7f63cd4ec700 1 --2- 172.21.15.98:0/2719535266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8100e70 0x7f63c8101250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009910 tx=0x7f63b8009c20 comp rx=0 tx=0).stop 2024-05-09T08:41:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:50.999+0000 7f63cd4ec700 1 -- 172.21.15.98:0/2719535266 shutdown_connections 2024-05-09T08:41:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:50.999+0000 7f63cd4ec700 1 --2- 172.21.15.98:0/2719535266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8100e70 0x7f63c8101250 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:50.999+0000 7f63cd4ec700 1 -- 172.21.15.98:0/2719535266 >> 172.21.15.98:0/2719535266 conn(0x7f63c80fc720 msgr2=0x7f63c80feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:50.999+0000 7f63cd4ec700 1 -- 172.21.15.98:0/2719535266 shutdown_connections 2024-05-09T08:41:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:50.999+0000 7f63cd4ec700 1 -- 172.21.15.98:0/2719535266 wait complete. 2024-05-09T08:41:50.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.000+0000 7f63cd4ec700 1 Processor -- start 2024-05-09T08:41:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.000+0000 7f63cd4ec700 1 -- start start 2024-05-09T08:41:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.000+0000 7f63cd4ec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8100e70 0x7f63c81986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.000+0000 7f63cd4ec700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63c810b240 con 0x7f63c8100e70 2024-05-09T08:41:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.000+0000 7f63c6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8100e70 0x7f63c81986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:51.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.000+0000 7f63c6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8100e70 0x7f63c81986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55796/0 (socket says 172.21.15.98:55796) 2024-05-09T08:41:51.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.000+0000 7f63c6ffd700 1 -- 172.21.15.98:0/3907177983 learned_addr learned my addr 172.21.15.98:0/3907177983 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:51.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.001+0000 7f63c6ffd700 1 -- 172.21.15.98:0/3907177983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c8198c00 con 0x7f63c8100e70 2024-05-09T08:41:51.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.001+0000 7f63c6ffd700 1 --2- 172.21.15.98:0/3907177983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8100e70 0x7f63c81986c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f63b8005850 tx=0x7f63b8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:51.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.001+0000 7f63affff700 1 -- 172.21.15.98:0/3907177983 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63b8026070 con 0x7f63c8100e70 2024-05-09T08:41:51.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.001+0000 7f63cd4ec700 1 -- 172.21.15.98:0/3907177983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c8198ef0 con 0x7f63c8100e70 2024-05-09T08:41:51.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.001+0000 7f63affff700 1 -- 172.21.15.98:0/3907177983 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63b800bbd0 con 0x7f63c8100e70 2024-05-09T08:41:51.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.001+0000 7f63affff700 1 -- 172.21.15.98:0/3907177983 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63b8003970 con 0x7f63c8100e70 2024-05-09T08:41:51.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.001+0000 7f63cd4ec700 1 -- 172.21.15.98:0/3907177983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c81993b0 con 0x7f63c8100e70 2024-05-09T08:41:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.002+0000 7f63affff700 1 -- 172.21.15.98:0/3907177983 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f63b8029070 con 0x7f63c8100e70 2024-05-09T08:41:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.002+0000 7f63cd4ec700 1 -- 172.21.15.98:0/3907177983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c8102350 con 0x7f63c8100e70 2024-05-09T08:41:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.002+0000 7f63affff700 1 --2- 172.21.15.98:0/3907177983 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f63b00390f0 0x7f63b003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.002+0000 7f63affff700 1 -- 172.21.15.98:0/3907177983 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f63b801e070 con 0x7f63c8100e70 2024-05-09T08:41:51.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.002+0000 7f63c67fc700 1 --2- 172.21.15.98:0/3907177983 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f63b00390f0 0x7f63b003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:51.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.003+0000 7f63c67fc700 1 --2- 172.21.15.98:0/3907177983 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f63b00390f0 0x7f63b003b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f63bc006fd0 tx=0x7f63bc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.007+0000 7f63affff700 1 -- 172.21.15.98:0/3907177983 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f63b8013070 con 0x7f63c8100e70 2024-05-09T08:41:51.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.207+0000 7f63cd4ec700 1 -- 172.21.15.98:0/3907177983 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f63c806a5c0 con 0x7f63b00390f0 2024-05-09T08:41:51.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.208+0000 7f63affff700 1 -- 172.21.15.98:0/3907177983 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f63c806a5c0 con 0x7f63b00390f0 2024-05-09T08:41:51.207 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:41:51.207 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}, {"addr": "172.21.15.142", "hostname": "smithi142", "labels": [], "status": ""}] 2024-05-09T08:41:51.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.210+0000 7f63cd4ec700 1 -- 172.21.15.98:0/3907177983 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f63b00390f0 msgr2=0x7f63b003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:51.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.210+0000 7f63cd4ec700 1 --2- 172.21.15.98:0/3907177983 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f63b00390f0 0x7f63b003b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f63bc006fd0 tx=0x7f63bc006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:51.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.210+0000 7f63cd4ec700 1 -- 172.21.15.98:0/3907177983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8100e70 msgr2=0x7f63c81986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:51.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.210+0000 7f63cd4ec700 1 --2- 172.21.15.98:0/3907177983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8100e70 0x7f63c81986c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f63b8005850 tx=0x7f63b8005030 comp rx=0 tx=0).stop 2024-05-09T08:41:51.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.210+0000 7f63cd4ec700 1 -- 172.21.15.98:0/3907177983 shutdown_connections 2024-05-09T08:41:51.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.210+0000 7f63cd4ec700 1 --2- 172.21.15.98:0/3907177983 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f63b00390f0 0x7f63b003b5b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:51.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.210+0000 7f63cd4ec700 1 --2- 172.21.15.98:0/3907177983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8100e70 0x7f63c81986c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:51.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.210+0000 7f63cd4ec700 1 -- 172.21.15.98:0/3907177983 >> 172.21.15.98:0/3907177983 conn(0x7f63c80fc720 msgr2=0x7f63c8102840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:51.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.210+0000 7f63cd4ec700 1 -- 172.21.15.98:0/3907177983 shutdown_connections 2024-05-09T08:41:51.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:51.210+0000 7f63cd4ec700 1 -- 172.21.15.98:0/3907177983 wait complete. 2024-05-09T08:41:51.860 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-09T08:41:51.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd crush tunables default 2024-05-09T08:41:52.094 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:41:52.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:52 smithi098 ceph-mon[30092]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:41:52.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:52 smithi098 ceph-mon[30092]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:41:52.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:53.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.562+0000 7f800d33f700 1 -- 172.21.15.98:0/1705548797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8008100e90 msgr2=0x7f8008101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:53.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.562+0000 7f800d33f700 1 --2- 172.21.15.98:0/1705548797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8008100e90 0x7f8008101270 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8009910 tx=0x7f7ff8009c20 comp rx=0 tx=0).stop 2024-05-09T08:41:53.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.562+0000 7f800d33f700 1 -- 172.21.15.98:0/1705548797 shutdown_connections 2024-05-09T08:41:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.562+0000 7f800d33f700 1 --2- 172.21.15.98:0/1705548797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8008100e90 0x7f8008101270 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.562+0000 7f800d33f700 1 -- 172.21.15.98:0/1705548797 >> 172.21.15.98:0/1705548797 conn(0x7f80080fc740 msgr2=0x7f80080feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.562+0000 7f800d33f700 1 -- 172.21.15.98:0/1705548797 shutdown_connections 2024-05-09T08:41:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.563+0000 7f800d33f700 1 -- 172.21.15.98:0/1705548797 wait complete. 2024-05-09T08:41:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.563+0000 7f800d33f700 1 Processor -- start 2024-05-09T08:41:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.563+0000 7f800d33f700 1 -- start start 2024-05-09T08:41:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.564+0000 7f800d33f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8008100e90 0x7f800819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.564+0000 7f800d33f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f800810b260 con 0x7f8008100e90 2024-05-09T08:41:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.564+0000 7f8006ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8008100e90 0x7f800819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.564+0000 7f8006ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8008100e90 0x7f800819c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55806/0 (socket says 172.21.15.98:55806) 2024-05-09T08:41:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.564+0000 7f8006ffd700 1 -- 172.21.15.98:0/20874310 learned_addr learned my addr 172.21.15.98:0/20874310 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:41:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.564+0000 7f8006ffd700 1 -- 172.21.15.98:0/20874310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f800819c920 con 0x7f8008100e90 2024-05-09T08:41:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.564+0000 7f8006ffd700 1 --2- 172.21.15.98:0/20874310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8008100e90 0x7f800819c3e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8005850 tx=0x7f7ff8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:53.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.564+0000 7f7feffff700 1 -- 172.21.15.98:0/20874310 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ff8026070 con 0x7f8008100e90 2024-05-09T08:41:53.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.564+0000 7f7feffff700 1 -- 172.21.15.98:0/20874310 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ff800bbd0 con 0x7f8008100e90 2024-05-09T08:41:53.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.564+0000 7f800d33f700 1 -- 172.21.15.98:0/20874310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f800819cc10 con 0x7f8008100e90 2024-05-09T08:41:53.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.564+0000 7f7feffff700 1 -- 172.21.15.98:0/20874310 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ff8003970 con 0x7f8008100e90 2024-05-09T08:41:53.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.564+0000 7f800d33f700 1 -- 172.21.15.98:0/20874310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80081965a0 con 0x7f8008100e90 2024-05-09T08:41:53.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.565+0000 7f800d33f700 1 -- 172.21.15.98:0/20874310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8008102370 con 0x7f8008100e90 2024-05-09T08:41:53.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.565+0000 7f7feffff700 1 -- 172.21.15.98:0/20874310 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f7ff8029070 con 0x7f8008100e90 2024-05-09T08:41:53.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.565+0000 7f7feffff700 1 --2- 172.21.15.98:0/20874310 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f7ff003b300 0x7f7ff003d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:53.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.565+0000 7f7feffff700 1 -- 172.21.15.98:0/20874310 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7ff801e070 con 0x7f8008100e90 2024-05-09T08:41:53.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.566+0000 7f80067fc700 1 --2- 172.21.15.98:0/20874310 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f7ff003b300 0x7f7ff003d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:53.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.566+0000 7f80067fc700 1 --2- 172.21.15.98:0/20874310 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f7ff003b300 0x7f7ff003d7c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc006fd0 tx=0x7f7ffc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:53.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.571+0000 7f7feffff700 1 -- 172.21.15.98:0/20874310 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7ff8013070 con 0x7f8008100e90 2024-05-09T08:41:53.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:53.768+0000 7f800d33f700 1 -- 172.21.15.98:0/20874310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f8008063980 con 0x7f8008100e90 2024-05-09T08:41:54.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:54.293+0000 7f7feffff700 1 -- 172.21.15.98:0/20874310 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f7ff8030090 con 0x7f8008100e90 2024-05-09T08:41:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:54.295+0000 7f800d33f700 1 -- 172.21.15.98:0/20874310 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f7ff003b300 msgr2=0x7f7ff003d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:54.295+0000 7f800d33f700 1 --2- 172.21.15.98:0/20874310 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f7ff003b300 0x7f7ff003d7c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc006fd0 tx=0x7f7ffc006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:54.295+0000 7f800d33f700 1 -- 172.21.15.98:0/20874310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8008100e90 msgr2=0x7f800819c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:54.295+0000 7f800d33f700 1 --2- 172.21.15.98:0/20874310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8008100e90 0x7f800819c3e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8005850 tx=0x7f7ff8005030 comp rx=0 tx=0).stop 2024-05-09T08:41:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:54.295+0000 7f800d33f700 1 -- 172.21.15.98:0/20874310 shutdown_connections 2024-05-09T08:41:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:54.295+0000 7f800d33f700 1 --2- 172.21.15.98:0/20874310 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f7ff003b300 0x7f7ff003d7c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:54.295+0000 7f800d33f700 1 --2- 172.21.15.98:0/20874310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8008100e90 0x7f800819c3e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:54.295+0000 7f800d33f700 1 -- 172.21.15.98:0/20874310 >> 172.21.15.98:0/20874310 conn(0x7f80080fc740 msgr2=0x7f8008102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:54.296+0000 7f800d33f700 1 -- 172.21.15.98:0/20874310 shutdown_connections 2024-05-09T08:41:54.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:41:54.296+0000 7f800d33f700 1 -- 172.21.15.98:0/20874310 wait complete. 2024-05-09T08:41:54.297 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-05-09T08:41:54.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:54 smithi098 ceph-mon[30092]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:41:54.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:54 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/20874310' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-09T08:41:54.975 INFO:tasks.cephadm:Adding mon.a on smithi098 2024-05-09T08:41:54.975 INFO:tasks.cephadm:Adding mon.b on smithi105 2024-05-09T08:41:54.975 INFO:tasks.cephadm:Adding mon.c on smithi142 2024-05-09T08:41:54.976 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch apply mon '3;smithi098:172.21.15.98=a;smithi105:172.21.15.105=b;smithi142:172.21.15.142=c' 2024-05-09T08:41:55.213 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:41:55.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:55 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/20874310' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-09T08:41:55.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:55 smithi098 ceph-mon[30092]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:41:55.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:55.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:55.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:55.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:55.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi142", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:41:55.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:41:55.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:41:56.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:56 smithi098 ceph-mon[30092]: Updating smithi142:/etc/ceph/ceph.conf 2024-05-09T08:41:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:56 smithi098 ceph-mon[30092]: Updating smithi142:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:41:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:56 smithi098 ceph-mon[30092]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:41:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:56 smithi098 ceph-mon[30092]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:41:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:56 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:56 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:56 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:56.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.740+0000 7f1f3898a700 1 -- 172.21.15.142:0/3542037978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f34106450 msgr2=0x7f1f34106830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.740+0000 7f1f3898a700 1 --2- 172.21.15.142:0/3542037978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f34106450 0x7f1f34106830 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009910 tx=0x7f1f24009c20 comp rx=0 tx=0).stop 2024-05-09T08:41:56.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.740+0000 7f1f3898a700 1 -- 172.21.15.142:0/3542037978 shutdown_connections 2024-05-09T08:41:56.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.740+0000 7f1f3898a700 1 --2- 172.21.15.142:0/3542037978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f34106450 0x7f1f34106830 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.740+0000 7f1f3898a700 1 -- 172.21.15.142:0/3542037978 >> 172.21.15.142:0/3542037978 conn(0x7f1f34078250 msgr2=0x7f1f34078660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:56.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.740+0000 7f1f3898a700 1 -- 172.21.15.142:0/3542037978 shutdown_connections 2024-05-09T08:41:56.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.740+0000 7f1f3898a700 1 -- 172.21.15.142:0/3542037978 wait complete. 2024-05-09T08:41:56.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.740+0000 7f1f3898a700 1 Processor -- start 2024-05-09T08:41:56.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.740+0000 7f1f3898a700 1 -- start start 2024-05-09T08:41:56.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.741+0000 7f1f3898a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f34106450 0x7f1f3419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.741+0000 7f1f3898a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3410a160 con 0x7f1f34106450 2024-05-09T08:41:56.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.741+0000 7f1f3259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f34106450 0x7f1f3419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.741+0000 7f1f3259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f34106450 0x7f1f3419f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.142:49836/0 (socket says 172.21.15.142:49836) 2024-05-09T08:41:56.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.741+0000 7f1f3259c700 1 -- 172.21.15.142:0/557603113 learned_addr learned my addr 172.21.15.142:0/557603113 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-05-09T08:41:56.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.741+0000 7f1f3259c700 1 -- 172.21.15.142:0/557603113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f3419fbe0 con 0x7f1f34106450 2024-05-09T08:41:56.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.741+0000 7f1f3259c700 1 --2- 172.21.15.142:0/557603113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f34106450 0x7f1f3419f6a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1f24005850 tx=0x7f1f24005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:56.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.742+0000 7f1f237fe700 1 -- 172.21.15.142:0/557603113 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f24026070 con 0x7f1f34106450 2024-05-09T08:41:56.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.742+0000 7f1f237fe700 1 -- 172.21.15.142:0/557603113 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f2400bbd0 con 0x7f1f34106450 2024-05-09T08:41:56.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.742+0000 7f1f3898a700 1 -- 172.21.15.142:0/557603113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f3419fed0 con 0x7f1f34106450 2024-05-09T08:41:56.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.742+0000 7f1f237fe700 1 -- 172.21.15.142:0/557603113 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f24003970 con 0x7f1f34106450 2024-05-09T08:41:56.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.742+0000 7f1f3898a700 1 -- 172.21.15.142:0/557603113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f34199860 con 0x7f1f34106450 2024-05-09T08:41:56.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.743+0000 7f1f3898a700 1 -- 172.21.15.142:0/557603113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f34101b00 con 0x7f1f34106450 2024-05-09T08:41:56.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.743+0000 7f1f237fe700 1 -- 172.21.15.142:0/557603113 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f1f24029070 con 0x7f1f34106450 2024-05-09T08:41:56.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.743+0000 7f1f237fe700 1 --2- 172.21.15.142:0/557603113 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1f1c039140 0x7f1f1c03b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.743+0000 7f1f237fe700 1 -- 172.21.15.142:0/557603113 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1f2401e070 con 0x7f1f34106450 2024-05-09T08:41:56.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.743+0000 7f1f31d9b700 1 --2- 172.21.15.142:0/557603113 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1f1c039140 0x7f1f1c03b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.743+0000 7f1f31d9b700 1 --2- 172.21.15.142:0/557603113 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1f1c039140 0x7f1f1c03b600 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1f28006fd0 tx=0x7f1f28006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:56.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.748+0000 7f1f237fe700 1 -- 172.21.15.142:0/557603113 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1f24051070 con 0x7f1f34106450 2024-05-09T08:41:56.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.954+0000 7f1f3898a700 1 -- 172.21.15.142:0/557603113 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi105:172.21.15.105=b;smithi142:172.21.15.142=c", "target": ["mon-mgr", ""]}) -- 0x7f1f3419ae00 con 0x7f1f1c039140 2024-05-09T08:41:56.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.960+0000 7f1f237fe700 1 -- 172.21.15.142:0/557603113 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1f3419ae00 con 0x7f1f1c039140 2024-05-09T08:41:56.959 INFO:teuthology.orchestra.run.smithi142.stdout:Scheduled mon update... 2024-05-09T08:41:56.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.962+0000 7f1f3898a700 1 -- 172.21.15.142:0/557603113 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1f1c039140 msgr2=0x7f1f1c03b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.962+0000 7f1f3898a700 1 --2- 172.21.15.142:0/557603113 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1f1c039140 0x7f1f1c03b600 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1f28006fd0 tx=0x7f1f28006e40 comp rx=0 tx=0).stop 2024-05-09T08:41:56.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.962+0000 7f1f3898a700 1 -- 172.21.15.142:0/557603113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f34106450 msgr2=0x7f1f3419f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.962+0000 7f1f3898a700 1 --2- 172.21.15.142:0/557603113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f34106450 0x7f1f3419f6a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1f24005850 tx=0x7f1f24005030 comp rx=0 tx=0).stop 2024-05-09T08:41:56.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.962+0000 7f1f3898a700 1 -- 172.21.15.142:0/557603113 shutdown_connections 2024-05-09T08:41:56.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.962+0000 7f1f3898a700 1 --2- 172.21.15.142:0/557603113 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1f1c039140 0x7f1f1c03b600 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.962+0000 7f1f3898a700 1 --2- 172.21.15.142:0/557603113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f34106450 0x7f1f3419f6a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.962+0000 7f1f3898a700 1 -- 172.21.15.142:0/557603113 >> 172.21.15.142:0/557603113 conn(0x7f1f34078250 msgr2=0x7f1f340ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:56.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.963+0000 7f1f3898a700 1 -- 172.21.15.142:0/557603113 shutdown_connections 2024-05-09T08:41:56.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:41:56.963+0000 7f1f3898a700 1 -- 172.21.15.142:0/557603113 wait complete. 2024-05-09T08:41:57.523 DEBUG:teuthology.orchestra.run.smithi105:mon.b> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.b.service 2024-05-09T08:41:57.526 DEBUG:teuthology.orchestra.run.smithi142:mon.c> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.c.service 2024-05-09T08:41:57.528 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-09T08:41:57.528 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-09T08:41:57.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:57 smithi098 ceph-mon[30092]: Updating smithi142:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:41:57.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:57 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:57.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:57 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:41:57.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:57 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:41:57.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:57 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:41:57.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:57 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:41:57.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:57 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:41:57.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:57 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:41:58.518 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:41:58.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:58 smithi098 ceph-mon[30092]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi105:172.21.15.105=b;smithi142:172.21.15.142=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:41:58.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:58 smithi098 ceph-mon[30092]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi105:172.21.15.105=b;smithi142:172.21.15.142=c;count:3 2024-05-09T08:41:58.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:58 smithi098 ceph-mon[30092]: Deploying daemon mon.c on smithi142 2024-05-09T08:41:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:41:58 smithi098 ceph-mon[30092]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:00.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:00 smithi098 ceph-mon[30092]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:01.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.015+0000 7f8798e94700 1 -- 172.21.15.142:0/903825702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8794100e90 msgr2=0x7f8794101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:01.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.015+0000 7f8798e94700 1 --2- 172.21.15.142:0/903825702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8794100e90 0x7f8794101270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8788009910 tx=0x7f8788009c20 comp rx=0 tx=0).stop 2024-05-09T08:42:01.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.016+0000 7f8798e94700 1 -- 172.21.15.142:0/903825702 shutdown_connections 2024-05-09T08:42:01.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.016+0000 7f8798e94700 1 --2- 172.21.15.142:0/903825702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8794100e90 0x7f8794101270 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:01.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.016+0000 7f8798e94700 1 -- 172.21.15.142:0/903825702 >> 172.21.15.142:0/903825702 conn(0x7f87940fc740 msgr2=0x7f87940feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:01.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.016+0000 7f8798e94700 1 -- 172.21.15.142:0/903825702 shutdown_connections 2024-05-09T08:42:01.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.016+0000 7f8798e94700 1 -- 172.21.15.142:0/903825702 wait complete. 2024-05-09T08:42:01.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.016+0000 7f8798e94700 1 Processor -- start 2024-05-09T08:42:01.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.016+0000 7f8798e94700 1 -- start start 2024-05-09T08:42:01.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.017+0000 7f8798e94700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8794100e90 0x7f8794102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:01.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.017+0000 7f8798e94700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f879410b260 con 0x7f8794100e90 2024-05-09T08:42:01.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.017+0000 7f879259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8794100e90 0x7f8794102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:01.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.017+0000 7f879259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8794100e90 0x7f8794102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.142:49860/0 (socket says 172.21.15.142:49860) 2024-05-09T08:42:01.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.017+0000 7f879259c700 1 -- 172.21.15.142:0/4273139154 learned_addr learned my addr 172.21.15.142:0/4273139154 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-05-09T08:42:01.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.017+0000 7f879259c700 1 -- 172.21.15.142:0/4273139154 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8794104820 con 0x7f8794100e90 2024-05-09T08:42:01.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.018+0000 7f879259c700 1 --2- 172.21.15.142:0/4273139154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8794100e90 0x7f8794102930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8788005850 tx=0x7f8788005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:01.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.018+0000 7f87837fe700 1 -- 172.21.15.142:0/4273139154 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8788026070 con 0x7f8794100e90 2024-05-09T08:42:01.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.018+0000 7f8798e94700 1 -- 172.21.15.142:0/4273139154 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8794102ed0 con 0x7f8794100e90 2024-05-09T08:42:01.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.018+0000 7f87837fe700 1 -- 172.21.15.142:0/4273139154 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f878800bc30 con 0x7f8794100e90 2024-05-09T08:42:01.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.018+0000 7f87837fe700 1 -- 172.21.15.142:0/4273139154 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8788003a20 con 0x7f8794100e90 2024-05-09T08:42:01.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.018+0000 7f8798e94700 1 -- 172.21.15.142:0/4273139154 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8794103390 con 0x7f8794100e90 2024-05-09T08:42:01.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.019+0000 7f87837fe700 1 -- 172.21.15.142:0/4273139154 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f8788029070 con 0x7f8794100e90 2024-05-09T08:42:01.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.019+0000 7f8798e94700 1 -- 172.21.15.142:0/4273139154 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8794102370 con 0x7f8794100e90 2024-05-09T08:42:01.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.019+0000 7f87837fe700 1 --2- 172.21.15.142:0/4273139154 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f877c0390f0 0x7f877c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:01.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.019+0000 7f87837fe700 1 -- 172.21.15.142:0/4273139154 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f878801e070 con 0x7f8794100e90 2024-05-09T08:42:01.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.019+0000 7f8791d9b700 1 --2- 172.21.15.142:0/4273139154 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f877c0390f0 0x7f877c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:01.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.020+0000 7f8791d9b700 1 --2- 172.21.15.142:0/4273139154 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f877c0390f0 0x7f877c03b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8784006fd0 tx=0x7f8784006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:01.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.025+0000 7f87837fe700 1 -- 172.21.15.142:0/4273139154 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8788013070 con 0x7f8794100e90 2024-05-09T08:42:01.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.322+0000 7f8798e94700 1 -- 172.21.15.142:0/4273139154 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f8794103f00 con 0x7f8794100e90 2024-05-09T08:42:01.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.323+0000 7f87837fe700 1 -- 172.21.15.142:0/4273139154 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f8788030020 con 0x7f8794100e90 2024-05-09T08:42:01.322 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:42:01.322 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":1,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","modified":"2024-05-09T08:40:14.468055Z","created":"2024-05-09T08:40:14.468055Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-09T08:42:01.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.325+0000 7f8798e94700 1 -- 172.21.15.142:0/4273139154 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f877c0390f0 msgr2=0x7f877c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:01.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.325+0000 7f8798e94700 1 --2- 172.21.15.142:0/4273139154 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f877c0390f0 0x7f877c03b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8784006fd0 tx=0x7f8784006e40 comp rx=0 tx=0).stop 2024-05-09T08:42:01.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.325+0000 7f8798e94700 1 -- 172.21.15.142:0/4273139154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8794100e90 msgr2=0x7f8794102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:01.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.325+0000 7f8798e94700 1 --2- 172.21.15.142:0/4273139154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8794100e90 0x7f8794102930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8788005850 tx=0x7f8788005030 comp rx=0 tx=0).stop 2024-05-09T08:42:01.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.325+0000 7f8798e94700 1 -- 172.21.15.142:0/4273139154 shutdown_connections 2024-05-09T08:42:01.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.325+0000 7f8798e94700 1 --2- 172.21.15.142:0/4273139154 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f877c0390f0 0x7f877c03b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:01.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.325+0000 7f8798e94700 1 --2- 172.21.15.142:0/4273139154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8794100e90 0x7f8794102930 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:01.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.325+0000 7f8798e94700 1 -- 172.21.15.142:0/4273139154 >> 172.21.15.142:0/4273139154 conn(0x7f87940fc740 msgr2=0x7f879406a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:01.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.326+0000 7f8798e94700 1 -- 172.21.15.142:0/4273139154 shutdown_connections 2024-05-09T08:42:01.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:01.326+0000 7f8798e94700 1 -- 172.21.15.142:0/4273139154 wait complete. 2024-05-09T08:42:01.325 INFO:teuthology.orchestra.run.smithi142.stderr:dumped monmap epoch 1 2024-05-09T08:42:01.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:01 smithi142 systemd[1]: Started Ceph mon.c for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:42:03.083 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-09T08:42:03.083 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-09T08:42:03.332 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:42:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:42:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: mon.a calling monitor election 2024-05-09T08:42:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: mon.c calling monitor election 2024-05-09T08:42:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-09T08:42:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: monmap epoch 2 2024-05-09T08:42:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: fsid a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:42:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: last_changed 2024-05-09T08:42:01.668574+0000 2024-05-09T08:42:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: created 2024-05-09T08:40:14.468055+0000 2024-05-09T08:42:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: min_mon_release 19 (squid) 2024-05-09T08:42:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: election_strategy: 1 2024-05-09T08:42:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-05-09T08:42:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: 1: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.c 2024-05-09T08:42:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: fsmap 2024-05-09T08:42:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:42:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: mgrmap e13: x(active, since 49s) 2024-05-09T08:42:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: overall HEALTH_OK 2024-05-09T08:42:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:42:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:07.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:42:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: mon.a calling monitor election 2024-05-09T08:42:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: mon.c calling monitor election 2024-05-09T08:42:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-09T08:42:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: monmap epoch 2 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: fsid a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: last_changed 2024-05-09T08:42:01.668574+0000 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: created 2024-05-09T08:40:14.468055+0000 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: min_mon_release 19 (squid) 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: election_strategy: 1 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: 1: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.c 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: fsmap 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: mgrmap e13: x(active, since 49s) 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: overall HEALTH_OK 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:42:07.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.896+0000 7f75e0af1700 1 -- 172.21.15.142:0/1439416280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75c40035c0 msgr2=0x7f75c4005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.896+0000 7f75e0af1700 1 --2- 172.21.15.142:0/1439416280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75c40035c0 0x7f75c4005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f75dc06a610 tx=0x7f75cc00feb0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.897+0000 7f75e0af1700 1 -- 172.21.15.142:0/1439416280 shutdown_connections 2024-05-09T08:42:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.897+0000 7f75e0af1700 1 --2- 172.21.15.142:0/1439416280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75c40035c0 0x7f75c4005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.897+0000 7f75e0af1700 1 -- 172.21.15.142:0/1439416280 >> 172.21.15.142:0/1439416280 conn(0x7f75dc0fc820 msgr2=0x7f75dc0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.897+0000 7f75e0af1700 1 -- 172.21.15.142:0/1439416280 shutdown_connections 2024-05-09T08:42:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.897+0000 7f75e0af1700 1 -- 172.21.15.142:0/1439416280 wait complete. 2024-05-09T08:42:07.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.898+0000 7f75e0af1700 1 Processor -- start 2024-05-09T08:42:07.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.898+0000 7f75e0af1700 1 -- start start 2024-05-09T08:42:07.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.898+0000 7f75e0af1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75c40035c0 0x7f75dc19d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.898+0000 7f75e0af1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75dc19dbe0 0x7f75dc197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.898+0000 7f75e0af1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc109ed0 con 0x7f75c40035c0 2024-05-09T08:42:07.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.898+0000 7f75e0af1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc109d50 con 0x7f75dc19dbe0 2024-05-09T08:42:07.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.898+0000 7f75d9d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75dc19dbe0 0x7f75dc197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.898+0000 7f75d9d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75dc19dbe0 0x7f75dc197860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43082/0 (socket says 172.21.15.142:43082) 2024-05-09T08:42:07.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.899+0000 7f75d9d9b700 1 -- 172.21.15.142:0/3270531969 learned_addr learned my addr 172.21.15.142:0/3270531969 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-05-09T08:42:07.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.899+0000 7f75d9d9b700 1 -- 172.21.15.142:0/3270531969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75dc19dbe0 msgr2=0x7f75dc197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-09T08:42:07.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.899+0000 7f75d9d9b700 1 -- 172.21.15.142:0/3270531969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75dc19dbe0 msgr2=0x7f75dc197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T08:42:07.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.899+0000 7f75d9d9b700 1 --2- 172.21.15.142:0/3270531969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75dc19dbe0 0x7f75dc197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T08:42:07.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.899+0000 7f75da59c700 1 --2- 172.21.15.142:0/3270531969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75c40035c0 0x7f75dc19d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.899+0000 7f75d9d9b700 1 --2- 172.21.15.142:0/3270531969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75dc19dbe0 0x7f75dc197860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:42:07.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.899+0000 7f75da59c700 1 -- 172.21.15.142:0/3270531969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75dc19dbe0 msgr2=0x7f75dc197860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:42:07.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.899+0000 7f75da59c700 1 --2- 172.21.15.142:0/3270531969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75dc19dbe0 0x7f75dc197860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.899+0000 7f75da59c700 1 -- 172.21.15.142:0/3270531969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75dc19e220 con 0x7f75c40035c0 2024-05-09T08:42:07.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.899+0000 7f75da59c700 1 --2- 172.21.15.142:0/3270531969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75c40035c0 0x7f75dc19d6a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f75cc010010 tx=0x7f75cc0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:07.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.899+0000 7f75cb7fe700 1 -- 172.21.15.142:0/3270531969 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f75cc004b20 con 0x7f75c40035c0 2024-05-09T08:42:07.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.899+0000 7f75cb7fe700 1 -- 172.21.15.142:0/3270531969 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75cc013e40 con 0x7f75c40035c0 2024-05-09T08:42:07.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.899+0000 7f75cb7fe700 1 -- 172.21.15.142:0/3270531969 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f75cc011b60 con 0x7f75c40035c0 2024-05-09T08:42:07.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.900+0000 7f75e0af1700 1 -- 172.21.15.142:0/3270531969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75dc197da0 con 0x7f75c40035c0 2024-05-09T08:42:07.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.900+0000 7f75e0af1700 1 -- 172.21.15.142:0/3270531969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75dc198330 con 0x7f75c40035c0 2024-05-09T08:42:07.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.900+0000 7f75cb7fe700 1 -- 172.21.15.142:0/3270531969 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f75cc023070 con 0x7f75c40035c0 2024-05-09T08:42:07.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.901+0000 7f75cb7fe700 1 --2- 172.21.15.142:0/3270531969 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f75bc038e90 0x7f75bc03b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.901+0000 7f75e0af1700 1 -- 172.21.15.142:0/3270531969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75dc06ae10 con 0x7f75c40035c0 2024-05-09T08:42:07.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.901+0000 7f75cb7fe700 1 -- 172.21.15.142:0/3270531969 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f75cc020050 con 0x7f75c40035c0 2024-05-09T08:42:07.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.901+0000 7f75d9d9b700 1 --2- 172.21.15.142:0/3270531969 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f75bc038e90 0x7f75bc03b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.901+0000 7f75d9d9b700 1 --2- 172.21.15.142:0/3270531969 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f75bc038e90 0x7f75bc03b350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009180 tx=0x7f75d0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:07.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:07.907+0000 7f75cb7fe700 1 -- 172.21.15.142:0/3270531969 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f75cc028020 con 0x7f75c40035c0 2024-05-09T08:42:08.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:07 smithi098 ceph-mon[30092]: Deploying daemon mon.b on smithi105 2024-05-09T08:42:08.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:07 smithi098 ceph-mon[30092]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:08.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:07 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:08.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:07 smithi142 ceph-mon[34625]: Deploying daemon mon.b on smithi105 2024-05-09T08:42:08.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:07 smithi142 ceph-mon[34625]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:08.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:07 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:08.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:08.187+0000 7f75e0af1700 1 -- 172.21.15.142:0/3270531969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f75dc198d90 con 0x7f75c40035c0 2024-05-09T08:42:08.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:08.188+0000 7f75cb7fe700 1 -- 172.21.15.142:0/3270531969 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f75cc01a550 con 0x7f75c40035c0 2024-05-09T08:42:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:42:08.187 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":2,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","modified":"2024-05-09T08:42:01.668574Z","created":"2024-05-09T08:40:14.468055Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-09T08:42:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:08.190+0000 7f75e0af1700 1 -- 172.21.15.142:0/3270531969 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f75bc038e90 msgr2=0x7f75bc03b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:08.190+0000 7f75e0af1700 1 --2- 172.21.15.142:0/3270531969 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f75bc038e90 0x7f75bc03b350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009180 tx=0x7f75d0008040 comp rx=0 tx=0).stop 2024-05-09T08:42:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:08.190+0000 7f75e0af1700 1 -- 172.21.15.142:0/3270531969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75c40035c0 msgr2=0x7f75dc19d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:08.190+0000 7f75e0af1700 1 --2- 172.21.15.142:0/3270531969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75c40035c0 0x7f75dc19d6a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f75cc010010 tx=0x7f75cc0055a0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:08.190+0000 7f75e0af1700 1 -- 172.21.15.142:0/3270531969 shutdown_connections 2024-05-09T08:42:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:08.190+0000 7f75e0af1700 1 --2- 172.21.15.142:0/3270531969 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f75bc038e90 0x7f75bc03b350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:08.190+0000 7f75e0af1700 1 --2- 172.21.15.142:0/3270531969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75c40035c0 0x7f75dc19d6a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:08.190+0000 7f75e0af1700 1 --2- 172.21.15.142:0/3270531969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75dc19dbe0 0x7f75dc197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:08.190+0000 7f75e0af1700 1 -- 172.21.15.142:0/3270531969 >> 172.21.15.142:0/3270531969 conn(0x7f75dc0fc820 msgr2=0x7f75dc0fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:08.190+0000 7f75e0af1700 1 -- 172.21.15.142:0/3270531969 shutdown_connections 2024-05-09T08:42:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:08.190+0000 7f75e0af1700 1 -- 172.21.15.142:0/3270531969 wait complete. 2024-05-09T08:42:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:dumped monmap epoch 2 2024-05-09T08:42:08.781 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:08 smithi142 ceph-mon[34625]: from='client.? 172.21.15.142:0/3270531969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T08:42:09.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:08 smithi098 ceph-mon[30092]: from='client.? 172.21.15.142:0/3270531969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T08:42:09.810 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-09T08:42:09.810 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-09T08:42:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:42:10.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:09 smithi098 ceph-mon[30092]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:10.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:09 smithi142 ceph-mon[34625]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:10.616 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:10 smithi105 ceph-mon[34498]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-09T08:42:10.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:10 smithi105 bash[34478]: 085dc4d67b8c2e9bd519efca9aad2a29ac53f7798167dd857e419d4d236cf23e 2024-05-09T08:42:10.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:10 smithi105 systemd[1]: Started Ceph mon.b for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:42:15.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.769+0000 7f0317142700 1 -- 172.21.15.142:0/2735554929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0310100f70 msgr2=0x7f02fc005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:15.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.769+0000 7f0317142700 1 --2- 172.21.15.142:0/2735554929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0310100f70 0x7f02fc005610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0308009c10 tx=0x7f0308009f20 comp rx=0 tx=0).stop 2024-05-09T08:42:15.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.769+0000 7f0317142700 1 -- 172.21.15.142:0/2735554929 shutdown_connections 2024-05-09T08:42:15.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.769+0000 7f0317142700 1 --2- 172.21.15.142:0/2735554929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0310100f70 0x7f02fc005610 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:15.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.769+0000 7f0317142700 1 -- 172.21.15.142:0/2735554929 >> 172.21.15.142:0/2735554929 conn(0x7f03100fc800 msgr2=0x7f03100fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:15.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.769+0000 7f0317142700 1 -- 172.21.15.142:0/2735554929 shutdown_connections 2024-05-09T08:42:15.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.769+0000 7f0317142700 1 -- 172.21.15.142:0/2735554929 wait complete. 2024-05-09T08:42:15.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.770+0000 7f0317142700 1 Processor -- start 2024-05-09T08:42:15.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.770+0000 7f0317142700 1 -- start start 2024-05-09T08:42:15.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.770+0000 7f0317142700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0310100f70 0x7f0310102a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:15.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.770+0000 7f0317142700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0310102f50 0x7f03101033d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:15.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.770+0000 7f0317142700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03101a3c40 0x7f03101a6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:15.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.770+0000 7f0317142700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0310108d50 con 0x7f0310100f70 2024-05-09T08:42:15.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.770+0000 7f0317142700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0310108bd0 con 0x7f03101a3c40 2024-05-09T08:42:15.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.770+0000 7f0317142700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0310108ed0 con 0x7f0310102f50 2024-05-09T08:42:15.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f030ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0310102f50 0x7f03101033d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:15.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f0314ede700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0310100f70 0x7f0310102a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:15.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f030ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0310102f50 0x7f03101033d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43088/0 (socket says 172.21.15.142:43088) 2024-05-09T08:42:15.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f0314ede700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0310100f70 0x7f0310102a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.142:36782/0 (socket says 172.21.15.142:36782) 2024-05-09T08:42:15.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f030ffff700 1 -- 172.21.15.142:0/2046987907 learned_addr learned my addr 172.21.15.142:0/2046987907 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-05-09T08:42:15.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f030ffff700 1 -- 172.21.15.142:0/2046987907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0310102f50 msgr2=0x7f03101033d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f030ffff700 1 -- 172.21.15.142:0/2046987907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0310102f50 msgr2=0x7f03101033d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f030ffff700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0310102f50 0x7f03101033d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f03156df700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03101a3c40 0x7f03101a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f030ffff700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0310102f50 0x7f03101033d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f0314ede700 1 -- 172.21.15.142:0/2046987907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0310102f50 msgr2=0x7f03101033d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f0314ede700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0310102f50 0x7f03101033d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f0314ede700 1 -- 172.21.15.142:0/2046987907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03101a3c40 msgr2=0x7f03101a6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f0314ede700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03101a3c40 0x7f03101a6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f0314ede700 1 -- 172.21.15.142:0/2046987907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03101a6760 con 0x7f0310100f70 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f0314ede700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0310100f70 0x7f0310102a10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0308000f80 tx=0x7f030801a720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f030dffb700 1 -- 172.21.15.142:0/2046987907 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0308003a00 con 0x7f0310100f70 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f030dffb700 1 -- 172.21.15.142:0/2046987907 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0308026070 con 0x7f0310100f70 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f0317142700 1 -- 172.21.15.142:0/2046987907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03101a69f0 con 0x7f0310100f70 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.771+0000 7f030dffb700 1 -- 172.21.15.142:0/2046987907 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0308022680 con 0x7f0310100f70 2024-05-09T08:42:15.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.772+0000 7f0317142700 1 -- 172.21.15.142:0/2046987907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03101a6f80 con 0x7f0310100f70 2024-05-09T08:42:15.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.773+0000 7f030dffb700 1 -- 172.21.15.142:0/2046987907 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f030801ad10 con 0x7f0310100f70 2024-05-09T08:42:15.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.773+0000 7f030dffb700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f02f0039580 0x7f02f003ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:15.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.773+0000 7f030dffb700 1 -- 172.21.15.142:0/2046987907 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f030804e560 con 0x7f0310100f70 2024-05-09T08:42:15.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.773+0000 7f0317142700 1 -- 172.21.15.142:0/2046987907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0310063980 con 0x7f0310100f70 2024-05-09T08:42:15.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.773+0000 7f030ffff700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f02f0039580 0x7f02f003ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:15.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.774+0000 7f030ffff700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f02f0039580 0x7f02f003ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009180 tx=0x7f02f8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:15.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:15.779+0000 7f030dffb700 1 -- 172.21.15.142:0/2046987907 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f03080098a0 con 0x7f0310100f70 2024-05-09T08:42:15.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: mon.a calling monitor election 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: mon.c calling monitor election 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: mon.b calling monitor election 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-09T08:42:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: monmap epoch 3 2024-05-09T08:42:15.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: fsid a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:42:15.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: last_changed 2024-05-09T08:42:10.598067+0000 2024-05-09T08:42:15.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: created 2024-05-09T08:40:14.468055+0000 2024-05-09T08:42:15.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: min_mon_release 19 (squid) 2024-05-09T08:42:15.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: election_strategy: 1 2024-05-09T08:42:15.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-05-09T08:42:15.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: 1: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.c 2024-05-09T08:42:15.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: 2: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.b 2024-05-09T08:42:15.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: fsmap 2024-05-09T08:42:15.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:42:15.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: mgrmap e13: x(active, since 58s) 2024-05-09T08:42:15.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: overall HEALTH_OK 2024-05-09T08:42:15.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:15.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:16.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:16.071+0000 7f0317142700 1 -- 172.21.15.142:0/2046987907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f03101a7180 con 0x7f0310100f70 2024-05-09T08:42:16.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:16.071+0000 7f030dffb700 1 -- 172.21.15.142:0/2046987907 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7f0308003ba0 con 0x7f0310100f70 2024-05-09T08:42:16.071 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-05-09T08:42:16.071 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":3,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","modified":"2024-05-09T08:42:10.598067Z","created":"2024-05-09T08:40:14.468055Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-05-09T08:42:16.071 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:42:16.071 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: mon.a calling monitor election 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: mon.c calling monitor election 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: mon.b calling monitor election 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: monmap epoch 3 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: fsid a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:42:16.072 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: last_changed 2024-05-09T08:42:10.598067+0000 2024-05-09T08:42:16.073 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: created 2024-05-09T08:40:14.468055+0000 2024-05-09T08:42:16.073 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: min_mon_release 19 (squid) 2024-05-09T08:42:16.073 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: election_strategy: 1 2024-05-09T08:42:16.073 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-05-09T08:42:16.073 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: 1: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.c 2024-05-09T08:42:16.073 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: 2: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.b 2024-05-09T08:42:16.073 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: fsmap 2024-05-09T08:42:16.073 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:42:16.073 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: mgrmap e13: x(active, since 58s) 2024-05-09T08:42:16.073 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: overall HEALTH_OK 2024-05-09T08:42:16.073 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:16.073 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:15 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:16.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:16.074+0000 7f0317142700 1 -- 172.21.15.142:0/2046987907 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f02f0039580 msgr2=0x7f02f003ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:16.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:16.074+0000 7f0317142700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f02f0039580 0x7f02f003ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009180 tx=0x7f02f8008040 comp rx=0 tx=0).stop 2024-05-09T08:42:16.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:16.074+0000 7f0317142700 1 -- 172.21.15.142:0/2046987907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0310100f70 msgr2=0x7f0310102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:16.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:16.074+0000 7f0317142700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0310100f70 0x7f0310102a10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0308000f80 tx=0x7f030801a720 comp rx=0 tx=0).stop 2024-05-09T08:42:16.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:16.074+0000 7f0317142700 1 -- 172.21.15.142:0/2046987907 shutdown_connections 2024-05-09T08:42:16.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:16.074+0000 7f0317142700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f02f0039580 0x7f02f003ba40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:16.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:16.074+0000 7f0317142700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03101a3c40 0x7f03101a6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:16.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:16.074+0000 7f0317142700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0310100f70 0x7f0310102a10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:16.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:16.074+0000 7f0317142700 1 --2- 172.21.15.142:0/2046987907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0310102f50 0x7f03101033d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:16.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:16.074+0000 7f0317142700 1 -- 172.21.15.142:0/2046987907 >> 172.21.15.142:0/2046987907 conn(0x7f03100fc800 msgr2=0x7f03100fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:16.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:16.074+0000 7f0317142700 1 -- 172.21.15.142:0/2046987907 shutdown_connections 2024-05-09T08:42:16.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:16.074+0000 7f0317142700 1 -- 172.21.15.142:0/2046987907 wait complete. 2024-05-09T08:42:16.075 INFO:teuthology.orchestra.run.smithi142.stderr:dumped monmap epoch 3 2024-05-09T08:42:16.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:42:16.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: mon.a calling monitor election 2024-05-09T08:42:16.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:16.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:42:16.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: mon.c calling monitor election 2024-05-09T08:42:16.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:16.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:16.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:16.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: mon.b calling monitor election 2024-05-09T08:42:16.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:16.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:16.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:16.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: monmap epoch 3 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: fsid a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: last_changed 2024-05-09T08:42:10.598067+0000 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: created 2024-05-09T08:40:14.468055+0000 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: min_mon_release 19 (squid) 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: election_strategy: 1 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: 1: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.c 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: 2: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.b 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: fsmap 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: mgrmap e13: x(active, since 58s) 2024-05-09T08:42:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: overall HEALTH_OK 2024-05-09T08:42:16.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:16.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:16.547 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-09T08:42:16.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph config generate-minimal-conf 2024-05-09T08:42:16.809 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:42:16.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:16 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:16.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:16 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:16.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:16 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:16.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:16 smithi098 ceph-mon[30092]: from='client.? 172.21.15.142:0/2046987907' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T08:42:16.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:16 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:16.940 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:16.941 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:16.941 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:16.941 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:16 smithi142 ceph-mon[34625]: from='client.? 172.21.15.142:0/2046987907' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T08:42:16.941 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:16.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:16 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:16.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:16 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:16.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:16 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:16.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:16 smithi105 ceph-mon[34498]: from='client.? 172.21.15.142:0/2046987907' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T08:42:16.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:16 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:42:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: Updating smithi098:/etc/ceph/ceph.conf 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: Updating smithi142:/etc/ceph/ceph.conf 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:42:18.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:42:18.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: Updating smithi098:/etc/ceph/ceph.conf 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: Updating smithi142:/etc/ceph/ceph.conf 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:42:18.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:42:18.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.475+0000 7efddbd39700 1 -- 172.21.15.98:0/739426432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd4100ea0 msgr2=0x7efdd4101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.475+0000 7efddbd39700 1 --2- 172.21.15.98:0/739426432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd4100ea0 0x7efdd4101280 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009910 tx=0x7efdd0009c20 comp rx=0 tx=0).stop 2024-05-09T08:42:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.475+0000 7efddbd39700 1 -- 172.21.15.98:0/739426432 shutdown_connections 2024-05-09T08:42:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.475+0000 7efddbd39700 1 --2- 172.21.15.98:0/739426432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd4100ea0 0x7efdd4101280 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.475+0000 7efddbd39700 1 -- 172.21.15.98:0/739426432 >> 172.21.15.98:0/739426432 conn(0x7efdd40fc770 msgr2=0x7efdd40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.475+0000 7efddbd39700 1 -- 172.21.15.98:0/739426432 shutdown_connections 2024-05-09T08:42:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.475+0000 7efddbd39700 1 -- 172.21.15.98:0/739426432 wait complete. 2024-05-09T08:42:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.476+0000 7efddbd39700 1 Processor -- start 2024-05-09T08:42:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.476+0000 7efddbd39700 1 -- start start 2024-05-09T08:42:18.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.476+0000 7efddbd39700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd4100ea0 0x7efdd419d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:18.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.476+0000 7efddbd39700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd419dc80 0x7efdd41979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:18.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.476+0000 7efddbd39700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdd4197ee0 0x7efdd4198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:18.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.476+0000 7efddbd39700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdd410b230 con 0x7efdd419dc80 2024-05-09T08:42:18.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.476+0000 7efddbd39700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efdd410b0b0 con 0x7efdd4197ee0 2024-05-09T08:42:18.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.476+0000 7efddbd39700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efdd410b3b0 con 0x7efdd4100ea0 2024-05-09T08:42:18.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.476+0000 7efdd92d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd419dc80 0x7efdd41979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:18.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.476+0000 7efdd92d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd419dc80 0x7efdd41979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57868/0 (socket says 172.21.15.98:57868) 2024-05-09T08:42:18.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.476+0000 7efdd92d4700 1 -- 172.21.15.98:0/2317618658 learned_addr learned my addr 172.21.15.98:0/2317618658 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:42:18.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.476+0000 7efdd9ad5700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd4100ea0 0x7efdd419d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:18.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.476+0000 7efdda2d6700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdd4197ee0 0x7efdd4198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:18.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.477+0000 7efdd92d4700 1 -- 172.21.15.98:0/2317618658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd4100ea0 msgr2=0x7efdd419d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:18.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.477+0000 7efdd92d4700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd4100ea0 0x7efdd419d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:18.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.477+0000 7efdd92d4700 1 -- 172.21.15.98:0/2317618658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdd4197ee0 msgr2=0x7efdd4198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:18.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.477+0000 7efdd92d4700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdd4197ee0 0x7efdd4198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:18.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.477+0000 7efdd92d4700 1 -- 172.21.15.98:0/2317618658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd41a4610 con 0x7efdd419dc80 2024-05-09T08:42:18.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.477+0000 7efdda2d6700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdd4197ee0 0x7efdd4198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:18.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.477+0000 7efdd92d4700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd419dc80 0x7efdd41979a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efdc400b780 tx=0x7efdc400ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:18.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.477+0000 7efdcaffd700 1 -- 172.21.15.98:0/2317618658 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc4004230 con 0x7efdd419dc80 2024-05-09T08:42:18.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.477+0000 7efdcaffd700 1 -- 172.21.15.98:0/2317618658 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc40043d0 con 0x7efdd419dc80 2024-05-09T08:42:18.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.477+0000 7efdcaffd700 1 -- 172.21.15.98:0/2317618658 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc4011c90 con 0x7efdd419dc80 2024-05-09T08:42:18.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.477+0000 7efddbd39700 1 -- 172.21.15.98:0/2317618658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd41a4840 con 0x7efdd419dc80 2024-05-09T08:42:18.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.477+0000 7efddbd39700 1 -- 172.21.15.98:0/2317618658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd41a4d80 con 0x7efdd419dc80 2024-05-09T08:42:18.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.478+0000 7efdcaffd700 1 -- 172.21.15.98:0/2317618658 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7efdc401a400 con 0x7efdd419dc80 2024-05-09T08:42:18.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.478+0000 7efdcaffd700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7efdc0039580 0x7efdc003ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:18.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.478+0000 7efdcaffd700 1 -- 172.21.15.98:0/2317618658 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7efdc404bd40 con 0x7efdd419dc80 2024-05-09T08:42:18.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.478+0000 7efddbd39700 1 -- 172.21.15.98:0/2317618658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdd4063980 con 0x7efdd419dc80 2024-05-09T08:42:18.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.478+0000 7efdd9ad5700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7efdc0039580 0x7efdc003ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:18.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.479+0000 7efdd9ad5700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7efdc0039580 0x7efdc003ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009750 tx=0x7efdd000b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:18.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.483+0000 7efdcaffd700 1 -- 172.21.15.98:0/2317618658 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efdc4020500 con 0x7efdd419dc80 2024-05-09T08:42:18.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:18.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: Updating smithi098:/etc/ceph/ceph.conf 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: Updating smithi142:/etc/ceph/ceph.conf 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:42:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:42:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:18.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.704+0000 7efddbd39700 1 -- 172.21.15.98:0/2317618658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7efdd41a50a0 con 0x7efdd419dc80 2024-05-09T08:42:18.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.704+0000 7efdcaffd700 1 -- 172.21.15.98:0/2317618658 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7efdc40202a0 con 0x7efdd419dc80 2024-05-09T08:42:18.703 INFO:teuthology.orchestra.run.smithi098.stdout:# minimal ceph.conf for a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:42:18.703 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-05-09T08:42:18.703 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:42:18.704 INFO:teuthology.orchestra.run.smithi098.stdout: mon_host = [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-05-09T08:42:18.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.706+0000 7efddbd39700 1 -- 172.21.15.98:0/2317618658 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7efdc0039580 msgr2=0x7efdc003ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.706+0000 7efddbd39700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7efdc0039580 0x7efdc003ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009750 tx=0x7efdd000b540 comp rx=0 tx=0).stop 2024-05-09T08:42:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.706+0000 7efddbd39700 1 -- 172.21.15.98:0/2317618658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd419dc80 msgr2=0x7efdd41979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.706+0000 7efddbd39700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd419dc80 0x7efdd41979a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efdc400b780 tx=0x7efdc400ba90 comp rx=0 tx=0).stop 2024-05-09T08:42:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.706+0000 7efddbd39700 1 -- 172.21.15.98:0/2317618658 shutdown_connections 2024-05-09T08:42:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.706+0000 7efddbd39700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7efdc0039580 0x7efdc003ba40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.706+0000 7efddbd39700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdd4197ee0 0x7efdd4198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.706+0000 7efddbd39700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd4100ea0 0x7efdd419d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.706+0000 7efddbd39700 1 --2- 172.21.15.98:0/2317618658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd419dc80 0x7efdd41979a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.706+0000 7efddbd39700 1 -- 172.21.15.98:0/2317618658 >> 172.21.15.98:0/2317618658 conn(0x7efdd40fc770 msgr2=0x7efdd406a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.706+0000 7efddbd39700 1 -- 172.21.15.98:0/2317618658 shutdown_connections 2024-05-09T08:42:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:18.706+0000 7efddbd39700 1 -- 172.21.15.98:0/2317618658 wait complete. 2024-05-09T08:42:19.268 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:19 smithi098 ceph-mon[30092]: Updating smithi098:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:42:19.268 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:19 smithi098 ceph-mon[30092]: Updating smithi142:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:42:19.268 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:19 smithi098 ceph-mon[30092]: Updating smithi105:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:42:19.268 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:19 smithi098 ceph-mon[30092]: Reconfiguring mon.a (unknown last config time)... 2024-05-09T08:42:19.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:19 smithi098 ceph-mon[30092]: Reconfiguring daemon mon.a on smithi098 2024-05-09T08:42:19.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:19 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2317618658' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:19.296 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-09T08:42:19.296 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:42:19.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-05-09T08:42:19.324 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:42:19.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:42:19.389 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-09T08:42:19.389 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.conf 2024-05-09T08:42:19.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:19 smithi142 ceph-mon[34625]: Updating smithi098:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:42:19.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:19 smithi142 ceph-mon[34625]: Updating smithi142:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:42:19.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:19 smithi142 ceph-mon[34625]: Updating smithi105:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:42:19.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:19 smithi142 ceph-mon[34625]: Reconfiguring mon.a (unknown last config time)... 2024-05-09T08:42:19.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:19 smithi142 ceph-mon[34625]: Reconfiguring daemon mon.a on smithi098 2024-05-09T08:42:19.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:19 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2317618658' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:19.415 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:19 smithi105 ceph-mon[34498]: Updating smithi098:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:42:19.415 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:19 smithi105 ceph-mon[34498]: Updating smithi142:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:42:19.415 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:19 smithi105 ceph-mon[34498]: Updating smithi105:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:42:19.415 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:19 smithi105 ceph-mon[34498]: Reconfiguring mon.a (unknown last config time)... 2024-05-09T08:42:19.415 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:19 smithi105 ceph-mon[34498]: Reconfiguring daemon mon.a on smithi098 2024-05-09T08:42:19.415 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:19 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2317618658' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:19.420 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-09T08:42:19.420 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:42:19.488 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-05-09T08:42:19.488 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.conf 2024-05-09T08:42:19.520 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-05-09T08:42:19.520 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:42:19.587 INFO:tasks.cephadm:Adding mgr.x on smithi098 2024-05-09T08:42:19.587 INFO:tasks.cephadm:Adding mgr.y on smithi105 2024-05-09T08:42:19.587 INFO:tasks.cephadm:Adding mgr.z on smithi142 2024-05-09T08:42:19.588 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch apply mgr '3;smithi098=x;smithi105=y;smithi142=z' 2024-05-09T08:42:19.860 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:42:20.114 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:20 smithi098 ceph-mon[30092]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:20.299 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:20 smithi142 ceph-mon[34625]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:20.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:20 smithi105 ceph-mon[34498]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:21.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.373+0000 7f0403447700 1 -- 172.21.15.142:0/3146338277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc101e60 msgr2=0x7f03fc102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:21.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.373+0000 7f0403447700 1 --2- 172.21.15.142:0/3146338277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc101e60 0x7f03fc102280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f03f8009910 tx=0x7f03f8009c20 comp rx=0 tx=0).stop 2024-05-09T08:42:21.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.373+0000 7f0403447700 1 -- 172.21.15.142:0/3146338277 shutdown_connections 2024-05-09T08:42:21.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.373+0000 7f0403447700 1 --2- 172.21.15.142:0/3146338277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc101e60 0x7f03fc102280 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:21.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.373+0000 7f0403447700 1 -- 172.21.15.142:0/3146338277 >> 172.21.15.142:0/3146338277 conn(0x7f03fc0fd450 msgr2=0x7f03fc0ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:21.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.373+0000 7f0403447700 1 -- 172.21.15.142:0/3146338277 shutdown_connections 2024-05-09T08:42:21.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.373+0000 7f0403447700 1 -- 172.21.15.142:0/3146338277 wait complete. 2024-05-09T08:42:21.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.374+0000 7f0403447700 1 Processor -- start 2024-05-09T08:42:21.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.374+0000 7f0403447700 1 -- start start 2024-05-09T08:42:21.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.374+0000 7f0403447700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc101e60 0x7f03fc1998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:21.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.374+0000 7f0403447700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03fc199e00 0x7f03fc19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:21.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.374+0000 7f0403447700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03fc19a280 0x7f03fc1a1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:21.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.374+0000 7f0403447700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03fc075a60 con 0x7f03fc101e60 2024-05-09T08:42:21.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.374+0000 7f0403447700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f03fc0758e0 con 0x7f03fc19a280 2024-05-09T08:42:21.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.374+0000 7f0403447700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f03fc075be0 con 0x7f03fc199e00 2024-05-09T08:42:21.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.374+0000 7f04009e2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03fc199e00 0x7f03fc19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:21.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.374+0000 7f04009e2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03fc199e00 0x7f03fc19ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52078/0 (socket says 172.21.15.142:52078) 2024-05-09T08:42:21.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.374+0000 7f04009e2700 1 -- 172.21.15.142:0/4281048971 learned_addr learned my addr 172.21.15.142:0/4281048971 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-05-09T08:42:21.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.374+0000 7f04011e3700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc101e60 0x7f03fc1998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:21.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.375+0000 7f04019e4700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03fc19a280 0x7f03fc1a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:21.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.375+0000 7f04009e2700 1 -- 172.21.15.142:0/4281048971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03fc19a280 msgr2=0x7f03fc1a1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:21.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.375+0000 7f04009e2700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03fc19a280 0x7f03fc1a1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:21.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.375+0000 7f04009e2700 1 -- 172.21.15.142:0/4281048971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc101e60 msgr2=0x7f03fc1998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:21.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.375+0000 7f04009e2700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc101e60 0x7f03fc1998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:21.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.375+0000 7f04009e2700 1 -- 172.21.15.142:0/4281048971 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03fc1a1cd0 con 0x7f03fc199e00 2024-05-09T08:42:21.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.375+0000 7f04019e4700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03fc19a280 0x7f03fc1a1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:42:21.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.375+0000 7f04009e2700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03fc199e00 0x7f03fc19ee70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00b780 tx=0x7f03ec00ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:21.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.375+0000 7f03f27fc700 1 -- 172.21.15.142:0/4281048971 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec004230 con 0x7f03fc199e00 2024-05-09T08:42:21.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.375+0000 7f03f27fc700 1 -- 172.21.15.142:0/4281048971 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03ec0043d0 con 0x7f03fc199e00 2024-05-09T08:42:21.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.375+0000 7f0403447700 1 -- 172.21.15.142:0/4281048971 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03fc1a1fc0 con 0x7f03fc199e00 2024-05-09T08:42:21.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.375+0000 7f03f27fc700 1 -- 172.21.15.142:0/4281048971 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec011c90 con 0x7f03fc199e00 2024-05-09T08:42:21.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.375+0000 7f0403447700 1 -- 172.21.15.142:0/4281048971 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03fc1a24f0 con 0x7f03fc199e00 2024-05-09T08:42:21.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.376+0000 7f03f27fc700 1 -- 172.21.15.142:0/4281048971 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f03ec01a400 con 0x7f03fc199e00 2024-05-09T08:42:21.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.376+0000 7f03f27fc700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f03e8039580 0x7f03e803ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:21.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.376+0000 7f03f27fc700 1 -- 172.21.15.142:0/4281048971 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f03ec04be90 con 0x7f03fc199e00 2024-05-09T08:42:21.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.376+0000 7f0403447700 1 -- 172.21.15.142:0/4281048971 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03fc063980 con 0x7f03fc199e00 2024-05-09T08:42:21.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.376+0000 7f04011e3700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f03e8039580 0x7f03e803ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:21.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.377+0000 7f04011e3700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f03e8039580 0x7f03e803ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f03f8005200 tx=0x7f03f801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:21.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.381+0000 7f03f27fc700 1 -- 172.21.15.142:0/4281048971 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f03ec018750 con 0x7f03fc199e00 2024-05-09T08:42:21.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:21 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:21.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:21 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:21.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:21 smithi142 ceph-mon[34625]: Reconfiguring mon.b (monmap changed)... 2024-05-09T08:42:21.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:21 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:42:21.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:21 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:42:21.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:21 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:21.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:21 smithi142 ceph-mon[34625]: Reconfiguring daemon mon.b on smithi105 2024-05-09T08:42:21.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:21 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:21.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:21 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:21.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:21 smithi105 ceph-mon[34498]: Reconfiguring mon.b (monmap changed)... 2024-05-09T08:42:21.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:21 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:42:21.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:21 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:42:21.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:21 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:21.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:21 smithi105 ceph-mon[34498]: Reconfiguring daemon mon.b on smithi105 2024-05-09T08:42:21.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.588+0000 7f0403447700 1 -- 172.21.15.142:0/4281048971 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi105=y;smithi142=z", "target": ["mon-mgr", ""]}) -- 0x7f03fc062820 con 0x7f03e8039580 2024-05-09T08:42:21.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:21 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:21.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:21 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:21.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:21 smithi098 ceph-mon[30092]: Reconfiguring mon.b (monmap changed)... 2024-05-09T08:42:21.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:21 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:42:21.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:21 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:42:21.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:21 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:21.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:21 smithi098 ceph-mon[30092]: Reconfiguring daemon mon.b on smithi105 2024-05-09T08:42:21.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.595+0000 7f03f27fc700 1 -- 172.21.15.142:0/4281048971 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f03fc062820 con 0x7f03e8039580 2024-05-09T08:42:21.595 INFO:teuthology.orchestra.run.smithi142.stdout:Scheduled mgr update... 2024-05-09T08:42:21.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.597+0000 7f0403447700 1 -- 172.21.15.142:0/4281048971 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f03e8039580 msgr2=0x7f03e803ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:21.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.597+0000 7f0403447700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f03e8039580 0x7f03e803ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f03f8005200 tx=0x7f03f801a040 comp rx=0 tx=0).stop 2024-05-09T08:42:21.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.597+0000 7f0403447700 1 -- 172.21.15.142:0/4281048971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03fc199e00 msgr2=0x7f03fc19ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:21.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.597+0000 7f0403447700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03fc199e00 0x7f03fc19ee70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00b780 tx=0x7f03ec00ba90 comp rx=0 tx=0).stop 2024-05-09T08:42:21.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.597+0000 7f0403447700 1 -- 172.21.15.142:0/4281048971 shutdown_connections 2024-05-09T08:42:21.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.597+0000 7f0403447700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f03e8039580 0x7f03e803ba40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:21.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.597+0000 7f0403447700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03fc19a280 0x7f03fc1a1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:21.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.597+0000 7f0403447700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03fc101e60 0x7f03fc1998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:21.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.597+0000 7f0403447700 1 --2- 172.21.15.142:0/4281048971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03fc199e00 0x7f03fc19ee70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:21.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.597+0000 7f0403447700 1 -- 172.21.15.142:0/4281048971 >> 172.21.15.142:0/4281048971 conn(0x7f03fc0fd450 msgr2=0x7f03fc0fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:21.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.598+0000 7f0403447700 1 -- 172.21.15.142:0/4281048971 shutdown_connections 2024-05-09T08:42:21.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:42:21.598+0000 7f0403447700 1 -- 172.21.15.142:0/4281048971 wait complete. 2024-05-09T08:42:22.242 DEBUG:teuthology.orchestra.run.smithi105:mgr.y> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mgr.y.service 2024-05-09T08:42:22.245 DEBUG:teuthology.orchestra.run.smithi142:mgr.z> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mgr.z.service 2024-05-09T08:42:22.247 INFO:tasks.cephadm:Deploying OSDs... 2024-05-09T08:42:22.247 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:42:22.247 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T08:42:22.266 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:22 smithi142 ceph-mon[34625]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:22.266 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:22.267 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:22.267 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:22.267 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:42:22.267 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:42:22.267 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:22.268 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T08:42:22.268 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-05-09T08:42:22.327 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T08:42:22.327 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:42:22.327 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-05-09T08:42:22.327 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:42:22.327 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:42:22.327 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-05-09 08:41:29.609298826 +0000 2024-05-09T08:42:22.327 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-05-09 08:36:22.232942720 +0000 2024-05-09T08:42:22.327 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-05-09 08:36:22.232942720 +0000 2024-05-09T08:42:22.327 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-05-09 08:36:22.232942720 +0000 2024-05-09T08:42:22.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T08:42:22.396 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-05-09T08:42:22.396 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-05-09T08:42:22.396 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000237233 s, 2.2 MB/s 2024-05-09T08:42:22.398 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T08:42:22.457 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-05-09T08:42:22.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:22 smithi105 ceph-mon[34498]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:22.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:22.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:22.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:22.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:42:22.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:42:22.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T08:42:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:42:22.515 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-09T08:42:22.516 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:42:22.516 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:42:22.516 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-05-09 08:41:29.634298516 +0000 2024-05-09T08:42:22.516 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-05-09 08:36:22.572938492 +0000 2024-05-09T08:42:22.516 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-05-09 08:36:22.572938492 +0000 2024-05-09T08:42:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-05-09 08:36:22.572938492 +0000 2024-05-09T08:42:22.516 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T08:42:22.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:22 smithi098 ceph-mon[30092]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:22.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:22.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:22.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:22.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:42:22.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:42:22.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:22.582 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-05-09T08:42:22.582 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-05-09T08:42:22.583 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000217964 s, 2.3 MB/s 2024-05-09T08:42:22.584 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T08:42:22.641 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-05-09T08:42:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T08:42:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:42:22.699 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-09T08:42:22.699 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:42:22.699 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:42:22.699 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-05-09 08:41:29.658298219 +0000 2024-05-09T08:42:22.699 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-05-09 08:36:22.909934302 +0000 2024-05-09T08:42:22.699 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-05-09 08:36:22.909934302 +0000 2024-05-09T08:42:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-05-09 08:36:22.909934302 +0000 2024-05-09T08:42:22.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T08:42:22.764 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-05-09T08:42:22.764 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-05-09T08:42:22.764 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000229705 s, 2.2 MB/s 2024-05-09T08:42:22.765 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T08:42:22.823 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-05-09T08:42:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T08:42:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:42:22.879 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-09T08:42:22.879 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:42:22.879 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:42:22.879 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-05-09 08:41:29.682297921 +0000 2024-05-09T08:42:22.879 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-05-09 08:36:23.243930149 +0000 2024-05-09T08:42:22.879 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-05-09 08:36:23.243930149 +0000 2024-05-09T08:42:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-05-09 08:36:23.243930149 +0000 2024-05-09T08:42:22.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T08:42:22.943 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-05-09T08:42:22.943 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-05-09T08:42:22.943 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000194806 s, 2.6 MB/s 2024-05-09T08:42:22.944 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T08:42:23.001 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-09T08:42:23.001 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T08:42:23.020 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T08:42:23.020 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2024-05-09T08:42:23.078 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T08:42:23.078 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:42:23.078 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-09T08:42:23.078 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:42:23.078 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:42:23.078 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-05-09 08:42:10.054540830 +0000 2024-05-09T08:42:23.078 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-05-09 08:36:33.253004028 +0000 2024-05-09T08:42:23.079 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-05-09 08:36:33.253004028 +0000 2024-05-09T08:42:23.079 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-05-09 08:36:33.253004028 +0000 2024-05-09T08:42:23.079 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T08:42:23.144 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-05-09T08:42:23.144 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-05-09T08:42:23.144 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000285905 s, 1.8 MB/s 2024-05-09T08:42:23.145 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T08:42:23.203 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2024-05-09T08:42:23.259 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T08:42:23.260 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:42:23.260 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-05-09T08:42:23.260 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:42:23.260 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:42:23.260 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-05-09 08:42:10.054540830 +0000 2024-05-09T08:42:23.260 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-05-09 08:36:33.594997208 +0000 2024-05-09T08:42:23.260 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-05-09 08:36:33.594997208 +0000 2024-05-09T08:42:23.260 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-05-09 08:36:33.594997208 +0000 2024-05-09T08:42:23.260 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T08:42:23.323 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-05-09T08:42:23.324 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-05-09T08:42:23.324 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000218785 s, 2.3 MB/s 2024-05-09T08:42:23.325 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T08:42:23.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:23 smithi098 ceph-mon[30092]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi105=y;smithi142=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:42:23.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:23 smithi098 ceph-mon[30092]: Saving service mgr spec with placement smithi098=x;smithi105=y;smithi142=z;count:3 2024-05-09T08:42:23.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:23 smithi098 ceph-mon[30092]: Reconfiguring mon.c (monmap changed)... 2024-05-09T08:42:23.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:23 smithi098 ceph-mon[30092]: Reconfiguring daemon mon.c on smithi142 2024-05-09T08:42:23.382 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2024-05-09T08:42:23.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:23 smithi142 ceph-mon[34625]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi105=y;smithi142=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:42:23.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:23 smithi142 ceph-mon[34625]: Saving service mgr spec with placement smithi098=x;smithi105=y;smithi142=z;count:3 2024-05-09T08:42:23.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:23 smithi142 ceph-mon[34625]: Reconfiguring mon.c (monmap changed)... 2024-05-09T08:42:23.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:23 smithi142 ceph-mon[34625]: Reconfiguring daemon mon.c on smithi142 2024-05-09T08:42:23.437 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T08:42:23.438 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:42:23.438 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-05-09T08:42:23.438 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:42:23.438 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:42:23.438 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-05-09 08:42:10.055540810 +0000 2024-05-09T08:42:23.438 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-05-09 08:36:33.922990666 +0000 2024-05-09T08:42:23.438 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-05-09 08:36:33.922990666 +0000 2024-05-09T08:42:23.438 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-05-09 08:36:33.922990666 +0000 2024-05-09T08:42:23.438 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T08:42:23.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:23 smithi105 ceph-mon[34498]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi105=y;smithi142=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:42:23.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:23 smithi105 ceph-mon[34498]: Saving service mgr spec with placement smithi098=x;smithi105=y;smithi142=z;count:3 2024-05-09T08:42:23.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:23 smithi105 ceph-mon[34498]: Reconfiguring mon.c (monmap changed)... 2024-05-09T08:42:23.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:23 smithi105 ceph-mon[34498]: Reconfiguring daemon mon.c on smithi142 2024-05-09T08:42:23.500 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-05-09T08:42:23.500 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-05-09T08:42:23.501 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000312015 s, 1.6 MB/s 2024-05-09T08:42:23.502 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T08:42:23.558 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2024-05-09T08:42:23.613 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T08:42:23.613 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:42:23.613 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-09T08:42:23.613 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:42:23.614 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:42:23.614 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-05-09 08:42:10.055540810 +0000 2024-05-09T08:42:23.614 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-05-09 08:36:34.256984006 +0000 2024-05-09T08:42:23.614 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-05-09 08:36:34.256984006 +0000 2024-05-09T08:42:23.614 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-05-09 08:36:34.256984006 +0000 2024-05-09T08:42:23.614 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T08:42:23.677 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-05-09T08:42:23.677 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-05-09T08:42:23.677 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.00026272 s, 1.9 MB/s 2024-05-09T08:42:23.678 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T08:42:23.734 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-05-09T08:42:23.734 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T08:42:23.753 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T08:42:23.753 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_1 2024-05-09T08:42:23.811 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T08:42:23.812 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:42:23.812 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-09T08:42:23.812 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:42:23.812 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:42:23.812 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-05-09 08:42:00.733046521 +0000 2024-05-09T08:42:23.812 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-05-09 08:36:41.015351593 +0000 2024-05-09T08:42:23.812 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-05-09 08:36:41.015351593 +0000 2024-05-09T08:42:23.812 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-05-09 08:36:41.015351593 +0000 2024-05-09T08:42:23.812 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T08:42:23.879 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-05-09T08:42:23.879 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-05-09T08:42:23.879 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000259339 s, 2.0 MB/s 2024-05-09T08:42:23.880 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T08:42:23.936 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_2 2024-05-09T08:42:23.992 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T08:42:23.992 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:42:23.992 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-09T08:42:23.992 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:42:23.992 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:42:23.992 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-05-09 08:42:00.733046521 +0000 2024-05-09T08:42:23.992 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-05-09 08:36:41.349345366 +0000 2024-05-09T08:42:23.993 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-05-09 08:36:41.349345366 +0000 2024-05-09T08:42:23.993 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-05-09 08:36:41.349345366 +0000 2024-05-09T08:42:23.993 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T08:42:24.054 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-05-09T08:42:24.054 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-05-09T08:42:24.054 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000207291 s, 2.5 MB/s 2024-05-09T08:42:24.056 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T08:42:24.111 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_3 2024-05-09T08:42:24.166 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T08:42:24.166 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:42:24.166 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-05-09T08:42:24.166 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:42:24.166 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:42:24.166 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-05-09 08:42:00.734046503 +0000 2024-05-09T08:42:24.166 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-05-09 08:36:41.698338858 +0000 2024-05-09T08:42:24.166 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-05-09 08:36:41.698338858 +0000 2024-05-09T08:42:24.166 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-05-09 08:36:41.698338858 +0000 2024-05-09T08:42:24.167 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T08:42:24.228 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-05-09T08:42:24.228 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-05-09T08:42:24.228 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000204741 s, 2.5 MB/s 2024-05-09T08:42:24.230 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T08:42:24.285 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_4 2024-05-09T08:42:24.341 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T08:42:24.341 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:42:24.341 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-09T08:42:24.341 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:42:24.341 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:42:24.341 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-05-09 08:42:00.734046503 +0000 2024-05-09T08:42:24.341 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-05-09 08:36:42.038332519 +0000 2024-05-09T08:42:24.341 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-05-09 08:36:42.038332519 +0000 2024-05-09T08:42:24.341 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-05-09 08:36:42.038332519 +0000 2024-05-09T08:42:24.341 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T08:42:24.404 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-05-09T08:42:24.404 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-05-09T08:42:24.404 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000219383 s, 2.3 MB/s 2024-05-09T08:42:24.405 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T08:42:24.461 INFO:tasks.cephadm:Deploying osd.0 on smithi098 with /dev/vg_nvme/lv_4... 2024-05-09T08:42:24.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-09T08:42:24.507 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:24 smithi142 ceph-mon[34625]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:24.507 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:24.507 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:24.507 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:24.507 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:24.507 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:24.507 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:24.507 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:42:24.507 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:42:24.507 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:42:24.507 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:24.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:24 smithi098 ceph-mon[30092]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:42:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:42:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:42:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:24.695 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:42:24.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:24 smithi105 ceph-mon[34498]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:24.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:24.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:24.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:24.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:24.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:24.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:24.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:42:24.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:42:24.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:42:24.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:25.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:25 smithi098 ceph-mon[30092]: Deploying daemon mgr.z on smithi142 2024-05-09T08:42:25.622 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:25 smithi142 ceph-mon[34625]: Deploying daemon mgr.z on smithi142 2024-05-09T08:42:25.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:25 smithi105 ceph-mon[34498]: Deploying daemon mgr.z on smithi142 2024-05-09T08:42:25.908 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:25 smithi142 podman[36463]: 2024-05-09 08:42:25.873765587 +0000 UTC m=+0.139772786 container create 725df350de80a739fcf38f4d6e7868bd7b5077ceb192783090cd71717b18f50d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-09T08:42:26.238 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:26 smithi142 podman[36463]: 2024-05-09 08:42:26.084443424 +0000 UTC m=+0.350450577 container init 725df350de80a739fcf38f4d6e7868bd7b5077ceb192783090cd71717b18f50d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, RELEASE=squid-96805dc) 2024-05-09T08:42:26.239 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:26 smithi142 podman[36463]: 2024-05-09 08:42:26.08698451 +0000 UTC m=+0.352991665 container start 725df350de80a739fcf38f4d6e7868bd7b5077ceb192783090cd71717b18f50d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:42:26.239 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:26 smithi142 ceph-mgr[36481]: -- 172.21.15.142:0/4054009452 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56208d6925a0 con 0x56208d667000 2024-05-09T08:42:26.239 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:26 smithi142 bash[36463]: 725df350de80a739fcf38f4d6e7868bd7b5077ceb192783090cd71717b18f50d 2024-05-09T08:42:26.239 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:26 smithi142 systemd[1]: Started Ceph mgr.z for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:42:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:26 smithi098 ceph-mon[30092]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:26 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:26 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:26 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:26 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:42:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:26 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:42:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:26 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:42:26.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:26 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:26.510 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-09T08:42:26.510 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-09T08:42:26.510 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-05-09T08:42:26.510 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-05-09T08:42:26.511 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126969 s, 826 MB/s 2024-05-09T08:42:26.511 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 2024-05-09T08:42:26.511 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-05-09T08:42:26.525 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:26 smithi142 ceph-mon[34625]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:26.526 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:26 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:26.526 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:26 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:26.526 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:26 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:26.526 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:26 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:42:26.526 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:26 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:42:26.526 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:26 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:42:26.526 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:26 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:26.526 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:26 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:26.239+0000 7fb766b22200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:42:26.526 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:26 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:26.425+0000 7fb766b22200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:42:26.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:26 smithi105 ceph-mon[34498]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:26.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:26 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:26.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:26 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:26.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:26 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:26.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:26 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:42:26.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:26 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:42:26.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:26 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:42:26.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:26 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:26.834 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:26 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:26.526+0000 7fb766b22200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:42:26.834 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:26 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:26.834+0000 7fb766b22200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:42:27.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-05-09T08:42:27.158 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:26 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:26.919+0000 7fb766b22200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:42:27.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:27 smithi098 ceph-mon[30092]: Deploying daemon mgr.y on smithi105 2024-05-09T08:42:27.348 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:42:27.539 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:27 smithi142 ceph-mon[34625]: Deploying daemon mgr.y on smithi105 2024-05-09T08:42:27.598 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:27 smithi105 ceph-mon[34498]: Deploying daemon mgr.y on smithi105 2024-05-09T08:42:28.236 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:28 smithi105 systemd[1]: Starting Ceph mgr.y for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:42:28.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:28 smithi142 ceph-mon[34625]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:28.408 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:28 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:28.133+0000 7fb766b22200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:42:28.408 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:28 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:28.304+0000 7fb766b22200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:42:28.460 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:28 smithi098 ceph-mon[30092]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:28.544 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:28 smithi105 ceph-mon[34498]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:28.545 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:28 smithi105 podman[35864]: 2024-05-09 08:42:28.187201567 +0000 UTC m=+0.017907404 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:42:28.545 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:28 smithi105 podman[35864]: 2024-05-09 08:42:28.29396643 +0000 UTC m=+0.124672277 container create 0ee24e7f39c9e532db2eac014de2805232f334fa689b914d68ac559653178a6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-09T08:42:28.545 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:28 smithi105 podman[35864]: 2024-05-09 08:42:28.487794367 +0000 UTC m=+0.318500179 container init 0ee24e7f39c9e532db2eac014de2805232f334fa689b914d68ac559653178a6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-09T08:42:28.545 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:28 smithi105 podman[35864]: 2024-05-09 08:42:28.490323807 +0000 UTC m=+0.321029613 container start 0ee24e7f39c9e532db2eac014de2805232f334fa689b914d68ac559653178a6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-09T08:42:28.545 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:28 smithi105 ceph-mgr[35883]: -- 172.21.15.105:0/3308967865 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cc364ec5a0 con 0x55cc364c0800 2024-05-09T08:42:28.545 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:28 smithi105 bash[35864]: 0ee24e7f39c9e532db2eac014de2805232f334fa689b914d68ac559653178a6f 2024-05-09T08:42:28.771 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:28 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:28.479+0000 7fb766b22200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:42:28.771 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:28 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:28.771+0000 7fb766b22200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:42:28.833 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:28 smithi105 systemd[1]: Started Ceph mgr.y for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:42:28.834 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:28 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:28.646+0000 7f38ff3ff200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:42:28.834 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:28 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:28.835+0000 7f38ff3ff200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:42:29.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.020+0000 7f5559f6a700 1 -- 172.21.15.98:0/412528888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55541080b0 msgr2=0x7f555410a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:29.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.020+0000 7f5559f6a700 1 --2- 172.21.15.98:0/412528888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55541080b0 0x7f555410a520 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f554c009990 tx=0x7f554c009ca0 comp rx=0 tx=0).stop 2024-05-09T08:42:29.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.020+0000 7f5559f6a700 1 -- 172.21.15.98:0/412528888 shutdown_connections 2024-05-09T08:42:29.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.020+0000 7f5559f6a700 1 --2- 172.21.15.98:0/412528888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55541080b0 0x7f555410a520 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:29.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.020+0000 7f5559f6a700 1 --2- 172.21.15.98:0/412528888 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5554102e50 0x7f5554105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:29.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.020+0000 7f5559f6a700 1 --2- 172.21.15.98:0/412528888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5554105780 0x7f5554107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:29.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.020+0000 7f5559f6a700 1 -- 172.21.15.98:0/412528888 >> 172.21.15.98:0/412528888 conn(0x7f55540fc880 msgr2=0x7f55540feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:29.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.020+0000 7f5559f6a700 1 -- 172.21.15.98:0/412528888 shutdown_connections 2024-05-09T08:42:29.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.020+0000 7f5559f6a700 1 -- 172.21.15.98:0/412528888 wait complete. 2024-05-09T08:42:29.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.021+0000 7f5559f6a700 1 Processor -- start 2024-05-09T08:42:29.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.021+0000 7f5559f6a700 1 -- start start 2024-05-09T08:42:29.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.021+0000 7f5559f6a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5554102e50 0x7f55541027b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:29.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.021+0000 7f5559f6a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5554105780 0x7f5554100e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:29.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.021+0000 7f5559f6a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55541080b0 0x7f5554101340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:29.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.021+0000 7f5559f6a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f555410dfd0 con 0x7f5554105780 2024-05-09T08:42:29.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.021+0000 7f5559f6a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f555410de50 con 0x7f5554102e50 2024-05-09T08:42:29.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.021+0000 7f5559f6a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f555410e150 con 0x7f55541080b0 2024-05-09T08:42:29.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.021+0000 7f5552ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5554105780 0x7f5554100e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:29.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.021+0000 7f55537fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5554102e50 0x7f55541027b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:29.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.021+0000 7f5553fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55541080b0 0x7f5554101340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:29.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.021+0000 7f55537fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5554102e50 0x7f55541027b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:43784/0 (socket says 172.21.15.98:43784) 2024-05-09T08:42:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.021+0000 7f5552ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5554105780 0x7f5554100e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48796/0 (socket says 172.21.15.98:48796) 2024-05-09T08:42:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.021+0000 7f55537fe700 1 -- 172.21.15.98:0/3240831118 learned_addr learned my addr 172.21.15.98:0/3240831118 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:42:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.022+0000 7f5552ffd700 1 -- 172.21.15.98:0/3240831118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55541080b0 msgr2=0x7f5554101340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.022+0000 7f5552ffd700 1 --2- 172.21.15.98:0/3240831118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55541080b0 0x7f5554101340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.022+0000 7f5552ffd700 1 -- 172.21.15.98:0/3240831118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5554102e50 msgr2=0x7f55541027b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.022+0000 7f5552ffd700 1 --2- 172.21.15.98:0/3240831118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5554102e50 0x7f55541027b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.022+0000 7f5552ffd700 1 -- 172.21.15.98:0/3240831118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5554101bf0 con 0x7f5554105780 2024-05-09T08:42:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.022+0000 7f5552ffd700 1 --2- 172.21.15.98:0/3240831118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5554105780 0x7f5554100e00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f554800d750 tx=0x7f554800da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.022+0000 7f5550ff9700 1 -- 172.21.15.98:0/3240831118 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5548014030 con 0x7f5554105780 2024-05-09T08:42:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.022+0000 7f5559f6a700 1 -- 172.21.15.98:0/3240831118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55541a2710 con 0x7f5554105780 2024-05-09T08:42:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.022+0000 7f5550ff9700 1 -- 172.21.15.98:0/3240831118 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5548004500 con 0x7f5554105780 2024-05-09T08:42:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.022+0000 7f5559f6a700 1 -- 172.21.15.98:0/3240831118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55541a2c00 con 0x7f5554105780 2024-05-09T08:42:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.022+0000 7f5550ff9700 1 -- 172.21.15.98:0/3240831118 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5548005420 con 0x7f5554105780 2024-05-09T08:42:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.023+0000 7f5550ff9700 1 -- 172.21.15.98:0/3240831118 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f554800f690 con 0x7f5554105780 2024-05-09T08:42:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.023+0000 7f5550ff9700 1 --2- 172.21.15.98:0/3240831118 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f553c039330 0x7f553c03b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.023+0000 7f5559f6a700 1 -- 172.21.15.98:0/3240831118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5554069010 con 0x7f5554105780 2024-05-09T08:42:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.023+0000 7f5550ff9700 1 -- 172.21.15.98:0/3240831118 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f554804bf80 con 0x7f5554105780 2024-05-09T08:42:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.023+0000 7f55537fe700 1 --2- 172.21.15.98:0/3240831118 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f553c039330 0x7f553c03b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.024+0000 7f55537fe700 1 --2- 172.21.15.98:0/3240831118 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f553c039330 0x7f553c03b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f55440094f0 tx=0x7f5544009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:29.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.029+0000 7f5550ff9700 1 -- 172.21.15.98:0/3240831118 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f554800ade0 con 0x7f5554105780 2024-05-09T08:42:29.079 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:28 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:28.868+0000 7fb766b22200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:42:29.080 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:29 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:29.080+0000 7fb766b22200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:42:29.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:29.236+0000 7f5559f6a700 1 -- 172.21.15.98:0/3240831118 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f5554062760 con 0x7f553c039330 2024-05-09T08:42:29.236 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:28 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:28.943+0000 7f38ff3ff200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:42:29.408 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:29 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:29.284+0000 7fb766b22200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:42:29.582 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:29 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:29.276+0000 7f38ff3ff200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:42:29.582 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:29 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:29.385+0000 7f38ff3ff200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:42:29.837 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:29.837 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:29.837 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:29.837 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:29.837 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:29.837 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:29 smithi105 ceph-mon[34498]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:29.837 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:29 smithi105 ceph-mon[34498]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:42:29.838 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:42:29.838 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:42:29.838 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:29.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:29.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:29.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:29.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:29.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:29.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:29 smithi142 ceph-mon[34625]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:29.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:29 smithi142 ceph-mon[34625]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:42:29.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:42:29.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:42:29.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:29.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:29.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:29.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:29.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:29.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:29.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:29 smithi098 ceph-mon[30092]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:29.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:29 smithi098 ceph-mon[30092]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:42:29.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:42:29.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:42:29.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:30.352 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:29 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:29.921+0000 7fb766b22200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:42:30.649 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:30 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:30.352+0000 7fb766b22200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:42:30.649 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:30 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:30.449+0000 7fb766b22200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:42:30.649 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:30 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:30.649+0000 7fb766b22200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:42:30.967 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:30 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:30.628+0000 7f38ff3ff200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:42:30.967 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:30 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:30.798+0000 7f38ff3ff200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:42:30.967 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:30 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:30.968+0000 7f38ff3ff200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:42:31.057 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.057 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.057 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.057 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.057 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:31.058 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:31.058 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.058 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:30 smithi098 ceph-mon[30092]: Reconfiguring mgr.x (unknown last config time)... 2024-05-09T08:42:31.058 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:42:31.058 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:42:31.058 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:31.058 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:30 smithi098 ceph-mon[30092]: Reconfiguring daemon mgr.x on smithi098 2024-05-09T08:42:31.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:30 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:30 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:30 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:30 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:30 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:31.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:30 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:31.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:30 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:30 smithi105 ceph-mon[34498]: Reconfiguring mgr.x (unknown last config time)... 2024-05-09T08:42:31.238 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:30 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:42:31.238 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:30 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:42:31.238 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:30 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:31.238 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:30 smithi105 ceph-mon[34498]: Reconfiguring daemon mgr.x on smithi098 2024-05-09T08:42:31.266 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:31.174+0000 7fb766b22200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:42:31.266 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:31.266+0000 7fb766b22200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:42:31.266 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:30 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.266 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:30 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.266 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:30 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.267 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:30 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.267 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:30 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:31.267 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:30 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:31.267 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:30 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:31.267 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:30 smithi142 ceph-mon[34625]: Reconfiguring mgr.x (unknown last config time)... 2024-05-09T08:42:31.267 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:30 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:42:31.267 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:30 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:42:31.267 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:30 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:31.267 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:30 smithi142 ceph-mon[34625]: Reconfiguring daemon mgr.x on smithi098 2024-05-09T08:42:31.545 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:31.255+0000 7f38ff3ff200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:42:31.545 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:31.346+0000 7f38ff3ff200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:42:31.545 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:31.546+0000 7f38ff3ff200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:42:31.640 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:31.508+0000 7fb766b22200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:42:31.640 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:31.641+0000 7fb766b22200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:42:31.908 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:31.737+0000 7fb766b22200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:42:31.908 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:31.852+0000 7fb766b22200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:42:31.986 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:31.735+0000 7f38ff3ff200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:42:32.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:31 smithi098 ceph-mon[30092]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:32.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:31 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1716653839' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aaf45c6-e99a-4a52-88f0-b24f52befeef"}]: dispatch 2024-05-09T08:42:32.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:31 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1716653839' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9aaf45c6-e99a-4a52-88f0-b24f52befeef"}]': finished 2024-05-09T08:42:32.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:31 smithi098 ceph-mon[30092]: osdmap e5: 1 total, 0 up, 1 in 2024-05-09T08:42:32.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:31 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:32.352 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:31 smithi105 ceph-mon[34498]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:32.352 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:31 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1716653839' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aaf45c6-e99a-4a52-88f0-b24f52befeef"}]: dispatch 2024-05-09T08:42:32.352 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:31 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1716653839' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9aaf45c6-e99a-4a52-88f0-b24f52befeef"}]': finished 2024-05-09T08:42:32.353 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:31 smithi105 ceph-mon[34498]: osdmap e5: 1 total, 0 up, 1 in 2024-05-09T08:42:32.353 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:31 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:32.408 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:32 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:32.164+0000 7fb766b22200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:42:32.418 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:31 smithi142 ceph-mon[34625]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:32.418 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:31 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1716653839' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aaf45c6-e99a-4a52-88f0-b24f52befeef"}]: dispatch 2024-05-09T08:42:32.418 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:31 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1716653839' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9aaf45c6-e99a-4a52-88f0-b24f52befeef"}]': finished 2024-05-09T08:42:32.418 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:31 smithi142 ceph-mon[34625]: osdmap e5: 1 total, 0 up, 1 in 2024-05-09T08:42:32.418 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:31 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:32.736 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:32 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:32.353+0000 7f38ff3ff200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:42:33.061 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:32 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:32.771+0000 7f38ff3ff200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:42:33.061 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:32 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:32.865+0000 7f38ff3ff200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:42:33.061 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:33.062+0000 7f38ff3ff200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:42:33.158 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:32 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:32.760+0000 7fb766b22200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:42:33.158 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:42:32 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:42:32.974+0000 7fb766b22200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:42:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:33.166+0000 7f5550ff9700 1 -- 172.21.15.98:0/3240831118 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7f5548004750 con 0x7f5554105780 2024-05-09T08:42:33.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:33 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:33.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:33 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2876963678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:42:33.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:33 smithi105 ceph-mon[34498]: Standby manager daemon z started 2024-05-09T08:42:33.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:33 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.142:0/247663962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:42:33.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:33 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.142:0/247663962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:42:33.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:33 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.142:0/247663962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:42:33.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:33 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.142:0/247663962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:42:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:33 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:33 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2876963678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:42:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:33 smithi098 ceph-mon[30092]: Standby manager daemon z started 2024-05-09T08:42:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:33 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.142:0/247663962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:42:33.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:33 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.142:0/247663962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:42:33.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:33 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.142:0/247663962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:42:33.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:33 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.142:0/247663962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:42:33.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:33 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:33.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:33 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2876963678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:42:33.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:33 smithi142 ceph-mon[34625]: Standby manager daemon z started 2024-05-09T08:42:33.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:33 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.142:0/247663962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:42:33.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:33 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.142:0/247663962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:42:33.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:33 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.142:0/247663962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:42:33.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:33 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.142:0/247663962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:42:33.898 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:33.572+0000 7f38ff3ff200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:42:33.898 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:33.662+0000 7f38ff3ff200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:42:34.163 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:33.899+0000 7f38ff3ff200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:42:34.163 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:34 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:34.030+0000 7f38ff3ff200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:42:34.163 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:34 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:34.122+0000 7f38ff3ff200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:42:34.486 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:34 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:34.236+0000 7f38ff3ff200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:42:34.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:34 smithi105 ceph-mon[34498]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:34 smithi105 ceph-mon[34498]: mgrmap e14: x(active, since 76s), standbys: z 2024-05-09T08:42:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:34 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:42:34.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:34 smithi098 ceph-mon[30092]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:34.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:34 smithi098 ceph-mon[30092]: mgrmap e14: x(active, since 76s), standbys: z 2024-05-09T08:42:34.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:34 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:42:34.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:34 smithi142 ceph-mon[34625]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:34.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:34 smithi142 ceph-mon[34625]: mgrmap e14: x(active, since 76s), standbys: z 2024-05-09T08:42:34.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:34 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:42:34.986 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:34 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:34.530+0000 7f38ff3ff200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:42:35.486 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:35 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:35.115+0000 7f38ff3ff200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:42:35.486 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:42:35 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:42:35.326+0000 7f38ff3ff200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:42:36.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:36.178+0000 7f5550ff9700 1 -- 172.21.15.98:0/3240831118 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f55480213a0 con 0x7f5554105780 2024-05-09T08:42:36.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:36 smithi098 ceph-mon[30092]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:36.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:36 smithi098 ceph-mon[30092]: Standby manager daemon y started 2024-05-09T08:42:36.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:36 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.105:0/3123896513' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:42:36.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:36 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.105:0/3123896513' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:42:36.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:36 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.105:0/3123896513' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:42:36.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:36 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.105:0/3123896513' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:42:36.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:36 smithi105 ceph-mon[34498]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:36.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:36 smithi105 ceph-mon[34498]: Standby manager daemon y started 2024-05-09T08:42:36.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:36 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.105:0/3123896513' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:42:36.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:36 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.105:0/3123896513' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:42:36.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:36 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.105:0/3123896513' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:42:36.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:36 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.105:0/3123896513' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:42:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:36 smithi142 ceph-mon[34625]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:36 smithi142 ceph-mon[34625]: Standby manager daemon y started 2024-05-09T08:42:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:36 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.105:0/3123896513' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:42:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:36 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.105:0/3123896513' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:42:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:36 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.105:0/3123896513' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:42:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:36 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.105:0/3123896513' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:42:37.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:37 smithi098 ceph-mon[30092]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-05-09T08:42:37.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:37 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:42:37.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:37 smithi105 ceph-mon[34498]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-05-09T08:42:37.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:37 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:42:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:37 smithi142 ceph-mon[34625]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-05-09T08:42:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:37 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:42:38.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:38 smithi098 ceph-mon[30092]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:38.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:38.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:38.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:38.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:38.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:38.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:38 smithi142 ceph-mon[34625]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:38.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:38 smithi105 ceph-mon[34498]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:38.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:38.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:38.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:38.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:38.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:38.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:40.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:40 smithi105 ceph-mon[34498]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:40.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:40 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-09T08:42:40.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:40 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:40.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:40 smithi098 ceph-mon[30092]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:40.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:40 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-09T08:42:40.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:40 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:40 smithi142 ceph-mon[34625]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:40 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-09T08:42:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:40 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:41.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:41 smithi098 ceph-mon[30092]: Deploying daemon osd.0 on smithi098 2024-05-09T08:42:41.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:41 smithi105 ceph-mon[34498]: Deploying daemon osd.0 on smithi098 2024-05-09T08:42:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:41 smithi142 ceph-mon[34625]: Deploying daemon osd.0 on smithi098 2024-05-09T08:42:42.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:42 smithi105 ceph-mon[34498]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:42.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:42 smithi098 ceph-mon[30092]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:42.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:42 smithi142 ceph-mon[34625]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:44.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:44 smithi098 ceph-mon[30092]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:44.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:44 smithi105 ceph-mon[34498]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:44.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:44 smithi142 ceph-mon[34625]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:46.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:45 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:46.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:45 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:46.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:45 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:46.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:45 smithi105 ceph-mon[34498]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:45 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:45 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:45 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:45 smithi098 ceph-mon[30092]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:46.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:45 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:46.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:45 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:46.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:45 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:46.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:45 smithi142 ceph-mon[34625]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:48.140 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 0 on host 'smithi098' 2024-05-09T08:42:48.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:48.141+0000 7f5550ff9700 1 -- 172.21.15.98:0/3240831118 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5554062760 con 0x7f553c039330 2024-05-09T08:42:48.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:48.144+0000 7f5559f6a700 1 -- 172.21.15.98:0/3240831118 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f553c039330 msgr2=0x7f553c03b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:48.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:48.144+0000 7f5559f6a700 1 --2- 172.21.15.98:0/3240831118 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f553c039330 0x7f553c03b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f55440094f0 tx=0x7f5544009040 comp rx=0 tx=0).stop 2024-05-09T08:42:48.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:48.144+0000 7f5559f6a700 1 -- 172.21.15.98:0/3240831118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5554105780 msgr2=0x7f5554100e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:48.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:48.144+0000 7f5559f6a700 1 --2- 172.21.15.98:0/3240831118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5554105780 0x7f5554100e00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f554800d750 tx=0x7f554800da60 comp rx=0 tx=0).stop 2024-05-09T08:42:48.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:48.144+0000 7f5559f6a700 1 -- 172.21.15.98:0/3240831118 shutdown_connections 2024-05-09T08:42:48.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:48.144+0000 7f5559f6a700 1 --2- 172.21.15.98:0/3240831118 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f553c039330 0x7f553c03b7f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:48.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:48.144+0000 7f5559f6a700 1 --2- 172.21.15.98:0/3240831118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5554102e50 0x7f55541027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:48.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:48.144+0000 7f5559f6a700 1 --2- 172.21.15.98:0/3240831118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55541080b0 0x7f5554101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:48.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:48.144+0000 7f5559f6a700 1 --2- 172.21.15.98:0/3240831118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5554105780 0x7f5554100e00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:48.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:48.144+0000 7f5559f6a700 1 -- 172.21.15.98:0/3240831118 >> 172.21.15.98:0/3240831118 conn(0x7f55540fc880 msgr2=0x7f5554103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:48.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:48.144+0000 7f5559f6a700 1 -- 172.21.15.98:0/3240831118 shutdown_connections 2024-05-09T08:42:48.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:42:48.144+0000 7f5559f6a700 1 -- 172.21.15.98:0/3240831118 wait complete. 2024-05-09T08:42:48.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:48 smithi098 ceph-mon[30092]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:48.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:48 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:48.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:48 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:48.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:48 smithi105 ceph-mon[34498]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:48.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:48 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:48.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:48 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:48.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:48 smithi142 ceph-mon[34625]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:48.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:48 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:48.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:48 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:48.795 DEBUG:teuthology.orchestra.run.smithi098:osd.0> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.0.service 2024-05-09T08:42:48.797 INFO:tasks.cephadm:Deploying osd.1 on smithi105 with /dev/vg_nvme/lv_4... 2024-05-09T08:42:48.797 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-09T08:42:49.038 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:42:49.049 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 08:42:48 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0[44312]: 2024-05-09T08:42:48.848+0000 7fdb58cdc7c0 -1 osd.0 0 log_to_monitors true 2024-05-09T08:42:49.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:49 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:49.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:49 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:49.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:49 smithi098 ceph-mon[30092]: from='osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-09T08:42:49.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:49 smithi098 ceph-mon[30092]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:49.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:49 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:49.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:49 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:49.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:49 smithi105 ceph-mon[34498]: from='osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-09T08:42:49.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:49 smithi105 ceph-mon[34498]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:49.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:49 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:49.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:49 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:49.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:49 smithi142 ceph-mon[34625]: from='osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-09T08:42:49.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:49 smithi142 ceph-mon[34625]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:50.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:50 smithi105 ceph-mon[34498]: from='osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-09T08:42:50.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:50 smithi105 ceph-mon[34498]: osdmap e6: 1 total, 0 up, 1 in 2024-05-09T08:42:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:50 smithi105 ceph-mon[34498]: from='osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-05-09T08:42:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:50.739 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-09T08:42:50.739 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-09T08:42:50.739 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-05-09T08:42:50.739 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-05-09T08:42:50.739 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132841 s, 789 MB/s 2024-05-09T08:42:50.740 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-05-09T08:42:50.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:50 smithi098 ceph-mon[30092]: from='osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-09T08:42:50.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:50 smithi098 ceph-mon[30092]: osdmap e6: 1 total, 0 up, 1 in 2024-05-09T08:42:50.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:50 smithi098 ceph-mon[30092]: from='osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-05-09T08:42:50.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:50.838 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 08:42:50 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0[44312]: 2024-05-09T08:42:50.432+0000 7fdb4e340700 -1 osd.0 0 waiting for initial osdmap 2024-05-09T08:42:50.839 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 08:42:50 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0[44312]: 2024-05-09T08:42:50.435+0000 7fdb4a133700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:42:50.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:50 smithi142 ceph-mon[34625]: from='osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-09T08:42:50.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:50 smithi142 ceph-mon[34625]: osdmap e6: 1 total, 0 up, 1 in 2024-05-09T08:42:50.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:50 smithi142 ceph-mon[34625]: from='osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-05-09T08:42:50.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:51.280 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_4 2024-05-09T08:42:51.527 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:42:51.729 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:51 smithi105 ceph-mon[34498]: from='osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-05-09T08:42:51.729 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:51 smithi105 ceph-mon[34498]: osdmap e7: 1 total, 0 up, 1 in 2024-05-09T08:42:51.729 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:51 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:51.729 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:51 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:51.729 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:51 smithi105 ceph-mon[34498]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:51.797 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:51 smithi098 ceph-mon[30092]: from='osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-05-09T08:42:51.797 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:51 smithi098 ceph-mon[30092]: osdmap e7: 1 total, 0 up, 1 in 2024-05-09T08:42:51.797 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:51 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:51.798 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:51 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:51.798 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:51 smithi098 ceph-mon[30092]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:51.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:51 smithi142 ceph-mon[34625]: from='osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-05-09T08:42:51.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:51 smithi142 ceph-mon[34625]: osdmap e7: 1 total, 0 up, 1 in 2024-05-09T08:42:51.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:51 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:51.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:51 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:51.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:51 smithi142 ceph-mon[34625]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:42:52.700 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: purged_snaps scrub starts 2024-05-09T08:42:52.700 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: purged_snaps scrub ok 2024-05-09T08:42:52.700 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214] boot 2024-05-09T08:42:52.701 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: osdmap e8: 1 total, 1 up, 1 in 2024-05-09T08:42:52.701 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:52.701 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: Detected new or changed devices on smithi098 2024-05-09T08:42:52.701 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:52.701 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:52.701 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:42:52.701 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi098 to 17088M 2024-05-09T08:42:52.701 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:52.701 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:52.701 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:52.701 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:52.701 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:52.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: purged_snaps scrub starts 2024-05-09T08:42:52.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: purged_snaps scrub ok 2024-05-09T08:42:52.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214] boot 2024-05-09T08:42:52.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: osdmap e8: 1 total, 1 up, 1 in 2024-05-09T08:42:52.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:52.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: Detected new or changed devices on smithi098 2024-05-09T08:42:52.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:52.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:52.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:42:52.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi098 to 17088M 2024-05-09T08:42:52.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:52.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:52.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:52.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:52.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:52.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: purged_snaps scrub starts 2024-05-09T08:42:52.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: purged_snaps scrub ok 2024-05-09T08:42:52.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: osd.0 [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214] boot 2024-05-09T08:42:52.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: osdmap e8: 1 total, 1 up, 1 in 2024-05-09T08:42:52.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:42:52.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: Detected new or changed devices on smithi098 2024-05-09T08:42:52.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:52.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:52.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:42:52.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi098 to 17088M 2024-05-09T08:42:52.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:52.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:52.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:52.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:52.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:53.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.032+0000 7f601f39c700 1 -- 172.21.15.105:0/3574124864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6018102470 msgr2=0x7f60181028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:53.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.032+0000 7f601f39c700 1 --2- 172.21.15.105:0/3574124864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6018102470 0x7f60181028d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f60140099c0 tx=0x7f6014009cd0 comp rx=0 tx=0).stop 2024-05-09T08:42:53.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.032+0000 7f601f39c700 1 -- 172.21.15.105:0/3574124864 shutdown_connections 2024-05-09T08:42:53.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.032+0000 7f601f39c700 1 --2- 172.21.15.105:0/3574124864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6018102e10 0x7f60181096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:53.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.032+0000 7f601f39c700 1 --2- 172.21.15.105:0/3574124864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6018101270 0x7f6018101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:53.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.032+0000 7f601f39c700 1 --2- 172.21.15.105:0/3574124864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6018102470 0x7f60181028d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:53.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.032+0000 7f601f39c700 1 -- 172.21.15.105:0/3574124864 >> 172.21.15.105:0/3574124864 conn(0x7f60180fc810 msgr2=0x7f60180fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:53.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.032+0000 7f601f39c700 1 -- 172.21.15.105:0/3574124864 shutdown_connections 2024-05-09T08:42:53.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.032+0000 7f601f39c700 1 -- 172.21.15.105:0/3574124864 wait complete. 2024-05-09T08:42:53.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601f39c700 1 Processor -- start 2024-05-09T08:42:53.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601f39c700 1 -- start start 2024-05-09T08:42:53.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601f39c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6018101270 0x7f601806b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:53.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601f39c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6018102470 0x7f601819d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:53.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601f39c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6018102e10 0x7f601819fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601f39c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f601810cf50 con 0x7f6018101270 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601f39c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f601810cdd0 con 0x7f6018102470 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601f39c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f601810d0d0 con 0x7f6018102e10 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601c937700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6018102470 0x7f601819d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601c937700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6018102470 0x7f601819d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:36828/0 (socket says 172.21.15.105:36828) 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601c937700 1 -- 172.21.15.105:0/2078584786 learned_addr learned my addr 172.21.15.105:0/2078584786 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601d138700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6018101270 0x7f601806b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601d939700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6018102e10 0x7f601819fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601c937700 1 -- 172.21.15.105:0/2078584786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6018102e10 msgr2=0x7f601819fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601c937700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6018102e10 0x7f601819fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601c937700 1 -- 172.21.15.105:0/2078584786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6018101270 msgr2=0x7f601806b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601c937700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6018101270 0x7f601806b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601c937700 1 -- 172.21.15.105:0/2078584786 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60181a0360 con 0x7f6018102470 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601d138700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6018101270 0x7f601806b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.033+0000 7f601d939700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6018102e10 0x7f601819fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:53.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.034+0000 7f601c937700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6018102470 0x7f601819d400 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6014003890 tx=0x7f6014003730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:53.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.034+0000 7f600e7fc700 1 -- 172.21.15.105:0/2078584786 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6014027070 con 0x7f6018102470 2024-05-09T08:42:53.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.034+0000 7f600e7fc700 1 -- 172.21.15.105:0/2078584786 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6014022470 con 0x7f6018102470 2024-05-09T08:42:53.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.034+0000 7f600e7fc700 1 -- 172.21.15.105:0/2078584786 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601400f6c0 con 0x7f6018102470 2024-05-09T08:42:53.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.034+0000 7f601f39c700 1 -- 172.21.15.105:0/2078584786 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60181a05f0 con 0x7f6018102470 2024-05-09T08:42:53.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.034+0000 7f601f39c700 1 -- 172.21.15.105:0/2078584786 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60181a0b20 con 0x7f6018102470 2024-05-09T08:42:53.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.035+0000 7f601f39c700 1 -- 172.21.15.105:0/2078584786 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6018069010 con 0x7f6018102470 2024-05-09T08:42:53.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.036+0000 7f600e7fc700 1 -- 172.21.15.105:0/2078584786 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f601402a070 con 0x7f6018102470 2024-05-09T08:42:53.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.037+0000 7f600e7fc700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f60040a2c10 0x7f60040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:53.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.037+0000 7f600e7fc700 1 -- 172.21.15.105:0/2078584786 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f60140cea80 con 0x7f6018102470 2024-05-09T08:42:53.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.037+0000 7f601d138700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f60040a2c10 0x7f60040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:53.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.038+0000 7f601d138700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f60040a2c10 0x7f60040a50d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f60181022d0 tx=0x7f6008008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:53.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.041+0000 7f600e7fc700 1 -- 172.21.15.105:0/2078584786 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6014031070 con 0x7f6018102470 2024-05-09T08:42:53.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:42:53.242+0000 7f601f39c700 1 -- 172.21.15.105:0/2078584786 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6018062760 con 0x7f60040a2c10 2024-05-09T08:42:53.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:53 smithi105 ceph-mon[34498]: osdmap e9: 1 total, 1 up, 1 in 2024-05-09T08:42:53.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:53.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:53.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:53.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:53 smithi105 ceph-mon[34498]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:42:53.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:53 smithi105 ceph-mon[34498]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:42:53.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:42:53.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:42:53.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:53.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:53 smithi098 ceph-mon[30092]: osdmap e9: 1 total, 1 up, 1 in 2024-05-09T08:42:53.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:53.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:53.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:53.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:53 smithi098 ceph-mon[30092]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:42:53.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:53 smithi098 ceph-mon[30092]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:42:53.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:42:53.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:42:53.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:53.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:53 smithi142 ceph-mon[34625]: osdmap e9: 1 total, 1 up, 1 in 2024-05-09T08:42:53.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:53.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:53.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:42:53.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:53 smithi142 ceph-mon[34625]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:42:53.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:53 smithi142 ceph-mon[34625]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:42:53.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:42:53.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:42:53.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:56.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:56 smithi105 ceph-mon[34498]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:42:56.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:56 smithi105 ceph-mon[34498]: from='client.? 172.21.15.105:0/1952612481' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fec79393-4941-4307-8146-dfbf97c37be3"}]: dispatch 2024-05-09T08:42:56.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:56 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fec79393-4941-4307-8146-dfbf97c37be3"}]: dispatch 2024-05-09T08:42:56.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:56 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fec79393-4941-4307-8146-dfbf97c37be3"}]': finished 2024-05-09T08:42:56.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:56 smithi105 ceph-mon[34498]: osdmap e10: 2 total, 1 up, 2 in 2024-05-09T08:42:56.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:56 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:42:56.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:56 smithi098 ceph-mon[30092]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:42:56.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:56 smithi098 ceph-mon[30092]: from='client.? 172.21.15.105:0/1952612481' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fec79393-4941-4307-8146-dfbf97c37be3"}]: dispatch 2024-05-09T08:42:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:56 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fec79393-4941-4307-8146-dfbf97c37be3"}]: dispatch 2024-05-09T08:42:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:56 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fec79393-4941-4307-8146-dfbf97c37be3"}]': finished 2024-05-09T08:42:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:56 smithi098 ceph-mon[30092]: osdmap e10: 2 total, 1 up, 2 in 2024-05-09T08:42:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:56 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:42:56.611 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:56 smithi142 ceph-mon[34625]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:42:56.611 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:56 smithi142 ceph-mon[34625]: from='client.? 172.21.15.105:0/1952612481' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fec79393-4941-4307-8146-dfbf97c37be3"}]: dispatch 2024-05-09T08:42:56.611 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:56 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fec79393-4941-4307-8146-dfbf97c37be3"}]: dispatch 2024-05-09T08:42:56.611 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:56 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fec79393-4941-4307-8146-dfbf97c37be3"}]': finished 2024-05-09T08:42:56.611 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:56 smithi142 ceph-mon[34625]: osdmap e10: 2 total, 1 up, 2 in 2024-05-09T08:42:56.611 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:56 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:42:57.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:57 smithi105 ceph-mon[34498]: from='client.? 172.21.15.105:0/201274877' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:42:57.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:57 smithi098 ceph-mon[30092]: from='client.? 172.21.15.105:0/201274877' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:42:57.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:57 smithi142 ceph-mon[34625]: from='client.? 172.21.15.105:0/201274877' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:42:58.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:42:58 smithi105 ceph-mon[34498]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:42:58.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:42:58 smithi098 ceph-mon[30092]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:42:58.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:42:58 smithi142 ceph-mon[34625]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:00.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:00 smithi105 ceph-mon[34498]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:00.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:00 smithi098 ceph-mon[30092]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:00 smithi142 ceph-mon[34625]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:02.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:02 smithi105 ceph-mon[34498]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:02.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:02 smithi098 ceph-mon[30092]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:02 smithi142 ceph-mon[34625]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:03.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:03 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-09T08:43:03.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:03 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:03.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:03 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-09T08:43:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:03 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:03 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-09T08:43:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:03 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:04 smithi105 ceph-mon[34498]: Deploying daemon osd.1 on smithi105 2024-05-09T08:43:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:04 smithi105 ceph-mon[34498]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:04.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:04 smithi098 ceph-mon[30092]: Deploying daemon osd.1 on smithi105 2024-05-09T08:43:04.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:04 smithi098 ceph-mon[30092]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:04 smithi142 ceph-mon[34625]: Deploying daemon osd.1 on smithi105 2024-05-09T08:43:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:04 smithi142 ceph-mon[34625]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:06.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:06 smithi105 ceph-mon[34498]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:06.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:06 smithi098 ceph-mon[30092]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:06.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:06 smithi142 ceph-mon[34625]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:08.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:08 smithi105 ceph-mon[34498]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:08.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:08 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:08.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:08 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:08.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:08 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:08.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:08 smithi098 ceph-mon[30092]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:08.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:08 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:08.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:08 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:08.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:08 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:08 smithi142 ceph-mon[34625]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:08 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:08 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:08 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:10.460 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:10 smithi105 ceph-mon[34498]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:10.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:10 smithi098 ceph-mon[30092]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:10.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:10 smithi142 ceph-mon[34625]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:11.462 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:11 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:11.462 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:11 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:11.462 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:11 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:11.462 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:11 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:11.463 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:11 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:11.554 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 1 on host 'smithi105' 2024-05-09T08:43:11.555 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:43:11.555+0000 7f600e7fc700 1 -- 172.21.15.105:0/2078584786 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6018062760 con 0x7f60040a2c10 2024-05-09T08:43:11.556 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:43:11.557+0000 7f601f39c700 1 -- 172.21.15.105:0/2078584786 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f60040a2c10 msgr2=0x7f60040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:11.556 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:43:11.557+0000 7f601f39c700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f60040a2c10 0x7f60040a50d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f60181022d0 tx=0x7f6008008040 comp rx=0 tx=0).stop 2024-05-09T08:43:11.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:43:11.557+0000 7f601f39c700 1 -- 172.21.15.105:0/2078584786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6018102470 msgr2=0x7f601819d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:11.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:43:11.557+0000 7f601f39c700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6018102470 0x7f601819d400 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6014003890 tx=0x7f6014003730 comp rx=0 tx=0).stop 2024-05-09T08:43:11.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:43:11.558+0000 7f601f39c700 1 -- 172.21.15.105:0/2078584786 shutdown_connections 2024-05-09T08:43:11.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:43:11.558+0000 7f601f39c700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f60040a2c10 0x7f60040a50d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:11.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:43:11.558+0000 7f601f39c700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6018102e10 0x7f601819fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:11.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:43:11.558+0000 7f601f39c700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6018101270 0x7f601806b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:11.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:43:11.558+0000 7f601f39c700 1 --2- 172.21.15.105:0/2078584786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6018102470 0x7f601819d400 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:11.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:43:11.558+0000 7f601f39c700 1 -- 172.21.15.105:0/2078584786 >> 172.21.15.105:0/2078584786 conn(0x7f60180fc810 msgr2=0x7f60181057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:11.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:43:11.558+0000 7f601f39c700 1 -- 172.21.15.105:0/2078584786 shutdown_connections 2024-05-09T08:43:11.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:43:11.558+0000 7f601f39c700 1 -- 172.21.15.105:0/2078584786 wait complete. 2024-05-09T08:43:11.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:11 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:11.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:11 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:11.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:11 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:11.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:11 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:11.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:11 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:11.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:11 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:11.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:11 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:11.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:11 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:11.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:11 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:11.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:11 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:12.183 DEBUG:teuthology.orchestra.run.smithi105:osd.1> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.1.service 2024-05-09T08:43:12.185 INFO:tasks.cephadm:Deploying osd.2 on smithi142 with /dev/vg_nvme/lv_4... 2024-05-09T08:43:12.186 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-09T08:43:12.432 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:43:12.464 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:12 smithi105 ceph-mon[34498]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:12.465 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:12 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:12.465 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:12 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:12.465 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:12 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:12.465 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:12 smithi105 ceph-mon[34498]: from='osd.1 [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:43:12.465 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:12 smithi105 ceph-mon[34498]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:43:12.554 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:12 smithi142 ceph-mon[34625]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:12.554 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:12 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:12.554 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:12 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:12.555 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:12 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:12.555 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:12 smithi142 ceph-mon[34625]: from='osd.1 [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:43:12.555 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:12 smithi142 ceph-mon[34625]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:43:12.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:12 smithi098 ceph-mon[30092]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:12.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:12 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:12.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:12 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:12.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:12 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:12.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:12 smithi098 ceph-mon[30092]: from='osd.1 [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:43:12.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:12 smithi098 ceph-mon[30092]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:43:13.701 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:13 smithi142 ceph-mon[34625]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-09T08:43:13.701 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:13 smithi142 ceph-mon[34625]: from='osd.1 [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:43:13.702 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:13 smithi142 ceph-mon[34625]: osdmap e11: 2 total, 1 up, 2 in 2024-05-09T08:43:13.702 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:13 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:13.702 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:13 smithi142 ceph-mon[34625]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:43:13.702 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:13 smithi142 ceph-mon[34625]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:13.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:13 smithi098 ceph-mon[30092]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-09T08:43:13.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:13 smithi098 ceph-mon[30092]: from='osd.1 [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:43:13.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:13 smithi098 ceph-mon[30092]: osdmap e11: 2 total, 1 up, 2 in 2024-05-09T08:43:13.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:13 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:13.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:13 smithi098 ceph-mon[30092]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:43:13.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:13 smithi098 ceph-mon[30092]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:13.981 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:13 smithi105 ceph-mon[34498]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-09T08:43:13.981 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:13 smithi105 ceph-mon[34498]: from='osd.1 [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:43:13.981 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:13 smithi105 ceph-mon[34498]: osdmap e11: 2 total, 1 up, 2 in 2024-05-09T08:43:13.981 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:13 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:13.982 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:13 smithi105 ceph-mon[34498]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:43:13.982 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:13 smithi105 ceph-mon[34498]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:14.237 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 08:43:13 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-1[39079]: 2024-05-09T08:43:13.981+0000 7f97e66bc700 -1 osd.1 0 waiting for initial osdmap 2024-05-09T08:43:14.237 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 08:43:13 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-1[39079]: 2024-05-09T08:43:13.985+0000 7f97ddca6700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:43:14.256 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-09T08:43:14.256 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-09T08:43:14.256 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2024-05-09T08:43:14.256 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2024-05-09T08:43:14.256 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115931 s, 904 MB/s 2024-05-09T08:43:14.256 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2024-05-09T08:43:14.688 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:14 smithi105 ceph-mon[34498]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-09T08:43:14.688 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:14 smithi105 ceph-mon[34498]: osdmap e12: 2 total, 1 up, 2 in 2024-05-09T08:43:14.688 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:14 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:14.688 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:14 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:14.688 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:14 smithi105 ceph-mon[34498]: from='osd.1 ' entity='osd.1' 2024-05-09T08:43:14.688 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:14 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:14.820 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:14 smithi142 ceph-mon[34625]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-09T08:43:14.820 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:14 smithi142 ceph-mon[34625]: osdmap e12: 2 total, 1 up, 2 in 2024-05-09T08:43:14.820 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:14 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:14.820 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:14 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:14.820 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:14 smithi142 ceph-mon[34625]: from='osd.1 ' entity='osd.1' 2024-05-09T08:43:14.820 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:14 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:14.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:14 smithi098 ceph-mon[30092]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-09T08:43:14.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:14 smithi098 ceph-mon[30092]: osdmap e12: 2 total, 1 up, 2 in 2024-05-09T08:43:14.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:14 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:14.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:14 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:14.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:14 smithi098 ceph-mon[30092]: from='osd.1 ' entity='osd.1' 2024-05-09T08:43:14.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:14 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:14.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi142:vg_nvme/lv_4 2024-05-09T08:43:15.091 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:43:16.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: purged_snaps scrub starts 2024-05-09T08:43:16.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: purged_snaps scrub ok 2024-05-09T08:43:16.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: osd.1 [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222] boot 2024-05-09T08:43:16.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: osdmap e13: 2 total, 2 up, 2 in 2024-05-09T08:43:16.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:16.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: Detected new or changed devices on smithi105 2024-05-09T08:43:16.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:16.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:16.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:16.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi105 to 17088M 2024-05-09T08:43:16.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:16.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:16.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:16.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:16.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:16 smithi142 ceph-mon[34625]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:16.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: purged_snaps scrub starts 2024-05-09T08:43:16.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: purged_snaps scrub ok 2024-05-09T08:43:16.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: osd.1 [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222] boot 2024-05-09T08:43:16.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: osdmap e13: 2 total, 2 up, 2 in 2024-05-09T08:43:16.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:16.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: Detected new or changed devices on smithi105 2024-05-09T08:43:16.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:16.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:16.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:16.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi105 to 17088M 2024-05-09T08:43:16.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:16.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:16.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:16.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:16.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:15 smithi098 ceph-mon[30092]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:16.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:15 smithi105 ceph-mon[34498]: purged_snaps scrub starts 2024-05-09T08:43:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:15 smithi105 ceph-mon[34498]: purged_snaps scrub ok 2024-05-09T08:43:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:15 smithi105 ceph-mon[34498]: osd.1 [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222] boot 2024-05-09T08:43:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:15 smithi105 ceph-mon[34498]: osdmap e13: 2 total, 2 up, 2 in 2024-05-09T08:43:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:43:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:15 smithi105 ceph-mon[34498]: Detected new or changed devices on smithi105 2024-05-09T08:43:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:15 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi105 to 17088M 2024-05-09T08:43:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:16 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:16 smithi105 ceph-mon[34498]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:43:16.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.681+0000 7f683afab700 1 -- 172.21.15.142:0/3680122215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6834103f20 msgr2=0x7f683410a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:16.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.681+0000 7f683afab700 1 --2- 172.21.15.142:0/3680122215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6834103f20 0x7f683410a7d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f682c00b0a0 tx=0x7f682c00b3b0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 -- 172.21.15.142:0/3680122215 shutdown_connections 2024-05-09T08:43:16.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 --2- 172.21.15.142:0/3680122215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6834103f20 0x7f683410a7d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 --2- 172.21.15.142:0/3680122215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6834102380 0x7f68341027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 --2- 172.21.15.142:0/3680122215 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6834103580 0x7f68341039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 -- 172.21.15.142:0/3680122215 >> 172.21.15.142:0/3680122215 conn(0x7f68340fd920 msgr2=0x7f68340ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:16.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 -- 172.21.15.142:0/3680122215 shutdown_connections 2024-05-09T08:43:16.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 -- 172.21.15.142:0/3680122215 wait complete. 2024-05-09T08:43:16.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 Processor -- start 2024-05-09T08:43:16.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 -- start start 2024-05-09T08:43:16.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6834102380 0x7f68341984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6834103580 0x7f6834198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6834103f20 0x7f683419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f683410dfc0 con 0x7f6834102380 2024-05-09T08:43:16.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f683410de40 con 0x7f6834103580 2024-05-09T08:43:16.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f683afab700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f683410e140 con 0x7f6834103f20 2024-05-09T08:43:16.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f6839548700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6834103f20 0x7f683419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f6839548700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6834103f20 0x7f683419faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41834/0 (socket says 172.21.15.142:41834) 2024-05-09T08:43:16.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f6839548700 1 -- 172.21.15.142:0/1391772362 learned_addr learned my addr 172.21.15.142:0/1391772362 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-05-09T08:43:16.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.682+0000 7f6838d47700 1 --2- 172.21.15.142:0/1391772362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6834102380 0x7f68341984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.683+0000 7f6833fff700 1 --2- 172.21.15.142:0/1391772362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6834103580 0x7f6834198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.683+0000 7f6839548700 1 -- 172.21.15.142:0/1391772362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6834103580 msgr2=0x7f6834198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:16.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.683+0000 7f6839548700 1 --2- 172.21.15.142:0/1391772362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6834103580 0x7f6834198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.683+0000 7f6839548700 1 -- 172.21.15.142:0/1391772362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6834102380 msgr2=0x7f68341984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:16.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.683+0000 7f6839548700 1 --2- 172.21.15.142:0/1391772362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6834102380 0x7f68341984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.683+0000 7f6839548700 1 -- 172.21.15.142:0/1391772362 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68341a0140 con 0x7f6834103f20 2024-05-09T08:43:16.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.683+0000 7f6839548700 1 --2- 172.21.15.142:0/1391772362 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6834103f20 0x7f683419faa0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f682c00be60 tx=0x7f682c00be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:16.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.683+0000 7f6831ffb700 1 -- 172.21.15.142:0/1391772362 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f682c021020 con 0x7f6834103f20 2024-05-09T08:43:16.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.683+0000 7f6831ffb700 1 -- 172.21.15.142:0/1391772362 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f682c005520 con 0x7f6834103f20 2024-05-09T08:43:16.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.683+0000 7f683afab700 1 -- 172.21.15.142:0/1391772362 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68341a03d0 con 0x7f6834103f20 2024-05-09T08:43:16.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.683+0000 7f6831ffb700 1 -- 172.21.15.142:0/1391772362 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f682c01b970 con 0x7f6834103f20 2024-05-09T08:43:16.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.683+0000 7f683afab700 1 -- 172.21.15.142:0/1391772362 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68341a0950 con 0x7f6834103f20 2024-05-09T08:43:16.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.684+0000 7f683afab700 1 -- 172.21.15.142:0/1391772362 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6834069010 con 0x7f6834103f20 2024-05-09T08:43:16.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.686+0000 7f6831ffb700 1 -- 172.21.15.142:0/1391772362 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f682c024020 con 0x7f6834103f20 2024-05-09T08:43:16.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.686+0000 7f6831ffb700 1 --2- 172.21.15.142:0/1391772362 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f68240a2c10 0x7f68240a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.687+0000 7f6831ffb700 1 -- 172.21.15.142:0/1391772362 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f682c00f020 con 0x7f6834103f20 2024-05-09T08:43:16.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.687+0000 7f6838d47700 1 --2- 172.21.15.142:0/1391772362 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f68240a2c10 0x7f68240a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.687+0000 7f6838d47700 1 --2- 172.21.15.142:0/1391772362 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f68240a2c10 0x7f68240a50d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f68341033e0 tx=0x7f682800a3b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:16.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.690+0000 7f6831ffb700 1 -- 172.21.15.142:0/1391772362 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f682c09d600 con 0x7f6834103f20 2024-05-09T08:43:16.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:16.891+0000 7f683afab700 1 -- 172.21.15.142:0/1391772362 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6834062760 con 0x7f68240a2c10 2024-05-09T08:43:17.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:17 smithi142 ceph-mon[34625]: osdmap e14: 2 total, 2 up, 2 in 2024-05-09T08:43:17.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:43:17.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:43:17.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:17.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:17 smithi098 ceph-mon[30092]: osdmap e14: 2 total, 2 up, 2 in 2024-05-09T08:43:17.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:43:17.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:43:17.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:17.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:17 smithi105 ceph-mon[34498]: osdmap e14: 2 total, 2 up, 2 in 2024-05-09T08:43:17.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:43:17.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:43:17.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:18.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:18 smithi142 ceph-mon[34625]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:43:18.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:18 smithi142 ceph-mon[34625]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:18.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:18 smithi098 ceph-mon[30092]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:43:18.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:18 smithi098 ceph-mon[30092]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:18.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:18 smithi105 ceph-mon[34498]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:43:18.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:18 smithi105 ceph-mon[34498]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:20.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:20 smithi105 ceph-mon[34498]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:20.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:20 smithi105 ceph-mon[34498]: from='client.? 172.21.15.142:0/1325394298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b659273-578f-462c-8089-3cad73e6f8d2"}]: dispatch 2024-05-09T08:43:20.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:20 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b659273-578f-462c-8089-3cad73e6f8d2"}]: dispatch 2024-05-09T08:43:20.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:20 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b659273-578f-462c-8089-3cad73e6f8d2"}]': finished 2024-05-09T08:43:20.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:20 smithi105 ceph-mon[34498]: osdmap e15: 3 total, 2 up, 3 in 2024-05-09T08:43:20.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:20 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:20.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:20 smithi105 ceph-mon[34498]: from='client.? 172.21.15.142:0/4082712232' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:43:20.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:20 smithi098 ceph-mon[30092]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:20 smithi098 ceph-mon[30092]: from='client.? 172.21.15.142:0/1325394298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b659273-578f-462c-8089-3cad73e6f8d2"}]: dispatch 2024-05-09T08:43:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:20 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b659273-578f-462c-8089-3cad73e6f8d2"}]: dispatch 2024-05-09T08:43:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:20 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b659273-578f-462c-8089-3cad73e6f8d2"}]': finished 2024-05-09T08:43:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:20 smithi098 ceph-mon[30092]: osdmap e15: 3 total, 2 up, 3 in 2024-05-09T08:43:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:20 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:20 smithi098 ceph-mon[30092]: from='client.? 172.21.15.142:0/4082712232' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:43:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:20 smithi142 ceph-mon[34625]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:20 smithi142 ceph-mon[34625]: from='client.? 172.21.15.142:0/1325394298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b659273-578f-462c-8089-3cad73e6f8d2"}]: dispatch 2024-05-09T08:43:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:20 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b659273-578f-462c-8089-3cad73e6f8d2"}]: dispatch 2024-05-09T08:43:20.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:20 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b659273-578f-462c-8089-3cad73e6f8d2"}]': finished 2024-05-09T08:43:20.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:20 smithi142 ceph-mon[34625]: osdmap e15: 3 total, 2 up, 3 in 2024-05-09T08:43:20.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:20 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:20.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:20 smithi142 ceph-mon[34625]: from='client.? 172.21.15.142:0/4082712232' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:43:22.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:22 smithi105 ceph-mon[34498]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:22.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:22 smithi098 ceph-mon[30092]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:22 smithi142 ceph-mon[34625]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:24.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:24 smithi142 ceph-mon[34625]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:24.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:24 smithi105 ceph-mon[34498]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:24.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:24 smithi098 ceph-mon[30092]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:26.353 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:26 smithi142 ceph-mon[34625]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:26.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:26 smithi105 ceph-mon[34498]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:26.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:26 smithi098 ceph-mon[30092]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:27.289 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:27 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-09T08:43:27.289 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:27 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:27.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:27 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-09T08:43:27.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:27 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:27.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:27 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-09T08:43:27.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:27 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:28.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:28 smithi105 ceph-mon[34498]: Deploying daemon osd.2 on smithi142 2024-05-09T08:43:28.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:28 smithi105 ceph-mon[34498]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:28.529 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:28 smithi142 ceph-mon[34625]: Deploying daemon osd.2 on smithi142 2024-05-09T08:43:28.529 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:28 smithi142 ceph-mon[34625]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:28.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:28 smithi098 ceph-mon[30092]: Deploying daemon osd.2 on smithi142 2024-05-09T08:43:28.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:28 smithi098 ceph-mon[30092]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:30.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:30 smithi142 ceph-mon[34625]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:30.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:30 smithi105 ceph-mon[34498]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:30.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:30 smithi098 ceph-mon[30092]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:32.335 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:32 smithi142 ceph-mon[34625]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:32.336 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:32 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:32.336 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:32 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:32.336 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:32 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:32.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:32 smithi105 ceph-mon[34498]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:32.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:32 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:32.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:32 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:32.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:32 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:32.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:32 smithi098 ceph-mon[30092]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:32.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:32 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:32.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:32 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:32.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:32 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:34.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:34 smithi105 ceph-mon[34498]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:34.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:34 smithi098 ceph-mon[30092]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:34.645 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:34 smithi142 ceph-mon[34625]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:35.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:35.254+0000 7f6831ffb700 1 -- 172.21.15.142:0/1391772362 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6834062760 con 0x7f68240a2c10 2024-05-09T08:43:35.254 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 2 on host 'smithi142' 2024-05-09T08:43:35.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:35.256+0000 7f683afab700 1 -- 172.21.15.142:0/1391772362 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f68240a2c10 msgr2=0x7f68240a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:35.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:35.256+0000 7f683afab700 1 --2- 172.21.15.142:0/1391772362 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f68240a2c10 0x7f68240a50d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f68341033e0 tx=0x7f682800a3b0 comp rx=0 tx=0).stop 2024-05-09T08:43:35.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:35.257+0000 7f683afab700 1 -- 172.21.15.142:0/1391772362 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6834103f20 msgr2=0x7f683419faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:35.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:35.257+0000 7f683afab700 1 --2- 172.21.15.142:0/1391772362 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6834103f20 0x7f683419faa0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f682c00be60 tx=0x7f682c00be90 comp rx=0 tx=0).stop 2024-05-09T08:43:35.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:35.257+0000 7f683afab700 1 -- 172.21.15.142:0/1391772362 shutdown_connections 2024-05-09T08:43:35.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:35.257+0000 7f683afab700 1 --2- 172.21.15.142:0/1391772362 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f68240a2c10 0x7f68240a50d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:35.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:35.257+0000 7f683afab700 1 --2- 172.21.15.142:0/1391772362 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6834103f20 0x7f683419faa0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:35.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:35.257+0000 7f683afab700 1 --2- 172.21.15.142:0/1391772362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6834102380 0x7f68341984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:35.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:35.257+0000 7f683afab700 1 --2- 172.21.15.142:0/1391772362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6834103580 0x7f6834198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:35.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:35.257+0000 7f683afab700 1 -- 172.21.15.142:0/1391772362 >> 172.21.15.142:0/1391772362 conn(0x7f68340fd920 msgr2=0x7f6834106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:35.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:35.257+0000 7f683afab700 1 -- 172.21.15.142:0/1391772362 shutdown_connections 2024-05-09T08:43:35.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:43:35.257+0000 7f683afab700 1 -- 172.21.15.142:0/1391772362 wait complete. 2024-05-09T08:43:35.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:35 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:35 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:35 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:35.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:35 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:35.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:35 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:35 smithi142 ceph-mon[34625]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:35.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:35 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.543 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:35 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:35.543 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:35 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:35 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:35 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:35 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:35.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:35 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:35.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:35 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:35 smithi105 ceph-mon[34498]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:35.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:35 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:35 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:35.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:35 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:35 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:35 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:35 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:35.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:35 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:35.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:35 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:35 smithi098 ceph-mon[30092]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:35.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:35 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:35 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:35.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:35 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:35.897 DEBUG:teuthology.orchestra.run.smithi142:osd.2> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.2.service 2024-05-09T08:43:35.900 INFO:tasks.cephadm:Deploying osd.3 on smithi098 with /dev/vg_nvme/lv_3... 2024-05-09T08:43:35.900 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-09T08:43:36.149 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:43:36.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:36 smithi098 ceph-mon[30092]: from='osd.2 [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:43:36.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:36 smithi098 ceph-mon[30092]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:43:36.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:36 smithi105 ceph-mon[34498]: from='osd.2 [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:43:36.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:36 smithi105 ceph-mon[34498]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:43:36.770 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:36 smithi142 ceph-mon[34625]: from='osd.2 [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:43:36.770 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:36 smithi142 ceph-mon[34625]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:43:37.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:37 smithi098 ceph-mon[30092]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-09T08:43:37.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:37 smithi098 ceph-mon[30092]: osdmap e16: 3 total, 2 up, 3 in 2024-05-09T08:43:37.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:37 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:37.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:37 smithi098 ceph-mon[30092]: from='osd.2 [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:43:37.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:37 smithi098 ceph-mon[30092]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:43:37.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:37 smithi098 ceph-mon[30092]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:37 smithi142 ceph-mon[34625]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-09T08:43:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:37 smithi142 ceph-mon[34625]: osdmap e16: 3 total, 2 up, 3 in 2024-05-09T08:43:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:37 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:37 smithi142 ceph-mon[34625]: from='osd.2 [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:43:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:37 smithi142 ceph-mon[34625]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:43:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:37 smithi142 ceph-mon[34625]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:37.659 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 08:43:37 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-2[39778]: 2024-05-09T08:43:37.455+0000 7f6d1d40d700 -1 osd.2 0 waiting for initial osdmap 2024-05-09T08:43:37.659 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 08:43:37 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-2[39778]: 2024-05-09T08:43:37.458+0000 7f6d19200700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:43:37.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:37 smithi105 ceph-mon[34498]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-09T08:43:37.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:37 smithi105 ceph-mon[34498]: osdmap e16: 3 total, 2 up, 3 in 2024-05-09T08:43:37.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:37 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:37.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:37 smithi105 ceph-mon[34498]: from='osd.2 [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:43:37.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:37 smithi105 ceph-mon[34498]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:43:37.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:37 smithi105 ceph-mon[34498]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:38.166 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-09T08:43:38.166 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-09T08:43:38.166 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-05-09T08:43:38.166 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-05-09T08:43:38.166 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113282 s, 926 MB/s 2024-05-09T08:43:38.166 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-05-09T08:43:38.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:38 smithi142 ceph-mon[34625]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-05-09T08:43:38.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:38 smithi142 ceph-mon[34625]: osdmap e17: 3 total, 2 up, 3 in 2024-05-09T08:43:38.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:38.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:38.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:38 smithi105 ceph-mon[34498]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-05-09T08:43:38.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:38 smithi105 ceph-mon[34498]: osdmap e17: 3 total, 2 up, 3 in 2024-05-09T08:43:38.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:38.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:38.778 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:38 smithi098 ceph-mon[30092]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-05-09T08:43:38.778 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:38 smithi098 ceph-mon[30092]: osdmap e17: 3 total, 2 up, 3 in 2024-05-09T08:43:38.778 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:38.778 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:38.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-05-09T08:43:39.055 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:43:39.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: purged_snaps scrub starts 2024-05-09T08:43:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: purged_snaps scrub ok 2024-05-09T08:43:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: osd.2 [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557] boot 2024-05-09T08:43:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: osdmap e18: 3 total, 3 up, 3 in 2024-05-09T08:43:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: Detected new or changed devices on smithi142 2024-05-09T08:43:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi142 to 17088M 2024-05-09T08:43:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:39.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:39 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:43:39.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: purged_snaps scrub starts 2024-05-09T08:43:39.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: purged_snaps scrub ok 2024-05-09T08:43:39.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: osd.2 [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557] boot 2024-05-09T08:43:39.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: osdmap e18: 3 total, 3 up, 3 in 2024-05-09T08:43:39.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:39.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: Detected new or changed devices on smithi142 2024-05-09T08:43:39.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:39.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:39.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:39.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi142 to 17088M 2024-05-09T08:43:39.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:39.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:39.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:39.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:39.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:39.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:39 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:43:39.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: purged_snaps scrub starts 2024-05-09T08:43:39.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: purged_snaps scrub ok 2024-05-09T08:43:39.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: osd.2 [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557] boot 2024-05-09T08:43:39.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: osdmap e18: 3 total, 3 up, 3 in 2024-05-09T08:43:39.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:43:39.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: Detected new or changed devices on smithi142 2024-05-09T08:43:39.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:39.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:39.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:39.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi142 to 17088M 2024-05-09T08:43:39.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:39.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:39.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:39.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:39.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:43:39.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:39 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:43:40.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.679+0000 7f81ef234700 1 -- 172.21.15.98:0/2610490721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e8101e80 msgr2=0x7f81e8109720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:40.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.679+0000 7f81ef234700 1 --2- 172.21.15.98:0/2610490721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e8101e80 0x7f81e8109720 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f81e000b0a0 tx=0x7f81e000b3b0 comp rx=0 tx=0).stop 2024-05-09T08:43:40.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.679+0000 7f81ef234700 1 -- 172.21.15.98:0/2610490721 shutdown_connections 2024-05-09T08:43:40.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.679+0000 7f81ef234700 1 --2- 172.21.15.98:0/2610490721 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e81028f0 0x7f81e8102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:40.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.679+0000 7f81ef234700 1 --2- 172.21.15.98:0/2610490721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e8101e80 0x7f81e8109720 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:40.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.679+0000 7f81ef234700 1 --2- 172.21.15.98:0/2610490721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f81e81014c0 0x7f81e8101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:40.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.679+0000 7f81ef234700 1 -- 172.21.15.98:0/2610490721 >> 172.21.15.98:0/2610490721 conn(0x7f81e80fbfb0 msgr2=0x7f81e80fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:40.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.679+0000 7f81ef234700 1 -- 172.21.15.98:0/2610490721 shutdown_connections 2024-05-09T08:43:40.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.679+0000 7f81ef234700 1 -- 172.21.15.98:0/2610490721 wait complete. 2024-05-09T08:43:40.679 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:40 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-09T08:43:40.679 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:40 smithi098 ceph-mon[30092]: osdmap e19: 3 total, 3 up, 3 in 2024-05-09T08:43:40.679 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:40 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:43:40.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.679+0000 7f81ef234700 1 Processor -- start 2024-05-09T08:43:40.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ef234700 1 -- start start 2024-05-09T08:43:40.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ef234700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e81014c0 0x7f81e81973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:40.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ef234700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f81e8101e80 0x7f81e8197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:40.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ef234700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e81028f0 0x7f81e819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:40.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ef234700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81e810ce30 con 0x7f81e81028f0 2024-05-09T08:43:40.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ef234700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f81e810ccb0 con 0x7f81e81014c0 2024-05-09T08:43:40.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ef234700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f81e810cfb0 con 0x7f81e8101e80 2024-05-09T08:43:40.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ecfd0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e81014c0 0x7f81e81973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:40.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ecfd0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e81014c0 0x7f81e81973c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45330/0 (socket says 172.21.15.98:45330) 2024-05-09T08:43:40.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ecfd0700 1 -- 172.21.15.98:0/1706643186 learned_addr learned my addr 172.21.15.98:0/1706643186 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:43:40.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ed7d1700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e81028f0 0x7f81e819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:40.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81e7fff700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f81e8101e80 0x7f81e8197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:40.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ecfd0700 1 -- 172.21.15.98:0/1706643186 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f81e8101e80 msgr2=0x7f81e8197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:40.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ecfd0700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f81e8101e80 0x7f81e8197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:40.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ecfd0700 1 -- 172.21.15.98:0/1706643186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e81028f0 msgr2=0x7f81e819e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:40.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ecfd0700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e81028f0 0x7f81e819e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:40.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ecfd0700 1 -- 172.21.15.98:0/1706643186 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e819f020 con 0x7f81e81014c0 2024-05-09T08:43:40.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.680+0000 7f81ed7d1700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e81028f0 0x7f81e819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:40.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.681+0000 7f81ecfd0700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e81014c0 0x7f81e81973c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f81dc009f60 tx=0x7f81dc009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:40.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.681+0000 7f81e5ffb700 1 -- 172.21.15.98:0/1706643186 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc00cbd0 con 0x7f81e81014c0 2024-05-09T08:43:40.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.681+0000 7f81ef234700 1 -- 172.21.15.98:0/1706643186 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e8100410 con 0x7f81e81014c0 2024-05-09T08:43:40.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.681+0000 7f81e5ffb700 1 -- 172.21.15.98:0/1706643186 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81dc004500 con 0x7f81e81014c0 2024-05-09T08:43:40.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.681+0000 7f81e5ffb700 1 -- 172.21.15.98:0/1706643186 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc0115d0 con 0x7f81e81014c0 2024-05-09T08:43:40.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.681+0000 7f81ef234700 1 -- 172.21.15.98:0/1706643186 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e81009a0 con 0x7f81e81014c0 2024-05-09T08:43:40.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.682+0000 7f81ef234700 1 -- 172.21.15.98:0/1706643186 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e8069010 con 0x7f81e81014c0 2024-05-09T08:43:40.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.684+0000 7f81e5ffb700 1 -- 172.21.15.98:0/1706643186 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f81dc020020 con 0x7f81e81014c0 2024-05-09T08:43:40.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.684+0000 7f81e5ffb700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f81d80a2c10 0x7f81d80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:40.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.685+0000 7f81e7fff700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f81d80a2c10 0x7f81d80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:40.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.685+0000 7f81e5ffb700 1 -- 172.21.15.98:0/1706643186 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7f81dc0cf890 con 0x7f81e81014c0 2024-05-09T08:43:40.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.685+0000 7f81e7fff700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f81d80a2c10 0x7f81d80a50d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f81d4005950 tx=0x7f81d40058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:40.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.687+0000 7f81e5ffb700 1 -- 172.21.15.98:0/1706643186 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f81dc09dca0 con 0x7f81e81014c0 2024-05-09T08:43:40.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:40 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-09T08:43:40.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:40 smithi105 ceph-mon[34498]: osdmap e19: 3 total, 3 up, 3 in 2024-05-09T08:43:40.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:40 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:43:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:40.895+0000 7f81ef234700 1 -- 172.21.15.98:0/1706643186 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f81e8100c00 con 0x7f81d80a2c10 2024-05-09T08:43:40.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:40 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-09T08:43:40.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:40 smithi142 ceph-mon[34625]: osdmap e19: 3 total, 3 up, 3 in 2024-05-09T08:43:40.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:40 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:43:41.086 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 08:43:40 smithi098 sudo[48986]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:43:41.086 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 08:43:40 smithi098 sudo[48986]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:43:41.087 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 08:43:40 smithi098 sudo[48986]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:43:41.087 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 08:43:40 smithi098 sudo[48986]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:43:41.337 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 08:43:41 smithi098 sudo[49056]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:43:41.338 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 08:43:41 smithi098 sudo[49056]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:43:41.338 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 08:43:41 smithi098 sudo[49056]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:43:41.338 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 08:43:41 smithi098 sudo[49056]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:43:41.609 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 08:43:41 smithi105 sudo[43219]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:43:41.609 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 08:43:41 smithi105 sudo[43219]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:43:41.609 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 08:43:41 smithi105 sudo[43219]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:43:41.609 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 08:43:41 smithi105 sudo[43219]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:43:41.609 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:43:41.609 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:41 smithi105 ceph-mon[34498]: osdmap e20: 3 total, 3 up, 3 in 2024-05-09T08:43:41.609 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:41 smithi105 ceph-mon[34498]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:43:41.609 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:43:41.609 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:43:41.610 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:41.610 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:41.610 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:41 smithi105 ceph-mon[34498]: pgmap v82: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:43:41.764 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:43:41.764 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:41 smithi098 ceph-mon[30092]: osdmap e20: 3 total, 3 up, 3 in 2024-05-09T08:43:41.764 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:41 smithi098 ceph-mon[30092]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:43:41.764 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:43:41.764 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:43:41.764 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:41.764 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:41.765 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:41 smithi098 ceph-mon[30092]: pgmap v82: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:43:41.876 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:43:41.876 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:41 smithi142 ceph-mon[34625]: osdmap e20: 3 total, 3 up, 3 in 2024-05-09T08:43:41.876 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:41 smithi142 ceph-mon[34625]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:43:41.876 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:43:41.876 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:43:41.876 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:41.876 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:41.876 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:41 smithi142 ceph-mon[34625]: pgmap v82: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:43:41.986 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 08:43:41 smithi105 sudo[43222]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:43:41.986 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 08:43:41 smithi105 sudo[43222]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:43:41.986 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 08:43:41 smithi105 sudo[43222]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:43:41.986 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 08:43:41 smithi105 sudo[43222]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:43:42.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:43:42.125+0000 7f81e5ffb700 1 -- 172.21.15.98:0/1706643186 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f81dc099c10 con 0x7f81e81014c0 2024-05-09T08:43:42.129 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 08:43:41 smithi142 sudo[43380]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:43:42.129 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 08:43:41 smithi142 sudo[43380]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:43:42.129 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 08:43:41 smithi142 sudo[43380]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:43:42.129 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 08:43:41 smithi142 sudo[43380]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:43:42.408 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 08:43:42 smithi142 sudo[43383]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:43:42.408 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 08:43:42 smithi142 sudo[43383]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:43:42.408 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 08:43:42 smithi142 sudo[43383]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:43:42.408 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 08:43:42 smithi142 sudo[43383]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:43:42.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:42 smithi098 sudo[49136]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:43:42.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:42 smithi098 sudo[49136]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:43:42.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:42 smithi098 sudo[49136]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:43:42.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:42 smithi098 sudo[49136]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:43:42.917 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:42 smithi142 ceph-mon[34625]: osdmap e21: 3 total, 3 up, 3 in 2024-05-09T08:43:42.917 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:42 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:42.918 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:42 smithi142 ceph-mon[34625]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-09T08:43:42.918 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:42 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:42.918 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:42 smithi142 ceph-mon[34625]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:43:42.918 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:42 smithi142 sudo[43386]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:43:42.918 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:42 smithi142 sudo[43386]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:43:42.918 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:42 smithi142 sudo[43386]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:43:42.918 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:42 smithi142 sudo[43386]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:43:42.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:42 smithi098 sudo[49139]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:43:42.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:42 smithi098 sudo[49139]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:43:42.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:42 smithi098 sudo[49139]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:43:42.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:42 smithi098 sudo[49139]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:43:42.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:42 smithi098 ceph-mon[30092]: osdmap e21: 3 total, 3 up, 3 in 2024-05-09T08:43:42.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:42 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:42.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:42 smithi098 ceph-mon[30092]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-09T08:43:42.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:42 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:42.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:42 smithi098 ceph-mon[30092]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:43:42.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:42 smithi105 ceph-mon[34498]: osdmap e21: 3 total, 3 up, 3 in 2024-05-09T08:43:42.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:42 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:42.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:42 smithi105 ceph-mon[34498]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-09T08:43:42.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:42 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:42.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:42 smithi105 ceph-mon[34498]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:43:43.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 sudo[43389]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:43:43.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 sudo[43389]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:43:43.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 sudo[43389]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:43:43.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 sudo[43389]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:43:43.654 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 sudo[43225]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:43:43.654 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 sudo[43225]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:43:43.654 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 sudo[43225]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:43:43.654 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 sudo[43225]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:43:43.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 sudo[43228]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:43:43.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 sudo[43228]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:43:43.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 sudo[43228]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:43:43.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 sudo[43228]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:43:43.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:43:43.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:43:43.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:43:43.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:43:43.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:43:43.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:43:43.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:43:43.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:43:43.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: pgmap v84: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:43:43.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:43:43.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:43:43.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:43:43.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:43:43.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:43:43.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:43 smithi105 ceph-mon[34498]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:43:44.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:43:44.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:43:44.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:43:44.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:43:44.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:43:44.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:43:44.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:43:44.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:43:44.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: pgmap v84: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:43:44.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:43:44.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:43:44.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:43:44.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:43:44.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:43:44.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:43 smithi098 ceph-mon[30092]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:43:44.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:43:44.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:43:44.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:43:44.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:43:44.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:43:44.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:43:44.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:43:44.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:43:44.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: pgmap v84: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:43:44.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:43:44.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:43:44.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:43:44.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:43:44.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:43:44.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:43 smithi142 ceph-mon[34625]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:43:44.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:44 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3533488662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d229212-2643-4c3a-93c1-25adad98e37d"}]: dispatch 2024-05-09T08:43:44.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:44 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3533488662' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d229212-2643-4c3a-93c1-25adad98e37d"}]': finished 2024-05-09T08:43:44.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:44 smithi105 ceph-mon[34498]: osdmap e22: 4 total, 3 up, 4 in 2024-05-09T08:43:44.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:44 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:43:44.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:44 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/4129342343' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:43:45.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:44 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3533488662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d229212-2643-4c3a-93c1-25adad98e37d"}]: dispatch 2024-05-09T08:43:45.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:44 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3533488662' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d229212-2643-4c3a-93c1-25adad98e37d"}]': finished 2024-05-09T08:43:45.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:44 smithi098 ceph-mon[30092]: osdmap e22: 4 total, 3 up, 4 in 2024-05-09T08:43:45.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:44 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:43:45.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:44 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4129342343' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:43:45.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:44 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3533488662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d229212-2643-4c3a-93c1-25adad98e37d"}]: dispatch 2024-05-09T08:43:45.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:44 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3533488662' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d229212-2643-4c3a-93c1-25adad98e37d"}]': finished 2024-05-09T08:43:45.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:44 smithi142 ceph-mon[34625]: osdmap e22: 4 total, 3 up, 4 in 2024-05-09T08:43:45.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:44 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:43:45.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:44 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/4129342343' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:43:45.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:45 smithi105 ceph-mon[34498]: pgmap v86: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:43:46.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:45 smithi098 ceph-mon[30092]: pgmap v86: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:43:46.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:45 smithi142 ceph-mon[34625]: pgmap v86: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:43:48.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:48 smithi105 ceph-mon[34498]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 13 KiB/s rd, 424 KiB/s wr, 35 op/s 2024-05-09T08:43:48.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:48 smithi098 ceph-mon[30092]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 13 KiB/s rd, 424 KiB/s wr, 35 op/s 2024-05-09T08:43:48.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:48 smithi142 ceph-mon[34625]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 13 KiB/s rd, 424 KiB/s wr, 35 op/s 2024-05-09T08:43:50.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:50 smithi105 ceph-mon[34498]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 318 KiB/s wr, 26 op/s 2024-05-09T08:43:50.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:50 smithi098 ceph-mon[30092]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 318 KiB/s wr, 26 op/s 2024-05-09T08:43:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:50 smithi142 ceph-mon[34625]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 318 KiB/s wr, 26 op/s 2024-05-09T08:43:51.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:51 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-09T08:43:51.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:51 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:51.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:51 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-09T08:43:51.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:51 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:51.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:51 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-09T08:43:51.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:51 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:52.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:52 smithi098 ceph-mon[30092]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.4 KiB/s rd, 266 KiB/s wr, 22 op/s 2024-05-09T08:43:52.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:52 smithi098 ceph-mon[30092]: Deploying daemon osd.3 on smithi098 2024-05-09T08:43:52.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:52 smithi105 ceph-mon[34498]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.4 KiB/s rd, 266 KiB/s wr, 22 op/s 2024-05-09T08:43:52.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:52 smithi105 ceph-mon[34498]: Deploying daemon osd.3 on smithi098 2024-05-09T08:43:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:52 smithi142 ceph-mon[34625]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.4 KiB/s rd, 266 KiB/s wr, 22 op/s 2024-05-09T08:43:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:52 smithi142 ceph-mon[34625]: Deploying daemon osd.3 on smithi098 2024-05-09T08:43:54.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:54 smithi098 ceph-mon[30092]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.0 KiB/s rd, 255 KiB/s wr, 21 op/s 2024-05-09T08:43:54.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:54 smithi105 ceph-mon[34498]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.0 KiB/s rd, 255 KiB/s wr, 21 op/s 2024-05-09T08:43:54.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:54 smithi142 ceph-mon[34625]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.0 KiB/s rd, 255 KiB/s wr, 21 op/s 2024-05-09T08:43:56.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:56 smithi142 ceph-mon[34625]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.1 KiB/s rd, 224 KiB/s wr, 18 op/s 2024-05-09T08:43:56.461 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:56 smithi098 ceph-mon[30092]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.1 KiB/s rd, 224 KiB/s wr, 18 op/s 2024-05-09T08:43:56.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:56 smithi105 ceph-mon[34498]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.1 KiB/s rd, 224 KiB/s wr, 18 op/s 2024-05-09T08:43:58.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:57 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:58.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:57 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:58.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:57 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:58.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:43:57 smithi098 ceph-mon[30092]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 212 KiB/s wr, 17 op/s 2024-05-09T08:43:58.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:57 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:58.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:57 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:58.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:57 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:58.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:43:57 smithi142 ceph-mon[34625]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 212 KiB/s wr, 17 op/s 2024-05-09T08:43:58.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:57 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:58.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:57 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:43:58.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:57 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:58.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:43:57 smithi105 ceph-mon[34498]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 212 KiB/s wr, 17 op/s 2024-05-09T08:44:00.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:00 smithi098 ceph-mon[30092]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:44:00.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:00 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:00.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:00 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:00.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:00 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:00.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:00 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:00.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:00 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:00.435 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:00 smithi105 ceph-mon[34498]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:44:00.435 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:00 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:00.435 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:00 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:00.435 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:00 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:00.435 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:00 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:00.436 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:00 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:00 smithi142 ceph-mon[34625]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:44:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:00 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:00 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:00 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:00 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:00.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:00 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:00.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:00.755+0000 7f81e5ffb700 1 -- 172.21.15.98:0/1706643186 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f81e8100c00 con 0x7f81d80a2c10 2024-05-09T08:44:00.755 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 3 on host 'smithi098' 2024-05-09T08:44:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:00.758+0000 7f81ef234700 1 -- 172.21.15.98:0/1706643186 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f81d80a2c10 msgr2=0x7f81d80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:00.758+0000 7f81ef234700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f81d80a2c10 0x7f81d80a50d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f81d4005950 tx=0x7f81d40058e0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:00.758+0000 7f81ef234700 1 -- 172.21.15.98:0/1706643186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e81014c0 msgr2=0x7f81e81973c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:00.758+0000 7f81ef234700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e81014c0 0x7f81e81973c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f81dc009f60 tx=0x7f81dc009f90 comp rx=0 tx=0).stop 2024-05-09T08:44:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:00.758+0000 7f81ef234700 1 -- 172.21.15.98:0/1706643186 shutdown_connections 2024-05-09T08:44:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:00.758+0000 7f81ef234700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f81d80a2c10 0x7f81d80a50d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:00.758+0000 7f81ef234700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e81014c0 0x7f81e81973c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:00.758+0000 7f81ef234700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e81028f0 0x7f81e819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:00.759+0000 7f81ef234700 1 --2- 172.21.15.98:0/1706643186 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f81e8101e80 0x7f81e8197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:00.759+0000 7f81ef234700 1 -- 172.21.15.98:0/1706643186 >> 172.21.15.98:0/1706643186 conn(0x7f81e80fbfb0 msgr2=0x7f81e8105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:00.759+0000 7f81ef234700 1 -- 172.21.15.98:0/1706643186 shutdown_connections 2024-05-09T08:44:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:00.759+0000 7f81ef234700 1 -- 172.21.15.98:0/1706643186 wait complete. 2024-05-09T08:44:01.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:01 smithi098 ceph-mon[30092]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-09T08:44:01.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:01 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:01.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:01 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:01.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:01 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:01.419 DEBUG:teuthology.orchestra.run.smithi098:osd.3> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.3.service 2024-05-09T08:44:01.422 INFO:tasks.cephadm:Deploying osd.4 on smithi105 with /dev/vg_nvme/lv_3... 2024-05-09T08:44:01.422 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-09T08:44:01.448 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:01 smithi105 ceph-mon[34498]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-09T08:44:01.448 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:01 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:01.448 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:01 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:01.448 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:01 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:01.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:01 smithi142 ceph-mon[34625]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-09T08:44:01.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:01 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:01.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:01 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:01.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:01 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:01.664 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:44:02.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:02 smithi105 ceph-mon[34498]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:44:02.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:02 smithi105 ceph-mon[34498]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-09T08:44:02.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:02 smithi105 ceph-mon[34498]: osdmap e23: 4 total, 3 up, 4 in 2024-05-09T08:44:02.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:02 smithi105 ceph-mon[34498]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-05-09T08:44:02.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:02 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:02.536 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:02 smithi098 ceph-mon[30092]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:44:02.536 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:02 smithi098 ceph-mon[30092]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-09T08:44:02.536 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:02 smithi098 ceph-mon[30092]: osdmap e23: 4 total, 3 up, 4 in 2024-05-09T08:44:02.536 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:02 smithi098 ceph-mon[30092]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-05-09T08:44:02.536 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:02 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:02 smithi142 ceph-mon[34625]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:44:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:02 smithi142 ceph-mon[34625]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-09T08:44:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:02 smithi142 ceph-mon[34625]: osdmap e23: 4 total, 3 up, 4 in 2024-05-09T08:44:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:02 smithi142 ceph-mon[34625]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-05-09T08:44:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:02 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:02.838 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 08:44:02 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3[51245]: 2024-05-09T08:44:02.658+0000 7f1aa3989700 -1 osd.3 0 waiting for initial osdmap 2024-05-09T08:44:02.838 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 08:44:02 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3[51245]: 2024-05-09T08:44:02.665+0000 7f1a9b774700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:44:03.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:03 smithi105 ceph-mon[34498]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-05-09T08:44:03.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:03 smithi105 ceph-mon[34498]: osdmap e24: 4 total, 3 up, 4 in 2024-05-09T08:44:03.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:03 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:03.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:03 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:03.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:03 smithi105 ceph-mon[34498]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' 2024-05-09T08:44:03.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:03 smithi105 ceph-mon[34498]: osdmap e25: 4 total, 3 up, 4 in 2024-05-09T08:44:03.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:03 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:03.552 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-09T08:44:03.552 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-09T08:44:03.552 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-05-09T08:44:03.552 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-05-09T08:44:03.552 INFO:teuthology.orchestra.run.smithi105.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113375 s, 925 MB/s 2024-05-09T08:44:03.552 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-05-09T08:44:03.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:03 smithi098 ceph-mon[30092]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-05-09T08:44:03.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:03 smithi098 ceph-mon[30092]: osdmap e24: 4 total, 3 up, 4 in 2024-05-09T08:44:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:03 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:03 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:03 smithi098 ceph-mon[30092]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' 2024-05-09T08:44:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:03 smithi098 ceph-mon[30092]: osdmap e25: 4 total, 3 up, 4 in 2024-05-09T08:44:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:03 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:03 smithi142 ceph-mon[34625]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-05-09T08:44:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:03 smithi142 ceph-mon[34625]: osdmap e24: 4 total, 3 up, 4 in 2024-05-09T08:44:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:03 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:03 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:03 smithi142 ceph-mon[34625]: from='osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287]' entity='osd.3' 2024-05-09T08:44:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:03 smithi142 ceph-mon[34625]: osdmap e25: 4 total, 3 up, 4 in 2024-05-09T08:44:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:03 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:04.186 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_3 2024-05-09T08:44:04.455 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:44:04.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:04 smithi098 ceph-mon[30092]: purged_snaps scrub starts 2024-05-09T08:44:04.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:04 smithi098 ceph-mon[30092]: purged_snaps scrub ok 2024-05-09T08:44:04.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:04 smithi098 ceph-mon[30092]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:44:04.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:04.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:04 smithi098 ceph-mon[30092]: osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287] boot 2024-05-09T08:44:04.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:04 smithi098 ceph-mon[30092]: osdmap e26: 4 total, 4 up, 4 in 2024-05-09T08:44:04.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:04 smithi105 ceph-mon[34498]: purged_snaps scrub starts 2024-05-09T08:44:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:04 smithi105 ceph-mon[34498]: purged_snaps scrub ok 2024-05-09T08:44:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:04 smithi105 ceph-mon[34498]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:44:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:04 smithi105 ceph-mon[34498]: osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287] boot 2024-05-09T08:44:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:04 smithi105 ceph-mon[34498]: osdmap e26: 4 total, 4 up, 4 in 2024-05-09T08:44:04.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:04 smithi142 ceph-mon[34625]: purged_snaps scrub starts 2024-05-09T08:44:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:04 smithi142 ceph-mon[34625]: purged_snaps scrub ok 2024-05-09T08:44:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:04 smithi142 ceph-mon[34625]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:44:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:04 smithi142 ceph-mon[34625]: osd.3 [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287] boot 2024-05-09T08:44:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:04 smithi142 ceph-mon[34625]: osdmap e26: 4 total, 4 up, 4 in 2024-05-09T08:44:04.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:44:05.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:05 smithi105 ceph-mon[34498]: Detected new or changed devices on smithi098 2024-05-09T08:44:05.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:05 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:05.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:05 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:05.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:05 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:05.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:05 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:05.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:05 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi098 to 8544M 2024-05-09T08:44:05.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:05 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:05.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:05 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:05.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:05 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:05.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:05 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:05.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:05 smithi105 ceph-mon[34498]: osdmap e27: 4 total, 4 up, 4 in 2024-05-09T08:44:05.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:05 smithi105 ceph-mon[34498]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:05.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:05 smithi098 ceph-mon[30092]: Detected new or changed devices on smithi098 2024-05-09T08:44:05.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:05 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:05.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:05 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:05.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:05 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:05.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:05 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:05.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:05 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi098 to 8544M 2024-05-09T08:44:05.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:05 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:05.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:05 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:05.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:05 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:05.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:05 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:05.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:05 smithi098 ceph-mon[30092]: osdmap e27: 4 total, 4 up, 4 in 2024-05-09T08:44:05.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:05 smithi098 ceph-mon[30092]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:05.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:05 smithi142 ceph-mon[34625]: Detected new or changed devices on smithi098 2024-05-09T08:44:05.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:05 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:05.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:05 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:05.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:05 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:05.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:05 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:05.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:05 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi098 to 8544M 2024-05-09T08:44:05.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:05 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:05.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:05 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:05.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:05 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:05.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:05 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:05.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:05 smithi142 ceph-mon[34625]: osdmap e27: 4 total, 4 up, 4 in 2024-05-09T08:44:05.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:05 smithi142 ceph-mon[34625]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:05.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.997+0000 7fb132a99700 1 -- 172.21.15.105:0/445183429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c101270 msgr2=0x7fb12c101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:05.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.997+0000 7fb132a99700 1 --2- 172.21.15.105:0/445183429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c101270 0x7fb12c101690 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb12800b0d0 tx=0x7fb12800b3e0 comp rx=0 tx=0).stop 2024-05-09T08:44:05.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.997+0000 7fb132a99700 1 -- 172.21.15.105:0/445183429 shutdown_connections 2024-05-09T08:44:05.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.997+0000 7fb132a99700 1 --2- 172.21.15.105:0/445183429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c101270 0x7fb12c101690 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:05.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.997+0000 7fb132a99700 1 --2- 172.21.15.105:0/445183429 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb12c102e10 0x7fb12c1096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:05.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.997+0000 7fb132a99700 1 --2- 172.21.15.105:0/445183429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb12c102470 0x7fb12c1028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:05.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.997+0000 7fb132a99700 1 -- 172.21.15.105:0/445183429 >> 172.21.15.105:0/445183429 conn(0x7fb12c0fc810 msgr2=0x7fb12c0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:05.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.997+0000 7fb132a99700 1 -- 172.21.15.105:0/445183429 shutdown_connections 2024-05-09T08:44:05.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.997+0000 7fb132a99700 1 -- 172.21.15.105:0/445183429 wait complete. 2024-05-09T08:44:05.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.997+0000 7fb132a99700 1 Processor -- start 2024-05-09T08:44:05.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb132a99700 1 -- start start 2024-05-09T08:44:05.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb132a99700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb12c101270 0x7fb12c1974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:05.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb132a99700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c102470 0x7fb12c1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:05.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb132a99700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb12c102e10 0x7fb12c19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:05.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb132a99700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c10ceb0 con 0x7fb12c102e10 2024-05-09T08:44:05.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb132a99700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c10cd30 con 0x7fb12c102470 2024-05-09T08:44:05.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb132a99700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c10d030 con 0x7fb12c101270 2024-05-09T08:44:05.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb123fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c102470 0x7fb12c1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:05.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb123fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c102470 0x7fb12c1979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58382/0 (socket says 172.21.15.105:58382) 2024-05-09T08:44:05.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb123fff700 1 -- 172.21.15.105:0/3183785201 learned_addr learned my addr 172.21.15.105:0/3183785201 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-05-09T08:44:05.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb131036700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb12c102e10 0x7fb12c19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:05.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb130835700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb12c101270 0x7fb12c1974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:05.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb123fff700 1 -- 172.21.15.105:0/3183785201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb12c101270 msgr2=0x7fb12c1974a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:05.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb123fff700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb12c101270 0x7fb12c1974a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:05.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb123fff700 1 -- 172.21.15.105:0/3183785201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb12c102e10 msgr2=0x7fb12c19ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:05.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb123fff700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb12c102e10 0x7fb12c19ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:06.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb123fff700 1 -- 172.21.15.105:0/3183785201 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12c19f100 con 0x7fb12c102470 2024-05-09T08:44:06.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb130835700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb12c101270 0x7fb12c1974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:44:06.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb123fff700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c102470 0x7fb12c1979e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb11c00d950 tx=0x7fb11c00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:06.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.998+0000 7fb121ffb700 1 -- 172.21.15.105:0/3183785201 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c016070 con 0x7fb12c102470 2024-05-09T08:44:06.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.999+0000 7fb132a99700 1 -- 172.21.15.105:0/3183785201 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb12c19f3f0 con 0x7fb12c102470 2024-05-09T08:44:06.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.999+0000 7fb121ffb700 1 -- 172.21.15.105:0/3183785201 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb11c009e20 con 0x7fb12c102470 2024-05-09T08:44:06.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.999+0000 7fb121ffb700 1 -- 172.21.15.105:0/3183785201 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c00f620 con 0x7fb12c102470 2024-05-09T08:44:06.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:05.999+0000 7fb132a99700 1 -- 172.21.15.105:0/3183785201 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb12c19f9b0 con 0x7fb12c102470 2024-05-09T08:44:06.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:06.000+0000 7fb132a99700 1 -- 172.21.15.105:0/3183785201 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb12c069010 con 0x7fb12c102470 2024-05-09T08:44:06.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:06.001+0000 7fb121ffb700 1 -- 172.21.15.105:0/3183785201 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb11c019070 con 0x7fb12c102470 2024-05-09T08:44:06.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:06.001+0000 7fb121ffb700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb1140a2c20 0x7fb1140a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:06.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:06.001+0000 7fb121ffb700 1 -- 172.21.15.105:0/3183785201 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3698+0+0 (secure 0 0 0) 0x7fb11c0cda80 con 0x7fb12c102470 2024-05-09T08:44:06.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:06.001+0000 7fb130835700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb1140a2c20 0x7fb1140a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:06.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:06.002+0000 7fb130835700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb1140a2c20 0x7fb1140a50e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb128007fa0 tx=0x7fb12800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:06.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:06.006+0000 7fb121ffb700 1 -- 172.21.15.105:0/3183785201 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb11c0cfe20 con 0x7fb12c102470 2024-05-09T08:44:06.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:06.212+0000 7fb132a99700 1 -- 172.21.15.105:0/3183785201 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb12c062760 con 0x7fb1140a2c20 2024-05-09T08:44:06.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:06 smithi105 ceph-mon[34498]: osdmap e28: 4 total, 4 up, 4 in 2024-05-09T08:44:06.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:06 smithi105 ceph-mon[34498]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:44:06.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:06 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:44:06.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:06 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:44:06.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:06 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:07.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:06 smithi098 ceph-mon[30092]: osdmap e28: 4 total, 4 up, 4 in 2024-05-09T08:44:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:06 smithi098 ceph-mon[30092]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:44:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:44:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:44:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:07.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:06 smithi142 ceph-mon[34625]: osdmap e28: 4 total, 4 up, 4 in 2024-05-09T08:44:07.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:06 smithi142 ceph-mon[34625]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:44:07.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:44:07.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:44:07.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:07.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:07 smithi105 ceph-mon[34498]: osdmap e29: 4 total, 4 up, 4 in 2024-05-09T08:44:07.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:07 smithi105 ceph-mon[34498]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:08.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:07 smithi098 ceph-mon[30092]: osdmap e29: 4 total, 4 up, 4 in 2024-05-09T08:44:08.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:07 smithi098 ceph-mon[30092]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:08.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:07 smithi142 ceph-mon[34625]: osdmap e29: 4 total, 4 up, 4 in 2024-05-09T08:44:08.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:07 smithi142 ceph-mon[34625]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:08.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:08 smithi105 ceph-mon[34498]: osdmap e30: 4 total, 4 up, 4 in 2024-05-09T08:44:09.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:08 smithi098 ceph-mon[30092]: osdmap e30: 4 total, 4 up, 4 in 2024-05-09T08:44:09.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:08 smithi142 ceph-mon[34625]: osdmap e30: 4 total, 4 up, 4 in 2024-05-09T08:44:09.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:09 smithi105 ceph-mon[34498]: from='client.? 172.21.15.105:0/1843585557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cf860bd-1c29-45e5-ba31-5dd49f76913a"}]: dispatch 2024-05-09T08:44:09.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:09 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cf860bd-1c29-45e5-ba31-5dd49f76913a"}]: dispatch 2024-05-09T08:44:09.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:09 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cf860bd-1c29-45e5-ba31-5dd49f76913a"}]': finished 2024-05-09T08:44:09.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:09 smithi105 ceph-mon[34498]: osdmap e31: 5 total, 4 up, 5 in 2024-05-09T08:44:09.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:09 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:09.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:09 smithi105 ceph-mon[34498]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:09.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:09 smithi105 ceph-mon[34498]: from='client.? 172.21.15.105:0/2522335713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:44:10.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:09 smithi098 ceph-mon[30092]: from='client.? 172.21.15.105:0/1843585557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cf860bd-1c29-45e5-ba31-5dd49f76913a"}]: dispatch 2024-05-09T08:44:10.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:09 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cf860bd-1c29-45e5-ba31-5dd49f76913a"}]: dispatch 2024-05-09T08:44:10.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:09 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cf860bd-1c29-45e5-ba31-5dd49f76913a"}]': finished 2024-05-09T08:44:10.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:09 smithi098 ceph-mon[30092]: osdmap e31: 5 total, 4 up, 5 in 2024-05-09T08:44:10.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:09 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:10.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:09 smithi098 ceph-mon[30092]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:10.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:09 smithi098 ceph-mon[30092]: from='client.? 172.21.15.105:0/2522335713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:44:10.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:09 smithi142 ceph-mon[34625]: from='client.? 172.21.15.105:0/1843585557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cf860bd-1c29-45e5-ba31-5dd49f76913a"}]: dispatch 2024-05-09T08:44:10.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:09 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cf860bd-1c29-45e5-ba31-5dd49f76913a"}]: dispatch 2024-05-09T08:44:10.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:09 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cf860bd-1c29-45e5-ba31-5dd49f76913a"}]': finished 2024-05-09T08:44:10.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:09 smithi142 ceph-mon[34625]: osdmap e31: 5 total, 4 up, 5 in 2024-05-09T08:44:10.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:09 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:10.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:09 smithi142 ceph-mon[34625]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:10.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:09 smithi142 ceph-mon[34625]: from='client.? 172.21.15.105:0/2522335713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:44:12.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:12 smithi105 ceph-mon[34498]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-05-09T08:44:12.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:12 smithi098 ceph-mon[30092]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-05-09T08:44:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:12 smithi142 ceph-mon[34625]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-05-09T08:44:14.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:14 smithi105 ceph-mon[34498]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-09T08:44:14.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:14 smithi098 ceph-mon[30092]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-09T08:44:14.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:14 smithi142 ceph-mon[34625]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-09T08:44:16.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:16 smithi105 ceph-mon[34498]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:44:16.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:16 smithi098 ceph-mon[30092]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:44:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:16 smithi142 ceph-mon[34625]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:44:17.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-09T08:44:17.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:17.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:17 smithi105 ceph-mon[34498]: Deploying daemon osd.4 on smithi105 2024-05-09T08:44:17.489 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-09T08:44:17.489 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:17.489 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:17 smithi098 ceph-mon[30092]: Deploying daemon osd.4 on smithi105 2024-05-09T08:44:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-09T08:44:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:17 smithi142 ceph-mon[34625]: Deploying daemon osd.4 on smithi105 2024-05-09T08:44:18.427 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:18 smithi105 ceph-mon[34498]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-09T08:44:18.508 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:18 smithi098 ceph-mon[30092]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-09T08:44:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:18 smithi142 ceph-mon[34625]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-09T08:44:20.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:20 smithi105 ceph-mon[34498]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-05-09T08:44:20.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:20 smithi098 ceph-mon[30092]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-05-09T08:44:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:20 smithi142 ceph-mon[34625]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-05-09T08:44:22.418 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:22 smithi105 ceph-mon[34498]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T08:44:22.418 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:22.418 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:22.418 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:22.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:22 smithi098 ceph-mon[30092]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T08:44:22.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:22.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:22.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:22 smithi142 ceph-mon[34625]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T08:44:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:24.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:24 smithi105 ceph-mon[34498]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:24.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:24.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:24.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:24.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:24.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:24.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:24 smithi098 ceph-mon[30092]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:24 smithi142 ceph-mon[34625]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:24.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:24.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:24.975+0000 7fb121ffb700 1 -- 172.21.15.105:0/3183785201 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb12c062760 con 0x7fb1140a2c20 2024-05-09T08:44:24.975 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 4 on host 'smithi105' 2024-05-09T08:44:24.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:24.977+0000 7fb132a99700 1 -- 172.21.15.105:0/3183785201 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb1140a2c20 msgr2=0x7fb1140a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:24.977+0000 7fb132a99700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb1140a2c20 0x7fb1140a50e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb128007fa0 tx=0x7fb12800b040 comp rx=0 tx=0).stop 2024-05-09T08:44:24.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:24.977+0000 7fb132a99700 1 -- 172.21.15.105:0/3183785201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c102470 msgr2=0x7fb12c1979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:24.977+0000 7fb132a99700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c102470 0x7fb12c1979e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb11c00d950 tx=0x7fb11c00dc60 comp rx=0 tx=0).stop 2024-05-09T08:44:24.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:24.977+0000 7fb132a99700 1 -- 172.21.15.105:0/3183785201 shutdown_connections 2024-05-09T08:44:24.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:24.977+0000 7fb132a99700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb1140a2c20 0x7fb1140a50e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:24.977+0000 7fb132a99700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb12c102e10 0x7fb12c19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:24.977+0000 7fb132a99700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb12c101270 0x7fb12c1974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:24.977+0000 7fb132a99700 1 --2- 172.21.15.105:0/3183785201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c102470 0x7fb12c1979e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:24.977+0000 7fb132a99700 1 -- 172.21.15.105:0/3183785201 >> 172.21.15.105:0/3183785201 conn(0x7fb12c0fc810 msgr2=0x7fb12c105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:24.978+0000 7fb132a99700 1 -- 172.21.15.105:0/3183785201 shutdown_connections 2024-05-09T08:44:24.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:44:24.978+0000 7fb132a99700 1 -- 172.21.15.105:0/3183785201 wait complete. 2024-05-09T08:44:25.645 DEBUG:teuthology.orchestra.run.smithi105:osd.4> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.4.service 2024-05-09T08:44:25.647 INFO:tasks.cephadm:Deploying osd.5 on smithi142 with /dev/vg_nvme/lv_3... 2024-05-09T08:44:25.647 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-09T08:44:25.893 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:44:26.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:25 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:26.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:25 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:26.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:25 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:26.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:25 smithi105 ceph-mon[34498]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:26.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:25 smithi105 ceph-mon[34498]: from='osd.4 [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:44:26.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:25 smithi105 ceph-mon[34498]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:44:26.295 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:25 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:26.295 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:25 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:26.295 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:25 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:26.295 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:25 smithi142 ceph-mon[34625]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:26.295 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:25 smithi142 ceph-mon[34625]: from='osd.4 [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:44:26.295 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:25 smithi142 ceph-mon[34625]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:44:26.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:25 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:25 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:25 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:25 smithi098 ceph-mon[30092]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:25 smithi098 ceph-mon[30092]: from='osd.4 [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:44:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:25 smithi098 ceph-mon[30092]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:44:27.088 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:26 smithi105 ceph-mon[34498]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-09T08:44:27.088 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:26 smithi105 ceph-mon[34498]: from='osd.4 [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:44:27.088 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:26 smithi105 ceph-mon[34498]: osdmap e32: 5 total, 4 up, 5 in 2024-05-09T08:44:27.089 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:26 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:27.089 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:26 smithi105 ceph-mon[34498]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:44:27.089 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 08:44:26 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-4[46017]: 2024-05-09T08:44:26.979+0000 7f03aaea6700 -1 osd.4 0 waiting for initial osdmap 2024-05-09T08:44:27.089 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 08:44:26 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-4[46017]: 2024-05-09T08:44:26.985+0000 7f03a6c99700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:44:27.293 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:26 smithi142 ceph-mon[34625]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-09T08:44:27.293 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:26 smithi142 ceph-mon[34625]: from='osd.4 [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:44:27.293 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:26 smithi142 ceph-mon[34625]: osdmap e32: 5 total, 4 up, 5 in 2024-05-09T08:44:27.293 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:26 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:27.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:26 smithi142 ceph-mon[34625]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:44:27.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:26 smithi098 ceph-mon[30092]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-09T08:44:27.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:26 smithi098 ceph-mon[30092]: from='osd.4 [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:44:27.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:26 smithi098 ceph-mon[30092]: osdmap e32: 5 total, 4 up, 5 in 2024-05-09T08:44:27.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:26 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:27.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:26 smithi098 ceph-mon[30092]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:44:27.998 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-09T08:44:27.999 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-09T08:44:27.999 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2024-05-09T08:44:27.999 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2024-05-09T08:44:27.999 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128154 s, 818 MB/s 2024-05-09T08:44:27.999 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2024-05-09T08:44:28.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:27 smithi105 ceph-mon[34498]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-09T08:44:28.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:27 smithi105 ceph-mon[34498]: osdmap e33: 5 total, 4 up, 5 in 2024-05-09T08:44:28.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:27 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:28.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:27 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:28.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:27 smithi105 ceph-mon[34498]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:28.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:27 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:28.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:27 smithi098 ceph-mon[30092]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-09T08:44:28.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:27 smithi098 ceph-mon[30092]: osdmap e33: 5 total, 4 up, 5 in 2024-05-09T08:44:28.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:27 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:28.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:27 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:28.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:27 smithi098 ceph-mon[30092]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:28.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:27 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:28.346 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:27 smithi142 ceph-mon[34625]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-09T08:44:28.346 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:27 smithi142 ceph-mon[34625]: osdmap e33: 5 total, 4 up, 5 in 2024-05-09T08:44:28.346 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:27 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:28.346 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:27 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:28.346 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:27 smithi142 ceph-mon[34625]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:44:28.346 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:27 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:28.645 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi142:vg_nvme/lv_3 2024-05-09T08:44:28.887 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:44:29.308 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:28 smithi142 ceph-mon[34625]: purged_snaps scrub starts 2024-05-09T08:44:29.308 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:28 smithi142 ceph-mon[34625]: purged_snaps scrub ok 2024-05-09T08:44:29.308 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:28 smithi142 ceph-mon[34625]: osd.4 [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474] boot 2024-05-09T08:44:29.308 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:28 smithi142 ceph-mon[34625]: osdmap e34: 5 total, 5 up, 5 in 2024-05-09T08:44:29.308 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:28 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:29.308 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:28 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:29.308 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:28 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:29.308 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:28 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:29.308 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:28 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:29.308 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:28 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:29.309 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:28 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:29.309 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:28 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:29.309 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:28 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:29.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:28 smithi098 ceph-mon[30092]: purged_snaps scrub starts 2024-05-09T08:44:29.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:28 smithi098 ceph-mon[30092]: purged_snaps scrub ok 2024-05-09T08:44:29.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:28 smithi098 ceph-mon[30092]: osd.4 [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474] boot 2024-05-09T08:44:29.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:28 smithi098 ceph-mon[30092]: osdmap e34: 5 total, 5 up, 5 in 2024-05-09T08:44:29.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:29.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:29.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:29.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:29.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:29.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:29.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:29.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:29.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:29.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:28 smithi105 ceph-mon[34498]: purged_snaps scrub starts 2024-05-09T08:44:29.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:28 smithi105 ceph-mon[34498]: purged_snaps scrub ok 2024-05-09T08:44:29.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:28 smithi105 ceph-mon[34498]: osd.4 [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474] boot 2024-05-09T08:44:29.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:28 smithi105 ceph-mon[34498]: osdmap e34: 5 total, 5 up, 5 in 2024-05-09T08:44:29.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:28 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:44:29.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:28 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:29.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:28 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:29.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:28 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:29.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:28 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:29.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:28 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:29.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:28 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:29.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:28 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:29.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:28 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:30.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:30 smithi098 ceph-mon[30092]: Detected new or changed devices on smithi105 2024-05-09T08:44:30.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:30 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi105 to 8544M 2024-05-09T08:44:30.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:30 smithi098 ceph-mon[30092]: osdmap e35: 5 total, 5 up, 5 in 2024-05-09T08:44:30.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:30 smithi098 ceph-mon[30092]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:30.273 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:30 smithi142 ceph-mon[34625]: Detected new or changed devices on smithi105 2024-05-09T08:44:30.273 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:30 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi105 to 8544M 2024-05-09T08:44:30.273 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:30 smithi142 ceph-mon[34625]: osdmap e35: 5 total, 5 up, 5 in 2024-05-09T08:44:30.273 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:30 smithi142 ceph-mon[34625]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:30.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.451+0000 7f99dbefd700 1 -- 172.21.15.142:0/4291796272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d4102380 msgr2=0x7f99d41027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:30.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.451+0000 7f99dbefd700 1 --2- 172.21.15.142:0/4291796272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d4102380 0x7f99d41027a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009990 tx=0x7f99d0009ca0 comp rx=0 tx=0).stop 2024-05-09T08:44:30.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.451+0000 7f99dbefd700 1 -- 172.21.15.142:0/4291796272 shutdown_connections 2024-05-09T08:44:30.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.451+0000 7f99dbefd700 1 --2- 172.21.15.142:0/4291796272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4103f20 0x7f99d410a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:30.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.451+0000 7f99dbefd700 1 --2- 172.21.15.142:0/4291796272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d4102380 0x7f99d41027a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:30.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.451+0000 7f99dbefd700 1 --2- 172.21.15.142:0/4291796272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99d4103580 0x7f99d41039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:30.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.451+0000 7f99dbefd700 1 -- 172.21.15.142:0/4291796272 >> 172.21.15.142:0/4291796272 conn(0x7f99d40fd920 msgr2=0x7f99d40ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:30.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.451+0000 7f99dbefd700 1 -- 172.21.15.142:0/4291796272 shutdown_connections 2024-05-09T08:44:30.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.451+0000 7f99dbefd700 1 -- 172.21.15.142:0/4291796272 wait complete. 2024-05-09T08:44:30.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.452+0000 7f99dbefd700 1 Processor -- start 2024-05-09T08:44:30.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.452+0000 7f99dbefd700 1 -- start start 2024-05-09T08:44:30.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.452+0000 7f99dbefd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d4102380 0x7f99d41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:30.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.452+0000 7f99dbefd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99d4103580 0x7f99d4198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:30.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.452+0000 7f99dbefd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4103f20 0x7f99d419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:30.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.452+0000 7f99dbefd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f99d410df60 con 0x7f99d4103f20 2024-05-09T08:44:30.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.452+0000 7f99dbefd700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f99d410dde0 con 0x7f99d4103580 2024-05-09T08:44:30.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.452+0000 7f99dbefd700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f99d410e0e0 con 0x7f99d4102380 2024-05-09T08:44:30.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.452+0000 7f99d9c99700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d4102380 0x7f99d41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:30.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99d9c99700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d4102380 0x7f99d41985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58638/0 (socket says 172.21.15.142:58638) 2024-05-09T08:44:30.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99d9c99700 1 -- 172.21.15.142:0/4032074875 learned_addr learned my addr 172.21.15.142:0/4032074875 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-05-09T08:44:30.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99d9498700 1 --2- 172.21.15.142:0/4032074875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99d4103580 0x7f99d4198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:30.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99da49a700 1 --2- 172.21.15.142:0/4032074875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4103f20 0x7f99d419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:30.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99d9c99700 1 -- 172.21.15.142:0/4032074875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99d4103580 msgr2=0x7f99d4198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:30.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99d9c99700 1 --2- 172.21.15.142:0/4032074875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99d4103580 0x7f99d4198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:30.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99d9c99700 1 -- 172.21.15.142:0/4032074875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4103f20 msgr2=0x7f99d419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:30.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99d9c99700 1 --2- 172.21.15.142:0/4032074875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4103f20 0x7f99d419fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:30.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99d9c99700 1 -- 172.21.15.142:0/4032074875 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d41a0230 con 0x7f99d4102380 2024-05-09T08:44:30.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99d9c99700 1 --2- 172.21.15.142:0/4032074875 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d4102380 0x7f99d41985d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f99d0005b40 tx=0x7f99d000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:30.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99caffd700 1 -- 172.21.15.142:0/4032074875 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d0026070 con 0x7f99d4102380 2024-05-09T08:44:30.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99caffd700 1 -- 172.21.15.142:0/4032074875 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99d000fe50 con 0x7f99d4102380 2024-05-09T08:44:30.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99dbefd700 1 -- 172.21.15.142:0/4032074875 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99d41a04c0 con 0x7f99d4102380 2024-05-09T08:44:30.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99caffd700 1 -- 172.21.15.142:0/4032074875 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d0017650 con 0x7f99d4102380 2024-05-09T08:44:30.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.453+0000 7f99dbefd700 1 -- 172.21.15.142:0/4032074875 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99d41a0910 con 0x7f99d4102380 2024-05-09T08:44:30.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.454+0000 7f99dbefd700 1 -- 172.21.15.142:0/4032074875 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99d4069010 con 0x7f99d4102380 2024-05-09T08:44:30.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.455+0000 7f99caffd700 1 -- 172.21.15.142:0/4032074875 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f99d0029070 con 0x7f99d4102380 2024-05-09T08:44:30.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.456+0000 7f99caffd700 1 --2- 172.21.15.142:0/4032074875 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f99c00a2c20 0x7f99c00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:30.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.456+0000 7f99caffd700 1 -- 172.21.15.142:0/4032074875 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7f99d00cee60 con 0x7f99d4102380 2024-05-09T08:44:30.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.456+0000 7f99d9498700 1 --2- 172.21.15.142:0/4032074875 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f99c00a2c20 0x7f99c00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:30.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.456+0000 7f99d9498700 1 --2- 172.21.15.142:0/4032074875 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f99c00a2c20 0x7f99c00a50e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009c60 tx=0x7f99c4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:30.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.461+0000 7f99caffd700 1 -- 172.21.15.142:0/4032074875 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f99d0013070 con 0x7f99d4102380 2024-05-09T08:44:30.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:30 smithi105 ceph-mon[34498]: Detected new or changed devices on smithi105 2024-05-09T08:44:30.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:30 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi105 to 8544M 2024-05-09T08:44:30.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:30 smithi105 ceph-mon[34498]: osdmap e35: 5 total, 5 up, 5 in 2024-05-09T08:44:30.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:30 smithi105 ceph-mon[34498]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:30.663+0000 7f99dbefd700 1 -- 172.21.15.142:0/4032074875 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f99d4062760 con 0x7f99c00a2c20 2024-05-09T08:44:31.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:31 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:44:31.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:31 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:44:31.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:31 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:31.268 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:31 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:44:31.269 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:31 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:44:31.269 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:31 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:31.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:31 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:44:31.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:31 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:44:31.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:31 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:32.277 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:32 smithi142 ceph-mon[34625]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:44:32.277 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:32 smithi142 ceph-mon[34625]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:32.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:32 smithi098 ceph-mon[30092]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:44:32.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:32 smithi098 ceph-mon[30092]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:32.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:32 smithi105 ceph-mon[34498]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:44:32.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:32 smithi105 ceph-mon[34498]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:34.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:34 smithi105 ceph-mon[34498]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:34 smithi105 ceph-mon[34498]: from='client.? 172.21.15.142:0/3820250544' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d2c5554-188a-45eb-b33b-cf929f0b40fa"}]: dispatch 2024-05-09T08:44:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:34 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d2c5554-188a-45eb-b33b-cf929f0b40fa"}]: dispatch 2024-05-09T08:44:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:34 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d2c5554-188a-45eb-b33b-cf929f0b40fa"}]': finished 2024-05-09T08:44:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:34 smithi105 ceph-mon[34498]: osdmap e36: 6 total, 5 up, 6 in 2024-05-09T08:44:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:34 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:34 smithi105 ceph-mon[34498]: from='client.? 172.21.15.142:0/1495412658' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:44:34.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:34 smithi098 ceph-mon[30092]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:34.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:34 smithi098 ceph-mon[30092]: from='client.? 172.21.15.142:0/3820250544' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d2c5554-188a-45eb-b33b-cf929f0b40fa"}]: dispatch 2024-05-09T08:44:34.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:34 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d2c5554-188a-45eb-b33b-cf929f0b40fa"}]: dispatch 2024-05-09T08:44:34.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:34 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d2c5554-188a-45eb-b33b-cf929f0b40fa"}]': finished 2024-05-09T08:44:34.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:34 smithi098 ceph-mon[30092]: osdmap e36: 6 total, 5 up, 6 in 2024-05-09T08:44:34.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:34 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:34.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:34 smithi098 ceph-mon[30092]: from='client.? 172.21.15.142:0/1495412658' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:44:34.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:34 smithi142 ceph-mon[34625]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:34.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:34 smithi142 ceph-mon[34625]: from='client.? 172.21.15.142:0/3820250544' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d2c5554-188a-45eb-b33b-cf929f0b40fa"}]: dispatch 2024-05-09T08:44:34.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:34 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d2c5554-188a-45eb-b33b-cf929f0b40fa"}]: dispatch 2024-05-09T08:44:34.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:34 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d2c5554-188a-45eb-b33b-cf929f0b40fa"}]': finished 2024-05-09T08:44:34.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:34 smithi142 ceph-mon[34625]: osdmap e36: 6 total, 5 up, 6 in 2024-05-09T08:44:34.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:34 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:34.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:34 smithi142 ceph-mon[34625]: from='client.? 172.21.15.142:0/1495412658' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:44:36.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:36 smithi105 ceph-mon[34498]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:36.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:36 smithi098 ceph-mon[30092]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:36 smithi142 ceph-mon[34625]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:38.348 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:38 smithi142 ceph-mon[34625]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:38.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:38 smithi105 ceph-mon[34498]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:38.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:38 smithi098 ceph-mon[30092]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:40.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:40 smithi105 ceph-mon[34498]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:40.568 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:40 smithi142 ceph-mon[34625]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:40.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:40 smithi098 ceph-mon[30092]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:41.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-09T08:44:41.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:41.512 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-09T08:44:41.513 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:41.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-09T08:44:41.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:42.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:42 smithi105 ceph-mon[34498]: Deploying daemon osd.5 on smithi142 2024-05-09T08:44:42.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:42 smithi105 ceph-mon[34498]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:42.519 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:42 smithi142 ceph-mon[34625]: Deploying daemon osd.5 on smithi142 2024-05-09T08:44:42.519 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:42 smithi142 ceph-mon[34625]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:42.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:42 smithi098 ceph-mon[30092]: Deploying daemon osd.5 on smithi142 2024-05-09T08:44:42.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:42 smithi098 ceph-mon[30092]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:44.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:44 smithi142 ceph-mon[34625]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:44.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:44 smithi105 ceph-mon[34498]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:44.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:44 smithi098 ceph-mon[30092]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:46.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:46 smithi105 ceph-mon[34498]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:46.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:46 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:46.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:46 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:46.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:46 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:46.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:46 smithi098 ceph-mon[30092]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:46.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:46.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:46.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:46 smithi142 ceph-mon[34625]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:46 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:46 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:46 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:48.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:48 smithi105 ceph-mon[34498]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:48.558 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:48 smithi142 ceph-mon[34625]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:48.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:48 smithi098 ceph-mon[30092]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:49.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:49.878+0000 7f99caffd700 1 -- 172.21.15.142:0/4032074875 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f99d4062760 con 0x7f99c00a2c20 2024-05-09T08:44:49.878 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 5 on host 'smithi142' 2024-05-09T08:44:49.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:49.880+0000 7f99dbefd700 1 -- 172.21.15.142:0/4032074875 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f99c00a2c20 msgr2=0x7f99c00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:49.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:49.880+0000 7f99dbefd700 1 --2- 172.21.15.142:0/4032074875 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f99c00a2c20 0x7f99c00a50e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009c60 tx=0x7f99c4009260 comp rx=0 tx=0).stop 2024-05-09T08:44:49.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:49.880+0000 7f99dbefd700 1 -- 172.21.15.142:0/4032074875 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d4102380 msgr2=0x7f99d41985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:49.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:49.880+0000 7f99dbefd700 1 --2- 172.21.15.142:0/4032074875 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d4102380 0x7f99d41985d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f99d0005b40 tx=0x7f99d000f690 comp rx=0 tx=0).stop 2024-05-09T08:44:49.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:49.880+0000 7f99dbefd700 1 -- 172.21.15.142:0/4032074875 shutdown_connections 2024-05-09T08:44:49.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:49.880+0000 7f99dbefd700 1 --2- 172.21.15.142:0/4032074875 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f99c00a2c20 0x7f99c00a50e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:49.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:49.880+0000 7f99dbefd700 1 --2- 172.21.15.142:0/4032074875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4103f20 0x7f99d419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:49.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:49.880+0000 7f99dbefd700 1 --2- 172.21.15.142:0/4032074875 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d4102380 0x7f99d41985d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:49.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:49.880+0000 7f99dbefd700 1 --2- 172.21.15.142:0/4032074875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99d4103580 0x7f99d4198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:49.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:49.880+0000 7f99dbefd700 1 -- 172.21.15.142:0/4032074875 >> 172.21.15.142:0/4032074875 conn(0x7f99d40fd920 msgr2=0x7f99d4106900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:49.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:49.881+0000 7f99dbefd700 1 -- 172.21.15.142:0/4032074875 shutdown_connections 2024-05-09T08:44:49.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:44:49.881+0000 7f99dbefd700 1 -- 172.21.15.142:0/4032074875 wait complete. 2024-05-09T08:44:50.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:50.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:50.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:50 smithi142 ceph-mon[34625]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:50.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:50.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:50 smithi142 ceph-mon[34625]: from='osd.5 [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:44:50.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:50 smithi142 ceph-mon[34625]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:44:50.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:50 smithi098 ceph-mon[30092]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:50 smithi098 ceph-mon[30092]: from='osd.5 [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:44:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:50 smithi098 ceph-mon[30092]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:44:50.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:50 smithi105 ceph-mon[34498]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:50 smithi105 ceph-mon[34498]: from='osd.5 [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:44:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:50 smithi105 ceph-mon[34498]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:44:50.591 DEBUG:teuthology.orchestra.run.smithi142:osd.5> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.5.service 2024-05-09T08:44:50.593 INFO:tasks.cephadm:Deploying osd.6 on smithi098 with /dev/vg_nvme/lv_2... 2024-05-09T08:44:50.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-09T08:44:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:44:51.979 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:51 smithi142 ceph-mon[34625]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-09T08:44:51.979 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:51 smithi142 ceph-mon[34625]: osdmap e37: 6 total, 5 up, 6 in 2024-05-09T08:44:51.979 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:51 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:51.979 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:51 smithi142 ceph-mon[34625]: from='osd.5 [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:44:51.980 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:51 smithi142 ceph-mon[34625]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:44:51.980 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:51 smithi142 ceph-mon[34625]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:52.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:51 smithi098 ceph-mon[30092]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-09T08:44:52.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:51 smithi098 ceph-mon[30092]: osdmap e37: 6 total, 5 up, 6 in 2024-05-09T08:44:52.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:51 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:52.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:51 smithi098 ceph-mon[30092]: from='osd.5 [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:44:52.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:51 smithi098 ceph-mon[30092]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:44:52.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:51 smithi098 ceph-mon[30092]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:52.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:51 smithi105 ceph-mon[34498]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-09T08:44:52.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:51 smithi105 ceph-mon[34498]: osdmap e37: 6 total, 5 up, 6 in 2024-05-09T08:44:52.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:51 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:52.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:51 smithi105 ceph-mon[34498]: from='osd.5 [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:44:52.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:51 smithi105 ceph-mon[34498]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:44:52.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:51 smithi105 ceph-mon[34498]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:44:52.658 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 08:44:52 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T08:44:52.311+0000 7fcdf608b700 -1 osd.5 0 waiting for initial osdmap 2024-05-09T08:44:52.659 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 08:44:52 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T08:44:52.317+0000 7fcdede76700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:44:52.919 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-09T08:44:52.919 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-09T08:44:52.919 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-05-09T08:44:52.919 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-05-09T08:44:52.919 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113155 s, 927 MB/s 2024-05-09T08:44:52.920 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-05-09T08:44:53.120 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:52 smithi142 ceph-mon[34625]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-05-09T08:44:53.120 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:52 smithi142 ceph-mon[34625]: osdmap e38: 6 total, 5 up, 6 in 2024-05-09T08:44:53.120 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:53.120 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:53.120 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:52 smithi142 ceph-mon[34625]: from='osd.5 ' entity='osd.5' 2024-05-09T08:44:53.120 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:52 smithi142 ceph-mon[34625]: osd.5 [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789] boot 2024-05-09T08:44:53.120 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:52 smithi142 ceph-mon[34625]: osdmap e39: 6 total, 6 up, 6 in 2024-05-09T08:44:53.120 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:53.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:52 smithi105 ceph-mon[34498]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-05-09T08:44:53.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:52 smithi105 ceph-mon[34498]: osdmap e38: 6 total, 5 up, 6 in 2024-05-09T08:44:53.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:53.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:53.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:52 smithi105 ceph-mon[34498]: from='osd.5 ' entity='osd.5' 2024-05-09T08:44:53.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:52 smithi105 ceph-mon[34498]: osd.5 [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789] boot 2024-05-09T08:44:53.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:52 smithi105 ceph-mon[34498]: osdmap e39: 6 total, 6 up, 6 in 2024-05-09T08:44:53.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:52 smithi098 ceph-mon[30092]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-05-09T08:44:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:52 smithi098 ceph-mon[30092]: osdmap e38: 6 total, 5 up, 6 in 2024-05-09T08:44:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:52 smithi098 ceph-mon[30092]: from='osd.5 ' entity='osd.5' 2024-05-09T08:44:53.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:52 smithi098 ceph-mon[30092]: osd.5 [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789] boot 2024-05-09T08:44:53.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:52 smithi098 ceph-mon[30092]: osdmap e39: 6 total, 6 up, 6 in 2024-05-09T08:44:53.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:44:53.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-05-09T08:44:53.860 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:44:54.130 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:53 smithi098 ceph-mon[30092]: purged_snaps scrub starts 2024-05-09T08:44:54.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:53 smithi098 ceph-mon[30092]: purged_snaps scrub ok 2024-05-09T08:44:54.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:53 smithi098 ceph-mon[30092]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:44:54.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:54.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:54.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:54.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:54.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:54.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:54.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:54.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:54.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:53 smithi098 ceph-mon[30092]: osdmap e40: 6 total, 6 up, 6 in 2024-05-09T08:44:54.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:53 smithi142 ceph-mon[34625]: purged_snaps scrub starts 2024-05-09T08:44:54.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:53 smithi142 ceph-mon[34625]: purged_snaps scrub ok 2024-05-09T08:44:54.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:53 smithi142 ceph-mon[34625]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:44:54.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:54.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:54.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:54.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:54.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:54.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:54.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:54.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:54.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:53 smithi142 ceph-mon[34625]: osdmap e40: 6 total, 6 up, 6 in 2024-05-09T08:44:54.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:53 smithi105 ceph-mon[34498]: purged_snaps scrub starts 2024-05-09T08:44:54.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:53 smithi105 ceph-mon[34498]: purged_snaps scrub ok 2024-05-09T08:44:54.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:53 smithi105 ceph-mon[34498]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:44:54.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:54.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:54.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:54.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:54.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:54.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:54.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:54.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:44:54.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:53 smithi105 ceph-mon[34498]: osdmap e40: 6 total, 6 up, 6 in 2024-05-09T08:44:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:54 smithi098 ceph-mon[30092]: Detected new or changed devices on smithi142 2024-05-09T08:44:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:54 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi142 to 8544M 2024-05-09T08:44:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:54 smithi098 ceph-mon[30092]: osdmap e41: 6 total, 6 up, 6 in 2024-05-09T08:44:55.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:54 smithi142 ceph-mon[34625]: Detected new or changed devices on smithi142 2024-05-09T08:44:55.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:54 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi142 to 8544M 2024-05-09T08:44:55.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:54 smithi142 ceph-mon[34625]: osdmap e41: 6 total, 6 up, 6 in 2024-05-09T08:44:55.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:54 smithi105 ceph-mon[34498]: Detected new or changed devices on smithi142 2024-05-09T08:44:55.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:54 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi142 to 8544M 2024-05-09T08:44:55.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:54 smithi105 ceph-mon[34498]: osdmap e41: 6 total, 6 up, 6 in 2024-05-09T08:44:55.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.657+0000 7f831370d700 1 -- 172.21.15.98:0/1433419815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f830c101ea0 msgr2=0x7f830c109740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:55.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.657+0000 7f831370d700 1 --2- 172.21.15.98:0/1433419815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f830c101ea0 0x7f830c109740 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f830400b0a0 tx=0x7f830400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:44:55.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.657+0000 7f831370d700 1 -- 172.21.15.98:0/1433419815 shutdown_connections 2024-05-09T08:44:55.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.657+0000 7f831370d700 1 --2- 172.21.15.98:0/1433419815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f830c101ea0 0x7f830c109740 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:55.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.657+0000 7f831370d700 1 --2- 172.21.15.98:0/1433419815 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f830c102880 0x7f830c102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:55.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.657+0000 7f831370d700 1 --2- 172.21.15.98:0/1433419815 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f830c1014e0 0x7f830c101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:55.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.657+0000 7f831370d700 1 -- 172.21.15.98:0/1433419815 >> 172.21.15.98:0/1433419815 conn(0x7f830c0fbfd0 msgr2=0x7f830c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:55.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.657+0000 7f831370d700 1 -- 172.21.15.98:0/1433419815 shutdown_connections 2024-05-09T08:44:55.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.657+0000 7f831370d700 1 -- 172.21.15.98:0/1433419815 wait complete. 2024-05-09T08:44:55.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.658+0000 7f831370d700 1 Processor -- start 2024-05-09T08:44:55.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.658+0000 7f831370d700 1 -- start start 2024-05-09T08:44:55.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.658+0000 7f831370d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f830c1014e0 0x7f830c100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:55.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.658+0000 7f831370d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f830c101ea0 0x7f830c19d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:55.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.658+0000 7f831370d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f830c102880 0x7f830c19f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:55.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.658+0000 7f831370d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f830c10cec0 con 0x7f830c102880 2024-05-09T08:44:55.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.658+0000 7f831370d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f830c10cd40 con 0x7f830c101ea0 2024-05-09T08:44:55.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.658+0000 7f831370d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f830c10d040 con 0x7f830c1014e0 2024-05-09T08:44:55.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.658+0000 7f8311caa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f830c102880 0x7f830c19f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:55.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.658+0000 7f8311caa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f830c102880 0x7f830c19f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40890/0 (socket says 172.21.15.98:40890) 2024-05-09T08:44:55.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.658+0000 7f8311caa700 1 -- 172.21.15.98:0/3916748641 learned_addr learned my addr 172.21.15.98:0/3916748641 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:44:55.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.658+0000 7f83114a9700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f830c1014e0 0x7f830c100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:55.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.658+0000 7f8310ca8700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f830c101ea0 0x7f830c19d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:55.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.659+0000 7f8311caa700 1 -- 172.21.15.98:0/3916748641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f830c1014e0 msgr2=0x7f830c100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:55.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.659+0000 7f8311caa700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f830c1014e0 0x7f830c100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:55.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.659+0000 7f8311caa700 1 -- 172.21.15.98:0/3916748641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f830c101ea0 msgr2=0x7f830c19d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:55.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.659+0000 7f8311caa700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f830c101ea0 0x7f830c19d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:55.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.659+0000 7f8311caa700 1 -- 172.21.15.98:0/3916748641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f830c1a0150 con 0x7f830c102880 2024-05-09T08:44:55.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.659+0000 7f8310ca8700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f830c101ea0 0x7f830c19d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:44:55.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.659+0000 7f8311caa700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f830c102880 0x7f830c19f880 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8304007d90 tx=0x7f8304007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:55.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.659+0000 7f83027fc700 1 -- 172.21.15.98:0/3916748641 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8304021020 con 0x7f830c102880 2024-05-09T08:44:55.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.659+0000 7f83027fc700 1 -- 172.21.15.98:0/3916748641 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8304004e40 con 0x7f830c102880 2024-05-09T08:44:55.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.659+0000 7f83027fc700 1 -- 172.21.15.98:0/3916748641 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83040055f0 con 0x7f830c102880 2024-05-09T08:44:55.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.659+0000 7f831370d700 1 -- 172.21.15.98:0/3916748641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f830c1a03e0 con 0x7f830c102880 2024-05-09T08:44:55.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.659+0000 7f831370d700 1 -- 172.21.15.98:0/3916748641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f830c197280 con 0x7f830c102880 2024-05-09T08:44:55.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.660+0000 7f831370d700 1 -- 172.21.15.98:0/3916748641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f830c069010 con 0x7f830c102880 2024-05-09T08:44:55.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.662+0000 7f83027fc700 1 -- 172.21.15.98:0/3916748641 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8304024020 con 0x7f830c102880 2024-05-09T08:44:55.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.662+0000 7f83027fc700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f82f80a2c20 0x7f82f80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:55.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.662+0000 7f83114a9700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f82f80a2c20 0x7f82f80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:55.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.662+0000 7f83027fc700 1 -- 172.21.15.98:0/3916748641 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4562+0+0 (secure 0 0 0) 0x7f830400f020 con 0x7f830c102880 2024-05-09T08:44:55.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.663+0000 7f83114a9700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f82f80a2c20 0x7f82f80a50e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f82fc005950 tx=0x7f82fc00a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:55.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.666+0000 7f83027fc700 1 -- 172.21.15.98:0/3916748641 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f830401ae20 con 0x7f830c102880 2024-05-09T08:44:55.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:44:55.870+0000 7f831370d700 1 -- 172.21.15.98:0/3916748641 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f830c062760 con 0x7f82f80a2c20 2024-05-09T08:44:56.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:55 smithi098 ceph-mon[30092]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:44:56.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:44:56.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:44:56.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:56.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:55 smithi142 ceph-mon[34625]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:44:56.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:55 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:44:56.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:55 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:44:56.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:55 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:56.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:55 smithi105 ceph-mon[34498]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:44:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:55 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:44:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:55 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:44:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:55 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:57.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:56 smithi098 ceph-mon[30092]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:44:57.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:56 smithi098 ceph-mon[30092]: osdmap e42: 6 total, 6 up, 6 in 2024-05-09T08:44:57.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:56 smithi142 ceph-mon[34625]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:44:57.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:56 smithi142 ceph-mon[34625]: osdmap e42: 6 total, 6 up, 6 in 2024-05-09T08:44:57.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:56 smithi105 ceph-mon[34498]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:44:57.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:56 smithi105 ceph-mon[34498]: osdmap e42: 6 total, 6 up, 6 in 2024-05-09T08:44:58.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:57 smithi142 ceph-mon[34625]: osdmap e43: 6 total, 6 up, 6 in 2024-05-09T08:44:58.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:57 smithi142 ceph-mon[34625]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T08:44:58.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:57 smithi105 ceph-mon[34498]: osdmap e43: 6 total, 6 up, 6 in 2024-05-09T08:44:58.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:57 smithi105 ceph-mon[34498]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T08:44:58.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:57 smithi098 ceph-mon[30092]: osdmap e43: 6 total, 6 up, 6 in 2024-05-09T08:44:58.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:57 smithi098 ceph-mon[30092]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T08:44:59.155 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:58 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1061053956' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa58eef-6903-41c6-9cb2-5b80475736ea"}]: dispatch 2024-05-09T08:44:59.155 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:58 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1061053956' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aa58eef-6903-41c6-9cb2-5b80475736ea"}]': finished 2024-05-09T08:44:59.155 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:58 smithi105 ceph-mon[34498]: osdmap e44: 7 total, 6 up, 7 in 2024-05-09T08:44:59.155 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:58 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:44:59.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:58 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1061053956' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa58eef-6903-41c6-9cb2-5b80475736ea"}]: dispatch 2024-05-09T08:44:59.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:58 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1061053956' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aa58eef-6903-41c6-9cb2-5b80475736ea"}]': finished 2024-05-09T08:44:59.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:58 smithi142 ceph-mon[34625]: osdmap e44: 7 total, 6 up, 7 in 2024-05-09T08:44:59.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:58 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:44:59.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:58 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1061053956' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa58eef-6903-41c6-9cb2-5b80475736ea"}]: dispatch 2024-05-09T08:44:59.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:58 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1061053956' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aa58eef-6903-41c6-9cb2-5b80475736ea"}]': finished 2024-05-09T08:44:59.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:58 smithi098 ceph-mon[30092]: osdmap e44: 7 total, 6 up, 7 in 2024-05-09T08:44:59.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:58 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:00.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:59 smithi142 ceph-mon[34625]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-09T08:45:00.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:44:59 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1525171115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:45:00.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:59 smithi105 ceph-mon[34498]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-09T08:45:00.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:44:59 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1525171115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:45:00.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:59 smithi098 ceph-mon[30092]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-09T08:45:00.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:44:59 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1525171115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:45:02.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:02 smithi105 ceph-mon[34498]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T08:45:02.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:02 smithi098 ceph-mon[30092]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T08:45:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:02 smithi142 ceph-mon[34625]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T08:45:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:04 smithi105 ceph-mon[34498]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:04 smithi098 ceph-mon[30092]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:04 smithi142 ceph-mon[34625]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:06.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:06 smithi105 ceph-mon[34498]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:06.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:06 smithi098 ceph-mon[30092]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:06.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:06 smithi142 ceph-mon[34625]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:07.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:07 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-09T08:45:07.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:07 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:07.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:07 smithi105 ceph-mon[34498]: Deploying daemon osd.6 on smithi098 2024-05-09T08:45:07.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:07 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-09T08:45:07.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:07 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:07.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:07 smithi098 ceph-mon[30092]: Deploying daemon osd.6 on smithi098 2024-05-09T08:45:07.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:07 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-09T08:45:07.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:07 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:07.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:07 smithi142 ceph-mon[34625]: Deploying daemon osd.6 on smithi098 2024-05-09T08:45:08.480 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:08 smithi098 ceph-mon[30092]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:08.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:08 smithi105 ceph-mon[34498]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:08 smithi142 ceph-mon[34625]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:10.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:10 smithi105 ceph-mon[34498]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:10.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:10 smithi098 ceph-mon[30092]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:10.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:10 smithi142 ceph-mon[34625]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:12.442 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:12 smithi098 ceph-mon[30092]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:12.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:12 smithi105 ceph-mon[34498]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:12 smithi142 ceph-mon[34625]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:13.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:13 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:13.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:13 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:13.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:13 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:13.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:13 smithi098 ceph-mon[30092]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:13.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:13 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:13.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:13 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:13.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:13 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:13.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:13 smithi142 ceph-mon[34625]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:13.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:13 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:13.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:13 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:13.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:13 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:13.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:13 smithi105 ceph-mon[34498]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:16.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:16 smithi105 ceph-mon[34498]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:16 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:16 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:16 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:16 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:16 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:16.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:16 smithi098 ceph-mon[30092]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:16.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:16 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:16.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:16 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:16.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:16 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:16.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:16 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:16.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:16 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:16.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:45:16.613+0000 7f83027fc700 1 -- 172.21.15.98:0/3916748641 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f830c062760 con 0x7f82f80a2c20 2024-05-09T08:45:16.612 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 6 on host 'smithi098' 2024-05-09T08:45:16.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:45:16.615+0000 7f831370d700 1 -- 172.21.15.98:0/3916748641 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f82f80a2c20 msgr2=0x7f82f80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:16.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:45:16.615+0000 7f831370d700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f82f80a2c20 0x7f82f80a50e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f82fc005950 tx=0x7f82fc00a300 comp rx=0 tx=0).stop 2024-05-09T08:45:16.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:45:16.615+0000 7f831370d700 1 -- 172.21.15.98:0/3916748641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f830c102880 msgr2=0x7f830c19f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:16.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:45:16.615+0000 7f831370d700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f830c102880 0x7f830c19f880 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8304007d90 tx=0x7f8304007dc0 comp rx=0 tx=0).stop 2024-05-09T08:45:16.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:45:16.615+0000 7f831370d700 1 -- 172.21.15.98:0/3916748641 shutdown_connections 2024-05-09T08:45:16.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:45:16.615+0000 7f831370d700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f82f80a2c20 0x7f82f80a50e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:16.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:45:16.615+0000 7f831370d700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f830c102880 0x7f830c19f880 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:16.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:45:16.615+0000 7f831370d700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f830c1014e0 0x7f830c100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:16.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:45:16.615+0000 7f831370d700 1 --2- 172.21.15.98:0/3916748641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f830c101ea0 0x7f830c19d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:16.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:45:16.615+0000 7f831370d700 1 -- 172.21.15.98:0/3916748641 >> 172.21.15.98:0/3916748641 conn(0x7f830c0fbfd0 msgr2=0x7f830c105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:16.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:45:16.615+0000 7f831370d700 1 -- 172.21.15.98:0/3916748641 shutdown_connections 2024-05-09T08:45:16.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:45:16.615+0000 7f831370d700 1 -- 172.21.15.98:0/3916748641 wait complete. 2024-05-09T08:45:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:16 smithi142 ceph-mon[34625]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:17.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:17 smithi098 ceph-mon[30092]: from='osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-09T08:45:17.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:17.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:17.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:17.303 DEBUG:teuthology.orchestra.run.smithi098:osd.6> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.6.service 2024-05-09T08:45:17.306 INFO:tasks.cephadm:Deploying osd.7 on smithi105 with /dev/vg_nvme/lv_2... 2024-05-09T08:45:17.306 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-09T08:45:17.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:17 smithi105 ceph-mon[34498]: from='osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-09T08:45:17.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:17.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:17.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:17.550 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:45:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:17 smithi142 ceph-mon[34625]: from='osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-09T08:45:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:18.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:18 smithi105 ceph-mon[34498]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:18.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:18 smithi105 ceph-mon[34498]: from='osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-09T08:45:18.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:18 smithi105 ceph-mon[34498]: osdmap e45: 7 total, 6 up, 7 in 2024-05-09T08:45:18.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:18 smithi105 ceph-mon[34498]: from='osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-05-09T08:45:18.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:18 smithi098 ceph-mon[30092]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:18 smithi098 ceph-mon[30092]: from='osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-09T08:45:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:18 smithi098 ceph-mon[30092]: osdmap e45: 7 total, 6 up, 7 in 2024-05-09T08:45:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:18 smithi098 ceph-mon[30092]: from='osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-05-09T08:45:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:18.588 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 08:45:18 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T08:45:18.209+0000 7fa84ac97700 -1 osd.6 0 waiting for initial osdmap 2024-05-09T08:45:18.589 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 08:45:18 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T08:45:18.218+0000 7fa846a8a700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:45:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:18 smithi142 ceph-mon[34625]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:18 smithi142 ceph-mon[34625]: from='osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-09T08:45:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:18 smithi142 ceph-mon[34625]: osdmap e45: 7 total, 6 up, 7 in 2024-05-09T08:45:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:18 smithi142 ceph-mon[34625]: from='osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-05-09T08:45:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:19.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:19 smithi105 ceph-mon[34498]: from='osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-05-09T08:45:19.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:19 smithi105 ceph-mon[34498]: osdmap e46: 7 total, 6 up, 7 in 2024-05-09T08:45:19.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:19 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:19.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:19 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:19.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:19 smithi098 ceph-mon[30092]: from='osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-05-09T08:45:19.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:19 smithi098 ceph-mon[30092]: osdmap e46: 7 total, 6 up, 7 in 2024-05-09T08:45:19.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:19 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:19.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:19 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:19.561 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-09T08:45:19.561 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-09T08:45:19.561 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-05-09T08:45:19.562 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-05-09T08:45:19.562 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115198 s, 910 MB/s 2024-05-09T08:45:19.562 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-05-09T08:45:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:19 smithi142 ceph-mon[34625]: from='osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-05-09T08:45:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:19 smithi142 ceph-mon[34625]: osdmap e46: 7 total, 6 up, 7 in 2024-05-09T08:45:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:19 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:19 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:20.141 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_2 2024-05-09T08:45:20.388 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:45:20.475 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:20 smithi105 ceph-mon[34498]: purged_snaps scrub starts 2024-05-09T08:45:20.476 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:20 smithi105 ceph-mon[34498]: purged_snaps scrub ok 2024-05-09T08:45:20.476 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:20 smithi105 ceph-mon[34498]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:20.476 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:20 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:20.476 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:20 smithi105 ceph-mon[34498]: osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836] boot 2024-05-09T08:45:20.476 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:20 smithi105 ceph-mon[34498]: osdmap e47: 7 total, 7 up, 7 in 2024-05-09T08:45:20.476 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:20 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:20.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:20 smithi098 ceph-mon[30092]: purged_snaps scrub starts 2024-05-09T08:45:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:20 smithi098 ceph-mon[30092]: purged_snaps scrub ok 2024-05-09T08:45:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:20 smithi098 ceph-mon[30092]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:20 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:20 smithi098 ceph-mon[30092]: osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836] boot 2024-05-09T08:45:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:20 smithi098 ceph-mon[30092]: osdmap e47: 7 total, 7 up, 7 in 2024-05-09T08:45:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:20 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:20 smithi142 ceph-mon[34625]: purged_snaps scrub starts 2024-05-09T08:45:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:20 smithi142 ceph-mon[34625]: purged_snaps scrub ok 2024-05-09T08:45:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:20 smithi142 ceph-mon[34625]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:45:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:20 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:20.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:20 smithi142 ceph-mon[34625]: osd.6 [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836] boot 2024-05-09T08:45:20.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:20 smithi142 ceph-mon[34625]: osdmap e47: 7 total, 7 up, 7 in 2024-05-09T08:45:20.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:20 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:45:21.454 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:21 smithi105 ceph-mon[34498]: osdmap e48: 7 total, 7 up, 7 in 2024-05-09T08:45:21.454 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:21 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:21.454 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:21 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:21.454 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:21 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:21.454 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:21 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:21.454 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:21 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:21.455 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:21 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:21.455 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:21 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:21.455 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:21 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:21.455 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:21 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:21.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:21 smithi098 ceph-mon[30092]: osdmap e48: 7 total, 7 up, 7 in 2024-05-09T08:45:21.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:21 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:21.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:21 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:21.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:21 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:21.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:21 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:21.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:21 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:21.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:21 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:21.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:21 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:21.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:21 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:21.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:21 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:21.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:21 smithi142 ceph-mon[34625]: osdmap e48: 7 total, 7 up, 7 in 2024-05-09T08:45:21.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:21 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:21.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:21 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:21.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:21 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:21.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:21 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:21.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:21 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:21.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:21 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:21.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:21 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:21.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:21 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:21.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:21 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:21.996 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.994+0000 7ffb03750700 1 -- 172.21.15.105:0/2246029918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc103f20 msgr2=0x7ffafc10a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:21.996 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.994+0000 7ffb03750700 1 --2- 172.21.15.105:0/2246029918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc103f20 0x7ffafc10a7d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400b0d0 tx=0x7ffaf400b3e0 comp rx=0 tx=0).stop 2024-05-09T08:45:21.996 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.994+0000 7ffb03750700 1 -- 172.21.15.105:0/2246029918 shutdown_connections 2024-05-09T08:45:21.996 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.994+0000 7ffb03750700 1 --2- 172.21.15.105:0/2246029918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc103f20 0x7ffafc10a7d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:21.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.994+0000 7ffb03750700 1 --2- 172.21.15.105:0/2246029918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc102380 0x7ffafc1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:21.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.994+0000 7ffb03750700 1 --2- 172.21.15.105:0/2246029918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc103580 0x7ffafc1039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:21.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.994+0000 7ffb03750700 1 -- 172.21.15.105:0/2246029918 >> 172.21.15.105:0/2246029918 conn(0x7ffafc0fd920 msgr2=0x7ffafc0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:21.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.994+0000 7ffb03750700 1 -- 172.21.15.105:0/2246029918 shutdown_connections 2024-05-09T08:45:21.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.994+0000 7ffb03750700 1 -- 172.21.15.105:0/2246029918 wait complete. 2024-05-09T08:45:21.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.995+0000 7ffb03750700 1 Processor -- start 2024-05-09T08:45:21.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.995+0000 7ffb03750700 1 -- start start 2024-05-09T08:45:21.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.995+0000 7ffb03750700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc102380 0x7ffafc198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:21.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.995+0000 7ffb03750700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc103580 0x7ffafc198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:21.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.995+0000 7ffb03750700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc103f20 0x7ffafc19fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:21.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.995+0000 7ffb03750700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc10df70 con 0x7ffafc103580 2024-05-09T08:45:21.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.995+0000 7ffb03750700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc10ddf0 con 0x7ffafc102380 2024-05-09T08:45:21.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.995+0000 7ffb03750700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc10e0f0 con 0x7ffafc103f20 2024-05-09T08:45:21.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.995+0000 7ffb014ec700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc102380 0x7ffafc198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:21.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.995+0000 7ffb014ec700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc102380 0x7ffafc198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43320/0 (socket says 172.21.15.105:43320) 2024-05-09T08:45:21.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.995+0000 7ffb014ec700 1 -- 172.21.15.105:0/1108089178 learned_addr learned my addr 172.21.15.105:0/1108089178 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-05-09T08:45:21.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.995+0000 7ffb00ceb700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc103580 0x7ffafc198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:21.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.995+0000 7ffb01ced700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc103f20 0x7ffafc19fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:21.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.995+0000 7ffb014ec700 1 -- 172.21.15.105:0/1108089178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc103f20 msgr2=0x7ffafc19fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:21.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.996+0000 7ffb014ec700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc103f20 0x7ffafc19fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:21.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.996+0000 7ffb014ec700 1 -- 172.21.15.105:0/1108089178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc103580 msgr2=0x7ffafc198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:21.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.996+0000 7ffb014ec700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc103580 0x7ffafc198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:21.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.996+0000 7ffb014ec700 1 -- 172.21.15.105:0/1108089178 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffafc1a01b0 con 0x7ffafc102380 2024-05-09T08:45:21.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.996+0000 7ffb01ced700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc103f20 0x7ffafc19fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:45:21.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.996+0000 7ffb014ec700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc102380 0x7ffafc198550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800e970 tx=0x7ffaf800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:21.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.996+0000 7ffaf27fc700 1 -- 172.21.15.105:0/1108089178 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf800cbd0 con 0x7ffafc102380 2024-05-09T08:45:21.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.996+0000 7ffb03750700 1 -- 172.21.15.105:0/1108089178 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffafc1a04a0 con 0x7ffafc102380 2024-05-09T08:45:21.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.996+0000 7ffaf27fc700 1 -- 172.21.15.105:0/1108089178 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaf8004500 con 0x7ffafc102380 2024-05-09T08:45:21.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.996+0000 7ffaf27fc700 1 -- 172.21.15.105:0/1108089178 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8010430 con 0x7ffafc102380 2024-05-09T08:45:21.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.996+0000 7ffb03750700 1 -- 172.21.15.105:0/1108089178 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffafc1a0a30 con 0x7ffafc102380 2024-05-09T08:45:22.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.997+0000 7ffb03750700 1 -- 172.21.15.105:0/1108089178 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffafc069010 con 0x7ffafc102380 2024-05-09T08:45:22.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.998+0000 7ffaf27fc700 1 -- 172.21.15.105:0/1108089178 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ffaf8003fa0 con 0x7ffafc102380 2024-05-09T08:45:22.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.998+0000 7ffaf27fc700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ffae80a2c20 0x7ffae80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.999+0000 7ffaf27fc700 1 -- 172.21.15.105:0/1108089178 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7ffaf80ce620 con 0x7ffafc102380 2024-05-09T08:45:22.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.999+0000 7ffb00ceb700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ffae80a2c20 0x7ffae80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:21.999+0000 7ffb00ceb700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ffae80a2c20 0x7ffae80a50e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffaec005950 tx=0x7ffaec0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:22.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:22.003+0000 7ffaf27fc700 1 -- 172.21.15.105:0/1108089178 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffaf809c290 con 0x7ffafc102380 2024-05-09T08:45:22.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:22.204+0000 7ffb03750700 1 -- 172.21.15.105:0/1108089178 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ffafc062760 con 0x7ffae80a2c20 2024-05-09T08:45:22.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:22 smithi105 ceph-mon[34498]: Detected new or changed devices on smithi098 2024-05-09T08:45:22.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:22 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi098 to 5696M 2024-05-09T08:45:22.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:22 smithi105 ceph-mon[34498]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:22.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:45:22.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:45:22.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:22.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:22 smithi098 ceph-mon[30092]: Detected new or changed devices on smithi098 2024-05-09T08:45:22.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:22 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi098 to 5696M 2024-05-09T08:45:22.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:22 smithi098 ceph-mon[30092]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:22.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:45:22.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:45:22.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:22 smithi142 ceph-mon[34625]: Detected new or changed devices on smithi098 2024-05-09T08:45:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:22 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi098 to 5696M 2024-05-09T08:45:22.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:22 smithi142 ceph-mon[34625]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:22.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:45:22.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:45:22.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:23.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:23 smithi105 ceph-mon[34498]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:45:23.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:23 smithi098 ceph-mon[30092]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:45:23.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:23 smithi142 ceph-mon[34625]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:45:24.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:24 smithi105 ceph-mon[34498]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:24.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:24 smithi098 ceph-mon[30092]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:24 smithi142 ceph-mon[34625]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:25.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:25 smithi105 ceph-mon[34498]: from='client.? 172.21.15.105:0/3172417805' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72436c50-ea38-4783-a40f-2312deff10e2"}]: dispatch 2024-05-09T08:45:25.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:25 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72436c50-ea38-4783-a40f-2312deff10e2"}]: dispatch 2024-05-09T08:45:25.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:25 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72436c50-ea38-4783-a40f-2312deff10e2"}]': finished 2024-05-09T08:45:25.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:25 smithi105 ceph-mon[34498]: osdmap e49: 8 total, 7 up, 8 in 2024-05-09T08:45:25.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:25 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:25.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:25 smithi098 ceph-mon[30092]: from='client.? 172.21.15.105:0/3172417805' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72436c50-ea38-4783-a40f-2312deff10e2"}]: dispatch 2024-05-09T08:45:25.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:25 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72436c50-ea38-4783-a40f-2312deff10e2"}]: dispatch 2024-05-09T08:45:25.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:25 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72436c50-ea38-4783-a40f-2312deff10e2"}]': finished 2024-05-09T08:45:25.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:25 smithi098 ceph-mon[30092]: osdmap e49: 8 total, 7 up, 8 in 2024-05-09T08:45:25.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:25 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:25.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:25 smithi142 ceph-mon[34625]: from='client.? 172.21.15.105:0/3172417805' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72436c50-ea38-4783-a40f-2312deff10e2"}]: dispatch 2024-05-09T08:45:25.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:25 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72436c50-ea38-4783-a40f-2312deff10e2"}]: dispatch 2024-05-09T08:45:25.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:25 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72436c50-ea38-4783-a40f-2312deff10e2"}]': finished 2024-05-09T08:45:25.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:25 smithi142 ceph-mon[34625]: osdmap e49: 8 total, 7 up, 8 in 2024-05-09T08:45:25.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:25 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:26.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:26 smithi105 ceph-mon[34498]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:26.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:26 smithi105 ceph-mon[34498]: from='client.? 172.21.15.105:0/2975903996' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:45:26.562 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:26 smithi142 ceph-mon[34625]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:26.562 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:26 smithi142 ceph-mon[34625]: from='client.? 172.21.15.105:0/2975903996' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:45:26.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:26 smithi098 ceph-mon[30092]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:26.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:26 smithi098 ceph-mon[30092]: from='client.? 172.21.15.105:0/2975903996' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:45:28.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:28 smithi105 ceph-mon[34498]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:28.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:28 smithi098 ceph-mon[30092]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:28.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:28 smithi142 ceph-mon[34625]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:30.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:30 smithi105 ceph-mon[34498]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:30.584 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:30 smithi098 ceph-mon[30092]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:30 smithi142 ceph-mon[34625]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:32.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:32 smithi105 ceph-mon[34498]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:32.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:32 smithi098 ceph-mon[30092]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:32.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:32 smithi142 ceph-mon[34625]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:33.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:33 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-09T08:45:33.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:33 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:33.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:33 smithi105 ceph-mon[34498]: Deploying daemon osd.7 on smithi105 2024-05-09T08:45:33.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:33 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-09T08:45:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:33 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:33.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:33 smithi098 ceph-mon[30092]: Deploying daemon osd.7 on smithi105 2024-05-09T08:45:33.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:33 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-09T08:45:33.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:33 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:33.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:33 smithi142 ceph-mon[34625]: Deploying daemon osd.7 on smithi105 2024-05-09T08:45:34.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:34 smithi105 ceph-mon[34498]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:34.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:34 smithi098 ceph-mon[30092]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:34.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:34 smithi142 ceph-mon[34625]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:36.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:36 smithi105 ceph-mon[34498]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:36.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:36 smithi098 ceph-mon[30092]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:36 smithi142 ceph-mon[34625]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:38.367 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:38 smithi105 ceph-mon[34498]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:38.367 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:38.367 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:38.368 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:38.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:38 smithi098 ceph-mon[30092]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:38.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:38.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:38.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:38.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:38 smithi142 ceph-mon[34625]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:38.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:38.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:38.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:40.323 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:40 smithi105 ceph-mon[34498]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:40.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:40 smithi098 ceph-mon[30092]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:40 smithi142 ceph-mon[34625]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:41.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:41.522+0000 7ffaf27fc700 1 -- 172.21.15.105:0/1108089178 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ffafc062760 con 0x7ffae80a2c20 2024-05-09T08:45:41.521 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 7 on host 'smithi105' 2024-05-09T08:45:41.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:41.524+0000 7ffb03750700 1 -- 172.21.15.105:0/1108089178 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ffae80a2c20 msgr2=0x7ffae80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:41.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:41.524+0000 7ffb03750700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ffae80a2c20 0x7ffae80a50e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffaec005950 tx=0x7ffaec0058e0 comp rx=0 tx=0).stop 2024-05-09T08:45:41.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:41.524+0000 7ffb03750700 1 -- 172.21.15.105:0/1108089178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc102380 msgr2=0x7ffafc198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:41.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:41.524+0000 7ffb03750700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc102380 0x7ffafc198550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800e970 tx=0x7ffaf800ed30 comp rx=0 tx=0).stop 2024-05-09T08:45:41.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:41.524+0000 7ffb03750700 1 -- 172.21.15.105:0/1108089178 shutdown_connections 2024-05-09T08:45:41.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:41.524+0000 7ffb03750700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ffae80a2c20 0x7ffae80a50e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:41.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:41.524+0000 7ffb03750700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc102380 0x7ffafc198550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:41.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:41.524+0000 7ffb03750700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc103f20 0x7ffafc19fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:41.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:41.524+0000 7ffb03750700 1 --2- 172.21.15.105:0/1108089178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc103580 0x7ffafc198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:41.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:41.524+0000 7ffb03750700 1 -- 172.21.15.105:0/1108089178 >> 172.21.15.105:0/1108089178 conn(0x7ffafc0fd920 msgr2=0x7ffafc106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:41.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:41.525+0000 7ffb03750700 1 -- 172.21.15.105:0/1108089178 shutdown_connections 2024-05-09T08:45:41.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:45:41.525+0000 7ffb03750700 1 -- 172.21.15.105:0/1108089178 wait complete. 2024-05-09T08:45:42.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:42.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:42.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:41 smithi098 ceph-mon[30092]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:42.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:42.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:41 smithi098 ceph-mon[30092]: from='osd.7 [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:45:42.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:41 smithi098 ceph-mon[30092]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:45:42.140 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.140 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.140 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:42.140 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:42.140 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.140 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:41 smithi105 ceph-mon[34498]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:42.140 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.140 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.140 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:42.140 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:41 smithi105 ceph-mon[34498]: from='osd.7 [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:45:42.141 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:41 smithi105 ceph-mon[34498]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:45:42.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:42.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:42.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:41 smithi142 ceph-mon[34625]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:42.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:42.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:42.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:41 smithi142 ceph-mon[34625]: from='osd.7 [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:45:42.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:41 smithi142 ceph-mon[34625]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:45:42.166 DEBUG:teuthology.orchestra.run.smithi105:osd.7> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.7.service 2024-05-09T08:45:42.168 INFO:tasks.cephadm:Deploying osd.8 on smithi142 with /dev/vg_nvme/lv_2... 2024-05-09T08:45:42.168 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-09T08:45:42.414 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:45:43.606 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:43 smithi105 ceph-mon[34498]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-09T08:45:43.606 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:43 smithi105 ceph-mon[34498]: from='osd.7 [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:45:43.606 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:43 smithi105 ceph-mon[34498]: osdmap e50: 8 total, 7 up, 8 in 2024-05-09T08:45:43.606 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:43 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:43.606 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:43 smithi105 ceph-mon[34498]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:45:43.606 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:43 smithi105 ceph-mon[34498]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:43.607 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 08:45:43 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T08:45:43.504+0000 7fcc8fd90700 -1 osd.7 0 waiting for initial osdmap 2024-05-09T08:45:43.607 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 08:45:43 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T08:45:43.512+0000 7fcc8bb83700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:45:43.826 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:43 smithi142 ceph-mon[34625]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-09T08:45:43.826 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:43 smithi142 ceph-mon[34625]: from='osd.7 [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:45:43.826 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:43 smithi142 ceph-mon[34625]: osdmap e50: 8 total, 7 up, 8 in 2024-05-09T08:45:43.826 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:43 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:43.826 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:43 smithi142 ceph-mon[34625]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:45:43.826 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:43 smithi142 ceph-mon[34625]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:43.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:43 smithi098 ceph-mon[30092]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-09T08:45:43.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:43 smithi098 ceph-mon[30092]: from='osd.7 [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:45:43.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:43 smithi098 ceph-mon[30092]: osdmap e50: 8 total, 7 up, 8 in 2024-05-09T08:45:43.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:43 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:43.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:43 smithi098 ceph-mon[30092]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:45:43.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:43 smithi098 ceph-mon[30092]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:45:44.528 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-09T08:45:44.528 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-09T08:45:44.528 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2024-05-09T08:45:44.528 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2024-05-09T08:45:44.528 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114279 s, 918 MB/s 2024-05-09T08:45:44.528 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2024-05-09T08:45:44.723 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:44 smithi105 ceph-mon[34498]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-09T08:45:44.723 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:44 smithi105 ceph-mon[34498]: osdmap e51: 8 total, 7 up, 8 in 2024-05-09T08:45:44.723 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:44 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:44.723 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:44 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:44 smithi098 ceph-mon[30092]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-09T08:45:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:44 smithi098 ceph-mon[30092]: osdmap e51: 8 total, 7 up, 8 in 2024-05-09T08:45:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:44 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:44 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:44.857 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:44 smithi142 ceph-mon[34625]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-09T08:45:44.857 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:44 smithi142 ceph-mon[34625]: osdmap e51: 8 total, 7 up, 8 in 2024-05-09T08:45:44.857 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:44 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:44.857 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:44 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:45.210 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi142:vg_nvme/lv_2 2024-05-09T08:45:45.451 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:45:45.828 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: purged_snaps scrub starts 2024-05-09T08:45:45.828 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: purged_snaps scrub ok 2024-05-09T08:45:45.828 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: osd.7 [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764] boot 2024-05-09T08:45:45.829 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: osdmap e52: 8 total, 8 up, 8 in 2024-05-09T08:45:45.829 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:45.829 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: Detected new or changed devices on smithi105 2024-05-09T08:45:45.829 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:45.829 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:45.829 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:45.829 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:45.829 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:45.829 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi105 to 5696M 2024-05-09T08:45:45.829 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:45.829 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:45.829 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:45.829 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:45.830 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:45 smithi142 ceph-mon[34625]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:45.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: purged_snaps scrub starts 2024-05-09T08:45:45.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: purged_snaps scrub ok 2024-05-09T08:45:45.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: osd.7 [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764] boot 2024-05-09T08:45:45.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: osdmap e52: 8 total, 8 up, 8 in 2024-05-09T08:45:45.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:45.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: Detected new or changed devices on smithi105 2024-05-09T08:45:45.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:45.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:45.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:45.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:45.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:45.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi105 to 5696M 2024-05-09T08:45:45.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:45.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:45.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:45.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:45.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:45 smithi098 ceph-mon[30092]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:45.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: purged_snaps scrub starts 2024-05-09T08:45:45.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: purged_snaps scrub ok 2024-05-09T08:45:45.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: osd.7 [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764] boot 2024-05-09T08:45:45.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: osdmap e52: 8 total, 8 up, 8 in 2024-05-09T08:45:45.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:45:45.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: Detected new or changed devices on smithi105 2024-05-09T08:45:45.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:45.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:45.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:45.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:45.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:45:45.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi105 to 5696M 2024-05-09T08:45:45.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:45.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:45.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:45.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:45:45.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:45 smithi105 ceph-mon[34498]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:46.826 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:46 smithi142 ceph-mon[34625]: osdmap e53: 8 total, 8 up, 8 in 2024-05-09T08:45:46.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:46 smithi098 ceph-mon[30092]: osdmap e53: 8 total, 8 up, 8 in 2024-05-09T08:45:46.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:46 smithi105 ceph-mon[34498]: osdmap e53: 8 total, 8 up, 8 in 2024-05-09T08:45:47.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.039+0000 7f85a5368700 1 -- 172.21.15.142:0/1316147901 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85a0100630 msgr2=0x7f85a0100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:47.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.039+0000 7f85a5368700 1 --2- 172.21.15.142:0/1316147901 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85a0100630 0x7f85a0100a50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8590009990 tx=0x7f8590009ca0 comp rx=0 tx=0).stop 2024-05-09T08:45:47.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.039+0000 7f85a5368700 1 -- 172.21.15.142:0/1316147901 shutdown_connections 2024-05-09T08:45:47.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.039+0000 7f85a5368700 1 --2- 172.21.15.142:0/1316147901 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a0103fb0 0x7f85a010a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:47.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.039+0000 7f85a5368700 1 --2- 172.21.15.142:0/1316147901 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85a0100630 0x7f85a0100a50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:47.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.039+0000 7f85a5368700 1 --2- 172.21.15.142:0/1316147901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0103610 0x7f85a0103a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:47.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.039+0000 7f85a5368700 1 -- 172.21.15.142:0/1316147901 >> 172.21.15.142:0/1316147901 conn(0x7f85a00fbfb0 msgr2=0x7f85a00fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:47.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.039+0000 7f85a5368700 1 -- 172.21.15.142:0/1316147901 shutdown_connections 2024-05-09T08:45:47.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.039+0000 7f85a5368700 1 -- 172.21.15.142:0/1316147901 wait complete. 2024-05-09T08:45:47.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.040+0000 7f85a5368700 1 Processor -- start 2024-05-09T08:45:47.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.040+0000 7f85a5368700 1 -- start start 2024-05-09T08:45:47.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.040+0000 7f85a5368700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0100630 0x7f85a0198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:47.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.040+0000 7f85a5368700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a0103610 0x7f85a0198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:47.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.040+0000 7f85a5368700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85a0103fb0 0x7f85a019fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:47.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.040+0000 7f85a5368700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85a010df30 con 0x7f85a0100630 2024-05-09T08:45:47.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.040+0000 7f85a5368700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f85a010ddb0 con 0x7f85a0103610 2024-05-09T08:45:47.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.040+0000 7f85a5368700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f85a010e0b0 con 0x7f85a0103fb0 2024-05-09T08:45:47.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.040+0000 7f859f7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85a0103fb0 0x7f85a019fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:47.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.040+0000 7f859f7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85a0103fb0 0x7f85a019fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34642/0 (socket says 172.21.15.142:34642) 2024-05-09T08:45:47.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.040+0000 7f859f7fe700 1 -- 172.21.15.142:0/2701082959 learned_addr learned my addr 172.21.15.142:0/2701082959 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-05-09T08:45:47.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.041+0000 7f859effd700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0100630 0x7f85a0198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:47.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.041+0000 7f85967fc700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a0103610 0x7f85a0198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:47.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.041+0000 7f859f7fe700 1 -- 172.21.15.142:0/2701082959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a0103610 msgr2=0x7f85a0198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:47.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.041+0000 7f859f7fe700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a0103610 0x7f85a0198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:47.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.041+0000 7f859f7fe700 1 -- 172.21.15.142:0/2701082959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0100630 msgr2=0x7f85a0198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:47.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.041+0000 7f859f7fe700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0100630 0x7f85a0198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:47.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.041+0000 7f859f7fe700 1 -- 172.21.15.142:0/2701082959 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a01a01a0 con 0x7f85a0103fb0 2024-05-09T08:45:47.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.041+0000 7f859effd700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0100630 0x7f85a0198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:45:47.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.041+0000 7f859f7fe700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85a0103fb0 0x7f85a019fb00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8598007f10 tx=0x7f859800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:47.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.041+0000 7f859cff9700 1 -- 172.21.15.142:0/2701082959 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598004bb0 con 0x7f85a0103fb0 2024-05-09T08:45:47.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.041+0000 7f85a5368700 1 -- 172.21.15.142:0/2701082959 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a01a0490 con 0x7f85a0103fb0 2024-05-09T08:45:47.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.041+0000 7f859cff9700 1 -- 172.21.15.142:0/2701082959 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8598004d50 con 0x7f85a0103fb0 2024-05-09T08:45:47.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.041+0000 7f859cff9700 1 -- 172.21.15.142:0/2701082959 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598005a60 con 0x7f85a0103fb0 2024-05-09T08:45:47.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.041+0000 7f85a5368700 1 -- 172.21.15.142:0/2701082959 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a01a09d0 con 0x7f85a0103fb0 2024-05-09T08:45:47.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.042+0000 7f85a5368700 1 -- 172.21.15.142:0/2701082959 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a0069010 con 0x7f85a0103fb0 2024-05-09T08:45:47.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.043+0000 7f859cff9700 1 -- 172.21.15.142:0/2701082959 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f859801a020 con 0x7f85a0103fb0 2024-05-09T08:45:47.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.044+0000 7f859cff9700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f858c0a2c20 0x7f858c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:47.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.044+0000 7f859cff9700 1 -- 172.21.15.142:0/2701082959 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7f85980cdcf0 con 0x7f85a0103fb0 2024-05-09T08:45:47.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.044+0000 7f859effd700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f858c0a2c20 0x7f858c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:47.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.045+0000 7f859effd700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f858c0a2c20 0x7f858c0a50e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f859000b7b0 tx=0x7f8590005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:47.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.048+0000 7f859cff9700 1 -- 172.21.15.142:0/2701082959 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f85980d2070 con 0x7f85a0103fb0 2024-05-09T08:45:47.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:45:47.252+0000 7f85a5368700 1 -- 172.21.15.142:0/2701082959 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f85a0062760 con 0x7f858c0a2c20 2024-05-09T08:45:47.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:47 smithi098 ceph-mon[30092]: osdmap e54: 8 total, 8 up, 8 in 2024-05-09T08:45:47.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:47 smithi098 ceph-mon[30092]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:47.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:47 smithi098 ceph-mon[30092]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:45:47.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:47 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:45:47.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:47 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:45:47.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:47 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:47.859 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:47 smithi142 ceph-mon[34625]: osdmap e54: 8 total, 8 up, 8 in 2024-05-09T08:45:47.859 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:47 smithi142 ceph-mon[34625]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:47.859 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:47 smithi142 ceph-mon[34625]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:45:47.859 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:47 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:45:47.859 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:47 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:45:47.859 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:47 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:47.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:47 smithi105 ceph-mon[34498]: osdmap e54: 8 total, 8 up, 8 in 2024-05-09T08:45:47.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:47 smithi105 ceph-mon[34498]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:47.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:47 smithi105 ceph-mon[34498]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:45:47.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:47 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:45:47.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:47 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:45:47.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:47 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:48.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:48 smithi098 ceph-mon[30092]: osdmap e55: 8 total, 8 up, 8 in 2024-05-09T08:45:48.849 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:48 smithi142 ceph-mon[34625]: osdmap e55: 8 total, 8 up, 8 in 2024-05-09T08:45:48.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:48 smithi105 ceph-mon[34498]: osdmap e55: 8 total, 8 up, 8 in 2024-05-09T08:45:49.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:49 smithi098 ceph-mon[30092]: osdmap e56: 8 total, 8 up, 8 in 2024-05-09T08:45:49.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:49 smithi098 ceph-mon[30092]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:49.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:49 smithi142 ceph-mon[34625]: osdmap e56: 8 total, 8 up, 8 in 2024-05-09T08:45:49.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:49 smithi142 ceph-mon[34625]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:49.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:49 smithi105 ceph-mon[34498]: osdmap e56: 8 total, 8 up, 8 in 2024-05-09T08:45:49.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:49 smithi105 ceph-mon[34498]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:50 smithi142 ceph-mon[34625]: from='client.? 172.21.15.142:0/2524996585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0716bad-82d8-4713-b945-f640664a0783"}]: dispatch 2024-05-09T08:45:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:50 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0716bad-82d8-4713-b945-f640664a0783"}]: dispatch 2024-05-09T08:45:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:50 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0716bad-82d8-4713-b945-f640664a0783"}]': finished 2024-05-09T08:45:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:50 smithi142 ceph-mon[34625]: osdmap e57: 9 total, 8 up, 9 in 2024-05-09T08:45:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:45:50.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:50 smithi098 ceph-mon[30092]: from='client.? 172.21.15.142:0/2524996585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0716bad-82d8-4713-b945-f640664a0783"}]: dispatch 2024-05-09T08:45:50.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:50 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0716bad-82d8-4713-b945-f640664a0783"}]: dispatch 2024-05-09T08:45:50.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:50 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0716bad-82d8-4713-b945-f640664a0783"}]': finished 2024-05-09T08:45:50.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:50 smithi098 ceph-mon[30092]: osdmap e57: 9 total, 8 up, 9 in 2024-05-09T08:45:50.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:45:50.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:50 smithi105 ceph-mon[34498]: from='client.? 172.21.15.142:0/2524996585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0716bad-82d8-4713-b945-f640664a0783"}]: dispatch 2024-05-09T08:45:50.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:50 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0716bad-82d8-4713-b945-f640664a0783"}]: dispatch 2024-05-09T08:45:50.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:50 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0716bad-82d8-4713-b945-f640664a0783"}]': finished 2024-05-09T08:45:50.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:50 smithi105 ceph-mon[34498]: osdmap e57: 9 total, 8 up, 9 in 2024-05-09T08:45:50.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:45:51.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:51 smithi098 ceph-mon[30092]: from='client.? 172.21.15.142:0/21767972' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:45:51.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:51 smithi098 ceph-mon[30092]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:51.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:51 smithi142 ceph-mon[34625]: from='client.? 172.21.15.142:0/21767972' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:45:51.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:51 smithi142 ceph-mon[34625]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:51.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:51 smithi105 ceph-mon[34498]: from='client.? 172.21.15.142:0/21767972' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:45:51.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:51 smithi105 ceph-mon[34498]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:54.368 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:54 smithi142 ceph-mon[34625]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:54.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:54 smithi105 ceph-mon[34498]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:54.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:54 smithi098 ceph-mon[30092]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:56.244 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:56 smithi142 ceph-mon[34625]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:56.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:56 smithi105 ceph-mon[34498]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:56 smithi098 ceph-mon[30092]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:58.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:58 smithi105 ceph-mon[34498]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:58.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:58 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-09T08:45:58.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:58 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:58.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:45:58 smithi105 ceph-mon[34498]: Deploying daemon osd.8 on smithi142 2024-05-09T08:45:58.522 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:58 smithi142 ceph-mon[34625]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:58.522 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:58 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-09T08:45:58.522 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:58 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:58.522 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:45:58 smithi142 ceph-mon[34625]: Deploying daemon osd.8 on smithi142 2024-05-09T08:45:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:58 smithi098 ceph-mon[30092]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:45:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:58 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-09T08:45:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:58 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:45:58 smithi098 ceph-mon[30092]: Deploying daemon osd.8 on smithi142 2024-05-09T08:46:00.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:00 smithi098 ceph-mon[30092]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:00.406 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:00 smithi142 ceph-mon[34625]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:00.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:00 smithi105 ceph-mon[34498]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:02.471 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:02 smithi142 ceph-mon[34625]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:02.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:02 smithi105 ceph-mon[34498]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:02 smithi098 ceph-mon[30092]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:04.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:03 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:04.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:03 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:04.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:03 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:04.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:03 smithi142 ceph-mon[34625]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:04.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:03 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:04.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:03 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:04.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:03 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:04.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:03 smithi105 ceph-mon[34498]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:04.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:03 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:04.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:03 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:04.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:03 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:04.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:03 smithi098 ceph-mon[30092]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:06.380 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:06 smithi142 ceph-mon[34625]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:06.380 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:06.381 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:06.381 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:06.381 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:06.381 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:06 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:06.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:06 smithi105 ceph-mon[34498]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:06.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:06 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:06.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:06 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:06.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:06 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:06.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:06 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:06.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:06 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:06.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:06 smithi098 ceph-mon[30092]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:06.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:06.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:06.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:06.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:06.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:06 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 8 on host 'smithi142' 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:46:06.774+0000 7f859cff9700 1 -- 172.21.15.142:0/2701082959 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f85a0062760 con 0x7f858c0a2c20 2024-05-09T08:46:06.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:46:06.777+0000 7f85a5368700 1 -- 172.21.15.142:0/2701082959 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f858c0a2c20 msgr2=0x7f858c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:06.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:46:06.777+0000 7f85a5368700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f858c0a2c20 0x7f858c0a50e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f859000b7b0 tx=0x7f8590005fd0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:46:06.777+0000 7f85a5368700 1 -- 172.21.15.142:0/2701082959 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85a0103fb0 msgr2=0x7f85a019fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:46:06.777+0000 7f85a5368700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85a0103fb0 0x7f85a019fb00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8598007f10 tx=0x7f859800d3b0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:46:06.777+0000 7f85a5368700 1 -- 172.21.15.142:0/2701082959 shutdown_connections 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:46:06.777+0000 7f85a5368700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f858c0a2c20 0x7f858c0a50e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:46:06.777+0000 7f85a5368700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85a0103fb0 0x7f85a019fb00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:46:06.777+0000 7f85a5368700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0100630 0x7f85a0198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:46:06.777+0000 7f85a5368700 1 --2- 172.21.15.142:0/2701082959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a0103610 0x7f85a0198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:46:06.777+0000 7f85a5368700 1 -- 172.21.15.142:0/2701082959 >> 172.21.15.142:0/2701082959 conn(0x7f85a00fbfb0 msgr2=0x7f85a0106910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:46:06.777+0000 7f85a5368700 1 -- 172.21.15.142:0/2701082959 shutdown_connections 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:46:06.777+0000 7f85a5368700 1 -- 172.21.15.142:0/2701082959 wait complete. 2024-05-09T08:46:07.453 DEBUG:teuthology.orchestra.run.smithi142:osd.8> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.8.service 2024-05-09T08:46:07.455 INFO:tasks.cephadm:Deploying osd.9 on smithi098 with /dev/vg_nvme/lv_1... 2024-05-09T08:46:07.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-09T08:46:07.706 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:46:07.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:07 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:07.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:07 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:07.831 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:07 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:07.831 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:07 smithi098 ceph-mon[30092]: from='osd.8 [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:46:07.831 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:07 smithi098 ceph-mon[30092]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:46:07.831 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:07 smithi098 ceph-mon[30092]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:07.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:07 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:07.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:07 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:07.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:07 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:07.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:07 smithi105 ceph-mon[34498]: from='osd.8 [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:46:07.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:07 smithi105 ceph-mon[34498]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:46:07.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:07 smithi105 ceph-mon[34498]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:08.004 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:07 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:08.004 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:07 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:08.004 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:07 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:08.004 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:07 smithi142 ceph-mon[34625]: from='osd.8 [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:46:08.004 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:07 smithi142 ceph-mon[34625]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:46:08.004 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:07 smithi142 ceph-mon[34625]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:09.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:08 smithi098 ceph-mon[30092]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-09T08:46:09.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:08 smithi098 ceph-mon[30092]: osdmap e58: 9 total, 8 up, 9 in 2024-05-09T08:46:09.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:08 smithi098 ceph-mon[30092]: from='osd.8 [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:46:09.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:08 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:09.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:08 smithi098 ceph-mon[30092]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:46:09.021 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:08 smithi142 ceph-mon[34625]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-09T08:46:09.021 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:08 smithi142 ceph-mon[34625]: osdmap e58: 9 total, 8 up, 9 in 2024-05-09T08:46:09.021 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:08 smithi142 ceph-mon[34625]: from='osd.8 [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:46:09.021 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:08 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:09.021 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:08 smithi142 ceph-mon[34625]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:46:09.021 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 08:46:08 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T08:46:08.742+0000 7fb2fd80e700 -1 osd.8 0 waiting for initial osdmap 2024-05-09T08:46:09.021 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 08:46:08 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T08:46:08.753+0000 7fb2f9601700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:46:09.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:08 smithi105 ceph-mon[34498]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-09T08:46:09.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:08 smithi105 ceph-mon[34498]: osdmap e58: 9 total, 8 up, 9 in 2024-05-09T08:46:09.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:08 smithi105 ceph-mon[34498]: from='osd.8 [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:46:09.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:08 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:09.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:08 smithi105 ceph-mon[34498]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:46:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-09T08:46:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-09T08:46:09.941 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-05-09T08:46:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-05-09T08:46:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115407 s, 909 MB/s 2024-05-09T08:46:09.942 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 2024-05-09T08:46:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-05-09T08:46:10.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:09 smithi098 ceph-mon[30092]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-05-09T08:46:10.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:09 smithi098 ceph-mon[30092]: osdmap e59: 9 total, 8 up, 9 in 2024-05-09T08:46:10.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:09 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:10.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:09 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:10.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:09 smithi098 ceph-mon[30092]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:10.099 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:09 smithi142 ceph-mon[34625]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-05-09T08:46:10.099 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:09 smithi142 ceph-mon[34625]: osdmap e59: 9 total, 8 up, 9 in 2024-05-09T08:46:10.099 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:09 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:10.099 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:09 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:10.099 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:09 smithi142 ceph-mon[34625]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:10.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:09 smithi105 ceph-mon[34498]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-05-09T08:46:10.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:09 smithi105 ceph-mon[34498]: osdmap e59: 9 total, 8 up, 9 in 2024-05-09T08:46:10.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:09 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:10.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:09 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:10.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:09 smithi105 ceph-mon[34498]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:46:10.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2024-05-09T08:46:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:46:10.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: purged_snaps scrub starts 2024-05-09T08:46:10.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: purged_snaps scrub ok 2024-05-09T08:46:10.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:10.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: osd.8 [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901] boot 2024-05-09T08:46:10.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: osdmap e60: 9 total, 9 up, 9 in 2024-05-09T08:46:10.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:10.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:10.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:10.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:10.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:10.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:10.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:10.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:10.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:10.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:10.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: purged_snaps scrub starts 2024-05-09T08:46:10.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: purged_snaps scrub ok 2024-05-09T08:46:10.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:10.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: osd.8 [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901] boot 2024-05-09T08:46:10.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: osdmap e60: 9 total, 9 up, 9 in 2024-05-09T08:46:10.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:10.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:10.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:10.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:10.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:10.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:10.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:10.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:10.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:10.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:11.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: purged_snaps scrub starts 2024-05-09T08:46:11.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: purged_snaps scrub ok 2024-05-09T08:46:11.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:11.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: osd.8 [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901] boot 2024-05-09T08:46:11.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: osdmap e60: 9 total, 9 up, 9 in 2024-05-09T08:46:11.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:46:11.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:11.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:11.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:11.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:11.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:11.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:11.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:11.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:11.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:12.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:11 smithi098 ceph-mon[30092]: Detected new or changed devices on smithi142 2024-05-09T08:46:12.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:11 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi142 to 5696M 2024-05-09T08:46:12.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:11 smithi098 ceph-mon[30092]: osdmap e61: 9 total, 9 up, 9 in 2024-05-09T08:46:12.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:11 smithi098 ceph-mon[30092]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:12.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:11 smithi142 ceph-mon[34625]: Detected new or changed devices on smithi142 2024-05-09T08:46:12.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:11 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi142 to 5696M 2024-05-09T08:46:12.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:11 smithi142 ceph-mon[34625]: osdmap e61: 9 total, 9 up, 9 in 2024-05-09T08:46:12.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:11 smithi142 ceph-mon[34625]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:12.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:11 smithi105 ceph-mon[34498]: Detected new or changed devices on smithi142 2024-05-09T08:46:12.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:11 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi142 to 5696M 2024-05-09T08:46:12.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:11 smithi105 ceph-mon[34498]: osdmap e61: 9 total, 9 up, 9 in 2024-05-09T08:46:12.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:11 smithi105 ceph-mon[34498]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/2240830032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6b0102e20 msgr2=0x7fd6b01096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 --2- 172.21.15.98:0/2240830032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6b0102e20 0x7fd6b01096d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800b0d0 tx=0x7fd6a800b3e0 comp rx=0 tx=0).stop 2024-05-09T08:46:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/2240830032 shutdown_connections 2024-05-09T08:46:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 --2- 172.21.15.98:0/2240830032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6b0102e20 0x7fd6b01096d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 --2- 172.21.15.98:0/2240830032 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6b0101280 0x7fd6b01016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 --2- 172.21.15.98:0/2240830032 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6b0102480 0x7fd6b01028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/2240830032 >> 172.21.15.98:0/2240830032 conn(0x7fd6b00fc840 msgr2=0x7fd6b00fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/2240830032 shutdown_connections 2024-05-09T08:46:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/2240830032 wait complete. 2024-05-09T08:46:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 Processor -- start 2024-05-09T08:46:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 -- start start 2024-05-09T08:46:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6b0101280 0x7fd6b0198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6b0102480 0x7fd6b0198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6b0102e20 0x7fd6b019fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b010ce90 con 0x7fd6b0101280 2024-05-09T08:46:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b010cd10 con 0x7fd6b0102480 2024-05-09T08:46:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.701+0000 7fd6b7d1e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b010d010 con 0x7fd6b0102e20 2024-05-09T08:46:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6b5aba700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6b0101280 0x7fd6b0198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6b5aba700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6b0101280 0x7fd6b0198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34546/0 (socket says 172.21.15.98:34546) 2024-05-09T08:46:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6b5aba700 1 -- 172.21.15.98:0/992694357 learned_addr learned my addr 172.21.15.98:0/992694357 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:46:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6b62bb700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6b0102e20 0x7fd6b019fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6b52b9700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6b0102480 0x7fd6b0198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6b5aba700 1 -- 172.21.15.98:0/992694357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6b0102e20 msgr2=0x7fd6b019fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6b5aba700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6b0102e20 0x7fd6b019fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6b5aba700 1 -- 172.21.15.98:0/992694357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6b0102480 msgr2=0x7fd6b0198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6b5aba700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6b0102480 0x7fd6b0198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6b5aba700 1 -- 172.21.15.98:0/992694357 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6b01a01c0 con 0x7fd6b0101280 2024-05-09T08:46:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6b52b9700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6b0102480 0x7fd6b0198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:46:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6b5aba700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6b0101280 0x7fd6b0198560 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00d9b0 tx=0x7fd6ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6a6ffd700 1 -- 172.21.15.98:0/992694357 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac017030 con 0x7fd6b0101280 2024-05-09T08:46:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/992694357 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6b01a04b0 con 0x7fd6b0101280 2024-05-09T08:46:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6a6ffd700 1 -- 172.21.15.98:0/992694357 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6ac004500 con 0x7fd6b0101280 2024-05-09T08:46:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6a6ffd700 1 -- 172.21.15.98:0/992694357 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac003aa0 con 0x7fd6b0101280 2024-05-09T08:46:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.702+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/992694357 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6b01a0a40 con 0x7fd6b0101280 2024-05-09T08:46:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.703+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/992694357 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6b0069010 con 0x7fd6b0101280 2024-05-09T08:46:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.704+0000 7fd6a6ffd700 1 -- 172.21.15.98:0/992694357 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd6ac01a070 con 0x7fd6b0101280 2024-05-09T08:46:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.705+0000 7fd6a6ffd700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd69c0a2c20 0x7fd69c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.705+0000 7fd6b52b9700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd69c0a2c20 0x7fd69c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.705+0000 7fd6a6ffd700 1 -- 172.21.15.98:0/992694357 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7fd6ac0ce380 con 0x7fd6b0101280 2024-05-09T08:46:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.705+0000 7fd6b52b9700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd69c0a2c20 0x7fd69c0a50e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000bc10 tx=0x7fd6a000b5f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:12.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.709+0000 7fd6a6ffd700 1 -- 172.21.15.98:0/992694357 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd6ac00e350 con 0x7fd6b0101280 2024-05-09T08:46:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:12.916+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/992694357 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd6b00fe930 con 0x7fd69c0a2c20 2024-05-09T08:46:13.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:12 smithi098 ceph-mon[30092]: osdmap e62: 9 total, 9 up, 9 in 2024-05-09T08:46:13.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:12 smithi098 ceph-mon[30092]: osdmap e63: 9 total, 9 up, 9 in 2024-05-09T08:46:13.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:12 smithi142 ceph-mon[34625]: osdmap e62: 9 total, 9 up, 9 in 2024-05-09T08:46:13.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:12 smithi142 ceph-mon[34625]: osdmap e63: 9 total, 9 up, 9 in 2024-05-09T08:46:13.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:12 smithi105 ceph-mon[34498]: osdmap e62: 9 total, 9 up, 9 in 2024-05-09T08:46:13.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:12 smithi105 ceph-mon[34498]: osdmap e63: 9 total, 9 up, 9 in 2024-05-09T08:46:13.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:13 smithi098 ceph-mon[30092]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:46:13.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:13 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:46:13.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:13 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:46:13.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:13 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:13.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:13 smithi098 ceph-mon[30092]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:13.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:13 smithi098 ceph-mon[30092]: osdmap e64: 9 total, 9 up, 9 in 2024-05-09T08:46:14.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:13 smithi142 ceph-mon[34625]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:46:14.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:13 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:46:14.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:13 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:46:14.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:13 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:14.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:13 smithi142 ceph-mon[34625]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:14.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:13 smithi142 ceph-mon[34625]: osdmap e64: 9 total, 9 up, 9 in 2024-05-09T08:46:14.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:13 smithi105 ceph-mon[34498]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:46:14.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:13 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:46:14.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:13 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:46:14.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:13 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:14.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:13 smithi105 ceph-mon[34498]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:14.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:13 smithi105 ceph-mon[34498]: osdmap e64: 9 total, 9 up, 9 in 2024-05-09T08:46:16.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:16 smithi105 ceph-mon[34498]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-05-09T08:46:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:16 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2887381662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae9746f9-c5b6-4574-8627-8cc50a41fa9e"}]: dispatch 2024-05-09T08:46:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:16 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2887381662' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae9746f9-c5b6-4574-8627-8cc50a41fa9e"}]': finished 2024-05-09T08:46:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:16 smithi105 ceph-mon[34498]: osdmap e65: 10 total, 9 up, 10 in 2024-05-09T08:46:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:16 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:16.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:16 smithi098 ceph-mon[30092]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-05-09T08:46:16.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:16 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2887381662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae9746f9-c5b6-4574-8627-8cc50a41fa9e"}]: dispatch 2024-05-09T08:46:16.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:16 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2887381662' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae9746f9-c5b6-4574-8627-8cc50a41fa9e"}]': finished 2024-05-09T08:46:16.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:16 smithi098 ceph-mon[30092]: osdmap e65: 10 total, 9 up, 10 in 2024-05-09T08:46:16.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:16 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:16 smithi142 ceph-mon[34625]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-05-09T08:46:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:16 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2887381662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae9746f9-c5b6-4574-8627-8cc50a41fa9e"}]: dispatch 2024-05-09T08:46:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:16 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2887381662' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae9746f9-c5b6-4574-8627-8cc50a41fa9e"}]': finished 2024-05-09T08:46:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:16 smithi142 ceph-mon[34625]: osdmap e65: 10 total, 9 up, 10 in 2024-05-09T08:46:16.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:17.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:17 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2560181574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:46:17.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:17 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2560181574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:46:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:17 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2560181574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:46:18.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:18 smithi105 ceph-mon[34498]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-05-09T08:46:18.489 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:18 smithi098 ceph-mon[30092]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-05-09T08:46:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:18 smithi142 ceph-mon[34625]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-05-09T08:46:20.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:20 smithi105 ceph-mon[34498]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-09T08:46:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:20 smithi098 ceph-mon[30092]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-09T08:46:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:20 smithi142 ceph-mon[34625]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-09T08:46:22.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:22 smithi098 ceph-mon[30092]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:46:22.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:22 smithi105 ceph-mon[34498]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:46:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:22 smithi142 ceph-mon[34625]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:46:24.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:24 smithi105 ceph-mon[34498]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-09T08:46:24.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-09T08:46:24.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:24 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:24 smithi098 ceph-mon[30092]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-09T08:46:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-09T08:46:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:24 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:24 smithi142 ceph-mon[34625]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-09T08:46:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-09T08:46:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:24 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:25.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:25 smithi105 ceph-mon[34498]: Deploying daemon osd.9 on smithi098 2024-05-09T08:46:25.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:25 smithi098 ceph-mon[30092]: Deploying daemon osd.9 on smithi098 2024-05-09T08:46:25.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:25 smithi142 ceph-mon[34625]: Deploying daemon osd.9 on smithi098 2024-05-09T08:46:26.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:26 smithi105 ceph-mon[34498]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:26.530 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:26 smithi098 ceph-mon[30092]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:26.559 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:26 smithi142 ceph-mon[34625]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:28.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:28 smithi105 ceph-mon[34498]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:28.528 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:28 smithi098 ceph-mon[30092]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:28.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:28 smithi142 ceph-mon[34625]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:30.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:30 smithi105 ceph-mon[34498]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:30.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:30 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:30.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:30 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:30.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:30 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:30.545 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:30 smithi098 ceph-mon[30092]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:30.545 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:30.546 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:30.546 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:30 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:30 smithi142 ceph-mon[34625]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:30 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:30 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:30 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:32.462 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:32 smithi098 ceph-mon[30092]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:32.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:32 smithi105 ceph-mon[34498]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:32.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:32 smithi142 ceph-mon[34625]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:33.941 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 9 on host 'smithi098' 2024-05-09T08:46:33.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:33.941+0000 7fd6a6ffd700 1 -- 172.21.15.98:0/992694357 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd6b00fe930 con 0x7fd69c0a2c20 2024-05-09T08:46:33.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:33.944+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/992694357 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd69c0a2c20 msgr2=0x7fd69c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:33.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:33.944+0000 7fd6b7d1e700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd69c0a2c20 0x7fd69c0a50e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000bc10 tx=0x7fd6a000b5f0 comp rx=0 tx=0).stop 2024-05-09T08:46:33.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:33.944+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/992694357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6b0101280 msgr2=0x7fd6b0198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:33.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:33.944+0000 7fd6b7d1e700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6b0101280 0x7fd6b0198560 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00d9b0 tx=0x7fd6ac00dcc0 comp rx=0 tx=0).stop 2024-05-09T08:46:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:33.944+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/992694357 shutdown_connections 2024-05-09T08:46:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:33.944+0000 7fd6b7d1e700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd69c0a2c20 0x7fd69c0a50e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:33.944+0000 7fd6b7d1e700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6b0102e20 0x7fd6b019fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:33.944+0000 7fd6b7d1e700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6b0101280 0x7fd6b0198560 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:33.944+0000 7fd6b7d1e700 1 --2- 172.21.15.98:0/992694357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6b0102480 0x7fd6b0198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:33.944+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/992694357 >> 172.21.15.98:0/992694357 conn(0x7fd6b00fc840 msgr2=0x7fd6b00fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:33.944+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/992694357 shutdown_connections 2024-05-09T08:46:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:46:33.944+0000 7fd6b7d1e700 1 -- 172.21.15.98:0/992694357 wait complete. 2024-05-09T08:46:34.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:34 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:34 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:34 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:34.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:34 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:34.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:34 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:34 smithi098 ceph-mon[30092]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:34.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:34 smithi098 ceph-mon[30092]: from='osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-09T08:46:34.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:34 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:34 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:34.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:34 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:34 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:34 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:34 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:34.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:34 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:34.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:34 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:34 smithi142 ceph-mon[34625]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:34.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:34 smithi142 ceph-mon[34625]: from='osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-09T08:46:34.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:34 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:34 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:34.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:34 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:34 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:34 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:34 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:34 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:34 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:34 smithi105 ceph-mon[34498]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:34 smithi105 ceph-mon[34498]: from='osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-09T08:46:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:34 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:34 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:34.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:34 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:34.593 DEBUG:teuthology.orchestra.run.smithi098:osd.9> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.9.service 2024-05-09T08:46:34.596 INFO:tasks.cephadm:Deploying osd.10 on smithi105 with /dev/vg_nvme/lv_1... 2024-05-09T08:46:34.596 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-09T08:46:34.840 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:46:35.122 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:35 smithi098 ceph-mon[30092]: from='osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-09T08:46:35.122 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:35 smithi098 ceph-mon[30092]: osdmap e66: 10 total, 9 up, 10 in 2024-05-09T08:46:35.122 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:35 smithi098 ceph-mon[30092]: from='osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-05-09T08:46:35.122 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:35 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:35.122 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 08:46:35 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T08:46:35.048+0000 7f346e8aa700 -1 osd.9 0 waiting for initial osdmap 2024-05-09T08:46:35.122 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 08:46:35 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T08:46:35.061+0000 7f346a69d700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:46:35.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:35 smithi105 ceph-mon[34498]: from='osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-09T08:46:35.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:35 smithi105 ceph-mon[34498]: osdmap e66: 10 total, 9 up, 10 in 2024-05-09T08:46:35.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:35 smithi105 ceph-mon[34498]: from='osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-05-09T08:46:35.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:35 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:35.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:35 smithi142 ceph-mon[34625]: from='osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-09T08:46:35.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:35 smithi142 ceph-mon[34625]: osdmap e66: 10 total, 9 up, 10 in 2024-05-09T08:46:35.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:35 smithi142 ceph-mon[34625]: from='osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-05-09T08:46:35.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:35 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:36.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:36 smithi098 ceph-mon[30092]: from='osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-05-09T08:46:36.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:36 smithi098 ceph-mon[30092]: osdmap e67: 10 total, 9 up, 10 in 2024-05-09T08:46:36.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:36 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:36.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:36 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:36.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:36 smithi098 ceph-mon[30092]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:36.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:36 smithi105 ceph-mon[34498]: from='osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-05-09T08:46:36.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:36 smithi105 ceph-mon[34498]: osdmap e67: 10 total, 9 up, 10 in 2024-05-09T08:46:36.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:36 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:36.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:36 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:36.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:36 smithi105 ceph-mon[34498]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:36.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:36 smithi142 ceph-mon[34625]: from='osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-05-09T08:46:36.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:36 smithi142 ceph-mon[34625]: osdmap e67: 10 total, 9 up, 10 in 2024-05-09T08:46:36.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:36 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:36.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:36 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:36.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:36 smithi142 ceph-mon[34625]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:46:36.867 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-09T08:46:36.867 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-09T08:46:36.867 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-05-09T08:46:36.867 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-05-09T08:46:36.867 INFO:teuthology.orchestra.run.smithi105.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115813 s, 905 MB/s 2024-05-09T08:46:36.867 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-05-09T08:46:37.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:37 smithi098 ceph-mon[30092]: purged_snaps scrub starts 2024-05-09T08:46:37.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:37 smithi098 ceph-mon[30092]: purged_snaps scrub ok 2024-05-09T08:46:37.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:37 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:37.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:37 smithi098 ceph-mon[30092]: osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414] boot 2024-05-09T08:46:37.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:37 smithi098 ceph-mon[30092]: osdmap e68: 10 total, 10 up, 10 in 2024-05-09T08:46:37.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:37 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:37.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:37 smithi142 ceph-mon[34625]: purged_snaps scrub starts 2024-05-09T08:46:37.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:37 smithi142 ceph-mon[34625]: purged_snaps scrub ok 2024-05-09T08:46:37.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:37 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:37.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:37 smithi142 ceph-mon[34625]: osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414] boot 2024-05-09T08:46:37.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:37 smithi142 ceph-mon[34625]: osdmap e68: 10 total, 10 up, 10 in 2024-05-09T08:46:37.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:37 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:37.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:37 smithi105 ceph-mon[34498]: purged_snaps scrub starts 2024-05-09T08:46:37.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:37 smithi105 ceph-mon[34498]: purged_snaps scrub ok 2024-05-09T08:46:37.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:37 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:37.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:37 smithi105 ceph-mon[34498]: osd.9 [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414] boot 2024-05-09T08:46:37.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:37 smithi105 ceph-mon[34498]: osdmap e68: 10 total, 10 up, 10 in 2024-05-09T08:46:37.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:37 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:46:37.542 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_1 2024-05-09T08:46:37.787 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:46:38.143 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:38 smithi105 ceph-mon[34498]: osdmap e69: 10 total, 10 up, 10 in 2024-05-09T08:46:38.143 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:38 smithi105 ceph-mon[34498]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:38.143 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:38.143 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:38.143 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:38.143 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:38.143 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:38.143 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:38.144 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:38.144 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:38.144 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:38.144 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:38.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:38 smithi142 ceph-mon[34625]: osdmap e69: 10 total, 10 up, 10 in 2024-05-09T08:46:38.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:38 smithi142 ceph-mon[34625]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:38.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:38.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:38.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:38.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:38.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:38.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:38.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:38.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:38.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:38.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:38 smithi098 ceph-mon[30092]: osdmap e69: 10 total, 10 up, 10 in 2024-05-09T08:46:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:38 smithi098 ceph-mon[30092]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:46:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:38.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:39.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:39 smithi105 ceph-mon[34498]: Detected new or changed devices on smithi098 2024-05-09T08:46:39.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:39 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi098 to 4272M 2024-05-09T08:46:39.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:39 smithi098 ceph-mon[30092]: Detected new or changed devices on smithi098 2024-05-09T08:46:39.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:39 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi098 to 4272M 2024-05-09T08:46:39.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:39 smithi142 ceph-mon[34625]: Detected new or changed devices on smithi098 2024-05-09T08:46:39.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:39 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi098 to 4272M 2024-05-09T08:46:39.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.430+0000 7f9104d5c700 1 -- 172.21.15.105:0/2814892006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91001080d0 msgr2=0x7f910010a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:39.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.430+0000 7f9104d5c700 1 --2- 172.21.15.105:0/2814892006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91001080d0 0x7f910010a4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f90f800b0a0 tx=0x7f90f800b3b0 comp rx=0 tx=0).stop 2024-05-09T08:46:39.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.431+0000 7f9104d5c700 1 -- 172.21.15.105:0/2814892006 shutdown_connections 2024-05-09T08:46:39.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.431+0000 7f9104d5c700 1 --2- 172.21.15.105:0/2814892006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91001080d0 0x7f910010a4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:39.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.431+0000 7f9104d5c700 1 --2- 172.21.15.105:0/2814892006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9100102e70 0x7f9100105260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:39.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.431+0000 7f9104d5c700 1 --2- 172.21.15.105:0/2814892006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91001057a0 0x7f9100107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:39.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.431+0000 7f9104d5c700 1 -- 172.21.15.105:0/2814892006 >> 172.21.15.105:0/2814892006 conn(0x7f91000fc840 msgr2=0x7f91000fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:39.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.431+0000 7f9104d5c700 1 -- 172.21.15.105:0/2814892006 shutdown_connections 2024-05-09T08:46:39.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.431+0000 7f9104d5c700 1 -- 172.21.15.105:0/2814892006 wait complete. 2024-05-09T08:46:39.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.431+0000 7f9104d5c700 1 Processor -- start 2024-05-09T08:46:39.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.431+0000 7f9104d5c700 1 -- start start 2024-05-09T08:46:39.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f9104d5c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9100102e70 0x7f91001027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:39.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f9104d5c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91001057a0 0x7f9100100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:39.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f9104d5c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91001080d0 0x7f9100101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:39.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f9104d5c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f910010df50 con 0x7f9100102e70 2024-05-09T08:46:39.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f9104d5c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f910010ddd0 con 0x7f91001057a0 2024-05-09T08:46:39.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f9104d5c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f910010e0d0 con 0x7f91001080d0 2024-05-09T08:46:39.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f90fdd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91001057a0 0x7f9100100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:39.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f90fdd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91001057a0 0x7f9100100e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50758/0 (socket says 172.21.15.105:50758) 2024-05-09T08:46:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f90fdd9b700 1 -- 172.21.15.105:0/2250924189 learned_addr learned my addr 172.21.15.105:0/2250924189 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-05-09T08:46:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f90fe59c700 1 --2- 172.21.15.105:0/2250924189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9100102e70 0x7f91001027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f90fed9d700 1 --2- 172.21.15.105:0/2250924189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91001080d0 0x7f9100101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f90fdd9b700 1 -- 172.21.15.105:0/2250924189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91001080d0 msgr2=0x7f9100101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f90fdd9b700 1 --2- 172.21.15.105:0/2250924189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91001080d0 0x7f9100101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f90fdd9b700 1 -- 172.21.15.105:0/2250924189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9100102e70 msgr2=0x7f91001027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f90fdd9b700 1 --2- 172.21.15.105:0/2250924189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9100102e70 0x7f91001027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f90fdd9b700 1 -- 172.21.15.105:0/2250924189 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9100101c10 con 0x7f91001057a0 2024-05-09T08:46:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f90fdd9b700 1 --2- 172.21.15.105:0/2250924189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91001057a0 0x7f9100100e20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f90f400d980 tx=0x7f90f400dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f90ef7fe700 1 -- 172.21.15.105:0/2250924189 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f4017070 con 0x7f91001057a0 2024-05-09T08:46:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f90ef7fe700 1 -- 172.21.15.105:0/2250924189 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90f4004d10 con 0x7f91001057a0 2024-05-09T08:46:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f90ef7fe700 1 -- 172.21.15.105:0/2250924189 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f40055c0 con 0x7f91001057a0 2024-05-09T08:46:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f9104d5c700 1 -- 172.21.15.105:0/2250924189 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91001a26c0 con 0x7f91001057a0 2024-05-09T08:46:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.432+0000 7f9104d5c700 1 -- 172.21.15.105:0/2250924189 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91001a2b80 con 0x7f91001057a0 2024-05-09T08:46:39.435 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.433+0000 7f9104d5c700 1 -- 172.21.15.105:0/2250924189 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9100069010 con 0x7f91001057a0 2024-05-09T08:46:39.435 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.434+0000 7f90ef7fe700 1 -- 172.21.15.105:0/2250924189 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f90f4004750 con 0x7f91001057a0 2024-05-09T08:46:39.435 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.435+0000 7f90ef7fe700 1 --2- 172.21.15.105:0/2250924189 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f90e80a2c20 0x7f90e80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:39.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.435+0000 7f90ef7fe700 1 -- 172.21.15.105:0/2250924189 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7f90f40ceb30 con 0x7f91001057a0 2024-05-09T08:46:39.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.435+0000 7f90fe59c700 1 --2- 172.21.15.105:0/2250924189 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f90e80a2c20 0x7f90e80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:39.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.436+0000 7f90fe59c700 1 --2- 172.21.15.105:0/2250924189 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f90e80a2c20 0x7f90e80a50e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f90f000bbd0 tx=0x7f90f000b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:39.437 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.439+0000 7f90ef7fe700 1 -- 172.21.15.105:0/2250924189 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f90f4020020 con 0x7f91001057a0 2024-05-09T08:46:39.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:39.643+0000 7f9104d5c700 1 -- 172.21.15.105:0/2250924189 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9100062760 con 0x7f90e80a2c20 2024-05-09T08:46:40.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:40 smithi098 ceph-mon[30092]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:40.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:40 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:46:40.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:40 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:46:40.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:40 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:40.361 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:40 smithi105 ceph-mon[34498]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:40.362 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:40 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:46:40.362 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:40 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:46:40.362 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:40 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:40.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:40 smithi142 ceph-mon[34625]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:40.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:40 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:46:40.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:40 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:46:40.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:40 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:41.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:41 smithi098 ceph-mon[30092]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:46:41.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:41 smithi142 ceph-mon[34625]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:46:41.436 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:41 smithi105 ceph-mon[34498]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:46:42.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:42 smithi098 ceph-mon[30092]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:42.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:42 smithi142 ceph-mon[34625]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:42.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:42 smithi105 ceph-mon[34498]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:43.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:43 smithi098 ceph-mon[30092]: from='client.? 172.21.15.105:0/2068313935' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1963b555-2424-4d7e-b769-95433251224f"}]: dispatch 2024-05-09T08:46:43.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:43 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1963b555-2424-4d7e-b769-95433251224f"}]: dispatch 2024-05-09T08:46:43.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:43 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1963b555-2424-4d7e-b769-95433251224f"}]': finished 2024-05-09T08:46:43.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:43 smithi098 ceph-mon[30092]: osdmap e70: 11 total, 10 up, 11 in 2024-05-09T08:46:43.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:43 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:46:43.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:43 smithi098 ceph-mon[30092]: from='client.? 172.21.15.105:0/157240144' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:46:43.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:43 smithi142 ceph-mon[34625]: from='client.? 172.21.15.105:0/2068313935' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1963b555-2424-4d7e-b769-95433251224f"}]: dispatch 2024-05-09T08:46:43.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:43 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1963b555-2424-4d7e-b769-95433251224f"}]: dispatch 2024-05-09T08:46:43.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:43 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1963b555-2424-4d7e-b769-95433251224f"}]': finished 2024-05-09T08:46:43.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:43 smithi142 ceph-mon[34625]: osdmap e70: 11 total, 10 up, 11 in 2024-05-09T08:46:43.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:43 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:46:43.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:43 smithi142 ceph-mon[34625]: from='client.? 172.21.15.105:0/157240144' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:46:43.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:43 smithi105 ceph-mon[34498]: from='client.? 172.21.15.105:0/2068313935' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1963b555-2424-4d7e-b769-95433251224f"}]: dispatch 2024-05-09T08:46:43.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:43 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1963b555-2424-4d7e-b769-95433251224f"}]: dispatch 2024-05-09T08:46:43.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:43 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1963b555-2424-4d7e-b769-95433251224f"}]': finished 2024-05-09T08:46:43.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:43 smithi105 ceph-mon[34498]: osdmap e70: 11 total, 10 up, 11 in 2024-05-09T08:46:43.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:43 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:46:43.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:43 smithi105 ceph-mon[34498]: from='client.? 172.21.15.105:0/157240144' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:46:44.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:44 smithi098 ceph-mon[30092]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:44.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:44 smithi142 ceph-mon[34625]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:44.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:44 smithi105 ceph-mon[34498]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:46.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:46 smithi105 ceph-mon[34498]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:46.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:46 smithi098 ceph-mon[30092]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:46 smithi142 ceph-mon[34625]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:48.465 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:48 smithi105 ceph-mon[34498]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:48.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:48 smithi098 ceph-mon[30092]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:48.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:48 smithi142 ceph-mon[34625]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:50.469 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:50 smithi105 ceph-mon[34498]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:50.470 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-09T08:46:50.470 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:50.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:50 smithi098 ceph-mon[30092]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:50.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-09T08:46:50.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:50 smithi142 ceph-mon[34625]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-09T08:46:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:51.463 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:51 smithi105 ceph-mon[34498]: Deploying daemon osd.10 on smithi105 2024-05-09T08:46:51.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:51 smithi098 ceph-mon[30092]: Deploying daemon osd.10 on smithi105 2024-05-09T08:46:51.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:51 smithi142 ceph-mon[34625]: Deploying daemon osd.10 on smithi105 2024-05-09T08:46:52.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:52 smithi105 ceph-mon[34498]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:52.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:52 smithi098 ceph-mon[30092]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:52 smithi142 ceph-mon[34625]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:54.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:54 smithi105 ceph-mon[34498]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:54.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:54 smithi098 ceph-mon[30092]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:54.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:54 smithi142 ceph-mon[34625]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:56.301 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:56 smithi105 ceph-mon[34498]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:56.301 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:56 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:56.301 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:56 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:56.301 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:56 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:56 smithi098 ceph-mon[30092]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:56 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:56 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:56 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:56.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:56 smithi142 ceph-mon[34625]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:56.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:56 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:56.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:56 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:46:56.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:56 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:58.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:46:58 smithi105 ceph-mon[34498]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:46:58 smithi098 ceph-mon[30092]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:58.653 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:46:58 smithi142 ceph-mon[34625]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:46:59.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:59.309+0000 7f90ef7fe700 1 -- 172.21.15.105:0/2250924189 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f9100062760 con 0x7f90e80a2c20 2024-05-09T08:46:59.310 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 10 on host 'smithi105' 2024-05-09T08:46:59.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:59.311+0000 7f9104d5c700 1 -- 172.21.15.105:0/2250924189 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f90e80a2c20 msgr2=0x7f90e80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:59.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:59.311+0000 7f9104d5c700 1 --2- 172.21.15.105:0/2250924189 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f90e80a2c20 0x7f90e80a50e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f90f000bbd0 tx=0x7f90f000b5b0 comp rx=0 tx=0).stop 2024-05-09T08:46:59.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:59.311+0000 7f9104d5c700 1 -- 172.21.15.105:0/2250924189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91001057a0 msgr2=0x7f9100100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:59.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:59.311+0000 7f9104d5c700 1 --2- 172.21.15.105:0/2250924189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91001057a0 0x7f9100100e20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f90f400d980 tx=0x7f90f400dd40 comp rx=0 tx=0).stop 2024-05-09T08:46:59.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:59.311+0000 7f9104d5c700 1 -- 172.21.15.105:0/2250924189 shutdown_connections 2024-05-09T08:46:59.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:59.311+0000 7f9104d5c700 1 --2- 172.21.15.105:0/2250924189 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f90e80a2c20 0x7f90e80a50e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:59.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:59.311+0000 7f9104d5c700 1 --2- 172.21.15.105:0/2250924189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91001080d0 0x7f9100101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:59.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:59.311+0000 7f9104d5c700 1 --2- 172.21.15.105:0/2250924189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9100102e70 0x7f91001027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:59.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:59.311+0000 7f9104d5c700 1 --2- 172.21.15.105:0/2250924189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91001057a0 0x7f9100100e20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:59.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:59.311+0000 7f9104d5c700 1 -- 172.21.15.105:0/2250924189 >> 172.21.15.105:0/2250924189 conn(0x7f91000fc840 msgr2=0x7f91001037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:59.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:59.311+0000 7f9104d5c700 1 -- 172.21.15.105:0/2250924189 shutdown_connections 2024-05-09T08:46:59.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-09T08:46:59.311+0000 7f9104d5c700 1 -- 172.21.15.105:0/2250924189 wait complete. 2024-05-09T08:46:59.944 DEBUG:teuthology.orchestra.run.smithi105:osd.10> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.10.service 2024-05-09T08:46:59.946 INFO:tasks.cephadm:Deploying osd.11 on smithi142 with /dev/vg_nvme/lv_1... 2024-05-09T08:46:59.947 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-09T08:47:00.192 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:47:00.549 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:00 smithi142 ceph-mon[34625]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:47:00.550 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:00 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:00.550 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:00 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:00.550 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:00 smithi142 ceph-mon[34625]: from='osd.10 [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:47:00.550 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:00 smithi142 ceph-mon[34625]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:47:00.550 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:00 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:00.550 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:00 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:00.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:00 smithi098 ceph-mon[30092]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:47:00.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:00 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:00.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:00 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:00.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:00 smithi098 ceph-mon[30092]: from='osd.10 [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:47:00.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:00 smithi098 ceph-mon[30092]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:47:00.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:00 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:00.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:00 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:00.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:00 smithi105 ceph-mon[34498]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:47:00.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:00 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:00.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:00 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:00.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:00 smithi105 ceph-mon[34498]: from='osd.10 [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:47:00.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:00 smithi105 ceph-mon[34498]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:47:00.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:00 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:00.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:00 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:01.579 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:01 smithi142 ceph-mon[34625]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-09T08:47:01.580 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:01 smithi142 ceph-mon[34625]: osdmap e71: 11 total, 10 up, 11 in 2024-05-09T08:47:01.580 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:01 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:01.580 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:01 smithi142 ceph-mon[34625]: from='osd.10 [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:47:01.580 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:01 smithi142 ceph-mon[34625]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:47:01.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:01 smithi098 ceph-mon[30092]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-09T08:47:01.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:01 smithi098 ceph-mon[30092]: osdmap e71: 11 total, 10 up, 11 in 2024-05-09T08:47:01.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:01 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:01.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:01 smithi098 ceph-mon[30092]: from='osd.10 [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:47:01.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:01 smithi098 ceph-mon[30092]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:47:01.642 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:01 smithi105 ceph-mon[34498]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-09T08:47:01.643 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:01 smithi105 ceph-mon[34498]: osdmap e71: 11 total, 10 up, 11 in 2024-05-09T08:47:01.643 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:01 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:01.643 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:01 smithi105 ceph-mon[34498]: from='osd.10 [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:47:01.643 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:01 smithi105 ceph-mon[34498]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-09T08:47:01.643 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 08:47:01 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T08:47:01.307+0000 7efcc3373700 -1 osd.10 0 waiting for initial osdmap 2024-05-09T08:47:01.643 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 08:47:01 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T08:47:01.317+0000 7efcbf166700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:47:02.302 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-09T08:47:02.302 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-09T08:47:02.302 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2024-05-09T08:47:02.302 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2024-05-09T08:47:02.302 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113808 s, 921 MB/s 2024-05-09T08:47:02.303 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2024-05-09T08:47:02.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:02 smithi098 ceph-mon[30092]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:47:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:02 smithi098 ceph-mon[30092]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-09T08:47:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:02 smithi098 ceph-mon[30092]: osdmap e72: 11 total, 10 up, 11 in 2024-05-09T08:47:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:02 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:02 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:02 smithi142 ceph-mon[34625]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:47:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:02 smithi142 ceph-mon[34625]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-09T08:47:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:02 smithi142 ceph-mon[34625]: osdmap e72: 11 total, 10 up, 11 in 2024-05-09T08:47:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:02 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:02 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:02.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:02 smithi105 ceph-mon[34498]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:47:02.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:02 smithi105 ceph-mon[34498]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-09T08:47:02.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:02 smithi105 ceph-mon[34498]: osdmap e72: 11 total, 10 up, 11 in 2024-05-09T08:47:02.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:02 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:02.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:02 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:03.031 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi142:vg_nvme/lv_1 2024-05-09T08:47:03.270 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:47:03.451 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:03 smithi105 ceph-mon[34498]: purged_snaps scrub starts 2024-05-09T08:47:03.451 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:03 smithi105 ceph-mon[34498]: purged_snaps scrub ok 2024-05-09T08:47:03.451 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:03 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:03.451 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:03 smithi105 ceph-mon[34498]: osd.10 [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218] boot 2024-05-09T08:47:03.451 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:03 smithi105 ceph-mon[34498]: osdmap e73: 11 total, 11 up, 11 in 2024-05-09T08:47:03.451 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:03 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:03.452 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:03 smithi105 ceph-mon[34498]: osdmap e74: 11 total, 11 up, 11 in 2024-05-09T08:47:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:03 smithi098 ceph-mon[30092]: purged_snaps scrub starts 2024-05-09T08:47:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:03 smithi098 ceph-mon[30092]: purged_snaps scrub ok 2024-05-09T08:47:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:03 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:03 smithi098 ceph-mon[30092]: osd.10 [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218] boot 2024-05-09T08:47:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:03 smithi098 ceph-mon[30092]: osdmap e73: 11 total, 11 up, 11 in 2024-05-09T08:47:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:03 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:03 smithi098 ceph-mon[30092]: osdmap e74: 11 total, 11 up, 11 in 2024-05-09T08:47:03.623 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:03 smithi142 ceph-mon[34625]: purged_snaps scrub starts 2024-05-09T08:47:03.623 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:03 smithi142 ceph-mon[34625]: purged_snaps scrub ok 2024-05-09T08:47:03.623 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:03 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:03.623 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:03 smithi142 ceph-mon[34625]: osd.10 [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218] boot 2024-05-09T08:47:03.623 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:03 smithi142 ceph-mon[34625]: osdmap e73: 11 total, 11 up, 11 in 2024-05-09T08:47:03.623 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:03 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:47:03.623 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:03 smithi142 ceph-mon[34625]: osdmap e74: 11 total, 11 up, 11 in 2024-05-09T08:47:04.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:47:04.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:04.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:04.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:04.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:04.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:04.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:04.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:04.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: osdmap e75: 11 total, 11 up, 11 in 2024-05-09T08:47:04.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:04.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:04.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:47:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:04.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:04.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:04.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: osdmap e75: 11 total, 11 up, 11 in 2024-05-09T08:47:04.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:04.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:04.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:47:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: osdmap e75: 11 total, 11 up, 11 in 2024-05-09T08:47:04.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:04.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:04.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:04.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.975+0000 7f174defb700 1 -- 172.21.15.142:0/1721552278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1748102e10 msgr2=0x7f17481096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:04.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.975+0000 7f174defb700 1 --2- 172.21.15.142:0/1721552278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1748102e10 0x7f17481096c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f174000b0a0 tx=0x7f174000b3b0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.975+0000 7f174defb700 1 -- 172.21.15.142:0/1721552278 shutdown_connections 2024-05-09T08:47:04.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.975+0000 7f174defb700 1 --2- 172.21.15.142:0/1721552278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1748102e10 0x7f17481096c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.975+0000 7f174defb700 1 --2- 172.21.15.142:0/1721552278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1748101270 0x7f1748101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.975+0000 7f174defb700 1 --2- 172.21.15.142:0/1721552278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1748102470 0x7f17481028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.975+0000 7f174defb700 1 -- 172.21.15.142:0/1721552278 >> 172.21.15.142:0/1721552278 conn(0x7f17480fc810 msgr2=0x7f17480fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:04.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.975+0000 7f174defb700 1 -- 172.21.15.142:0/1721552278 shutdown_connections 2024-05-09T08:47:04.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.975+0000 7f174defb700 1 -- 172.21.15.142:0/1721552278 wait complete. 2024-05-09T08:47:04.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.976+0000 7f174defb700 1 Processor -- start 2024-05-09T08:47:04.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.976+0000 7f174defb700 1 -- start start 2024-05-09T08:47:04.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.976+0000 7f174defb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1748101270 0x7f1748105270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:04.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.976+0000 7f174defb700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1748102470 0x7f17481038c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:04.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.976+0000 7f174defb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1748102e10 0x7f1748103e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:04.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.976+0000 7f174defb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f174810ce50 con 0x7f1748102e10 2024-05-09T08:47:04.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.976+0000 7f174defb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f174810ccd0 con 0x7f1748101270 2024-05-09T08:47:04.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.976+0000 7f174defb700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f174810cfd0 con 0x7f1748102470 2024-05-09T08:47:04.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.976+0000 7f1746ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1748102470 0x7f17481038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:04.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.976+0000 7f1746ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1748102470 0x7f17481038c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53726/0 (socket says 172.21.15.142:53726) 2024-05-09T08:47:04.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.976+0000 7f1746ffd700 1 -- 172.21.15.142:0/2968090564 learned_addr learned my addr 172.21.15.142:0/2968090564 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-05-09T08:47:04.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.977+0000 7f1747fff700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1748102e10 0x7f1748103e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:04.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.977+0000 7f17477fe700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1748101270 0x7f1748105270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:04.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.977+0000 7f1746ffd700 1 -- 172.21.15.142:0/2968090564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1748101270 msgr2=0x7f1748105270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:04.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.977+0000 7f1746ffd700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1748101270 0x7f1748105270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.977+0000 7f1746ffd700 1 -- 172.21.15.142:0/2968090564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1748102e10 msgr2=0x7f1748103e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:04.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.977+0000 7f1746ffd700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1748102e10 0x7f1748103e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.977+0000 7f1746ffd700 1 -- 172.21.15.142:0/2968090564 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17481046b0 con 0x7f1748102470 2024-05-09T08:47:04.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.977+0000 7f1747fff700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1748102e10 0x7f1748103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:47:04.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.977+0000 7f1746ffd700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1748102470 0x7f17481038c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f173c00c970 tx=0x7f173c00cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:04.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.977+0000 7f1744ff9700 1 -- 172.21.15.142:0/2968090564 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c007aa0 con 0x7f1748102470 2024-05-09T08:47:04.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.977+0000 7f1744ff9700 1 -- 172.21.15.142:0/2968090564 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f173c004d10 con 0x7f1748102470 2024-05-09T08:47:04.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.977+0000 7f174defb700 1 -- 172.21.15.142:0/2968090564 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17481a26b0 con 0x7f1748102470 2024-05-09T08:47:04.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.977+0000 7f1744ff9700 1 -- 172.21.15.142:0/2968090564 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c005690 con 0x7f1748102470 2024-05-09T08:47:04.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.977+0000 7f174defb700 1 -- 172.21.15.142:0/2968090564 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17481a2b70 con 0x7f1748102470 2024-05-09T08:47:04.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.978+0000 7f174defb700 1 -- 172.21.15.142:0/2968090564 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1748069010 con 0x7f1748102470 2024-05-09T08:47:04.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.979+0000 7f1744ff9700 1 -- 172.21.15.142:0/2968090564 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f173c004750 con 0x7f1748102470 2024-05-09T08:47:04.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.980+0000 7f1744ff9700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f17300a2c20 0x7f17300a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:04.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.980+0000 7f1744ff9700 1 -- 172.21.15.142:0/2968090564 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7f173c0ceb30 con 0x7f1748102470 2024-05-09T08:47:04.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.980+0000 7f17477fe700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f17300a2c20 0x7f17300a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:04.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.981+0000 7f17477fe700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f17300a2c20 0x7f17300a50e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1738005fd0 tx=0x7f1738005dd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:04.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:04.983+0000 7f1744ff9700 1 -- 172.21.15.142:0/2968090564 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f173c00de20 con 0x7f1748102470 2024-05-09T08:47:05.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:05.186+0000 7f174defb700 1 -- 172.21.15.142:0/2968090564 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f17481a2e30 con 0x7f17300a2c20 2024-05-09T08:47:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:05 smithi098 ceph-mon[30092]: Detected new or changed devices on smithi105 2024-05-09T08:47:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:05 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi105 to 4272M 2024-05-09T08:47:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:05 smithi098 ceph-mon[30092]: osdmap e76: 11 total, 11 up, 11 in 2024-05-09T08:47:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:05 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:47:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:05 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:47:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:05 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:05 smithi142 ceph-mon[34625]: Detected new or changed devices on smithi105 2024-05-09T08:47:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:05 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi105 to 4272M 2024-05-09T08:47:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:05 smithi142 ceph-mon[34625]: osdmap e76: 11 total, 11 up, 11 in 2024-05-09T08:47:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:05 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:47:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:05 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:47:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:05 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:05 smithi105 ceph-mon[34498]: Detected new or changed devices on smithi105 2024-05-09T08:47:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:05 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi105 to 4272M 2024-05-09T08:47:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:05 smithi105 ceph-mon[34498]: osdmap e76: 11 total, 11 up, 11 in 2024-05-09T08:47:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:05 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:47:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:05 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:47:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:05 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:06.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:06 smithi142 ceph-mon[34625]: pgmap v240: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail; 149 KiB/s, 0 objects/s recovering 2024-05-09T08:47:06.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:06 smithi142 ceph-mon[34625]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:47:06.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:06 smithi142 ceph-mon[34625]: osdmap e77: 11 total, 11 up, 11 in 2024-05-09T08:47:06.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:06 smithi098 ceph-mon[30092]: pgmap v240: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail; 149 KiB/s, 0 objects/s recovering 2024-05-09T08:47:06.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:06 smithi098 ceph-mon[30092]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:47:06.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:06 smithi098 ceph-mon[30092]: osdmap e77: 11 total, 11 up, 11 in 2024-05-09T08:47:06.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:06 smithi105 ceph-mon[34498]: pgmap v240: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail; 149 KiB/s, 0 objects/s recovering 2024-05-09T08:47:06.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:06 smithi105 ceph-mon[34498]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:47:06.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:06 smithi105 ceph-mon[34498]: osdmap e77: 11 total, 11 up, 11 in 2024-05-09T08:47:08.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:08 smithi098 ceph-mon[30092]: pgmap v242: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-09T08:47:08.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:08 smithi098 ceph-mon[30092]: from='client.? 172.21.15.142:0/3284133483' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d685640-23a3-4c93-8a66-1db5b01c3252"}]: dispatch 2024-05-09T08:47:08.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:08 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d685640-23a3-4c93-8a66-1db5b01c3252"}]: dispatch 2024-05-09T08:47:08.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:08 smithi098 ceph-mon[30092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d685640-23a3-4c93-8a66-1db5b01c3252"}]': finished 2024-05-09T08:47:08.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:08 smithi098 ceph-mon[30092]: osdmap e78: 12 total, 11 up, 12 in 2024-05-09T08:47:08.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:08 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:08 smithi142 ceph-mon[34625]: pgmap v242: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-09T08:47:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:08 smithi142 ceph-mon[34625]: from='client.? 172.21.15.142:0/3284133483' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d685640-23a3-4c93-8a66-1db5b01c3252"}]: dispatch 2024-05-09T08:47:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:08 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d685640-23a3-4c93-8a66-1db5b01c3252"}]: dispatch 2024-05-09T08:47:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:08 smithi142 ceph-mon[34625]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d685640-23a3-4c93-8a66-1db5b01c3252"}]': finished 2024-05-09T08:47:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:08 smithi142 ceph-mon[34625]: osdmap e78: 12 total, 11 up, 12 in 2024-05-09T08:47:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:08 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:08.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:08 smithi105 ceph-mon[34498]: pgmap v242: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-09T08:47:08.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:08 smithi105 ceph-mon[34498]: from='client.? 172.21.15.142:0/3284133483' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d685640-23a3-4c93-8a66-1db5b01c3252"}]: dispatch 2024-05-09T08:47:08.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:08 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d685640-23a3-4c93-8a66-1db5b01c3252"}]: dispatch 2024-05-09T08:47:08.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:08 smithi105 ceph-mon[34498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d685640-23a3-4c93-8a66-1db5b01c3252"}]': finished 2024-05-09T08:47:08.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:08 smithi105 ceph-mon[34498]: osdmap e78: 12 total, 11 up, 12 in 2024-05-09T08:47:08.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:08 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:09.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:09 smithi105 ceph-mon[34498]: from='client.? 172.21.15.142:0/3939455189' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:47:09.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:09 smithi098 ceph-mon[30092]: from='client.? 172.21.15.142:0/3939455189' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:47:09.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:09 smithi142 ceph-mon[34625]: from='client.? 172.21.15.142:0/3939455189' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:47:10.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:10 smithi098 ceph-mon[30092]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-05-09T08:47:10.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:10 smithi142 ceph-mon[34625]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-05-09T08:47:10.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:10 smithi105 ceph-mon[34498]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-05-09T08:47:12.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:12 smithi098 ceph-mon[30092]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:12 smithi142 ceph-mon[34625]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:12.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:12 smithi105 ceph-mon[34498]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:14.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:14 smithi098 ceph-mon[30092]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:14.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:14 smithi142 ceph-mon[34625]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:14.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:14 smithi105 ceph-mon[34498]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:16.407 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:16 smithi142 ceph-mon[34625]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:16.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-09T08:47:16.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:16 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:16.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:16 smithi098 ceph-mon[30092]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:16.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:16 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-09T08:47:16.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:16 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:16.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:16 smithi105 ceph-mon[34498]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:16.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:16 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-09T08:47:16.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:16 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:17.434 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:17 smithi142 ceph-mon[34625]: Deploying daemon osd.11 on smithi142 2024-05-09T08:47:17.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:17 smithi098 ceph-mon[30092]: Deploying daemon osd.11 on smithi142 2024-05-09T08:47:17.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:17 smithi105 ceph-mon[34498]: Deploying daemon osd.11 on smithi142 2024-05-09T08:47:18.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:18 smithi098 ceph-mon[30092]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:18.620 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:18 smithi142 ceph-mon[34625]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:18.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:18 smithi105 ceph-mon[34498]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:20.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:20 smithi142 ceph-mon[34625]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:20.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:20 smithi098 ceph-mon[30092]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:20.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:20 smithi105 ceph-mon[34498]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:22.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:22 smithi142 ceph-mon[34625]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:22.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:22.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:22.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:22 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:22.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:22 smithi105 ceph-mon[34498]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:22.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:22.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:22.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:22 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:22.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:22 smithi098 ceph-mon[30092]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:22.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:22.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:22.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:22 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:24 smithi142 ceph-mon[34625]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:24.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:24 smithi105 ceph-mon[34498]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:24.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:24 smithi098 ceph-mon[30092]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:25.503 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 11 on host 'smithi142' 2024-05-09T08:47:25.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:25.504+0000 7f1744ff9700 1 -- 172.21.15.142:0/2968090564 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f17481a2e30 con 0x7f17300a2c20 2024-05-09T08:47:25.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:25.507+0000 7f174defb700 1 -- 172.21.15.142:0/2968090564 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f17300a2c20 msgr2=0x7f17300a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:25.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:25.507+0000 7f174defb700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f17300a2c20 0x7f17300a50e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1738005fd0 tx=0x7f1738005dd0 comp rx=0 tx=0).stop 2024-05-09T08:47:25.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:25.507+0000 7f174defb700 1 -- 172.21.15.142:0/2968090564 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1748102470 msgr2=0x7f17481038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:25.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:25.507+0000 7f174defb700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1748102470 0x7f17481038c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f173c00c970 tx=0x7f173c00cd30 comp rx=0 tx=0).stop 2024-05-09T08:47:25.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:25.507+0000 7f174defb700 1 -- 172.21.15.142:0/2968090564 shutdown_connections 2024-05-09T08:47:25.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:25.507+0000 7f174defb700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f17300a2c20 0x7f17300a50e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:25.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:25.507+0000 7f174defb700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1748101270 0x7f1748105270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:25.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:25.507+0000 7f174defb700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1748102e10 0x7f1748103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:25.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:25.507+0000 7f174defb700 1 --2- 172.21.15.142:0/2968090564 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1748102470 0x7f17481038c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:25.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:25.507+0000 7f174defb700 1 -- 172.21.15.142:0/2968090564 >> 172.21.15.142:0/2968090564 conn(0x7f17480fc810 msgr2=0x7f174806a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:25.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:25.507+0000 7f174defb700 1 -- 172.21.15.142:0/2968090564 shutdown_connections 2024-05-09T08:47:25.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:25.507+0000 7f174defb700 1 -- 172.21.15.142:0/2968090564 wait complete. 2024-05-09T08:47:25.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:25 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:25.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:25 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:25.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:25 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:25.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:25 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:25.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:25 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:25.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:25 smithi142 ceph-mon[34625]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:25.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:25 smithi142 ceph-mon[34625]: from='osd.11 [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:47:25.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:25 smithi142 ceph-mon[34625]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:47:25.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:25 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:25.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:25 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:25.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:25 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:25.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:25 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:25.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:25 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:25.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:25 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:25.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:25 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:25.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:25 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:25.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:25 smithi105 ceph-mon[34498]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:25.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:25 smithi105 ceph-mon[34498]: from='osd.11 [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:47:25.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:25 smithi105 ceph-mon[34498]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:47:25.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:25 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:25.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:25 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:25.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:25 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:26.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:25 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:26.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:25 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:26.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:25 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:26.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:25 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:26.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:25 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:26.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:25 smithi098 ceph-mon[30092]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:26.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:25 smithi098 ceph-mon[30092]: from='osd.11 [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:47:26.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:25 smithi098 ceph-mon[30092]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:47:26.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:25 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:26.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:25 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:26.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:25 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:26.150 DEBUG:teuthology.orchestra.run.smithi142:osd.11> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.11.service 2024-05-09T08:47:26.152 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-09T08:47:26.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd stat -f json 2024-05-09T08:47:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:47:26.701 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:26 smithi142 ceph-mon[34625]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-09T08:47:26.702 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:26 smithi142 ceph-mon[34625]: from='osd.11 [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:47:26.702 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:26 smithi142 ceph-mon[34625]: osdmap e79: 12 total, 11 up, 12 in 2024-05-09T08:47:26.702 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:26 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:26.702 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:26 smithi142 ceph-mon[34625]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:47:26.702 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 08:47:26 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T08:47:26.632+0000 7fa71c16f700 -1 osd.11 0 waiting for initial osdmap 2024-05-09T08:47:26.702 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 08:47:26 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T08:47:26.650+0000 7fa717f62700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:47:26.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:26 smithi098 ceph-mon[30092]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-09T08:47:26.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:26 smithi098 ceph-mon[30092]: from='osd.11 [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:47:26.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:26 smithi098 ceph-mon[30092]: osdmap e79: 12 total, 11 up, 12 in 2024-05-09T08:47:26.807 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:26 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:26.807 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:26 smithi098 ceph-mon[30092]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:47:26.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:26 smithi105 ceph-mon[34498]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-09T08:47:26.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:26 smithi105 ceph-mon[34498]: from='osd.11 [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:47:26.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:26 smithi105 ceph-mon[34498]: osdmap e79: 12 total, 11 up, 12 in 2024-05-09T08:47:26.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:26 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:26.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:26 smithi105 ceph-mon[34498]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-05-09T08:47:27.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:27 smithi098 ceph-mon[30092]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-05-09T08:47:27.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:27 smithi098 ceph-mon[30092]: osdmap e80: 12 total, 11 up, 12 in 2024-05-09T08:47:27.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:27 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:27.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:27 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:27.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:27 smithi098 ceph-mon[30092]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:27.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:27 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:27.903 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:27 smithi142 ceph-mon[34625]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-05-09T08:47:27.903 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:27 smithi142 ceph-mon[34625]: osdmap e80: 12 total, 11 up, 12 in 2024-05-09T08:47:27.903 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:27 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:27.903 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:27 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:27.903 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:27 smithi142 ceph-mon[34625]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:27.903 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:27 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:27.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:27 smithi105 ceph-mon[34498]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-05-09T08:47:27.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:27 smithi105 ceph-mon[34498]: osdmap e80: 12 total, 11 up, 12 in 2024-05-09T08:47:27.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:27 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:27.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:27 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:27.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:27 smithi105 ceph-mon[34498]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:47:27.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:27 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.223+0000 7f3596100700 1 -- 172.21.15.98:0/2540059969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3590101280 msgr2=0x7f35901016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.223+0000 7f3596100700 1 --2- 172.21.15.98:0/2540059969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3590101280 0x7f35901016a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f358000b0d0 tx=0x7f358000b3e0 comp rx=0 tx=0).stop 2024-05-09T08:47:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.223+0000 7f3596100700 1 -- 172.21.15.98:0/2540059969 shutdown_connections 2024-05-09T08:47:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.223+0000 7f3596100700 1 --2- 172.21.15.98:0/2540059969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3590102e20 0x7f35901096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.223+0000 7f3596100700 1 --2- 172.21.15.98:0/2540059969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3590101280 0x7f35901016a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.223+0000 7f3596100700 1 --2- 172.21.15.98:0/2540059969 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3590102480 0x7f35901028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.223+0000 7f3596100700 1 -- 172.21.15.98:0/2540059969 >> 172.21.15.98:0/2540059969 conn(0x7f35900fc820 msgr2=0x7f35900fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.223+0000 7f3596100700 1 -- 172.21.15.98:0/2540059969 shutdown_connections 2024-05-09T08:47:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.223+0000 7f3596100700 1 -- 172.21.15.98:0/2540059969 wait complete. 2024-05-09T08:47:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.224+0000 7f3596100700 1 Processor -- start 2024-05-09T08:47:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.224+0000 7f3596100700 1 -- start start 2024-05-09T08:47:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.224+0000 7f3596100700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3590101280 0x7f359006b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.224+0000 7f3596100700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3590102480 0x7f359019d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.224+0000 7f3596100700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3590102e20 0x7f359019f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.224+0000 7f3596100700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f359010cec0 con 0x7f3590101280 2024-05-09T08:47:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.224+0000 7f3596100700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f359010cd40 con 0x7f3590102e20 2024-05-09T08:47:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.224+0000 7f3596100700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f359010d040 con 0x7f3590102480 2024-05-09T08:47:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.224+0000 7f358f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3590101280 0x7f359006b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.224+0000 7f358f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3590101280 0x7f359006b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48264/0 (socket says 172.21.15.98:48264) 2024-05-09T08:47:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.224+0000 7f358f7fe700 1 -- 172.21.15.98:0/1752345655 learned_addr learned my addr 172.21.15.98:0/1752345655 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:47:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.224+0000 7f358effd700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3590102480 0x7f359019d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.224+0000 7f358ffff700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3590102e20 0x7f359019f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.225+0000 7f358f7fe700 1 -- 172.21.15.98:0/1752345655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3590102480 msgr2=0x7f359019d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.225+0000 7f358f7fe700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3590102480 0x7f359019d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.225+0000 7f358f7fe700 1 -- 172.21.15.98:0/1752345655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3590102e20 msgr2=0x7f359019f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.225+0000 7f358f7fe700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3590102e20 0x7f359019f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.225+0000 7f358f7fe700 1 -- 172.21.15.98:0/1752345655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35901a0210 con 0x7f3590101280 2024-05-09T08:47:28.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.225+0000 7f358ffff700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3590102e20 0x7f359019f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:47:28.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.225+0000 7f358f7fe700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3590101280 0x7f359006b5d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3580006130 tx=0x7f3580007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:28.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.225+0000 7f358cff9700 1 -- 172.21.15.98:0/1752345655 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3580021020 con 0x7f3590101280 2024-05-09T08:47:28.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.225+0000 7f3596100700 1 -- 172.21.15.98:0/1752345655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35901a04a0 con 0x7f3590101280 2024-05-09T08:47:28.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.225+0000 7f358cff9700 1 -- 172.21.15.98:0/1752345655 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f358001d950 con 0x7f3590101280 2024-05-09T08:47:28.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.225+0000 7f358cff9700 1 -- 172.21.15.98:0/1752345655 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35800054c0 con 0x7f3590101280 2024-05-09T08:47:28.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.225+0000 7f3596100700 1 -- 172.21.15.98:0/1752345655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35901a0a60 con 0x7f3590101280 2024-05-09T08:47:28.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.226+0000 7f3596100700 1 -- 172.21.15.98:0/1752345655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3590069010 con 0x7f3590101280 2024-05-09T08:47:28.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.227+0000 7f358cff9700 1 -- 172.21.15.98:0/1752345655 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3580024020 con 0x7f3590101280 2024-05-09T08:47:28.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.228+0000 7f358cff9700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f35780a2c20 0x7f35780a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:28.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.228+0000 7f358effd700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f35780a2c20 0x7f35780a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:28.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.228+0000 7f358cff9700 1 -- 172.21.15.98:0/1752345655 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f358000f020 con 0x7f3590101280 2024-05-09T08:47:28.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.228+0000 7f358effd700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f35780a2c20 0x7f35780a50e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f359006bb30 tx=0x7f3584009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:28.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.232+0000 7f358cff9700 1 -- 172.21.15.98:0/1752345655 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3580011e50 con 0x7f3590101280 2024-05-09T08:47:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.428+0000 7f3596100700 1 -- 172.21.15.98:0/1752345655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f3590062f30 con 0x7f3590101280 2024-05-09T08:47:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.428+0000 7f358cff9700 1 -- 172.21.15.98:0/1752345655 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f358009d360 con 0x7f3590101280 2024-05-09T08:47:28.430 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:47:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.430+0000 7f3596100700 1 -- 172.21.15.98:0/1752345655 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f35780a2c20 msgr2=0x7f35780a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.430+0000 7f3596100700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f35780a2c20 0x7f35780a50e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f359006bb30 tx=0x7f3584009190 comp rx=0 tx=0).stop 2024-05-09T08:47:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.430+0000 7f3596100700 1 -- 172.21.15.98:0/1752345655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3590101280 msgr2=0x7f359006b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.430+0000 7f3596100700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3590101280 0x7f359006b5d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3580006130 tx=0x7f3580007d90 comp rx=0 tx=0).stop 2024-05-09T08:47:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.431+0000 7f3596100700 1 -- 172.21.15.98:0/1752345655 shutdown_connections 2024-05-09T08:47:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.431+0000 7f3596100700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f35780a2c20 0x7f35780a50e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.431+0000 7f3596100700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3590102e20 0x7f359019f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.431+0000 7f3596100700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3590101280 0x7f359006b5d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.431+0000 7f3596100700 1 --2- 172.21.15.98:0/1752345655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3590102480 0x7f359019d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.431+0000 7f3596100700 1 -- 172.21.15.98:0/1752345655 >> 172.21.15.98:0/1752345655 conn(0x7f35900fc820 msgr2=0x7f3590105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:28.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.431+0000 7f3596100700 1 -- 172.21.15.98:0/1752345655 shutdown_connections 2024-05-09T08:47:28.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:28.431+0000 7f3596100700 1 -- 172.21.15.98:0/1752345655 wait complete. 2024-05-09T08:47:28.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:28 smithi098 ceph-mon[30092]: purged_snaps scrub starts 2024-05-09T08:47:28.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:28 smithi098 ceph-mon[30092]: purged_snaps scrub ok 2024-05-09T08:47:28.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:28 smithi098 ceph-mon[30092]: osd.11 [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207] boot 2024-05-09T08:47:28.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:28 smithi098 ceph-mon[30092]: osdmap e81: 12 total, 12 up, 12 in 2024-05-09T08:47:28.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:28 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:28.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:28 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1752345655' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-09T08:47:28.881 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:28 smithi142 ceph-mon[34625]: purged_snaps scrub starts 2024-05-09T08:47:28.881 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:28 smithi142 ceph-mon[34625]: purged_snaps scrub ok 2024-05-09T08:47:28.881 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:28 smithi142 ceph-mon[34625]: osd.11 [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207] boot 2024-05-09T08:47:28.881 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:28 smithi142 ceph-mon[34625]: osdmap e81: 12 total, 12 up, 12 in 2024-05-09T08:47:28.881 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:28 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:28.881 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:28 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1752345655' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-09T08:47:28.907 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:28 smithi105 ceph-mon[34498]: purged_snaps scrub starts 2024-05-09T08:47:28.907 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:28 smithi105 ceph-mon[34498]: purged_snaps scrub ok 2024-05-09T08:47:28.907 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:28 smithi105 ceph-mon[34498]: osd.11 [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207] boot 2024-05-09T08:47:28.907 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:28 smithi105 ceph-mon[34498]: osdmap e81: 12 total, 12 up, 12 in 2024-05-09T08:47:28.907 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:28 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:47:28.907 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:28 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1752345655' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-09T08:47:29.093 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1715244447,"num_in_osds":12,"osd_in_since":1715244427,"num_remapped_pgs":0} 2024-05-09T08:47:29.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-09T08:47:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:47:29.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:29 smithi098 ceph-mon[30092]: osdmap e82: 12 total, 12 up, 12 in 2024-05-09T08:47:29.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:29 smithi098 ceph-mon[30092]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:29.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:29 smithi098 ceph-mon[30092]: Detected new or changed devices on smithi142 2024-05-09T08:47:29.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:29.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:29.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:29.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:29.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:29.727 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:29.727 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:29 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi142 to 4272M 2024-05-09T08:47:29.727 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:29.727 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:29.727 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:29.727 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:29 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:29.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:29 smithi142 ceph-mon[34625]: osdmap e82: 12 total, 12 up, 12 in 2024-05-09T08:47:29.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:29 smithi142 ceph-mon[34625]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:29.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:29 smithi142 ceph-mon[34625]: Detected new or changed devices on smithi142 2024-05-09T08:47:29.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:29.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:29.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:29.910 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:29.910 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:29.910 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:29.910 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:29 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi142 to 4272M 2024-05-09T08:47:29.910 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:29.910 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:29.911 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:29.911 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:29 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:29.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:29 smithi105 ceph-mon[34498]: osdmap e82: 12 total, 12 up, 12 in 2024-05-09T08:47:29.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:29 smithi105 ceph-mon[34498]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:29.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:29 smithi105 ceph-mon[34498]: Detected new or changed devices on smithi142 2024-05-09T08:47:29.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:29.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:29.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:29.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:29.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:29.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:47:29.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:29 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi142 to 4272M 2024-05-09T08:47:29.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:29.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:29.989 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:29.989 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:29 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:31.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.155+0000 7f0c05ce7700 1 -- 172.21.15.98:0/1874151938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00101be0 msgr2=0x7f0c0010e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:31.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.155+0000 7f0c05ce7700 1 --2- 172.21.15.98:0/1874151938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00101be0 0x7f0c0010e0d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800b0a0 tx=0x7f0bf800b3b0 comp rx=0 tx=0).stop 2024-05-09T08:47:31.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.155+0000 7f0c05ce7700 1 -- 172.21.15.98:0/1874151938 shutdown_connections 2024-05-09T08:47:31.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.155+0000 7f0c05ce7700 1 --2- 172.21.15.98:0/1874151938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00101be0 0x7f0c0010e0d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:31.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.155+0000 7f0c05ce7700 1 --2- 172.21.15.98:0/1874151938 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c00107290 0x7f0c00107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:31.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.155+0000 7f0c05ce7700 1 --2- 172.21.15.98:0/1874151938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c00101220 0x7f0c001016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:31.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.155+0000 7f0c05ce7700 1 -- 172.21.15.98:0/1874151938 >> 172.21.15.98:0/1874151938 conn(0x7f0c000781d0 msgr2=0x7f0c000785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:31.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.155+0000 7f0c05ce7700 1 -- 172.21.15.98:0/1874151938 shutdown_connections 2024-05-09T08:47:31.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.155+0000 7f0c05ce7700 1 -- 172.21.15.98:0/1874151938 wait complete. 2024-05-09T08:47:31.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.156+0000 7f0c05ce7700 1 Processor -- start 2024-05-09T08:47:31.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.156+0000 7f0c05ce7700 1 -- start start 2024-05-09T08:47:31.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.156+0000 7f0c05ce7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00101220 0x7f0c00102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:31.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.156+0000 7f0c05ce7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c00101be0 0x7f0c00103440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:31.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.156+0000 7f0c05ce7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c00107290 0x7f0c00103980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:31.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.156+0000 7f0c05ce7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c00111600 con 0x7f0c00101220 2024-05-09T08:47:31.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.156+0000 7f0c05ce7700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0c00111480 con 0x7f0c00101be0 2024-05-09T08:47:31.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.156+0000 7f0c05ce7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0c00111780 con 0x7f0c00107290 2024-05-09T08:47:31.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0bff7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00101220 0x7f0c00102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:31.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0bff7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00101220 0x7f0c00102f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48278/0 (socket says 172.21.15.98:48278) 2024-05-09T08:47:31.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0bff7fe700 1 -- 172.21.15.98:0/4081553252 learned_addr learned my addr 172.21.15.98:0/4081553252 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:47:31.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0bfeffd700 1 --2- 172.21.15.98:0/4081553252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c00101be0 0x7f0c00103440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:31.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0bfffff700 1 --2- 172.21.15.98:0/4081553252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c00107290 0x7f0c00103980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:31.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0bff7fe700 1 -- 172.21.15.98:0/4081553252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c00107290 msgr2=0x7f0c00103980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:31.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0bff7fe700 1 --2- 172.21.15.98:0/4081553252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c00107290 0x7f0c00103980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:31.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0bff7fe700 1 -- 172.21.15.98:0/4081553252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c00101be0 msgr2=0x7f0c00103440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:31.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0bff7fe700 1 --2- 172.21.15.98:0/4081553252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c00101be0 0x7f0c00103440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:31.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0bff7fe700 1 -- 172.21.15.98:0/4081553252 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c001a7040 con 0x7f0c00101220 2024-05-09T08:47:31.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0bff7fe700 1 --2- 172.21.15.98:0/4081553252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00101220 0x7f0c00102f00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0bf000b530 tx=0x7f0bf000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:31.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0bfcff9700 1 -- 172.21.15.98:0/4081553252 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0015020 con 0x7f0c00101220 2024-05-09T08:47:31.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0c05ce7700 1 -- 172.21.15.98:0/4081553252 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c001a7330 con 0x7f0c00101220 2024-05-09T08:47:31.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0bfcff9700 1 -- 172.21.15.98:0/4081553252 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bf0011d10 con 0x7f0c00101220 2024-05-09T08:47:31.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0bfcff9700 1 -- 172.21.15.98:0/4081553252 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf000f2e0 con 0x7f0c00101220 2024-05-09T08:47:31.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.157+0000 7f0c05ce7700 1 -- 172.21.15.98:0/4081553252 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c001a7870 con 0x7f0c00101220 2024-05-09T08:47:31.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.158+0000 7f0c05ce7700 1 -- 172.21.15.98:0/4081553252 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c00102940 con 0x7f0c00101220 2024-05-09T08:47:31.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.159+0000 7f0bfcff9700 1 -- 172.21.15.98:0/4081553252 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0bf0018020 con 0x7f0c00101220 2024-05-09T08:47:31.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.160+0000 7f0bfcff9700 1 --2- 172.21.15.98:0/4081553252 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f0be80a2c20 0x7f0be80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:31.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.160+0000 7f0bfeffd700 1 --2- 172.21.15.98:0/4081553252 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f0be80a2c20 0x7f0be80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:31.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.160+0000 7f0bfcff9700 1 -- 172.21.15.98:0/4081553252 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0bf00ce9c0 con 0x7f0c00101220 2024-05-09T08:47:31.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.161+0000 7f0bfeffd700 1 --2- 172.21.15.98:0/4081553252 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f0be80a2c20 0x7f0be80a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0c000ff8c0 tx=0x7f0bf4009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:31.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.164+0000 7f0bfcff9700 1 -- 172.21.15.98:0/4081553252 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0bf001f020 con 0x7f0c00101220 2024-05-09T08:47:31.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.360+0000 7f0c05ce7700 1 -- 172.21.15.98:0/4081553252 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0c001a7b90 con 0x7f0c00101220 2024-05-09T08:47:31.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.361+0000 7f0bfcff9700 1 -- 172.21.15.98:0/4081553252 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18478 (secure 0 0 0) 0x7f0bf009bdc0 con 0x7f0c00101220 2024-05-09T08:47:31.360 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:47:31.361 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":82,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:47:28.644215+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:47:31.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.363+0000 7f0c05ce7700 1 -- 172.21.15.98:0/4081553252 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f0be80a2c20 msgr2=0x7f0be80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:31.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.363+0000 7f0c05ce7700 1 --2- 172.21.15.98:0/4081553252 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f0be80a2c20 0x7f0be80a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0c000ff8c0 tx=0x7f0bf4009190 comp rx=0 tx=0).stop 2024-05-09T08:47:31.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.363+0000 7f0c05ce7700 1 -- 172.21.15.98:0/4081553252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00101220 msgr2=0x7f0c00102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:31.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.363+0000 7f0c05ce7700 1 --2- 172.21.15.98:0/4081553252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00101220 0x7f0c00102f00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0bf000b530 tx=0x7f0bf000b840 comp rx=0 tx=0).stop 2024-05-09T08:47:31.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.363+0000 7f0c05ce7700 1 -- 172.21.15.98:0/4081553252 shutdown_connections 2024-05-09T08:47:31.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.363+0000 7f0c05ce7700 1 --2- 172.21.15.98:0/4081553252 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f0be80a2c20 0x7f0be80a50e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:31.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.363+0000 7f0c05ce7700 1 --2- 172.21.15.98:0/4081553252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c00107290 0x7f0c00103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:31.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.363+0000 7f0c05ce7700 1 --2- 172.21.15.98:0/4081553252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00101220 0x7f0c00102f00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:31.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.363+0000 7f0c05ce7700 1 --2- 172.21.15.98:0/4081553252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c00101be0 0x7f0c00103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:31.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.363+0000 7f0c05ce7700 1 -- 172.21.15.98:0/4081553252 >> 172.21.15.98:0/4081553252 conn(0x7f0c000781d0 msgr2=0x7f0c00100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:31.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.363+0000 7f0c05ce7700 1 -- 172.21.15.98:0/4081553252 shutdown_connections 2024-05-09T08:47:31.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:31.363+0000 7f0c05ce7700 1 -- 172.21.15.98:0/4081553252 wait complete. 2024-05-09T08:47:32.037 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:43:39.424443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T08:47:32.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-09T08:47:32.277 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:47:32.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:32 smithi098 ceph-mon[30092]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:32.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:32 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4081553252' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:47:32.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:32 smithi105 ceph-mon[34498]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:32.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:32 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/4081553252' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:47:32.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:32 smithi142 ceph-mon[34625]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:32.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:32 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/4081553252' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:47:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.967+0000 7fb931931700 1 -- 172.21.15.98:0/2311684201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c101be0 msgr2=0x7fb92c10e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.967+0000 7fb931931700 1 --2- 172.21.15.98:0/2311684201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c101be0 0x7fb92c10e0d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb92400b0a0 tx=0x7fb92400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:47:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.967+0000 7fb931931700 1 -- 172.21.15.98:0/2311684201 shutdown_connections 2024-05-09T08:47:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.967+0000 7fb931931700 1 --2- 172.21.15.98:0/2311684201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c101be0 0x7fb92c10e0d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.967+0000 7fb931931700 1 --2- 172.21.15.98:0/2311684201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb92c107290 0x7fb92c107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.967+0000 7fb931931700 1 --2- 172.21.15.98:0/2311684201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb92c101220 0x7fb92c1016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.967+0000 7fb931931700 1 -- 172.21.15.98:0/2311684201 >> 172.21.15.98:0/2311684201 conn(0x7fb92c0781d0 msgr2=0x7fb92c0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.968+0000 7fb931931700 1 -- 172.21.15.98:0/2311684201 shutdown_connections 2024-05-09T08:47:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.968+0000 7fb931931700 1 -- 172.21.15.98:0/2311684201 wait complete. 2024-05-09T08:47:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.968+0000 7fb931931700 1 Processor -- start 2024-05-09T08:47:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.968+0000 7fb931931700 1 -- start start 2024-05-09T08:47:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb931931700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c101220 0x7fb92c19d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb931931700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb92c101be0 0x7fb92c19db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb931931700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb92c107290 0x7fb92c1977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb931931700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c111580 con 0x7fb92c101220 2024-05-09T08:47:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb931931700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c111400 con 0x7fb92c101be0 2024-05-09T08:47:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb931931700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c111700 con 0x7fb92c107290 2024-05-09T08:47:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb92affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c101220 0x7fb92c19d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb92affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c101220 0x7fb92c19d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48284/0 (socket says 172.21.15.98:48284) 2024-05-09T08:47:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb92affd700 1 -- 172.21.15.98:0/2144396233 learned_addr learned my addr 172.21.15.98:0/2144396233 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:47:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb92b7fe700 1 --2- 172.21.15.98:0/2144396233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb92c107290 0x7fb92c1977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb92affd700 1 -- 172.21.15.98:0/2144396233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb92c107290 msgr2=0x7fb92c1977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb92affd700 1 --2- 172.21.15.98:0/2144396233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb92c107290 0x7fb92c1977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb92a7fc700 1 --2- 172.21.15.98:0/2144396233 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb92c101be0 0x7fb92c19db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb92affd700 1 -- 172.21.15.98:0/2144396233 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb92c101be0 msgr2=0x7fb92c19db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb92affd700 1 --2- 172.21.15.98:0/2144396233 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb92c101be0 0x7fb92c19db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb92affd700 1 -- 172.21.15.98:0/2144396233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92c197fa0 con 0x7fb92c101220 2024-05-09T08:47:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.969+0000 7fb92affd700 1 --2- 172.21.15.98:0/2144396233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c101220 0x7fb92c19d640 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00c9b0 tx=0x7fb91c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.970+0000 7fb93092f700 1 -- 172.21.15.98:0/2144396233 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c007b00 con 0x7fb92c101220 2024-05-09T08:47:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.970+0000 7fb93092f700 1 -- 172.21.15.98:0/2144396233 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb91c007ca0 con 0x7fb92c101220 2024-05-09T08:47:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.970+0000 7fb931931700 1 -- 172.21.15.98:0/2144396233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb92c198290 con 0x7fb92c101220 2024-05-09T08:47:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.970+0000 7fb93092f700 1 -- 172.21.15.98:0/2144396233 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c018730 con 0x7fb92c101220 2024-05-09T08:47:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.970+0000 7fb931931700 1 -- 172.21.15.98:0/2144396233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb92c1a44a0 con 0x7fb92c101220 2024-05-09T08:47:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.971+0000 7fb931931700 1 -- 172.21.15.98:0/2144396233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb92c102940 con 0x7fb92c101220 2024-05-09T08:47:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.972+0000 7fb93092f700 1 -- 172.21.15.98:0/2144396233 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb91c0188d0 con 0x7fb92c101220 2024-05-09T08:47:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.972+0000 7fb93092f700 1 --2- 172.21.15.98:0/2144396233 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb9140a2c20 0x7fb9140a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.973+0000 7fb92a7fc700 1 --2- 172.21.15.98:0/2144396233 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb9140a2c20 0x7fb9140a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.973+0000 7fb93092f700 1 -- 172.21.15.98:0/2144396233 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fb91c0cee00 con 0x7fb92c101220 2024-05-09T08:47:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.973+0000 7fb92a7fc700 1 --2- 172.21.15.98:0/2144396233 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb9140a2c20 0x7fb9140a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb920006fd0 tx=0x7fb920008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:33.976+0000 7fb93092f700 1 -- 172.21.15.98:0/2144396233 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb91c014030 con 0x7fb92c101220 2024-05-09T08:47:34.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:34.170+0000 7fb931931700 1 -- 172.21.15.98:0/2144396233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb92c02db80 con 0x7fb92c101220 2024-05-09T08:47:34.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:34.170+0000 7fb93092f700 1 -- 172.21.15.98:0/2144396233 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fb91c09c290 con 0x7fb92c101220 2024-05-09T08:47:34.170 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-09T08:47:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:34.172+0000 7fb931931700 1 -- 172.21.15.98:0/2144396233 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb9140a2c20 msgr2=0x7fb9140a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:34.172+0000 7fb931931700 1 --2- 172.21.15.98:0/2144396233 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb9140a2c20 0x7fb9140a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb920006fd0 tx=0x7fb920008040 comp rx=0 tx=0).stop 2024-05-09T08:47:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:34.172+0000 7fb931931700 1 -- 172.21.15.98:0/2144396233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c101220 msgr2=0x7fb92c19d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:34.172+0000 7fb931931700 1 --2- 172.21.15.98:0/2144396233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c101220 0x7fb92c19d640 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00c9b0 tx=0x7fb91c00ccc0 comp rx=0 tx=0).stop 2024-05-09T08:47:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:34.172+0000 7fb931931700 1 -- 172.21.15.98:0/2144396233 shutdown_connections 2024-05-09T08:47:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:34.172+0000 7fb931931700 1 --2- 172.21.15.98:0/2144396233 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb9140a2c20 0x7fb9140a50e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:34.172+0000 7fb931931700 1 --2- 172.21.15.98:0/2144396233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb92c107290 0x7fb92c1977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:34.172+0000 7fb931931700 1 --2- 172.21.15.98:0/2144396233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c101220 0x7fb92c19d640 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:34.172+0000 7fb931931700 1 --2- 172.21.15.98:0/2144396233 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb92c101be0 0x7fb92c19db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:34.172+0000 7fb931931700 1 -- 172.21.15.98:0/2144396233 >> 172.21.15.98:0/2144396233 conn(0x7fb92c0781d0 msgr2=0x7fb92c0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:34.172+0000 7fb931931700 1 -- 172.21.15.98:0/2144396233 shutdown_connections 2024-05-09T08:47:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:34.172+0000 7fb931931700 1 -- 172.21.15.98:0/2144396233 wait complete. 2024-05-09T08:47:34.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:34 smithi105 ceph-mon[34498]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:34.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:34 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2144396233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:47:34.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:34 smithi098 ceph-mon[30092]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:34.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:34 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2144396233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:47:34.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:34 smithi142 ceph-mon[34625]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:34.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:34 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2144396233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:47:34.887 INFO:tasks.cephadm:Adding mds.a on smithi098 2024-05-09T08:47:34.888 INFO:tasks.cephadm:Adding mds.d on smithi098 2024-05-09T08:47:34.888 INFO:tasks.cephadm:Adding mds.g on smithi098 2024-05-09T08:47:34.888 INFO:tasks.cephadm:Adding mds.j on smithi098 2024-05-09T08:47:34.888 INFO:tasks.cephadm:Adding mds.b on smithi105 2024-05-09T08:47:34.888 INFO:tasks.cephadm:Adding mds.e on smithi105 2024-05-09T08:47:34.888 INFO:tasks.cephadm:Adding mds.h on smithi105 2024-05-09T08:47:34.888 INFO:tasks.cephadm:Adding mds.k on smithi105 2024-05-09T08:47:34.888 INFO:tasks.cephadm:Adding mds.c on smithi142 2024-05-09T08:47:34.888 INFO:tasks.cephadm:Adding mds.f on smithi142 2024-05-09T08:47:34.888 INFO:tasks.cephadm:Adding mds.i on smithi142 2024-05-09T08:47:34.888 INFO:tasks.cephadm:Adding mds.l on smithi142 2024-05-09T08:47:34.889 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph orch apply mds all '12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l' 2024-05-09T08:47:35.139 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:47:36.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:36 smithi105 ceph-mon[34498]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:36.549 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:36 smithi142 ceph-mon[34625]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:36.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:36 smithi098 ceph-mon[30092]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.778+0000 7fa7a5e89700 1 -- 172.21.15.142:0/3153173108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7a0102480 msgr2=0x7fa7a01028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.778+0000 7fa7a5e89700 1 --2- 172.21.15.142:0/3153173108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7a0102480 0x7fa7a01028e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa788009990 tx=0x7fa788009ca0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.778+0000 7fa7a5e89700 1 -- 172.21.15.142:0/3153173108 shutdown_connections 2024-05-09T08:47:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.778+0000 7fa7a5e89700 1 --2- 172.21.15.142:0/3153173108 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7a0101280 0x7fa7a01016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.778+0000 7fa7a5e89700 1 --2- 172.21.15.142:0/3153173108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0102e20 0x7fa7a01096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.778+0000 7fa7a5e89700 1 --2- 172.21.15.142:0/3153173108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7a0102480 0x7fa7a01028e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.778+0000 7fa7a5e89700 1 -- 172.21.15.142:0/3153173108 >> 172.21.15.142:0/3153173108 conn(0x7fa7a00fc820 msgr2=0x7fa7a00fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:36.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.778+0000 7fa7a5e89700 1 -- 172.21.15.142:0/3153173108 shutdown_connections 2024-05-09T08:47:36.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.778+0000 7fa7a5e89700 1 -- 172.21.15.142:0/3153173108 wait complete. 2024-05-09T08:47:36.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.779+0000 7fa7a5e89700 1 Processor -- start 2024-05-09T08:47:36.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.779+0000 7fa7a5e89700 1 -- start start 2024-05-09T08:47:36.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.779+0000 7fa7a5e89700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0101280 0x7fa7a01974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.779+0000 7fa7a5e89700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7a0102480 0x7fa7a0197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.779+0000 7fa7a5e89700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7a0102e20 0x7fa7a019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.779+0000 7fa7a5e89700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a010ce40 con 0x7fa7a0101280 2024-05-09T08:47:36.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.779+0000 7fa7a5e89700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a010ccc0 con 0x7fa7a0102480 2024-05-09T08:47:36.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.779+0000 7fa7a5e89700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a010cfc0 con 0x7fa7a0102e20 2024-05-09T08:47:36.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.779+0000 7fa79ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7a0102e20 0x7fa7a019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:36.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.779+0000 7fa79ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7a0102e20 0x7fa7a019eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:32866/0 (socket says 172.21.15.142:32866) 2024-05-09T08:47:36.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.779+0000 7fa79ffff700 1 -- 172.21.15.142:0/4118480759 learned_addr learned my addr 172.21.15.142:0/4118480759 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-05-09T08:47:36.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.779+0000 7fa79f7fe700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0101280 0x7fa7a01974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:36.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.779+0000 7fa796ffd700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7a0102480 0x7fa7a0197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:36.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.780+0000 7fa79ffff700 1 -- 172.21.15.142:0/4118480759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7a0102480 msgr2=0x7fa7a0197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.780+0000 7fa79ffff700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7a0102480 0x7fa7a0197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.780+0000 7fa79ffff700 1 -- 172.21.15.142:0/4118480759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0101280 msgr2=0x7fa7a01974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.780+0000 7fa79ffff700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0101280 0x7fa7a01974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.780+0000 7fa79ffff700 1 -- 172.21.15.142:0/4118480759 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a019f150 con 0x7fa7a0102e20 2024-05-09T08:47:36.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.780+0000 7fa796ffd700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7a0102480 0x7fa7a0197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:47:36.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.780+0000 7fa79ffff700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7a0102e20 0x7fa7a019eab0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa79800add0 tx=0x7fa79800c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:36.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.780+0000 7fa79d7fa700 1 -- 172.21.15.142:0/4118480759 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798017020 con 0x7fa7a0102e20 2024-05-09T08:47:36.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.780+0000 7fa7a5e89700 1 -- 172.21.15.142:0/4118480759 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a019f440 con 0x7fa7a0102e20 2024-05-09T08:47:36.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.780+0000 7fa79d7fa700 1 -- 172.21.15.142:0/4118480759 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa79800ce20 con 0x7fa7a0102e20 2024-05-09T08:47:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.780+0000 7fa79d7fa700 1 -- 172.21.15.142:0/4118480759 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7980053e0 con 0x7fa7a0102e20 2024-05-09T08:47:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.780+0000 7fa7a5e89700 1 -- 172.21.15.142:0/4118480759 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a019f930 con 0x7fa7a0102e20 2024-05-09T08:47:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.781+0000 7fa7a5e89700 1 -- 172.21.15.142:0/4118480759 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a0069010 con 0x7fa7a0102e20 2024-05-09T08:47:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.782+0000 7fa79d7fa700 1 -- 172.21.15.142:0/4118480759 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa79800c940 con 0x7fa7a0102e20 2024-05-09T08:47:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.782+0000 7fa79d7fa700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa78c0a2c20 0x7fa78c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.783+0000 7fa79d7fa700 1 -- 172.21.15.142:0/4118480759 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa7980ce3a0 con 0x7fa7a0102e20 2024-05-09T08:47:36.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.783+0000 7fa79f7fe700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa78c0a2c20 0x7fa78c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:36.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.783+0000 7fa79f7fe700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa78c0a2c20 0x7fa78c0a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa7a01022e0 tx=0x7fa79000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.787+0000 7fa79d7fa700 1 -- 172.21.15.142:0/4118480759 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa798021020 con 0x7fa7a0102e20 2024-05-09T08:47:36.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:36.992+0000 7fa7a5e89700 1 -- 172.21.15.142:0/4118480759 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l", "target": ["mon-mgr", ""]}) -- 0x7fa7a0062a20 con 0x7fa78c0a2c20 2024-05-09T08:47:37.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:37.000+0000 7fa79d7fa700 1 -- 172.21.15.142:0/4118480759 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fa7a0062a20 con 0x7fa78c0a2c20 2024-05-09T08:47:37.000 INFO:teuthology.orchestra.run.smithi142.stdout:Scheduled mds.all update... 2024-05-09T08:47:37.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:37.003+0000 7fa7a5e89700 1 -- 172.21.15.142:0/4118480759 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa78c0a2c20 msgr2=0x7fa78c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:37.003+0000 7fa7a5e89700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa78c0a2c20 0x7fa78c0a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa7a01022e0 tx=0x7fa79000b410 comp rx=0 tx=0).stop 2024-05-09T08:47:37.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:37.003+0000 7fa7a5e89700 1 -- 172.21.15.142:0/4118480759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7a0102e20 msgr2=0x7fa7a019eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:37.003+0000 7fa7a5e89700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7a0102e20 0x7fa7a019eab0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa79800add0 tx=0x7fa79800c330 comp rx=0 tx=0).stop 2024-05-09T08:47:37.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:37.003+0000 7fa7a5e89700 1 -- 172.21.15.142:0/4118480759 shutdown_connections 2024-05-09T08:47:37.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:37.003+0000 7fa7a5e89700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa78c0a2c20 0x7fa78c0a50e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:37.003+0000 7fa7a5e89700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7a0102e20 0x7fa7a019eab0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:37.003+0000 7fa7a5e89700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7a0101280 0x7fa7a01974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:37.003+0000 7fa7a5e89700 1 --2- 172.21.15.142:0/4118480759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7a0102480 0x7fa7a0197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:37.003+0000 7fa7a5e89700 1 -- 172.21.15.142:0/4118480759 >> 172.21.15.142:0/4118480759 conn(0x7fa7a00fc820 msgr2=0x7fa7a0105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:37.003+0000 7fa7a5e89700 1 -- 172.21.15.142:0/4118480759 shutdown_connections 2024-05-09T08:47:37.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-09T08:47:37.003+0000 7fa7a5e89700 1 -- 172.21.15.142:0/4118480759 wait complete. 2024-05-09T08:47:37.707 DEBUG:teuthology.orchestra.run.smithi098:mds.a> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.a.service 2024-05-09T08:47:37.710 DEBUG:teuthology.orchestra.run.smithi098:mds.d> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.d.service 2024-05-09T08:47:37.755 DEBUG:teuthology.orchestra.run.smithi098:mds.g> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.g.service 2024-05-09T08:47:37.798 DEBUG:teuthology.orchestra.run.smithi098:mds.j> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.j.service 2024-05-09T08:47:37.841 DEBUG:teuthology.orchestra.run.smithi105:mds.b> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.b.service 2024-05-09T08:47:37.843 DEBUG:teuthology.orchestra.run.smithi105:mds.e> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.e.service 2024-05-09T08:47:37.887 DEBUG:teuthology.orchestra.run.smithi105:mds.h> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.h.service 2024-05-09T08:47:37.930 DEBUG:teuthology.orchestra.run.smithi105:mds.k> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.k.service 2024-05-09T08:47:37.973 DEBUG:teuthology.orchestra.run.smithi142:mds.c> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.c.service 2024-05-09T08:47:37.976 DEBUG:teuthology.orchestra.run.smithi142:mds.f> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.f.service 2024-05-09T08:47:38.019 DEBUG:teuthology.orchestra.run.smithi142:mds.i> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.i.service 2024-05-09T08:47:38.064 DEBUG:teuthology.orchestra.run.smithi142:mds.l> sudo journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.l.service 2024-05-09T08:47:38.106 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-09T08:47:38.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:47:38.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.228+0000 7fdf060d4640 1 -- 172.21.15.98:0/4263197852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf00074290 msgr2=0x7fdf00074710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.228+0000 7fdf060d4640 1 --2- 172.21.15.98:0/4263197852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf00074290 0x7fdf00074710 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009930 tx=0x7fdee802a420 comp rx=0 tx=0).stop 2024-05-09T08:47:38.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 -- 172.21.15.98:0/4263197852 shutdown_connections 2024-05-09T08:47:38.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 --2- 172.21.15.98:0/4263197852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf0010de50 0x7fdf00110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 --2- 172.21.15.98:0/4263197852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf00074290 0x7fdf00074710 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 --2- 172.21.15.98:0/4263197852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf00075870 0x7fdf00073cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 -- 172.21.15.98:0/4263197852 >> 172.21.15.98:0/4263197852 conn(0x7fdf000fce10 msgr2=0x7fdf000ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 -- 172.21.15.98:0/4263197852 shutdown_connections 2024-05-09T08:47:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 -- 172.21.15.98:0/4263197852 wait complete. 2024-05-09T08:47:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 Processor -- start 2024-05-09T08:47:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 -- start start 2024-05-09T08:47:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf00074290 0x7fdf0019e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf00075870 0x7fdf0019e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf0010de50 0x7fdf001a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf00113ad0 con 0x7fdf00074290 2024-05-09T08:47:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdf00113950 con 0x7fdf0010de50 2024-05-09T08:47:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdf060d4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdf00113c50 con 0x7fdf00075870 2024-05-09T08:47:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdeff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf00074290 0x7fdf0019e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdeff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf00074290 0x7fdf0019e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46564/0 (socket says 172.21.15.98:46564) 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.229+0000 7fdeff7fe640 1 -- 172.21.15.98:0/1114999913 learned_addr learned my addr 172.21.15.98:0/1114999913 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.230+0000 7fdef7fff640 1 --2- 172.21.15.98:0/1114999913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf00075870 0x7fdf0019e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.230+0000 7fdefffff640 1 --2- 172.21.15.98:0/1114999913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf0010de50 0x7fdf001a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.230+0000 7fdeff7fe640 1 -- 172.21.15.98:0/1114999913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf00075870 msgr2=0x7fdf0019e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.230+0000 7fdeff7fe640 1 --2- 172.21.15.98:0/1114999913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf00075870 0x7fdf0019e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.230+0000 7fdeff7fe640 1 -- 172.21.15.98:0/1114999913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf0010de50 msgr2=0x7fdf001a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.230+0000 7fdeff7fe640 1 --2- 172.21.15.98:0/1114999913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf0010de50 0x7fdf001a57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.230+0000 7fdeff7fe640 1 -- 172.21.15.98:0/1114999913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf001a5e10 con 0x7fdf00074290 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.230+0000 7fdeff7fe640 1 --2- 172.21.15.98:0/1114999913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf00074290 0x7fdf0019e1f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdef000d950 tx=0x7fdef000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.230+0000 7fdefd7fa640 1 -- 172.21.15.98:0/1114999913 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0017030 con 0x7fdf00074290 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.230+0000 7fdf060d4640 1 -- 172.21.15.98:0/1114999913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf001a6100 con 0x7fdf00074290 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.230+0000 7fdefd7fa640 1 -- 172.21.15.98:0/1114999913 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdef0004020 con 0x7fdf00074290 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.230+0000 7fdefd7fa640 1 -- 172.21.15.98:0/1114999913 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef00047a0 con 0x7fdf00074290 2024-05-09T08:47:38.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.230+0000 7fdf060d4640 1 -- 172.21.15.98:0/1114999913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf001a6620 con 0x7fdf00074290 2024-05-09T08:47:38.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.231+0000 7fdf060d4640 1 -- 172.21.15.98:0/1114999913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf00104020 con 0x7fdf00074290 2024-05-09T08:47:38.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.232+0000 7fdefd7fa640 1 -- 172.21.15.98:0/1114999913 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fdef00041d0 con 0x7fdf00074290 2024-05-09T08:47:38.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.233+0000 7fdefd7fa640 1 --2- 172.21.15.98:0/1114999913 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdeec0a2b90 0x7fdeec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.233+0000 7fdef7fff640 1 --2- 172.21.15.98:0/1114999913 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdeec0a2b90 0x7fdeec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.233+0000 7fdefd7fa640 1 -- 172.21.15.98:0/1114999913 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fdef00ce610 con 0x7fdf00074290 2024-05-09T08:47:38.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.233+0000 7fdef7fff640 1 --2- 172.21.15.98:0/1114999913 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdeec0a2b90 0x7fdeec0a5050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdee8005ab0 tx=0x7fdee8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:38.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:38 smithi105 ceph-mon[34498]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:47:38.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:38 smithi105 ceph-mon[34498]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l;count:12 2024-05-09T08:47:38.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:38.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:38.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:38 smithi105 ceph-mon[34498]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:38.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:38.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:38.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:38.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:38.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:38.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:38 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:38.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.238+0000 7fdefd7fa640 1 -- 172.21.15.98:0/1114999913 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdef009ba10 con 0x7fdf00074290 2024-05-09T08:47:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:38 smithi098 ceph-mon[30092]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:47:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:38 smithi098 ceph-mon[30092]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l;count:12 2024-05-09T08:47:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:38 smithi098 ceph-mon[30092]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:38.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:38 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:38 smithi142 ceph-mon[34625]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:47:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:38 smithi142 ceph-mon[34625]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l;count:12 2024-05-09T08:47:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:38 smithi142 ceph-mon[34625]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:38.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:38.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:38.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:38.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:38.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:38 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.417+0000 7fdf060d4640 1 -- 172.21.15.98:0/1114999913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdf00062430 con 0x7fdf00074290 2024-05-09T08:47:38.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.418+0000 7fdefd7fa640 1 -- 172.21.15.98:0/1114999913 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18478 (secure 0 0 0) 0x7fdef009b310 con 0x7fdf00074290 2024-05-09T08:47:38.418 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:47:38.418 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":82,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:47:28.644215+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:47:38.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.420+0000 7fdf060d4640 1 -- 172.21.15.98:0/1114999913 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdeec0a2b90 msgr2=0x7fdeec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.420+0000 7fdf060d4640 1 --2- 172.21.15.98:0/1114999913 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdeec0a2b90 0x7fdeec0a5050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdee8005ab0 tx=0x7fdee8000bc0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.420+0000 7fdf060d4640 1 -- 172.21.15.98:0/1114999913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf00074290 msgr2=0x7fdf0019e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.420+0000 7fdf060d4640 1 --2- 172.21.15.98:0/1114999913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf00074290 0x7fdf0019e1f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdef000d950 tx=0x7fdef000de10 comp rx=0 tx=0).stop 2024-05-09T08:47:38.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.420+0000 7fdf060d4640 1 -- 172.21.15.98:0/1114999913 shutdown_connections 2024-05-09T08:47:38.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.420+0000 7fdf060d4640 1 --2- 172.21.15.98:0/1114999913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf0010de50 0x7fdf001a57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.420+0000 7fdf060d4640 1 --2- 172.21.15.98:0/1114999913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf00075870 0x7fdf0019e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.421+0000 7fdf060d4640 1 --2- 172.21.15.98:0/1114999913 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdeec0a2b90 0x7fdeec0a5050 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.421+0000 7fdf060d4640 1 --2- 172.21.15.98:0/1114999913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf00074290 0x7fdf0019e1f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.421+0000 7fdf060d4640 1 -- 172.21.15.98:0/1114999913 >> 172.21.15.98:0/1114999913 conn(0x7fdf000fce10 msgr2=0x7fdf001011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:38.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.421+0000 7fdf060d4640 1 -- 172.21.15.98:0/1114999913 shutdown_connections 2024-05-09T08:47:38.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.421+0000 7fdf060d4640 1 -- 172.21.15.98:0/1114999913 wait complete. 2024-05-09T08:47:38.433 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:43:39.424443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T08:47:38.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:47:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.538+0000 7f67b0e38640 1 -- 172.21.15.98:0/2814513189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac100770 msgr2=0x7f67ac100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.538+0000 7f67b0e38640 1 --2- 172.21.15.98:0/2814513189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac100770 0x7f67ac100b90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f679c009900 tx=0x7f679c02a410 comp rx=0 tx=0).stop 2024-05-09T08:47:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.539+0000 7f67b0e38640 1 -- 172.21.15.98:0/2814513189 shutdown_connections 2024-05-09T08:47:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.539+0000 7f67b0e38640 1 --2- 172.21.15.98:0/2814513189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67ac102320 0x7f67ac108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.539+0000 7f67b0e38640 1 --2- 172.21.15.98:0/2814513189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67ac101960 0x7f67ac101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.539+0000 7f67b0e38640 1 --2- 172.21.15.98:0/2814513189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac100770 0x7f67ac100b90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.539+0000 7f67b0e38640 1 -- 172.21.15.98:0/2814513189 >> 172.21.15.98:0/2814513189 conn(0x7f67ac0fbce0 msgr2=0x7f67ac0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.539+0000 7f67b0e38640 1 -- 172.21.15.98:0/2814513189 shutdown_connections 2024-05-09T08:47:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.539+0000 7f67b0e38640 1 -- 172.21.15.98:0/2814513189 wait complete. 2024-05-09T08:47:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.539+0000 7f67b0e38640 1 Processor -- start 2024-05-09T08:47:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.539+0000 7f67b0e38640 1 -- start start 2024-05-09T08:47:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.539+0000 7f67b0e38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac100770 0x7f67ac069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.539+0000 7f67b0e38640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67ac101960 0x7f67ac19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.539+0000 7f67b0e38640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67ac102320 0x7f67ac19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.539+0000 7f67b0e38640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f67ac10c3c0 con 0x7f67ac100770 2024-05-09T08:47:38.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67b0e38640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f67ac10c240 con 0x7f67ac101960 2024-05-09T08:47:38.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67b0e38640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f67ac10c540 con 0x7f67ac102320 2024-05-09T08:47:38.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67aa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac100770 0x7f67ac069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67aa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac100770 0x7f67ac069530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46578/0 (socket says 172.21.15.98:46578) 2024-05-09T08:47:38.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67aa59c640 1 -- 172.21.15.98:0/499452206 learned_addr learned my addr 172.21.15.98:0/499452206 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:47:38.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67a3fff640 1 --2- 172.21.15.98:0/499452206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67ac101960 0x7f67ac19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67aa59c640 1 -- 172.21.15.98:0/499452206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67ac102320 msgr2=0x7f67ac19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67aa59c640 1 --2- 172.21.15.98:0/499452206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67ac102320 0x7f67ac19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67aa59c640 1 -- 172.21.15.98:0/499452206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67ac101960 msgr2=0x7f67ac19c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67aa59c640 1 --2- 172.21.15.98:0/499452206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67ac101960 0x7f67ac19c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67aa59c640 1 -- 172.21.15.98:0/499452206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67ac19f840 con 0x7f67ac100770 2024-05-09T08:47:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67aa59c640 1 --2- 172.21.15.98:0/499452206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac100770 0x7f67ac069530 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f679c031610 tx=0x7f679c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67a37fe640 1 -- 172.21.15.98:0/499452206 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679c03f030 con 0x7f67ac100770 2024-05-09T08:47:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67a37fe640 1 -- 172.21.15.98:0/499452206 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f679c003ad0 con 0x7f67ac100770 2024-05-09T08:47:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67b0e38640 1 -- 172.21.15.98:0/499452206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67ac19fa70 con 0x7f67ac100770 2024-05-09T08:47:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67a37fe640 1 -- 172.21.15.98:0/499452206 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679c035070 con 0x7f67ac100770 2024-05-09T08:47:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.540+0000 7f67b0e38640 1 -- 172.21.15.98:0/499452206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67ac19ffb0 con 0x7f67ac100770 2024-05-09T08:47:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.541+0000 7f67b0e38640 1 -- 172.21.15.98:0/499452206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67ac107b60 con 0x7f67ac100770 2024-05-09T08:47:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.542+0000 7f67a37fe640 1 -- 172.21.15.98:0/499452206 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f679c042030 con 0x7f67ac100770 2024-05-09T08:47:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.543+0000 7f67a37fe640 1 --2- 172.21.15.98:0/499452206 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f67980a2c10 0x7f67980a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.543+0000 7f67a37fe640 1 -- 172.21.15.98:0/499452206 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f679c0ef590 con 0x7f67ac100770 2024-05-09T08:47:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.543+0000 7f67a3fff640 1 --2- 172.21.15.98:0/499452206 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f67980a2c10 0x7f67980a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.543+0000 7f67a3fff640 1 --2- 172.21.15.98:0/499452206 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f67980a2c10 0x7f67980a50d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6794005fd0 tx=0x7f6794005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:38.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.546+0000 7f67a37fe640 1 -- 172.21.15.98:0/499452206 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f679c0bc990 con 0x7f67ac100770 2024-05-09T08:47:38.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.722+0000 7f67b0e38640 1 -- 172.21.15.98:0/499452206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f67ac02d1c0 con 0x7f67ac100770 2024-05-09T08:47:38.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.723+0000 7f67a37fe640 1 -- 172.21.15.98:0/499452206 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f679c0bc290 con 0x7f67ac100770 2024-05-09T08:47:38.722 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-09T08:47:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.724+0000 7f67b0e38640 1 -- 172.21.15.98:0/499452206 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f67980a2c10 msgr2=0x7f67980a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.724+0000 7f67b0e38640 1 --2- 172.21.15.98:0/499452206 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f67980a2c10 0x7f67980a50d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6794005fd0 tx=0x7f6794005850 comp rx=0 tx=0).stop 2024-05-09T08:47:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.724+0000 7f67b0e38640 1 -- 172.21.15.98:0/499452206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac100770 msgr2=0x7f67ac069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.725+0000 7f67b0e38640 1 --2- 172.21.15.98:0/499452206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac100770 0x7f67ac069530 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f679c031610 tx=0x7f679c004270 comp rx=0 tx=0).stop 2024-05-09T08:47:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.725+0000 7f67b0e38640 1 -- 172.21.15.98:0/499452206 shutdown_connections 2024-05-09T08:47:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.725+0000 7f67b0e38640 1 --2- 172.21.15.98:0/499452206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67ac102320 0x7f67ac19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.725+0000 7f67b0e38640 1 --2- 172.21.15.98:0/499452206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67ac101960 0x7f67ac19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.725+0000 7f67b0e38640 1 --2- 172.21.15.98:0/499452206 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f67980a2c10 0x7f67980a50d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.725+0000 7f67b0e38640 1 --2- 172.21.15.98:0/499452206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac100770 0x7f67ac069530 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.725+0000 7f67b0e38640 1 -- 172.21.15.98:0/499452206 >> 172.21.15.98:0/499452206 conn(0x7f67ac0fbce0 msgr2=0x7f67ac104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.725+0000 7f67b0e38640 1 -- 172.21.15.98:0/499452206 shutdown_connections 2024-05-09T08:47:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.725+0000 7f67b0e38640 1 -- 172.21.15.98:0/499452206 wait complete. 2024-05-09T08:47:38.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:47:38.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.837+0000 7f8bb3cba640 1 -- 172.21.15.98:0/2411955495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bac100770 msgr2=0x7f8bac100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.837+0000 7f8bb3cba640 1 --2- 172.21.15.98:0/2411955495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bac100770 0x7f8bac100b90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009900 tx=0x7f8ba802a410 comp rx=0 tx=0).stop 2024-05-09T08:47:38.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.837+0000 7f8bb3cba640 1 -- 172.21.15.98:0/2411955495 shutdown_connections 2024-05-09T08:47:38.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.837+0000 7f8bb3cba640 1 --2- 172.21.15.98:0/2411955495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bac102320 0x7f8bac108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.837+0000 7f8bb3cba640 1 --2- 172.21.15.98:0/2411955495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bac101960 0x7f8bac101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.837+0000 7f8bb3cba640 1 --2- 172.21.15.98:0/2411955495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bac100770 0x7f8bac100b90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.837+0000 7f8bb3cba640 1 -- 172.21.15.98:0/2411955495 >> 172.21.15.98:0/2411955495 conn(0x7f8bac0fbce0 msgr2=0x7f8bac0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.837+0000 7f8bb3cba640 1 -- 172.21.15.98:0/2411955495 shutdown_connections 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.838+0000 7f8bb3cba640 1 -- 172.21.15.98:0/2411955495 wait complete. 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.838+0000 7f8bb3cba640 1 Processor -- start 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.838+0000 7f8bb3cba640 1 -- start start 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.838+0000 7f8bb3cba640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bac100770 0x7f8bac069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.838+0000 7f8bb3cba640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bac101960 0x7f8bac19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.838+0000 7f8bb3cba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bac102320 0x7f8bac19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.838+0000 7f8bb3cba640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac10c3e0 con 0x7f8bac102320 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.838+0000 7f8bb3cba640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac10c260 con 0x7f8bac101960 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.838+0000 7f8bb3cba640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac10c560 con 0x7f8bac100770 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.838+0000 7f8bb2257640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bac102320 0x7f8bac19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.838+0000 7f8bb2257640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bac102320 0x7f8bac19ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46590/0 (socket says 172.21.15.98:46590) 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.838+0000 7f8bb2257640 1 -- 172.21.15.98:0/1401748236 learned_addr learned my addr 172.21.15.98:0/1401748236 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.838+0000 7f8bb1a56640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bac100770 0x7f8bac069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.839+0000 7f8bb1255640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bac101960 0x7f8bac19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.839+0000 7f8bb2257640 1 -- 172.21.15.98:0/1401748236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bac100770 msgr2=0x7f8bac069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.839+0000 7f8bb2257640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bac100770 0x7f8bac069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.839+0000 7f8bb2257640 1 -- 172.21.15.98:0/1401748236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bac101960 msgr2=0x7f8bac19c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.839+0000 7f8bb2257640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bac101960 0x7f8bac19c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.839+0000 7f8bb2257640 1 -- 172.21.15.98:0/1401748236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bac19f890 con 0x7f8bac102320 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.839+0000 7f8bb1255640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bac101960 0x7f8bac19c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.839+0000 7f8bb2257640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bac102320 0x7f8bac19ef80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400cad0 tx=0x7f8ba400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.839+0000 7f8ba2ffd640 1 -- 172.21.15.98:0/1401748236 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba400ae20 con 0x7f8bac102320 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.839+0000 7f8bb3cba640 1 -- 172.21.15.98:0/1401748236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bac19fb20 con 0x7f8bac102320 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.839+0000 7f8ba2ffd640 1 -- 172.21.15.98:0/1401748236 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ba4005020 con 0x7f8bac102320 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.839+0000 7f8ba2ffd640 1 -- 172.21.15.98:0/1401748236 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba400f6f0 con 0x7f8bac102320 2024-05-09T08:47:38.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.839+0000 7f8bb3cba640 1 -- 172.21.15.98:0/1401748236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bac1a0020 con 0x7f8bac102320 2024-05-09T08:47:38.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.840+0000 7f8bb3cba640 1 -- 172.21.15.98:0/1401748236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bac107b80 con 0x7f8bac102320 2024-05-09T08:47:38.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.842+0000 7f8ba2ffd640 1 -- 172.21.15.98:0/1401748236 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8ba40051d0 con 0x7f8bac102320 2024-05-09T08:47:38.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.842+0000 7f8ba2ffd640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8b980a2b90 0x7f8b980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.842+0000 7f8bb1a56640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8b980a2b90 0x7f8b980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.842+0000 7f8ba2ffd640 1 -- 172.21.15.98:0/1401748236 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f8ba40cef80 con 0x7f8bac102320 2024-05-09T08:47:38.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.843+0000 7f8bb1a56640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8b980a2b90 0x7f8b980a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8005ab0 tx=0x7f8ba8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:38.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:38.846+0000 7f8ba2ffd640 1 -- 172.21.15.98:0/1401748236 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8ba4014030 con 0x7f8bac102320 2024-05-09T08:47:39.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:39.058+0000 7f8bb3cba640 1 -- 172.21.15.98:0/1401748236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bac02d1c0 con 0x7f8bac102320 2024-05-09T08:47:39.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:39.058+0000 7f8ba2ffd640 1 -- 172.21.15.98:0/1401748236 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f8ba409c380 con 0x7f8bac102320 2024-05-09T08:47:39.057 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:47:39.058 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-05-09T08:47:39.058 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 1 2024-05-09T08:47:39.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:39.060+0000 7f8bb3cba640 1 -- 172.21.15.98:0/1401748236 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8b980a2b90 msgr2=0x7f8b980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:39.060+0000 7f8bb3cba640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8b980a2b90 0x7f8b980a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8005ab0 tx=0x7f8ba8000bc0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:39.060+0000 7f8bb3cba640 1 -- 172.21.15.98:0/1401748236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bac102320 msgr2=0x7f8bac19ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:39.060+0000 7f8bb3cba640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bac102320 0x7f8bac19ef80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400cad0 tx=0x7f8ba400cf90 comp rx=0 tx=0).stop 2024-05-09T08:47:39.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:39.060+0000 7f8bb3cba640 1 -- 172.21.15.98:0/1401748236 shutdown_connections 2024-05-09T08:47:39.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:39.060+0000 7f8bb3cba640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8b980a2b90 0x7f8b980a5050 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:39.060+0000 7f8bb3cba640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bac102320 0x7f8bac19ef80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:39.060+0000 7f8bb3cba640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bac101960 0x7f8bac19c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:39.060+0000 7f8bb3cba640 1 --2- 172.21.15.98:0/1401748236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bac100770 0x7f8bac069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:39.060+0000 7f8bb3cba640 1 -- 172.21.15.98:0/1401748236 >> 172.21.15.98:0/1401748236 conn(0x7f8bac0fbce0 msgr2=0x7f8bac104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:39.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:39.060+0000 7f8bb3cba640 1 -- 172.21.15.98:0/1401748236 shutdown_connections 2024-05-09T08:47:39.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:39.060+0000 7f8bb3cba640 1 -- 172.21.15.98:0/1401748236 wait complete. 2024-05-09T08:47:39.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:39 smithi098 ceph-mon[30092]: Deploying daemon mds.l on smithi142 2024-05-09T08:47:39.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:39 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1114999913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:47:39.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:39 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/499452206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:47:39.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:39 smithi142 ceph-mon[34625]: Deploying daemon mds.l on smithi142 2024-05-09T08:47:39.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:39 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1114999913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:47:39.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:39 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/499452206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:47:39.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:39 smithi105 ceph-mon[34498]: Deploying daemon mds.l on smithi142 2024-05-09T08:47:39.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:39 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1114999913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:47:39.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:39 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/499452206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:47:40.009 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 08:47:39 smithi142 systemd[1]: Starting Ceph mds.l for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:47:40.273 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:40 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1401748236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:40.273 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:40 smithi142 ceph-mon[34625]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:40.273 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 08:47:40 smithi142 podman[63953]: 2024-05-09 08:47:40.052167875 +0000 UTC m=+0.017953096 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:47:40.273 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 08:47:40 smithi142 podman[63953]: 2024-05-09 08:47:40.165517021 +0000 UTC m=+0.131302243 container create 7a77746dd9b0e33991462528211ed3fbb9e329e638d1fb05669721c25ca0ae89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:47:40.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:40 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1401748236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:40.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:40 smithi098 ceph-mon[30092]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:40.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:40 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1401748236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:40.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:40 smithi105 ceph-mon[34498]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:40.658 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 08:47:40 smithi142 podman[63953]: 2024-05-09 08:47:40.36817581 +0000 UTC m=+0.333961032 container init 7a77746dd9b0e33991462528211ed3fbb9e329e638d1fb05669721c25ca0ae89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2) 2024-05-09T08:47:40.659 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 08:47:40 smithi142 podman[63953]: 2024-05-09 08:47:40.370661487 +0000 UTC m=+0.336446710 container start 7a77746dd9b0e33991462528211ed3fbb9e329e638d1fb05669721c25ca0ae89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-l, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507) 2024-05-09T08:47:40.659 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 08:47:40 smithi142 ceph-mds[63972]: -- 172.21.15.142:0/2524961742 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556103423a40 con 0x5561033f6800 2024-05-09T08:47:40.659 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 08:47:40 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-l[63968]: starting mds.l at 2024-05-09T08:47:40.659 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 08:47:40 smithi142 bash[63953]: 7a77746dd9b0e33991462528211ed3fbb9e329e638d1fb05669721c25ca0ae89 2024-05-09T08:47:40.659 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 08:47:40 smithi142 systemd[1]: Started Ceph mds.l for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:47:41.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:47:41.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.184+0000 7ff6b36fc640 1 -- 172.21.15.98:0/4178288165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ac102330 msgr2=0x7ff6ac108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:41.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.184+0000 7ff6b36fc640 1 --2- 172.21.15.98:0/4178288165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ac102330 0x7ff6ac108bc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4009ca0 tx=0x7ff6a402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:47:41.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.184+0000 7ff6b36fc640 1 -- 172.21.15.98:0/4178288165 shutdown_connections 2024-05-09T08:47:41.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.184+0000 7ff6b36fc640 1 --2- 172.21.15.98:0/4178288165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ac102330 0x7ff6ac108bc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:41.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.184+0000 7ff6b36fc640 1 --2- 172.21.15.98:0/4178288165 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6ac101970 0x7ff6ac101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:41.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.184+0000 7ff6b36fc640 1 --2- 172.21.15.98:0/4178288165 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6ac100780 0x7ff6ac100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.184+0000 7ff6b36fc640 1 -- 172.21.15.98:0/4178288165 >> 172.21.15.98:0/4178288165 conn(0x7ff6ac0fbd10 msgr2=0x7ff6ac0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.184+0000 7ff6b36fc640 1 -- 172.21.15.98:0/4178288165 shutdown_connections 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.184+0000 7ff6b36fc640 1 -- 172.21.15.98:0/4178288165 wait complete. 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b36fc640 1 Processor -- start 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b36fc640 1 -- start start 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b36fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6ac100780 0x7ff6ac197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b36fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6ac101970 0x7ff6ac1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b36fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ac102330 0x7ff6ac19f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b36fc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac10d580 con 0x7ff6ac102330 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b36fc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac10d400 con 0x7ff6ac100780 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b36fc640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac10d700 con 0x7ff6ac101970 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b1c99640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ac102330 0x7ff6ac19f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b1c99640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ac102330 0x7ff6ac19f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46612/0 (socket says 172.21.15.98:46612) 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b1c99640 1 -- 172.21.15.98:0/1972618939 learned_addr learned my addr 172.21.15.98:0/1972618939 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:47:41.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b1498640 1 --2- 172.21.15.98:0/1972618939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6ac100780 0x7ff6ac197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:41.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b0c97640 1 --2- 172.21.15.98:0/1972618939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6ac101970 0x7ff6ac1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:41.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b1c99640 1 -- 172.21.15.98:0/1972618939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6ac101970 msgr2=0x7ff6ac1980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:41.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b1c99640 1 --2- 172.21.15.98:0/1972618939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6ac101970 0x7ff6ac1980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:41.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b1c99640 1 -- 172.21.15.98:0/1972618939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6ac100780 msgr2=0x7ff6ac197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:41.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.185+0000 7ff6b1c99640 1 --2- 172.21.15.98:0/1972618939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6ac100780 0x7ff6ac197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:41.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.186+0000 7ff6b1c99640 1 -- 172.21.15.98:0/1972618939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ac19f7b0 con 0x7ff6ac102330 2024-05-09T08:47:41.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.186+0000 7ff6b1c99640 1 --2- 172.21.15.98:0/1972618939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ac102330 0x7ff6ac19f140 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff6a40098d0 tx=0x7ff6a402f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:41.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.186+0000 7ff6a27fc640 1 -- 172.21.15.98:0/1972618939 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a402fc30 con 0x7ff6ac102330 2024-05-09T08:47:41.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.186+0000 7ff6b36fc640 1 -- 172.21.15.98:0/1972618939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6ac19fa40 con 0x7ff6ac102330 2024-05-09T08:47:41.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.186+0000 7ff6a27fc640 1 -- 172.21.15.98:0/1972618939 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6a4003e50 con 0x7ff6ac102330 2024-05-09T08:47:41.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.186+0000 7ff6a27fc640 1 -- 172.21.15.98:0/1972618939 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4004810 con 0x7ff6ac102330 2024-05-09T08:47:41.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.186+0000 7ff6b36fc640 1 -- 172.21.15.98:0/1972618939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6ac19ff90 con 0x7ff6ac102330 2024-05-09T08:47:41.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.187+0000 7ff6b36fc640 1 -- 172.21.15.98:0/1972618939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6ac107aa0 con 0x7ff6ac102330 2024-05-09T08:47:41.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.188+0000 7ff6a27fc640 1 -- 172.21.15.98:0/1972618939 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff6a4003970 con 0x7ff6ac102330 2024-05-09T08:47:41.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.189+0000 7ff6a27fc640 1 --2- 172.21.15.98:0/1972618939 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6980a2b90 0x7ff6980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:41.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.189+0000 7ff6b1498640 1 --2- 172.21.15.98:0/1972618939 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6980a2b90 0x7ff6980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:41.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.189+0000 7ff6a27fc640 1 -- 172.21.15.98:0/1972618939 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff6a40eff70 con 0x7ff6ac102330 2024-05-09T08:47:41.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.189+0000 7ff6b1498640 1 --2- 172.21.15.98:0/1972618939 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6980a2b90 0x7ff6980a5050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac1017d0 tx=0x7ff69c009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:41.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.193+0000 7ff6a27fc640 1 -- 172.21.15.98:0/1972618939 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff6a40bd370 con 0x7ff6ac102330 2024-05-09T08:47:41.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.405+0000 7ff6b36fc640 1 -- 172.21.15.98:0/1972618939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6ac02d3e0 con 0x7ff6ac102330 2024-05-09T08:47:41.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.405+0000 7ff6a27fc640 1 -- 172.21.15.98:0/1972618939 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7ff6a40bcc70 con 0x7ff6ac102330 2024-05-09T08:47:41.404 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:47:41.405 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T08:47:41.405 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 2 2024-05-09T08:47:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.407+0000 7ff6b36fc640 1 -- 172.21.15.98:0/1972618939 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6980a2b90 msgr2=0x7ff6980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.407+0000 7ff6b36fc640 1 --2- 172.21.15.98:0/1972618939 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6980a2b90 0x7ff6980a5050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac1017d0 tx=0x7ff69c009110 comp rx=0 tx=0).stop 2024-05-09T08:47:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.407+0000 7ff6b36fc640 1 -- 172.21.15.98:0/1972618939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ac102330 msgr2=0x7ff6ac19f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.407+0000 7ff6b36fc640 1 --2- 172.21.15.98:0/1972618939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ac102330 0x7ff6ac19f140 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff6a40098d0 tx=0x7ff6a402f840 comp rx=0 tx=0).stop 2024-05-09T08:47:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.407+0000 7ff6b36fc640 1 -- 172.21.15.98:0/1972618939 shutdown_connections 2024-05-09T08:47:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.407+0000 7ff6b36fc640 1 --2- 172.21.15.98:0/1972618939 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6980a2b90 0x7ff6980a5050 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.407+0000 7ff6b36fc640 1 --2- 172.21.15.98:0/1972618939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ac102330 0x7ff6ac19f140 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.407+0000 7ff6b36fc640 1 --2- 172.21.15.98:0/1972618939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6ac101970 0x7ff6ac1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.407+0000 7ff6b36fc640 1 --2- 172.21.15.98:0/1972618939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6ac100780 0x7ff6ac197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.407+0000 7ff6b36fc640 1 -- 172.21.15.98:0/1972618939 >> 172.21.15.98:0/1972618939 conn(0x7ff6ac0fbd10 msgr2=0x7ff6ac104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.407+0000 7ff6b36fc640 1 -- 172.21.15.98:0/1972618939 shutdown_connections 2024-05-09T08:47:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:41.407+0000 7ff6b36fc640 1 -- 172.21.15.98:0/1972618939 wait complete. 2024-05-09T08:47:41.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:41 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] up:boot 2024-05-09T08:47:41.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:41 smithi105 ceph-mon[34498]: fsmap 1 up:standby 2024-05-09T08:47:41.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:47:41.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:41.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:41.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:41.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:41.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:41.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:41 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:41.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:41 smithi105 ceph-mon[34498]: Deploying daemon mds.k on smithi105 2024-05-09T08:47:41.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:41 smithi105 ceph-mon[34498]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:41 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] up:boot 2024-05-09T08:47:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:41 smithi142 ceph-mon[34625]: fsmap 1 up:standby 2024-05-09T08:47:41.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:47:41.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:41.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:41.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:41.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:41.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:41.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:41 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:41.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:41 smithi142 ceph-mon[34625]: Deploying daemon mds.k on smithi105 2024-05-09T08:47:41.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:41 smithi142 ceph-mon[34625]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:41.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:41 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] up:boot 2024-05-09T08:47:41.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:41 smithi098 ceph-mon[30092]: fsmap 1 up:standby 2024-05-09T08:47:41.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:47:41.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:41.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:41.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:41.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:41.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:41.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:41 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:41.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:41 smithi098 ceph-mon[30092]: Deploying daemon mds.k on smithi105 2024-05-09T08:47:41.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:41 smithi098 ceph-mon[30092]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:42.577 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:42 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1972618939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:42.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:42 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1972618939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:42.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:42 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1972618939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:42.977 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 08:47:42 smithi105 systemd[1]: Starting Ceph mds.k for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:47:43.237 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 08:47:42 smithi105 podman[63881]: 2024-05-09 08:47:42.880979329 +0000 UTC m=+0.018548291 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:47:43.237 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 08:47:42 smithi105 podman[63881]: 2024-05-09 08:47:42.996373696 +0000 UTC m=+0.133942658 container create 39d1fef78d2833387f20f9d7a11b9f7c77b33c68f6d3274c7d303167b83489b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507) 2024-05-09T08:47:43.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:47:43.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.535+0000 7f797e295640 1 -- 172.21.15.98:0/2049710238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79780ffb50 msgr2=0x7f79780fff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:43.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.535+0000 7f797e295640 1 --2- 172.21.15.98:0/2049710238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79780ffb50 0x7f79780fff70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7968009930 tx=0x7f796802a420 comp rx=0 tx=0).stop 2024-05-09T08:47:43.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.535+0000 7f797e295640 1 -- 172.21.15.98:0/2049710238 shutdown_connections 2024-05-09T08:47:43.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.535+0000 7f797e295640 1 --2- 172.21.15.98:0/2049710238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79781034e0 0x7f7978109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:43.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.535+0000 7f797e295640 1 --2- 172.21.15.98:0/2049710238 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7978102b20 0x7f7978102fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:43.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.535+0000 7f797e295640 1 --2- 172.21.15.98:0/2049710238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79780ffb50 0x7f79780fff70 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:43.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.535+0000 7f797e295640 1 -- 172.21.15.98:0/2049710238 >> 172.21.15.98:0/2049710238 conn(0x7f79780fb4a0 msgr2=0x7f79780fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:43.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.535+0000 7f797e295640 1 -- 172.21.15.98:0/2049710238 shutdown_connections 2024-05-09T08:47:43.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.535+0000 7f797e295640 1 -- 172.21.15.98:0/2049710238 wait complete. 2024-05-09T08:47:43.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.536+0000 7f797e295640 1 Processor -- start 2024-05-09T08:47:43.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.536+0000 7f797e295640 1 -- start start 2024-05-09T08:47:43.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.536+0000 7f797e295640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79780ffb50 0x7f7978199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:43.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.536+0000 7f797e295640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7978102b20 0x7f797819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:43.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.536+0000 7f797e295640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79781034e0 0x7f79781a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:43.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.536+0000 7f797e295640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f797810d520 con 0x7f7978102b20 2024-05-09T08:47:43.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.536+0000 7f797e295640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f797810d3a0 con 0x7f79781034e0 2024-05-09T08:47:43.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.536+0000 7f797e295640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f797810d6a0 con 0x7f79780ffb50 2024-05-09T08:47:43.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.536+0000 7f79777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7978102b20 0x7f797819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:43.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.536+0000 7f79777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7978102b20 0x7f797819a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46626/0 (socket says 172.21.15.98:46626) 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.536+0000 7f79777fe640 1 -- 172.21.15.98:0/1076342391 learned_addr learned my addr 172.21.15.98:0/1076342391 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.536+0000 7f797c832640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79781034e0 0x7f79781a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.536+0000 7f7977fff640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79780ffb50 0x7f7978199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.537+0000 7f79777fe640 1 -- 172.21.15.98:0/1076342391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79780ffb50 msgr2=0x7f7978199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.537+0000 7f79777fe640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79780ffb50 0x7f7978199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.537+0000 7f79777fe640 1 -- 172.21.15.98:0/1076342391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79781034e0 msgr2=0x7f79781a13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.537+0000 7f79777fe640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79781034e0 0x7f79781a13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.537+0000 7f79777fe640 1 -- 172.21.15.98:0/1076342391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79781a1a20 con 0x7f7978102b20 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.537+0000 7f797c832640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79781034e0 0x7f79781a13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.537+0000 7f79777fe640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7978102b20 0x7f797819a340 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f796000e9f0 tx=0x7f796000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.537+0000 7f79757fa640 1 -- 172.21.15.98:0/1076342391 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796000cd80 con 0x7f7978102b20 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.537+0000 7f79757fa640 1 -- 172.21.15.98:0/1076342391 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7960010910 con 0x7f7978102b20 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.537+0000 7f79757fa640 1 -- 172.21.15.98:0/1076342391 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7960018990 con 0x7f7978102b20 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.537+0000 7f797e295640 1 -- 172.21.15.98:0/1076342391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79781a1d10 con 0x7f7978102b20 2024-05-09T08:47:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.537+0000 7f797e295640 1 -- 172.21.15.98:0/1076342391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79781a2200 con 0x7f7978102b20 2024-05-09T08:47:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.538+0000 7f797e295640 1 -- 172.21.15.98:0/1076342391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79781a2430 con 0x7f7978102b20 2024-05-09T08:47:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.539+0000 7f79757fa640 1 -- 172.21.15.98:0/1076342391 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7960010ac0 con 0x7f7978102b20 2024-05-09T08:47:43.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.540+0000 7f79757fa640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f79640a2b90 0x7f79640a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:43.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.540+0000 7f7977fff640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f79640a2b90 0x7f79640a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:43.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.540+0000 7f79757fa640 1 -- 172.21.15.98:0/1076342391 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f79600cf280 con 0x7f7978102b20 2024-05-09T08:47:43.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.541+0000 7f7977fff640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f79640a2b90 0x7f79640a5050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7968031440 tx=0x7f7968000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:43.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.544+0000 7f79757fa640 1 -- 172.21.15.98:0/1076342391 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f796009c030 con 0x7f7978102b20 2024-05-09T08:47:43.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:43 smithi105 ceph-mon[34498]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:43.737 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 08:47:43 smithi105 podman[63881]: 2024-05-09 08:47:43.274003587 +0000 UTC m=+0.411572549 container init 39d1fef78d2833387f20f9d7a11b9f7c77b33c68f6d3274c7d303167b83489b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-05-09T08:47:43.737 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 08:47:43 smithi105 podman[63881]: 2024-05-09 08:47:43.276482335 +0000 UTC m=+0.414051300 container start 39d1fef78d2833387f20f9d7a11b9f7c77b33c68f6d3274c7d303167b83489b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-k, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_CLEAN=True) 2024-05-09T08:47:43.737 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 08:47:43 smithi105 ceph-mds[63900]: -- 172.21.15.105:0/957609310 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c3873fba40 con 0x55c3873ce800 2024-05-09T08:47:43.737 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 08:47:43 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-k[63896]: starting mds.k at 2024-05-09T08:47:43.737 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 08:47:43 smithi105 bash[63881]: 39d1fef78d2833387f20f9d7a11b9f7c77b33c68f6d3274c7d303167b83489b5 2024-05-09T08:47:43.737 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 08:47:43 smithi105 systemd[1]: Started Ceph mds.k for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:47:43.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.766+0000 7f797e295640 1 -- 172.21.15.98:0/1076342391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79781a2600 con 0x7f7978102b20 2024-05-09T08:47:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.767+0000 7f79757fa640 1 -- 172.21.15.98:0/1076342391 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f79600a0330 con 0x7f7978102b20 2024-05-09T08:47:43.766 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:47:43.766 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-09T08:47:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 3 2024-05-09T08:47:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.769+0000 7f797e295640 1 -- 172.21.15.98:0/1076342391 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f79640a2b90 msgr2=0x7f79640a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.769+0000 7f797e295640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f79640a2b90 0x7f79640a5050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7968031440 tx=0x7f7968000bc0 comp rx=0 tx=0).stop 2024-05-09T08:47:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.769+0000 7f797e295640 1 -- 172.21.15.98:0/1076342391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7978102b20 msgr2=0x7f797819a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.769+0000 7f797e295640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7978102b20 0x7f797819a340 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f796000e9f0 tx=0x7f796000eeb0 comp rx=0 tx=0).stop 2024-05-09T08:47:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.769+0000 7f797e295640 1 -- 172.21.15.98:0/1076342391 shutdown_connections 2024-05-09T08:47:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.769+0000 7f797e295640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79781034e0 0x7f79781a13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.769+0000 7f797e295640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f79640a2b90 0x7f79640a5050 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.769+0000 7f797e295640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7978102b20 0x7f797819a340 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.769+0000 7f797e295640 1 --2- 172.21.15.98:0/1076342391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79780ffb50 0x7f7978199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.769+0000 7f797e295640 1 -- 172.21.15.98:0/1076342391 >> 172.21.15.98:0/1076342391 conn(0x7f79780fb4a0 msgr2=0x7f7978100910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.769+0000 7f797e295640 1 -- 172.21.15.98:0/1076342391 shutdown_connections 2024-05-09T08:47:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:43.769+0000 7f797e295640 1 -- 172.21.15.98:0/1076342391 wait complete. 2024-05-09T08:47:43.778 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:43 smithi098 ceph-mon[30092]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:43.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:43 smithi142 ceph-mon[34625]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:44.675 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:44 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:44.675 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:44 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.105:6832/1472919356,v1:172.21.15.105:6833/1472919356] up:boot 2024-05-09T08:47:44.675 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:44 smithi105 ceph-mon[34498]: fsmap 2 up:standby 2024-05-09T08:47:44.675 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:44 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:47:44.675 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:44 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:44.675 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:44 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:44.675 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:44 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:44.675 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:44 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:44.676 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:44 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:44.676 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:44 smithi105 ceph-mon[34498]: Deploying daemon mds.j on smithi098 2024-05-09T08:47:44.676 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:44 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1076342391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:44 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:44 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.105:6832/1472919356,v1:172.21.15.105:6833/1472919356] up:boot 2024-05-09T08:47:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:44 smithi098 ceph-mon[30092]: fsmap 2 up:standby 2024-05-09T08:47:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:44 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:47:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:44 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:44 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:44 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:44.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:44 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:44.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:44 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:44.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:44 smithi098 ceph-mon[30092]: Deploying daemon mds.j on smithi098 2024-05-09T08:47:44.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:44 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1076342391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:44.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:44 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:44.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:44 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.105:6832/1472919356,v1:172.21.15.105:6833/1472919356] up:boot 2024-05-09T08:47:44.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:44 smithi142 ceph-mon[34625]: fsmap 2 up:standby 2024-05-09T08:47:44.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:44 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:47:44.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:44 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:44.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:44 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:44.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:44 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:44.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:44 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:44.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:44 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:44.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:44 smithi142 ceph-mon[34625]: Deploying daemon mds.j on smithi098 2024-05-09T08:47:44.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:44 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1076342391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:45.695 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:45 smithi098 ceph-mon[30092]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:45.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:45 smithi105 ceph-mon[34498]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:45.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:47:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.896+0000 7f21db3a5640 1 -- 172.21.15.98:0/4267424275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d4100770 msgr2=0x7f21d4100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.896+0000 7f21db3a5640 1 --2- 172.21.15.98:0/4267424275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d4100770 0x7f21d4100b90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f21c4009900 tx=0x7f21c402a410 comp rx=0 tx=0).stop 2024-05-09T08:47:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.896+0000 7f21db3a5640 1 -- 172.21.15.98:0/4267424275 shutdown_connections 2024-05-09T08:47:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.896+0000 7f21db3a5640 1 --2- 172.21.15.98:0/4267424275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21d4102320 0x7f21d4108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.896+0000 7f21db3a5640 1 --2- 172.21.15.98:0/4267424275 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21d4101960 0x7f21d4101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.896+0000 7f21db3a5640 1 --2- 172.21.15.98:0/4267424275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d4100770 0x7f21d4100b90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.896+0000 7f21db3a5640 1 -- 172.21.15.98:0/4267424275 >> 172.21.15.98:0/4267424275 conn(0x7f21d40fbce0 msgr2=0x7f21d40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.896+0000 7f21db3a5640 1 -- 172.21.15.98:0/4267424275 shutdown_connections 2024-05-09T08:47:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.896+0000 7f21db3a5640 1 -- 172.21.15.98:0/4267424275 wait complete. 2024-05-09T08:47:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.896+0000 7f21db3a5640 1 Processor -- start 2024-05-09T08:47:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.896+0000 7f21db3a5640 1 -- start start 2024-05-09T08:47:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21db3a5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21d4100770 0x7f21d4197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21db3a5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21d4101960 0x7f21d41980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21db3a5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d4102320 0x7f21d419f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21db3a5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21d410d4d0 con 0x7f21d4102320 2024-05-09T08:47:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21db3a5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f21d410d350 con 0x7f21d4100770 2024-05-09T08:47:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21db3a5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f21d410d650 con 0x7f21d4101960 2024-05-09T08:47:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21d9942640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d4102320 0x7f21d419f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21d9942640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d4102320 0x7f21d419f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54658/0 (socket says 172.21.15.98:54658) 2024-05-09T08:47:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21d9942640 1 -- 172.21.15.98:0/2500950707 learned_addr learned my addr 172.21.15.98:0/2500950707 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:47:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21d9141640 1 --2- 172.21.15.98:0/2500950707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21d4100770 0x7f21d4197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21d9942640 1 -- 172.21.15.98:0/2500950707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21d4101960 msgr2=0x7f21d41980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21d9942640 1 --2- 172.21.15.98:0/2500950707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21d4101960 0x7f21d41980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21d9942640 1 -- 172.21.15.98:0/2500950707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21d4100770 msgr2=0x7f21d4197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21d9942640 1 --2- 172.21.15.98:0/2500950707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21d4100770 0x7f21d4197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21d9942640 1 -- 172.21.15.98:0/2500950707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d419f7c0 con 0x7f21d4102320 2024-05-09T08:47:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21d9942640 1 --2- 172.21.15.98:0/2500950707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d4102320 0x7f21d419f150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f21cc00d750 tx=0x7f21cc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21ca7fc640 1 -- 172.21.15.98:0/2500950707 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc017070 con 0x7f21d4102320 2024-05-09T08:47:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21ca7fc640 1 -- 172.21.15.98:0/2500950707 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21cc005500 con 0x7f21d4102320 2024-05-09T08:47:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21db3a5640 1 -- 172.21.15.98:0/2500950707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d419fab0 con 0x7f21d4102320 2024-05-09T08:47:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21ca7fc640 1 -- 172.21.15.98:0/2500950707 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc004d90 con 0x7f21d4102320 2024-05-09T08:47:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.897+0000 7f21db3a5640 1 -- 172.21.15.98:0/2500950707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d419ffb0 con 0x7f21d4102320 2024-05-09T08:47:45.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.898+0000 7f21db3a5640 1 -- 172.21.15.98:0/2500950707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21d4107ae0 con 0x7f21d4102320 2024-05-09T08:47:45.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.899+0000 7f21ca7fc640 1 -- 172.21.15.98:0/2500950707 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f21cc004f30 con 0x7f21d4102320 2024-05-09T08:47:45.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.900+0000 7f21ca7fc640 1 --2- 172.21.15.98:0/2500950707 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f21c00a2b90 0x7f21c00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:45.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.900+0000 7f21d9141640 1 --2- 172.21.15.98:0/2500950707 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f21c00a2b90 0x7f21c00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:45.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.900+0000 7f21ca7fc640 1 -- 172.21.15.98:0/2500950707 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f21cc0ce5e0 con 0x7f21d4102320 2024-05-09T08:47:45.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.901+0000 7f21d9141640 1 --2- 172.21.15.98:0/2500950707 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f21c00a2b90 0x7f21c00a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f21c4005ab0 tx=0x7f21c4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:45.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:45.905+0000 7f21ca7fc640 1 -- 172.21.15.98:0/2500950707 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f21cc09b9e0 con 0x7f21d4102320 2024-05-09T08:47:45.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:45 smithi142 ceph-mon[34625]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:45.973 INFO:journalctl@ceph.mds.j.smithi098.stdout:May 09 08:47:45 smithi098 podman[70634]: 2024-05-09 08:47:45.97405978 +0000 UTC m=+0.169064691 container create ecf3da05d375be0db8c2e9a74c98864bd3bbb51812e4eb0c938edbd1e43b8f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-j, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc) 2024-05-09T08:47:46.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:46.127+0000 7f21db3a5640 1 -- 172.21.15.98:0/2500950707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21d402d1c0 con 0x7f21d4102320 2024-05-09T08:47:46.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:46.128+0000 7f21ca7fc640 1 -- 172.21.15.98:0/2500950707 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f21cc09b2e0 con 0x7f21d4102320 2024-05-09T08:47:46.127 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:47:46.127 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-09T08:47:46.127 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 3 2024-05-09T08:47:46.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:46.129+0000 7f21db3a5640 1 -- 172.21.15.98:0/2500950707 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f21c00a2b90 msgr2=0x7f21c00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:46.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:46.129+0000 7f21db3a5640 1 --2- 172.21.15.98:0/2500950707 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f21c00a2b90 0x7f21c00a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f21c4005ab0 tx=0x7f21c4000bc0 comp rx=0 tx=0).stop 2024-05-09T08:47:46.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:46.129+0000 7f21db3a5640 1 -- 172.21.15.98:0/2500950707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d4102320 msgr2=0x7f21d419f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:46.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:46.130+0000 7f21db3a5640 1 --2- 172.21.15.98:0/2500950707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d4102320 0x7f21d419f150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f21cc00d750 tx=0x7f21cc00dc10 comp rx=0 tx=0).stop 2024-05-09T08:47:46.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:46.130+0000 7f21db3a5640 1 -- 172.21.15.98:0/2500950707 shutdown_connections 2024-05-09T08:47:46.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:46.130+0000 7f21db3a5640 1 --2- 172.21.15.98:0/2500950707 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f21c00a2b90 0x7f21c00a5050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:46.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:46.130+0000 7f21db3a5640 1 --2- 172.21.15.98:0/2500950707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d4102320 0x7f21d419f150 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:46.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:46.130+0000 7f21db3a5640 1 --2- 172.21.15.98:0/2500950707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21d4101960 0x7f21d41980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:46.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:46.130+0000 7f21db3a5640 1 --2- 172.21.15.98:0/2500950707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21d4100770 0x7f21d4197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:46.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:46.130+0000 7f21db3a5640 1 -- 172.21.15.98:0/2500950707 >> 172.21.15.98:0/2500950707 conn(0x7f21d40fbce0 msgr2=0x7f21d4104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:46.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:46.130+0000 7f21db3a5640 1 -- 172.21.15.98:0/2500950707 shutdown_connections 2024-05-09T08:47:46.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:46.130+0000 7f21db3a5640 1 -- 172.21.15.98:0/2500950707 wait complete. 2024-05-09T08:47:46.238 INFO:journalctl@ceph.mds.j.smithi098.stdout:May 09 08:47:46 smithi098 podman[70634]: 2024-05-09 08:47:46.210562141 +0000 UTC m=+0.405567063 container init ecf3da05d375be0db8c2e9a74c98864bd3bbb51812e4eb0c938edbd1e43b8f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-j, RELEASE=squid-96805dc, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:47:46.238 INFO:journalctl@ceph.mds.j.smithi098.stdout:May 09 08:47:46 smithi098 podman[70634]: 2024-05-09 08:47:46.214186001 +0000 UTC m=+0.409190910 container start ecf3da05d375be0db8c2e9a74c98864bd3bbb51812e4eb0c938edbd1e43b8f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-j, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:47:46.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:46 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2500950707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:46.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:46.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:46.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:46.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:46.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:46.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:46 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:46.589 INFO:journalctl@ceph.mds.j.smithi098.stdout:May 09 08:47:46 smithi098 ceph-mds[70678]: -- 172.21.15.98:0/3661316244 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559ac9391a40 con 0x559ac9365000 2024-05-09T08:47:46.589 INFO:journalctl@ceph.mds.j.smithi098.stdout:May 09 08:47:46 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-j[70671]: starting mds.j at 2024-05-09T08:47:46.590 INFO:journalctl@ceph.mds.j.smithi098.stdout:May 09 08:47:46 smithi098 bash[70634]: ecf3da05d375be0db8c2e9a74c98864bd3bbb51812e4eb0c938edbd1e43b8f20 2024-05-09T08:47:46.590 INFO:journalctl@ceph.mds.j.smithi098.stdout:May 09 08:47:46 smithi098 systemd[1]: Started Ceph mds.j for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:47:46.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:46 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2500950707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:46.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:46 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:46.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:46 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:46.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:46 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:46.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:46 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:46.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:46 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:46.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:46 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:46.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:46 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2500950707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:46.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:46 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:46.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:46 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:46.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:46 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:46.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:46 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:46.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:46 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:46.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:46 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:47.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:47 smithi098 ceph-mon[30092]: Deploying daemon mds.d on smithi098 2024-05-09T08:47:47.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:47 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.98:6834/3172071169,v1:172.21.15.98:6835/3172071169] up:boot 2024-05-09T08:47:47.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:47 smithi098 ceph-mon[30092]: fsmap 3 up:standby 2024-05-09T08:47:47.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:47 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:47:47.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:47 smithi098 ceph-mon[30092]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:47.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:47 smithi105 ceph-mon[34498]: Deploying daemon mds.d on smithi098 2024-05-09T08:47:47.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:47 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.98:6834/3172071169,v1:172.21.15.98:6835/3172071169] up:boot 2024-05-09T08:47:47.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:47 smithi105 ceph-mon[34498]: fsmap 3 up:standby 2024-05-09T08:47:47.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:47 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:47:47.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:47 smithi105 ceph-mon[34498]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:47.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:47 smithi142 ceph-mon[34625]: Deploying daemon mds.d on smithi098 2024-05-09T08:47:47.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:47 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.98:6834/3172071169,v1:172.21.15.98:6835/3172071169] up:boot 2024-05-09T08:47:47.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:47 smithi142 ceph-mon[34625]: fsmap 3 up:standby 2024-05-09T08:47:47.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:47 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:47:47.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:47 smithi142 ceph-mon[34625]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:48.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:47:48.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.265+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/191386992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b4101080 msgr2=0x7fa9b4101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:48.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.265+0000 7fa9bb8c3640 1 --2- 172.21.15.98:0/191386992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b4101080 0x7fa9b4101500 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0009900 tx=0x7fa9b002a410 comp rx=0 tx=0).stop 2024-05-09T08:47:48.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.265+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/191386992 shutdown_connections 2024-05-09T08:47:48.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.265+0000 7fa9bb8c3640 1 --2- 172.21.15.98:0/191386992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4101a40 0x7fa9b4108360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:48.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.265+0000 7fa9bb8c3640 1 --2- 172.21.15.98:0/191386992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b4101080 0x7fa9b4101500 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:48.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.265+0000 7fa9bb8c3640 1 --2- 172.21.15.98:0/191386992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9b40ffea0 0x7fa9b41002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:48.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.265+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/191386992 >> 172.21.15.98:0/191386992 conn(0x7fa9b40fb460 msgr2=0x7fa9b40fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:48.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.265+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/191386992 shutdown_connections 2024-05-09T08:47:48.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.265+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/191386992 wait complete. 2024-05-09T08:47:48.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.266+0000 7fa9bb8c3640 1 Processor -- start 2024-05-09T08:47:48.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.266+0000 7fa9bb8c3640 1 -- start start 2024-05-09T08:47:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.266+0000 7fa9bb8c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40ffea0 0x7fa9b41959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.266+0000 7fa9bb8c3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4101080 0x7fa9b4195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.266+0000 7fa9bb8c3640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9b4101a40 0x7fa9b419cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.266+0000 7fa9bb8c3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b410b360 con 0x7fa9b40ffea0 2024-05-09T08:47:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.266+0000 7fa9bb8c3640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b410b1e0 con 0x7fa9b4101080 2024-05-09T08:47:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.266+0000 7fa9bb8c3640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b410b4e0 con 0x7fa9b4101a40 2024-05-09T08:47:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.266+0000 7fa9b965f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40ffea0 0x7fa9b41959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:48.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.266+0000 7fa9b965f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40ffea0 0x7fa9b41959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54686/0 (socket says 172.21.15.98:54686) 2024-05-09T08:47:48.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.266+0000 7fa9b965f640 1 -- 172.21.15.98:0/2524632751 learned_addr learned my addr 172.21.15.98:0/2524632751 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:47:48.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.267+0000 7fa9b8e5e640 1 --2- 172.21.15.98:0/2524632751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4101080 0x7fa9b4195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:48.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.267+0000 7fa9b9e60640 1 --2- 172.21.15.98:0/2524632751 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9b4101a40 0x7fa9b419cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:48.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.267+0000 7fa9b965f640 1 -- 172.21.15.98:0/2524632751 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9b4101a40 msgr2=0x7fa9b419cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.267+0000 7fa9b965f640 1 --2- 172.21.15.98:0/2524632751 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9b4101a40 0x7fa9b419cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.267+0000 7fa9b965f640 1 -- 172.21.15.98:0/2524632751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4101080 msgr2=0x7fa9b4195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.267+0000 7fa9b965f640 1 --2- 172.21.15.98:0/2524632751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4101080 0x7fa9b4195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.267+0000 7fa9b965f640 1 -- 172.21.15.98:0/2524632751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b419f7d0 con 0x7fa9b40ffea0 2024-05-09T08:47:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.267+0000 7fa9b965f640 1 --2- 172.21.15.98:0/2524632751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40ffea0 0x7fa9b41959a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400e9c0 tx=0x7fa9a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.267+0000 7fa9aa7fc640 1 -- 172.21.15.98:0/2524632751 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a400cd80 con 0x7fa9b40ffea0 2024-05-09T08:47:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.267+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/2524632751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b419fac0 con 0x7fa9b40ffea0 2024-05-09T08:47:48.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.267+0000 7fa9aa7fc640 1 -- 172.21.15.98:0/2524632751 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9a40035f0 con 0x7fa9b40ffea0 2024-05-09T08:47:48.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.267+0000 7fa9aa7fc640 1 -- 172.21.15.98:0/2524632751 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4010690 con 0x7fa9b40ffea0 2024-05-09T08:47:48.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.267+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/2524632751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b419ff90 con 0x7fa9b40ffea0 2024-05-09T08:47:48.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.268+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/2524632751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b4067dc0 con 0x7fa9b40ffea0 2024-05-09T08:47:48.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.269+0000 7fa9aa7fc640 1 -- 172.21.15.98:0/2524632751 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa9a40037a0 con 0x7fa9b40ffea0 2024-05-09T08:47:48.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.270+0000 7fa9aa7fc640 1 --2- 172.21.15.98:0/2524632751 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa9a00a2be0 0x7fa9a00a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:48.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.270+0000 7fa9b8e5e640 1 --2- 172.21.15.98:0/2524632751 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa9a00a2be0 0x7fa9a00a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:48.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.270+0000 7fa9aa7fc640 1 -- 172.21.15.98:0/2524632751 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa9a40cf130 con 0x7fa9b40ffea0 2024-05-09T08:47:48.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.271+0000 7fa9b8e5e640 1 --2- 172.21.15.98:0/2524632751 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa9a00a2be0 0x7fa9a00a50a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0031440 tx=0x7fa9b0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:48.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.274+0000 7fa9aa7fc640 1 -- 172.21.15.98:0/2524632751 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa9a409c490 con 0x7fa9b40ffea0 2024-05-09T08:47:48.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.503+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/2524632751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9b41071e0 con 0x7fa9b40ffea0 2024-05-09T08:47:48.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.503+0000 7fa9aa7fc640 1 -- 172.21.15.98:0/2524632751 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7fa9a409bd90 con 0x7fa9b40ffea0 2024-05-09T08:47:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:47:48.503 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-09T08:47:48.503 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 4 2024-05-09T08:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.505+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/2524632751 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa9a00a2be0 msgr2=0x7fa9a00a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.505+0000 7fa9bb8c3640 1 --2- 172.21.15.98:0/2524632751 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa9a00a2be0 0x7fa9a00a50a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0031440 tx=0x7fa9b0000bc0 comp rx=0 tx=0).stop 2024-05-09T08:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.505+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/2524632751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40ffea0 msgr2=0x7fa9b41959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.505+0000 7fa9bb8c3640 1 --2- 172.21.15.98:0/2524632751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40ffea0 0x7fa9b41959a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400e9c0 tx=0x7fa9a400ee80 comp rx=0 tx=0).stop 2024-05-09T08:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.505+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/2524632751 shutdown_connections 2024-05-09T08:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.505+0000 7fa9bb8c3640 1 --2- 172.21.15.98:0/2524632751 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9b4101a40 0x7fa9b419cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.505+0000 7fa9bb8c3640 1 --2- 172.21.15.98:0/2524632751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4101080 0x7fa9b4195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.505+0000 7fa9bb8c3640 1 --2- 172.21.15.98:0/2524632751 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa9a00a2be0 0x7fa9a00a50a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.505+0000 7fa9bb8c3640 1 --2- 172.21.15.98:0/2524632751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b40ffea0 0x7fa9b41959a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.505+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/2524632751 >> 172.21.15.98:0/2524632751 conn(0x7fa9b40fb460 msgr2=0x7fa9b41043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:48.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.506+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/2524632751 shutdown_connections 2024-05-09T08:47:48.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:48.506+0000 7fa9bb8c3640 1 -- 172.21.15.98:0/2524632751 wait complete. 2024-05-09T08:47:48.807 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:48 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2524632751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:48.807 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 08:47:48 smithi098 systemd[1]: Starting Ceph mds.d for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:47:48.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:48 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2524632751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:48.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:48 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2524632751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:49.066 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 08:47:48 smithi098 podman[71005]: 2024-05-09 08:47:48.711186902 +0000 UTC m=+0.017792315 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:47:49.066 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 08:47:48 smithi098 podman[71005]: 2024-05-09 08:47:48.848232149 +0000 UTC m=+0.154837559 container create c7c51b9ae468a49a9f13d1b19d9d753d2a5c61739d4f9bab59428e0213a53352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-09T08:47:49.066 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 08:47:49 smithi098 podman[71005]: 2024-05-09 08:47:49.067562665 +0000 UTC m=+0.374168075 container init c7c51b9ae468a49a9f13d1b19d9d753d2a5c61739d4f9bab59428e0213a53352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, ceph=True, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS) 2024-05-09T08:47:49.338 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 08:47:49 smithi098 podman[71005]: 2024-05-09 08:47:49.070473571 +0000 UTC m=+0.377078982 container start c7c51b9ae468a49a9f13d1b19d9d753d2a5c61739d4f9bab59428e0213a53352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-09T08:47:49.338 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 08:47:49 smithi098 ceph-mds[71024]: -- 172.21.15.98:0/1048176816 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fa1fdc3a40 con 0x55fa1fd96800 2024-05-09T08:47:49.338 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 08:47:49 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d[71020]: starting mds.d at 2024-05-09T08:47:49.338 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 08:47:49 smithi098 bash[71005]: c7c51b9ae468a49a9f13d1b19d9d753d2a5c61739d4f9bab59428e0213a53352 2024-05-09T08:47:49.339 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 08:47:49 smithi098 systemd[1]: Started Ceph mds.d for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:47:50.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:50 smithi105 ceph-mon[34498]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:50 smithi105 ceph-mon[34498]: Deploying daemon mds.e on smithi105 2024-05-09T08:47:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:50 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.98:6836/3134006534,v1:172.21.15.98:6837/3134006534] up:boot 2024-05-09T08:47:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:50 smithi105 ceph-mon[34498]: fsmap 4 up:standby 2024-05-09T08:47:50.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:47:50.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:47:50.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:50 smithi098 ceph-mon[30092]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:50.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:50.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:50.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:50.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:50.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:50.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:50.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:50 smithi098 ceph-mon[30092]: Deploying daemon mds.e on smithi105 2024-05-09T08:47:50.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:50 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.98:6836/3134006534,v1:172.21.15.98:6837/3134006534] up:boot 2024-05-09T08:47:50.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:50 smithi098 ceph-mon[30092]: fsmap 4 up:standby 2024-05-09T08:47:50.543 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:47:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.623+0000 7f3174f12640 1 -- 172.21.15.98:0/205665534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3170101970 msgr2=0x7f3170101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.623+0000 7f3174f12640 1 --2- 172.21.15.98:0/205665534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3170101970 0x7f3170101df0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3158009900 tx=0x7f315802a410 comp rx=0 tx=0).stop 2024-05-09T08:47:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.623+0000 7f3174f12640 1 -- 172.21.15.98:0/205665534 shutdown_connections 2024-05-09T08:47:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.623+0000 7f3174f12640 1 --2- 172.21.15.98:0/205665534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3170102330 0x7f3170108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.623+0000 7f3174f12640 1 --2- 172.21.15.98:0/205665534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3170101970 0x7f3170101df0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.623+0000 7f3174f12640 1 --2- 172.21.15.98:0/205665534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3170100780 0x7f3170100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.623+0000 7f3174f12640 1 -- 172.21.15.98:0/205665534 >> 172.21.15.98:0/205665534 conn(0x7f31700fbd10 msgr2=0x7f31700fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.623+0000 7f3174f12640 1 -- 172.21.15.98:0/205665534 shutdown_connections 2024-05-09T08:47:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.623+0000 7f3174f12640 1 -- 172.21.15.98:0/205665534 wait complete. 2024-05-09T08:47:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.624+0000 7f3174f12640 1 Processor -- start 2024-05-09T08:47:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.624+0000 7f3174f12640 1 -- start start 2024-05-09T08:47:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.624+0000 7f3174f12640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3170100780 0x7f3170199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.624+0000 7f3174f12640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3170101970 0x7f317019a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.624+0000 7f3174f12640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3170102330 0x7f31701a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.624+0000 7f3174f12640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f317010d560 con 0x7f3170101970 2024-05-09T08:47:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.624+0000 7f3174f12640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f317010d3e0 con 0x7f3170102330 2024-05-09T08:47:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.624+0000 7f3174f12640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f317010d6e0 con 0x7f3170100780 2024-05-09T08:47:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.624+0000 7f3167fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3170101970 0x7f317019a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.624+0000 7f3167fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3170101970 0x7f317019a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54740/0 (socket says 172.21.15.98:54740) 2024-05-09T08:47:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.624+0000 7f3167fff640 1 -- 172.21.15.98:0/2753150620 learned_addr learned my addr 172.21.15.98:0/2753150620 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:47:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f316e59c640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3170100780 0x7f3170199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f316ed9d640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3170102330 0x7f31701a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f3167fff640 1 -- 172.21.15.98:0/2753150620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3170100780 msgr2=0x7f3170199e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f3167fff640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3170100780 0x7f3170199e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f3167fff640 1 -- 172.21.15.98:0/2753150620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3170102330 msgr2=0x7f31701a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f3167fff640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3170102330 0x7f31701a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:50.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f3167fff640 1 -- 172.21.15.98:0/2753150620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31701a1a40 con 0x7f3170101970 2024-05-09T08:47:50.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f316e59c640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3170100780 0x7f3170199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:47:50.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f316ed9d640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3170102330 0x7f31701a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:47:50.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f3167fff640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3170101970 0x7f317019a360 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3158009230 tx=0x7f3158004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:50.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f31677fe640 1 -- 172.21.15.98:0/2753150620 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315803f030 con 0x7f3170101970 2024-05-09T08:47:50.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f3174f12640 1 -- 172.21.15.98:0/2753150620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31701a1cd0 con 0x7f3170101970 2024-05-09T08:47:50.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f31677fe640 1 -- 172.21.15.98:0/2753150620 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3158003ce0 con 0x7f3170101970 2024-05-09T08:47:50.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f31677fe640 1 -- 172.21.15.98:0/2753150620 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3158035070 con 0x7f3170101970 2024-05-09T08:47:50.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.625+0000 7f3174f12640 1 -- 172.21.15.98:0/2753150620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31701a21c0 con 0x7f3170101970 2024-05-09T08:47:50.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.626+0000 7f3174f12640 1 -- 172.21.15.98:0/2753150620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31701a23f0 con 0x7f3170101970 2024-05-09T08:47:50.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.627+0000 7f31677fe640 1 -- 172.21.15.98:0/2753150620 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3158042030 con 0x7f3170101970 2024-05-09T08:47:50.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.628+0000 7f31677fe640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f315c0a2b90 0x7f315c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:50.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.628+0000 7f316e59c640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f315c0a2b90 0x7f315c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:50.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.628+0000 7f31677fe640 1 -- 172.21.15.98:0/2753150620 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f31580ef030 con 0x7f3170101970 2024-05-09T08:47:50.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.628+0000 7f316e59c640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f315c0a2b90 0x7f315c0a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f31701017d0 tx=0x7f3160005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:50.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.632+0000 7f31677fe640 1 -- 172.21.15.98:0/2753150620 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f31580bc430 con 0x7f3170101970 2024-05-09T08:47:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:50 smithi142 ceph-mon[34625]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:50.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:50.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:50.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:50.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:50.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:50.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:50 smithi142 ceph-mon[34625]: Deploying daemon mds.e on smithi105 2024-05-09T08:47:50.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:50 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.98:6836/3134006534,v1:172.21.15.98:6837/3134006534] up:boot 2024-05-09T08:47:50.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:50 smithi142 ceph-mon[34625]: fsmap 4 up:standby 2024-05-09T08:47:50.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:47:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.848+0000 7f3174f12640 1 -- 172.21.15.98:0/2753150620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31701a2780 con 0x7f3170101970 2024-05-09T08:47:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.849+0000 7f31677fe640 1 -- 172.21.15.98:0/2753150620 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f31580bbd30 con 0x7f3170101970 2024-05-09T08:47:50.849 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:47:50.849 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-09T08:47:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 5 2024-05-09T08:47:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.851+0000 7f3174f12640 1 -- 172.21.15.98:0/2753150620 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f315c0a2b90 msgr2=0x7f315c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.851+0000 7f3174f12640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f315c0a2b90 0x7f315c0a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f31701017d0 tx=0x7f3160005d30 comp rx=0 tx=0).stop 2024-05-09T08:47:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.851+0000 7f3174f12640 1 -- 172.21.15.98:0/2753150620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3170101970 msgr2=0x7f317019a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.851+0000 7f3174f12640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3170101970 0x7f317019a360 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3158009230 tx=0x7f3158004430 comp rx=0 tx=0).stop 2024-05-09T08:47:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.851+0000 7f3174f12640 1 -- 172.21.15.98:0/2753150620 shutdown_connections 2024-05-09T08:47:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.851+0000 7f3174f12640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3170102330 0x7f31701a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.851+0000 7f3174f12640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f315c0a2b90 0x7f315c0a5050 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.851+0000 7f3174f12640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3170101970 0x7f317019a360 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.851+0000 7f3174f12640 1 --2- 172.21.15.98:0/2753150620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3170100780 0x7f3170199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.851+0000 7f3174f12640 1 -- 172.21.15.98:0/2753150620 >> 172.21.15.98:0/2753150620 conn(0x7f31700fbd10 msgr2=0x7f3170109580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.851+0000 7f3174f12640 1 -- 172.21.15.98:0/2753150620 shutdown_connections 2024-05-09T08:47:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:50.851+0000 7f3174f12640 1 -- 172.21.15.98:0/2753150620 wait complete. 2024-05-09T08:47:51.484 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:51 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2753150620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:51.484 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:47:51 smithi105 systemd[1]: Starting Ceph mds.e for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:47:51.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:51 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2753150620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:51.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:51 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2753150620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:51.736 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:47:51 smithi105 podman[64202]: 2024-05-09 08:47:51.604686004 +0000 UTC m=+0.017799352 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:47:51.737 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:47:51 smithi105 podman[64202]: 2024-05-09 08:47:51.717967466 +0000 UTC m=+0.131080799 container create ce228e5ed3306a98be0e32cc00de77728e586e2cbea1c77e4c15d6c8df7e56b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-09T08:47:52.235 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:47:51 smithi105 podman[64202]: 2024-05-09 08:47:51.977684384 +0000 UTC m=+0.390797716 container init ce228e5ed3306a98be0e32cc00de77728e586e2cbea1c77e4c15d6c8df7e56b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-09T08:47:52.235 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:47:51 smithi105 podman[64202]: 2024-05-09 08:47:51.980336942 +0000 UTC m=+0.393450277 container start ce228e5ed3306a98be0e32cc00de77728e586e2cbea1c77e4c15d6c8df7e56b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-96805dc, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-09T08:47:52.235 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:47:52 smithi105 ceph-mds[64221]: -- 172.21.15.105:0/800420766 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563cd24fba40 con 0x563cd24ce800 2024-05-09T08:47:52.235 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:47:52 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: starting mds.e at 2024-05-09T08:47:52.235 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:47:52 smithi105 bash[64202]: ce228e5ed3306a98be0e32cc00de77728e586e2cbea1c77e4c15d6c8df7e56b4 2024-05-09T08:47:52.235 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:47:52 smithi105 systemd[1]: Started Ceph mds.e for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:47:52.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:52 smithi105 ceph-mon[34498]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:52.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:52.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:52.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:52.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:52.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:52.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:52 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:52.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:52 smithi098 ceph-mon[30092]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:52.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:52.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:52.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:52.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:52.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:52.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:52 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:52 smithi142 ceph-mon[34625]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:52 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:52.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:47:52.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.975+0000 7f06b9d6a640 1 -- 172.21.15.98:0/3227194150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b4107570 msgr2=0x7f06b41099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:52.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.975+0000 7f06b9d6a640 1 --2- 172.21.15.98:0/3227194150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b4107570 0x7f06b41099c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009900 tx=0x7f06ac02a410 comp rx=0 tx=0).stop 2024-05-09T08:47:52.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.975+0000 7f06b9d6a640 1 -- 172.21.15.98:0/3227194150 shutdown_connections 2024-05-09T08:47:52.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.975+0000 7f06b9d6a640 1 --2- 172.21.15.98:0/3227194150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b4107570 0x7f06b41099c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:52.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.975+0000 7f06b9d6a640 1 --2- 172.21.15.98:0/3227194150 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06b4104c40 0x7f06b4107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:52.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.975+0000 7f06b9d6a640 1 --2- 172.21.15.98:0/3227194150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06b4102310 0x7f06b4104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:52.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.975+0000 7f06b9d6a640 1 -- 172.21.15.98:0/3227194150 >> 172.21.15.98:0/3227194150 conn(0x7f06b40fbd20 msgr2=0x7f06b40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:52.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.975+0000 7f06b9d6a640 1 -- 172.21.15.98:0/3227194150 shutdown_connections 2024-05-09T08:47:52.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.975+0000 7f06b9d6a640 1 -- 172.21.15.98:0/3227194150 wait complete. 2024-05-09T08:47:52.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.976+0000 7f06b9d6a640 1 Processor -- start 2024-05-09T08:47:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.976+0000 7f06b9d6a640 1 -- start start 2024-05-09T08:47:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.976+0000 7f06b9d6a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06b4102310 0x7f06b4101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.976+0000 7f06b9d6a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b4104c40 0x7f06b41002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.976+0000 7f06b9d6a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06b4107570 0x7f06b4100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.976+0000 7f06b9d6a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f06b410d530 con 0x7f06b4104c40 2024-05-09T08:47:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.976+0000 7f06b9d6a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f06b410d3b0 con 0x7f06b4107570 2024-05-09T08:47:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.976+0000 7f06b9d6a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f06b410d6b0 con 0x7f06b4102310 2024-05-09T08:47:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.976+0000 7f06b2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b4104c40 0x7f06b41002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.976+0000 7f06b37fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06b4102310 0x7f06b4101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.976+0000 7f06b2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b4104c40 0x7f06b41002c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54746/0 (socket says 172.21.15.98:54746) 2024-05-09T08:47:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.976+0000 7f06b37fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06b4102310 0x7f06b4101c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:43984/0 (socket says 172.21.15.98:43984) 2024-05-09T08:47:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.976+0000 7f06b2ffd640 1 -- 172.21.15.98:0/2592955810 learned_addr learned my addr 172.21.15.98:0/2592955810 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:47:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.977+0000 7f06b3fff640 1 --2- 172.21.15.98:0/2592955810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06b4107570 0x7f06b4100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.977+0000 7f06b2ffd640 1 -- 172.21.15.98:0/2592955810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06b4102310 msgr2=0x7f06b4101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.977+0000 7f06b2ffd640 1 --2- 172.21.15.98:0/2592955810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06b4102310 0x7f06b4101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.977+0000 7f06b2ffd640 1 -- 172.21.15.98:0/2592955810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06b4107570 msgr2=0x7f06b4100800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.977+0000 7f06b2ffd640 1 --2- 172.21.15.98:0/2592955810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06b4107570 0x7f06b4100800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.977+0000 7f06b2ffd640 1 -- 172.21.15.98:0/2592955810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b4101090 con 0x7f06b4104c40 2024-05-09T08:47:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.977+0000 7f06b3fff640 1 --2- 172.21.15.98:0/2592955810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06b4107570 0x7f06b4100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:47:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.977+0000 7f06b2ffd640 1 --2- 172.21.15.98:0/2592955810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b4104c40 0x7f06b41002c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f06a800e9c0 tx=0x7f06a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.977+0000 7f06b0ff9640 1 -- 172.21.15.98:0/2592955810 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a800cd80 con 0x7f06b4104c40 2024-05-09T08:47:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.977+0000 7f06b0ff9640 1 -- 172.21.15.98:0/2592955810 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06a8005500 con 0x7f06b4104c40 2024-05-09T08:47:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.977+0000 7f06b9d6a640 1 -- 172.21.15.98:0/2592955810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b41a1c80 con 0x7f06b4104c40 2024-05-09T08:47:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.977+0000 7f06b0ff9640 1 -- 172.21.15.98:0/2592955810 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8010690 con 0x7f06b4104c40 2024-05-09T08:47:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.977+0000 7f06b9d6a640 1 -- 172.21.15.98:0/2592955810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b41a21d0 con 0x7f06b4104c40 2024-05-09T08:47:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.978+0000 7f06b9d6a640 1 -- 172.21.15.98:0/2592955810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b40fd810 con 0x7f06b4104c40 2024-05-09T08:47:52.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.979+0000 7f06b0ff9640 1 -- 172.21.15.98:0/2592955810 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f06a80046c0 con 0x7f06b4104c40 2024-05-09T08:47:52.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.980+0000 7f06b0ff9640 1 --2- 172.21.15.98:0/2592955810 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f069c0a2b90 0x7f069c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:52.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.980+0000 7f06b37fe640 1 --2- 172.21.15.98:0/2592955810 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f069c0a2b90 0x7f069c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:52.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.980+0000 7f06b0ff9640 1 -- 172.21.15.98:0/2592955810 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f06a80cf030 con 0x7f06b4104c40 2024-05-09T08:47:52.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.980+0000 7f06b37fe640 1 --2- 172.21.15.98:0/2592955810 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f069c0a2b90 0x7f069c0a5050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f06a4005fd0 tx=0x7f06a4009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:52.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:52.984+0000 7f06b0ff9640 1 -- 172.21.15.98:0/2592955810 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f06a809c430 con 0x7f06b4104c40 2024-05-09T08:47:53.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:53.195+0000 7f06b9d6a640 1 -- 172.21.15.98:0/2592955810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06b402d3e0 con 0x7f06b4104c40 2024-05-09T08:47:53.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:53.196+0000 7f06b0ff9640 1 -- 172.21.15.98:0/2592955810 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7f06a809bd30 con 0x7f06b4104c40 2024-05-09T08:47:53.198 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:47:53.199 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-09T08:47:53.199 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 6 2024-05-09T08:47:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:53.198+0000 7f06b9d6a640 1 -- 172.21.15.98:0/2592955810 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f069c0a2b90 msgr2=0x7f069c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:53.198+0000 7f06b9d6a640 1 --2- 172.21.15.98:0/2592955810 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f069c0a2b90 0x7f069c0a5050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f06a4005fd0 tx=0x7f06a4009680 comp rx=0 tx=0).stop 2024-05-09T08:47:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:53.198+0000 7f06b9d6a640 1 -- 172.21.15.98:0/2592955810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b4104c40 msgr2=0x7f06b41002c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:53.198+0000 7f06b9d6a640 1 --2- 172.21.15.98:0/2592955810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b4104c40 0x7f06b41002c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f06a800e9c0 tx=0x7f06a800ee80 comp rx=0 tx=0).stop 2024-05-09T08:47:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:53.198+0000 7f06b9d6a640 1 -- 172.21.15.98:0/2592955810 shutdown_connections 2024-05-09T08:47:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:53.198+0000 7f06b9d6a640 1 --2- 172.21.15.98:0/2592955810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06b4107570 0x7f06b4100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:53.198+0000 7f06b9d6a640 1 --2- 172.21.15.98:0/2592955810 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f069c0a2b90 0x7f069c0a5050 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:53.198+0000 7f06b9d6a640 1 --2- 172.21.15.98:0/2592955810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06b4104c40 0x7f06b41002c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:53.198+0000 7f06b9d6a640 1 --2- 172.21.15.98:0/2592955810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06b4102310 0x7f06b4101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:53.198+0000 7f06b9d6a640 1 -- 172.21.15.98:0/2592955810 >> 172.21.15.98:0/2592955810 conn(0x7f06b40fbd20 msgr2=0x7f06b4102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:53.198+0000 7f06b9d6a640 1 -- 172.21.15.98:0/2592955810 shutdown_connections 2024-05-09T08:47:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:53.198+0000 7f06b9d6a640 1 -- 172.21.15.98:0/2592955810 wait complete. 2024-05-09T08:47:53.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:53 smithi142 ceph-mon[34625]: Deploying daemon mds.f on smithi142 2024-05-09T08:47:53.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:53 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.105:6834/588012490,v1:172.21.15.105:6835/588012490] up:boot 2024-05-09T08:47:53.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:53 smithi142 ceph-mon[34625]: fsmap 5 up:standby 2024-05-09T08:47:53.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:53 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:47:53.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:53 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2592955810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:53.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:53 smithi098 ceph-mon[30092]: Deploying daemon mds.f on smithi142 2024-05-09T08:47:53.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:53 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.105:6834/588012490,v1:172.21.15.105:6835/588012490] up:boot 2024-05-09T08:47:53.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:53 smithi098 ceph-mon[30092]: fsmap 5 up:standby 2024-05-09T08:47:53.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:53 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:47:53.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:53 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2592955810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:53.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:53 smithi105 ceph-mon[34498]: Deploying daemon mds.f on smithi142 2024-05-09T08:47:53.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:53 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.105:6834/588012490,v1:172.21.15.105:6835/588012490] up:boot 2024-05-09T08:47:53.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:53 smithi105 ceph-mon[34498]: fsmap 5 up:standby 2024-05-09T08:47:53.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:53 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:47:53.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:53 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2592955810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:54.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:54 smithi142 ceph-mon[34625]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:54.408 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 08:47:54 smithi142 systemd[1]: Starting Ceph mds.f for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:47:54.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:54 smithi098 ceph-mon[30092]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:54.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:54 smithi105 ceph-mon[34498]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:54.788 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 08:47:54 smithi142 podman[64263]: 2024-05-09 08:47:54.423865762 +0000 UTC m=+0.018049482 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:47:54.788 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 08:47:54 smithi142 podman[64263]: 2024-05-09 08:47:54.545011089 +0000 UTC m=+0.139194782 container create ea75d301ead0b9caa8bad2fccafe84480eb678f203fd2f7e976b03152bc12dfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-09T08:47:54.788 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 08:47:54 smithi142 podman[64263]: 2024-05-09 08:47:54.789310256 +0000 UTC m=+0.383493951 container init ea75d301ead0b9caa8bad2fccafe84480eb678f203fd2f7e976b03152bc12dfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-f, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-09T08:47:55.158 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 08:47:54 smithi142 podman[64263]: 2024-05-09 08:47:54.792019641 +0000 UTC m=+0.386203345 container start ea75d301ead0b9caa8bad2fccafe84480eb678f203fd2f7e976b03152bc12dfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0) 2024-05-09T08:47:55.158 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 08:47:54 smithi142 ceph-mds[64283]: -- 172.21.15.142:0/2903361045 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556d26c59a40 con 0x556d26c2cc00 2024-05-09T08:47:55.158 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 08:47:54 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-f[64279]: starting mds.f at 2024-05-09T08:47:55.158 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 08:47:54 smithi142 bash[64263]: ea75d301ead0b9caa8bad2fccafe84480eb678f203fd2f7e976b03152bc12dfc 2024-05-09T08:47:55.158 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 08:47:54 smithi142 systemd[1]: Started Ceph mds.f for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:47:55.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:47:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.325+0000 7f8fdf08c640 1 -- 172.21.15.98:0/691706353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8103430 msgr2=0x7f8fd8109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.325+0000 7f8fdf08c640 1 --2- 172.21.15.98:0/691706353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8103430 0x7f8fd8109cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009ca0 tx=0x7f8fd002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:47:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.325+0000 7f8fdf08c640 1 -- 172.21.15.98:0/691706353 shutdown_connections 2024-05-09T08:47:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.325+0000 7f8fdf08c640 1 --2- 172.21.15.98:0/691706353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8103430 0x7f8fd8109cc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.325+0000 7f8fdf08c640 1 --2- 172.21.15.98:0/691706353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fd8102a70 0x7f8fd8102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.325+0000 7f8fdf08c640 1 --2- 172.21.15.98:0/691706353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fd8101880 0x7f8fd8101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.325+0000 7f8fdf08c640 1 -- 172.21.15.98:0/691706353 >> 172.21.15.98:0/691706353 conn(0x7f8fd80fce10 msgr2=0x7f8fd80ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.325+0000 7f8fdf08c640 1 -- 172.21.15.98:0/691706353 shutdown_connections 2024-05-09T08:47:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.325+0000 7f8fdf08c640 1 -- 172.21.15.98:0/691706353 wait complete. 2024-05-09T08:47:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fdf08c640 1 Processor -- start 2024-05-09T08:47:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fdf08c640 1 -- start start 2024-05-09T08:47:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fdf08c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fd8101880 0x7f8fd8197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fdf08c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8102a70 0x7f8fd8198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fdf08c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fd8103430 0x7f8fd819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fdf08c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd810d4e0 con 0x7f8fd8102a70 2024-05-09T08:47:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fdf08c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd810d360 con 0x7f8fd8101880 2024-05-09T08:47:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fdf08c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd810d660 con 0x7f8fd8103430 2024-05-09T08:47:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8102a70 0x7f8fd8198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8102a70 0x7f8fd8198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54760/0 (socket says 172.21.15.98:54760) 2024-05-09T08:47:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fd7fff640 1 -- 172.21.15.98:0/2077284699 learned_addr learned my addr 172.21.15.98:0/2077284699 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:47:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fdd629640 1 --2- 172.21.15.98:0/2077284699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fd8103430 0x7f8fd819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fdce28640 1 --2- 172.21.15.98:0/2077284699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fd8101880 0x7f8fd8197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fd7fff640 1 -- 172.21.15.98:0/2077284699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fd8103430 msgr2=0x7f8fd819f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fd7fff640 1 --2- 172.21.15.98:0/2077284699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fd8103430 0x7f8fd819f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fd7fff640 1 -- 172.21.15.98:0/2077284699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fd8101880 msgr2=0x7f8fd8197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fd7fff640 1 --2- 172.21.15.98:0/2077284699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fd8101880 0x7f8fd8197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fd7fff640 1 -- 172.21.15.98:0/2077284699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd819f710 con 0x7f8fd8102a70 2024-05-09T08:47:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.326+0000 7f8fd7fff640 1 --2- 172.21.15.98:0/2077284699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8102a70 0x7f8fd8198030 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400c890 tx=0x7f8fc400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.327+0000 7f8fd5ffb640 1 -- 172.21.15.98:0/2077284699 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4004220 con 0x7f8fd8102a70 2024-05-09T08:47:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.327+0000 7f8fd5ffb640 1 -- 172.21.15.98:0/2077284699 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fc40043c0 con 0x7f8fd8102a70 2024-05-09T08:47:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.327+0000 7f8fd5ffb640 1 -- 172.21.15.98:0/2077284699 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40049e0 con 0x7f8fd8102a70 2024-05-09T08:47:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.327+0000 7f8fdf08c640 1 -- 172.21.15.98:0/2077284699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd819fa00 con 0x7f8fd8102a70 2024-05-09T08:47:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.327+0000 7f8fdf08c640 1 -- 172.21.15.98:0/2077284699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd819ff50 con 0x7f8fd8102a70 2024-05-09T08:47:55.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.328+0000 7f8fdf08c640 1 -- 172.21.15.98:0/2077284699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fd8108b40 con 0x7f8fd8102a70 2024-05-09T08:47:55.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.329+0000 7f8fd5ffb640 1 -- 172.21.15.98:0/2077284699 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8fc40079b0 con 0x7f8fd8102a70 2024-05-09T08:47:55.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.330+0000 7f8fd5ffb640 1 --2- 172.21.15.98:0/2077284699 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8fc80a2b90 0x7f8fc80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:55.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.330+0000 7f8fdce28640 1 --2- 172.21.15.98:0/2077284699 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8fc80a2b90 0x7f8fc80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:55.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.330+0000 7f8fd5ffb640 1 -- 172.21.15.98:0/2077284699 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f8fc40cec00 con 0x7f8fd8102a70 2024-05-09T08:47:55.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.330+0000 7f8fdce28640 1 --2- 172.21.15.98:0/2077284699 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8fc80a2b90 0x7f8fc80a5050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8fd81028d0 tx=0x7f8fcc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:55.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.334+0000 7f8fd5ffb640 1 -- 172.21.15.98:0/2077284699 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8fc40d5080 con 0x7f8fd8102a70 2024-05-09T08:47:55.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.549+0000 7f8fdf08c640 1 -- 172.21.15.98:0/2077284699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fd802d1c0 con 0x7f8fd8102a70 2024-05-09T08:47:55.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.550+0000 7f8fd5ffb640 1 -- 172.21.15.98:0/2077284699 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7f8fc4014030 con 0x7f8fd8102a70 2024-05-09T08:47:55.549 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:47:55.549 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-09T08:47:55.549 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 7 2024-05-09T08:47:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.552+0000 7f8fdf08c640 1 -- 172.21.15.98:0/2077284699 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8fc80a2b90 msgr2=0x7f8fc80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.552+0000 7f8fdf08c640 1 --2- 172.21.15.98:0/2077284699 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8fc80a2b90 0x7f8fc80a5050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8fd81028d0 tx=0x7f8fcc005f90 comp rx=0 tx=0).stop 2024-05-09T08:47:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.552+0000 7f8fdf08c640 1 -- 172.21.15.98:0/2077284699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8102a70 msgr2=0x7f8fd8198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.552+0000 7f8fdf08c640 1 --2- 172.21.15.98:0/2077284699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8102a70 0x7f8fd8198030 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400c890 tx=0x7f8fc400cd50 comp rx=0 tx=0).stop 2024-05-09T08:47:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.552+0000 7f8fdf08c640 1 -- 172.21.15.98:0/2077284699 shutdown_connections 2024-05-09T08:47:55.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.552+0000 7f8fdf08c640 1 --2- 172.21.15.98:0/2077284699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fd8103430 0x7f8fd819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:55.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.552+0000 7f8fdf08c640 1 --2- 172.21.15.98:0/2077284699 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8fc80a2b90 0x7f8fc80a5050 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:55.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.552+0000 7f8fdf08c640 1 --2- 172.21.15.98:0/2077284699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fd8102a70 0x7f8fd8198030 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:55.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.552+0000 7f8fdf08c640 1 --2- 172.21.15.98:0/2077284699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fd8101880 0x7f8fd8197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:55.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.552+0000 7f8fdf08c640 1 -- 172.21.15.98:0/2077284699 >> 172.21.15.98:0/2077284699 conn(0x7f8fd80fce10 msgr2=0x7f8fd8105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:55.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.552+0000 7f8fdf08c640 1 -- 172.21.15.98:0/2077284699 shutdown_connections 2024-05-09T08:47:55.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:55.552+0000 7f8fdf08c640 1 -- 172.21.15.98:0/2077284699 wait complete. 2024-05-09T08:47:56.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:55 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:56.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:55 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:56.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:55 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:56.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:55 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:56.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:55 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:56.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:55 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:56.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:55 smithi142 ceph-mon[34625]: Deploying daemon mds.c on smithi142 2024-05-09T08:47:56.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:55 smithi142 ceph-mon[34625]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:56.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:55 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.142:6834/2791041858,v1:172.21.15.142:6835/2791041858] up:boot 2024-05-09T08:47:56.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:55 smithi142 ceph-mon[34625]: fsmap 6 up:standby 2024-05-09T08:47:56.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:55 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:47:56.160 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:55 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2077284699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:55 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:55 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:55 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:55 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:55 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:55 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:55 smithi105 ceph-mon[34498]: Deploying daemon mds.c on smithi142 2024-05-09T08:47:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:55 smithi105 ceph-mon[34498]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:55 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.142:6834/2791041858,v1:172.21.15.142:6835/2791041858] up:boot 2024-05-09T08:47:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:55 smithi105 ceph-mon[34498]: fsmap 6 up:standby 2024-05-09T08:47:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:55 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:47:56.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:55 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2077284699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:56.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:56.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:56.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:56.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:56.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:56.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:56.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:55 smithi098 ceph-mon[30092]: Deploying daemon mds.c on smithi142 2024-05-09T08:47:56.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:55 smithi098 ceph-mon[30092]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:56.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:55 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.142:6834/2791041858,v1:172.21.15.142:6835/2791041858] up:boot 2024-05-09T08:47:56.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:55 smithi098 ceph-mon[30092]: fsmap 6 up:standby 2024-05-09T08:47:56.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:55 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:47:56.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:55 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2077284699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:57.357 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 08:47:57 smithi142 systemd[1]: Starting Ceph mds.c for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:47:57.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:47:57.658 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 08:47:57 smithi142 podman[64570]: 2024-05-09 08:47:57.260998746 +0000 UTC m=+0.018528380 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:47:57.658 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 08:47:57 smithi142 podman[64570]: 2024-05-09 08:47:57.402395837 +0000 UTC m=+0.159925467 container create a83c33cdbf6a3a0741786ea6c5356f2009592b745537d2a91fd5d0e239f286df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-c, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-05-09T08:47:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.677+0000 7f74827ce640 1 -- 172.21.15.98:0/746524460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f747c1013f0 msgr2=0x7f747c108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.677+0000 7f74827ce640 1 --2- 172.21.15.98:0/746524460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f747c1013f0 0x7f747c108c90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7474009cd0 tx=0x7f747402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:47:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.677+0000 7f74827ce640 1 -- 172.21.15.98:0/746524460 shutdown_connections 2024-05-09T08:47:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.677+0000 7f74827ce640 1 --2- 172.21.15.98:0/746524460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f747c1013f0 0x7f747c108c90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.677+0000 7f74827ce640 1 --2- 172.21.15.98:0/746524460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f747c100a30 0x7f747c100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.677+0000 7f74827ce640 1 --2- 172.21.15.98:0/746524460 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f747c101d90 0x7f747c1021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.677+0000 7f74827ce640 1 -- 172.21.15.98:0/746524460 >> 172.21.15.98:0/746524460 conn(0x7f747c0fb4a0 msgr2=0x7f747c0fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.677+0000 7f74827ce640 1 -- 172.21.15.98:0/746524460 shutdown_connections 2024-05-09T08:47:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.677+0000 7f74827ce640 1 -- 172.21.15.98:0/746524460 wait complete. 2024-05-09T08:47:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.677+0000 7f74827ce640 1 Processor -- start 2024-05-09T08:47:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.677+0000 7f74827ce640 1 -- start start 2024-05-09T08:47:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f74827ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f747c100a30 0x7f747c196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f74827ce640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f747c1013f0 0x7f747c196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f74827ce640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f747c101d90 0x7f747c19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f74827ce640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f747c10c400 con 0x7f747c100a30 2024-05-09T08:47:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f74827ce640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f747c10c280 con 0x7f747c101d90 2024-05-09T08:47:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f74827ce640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f747c10c580 con 0x7f747c1013f0 2024-05-09T08:47:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f747bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f747c100a30 0x7f747c196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f747bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f747c100a30 0x7f747c196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43276/0 (socket says 172.21.15.98:43276) 2024-05-09T08:47:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f747bfff640 1 -- 172.21.15.98:0/1173167492 learned_addr learned my addr 172.21.15.98:0/1173167492 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:47:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f747b7fe640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f747c1013f0 0x7f747c196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f7480d6b640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f747c101d90 0x7f747c19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f747bfff640 1 -- 172.21.15.98:0/1173167492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f747c1013f0 msgr2=0x7f747c196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f747bfff640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f747c1013f0 0x7f747c196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f747bfff640 1 -- 172.21.15.98:0/1173167492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f747c101d90 msgr2=0x7f747c19dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f747bfff640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f747c101d90 0x7f747c19dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f747bfff640 1 -- 172.21.15.98:0/1173167492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747c19e630 con 0x7f747c100a30 2024-05-09T08:47:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f7480d6b640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f747c101d90 0x7f747c19dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:47:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f747bfff640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f747c100a30 0x7f747c196a10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f746c009970 tx=0x7f746c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f74797fa640 1 -- 172.21.15.98:0/1173167492 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c00d030 con 0x7f747c100a30 2024-05-09T08:47:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f74797fa640 1 -- 172.21.15.98:0/1173167492 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f746c007d40 con 0x7f747c100a30 2024-05-09T08:47:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.678+0000 7f74827ce640 1 -- 172.21.15.98:0/1173167492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f747c0ff980 con 0x7f747c100a30 2024-05-09T08:47:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.679+0000 7f74797fa640 1 -- 172.21.15.98:0/1173167492 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c00cc50 con 0x7f747c100a30 2024-05-09T08:47:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.679+0000 7f74827ce640 1 -- 172.21.15.98:0/1173167492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f747c0fff20 con 0x7f747c100a30 2024-05-09T08:47:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.679+0000 7f74827ce640 1 -- 172.21.15.98:0/1173167492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f747c1075e0 con 0x7f747c100a30 2024-05-09T08:47:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.681+0000 7f74797fa640 1 -- 172.21.15.98:0/1173167492 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f746c007860 con 0x7f747c100a30 2024-05-09T08:47:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.681+0000 7f74797fa640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f74680a2c10 0x7f74680a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.681+0000 7f747b7fe640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f74680a2c10 0x7f74680a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.682+0000 7f74797fa640 1 -- 172.21.15.98:0/1173167492 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f746c0d0550 con 0x7f747c100a30 2024-05-09T08:47:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.682+0000 7f747b7fe640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f74680a2c10 0x7f74680a50d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f747c197f20 tx=0x7f746400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.685+0000 7f74797fa640 1 -- 172.21.15.98:0/1173167492 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f746c09d950 con 0x7f747c100a30 2024-05-09T08:47:57.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.901+0000 7f74827ce640 1 -- 172.21.15.98:0/1173167492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f747c107800 con 0x7f747c100a30 2024-05-09T08:47:57.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.902+0000 7f74797fa640 1 -- 172.21.15.98:0/1173167492 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7f746c09d250 con 0x7f747c100a30 2024-05-09T08:47:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:47:57.901 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-09T08:47:57.901 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 7 2024-05-09T08:47:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.904+0000 7f74827ce640 1 -- 172.21.15.98:0/1173167492 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f74680a2c10 msgr2=0x7f74680a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.904+0000 7f74827ce640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f74680a2c10 0x7f74680a50d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f747c197f20 tx=0x7f746400b390 comp rx=0 tx=0).stop 2024-05-09T08:47:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.904+0000 7f74827ce640 1 -- 172.21.15.98:0/1173167492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f747c100a30 msgr2=0x7f747c196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.904+0000 7f74827ce640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f747c100a30 0x7f747c196a10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f746c009970 tx=0x7f746c009e30 comp rx=0 tx=0).stop 2024-05-09T08:47:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.904+0000 7f74827ce640 1 -- 172.21.15.98:0/1173167492 shutdown_connections 2024-05-09T08:47:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.904+0000 7f74827ce640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f747c101d90 0x7f747c19dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.904+0000 7f74827ce640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f747c1013f0 0x7f747c196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.904+0000 7f74827ce640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f74680a2c10 0x7f74680a50d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.904+0000 7f74827ce640 1 --2- 172.21.15.98:0/1173167492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f747c100a30 0x7f747c196a10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.904+0000 7f74827ce640 1 -- 172.21.15.98:0/1173167492 >> 172.21.15.98:0/1173167492 conn(0x7f747c0fb4a0 msgr2=0x7f747c104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.904+0000 7f74827ce640 1 -- 172.21.15.98:0/1173167492 shutdown_connections 2024-05-09T08:47:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:47:57.904+0000 7f74827ce640 1 -- 172.21.15.98:0/1173167492 wait complete. 2024-05-09T08:47:58.158 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 08:47:57 smithi142 podman[64570]: 2024-05-09 08:47:57.669719122 +0000 UTC m=+0.427248760 container init a83c33cdbf6a3a0741786ea6c5356f2009592b745537d2a91fd5d0e239f286df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:47:58.158 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 08:47:57 smithi142 podman[64570]: 2024-05-09 08:47:57.672626136 +0000 UTC m=+0.430155770 container start a83c33cdbf6a3a0741786ea6c5356f2009592b745537d2a91fd5d0e239f286df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-09T08:47:58.158 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 08:47:57 smithi142 ceph-mds[64597]: -- 172.21.15.142:0/3989557669 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e8b3c35a40 con 0x55e8b3c08800 2024-05-09T08:47:58.158 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 08:47:57 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-c[64588]: starting mds.c at 2024-05-09T08:47:58.158 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 08:47:57 smithi142 bash[64570]: a83c33cdbf6a3a0741786ea6c5356f2009592b745537d2a91fd5d0e239f286df 2024-05-09T08:47:58.158 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 08:47:57 smithi142 systemd[1]: Started Ceph mds.c for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:47:58.477 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:58 smithi105 ceph-mon[34498]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:58.477 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:58 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:58.477 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:58 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:58.477 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:58 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:58.477 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:58 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:58.477 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:58 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:58.477 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:58 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:58.478 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:58 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1173167492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:58 smithi098 ceph-mon[30092]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:58 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:58 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:58 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:58 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:58 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:58 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:58 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1173167492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:58.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:58 smithi142 ceph-mon[34625]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:58.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:58 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:58.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:58 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:58.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:58 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:47:58.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:58 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:47:58.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:58 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:47:58.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:58 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:58.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:58 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1173167492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:59.495 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:59 smithi105 ceph-mon[34498]: Deploying daemon mds.h on smithi105 2024-05-09T08:47:59.495 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:59 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] up:boot 2024-05-09T08:47:59.495 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:59 smithi105 ceph-mon[34498]: fsmap 7 up:standby 2024-05-09T08:47:59.495 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:47:59 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:47:59.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:59 smithi098 ceph-mon[30092]: Deploying daemon mds.h on smithi105 2024-05-09T08:47:59.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:59 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] up:boot 2024-05-09T08:47:59.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:59 smithi098 ceph-mon[30092]: fsmap 7 up:standby 2024-05-09T08:47:59.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:47:59 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:47:59.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:59 smithi142 ceph-mon[34625]: Deploying daemon mds.h on smithi105 2024-05-09T08:47:59.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:59 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] up:boot 2024-05-09T08:47:59.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:59 smithi142 ceph-mon[34625]: fsmap 7 up:standby 2024-05-09T08:47:59.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:47:59 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:47:59.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:48:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.023+0000 7f7a91910640 1 -- 172.21.15.98:0/1332494506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c102330 msgr2=0x7f7a8c108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.023+0000 7f7a91910640 1 --2- 172.21.15.98:0/1332494506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c102330 0x7f7a8c108bc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7a84009ca0 tx=0x7f7a8402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.023+0000 7f7a91910640 1 -- 172.21.15.98:0/1332494506 shutdown_connections 2024-05-09T08:48:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.023+0000 7f7a91910640 1 --2- 172.21.15.98:0/1332494506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c102330 0x7f7a8c108bc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.023+0000 7f7a91910640 1 --2- 172.21.15.98:0/1332494506 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a8c101970 0x7f7a8c101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.023+0000 7f7a91910640 1 --2- 172.21.15.98:0/1332494506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a8c100780 0x7f7a8c100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.023+0000 7f7a91910640 1 -- 172.21.15.98:0/1332494506 >> 172.21.15.98:0/1332494506 conn(0x7f7a8c0fbd30 msgr2=0x7f7a8c0fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.023+0000 7f7a91910640 1 -- 172.21.15.98:0/1332494506 shutdown_connections 2024-05-09T08:48:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.023+0000 7f7a91910640 1 -- 172.21.15.98:0/1332494506 wait complete. 2024-05-09T08:48:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.024+0000 7f7a91910640 1 Processor -- start 2024-05-09T08:48:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.024+0000 7f7a91910640 1 -- start start 2024-05-09T08:48:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.024+0000 7f7a91910640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c100780 0x7f7a8c06aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.024+0000 7f7a91910640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a8c101970 0x7f7a8c19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.024+0000 7f7a91910640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a8c102330 0x7f7a8c19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.024+0000 7f7a91910640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c10c460 con 0x7f7a8c100780 2024-05-09T08:48:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.024+0000 7f7a91910640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c10c2e0 con 0x7f7a8c101970 2024-05-09T08:48:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.024+0000 7f7a91910640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c10c5e0 con 0x7f7a8c102330 2024-05-09T08:48:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.024+0000 7f7a8affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c100780 0x7f7a8c06aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.024+0000 7f7a8affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c100780 0x7f7a8c06aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43294/0 (socket says 172.21.15.98:43294) 2024-05-09T08:48:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.024+0000 7f7a8affd640 1 -- 172.21.15.98:0/4189240508 learned_addr learned my addr 172.21.15.98:0/4189240508 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.024+0000 7f7a8a7fc640 1 --2- 172.21.15.98:0/4189240508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a8c101970 0x7f7a8c19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.025+0000 7f7a8b7fe640 1 --2- 172.21.15.98:0/4189240508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a8c102330 0x7f7a8c19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.025+0000 7f7a8affd640 1 -- 172.21.15.98:0/4189240508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a8c102330 msgr2=0x7f7a8c19eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.025+0000 7f7a8affd640 1 --2- 172.21.15.98:0/4189240508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a8c102330 0x7f7a8c19eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.025+0000 7f7a8affd640 1 -- 172.21.15.98:0/4189240508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a8c101970 msgr2=0x7f7a8c19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.025+0000 7f7a8affd640 1 --2- 172.21.15.98:0/4189240508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a8c101970 0x7f7a8c19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.025+0000 7f7a8affd640 1 -- 172.21.15.98:0/4189240508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a8c19f7a0 con 0x7f7a8c100780 2024-05-09T08:48:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.025+0000 7f7a8affd640 1 --2- 172.21.15.98:0/4189240508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c100780 0x7f7a8c06aa60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00e9c0 tx=0x7f7a7c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.025+0000 7f7a9090e640 1 -- 172.21.15.98:0/4189240508 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c00ccc0 con 0x7f7a8c100780 2024-05-09T08:48:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.025+0000 7f7a9090e640 1 -- 172.21.15.98:0/4189240508 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a7c0035f0 con 0x7f7a8c100780 2024-05-09T08:48:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.025+0000 7f7a91910640 1 -- 172.21.15.98:0/4189240508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a8c19fa90 con 0x7f7a8c100780 2024-05-09T08:48:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.025+0000 7f7a9090e640 1 -- 172.21.15.98:0/4189240508 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c00ce60 con 0x7f7a8c100780 2024-05-09T08:48:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.025+0000 7f7a91910640 1 -- 172.21.15.98:0/4189240508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a8c19ffe0 con 0x7f7a8c100780 2024-05-09T08:48:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.026+0000 7f7a91910640 1 -- 172.21.15.98:0/4189240508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a8c068aa0 con 0x7f7a8c100780 2024-05-09T08:48:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.027+0000 7f7a9090e640 1 -- 172.21.15.98:0/4189240508 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7a7c0037a0 con 0x7f7a8c100780 2024-05-09T08:48:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.028+0000 7f7a9090e640 1 --2- 172.21.15.98:0/4189240508 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f7a740a2b90 0x7f7a740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.028+0000 7f7a8a7fc640 1 --2- 172.21.15.98:0/4189240508 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f7a740a2b90 0x7f7a740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.028+0000 7f7a9090e640 1 -- 172.21.15.98:0/4189240508 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f7a7c0cefb0 con 0x7f7a8c100780 2024-05-09T08:48:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.028+0000 7f7a8a7fc640 1 --2- 172.21.15.98:0/4189240508 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f7a740a2b90 0x7f7a740a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c06a7d0 tx=0x7f7a80009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.032+0000 7f7a9090e640 1 -- 172.21.15.98:0/4189240508 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7a7c09c3b0 con 0x7f7a8c100780 2024-05-09T08:48:00.237 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 08:48:00 smithi105 systemd[1]: Starting Ceph mds.h for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:48:00.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.241+0000 7f7a91910640 1 -- 172.21.15.98:0/4189240508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a8c02d1f0 con 0x7f7a8c100780 2024-05-09T08:48:00.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.244+0000 7f7a9090e640 1 -- 172.21.15.98:0/4189240508 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7f7a7c09bcb0 con 0x7f7a8c100780 2024-05-09T08:48:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:00.243 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-05-09T08:48:00.244 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-09T08:48:00.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.246+0000 7f7a91910640 1 -- 172.21.15.98:0/4189240508 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f7a740a2b90 msgr2=0x7f7a740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:00.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.246+0000 7f7a91910640 1 --2- 172.21.15.98:0/4189240508 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f7a740a2b90 0x7f7a740a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c06a7d0 tx=0x7f7a80009110 comp rx=0 tx=0).stop 2024-05-09T08:48:00.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.246+0000 7f7a91910640 1 -- 172.21.15.98:0/4189240508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c100780 msgr2=0x7f7a8c06aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:00.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.246+0000 7f7a91910640 1 --2- 172.21.15.98:0/4189240508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c100780 0x7f7a8c06aa60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00e9c0 tx=0x7f7a7c00edc0 comp rx=0 tx=0).stop 2024-05-09T08:48:00.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.246+0000 7f7a91910640 1 -- 172.21.15.98:0/4189240508 shutdown_connections 2024-05-09T08:48:00.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.246+0000 7f7a91910640 1 --2- 172.21.15.98:0/4189240508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a8c102330 0x7f7a8c19eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:00.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.246+0000 7f7a91910640 1 --2- 172.21.15.98:0/4189240508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a8c101970 0x7f7a8c19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:00.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.246+0000 7f7a91910640 1 --2- 172.21.15.98:0/4189240508 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f7a740a2b90 0x7f7a740a5050 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:00.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.246+0000 7f7a91910640 1 --2- 172.21.15.98:0/4189240508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c100780 0x7f7a8c06aa60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:00.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.246+0000 7f7a91910640 1 -- 172.21.15.98:0/4189240508 >> 172.21.15.98:0/4189240508 conn(0x7f7a8c0fbd30 msgr2=0x7f7a8c104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:00.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.246+0000 7f7a91910640 1 -- 172.21.15.98:0/4189240508 shutdown_connections 2024-05-09T08:48:00.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:00.246+0000 7f7a91910640 1 -- 172.21.15.98:0/4189240508 wait complete. 2024-05-09T08:48:00.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:00 smithi098 ceph-mon[30092]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:00.613 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:00 smithi105 ceph-mon[34498]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:00.614 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 08:48:00 smithi105 podman[64521]: 2024-05-09 08:48:00.227787215 +0000 UTC m=+0.017914366 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:48:00.614 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 08:48:00 smithi105 podman[64521]: 2024-05-09 08:48:00.342127954 +0000 UTC m=+0.132255138 container create d2eef274aff1aa3f88ce3d8b1294e846edd32ee2c37658d560c0e94db276c1ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-96805dc) 2024-05-09T08:48:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:00 smithi142 ceph-mon[34625]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:00.986 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 08:48:00 smithi105 podman[64521]: 2024-05-09 08:48:00.615321521 +0000 UTC m=+0.405448663 container init d2eef274aff1aa3f88ce3d8b1294e846edd32ee2c37658d560c0e94db276c1ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-05-09T08:48:00.987 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 08:48:00 smithi105 podman[64521]: 2024-05-09 08:48:00.618189351 +0000 UTC m=+0.408316493 container start d2eef274aff1aa3f88ce3d8b1294e846edd32ee2c37658d560c0e94db276c1ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-09T08:48:00.987 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 08:48:00 smithi105 ceph-mds[64540]: -- 172.21.15.105:0/488268546 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562257a9ba40 con 0x562257a6f000 2024-05-09T08:48:00.987 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 08:48:00 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h[64536]: starting mds.h at 2024-05-09T08:48:00.987 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 08:48:00 smithi105 bash[64521]: d2eef274aff1aa3f88ce3d8b1294e846edd32ee2c37658d560c0e94db276c1ed 2024-05-09T08:48:00.987 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 08:48:00 smithi105 systemd[1]: Started Ceph mds.h for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:48:01.567 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:01 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/4189240508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:01.568 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:01 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:01.568 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:01 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:01.568 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:01 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:01.568 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:01 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:48:01.568 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:01 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:48:01.568 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:01 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:01.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:01 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4189240508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:01.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:01 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:01.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:01 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:01.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:01 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:01.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:01 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:48:01.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:01 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:48:01.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:01 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:01.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:01 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/4189240508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:01.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:01 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:01.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:01 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:01.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:01 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:01.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:01 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:48:01.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:01 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:48:01.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:01 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:02.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:48:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.370+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/687604515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b28101980 msgr2=0x7f2b28101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.370+0000 7f2b2e6d4640 1 --2- 172.21.15.98:0/687604515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b28101980 0x7f2b28101e00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2b10009900 tx=0x7f2b1002a410 comp rx=0 tx=0).stop 2024-05-09T08:48:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.370+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/687604515 shutdown_connections 2024-05-09T08:48:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.370+0000 7f2b2e6d4640 1 --2- 172.21.15.98:0/687604515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b28102340 0x7f2b28108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.370+0000 7f2b2e6d4640 1 --2- 172.21.15.98:0/687604515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b28101980 0x7f2b28101e00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.370+0000 7f2b2e6d4640 1 --2- 172.21.15.98:0/687604515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28100790 0x7f2b28100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.370+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/687604515 >> 172.21.15.98:0/687604515 conn(0x7f2b280fbd20 msgr2=0x7f2b280fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.370+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/687604515 shutdown_connections 2024-05-09T08:48:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.370+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/687604515 wait complete. 2024-05-09T08:48:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.370+0000 7f2b2e6d4640 1 Processor -- start 2024-05-09T08:48:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.370+0000 7f2b2e6d4640 1 -- start start 2024-05-09T08:48:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b2e6d4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b28100790 0x7f2b28197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b2e6d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b28101980 0x7f2b281981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b2e6d4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28102340 0x7f2b2819f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b2e6d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2810d590 con 0x7f2b28101980 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b2e6d4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2810d410 con 0x7f2b28102340 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b2e6d4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2810d710 con 0x7f2b28100790 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b28101980 0x7f2b281981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b28101980 0x7f2b281981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43324/0 (socket says 172.21.15.98:43324) 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b277fe640 1 -- 172.21.15.98:0/2741188513 learned_addr learned my addr 172.21.15.98:0/2741188513 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b27fff640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b28100790 0x7f2b28197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b2cc71640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28102340 0x7f2b2819f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b277fe640 1 -- 172.21.15.98:0/2741188513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b28100790 msgr2=0x7f2b28197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b277fe640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b28100790 0x7f2b28197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b277fe640 1 -- 172.21.15.98:0/2741188513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28102340 msgr2=0x7f2b2819f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b277fe640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28102340 0x7f2b2819f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b277fe640 1 -- 172.21.15.98:0/2741188513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b2819f890 con 0x7f2b28101980 2024-05-09T08:48:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b2cc71640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28102340 0x7f2b2819f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b277fe640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b28101980 0x7f2b281981c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2b10009230 tx=0x7f2b10004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b257fa640 1 -- 172.21.15.98:0/2741188513 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1003f030 con 0x7f2b28101980 2024-05-09T08:48:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b257fa640 1 -- 172.21.15.98:0/2741188513 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b10003ce0 con 0x7f2b28101980 2024-05-09T08:48:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b257fa640 1 -- 172.21.15.98:0/2741188513 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b10035070 con 0x7f2b28101980 2024-05-09T08:48:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.371+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/2741188513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b2819fb20 con 0x7f2b28101980 2024-05-09T08:48:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.372+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/2741188513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b281a0010 con 0x7f2b28101980 2024-05-09T08:48:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.372+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/2741188513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b28107b50 con 0x7f2b28101980 2024-05-09T08:48:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.374+0000 7f2b257fa640 1 -- 172.21.15.98:0/2741188513 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2b10042030 con 0x7f2b28101980 2024-05-09T08:48:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.374+0000 7f2b257fa640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b140a2b90 0x7f2b140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.374+0000 7f2b27fff640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b140a2b90 0x7f2b140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.374+0000 7f2b257fa640 1 -- 172.21.15.98:0/2741188513 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f2b100ef620 con 0x7f2b28101980 2024-05-09T08:48:02.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.375+0000 7f2b27fff640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b140a2b90 0x7f2b140a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2b28199150 tx=0x7f2b18009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.379+0000 7f2b257fa640 1 -- 172.21.15.98:0/2741188513 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2b100bca20 con 0x7f2b28101980 2024-05-09T08:48:02.551 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:02 smithi142 ceph-mon[34625]: Deploying daemon mds.i on smithi142 2024-05-09T08:48:02.551 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:02 smithi142 ceph-mon[34625]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:02.551 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:02 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] up:boot 2024-05-09T08:48:02.551 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:02 smithi142 ceph-mon[34625]: fsmap 8 up:standby 2024-05-09T08:48:02.551 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:02 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:48:02.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:02 smithi098 ceph-mon[30092]: Deploying daemon mds.i on smithi142 2024-05-09T08:48:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:02 smithi098 ceph-mon[30092]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:02 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] up:boot 2024-05-09T08:48:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:02 smithi098 ceph-mon[30092]: fsmap 8 up:standby 2024-05-09T08:48:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:02 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:48:02.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.590+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/2741188513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b2802d3e0 con 0x7f2b28101980 2024-05-09T08:48:02.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.590+0000 7f2b257fa640 1 -- 172.21.15.98:0/2741188513 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7f2b100bc320 con 0x7f2b28101980 2024-05-09T08:48:02.589 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:02.589 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-09T08:48:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 9 2024-05-09T08:48:02.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.592+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/2741188513 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b140a2b90 msgr2=0x7f2b140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.592+0000 7f2b2e6d4640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b140a2b90 0x7f2b140a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2b28199150 tx=0x7f2b18009110 comp rx=0 tx=0).stop 2024-05-09T08:48:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.592+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/2741188513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b28101980 msgr2=0x7f2b281981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.592+0000 7f2b2e6d4640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b28101980 0x7f2b281981c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2b10009230 tx=0x7f2b10004360 comp rx=0 tx=0).stop 2024-05-09T08:48:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.592+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/2741188513 shutdown_connections 2024-05-09T08:48:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.592+0000 7f2b2e6d4640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28102340 0x7f2b2819f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.592+0000 7f2b2e6d4640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b140a2b90 0x7f2b140a5050 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.592+0000 7f2b2e6d4640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b28101980 0x7f2b281981c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.592+0000 7f2b2e6d4640 1 --2- 172.21.15.98:0/2741188513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b28100790 0x7f2b28197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.592+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/2741188513 >> 172.21.15.98:0/2741188513 conn(0x7f2b280fbd20 msgr2=0x7f2b28104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.592+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/2741188513 shutdown_connections 2024-05-09T08:48:02.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:02.592+0000 7f2b2e6d4640 1 -- 172.21.15.98:0/2741188513 wait complete. 2024-05-09T08:48:02.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:02 smithi105 ceph-mon[34498]: Deploying daemon mds.i on smithi142 2024-05-09T08:48:02.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:02 smithi105 ceph-mon[34498]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:02.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:02 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] up:boot 2024-05-09T08:48:02.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:02 smithi105 ceph-mon[34498]: fsmap 8 up:standby 2024-05-09T08:48:02.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:02 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:48:03.257 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 08:48:03 smithi142 podman[64892]: 2024-05-09 08:48:03.163123339 +0000 UTC m=+0.117827821 container create 8d37654b0fb09f1bbdce58a9804c3af29b9b8a25172378bf87fcc5b58093e68c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-i, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, RELEASE=squid-96805dc) 2024-05-09T08:48:03.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:03 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2741188513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:03 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2741188513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:03.658 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 08:48:03 smithi142 podman[64892]: 2024-05-09 08:48:03.399433046 +0000 UTC m=+0.354137471 container init 8d37654b0fb09f1bbdce58a9804c3af29b9b8a25172378bf87fcc5b58093e68c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-i, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0) 2024-05-09T08:48:03.658 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 08:48:03 smithi142 podman[64892]: 2024-05-09 08:48:03.402096517 +0000 UTC m=+0.356800948 container start 8d37654b0fb09f1bbdce58a9804c3af29b9b8a25172378bf87fcc5b58093e68c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-09T08:48:03.659 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 08:48:03 smithi142 ceph-mds[64910]: -- 172.21.15.142:0/166394222 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d163e07a40 con 0x55d163dda800 2024-05-09T08:48:03.659 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 08:48:03 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-i[64906]: starting mds.i at 2024-05-09T08:48:03.659 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 08:48:03 smithi142 bash[64892]: 8d37654b0fb09f1bbdce58a9804c3af29b9b8a25172378bf87fcc5b58093e68c 2024-05-09T08:48:03.659 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 08:48:03 smithi142 systemd[1]: Started Ceph mds.i for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:48:03.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:03 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2741188513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:04.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:04 smithi098 ceph-mon[30092]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:04.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:04.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:04.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:04.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:48:04.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:48:04.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:04 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:04.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:48:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:04 smithi142 ceph-mon[34625]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:48:04.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:48:04.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:04 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:04.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.711+0000 7f5ad02df640 1 -- 172.21.15.98:0/2022794148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8100770 msgr2=0x7f5ac8100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:04.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.711+0000 7f5ad02df640 1 --2- 172.21.15.98:0/2022794148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8100770 0x7f5ac8100b90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009ca0 tx=0x7f5ac402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:04.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.711+0000 7f5ad02df640 1 -- 172.21.15.98:0/2022794148 shutdown_connections 2024-05-09T08:48:04.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.711+0000 7f5ad02df640 1 --2- 172.21.15.98:0/2022794148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ac8102320 0x7f5ac8108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:04.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.711+0000 7f5ad02df640 1 --2- 172.21.15.98:0/2022794148 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac8101960 0x7f5ac8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:04.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.711+0000 7f5ad02df640 1 --2- 172.21.15.98:0/2022794148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8100770 0x7f5ac8100b90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:04.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.711+0000 7f5ad02df640 1 -- 172.21.15.98:0/2022794148 >> 172.21.15.98:0/2022794148 conn(0x7f5ac80fbce0 msgr2=0x7f5ac80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:04.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.711+0000 7f5ad02df640 1 -- 172.21.15.98:0/2022794148 shutdown_connections 2024-05-09T08:48:04.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.711+0000 7f5ad02df640 1 -- 172.21.15.98:0/2022794148 wait complete. 2024-05-09T08:48:04.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.711+0000 7f5ad02df640 1 Processor -- start 2024-05-09T08:48:04.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5ad02df640 1 -- start start 2024-05-09T08:48:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5ad02df640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac8100770 0x7f5ac8196ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5ad02df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8101960 0x7f5ac8197020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5ad02df640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ac8102320 0x7f5ac819e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5ad02df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac810c440 con 0x7f5ac8101960 2024-05-09T08:48:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5ad02df640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac810c2c0 con 0x7f5ac8100770 2024-05-09T08:48:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5ad02df640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac810c5c0 con 0x7f5ac8102320 2024-05-09T08:48:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5acd87a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8101960 0x7f5ac8197020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5acd87a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8101960 0x7f5ac8197020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43344/0 (socket says 172.21.15.98:43344) 2024-05-09T08:48:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5acd87a640 1 -- 172.21.15.98:0/2910470033 learned_addr learned my addr 172.21.15.98:0/2910470033 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5ace07b640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac8100770 0x7f5ac8196ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5acd87a640 1 -- 172.21.15.98:0/2910470033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ac8102320 msgr2=0x7f5ac819e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5ace87c640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ac8102320 0x7f5ac819e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5acd87a640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ac8102320 0x7f5ac819e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5acd87a640 1 -- 172.21.15.98:0/2910470033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac8100770 msgr2=0x7f5ac8196ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:04.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5acd87a640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac8100770 0x7f5ac8196ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:04.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5acd87a640 1 -- 172.21.15.98:0/2910470033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac819e700 con 0x7f5ac8101960 2024-05-09T08:48:04.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5ace07b640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac8100770 0x7f5ac8196ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:04.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5acd87a640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8101960 0x7f5ac8197020 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800c890 tx=0x7f5ab800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:04.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5abf7fe640 1 -- 172.21.15.98:0/2910470033 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8004220 con 0x7f5ac8101960 2024-05-09T08:48:04.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5ad02df640 1 -- 172.21.15.98:0/2910470033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac819e9f0 con 0x7f5ac8101960 2024-05-09T08:48:04.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5abf7fe640 1 -- 172.21.15.98:0/2910470033 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ab80043c0 con 0x7f5ac8101960 2024-05-09T08:48:04.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5abf7fe640 1 -- 172.21.15.98:0/2910470033 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab80049e0 con 0x7f5ac8101960 2024-05-09T08:48:04.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.712+0000 7f5ad02df640 1 -- 172.21.15.98:0/2910470033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac819efc0 con 0x7f5ac8101960 2024-05-09T08:48:04.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.713+0000 7f5ad02df640 1 -- 172.21.15.98:0/2910470033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac8107b10 con 0x7f5ac8101960 2024-05-09T08:48:04.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.715+0000 7f5abf7fe640 1 -- 172.21.15.98:0/2910470033 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5ab80079b0 con 0x7f5ac8101960 2024-05-09T08:48:04.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.715+0000 7f5abf7fe640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5ab40a2b90 0x7f5ab40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:04.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.715+0000 7f5ace07b640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5ab40a2b90 0x7f5ab40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:04.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.715+0000 7f5abf7fe640 1 -- 172.21.15.98:0/2910470033 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f5ab80cece0 con 0x7f5ac8101960 2024-05-09T08:48:04.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.716+0000 7f5ace07b640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5ab40a2b90 0x7f5ab40a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4006010 tx=0x7f5ac4000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:04.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.719+0000 7f5abf7fe640 1 -- 172.21.15.98:0/2910470033 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5ab8014030 con 0x7f5ac8101960 2024-05-09T08:48:04.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:04 smithi105 ceph-mon[34498]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:48:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:48:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:04 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.940+0000 7f5ad02df640 1 -- 172.21.15.98:0/2910470033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ac802d3f0 con 0x7f5ac8101960 2024-05-09T08:48:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.941+0000 7f5abf7fe640 1 -- 172.21.15.98:0/2910470033 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7f5ab809c0e0 con 0x7f5ac8101960 2024-05-09T08:48:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:04.940 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-09T08:48:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 10 2024-05-09T08:48:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.943+0000 7f5ad02df640 1 -- 172.21.15.98:0/2910470033 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5ab40a2b90 msgr2=0x7f5ab40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.943+0000 7f5ad02df640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5ab40a2b90 0x7f5ab40a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4006010 tx=0x7f5ac4000f40 comp rx=0 tx=0).stop 2024-05-09T08:48:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.943+0000 7f5ad02df640 1 -- 172.21.15.98:0/2910470033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8101960 msgr2=0x7f5ac8197020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.943+0000 7f5ad02df640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8101960 0x7f5ac8197020 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800c890 tx=0x7f5ab800cd50 comp rx=0 tx=0).stop 2024-05-09T08:48:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.943+0000 7f5ad02df640 1 -- 172.21.15.98:0/2910470033 shutdown_connections 2024-05-09T08:48:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.943+0000 7f5ad02df640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ac8102320 0x7f5ac819e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.943+0000 7f5ad02df640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5ab40a2b90 0x7f5ab40a5050 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.943+0000 7f5ad02df640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8101960 0x7f5ac8197020 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.943+0000 7f5ad02df640 1 --2- 172.21.15.98:0/2910470033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac8100770 0x7f5ac8196ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.943+0000 7f5ad02df640 1 -- 172.21.15.98:0/2910470033 >> 172.21.15.98:0/2910470033 conn(0x7f5ac80fbce0 msgr2=0x7f5ac8104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.943+0000 7f5ad02df640 1 -- 172.21.15.98:0/2910470033 shutdown_connections 2024-05-09T08:48:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:04.943+0000 7f5ad02df640 1 -- 172.21.15.98:0/2910470033 wait complete. 2024-05-09T08:48:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:05 smithi098 ceph-mon[30092]: Deploying daemon mds.g on smithi098 2024-05-09T08:48:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:05 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] up:boot 2024-05-09T08:48:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:05 smithi098 ceph-mon[30092]: fsmap 9 up:standby 2024-05-09T08:48:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:05 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:48:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:05 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2910470033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:05 smithi142 ceph-mon[34625]: Deploying daemon mds.g on smithi098 2024-05-09T08:48:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:05 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] up:boot 2024-05-09T08:48:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:05 smithi142 ceph-mon[34625]: fsmap 9 up:standby 2024-05-09T08:48:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:05 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:48:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:05 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2910470033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:05.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:05 smithi105 ceph-mon[34498]: Deploying daemon mds.g on smithi098 2024-05-09T08:48:05.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:05 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] up:boot 2024-05-09T08:48:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:05 smithi105 ceph-mon[34498]: fsmap 9 up:standby 2024-05-09T08:48:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:05 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:48:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:05 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2910470033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:06.021 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 08:48:05 smithi098 systemd[1]: Starting Ceph mds.g for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:48:06.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:06 smithi098 ceph-mon[30092]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:06.293 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 08:48:06 smithi098 podman[71610]: 2024-05-09 08:48:05.925792421 +0000 UTC m=+0.018164081 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:48:06.293 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 08:48:06 smithi098 podman[71610]: 2024-05-09 08:48:06.024941739 +0000 UTC m=+0.117313459 container create a66220490dfca0510895d4bdf85a7867bc74cc1d1928d278abcb78ab3ec9265a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-g, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:48:06.293 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 08:48:06 smithi098 podman[71610]: 2024-05-09 08:48:06.294535854 +0000 UTC m=+0.386907498 container init a66220490dfca0510895d4bdf85a7867bc74cc1d1928d278abcb78ab3ec9265a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-g, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240507) 2024-05-09T08:48:06.588 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 08:48:06 smithi098 podman[71610]: 2024-05-09 08:48:06.297861077 +0000 UTC m=+0.390232727 container start a66220490dfca0510895d4bdf85a7867bc74cc1d1928d278abcb78ab3ec9265a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:48:06.588 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 08:48:06 smithi098 ceph-mds[71629]: -- 172.21.15.98:0/3042233697 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55778e509a40 con 0x55778e4dc800 2024-05-09T08:48:06.588 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 08:48:06 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-g[71625]: starting mds.g at 2024-05-09T08:48:06.588 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 08:48:06 smithi098 bash[71610]: a66220490dfca0510895d4bdf85a7867bc74cc1d1928d278abcb78ab3ec9265a 2024-05-09T08:48:06.588 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 08:48:06 smithi098 systemd[1]: Started Ceph mds.g for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:48:06.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:06 smithi142 ceph-mon[34625]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:06.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:06 smithi105 ceph-mon[34498]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:06.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:48:07.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.064+0000 7ff733038640 1 -- 172.21.15.98:0/1909609558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff72c1075c0 msgr2=0x7ff72c1099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:07.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.064+0000 7ff733038640 1 --2- 172.21.15.98:0/1909609558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff72c1075c0 0x7ff72c1099b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff724009ca0 tx=0x7ff72402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:07.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.064+0000 7ff733038640 1 -- 172.21.15.98:0/1909609558 shutdown_connections 2024-05-09T08:48:07.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.064+0000 7ff733038640 1 --2- 172.21.15.98:0/1909609558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff72c1075c0 0x7ff72c1099b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:07.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.064+0000 7ff733038640 1 --2- 172.21.15.98:0/1909609558 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff72c104c90 0x7ff72c107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:07.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.064+0000 7ff733038640 1 --2- 172.21.15.98:0/1909609558 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff72c102360 0x7ff72c104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:07.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.064+0000 7ff733038640 1 -- 172.21.15.98:0/1909609558 >> 172.21.15.98:0/1909609558 conn(0x7ff72c0fbd10 msgr2=0x7ff72c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:07.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.064+0000 7ff733038640 1 -- 172.21.15.98:0/1909609558 shutdown_connections 2024-05-09T08:48:07.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.064+0000 7ff733038640 1 -- 172.21.15.98:0/1909609558 wait complete. 2024-05-09T08:48:07.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.064+0000 7ff733038640 1 Processor -- start 2024-05-09T08:48:07.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff733038640 1 -- start start 2024-05-09T08:48:07.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff733038640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff72c102360 0x7ff72c197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:07.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff733038640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff72c104c90 0x7ff72c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:07.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff733038640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff72c1075c0 0x7ff72c19f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:07.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff733038640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff72c10d520 con 0x7ff72c1075c0 2024-05-09T08:48:07.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff733038640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff72c10d3a0 con 0x7ff72c104c90 2024-05-09T08:48:07.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff733038640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff72c10d6a0 con 0x7ff72c102360 2024-05-09T08:48:07.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff7315d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff72c1075c0 0x7ff72c19f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:07.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff7315d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff72c1075c0 0x7ff72c19f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34542/0 (socket says 172.21.15.98:34542) 2024-05-09T08:48:07.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff7315d5640 1 -- 172.21.15.98:0/107519657 learned_addr learned my addr 172.21.15.98:0/107519657 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:07.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff730dd4640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff72c102360 0x7ff72c197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:07.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff72bfff640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff72c104c90 0x7ff72c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:07.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff7315d5640 1 -- 172.21.15.98:0/107519657 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff72c102360 msgr2=0x7ff72c197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:07.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff7315d5640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff72c102360 0x7ff72c197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:07.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff7315d5640 1 -- 172.21.15.98:0/107519657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff72c104c90 msgr2=0x7ff72c1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:07.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff7315d5640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff72c104c90 0x7ff72c1980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:07.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff7315d5640 1 -- 172.21.15.98:0/107519657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff72c19f7a0 con 0x7ff72c1075c0 2024-05-09T08:48:07.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff730dd4640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff72c102360 0x7ff72c197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:07.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff72bfff640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff72c104c90 0x7ff72c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:07.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff7315d5640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff72c1075c0 0x7ff72c19f130 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff724031ae0 tx=0x7ff724031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:07.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff729ffb640 1 -- 172.21.15.98:0/107519657 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff724003c30 con 0x7ff72c1075c0 2024-05-09T08:48:07.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff729ffb640 1 -- 172.21.15.98:0/107519657 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff724003dd0 con 0x7ff72c1075c0 2024-05-09T08:48:07.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff733038640 1 -- 172.21.15.98:0/107519657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff72c19fa30 con 0x7ff72c1075c0 2024-05-09T08:48:07.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.065+0000 7ff729ffb640 1 -- 172.21.15.98:0/107519657 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72402fcb0 con 0x7ff72c1075c0 2024-05-09T08:48:07.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.066+0000 7ff733038640 1 -- 172.21.15.98:0/107519657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff72c19ffd0 con 0x7ff72c1075c0 2024-05-09T08:48:07.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.067+0000 7ff733038640 1 -- 172.21.15.98:0/107519657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff72c0fd800 con 0x7ff72c1075c0 2024-05-09T08:48:07.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.068+0000 7ff729ffb640 1 -- 172.21.15.98:0/107519657 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff724004050 con 0x7ff72c1075c0 2024-05-09T08:48:07.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.068+0000 7ff729ffb640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff71c0a2c10 0x7ff71c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:07.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.068+0000 7ff729ffb640 1 -- 172.21.15.98:0/107519657 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff7240efe90 con 0x7ff72c1075c0 2024-05-09T08:48:07.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.068+0000 7ff730dd4640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff71c0a2c10 0x7ff71c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:07.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.069+0000 7ff730dd4640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff71c0a2c10 0x7ff71c0a50d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff7200058c0 tx=0x7ff720005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:07.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.073+0000 7ff729ffb640 1 -- 172.21.15.98:0/107519657 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff7240bd290 con 0x7ff72c1075c0 2024-05-09T08:48:07.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.300+0000 7ff733038640 1 -- 172.21.15.98:0/107519657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff72c02d1c0 con 0x7ff72c1075c0 2024-05-09T08:48:07.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.301+0000 7ff729ffb640 1 -- 172.21.15.98:0/107519657 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7ff7240bcb90 con 0x7ff72c1075c0 2024-05-09T08:48:07.302 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:07.303 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-09T08:48:07.303 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 11 2024-05-09T08:48:07.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.303+0000 7ff733038640 1 -- 172.21.15.98:0/107519657 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff71c0a2c10 msgr2=0x7ff71c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:07.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.303+0000 7ff733038640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff71c0a2c10 0x7ff71c0a50d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff7200058c0 tx=0x7ff720005850 comp rx=0 tx=0).stop 2024-05-09T08:48:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.303+0000 7ff733038640 1 -- 172.21.15.98:0/107519657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff72c1075c0 msgr2=0x7ff72c19f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.303+0000 7ff733038640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff72c1075c0 0x7ff72c19f130 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff724031ae0 tx=0x7ff724031ee0 comp rx=0 tx=0).stop 2024-05-09T08:48:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.303+0000 7ff733038640 1 -- 172.21.15.98:0/107519657 shutdown_connections 2024-05-09T08:48:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.303+0000 7ff733038640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff71c0a2c10 0x7ff71c0a50d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.303+0000 7ff733038640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff72c1075c0 0x7ff72c19f130 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.303+0000 7ff733038640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff72c104c90 0x7ff72c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.303+0000 7ff733038640 1 --2- 172.21.15.98:0/107519657 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff72c102360 0x7ff72c197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.303+0000 7ff733038640 1 -- 172.21.15.98:0/107519657 >> 172.21.15.98:0/107519657 conn(0x7ff72c0fbd10 msgr2=0x7ff72c102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.303+0000 7ff733038640 1 -- 172.21.15.98:0/107519657 shutdown_connections 2024-05-09T08:48:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:07.303+0000 7ff733038640 1 -- 172.21.15.98:0/107519657 wait complete. 2024-05-09T08:48:07.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:07 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:07.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:07 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:07.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:07 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:07.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:07 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:48:07.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:07 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:48:07.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:07 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:07.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:07 smithi098 ceph-mon[30092]: Deploying daemon mds.a on smithi098 2024-05-09T08:48:07.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:07 smithi098 ceph-mon[30092]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:07.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:07 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.98:6838/3981776050,v1:172.21.15.98:6839/3981776050] up:boot 2024-05-09T08:48:07.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:07 smithi098 ceph-mon[30092]: fsmap 10 up:standby 2024-05-09T08:48:07.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:07 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:48:07.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:07 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/107519657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:07.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:07 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:07.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:07 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:07.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:07 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:07.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:07 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:48:07.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:07 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:48:07.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:07 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:07.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:07 smithi105 ceph-mon[34498]: Deploying daemon mds.a on smithi098 2024-05-09T08:48:07.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:07 smithi105 ceph-mon[34498]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:07.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:07 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.98:6838/3981776050,v1:172.21.15.98:6839/3981776050] up:boot 2024-05-09T08:48:07.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:07 smithi105 ceph-mon[34498]: fsmap 10 up:standby 2024-05-09T08:48:07.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:07 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:48:07.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:07 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/107519657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:07.833 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:07 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:07.833 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:07 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:07.833 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:07 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:07.833 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:07 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:48:07.834 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:07 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:48:07.834 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:07 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:07.834 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:07 smithi142 ceph-mon[34625]: Deploying daemon mds.a on smithi098 2024-05-09T08:48:07.834 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:07 smithi142 ceph-mon[34625]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:07.834 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:07 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.98:6838/3981776050,v1:172.21.15.98:6839/3981776050] up:boot 2024-05-09T08:48:07.834 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:07 smithi142 ceph-mon[34625]: fsmap 10 up:standby 2024-05-09T08:48:07.834 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:07 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:48:07.834 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:07 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/107519657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:08.838 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 08:48:08 smithi098 systemd[1]: Starting Ceph mds.a for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:48:09.128 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 08:48:08 smithi098 podman[71972]: 2024-05-09 08:48:08.755974099 +0000 UTC m=+0.018248450 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:48:09.128 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 08:48:08 smithi098 podman[71972]: 2024-05-09 08:48:08.885234735 +0000 UTC m=+0.147509070 container create b659e2da800b915f5d8046e515147a8b6527b2367f9522604c863502873ea1fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:48:09.128 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 08:48:09 smithi098 podman[71972]: 2024-05-09 08:48:09.129873588 +0000 UTC m=+0.392147930 container init b659e2da800b915f5d8046e515147a8b6527b2367f9522604c863502873ea1fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-09T08:48:09.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:48:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3405812362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1adc1009d0 msgr2=0x7f1adc100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 --2- 172.21.15.98:0/3405812362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1adc1009d0 0x7f1adc100e50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009900 tx=0x7f1ac802a400 comp rx=0 tx=0).stop 2024-05-09T08:48:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3405812362 shutdown_connections 2024-05-09T08:48:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 --2- 172.21.15.98:0/3405812362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1adc101390 0x7f1adc108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 --2- 172.21.15.98:0/3405812362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1adc1009d0 0x7f1adc100e50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 --2- 172.21.15.98:0/3405812362 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1adc101e10 0x7f1adc102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3405812362 >> 172.21.15.98:0/3405812362 conn(0x7f1adc0fb4a0 msgr2=0x7f1adc0fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3405812362 shutdown_connections 2024-05-09T08:48:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3405812362 wait complete. 2024-05-09T08:48:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 Processor -- start 2024-05-09T08:48:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 -- start start 2024-05-09T08:48:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1adc1009d0 0x7f1adc100450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1adc101390 0x7f1adc19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1adc101e10 0x7f1adc19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc10c420 con 0x7f1adc1009d0 2024-05-09T08:48:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc10c2a0 con 0x7f1adc101e10 2024-05-09T08:48:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae2c76640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc10c5a0 con 0x7f1adc101390 2024-05-09T08:48:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.421+0000 7f1ae0a12640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1adc1009d0 0x7f1adc100450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1ae0a12640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1adc1009d0 0x7f1adc100450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34572/0 (socket says 172.21.15.98:34572) 2024-05-09T08:48:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1ae0a12640 1 -- 172.21.15.98:0/3845622048 learned_addr learned my addr 172.21.15.98:0/3845622048 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1adbfff640 1 --2- 172.21.15.98:0/3845622048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1adc101390 0x7f1adc19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1ae1213640 1 --2- 172.21.15.98:0/3845622048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1adc101e10 0x7f1adc19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1ae0a12640 1 -- 172.21.15.98:0/3845622048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1adc101390 msgr2=0x7f1adc19c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1ae0a12640 1 --2- 172.21.15.98:0/3845622048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1adc101390 0x7f1adc19c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1ae0a12640 1 -- 172.21.15.98:0/3845622048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1adc101e10 msgr2=0x7f1adc19ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1ae0a12640 1 --2- 172.21.15.98:0/3845622048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1adc101e10 0x7f1adc19ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1ae0a12640 1 -- 172.21.15.98:0/3845622048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1adc19f7d0 con 0x7f1adc1009d0 2024-05-09T08:48:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1ae0a12640 1 --2- 172.21.15.98:0/3845622048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1adc1009d0 0x7f1adc100450 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000c970 tx=0x7f1ad000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1ad9ffb640 1 -- 172.21.15.98:0/3845622048 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0007bb0 con 0x7f1adc1009d0 2024-05-09T08:48:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1ad9ffb640 1 -- 172.21.15.98:0/3845622048 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad0007d50 con 0x7f1adc1009d0 2024-05-09T08:48:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3845622048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1adc19fac0 con 0x7f1adc1009d0 2024-05-09T08:48:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1ad9ffb640 1 -- 172.21.15.98:0/3845622048 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad000f450 con 0x7f1adc1009d0 2024-05-09T08:48:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.422+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3845622048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1adc19ff90 con 0x7f1adc1009d0 2024-05-09T08:48:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.423+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3845622048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1adc107b90 con 0x7f1adc1009d0 2024-05-09T08:48:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.424+0000 7f1ad9ffb640 1 -- 172.21.15.98:0/3845622048 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1ad0005020 con 0x7f1adc1009d0 2024-05-09T08:48:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.425+0000 7f1ad9ffb640 1 --2- 172.21.15.98:0/3845622048 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1acc0a2b90 0x7f1acc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.425+0000 7f1adbfff640 1 --2- 172.21.15.98:0/3845622048 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1acc0a2b90 0x7f1acc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.425+0000 7f1ad9ffb640 1 -- 172.21.15.98:0/3845622048 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f1ad00cea40 con 0x7f1adc1009d0 2024-05-09T08:48:09.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.425+0000 7f1adbfff640 1 --2- 172.21.15.98:0/3845622048 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1acc0a2b90 0x7f1acc0a5050 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8006010 tx=0x7f1ac8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:09.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.429+0000 7f1ad9ffb640 1 -- 172.21.15.98:0/3845622048 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1ad0014030 con 0x7f1adc1009d0 2024-05-09T08:48:09.588 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 08:48:09 smithi098 podman[71972]: 2024-05-09 08:48:09.132746462 +0000 UTC m=+0.395020794 container start b659e2da800b915f5d8046e515147a8b6527b2367f9522604c863502873ea1fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-09T08:48:09.588 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 08:48:09 smithi098 ceph-mds[71992]: -- 172.21.15.98:0/618278034 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dfcdc89a40 con 0x55dfcdc5c800 2024-05-09T08:48:09.588 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 08:48:09 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-a[71988]: starting mds.a at 2024-05-09T08:48:09.588 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 08:48:09 smithi098 bash[71972]: b659e2da800b915f5d8046e515147a8b6527b2367f9522604c863502873ea1fe 2024-05-09T08:48:09.588 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 08:48:09 smithi098 systemd[1]: Started Ceph mds.a for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:48:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.641+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3845622048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1adc02d3e0 con 0x7f1adc1009d0 2024-05-09T08:48:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.641+0000 7f1ad9ffb640 1 -- 172.21.15.98:0/3845622048 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9659 (secure 0 0 0) 0x7f1ad009be40 con 0x7f1adc1009d0 2024-05-09T08:48:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:09.641 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-09T08:48:09.641 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 12 2024-05-09T08:48:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.643+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3845622048 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1acc0a2b90 msgr2=0x7f1acc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.643+0000 7f1ae2c76640 1 --2- 172.21.15.98:0/3845622048 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1acc0a2b90 0x7f1acc0a5050 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8006010 tx=0x7f1ac8000bc0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.643+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3845622048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1adc1009d0 msgr2=0x7f1adc100450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.643+0000 7f1ae2c76640 1 --2- 172.21.15.98:0/3845622048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1adc1009d0 0x7f1adc100450 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000c970 tx=0x7f1ad000ce30 comp rx=0 tx=0).stop 2024-05-09T08:48:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.644+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3845622048 shutdown_connections 2024-05-09T08:48:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.644+0000 7f1ae2c76640 1 --2- 172.21.15.98:0/3845622048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1adc101e10 0x7f1adc19ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.644+0000 7f1ae2c76640 1 --2- 172.21.15.98:0/3845622048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1adc101390 0x7f1adc19c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.644+0000 7f1ae2c76640 1 --2- 172.21.15.98:0/3845622048 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1acc0a2b90 0x7f1acc0a5050 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.644+0000 7f1ae2c76640 1 --2- 172.21.15.98:0/3845622048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1adc1009d0 0x7f1adc100450 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.644+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3845622048 >> 172.21.15.98:0/3845622048 conn(0x7f1adc0fb4a0 msgr2=0x7f1adc104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.644+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3845622048 shutdown_connections 2024-05-09T08:48:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:09.644+0000 7f1ae2c76640 1 -- 172.21.15.98:0/3845622048 wait complete. 2024-05-09T08:48:10.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:10 smithi105 ceph-mon[34498]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:10.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:10 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.98:6840/2714016730,v1:172.21.15.98:6841/2714016730] up:boot 2024-05-09T08:48:10.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:10 smithi105 ceph-mon[34498]: fsmap 11 up:standby 2024-05-09T08:48:10.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:48:10.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:10.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:10.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:10.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:48:10.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:48:10.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:10 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:10.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:10 smithi105 ceph-mon[34498]: Deploying daemon mds.b on smithi105 2024-05-09T08:48:10.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:10 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3845622048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:10.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:10 smithi098 ceph-mon[30092]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:10.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:10 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.98:6840/2714016730,v1:172.21.15.98:6841/2714016730] up:boot 2024-05-09T08:48:10.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:10 smithi098 ceph-mon[30092]: fsmap 11 up:standby 2024-05-09T08:48:10.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:48:10.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:10.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:10.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:10.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:48:10.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:48:10.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:10 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:10.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:10 smithi098 ceph-mon[30092]: Deploying daemon mds.b on smithi105 2024-05-09T08:48:10.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:10 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3845622048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:10.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:10 smithi142 ceph-mon[34625]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:10.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:10 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.98:6840/2714016730,v1:172.21.15.98:6841/2714016730] up:boot 2024-05-09T08:48:10.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:10 smithi142 ceph-mon[34625]: fsmap 11 up:standby 2024-05-09T08:48:10.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:48:10.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:10.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:10.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:10.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:48:10.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:48:10.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:10 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:10.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:10 smithi142 ceph-mon[34625]: Deploying daemon mds.b on smithi105 2024-05-09T08:48:10.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:10 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3845622048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:11.486 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 08:48:11 smithi105 systemd[1]: Starting Ceph mds.b for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:48:11.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:48:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.769+0000 7f1cf31d5640 1 -- 172.21.15.98:0/3025814672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cec100180 msgr2=0x7f1cec1005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.769+0000 7f1cf31d5640 1 --2- 172.21.15.98:0/3025814672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cec100180 0x7f1cec1005a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009900 tx=0x7f1cdc02a4a0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.769+0000 7f1cf31d5640 1 -- 172.21.15.98:0/3025814672 shutdown_connections 2024-05-09T08:48:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.769+0000 7f1cf31d5640 1 --2- 172.21.15.98:0/3025814672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1cec101480 0x7f1cec101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.769+0000 7f1cf31d5640 1 --2- 172.21.15.98:0/3025814672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cec100ae0 0x7f1cec100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.769+0000 7f1cf31d5640 1 --2- 172.21.15.98:0/3025814672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cec100180 0x7f1cec1005a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.769+0000 7f1cf31d5640 1 -- 172.21.15.98:0/3025814672 >> 172.21.15.98:0/3025814672 conn(0x7f1cec0fbd20 msgr2=0x7f1cec0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.769+0000 7f1cf31d5640 1 -- 172.21.15.98:0/3025814672 shutdown_connections 2024-05-09T08:48:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.769+0000 7f1cf31d5640 1 -- 172.21.15.98:0/3025814672 wait complete. 2024-05-09T08:48:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.770+0000 7f1cf31d5640 1 Processor -- start 2024-05-09T08:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.770+0000 7f1cf31d5640 1 -- start start 2024-05-09T08:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.770+0000 7f1cf31d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cec100180 0x7f1cec199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.770+0000 7f1cf31d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cec100ae0 0x7f1cec19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.770+0000 7f1cf31d5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1cec101480 0x7f1cec1a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.770+0000 7f1cf31d5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec10f850 con 0x7f1cec100ae0 2024-05-09T08:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.770+0000 7f1cf31d5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec10f6d0 con 0x7f1cec100180 2024-05-09T08:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.770+0000 7f1cf31d5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec10f9d0 con 0x7f1cec101480 2024-05-09T08:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.770+0000 7f1cebfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cec100ae0 0x7f1cec19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.770+0000 7f1cebfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cec100ae0 0x7f1cec19a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34594/0 (socket says 172.21.15.98:34594) 2024-05-09T08:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.770+0000 7f1cebfff640 1 -- 172.21.15.98:0/1203759674 learned_addr learned my addr 172.21.15.98:0/1203759674 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.770+0000 7f1cf1772640 1 --2- 172.21.15.98:0/1203759674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1cec101480 0x7f1cec1a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.771+0000 7f1cebfff640 1 -- 172.21.15.98:0/1203759674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1cec101480 msgr2=0x7f1cec1a14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.771+0000 7f1cebfff640 1 --2- 172.21.15.98:0/1203759674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1cec101480 0x7f1cec1a14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.771+0000 7f1cebfff640 1 -- 172.21.15.98:0/1203759674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cec100180 msgr2=0x7f1cec199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.771+0000 7f1cebfff640 1 --2- 172.21.15.98:0/1203759674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cec100180 0x7f1cec199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.771+0000 7f1cebfff640 1 -- 172.21.15.98:0/1203759674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cec1a1b40 con 0x7f1cec100ae0 2024-05-09T08:48:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.771+0000 7f1cf1772640 1 --2- 172.21.15.98:0/1203759674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1cec101480 0x7f1cec1a14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.771+0000 7f1cebfff640 1 --2- 172.21.15.98:0/1203759674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cec100ae0 0x7f1cec19a460 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000b450 tx=0x7f1ce000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.771+0000 7f1ce9ffb640 1 -- 172.21.15.98:0/1203759674 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0015020 con 0x7f1cec100ae0 2024-05-09T08:48:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.771+0000 7f1ce9ffb640 1 -- 172.21.15.98:0/1203759674 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ce0011d70 con 0x7f1cec100ae0 2024-05-09T08:48:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.771+0000 7f1cf31d5640 1 -- 172.21.15.98:0/1203759674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cec1a1e30 con 0x7f1cec100ae0 2024-05-09T08:48:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.771+0000 7f1ce9ffb640 1 -- 172.21.15.98:0/1203759674 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce000f340 con 0x7f1cec100ae0 2024-05-09T08:48:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.771+0000 7f1cf31d5640 1 -- 172.21.15.98:0/1203759674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cec1a2310 con 0x7f1cec100ae0 2024-05-09T08:48:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.772+0000 7f1cf31d5640 1 -- 172.21.15.98:0/1203759674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cec10b0c0 con 0x7f1cec100ae0 2024-05-09T08:48:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.773+0000 7f1ce9ffb640 1 -- 172.21.15.98:0/1203759674 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1ce0018020 con 0x7f1cec100ae0 2024-05-09T08:48:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.774+0000 7f1ce9ffb640 1 --2- 172.21.15.98:0/1203759674 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1cd40a2b90 0x7f1cd40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.774+0000 7f1cf0f71640 1 --2- 172.21.15.98:0/1203759674 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1cd40a2b90 0x7f1cd40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.774+0000 7f1ce9ffb640 1 -- 172.21.15.98:0/1203759674 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f1ce00ce700 con 0x7f1cec100ae0 2024-05-09T08:48:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.774+0000 7f1cf0f71640 1 --2- 172.21.15.98:0/1203759674 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1cd40a2b90 0x7f1cd40a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0096c0 tx=0x7f1cdc031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:11.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.778+0000 7f1ce9ffb640 1 -- 172.21.15.98:0/1203759674 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1ce001f020 con 0x7f1cec100ae0 2024-05-09T08:48:11.869 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 08:48:11 smithi105 podman[64828]: 2024-05-09 08:48:11.512836985 +0000 UTC m=+0.018340569 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:48:11.869 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 08:48:11 smithi105 podman[64828]: 2024-05-09 08:48:11.617567195 +0000 UTC m=+0.123070757 container create 1a83673e3f3356270df53c9ac846e49a5ef15ca4fa84cfd71aaf73537586d233 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-b, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD) 2024-05-09T08:48:11.869 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 08:48:11 smithi105 podman[64828]: 2024-05-09 08:48:11.870668729 +0000 UTC m=+0.376172290 container init 1a83673e3f3356270df53c9ac846e49a5ef15ca4fa84cfd71aaf73537586d233 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD) 2024-05-09T08:48:11.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.993+0000 7f1cf31d5640 1 -- 172.21.15.98:0/1203759674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cec02d1c0 con 0x7f1cec100ae0 2024-05-09T08:48:11.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.994+0000 7f1ce9ffb640 1 -- 172.21.15.98:0/1203759674 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9659 (secure 0 0 0) 0x7f1ce000d040 con 0x7f1cec100ae0 2024-05-09T08:48:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:11.993 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-09T08:48:11.993 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 12 2024-05-09T08:48:11.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.996+0000 7f1cf31d5640 1 -- 172.21.15.98:0/1203759674 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1cd40a2b90 msgr2=0x7f1cd40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.996+0000 7f1cf31d5640 1 --2- 172.21.15.98:0/1203759674 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1cd40a2b90 0x7f1cd40a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0096c0 tx=0x7f1cdc031400 comp rx=0 tx=0).stop 2024-05-09T08:48:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.996+0000 7f1cf31d5640 1 -- 172.21.15.98:0/1203759674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cec100ae0 msgr2=0x7f1cec19a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.996+0000 7f1cf31d5640 1 --2- 172.21.15.98:0/1203759674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cec100ae0 0x7f1cec19a460 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000b450 tx=0x7f1ce000b910 comp rx=0 tx=0).stop 2024-05-09T08:48:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.996+0000 7f1cf31d5640 1 -- 172.21.15.98:0/1203759674 shutdown_connections 2024-05-09T08:48:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.996+0000 7f1cf31d5640 1 --2- 172.21.15.98:0/1203759674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1cec101480 0x7f1cec1a14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.996+0000 7f1cf31d5640 1 --2- 172.21.15.98:0/1203759674 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1cd40a2b90 0x7f1cd40a5050 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.996+0000 7f1cf31d5640 1 --2- 172.21.15.98:0/1203759674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cec100ae0 0x7f1cec19a460 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.996+0000 7f1cf31d5640 1 --2- 172.21.15.98:0/1203759674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cec100180 0x7f1cec199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.996+0000 7f1cf31d5640 1 -- 172.21.15.98:0/1203759674 >> 172.21.15.98:0/1203759674 conn(0x7f1cec0fbd20 msgr2=0x7f1cec0fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.996+0000 7f1cf31d5640 1 -- 172.21.15.98:0/1203759674 shutdown_connections 2024-05-09T08:48:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:11.996+0000 7f1cf31d5640 1 -- 172.21.15.98:0/1203759674 wait complete. 2024-05-09T08:48:12.215 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 08:48:11 smithi105 podman[64828]: 2024-05-09 08:48:11.873141964 +0000 UTC m=+0.378645534 container start 1a83673e3f3356270df53c9ac846e49a5ef15ca4fa84cfd71aaf73537586d233 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-05-09T08:48:12.215 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 08:48:11 smithi105 ceph-mds[64847]: -- 172.21.15.105:0/778464164 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559c5f40fa40 con 0x559c5f3e2800 2024-05-09T08:48:12.215 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 08:48:11 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-b[64843]: starting mds.b at 2024-05-09T08:48:12.215 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 08:48:11 smithi105 bash[64828]: 1a83673e3f3356270df53c9ac846e49a5ef15ca4fa84cfd71aaf73537586d233 2024-05-09T08:48:12.216 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 08:48:11 smithi105 systemd[1]: Started Ceph mds.b for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:48:12.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:12 smithi098 ceph-mon[30092]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:12.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:12 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1203759674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:12.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:12 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:12 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:12 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:12 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:12 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:12 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:12 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:48:12.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:12 smithi105 ceph-mon[34498]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:12.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:12 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1203759674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:12.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:12 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:12 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:12 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:12 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:12 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:12 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:12 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:48:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:12 smithi142 ceph-mon[34625]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:12 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1203759674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:12 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:12 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:12 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:12 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:12 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:12 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:12.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:12 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:48:13.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:13 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] up:boot 2024-05-09T08:48:13.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:13 smithi098 ceph-mon[30092]: fsmap 12 up:standby 2024-05-09T08:48:13.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:13 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:48:13.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:13 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] up:boot 2024-05-09T08:48:13.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:13 smithi142 ceph-mon[34625]: fsmap 12 up:standby 2024-05-09T08:48:13.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:13 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:48:13.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:13 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] up:boot 2024-05-09T08:48:13.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:13 smithi105 ceph-mon[34498]: fsmap 12 up:standby 2024-05-09T08:48:13.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:13 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:48:14.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:48:14.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.116+0000 7fe3ff26b640 1 -- 172.21.15.98:0/65513634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f8102320 msgr2=0x7fe3f8108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.116+0000 7fe3ff26b640 1 --2- 172.21.15.98:0/65513634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f8102320 0x7fe3f8108bb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0009be0 tx=0x7fe3f002a6c0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.116+0000 7fe3ff26b640 1 -- 172.21.15.98:0/65513634 shutdown_connections 2024-05-09T08:48:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.116+0000 7fe3ff26b640 1 --2- 172.21.15.98:0/65513634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f8102320 0x7fe3f8108bb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.116+0000 7fe3ff26b640 1 --2- 172.21.15.98:0/65513634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3f8101960 0x7fe3f8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.116+0000 7fe3ff26b640 1 --2- 172.21.15.98:0/65513634 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3f8100770 0x7fe3f8100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.116+0000 7fe3ff26b640 1 -- 172.21.15.98:0/65513634 >> 172.21.15.98:0/65513634 conn(0x7fe3f80fbce0 msgr2=0x7fe3f80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:14.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.116+0000 7fe3ff26b640 1 -- 172.21.15.98:0/65513634 shutdown_connections 2024-05-09T08:48:14.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.116+0000 7fe3ff26b640 1 -- 172.21.15.98:0/65513634 wait complete. 2024-05-09T08:48:14.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.117+0000 7fe3ff26b640 1 Processor -- start 2024-05-09T08:48:14.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.117+0000 7fe3ff26b640 1 -- start start 2024-05-09T08:48:14.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.117+0000 7fe3ff26b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f8100770 0x7fe3f8069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.117+0000 7fe3ff26b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3f8101960 0x7fe3f819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.117+0000 7fe3ff26b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3f8102320 0x7fe3f819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.117+0000 7fe3ff26b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f810c3a0 con 0x7fe3f8102320 2024-05-09T08:48:14.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.117+0000 7fe3ff26b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f810c220 con 0x7fe3f8100770 2024-05-09T08:48:14.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.117+0000 7fe3ff26b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f810c520 con 0x7fe3f8101960 2024-05-09T08:48:14.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.117+0000 7fe3fd808640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3f8102320 0x7fe3f819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:14.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.117+0000 7fe3fd808640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3f8102320 0x7fe3f819ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34614/0 (socket says 172.21.15.98:34614) 2024-05-09T08:48:14.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.117+0000 7fe3fd808640 1 -- 172.21.15.98:0/2437771423 learned_addr learned my addr 172.21.15.98:0/2437771423 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:14.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.117+0000 7fe3fd007640 1 --2- 172.21.15.98:0/2437771423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f8100770 0x7fe3f8069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:14.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.117+0000 7fe3fc806640 1 --2- 172.21.15.98:0/2437771423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3f8101960 0x7fe3f819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:14.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.118+0000 7fe3fd808640 1 -- 172.21.15.98:0/2437771423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3f8101960 msgr2=0x7fe3f819c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.118+0000 7fe3fd808640 1 --2- 172.21.15.98:0/2437771423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3f8101960 0x7fe3f819c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.118+0000 7fe3fd808640 1 -- 172.21.15.98:0/2437771423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f8100770 msgr2=0x7fe3f8069550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.118+0000 7fe3fd808640 1 --2- 172.21.15.98:0/2437771423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f8100770 0x7fe3f8069550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.118+0000 7fe3fd808640 1 -- 172.21.15.98:0/2437771423 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f819f740 con 0x7fe3f8102320 2024-05-09T08:48:14.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.118+0000 7fe3fd808640 1 --2- 172.21.15.98:0/2437771423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3f8102320 0x7fe3f819ee80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe3f00305b0 tx=0x7fe3f00305e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:14.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.118+0000 7fe3ee7fc640 1 -- 172.21.15.98:0/2437771423 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0041020 con 0x7fe3f8102320 2024-05-09T08:48:14.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.118+0000 7fe3ee7fc640 1 -- 172.21.15.98:0/2437771423 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3f0037040 con 0x7fe3f8102320 2024-05-09T08:48:14.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.118+0000 7fe3ff26b640 1 -- 172.21.15.98:0/2437771423 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f819f9d0 con 0x7fe3f8102320 2024-05-09T08:48:14.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.118+0000 7fe3ee7fc640 1 -- 172.21.15.98:0/2437771423 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0036070 con 0x7fe3f8102320 2024-05-09T08:48:14.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.118+0000 7fe3ff26b640 1 -- 172.21.15.98:0/2437771423 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f819fec0 con 0x7fe3f8102320 2024-05-09T08:48:14.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.119+0000 7fe3ff26b640 1 -- 172.21.15.98:0/2437771423 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3f8107b10 con 0x7fe3f8102320 2024-05-09T08:48:14.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.120+0000 7fe3ee7fc640 1 -- 172.21.15.98:0/2437771423 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe3f0044020 con 0x7fe3f8102320 2024-05-09T08:48:14.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.121+0000 7fe3ee7fc640 1 --2- 172.21.15.98:0/2437771423 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe3e40a2b90 0x7fe3e40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.121+0000 7fe3fd007640 1 --2- 172.21.15.98:0/2437771423 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe3e40a2b90 0x7fe3e40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:14.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.121+0000 7fe3ee7fc640 1 -- 172.21.15.98:0/2437771423 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe3f00efb70 con 0x7fe3f8102320 2024-05-09T08:48:14.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.122+0000 7fe3fd007640 1 --2- 172.21.15.98:0/2437771423 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe3e40a2b90 0x7fe3e40a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe3f81017c0 tx=0x7fe3f4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:14.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.125+0000 7fe3ee7fc640 1 -- 172.21.15.98:0/2437771423 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe3f004b020 con 0x7fe3f8102320 2024-05-09T08:48:14.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:14 smithi098 ceph-mon[30092]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:14.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:14 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:14 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:14 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:14 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:14 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:14 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.350+0000 7fe3ff26b640 1 -- 172.21.15.98:0/2437771423 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3f802d380 con 0x7fe3f8102320 2024-05-09T08:48:14.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.351+0000 7fe3ee7fc640 1 -- 172.21.15.98:0/2437771423 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10488 (secure 0 0 0) 0x7fe3f0004670 con 0x7fe3f8102320 2024-05-09T08:48:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:14.351 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":14667,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-09T08:48:14.351 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-05-09T08:48:14.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.353+0000 7fe3ff26b640 1 -- 172.21.15.98:0/2437771423 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe3e40a2b90 msgr2=0x7fe3e40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.353+0000 7fe3ff26b640 1 --2- 172.21.15.98:0/2437771423 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe3e40a2b90 0x7fe3e40a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe3f81017c0 tx=0x7fe3f4005f90 comp rx=0 tx=0).stop 2024-05-09T08:48:14.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.353+0000 7fe3ff26b640 1 -- 172.21.15.98:0/2437771423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3f8102320 msgr2=0x7fe3f819ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.353+0000 7fe3ff26b640 1 --2- 172.21.15.98:0/2437771423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3f8102320 0x7fe3f819ee80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe3f00305b0 tx=0x7fe3f00305e0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.353+0000 7fe3ff26b640 1 -- 172.21.15.98:0/2437771423 shutdown_connections 2024-05-09T08:48:14.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.353+0000 7fe3ff26b640 1 --2- 172.21.15.98:0/2437771423 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe3e40a2b90 0x7fe3e40a5050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.353+0000 7fe3ff26b640 1 --2- 172.21.15.98:0/2437771423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3f8102320 0x7fe3f819ee80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.353+0000 7fe3ff26b640 1 --2- 172.21.15.98:0/2437771423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3f8101960 0x7fe3f819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.353+0000 7fe3ff26b640 1 --2- 172.21.15.98:0/2437771423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f8100770 0x7fe3f8069550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.353+0000 7fe3ff26b640 1 -- 172.21.15.98:0/2437771423 >> 172.21.15.98:0/2437771423 conn(0x7fe3f80fbce0 msgr2=0x7fe3f8104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:14.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.353+0000 7fe3ff26b640 1 -- 172.21.15.98:0/2437771423 shutdown_connections 2024-05-09T08:48:14.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.353+0000 7fe3ff26b640 1 -- 172.21.15.98:0/2437771423 wait complete. 2024-05-09T08:48:14.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:48:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.471+0000 7fd4d6838640 1 -- 172.21.15.98:0/4156099491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d0102a90 msgr2=0x7fd4d0102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.471+0000 7fd4d6838640 1 --2- 172.21.15.98:0/4156099491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d0102a90 0x7fd4d0102f10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009900 tx=0x7fd4b802a410 comp rx=0 tx=0).stop 2024-05-09T08:48:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.471+0000 7fd4d6838640 1 -- 172.21.15.98:0/4156099491 shutdown_connections 2024-05-09T08:48:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.471+0000 7fd4d6838640 1 --2- 172.21.15.98:0/4156099491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4d0103450 0x7fd4d0109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.471+0000 7fd4d6838640 1 --2- 172.21.15.98:0/4156099491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d0102a90 0x7fd4d0102f10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.471+0000 7fd4d6838640 1 --2- 172.21.15.98:0/4156099491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4d01018a0 0x7fd4d0101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.471+0000 7fd4d6838640 1 -- 172.21.15.98:0/4156099491 >> 172.21.15.98:0/4156099491 conn(0x7fd4d00fce30 msgr2=0x7fd4d00ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.471+0000 7fd4d6838640 1 -- 172.21.15.98:0/4156099491 shutdown_connections 2024-05-09T08:48:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.471+0000 7fd4d6838640 1 -- 172.21.15.98:0/4156099491 wait complete. 2024-05-09T08:48:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.471+0000 7fd4d6838640 1 Processor -- start 2024-05-09T08:48:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.471+0000 7fd4d6838640 1 -- start start 2024-05-09T08:48:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.471+0000 7fd4d6838640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4d01018a0 0x7fd4d0197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.471+0000 7fd4d6838640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4d0102a90 0x7fd4d01980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4d6838640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d0103450 0x7fd4d019f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4d6838640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d010d500 con 0x7fd4d0103450 2024-05-09T08:48:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4d6838640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d010d380 con 0x7fd4d01018a0 2024-05-09T08:48:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4d6838640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d010d680 con 0x7fd4d0102a90 2024-05-09T08:48:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4d4dd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d0103450 0x7fd4d019f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4cffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4d01018a0 0x7fd4d0197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4d4dd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d0103450 0x7fd4d019f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34642/0 (socket says 172.21.15.98:34642) 2024-05-09T08:48:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4d4dd5640 1 -- 172.21.15.98:0/1369211449 learned_addr learned my addr 172.21.15.98:0/1369211449 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4c7fff640 1 --2- 172.21.15.98:0/1369211449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4d0102a90 0x7fd4d01980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4cffff640 1 -- 172.21.15.98:0/1369211449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4d0102a90 msgr2=0x7fd4d01980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4cffff640 1 --2- 172.21.15.98:0/1369211449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4d0102a90 0x7fd4d01980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4cffff640 1 -- 172.21.15.98:0/1369211449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d0103450 msgr2=0x7fd4d019f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4cffff640 1 --2- 172.21.15.98:0/1369211449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d0103450 0x7fd4d019f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4cffff640 1 -- 172.21.15.98:0/1369211449 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d019f7c0 con 0x7fd4d01018a0 2024-05-09T08:48:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4cffff640 1 --2- 172.21.15.98:0/1369211449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4d01018a0 0x7fd4d0197ba0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000e9c0 tx=0x7fd4c000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4cdffb640 1 -- 172.21.15.98:0/1369211449 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c000ccc0 con 0x7fd4d01018a0 2024-05-09T08:48:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.472+0000 7fd4d6838640 1 -- 172.21.15.98:0/1369211449 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d019fab0 con 0x7fd4d01018a0 2024-05-09T08:48:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.473+0000 7fd4cdffb640 1 -- 172.21.15.98:0/1369211449 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4c00035f0 con 0x7fd4d01018a0 2024-05-09T08:48:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.473+0000 7fd4cdffb640 1 -- 172.21.15.98:0/1369211449 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c000ce60 con 0x7fd4d01018a0 2024-05-09T08:48:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.473+0000 7fd4d6838640 1 -- 172.21.15.98:0/1369211449 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d019ffd0 con 0x7fd4d01018a0 2024-05-09T08:48:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.474+0000 7fd4d6838640 1 -- 172.21.15.98:0/1369211449 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4d0108bc0 con 0x7fd4d01018a0 2024-05-09T08:48:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.475+0000 7fd4cdffb640 1 -- 172.21.15.98:0/1369211449 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd4c00037a0 con 0x7fd4d01018a0 2024-05-09T08:48:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.475+0000 7fd4cdffb640 1 --2- 172.21.15.98:0/1369211449 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd4bc0a2b90 0x7fd4bc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.475+0000 7fd4c7fff640 1 --2- 172.21.15.98:0/1369211449 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd4bc0a2b90 0x7fd4bc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.475+0000 7fd4cdffb640 1 -- 172.21.15.98:0/1369211449 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd4c00ceff0 con 0x7fd4d01018a0 2024-05-09T08:48:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.476+0000 7fd4c7fff640 1 --2- 172.21.15.98:0/1369211449 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd4bc0a2b90 0x7fd4bc0a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80314e0 tx=0x7fd4b8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:14.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.480+0000 7fd4cdffb640 1 -- 172.21.15.98:0/1369211449 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd4c009c3b0 con 0x7fd4d01018a0 2024-05-09T08:48:14.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:14 smithi105 ceph-mon[34498]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:14.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:14 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:14 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:14 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:14 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:14 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:14 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:14 smithi142 ceph-mon[34625]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:14.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:14 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:14 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:14 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:14 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:14 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:14 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.664+0000 7fd4d6838640 1 -- 172.21.15.98:0/1369211449 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4d002d1c0 con 0x7fd4d01018a0 2024-05-09T08:48:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.666+0000 7fd4cdffb640 1 -- 172.21.15.98:0/1369211449 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18478 (secure 0 0 0) 0x7fd4c009bcb0 con 0x7fd4d01018a0 2024-05-09T08:48:14.665 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:14.665 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":82,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:47:28.644215+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:48:14.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.668+0000 7fd4d6838640 1 -- 172.21.15.98:0/1369211449 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd4bc0a2b90 msgr2=0x7fd4bc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.668+0000 7fd4d6838640 1 --2- 172.21.15.98:0/1369211449 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd4bc0a2b90 0x7fd4bc0a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80314e0 tx=0x7fd4b8000bc0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.668+0000 7fd4d6838640 1 -- 172.21.15.98:0/1369211449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4d01018a0 msgr2=0x7fd4d0197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.668+0000 7fd4d6838640 1 --2- 172.21.15.98:0/1369211449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4d01018a0 0x7fd4d0197ba0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000e9c0 tx=0x7fd4c000ee80 comp rx=0 tx=0).stop 2024-05-09T08:48:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.668+0000 7fd4d6838640 1 -- 172.21.15.98:0/1369211449 shutdown_connections 2024-05-09T08:48:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.668+0000 7fd4d6838640 1 --2- 172.21.15.98:0/1369211449 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd4bc0a2b90 0x7fd4bc0a5050 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.668+0000 7fd4d6838640 1 --2- 172.21.15.98:0/1369211449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d0103450 0x7fd4d019f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.668+0000 7fd4d6838640 1 --2- 172.21.15.98:0/1369211449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4d0102a90 0x7fd4d01980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.668+0000 7fd4d6838640 1 --2- 172.21.15.98:0/1369211449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4d01018a0 0x7fd4d0197ba0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.668+0000 7fd4d6838640 1 -- 172.21.15.98:0/1369211449 >> 172.21.15.98:0/1369211449 conn(0x7fd4d00fce30 msgr2=0x7fd4d0105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.668+0000 7fd4d6838640 1 -- 172.21.15.98:0/1369211449 shutdown_connections 2024-05-09T08:48:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.668+0000 7fd4d6838640 1 -- 172.21.15.98:0/1369211449 wait complete. 2024-05-09T08:48:14.679 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:43:39.424443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T08:48:14.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:48:14.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.787+0000 7f4ac4f89640 1 -- 172.21.15.98:0/3329965238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ac0102320 msgr2=0x7f4ac0108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.787+0000 7f4ac4f89640 1 --2- 172.21.15.98:0/3329965238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ac0102320 0x7f4ac0108bb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009ca0 tx=0x7f4ab802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.787+0000 7f4ac4f89640 1 -- 172.21.15.98:0/3329965238 shutdown_connections 2024-05-09T08:48:14.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.787+0000 7f4ac4f89640 1 --2- 172.21.15.98:0/3329965238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ac0102320 0x7f4ac0108bb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.787+0000 7f4ac4f89640 1 --2- 172.21.15.98:0/3329965238 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ac0101960 0x7f4ac0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.787+0000 7f4ac4f89640 1 --2- 172.21.15.98:0/3329965238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ac0100770 0x7f4ac0100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.787+0000 7f4ac4f89640 1 -- 172.21.15.98:0/3329965238 >> 172.21.15.98:0/3329965238 conn(0x7f4ac00fbce0 msgr2=0x7f4ac00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:14.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.787+0000 7f4ac4f89640 1 -- 172.21.15.98:0/3329965238 shutdown_connections 2024-05-09T08:48:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.787+0000 7f4ac4f89640 1 -- 172.21.15.98:0/3329965238 wait complete. 2024-05-09T08:48:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.787+0000 7f4ac4f89640 1 Processor -- start 2024-05-09T08:48:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.787+0000 7f4ac4f89640 1 -- start start 2024-05-09T08:48:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4ac4f89640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ac0100770 0x7f4ac0197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4ac4f89640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ac0101960 0x7f4ac01980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4ac4f89640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ac0102320 0x7f4ac019f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4ac4f89640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac010d510 con 0x7f4ac0102320 2024-05-09T08:48:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4ac4f89640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac010d390 con 0x7f4ac0100770 2024-05-09T08:48:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4ac4f89640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac010d690 con 0x7f4ac0101960 2024-05-09T08:48:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4abed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ac0102320 0x7f4ac019f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4abed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ac0102320 0x7f4ac019f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34662/0 (socket says 172.21.15.98:34662) 2024-05-09T08:48:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4abed9d640 1 -- 172.21.15.98:0/2626739018 learned_addr learned my addr 172.21.15.98:0/2626739018 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:14.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4abe59c640 1 --2- 172.21.15.98:0/2626739018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ac0100770 0x7f4ac0197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4ab7fff640 1 --2- 172.21.15.98:0/2626739018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ac0101960 0x7f4ac01980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4abed9d640 1 -- 172.21.15.98:0/2626739018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ac0101960 msgr2=0x7f4ac01980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4abed9d640 1 --2- 172.21.15.98:0/2626739018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ac0101960 0x7f4ac01980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4abed9d640 1 -- 172.21.15.98:0/2626739018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ac0100770 msgr2=0x7f4ac0197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4abed9d640 1 --2- 172.21.15.98:0/2626739018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ac0100770 0x7f4ac0197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4abed9d640 1 -- 172.21.15.98:0/2626739018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac019f780 con 0x7f4ac0102320 2024-05-09T08:48:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4abed9d640 1 --2- 172.21.15.98:0/2626739018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ac0102320 0x7f4ac019f110 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8005e30 tx=0x7f4ab8031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4ab77fe640 1 -- 172.21.15.98:0/2626739018 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8003c30 con 0x7f4ac0102320 2024-05-09T08:48:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4ab77fe640 1 -- 172.21.15.98:0/2626739018 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ab8003dd0 con 0x7f4ac0102320 2024-05-09T08:48:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4ac4f89640 1 -- 172.21.15.98:0/2626739018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac019fa10 con 0x7f4ac0102320 2024-05-09T08:48:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4ab77fe640 1 -- 172.21.15.98:0/2626739018 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab802fde0 con 0x7f4ac0102320 2024-05-09T08:48:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.788+0000 7f4ac4f89640 1 -- 172.21.15.98:0/2626739018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac019ff60 con 0x7f4ac0102320 2024-05-09T08:48:14.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.789+0000 7f4ac4f89640 1 -- 172.21.15.98:0/2626739018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ac0107ae0 con 0x7f4ac0102320 2024-05-09T08:48:14.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.791+0000 7f4ab77fe640 1 -- 172.21.15.98:0/2626739018 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4ab8004050 con 0x7f4ac0102320 2024-05-09T08:48:14.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.791+0000 7f4ab77fe640 1 --2- 172.21.15.98:0/2626739018 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4aac0a2b90 0x7f4aac0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.791+0000 7f4abe59c640 1 --2- 172.21.15.98:0/2626739018 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4aac0a2b90 0x7f4aac0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:14.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.791+0000 7f4ab77fe640 1 -- 172.21.15.98:0/2626739018 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4ab80f00a0 con 0x7f4ac0102320 2024-05-09T08:48:14.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.792+0000 7f4abe59c640 1 --2- 172.21.15.98:0/2626739018 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4aac0a2b90 0x7f4aac0a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0005fd0 tx=0x7f4ab000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:14.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.795+0000 7f4ab77fe640 1 -- 172.21.15.98:0/2626739018 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4ab80bd4a0 con 0x7f4ac0102320 2024-05-09T08:48:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.983+0000 7f4ac4f89640 1 -- 172.21.15.98:0/2626739018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4ac002d350 con 0x7f4ac0102320 2024-05-09T08:48:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.983+0000 7f4ab77fe640 1 -- 172.21.15.98:0/2626739018 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f4ab80bcda0 con 0x7f4ac0102320 2024-05-09T08:48:14.983 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-09T08:48:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.985+0000 7f4ac4f89640 1 -- 172.21.15.98:0/2626739018 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4aac0a2b90 msgr2=0x7f4aac0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.985+0000 7f4ac4f89640 1 --2- 172.21.15.98:0/2626739018 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4aac0a2b90 0x7f4aac0a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0005fd0 tx=0x7f4ab000b390 comp rx=0 tx=0).stop 2024-05-09T08:48:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.985+0000 7f4ac4f89640 1 -- 172.21.15.98:0/2626739018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ac0102320 msgr2=0x7f4ac019f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.985+0000 7f4ac4f89640 1 --2- 172.21.15.98:0/2626739018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ac0102320 0x7f4ac019f110 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8005e30 tx=0x7f4ab8031ee0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.986+0000 7f4ac4f89640 1 -- 172.21.15.98:0/2626739018 shutdown_connections 2024-05-09T08:48:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.986+0000 7f4ac4f89640 1 --2- 172.21.15.98:0/2626739018 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4aac0a2b90 0x7f4aac0a5050 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.986+0000 7f4ac4f89640 1 --2- 172.21.15.98:0/2626739018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ac0102320 0x7f4ac019f110 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.986+0000 7f4ac4f89640 1 --2- 172.21.15.98:0/2626739018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ac0101960 0x7f4ac01980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.986+0000 7f4ac4f89640 1 --2- 172.21.15.98:0/2626739018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ac0100770 0x7f4ac0197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.986+0000 7f4ac4f89640 1 -- 172.21.15.98:0/2626739018 >> 172.21.15.98:0/2626739018 conn(0x7f4ac00fbce0 msgr2=0x7f4ac0104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.986+0000 7f4ac4f89640 1 -- 172.21.15.98:0/2626739018 shutdown_connections 2024-05-09T08:48:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:14.986+0000 7f4ac4f89640 1 -- 172.21.15.98:0/2626739018 wait complete. 2024-05-09T08:48:14.997 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T08:48:15.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.103+0000 7fa971fce640 1 -- 172.21.15.98:0/3086761891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101960 msgr2=0x7fa96c101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.103+0000 7fa971fce640 1 --2- 172.21.15.98:0/3086761891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101960 0x7fa96c101de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa954009900 tx=0x7fa95402a410 comp rx=0 tx=0).stop 2024-05-09T08:48:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.103+0000 7fa971fce640 1 -- 172.21.15.98:0/3086761891 shutdown_connections 2024-05-09T08:48:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.103+0000 7fa971fce640 1 --2- 172.21.15.98:0/3086761891 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96c102320 0x7fa96c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.103+0000 7fa971fce640 1 --2- 172.21.15.98:0/3086761891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101960 0x7fa96c101de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.103+0000 7fa971fce640 1 --2- 172.21.15.98:0/3086761891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa96c100770 0x7fa96c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.103+0000 7fa971fce640 1 -- 172.21.15.98:0/3086761891 >> 172.21.15.98:0/3086761891 conn(0x7fa96c0fbce0 msgr2=0x7fa96c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.103+0000 7fa971fce640 1 -- 172.21.15.98:0/3086761891 shutdown_connections 2024-05-09T08:48:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.103+0000 7fa971fce640 1 -- 172.21.15.98:0/3086761891 wait complete. 2024-05-09T08:48:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.103+0000 7fa971fce640 1 Processor -- start 2024-05-09T08:48:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.103+0000 7fa971fce640 1 -- start start 2024-05-09T08:48:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa971fce640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96c100770 0x7fa96c197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa971fce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101960 0x7fa96c198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa971fce640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa96c102320 0x7fa96c19f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa971fce640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c10c3d0 con 0x7fa96c101960 2024-05-09T08:48:15.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa971fce640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c10c250 con 0x7fa96c102320 2024-05-09T08:48:15.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa971fce640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c10c550 con 0x7fa96c100770 2024-05-09T08:48:15.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa963fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101960 0x7fa96c198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:15.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa963fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101960 0x7fa96c198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34686/0 (socket says 172.21.15.98:34686) 2024-05-09T08:48:15.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa963fff640 1 -- 172.21.15.98:0/4030282407 learned_addr learned my addr 172.21.15.98:0/4030282407 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:15.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa96b7fe640 1 --2- 172.21.15.98:0/4030282407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96c100770 0x7fa96c197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:15.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa96bfff640 1 --2- 172.21.15.98:0/4030282407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa96c102320 0x7fa96c19f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:15.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa963fff640 1 -- 172.21.15.98:0/4030282407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96c100770 msgr2=0x7fa96c197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:15.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa963fff640 1 --2- 172.21.15.98:0/4030282407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96c100770 0x7fa96c197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa963fff640 1 -- 172.21.15.98:0/4030282407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa96c102320 msgr2=0x7fa96c19f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:15.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa963fff640 1 --2- 172.21.15.98:0/4030282407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa96c102320 0x7fa96c19f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa963fff640 1 -- 172.21.15.98:0/4030282407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa96c19f7f0 con 0x7fa96c101960 2024-05-09T08:48:15.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa963fff640 1 --2- 172.21.15.98:0/4030282407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101960 0x7fa96c198110 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa954009230 tx=0x7fa954004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:15.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa9697fa640 1 -- 172.21.15.98:0/4030282407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95403f030 con 0x7fa96c101960 2024-05-09T08:48:15.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa9697fa640 1 -- 172.21.15.98:0/4030282407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa954003ce0 con 0x7fa96c101960 2024-05-09T08:48:15.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa971fce640 1 -- 172.21.15.98:0/4030282407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa96c19fa80 con 0x7fa96c101960 2024-05-09T08:48:15.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa9697fa640 1 -- 172.21.15.98:0/4030282407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa954035070 con 0x7fa96c101960 2024-05-09T08:48:15.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.104+0000 7fa971fce640 1 -- 172.21.15.98:0/4030282407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa96c19ff70 con 0x7fa96c101960 2024-05-09T08:48:15.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.105+0000 7fa971fce640 1 -- 172.21.15.98:0/4030282407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa96c1a01a0 con 0x7fa96c101960 2024-05-09T08:48:15.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.107+0000 7fa9697fa640 1 -- 172.21.15.98:0/4030282407 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa954042030 con 0x7fa96c101960 2024-05-09T08:48:15.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.107+0000 7fa9697fa640 1 --2- 172.21.15.98:0/4030282407 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa9580a2b90 0x7fa9580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:15.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.107+0000 7fa96b7fe640 1 --2- 172.21.15.98:0/4030282407 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa9580a2b90 0x7fa9580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:15.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.107+0000 7fa9697fa640 1 -- 172.21.15.98:0/4030282407 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa9540ef4c0 con 0x7fa96c101960 2024-05-09T08:48:15.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.108+0000 7fa96b7fe640 1 --2- 172.21.15.98:0/4030282407 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa9580a2b90 0x7fa9580a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa95c005fd0 tx=0x7fa95c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:15.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.115+0000 7fa9697fa640 1 -- 172.21.15.98:0/4030282407 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa9540bc270 con 0x7fa96c101960 2024-05-09T08:48:15.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.304+0000 7fa971fce640 1 -- 172.21.15.98:0/4030282407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa96c1a0370 con 0x7fa96c101960 2024-05-09T08:48:15.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.305+0000 7fa9697fa640 1 -- 172.21.15.98:0/4030282407 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7fa9540c0570 con 0x7fa96c101960 2024-05-09T08:48:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:15.304 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2024-05-09T08:48:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T08:48:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2024-05-09T08:48:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2024-05-09T08:48:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:48:15.305 INFO:teuthology.orchestra.run.smithi098.stdout:] 2024-05-09T08:48:15.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.307+0000 7fa971fce640 1 -- 172.21.15.98:0/4030282407 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa9580a2b90 msgr2=0x7fa9580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:15.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.307+0000 7fa971fce640 1 --2- 172.21.15.98:0/4030282407 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa9580a2b90 0x7fa9580a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa95c005fd0 tx=0x7fa95c00b410 comp rx=0 tx=0).stop 2024-05-09T08:48:15.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.307+0000 7fa971fce640 1 -- 172.21.15.98:0/4030282407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101960 msgr2=0x7fa96c198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:15.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.307+0000 7fa971fce640 1 --2- 172.21.15.98:0/4030282407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101960 0x7fa96c198110 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa954009230 tx=0x7fa954004360 comp rx=0 tx=0).stop 2024-05-09T08:48:15.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.307+0000 7fa971fce640 1 -- 172.21.15.98:0/4030282407 shutdown_connections 2024-05-09T08:48:15.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.307+0000 7fa971fce640 1 --2- 172.21.15.98:0/4030282407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa96c102320 0x7fa96c19f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.307+0000 7fa971fce640 1 --2- 172.21.15.98:0/4030282407 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa9580a2b90 0x7fa9580a5050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.307+0000 7fa971fce640 1 --2- 172.21.15.98:0/4030282407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101960 0x7fa96c198110 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.307+0000 7fa971fce640 1 --2- 172.21.15.98:0/4030282407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96c100770 0x7fa96c197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.307+0000 7fa971fce640 1 -- 172.21.15.98:0/4030282407 >> 172.21.15.98:0/4030282407 conn(0x7fa96c0fbce0 msgr2=0x7fa96c0689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:15.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.307+0000 7fa971fce640 1 -- 172.21.15.98:0/4030282407 shutdown_connections 2024-05-09T08:48:15.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.307+0000 7fa971fce640 1 -- 172.21.15.98:0/4030282407 wait complete. 2024-05-09T08:48:15.317 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:15 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2437771423' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:15.317 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:15.317 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:48:15.317 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:15.317 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:15 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1369211449' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:15.317 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:15 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:48:15.317 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:15 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2626739018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:48:15.318 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T08:48:15.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T08:48:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.436+0000 7fc61fd42640 1 -- 172.21.15.98:0/132241553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc618104ce0 msgr2=0x7fc6181070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.436+0000 7fc61fd42640 1 --2- 172.21.15.98:0/132241553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc618104ce0 0x7fc6181070d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc608009900 tx=0x7fc60802a410 comp rx=0 tx=0).stop 2024-05-09T08:48:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.436+0000 7fc61fd42640 1 -- 172.21.15.98:0/132241553 shutdown_connections 2024-05-09T08:48:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.436+0000 7fc61fd42640 1 --2- 172.21.15.98:0/132241553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc618107610 0x7fc618109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.436+0000 7fc61fd42640 1 --2- 172.21.15.98:0/132241553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc618104ce0 0x7fc6181070d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.436+0000 7fc61fd42640 1 --2- 172.21.15.98:0/132241553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6181023b0 0x7fc6181047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.436+0000 7fc61fd42640 1 -- 172.21.15.98:0/132241553 >> 172.21.15.98:0/132241553 conn(0x7fc6180fbce0 msgr2=0x7fc6180fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.436+0000 7fc61fd42640 1 -- 172.21.15.98:0/132241553 shutdown_connections 2024-05-09T08:48:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.437+0000 7fc61fd42640 1 -- 172.21.15.98:0/132241553 wait complete. 2024-05-09T08:48:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.437+0000 7fc61fd42640 1 Processor -- start 2024-05-09T08:48:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.437+0000 7fc61fd42640 1 -- start start 2024-05-09T08:48:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.437+0000 7fc61fd42640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6181023b0 0x7fc618197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.437+0000 7fc61fd42640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc618104ce0 0x7fc6181981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.437+0000 7fc61fd42640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc618107610 0x7fc61819f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.437+0000 7fc61fd42640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc61810d570 con 0x7fc6181023b0 2024-05-09T08:48:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.437+0000 7fc61fd42640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc61810d3f0 con 0x7fc618104ce0 2024-05-09T08:48:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.437+0000 7fc61fd42640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc61810d6f0 con 0x7fc618107610 2024-05-09T08:48:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.437+0000 7fc61dade640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6181023b0 0x7fc618197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.437+0000 7fc61dade640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6181023b0 0x7fc618197c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34716/0 (socket says 172.21.15.98:34716) 2024-05-09T08:48:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.437+0000 7fc61dade640 1 -- 172.21.15.98:0/3663786078 learned_addr learned my addr 172.21.15.98:0/3663786078 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.437+0000 7fc61e2df640 1 --2- 172.21.15.98:0/3663786078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc618107610 0x7fc61819f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.437+0000 7fc61d2dd640 1 --2- 172.21.15.98:0/3663786078 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc618104ce0 0x7fc6181981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.438+0000 7fc61dade640 1 -- 172.21.15.98:0/3663786078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc618107610 msgr2=0x7fc61819f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.438+0000 7fc61dade640 1 --2- 172.21.15.98:0/3663786078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc618107610 0x7fc61819f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.438+0000 7fc61dade640 1 -- 172.21.15.98:0/3663786078 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc618104ce0 msgr2=0x7fc6181981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.438+0000 7fc61dade640 1 --2- 172.21.15.98:0/3663786078 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc618104ce0 0x7fc6181981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.438+0000 7fc61dade640 1 -- 172.21.15.98:0/3663786078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61819f8b0 con 0x7fc6181023b0 2024-05-09T08:48:15.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.438+0000 7fc61dade640 1 --2- 172.21.15.98:0/3663786078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6181023b0 0x7fc618197c70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc61400d770 tx=0x7fc61400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:15.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.438+0000 7fc60effd640 1 -- 172.21.15.98:0/3663786078 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614017070 con 0x7fc6181023b0 2024-05-09T08:48:15.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.438+0000 7fc61fd42640 1 -- 172.21.15.98:0/3663786078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61819fba0 con 0x7fc6181023b0 2024-05-09T08:48:15.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.438+0000 7fc60effd640 1 -- 172.21.15.98:0/3663786078 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc614005500 con 0x7fc6181023b0 2024-05-09T08:48:15.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.438+0000 7fc60effd640 1 -- 172.21.15.98:0/3663786078 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614004e00 con 0x7fc6181023b0 2024-05-09T08:48:15.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.438+0000 7fc61fd42640 1 -- 172.21.15.98:0/3663786078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6181a0070 con 0x7fc6181023b0 2024-05-09T08:48:15.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.439+0000 7fc61fd42640 1 -- 172.21.15.98:0/3663786078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6180fd7d0 con 0x7fc6181023b0 2024-05-09T08:48:15.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.440+0000 7fc60effd640 1 -- 172.21.15.98:0/3663786078 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc614004fa0 con 0x7fc6181023b0 2024-05-09T08:48:15.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.440+0000 7fc60effd640 1 --2- 172.21.15.98:0/3663786078 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc6040a2b90 0x7fc6040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:15.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.441+0000 7fc61d2dd640 1 --2- 172.21.15.98:0/3663786078 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc6040a2b90 0x7fc6040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:15.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.441+0000 7fc60effd640 1 -- 172.21.15.98:0/3663786078 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc6140ce6d0 con 0x7fc6181023b0 2024-05-09T08:48:15.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.441+0000 7fc61d2dd640 1 --2- 172.21.15.98:0/3663786078 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc6040a2b90 0x7fc6040a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc608006010 tx=0x7fc608000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:15.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.445+0000 7fc60effd640 1 -- 172.21.15.98:0/3663786078 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc61409bad0 con 0x7fc6181023b0 2024-05-09T08:48:15.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:15 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2437771423' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:15.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:15.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:48:15.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:15.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:15 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1369211449' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:15.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:15 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:48:15.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:15 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2626739018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:48:15.621 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:15 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2437771423' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:15.621 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:15 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:15.621 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:15 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:48:15.621 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:15 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:15.622 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:15 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1369211449' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:15.622 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:15 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:48:15.622 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:15 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2626739018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:48:15.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:15.624+0000 7fc61fd42640 1 -- 172.21.15.98:0/3663786078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc61802d410 con 0x7fc6181023b0 2024-05-09T08:48:16.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:16 smithi105 ceph-mon[34498]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:16.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:16 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/4030282407' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-09T08:48:16.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:16 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3663786078' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-09T08:48:16.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.572+0000 7fc60effd640 1 -- 172.21.15.98:0/3663786078 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7fc61409b3d0 con 0x7fc6181023b0 2024-05-09T08:48:16.571 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2024-05-09T08:48:16.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.574+0000 7fc61fd42640 1 -- 172.21.15.98:0/3663786078 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc6040a2b90 msgr2=0x7fc6040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:16.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.574+0000 7fc61fd42640 1 --2- 172.21.15.98:0/3663786078 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc6040a2b90 0x7fc6040a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc608006010 tx=0x7fc608000bc0 comp rx=0 tx=0).stop 2024-05-09T08:48:16.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.574+0000 7fc61fd42640 1 -- 172.21.15.98:0/3663786078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6181023b0 msgr2=0x7fc618197c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:16.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.574+0000 7fc61fd42640 1 --2- 172.21.15.98:0/3663786078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6181023b0 0x7fc618197c70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc61400d770 tx=0x7fc61400dc30 comp rx=0 tx=0).stop 2024-05-09T08:48:16.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.574+0000 7fc61fd42640 1 -- 172.21.15.98:0/3663786078 shutdown_connections 2024-05-09T08:48:16.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.574+0000 7fc61fd42640 1 --2- 172.21.15.98:0/3663786078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc618107610 0x7fc61819f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:16.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.574+0000 7fc61fd42640 1 --2- 172.21.15.98:0/3663786078 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc618104ce0 0x7fc6181981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:16.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.574+0000 7fc61fd42640 1 --2- 172.21.15.98:0/3663786078 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc6040a2b90 0x7fc6040a5050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:16.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.574+0000 7fc61fd42640 1 --2- 172.21.15.98:0/3663786078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6181023b0 0x7fc618197c70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:16.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.574+0000 7fc61fd42640 1 -- 172.21.15.98:0/3663786078 >> 172.21.15.98:0/3663786078 conn(0x7fc6180fbce0 msgr2=0x7fc618102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:16.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.575+0000 7fc61fd42640 1 -- 172.21.15.98:0/3663786078 shutdown_connections 2024-05-09T08:48:16.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.575+0000 7fc61fd42640 1 -- 172.21.15.98:0/3663786078 wait complete. 2024-05-09T08:48:16.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:16 smithi098 ceph-mon[30092]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:16.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:16 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4030282407' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-09T08:48:16.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:16 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3663786078' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-09T08:48:16.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-05-09T08:48:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:16 smithi142 ceph-mon[34625]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:16 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/4030282407' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-09T08:48:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:16 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3663786078' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-09T08:48:16.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.702+0000 7fd93ca38640 1 -- 172.21.15.98:0/1709067156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd938107570 msgr2=0x7fd9381099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:16.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.702+0000 7fd93ca38640 1 --2- 172.21.15.98:0/1709067156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd938107570 0x7fd9381099c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd930009ca0 tx=0x7fd93002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:16.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.702+0000 7fd93ca38640 1 -- 172.21.15.98:0/1709067156 shutdown_connections 2024-05-09T08:48:16.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.702+0000 7fd93ca38640 1 --2- 172.21.15.98:0/1709067156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd938107570 0x7fd9381099c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:16.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.702+0000 7fd93ca38640 1 --2- 172.21.15.98:0/1709067156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd938104c40 0x7fd938107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:16.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.702+0000 7fd93ca38640 1 --2- 172.21.15.98:0/1709067156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd938102310 0x7fd938104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:16.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.702+0000 7fd93ca38640 1 -- 172.21.15.98:0/1709067156 >> 172.21.15.98:0/1709067156 conn(0x7fd9380fbd20 msgr2=0x7fd9380fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:16.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.702+0000 7fd93ca38640 1 -- 172.21.15.98:0/1709067156 shutdown_connections 2024-05-09T08:48:16.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.702+0000 7fd93ca38640 1 -- 172.21.15.98:0/1709067156 wait complete. 2024-05-09T08:48:16.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.703+0000 7fd93ca38640 1 Processor -- start 2024-05-09T08:48:16.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.703+0000 7fd93ca38640 1 -- start start 2024-05-09T08:48:16.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.703+0000 7fd93ca38640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd938102310 0x7fd938101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:16.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.703+0000 7fd93ca38640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd938104c40 0x7fd9381002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:16.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.703+0000 7fd93ca38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd938107570 0x7fd938100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:16.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.703+0000 7fd93ca38640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd93810d530 con 0x7fd938107570 2024-05-09T08:48:16.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.703+0000 7fd93ca38640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd93810d3b0 con 0x7fd938102310 2024-05-09T08:48:16.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.703+0000 7fd93ca38640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd93810d6b0 con 0x7fd938104c40 2024-05-09T08:48:16.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd936d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd938107570 0x7fd938100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:16.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd936d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd938107570 0x7fd938100800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39280/0 (socket says 172.21.15.98:39280) 2024-05-09T08:48:16.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd936d9d640 1 -- 172.21.15.98:0/1087219207 learned_addr learned my addr 172.21.15.98:0/1087219207 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:16.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd92dd9b640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd938104c40 0x7fd9381002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:16.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd93659c640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd938102310 0x7fd938101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:16.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd936d9d640 1 -- 172.21.15.98:0/1087219207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd938104c40 msgr2=0x7fd9381002c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:16.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd936d9d640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd938104c40 0x7fd9381002c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd936d9d640 1 -- 172.21.15.98:0/1087219207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd938102310 msgr2=0x7fd938101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd936d9d640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd938102310 0x7fd938101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd936d9d640 1 -- 172.21.15.98:0/1087219207 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd938101090 con 0x7fd938107570 2024-05-09T08:48:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd92dd9b640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd938104c40 0x7fd9381002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:48:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd936d9d640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd938107570 0x7fd938100800 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd930031ae0 tx=0x7fd930031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd92ffff640 1 -- 172.21.15.98:0/1087219207 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd930003c30 con 0x7fd938107570 2024-05-09T08:48:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd93ca38640 1 -- 172.21.15.98:0/1087219207 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9381a1c70 con 0x7fd938107570 2024-05-09T08:48:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd92ffff640 1 -- 172.21.15.98:0/1087219207 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd930003dd0 con 0x7fd938107570 2024-05-09T08:48:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd92ffff640 1 -- 172.21.15.98:0/1087219207 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93002fcb0 con 0x7fd938107570 2024-05-09T08:48:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.704+0000 7fd93ca38640 1 -- 172.21.15.98:0/1087219207 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9381a21c0 con 0x7fd938107570 2024-05-09T08:48:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.705+0000 7fd93ca38640 1 -- 172.21.15.98:0/1087219207 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9380fd810 con 0x7fd938107570 2024-05-09T08:48:16.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.707+0000 7fd92ffff640 1 -- 172.21.15.98:0/1087219207 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd930004050 con 0x7fd938107570 2024-05-09T08:48:16.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.707+0000 7fd92ffff640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd9240a2b90 0x7fd9240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:16.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.708+0000 7fd93659c640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd9240a2b90 0x7fd9240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:16.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.708+0000 7fd92ffff640 1 -- 172.21.15.98:0/1087219207 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7fd9300f02b0 con 0x7fd938107570 2024-05-09T08:48:16.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.708+0000 7fd93659c640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd9240a2b90 0x7fd9240a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd928009b60 tx=0x7fd928005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:16.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.711+0000 7fd92ffff640 1 -- 172.21.15.98:0/1087219207 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd9300bd4a0 con 0x7fd938107570 2024-05-09T08:48:16.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:16.886+0000 7fd93ca38640 1 -- 172.21.15.98:0/1087219207 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fd93802d1c0 con 0x7fd938107570 2024-05-09T08:48:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.577+0000 7fd92ffff640 1 -- 172.21.15.98:0/1087219207 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v84) ==== 171+0+0 (secure 0 0 0) 0x7fd9300bcda0 con 0x7fd938107570 2024-05-09T08:48:17.577 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2024-05-09T08:48:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.580+0000 7fd93ca38640 1 -- 172.21.15.98:0/1087219207 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd9240a2b90 msgr2=0x7fd9240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.580+0000 7fd93ca38640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd9240a2b90 0x7fd9240a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd928009b60 tx=0x7fd928005f90 comp rx=0 tx=0).stop 2024-05-09T08:48:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.580+0000 7fd93ca38640 1 -- 172.21.15.98:0/1087219207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd938107570 msgr2=0x7fd938100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.580+0000 7fd93ca38640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd938107570 0x7fd938100800 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd930031ae0 tx=0x7fd930031ee0 comp rx=0 tx=0).stop 2024-05-09T08:48:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.580+0000 7fd93ca38640 1 -- 172.21.15.98:0/1087219207 shutdown_connections 2024-05-09T08:48:17.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.580+0000 7fd93ca38640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd9240a2b90 0x7fd9240a5050 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:17.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.580+0000 7fd93ca38640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd938107570 0x7fd938100800 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:17.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.580+0000 7fd93ca38640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd938104c40 0x7fd9381002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:17.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.580+0000 7fd93ca38640 1 --2- 172.21.15.98:0/1087219207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd938102310 0x7fd938101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:17.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.580+0000 7fd93ca38640 1 -- 172.21.15.98:0/1087219207 >> 172.21.15.98:0/1087219207 conn(0x7fd9380fbd20 msgr2=0x7fd938102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:17.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.580+0000 7fd93ca38640 1 -- 172.21.15.98:0/1087219207 shutdown_connections 2024-05-09T08:48:17.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.580+0000 7fd93ca38640 1 -- 172.21.15.98:0/1087219207 wait complete. 2024-05-09T08:48:17.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T08:48:17.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:17.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:48:17.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:17 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3663786078' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-09T08:48:17.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:17 smithi098 ceph-mon[30092]: osdmap e83: 12 total, 12 up, 12 in 2024-05-09T08:48:17.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:17 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1087219207' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T08:48:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:17.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:48:17.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:17 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3663786078' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-09T08:48:17.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:17 smithi142 ceph-mon[34625]: osdmap e83: 12 total, 12 up, 12 in 2024-05-09T08:48:17.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:17 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1087219207' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T08:48:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.707+0000 7f8b47fff640 1 -- 172.21.15.98:0/4143617074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b48101840 msgr2=0x7f8b48101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.707+0000 7f8b47fff640 1 --2- 172.21.15.98:0/4143617074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b48101840 0x7f8b48101cc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c009900 tx=0x7f8b3c02a410 comp rx=0 tx=0).stop 2024-05-09T08:48:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.707+0000 7f8b47fff640 1 -- 172.21.15.98:0/4143617074 shutdown_connections 2024-05-09T08:48:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.707+0000 7f8b47fff640 1 --2- 172.21.15.98:0/4143617074 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b48102200 0x7f8b48108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.707+0000 7f8b47fff640 1 --2- 172.21.15.98:0/4143617074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b48101840 0x7f8b48101cc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.707+0000 7f8b47fff640 1 --2- 172.21.15.98:0/4143617074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b48100650 0x7f8b48100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.707+0000 7f8b47fff640 1 -- 172.21.15.98:0/4143617074 >> 172.21.15.98:0/4143617074 conn(0x7f8b480fbbe0 msgr2=0x7f8b480fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.707+0000 7f8b47fff640 1 -- 172.21.15.98:0/4143617074 shutdown_connections 2024-05-09T08:48:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.707+0000 7f8b47fff640 1 -- 172.21.15.98:0/4143617074 wait complete. 2024-05-09T08:48:17.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.708+0000 7f8b47fff640 1 Processor -- start 2024-05-09T08:48:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.708+0000 7f8b47fff640 1 -- start start 2024-05-09T08:48:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.708+0000 7f8b47fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b48100650 0x7f8b481979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.708+0000 7f8b47fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b48101840 0x7f8b48197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.708+0000 7f8b47fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b48102200 0x7f8b4819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.708+0000 7f8b47fff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4810c2c0 con 0x7f8b48102200 2024-05-09T08:48:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.708+0000 7f8b47fff640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4810c140 con 0x7f8b48100650 2024-05-09T08:48:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.708+0000 7f8b47fff640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4810c440 con 0x7f8b48101840 2024-05-09T08:48:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b477fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b48102200 0x7f8b4819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b477fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b48102200 0x7f8b4819ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39310/0 (socket says 172.21.15.98:39310) 2024-05-09T08:48:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b477fe640 1 -- 172.21.15.98:0/4168163315 learned_addr learned my addr 172.21.15.98:0/4168163315 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b467fc640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b48101840 0x7f8b48197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b46ffd640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b48100650 0x7f8b481979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b477fe640 1 -- 172.21.15.98:0/4168163315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b48101840 msgr2=0x7f8b48197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b477fe640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b48101840 0x7f8b48197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b477fe640 1 -- 172.21.15.98:0/4168163315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b48100650 msgr2=0x7f8b481979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b477fe640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b48100650 0x7f8b481979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b477fe640 1 -- 172.21.15.98:0/4168163315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4819f5d0 con 0x7f8b48102200 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b46ffd640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b48100650 0x7f8b481979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b477fe640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b48102200 0x7f8b4819ef60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000d750 tx=0x7f8b4000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b4c9fe640 1 -- 172.21.15.98:0/4168163315 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b40017070 con 0x7f8b48102200 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b47fff640 1 -- 172.21.15.98:0/4168163315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b4819f8c0 con 0x7f8b48102200 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b4c9fe640 1 -- 172.21.15.98:0/4168163315 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b40005500 con 0x7f8b48102200 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b4c9fe640 1 -- 172.21.15.98:0/4168163315 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b40004d90 con 0x7f8b48102200 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.709+0000 7f8b47fff640 1 -- 172.21.15.98:0/4168163315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b4819fd90 con 0x7f8b48102200 2024-05-09T08:48:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.710+0000 7f8b47fff640 1 -- 172.21.15.98:0/4168163315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b4819fff0 con 0x7f8b48102200 2024-05-09T08:48:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.711+0000 7f8b4c9fe640 1 -- 172.21.15.98:0/4168163315 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8b40004f30 con 0x7f8b48102200 2024-05-09T08:48:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.712+0000 7f8b4c9fe640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8b300a2b90 0x7f8b300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.712+0000 7f8b46ffd640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8b300a2b90 0x7f8b300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.712+0000 7f8b4c9fe640 1 -- 172.21.15.98:0/4168163315 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f8b400ce980 con 0x7f8b48102200 2024-05-09T08:48:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.712+0000 7f8b46ffd640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8b300a2b90 0x7f8b300a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8b481016a0 tx=0x7f8b38008370 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.716+0000 7f8b4c9fe640 1 -- 172.21.15.98:0/4168163315 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8b4009ba90 con 0x7f8b48102200 2024-05-09T08:48:17.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:17.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:48:17.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:48:17.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:17 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3663786078' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-09T08:48:17.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:17 smithi105 ceph-mon[34498]: osdmap e83: 12 total, 12 up, 12 in 2024-05-09T08:48:17.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:17 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1087219207' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T08:48:17.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:17.931+0000 7f8b47fff640 1 -- 172.21.15.98:0/4168163315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f8b481a02e0 con 0x7f8b48102200 2024-05-09T08:48:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:18 smithi142 ceph-mon[34625]: pgmap v283: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:18 smithi142 ceph-mon[34625]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:48:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:18 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1087219207' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-09T08:48:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:18 smithi142 ceph-mon[34625]: osdmap e84: 12 total, 12 up, 12 in 2024-05-09T08:48:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:18 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/4168163315' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-09T08:48:18.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:18 smithi142 ceph-mon[34625]: daemon mds.h assigned to filesystem cephfs as rank 0 2024-05-09T08:48:18.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.725+0000 7f8b4c9fe640 1 -- 172.21.15.98:0/4168163315 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f8b4009b390 con 0x7f8b48102200 2024-05-09T08:48:18.725 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 2 and data pool 3 2024-05-09T08:48:18.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.729+0000 7f8b47fff640 1 -- 172.21.15.98:0/4168163315 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8b300a2b90 msgr2=0x7f8b300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:18.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.729+0000 7f8b47fff640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8b300a2b90 0x7f8b300a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8b481016a0 tx=0x7f8b38008370 comp rx=0 tx=0).stop 2024-05-09T08:48:18.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.729+0000 7f8b47fff640 1 -- 172.21.15.98:0/4168163315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b48102200 msgr2=0x7f8b4819ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:18.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.729+0000 7f8b47fff640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b48102200 0x7f8b4819ef60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000d750 tx=0x7f8b4000dc10 comp rx=0 tx=0).stop 2024-05-09T08:48:18.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.729+0000 7f8b47fff640 1 -- 172.21.15.98:0/4168163315 shutdown_connections 2024-05-09T08:48:18.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.729+0000 7f8b47fff640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8b300a2b90 0x7f8b300a5050 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:18.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.729+0000 7f8b47fff640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b48102200 0x7f8b4819ef60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:18.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.729+0000 7f8b47fff640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b48101840 0x7f8b48197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:18.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.729+0000 7f8b47fff640 1 --2- 172.21.15.98:0/4168163315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b48100650 0x7f8b481979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:18.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.729+0000 7f8b47fff640 1 -- 172.21.15.98:0/4168163315 >> 172.21.15.98:0/4168163315 conn(0x7f8b480fbbe0 msgr2=0x7f8b48069db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:18.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.729+0000 7f8b47fff640 1 -- 172.21.15.98:0/4168163315 shutdown_connections 2024-05-09T08:48:18.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.729+0000 7f8b47fff640 1 -- 172.21.15.98:0/4168163315 wait complete. 2024-05-09T08:48:18.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:18 smithi105 ceph-mon[34498]: pgmap v283: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:18.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:18 smithi105 ceph-mon[34498]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:48:18.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:18 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1087219207' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-09T08:48:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:18 smithi105 ceph-mon[34498]: osdmap e84: 12 total, 12 up, 12 in 2024-05-09T08:48:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:18 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/4168163315' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-09T08:48:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:18 smithi105 ceph-mon[34498]: daemon mds.h assigned to filesystem cephfs as rank 0 2024-05-09T08:48:18.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:18 smithi098 ceph-mon[30092]: pgmap v283: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:48:18.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:18 smithi098 ceph-mon[30092]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:48:18.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:18 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1087219207' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-09T08:48:18.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:18 smithi098 ceph-mon[30092]: osdmap e84: 12 total, 12 up, 12 in 2024-05-09T08:48:18.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:18 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4168163315' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-09T08:48:18.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:18 smithi098 ceph-mon[30092]: daemon mds.h assigned to filesystem cephfs as rank 0 2024-05-09T08:48:18.743 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-05-09T08:48:18.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-05-09T08:48:18.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.852+0000 7fbf90942640 1 -- 172.21.15.98:0/3762697777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c101960 msgr2=0x7fbf8c101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:18.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.852+0000 7fbf90942640 1 --2- 172.21.15.98:0/3762697777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c101960 0x7fbf8c101de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbf74009900 tx=0x7fbf7402a410 comp rx=0 tx=0).stop 2024-05-09T08:48:18.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.852+0000 7fbf90942640 1 -- 172.21.15.98:0/3762697777 shutdown_connections 2024-05-09T08:48:18.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.852+0000 7fbf90942640 1 --2- 172.21.15.98:0/3762697777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf8c102320 0x7fbf8c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:18.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.852+0000 7fbf90942640 1 --2- 172.21.15.98:0/3762697777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c101960 0x7fbf8c101de0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:18.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.852+0000 7fbf90942640 1 --2- 172.21.15.98:0/3762697777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf8c100770 0x7fbf8c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:18.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.852+0000 7fbf90942640 1 -- 172.21.15.98:0/3762697777 >> 172.21.15.98:0/3762697777 conn(0x7fbf8c0fbce0 msgr2=0x7fbf8c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:18.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.852+0000 7fbf90942640 1 -- 172.21.15.98:0/3762697777 shutdown_connections 2024-05-09T08:48:18.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.852+0000 7fbf90942640 1 -- 172.21.15.98:0/3762697777 wait complete. 2024-05-09T08:48:18.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf90942640 1 Processor -- start 2024-05-09T08:48:18.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf90942640 1 -- start start 2024-05-09T08:48:18.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf90942640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf8c100770 0x7fbf8c196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:18.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf90942640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c101960 0x7fbf8c197000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:18.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf90942640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf8c102320 0x7fbf8c19e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:18.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf90942640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c10c3f0 con 0x7fbf8c101960 2024-05-09T08:48:18.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf90942640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c10c270 con 0x7fbf8c100770 2024-05-09T08:48:18.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf90942640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c10c570 con 0x7fbf8c102320 2024-05-09T08:48:18.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf83fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c101960 0x7fbf8c197000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf83fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c101960 0x7fbf8c197000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39320/0 (socket says 172.21.15.98:39320) 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf83fff640 1 -- 172.21.15.98:0/3915685245 learned_addr learned my addr 172.21.15.98:0/3915685245 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf8a59c640 1 --2- 172.21.15.98:0/3915685245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf8c100770 0x7fbf8c196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf8ad9d640 1 --2- 172.21.15.98:0/3915685245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf8c102320 0x7fbf8c19e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf83fff640 1 -- 172.21.15.98:0/3915685245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf8c102320 msgr2=0x7fbf8c19e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf83fff640 1 --2- 172.21.15.98:0/3915685245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf8c102320 0x7fbf8c19e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf83fff640 1 -- 172.21.15.98:0/3915685245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf8c100770 msgr2=0x7fbf8c196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf83fff640 1 --2- 172.21.15.98:0/3915685245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf8c100770 0x7fbf8c196ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf83fff640 1 -- 172.21.15.98:0/3915685245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8c19e6e0 con 0x7fbf8c101960 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.853+0000 7fbf83fff640 1 --2- 172.21.15.98:0/3915685245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c101960 0x7fbf8c197000 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbf74009230 tx=0x7fbf74004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.854+0000 7fbf837fe640 1 -- 172.21.15.98:0/3915685245 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7403f030 con 0x7fbf8c101960 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.854+0000 7fbf837fe640 1 -- 172.21.15.98:0/3915685245 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf74003ce0 con 0x7fbf8c101960 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.854+0000 7fbf90942640 1 -- 172.21.15.98:0/3915685245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8c19e970 con 0x7fbf8c101960 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.854+0000 7fbf837fe640 1 -- 172.21.15.98:0/3915685245 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf74035070 con 0x7fbf8c101960 2024-05-09T08:48:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.854+0000 7fbf90942640 1 -- 172.21.15.98:0/3915685245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf8c19eeb0 con 0x7fbf8c101960 2024-05-09T08:48:18.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.855+0000 7fbf90942640 1 -- 172.21.15.98:0/3915685245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf8c107ae0 con 0x7fbf8c101960 2024-05-09T08:48:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.856+0000 7fbf837fe640 1 -- 172.21.15.98:0/3915685245 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbf74042030 con 0x7fbf8c101960 2024-05-09T08:48:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.857+0000 7fbf837fe640 1 --2- 172.21.15.98:0/3915685245 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbf780a2c10 0x7fbf780a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.857+0000 7fbf8a59c640 1 --2- 172.21.15.98:0/3915685245 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbf780a2c10 0x7fbf780a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.857+0000 7fbf837fe640 1 -- 172.21.15.98:0/3915685245 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fbf740ef8b0 con 0x7fbf8c101960 2024-05-09T08:48:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.857+0000 7fbf8a59c640 1 --2- 172.21.15.98:0/3915685245 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbf780a2c10 0x7fbf780a50d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c0058c0 tx=0x7fbf7c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:18.860+0000 7fbf837fe640 1 -- 172.21.15.98:0/3915685245 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbf740bca20 con 0x7fbf8c101960 2024-05-09T08:48:19.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.052+0000 7fbf90942640 1 -- 172.21.15.98:0/3915685245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fbf8c02d410 con 0x7fbf8c101960 2024-05-09T08:48:19.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.346+0000 7fbf837fe640 1 -- 172.21.15.98:0/3915685245 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) ==== 157+0+0 (secure 0 0 0) 0x7fbf740bc320 con 0x7fbf8c101960 2024-05-09T08:48:19.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.349+0000 7fbf90942640 1 -- 172.21.15.98:0/3915685245 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbf780a2c10 msgr2=0x7fbf780a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:19.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.349+0000 7fbf90942640 1 --2- 172.21.15.98:0/3915685245 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbf780a2c10 0x7fbf780a50d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c0058c0 tx=0x7fbf7c00a2d0 comp rx=0 tx=0).stop 2024-05-09T08:48:19.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.349+0000 7fbf90942640 1 -- 172.21.15.98:0/3915685245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c101960 msgr2=0x7fbf8c197000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:19.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.349+0000 7fbf90942640 1 --2- 172.21.15.98:0/3915685245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c101960 0x7fbf8c197000 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbf74009230 tx=0x7fbf74004360 comp rx=0 tx=0).stop 2024-05-09T08:48:19.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.349+0000 7fbf90942640 1 -- 172.21.15.98:0/3915685245 shutdown_connections 2024-05-09T08:48:19.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.349+0000 7fbf90942640 1 --2- 172.21.15.98:0/3915685245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf8c102320 0x7fbf8c19e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:19.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.349+0000 7fbf90942640 1 --2- 172.21.15.98:0/3915685245 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbf780a2c10 0x7fbf780a50d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:19.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.349+0000 7fbf90942640 1 --2- 172.21.15.98:0/3915685245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c101960 0x7fbf8c197000 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:19.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.349+0000 7fbf90942640 1 --2- 172.21.15.98:0/3915685245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf8c100770 0x7fbf8c196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:19.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.349+0000 7fbf90942640 1 -- 172.21.15.98:0/3915685245 >> 172.21.15.98:0/3915685245 conn(0x7fbf8c0fbce0 msgr2=0x7fbf8c104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:19.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.350+0000 7fbf90942640 1 -- 172.21.15.98:0/3915685245 shutdown_connections 2024-05-09T08:48:19.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.350+0000 7fbf90942640 1 -- 172.21.15.98:0/3915685245 wait complete. 2024-05-09T08:48:19.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T08:48:19.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.474+0000 7f6ff0476640 1 -- 172.21.15.98:0/1010670908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe8100770 msgr2=0x7f6fe8100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:19.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.474+0000 7f6ff0476640 1 --2- 172.21.15.98:0/1010670908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe8100770 0x7f6fe8100b90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009900 tx=0x7f6fe402a410 comp rx=0 tx=0).stop 2024-05-09T08:48:19.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 -- 172.21.15.98:0/1010670908 shutdown_connections 2024-05-09T08:48:19.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 --2- 172.21.15.98:0/1010670908 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe8102320 0x7f6fe8108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:19.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 --2- 172.21.15.98:0/1010670908 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6fe8101960 0x7f6fe8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:19.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 --2- 172.21.15.98:0/1010670908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe8100770 0x7f6fe8100b90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:19.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 -- 172.21.15.98:0/1010670908 >> 172.21.15.98:0/1010670908 conn(0x7f6fe80fbce0 msgr2=0x7f6fe80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:19.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 -- 172.21.15.98:0/1010670908 shutdown_connections 2024-05-09T08:48:19.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 -- 172.21.15.98:0/1010670908 wait complete. 2024-05-09T08:48:19.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 Processor -- start 2024-05-09T08:48:19.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 -- start start 2024-05-09T08:48:19.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe8100770 0x7f6fe8197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:19.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6fe8101960 0x7f6fe8198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:19.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe8102320 0x7f6fe819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:19.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe810d5d0 con 0x7f6fe8102320 2024-05-09T08:48:19.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe810d450 con 0x7f6fe8101960 2024-05-09T08:48:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.475+0000 7f6ff0476640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe810d750 con 0x7f6fe8100770 2024-05-09T08:48:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6feea13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe8102320 0x7f6fe819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6feea13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe8102320 0x7f6fe819f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39330/0 (socket says 172.21.15.98:39330) 2024-05-09T08:48:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6feea13640 1 -- 172.21.15.98:0/2779832070 learned_addr learned my addr 172.21.15.98:0/2779832070 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6fee212640 1 --2- 172.21.15.98:0/2779832070 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe8100770 0x7f6fe8197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6feea13640 1 -- 172.21.15.98:0/2779832070 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe8100770 msgr2=0x7f6fe8197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6feea13640 1 --2- 172.21.15.98:0/2779832070 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe8100770 0x7f6fe8197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6feea13640 1 -- 172.21.15.98:0/2779832070 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6fe8101960 msgr2=0x7f6fe8198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6feda11640 1 --2- 172.21.15.98:0/2779832070 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6fe8101960 0x7f6fe8198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6feea13640 1 --2- 172.21.15.98:0/2779832070 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6fe8101960 0x7f6fe8198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6feea13640 1 -- 172.21.15.98:0/2779832070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe819f740 con 0x7f6fe8102320 2024-05-09T08:48:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6feea13640 1 --2- 172.21.15.98:0/2779832070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe8102320 0x7f6fe819f0d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6fe000b970 tx=0x7f6fe000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6fdf7fe640 1 -- 172.21.15.98:0/2779832070 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0019070 con 0x7f6fe8102320 2024-05-09T08:48:19.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6fdf7fe640 1 -- 172.21.15.98:0/2779832070 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe000cb40 con 0x7f6fe8102320 2024-05-09T08:48:19.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6ff0476640 1 -- 172.21.15.98:0/2779832070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe819fa30 con 0x7f6fe8102320 2024-05-09T08:48:19.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6fdf7fe640 1 -- 172.21.15.98:0/2779832070 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0007650 con 0x7f6fe8102320 2024-05-09T08:48:19.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.476+0000 7f6ff0476640 1 -- 172.21.15.98:0/2779832070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe81a0000 con 0x7f6fe8102320 2024-05-09T08:48:19.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.477+0000 7f6ff0476640 1 -- 172.21.15.98:0/2779832070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe81079c0 con 0x7f6fe8102320 2024-05-09T08:48:19.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.478+0000 7f6fdf7fe640 1 -- 172.21.15.98:0/2779832070 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6fe001c070 con 0x7f6fe8102320 2024-05-09T08:48:19.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.479+0000 7f6fdf7fe640 1 --2- 172.21.15.98:0/2779832070 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6fd40a2b90 0x7f6fd40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:19.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.479+0000 7f6fee212640 1 --2- 172.21.15.98:0/2779832070 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6fd40a2b90 0x7f6fd40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:19.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.479+0000 7f6fdf7fe640 1 -- 172.21.15.98:0/2779832070 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8277+0+0 (secure 0 0 0) 0x7f6fe00ce6b0 con 0x7f6fe8102320 2024-05-09T08:48:19.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.479+0000 7f6fee212640 1 --2- 172.21.15.98:0/2779832070 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6fd40a2b90 0x7f6fd40a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40314e0 tx=0x7f6fe4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:19.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.483+0000 7f6fdf7fe640 1 -- 172.21.15.98:0/2779832070 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6fe009b5e0 con 0x7f6fe8102320 2024-05-09T08:48:19.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:19 smithi098 ceph-mon[30092]: osdmap e85: 12 total, 12 up, 12 in 2024-05-09T08:48:19.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:19 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4168163315' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-09T08:48:19.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:19 smithi098 ceph-mon[30092]: fsmap cephfs:1 {0=h=up:creating} 11 up:standby 2024-05-09T08:48:19.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:19 smithi098 ceph-mon[30092]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-05-09T08:48:19.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:19 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3915685245' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-09T08:48:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:19 smithi142 ceph-mon[34625]: osdmap e85: 12 total, 12 up, 12 in 2024-05-09T08:48:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:19 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/4168163315' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-09T08:48:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:19 smithi142 ceph-mon[34625]: fsmap cephfs:1 {0=h=up:creating} 11 up:standby 2024-05-09T08:48:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:19 smithi142 ceph-mon[34625]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-05-09T08:48:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:19 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3915685245' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-09T08:48:19.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:19.660+0000 7f6ff0476640 1 -- 172.21.15.98:0/2779832070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f6fe8062430 con 0x7f6fe8102320 2024-05-09T08:48:19.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:19 smithi105 ceph-mon[34498]: osdmap e85: 12 total, 12 up, 12 in 2024-05-09T08:48:19.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:19 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/4168163315' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-09T08:48:19.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:19 smithi105 ceph-mon[34498]: fsmap cephfs:1 {0=h=up:creating} 11 up:standby 2024-05-09T08:48:19.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:19 smithi105 ceph-mon[34498]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-05-09T08:48:19.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:19 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3915685245' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-09T08:48:20.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:20 smithi142 ceph-mon[34625]: pgmap v286: 97 pgs: 12 creating+peering, 21 active+clean, 64 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-09T08:48:20.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:20 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3915685245' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-09T08:48:20.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:20 smithi142 ceph-mon[34625]: osdmap e86: 12 total, 12 up, 12 in 2024-05-09T08:48:20.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:20 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2779832070' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-09T08:48:20.910 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:20 smithi142 ceph-mon[34625]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:48:20.910 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:20 smithi142 ceph-mon[34625]: Cluster is now healthy 2024-05-09T08:48:20.910 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:20 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] up:active 2024-05-09T08:48:20.910 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:20 smithi142 ceph-mon[34625]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:20.910 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:20 smithi142 ceph-mon[34625]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:20.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:20 smithi105 ceph-mon[34498]: pgmap v286: 97 pgs: 12 creating+peering, 21 active+clean, 64 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-09T08:48:20.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:20 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3915685245' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-09T08:48:20.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:20 smithi105 ceph-mon[34498]: osdmap e86: 12 total, 12 up, 12 in 2024-05-09T08:48:20.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:20 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2779832070' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-09T08:48:20.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:20 smithi105 ceph-mon[34498]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:48:20.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:20 smithi105 ceph-mon[34498]: Cluster is now healthy 2024-05-09T08:48:20.988 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:20 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] up:active 2024-05-09T08:48:20.989 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:20 smithi105 ceph-mon[34498]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:20.989 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:20 smithi105 ceph-mon[34498]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:21.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:20 smithi098 ceph-mon[30092]: pgmap v286: 97 pgs: 12 creating+peering, 21 active+clean, 64 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-09T08:48:21.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:20 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3915685245' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-09T08:48:21.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:20 smithi098 ceph-mon[30092]: osdmap e86: 12 total, 12 up, 12 in 2024-05-09T08:48:21.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:20 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2779832070' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-09T08:48:21.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:20 smithi098 ceph-mon[30092]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:48:21.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:20 smithi098 ceph-mon[30092]: Cluster is now healthy 2024-05-09T08:48:21.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:20 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] up:active 2024-05-09T08:48:21.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:20 smithi098 ceph-mon[30092]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:21.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:20 smithi098 ceph-mon[30092]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:21.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.357+0000 7f6fdf7fe640 1 -- 172.21.15.98:0/2779832070 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) ==== 228+0+0 (secure 0 0 0) 0x7f6fe009aee0 con 0x7f6fe8102320 2024-05-09T08:48:21.356 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data_ec' created 2024-05-09T08:48:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.360+0000 7f6ff0476640 1 -- 172.21.15.98:0/2779832070 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6fd40a2b90 msgr2=0x7f6fd40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.360+0000 7f6ff0476640 1 --2- 172.21.15.98:0/2779832070 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6fd40a2b90 0x7f6fd40a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40314e0 tx=0x7f6fe4000bc0 comp rx=0 tx=0).stop 2024-05-09T08:48:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.360+0000 7f6ff0476640 1 -- 172.21.15.98:0/2779832070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe8102320 msgr2=0x7f6fe819f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.360+0000 7f6ff0476640 1 --2- 172.21.15.98:0/2779832070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe8102320 0x7f6fe819f0d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6fe000b970 tx=0x7f6fe000be30 comp rx=0 tx=0).stop 2024-05-09T08:48:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.360+0000 7f6ff0476640 1 -- 172.21.15.98:0/2779832070 shutdown_connections 2024-05-09T08:48:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.360+0000 7f6ff0476640 1 --2- 172.21.15.98:0/2779832070 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6fd40a2b90 0x7f6fd40a5050 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.360+0000 7f6ff0476640 1 --2- 172.21.15.98:0/2779832070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe8102320 0x7f6fe819f0d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.360+0000 7f6ff0476640 1 --2- 172.21.15.98:0/2779832070 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6fe8101960 0x7f6fe8198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.360+0000 7f6ff0476640 1 --2- 172.21.15.98:0/2779832070 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe8100770 0x7f6fe8197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.360+0000 7f6ff0476640 1 -- 172.21.15.98:0/2779832070 >> 172.21.15.98:0/2779832070 conn(0x7f6fe80fbce0 msgr2=0x7f6fe8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.360+0000 7f6ff0476640 1 -- 172.21.15.98:0/2779832070 shutdown_connections 2024-05-09T08:48:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.360+0000 7f6ff0476640 1 -- 172.21.15.98:0/2779832070 wait complete. 2024-05-09T08:48:21.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-05-09T08:48:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.489+0000 7f1034458640 1 -- 172.21.15.98:0/3365314964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f102c0ff4c0 msgr2=0x7f102c0ff8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.489+0000 7f1034458640 1 --2- 172.21.15.98:0/3365314964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f102c0ff4c0 0x7f102c0ff8e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1028009ca0 tx=0x7f102802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.489+0000 7f1034458640 1 -- 172.21.15.98:0/3365314964 shutdown_connections 2024-05-09T08:48:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.489+0000 7f1034458640 1 --2- 172.21.15.98:0/3365314964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f102c107540 0x7f102c109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.489+0000 7f1034458640 1 --2- 172.21.15.98:0/3365314964 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f102c0ffeb0 0x7f102c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.489+0000 7f1034458640 1 --2- 172.21.15.98:0/3365314964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f102c0ff4c0 0x7f102c0ff8e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.489+0000 7f1034458640 1 -- 172.21.15.98:0/3365314964 >> 172.21.15.98:0/3365314964 conn(0x7f102c0fb030 msgr2=0x7f102c0fd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.489+0000 7f1034458640 1 -- 172.21.15.98:0/3365314964 shutdown_connections 2024-05-09T08:48:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.489+0000 7f1034458640 1 -- 172.21.15.98:0/3365314964 wait complete. 2024-05-09T08:48:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.489+0000 7f1034458640 1 Processor -- start 2024-05-09T08:48:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f1034458640 1 -- start start 2024-05-09T08:48:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f1034458640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f102c0ff4c0 0x7f102c197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f1034458640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f102c0ffeb0 0x7f102c198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f1034458640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f102c107540 0x7f102c19f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f1034458640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f102c10d5d0 con 0x7f102c107540 2024-05-09T08:48:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f1034458640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f102c10d450 con 0x7f102c0ffeb0 2024-05-09T08:48:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f1034458640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f102c10d750 con 0x7f102c0ff4c0 2024-05-09T08:48:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f10329f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f102c107540 0x7f102c19f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f10329f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f102c107540 0x7f102c19f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39360/0 (socket says 172.21.15.98:39360) 2024-05-09T08:48:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f10329f5640 1 -- 172.21.15.98:0/750176395 learned_addr learned my addr 172.21.15.98:0/750176395 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f10321f4640 1 --2- 172.21.15.98:0/750176395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f102c0ff4c0 0x7f102c197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f10329f5640 1 -- 172.21.15.98:0/750176395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f102c0ff4c0 msgr2=0x7f102c197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f10329f5640 1 --2- 172.21.15.98:0/750176395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f102c0ff4c0 0x7f102c197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f10329f5640 1 -- 172.21.15.98:0/750176395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f102c0ffeb0 msgr2=0x7f102c198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f10329f5640 1 --2- 172.21.15.98:0/750176395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f102c0ffeb0 0x7f102c198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f10329f5640 1 -- 172.21.15.98:0/750176395 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102c19f930 con 0x7f102c107540 2024-05-09T08:48:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f10329f5640 1 --2- 172.21.15.98:0/750176395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f102c107540 0x7f102c19f290 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f102400b3f0 tx=0x7f102400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f10237fe640 1 -- 172.21.15.98:0/750176395 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1024015020 con 0x7f102c107540 2024-05-09T08:48:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f1034458640 1 -- 172.21.15.98:0/750176395 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f102c19fc20 con 0x7f102c107540 2024-05-09T08:48:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f10237fe640 1 -- 172.21.15.98:0/750176395 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1024011d10 con 0x7f102c107540 2024-05-09T08:48:21.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.490+0000 7f10237fe640 1 -- 172.21.15.98:0/750176395 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102400f2e0 con 0x7f102c107540 2024-05-09T08:48:21.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.491+0000 7f1034458640 1 -- 172.21.15.98:0/750176395 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f102c1a0240 con 0x7f102c107540 2024-05-09T08:48:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.492+0000 7f1034458640 1 -- 172.21.15.98:0/750176395 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f102c105e10 con 0x7f102c107540 2024-05-09T08:48:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.493+0000 7f10237fe640 1 -- 172.21.15.98:0/750176395 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1024018020 con 0x7f102c107540 2024-05-09T08:48:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.494+0000 7f10237fe640 1 --2- 172.21.15.98:0/750176395 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f10180a2b90 0x7f10180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.494+0000 7f10321f4640 1 --2- 172.21.15.98:0/750176395 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f10180a2b90 0x7f10180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.494+0000 7f10237fe640 1 -- 172.21.15.98:0/750176395 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f10240cf130 con 0x7f102c107540 2024-05-09T08:48:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.494+0000 7f10321f4640 1 --2- 172.21.15.98:0/750176395 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f10180a2b90 0x7f10180a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1028006010 tx=0x7f1028005f80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.498+0000 7f10237fe640 1 -- 172.21.15.98:0/750176395 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f102401f020 con 0x7f102c107540 2024-05-09T08:48:21.593 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 08:48:21 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h[64536]: 2024-05-09T08:48:21.260+0000 7f19d652a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T08:48:21.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:21.676+0000 7f1034458640 1 -- 172.21.15.98:0/750176395 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f102c02d1c0 con 0x7f102c107540 2024-05-09T08:48:21.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:21 smithi098 ceph-mon[30092]: osdmap e87: 12 total, 12 up, 12 in 2024-05-09T08:48:21.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:21 smithi098 ceph-mon[30092]: pgmap v289: 97 pgs: 12 creating+peering, 75 active+clean, 10 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 7 op/s 2024-05-09T08:48:21.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:21 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2779832070' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-09T08:48:21.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:21 smithi098 ceph-mon[30092]: osdmap e88: 12 total, 12 up, 12 in 2024-05-09T08:48:21.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:21 smithi142 ceph-mon[34625]: osdmap e87: 12 total, 12 up, 12 in 2024-05-09T08:48:21.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:21 smithi142 ceph-mon[34625]: pgmap v289: 97 pgs: 12 creating+peering, 75 active+clean, 10 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 7 op/s 2024-05-09T08:48:21.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:21 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2779832070' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-09T08:48:21.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:21 smithi142 ceph-mon[34625]: osdmap e88: 12 total, 12 up, 12 in 2024-05-09T08:48:21.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:21 smithi105 ceph-mon[34498]: osdmap e87: 12 total, 12 up, 12 in 2024-05-09T08:48:21.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:21 smithi105 ceph-mon[34498]: pgmap v289: 97 pgs: 12 creating+peering, 75 active+clean, 10 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 7 op/s 2024-05-09T08:48:21.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:21 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2779832070' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-09T08:48:21.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:21 smithi105 ceph-mon[34498]: osdmap e88: 12 total, 12 up, 12 in 2024-05-09T08:48:22.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.359+0000 7f10237fe640 1 -- 172.21.15.98:0/750176395 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v89) ==== 169+0+0 (secure 0 0 0) 0x7f102409be70 con 0x7f102c107540 2024-05-09T08:48:22.358 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 4 allow_ec_overwrites to true 2024-05-09T08:48:22.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.362+0000 7f1034458640 1 -- 172.21.15.98:0/750176395 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f10180a2b90 msgr2=0x7f10180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:22.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.362+0000 7f1034458640 1 --2- 172.21.15.98:0/750176395 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f10180a2b90 0x7f10180a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1028006010 tx=0x7f1028005f80 comp rx=0 tx=0).stop 2024-05-09T08:48:22.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.362+0000 7f1034458640 1 -- 172.21.15.98:0/750176395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f102c107540 msgr2=0x7f102c19f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.362+0000 7f1034458640 1 --2- 172.21.15.98:0/750176395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f102c107540 0x7f102c19f290 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f102400b3f0 tx=0x7f102400b8b0 comp rx=0 tx=0).stop 2024-05-09T08:48:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.363+0000 7f1034458640 1 -- 172.21.15.98:0/750176395 shutdown_connections 2024-05-09T08:48:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.363+0000 7f1034458640 1 --2- 172.21.15.98:0/750176395 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f10180a2b90 0x7f10180a5050 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.363+0000 7f1034458640 1 --2- 172.21.15.98:0/750176395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f102c107540 0x7f102c19f290 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.363+0000 7f1034458640 1 --2- 172.21.15.98:0/750176395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f102c0ffeb0 0x7f102c198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.363+0000 7f1034458640 1 --2- 172.21.15.98:0/750176395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f102c0ff4c0 0x7f102c197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.363+0000 7f1034458640 1 -- 172.21.15.98:0/750176395 >> 172.21.15.98:0/750176395 conn(0x7f102c0fb030 msgr2=0x7f102c102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.363+0000 7f1034458640 1 -- 172.21.15.98:0/750176395 shutdown_connections 2024-05-09T08:48:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.363+0000 7f1034458640 1 -- 172.21.15.98:0/750176395 wait complete. 2024-05-09T08:48:22.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-05-09T08:48:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.493+0000 7fc2fefc1640 1 -- 172.21.15.98:0/1363389094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8102320 msgr2=0x7fc2f8108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.493+0000 7fc2fefc1640 1 --2- 172.21.15.98:0/1363389094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8102320 0x7fc2f8108bb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009900 tx=0x7fc2f002a410 comp rx=0 tx=0).stop 2024-05-09T08:48:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 -- 172.21.15.98:0/1363389094 shutdown_connections 2024-05-09T08:48:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 --2- 172.21.15.98:0/1363389094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8102320 0x7fc2f8108bb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 --2- 172.21.15.98:0/1363389094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2f8101960 0x7fc2f8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:22.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 --2- 172.21.15.98:0/1363389094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2f8100770 0x7fc2f8100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 -- 172.21.15.98:0/1363389094 >> 172.21.15.98:0/1363389094 conn(0x7fc2f80fbce0 msgr2=0x7fc2f80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 -- 172.21.15.98:0/1363389094 shutdown_connections 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 -- 172.21.15.98:0/1363389094 wait complete. 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 Processor -- start 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 -- start start 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8100770 0x7fc2f81969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2f8101960 0x7fc2f8196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2f8102320 0x7fc2f819df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f810c3d0 con 0x7fc2f8100770 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f810c250 con 0x7fc2f8101960 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fefc1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f810c550 con 0x7fc2f8102320 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fcd5d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8100770 0x7fc2f81969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fcd5d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8100770 0x7fc2f81969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39364/0 (socket says 172.21.15.98:39364) 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.494+0000 7fc2fcd5d640 1 -- 172.21.15.98:0/2863358417 learned_addr learned my addr 172.21.15.98:0/2863358417 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.495+0000 7fc2f7fff640 1 --2- 172.21.15.98:0/2863358417 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2f8101960 0x7fc2f8196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:22.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.495+0000 7fc2fd55e640 1 --2- 172.21.15.98:0/2863358417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2f8102320 0x7fc2f819df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:22.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.495+0000 7fc2fcd5d640 1 -- 172.21.15.98:0/2863358417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2f8102320 msgr2=0x7fc2f819df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:22.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.495+0000 7fc2fcd5d640 1 --2- 172.21.15.98:0/2863358417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2f8102320 0x7fc2f819df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:22.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.495+0000 7fc2fcd5d640 1 -- 172.21.15.98:0/2863358417 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2f8101960 msgr2=0x7fc2f8196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:22.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.495+0000 7fc2fcd5d640 1 --2- 172.21.15.98:0/2863358417 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2f8101960 0x7fc2f8196f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:22.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.495+0000 7fc2fcd5d640 1 -- 172.21.15.98:0/2863358417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f819e5e0 con 0x7fc2f8100770 2024-05-09T08:48:22.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.495+0000 7fc2fcd5d640 1 --2- 172.21.15.98:0/2863358417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8100770 0x7fc2f81969c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800ea80 tx=0x7fc2e800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:22.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.495+0000 7fc2f5ffb640 1 -- 172.21.15.98:0/2863358417 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e800cd80 con 0x7fc2f8100770 2024-05-09T08:48:22.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.495+0000 7fc2f5ffb640 1 -- 172.21.15.98:0/2863358417 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2e8011950 con 0x7fc2f8100770 2024-05-09T08:48:22.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.495+0000 7fc2fefc1640 1 -- 172.21.15.98:0/2863358417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f819e8d0 con 0x7fc2f8100770 2024-05-09T08:48:22.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.495+0000 7fc2f5ffb640 1 -- 172.21.15.98:0/2863358417 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e801aac0 con 0x7fc2f8100770 2024-05-09T08:48:22.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.495+0000 7fc2fefc1640 1 -- 172.21.15.98:0/2863358417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f819ee20 con 0x7fc2f8100770 2024-05-09T08:48:22.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.496+0000 7fc2fefc1640 1 -- 172.21.15.98:0/2863358417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f8068650 con 0x7fc2f8100770 2024-05-09T08:48:22.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.497+0000 7fc2f5ffb640 1 -- 172.21.15.98:0/2863358417 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc2e8011b00 con 0x7fc2f8100770 2024-05-09T08:48:22.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.498+0000 7fc2f5ffb640 1 --2- 172.21.15.98:0/2863358417 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc2e40a2b90 0x7fc2e40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:22.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.498+0000 7fc2f7fff640 1 --2- 172.21.15.98:0/2863358417 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc2e40a2b90 0x7fc2e40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:22.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.498+0000 7fc2f5ffb640 1 -- 172.21.15.98:0/2863358417 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7fc2e80cf840 con 0x7fc2f8100770 2024-05-09T08:48:22.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.498+0000 7fc2f7fff640 1 --2- 172.21.15.98:0/2863358417 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc2e40a2b90 0x7fc2e40a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8197ed0 tx=0x7fc2e000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.503+0000 7fc2f5ffb640 1 -- 172.21.15.98:0/2863358417 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc2e8016030 con 0x7fc2f8100770 2024-05-09T08:48:22.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:22.718+0000 7fc2fefc1640 1 -- 172.21.15.98:0/2863358417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fc2f8107a90 con 0x7fc2f8100770 2024-05-09T08:48:22.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:22 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/750176395' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-09T08:48:22.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:22 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/750176395' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-09T08:48:22.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:22 smithi098 ceph-mon[30092]: osdmap e89: 12 total, 12 up, 12 in 2024-05-09T08:48:22.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:22 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/750176395' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-09T08:48:22.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:22 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/750176395' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-09T08:48:22.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:22 smithi142 ceph-mon[34625]: osdmap e89: 12 total, 12 up, 12 in 2024-05-09T08:48:22.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:22 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/750176395' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-09T08:48:22.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:22 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/750176395' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-09T08:48:22.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:22 smithi105 ceph-mon[34498]: osdmap e89: 12 total, 12 up, 12 in 2024-05-09T08:48:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.611+0000 7fc2f5ffb640 1 -- 172.21.15.98:0/2863358417 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7fc2f8107a90 con 0x7fc2f8100770 2024-05-09T08:48:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:added data pool 4 to fsmap 2024-05-09T08:48:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.614+0000 7fc2fefc1640 1 -- 172.21.15.98:0/2863358417 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc2e40a2b90 msgr2=0x7fc2e40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.614+0000 7fc2fefc1640 1 --2- 172.21.15.98:0/2863358417 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc2e40a2b90 0x7fc2e40a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8197ed0 tx=0x7fc2e000a2d0 comp rx=0 tx=0).stop 2024-05-09T08:48:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.614+0000 7fc2fefc1640 1 -- 172.21.15.98:0/2863358417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8100770 msgr2=0x7fc2f81969c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.614+0000 7fc2fefc1640 1 --2- 172.21.15.98:0/2863358417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8100770 0x7fc2f81969c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800ea80 tx=0x7fc2e800ee80 comp rx=0 tx=0).stop 2024-05-09T08:48:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.614+0000 7fc2fefc1640 1 -- 172.21.15.98:0/2863358417 shutdown_connections 2024-05-09T08:48:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.614+0000 7fc2fefc1640 1 --2- 172.21.15.98:0/2863358417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2f8102320 0x7fc2f819df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.614+0000 7fc2fefc1640 1 --2- 172.21.15.98:0/2863358417 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2f8101960 0x7fc2f8196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.614+0000 7fc2fefc1640 1 --2- 172.21.15.98:0/2863358417 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc2e40a2b90 0x7fc2e40a5050 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.614+0000 7fc2fefc1640 1 --2- 172.21.15.98:0/2863358417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8100770 0x7fc2f81969c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.614+0000 7fc2fefc1640 1 -- 172.21.15.98:0/2863358417 >> 172.21.15.98:0/2863358417 conn(0x7fc2f80fbce0 msgr2=0x7fc2f8104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.614+0000 7fc2fefc1640 1 -- 172.21.15.98:0/2863358417 shutdown_connections 2024-05-09T08:48:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.614+0000 7fc2fefc1640 1 -- 172.21.15.98:0/2863358417 wait complete. 2024-05-09T08:48:23.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:48:23.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.734+0000 7f930ff7e640 1 -- 172.21.15.98:0/3077223650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9308102a70 msgr2=0x7f9308102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:23.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.734+0000 7f930ff7e640 1 --2- 172.21.15.98:0/3077223650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9308102a70 0x7f9308102ef0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009900 tx=0x7f92fc02a410 comp rx=0 tx=0).stop 2024-05-09T08:48:23.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.734+0000 7f930ff7e640 1 -- 172.21.15.98:0/3077223650 shutdown_connections 2024-05-09T08:48:23.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.734+0000 7f930ff7e640 1 --2- 172.21.15.98:0/3077223650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9308103430 0x7f9308109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:23.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.734+0000 7f930ff7e640 1 --2- 172.21.15.98:0/3077223650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9308102a70 0x7f9308102ef0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:23.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.734+0000 7f930ff7e640 1 --2- 172.21.15.98:0/3077223650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9308101880 0x7f9308101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.734+0000 7f930ff7e640 1 -- 172.21.15.98:0/3077223650 >> 172.21.15.98:0/3077223650 conn(0x7f93080fce10 msgr2=0x7f93080ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.734+0000 7f930ff7e640 1 -- 172.21.15.98:0/3077223650 shutdown_connections 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.734+0000 7f930ff7e640 1 -- 172.21.15.98:0/3077223650 wait complete. 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930ff7e640 1 Processor -- start 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930ff7e640 1 -- start start 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930ff7e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9308101880 0x7f9308197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930ff7e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9308102a70 0x7f9308198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930ff7e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9308103430 0x7f930819f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930ff7e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f930810d560 con 0x7f9308103430 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930ff7e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f930810d3e0 con 0x7f9308101880 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930ff7e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f930810d6e0 con 0x7f9308102a70 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930e51b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9308103430 0x7f930819f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930e51b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9308103430 0x7f930819f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39370/0 (socket says 172.21.15.98:39370) 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930e51b640 1 -- 172.21.15.98:0/2604398633 learned_addr learned my addr 172.21.15.98:0/2604398633 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930d519640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9308102a70 0x7f9308198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930dd1a640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9308101880 0x7f9308197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930e51b640 1 -- 172.21.15.98:0/2604398633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9308102a70 msgr2=0x7f9308198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930e51b640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9308102a70 0x7f9308198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930e51b640 1 -- 172.21.15.98:0/2604398633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9308101880 msgr2=0x7f9308197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930e51b640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9308101880 0x7f9308197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930e51b640 1 -- 172.21.15.98:0/2604398633 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f930819f840 con 0x7f9308103430 2024-05-09T08:48:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930dd1a640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9308101880 0x7f9308197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.735+0000 7f930e51b640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9308103430 0x7f930819f1a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f92f800b970 tx=0x7f92f800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.736+0000 7f9306ffd640 1 -- 172.21.15.98:0/2604398633 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f8019070 con 0x7f9308103430 2024-05-09T08:48:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.736+0000 7f930ff7e640 1 -- 172.21.15.98:0/2604398633 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f930819fb30 con 0x7f9308103430 2024-05-09T08:48:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.736+0000 7f9306ffd640 1 -- 172.21.15.98:0/2604398633 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92f800cb40 con 0x7f9308103430 2024-05-09T08:48:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.736+0000 7f9306ffd640 1 -- 172.21.15.98:0/2604398633 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f8007610 con 0x7f9308103430 2024-05-09T08:48:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.736+0000 7f930ff7e640 1 -- 172.21.15.98:0/2604398633 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93081a00d0 con 0x7f9308103430 2024-05-09T08:48:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.737+0000 7f930ff7e640 1 -- 172.21.15.98:0/2604398633 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9308108ba0 con 0x7f9308103430 2024-05-09T08:48:23.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.738+0000 7f9306ffd640 1 -- 172.21.15.98:0/2604398633 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f92f801c070 con 0x7f9308103430 2024-05-09T08:48:23.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.738+0000 7f9306ffd640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f92f00a2b90 0x7f92f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:23.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.738+0000 7f930dd1a640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f92f00a2b90 0x7f92f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:23.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.739+0000 7f9306ffd640 1 -- 172.21.15.98:0/2604398633 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f92f80cede0 con 0x7f9308103430 2024-05-09T08:48:23.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.739+0000 7f930dd1a640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f92f00a2b90 0x7f92f00a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9300005fd0 tx=0x7f9300005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:23.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.742+0000 7f9306ffd640 1 -- 172.21.15.98:0/2604398633 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f92f809bb00 con 0x7f9308103430 2024-05-09T08:48:23.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:23 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2863358417' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-09T08:48:23.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:23 smithi142 ceph-mon[34625]: pgmap v292: 129 pgs: 8 creating+peering, 99 active+clean, 22 unknown; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-05-09T08:48:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.960+0000 7f930ff7e640 1 -- 172.21.15.98:0/2604398633 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f930802d1c0 con 0x7f9308103430 2024-05-09T08:48:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.961+0000 7f9306ffd640 1 -- 172.21.15.98:0/2604398633 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11807 (secure 0 0 0) 0x7f92f809b400 con 0x7f9308103430 2024-05-09T08:48:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:23.960 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":14667,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:48:22.719887+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24461,"qdb_cluster":[24461]},"id":1}]} 2024-05-09T08:48:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2024-05-09T08:48:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.963+0000 7f930ff7e640 1 -- 172.21.15.98:0/2604398633 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f92f00a2b90 msgr2=0x7f92f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.963+0000 7f930ff7e640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f92f00a2b90 0x7f92f00a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9300005fd0 tx=0x7f9300005850 comp rx=0 tx=0).stop 2024-05-09T08:48:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.963+0000 7f930ff7e640 1 -- 172.21.15.98:0/2604398633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9308103430 msgr2=0x7f930819f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.963+0000 7f930ff7e640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9308103430 0x7f930819f1a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f92f800b970 tx=0x7f92f800be30 comp rx=0 tx=0).stop 2024-05-09T08:48:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.963+0000 7f930ff7e640 1 -- 172.21.15.98:0/2604398633 shutdown_connections 2024-05-09T08:48:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.963+0000 7f930ff7e640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f92f00a2b90 0x7f92f00a5050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.963+0000 7f930ff7e640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9308103430 0x7f930819f1a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.963+0000 7f930ff7e640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9308102a70 0x7f9308198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.963+0000 7f930ff7e640 1 --2- 172.21.15.98:0/2604398633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9308101880 0x7f9308197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.963+0000 7f930ff7e640 1 -- 172.21.15.98:0/2604398633 >> 172.21.15.98:0/2604398633 conn(0x7f93080fce10 msgr2=0x7f9308105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:23.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.963+0000 7f930ff7e640 1 -- 172.21.15.98:0/2604398633 shutdown_connections 2024-05-09T08:48:23.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:23.963+0000 7f930ff7e640 1 -- 172.21.15.98:0/2604398633 wait complete. 2024-05-09T08:48:23.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:23 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2863358417' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-09T08:48:23.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:23 smithi098 ceph-mon[30092]: pgmap v292: 129 pgs: 8 creating+peering, 99 active+clean, 22 unknown; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-05-09T08:48:23.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:48:23.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:23 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2863358417' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-09T08:48:23.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:23 smithi105 ceph-mon[34498]: pgmap v292: 129 pgs: 8 creating+peering, 99 active+clean, 22 unknown; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-05-09T08:48:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.075+0000 7f84da46b640 1 -- 172.21.15.98:0/866264179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84d4101970 msgr2=0x7f84d4101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.075+0000 7f84da46b640 1 --2- 172.21.15.98:0/866264179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84d4101970 0x7f84d4101df0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f84bc009900 tx=0x7f84bc02a410 comp rx=0 tx=0).stop 2024-05-09T08:48:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.075+0000 7f84da46b640 1 -- 172.21.15.98:0/866264179 shutdown_connections 2024-05-09T08:48:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.075+0000 7f84da46b640 1 --2- 172.21.15.98:0/866264179 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84d4102330 0x7f84d4108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.075+0000 7f84da46b640 1 --2- 172.21.15.98:0/866264179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84d4101970 0x7f84d4101df0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.075+0000 7f84da46b640 1 --2- 172.21.15.98:0/866264179 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84d4100780 0x7f84d4100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.075+0000 7f84da46b640 1 -- 172.21.15.98:0/866264179 >> 172.21.15.98:0/866264179 conn(0x7f84d40fbd10 msgr2=0x7f84d40fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.075+0000 7f84da46b640 1 -- 172.21.15.98:0/866264179 shutdown_connections 2024-05-09T08:48:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.075+0000 7f84da46b640 1 -- 172.21.15.98:0/866264179 wait complete. 2024-05-09T08:48:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.075+0000 7f84da46b640 1 Processor -- start 2024-05-09T08:48:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84da46b640 1 -- start start 2024-05-09T08:48:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84da46b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84d4100780 0x7f84d4197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84da46b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84d4101970 0x7f84d4198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84da46b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84d4102330 0x7f84d419f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84da46b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f84d410d600 con 0x7f84d4101970 2024-05-09T08:48:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84da46b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f84d410d480 con 0x7f84d4102330 2024-05-09T08:48:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84da46b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f84d410d780 con 0x7f84d4100780 2024-05-09T08:48:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84d4101970 0x7f84d4198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84d4101970 0x7f84d4198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39380/0 (socket says 172.21.15.98:39380) 2024-05-09T08:48:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84cbfff640 1 -- 172.21.15.98:0/1696670317 learned_addr learned my addr 172.21.15.98:0/1696670317 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84d3fff640 1 --2- 172.21.15.98:0/1696670317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84d4100780 0x7f84d4197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84cbfff640 1 -- 172.21.15.98:0/1696670317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84d4100780 msgr2=0x7f84d4197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84cbfff640 1 --2- 172.21.15.98:0/1696670317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84d4100780 0x7f84d4197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84cbfff640 1 -- 172.21.15.98:0/1696670317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84d4102330 msgr2=0x7f84d419f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84d8a08640 1 --2- 172.21.15.98:0/1696670317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84d4102330 0x7f84d419f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84cbfff640 1 --2- 172.21.15.98:0/1696670317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84d4102330 0x7f84d419f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84cbfff640 1 -- 172.21.15.98:0/1696670317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d419f950 con 0x7f84d4101970 2024-05-09T08:48:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84cbfff640 1 --2- 172.21.15.98:0/1696670317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84d4101970 0x7f84d4198280 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f84bc009230 tx=0x7f84bc004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84d1ffb640 1 -- 172.21.15.98:0/1696670317 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc03f030 con 0x7f84d4101970 2024-05-09T08:48:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84d1ffb640 1 -- 172.21.15.98:0/1696670317 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84bc003ce0 con 0x7f84d4101970 2024-05-09T08:48:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84da46b640 1 -- 172.21.15.98:0/1696670317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d419fbe0 con 0x7f84d4101970 2024-05-09T08:48:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84d1ffb640 1 -- 172.21.15.98:0/1696670317 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc035070 con 0x7f84d4101970 2024-05-09T08:48:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.076+0000 7f84da46b640 1 -- 172.21.15.98:0/1696670317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d41a0120 con 0x7f84d4101970 2024-05-09T08:48:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.077+0000 7f84da46b640 1 -- 172.21.15.98:0/1696670317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d4107b70 con 0x7f84d4101970 2024-05-09T08:48:24.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.078+0000 7f84d1ffb640 1 -- 172.21.15.98:0/1696670317 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f84bc042030 con 0x7f84d4101970 2024-05-09T08:48:24.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.079+0000 7f84d1ffb640 1 --2- 172.21.15.98:0/1696670317 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f84c00a2c10 0x7f84c00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:24.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.079+0000 7f84d3fff640 1 --2- 172.21.15.98:0/1696670317 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f84c00a2c10 0x7f84c00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:24.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.079+0000 7f84d1ffb640 1 -- 172.21.15.98:0/1696670317 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f84bc0efab0 con 0x7f84d4101970 2024-05-09T08:48:24.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.079+0000 7f84d3fff640 1 --2- 172.21.15.98:0/1696670317 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f84c00a2c10 0x7f84c00a50d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f84d4199210 tx=0x7f84c4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:24.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.083+0000 7f84d1ffb640 1 -- 172.21.15.98:0/1696670317 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f84bc0bc7d0 con 0x7f84d4101970 2024-05-09T08:48:24.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.263+0000 7f84da46b640 1 -- 172.21.15.98:0/1696670317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84d40624d0 con 0x7f84d4101970 2024-05-09T08:48:24.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.264+0000 7f84d1ffb640 1 -- 172.21.15.98:0/1696670317 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24150 (secure 0 0 0) 0x7f84bc0bc0d0 con 0x7f84d4101970 2024-05-09T08:48:24.264 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:24.264 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":90,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:48:22.719729+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:48:24.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.266+0000 7f84da46b640 1 -- 172.21.15.98:0/1696670317 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f84c00a2c10 msgr2=0x7f84c00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.266+0000 7f84da46b640 1 --2- 172.21.15.98:0/1696670317 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f84c00a2c10 0x7f84c00a50d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f84d4199210 tx=0x7f84c4008040 comp rx=0 tx=0).stop 2024-05-09T08:48:24.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.266+0000 7f84da46b640 1 -- 172.21.15.98:0/1696670317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84d4101970 msgr2=0x7f84d4198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.266+0000 7f84da46b640 1 --2- 172.21.15.98:0/1696670317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84d4101970 0x7f84d4198280 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f84bc009230 tx=0x7f84bc004430 comp rx=0 tx=0).stop 2024-05-09T08:48:24.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.266+0000 7f84da46b640 1 -- 172.21.15.98:0/1696670317 shutdown_connections 2024-05-09T08:48:24.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.266+0000 7f84da46b640 1 --2- 172.21.15.98:0/1696670317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84d4102330 0x7f84d419f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.266+0000 7f84da46b640 1 --2- 172.21.15.98:0/1696670317 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f84c00a2c10 0x7f84c00a50d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.266+0000 7f84da46b640 1 --2- 172.21.15.98:0/1696670317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84d4101970 0x7f84d4198280 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.266+0000 7f84da46b640 1 --2- 172.21.15.98:0/1696670317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84d4100780 0x7f84d4197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.266+0000 7f84da46b640 1 -- 172.21.15.98:0/1696670317 >> 172.21.15.98:0/1696670317 conn(0x7f84d40fbd10 msgr2=0x7f84d4104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:24.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.266+0000 7f84da46b640 1 -- 172.21.15.98:0/1696670317 shutdown_connections 2024-05-09T08:48:24.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.266+0000 7f84da46b640 1 -- 172.21.15.98:0/1696670317 wait complete. 2024-05-09T08:48:24.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:48:24.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 -- 172.21.15.98:0/1993009583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1114103440 msgr2=0x7f1114109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 --2- 172.21.15.98:0/1993009583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1114103440 0x7f1114109cd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f110c009ca0 tx=0x7f110c02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 -- 172.21.15.98:0/1993009583 shutdown_connections 2024-05-09T08:48:24.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 --2- 172.21.15.98:0/1993009583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1114103440 0x7f1114109cd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 --2- 172.21.15.98:0/1993009583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1114102a80 0x7f1114102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 --2- 172.21.15.98:0/1993009583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1114101890 0x7f1114101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 -- 172.21.15.98:0/1993009583 >> 172.21.15.98:0/1993009583 conn(0x7f11140fcde0 msgr2=0x7f11140ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 -- 172.21.15.98:0/1993009583 shutdown_connections 2024-05-09T08:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 -- 172.21.15.98:0/1993009583 wait complete. 2024-05-09T08:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 Processor -- start 2024-05-09T08:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 -- start start 2024-05-09T08:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1114101890 0x7f1114197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1114102a80 0x7f1114198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1114103440 0x7f111419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f111410d5d0 con 0x7f1114101890 2024-05-09T08:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f111410d450 con 0x7f1114103440 2024-05-09T08:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f111a59e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f111410d750 con 0x7f1114102a80 2024-05-09T08:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f1113fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1114101890 0x7f1114197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f1113fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1114101890 0x7f1114197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39398/0 (socket says 172.21.15.98:39398) 2024-05-09T08:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.379+0000 7f1113fff640 1 -- 172.21.15.98:0/1265497465 learned_addr learned my addr 172.21.15.98:0/1265497465 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.380+0000 7f1118b3b640 1 --2- 172.21.15.98:0/1265497465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1114103440 0x7f111419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.380+0000 7f1113fff640 1 -- 172.21.15.98:0/1265497465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1114102a80 msgr2=0x7f1114198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.380+0000 7f1113fff640 1 --2- 172.21.15.98:0/1265497465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1114102a80 0x7f1114198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.380+0000 7f1113fff640 1 -- 172.21.15.98:0/1265497465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1114103440 msgr2=0x7f111419f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.380+0000 7f1113fff640 1 --2- 172.21.15.98:0/1265497465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1114103440 0x7f111419f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.380+0000 7f1113fff640 1 -- 172.21.15.98:0/1265497465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111419f740 con 0x7f1114101890 2024-05-09T08:48:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.380+0000 7f1113fff640 1 --2- 172.21.15.98:0/1265497465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1114101890 0x7f1114197b20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f110400b450 tx=0x7f110400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.380+0000 7f1111ffb640 1 -- 172.21.15.98:0/1265497465 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1104015020 con 0x7f1114101890 2024-05-09T08:48:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.380+0000 7f1111ffb640 1 -- 172.21.15.98:0/1265497465 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1104011d70 con 0x7f1114101890 2024-05-09T08:48:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.380+0000 7f111a59e640 1 -- 172.21.15.98:0/1265497465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f111419fa30 con 0x7f1114101890 2024-05-09T08:48:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.380+0000 7f1111ffb640 1 -- 172.21.15.98:0/1265497465 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110400f340 con 0x7f1114101890 2024-05-09T08:48:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.380+0000 7f111a59e640 1 -- 172.21.15.98:0/1265497465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f111419ffb0 con 0x7f1114101890 2024-05-09T08:48:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.381+0000 7f111a59e640 1 -- 172.21.15.98:0/1265497465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1114108ae0 con 0x7f1114101890 2024-05-09T08:48:24.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.382+0000 7f1111ffb640 1 -- 172.21.15.98:0/1265497465 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1104018020 con 0x7f1114101890 2024-05-09T08:48:24.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.383+0000 7f1111ffb640 1 --2- 172.21.15.98:0/1265497465 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f11000a2b90 0x7f11000a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:24.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.383+0000 7f110bfff640 1 --2- 172.21.15.98:0/1265497465 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f11000a2b90 0x7f11000a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:24.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.383+0000 7f1111ffb640 1 -- 172.21.15.98:0/1265497465 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f11040cf2f0 con 0x7f1114101890 2024-05-09T08:48:24.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.383+0000 7f110bfff640 1 --2- 172.21.15.98:0/1265497465 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f11000a2b90 0x7f11000a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1114199030 tx=0x7f10fc005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:24.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.386+0000 7f1111ffb640 1 -- 172.21.15.98:0/1265497465 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f110401f020 con 0x7f1114101890 2024-05-09T08:48:24.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.562+0000 7f111a59e640 1 -- 172.21.15.98:0/1265497465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f111402d3e0 con 0x7f1114101890 2024-05-09T08:48:24.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.564+0000 7f1111ffb640 1 -- 172.21.15.98:0/1265497465 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24150 (secure 0 0 0) 0x7f110400d040 con 0x7f1114101890 2024-05-09T08:48:24.563 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:24.563 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":90,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:48:22.719729+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:48:24.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.566+0000 7f111a59e640 1 -- 172.21.15.98:0/1265497465 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f11000a2b90 msgr2=0x7f11000a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.566+0000 7f111a59e640 1 --2- 172.21.15.98:0/1265497465 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f11000a2b90 0x7f11000a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1114199030 tx=0x7f10fc005c70 comp rx=0 tx=0).stop 2024-05-09T08:48:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.566+0000 7f111a59e640 1 -- 172.21.15.98:0/1265497465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1114101890 msgr2=0x7f1114197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.566+0000 7f111a59e640 1 --2- 172.21.15.98:0/1265497465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1114101890 0x7f1114197b20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f110400b450 tx=0x7f110400b910 comp rx=0 tx=0).stop 2024-05-09T08:48:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.566+0000 7f111a59e640 1 -- 172.21.15.98:0/1265497465 shutdown_connections 2024-05-09T08:48:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.566+0000 7f111a59e640 1 --2- 172.21.15.98:0/1265497465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1114103440 0x7f111419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.566+0000 7f111a59e640 1 --2- 172.21.15.98:0/1265497465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1114102a80 0x7f1114198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.566+0000 7f111a59e640 1 --2- 172.21.15.98:0/1265497465 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f11000a2b90 0x7f11000a5050 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.566+0000 7f111a59e640 1 --2- 172.21.15.98:0/1265497465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1114101890 0x7f1114197b20 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.566+0000 7f111a59e640 1 -- 172.21.15.98:0/1265497465 >> 172.21.15.98:0/1265497465 conn(0x7f11140fcde0 msgr2=0x7f1114105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.566+0000 7f111a59e640 1 -- 172.21.15.98:0/1265497465 shutdown_connections 2024-05-09T08:48:24.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.566+0000 7f111a59e640 1 -- 172.21.15.98:0/1265497465 wait complete. 2024-05-09T08:48:24.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:48:24.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.696+0000 7ff096cba640 1 -- 172.21.15.98:0/2327576398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff090101980 msgr2=0x7ff090101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.696+0000 7ff096cba640 1 --2- 172.21.15.98:0/2327576398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff090101980 0x7ff090101e00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009900 tx=0x7ff07c02a410 comp rx=0 tx=0).stop 2024-05-09T08:48:24.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.696+0000 7ff096cba640 1 -- 172.21.15.98:0/2327576398 shutdown_connections 2024-05-09T08:48:24.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.696+0000 7ff096cba640 1 --2- 172.21.15.98:0/2327576398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff090102340 0x7ff090108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.696+0000 7ff096cba640 1 --2- 172.21.15.98:0/2327576398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff090101980 0x7ff090101e00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.696+0000 7ff096cba640 1 --2- 172.21.15.98:0/2327576398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff090100790 0x7ff090100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.696+0000 7ff096cba640 1 -- 172.21.15.98:0/2327576398 >> 172.21.15.98:0/2327576398 conn(0x7ff0900fbd20 msgr2=0x7ff0900fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.696+0000 7ff096cba640 1 -- 172.21.15.98:0/2327576398 shutdown_connections 2024-05-09T08:48:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.696+0000 7ff096cba640 1 -- 172.21.15.98:0/2327576398 wait complete. 2024-05-09T08:48:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.696+0000 7ff096cba640 1 Processor -- start 2024-05-09T08:48:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.696+0000 7ff096cba640 1 -- start start 2024-05-09T08:48:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff096cba640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff090100790 0x7ff09006aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff096cba640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff090101980 0x7ff09019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff096cba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff090102340 0x7ff09019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff096cba640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff09010c550 con 0x7ff090102340 2024-05-09T08:48:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff096cba640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff09010c3d0 con 0x7ff090100790 2024-05-09T08:48:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff096cba640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff09010c6d0 con 0x7ff090101980 2024-05-09T08:48:24.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff095257640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff090102340 0x7ff09019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:24.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff095257640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff090102340 0x7ff09019f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39426/0 (socket says 172.21.15.98:39426) 2024-05-09T08:48:24.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff095257640 1 -- 172.21.15.98:0/1263346748 learned_addr learned my addr 172.21.15.98:0/1263346748 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:24.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff095257640 1 -- 172.21.15.98:0/1263346748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff090101980 msgr2=0x7ff09019ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff095257640 1 --2- 172.21.15.98:0/1263346748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff090101980 0x7ff09019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff095257640 1 -- 172.21.15.98:0/1263346748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff090100790 msgr2=0x7ff09006aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff094a56640 1 --2- 172.21.15.98:0/1263346748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff090100790 0x7ff09006aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:24.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff095257640 1 --2- 172.21.15.98:0/1263346748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff090100790 0x7ff09006aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff095257640 1 -- 172.21.15.98:0/1263346748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff09019f950 con 0x7ff090102340 2024-05-09T08:48:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff094a56640 1 --2- 172.21.15.98:0/1263346748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff090100790 0x7ff09006aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff095257640 1 --2- 172.21.15.98:0/1263346748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff090102340 0x7ff09019f090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff08800b970 tx=0x7ff08800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff08dffb640 1 -- 172.21.15.98:0/1263346748 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088019070 con 0x7ff090102340 2024-05-09T08:48:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff08dffb640 1 -- 172.21.15.98:0/1263346748 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff08800cb40 con 0x7ff090102340 2024-05-09T08:48:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff096cba640 1 -- 172.21.15.98:0/1263346748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff09019fc40 con 0x7ff090102340 2024-05-09T08:48:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.697+0000 7ff08dffb640 1 -- 172.21.15.98:0/1263346748 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088007650 con 0x7ff090102340 2024-05-09T08:48:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.698+0000 7ff096cba640 1 -- 172.21.15.98:0/1263346748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0901a01c0 con 0x7ff090102340 2024-05-09T08:48:24.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.698+0000 7ff096cba640 1 -- 172.21.15.98:0/1263346748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff090107b00 con 0x7ff090102340 2024-05-09T08:48:24.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.699+0000 7ff08dffb640 1 -- 172.21.15.98:0/1263346748 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff08801c070 con 0x7ff090102340 2024-05-09T08:48:24.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.700+0000 7ff08dffb640 1 --2- 172.21.15.98:0/1263346748 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff0800a2b90 0x7ff0800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:24.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.700+0000 7ff094a56640 1 --2- 172.21.15.98:0/1263346748 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff0800a2b90 0x7ff0800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:24.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.700+0000 7ff08dffb640 1 -- 172.21.15.98:0/1263346748 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff0880ce910 con 0x7ff090102340 2024-05-09T08:48:24.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.700+0000 7ff094a56640 1 --2- 172.21.15.98:0/1263346748 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff0800a2b90 0x7ff0800a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff084006fd0 tx=0x7ff084008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:24.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.704+0000 7ff08dffb640 1 -- 172.21.15.98:0/1263346748 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff08809b630 con 0x7ff090102340 2024-05-09T08:48:24.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:24 smithi098 ceph-mon[30092]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:48:24.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:24 smithi098 ceph-mon[30092]: osdmap e90: 12 total, 12 up, 12 in 2024-05-09T08:48:24.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:24 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2863358417' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-09T08:48:24.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:24 smithi098 ceph-mon[30092]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:24.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:24 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2604398633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:24.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:24 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1696670317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:24.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:24 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1265497465' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:24.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:24 smithi142 ceph-mon[34625]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:48:24.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:24 smithi142 ceph-mon[34625]: osdmap e90: 12 total, 12 up, 12 in 2024-05-09T08:48:24.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:24 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2863358417' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-09T08:48:24.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:24 smithi142 ceph-mon[34625]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:24.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:24 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2604398633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:24.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:24 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1696670317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:24.909 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:24 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1265497465' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:24.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.919+0000 7ff096cba640 1 -- 172.21.15.98:0/1263346748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff090062430 con 0x7ff090102340 2024-05-09T08:48:24.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.919+0000 7ff08dffb640 1 -- 172.21.15.98:0/1263346748 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11807 (secure 0 0 0) 0x7ff08809af30 con 0x7ff090102340 2024-05-09T08:48:24.919 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:24.920 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":14667,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:48:22.719887+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24461,"qdb_cluster":[24461]},"id":1}]} 2024-05-09T08:48:24.921 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2024-05-09T08:48:24.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.921+0000 7ff096cba640 1 -- 172.21.15.98:0/1263346748 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff0800a2b90 msgr2=0x7ff0800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.921+0000 7ff096cba640 1 --2- 172.21.15.98:0/1263346748 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff0800a2b90 0x7ff0800a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff084006fd0 tx=0x7ff084008040 comp rx=0 tx=0).stop 2024-05-09T08:48:24.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.921+0000 7ff096cba640 1 -- 172.21.15.98:0/1263346748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff090102340 msgr2=0x7ff09019f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:24.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.921+0000 7ff096cba640 1 --2- 172.21.15.98:0/1263346748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff090102340 0x7ff09019f090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff08800b970 tx=0x7ff08800be30 comp rx=0 tx=0).stop 2024-05-09T08:48:24.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.921+0000 7ff096cba640 1 -- 172.21.15.98:0/1263346748 shutdown_connections 2024-05-09T08:48:24.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.921+0000 7ff096cba640 1 --2- 172.21.15.98:0/1263346748 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff0800a2b90 0x7ff0800a5050 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.921+0000 7ff096cba640 1 --2- 172.21.15.98:0/1263346748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff090102340 0x7ff09019f090 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.921+0000 7ff096cba640 1 --2- 172.21.15.98:0/1263346748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff090101980 0x7ff09019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.921+0000 7ff096cba640 1 --2- 172.21.15.98:0/1263346748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff090100790 0x7ff09006aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:24.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.921+0000 7ff096cba640 1 -- 172.21.15.98:0/1263346748 >> 172.21.15.98:0/1263346748 conn(0x7ff0900fbd20 msgr2=0x7ff090104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:24.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.921+0000 7ff096cba640 1 -- 172.21.15.98:0/1263346748 shutdown_connections 2024-05-09T08:48:24.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:24.921+0000 7ff096cba640 1 -- 172.21.15.98:0/1263346748 wait complete. 2024-05-09T08:48:24.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:48:24.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:24 smithi105 ceph-mon[34498]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:48:24.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:24 smithi105 ceph-mon[34498]: osdmap e90: 12 total, 12 up, 12 in 2024-05-09T08:48:24.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:24 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2863358417' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-09T08:48:24.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:24 smithi105 ceph-mon[34498]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:24.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:24 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2604398633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:24.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:24 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1696670317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:24.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:24 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1265497465' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:25.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.034+0000 7f07692fc640 1 -- 172.21.15.98:0/3032862985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07641007b0 msgr2=0x7f0764100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:25.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.034+0000 7f07692fc640 1 --2- 172.21.15.98:0/3032862985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07641007b0 0x7f0764100bd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0754009900 tx=0x7f075402a410 comp rx=0 tx=0).stop 2024-05-09T08:48:25.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.034+0000 7f07692fc640 1 -- 172.21.15.98:0/3032862985 shutdown_connections 2024-05-09T08:48:25.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.034+0000 7f07692fc640 1 --2- 172.21.15.98:0/3032862985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0764102360 0x7f0764108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.034+0000 7f07692fc640 1 --2- 172.21.15.98:0/3032862985 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07641019a0 0x7f0764101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.034+0000 7f07692fc640 1 --2- 172.21.15.98:0/3032862985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07641007b0 0x7f0764100bd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.034+0000 7f07692fc640 1 -- 172.21.15.98:0/3032862985 >> 172.21.15.98:0/3032862985 conn(0x7f07640fbd20 msgr2=0x7f07640fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:25.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.034+0000 7f07692fc640 1 -- 172.21.15.98:0/3032862985 shutdown_connections 2024-05-09T08:48:25.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.034+0000 7f07692fc640 1 -- 172.21.15.98:0/3032862985 wait complete. 2024-05-09T08:48:25.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.034+0000 7f07692fc640 1 Processor -- start 2024-05-09T08:48:25.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07692fc640 1 -- start start 2024-05-09T08:48:25.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07692fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07641007b0 0x7f076406a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07692fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07641019a0 0x7f076419c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07692fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0764102360 0x7f076419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07692fc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f076410c480 con 0x7f0764102360 2024-05-09T08:48:25.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07692fc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f076410c300 con 0x7f07641007b0 2024-05-09T08:48:25.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07692fc640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f076410c600 con 0x7f07641019a0 2024-05-09T08:48:25.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07637fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0764102360 0x7f076419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07637fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0764102360 0x7f076419efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39432/0 (socket says 172.21.15.98:39432) 2024-05-09T08:48:25.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07637fe640 1 -- 172.21.15.98:0/1331945364 learned_addr learned my addr 172.21.15.98:0/1331945364 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:25.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f075a7fc640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07641019a0 0x7f076419c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f0762ffd640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07641007b0 0x7f076406a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07637fe640 1 -- 172.21.15.98:0/1331945364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07641019a0 msgr2=0x7f076419c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:25.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07637fe640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07641019a0 0x7f076419c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07637fe640 1 -- 172.21.15.98:0/1331945364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07641007b0 msgr2=0x7f076406a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:25.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07637fe640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07641007b0 0x7f076406a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07637fe640 1 -- 172.21.15.98:0/1331945364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076419f8a0 con 0x7f0764102360 2024-05-09T08:48:25.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f075a7fc640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07641019a0 0x7f076419c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:25.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f07637fe640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0764102360 0x7f076419efe0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f075c00b970 tx=0x7f075c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:25.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.035+0000 7f0760ff9640 1 -- 172.21.15.98:0/1331945364 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075c019070 con 0x7f0764102360 2024-05-09T08:48:25.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.036+0000 7f0760ff9640 1 -- 172.21.15.98:0/1331945364 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f075c00cb40 con 0x7f0764102360 2024-05-09T08:48:25.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.036+0000 7f07692fc640 1 -- 172.21.15.98:0/1331945364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f076419fb90 con 0x7f0764102360 2024-05-09T08:48:25.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.036+0000 7f0760ff9640 1 -- 172.21.15.98:0/1331945364 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075c007650 con 0x7f0764102360 2024-05-09T08:48:25.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.036+0000 7f07692fc640 1 -- 172.21.15.98:0/1331945364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07641a0160 con 0x7f0764102360 2024-05-09T08:48:25.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.036+0000 7f07692fc640 1 -- 172.21.15.98:0/1331945364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0764107ad0 con 0x7f0764102360 2024-05-09T08:48:25.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.038+0000 7f0760ff9640 1 -- 172.21.15.98:0/1331945364 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f075c01c070 con 0x7f0764102360 2024-05-09T08:48:25.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.038+0000 7f0760ff9640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f07500a2b90 0x7f07500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.038+0000 7f0762ffd640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f07500a2b90 0x7f07500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.038+0000 7f0760ff9640 1 -- 172.21.15.98:0/1331945364 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f075c0ce8c0 con 0x7f0764102360 2024-05-09T08:48:25.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.039+0000 7f0762ffd640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f07500a2b90 0x7f07500a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f07540096c0 tx=0x7f0754000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:25.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.043+0000 7f0760ff9640 1 -- 172.21.15.98:0/1331945364 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f075c09b5e0 con 0x7f0764102360 2024-05-09T08:48:25.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.219+0000 7f07692fc640 1 -- 172.21.15.98:0/1331945364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0764062430 con 0x7f0764102360 2024-05-09T08:48:25.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.220+0000 7f0760ff9640 1 -- 172.21.15.98:0/1331945364 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24150 (secure 0 0 0) 0x7f075c09aee0 con 0x7f0764102360 2024-05-09T08:48:25.220 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:25.220 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":90,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:48:22.719729+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:48:25.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.222+0000 7f07692fc640 1 -- 172.21.15.98:0/1331945364 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f07500a2b90 msgr2=0x7f07500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:25.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.222+0000 7f07692fc640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f07500a2b90 0x7f07500a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f07540096c0 tx=0x7f0754000bc0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.222+0000 7f07692fc640 1 -- 172.21.15.98:0/1331945364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0764102360 msgr2=0x7f076419efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:25.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.223+0000 7f07692fc640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0764102360 0x7f076419efe0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f075c00b970 tx=0x7f075c00be30 comp rx=0 tx=0).stop 2024-05-09T08:48:25.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.223+0000 7f07692fc640 1 -- 172.21.15.98:0/1331945364 shutdown_connections 2024-05-09T08:48:25.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.223+0000 7f07692fc640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f07500a2b90 0x7f07500a5050 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.223+0000 7f07692fc640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0764102360 0x7f076419efe0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.223+0000 7f07692fc640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07641019a0 0x7f076419c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.223+0000 7f07692fc640 1 --2- 172.21.15.98:0/1331945364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07641007b0 0x7f076406a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.223+0000 7f07692fc640 1 -- 172.21.15.98:0/1331945364 >> 172.21.15.98:0/1331945364 conn(0x7f07640fbd20 msgr2=0x7f0764104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:25.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.223+0000 7f07692fc640 1 -- 172.21.15.98:0/1331945364 shutdown_connections 2024-05-09T08:48:25.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.223+0000 7f07692fc640 1 -- 172.21.15.98:0/1331945364 wait complete. 2024-05-09T08:48:25.234 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T08:48:17.933763+0000', 'modified': '2024-05-09T08:48:22.719887+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24461}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24461': {'gid': 24461, 'name': 'h', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.105:6837/992088454', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6836', 'nonce': 992088454}, {'type': 'v1', 'addr': '172.21.15.105:6837', 'nonce': 992088454}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24461, 'qdb_cluster': [24461]} 2024-05-09T08:48:25.234 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T08:48:25.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-05-09T08:48:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.335+0000 7f8b137dc640 1 -- 172.21.15.98:0/3003278896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b0c107570 msgr2=0x7f8b0c1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.335+0000 7f8b137dc640 1 --2- 172.21.15.98:0/3003278896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b0c107570 0x7f8b0c1099c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009ca0 tx=0x7f8b0402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.335+0000 7f8b137dc640 1 -- 172.21.15.98:0/3003278896 shutdown_connections 2024-05-09T08:48:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.335+0000 7f8b137dc640 1 --2- 172.21.15.98:0/3003278896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b0c107570 0x7f8b0c1099c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.335+0000 7f8b137dc640 1 --2- 172.21.15.98:0/3003278896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b0c104c40 0x7f8b0c107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.335+0000 7f8b137dc640 1 --2- 172.21.15.98:0/3003278896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b0c102310 0x7f8b0c104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.335+0000 7f8b137dc640 1 -- 172.21.15.98:0/3003278896 >> 172.21.15.98:0/3003278896 conn(0x7f8b0c0fbd20 msgr2=0x7f8b0c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.335+0000 7f8b137dc640 1 -- 172.21.15.98:0/3003278896 shutdown_connections 2024-05-09T08:48:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.335+0000 7f8b137dc640 1 -- 172.21.15.98:0/3003278896 wait complete. 2024-05-09T08:48:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b137dc640 1 Processor -- start 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b137dc640 1 -- start start 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b137dc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b0c102310 0x7f8b0c197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b137dc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b0c104c40 0x7f8b0c1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b137dc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b0c107570 0x7f8b0c19f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b137dc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0c10d580 con 0x7f8b0c104c40 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b137dc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0c10d400 con 0x7f8b0c107570 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b137dc640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0c10d700 con 0x7f8b0c102310 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b10d77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b0c104c40 0x7f8b0c1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b10d77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b0c104c40 0x7f8b0c1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39456/0 (socket says 172.21.15.98:39456) 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b10d77640 1 -- 172.21.15.98:0/1260159987 learned_addr learned my addr 172.21.15.98:0/1260159987 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b11578640 1 --2- 172.21.15.98:0/1260159987 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b0c102310 0x7f8b0c197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b11d79640 1 --2- 172.21.15.98:0/1260159987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b0c107570 0x7f8b0c19f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b10d77640 1 -- 172.21.15.98:0/1260159987 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b0c102310 msgr2=0x7f8b0c197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b10d77640 1 --2- 172.21.15.98:0/1260159987 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b0c102310 0x7f8b0c197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b10d77640 1 -- 172.21.15.98:0/1260159987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b0c107570 msgr2=0x7f8b0c19f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b10d77640 1 --2- 172.21.15.98:0/1260159987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b0c107570 0x7f8b0c19f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.336+0000 7f8b10d77640 1 -- 172.21.15.98:0/1260159987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b0c19f7c0 con 0x7f8b0c104c40 2024-05-09T08:48:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.337+0000 7f8b10d77640 1 --2- 172.21.15.98:0/1260159987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b0c104c40 0x7f8b0c1980e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800b450 tx=0x7f8b0800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.337+0000 7f8b027fc640 1 -- 172.21.15.98:0/1260159987 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08015020 con 0x7f8b0c104c40 2024-05-09T08:48:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.337+0000 7f8b027fc640 1 -- 172.21.15.98:0/1260159987 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b08011d70 con 0x7f8b0c104c40 2024-05-09T08:48:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.337+0000 7f8b137dc640 1 -- 172.21.15.98:0/1260159987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b0c19fab0 con 0x7f8b0c104c40 2024-05-09T08:48:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.337+0000 7f8b027fc640 1 -- 172.21.15.98:0/1260159987 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0800f340 con 0x7f8b0c104c40 2024-05-09T08:48:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.337+0000 7f8b137dc640 1 -- 172.21.15.98:0/1260159987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b0c1a0000 con 0x7f8b0c104c40 2024-05-09T08:48:25.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.339+0000 7f8b027fc640 1 -- 172.21.15.98:0/1260159987 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8b08018020 con 0x7f8b0c104c40 2024-05-09T08:48:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.339+0000 7f8b165c8740 1 Processor -- start 2024-05-09T08:48:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.339+0000 7f8b165c8740 1 -- start start 2024-05-09T08:48:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.339+0000 7f8b165c8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d5498c0 0x558a4d549ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.339+0000 7f8b165c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d54a2b0 0x558a4d54edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.339+0000 7f8b165c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d5518b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b165c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558a4d4ad4d0 con 0x558a4d54f420 2024-05-09T08:48:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b165c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558a4d4553e0 con 0x558a4d54a2b0 2024-05-09T08:48:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b165c8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558a4d4a9490 con 0x558a4d5498c0 2024-05-09T08:48:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b10d77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d5518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b10d77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d5518b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39462/0 (socket says 172.21.15.98:39462) 2024-05-09T08:48:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b10d77640 1 -- 172.21.15.98:0/4281267198 learned_addr learned my addr 172.21.15.98:0/4281267198 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b027fc640 1 --2- 172.21.15.98:0/1260159987 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8af80a2c10 0x7f8af80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b11578640 1 --2- 172.21.15.98:0/4281267198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d54a2b0 0x558a4d54edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b11d79640 1 --2- 172.21.15.98:0/4281267198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d5498c0 0x558a4d549ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b11d79640 1 --2- 172.21.15.98:0/1260159987 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8af80a2c10 0x7f8af80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b10d77640 1 -- 172.21.15.98:0/4281267198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d5498c0 msgr2=0x558a4d549ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b10d77640 1 --2- 172.21.15.98:0/4281267198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d5498c0 0x558a4d549ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b10d77640 1 -- 172.21.15.98:0/4281267198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d54a2b0 msgr2=0x558a4d54edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b10d77640 1 --2- 172.21.15.98:0/4281267198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d54a2b0 0x558a4d54edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b027fc640 1 -- 172.21.15.98:0/1260159987 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8b080ced60 con 0x7f8b0c104c40 2024-05-09T08:48:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b10d77640 1 -- 172.21.15.98:0/4281267198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a4d1c12d0 con 0x558a4d54f420 2024-05-09T08:48:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b10d77640 1 --2- 172.21.15.98:0/4281267198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d5518b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8b080cc5c0 tx=0x7f8b08010970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d7b7ab295a5f1e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8b11d79640 1 --2- 172.21.15.98:0/1260159987 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8af80a2c10 0x7f8af80a50d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8b0402ade0 tx=0x7f8b040095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8af77fe640 1 -- 172.21.15.98:0/4281267198 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0809ba20 con 0x558a4d54f420 2024-05-09T08:48:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8af77fe640 1 -- 172.21.15.98:0/4281267198 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b080a3f90 con 0x558a4d54f420 2024-05-09T08:48:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.340+0000 7f8af77fe640 1 -- 172.21.15.98:0/4281267198 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08097990 con 0x558a4d54f420 2024-05-09T08:48:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.341+0000 7f8b165c8740 1 -- 172.21.15.98:0/4281267198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 msgr2=0x558a4d5518b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.341+0000 7f8b165c8740 1 --2- 172.21.15.98:0/4281267198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d5518b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8b080cc5c0 tx=0x7f8b08010970 comp rx=0 tx=0).stop 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.341+0000 7f8b165c8740 1 -- 172.21.15.98:0/4281267198 shutdown_connections 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.341+0000 7f8b165c8740 1 --2- 172.21.15.98:0/4281267198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d5518b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.341+0000 7f8b165c8740 1 --2- 172.21.15.98:0/4281267198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d54a2b0 0x558a4d54edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.341+0000 7f8b165c8740 1 --2- 172.21.15.98:0/4281267198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d5498c0 0x558a4d549ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.341+0000 7f8b165c8740 1 -- 172.21.15.98:0/4281267198 >> 172.21.15.98:0/4281267198 conn(0x558a4d544f30 msgr2=0x558a4d545330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.341+0000 7f8b165c8740 1 -- 172.21.15.98:0/4281267198 shutdown_connections 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.341+0000 7f8b165c8740 1 -- 172.21.15.98:0/4281267198 wait complete. 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.341+0000 7f8b165c8740 1 -- start start 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.341+0000 7f8b165c8740 1 Processor -- start 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b165c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d5498c0 0x558a4d553ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b165c8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d54a2b0 0x558a4d554220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b165c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d55b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b165c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558a4d552370 con 0x558a4d54f420 2024-05-09T08:48:25.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b165c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558a4d4688e0 con 0x558a4d5498c0 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b165c8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558a4d5524f0 con 0x558a4d54a2b0 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b165c8740 10 client.0 _ll_register_callbacks cb 0x558a4d4359c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b11d79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d55b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b11d79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d55b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39470/0 (socket says 172.21.15.98:39470) 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b11d79640 1 -- 172.21.15.98:0/1080129597 learned_addr learned my addr 172.21.15.98:0/1080129597 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b10d77640 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d54a2b0 0x558a4d554220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b11578640 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d5498c0 0x558a4d553ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b11d79640 1 -- 172.21.15.98:0/1080129597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d54a2b0 msgr2=0x558a4d554220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b11d79640 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d54a2b0 0x558a4d554220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b11d79640 1 -- 172.21.15.98:0/1080129597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d5498c0 msgr2=0x558a4d553ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b11d79640 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d5498c0 0x558a4d553ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b11d79640 1 -- 172.21.15.98:0/1080129597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a4d28b6b0 con 0x558a4d54f420 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b11578640 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d5498c0 0x558a4d553ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b10d77640 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d54a2b0 0x558a4d554220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:25.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8b11d79640 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d55b250 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8b04031f30 tx=0x7f8b0402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982b3d859cfd80b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.342+0000 7f8af4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8af4ff9640 1 -- 172.21.15.98:0/1080129597 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b04003970 con 0x558a4d54f420 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8af4ff9640 1 -- 172.21.15.98:0/1080129597 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8b04003b10 con 0x558a4d54f420 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8b165c8740 1 -- 172.21.15.98:0/1080129597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b080a3f90 con 0x558a4d54f420 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8af4ff9640 1 -- 172.21.15.98:0/1080129597 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f8b04003d10 con 0x558a4d54f420 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8b165c8740 10 client.14775 fetch_fsmap learned FSMap version 17 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8b165c8740 1 -- 172.21.15.98:0/1080129597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x558a4d55e230 con 0x558a4d54f420 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8af4ff9640 1 -- 172.21.15.98:0/1080129597 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b080a3f90 con 0x558a4d54f420 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8af4ff9640 1 -- 172.21.15.98:0/1080129597 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f8b040305c0 con 0x558a4d54f420 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8b165c8740 10 client.14775 fetch_fsmap finished waiting for FSMap version 17 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8b165c8740 10 client.14775 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8b165c8740 20 client.14775 populate_metadata read hostname 'smithi098' 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8b165c8740 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a4d560940 0x558a4d561db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8b165c8740 4 client.14775 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:48:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8b165c8740 1 -- 172.21.15.98:0/1080129597 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a4d4bac00 con 0x558a4d560940 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.343+0000 7f8b11578640 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a4d560940 0x558a4d561db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.344+0000 7f8af4ff9640 10 client.14775 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.344+0000 7f8b11578640 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a4d560940 0x558a4d561db0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4e8342f1dd879e server_cookie=bc5f6d9848ad85ff in_seq=0 out_seq=0 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8af4ff9640 1 -- 172.21.15.98:0/1080129597 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558a4d4bac00 con 0x558a4d560940 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8af4ff9640 10 client.14775 handle_command_reply: tid=0 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8b165c8740 1 client.14775 shutdown 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8b165c8740 1 -- 172.21.15.98:0/1080129597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 msgr2=0x558a4d55b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8b165c8740 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d55b250 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8b04031f30 tx=0x7f8b0402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8b165c8740 1 -- 172.21.15.98:0/1080129597 shutdown_connections 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8b165c8740 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a4d560940 0x558a4d561db0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8b165c8740 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d55b250 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8af4ff9640 0 client.14775 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8b165c8740 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d54a2b0 0x558a4d554220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8b165c8740 1 --2- 172.21.15.98:0/1080129597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d5498c0 0x558a4d553ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8b165c8740 1 -- 172.21.15.98:0/1080129597 >> 172.21.15.98:0/1080129597 conn(0x558a4d544f30 msgr2=0x558a4d544000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:25.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8b165c8740 1 -- 172.21.15.98:0/1080129597 shutdown_connections 2024-05-09T08:48:25.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8b165c8740 1 -- 172.21.15.98:0/1080129597 wait complete. 2024-05-09T08:48:25.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.345+0000 7f8b165c8740 20 client.14775 trim_cache size 0 max 16384 2024-05-09T08:48:25.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b165c8740 1 Processor -- start 2024-05-09T08:48:25.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b165c8740 1 -- start start 2024-05-09T08:48:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b165c8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d5498c0 0x558a4d54ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b165c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d54a2b0 0x558a4d553ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b165c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d554220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b165c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558a4d5600f0 con 0x558a4d54f420 2024-05-09T08:48:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b165c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558a4d5607c0 con 0x558a4d54a2b0 2024-05-09T08:48:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b165c8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558a4d560e90 con 0x558a4d5498c0 2024-05-09T08:48:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b11d79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d554220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b11d79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d554220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39476/0 (socket says 172.21.15.98:39476) 2024-05-09T08:48:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b11d79640 1 -- 172.21.15.98:0/2953970670 learned_addr learned my addr 172.21.15.98:0/2953970670 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b11578640 1 --2- 172.21.15.98:0/2953970670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d5498c0 0x558a4d54ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b10d77640 1 --2- 172.21.15.98:0/2953970670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d54a2b0 0x558a4d553ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b11d79640 1 -- 172.21.15.98:0/2953970670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d5498c0 msgr2=0x558a4d54ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b11d79640 1 --2- 172.21.15.98:0/2953970670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d5498c0 0x558a4d54ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b11d79640 1 -- 172.21.15.98:0/2953970670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d54a2b0 msgr2=0x558a4d553ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b11d79640 1 --2- 172.21.15.98:0/2953970670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d54a2b0 0x558a4d553ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b11d79640 1 -- 172.21.15.98:0/2953970670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a4d545ca0 con 0x558a4d54f420 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8b11d79640 1 --2- 172.21.15.98:0/2953970670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d554220 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8b04030990 tx=0x7f8b04009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b65896180db593 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.363+0000 7f8af6ffd640 1 -- 172.21.15.98:0/2953970670 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0402f9e0 con 0x558a4d54f420 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.364+0000 7f8af6ffd640 1 -- 172.21.15.98:0/2953970670 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b04036040 con 0x558a4d54f420 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.364+0000 7f8af6ffd640 1 -- 172.21.15.98:0/2953970670 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b04030e60 con 0x558a4d54f420 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.364+0000 7f8b165c8740 1 -- 172.21.15.98:0/2953970670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 msgr2=0x558a4d554220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.364+0000 7f8b165c8740 1 --2- 172.21.15.98:0/2953970670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d554220 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8b04030990 tx=0x7f8b04009bf0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.364+0000 7f8b165c8740 1 -- 172.21.15.98:0/2953970670 shutdown_connections 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.364+0000 7f8b165c8740 1 --2- 172.21.15.98:0/2953970670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d54f420 0x558a4d554220 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.364+0000 7f8b165c8740 1 --2- 172.21.15.98:0/2953970670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d54a2b0 0x558a4d553ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.364+0000 7f8b165c8740 1 --2- 172.21.15.98:0/2953970670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d5498c0 0x558a4d54ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.364+0000 7f8b165c8740 1 -- 172.21.15.98:0/2953970670 >> 172.21.15.98:0/2953970670 conn(0x558a4d544f30 msgr2=0x558a4d54b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.364+0000 7f8b165c8740 1 -- 172.21.15.98:0/2953970670 shutdown_connections 2024-05-09T08:48:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.364+0000 7f8b165c8740 1 -- 172.21.15.98:0/2953970670 wait complete. 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.364+0000 7f8b165c8740 1 -- start start 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.364+0000 7f8b165c8740 1 Processor -- start 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b165c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d5498c0 0x558a4d5c2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b165c8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d54a2b0 0x558a4d5c3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b165c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d54f420 0x558a4d5c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b165c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558a4d561c30 con 0x558a4d5498c0 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b165c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558a4d561560 con 0x558a4d54f420 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b165c8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558a4d562300 con 0x558a4d54a2b0 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b165c8740 10 client.0 _ll_register_callbacks cb 0x7f8b04009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b11578640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d5498c0 0x558a4d5c2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b11578640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d5498c0 0x558a4d5c2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39482/0 (socket says 172.21.15.98:39482) 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b11578640 1 -- 172.21.15.98:0/1273913523 learned_addr learned my addr 172.21.15.98:0/1273913523 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b10d77640 1 --2- 172.21.15.98:0/1273913523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d54a2b0 0x558a4d5c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b11578640 1 -- 172.21.15.98:0/1273913523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d54a2b0 msgr2=0x558a4d5c3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b11578640 1 --2- 172.21.15.98:0/1273913523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d54a2b0 0x558a4d5c3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b11578640 1 -- 172.21.15.98:0/1273913523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d54f420 msgr2=0x558a4d5c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:25.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b11578640 1 --2- 172.21.15.98:0/1273913523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d54f420 0x558a4d5c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b11578640 1 -- 172.21.15.98:0/1273913523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a4d5c9f90 con 0x558a4d5498c0 2024-05-09T08:48:25.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8b11578640 1 --2- 172.21.15.98:0/1273913523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d5498c0 0x558a4d5c2b20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8afc000c00 tx=0x7f8afc01fa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad667a90905a1e53 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:25.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8af4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:48:25.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.365+0000 7f8af4ff9640 1 -- 172.21.15.98:0/1273913523 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc004020 con 0x558a4d5498c0 2024-05-09T08:48:25.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.366+0000 7f8af4ff9640 1 -- 172.21.15.98:0/1273913523 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8afc0041c0 con 0x558a4d5498c0 2024-05-09T08:48:25.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.366+0000 7f8b165c8740 1 -- 172.21.15.98:0/1273913523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b04036040 con 0x558a4d5498c0 2024-05-09T08:48:25.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.366+0000 7f8af4ff9640 1 -- 172.21.15.98:0/1273913523 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f8afc0043c0 con 0x558a4d5498c0 2024-05-09T08:48:25.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.366+0000 7f8b165c8740 10 client.14781 fetch_fsmap learned FSMap version 17 2024-05-09T08:48:25.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.366+0000 7f8b165c8740 1 -- 172.21.15.98:0/1273913523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x558a4d5cb360 con 0x558a4d5498c0 2024-05-09T08:48:25.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.366+0000 7f8af4ff9640 1 -- 172.21.15.98:0/1273913523 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b04036040 con 0x558a4d5498c0 2024-05-09T08:48:25.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.366+0000 7f8af4ff9640 1 -- 172.21.15.98:0/1273913523 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f8afc026070 con 0x558a4d5498c0 2024-05-09T08:48:25.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.366+0000 7f8b165c8740 10 client.14781 fetch_fsmap finished waiting for FSMap version 17 2024-05-09T08:48:25.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.366+0000 7f8b165c8740 10 client.14781 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-09T08:48:25.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.366+0000 7f8b165c8740 20 client.14781 populate_metadata read hostname 'smithi098' 2024-05-09T08:48:25.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.366+0000 7f8b165c8740 1 --2- 172.21.15.98:0/1273913523 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a4d5d0010 0x558a4d5ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:25.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.366+0000 7f8b165c8740 4 client.14781 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:48:25.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.366+0000 7f8b165c8740 1 -- 172.21.15.98:0/1273913523 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "status"}) -- 0x558a4d4b4f50 con 0x558a4d5d0010 2024-05-09T08:48:25.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.366+0000 7f8b10d77640 1 --2- 172.21.15.98:0/1273913523 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a4d5d0010 0x558a4d5ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:25.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8af4ff9640 10 client.14781 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:48:25.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8b10d77640 1 --2- 172.21.15.98:0/1273913523 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a4d5d0010 0x558a4d5ced20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54a5661a9b107095 server_cookie=d120e5c5c91a243d in_seq=0 out_seq=0 2024-05-09T08:48:25.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8af4ff9640 1 -- 172.21.15.98:0/1273913523 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558a4d4b4f50 con 0x558a4d5d0010 2024-05-09T08:48:25.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8af4ff9640 10 client.14781 handle_command_reply: tid=0 2024-05-09T08:48:25.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8b165c8740 1 client.14781 shutdown 2024-05-09T08:48:25.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8b165c8740 1 -- 172.21.15.98:0/1273913523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d5498c0 msgr2=0x558a4d5c2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:25.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8b165c8740 1 --2- 172.21.15.98:0/1273913523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d5498c0 0x558a4d5c2b20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8afc000c00 tx=0x7f8afc01fa20 comp rx=0 tx=0).stop 2024-05-09T08:48:25.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8b165c8740 1 -- 172.21.15.98:0/1273913523 shutdown_connections 2024-05-09T08:48:25.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8b165c8740 1 --2- 172.21.15.98:0/1273913523 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a4d5d0010 0x558a4d5ced20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8b165c8740 1 --2- 172.21.15.98:0/1273913523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d54f420 0x558a4d5c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8af4ff9640 0 client.14781 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:48:25.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8b165c8740 1 --2- 172.21.15.98:0/1273913523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a4d54a2b0 0x558a4d5c3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8b165c8740 1 --2- 172.21.15.98:0/1273913523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a4d5498c0 0x558a4d5c2b20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8b165c8740 1 -- 172.21.15.98:0/1273913523 >> 172.21.15.98:0/1273913523 conn(0x558a4d544f30 msgr2=0x558a4d55f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:25.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8b165c8740 1 -- 172.21.15.98:0/1273913523 shutdown_connections 2024-05-09T08:48:25.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8b165c8740 1 -- 172.21.15.98:0/1273913523 wait complete. 2024-05-09T08:48:25.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.367+0000 7f8b165c8740 20 client.14781 trim_cache size 0 max 16384 2024-05-09T08:48:25.373 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:48:25.373 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a57d9712-0ddf-11ef-bc98-c7b262605968", 2024-05-09T08:48:25.374 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-05-09T08:48:25.374 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24461, 2024-05-09T08:48:25.374 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-05-09T08:48:25.374 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-05-09T08:48:25.374 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-05-09T08:48:25.374 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 6.6404099069999996, 2024-05-09T08:48:25.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 17, 2024-05-09T08:48:25.374 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 90, 2024-05-09T08:48:25.374 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 90, 2024-05-09T08:48:25.374 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 24.724302169000001 2024-05-09T08:48:25.374 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:48:25.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.368+0000 7f8b137dc640 1 -- 172.21.15.98:0/1260159987 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8af80a2c10 msgr2=0x7f8af80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:25.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.368+0000 7f8b137dc640 1 --2- 172.21.15.98:0/1260159987 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8af80a2c10 0x7f8af80a50d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8b0402ade0 tx=0x7f8b040095c0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.368+0000 7f8b137dc640 1 -- 172.21.15.98:0/1260159987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b0c104c40 msgr2=0x7f8b0c1980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:25.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.368+0000 7f8b137dc640 1 --2- 172.21.15.98:0/1260159987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b0c104c40 0x7f8b0c1980e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800b450 tx=0x7f8b0800b910 comp rx=0 tx=0).stop 2024-05-09T08:48:25.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.368+0000 7f8b137dc640 1 -- 172.21.15.98:0/1260159987 shutdown_connections 2024-05-09T08:48:25.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.368+0000 7f8b137dc640 1 --2- 172.21.15.98:0/1260159987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b0c107570 0x7f8b0c19f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.368+0000 7f8b137dc640 1 --2- 172.21.15.98:0/1260159987 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8af80a2c10 0x7f8af80a50d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.368+0000 7f8b137dc640 1 --2- 172.21.15.98:0/1260159987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b0c104c40 0x7f8b0c1980e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.368+0000 7f8b137dc640 1 --2- 172.21.15.98:0/1260159987 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b0c102310 0x7f8b0c197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:25.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.368+0000 7f8b137dc640 1 -- 172.21.15.98:0/1260159987 >> 172.21.15.98:0/1260159987 conn(0x7f8b0c0fbd20 msgr2=0x7f8b0c102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:25.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.368+0000 7f8b137dc640 1 -- 172.21.15.98:0/1260159987 shutdown_connections 2024-05-09T08:48:25.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:25.368+0000 7f8b137dc640 1 -- 172.21.15.98:0/1260159987 wait complete. 2024-05-09T08:48:25.378 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T08:48:25.378 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:48:25.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph fs ls 2024-05-09T08:48:25.621 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:25.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:25 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1263346748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:25.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:25 smithi098 ceph-mon[30092]: pgmap v294: 129 pgs: 4 creating+peering, 120 active+clean, 5 unknown; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 7 op/s 2024-05-09T08:48:25.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:25 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1331945364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:25.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:25 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1263346748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:25.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:25 smithi142 ceph-mon[34625]: pgmap v294: 129 pgs: 4 creating+peering, 120 active+clean, 5 unknown; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 7 op/s 2024-05-09T08:48:25.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:25 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1331945364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:25.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:25 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1263346748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:25.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:25 smithi105 ceph-mon[34498]: pgmap v294: 129 pgs: 4 creating+peering, 120 active+clean, 5 unknown; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 7 op/s 2024-05-09T08:48:25.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:25 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1331945364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:26.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:26 smithi142 ceph-mon[34625]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:48:26.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:26 smithi142 ceph-mon[34625]: Cluster is now healthy 2024-05-09T08:48:26.913 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:26 smithi098 ceph-mon[30092]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:48:26.913 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:26 smithi098 ceph-mon[30092]: Cluster is now healthy 2024-05-09T08:48:26.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:26 smithi105 ceph-mon[34498]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:48:26.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:26 smithi105 ceph-mon[34498]: Cluster is now healthy 2024-05-09T08:48:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.289+0000 7f5b42b90700 1 -- 172.21.15.98:0/3809656923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b3c1083a0 msgr2=0x7f5b3c108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.289+0000 7f5b42b90700 1 --2- 172.21.15.98:0/3809656923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b3c1083a0 0x7f5b3c108780 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009990 tx=0x7f5b30009ca0 comp rx=0 tx=0).stop 2024-05-09T08:48:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.289+0000 7f5b42b90700 1 -- 172.21.15.98:0/3809656923 shutdown_connections 2024-05-09T08:48:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.289+0000 7f5b42b90700 1 --2- 172.21.15.98:0/3809656923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b3c06a9d0 0x7f5b3c10f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.289+0000 7f5b42b90700 1 --2- 172.21.15.98:0/3809656923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b3c1083a0 0x7f5b3c108780 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.289+0000 7f5b42b90700 1 --2- 172.21.15.98:0/3809656923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b3c06a030 0x7f5b3c06a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.289+0000 7f5b42b90700 1 -- 172.21.15.98:0/3809656923 >> 172.21.15.98:0/3809656923 conn(0x7f5b3c0781d0 msgr2=0x7f5b3c0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.289+0000 7f5b42b90700 1 -- 172.21.15.98:0/3809656923 shutdown_connections 2024-05-09T08:48:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.289+0000 7f5b42b90700 1 -- 172.21.15.98:0/3809656923 wait complete. 2024-05-09T08:48:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.290+0000 7f5b42b90700 1 Processor -- start 2024-05-09T08:48:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.290+0000 7f5b42b90700 1 -- start start 2024-05-09T08:48:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.290+0000 7f5b42b90700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b3c06a030 0x7f5b3c19e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.290+0000 7f5b42b90700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b3c06a9d0 0x7f5b3c19ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.290+0000 7f5b42b90700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b3c1083a0 0x7f5b3c198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.290+0000 7f5b42b90700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c1125e0 con 0x7f5b3c1083a0 2024-05-09T08:48:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.290+0000 7f5b42b90700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c112460 con 0x7f5b3c06a030 2024-05-09T08:48:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.290+0000 7f5b42b90700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c112760 con 0x7f5b3c06a9d0 2024-05-09T08:48:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.290+0000 7f5b4112d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b3c1083a0 0x7f5b3c198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.290+0000 7f5b4112d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b3c1083a0 0x7f5b3c198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36606/0 (socket says 172.21.15.98:36606) 2024-05-09T08:48:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.290+0000 7f5b4112d700 1 -- 172.21.15.98:0/3101600639 learned_addr learned my addr 172.21.15.98:0/3101600639 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.290+0000 7f5b3bfff700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b3c06a9d0 0x7f5b3c19ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.290+0000 7f5b4092c700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b3c06a030 0x7f5b3c19e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.291+0000 7f5b3bfff700 1 -- 172.21.15.98:0/3101600639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b3c06a030 msgr2=0x7f5b3c19e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.291+0000 7f5b3bfff700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b3c06a030 0x7f5b3c19e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.291+0000 7f5b3bfff700 1 -- 172.21.15.98:0/3101600639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b3c1083a0 msgr2=0x7f5b3c198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.291+0000 7f5b3bfff700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b3c1083a0 0x7f5b3c198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.291+0000 7f5b3bfff700 1 -- 172.21.15.98:0/3101600639 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3c1990c0 con 0x7f5b3c06a9d0 2024-05-09T08:48:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.291+0000 7f5b4112d700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b3c1083a0 0x7f5b3c198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:48:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.291+0000 7f5b3bfff700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b3c06a9d0 0x7f5b3c19ece0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800d950 tx=0x7f5b2800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.291+0000 7f5b39ffb700 1 -- 172.21.15.98:0/3101600639 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b28016070 con 0x7f5b3c06a9d0 2024-05-09T08:48:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.291+0000 7f5b42b90700 1 -- 172.21.15.98:0/3101600639 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b3c1993b0 con 0x7f5b3c06a9d0 2024-05-09T08:48:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.291+0000 7f5b39ffb700 1 -- 172.21.15.98:0/3101600639 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b2800fd70 con 0x7f5b3c06a9d0 2024-05-09T08:48:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.291+0000 7f5b39ffb700 1 -- 172.21.15.98:0/3101600639 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2800e3b0 con 0x7f5b3c06a9d0 2024-05-09T08:48:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.291+0000 7f5b42b90700 1 -- 172.21.15.98:0/3101600639 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b3c1a5670 con 0x7f5b3c06a9d0 2024-05-09T08:48:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.292+0000 7f5b42b90700 1 -- 172.21.15.98:0/3101600639 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b3c069010 con 0x7f5b3c06a9d0 2024-05-09T08:48:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.293+0000 7f5b39ffb700 1 -- 172.21.15.98:0/3101600639 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5b28019070 con 0x7f5b3c06a9d0 2024-05-09T08:48:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.294+0000 7f5b39ffb700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5b2c0a2c20 0x7f5b2c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.294+0000 7f5b4092c700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5b2c0a2c20 0x7f5b2c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.294+0000 7f5b39ffb700 1 -- 172.21.15.98:0/3101600639 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f5b280ced70 con 0x7f5b3c06a9d0 2024-05-09T08:48:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.294+0000 7f5b4092c700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5b2c0a2c20 0x7f5b2c0a50e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000b2e0 tx=0x7f5b3001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:27.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.297+0000 7f5b39ffb700 1 -- 172.21.15.98:0/3101600639 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5b28010460 con 0x7f5b3c06a9d0 2024-05-09T08:48:27.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.536+0000 7f5b42b90700 1 -- 172.21.15.98:0/3101600639 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5b3c199e90 con 0x7f5b3c06a9d0 2024-05-09T08:48:27.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.536+0000 7f5b39ffb700 1 -- 172.21.15.98:0/3101600639 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f5b2809ba90 con 0x7f5b3c06a9d0 2024-05-09T08:48:27.536 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-09T08:48:27.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.539+0000 7f5b42b90700 1 -- 172.21.15.98:0/3101600639 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5b2c0a2c20 msgr2=0x7f5b2c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:27.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.539+0000 7f5b42b90700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5b2c0a2c20 0x7f5b2c0a50e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000b2e0 tx=0x7f5b3001a040 comp rx=0 tx=0).stop 2024-05-09T08:48:27.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.539+0000 7f5b42b90700 1 -- 172.21.15.98:0/3101600639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b3c06a9d0 msgr2=0x7f5b3c19ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:27.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.539+0000 7f5b42b90700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b3c06a9d0 0x7f5b3c19ece0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800d950 tx=0x7f5b2800dc60 comp rx=0 tx=0).stop 2024-05-09T08:48:27.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.539+0000 7f5b42b90700 1 -- 172.21.15.98:0/3101600639 shutdown_connections 2024-05-09T08:48:27.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.539+0000 7f5b42b90700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5b2c0a2c20 0x7f5b2c0a50e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:27.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.539+0000 7f5b42b90700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b3c06a030 0x7f5b3c19e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:27.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.539+0000 7f5b42b90700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b3c1083a0 0x7f5b3c198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:27.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.539+0000 7f5b42b90700 1 --2- 172.21.15.98:0/3101600639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b3c06a9d0 0x7f5b3c19ece0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:27.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.539+0000 7f5b42b90700 1 -- 172.21.15.98:0/3101600639 >> 172.21.15.98:0/3101600639 conn(0x7f5b3c0781d0 msgr2=0x7f5b3c0fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:27.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.539+0000 7f5b42b90700 1 -- 172.21.15.98:0/3101600639 shutdown_connections 2024-05-09T08:48:27.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:27.539+0000 7f5b42b90700 1 -- 172.21.15.98:0/3101600639 wait complete. 2024-05-09T08:48:27.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:27 smithi098 ceph-mon[30092]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-05-09T08:48:27.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:27 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3101600639' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:48:27.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:27 smithi142 ceph-mon[34625]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-05-09T08:48:27.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:27 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3101600639' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:48:27.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:27 smithi105 ceph-mon[34498]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-05-09T08:48:27.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:27 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3101600639' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:48:28.257 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-09T08:48:28.257 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T08:48:28.258 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-05-09T08:48:28.258 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-05-09T08:48:28.258 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:48:28.258 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-09T08:48:28.258 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T08:48:28.258 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T08:48:28.258 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-05-09T08:48:28.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:48:28.258 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-05-09T08:48:28.276 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T08:48:28.276 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T08:48:28.276 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T08:48:28.276 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:48:28.276 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-05-09T08:48:28.276 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:48:28.276 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:48:28.276 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-05-09T08:48:28.276 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-05-09T08:48:28.276 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:48:28.277 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-05-09T08:48:28.277 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-05-09T08:48:28.277 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:48:28.277 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-05-09T08:48:28.277 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T08:48:28.277 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-05-09T08:48:28.277 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-09T08:48:28.277 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42245sec preferred_lft 42245sec 2024-05-09T08:48:28.277 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-05-09T08:48:28.277 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:48:28.278 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T08:48:28.278 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:48:28.278 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-09T08:48:28.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2024-05-09T08:48:28.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2024-05-09T08:48:28.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2024-05-09T08:48:28.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T08:48:28.278 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-09T08:48:28.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:48:28.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:48:28.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:48:28.489 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T08:48:28.514 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-05-09T08:48:28.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:48:28.515 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-05-09T08:48:28.578 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-05-09T08:48:28.579 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T08:48:28.579 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T08:48:28.579 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T08:48:28.579 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T08:48:28.579 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:48:28.579 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-09T08:48:28.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T08:48:28.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T08:48:28.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T08:48:28.580 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-09T08:48:28.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:48:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:48:28.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:48:28.850 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-05-09T08:48:28.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:48:28.866 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-05-09T08:48:28.922 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:48:28.922 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-09T08:48:28.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:48:28.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-05-09T08:48:28.922 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-09T08:48:29.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:48:29.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:48:29.041 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-05-09T08:48:29.041 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:48:29.041 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-09T08:48:29.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-05-09T08:48:29.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T08:48:29.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-05-09T08:48:29.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-05-09T08:48:29.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-05-09T08:48:29.042 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-09T08:48:29.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:48:29.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:48:29.165 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:48:29.165 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-09T08:48:29.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-05-09T08:48:29.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T08:48:29.165 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-09T08:48:29.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:48:29.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:48:29.245 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-05-09T08:48:29.245 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T08:48:29.246 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:48:29.260 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-05-09T08:48:29.261 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T08:48:29.261 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:48:29.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-05-09T08:48:29.381 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T08:48:29.437 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-05-09T08:48:29.437 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-05-09T08:48:29.437 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-05-09T08:48:29.438 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-05-09T08:48:29.438 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-05-09T08:48:29.438 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-05-09T08:48:29.438 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-05-09T08:48:29.438 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-05-09T08:48:29.438 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-05-09T08:48:29.438 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-05-09T08:48:29.438 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-05-09T08:48:29.438 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/selinux 2024-05-09T08:48:29.438 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:48:29.438 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-05-09T08:48:29.438 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-05-09T08:48:29.438 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-05-09T08:48:29.438 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/06234a570fa54b0c907ecd1beb6714cda70bd7fd0e0fa7d0036ac58b385534ab/merged 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/63cbc1b459868410a8d2402ec2be153eef777b13455d64415c0651585d58c692/merged 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/0 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/39e945433b97066de337a28ca2696c4b91ca998ff142a42e94f26c77bba5fbf0/merged 2024-05-09T08:48:29.439 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/0da3bb0620a70c1beae2808051217bb1c8cfd06d2e8ed2621d51041c8e471169/merged 2024-05-09T08:48:29.440 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/c675c4886ed88d81d05e733e4d39b310f011ca6569cd80a5176b06b0a0119be0/merged 2024-05-09T08:48:29.440 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/3b3a3c186b81a531e195eb5fbc115ec2e6367b7a610950c8620acd1b7ddcace8/merged 2024-05-09T08:48:29.440 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/94c656271fb5cc5476dbac54a7bd28ab147d21c8b485817572993f234edeffdb/merged 2024-05-09T08:48:29.440 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/c2d875c591c5a88aaf241004b6ee5aa48690cbeac157a2acf690e4f85d2c8031/merged 2024-05-09T08:48:29.440 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/3099ed1a271bb57d85223b477dfc4a5f450997ff242c6275c573127729a8ce43/merged 2024-05-09T08:48:29.440 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/66452991546e6d6216295f6906932b90287eb9852a764d04bd070cf60ba43f35/merged 2024-05-09T08:48:29.440 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-05-09T08:48:29.440 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:48:29.440 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:48:29.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:48:29.440 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-05-09T08:48:29.493 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T08:48:29.493 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-05-09T08:48:29.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-05-09T08:48:29.560 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T08:48:29.616 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-05-09T08:48:29.616 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-05-09T08:48:29.616 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-05-09T08:48:29.616 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-05-09T08:48:29.616 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-05-09T08:48:29.617 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-05-09T08:48:29.617 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-05-09T08:48:29.617 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-05-09T08:48:29.617 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-05-09T08:48:29.617 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-05-09T08:48:29.617 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-05-09T08:48:29.617 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/selinux 2024-05-09T08:48:29.617 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:48:29.617 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-05-09T08:48:29.617 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-05-09T08:48:29.617 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-05-09T08:48:29.617 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-05-09T08:48:29.617 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T08:48:29.617 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-05-09T08:48:29.618 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-05-09T08:48:29.618 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T08:48:29.618 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T08:48:29.618 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T08:48:29.618 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-05-09T08:48:29.618 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-05-09T08:48:29.618 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay 2024-05-09T08:48:29.618 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/06234a570fa54b0c907ecd1beb6714cda70bd7fd0e0fa7d0036ac58b385534ab/merged 2024-05-09T08:48:29.618 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/63cbc1b459868410a8d2402ec2be153eef777b13455d64415c0651585d58c692/merged 2024-05-09T08:48:29.618 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/0 2024-05-09T08:48:29.618 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:48:29.618 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/39e945433b97066de337a28ca2696c4b91ca998ff142a42e94f26c77bba5fbf0/merged 2024-05-09T08:48:29.618 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/0da3bb0620a70c1beae2808051217bb1c8cfd06d2e8ed2621d51041c8e471169/merged 2024-05-09T08:48:29.618 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/c675c4886ed88d81d05e733e4d39b310f011ca6569cd80a5176b06b0a0119be0/merged 2024-05-09T08:48:29.619 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/3b3a3c186b81a531e195eb5fbc115ec2e6367b7a610950c8620acd1b7ddcace8/merged 2024-05-09T08:48:29.619 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/94c656271fb5cc5476dbac54a7bd28ab147d21c8b485817572993f234edeffdb/merged 2024-05-09T08:48:29.619 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/c2d875c591c5a88aaf241004b6ee5aa48690cbeac157a2acf690e4f85d2c8031/merged 2024-05-09T08:48:29.619 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/3099ed1a271bb57d85223b477dfc4a5f450997ff242c6275c573127729a8ce43/merged 2024-05-09T08:48:29.619 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/66452991546e6d6216295f6906932b90287eb9852a764d04bd070cf60ba43f35/merged 2024-05-09T08:48:29.619 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-05-09T08:48:29.619 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:48:29.619 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:48:29.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:48:29.619 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-05-09T08:48:29.637 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:2024-05-09T08:48:29.638+0000 7faec0521480 -1 init, newargv = 0x55af7be03480 newargc=15 2024-05-09T08:48:29.638 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[74261]: starting ceph client 2024-05-09T08:48:29.642 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[74261]: starting fuse 2024-05-09T08:48:29.653 INFO:teuthology.orchestra.run.smithi098.stdout:114 2024-05-09T08:48:29.653 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-05-09T08:48:29.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo stdin-killer -- python3 -c ' 2024-05-09T08:48:29.653 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-05-09T08:48:29.653 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-05-09T08:48:29.653 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-05-09T08:48:29.653 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-05-09T08:48:29.653 DEBUG:teuthology.orchestra.run.smithi098:> 2024-05-09T08:48:29.653 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-05-09T08:48:29.653 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-05-09T08:48:29.654 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-05-09T08:48:29.655 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T08:48:29.655 DEBUG:teuthology.orchestra.run.smithi098:> 2024-05-09T08:48:29.655 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.admin")) 2024-05-09T08:48:29.655 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-05-09T08:48:29.772 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.admin.74261.asok 2024-05-09T08:48:29.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:48:29.783 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.74261.asok 2024-05-09T08:48:29.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:48:29.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.74261.asok status 2024-05-09T08:48:29.918 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:48:29.918 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-05-09T08:48:29.918 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "96805dcf909541c4ca9a1e90258de8ae5b6fa1bc", 2024-05-09T08:48:29.918 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T08:48:29.918 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "admin", 2024-05-09T08:48:29.918 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-05-09T08:48:29.918 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-05-09T08:48:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "74261", 2024-05-09T08:48:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2024-05-09T08:48:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T08:48:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-05-09T08:48:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-05-09T08:48:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 14793, 2024-05-09T08:48:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-05-09T08:48:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-05-09T08:48:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-05-09T08:48:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 14793 2024-05-09T08:48:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T08:48:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-05-09T08:48:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-05-09T08:48:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 621801090 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 621801090 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.14793 192.168.0.1:0/621801090", 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/621801090", 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 17, 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 90, 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2024-05-09T08:48:29.920 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-05-09T08:48:29.921 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-05-09T08:48:29.921 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:48:29.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:48:29.926 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-05-09T08:48:29.926 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:48:29.942 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-05-09T08:48:29.942 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:48:29.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:48:29.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:48:30.009 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-05-09T08:48:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T08:48:30.117 INFO:teuthology.orchestra.run.smithi098.stdout:# file: . 2024-05-09T08:48:30.117 INFO:teuthology.orchestra.run.smithi098.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-05-09T08:48:30.118 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:30.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:48:30.123 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:48:30.124 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T08:48:30.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:48:30.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.284+0000 7f5068bc8640 1 -- 172.21.15.98:0/3482932458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5064103570 msgr2=0x7f5064109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:30.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.284+0000 7f5068bc8640 1 --2- 172.21.15.98:0/3482932458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5064103570 0x7f5064109e00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5054009930 tx=0x7f505402a420 comp rx=0 tx=0).stop 2024-05-09T08:48:30.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.284+0000 7f5068bc8640 1 -- 172.21.15.98:0/3482932458 shutdown_connections 2024-05-09T08:48:30.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.284+0000 7f5068bc8640 1 --2- 172.21.15.98:0/3482932458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5064103570 0x7f5064109e00 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.284+0000 7f5068bc8640 1 --2- 172.21.15.98:0/3482932458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5064102bb0 0x7f5064103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.284+0000 7f5068bc8640 1 --2- 172.21.15.98:0/3482932458 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50640ffbe0 0x7f5064100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.284+0000 7f5068bc8640 1 -- 172.21.15.98:0/3482932458 >> 172.21.15.98:0/3482932458 conn(0x7f50640fb510 msgr2=0x7f50640fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.284+0000 7f5068bc8640 1 -- 172.21.15.98:0/3482932458 shutdown_connections 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.284+0000 7f5068bc8640 1 -- 172.21.15.98:0/3482932458 wait complete. 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.284+0000 7f5068bc8640 1 Processor -- start 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.284+0000 7f5068bc8640 1 -- start start 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5068bc8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50640ffbe0 0x7f5064197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5068bc8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5064102bb0 0x7f5064198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5068bc8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5064103570 0x7f506419f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5068bc8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f506410d610 con 0x7f5064102bb0 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5068bc8640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f506410d490 con 0x7f50640ffbe0 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5068bc8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f506410d790 con 0x7f5064103570 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5061d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5064102bb0 0x7f5064198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5061d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5064102bb0 0x7f5064198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36632/0 (socket says 172.21.15.98:36632) 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5061d9b640 1 -- 172.21.15.98:0/1030144050 learned_addr learned my addr 172.21.15.98:0/1030144050 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f506259c640 1 --2- 172.21.15.98:0/1030144050 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50640ffbe0 0x7f5064197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5062d9d640 1 --2- 172.21.15.98:0/1030144050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5064103570 0x7f506419f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5061d9b640 1 -- 172.21.15.98:0/1030144050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5064103570 msgr2=0x7f506419f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5061d9b640 1 --2- 172.21.15.98:0/1030144050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5064103570 0x7f506419f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5061d9b640 1 -- 172.21.15.98:0/1030144050 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50640ffbe0 msgr2=0x7f5064197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5061d9b640 1 --2- 172.21.15.98:0/1030144050 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50640ffbe0 0x7f5064197c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5061d9b640 1 -- 172.21.15.98:0/1030144050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f506419f860 con 0x7f5064102bb0 2024-05-09T08:48:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5061d9b640 1 --2- 172.21.15.98:0/1030144050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5064102bb0 0x7f5064198190 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f505800ba50 tx=0x7f505800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f50537fe640 1 -- 172.21.15.98:0/1030144050 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5058019070 con 0x7f5064102bb0 2024-05-09T08:48:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f50537fe640 1 -- 172.21.15.98:0/1030144050 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f505800cbe0 con 0x7f5064102bb0 2024-05-09T08:48:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5068bc8640 1 -- 172.21.15.98:0/1030144050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f506419fb50 con 0x7f5064102bb0 2024-05-09T08:48:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f50537fe640 1 -- 172.21.15.98:0/1030144050 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50580076e0 con 0x7f5064102bb0 2024-05-09T08:48:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.285+0000 7f5068bc8640 1 -- 172.21.15.98:0/1030144050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50641a0050 con 0x7f5064102bb0 2024-05-09T08:48:30.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.286+0000 7f5068bc8640 1 -- 172.21.15.98:0/1030144050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5064108c80 con 0x7f5064102bb0 2024-05-09T08:48:30.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.287+0000 7f50537fe640 1 -- 172.21.15.98:0/1030144050 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f505801c070 con 0x7f5064102bb0 2024-05-09T08:48:30.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.288+0000 7f50537fe640 1 --2- 172.21.15.98:0/1030144050 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f504c0a2b90 0x7f504c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:30.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.288+0000 7f506259c640 1 --2- 172.21.15.98:0/1030144050 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f504c0a2b90 0x7f504c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:30.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.288+0000 7f50537fe640 1 -- 172.21.15.98:0/1030144050 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f50580ce910 con 0x7f5064102bb0 2024-05-09T08:48:30.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.289+0000 7f506259c640 1 --2- 172.21.15.98:0/1030144050 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f504c0a2b90 0x7f504c0a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5064199120 tx=0x7f505c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:30.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.293+0000 7f50537fe640 1 -- 172.21.15.98:0/1030144050 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f505809afe0 con 0x7f5064102bb0 2024-05-09T08:48:30.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:30 smithi098 ceph-mon[30092]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 785 B/s wr, 2 op/s 2024-05-09T08:48:30.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.474+0000 7f5068bc8640 1 -- 172.21.15.98:0/1030144050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f50640624d0 con 0x7f5064102bb0 2024-05-09T08:48:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.476+0000 7f50537fe640 1 -- 172.21.15.98:0/1030144050 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24150 (secure 0 0 0) 0x7f505809f2e0 con 0x7f5064102bb0 2024-05-09T08:48:30.475 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:30.475 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":90,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:48:22.719729+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:48:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.477+0000 7f5068bc8640 1 -- 172.21.15.98:0/1030144050 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f504c0a2b90 msgr2=0x7f504c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.477+0000 7f5068bc8640 1 --2- 172.21.15.98:0/1030144050 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f504c0a2b90 0x7f504c0a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5064199120 tx=0x7f505c00b390 comp rx=0 tx=0).stop 2024-05-09T08:48:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.477+0000 7f5068bc8640 1 -- 172.21.15.98:0/1030144050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5064102bb0 msgr2=0x7f5064198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.478+0000 7f5068bc8640 1 --2- 172.21.15.98:0/1030144050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5064102bb0 0x7f5064198190 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f505800ba50 tx=0x7f505800bf10 comp rx=0 tx=0).stop 2024-05-09T08:48:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.478+0000 7f5068bc8640 1 -- 172.21.15.98:0/1030144050 shutdown_connections 2024-05-09T08:48:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.478+0000 7f5068bc8640 1 --2- 172.21.15.98:0/1030144050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5064103570 0x7f506419f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.478+0000 7f5068bc8640 1 --2- 172.21.15.98:0/1030144050 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f504c0a2b90 0x7f504c0a5050 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.478+0000 7f5068bc8640 1 --2- 172.21.15.98:0/1030144050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5064102bb0 0x7f5064198190 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.478+0000 7f5068bc8640 1 --2- 172.21.15.98:0/1030144050 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50640ffbe0 0x7f5064197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.478+0000 7f5068bc8640 1 -- 172.21.15.98:0/1030144050 >> 172.21.15.98:0/1030144050 conn(0x7f50640fb510 msgr2=0x7f5064105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.478+0000 7f5068bc8640 1 -- 172.21.15.98:0/1030144050 shutdown_connections 2024-05-09T08:48:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.478+0000 7f5068bc8640 1 -- 172.21.15.98:0/1030144050 wait complete. 2024-05-09T08:48:30.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:30 smithi105 ceph-mon[34498]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 785 B/s wr, 2 op/s 2024-05-09T08:48:30.488 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:43:39.424443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:48:15.866450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:48:17.122162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:48:20.591475+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:48:30.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:48:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.593+0000 7ff7169ab640 1 -- 172.21.15.98:0/855975110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff710102310 msgr2=0x7ff710104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.593+0000 7ff7169ab640 1 --2- 172.21.15.98:0/855975110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff710102310 0x7ff710104700 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff700009900 tx=0x7ff70002a410 comp rx=0 tx=0).stop 2024-05-09T08:48:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.593+0000 7ff7169ab640 1 -- 172.21.15.98:0/855975110 shutdown_connections 2024-05-09T08:48:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.593+0000 7ff7169ab640 1 --2- 172.21.15.98:0/855975110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff710107570 0x7ff7101099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.593+0000 7ff7169ab640 1 --2- 172.21.15.98:0/855975110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff710104c40 0x7ff710107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.593+0000 7ff7169ab640 1 --2- 172.21.15.98:0/855975110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff710102310 0x7ff710104700 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.593+0000 7ff7169ab640 1 -- 172.21.15.98:0/855975110 >> 172.21.15.98:0/855975110 conn(0x7ff7100fbd20 msgr2=0x7ff7100fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.593+0000 7ff7169ab640 1 -- 172.21.15.98:0/855975110 shutdown_connections 2024-05-09T08:48:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.593+0000 7ff7169ab640 1 -- 172.21.15.98:0/855975110 wait complete. 2024-05-09T08:48:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.593+0000 7ff7169ab640 1 Processor -- start 2024-05-09T08:48:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.593+0000 7ff7169ab640 1 -- start start 2024-05-09T08:48:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff7169ab640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff710102310 0x7ff710199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff7169ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff710104c40 0x7ff71019a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff7169ab640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff710107570 0x7ff7101a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff7169ab640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff71010d510 con 0x7ff710104c40 2024-05-09T08:48:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff7169ab640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff71010d390 con 0x7ff710102310 2024-05-09T08:48:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff7169ab640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff71010d690 con 0x7ff710107570 2024-05-09T08:48:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff70f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff710104c40 0x7ff71019a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff70f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff710104c40 0x7ff71019a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36656/0 (socket says 172.21.15.98:36656) 2024-05-09T08:48:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff70f7fe640 1 -- 172.21.15.98:0/2060746397 learned_addr learned my addr 172.21.15.98:0/2060746397 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff70ffff640 1 --2- 172.21.15.98:0/2060746397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff710102310 0x7ff710199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff714f48640 1 --2- 172.21.15.98:0/2060746397 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff710107570 0x7ff7101a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff70f7fe640 1 -- 172.21.15.98:0/2060746397 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff710107570 msgr2=0x7ff7101a13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff70f7fe640 1 --2- 172.21.15.98:0/2060746397 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff710107570 0x7ff7101a13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff70f7fe640 1 -- 172.21.15.98:0/2060746397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff710102310 msgr2=0x7ff710199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff70f7fe640 1 --2- 172.21.15.98:0/2060746397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff710102310 0x7ff710199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff70f7fe640 1 -- 172.21.15.98:0/2060746397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7101a1a30 con 0x7ff710104c40 2024-05-09T08:48:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff70f7fe640 1 --2- 172.21.15.98:0/2060746397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff710104c40 0x7ff71019a350 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff70400d770 tx=0x7ff70400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.594+0000 7ff70d7fa640 1 -- 172.21.15.98:0/2060746397 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff704017070 con 0x7ff710104c40 2024-05-09T08:48:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.595+0000 7ff7169ab640 1 -- 172.21.15.98:0/2060746397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7101a1d20 con 0x7ff710104c40 2024-05-09T08:48:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.595+0000 7ff70d7fa640 1 -- 172.21.15.98:0/2060746397 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff704009c30 con 0x7ff710104c40 2024-05-09T08:48:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.595+0000 7ff70d7fa640 1 -- 172.21.15.98:0/2060746397 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70400f620 con 0x7ff710104c40 2024-05-09T08:48:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.595+0000 7ff7169ab640 1 -- 172.21.15.98:0/2060746397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7101a2210 con 0x7ff710104c40 2024-05-09T08:48:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.595+0000 7ff7169ab640 1 -- 172.21.15.98:0/2060746397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7101090c0 con 0x7ff710104c40 2024-05-09T08:48:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.597+0000 7ff70d7fa640 1 -- 172.21.15.98:0/2060746397 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff70401a070 con 0x7ff710104c40 2024-05-09T08:48:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.597+0000 7ff70d7fa640 1 --2- 172.21.15.98:0/2060746397 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6f80a2b90 0x7ff6f80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.597+0000 7ff70ffff640 1 --2- 172.21.15.98:0/2060746397 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6f80a2b90 0x7ff6f80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.598+0000 7ff70d7fa640 1 -- 172.21.15.98:0/2060746397 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff7040cf040 con 0x7ff710104c40 2024-05-09T08:48:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.598+0000 7ff70ffff640 1 --2- 172.21.15.98:0/2060746397 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6f80a2b90 0x7ff6f80a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff7000314e0 tx=0x7ff700000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.601+0000 7ff70d7fa640 1 -- 172.21.15.98:0/2060746397 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff704009de0 con 0x7ff710104c40 2024-05-09T08:48:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:30 smithi142 ceph-mon[34625]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 785 B/s wr, 2 op/s 2024-05-09T08:48:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.777+0000 7ff7169ab640 1 -- 172.21.15.98:0/2060746397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff71002d410 con 0x7ff710104c40 2024-05-09T08:48:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.777+0000 7ff70d7fa640 1 -- 172.21.15.98:0/2060746397 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7ff70409b710 con 0x7ff710104c40 2024-05-09T08:48:30.777 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-09T08:48:30.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.779+0000 7ff7169ab640 1 -- 172.21.15.98:0/2060746397 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6f80a2b90 msgr2=0x7ff6f80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:30.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.779+0000 7ff7169ab640 1 --2- 172.21.15.98:0/2060746397 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6f80a2b90 0x7ff6f80a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff7000314e0 tx=0x7ff700000bc0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.779+0000 7ff7169ab640 1 -- 172.21.15.98:0/2060746397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff710104c40 msgr2=0x7ff71019a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:30.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.779+0000 7ff7169ab640 1 --2- 172.21.15.98:0/2060746397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff710104c40 0x7ff71019a350 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff70400d770 tx=0x7ff70400dc30 comp rx=0 tx=0).stop 2024-05-09T08:48:30.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.779+0000 7ff7169ab640 1 -- 172.21.15.98:0/2060746397 shutdown_connections 2024-05-09T08:48:30.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.779+0000 7ff7169ab640 1 --2- 172.21.15.98:0/2060746397 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff710107570 0x7ff7101a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.779+0000 7ff7169ab640 1 --2- 172.21.15.98:0/2060746397 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff6f80a2b90 0x7ff6f80a5050 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.779+0000 7ff7169ab640 1 --2- 172.21.15.98:0/2060746397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff710104c40 0x7ff71019a350 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.779+0000 7ff7169ab640 1 --2- 172.21.15.98:0/2060746397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff710102310 0x7ff710199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.779+0000 7ff7169ab640 1 -- 172.21.15.98:0/2060746397 >> 172.21.15.98:0/2060746397 conn(0x7ff7100fbd20 msgr2=0x7ff710101b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:30.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.779+0000 7ff7169ab640 1 -- 172.21.15.98:0/2060746397 shutdown_connections 2024-05-09T08:48:30.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.779+0000 7ff7169ab640 1 -- 172.21.15.98:0/2060746397 wait complete. 2024-05-09T08:48:30.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:48:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.895+0000 7fc2207d5640 1 -- 172.21.15.98:0/3958140654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc218104ca0 msgr2=0x7fc218107090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.895+0000 7fc2207d5640 1 --2- 172.21.15.98:0/3958140654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc218104ca0 0x7fc218107090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc208009900 tx=0x7fc20802a410 comp rx=0 tx=0).stop 2024-05-09T08:48:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.895+0000 7fc2207d5640 1 -- 172.21.15.98:0/3958140654 shutdown_connections 2024-05-09T08:48:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.895+0000 7fc2207d5640 1 --2- 172.21.15.98:0/3958140654 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2181075d0 0x7fc2181099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.895+0000 7fc2207d5640 1 --2- 172.21.15.98:0/3958140654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc218104ca0 0x7fc218107090 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.895+0000 7fc2207d5640 1 --2- 172.21.15.98:0/3958140654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc218102370 0x7fc218104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.895+0000 7fc2207d5640 1 -- 172.21.15.98:0/3958140654 >> 172.21.15.98:0/3958140654 conn(0x7fc2180fbd20 msgr2=0x7fc2180fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.895+0000 7fc2207d5640 1 -- 172.21.15.98:0/3958140654 shutdown_connections 2024-05-09T08:48:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.895+0000 7fc2207d5640 1 -- 172.21.15.98:0/3958140654 wait complete. 2024-05-09T08:48:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc2207d5640 1 Processor -- start 2024-05-09T08:48:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc2207d5640 1 -- start start 2024-05-09T08:48:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc2207d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc218102370 0x7fc218197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc2207d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc218104ca0 0x7fc218198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc2207d5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2181075d0 0x7fc21819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc2207d5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc21810d510 con 0x7fc218102370 2024-05-09T08:48:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc2207d5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc21810d390 con 0x7fc218104ca0 2024-05-09T08:48:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc2207d5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc21810d690 con 0x7fc2181075d0 2024-05-09T08:48:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc21e571640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc218102370 0x7fc218197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc21e571640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc218102370 0x7fc218197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36676/0 (socket says 172.21.15.98:36676) 2024-05-09T08:48:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc21e571640 1 -- 172.21.15.98:0/4015133599 learned_addr learned my addr 172.21.15.98:0/4015133599 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc21dd70640 1 --2- 172.21.15.98:0/4015133599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc218104ca0 0x7fc218198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc21ed72640 1 --2- 172.21.15.98:0/4015133599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2181075d0 0x7fc21819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc21e571640 1 -- 172.21.15.98:0/4015133599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2181075d0 msgr2=0x7fc21819f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc21e571640 1 --2- 172.21.15.98:0/4015133599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2181075d0 0x7fc21819f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc21e571640 1 -- 172.21.15.98:0/4015133599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc218104ca0 msgr2=0x7fc218198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:30.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc21e571640 1 --2- 172.21.15.98:0/4015133599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc218104ca0 0x7fc218198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:30.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc21e571640 1 -- 172.21.15.98:0/4015133599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc21819f800 con 0x7fc218102370 2024-05-09T08:48:30.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.896+0000 7fc21e571640 1 --2- 172.21.15.98:0/4015133599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc218102370 0x7fc218197be0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc21400d950 tx=0x7fc21400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:30.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.897+0000 7fc20f7fe640 1 -- 172.21.15.98:0/4015133599 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc214017030 con 0x7fc218102370 2024-05-09T08:48:30.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.897+0000 7fc20f7fe640 1 -- 172.21.15.98:0/4015133599 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc214004020 con 0x7fc218102370 2024-05-09T08:48:30.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.897+0000 7fc2207d5640 1 -- 172.21.15.98:0/4015133599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc21819faf0 con 0x7fc218102370 2024-05-09T08:48:30.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.897+0000 7fc20f7fe640 1 -- 172.21.15.98:0/4015133599 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2140047a0 con 0x7fc218102370 2024-05-09T08:48:30.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.897+0000 7fc2207d5640 1 -- 172.21.15.98:0/4015133599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2181a0060 con 0x7fc218102370 2024-05-09T08:48:30.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.898+0000 7fc2207d5640 1 -- 172.21.15.98:0/4015133599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2180fd810 con 0x7fc218102370 2024-05-09T08:48:30.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.899+0000 7fc20f7fe640 1 -- 172.21.15.98:0/4015133599 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc2140041d0 con 0x7fc218102370 2024-05-09T08:48:30.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.899+0000 7fc20f7fe640 1 --2- 172.21.15.98:0/4015133599 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc2040a2c10 0x7fc2040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:30.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.900+0000 7fc21dd70640 1 --2- 172.21.15.98:0/4015133599 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc2040a2c10 0x7fc2040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:30.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.900+0000 7fc20f7fe640 1 -- 172.21.15.98:0/4015133599 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc2140cf230 con 0x7fc218102370 2024-05-09T08:48:30.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.900+0000 7fc21dd70640 1 --2- 172.21.15.98:0/4015133599 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc2040a2c10 0x7fc2040a50d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc208005ab0 tx=0x7fc208000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:30.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:30.903+0000 7fc20f7fe640 1 -- 172.21.15.98:0/4015133599 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc21409c070 con 0x7fc218102370 2024-05-09T08:48:31.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.076+0000 7fc2207d5640 1 -- 172.21.15.98:0/4015133599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc21802d470 con 0x7fc218102370 2024-05-09T08:48:31.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.077+0000 7fc20f7fe640 1 -- 172.21.15.98:0/4015133599 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fc21409b970 con 0x7fc218102370 2024-05-09T08:48:31.076 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:48:31.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.078+0000 7fc2207d5640 1 -- 172.21.15.98:0/4015133599 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc2040a2c10 msgr2=0x7fc2040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.078+0000 7fc2207d5640 1 --2- 172.21.15.98:0/4015133599 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc2040a2c10 0x7fc2040a50d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc208005ab0 tx=0x7fc208000bc0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.079+0000 7fc2207d5640 1 -- 172.21.15.98:0/4015133599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc218102370 msgr2=0x7fc218197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.079+0000 7fc2207d5640 1 --2- 172.21.15.98:0/4015133599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc218102370 0x7fc218197be0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc21400d950 tx=0x7fc21400de10 comp rx=0 tx=0).stop 2024-05-09T08:48:31.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.079+0000 7fc2207d5640 1 -- 172.21.15.98:0/4015133599 shutdown_connections 2024-05-09T08:48:31.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.079+0000 7fc2207d5640 1 --2- 172.21.15.98:0/4015133599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2181075d0 0x7fc21819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.079+0000 7fc2207d5640 1 --2- 172.21.15.98:0/4015133599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc218104ca0 0x7fc218198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.079+0000 7fc2207d5640 1 --2- 172.21.15.98:0/4015133599 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc2040a2c10 0x7fc2040a50d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.079+0000 7fc2207d5640 1 --2- 172.21.15.98:0/4015133599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc218102370 0x7fc218197be0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.079+0000 7fc2207d5640 1 -- 172.21.15.98:0/4015133599 >> 172.21.15.98:0/4015133599 conn(0x7fc2180fbd20 msgr2=0x7fc218102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:31.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.079+0000 7fc2207d5640 1 -- 172.21.15.98:0/4015133599 shutdown_connections 2024-05-09T08:48:31.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.079+0000 7fc2207d5640 1 -- 172.21.15.98:0/4015133599 wait complete. 2024-05-09T08:48:31.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:48:31.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.195+0000 7f2dd60d4640 1 -- 172.21.15.98:0/2535424432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dd01019e0 msgr2=0x7f2dd0101e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.195+0000 7f2dd60d4640 1 --2- 172.21.15.98:0/2535424432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dd01019e0 0x7f2dd0101e60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009900 tx=0x7f2db802a410 comp rx=0 tx=0).stop 2024-05-09T08:48:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.195+0000 7f2dd60d4640 1 -- 172.21.15.98:0/2535424432 shutdown_connections 2024-05-09T08:48:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.195+0000 7f2dd60d4640 1 --2- 172.21.15.98:0/2535424432 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2dd01023a0 0x7f2dd0108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.195+0000 7f2dd60d4640 1 --2- 172.21.15.98:0/2535424432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dd01019e0 0x7f2dd0101e60 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.195+0000 7f2dd60d4640 1 --2- 172.21.15.98:0/2535424432 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dd01007f0 0x7f2dd0100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.195+0000 7f2dd60d4640 1 -- 172.21.15.98:0/2535424432 >> 172.21.15.98:0/2535424432 conn(0x7f2dd00fbd80 msgr2=0x7f2dd00fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.195+0000 7f2dd60d4640 1 -- 172.21.15.98:0/2535424432 shutdown_connections 2024-05-09T08:48:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.195+0000 7f2dd60d4640 1 -- 172.21.15.98:0/2535424432 wait complete. 2024-05-09T08:48:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.195+0000 7f2dd60d4640 1 Processor -- start 2024-05-09T08:48:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.195+0000 7f2dd60d4640 1 -- start start 2024-05-09T08:48:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dd60d4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dd01007f0 0x7f2dd0197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dd60d4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2dd01019e0 0x7f2dd0198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dd60d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dd01023a0 0x7f2dd019f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dd60d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd010c400 con 0x7f2dd01023a0 2024-05-09T08:48:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dd60d4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd010c280 con 0x7f2dd01007f0 2024-05-09T08:48:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dd60d4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd010c580 con 0x7f2dd01019e0 2024-05-09T08:48:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dcffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dd01023a0 0x7f2dd019f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dcf7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dd01007f0 0x7f2dd0197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dcffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dd01023a0 0x7f2dd019f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36686/0 (socket says 172.21.15.98:36686) 2024-05-09T08:48:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dcffff640 1 -- 172.21.15.98:0/3287498041 learned_addr learned my addr 172.21.15.98:0/3287498041 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dc6ffd640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2dd01019e0 0x7f2dd0198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dcffff640 1 -- 172.21.15.98:0/3287498041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2dd01019e0 msgr2=0x7f2dd0198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dcffff640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2dd01019e0 0x7f2dd0198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dcffff640 1 -- 172.21.15.98:0/3287498041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dd01007f0 msgr2=0x7f2dd0197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dcffff640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dd01007f0 0x7f2dd0197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dcffff640 1 -- 172.21.15.98:0/3287498041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd019f7b0 con 0x7f2dd01023a0 2024-05-09T08:48:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dc6ffd640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2dd01019e0 0x7f2dd0198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dcf7fe640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dd01007f0 0x7f2dd0197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:48:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dcffff640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dd01023a0 0x7f2dd019f140 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2dc800eaa0 tx=0x7f2dc800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dcd7fa640 1 -- 172.21.15.98:0/3287498041 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc800cdb0 con 0x7f2dd01023a0 2024-05-09T08:48:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dd60d4640 1 -- 172.21.15.98:0/3287498041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd019faa0 con 0x7f2dd01023a0 2024-05-09T08:48:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dcd7fa640 1 -- 172.21.15.98:0/3287498041 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dc8005020 con 0x7f2dd01023a0 2024-05-09T08:48:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dcd7fa640 1 -- 172.21.15.98:0/3287498041 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc80116d0 con 0x7f2dd01023a0 2024-05-09T08:48:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.196+0000 7f2dd60d4640 1 -- 172.21.15.98:0/3287498041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd01a0010 con 0x7f2dd01023a0 2024-05-09T08:48:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.197+0000 7f2dd60d4640 1 -- 172.21.15.98:0/3287498041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dd01a02a0 con 0x7f2dd01023a0 2024-05-09T08:48:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.198+0000 7f2dcd7fa640 1 -- 172.21.15.98:0/3287498041 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2dc80054d0 con 0x7f2dd01023a0 2024-05-09T08:48:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.199+0000 7f2dcd7fa640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2dbc0a2b90 0x7f2dbc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.199+0000 7f2dcf7fe640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2dbc0a2b90 0x7f2dbc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.199+0000 7f2dcd7fa640 1 -- 172.21.15.98:0/3287498041 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f2dc80cf4a0 con 0x7f2dd01023a0 2024-05-09T08:48:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.200+0000 7f2dcf7fe640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2dbc0a2b90 0x7f2dbc0a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0101840 tx=0x7f2dc0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.204+0000 7f2dcd7fa640 1 -- 172.21.15.98:0/3287498041 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2dc8016030 con 0x7f2dd01023a0 2024-05-09T08:48:31.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.382+0000 7f2dd60d4640 1 -- 172.21.15.98:0/3287498041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2dd01a0470 con 0x7f2dd01023a0 2024-05-09T08:48:31.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.382+0000 7f2dcd7fa640 1 -- 172.21.15.98:0/3287498041 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f2dc809c1c0 con 0x7f2dd01023a0 2024-05-09T08:48:31.384 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-09T08:48:31.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.384+0000 7f2dd60d4640 1 -- 172.21.15.98:0/3287498041 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2dbc0a2b90 msgr2=0x7f2dbc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.384+0000 7f2dd60d4640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2dbc0a2b90 0x7f2dbc0a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0101840 tx=0x7f2dc0005f90 comp rx=0 tx=0).stop 2024-05-09T08:48:31.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.384+0000 7f2dd60d4640 1 -- 172.21.15.98:0/3287498041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dd01023a0 msgr2=0x7f2dd019f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.384+0000 7f2dd60d4640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dd01023a0 0x7f2dd019f140 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2dc800eaa0 tx=0x7f2dc800ef60 comp rx=0 tx=0).stop 2024-05-09T08:48:31.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.384+0000 7f2dd60d4640 1 -- 172.21.15.98:0/3287498041 shutdown_connections 2024-05-09T08:48:31.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.384+0000 7f2dd60d4640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2dbc0a2b90 0x7f2dbc0a5050 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.384+0000 7f2dd60d4640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dd01023a0 0x7f2dd019f140 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.384+0000 7f2dd60d4640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2dd01019e0 0x7f2dd0198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.384+0000 7f2dd60d4640 1 --2- 172.21.15.98:0/3287498041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dd01007f0 0x7f2dd0197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.384+0000 7f2dd60d4640 1 -- 172.21.15.98:0/3287498041 >> 172.21.15.98:0/3287498041 conn(0x7f2dd00fbd80 msgr2=0x7f2dd00fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:31.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.385+0000 7f2dd60d4640 1 -- 172.21.15.98:0/3287498041 shutdown_connections 2024-05-09T08:48:31.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.385+0000 7f2dd60d4640 1 -- 172.21.15.98:0/3287498041 wait complete. 2024-05-09T08:48:31.394 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:31 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1030144050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:31.394 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:31 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2060746397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:48:31.394 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:31 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4015133599' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:48:31.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:48:31.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.500+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/2946252460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd58100770 msgr2=0x7fcd58100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.500+0000 7fcd5e9c1640 1 --2- 172.21.15.98:0/2946252460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd58100770 0x7fcd58100b90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009900 tx=0x7fcd4802a410 comp rx=0 tx=0).stop 2024-05-09T08:48:31.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.500+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/2946252460 shutdown_connections 2024-05-09T08:48:31.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.500+0000 7fcd5e9c1640 1 --2- 172.21.15.98:0/2946252460 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd58102320 0x7fcd58108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.500+0000 7fcd5e9c1640 1 --2- 172.21.15.98:0/2946252460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd58101960 0x7fcd58101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.501+0000 7fcd5e9c1640 1 --2- 172.21.15.98:0/2946252460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd58100770 0x7fcd58100b90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.501+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/2946252460 >> 172.21.15.98:0/2946252460 conn(0x7fcd580fbd20 msgr2=0x7fcd580fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:31.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.501+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/2946252460 shutdown_connections 2024-05-09T08:48:31.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.501+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/2946252460 wait complete. 2024-05-09T08:48:31.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.501+0000 7fcd5e9c1640 1 Processor -- start 2024-05-09T08:48:31.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.501+0000 7fcd5e9c1640 1 -- start start 2024-05-09T08:48:31.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.501+0000 7fcd5e9c1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd58100770 0x7fcd5806a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:31.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.501+0000 7fcd5e9c1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd58101960 0x7fcd5819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:31.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.501+0000 7fcd5e9c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd58102320 0x7fcd5819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:31.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.501+0000 7fcd5e9c1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5810c3c0 con 0x7fcd58102320 2024-05-09T08:48:31.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd5e9c1640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5810c240 con 0x7fcd58100770 2024-05-09T08:48:31.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd5e9c1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5810c540 con 0x7fcd58101960 2024-05-09T08:48:31.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd5cf5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd58102320 0x7fcd5819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:31.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd5cf5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd58102320 0x7fcd5819eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36706/0 (socket says 172.21.15.98:36706) 2024-05-09T08:48:31.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd5cf5e640 1 -- 172.21.15.98:0/561280646 learned_addr learned my addr 172.21.15.98:0/561280646 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:31.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd57fff640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd58100770 0x7fcd5806a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:31.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd577fe640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd58101960 0x7fcd5819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:31.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd5cf5e640 1 -- 172.21.15.98:0/561280646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd58101960 msgr2=0x7fcd5819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd5cf5e640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd58101960 0x7fcd5819c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd5cf5e640 1 -- 172.21.15.98:0/561280646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd58100770 msgr2=0x7fcd5806a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd5cf5e640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd58100770 0x7fcd5806a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd5cf5e640 1 -- 172.21.15.98:0/561280646 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5819f7f0 con 0x7fcd58102320 2024-05-09T08:48:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd57fff640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd58100770 0x7fcd5806a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:48:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd5cf5e640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd58102320 0x7fcd5819eee0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcd5000cce0 tx=0x7fcd5000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd557fa640 1 -- 172.21.15.98:0/561280646 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5000f450 con 0x7fcd58102320 2024-05-09T08:48:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd557fa640 1 -- 172.21.15.98:0/561280646 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd5000fad0 con 0x7fcd58102320 2024-05-09T08:48:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/561280646 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd5819fa80 con 0x7fcd58102320 2024-05-09T08:48:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd557fa640 1 -- 172.21.15.98:0/561280646 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd50018c10 con 0x7fcd58102320 2024-05-09T08:48:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.502+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/561280646 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd5819ff80 con 0x7fcd58102320 2024-05-09T08:48:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.503+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/561280646 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd58107b10 con 0x7fcd58102320 2024-05-09T08:48:31.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.504+0000 7fcd557fa640 1 -- 172.21.15.98:0/561280646 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcd5000f5f0 con 0x7fcd58102320 2024-05-09T08:48:31.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.505+0000 7fcd557fa640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcd440a2b90 0x7fcd440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:31.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.505+0000 7fcd57fff640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcd440a2b90 0x7fcd440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:31.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.505+0000 7fcd557fa640 1 -- 172.21.15.98:0/561280646 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fcd500cf970 con 0x7fcd58102320 2024-05-09T08:48:31.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.505+0000 7fcd57fff640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcd440a2b90 0x7fcd440a5050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcd48005ab0 tx=0x7fcd48000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:31.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.509+0000 7fcd557fa640 1 -- 172.21.15.98:0/561280646 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcd50014030 con 0x7fcd58102320 2024-05-09T08:48:31.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:31 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1030144050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:31.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:31 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2060746397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:48:31.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:31 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/4015133599' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:48:31.681 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:31 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1030144050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:31.681 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:31 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2060746397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:48:31.681 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:31 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/4015133599' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:48:31.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.685+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/561280646 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcd58062430 con 0x7fcd58102320 2024-05-09T08:48:31.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.685+0000 7fcd557fa640 1 -- 172.21.15.98:0/561280646 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7fcd5009c690 con 0x7fcd58102320 2024-05-09T08:48:31.684 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:48:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.687+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/561280646 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcd440a2b90 msgr2=0x7fcd440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.687+0000 7fcd5e9c1640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcd440a2b90 0x7fcd440a5050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcd48005ab0 tx=0x7fcd48000bc0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.687+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/561280646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd58102320 msgr2=0x7fcd5819eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.687+0000 7fcd5e9c1640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd58102320 0x7fcd5819eee0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcd5000cce0 tx=0x7fcd5000a620 comp rx=0 tx=0).stop 2024-05-09T08:48:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.687+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/561280646 shutdown_connections 2024-05-09T08:48:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.687+0000 7fcd5e9c1640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcd440a2b90 0x7fcd440a5050 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.687+0000 7fcd5e9c1640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd58102320 0x7fcd5819eee0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.687+0000 7fcd5e9c1640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd58101960 0x7fcd5819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.687+0000 7fcd5e9c1640 1 --2- 172.21.15.98:0/561280646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd58100770 0x7fcd5806a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.687+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/561280646 >> 172.21.15.98:0/561280646 conn(0x7fcd580fbd20 msgr2=0x7fcd58104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:31.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.687+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/561280646 shutdown_connections 2024-05-09T08:48:31.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.687+0000 7fcd5e9c1640 1 -- 172.21.15.98:0/561280646 wait complete. 2024-05-09T08:48:31.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T08:48:31.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.800+0000 7f11330bd640 1 -- 172.21.15.98:0/2432875990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f112c101980 msgr2=0x7f112c101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.800+0000 7f11330bd640 1 --2- 172.21.15.98:0/2432875990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f112c101980 0x7f112c101e00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1118009900 tx=0x7f111802a410 comp rx=0 tx=0).stop 2024-05-09T08:48:31.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.800+0000 7f11330bd640 1 -- 172.21.15.98:0/2432875990 shutdown_connections 2024-05-09T08:48:31.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.800+0000 7f11330bd640 1 --2- 172.21.15.98:0/2432875990 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f112c102340 0x7f112c108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.800+0000 7f11330bd640 1 --2- 172.21.15.98:0/2432875990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f112c101980 0x7f112c101e00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.800+0000 7f11330bd640 1 --2- 172.21.15.98:0/2432875990 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f112c100790 0x7f112c100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.800+0000 7f11330bd640 1 -- 172.21.15.98:0/2432875990 >> 172.21.15.98:0/2432875990 conn(0x7f112c0fbd20 msgr2=0x7f112c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.800+0000 7f11330bd640 1 -- 172.21.15.98:0/2432875990 shutdown_connections 2024-05-09T08:48:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.800+0000 7f11330bd640 1 -- 172.21.15.98:0/2432875990 wait complete. 2024-05-09T08:48:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.800+0000 7f11330bd640 1 Processor -- start 2024-05-09T08:48:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.800+0000 7f11330bd640 1 -- start start 2024-05-09T08:48:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f11330bd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f112c100790 0x7f112c196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f11330bd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f112c101980 0x7f112c196fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f11330bd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f112c102340 0x7f112c19e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f11330bd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f112c10c470 con 0x7f112c102340 2024-05-09T08:48:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f11330bd640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f112c10c2f0 con 0x7f112c101980 2024-05-09T08:48:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f11330bd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f112c10c5f0 con 0x7f112c100790 2024-05-09T08:48:31.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f113165a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f112c102340 0x7f112c19e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f113165a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f112c102340 0x7f112c19e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36726/0 (socket says 172.21.15.98:36726) 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f113165a640 1 -- 172.21.15.98:0/2041034180 learned_addr learned my addr 172.21.15.98:0/2041034180 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f112bfff640 1 --2- 172.21.15.98:0/2041034180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f112c101980 0x7f112c196fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f1130e59640 1 --2- 172.21.15.98:0/2041034180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f112c100790 0x7f112c196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f113165a640 1 -- 172.21.15.98:0/2041034180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f112c100790 msgr2=0x7f112c196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f113165a640 1 --2- 172.21.15.98:0/2041034180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f112c100790 0x7f112c196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f113165a640 1 -- 172.21.15.98:0/2041034180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f112c101980 msgr2=0x7f112c196fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f113165a640 1 --2- 172.21.15.98:0/2041034180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f112c101980 0x7f112c196fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f113165a640 1 -- 172.21.15.98:0/2041034180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112c19e6a0 con 0x7f112c102340 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f113165a640 1 --2- 172.21.15.98:0/2041034180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f112c102340 0x7f112c19e030 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f112400db90 tx=0x7f112400dc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f1129ffb640 1 -- 172.21.15.98:0/2041034180 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112400e980 con 0x7f112c102340 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f11330bd640 1 -- 172.21.15.98:0/2041034180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f112c069e30 con 0x7f112c102340 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f1129ffb640 1 -- 172.21.15.98:0/2041034180 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1124009580 con 0x7f112c102340 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.801+0000 7f1129ffb640 1 -- 172.21.15.98:0/2041034180 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1124016650 con 0x7f112c102340 2024-05-09T08:48:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.802+0000 7f11330bd640 1 -- 172.21.15.98:0/2041034180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f112c06a420 con 0x7f112c102340 2024-05-09T08:48:31.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.803+0000 7f11330bd640 1 -- 172.21.15.98:0/2041034180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f112c068aa0 con 0x7f112c102340 2024-05-09T08:48:31.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.804+0000 7f1129ffb640 1 -- 172.21.15.98:0/2041034180 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1124009730 con 0x7f112c102340 2024-05-09T08:48:31.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.804+0000 7f1129ffb640 1 --2- 172.21.15.98:0/2041034180 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f111c0a2b90 0x7f111c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:31.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.805+0000 7f1130e59640 1 --2- 172.21.15.98:0/2041034180 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f111c0a2b90 0x7f111c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:31.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.805+0000 7f1129ffb640 1 -- 172.21.15.98:0/2041034180 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f11240d0860 con 0x7f112c102340 2024-05-09T08:48:31.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.805+0000 7f1130e59640 1 --2- 172.21.15.98:0/2041034180 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f111c0a2b90 0x7f111c0a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f112c1017e0 tx=0x7f1120005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:31.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.809+0000 7f1129ffb640 1 -- 172.21.15.98:0/2041034180 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f112409cf30 con 0x7f112c102340 2024-05-09T08:48:31.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.998+0000 7f11330bd640 1 -- 172.21.15.98:0/2041034180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f112c107b00 con 0x7f112c102340 2024-05-09T08:48:31.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:31.999+0000 7f1129ffb640 1 -- 172.21.15.98:0/2041034180 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+588 (secure 0 0 0) 0x7f112c107b00 con 0x7f112c102340 2024-05-09T08:48:31.998 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2163305674 2024-05-10T08:41:17.098271+0000 2024-05-09T08:48:31.998 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1385598801 2024-05-10T08:41:17.098271+0000 2024-05-09T08:48:31.999 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/498605085 2024-05-10T08:41:17.098271+0000 2024-05-09T08:48:31.999 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/1313924889 2024-05-10T08:40:50.793131+0000 2024-05-09T08:48:31.999 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/498605085 2024-05-10T08:41:17.098271+0000 2024-05-09T08:48:31.999 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/812399881 2024-05-10T08:40:50.793131+0000 2024-05-09T08:48:31.999 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2540720935 2024-05-10T08:40:50.793131+0000 2024-05-09T08:48:31.999 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/1313924889 2024-05-10T08:40:50.793131+0000 2024-05-09T08:48:31.999 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3487867282 2024-05-10T08:41:17.098271+0000 2024-05-09T08:48:31.999 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/645511714 2024-05-10T08:40:50.793131+0000 2024-05-09T08:48:31.999 INFO:teuthology.orchestra.run.smithi098.stderr:listed 10 entries 2024-05-09T08:48:32.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.001+0000 7f11330bd640 1 -- 172.21.15.98:0/2041034180 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f111c0a2b90 msgr2=0x7f111c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.001+0000 7f11330bd640 1 --2- 172.21.15.98:0/2041034180 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f111c0a2b90 0x7f111c0a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f112c1017e0 tx=0x7f1120005f90 comp rx=0 tx=0).stop 2024-05-09T08:48:32.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.001+0000 7f11330bd640 1 -- 172.21.15.98:0/2041034180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f112c102340 msgr2=0x7f112c19e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.001+0000 7f11330bd640 1 --2- 172.21.15.98:0/2041034180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f112c102340 0x7f112c19e030 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f112400db90 tx=0x7f112400dc80 comp rx=0 tx=0).stop 2024-05-09T08:48:32.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.001+0000 7f11330bd640 1 -- 172.21.15.98:0/2041034180 shutdown_connections 2024-05-09T08:48:32.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.001+0000 7f11330bd640 1 --2- 172.21.15.98:0/2041034180 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f111c0a2b90 0x7f111c0a5050 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.001+0000 7f11330bd640 1 --2- 172.21.15.98:0/2041034180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f112c102340 0x7f112c19e030 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.001+0000 7f11330bd640 1 --2- 172.21.15.98:0/2041034180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f112c101980 0x7f112c196fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.001+0000 7f11330bd640 1 --2- 172.21.15.98:0/2041034180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f112c100790 0x7f112c196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.001+0000 7f11330bd640 1 -- 172.21.15.98:0/2041034180 >> 172.21.15.98:0/2041034180 conn(0x7f112c0fbd20 msgr2=0x7f112c104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:32.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.001+0000 7f11330bd640 1 -- 172.21.15.98:0/2041034180 shutdown_connections 2024-05-09T08:48:32.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.001+0000 7f11330bd640 1 -- 172.21.15.98:0/2041034180 wait complete. 2024-05-09T08:48:32.011 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:48:32.012 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T08:48:32.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:48:32.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.169+0000 7f986a55a640 1 -- 172.21.15.98:0/1363364807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9864102340 msgr2=0x7f9864108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.169+0000 7f986a55a640 1 --2- 172.21.15.98:0/1363364807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9864102340 0x7f9864108bd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f985c009ca0 tx=0x7f985c02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.169+0000 7f986a55a640 1 -- 172.21.15.98:0/1363364807 shutdown_connections 2024-05-09T08:48:32.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.169+0000 7f986a55a640 1 --2- 172.21.15.98:0/1363364807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9864102340 0x7f9864108bd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.169+0000 7f986a55a640 1 --2- 172.21.15.98:0/1363364807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9864101980 0x7f9864101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.169+0000 7f986a55a640 1 --2- 172.21.15.98:0/1363364807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9864100790 0x7f9864100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.169+0000 7f986a55a640 1 -- 172.21.15.98:0/1363364807 >> 172.21.15.98:0/1363364807 conn(0x7f98640fbd20 msgr2=0x7f98640fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:32.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.169+0000 7f986a55a640 1 -- 172.21.15.98:0/1363364807 shutdown_connections 2024-05-09T08:48:32.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.169+0000 7f986a55a640 1 -- 172.21.15.98:0/1363364807 wait complete. 2024-05-09T08:48:32.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f986a55a640 1 Processor -- start 2024-05-09T08:48:32.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f986a55a640 1 -- start start 2024-05-09T08:48:32.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f986a55a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9864100790 0x7f9864197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:32.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f986a55a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9864101980 0x7f9864197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:32.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f986a55a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9864102340 0x7f986419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:32.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f986a55a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f986410d4f0 con 0x7f9864102340 2024-05-09T08:48:32.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f986a55a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f986410d370 con 0x7f9864101980 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f986a55a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f986410d670 con 0x7f9864100790 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f9868af7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9864102340 0x7f986419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f9868af7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9864102340 0x7f986419f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36730/0 (socket says 172.21.15.98:36730) 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f9868af7640 1 -- 172.21.15.98:0/4189624105 learned_addr learned my addr 172.21.15.98:0/4189624105 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f98637fe640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9864101980 0x7f9864197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f9863fff640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9864100790 0x7f9864197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f9868af7640 1 -- 172.21.15.98:0/4189624105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9864100790 msgr2=0x7f9864197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f9868af7640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9864100790 0x7f9864197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f9868af7640 1 -- 172.21.15.98:0/4189624105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9864101980 msgr2=0x7f9864197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f9868af7640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9864101980 0x7f9864197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f9868af7640 1 -- 172.21.15.98:0/4189624105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f986419f6c0 con 0x7f9864102340 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.170+0000 7f98637fe640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9864101980 0x7f9864197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.171+0000 7f9868af7640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9864102340 0x7f986419f050 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f985c031ae0 tx=0x7f985c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.171+0000 7f98617fa640 1 -- 172.21.15.98:0/4189624105 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c003c30 con 0x7f9864102340 2024-05-09T08:48:32.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.171+0000 7f986a55a640 1 -- 172.21.15.98:0/4189624105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f986419f950 con 0x7f9864102340 2024-05-09T08:48:32.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.171+0000 7f98617fa640 1 -- 172.21.15.98:0/4189624105 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f985c003dd0 con 0x7f9864102340 2024-05-09T08:48:32.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.171+0000 7f98617fa640 1 -- 172.21.15.98:0/4189624105 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c02fcb0 con 0x7f9864102340 2024-05-09T08:48:32.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.171+0000 7f986a55a640 1 -- 172.21.15.98:0/4189624105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f986419fea0 con 0x7f9864102340 2024-05-09T08:48:32.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.172+0000 7f986a55a640 1 -- 172.21.15.98:0/4189624105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9864107a50 con 0x7f9864102340 2024-05-09T08:48:32.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.173+0000 7f98617fa640 1 -- 172.21.15.98:0/4189624105 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f985c004050 con 0x7f9864102340 2024-05-09T08:48:32.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.174+0000 7f98617fa640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f984c0a2b90 0x7f984c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:32.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.174+0000 7f9863fff640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f984c0a2b90 0x7f984c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:32.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.174+0000 7f98617fa640 1 -- 172.21.15.98:0/4189624105 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f985c0f0580 con 0x7f9864102340 2024-05-09T08:48:32.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.174+0000 7f9863fff640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f984c0a2b90 0x7f984c0a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f98540058c0 tx=0x7f9854005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:32.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.178+0000 7f98617fa640 1 -- 172.21.15.98:0/4189624105 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f985c0bd3c0 con 0x7f9864102340 2024-05-09T08:48:32.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:32 smithi098 ceph-mon[30092]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 580 B/s wr, 1 op/s 2024-05-09T08:48:32.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:32 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3287498041' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:48:32.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:32 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/561280646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:48:32.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:32 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2041034180' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:48:32.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.359+0000 7f986a55a640 1 -- 172.21.15.98:0/4189624105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9864062430 con 0x7f9864102340 2024-05-09T08:48:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.361+0000 7f98617fa640 1 -- 172.21.15.98:0/4189624105 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24150 (secure 0 0 0) 0x7f985c0bccc0 con 0x7f9864102340 2024-05-09T08:48:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:32.360 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":90,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:48:22.719729+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:48:32.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.363+0000 7f986a55a640 1 -- 172.21.15.98:0/4189624105 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f984c0a2b90 msgr2=0x7f984c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.363+0000 7f986a55a640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f984c0a2b90 0x7f984c0a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f98540058c0 tx=0x7f9854005850 comp rx=0 tx=0).stop 2024-05-09T08:48:32.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.363+0000 7f986a55a640 1 -- 172.21.15.98:0/4189624105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9864102340 msgr2=0x7f986419f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.363+0000 7f986a55a640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9864102340 0x7f986419f050 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f985c031ae0 tx=0x7f985c031ee0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.363+0000 7f986a55a640 1 -- 172.21.15.98:0/4189624105 shutdown_connections 2024-05-09T08:48:32.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.363+0000 7f986a55a640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f984c0a2b90 0x7f984c0a5050 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.363+0000 7f986a55a640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9864102340 0x7f986419f050 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.363+0000 7f986a55a640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9864101980 0x7f9864197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.363+0000 7f986a55a640 1 --2- 172.21.15.98:0/4189624105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9864100790 0x7f9864197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.363+0000 7f986a55a640 1 -- 172.21.15.98:0/4189624105 >> 172.21.15.98:0/4189624105 conn(0x7f98640fbd20 msgr2=0x7f9864104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:32.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.363+0000 7f986a55a640 1 -- 172.21.15.98:0/4189624105 shutdown_connections 2024-05-09T08:48:32.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.363+0000 7f986a55a640 1 -- 172.21.15.98:0/4189624105 wait complete. 2024-05-09T08:48:32.374 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:43:39.424443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:48:15.866450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:48:17.122162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:48:20.591475+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:48:32.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:48:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.480+0000 7fbe6fe93640 1 -- 172.21.15.98:0/1121509767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe68100770 msgr2=0x7fbe68100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.480+0000 7fbe6fe93640 1 --2- 172.21.15.98:0/1121509767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe68100770 0x7fbe68100b90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009900 tx=0x7fbe6402a410 comp rx=0 tx=0).stop 2024-05-09T08:48:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.480+0000 7fbe6fe93640 1 -- 172.21.15.98:0/1121509767 shutdown_connections 2024-05-09T08:48:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.480+0000 7fbe6fe93640 1 --2- 172.21.15.98:0/1121509767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe68102320 0x7fbe68108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.480+0000 7fbe6fe93640 1 --2- 172.21.15.98:0/1121509767 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe68101960 0x7fbe68101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.480+0000 7fbe6fe93640 1 --2- 172.21.15.98:0/1121509767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe68100770 0x7fbe68100b90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.480+0000 7fbe6fe93640 1 -- 172.21.15.98:0/1121509767 >> 172.21.15.98:0/1121509767 conn(0x7fbe680fbce0 msgr2=0x7fbe680fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.480+0000 7fbe6fe93640 1 -- 172.21.15.98:0/1121509767 shutdown_connections 2024-05-09T08:48:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.480+0000 7fbe6fe93640 1 -- 172.21.15.98:0/1121509767 wait complete. 2024-05-09T08:48:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.480+0000 7fbe6fe93640 1 Processor -- start 2024-05-09T08:48:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.480+0000 7fbe6fe93640 1 -- start start 2024-05-09T08:48:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.480+0000 7fbe6fe93640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe68100770 0x7fbe68197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6fe93640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe68101960 0x7fbe681981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6fe93640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe68102320 0x7fbe6819f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6fe93640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6810c440 con 0x7fbe68102320 2024-05-09T08:48:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6fe93640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6810c2c0 con 0x7fbe68101960 2024-05-09T08:48:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6fe93640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6810c5c0 con 0x7fbe68100770 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6e430640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe68102320 0x7fbe6819f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6e430640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe68102320 0x7fbe6819f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36762/0 (socket says 172.21.15.98:36762) 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6e430640 1 -- 172.21.15.98:0/178898971 learned_addr learned my addr 172.21.15.98:0/178898971 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6dc2f640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe68100770 0x7fbe68197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6e430640 1 -- 172.21.15.98:0/178898971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe68100770 msgr2=0x7fbe68197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6d42e640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe68101960 0x7fbe681981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6e430640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe68100770 0x7fbe68197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6e430640 1 -- 172.21.15.98:0/178898971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe68101960 msgr2=0x7fbe681981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6e430640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe68101960 0x7fbe681981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6e430640 1 -- 172.21.15.98:0/178898971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe6819f8d0 con 0x7fbe68102320 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6dc2f640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe68100770 0x7fbe68197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6e430640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe68102320 0x7fbe6819f230 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000b970 tx=0x7fbe6000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe5effd640 1 -- 172.21.15.98:0/178898971 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60019070 con 0x7fbe68102320 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe5effd640 1 -- 172.21.15.98:0/178898971 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe6000cb40 con 0x7fbe68102320 2024-05-09T08:48:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe5effd640 1 -- 172.21.15.98:0/178898971 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60007650 con 0x7fbe68102320 2024-05-09T08:48:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6fe93640 1 -- 172.21.15.98:0/178898971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe6819fbc0 con 0x7fbe68102320 2024-05-09T08:48:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.481+0000 7fbe6fe93640 1 -- 172.21.15.98:0/178898971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe681a0110 con 0x7fbe68102320 2024-05-09T08:48:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.482+0000 7fbe6fe93640 1 -- 172.21.15.98:0/178898971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe681a0340 con 0x7fbe68102320 2024-05-09T08:48:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.483+0000 7fbe5effd640 1 -- 172.21.15.98:0/178898971 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbe6001c070 con 0x7fbe68102320 2024-05-09T08:48:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.484+0000 7fbe5effd640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbe540a2b90 0x7fbe540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.484+0000 7fbe6dc2f640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbe540a2b90 0x7fbe540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.484+0000 7fbe5effd640 1 -- 172.21.15.98:0/178898971 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fbe600ce8c0 con 0x7fbe68102320 2024-05-09T08:48:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.484+0000 7fbe6dc2f640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbe540a2b90 0x7fbe540a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbe64000c00 tx=0x7fbe64031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:32.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.488+0000 7fbe5effd640 1 -- 172.21.15.98:0/178898971 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbe6009b5e0 con 0x7fbe68102320 2024-05-09T08:48:32.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:32 smithi142 ceph-mon[34625]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 580 B/s wr, 1 op/s 2024-05-09T08:48:32.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:32 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3287498041' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:48:32.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:32 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/561280646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:48:32.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:32 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2041034180' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:48:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.664+0000 7fbe6fe93640 1 -- 172.21.15.98:0/178898971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbe681a0510 con 0x7fbe68102320 2024-05-09T08:48:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.665+0000 7fbe5effd640 1 -- 172.21.15.98:0/178898971 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fbe6009aee0 con 0x7fbe68102320 2024-05-09T08:48:32.664 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-09T08:48:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.667+0000 7fbe6fe93640 1 -- 172.21.15.98:0/178898971 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbe540a2b90 msgr2=0x7fbe540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.667+0000 7fbe6fe93640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbe540a2b90 0x7fbe540a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbe64000c00 tx=0x7fbe64031400 comp rx=0 tx=0).stop 2024-05-09T08:48:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.667+0000 7fbe6fe93640 1 -- 172.21.15.98:0/178898971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe68102320 msgr2=0x7fbe6819f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.667+0000 7fbe6fe93640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe68102320 0x7fbe6819f230 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000b970 tx=0x7fbe6000be30 comp rx=0 tx=0).stop 2024-05-09T08:48:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.667+0000 7fbe6fe93640 1 -- 172.21.15.98:0/178898971 shutdown_connections 2024-05-09T08:48:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.667+0000 7fbe6fe93640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbe540a2b90 0x7fbe540a5050 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.667+0000 7fbe6fe93640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe68102320 0x7fbe6819f230 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.667+0000 7fbe6fe93640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe68101960 0x7fbe681981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.667+0000 7fbe6fe93640 1 --2- 172.21.15.98:0/178898971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe68100770 0x7fbe68197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.667+0000 7fbe6fe93640 1 -- 172.21.15.98:0/178898971 >> 172.21.15.98:0/178898971 conn(0x7fbe680fbce0 msgr2=0x7fbe680fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.667+0000 7fbe6fe93640 1 -- 172.21.15.98:0/178898971 shutdown_connections 2024-05-09T08:48:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.667+0000 7fbe6fe93640 1 -- 172.21.15.98:0/178898971 wait complete. 2024-05-09T08:48:32.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:48:32.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:32 smithi105 ceph-mon[34498]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 580 B/s wr, 1 op/s 2024-05-09T08:48:32.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:32 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3287498041' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:48:32.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:32 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/561280646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:48:32.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:32 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2041034180' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:48:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.784+0000 7f3065963640 1 -- 172.21.15.98:0/983359630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3060102320 msgr2=0x7f3060108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.784+0000 7f3065963640 1 --2- 172.21.15.98:0/983359630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3060102320 0x7f3060108bb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3058009ca0 tx=0x7f305802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.784+0000 7f3065963640 1 -- 172.21.15.98:0/983359630 shutdown_connections 2024-05-09T08:48:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.784+0000 7f3065963640 1 --2- 172.21.15.98:0/983359630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3060102320 0x7f3060108bb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.784+0000 7f3065963640 1 --2- 172.21.15.98:0/983359630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3060101960 0x7f3060101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.784+0000 7f3065963640 1 --2- 172.21.15.98:0/983359630 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3060100770 0x7f3060100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.784+0000 7f3065963640 1 -- 172.21.15.98:0/983359630 >> 172.21.15.98:0/983359630 conn(0x7f30600fbce0 msgr2=0x7f30600fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.784+0000 7f3065963640 1 -- 172.21.15.98:0/983359630 shutdown_connections 2024-05-09T08:48:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.784+0000 7f3065963640 1 -- 172.21.15.98:0/983359630 wait complete. 2024-05-09T08:48:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.785+0000 7f3065963640 1 Processor -- start 2024-05-09T08:48:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.785+0000 7f3065963640 1 -- start start 2024-05-09T08:48:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.785+0000 7f3065963640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3060100770 0x7f306006a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.785+0000 7f3065963640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3060101960 0x7f306019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.785+0000 7f3065963640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3060102320 0x7f306019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.785+0000 7f3065963640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f306010c3f0 con 0x7f3060100770 2024-05-09T08:48:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.785+0000 7f3065963640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f306010c270 con 0x7f3060101960 2024-05-09T08:48:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.785+0000 7f3065963640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f306010c570 con 0x7f3060102320 2024-05-09T08:48:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.785+0000 7f305effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3060100770 0x7f306006a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.785+0000 7f305effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3060100770 0x7f306006a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36780/0 (socket says 172.21.15.98:36780) 2024-05-09T08:48:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.785+0000 7f305effd640 1 -- 172.21.15.98:0/556822674 learned_addr learned my addr 172.21.15.98:0/556822674 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.785+0000 7f305e7fc640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3060101960 0x7f306019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.785+0000 7f305f7fe640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3060102320 0x7f306019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.786+0000 7f305effd640 1 -- 172.21.15.98:0/556822674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3060102320 msgr2=0x7f306019eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.786+0000 7f305effd640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3060102320 0x7f306019eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.786+0000 7f305effd640 1 -- 172.21.15.98:0/556822674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3060101960 msgr2=0x7f306019c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.786+0000 7f305effd640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3060101960 0x7f306019c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.786+0000 7f305effd640 1 -- 172.21.15.98:0/556822674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f306019f760 con 0x7f3060100770 2024-05-09T08:48:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.786+0000 7f305f7fe640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3060102320 0x7f306019eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.786+0000 7f305effd640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3060100770 0x7f306006a9e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f305000d950 tx=0x7f305000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.786+0000 7f3064961640 1 -- 172.21.15.98:0/556822674 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3050017020 con 0x7f3060100770 2024-05-09T08:48:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.786+0000 7f3064961640 1 -- 172.21.15.98:0/556822674 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3050009c30 con 0x7f3060100770 2024-05-09T08:48:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.786+0000 7f3065963640 1 -- 172.21.15.98:0/556822674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f306019fa50 con 0x7f3060100770 2024-05-09T08:48:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.786+0000 7f3064961640 1 -- 172.21.15.98:0/556822674 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305000f740 con 0x7f3060100770 2024-05-09T08:48:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.786+0000 7f3065963640 1 -- 172.21.15.98:0/556822674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f306019ffa0 con 0x7f3060100770 2024-05-09T08:48:32.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.787+0000 7f3065963640 1 -- 172.21.15.98:0/556822674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3060107a90 con 0x7f3060100770 2024-05-09T08:48:32.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.788+0000 7f3064961640 1 -- 172.21.15.98:0/556822674 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f305001a020 con 0x7f3060100770 2024-05-09T08:48:32.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.789+0000 7f3064961640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f30480a2b90 0x7f30480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:32.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.789+0000 7f305e7fc640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f30480a2b90 0x7f30480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:32.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.789+0000 7f3064961640 1 -- 172.21.15.98:0/556822674 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f30500cecb0 con 0x7f3060100770 2024-05-09T08:48:32.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.789+0000 7f305e7fc640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f30480a2b90 0x7f30480a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f306006a750 tx=0x7f305400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:32.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.793+0000 7f3064961640 1 -- 172.21.15.98:0/556822674 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3050021070 con 0x7f3060100770 2024-05-09T08:48:32.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.969+0000 7f3065963640 1 -- 172.21.15.98:0/556822674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3060062430 con 0x7f3060100770 2024-05-09T08:48:32.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.970+0000 7f3064961640 1 -- 172.21.15.98:0/556822674 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f3050009de0 con 0x7f3060100770 2024-05-09T08:48:32.969 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:48:32.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.971+0000 7f3065963640 1 -- 172.21.15.98:0/556822674 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f30480a2b90 msgr2=0x7f30480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.971+0000 7f3065963640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f30480a2b90 0x7f30480a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f306006a750 tx=0x7f305400b390 comp rx=0 tx=0).stop 2024-05-09T08:48:32.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.971+0000 7f3065963640 1 -- 172.21.15.98:0/556822674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3060100770 msgr2=0x7f306006a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:32.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.971+0000 7f3065963640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3060100770 0x7f306006a9e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f305000d950 tx=0x7f305000de10 comp rx=0 tx=0).stop 2024-05-09T08:48:32.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.971+0000 7f3065963640 1 -- 172.21.15.98:0/556822674 shutdown_connections 2024-05-09T08:48:32.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.971+0000 7f3065963640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3060102320 0x7f306019eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.971+0000 7f3065963640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3060101960 0x7f306019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.972+0000 7f3065963640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f30480a2b90 0x7f30480a5050 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.972+0000 7f3065963640 1 --2- 172.21.15.98:0/556822674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3060100770 0x7f306006a9e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.972+0000 7f3065963640 1 -- 172.21.15.98:0/556822674 >> 172.21.15.98:0/556822674 conn(0x7f30600fbce0 msgr2=0x7f3060104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.972+0000 7f3065963640 1 -- 172.21.15.98:0/556822674 shutdown_connections 2024-05-09T08:48:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:32.972+0000 7f3065963640 1 -- 172.21.15.98:0/556822674 wait complete. 2024-05-09T08:48:32.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:48:33.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.086+0000 7f2b70cd4640 1 -- 172.21.15.98:0/1357230278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b6c0ffeb0 msgr2=0x7f2b6c1002d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.086+0000 7f2b70cd4640 1 --2- 172.21.15.98:0/1357230278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b6c0ffeb0 0x7f2b6c1002d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009900 tx=0x7f2b5c02a410 comp rx=0 tx=0).stop 2024-05-09T08:48:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.086+0000 7f2b70cd4640 1 -- 172.21.15.98:0/1357230278 shutdown_connections 2024-05-09T08:48:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.086+0000 7f2b70cd4640 1 --2- 172.21.15.98:0/1357230278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b6c101a50 0x7f2b6c108370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.086+0000 7f2b70cd4640 1 --2- 172.21.15.98:0/1357230278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b6c101090 0x7f2b6c101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.086+0000 7f2b70cd4640 1 --2- 172.21.15.98:0/1357230278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b6c0ffeb0 0x7f2b6c1002d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.086+0000 7f2b70cd4640 1 -- 172.21.15.98:0/1357230278 >> 172.21.15.98:0/1357230278 conn(0x7f2b6c0fb470 msgr2=0x7f2b6c0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.086+0000 7f2b70cd4640 1 -- 172.21.15.98:0/1357230278 shutdown_connections 2024-05-09T08:48:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.086+0000 7f2b70cd4640 1 -- 172.21.15.98:0/1357230278 wait complete. 2024-05-09T08:48:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.087+0000 7f2b70cd4640 1 Processor -- start 2024-05-09T08:48:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.087+0000 7f2b70cd4640 1 -- start start 2024-05-09T08:48:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.087+0000 7f2b70cd4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b6c0ffeb0 0x7f2b6c195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.087+0000 7f2b70cd4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b6c101090 0x7f2b6c195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.087+0000 7f2b70cd4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b6c101a50 0x7f2b6c19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.087+0000 7f2b70cd4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6c10b310 con 0x7f2b6c0ffeb0 2024-05-09T08:48:33.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.087+0000 7f2b70cd4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6c10b190 con 0x7f2b6c101090 2024-05-09T08:48:33.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.087+0000 7f2b70cd4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6c10b490 con 0x7f2b6c101a50 2024-05-09T08:48:33.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.087+0000 7f2b6a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b6c0ffeb0 0x7f2b6c195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:33.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.087+0000 7f2b6a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b6c0ffeb0 0x7f2b6c195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36806/0 (socket says 172.21.15.98:36806) 2024-05-09T08:48:33.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.087+0000 7f2b6a59c640 1 -- 172.21.15.98:0/3652003437 learned_addr learned my addr 172.21.15.98:0/3652003437 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:33.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.087+0000 7f2b61d9b640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b6c101090 0x7f2b6c195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:33.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.087+0000 7f2b6ad9d640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b6c101a50 0x7f2b6c19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:33.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.087+0000 7f2b6a59c640 1 -- 172.21.15.98:0/3652003437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b6c101a50 msgr2=0x7f2b6c19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.088+0000 7f2b6a59c640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b6c101a50 0x7f2b6c19cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.088+0000 7f2b6a59c640 1 -- 172.21.15.98:0/3652003437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b6c101090 msgr2=0x7f2b6c195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.088+0000 7f2b6a59c640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b6c101090 0x7f2b6c195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.088+0000 7f2b6a59c640 1 -- 172.21.15.98:0/3652003437 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b6c19d5b0 con 0x7f2b6c0ffeb0 2024-05-09T08:48:33.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.088+0000 7f2b6ad9d640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b6c101a50 0x7f2b6c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:33.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.088+0000 7f2b6a59c640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b6c0ffeb0 0x7f2b6c195990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c031610 tx=0x7f2b5c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:33.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.088+0000 7f2b63fff640 1 -- 172.21.15.98:0/3652003437 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c03f030 con 0x7f2b6c0ffeb0 2024-05-09T08:48:33.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.088+0000 7f2b70cd4640 1 -- 172.21.15.98:0/3652003437 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b6c19d840 con 0x7f2b6c0ffeb0 2024-05-09T08:48:33.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.088+0000 7f2b63fff640 1 -- 172.21.15.98:0/3652003437 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b5c003ad0 con 0x7f2b6c0ffeb0 2024-05-09T08:48:33.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.088+0000 7f2b63fff640 1 -- 172.21.15.98:0/3652003437 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c035070 con 0x7f2b6c0ffeb0 2024-05-09T08:48:33.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.088+0000 7f2b70cd4640 1 -- 172.21.15.98:0/3652003437 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b6c19dd30 con 0x7f2b6c0ffeb0 2024-05-09T08:48:33.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.089+0000 7f2b70cd4640 1 -- 172.21.15.98:0/3652003437 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b6c107320 con 0x7f2b6c0ffeb0 2024-05-09T08:48:33.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.092+0000 7f2b63fff640 1 -- 172.21.15.98:0/3652003437 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2b5c042030 con 0x7f2b6c0ffeb0 2024-05-09T08:48:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.093+0000 7f2b63fff640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b580a2b90 0x7f2b580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.093+0000 7f2b63fff640 1 -- 172.21.15.98:0/3652003437 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f2b5c0efc20 con 0x7f2b6c0ffeb0 2024-05-09T08:48:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.093+0000 7f2b61d9b640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b580a2b90 0x7f2b580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.093+0000 7f2b61d9b640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b580a2b90 0x7f2b580a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c196ea0 tx=0x7f2b5400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:33.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.096+0000 7f2b63fff640 1 -- 172.21.15.98:0/3652003437 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2b5c0bc940 con 0x7f2b6c0ffeb0 2024-05-09T08:48:33.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.274+0000 7f2b70cd4640 1 -- 172.21.15.98:0/3652003437 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2b6c062430 con 0x7f2b6c0ffeb0 2024-05-09T08:48:33.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.274+0000 7f2b63fff640 1 -- 172.21.15.98:0/3652003437 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f2b5c0bc240 con 0x7f2b6c0ffeb0 2024-05-09T08:48:33.274 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-09T08:48:33.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.278+0000 7f2b70cd4640 1 -- 172.21.15.98:0/3652003437 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b580a2b90 msgr2=0x7f2b580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.278+0000 7f2b70cd4640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b580a2b90 0x7f2b580a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c196ea0 tx=0x7f2b5400b390 comp rx=0 tx=0).stop 2024-05-09T08:48:33.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.278+0000 7f2b70cd4640 1 -- 172.21.15.98:0/3652003437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b6c0ffeb0 msgr2=0x7f2b6c195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.278+0000 7f2b70cd4640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b6c0ffeb0 0x7f2b6c195990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c031610 tx=0x7f2b5c004270 comp rx=0 tx=0).stop 2024-05-09T08:48:33.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.278+0000 7f2b70cd4640 1 -- 172.21.15.98:0/3652003437 shutdown_connections 2024-05-09T08:48:33.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.278+0000 7f2b70cd4640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b6c101a50 0x7f2b6c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.278+0000 7f2b70cd4640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b6c101090 0x7f2b6c195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.278+0000 7f2b70cd4640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b580a2b90 0x7f2b580a5050 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.278+0000 7f2b70cd4640 1 --2- 172.21.15.98:0/3652003437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b6c0ffeb0 0x7f2b6c195990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.278+0000 7f2b70cd4640 1 -- 172.21.15.98:0/3652003437 >> 172.21.15.98:0/3652003437 conn(0x7f2b6c0fb470 msgr2=0x7f2b6c1044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:33.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.278+0000 7f2b70cd4640 1 -- 172.21.15.98:0/3652003437 shutdown_connections 2024-05-09T08:48:33.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.278+0000 7f2b70cd4640 1 -- 172.21.15.98:0/3652003437 wait complete. 2024-05-09T08:48:33.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:48:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.392+0000 7f7305114640 1 -- 172.21.15.98:0/1965544245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7300104c40 msgr2=0x7f7300107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.392+0000 7f7305114640 1 --2- 172.21.15.98:0/1965544245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7300104c40 0x7f7300107030 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f72e8009900 tx=0x7f72e802a410 comp rx=0 tx=0).stop 2024-05-09T08:48:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.392+0000 7f7305114640 1 -- 172.21.15.98:0/1965544245 shutdown_connections 2024-05-09T08:48:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.392+0000 7f7305114640 1 --2- 172.21.15.98:0/1965544245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7300107570 0x7f73001099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.392+0000 7f7305114640 1 --2- 172.21.15.98:0/1965544245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7300104c40 0x7f7300107030 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.392+0000 7f7305114640 1 --2- 172.21.15.98:0/1965544245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7300102310 0x7f7300104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.392+0000 7f7305114640 1 -- 172.21.15.98:0/1965544245 >> 172.21.15.98:0/1965544245 conn(0x7f73000fbd00 msgr2=0x7f73000fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.392+0000 7f7305114640 1 -- 172.21.15.98:0/1965544245 shutdown_connections 2024-05-09T08:48:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.392+0000 7f7305114640 1 -- 172.21.15.98:0/1965544245 wait complete. 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f7305114640 1 Processor -- start 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f7305114640 1 -- start start 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f7305114640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7300102310 0x7f7300197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f7305114640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7300104c40 0x7f73001981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f7305114640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7300107570 0x7f730019f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f7305114640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f730010d570 con 0x7f7300107570 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f7305114640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f730010d3f0 con 0x7f7300102310 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f7305114640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f730010d6f0 con 0x7f7300104c40 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f72ff59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7300107570 0x7f730019f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f72ff59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7300107570 0x7f730019f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36828/0 (socket says 172.21.15.98:36828) 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f72ff59e640 1 -- 172.21.15.98:0/2191738319 learned_addr learned my addr 172.21.15.98:0/2191738319 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f72fed9d640 1 --2- 172.21.15.98:0/2191738319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7300102310 0x7f7300197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f72f7fff640 1 --2- 172.21.15.98:0/2191738319 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7300104c40 0x7f73001981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f72ff59e640 1 -- 172.21.15.98:0/2191738319 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7300104c40 msgr2=0x7f73001981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f72ff59e640 1 --2- 172.21.15.98:0/2191738319 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7300104c40 0x7f73001981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f72ff59e640 1 -- 172.21.15.98:0/2191738319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7300102310 msgr2=0x7f7300197c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f72ff59e640 1 --2- 172.21.15.98:0/2191738319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7300102310 0x7f7300197c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.393+0000 7f72ff59e640 1 -- 172.21.15.98:0/2191738319 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730019f870 con 0x7f7300107570 2024-05-09T08:48:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.394+0000 7f72ff59e640 1 --2- 172.21.15.98:0/2191738319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7300107570 0x7f730019f1d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f72f800eaa0 tx=0x7f72f800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.394+0000 7f72fcd99640 1 -- 172.21.15.98:0/2191738319 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f8004d20 con 0x7f7300107570 2024-05-09T08:48:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.394+0000 7f7305114640 1 -- 172.21.15.98:0/2191738319 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f730019fb60 con 0x7f7300107570 2024-05-09T08:48:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.394+0000 7f72fcd99640 1 -- 172.21.15.98:0/2191738319 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72f8005020 con 0x7f7300107570 2024-05-09T08:48:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.394+0000 7f72fcd99640 1 -- 172.21.15.98:0/2191738319 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f80116d0 con 0x7f7300107570 2024-05-09T08:48:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.394+0000 7f7305114640 1 -- 172.21.15.98:0/2191738319 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73001a0060 con 0x7f7300107570 2024-05-09T08:48:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.395+0000 7f7305114640 1 -- 172.21.15.98:0/2191738319 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73000fd7f0 con 0x7f7300107570 2024-05-09T08:48:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.396+0000 7f72fcd99640 1 -- 172.21.15.98:0/2191738319 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f72f80051d0 con 0x7f7300107570 2024-05-09T08:48:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.397+0000 7f72fcd99640 1 --2- 172.21.15.98:0/2191738319 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f72ec0a2b90 0x7f72ec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.397+0000 7f72fed9d640 1 --2- 172.21.15.98:0/2191738319 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f72ec0a2b90 0x7f72ec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.397+0000 7f72fcd99640 1 -- 172.21.15.98:0/2191738319 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f72f80cf910 con 0x7f7300107570 2024-05-09T08:48:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.397+0000 7f72fed9d640 1 --2- 172.21.15.98:0/2191738319 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f72ec0a2b90 0x7f72ec0a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7300199130 tx=0x7f72f0009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.400+0000 7f72fcd99640 1 -- 172.21.15.98:0/2191738319 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f72f8016030 con 0x7f7300107570 2024-05-09T08:48:33.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.576+0000 7f7305114640 1 -- 172.21.15.98:0/2191738319 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f730002d410 con 0x7f7300107570 2024-05-09T08:48:33.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.576+0000 7f72fcd99640 1 -- 172.21.15.98:0/2191738319 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f72f809c630 con 0x7f7300107570 2024-05-09T08:48:33.576 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:48:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.578+0000 7f7305114640 1 -- 172.21.15.98:0/2191738319 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f72ec0a2b90 msgr2=0x7f72ec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.578+0000 7f7305114640 1 --2- 172.21.15.98:0/2191738319 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f72ec0a2b90 0x7f72ec0a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7300199130 tx=0x7f72f0009110 comp rx=0 tx=0).stop 2024-05-09T08:48:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.578+0000 7f7305114640 1 -- 172.21.15.98:0/2191738319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7300107570 msgr2=0x7f730019f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.578+0000 7f7305114640 1 --2- 172.21.15.98:0/2191738319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7300107570 0x7f730019f1d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f72f800eaa0 tx=0x7f72f800eea0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.578+0000 7f7305114640 1 -- 172.21.15.98:0/2191738319 shutdown_connections 2024-05-09T08:48:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.578+0000 7f7305114640 1 --2- 172.21.15.98:0/2191738319 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f72ec0a2b90 0x7f72ec0a5050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.578+0000 7f7305114640 1 --2- 172.21.15.98:0/2191738319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7300107570 0x7f730019f1d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.578+0000 7f7305114640 1 --2- 172.21.15.98:0/2191738319 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7300104c40 0x7f73001981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.578+0000 7f7305114640 1 --2- 172.21.15.98:0/2191738319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7300102310 0x7f7300197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.578+0000 7f7305114640 1 -- 172.21.15.98:0/2191738319 >> 172.21.15.98:0/2191738319 conn(0x7f73000fbd00 msgr2=0x7f7300102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.578+0000 7f7305114640 1 -- 172.21.15.98:0/2191738319 shutdown_connections 2024-05-09T08:48:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.578+0000 7f7305114640 1 -- 172.21.15.98:0/2191738319 wait complete. 2024-05-09T08:48:33.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:33 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4189624105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:33.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:33 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/178898971' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:48:33.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:33 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/556822674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:48:33.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T08:48:33.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:33 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/4189624105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:33.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:33 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/178898971' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:48:33.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:33 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/556822674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:48:33.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.691+0000 7f2bafbdb640 1 -- 172.21.15.98:0/1401231633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ba8100770 msgr2=0x7f2ba8100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.691+0000 7f2bafbdb640 1 --2- 172.21.15.98:0/1401231633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ba8100770 0x7f2ba8100b90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4009900 tx=0x7f2ba402a410 comp rx=0 tx=0).stop 2024-05-09T08:48:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 -- 172.21.15.98:0/1401231633 shutdown_connections 2024-05-09T08:48:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 --2- 172.21.15.98:0/1401231633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ba8102320 0x7f2ba8108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 --2- 172.21.15.98:0/1401231633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ba8101960 0x7f2ba8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 --2- 172.21.15.98:0/1401231633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ba8100770 0x7f2ba8100b90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 -- 172.21.15.98:0/1401231633 >> 172.21.15.98:0/1401231633 conn(0x7f2ba80fbce0 msgr2=0x7f2ba80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 -- 172.21.15.98:0/1401231633 shutdown_connections 2024-05-09T08:48:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 -- 172.21.15.98:0/1401231633 wait complete. 2024-05-09T08:48:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 Processor -- start 2024-05-09T08:48:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 -- start start 2024-05-09T08:48:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ba8100770 0x7f2ba8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ba8101960 0x7f2ba819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ba8102320 0x7f2ba819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba810c3c0 con 0x7f2ba8100770 2024-05-09T08:48:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba810c240 con 0x7f2ba8101960 2024-05-09T08:48:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bafbdb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba810c540 con 0x7f2ba8102320 2024-05-09T08:48:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.692+0000 7f2bad977640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ba8100770 0x7f2ba8069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2bad977640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ba8100770 0x7f2ba8069590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36844/0 (socket says 172.21.15.98:36844) 2024-05-09T08:48:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2bad977640 1 -- 172.21.15.98:0/3780172941 learned_addr learned my addr 172.21.15.98:0/3780172941 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2bad176640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ba8101960 0x7f2ba819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2bae178640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ba8102320 0x7f2ba819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2bad977640 1 -- 172.21.15.98:0/3780172941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ba8102320 msgr2=0x7f2ba819eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2bad977640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ba8102320 0x7f2ba819eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2bad977640 1 -- 172.21.15.98:0/3780172941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ba8101960 msgr2=0x7f2ba819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2bad977640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ba8101960 0x7f2ba819c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2bad977640 1 -- 172.21.15.98:0/3780172941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba819f780 con 0x7f2ba8100770 2024-05-09T08:48:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2bae178640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ba8102320 0x7f2ba819eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2bad176640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ba8101960 0x7f2ba819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2bad977640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ba8100770 0x7f2ba8069590 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4031610 tx=0x7f2ba40043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2b9affd640 1 -- 172.21.15.98:0/3780172941 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba403f030 con 0x7f2ba8100770 2024-05-09T08:48:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2bafbdb640 1 -- 172.21.15.98:0/3780172941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba819fa10 con 0x7f2ba8100770 2024-05-09T08:48:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2bafbdb640 1 -- 172.21.15.98:0/3780172941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba819ff50 con 0x7f2ba8100770 2024-05-09T08:48:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2b9affd640 1 -- 172.21.15.98:0/3780172941 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ba4003ce0 con 0x7f2ba8100770 2024-05-09T08:48:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.693+0000 7f2b9affd640 1 -- 172.21.15.98:0/3780172941 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4035070 con 0x7f2ba8100770 2024-05-09T08:48:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.694+0000 7f2bafbdb640 1 -- 172.21.15.98:0/3780172941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ba8107a90 con 0x7f2ba8100770 2024-05-09T08:48:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.695+0000 7f2b9affd640 1 -- 172.21.15.98:0/3780172941 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2ba4042030 con 0x7f2ba8100770 2024-05-09T08:48:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.696+0000 7f2b9affd640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b940a2b90 0x7f2b940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.696+0000 7f2bad176640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b940a2b90 0x7f2b940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.696+0000 7f2b9affd640 1 -- 172.21.15.98:0/3780172941 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f2ba40efc70 con 0x7f2ba8100770 2024-05-09T08:48:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.696+0000 7f2bad176640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b940a2b90 0x7f2b940a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8069300 tx=0x7f2b9c005ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.700+0000 7f2b9affd640 1 -- 172.21.15.98:0/3780172941 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2ba40bc990 con 0x7f2ba8100770 2024-05-09T08:48:33.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:33 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/4189624105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:33.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:33 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/178898971' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:48:33.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:33 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/556822674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:48:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.886+0000 7f2bafbdb640 1 -- 172.21.15.98:0/3780172941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2ba8062430 con 0x7f2ba8100770 2024-05-09T08:48:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.886+0000 7f2b9affd640 1 -- 172.21.15.98:0/3780172941 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+588 (secure 0 0 0) 0x7f2ba40bc290 con 0x7f2ba8100770 2024-05-09T08:48:33.885 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2163305674 2024-05-10T08:41:17.098271+0000 2024-05-09T08:48:33.886 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1385598801 2024-05-10T08:41:17.098271+0000 2024-05-09T08:48:33.886 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/498605085 2024-05-10T08:41:17.098271+0000 2024-05-09T08:48:33.886 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/1313924889 2024-05-10T08:40:50.793131+0000 2024-05-09T08:48:33.886 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/498605085 2024-05-10T08:41:17.098271+0000 2024-05-09T08:48:33.886 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/812399881 2024-05-10T08:40:50.793131+0000 2024-05-09T08:48:33.886 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2540720935 2024-05-10T08:40:50.793131+0000 2024-05-09T08:48:33.886 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/1313924889 2024-05-10T08:40:50.793131+0000 2024-05-09T08:48:33.886 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3487867282 2024-05-10T08:41:17.098271+0000 2024-05-09T08:48:33.886 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/645511714 2024-05-10T08:40:50.793131+0000 2024-05-09T08:48:33.886 INFO:teuthology.orchestra.run.smithi098.stderr:listed 10 entries 2024-05-09T08:48:33.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.888+0000 7f2bafbdb640 1 -- 172.21.15.98:0/3780172941 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b940a2b90 msgr2=0x7f2b940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.888+0000 7f2bafbdb640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b940a2b90 0x7f2b940a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8069300 tx=0x7f2b9c005ec0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.888+0000 7f2bafbdb640 1 -- 172.21.15.98:0/3780172941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ba8100770 msgr2=0x7f2ba8069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.888+0000 7f2bafbdb640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ba8100770 0x7f2ba8069590 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4031610 tx=0x7f2ba40043b0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.888+0000 7f2bafbdb640 1 -- 172.21.15.98:0/3780172941 shutdown_connections 2024-05-09T08:48:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.888+0000 7f2bafbdb640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ba8102320 0x7f2ba819eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.888+0000 7f2bafbdb640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ba8101960 0x7f2ba819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.888+0000 7f2bafbdb640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2b940a2b90 0x7f2b940a5050 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.888+0000 7f2bafbdb640 1 --2- 172.21.15.98:0/3780172941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ba8100770 0x7f2ba8069590 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.888+0000 7f2bafbdb640 1 -- 172.21.15.98:0/3780172941 >> 172.21.15.98:0/3780172941 conn(0x7f2ba80fbce0 msgr2=0x7f2ba8104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.888+0000 7f2bafbdb640 1 -- 172.21.15.98:0/3780172941 shutdown_connections 2024-05-09T08:48:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:33.888+0000 7f2bafbdb640 1 -- 172.21.15.98:0/3780172941 wait complete. 2024-05-09T08:48:33.901 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-05-09T08:48:33.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:48:33.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:48:33.925 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[74261]: fuse finished with error 0 and tester_r 0 2024-05-09T08:48:33.927 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T08:48:34.025 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-05-09T08:48:34.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:48:34.025 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-05-09T08:48:34.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:48:34.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.194+0000 7f933797e640 1 -- 172.21.15.98:0/408698142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93301000e0 msgr2=0x7f9330102510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.194+0000 7f933797e640 1 --2- 172.21.15.98:0/408698142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93301000e0 0x7f9330102510 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9328009900 tx=0x7f932802a410 comp rx=0 tx=0).stop 2024-05-09T08:48:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.194+0000 7f933797e640 1 -- 172.21.15.98:0/408698142 shutdown_connections 2024-05-09T08:48:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.194+0000 7f933797e640 1 --2- 172.21.15.98:0/408698142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9330105400 0x7f9330105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.194+0000 7f933797e640 1 --2- 172.21.15.98:0/408698142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9330102a50 0x7f9330104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.194+0000 7f933797e640 1 --2- 172.21.15.98:0/408698142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93301000e0 0x7f9330102510 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.194+0000 7f933797e640 1 -- 172.21.15.98:0/408698142 >> 172.21.15.98:0/408698142 conn(0x7f93300fbce0 msgr2=0x7f93300fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.194+0000 7f933797e640 1 -- 172.21.15.98:0/408698142 shutdown_connections 2024-05-09T08:48:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.194+0000 7f933797e640 1 -- 172.21.15.98:0/408698142 wait complete. 2024-05-09T08:48:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.194+0000 7f933797e640 1 Processor -- start 2024-05-09T08:48:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.194+0000 7f933797e640 1 -- start start 2024-05-09T08:48:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933797e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93301000e0 0x7f933019e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933797e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9330102a50 0x7f933019e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933797e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9330105400 0x7f93301a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933797e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9330113b30 con 0x7f93301000e0 2024-05-09T08:48:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933797e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f93301139b0 con 0x7f9330102a50 2024-05-09T08:48:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933797e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9330113cb0 con 0x7f9330105400 2024-05-09T08:48:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933571a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93301000e0 0x7f933019e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933571a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93301000e0 0x7f933019e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36860/0 (socket says 172.21.15.98:36860) 2024-05-09T08:48:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933571a640 1 -- 172.21.15.98:0/2974243946 learned_addr learned my addr 172.21.15.98:0/2974243946 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f9334f19640 1 --2- 172.21.15.98:0/2974243946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9330102a50 0x7f933019e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933571a640 1 -- 172.21.15.98:0/2974243946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9330105400 msgr2=0x7f93301a57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f9335f1b640 1 --2- 172.21.15.98:0/2974243946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9330105400 0x7f93301a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933571a640 1 --2- 172.21.15.98:0/2974243946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9330105400 0x7f93301a57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933571a640 1 -- 172.21.15.98:0/2974243946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9330102a50 msgr2=0x7f933019e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933571a640 1 --2- 172.21.15.98:0/2974243946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9330102a50 0x7f933019e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933571a640 1 -- 172.21.15.98:0/2974243946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93301a5e20 con 0x7f93301000e0 2024-05-09T08:48:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933571a640 1 --2- 172.21.15.98:0/2974243946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93301000e0 0x7f933019e250 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9328009230 tx=0x7f9328004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f932e7fc640 1 -- 172.21.15.98:0/2974243946 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932803f030 con 0x7f93301000e0 2024-05-09T08:48:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f932e7fc640 1 -- 172.21.15.98:0/2974243946 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9328003ce0 con 0x7f93301000e0 2024-05-09T08:48:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f933797e640 1 -- 172.21.15.98:0/2974243946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93301a60b0 con 0x7f93301000e0 2024-05-09T08:48:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.195+0000 7f932e7fc640 1 -- 172.21.15.98:0/2974243946 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328035070 con 0x7f93301000e0 2024-05-09T08:48:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.196+0000 7f933797e640 1 -- 172.21.15.98:0/2974243946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93301a65a0 con 0x7f93301000e0 2024-05-09T08:48:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.196+0000 7f933797e640 1 -- 172.21.15.98:0/2974243946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93300fd7d0 con 0x7f93301000e0 2024-05-09T08:48:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.198+0000 7f932e7fc640 1 -- 172.21.15.98:0/2974243946 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9328042030 con 0x7f93301000e0 2024-05-09T08:48:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.198+0000 7f932e7fc640 1 --2- 172.21.15.98:0/2974243946 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f93180a2b90 0x7f93180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.198+0000 7f9334f19640 1 --2- 172.21.15.98:0/2974243946 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f93180a2b90 0x7f93180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.198+0000 7f932e7fc640 1 -- 172.21.15.98:0/2974243946 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f93280efc90 con 0x7f93301000e0 2024-05-09T08:48:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.199+0000 7f9334f19640 1 --2- 172.21.15.98:0/2974243946 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f93180a2b90 0x7f93180a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f933019f710 tx=0x7f9324009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.202+0000 7f932e7fc640 1 -- 172.21.15.98:0/2974243946 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f93280bc9b0 con 0x7f93301000e0 2024-05-09T08:48:34.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.385+0000 7f933797e640 1 -- 172.21.15.98:0/2974243946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9330062430 con 0x7f93301000e0 2024-05-09T08:48:34.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.386+0000 7f932e7fc640 1 -- 172.21.15.98:0/2974243946 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24150 (secure 0 0 0) 0x7f93280b6e20 con 0x7f93301000e0 2024-05-09T08:48:34.386 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:34.386 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":90,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:48:22.719729+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:48:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.388+0000 7f933797e640 1 -- 172.21.15.98:0/2974243946 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f93180a2b90 msgr2=0x7f93180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.388+0000 7f933797e640 1 --2- 172.21.15.98:0/2974243946 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f93180a2b90 0x7f93180a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f933019f710 tx=0x7f9324009110 comp rx=0 tx=0).stop 2024-05-09T08:48:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.388+0000 7f933797e640 1 -- 172.21.15.98:0/2974243946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93301000e0 msgr2=0x7f933019e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.388+0000 7f933797e640 1 --2- 172.21.15.98:0/2974243946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93301000e0 0x7f933019e250 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9328009230 tx=0x7f9328004360 comp rx=0 tx=0).stop 2024-05-09T08:48:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.388+0000 7f933797e640 1 -- 172.21.15.98:0/2974243946 shutdown_connections 2024-05-09T08:48:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.388+0000 7f933797e640 1 --2- 172.21.15.98:0/2974243946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9330105400 0x7f93301a57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.388+0000 7f933797e640 1 --2- 172.21.15.98:0/2974243946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9330102a50 0x7f933019e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.388+0000 7f933797e640 1 --2- 172.21.15.98:0/2974243946 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f93180a2b90 0x7f93180a5050 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.388+0000 7f933797e640 1 --2- 172.21.15.98:0/2974243946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93301000e0 0x7f933019e250 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.389+0000 7f933797e640 1 -- 172.21.15.98:0/2974243946 >> 172.21.15.98:0/2974243946 conn(0x7f93300fbce0 msgr2=0x7f9330103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.389+0000 7f933797e640 1 -- 172.21.15.98:0/2974243946 shutdown_connections 2024-05-09T08:48:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.389+0000 7f933797e640 1 -- 172.21.15.98:0/2974243946 wait complete. 2024-05-09T08:48:34.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:48:34.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.500+0000 7ff3c117e640 1 -- 172.21.15.98:0/3240308571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3bc102a80 msgr2=0x7ff3bc102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:34.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.500+0000 7ff3c117e640 1 --2- 172.21.15.98:0/3240308571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3bc102a80 0x7ff3bc102f00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009900 tx=0x7ff3b002a410 comp rx=0 tx=0).stop 2024-05-09T08:48:34.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.500+0000 7ff3c117e640 1 -- 172.21.15.98:0/3240308571 shutdown_connections 2024-05-09T08:48:34.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.500+0000 7ff3c117e640 1 --2- 172.21.15.98:0/3240308571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3bc103440 0x7ff3bc109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.500+0000 7ff3c117e640 1 --2- 172.21.15.98:0/3240308571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3bc102a80 0x7ff3bc102f00 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.500+0000 7ff3c117e640 1 --2- 172.21.15.98:0/3240308571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3bc101890 0x7ff3bc101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.500+0000 7ff3c117e640 1 -- 172.21.15.98:0/3240308571 >> 172.21.15.98:0/3240308571 conn(0x7ff3bc0fcde0 msgr2=0x7ff3bc0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.500+0000 7ff3c117e640 1 -- 172.21.15.98:0/3240308571 shutdown_connections 2024-05-09T08:48:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.500+0000 7ff3c117e640 1 -- 172.21.15.98:0/3240308571 wait complete. 2024-05-09T08:48:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.501+0000 7ff3c117e640 1 Processor -- start 2024-05-09T08:48:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.501+0000 7ff3c117e640 1 -- start start 2024-05-09T08:48:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.501+0000 7ff3c117e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3bc101890 0x7ff3bc197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.501+0000 7ff3c117e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3bc102a80 0x7ff3bc198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.501+0000 7ff3c117e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3bc103440 0x7ff3bc19f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.501+0000 7ff3c117e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc10d570 con 0x7ff3bc103440 2024-05-09T08:48:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.501+0000 7ff3c117e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc10d3f0 con 0x7ff3bc101890 2024-05-09T08:48:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.501+0000 7ff3c117e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc10d6f0 con 0x7ff3bc102a80 2024-05-09T08:48:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3bb59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3bc103440 0x7ff3bc19f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3bb59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3bc103440 0x7ff3bc19f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36882/0 (socket says 172.21.15.98:36882) 2024-05-09T08:48:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3bb59e640 1 -- 172.21.15.98:0/1625516539 learned_addr learned my addr 172.21.15.98:0/1625516539 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3ba59c640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3bc102a80 0x7ff3bc198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3bad9d640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3bc101890 0x7ff3bc197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3bb59e640 1 -- 172.21.15.98:0/1625516539 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3bc102a80 msgr2=0x7ff3bc198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3bb59e640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3bc102a80 0x7ff3bc198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3bb59e640 1 -- 172.21.15.98:0/1625516539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3bc101890 msgr2=0x7ff3bc197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3bb59e640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3bc101890 0x7ff3bc197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3bb59e640 1 -- 172.21.15.98:0/1625516539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3bc19f7f0 con 0x7ff3bc103440 2024-05-09T08:48:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3bad9d640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3bc101890 0x7ff3bc197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3bb59e640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3bc103440 0x7ff3bc19f180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009fd0 tx=0x7ff3b400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3a3fff640 1 -- 172.21.15.98:0/1625516539 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4011830 con 0x7ff3bc103440 2024-05-09T08:48:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3c117e640 1 -- 172.21.15.98:0/1625516539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3bc19fae0 con 0x7ff3bc103440 2024-05-09T08:48:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3a3fff640 1 -- 172.21.15.98:0/1625516539 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3b4016070 con 0x7ff3bc103440 2024-05-09T08:48:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3a3fff640 1 -- 172.21.15.98:0/1625516539 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b400f2e0 con 0x7ff3bc103440 2024-05-09T08:48:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.502+0000 7ff3c117e640 1 -- 172.21.15.98:0/1625516539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3bc19ffb0 con 0x7ff3bc103440 2024-05-09T08:48:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.503+0000 7ff3c117e640 1 -- 172.21.15.98:0/1625516539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3bc108b50 con 0x7ff3bc103440 2024-05-09T08:48:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.505+0000 7ff3a3fff640 1 -- 172.21.15.98:0/1625516539 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff3b40119d0 con 0x7ff3bc103440 2024-05-09T08:48:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.505+0000 7ff3a3fff640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff3a40a2b90 0x7ff3a40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.505+0000 7ff3bad9d640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff3a40a2b90 0x7ff3a40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.505+0000 7ff3a3fff640 1 -- 172.21.15.98:0/1625516539 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff3b40cefe0 con 0x7ff3bc103440 2024-05-09T08:48:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.506+0000 7ff3bad9d640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff3a40a2b90 0x7ff3a40a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc1028e0 tx=0x7ff3ac007870 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:34.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.509+0000 7ff3a3fff640 1 -- 172.21.15.98:0/1625516539 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff3b409bd00 con 0x7ff3bc103440 2024-05-09T08:48:34.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:34 smithi098 ceph-mon[30092]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-05-09T08:48:34.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:34 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3652003437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:48:34.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:34 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2191738319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:48:34.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:34 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3780172941' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:48:34.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:34 smithi142 ceph-mon[34625]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-05-09T08:48:34.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:34 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3652003437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:48:34.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:34 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2191738319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:48:34.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:34 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3780172941' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:48:34.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.685+0000 7ff3c117e640 1 -- 172.21.15.98:0/1625516539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3bc02d390 con 0x7ff3bc103440 2024-05-09T08:48:34.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.688+0000 7ff3a3fff640 1 -- 172.21.15.98:0/1625516539 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24150 (secure 0 0 0) 0x7ff3b409b600 con 0x7ff3bc103440 2024-05-09T08:48:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:34.687 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":90,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:48:22.719729+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:48:34.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.689+0000 7ff3c117e640 1 -- 172.21.15.98:0/1625516539 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff3a40a2b90 msgr2=0x7ff3a40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:34.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.689+0000 7ff3c117e640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff3a40a2b90 0x7ff3a40a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc1028e0 tx=0x7ff3ac007870 comp rx=0 tx=0).stop 2024-05-09T08:48:34.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.689+0000 7ff3c117e640 1 -- 172.21.15.98:0/1625516539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3bc103440 msgr2=0x7ff3bc19f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:34.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.689+0000 7ff3c117e640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3bc103440 0x7ff3bc19f180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009fd0 tx=0x7ff3b400b8e0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.689+0000 7ff3c117e640 1 -- 172.21.15.98:0/1625516539 shutdown_connections 2024-05-09T08:48:34.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.689+0000 7ff3c117e640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff3a40a2b90 0x7ff3a40a5050 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.689+0000 7ff3c117e640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3bc103440 0x7ff3bc19f180 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.689+0000 7ff3c117e640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3bc102a80 0x7ff3bc198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.689+0000 7ff3c117e640 1 --2- 172.21.15.98:0/1625516539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3bc101890 0x7ff3bc197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.689+0000 7ff3c117e640 1 -- 172.21.15.98:0/1625516539 >> 172.21.15.98:0/1625516539 conn(0x7ff3bc0fcde0 msgr2=0x7ff3bc105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:34.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.689+0000 7ff3c117e640 1 -- 172.21.15.98:0/1625516539 shutdown_connections 2024-05-09T08:48:34.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.689+0000 7ff3c117e640 1 -- 172.21.15.98:0/1625516539 wait complete. 2024-05-09T08:48:34.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T08:48:34.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:34 smithi105 ceph-mon[34498]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-05-09T08:48:34.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:34 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3652003437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:48:34.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:34 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2191738319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:48:34.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:34 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3780172941' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:48:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.805+0000 7fb6cfefc640 1 -- 172.21.15.98:0/4210449822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c8105420 msgr2=0x7fb6c81058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.805+0000 7fb6cfefc640 1 --2- 172.21.15.98:0/4210449822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c8105420 0x7fb6c81058a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0009900 tx=0x7fb6c002a410 comp rx=0 tx=0).stop 2024-05-09T08:48:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.805+0000 7fb6cfefc640 1 -- 172.21.15.98:0/4210449822 shutdown_connections 2024-05-09T08:48:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.805+0000 7fb6cfefc640 1 --2- 172.21.15.98:0/4210449822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c8105420 0x7fb6c81058a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.805+0000 7fb6cfefc640 1 --2- 172.21.15.98:0/4210449822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c8102a70 0x7fb6c8104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.805+0000 7fb6cfefc640 1 --2- 172.21.15.98:0/4210449822 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6c8100100 0x7fb6c8102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.805+0000 7fb6cfefc640 1 -- 172.21.15.98:0/4210449822 >> 172.21.15.98:0/4210449822 conn(0x7fb6c80fbd20 msgr2=0x7fb6c80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.805+0000 7fb6cfefc640 1 -- 172.21.15.98:0/4210449822 shutdown_connections 2024-05-09T08:48:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.805+0000 7fb6cfefc640 1 -- 172.21.15.98:0/4210449822 wait complete. 2024-05-09T08:48:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.806+0000 7fb6cfefc640 1 Processor -- start 2024-05-09T08:48:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.806+0000 7fb6cfefc640 1 -- start start 2024-05-09T08:48:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.806+0000 7fb6cfefc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6c8100100 0x7fb6c819e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.806+0000 7fb6cfefc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c8102a70 0x7fb6c819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.806+0000 7fb6cfefc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c8105420 0x7fb6c81a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.806+0000 7fb6cfefc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c8113b00 con 0x7fb6c8102a70 2024-05-09T08:48:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.806+0000 7fb6cfefc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c8113980 con 0x7fb6c8105420 2024-05-09T08:48:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.806+0000 7fb6cfefc640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c8113c80 con 0x7fb6c8100100 2024-05-09T08:48:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.806+0000 7fb6cd497640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c8102a70 0x7fb6c819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.806+0000 7fb6cd497640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c8102a70 0x7fb6c819e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36898/0 (socket says 172.21.15.98:36898) 2024-05-09T08:48:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.806+0000 7fb6cd497640 1 -- 172.21.15.98:0/678409443 learned_addr learned my addr 172.21.15.98:0/678409443 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.806+0000 7fb6ce499640 1 --2- 172.21.15.98:0/678409443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c8105420 0x7fb6c81a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.806+0000 7fb6cdc98640 1 --2- 172.21.15.98:0/678409443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6c8100100 0x7fb6c819e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.807+0000 7fb6cd497640 1 -- 172.21.15.98:0/678409443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6c8100100 msgr2=0x7fb6c819e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.807+0000 7fb6cd497640 1 --2- 172.21.15.98:0/678409443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6c8100100 0x7fb6c819e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.807+0000 7fb6cd497640 1 -- 172.21.15.98:0/678409443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c8105420 msgr2=0x7fb6c81a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.807+0000 7fb6cd497640 1 --2- 172.21.15.98:0/678409443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c8105420 0x7fb6c81a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.807+0000 7fb6cd497640 1 -- 172.21.15.98:0/678409443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c81a5d00 con 0x7fb6c8102a70 2024-05-09T08:48:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.807+0000 7fb6cd497640 1 --2- 172.21.15.98:0/678409443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c8102a70 0x7fb6c819e620 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800c920 tx=0x7fb6b800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.807+0000 7fb6beffd640 1 -- 172.21.15.98:0/678409443 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8007c70 con 0x7fb6c8102a70 2024-05-09T08:48:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.807+0000 7fb6beffd640 1 -- 172.21.15.98:0/678409443 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6b8007e10 con 0x7fb6c8102a70 2024-05-09T08:48:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.807+0000 7fb6cfefc640 1 -- 172.21.15.98:0/678409443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c81a5ff0 con 0x7fb6c8102a70 2024-05-09T08:48:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.807+0000 7fb6beffd640 1 -- 172.21.15.98:0/678409443 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8018890 con 0x7fb6c8102a70 2024-05-09T08:48:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.807+0000 7fb6cfefc640 1 -- 172.21.15.98:0/678409443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c81a6500 con 0x7fb6c8102a70 2024-05-09T08:48:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.808+0000 7fb6cfefc640 1 -- 172.21.15.98:0/678409443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6c80fd810 con 0x7fb6c8102a70 2024-05-09T08:48:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.809+0000 7fb6beffd640 1 -- 172.21.15.98:0/678409443 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb6b8018a70 con 0x7fb6c8102a70 2024-05-09T08:48:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.809+0000 7fb6beffd640 1 --2- 172.21.15.98:0/678409443 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb6b40a2c10 0x7fb6b40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.810+0000 7fb6cdc98640 1 --2- 172.21.15.98:0/678409443 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb6b40a2c10 0x7fb6b40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.810+0000 7fb6beffd640 1 -- 172.21.15.98:0/678409443 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb6b80cf1e0 con 0x7fb6c8102a70 2024-05-09T08:48:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.810+0000 7fb6cdc98640 1 --2- 172.21.15.98:0/678409443 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb6b40a2c10 0x7fb6b40a50d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4005fd0 tx=0x7fb6c4009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:34.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:34.814+0000 7fb6beffd640 1 -- 172.21.15.98:0/678409443 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb6b8014030 con 0x7fb6c8102a70 2024-05-09T08:48:35.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.027+0000 7fb6cfefc640 1 -- 172.21.15.98:0/678409443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb6c802d410 con 0x7fb6c8102a70 2024-05-09T08:48:35.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.257+0000 7fb6beffd640 1 -- 172.21.15.98:0/678409443 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fb6b809bfa0 con 0x7fb6c8102a70 2024-05-09T08:48:35.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.259+0000 7fb6cfefc640 1 -- 172.21.15.98:0/678409443 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb6b40a2c10 msgr2=0x7fb6b40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.259+0000 7fb6cfefc640 1 --2- 172.21.15.98:0/678409443 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb6b40a2c10 0x7fb6b40a50d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4005fd0 tx=0x7fb6c4009680 comp rx=0 tx=0).stop 2024-05-09T08:48:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.259+0000 7fb6cfefc640 1 -- 172.21.15.98:0/678409443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c8102a70 msgr2=0x7fb6c819e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.259+0000 7fb6cfefc640 1 --2- 172.21.15.98:0/678409443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c8102a70 0x7fb6c819e620 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800c920 tx=0x7fb6b800cde0 comp rx=0 tx=0).stop 2024-05-09T08:48:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.259+0000 7fb6cfefc640 1 -- 172.21.15.98:0/678409443 shutdown_connections 2024-05-09T08:48:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.259+0000 7fb6cfefc640 1 --2- 172.21.15.98:0/678409443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c8105420 0x7fb6c81a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.259+0000 7fb6cfefc640 1 --2- 172.21.15.98:0/678409443 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb6b40a2c10 0x7fb6b40a50d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.259+0000 7fb6cfefc640 1 --2- 172.21.15.98:0/678409443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c8102a70 0x7fb6c819e620 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.259+0000 7fb6cfefc640 1 --2- 172.21.15.98:0/678409443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6c8100100 0x7fb6c819e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.259+0000 7fb6cfefc640 1 -- 172.21.15.98:0/678409443 >> 172.21.15.98:0/678409443 conn(0x7fb6c80fbd20 msgr2=0x7fb6c8100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.259+0000 7fb6cfefc640 1 -- 172.21.15.98:0/678409443 shutdown_connections 2024-05-09T08:48:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.259+0000 7fb6cfefc640 1 -- 172.21.15.98:0/678409443 wait complete. 2024-05-09T08:48:35.270 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-05-09T08:48:35.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-05-09T08:48:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.375+0000 7f4dab3e7640 1 -- 172.21.15.98:0/3327328819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da4101960 msgr2=0x7f4da4101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.375+0000 7f4dab3e7640 1 --2- 172.21.15.98:0/3327328819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da4101960 0x7f4da4101de0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4d94009900 tx=0x7f4d9402a410 comp rx=0 tx=0).stop 2024-05-09T08:48:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.375+0000 7f4dab3e7640 1 -- 172.21.15.98:0/3327328819 shutdown_connections 2024-05-09T08:48:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.375+0000 7f4dab3e7640 1 --2- 172.21.15.98:0/3327328819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4da4102320 0x7f4da4108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.375+0000 7f4dab3e7640 1 --2- 172.21.15.98:0/3327328819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da4101960 0x7f4da4101de0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.375+0000 7f4dab3e7640 1 --2- 172.21.15.98:0/3327328819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4da4100770 0x7f4da4100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.375+0000 7f4dab3e7640 1 -- 172.21.15.98:0/3327328819 >> 172.21.15.98:0/3327328819 conn(0x7f4da40fbce0 msgr2=0x7f4da40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.375+0000 7f4dab3e7640 1 -- 172.21.15.98:0/3327328819 shutdown_connections 2024-05-09T08:48:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.375+0000 7f4dab3e7640 1 -- 172.21.15.98:0/3327328819 wait complete. 2024-05-09T08:48:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.375+0000 7f4dab3e7640 1 Processor -- start 2024-05-09T08:48:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4dab3e7640 1 -- start start 2024-05-09T08:48:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4dab3e7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4da4100770 0x7f4da4196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4dab3e7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da4101960 0x7f4da4196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4dab3e7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4da4102320 0x7f4da419e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4dab3e7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4da410c460 con 0x7f4da4101960 2024-05-09T08:48:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4dab3e7640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4da410c2e0 con 0x7f4da4100770 2024-05-09T08:48:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4dab3e7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4da410c5e0 con 0x7f4da4102320 2024-05-09T08:48:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4da8982640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da4101960 0x7f4da4196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4da8982640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da4101960 0x7f4da4196ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36912/0 (socket says 172.21.15.98:36912) 2024-05-09T08:48:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4da8982640 1 -- 172.21.15.98:0/2147668379 learned_addr learned my addr 172.21.15.98:0/2147668379 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4da9183640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4da4100770 0x7f4da4196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4da9984640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4da4102320 0x7f4da419e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4da8982640 1 -- 172.21.15.98:0/2147668379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4da4102320 msgr2=0x7f4da419e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4da8982640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4da4102320 0x7f4da419e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4da8982640 1 -- 172.21.15.98:0/2147668379 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4da4100770 msgr2=0x7f4da4196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4da8982640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4da4100770 0x7f4da4196ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4da8982640 1 -- 172.21.15.98:0/2147668379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da419e6d0 con 0x7f4da4101960 2024-05-09T08:48:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4da9183640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4da4100770 0x7f4da4196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4da9984640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4da4102320 0x7f4da419e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4da8982640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da4101960 0x7f4da4196ff0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4d94009230 tx=0x7f4d94004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4d9a7fc640 1 -- 172.21.15.98:0/2147668379 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9403f030 con 0x7f4da4101960 2024-05-09T08:48:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4d9a7fc640 1 -- 172.21.15.98:0/2147668379 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d94003ce0 con 0x7f4da4101960 2024-05-09T08:48:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.376+0000 7f4d9a7fc640 1 -- 172.21.15.98:0/2147668379 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d94035070 con 0x7f4da4101960 2024-05-09T08:48:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.377+0000 7f4dab3e7640 1 -- 172.21.15.98:0/2147668379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da419e960 con 0x7f4da4101960 2024-05-09T08:48:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.377+0000 7f4dab3e7640 1 -- 172.21.15.98:0/2147668379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da419ef20 con 0x7f4da4101960 2024-05-09T08:48:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.377+0000 7f4dab3e7640 1 -- 172.21.15.98:0/2147668379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da4107a90 con 0x7f4da4101960 2024-05-09T08:48:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.379+0000 7f4d9a7fc640 1 -- 172.21.15.98:0/2147668379 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4d94042030 con 0x7f4da4101960 2024-05-09T08:48:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.379+0000 7f4d9a7fc640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4d900a2c10 0x7f4d900a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.379+0000 7f4da9183640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4d900a2c10 0x7f4d900a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.379+0000 7f4d9a7fc640 1 -- 172.21.15.98:0/2147668379 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4d940efa20 con 0x7f4da4101960 2024-05-09T08:48:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.380+0000 7f4da9183640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4d900a2c10 0x7f4d900a50d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4da41017c0 tx=0x7f4da0009570 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.384+0000 7f4d9a7fc640 1 -- 172.21.15.98:0/2147668379 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4d940bc740 con 0x7f4da4101960 2024-05-09T08:48:35.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:35 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2974243946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:35.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:35 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1625516539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:35.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:35 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/678409443' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-09T08:48:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:35.601+0000 7f4dab3e7640 1 -- 172.21.15.98:0/2147668379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f4da4062430 con 0x7f4da4101960 2024-05-09T08:48:35.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:35 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2974243946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:35.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:35 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1625516539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:35.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:35 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/678409443' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-09T08:48:35.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:35 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2974243946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:35.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:35 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1625516539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:35.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:35 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/678409443' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-09T08:48:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.259+0000 7f4d9a7fc640 1 -- 172.21.15.98:0/2147668379 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f4d940dc100 con 0x7f4da4101960 2024-05-09T08:48:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.262+0000 7f4dab3e7640 1 -- 172.21.15.98:0/2147668379 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4d900a2c10 msgr2=0x7f4d900a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.262+0000 7f4dab3e7640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4d900a2c10 0x7f4d900a50d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4da41017c0 tx=0x7f4da0009570 comp rx=0 tx=0).stop 2024-05-09T08:48:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.262+0000 7f4dab3e7640 1 -- 172.21.15.98:0/2147668379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da4101960 msgr2=0x7f4da4196ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.262+0000 7f4dab3e7640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da4101960 0x7f4da4196ff0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4d94009230 tx=0x7f4d94004430 comp rx=0 tx=0).stop 2024-05-09T08:48:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.262+0000 7f4dab3e7640 1 -- 172.21.15.98:0/2147668379 shutdown_connections 2024-05-09T08:48:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.262+0000 7f4dab3e7640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4da4102320 0x7f4da419e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.262+0000 7f4dab3e7640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4d900a2c10 0x7f4d900a50d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.262+0000 7f4dab3e7640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4da4101960 0x7f4da4196ff0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.262+0000 7f4dab3e7640 1 --2- 172.21.15.98:0/2147668379 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4da4100770 0x7f4da4196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.262+0000 7f4dab3e7640 1 -- 172.21.15.98:0/2147668379 >> 172.21.15.98:0/2147668379 conn(0x7f4da40fbce0 msgr2=0x7f4da4104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.262+0000 7f4dab3e7640 1 -- 172.21.15.98:0/2147668379 shutdown_connections 2024-05-09T08:48:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.262+0000 7f4dab3e7640 1 -- 172.21.15.98:0/2147668379 wait complete. 2024-05-09T08:48:36.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-05-09T08:48:36.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.387+0000 7fcea1258640 1 -- 172.21.15.98:0/1808763838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce9c101880 msgr2=0x7fce9c101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:36.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.387+0000 7fcea1258640 1 --2- 172.21.15.98:0/1808763838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce9c101880 0x7fce9c101ca0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fce8c009900 tx=0x7fce8c02a410 comp rx=0 tx=0).stop 2024-05-09T08:48:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.387+0000 7fcea1258640 1 -- 172.21.15.98:0/1808763838 shutdown_connections 2024-05-09T08:48:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.387+0000 7fcea1258640 1 --2- 172.21.15.98:0/1808763838 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce9c103430 0x7fce9c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.387+0000 7fcea1258640 1 --2- 172.21.15.98:0/1808763838 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce9c102a70 0x7fce9c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.387+0000 7fcea1258640 1 --2- 172.21.15.98:0/1808763838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce9c101880 0x7fce9c101ca0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.387+0000 7fcea1258640 1 -- 172.21.15.98:0/1808763838 >> 172.21.15.98:0/1808763838 conn(0x7fce9c0fce10 msgr2=0x7fce9c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.387+0000 7fcea1258640 1 -- 172.21.15.98:0/1808763838 shutdown_connections 2024-05-09T08:48:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.387+0000 7fcea1258640 1 -- 172.21.15.98:0/1808763838 wait complete. 2024-05-09T08:48:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fcea1258640 1 Processor -- start 2024-05-09T08:48:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fcea1258640 1 -- start start 2024-05-09T08:48:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fcea1258640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce9c101880 0x7fce9c197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fcea1258640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce9c102a70 0x7fce9c198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fcea1258640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce9c103430 0x7fce9c19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:36.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fcea1258640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fce9c10d4f0 con 0x7fce9c103430 2024-05-09T08:48:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fcea1258640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fce9c10d370 con 0x7fce9c101880 2024-05-09T08:48:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fcea1258640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fce9c10d670 con 0x7fce9c102a70 2024-05-09T08:48:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fce9b59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce9c103430 0x7fce9c19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fce9b59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce9c103430 0x7fce9c19f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55236/0 (socket says 172.21.15.98:55236) 2024-05-09T08:48:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fce9b59e640 1 -- 172.21.15.98:0/3229795383 learned_addr learned my addr 172.21.15.98:0/3229795383 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fce9ad9d640 1 --2- 172.21.15.98:0/3229795383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce9c101880 0x7fce9c197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fce9a59c640 1 --2- 172.21.15.98:0/3229795383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce9c102a70 0x7fce9c198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fce9b59e640 1 -- 172.21.15.98:0/3229795383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce9c102a70 msgr2=0x7fce9c198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fce9b59e640 1 --2- 172.21.15.98:0/3229795383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce9c102a70 0x7fce9c198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fce9b59e640 1 -- 172.21.15.98:0/3229795383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce9c101880 msgr2=0x7fce9c197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:36.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fce9b59e640 1 --2- 172.21.15.98:0/3229795383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce9c101880 0x7fce9c197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:36.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.388+0000 7fce9b59e640 1 -- 172.21.15.98:0/3229795383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce9c19f810 con 0x7fce9c103430 2024-05-09T08:48:36.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.389+0000 7fce9b59e640 1 --2- 172.21.15.98:0/3229795383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce9c103430 0x7fce9c19f1a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fce9400cad0 tx=0x7fce9400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:36.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.389+0000 7fce83fff640 1 -- 172.21.15.98:0/3229795383 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9400ae20 con 0x7fce9c103430 2024-05-09T08:48:36.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.389+0000 7fcea1258640 1 -- 172.21.15.98:0/3229795383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce9c19fb00 con 0x7fce9c103430 2024-05-09T08:48:36.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.389+0000 7fce83fff640 1 -- 172.21.15.98:0/3229795383 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce94005020 con 0x7fce9c103430 2024-05-09T08:48:36.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.389+0000 7fce83fff640 1 -- 172.21.15.98:0/3229795383 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9400f6f0 con 0x7fce9c103430 2024-05-09T08:48:36.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.389+0000 7fcea1258640 1 -- 172.21.15.98:0/3229795383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce9c1a0000 con 0x7fce9c103430 2024-05-09T08:48:36.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.390+0000 7fcea1258640 1 -- 172.21.15.98:0/3229795383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce9c108c60 con 0x7fce9c103430 2024-05-09T08:48:36.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.391+0000 7fce83fff640 1 -- 172.21.15.98:0/3229795383 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fce940051d0 con 0x7fce9c103430 2024-05-09T08:48:36.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.391+0000 7fce83fff640 1 --2- 172.21.15.98:0/3229795383 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fce840a2b90 0x7fce840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:36.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.392+0000 7fce9ad9d640 1 --2- 172.21.15.98:0/3229795383 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fce840a2b90 0x7fce840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:36.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.392+0000 7fce83fff640 1 -- 172.21.15.98:0/3229795383 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fce940cf1f0 con 0x7fce9c103430 2024-05-09T08:48:36.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.392+0000 7fce9ad9d640 1 --2- 172.21.15.98:0/3229795383 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fce840a2b90 0x7fce840a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fce8c005ab0 tx=0x7fce8c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.396+0000 7fce83fff640 1 -- 172.21.15.98:0/3229795383 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fce94014030 con 0x7fce9c103430 2024-05-09T08:48:36.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:36 smithi098 ceph-mon[30092]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 265 B/s wr, 0 op/s 2024-05-09T08:48:36.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:36 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/678409443' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-09T08:48:36.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:36 smithi098 ceph-mon[30092]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:36.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:36 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2147668379' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-09T08:48:36.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:36.611+0000 7fcea1258640 1 -- 172.21.15.98:0/3229795383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fce9c02d410 con 0x7fce9c103430 2024-05-09T08:48:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:36 smithi142 ceph-mon[34625]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 265 B/s wr, 0 op/s 2024-05-09T08:48:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:36 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/678409443' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-09T08:48:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:36 smithi142 ceph-mon[34625]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:36 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2147668379' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-09T08:48:36.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:36 smithi105 ceph-mon[34498]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 265 B/s wr, 0 op/s 2024-05-09T08:48:36.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:36 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/678409443' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-09T08:48:36.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:36 smithi105 ceph-mon[34498]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:36.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:36 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2147668379' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-09T08:48:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.272+0000 7fce83fff640 1 -- 172.21.15.98:0/3229795383 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7fce9409b960 con 0x7fce9c103430 2024-05-09T08:48:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.275+0000 7fcea1258640 1 -- 172.21.15.98:0/3229795383 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fce840a2b90 msgr2=0x7fce840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.275+0000 7fcea1258640 1 --2- 172.21.15.98:0/3229795383 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fce840a2b90 0x7fce840a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fce8c005ab0 tx=0x7fce8c000bc0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.275+0000 7fcea1258640 1 -- 172.21.15.98:0/3229795383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce9c103430 msgr2=0x7fce9c19f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.275+0000 7fcea1258640 1 --2- 172.21.15.98:0/3229795383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce9c103430 0x7fce9c19f1a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fce9400cad0 tx=0x7fce9400cf90 comp rx=0 tx=0).stop 2024-05-09T08:48:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.275+0000 7fcea1258640 1 -- 172.21.15.98:0/3229795383 shutdown_connections 2024-05-09T08:48:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.275+0000 7fcea1258640 1 --2- 172.21.15.98:0/3229795383 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fce840a2b90 0x7fce840a5050 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.275+0000 7fcea1258640 1 --2- 172.21.15.98:0/3229795383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce9c103430 0x7fce9c19f1a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.275+0000 7fcea1258640 1 --2- 172.21.15.98:0/3229795383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce9c102a70 0x7fce9c198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.275+0000 7fcea1258640 1 --2- 172.21.15.98:0/3229795383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce9c101880 0x7fce9c197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.275+0000 7fcea1258640 1 -- 172.21.15.98:0/3229795383 >> 172.21.15.98:0/3229795383 conn(0x7fce9c0fce10 msgr2=0x7fce9c105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.275+0000 7fcea1258640 1 -- 172.21.15.98:0/3229795383 shutdown_connections 2024-05-09T08:48:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.275+0000 7fcea1258640 1 -- 172.21.15.98:0/3229795383 wait complete. 2024-05-09T08:48:37.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-05-09T08:48:37.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.402+0000 7fb2758a5640 1 -- 172.21.15.98:0/3291818376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2701075f0 msgr2=0x7fb2701099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:37.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.402+0000 7fb2758a5640 1 --2- 172.21.15.98:0/3291818376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2701075f0 0x7fb2701099e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb268009ca0 tx=0x7fb26802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.402+0000 7fb2758a5640 1 -- 172.21.15.98:0/3291818376 shutdown_connections 2024-05-09T08:48:37.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.402+0000 7fb2758a5640 1 --2- 172.21.15.98:0/3291818376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2701075f0 0x7fb2701099e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.402+0000 7fb2758a5640 1 --2- 172.21.15.98:0/3291818376 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb270104cc0 0x7fb2701070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.402+0000 7fb2758a5640 1 --2- 172.21.15.98:0/3291818376 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb270102390 0x7fb270104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.402+0000 7fb2758a5640 1 -- 172.21.15.98:0/3291818376 >> 172.21.15.98:0/3291818376 conn(0x7fb2700fbd00 msgr2=0x7fb2700fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:37.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.402+0000 7fb2758a5640 1 -- 172.21.15.98:0/3291818376 shutdown_connections 2024-05-09T08:48:37.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.402+0000 7fb2758a5640 1 -- 172.21.15.98:0/3291818376 wait complete. 2024-05-09T08:48:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.402+0000 7fb2758a5640 1 Processor -- start 2024-05-09T08:48:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb2758a5640 1 -- start start 2024-05-09T08:48:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb2758a5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb270102390 0x7fb270197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb2758a5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb270104cc0 0x7fb2701980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb2758a5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2701075f0 0x7fb27019f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb2758a5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb27010d530 con 0x7fb270102390 2024-05-09T08:48:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb2758a5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb27010d3b0 con 0x7fb2701075f0 2024-05-09T08:48:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb2758a5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb27010d6b0 con 0x7fb270104cc0 2024-05-09T08:48:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb26effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb270102390 0x7fb270197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb26effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb270102390 0x7fb270197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55262/0 (socket says 172.21.15.98:55262) 2024-05-09T08:48:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb26effd640 1 -- 172.21.15.98:0/3302286901 learned_addr learned my addr 172.21.15.98:0/3302286901 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb26f7fe640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2701075f0 0x7fb27019f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb26e7fc640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb270104cc0 0x7fb2701980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:37.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb26effd640 1 -- 172.21.15.98:0/3302286901 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb270104cc0 msgr2=0x7fb2701980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:37.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb26effd640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb270104cc0 0x7fb2701980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb26effd640 1 -- 172.21.15.98:0/3302286901 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2701075f0 msgr2=0x7fb27019f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:37.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb26effd640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2701075f0 0x7fb27019f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb26effd640 1 -- 172.21.15.98:0/3302286901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27019f7c0 con 0x7fb270102390 2024-05-09T08:48:37.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb26e7fc640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb270104cc0 0x7fb2701980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:37.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb26effd640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb270102390 0x7fb270197ba0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb26000d770 tx=0x7fb26000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:37.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.403+0000 7fb2748a3640 1 -- 172.21.15.98:0/3302286901 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb260017070 con 0x7fb270102390 2024-05-09T08:48:37.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.404+0000 7fb2748a3640 1 -- 172.21.15.98:0/3302286901 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb260005500 con 0x7fb270102390 2024-05-09T08:48:37.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.404+0000 7fb2758a5640 1 -- 172.21.15.98:0/3302286901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27019fab0 con 0x7fb270102390 2024-05-09T08:48:37.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.404+0000 7fb2748a3640 1 -- 172.21.15.98:0/3302286901 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb260004e00 con 0x7fb270102390 2024-05-09T08:48:37.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.404+0000 7fb2758a5640 1 -- 172.21.15.98:0/3302286901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb27019ffb0 con 0x7fb270102390 2024-05-09T08:48:37.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.405+0000 7fb2758a5640 1 -- 172.21.15.98:0/3302286901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2700fd7f0 con 0x7fb270102390 2024-05-09T08:48:37.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.406+0000 7fb2748a3640 1 -- 172.21.15.98:0/3302286901 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb260004fa0 con 0x7fb270102390 2024-05-09T08:48:37.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.406+0000 7fb2748a3640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb2580a2b90 0x7fb2580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:37.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.407+0000 7fb26e7fc640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb2580a2b90 0x7fb2580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:37.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.407+0000 7fb2748a3640 1 -- 172.21.15.98:0/3302286901 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb2600cf130 con 0x7fb270102390 2024-05-09T08:48:37.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.407+0000 7fb26e7fc640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb2580a2b90 0x7fb2580a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb2701990b0 tx=0x7fb264005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:37.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.410+0000 7fb2748a3640 1 -- 172.21.15.98:0/3302286901 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb26009be50 con 0x7fb270102390 2024-05-09T08:48:37.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:37 smithi098 ceph-mon[30092]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-09T08:48:37.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:37 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2147668379' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-09T08:48:37.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:37 smithi098 ceph-mon[30092]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-09T08:48:37.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:37 smithi098 ceph-mon[30092]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:37.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:37 smithi098 ceph-mon[30092]: fsmap cephfs:2 {0=h=up:active,1=i=up:creating} 10 up:standby 2024-05-09T08:48:37.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:37 smithi098 ceph-mon[30092]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-05-09T08:48:37.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:37 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3229795383' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-09T08:48:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.627+0000 7fb2758a5640 1 -- 172.21.15.98:0/3302286901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fb27002d1c0 con 0x7fb270102390 2024-05-09T08:48:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:37 smithi142 ceph-mon[34625]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-09T08:48:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:37 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2147668379' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-09T08:48:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:37 smithi142 ceph-mon[34625]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-09T08:48:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:37 smithi142 ceph-mon[34625]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:37 smithi142 ceph-mon[34625]: fsmap cephfs:2 {0=h=up:active,1=i=up:creating} 10 up:standby 2024-05-09T08:48:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:37 smithi142 ceph-mon[34625]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-05-09T08:48:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:37 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3229795383' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-09T08:48:37.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.723+0000 7fb2748a3640 1 -- 172.21.15.98:0/3302286901 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7fb26009b750 con 0x7fb270102390 2024-05-09T08:48:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.725+0000 7fb2758a5640 1 -- 172.21.15.98:0/3302286901 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb2580a2b90 msgr2=0x7fb2580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.725+0000 7fb2758a5640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb2580a2b90 0x7fb2580a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb2701990b0 tx=0x7fb264005f90 comp rx=0 tx=0).stop 2024-05-09T08:48:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.725+0000 7fb2758a5640 1 -- 172.21.15.98:0/3302286901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb270102390 msgr2=0x7fb270197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.725+0000 7fb2758a5640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb270102390 0x7fb270197ba0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb26000d770 tx=0x7fb26000dc30 comp rx=0 tx=0).stop 2024-05-09T08:48:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.725+0000 7fb2758a5640 1 -- 172.21.15.98:0/3302286901 shutdown_connections 2024-05-09T08:48:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.725+0000 7fb2758a5640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2701075f0 0x7fb27019f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.725+0000 7fb2758a5640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb270104cc0 0x7fb2701980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.725+0000 7fb2758a5640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb2580a2b90 0x7fb2580a5050 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.725+0000 7fb2758a5640 1 --2- 172.21.15.98:0/3302286901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb270102390 0x7fb270197ba0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.725+0000 7fb2758a5640 1 -- 172.21.15.98:0/3302286901 >> 172.21.15.98:0/3302286901 conn(0x7fb2700fbd00 msgr2=0x7fb270102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.725+0000 7fb2758a5640 1 -- 172.21.15.98:0/3302286901 shutdown_connections 2024-05-09T08:48:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.725+0000 7fb2758a5640 1 -- 172.21.15.98:0/3302286901 wait complete. 2024-05-09T08:48:37.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:37 smithi105 ceph-mon[34498]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-09T08:48:37.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:37 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2147668379' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-09T08:48:37.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:37 smithi105 ceph-mon[34498]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-09T08:48:37.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:37 smithi105 ceph-mon[34498]: fsmap cephfs:1 {0=h=up:active} 11 up:standby 2024-05-09T08:48:37.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:37 smithi105 ceph-mon[34498]: fsmap cephfs:2 {0=h=up:active,1=i=up:creating} 10 up:standby 2024-05-09T08:48:37.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:37 smithi105 ceph-mon[34498]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-05-09T08:48:37.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:37 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3229795383' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-09T08:48:37.738 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-05-09T08:48:37.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-05-09T08:48:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.843+0000 7f4cef0f5640 1 -- 172.21.15.98:0/2106342116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8103430 msgr2=0x7f4ce8109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.843+0000 7f4cef0f5640 1 --2- 172.21.15.98:0/2106342116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8103430 0x7f4ce8109cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009ca0 tx=0x7f4ce002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.843+0000 7f4cef0f5640 1 -- 172.21.15.98:0/2106342116 shutdown_connections 2024-05-09T08:48:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.843+0000 7f4cef0f5640 1 --2- 172.21.15.98:0/2106342116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8103430 0x7f4ce8109cc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.843+0000 7f4cef0f5640 1 --2- 172.21.15.98:0/2106342116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ce8102a70 0x7f4ce8102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.843+0000 7f4cef0f5640 1 --2- 172.21.15.98:0/2106342116 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce8101880 0x7f4ce8101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.843+0000 7f4cef0f5640 1 -- 172.21.15.98:0/2106342116 >> 172.21.15.98:0/2106342116 conn(0x7f4ce80fce10 msgr2=0x7f4ce80ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.843+0000 7f4cef0f5640 1 -- 172.21.15.98:0/2106342116 shutdown_connections 2024-05-09T08:48:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.843+0000 7f4cef0f5640 1 -- 172.21.15.98:0/2106342116 wait complete. 2024-05-09T08:48:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cef0f5640 1 Processor -- start 2024-05-09T08:48:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cef0f5640 1 -- start start 2024-05-09T08:48:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cef0f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8101880 0x7f4ce8197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cef0f5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce8102a70 0x7f4ce8198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cef0f5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ce8103430 0x7f4ce819f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cef0f5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce810d570 con 0x7f4ce8101880 2024-05-09T08:48:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cef0f5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce810d3f0 con 0x7f4ce8103430 2024-05-09T08:48:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cef0f5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce810d6f0 con 0x7f4ce8102a70 2024-05-09T08:48:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cece91640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8101880 0x7f4ce8197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cece91640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8101880 0x7f4ce8197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55284/0 (socket says 172.21.15.98:55284) 2024-05-09T08:48:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cece91640 1 -- 172.21.15.98:0/1362951062 learned_addr learned my addr 172.21.15.98:0/1362951062 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4ced692640 1 --2- 172.21.15.98:0/1362951062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ce8103430 0x7f4ce819f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cdffff640 1 --2- 172.21.15.98:0/1362951062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce8102a70 0x7f4ce8198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cece91640 1 -- 172.21.15.98:0/1362951062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce8102a70 msgr2=0x7f4ce8198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cece91640 1 --2- 172.21.15.98:0/1362951062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce8102a70 0x7f4ce8198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cece91640 1 -- 172.21.15.98:0/1362951062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ce8103430 msgr2=0x7f4ce819f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cece91640 1 --2- 172.21.15.98:0/1362951062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ce8103430 0x7f4ce819f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cece91640 1 -- 172.21.15.98:0/1362951062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce819f840 con 0x7f4ce8101880 2024-05-09T08:48:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cece91640 1 --2- 172.21.15.98:0/1362951062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8101880 0x7f4ce8197c30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4ce400b450 tx=0x7f4ce400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.844+0000 7f4cddffb640 1 -- 172.21.15.98:0/1362951062 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4015020 con 0x7f4ce8101880 2024-05-09T08:48:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.845+0000 7f4cddffb640 1 -- 172.21.15.98:0/1362951062 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ce4011d70 con 0x7f4ce8101880 2024-05-09T08:48:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.845+0000 7f4cef0f5640 1 -- 172.21.15.98:0/1362951062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce819fb30 con 0x7f4ce8101880 2024-05-09T08:48:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.845+0000 7f4cddffb640 1 -- 172.21.15.98:0/1362951062 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce400f340 con 0x7f4ce8101880 2024-05-09T08:48:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.845+0000 7f4cef0f5640 1 -- 172.21.15.98:0/1362951062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce81a00d0 con 0x7f4ce8101880 2024-05-09T08:48:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.846+0000 7f4cef0f5640 1 -- 172.21.15.98:0/1362951062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ce8108ba0 con 0x7f4ce8101880 2024-05-09T08:48:37.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.847+0000 7f4cddffb640 1 -- 172.21.15.98:0/1362951062 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4ce4018020 con 0x7f4ce8101880 2024-05-09T08:48:37.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.847+0000 7f4cddffb640 1 --2- 172.21.15.98:0/1362951062 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4cd00a2c10 0x7f4cd00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:37.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.847+0000 7f4cdffff640 1 --2- 172.21.15.98:0/1362951062 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4cd00a2c10 0x7f4cd00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:37.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.847+0000 7f4cddffb640 1 -- 172.21.15.98:0/1362951062 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4ce40ced30 con 0x7f4ce8101880 2024-05-09T08:48:37.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.848+0000 7f4cdffff640 1 --2- 172.21.15.98:0/1362951062 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4cd00a2c10 0x7f4cd00a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009aa0 tx=0x7f4cd8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:37.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:37.852+0000 7f4cddffb640 1 -- 172.21.15.98:0/1362951062 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4ce401f020 con 0x7f4ce8101880 2024-05-09T08:48:38.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.054+0000 7f4cef0f5640 1 -- 172.21.15.98:0/1362951062 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f4ce8061c70 con 0x7f4cd00a2c10 2024-05-09T08:48:38.338 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:48:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:48:38.059+0000 7f4e6065c700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:48:38.338 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:48:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:48:38.059+0000 7f4e6065c700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:48:38.338 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:48:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:48:38.059+0000 7f4e6065c700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:48:38.338 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:48:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:48:38.059+0000 7f4e6065c700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:48:38.338 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:48:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:48:38.059+0000 7f4e6065c700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:48:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:38 smithi098 ceph-mon[30092]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-09T08:48:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:38 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3229795383' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-09T08:48:38.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:38 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] up:active 2024-05-09T08:48:38.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:38 smithi098 ceph-mon[30092]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-09T08:48:38.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:38 smithi098 ceph-mon[30092]: fsmap cephfs:2 {0=h=up:active,1=i=up:active} 10 up:standby 2024-05-09T08:48:38.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:38 smithi098 ceph-mon[30092]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-05-09T08:48:38.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:38 smithi098 ceph-mon[30092]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-05-09T08:48:38.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:38 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3302286901' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-09T08:48:38.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:38 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3302286901' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-09T08:48:38.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:38 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] up:active 2024-05-09T08:48:38.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:38 smithi098 ceph-mon[30092]: daemon mds.l assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-09T08:48:38.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:38 smithi098 ceph-mon[30092]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:48:38.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:38 smithi098 ceph-mon[30092]: fsmap cephfs:4 {0=h=up:active,1=i=up:active,2=c=up:active,3=l=up:creating} 3 up:standby-replay 5 up:standby 2024-05-09T08:48:38.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:38 smithi098 ceph-mon[30092]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-05-09T08:48:38.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:38 smithi142 ceph-mon[34625]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-09T08:48:38.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:38 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3229795383' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-09T08:48:38.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:38 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] up:active 2024-05-09T08:48:38.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:38 smithi142 ceph-mon[34625]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-09T08:48:38.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:38 smithi142 ceph-mon[34625]: fsmap cephfs:2 {0=h=up:active,1=i=up:active} 10 up:standby 2024-05-09T08:48:38.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:38 smithi142 ceph-mon[34625]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-05-09T08:48:38.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:38 smithi142 ceph-mon[34625]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-05-09T08:48:38.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:38 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3302286901' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-09T08:48:38.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:38 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3302286901' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-09T08:48:38.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:38 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] up:active 2024-05-09T08:48:38.411 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:38 smithi142 ceph-mon[34625]: daemon mds.l assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-09T08:48:38.411 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:38 smithi142 ceph-mon[34625]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:48:38.411 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:38 smithi142 ceph-mon[34625]: fsmap cephfs:4 {0=h=up:active,1=i=up:active,2=c=up:active,3=l=up:creating} 3 up:standby-replay 5 up:standby 2024-05-09T08:48:38.411 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:38 smithi142 ceph-mon[34625]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-05-09T08:48:38.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:38 smithi105 ceph-mon[34498]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-09T08:48:38.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:38 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3229795383' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-09T08:48:38.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:38 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] up:active 2024-05-09T08:48:38.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:38 smithi105 ceph-mon[34498]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-09T08:48:38.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:38 smithi105 ceph-mon[34498]: fsmap cephfs:2 {0=h=up:active,1=i=up:active} 10 up:standby 2024-05-09T08:48:38.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:38 smithi105 ceph-mon[34498]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-05-09T08:48:38.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:38 smithi105 ceph-mon[34498]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-05-09T08:48:38.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:38 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3302286901' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-09T08:48:38.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:38 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3302286901' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-09T08:48:38.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:38 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] up:active 2024-05-09T08:48:38.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:38 smithi105 ceph-mon[34498]: daemon mds.l assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-09T08:48:38.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:38 smithi105 ceph-mon[34498]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:48:38.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:38 smithi105 ceph-mon[34498]: fsmap cephfs:4 {0=h=up:active,1=i=up:active,2=c=up:active,3=l=up:creating} 3 up:standby-replay 5 up:standby 2024-05-09T08:48:38.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:38 smithi105 ceph-mon[34498]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-05-09T08:48:38.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.761+0000 7f4cddffb640 1 -- 172.21.15.98:0/1362951062 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4ce8061c70 con 0x7f4cd00a2c10 2024-05-09T08:48:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.763+0000 7f4cef0f5640 1 -- 172.21.15.98:0/1362951062 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4cd00a2c10 msgr2=0x7f4cd00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.763+0000 7f4cef0f5640 1 --2- 172.21.15.98:0/1362951062 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4cd00a2c10 0x7f4cd00a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009aa0 tx=0x7f4cd8005f90 comp rx=0 tx=0).stop 2024-05-09T08:48:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.763+0000 7f4cef0f5640 1 -- 172.21.15.98:0/1362951062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8101880 msgr2=0x7f4ce8197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.763+0000 7f4cef0f5640 1 --2- 172.21.15.98:0/1362951062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8101880 0x7f4ce8197c30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4ce400b450 tx=0x7f4ce400b910 comp rx=0 tx=0).stop 2024-05-09T08:48:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.763+0000 7f4cef0f5640 1 -- 172.21.15.98:0/1362951062 shutdown_connections 2024-05-09T08:48:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.763+0000 7f4cef0f5640 1 --2- 172.21.15.98:0/1362951062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ce8103430 0x7f4ce819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.763+0000 7f4cef0f5640 1 --2- 172.21.15.98:0/1362951062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce8102a70 0x7f4ce8198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.763+0000 7f4cef0f5640 1 --2- 172.21.15.98:0/1362951062 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f4cd00a2c10 0x7f4cd00a50d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.763+0000 7f4cef0f5640 1 --2- 172.21.15.98:0/1362951062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ce8101880 0x7f4ce8197c30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:38.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.763+0000 7f4cef0f5640 1 -- 172.21.15.98:0/1362951062 >> 172.21.15.98:0/1362951062 conn(0x7f4ce80fce10 msgr2=0x7f4ce8105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:38.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.763+0000 7f4cef0f5640 1 -- 172.21.15.98:0/1362951062 shutdown_connections 2024-05-09T08:48:38.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.763+0000 7f4cef0f5640 1 -- 172.21.15.98:0/1362951062 wait complete. 2024-05-09T08:48:38.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-09T08:48:38.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.882+0000 7f1388fe3640 1 -- 172.21.15.98:0/3275551378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1384101980 msgr2=0x7f1384101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:38.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.882+0000 7f1388fe3640 1 --2- 172.21.15.98:0/3275551378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1384101980 0x7f1384101e00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f136c009900 tx=0x7f136c02a410 comp rx=0 tx=0).stop 2024-05-09T08:48:38.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.882+0000 7f1388fe3640 1 -- 172.21.15.98:0/3275551378 shutdown_connections 2024-05-09T08:48:38.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.882+0000 7f1388fe3640 1 --2- 172.21.15.98:0/3275551378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1384102340 0x7f1384108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:38.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.882+0000 7f1388fe3640 1 --2- 172.21.15.98:0/3275551378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1384101980 0x7f1384101e00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:38.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.882+0000 7f1388fe3640 1 --2- 172.21.15.98:0/3275551378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1384100790 0x7f1384100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:38.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.882+0000 7f1388fe3640 1 -- 172.21.15.98:0/3275551378 >> 172.21.15.98:0/3275551378 conn(0x7f13840fbd20 msgr2=0x7f13840fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.882+0000 7f1388fe3640 1 -- 172.21.15.98:0/3275551378 shutdown_connections 2024-05-09T08:48:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.882+0000 7f1388fe3640 1 -- 172.21.15.98:0/3275551378 wait complete. 2024-05-09T08:48:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.882+0000 7f1388fe3640 1 Processor -- start 2024-05-09T08:48:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1388fe3640 1 -- start start 2024-05-09T08:48:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1388fe3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1384100790 0x7f138406aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1388fe3640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1384101980 0x7f138419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1388fe3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1384102340 0x7f138419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1388fe3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f138410c470 con 0x7f1384102340 2024-05-09T08:48:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1388fe3640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f138410c2f0 con 0x7f1384100790 2024-05-09T08:48:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1388fe3640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f138410c5f0 con 0x7f1384101980 2024-05-09T08:48:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1382d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1384102340 0x7f138419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f138259c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1384100790 0x7f138406aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1382d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1384102340 0x7f138419ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55314/0 (socket says 172.21.15.98:55314) 2024-05-09T08:48:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f138259c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1384100790 0x7f138406aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:48754/0 (socket says 172.21.15.98:48754) 2024-05-09T08:48:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1382d9d640 1 -- 172.21.15.98:0/2781490603 learned_addr learned my addr 172.21.15.98:0/2781490603 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1379d9b640 1 --2- 172.21.15.98:0/2781490603 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1384101980 0x7f138419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1382d9d640 1 -- 172.21.15.98:0/2781490603 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1384101980 msgr2=0x7f138419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1382d9d640 1 --2- 172.21.15.98:0/2781490603 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1384101980 0x7f138419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1382d9d640 1 -- 172.21.15.98:0/2781490603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1384100790 msgr2=0x7f138406aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1382d9d640 1 --2- 172.21.15.98:0/2781490603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1384100790 0x7f138406aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1382d9d640 1 -- 172.21.15.98:0/2781490603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138419f830 con 0x7f1384102340 2024-05-09T08:48:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f138259c640 1 --2- 172.21.15.98:0/2781490603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1384100790 0x7f138406aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:48:38.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.883+0000 7f1382d9d640 1 --2- 172.21.15.98:0/2781490603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1384102340 0x7f138419ef70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f137c00b970 tx=0x7f137c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:38.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.884+0000 7f137bfff640 1 -- 172.21.15.98:0/2781490603 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c019070 con 0x7f1384102340 2024-05-09T08:48:38.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.884+0000 7f137bfff640 1 -- 172.21.15.98:0/2781490603 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f137c00cb40 con 0x7f1384102340 2024-05-09T08:48:38.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.884+0000 7f1388fe3640 1 -- 172.21.15.98:0/2781490603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f138419fb20 con 0x7f1384102340 2024-05-09T08:48:38.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.884+0000 7f137bfff640 1 -- 172.21.15.98:0/2781490603 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c007650 con 0x7f1384102340 2024-05-09T08:48:38.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.884+0000 7f1388fe3640 1 -- 172.21.15.98:0/2781490603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f138419fff0 con 0x7f1384102340 2024-05-09T08:48:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.885+0000 7f1388fe3640 1 -- 172.21.15.98:0/2781490603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1384107b80 con 0x7f1384102340 2024-05-09T08:48:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.886+0000 7f137bfff640 1 -- 172.21.15.98:0/2781490603 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f137c01c070 con 0x7f1384102340 2024-05-09T08:48:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.887+0000 7f137bfff640 1 --2- 172.21.15.98:0/2781490603 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f13700a2b90 0x7f13700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.887+0000 7f138259c640 1 --2- 172.21.15.98:0/2781490603 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f13700a2b90 0x7f13700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.887+0000 7f137bfff640 1 -- 172.21.15.98:0/2781490603 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f137c0ce860 con 0x7f1384102340 2024-05-09T08:48:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.887+0000 7f138259c640 1 --2- 172.21.15.98:0/2781490603 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f13700a2b90 0x7f13700a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f13841017e0 tx=0x7f1374005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:38.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:38.891+0000 7f137bfff640 1 -- 172.21.15.98:0/2781490603 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f137c09af30 con 0x7f1384102340 2024-05-09T08:48:39.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.097+0000 7f1388fe3640 1 -- 172.21.15.98:0/2781490603 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f1384061c70 con 0x7f13700a2b90 2024-05-09T08:48:39.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.122+0000 7f137bfff640 1 -- 172.21.15.98:0/2781490603 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1384061c70 con 0x7f13700a2b90 2024-05-09T08:48:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.124+0000 7f1388fe3640 1 -- 172.21.15.98:0/2781490603 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f13700a2b90 msgr2=0x7f13700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.124+0000 7f1388fe3640 1 --2- 172.21.15.98:0/2781490603 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f13700a2b90 0x7f13700a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f13841017e0 tx=0x7f1374005f90 comp rx=0 tx=0).stop 2024-05-09T08:48:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.124+0000 7f1388fe3640 1 -- 172.21.15.98:0/2781490603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1384102340 msgr2=0x7f138419ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.124+0000 7f1388fe3640 1 --2- 172.21.15.98:0/2781490603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1384102340 0x7f138419ef70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f137c00b970 tx=0x7f137c00be30 comp rx=0 tx=0).stop 2024-05-09T08:48:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.124+0000 7f1388fe3640 1 -- 172.21.15.98:0/2781490603 shutdown_connections 2024-05-09T08:48:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.124+0000 7f1388fe3640 1 --2- 172.21.15.98:0/2781490603 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f13700a2b90 0x7f13700a5050 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.124+0000 7f1388fe3640 1 --2- 172.21.15.98:0/2781490603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1384102340 0x7f138419ef70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.124+0000 7f1388fe3640 1 --2- 172.21.15.98:0/2781490603 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1384101980 0x7f138419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.124+0000 7f1388fe3640 1 --2- 172.21.15.98:0/2781490603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1384100790 0x7f138406aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.124+0000 7f1388fe3640 1 -- 172.21.15.98:0/2781490603 >> 172.21.15.98:0/2781490603 conn(0x7f13840fbd20 msgr2=0x7f1384104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:39.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.124+0000 7f1388fe3640 1 -- 172.21.15.98:0/2781490603 shutdown_connections 2024-05-09T08:48:39.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.124+0000 7f1388fe3640 1 -- 172.21.15.98:0/2781490603 wait complete. 2024-05-09T08:48:39.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-09T08:48:39.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.242+0000 7f1bfd990640 1 -- 172.21.15.98:0/2207879291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bf8102320 msgr2=0x7f1bf8108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.242+0000 7f1bfd990640 1 --2- 172.21.15.98:0/2207879291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bf8102320 0x7f1bf8108bb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0009ca0 tx=0x7f1bf002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.242+0000 7f1bfd990640 1 -- 172.21.15.98:0/2207879291 shutdown_connections 2024-05-09T08:48:39.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.242+0000 7f1bfd990640 1 --2- 172.21.15.98:0/2207879291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bf8102320 0x7f1bf8108bb0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.242+0000 7f1bfd990640 1 --2- 172.21.15.98:0/2207879291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bf8101960 0x7f1bf8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.242+0000 7f1bfd990640 1 --2- 172.21.15.98:0/2207879291 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bf8100770 0x7f1bf8100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.242+0000 7f1bfd990640 1 -- 172.21.15.98:0/2207879291 >> 172.21.15.98:0/2207879291 conn(0x7f1bf80fbce0 msgr2=0x7f1bf80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:39.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.242+0000 7f1bfd990640 1 -- 172.21.15.98:0/2207879291 shutdown_connections 2024-05-09T08:48:39.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.242+0000 7f1bfd990640 1 -- 172.21.15.98:0/2207879291 wait complete. 2024-05-09T08:48:39.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.242+0000 7f1bfd990640 1 Processor -- start 2024-05-09T08:48:39.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.242+0000 7f1bfd990640 1 -- start start 2024-05-09T08:48:39.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1bfd990640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bf8100770 0x7f1bf8104700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:39.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1bfd990640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bf8101960 0x7f1bf8102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:39.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1bfd990640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bf8102320 0x7f1bf8103280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:39.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1bfd990640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf810c3f0 con 0x7f1bf8101960 2024-05-09T08:48:39.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1bfd990640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf810c270 con 0x7f1bf8100770 2024-05-09T08:48:39.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1bfd990640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf810c570 con 0x7f1bf8102320 2024-05-09T08:48:39.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1beffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bf8101960 0x7f1bf8102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:39.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1beffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bf8101960 0x7f1bf8102d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55318/0 (socket says 172.21.15.98:55318) 2024-05-09T08:48:39.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1beffff640 1 -- 172.21.15.98:0/999271420 learned_addr learned my addr 172.21.15.98:0/999271420 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:39.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1bf6ffd640 1 --2- 172.21.15.98:0/999271420 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bf8100770 0x7f1bf8104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:39.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1beffff640 1 -- 172.21.15.98:0/999271420 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bf8102320 msgr2=0x7f1bf8103280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1beffff640 1 --2- 172.21.15.98:0/999271420 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bf8102320 0x7f1bf8103280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1beffff640 1 -- 172.21.15.98:0/999271420 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bf8100770 msgr2=0x7f1bf8104700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1beffff640 1 --2- 172.21.15.98:0/999271420 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bf8100770 0x7f1bf8104700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1beffff640 1 -- 172.21.15.98:0/999271420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf8103b10 con 0x7f1bf8101960 2024-05-09T08:48:39.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1beffff640 1 --2- 172.21.15.98:0/999271420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bf8101960 0x7f1bf8102d40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1be000d950 tx=0x7f1be000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:39.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1bfc98e640 1 -- 172.21.15.98:0/999271420 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be0017030 con 0x7f1bf8101960 2024-05-09T08:48:39.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1bfd990640 1 -- 172.21.15.98:0/999271420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf81a1c70 con 0x7f1bf8101960 2024-05-09T08:48:39.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1bfc98e640 1 -- 172.21.15.98:0/999271420 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1be0004020 con 0x7f1bf8101960 2024-05-09T08:48:39.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1bfc98e640 1 -- 172.21.15.98:0/999271420 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be00047a0 con 0x7f1bf8101960 2024-05-09T08:48:39.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.243+0000 7f1bfd990640 1 -- 172.21.15.98:0/999271420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf81a21c0 con 0x7f1bf8101960 2024-05-09T08:48:39.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.244+0000 7f1bfd990640 1 -- 172.21.15.98:0/999271420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bf81a23f0 con 0x7f1bf8101960 2024-05-09T08:48:39.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.245+0000 7f1bfc98e640 1 -- 172.21.15.98:0/999271420 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1be00041d0 con 0x7f1bf8101960 2024-05-09T08:48:39.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.246+0000 7f1bfc98e640 1 --2- 172.21.15.98:0/999271420 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1be40a2b90 0x7f1be40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:39.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.246+0000 7f1bf6ffd640 1 --2- 172.21.15.98:0/999271420 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1be40a2b90 0x7f1be40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:39.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.246+0000 7f1bfc98e640 1 -- 172.21.15.98:0/999271420 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f1be00cecf0 con 0x7f1bf8101960 2024-05-09T08:48:39.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.246+0000 7f1bf6ffd640 1 --2- 172.21.15.98:0/999271420 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1be40a2b90 0x7f1be40a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1be80058c0 tx=0x7f1be800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:39.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.250+0000 7f1bfc98e640 1 -- 172.21.15.98:0/999271420 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1be00d3070 con 0x7f1bf8101960 2024-05-09T08:48:39.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.454+0000 7f1bfd990640 1 -- 172.21.15.98:0/999271420 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f1bf80fde90 con 0x7f1be40a2b90 2024-05-09T08:48:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.460+0000 7f1bfc98e640 1 -- 172.21.15.98:0/999271420 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f1bf80fde90 con 0x7f1be40a2b90 2024-05-09T08:48:39.459 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/qa/sv_0/f833bb58-6048-4e11-b4cf-e3f1b017d1cf 2024-05-09T08:48:39.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.462+0000 7f1bfd990640 1 -- 172.21.15.98:0/999271420 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1be40a2b90 msgr2=0x7f1be40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.462+0000 7f1bfd990640 1 --2- 172.21.15.98:0/999271420 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1be40a2b90 0x7f1be40a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1be80058c0 tx=0x7f1be800a2d0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.462+0000 7f1bfd990640 1 -- 172.21.15.98:0/999271420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bf8101960 msgr2=0x7f1bf8102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.462+0000 7f1bfd990640 1 --2- 172.21.15.98:0/999271420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bf8101960 0x7f1bf8102d40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1be000d950 tx=0x7f1be000de10 comp rx=0 tx=0).stop 2024-05-09T08:48:39.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.462+0000 7f1bfd990640 1 -- 172.21.15.98:0/999271420 shutdown_connections 2024-05-09T08:48:39.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.462+0000 7f1bfd990640 1 --2- 172.21.15.98:0/999271420 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bf8102320 0x7f1bf8103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.462+0000 7f1bfd990640 1 --2- 172.21.15.98:0/999271420 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1be40a2b90 0x7f1be40a5050 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.462+0000 7f1bfd990640 1 --2- 172.21.15.98:0/999271420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bf8101960 0x7f1bf8102d40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.462+0000 7f1bfd990640 1 --2- 172.21.15.98:0/999271420 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bf8100770 0x7f1bf8104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.462+0000 7f1bfd990640 1 -- 172.21.15.98:0/999271420 >> 172.21.15.98:0/999271420 conn(0x7f1bf80fbce0 msgr2=0x7f1bf80fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:39.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.462+0000 7f1bfd990640 1 -- 172.21.15.98:0/999271420 shutdown_connections 2024-05-09T08:48:39.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.462+0000 7f1bfd990640 1 -- 172.21.15.98:0/999271420 wait complete. 2024-05-09T08:48:39.471 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:39 smithi098 ceph-mon[30092]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:39.471 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:39 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] up:active 2024-05-09T08:48:39.471 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:39 smithi098 ceph-mon[30092]: daemon mds.b assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-09T08:48:39.471 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:39 smithi098 ceph-mon[30092]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-09T08:48:39.472 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:39 smithi098 ceph-mon[30092]: Cluster is now healthy 2024-05-09T08:48:39.472 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:39 smithi098 ceph-mon[30092]: fsmap cephfs:4 {0=h=up:active,1=i=up:active,2=c=up:active,3=l=up:active} 3 up:standby-replay 5 up:standby 2024-05-09T08:48:39.472 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:39 smithi098 ceph-mon[30092]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-05-09T08:48:39.472 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:39 smithi098 ceph-mon[30092]: daemon mds.b is now active in filesystem cephfs as rank 4 2024-05-09T08:48:39.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-09T08:48:39.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.580+0000 7fc9b6d42640 1 -- 172.21.15.98:0/2447051841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0100770 msgr2=0x7fc9b0100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.580+0000 7fc9b6d42640 1 --2- 172.21.15.98:0/2447051841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0100770 0x7fc9b0100b90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009900 tx=0x7fc9a002a410 comp rx=0 tx=0).stop 2024-05-09T08:48:39.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.580+0000 7fc9b6d42640 1 -- 172.21.15.98:0/2447051841 shutdown_connections 2024-05-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.580+0000 7fc9b6d42640 1 --2- 172.21.15.98:0/2447051841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9b0102320 0x7fc9b0108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.580+0000 7fc9b6d42640 1 --2- 172.21.15.98:0/2447051841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9b0101960 0x7fc9b0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.580+0000 7fc9b6d42640 1 --2- 172.21.15.98:0/2447051841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0100770 0x7fc9b0100b90 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.580+0000 7fc9b6d42640 1 -- 172.21.15.98:0/2447051841 >> 172.21.15.98:0/2447051841 conn(0x7fc9b00fbd20 msgr2=0x7fc9b00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.580+0000 7fc9b6d42640 1 -- 172.21.15.98:0/2447051841 shutdown_connections 2024-05-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.580+0000 7fc9b6d42640 1 -- 172.21.15.98:0/2447051841 wait complete. 2024-05-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.580+0000 7fc9b6d42640 1 Processor -- start 2024-05-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.580+0000 7fc9b6d42640 1 -- start start 2024-05-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b6d42640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0100770 0x7fc9b0196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b6d42640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9b0101960 0x7fc9b0197060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b6d42640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9b0102320 0x7fc9b019e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b6d42640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b010c440 con 0x7fc9b0100770 2024-05-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b6d42640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b010c2c0 con 0x7fc9b0101960 2024-05-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b6d42640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b010c5c0 con 0x7fc9b0102320 2024-05-09T08:48:39.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b4ade640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0100770 0x7fc9b0196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:39.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b4ade640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0100770 0x7fc9b0196b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55336/0 (socket says 172.21.15.98:55336) 2024-05-09T08:48:39.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b4ade640 1 -- 172.21.15.98:0/3081844502 learned_addr learned my addr 172.21.15.98:0/3081844502 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:39.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9affff640 1 --2- 172.21.15.98:0/3081844502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9b0101960 0x7fc9b0197060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:39.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b52df640 1 --2- 172.21.15.98:0/3081844502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9b0102320 0x7fc9b019e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:39.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b4ade640 1 -- 172.21.15.98:0/3081844502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9b0102320 msgr2=0x7fc9b019e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b4ade640 1 --2- 172.21.15.98:0/3081844502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9b0102320 0x7fc9b019e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b4ade640 1 -- 172.21.15.98:0/3081844502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9b0101960 msgr2=0x7fc9b0197060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b4ade640 1 --2- 172.21.15.98:0/3081844502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9b0101960 0x7fc9b0197060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b4ade640 1 -- 172.21.15.98:0/3081844502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b019e740 con 0x7fc9b0100770 2024-05-09T08:48:39.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b4ade640 1 --2- 172.21.15.98:0/3081844502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0100770 0x7fc9b0196b20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0031610 tx=0x7fc9a0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:39.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9adffb640 1 -- 172.21.15.98:0/3081844502 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a003f030 con 0x7fc9b0100770 2024-05-09T08:48:39.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9b6d42640 1 -- 172.21.15.98:0/3081844502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b019e9d0 con 0x7fc9b0100770 2024-05-09T08:48:39.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9adffb640 1 -- 172.21.15.98:0/3081844502 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9a0003ad0 con 0x7fc9b0100770 2024-05-09T08:48:39.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.581+0000 7fc9adffb640 1 -- 172.21.15.98:0/3081844502 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0035070 con 0x7fc9b0100770 2024-05-09T08:48:39.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.582+0000 7fc9b6d42640 1 -- 172.21.15.98:0/3081844502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b019ef40 con 0x7fc9b0100770 2024-05-09T08:48:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.583+0000 7fc9b6d42640 1 -- 172.21.15.98:0/3081844502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b0107b10 con 0x7fc9b0100770 2024-05-09T08:48:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.584+0000 7fc9adffb640 1 -- 172.21.15.98:0/3081844502 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc9a0042030 con 0x7fc9b0100770 2024-05-09T08:48:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.585+0000 7fc9adffb640 1 --2- 172.21.15.98:0/3081844502 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc9980a2b90 0x7fc9980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.585+0000 7fc9affff640 1 --2- 172.21.15.98:0/3081844502 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc9980a2b90 0x7fc9980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.585+0000 7fc9adffb640 1 -- 172.21.15.98:0/3081844502 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc9a00ef950 con 0x7fc9b0100770 2024-05-09T08:48:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.585+0000 7fc9affff640 1 --2- 172.21.15.98:0/3081844502 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc9980a2b90 0x7fc9980a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0198030 tx=0x7fc9a4009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:39.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.589+0000 7fc9adffb640 1 -- 172.21.15.98:0/3081844502 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc9a00bc670 con 0x7fc9b0100770 2024-05-09T08:48:39.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:39 smithi142 ceph-mon[34625]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:39.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:39 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] up:active 2024-05-09T08:48:39.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:39 smithi142 ceph-mon[34625]: daemon mds.b assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-09T08:48:39.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:39 smithi142 ceph-mon[34625]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-09T08:48:39.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:39 smithi142 ceph-mon[34625]: Cluster is now healthy 2024-05-09T08:48:39.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:39 smithi142 ceph-mon[34625]: fsmap cephfs:4 {0=h=up:active,1=i=up:active,2=c=up:active,3=l=up:active} 3 up:standby-replay 5 up:standby 2024-05-09T08:48:39.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:39 smithi142 ceph-mon[34625]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-05-09T08:48:39.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:39 smithi142 ceph-mon[34625]: daemon mds.b is now active in filesystem cephfs as rank 4 2024-05-09T08:48:39.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:39 smithi105 ceph-mon[34498]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:39 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] up:active 2024-05-09T08:48:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:39 smithi105 ceph-mon[34498]: daemon mds.b assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-09T08:48:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:39 smithi105 ceph-mon[34498]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-09T08:48:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:39 smithi105 ceph-mon[34498]: Cluster is now healthy 2024-05-09T08:48:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:39 smithi105 ceph-mon[34498]: fsmap cephfs:4 {0=h=up:active,1=i=up:active,2=c=up:active,3=l=up:active} 3 up:standby-replay 5 up:standby 2024-05-09T08:48:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:39 smithi105 ceph-mon[34498]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-05-09T08:48:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:39 smithi105 ceph-mon[34498]: daemon mds.b is now active in filesystem cephfs as rank 4 2024-05-09T08:48:39.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.756+0000 7fc9adffb640 1 -- 172.21.15.98:0/3081844502 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc9a00b85e0 con 0x7fc9b0100770 2024-05-09T08:48:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.799+0000 7fc9b6d42640 1 -- 172.21.15.98:0/3081844502 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc9b0061d00 con 0x7fc9980a2b90 2024-05-09T08:48:39.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.821+0000 7fc9adffb640 1 -- 172.21.15.98:0/3081844502 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc9b0061d00 con 0x7fc9980a2b90 2024-05-09T08:48:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.823+0000 7fc9b6d42640 1 -- 172.21.15.98:0/3081844502 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc9980a2b90 msgr2=0x7fc9980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.823+0000 7fc9b6d42640 1 --2- 172.21.15.98:0/3081844502 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc9980a2b90 0x7fc9980a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0198030 tx=0x7fc9a4009110 comp rx=0 tx=0).stop 2024-05-09T08:48:39.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.823+0000 7fc9b6d42640 1 -- 172.21.15.98:0/3081844502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0100770 msgr2=0x7fc9b0196b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.823+0000 7fc9b6d42640 1 --2- 172.21.15.98:0/3081844502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0100770 0x7fc9b0196b20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0031610 tx=0x7fc9a0004270 comp rx=0 tx=0).stop 2024-05-09T08:48:39.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.823+0000 7fc9b6d42640 1 -- 172.21.15.98:0/3081844502 shutdown_connections 2024-05-09T08:48:39.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.823+0000 7fc9b6d42640 1 --2- 172.21.15.98:0/3081844502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9b0102320 0x7fc9b019e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.823+0000 7fc9b6d42640 1 --2- 172.21.15.98:0/3081844502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9b0101960 0x7fc9b0197060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.823+0000 7fc9b6d42640 1 --2- 172.21.15.98:0/3081844502 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc9980a2b90 0x7fc9980a5050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.823+0000 7fc9b6d42640 1 --2- 172.21.15.98:0/3081844502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0100770 0x7fc9b0196b20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.823+0000 7fc9b6d42640 1 -- 172.21.15.98:0/3081844502 >> 172.21.15.98:0/3081844502 conn(0x7fc9b00fbd20 msgr2=0x7fc9b0104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:39.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.823+0000 7fc9b6d42640 1 -- 172.21.15.98:0/3081844502 shutdown_connections 2024-05-09T08:48:39.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.823+0000 7fc9b6d42640 1 -- 172.21.15.98:0/3081844502 wait complete. 2024-05-09T08:48:39.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-09T08:48:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.940+0000 7f1cdcabd640 1 -- 172.21.15.98:0/1805093093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cd8102350 msgr2=0x7f1cd8104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.940+0000 7f1cdcabd640 1 --2- 172.21.15.98:0/1805093093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cd8102350 0x7f1cd8104740 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009860 tx=0x7f1cc802a340 comp rx=0 tx=0).stop 2024-05-09T08:48:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.940+0000 7f1cdcabd640 1 -- 172.21.15.98:0/1805093093 shutdown_connections 2024-05-09T08:48:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.940+0000 7f1cdcabd640 1 --2- 172.21.15.98:0/1805093093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cd81075b0 0x7f1cd8109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.940+0000 7f1cdcabd640 1 --2- 172.21.15.98:0/1805093093 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1cd8104c80 0x7f1cd8107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.940+0000 7f1cdcabd640 1 --2- 172.21.15.98:0/1805093093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cd8102350 0x7f1cd8104740 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.940+0000 7f1cdcabd640 1 -- 172.21.15.98:0/1805093093 >> 172.21.15.98:0/1805093093 conn(0x7f1cd80fbd80 msgr2=0x7f1cd80fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.940+0000 7f1cdcabd640 1 -- 172.21.15.98:0/1805093093 shutdown_connections 2024-05-09T08:48:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.940+0000 7f1cdcabd640 1 -- 172.21.15.98:0/1805093093 wait complete. 2024-05-09T08:48:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cdcabd640 1 Processor -- start 2024-05-09T08:48:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cdcabd640 1 -- start start 2024-05-09T08:48:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cdcabd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cd8102350 0x7f1cd8197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cdcabd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1cd8104c80 0x7f1cd81980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cdcabd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cd81075b0 0x7f1cd819f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cdcabd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd810d4e0 con 0x7f1cd8102350 2024-05-09T08:48:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cdcabd640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd810d360 con 0x7f1cd81075b0 2024-05-09T08:48:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cdcabd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd810d660 con 0x7f1cd8104c80 2024-05-09T08:48:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cd659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cd8102350 0x7f1cd8197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cd659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cd8102350 0x7f1cd8197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55356/0 (socket says 172.21.15.98:55356) 2024-05-09T08:48:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cd659c640 1 -- 172.21.15.98:0/3472660234 learned_addr learned my addr 172.21.15.98:0/3472660234 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1ccffff640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1cd8104c80 0x7f1cd81980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cd6d9d640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cd81075b0 0x7f1cd819f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cd659c640 1 -- 172.21.15.98:0/3472660234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1cd8104c80 msgr2=0x7f1cd81980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cd659c640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1cd8104c80 0x7f1cd81980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cd659c640 1 -- 172.21.15.98:0/3472660234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cd81075b0 msgr2=0x7f1cd819f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cd659c640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cd81075b0 0x7f1cd819f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cd659c640 1 -- 172.21.15.98:0/3472660234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd819f7d0 con 0x7f1cd8102350 2024-05-09T08:48:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.941+0000 7f1cd6d9d640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cd81075b0 0x7f1cd819f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.942+0000 7f1cd659c640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cd8102350 0x7f1cd8197bb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8005940 tx=0x7f1cc8033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.942+0000 7f1ccf7fe640 1 -- 172.21.15.98:0/3472660234 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc803f020 con 0x7f1cd8102350 2024-05-09T08:48:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.942+0000 7f1cdcabd640 1 -- 172.21.15.98:0/3472660234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cd819fa60 con 0x7f1cd8102350 2024-05-09T08:48:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.942+0000 7f1ccf7fe640 1 -- 172.21.15.98:0/3472660234 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cc8030850 con 0x7f1cd8102350 2024-05-09T08:48:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.942+0000 7f1ccf7fe640 1 -- 172.21.15.98:0/3472660234 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8037040 con 0x7f1cd8102350 2024-05-09T08:48:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.942+0000 7f1cdcabd640 1 -- 172.21.15.98:0/3472660234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cd819ffa0 con 0x7f1cd8102350 2024-05-09T08:48:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.943+0000 7f1cdcabd640 1 -- 172.21.15.98:0/3472660234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cd80fd870 con 0x7f1cd8102350 2024-05-09T08:48:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.944+0000 7f1ccf7fe640 1 -- 172.21.15.98:0/3472660234 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1cc8042020 con 0x7f1cd8102350 2024-05-09T08:48:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.945+0000 7f1ccf7fe640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1cc40a2c20 0x7f1cc40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.945+0000 7f1ccffff640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1cc40a2c20 0x7f1cc40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.945+0000 7f1ccf7fe640 1 -- 172.21.15.98:0/3472660234 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f1cc80efb40 con 0x7f1cd8102350 2024-05-09T08:48:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.945+0000 7f1ccffff640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1cc40a2c20 0x7f1cc40a50e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1cd81990c0 tx=0x7f1cc000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:39.948+0000 7f1ccf7fe640 1 -- 172.21.15.98:0/3472660234 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1cc8049020 con 0x7f1cd8102350 2024-05-09T08:48:40.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.152+0000 7f1cdcabd640 1 -- 172.21.15.98:0/3472660234 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f1cd8061c70 con 0x7f1cc40a2c20 2024-05-09T08:48:40.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.157+0000 7f1ccf7fe640 1 -- 172.21.15.98:0/3472660234 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f1cd8061c70 con 0x7f1cc40a2c20 2024-05-09T08:48:40.156 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/qa/sv_1/04326e37-0f86-4776-bf48-99735d3687da 2024-05-09T08:48:40.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.159+0000 7f1cdcabd640 1 -- 172.21.15.98:0/3472660234 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1cc40a2c20 msgr2=0x7f1cc40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.159+0000 7f1cdcabd640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1cc40a2c20 0x7f1cc40a50e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1cd81990c0 tx=0x7f1cc000b390 comp rx=0 tx=0).stop 2024-05-09T08:48:40.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.159+0000 7f1cdcabd640 1 -- 172.21.15.98:0/3472660234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cd8102350 msgr2=0x7f1cd8197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.159+0000 7f1cdcabd640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cd8102350 0x7f1cd8197bb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8005940 tx=0x7f1cc8033d90 comp rx=0 tx=0).stop 2024-05-09T08:48:40.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.159+0000 7f1cdcabd640 1 -- 172.21.15.98:0/3472660234 shutdown_connections 2024-05-09T08:48:40.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.159+0000 7f1cdcabd640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cd81075b0 0x7f1cd819f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.159+0000 7f1cdcabd640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1cd8104c80 0x7f1cd81980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.159+0000 7f1cdcabd640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f1cc40a2c20 0x7f1cc40a50e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.159+0000 7f1cdcabd640 1 --2- 172.21.15.98:0/3472660234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cd8102350 0x7f1cd8197bb0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.159+0000 7f1cdcabd640 1 -- 172.21.15.98:0/3472660234 >> 172.21.15.98:0/3472660234 conn(0x7f1cd80fbd80 msgr2=0x7f1cd8102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:40.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.159+0000 7f1cdcabd640 1 -- 172.21.15.98:0/3472660234 shutdown_connections 2024-05-09T08:48:40.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.159+0000 7f1cdcabd640 1 -- 172.21.15.98:0/3472660234 wait complete. 2024-05-09T08:48:40.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:48:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.272+0000 7fa130d61640 1 -- 172.21.15.98:0/2847983901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa12c101880 msgr2=0x7fa12c101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.272+0000 7fa130d61640 1 --2- 172.21.15.98:0/2847983901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa12c101880 0x7fa12c101ca0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa124009ca0 tx=0x7fa12402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.272+0000 7fa130d61640 1 -- 172.21.15.98:0/2847983901 shutdown_connections 2024-05-09T08:48:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.272+0000 7fa130d61640 1 --2- 172.21.15.98:0/2847983901 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa12c103430 0x7fa12c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.272+0000 7fa130d61640 1 --2- 172.21.15.98:0/2847983901 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa12c102a70 0x7fa12c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.272+0000 7fa130d61640 1 --2- 172.21.15.98:0/2847983901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa12c101880 0x7fa12c101ca0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.272+0000 7fa130d61640 1 -- 172.21.15.98:0/2847983901 >> 172.21.15.98:0/2847983901 conn(0x7fa12c0fce10 msgr2=0x7fa12c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.272+0000 7fa130d61640 1 -- 172.21.15.98:0/2847983901 shutdown_connections 2024-05-09T08:48:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.272+0000 7fa130d61640 1 -- 172.21.15.98:0/2847983901 wait complete. 2024-05-09T08:48:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa130d61640 1 Processor -- start 2024-05-09T08:48:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa130d61640 1 -- start start 2024-05-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa130d61640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa12c101880 0x7fa12c197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa130d61640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa12c102a70 0x7fa12c1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa130d61640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa12c103430 0x7fa12c19f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa130d61640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c10d540 con 0x7fa12c101880 2024-05-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa130d61640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c10d3c0 con 0x7fa12c103430 2024-05-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa130d61640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c10d6c0 con 0x7fa12c102a70 2024-05-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa12a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa12c101880 0x7fa12c197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa12a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa12c101880 0x7fa12c197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55378/0 (socket says 172.21.15.98:55378) 2024-05-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa12a59c640 1 -- 172.21.15.98:0/2726581038 learned_addr learned my addr 172.21.15.98:0/2726581038 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa12ad9d640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa12c103430 0x7fa12c19f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa129d9b640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa12c102a70 0x7fa12c1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa12a59c640 1 -- 172.21.15.98:0/2726581038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa12c102a70 msgr2=0x7fa12c1980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa12a59c640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa12c102a70 0x7fa12c1980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa12a59c640 1 -- 172.21.15.98:0/2726581038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa12c103430 msgr2=0x7fa12c19f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa12a59c640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa12c103430 0x7fa12c19f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa12a59c640 1 -- 172.21.15.98:0/2726581038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa12c19f7b0 con 0x7fa12c101880 2024-05-09T08:48:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa129d9b640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa12c102a70 0x7fa12c1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa12a59c640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa12c101880 0x7fa12c197b90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa124000f80 tx=0x7fa1240319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.273+0000 7fa1237fe640 1 -- 172.21.15.98:0/2726581038 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa124003b70 con 0x7fa12c101880 2024-05-09T08:48:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.274+0000 7fa1237fe640 1 -- 172.21.15.98:0/2726581038 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa124003d10 con 0x7fa12c101880 2024-05-09T08:48:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.274+0000 7fa1237fe640 1 -- 172.21.15.98:0/2726581038 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12402fc50 con 0x7fa12c101880 2024-05-09T08:48:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.274+0000 7fa130d61640 1 -- 172.21.15.98:0/2726581038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa12c19fa40 con 0x7fa12c101880 2024-05-09T08:48:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.274+0000 7fa130d61640 1 -- 172.21.15.98:0/2726581038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa12c19ff90 con 0x7fa12c101880 2024-05-09T08:48:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.274+0000 7fa130d61640 1 -- 172.21.15.98:0/2726581038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa12c108ba0 con 0x7fa12c101880 2024-05-09T08:48:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.276+0000 7fa1237fe640 1 -- 172.21.15.98:0/2726581038 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa124004040 con 0x7fa12c101880 2024-05-09T08:48:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.276+0000 7fa1237fe640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa1180a2c20 0x7fa1180a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.277+0000 7fa129d9b640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa1180a2c20 0x7fa1180a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.277+0000 7fa1237fe640 1 -- 172.21.15.98:0/2726581038 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa1240f08c0 con 0x7fa12c101880 2024-05-09T08:48:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.277+0000 7fa129d9b640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa1180a2c20 0x7fa1180a50e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa12c1990a0 tx=0x7fa114005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:40.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.280+0000 7fa1237fe640 1 -- 172.21.15.98:0/2726581038 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa1240bd590 con 0x7fa12c101880 2024-05-09T08:48:40.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.491+0000 7fa130d61640 1 -- 172.21.15.98:0/2726581038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa12c02d1c0 con 0x7fa12c101880 2024-05-09T08:48:40.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.492+0000 7fa1237fe640 1 -- 172.21.15.98:0/2726581038 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11940 (secure 0 0 0) 0x7fa1240bce90 con 0x7fa12c101880 2024-05-09T08:48:40.492 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:40.492 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:48:39.758226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24457,24461,24388,24439,14667]},"id":1}]} 2024-05-09T08:48:40.492 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-05-09T08:48:40.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.494+0000 7fa130d61640 1 -- 172.21.15.98:0/2726581038 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa1180a2c20 msgr2=0x7fa1180a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.494+0000 7fa130d61640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa1180a2c20 0x7fa1180a50e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa12c1990a0 tx=0x7fa114005f90 comp rx=0 tx=0).stop 2024-05-09T08:48:40.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.494+0000 7fa130d61640 1 -- 172.21.15.98:0/2726581038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa12c101880 msgr2=0x7fa12c197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.494+0000 7fa130d61640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa12c101880 0x7fa12c197b90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa124000f80 tx=0x7fa1240319f0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.494+0000 7fa130d61640 1 -- 172.21.15.98:0/2726581038 shutdown_connections 2024-05-09T08:48:40.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.494+0000 7fa130d61640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa12c103430 0x7fa12c19f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.494+0000 7fa130d61640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa12c102a70 0x7fa12c1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.494+0000 7fa130d61640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa1180a2c20 0x7fa1180a50e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.494+0000 7fa130d61640 1 --2- 172.21.15.98:0/2726581038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa12c101880 0x7fa12c197b90 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.494+0000 7fa130d61640 1 -- 172.21.15.98:0/2726581038 >> 172.21.15.98:0/2726581038 conn(0x7fa12c0fce10 msgr2=0x7fa12c105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:40.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.494+0000 7fa130d61640 1 -- 172.21.15.98:0/2726581038 shutdown_connections 2024-05-09T08:48:40.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.494+0000 7fa130d61640 1 -- 172.21.15.98:0/2726581038 wait complete. 2024-05-09T08:48:40.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:40 smithi098 ceph-mon[30092]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:40.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:40 smithi098 ceph-mon[30092]: pgmap v301: 129 pgs: 129 active+clean; 584 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-05-09T08:48:40.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:40 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] up:active 2024-05-09T08:48:40.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:40 smithi098 ceph-mon[30092]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-05-09T08:48:40.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:40 smithi098 ceph-mon[30092]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-09T08:48:40.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:40 smithi098 ceph-mon[30092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:48:40.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:48:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.606+0000 7fea82dee640 1 -- 172.21.15.98:0/1539042704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea7c100790 msgr2=0x7fea7c100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.606+0000 7fea82dee640 1 --2- 172.21.15.98:0/1539042704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea7c100790 0x7fea7c100bb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009900 tx=0x7fea6c02a410 comp rx=0 tx=0).stop 2024-05-09T08:48:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.606+0000 7fea82dee640 1 -- 172.21.15.98:0/1539042704 shutdown_connections 2024-05-09T08:48:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.606+0000 7fea82dee640 1 --2- 172.21.15.98:0/1539042704 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea7c102340 0x7fea7c108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.606+0000 7fea82dee640 1 --2- 172.21.15.98:0/1539042704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea7c101980 0x7fea7c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.606+0000 7fea82dee640 1 --2- 172.21.15.98:0/1539042704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea7c100790 0x7fea7c100bb0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.606+0000 7fea82dee640 1 -- 172.21.15.98:0/1539042704 >> 172.21.15.98:0/1539042704 conn(0x7fea7c0fbd20 msgr2=0x7fea7c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.606+0000 7fea82dee640 1 -- 172.21.15.98:0/1539042704 shutdown_connections 2024-05-09T08:48:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.606+0000 7fea82dee640 1 -- 172.21.15.98:0/1539042704 wait complete. 2024-05-09T08:48:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea82dee640 1 Processor -- start 2024-05-09T08:48:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea82dee640 1 -- start start 2024-05-09T08:48:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea82dee640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea7c100790 0x7fea7c06aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea82dee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea7c101980 0x7fea7c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea82dee640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea7c102340 0x7fea7c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea82dee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fea7c10c3e0 con 0x7fea7c101980 2024-05-09T08:48:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea82dee640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fea7c10c260 con 0x7fea7c100790 2024-05-09T08:48:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea82dee640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fea7c10c560 con 0x7fea7c102340 2024-05-09T08:48:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea7bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea7c101980 0x7fea7c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea7bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea7c101980 0x7fea7c19c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55394/0 (socket says 172.21.15.98:55394) 2024-05-09T08:48:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea7bfff640 1 -- 172.21.15.98:0/3686423068 learned_addr learned my addr 172.21.15.98:0/3686423068 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea80b8a640 1 --2- 172.21.15.98:0/3686423068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea7c100790 0x7fea7c06aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea8138b640 1 --2- 172.21.15.98:0/3686423068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea7c102340 0x7fea7c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea7bfff640 1 -- 172.21.15.98:0/3686423068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea7c102340 msgr2=0x7fea7c19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea7bfff640 1 --2- 172.21.15.98:0/3686423068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea7c102340 0x7fea7c19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea7bfff640 1 -- 172.21.15.98:0/3686423068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea7c100790 msgr2=0x7fea7c06aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea7bfff640 1 --2- 172.21.15.98:0/3686423068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea7c100790 0x7fea7c06aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.607+0000 7fea7bfff640 1 -- 172.21.15.98:0/3686423068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7c19f7f0 con 0x7fea7c101980 2024-05-09T08:48:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.608+0000 7fea7bfff640 1 --2- 172.21.15.98:0/3686423068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea7c101980 0x7fea7c19c8f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fea6400db80 tx=0x7fea6400b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.608+0000 7fea79ffb640 1 -- 172.21.15.98:0/3686423068 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea64017070 con 0x7fea7c101980 2024-05-09T08:48:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.608+0000 7fea82dee640 1 -- 172.21.15.98:0/3686423068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea7c19fae0 con 0x7fea7c101980 2024-05-09T08:48:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.608+0000 7fea79ffb640 1 -- 172.21.15.98:0/3686423068 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea64003b70 con 0x7fea7c101980 2024-05-09T08:48:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.608+0000 7fea79ffb640 1 -- 172.21.15.98:0/3686423068 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea640106c0 con 0x7fea7c101980 2024-05-09T08:48:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.608+0000 7fea82dee640 1 -- 172.21.15.98:0/3686423068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea7c1a0080 con 0x7fea7c101980 2024-05-09T08:48:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.609+0000 7fea82dee640 1 -- 172.21.15.98:0/3686423068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea7c107ad0 con 0x7fea7c101980 2024-05-09T08:48:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.610+0000 7fea79ffb640 1 -- 172.21.15.98:0/3686423068 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fea64004020 con 0x7fea7c101980 2024-05-09T08:48:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.610+0000 7fea79ffb640 1 --2- 172.21.15.98:0/3686423068 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fea680a2c20 0x7fea680a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.611+0000 7fea80b8a640 1 --2- 172.21.15.98:0/3686423068 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fea680a2c20 0x7fea680a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.611+0000 7fea79ffb640 1 -- 172.21.15.98:0/3686423068 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fea640cf1b0 con 0x7fea7c101980 2024-05-09T08:48:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.611+0000 7fea80b8a640 1 --2- 172.21.15.98:0/3686423068 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fea680a2c20 0x7fea680a50e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fea6c005ab0 tx=0x7fea6c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:40.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.615+0000 7fea79ffb640 1 -- 172.21.15.98:0/3686423068 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fea64020070 con 0x7fea7c101980 2024-05-09T08:48:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:40 smithi142 ceph-mon[34625]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:40.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:40 smithi142 ceph-mon[34625]: pgmap v301: 129 pgs: 129 active+clean; 584 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-05-09T08:48:40.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:40 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] up:active 2024-05-09T08:48:40.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:40 smithi142 ceph-mon[34625]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-05-09T08:48:40.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:40 smithi142 ceph-mon[34625]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-09T08:48:40.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:40 smithi142 ceph-mon[34625]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:48:40.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:40 smithi105 ceph-mon[34498]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:40.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:40 smithi105 ceph-mon[34498]: pgmap v301: 129 pgs: 129 active+clean; 584 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-05-09T08:48:40.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:40 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] up:active 2024-05-09T08:48:40.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:40 smithi105 ceph-mon[34498]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-05-09T08:48:40.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:40 smithi105 ceph-mon[34498]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-09T08:48:40.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:40 smithi105 ceph-mon[34498]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:48:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.796+0000 7fea82dee640 1 -- 172.21.15.98:0/3686423068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea7c02d370 con 0x7fea7c101980 2024-05-09T08:48:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.797+0000 7fea79ffb640 1 -- 172.21.15.98:0/3686423068 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24150 (secure 0 0 0) 0x7fea6409bd70 con 0x7fea7c101980 2024-05-09T08:48:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:40.797 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":90,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:48:22.719729+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:48:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.799+0000 7fea82dee640 1 -- 172.21.15.98:0/3686423068 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fea680a2c20 msgr2=0x7fea680a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.799+0000 7fea82dee640 1 --2- 172.21.15.98:0/3686423068 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fea680a2c20 0x7fea680a50e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fea6c005ab0 tx=0x7fea6c000bc0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.799+0000 7fea82dee640 1 -- 172.21.15.98:0/3686423068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea7c101980 msgr2=0x7fea7c19c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.799+0000 7fea82dee640 1 --2- 172.21.15.98:0/3686423068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea7c101980 0x7fea7c19c8f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fea6400db80 tx=0x7fea6400b630 comp rx=0 tx=0).stop 2024-05-09T08:48:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.799+0000 7fea82dee640 1 -- 172.21.15.98:0/3686423068 shutdown_connections 2024-05-09T08:48:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.799+0000 7fea82dee640 1 --2- 172.21.15.98:0/3686423068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea7c102340 0x7fea7c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.799+0000 7fea82dee640 1 --2- 172.21.15.98:0/3686423068 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fea680a2c20 0x7fea680a50e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.799+0000 7fea82dee640 1 --2- 172.21.15.98:0/3686423068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea7c101980 0x7fea7c19c8f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.799+0000 7fea82dee640 1 --2- 172.21.15.98:0/3686423068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea7c100790 0x7fea7c06aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.799+0000 7fea82dee640 1 -- 172.21.15.98:0/3686423068 >> 172.21.15.98:0/3686423068 conn(0x7fea7c0fbd20 msgr2=0x7fea7c104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.799+0000 7fea82dee640 1 -- 172.21.15.98:0/3686423068 shutdown_connections 2024-05-09T08:48:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.799+0000 7fea82dee640 1 -- 172.21.15.98:0/3686423068 wait complete. 2024-05-09T08:48:40.810 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-09T08:48:40.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:48:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.913+0000 7fe892693640 1 -- 172.21.15.98:0/1789500692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c102a70 msgr2=0x7fe88c102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.913+0000 7fe892693640 1 --2- 172.21.15.98:0/1789500692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c102a70 0x7fe88c102ef0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe880009900 tx=0x7fe88002a410 comp rx=0 tx=0).stop 2024-05-09T08:48:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.913+0000 7fe892693640 1 -- 172.21.15.98:0/1789500692 shutdown_connections 2024-05-09T08:48:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.913+0000 7fe892693640 1 --2- 172.21.15.98:0/1789500692 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe88c103430 0x7fe88c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.913+0000 7fe892693640 1 --2- 172.21.15.98:0/1789500692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c102a70 0x7fe88c102ef0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.913+0000 7fe892693640 1 --2- 172.21.15.98:0/1789500692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe88c101880 0x7fe88c101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.913+0000 7fe892693640 1 -- 172.21.15.98:0/1789500692 >> 172.21.15.98:0/1789500692 conn(0x7fe88c0fce10 msgr2=0x7fe88c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.913+0000 7fe892693640 1 -- 172.21.15.98:0/1789500692 shutdown_connections 2024-05-09T08:48:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.913+0000 7fe892693640 1 -- 172.21.15.98:0/1789500692 wait complete. 2024-05-09T08:48:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.913+0000 7fe892693640 1 Processor -- start 2024-05-09T08:48:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe892693640 1 -- start start 2024-05-09T08:48:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe892693640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe88c101880 0x7fe88c197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe892693640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c102a70 0x7fe88c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe892693640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe88c103430 0x7fe88c19f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe892693640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c10d500 con 0x7fe88c102a70 2024-05-09T08:48:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe892693640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c10d380 con 0x7fe88c101880 2024-05-09T08:48:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe892693640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c10d680 con 0x7fe88c103430 2024-05-09T08:48:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe88b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c102a70 0x7fe88c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe88b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c102a70 0x7fe88c1980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55426/0 (socket says 172.21.15.98:55426) 2024-05-09T08:48:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe88b7fe640 1 -- 172.21.15.98:0/2639472103 learned_addr learned my addr 172.21.15.98:0/2639472103 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe890c30640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe88c103430 0x7fe88c19f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe88bfff640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe88c101880 0x7fe88c197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe88b7fe640 1 -- 172.21.15.98:0/2639472103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe88c103430 msgr2=0x7fe88c19f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe88b7fe640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe88c103430 0x7fe88c19f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe88b7fe640 1 -- 172.21.15.98:0/2639472103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe88c101880 msgr2=0x7fe88c197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe88b7fe640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe88c101880 0x7fe88c197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe88b7fe640 1 -- 172.21.15.98:0/2639472103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88c19f7a0 con 0x7fe88c102a70 2024-05-09T08:48:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe890c30640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe88c103430 0x7fe88c19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe88bfff640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe88c101880 0x7fe88c197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.914+0000 7fe88b7fe640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c102a70 0x7fe88c1980c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe880009230 tx=0x7fe8800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.915+0000 7fe8897fa640 1 -- 172.21.15.98:0/2639472103 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88003f030 con 0x7fe88c102a70 2024-05-09T08:48:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.915+0000 7fe892693640 1 -- 172.21.15.98:0/2639472103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe88c19fa30 con 0x7fe88c102a70 2024-05-09T08:48:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.915+0000 7fe8897fa640 1 -- 172.21.15.98:0/2639472103 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe880003ce0 con 0x7fe88c102a70 2024-05-09T08:48:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.915+0000 7fe8897fa640 1 -- 172.21.15.98:0/2639472103 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe880035070 con 0x7fe88c102a70 2024-05-09T08:48:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.915+0000 7fe892693640 1 -- 172.21.15.98:0/2639472103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe88c19ff70 con 0x7fe88c102a70 2024-05-09T08:48:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.916+0000 7fe892693640 1 -- 172.21.15.98:0/2639472103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe88c108ba0 con 0x7fe88c102a70 2024-05-09T08:48:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.917+0000 7fe8897fa640 1 -- 172.21.15.98:0/2639472103 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe880042030 con 0x7fe88c102a70 2024-05-09T08:48:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.917+0000 7fe8897fa640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe8740a2c20 0x7fe8740a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.917+0000 7fe88bfff640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe8740a2c20 0x7fe8740a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.917+0000 7fe8897fa640 1 -- 172.21.15.98:0/2639472103 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe8800ef9e0 con 0x7fe88c102a70 2024-05-09T08:48:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.918+0000 7fe88bfff640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe8740a2c20 0x7fe8740a50e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe88c1028d0 tx=0x7fe87c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:40.922+0000 7fe8897fa640 1 -- 172.21.15.98:0/2639472103 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe8800bc5a0 con 0x7fe88c102a70 2024-05-09T08:48:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.094+0000 7fe892693640 1 -- 172.21.15.98:0/2639472103 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe88c061c70 con 0x7fe8740a2c20 2024-05-09T08:48:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.104+0000 7fe8897fa640 1 -- 172.21.15.98:0/2639472103 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340471 (secure 0 0 0) 0x7fe88c061c70 con 0x7fe8740a2c20 2024-05-09T08:48:41.107 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-09T08:48:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.107+0000 7fe892693640 1 -- 172.21.15.98:0/2639472103 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe8740a2c20 msgr2=0x7fe8740a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.107+0000 7fe892693640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe8740a2c20 0x7fe8740a50e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe88c1028d0 tx=0x7fe87c005f90 comp rx=0 tx=0).stop 2024-05-09T08:48:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.107+0000 7fe892693640 1 -- 172.21.15.98:0/2639472103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c102a70 msgr2=0x7fe88c1980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.107+0000 7fe892693640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c102a70 0x7fe88c1980c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe880009230 tx=0x7fe8800042f0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.107+0000 7fe892693640 1 -- 172.21.15.98:0/2639472103 shutdown_connections 2024-05-09T08:48:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.107+0000 7fe892693640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe88c103430 0x7fe88c19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.107+0000 7fe892693640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe8740a2c20 0x7fe8740a50e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.107+0000 7fe892693640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c102a70 0x7fe88c1980c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.107+0000 7fe892693640 1 --2- 172.21.15.98:0/2639472103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe88c101880 0x7fe88c197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.107+0000 7fe892693640 1 -- 172.21.15.98:0/2639472103 >> 172.21.15.98:0/2639472103 conn(0x7fe88c0fce10 msgr2=0x7fe88c105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.108+0000 7fe892693640 1 -- 172.21.15.98:0/2639472103 shutdown_connections 2024-05-09T08:48:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.108+0000 7fe892693640 1 -- 172.21.15.98:0/2639472103 wait complete. 2024-05-09T08:48:41.118 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-05-09T08:48:39.184206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598133,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":154,"num_read_kb":136,"num_write":278,"num_write_kb":2981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":352556,"kb_used_data":11696,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124589268,"statfs":{"total":1151940427776,"available":1151579410432,"internally_reserved":0,"allocated":11976704,"data_stored":7560343,"data_compressed":282369,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5355,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":18,"num_read_kb":17,"num_write":52,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003294"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538367+0000","last_change":"2024-05-09T08:48:18.373685+0000","last_active":"2024-05-09T08:48:25.538367+0000","last_peered":"2024-05-09T08:48:25.538367+0000","last_clean":"2024-05-09T08:48:25.538367+0000","last_became_active":"2024-05-09T08:48:18.373305+0000","last_became_peered":"2024-05-09T08:48:18.373305+0000","last_unstale":"2024-05-09T08:48:25.538367+0000","last_undegraded":"2024-05-09T08:48:25.538367+0000","last_fullsized":"2024-05-09T08:48:25.538367+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:30:45.261706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782689+0000","last_change":"2024-05-09T08:48:18.373152+0000","last_active":"2024-05-09T08:48:25.782689+0000","last_peered":"2024-05-09T08:48:25.782689+0000","last_clean":"2024-05-09T08:48:25.782689+0000","last_became_active":"2024-05-09T08:48:18.372015+0000","last_became_peered":"2024-05-09T08:48:18.372015+0000","last_unstale":"2024-05-09T08:48:25.782689+0000","last_undegraded":"2024-05-09T08:48:25.782689+0000","last_fullsized":"2024-05-09T08:48:25.782689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:47.512778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782656+0000","last_change":"2024-05-09T08:48:18.370177+0000","last_active":"2024-05-09T08:48:25.782656+0000","last_peered":"2024-05-09T08:48:25.782656+0000","last_clean":"2024-05-09T08:48:25.782656+0000","last_became_active":"2024-05-09T08:48:18.369842+0000","last_became_peered":"2024-05-09T08:48:18.369842+0000","last_unstale":"2024-05-09T08:48:25.782656+0000","last_undegraded":"2024-05-09T08:48:25.782656+0000","last_fullsized":"2024-05-09T08:48:25.782656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:00:24.015500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538420+0000","last_change":"2024-05-09T08:48:18.375364+0000","last_active":"2024-05-09T08:48:25.538420+0000","last_peered":"2024-05-09T08:48:25.538420+0000","last_clean":"2024-05-09T08:48:25.538420+0000","last_became_active":"2024-05-09T08:48:18.375030+0000","last_became_peered":"2024-05-09T08:48:18.375030+0000","last_unstale":"2024-05-09T08:48:25.538420+0000","last_undegraded":"2024-05-09T08:48:25.538420+0000","last_fullsized":"2024-05-09T08:48:25.538420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:43:57.684123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.681816+0000","last_change":"2024-05-09T08:48:18.374548+0000","last_active":"2024-05-09T08:48:24.681816+0000","last_peered":"2024-05-09T08:48:24.681816+0000","last_clean":"2024-05-09T08:48:24.681816+0000","last_became_active":"2024-05-09T08:48:18.374190+0000","last_became_peered":"2024-05-09T08:48:18.374190+0000","last_unstale":"2024-05-09T08:48:24.681816+0000","last_undegraded":"2024-05-09T08:48:24.681816+0000","last_fullsized":"2024-05-09T08:48:24.681816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:43:48.456839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538724+0000","last_change":"2024-05-09T08:48:18.371234+0000","last_active":"2024-05-09T08:48:25.538724+0000","last_peered":"2024-05-09T08:48:25.538724+0000","last_clean":"2024-05-09T08:48:25.538724+0000","last_became_active":"2024-05-09T08:48:18.370218+0000","last_became_peered":"2024-05-09T08:48:18.370218+0000","last_unstale":"2024-05-09T08:48:25.538724+0000","last_undegraded":"2024-05-09T08:48:25.538724+0000","last_fullsized":"2024-05-09T08:48:25.538724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:02:09.923146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608500+0000","last_change":"2024-05-09T08:48:18.368731+0000","last_active":"2024-05-09T08:48:23.608500+0000","last_peered":"2024-05-09T08:48:23.608500+0000","last_clean":"2024-05-09T08:48:23.608500+0000","last_became_active":"2024-05-09T08:48:18.368389+0000","last_became_peered":"2024-05-09T08:48:18.368389+0000","last_unstale":"2024-05-09T08:48:23.608500+0000","last_undegraded":"2024-05-09T08:48:23.608500+0000","last_fullsized":"2024-05-09T08:48:23.608500+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:21:45.177845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537684+0000","last_change":"2024-05-09T08:48:18.374296+0000","last_active":"2024-05-09T08:48:25.537684+0000","last_peered":"2024-05-09T08:48:25.537684+0000","last_clean":"2024-05-09T08:48:25.537684+0000","last_became_active":"2024-05-09T08:48:18.369805+0000","last_became_peered":"2024-05-09T08:48:18.369805+0000","last_unstale":"2024-05-09T08:48:25.537684+0000","last_undegraded":"2024-05-09T08:48:25.537684+0000","last_fullsized":"2024-05-09T08:48:25.537684+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:27:31.476940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.835158+0000","last_change":"2024-05-09T08:48:18.372686+0000","last_active":"2024-05-09T08:48:24.835158+0000","last_peered":"2024-05-09T08:48:24.835158+0000","last_clean":"2024-05-09T08:48:24.835158+0000","last_became_active":"2024-05-09T08:48:18.372349+0000","last_became_peered":"2024-05-09T08:48:18.372349+0000","last_unstale":"2024-05-09T08:48:24.835158+0000","last_undegraded":"2024-05-09T08:48:24.835158+0000","last_fullsized":"2024-05-09T08:48:24.835158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:15.177389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.539112+0000","last_change":"2024-05-09T08:48:18.363668+0000","last_active":"2024-05-09T08:48:25.539112+0000","last_peered":"2024-05-09T08:48:25.539112+0000","last_clean":"2024-05-09T08:48:25.539112+0000","last_became_active":"2024-05-09T08:48:18.363248+0000","last_became_peered":"2024-05-09T08:48:18.363248+0000","last_unstale":"2024-05-09T08:48:25.539112+0000","last_undegraded":"2024-05-09T08:48:25.539112+0000","last_fullsized":"2024-05-09T08:48:25.539112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:19:33.802350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.609133+0000","last_change":"2024-05-09T08:48:18.376401+0000","last_active":"2024-05-09T08:48:23.609133+0000","last_peered":"2024-05-09T08:48:23.609133+0000","last_clean":"2024-05-09T08:48:23.609133+0000","last_became_active":"2024-05-09T08:48:18.375520+0000","last_became_peered":"2024-05-09T08:48:18.375520+0000","last_unstale":"2024-05-09T08:48:23.609133+0000","last_undegraded":"2024-05-09T08:48:23.609133+0000","last_fullsized":"2024-05-09T08:48:23.609133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:40:36.145071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538314+0000","last_change":"2024-05-09T08:48:18.372847+0000","last_active":"2024-05-09T08:48:25.538314+0000","last_peered":"2024-05-09T08:48:25.538314+0000","last_clean":"2024-05-09T08:48:25.538314+0000","last_became_active":"2024-05-09T08:48:18.370671+0000","last_became_peered":"2024-05-09T08:48:18.370671+0000","last_unstale":"2024-05-09T08:48:25.538314+0000","last_undegraded":"2024-05-09T08:48:25.538314+0000","last_fullsized":"2024-05-09T08:48:25.538314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:06:22.438515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603752+0000","last_change":"2024-05-09T08:48:18.365810+0000","last_active":"2024-05-09T08:48:23.603752+0000","last_peered":"2024-05-09T08:48:23.603752+0000","last_clean":"2024-05-09T08:48:23.603752+0000","last_became_active":"2024-05-09T08:48:18.365214+0000","last_became_peered":"2024-05-09T08:48:18.365214+0000","last_unstale":"2024-05-09T08:48:23.603752+0000","last_undegraded":"2024-05-09T08:48:23.603752+0000","last_fullsized":"2024-05-09T08:48:23.603752+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:00.037153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782218+0000","last_change":"2024-05-09T08:48:18.375279+0000","last_active":"2024-05-09T08:48:25.782218+0000","last_peered":"2024-05-09T08:48:25.782218+0000","last_clean":"2024-05-09T08:48:25.782218+0000","last_became_active":"2024-05-09T08:48:18.374918+0000","last_became_peered":"2024-05-09T08:48:18.374918+0000","last_unstale":"2024-05-09T08:48:25.782218+0000","last_undegraded":"2024-05-09T08:48:25.782218+0000","last_fullsized":"2024-05-09T08:48:25.782218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:23:12.889283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782336+0000","last_change":"2024-05-09T08:48:18.377722+0000","last_active":"2024-05-09T08:48:25.782336+0000","last_peered":"2024-05-09T08:48:25.782336+0000","last_clean":"2024-05-09T08:48:25.782336+0000","last_became_active":"2024-05-09T08:48:18.377416+0000","last_became_peered":"2024-05-09T08:48:18.377416+0000","last_unstale":"2024-05-09T08:48:25.782336+0000","last_undegraded":"2024-05-09T08:48:25.782336+0000","last_fullsized":"2024-05-09T08:48:25.782336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:27.093800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.834636+0000","last_change":"2024-05-09T08:48:18.373905+0000","last_active":"2024-05-09T08:48:24.834636+0000","last_peered":"2024-05-09T08:48:24.834636+0000","last_clean":"2024-05-09T08:48:24.834636+0000","last_became_active":"2024-05-09T08:48:18.373566+0000","last_became_peered":"2024-05-09T08:48:18.373566+0000","last_unstale":"2024-05-09T08:48:24.834636+0000","last_undegraded":"2024-05-09T08:48:24.834636+0000","last_fullsized":"2024-05-09T08:48:24.834636+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:33.905686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538287+0000","last_change":"2024-05-09T08:48:18.373970+0000","last_active":"2024-05-09T08:48:25.538287+0000","last_peered":"2024-05-09T08:48:25.538287+0000","last_clean":"2024-05-09T08:48:25.538287+0000","last_became_active":"2024-05-09T08:48:18.372696+0000","last_became_peered":"2024-05-09T08:48:18.372696+0000","last_unstale":"2024-05-09T08:48:25.538287+0000","last_undegraded":"2024-05-09T08:48:25.538287+0000","last_fullsized":"2024-05-09T08:48:25.538287+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:09:33.751950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.779162+0000","last_change":"2024-05-09T08:48:18.376851+0000","last_active":"2024-05-09T08:48:23.779162+0000","last_peered":"2024-05-09T08:48:23.779162+0000","last_clean":"2024-05-09T08:48:23.779162+0000","last_became_active":"2024-05-09T08:48:18.376230+0000","last_became_peered":"2024-05-09T08:48:18.376230+0000","last_unstale":"2024-05-09T08:48:23.779162+0000","last_undegraded":"2024-05-09T08:48:23.779162+0000","last_fullsized":"2024-05-09T08:48:23.779162+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:34.959179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888910+0000","last_change":"2024-05-09T08:48:18.370287+0000","last_active":"2024-05-09T08:48:23.888910+0000","last_peered":"2024-05-09T08:48:23.888910+0000","last_clean":"2024-05-09T08:48:23.888910+0000","last_became_active":"2024-05-09T08:48:18.369956+0000","last_became_peered":"2024-05-09T08:48:18.369956+0000","last_unstale":"2024-05-09T08:48:23.888910+0000","last_undegraded":"2024-05-09T08:48:23.888910+0000","last_fullsized":"2024-05-09T08:48:23.888910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:19:34.746216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888612+0000","last_change":"2024-05-09T08:48:22.377973+0000","last_active":"2024-05-09T08:48:23.888612+0000","last_peered":"2024-05-09T08:48:23.888612+0000","last_clean":"2024-05-09T08:48:23.888612+0000","last_became_active":"2024-05-09T08:48:22.377325+0000","last_became_peered":"2024-05-09T08:48:22.377325+0000","last_unstale":"2024-05-09T08:48:23.888612+0000","last_undegraded":"2024-05-09T08:48:23.888612+0000","last_fullsized":"2024-05-09T08:48:23.888612+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:32:00.873082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.779296+0000","last_change":"2024-05-09T08:48:17.595784+0000","last_active":"2024-05-09T08:48:23.779296+0000","last_peered":"2024-05-09T08:48:23.779296+0000","last_clean":"2024-05-09T08:48:23.779296+0000","last_became_active":"2024-05-09T08:48:17.595395+0000","last_became_peered":"2024-05-09T08:48:17.595395+0000","last_unstale":"2024-05-09T08:48:23.779296+0000","last_undegraded":"2024-05-09T08:48:23.779296+0000","last_fullsized":"2024-05-09T08:48:23.779296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:47.475148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537907+0000","last_change":"2024-05-09T08:48:18.382332+0000","last_active":"2024-05-09T08:48:25.537907+0000","last_peered":"2024-05-09T08:48:25.537907+0000","last_clean":"2024-05-09T08:48:25.537907+0000","last_became_active":"2024-05-09T08:48:18.381536+0000","last_became_peered":"2024-05-09T08:48:18.381536+0000","last_unstale":"2024-05-09T08:48:25.537907+0000","last_undegraded":"2024-05-09T08:48:25.537907+0000","last_fullsized":"2024-05-09T08:48:25.537907+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:52:48.327000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537819+0000","last_change":"2024-05-09T08:48:22.379541+0000","last_active":"2024-05-09T08:48:25.537819+0000","last_peered":"2024-05-09T08:48:25.537819+0000","last_clean":"2024-05-09T08:48:25.537819+0000","last_became_active":"2024-05-09T08:48:22.379135+0000","last_became_peered":"2024-05-09T08:48:22.379135+0000","last_unstale":"2024-05-09T08:48:25.537819+0000","last_undegraded":"2024-05-09T08:48:25.537819+0000","last_fullsized":"2024-05-09T08:48:25.537819+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:51.365075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538085+0000","last_change":"2024-05-09T08:48:17.592168+0000","last_active":"2024-05-09T08:48:25.538085+0000","last_peered":"2024-05-09T08:48:25.538085+0000","last_clean":"2024-05-09T08:48:25.538085+0000","last_became_active":"2024-05-09T08:48:17.591822+0000","last_became_peered":"2024-05-09T08:48:17.591822+0000","last_unstale":"2024-05-09T08:48:25.538085+0000","last_undegraded":"2024-05-09T08:48:25.538085+0000","last_fullsized":"2024-05-09T08:48:25.538085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:45:15.162240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782715+0000","last_change":"2024-05-09T08:48:18.379396+0000","last_active":"2024-05-09T08:48:25.782715+0000","last_peered":"2024-05-09T08:48:25.782715+0000","last_clean":"2024-05-09T08:48:25.782715+0000","last_became_active":"2024-05-09T08:48:18.378770+0000","last_became_peered":"2024-05-09T08:48:18.378770+0000","last_unstale":"2024-05-09T08:48:25.782715+0000","last_undegraded":"2024-05-09T08:48:25.782715+0000","last_fullsized":"2024-05-09T08:48:25.782715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:22:22.699186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538058+0000","last_change":"2024-05-09T08:48:22.387777+0000","last_active":"2024-05-09T08:48:25.538058+0000","last_peered":"2024-05-09T08:48:25.538058+0000","last_clean":"2024-05-09T08:48:25.538058+0000","last_became_active":"2024-05-09T08:48:22.387300+0000","last_became_peered":"2024-05-09T08:48:22.387300+0000","last_unstale":"2024-05-09T08:48:25.538058+0000","last_undegraded":"2024-05-09T08:48:25.538058+0000","last_fullsized":"2024-05-09T08:48:25.538058+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:08:35.646374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.682429+0000","last_change":"2024-05-09T08:48:17.602493+0000","last_active":"2024-05-09T08:48:24.682429+0000","last_peered":"2024-05-09T08:48:24.682429+0000","last_clean":"2024-05-09T08:48:24.682429+0000","last_became_active":"2024-05-09T08:48:17.601625+0000","last_became_peered":"2024-05-09T08:48:17.601625+0000","last_unstale":"2024-05-09T08:48:24.682429+0000","last_undegraded":"2024-05-09T08:48:24.682429+0000","last_fullsized":"2024-05-09T08:48:24.682429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:33.044532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.834873+0000","last_change":"2024-05-09T08:48:18.380615+0000","last_active":"2024-05-09T08:48:24.834873+0000","last_peered":"2024-05-09T08:48:24.834873+0000","last_clean":"2024-05-09T08:48:24.834873+0000","last_became_active":"2024-05-09T08:48:18.380196+0000","last_became_peered":"2024-05-09T08:48:18.380196+0000","last_unstale":"2024-05-09T08:48:24.834873+0000","last_undegraded":"2024-05-09T08:48:24.834873+0000","last_fullsized":"2024-05-09T08:48:24.834873+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:27.299263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537812+0000","last_change":"2024-05-09T08:48:22.381954+0000","last_active":"2024-05-09T08:48:25.537812+0000","last_peered":"2024-05-09T08:48:25.537812+0000","last_clean":"2024-05-09T08:48:25.537812+0000","last_became_active":"2024-05-09T08:48:22.381527+0000","last_became_peered":"2024-05-09T08:48:22.381527+0000","last_unstale":"2024-05-09T08:48:25.537812+0000","last_undegraded":"2024-05-09T08:48:25.537812+0000","last_fullsized":"2024-05-09T08:48:25.537812+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:35.170753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.681941+0000","last_change":"2024-05-09T08:48:17.592795+0000","last_active":"2024-05-09T08:48:24.681941+0000","last_peered":"2024-05-09T08:48:24.681941+0000","last_clean":"2024-05-09T08:48:24.681941+0000","last_became_active":"2024-05-09T08:48:17.592240+0000","last_became_peered":"2024-05-09T08:48:17.592240+0000","last_unstale":"2024-05-09T08:48:24.681941+0000","last_undegraded":"2024-05-09T08:48:24.681941+0000","last_fullsized":"2024-05-09T08:48:24.681941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:27:30.056805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608521+0000","last_change":"2024-05-09T08:48:18.373853+0000","last_active":"2024-05-09T08:48:23.608521+0000","last_peered":"2024-05-09T08:48:23.608521+0000","last_clean":"2024-05-09T08:48:23.608521+0000","last_became_active":"2024-05-09T08:48:18.373482+0000","last_became_peered":"2024-05-09T08:48:18.373482+0000","last_unstale":"2024-05-09T08:48:23.608521+0000","last_undegraded":"2024-05-09T08:48:23.608521+0000","last_fullsized":"2024-05-09T08:48:23.608521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:14:49.129927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888800+0000","last_change":"2024-05-09T08:48:22.374267+0000","last_active":"2024-05-09T08:48:23.888800+0000","last_peered":"2024-05-09T08:48:23.888800+0000","last_clean":"2024-05-09T08:48:23.888800+0000","last_became_active":"2024-05-09T08:48:22.373824+0000","last_became_peered":"2024-05-09T08:48:22.373824+0000","last_unstale":"2024-05-09T08:48:23.888800+0000","last_undegraded":"2024-05-09T08:48:23.888800+0000","last_fullsized":"2024-05-09T08:48:23.888800+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:27.407543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.733155+0000","last_change":"2024-05-09T08:48:17.596851+0000","last_active":"2024-05-09T08:48:37.733155+0000","last_peered":"2024-05-09T08:48:37.733155+0000","last_clean":"2024-05-09T08:48:37.733155+0000","last_became_active":"2024-05-09T08:48:17.596532+0000","last_became_peered":"2024-05-09T08:48:17.596532+0000","last_unstale":"2024-05-09T08:48:37.733155+0000","last_undegraded":"2024-05-09T08:48:37.733155+0000","last_fullsized":"2024-05-09T08:48:37.733155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:58:51.923188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.682002+0000","last_change":"2024-05-09T08:48:18.372214+0000","last_active":"2024-05-09T08:48:24.682002+0000","last_peered":"2024-05-09T08:48:24.682002+0000","last_clean":"2024-05-09T08:48:24.682002+0000","last_became_active":"2024-05-09T08:48:18.371850+0000","last_became_peered":"2024-05-09T08:48:18.371850+0000","last_unstale":"2024-05-09T08:48:24.682002+0000","last_undegraded":"2024-05-09T08:48:24.682002+0000","last_fullsized":"2024-05-09T08:48:24.682002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:56.971735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888674+0000","last_change":"2024-05-09T08:48:22.374755+0000","last_active":"2024-05-09T08:48:23.888674+0000","last_peered":"2024-05-09T08:48:23.888674+0000","last_clean":"2024-05-09T08:48:23.888674+0000","last_became_active":"2024-05-09T08:48:22.374139+0000","last_became_peered":"2024-05-09T08:48:22.374139+0000","last_unstale":"2024-05-09T08:48:23.888674+0000","last_undegraded":"2024-05-09T08:48:23.888674+0000","last_fullsized":"2024-05-09T08:48:23.888674+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:57:57.724212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.743649+0000","last_change":"2024-05-09T08:48:17.601726+0000","last_active":"2024-05-09T08:48:37.743649+0000","last_peered":"2024-05-09T08:48:37.743649+0000","last_clean":"2024-05-09T08:48:37.743649+0000","last_became_active":"2024-05-09T08:48:17.601372+0000","last_became_peered":"2024-05-09T08:48:17.601372+0000","last_unstale":"2024-05-09T08:48:37.743649+0000","last_undegraded":"2024-05-09T08:48:37.743649+0000","last_fullsized":"2024-05-09T08:48:37.743649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:27:56.683821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608262+0000","last_change":"2024-05-09T08:48:18.369767+0000","last_active":"2024-05-09T08:48:23.608262+0000","last_peered":"2024-05-09T08:48:23.608262+0000","last_clean":"2024-05-09T08:48:23.608262+0000","last_became_active":"2024-05-09T08:48:18.369388+0000","last_became_peered":"2024-05-09T08:48:18.369388+0000","last_unstale":"2024-05-09T08:48:23.608262+0000","last_undegraded":"2024-05-09T08:48:23.608262+0000","last_fullsized":"2024-05-09T08:48:23.608262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:31:44.727770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538260+0000","last_change":"2024-05-09T08:48:22.390990+0000","last_active":"2024-05-09T08:48:25.538260+0000","last_peered":"2024-05-09T08:48:25.538260+0000","last_clean":"2024-05-09T08:48:25.538260+0000","last_became_active":"2024-05-09T08:48:22.390601+0000","last_became_peered":"2024-05-09T08:48:22.390601+0000","last_unstale":"2024-05-09T08:48:25.538260+0000","last_undegraded":"2024-05-09T08:48:25.538260+0000","last_fullsized":"2024-05-09T08:48:25.538260+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:47:24.644408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603399+0000","last_change":"2024-05-09T08:48:17.598008+0000","last_active":"2024-05-09T08:48:23.603399+0000","last_peered":"2024-05-09T08:48:23.603399+0000","last_clean":"2024-05-09T08:48:23.603399+0000","last_became_active":"2024-05-09T08:48:17.597514+0000","last_became_peered":"2024-05-09T08:48:17.597514+0000","last_unstale":"2024-05-09T08:48:23.603399+0000","last_undegraded":"2024-05-09T08:48:23.603399+0000","last_fullsized":"2024-05-09T08:48:23.603399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:18:32.788536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537547+0000","last_change":"2024-05-09T08:48:18.378061+0000","last_active":"2024-05-09T08:48:25.537547+0000","last_peered":"2024-05-09T08:48:25.537547+0000","last_clean":"2024-05-09T08:48:25.537547+0000","last_became_active":"2024-05-09T08:48:18.377684+0000","last_became_peered":"2024-05-09T08:48:18.377684+0000","last_unstale":"2024-05-09T08:48:25.537547+0000","last_undegraded":"2024-05-09T08:48:25.537547+0000","last_fullsized":"2024-05-09T08:48:25.537547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:13:08.338307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782722+0000","last_change":"2024-05-09T08:48:22.385779+0000","last_active":"2024-05-09T08:48:25.782722+0000","last_peered":"2024-05-09T08:48:25.782722+0000","last_clean":"2024-05-09T08:48:25.782722+0000","last_became_active":"2024-05-09T08:48:22.385021+0000","last_became_peered":"2024-05-09T08:48:22.385021+0000","last_unstale":"2024-05-09T08:48:25.782722+0000","last_undegraded":"2024-05-09T08:48:25.782722+0000","last_fullsized":"2024-05-09T08:48:25.782722+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:51.350754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:36.279252+0000","last_change":"2024-05-09T08:48:17.595750+0000","last_active":"2024-05-09T08:48:36.279252+0000","last_peered":"2024-05-09T08:48:36.279252+0000","last_clean":"2024-05-09T08:48:36.279252+0000","last_became_active":"2024-05-09T08:48:17.595394+0000","last_became_peered":"2024-05-09T08:48:17.595394+0000","last_unstale":"2024-05-09T08:48:36.279252+0000","last_undegraded":"2024-05-09T08:48:36.279252+0000","last_fullsized":"2024-05-09T08:48:36.279252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:39:21.975059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.778932+0000","last_change":"2024-05-09T08:48:18.370750+0000","last_active":"2024-05-09T08:48:23.778932+0000","last_peered":"2024-05-09T08:48:23.778932+0000","last_clean":"2024-05-09T08:48:23.778932+0000","last_became_active":"2024-05-09T08:48:18.370383+0000","last_became_peered":"2024-05-09T08:48:18.370383+0000","last_unstale":"2024-05-09T08:48:23.778932+0000","last_undegraded":"2024-05-09T08:48:23.778932+0000","last_fullsized":"2024-05-09T08:48:23.778932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:36.049350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537648+0000","last_change":"2024-05-09T08:48:22.377157+0000","last_active":"2024-05-09T08:48:25.537648+0000","last_peered":"2024-05-09T08:48:25.537648+0000","last_clean":"2024-05-09T08:48:25.537648+0000","last_became_active":"2024-05-09T08:48:22.376730+0000","last_became_peered":"2024-05-09T08:48:22.376730+0000","last_unstale":"2024-05-09T08:48:25.537648+0000","last_undegraded":"2024-05-09T08:48:25.537648+0000","last_fullsized":"2024-05-09T08:48:25.537648+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:41:52.080730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.732425+0000","last_change":"2024-05-09T08:48:17.598528+0000","last_active":"2024-05-09T08:48:37.732425+0000","last_peered":"2024-05-09T08:48:37.732425+0000","last_clean":"2024-05-09T08:48:37.732425+0000","last_became_active":"2024-05-09T08:48:17.598072+0000","last_became_peered":"2024-05-09T08:48:17.598072+0000","last_unstale":"2024-05-09T08:48:37.732425+0000","last_undegraded":"2024-05-09T08:48:37.732425+0000","last_fullsized":"2024-05-09T08:48:37.732425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:35.697154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.835104+0000","last_change":"2024-05-09T08:48:18.380919+0000","last_active":"2024-05-09T08:48:24.835104+0000","last_peered":"2024-05-09T08:48:24.835104+0000","last_clean":"2024-05-09T08:48:24.835104+0000","last_became_active":"2024-05-09T08:48:18.380485+0000","last_became_peered":"2024-05-09T08:48:18.380485+0000","last_unstale":"2024-05-09T08:48:24.835104+0000","last_undegraded":"2024-05-09T08:48:24.835104+0000","last_fullsized":"2024-05-09T08:48:24.835104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:45.079173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.778704+0000","last_change":"2024-05-09T08:48:22.389667+0000","last_active":"2024-05-09T08:48:23.778704+0000","last_peered":"2024-05-09T08:48:23.778704+0000","last_clean":"2024-05-09T08:48:23.778704+0000","last_became_active":"2024-05-09T08:48:22.389268+0000","last_became_peered":"2024-05-09T08:48:22.389268+0000","last_unstale":"2024-05-09T08:48:23.778704+0000","last_undegraded":"2024-05-09T08:48:23.778704+0000","last_fullsized":"2024-05-09T08:48:23.778704+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:33:14.861855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.609016+0000","last_change":"2024-05-09T08:48:17.596028+0000","last_active":"2024-05-09T08:48:23.609016+0000","last_peered":"2024-05-09T08:48:23.609016+0000","last_clean":"2024-05-09T08:48:23.609016+0000","last_became_active":"2024-05-09T08:48:17.595053+0000","last_became_peered":"2024-05-09T08:48:17.595053+0000","last_unstale":"2024-05-09T08:48:23.609016+0000","last_undegraded":"2024-05-09T08:48:23.609016+0000","last_fullsized":"2024-05-09T08:48:23.609016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:48:31.323595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782741+0000","last_change":"2024-05-09T08:48:18.373472+0000","last_active":"2024-05-09T08:48:25.782741+0000","last_peered":"2024-05-09T08:48:25.782741+0000","last_clean":"2024-05-09T08:48:25.782741+0000","last_became_active":"2024-05-09T08:48:18.373178+0000","last_became_peered":"2024-05-09T08:48:18.373178+0000","last_unstale":"2024-05-09T08:48:25.782741+0000","last_undegraded":"2024-05-09T08:48:25.782741+0000","last_fullsized":"2024-05-09T08:48:25.782741+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:18.048722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538490+0000","last_change":"2024-05-09T08:48:22.386551+0000","last_active":"2024-05-09T08:48:25.538490+0000","last_peered":"2024-05-09T08:48:25.538490+0000","last_clean":"2024-05-09T08:48:25.538490+0000","last_became_active":"2024-05-09T08:48:22.386018+0000","last_became_peered":"2024-05-09T08:48:22.386018+0000","last_unstale":"2024-05-09T08:48:25.538490+0000","last_undegraded":"2024-05-09T08:48:25.538490+0000","last_fullsized":"2024-05-09T08:48:25.538490+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:59.812030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.733781+0000","last_change":"2024-05-09T08:48:17.601207+0000","last_active":"2024-05-09T08:48:37.733781+0000","last_peered":"2024-05-09T08:48:37.733781+0000","last_clean":"2024-05-09T08:48:37.733781+0000","last_became_active":"2024-05-09T08:48:17.600878+0000","last_became_peered":"2024-05-09T08:48:17.600878+0000","last_unstale":"2024-05-09T08:48:37.733781+0000","last_undegraded":"2024-05-09T08:48:37.733781+0000","last_fullsized":"2024-05-09T08:48:37.733781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:30.040527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.835063+0000","last_change":"2024-05-09T08:48:18.381577+0000","last_active":"2024-05-09T08:48:24.835063+0000","last_peered":"2024-05-09T08:48:24.835063+0000","last_clean":"2024-05-09T08:48:24.835063+0000","last_became_active":"2024-05-09T08:48:18.381223+0000","last_became_peered":"2024-05-09T08:48:18.381223+0000","last_unstale":"2024-05-09T08:48:24.835063+0000","last_undegraded":"2024-05-09T08:48:24.835063+0000","last_fullsized":"2024-05-09T08:48:24.835063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:57.038392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.778720+0000","last_change":"2024-05-09T08:48:22.385885+0000","last_active":"2024-05-09T08:48:23.778720+0000","last_peered":"2024-05-09T08:48:23.778720+0000","last_clean":"2024-05-09T08:48:23.778720+0000","last_became_active":"2024-05-09T08:48:22.385403+0000","last_became_peered":"2024-05-09T08:48:22.385403+0000","last_unstale":"2024-05-09T08:48:23.778720+0000","last_undegraded":"2024-05-09T08:48:23.778720+0000","last_fullsized":"2024-05-09T08:48:23.778720+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:53:20.733523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'12","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.732070+0000","last_change":"2024-05-09T08:48:17.597848+0000","last_active":"2024-05-09T08:48:37.732070+0000","last_peered":"2024-05-09T08:48:37.732070+0000","last_clean":"2024-05-09T08:48:37.732070+0000","last_became_active":"2024-05-09T08:48:17.596989+0000","last_became_peered":"2024-05-09T08:48:17.596989+0000","last_unstale":"2024-05-09T08:48:37.732070+0000","last_undegraded":"2024-05-09T08:48:37.732070+0000","last_fullsized":"2024-05-09T08:48:37.732070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:07.256301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.682156+0000","last_change":"2024-05-09T08:48:18.365308+0000","last_active":"2024-05-09T08:48:24.682156+0000","last_peered":"2024-05-09T08:48:24.682156+0000","last_clean":"2024-05-09T08:48:24.682156+0000","last_became_active":"2024-05-09T08:48:18.364845+0000","last_became_peered":"2024-05-09T08:48:18.364845+0000","last_unstale":"2024-05-09T08:48:24.682156+0000","last_undegraded":"2024-05-09T08:48:24.682156+0000","last_fullsized":"2024-05-09T08:48:24.682156+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:07.277730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608825+0000","last_change":"2024-05-09T08:48:22.377333+0000","last_active":"2024-05-09T08:48:23.608825+0000","last_peered":"2024-05-09T08:48:23.608825+0000","last_clean":"2024-05-09T08:48:23.608825+0000","last_became_active":"2024-05-09T08:48:22.376494+0000","last_became_peered":"2024-05-09T08:48:22.376494+0000","last_unstale":"2024-05-09T08:48:23.608825+0000","last_undegraded":"2024-05-09T08:48:23.608825+0000","last_fullsized":"2024-05-09T08:48:23.608825+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:49:17.693977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888419+0000","last_change":"2024-05-09T08:48:17.600749+0000","last_active":"2024-05-09T08:48:23.888419+0000","last_peered":"2024-05-09T08:48:23.888419+0000","last_clean":"2024-05-09T08:48:23.888419+0000","last_became_active":"2024-05-09T08:48:17.600402+0000","last_became_peered":"2024-05-09T08:48:17.600402+0000","last_unstale":"2024-05-09T08:48:23.888419+0000","last_undegraded":"2024-05-09T08:48:23.888419+0000","last_fullsized":"2024-05-09T08:48:23.888419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:42:14.889227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603573+0000","last_change":"2024-05-09T08:48:18.373652+0000","last_active":"2024-05-09T08:48:23.603573+0000","last_peered":"2024-05-09T08:48:23.603573+0000","last_clean":"2024-05-09T08:48:23.603573+0000","last_became_active":"2024-05-09T08:48:18.373345+0000","last_became_peered":"2024-05-09T08:48:18.373345+0000","last_unstale":"2024-05-09T08:48:23.603573+0000","last_undegraded":"2024-05-09T08:48:23.603573+0000","last_fullsized":"2024-05-09T08:48:23.603573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:14:34.843305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538026+0000","last_change":"2024-05-09T08:48:22.378946+0000","last_active":"2024-05-09T08:48:25.538026+0000","last_peered":"2024-05-09T08:48:25.538026+0000","last_clean":"2024-05-09T08:48:25.538026+0000","last_became_active":"2024-05-09T08:48:22.378184+0000","last_became_peered":"2024-05-09T08:48:22.378184+0000","last_unstale":"2024-05-09T08:48:25.538026+0000","last_undegraded":"2024-05-09T08:48:25.538026+0000","last_fullsized":"2024-05-09T08:48:25.538026+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:04.063287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:36.279310+0000","last_change":"2024-05-09T08:48:17.592658+0000","last_active":"2024-05-09T08:48:36.279310+0000","last_peered":"2024-05-09T08:48:36.279310+0000","last_clean":"2024-05-09T08:48:36.279310+0000","last_became_active":"2024-05-09T08:48:17.592275+0000","last_became_peered":"2024-05-09T08:48:17.592275+0000","last_unstale":"2024-05-09T08:48:36.279310+0000","last_undegraded":"2024-05-09T08:48:36.279310+0000","last_fullsized":"2024-05-09T08:48:36.279310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:26:56.742079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608836+0000","last_change":"2024-05-09T08:48:18.374643+0000","last_active":"2024-05-09T08:48:23.608836+0000","last_peered":"2024-05-09T08:48:23.608836+0000","last_clean":"2024-05-09T08:48:23.608836+0000","last_became_active":"2024-05-09T08:48:18.374291+0000","last_became_peered":"2024-05-09T08:48:18.374291+0000","last_unstale":"2024-05-09T08:48:23.608836+0000","last_undegraded":"2024-05-09T08:48:23.608836+0000","last_fullsized":"2024-05-09T08:48:23.608836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:19:55.390772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.779227+0000","last_change":"2024-05-09T08:48:22.389568+0000","last_active":"2024-05-09T08:48:23.779227+0000","last_peered":"2024-05-09T08:48:23.779227+0000","last_clean":"2024-05-09T08:48:23.779227+0000","last_became_active":"2024-05-09T08:48:22.387192+0000","last_became_peered":"2024-05-09T08:48:22.387192+0000","last_unstale":"2024-05-09T08:48:23.779227+0000","last_undegraded":"2024-05-09T08:48:23.779227+0000","last_fullsized":"2024-05-09T08:48:23.779227+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:56.794208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:38.734761+0000","last_change":"2024-05-09T08:48:17.598760+0000","last_active":"2024-05-09T08:48:38.734761+0000","last_peered":"2024-05-09T08:48:38.734761+0000","last_clean":"2024-05-09T08:48:38.734761+0000","last_became_active":"2024-05-09T08:48:17.598192+0000","last_became_peered":"2024-05-09T08:48:17.598192+0000","last_unstale":"2024-05-09T08:48:38.734761+0000","last_undegraded":"2024-05-09T08:48:38.734761+0000","last_fullsized":"2024-05-09T08:48:38.734761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:29.007644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608634+0000","last_change":"2024-05-09T08:48:18.364570+0000","last_active":"2024-05-09T08:48:23.608634+0000","last_peered":"2024-05-09T08:48:23.608634+0000","last_clean":"2024-05-09T08:48:23.608634+0000","last_became_active":"2024-05-09T08:48:18.364191+0000","last_became_peered":"2024-05-09T08:48:18.364191+0000","last_unstale":"2024-05-09T08:48:23.608634+0000","last_undegraded":"2024-05-09T08:48:23.608634+0000","last_fullsized":"2024-05-09T08:48:23.608634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:49.267954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.604183+0000","last_change":"2024-05-09T08:48:22.380710+0000","last_active":"2024-05-09T08:48:23.604183+0000","last_peered":"2024-05-09T08:48:23.604183+0000","last_clean":"2024-05-09T08:48:23.604183+0000","last_became_active":"2024-05-09T08:48:22.380322+0000","last_became_peered":"2024-05-09T08:48:22.380322+0000","last_unstale":"2024-05-09T08:48:23.604183+0000","last_undegraded":"2024-05-09T08:48:23.604183+0000","last_fullsized":"2024-05-09T08:48:23.604183+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:36:44.479999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.681749+0000","last_change":"2024-05-09T08:48:17.599762+0000","last_active":"2024-05-09T08:48:24.681749+0000","last_peered":"2024-05-09T08:48:24.681749+0000","last_clean":"2024-05-09T08:48:24.681749+0000","last_became_active":"2024-05-09T08:48:17.599355+0000","last_became_peered":"2024-05-09T08:48:17.599355+0000","last_unstale":"2024-05-09T08:48:24.681749+0000","last_undegraded":"2024-05-09T08:48:24.681749+0000","last_fullsized":"2024-05-09T08:48:24.681749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:28:36.836403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608133+0000","last_change":"2024-05-09T08:48:18.372810+0000","last_active":"2024-05-09T08:48:23.608133+0000","last_peered":"2024-05-09T08:48:23.608133+0000","last_clean":"2024-05-09T08:48:23.608133+0000","last_became_active":"2024-05-09T08:48:18.372437+0000","last_became_peered":"2024-05-09T08:48:18.372437+0000","last_unstale":"2024-05-09T08:48:23.608133+0000","last_undegraded":"2024-05-09T08:48:23.608133+0000","last_fullsized":"2024-05-09T08:48:23.608133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:03:36.879719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608510+0000","last_change":"2024-05-09T08:48:22.379370+0000","last_active":"2024-05-09T08:48:23.608510+0000","last_peered":"2024-05-09T08:48:23.608510+0000","last_clean":"2024-05-09T08:48:23.608510+0000","last_became_active":"2024-05-09T08:48:22.378994+0000","last_became_peered":"2024-05-09T08:48:22.378994+0000","last_unstale":"2024-05-09T08:48:23.608510+0000","last_undegraded":"2024-05-09T08:48:23.608510+0000","last_fullsized":"2024-05-09T08:48:23.608510+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:08.005647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.682341+0000","last_change":"2024-05-09T08:48:17.594236+0000","last_active":"2024-05-09T08:48:24.682341+0000","last_peered":"2024-05-09T08:48:24.682341+0000","last_clean":"2024-05-09T08:48:24.682341+0000","last_became_active":"2024-05-09T08:48:17.593418+0000","last_became_peered":"2024-05-09T08:48:17.593418+0000","last_unstale":"2024-05-09T08:48:24.682341+0000","last_undegraded":"2024-05-09T08:48:24.682341+0000","last_fullsized":"2024-05-09T08:48:24.682341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:23.930421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538064+0000","last_change":"2024-05-09T08:48:18.366767+0000","last_active":"2024-05-09T08:48:25.538064+0000","last_peered":"2024-05-09T08:48:25.538064+0000","last_clean":"2024-05-09T08:48:25.538064+0000","last_became_active":"2024-05-09T08:48:18.366435+0000","last_became_peered":"2024-05-09T08:48:18.366435+0000","last_unstale":"2024-05-09T08:48:25.538064+0000","last_undegraded":"2024-05-09T08:48:25.538064+0000","last_fullsized":"2024-05-09T08:48:25.538064+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:45.111510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.682057+0000","last_change":"2024-05-09T08:48:22.378642+0000","last_active":"2024-05-09T08:48:24.682057+0000","last_peered":"2024-05-09T08:48:24.682057+0000","last_clean":"2024-05-09T08:48:24.682057+0000","last_became_active":"2024-05-09T08:48:22.378161+0000","last_became_peered":"2024-05-09T08:48:22.378161+0000","last_unstale":"2024-05-09T08:48:24.682057+0000","last_undegraded":"2024-05-09T08:48:24.682057+0000","last_fullsized":"2024-05-09T08:48:24.682057+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:53.149728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.778823+0000","last_change":"2024-05-09T08:48:17.599499+0000","last_active":"2024-05-09T08:48:23.778823+0000","last_peered":"2024-05-09T08:48:23.778823+0000","last_clean":"2024-05-09T08:48:23.778823+0000","last_became_active":"2024-05-09T08:48:17.599145+0000","last_became_peered":"2024-05-09T08:48:17.599145+0000","last_unstale":"2024-05-09T08:48:23.778823+0000","last_undegraded":"2024-05-09T08:48:23.778823+0000","last_fullsized":"2024-05-09T08:48:23.778823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:31:12.502004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538875+0000","last_change":"2024-05-09T08:48:18.372355+0000","last_active":"2024-05-09T08:48:25.538875+0000","last_peered":"2024-05-09T08:48:25.538875+0000","last_clean":"2024-05-09T08:48:25.538875+0000","last_became_active":"2024-05-09T08:48:18.367743+0000","last_became_peered":"2024-05-09T08:48:18.367743+0000","last_unstale":"2024-05-09T08:48:25.538875+0000","last_undegraded":"2024-05-09T08:48:25.538875+0000","last_fullsized":"2024-05-09T08:48:25.538875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:31:12.174350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603861+0000","last_change":"2024-05-09T08:48:22.384064+0000","last_active":"2024-05-09T08:48:23.603861+0000","last_peered":"2024-05-09T08:48:23.603861+0000","last_clean":"2024-05-09T08:48:23.603861+0000","last_became_active":"2024-05-09T08:48:22.383670+0000","last_became_peered":"2024-05-09T08:48:22.383670+0000","last_unstale":"2024-05-09T08:48:23.603861+0000","last_undegraded":"2024-05-09T08:48:23.603861+0000","last_fullsized":"2024-05-09T08:48:23.603861+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:51:35.000827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608520+0000","last_change":"2024-05-09T08:48:17.601596+0000","last_active":"2024-05-09T08:48:23.608520+0000","last_peered":"2024-05-09T08:48:23.608520+0000","last_clean":"2024-05-09T08:48:23.608520+0000","last_became_active":"2024-05-09T08:48:17.601250+0000","last_became_peered":"2024-05-09T08:48:17.601250+0000","last_unstale":"2024-05-09T08:48:23.608520+0000","last_undegraded":"2024-05-09T08:48:23.608520+0000","last_fullsized":"2024-05-09T08:48:23.608520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:48.237336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538623+0000","last_change":"2024-05-09T08:48:18.368751+0000","last_active":"2024-05-09T08:48:25.538623+0000","last_peered":"2024-05-09T08:48:25.538623+0000","last_clean":"2024-05-09T08:48:25.538623+0000","last_became_active":"2024-05-09T08:48:18.367463+0000","last_became_peered":"2024-05-09T08:48:18.367463+0000","last_unstale":"2024-05-09T08:48:25.538623+0000","last_undegraded":"2024-05-09T08:48:25.538623+0000","last_fullsized":"2024-05-09T08:48:25.538623+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:19:17.696602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.604032+0000","last_change":"2024-05-09T08:48:22.385025+0000","last_active":"2024-05-09T08:48:23.604032+0000","last_peered":"2024-05-09T08:48:23.604032+0000","last_clean":"2024-05-09T08:48:23.604032+0000","last_became_active":"2024-05-09T08:48:22.384633+0000","last_became_peered":"2024-05-09T08:48:22.384633+0000","last_unstale":"2024-05-09T08:48:23.604032+0000","last_undegraded":"2024-05-09T08:48:23.604032+0000","last_fullsized":"2024-05-09T08:48:23.604032+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:17.804075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608209+0000","last_change":"2024-05-09T08:48:18.376741+0000","last_active":"2024-05-09T08:48:23.608209+0000","last_peered":"2024-05-09T08:48:23.608209+0000","last_clean":"2024-05-09T08:48:23.608209+0000","last_became_active":"2024-05-09T08:48:18.376376+0000","last_became_peered":"2024-05-09T08:48:18.376376+0000","last_unstale":"2024-05-09T08:48:23.608209+0000","last_undegraded":"2024-05-09T08:48:23.608209+0000","last_fullsized":"2024-05-09T08:48:23.608209+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:58:31.717555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603257+0000","last_change":"2024-05-09T08:48:17.598316+0000","last_active":"2024-05-09T08:48:23.603257+0000","last_peered":"2024-05-09T08:48:23.603257+0000","last_clean":"2024-05-09T08:48:23.603257+0000","last_became_active":"2024-05-09T08:48:17.598022+0000","last_became_peered":"2024-05-09T08:48:17.598022+0000","last_unstale":"2024-05-09T08:48:23.603257+0000","last_undegraded":"2024-05-09T08:48:23.603257+0000","last_fullsized":"2024-05-09T08:48:23.603257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:04:10.454894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.609157+0000","last_change":"2024-05-09T08:48:22.374123+0000","last_active":"2024-05-09T08:48:23.609157+0000","last_peered":"2024-05-09T08:48:23.609157+0000","last_clean":"2024-05-09T08:48:23.609157+0000","last_became_active":"2024-05-09T08:48:22.373725+0000","last_became_peered":"2024-05-09T08:48:22.373725+0000","last_unstale":"2024-05-09T08:48:23.609157+0000","last_undegraded":"2024-05-09T08:48:23.609157+0000","last_fullsized":"2024-05-09T08:48:23.609157+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:32:53.171556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537571+0000","last_change":"2024-05-09T08:48:18.379203+0000","last_active":"2024-05-09T08:48:25.537571+0000","last_peered":"2024-05-09T08:48:25.537571+0000","last_clean":"2024-05-09T08:48:25.537571+0000","last_became_active":"2024-05-09T08:48:18.378885+0000","last_became_peered":"2024-05-09T08:48:18.378885+0000","last_unstale":"2024-05-09T08:48:25.537571+0000","last_undegraded":"2024-05-09T08:48:25.537571+0000","last_fullsized":"2024-05-09T08:48:25.537571+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:40.588263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.742433+0000","last_change":"2024-05-09T08:48:17.593520+0000","last_active":"2024-05-09T08:48:37.742433+0000","last_peered":"2024-05-09T08:48:37.742433+0000","last_clean":"2024-05-09T08:48:37.742433+0000","last_became_active":"2024-05-09T08:48:17.593127+0000","last_became_peered":"2024-05-09T08:48:17.593127+0000","last_unstale":"2024-05-09T08:48:37.742433+0000","last_undegraded":"2024-05-09T08:48:37.742433+0000","last_fullsized":"2024-05-09T08:48:37.742433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:34:29.567166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":356,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608435+0000","last_change":"2024-05-09T08:47:05.718649+0000","last_active":"2024-05-09T08:48:23.608435+0000","last_peered":"2024-05-09T08:48:23.608435+0000","last_clean":"2024-05-09T08:48:23.608435+0000","last_became_active":"2024-05-09T08:47:05.718088+0000","last_became_peered":"2024-05-09T08:47:05.718088+0000","last_unstale":"2024-05-09T08:48:23.608435+0000","last_undegraded":"2024-05-09T08:48:23.608435+0000","last_fullsized":"2024-05-09T08:48:23.608435+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:43:39.457880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:43:39.457880+0000","last_clean_scrub_stamp":"2024-05-09T08:43:39.457880+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:14.227590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608605+0000","last_change":"2024-05-09T08:48:22.376751+0000","last_active":"2024-05-09T08:48:23.608605+0000","last_peered":"2024-05-09T08:48:23.608605+0000","last_clean":"2024-05-09T08:48:23.608605+0000","last_became_active":"2024-05-09T08:48:22.376322+0000","last_became_peered":"2024-05-09T08:48:22.376322+0000","last_unstale":"2024-05-09T08:48:23.608605+0000","last_undegraded":"2024-05-09T08:48:23.608605+0000","last_fullsized":"2024-05-09T08:48:23.608605+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:08:07.859034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603239+0000","last_change":"2024-05-09T08:48:18.364063+0000","last_active":"2024-05-09T08:48:23.603239+0000","last_peered":"2024-05-09T08:48:23.603239+0000","last_clean":"2024-05-09T08:48:23.603239+0000","last_became_active":"2024-05-09T08:48:18.363694+0000","last_became_peered":"2024-05-09T08:48:18.363694+0000","last_unstale":"2024-05-09T08:48:23.603239+0000","last_undegraded":"2024-05-09T08:48:23.603239+0000","last_fullsized":"2024-05-09T08:48:23.603239+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:42.536329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608626+0000","last_change":"2024-05-09T08:48:17.599787+0000","last_active":"2024-05-09T08:48:23.608626+0000","last_peered":"2024-05-09T08:48:23.608626+0000","last_clean":"2024-05-09T08:48:23.608626+0000","last_became_active":"2024-05-09T08:48:17.599248+0000","last_became_peered":"2024-05-09T08:48:17.599248+0000","last_unstale":"2024-05-09T08:48:23.608626+0000","last_undegraded":"2024-05-09T08:48:23.608626+0000","last_fullsized":"2024-05-09T08:48:23.608626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:17.448404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537914+0000","last_change":"2024-05-09T08:48:22.378913+0000","last_active":"2024-05-09T08:48:25.537914+0000","last_peered":"2024-05-09T08:48:25.537914+0000","last_clean":"2024-05-09T08:48:25.537914+0000","last_became_active":"2024-05-09T08:48:22.378441+0000","last_became_peered":"2024-05-09T08:48:22.378441+0000","last_unstale":"2024-05-09T08:48:25.537914+0000","last_undegraded":"2024-05-09T08:48:25.537914+0000","last_fullsized":"2024-05-09T08:48:25.537914+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:49.623778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.682336+0000","last_change":"2024-05-09T08:48:17.600121+0000","last_active":"2024-05-09T08:48:24.682336+0000","last_peered":"2024-05-09T08:48:24.682336+0000","last_clean":"2024-05-09T08:48:24.682336+0000","last_became_active":"2024-05-09T08:48:17.599800+0000","last_became_peered":"2024-05-09T08:48:17.599800+0000","last_unstale":"2024-05-09T08:48:24.682336+0000","last_undegraded":"2024-05-09T08:48:24.682336+0000","last_fullsized":"2024-05-09T08:48:24.682336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:56.607393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603862+0000","last_change":"2024-05-09T08:48:18.366612+0000","last_active":"2024-05-09T08:48:23.603862+0000","last_peered":"2024-05-09T08:48:23.603862+0000","last_clean":"2024-05-09T08:48:23.603862+0000","last_became_active":"2024-05-09T08:48:18.366266+0000","last_became_peered":"2024-05-09T08:48:18.366266+0000","last_unstale":"2024-05-09T08:48:23.603862+0000","last_undegraded":"2024-05-09T08:48:23.603862+0000","last_fullsized":"2024-05-09T08:48:23.603862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:35:01.663456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537615+0000","last_change":"2024-05-09T08:48:22.384096+0000","last_active":"2024-05-09T08:48:25.537615+0000","last_peered":"2024-05-09T08:48:25.537615+0000","last_clean":"2024-05-09T08:48:25.537615+0000","last_became_active":"2024-05-09T08:48:22.383656+0000","last_became_peered":"2024-05-09T08:48:22.383656+0000","last_unstale":"2024-05-09T08:48:25.537615+0000","last_undegraded":"2024-05-09T08:48:25.537615+0000","last_fullsized":"2024-05-09T08:48:25.537615+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:13.622377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.779204+0000","last_change":"2024-05-09T08:48:17.593684+0000","last_active":"2024-05-09T08:48:23.779204+0000","last_peered":"2024-05-09T08:48:23.779204+0000","last_clean":"2024-05-09T08:48:23.779204+0000","last_became_active":"2024-05-09T08:48:17.593358+0000","last_became_peered":"2024-05-09T08:48:17.593358+0000","last_unstale":"2024-05-09T08:48:23.779204+0000","last_undegraded":"2024-05-09T08:48:23.779204+0000","last_fullsized":"2024-05-09T08:48:23.779204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:51:36.023421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538003+0000","last_change":"2024-05-09T08:48:18.374498+0000","last_active":"2024-05-09T08:48:25.538003+0000","last_peered":"2024-05-09T08:48:25.538003+0000","last_clean":"2024-05-09T08:48:25.538003+0000","last_became_active":"2024-05-09T08:48:18.374142+0000","last_became_peered":"2024-05-09T08:48:18.374142+0000","last_unstale":"2024-05-09T08:48:25.538003+0000","last_undegraded":"2024-05-09T08:48:25.538003+0000","last_fullsized":"2024-05-09T08:48:25.538003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:53.874419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888360+0000","last_change":"2024-05-09T08:48:22.378330+0000","last_active":"2024-05-09T08:48:23.888360+0000","last_peered":"2024-05-09T08:48:23.888360+0000","last_clean":"2024-05-09T08:48:23.888360+0000","last_became_active":"2024-05-09T08:48:22.377218+0000","last_became_peered":"2024-05-09T08:48:22.377218+0000","last_unstale":"2024-05-09T08:48:23.888360+0000","last_undegraded":"2024-05-09T08:48:23.888360+0000","last_fullsized":"2024-05-09T08:48:23.888360+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:41.650168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:38.747162+0000","last_change":"2024-05-09T08:48:17.601410+0000","last_active":"2024-05-09T08:48:38.747162+0000","last_peered":"2024-05-09T08:48:38.747162+0000","last_clean":"2024-05-09T08:48:38.747162+0000","last_became_active":"2024-05-09T08:48:17.601075+0000","last_became_peered":"2024-05-09T08:48:17.601075+0000","last_unstale":"2024-05-09T08:48:38.747162+0000","last_undegraded":"2024-05-09T08:48:38.747162+0000","last_fullsized":"2024-05-09T08:48:38.747162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:45.760280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782803+0000","last_change":"2024-05-09T08:48:18.379590+0000","last_active":"2024-05-09T08:48:25.782803+0000","last_peered":"2024-05-09T08:48:25.782803+0000","last_clean":"2024-05-09T08:48:25.782803+0000","last_became_active":"2024-05-09T08:48:18.379187+0000","last_became_peered":"2024-05-09T08:48:18.379187+0000","last_unstale":"2024-05-09T08:48:25.782803+0000","last_undegraded":"2024-05-09T08:48:25.782803+0000","last_fullsized":"2024-05-09T08:48:25.782803+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:40:02.630449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538370+0000","last_change":"2024-05-09T08:48:22.386437+0000","last_active":"2024-05-09T08:48:25.538370+0000","last_peered":"2024-05-09T08:48:25.538370+0000","last_clean":"2024-05-09T08:48:25.538370+0000","last_became_active":"2024-05-09T08:48:22.385973+0000","last_became_peered":"2024-05-09T08:48:22.385973+0000","last_unstale":"2024-05-09T08:48:25.538370+0000","last_undegraded":"2024-05-09T08:48:25.538370+0000","last_fullsized":"2024-05-09T08:48:25.538370+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:11:37.338641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.834442+0000","last_change":"2024-05-09T08:48:17.598586+0000","last_active":"2024-05-09T08:48:24.834442+0000","last_peered":"2024-05-09T08:48:24.834442+0000","last_clean":"2024-05-09T08:48:24.834442+0000","last_became_active":"2024-05-09T08:48:17.597851+0000","last_became_peered":"2024-05-09T08:48:17.597851+0000","last_unstale":"2024-05-09T08:48:24.834442+0000","last_undegraded":"2024-05-09T08:48:24.834442+0000","last_fullsized":"2024-05-09T08:48:24.834442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:28:22.996999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.681748+0000","last_change":"2024-05-09T08:48:18.367319+0000","last_active":"2024-05-09T08:48:24.681748+0000","last_peered":"2024-05-09T08:48:24.681748+0000","last_clean":"2024-05-09T08:48:24.681748+0000","last_became_active":"2024-05-09T08:48:18.366034+0000","last_became_peered":"2024-05-09T08:48:18.366034+0000","last_unstale":"2024-05-09T08:48:24.681748+0000","last_undegraded":"2024-05-09T08:48:24.681748+0000","last_fullsized":"2024-05-09T08:48:24.681748+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:22.998089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.779329+0000","last_change":"2024-05-09T08:48:22.389129+0000","last_active":"2024-05-09T08:48:23.779329+0000","last_peered":"2024-05-09T08:48:23.779329+0000","last_clean":"2024-05-09T08:48:23.779329+0000","last_became_active":"2024-05-09T08:48:22.386639+0000","last_became_peered":"2024-05-09T08:48:22.386639+0000","last_unstale":"2024-05-09T08:48:23.779329+0000","last_undegraded":"2024-05-09T08:48:23.779329+0000","last_fullsized":"2024-05-09T08:48:23.779329+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:47:07.257464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537637+0000","last_change":"2024-05-09T08:48:17.595255+0000","last_active":"2024-05-09T08:48:25.537637+0000","last_peered":"2024-05-09T08:48:25.537637+0000","last_clean":"2024-05-09T08:48:25.537637+0000","last_became_active":"2024-05-09T08:48:17.594882+0000","last_became_peered":"2024-05-09T08:48:17.594882+0000","last_unstale":"2024-05-09T08:48:25.537637+0000","last_undegraded":"2024-05-09T08:48:25.537637+0000","last_fullsized":"2024-05-09T08:48:25.537637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:49.845326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537798+0000","last_change":"2024-05-09T08:48:18.378580+0000","last_active":"2024-05-09T08:48:25.537798+0000","last_peered":"2024-05-09T08:48:25.537798+0000","last_clean":"2024-05-09T08:48:25.537798+0000","last_became_active":"2024-05-09T08:48:18.378205+0000","last_became_peered":"2024-05-09T08:48:18.378205+0000","last_unstale":"2024-05-09T08:48:25.537798+0000","last_undegraded":"2024-05-09T08:48:25.537798+0000","last_fullsized":"2024-05-09T08:48:25.537798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:50.103195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538026+0000","last_change":"2024-05-09T08:48:22.378494+0000","last_active":"2024-05-09T08:48:25.538026+0000","last_peered":"2024-05-09T08:48:25.538026+0000","last_clean":"2024-05-09T08:48:25.538026+0000","last_became_active":"2024-05-09T08:48:22.378076+0000","last_became_peered":"2024-05-09T08:48:22.378076+0000","last_unstale":"2024-05-09T08:48:25.538026+0000","last_undegraded":"2024-05-09T08:48:25.538026+0000","last_fullsized":"2024-05-09T08:48:25.538026+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:40:09.079968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888247+0000","last_change":"2024-05-09T08:48:17.601433+0000","last_active":"2024-05-09T08:48:23.888247+0000","last_peered":"2024-05-09T08:48:23.888247+0000","last_clean":"2024-05-09T08:48:23.888247+0000","last_became_active":"2024-05-09T08:48:17.601128+0000","last_became_peered":"2024-05-09T08:48:17.601128+0000","last_unstale":"2024-05-09T08:48:23.888247+0000","last_undegraded":"2024-05-09T08:48:23.888247+0000","last_fullsized":"2024-05-09T08:48:23.888247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:33:26.207437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537889+0000","last_change":"2024-05-09T08:48:18.373569+0000","last_active":"2024-05-09T08:48:25.537889+0000","last_peered":"2024-05-09T08:48:25.537889+0000","last_clean":"2024-05-09T08:48:25.537889+0000","last_became_active":"2024-05-09T08:48:18.373238+0000","last_became_peered":"2024-05-09T08:48:18.373238+0000","last_unstale":"2024-05-09T08:48:25.537889+0000","last_undegraded":"2024-05-09T08:48:25.537889+0000","last_fullsized":"2024-05-09T08:48:25.537889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:17:41.629419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538071+0000","last_change":"2024-05-09T08:48:22.381523+0000","last_active":"2024-05-09T08:48:25.538071+0000","last_peered":"2024-05-09T08:48:25.538071+0000","last_clean":"2024-05-09T08:48:25.538071+0000","last_became_active":"2024-05-09T08:48:22.380626+0000","last_became_peered":"2024-05-09T08:48:22.380626+0000","last_unstale":"2024-05-09T08:48:25.538071+0000","last_undegraded":"2024-05-09T08:48:25.538071+0000","last_fullsized":"2024-05-09T08:48:25.538071+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:47:45.092232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.681860+0000","last_change":"2024-05-09T08:48:17.601479+0000","last_active":"2024-05-09T08:48:24.681860+0000","last_peered":"2024-05-09T08:48:24.681860+0000","last_clean":"2024-05-09T08:48:24.681860+0000","last_became_active":"2024-05-09T08:48:17.601140+0000","last_became_peered":"2024-05-09T08:48:17.601140+0000","last_unstale":"2024-05-09T08:48:24.681860+0000","last_undegraded":"2024-05-09T08:48:24.681860+0000","last_fullsized":"2024-05-09T08:48:24.681860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:38:24.011209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608523+0000","last_change":"2024-05-09T08:48:18.376890+0000","last_active":"2024-05-09T08:48:23.608523+0000","last_peered":"2024-05-09T08:48:23.608523+0000","last_clean":"2024-05-09T08:48:23.608523+0000","last_became_active":"2024-05-09T08:48:18.376472+0000","last_became_peered":"2024-05-09T08:48:18.376472+0000","last_unstale":"2024-05-09T08:48:23.608523+0000","last_undegraded":"2024-05-09T08:48:23.608523+0000","last_fullsized":"2024-05-09T08:48:23.608523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:40:16.238539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.834831+0000","last_change":"2024-05-09T08:48:22.374743+0000","last_active":"2024-05-09T08:48:24.834831+0000","last_peered":"2024-05-09T08:48:24.834831+0000","last_clean":"2024-05-09T08:48:24.834831+0000","last_became_active":"2024-05-09T08:48:22.374324+0000","last_became_peered":"2024-05-09T08:48:22.374324+0000","last_unstale":"2024-05-09T08:48:24.834831+0000","last_undegraded":"2024-05-09T08:48:24.834831+0000","last_fullsized":"2024-05-09T08:48:24.834831+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:48.073647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888765+0000","last_change":"2024-05-09T08:48:17.604700+0000","last_active":"2024-05-09T08:48:23.888765+0000","last_peered":"2024-05-09T08:48:23.888765+0000","last_clean":"2024-05-09T08:48:23.888765+0000","last_became_active":"2024-05-09T08:48:17.604334+0000","last_became_peered":"2024-05-09T08:48:17.604334+0000","last_unstale":"2024-05-09T08:48:23.888765+0000","last_undegraded":"2024-05-09T08:48:23.888765+0000","last_fullsized":"2024-05-09T08:48:23.888765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:15:37.346232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.609221+0000","last_change":"2024-05-09T08:48:18.375769+0000","last_active":"2024-05-09T08:48:23.609221+0000","last_peered":"2024-05-09T08:48:23.609221+0000","last_clean":"2024-05-09T08:48:23.609221+0000","last_became_active":"2024-05-09T08:48:18.375059+0000","last_became_peered":"2024-05-09T08:48:18.375059+0000","last_unstale":"2024-05-09T08:48:23.609221+0000","last_undegraded":"2024-05-09T08:48:23.609221+0000","last_fullsized":"2024-05-09T08:48:23.609221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:51.001240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782227+0000","last_change":"2024-05-09T08:48:22.385306+0000","last_active":"2024-05-09T08:48:25.782227+0000","last_peered":"2024-05-09T08:48:25.782227+0000","last_clean":"2024-05-09T08:48:25.782227+0000","last_became_active":"2024-05-09T08:48:22.383890+0000","last_became_peered":"2024-05-09T08:48:22.383890+0000","last_unstale":"2024-05-09T08:48:25.782227+0000","last_undegraded":"2024-05-09T08:48:25.782227+0000","last_fullsized":"2024-05-09T08:48:25.782227+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:50:52.369399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.729727+0000","last_change":"2024-05-09T08:48:17.591627+0000","last_active":"2024-05-09T08:48:37.729727+0000","last_peered":"2024-05-09T08:48:37.729727+0000","last_clean":"2024-05-09T08:48:37.729727+0000","last_became_active":"2024-05-09T08:48:17.591250+0000","last_became_peered":"2024-05-09T08:48:17.591250+0000","last_unstale":"2024-05-09T08:48:37.729727+0000","last_undegraded":"2024-05-09T08:48:37.729727+0000","last_fullsized":"2024-05-09T08:48:37.729727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:40:59.748892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608916+0000","last_change":"2024-05-09T08:48:18.372256+0000","last_active":"2024-05-09T08:48:23.608916+0000","last_peered":"2024-05-09T08:48:23.608916+0000","last_clean":"2024-05-09T08:48:23.608916+0000","last_became_active":"2024-05-09T08:48:18.371895+0000","last_became_peered":"2024-05-09T08:48:18.371895+0000","last_unstale":"2024-05-09T08:48:23.608916+0000","last_undegraded":"2024-05-09T08:48:23.608916+0000","last_fullsized":"2024-05-09T08:48:23.608916+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:25.067587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.607835+0000","last_change":"2024-05-09T08:48:22.373684+0000","last_active":"2024-05-09T08:48:23.607835+0000","last_peered":"2024-05-09T08:48:23.607835+0000","last_clean":"2024-05-09T08:48:23.607835+0000","last_became_active":"2024-05-09T08:48:22.373000+0000","last_became_peered":"2024-05-09T08:48:22.373000+0000","last_unstale":"2024-05-09T08:48:23.607835+0000","last_undegraded":"2024-05-09T08:48:23.607835+0000","last_fullsized":"2024-05-09T08:48:23.607835+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:53:14.416837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603986+0000","last_change":"2024-05-09T08:48:17.596910+0000","last_active":"2024-05-09T08:48:23.603986+0000","last_peered":"2024-05-09T08:48:23.603986+0000","last_clean":"2024-05-09T08:48:23.603986+0000","last_became_active":"2024-05-09T08:48:17.596605+0000","last_became_peered":"2024-05-09T08:48:17.596605+0000","last_unstale":"2024-05-09T08:48:23.603986+0000","last_undegraded":"2024-05-09T08:48:23.603986+0000","last_fullsized":"2024-05-09T08:48:23.603986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:05:25.880082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.609245+0000","last_change":"2024-05-09T08:48:18.370354+0000","last_active":"2024-05-09T08:48:23.609245+0000","last_peered":"2024-05-09T08:48:23.609245+0000","last_clean":"2024-05-09T08:48:23.609245+0000","last_became_active":"2024-05-09T08:48:18.370043+0000","last_became_peered":"2024-05-09T08:48:18.370043+0000","last_unstale":"2024-05-09T08:48:23.609245+0000","last_undegraded":"2024-05-09T08:48:23.609245+0000","last_fullsized":"2024-05-09T08:48:23.609245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:21:34.066162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537971+0000","last_change":"2024-05-09T08:48:18.376431+0000","last_active":"2024-05-09T08:48:25.537971+0000","last_peered":"2024-05-09T08:48:25.537971+0000","last_clean":"2024-05-09T08:48:25.537971+0000","last_became_active":"2024-05-09T08:48:18.375598+0000","last_became_peered":"2024-05-09T08:48:18.375598+0000","last_unstale":"2024-05-09T08:48:25.537971+0000","last_undegraded":"2024-05-09T08:48:25.537971+0000","last_fullsized":"2024-05-09T08:48:25.537971+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:51:13.481219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888610+0000","last_change":"2024-05-09T08:48:18.371460+0000","last_active":"2024-05-09T08:48:23.888610+0000","last_peered":"2024-05-09T08:48:23.888610+0000","last_clean":"2024-05-09T08:48:23.888610+0000","last_became_active":"2024-05-09T08:48:18.371091+0000","last_became_peered":"2024-05-09T08:48:18.371091+0000","last_unstale":"2024-05-09T08:48:23.888610+0000","last_undegraded":"2024-05-09T08:48:23.888610+0000","last_fullsized":"2024-05-09T08:48:23.888610+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:02.829902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888201+0000","last_change":"2024-05-09T08:48:18.369453+0000","last_active":"2024-05-09T08:48:23.888201+0000","last_peered":"2024-05-09T08:48:23.888201+0000","last_clean":"2024-05-09T08:48:23.888201+0000","last_became_active":"2024-05-09T08:48:18.368966+0000","last_became_peered":"2024-05-09T08:48:18.368966+0000","last_unstale":"2024-05-09T08:48:23.888201+0000","last_undegraded":"2024-05-09T08:48:23.888201+0000","last_fullsized":"2024-05-09T08:48:23.888201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:23:14.359222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782577+0000","last_change":"2024-05-09T08:48:18.378815+0000","last_active":"2024-05-09T08:48:25.782577+0000","last_peered":"2024-05-09T08:48:25.782577+0000","last_clean":"2024-05-09T08:48:25.782577+0000","last_became_active":"2024-05-09T08:48:18.378479+0000","last_became_peered":"2024-05-09T08:48:18.378479+0000","last_unstale":"2024-05-09T08:48:25.782577+0000","last_undegraded":"2024-05-09T08:48:25.782577+0000","last_fullsized":"2024-05-09T08:48:25.782577+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:38:50.133239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603926+0000","last_change":"2024-05-09T08:48:18.365696+0000","last_active":"2024-05-09T08:48:23.603926+0000","last_peered":"2024-05-09T08:48:23.603926+0000","last_clean":"2024-05-09T08:48:23.603926+0000","last_became_active":"2024-05-09T08:48:18.365358+0000","last_became_peered":"2024-05-09T08:48:18.365358+0000","last_unstale":"2024-05-09T08:48:23.603926+0000","last_undegraded":"2024-05-09T08:48:23.603926+0000","last_fullsized":"2024-05-09T08:48:23.603926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:04.064150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538124+0000","last_change":"2024-05-09T08:48:18.368911+0000","last_active":"2024-05-09T08:48:25.538124+0000","last_peered":"2024-05-09T08:48:25.538124+0000","last_clean":"2024-05-09T08:48:25.538124+0000","last_became_active":"2024-05-09T08:48:18.368591+0000","last_became_peered":"2024-05-09T08:48:18.368591+0000","last_unstale":"2024-05-09T08:48:25.538124+0000","last_undegraded":"2024-05-09T08:48:25.538124+0000","last_fullsized":"2024-05-09T08:48:25.538124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:34:09.692168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888820+0000","last_change":"2024-05-09T08:48:18.373629+0000","last_active":"2024-05-09T08:48:23.888820+0000","last_peered":"2024-05-09T08:48:23.888820+0000","last_clean":"2024-05-09T08:48:23.888820+0000","last_became_active":"2024-05-09T08:48:18.372992+0000","last_became_peered":"2024-05-09T08:48:18.372992+0000","last_unstale":"2024-05-09T08:48:23.888820+0000","last_undegraded":"2024-05-09T08:48:23.888820+0000","last_fullsized":"2024-05-09T08:48:23.888820+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:42.532250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538104+0000","last_change":"2024-05-09T08:48:18.382423+0000","last_active":"2024-05-09T08:48:25.538104+0000","last_peered":"2024-05-09T08:48:25.538104+0000","last_clean":"2024-05-09T08:48:25.538104+0000","last_became_active":"2024-05-09T08:48:18.382088+0000","last_became_peered":"2024-05-09T08:48:18.382088+0000","last_unstale":"2024-05-09T08:48:25.538104+0000","last_undegraded":"2024-05-09T08:48:25.538104+0000","last_fullsized":"2024-05-09T08:48:25.538104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:09:06.342557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608067+0000","last_change":"2024-05-09T08:48:18.370460+0000","last_active":"2024-05-09T08:48:23.608067+0000","last_peered":"2024-05-09T08:48:23.608067+0000","last_clean":"2024-05-09T08:48:23.608067+0000","last_became_active":"2024-05-09T08:48:18.370033+0000","last_became_peered":"2024-05-09T08:48:18.370033+0000","last_unstale":"2024-05-09T08:48:23.608067+0000","last_undegraded":"2024-05-09T08:48:23.608067+0000","last_fullsized":"2024-05-09T08:48:23.608067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:21:28.622511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537729+0000","last_change":"2024-05-09T08:48:18.371410+0000","last_active":"2024-05-09T08:48:25.537729+0000","last_peered":"2024-05-09T08:48:25.537729+0000","last_clean":"2024-05-09T08:48:25.537729+0000","last_became_active":"2024-05-09T08:48:18.371067+0000","last_became_peered":"2024-05-09T08:48:18.371067+0000","last_unstale":"2024-05-09T08:48:25.537729+0000","last_undegraded":"2024-05-09T08:48:25.537729+0000","last_fullsized":"2024-05-09T08:48:25.537729+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:35:11.445992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.834876+0000","last_change":"2024-05-09T08:48:18.382873+0000","last_active":"2024-05-09T08:48:24.834876+0000","last_peered":"2024-05-09T08:48:24.834876+0000","last_clean":"2024-05-09T08:48:24.834876+0000","last_became_active":"2024-05-09T08:48:18.382366+0000","last_became_peered":"2024-05-09T08:48:18.382366+0000","last_unstale":"2024-05-09T08:48:24.834876+0000","last_undegraded":"2024-05-09T08:48:24.834876+0000","last_fullsized":"2024-05-09T08:48:24.834876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:23.343131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603590+0000","last_change":"2024-05-09T08:48:18.370821+0000","last_active":"2024-05-09T08:48:23.603590+0000","last_peered":"2024-05-09T08:48:23.603590+0000","last_clean":"2024-05-09T08:48:23.603590+0000","last_became_active":"2024-05-09T08:48:18.368208+0000","last_became_peered":"2024-05-09T08:48:18.368208+0000","last_unstale":"2024-05-09T08:48:23.603590+0000","last_undegraded":"2024-05-09T08:48:23.603590+0000","last_fullsized":"2024-05-09T08:48:23.603590+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:14.633938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538184+0000","last_change":"2024-05-09T08:48:18.371626+0000","last_active":"2024-05-09T08:48:25.538184+0000","last_peered":"2024-05-09T08:48:25.538184+0000","last_clean":"2024-05-09T08:48:25.538184+0000","last_became_active":"2024-05-09T08:48:18.371081+0000","last_became_peered":"2024-05-09T08:48:18.371081+0000","last_unstale":"2024-05-09T08:48:25.538184+0000","last_undegraded":"2024-05-09T08:48:25.538184+0000","last_fullsized":"2024-05-09T08:48:25.538184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:27.372934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":210578,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7765,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":18,"num_read_kb":17,"num_write":97,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":21335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350993,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":482037,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612630,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":1076633,"data_compressed":50263,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776155,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":481629,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037792,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":481449,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299428,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1024000,"data_stored":1071795,"data_compressed":50263,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462954,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":480845,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":484687,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":481539,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":484580,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":978944,"data_stored":480949,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888117,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":481511,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724591,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":1072689,"data_compressed":50263,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34003,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34003,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:48:41.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:48:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.260+0000 7fa277a8c640 1 -- 172.21.15.98:0/3754961394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2701023a0 msgr2=0x7fa270108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.260+0000 7fa277a8c640 1 --2- 172.21.15.98:0/3754961394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2701023a0 0x7fa270108c30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa268009ca0 tx=0x7fa26802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.260+0000 7fa277a8c640 1 -- 172.21.15.98:0/3754961394 shutdown_connections 2024-05-09T08:48:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.260+0000 7fa277a8c640 1 --2- 172.21.15.98:0/3754961394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2701023a0 0x7fa270108c30 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.260+0000 7fa277a8c640 1 --2- 172.21.15.98:0/3754961394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2701019e0 0x7fa270101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.260+0000 7fa277a8c640 1 --2- 172.21.15.98:0/3754961394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2701007f0 0x7fa270100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.260+0000 7fa277a8c640 1 -- 172.21.15.98:0/3754961394 >> 172.21.15.98:0/3754961394 conn(0x7fa2700fbd80 msgr2=0x7fa2700fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.260+0000 7fa277a8c640 1 -- 172.21.15.98:0/3754961394 shutdown_connections 2024-05-09T08:48:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.260+0000 7fa277a8c640 1 -- 172.21.15.98:0/3754961394 wait complete. 2024-05-09T08:48:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa277a8c640 1 Processor -- start 2024-05-09T08:48:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa277a8c640 1 -- start start 2024-05-09T08:48:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa277a8c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2701007f0 0x7fa270196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa277a8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2701019e0 0x7fa270196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa277a8c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2701023a0 0x7fa27019dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa277a8c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa27010c3e0 con 0x7fa2701019e0 2024-05-09T08:48:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa277a8c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa27010c260 con 0x7fa2701007f0 2024-05-09T08:48:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa277a8c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa27010c560 con 0x7fa2701023a0 2024-05-09T08:48:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa275027640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2701019e0 0x7fa270196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa275027640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2701019e0 0x7fa270196f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55438/0 (socket says 172.21.15.98:55438) 2024-05-09T08:48:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa275027640 1 -- 172.21.15.98:0/3600181480 learned_addr learned my addr 172.21.15.98:0/3600181480 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa275828640 1 --2- 172.21.15.98:0/3600181480 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2701007f0 0x7fa270196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa276029640 1 --2- 172.21.15.98:0/3600181480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2701023a0 0x7fa27019dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa275027640 1 -- 172.21.15.98:0/3600181480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2701023a0 msgr2=0x7fa27019dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa275027640 1 --2- 172.21.15.98:0/3600181480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2701023a0 0x7fa27019dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa275027640 1 -- 172.21.15.98:0/3600181480 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2701007f0 msgr2=0x7fa270196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa275027640 1 --2- 172.21.15.98:0/3600181480 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2701007f0 0x7fa270196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa275027640 1 -- 172.21.15.98:0/3600181480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa27019e650 con 0x7fa2701019e0 2024-05-09T08:48:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.261+0000 7fa275027640 1 --2- 172.21.15.98:0/3600181480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2701019e0 0x7fa270196f70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa26c00d950 tx=0x7fa26c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.262+0000 7fa266ffd640 1 -- 172.21.15.98:0/3600181480 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c017030 con 0x7fa2701019e0 2024-05-09T08:48:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.262+0000 7fa266ffd640 1 -- 172.21.15.98:0/3600181480 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa26c004020 con 0x7fa2701019e0 2024-05-09T08:48:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.262+0000 7fa266ffd640 1 -- 172.21.15.98:0/3600181480 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c0047a0 con 0x7fa2701019e0 2024-05-09T08:48:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.262+0000 7fa277a8c640 1 -- 172.21.15.98:0/3600181480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa27019e940 con 0x7fa2701019e0 2024-05-09T08:48:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.262+0000 7fa277a8c640 1 -- 172.21.15.98:0/3600181480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa27019eee0 con 0x7fa2701019e0 2024-05-09T08:48:41.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.263+0000 7fa277a8c640 1 -- 172.21.15.98:0/3600181480 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa270068aa0 con 0x7fa2701019e0 2024-05-09T08:48:41.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.264+0000 7fa266ffd640 1 -- 172.21.15.98:0/3600181480 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa26c0041d0 con 0x7fa2701019e0 2024-05-09T08:48:41.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.264+0000 7fa266ffd640 1 --2- 172.21.15.98:0/3600181480 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa25c0a2ca0 0x7fa25c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:41.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.264+0000 7fa275828640 1 --2- 172.21.15.98:0/3600181480 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa25c0a2ca0 0x7fa25c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:41.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.264+0000 7fa266ffd640 1 -- 172.21.15.98:0/3600181480 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa26c0cecd0 con 0x7fa2701019e0 2024-05-09T08:48:41.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.265+0000 7fa275828640 1 --2- 172.21.15.98:0/3600181480 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa25c0a2ca0 0x7fa25c0a5160 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa270101840 tx=0x7fa26000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:41.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.269+0000 7fa266ffd640 1 -- 172.21.15.98:0/3600181480 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa26c09b890 con 0x7fa2701019e0 2024-05-09T08:48:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.444+0000 7fa277a8c640 1 -- 172.21.15.98:0/3600181480 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa270061c70 con 0x7fa25c0a2ca0 2024-05-09T08:48:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.453+0000 7fa266ffd640 1 -- 172.21.15.98:0/3600181480 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340589 (secure 0 0 0) 0x7fa270061c70 con 0x7fa25c0a2ca0 2024-05-09T08:48:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:41.458 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-09T08:48:41.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.457+0000 7fa277a8c640 1 -- 172.21.15.98:0/3600181480 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa25c0a2ca0 msgr2=0x7fa25c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:41.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.457+0000 7fa277a8c640 1 --2- 172.21.15.98:0/3600181480 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa25c0a2ca0 0x7fa25c0a5160 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa270101840 tx=0x7fa26000b390 comp rx=0 tx=0).stop 2024-05-09T08:48:41.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.457+0000 7fa277a8c640 1 -- 172.21.15.98:0/3600181480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2701019e0 msgr2=0x7fa270196f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:41.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.457+0000 7fa277a8c640 1 --2- 172.21.15.98:0/3600181480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2701019e0 0x7fa270196f70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa26c00d950 tx=0x7fa26c00de10 comp rx=0 tx=0).stop 2024-05-09T08:48:41.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.457+0000 7fa277a8c640 1 -- 172.21.15.98:0/3600181480 shutdown_connections 2024-05-09T08:48:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.457+0000 7fa277a8c640 1 --2- 172.21.15.98:0/3600181480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2701023a0 0x7fa27019dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.457+0000 7fa277a8c640 1 --2- 172.21.15.98:0/3600181480 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fa25c0a2ca0 0x7fa25c0a5160 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.457+0000 7fa277a8c640 1 --2- 172.21.15.98:0/3600181480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2701019e0 0x7fa270196f70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.457+0000 7fa277a8c640 1 --2- 172.21.15.98:0/3600181480 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2701007f0 0x7fa270196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.457+0000 7fa277a8c640 1 -- 172.21.15.98:0/3600181480 >> 172.21.15.98:0/3600181480 conn(0x7fa2700fbd80 msgr2=0x7fa270104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.457+0000 7fa277a8c640 1 -- 172.21.15.98:0/3600181480 shutdown_connections 2024-05-09T08:48:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:41.457+0000 7fa277a8c640 1 -- 172.21.15.98:0/3600181480 wait complete. 2024-05-09T08:48:41.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:41 smithi098 ceph-mon[30092]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:41.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:41 smithi098 ceph-mon[30092]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:41.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:41 smithi098 ceph-mon[30092]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:41.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:41 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2726581038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:41.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:41 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3686423068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:41.468 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-05-09T08:48:41.184698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601050,"num_objects":92,"num_object_clones":0,"num_object_copies":277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":168,"num_read_kb":150,"num_write":369,"num_write_kb":3006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":352692,"kb_used_data":11860,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124589132,"statfs":{"total":1151940427776,"available":1151579271168,"internally_reserved":0,"allocated":12144640,"data_stored":7610163,"data_compressed":282369,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8272,"num_objects":68,"num_object_clones":0,"num_object_copies":205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":32,"num_read_kb":31,"num_write":143,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003259"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538367+0000","last_change":"2024-05-09T08:48:18.373685+0000","last_active":"2024-05-09T08:48:25.538367+0000","last_peered":"2024-05-09T08:48:25.538367+0000","last_clean":"2024-05-09T08:48:25.538367+0000","last_became_active":"2024-05-09T08:48:18.373305+0000","last_became_peered":"2024-05-09T08:48:18.373305+0000","last_unstale":"2024-05-09T08:48:25.538367+0000","last_undegraded":"2024-05-09T08:48:25.538367+0000","last_fullsized":"2024-05-09T08:48:25.538367+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:30:45.261706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782689+0000","last_change":"2024-05-09T08:48:18.373152+0000","last_active":"2024-05-09T08:48:25.782689+0000","last_peered":"2024-05-09T08:48:25.782689+0000","last_clean":"2024-05-09T08:48:25.782689+0000","last_became_active":"2024-05-09T08:48:18.372015+0000","last_became_peered":"2024-05-09T08:48:18.372015+0000","last_unstale":"2024-05-09T08:48:25.782689+0000","last_undegraded":"2024-05-09T08:48:25.782689+0000","last_fullsized":"2024-05-09T08:48:25.782689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:47.512778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782656+0000","last_change":"2024-05-09T08:48:18.370177+0000","last_active":"2024-05-09T08:48:25.782656+0000","last_peered":"2024-05-09T08:48:25.782656+0000","last_clean":"2024-05-09T08:48:25.782656+0000","last_became_active":"2024-05-09T08:48:18.369842+0000","last_became_peered":"2024-05-09T08:48:18.369842+0000","last_unstale":"2024-05-09T08:48:25.782656+0000","last_undegraded":"2024-05-09T08:48:25.782656+0000","last_fullsized":"2024-05-09T08:48:25.782656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:00:24.015500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538420+0000","last_change":"2024-05-09T08:48:18.375364+0000","last_active":"2024-05-09T08:48:25.538420+0000","last_peered":"2024-05-09T08:48:25.538420+0000","last_clean":"2024-05-09T08:48:25.538420+0000","last_became_active":"2024-05-09T08:48:18.375030+0000","last_became_peered":"2024-05-09T08:48:18.375030+0000","last_unstale":"2024-05-09T08:48:25.538420+0000","last_undegraded":"2024-05-09T08:48:25.538420+0000","last_fullsized":"2024-05-09T08:48:25.538420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:43:57.684123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.681816+0000","last_change":"2024-05-09T08:48:18.374548+0000","last_active":"2024-05-09T08:48:24.681816+0000","last_peered":"2024-05-09T08:48:24.681816+0000","last_clean":"2024-05-09T08:48:24.681816+0000","last_became_active":"2024-05-09T08:48:18.374190+0000","last_became_peered":"2024-05-09T08:48:18.374190+0000","last_unstale":"2024-05-09T08:48:24.681816+0000","last_undegraded":"2024-05-09T08:48:24.681816+0000","last_fullsized":"2024-05-09T08:48:24.681816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:43:48.456839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538724+0000","last_change":"2024-05-09T08:48:18.371234+0000","last_active":"2024-05-09T08:48:25.538724+0000","last_peered":"2024-05-09T08:48:25.538724+0000","last_clean":"2024-05-09T08:48:25.538724+0000","last_became_active":"2024-05-09T08:48:18.370218+0000","last_became_peered":"2024-05-09T08:48:18.370218+0000","last_unstale":"2024-05-09T08:48:25.538724+0000","last_undegraded":"2024-05-09T08:48:25.538724+0000","last_fullsized":"2024-05-09T08:48:25.538724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:02:09.923146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608500+0000","last_change":"2024-05-09T08:48:18.368731+0000","last_active":"2024-05-09T08:48:23.608500+0000","last_peered":"2024-05-09T08:48:23.608500+0000","last_clean":"2024-05-09T08:48:23.608500+0000","last_became_active":"2024-05-09T08:48:18.368389+0000","last_became_peered":"2024-05-09T08:48:18.368389+0000","last_unstale":"2024-05-09T08:48:23.608500+0000","last_undegraded":"2024-05-09T08:48:23.608500+0000","last_fullsized":"2024-05-09T08:48:23.608500+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:21:45.177845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537684+0000","last_change":"2024-05-09T08:48:18.374296+0000","last_active":"2024-05-09T08:48:25.537684+0000","last_peered":"2024-05-09T08:48:25.537684+0000","last_clean":"2024-05-09T08:48:25.537684+0000","last_became_active":"2024-05-09T08:48:18.369805+0000","last_became_peered":"2024-05-09T08:48:18.369805+0000","last_unstale":"2024-05-09T08:48:25.537684+0000","last_undegraded":"2024-05-09T08:48:25.537684+0000","last_fullsized":"2024-05-09T08:48:25.537684+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:27:31.476940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.835158+0000","last_change":"2024-05-09T08:48:18.372686+0000","last_active":"2024-05-09T08:48:24.835158+0000","last_peered":"2024-05-09T08:48:24.835158+0000","last_clean":"2024-05-09T08:48:24.835158+0000","last_became_active":"2024-05-09T08:48:18.372349+0000","last_became_peered":"2024-05-09T08:48:18.372349+0000","last_unstale":"2024-05-09T08:48:24.835158+0000","last_undegraded":"2024-05-09T08:48:24.835158+0000","last_fullsized":"2024-05-09T08:48:24.835158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:15.177389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.539112+0000","last_change":"2024-05-09T08:48:18.363668+0000","last_active":"2024-05-09T08:48:25.539112+0000","last_peered":"2024-05-09T08:48:25.539112+0000","last_clean":"2024-05-09T08:48:25.539112+0000","last_became_active":"2024-05-09T08:48:18.363248+0000","last_became_peered":"2024-05-09T08:48:18.363248+0000","last_unstale":"2024-05-09T08:48:25.539112+0000","last_undegraded":"2024-05-09T08:48:25.539112+0000","last_fullsized":"2024-05-09T08:48:25.539112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:19:33.802350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.609133+0000","last_change":"2024-05-09T08:48:18.376401+0000","last_active":"2024-05-09T08:48:23.609133+0000","last_peered":"2024-05-09T08:48:23.609133+0000","last_clean":"2024-05-09T08:48:23.609133+0000","last_became_active":"2024-05-09T08:48:18.375520+0000","last_became_peered":"2024-05-09T08:48:18.375520+0000","last_unstale":"2024-05-09T08:48:23.609133+0000","last_undegraded":"2024-05-09T08:48:23.609133+0000","last_fullsized":"2024-05-09T08:48:23.609133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:40:36.145071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538314+0000","last_change":"2024-05-09T08:48:18.372847+0000","last_active":"2024-05-09T08:48:25.538314+0000","last_peered":"2024-05-09T08:48:25.538314+0000","last_clean":"2024-05-09T08:48:25.538314+0000","last_became_active":"2024-05-09T08:48:18.370671+0000","last_became_peered":"2024-05-09T08:48:18.370671+0000","last_unstale":"2024-05-09T08:48:25.538314+0000","last_undegraded":"2024-05-09T08:48:25.538314+0000","last_fullsized":"2024-05-09T08:48:25.538314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:06:22.438515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603752+0000","last_change":"2024-05-09T08:48:18.365810+0000","last_active":"2024-05-09T08:48:23.603752+0000","last_peered":"2024-05-09T08:48:23.603752+0000","last_clean":"2024-05-09T08:48:23.603752+0000","last_became_active":"2024-05-09T08:48:18.365214+0000","last_became_peered":"2024-05-09T08:48:18.365214+0000","last_unstale":"2024-05-09T08:48:23.603752+0000","last_undegraded":"2024-05-09T08:48:23.603752+0000","last_fullsized":"2024-05-09T08:48:23.603752+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:00.037153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782218+0000","last_change":"2024-05-09T08:48:18.375279+0000","last_active":"2024-05-09T08:48:25.782218+0000","last_peered":"2024-05-09T08:48:25.782218+0000","last_clean":"2024-05-09T08:48:25.782218+0000","last_became_active":"2024-05-09T08:48:18.374918+0000","last_became_peered":"2024-05-09T08:48:18.374918+0000","last_unstale":"2024-05-09T08:48:25.782218+0000","last_undegraded":"2024-05-09T08:48:25.782218+0000","last_fullsized":"2024-05-09T08:48:25.782218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:23:12.889283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782336+0000","last_change":"2024-05-09T08:48:18.377722+0000","last_active":"2024-05-09T08:48:25.782336+0000","last_peered":"2024-05-09T08:48:25.782336+0000","last_clean":"2024-05-09T08:48:25.782336+0000","last_became_active":"2024-05-09T08:48:18.377416+0000","last_became_peered":"2024-05-09T08:48:18.377416+0000","last_unstale":"2024-05-09T08:48:25.782336+0000","last_undegraded":"2024-05-09T08:48:25.782336+0000","last_fullsized":"2024-05-09T08:48:25.782336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:27.093800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.834636+0000","last_change":"2024-05-09T08:48:18.373905+0000","last_active":"2024-05-09T08:48:24.834636+0000","last_peered":"2024-05-09T08:48:24.834636+0000","last_clean":"2024-05-09T08:48:24.834636+0000","last_became_active":"2024-05-09T08:48:18.373566+0000","last_became_peered":"2024-05-09T08:48:18.373566+0000","last_unstale":"2024-05-09T08:48:24.834636+0000","last_undegraded":"2024-05-09T08:48:24.834636+0000","last_fullsized":"2024-05-09T08:48:24.834636+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:33.905686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538287+0000","last_change":"2024-05-09T08:48:18.373970+0000","last_active":"2024-05-09T08:48:25.538287+0000","last_peered":"2024-05-09T08:48:25.538287+0000","last_clean":"2024-05-09T08:48:25.538287+0000","last_became_active":"2024-05-09T08:48:18.372696+0000","last_became_peered":"2024-05-09T08:48:18.372696+0000","last_unstale":"2024-05-09T08:48:25.538287+0000","last_undegraded":"2024-05-09T08:48:25.538287+0000","last_fullsized":"2024-05-09T08:48:25.538287+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:09:33.751950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.779162+0000","last_change":"2024-05-09T08:48:18.376851+0000","last_active":"2024-05-09T08:48:23.779162+0000","last_peered":"2024-05-09T08:48:23.779162+0000","last_clean":"2024-05-09T08:48:23.779162+0000","last_became_active":"2024-05-09T08:48:18.376230+0000","last_became_peered":"2024-05-09T08:48:18.376230+0000","last_unstale":"2024-05-09T08:48:23.779162+0000","last_undegraded":"2024-05-09T08:48:23.779162+0000","last_fullsized":"2024-05-09T08:48:23.779162+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:34.959179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888910+0000","last_change":"2024-05-09T08:48:18.370287+0000","last_active":"2024-05-09T08:48:23.888910+0000","last_peered":"2024-05-09T08:48:23.888910+0000","last_clean":"2024-05-09T08:48:23.888910+0000","last_became_active":"2024-05-09T08:48:18.369956+0000","last_became_peered":"2024-05-09T08:48:18.369956+0000","last_unstale":"2024-05-09T08:48:23.888910+0000","last_undegraded":"2024-05-09T08:48:23.888910+0000","last_fullsized":"2024-05-09T08:48:23.888910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:19:34.746216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888612+0000","last_change":"2024-05-09T08:48:22.377973+0000","last_active":"2024-05-09T08:48:23.888612+0000","last_peered":"2024-05-09T08:48:23.888612+0000","last_clean":"2024-05-09T08:48:23.888612+0000","last_became_active":"2024-05-09T08:48:22.377325+0000","last_became_peered":"2024-05-09T08:48:22.377325+0000","last_unstale":"2024-05-09T08:48:23.888612+0000","last_undegraded":"2024-05-09T08:48:23.888612+0000","last_fullsized":"2024-05-09T08:48:23.888612+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:32:00.873082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.779296+0000","last_change":"2024-05-09T08:48:17.595784+0000","last_active":"2024-05-09T08:48:23.779296+0000","last_peered":"2024-05-09T08:48:23.779296+0000","last_clean":"2024-05-09T08:48:23.779296+0000","last_became_active":"2024-05-09T08:48:17.595395+0000","last_became_peered":"2024-05-09T08:48:17.595395+0000","last_unstale":"2024-05-09T08:48:23.779296+0000","last_undegraded":"2024-05-09T08:48:23.779296+0000","last_fullsized":"2024-05-09T08:48:23.779296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:47.475148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537907+0000","last_change":"2024-05-09T08:48:18.382332+0000","last_active":"2024-05-09T08:48:25.537907+0000","last_peered":"2024-05-09T08:48:25.537907+0000","last_clean":"2024-05-09T08:48:25.537907+0000","last_became_active":"2024-05-09T08:48:18.381536+0000","last_became_peered":"2024-05-09T08:48:18.381536+0000","last_unstale":"2024-05-09T08:48:25.537907+0000","last_undegraded":"2024-05-09T08:48:25.537907+0000","last_fullsized":"2024-05-09T08:48:25.537907+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:52:48.327000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537819+0000","last_change":"2024-05-09T08:48:22.379541+0000","last_active":"2024-05-09T08:48:25.537819+0000","last_peered":"2024-05-09T08:48:25.537819+0000","last_clean":"2024-05-09T08:48:25.537819+0000","last_became_active":"2024-05-09T08:48:22.379135+0000","last_became_peered":"2024-05-09T08:48:22.379135+0000","last_unstale":"2024-05-09T08:48:25.537819+0000","last_undegraded":"2024-05-09T08:48:25.537819+0000","last_fullsized":"2024-05-09T08:48:25.537819+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:51.365075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538116+0000","last_change":"2024-05-09T08:48:17.592168+0000","last_active":"2024-05-09T08:48:25.538116+0000","last_peered":"2024-05-09T08:48:25.538116+0000","last_clean":"2024-05-09T08:48:25.538116+0000","last_became_active":"2024-05-09T08:48:17.591822+0000","last_became_peered":"2024-05-09T08:48:17.591822+0000","last_unstale":"2024-05-09T08:48:25.538116+0000","last_undegraded":"2024-05-09T08:48:25.538116+0000","last_fullsized":"2024-05-09T08:48:25.538116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:45:15.162240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782715+0000","last_change":"2024-05-09T08:48:18.379396+0000","last_active":"2024-05-09T08:48:25.782715+0000","last_peered":"2024-05-09T08:48:25.782715+0000","last_clean":"2024-05-09T08:48:25.782715+0000","last_became_active":"2024-05-09T08:48:18.378770+0000","last_became_peered":"2024-05-09T08:48:18.378770+0000","last_unstale":"2024-05-09T08:48:25.782715+0000","last_undegraded":"2024-05-09T08:48:25.782715+0000","last_fullsized":"2024-05-09T08:48:25.782715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:22:22.699186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538058+0000","last_change":"2024-05-09T08:48:22.387777+0000","last_active":"2024-05-09T08:48:25.538058+0000","last_peered":"2024-05-09T08:48:25.538058+0000","last_clean":"2024-05-09T08:48:25.538058+0000","last_became_active":"2024-05-09T08:48:22.387300+0000","last_became_peered":"2024-05-09T08:48:22.387300+0000","last_unstale":"2024-05-09T08:48:25.538058+0000","last_undegraded":"2024-05-09T08:48:25.538058+0000","last_fullsized":"2024-05-09T08:48:25.538058+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:08:35.646374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:39.764399+0000","last_change":"2024-05-09T08:48:17.602493+0000","last_active":"2024-05-09T08:48:39.764399+0000","last_peered":"2024-05-09T08:48:39.764399+0000","last_clean":"2024-05-09T08:48:39.764399+0000","last_became_active":"2024-05-09T08:48:17.601625+0000","last_became_peered":"2024-05-09T08:48:17.601625+0000","last_unstale":"2024-05-09T08:48:39.764399+0000","last_undegraded":"2024-05-09T08:48:39.764399+0000","last_fullsized":"2024-05-09T08:48:39.764399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:33.044532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.834873+0000","last_change":"2024-05-09T08:48:18.380615+0000","last_active":"2024-05-09T08:48:24.834873+0000","last_peered":"2024-05-09T08:48:24.834873+0000","last_clean":"2024-05-09T08:48:24.834873+0000","last_became_active":"2024-05-09T08:48:18.380196+0000","last_became_peered":"2024-05-09T08:48:18.380196+0000","last_unstale":"2024-05-09T08:48:24.834873+0000","last_undegraded":"2024-05-09T08:48:24.834873+0000","last_fullsized":"2024-05-09T08:48:24.834873+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:27.299263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537812+0000","last_change":"2024-05-09T08:48:22.381954+0000","last_active":"2024-05-09T08:48:25.537812+0000","last_peered":"2024-05-09T08:48:25.537812+0000","last_clean":"2024-05-09T08:48:25.537812+0000","last_became_active":"2024-05-09T08:48:22.381527+0000","last_became_peered":"2024-05-09T08:48:22.381527+0000","last_unstale":"2024-05-09T08:48:25.537812+0000","last_undegraded":"2024-05-09T08:48:25.537812+0000","last_fullsized":"2024-05-09T08:48:25.537812+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:35.170753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:38.748946+0000","last_change":"2024-05-09T08:48:17.592795+0000","last_active":"2024-05-09T08:48:38.748946+0000","last_peered":"2024-05-09T08:48:38.748946+0000","last_clean":"2024-05-09T08:48:38.748946+0000","last_became_active":"2024-05-09T08:48:17.592240+0000","last_became_peered":"2024-05-09T08:48:17.592240+0000","last_unstale":"2024-05-09T08:48:38.748946+0000","last_undegraded":"2024-05-09T08:48:38.748946+0000","last_fullsized":"2024-05-09T08:48:38.748946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:27:30.056805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608521+0000","last_change":"2024-05-09T08:48:18.373853+0000","last_active":"2024-05-09T08:48:23.608521+0000","last_peered":"2024-05-09T08:48:23.608521+0000","last_clean":"2024-05-09T08:48:23.608521+0000","last_became_active":"2024-05-09T08:48:18.373482+0000","last_became_peered":"2024-05-09T08:48:18.373482+0000","last_unstale":"2024-05-09T08:48:23.608521+0000","last_undegraded":"2024-05-09T08:48:23.608521+0000","last_fullsized":"2024-05-09T08:48:23.608521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:14:49.129927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888800+0000","last_change":"2024-05-09T08:48:22.374267+0000","last_active":"2024-05-09T08:48:23.888800+0000","last_peered":"2024-05-09T08:48:23.888800+0000","last_clean":"2024-05-09T08:48:23.888800+0000","last_became_active":"2024-05-09T08:48:22.373824+0000","last_became_peered":"2024-05-09T08:48:22.373824+0000","last_unstale":"2024-05-09T08:48:23.888800+0000","last_undegraded":"2024-05-09T08:48:23.888800+0000","last_fullsized":"2024-05-09T08:48:23.888800+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:27.407543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.733155+0000","last_change":"2024-05-09T08:48:17.596851+0000","last_active":"2024-05-09T08:48:37.733155+0000","last_peered":"2024-05-09T08:48:37.733155+0000","last_clean":"2024-05-09T08:48:37.733155+0000","last_became_active":"2024-05-09T08:48:17.596532+0000","last_became_peered":"2024-05-09T08:48:17.596532+0000","last_unstale":"2024-05-09T08:48:37.733155+0000","last_undegraded":"2024-05-09T08:48:37.733155+0000","last_fullsized":"2024-05-09T08:48:37.733155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:58:51.923188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.682002+0000","last_change":"2024-05-09T08:48:18.372214+0000","last_active":"2024-05-09T08:48:24.682002+0000","last_peered":"2024-05-09T08:48:24.682002+0000","last_clean":"2024-05-09T08:48:24.682002+0000","last_became_active":"2024-05-09T08:48:18.371850+0000","last_became_peered":"2024-05-09T08:48:18.371850+0000","last_unstale":"2024-05-09T08:48:24.682002+0000","last_undegraded":"2024-05-09T08:48:24.682002+0000","last_fullsized":"2024-05-09T08:48:24.682002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:56.971735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888674+0000","last_change":"2024-05-09T08:48:22.374755+0000","last_active":"2024-05-09T08:48:23.888674+0000","last_peered":"2024-05-09T08:48:23.888674+0000","last_clean":"2024-05-09T08:48:23.888674+0000","last_became_active":"2024-05-09T08:48:22.374139+0000","last_became_peered":"2024-05-09T08:48:22.374139+0000","last_unstale":"2024-05-09T08:48:23.888674+0000","last_undegraded":"2024-05-09T08:48:23.888674+0000","last_fullsized":"2024-05-09T08:48:23.888674+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:57:57.724212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.743649+0000","last_change":"2024-05-09T08:48:17.601726+0000","last_active":"2024-05-09T08:48:37.743649+0000","last_peered":"2024-05-09T08:48:37.743649+0000","last_clean":"2024-05-09T08:48:37.743649+0000","last_became_active":"2024-05-09T08:48:17.601372+0000","last_became_peered":"2024-05-09T08:48:17.601372+0000","last_unstale":"2024-05-09T08:48:37.743649+0000","last_undegraded":"2024-05-09T08:48:37.743649+0000","last_fullsized":"2024-05-09T08:48:37.743649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:27:56.683821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608262+0000","last_change":"2024-05-09T08:48:18.369767+0000","last_active":"2024-05-09T08:48:23.608262+0000","last_peered":"2024-05-09T08:48:23.608262+0000","last_clean":"2024-05-09T08:48:23.608262+0000","last_became_active":"2024-05-09T08:48:18.369388+0000","last_became_peered":"2024-05-09T08:48:18.369388+0000","last_unstale":"2024-05-09T08:48:23.608262+0000","last_undegraded":"2024-05-09T08:48:23.608262+0000","last_fullsized":"2024-05-09T08:48:23.608262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:31:44.727770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538260+0000","last_change":"2024-05-09T08:48:22.390990+0000","last_active":"2024-05-09T08:48:25.538260+0000","last_peered":"2024-05-09T08:48:25.538260+0000","last_clean":"2024-05-09T08:48:25.538260+0000","last_became_active":"2024-05-09T08:48:22.390601+0000","last_became_peered":"2024-05-09T08:48:22.390601+0000","last_unstale":"2024-05-09T08:48:25.538260+0000","last_undegraded":"2024-05-09T08:48:25.538260+0000","last_fullsized":"2024-05-09T08:48:25.538260+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:47:24.644408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.290497+0000","last_change":"2024-05-09T08:48:17.598008+0000","last_active":"2024-05-09T08:48:37.290497+0000","last_peered":"2024-05-09T08:48:37.290497+0000","last_clean":"2024-05-09T08:48:37.290497+0000","last_became_active":"2024-05-09T08:48:17.597514+0000","last_became_peered":"2024-05-09T08:48:17.597514+0000","last_unstale":"2024-05-09T08:48:37.290497+0000","last_undegraded":"2024-05-09T08:48:37.290497+0000","last_fullsized":"2024-05-09T08:48:37.290497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:18:32.788536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537547+0000","last_change":"2024-05-09T08:48:18.378061+0000","last_active":"2024-05-09T08:48:25.537547+0000","last_peered":"2024-05-09T08:48:25.537547+0000","last_clean":"2024-05-09T08:48:25.537547+0000","last_became_active":"2024-05-09T08:48:18.377684+0000","last_became_peered":"2024-05-09T08:48:18.377684+0000","last_unstale":"2024-05-09T08:48:25.537547+0000","last_undegraded":"2024-05-09T08:48:25.537547+0000","last_fullsized":"2024-05-09T08:48:25.537547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:13:08.338307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782722+0000","last_change":"2024-05-09T08:48:22.385779+0000","last_active":"2024-05-09T08:48:25.782722+0000","last_peered":"2024-05-09T08:48:25.782722+0000","last_clean":"2024-05-09T08:48:25.782722+0000","last_became_active":"2024-05-09T08:48:22.385021+0000","last_became_peered":"2024-05-09T08:48:22.385021+0000","last_unstale":"2024-05-09T08:48:25.782722+0000","last_undegraded":"2024-05-09T08:48:25.782722+0000","last_fullsized":"2024-05-09T08:48:25.782722+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:51.350754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:36.279252+0000","last_change":"2024-05-09T08:48:17.595750+0000","last_active":"2024-05-09T08:48:36.279252+0000","last_peered":"2024-05-09T08:48:36.279252+0000","last_clean":"2024-05-09T08:48:36.279252+0000","last_became_active":"2024-05-09T08:48:17.595394+0000","last_became_peered":"2024-05-09T08:48:17.595394+0000","last_unstale":"2024-05-09T08:48:36.279252+0000","last_undegraded":"2024-05-09T08:48:36.279252+0000","last_fullsized":"2024-05-09T08:48:36.279252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:39:21.975059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.778932+0000","last_change":"2024-05-09T08:48:18.370750+0000","last_active":"2024-05-09T08:48:23.778932+0000","last_peered":"2024-05-09T08:48:23.778932+0000","last_clean":"2024-05-09T08:48:23.778932+0000","last_became_active":"2024-05-09T08:48:18.370383+0000","last_became_peered":"2024-05-09T08:48:18.370383+0000","last_unstale":"2024-05-09T08:48:23.778932+0000","last_undegraded":"2024-05-09T08:48:23.778932+0000","last_fullsized":"2024-05-09T08:48:23.778932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:36.049350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537648+0000","last_change":"2024-05-09T08:48:22.377157+0000","last_active":"2024-05-09T08:48:25.537648+0000","last_peered":"2024-05-09T08:48:25.537648+0000","last_clean":"2024-05-09T08:48:25.537648+0000","last_became_active":"2024-05-09T08:48:22.376730+0000","last_became_peered":"2024-05-09T08:48:22.376730+0000","last_unstale":"2024-05-09T08:48:25.537648+0000","last_undegraded":"2024-05-09T08:48:25.537648+0000","last_fullsized":"2024-05-09T08:48:25.537648+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:41:52.080730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.732425+0000","last_change":"2024-05-09T08:48:17.598528+0000","last_active":"2024-05-09T08:48:37.732425+0000","last_peered":"2024-05-09T08:48:37.732425+0000","last_clean":"2024-05-09T08:48:37.732425+0000","last_became_active":"2024-05-09T08:48:17.598072+0000","last_became_peered":"2024-05-09T08:48:17.598072+0000","last_unstale":"2024-05-09T08:48:37.732425+0000","last_undegraded":"2024-05-09T08:48:37.732425+0000","last_fullsized":"2024-05-09T08:48:37.732425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:35.697154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.835104+0000","last_change":"2024-05-09T08:48:18.380919+0000","last_active":"2024-05-09T08:48:24.835104+0000","last_peered":"2024-05-09T08:48:24.835104+0000","last_clean":"2024-05-09T08:48:24.835104+0000","last_became_active":"2024-05-09T08:48:18.380485+0000","last_became_peered":"2024-05-09T08:48:18.380485+0000","last_unstale":"2024-05-09T08:48:24.835104+0000","last_undegraded":"2024-05-09T08:48:24.835104+0000","last_fullsized":"2024-05-09T08:48:24.835104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:45.079173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.778704+0000","last_change":"2024-05-09T08:48:22.389667+0000","last_active":"2024-05-09T08:48:23.778704+0000","last_peered":"2024-05-09T08:48:23.778704+0000","last_clean":"2024-05-09T08:48:23.778704+0000","last_became_active":"2024-05-09T08:48:22.389268+0000","last_became_peered":"2024-05-09T08:48:22.389268+0000","last_unstale":"2024-05-09T08:48:23.778704+0000","last_undegraded":"2024-05-09T08:48:23.778704+0000","last_fullsized":"2024-05-09T08:48:23.778704+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:33:14.861855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:38.748642+0000","last_change":"2024-05-09T08:48:17.596028+0000","last_active":"2024-05-09T08:48:38.748642+0000","last_peered":"2024-05-09T08:48:38.748642+0000","last_clean":"2024-05-09T08:48:38.748642+0000","last_became_active":"2024-05-09T08:48:17.595053+0000","last_became_peered":"2024-05-09T08:48:17.595053+0000","last_unstale":"2024-05-09T08:48:38.748642+0000","last_undegraded":"2024-05-09T08:48:38.748642+0000","last_fullsized":"2024-05-09T08:48:38.748642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:48:31.323595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782741+0000","last_change":"2024-05-09T08:48:18.373472+0000","last_active":"2024-05-09T08:48:25.782741+0000","last_peered":"2024-05-09T08:48:25.782741+0000","last_clean":"2024-05-09T08:48:25.782741+0000","last_became_active":"2024-05-09T08:48:18.373178+0000","last_became_peered":"2024-05-09T08:48:18.373178+0000","last_unstale":"2024-05-09T08:48:25.782741+0000","last_undegraded":"2024-05-09T08:48:25.782741+0000","last_fullsized":"2024-05-09T08:48:25.782741+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:18.048722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538490+0000","last_change":"2024-05-09T08:48:22.386551+0000","last_active":"2024-05-09T08:48:25.538490+0000","last_peered":"2024-05-09T08:48:25.538490+0000","last_clean":"2024-05-09T08:48:25.538490+0000","last_became_active":"2024-05-09T08:48:22.386018+0000","last_became_peered":"2024-05-09T08:48:22.386018+0000","last_unstale":"2024-05-09T08:48:25.538490+0000","last_undegraded":"2024-05-09T08:48:25.538490+0000","last_fullsized":"2024-05-09T08:48:25.538490+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:59.812030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.733781+0000","last_change":"2024-05-09T08:48:17.601207+0000","last_active":"2024-05-09T08:48:37.733781+0000","last_peered":"2024-05-09T08:48:37.733781+0000","last_clean":"2024-05-09T08:48:37.733781+0000","last_became_active":"2024-05-09T08:48:17.600878+0000","last_became_peered":"2024-05-09T08:48:17.600878+0000","last_unstale":"2024-05-09T08:48:37.733781+0000","last_undegraded":"2024-05-09T08:48:37.733781+0000","last_fullsized":"2024-05-09T08:48:37.733781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:30.040527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.835063+0000","last_change":"2024-05-09T08:48:18.381577+0000","last_active":"2024-05-09T08:48:24.835063+0000","last_peered":"2024-05-09T08:48:24.835063+0000","last_clean":"2024-05-09T08:48:24.835063+0000","last_became_active":"2024-05-09T08:48:18.381223+0000","last_became_peered":"2024-05-09T08:48:18.381223+0000","last_unstale":"2024-05-09T08:48:24.835063+0000","last_undegraded":"2024-05-09T08:48:24.835063+0000","last_fullsized":"2024-05-09T08:48:24.835063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:57.038392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.778720+0000","last_change":"2024-05-09T08:48:22.385885+0000","last_active":"2024-05-09T08:48:23.778720+0000","last_peered":"2024-05-09T08:48:23.778720+0000","last_clean":"2024-05-09T08:48:23.778720+0000","last_became_active":"2024-05-09T08:48:22.385403+0000","last_became_peered":"2024-05-09T08:48:22.385403+0000","last_unstale":"2024-05-09T08:48:23.778720+0000","last_undegraded":"2024-05-09T08:48:23.778720+0000","last_fullsized":"2024-05-09T08:48:23.778720+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:53:20.733523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'12","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.732070+0000","last_change":"2024-05-09T08:48:17.597848+0000","last_active":"2024-05-09T08:48:37.732070+0000","last_peered":"2024-05-09T08:48:37.732070+0000","last_clean":"2024-05-09T08:48:37.732070+0000","last_became_active":"2024-05-09T08:48:17.596989+0000","last_became_peered":"2024-05-09T08:48:17.596989+0000","last_unstale":"2024-05-09T08:48:37.732070+0000","last_undegraded":"2024-05-09T08:48:37.732070+0000","last_fullsized":"2024-05-09T08:48:37.732070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:07.256301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.682156+0000","last_change":"2024-05-09T08:48:18.365308+0000","last_active":"2024-05-09T08:48:24.682156+0000","last_peered":"2024-05-09T08:48:24.682156+0000","last_clean":"2024-05-09T08:48:24.682156+0000","last_became_active":"2024-05-09T08:48:18.364845+0000","last_became_peered":"2024-05-09T08:48:18.364845+0000","last_unstale":"2024-05-09T08:48:24.682156+0000","last_undegraded":"2024-05-09T08:48:24.682156+0000","last_fullsized":"2024-05-09T08:48:24.682156+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:07.277730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608825+0000","last_change":"2024-05-09T08:48:22.377333+0000","last_active":"2024-05-09T08:48:23.608825+0000","last_peered":"2024-05-09T08:48:23.608825+0000","last_clean":"2024-05-09T08:48:23.608825+0000","last_became_active":"2024-05-09T08:48:22.376494+0000","last_became_peered":"2024-05-09T08:48:22.376494+0000","last_unstale":"2024-05-09T08:48:23.608825+0000","last_undegraded":"2024-05-09T08:48:23.608825+0000","last_fullsized":"2024-05-09T08:48:23.608825+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:49:17.693977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:38.748975+0000","last_change":"2024-05-09T08:48:17.600749+0000","last_active":"2024-05-09T08:48:38.748975+0000","last_peered":"2024-05-09T08:48:38.748975+0000","last_clean":"2024-05-09T08:48:38.748975+0000","last_became_active":"2024-05-09T08:48:17.600402+0000","last_became_peered":"2024-05-09T08:48:17.600402+0000","last_unstale":"2024-05-09T08:48:38.748975+0000","last_undegraded":"2024-05-09T08:48:38.748975+0000","last_fullsized":"2024-05-09T08:48:38.748975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:42:14.889227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603573+0000","last_change":"2024-05-09T08:48:18.373652+0000","last_active":"2024-05-09T08:48:23.603573+0000","last_peered":"2024-05-09T08:48:23.603573+0000","last_clean":"2024-05-09T08:48:23.603573+0000","last_became_active":"2024-05-09T08:48:18.373345+0000","last_became_peered":"2024-05-09T08:48:18.373345+0000","last_unstale":"2024-05-09T08:48:23.603573+0000","last_undegraded":"2024-05-09T08:48:23.603573+0000","last_fullsized":"2024-05-09T08:48:23.603573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:14:34.843305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538026+0000","last_change":"2024-05-09T08:48:22.378946+0000","last_active":"2024-05-09T08:48:25.538026+0000","last_peered":"2024-05-09T08:48:25.538026+0000","last_clean":"2024-05-09T08:48:25.538026+0000","last_became_active":"2024-05-09T08:48:22.378184+0000","last_became_peered":"2024-05-09T08:48:22.378184+0000","last_unstale":"2024-05-09T08:48:25.538026+0000","last_undegraded":"2024-05-09T08:48:25.538026+0000","last_fullsized":"2024-05-09T08:48:25.538026+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:04.063287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:36.279310+0000","last_change":"2024-05-09T08:48:17.592658+0000","last_active":"2024-05-09T08:48:36.279310+0000","last_peered":"2024-05-09T08:48:36.279310+0000","last_clean":"2024-05-09T08:48:36.279310+0000","last_became_active":"2024-05-09T08:48:17.592275+0000","last_became_peered":"2024-05-09T08:48:17.592275+0000","last_unstale":"2024-05-09T08:48:36.279310+0000","last_undegraded":"2024-05-09T08:48:36.279310+0000","last_fullsized":"2024-05-09T08:48:36.279310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:26:56.742079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608836+0000","last_change":"2024-05-09T08:48:18.374643+0000","last_active":"2024-05-09T08:48:23.608836+0000","last_peered":"2024-05-09T08:48:23.608836+0000","last_clean":"2024-05-09T08:48:23.608836+0000","last_became_active":"2024-05-09T08:48:18.374291+0000","last_became_peered":"2024-05-09T08:48:18.374291+0000","last_unstale":"2024-05-09T08:48:23.608836+0000","last_undegraded":"2024-05-09T08:48:23.608836+0000","last_fullsized":"2024-05-09T08:48:23.608836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:19:55.390772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.779227+0000","last_change":"2024-05-09T08:48:22.389568+0000","last_active":"2024-05-09T08:48:23.779227+0000","last_peered":"2024-05-09T08:48:23.779227+0000","last_clean":"2024-05-09T08:48:23.779227+0000","last_became_active":"2024-05-09T08:48:22.387192+0000","last_became_peered":"2024-05-09T08:48:22.387192+0000","last_unstale":"2024-05-09T08:48:23.779227+0000","last_undegraded":"2024-05-09T08:48:23.779227+0000","last_fullsized":"2024-05-09T08:48:23.779227+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:56.794208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:38.734761+0000","last_change":"2024-05-09T08:48:17.598760+0000","last_active":"2024-05-09T08:48:38.734761+0000","last_peered":"2024-05-09T08:48:38.734761+0000","last_clean":"2024-05-09T08:48:38.734761+0000","last_became_active":"2024-05-09T08:48:17.598192+0000","last_became_peered":"2024-05-09T08:48:17.598192+0000","last_unstale":"2024-05-09T08:48:38.734761+0000","last_undegraded":"2024-05-09T08:48:38.734761+0000","last_fullsized":"2024-05-09T08:48:38.734761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:29.007644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608634+0000","last_change":"2024-05-09T08:48:18.364570+0000","last_active":"2024-05-09T08:48:23.608634+0000","last_peered":"2024-05-09T08:48:23.608634+0000","last_clean":"2024-05-09T08:48:23.608634+0000","last_became_active":"2024-05-09T08:48:18.364191+0000","last_became_peered":"2024-05-09T08:48:18.364191+0000","last_unstale":"2024-05-09T08:48:23.608634+0000","last_undegraded":"2024-05-09T08:48:23.608634+0000","last_fullsized":"2024-05-09T08:48:23.608634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:49.267954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.604183+0000","last_change":"2024-05-09T08:48:22.380710+0000","last_active":"2024-05-09T08:48:23.604183+0000","last_peered":"2024-05-09T08:48:23.604183+0000","last_clean":"2024-05-09T08:48:23.604183+0000","last_became_active":"2024-05-09T08:48:22.380322+0000","last_became_peered":"2024-05-09T08:48:22.380322+0000","last_unstale":"2024-05-09T08:48:23.604183+0000","last_undegraded":"2024-05-09T08:48:23.604183+0000","last_fullsized":"2024-05-09T08:48:23.604183+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:36:44.479999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.743507+0000","last_change":"2024-05-09T08:48:17.599762+0000","last_active":"2024-05-09T08:48:37.743507+0000","last_peered":"2024-05-09T08:48:37.743507+0000","last_clean":"2024-05-09T08:48:37.743507+0000","last_became_active":"2024-05-09T08:48:17.599355+0000","last_became_peered":"2024-05-09T08:48:17.599355+0000","last_unstale":"2024-05-09T08:48:37.743507+0000","last_undegraded":"2024-05-09T08:48:37.743507+0000","last_fullsized":"2024-05-09T08:48:37.743507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:28:36.836403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608133+0000","last_change":"2024-05-09T08:48:18.372810+0000","last_active":"2024-05-09T08:48:23.608133+0000","last_peered":"2024-05-09T08:48:23.608133+0000","last_clean":"2024-05-09T08:48:23.608133+0000","last_became_active":"2024-05-09T08:48:18.372437+0000","last_became_peered":"2024-05-09T08:48:18.372437+0000","last_unstale":"2024-05-09T08:48:23.608133+0000","last_undegraded":"2024-05-09T08:48:23.608133+0000","last_fullsized":"2024-05-09T08:48:23.608133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:03:36.879719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608510+0000","last_change":"2024-05-09T08:48:22.379370+0000","last_active":"2024-05-09T08:48:23.608510+0000","last_peered":"2024-05-09T08:48:23.608510+0000","last_clean":"2024-05-09T08:48:23.608510+0000","last_became_active":"2024-05-09T08:48:22.378994+0000","last_became_peered":"2024-05-09T08:48:22.378994+0000","last_unstale":"2024-05-09T08:48:23.608510+0000","last_undegraded":"2024-05-09T08:48:23.608510+0000","last_fullsized":"2024-05-09T08:48:23.608510+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:08.005647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.729878+0000","last_change":"2024-05-09T08:48:17.594236+0000","last_active":"2024-05-09T08:48:37.729878+0000","last_peered":"2024-05-09T08:48:37.729878+0000","last_clean":"2024-05-09T08:48:37.729878+0000","last_became_active":"2024-05-09T08:48:17.593418+0000","last_became_peered":"2024-05-09T08:48:17.593418+0000","last_unstale":"2024-05-09T08:48:37.729878+0000","last_undegraded":"2024-05-09T08:48:37.729878+0000","last_fullsized":"2024-05-09T08:48:37.729878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:23.930421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538064+0000","last_change":"2024-05-09T08:48:18.366767+0000","last_active":"2024-05-09T08:48:25.538064+0000","last_peered":"2024-05-09T08:48:25.538064+0000","last_clean":"2024-05-09T08:48:25.538064+0000","last_became_active":"2024-05-09T08:48:18.366435+0000","last_became_peered":"2024-05-09T08:48:18.366435+0000","last_unstale":"2024-05-09T08:48:25.538064+0000","last_undegraded":"2024-05-09T08:48:25.538064+0000","last_fullsized":"2024-05-09T08:48:25.538064+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:45.111510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.682057+0000","last_change":"2024-05-09T08:48:22.378642+0000","last_active":"2024-05-09T08:48:24.682057+0000","last_peered":"2024-05-09T08:48:24.682057+0000","last_clean":"2024-05-09T08:48:24.682057+0000","last_became_active":"2024-05-09T08:48:22.378161+0000","last_became_peered":"2024-05-09T08:48:22.378161+0000","last_unstale":"2024-05-09T08:48:24.682057+0000","last_undegraded":"2024-05-09T08:48:24.682057+0000","last_fullsized":"2024-05-09T08:48:24.682057+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:53.149728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.778823+0000","last_change":"2024-05-09T08:48:17.599499+0000","last_active":"2024-05-09T08:48:23.778823+0000","last_peered":"2024-05-09T08:48:23.778823+0000","last_clean":"2024-05-09T08:48:23.778823+0000","last_became_active":"2024-05-09T08:48:17.599145+0000","last_became_peered":"2024-05-09T08:48:17.599145+0000","last_unstale":"2024-05-09T08:48:23.778823+0000","last_undegraded":"2024-05-09T08:48:23.778823+0000","last_fullsized":"2024-05-09T08:48:23.778823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:31:12.502004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538875+0000","last_change":"2024-05-09T08:48:18.372355+0000","last_active":"2024-05-09T08:48:25.538875+0000","last_peered":"2024-05-09T08:48:25.538875+0000","last_clean":"2024-05-09T08:48:25.538875+0000","last_became_active":"2024-05-09T08:48:18.367743+0000","last_became_peered":"2024-05-09T08:48:18.367743+0000","last_unstale":"2024-05-09T08:48:25.538875+0000","last_undegraded":"2024-05-09T08:48:25.538875+0000","last_fullsized":"2024-05-09T08:48:25.538875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:31:12.174350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603861+0000","last_change":"2024-05-09T08:48:22.384064+0000","last_active":"2024-05-09T08:48:23.603861+0000","last_peered":"2024-05-09T08:48:23.603861+0000","last_clean":"2024-05-09T08:48:23.603861+0000","last_became_active":"2024-05-09T08:48:22.383670+0000","last_became_peered":"2024-05-09T08:48:22.383670+0000","last_unstale":"2024-05-09T08:48:23.603861+0000","last_undegraded":"2024-05-09T08:48:23.603861+0000","last_fullsized":"2024-05-09T08:48:23.603861+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:51:35.000827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:38.750465+0000","last_change":"2024-05-09T08:48:17.601596+0000","last_active":"2024-05-09T08:48:38.750465+0000","last_peered":"2024-05-09T08:48:38.750465+0000","last_clean":"2024-05-09T08:48:38.750465+0000","last_became_active":"2024-05-09T08:48:17.601250+0000","last_became_peered":"2024-05-09T08:48:17.601250+0000","last_unstale":"2024-05-09T08:48:38.750465+0000","last_undegraded":"2024-05-09T08:48:38.750465+0000","last_fullsized":"2024-05-09T08:48:38.750465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:48.237336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538623+0000","last_change":"2024-05-09T08:48:18.368751+0000","last_active":"2024-05-09T08:48:25.538623+0000","last_peered":"2024-05-09T08:48:25.538623+0000","last_clean":"2024-05-09T08:48:25.538623+0000","last_became_active":"2024-05-09T08:48:18.367463+0000","last_became_peered":"2024-05-09T08:48:18.367463+0000","last_unstale":"2024-05-09T08:48:25.538623+0000","last_undegraded":"2024-05-09T08:48:25.538623+0000","last_fullsized":"2024-05-09T08:48:25.538623+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:19:17.696602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.604032+0000","last_change":"2024-05-09T08:48:22.385025+0000","last_active":"2024-05-09T08:48:23.604032+0000","last_peered":"2024-05-09T08:48:23.604032+0000","last_clean":"2024-05-09T08:48:23.604032+0000","last_became_active":"2024-05-09T08:48:22.384633+0000","last_became_peered":"2024-05-09T08:48:22.384633+0000","last_unstale":"2024-05-09T08:48:23.604032+0000","last_undegraded":"2024-05-09T08:48:23.604032+0000","last_fullsized":"2024-05-09T08:48:23.604032+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:17.804075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608209+0000","last_change":"2024-05-09T08:48:18.376741+0000","last_active":"2024-05-09T08:48:23.608209+0000","last_peered":"2024-05-09T08:48:23.608209+0000","last_clean":"2024-05-09T08:48:23.608209+0000","last_became_active":"2024-05-09T08:48:18.376376+0000","last_became_peered":"2024-05-09T08:48:18.376376+0000","last_unstale":"2024-05-09T08:48:23.608209+0000","last_undegraded":"2024-05-09T08:48:23.608209+0000","last_fullsized":"2024-05-09T08:48:23.608209+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:58:31.717555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:39.735437+0000","last_change":"2024-05-09T08:48:17.598316+0000","last_active":"2024-05-09T08:48:39.735437+0000","last_peered":"2024-05-09T08:48:39.735437+0000","last_clean":"2024-05-09T08:48:39.735437+0000","last_became_active":"2024-05-09T08:48:17.598022+0000","last_became_peered":"2024-05-09T08:48:17.598022+0000","last_unstale":"2024-05-09T08:48:39.735437+0000","last_undegraded":"2024-05-09T08:48:39.735437+0000","last_fullsized":"2024-05-09T08:48:39.735437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:04:10.454894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.528407+0000","last_change":"2024-05-09T08:48:22.374123+0000","last_active":"2024-05-09T08:48:24.528407+0000","last_peered":"2024-05-09T08:48:24.528407+0000","last_clean":"2024-05-09T08:48:24.528407+0000","last_became_active":"2024-05-09T08:48:22.373725+0000","last_became_peered":"2024-05-09T08:48:22.373725+0000","last_unstale":"2024-05-09T08:48:24.528407+0000","last_undegraded":"2024-05-09T08:48:24.528407+0000","last_fullsized":"2024-05-09T08:48:24.528407+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:32:53.171556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537571+0000","last_change":"2024-05-09T08:48:18.379203+0000","last_active":"2024-05-09T08:48:25.537571+0000","last_peered":"2024-05-09T08:48:25.537571+0000","last_clean":"2024-05-09T08:48:25.537571+0000","last_became_active":"2024-05-09T08:48:18.378885+0000","last_became_peered":"2024-05-09T08:48:18.378885+0000","last_unstale":"2024-05-09T08:48:25.537571+0000","last_undegraded":"2024-05-09T08:48:25.537571+0000","last_fullsized":"2024-05-09T08:48:25.537571+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:40.588263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.742433+0000","last_change":"2024-05-09T08:48:17.593520+0000","last_active":"2024-05-09T08:48:37.742433+0000","last_peered":"2024-05-09T08:48:37.742433+0000","last_clean":"2024-05-09T08:48:37.742433+0000","last_became_active":"2024-05-09T08:48:17.593127+0000","last_became_peered":"2024-05-09T08:48:17.593127+0000","last_unstale":"2024-05-09T08:48:37.742433+0000","last_undegraded":"2024-05-09T08:48:37.742433+0000","last_fullsized":"2024-05-09T08:48:37.742433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:34:29.567166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":356,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608435+0000","last_change":"2024-05-09T08:47:05.718649+0000","last_active":"2024-05-09T08:48:23.608435+0000","last_peered":"2024-05-09T08:48:23.608435+0000","last_clean":"2024-05-09T08:48:23.608435+0000","last_became_active":"2024-05-09T08:47:05.718088+0000","last_became_peered":"2024-05-09T08:47:05.718088+0000","last_unstale":"2024-05-09T08:48:23.608435+0000","last_undegraded":"2024-05-09T08:48:23.608435+0000","last_fullsized":"2024-05-09T08:48:23.608435+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:43:39.457880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:43:39.457880+0000","last_clean_scrub_stamp":"2024-05-09T08:43:39.457880+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:14.227590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608605+0000","last_change":"2024-05-09T08:48:22.376751+0000","last_active":"2024-05-09T08:48:23.608605+0000","last_peered":"2024-05-09T08:48:23.608605+0000","last_clean":"2024-05-09T08:48:23.608605+0000","last_became_active":"2024-05-09T08:48:22.376322+0000","last_became_peered":"2024-05-09T08:48:22.376322+0000","last_unstale":"2024-05-09T08:48:23.608605+0000","last_undegraded":"2024-05-09T08:48:23.608605+0000","last_fullsized":"2024-05-09T08:48:23.608605+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:08:07.859034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603239+0000","last_change":"2024-05-09T08:48:18.364063+0000","last_active":"2024-05-09T08:48:23.603239+0000","last_peered":"2024-05-09T08:48:23.603239+0000","last_clean":"2024-05-09T08:48:23.603239+0000","last_became_active":"2024-05-09T08:48:18.363694+0000","last_became_peered":"2024-05-09T08:48:18.363694+0000","last_unstale":"2024-05-09T08:48:23.603239+0000","last_undegraded":"2024-05-09T08:48:23.603239+0000","last_fullsized":"2024-05-09T08:48:23.603239+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:42.536329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.289922+0000","last_change":"2024-05-09T08:48:17.599787+0000","last_active":"2024-05-09T08:48:37.289922+0000","last_peered":"2024-05-09T08:48:37.289922+0000","last_clean":"2024-05-09T08:48:37.289922+0000","last_became_active":"2024-05-09T08:48:17.599248+0000","last_became_peered":"2024-05-09T08:48:17.599248+0000","last_unstale":"2024-05-09T08:48:37.289922+0000","last_undegraded":"2024-05-09T08:48:37.289922+0000","last_fullsized":"2024-05-09T08:48:37.289922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:17.448404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537914+0000","last_change":"2024-05-09T08:48:22.378913+0000","last_active":"2024-05-09T08:48:25.537914+0000","last_peered":"2024-05-09T08:48:25.537914+0000","last_clean":"2024-05-09T08:48:25.537914+0000","last_became_active":"2024-05-09T08:48:22.378441+0000","last_became_peered":"2024-05-09T08:48:22.378441+0000","last_unstale":"2024-05-09T08:48:25.537914+0000","last_undegraded":"2024-05-09T08:48:25.537914+0000","last_fullsized":"2024-05-09T08:48:25.537914+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:49.623778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.744116+0000","last_change":"2024-05-09T08:48:17.600121+0000","last_active":"2024-05-09T08:48:37.744116+0000","last_peered":"2024-05-09T08:48:37.744116+0000","last_clean":"2024-05-09T08:48:37.744116+0000","last_became_active":"2024-05-09T08:48:17.599800+0000","last_became_peered":"2024-05-09T08:48:17.599800+0000","last_unstale":"2024-05-09T08:48:37.744116+0000","last_undegraded":"2024-05-09T08:48:37.744116+0000","last_fullsized":"2024-05-09T08:48:37.744116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:56.607393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603862+0000","last_change":"2024-05-09T08:48:18.366612+0000","last_active":"2024-05-09T08:48:23.603862+0000","last_peered":"2024-05-09T08:48:23.603862+0000","last_clean":"2024-05-09T08:48:23.603862+0000","last_became_active":"2024-05-09T08:48:18.366266+0000","last_became_peered":"2024-05-09T08:48:18.366266+0000","last_unstale":"2024-05-09T08:48:23.603862+0000","last_undegraded":"2024-05-09T08:48:23.603862+0000","last_fullsized":"2024-05-09T08:48:23.603862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:35:01.663456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537615+0000","last_change":"2024-05-09T08:48:22.384096+0000","last_active":"2024-05-09T08:48:25.537615+0000","last_peered":"2024-05-09T08:48:25.537615+0000","last_clean":"2024-05-09T08:48:25.537615+0000","last_became_active":"2024-05-09T08:48:22.383656+0000","last_became_peered":"2024-05-09T08:48:22.383656+0000","last_unstale":"2024-05-09T08:48:25.537615+0000","last_undegraded":"2024-05-09T08:48:25.537615+0000","last_fullsized":"2024-05-09T08:48:25.537615+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:13.622377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.779204+0000","last_change":"2024-05-09T08:48:17.593684+0000","last_active":"2024-05-09T08:48:23.779204+0000","last_peered":"2024-05-09T08:48:23.779204+0000","last_clean":"2024-05-09T08:48:23.779204+0000","last_became_active":"2024-05-09T08:48:17.593358+0000","last_became_peered":"2024-05-09T08:48:17.593358+0000","last_unstale":"2024-05-09T08:48:23.779204+0000","last_undegraded":"2024-05-09T08:48:23.779204+0000","last_fullsized":"2024-05-09T08:48:23.779204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:51:36.023421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538003+0000","last_change":"2024-05-09T08:48:18.374498+0000","last_active":"2024-05-09T08:48:25.538003+0000","last_peered":"2024-05-09T08:48:25.538003+0000","last_clean":"2024-05-09T08:48:25.538003+0000","last_became_active":"2024-05-09T08:48:18.374142+0000","last_became_peered":"2024-05-09T08:48:18.374142+0000","last_unstale":"2024-05-09T08:48:25.538003+0000","last_undegraded":"2024-05-09T08:48:25.538003+0000","last_fullsized":"2024-05-09T08:48:25.538003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:53.874419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888360+0000","last_change":"2024-05-09T08:48:22.378330+0000","last_active":"2024-05-09T08:48:23.888360+0000","last_peered":"2024-05-09T08:48:23.888360+0000","last_clean":"2024-05-09T08:48:23.888360+0000","last_became_active":"2024-05-09T08:48:22.377218+0000","last_became_peered":"2024-05-09T08:48:22.377218+0000","last_unstale":"2024-05-09T08:48:23.888360+0000","last_undegraded":"2024-05-09T08:48:23.888360+0000","last_fullsized":"2024-05-09T08:48:23.888360+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:41.650168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:38.747162+0000","last_change":"2024-05-09T08:48:17.601410+0000","last_active":"2024-05-09T08:48:38.747162+0000","last_peered":"2024-05-09T08:48:38.747162+0000","last_clean":"2024-05-09T08:48:38.747162+0000","last_became_active":"2024-05-09T08:48:17.601075+0000","last_became_peered":"2024-05-09T08:48:17.601075+0000","last_unstale":"2024-05-09T08:48:38.747162+0000","last_undegraded":"2024-05-09T08:48:38.747162+0000","last_fullsized":"2024-05-09T08:48:38.747162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:45.760280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782803+0000","last_change":"2024-05-09T08:48:18.379590+0000","last_active":"2024-05-09T08:48:25.782803+0000","last_peered":"2024-05-09T08:48:25.782803+0000","last_clean":"2024-05-09T08:48:25.782803+0000","last_became_active":"2024-05-09T08:48:18.379187+0000","last_became_peered":"2024-05-09T08:48:18.379187+0000","last_unstale":"2024-05-09T08:48:25.782803+0000","last_undegraded":"2024-05-09T08:48:25.782803+0000","last_fullsized":"2024-05-09T08:48:25.782803+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:40:02.630449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538370+0000","last_change":"2024-05-09T08:48:22.386437+0000","last_active":"2024-05-09T08:48:25.538370+0000","last_peered":"2024-05-09T08:48:25.538370+0000","last_clean":"2024-05-09T08:48:25.538370+0000","last_became_active":"2024-05-09T08:48:22.385973+0000","last_became_peered":"2024-05-09T08:48:22.385973+0000","last_unstale":"2024-05-09T08:48:25.538370+0000","last_undegraded":"2024-05-09T08:48:25.538370+0000","last_fullsized":"2024-05-09T08:48:25.538370+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:11:37.338641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:36.277471+0000","last_change":"2024-05-09T08:48:17.598586+0000","last_active":"2024-05-09T08:48:36.277471+0000","last_peered":"2024-05-09T08:48:36.277471+0000","last_clean":"2024-05-09T08:48:36.277471+0000","last_became_active":"2024-05-09T08:48:17.597851+0000","last_became_peered":"2024-05-09T08:48:17.597851+0000","last_unstale":"2024-05-09T08:48:36.277471+0000","last_undegraded":"2024-05-09T08:48:36.277471+0000","last_fullsized":"2024-05-09T08:48:36.277471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:28:22.996999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.681748+0000","last_change":"2024-05-09T08:48:18.367319+0000","last_active":"2024-05-09T08:48:24.681748+0000","last_peered":"2024-05-09T08:48:24.681748+0000","last_clean":"2024-05-09T08:48:24.681748+0000","last_became_active":"2024-05-09T08:48:18.366034+0000","last_became_peered":"2024-05-09T08:48:18.366034+0000","last_unstale":"2024-05-09T08:48:24.681748+0000","last_undegraded":"2024-05-09T08:48:24.681748+0000","last_fullsized":"2024-05-09T08:48:24.681748+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:22.998089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.779329+0000","last_change":"2024-05-09T08:48:22.389129+0000","last_active":"2024-05-09T08:48:23.779329+0000","last_peered":"2024-05-09T08:48:23.779329+0000","last_clean":"2024-05-09T08:48:23.779329+0000","last_became_active":"2024-05-09T08:48:22.386639+0000","last_became_peered":"2024-05-09T08:48:22.386639+0000","last_unstale":"2024-05-09T08:48:23.779329+0000","last_undegraded":"2024-05-09T08:48:23.779329+0000","last_fullsized":"2024-05-09T08:48:23.779329+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:47:07.257464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:38.747940+0000","last_change":"2024-05-09T08:48:17.595255+0000","last_active":"2024-05-09T08:48:38.747940+0000","last_peered":"2024-05-09T08:48:38.747940+0000","last_clean":"2024-05-09T08:48:38.747940+0000","last_became_active":"2024-05-09T08:48:17.594882+0000","last_became_peered":"2024-05-09T08:48:17.594882+0000","last_unstale":"2024-05-09T08:48:38.747940+0000","last_undegraded":"2024-05-09T08:48:38.747940+0000","last_fullsized":"2024-05-09T08:48:38.747940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:49.845326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537798+0000","last_change":"2024-05-09T08:48:18.378580+0000","last_active":"2024-05-09T08:48:25.537798+0000","last_peered":"2024-05-09T08:48:25.537798+0000","last_clean":"2024-05-09T08:48:25.537798+0000","last_became_active":"2024-05-09T08:48:18.378205+0000","last_became_peered":"2024-05-09T08:48:18.378205+0000","last_unstale":"2024-05-09T08:48:25.537798+0000","last_undegraded":"2024-05-09T08:48:25.537798+0000","last_fullsized":"2024-05-09T08:48:25.537798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:50.103195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538026+0000","last_change":"2024-05-09T08:48:22.378494+0000","last_active":"2024-05-09T08:48:25.538026+0000","last_peered":"2024-05-09T08:48:25.538026+0000","last_clean":"2024-05-09T08:48:25.538026+0000","last_became_active":"2024-05-09T08:48:22.378076+0000","last_became_peered":"2024-05-09T08:48:22.378076+0000","last_unstale":"2024-05-09T08:48:25.538026+0000","last_undegraded":"2024-05-09T08:48:25.538026+0000","last_fullsized":"2024-05-09T08:48:25.538026+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:40:09.079968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:38.273298+0000","last_change":"2024-05-09T08:48:17.601433+0000","last_active":"2024-05-09T08:48:38.273298+0000","last_peered":"2024-05-09T08:48:38.273298+0000","last_clean":"2024-05-09T08:48:38.273298+0000","last_became_active":"2024-05-09T08:48:17.601128+0000","last_became_peered":"2024-05-09T08:48:17.601128+0000","last_unstale":"2024-05-09T08:48:38.273298+0000","last_undegraded":"2024-05-09T08:48:38.273298+0000","last_fullsized":"2024-05-09T08:48:38.273298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:33:26.207437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537889+0000","last_change":"2024-05-09T08:48:18.373569+0000","last_active":"2024-05-09T08:48:25.537889+0000","last_peered":"2024-05-09T08:48:25.537889+0000","last_clean":"2024-05-09T08:48:25.537889+0000","last_became_active":"2024-05-09T08:48:18.373238+0000","last_became_peered":"2024-05-09T08:48:18.373238+0000","last_unstale":"2024-05-09T08:48:25.537889+0000","last_undegraded":"2024-05-09T08:48:25.537889+0000","last_fullsized":"2024-05-09T08:48:25.537889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:17:41.629419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538071+0000","last_change":"2024-05-09T08:48:22.381523+0000","last_active":"2024-05-09T08:48:25.538071+0000","last_peered":"2024-05-09T08:48:25.538071+0000","last_clean":"2024-05-09T08:48:25.538071+0000","last_became_active":"2024-05-09T08:48:22.380626+0000","last_became_peered":"2024-05-09T08:48:22.380626+0000","last_unstale":"2024-05-09T08:48:25.538071+0000","last_undegraded":"2024-05-09T08:48:25.538071+0000","last_fullsized":"2024-05-09T08:48:25.538071+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:47:45.092232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:36.278365+0000","last_change":"2024-05-09T08:48:17.601479+0000","last_active":"2024-05-09T08:48:36.278365+0000","last_peered":"2024-05-09T08:48:36.278365+0000","last_clean":"2024-05-09T08:48:36.278365+0000","last_became_active":"2024-05-09T08:48:17.601140+0000","last_became_peered":"2024-05-09T08:48:17.601140+0000","last_unstale":"2024-05-09T08:48:36.278365+0000","last_undegraded":"2024-05-09T08:48:36.278365+0000","last_fullsized":"2024-05-09T08:48:36.278365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:38:24.011209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608523+0000","last_change":"2024-05-09T08:48:18.376890+0000","last_active":"2024-05-09T08:48:23.608523+0000","last_peered":"2024-05-09T08:48:23.608523+0000","last_clean":"2024-05-09T08:48:23.608523+0000","last_became_active":"2024-05-09T08:48:18.376472+0000","last_became_peered":"2024-05-09T08:48:18.376472+0000","last_unstale":"2024-05-09T08:48:23.608523+0000","last_undegraded":"2024-05-09T08:48:23.608523+0000","last_fullsized":"2024-05-09T08:48:23.608523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:40:16.238539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.834831+0000","last_change":"2024-05-09T08:48:22.374743+0000","last_active":"2024-05-09T08:48:24.834831+0000","last_peered":"2024-05-09T08:48:24.834831+0000","last_clean":"2024-05-09T08:48:24.834831+0000","last_became_active":"2024-05-09T08:48:22.374324+0000","last_became_peered":"2024-05-09T08:48:22.374324+0000","last_unstale":"2024-05-09T08:48:24.834831+0000","last_undegraded":"2024-05-09T08:48:24.834831+0000","last_fullsized":"2024-05-09T08:48:24.834831+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:48.073647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.741649+0000","last_change":"2024-05-09T08:48:17.604700+0000","last_active":"2024-05-09T08:48:37.741649+0000","last_peered":"2024-05-09T08:48:37.741649+0000","last_clean":"2024-05-09T08:48:37.741649+0000","last_became_active":"2024-05-09T08:48:17.604334+0000","last_became_peered":"2024-05-09T08:48:17.604334+0000","last_unstale":"2024-05-09T08:48:37.741649+0000","last_undegraded":"2024-05-09T08:48:37.741649+0000","last_fullsized":"2024-05-09T08:48:37.741649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:15:37.346232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.609221+0000","last_change":"2024-05-09T08:48:18.375769+0000","last_active":"2024-05-09T08:48:23.609221+0000","last_peered":"2024-05-09T08:48:23.609221+0000","last_clean":"2024-05-09T08:48:23.609221+0000","last_became_active":"2024-05-09T08:48:18.375059+0000","last_became_peered":"2024-05-09T08:48:18.375059+0000","last_unstale":"2024-05-09T08:48:23.609221+0000","last_undegraded":"2024-05-09T08:48:23.609221+0000","last_fullsized":"2024-05-09T08:48:23.609221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:51.001240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782227+0000","last_change":"2024-05-09T08:48:22.385306+0000","last_active":"2024-05-09T08:48:25.782227+0000","last_peered":"2024-05-09T08:48:25.782227+0000","last_clean":"2024-05-09T08:48:25.782227+0000","last_became_active":"2024-05-09T08:48:22.383890+0000","last_became_peered":"2024-05-09T08:48:22.383890+0000","last_unstale":"2024-05-09T08:48:25.782227+0000","last_undegraded":"2024-05-09T08:48:25.782227+0000","last_fullsized":"2024-05-09T08:48:25.782227+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:50:52.369399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:37.729727+0000","last_change":"2024-05-09T08:48:17.591627+0000","last_active":"2024-05-09T08:48:37.729727+0000","last_peered":"2024-05-09T08:48:37.729727+0000","last_clean":"2024-05-09T08:48:37.729727+0000","last_became_active":"2024-05-09T08:48:17.591250+0000","last_became_peered":"2024-05-09T08:48:17.591250+0000","last_unstale":"2024-05-09T08:48:37.729727+0000","last_undegraded":"2024-05-09T08:48:37.729727+0000","last_fullsized":"2024-05-09T08:48:37.729727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:40:59.748892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608916+0000","last_change":"2024-05-09T08:48:18.372256+0000","last_active":"2024-05-09T08:48:23.608916+0000","last_peered":"2024-05-09T08:48:23.608916+0000","last_clean":"2024-05-09T08:48:23.608916+0000","last_became_active":"2024-05-09T08:48:18.371895+0000","last_became_peered":"2024-05-09T08:48:18.371895+0000","last_unstale":"2024-05-09T08:48:23.608916+0000","last_undegraded":"2024-05-09T08:48:23.608916+0000","last_fullsized":"2024-05-09T08:48:23.608916+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:25.067587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.607835+0000","last_change":"2024-05-09T08:48:22.373684+0000","last_active":"2024-05-09T08:48:23.607835+0000","last_peered":"2024-05-09T08:48:23.607835+0000","last_clean":"2024-05-09T08:48:23.607835+0000","last_became_active":"2024-05-09T08:48:22.373000+0000","last_became_peered":"2024-05-09T08:48:22.373000+0000","last_unstale":"2024-05-09T08:48:23.607835+0000","last_undegraded":"2024-05-09T08:48:23.607835+0000","last_fullsized":"2024-05-09T08:48:23.607835+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:53:14.416837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:38.746641+0000","last_change":"2024-05-09T08:48:17.596910+0000","last_active":"2024-05-09T08:48:38.746641+0000","last_peered":"2024-05-09T08:48:38.746641+0000","last_clean":"2024-05-09T08:48:38.746641+0000","last_became_active":"2024-05-09T08:48:17.596605+0000","last_became_peered":"2024-05-09T08:48:17.596605+0000","last_unstale":"2024-05-09T08:48:38.746641+0000","last_undegraded":"2024-05-09T08:48:38.746641+0000","last_fullsized":"2024-05-09T08:48:38.746641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:05:25.880082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.609245+0000","last_change":"2024-05-09T08:48:18.370354+0000","last_active":"2024-05-09T08:48:23.609245+0000","last_peered":"2024-05-09T08:48:23.609245+0000","last_clean":"2024-05-09T08:48:23.609245+0000","last_became_active":"2024-05-09T08:48:18.370043+0000","last_became_peered":"2024-05-09T08:48:18.370043+0000","last_unstale":"2024-05-09T08:48:23.609245+0000","last_undegraded":"2024-05-09T08:48:23.609245+0000","last_fullsized":"2024-05-09T08:48:23.609245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:21:34.066162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537971+0000","last_change":"2024-05-09T08:48:18.376431+0000","last_active":"2024-05-09T08:48:25.537971+0000","last_peered":"2024-05-09T08:48:25.537971+0000","last_clean":"2024-05-09T08:48:25.537971+0000","last_became_active":"2024-05-09T08:48:18.375598+0000","last_became_peered":"2024-05-09T08:48:18.375598+0000","last_unstale":"2024-05-09T08:48:25.537971+0000","last_undegraded":"2024-05-09T08:48:25.537971+0000","last_fullsized":"2024-05-09T08:48:25.537971+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:51:13.481219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888610+0000","last_change":"2024-05-09T08:48:18.371460+0000","last_active":"2024-05-09T08:48:23.888610+0000","last_peered":"2024-05-09T08:48:23.888610+0000","last_clean":"2024-05-09T08:48:23.888610+0000","last_became_active":"2024-05-09T08:48:18.371091+0000","last_became_peered":"2024-05-09T08:48:18.371091+0000","last_unstale":"2024-05-09T08:48:23.888610+0000","last_undegraded":"2024-05-09T08:48:23.888610+0000","last_fullsized":"2024-05-09T08:48:23.888610+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:02.829902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888201+0000","last_change":"2024-05-09T08:48:18.369453+0000","last_active":"2024-05-09T08:48:23.888201+0000","last_peered":"2024-05-09T08:48:23.888201+0000","last_clean":"2024-05-09T08:48:23.888201+0000","last_became_active":"2024-05-09T08:48:18.368966+0000","last_became_peered":"2024-05-09T08:48:18.368966+0000","last_unstale":"2024-05-09T08:48:23.888201+0000","last_undegraded":"2024-05-09T08:48:23.888201+0000","last_fullsized":"2024-05-09T08:48:23.888201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:23:14.359222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.782577+0000","last_change":"2024-05-09T08:48:18.378815+0000","last_active":"2024-05-09T08:48:25.782577+0000","last_peered":"2024-05-09T08:48:25.782577+0000","last_clean":"2024-05-09T08:48:25.782577+0000","last_became_active":"2024-05-09T08:48:18.378479+0000","last_became_peered":"2024-05-09T08:48:18.378479+0000","last_unstale":"2024-05-09T08:48:25.782577+0000","last_undegraded":"2024-05-09T08:48:25.782577+0000","last_fullsized":"2024-05-09T08:48:25.782577+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:38:50.133239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603926+0000","last_change":"2024-05-09T08:48:18.365696+0000","last_active":"2024-05-09T08:48:23.603926+0000","last_peered":"2024-05-09T08:48:23.603926+0000","last_clean":"2024-05-09T08:48:23.603926+0000","last_became_active":"2024-05-09T08:48:18.365358+0000","last_became_peered":"2024-05-09T08:48:18.365358+0000","last_unstale":"2024-05-09T08:48:23.603926+0000","last_undegraded":"2024-05-09T08:48:23.603926+0000","last_fullsized":"2024-05-09T08:48:23.603926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:04.064150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538124+0000","last_change":"2024-05-09T08:48:18.368911+0000","last_active":"2024-05-09T08:48:25.538124+0000","last_peered":"2024-05-09T08:48:25.538124+0000","last_clean":"2024-05-09T08:48:25.538124+0000","last_became_active":"2024-05-09T08:48:18.368591+0000","last_became_peered":"2024-05-09T08:48:18.368591+0000","last_unstale":"2024-05-09T08:48:25.538124+0000","last_undegraded":"2024-05-09T08:48:25.538124+0000","last_fullsized":"2024-05-09T08:48:25.538124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:34:09.692168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.888820+0000","last_change":"2024-05-09T08:48:18.373629+0000","last_active":"2024-05-09T08:48:23.888820+0000","last_peered":"2024-05-09T08:48:23.888820+0000","last_clean":"2024-05-09T08:48:23.888820+0000","last_became_active":"2024-05-09T08:48:18.372992+0000","last_became_peered":"2024-05-09T08:48:18.372992+0000","last_unstale":"2024-05-09T08:48:23.888820+0000","last_undegraded":"2024-05-09T08:48:23.888820+0000","last_fullsized":"2024-05-09T08:48:23.888820+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:42.532250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538104+0000","last_change":"2024-05-09T08:48:18.382423+0000","last_active":"2024-05-09T08:48:25.538104+0000","last_peered":"2024-05-09T08:48:25.538104+0000","last_clean":"2024-05-09T08:48:25.538104+0000","last_became_active":"2024-05-09T08:48:18.382088+0000","last_became_peered":"2024-05-09T08:48:18.382088+0000","last_unstale":"2024-05-09T08:48:25.538104+0000","last_undegraded":"2024-05-09T08:48:25.538104+0000","last_fullsized":"2024-05-09T08:48:25.538104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:09:06.342557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.608067+0000","last_change":"2024-05-09T08:48:18.370460+0000","last_active":"2024-05-09T08:48:23.608067+0000","last_peered":"2024-05-09T08:48:23.608067+0000","last_clean":"2024-05-09T08:48:23.608067+0000","last_became_active":"2024-05-09T08:48:18.370033+0000","last_became_peered":"2024-05-09T08:48:18.370033+0000","last_unstale":"2024-05-09T08:48:23.608067+0000","last_undegraded":"2024-05-09T08:48:23.608067+0000","last_fullsized":"2024-05-09T08:48:23.608067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:21:28.622511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.537729+0000","last_change":"2024-05-09T08:48:18.371410+0000","last_active":"2024-05-09T08:48:25.537729+0000","last_peered":"2024-05-09T08:48:25.537729+0000","last_clean":"2024-05-09T08:48:25.537729+0000","last_became_active":"2024-05-09T08:48:18.371067+0000","last_became_peered":"2024-05-09T08:48:18.371067+0000","last_unstale":"2024-05-09T08:48:25.537729+0000","last_undegraded":"2024-05-09T08:48:25.537729+0000","last_fullsized":"2024-05-09T08:48:25.537729+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:35:11.445992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:24.834876+0000","last_change":"2024-05-09T08:48:18.382873+0000","last_active":"2024-05-09T08:48:24.834876+0000","last_peered":"2024-05-09T08:48:24.834876+0000","last_clean":"2024-05-09T08:48:24.834876+0000","last_became_active":"2024-05-09T08:48:18.382366+0000","last_became_peered":"2024-05-09T08:48:18.382366+0000","last_unstale":"2024-05-09T08:48:24.834876+0000","last_undegraded":"2024-05-09T08:48:24.834876+0000","last_fullsized":"2024-05-09T08:48:24.834876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:23.343131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:23.603590+0000","last_change":"2024-05-09T08:48:18.370821+0000","last_active":"2024-05-09T08:48:23.603590+0000","last_peered":"2024-05-09T08:48:23.603590+0000","last_clean":"2024-05-09T08:48:23.603590+0000","last_became_active":"2024-05-09T08:48:18.368208+0000","last_became_peered":"2024-05-09T08:48:18.368208+0000","last_unstale":"2024-05-09T08:48:23.603590+0000","last_undegraded":"2024-05-09T08:48:23.603590+0000","last_fullsized":"2024-05-09T08:48:23.603590+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:14.633938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-09T08:48:25.538184+0000","last_change":"2024-05-09T08:48:18.371626+0000","last_active":"2024-05-09T08:48:25.538184+0000","last_peered":"2024-05-09T08:48:25.538184+0000","last_clean":"2024-05-09T08:48:25.538184+0000","last_became_active":"2024-05-09T08:48:18.371081+0000","last_became_peered":"2024-05-09T08:48:18.371081+0000","last_unstale":"2024-05-09T08:48:25.538184+0000","last_undegraded":"2024-05-09T08:48:25.538184+0000","last_fullsized":"2024-05-09T08:48:25.538184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:27.372934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":210578,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10565,"num_objects":89,"num_object_clones":0,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":32,"num_read_kb":31,"num_write":187,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":86630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163,"ondisk_log_size":163,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350993,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":482037,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612630,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":1076633,"data_compressed":50263,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776155,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":481629,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037793,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":486441,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299429,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1075137,"data_compressed":50263,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":481567,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1032192,"data_stored":523068,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":481539,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":486659,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":481253,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888117,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":481511,"data_compressed":14620,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724591,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":1072689,"data_compressed":50263,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34003,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34003,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":42223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":25504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:48:41.473 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-09T08:48:41.473 INFO:tasks.cephadm:Setting up client nodes... 2024-05-09T08:48:41.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-09T08:48:41.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:41 smithi105 ceph-mon[34498]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:41.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:41 smithi105 ceph-mon[34498]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:41.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:41 smithi105 ceph-mon[34498]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:41.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:41 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2726581038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:41.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:41 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3686423068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:41.487 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 08:48:41 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h[64536]: 2024-05-09T08:48:41.260+0000 7f19d652a700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-05-09T08:48:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:41 smithi142 ceph-mon[34625]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:41 smithi142 ceph-mon[34625]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:41 smithi142 ceph-mon[34625]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:41 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2726581038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:41 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3686423068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:41.753 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:42.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:42 smithi098 ceph-mon[30092]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:48:42.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:42 smithi098 ceph-mon[30092]: pgmap v302: 129 pgs: 129 active+clean; 587 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-05-09T08:48:42.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:42 smithi142 ceph-mon[34625]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:48:42.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:42 smithi142 ceph-mon[34625]: pgmap v302: 129 pgs: 129 active+clean; 587 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-05-09T08:48:42.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:42 smithi105 ceph-mon[34498]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:48:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:42 smithi105 ceph-mon[34498]: pgmap v302: 129 pgs: 129 active+clean; 587 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-05-09T08:48:43.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:43 smithi098 ceph-mon[30092]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.582+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/1813553469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d88104000 msgr2=0x7f2d8810a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.582+0000 7f2d8cbd2700 1 --2- 172.21.15.98:0/1813553469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d88104000 0x7f2d8810a8b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2d8000b0a0 tx=0x7f2d8000b3b0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.582+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/1813553469 shutdown_connections 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.582+0000 7f2d8cbd2700 1 --2- 172.21.15.98:0/1813553469 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d88100680 0x7f2d88100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.582+0000 7f2d8cbd2700 1 --2- 172.21.15.98:0/1813553469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d88104000 0x7f2d8810a8b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.582+0000 7f2d8cbd2700 1 --2- 172.21.15.98:0/1813553469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d88103660 0x7f2d88103ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.582+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/1813553469 >> 172.21.15.98:0/1813553469 conn(0x7f2d880fc000 msgr2=0x7f2d880fe420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.582+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/1813553469 shutdown_connections 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.582+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/1813553469 wait complete. 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.582+0000 7f2d8cbd2700 1 Processor -- start 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d8cbd2700 1 -- start start 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d8cbd2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d88100680 0x7f2d881983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d8cbd2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d88103660 0x7f2d881988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d8cbd2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d88104000 0x7f2d8819f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d8cbd2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8810dfa0 con 0x7f2d88104000 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d8cbd2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8810de20 con 0x7f2d88100680 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d8cbd2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8810e120 con 0x7f2d88103660 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d86d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d88104000 0x7f2d8819f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d86d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d88104000 0x7f2d8819f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55452/0 (socket says 172.21.15.98:55452) 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d86d9d700 1 -- 172.21.15.98:0/538403582 learned_addr learned my addr 172.21.15.98:0/538403582 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d85d9b700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d88103660 0x7f2d881988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d8659c700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d88100680 0x7f2d881983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d86d9d700 1 -- 172.21.15.98:0/538403582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d88103660 msgr2=0x7f2d881988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d86d9d700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d88103660 0x7f2d881988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d86d9d700 1 -- 172.21.15.98:0/538403582 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d88100680 msgr2=0x7f2d881983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d86d9d700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d88100680 0x7f2d881983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d86d9d700 1 -- 172.21.15.98:0/538403582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d881a0000 con 0x7f2d88104000 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d8659c700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d88100680 0x7f2d881983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d85d9b700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d88103660 0x7f2d881988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d86d9d700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d88104000 0x7f2d8819f960 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2d8000d010 tx=0x7f2d80004e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d777fe700 1 -- 172.21.15.98:0/538403582 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d80021070 con 0x7f2d88104000 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/538403582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d881a0290 con 0x7f2d88104000 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d777fe700 1 -- 172.21.15.98:0/538403582 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d80004020 con 0x7f2d88104000 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.583+0000 7f2d777fe700 1 -- 172.21.15.98:0/538403582 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8001d6b0 con 0x7f2d88104000 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.584+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/538403582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d881a0740 con 0x7f2d88104000 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.585+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/538403582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d88050890 con 0x7f2d88104000 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.586+0000 7f2d777fe700 1 -- 172.21.15.98:0/538403582 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2d800041d0 con 0x7f2d88104000 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.587+0000 7f2d777fe700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2d700a2d60 0x7f2d700a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.587+0000 7f2d8659c700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2d700a2d60 0x7f2d700a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.587+0000 7f2d777fe700 1 -- 172.21.15.98:0/538403582 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f2d800d0ae0 con 0x7f2d88104000 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.587+0000 7f2d8659c700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2d700a2d60 0x7f2d700a5220 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2d8806a8b0 tx=0x7f2d7c00a680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.590+0000 7f2d777fe700 1 -- 172.21.15.98:0/538403582 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2d8009d730 con 0x7f2d88104000 2024-05-09T08:48:43.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:43 smithi142 ceph-mon[34625]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:48:43.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:43 smithi105 ceph-mon[34498]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:48:43.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.869+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/538403582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f2d8802db80 con 0x7f2d88104000 2024-05-09T08:48:43.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.871+0000 7f2d777fe700 1 -- 172.21.15.98:0/538403582 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f2d8009d030 con 0x7f2d88104000 2024-05-09T08:48:43.871 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-05-09T08:48:43.871 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDrjTxm8t3qMxAAYjuII2a+Al1ViBX9mkXZQA== 2024-05-09T08:48:43.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.874+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/538403582 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2d700a2d60 msgr2=0x7f2d700a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.874+0000 7f2d8cbd2700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2d700a2d60 0x7f2d700a5220 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2d8806a8b0 tx=0x7f2d7c00a680 comp rx=0 tx=0).stop 2024-05-09T08:48:43.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.874+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/538403582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d88104000 msgr2=0x7f2d8819f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.874+0000 7f2d8cbd2700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d88104000 0x7f2d8819f960 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2d8000d010 tx=0x7f2d80004e80 comp rx=0 tx=0).stop 2024-05-09T08:48:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.874+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/538403582 shutdown_connections 2024-05-09T08:48:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.874+0000 7f2d8cbd2700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2d700a2d60 0x7f2d700a5220 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.874+0000 7f2d8cbd2700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d88100680 0x7f2d881983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.874+0000 7f2d8cbd2700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d88104000 0x7f2d8819f960 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.874+0000 7f2d8cbd2700 1 --2- 172.21.15.98:0/538403582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d88103660 0x7f2d881988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.874+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/538403582 >> 172.21.15.98:0/538403582 conn(0x7f2d880fc000 msgr2=0x7f2d88106890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.874+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/538403582 shutdown_connections 2024-05-09T08:48:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:43.874+0000 7f2d8cbd2700 1 -- 172.21.15.98:0/538403582 wait complete. 2024-05-09T08:48:44.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:44 smithi098 ceph-mon[30092]: pgmap v303: 129 pgs: 129 active+clean; 587 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 4.8 KiB/s wr, 17 op/s 2024-05-09T08:48:44.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:44 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/538403582' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-09T08:48:44.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:44 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/538403582' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-09T08:48:44.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:44 smithi142 ceph-mon[34625]: pgmap v303: 129 pgs: 129 active+clean; 587 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 4.8 KiB/s wr, 17 op/s 2024-05-09T08:48:44.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:44 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/538403582' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-09T08:48:44.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:44 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/538403582' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-09T08:48:44.682 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:48:44.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-09T08:48:44.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-09T08:48:44.725 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-09T08:48:44.725 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-09T08:48:44.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph mgr dump --format=json 2024-05-09T08:48:44.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:44 smithi105 ceph-mon[34498]: pgmap v303: 129 pgs: 129 active+clean; 587 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 4.8 KiB/s wr, 17 op/s 2024-05-09T08:48:44.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:44 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/538403582' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-09T08:48:44.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:44 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/538403582' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-09T08:48:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:46.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:46 smithi098 ceph-mon[30092]: pgmap v304: 129 pgs: 129 active+clean; 628 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.7 KiB/s wr, 22 op/s 2024-05-09T08:48:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:46 smithi142 ceph-mon[34625]: pgmap v304: 129 pgs: 129 active+clean; 628 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.7 KiB/s wr, 22 op/s 2024-05-09T08:48:46.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:46 smithi105 ceph-mon[34498]: pgmap v304: 129 pgs: 129 active+clean; 628 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.7 KiB/s wr, 22 op/s 2024-05-09T08:48:46.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.749+0000 7f690854d700 1 -- 172.21.15.98:0/2926242327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6900101bd0 msgr2=0x7f690010e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.749+0000 7f690854d700 1 --2- 172.21.15.98:0/2926242327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6900101bd0 0x7f690010e0c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f68f800b0a0 tx=0x7f68f800b3b0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.749+0000 7f690854d700 1 -- 172.21.15.98:0/2926242327 shutdown_connections 2024-05-09T08:48:46.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.749+0000 7f690854d700 1 --2- 172.21.15.98:0/2926242327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6900107280 0x7f6900107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.749+0000 7f690854d700 1 --2- 172.21.15.98:0/2926242327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6900101bd0 0x7f690010e0c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.749+0000 7f690854d700 1 --2- 172.21.15.98:0/2926242327 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6900101210 0x7f6900101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.749+0000 7f690854d700 1 -- 172.21.15.98:0/2926242327 >> 172.21.15.98:0/2926242327 conn(0x7f6900078190 msgr2=0x7f69000785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:46.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.749+0000 7f690854d700 1 -- 172.21.15.98:0/2926242327 shutdown_connections 2024-05-09T08:48:46.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.749+0000 7f690854d700 1 -- 172.21.15.98:0/2926242327 wait complete. 2024-05-09T08:48:46.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.750+0000 7f690854d700 1 Processor -- start 2024-05-09T08:48:46.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.750+0000 7f690854d700 1 -- start start 2024-05-09T08:48:46.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.750+0000 7f690854d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6900101210 0x7f690019e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.750+0000 7f690854d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6900101bd0 0x7f690019ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.750+0000 7f690854d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6900107280 0x7f69001988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.750+0000 7f690854d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f69001115a0 con 0x7f6900101210 2024-05-09T08:48:46.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.750+0000 7f690854d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6900111420 con 0x7f6900101bd0 2024-05-09T08:48:46.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.750+0000 7f690854d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6900111720 con 0x7f6900107280 2024-05-09T08:48:46.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.750+0000 7f69062e9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6900101210 0x7f690019e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.750+0000 7f69062e9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6900101210 0x7f690019e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52832/0 (socket says 172.21.15.98:52832) 2024-05-09T08:48:46.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.750+0000 7f69062e9700 1 -- 172.21.15.98:0/2772739607 learned_addr learned my addr 172.21.15.98:0/2772739607 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.750+0000 7f6905ae8700 1 --2- 172.21.15.98:0/2772739607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6900101bd0 0x7f690019ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.750+0000 7f6906aea700 1 --2- 172.21.15.98:0/2772739607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6900107280 0x7f69001988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.751+0000 7f69062e9700 1 -- 172.21.15.98:0/2772739607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6900107280 msgr2=0x7f69001988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.751+0000 7f69062e9700 1 --2- 172.21.15.98:0/2772739607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6900107280 0x7f69001988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.751+0000 7f69062e9700 1 -- 172.21.15.98:0/2772739607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6900101bd0 msgr2=0x7f690019ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.751+0000 7f69062e9700 1 --2- 172.21.15.98:0/2772739607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6900101bd0 0x7f690019ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.751+0000 7f69062e9700 1 -- 172.21.15.98:0/2772739607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6900199050 con 0x7f6900101210 2024-05-09T08:48:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.751+0000 7f69062e9700 1 --2- 172.21.15.98:0/2772739607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6900101210 0x7f690019e730 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f68fc00b530 tx=0x7f68fc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.751+0000 7f68f77fe700 1 -- 172.21.15.98:0/2772739607 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc015020 con 0x7f6900101210 2024-05-09T08:48:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.751+0000 7f690854d700 1 -- 172.21.15.98:0/2772739607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6900199340 con 0x7f6900101210 2024-05-09T08:48:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.751+0000 7f68f77fe700 1 -- 172.21.15.98:0/2772739607 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68fc011d10 con 0x7f6900101210 2024-05-09T08:48:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.751+0000 7f68f77fe700 1 -- 172.21.15.98:0/2772739607 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc00f2e0 con 0x7f6900101210 2024-05-09T08:48:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.751+0000 7f690854d700 1 -- 172.21.15.98:0/2772739607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69001a5600 con 0x7f6900101210 2024-05-09T08:48:46.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.752+0000 7f690854d700 1 -- 172.21.15.98:0/2772739607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6900069010 con 0x7f6900101210 2024-05-09T08:48:46.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.753+0000 7f68f77fe700 1 -- 172.21.15.98:0/2772739607 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f68fc018020 con 0x7f6900101210 2024-05-09T08:48:46.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.754+0000 7f68f77fe700 1 --2- 172.21.15.98:0/2772739607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f68ec0a2cb0 0x7f68ec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.754+0000 7f6905ae8700 1 --2- 172.21.15.98:0/2772739607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f68ec0a2cb0 0x7f68ec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.754+0000 7f68f77fe700 1 -- 172.21.15.98:0/2772739607 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f68fc0cee10 con 0x7f6900101210 2024-05-09T08:48:46.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.754+0000 7f6905ae8700 1 --2- 172.21.15.98:0/2772739607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f68ec0a2cb0 0x7f68ec0a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6900107a10 tx=0x7f68f000b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:46.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:46.757+0000 7f68f77fe700 1 -- 172.21.15.98:0/2772739607 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f68fc01f020 con 0x7f6900101210 2024-05-09T08:48:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:47.012+0000 7f690854d700 1 -- 172.21.15.98:0/2772739607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f69001a5890 con 0x7f6900101210 2024-05-09T08:48:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:47.017+0000 7f68f77fe700 1 -- 172.21.15.98:0/2772739607 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263200 (secure 0 0 0) 0x7f68fc09bae0 con 0x7f6900101210 2024-05-09T08:48:47.020 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:47.020+0000 7f690854d700 1 -- 172.21.15.98:0/2772739607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f68ec0a2cb0 msgr2=0x7f68ec0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:47.020+0000 7f690854d700 1 --2- 172.21.15.98:0/2772739607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f68ec0a2cb0 0x7f68ec0a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6900107a10 tx=0x7f68f000b3c0 comp rx=0 tx=0).stop 2024-05-09T08:48:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:47.020+0000 7f690854d700 1 -- 172.21.15.98:0/2772739607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6900101210 msgr2=0x7f690019e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:47.020+0000 7f690854d700 1 --2- 172.21.15.98:0/2772739607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6900101210 0x7f690019e730 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f68fc00b530 tx=0x7f68fc00b840 comp rx=0 tx=0).stop 2024-05-09T08:48:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:47.020+0000 7f690854d700 1 -- 172.21.15.98:0/2772739607 shutdown_connections 2024-05-09T08:48:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:47.020+0000 7f690854d700 1 --2- 172.21.15.98:0/2772739607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f68ec0a2cb0 0x7f68ec0a5170 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:47.020+0000 7f690854d700 1 --2- 172.21.15.98:0/2772739607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6900107280 0x7f69001988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:47.020+0000 7f690854d700 1 --2- 172.21.15.98:0/2772739607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6900101210 0x7f690019e730 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:47.020+0000 7f690854d700 1 --2- 172.21.15.98:0/2772739607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6900101bd0 0x7f690019ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:47.020+0000 7f690854d700 1 -- 172.21.15.98:0/2772739607 >> 172.21.15.98:0/2772739607 conn(0x7f6900078190 msgr2=0x7f6900100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:47.020+0000 7f690854d700 1 -- 172.21.15.98:0/2772739607 shutdown_connections 2024-05-09T08:48:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:47.020+0000 7f690854d700 1 -- 172.21.15.98:0/2772739607 wait complete. 2024-05-09T08:48:47.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:47 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2772739607' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-09T08:48:47.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:47 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2772739607' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-09T08:48:47.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:47 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2772739607' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1228297308},{"type":"v1","addr":"172.21.15.98:6801","nonce":1228297308}]},"active_addr":"172.21.15.98:6801/1228297308","active_change":"2024-05-09T08:41:17.098382+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.98:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1734294041}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1013153207}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":2113007677}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":808614683}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":4168866683}]}]} 2024-05-09T08:48:47.883 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-09T08:48:47.883 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-09T08:48:47.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:48.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:48 smithi098 ceph-mon[30092]: pgmap v305: 129 pgs: 129 active+clean; 628 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2024-05-09T08:48:48.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:48 smithi142 ceph-mon[34625]: pgmap v305: 129 pgs: 129 active+clean; 628 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2024-05-09T08:48:48.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:48 smithi105 ceph-mon[34498]: pgmap v305: 129 pgs: 129 active+clean; 628 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2024-05-09T08:48:49.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.947+0000 7fc6d2b52700 1 -- 172.21.15.98:0/857749811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6cc1069c0 msgr2=0x7fc6cc106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.947+0000 7fc6d2b52700 1 --2- 172.21.15.98:0/857749811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6cc1069c0 0x7fc6cc106da0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc6c00099c0 tx=0x7fc6c0009cd0 comp rx=0 tx=0).stop 2024-05-09T08:48:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 -- 172.21.15.98:0/857749811 shutdown_connections 2024-05-09T08:48:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 --2- 172.21.15.98:0/857749811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6cc1016f0 0x7fc6cc10d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 --2- 172.21.15.98:0/857749811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6cc1069c0 0x7fc6cc106da0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 --2- 172.21.15.98:0/857749811 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6cc100d30 0x7fc6cc1011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 -- 172.21.15.98:0/857749811 >> 172.21.15.98:0/857749811 conn(0x7fc6cc077420 msgr2=0x7fc6cc0fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 -- 172.21.15.98:0/857749811 shutdown_connections 2024-05-09T08:48:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 -- 172.21.15.98:0/857749811 wait complete. 2024-05-09T08:48:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 Processor -- start 2024-05-09T08:48:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 -- start start 2024-05-09T08:48:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6cc100d30 0x7fc6cc19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6cc1016f0 0x7fc6cc19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6cc1069c0 0x7fc6cc198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc6cc110460 con 0x7fc6cc1016f0 2024-05-09T08:48:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc6cc1102e0 con 0x7fc6cc1069c0 2024-05-09T08:48:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.948+0000 7fc6d2b52700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc6cc1105e0 con 0x7fc6cc100d30 2024-05-09T08:48:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6cc1016f0 0x7fc6cc19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6cc1016f0 0x7fc6cc19ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52856/0 (socket says 172.21.15.98:52856) 2024-05-09T08:48:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6cbfff700 1 -- 172.21.15.98:0/1128196101 learned_addr learned my addr 172.21.15.98:0/1128196101 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6d08ee700 1 --2- 172.21.15.98:0/1128196101 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6cc100d30 0x7fc6cc19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6d10ef700 1 --2- 172.21.15.98:0/1128196101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6cc1069c0 0x7fc6cc198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6cbfff700 1 -- 172.21.15.98:0/1128196101 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6cc100d30 msgr2=0x7fc6cc19e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6cbfff700 1 --2- 172.21.15.98:0/1128196101 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6cc100d30 0x7fc6cc19e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6cbfff700 1 -- 172.21.15.98:0/1128196101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6cc1069c0 msgr2=0x7fc6cc198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:49.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6cbfff700 1 --2- 172.21.15.98:0/1128196101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6cc1069c0 0x7fc6cc198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6cbfff700 1 -- 172.21.15.98:0/1128196101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6cc1990d0 con 0x7fc6cc1016f0 2024-05-09T08:48:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6cbfff700 1 --2- 172.21.15.98:0/1128196101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6cc1016f0 0x7fc6cc19ec50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800ddc0 tx=0x7fc6b800a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6c9ffb700 1 -- 172.21.15.98:0/1128196101 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b801b070 con 0x7fc6cc1016f0 2024-05-09T08:48:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6d2b52700 1 -- 172.21.15.98:0/1128196101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6cc1993c0 con 0x7fc6cc1016f0 2024-05-09T08:48:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6c9ffb700 1 -- 172.21.15.98:0/1128196101 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6b800ad40 con 0x7fc6cc1016f0 2024-05-09T08:48:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6c9ffb700 1 -- 172.21.15.98:0/1128196101 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b80167d0 con 0x7fc6cc1016f0 2024-05-09T08:48:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.949+0000 7fc6d2b52700 1 -- 172.21.15.98:0/1128196101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6cc1a55e0 con 0x7fc6cc1016f0 2024-05-09T08:48:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.950+0000 7fc6d2b52700 1 -- 172.21.15.98:0/1128196101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6cc102090 con 0x7fc6cc1016f0 2024-05-09T08:48:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.951+0000 7fc6c9ffb700 1 -- 172.21.15.98:0/1128196101 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc6b801e070 con 0x7fc6cc1016f0 2024-05-09T08:48:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.952+0000 7fc6c9ffb700 1 --2- 172.21.15.98:0/1128196101 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc6bc0a2cb0 0x7fc6bc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.952+0000 7fc6d08ee700 1 --2- 172.21.15.98:0/1128196101 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc6bc0a2cb0 0x7fc6bc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.952+0000 7fc6c9ffb700 1 -- 172.21.15.98:0/1128196101 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc6b80ceec0 con 0x7fc6cc1016f0 2024-05-09T08:48:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.952+0000 7fc6d08ee700 1 --2- 172.21.15.98:0/1128196101 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc6bc0a2cb0 0x7fc6bc0a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0000c00 tx=0x7fc6c0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:49.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:49.956+0000 7fc6c9ffb700 1 -- 172.21.15.98:0/1128196101 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc6b809bb90 con 0x7fc6cc1016f0 2024-05-09T08:48:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:50.155+0000 7fc6d2b52700 1 -- 172.21.15.98:0/1128196101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6cc02db80 con 0x7fc6cc1016f0 2024-05-09T08:48:50.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:50.156+0000 7fc6c9ffb700 1 -- 172.21.15.98:0/1128196101 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24150 (secure 0 0 0) 0x7fc6b809b490 con 0x7fc6cc1016f0 2024-05-09T08:48:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:50.156 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":90,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:48:22.719729+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:48:50.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:50.159+0000 7fc6d2b52700 1 -- 172.21.15.98:0/1128196101 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc6bc0a2cb0 msgr2=0x7fc6bc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:50.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:50.159+0000 7fc6d2b52700 1 --2- 172.21.15.98:0/1128196101 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc6bc0a2cb0 0x7fc6bc0a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0000c00 tx=0x7fc6c0005fb0 comp rx=0 tx=0).stop 2024-05-09T08:48:50.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:50.159+0000 7fc6d2b52700 1 -- 172.21.15.98:0/1128196101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6cc1016f0 msgr2=0x7fc6cc19ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:50.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:50.159+0000 7fc6d2b52700 1 --2- 172.21.15.98:0/1128196101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6cc1016f0 0x7fc6cc19ec50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800ddc0 tx=0x7fc6b800a530 comp rx=0 tx=0).stop 2024-05-09T08:48:50.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:50.159+0000 7fc6d2b52700 1 -- 172.21.15.98:0/1128196101 shutdown_connections 2024-05-09T08:48:50.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:50.159+0000 7fc6d2b52700 1 --2- 172.21.15.98:0/1128196101 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc6bc0a2cb0 0x7fc6bc0a5170 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:50.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:50.159+0000 7fc6d2b52700 1 --2- 172.21.15.98:0/1128196101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6cc1069c0 0x7fc6cc198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:50.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:50.159+0000 7fc6d2b52700 1 --2- 172.21.15.98:0/1128196101 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6cc100d30 0x7fc6cc19e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:50.159+0000 7fc6d2b52700 1 --2- 172.21.15.98:0/1128196101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6cc1016f0 0x7fc6cc19ec50 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:50.159+0000 7fc6d2b52700 1 -- 172.21.15.98:0/1128196101 >> 172.21.15.98:0/1128196101 conn(0x7fc6cc077420 msgr2=0x7fc6cc103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:50.159+0000 7fc6d2b52700 1 -- 172.21.15.98:0/1128196101 shutdown_connections 2024-05-09T08:48:50.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:50.159+0000 7fc6d2b52700 1 -- 172.21.15.98:0/1128196101 wait complete. 2024-05-09T08:48:50.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:50 smithi098 ceph-mon[30092]: pgmap v306: 129 pgs: 129 active+clean; 637 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 KiB/s wr, 26 op/s 2024-05-09T08:48:50.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:50 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1128196101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:50 smithi142 ceph-mon[34625]: pgmap v306: 129 pgs: 129 active+clean; 637 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 KiB/s wr, 26 op/s 2024-05-09T08:48:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:50 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1128196101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:50.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:50 smithi105 ceph-mon[34498]: pgmap v306: 129 pgs: 129 active+clean; 637 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 KiB/s wr, 26 op/s 2024-05-09T08:48:50.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:50 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1128196101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:50.986 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-09T08:48:50.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-09T08:48:51.233 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:52.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:52 smithi098 ceph-mon[30092]: pgmap v307: 129 pgs: 129 active+clean; 637 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 KiB/s wr, 22 op/s 2024-05-09T08:48:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:52 smithi142 ceph-mon[34625]: pgmap v307: 129 pgs: 129 active+clean; 637 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 KiB/s wr, 22 op/s 2024-05-09T08:48:52.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:52 smithi105 ceph-mon[34498]: pgmap v307: 129 pgs: 129 active+clean; 637 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 KiB/s wr, 22 op/s 2024-05-09T08:48:52.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.985+0000 7f0307a24700 1 -- 172.21.15.98:0/1854559733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f030006a9d0 msgr2=0x7f030010f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:52.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.985+0000 7f0307a24700 1 --2- 172.21.15.98:0/1854559733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f030006a9d0 0x7f030010f1e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f02f800b0a0 tx=0x7f02f800b3b0 comp rx=0 tx=0).stop 2024-05-09T08:48:52.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.985+0000 7f0307a24700 1 -- 172.21.15.98:0/1854559733 shutdown_connections 2024-05-09T08:48:52.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.985+0000 7f0307a24700 1 --2- 172.21.15.98:0/1854559733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03001083a0 0x7f0300108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:52.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.985+0000 7f0307a24700 1 --2- 172.21.15.98:0/1854559733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f030006a9d0 0x7f030010f1e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.985+0000 7f0307a24700 1 --2- 172.21.15.98:0/1854559733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f030006a030 0x7f030006a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.985+0000 7f0307a24700 1 -- 172.21.15.98:0/1854559733 >> 172.21.15.98:0/1854559733 conn(0x7f03000781d0 msgr2=0x7f03000785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.985+0000 7f0307a24700 1 -- 172.21.15.98:0/1854559733 shutdown_connections 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.985+0000 7f0307a24700 1 -- 172.21.15.98:0/1854559733 wait complete. 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0307a24700 1 Processor -- start 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0307a24700 1 -- start start 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0307a24700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f030006a030 0x7f03001012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0307a24700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f030006a9d0 0x7f0300101830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0307a24700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03001083a0 0x7f0300101d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0307a24700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03001125f0 con 0x7f03001083a0 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0307a24700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0300112470 con 0x7f030006a030 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0307a24700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0300112770 con 0x7f030006a9d0 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0305fc1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03001083a0 0x7f0300101d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0305fc1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03001083a0 0x7f0300101d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52888/0 (socket says 172.21.15.98:52888) 2024-05-09T08:48:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0305fc1700 1 -- 172.21.15.98:0/1797206527 learned_addr learned my addr 172.21.15.98:0/1797206527 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:48:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0304fbf700 1 --2- 172.21.15.98:0/1797206527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f030006a9d0 0x7f0300101830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f03057c0700 1 --2- 172.21.15.98:0/1797206527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f030006a030 0x7f03001012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0305fc1700 1 -- 172.21.15.98:0/1797206527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f030006a9d0 msgr2=0x7f0300101830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0305fc1700 1 --2- 172.21.15.98:0/1797206527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f030006a9d0 0x7f0300101830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0305fc1700 1 -- 172.21.15.98:0/1797206527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f030006a030 msgr2=0x7f03001012f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0305fc1700 1 --2- 172.21.15.98:0/1797206527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f030006a030 0x7f03001012f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.986+0000 7f0305fc1700 1 -- 172.21.15.98:0/1797206527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03001a6f30 con 0x7f03001083a0 2024-05-09T08:48:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.987+0000 7f0305fc1700 1 --2- 172.21.15.98:0/1797206527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03001083a0 0x7f0300101d70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f02f80062a0 tx=0x7f02f800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.987+0000 7f02f67fc700 1 -- 172.21.15.98:0/1797206527 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f8021070 con 0x7f03001083a0 2024-05-09T08:48:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.987+0000 7f02f67fc700 1 -- 172.21.15.98:0/1797206527 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f8007d00 con 0x7f03001083a0 2024-05-09T08:48:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.987+0000 7f0307a24700 1 -- 172.21.15.98:0/1797206527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03001a71c0 con 0x7f03001083a0 2024-05-09T08:48:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.987+0000 7f02f67fc700 1 -- 172.21.15.98:0/1797206527 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f8005460 con 0x7f03001083a0 2024-05-09T08:48:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.987+0000 7f0307a24700 1 -- 172.21.15.98:0/1797206527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03001a7620 con 0x7f03001083a0 2024-05-09T08:48:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.988+0000 7f0307a24700 1 -- 172.21.15.98:0/1797206527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0300069010 con 0x7f03001083a0 2024-05-09T08:48:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.989+0000 7f02f67fc700 1 -- 172.21.15.98:0/1797206527 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f02f8024070 con 0x7f03001083a0 2024-05-09T08:48:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.989+0000 7f02f67fc700 1 --2- 172.21.15.98:0/1797206527 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f02ec0a2cb0 0x7f02ec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.989+0000 7f03057c0700 1 --2- 172.21.15.98:0/1797206527 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f02ec0a2cb0 0x7f02ec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:52.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.990+0000 7f02f67fc700 1 -- 172.21.15.98:0/1797206527 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f02f80d0640 con 0x7f03001083a0 2024-05-09T08:48:52.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.990+0000 7f03057c0700 1 --2- 172.21.15.98:0/1797206527 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f02ec0a2cb0 0x7f02ec0a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f02fc005fd0 tx=0x7f02fc00af20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:52.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:52.994+0000 7f02f67fc700 1 -- 172.21.15.98:0/1797206527 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f02f809d290 con 0x7f03001083a0 2024-05-09T08:48:53.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:53.190+0000 7f0307a24700 1 -- 172.21.15.98:0/1797206527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f03001a7880 con 0x7f03001083a0 2024-05-09T08:48:53.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:53.192+0000 7f02f67fc700 1 -- 172.21.15.98:0/1797206527 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24150 (secure 0 0 0) 0x7f02f809cb90 con 0x7f03001083a0 2024-05-09T08:48:53.191 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:48:53.192 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":90,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:48:22.719729+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:48:53.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:53.194+0000 7f0307a24700 1 -- 172.21.15.98:0/1797206527 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f02ec0a2cb0 msgr2=0x7f02ec0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:53.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:53.194+0000 7f0307a24700 1 --2- 172.21.15.98:0/1797206527 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f02ec0a2cb0 0x7f02ec0a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f02fc005fd0 tx=0x7f02fc00af20 comp rx=0 tx=0).stop 2024-05-09T08:48:53.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:53.194+0000 7f0307a24700 1 -- 172.21.15.98:0/1797206527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03001083a0 msgr2=0x7f0300101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:53.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:53.194+0000 7f0307a24700 1 --2- 172.21.15.98:0/1797206527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03001083a0 0x7f0300101d70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f02f80062a0 tx=0x7f02f800be60 comp rx=0 tx=0).stop 2024-05-09T08:48:53.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:53.195+0000 7f0307a24700 1 -- 172.21.15.98:0/1797206527 shutdown_connections 2024-05-09T08:48:53.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:53.195+0000 7f0307a24700 1 --2- 172.21.15.98:0/1797206527 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f02ec0a2cb0 0x7f02ec0a5170 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:53.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:53.195+0000 7f0307a24700 1 --2- 172.21.15.98:0/1797206527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f030006a030 0x7f03001012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:53.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:53.195+0000 7f0307a24700 1 --2- 172.21.15.98:0/1797206527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03001083a0 0x7f0300101d70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:53.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:53.195+0000 7f0307a24700 1 --2- 172.21.15.98:0/1797206527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f030006a9d0 0x7f0300101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:53.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:53.195+0000 7f0307a24700 1 -- 172.21.15.98:0/1797206527 >> 172.21.15.98:0/1797206527 conn(0x7f03000781d0 msgr2=0x7f03000ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:53.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:53.195+0000 7f0307a24700 1 -- 172.21.15.98:0/1797206527 shutdown_connections 2024-05-09T08:48:53.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:48:53.195+0000 7f0307a24700 1 -- 172.21.15.98:0/1797206527 wait complete. 2024-05-09T08:48:53.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:53 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1797206527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:53.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:53 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1797206527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:53.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:53 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1797206527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:54.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-09T08:48:54.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-09T08:48:54.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-09T08:48:54.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-09T08:48:54.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-09T08:48:54.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-09T08:48:54.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-09T08:48:54.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-09T08:48:54.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-09T08:48:54.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-09T08:48:54.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-09T08:48:54.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-09T08:48:54.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:54 smithi098 ceph-mon[30092]: pgmap v308: 129 pgs: 129 active+clean; 637 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-05-09T08:48:54.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:54 smithi142 ceph-mon[34625]: pgmap v308: 129 pgs: 129 active+clean; 637 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-05-09T08:48:54.725 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:54.731 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:54.731 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:54.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:54 smithi105 ceph-mon[34498]: pgmap v308: 129 pgs: 129 active+clean; 637 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-05-09T08:48:56.530 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:56 smithi098 ceph-mon[30092]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.3 KiB/s wr, 14 op/s 2024-05-09T08:48:56.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:56 smithi142 ceph-mon[34625]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.3 KiB/s wr, 14 op/s 2024-05-09T08:48:56.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:56 smithi105 ceph-mon[34498]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.3 KiB/s wr, 14 op/s 2024-05-09T08:48:56.871 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:48:58 smithi098 ceph-mon[30092]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-05-09T08:48:58.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:48:58 smithi142 ceph-mon[34625]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-05-09T08:48:58.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:48:58 smithi105 ceph-mon[34498]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-05-09T08:49:00.462 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:00 smithi098 ceph-mon[30092]: pgmap v311: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-05-09T08:49:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:00 smithi142 ceph-mon[34625]: pgmap v311: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-05-09T08:49:00.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:00 smithi105 ceph-mon[34498]: pgmap v311: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-05-09T08:49:02.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:02 smithi098 ceph-mon[30092]: pgmap v312: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-05-09T08:49:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:02 smithi142 ceph-mon[34625]: pgmap v312: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-05-09T08:49:02.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:02 smithi105 ceph-mon[34498]: pgmap v312: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-05-09T08:49:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:04.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.100+0000 7fd1da223700 1 -- 172.21.15.98:0/1652799726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d406b260 msgr2=0x7fd1d410f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:04.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.100+0000 7fd1da223700 1 --2- 172.21.15.98:0/1652799726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d406b260 0x7fd1d410f1d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00b0a0 tx=0x7fd1cc00b3b0 comp rx=0 tx=0).stop 2024-05-09T08:49:04.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.101+0000 7fd1da223700 1 -- 172.21.15.98:0/1652799726 shutdown_connections 2024-05-09T08:49:04.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.101+0000 7fd1da223700 1 --2- 172.21.15.98:0/1652799726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d406b260 0x7fd1d410f1d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:04.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.101+0000 7fd1da223700 1 --2- 172.21.15.98:0/1652799726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1d4108390 0x7fd1d4108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:04.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.101+0000 7fd1da223700 1 --2- 172.21.15.98:0/1652799726 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1d406a8c0 0x7fd1d406ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:04.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.101+0000 7fd1da223700 1 -- 172.21.15.98:0/1652799726 >> 172.21.15.98:0/1652799726 conn(0x7fd1d4078190 msgr2=0x7fd1d40785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:04.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.101+0000 7fd1da223700 1 -- 172.21.15.98:0/1652799726 shutdown_connections 2024-05-09T08:49:04.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.101+0000 7fd1da223700 1 -- 172.21.15.98:0/1652799726 wait complete. 2024-05-09T08:49:04.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.102+0000 7fd1da223700 1 Processor -- start 2024-05-09T08:49:04.102 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:04.103 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:04.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.102+0000 7fd1da223700 1 -- start start 2024-05-09T08:49:04.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.102+0000 7fd1da223700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1d406a8c0 0x7fd1d419e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:04.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.102+0000 7fd1da223700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d406b260 0x7fd1d419ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:04.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.102+0000 7fd1da223700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1d4108390 0x7fd1d4198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:04.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.102+0000 7fd1da223700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d41125e0 con 0x7fd1d406b260 2024-05-09T08:49:04.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.102+0000 7fd1da223700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d4112460 con 0x7fd1d406a8c0 2024-05-09T08:49:04.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.102+0000 7fd1da223700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d4112760 con 0x7fd1d4108390 2024-05-09T08:49:04.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.102+0000 7fd1d2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d406b260 0x7fd1d419ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:04.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.102+0000 7fd1d2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d406b260 0x7fd1d419ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37412/0 (socket says 172.21.15.98:37412) 2024-05-09T08:49:04.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.102+0000 7fd1d2ffd700 1 -- 172.21.15.98:0/4197172415 learned_addr learned my addr 172.21.15.98:0/4197172415 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:04.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.102+0000 7fd1d3fff700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1d4108390 0x7fd1d4198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:04.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.102+0000 7fd1d37fe700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1d406a8c0 0x7fd1d419e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.103+0000 7fd1d2ffd700 1 -- 172.21.15.98:0/4197172415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1d4108390 msgr2=0x7fd1d4198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.103+0000 7fd1d2ffd700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1d4108390 0x7fd1d4198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.103+0000 7fd1d2ffd700 1 -- 172.21.15.98:0/4197172415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1d406a8c0 msgr2=0x7fd1d419e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.103+0000 7fd1d2ffd700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1d406a8c0 0x7fd1d419e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.103+0000 7fd1d2ffd700 1 -- 172.21.15.98:0/4197172415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d41990a0 con 0x7fd1d406b260 2024-05-09T08:49:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.103+0000 7fd1d3fff700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1d4108390 0x7fd1d4198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:49:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.103+0000 7fd1d2ffd700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d406b260 0x7fd1d419ecc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800c920 tx=0x7fd1c800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.103+0000 7fd1d0ff9700 1 -- 172.21.15.98:0/4197172415 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8007b00 con 0x7fd1d406b260 2024-05-09T08:49:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.103+0000 7fd1d0ff9700 1 -- 172.21.15.98:0/4197172415 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1c8004d10 con 0x7fd1d406b260 2024-05-09T08:49:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.103+0000 7fd1da223700 1 -- 172.21.15.98:0/4197172415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d4199390 con 0x7fd1d406b260 2024-05-09T08:49:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.103+0000 7fd1d0ff9700 1 -- 172.21.15.98:0/4197172415 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c800f450 con 0x7fd1d406b260 2024-05-09T08:49:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.103+0000 7fd1da223700 1 -- 172.21.15.98:0/4197172415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d41a5650 con 0x7fd1d406b260 2024-05-09T08:49:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.104+0000 7fd1da223700 1 -- 172.21.15.98:0/4197172415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd1d40629f0 con 0x7fd1d406b260 2024-05-09T08:49:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.105+0000 7fd1d0ff9700 1 -- 172.21.15.98:0/4197172415 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd1c8004750 con 0x7fd1d406b260 2024-05-09T08:49:04.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.106+0000 7fd1d0ff9700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd1bc0a2cb0 0x7fd1bc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:04.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.106+0000 7fd1d37fe700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd1bc0a2cb0 0x7fd1bc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:04.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.106+0000 7fd1d0ff9700 1 -- 172.21.15.98:0/4197172415 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd1c80cf610 con 0x7fd1d406b260 2024-05-09T08:49:04.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.106+0000 7fd1d0ff9700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287] conn(0x7fd1bc0ad960 0x7fd1bc0afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:04.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.106+0000 7fd1d0ff9700 1 -- 172.21.15.98:0/4197172415 --> [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd1c8023e50 con 0x7fd1bc0ad960 2024-05-09T08:49:04.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.106+0000 7fd1d0ff9700 1 -- 172.21.15.98:0/4197172415 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fd1c80cfa60 con 0x7fd1d406b260 2024-05-09T08:49:04.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.107+0000 7fd1d3fff700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287] conn(0x7fd1bc0ad960 0x7fd1bc0afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:04.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.107+0000 7fd1d37fe700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd1bc0a2cb0 0x7fd1bc0a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4199a80 tx=0x7fd1c400b4d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:04.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.107+0000 7fd1d3fff700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287] conn(0x7fd1bc0ad960 0x7fd1bc0afda0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:04.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.108+0000 7fd1d0ff9700 1 -- 172.21.15.98:0/4197172415 <== osd.3 v2:172.21.15.98:6810/2745237287 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd1c8023e50 con 0x7fd1bc0ad960 2024-05-09T08:49:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.131+0000 7fd1da223700 1 -- 172.21.15.98:0/4197172415 --> [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd1d4062e70 con 0x7fd1bc0ad960 2024-05-09T08:49:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.132+0000 7fd1d0ff9700 1 -- 172.21.15.98:0/4197172415 <== osd.3 v2:172.21.15.98:6810/2745237287 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd1d4062e70 con 0x7fd1bc0ad960 2024-05-09T08:49:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.132+0000 7fd1da223700 1 -- 172.21.15.98:0/4197172415 >> [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287] conn(0x7fd1bc0ad960 msgr2=0x7fd1bc0afda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.132+0000 7fd1da223700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287] conn(0x7fd1bc0ad960 0x7fd1bc0afda0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.132+0000 7fd1da223700 1 -- 172.21.15.98:0/4197172415 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd1bc0a2cb0 msgr2=0x7fd1bc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.132+0000 7fd1da223700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd1bc0a2cb0 0x7fd1bc0a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4199a80 tx=0x7fd1c400b4d0 comp rx=0 tx=0).stop 2024-05-09T08:49:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.132+0000 7fd1da223700 1 -- 172.21.15.98:0/4197172415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d406b260 msgr2=0x7fd1d419ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.132+0000 7fd1da223700 1 --2- 172.21.15.98:0/4197172415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d406b260 0x7fd1d419ecc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800c920 tx=0x7fd1c800cce0 comp rx=0 tx=0).stop 2024-05-09T08:49:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.132+0000 7fd1d3fff700 1 -- 172.21.15.98:0/4197172415 reap_dead start 2024-05-09T08:49:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.132+0000 7fd1da223700 1 -- 172.21.15.98:0/4197172415 shutdown_connections 2024-05-09T08:49:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.132+0000 7fd1da223700 1 -- 172.21.15.98:0/4197172415 >> 172.21.15.98:0/4197172415 conn(0x7fd1d4078190 msgr2=0x7fd1d40ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.132+0000 7fd1da223700 1 -- 172.21.15.98:0/4197172415 shutdown_connections 2024-05-09T08:49:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:04.132+0000 7fd1da223700 1 -- 172.21.15.98:0/4197172415 wait complete. 2024-05-09T08:49:04.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:04 smithi098 ceph-mon[30092]: pgmap v313: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-05-09T08:49:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:04 smithi142 ceph-mon[34625]: pgmap v313: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-05-09T08:49:04.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:04 smithi105 ceph-mon[34498]: pgmap v313: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-05-09T08:49:05.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.286+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1432264049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34c01072c0 msgr2=0x7f34c01076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:05.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.286+0000 7f34c5c1a700 1 --2- 172.21.15.98:0/1432264049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34c01072c0 0x7f34c01076a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f34b000b0a0 tx=0x7f34b000b3b0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.286+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1432264049 shutdown_connections 2024-05-09T08:49:05.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.286+0000 7f34c5c1a700 1 --2- 172.21.15.98:0/1432264049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34c0101c10 0x7f34c010e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.286+0000 7f34c5c1a700 1 --2- 172.21.15.98:0/1432264049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34c01072c0 0x7f34c01076a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.286+0000 7f34c5c1a700 1 --2- 172.21.15.98:0/1432264049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34c0101250 0x7f34c01016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.286+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1432264049 >> 172.21.15.98:0/1432264049 conn(0x7f34c0078260 msgr2=0x7f34c0078670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:05.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.286+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1432264049 shutdown_connections 2024-05-09T08:49:05.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.286+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1432264049 wait complete. 2024-05-09T08:49:05.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.287+0000 7f34c5c1a700 1 Processor -- start 2024-05-09T08:49:05.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.287+0000 7f34c5c1a700 1 -- start start 2024-05-09T08:49:05.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.287+0000 7f34c5c1a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34c0101250 0x7f34c0199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:05.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.287+0000 7f34c5c1a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34c0101c10 0x7f34c0199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.287+0000 7f34c5c1a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34c01072c0 0x7f34c01a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.287+0000 7f34c5c1a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01115b0 con 0x7f34c0101c10 2024-05-09T08:49:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.287+0000 7f34c5c1a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f34c0111430 con 0x7f34c01072c0 2024-05-09T08:49:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.287+0000 7f34c5c1a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f34c0111730 con 0x7f34c0101250 2024-05-09T08:49:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.287+0000 7f34beffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34c0101c10 0x7f34c0199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.287+0000 7f34beffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34c0101c10 0x7f34c0199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37432/0 (socket says 172.21.15.98:37432) 2024-05-09T08:49:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.287+0000 7f34beffd700 1 -- 172.21.15.98:0/1362649995 learned_addr learned my addr 172.21.15.98:0/1362649995 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.287+0000 7f34bf7fe700 1 --2- 172.21.15.98:0/1362649995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34c0101250 0x7f34c0199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.288+0000 7f34bffff700 1 --2- 172.21.15.98:0/1362649995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34c01072c0 0x7f34c01a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.288+0000 7f34beffd700 1 -- 172.21.15.98:0/1362649995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34c0101250 msgr2=0x7f34c0199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.288+0000 7f34beffd700 1 --2- 172.21.15.98:0/1362649995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34c0101250 0x7f34c0199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.288+0000 7f34beffd700 1 -- 172.21.15.98:0/1362649995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34c01072c0 msgr2=0x7f34c01a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.288+0000 7f34beffd700 1 --2- 172.21.15.98:0/1362649995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34c01072c0 0x7f34c01a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.288+0000 7f34beffd700 1 -- 172.21.15.98:0/1362649995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c01a4ed0 con 0x7f34c0101c10 2024-05-09T08:49:05.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.288+0000 7f34beffd700 1 --2- 172.21.15.98:0/1362649995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34c0101c10 0x7f34c0199e50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f34a800c9b0 tx=0x7f34a800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:05.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.288+0000 7f34bcff9700 1 -- 172.21.15.98:0/1362649995 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8007b00 con 0x7f34c0101c10 2024-05-09T08:49:05.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.288+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1362649995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c01a5100 con 0x7f34c0101c10 2024-05-09T08:49:05.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.288+0000 7f34bcff9700 1 -- 172.21.15.98:0/1362649995 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34a800ce50 con 0x7f34c0101c10 2024-05-09T08:49:05.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.288+0000 7f34bcff9700 1 -- 172.21.15.98:0/1362649995 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8018660 con 0x7f34c0101c10 2024-05-09T08:49:05.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.288+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1362649995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c01a5640 con 0x7f34c0101c10 2024-05-09T08:49:05.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.289+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1362649995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f34c0062a30 con 0x7f34c0101c10 2024-05-09T08:49:05.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.290+0000 7f34bcff9700 1 -- 172.21.15.98:0/1362649995 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f34a800f450 con 0x7f34c0101c10 2024-05-09T08:49:05.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.291+0000 7f34bcff9700 1 --2- 172.21.15.98:0/1362649995 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f34ac0a2cb0 0x7f34ac0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:05.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.291+0000 7f34bf7fe700 1 --2- 172.21.15.98:0/1362649995 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f34ac0a2cb0 0x7f34ac0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:05.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.291+0000 7f34bcff9700 1 -- 172.21.15.98:0/1362649995 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f34a80cf720 con 0x7f34c0101c10 2024-05-09T08:49:05.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.291+0000 7f34bcff9700 1 --2- 172.21.15.98:0/1362649995 >> [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836] conn(0x7f34ac0ad970 0x7f34ac0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:05.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.291+0000 7f34bcff9700 1 -- 172.21.15.98:0/1362649995 --> [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34ac0b0480 con 0x7f34ac0ad970 2024-05-09T08:49:05.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.291+0000 7f34bcff9700 1 -- 172.21.15.98:0/1362649995 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f34a80cfb70 con 0x7f34c0101c10 2024-05-09T08:49:05.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.292+0000 7f34bffff700 1 --2- 172.21.15.98:0/1362649995 >> [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836] conn(0x7f34ac0ad970 0x7f34ac0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:05.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.292+0000 7f34bf7fe700 1 --2- 172.21.15.98:0/1362649995 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f34ac0a2cb0 0x7f34ac0a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f34b000b070 tx=0x7f34b000b7b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:05.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.292+0000 7f34bffff700 1 --2- 172.21.15.98:0/1362649995 >> [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836] conn(0x7f34ac0ad970 0x7f34ac0afdb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:05.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.293+0000 7f34bcff9700 1 -- 172.21.15.98:0/1362649995 <== osd.6 v2:172.21.15.98:6818/1723462836 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f34ac0b0480 con 0x7f34ac0ad970 2024-05-09T08:49:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.309+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1362649995 --> [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f34c0062e30 con 0x7f34ac0ad970 2024-05-09T08:49:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.309+0000 7f34bcff9700 1 -- 172.21.15.98:0/1362649995 <== osd.6 v2:172.21.15.98:6818/1723462836 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f34c0062e30 con 0x7f34ac0ad970 2024-05-09T08:49:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.309+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1362649995 >> [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836] conn(0x7f34ac0ad970 msgr2=0x7f34ac0afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.309+0000 7f34c5c1a700 1 --2- 172.21.15.98:0/1362649995 >> [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836] conn(0x7f34ac0ad970 0x7f34ac0afdb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.309+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1362649995 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f34ac0a2cb0 msgr2=0x7f34ac0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.309+0000 7f34c5c1a700 1 --2- 172.21.15.98:0/1362649995 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f34ac0a2cb0 0x7f34ac0a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f34b000b070 tx=0x7f34b000b7b0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.309+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1362649995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34c0101c10 msgr2=0x7f34c0199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.309+0000 7f34c5c1a700 1 --2- 172.21.15.98:0/1362649995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34c0101c10 0x7f34c0199e50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f34a800c9b0 tx=0x7f34a800ccc0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.309+0000 7f34bffff700 1 -- 172.21.15.98:0/1362649995 reap_dead start 2024-05-09T08:49:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.309+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1362649995 shutdown_connections 2024-05-09T08:49:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.309+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1362649995 >> 172.21.15.98:0/1362649995 conn(0x7f34c0078260 msgr2=0x7f34c00fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.310+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1362649995 shutdown_connections 2024-05-09T08:49:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.310+0000 7f34c5c1a700 1 -- 172.21.15.98:0/1362649995 wait complete. 2024-05-09T08:49:05.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.917+0000 7f533e405700 1 -- 172.21.15.98:0/3333236877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53381083a0 msgr2=0x7f5338108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:05.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.917+0000 7f533e405700 1 --2- 172.21.15.98:0/3333236877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53381083a0 0x7f5338108780 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f53280099c0 tx=0x7f5328009cd0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.917+0000 7f533e405700 1 -- 172.21.15.98:0/3333236877 shutdown_connections 2024-05-09T08:49:05.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.917+0000 7f533e405700 1 --2- 172.21.15.98:0/3333236877 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f533806b3f0 0x7f533810f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.917+0000 7f533e405700 1 --2- 172.21.15.98:0/3333236877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53381083a0 0x7f5338108780 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.917+0000 7f533e405700 1 --2- 172.21.15.98:0/3333236877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f533806aa50 0x7f533806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.917+0000 7f533e405700 1 -- 172.21.15.98:0/3333236877 >> 172.21.15.98:0/3333236877 conn(0x7f53380781c0 msgr2=0x7f53380785d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:05.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.917+0000 7f533e405700 1 -- 172.21.15.98:0/3333236877 shutdown_connections 2024-05-09T08:49:05.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.917+0000 7f533e405700 1 -- 172.21.15.98:0/3333236877 wait complete. 2024-05-09T08:49:05.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f533e405700 1 Processor -- start 2024-05-09T08:49:05.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f533e405700 1 -- start start 2024-05-09T08:49:05.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f533e405700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f533806aa50 0x7f5338198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:05.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f533e405700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f533806b3f0 0x7f5338199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:05.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f533e405700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53381083a0 0x7f533819d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:05.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f533e405700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5338112600 con 0x7f533806aa50 2024-05-09T08:49:05.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f533e405700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5338112480 con 0x7f533806b3f0 2024-05-09T08:49:05.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f533e405700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5338112780 con 0x7f53381083a0 2024-05-09T08:49:05.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f5337fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f533806aa50 0x7f5338198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:05.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f5337fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f533806aa50 0x7f5338198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48554/0 (socket says 172.21.15.98:48554) 2024-05-09T08:49:05.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f5337fff700 1 -- 172.21.15.98:0/2332796024 learned_addr learned my addr 172.21.15.98:0/2332796024 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:05.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f532ffff700 1 --2- 172.21.15.98:0/2332796024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f533806b3f0 0x7f5338199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:05.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f533c9a2700 1 --2- 172.21.15.98:0/2332796024 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53381083a0 0x7f533819d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:05.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f5337fff700 1 -- 172.21.15.98:0/2332796024 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53381083a0 msgr2=0x7f533819d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:05.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f5337fff700 1 --2- 172.21.15.98:0/2332796024 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53381083a0 0x7f533819d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f5337fff700 1 -- 172.21.15.98:0/2332796024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f533806b3f0 msgr2=0x7f5338199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:05.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.918+0000 7f5337fff700 1 --2- 172.21.15.98:0/2332796024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f533806b3f0 0x7f5338199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.919+0000 7f5337fff700 1 -- 172.21.15.98:0/2332796024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f533819dc30 con 0x7f533806aa50 2024-05-09T08:49:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.919+0000 7f5337fff700 1 --2- 172.21.15.98:0/2332796024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f533806aa50 0x7f5338198c30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f53280059d0 tx=0x7f53280038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.919+0000 7f5335ffb700 1 -- 172.21.15.98:0/2332796024 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5328027070 con 0x7f533806aa50 2024-05-09T08:49:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.919+0000 7f533e405700 1 -- 172.21.15.98:0/2332796024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f533819dec0 con 0x7f533806aa50 2024-05-09T08:49:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.919+0000 7f5335ffb700 1 -- 172.21.15.98:0/2332796024 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5328003e40 con 0x7f533806aa50 2024-05-09T08:49:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.919+0000 7f5335ffb700 1 -- 172.21.15.98:0/2332796024 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5328022d00 con 0x7f533806aa50 2024-05-09T08:49:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.919+0000 7f533e405700 1 -- 172.21.15.98:0/2332796024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53381a57e0 con 0x7f533806aa50 2024-05-09T08:49:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.920+0000 7f533e405700 1 -- 172.21.15.98:0/2332796024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5338062a30 con 0x7f533806aa50 2024-05-09T08:49:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.921+0000 7f5335ffb700 1 -- 172.21.15.98:0/2332796024 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f532802a070 con 0x7f533806aa50 2024-05-09T08:49:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.921+0000 7f5335ffb700 1 --2- 172.21.15.98:0/2332796024 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f53240a2cb0 0x7f53240a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.922+0000 7f532ffff700 1 --2- 172.21.15.98:0/2332796024 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f53240a2cb0 0x7f53240a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.922+0000 7f5335ffb700 1 -- 172.21.15.98:0/2332796024 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f53280d0d40 con 0x7f533806aa50 2024-05-09T08:49:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.922+0000 7f5335ffb700 1 --2- 172.21.15.98:0/2332796024 >> [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474] conn(0x7f53240ad970 0x7f53240afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:05.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.922+0000 7f5335ffb700 1 -- 172.21.15.98:0/2332796024 --> [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53280095d0 con 0x7f53240ad970 2024-05-09T08:49:05.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.922+0000 7f5335ffb700 1 -- 172.21.15.98:0/2332796024 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f53280d1190 con 0x7f533806aa50 2024-05-09T08:49:05.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.922+0000 7f532ffff700 1 --2- 172.21.15.98:0/2332796024 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f53240a2cb0 0x7f53240a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5338078720 tx=0x7f532000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:05.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.922+0000 7f533c9a2700 1 --2- 172.21.15.98:0/2332796024 >> [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474] conn(0x7f53240ad970 0x7f53240afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:05.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.923+0000 7f533c9a2700 1 --2- 172.21.15.98:0/2332796024 >> [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474] conn(0x7f53240ad970 0x7f53240afdb0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:05.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.923+0000 7f5335ffb700 1 -- 172.21.15.98:0/2332796024 <== osd.4 v2:172.21.15.105:6808/3797460474 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53280095d0 con 0x7f53240ad970 2024-05-09T08:49:05.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.939+0000 7f533e405700 1 -- 172.21.15.98:0/2332796024 --> [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5338062e30 con 0x7f53240ad970 2024-05-09T08:49:05.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.940+0000 7f5335ffb700 1 -- 172.21.15.98:0/2332796024 <== osd.4 v2:172.21.15.105:6808/3797460474 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5338062e30 con 0x7f53240ad970 2024-05-09T08:49:05.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.940+0000 7f533e405700 1 -- 172.21.15.98:0/2332796024 >> [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474] conn(0x7f53240ad970 msgr2=0x7f53240afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:05.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.940+0000 7f533e405700 1 --2- 172.21.15.98:0/2332796024 >> [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474] conn(0x7f53240ad970 0x7f53240afdb0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.940+0000 7f533e405700 1 -- 172.21.15.98:0/2332796024 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f53240a2cb0 msgr2=0x7f53240a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:05.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.940+0000 7f533e405700 1 --2- 172.21.15.98:0/2332796024 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f53240a2cb0 0x7f53240a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5338078720 tx=0x7f532000b410 comp rx=0 tx=0).stop 2024-05-09T08:49:05.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.940+0000 7f533e405700 1 -- 172.21.15.98:0/2332796024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f533806aa50 msgr2=0x7f5338198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:05.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.940+0000 7f533e405700 1 --2- 172.21.15.98:0/2332796024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f533806aa50 0x7f5338198c30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f53280059d0 tx=0x7f53280038b0 comp rx=0 tx=0).stop 2024-05-09T08:49:05.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.940+0000 7f533c9a2700 1 -- 172.21.15.98:0/2332796024 reap_dead start 2024-05-09T08:49:05.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.940+0000 7f533e405700 1 -- 172.21.15.98:0/2332796024 shutdown_connections 2024-05-09T08:49:05.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.940+0000 7f533e405700 1 -- 172.21.15.98:0/2332796024 >> 172.21.15.98:0/2332796024 conn(0x7f53380781c0 msgr2=0x7f53380fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:05.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.940+0000 7f533e405700 1 -- 172.21.15.98:0/2332796024 shutdown_connections 2024-05-09T08:49:05.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:05.940+0000 7f533e405700 1 -- 172.21.15.98:0/2332796024 wait complete. 2024-05-09T08:49:06.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:06 smithi098 ceph-mon[30092]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 682 B/s wr, 11 op/s 2024-05-09T08:49:06.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:06 smithi142 ceph-mon[34625]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 682 B/s wr, 11 op/s 2024-05-09T08:49:06.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:06 smithi105 ceph-mon[34498]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 682 B/s wr, 11 op/s 2024-05-09T08:49:08.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.317+0000 7f322de27700 1 -- 172.21.15.98:0/3000278451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32281083b0 msgr2=0x7f3228108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:08.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.317+0000 7f322de27700 1 --2- 172.21.15.98:0/3000278451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32281083b0 0x7f3228108790 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3218009990 tx=0x7f3218009ca0 comp rx=0 tx=0).stop 2024-05-09T08:49:08.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.317+0000 7f322de27700 1 -- 172.21.15.98:0/3000278451 shutdown_connections 2024-05-09T08:49:08.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.317+0000 7f322de27700 1 --2- 172.21.15.98:0/3000278451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32281012b0 0x7f322810f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:08.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.317+0000 7f322de27700 1 --2- 172.21.15.98:0/3000278451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32281083b0 0x7f3228108790 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:08.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.317+0000 7f322de27700 1 --2- 172.21.15.98:0/3000278451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3228100910 0x7f3228100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:08.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.317+0000 7f322de27700 1 -- 172.21.15.98:0/3000278451 >> 172.21.15.98:0/3000278451 conn(0x7f3228077890 msgr2=0x7f32280fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:08.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.317+0000 7f322de27700 1 -- 172.21.15.98:0/3000278451 shutdown_connections 2024-05-09T08:49:08.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.317+0000 7f322de27700 1 -- 172.21.15.98:0/3000278451 wait complete. 2024-05-09T08:49:08.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.318+0000 7f322de27700 1 Processor -- start 2024-05-09T08:49:08.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.318+0000 7f322de27700 1 -- start start 2024-05-09T08:49:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.318+0000 7f322de27700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3228100910 0x7f322819e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.318+0000 7f322de27700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32281012b0 0x7f322819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.318+0000 7f322de27700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32281083b0 0x7f3228198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.318+0000 7f322de27700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f32281126f0 con 0x7f32281012b0 2024-05-09T08:49:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.318+0000 7f322de27700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3228112570 con 0x7f32281083b0 2024-05-09T08:49:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.318+0000 7f322de27700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3228112870 con 0x7f3228100910 2024-05-09T08:49:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.318+0000 7f3226ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32281012b0 0x7f322819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.318+0000 7f3226ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32281012b0 0x7f322819ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48576/0 (socket says 172.21.15.98:48576) 2024-05-09T08:49:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.318+0000 7f3226ffd700 1 -- 172.21.15.98:0/282307275 learned_addr learned my addr 172.21.15.98:0/282307275 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.318+0000 7f3227fff700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32281083b0 0x7f3228198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.318+0000 7f32277fe700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3228100910 0x7f322819e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.319+0000 7f3226ffd700 1 -- 172.21.15.98:0/282307275 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3228100910 msgr2=0x7f322819e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.319+0000 7f3226ffd700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3228100910 0x7f322819e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.319+0000 7f3226ffd700 1 -- 172.21.15.98:0/282307275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32281083b0 msgr2=0x7f3228198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.319+0000 7f3226ffd700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32281083b0 0x7f3228198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.319+0000 7f3226ffd700 1 -- 172.21.15.98:0/282307275 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32281991f0 con 0x7f32281012b0 2024-05-09T08:49:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.319+0000 7f3227fff700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32281083b0 0x7f3228198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:49:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.319+0000 7f3226ffd700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32281012b0 0x7f322819ee10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f321c00b530 tx=0x7f321c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.319+0000 7f3224ff9700 1 -- 172.21.15.98:0/282307275 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c015020 con 0x7f32281012b0 2024-05-09T08:49:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.319+0000 7f3224ff9700 1 -- 172.21.15.98:0/282307275 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f321c011d10 con 0x7f32281012b0 2024-05-09T08:49:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.319+0000 7f322de27700 1 -- 172.21.15.98:0/282307275 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32281994e0 con 0x7f32281012b0 2024-05-09T08:49:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.319+0000 7f3224ff9700 1 -- 172.21.15.98:0/282307275 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c010e50 con 0x7f32281012b0 2024-05-09T08:49:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.319+0000 7f322de27700 1 -- 172.21.15.98:0/282307275 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32281a57a0 con 0x7f32281012b0 2024-05-09T08:49:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.320+0000 7f322de27700 1 -- 172.21.15.98:0/282307275 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3228062a70 con 0x7f32281012b0 2024-05-09T08:49:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.321+0000 7f3224ff9700 1 -- 172.21.15.98:0/282307275 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f321c018020 con 0x7f32281012b0 2024-05-09T08:49:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.321+0000 7f3224ff9700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32100a2e50 0x7f32100a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.322+0000 7f32277fe700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32100a2e50 0x7f32100a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.322+0000 7f3224ff9700 1 -- 172.21.15.98:0/282307275 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f321c0cf1e0 con 0x7f32281012b0 2024-05-09T08:49:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.322+0000 7f3224ff9700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414] conn(0x7f32100adb10 0x7f32100aff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.322+0000 7f3224ff9700 1 -- 172.21.15.98:0/282307275 --> [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f321c013ec0 con 0x7f32100adb10 2024-05-09T08:49:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.322+0000 7f3224ff9700 1 -- 172.21.15.98:0/282307275 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f321c0cf630 con 0x7f32281012b0 2024-05-09T08:49:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.322+0000 7f32277fe700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32100a2e50 0x7f32100a5310 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3218006010 tx=0x7f321800b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.322+0000 7f3227fff700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414] conn(0x7f32100adb10 0x7f32100aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.322+0000 7f3227fff700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414] conn(0x7f32100adb10 0x7f32100aff50 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.323+0000 7f3224ff9700 1 -- 172.21.15.98:0/282307275 <== osd.9 v2:172.21.15.98:6826/1742411414 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f321c013ec0 con 0x7f32100adb10 2024-05-09T08:49:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.339+0000 7f322de27700 1 -- 172.21.15.98:0/282307275 --> [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3228062ef0 con 0x7f32100adb10 2024-05-09T08:49:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.340+0000 7f3224ff9700 1 -- 172.21.15.98:0/282307275 <== osd.9 v2:172.21.15.98:6826/1742411414 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3228062ef0 con 0x7f32100adb10 2024-05-09T08:49:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.340+0000 7f322de27700 1 -- 172.21.15.98:0/282307275 >> [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414] conn(0x7f32100adb10 msgr2=0x7f32100aff50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.340+0000 7f322de27700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414] conn(0x7f32100adb10 0x7f32100aff50 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.340+0000 7f322de27700 1 -- 172.21.15.98:0/282307275 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32100a2e50 msgr2=0x7f32100a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.340+0000 7f322de27700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32100a2e50 0x7f32100a5310 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3218006010 tx=0x7f321800b540 comp rx=0 tx=0).stop 2024-05-09T08:49:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.340+0000 7f322de27700 1 -- 172.21.15.98:0/282307275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32281012b0 msgr2=0x7f322819ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.340+0000 7f322de27700 1 --2- 172.21.15.98:0/282307275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32281012b0 0x7f322819ee10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f321c00b530 tx=0x7f321c00b8f0 comp rx=0 tx=0).stop 2024-05-09T08:49:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.340+0000 7f3227fff700 1 -- 172.21.15.98:0/282307275 reap_dead start 2024-05-09T08:49:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.340+0000 7f322de27700 1 -- 172.21.15.98:0/282307275 shutdown_connections 2024-05-09T08:49:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.340+0000 7f322de27700 1 -- 172.21.15.98:0/282307275 >> 172.21.15.98:0/282307275 conn(0x7f3228077890 msgr2=0x7f32280ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.340+0000 7f322de27700 1 -- 172.21.15.98:0/282307275 shutdown_connections 2024-05-09T08:49:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:08.340+0000 7f322de27700 1 -- 172.21.15.98:0/282307275 wait complete. 2024-05-09T08:49:08.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:08 smithi098 ceph-mon[30092]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-05-09T08:49:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:08 smithi142 ceph-mon[34625]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-05-09T08:49:08.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:08 smithi105 ceph-mon[34498]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-05-09T08:49:09.041 INFO:teuthology.orchestra.run.smithi098.stdout:292057776162 2024-05-09T08:49:09.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-09T08:49:09.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.565+0000 7fafd2e70700 1 -- 172.21.15.98:0/3000984296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafcc101be0 msgr2=0x7fafcc10e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:09.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.565+0000 7fafd2e70700 1 --2- 172.21.15.98:0/3000984296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafcc101be0 0x7fafcc10e0d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009990 tx=0x7fafb4009ca0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.565+0000 7fafd2e70700 1 -- 172.21.15.98:0/3000984296 shutdown_connections 2024-05-09T08:49:09.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.565+0000 7fafd2e70700 1 --2- 172.21.15.98:0/3000984296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafcc101be0 0x7fafcc10e0d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.565+0000 7fafd2e70700 1 --2- 172.21.15.98:0/3000984296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafcc107290 0x7fafcc107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.565+0000 7fafd2e70700 1 --2- 172.21.15.98:0/3000984296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafcc101220 0x7fafcc1016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.565+0000 7fafd2e70700 1 -- 172.21.15.98:0/3000984296 >> 172.21.15.98:0/3000984296 conn(0x7fafcc0781d0 msgr2=0x7fafcc0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:09.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.565+0000 7fafd2e70700 1 -- 172.21.15.98:0/3000984296 shutdown_connections 2024-05-09T08:49:09.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.565+0000 7fafd2e70700 1 -- 172.21.15.98:0/3000984296 wait complete. 2024-05-09T08:49:09.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.565+0000 7fafd2e70700 1 Processor -- start 2024-05-09T08:49:09.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.565+0000 7fafd2e70700 1 -- start start 2024-05-09T08:49:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd2e70700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafcc101220 0x7fafcc199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd2e70700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafcc101be0 0x7fafcc199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd2e70700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafcc107290 0x7fafcc1a49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd2e70700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc111520 con 0x7fafcc107290 2024-05-09T08:49:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd2e70700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc1113a0 con 0x7fafcc101be0 2024-05-09T08:49:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd2e70700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc1116a0 con 0x7fafcc101220 2024-05-09T08:49:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd140d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafcc107290 0x7fafcc1a49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd140d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafcc107290 0x7fafcc1a49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48592/0 (socket says 172.21.15.98:48592) 2024-05-09T08:49:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd140d700 1 -- 172.21.15.98:0/1645304397 learned_addr learned my addr 172.21.15.98:0/1645304397 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafcbfff700 1 --2- 172.21.15.98:0/1645304397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafcc101be0 0x7fafcc199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd0c0c700 1 --2- 172.21.15.98:0/1645304397 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafcc101220 0x7fafcc199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd140d700 1 -- 172.21.15.98:0/1645304397 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafcc101220 msgr2=0x7fafcc199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd140d700 1 --2- 172.21.15.98:0/1645304397 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafcc101220 0x7fafcc199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd140d700 1 -- 172.21.15.98:0/1645304397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafcc101be0 msgr2=0x7fafcc199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd140d700 1 --2- 172.21.15.98:0/1645304397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafcc101be0 0x7fafcc199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd140d700 1 -- 172.21.15.98:0/1645304397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafcc1a4ee0 con 0x7fafcc107290 2024-05-09T08:49:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd140d700 1 --2- 172.21.15.98:0/1645304397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafcc107290 0x7fafcc1a49a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fafb40047b0 tx=0x7fafb4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafc9ffb700 1 -- 172.21.15.98:0/1645304397 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb4027070 con 0x7fafcc107290 2024-05-09T08:49:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafd2e70700 1 -- 172.21.15.98:0/1645304397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafcc1a50b0 con 0x7fafcc107290 2024-05-09T08:49:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafc9ffb700 1 -- 172.21.15.98:0/1645304397 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb4003e40 con 0x7fafcc107290 2024-05-09T08:49:09.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.566+0000 7fafc9ffb700 1 -- 172.21.15.98:0/1645304397 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb4022c50 con 0x7fafcc107290 2024-05-09T08:49:09.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.567+0000 7fafd2e70700 1 -- 172.21.15.98:0/1645304397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafcc1a54a0 con 0x7fafcc107290 2024-05-09T08:49:09.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.568+0000 7fafd2e70700 1 -- 172.21.15.98:0/1645304397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fafcc062a30 con 0x7fafcc107290 2024-05-09T08:49:09.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.569+0000 7fafc9ffb700 1 -- 172.21.15.98:0/1645304397 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fafb402a070 con 0x7fafcc107290 2024-05-09T08:49:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.569+0000 7fafc9ffb700 1 --2- 172.21.15.98:0/1645304397 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fafb80a2cb0 0x7fafb80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.570+0000 7fafd0c0c700 1 --2- 172.21.15.98:0/1645304397 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fafb80a2cb0 0x7fafb80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.570+0000 7fafc9ffb700 1 -- 172.21.15.98:0/1645304397 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fafb40d0de0 con 0x7fafcc107290 2024-05-09T08:49:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.570+0000 7fafc9ffb700 1 --2- 172.21.15.98:0/1645304397 >> [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207] conn(0x7fafb80ad970 0x7fafb80afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.570+0000 7fafc9ffb700 1 -- 172.21.15.98:0/1645304397 --> [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fafb4001d30 con 0x7fafb80ad970 2024-05-09T08:49:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.570+0000 7fafc9ffb700 1 -- 172.21.15.98:0/1645304397 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fafb40d1230 con 0x7fafcc107290 2024-05-09T08:49:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.570+0000 7fafd0c0c700 1 --2- 172.21.15.98:0/1645304397 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fafb80a2cb0 0x7fafb80a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fafc4005950 tx=0x7fafc40058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.570+0000 7fafcbfff700 1 --2- 172.21.15.98:0/1645304397 >> [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207] conn(0x7fafb80ad970 0x7fafb80afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.571+0000 7fafcbfff700 1 --2- 172.21.15.98:0/1645304397 >> [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207] conn(0x7fafb80ad970 0x7fafb80afdb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.571+0000 7fafc9ffb700 1 -- 172.21.15.98:0/1645304397 <== osd.11 v2:172.21.15.142:6824/4047948207 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fafb4001d30 con 0x7fafb80ad970 2024-05-09T08:49:09.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.588+0000 7fafd2e70700 1 -- 172.21.15.98:0/1645304397 --> [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fafcc062eb0 con 0x7fafb80ad970 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.588+0000 7fafc9ffb700 1 -- 172.21.15.98:0/1645304397 <== osd.11 v2:172.21.15.142:6824/4047948207 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fafcc062eb0 con 0x7fafb80ad970 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.588+0000 7fafd2e70700 1 -- 172.21.15.98:0/1645304397 >> [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207] conn(0x7fafb80ad970 msgr2=0x7fafb80afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.588+0000 7fafd2e70700 1 --2- 172.21.15.98:0/1645304397 >> [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207] conn(0x7fafb80ad970 0x7fafb80afdb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.589+0000 7fafd2e70700 1 -- 172.21.15.98:0/1645304397 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fafb80a2cb0 msgr2=0x7fafb80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.589+0000 7fafd2e70700 1 --2- 172.21.15.98:0/1645304397 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fafb80a2cb0 0x7fafb80a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fafc4005950 tx=0x7fafc40058e0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.589+0000 7fafd2e70700 1 -- 172.21.15.98:0/1645304397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafcc107290 msgr2=0x7fafcc1a49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.589+0000 7fafd2e70700 1 --2- 172.21.15.98:0/1645304397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafcc107290 0x7fafcc1a49a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fafb40047b0 tx=0x7fafb4005e80 comp rx=0 tx=0).stop 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.589+0000 7fafd140d700 1 -- 172.21.15.98:0/1645304397 reap_dead start 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.589+0000 7fafd2e70700 1 -- 172.21.15.98:0/1645304397 shutdown_connections 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.589+0000 7fafd2e70700 1 -- 172.21.15.98:0/1645304397 >> 172.21.15.98:0/1645304397 conn(0x7fafcc0781d0 msgr2=0x7fafcc0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.589+0000 7fafd2e70700 1 -- 172.21.15.98:0/1645304397 shutdown_connections 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.589+0000 7fafd2e70700 1 -- 172.21.15.98:0/1645304397 wait complete. 2024-05-09T08:49:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.791+0000 7f8899097700 1 -- 172.21.15.98:0/2923069293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8894101c10 msgr2=0x7f889410e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.791+0000 7f8899097700 1 --2- 172.21.15.98:0/2923069293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8894101c10 0x7f889410e100 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f888c00b0a0 tx=0x7f888c00b3b0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.791+0000 7f8899097700 1 -- 172.21.15.98:0/2923069293 shutdown_connections 2024-05-09T08:49:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.791+0000 7f8899097700 1 --2- 172.21.15.98:0/2923069293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8894101c10 0x7f889410e100 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.791+0000 7f8899097700 1 --2- 172.21.15.98:0/2923069293 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88941072c0 0x7f88941076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.791+0000 7f8899097700 1 --2- 172.21.15.98:0/2923069293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8894101250 0x7f88941016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.791+0000 7f8899097700 1 -- 172.21.15.98:0/2923069293 >> 172.21.15.98:0/2923069293 conn(0x7f8894078260 msgr2=0x7f8894078670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.792+0000 7f8899097700 1 -- 172.21.15.98:0/2923069293 shutdown_connections 2024-05-09T08:49:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.792+0000 7f8899097700 1 -- 172.21.15.98:0/2923069293 wait complete. 2024-05-09T08:49:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.792+0000 7f8899097700 1 Processor -- start 2024-05-09T08:49:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.792+0000 7f8899097700 1 -- start start 2024-05-09T08:49:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8899097700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8894101250 0x7f8894199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8899097700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8894101c10 0x7f8894199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8899097700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88941072c0 0x7f88941a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8899097700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f88941115b0 con 0x7f8894101250 2024-05-09T08:49:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8899097700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8894111430 con 0x7f8894101c10 2024-05-09T08:49:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8899097700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8894111730 con 0x7f88941072c0 2024-05-09T08:49:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8892d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8894101250 0x7f8894199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8892d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8894101250 0x7f8894199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48608/0 (socket says 172.21.15.98:48608) 2024-05-09T08:49:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8892d9d700 1 -- 172.21.15.98:0/1436359520 learned_addr learned my addr 172.21.15.98:0/1436359520 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f889359e700 1 --2- 172.21.15.98:0/1436359520 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88941072c0 0x7f88941a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f889259c700 1 --2- 172.21.15.98:0/1436359520 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8894101c10 0x7f8894199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8892d9d700 1 -- 172.21.15.98:0/1436359520 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88941072c0 msgr2=0x7f88941a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8892d9d700 1 --2- 172.21.15.98:0/1436359520 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88941072c0 0x7f88941a4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8892d9d700 1 -- 172.21.15.98:0/1436359520 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8894101c10 msgr2=0x7f8894199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8892d9d700 1 --2- 172.21.15.98:0/1436359520 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8894101c10 0x7f8894199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8892d9d700 1 -- 172.21.15.98:0/1436359520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88941a4fd0 con 0x7f8894101250 2024-05-09T08:49:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f8892d9d700 1 --2- 172.21.15.98:0/1436359520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8894101250 0x7f8894199a10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f888400b530 tx=0x7f888400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.793+0000 7f887bfff700 1 -- 172.21.15.98:0/1436359520 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8884015020 con 0x7f8894101250 2024-05-09T08:49:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.794+0000 7f8899097700 1 -- 172.21.15.98:0/1436359520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88941a51a0 con 0x7f8894101250 2024-05-09T08:49:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.794+0000 7f887bfff700 1 -- 172.21.15.98:0/1436359520 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8884011d10 con 0x7f8894101250 2024-05-09T08:49:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.794+0000 7f887bfff700 1 -- 172.21.15.98:0/1436359520 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888400f2e0 con 0x7f8894101250 2024-05-09T08:49:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.794+0000 7f8899097700 1 -- 172.21.15.98:0/1436359520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88941a5730 con 0x7f8894101250 2024-05-09T08:49:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.794+0000 7f8899097700 1 -- 172.21.15.98:0/1436359520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8894062a30 con 0x7f8894101250 2024-05-09T08:49:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.796+0000 7f887bfff700 1 -- 172.21.15.98:0/1436359520 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8884018020 con 0x7f8894101250 2024-05-09T08:49:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.796+0000 7f887bfff700 1 --2- 172.21.15.98:0/1436359520 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f887c0a2cb0 0x7f887c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.796+0000 7f889259c700 1 --2- 172.21.15.98:0/1436359520 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f887c0a2cb0 0x7f887c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.796+0000 7f887bfff700 1 -- 172.21.15.98:0/1436359520 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f88840cf160 con 0x7f8894101250 2024-05-09T08:49:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.797+0000 7f887bfff700 1 --2- 172.21.15.98:0/1436359520 >> [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214] conn(0x7f887c0ad970 0x7f887c0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.797+0000 7f887bfff700 1 -- 172.21.15.98:0/1436359520 --> [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88840044e0 con 0x7f887c0ad970 2024-05-09T08:49:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.797+0000 7f887bfff700 1 -- 172.21.15.98:0/1436359520 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f88840cf5b0 con 0x7f8894101250 2024-05-09T08:49:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.797+0000 7f889259c700 1 --2- 172.21.15.98:0/1436359520 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f887c0a2cb0 0x7f887c0a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8888005950 tx=0x7f88880058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.797+0000 7f889359e700 1 --2- 172.21.15.98:0/1436359520 >> [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214] conn(0x7f887c0ad970 0x7f887c0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.797+0000 7f889359e700 1 --2- 172.21.15.98:0/1436359520 >> [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214] conn(0x7f887c0ad970 0x7f887c0afdb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.798+0000 7f887bfff700 1 -- 172.21.15.98:0/1436359520 <== osd.0 v2:172.21.15.98:6802/2307851214 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f88840044e0 con 0x7f887c0ad970 2024-05-09T08:49:09.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.814+0000 7f8899097700 1 -- 172.21.15.98:0/1436359520 --> [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8894062e30 con 0x7f887c0ad970 2024-05-09T08:49:09.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.814+0000 7f887bfff700 1 -- 172.21.15.98:0/1436359520 <== osd.0 v2:172.21.15.98:6802/2307851214 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8894062e30 con 0x7f887c0ad970 2024-05-09T08:49:09.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.815+0000 7f8899097700 1 -- 172.21.15.98:0/1436359520 >> [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214] conn(0x7f887c0ad970 msgr2=0x7f887c0afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:09.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.815+0000 7f8899097700 1 --2- 172.21.15.98:0/1436359520 >> [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214] conn(0x7f887c0ad970 0x7f887c0afdb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.815+0000 7f8899097700 1 -- 172.21.15.98:0/1436359520 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f887c0a2cb0 msgr2=0x7f887c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:09.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.815+0000 7f8899097700 1 --2- 172.21.15.98:0/1436359520 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f887c0a2cb0 0x7f887c0a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8888005950 tx=0x7f88880058e0 comp rx=0 tx=0).stop 2024-05-09T08:49:09.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.815+0000 7f8899097700 1 -- 172.21.15.98:0/1436359520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8894101250 msgr2=0x7f8894199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:09.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.815+0000 7f8899097700 1 --2- 172.21.15.98:0/1436359520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8894101250 0x7f8894199a10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f888400b530 tx=0x7f888400b840 comp rx=0 tx=0).stop 2024-05-09T08:49:09.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.815+0000 7f889359e700 1 -- 172.21.15.98:0/1436359520 reap_dead start 2024-05-09T08:49:09.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.815+0000 7f8899097700 1 -- 172.21.15.98:0/1436359520 shutdown_connections 2024-05-09T08:49:09.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.815+0000 7f8899097700 1 -- 172.21.15.98:0/1436359520 >> 172.21.15.98:0/1436359520 conn(0x7f8894078260 msgr2=0x7f88940ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:09.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.815+0000 7f8899097700 1 -- 172.21.15.98:0/1436359520 shutdown_connections 2024-05-09T08:49:09.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:09.815+0000 7f8899097700 1 -- 172.21.15.98:0/1436359520 wait complete. 2024-05-09T08:49:10.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:10 smithi098 ceph-mon[30092]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 11 op/s 2024-05-09T08:49:10.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:10 smithi142 ceph-mon[34625]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 11 op/s 2024-05-09T08:49:10.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:10 smithi105 ceph-mon[34498]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 11 op/s 2024-05-09T08:49:11.731 INFO:teuthology.orchestra.run.smithi098.stdout:34359738446 2024-05-09T08:49:11.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-09T08:49:12.464 INFO:teuthology.orchestra.run.smithi098.stdout:146028888123 2024-05-09T08:49:12.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-09T08:49:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:12 smithi142 ceph-mon[34625]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-09T08:49:12.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:12 smithi105 ceph-mon[34498]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-09T08:49:12.833 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:12 smithi098 ceph-mon[30092]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-09T08:49:12.859 INFO:teuthology.orchestra.run.smithi098.stdout:111669149759 2024-05-09T08:49:12.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-09T08:49:14.015 INFO:teuthology.orchestra.run.smithi098.stdout:347892351000 2024-05-09T08:49:14.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-09T08:49:14.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:14 smithi098 ceph-mon[30092]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:49:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.642+0000 7f31a55c4700 1 -- 172.21.15.98:0/3967823735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a01023d0 msgr2=0x7f31a0102850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.642+0000 7f31a55c4700 1 --2- 172.21.15.98:0/3967823735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a01023d0 0x7f31a0102850 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f31900099c0 tx=0x7f3190009cd0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.642+0000 7f31a55c4700 1 -- 172.21.15.98:0/3967823735 shutdown_connections 2024-05-09T08:49:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.642+0000 7f31a55c4700 1 --2- 172.21.15.98:0/3967823735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31a0102d90 0x7f31a010f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.642+0000 7f31a55c4700 1 --2- 172.21.15.98:0/3967823735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31a01083e0 0x7f31a01087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.642+0000 7f31a55c4700 1 --2- 172.21.15.98:0/3967823735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a01023d0 0x7f31a0102850 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.642+0000 7f31a55c4700 1 -- 172.21.15.98:0/3967823735 >> 172.21.15.98:0/3967823735 conn(0x7f31a0079480 msgr2=0x7f31a0079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.642+0000 7f31a55c4700 1 -- 172.21.15.98:0/3967823735 shutdown_connections 2024-05-09T08:49:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.642+0000 7f31a55c4700 1 -- 172.21.15.98:0/3967823735 wait complete. 2024-05-09T08:49:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.643+0000 7f31a55c4700 1 Processor -- start 2024-05-09T08:49:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.643+0000 7f31a55c4700 1 -- start start 2024-05-09T08:49:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.643+0000 7f31a55c4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31a01023d0 0x7f31a0074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.643+0000 7f31a55c4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a0102d90 0x7f31a0074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.643+0000 7f31a55c4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31a01083e0 0x7f31a00750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.643+0000 7f31a55c4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31a01126d0 con 0x7f31a0102d90 2024-05-09T08:49:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.643+0000 7f31a55c4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0112550 con 0x7f31a01083e0 2024-05-09T08:49:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.643+0000 7f31a55c4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0112850 con 0x7f31a01023d0 2024-05-09T08:49:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.643+0000 7f319e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a0102d90 0x7f31a0074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.643+0000 7f319e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a0102d90 0x7f31a0074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48634/0 (socket says 172.21.15.98:48634) 2024-05-09T08:49:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.643+0000 7f319e7fc700 1 -- 172.21.15.98:0/3597795002 learned_addr learned my addr 172.21.15.98:0/3597795002 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.643+0000 7f319effd700 1 --2- 172.21.15.98:0/3597795002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31a01023d0 0x7f31a0074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.643+0000 7f319f7fe700 1 --2- 172.21.15.98:0/3597795002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31a01083e0 0x7f31a00750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.644+0000 7f319e7fc700 1 -- 172.21.15.98:0/3597795002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31a01023d0 msgr2=0x7f31a0074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.644+0000 7f319e7fc700 1 --2- 172.21.15.98:0/3597795002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31a01023d0 0x7f31a0074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.644+0000 7f319e7fc700 1 -- 172.21.15.98:0/3597795002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31a01083e0 msgr2=0x7f31a00750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.644+0000 7f319e7fc700 1 --2- 172.21.15.98:0/3597795002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31a01083e0 0x7f31a00750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.644+0000 7f319e7fc700 1 -- 172.21.15.98:0/3597795002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a01a6f80 con 0x7f31a0102d90 2024-05-09T08:49:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.644+0000 7f319e7fc700 1 --2- 172.21.15.98:0/3597795002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a0102d90 0x7f31a0074b60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3190009990 tx=0x7f3190009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.644+0000 7f3197fff700 1 -- 172.21.15.98:0/3597795002 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190027070 con 0x7f31a0102d90 2024-05-09T08:49:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.644+0000 7f31a55c4700 1 -- 172.21.15.98:0/3597795002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a01a7210 con 0x7f31a0102d90 2024-05-09T08:49:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.644+0000 7f3197fff700 1 -- 172.21.15.98:0/3597795002 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3190003e40 con 0x7f31a0102d90 2024-05-09T08:49:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.644+0000 7f3197fff700 1 -- 172.21.15.98:0/3597795002 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190022bf0 con 0x7f31a0102d90 2024-05-09T08:49:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.644+0000 7f31a55c4700 1 -- 172.21.15.98:0/3597795002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a01a76c0 con 0x7f31a0102d90 2024-05-09T08:49:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.645+0000 7f31a55c4700 1 -- 172.21.15.98:0/3597795002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f31a0062ab0 con 0x7f31a0102d90 2024-05-09T08:49:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.646+0000 7f3197fff700 1 -- 172.21.15.98:0/3597795002 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f319002a070 con 0x7f31a0102d90 2024-05-09T08:49:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.647+0000 7f3197fff700 1 --2- 172.21.15.98:0/3597795002 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f318c0a2cb0 0x7f318c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.647+0000 7f319effd700 1 --2- 172.21.15.98:0/3597795002 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f318c0a2cb0 0x7f318c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.647+0000 7f3197fff700 1 -- 172.21.15.98:0/3597795002 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f31900d0d80 con 0x7f31a0102d90 2024-05-09T08:49:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.647+0000 7f3197fff700 1 --2- 172.21.15.98:0/3597795002 >> [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557] conn(0x7f318c0ad970 0x7f318c0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.647+0000 7f3197fff700 1 -- 172.21.15.98:0/3597795002 --> [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31900043d0 con 0x7f318c0ad970 2024-05-09T08:49:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.647+0000 7f3197fff700 1 -- 172.21.15.98:0/3597795002 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f31900d11d0 con 0x7f31a0102d90 2024-05-09T08:49:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.647+0000 7f319f7fe700 1 --2- 172.21.15.98:0/3597795002 >> [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557] conn(0x7f318c0ad970 0x7f318c0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.647+0000 7f319effd700 1 --2- 172.21.15.98:0/3597795002 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f318c0a2cb0 0x7f318c0a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f31880045b0 tx=0x7f318800b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.648+0000 7f319f7fe700 1 --2- 172.21.15.98:0/3597795002 >> [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557] conn(0x7f318c0ad970 0x7f318c0afdb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.649+0000 7f3197fff700 1 -- 172.21.15.98:0/3597795002 <== osd.2 v2:172.21.15.142:6800/1020582557 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f31900043d0 con 0x7f318c0ad970 2024-05-09T08:49:14.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:14 smithi142 ceph-mon[34625]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:49:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.665+0000 7f31a55c4700 1 -- 172.21.15.98:0/3597795002 --> [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f31a0062f30 con 0x7f318c0ad970 2024-05-09T08:49:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.666+0000 7f3197fff700 1 -- 172.21.15.98:0/3597795002 <== osd.2 v2:172.21.15.142:6800/1020582557 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f31a0062f30 con 0x7f318c0ad970 2024-05-09T08:49:14.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.666+0000 7f31a55c4700 1 -- 172.21.15.98:0/3597795002 >> [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557] conn(0x7f318c0ad970 msgr2=0x7f318c0afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:14.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.666+0000 7f31a55c4700 1 --2- 172.21.15.98:0/3597795002 >> [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557] conn(0x7f318c0ad970 0x7f318c0afdb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.666+0000 7f31a55c4700 1 -- 172.21.15.98:0/3597795002 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f318c0a2cb0 msgr2=0x7f318c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:14.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.666+0000 7f31a55c4700 1 --2- 172.21.15.98:0/3597795002 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f318c0a2cb0 0x7f318c0a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f31880045b0 tx=0x7f318800b3c0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.666+0000 7f31a55c4700 1 -- 172.21.15.98:0/3597795002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a0102d90 msgr2=0x7f31a0074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:14.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.666+0000 7f31a55c4700 1 --2- 172.21.15.98:0/3597795002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a0102d90 0x7f31a0074b60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3190009990 tx=0x7f3190009f80 comp rx=0 tx=0).stop 2024-05-09T08:49:14.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.666+0000 7f319f7fe700 1 -- 172.21.15.98:0/3597795002 reap_dead start 2024-05-09T08:49:14.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.666+0000 7f31a55c4700 1 -- 172.21.15.98:0/3597795002 shutdown_connections 2024-05-09T08:49:14.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.666+0000 7f31a55c4700 1 -- 172.21.15.98:0/3597795002 >> 172.21.15.98:0/3597795002 conn(0x7f31a0079480 msgr2=0x7f31a0100e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:14.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.667+0000 7f31a55c4700 1 -- 172.21.15.98:0/3597795002 shutdown_connections 2024-05-09T08:49:14.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.667+0000 7f31a55c4700 1 -- 172.21.15.98:0/3597795002 wait complete. 2024-05-09T08:49:14.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:14 smithi105 ceph-mon[34498]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:49:14.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.850+0000 7f5da8ec6700 1 -- 172.21.15.98:0/887580343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5da41065a0 msgr2=0x7f5da410e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:14.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.850+0000 7f5da8ec6700 1 --2- 172.21.15.98:0/887580343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5da41065a0 0x7f5da410e1b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00b0d0 tx=0x7f5d9c00b3e0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.850+0000 7f5da8ec6700 1 -- 172.21.15.98:0/887580343 shutdown_connections 2024-05-09T08:49:14.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.850+0000 7f5da8ec6700 1 --2- 172.21.15.98:0/887580343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5da4101c80 0x7f5da4102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.850+0000 7f5da8ec6700 1 --2- 172.21.15.98:0/887580343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5da41065a0 0x7f5da410e1b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.850+0000 7f5da8ec6700 1 --2- 172.21.15.98:0/887580343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5da4102630 0x7f5da4105ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.850+0000 7f5da8ec6700 1 -- 172.21.15.98:0/887580343 >> 172.21.15.98:0/887580343 conn(0x7f5da4077890 msgr2=0x7f5da40fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:14.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.850+0000 7f5da8ec6700 1 -- 172.21.15.98:0/887580343 shutdown_connections 2024-05-09T08:49:14.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.850+0000 7f5da8ec6700 1 -- 172.21.15.98:0/887580343 wait complete. 2024-05-09T08:49:14.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.851+0000 7f5da8ec6700 1 Processor -- start 2024-05-09T08:49:14.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.851+0000 7f5da8ec6700 1 -- start start 2024-05-09T08:49:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.851+0000 7f5da8ec6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5da4101c80 0x7f5da40fcd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.851+0000 7f5da8ec6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5da4102630 0x7f5da40fd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.851+0000 7f5da8ec6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5da41065a0 0x7f5da40fd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.851+0000 7f5da8ec6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5da41115e0 con 0x7f5da4102630 2024-05-09T08:49:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.851+0000 7f5da8ec6700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5da4111460 con 0x7f5da41065a0 2024-05-09T08:49:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.851+0000 7f5da8ec6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5da4111760 con 0x7f5da4101c80 2024-05-09T08:49:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5da1d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5da4102630 0x7f5da40fd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5da1d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5da4102630 0x7f5da40fd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48648/0 (socket says 172.21.15.98:48648) 2024-05-09T08:49:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5da1d9b700 1 -- 172.21.15.98:0/2627712353 learned_addr learned my addr 172.21.15.98:0/2627712353 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5da259c700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5da4101c80 0x7f5da40fcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5da2d9d700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5da41065a0 0x7f5da40fd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5da1d9b700 1 -- 172.21.15.98:0/2627712353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5da4101c80 msgr2=0x7f5da40fcd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5da1d9b700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5da4101c80 0x7f5da40fcd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5da1d9b700 1 -- 172.21.15.98:0/2627712353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5da41065a0 msgr2=0x7f5da40fd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:14.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5da1d9b700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5da41065a0 0x7f5da40fd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5da1d9b700 1 -- 172.21.15.98:0/2627712353 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5da41a7080 con 0x7f5da4102630 2024-05-09T08:49:14.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5da2d9d700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5da41065a0 0x7f5da40fd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:49:14.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5da1d9b700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5da4102630 0x7f5da40fd240 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5d9800d950 tx=0x7f5d9800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:14.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5d937fe700 1 -- 172.21.15.98:0/2627712353 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d98017070 con 0x7f5da4102630 2024-05-09T08:49:14.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5d937fe700 1 -- 172.21.15.98:0/2627712353 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d98005020 con 0x7f5da4102630 2024-05-09T08:49:14.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5d937fe700 1 -- 172.21.15.98:0/2627712353 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d98009b20 con 0x7f5da4102630 2024-05-09T08:49:14.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5da8ec6700 1 -- 172.21.15.98:0/2627712353 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5da41a7370 con 0x7f5da4102630 2024-05-09T08:49:14.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.852+0000 7f5da8ec6700 1 -- 172.21.15.98:0/2627712353 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5da41a7860 con 0x7f5da4102630 2024-05-09T08:49:14.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.853+0000 7f5da8ec6700 1 -- 172.21.15.98:0/2627712353 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5da4062a30 con 0x7f5da4102630 2024-05-09T08:49:14.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.855+0000 7f5d937fe700 1 -- 172.21.15.98:0/2627712353 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5d980051d0 con 0x7f5da4102630 2024-05-09T08:49:14.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.855+0000 7f5d937fe700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5d8c0a2cb0 0x7f5d8c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:14.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.855+0000 7f5da259c700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5d8c0a2cb0 0x7f5d8c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:14.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.855+0000 7f5d937fe700 1 -- 172.21.15.98:0/2627712353 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f5d980cf060 con 0x7f5da4102630 2024-05-09T08:49:14.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.856+0000 7f5d937fe700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764] conn(0x7f5d8c0ad970 0x7f5d8c0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:14.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.856+0000 7f5d937fe700 1 -- 172.21.15.98:0/2627712353 --> [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d980044e0 con 0x7f5d8c0ad970 2024-05-09T08:49:14.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.856+0000 7f5d937fe700 1 -- 172.21.15.98:0/2627712353 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f5d980cf4b0 con 0x7f5da4102630 2024-05-09T08:49:14.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.856+0000 7f5da259c700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5d8c0a2cb0 0x7f5d8c0a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400ac50 tx=0x7f5d9400a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:14.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.856+0000 7f5da2d9d700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764] conn(0x7f5d8c0ad970 0x7f5d8c0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:14.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.856+0000 7f5da2d9d700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764] conn(0x7f5d8c0ad970 0x7f5d8c0afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:14.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.857+0000 7f5d937fe700 1 -- 172.21.15.98:0/2627712353 <== osd.7 v2:172.21.15.105:6816/352668764 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5d980044e0 con 0x7f5d8c0ad970 2024-05-09T08:49:14.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.873+0000 7f5da8ec6700 1 -- 172.21.15.98:0/2627712353 --> [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5da4062e30 con 0x7f5d8c0ad970 2024-05-09T08:49:14.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.874+0000 7f5d937fe700 1 -- 172.21.15.98:0/2627712353 <== osd.7 v2:172.21.15.105:6816/352668764 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5da4062e30 con 0x7f5d8c0ad970 2024-05-09T08:49:14.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.874+0000 7f5da8ec6700 1 -- 172.21.15.98:0/2627712353 >> [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764] conn(0x7f5d8c0ad970 msgr2=0x7f5d8c0afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:14.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.874+0000 7f5da8ec6700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764] conn(0x7f5d8c0ad970 0x7f5d8c0afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:14.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.874+0000 7f5da8ec6700 1 -- 172.21.15.98:0/2627712353 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5d8c0a2cb0 msgr2=0x7f5d8c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:14.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.874+0000 7f5da8ec6700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f5d8c0a2cb0 0x7f5d8c0a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400ac50 tx=0x7f5d9400a380 comp rx=0 tx=0).stop 2024-05-09T08:49:14.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.874+0000 7f5da8ec6700 1 -- 172.21.15.98:0/2627712353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5da4102630 msgr2=0x7f5da40fd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:14.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.874+0000 7f5da8ec6700 1 --2- 172.21.15.98:0/2627712353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5da4102630 0x7f5da40fd240 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5d9800d950 tx=0x7f5d9800dc60 comp rx=0 tx=0).stop 2024-05-09T08:49:14.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.874+0000 7f5da2d9d700 1 -- 172.21.15.98:0/2627712353 reap_dead start 2024-05-09T08:49:14.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.875+0000 7f5da8ec6700 1 -- 172.21.15.98:0/2627712353 shutdown_connections 2024-05-09T08:49:14.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.875+0000 7f5da8ec6700 1 -- 172.21.15.98:0/2627712353 >> 172.21.15.98:0/2627712353 conn(0x7f5da4077890 msgr2=0x7f5da4100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:14.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.875+0000 7f5da8ec6700 1 -- 172.21.15.98:0/2627712353 shutdown_connections 2024-05-09T08:49:14.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:14.875+0000 7f5da8ec6700 1 -- 172.21.15.98:0/2627712353 wait complete. 2024-05-09T08:49:15.772 INFO:teuthology.orchestra.run.smithi098.stdout:223338299437 2024-05-09T08:49:15.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-09T08:49:16.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:16 smithi098 ceph-mon[30092]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-09T08:49:16.637 INFO:teuthology.orchestra.run.smithi098.stdout:77309411399 2024-05-09T08:49:16.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-09T08:49:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:16 smithi142 ceph-mon[34625]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-09T08:49:16.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:16 smithi105 ceph-mon[34498]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-09T08:49:17.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.601+0000 7fe162a87700 1 -- 172.21.15.98:0/1945542020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c107660 msgr2=0x7fe15c10f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:17.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.601+0000 7fe162a87700 1 --2- 172.21.15.98:0/1945542020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c107660 0x7fe15c10f230 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe15400b0a0 tx=0x7fe15400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:49:17.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.601+0000 7fe162a87700 1 -- 172.21.15.98:0/1945542020 shutdown_connections 2024-05-09T08:49:17.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.601+0000 7fe162a87700 1 --2- 172.21.15.98:0/1945542020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c107660 0x7fe15c10f230 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:17.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.601+0000 7fe162a87700 1 --2- 172.21.15.98:0/1945542020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe15c1025d0 0x7fe15c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:17.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.601+0000 7fe162a87700 1 --2- 172.21.15.98:0/1945542020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe15c102f80 0x7fe15c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:17.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.601+0000 7fe162a87700 1 -- 172.21.15.98:0/1945542020 >> 172.21.15.98:0/1945542020 conn(0x7fe15c079480 msgr2=0x7fe15c079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:17.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.601+0000 7fe162a87700 1 -- 172.21.15.98:0/1945542020 shutdown_connections 2024-05-09T08:49:17.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.601+0000 7fe162a87700 1 -- 172.21.15.98:0/1945542020 wait complete. 2024-05-09T08:49:17.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe162a87700 1 Processor -- start 2024-05-09T08:49:17.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe162a87700 1 -- start start 2024-05-09T08:49:17.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe162a87700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c1025d0 0x7fe15c19e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:17.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe162a87700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe15c102f80 0x7fe15c19ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:17.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe162a87700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe15c107660 0x7fe15c198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:17.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe162a87700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c112640 con 0x7fe15c1025d0 2024-05-09T08:49:17.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe162a87700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c1124c0 con 0x7fe15c102f80 2024-05-09T08:49:17.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe162a87700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c1127c0 con 0x7fe15c107660 2024-05-09T08:49:17.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe160823700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c1025d0 0x7fe15c19e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:17.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe160823700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c1025d0 0x7fe15c19e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43836/0 (socket says 172.21.15.98:43836) 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe160823700 1 -- 172.21.15.98:0/762067083 learned_addr learned my addr 172.21.15.98:0/762067083 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe15bfff700 1 --2- 172.21.15.98:0/762067083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe15c102f80 0x7fe15c19ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe161024700 1 --2- 172.21.15.98:0/762067083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe15c107660 0x7fe15c198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe160823700 1 -- 172.21.15.98:0/762067083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe15c107660 msgr2=0x7fe15c198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe160823700 1 --2- 172.21.15.98:0/762067083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe15c107660 0x7fe15c198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe160823700 1 -- 172.21.15.98:0/762067083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe15c102f80 msgr2=0x7fe15c19ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe160823700 1 --2- 172.21.15.98:0/762067083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe15c102f80 0x7fe15c19ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.602+0000 7fe160823700 1 -- 172.21.15.98:0/762067083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe15c199020 con 0x7fe15c1025d0 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.603+0000 7fe160823700 1 --2- 172.21.15.98:0/762067083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c1025d0 0x7fe15c19e6c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe15000d980 tx=0x7fe15000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.603+0000 7fe159ffb700 1 -- 172.21.15.98:0/762067083 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe150017070 con 0x7fe15c1025d0 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.603+0000 7fe162a87700 1 -- 172.21.15.98:0/762067083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe15c199310 con 0x7fe15c1025d0 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.603+0000 7fe159ffb700 1 -- 172.21.15.98:0/762067083 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe150005020 con 0x7fe15c1025d0 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.603+0000 7fe159ffb700 1 -- 172.21.15.98:0/762067083 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15000fbf0 con 0x7fe15c1025d0 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.603+0000 7fe162a87700 1 -- 172.21.15.98:0/762067083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe15c1a5520 con 0x7fe15c1025d0 2024-05-09T08:49:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.604+0000 7fe162a87700 1 -- 172.21.15.98:0/762067083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe15c062ab0 con 0x7fe15c1025d0 2024-05-09T08:49:17.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.605+0000 7fe159ffb700 1 -- 172.21.15.98:0/762067083 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe1500051d0 con 0x7fe15c1025d0 2024-05-09T08:49:17.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.606+0000 7fe159ffb700 1 --2- 172.21.15.98:0/762067083 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe14c0a2cb0 0x7fe14c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:17.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.606+0000 7fe15bfff700 1 --2- 172.21.15.98:0/762067083 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe14c0a2cb0 0x7fe14c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:17.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.606+0000 7fe159ffb700 1 -- 172.21.15.98:0/762067083 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe1500cf900 con 0x7fe15c1025d0 2024-05-09T08:49:17.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.606+0000 7fe159ffb700 1 --2- 172.21.15.98:0/762067083 >> [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222] conn(0x7fe14c0ade10 0x7fe14c0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:17.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.606+0000 7fe159ffb700 1 -- 172.21.15.98:0/762067083 --> [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe14c0b0980 con 0x7fe14c0ade10 2024-05-09T08:49:17.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.606+0000 7fe159ffb700 1 -- 172.21.15.98:0/762067083 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fe1500cfd50 con 0x7fe15c1025d0 2024-05-09T08:49:17.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.607+0000 7fe161024700 1 --2- 172.21.15.98:0/762067083 >> [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222] conn(0x7fe14c0ade10 0x7fe14c0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:17.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.607+0000 7fe15bfff700 1 --2- 172.21.15.98:0/762067083 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe14c0a2cb0 0x7fe14c0a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe15c108970 tx=0x7fe14800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:17.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.607+0000 7fe161024700 1 --2- 172.21.15.98:0/762067083 >> [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222] conn(0x7fe14c0ade10 0x7fe14c0b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:17.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.608+0000 7fe159ffb700 1 -- 172.21.15.98:0/762067083 <== osd.1 v2:172.21.15.105:6800/2333305222 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe14c0b0980 con 0x7fe14c0ade10 2024-05-09T08:49:17.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:49:17.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-09T08:49:17.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-05-09T08:49:17.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-05-09T08:49:17.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2024-05-09T08:49:17.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-05-09T08:49:17.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-05-09T08:49:17.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-05-09T08:49:17.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-05-09T08:49:17.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-05-09T08:49:17.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:17 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 8]}]: dispatch 2024-05-09T08:49:17.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.625+0000 7fe162a87700 1 -- 172.21.15.98:0/762067083 --> [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe15c062f30 con 0x7fe14c0ade10 2024-05-09T08:49:17.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.626+0000 7fe159ffb700 1 -- 172.21.15.98:0/762067083 <== osd.1 v2:172.21.15.105:6800/2333305222 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe15c062f30 con 0x7fe14c0ade10 2024-05-09T08:49:17.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.626+0000 7fe162a87700 1 -- 172.21.15.98:0/762067083 >> [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222] conn(0x7fe14c0ade10 msgr2=0x7fe14c0b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:17.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.626+0000 7fe162a87700 1 --2- 172.21.15.98:0/762067083 >> [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222] conn(0x7fe14c0ade10 0x7fe14c0b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:17.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.626+0000 7fe162a87700 1 -- 172.21.15.98:0/762067083 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe14c0a2cb0 msgr2=0x7fe14c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:17.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.626+0000 7fe162a87700 1 --2- 172.21.15.98:0/762067083 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe14c0a2cb0 0x7fe14c0a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe15c108970 tx=0x7fe14800a380 comp rx=0 tx=0).stop 2024-05-09T08:49:17.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.626+0000 7fe162a87700 1 -- 172.21.15.98:0/762067083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c1025d0 msgr2=0x7fe15c19e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:17.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.626+0000 7fe162a87700 1 --2- 172.21.15.98:0/762067083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c1025d0 0x7fe15c19e6c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe15000d980 tx=0x7fe15000dc90 comp rx=0 tx=0).stop 2024-05-09T08:49:17.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.626+0000 7fe161024700 1 -- 172.21.15.98:0/762067083 reap_dead start 2024-05-09T08:49:17.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.626+0000 7fe162a87700 1 -- 172.21.15.98:0/762067083 shutdown_connections 2024-05-09T08:49:17.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.626+0000 7fe162a87700 1 -- 172.21.15.98:0/762067083 >> 172.21.15.98:0/762067083 conn(0x7fe15c079480 msgr2=0x7fe15c100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:17.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.626+0000 7fe162a87700 1 -- 172.21.15.98:0/762067083 shutdown_connections 2024-05-09T08:49:17.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:17.626+0000 7fe162a87700 1 -- 172.21.15.98:0/762067083 wait complete. 2024-05-09T08:49:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:49:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-09T08:49:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-05-09T08:49:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-05-09T08:49:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2024-05-09T08:49:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-05-09T08:49:17.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-05-09T08:49:17.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-05-09T08:49:17.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-05-09T08:49:17.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-05-09T08:49:17.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:17 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 8]}]: dispatch 2024-05-09T08:49:17.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:49:17.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-09T08:49:17.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-05-09T08:49:17.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-05-09T08:49:17.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2024-05-09T08:49:17.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-05-09T08:49:17.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-05-09T08:49:17.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-05-09T08:49:17.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-05-09T08:49:17.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-05-09T08:49:17.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:17 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 8]}]: dispatch 2024-05-09T08:49:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:18 smithi098 ceph-mon[30092]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:49:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-09T08:49:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-05-09T08:49:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-05-09T08:49:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2024-05-09T08:49:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-05-09T08:49:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-05-09T08:49:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-05-09T08:49:18.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-05-09T08:49:18.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-05-09T08:49:18.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 8]}]': finished 2024-05-09T08:49:18.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:18 smithi098 ceph-mon[30092]: osdmap e91: 12 total, 12 up, 12 in 2024-05-09T08:49:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:18 smithi142 ceph-mon[34625]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:49:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-09T08:49:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-05-09T08:49:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-05-09T08:49:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2024-05-09T08:49:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-05-09T08:49:18.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-05-09T08:49:18.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-05-09T08:49:18.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-05-09T08:49:18.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-05-09T08:49:18.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 8]}]': finished 2024-05-09T08:49:18.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:18 smithi142 ceph-mon[34625]: osdmap e91: 12 total, 12 up, 12 in 2024-05-09T08:49:18.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:18 smithi105 ceph-mon[34498]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:49:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-09T08:49:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-05-09T08:49:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-05-09T08:49:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2024-05-09T08:49:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-05-09T08:49:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-05-09T08:49:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-05-09T08:49:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-05-09T08:49:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-05-09T08:49:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 8]}]': finished 2024-05-09T08:49:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:18 smithi105 ceph-mon[34498]: osdmap e91: 12 total, 12 up, 12 in 2024-05-09T08:49:18.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.884+0000 7fb706cd8700 1 -- 172.21.15.98:0/1515833683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb700101260 msgr2=0x7fb7001016c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:18.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.884+0000 7fb706cd8700 1 --2- 172.21.15.98:0/1515833683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb700101260 0x7fb7001016c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009990 tx=0x7fb6f0009ca0 comp rx=0 tx=0).stop 2024-05-09T08:49:18.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.884+0000 7fb706cd8700 1 -- 172.21.15.98:0/1515833683 shutdown_connections 2024-05-09T08:49:18.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.884+0000 7fb706cd8700 1 --2- 172.21.15.98:0/1515833683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb700101c00 0x7fb70010e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:18.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.884+0000 7fb706cd8700 1 --2- 172.21.15.98:0/1515833683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7001072a0 0x7fb700107680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:18.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.884+0000 7fb706cd8700 1 --2- 172.21.15.98:0/1515833683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb700101260 0x7fb7001016c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:18.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.884+0000 7fb706cd8700 1 -- 172.21.15.98:0/1515833683 >> 172.21.15.98:0/1515833683 conn(0x7fb700078150 msgr2=0x7fb700078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:18.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.884+0000 7fb706cd8700 1 -- 172.21.15.98:0/1515833683 shutdown_connections 2024-05-09T08:49:18.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.884+0000 7fb706cd8700 1 -- 172.21.15.98:0/1515833683 wait complete. 2024-05-09T08:49:18.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.884+0000 7fb706cd8700 1 Processor -- start 2024-05-09T08:49:18.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb706cd8700 1 -- start start 2024-05-09T08:49:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb706cd8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb700101260 0x7fb700197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb706cd8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb700101c00 0x7fb700197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb706cd8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7001072a0 0x7fb700198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb706cd8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb700111500 con 0x7fb700101c00 2024-05-09T08:49:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb706cd8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb700111380 con 0x7fb7001072a0 2024-05-09T08:49:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb706cd8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb700111680 con 0x7fb700101260 2024-05-09T08:49:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb6f7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb700101c00 0x7fb700197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb6f7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb700101c00 0x7fb700197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43850/0 (socket says 172.21.15.98:43850) 2024-05-09T08:49:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb6f7fff700 1 -- 172.21.15.98:0/449860324 learned_addr learned my addr 172.21.15.98:0/449860324 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb704a74700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb700101260 0x7fb700197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb705275700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7001072a0 0x7fb700198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb6f7fff700 1 -- 172.21.15.98:0/449860324 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb700101260 msgr2=0x7fb700197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb6f7fff700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb700101260 0x7fb700197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:18.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb6f7fff700 1 -- 172.21.15.98:0/449860324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7001072a0 msgr2=0x7fb700198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb6f7fff700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7001072a0 0x7fb700198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb6f7fff700 1 -- 172.21.15.98:0/449860324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7001a6f80 con 0x7fb700101c00 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb705275700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7001072a0 0x7fb700198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb6f7fff700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb700101c00 0x7fb700197c50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0000c00 tx=0x7fb6f000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb6f5ffb700 1 -- 172.21.15.98:0/449860324 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0027070 con 0x7fb700101c00 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb6f5ffb700 1 -- 172.21.15.98:0/449860324 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6f0003d70 con 0x7fb700101c00 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.885+0000 7fb706cd8700 1 -- 172.21.15.98:0/449860324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7001a7210 con 0x7fb700101c00 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.886+0000 7fb6f5ffb700 1 -- 172.21.15.98:0/449860324 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f000f460 con 0x7fb700101c00 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.886+0000 7fb706cd8700 1 -- 172.21.15.98:0/449860324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7001a7660 con 0x7fb700101c00 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.887+0000 7fb706cd8700 1 -- 172.21.15.98:0/449860324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb700062ab0 con 0x7fb700101c00 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.888+0000 7fb6f5ffb700 1 -- 172.21.15.98:0/449860324 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb6f002a070 con 0x7fb700101c00 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.889+0000 7fb6f5ffb700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb6e80a2cb0 0x7fb6e80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.889+0000 7fb704a74700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb6e80a2cb0 0x7fb6e80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.889+0000 7fb6f5ffb700 1 -- 172.21.15.98:0/449860324 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fb6f00d0a60 con 0x7fb700101c00 2024-05-09T08:49:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.889+0000 7fb6f5ffb700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218] conn(0x7fb6e80ade10 0x7fb6e80b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.889+0000 7fb6f5ffb700 1 -- 172.21.15.98:0/449860324 --> [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6f00043d0 con 0x7fb6e80ade10 2024-05-09T08:49:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.889+0000 7fb6f5ffb700 1 -- 172.21.15.98:0/449860324 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fb6f00d0eb0 con 0x7fb700101c00 2024-05-09T08:49:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.889+0000 7fb704a74700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb6e80a2cb0 0x7fb6e80a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00f540 tx=0x7fb6fc00b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.889+0000 7fb705275700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218] conn(0x7fb6e80ade10 0x7fb6e80b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.890+0000 7fb705275700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218] conn(0x7fb6e80ade10 0x7fb6e80b02b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.891+0000 7fb6f5ffb700 1 -- 172.21.15.98:0/449860324 <== osd.10 v2:172.21.15.105:6824/1999822218 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb6f00043d0 con 0x7fb6e80ade10 2024-05-09T08:49:18.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.908+0000 7fb706cd8700 1 -- 172.21.15.98:0/449860324 --> [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb700062f30 con 0x7fb6e80ade10 2024-05-09T08:49:18.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.908+0000 7fb6f5ffb700 1 -- 172.21.15.98:0/449860324 <== osd.10 v2:172.21.15.105:6824/1999822218 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb700062f30 con 0x7fb6e80ade10 2024-05-09T08:49:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.908+0000 7fb706cd8700 1 -- 172.21.15.98:0/449860324 >> [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218] conn(0x7fb6e80ade10 msgr2=0x7fb6e80b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.908+0000 7fb706cd8700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218] conn(0x7fb6e80ade10 0x7fb6e80b02b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.909+0000 7fb706cd8700 1 -- 172.21.15.98:0/449860324 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb6e80a2cb0 msgr2=0x7fb6e80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.909+0000 7fb706cd8700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb6e80a2cb0 0x7fb6e80a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00f540 tx=0x7fb6fc00b040 comp rx=0 tx=0).stop 2024-05-09T08:49:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.909+0000 7fb706cd8700 1 -- 172.21.15.98:0/449860324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb700101c00 msgr2=0x7fb700197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.909+0000 7fb706cd8700 1 --2- 172.21.15.98:0/449860324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb700101c00 0x7fb700197c50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0000c00 tx=0x7fb6f000bab0 comp rx=0 tx=0).stop 2024-05-09T08:49:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.909+0000 7fb705275700 1 -- 172.21.15.98:0/449860324 reap_dead start 2024-05-09T08:49:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.909+0000 7fb706cd8700 1 -- 172.21.15.98:0/449860324 shutdown_connections 2024-05-09T08:49:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.909+0000 7fb706cd8700 1 -- 172.21.15.98:0/449860324 >> 172.21.15.98:0/449860324 conn(0x7fb700078150 msgr2=0x7fb7000ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.909+0000 7fb706cd8700 1 -- 172.21.15.98:0/449860324 shutdown_connections 2024-05-09T08:49:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:18.909+0000 7fb706cd8700 1 -- 172.21.15.98:0/449860324 wait complete. 2024-05-09T08:49:19.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.134+0000 7f81586bc700 1 -- 172.21.15.98:0/1276343753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8150107280 msgr2=0x7f8150107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:19.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.134+0000 7f81586bc700 1 --2- 172.21.15.98:0/1276343753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8150107280 0x7f8150107660 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8140009990 tx=0x7f8140009ca0 comp rx=0 tx=0).stop 2024-05-09T08:49:19.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.134+0000 7f81586bc700 1 -- 172.21.15.98:0/1276343753 shutdown_connections 2024-05-09T08:49:19.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.134+0000 7f81586bc700 1 --2- 172.21.15.98:0/1276343753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8150101bd0 0x7f815010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:19.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.134+0000 7f81586bc700 1 --2- 172.21.15.98:0/1276343753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8150107280 0x7f8150107660 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:19.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.134+0000 7f81586bc700 1 --2- 172.21.15.98:0/1276343753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8150101210 0x7f8150101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:19.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.134+0000 7f81586bc700 1 -- 172.21.15.98:0/1276343753 >> 172.21.15.98:0/1276343753 conn(0x7f8150078190 msgr2=0x7f81500785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:19.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.135+0000 7f81586bc700 1 -- 172.21.15.98:0/1276343753 shutdown_connections 2024-05-09T08:49:19.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.135+0000 7f81586bc700 1 -- 172.21.15.98:0/1276343753 wait complete. 2024-05-09T08:49:19.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.135+0000 7f81586bc700 1 Processor -- start 2024-05-09T08:49:19.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.135+0000 7f81586bc700 1 -- start start 2024-05-09T08:49:19.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f81586bc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8150101210 0x7f815019e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:19.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f81586bc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8150101bd0 0x7f815019ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:19.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f81586bc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8150107280 0x7f8150198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:19.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f81586bc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8150111520 con 0x7f8150101bd0 2024-05-09T08:49:19.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f81586bc700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f81501113a0 con 0x7f8150107280 2024-05-09T08:49:19.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f81586bc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f81501116a0 con 0x7f8150101210 2024-05-09T08:49:19.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f8155c57700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8150101bd0 0x7f815019ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f8155c57700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8150101bd0 0x7f815019ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43862/0 (socket says 172.21.15.98:43862) 2024-05-09T08:49:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f8155c57700 1 -- 172.21.15.98:0/2595353382 learned_addr learned my addr 172.21.15.98:0/2595353382 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f8156458700 1 --2- 172.21.15.98:0/2595353382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8150101210 0x7f815019e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f8156c59700 1 --2- 172.21.15.98:0/2595353382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8150107280 0x7f8150198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f8155c57700 1 -- 172.21.15.98:0/2595353382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8150101210 msgr2=0x7f815019e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f8155c57700 1 --2- 172.21.15.98:0/2595353382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8150101210 0x7f815019e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f8155c57700 1 -- 172.21.15.98:0/2595353382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8150107280 msgr2=0x7f8150198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f8155c57700 1 --2- 172.21.15.98:0/2595353382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8150107280 0x7f8150198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f8155c57700 1 -- 172.21.15.98:0/2595353382 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8150199080 con 0x7f8150101bd0 2024-05-09T08:49:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f8155c57700 1 --2- 172.21.15.98:0/2595353382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8150101bd0 0x7f815019ec50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f814c00c9b0 tx=0x7f814c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f81477fe700 1 -- 172.21.15.98:0/2595353382 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f814c007b00 con 0x7f8150101bd0 2024-05-09T08:49:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f81477fe700 1 -- 172.21.15.98:0/2595353382 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f814c00ce50 con 0x7f8150101bd0 2024-05-09T08:49:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.136+0000 7f81586bc700 1 -- 172.21.15.98:0/2595353382 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8150199370 con 0x7f8150101bd0 2024-05-09T08:49:19.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.137+0000 7f81477fe700 1 -- 172.21.15.98:0/2595353382 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f814c018660 con 0x7f8150101bd0 2024-05-09T08:49:19.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.137+0000 7f81586bc700 1 -- 172.21.15.98:0/2595353382 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81501a55e0 con 0x7f8150101bd0 2024-05-09T08:49:19.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.138+0000 7f81586bc700 1 -- 172.21.15.98:0/2595353382 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f81500ffe80 con 0x7f8150101bd0 2024-05-09T08:49:19.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.140+0000 7f81477fe700 1 -- 172.21.15.98:0/2595353382 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f814c00f450 con 0x7f8150101bd0 2024-05-09T08:49:19.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.140+0000 7f81477fe700 1 --2- 172.21.15.98:0/2595353382 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f813c0a2cb0 0x7f813c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:19.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.140+0000 7f8156458700 1 --2- 172.21.15.98:0/2595353382 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f813c0a2cb0 0x7f813c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:19.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.140+0000 7f81477fe700 1 -- 172.21.15.98:0/2595353382 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f814c0cf480 con 0x7f8150101bd0 2024-05-09T08:49:19.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.141+0000 7f81477fe700 1 --2- 172.21.15.98:0/2595353382 >> [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789] conn(0x7f813c0ade10 0x7f813c0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:19.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.141+0000 7f81477fe700 1 -- 172.21.15.98:0/2595353382 --> [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f814c01be50 con 0x7f813c0ade10 2024-05-09T08:49:19.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.141+0000 7f81477fe700 1 -- 172.21.15.98:0/2595353382 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f814c0d6050 con 0x7f8150101bd0 2024-05-09T08:49:19.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.141+0000 7f8156458700 1 --2- 172.21.15.98:0/2595353382 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f813c0a2cb0 0x7f813c0a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f814000b5c0 tx=0x7f8140005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:19.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.141+0000 7f8156c59700 1 --2- 172.21.15.98:0/2595353382 >> [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789] conn(0x7f813c0ade10 0x7f813c0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:19.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.141+0000 7f8156c59700 1 --2- 172.21.15.98:0/2595353382 >> [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789] conn(0x7f813c0ade10 0x7f813c0b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:19.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.142+0000 7f81477fe700 1 -- 172.21.15.98:0/2595353382 <== osd.5 v2:172.21.15.142:6808/1058034789 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f814c01be50 con 0x7f813c0ade10 2024-05-09T08:49:19.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.159+0000 7f81586bc700 1 -- 172.21.15.98:0/2595353382 --> [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8150102930 con 0x7f813c0ade10 2024-05-09T08:49:19.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.159+0000 7f81477fe700 1 -- 172.21.15.98:0/2595353382 <== osd.5 v2:172.21.15.142:6808/1058034789 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8150102930 con 0x7f813c0ade10 2024-05-09T08:49:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.159+0000 7f81586bc700 1 -- 172.21.15.98:0/2595353382 >> [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789] conn(0x7f813c0ade10 msgr2=0x7f813c0b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.159+0000 7f81586bc700 1 --2- 172.21.15.98:0/2595353382 >> [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789] conn(0x7f813c0ade10 0x7f813c0b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.159+0000 7f81586bc700 1 -- 172.21.15.98:0/2595353382 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f813c0a2cb0 msgr2=0x7f813c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.159+0000 7f81586bc700 1 --2- 172.21.15.98:0/2595353382 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f813c0a2cb0 0x7f813c0a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f814000b5c0 tx=0x7f8140005960 comp rx=0 tx=0).stop 2024-05-09T08:49:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.159+0000 7f81586bc700 1 -- 172.21.15.98:0/2595353382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8150101bd0 msgr2=0x7f815019ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.159+0000 7f81586bc700 1 --2- 172.21.15.98:0/2595353382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8150101bd0 0x7f815019ec50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f814c00c9b0 tx=0x7f814c00ccc0 comp rx=0 tx=0).stop 2024-05-09T08:49:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.159+0000 7f8156c59700 1 -- 172.21.15.98:0/2595353382 reap_dead start 2024-05-09T08:49:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.160+0000 7f81586bc700 1 -- 172.21.15.98:0/2595353382 shutdown_connections 2024-05-09T08:49:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.160+0000 7f81586bc700 1 -- 172.21.15.98:0/2595353382 >> 172.21.15.98:0/2595353382 conn(0x7f8150078190 msgr2=0x7f81501054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.160+0000 7f81586bc700 1 -- 172.21.15.98:0/2595353382 shutdown_connections 2024-05-09T08:49:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:19.160+0000 7f81586bc700 1 -- 172.21.15.98:0/2595353382 wait complete. 2024-05-09T08:49:19.393 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:19 smithi098 ceph-mon[30092]: osdmap e92: 12 total, 12 up, 12 in 2024-05-09T08:49:19.393 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:19 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:19.393 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:19 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:19.393 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:19 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:19.393 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:19 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:19.394 INFO:teuthology.orchestra.run.smithi098.stdout:201863462961 2024-05-09T08:49:19.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-09T08:49:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:19 smithi142 ceph-mon[34625]: osdmap e92: 12 total, 12 up, 12 in 2024-05-09T08:49:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:19 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:19 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:19 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:19 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:19.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:19 smithi105 ceph-mon[34498]: osdmap e92: 12 total, 12 up, 12 in 2024-05-09T08:49:19.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:19 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:19.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:19 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:19.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:19 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:19.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:19 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:20.010 INFO:teuthology.orchestra.run.smithi098.stdout:55834574924 2024-05-09T08:49:20.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-09T08:49:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:20 smithi142 ceph-mon[34625]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-05-09T08:49:20.662 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:20 smithi098 ceph-mon[30092]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-05-09T08:49:20.690 INFO:teuthology.orchestra.run.smithi098.stdout:313532612639 2024-05-09T08:49:20.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-09T08:49:20.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:20 smithi105 ceph-mon[34498]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-05-09T08:49:21.419 INFO:teuthology.orchestra.run.smithi098.stdout:167503724600 2024-05-09T08:49:21.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-09T08:49:21.458 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:21.472 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:21.653 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:21.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.681+0000 7ff906c9f700 1 -- 172.21.15.98:0/1721474273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff900106990 msgr2=0x7ff900106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:21.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.681+0000 7ff906c9f700 1 --2- 172.21.15.98:0/1721474273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff900106990 0x7ff900106d70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff8f400aaa0 tx=0x7ff8f400adb0 comp rx=0 tx=0).stop 2024-05-09T08:49:21.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.681+0000 7ff906c9f700 1 -- 172.21.15.98:0/1721474273 shutdown_connections 2024-05-09T08:49:21.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.681+0000 7ff906c9f700 1 --2- 172.21.15.98:0/1721474273 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9001013a0 0x7ff90010d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:21.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.681+0000 7ff906c9f700 1 --2- 172.21.15.98:0/1721474273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff900106990 0x7ff900106d70 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:21.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.681+0000 7ff906c9f700 1 --2- 172.21.15.98:0/1721474273 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9001009e0 0x7ff900100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:21.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.681+0000 7ff906c9f700 1 -- 172.21.15.98:0/1721474273 >> 172.21.15.98:0/1721474273 conn(0x7ff900077890 msgr2=0x7ff9000fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:21.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.682+0000 7ff906c9f700 1 -- 172.21.15.98:0/1721474273 shutdown_connections 2024-05-09T08:49:21.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.682+0000 7ff906c9f700 1 -- 172.21.15.98:0/1721474273 wait complete. 2024-05-09T08:49:21.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.682+0000 7ff906c9f700 1 Processor -- start 2024-05-09T08:49:21.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff906c9f700 1 -- start start 2024-05-09T08:49:21.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff906c9f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9001009e0 0x7ff900198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:21.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff906c9f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9001013a0 0x7ff9001992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:21.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff906c9f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff90019d630 0x7ff90019dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:21.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff906c9f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff900110490 con 0x7ff9001009e0 2024-05-09T08:49:21.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff906c9f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff900110310 con 0x7ff9001013a0 2024-05-09T08:49:21.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff906c9f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff900110610 con 0x7ff90019d630 2024-05-09T08:49:21.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff904a3b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9001009e0 0x7ff900198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:21.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff904a3b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9001009e0 0x7ff900198d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43880/0 (socket says 172.21.15.98:43880) 2024-05-09T08:49:21.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff904a3b700 1 -- 172.21.15.98:0/3673966255 learned_addr learned my addr 172.21.15.98:0/3673966255 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:21.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff8fffff700 1 --2- 172.21.15.98:0/3673966255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9001013a0 0x7ff9001992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:21.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff90523c700 1 --2- 172.21.15.98:0/3673966255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff90019d630 0x7ff90019dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:21.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff904a3b700 1 -- 172.21.15.98:0/3673966255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff90019d630 msgr2=0x7ff90019dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:21.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff904a3b700 1 --2- 172.21.15.98:0/3673966255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff90019d630 0x7ff90019dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:21.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff904a3b700 1 -- 172.21.15.98:0/3673966255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9001013a0 msgr2=0x7ff9001992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:21.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff904a3b700 1 --2- 172.21.15.98:0/3673966255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9001013a0 0x7ff9001992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:21.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.683+0000 7ff904a3b700 1 -- 172.21.15.98:0/3673966255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff90019e150 con 0x7ff9001009e0 2024-05-09T08:49:21.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.684+0000 7ff904a3b700 1 --2- 172.21.15.98:0/3673966255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9001009e0 0x7ff900198d60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4000c00 tx=0x7ff8f4003930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:21.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.684+0000 7ff8fdffb700 1 -- 172.21.15.98:0/3673966255 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f4003dc0 con 0x7ff9001009e0 2024-05-09T08:49:21.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.684+0000 7ff906c9f700 1 -- 172.21.15.98:0/3673966255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff90019e3e0 con 0x7ff9001009e0 2024-05-09T08:49:21.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.684+0000 7ff8fdffb700 1 -- 172.21.15.98:0/3673966255 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8f4024070 con 0x7ff9001009e0 2024-05-09T08:49:21.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.684+0000 7ff8fdffb700 1 -- 172.21.15.98:0/3673966255 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f4016420 con 0x7ff9001009e0 2024-05-09T08:49:21.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.684+0000 7ff906c9f700 1 -- 172.21.15.98:0/3673966255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff900110e30 con 0x7ff9001009e0 2024-05-09T08:49:21.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.685+0000 7ff906c9f700 1 -- 172.21.15.98:0/3673966255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff900062ab0 con 0x7ff9001009e0 2024-05-09T08:49:21.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.687+0000 7ff8fdffb700 1 -- 172.21.15.98:0/3673966255 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff8f401e070 con 0x7ff9001009e0 2024-05-09T08:49:21.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.687+0000 7ff8fdffb700 1 --2- 172.21.15.98:0/3673966255 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff8f00a2cb0 0x7ff8f00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:21.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.687+0000 7ff8fdffb700 1 -- 172.21.15.98:0/3673966255 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7ff8f40d0c50 con 0x7ff9001009e0 2024-05-09T08:49:21.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.688+0000 7ff8fffff700 1 --2- 172.21.15.98:0/3673966255 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff8f00a2cb0 0x7ff8f00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:21.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.688+0000 7ff8fdffb700 1 --2- 172.21.15.98:0/3673966255 >> [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901] conn(0x7ff8f00ade10 0x7ff8f00b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:21.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.688+0000 7ff8fdffb700 1 -- 172.21.15.98:0/3673966255 --> [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8f400c1c0 con 0x7ff8f00ade10 2024-05-09T08:49:21.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.688+0000 7ff8fdffb700 1 -- 172.21.15.98:0/3673966255 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7ff8f40d10a0 con 0x7ff9001009e0 2024-05-09T08:49:21.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.688+0000 7ff8fffff700 1 --2- 172.21.15.98:0/3673966255 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff8f00a2cb0 0x7ff8f00a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff9000fffc0 tx=0x7ff8ec00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:21.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.688+0000 7ff90523c700 1 --2- 172.21.15.98:0/3673966255 >> [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901] conn(0x7ff8f00ade10 0x7ff8f00b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:21.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.688+0000 7ff90523c700 1 --2- 172.21.15.98:0/3673966255 >> [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901] conn(0x7ff8f00ade10 0x7ff8f00b02b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:21.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.689+0000 7ff8fdffb700 1 -- 172.21.15.98:0/3673966255 <== osd.8 v2:172.21.15.142:6816/1607245901 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8f400c1c0 con 0x7ff8f00ade10 2024-05-09T08:49:21.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.705+0000 7ff906c9f700 1 -- 172.21.15.98:0/3673966255 --> [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff900062f30 con 0x7ff8f00ade10 2024-05-09T08:49:21.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.706+0000 7ff8fdffb700 1 -- 172.21.15.98:0/3673966255 <== osd.8 v2:172.21.15.142:6816/1607245901 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff900062f30 con 0x7ff8f00ade10 2024-05-09T08:49:21.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.706+0000 7ff906c9f700 1 -- 172.21.15.98:0/3673966255 >> [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901] conn(0x7ff8f00ade10 msgr2=0x7ff8f00b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:21.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.706+0000 7ff906c9f700 1 --2- 172.21.15.98:0/3673966255 >> [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901] conn(0x7ff8f00ade10 0x7ff8f00b02b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:21.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.707+0000 7ff906c9f700 1 -- 172.21.15.98:0/3673966255 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff8f00a2cb0 msgr2=0x7ff8f00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:21.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.707+0000 7ff906c9f700 1 --2- 172.21.15.98:0/3673966255 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff8f00a2cb0 0x7ff8f00a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff9000fffc0 tx=0x7ff8ec00b410 comp rx=0 tx=0).stop 2024-05-09T08:49:21.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.707+0000 7ff906c9f700 1 -- 172.21.15.98:0/3673966255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9001009e0 msgr2=0x7ff900198d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:21.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.707+0000 7ff906c9f700 1 --2- 172.21.15.98:0/3673966255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9001009e0 0x7ff900198d60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4000c00 tx=0x7ff8f4003930 comp rx=0 tx=0).stop 2024-05-09T08:49:21.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.707+0000 7ff90523c700 1 -- 172.21.15.98:0/3673966255 reap_dead start 2024-05-09T08:49:21.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.707+0000 7ff906c9f700 1 -- 172.21.15.98:0/3673966255 shutdown_connections 2024-05-09T08:49:21.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.707+0000 7ff906c9f700 1 -- 172.21.15.98:0/3673966255 >> 172.21.15.98:0/3673966255 conn(0x7ff900077890 msgr2=0x7ff9001039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:21.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.707+0000 7ff906c9f700 1 -- 172.21.15.98:0/3673966255 shutdown_connections 2024-05-09T08:49:21.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:21.707+0000 7ff906c9f700 1 -- 172.21.15.98:0/3673966255 wait complete. 2024-05-09T08:49:22.107 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:22.108 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:22.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:22 smithi098 ceph-mon[30092]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s; 14 B/s, 0 objects/s recovering 2024-05-09T08:49:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:22 smithi142 ceph-mon[34625]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s; 14 B/s, 0 objects/s recovering 2024-05-09T08:49:22.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:22 smithi105 ceph-mon[34498]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s; 14 B/s, 0 objects/s recovering 2024-05-09T08:49:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:24 smithi142 ceph-mon[34625]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s; 14 B/s, 0 objects/s recovering 2024-05-09T08:49:24.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:24 smithi105 ceph-mon[34498]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s; 14 B/s, 0 objects/s recovering 2024-05-09T08:49:24.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:24 smithi098 ceph-mon[30092]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s; 14 B/s, 0 objects/s recovering 2024-05-09T08:49:26.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:26 smithi098 ceph-mon[30092]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 11 op/s; 14 B/s, 0 objects/s recovering 2024-05-09T08:49:26.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:26 smithi142 ceph-mon[34625]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 11 op/s; 14 B/s, 0 objects/s recovering 2024-05-09T08:49:26.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:26 smithi105 ceph-mon[34498]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 11 op/s; 14 B/s, 0 objects/s recovering 2024-05-09T08:49:28.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.497+0000 7fcff76c8700 1 -- 172.21.15.98:0/1482746824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff0106990 msgr2=0x7fcff0106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:28.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.497+0000 7fcff76c8700 1 --2- 172.21.15.98:0/1482746824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff0106990 0x7fcff0106d70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcfec00b0a0 tx=0x7fcfec00b3b0 comp rx=0 tx=0).stop 2024-05-09T08:49:28.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.497+0000 7fcff76c8700 1 -- 172.21.15.98:0/1482746824 shutdown_connections 2024-05-09T08:49:28.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.497+0000 7fcff76c8700 1 --2- 172.21.15.98:0/1482746824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcff01013a0 0x7fcff010d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:28.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.497+0000 7fcff76c8700 1 --2- 172.21.15.98:0/1482746824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff0106990 0x7fcff0106d70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:28.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.497+0000 7fcff76c8700 1 --2- 172.21.15.98:0/1482746824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcff01009e0 0x7fcff0100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:28.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.497+0000 7fcff76c8700 1 -- 172.21.15.98:0/1482746824 >> 172.21.15.98:0/1482746824 conn(0x7fcff0077890 msgr2=0x7fcff00fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:28.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.497+0000 7fcff76c8700 1 -- 172.21.15.98:0/1482746824 shutdown_connections 2024-05-09T08:49:28.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.497+0000 7fcff76c8700 1 -- 172.21.15.98:0/1482746824 wait complete. 2024-05-09T08:49:28.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.498+0000 7fcff76c8700 1 Processor -- start 2024-05-09T08:49:28.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.498+0000 7fcff76c8700 1 -- start start 2024-05-09T08:49:28.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff76c8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcff01009e0 0x7fcff019e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:28.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff76c8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcff01013a0 0x7fcff019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:28.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff76c8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff0106990 0x7fcff0198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:28.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff76c8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcff0110450 con 0x7fcff0106990 2024-05-09T08:49:28.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff76c8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcff01102d0 con 0x7fcff01009e0 2024-05-09T08:49:28.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff76c8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcff01105d0 con 0x7fcff01013a0 2024-05-09T08:49:28.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff5c65700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff0106990 0x7fcff0198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:28.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff5c65700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff0106990 0x7fcff0198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52222/0 (socket says 172.21.15.98:52222) 2024-05-09T08:49:28.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff5c65700 1 -- 172.21.15.98:0/2445153697 learned_addr learned my addr 172.21.15.98:0/2445153697 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:28.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff5464700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcff01009e0 0x7fcff019e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:28.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff4c63700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcff01013a0 0x7fcff019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:28.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff5c65700 1 -- 172.21.15.98:0/2445153697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcff01013a0 msgr2=0x7fcff019ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:28.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff5c65700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcff01013a0 0x7fcff019ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:28.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff5c65700 1 -- 172.21.15.98:0/2445153697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcff01009e0 msgr2=0x7fcff019e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:28.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff5c65700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcff01009e0 0x7fcff019e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:28.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff5c65700 1 -- 172.21.15.98:0/2445153697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff01990c0 con 0x7fcff0106990 2024-05-09T08:49:28.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff4c63700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcff01013a0 0x7fcff019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:49:28.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff5c65700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff0106990 0x7fcff0198960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800d980 tx=0x7fcfe800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:28.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcfe67fc700 1 -- 172.21.15.98:0/2445153697 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8017070 con 0x7fcff0106990 2024-05-09T08:49:28.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff76c8700 1 -- 172.21.15.98:0/2445153697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff01993b0 con 0x7fcff0106990 2024-05-09T08:49:28.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcfe67fc700 1 -- 172.21.15.98:0/2445153697 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfe8005020 con 0x7fcff0106990 2024-05-09T08:49:28.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcfe67fc700 1 -- 172.21.15.98:0/2445153697 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8009b50 con 0x7fcff0106990 2024-05-09T08:49:28.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.499+0000 7fcff76c8700 1 -- 172.21.15.98:0/2445153697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff01a55d0 con 0x7fcff0106990 2024-05-09T08:49:28.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.500+0000 7fcff76c8700 1 -- 172.21.15.98:0/2445153697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcff0069010 con 0x7fcff0106990 2024-05-09T08:49:28.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.502+0000 7fcfe67fc700 1 -- 172.21.15.98:0/2445153697 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fcfe8009cf0 con 0x7fcff0106990 2024-05-09T08:49:28.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.502+0000 7fcfe67fc700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcfdc0a2cb0 0x7fcfdc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:28.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.502+0000 7fcff5464700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcfdc0a2cb0 0x7fcfdc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:28.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.502+0000 7fcfe67fc700 1 -- 172.21.15.98:0/2445153697 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fcfe80cf680 con 0x7fcff0106990 2024-05-09T08:49:28.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.503+0000 7fcff5464700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcfdc0a2cb0 0x7fcfdc0a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009e60 tx=0x7fcfec00b700 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:28.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.506+0000 7fcfe67fc700 1 -- 172.21.15.98:0/2445153697 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcfe8020020 con 0x7fcff0106990 2024-05-09T08:49:28.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:28 smithi142 ceph-mon[34625]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 11 B/s, 0 objects/s recovering 2024-05-09T08:49:28.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.705+0000 7fcff76c8700 1 -- 172.21.15.98:0/2445153697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fcff0199e90 con 0x7fcff0106990 2024-05-09T08:49:28.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:28 smithi098 ceph-mon[30092]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 11 B/s, 0 objects/s recovering 2024-05-09T08:49:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.706+0000 7fcfe67fc700 1 -- 172.21.15.98:0/2445153697 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fcfe809c120 con 0x7fcff0106990 2024-05-09T08:49:28.706 INFO:teuthology.orchestra.run.smithi098.stdout:347892351003 2024-05-09T08:49:28.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.708+0000 7fcff76c8700 1 -- 172.21.15.98:0/2445153697 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcfdc0a2cb0 msgr2=0x7fcfdc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:28.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.708+0000 7fcff76c8700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcfdc0a2cb0 0x7fcfdc0a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009e60 tx=0x7fcfec00b700 comp rx=0 tx=0).stop 2024-05-09T08:49:28.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.708+0000 7fcff76c8700 1 -- 172.21.15.98:0/2445153697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff0106990 msgr2=0x7fcff0198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:28.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.708+0000 7fcff76c8700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff0106990 0x7fcff0198960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800d980 tx=0x7fcfe800dc90 comp rx=0 tx=0).stop 2024-05-09T08:49:28.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.708+0000 7fcff76c8700 1 -- 172.21.15.98:0/2445153697 shutdown_connections 2024-05-09T08:49:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.708+0000 7fcff76c8700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcfdc0a2cb0 0x7fcfdc0a5170 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.708+0000 7fcff76c8700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcff01009e0 0x7fcff019e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.708+0000 7fcff76c8700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff0106990 0x7fcff0198960 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.708+0000 7fcff76c8700 1 --2- 172.21.15.98:0/2445153697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcff01013a0 0x7fcff019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.708+0000 7fcff76c8700 1 -- 172.21.15.98:0/2445153697 >> 172.21.15.98:0/2445153697 conn(0x7fcff0077890 msgr2=0x7fcff0103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.708+0000 7fcff76c8700 1 -- 172.21.15.98:0/2445153697 shutdown_connections 2024-05-09T08:49:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:28.708+0000 7fcff76c8700 1 -- 172.21.15.98:0/2445153697 wait complete. 2024-05-09T08:49:28.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:28 smithi105 ceph-mon[34498]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 11 B/s, 0 objects/s recovering 2024-05-09T08:49:28.914 INFO:teuthology.orchestra.run.smithi098.stdout:257698037802 2024-05-09T08:49:28.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-09T08:49:29.554 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:29 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2445153697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-09T08:49:29.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:29 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2445153697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-09T08:49:29.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:29 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2445153697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-09T08:49:29.900 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.599+0000 7fcef6cc2700 1 -- 172.21.15.98:0/996311299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01025d0 msgr2=0x7fcef01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.599+0000 7fcef6cc2700 1 --2- 172.21.15.98:0/996311299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01025d0 0x7fcef01029b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcee4009990 tx=0x7fcee4009ca0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.599+0000 7fcef6cc2700 1 -- 172.21.15.98:0/996311299 shutdown_connections 2024-05-09T08:49:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.599+0000 7fcef6cc2700 1 --2- 172.21.15.98:0/996311299 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcef0107660 0x7fcef010f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.599+0000 7fcef6cc2700 1 --2- 172.21.15.98:0/996311299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01025d0 0x7fcef01029b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.599+0000 7fcef6cc2700 1 --2- 172.21.15.98:0/996311299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcef0102f80 0x7fcef0106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.599+0000 7fcef6cc2700 1 -- 172.21.15.98:0/996311299 >> 172.21.15.98:0/996311299 conn(0x7fcef0079480 msgr2=0x7fcef0079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.599+0000 7fcef6cc2700 1 -- 172.21.15.98:0/996311299 shutdown_connections 2024-05-09T08:49:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.599+0000 7fcef6cc2700 1 -- 172.21.15.98:0/996311299 wait complete. 2024-05-09T08:49:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.599+0000 7fcef6cc2700 1 Processor -- start 2024-05-09T08:49:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.599+0000 7fcef6cc2700 1 -- start start 2024-05-09T08:49:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef6cc2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01025d0 0x7fcef019e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef6cc2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcef0102f80 0x7fcef019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef6cc2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcef0107660 0x7fcef0198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef6cc2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcef0112620 con 0x7fcef01025d0 2024-05-09T08:49:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef6cc2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcef01124a0 con 0x7fcef0107660 2024-05-09T08:49:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef6cc2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcef01127a0 con 0x7fcef0102f80 2024-05-09T08:49:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef4a5e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01025d0 0x7fcef019e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef4a5e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01025d0 0x7fcef019e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52250/0 (socket says 172.21.15.98:52250) 2024-05-09T08:49:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef4a5e700 1 -- 172.21.15.98:0/1093010292 learned_addr learned my addr 172.21.15.98:0/1093010292 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef525f700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcef0107660 0x7fcef0198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fceeffff700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcef0102f80 0x7fcef019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef4a5e700 1 -- 172.21.15.98:0/1093010292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcef0102f80 msgr2=0x7fcef019ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef4a5e700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcef0102f80 0x7fcef019ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef4a5e700 1 -- 172.21.15.98:0/1093010292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcef0107660 msgr2=0x7fcef0198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef4a5e700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcef0107660 0x7fcef0198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef4a5e700 1 -- 172.21.15.98:0/1093010292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef0199060 con 0x7fcef01025d0 2024-05-09T08:49:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef525f700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcef0107660 0x7fcef0198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:49:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fceeffff700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcef0102f80 0x7fcef019ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:49:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fcef4a5e700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01025d0 0x7fcef019e6f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fcee400bae0 tx=0x7fcee400bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fceedffb700 1 -- 172.21.15.98:0/1093010292 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee4026070 con 0x7fcef01025d0 2024-05-09T08:49:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fceedffb700 1 -- 172.21.15.98:0/1093010292 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcee400f460 con 0x7fcef01025d0 2024-05-09T08:49:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.600+0000 7fceedffb700 1 -- 172.21.15.98:0/1093010292 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee4021670 con 0x7fcef01025d0 2024-05-09T08:49:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.601+0000 7fcef6cc2700 1 -- 172.21.15.98:0/1093010292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef01992f0 con 0x7fcef01025d0 2024-05-09T08:49:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.601+0000 7fcef6cc2700 1 -- 172.21.15.98:0/1093010292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef01a55c0 con 0x7fcef01025d0 2024-05-09T08:49:30.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.602+0000 7fcef6cc2700 1 -- 172.21.15.98:0/1093010292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef0069010 con 0x7fcef01025d0 2024-05-09T08:49:30.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.603+0000 7fceedffb700 1 -- 172.21.15.98:0/1093010292 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fcee4009d10 con 0x7fcef01025d0 2024-05-09T08:49:30.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.604+0000 7fceedffb700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcee00a2d30 0x7fcee00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.604+0000 7fceeffff700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcee00a2d30 0x7fcee00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.604+0000 7fceedffb700 1 -- 172.21.15.98:0/1093010292 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fcee40cff70 con 0x7fcef01025d0 2024-05-09T08:49:30.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.604+0000 7fceeffff700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcee00a2d30 0x7fcee00a51f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcef0108b90 tx=0x7fcedc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.608+0000 7fceedffb700 1 -- 172.21.15.98:0/1093010292 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcee405c740 con 0x7fcef01025d0 2024-05-09T08:49:30.638 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:30 smithi105 ceph-mon[34498]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-05-09T08:49:30.638 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:30 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3910145553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-09T08:49:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:30 smithi142 ceph-mon[34625]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-05-09T08:49:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:30 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3910145553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-09T08:49:30.716 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:30 smithi098 ceph-mon[30092]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-05-09T08:49:30.716 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:30 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3910145553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-09T08:49:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.816+0000 7fcef6cc2700 1 -- 172.21.15.98:0/1093010292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fcef002db80 con 0x7fcef01025d0 2024-05-09T08:49:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.817+0000 7fceedffb700 1 -- 172.21.15.98:0/1093010292 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fcee4013070 con 0x7fcef01025d0 2024-05-09T08:49:30.816 INFO:teuthology.orchestra.run.smithi098.stdout:292057776166 2024-05-09T08:49:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.819+0000 7fcef6cc2700 1 -- 172.21.15.98:0/1093010292 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcee00a2d30 msgr2=0x7fcee00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.819+0000 7fcef6cc2700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcee00a2d30 0x7fcee00a51f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcef0108b90 tx=0x7fcedc009260 comp rx=0 tx=0).stop 2024-05-09T08:49:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.819+0000 7fcef6cc2700 1 -- 172.21.15.98:0/1093010292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01025d0 msgr2=0x7fcef019e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.819+0000 7fcef6cc2700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01025d0 0x7fcef019e6f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fcee400bae0 tx=0x7fcee400bb10 comp rx=0 tx=0).stop 2024-05-09T08:49:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.819+0000 7fcef6cc2700 1 -- 172.21.15.98:0/1093010292 shutdown_connections 2024-05-09T08:49:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.819+0000 7fcef6cc2700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcee00a2d30 0x7fcee00a51f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.819+0000 7fcef6cc2700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcef0107660 0x7fcef0198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.819+0000 7fcef6cc2700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01025d0 0x7fcef019e6f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.819+0000 7fcef6cc2700 1 --2- 172.21.15.98:0/1093010292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcef0102f80 0x7fcef019ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.819+0000 7fcef6cc2700 1 -- 172.21.15.98:0/1093010292 >> 172.21.15.98:0/1093010292 conn(0x7fcef0079480 msgr2=0x7fcef0101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.819+0000 7fcef6cc2700 1 -- 172.21.15.98:0/1093010292 shutdown_connections 2024-05-09T08:49:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.819+0000 7fcef6cc2700 1 -- 172.21.15.98:0/1093010292 wait complete. 2024-05-09T08:49:31.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:31 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1093010292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-09T08:49:31.485 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776162 got 292057776166 for osd.9 2024-05-09T08:49:31.485 DEBUG:teuthology.parallel:result is None 2024-05-09T08:49:31.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:31 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1093010292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-09T08:49:31.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:31 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1093010292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-09T08:49:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.049+0000 7f32f7415700 1 -- 172.21.15.98:0/2370830824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32f0107270 msgr2=0x7f32f006aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.049+0000 7f32f7415700 1 --2- 172.21.15.98:0/2370830824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32f0107270 0x7f32f006aa90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009990 tx=0x7f32ec009ca0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.049+0000 7f32f7415700 1 -- 172.21.15.98:0/2370830824 shutdown_connections 2024-05-09T08:49:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.049+0000 7f32f7415700 1 --2- 172.21.15.98:0/2370830824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32f0104d90 0x7f32f0105210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.049+0000 7f32f7415700 1 --2- 172.21.15.98:0/2370830824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32f0107270 0x7f32f006aa90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.049+0000 7f32f7415700 1 --2- 172.21.15.98:0/2370830824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f006afd0 0x7f32f006b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.049+0000 7f32f7415700 1 -- 172.21.15.98:0/2370830824 >> 172.21.15.98:0/2370830824 conn(0x7f32f0077290 msgr2=0x7f32f00776a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.049+0000 7f32f7415700 1 -- 172.21.15.98:0/2370830824 shutdown_connections 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.049+0000 7f32f7415700 1 -- 172.21.15.98:0/2370830824 wait complete. 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.049+0000 7f32f7415700 1 Processor -- start 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.049+0000 7f32f7415700 1 -- start start 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f7415700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32f006afd0 0x7f32f019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f7415700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f0104d90 0x7f32f019f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f7415700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32f0107270 0x7f32f01a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f7415700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f32f0118bd0 con 0x7f32f0107270 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f7415700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f32f0118a50 con 0x7f32f006afd0 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f7415700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f32f0118d50 con 0x7f32f0104d90 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f59b2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32f0107270 0x7f32f01a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f59b2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32f0107270 0x7f32f01a3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52270/0 (socket says 172.21.15.98:52270) 2024-05-09T08:49:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f59b2700 1 -- 172.21.15.98:0/2743396098 learned_addr learned my addr 172.21.15.98:0/2743396098 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f51b1700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32f006afd0 0x7f32f019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f49b0700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f0104d90 0x7f32f019f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f59b2700 1 -- 172.21.15.98:0/2743396098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f0104d90 msgr2=0x7f32f019f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f59b2700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f0104d90 0x7f32f019f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f59b2700 1 -- 172.21.15.98:0/2743396098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32f006afd0 msgr2=0x7f32f019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f59b2700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32f006afd0 0x7f32f019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f59b2700 1 -- 172.21.15.98:0/2743396098 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f01a4070 con 0x7f32f0107270 2024-05-09T08:49:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f49b0700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f0104d90 0x7f32f019f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:49:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f59b2700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32f0107270 0x7f32f01a3930 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f32e8007f10 tx=0x7f32e800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32e67fc700 1 -- 172.21.15.98:0/2743396098 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e800dbd0 con 0x7f32f0107270 2024-05-09T08:49:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32f7415700 1 -- 172.21.15.98:0/2743396098 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32f01a4360 con 0x7f32f0107270 2024-05-09T08:49:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.050+0000 7f32e67fc700 1 -- 172.21.15.98:0/2743396098 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32e800dd70 con 0x7f32f0107270 2024-05-09T08:49:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.051+0000 7f32e67fc700 1 -- 172.21.15.98:0/2743396098 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e80133f0 con 0x7f32f0107270 2024-05-09T08:49:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.051+0000 7f32f7415700 1 -- 172.21.15.98:0/2743396098 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32f01abbe0 con 0x7f32f0107270 2024-05-09T08:49:32.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.051+0000 7f32f7415700 1 -- 172.21.15.98:0/2743396098 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32f0101870 con 0x7f32f0107270 2024-05-09T08:49:32.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.053+0000 7f32e67fc700 1 -- 172.21.15.98:0/2743396098 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f32e801a020 con 0x7f32f0107270 2024-05-09T08:49:32.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.053+0000 7f32e67fc700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32dc0a2cb0 0x7f32dc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.053+0000 7f32f51b1700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32dc0a2cb0 0x7f32dc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.053+0000 7f32e67fc700 1 -- 172.21.15.98:0/2743396098 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f32e80cec40 con 0x7f32f0107270 2024-05-09T08:49:32.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.054+0000 7f32f51b1700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32dc0a2cb0 0x7f32dc0a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00b5c0 tx=0x7f32ec005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:32.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.058+0000 7f32e67fc700 1 -- 172.21.15.98:0/2743396098 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f32e80d5080 con 0x7f32f0107270 2024-05-09T08:49:32.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.123+0000 7feea599b700 1 -- 172.21.15.98:0/2254472740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea0107290 msgr2=0x7feea0107670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.123+0000 7feea599b700 1 --2- 172.21.15.98:0/2254472740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea0107290 0x7feea0107670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fee9000b0a0 tx=0x7fee9000b3b0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.123+0000 7feea599b700 1 -- 172.21.15.98:0/2254472740 shutdown_connections 2024-05-09T08:49:32.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.123+0000 7feea599b700 1 --2- 172.21.15.98:0/2254472740 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feea0101be0 0x7feea010e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.123+0000 7feea599b700 1 --2- 172.21.15.98:0/2254472740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea0107290 0x7feea0107670 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.123+0000 7feea599b700 1 --2- 172.21.15.98:0/2254472740 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feea0101220 0x7feea01016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.123+0000 7feea599b700 1 -- 172.21.15.98:0/2254472740 >> 172.21.15.98:0/2254472740 conn(0x7feea00781d0 msgr2=0x7feea00785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:32.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.123+0000 7feea599b700 1 -- 172.21.15.98:0/2254472740 shutdown_connections 2024-05-09T08:49:32.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.123+0000 7feea599b700 1 -- 172.21.15.98:0/2254472740 wait complete. 2024-05-09T08:49:32.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.124+0000 7feea599b700 1 Processor -- start 2024-05-09T08:49:32.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.124+0000 7feea599b700 1 -- start start 2024-05-09T08:49:32.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.124+0000 7feea599b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feea0101220 0x7feea019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.124+0000 7feea599b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feea0101be0 0x7feea019ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.124+0000 7feea599b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea0107290 0x7feea0198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.124+0000 7feea599b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feea0111530 con 0x7feea0107290 2024-05-09T08:49:32.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.124+0000 7feea599b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feea01113b0 con 0x7feea0101be0 2024-05-09T08:49:32.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.124+0000 7feea599b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7feea01116b0 con 0x7feea0101220 2024-05-09T08:49:32.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.124+0000 7fee9f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea0107290 0x7feea0198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.124+0000 7fee9f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea0107290 0x7feea0198890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52240/0 (socket says 172.21.15.98:52240) 2024-05-09T08:49:32.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.124+0000 7fee9f7fe700 1 -- 172.21.15.98:0/3910145553 learned_addr learned my addr 172.21.15.98:0/3910145553 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:32.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.124+0000 7fee9e7fc700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feea0101be0 0x7feea019ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.125+0000 7fee9effd700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feea0101220 0x7feea019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.147 INFO:teuthology.orchestra.run.smithi098.stdout:77309411401 2024-05-09T08:49:32.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.125+0000 7fee9f7fe700 1 -- 172.21.15.98:0/3910145553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feea0101220 msgr2=0x7feea019e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.125+0000 7fee9f7fe700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feea0101220 0x7feea019e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.125+0000 7fee9f7fe700 1 -- 172.21.15.98:0/3910145553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feea0101be0 msgr2=0x7feea019ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.125+0000 7fee9f7fe700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feea0101be0 0x7feea019ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.125+0000 7fee9f7fe700 1 -- 172.21.15.98:0/3910145553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea0198ff0 con 0x7feea0107290 2024-05-09T08:49:32.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.125+0000 7fee9effd700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feea0101220 0x7feea019e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:49:32.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.125+0000 7fee9f7fe700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea0107290 0x7feea0198890 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fee9800c900 tx=0x7fee9800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:32.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.125+0000 7feea4999700 1 -- 172.21.15.98:0/3910145553 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee98007b00 con 0x7feea0107290 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.125+0000 7feea4999700 1 -- 172.21.15.98:0/3910145553 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee9800ce50 con 0x7feea0107290 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.125+0000 7feea599b700 1 -- 172.21.15.98:0/3910145553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feea01992e0 con 0x7feea0107290 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.125+0000 7feea4999700 1 -- 172.21.15.98:0/3910145553 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee98018660 con 0x7feea0107290 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.125+0000 7feea599b700 1 -- 172.21.15.98:0/3910145553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feea01a55a0 con 0x7feea0107290 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.126+0000 7feea599b700 1 -- 172.21.15.98:0/3910145553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feea0069010 con 0x7feea0107290 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.127+0000 7feea4999700 1 -- 172.21.15.98:0/3910145553 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fee9800f450 con 0x7feea0107290 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.128+0000 7feea4999700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fee8c0a2cb0 0x7fee8c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.128+0000 7fee9effd700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fee8c0a2cb0 0x7fee8c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.128+0000 7feea4999700 1 -- 172.21.15.98:0/3910145553 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fee980cf470 con 0x7feea0107290 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.128+0000 7fee9effd700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fee8c0a2cb0 0x7fee8c0a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fee9000b070 tx=0x7fee90009470 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.133+0000 7feea4999700 1 -- 172.21.15.98:0/3910145553 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fee98014030 con 0x7feea0107290 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.334+0000 7feea599b700 1 -- 172.21.15.98:0/3910145553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7feea01a57d0 con 0x7feea0107290 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.334+0000 7feea4999700 1 -- 172.21.15.98:0/3910145553 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fee9809bf10 con 0x7feea0107290 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.336+0000 7feea599b700 1 -- 172.21.15.98:0/3910145553 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fee8c0a2cb0 msgr2=0x7fee8c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.336+0000 7feea599b700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fee8c0a2cb0 0x7fee8c0a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fee9000b070 tx=0x7fee90009470 comp rx=0 tx=0).stop 2024-05-09T08:49:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.336+0000 7feea599b700 1 -- 172.21.15.98:0/3910145553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea0107290 msgr2=0x7feea0198890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.336+0000 7feea599b700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea0107290 0x7feea0198890 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fee9800c900 tx=0x7fee9800cc10 comp rx=0 tx=0).stop 2024-05-09T08:49:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.336+0000 7feea599b700 1 -- 172.21.15.98:0/3910145553 shutdown_connections 2024-05-09T08:49:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.336+0000 7feea599b700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fee8c0a2cb0 0x7fee8c0a5170 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.336+0000 7feea599b700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea0107290 0x7feea0198890 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.336+0000 7feea599b700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feea0101220 0x7feea019e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.336+0000 7feea599b700 1 --2- 172.21.15.98:0/3910145553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feea0101be0 0x7feea019ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.336+0000 7feea599b700 1 -- 172.21.15.98:0/3910145553 >> 172.21.15.98:0/3910145553 conn(0x7feea00781d0 msgr2=0x7feea0105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.336+0000 7feea599b700 1 -- 172.21.15.98:0/3910145553 shutdown_connections 2024-05-09T08:49:32.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:30.336+0000 7feea599b700 1 -- 172.21.15.98:0/3910145553 wait complete. 2024-05-09T08:49:32.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.257+0000 7f32f7415700 1 -- 172.21.15.98:0/2743396098 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f32f01abe10 con 0x7f32f0107270 2024-05-09T08:49:32.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.258+0000 7f32e67fc700 1 -- 172.21.15.98:0/2743396098 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f32e8010070 con 0x7f32f0107270 2024-05-09T08:49:32.257 INFO:teuthology.orchestra.run.smithi098.stdout:146028888128 2024-05-09T08:49:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.260+0000 7f32f7415700 1 -- 172.21.15.98:0/2743396098 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32dc0a2cb0 msgr2=0x7f32dc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.260+0000 7f32f7415700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32dc0a2cb0 0x7f32dc0a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00b5c0 tx=0x7f32ec005960 comp rx=0 tx=0).stop 2024-05-09T08:49:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.260+0000 7f32f7415700 1 -- 172.21.15.98:0/2743396098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32f0107270 msgr2=0x7f32f01a3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.260+0000 7f32f7415700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32f0107270 0x7f32f01a3930 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f32e8007f10 tx=0x7f32e800d3b0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.260+0000 7f32f7415700 1 -- 172.21.15.98:0/2743396098 shutdown_connections 2024-05-09T08:49:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.260+0000 7f32f7415700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32dc0a2cb0 0x7f32dc0a5170 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.260+0000 7f32f7415700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32f006afd0 0x7f32f019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.260+0000 7f32f7415700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32f0107270 0x7f32f01a3930 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.260+0000 7f32f7415700 1 --2- 172.21.15.98:0/2743396098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f0104d90 0x7f32f019f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.260+0000 7f32f7415700 1 -- 172.21.15.98:0/2743396098 >> 172.21.15.98:0/2743396098 conn(0x7f32f0077290 msgr2=0x7f32f0103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.260+0000 7f32f7415700 1 -- 172.21.15.98:0/2743396098 shutdown_connections 2024-05-09T08:49:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:32.260+0000 7f32f7415700 1 -- 172.21.15.98:0/2743396098 wait complete. 2024-05-09T08:49:32.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:32 smithi098 ceph-mon[30092]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-05-09T08:49:32.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:32 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2743396098' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-09T08:49:32.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:32 smithi142 ceph-mon[34625]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-05-09T08:49:32.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:32 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2743396098' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-09T08:49:32.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:32 smithi105 ceph-mon[34498]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-05-09T08:49:32.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:32 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2743396098' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-09T08:49:33.032 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888123 got 146028888128 for osd.4 2024-05-09T08:49:33.032 DEBUG:teuthology.parallel:result is None 2024-05-09T08:49:33.974 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411399 got 77309411401 for osd.2 2024-05-09T08:49:33.974 DEBUG:teuthology.parallel:result is None 2024-05-09T08:49:34.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:34 smithi142 ceph-mon[34625]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-05-09T08:49:34.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:34 smithi105 ceph-mon[34498]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-05-09T08:49:34.835 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:34 smithi098 ceph-mon[30092]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-05-09T08:49:35.369 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351000 got 347892351003 for osd.11 2024-05-09T08:49:35.369 DEBUG:teuthology.parallel:result is None 2024-05-09T08:49:35.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.620+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2661917166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0074e70 msgr2=0x7f0ab010e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.620+0000 7f0ab6e51700 1 --2- 172.21.15.98:0/2661917166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0074e70 0x7f0ab010e840 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000b0a0 tx=0x7f0aa000b3b0 comp rx=0 tx=0).stop 2024-05-09T08:49:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.620+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2661917166 shutdown_connections 2024-05-09T08:49:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.620+0000 7f0ab6e51700 1 --2- 172.21.15.98:0/2661917166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ab010ed80 0x7f0ab0111170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.620+0000 7f0ab6e51700 1 --2- 172.21.15.98:0/2661917166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ab0074550 0x7f0ab0074930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.620+0000 7f0ab6e51700 1 --2- 172.21.15.98:0/2661917166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0074e70 0x7f0ab010e840 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.620+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2661917166 >> 172.21.15.98:0/2661917166 conn(0x7f0ab00793b0 msgr2=0x7f0ab00797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.620+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2661917166 shutdown_connections 2024-05-09T08:49:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.620+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2661917166 wait complete. 2024-05-09T08:49:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.621+0000 7f0ab6e51700 1 Processor -- start 2024-05-09T08:49:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.621+0000 7f0ab6e51700 1 -- start start 2024-05-09T08:49:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.621+0000 7f0ab6e51700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0074550 0x7f0ab01a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.621+0000 7f0ab6e51700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ab0074e70 0x7f0ab01a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.621+0000 7f0ab6e51700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ab010ed80 0x7f0ab019aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.621+0000 7f0ab6e51700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab0114830 con 0x7f0ab0074550 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.621+0000 7f0ab6e51700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab01146b0 con 0x7f0ab010ed80 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.621+0000 7f0ab6e51700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab01149b0 con 0x7f0ab0074e70 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.621+0000 7f0ab4bed700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0074550 0x7f0ab01a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.621+0000 7f0ab4bed700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0074550 0x7f0ab01a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45746/0 (socket says 172.21.15.98:45746) 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.621+0000 7f0ab4bed700 1 -- 172.21.15.98:0/2144855988 learned_addr learned my addr 172.21.15.98:0/2144855988 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.621+0000 7f0aaffff700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ab0074e70 0x7f0ab01a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.621+0000 7f0ab53ee700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ab010ed80 0x7f0ab019aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.622+0000 7f0ab4bed700 1 -- 172.21.15.98:0/2144855988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ab0074e70 msgr2=0x7f0ab01a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.622+0000 7f0ab4bed700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ab0074e70 0x7f0ab01a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.622+0000 7f0ab4bed700 1 -- 172.21.15.98:0/2144855988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ab010ed80 msgr2=0x7f0ab019aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.622+0000 7f0ab4bed700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ab010ed80 0x7f0ab019aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.622+0000 7f0ab4bed700 1 -- 172.21.15.98:0/2144855988 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ab019b210 con 0x7f0ab0074550 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.622+0000 7f0aaffff700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ab0074e70 0x7f0ab01a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:49:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.622+0000 7f0ab4bed700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0074550 0x7f0ab01a08f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800d950 tx=0x7f0aa800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.622+0000 7f0aadffb700 1 -- 172.21.15.98:0/2144855988 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8016070 con 0x7f0ab0074550 2024-05-09T08:49:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.622+0000 7f0aadffb700 1 -- 172.21.15.98:0/2144855988 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0aa800fd70 con 0x7f0ab0074550 2024-05-09T08:49:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.622+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2144855988 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ab019b500 con 0x7f0ab0074550 2024-05-09T08:49:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.622+0000 7f0aadffb700 1 -- 172.21.15.98:0/2144855988 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa800e610 con 0x7f0ab0074550 2024-05-09T08:49:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.622+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2144855988 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ab01a77c0 con 0x7f0ab0074550 2024-05-09T08:49:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.624+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2144855988 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ab0069010 con 0x7f0ab0074550 2024-05-09T08:49:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.625+0000 7f0aadffb700 1 -- 172.21.15.98:0/2144855988 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0aa8019070 con 0x7f0ab0074550 2024-05-09T08:49:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.626+0000 7f0aadffb700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f0a9c0a2d30 0x7f0a9c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.626+0000 7f0aadffb700 1 -- 172.21.15.98:0/2144855988 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f0aa80cef60 con 0x7f0ab0074550 2024-05-09T08:49:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.626+0000 7f0aaffff700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f0a9c0a2d30 0x7f0a9c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:35.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.627+0000 7f0aaffff700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f0a9c0a2d30 0x7f0a9c0a51f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0110940 tx=0x7f0aa000b7b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.632+0000 7f0aadffb700 1 -- 172.21.15.98:0/2144855988 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0aa809ba00 con 0x7f0ab0074550 2024-05-09T08:49:35.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.847+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2144855988 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0ab01a79f0 con 0x7f0ab0074550 2024-05-09T08:49:35.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.848+0000 7f0aadffb700 1 -- 172.21.15.98:0/2144855988 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0aa809b300 con 0x7f0ab0074550 2024-05-09T08:49:35.847 INFO:teuthology.orchestra.run.smithi098.stdout:34359738451 2024-05-09T08:49:35.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.850+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2144855988 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f0a9c0a2d30 msgr2=0x7f0a9c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:35.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.850+0000 7f0ab6e51700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f0a9c0a2d30 0x7f0a9c0a51f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0110940 tx=0x7f0aa000b7b0 comp rx=0 tx=0).stop 2024-05-09T08:49:35.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.850+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2144855988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0074550 msgr2=0x7f0ab01a08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:35.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.850+0000 7f0ab6e51700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0074550 0x7f0ab01a08f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800d950 tx=0x7f0aa800dc60 comp rx=0 tx=0).stop 2024-05-09T08:49:35.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.850+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2144855988 shutdown_connections 2024-05-09T08:49:35.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.850+0000 7f0ab6e51700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f0a9c0a2d30 0x7f0a9c0a51f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:35.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.850+0000 7f0ab6e51700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ab010ed80 0x7f0ab019aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:35.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.850+0000 7f0ab6e51700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0074550 0x7f0ab01a08f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:35.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.850+0000 7f0ab6e51700 1 --2- 172.21.15.98:0/2144855988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ab0074e70 0x7f0ab01a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:35.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.850+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2144855988 >> 172.21.15.98:0/2144855988 conn(0x7f0ab00793b0 msgr2=0x7f0ab00ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:35.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.850+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2144855988 shutdown_connections 2024-05-09T08:49:35.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:35.850+0000 7f0ab6e51700 1 -- 172.21.15.98:0/2144855988 wait complete. 2024-05-09T08:49:36.296 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:36.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:36 smithi098 ceph-mon[30092]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-05-09T08:49:36.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:36 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2144855988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-09T08:49:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:36 smithi142 ceph-mon[34625]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-05-09T08:49:36.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:36 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2144855988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-09T08:49:36.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:36 smithi105 ceph-mon[34498]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-05-09T08:49:36.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:36 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2144855988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-09T08:49:36.738 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738446 got 34359738451 for osd.0 2024-05-09T08:49:36.739 DEBUG:teuthology.parallel:result is None 2024-05-09T08:49:38.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:38 smithi142 ceph-mon[34625]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:49:38.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:38 smithi105 ceph-mon[34498]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:49:38.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:38 smithi098 ceph-mon[30092]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:49:39.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:39 smithi142 ceph-mon[34625]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-09T08:49:39.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:39 smithi098 ceph-mon[30092]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-09T08:49:39.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:39 smithi105 ceph-mon[34498]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-09T08:49:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.895+0000 7fdb4b626700 1 -- 172.21.15.98:0/3002332698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb441069c0 msgr2=0x7fdb44106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.895+0000 7fdb4b626700 1 --2- 172.21.15.98:0/3002332698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb441069c0 0x7fdb44106da0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00c210 tx=0x7fdb3c00c520 comp rx=0 tx=0).stop 2024-05-09T08:49:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.895+0000 7fdb4b626700 1 -- 172.21.15.98:0/3002332698 shutdown_connections 2024-05-09T08:49:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.895+0000 7fdb4b626700 1 --2- 172.21.15.98:0/3002332698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb44101380 0x7fdb4410d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.895+0000 7fdb4b626700 1 --2- 172.21.15.98:0/3002332698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb441069c0 0x7fdb44106da0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.895+0000 7fdb4b626700 1 --2- 172.21.15.98:0/3002332698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb441009c0 0x7fdb44100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.895+0000 7fdb4b626700 1 -- 172.21.15.98:0/3002332698 >> 172.21.15.98:0/3002332698 conn(0x7fdb44077820 msgr2=0x7fdb440feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.895+0000 7fdb4b626700 1 -- 172.21.15.98:0/3002332698 shutdown_connections 2024-05-09T08:49:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.895+0000 7fdb4b626700 1 -- 172.21.15.98:0/3002332698 wait complete. 2024-05-09T08:49:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.896+0000 7fdb4b626700 1 Processor -- start 2024-05-09T08:49:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.896+0000 7fdb4b626700 1 -- start start 2024-05-09T08:49:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.896+0000 7fdb4b626700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb441009c0 0x7fdb441966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.896+0000 7fdb4b626700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb44101380 0x7fdb44196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:39.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.896+0000 7fdb4b626700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb441069c0 0x7fdb44197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:39.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.896+0000 7fdb4b626700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb44110490 con 0x7fdb44101380 2024-05-09T08:49:39.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.896+0000 7fdb4b626700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdb44110310 con 0x7fdb441009c0 2024-05-09T08:49:39.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.896+0000 7fdb4b626700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdb44110610 con 0x7fdb441069c0 2024-05-09T08:49:39.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.896+0000 7fdb48bc1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb44101380 0x7fdb44196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:39.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.896+0000 7fdb48bc1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb44101380 0x7fdb44196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45762/0 (socket says 172.21.15.98:45762) 2024-05-09T08:49:39.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.896+0000 7fdb48bc1700 1 -- 172.21.15.98:0/3966280396 learned_addr learned my addr 172.21.15.98:0/3966280396 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:39.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.897+0000 7fdb49bc3700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb441069c0 0x7fdb44197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:39.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.897+0000 7fdb493c2700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb441009c0 0x7fdb441966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:39.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.897+0000 7fdb48bc1700 1 -- 172.21.15.98:0/3966280396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb441069c0 msgr2=0x7fdb44197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:39.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.897+0000 7fdb48bc1700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb441069c0 0x7fdb44197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:39.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.897+0000 7fdb48bc1700 1 -- 172.21.15.98:0/3966280396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb441009c0 msgr2=0x7fdb441966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:39.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.897+0000 7fdb48bc1700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb441009c0 0x7fdb441966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:39.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.897+0000 7fdb48bc1700 1 -- 172.21.15.98:0/3966280396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb441a6fd0 con 0x7fdb44101380 2024-05-09T08:49:39.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.897+0000 7fdb49bc3700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb441069c0 0x7fdb44197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:49:39.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.897+0000 7fdb48bc1700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb44101380 0x7fdb44196c30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdb3800c9b0 tx=0x7fdb3800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:39.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.897+0000 7fdb427fc700 1 -- 172.21.15.98:0/3966280396 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb38007b00 con 0x7fdb44101380 2024-05-09T08:49:39.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.897+0000 7fdb427fc700 1 -- 172.21.15.98:0/3966280396 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb38007ca0 con 0x7fdb44101380 2024-05-09T08:49:39.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.897+0000 7fdb4b626700 1 -- 172.21.15.98:0/3966280396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb441a72c0 con 0x7fdb44101380 2024-05-09T08:49:39.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.897+0000 7fdb427fc700 1 -- 172.21.15.98:0/3966280396 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb38018780 con 0x7fdb44101380 2024-05-09T08:49:39.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.897+0000 7fdb4b626700 1 -- 172.21.15.98:0/3966280396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb441a7800 con 0x7fdb44101380 2024-05-09T08:49:39.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.898+0000 7fdb4b626700 1 -- 172.21.15.98:0/3966280396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb44069010 con 0x7fdb44101380 2024-05-09T08:49:39.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.899+0000 7fdb427fc700 1 -- 172.21.15.98:0/3966280396 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fdb3800f450 con 0x7fdb44101380 2024-05-09T08:49:39.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.900+0000 7fdb427fc700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdb2c0a2cb0 0x7fdb2c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:39.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.900+0000 7fdb493c2700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdb2c0a2cb0 0x7fdb2c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:39.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.900+0000 7fdb427fc700 1 -- 172.21.15.98:0/3966280396 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fdb380cf5a0 con 0x7fdb44101380 2024-05-09T08:49:39.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.900+0000 7fdb493c2700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdb2c0a2cb0 0x7fdb2c0a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00c9a0 tx=0x7fdb3c009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:39.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:39.904+0000 7fdb427fc700 1 -- 172.21.15.98:0/3966280396 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdb38014030 con 0x7fdb44101380 2024-05-09T08:49:40.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:40.116+0000 7fdb4b626700 1 -- 172.21.15.98:0/3966280396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fdb44197bd0 con 0x7fdb44101380 2024-05-09T08:49:40.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:40.117+0000 7fdb427fc700 1 -- 172.21.15.98:0/3966280396 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fdb3809c040 con 0x7fdb44101380 2024-05-09T08:49:40.116 INFO:teuthology.orchestra.run.smithi098.stdout:111669149766 2024-05-09T08:49:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:40.119+0000 7fdb4b626700 1 -- 172.21.15.98:0/3966280396 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdb2c0a2cb0 msgr2=0x7fdb2c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:40.119+0000 7fdb4b626700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdb2c0a2cb0 0x7fdb2c0a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00c9a0 tx=0x7fdb3c009040 comp rx=0 tx=0).stop 2024-05-09T08:49:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:40.119+0000 7fdb4b626700 1 -- 172.21.15.98:0/3966280396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb44101380 msgr2=0x7fdb44196c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:40.119+0000 7fdb4b626700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb44101380 0x7fdb44196c30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdb3800c9b0 tx=0x7fdb3800cd70 comp rx=0 tx=0).stop 2024-05-09T08:49:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:40.119+0000 7fdb4b626700 1 -- 172.21.15.98:0/3966280396 shutdown_connections 2024-05-09T08:49:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:40.119+0000 7fdb4b626700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdb2c0a2cb0 0x7fdb2c0a5170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:40.119+0000 7fdb4b626700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb441009c0 0x7fdb441966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:40.119+0000 7fdb4b626700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb441069c0 0x7fdb44197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:40.119+0000 7fdb4b626700 1 --2- 172.21.15.98:0/3966280396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb44101380 0x7fdb44196c30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:40.119+0000 7fdb4b626700 1 -- 172.21.15.98:0/3966280396 >> 172.21.15.98:0/3966280396 conn(0x7fdb44077820 msgr2=0x7fdb44103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:40.119+0000 7fdb4b626700 1 -- 172.21.15.98:0/3966280396 shutdown_connections 2024-05-09T08:49:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:40.119+0000 7fdb4b626700 1 -- 172.21.15.98:0/3966280396 wait complete. 2024-05-09T08:49:40.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:40 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3966280396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-09T08:49:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:40 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3966280396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-09T08:49:40.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:40 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3966280396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-09T08:49:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.200+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/34423080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4100d50 msgr2=0x7f6cf41011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.200+0000 7f6cf8ea7700 1 --2- 172.21.15.98:0/34423080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4100d50 0x7f6cf41011d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009990 tx=0x7f6cdc009ca0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.200+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/34423080 shutdown_connections 2024-05-09T08:49:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.200+0000 7f6cf8ea7700 1 --2- 172.21.15.98:0/34423080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cf4101710 0x7f6cf410f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.200+0000 7f6cf8ea7700 1 --2- 172.21.15.98:0/34423080 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cf4108500 0x7f6cf41088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.200+0000 7f6cf8ea7700 1 --2- 172.21.15.98:0/34423080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4100d50 0x7f6cf41011d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.200+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/34423080 >> 172.21.15.98:0/34423080 conn(0x7f6cf4076490 msgr2=0x7f6cf40768a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.201+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/34423080 shutdown_connections 2024-05-09T08:49:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.201+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/34423080 wait complete. 2024-05-09T08:49:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf8ea7700 1 Processor -- start 2024-05-09T08:49:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf8ea7700 1 -- start start 2024-05-09T08:49:41.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf8ea7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4100d50 0x7f6cf4101f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf8ea7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cf4101710 0x7f6cf4102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf8ea7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cf4108500 0x7f6cf41029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf8ea7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf4112680 con 0x7f6cf4100d50 2024-05-09T08:49:41.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf8ea7700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf4112500 con 0x7f6cf4101710 2024-05-09T08:49:41.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf8ea7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf4112800 con 0x7f6cf4108500 2024-05-09T08:49:41.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4100d50 0x7f6cf4101f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4100d50 0x7f6cf4101f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45778/0 (socket says 172.21.15.98:45778) 2024-05-09T08:49:41.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf259c700 1 -- 172.21.15.98:0/1356084861 learned_addr learned my addr 172.21.15.98:0/1356084861 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:41.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6ce9d9b700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cf4101710 0x7f6cf4102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf2d9d700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cf4108500 0x7f6cf41029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf259c700 1 -- 172.21.15.98:0/1356084861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cf4108500 msgr2=0x7f6cf41029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf259c700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cf4108500 0x7f6cf41029b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf259c700 1 -- 172.21.15.98:0/1356084861 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cf4101710 msgr2=0x7f6cf4102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf259c700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cf4101710 0x7f6cf4102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf259c700 1 -- 172.21.15.98:0/1356084861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf41a6fd0 con 0x7f6cf4100d50 2024-05-09T08:49:41.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.202+0000 7f6cf2d9d700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cf4108500 0x7f6cf41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:49:41.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.203+0000 7f6cf259c700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4100d50 0x7f6cf4101f30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6ce400b530 tx=0x7f6ce400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:41.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.203+0000 7f6cebfff700 1 -- 172.21.15.98:0/1356084861 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4015020 con 0x7f6cf4100d50 2024-05-09T08:49:41.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.203+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/1356084861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf41a72c0 con 0x7f6cf4100d50 2024-05-09T08:49:41.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.203+0000 7f6cebfff700 1 -- 172.21.15.98:0/1356084861 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ce4011d10 con 0x7f6cf4100d50 2024-05-09T08:49:41.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.203+0000 7f6cebfff700 1 -- 172.21.15.98:0/1356084861 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce400f2e0 con 0x7f6cf4100d50 2024-05-09T08:49:41.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.203+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/1356084861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf41a77d0 con 0x7f6cf4100d50 2024-05-09T08:49:41.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.204+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/1356084861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf410a940 con 0x7f6cf4100d50 2024-05-09T08:49:41.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.205+0000 7f6cebfff700 1 -- 172.21.15.98:0/1356084861 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6ce4018020 con 0x7f6cf4100d50 2024-05-09T08:49:41.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.206+0000 7f6cebfff700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6ce00a2cb0 0x7f6ce00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.206+0000 7f6ce9d9b700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6ce00a2cb0 0x7f6ce00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.206+0000 7f6cebfff700 1 -- 172.21.15.98:0/1356084861 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f6ce40cf260 con 0x7f6cf4100d50 2024-05-09T08:49:41.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.206+0000 7f6ce9d9b700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6ce00a2cb0 0x7f6ce00a5170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00b5c0 tx=0x7f6cdc005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:41.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.209+0000 7f6cebfff700 1 -- 172.21.15.98:0/1356084861 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6ce401f020 con 0x7f6cf4100d50 2024-05-09T08:49:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.416+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/1356084861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f6cf410a320 con 0x7f6cf4100d50 2024-05-09T08:49:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.416+0000 7f6cebfff700 1 -- 172.21.15.98:0/1356084861 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f6ce409be10 con 0x7f6cf4100d50 2024-05-09T08:49:41.416 INFO:teuthology.orchestra.run.smithi098.stdout:313532612643 2024-05-09T08:49:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.419+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/1356084861 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6ce00a2cb0 msgr2=0x7f6ce00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.419+0000 7f6cf8ea7700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6ce00a2cb0 0x7f6ce00a5170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00b5c0 tx=0x7f6cdc005960 comp rx=0 tx=0).stop 2024-05-09T08:49:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.419+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/1356084861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4100d50 msgr2=0x7f6cf4101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.419+0000 7f6cf8ea7700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4100d50 0x7f6cf4101f30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6ce400b530 tx=0x7f6ce400b840 comp rx=0 tx=0).stop 2024-05-09T08:49:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.419+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/1356084861 shutdown_connections 2024-05-09T08:49:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.419+0000 7f6cf8ea7700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6ce00a2cb0 0x7f6ce00a5170 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.419+0000 7f6cf8ea7700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cf4108500 0x7f6cf41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.419+0000 7f6cf8ea7700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4100d50 0x7f6cf4101f30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.419+0000 7f6cf8ea7700 1 --2- 172.21.15.98:0/1356084861 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cf4101710 0x7f6cf4102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.419+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/1356084861 >> 172.21.15.98:0/1356084861 conn(0x7f6cf4076490 msgr2=0x7f6cf40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.419+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/1356084861 shutdown_connections 2024-05-09T08:49:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:41.419+0000 7f6cf8ea7700 1 -- 172.21.15.98:0/1356084861 wait complete. 2024-05-09T08:49:41.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:41 smithi098 ceph-mon[30092]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:49:41.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:41 smithi105 ceph-mon[34498]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:49:41.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:41 smithi142 ceph-mon[34625]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:49:42.525 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:42.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:42 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1356084861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-09T08:49:42.550 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612639 got 313532612643 for osd.10 2024-05-09T08:49:42.550 DEBUG:teuthology.parallel:result is None 2024-05-09T08:49:42.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:42 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1356084861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-09T08:49:42.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:42 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1356084861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-09T08:49:43.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.067+0000 7f9520535700 1 -- 172.21.15.98:0/2366473133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9518101260 msgr2=0x7f95181016c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:43.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.067+0000 7f9520535700 1 --2- 172.21.15.98:0/2366473133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9518101260 0x7f95181016c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9508009990 tx=0x7f9508009ca0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.067+0000 7f9520535700 1 -- 172.21.15.98:0/2366473133 shutdown_connections 2024-05-09T08:49:43.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.067+0000 7f9520535700 1 --2- 172.21.15.98:0/2366473133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9518101c00 0x7f951810e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.067+0000 7f9520535700 1 --2- 172.21.15.98:0/2366473133 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95181072a0 0x7f9518107680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.067+0000 7f9520535700 1 --2- 172.21.15.98:0/2366473133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9518101260 0x7f95181016c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.067+0000 7f9520535700 1 -- 172.21.15.98:0/2366473133 >> 172.21.15.98:0/2366473133 conn(0x7f9518078150 msgr2=0x7f9518078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:43.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.067+0000 7f9520535700 1 -- 172.21.15.98:0/2366473133 shutdown_connections 2024-05-09T08:49:43.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.067+0000 7f9520535700 1 -- 172.21.15.98:0/2366473133 wait complete. 2024-05-09T08:49:43.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f9520535700 1 Processor -- start 2024-05-09T08:49:43.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f9520535700 1 -- start start 2024-05-09T08:49:43.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f9520535700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9518101260 0x7f9518102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:43.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f9520535700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9518101c00 0x7f9518103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:43.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f9520535700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95181072a0 0x7f9518103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:43.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f9520535700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f95181115e0 con 0x7f9518101260 2024-05-09T08:49:43.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f9520535700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9518111460 con 0x7f9518101c00 2024-05-09T08:49:43.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f9520535700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9518111760 con 0x7f95181072a0 2024-05-09T08:49:43.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f951e2d1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9518101260 0x7f9518102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:43.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f951e2d1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9518101260 0x7f9518102f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45798/0 (socket says 172.21.15.98:45798) 2024-05-09T08:49:43.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f951e2d1700 1 -- 172.21.15.98:0/3897030369 learned_addr learned my addr 172.21.15.98:0/3897030369 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:43.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f951dad0700 1 --2- 172.21.15.98:0/3897030369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9518101c00 0x7f9518103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:43.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f951e2d1700 1 -- 172.21.15.98:0/3897030369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95181072a0 msgr2=0x7f9518103990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:43.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f951e2d1700 1 --2- 172.21.15.98:0/3897030369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95181072a0 0x7f9518103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f951e2d1700 1 -- 172.21.15.98:0/3897030369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9518101c00 msgr2=0x7f9518103450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:43.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f951e2d1700 1 --2- 172.21.15.98:0/3897030369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9518101c00 0x7f9518103450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.068+0000 7f951e2d1700 1 -- 172.21.15.98:0/3897030369 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95181a71a0 con 0x7f9518101260 2024-05-09T08:49:43.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.069+0000 7f951e2d1700 1 --2- 172.21.15.98:0/3897030369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9518101260 0x7f9518102f10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f951400b530 tx=0x7f951400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:43.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.069+0000 7f950f7fe700 1 -- 172.21.15.98:0/3897030369 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514015020 con 0x7f9518101260 2024-05-09T08:49:43.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.069+0000 7f950f7fe700 1 -- 172.21.15.98:0/3897030369 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9514011d10 con 0x7f9518101260 2024-05-09T08:49:43.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.069+0000 7f9520535700 1 -- 172.21.15.98:0/3897030369 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95181a7490 con 0x7f9518101260 2024-05-09T08:49:43.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.069+0000 7f950f7fe700 1 -- 172.21.15.98:0/3897030369 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951400f2e0 con 0x7f9518101260 2024-05-09T08:49:43.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.069+0000 7f9520535700 1 -- 172.21.15.98:0/3897030369 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95181a79a0 con 0x7f9518101260 2024-05-09T08:49:43.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.070+0000 7f9520535700 1 -- 172.21.15.98:0/3897030369 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9518102950 con 0x7f9518101260 2024-05-09T08:49:43.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.071+0000 7f950f7fe700 1 -- 172.21.15.98:0/3897030369 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9514018020 con 0x7f9518101260 2024-05-09T08:49:43.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.072+0000 7f950f7fe700 1 --2- 172.21.15.98:0/3897030369 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f95040a2cb0 0x7f95040a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:43.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.072+0000 7f951dad0700 1 --2- 172.21.15.98:0/3897030369 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f95040a2cb0 0x7f95040a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:43.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.072+0000 7f950f7fe700 1 -- 172.21.15.98:0/3897030369 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f95140cf100 con 0x7f9518101260 2024-05-09T08:49:43.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.072+0000 7f951dad0700 1 --2- 172.21.15.98:0/3897030369 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f95040a2cb0 0x7f95040a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f950800b5c0 tx=0x7f9508005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:43.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.076+0000 7f950f7fe700 1 -- 172.21.15.98:0/3897030369 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f951401f020 con 0x7f9518101260 2024-05-09T08:49:43.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.280+0000 7f9520535700 1 -- 172.21.15.98:0/3897030369 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f9518069010 con 0x7f9518101260 2024-05-09T08:49:43.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.280+0000 7f950f7fe700 1 -- 172.21.15.98:0/3897030369 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9518069010 con 0x7f9518101260 2024-05-09T08:49:43.280 INFO:teuthology.orchestra.run.smithi098.stdout:201863462968 2024-05-09T08:49:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.283+0000 7f9520535700 1 -- 172.21.15.98:0/3897030369 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f95040a2cb0 msgr2=0x7f95040a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.283+0000 7f9520535700 1 --2- 172.21.15.98:0/3897030369 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f95040a2cb0 0x7f95040a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f950800b5c0 tx=0x7f9508005960 comp rx=0 tx=0).stop 2024-05-09T08:49:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.283+0000 7f9520535700 1 -- 172.21.15.98:0/3897030369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9518101260 msgr2=0x7f9518102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.283+0000 7f9520535700 1 --2- 172.21.15.98:0/3897030369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9518101260 0x7f9518102f10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f951400b530 tx=0x7f951400b840 comp rx=0 tx=0).stop 2024-05-09T08:49:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.283+0000 7f9520535700 1 -- 172.21.15.98:0/3897030369 shutdown_connections 2024-05-09T08:49:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.283+0000 7f9520535700 1 --2- 172.21.15.98:0/3897030369 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f95040a2cb0 0x7f95040a5170 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.283+0000 7f9520535700 1 --2- 172.21.15.98:0/3897030369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95181072a0 0x7f9518103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.283+0000 7f9520535700 1 --2- 172.21.15.98:0/3897030369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9518101260 0x7f9518102f10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.283+0000 7f9520535700 1 --2- 172.21.15.98:0/3897030369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9518101c00 0x7f9518103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.283+0000 7f9520535700 1 -- 172.21.15.98:0/3897030369 >> 172.21.15.98:0/3897030369 conn(0x7f9518078150 msgr2=0x7f951806ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.283+0000 7f9520535700 1 -- 172.21.15.98:0/3897030369 shutdown_connections 2024-05-09T08:49:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:43.283+0000 7f9520535700 1 -- 172.21.15.98:0/3897030369 wait complete. 2024-05-09T08:49:43.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:43 smithi098 ceph-mon[30092]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:49:43.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:43 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3897030369' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-09T08:49:43.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:43 smithi105 ceph-mon[34498]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:49:43.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:43 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3897030369' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-09T08:49:43.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:43 smithi142 ceph-mon[34625]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:49:43.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:43 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3897030369' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-09T08:49:44.387 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462961 got 201863462968 for osd.6 2024-05-09T08:49:44.387 DEBUG:teuthology.parallel:result is None 2024-05-09T08:49:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.599+0000 7fbb3ea83700 1 -- 172.21.15.98:0/1241600821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38107280 msgr2=0x7fbb38107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.599+0000 7fbb3ea83700 1 --2- 172.21.15.98:0/1241600821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38107280 0x7fbb38107660 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009990 tx=0x7fbb2c009ca0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.599+0000 7fbb3ea83700 1 -- 172.21.15.98:0/1241600821 shutdown_connections 2024-05-09T08:49:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.599+0000 7fbb3ea83700 1 --2- 172.21.15.98:0/1241600821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb38101bd0 0x7fbb3810e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.599+0000 7fbb3ea83700 1 --2- 172.21.15.98:0/1241600821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38107280 0x7fbb38107660 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.599+0000 7fbb3ea83700 1 --2- 172.21.15.98:0/1241600821 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb38101210 0x7fbb38101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.599+0000 7fbb3ea83700 1 -- 172.21.15.98:0/1241600821 >> 172.21.15.98:0/1241600821 conn(0x7fbb38078190 msgr2=0x7fbb380785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.599+0000 7fbb3ea83700 1 -- 172.21.15.98:0/1241600821 shutdown_connections 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.599+0000 7fbb3ea83700 1 -- 172.21.15.98:0/1241600821 wait complete. 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.600+0000 7fbb3ea83700 1 Processor -- start 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.600+0000 7fbb3ea83700 1 -- start start 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.600+0000 7fbb3ea83700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb38101210 0x7fbb3810cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.600+0000 7fbb3ea83700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38101bd0 0x7fbb3810d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.600+0000 7fbb3ea83700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb38107280 0x7fbb381a39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.600+0000 7fbb3ea83700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbb38110400 con 0x7fbb38101bd0 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.600+0000 7fbb3ea83700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbb38110280 con 0x7fbb38107280 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.600+0000 7fbb3ea83700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbb38110580 con 0x7fbb38101210 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.601+0000 7fbb3c81f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb38101210 0x7fbb3810cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.601+0000 7fbb37fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38101bd0 0x7fbb3810d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.601+0000 7fbb3c81f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb38101210 0x7fbb3810cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55642/0 (socket says 172.21.15.98:55642) 2024-05-09T08:49:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.601+0000 7fbb3c81f700 1 -- 172.21.15.98:0/263081382 learned_addr learned my addr 172.21.15.98:0/263081382 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.601+0000 7fbb3d020700 1 --2- 172.21.15.98:0/263081382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb38107280 0x7fbb381a39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.601+0000 7fbb3d020700 1 -- 172.21.15.98:0/263081382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb38101210 msgr2=0x7fbb3810cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.601+0000 7fbb3d020700 1 --2- 172.21.15.98:0/263081382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb38101210 0x7fbb3810cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.601+0000 7fbb3d020700 1 -- 172.21.15.98:0/263081382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38101bd0 msgr2=0x7fbb3810d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.601+0000 7fbb3d020700 1 --2- 172.21.15.98:0/263081382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38101bd0 0x7fbb3810d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.601+0000 7fbb3d020700 1 -- 172.21.15.98:0/263081382 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb381a4100 con 0x7fbb38107280 2024-05-09T08:49:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.601+0000 7fbb3d020700 1 --2- 172.21.15.98:0/263081382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb38107280 0x7fbb381a39f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000efc0 tx=0x7fbb3000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.601+0000 7fbb35ffb700 1 -- 172.21.15.98:0/263081382 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb300092e0 con 0x7fbb38107280 2024-05-09T08:49:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.602+0000 7fbb35ffb700 1 -- 172.21.15.98:0/263081382 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb30004bb0 con 0x7fbb38107280 2024-05-09T08:49:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.602+0000 7fbb35ffb700 1 -- 172.21.15.98:0/263081382 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb30004280 con 0x7fbb38107280 2024-05-09T08:49:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.602+0000 7fbb3ea83700 1 -- 172.21.15.98:0/263081382 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb381a4390 con 0x7fbb38107280 2024-05-09T08:49:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.602+0000 7fbb3ea83700 1 -- 172.21.15.98:0/263081382 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb381abca0 con 0x7fbb38107280 2024-05-09T08:49:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.603+0000 7fbb237fe700 1 -- 172.21.15.98:0/263081382 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb18005330 con 0x7fbb38107280 2024-05-09T08:49:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.605+0000 7fbb35ffb700 1 -- 172.21.15.98:0/263081382 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fbb3001b030 con 0x7fbb38107280 2024-05-09T08:49:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.605+0000 7fbb35ffb700 1 --2- 172.21.15.98:0/263081382 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbb280a2d30 0x7fbb280a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:45.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.606+0000 7fbb3c81f700 1 --2- 172.21.15.98:0/263081382 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbb280a2d30 0x7fbb280a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:45.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.606+0000 7fbb35ffb700 1 -- 172.21.15.98:0/263081382 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fbb300cf1d0 con 0x7fbb38107280 2024-05-09T08:49:45.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.606+0000 7fbb3c81f700 1 --2- 172.21.15.98:0/263081382 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbb280a2d30 0x7fbb280a51f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c006010 tx=0x7fbb2c009590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:45.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.609+0000 7fbb35ffb700 1 -- 172.21.15.98:0/263081382 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbb30022020 con 0x7fbb38107280 2024-05-09T08:49:45.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.487+0000 7fbec0ca3700 1 -- 172.21.15.98:0/2547455211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbebc102f80 msgr2=0x7fbebc106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:45.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.487+0000 7fbec0ca3700 1 --2- 172.21.15.98:0/2547455211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbebc102f80 0x7fbebc106f80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009990 tx=0x7fbeb0009ca0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.487+0000 7fbec0ca3700 1 -- 172.21.15.98:0/2547455211 shutdown_connections 2024-05-09T08:49:45.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.487+0000 7fbec0ca3700 1 --2- 172.21.15.98:0/2547455211 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbebc1025d0 0x7fbebc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.487+0000 7fbec0ca3700 1 --2- 172.21.15.98:0/2547455211 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbebc107660 0x7fbebc10f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.487+0000 7fbec0ca3700 1 --2- 172.21.15.98:0/2547455211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbebc102f80 0x7fbebc106f80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.487+0000 7fbec0ca3700 1 -- 172.21.15.98:0/2547455211 >> 172.21.15.98:0/2547455211 conn(0x7fbebc079480 msgr2=0x7fbebc079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.487+0000 7fbec0ca3700 1 -- 172.21.15.98:0/2547455211 shutdown_connections 2024-05-09T08:49:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.487+0000 7fbec0ca3700 1 -- 172.21.15.98:0/2547455211 wait complete. 2024-05-09T08:49:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.488+0000 7fbec0ca3700 1 Processor -- start 2024-05-09T08:49:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.488+0000 7fbec0ca3700 1 -- start start 2024-05-09T08:49:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.488+0000 7fbec0ca3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbebc1025d0 0x7fbebc19e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.488+0000 7fbec0ca3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbebc102f80 0x7fbebc19ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.488+0000 7fbec0ca3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbebc107660 0x7fbebc198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.488+0000 7fbec0ca3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbebc1126d0 con 0x7fbebc102f80 2024-05-09T08:49:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.488+0000 7fbec0ca3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbebc112550 con 0x7fbebc1025d0 2024-05-09T08:49:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.488+0000 7fbec0ca3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbebc112850 con 0x7fbebc107660 2024-05-09T08:49:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.488+0000 7fbeb9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbebc102f80 0x7fbebc19ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.488+0000 7fbeb9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbebc102f80 0x7fbebc19ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45810/0 (socket says 172.21.15.98:45810) 2024-05-09T08:49:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.488+0000 7fbeb9d9b700 1 -- 172.21.15.98:0/1995051247 learned_addr learned my addr 172.21.15.98:0/1995051247 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.489+0000 7fbebad9d700 1 --2- 172.21.15.98:0/1995051247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbebc107660 0x7fbebc198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.489+0000 7fbeba59c700 1 --2- 172.21.15.98:0/1995051247 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbebc1025d0 0x7fbebc19e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.489+0000 7fbeb9d9b700 1 -- 172.21.15.98:0/1995051247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbebc107660 msgr2=0x7fbebc198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.489+0000 7fbeb9d9b700 1 --2- 172.21.15.98:0/1995051247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbebc107660 0x7fbebc198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.489+0000 7fbeb9d9b700 1 -- 172.21.15.98:0/1995051247 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbebc1025d0 msgr2=0x7fbebc19e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.489+0000 7fbeb9d9b700 1 --2- 172.21.15.98:0/1995051247 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbebc1025d0 0x7fbebc19e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.489+0000 7fbeb9d9b700 1 -- 172.21.15.98:0/1995051247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbebc1990d0 con 0x7fbebc102f80 2024-05-09T08:49:45.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.489+0000 7fbeb9d9b700 1 --2- 172.21.15.98:0/1995051247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbebc102f80 0x7fbebc19ecf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0000c00 tx=0x7fbeb00038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.489+0000 7fbeab7fe700 1 -- 172.21.15.98:0/1995051247 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0027070 con 0x7fbebc102f80 2024-05-09T08:49:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.489+0000 7fbeab7fe700 1 -- 172.21.15.98:0/1995051247 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbeb0004050 con 0x7fbebc102f80 2024-05-09T08:49:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.489+0000 7fbec0ca3700 1 -- 172.21.15.98:0/1995051247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbebc199360 con 0x7fbebc102f80 2024-05-09T08:49:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.489+0000 7fbeab7fe700 1 -- 172.21.15.98:0/1995051247 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb000f460 con 0x7fbebc102f80 2024-05-09T08:49:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.489+0000 7fbec0ca3700 1 -- 172.21.15.98:0/1995051247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbebc1a5680 con 0x7fbebc102f80 2024-05-09T08:49:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.490+0000 7fbec0ca3700 1 -- 172.21.15.98:0/1995051247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbebc103a60 con 0x7fbebc102f80 2024-05-09T08:49:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.491+0000 7fbeab7fe700 1 -- 172.21.15.98:0/1995051247 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fbeb002a070 con 0x7fbebc102f80 2024-05-09T08:49:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.492+0000 7fbeab7fe700 1 --2- 172.21.15.98:0/1995051247 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbea40a2cb0 0x7fbea40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.492+0000 7fbeba59c700 1 --2- 172.21.15.98:0/1995051247 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbea40a2cb0 0x7fbea40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.492+0000 7fbeab7fe700 1 -- 172.21.15.98:0/1995051247 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fbeb00d0410 con 0x7fbebc102f80 2024-05-09T08:49:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.492+0000 7fbeba59c700 1 --2- 172.21.15.98:0/1995051247 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbea40a2cb0 0x7fbea40a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbeac006fd0 tx=0x7fbeac008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.496+0000 7fbeab7fe700 1 -- 172.21.15.98:0/1995051247 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbeb005e740 con 0x7fbebc102f80 2024-05-09T08:49:45.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.717+0000 7fbec0ca3700 1 -- 172.21.15.98:0/1995051247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fbebc199ea0 con 0x7fbebc102f80 2024-05-09T08:49:45.724 INFO:teuthology.orchestra.run.smithi098.stdout:223338299443 2024-05-09T08:49:45.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.718+0000 7fbeab7fe700 1 -- 172.21.15.98:0/1995051247 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbeb0013070 con 0x7fbebc102f80 2024-05-09T08:49:45.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.720+0000 7fbec0ca3700 1 -- 172.21.15.98:0/1995051247 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbea40a2cb0 msgr2=0x7fbea40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:45.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.720+0000 7fbec0ca3700 1 --2- 172.21.15.98:0/1995051247 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbea40a2cb0 0x7fbea40a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbeac006fd0 tx=0x7fbeac008040 comp rx=0 tx=0).stop 2024-05-09T08:49:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.720+0000 7fbec0ca3700 1 -- 172.21.15.98:0/1995051247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbebc102f80 msgr2=0x7fbebc19ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.720+0000 7fbec0ca3700 1 --2- 172.21.15.98:0/1995051247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbebc102f80 0x7fbebc19ecf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0000c00 tx=0x7fbeb00038b0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.720+0000 7fbec0ca3700 1 -- 172.21.15.98:0/1995051247 shutdown_connections 2024-05-09T08:49:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.720+0000 7fbec0ca3700 1 --2- 172.21.15.98:0/1995051247 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbea40a2cb0 0x7fbea40a5170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.720+0000 7fbec0ca3700 1 --2- 172.21.15.98:0/1995051247 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbebc1025d0 0x7fbebc19e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.720+0000 7fbec0ca3700 1 --2- 172.21.15.98:0/1995051247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbebc107660 0x7fbebc198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.720+0000 7fbec0ca3700 1 --2- 172.21.15.98:0/1995051247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbebc102f80 0x7fbebc19ecf0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.720+0000 7fbec0ca3700 1 -- 172.21.15.98:0/1995051247 >> 172.21.15.98:0/1995051247 conn(0x7fbebc079480 msgr2=0x7fbebc101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.720+0000 7fbec0ca3700 1 -- 172.21.15.98:0/1995051247 shutdown_connections 2024-05-09T08:49:45.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.720+0000 7fbec0ca3700 1 -- 172.21.15.98:0/1995051247 wait complete. 2024-05-09T08:49:45.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.823+0000 7fbb237fe700 1 -- 172.21.15.98:0/263081382 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fbb180058f0 con 0x7fbb38107280 2024-05-09T08:49:45.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.824+0000 7fbb35ffb700 1 -- 172.21.15.98:0/263081382 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbb300bb020 con 0x7fbb38107280 2024-05-09T08:49:45.823 INFO:teuthology.orchestra.run.smithi098.stdout:167503724605 2024-05-09T08:49:45.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.826+0000 7fbb237fe700 1 -- 172.21.15.98:0/263081382 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbb280a2d30 msgr2=0x7fbb280a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.826+0000 7fbb237fe700 1 --2- 172.21.15.98:0/263081382 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbb280a2d30 0x7fbb280a51f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c006010 tx=0x7fbb2c009590 comp rx=0 tx=0).stop 2024-05-09T08:49:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.826+0000 7fbb237fe700 1 -- 172.21.15.98:0/263081382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb38107280 msgr2=0x7fbb381a39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.826+0000 7fbb237fe700 1 --2- 172.21.15.98:0/263081382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb38107280 0x7fbb381a39f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000efc0 tx=0x7fbb3000c3a0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.826+0000 7fbb237fe700 1 -- 172.21.15.98:0/263081382 shutdown_connections 2024-05-09T08:49:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.826+0000 7fbb237fe700 1 --2- 172.21.15.98:0/263081382 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbb280a2d30 0x7fbb280a51f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.826+0000 7fbb237fe700 1 --2- 172.21.15.98:0/263081382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb38107280 0x7fbb381a39f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.826+0000 7fbb237fe700 1 --2- 172.21.15.98:0/263081382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb38101210 0x7fbb3810cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.826+0000 7fbb237fe700 1 --2- 172.21.15.98:0/263081382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb38101bd0 0x7fbb3810d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.826+0000 7fbb237fe700 1 -- 172.21.15.98:0/263081382 >> 172.21.15.98:0/263081382 conn(0x7fbb38078190 msgr2=0x7fbb38102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.826+0000 7fbb237fe700 1 -- 172.21.15.98:0/263081382 shutdown_connections 2024-05-09T08:49:45.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:45.826+0000 7fbb237fe700 1 -- 172.21.15.98:0/263081382 wait complete. 2024-05-09T08:49:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:46 smithi098 ceph-mon[30092]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-09T08:49:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:46 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1995051247' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-09T08:49:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:46 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/263081382' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-09T08:49:46.402 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299437 got 223338299443 for osd.7 2024-05-09T08:49:46.403 DEBUG:teuthology.parallel:result is None 2024-05-09T08:49:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:46 smithi142 ceph-mon[34625]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-09T08:49:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:46 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1995051247' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-09T08:49:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:46 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/263081382' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-09T08:49:46.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:46 smithi105 ceph-mon[34498]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-09T08:49:46.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:46 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1995051247' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-09T08:49:46.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:46 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/263081382' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-09T08:49:47.328 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149759 got 111669149766 for osd.3 2024-05-09T08:49:47.328 DEBUG:teuthology.parallel:result is None 2024-05-09T08:49:48.366 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724600 got 167503724605 for osd.5 2024-05-09T08:49:48.366 DEBUG:teuthology.parallel:result is None 2024-05-09T08:49:48.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:48 smithi098 ceph-mon[30092]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:49:48.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:48 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2536668439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-09T08:49:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.549+0000 7f328dbd7700 1 -- 172.21.15.98:0/1154992177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3288101c10 msgr2=0x7f328810e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.549+0000 7f328dbd7700 1 --2- 172.21.15.98:0/1154992177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3288101c10 0x7f328810e0d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f328000b0a0 tx=0x7f328000b3b0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.549+0000 7f328dbd7700 1 -- 172.21.15.98:0/1154992177 shutdown_connections 2024-05-09T08:49:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.549+0000 7f328dbd7700 1 --2- 172.21.15.98:0/1154992177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3288101c10 0x7f328810e0d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.549+0000 7f328dbd7700 1 --2- 172.21.15.98:0/1154992177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32881072d0 0x7f32881076b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.549+0000 7f328dbd7700 1 --2- 172.21.15.98:0/1154992177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3288101250 0x7f32881016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.549+0000 7f328dbd7700 1 -- 172.21.15.98:0/1154992177 >> 172.21.15.98:0/1154992177 conn(0x7f3288077290 msgr2=0x7f32880776a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.549+0000 7f328dbd7700 1 -- 172.21.15.98:0/1154992177 shutdown_connections 2024-05-09T08:49:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.549+0000 7f328dbd7700 1 -- 172.21.15.98:0/1154992177 wait complete. 2024-05-09T08:49:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.549+0000 7f328dbd7700 1 Processor -- start 2024-05-09T08:49:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f328dbd7700 1 -- start start 2024-05-09T08:49:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f328dbd7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3288101250 0x7f328810e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f328dbd7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3288101c10 0x7f328810ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f328dbd7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32881072d0 0x7f328810f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f328dbd7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3288112690 con 0x7f3288101c10 2024-05-09T08:49:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f328dbd7700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3288112510 con 0x7f3288101250 2024-05-09T08:49:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f328dbd7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3288112810 con 0x7f32881072d0 2024-05-09T08:49:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f3286ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3288101c10 0x7f328810ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f3286ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3288101c10 0x7f328810ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56484/0 (socket says 172.21.15.98:56484) 2024-05-09T08:49:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f3286ffd700 1 -- 172.21.15.98:0/1744916687 learned_addr learned my addr 172.21.15.98:0/1744916687 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f3287fff700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32881072d0 0x7f328810f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f32877fe700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3288101250 0x7f328810e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f3286ffd700 1 -- 172.21.15.98:0/1744916687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32881072d0 msgr2=0x7f328810f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f3286ffd700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32881072d0 0x7f328810f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f3286ffd700 1 -- 172.21.15.98:0/1744916687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3288101250 msgr2=0x7f328810e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f3286ffd700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3288101250 0x7f328810e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f3286ffd700 1 -- 172.21.15.98:0/1744916687 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32881a6fd0 con 0x7f3288101c10 2024-05-09T08:49:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f32877fe700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3288101250 0x7f328810e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:49:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.550+0000 7f3286ffd700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3288101c10 0x7f328810ef20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f327c00d950 tx=0x7f327c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.551+0000 7f3284ff9700 1 -- 172.21.15.98:0/1744916687 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327c016070 con 0x7f3288101c10 2024-05-09T08:49:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.551+0000 7f328dbd7700 1 -- 172.21.15.98:0/1744916687 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32881a72c0 con 0x7f3288101c10 2024-05-09T08:49:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.551+0000 7f3284ff9700 1 -- 172.21.15.98:0/1744916687 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f327c009e20 con 0x7f3288101c10 2024-05-09T08:49:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.551+0000 7f3284ff9700 1 -- 172.21.15.98:0/1744916687 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327c00f620 con 0x7f3288101c10 2024-05-09T08:49:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.551+0000 7f328dbd7700 1 -- 172.21.15.98:0/1744916687 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32881a7850 con 0x7f3288101c10 2024-05-09T08:49:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.552+0000 7f328dbd7700 1 -- 172.21.15.98:0/1744916687 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3288109790 con 0x7f3288101c10 2024-05-09T08:49:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.553+0000 7f3284ff9700 1 -- 172.21.15.98:0/1744916687 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f327c019070 con 0x7f3288101c10 2024-05-09T08:49:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.553+0000 7f3284ff9700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32700a2c20 0x7f32700a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.553+0000 7f32877fe700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32700a2c20 0x7f32700a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.554+0000 7f3284ff9700 1 -- 172.21.15.98:0/1744916687 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f327c0cefa0 con 0x7f3288101c10 2024-05-09T08:49:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.554+0000 7f32877fe700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32700a2c20 0x7f32700a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3278005e10 tx=0x7f3278005d80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.557+0000 7f3284ff9700 1 -- 172.21.15.98:0/1744916687 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f327c09bba0 con 0x7f3288101c10 2024-05-09T08:49:48.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:48 smithi142 ceph-mon[34625]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:49:48.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:48 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2536668439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-09T08:49:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.989+0000 7f8f3830d700 1 -- 172.21.15.98:0/1445444014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3006a8c0 msgr2=0x7f8f3006ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.989+0000 7f8f3830d700 1 --2- 172.21.15.98:0/1445444014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3006a8c0 0x7f8f3006ad20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8f20009990 tx=0x7f8f20009ca0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.989+0000 7f8f3830d700 1 -- 172.21.15.98:0/1445444014 shutdown_connections 2024-05-09T08:49:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.989+0000 7f8f3830d700 1 --2- 172.21.15.98:0/1445444014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f30108390 0x7f8f30108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.989+0000 7f8f3830d700 1 --2- 172.21.15.98:0/1445444014 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f3006b260 0x7f8f3010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.989+0000 7f8f3830d700 1 --2- 172.21.15.98:0/1445444014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3006a8c0 0x7f8f3006ad20 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.989+0000 7f8f3830d700 1 -- 172.21.15.98:0/1445444014 >> 172.21.15.98:0/1445444014 conn(0x7f8f30078190 msgr2=0x7f8f300785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.989+0000 7f8f3830d700 1 -- 172.21.15.98:0/1445444014 shutdown_connections 2024-05-09T08:49:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.989+0000 7f8f3830d700 1 -- 172.21.15.98:0/1445444014 wait complete. 2024-05-09T08:49:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.990+0000 7f8f3830d700 1 Processor -- start 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.990+0000 7f8f3830d700 1 -- start start 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.990+0000 7f8f3830d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f3006a8c0 0x7f8f301012e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.990+0000 7f8f3830d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3006b260 0x7f8f30101820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.990+0000 7f8f3830d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f30108390 0x7f8f30101d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.990+0000 7f8f3830d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f301126d0 con 0x7f8f3006b260 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.990+0000 7f8f3830d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8f30112550 con 0x7f8f3006a8c0 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.990+0000 7f8f3830d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8f30112850 con 0x7f8f30108390 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.990+0000 7f8f358a8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3006b260 0x7f8f30101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.990+0000 7f8f358a8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3006b260 0x7f8f30101820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56460/0 (socket says 172.21.15.98:56460) 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.990+0000 7f8f358a8700 1 -- 172.21.15.98:0/2536668439 learned_addr learned my addr 172.21.15.98:0/2536668439 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.990+0000 7f8f360a9700 1 --2- 172.21.15.98:0/2536668439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f3006a8c0 0x7f8f301012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.991+0000 7f8f368aa700 1 --2- 172.21.15.98:0/2536668439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f30108390 0x7f8f30101d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.991+0000 7f8f358a8700 1 -- 172.21.15.98:0/2536668439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f30108390 msgr2=0x7f8f30101d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.991+0000 7f8f358a8700 1 --2- 172.21.15.98:0/2536668439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f30108390 0x7f8f30101d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.991+0000 7f8f358a8700 1 -- 172.21.15.98:0/2536668439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f3006a8c0 msgr2=0x7f8f301012e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.991+0000 7f8f358a8700 1 --2- 172.21.15.98:0/2536668439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f3006a8c0 0x7f8f301012e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.991+0000 7f8f358a8700 1 -- 172.21.15.98:0/2536668439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f301a71f0 con 0x7f8f3006b260 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.991+0000 7f8f358a8700 1 --2- 172.21.15.98:0/2536668439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3006b260 0x7f8f30101820 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8f2000bff0 tx=0x7f8f20005590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.991+0000 7f8f277fe700 1 -- 172.21.15.98:0/2536668439 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f20027070 con 0x7f8f3006b260 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.991+0000 7f8f277fe700 1 -- 172.21.15.98:0/2536668439 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f200040d0 con 0x7f8f3006b260 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.991+0000 7f8f3830d700 1 -- 172.21.15.98:0/2536668439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f301a7480 con 0x7f8f3006b260 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.991+0000 7f8f277fe700 1 -- 172.21.15.98:0/2536668439 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f20022e00 con 0x7f8f3006b260 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.991+0000 7f8f3830d700 1 -- 172.21.15.98:0/2536668439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f301a7960 con 0x7f8f3006b260 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.992+0000 7f8f3830d700 1 -- 172.21.15.98:0/2536668439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f3010a900 con 0x7f8f3006b260 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.993+0000 7f8f277fe700 1 -- 172.21.15.98:0/2536668439 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8f2002a070 con 0x7f8f3006b260 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.994+0000 7f8f277fe700 1 --2- 172.21.15.98:0/2536668439 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8f1c0a2cb0 0x7f8f1c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.994+0000 7f8f360a9700 1 --2- 172.21.15.98:0/2536668439 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8f1c0a2cb0 0x7f8f1c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.994+0000 7f8f277fe700 1 -- 172.21.15.98:0/2536668439 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f8f200d0340 con 0x7f8f3006b260 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.994+0000 7f8f360a9700 1 --2- 172.21.15.98:0/2536668439 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8f1c0a2cb0 0x7f8f1c0a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009a10 tx=0x7f8f2c009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:47.998+0000 7f8f277fe700 1 -- 172.21.15.98:0/2536668439 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8f2005e740 con 0x7f8f3006b260 2024-05-09T08:49:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.198+0000 7f8f3830d700 1 -- 172.21.15.98:0/2536668439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f8f30102810 con 0x7f8f3006b260 2024-05-09T08:49:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.199+0000 7f8f277fe700 1 -- 172.21.15.98:0/2536668439 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8f20013070 con 0x7f8f3006b260 2024-05-09T08:49:48.735 INFO:teuthology.orchestra.run.smithi098.stdout:55834574930 2024-05-09T08:49:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.201+0000 7f8f3830d700 1 -- 172.21.15.98:0/2536668439 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8f1c0a2cb0 msgr2=0x7f8f1c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.201+0000 7f8f3830d700 1 --2- 172.21.15.98:0/2536668439 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8f1c0a2cb0 0x7f8f1c0a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009a10 tx=0x7f8f2c009190 comp rx=0 tx=0).stop 2024-05-09T08:49:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.201+0000 7f8f3830d700 1 -- 172.21.15.98:0/2536668439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3006b260 msgr2=0x7f8f30101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.201+0000 7f8f3830d700 1 --2- 172.21.15.98:0/2536668439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3006b260 0x7f8f30101820 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8f2000bff0 tx=0x7f8f20005590 comp rx=0 tx=0).stop 2024-05-09T08:49:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.201+0000 7f8f3830d700 1 -- 172.21.15.98:0/2536668439 shutdown_connections 2024-05-09T08:49:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.201+0000 7f8f3830d700 1 --2- 172.21.15.98:0/2536668439 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f8f1c0a2cb0 0x7f8f1c0a5170 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.201+0000 7f8f3830d700 1 --2- 172.21.15.98:0/2536668439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f3006a8c0 0x7f8f301012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.201+0000 7f8f3830d700 1 --2- 172.21.15.98:0/2536668439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f30108390 0x7f8f30101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.201+0000 7f8f3830d700 1 --2- 172.21.15.98:0/2536668439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3006b260 0x7f8f30101820 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.201+0000 7f8f3830d700 1 -- 172.21.15.98:0/2536668439 >> 172.21.15.98:0/2536668439 conn(0x7f8f30078190 msgr2=0x7f8f30100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:48.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.201+0000 7f8f3830d700 1 -- 172.21.15.98:0/2536668439 shutdown_connections 2024-05-09T08:49:48.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.201+0000 7f8f3830d700 1 -- 172.21.15.98:0/2536668439 wait complete. 2024-05-09T08:49:48.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:48 smithi105 ceph-mon[34498]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:49:48.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:48 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2536668439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-09T08:49:48.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.778+0000 7f328dbd7700 1 -- 172.21.15.98:0/1744916687 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f3288109170 con 0x7f3288101c10 2024-05-09T08:49:48.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.778+0000 7f3284ff9700 1 -- 172.21.15.98:0/1744916687 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f327c09b4a0 con 0x7f3288101c10 2024-05-09T08:49:48.777 INFO:teuthology.orchestra.run.smithi098.stdout:257698037807 2024-05-09T08:49:48.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.781+0000 7f328dbd7700 1 -- 172.21.15.98:0/1744916687 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32700a2c20 msgr2=0x7f32700a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:48.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.781+0000 7f328dbd7700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32700a2c20 0x7f32700a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3278005e10 tx=0x7f3278005d80 comp rx=0 tx=0).stop 2024-05-09T08:49:48.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.781+0000 7f328dbd7700 1 -- 172.21.15.98:0/1744916687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3288101c10 msgr2=0x7f328810ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:48.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.781+0000 7f328dbd7700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3288101c10 0x7f328810ef20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f327c00d950 tx=0x7f327c00dc60 comp rx=0 tx=0).stop 2024-05-09T08:49:48.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.781+0000 7f328dbd7700 1 -- 172.21.15.98:0/1744916687 shutdown_connections 2024-05-09T08:49:48.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.781+0000 7f328dbd7700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32700a2c20 0x7f32700a50e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.781+0000 7f328dbd7700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3288101250 0x7f328810e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.781+0000 7f328dbd7700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32881072d0 0x7f328810f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.781+0000 7f328dbd7700 1 --2- 172.21.15.98:0/1744916687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3288101c10 0x7f328810ef20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:48.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.781+0000 7f328dbd7700 1 -- 172.21.15.98:0/1744916687 >> 172.21.15.98:0/1744916687 conn(0x7f3288077290 msgr2=0x7f32880fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:48.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.781+0000 7f328dbd7700 1 -- 172.21.15.98:0/1744916687 shutdown_connections 2024-05-09T08:49:48.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:48.781+0000 7f328dbd7700 1 -- 172.21.15.98:0/1744916687 wait complete. 2024-05-09T08:49:49.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:49 smithi098 ceph-mon[30092]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-09T08:49:49.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:49 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:49.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:49 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:49.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:49 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1744916687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-09T08:49:49.357 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574924 got 55834574930 for osd.1 2024-05-09T08:49:49.357 DEBUG:teuthology.parallel:result is None 2024-05-09T08:49:49.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:49 smithi142 ceph-mon[34625]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-09T08:49:49.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:49 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:49.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:49 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:49.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:49 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1744916687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-09T08:49:49.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:49 smithi105 ceph-mon[34498]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-09T08:49:49.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:49 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:49.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:49 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:49.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:49 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1744916687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-09T08:49:50.112 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037802 got 257698037807 for osd.8 2024-05-09T08:49:50.112 DEBUG:teuthology.parallel:result is None 2024-05-09T08:49:50.112 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-09T08:49:50.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-09T08:49:50.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:50 smithi098 ceph-mon[30092]: pgmap v338: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 511 B/s wr, 11 op/s 2024-05-09T08:49:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:49:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:49:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:50 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:50.355 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:50.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:50 smithi142 ceph-mon[34625]: pgmap v338: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 511 B/s wr, 11 op/s 2024-05-09T08:49:50.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:49:50.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:49:50.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:50 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:50.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:50 smithi105 ceph-mon[34498]: pgmap v338: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 511 B/s wr, 11 op/s 2024-05-09T08:49:50.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:49:50.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:49:50.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:50 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' 2024-05-09T08:49:52.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.217+0000 7fdf26973700 1 -- 172.21.15.98:0/1971292489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf20107290 msgr2=0x7fdf20107670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:52.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.217+0000 7fdf26973700 1 --2- 172.21.15.98:0/1971292489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf20107290 0x7fdf20107670 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdf10009990 tx=0x7fdf10009ca0 comp rx=0 tx=0).stop 2024-05-09T08:49:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.217+0000 7fdf26973700 1 -- 172.21.15.98:0/1971292489 shutdown_connections 2024-05-09T08:49:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.217+0000 7fdf26973700 1 --2- 172.21.15.98:0/1971292489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf20101be0 0x7fdf2010e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.217+0000 7fdf26973700 1 --2- 172.21.15.98:0/1971292489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf20107290 0x7fdf20107670 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.217+0000 7fdf26973700 1 --2- 172.21.15.98:0/1971292489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf20101220 0x7fdf201016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.217+0000 7fdf26973700 1 -- 172.21.15.98:0/1971292489 >> 172.21.15.98:0/1971292489 conn(0x7fdf200781d0 msgr2=0x7fdf200785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.217+0000 7fdf26973700 1 -- 172.21.15.98:0/1971292489 shutdown_connections 2024-05-09T08:49:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.218+0000 7fdf26973700 1 -- 172.21.15.98:0/1971292489 wait complete. 2024-05-09T08:49:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.218+0000 7fdf26973700 1 Processor -- start 2024-05-09T08:49:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.218+0000 7fdf26973700 1 -- start start 2024-05-09T08:49:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.218+0000 7fdf26973700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf20101220 0x7fdf20199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.218+0000 7fdf26973700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf20101be0 0x7fdf20199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.218+0000 7fdf26973700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf20107290 0x7fdf2019a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.218+0000 7fdf26973700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf20111530 con 0x7fdf20101220 2024-05-09T08:49:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.218+0000 7fdf26973700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdf201113b0 con 0x7fdf20101be0 2024-05-09T08:49:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.218+0000 7fdf26973700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdf201116b0 con 0x7fdf20107290 2024-05-09T08:49:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf1ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf20101220 0x7fdf20199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf1ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf20101220 0x7fdf20199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56510/0 (socket says 172.21.15.98:56510) 2024-05-09T08:49:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf1ffff700 1 -- 172.21.15.98:0/1627496388 learned_addr learned my addr 172.21.15.98:0/1627496388 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf24f10700 1 --2- 172.21.15.98:0/1627496388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf20107290 0x7fdf2019a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf1f7fe700 1 --2- 172.21.15.98:0/1627496388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf20101be0 0x7fdf20199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf1ffff700 1 -- 172.21.15.98:0/1627496388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf20107290 msgr2=0x7fdf2019a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf1ffff700 1 --2- 172.21.15.98:0/1627496388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf20107290 0x7fdf2019a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf1ffff700 1 -- 172.21.15.98:0/1627496388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf20101be0 msgr2=0x7fdf20199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf1ffff700 1 --2- 172.21.15.98:0/1627496388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf20101be0 0x7fdf20199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf1ffff700 1 -- 172.21.15.98:0/1627496388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf201a4e30 con 0x7fdf20101220 2024-05-09T08:49:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf1ffff700 1 --2- 172.21.15.98:0/1627496388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf20101220 0x7fdf20199a10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdf10000c00 tx=0x7fdf100047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf1d7fa700 1 -- 172.21.15.98:0/1627496388 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf10027070 con 0x7fdf20101220 2024-05-09T08:49:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf26973700 1 -- 172.21.15.98:0/1627496388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf201a50c0 con 0x7fdf20101220 2024-05-09T08:49:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf1d7fa700 1 -- 172.21.15.98:0/1627496388 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf10003e40 con 0x7fdf20101220 2024-05-09T08:49:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf1d7fa700 1 -- 172.21.15.98:0/1627496388 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf10022d10 con 0x7fdf20101220 2024-05-09T08:49:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.219+0000 7fdf26973700 1 -- 172.21.15.98:0/1627496388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf201a5510 con 0x7fdf20101220 2024-05-09T08:49:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.220+0000 7fdf26973700 1 -- 172.21.15.98:0/1627496388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf20069010 con 0x7fdf20101220 2024-05-09T08:49:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.221+0000 7fdf1d7fa700 1 -- 172.21.15.98:0/1627496388 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fdf1002a070 con 0x7fdf20101220 2024-05-09T08:49:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.222+0000 7fdf1d7fa700 1 --2- 172.21.15.98:0/1627496388 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdf080a2c20 0x7fdf080a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.222+0000 7fdf1f7fe700 1 --2- 172.21.15.98:0/1627496388 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdf080a2c20 0x7fdf080a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.222+0000 7fdf1d7fa700 1 -- 172.21.15.98:0/1627496388 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fdf100d0880 con 0x7fdf20101220 2024-05-09T08:49:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.223+0000 7fdf1f7fe700 1 --2- 172.21.15.98:0/1627496388 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdf080a2c20 0x7fdf080a50e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdf200786d0 tx=0x7fdf1400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:52.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.226+0000 7fdf1d7fa700 1 -- 172.21.15.98:0/1627496388 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdf10031070 con 0x7fdf20101220 2024-05-09T08:49:52.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.418+0000 7fdf26973700 1 -- 172.21.15.98:0/1627496388 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdf20062760 con 0x7fdf080a2c20 2024-05-09T08:49:52.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:52 smithi098 ceph-mon[30092]: pgmap v339: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-09T08:49:52.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.427+0000 7fdf1d7fa700 1 -- 172.21.15.98:0/1627496388 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340923 (secure 0 0 0) 0x7fdf20062760 con 0x7fdf080a2c20 2024-05-09T08:49:52.432 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:49:52.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.430+0000 7fdf26973700 1 -- 172.21.15.98:0/1627496388 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdf080a2c20 msgr2=0x7fdf080a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:52.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.430+0000 7fdf26973700 1 --2- 172.21.15.98:0/1627496388 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdf080a2c20 0x7fdf080a50e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdf200786d0 tx=0x7fdf1400b410 comp rx=0 tx=0).stop 2024-05-09T08:49:52.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.430+0000 7fdf26973700 1 -- 172.21.15.98:0/1627496388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf20101220 msgr2=0x7fdf20199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:52.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.430+0000 7fdf26973700 1 --2- 172.21.15.98:0/1627496388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf20101220 0x7fdf20199a10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdf10000c00 tx=0x7fdf100047b0 comp rx=0 tx=0).stop 2024-05-09T08:49:52.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.431+0000 7fdf26973700 1 -- 172.21.15.98:0/1627496388 shutdown_connections 2024-05-09T08:49:52.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.431+0000 7fdf26973700 1 --2- 172.21.15.98:0/1627496388 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdf080a2c20 0x7fdf080a50e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:52.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.431+0000 7fdf26973700 1 --2- 172.21.15.98:0/1627496388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf20107290 0x7fdf2019a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:52.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.431+0000 7fdf26973700 1 --2- 172.21.15.98:0/1627496388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf20101220 0x7fdf20199a10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:52.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.431+0000 7fdf26973700 1 --2- 172.21.15.98:0/1627496388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf20101be0 0x7fdf20199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:52.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.431+0000 7fdf26973700 1 -- 172.21.15.98:0/1627496388 >> 172.21.15.98:0/1627496388 conn(0x7fdf200781d0 msgr2=0x7fdf200ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:52.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.431+0000 7fdf26973700 1 -- 172.21.15.98:0/1627496388 shutdown_connections 2024-05-09T08:49:52.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:52.431+0000 7fdf26973700 1 -- 172.21.15.98:0/1627496388 wait complete. 2024-05-09T08:49:52.436 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-09T08:49:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:52 smithi142 ceph-mon[34625]: pgmap v339: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-09T08:49:52.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:52 smithi105 ceph-mon[34498]: pgmap v339: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-09T08:49:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-05-09T08:49:51.204715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658979,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":884,"num_read_kb":569,"num_write":428,"num_write_kb":3083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180853,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":824,"ondisk_log_size":824,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353236,"kb_used_data":12392,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588588,"statfs":{"total":1151940427776,"available":1151578714112,"internally_reserved":0,"allocated":12689408,"data_stored":8006399,"data_compressed":353709,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,27,40,18,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":58,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003464"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.213936+0000","last_change":"2024-05-09T08:48:18.373685+0000","last_active":"2024-05-09T08:49:47.213936+0000","last_peered":"2024-05-09T08:49:47.213936+0000","last_clean":"2024-05-09T08:49:47.213936+0000","last_became_active":"2024-05-09T08:48:18.373305+0000","last_became_peered":"2024-05-09T08:48:18.373305+0000","last_unstale":"2024-05-09T08:49:47.213936+0000","last_undegraded":"2024-05-09T08:49:47.213936+0000","last_fullsized":"2024-05-09T08:49:47.213936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:30:45.261706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.201601+0000","last_change":"2024-05-09T08:48:18.373152+0000","last_active":"2024-05-09T08:49:47.201601+0000","last_peered":"2024-05-09T08:49:47.201601+0000","last_clean":"2024-05-09T08:49:47.201601+0000","last_became_active":"2024-05-09T08:48:18.372015+0000","last_became_peered":"2024-05-09T08:48:18.372015+0000","last_unstale":"2024-05-09T08:49:47.201601+0000","last_undegraded":"2024-05-09T08:49:47.201601+0000","last_fullsized":"2024-05-09T08:49:47.201601+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:47.512778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.217042+0000","last_change":"2024-05-09T08:48:18.370177+0000","last_active":"2024-05-09T08:49:47.217042+0000","last_peered":"2024-05-09T08:49:47.217042+0000","last_clean":"2024-05-09T08:49:47.217042+0000","last_became_active":"2024-05-09T08:48:18.369842+0000","last_became_peered":"2024-05-09T08:48:18.369842+0000","last_unstale":"2024-05-09T08:49:47.217042+0000","last_undegraded":"2024-05-09T08:49:47.217042+0000","last_fullsized":"2024-05-09T08:49:47.217042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:00:24.015500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.202870+0000","last_change":"2024-05-09T08:48:18.375364+0000","last_active":"2024-05-09T08:49:47.202870+0000","last_peered":"2024-05-09T08:49:47.202870+0000","last_clean":"2024-05-09T08:49:47.202870+0000","last_became_active":"2024-05-09T08:48:18.375030+0000","last_became_peered":"2024-05-09T08:48:18.375030+0000","last_unstale":"2024-05-09T08:49:47.202870+0000","last_undegraded":"2024-05-09T08:49:47.202870+0000","last_fullsized":"2024-05-09T08:49:47.202870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:43:57.684123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.205041+0000","last_change":"2024-05-09T08:48:18.374548+0000","last_active":"2024-05-09T08:49:47.205041+0000","last_peered":"2024-05-09T08:49:47.205041+0000","last_clean":"2024-05-09T08:49:47.205041+0000","last_became_active":"2024-05-09T08:48:18.374190+0000","last_became_peered":"2024-05-09T08:48:18.374190+0000","last_unstale":"2024-05-09T08:49:47.205041+0000","last_undegraded":"2024-05-09T08:49:47.205041+0000","last_fullsized":"2024-05-09T08:49:47.205041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:43:48.456839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.213257+0000","last_change":"2024-05-09T08:48:18.371234+0000","last_active":"2024-05-09T08:49:47.213257+0000","last_peered":"2024-05-09T08:49:47.213257+0000","last_clean":"2024-05-09T08:49:47.213257+0000","last_became_active":"2024-05-09T08:48:18.370218+0000","last_became_peered":"2024-05-09T08:48:18.370218+0000","last_unstale":"2024-05-09T08:49:47.213257+0000","last_undegraded":"2024-05-09T08:49:47.213257+0000","last_fullsized":"2024-05-09T08:49:47.213257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:02:09.923146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.215407+0000","last_change":"2024-05-09T08:49:18.366083+0000","last_active":"2024-05-09T08:49:47.215407+0000","last_peered":"2024-05-09T08:49:47.215407+0000","last_clean":"2024-05-09T08:49:47.215407+0000","last_became_active":"2024-05-09T08:49:18.365755+0000","last_became_peered":"2024-05-09T08:49:18.365755+0000","last_unstale":"2024-05-09T08:49:47.215407+0000","last_undegraded":"2024-05-09T08:49:47.215407+0000","last_fullsized":"2024-05-09T08:49:47.215407+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:40:18.052955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.489001+0000","last_change":"2024-05-09T08:49:18.488640+0000","last_active":"2024-05-09T08:49:18.489001+0000","last_peered":"2024-05-09T08:49:18.489001+0000","last_clean":"2024-05-09T08:49:18.489001+0000","last_became_active":"2024-05-09T08:49:18.488404+0000","last_became_peered":"2024-05-09T08:49:18.488404+0000","last_unstale":"2024-05-09T08:49:18.489001+0000","last_undegraded":"2024-05-09T08:49:18.489001+0000","last_fullsized":"2024-05-09T08:49:18.489001+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:47.317453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.218289+0000","last_change":"2024-05-09T08:49:18.368485+0000","last_active":"2024-05-09T08:49:47.218289+0000","last_peered":"2024-05-09T08:49:47.218289+0000","last_clean":"2024-05-09T08:49:47.218289+0000","last_became_active":"2024-05-09T08:49:18.368175+0000","last_became_peered":"2024-05-09T08:49:18.368175+0000","last_unstale":"2024-05-09T08:49:47.218289+0000","last_undegraded":"2024-05-09T08:49:47.218289+0000","last_fullsized":"2024-05-09T08:49:47.218289+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:39:32.399209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.213082+0000","last_change":"2024-05-09T08:48:18.363668+0000","last_active":"2024-05-09T08:49:47.213082+0000","last_peered":"2024-05-09T08:49:47.213082+0000","last_clean":"2024-05-09T08:49:47.213082+0000","last_became_active":"2024-05-09T08:48:18.363248+0000","last_became_peered":"2024-05-09T08:48:18.363248+0000","last_unstale":"2024-05-09T08:49:47.213082+0000","last_undegraded":"2024-05-09T08:49:47.213082+0000","last_fullsized":"2024-05-09T08:49:47.213082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:19:33.802350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.219443+0000","last_change":"2024-05-09T08:48:18.376401+0000","last_active":"2024-05-09T08:49:47.219443+0000","last_peered":"2024-05-09T08:49:47.219443+0000","last_clean":"2024-05-09T08:49:47.219443+0000","last_became_active":"2024-05-09T08:48:18.375520+0000","last_became_peered":"2024-05-09T08:48:18.375520+0000","last_unstale":"2024-05-09T08:49:47.219443+0000","last_undegraded":"2024-05-09T08:49:47.219443+0000","last_fullsized":"2024-05-09T08:49:47.219443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:40:36.145071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.209825+0000","last_change":"2024-05-09T08:48:18.372847+0000","last_active":"2024-05-09T08:49:47.209825+0000","last_peered":"2024-05-09T08:49:47.209825+0000","last_clean":"2024-05-09T08:49:47.209825+0000","last_became_active":"2024-05-09T08:48:18.370671+0000","last_became_peered":"2024-05-09T08:48:18.370671+0000","last_unstale":"2024-05-09T08:49:47.209825+0000","last_undegraded":"2024-05-09T08:49:47.209825+0000","last_fullsized":"2024-05-09T08:49:47.209825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:06:22.438515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.208023+0000","last_change":"2024-05-09T08:48:18.365810+0000","last_active":"2024-05-09T08:49:47.208023+0000","last_peered":"2024-05-09T08:49:47.208023+0000","last_clean":"2024-05-09T08:49:47.208023+0000","last_became_active":"2024-05-09T08:48:18.365214+0000","last_became_peered":"2024-05-09T08:48:18.365214+0000","last_unstale":"2024-05-09T08:49:47.208023+0000","last_undegraded":"2024-05-09T08:49:47.208023+0000","last_fullsized":"2024-05-09T08:49:47.208023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:00.037153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.209125+0000","last_change":"2024-05-09T08:48:18.375279+0000","last_active":"2024-05-09T08:49:47.209125+0000","last_peered":"2024-05-09T08:49:47.209125+0000","last_clean":"2024-05-09T08:49:47.209125+0000","last_became_active":"2024-05-09T08:48:18.374918+0000","last_became_peered":"2024-05-09T08:48:18.374918+0000","last_unstale":"2024-05-09T08:49:47.209125+0000","last_undegraded":"2024-05-09T08:49:47.209125+0000","last_fullsized":"2024-05-09T08:49:47.209125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:23:12.889283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.215473+0000","last_change":"2024-05-09T08:49:18.369768+0000","last_active":"2024-05-09T08:49:47.215473+0000","last_peered":"2024-05-09T08:49:47.215473+0000","last_clean":"2024-05-09T08:49:47.215473+0000","last_became_active":"2024-05-09T08:49:18.369520+0000","last_became_peered":"2024-05-09T08:49:18.369520+0000","last_unstale":"2024-05-09T08:49:47.215473+0000","last_undegraded":"2024-05-09T08:49:47.215473+0000","last_fullsized":"2024-05-09T08:49:47.215473+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:44:19.502358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.217231+0000","last_change":"2024-05-09T08:48:18.373905+0000","last_active":"2024-05-09T08:49:47.217231+0000","last_peered":"2024-05-09T08:49:47.217231+0000","last_clean":"2024-05-09T08:49:47.217231+0000","last_became_active":"2024-05-09T08:48:18.373566+0000","last_became_peered":"2024-05-09T08:48:18.373566+0000","last_unstale":"2024-05-09T08:49:47.217231+0000","last_undegraded":"2024-05-09T08:49:47.217231+0000","last_fullsized":"2024-05-09T08:49:47.217231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:33.905686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.214154+0000","last_change":"2024-05-09T08:48:18.373970+0000","last_active":"2024-05-09T08:49:47.214154+0000","last_peered":"2024-05-09T08:49:47.214154+0000","last_clean":"2024-05-09T08:49:47.214154+0000","last_became_active":"2024-05-09T08:48:18.372696+0000","last_became_peered":"2024-05-09T08:48:18.372696+0000","last_unstale":"2024-05-09T08:49:47.214154+0000","last_undegraded":"2024-05-09T08:49:47.214154+0000","last_fullsized":"2024-05-09T08:49:47.214154+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:09:33.751950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359812+0000","last_change":"2024-05-09T08:48:18.376851+0000","last_active":"2024-05-09T08:49:18.359812+0000","last_peered":"2024-05-09T08:49:18.359812+0000","last_clean":"2024-05-09T08:49:18.359812+0000","last_became_active":"2024-05-09T08:48:18.376230+0000","last_became_peered":"2024-05-09T08:48:18.376230+0000","last_unstale":"2024-05-09T08:49:18.359812+0000","last_undegraded":"2024-05-09T08:49:18.359812+0000","last_fullsized":"2024-05-09T08:49:18.359812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:34.959179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.199408+0000","last_change":"2024-05-09T08:48:18.370287+0000","last_active":"2024-05-09T08:49:47.199408+0000","last_peered":"2024-05-09T08:49:47.199408+0000","last_clean":"2024-05-09T08:49:47.199408+0000","last_became_active":"2024-05-09T08:48:18.369956+0000","last_became_peered":"2024-05-09T08:48:18.369956+0000","last_unstale":"2024-05-09T08:49:47.199408+0000","last_undegraded":"2024-05-09T08:49:47.199408+0000","last_fullsized":"2024-05-09T08:49:47.199408+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:19:34.746216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361884+0000","last_change":"2024-05-09T08:48:22.377973+0000","last_active":"2024-05-09T08:49:18.361884+0000","last_peered":"2024-05-09T08:49:18.361884+0000","last_clean":"2024-05-09T08:49:18.361884+0000","last_became_active":"2024-05-09T08:48:22.377325+0000","last_became_peered":"2024-05-09T08:48:22.377325+0000","last_unstale":"2024-05-09T08:49:18.361884+0000","last_undegraded":"2024-05-09T08:49:18.361884+0000","last_fullsized":"2024-05-09T08:49:18.361884+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:32:00.873082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":109,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:45.854520+0000","last_change":"2024-05-09T08:48:17.595784+0000","last_active":"2024-05-09T08:49:45.854520+0000","last_peered":"2024-05-09T08:49:45.854520+0000","last_clean":"2024-05-09T08:49:45.854520+0000","last_became_active":"2024-05-09T08:48:17.595395+0000","last_became_peered":"2024-05-09T08:48:17.595395+0000","last_unstale":"2024-05-09T08:49:45.854520+0000","last_undegraded":"2024-05-09T08:49:45.854520+0000","last_fullsized":"2024-05-09T08:49:45.854520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:47.475148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.209386+0000","last_change":"2024-05-09T08:48:18.382332+0000","last_active":"2024-05-09T08:49:47.209386+0000","last_peered":"2024-05-09T08:49:47.209386+0000","last_clean":"2024-05-09T08:49:47.209386+0000","last_became_active":"2024-05-09T08:48:18.381536+0000","last_became_peered":"2024-05-09T08:48:18.381536+0000","last_unstale":"2024-05-09T08:49:47.209386+0000","last_undegraded":"2024-05-09T08:49:47.209386+0000","last_fullsized":"2024-05-09T08:49:47.209386+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:52:48.327000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359199+0000","last_change":"2024-05-09T08:48:22.379541+0000","last_active":"2024-05-09T08:49:18.359199+0000","last_peered":"2024-05-09T08:49:18.359199+0000","last_clean":"2024-05-09T08:49:18.359199+0000","last_became_active":"2024-05-09T08:48:22.379135+0000","last_became_peered":"2024-05-09T08:48:22.379135+0000","last_unstale":"2024-05-09T08:49:18.359199+0000","last_undegraded":"2024-05-09T08:49:18.359199+0000","last_fullsized":"2024-05-09T08:49:18.359199+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:51.365075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.693649+0000","last_change":"2024-05-09T08:48:17.592168+0000","last_active":"2024-05-09T08:49:18.693649+0000","last_peered":"2024-05-09T08:49:18.693649+0000","last_clean":"2024-05-09T08:49:18.693649+0000","last_became_active":"2024-05-09T08:48:17.591822+0000","last_became_peered":"2024-05-09T08:48:17.591822+0000","last_unstale":"2024-05-09T08:49:18.693649+0000","last_undegraded":"2024-05-09T08:49:18.693649+0000","last_fullsized":"2024-05-09T08:49:18.693649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:45:15.162240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.213872+0000","last_change":"2024-05-09T08:48:18.379396+0000","last_active":"2024-05-09T08:49:47.213872+0000","last_peered":"2024-05-09T08:49:47.213872+0000","last_clean":"2024-05-09T08:49:47.213872+0000","last_became_active":"2024-05-09T08:48:18.378770+0000","last_became_peered":"2024-05-09T08:48:18.378770+0000","last_unstale":"2024-05-09T08:49:47.213872+0000","last_undegraded":"2024-05-09T08:49:47.213872+0000","last_fullsized":"2024-05-09T08:49:47.213872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:22:22.699186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359324+0000","last_change":"2024-05-09T08:48:22.387777+0000","last_active":"2024-05-09T08:49:18.359324+0000","last_peered":"2024-05-09T08:49:18.359324+0000","last_clean":"2024-05-09T08:49:18.359324+0000","last_became_active":"2024-05-09T08:48:22.387300+0000","last_became_peered":"2024-05-09T08:48:22.387300+0000","last_unstale":"2024-05-09T08:49:18.359324+0000","last_undegraded":"2024-05-09T08:49:18.359324+0000","last_fullsized":"2024-05-09T08:49:18.359324+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:08:35.646374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.367409+0000","last_change":"2024-05-09T08:48:17.602493+0000","last_active":"2024-05-09T08:49:18.367409+0000","last_peered":"2024-05-09T08:49:18.367409+0000","last_clean":"2024-05-09T08:49:18.367409+0000","last_became_active":"2024-05-09T08:48:17.601625+0000","last_became_peered":"2024-05-09T08:48:17.601625+0000","last_unstale":"2024-05-09T08:49:18.367409+0000","last_undegraded":"2024-05-09T08:49:18.367409+0000","last_fullsized":"2024-05-09T08:49:18.367409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:33.044532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.217229+0000","last_change":"2024-05-09T08:48:18.380615+0000","last_active":"2024-05-09T08:49:47.217229+0000","last_peered":"2024-05-09T08:49:47.217229+0000","last_clean":"2024-05-09T08:49:47.217229+0000","last_became_active":"2024-05-09T08:48:18.380196+0000","last_became_peered":"2024-05-09T08:48:18.380196+0000","last_unstale":"2024-05-09T08:49:47.217229+0000","last_undegraded":"2024-05-09T08:49:47.217229+0000","last_fullsized":"2024-05-09T08:49:47.217229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:27.299263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359229+0000","last_change":"2024-05-09T08:48:22.381954+0000","last_active":"2024-05-09T08:49:18.359229+0000","last_peered":"2024-05-09T08:49:18.359229+0000","last_clean":"2024-05-09T08:49:18.359229+0000","last_became_active":"2024-05-09T08:48:22.381527+0000","last_became_peered":"2024-05-09T08:48:22.381527+0000","last_unstale":"2024-05-09T08:49:18.359229+0000","last_undegraded":"2024-05-09T08:49:18.359229+0000","last_fullsized":"2024-05-09T08:49:18.359229+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:35.170753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.366013+0000","last_change":"2024-05-09T08:48:17.592795+0000","last_active":"2024-05-09T08:49:18.366013+0000","last_peered":"2024-05-09T08:49:18.366013+0000","last_clean":"2024-05-09T08:49:18.366013+0000","last_became_active":"2024-05-09T08:48:17.592240+0000","last_became_peered":"2024-05-09T08:48:17.592240+0000","last_unstale":"2024-05-09T08:49:18.366013+0000","last_undegraded":"2024-05-09T08:49:18.366013+0000","last_fullsized":"2024-05-09T08:49:18.366013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:27:30.056805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.213858+0000","last_change":"2024-05-09T08:48:18.373853+0000","last_active":"2024-05-09T08:49:47.213858+0000","last_peered":"2024-05-09T08:49:47.213858+0000","last_clean":"2024-05-09T08:49:47.213858+0000","last_became_active":"2024-05-09T08:48:18.373482+0000","last_became_peered":"2024-05-09T08:48:18.373482+0000","last_unstale":"2024-05-09T08:49:47.213858+0000","last_undegraded":"2024-05-09T08:49:47.213858+0000","last_fullsized":"2024-05-09T08:49:47.213858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:14:49.129927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362680+0000","last_change":"2024-05-09T08:48:22.374267+0000","last_active":"2024-05-09T08:49:18.362680+0000","last_peered":"2024-05-09T08:49:18.362680+0000","last_clean":"2024-05-09T08:49:18.362680+0000","last_became_active":"2024-05-09T08:48:22.373824+0000","last_became_peered":"2024-05-09T08:48:22.373824+0000","last_unstale":"2024-05-09T08:49:18.362680+0000","last_undegraded":"2024-05-09T08:49:18.362680+0000","last_fullsized":"2024-05-09T08:49:18.362680+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:27.407543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":109,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.827138+0000","last_change":"2024-05-09T08:48:17.596851+0000","last_active":"2024-05-09T08:49:47.827138+0000","last_peered":"2024-05-09T08:49:47.827138+0000","last_clean":"2024-05-09T08:49:47.827138+0000","last_became_active":"2024-05-09T08:48:17.596532+0000","last_became_peered":"2024-05-09T08:48:17.596532+0000","last_unstale":"2024-05-09T08:49:47.827138+0000","last_undegraded":"2024-05-09T08:49:47.827138+0000","last_fullsized":"2024-05-09T08:49:47.827138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:58:51.923188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.205763+0000","last_change":"2024-05-09T08:48:18.372214+0000","last_active":"2024-05-09T08:49:47.205763+0000","last_peered":"2024-05-09T08:49:47.205763+0000","last_clean":"2024-05-09T08:49:47.205763+0000","last_became_active":"2024-05-09T08:48:18.371850+0000","last_became_peered":"2024-05-09T08:48:18.371850+0000","last_unstale":"2024-05-09T08:49:47.205763+0000","last_undegraded":"2024-05-09T08:49:47.205763+0000","last_fullsized":"2024-05-09T08:49:47.205763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:56.971735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.371344+0000","last_change":"2024-05-09T08:49:18.371344+0000","last_active":"2024-05-09T08:49:18.371344+0000","last_peered":"2024-05-09T08:49:18.371344+0000","last_clean":"2024-05-09T08:49:18.371344+0000","last_became_active":"2024-05-09T08:49:18.370911+0000","last_became_peered":"2024-05-09T08:49:18.370911+0000","last_unstale":"2024-05-09T08:49:18.371344+0000","last_undegraded":"2024-05-09T08:49:18.371344+0000","last_fullsized":"2024-05-09T08:49:18.371344+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:53:49.122965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,8],"acting":[2,6,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.488066+0000","last_change":"2024-05-09T08:48:17.601726+0000","last_active":"2024-05-09T08:49:18.488066+0000","last_peered":"2024-05-09T08:49:18.488066+0000","last_clean":"2024-05-09T08:49:18.488066+0000","last_became_active":"2024-05-09T08:48:17.601372+0000","last_became_peered":"2024-05-09T08:48:17.601372+0000","last_unstale":"2024-05-09T08:49:18.488066+0000","last_undegraded":"2024-05-09T08:49:18.488066+0000","last_fullsized":"2024-05-09T08:49:18.488066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:27:56.683821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.215540+0000","last_change":"2024-05-09T08:49:18.694713+0000","last_active":"2024-05-09T08:49:47.215540+0000","last_peered":"2024-05-09T08:49:47.215540+0000","last_clean":"2024-05-09T08:49:47.215540+0000","last_became_active":"2024-05-09T08:49:18.694246+0000","last_became_peered":"2024-05-09T08:49:18.694246+0000","last_unstale":"2024-05-09T08:49:47.215540+0000","last_undegraded":"2024-05-09T08:49:47.215540+0000","last_fullsized":"2024-05-09T08:49:47.215540+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:05:30.818965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.363420+0000","last_change":"2024-05-09T08:48:22.390990+0000","last_active":"2024-05-09T08:49:18.363420+0000","last_peered":"2024-05-09T08:49:18.363420+0000","last_clean":"2024-05-09T08:49:18.363420+0000","last_became_active":"2024-05-09T08:48:22.390601+0000","last_became_peered":"2024-05-09T08:48:22.390601+0000","last_unstale":"2024-05-09T08:49:18.363420+0000","last_undegraded":"2024-05-09T08:49:18.363420+0000","last_fullsized":"2024-05-09T08:49:18.363420+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:47:24.644408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361959+0000","last_change":"2024-05-09T08:48:17.598008+0000","last_active":"2024-05-09T08:49:18.361959+0000","last_peered":"2024-05-09T08:49:18.361959+0000","last_clean":"2024-05-09T08:49:18.361959+0000","last_became_active":"2024-05-09T08:48:17.597514+0000","last_became_peered":"2024-05-09T08:48:17.597514+0000","last_unstale":"2024-05-09T08:49:18.361959+0000","last_undegraded":"2024-05-09T08:49:18.361959+0000","last_fullsized":"2024-05-09T08:49:18.361959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:18:32.788536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.215821+0000","last_change":"2024-05-09T08:48:18.378061+0000","last_active":"2024-05-09T08:49:47.215821+0000","last_peered":"2024-05-09T08:49:47.215821+0000","last_clean":"2024-05-09T08:49:47.215821+0000","last_became_active":"2024-05-09T08:48:18.377684+0000","last_became_peered":"2024-05-09T08:48:18.377684+0000","last_unstale":"2024-05-09T08:49:47.215821+0000","last_undegraded":"2024-05-09T08:49:47.215821+0000","last_fullsized":"2024-05-09T08:49:47.215821+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:13:08.338307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361739+0000","last_change":"2024-05-09T08:48:22.385779+0000","last_active":"2024-05-09T08:49:18.361739+0000","last_peered":"2024-05-09T08:49:18.361739+0000","last_clean":"2024-05-09T08:49:18.361739+0000","last_became_active":"2024-05-09T08:48:22.385021+0000","last_became_peered":"2024-05-09T08:48:22.385021+0000","last_unstale":"2024-05-09T08:49:18.361739+0000","last_undegraded":"2024-05-09T08:49:18.361739+0000","last_fullsized":"2024-05-09T08:49:18.361739+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:51.350754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361862+0000","last_change":"2024-05-09T08:48:17.595750+0000","last_active":"2024-05-09T08:49:18.361862+0000","last_peered":"2024-05-09T08:49:18.361862+0000","last_clean":"2024-05-09T08:49:18.361862+0000","last_became_active":"2024-05-09T08:48:17.595394+0000","last_became_peered":"2024-05-09T08:48:17.595394+0000","last_unstale":"2024-05-09T08:49:18.361862+0000","last_undegraded":"2024-05-09T08:49:18.361862+0000","last_fullsized":"2024-05-09T08:49:18.361862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:39:21.975059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359486+0000","last_change":"2024-05-09T08:48:18.370750+0000","last_active":"2024-05-09T08:49:18.359486+0000","last_peered":"2024-05-09T08:49:18.359486+0000","last_clean":"2024-05-09T08:49:18.359486+0000","last_became_active":"2024-05-09T08:48:18.370383+0000","last_became_peered":"2024-05-09T08:48:18.370383+0000","last_unstale":"2024-05-09T08:49:18.359486+0000","last_undegraded":"2024-05-09T08:49:18.359486+0000","last_fullsized":"2024-05-09T08:49:18.359486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:36.049350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.692574+0000","last_change":"2024-05-09T08:48:22.377157+0000","last_active":"2024-05-09T08:49:18.692574+0000","last_peered":"2024-05-09T08:49:18.692574+0000","last_clean":"2024-05-09T08:49:18.692574+0000","last_became_active":"2024-05-09T08:48:22.376730+0000","last_became_peered":"2024-05-09T08:48:22.376730+0000","last_unstale":"2024-05-09T08:49:18.692574+0000","last_undegraded":"2024-05-09T08:49:18.692574+0000","last_fullsized":"2024-05-09T08:49:18.692574+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:41:52.080730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'5","reported_seq":172,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.826405+0000","last_change":"2024-05-09T08:48:17.598528+0000","last_active":"2024-05-09T08:49:47.826405+0000","last_peered":"2024-05-09T08:49:47.826405+0000","last_clean":"2024-05-09T08:49:47.826405+0000","last_became_active":"2024-05-09T08:48:17.598072+0000","last_became_peered":"2024-05-09T08:48:17.598072+0000","last_unstale":"2024-05-09T08:49:47.826405+0000","last_undegraded":"2024-05-09T08:49:47.826405+0000","last_fullsized":"2024-05-09T08:49:47.826405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:35.697154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.218018+0000","last_change":"2024-05-09T08:48:18.380919+0000","last_active":"2024-05-09T08:49:47.218018+0000","last_peered":"2024-05-09T08:49:47.218018+0000","last_clean":"2024-05-09T08:49:47.218018+0000","last_became_active":"2024-05-09T08:48:18.380485+0000","last_became_peered":"2024-05-09T08:48:18.380485+0000","last_unstale":"2024-05-09T08:49:47.218018+0000","last_undegraded":"2024-05-09T08:49:47.218018+0000","last_fullsized":"2024-05-09T08:49:47.218018+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:45.079173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359369+0000","last_change":"2024-05-09T08:48:22.389667+0000","last_active":"2024-05-09T08:49:18.359369+0000","last_peered":"2024-05-09T08:49:18.359369+0000","last_clean":"2024-05-09T08:49:18.359369+0000","last_became_active":"2024-05-09T08:48:22.389268+0000","last_became_peered":"2024-05-09T08:48:22.389268+0000","last_unstale":"2024-05-09T08:49:18.359369+0000","last_undegraded":"2024-05-09T08:49:18.359369+0000","last_fullsized":"2024-05-09T08:49:18.359369+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:33:14.861855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361320+0000","last_change":"2024-05-09T08:48:17.596028+0000","last_active":"2024-05-09T08:49:18.361320+0000","last_peered":"2024-05-09T08:49:18.361320+0000","last_clean":"2024-05-09T08:49:18.361320+0000","last_became_active":"2024-05-09T08:48:17.595053+0000","last_became_peered":"2024-05-09T08:48:17.595053+0000","last_unstale":"2024-05-09T08:49:18.361320+0000","last_undegraded":"2024-05-09T08:49:18.361320+0000","last_fullsized":"2024-05-09T08:49:18.361320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:48:31.323595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.208956+0000","last_change":"2024-05-09T08:48:18.373472+0000","last_active":"2024-05-09T08:49:47.208956+0000","last_peered":"2024-05-09T08:49:47.208956+0000","last_clean":"2024-05-09T08:49:47.208956+0000","last_became_active":"2024-05-09T08:48:18.373178+0000","last_became_peered":"2024-05-09T08:48:18.373178+0000","last_unstale":"2024-05-09T08:49:47.208956+0000","last_undegraded":"2024-05-09T08:49:47.208956+0000","last_fullsized":"2024-05-09T08:49:47.208956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:18.048722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359521+0000","last_change":"2024-05-09T08:48:22.386551+0000","last_active":"2024-05-09T08:49:18.359521+0000","last_peered":"2024-05-09T08:49:18.359521+0000","last_clean":"2024-05-09T08:49:18.359521+0000","last_became_active":"2024-05-09T08:48:22.386018+0000","last_became_peered":"2024-05-09T08:48:22.386018+0000","last_unstale":"2024-05-09T08:49:18.359521+0000","last_undegraded":"2024-05-09T08:49:18.359521+0000","last_fullsized":"2024-05-09T08:49:18.359521+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:59.812030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.363858+0000","last_change":"2024-05-09T08:48:17.601207+0000","last_active":"2024-05-09T08:49:18.363858+0000","last_peered":"2024-05-09T08:49:18.363858+0000","last_clean":"2024-05-09T08:49:18.363858+0000","last_became_active":"2024-05-09T08:48:17.600878+0000","last_became_peered":"2024-05-09T08:48:17.600878+0000","last_unstale":"2024-05-09T08:49:18.363858+0000","last_undegraded":"2024-05-09T08:49:18.363858+0000","last_fullsized":"2024-05-09T08:49:18.363858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:30.040527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.218089+0000","last_change":"2024-05-09T08:48:18.381577+0000","last_active":"2024-05-09T08:49:47.218089+0000","last_peered":"2024-05-09T08:49:47.218089+0000","last_clean":"2024-05-09T08:49:47.218089+0000","last_became_active":"2024-05-09T08:48:18.381223+0000","last_became_peered":"2024-05-09T08:48:18.381223+0000","last_unstale":"2024-05-09T08:49:47.218089+0000","last_undegraded":"2024-05-09T08:49:47.218089+0000","last_fullsized":"2024-05-09T08:49:47.218089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:57.038392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359370+0000","last_change":"2024-05-09T08:48:22.385885+0000","last_active":"2024-05-09T08:49:18.359370+0000","last_peered":"2024-05-09T08:49:18.359370+0000","last_clean":"2024-05-09T08:49:18.359370+0000","last_became_active":"2024-05-09T08:48:22.385403+0000","last_became_peered":"2024-05-09T08:48:22.385403+0000","last_unstale":"2024-05-09T08:49:18.359370+0000","last_undegraded":"2024-05-09T08:49:18.359370+0000","last_fullsized":"2024-05-09T08:49:18.359370+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:53:20.733523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":213,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.826864+0000","last_change":"2024-05-09T08:48:17.597848+0000","last_active":"2024-05-09T08:49:47.826864+0000","last_peered":"2024-05-09T08:49:47.826864+0000","last_clean":"2024-05-09T08:49:47.826864+0000","last_became_active":"2024-05-09T08:48:17.596989+0000","last_became_peered":"2024-05-09T08:48:17.596989+0000","last_unstale":"2024-05-09T08:49:47.826864+0000","last_undegraded":"2024-05-09T08:49:47.826864+0000","last_fullsized":"2024-05-09T08:49:47.826864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:07.256301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60891,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":150,"num_read_kb":135,"num_write":38,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.219510+0000","last_change":"2024-05-09T08:48:18.365308+0000","last_active":"2024-05-09T08:49:47.219510+0000","last_peered":"2024-05-09T08:49:47.219510+0000","last_clean":"2024-05-09T08:49:47.219510+0000","last_became_active":"2024-05-09T08:48:18.364845+0000","last_became_peered":"2024-05-09T08:48:18.364845+0000","last_unstale":"2024-05-09T08:49:47.219510+0000","last_undegraded":"2024-05-09T08:49:47.219510+0000","last_fullsized":"2024-05-09T08:49:47.219510+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:07.277730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.364236+0000","last_change":"2024-05-09T08:48:22.377333+0000","last_active":"2024-05-09T08:49:18.364236+0000","last_peered":"2024-05-09T08:49:18.364236+0000","last_clean":"2024-05-09T08:49:18.364236+0000","last_became_active":"2024-05-09T08:48:22.376494+0000","last_became_peered":"2024-05-09T08:48:22.376494+0000","last_unstale":"2024-05-09T08:49:18.364236+0000","last_undegraded":"2024-05-09T08:49:18.364236+0000","last_fullsized":"2024-05-09T08:49:18.364236+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:49:17.693977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362494+0000","last_change":"2024-05-09T08:48:17.600749+0000","last_active":"2024-05-09T08:49:18.362494+0000","last_peered":"2024-05-09T08:49:18.362494+0000","last_clean":"2024-05-09T08:49:18.362494+0000","last_became_active":"2024-05-09T08:48:17.600402+0000","last_became_peered":"2024-05-09T08:48:17.600402+0000","last_unstale":"2024-05-09T08:49:18.362494+0000","last_undegraded":"2024-05-09T08:49:18.362494+0000","last_fullsized":"2024-05-09T08:49:18.362494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:42:14.889227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.206098+0000","last_change":"2024-05-09T08:48:18.373652+0000","last_active":"2024-05-09T08:49:47.206098+0000","last_peered":"2024-05-09T08:49:47.206098+0000","last_clean":"2024-05-09T08:49:47.206098+0000","last_became_active":"2024-05-09T08:48:18.373345+0000","last_became_peered":"2024-05-09T08:48:18.373345+0000","last_unstale":"2024-05-09T08:49:47.206098+0000","last_undegraded":"2024-05-09T08:49:47.206098+0000","last_fullsized":"2024-05-09T08:49:47.206098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:14:34.843305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.369432+0000","last_change":"2024-05-09T08:49:18.369432+0000","last_active":"2024-05-09T08:49:18.369432+0000","last_peered":"2024-05-09T08:49:18.369432+0000","last_clean":"2024-05-09T08:49:18.369432+0000","last_became_active":"2024-05-09T08:49:18.368978+0000","last_became_peered":"2024-05-09T08:49:18.368978+0000","last_unstale":"2024-05-09T08:49:18.369432+0000","last_undegraded":"2024-05-09T08:49:18.369432+0000","last_fullsized":"2024-05-09T08:49:18.369432+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:20.969162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.363543+0000","last_change":"2024-05-09T08:48:17.592658+0000","last_active":"2024-05-09T08:49:18.363543+0000","last_peered":"2024-05-09T08:49:18.363543+0000","last_clean":"2024-05-09T08:49:18.363543+0000","last_became_active":"2024-05-09T08:48:17.592275+0000","last_became_peered":"2024-05-09T08:48:17.592275+0000","last_unstale":"2024-05-09T08:49:18.363543+0000","last_undegraded":"2024-05-09T08:49:18.363543+0000","last_fullsized":"2024-05-09T08:49:18.363543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:26:56.742079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.218875+0000","last_change":"2024-05-09T08:49:18.364399+0000","last_active":"2024-05-09T08:49:47.218875+0000","last_peered":"2024-05-09T08:49:47.218875+0000","last_clean":"2024-05-09T08:49:47.218875+0000","last_became_active":"2024-05-09T08:49:18.364166+0000","last_became_peered":"2024-05-09T08:49:18.364166+0000","last_unstale":"2024-05-09T08:49:47.218875+0000","last_undegraded":"2024-05-09T08:49:47.218875+0000","last_fullsized":"2024-05-09T08:49:47.218875+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:49:36.480337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359578+0000","last_change":"2024-05-09T08:48:22.389568+0000","last_active":"2024-05-09T08:49:18.359578+0000","last_peered":"2024-05-09T08:49:18.359578+0000","last_clean":"2024-05-09T08:49:18.359578+0000","last_became_active":"2024-05-09T08:48:22.387192+0000","last_became_peered":"2024-05-09T08:48:22.387192+0000","last_unstale":"2024-05-09T08:49:18.359578+0000","last_undegraded":"2024-05-09T08:49:18.359578+0000","last_fullsized":"2024-05-09T08:49:18.359578+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:56.794208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"92'7","reported_seq":110,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.828167+0000","last_change":"2024-05-09T08:48:17.598760+0000","last_active":"2024-05-09T08:49:47.828167+0000","last_peered":"2024-05-09T08:49:47.828167+0000","last_clean":"2024-05-09T08:49:47.828167+0000","last_became_active":"2024-05-09T08:48:17.598192+0000","last_became_peered":"2024-05-09T08:48:17.598192+0000","last_unstale":"2024-05-09T08:49:47.828167+0000","last_undegraded":"2024-05-09T08:49:47.828167+0000","last_fullsized":"2024-05-09T08:49:47.828167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:29.007644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":77,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.216235+0000","last_change":"2024-05-09T08:48:18.364570+0000","last_active":"2024-05-09T08:49:47.216235+0000","last_peered":"2024-05-09T08:49:47.216235+0000","last_clean":"2024-05-09T08:49:47.216235+0000","last_became_active":"2024-05-09T08:48:18.364191+0000","last_became_peered":"2024-05-09T08:48:18.364191+0000","last_unstale":"2024-05-09T08:49:47.216235+0000","last_undegraded":"2024-05-09T08:49:47.216235+0000","last_fullsized":"2024-05-09T08:49:47.216235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:49.267954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.694742+0000","last_change":"2024-05-09T08:49:18.694742+0000","last_active":"2024-05-09T08:49:18.694742+0000","last_peered":"2024-05-09T08:49:18.694742+0000","last_clean":"2024-05-09T08:49:18.694742+0000","last_became_active":"2024-05-09T08:49:18.694241+0000","last_became_peered":"2024-05-09T08:49:18.694241+0000","last_unstale":"2024-05-09T08:49:18.694742+0000","last_undegraded":"2024-05-09T08:49:18.694742+0000","last_fullsized":"2024-05-09T08:49:18.694742+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:20.289651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.367007+0000","last_change":"2024-05-09T08:48:17.599762+0000","last_active":"2024-05-09T08:49:18.367007+0000","last_peered":"2024-05-09T08:49:18.367007+0000","last_clean":"2024-05-09T08:49:18.367007+0000","last_became_active":"2024-05-09T08:48:17.599355+0000","last_became_peered":"2024-05-09T08:48:17.599355+0000","last_unstale":"2024-05-09T08:49:18.367007+0000","last_undegraded":"2024-05-09T08:49:18.367007+0000","last_fullsized":"2024-05-09T08:49:18.367007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:28:36.836403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.208503+0000","last_change":"2024-05-09T08:48:18.372810+0000","last_active":"2024-05-09T08:49:47.208503+0000","last_peered":"2024-05-09T08:49:47.208503+0000","last_clean":"2024-05-09T08:49:47.208503+0000","last_became_active":"2024-05-09T08:48:18.372437+0000","last_became_peered":"2024-05-09T08:48:18.372437+0000","last_unstale":"2024-05-09T08:49:47.208503+0000","last_undegraded":"2024-05-09T08:49:47.208503+0000","last_fullsized":"2024-05-09T08:49:47.208503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:03:36.879719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.363895+0000","last_change":"2024-05-09T08:48:22.379370+0000","last_active":"2024-05-09T08:49:18.363895+0000","last_peered":"2024-05-09T08:49:18.363895+0000","last_clean":"2024-05-09T08:49:18.363895+0000","last_became_active":"2024-05-09T08:48:22.378994+0000","last_became_peered":"2024-05-09T08:48:22.378994+0000","last_unstale":"2024-05-09T08:49:18.363895+0000","last_undegraded":"2024-05-09T08:49:18.363895+0000","last_fullsized":"2024-05-09T08:49:18.363895+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:08.005647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.366906+0000","last_change":"2024-05-09T08:48:17.594236+0000","last_active":"2024-05-09T08:49:18.366906+0000","last_peered":"2024-05-09T08:49:18.366906+0000","last_clean":"2024-05-09T08:49:18.366906+0000","last_became_active":"2024-05-09T08:48:17.593418+0000","last_became_peered":"2024-05-09T08:48:17.593418+0000","last_unstale":"2024-05-09T08:49:18.366906+0000","last_undegraded":"2024-05-09T08:49:18.366906+0000","last_fullsized":"2024-05-09T08:49:18.366906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:23.930421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.203573+0000","last_change":"2024-05-09T08:48:18.366767+0000","last_active":"2024-05-09T08:49:47.203573+0000","last_peered":"2024-05-09T08:49:47.203573+0000","last_clean":"2024-05-09T08:49:47.203573+0000","last_became_active":"2024-05-09T08:48:18.366435+0000","last_became_peered":"2024-05-09T08:48:18.366435+0000","last_unstale":"2024-05-09T08:49:47.203573+0000","last_undegraded":"2024-05-09T08:49:47.203573+0000","last_fullsized":"2024-05-09T08:49:47.203573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:45.111510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.366355+0000","last_change":"2024-05-09T08:48:22.378642+0000","last_active":"2024-05-09T08:49:18.366355+0000","last_peered":"2024-05-09T08:49:18.366355+0000","last_clean":"2024-05-09T08:49:18.366355+0000","last_became_active":"2024-05-09T08:48:22.378161+0000","last_became_peered":"2024-05-09T08:48:22.378161+0000","last_unstale":"2024-05-09T08:49:18.366355+0000","last_undegraded":"2024-05-09T08:49:18.366355+0000","last_fullsized":"2024-05-09T08:49:18.366355+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:53.149728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359658+0000","last_change":"2024-05-09T08:48:17.599499+0000","last_active":"2024-05-09T08:49:18.359658+0000","last_peered":"2024-05-09T08:49:18.359658+0000","last_clean":"2024-05-09T08:49:18.359658+0000","last_became_active":"2024-05-09T08:48:17.599145+0000","last_became_peered":"2024-05-09T08:48:17.599145+0000","last_unstale":"2024-05-09T08:49:18.359658+0000","last_undegraded":"2024-05-09T08:49:18.359658+0000","last_fullsized":"2024-05-09T08:49:18.359658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:31:12.502004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.210067+0000","last_change":"2024-05-09T08:48:18.372355+0000","last_active":"2024-05-09T08:49:47.210067+0000","last_peered":"2024-05-09T08:49:47.210067+0000","last_clean":"2024-05-09T08:49:47.210067+0000","last_became_active":"2024-05-09T08:48:18.367743+0000","last_became_peered":"2024-05-09T08:48:18.367743+0000","last_unstale":"2024-05-09T08:49:47.210067+0000","last_undegraded":"2024-05-09T08:49:47.210067+0000","last_fullsized":"2024-05-09T08:49:47.210067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:31:12.174350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.363563+0000","last_change":"2024-05-09T08:48:22.384064+0000","last_active":"2024-05-09T08:49:18.363563+0000","last_peered":"2024-05-09T08:49:18.363563+0000","last_clean":"2024-05-09T08:49:18.363563+0000","last_became_active":"2024-05-09T08:48:22.383670+0000","last_became_peered":"2024-05-09T08:48:22.383670+0000","last_unstale":"2024-05-09T08:49:18.363563+0000","last_undegraded":"2024-05-09T08:49:18.363563+0000","last_fullsized":"2024-05-09T08:49:18.363563+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:51:35.000827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":103,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.857481+0000","last_change":"2024-05-09T08:48:17.601596+0000","last_active":"2024-05-09T08:49:47.857481+0000","last_peered":"2024-05-09T08:49:47.857481+0000","last_clean":"2024-05-09T08:49:47.857481+0000","last_became_active":"2024-05-09T08:48:17.601250+0000","last_became_peered":"2024-05-09T08:48:17.601250+0000","last_unstale":"2024-05-09T08:49:47.857481+0000","last_undegraded":"2024-05-09T08:49:47.857481+0000","last_fullsized":"2024-05-09T08:49:47.857481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:48.237336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.213252+0000","last_change":"2024-05-09T08:48:18.368751+0000","last_active":"2024-05-09T08:49:47.213252+0000","last_peered":"2024-05-09T08:49:47.213252+0000","last_clean":"2024-05-09T08:49:47.213252+0000","last_became_active":"2024-05-09T08:48:18.367463+0000","last_became_peered":"2024-05-09T08:48:18.367463+0000","last_unstale":"2024-05-09T08:49:47.213252+0000","last_undegraded":"2024-05-09T08:49:47.213252+0000","last_fullsized":"2024-05-09T08:49:47.213252+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:19:17.696602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.363573+0000","last_change":"2024-05-09T08:48:22.385025+0000","last_active":"2024-05-09T08:49:18.363573+0000","last_peered":"2024-05-09T08:49:18.363573+0000","last_clean":"2024-05-09T08:49:18.363573+0000","last_became_active":"2024-05-09T08:48:22.384633+0000","last_became_peered":"2024-05-09T08:48:22.384633+0000","last_unstale":"2024-05-09T08:49:18.363573+0000","last_undegraded":"2024-05-09T08:49:18.363573+0000","last_fullsized":"2024-05-09T08:49:18.363573+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:17.804075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.210920+0000","last_change":"2024-05-09T08:48:18.376741+0000","last_active":"2024-05-09T08:49:47.210920+0000","last_peered":"2024-05-09T08:49:47.210920+0000","last_clean":"2024-05-09T08:49:47.210920+0000","last_became_active":"2024-05-09T08:48:18.376376+0000","last_became_peered":"2024-05-09T08:48:18.376376+0000","last_unstale":"2024-05-09T08:49:47.210920+0000","last_undegraded":"2024-05-09T08:49:47.210920+0000","last_fullsized":"2024-05-09T08:49:47.210920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:58:31.717555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'6","reported_seq":107,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:48.828866+0000","last_change":"2024-05-09T08:48:17.598316+0000","last_active":"2024-05-09T08:49:48.828866+0000","last_peered":"2024-05-09T08:49:48.828866+0000","last_clean":"2024-05-09T08:49:48.828866+0000","last_became_active":"2024-05-09T08:48:17.598022+0000","last_became_peered":"2024-05-09T08:48:17.598022+0000","last_unstale":"2024-05-09T08:49:48.828866+0000","last_undegraded":"2024-05-09T08:49:48.828866+0000","last_fullsized":"2024-05-09T08:49:48.828866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:04:10.454894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.492981+0000","last_change":"2024-05-09T08:49:18.492981+0000","last_active":"2024-05-09T08:49:18.492981+0000","last_peered":"2024-05-09T08:49:18.492981+0000","last_clean":"2024-05-09T08:49:18.492981+0000","last_became_active":"2024-05-09T08:49:18.489169+0000","last_became_peered":"2024-05-09T08:49:18.489169+0000","last_unstale":"2024-05-09T08:49:18.492981+0000","last_undegraded":"2024-05-09T08:49:18.492981+0000","last_fullsized":"2024-05-09T08:49:18.492981+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:02:53.955587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.209575+0000","last_change":"2024-05-09T08:48:18.379203+0000","last_active":"2024-05-09T08:49:47.209575+0000","last_peered":"2024-05-09T08:49:47.209575+0000","last_clean":"2024-05-09T08:49:47.209575+0000","last_became_active":"2024-05-09T08:48:18.378885+0000","last_became_peered":"2024-05-09T08:48:18.378885+0000","last_unstale":"2024-05-09T08:49:47.209575+0000","last_undegraded":"2024-05-09T08:49:47.209575+0000","last_fullsized":"2024-05-09T08:49:47.209575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:40.588263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359336+0000","last_change":"2024-05-09T08:48:17.593520+0000","last_active":"2024-05-09T08:49:18.359336+0000","last_peered":"2024-05-09T08:49:18.359336+0000","last_clean":"2024-05-09T08:49:18.359336+0000","last_became_active":"2024-05-09T08:48:17.593127+0000","last_became_peered":"2024-05-09T08:48:17.593127+0000","last_unstale":"2024-05-09T08:49:18.359336+0000","last_undegraded":"2024-05-09T08:49:18.359336+0000","last_fullsized":"2024-05-09T08:49:18.359336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:34:29.567166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":361,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361176+0000","last_change":"2024-05-09T08:47:05.718649+0000","last_active":"2024-05-09T08:49:18.361176+0000","last_peered":"2024-05-09T08:49:18.361176+0000","last_clean":"2024-05-09T08:49:18.361176+0000","last_became_active":"2024-05-09T08:47:05.718088+0000","last_became_peered":"2024-05-09T08:47:05.718088+0000","last_unstale":"2024-05-09T08:49:18.361176+0000","last_undegraded":"2024-05-09T08:49:18.361176+0000","last_fullsized":"2024-05-09T08:49:18.361176+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:43:39.457880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:43:39.457880+0000","last_clean_scrub_stamp":"2024-05-09T08:43:39.457880+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:14.227590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362973+0000","last_change":"2024-05-09T08:48:22.376751+0000","last_active":"2024-05-09T08:49:18.362973+0000","last_peered":"2024-05-09T08:49:18.362973+0000","last_clean":"2024-05-09T08:49:18.362973+0000","last_became_active":"2024-05-09T08:48:22.376322+0000","last_became_peered":"2024-05-09T08:48:22.376322+0000","last_unstale":"2024-05-09T08:49:18.362973+0000","last_undegraded":"2024-05-09T08:49:18.362973+0000","last_fullsized":"2024-05-09T08:49:18.362973+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:08:07.859034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.197130+0000","last_change":"2024-05-09T08:48:18.364063+0000","last_active":"2024-05-09T08:49:47.197130+0000","last_peered":"2024-05-09T08:49:47.197130+0000","last_clean":"2024-05-09T08:49:47.197130+0000","last_became_active":"2024-05-09T08:48:18.363694+0000","last_became_peered":"2024-05-09T08:48:18.363694+0000","last_unstale":"2024-05-09T08:49:47.197130+0000","last_undegraded":"2024-05-09T08:49:47.197130+0000","last_fullsized":"2024-05-09T08:49:47.197130+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:42.536329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361351+0000","last_change":"2024-05-09T08:48:17.599787+0000","last_active":"2024-05-09T08:49:18.361351+0000","last_peered":"2024-05-09T08:49:18.361351+0000","last_clean":"2024-05-09T08:49:18.361351+0000","last_became_active":"2024-05-09T08:48:17.599248+0000","last_became_peered":"2024-05-09T08:48:17.599248+0000","last_unstale":"2024-05-09T08:49:18.361351+0000","last_undegraded":"2024-05-09T08:49:18.361351+0000","last_fullsized":"2024-05-09T08:49:18.361351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:17.448404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362948+0000","last_change":"2024-05-09T08:48:22.378913+0000","last_active":"2024-05-09T08:49:18.362948+0000","last_peered":"2024-05-09T08:49:18.362948+0000","last_clean":"2024-05-09T08:49:18.362948+0000","last_became_active":"2024-05-09T08:48:22.378441+0000","last_became_peered":"2024-05-09T08:48:22.378441+0000","last_unstale":"2024-05-09T08:49:18.362948+0000","last_undegraded":"2024-05-09T08:49:18.362948+0000","last_fullsized":"2024-05-09T08:49:18.362948+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:49.623778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.367313+0000","last_change":"2024-05-09T08:48:17.600121+0000","last_active":"2024-05-09T08:49:18.367313+0000","last_peered":"2024-05-09T08:49:18.367313+0000","last_clean":"2024-05-09T08:49:18.367313+0000","last_became_active":"2024-05-09T08:48:17.599800+0000","last_became_peered":"2024-05-09T08:48:17.599800+0000","last_unstale":"2024-05-09T08:49:18.367313+0000","last_undegraded":"2024-05-09T08:49:18.367313+0000","last_fullsized":"2024-05-09T08:49:18.367313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:56.607393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.197288+0000","last_change":"2024-05-09T08:48:18.366612+0000","last_active":"2024-05-09T08:49:47.197288+0000","last_peered":"2024-05-09T08:49:47.197288+0000","last_clean":"2024-05-09T08:49:47.197288+0000","last_became_active":"2024-05-09T08:48:18.366266+0000","last_became_peered":"2024-05-09T08:48:18.366266+0000","last_unstale":"2024-05-09T08:49:47.197288+0000","last_undegraded":"2024-05-09T08:49:47.197288+0000","last_fullsized":"2024-05-09T08:49:47.197288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:35:01.663456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.692585+0000","last_change":"2024-05-09T08:48:22.384096+0000","last_active":"2024-05-09T08:49:18.692585+0000","last_peered":"2024-05-09T08:49:18.692585+0000","last_clean":"2024-05-09T08:49:18.692585+0000","last_became_active":"2024-05-09T08:48:22.383656+0000","last_became_peered":"2024-05-09T08:48:22.383656+0000","last_unstale":"2024-05-09T08:49:18.692585+0000","last_undegraded":"2024-05-09T08:49:18.692585+0000","last_fullsized":"2024-05-09T08:49:18.692585+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:13.622377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'11","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359899+0000","last_change":"2024-05-09T08:48:17.593684+0000","last_active":"2024-05-09T08:49:18.359899+0000","last_peered":"2024-05-09T08:49:18.359899+0000","last_clean":"2024-05-09T08:49:18.359899+0000","last_became_active":"2024-05-09T08:48:17.593358+0000","last_became_peered":"2024-05-09T08:48:17.593358+0000","last_unstale":"2024-05-09T08:49:18.359899+0000","last_undegraded":"2024-05-09T08:49:18.359899+0000","last_fullsized":"2024-05-09T08:49:18.359899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:51:36.023421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.216137+0000","last_change":"2024-05-09T08:48:18.374498+0000","last_active":"2024-05-09T08:49:47.216137+0000","last_peered":"2024-05-09T08:49:47.216137+0000","last_clean":"2024-05-09T08:49:47.216137+0000","last_became_active":"2024-05-09T08:48:18.374142+0000","last_became_peered":"2024-05-09T08:48:18.374142+0000","last_unstale":"2024-05-09T08:49:47.216137+0000","last_undegraded":"2024-05-09T08:49:47.216137+0000","last_fullsized":"2024-05-09T08:49:47.216137+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:53.874419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361533+0000","last_change":"2024-05-09T08:48:22.378330+0000","last_active":"2024-05-09T08:49:18.361533+0000","last_peered":"2024-05-09T08:49:18.361533+0000","last_clean":"2024-05-09T08:49:18.361533+0000","last_became_active":"2024-05-09T08:48:22.377218+0000","last_became_peered":"2024-05-09T08:48:22.377218+0000","last_unstale":"2024-05-09T08:49:18.361533+0000","last_undegraded":"2024-05-09T08:49:18.361533+0000","last_fullsized":"2024-05-09T08:49:18.361533+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:41.650168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359666+0000","last_change":"2024-05-09T08:48:17.601410+0000","last_active":"2024-05-09T08:49:18.359666+0000","last_peered":"2024-05-09T08:49:18.359666+0000","last_clean":"2024-05-09T08:49:18.359666+0000","last_became_active":"2024-05-09T08:48:17.601075+0000","last_became_peered":"2024-05-09T08:48:17.601075+0000","last_unstale":"2024-05-09T08:49:18.359666+0000","last_undegraded":"2024-05-09T08:49:18.359666+0000","last_fullsized":"2024-05-09T08:49:18.359666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:45.760280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.213517+0000","last_change":"2024-05-09T08:48:18.379590+0000","last_active":"2024-05-09T08:49:47.213517+0000","last_peered":"2024-05-09T08:49:47.213517+0000","last_clean":"2024-05-09T08:49:47.213517+0000","last_became_active":"2024-05-09T08:48:18.379187+0000","last_became_peered":"2024-05-09T08:48:18.379187+0000","last_unstale":"2024-05-09T08:49:47.213517+0000","last_undegraded":"2024-05-09T08:49:47.213517+0000","last_fullsized":"2024-05-09T08:49:47.213517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:40:02.630449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359474+0000","last_change":"2024-05-09T08:48:22.386437+0000","last_active":"2024-05-09T08:49:18.359474+0000","last_peered":"2024-05-09T08:49:18.359474+0000","last_clean":"2024-05-09T08:49:18.359474+0000","last_became_active":"2024-05-09T08:48:22.385973+0000","last_became_peered":"2024-05-09T08:48:22.385973+0000","last_unstale":"2024-05-09T08:49:18.359474+0000","last_undegraded":"2024-05-09T08:49:18.359474+0000","last_fullsized":"2024-05-09T08:49:18.359474+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:11:37.338641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362495+0000","last_change":"2024-05-09T08:48:17.598586+0000","last_active":"2024-05-09T08:49:18.362495+0000","last_peered":"2024-05-09T08:49:18.362495+0000","last_clean":"2024-05-09T08:49:18.362495+0000","last_became_active":"2024-05-09T08:48:17.597851+0000","last_became_peered":"2024-05-09T08:48:17.597851+0000","last_unstale":"2024-05-09T08:49:18.362495+0000","last_undegraded":"2024-05-09T08:49:18.362495+0000","last_fullsized":"2024-05-09T08:49:18.362495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:28:22.996999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.173783+0000","last_change":"2024-05-09T08:48:18.367319+0000","last_active":"2024-05-09T08:49:47.173783+0000","last_peered":"2024-05-09T08:49:47.173783+0000","last_clean":"2024-05-09T08:49:47.173783+0000","last_became_active":"2024-05-09T08:48:18.366034+0000","last_became_peered":"2024-05-09T08:48:18.366034+0000","last_unstale":"2024-05-09T08:49:47.173783+0000","last_undegraded":"2024-05-09T08:49:47.173783+0000","last_fullsized":"2024-05-09T08:49:47.173783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:22.998089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359806+0000","last_change":"2024-05-09T08:48:22.389129+0000","last_active":"2024-05-09T08:49:18.359806+0000","last_peered":"2024-05-09T08:49:18.359806+0000","last_clean":"2024-05-09T08:49:18.359806+0000","last_became_active":"2024-05-09T08:48:22.386639+0000","last_became_peered":"2024-05-09T08:48:22.386639+0000","last_unstale":"2024-05-09T08:49:18.359806+0000","last_undegraded":"2024-05-09T08:49:18.359806+0000","last_fullsized":"2024-05-09T08:49:18.359806+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:47:07.257464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.693551+0000","last_change":"2024-05-09T08:48:17.595255+0000","last_active":"2024-05-09T08:49:18.693551+0000","last_peered":"2024-05-09T08:49:18.693551+0000","last_clean":"2024-05-09T08:49:18.693551+0000","last_became_active":"2024-05-09T08:48:17.594882+0000","last_became_peered":"2024-05-09T08:48:17.594882+0000","last_unstale":"2024-05-09T08:49:18.693551+0000","last_undegraded":"2024-05-09T08:49:18.693551+0000","last_fullsized":"2024-05-09T08:49:18.693551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:49.845326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.211374+0000","last_change":"2024-05-09T08:48:18.378580+0000","last_active":"2024-05-09T08:49:47.211374+0000","last_peered":"2024-05-09T08:49:47.211374+0000","last_clean":"2024-05-09T08:49:47.211374+0000","last_became_active":"2024-05-09T08:48:18.378205+0000","last_became_peered":"2024-05-09T08:48:18.378205+0000","last_unstale":"2024-05-09T08:49:47.211374+0000","last_undegraded":"2024-05-09T08:49:47.211374+0000","last_fullsized":"2024-05-09T08:49:47.211374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:50.103195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359371+0000","last_change":"2024-05-09T08:48:22.378494+0000","last_active":"2024-05-09T08:49:18.359371+0000","last_peered":"2024-05-09T08:49:18.359371+0000","last_clean":"2024-05-09T08:49:18.359371+0000","last_became_active":"2024-05-09T08:48:22.378076+0000","last_became_peered":"2024-05-09T08:48:22.378076+0000","last_unstale":"2024-05-09T08:49:18.359371+0000","last_undegraded":"2024-05-09T08:49:18.359371+0000","last_fullsized":"2024-05-09T08:49:18.359371+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:40:09.079968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362143+0000","last_change":"2024-05-09T08:48:17.601433+0000","last_active":"2024-05-09T08:49:18.362143+0000","last_peered":"2024-05-09T08:49:18.362143+0000","last_clean":"2024-05-09T08:49:18.362143+0000","last_became_active":"2024-05-09T08:48:17.601128+0000","last_became_peered":"2024-05-09T08:48:17.601128+0000","last_unstale":"2024-05-09T08:49:18.362143+0000","last_undegraded":"2024-05-09T08:49:18.362143+0000","last_fullsized":"2024-05-09T08:49:18.362143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:33:26.207437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.216272+0000","last_change":"2024-05-09T08:48:18.373569+0000","last_active":"2024-05-09T08:49:47.216272+0000","last_peered":"2024-05-09T08:49:47.216272+0000","last_clean":"2024-05-09T08:49:47.216272+0000","last_became_active":"2024-05-09T08:48:18.373238+0000","last_became_peered":"2024-05-09T08:48:18.373238+0000","last_unstale":"2024-05-09T08:49:47.216272+0000","last_undegraded":"2024-05-09T08:49:47.216272+0000","last_fullsized":"2024-05-09T08:49:47.216272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:17:41.629419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359480+0000","last_change":"2024-05-09T08:48:22.381523+0000","last_active":"2024-05-09T08:49:18.359480+0000","last_peered":"2024-05-09T08:49:18.359480+0000","last_clean":"2024-05-09T08:49:18.359480+0000","last_became_active":"2024-05-09T08:48:22.380626+0000","last_became_peered":"2024-05-09T08:48:22.380626+0000","last_unstale":"2024-05-09T08:49:18.359480+0000","last_undegraded":"2024-05-09T08:49:18.359480+0000","last_fullsized":"2024-05-09T08:49:18.359480+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:47:45.092232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.366669+0000","last_change":"2024-05-09T08:48:17.601479+0000","last_active":"2024-05-09T08:49:18.366669+0000","last_peered":"2024-05-09T08:49:18.366669+0000","last_clean":"2024-05-09T08:49:18.366669+0000","last_became_active":"2024-05-09T08:48:17.601140+0000","last_became_peered":"2024-05-09T08:48:17.601140+0000","last_unstale":"2024-05-09T08:49:18.366669+0000","last_undegraded":"2024-05-09T08:49:18.366669+0000","last_fullsized":"2024-05-09T08:49:18.366669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:38:24.011209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.219155+0000","last_change":"2024-05-09T08:48:18.376890+0000","last_active":"2024-05-09T08:49:47.219155+0000","last_peered":"2024-05-09T08:49:47.219155+0000","last_clean":"2024-05-09T08:49:47.219155+0000","last_became_active":"2024-05-09T08:48:18.376472+0000","last_became_peered":"2024-05-09T08:48:18.376472+0000","last_unstale":"2024-05-09T08:49:47.219155+0000","last_undegraded":"2024-05-09T08:49:47.219155+0000","last_fullsized":"2024-05-09T08:49:47.219155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:40:16.238539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362579+0000","last_change":"2024-05-09T08:48:22.374743+0000","last_active":"2024-05-09T08:49:18.362579+0000","last_peered":"2024-05-09T08:49:18.362579+0000","last_clean":"2024-05-09T08:49:18.362579+0000","last_became_active":"2024-05-09T08:48:22.374324+0000","last_became_peered":"2024-05-09T08:48:22.374324+0000","last_unstale":"2024-05-09T08:49:18.362579+0000","last_undegraded":"2024-05-09T08:49:18.362579+0000","last_fullsized":"2024-05-09T08:49:18.362579+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:48.073647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":104,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.859977+0000","last_change":"2024-05-09T08:48:17.604700+0000","last_active":"2024-05-09T08:49:47.859977+0000","last_peered":"2024-05-09T08:49:47.859977+0000","last_clean":"2024-05-09T08:49:47.859977+0000","last_became_active":"2024-05-09T08:48:17.604334+0000","last_became_peered":"2024-05-09T08:48:17.604334+0000","last_unstale":"2024-05-09T08:49:47.859977+0000","last_undegraded":"2024-05-09T08:49:47.859977+0000","last_fullsized":"2024-05-09T08:49:47.859977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:15:37.346232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.219459+0000","last_change":"2024-05-09T08:48:18.375769+0000","last_active":"2024-05-09T08:49:47.219459+0000","last_peered":"2024-05-09T08:49:47.219459+0000","last_clean":"2024-05-09T08:49:47.219459+0000","last_became_active":"2024-05-09T08:48:18.375059+0000","last_became_peered":"2024-05-09T08:48:18.375059+0000","last_unstale":"2024-05-09T08:49:47.219459+0000","last_undegraded":"2024-05-09T08:49:47.219459+0000","last_fullsized":"2024-05-09T08:49:47.219459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:51.001240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362302+0000","last_change":"2024-05-09T08:48:22.385306+0000","last_active":"2024-05-09T08:49:18.362302+0000","last_peered":"2024-05-09T08:49:18.362302+0000","last_clean":"2024-05-09T08:49:18.362302+0000","last_became_active":"2024-05-09T08:48:22.383890+0000","last_became_peered":"2024-05-09T08:48:22.383890+0000","last_unstale":"2024-05-09T08:49:18.362302+0000","last_undegraded":"2024-05-09T08:49:18.362302+0000","last_fullsized":"2024-05-09T08:49:18.362302+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:50:52.369399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.364051+0000","last_change":"2024-05-09T08:48:17.591627+0000","last_active":"2024-05-09T08:49:18.364051+0000","last_peered":"2024-05-09T08:49:18.364051+0000","last_clean":"2024-05-09T08:49:18.364051+0000","last_became_active":"2024-05-09T08:48:17.591250+0000","last_became_peered":"2024-05-09T08:48:17.591250+0000","last_unstale":"2024-05-09T08:49:18.364051+0000","last_undegraded":"2024-05-09T08:49:18.364051+0000","last_fullsized":"2024-05-09T08:49:18.364051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:40:59.748892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.218081+0000","last_change":"2024-05-09T08:48:18.372256+0000","last_active":"2024-05-09T08:49:47.218081+0000","last_peered":"2024-05-09T08:49:47.218081+0000","last_clean":"2024-05-09T08:49:47.218081+0000","last_became_active":"2024-05-09T08:48:18.371895+0000","last_became_peered":"2024-05-09T08:48:18.371895+0000","last_unstale":"2024-05-09T08:49:47.218081+0000","last_undegraded":"2024-05-09T08:49:47.218081+0000","last_fullsized":"2024-05-09T08:49:47.218081+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:25.067587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361711+0000","last_change":"2024-05-09T08:48:22.373684+0000","last_active":"2024-05-09T08:49:18.361711+0000","last_peered":"2024-05-09T08:49:18.361711+0000","last_clean":"2024-05-09T08:49:18.361711+0000","last_became_active":"2024-05-09T08:48:22.373000+0000","last_became_peered":"2024-05-09T08:48:22.373000+0000","last_unstale":"2024-05-09T08:49:18.361711+0000","last_undegraded":"2024-05-09T08:49:18.361711+0000","last_fullsized":"2024-05-09T08:49:18.361711+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:53:14.416837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.363742+0000","last_change":"2024-05-09T08:48:17.596910+0000","last_active":"2024-05-09T08:49:18.363742+0000","last_peered":"2024-05-09T08:49:18.363742+0000","last_clean":"2024-05-09T08:49:18.363742+0000","last_became_active":"2024-05-09T08:48:17.596605+0000","last_became_peered":"2024-05-09T08:48:17.596605+0000","last_unstale":"2024-05-09T08:49:18.363742+0000","last_undegraded":"2024-05-09T08:49:18.363742+0000","last_fullsized":"2024-05-09T08:49:18.363742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:05:25.880082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.218708+0000","last_change":"2024-05-09T08:48:18.370354+0000","last_active":"2024-05-09T08:49:47.218708+0000","last_peered":"2024-05-09T08:49:47.218708+0000","last_clean":"2024-05-09T08:49:47.218708+0000","last_became_active":"2024-05-09T08:48:18.370043+0000","last_became_peered":"2024-05-09T08:48:18.370043+0000","last_unstale":"2024-05-09T08:49:47.218708+0000","last_undegraded":"2024-05-09T08:49:47.218708+0000","last_fullsized":"2024-05-09T08:49:47.218708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:21:34.066162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.209350+0000","last_change":"2024-05-09T08:48:18.376431+0000","last_active":"2024-05-09T08:49:47.209350+0000","last_peered":"2024-05-09T08:49:47.209350+0000","last_clean":"2024-05-09T08:49:47.209350+0000","last_became_active":"2024-05-09T08:48:18.375598+0000","last_became_peered":"2024-05-09T08:48:18.375598+0000","last_unstale":"2024-05-09T08:49:47.209350+0000","last_undegraded":"2024-05-09T08:49:47.209350+0000","last_fullsized":"2024-05-09T08:49:47.209350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:51:13.481219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.211305+0000","last_change":"2024-05-09T08:48:18.371460+0000","last_active":"2024-05-09T08:49:47.211305+0000","last_peered":"2024-05-09T08:49:47.211305+0000","last_clean":"2024-05-09T08:49:47.211305+0000","last_became_active":"2024-05-09T08:48:18.371091+0000","last_became_peered":"2024-05-09T08:48:18.371091+0000","last_unstale":"2024-05-09T08:49:47.211305+0000","last_undegraded":"2024-05-09T08:49:47.211305+0000","last_fullsized":"2024-05-09T08:49:47.211305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:02.829902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.215028+0000","last_change":"2024-05-09T08:48:18.369453+0000","last_active":"2024-05-09T08:49:47.215028+0000","last_peered":"2024-05-09T08:49:47.215028+0000","last_clean":"2024-05-09T08:49:47.215028+0000","last_became_active":"2024-05-09T08:48:18.368966+0000","last_became_peered":"2024-05-09T08:48:18.368966+0000","last_unstale":"2024-05-09T08:49:47.215028+0000","last_undegraded":"2024-05-09T08:49:47.215028+0000","last_fullsized":"2024-05-09T08:49:47.215028+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:23:14.359222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.210404+0000","last_change":"2024-05-09T08:48:18.378815+0000","last_active":"2024-05-09T08:49:47.210404+0000","last_peered":"2024-05-09T08:49:47.210404+0000","last_clean":"2024-05-09T08:49:47.210404+0000","last_became_active":"2024-05-09T08:48:18.378479+0000","last_became_peered":"2024-05-09T08:48:18.378479+0000","last_unstale":"2024-05-09T08:49:47.210404+0000","last_undegraded":"2024-05-09T08:49:47.210404+0000","last_fullsized":"2024-05-09T08:49:47.210404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:38:50.133239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.214360+0000","last_change":"2024-05-09T08:48:18.365696+0000","last_active":"2024-05-09T08:49:47.214360+0000","last_peered":"2024-05-09T08:49:47.214360+0000","last_clean":"2024-05-09T08:49:47.214360+0000","last_became_active":"2024-05-09T08:48:18.365358+0000","last_became_peered":"2024-05-09T08:48:18.365358+0000","last_unstale":"2024-05-09T08:49:47.214360+0000","last_undegraded":"2024-05-09T08:49:47.214360+0000","last_fullsized":"2024-05-09T08:49:47.214360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:04.064150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.204531+0000","last_change":"2024-05-09T08:48:18.368911+0000","last_active":"2024-05-09T08:49:47.204531+0000","last_peered":"2024-05-09T08:49:47.204531+0000","last_clean":"2024-05-09T08:49:47.204531+0000","last_became_active":"2024-05-09T08:48:18.368591+0000","last_became_peered":"2024-05-09T08:48:18.368591+0000","last_unstale":"2024-05-09T08:49:47.204531+0000","last_undegraded":"2024-05-09T08:49:47.204531+0000","last_fullsized":"2024-05-09T08:49:47.204531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:34:09.692168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.203098+0000","last_change":"2024-05-09T08:48:18.373629+0000","last_active":"2024-05-09T08:49:47.203098+0000","last_peered":"2024-05-09T08:49:47.203098+0000","last_clean":"2024-05-09T08:49:47.203098+0000","last_became_active":"2024-05-09T08:48:18.372992+0000","last_became_peered":"2024-05-09T08:48:18.372992+0000","last_unstale":"2024-05-09T08:49:47.203098+0000","last_undegraded":"2024-05-09T08:49:47.203098+0000","last_fullsized":"2024-05-09T08:49:47.203098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:42.532250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.199157+0000","last_change":"2024-05-09T08:48:18.382423+0000","last_active":"2024-05-09T08:49:47.199157+0000","last_peered":"2024-05-09T08:49:47.199157+0000","last_clean":"2024-05-09T08:49:47.199157+0000","last_became_active":"2024-05-09T08:48:18.382088+0000","last_became_peered":"2024-05-09T08:48:18.382088+0000","last_unstale":"2024-05-09T08:49:47.199157+0000","last_undegraded":"2024-05-09T08:49:47.199157+0000","last_fullsized":"2024-05-09T08:49:47.199157+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:09:06.342557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.201504+0000","last_change":"2024-05-09T08:48:18.370460+0000","last_active":"2024-05-09T08:49:47.201504+0000","last_peered":"2024-05-09T08:49:47.201504+0000","last_clean":"2024-05-09T08:49:47.201504+0000","last_became_active":"2024-05-09T08:48:18.370033+0000","last_became_peered":"2024-05-09T08:48:18.370033+0000","last_unstale":"2024-05-09T08:49:47.201504+0000","last_undegraded":"2024-05-09T08:49:47.201504+0000","last_fullsized":"2024-05-09T08:49:47.201504+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:21:28.622511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.215015+0000","last_change":"2024-05-09T08:48:18.371410+0000","last_active":"2024-05-09T08:49:47.215015+0000","last_peered":"2024-05-09T08:49:47.215015+0000","last_clean":"2024-05-09T08:49:47.215015+0000","last_became_active":"2024-05-09T08:48:18.371067+0000","last_became_peered":"2024-05-09T08:48:18.371067+0000","last_unstale":"2024-05-09T08:49:47.215015+0000","last_undegraded":"2024-05-09T08:49:47.215015+0000","last_fullsized":"2024-05-09T08:49:47.215015+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:35:11.445992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.216959+0000","last_change":"2024-05-09T08:48:18.382873+0000","last_active":"2024-05-09T08:49:47.216959+0000","last_peered":"2024-05-09T08:49:47.216959+0000","last_clean":"2024-05-09T08:49:47.216959+0000","last_became_active":"2024-05-09T08:48:18.382366+0000","last_became_peered":"2024-05-09T08:48:18.382366+0000","last_unstale":"2024-05-09T08:49:47.216959+0000","last_undegraded":"2024-05-09T08:49:47.216959+0000","last_fullsized":"2024-05-09T08:49:47.216959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:23.343131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.216253+0000","last_change":"2024-05-09T08:48:18.370821+0000","last_active":"2024-05-09T08:49:47.216253+0000","last_peered":"2024-05-09T08:49:47.216253+0000","last_clean":"2024-05-09T08:49:47.216253+0000","last_became_active":"2024-05-09T08:48:18.368208+0000","last_became_peered":"2024-05-09T08:48:18.368208+0000","last_unstale":"2024-05-09T08:49:47.216253+0000","last_undegraded":"2024-05-09T08:49:47.216253+0000","last_fullsized":"2024-05-09T08:49:47.216253+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:14.633938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.204816+0000","last_change":"2024-05-09T08:48:18.371626+0000","last_active":"2024-05-09T08:49:47.204816+0000","last_peered":"2024-05-09T08:49:47.204816+0000","last_clean":"2024-05-09T08:49:47.204816+0000","last_became_active":"2024-05-09T08:48:18.371081+0000","last_became_peered":"2024-05-09T08:48:18.371081+0000","last_unstale":"2024-05-09T08:49:47.204816+0000","last_undegraded":"2024-05-09T08:49:47.204816+0000","last_fullsized":"2024-05-09T08:49:47.204816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:27.372934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":470,"ondisk_log_size":470,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":210578,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68377,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":748,"num_read_kb":450,"num_write":245,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":205131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351008,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":505071,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612645,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1163264,"data_stored":1157406,"data_compressed":56208,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776170,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":505001,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":505379,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299444,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1102549,"data_compressed":56208,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462970,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":500505,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":562426,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":501961,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":563178,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":500213,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888132,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":505111,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,7,18,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724606,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1097599,"data_compressed":56208,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,20,22,13,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34003,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34003,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:49:53.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-09T08:49:53.497 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:54.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:54 smithi098 ceph-mon[30092]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:49:54.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:54 smithi098 ceph-mon[30092]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-09T08:49:54.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:54 smithi142 ceph-mon[34625]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:49:54.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:54 smithi142 ceph-mon[34625]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-09T08:49:54.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:54 smithi105 ceph-mon[34498]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:49:54.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:54 smithi105 ceph-mon[34498]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-09T08:49:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.333+0000 7f059db78700 1 -- 172.21.15.98:0/83813451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0598101220 msgr2=0x7f05981016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.333+0000 7f059db78700 1 --2- 172.21.15.98:0/83813451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0598101220 0x7f05981016a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f058c009990 tx=0x7f058c009ca0 comp rx=0 tx=0).stop 2024-05-09T08:49:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.333+0000 7f059db78700 1 -- 172.21.15.98:0/83813451 shutdown_connections 2024-05-09T08:49:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.333+0000 7f059db78700 1 --2- 172.21.15.98:0/83813451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0598107290 0x7f0598107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.333+0000 7f059db78700 1 --2- 172.21.15.98:0/83813451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0598101be0 0x7f059810e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.333+0000 7f059db78700 1 --2- 172.21.15.98:0/83813451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0598101220 0x7f05981016a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.333+0000 7f059db78700 1 -- 172.21.15.98:0/83813451 >> 172.21.15.98:0/83813451 conn(0x7f05980781d0 msgr2=0x7f05980785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:55.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.333+0000 7f059db78700 1 -- 172.21.15.98:0/83813451 shutdown_connections 2024-05-09T08:49:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.333+0000 7f059db78700 1 -- 172.21.15.98:0/83813451 wait complete. 2024-05-09T08:49:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.333+0000 7f059db78700 1 Processor -- start 2024-05-09T08:49:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f059db78700 1 -- start start 2024-05-09T08:49:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f059db78700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0598101220 0x7f0598199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f059db78700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0598101be0 0x7f059819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f059db78700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0598107290 0x7f05981a4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f059db78700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0598111600 con 0x7f0598107290 2024-05-09T08:49:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f059db78700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0598111480 con 0x7f0598101220 2024-05-09T08:49:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f059db78700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0598111780 con 0x7f0598101be0 2024-05-09T08:49:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f0597fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0598107290 0x7f05981a4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f0597fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0598107290 0x7f05981a4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56530/0 (socket says 172.21.15.98:56530) 2024-05-09T08:49:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f0597fff700 1 -- 172.21.15.98:0/147442189 learned_addr learned my addr 172.21.15.98:0/147442189 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f05977fe700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0598101220 0x7f0598199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f0596ffd700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0598101be0 0x7f059819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f0597fff700 1 -- 172.21.15.98:0/147442189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0598101be0 msgr2=0x7f059819a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f0597fff700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0598101be0 0x7f059819a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f0597fff700 1 -- 172.21.15.98:0/147442189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0598101220 msgr2=0x7f0598199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f0597fff700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0598101220 0x7f0598199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f0597fff700 1 -- 172.21.15.98:0/147442189 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05981a50e0 con 0x7f0598107290 2024-05-09T08:49:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f0596ffd700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0598101be0 0x7f059819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:49:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.334+0000 7f05977fe700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0598101220 0x7f0598199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:49:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.335+0000 7f0597fff700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0598107290 0x7f05981a4ba0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f059000b970 tx=0x7f059000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.335+0000 7f0594ff9700 1 -- 172.21.15.98:0/147442189 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0590019070 con 0x7f0598107290 2024-05-09T08:49:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.335+0000 7f0594ff9700 1 -- 172.21.15.98:0/147442189 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0590014440 con 0x7f0598107290 2024-05-09T08:49:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.335+0000 7f059db78700 1 -- 172.21.15.98:0/147442189 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05981a52b0 con 0x7f0598107290 2024-05-09T08:49:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.335+0000 7f0594ff9700 1 -- 172.21.15.98:0/147442189 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05900075d0 con 0x7f0598107290 2024-05-09T08:49:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.335+0000 7f059db78700 1 -- 172.21.15.98:0/147442189 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05981a57c0 con 0x7f0598107290 2024-05-09T08:49:55.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.336+0000 7f059db78700 1 -- 172.21.15.98:0/147442189 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0598109800 con 0x7f0598107290 2024-05-09T08:49:55.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.337+0000 7f0594ff9700 1 -- 172.21.15.98:0/147442189 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f059001c070 con 0x7f0598107290 2024-05-09T08:49:55.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.338+0000 7f0594ff9700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f05800a2c20 0x7f05800a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:55.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.338+0000 7f05977fe700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f05800a2c20 0x7f05800a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:55.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.338+0000 7f0594ff9700 1 -- 172.21.15.98:0/147442189 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f05900ce7b0 con 0x7f0598107290 2024-05-09T08:49:55.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.338+0000 7f05977fe700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f05800a2c20 0x7f05800a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f05880099d0 tx=0x7f0588008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:55.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.343+0000 7f0594ff9700 1 -- 172.21.15.98:0/147442189 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f059009b3b0 con 0x7f0598107290 2024-05-09T08:49:55.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.535+0000 7f059db78700 1 -- 172.21.15.98:0/147442189 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0598062760 con 0x7f05800a2c20 2024-05-09T08:49:55.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.545+0000 7f0594ff9700 1 -- 172.21.15.98:0/147442189 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340801 (secure 0 0 0) 0x7f0598062760 con 0x7f05800a2c20 2024-05-09T08:49:55.548 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:49:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.548+0000 7f059db78700 1 -- 172.21.15.98:0/147442189 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f05800a2c20 msgr2=0x7f05800a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.548+0000 7f059db78700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f05800a2c20 0x7f05800a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f05880099d0 tx=0x7f0588008040 comp rx=0 tx=0).stop 2024-05-09T08:49:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.548+0000 7f059db78700 1 -- 172.21.15.98:0/147442189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0598107290 msgr2=0x7f05981a4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.548+0000 7f059db78700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0598107290 0x7f05981a4ba0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f059000b970 tx=0x7f059000bd30 comp rx=0 tx=0).stop 2024-05-09T08:49:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.548+0000 7f059db78700 1 -- 172.21.15.98:0/147442189 shutdown_connections 2024-05-09T08:49:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.548+0000 7f059db78700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f05800a2c20 0x7f05800a50e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.548+0000 7f059db78700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0598101220 0x7f0598199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.548+0000 7f059db78700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0598107290 0x7f05981a4ba0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.548+0000 7f059db78700 1 --2- 172.21.15.98:0/147442189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0598101be0 0x7f059819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.548+0000 7f059db78700 1 -- 172.21.15.98:0/147442189 >> 172.21.15.98:0/147442189 conn(0x7f05980781d0 msgr2=0x7f05980ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.548+0000 7f059db78700 1 -- 172.21.15.98:0/147442189 shutdown_connections 2024-05-09T08:49:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:55.548+0000 7f059db78700 1 -- 172.21.15.98:0/147442189 wait complete. 2024-05-09T08:49:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-09T08:49:56.329 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-05-09T08:49:55.206051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658979,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":929,"num_read_kb":589,"num_write":430,"num_write_kb":3083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180853,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":855,"ondisk_log_size":855,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353224,"kb_used_data":12392,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588600,"statfs":{"total":1151940427776,"available":1151578726400,"internally_reserved":0,"allocated":12689408,"data_stored":8006399,"data_compressed":353709,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":138,"num_read_kb":68,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003713"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.213936+0000","last_change":"2024-05-09T08:48:18.373685+0000","last_active":"2024-05-09T08:49:47.213936+0000","last_peered":"2024-05-09T08:49:47.213936+0000","last_clean":"2024-05-09T08:49:47.213936+0000","last_became_active":"2024-05-09T08:48:18.373305+0000","last_became_peered":"2024-05-09T08:48:18.373305+0000","last_unstale":"2024-05-09T08:49:47.213936+0000","last_undegraded":"2024-05-09T08:49:47.213936+0000","last_fullsized":"2024-05-09T08:49:47.213936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:30:45.261706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.201601+0000","last_change":"2024-05-09T08:48:18.373152+0000","last_active":"2024-05-09T08:49:47.201601+0000","last_peered":"2024-05-09T08:49:47.201601+0000","last_clean":"2024-05-09T08:49:47.201601+0000","last_became_active":"2024-05-09T08:48:18.372015+0000","last_became_peered":"2024-05-09T08:48:18.372015+0000","last_unstale":"2024-05-09T08:49:47.201601+0000","last_undegraded":"2024-05-09T08:49:47.201601+0000","last_fullsized":"2024-05-09T08:49:47.201601+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:47.512778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.217042+0000","last_change":"2024-05-09T08:48:18.370177+0000","last_active":"2024-05-09T08:49:47.217042+0000","last_peered":"2024-05-09T08:49:47.217042+0000","last_clean":"2024-05-09T08:49:47.217042+0000","last_became_active":"2024-05-09T08:48:18.369842+0000","last_became_peered":"2024-05-09T08:48:18.369842+0000","last_unstale":"2024-05-09T08:49:47.217042+0000","last_undegraded":"2024-05-09T08:49:47.217042+0000","last_fullsized":"2024-05-09T08:49:47.217042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:00:24.015500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.202870+0000","last_change":"2024-05-09T08:48:18.375364+0000","last_active":"2024-05-09T08:49:47.202870+0000","last_peered":"2024-05-09T08:49:47.202870+0000","last_clean":"2024-05-09T08:49:47.202870+0000","last_became_active":"2024-05-09T08:48:18.375030+0000","last_became_peered":"2024-05-09T08:48:18.375030+0000","last_unstale":"2024-05-09T08:49:47.202870+0000","last_undegraded":"2024-05-09T08:49:47.202870+0000","last_fullsized":"2024-05-09T08:49:47.202870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:43:57.684123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.205041+0000","last_change":"2024-05-09T08:48:18.374548+0000","last_active":"2024-05-09T08:49:47.205041+0000","last_peered":"2024-05-09T08:49:47.205041+0000","last_clean":"2024-05-09T08:49:47.205041+0000","last_became_active":"2024-05-09T08:48:18.374190+0000","last_became_peered":"2024-05-09T08:48:18.374190+0000","last_unstale":"2024-05-09T08:49:47.205041+0000","last_undegraded":"2024-05-09T08:49:47.205041+0000","last_fullsized":"2024-05-09T08:49:47.205041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:43:48.456839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.213257+0000","last_change":"2024-05-09T08:48:18.371234+0000","last_active":"2024-05-09T08:49:47.213257+0000","last_peered":"2024-05-09T08:49:47.213257+0000","last_clean":"2024-05-09T08:49:47.213257+0000","last_became_active":"2024-05-09T08:48:18.370218+0000","last_became_peered":"2024-05-09T08:48:18.370218+0000","last_unstale":"2024-05-09T08:49:47.213257+0000","last_undegraded":"2024-05-09T08:49:47.213257+0000","last_fullsized":"2024-05-09T08:49:47.213257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:02:09.923146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.215407+0000","last_change":"2024-05-09T08:49:18.366083+0000","last_active":"2024-05-09T08:49:47.215407+0000","last_peered":"2024-05-09T08:49:47.215407+0000","last_clean":"2024-05-09T08:49:47.215407+0000","last_became_active":"2024-05-09T08:49:18.365755+0000","last_became_peered":"2024-05-09T08:49:18.365755+0000","last_unstale":"2024-05-09T08:49:47.215407+0000","last_undegraded":"2024-05-09T08:49:47.215407+0000","last_fullsized":"2024-05-09T08:49:47.215407+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:40:18.052955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.204209+0000","last_change":"2024-05-09T08:49:18.488640+0000","last_active":"2024-05-09T08:49:47.204209+0000","last_peered":"2024-05-09T08:49:47.204209+0000","last_clean":"2024-05-09T08:49:47.204209+0000","last_became_active":"2024-05-09T08:49:18.488404+0000","last_became_peered":"2024-05-09T08:49:18.488404+0000","last_unstale":"2024-05-09T08:49:47.204209+0000","last_undegraded":"2024-05-09T08:49:47.204209+0000","last_fullsized":"2024-05-09T08:49:47.204209+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:47.317453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.218289+0000","last_change":"2024-05-09T08:49:18.368485+0000","last_active":"2024-05-09T08:49:47.218289+0000","last_peered":"2024-05-09T08:49:47.218289+0000","last_clean":"2024-05-09T08:49:47.218289+0000","last_became_active":"2024-05-09T08:49:18.368175+0000","last_became_peered":"2024-05-09T08:49:18.368175+0000","last_unstale":"2024-05-09T08:49:47.218289+0000","last_undegraded":"2024-05-09T08:49:47.218289+0000","last_fullsized":"2024-05-09T08:49:47.218289+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:39:32.399209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.213082+0000","last_change":"2024-05-09T08:48:18.363668+0000","last_active":"2024-05-09T08:49:47.213082+0000","last_peered":"2024-05-09T08:49:47.213082+0000","last_clean":"2024-05-09T08:49:47.213082+0000","last_became_active":"2024-05-09T08:48:18.363248+0000","last_became_peered":"2024-05-09T08:48:18.363248+0000","last_unstale":"2024-05-09T08:49:47.213082+0000","last_undegraded":"2024-05-09T08:49:47.213082+0000","last_fullsized":"2024-05-09T08:49:47.213082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:19:33.802350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.219443+0000","last_change":"2024-05-09T08:48:18.376401+0000","last_active":"2024-05-09T08:49:47.219443+0000","last_peered":"2024-05-09T08:49:47.219443+0000","last_clean":"2024-05-09T08:49:47.219443+0000","last_became_active":"2024-05-09T08:48:18.375520+0000","last_became_peered":"2024-05-09T08:48:18.375520+0000","last_unstale":"2024-05-09T08:49:47.219443+0000","last_undegraded":"2024-05-09T08:49:47.219443+0000","last_fullsized":"2024-05-09T08:49:47.219443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:40:36.145071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.209825+0000","last_change":"2024-05-09T08:48:18.372847+0000","last_active":"2024-05-09T08:49:47.209825+0000","last_peered":"2024-05-09T08:49:47.209825+0000","last_clean":"2024-05-09T08:49:47.209825+0000","last_became_active":"2024-05-09T08:48:18.370671+0000","last_became_peered":"2024-05-09T08:48:18.370671+0000","last_unstale":"2024-05-09T08:49:47.209825+0000","last_undegraded":"2024-05-09T08:49:47.209825+0000","last_fullsized":"2024-05-09T08:49:47.209825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:06:22.438515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.208023+0000","last_change":"2024-05-09T08:48:18.365810+0000","last_active":"2024-05-09T08:49:47.208023+0000","last_peered":"2024-05-09T08:49:47.208023+0000","last_clean":"2024-05-09T08:49:47.208023+0000","last_became_active":"2024-05-09T08:48:18.365214+0000","last_became_peered":"2024-05-09T08:48:18.365214+0000","last_unstale":"2024-05-09T08:49:47.208023+0000","last_undegraded":"2024-05-09T08:49:47.208023+0000","last_fullsized":"2024-05-09T08:49:47.208023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:00.037153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.209125+0000","last_change":"2024-05-09T08:48:18.375279+0000","last_active":"2024-05-09T08:49:47.209125+0000","last_peered":"2024-05-09T08:49:47.209125+0000","last_clean":"2024-05-09T08:49:47.209125+0000","last_became_active":"2024-05-09T08:48:18.374918+0000","last_became_peered":"2024-05-09T08:48:18.374918+0000","last_unstale":"2024-05-09T08:49:47.209125+0000","last_undegraded":"2024-05-09T08:49:47.209125+0000","last_fullsized":"2024-05-09T08:49:47.209125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:23:12.889283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.215473+0000","last_change":"2024-05-09T08:49:18.369768+0000","last_active":"2024-05-09T08:49:47.215473+0000","last_peered":"2024-05-09T08:49:47.215473+0000","last_clean":"2024-05-09T08:49:47.215473+0000","last_became_active":"2024-05-09T08:49:18.369520+0000","last_became_peered":"2024-05-09T08:49:18.369520+0000","last_unstale":"2024-05-09T08:49:47.215473+0000","last_undegraded":"2024-05-09T08:49:47.215473+0000","last_fullsized":"2024-05-09T08:49:47.215473+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:44:19.502358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.217231+0000","last_change":"2024-05-09T08:48:18.373905+0000","last_active":"2024-05-09T08:49:47.217231+0000","last_peered":"2024-05-09T08:49:47.217231+0000","last_clean":"2024-05-09T08:49:47.217231+0000","last_became_active":"2024-05-09T08:48:18.373566+0000","last_became_peered":"2024-05-09T08:48:18.373566+0000","last_unstale":"2024-05-09T08:49:47.217231+0000","last_undegraded":"2024-05-09T08:49:47.217231+0000","last_fullsized":"2024-05-09T08:49:47.217231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:33.905686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.214154+0000","last_change":"2024-05-09T08:48:18.373970+0000","last_active":"2024-05-09T08:49:47.214154+0000","last_peered":"2024-05-09T08:49:47.214154+0000","last_clean":"2024-05-09T08:49:47.214154+0000","last_became_active":"2024-05-09T08:48:18.372696+0000","last_became_peered":"2024-05-09T08:48:18.372696+0000","last_unstale":"2024-05-09T08:49:47.214154+0000","last_undegraded":"2024-05-09T08:49:47.214154+0000","last_fullsized":"2024-05-09T08:49:47.214154+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:09:33.751950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.214834+0000","last_change":"2024-05-09T08:48:18.376851+0000","last_active":"2024-05-09T08:49:47.214834+0000","last_peered":"2024-05-09T08:49:47.214834+0000","last_clean":"2024-05-09T08:49:47.214834+0000","last_became_active":"2024-05-09T08:48:18.376230+0000","last_became_peered":"2024-05-09T08:48:18.376230+0000","last_unstale":"2024-05-09T08:49:47.214834+0000","last_undegraded":"2024-05-09T08:49:47.214834+0000","last_fullsized":"2024-05-09T08:49:47.214834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:34.959179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.199408+0000","last_change":"2024-05-09T08:48:18.370287+0000","last_active":"2024-05-09T08:49:47.199408+0000","last_peered":"2024-05-09T08:49:47.199408+0000","last_clean":"2024-05-09T08:49:47.199408+0000","last_became_active":"2024-05-09T08:48:18.369956+0000","last_became_peered":"2024-05-09T08:48:18.369956+0000","last_unstale":"2024-05-09T08:49:47.199408+0000","last_undegraded":"2024-05-09T08:49:47.199408+0000","last_fullsized":"2024-05-09T08:49:47.199408+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:19:34.746216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361884+0000","last_change":"2024-05-09T08:48:22.377973+0000","last_active":"2024-05-09T08:49:18.361884+0000","last_peered":"2024-05-09T08:49:18.361884+0000","last_clean":"2024-05-09T08:49:18.361884+0000","last_became_active":"2024-05-09T08:48:22.377325+0000","last_became_peered":"2024-05-09T08:48:22.377325+0000","last_unstale":"2024-05-09T08:49:18.361884+0000","last_undegraded":"2024-05-09T08:49:18.361884+0000","last_fullsized":"2024-05-09T08:49:18.361884+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:32:00.873082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":114,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:50.861067+0000","last_change":"2024-05-09T08:48:17.595784+0000","last_active":"2024-05-09T08:49:50.861067+0000","last_peered":"2024-05-09T08:49:50.861067+0000","last_clean":"2024-05-09T08:49:50.861067+0000","last_became_active":"2024-05-09T08:48:17.595395+0000","last_became_peered":"2024-05-09T08:48:17.595395+0000","last_unstale":"2024-05-09T08:49:50.861067+0000","last_undegraded":"2024-05-09T08:49:50.861067+0000","last_fullsized":"2024-05-09T08:49:50.861067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:47.475148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.209386+0000","last_change":"2024-05-09T08:48:18.382332+0000","last_active":"2024-05-09T08:49:47.209386+0000","last_peered":"2024-05-09T08:49:47.209386+0000","last_clean":"2024-05-09T08:49:47.209386+0000","last_became_active":"2024-05-09T08:48:18.381536+0000","last_became_peered":"2024-05-09T08:48:18.381536+0000","last_unstale":"2024-05-09T08:49:47.209386+0000","last_undegraded":"2024-05-09T08:49:47.209386+0000","last_fullsized":"2024-05-09T08:49:47.209386+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:52:48.327000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359199+0000","last_change":"2024-05-09T08:48:22.379541+0000","last_active":"2024-05-09T08:49:18.359199+0000","last_peered":"2024-05-09T08:49:18.359199+0000","last_clean":"2024-05-09T08:49:18.359199+0000","last_became_active":"2024-05-09T08:48:22.379135+0000","last_became_peered":"2024-05-09T08:48:22.379135+0000","last_unstale":"2024-05-09T08:49:18.359199+0000","last_undegraded":"2024-05-09T08:49:18.359199+0000","last_fullsized":"2024-05-09T08:49:18.359199+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:51.365075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.693649+0000","last_change":"2024-05-09T08:48:17.592168+0000","last_active":"2024-05-09T08:49:18.693649+0000","last_peered":"2024-05-09T08:49:18.693649+0000","last_clean":"2024-05-09T08:49:18.693649+0000","last_became_active":"2024-05-09T08:48:17.591822+0000","last_became_peered":"2024-05-09T08:48:17.591822+0000","last_unstale":"2024-05-09T08:49:18.693649+0000","last_undegraded":"2024-05-09T08:49:18.693649+0000","last_fullsized":"2024-05-09T08:49:18.693649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:45:15.162240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.213872+0000","last_change":"2024-05-09T08:48:18.379396+0000","last_active":"2024-05-09T08:49:47.213872+0000","last_peered":"2024-05-09T08:49:47.213872+0000","last_clean":"2024-05-09T08:49:47.213872+0000","last_became_active":"2024-05-09T08:48:18.378770+0000","last_became_peered":"2024-05-09T08:48:18.378770+0000","last_unstale":"2024-05-09T08:49:47.213872+0000","last_undegraded":"2024-05-09T08:49:47.213872+0000","last_fullsized":"2024-05-09T08:49:47.213872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:22:22.699186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359324+0000","last_change":"2024-05-09T08:48:22.387777+0000","last_active":"2024-05-09T08:49:18.359324+0000","last_peered":"2024-05-09T08:49:18.359324+0000","last_clean":"2024-05-09T08:49:18.359324+0000","last_became_active":"2024-05-09T08:48:22.387300+0000","last_became_peered":"2024-05-09T08:48:22.387300+0000","last_unstale":"2024-05-09T08:49:18.359324+0000","last_undegraded":"2024-05-09T08:49:18.359324+0000","last_fullsized":"2024-05-09T08:49:18.359324+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:08:35.646374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.367409+0000","last_change":"2024-05-09T08:48:17.602493+0000","last_active":"2024-05-09T08:49:18.367409+0000","last_peered":"2024-05-09T08:49:18.367409+0000","last_clean":"2024-05-09T08:49:18.367409+0000","last_became_active":"2024-05-09T08:48:17.601625+0000","last_became_peered":"2024-05-09T08:48:17.601625+0000","last_unstale":"2024-05-09T08:49:18.367409+0000","last_undegraded":"2024-05-09T08:49:18.367409+0000","last_fullsized":"2024-05-09T08:49:18.367409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:33.044532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.217229+0000","last_change":"2024-05-09T08:48:18.380615+0000","last_active":"2024-05-09T08:49:47.217229+0000","last_peered":"2024-05-09T08:49:47.217229+0000","last_clean":"2024-05-09T08:49:47.217229+0000","last_became_active":"2024-05-09T08:48:18.380196+0000","last_became_peered":"2024-05-09T08:48:18.380196+0000","last_unstale":"2024-05-09T08:49:47.217229+0000","last_undegraded":"2024-05-09T08:49:47.217229+0000","last_fullsized":"2024-05-09T08:49:47.217229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:27.299263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359229+0000","last_change":"2024-05-09T08:48:22.381954+0000","last_active":"2024-05-09T08:49:18.359229+0000","last_peered":"2024-05-09T08:49:18.359229+0000","last_clean":"2024-05-09T08:49:18.359229+0000","last_became_active":"2024-05-09T08:48:22.381527+0000","last_became_peered":"2024-05-09T08:48:22.381527+0000","last_unstale":"2024-05-09T08:49:18.359229+0000","last_undegraded":"2024-05-09T08:49:18.359229+0000","last_fullsized":"2024-05-09T08:49:18.359229+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:35.170753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.366013+0000","last_change":"2024-05-09T08:48:17.592795+0000","last_active":"2024-05-09T08:49:18.366013+0000","last_peered":"2024-05-09T08:49:18.366013+0000","last_clean":"2024-05-09T08:49:18.366013+0000","last_became_active":"2024-05-09T08:48:17.592240+0000","last_became_peered":"2024-05-09T08:48:17.592240+0000","last_unstale":"2024-05-09T08:49:18.366013+0000","last_undegraded":"2024-05-09T08:49:18.366013+0000","last_fullsized":"2024-05-09T08:49:18.366013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:27:30.056805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.213858+0000","last_change":"2024-05-09T08:48:18.373853+0000","last_active":"2024-05-09T08:49:47.213858+0000","last_peered":"2024-05-09T08:49:47.213858+0000","last_clean":"2024-05-09T08:49:47.213858+0000","last_became_active":"2024-05-09T08:48:18.373482+0000","last_became_peered":"2024-05-09T08:48:18.373482+0000","last_unstale":"2024-05-09T08:49:47.213858+0000","last_undegraded":"2024-05-09T08:49:47.213858+0000","last_fullsized":"2024-05-09T08:49:47.213858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:14:49.129927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362680+0000","last_change":"2024-05-09T08:48:22.374267+0000","last_active":"2024-05-09T08:49:18.362680+0000","last_peered":"2024-05-09T08:49:18.362680+0000","last_clean":"2024-05-09T08:49:18.362680+0000","last_became_active":"2024-05-09T08:48:22.373824+0000","last_became_peered":"2024-05-09T08:48:22.373824+0000","last_unstale":"2024-05-09T08:49:18.362680+0000","last_undegraded":"2024-05-09T08:49:18.362680+0000","last_fullsized":"2024-05-09T08:49:18.362680+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:27.407543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":114,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:51.832885+0000","last_change":"2024-05-09T08:48:17.596851+0000","last_active":"2024-05-09T08:49:51.832885+0000","last_peered":"2024-05-09T08:49:51.832885+0000","last_clean":"2024-05-09T08:49:51.832885+0000","last_became_active":"2024-05-09T08:48:17.596532+0000","last_became_peered":"2024-05-09T08:48:17.596532+0000","last_unstale":"2024-05-09T08:49:51.832885+0000","last_undegraded":"2024-05-09T08:49:51.832885+0000","last_fullsized":"2024-05-09T08:49:51.832885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:58:51.923188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.205763+0000","last_change":"2024-05-09T08:48:18.372214+0000","last_active":"2024-05-09T08:49:47.205763+0000","last_peered":"2024-05-09T08:49:47.205763+0000","last_clean":"2024-05-09T08:49:47.205763+0000","last_became_active":"2024-05-09T08:48:18.371850+0000","last_became_peered":"2024-05-09T08:48:18.371850+0000","last_unstale":"2024-05-09T08:49:47.205763+0000","last_undegraded":"2024-05-09T08:49:47.205763+0000","last_fullsized":"2024-05-09T08:49:47.205763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:56.971735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.371344+0000","last_change":"2024-05-09T08:49:18.371344+0000","last_active":"2024-05-09T08:49:18.371344+0000","last_peered":"2024-05-09T08:49:18.371344+0000","last_clean":"2024-05-09T08:49:18.371344+0000","last_became_active":"2024-05-09T08:49:18.370911+0000","last_became_peered":"2024-05-09T08:49:18.370911+0000","last_unstale":"2024-05-09T08:49:18.371344+0000","last_undegraded":"2024-05-09T08:49:18.371344+0000","last_fullsized":"2024-05-09T08:49:18.371344+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:53:49.122965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,8],"acting":[2,6,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.488066+0000","last_change":"2024-05-09T08:48:17.601726+0000","last_active":"2024-05-09T08:49:18.488066+0000","last_peered":"2024-05-09T08:49:18.488066+0000","last_clean":"2024-05-09T08:49:18.488066+0000","last_became_active":"2024-05-09T08:48:17.601372+0000","last_became_peered":"2024-05-09T08:48:17.601372+0000","last_unstale":"2024-05-09T08:49:18.488066+0000","last_undegraded":"2024-05-09T08:49:18.488066+0000","last_fullsized":"2024-05-09T08:49:18.488066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:27:56.683821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.215540+0000","last_change":"2024-05-09T08:49:18.694713+0000","last_active":"2024-05-09T08:49:47.215540+0000","last_peered":"2024-05-09T08:49:47.215540+0000","last_clean":"2024-05-09T08:49:47.215540+0000","last_became_active":"2024-05-09T08:49:18.694246+0000","last_became_peered":"2024-05-09T08:49:18.694246+0000","last_unstale":"2024-05-09T08:49:47.215540+0000","last_undegraded":"2024-05-09T08:49:47.215540+0000","last_fullsized":"2024-05-09T08:49:47.215540+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:05:30.818965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.363420+0000","last_change":"2024-05-09T08:48:22.390990+0000","last_active":"2024-05-09T08:49:18.363420+0000","last_peered":"2024-05-09T08:49:18.363420+0000","last_clean":"2024-05-09T08:49:18.363420+0000","last_became_active":"2024-05-09T08:48:22.390601+0000","last_became_peered":"2024-05-09T08:48:22.390601+0000","last_unstale":"2024-05-09T08:49:18.363420+0000","last_undegraded":"2024-05-09T08:49:18.363420+0000","last_fullsized":"2024-05-09T08:49:18.363420+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:47:24.644408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361959+0000","last_change":"2024-05-09T08:48:17.598008+0000","last_active":"2024-05-09T08:49:18.361959+0000","last_peered":"2024-05-09T08:49:18.361959+0000","last_clean":"2024-05-09T08:49:18.361959+0000","last_became_active":"2024-05-09T08:48:17.597514+0000","last_became_peered":"2024-05-09T08:48:17.597514+0000","last_unstale":"2024-05-09T08:49:18.361959+0000","last_undegraded":"2024-05-09T08:49:18.361959+0000","last_fullsized":"2024-05-09T08:49:18.361959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:18:32.788536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.215821+0000","last_change":"2024-05-09T08:48:18.378061+0000","last_active":"2024-05-09T08:49:47.215821+0000","last_peered":"2024-05-09T08:49:47.215821+0000","last_clean":"2024-05-09T08:49:47.215821+0000","last_became_active":"2024-05-09T08:48:18.377684+0000","last_became_peered":"2024-05-09T08:48:18.377684+0000","last_unstale":"2024-05-09T08:49:47.215821+0000","last_undegraded":"2024-05-09T08:49:47.215821+0000","last_fullsized":"2024-05-09T08:49:47.215821+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:13:08.338307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361739+0000","last_change":"2024-05-09T08:48:22.385779+0000","last_active":"2024-05-09T08:49:18.361739+0000","last_peered":"2024-05-09T08:49:18.361739+0000","last_clean":"2024-05-09T08:49:18.361739+0000","last_became_active":"2024-05-09T08:48:22.385021+0000","last_became_peered":"2024-05-09T08:48:22.385021+0000","last_unstale":"2024-05-09T08:49:18.361739+0000","last_undegraded":"2024-05-09T08:49:18.361739+0000","last_fullsized":"2024-05-09T08:49:18.361739+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:51.350754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361862+0000","last_change":"2024-05-09T08:48:17.595750+0000","last_active":"2024-05-09T08:49:18.361862+0000","last_peered":"2024-05-09T08:49:18.361862+0000","last_clean":"2024-05-09T08:49:18.361862+0000","last_became_active":"2024-05-09T08:48:17.595394+0000","last_became_peered":"2024-05-09T08:48:17.595394+0000","last_unstale":"2024-05-09T08:49:18.361862+0000","last_undegraded":"2024-05-09T08:49:18.361862+0000","last_fullsized":"2024-05-09T08:49:18.361862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:39:21.975059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.204858+0000","last_change":"2024-05-09T08:48:18.370750+0000","last_active":"2024-05-09T08:49:47.204858+0000","last_peered":"2024-05-09T08:49:47.204858+0000","last_clean":"2024-05-09T08:49:47.204858+0000","last_became_active":"2024-05-09T08:48:18.370383+0000","last_became_peered":"2024-05-09T08:48:18.370383+0000","last_unstale":"2024-05-09T08:49:47.204858+0000","last_undegraded":"2024-05-09T08:49:47.204858+0000","last_fullsized":"2024-05-09T08:49:47.204858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:36.049350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.692574+0000","last_change":"2024-05-09T08:48:22.377157+0000","last_active":"2024-05-09T08:49:18.692574+0000","last_peered":"2024-05-09T08:49:18.692574+0000","last_clean":"2024-05-09T08:49:18.692574+0000","last_became_active":"2024-05-09T08:48:22.376730+0000","last_became_peered":"2024-05-09T08:48:22.376730+0000","last_unstale":"2024-05-09T08:49:18.692574+0000","last_undegraded":"2024-05-09T08:49:18.692574+0000","last_fullsized":"2024-05-09T08:49:18.692574+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:41:52.080730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'5","reported_seq":182,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:52.833749+0000","last_change":"2024-05-09T08:48:17.598528+0000","last_active":"2024-05-09T08:49:52.833749+0000","last_peered":"2024-05-09T08:49:52.833749+0000","last_clean":"2024-05-09T08:49:52.833749+0000","last_became_active":"2024-05-09T08:48:17.598072+0000","last_became_peered":"2024-05-09T08:48:17.598072+0000","last_unstale":"2024-05-09T08:49:52.833749+0000","last_undegraded":"2024-05-09T08:49:52.833749+0000","last_fullsized":"2024-05-09T08:49:52.833749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:35.697154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":150,"num_read_kb":150,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.218018+0000","last_change":"2024-05-09T08:48:18.380919+0000","last_active":"2024-05-09T08:49:47.218018+0000","last_peered":"2024-05-09T08:49:47.218018+0000","last_clean":"2024-05-09T08:49:47.218018+0000","last_became_active":"2024-05-09T08:48:18.380485+0000","last_became_peered":"2024-05-09T08:48:18.380485+0000","last_unstale":"2024-05-09T08:49:47.218018+0000","last_undegraded":"2024-05-09T08:49:47.218018+0000","last_fullsized":"2024-05-09T08:49:47.218018+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:45.079173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359369+0000","last_change":"2024-05-09T08:48:22.389667+0000","last_active":"2024-05-09T08:49:18.359369+0000","last_peered":"2024-05-09T08:49:18.359369+0000","last_clean":"2024-05-09T08:49:18.359369+0000","last_became_active":"2024-05-09T08:48:22.389268+0000","last_became_peered":"2024-05-09T08:48:22.389268+0000","last_unstale":"2024-05-09T08:49:18.359369+0000","last_undegraded":"2024-05-09T08:49:18.359369+0000","last_fullsized":"2024-05-09T08:49:18.359369+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:33:14.861855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361320+0000","last_change":"2024-05-09T08:48:17.596028+0000","last_active":"2024-05-09T08:49:18.361320+0000","last_peered":"2024-05-09T08:49:18.361320+0000","last_clean":"2024-05-09T08:49:18.361320+0000","last_became_active":"2024-05-09T08:48:17.595053+0000","last_became_peered":"2024-05-09T08:48:17.595053+0000","last_unstale":"2024-05-09T08:49:18.361320+0000","last_undegraded":"2024-05-09T08:49:18.361320+0000","last_fullsized":"2024-05-09T08:49:18.361320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:48:31.323595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.208956+0000","last_change":"2024-05-09T08:48:18.373472+0000","last_active":"2024-05-09T08:49:47.208956+0000","last_peered":"2024-05-09T08:49:47.208956+0000","last_clean":"2024-05-09T08:49:47.208956+0000","last_became_active":"2024-05-09T08:48:18.373178+0000","last_became_peered":"2024-05-09T08:48:18.373178+0000","last_unstale":"2024-05-09T08:49:47.208956+0000","last_undegraded":"2024-05-09T08:49:47.208956+0000","last_fullsized":"2024-05-09T08:49:47.208956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:18.048722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359521+0000","last_change":"2024-05-09T08:48:22.386551+0000","last_active":"2024-05-09T08:49:18.359521+0000","last_peered":"2024-05-09T08:49:18.359521+0000","last_clean":"2024-05-09T08:49:18.359521+0000","last_became_active":"2024-05-09T08:48:22.386018+0000","last_became_peered":"2024-05-09T08:48:22.386018+0000","last_unstale":"2024-05-09T08:49:18.359521+0000","last_undegraded":"2024-05-09T08:49:18.359521+0000","last_fullsized":"2024-05-09T08:49:18.359521+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:59.812030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.363858+0000","last_change":"2024-05-09T08:48:17.601207+0000","last_active":"2024-05-09T08:49:18.363858+0000","last_peered":"2024-05-09T08:49:18.363858+0000","last_clean":"2024-05-09T08:49:18.363858+0000","last_became_active":"2024-05-09T08:48:17.600878+0000","last_became_peered":"2024-05-09T08:48:17.600878+0000","last_unstale":"2024-05-09T08:49:18.363858+0000","last_undegraded":"2024-05-09T08:49:18.363858+0000","last_fullsized":"2024-05-09T08:49:18.363858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:30.040527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.218089+0000","last_change":"2024-05-09T08:48:18.381577+0000","last_active":"2024-05-09T08:49:47.218089+0000","last_peered":"2024-05-09T08:49:47.218089+0000","last_clean":"2024-05-09T08:49:47.218089+0000","last_became_active":"2024-05-09T08:48:18.381223+0000","last_became_peered":"2024-05-09T08:48:18.381223+0000","last_unstale":"2024-05-09T08:49:47.218089+0000","last_undegraded":"2024-05-09T08:49:47.218089+0000","last_fullsized":"2024-05-09T08:49:47.218089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:57.038392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359370+0000","last_change":"2024-05-09T08:48:22.385885+0000","last_active":"2024-05-09T08:49:18.359370+0000","last_peered":"2024-05-09T08:49:18.359370+0000","last_clean":"2024-05-09T08:49:18.359370+0000","last_became_active":"2024-05-09T08:48:22.385403+0000","last_became_peered":"2024-05-09T08:48:22.385403+0000","last_unstale":"2024-05-09T08:49:18.359370+0000","last_undegraded":"2024-05-09T08:49:18.359370+0000","last_fullsized":"2024-05-09T08:49:18.359370+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:53:20.733523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":223,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:52.833599+0000","last_change":"2024-05-09T08:48:17.597848+0000","last_active":"2024-05-09T08:49:52.833599+0000","last_peered":"2024-05-09T08:49:52.833599+0000","last_clean":"2024-05-09T08:49:52.833599+0000","last_became_active":"2024-05-09T08:48:17.596989+0000","last_became_peered":"2024-05-09T08:48:17.596989+0000","last_unstale":"2024-05-09T08:49:52.833599+0000","last_undegraded":"2024-05-09T08:49:52.833599+0000","last_fullsized":"2024-05-09T08:49:52.833599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:07.256301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60891,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":160,"num_read_kb":140,"num_write":38,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.219510+0000","last_change":"2024-05-09T08:48:18.365308+0000","last_active":"2024-05-09T08:49:47.219510+0000","last_peered":"2024-05-09T08:49:47.219510+0000","last_clean":"2024-05-09T08:49:47.219510+0000","last_became_active":"2024-05-09T08:48:18.364845+0000","last_became_peered":"2024-05-09T08:48:18.364845+0000","last_unstale":"2024-05-09T08:49:47.219510+0000","last_undegraded":"2024-05-09T08:49:47.219510+0000","last_fullsized":"2024-05-09T08:49:47.219510+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:07.277730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.364236+0000","last_change":"2024-05-09T08:48:22.377333+0000","last_active":"2024-05-09T08:49:18.364236+0000","last_peered":"2024-05-09T08:49:18.364236+0000","last_clean":"2024-05-09T08:49:18.364236+0000","last_became_active":"2024-05-09T08:48:22.376494+0000","last_became_peered":"2024-05-09T08:48:22.376494+0000","last_unstale":"2024-05-09T08:49:18.364236+0000","last_undegraded":"2024-05-09T08:49:18.364236+0000","last_fullsized":"2024-05-09T08:49:18.364236+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:49:17.693977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362494+0000","last_change":"2024-05-09T08:48:17.600749+0000","last_active":"2024-05-09T08:49:18.362494+0000","last_peered":"2024-05-09T08:49:18.362494+0000","last_clean":"2024-05-09T08:49:18.362494+0000","last_became_active":"2024-05-09T08:48:17.600402+0000","last_became_peered":"2024-05-09T08:48:17.600402+0000","last_unstale":"2024-05-09T08:49:18.362494+0000","last_undegraded":"2024-05-09T08:49:18.362494+0000","last_fullsized":"2024-05-09T08:49:18.362494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:42:14.889227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.206098+0000","last_change":"2024-05-09T08:48:18.373652+0000","last_active":"2024-05-09T08:49:47.206098+0000","last_peered":"2024-05-09T08:49:47.206098+0000","last_clean":"2024-05-09T08:49:47.206098+0000","last_became_active":"2024-05-09T08:48:18.373345+0000","last_became_peered":"2024-05-09T08:48:18.373345+0000","last_unstale":"2024-05-09T08:49:47.206098+0000","last_undegraded":"2024-05-09T08:49:47.206098+0000","last_fullsized":"2024-05-09T08:49:47.206098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:14:34.843305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.369432+0000","last_change":"2024-05-09T08:49:18.369432+0000","last_active":"2024-05-09T08:49:18.369432+0000","last_peered":"2024-05-09T08:49:18.369432+0000","last_clean":"2024-05-09T08:49:18.369432+0000","last_became_active":"2024-05-09T08:49:18.368978+0000","last_became_peered":"2024-05-09T08:49:18.368978+0000","last_unstale":"2024-05-09T08:49:18.369432+0000","last_undegraded":"2024-05-09T08:49:18.369432+0000","last_fullsized":"2024-05-09T08:49:18.369432+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:20.969162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.363543+0000","last_change":"2024-05-09T08:48:17.592658+0000","last_active":"2024-05-09T08:49:18.363543+0000","last_peered":"2024-05-09T08:49:18.363543+0000","last_clean":"2024-05-09T08:49:18.363543+0000","last_became_active":"2024-05-09T08:48:17.592275+0000","last_became_peered":"2024-05-09T08:48:17.592275+0000","last_unstale":"2024-05-09T08:49:18.363543+0000","last_undegraded":"2024-05-09T08:49:18.363543+0000","last_fullsized":"2024-05-09T08:49:18.363543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:26:56.742079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.218875+0000","last_change":"2024-05-09T08:49:18.364399+0000","last_active":"2024-05-09T08:49:47.218875+0000","last_peered":"2024-05-09T08:49:47.218875+0000","last_clean":"2024-05-09T08:49:47.218875+0000","last_became_active":"2024-05-09T08:49:18.364166+0000","last_became_peered":"2024-05-09T08:49:18.364166+0000","last_unstale":"2024-05-09T08:49:47.218875+0000","last_undegraded":"2024-05-09T08:49:47.218875+0000","last_fullsized":"2024-05-09T08:49:47.218875+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:49:36.480337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359578+0000","last_change":"2024-05-09T08:48:22.389568+0000","last_active":"2024-05-09T08:49:18.359578+0000","last_peered":"2024-05-09T08:49:18.359578+0000","last_clean":"2024-05-09T08:49:18.359578+0000","last_became_active":"2024-05-09T08:48:22.387192+0000","last_became_peered":"2024-05-09T08:48:22.387192+0000","last_unstale":"2024-05-09T08:49:18.359578+0000","last_undegraded":"2024-05-09T08:49:18.359578+0000","last_fullsized":"2024-05-09T08:49:18.359578+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:56.794208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"92'7","reported_seq":115,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:52.834135+0000","last_change":"2024-05-09T08:48:17.598760+0000","last_active":"2024-05-09T08:49:52.834135+0000","last_peered":"2024-05-09T08:49:52.834135+0000","last_clean":"2024-05-09T08:49:52.834135+0000","last_became_active":"2024-05-09T08:48:17.598192+0000","last_became_peered":"2024-05-09T08:48:17.598192+0000","last_unstale":"2024-05-09T08:49:52.834135+0000","last_undegraded":"2024-05-09T08:49:52.834135+0000","last_fullsized":"2024-05-09T08:49:52.834135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:29.007644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.216235+0000","last_change":"2024-05-09T08:48:18.364570+0000","last_active":"2024-05-09T08:49:47.216235+0000","last_peered":"2024-05-09T08:49:47.216235+0000","last_clean":"2024-05-09T08:49:47.216235+0000","last_became_active":"2024-05-09T08:48:18.364191+0000","last_became_peered":"2024-05-09T08:48:18.364191+0000","last_unstale":"2024-05-09T08:49:47.216235+0000","last_undegraded":"2024-05-09T08:49:47.216235+0000","last_fullsized":"2024-05-09T08:49:47.216235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:49.267954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.694742+0000","last_change":"2024-05-09T08:49:18.694742+0000","last_active":"2024-05-09T08:49:18.694742+0000","last_peered":"2024-05-09T08:49:18.694742+0000","last_clean":"2024-05-09T08:49:18.694742+0000","last_became_active":"2024-05-09T08:49:18.694241+0000","last_became_peered":"2024-05-09T08:49:18.694241+0000","last_unstale":"2024-05-09T08:49:18.694742+0000","last_undegraded":"2024-05-09T08:49:18.694742+0000","last_fullsized":"2024-05-09T08:49:18.694742+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:20.289651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.367007+0000","last_change":"2024-05-09T08:48:17.599762+0000","last_active":"2024-05-09T08:49:18.367007+0000","last_peered":"2024-05-09T08:49:18.367007+0000","last_clean":"2024-05-09T08:49:18.367007+0000","last_became_active":"2024-05-09T08:48:17.599355+0000","last_became_peered":"2024-05-09T08:48:17.599355+0000","last_unstale":"2024-05-09T08:49:18.367007+0000","last_undegraded":"2024-05-09T08:49:18.367007+0000","last_fullsized":"2024-05-09T08:49:18.367007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:28:36.836403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.208503+0000","last_change":"2024-05-09T08:48:18.372810+0000","last_active":"2024-05-09T08:49:47.208503+0000","last_peered":"2024-05-09T08:49:47.208503+0000","last_clean":"2024-05-09T08:49:47.208503+0000","last_became_active":"2024-05-09T08:48:18.372437+0000","last_became_peered":"2024-05-09T08:48:18.372437+0000","last_unstale":"2024-05-09T08:49:47.208503+0000","last_undegraded":"2024-05-09T08:49:47.208503+0000","last_fullsized":"2024-05-09T08:49:47.208503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:03:36.879719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.363895+0000","last_change":"2024-05-09T08:48:22.379370+0000","last_active":"2024-05-09T08:49:18.363895+0000","last_peered":"2024-05-09T08:49:18.363895+0000","last_clean":"2024-05-09T08:49:18.363895+0000","last_became_active":"2024-05-09T08:48:22.378994+0000","last_became_peered":"2024-05-09T08:48:22.378994+0000","last_unstale":"2024-05-09T08:49:18.363895+0000","last_undegraded":"2024-05-09T08:49:18.363895+0000","last_fullsized":"2024-05-09T08:49:18.363895+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:08.005647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.366906+0000","last_change":"2024-05-09T08:48:17.594236+0000","last_active":"2024-05-09T08:49:18.366906+0000","last_peered":"2024-05-09T08:49:18.366906+0000","last_clean":"2024-05-09T08:49:18.366906+0000","last_became_active":"2024-05-09T08:48:17.593418+0000","last_became_peered":"2024-05-09T08:48:17.593418+0000","last_unstale":"2024-05-09T08:49:18.366906+0000","last_undegraded":"2024-05-09T08:49:18.366906+0000","last_fullsized":"2024-05-09T08:49:18.366906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:23.930421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.203573+0000","last_change":"2024-05-09T08:48:18.366767+0000","last_active":"2024-05-09T08:49:47.203573+0000","last_peered":"2024-05-09T08:49:47.203573+0000","last_clean":"2024-05-09T08:49:47.203573+0000","last_became_active":"2024-05-09T08:48:18.366435+0000","last_became_peered":"2024-05-09T08:48:18.366435+0000","last_unstale":"2024-05-09T08:49:47.203573+0000","last_undegraded":"2024-05-09T08:49:47.203573+0000","last_fullsized":"2024-05-09T08:49:47.203573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:45.111510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.366355+0000","last_change":"2024-05-09T08:48:22.378642+0000","last_active":"2024-05-09T08:49:18.366355+0000","last_peered":"2024-05-09T08:49:18.366355+0000","last_clean":"2024-05-09T08:49:18.366355+0000","last_became_active":"2024-05-09T08:48:22.378161+0000","last_became_peered":"2024-05-09T08:48:22.378161+0000","last_unstale":"2024-05-09T08:49:18.366355+0000","last_undegraded":"2024-05-09T08:49:18.366355+0000","last_fullsized":"2024-05-09T08:49:18.366355+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:53.149728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359658+0000","last_change":"2024-05-09T08:48:17.599499+0000","last_active":"2024-05-09T08:49:18.359658+0000","last_peered":"2024-05-09T08:49:18.359658+0000","last_clean":"2024-05-09T08:49:18.359658+0000","last_became_active":"2024-05-09T08:48:17.599145+0000","last_became_peered":"2024-05-09T08:48:17.599145+0000","last_unstale":"2024-05-09T08:49:18.359658+0000","last_undegraded":"2024-05-09T08:49:18.359658+0000","last_fullsized":"2024-05-09T08:49:18.359658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:31:12.502004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.210067+0000","last_change":"2024-05-09T08:48:18.372355+0000","last_active":"2024-05-09T08:49:47.210067+0000","last_peered":"2024-05-09T08:49:47.210067+0000","last_clean":"2024-05-09T08:49:47.210067+0000","last_became_active":"2024-05-09T08:48:18.367743+0000","last_became_peered":"2024-05-09T08:48:18.367743+0000","last_unstale":"2024-05-09T08:49:47.210067+0000","last_undegraded":"2024-05-09T08:49:47.210067+0000","last_fullsized":"2024-05-09T08:49:47.210067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:31:12.174350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.363563+0000","last_change":"2024-05-09T08:48:22.384064+0000","last_active":"2024-05-09T08:49:18.363563+0000","last_peered":"2024-05-09T08:49:18.363563+0000","last_clean":"2024-05-09T08:49:18.363563+0000","last_became_active":"2024-05-09T08:48:22.383670+0000","last_became_peered":"2024-05-09T08:48:22.383670+0000","last_unstale":"2024-05-09T08:49:18.363563+0000","last_undegraded":"2024-05-09T08:49:18.363563+0000","last_fullsized":"2024-05-09T08:49:18.363563+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:51:35.000827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":108,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:52.864161+0000","last_change":"2024-05-09T08:48:17.601596+0000","last_active":"2024-05-09T08:49:52.864161+0000","last_peered":"2024-05-09T08:49:52.864161+0000","last_clean":"2024-05-09T08:49:52.864161+0000","last_became_active":"2024-05-09T08:48:17.601250+0000","last_became_peered":"2024-05-09T08:48:17.601250+0000","last_unstale":"2024-05-09T08:49:52.864161+0000","last_undegraded":"2024-05-09T08:49:52.864161+0000","last_fullsized":"2024-05-09T08:49:52.864161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:48.237336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.213252+0000","last_change":"2024-05-09T08:48:18.368751+0000","last_active":"2024-05-09T08:49:47.213252+0000","last_peered":"2024-05-09T08:49:47.213252+0000","last_clean":"2024-05-09T08:49:47.213252+0000","last_became_active":"2024-05-09T08:48:18.367463+0000","last_became_peered":"2024-05-09T08:48:18.367463+0000","last_unstale":"2024-05-09T08:49:47.213252+0000","last_undegraded":"2024-05-09T08:49:47.213252+0000","last_fullsized":"2024-05-09T08:49:47.213252+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:19:17.696602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.363573+0000","last_change":"2024-05-09T08:48:22.385025+0000","last_active":"2024-05-09T08:49:18.363573+0000","last_peered":"2024-05-09T08:49:18.363573+0000","last_clean":"2024-05-09T08:49:18.363573+0000","last_became_active":"2024-05-09T08:48:22.384633+0000","last_became_peered":"2024-05-09T08:48:22.384633+0000","last_unstale":"2024-05-09T08:49:18.363573+0000","last_undegraded":"2024-05-09T08:49:18.363573+0000","last_fullsized":"2024-05-09T08:49:18.363573+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:17.804075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.210920+0000","last_change":"2024-05-09T08:48:18.376741+0000","last_active":"2024-05-09T08:49:47.210920+0000","last_peered":"2024-05-09T08:49:47.210920+0000","last_clean":"2024-05-09T08:49:47.210920+0000","last_became_active":"2024-05-09T08:48:18.376376+0000","last_became_peered":"2024-05-09T08:48:18.376376+0000","last_unstale":"2024-05-09T08:49:47.210920+0000","last_undegraded":"2024-05-09T08:49:47.210920+0000","last_fullsized":"2024-05-09T08:49:47.210920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:58:31.717555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'6","reported_seq":107,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:48.828866+0000","last_change":"2024-05-09T08:48:17.598316+0000","last_active":"2024-05-09T08:49:48.828866+0000","last_peered":"2024-05-09T08:49:48.828866+0000","last_clean":"2024-05-09T08:49:48.828866+0000","last_became_active":"2024-05-09T08:48:17.598022+0000","last_became_peered":"2024-05-09T08:48:17.598022+0000","last_unstale":"2024-05-09T08:49:48.828866+0000","last_undegraded":"2024-05-09T08:49:48.828866+0000","last_fullsized":"2024-05-09T08:49:48.828866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:04:10.454894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.492981+0000","last_change":"2024-05-09T08:49:18.492981+0000","last_active":"2024-05-09T08:49:18.492981+0000","last_peered":"2024-05-09T08:49:18.492981+0000","last_clean":"2024-05-09T08:49:18.492981+0000","last_became_active":"2024-05-09T08:49:18.489169+0000","last_became_peered":"2024-05-09T08:49:18.489169+0000","last_unstale":"2024-05-09T08:49:18.492981+0000","last_undegraded":"2024-05-09T08:49:18.492981+0000","last_fullsized":"2024-05-09T08:49:18.492981+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:02:53.955587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.209575+0000","last_change":"2024-05-09T08:48:18.379203+0000","last_active":"2024-05-09T08:49:47.209575+0000","last_peered":"2024-05-09T08:49:47.209575+0000","last_clean":"2024-05-09T08:49:47.209575+0000","last_became_active":"2024-05-09T08:48:18.378885+0000","last_became_peered":"2024-05-09T08:48:18.378885+0000","last_unstale":"2024-05-09T08:49:47.209575+0000","last_undegraded":"2024-05-09T08:49:47.209575+0000","last_fullsized":"2024-05-09T08:49:47.209575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:40.588263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359336+0000","last_change":"2024-05-09T08:48:17.593520+0000","last_active":"2024-05-09T08:49:18.359336+0000","last_peered":"2024-05-09T08:49:18.359336+0000","last_clean":"2024-05-09T08:49:18.359336+0000","last_became_active":"2024-05-09T08:48:17.593127+0000","last_became_peered":"2024-05-09T08:48:17.593127+0000","last_unstale":"2024-05-09T08:49:18.359336+0000","last_undegraded":"2024-05-09T08:49:18.359336+0000","last_fullsized":"2024-05-09T08:49:18.359336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:34:29.567166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":361,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361176+0000","last_change":"2024-05-09T08:47:05.718649+0000","last_active":"2024-05-09T08:49:18.361176+0000","last_peered":"2024-05-09T08:49:18.361176+0000","last_clean":"2024-05-09T08:49:18.361176+0000","last_became_active":"2024-05-09T08:47:05.718088+0000","last_became_peered":"2024-05-09T08:47:05.718088+0000","last_unstale":"2024-05-09T08:49:18.361176+0000","last_undegraded":"2024-05-09T08:49:18.361176+0000","last_fullsized":"2024-05-09T08:49:18.361176+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:43:39.457880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:43:39.457880+0000","last_clean_scrub_stamp":"2024-05-09T08:43:39.457880+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:14.227590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362973+0000","last_change":"2024-05-09T08:48:22.376751+0000","last_active":"2024-05-09T08:49:18.362973+0000","last_peered":"2024-05-09T08:49:18.362973+0000","last_clean":"2024-05-09T08:49:18.362973+0000","last_became_active":"2024-05-09T08:48:22.376322+0000","last_became_peered":"2024-05-09T08:48:22.376322+0000","last_unstale":"2024-05-09T08:49:18.362973+0000","last_undegraded":"2024-05-09T08:49:18.362973+0000","last_fullsized":"2024-05-09T08:49:18.362973+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:08:07.859034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.197130+0000","last_change":"2024-05-09T08:48:18.364063+0000","last_active":"2024-05-09T08:49:47.197130+0000","last_peered":"2024-05-09T08:49:47.197130+0000","last_clean":"2024-05-09T08:49:47.197130+0000","last_became_active":"2024-05-09T08:48:18.363694+0000","last_became_peered":"2024-05-09T08:48:18.363694+0000","last_unstale":"2024-05-09T08:49:47.197130+0000","last_undegraded":"2024-05-09T08:49:47.197130+0000","last_fullsized":"2024-05-09T08:49:47.197130+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:42.536329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361351+0000","last_change":"2024-05-09T08:48:17.599787+0000","last_active":"2024-05-09T08:49:18.361351+0000","last_peered":"2024-05-09T08:49:18.361351+0000","last_clean":"2024-05-09T08:49:18.361351+0000","last_became_active":"2024-05-09T08:48:17.599248+0000","last_became_peered":"2024-05-09T08:48:17.599248+0000","last_unstale":"2024-05-09T08:49:18.361351+0000","last_undegraded":"2024-05-09T08:49:18.361351+0000","last_fullsized":"2024-05-09T08:49:18.361351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:17.448404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362948+0000","last_change":"2024-05-09T08:48:22.378913+0000","last_active":"2024-05-09T08:49:18.362948+0000","last_peered":"2024-05-09T08:49:18.362948+0000","last_clean":"2024-05-09T08:49:18.362948+0000","last_became_active":"2024-05-09T08:48:22.378441+0000","last_became_peered":"2024-05-09T08:48:22.378441+0000","last_unstale":"2024-05-09T08:49:18.362948+0000","last_undegraded":"2024-05-09T08:49:18.362948+0000","last_fullsized":"2024-05-09T08:49:18.362948+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:49.623778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.367313+0000","last_change":"2024-05-09T08:48:17.600121+0000","last_active":"2024-05-09T08:49:18.367313+0000","last_peered":"2024-05-09T08:49:18.367313+0000","last_clean":"2024-05-09T08:49:18.367313+0000","last_became_active":"2024-05-09T08:48:17.599800+0000","last_became_peered":"2024-05-09T08:48:17.599800+0000","last_unstale":"2024-05-09T08:49:18.367313+0000","last_undegraded":"2024-05-09T08:49:18.367313+0000","last_fullsized":"2024-05-09T08:49:18.367313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:56.607393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.197288+0000","last_change":"2024-05-09T08:48:18.366612+0000","last_active":"2024-05-09T08:49:47.197288+0000","last_peered":"2024-05-09T08:49:47.197288+0000","last_clean":"2024-05-09T08:49:47.197288+0000","last_became_active":"2024-05-09T08:48:18.366266+0000","last_became_peered":"2024-05-09T08:48:18.366266+0000","last_unstale":"2024-05-09T08:49:47.197288+0000","last_undegraded":"2024-05-09T08:49:47.197288+0000","last_fullsized":"2024-05-09T08:49:47.197288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:35:01.663456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.692585+0000","last_change":"2024-05-09T08:48:22.384096+0000","last_active":"2024-05-09T08:49:18.692585+0000","last_peered":"2024-05-09T08:49:18.692585+0000","last_clean":"2024-05-09T08:49:18.692585+0000","last_became_active":"2024-05-09T08:48:22.383656+0000","last_became_peered":"2024-05-09T08:48:22.383656+0000","last_unstale":"2024-05-09T08:49:18.692585+0000","last_undegraded":"2024-05-09T08:49:18.692585+0000","last_fullsized":"2024-05-09T08:49:18.692585+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:13.622377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'12","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359958+0000","last_change":"2024-05-09T08:48:17.593684+0000","last_active":"2024-05-09T08:49:18.359958+0000","last_peered":"2024-05-09T08:49:18.359958+0000","last_clean":"2024-05-09T08:49:18.359958+0000","last_became_active":"2024-05-09T08:48:17.593358+0000","last_became_peered":"2024-05-09T08:48:17.593358+0000","last_unstale":"2024-05-09T08:49:18.359958+0000","last_undegraded":"2024-05-09T08:49:18.359958+0000","last_fullsized":"2024-05-09T08:49:18.359958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:51:36.023421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.216137+0000","last_change":"2024-05-09T08:48:18.374498+0000","last_active":"2024-05-09T08:49:47.216137+0000","last_peered":"2024-05-09T08:49:47.216137+0000","last_clean":"2024-05-09T08:49:47.216137+0000","last_became_active":"2024-05-09T08:48:18.374142+0000","last_became_peered":"2024-05-09T08:48:18.374142+0000","last_unstale":"2024-05-09T08:49:47.216137+0000","last_undegraded":"2024-05-09T08:49:47.216137+0000","last_fullsized":"2024-05-09T08:49:47.216137+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:53.874419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361533+0000","last_change":"2024-05-09T08:48:22.378330+0000","last_active":"2024-05-09T08:49:18.361533+0000","last_peered":"2024-05-09T08:49:18.361533+0000","last_clean":"2024-05-09T08:49:18.361533+0000","last_became_active":"2024-05-09T08:48:22.377218+0000","last_became_peered":"2024-05-09T08:48:22.377218+0000","last_unstale":"2024-05-09T08:49:18.361533+0000","last_undegraded":"2024-05-09T08:49:18.361533+0000","last_fullsized":"2024-05-09T08:49:18.361533+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:41.650168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359666+0000","last_change":"2024-05-09T08:48:17.601410+0000","last_active":"2024-05-09T08:49:18.359666+0000","last_peered":"2024-05-09T08:49:18.359666+0000","last_clean":"2024-05-09T08:49:18.359666+0000","last_became_active":"2024-05-09T08:48:17.601075+0000","last_became_peered":"2024-05-09T08:48:17.601075+0000","last_unstale":"2024-05-09T08:49:18.359666+0000","last_undegraded":"2024-05-09T08:49:18.359666+0000","last_fullsized":"2024-05-09T08:49:18.359666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:45.760280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.213517+0000","last_change":"2024-05-09T08:48:18.379590+0000","last_active":"2024-05-09T08:49:47.213517+0000","last_peered":"2024-05-09T08:49:47.213517+0000","last_clean":"2024-05-09T08:49:47.213517+0000","last_became_active":"2024-05-09T08:48:18.379187+0000","last_became_peered":"2024-05-09T08:48:18.379187+0000","last_unstale":"2024-05-09T08:49:47.213517+0000","last_undegraded":"2024-05-09T08:49:47.213517+0000","last_fullsized":"2024-05-09T08:49:47.213517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:40:02.630449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359474+0000","last_change":"2024-05-09T08:48:22.386437+0000","last_active":"2024-05-09T08:49:18.359474+0000","last_peered":"2024-05-09T08:49:18.359474+0000","last_clean":"2024-05-09T08:49:18.359474+0000","last_became_active":"2024-05-09T08:48:22.385973+0000","last_became_peered":"2024-05-09T08:48:22.385973+0000","last_unstale":"2024-05-09T08:49:18.359474+0000","last_undegraded":"2024-05-09T08:49:18.359474+0000","last_fullsized":"2024-05-09T08:49:18.359474+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:11:37.338641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362495+0000","last_change":"2024-05-09T08:48:17.598586+0000","last_active":"2024-05-09T08:49:18.362495+0000","last_peered":"2024-05-09T08:49:18.362495+0000","last_clean":"2024-05-09T08:49:18.362495+0000","last_became_active":"2024-05-09T08:48:17.597851+0000","last_became_peered":"2024-05-09T08:48:17.597851+0000","last_unstale":"2024-05-09T08:49:18.362495+0000","last_undegraded":"2024-05-09T08:49:18.362495+0000","last_fullsized":"2024-05-09T08:49:18.362495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:28:22.996999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.173783+0000","last_change":"2024-05-09T08:48:18.367319+0000","last_active":"2024-05-09T08:49:47.173783+0000","last_peered":"2024-05-09T08:49:47.173783+0000","last_clean":"2024-05-09T08:49:47.173783+0000","last_became_active":"2024-05-09T08:48:18.366034+0000","last_became_peered":"2024-05-09T08:48:18.366034+0000","last_unstale":"2024-05-09T08:49:47.173783+0000","last_undegraded":"2024-05-09T08:49:47.173783+0000","last_fullsized":"2024-05-09T08:49:47.173783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:22.998089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359806+0000","last_change":"2024-05-09T08:48:22.389129+0000","last_active":"2024-05-09T08:49:18.359806+0000","last_peered":"2024-05-09T08:49:18.359806+0000","last_clean":"2024-05-09T08:49:18.359806+0000","last_became_active":"2024-05-09T08:48:22.386639+0000","last_became_peered":"2024-05-09T08:48:22.386639+0000","last_unstale":"2024-05-09T08:49:18.359806+0000","last_undegraded":"2024-05-09T08:49:18.359806+0000","last_fullsized":"2024-05-09T08:49:18.359806+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:47:07.257464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.693551+0000","last_change":"2024-05-09T08:48:17.595255+0000","last_active":"2024-05-09T08:49:18.693551+0000","last_peered":"2024-05-09T08:49:18.693551+0000","last_clean":"2024-05-09T08:49:18.693551+0000","last_became_active":"2024-05-09T08:48:17.594882+0000","last_became_peered":"2024-05-09T08:48:17.594882+0000","last_unstale":"2024-05-09T08:49:18.693551+0000","last_undegraded":"2024-05-09T08:49:18.693551+0000","last_fullsized":"2024-05-09T08:49:18.693551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:49.845326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.211374+0000","last_change":"2024-05-09T08:48:18.378580+0000","last_active":"2024-05-09T08:49:47.211374+0000","last_peered":"2024-05-09T08:49:47.211374+0000","last_clean":"2024-05-09T08:49:47.211374+0000","last_became_active":"2024-05-09T08:48:18.378205+0000","last_became_peered":"2024-05-09T08:48:18.378205+0000","last_unstale":"2024-05-09T08:49:47.211374+0000","last_undegraded":"2024-05-09T08:49:47.211374+0000","last_fullsized":"2024-05-09T08:49:47.211374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:50.103195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359371+0000","last_change":"2024-05-09T08:48:22.378494+0000","last_active":"2024-05-09T08:49:18.359371+0000","last_peered":"2024-05-09T08:49:18.359371+0000","last_clean":"2024-05-09T08:49:18.359371+0000","last_became_active":"2024-05-09T08:48:22.378076+0000","last_became_peered":"2024-05-09T08:48:22.378076+0000","last_unstale":"2024-05-09T08:49:18.359371+0000","last_undegraded":"2024-05-09T08:49:18.359371+0000","last_fullsized":"2024-05-09T08:49:18.359371+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:40:09.079968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362143+0000","last_change":"2024-05-09T08:48:17.601433+0000","last_active":"2024-05-09T08:49:18.362143+0000","last_peered":"2024-05-09T08:49:18.362143+0000","last_clean":"2024-05-09T08:49:18.362143+0000","last_became_active":"2024-05-09T08:48:17.601128+0000","last_became_peered":"2024-05-09T08:48:17.601128+0000","last_unstale":"2024-05-09T08:49:18.362143+0000","last_undegraded":"2024-05-09T08:49:18.362143+0000","last_fullsized":"2024-05-09T08:49:18.362143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:33:26.207437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.216272+0000","last_change":"2024-05-09T08:48:18.373569+0000","last_active":"2024-05-09T08:49:47.216272+0000","last_peered":"2024-05-09T08:49:47.216272+0000","last_clean":"2024-05-09T08:49:47.216272+0000","last_became_active":"2024-05-09T08:48:18.373238+0000","last_became_peered":"2024-05-09T08:48:18.373238+0000","last_unstale":"2024-05-09T08:49:47.216272+0000","last_undegraded":"2024-05-09T08:49:47.216272+0000","last_fullsized":"2024-05-09T08:49:47.216272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:17:41.629419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.359480+0000","last_change":"2024-05-09T08:48:22.381523+0000","last_active":"2024-05-09T08:49:18.359480+0000","last_peered":"2024-05-09T08:49:18.359480+0000","last_clean":"2024-05-09T08:49:18.359480+0000","last_became_active":"2024-05-09T08:48:22.380626+0000","last_became_peered":"2024-05-09T08:48:22.380626+0000","last_unstale":"2024-05-09T08:49:18.359480+0000","last_undegraded":"2024-05-09T08:49:18.359480+0000","last_fullsized":"2024-05-09T08:49:18.359480+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:47:45.092232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.366669+0000","last_change":"2024-05-09T08:48:17.601479+0000","last_active":"2024-05-09T08:49:18.366669+0000","last_peered":"2024-05-09T08:49:18.366669+0000","last_clean":"2024-05-09T08:49:18.366669+0000","last_became_active":"2024-05-09T08:48:17.601140+0000","last_became_peered":"2024-05-09T08:48:17.601140+0000","last_unstale":"2024-05-09T08:49:18.366669+0000","last_undegraded":"2024-05-09T08:49:18.366669+0000","last_fullsized":"2024-05-09T08:49:18.366669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:38:24.011209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.219155+0000","last_change":"2024-05-09T08:48:18.376890+0000","last_active":"2024-05-09T08:49:47.219155+0000","last_peered":"2024-05-09T08:49:47.219155+0000","last_clean":"2024-05-09T08:49:47.219155+0000","last_became_active":"2024-05-09T08:48:18.376472+0000","last_became_peered":"2024-05-09T08:48:18.376472+0000","last_unstale":"2024-05-09T08:49:47.219155+0000","last_undegraded":"2024-05-09T08:49:47.219155+0000","last_fullsized":"2024-05-09T08:49:47.219155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:40:16.238539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362579+0000","last_change":"2024-05-09T08:48:22.374743+0000","last_active":"2024-05-09T08:49:18.362579+0000","last_peered":"2024-05-09T08:49:18.362579+0000","last_clean":"2024-05-09T08:49:18.362579+0000","last_became_active":"2024-05-09T08:48:22.374324+0000","last_became_peered":"2024-05-09T08:48:22.374324+0000","last_unstale":"2024-05-09T08:49:18.362579+0000","last_undegraded":"2024-05-09T08:49:18.362579+0000","last_fullsized":"2024-05-09T08:49:18.362579+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:48.073647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":109,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:52.867309+0000","last_change":"2024-05-09T08:48:17.604700+0000","last_active":"2024-05-09T08:49:52.867309+0000","last_peered":"2024-05-09T08:49:52.867309+0000","last_clean":"2024-05-09T08:49:52.867309+0000","last_became_active":"2024-05-09T08:48:17.604334+0000","last_became_peered":"2024-05-09T08:48:17.604334+0000","last_unstale":"2024-05-09T08:49:52.867309+0000","last_undegraded":"2024-05-09T08:49:52.867309+0000","last_fullsized":"2024-05-09T08:49:52.867309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:15:37.346232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":77,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.219459+0000","last_change":"2024-05-09T08:48:18.375769+0000","last_active":"2024-05-09T08:49:47.219459+0000","last_peered":"2024-05-09T08:49:47.219459+0000","last_clean":"2024-05-09T08:49:47.219459+0000","last_became_active":"2024-05-09T08:48:18.375059+0000","last_became_peered":"2024-05-09T08:48:18.375059+0000","last_unstale":"2024-05-09T08:49:47.219459+0000","last_undegraded":"2024-05-09T08:49:47.219459+0000","last_fullsized":"2024-05-09T08:49:47.219459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:51.001240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.362302+0000","last_change":"2024-05-09T08:48:22.385306+0000","last_active":"2024-05-09T08:49:18.362302+0000","last_peered":"2024-05-09T08:49:18.362302+0000","last_clean":"2024-05-09T08:49:18.362302+0000","last_became_active":"2024-05-09T08:48:22.383890+0000","last_became_peered":"2024-05-09T08:48:22.383890+0000","last_unstale":"2024-05-09T08:49:18.362302+0000","last_undegraded":"2024-05-09T08:49:18.362302+0000","last_fullsized":"2024-05-09T08:49:18.362302+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:50:52.369399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.364051+0000","last_change":"2024-05-09T08:48:17.591627+0000","last_active":"2024-05-09T08:49:18.364051+0000","last_peered":"2024-05-09T08:49:18.364051+0000","last_clean":"2024-05-09T08:49:18.364051+0000","last_became_active":"2024-05-09T08:48:17.591250+0000","last_became_peered":"2024-05-09T08:48:17.591250+0000","last_unstale":"2024-05-09T08:49:18.364051+0000","last_undegraded":"2024-05-09T08:49:18.364051+0000","last_fullsized":"2024-05-09T08:49:18.364051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:40:59.748892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.218081+0000","last_change":"2024-05-09T08:48:18.372256+0000","last_active":"2024-05-09T08:49:47.218081+0000","last_peered":"2024-05-09T08:49:47.218081+0000","last_clean":"2024-05-09T08:49:47.218081+0000","last_became_active":"2024-05-09T08:48:18.371895+0000","last_became_peered":"2024-05-09T08:48:18.371895+0000","last_unstale":"2024-05-09T08:49:47.218081+0000","last_undegraded":"2024-05-09T08:49:47.218081+0000","last_fullsized":"2024-05-09T08:49:47.218081+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:25.067587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.361711+0000","last_change":"2024-05-09T08:48:22.373684+0000","last_active":"2024-05-09T08:49:18.361711+0000","last_peered":"2024-05-09T08:49:18.361711+0000","last_clean":"2024-05-09T08:49:18.361711+0000","last_became_active":"2024-05-09T08:48:22.373000+0000","last_became_peered":"2024-05-09T08:48:22.373000+0000","last_unstale":"2024-05-09T08:49:18.361711+0000","last_undegraded":"2024-05-09T08:49:18.361711+0000","last_fullsized":"2024-05-09T08:49:18.361711+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:53:14.416837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:18.363742+0000","last_change":"2024-05-09T08:48:17.596910+0000","last_active":"2024-05-09T08:49:18.363742+0000","last_peered":"2024-05-09T08:49:18.363742+0000","last_clean":"2024-05-09T08:49:18.363742+0000","last_became_active":"2024-05-09T08:48:17.596605+0000","last_became_peered":"2024-05-09T08:48:17.596605+0000","last_unstale":"2024-05-09T08:49:18.363742+0000","last_undegraded":"2024-05-09T08:49:18.363742+0000","last_fullsized":"2024-05-09T08:49:18.363742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:05:25.880082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.218708+0000","last_change":"2024-05-09T08:48:18.370354+0000","last_active":"2024-05-09T08:49:47.218708+0000","last_peered":"2024-05-09T08:49:47.218708+0000","last_clean":"2024-05-09T08:49:47.218708+0000","last_became_active":"2024-05-09T08:48:18.370043+0000","last_became_peered":"2024-05-09T08:48:18.370043+0000","last_unstale":"2024-05-09T08:49:47.218708+0000","last_undegraded":"2024-05-09T08:49:47.218708+0000","last_fullsized":"2024-05-09T08:49:47.218708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:21:34.066162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.209350+0000","last_change":"2024-05-09T08:48:18.376431+0000","last_active":"2024-05-09T08:49:47.209350+0000","last_peered":"2024-05-09T08:49:47.209350+0000","last_clean":"2024-05-09T08:49:47.209350+0000","last_became_active":"2024-05-09T08:48:18.375598+0000","last_became_peered":"2024-05-09T08:48:18.375598+0000","last_unstale":"2024-05-09T08:49:47.209350+0000","last_undegraded":"2024-05-09T08:49:47.209350+0000","last_fullsized":"2024-05-09T08:49:47.209350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:51:13.481219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.211305+0000","last_change":"2024-05-09T08:48:18.371460+0000","last_active":"2024-05-09T08:49:47.211305+0000","last_peered":"2024-05-09T08:49:47.211305+0000","last_clean":"2024-05-09T08:49:47.211305+0000","last_became_active":"2024-05-09T08:48:18.371091+0000","last_became_peered":"2024-05-09T08:48:18.371091+0000","last_unstale":"2024-05-09T08:49:47.211305+0000","last_undegraded":"2024-05-09T08:49:47.211305+0000","last_fullsized":"2024-05-09T08:49:47.211305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:02.829902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.215028+0000","last_change":"2024-05-09T08:48:18.369453+0000","last_active":"2024-05-09T08:49:47.215028+0000","last_peered":"2024-05-09T08:49:47.215028+0000","last_clean":"2024-05-09T08:49:47.215028+0000","last_became_active":"2024-05-09T08:48:18.368966+0000","last_became_peered":"2024-05-09T08:48:18.368966+0000","last_unstale":"2024-05-09T08:49:47.215028+0000","last_undegraded":"2024-05-09T08:49:47.215028+0000","last_fullsized":"2024-05-09T08:49:47.215028+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:23:14.359222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.210404+0000","last_change":"2024-05-09T08:48:18.378815+0000","last_active":"2024-05-09T08:49:47.210404+0000","last_peered":"2024-05-09T08:49:47.210404+0000","last_clean":"2024-05-09T08:49:47.210404+0000","last_became_active":"2024-05-09T08:48:18.378479+0000","last_became_peered":"2024-05-09T08:48:18.378479+0000","last_unstale":"2024-05-09T08:49:47.210404+0000","last_undegraded":"2024-05-09T08:49:47.210404+0000","last_fullsized":"2024-05-09T08:49:47.210404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:38:50.133239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.214360+0000","last_change":"2024-05-09T08:48:18.365696+0000","last_active":"2024-05-09T08:49:47.214360+0000","last_peered":"2024-05-09T08:49:47.214360+0000","last_clean":"2024-05-09T08:49:47.214360+0000","last_became_active":"2024-05-09T08:48:18.365358+0000","last_became_peered":"2024-05-09T08:48:18.365358+0000","last_unstale":"2024-05-09T08:49:47.214360+0000","last_undegraded":"2024-05-09T08:49:47.214360+0000","last_fullsized":"2024-05-09T08:49:47.214360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:04.064150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.204531+0000","last_change":"2024-05-09T08:48:18.368911+0000","last_active":"2024-05-09T08:49:47.204531+0000","last_peered":"2024-05-09T08:49:47.204531+0000","last_clean":"2024-05-09T08:49:47.204531+0000","last_became_active":"2024-05-09T08:48:18.368591+0000","last_became_peered":"2024-05-09T08:48:18.368591+0000","last_unstale":"2024-05-09T08:49:47.204531+0000","last_undegraded":"2024-05-09T08:49:47.204531+0000","last_fullsized":"2024-05-09T08:49:47.204531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:34:09.692168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.203098+0000","last_change":"2024-05-09T08:48:18.373629+0000","last_active":"2024-05-09T08:49:47.203098+0000","last_peered":"2024-05-09T08:49:47.203098+0000","last_clean":"2024-05-09T08:49:47.203098+0000","last_became_active":"2024-05-09T08:48:18.372992+0000","last_became_peered":"2024-05-09T08:48:18.372992+0000","last_unstale":"2024-05-09T08:49:47.203098+0000","last_undegraded":"2024-05-09T08:49:47.203098+0000","last_fullsized":"2024-05-09T08:49:47.203098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:42.532250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.199157+0000","last_change":"2024-05-09T08:48:18.382423+0000","last_active":"2024-05-09T08:49:47.199157+0000","last_peered":"2024-05-09T08:49:47.199157+0000","last_clean":"2024-05-09T08:49:47.199157+0000","last_became_active":"2024-05-09T08:48:18.382088+0000","last_became_peered":"2024-05-09T08:48:18.382088+0000","last_unstale":"2024-05-09T08:49:47.199157+0000","last_undegraded":"2024-05-09T08:49:47.199157+0000","last_fullsized":"2024-05-09T08:49:47.199157+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:09:06.342557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.201504+0000","last_change":"2024-05-09T08:48:18.370460+0000","last_active":"2024-05-09T08:49:47.201504+0000","last_peered":"2024-05-09T08:49:47.201504+0000","last_clean":"2024-05-09T08:49:47.201504+0000","last_became_active":"2024-05-09T08:48:18.370033+0000","last_became_peered":"2024-05-09T08:48:18.370033+0000","last_unstale":"2024-05-09T08:49:47.201504+0000","last_undegraded":"2024-05-09T08:49:47.201504+0000","last_fullsized":"2024-05-09T08:49:47.201504+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:21:28.622511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.215015+0000","last_change":"2024-05-09T08:48:18.371410+0000","last_active":"2024-05-09T08:49:47.215015+0000","last_peered":"2024-05-09T08:49:47.215015+0000","last_clean":"2024-05-09T08:49:47.215015+0000","last_became_active":"2024-05-09T08:48:18.371067+0000","last_became_peered":"2024-05-09T08:48:18.371067+0000","last_unstale":"2024-05-09T08:49:47.215015+0000","last_undegraded":"2024-05-09T08:49:47.215015+0000","last_fullsized":"2024-05-09T08:49:47.215015+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:35:11.445992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.216959+0000","last_change":"2024-05-09T08:48:18.382873+0000","last_active":"2024-05-09T08:49:47.216959+0000","last_peered":"2024-05-09T08:49:47.216959+0000","last_clean":"2024-05-09T08:49:47.216959+0000","last_became_active":"2024-05-09T08:48:18.382366+0000","last_became_peered":"2024-05-09T08:48:18.382366+0000","last_unstale":"2024-05-09T08:49:47.216959+0000","last_undegraded":"2024-05-09T08:49:47.216959+0000","last_fullsized":"2024-05-09T08:49:47.216959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:23.343131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.216253+0000","last_change":"2024-05-09T08:48:18.370821+0000","last_active":"2024-05-09T08:49:47.216253+0000","last_peered":"2024-05-09T08:49:47.216253+0000","last_clean":"2024-05-09T08:49:47.216253+0000","last_became_active":"2024-05-09T08:48:18.368208+0000","last_became_peered":"2024-05-09T08:48:18.368208+0000","last_unstale":"2024-05-09T08:49:47.216253+0000","last_undegraded":"2024-05-09T08:49:47.216253+0000","last_fullsized":"2024-05-09T08:49:47.216253+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:14.633938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-05-09T08:49:47.204816+0000","last_change":"2024-05-09T08:48:18.371626+0000","last_active":"2024-05-09T08:49:47.204816+0000","last_peered":"2024-05-09T08:49:47.204816+0000","last_clean":"2024-05-09T08:49:47.204816+0000","last_became_active":"2024-05-09T08:48:18.371081+0000","last_became_peered":"2024-05-09T08:48:18.371081+0000","last_unstale":"2024-05-09T08:49:47.204816+0000","last_undegraded":"2024-05-09T08:49:47.204816+0000","last_fullsized":"2024-05-09T08:49:47.204816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:27.372934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":210578,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68377,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":793,"num_read_kb":470,"num_write":247,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":205131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351009,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":505071,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612646,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1163264,"data_stored":1157406,"data_compressed":56208,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776171,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":505001,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":505379,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299445,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1102549,"data_compressed":56208,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462970,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":500505,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":562426,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":501961,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":563178,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":500213,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888133,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":505111,"data_compressed":20565,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724607,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1097599,"data_compressed":56208,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34003,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34003,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:49:56.335 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-09T08:49:56.335 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-09T08:49:56.335 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-09T08:49:56.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph health --format=json 2024-05-09T08:49:56.578 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:49:56.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:56 smithi098 ceph-mon[30092]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-05-09T08:49:56.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:56 smithi142 ceph-mon[34625]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-05-09T08:49:56.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:56 smithi105 ceph-mon[34498]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-05-09T08:49:57.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:57 smithi098 ceph-mon[30092]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:49:57.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:57 smithi142 ceph-mon[34625]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:49:57.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:57 smithi105 ceph-mon[34498]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:49:58.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:58 smithi098 ceph-mon[30092]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-09T08:49:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.332+0000 7fd01a840700 1 -- 172.21.15.98:0/3811480259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0141083a0 msgr2=0x7fd014108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.332+0000 7fd01a840700 1 --2- 172.21.15.98:0/3811480259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0141083a0 0x7fd014108780 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd004009990 tx=0x7fd004009ca0 comp rx=0 tx=0).stop 2024-05-09T08:49:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.332+0000 7fd01a840700 1 -- 172.21.15.98:0/3811480259 shutdown_connections 2024-05-09T08:49:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.332+0000 7fd01a840700 1 --2- 172.21.15.98:0/3811480259 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01406a9d0 0x7fd01410f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.332+0000 7fd01a840700 1 --2- 172.21.15.98:0/3811480259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0141083a0 0x7fd014108780 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.332+0000 7fd01a840700 1 --2- 172.21.15.98:0/3811480259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd01406a030 0x7fd01406a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.332+0000 7fd01a840700 1 -- 172.21.15.98:0/3811480259 >> 172.21.15.98:0/3811480259 conn(0x7fd0140781d0 msgr2=0x7fd0140785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.332+0000 7fd01a840700 1 -- 172.21.15.98:0/3811480259 shutdown_connections 2024-05-09T08:49:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.332+0000 7fd01a840700 1 -- 172.21.15.98:0/3811480259 wait complete. 2024-05-09T08:49:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.333+0000 7fd01a840700 1 Processor -- start 2024-05-09T08:49:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.333+0000 7fd01a840700 1 -- start start 2024-05-09T08:49:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.333+0000 7fd01a840700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd01406a030 0x7fd0141012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.333+0000 7fd01a840700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01406a9d0 0x7fd014101830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.333+0000 7fd01a840700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0141083a0 0x7fd014101d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.333+0000 7fd01a840700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd0141126e0 con 0x7fd01406a030 2024-05-09T08:49:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.333+0000 7fd01a840700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd014112560 con 0x7fd0141083a0 2024-05-09T08:49:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.333+0000 7fd01a840700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd014112860 con 0x7fd01406a9d0 2024-05-09T08:49:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd013fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd01406a030 0x7fd0141012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd013fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd01406a030 0x7fd0141012f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51924/0 (socket says 172.21.15.98:51924) 2024-05-09T08:49:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd013fff700 1 -- 172.21.15.98:0/159858320 learned_addr learned my addr 172.21.15.98:0/159858320 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd0137fe700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01406a9d0 0x7fd014101830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd018ddd700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0141083a0 0x7fd014101d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd013fff700 1 -- 172.21.15.98:0/159858320 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01406a9d0 msgr2=0x7fd014101830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd013fff700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01406a9d0 0x7fd014101830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd013fff700 1 -- 172.21.15.98:0/159858320 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0141083a0 msgr2=0x7fd014101d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd013fff700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0141083a0 0x7fd014101d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd013fff700 1 -- 172.21.15.98:0/159858320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0141a7100 con 0x7fd01406a030 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd0137fe700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01406a9d0 0x7fd014101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd013fff700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd01406a030 0x7fd0141012f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd004000c00 tx=0x7fd004003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd0117fa700 1 -- 172.21.15.98:0/159858320 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd004027070 con 0x7fd01406a030 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd01a840700 1 -- 172.21.15.98:0/159858320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0141a7390 con 0x7fd01406a030 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd01a840700 1 -- 172.21.15.98:0/159858320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0141a7840 con 0x7fd01406a030 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd0117fa700 1 -- 172.21.15.98:0/159858320 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd004004190 con 0x7fd01406a030 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.334+0000 7fd0117fa700 1 -- 172.21.15.98:0/159858320 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00400f460 con 0x7fd01406a030 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.335+0000 7fd01a840700 1 -- 172.21.15.98:0/159858320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd014069010 con 0x7fd01406a030 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.336+0000 7fd0117fa700 1 -- 172.21.15.98:0/159858320 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd00402a070 con 0x7fd01406a030 2024-05-09T08:49:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.337+0000 7fd0117fa700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcffc0a2c20 0x7fcffc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:58.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.337+0000 7fd0137fe700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcffc0a2c20 0x7fcffc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:58.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.337+0000 7fd0117fa700 1 -- 172.21.15.98:0/159858320 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fd0040d06d0 con 0x7fd01406a030 2024-05-09T08:49:58.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.337+0000 7fd0137fe700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcffc0a2c20 0x7fcffc0a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd014078880 tx=0x7fd008008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:58.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.341+0000 7fd0117fa700 1 -- 172.21.15.98:0/159858320 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd004013070 con 0x7fd01406a030 2024-05-09T08:49:58.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.585+0000 7fd01a840700 1 -- 172.21.15.98:0/159858320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd01402db80 con 0x7fd01406a030 2024-05-09T08:49:58.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.585+0000 7fd0117fa700 1 -- 172.21.15.98:0/159858320 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd00409d360 con 0x7fd01406a030 2024-05-09T08:49:58.585 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:49:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-09T08:49:58.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.588+0000 7fd01a840700 1 -- 172.21.15.98:0/159858320 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcffc0a2c20 msgr2=0x7fcffc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:58.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.588+0000 7fd01a840700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcffc0a2c20 0x7fcffc0a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd014078880 tx=0x7fd008008040 comp rx=0 tx=0).stop 2024-05-09T08:49:58.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.588+0000 7fd01a840700 1 -- 172.21.15.98:0/159858320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd01406a030 msgr2=0x7fd0141012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:58.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.588+0000 7fd01a840700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd01406a030 0x7fd0141012f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd004000c00 tx=0x7fd004003730 comp rx=0 tx=0).stop 2024-05-09T08:49:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.588+0000 7fd01a840700 1 -- 172.21.15.98:0/159858320 shutdown_connections 2024-05-09T08:49:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.588+0000 7fd01a840700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcffc0a2c20 0x7fcffc0a50e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.588+0000 7fd01a840700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0141083a0 0x7fd014101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.588+0000 7fd01a840700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd01406a030 0x7fd0141012f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.588+0000 7fd01a840700 1 --2- 172.21.15.98:0/159858320 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01406a9d0 0x7fd014101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.588+0000 7fd01a840700 1 -- 172.21.15.98:0/159858320 >> 172.21.15.98:0/159858320 conn(0x7fd0140781d0 msgr2=0x7fd014100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.588+0000 7fd01a840700 1 -- 172.21.15.98:0/159858320 shutdown_connections 2024-05-09T08:49:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:58.588+0000 7fd01a840700 1 -- 172.21.15.98:0/159858320 wait complete. 2024-05-09T08:49:58.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:58 smithi142 ceph-mon[34625]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-09T08:49:59.243 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:58 smithi105 ceph-mon[34498]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-09T08:49:59.258 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-09T08:49:59.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:49:59.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.366+0000 7f1176716640 1 -- 172.21.15.98:0/1494136052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11700ffb50 msgr2=0x7f11700fff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:59.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.366+0000 7f1176716640 1 --2- 172.21.15.98:0/1494136052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11700ffb50 0x7f11700fff70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1160009930 tx=0x7f116002a420 comp rx=0 tx=0).stop 2024-05-09T08:49:59.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.367+0000 7f1176716640 1 -- 172.21.15.98:0/1494136052 shutdown_connections 2024-05-09T08:49:59.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.367+0000 7f1176716640 1 --2- 172.21.15.98:0/1494136052 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11701034e0 0x7f1170109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.367+0000 7f1176716640 1 --2- 172.21.15.98:0/1494136052 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1170102b20 0x7f1170102fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.367+0000 7f1176716640 1 --2- 172.21.15.98:0/1494136052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11700ffb50 0x7f11700fff70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.367+0000 7f1176716640 1 -- 172.21.15.98:0/1494136052 >> 172.21.15.98:0/1494136052 conn(0x7f11700fb4a0 msgr2=0x7f11700fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.367+0000 7f1176716640 1 -- 172.21.15.98:0/1494136052 shutdown_connections 2024-05-09T08:49:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.367+0000 7f1176716640 1 -- 172.21.15.98:0/1494136052 wait complete. 2024-05-09T08:49:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.367+0000 7f1176716640 1 Processor -- start 2024-05-09T08:49:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.367+0000 7f1176716640 1 -- start start 2024-05-09T08:49:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.367+0000 7f1176716640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11700ffb50 0x7f1170197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.367+0000 7f1176716640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1170102b20 0x7f1170198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.367+0000 7f1176716640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11701034e0 0x7f117019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.367+0000 7f1176716640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f117010d500 con 0x7f11701034e0 2024-05-09T08:49:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f1176716640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f117010d380 con 0x7f1170102b20 2024-05-09T08:49:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f1176716640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f117010d680 con 0x7f11700ffb50 2024-05-09T08:49:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f1174cb3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11701034e0 0x7f117019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f1174cb3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11701034e0 0x7f117019f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51946/0 (socket says 172.21.15.98:51946) 2024-05-09T08:49:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f1174cb3640 1 -- 172.21.15.98:0/2998317563 learned_addr learned my addr 172.21.15.98:0/2998317563 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f116ffff640 1 --2- 172.21.15.98:0/2998317563 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11700ffb50 0x7f1170197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f1174cb3640 1 -- 172.21.15.98:0/2998317563 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11700ffb50 msgr2=0x7f1170197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f1174cb3640 1 --2- 172.21.15.98:0/2998317563 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11700ffb50 0x7f1170197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f1174cb3640 1 -- 172.21.15.98:0/2998317563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1170102b20 msgr2=0x7f1170198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f1174cb3640 1 --2- 172.21.15.98:0/2998317563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1170102b20 0x7f1170198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f1174cb3640 1 -- 172.21.15.98:0/2998317563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f117019f770 con 0x7f11701034e0 2024-05-09T08:49:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f116ffff640 1 --2- 172.21.15.98:0/2998317563 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11700ffb50 0x7f1170197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:49:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f1174cb3640 1 --2- 172.21.15.98:0/2998317563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11701034e0 0x7f117019f100 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f116800cce0 tx=0x7f116800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f116d7fa640 1 -- 172.21.15.98:0/2998317563 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116800f450 con 0x7f11701034e0 2024-05-09T08:49:59.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f116d7fa640 1 -- 172.21.15.98:0/2998317563 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f116800fad0 con 0x7f11701034e0 2024-05-09T08:49:59.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f1176716640 1 -- 172.21.15.98:0/2998317563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f117019fa60 con 0x7f11701034e0 2024-05-09T08:49:59.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f116d7fa640 1 -- 172.21.15.98:0/2998317563 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1168018c10 con 0x7f11701034e0 2024-05-09T08:49:59.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.368+0000 7f1176716640 1 -- 172.21.15.98:0/2998317563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f117019ff60 con 0x7f11701034e0 2024-05-09T08:49:59.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.369+0000 7f1176716640 1 -- 172.21.15.98:0/2998317563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1170108cf0 con 0x7f11701034e0 2024-05-09T08:49:59.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.370+0000 7f116d7fa640 1 -- 172.21.15.98:0/2998317563 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f116800f5f0 con 0x7f11701034e0 2024-05-09T08:49:59.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.371+0000 7f116d7fa640 1 --2- 172.21.15.98:0/2998317563 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f11580a2b90 0x7f11580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:59.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.371+0000 7f116ffff640 1 --2- 172.21.15.98:0/2998317563 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f11580a2b90 0x7f11580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:59.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.371+0000 7f116d7fa640 1 -- 172.21.15.98:0/2998317563 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f11680cfb10 con 0x7f11701034e0 2024-05-09T08:49:59.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.371+0000 7f116ffff640 1 --2- 172.21.15.98:0/2998317563 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f11580a2b90 0x7f11580a5050 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1160006010 tx=0x7f1160000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:59.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.375+0000 7f116d7fa640 1 -- 172.21.15.98:0/2998317563 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1168014030 con 0x7f11701034e0 2024-05-09T08:49:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.553+0000 7f1176716640 1 -- 172.21.15.98:0/2998317563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f11700624d0 con 0x7f11701034e0 2024-05-09T08:49:59.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.554+0000 7f116d7fa640 1 -- 172.21.15.98:0/2998317563 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24622 (secure 0 0 0) 0x7f116809c710 con 0x7f11701034e0 2024-05-09T08:49:59.554 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:49:59.554 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":92,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:49:18.355914+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:49:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.556+0000 7f1176716640 1 -- 172.21.15.98:0/2998317563 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f11580a2b90 msgr2=0x7f11580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.556+0000 7f1176716640 1 --2- 172.21.15.98:0/2998317563 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f11580a2b90 0x7f11580a5050 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1160006010 tx=0x7f1160000bc0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.556+0000 7f1176716640 1 -- 172.21.15.98:0/2998317563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11701034e0 msgr2=0x7f117019f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.556+0000 7f1176716640 1 --2- 172.21.15.98:0/2998317563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11701034e0 0x7f117019f100 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f116800cce0 tx=0x7f116800a620 comp rx=0 tx=0).stop 2024-05-09T08:49:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.556+0000 7f1176716640 1 -- 172.21.15.98:0/2998317563 shutdown_connections 2024-05-09T08:49:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.556+0000 7f1176716640 1 --2- 172.21.15.98:0/2998317563 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f11580a2b90 0x7f11580a5050 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.556+0000 7f1176716640 1 --2- 172.21.15.98:0/2998317563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11701034e0 0x7f117019f100 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.556+0000 7f1176716640 1 --2- 172.21.15.98:0/2998317563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1170102b20 0x7f1170198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.556+0000 7f1176716640 1 --2- 172.21.15.98:0/2998317563 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11700ffb50 0x7f1170197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.556+0000 7f1176716640 1 -- 172.21.15.98:0/2998317563 >> 172.21.15.98:0/2998317563 conn(0x7f11700fb4a0 msgr2=0x7f1170105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.556+0000 7f1176716640 1 -- 172.21.15.98:0/2998317563 shutdown_connections 2024-05-09T08:49:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.556+0000 7f1176716640 1 -- 172.21.15.98:0/2998317563 wait complete. 2024-05-09T08:49:59.566 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:49:59 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/159858320' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-09T08:49:59.567 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:43:39.424443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:48:15.866450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:48:17.122162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:48:20.591475+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:49:59.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:49:59.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:49:59 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/159858320' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-09T08:49:59.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.675+0000 7f59faaa5640 1 -- 172.21.15.98:0/4149081877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4101880 msgr2=0x7f59f4101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:59.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.675+0000 7f59faaa5640 1 --2- 172.21.15.98:0/4149081877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4101880 0x7f59f4101ca0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f59e8009900 tx=0x7f59e802a410 comp rx=0 tx=0).stop 2024-05-09T08:49:59.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.675+0000 7f59faaa5640 1 -- 172.21.15.98:0/4149081877 shutdown_connections 2024-05-09T08:49:59.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.675+0000 7f59faaa5640 1 --2- 172.21.15.98:0/4149081877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f4103430 0x7f59f4109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.675+0000 7f59faaa5640 1 --2- 172.21.15.98:0/4149081877 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59f4102a70 0x7f59f4102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.675+0000 7f59faaa5640 1 --2- 172.21.15.98:0/4149081877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4101880 0x7f59f4101ca0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.675+0000 7f59faaa5640 1 -- 172.21.15.98:0/4149081877 >> 172.21.15.98:0/4149081877 conn(0x7f59f40fce10 msgr2=0x7f59f40ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.675+0000 7f59faaa5640 1 -- 172.21.15.98:0/4149081877 shutdown_connections 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.675+0000 7f59faaa5640 1 -- 172.21.15.98:0/4149081877 wait complete. 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.675+0000 7f59faaa5640 1 Processor -- start 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.675+0000 7f59faaa5640 1 -- start start 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59faaa5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4101880 0x7f59f4197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59faaa5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59f4102a70 0x7f59f4198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59faaa5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f4103430 0x7f59f419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59faaa5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59f410d4f0 con 0x7f59f4101880 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59faaa5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f59f410d370 con 0x7f59f4103430 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59faaa5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f59f410d670 con 0x7f59f4102a70 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59f8841640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4101880 0x7f59f4197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59f8841640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4101880 0x7f59f4197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51960/0 (socket says 172.21.15.98:51960) 2024-05-09T08:49:59.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59f8841640 1 -- 172.21.15.98:0/1453453519 learned_addr learned my addr 172.21.15.98:0/1453453519 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:59.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59f3fff640 1 --2- 172.21.15.98:0/1453453519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59f4102a70 0x7f59f4198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:59.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59f8841640 1 -- 172.21.15.98:0/1453453519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59f4102a70 msgr2=0x7f59f4198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:59.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59f8841640 1 --2- 172.21.15.98:0/1453453519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59f4102a70 0x7f59f4198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59f8841640 1 -- 172.21.15.98:0/1453453519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f4103430 msgr2=0x7f59f419f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:59.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59f8841640 1 --2- 172.21.15.98:0/1453453519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f4103430 0x7f59f419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59f8841640 1 -- 172.21.15.98:0/1453453519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f419f7e0 con 0x7f59f4101880 2024-05-09T08:49:59.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59f8841640 1 --2- 172.21.15.98:0/1453453519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4101880 0x7f59f4197bc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f59e8031610 tx=0x7f59e80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:59.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59f1ffb640 1 -- 172.21.15.98:0/1453453519 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e803f030 con 0x7f59f4101880 2024-05-09T08:49:59.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59faaa5640 1 -- 172.21.15.98:0/1453453519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f419fa70 con 0x7f59f4101880 2024-05-09T08:49:59.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.676+0000 7f59f1ffb640 1 -- 172.21.15.98:0/1453453519 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59e8003ce0 con 0x7f59f4101880 2024-05-09T08:49:59.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.677+0000 7f59faaa5640 1 -- 172.21.15.98:0/1453453519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f419ffb0 con 0x7f59f4101880 2024-05-09T08:49:59.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.677+0000 7f59f1ffb640 1 -- 172.21.15.98:0/1453453519 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e8049060 con 0x7f59f4101880 2024-05-09T08:49:59.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.677+0000 7f59faaa5640 1 -- 172.21.15.98:0/1453453519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59f4108cc0 con 0x7f59f4101880 2024-05-09T08:49:59.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.678+0000 7f59f1ffb640 1 -- 172.21.15.98:0/1453453519 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f59e8042030 con 0x7f59f4101880 2024-05-09T08:49:59.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.679+0000 7f59f1ffb640 1 --2- 172.21.15.98:0/1453453519 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f59e40a2b90 0x7f59e40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:59.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.679+0000 7f59f3fff640 1 --2- 172.21.15.98:0/1453453519 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f59e40a2b90 0x7f59e40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:59.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.680+0000 7f59f1ffb640 1 -- 172.21.15.98:0/1453453519 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f59e80ef9a0 con 0x7f59f4101880 2024-05-09T08:49:59.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.680+0000 7f59f3fff640 1 --2- 172.21.15.98:0/1453453519 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f59e40a2b90 0x7f59e40a5050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f59e0005fd0 tx=0x7f59e0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:59.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.684+0000 7f59f1ffb640 1 -- 172.21.15.98:0/1453453519 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f59e80bc5a0 con 0x7f59f4101880 2024-05-09T08:49:59.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:49:59 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/159858320' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-09T08:49:59.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.860+0000 7f59faaa5640 1 -- 172.21.15.98:0/1453453519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f59f40624d0 con 0x7f59f4101880 2024-05-09T08:49:59.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.860+0000 7f59f1ffb640 1 -- 172.21.15.98:0/1453453519 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f59e80bbea0 con 0x7f59f4101880 2024-05-09T08:49:59.860 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-09T08:49:59.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.863+0000 7f59faaa5640 1 -- 172.21.15.98:0/1453453519 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f59e40a2b90 msgr2=0x7f59e40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:59.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.863+0000 7f59faaa5640 1 --2- 172.21.15.98:0/1453453519 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f59e40a2b90 0x7f59e40a5050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f59e0005fd0 tx=0x7f59e0005850 comp rx=0 tx=0).stop 2024-05-09T08:49:59.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.863+0000 7f59faaa5640 1 -- 172.21.15.98:0/1453453519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4101880 msgr2=0x7f59f4197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:59.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.863+0000 7f59faaa5640 1 --2- 172.21.15.98:0/1453453519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4101880 0x7f59f4197bc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f59e8031610 tx=0x7f59e80043b0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.863+0000 7f59faaa5640 1 -- 172.21.15.98:0/1453453519 shutdown_connections 2024-05-09T08:49:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.863+0000 7f59faaa5640 1 --2- 172.21.15.98:0/1453453519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f4103430 0x7f59f419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.863+0000 7f59faaa5640 1 --2- 172.21.15.98:0/1453453519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59f4102a70 0x7f59f4198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.863+0000 7f59faaa5640 1 --2- 172.21.15.98:0/1453453519 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f59e40a2b90 0x7f59e40a5050 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.863+0000 7f59faaa5640 1 --2- 172.21.15.98:0/1453453519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59f4101880 0x7f59f4197bc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.863+0000 7f59faaa5640 1 -- 172.21.15.98:0/1453453519 >> 172.21.15.98:0/1453453519 conn(0x7f59f40fce10 msgr2=0x7f59f4105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.863+0000 7f59faaa5640 1 -- 172.21.15.98:0/1453453519 shutdown_connections 2024-05-09T08:49:59.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.863+0000 7f59faaa5640 1 -- 172.21.15.98:0/1453453519 wait complete. 2024-05-09T08:49:59.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:49:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.977+0000 7fc0108b2640 1 -- 172.21.15.98:0/1717044035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c101890 msgr2=0x7fc00c101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.977+0000 7fc0108b2640 1 --2- 172.21.15.98:0/1717044035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c101890 0x7fc00c101cb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009900 tx=0x7fbffc02a410 comp rx=0 tx=0).stop 2024-05-09T08:49:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.977+0000 7fc0108b2640 1 -- 172.21.15.98:0/1717044035 shutdown_connections 2024-05-09T08:49:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.977+0000 7fc0108b2640 1 --2- 172.21.15.98:0/1717044035 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc00c103440 0x7fc00c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.977+0000 7fc0108b2640 1 --2- 172.21.15.98:0/1717044035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc00c102a80 0x7fc00c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.977+0000 7fc0108b2640 1 --2- 172.21.15.98:0/1717044035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c101890 0x7fc00c101cb0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.977+0000 7fc0108b2640 1 -- 172.21.15.98:0/1717044035 >> 172.21.15.98:0/1717044035 conn(0x7fc00c0fce00 msgr2=0x7fc00c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.977+0000 7fc0108b2640 1 -- 172.21.15.98:0/1717044035 shutdown_connections 2024-05-09T08:49:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.977+0000 7fc0108b2640 1 -- 172.21.15.98:0/1717044035 wait complete. 2024-05-09T08:49:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc0108b2640 1 Processor -- start 2024-05-09T08:49:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc0108b2640 1 -- start start 2024-05-09T08:49:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc0108b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c101890 0x7fc00c197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc0108b2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc00c102a80 0x7fc00c198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc0108b2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc00c103440 0x7fc00c19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc0108b2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c10d530 con 0x7fc00c101890 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc0108b2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c10d3b0 con 0x7fc00c103440 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc0108b2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c10d6b0 con 0x7fc00c102a80 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc00a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c101890 0x7fc00c197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc00a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c101890 0x7fc00c197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51982/0 (socket says 172.21.15.98:51982) 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc00a59c640 1 -- 172.21.15.98:0/1526346343 learned_addr learned my addr 172.21.15.98:0/1526346343 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc003fff640 1 --2- 172.21.15.98:0/1526346343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc00c102a80 0x7fc00c198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc00ad9d640 1 --2- 172.21.15.98:0/1526346343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc00c103440 0x7fc00c19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc00a59c640 1 -- 172.21.15.98:0/1526346343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc00c102a80 msgr2=0x7fc00c198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc00a59c640 1 --2- 172.21.15.98:0/1526346343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc00c102a80 0x7fc00c198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc00a59c640 1 -- 172.21.15.98:0/1526346343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc00c103440 msgr2=0x7fc00c19f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc00a59c640 1 --2- 172.21.15.98:0/1526346343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc00c103440 0x7fc00c19f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc00a59c640 1 -- 172.21.15.98:0/1526346343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00c19f810 con 0x7fc00c101890 2024-05-09T08:49:59.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.978+0000 7fc00a59c640 1 --2- 172.21.15.98:0/1526346343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c101890 0x7fc00c197bf0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbffc031610 tx=0x7fbffc0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:59.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.979+0000 7fc0037fe640 1 -- 172.21.15.98:0/1526346343 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc03f030 con 0x7fc00c101890 2024-05-09T08:49:59.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.979+0000 7fc0037fe640 1 -- 172.21.15.98:0/1526346343 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbffc003ce0 con 0x7fc00c101890 2024-05-09T08:49:59.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.979+0000 7fc0037fe640 1 -- 172.21.15.98:0/1526346343 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc035070 con 0x7fc00c101890 2024-05-09T08:49:59.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.979+0000 7fc0108b2640 1 -- 172.21.15.98:0/1526346343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc00c19faa0 con 0x7fc00c101890 2024-05-09T08:49:59.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.979+0000 7fc0108b2640 1 -- 172.21.15.98:0/1526346343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc00c19ff90 con 0x7fc00c101890 2024-05-09T08:49:59.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.980+0000 7fc0108b2640 1 -- 172.21.15.98:0/1526346343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc00c108c80 con 0x7fc00c101890 2024-05-09T08:49:59.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.981+0000 7fc0037fe640 1 -- 172.21.15.98:0/1526346343 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbffc042030 con 0x7fc00c101890 2024-05-09T08:49:59.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.981+0000 7fc0037fe640 1 --2- 172.21.15.98:0/1526346343 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbff80a2b90 0x7fbff80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:59.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.981+0000 7fc003fff640 1 --2- 172.21.15.98:0/1526346343 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbff80a2b90 0x7fbff80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:59.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.981+0000 7fc0037fe640 1 -- 172.21.15.98:0/1526346343 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fbffc0ef9f0 con 0x7fc00c101890 2024-05-09T08:49:59.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.982+0000 7fc003fff640 1 --2- 172.21.15.98:0/1526346343 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbff80a2b90 0x7fbff80a5050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbff4005fd0 tx=0x7fbff4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:59.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:49:59.986+0000 7fc0037fe640 1 -- 172.21.15.98:0/1526346343 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbffc0bc5f0 con 0x7fc00c101890 2024-05-09T08:50:00.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.160+0000 7fc0108b2640 1 -- 172.21.15.98:0/1526346343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc00c062430 con 0x7fc00c101890 2024-05-09T08:50:00.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.161+0000 7fc0037fe640 1 -- 172.21.15.98:0/1526346343 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7fbffc0bbef0 con 0x7fc00c101890 2024-05-09T08:50:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:50:00.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.163+0000 7fc0108b2640 1 -- 172.21.15.98:0/1526346343 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbff80a2b90 msgr2=0x7fbff80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.163+0000 7fc0108b2640 1 --2- 172.21.15.98:0/1526346343 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbff80a2b90 0x7fbff80a5050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbff4005fd0 tx=0x7fbff4005850 comp rx=0 tx=0).stop 2024-05-09T08:50:00.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.163+0000 7fc0108b2640 1 -- 172.21.15.98:0/1526346343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c101890 msgr2=0x7fc00c197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.163+0000 7fc0108b2640 1 --2- 172.21.15.98:0/1526346343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c101890 0x7fc00c197bf0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbffc031610 tx=0x7fbffc0043b0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.163+0000 7fc0108b2640 1 -- 172.21.15.98:0/1526346343 shutdown_connections 2024-05-09T08:50:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.163+0000 7fc0108b2640 1 --2- 172.21.15.98:0/1526346343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc00c103440 0x7fc00c19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.163+0000 7fc0108b2640 1 --2- 172.21.15.98:0/1526346343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc00c102a80 0x7fc00c198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.163+0000 7fc0108b2640 1 --2- 172.21.15.98:0/1526346343 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbff80a2b90 0x7fbff80a5050 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.163+0000 7fc0108b2640 1 --2- 172.21.15.98:0/1526346343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc00c101890 0x7fc00c197bf0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.163+0000 7fc0108b2640 1 -- 172.21.15.98:0/1526346343 >> 172.21.15.98:0/1526346343 conn(0x7fc00c0fce00 msgr2=0x7fc00c105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.163+0000 7fc0108b2640 1 -- 172.21.15.98:0/1526346343 shutdown_connections 2024-05-09T08:50:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.163+0000 7fc0108b2640 1 -- 172.21.15.98:0/1526346343 wait complete. 2024-05-09T08:50:00.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:50:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.280+0000 7f344076a640 1 -- 172.21.15.98:0/2309047868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3438102a70 msgr2=0x7f3438102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.280+0000 7f344076a640 1 --2- 172.21.15.98:0/2309047868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3438102a70 0x7f3438102ef0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3434009900 tx=0x7f343402a410 comp rx=0 tx=0).stop 2024-05-09T08:50:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.280+0000 7f344076a640 1 -- 172.21.15.98:0/2309047868 shutdown_connections 2024-05-09T08:50:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.280+0000 7f344076a640 1 --2- 172.21.15.98:0/2309047868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3438103430 0x7f3438109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.280+0000 7f344076a640 1 --2- 172.21.15.98:0/2309047868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3438102a70 0x7f3438102ef0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.280+0000 7f344076a640 1 --2- 172.21.15.98:0/2309047868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3438101880 0x7f3438101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.280+0000 7f344076a640 1 -- 172.21.15.98:0/2309047868 >> 172.21.15.98:0/2309047868 conn(0x7f34380fce10 msgr2=0x7f34380ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.280+0000 7f344076a640 1 -- 172.21.15.98:0/2309047868 shutdown_connections 2024-05-09T08:50:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.280+0000 7f344076a640 1 -- 172.21.15.98:0/2309047868 wait complete. 2024-05-09T08:50:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.280+0000 7f344076a640 1 Processor -- start 2024-05-09T08:50:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f344076a640 1 -- start start 2024-05-09T08:50:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f344076a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3438101880 0x7f3438197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f344076a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3438102a70 0x7f3438198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f344076a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3438103430 0x7f343819f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f344076a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f343810d500 con 0x7f3438102a70 2024-05-09T08:50:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f344076a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f343810d380 con 0x7f3438101880 2024-05-09T08:50:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f344076a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f343810d680 con 0x7f3438103430 2024-05-09T08:50:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f343dd05640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3438102a70 0x7f3438198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f343dd05640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3438102a70 0x7f3438198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52000/0 (socket says 172.21.15.98:52000) 2024-05-09T08:50:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f343dd05640 1 -- 172.21.15.98:0/1616376387 learned_addr learned my addr 172.21.15.98:0/1616376387 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f343e506640 1 --2- 172.21.15.98:0/1616376387 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3438101880 0x7f3438197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f343dd05640 1 -- 172.21.15.98:0/1616376387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3438103430 msgr2=0x7f343819f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f343dd05640 1 --2- 172.21.15.98:0/1616376387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3438103430 0x7f343819f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f343dd05640 1 -- 172.21.15.98:0/1616376387 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3438101880 msgr2=0x7f3438197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f343dd05640 1 --2- 172.21.15.98:0/1616376387 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3438101880 0x7f3438197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f343dd05640 1 -- 172.21.15.98:0/1616376387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343819f7f0 con 0x7f3438102a70 2024-05-09T08:50:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f343dd05640 1 --2- 172.21.15.98:0/1616376387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3438102a70 0x7f3438198110 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3434009230 tx=0x7f3434004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f342b7fe640 1 -- 172.21.15.98:0/1616376387 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343403f030 con 0x7f3438102a70 2024-05-09T08:50:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f342b7fe640 1 -- 172.21.15.98:0/1616376387 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3434003ce0 con 0x7f3438102a70 2024-05-09T08:50:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f344076a640 1 -- 172.21.15.98:0/1616376387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343819fa80 con 0x7f3438102a70 2024-05-09T08:50:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f342b7fe640 1 -- 172.21.15.98:0/1616376387 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434035070 con 0x7f3438102a70 2024-05-09T08:50:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.281+0000 7f344076a640 1 -- 172.21.15.98:0/1616376387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f343819ff70 con 0x7f3438102a70 2024-05-09T08:50:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.282+0000 7f344076a640 1 -- 172.21.15.98:0/1616376387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3438108c70 con 0x7f3438102a70 2024-05-09T08:50:00.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.284+0000 7f342b7fe640 1 -- 172.21.15.98:0/1616376387 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3434042030 con 0x7f3438102a70 2024-05-09T08:50:00.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.284+0000 7f342b7fe640 1 --2- 172.21.15.98:0/1616376387 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f34240a2b90 0x7f34240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:00.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.284+0000 7f343e506640 1 --2- 172.21.15.98:0/1616376387 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f34240a2b90 0x7f34240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:00.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.284+0000 7f342b7fe640 1 -- 172.21.15.98:0/1616376387 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f34340efbc0 con 0x7f3438102a70 2024-05-09T08:50:00.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.285+0000 7f343e506640 1 --2- 172.21.15.98:0/1616376387 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f34240a2b90 0x7f34240a5050 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f342c0058c0 tx=0x7f342c00a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:00.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.289+0000 7f342b7fe640 1 -- 172.21.15.98:0/1616376387 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f34340bc7c0 con 0x7f3438102a70 2024-05-09T08:50:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.467+0000 7f344076a640 1 -- 172.21.15.98:0/1616376387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f34380624d0 con 0x7f3438102a70 2024-05-09T08:50:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.468+0000 7f342b7fe640 1 -- 172.21.15.98:0/1616376387 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f34340bc0c0 con 0x7f3438102a70 2024-05-09T08:50:00.467 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-09T08:50:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.469+0000 7f344076a640 1 -- 172.21.15.98:0/1616376387 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f34240a2b90 msgr2=0x7f34240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.469+0000 7f344076a640 1 --2- 172.21.15.98:0/1616376387 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f34240a2b90 0x7f34240a5050 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f342c0058c0 tx=0x7f342c00a4c0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.469+0000 7f344076a640 1 -- 172.21.15.98:0/1616376387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3438102a70 msgr2=0x7f3438198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.469+0000 7f344076a640 1 --2- 172.21.15.98:0/1616376387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3438102a70 0x7f3438198110 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3434009230 tx=0x7f3434004430 comp rx=0 tx=0).stop 2024-05-09T08:50:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.470+0000 7f344076a640 1 -- 172.21.15.98:0/1616376387 shutdown_connections 2024-05-09T08:50:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.470+0000 7f344076a640 1 --2- 172.21.15.98:0/1616376387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3438103430 0x7f343819f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.470+0000 7f344076a640 1 --2- 172.21.15.98:0/1616376387 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f34240a2b90 0x7f34240a5050 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.470+0000 7f344076a640 1 --2- 172.21.15.98:0/1616376387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3438102a70 0x7f3438198110 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.470+0000 7f344076a640 1 --2- 172.21.15.98:0/1616376387 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3438101880 0x7f3438197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.470+0000 7f344076a640 1 -- 172.21.15.98:0/1616376387 >> 172.21.15.98:0/1616376387 conn(0x7f34380fce10 msgr2=0x7f3438105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.470+0000 7f344076a640 1 -- 172.21.15.98:0/1616376387 shutdown_connections 2024-05-09T08:50:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.470+0000 7f344076a640 1 -- 172.21.15.98:0/1616376387 wait complete. 2024-05-09T08:50:00.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:00 smithi098 ceph-mon[30092]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-05-09T08:50:00.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:00 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2998317563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:00.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:00 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1453453519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:00.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:00 smithi098 ceph-mon[30092]: overall HEALTH_OK 2024-05-09T08:50:00.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:00 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1526346343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:00.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:50:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.586+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/422133161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7a0100770 msgr2=0x7fb7a0100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.586+0000 7fb7a4dd5640 1 --2- 172.21.15.98:0/422133161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7a0100770 0x7fb7a0100b90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb788009900 tx=0x7fb78802a410 comp rx=0 tx=0).stop 2024-05-09T08:50:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.586+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/422133161 shutdown_connections 2024-05-09T08:50:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.586+0000 7fb7a4dd5640 1 --2- 172.21.15.98:0/422133161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7a0102320 0x7fb7a0108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.586+0000 7fb7a4dd5640 1 --2- 172.21.15.98:0/422133161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7a0101960 0x7fb7a0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.586+0000 7fb7a4dd5640 1 --2- 172.21.15.98:0/422133161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7a0100770 0x7fb7a0100b90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.586+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/422133161 >> 172.21.15.98:0/422133161 conn(0x7fb7a00fbd20 msgr2=0x7fb7a00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.586+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/422133161 shutdown_connections 2024-05-09T08:50:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.586+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/422133161 wait complete. 2024-05-09T08:50:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.587+0000 7fb7a4dd5640 1 Processor -- start 2024-05-09T08:50:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.587+0000 7fb7a4dd5640 1 -- start start 2024-05-09T08:50:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.587+0000 7fb7a4dd5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7a0100770 0x7fb7a0197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.587+0000 7fb7a4dd5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7a0101960 0x7fb7a0198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.587+0000 7fb7a4dd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7a0102320 0x7fb7a019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.587+0000 7fb7a4dd5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a010c3c0 con 0x7fb7a0102320 2024-05-09T08:50:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.587+0000 7fb7a4dd5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a010c240 con 0x7fb7a0100770 2024-05-09T08:50:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.587+0000 7fb7a4dd5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a010c540 con 0x7fb7a0101960 2024-05-09T08:50:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.587+0000 7fb79ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7a0102320 0x7fb7a019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.587+0000 7fb79ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7a0102320 0x7fb7a019f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52018/0 (socket says 172.21.15.98:52018) 2024-05-09T08:50:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.587+0000 7fb79ed9d640 1 -- 172.21.15.98:0/3420900450 learned_addr learned my addr 172.21.15.98:0/3420900450 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.587+0000 7fb79e59c640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7a0100770 0x7fb7a0197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.587+0000 7fb79dd9b640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7a0101960 0x7fb7a0198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.588+0000 7fb79ed9d640 1 -- 172.21.15.98:0/3420900450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7a0101960 msgr2=0x7fb7a0198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.588+0000 7fb79ed9d640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7a0101960 0x7fb7a0198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.588+0000 7fb79ed9d640 1 -- 172.21.15.98:0/3420900450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7a0100770 msgr2=0x7fb7a0197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.588+0000 7fb79ed9d640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7a0100770 0x7fb7a0197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.588+0000 7fb79ed9d640 1 -- 172.21.15.98:0/3420900450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a019f840 con 0x7fb7a0102320 2024-05-09T08:50:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.588+0000 7fb79dd9b640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7a0101960 0x7fb7a0198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.588+0000 7fb79ed9d640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7a0102320 0x7fb7a019f1a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb79800cad0 tx=0x7fb79800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.588+0000 7fb7977fe640 1 -- 172.21.15.98:0/3420900450 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79800ae20 con 0x7fb7a0102320 2024-05-09T08:50:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.588+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/3420900450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7a019fb30 con 0x7fb7a0102320 2024-05-09T08:50:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.588+0000 7fb7977fe640 1 -- 172.21.15.98:0/3420900450 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb798005020 con 0x7fb7a0102320 2024-05-09T08:50:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.588+0000 7fb7977fe640 1 -- 172.21.15.98:0/3420900450 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79800f6f0 con 0x7fb7a0102320 2024-05-09T08:50:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.588+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/3420900450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7a01a0000 con 0x7fb7a0102320 2024-05-09T08:50:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.589+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/3420900450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7a01a0230 con 0x7fb7a0102320 2024-05-09T08:50:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.590+0000 7fb7977fe640 1 -- 172.21.15.98:0/3420900450 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb7980051d0 con 0x7fb7a0102320 2024-05-09T08:50:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.591+0000 7fb7977fe640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb78c0a2b90 0x7fb78c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.591+0000 7fb7977fe640 1 -- 172.21.15.98:0/3420900450 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fb7980cf460 con 0x7fb7a0102320 2024-05-09T08:50:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.591+0000 7fb79e59c640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb78c0a2b90 0x7fb78c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.591+0000 7fb79e59c640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb78c0a2b90 0x7fb78c0a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb788005ab0 tx=0x7fb788000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.595+0000 7fb7977fe640 1 -- 172.21.15.98:0/3420900450 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb798014030 con 0x7fb7a0102320 2024-05-09T08:50:00.638 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:00 smithi105 ceph-mon[34498]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-05-09T08:50:00.638 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:00 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2998317563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:00.638 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:00 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1453453519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:00.638 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:00 smithi105 ceph-mon[34498]: overall HEALTH_OK 2024-05-09T08:50:00.638 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:00 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1526346343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:00 smithi142 ceph-mon[34625]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-05-09T08:50:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:00 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2998317563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:00 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1453453519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:00 smithi142 ceph-mon[34625]: overall HEALTH_OK 2024-05-09T08:50:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:00 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1526346343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.771+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/3420900450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb7a01a0400 con 0x7fb7a0102320 2024-05-09T08:50:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.771+0000 7fb7977fe640 1 -- 172.21.15.98:0/3420900450 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7fb79809c060 con 0x7fb7a0102320 2024-05-09T08:50:00.771 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:50:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.774+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/3420900450 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb78c0a2b90 msgr2=0x7fb78c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.774+0000 7fb7a4dd5640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb78c0a2b90 0x7fb78c0a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb788005ab0 tx=0x7fb788000bc0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.774+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/3420900450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7a0102320 msgr2=0x7fb7a019f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.774+0000 7fb7a4dd5640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7a0102320 0x7fb7a019f1a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb79800cad0 tx=0x7fb79800cf90 comp rx=0 tx=0).stop 2024-05-09T08:50:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.774+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/3420900450 shutdown_connections 2024-05-09T08:50:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.774+0000 7fb7a4dd5640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb78c0a2b90 0x7fb78c0a5050 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.774+0000 7fb7a4dd5640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7a0102320 0x7fb7a019f1a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.774+0000 7fb7a4dd5640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7a0101960 0x7fb7a0198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.774+0000 7fb7a4dd5640 1 --2- 172.21.15.98:0/3420900450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7a0100770 0x7fb7a0197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.774+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/3420900450 >> 172.21.15.98:0/3420900450 conn(0x7fb7a00fbd20 msgr2=0x7fb7a0069f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.774+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/3420900450 shutdown_connections 2024-05-09T08:50:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.774+0000 7fb7a4dd5640 1 -- 172.21.15.98:0/3420900450 wait complete. 2024-05-09T08:50:00.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:50:00.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.886+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1994653326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe654101960 msgr2=0x7fe654101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.886+0000 7fe65a9d5640 1 --2- 172.21.15.98:0/1994653326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe654101960 0x7fe654101de0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009900 tx=0x7fe63c02a410 comp rx=0 tx=0).stop 2024-05-09T08:50:00.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.886+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1994653326 shutdown_connections 2024-05-09T08:50:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.886+0000 7fe65a9d5640 1 --2- 172.21.15.98:0/1994653326 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe654102320 0x7fe654108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.886+0000 7fe65a9d5640 1 --2- 172.21.15.98:0/1994653326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe654101960 0x7fe654101de0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.886+0000 7fe65a9d5640 1 --2- 172.21.15.98:0/1994653326 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe654100770 0x7fe654100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.886+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1994653326 >> 172.21.15.98:0/1994653326 conn(0x7fe6540fbce0 msgr2=0x7fe6540fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.886+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1994653326 shutdown_connections 2024-05-09T08:50:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.886+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1994653326 wait complete. 2024-05-09T08:50:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.886+0000 7fe65a9d5640 1 Processor -- start 2024-05-09T08:50:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.886+0000 7fe65a9d5640 1 -- start start 2024-05-09T08:50:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.886+0000 7fe65a9d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe654100770 0x7fe654069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe65a9d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe654101960 0x7fe65419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe65a9d5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe654102320 0x7fe65419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe65a9d5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe65410c460 con 0x7fe654101960 2024-05-09T08:50:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe65a9d5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe65410c2e0 con 0x7fe654100770 2024-05-09T08:50:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe65a9d5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe65410c5e0 con 0x7fe654102320 2024-05-09T08:50:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe64bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe654101960 0x7fe65419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe64bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe654101960 0x7fe65419c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52042/0 (socket says 172.21.15.98:52042) 2024-05-09T08:50:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe64bfff640 1 -- 172.21.15.98:0/1081411408 learned_addr learned my addr 172.21.15.98:0/1081411408 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe653fff640 1 --2- 172.21.15.98:0/1081411408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe654100770 0x7fe654069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe64bfff640 1 -- 172.21.15.98:0/1081411408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe654102320 msgr2=0x7fe65419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe64bfff640 1 --2- 172.21.15.98:0/1081411408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe654102320 0x7fe65419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe64bfff640 1 -- 172.21.15.98:0/1081411408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe654100770 msgr2=0x7fe654069550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe64bfff640 1 --2- 172.21.15.98:0/1081411408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe654100770 0x7fe654069550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe64bfff640 1 -- 172.21.15.98:0/1081411408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe65419f860 con 0x7fe654101960 2024-05-09T08:50:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe64bfff640 1 --2- 172.21.15.98:0/1081411408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe654101960 0x7fe65419c960 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009230 tx=0x7fe63c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe651ffb640 1 -- 172.21.15.98:0/1081411408 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c03f030 con 0x7fe654101960 2024-05-09T08:50:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe651ffb640 1 -- 172.21.15.98:0/1081411408 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe63c003ce0 con 0x7fe654101960 2024-05-09T08:50:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1081411408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe65419faf0 con 0x7fe654101960 2024-05-09T08:50:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe651ffb640 1 -- 172.21.15.98:0/1081411408 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c035070 con 0x7fe654101960 2024-05-09T08:50:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.887+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1081411408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6541a0060 con 0x7fe654101960 2024-05-09T08:50:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.888+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1081411408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe654107a90 con 0x7fe654101960 2024-05-09T08:50:00.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.889+0000 7fe651ffb640 1 -- 172.21.15.98:0/1081411408 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe63c042030 con 0x7fe654101960 2024-05-09T08:50:00.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.890+0000 7fe651ffb640 1 --2- 172.21.15.98:0/1081411408 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe6400a2b90 0x7fe6400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:00.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.890+0000 7fe653fff640 1 --2- 172.21.15.98:0/1081411408 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe6400a2b90 0x7fe6400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:00.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.890+0000 7fe651ffb640 1 -- 172.21.15.98:0/1081411408 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fe63c0efbd0 con 0x7fe654101960 2024-05-09T08:50:00.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.891+0000 7fe653fff640 1 --2- 172.21.15.98:0/1081411408 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe6400a2b90 0x7fe6400a5050 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe6541017c0 tx=0x7fe644008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:00.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:00.895+0000 7fe651ffb640 1 -- 172.21.15.98:0/1081411408 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe63c0bc7d0 con 0x7fe654101960 2024-05-09T08:50:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.104+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1081411408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe654062430 con 0x7fe654101960 2024-05-09T08:50:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.105+0000 7fe651ffb640 1 -- 172.21.15.98:0/1081411408 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11940 (secure 0 0 0) 0x7fe63c0bc0d0 con 0x7fe654101960 2024-05-09T08:50:01.104 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:01.104 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:48:39.758226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24457,24461,24388,24439,14667]},"id":1}]} 2024-05-09T08:50:01.105 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-05-09T08:50:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.107+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1081411408 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe6400a2b90 msgr2=0x7fe6400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.107+0000 7fe65a9d5640 1 --2- 172.21.15.98:0/1081411408 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe6400a2b90 0x7fe6400a5050 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe6541017c0 tx=0x7fe644008040 comp rx=0 tx=0).stop 2024-05-09T08:50:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.107+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1081411408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe654101960 msgr2=0x7fe65419c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.107+0000 7fe65a9d5640 1 --2- 172.21.15.98:0/1081411408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe654101960 0x7fe65419c960 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009230 tx=0x7fe63c004430 comp rx=0 tx=0).stop 2024-05-09T08:50:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.107+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1081411408 shutdown_connections 2024-05-09T08:50:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.107+0000 7fe65a9d5640 1 --2- 172.21.15.98:0/1081411408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe654102320 0x7fe65419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.107+0000 7fe65a9d5640 1 --2- 172.21.15.98:0/1081411408 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe6400a2b90 0x7fe6400a5050 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.107+0000 7fe65a9d5640 1 --2- 172.21.15.98:0/1081411408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe654101960 0x7fe65419c960 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.107+0000 7fe65a9d5640 1 --2- 172.21.15.98:0/1081411408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe654100770 0x7fe654069550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.107+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1081411408 >> 172.21.15.98:0/1081411408 conn(0x7fe6540fbce0 msgr2=0x7fe654104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.107+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1081411408 shutdown_connections 2024-05-09T08:50:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.107+0000 7fe65a9d5640 1 -- 172.21.15.98:0/1081411408 wait complete. 2024-05-09T08:50:01.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:50:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.220+0000 7f198e8bc640 1 -- 172.21.15.98:0/3653261550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1988102b20 msgr2=0x7f1988102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.220+0000 7f198e8bc640 1 --2- 172.21.15.98:0/3653261550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1988102b20 0x7f1988102fa0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f197c009930 tx=0x7f197c02a420 comp rx=0 tx=0).stop 2024-05-09T08:50:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.220+0000 7f198e8bc640 1 -- 172.21.15.98:0/3653261550 shutdown_connections 2024-05-09T08:50:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.220+0000 7f198e8bc640 1 --2- 172.21.15.98:0/3653261550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19881034e0 0x7f1988109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.220+0000 7f198e8bc640 1 --2- 172.21.15.98:0/3653261550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1988102b20 0x7f1988102fa0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.220+0000 7f198e8bc640 1 --2- 172.21.15.98:0/3653261550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19880ffb50 0x7f19880fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.220+0000 7f198e8bc640 1 -- 172.21.15.98:0/3653261550 >> 172.21.15.98:0/3653261550 conn(0x7f19880fb4a0 msgr2=0x7f19880fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.220+0000 7f198e8bc640 1 -- 172.21.15.98:0/3653261550 shutdown_connections 2024-05-09T08:50:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.220+0000 7f198e8bc640 1 -- 172.21.15.98:0/3653261550 wait complete. 2024-05-09T08:50:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.221+0000 7f198e8bc640 1 Processor -- start 2024-05-09T08:50:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.221+0000 7f198e8bc640 1 -- start start 2024-05-09T08:50:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.221+0000 7f198e8bc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19880ffb50 0x7f1988197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.221+0000 7f198e8bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1988102b20 0x7f1988198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.221+0000 7f198e8bc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19881034e0 0x7f198819f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.221+0000 7f198e8bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f198810d580 con 0x7f1988102b20 2024-05-09T08:50:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.221+0000 7f198e8bc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f198810d400 con 0x7f19881034e0 2024-05-09T08:50:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.221+0000 7f198e8bc640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f198810d700 con 0x7f19880ffb50 2024-05-09T08:50:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.221+0000 7f19877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1988102b20 0x7f1988198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.221+0000 7f19877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1988102b20 0x7f1988198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52062/0 (socket says 172.21.15.98:52062) 2024-05-09T08:50:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.221+0000 7f19877fe640 1 -- 172.21.15.98:0/989541866 learned_addr learned my addr 172.21.15.98:0/989541866 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f198ce59640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19881034e0 0x7f198819f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f19877fe640 1 -- 172.21.15.98:0/989541866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19880ffb50 msgr2=0x7f1988197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f1987fff640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19880ffb50 0x7f1988197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f19877fe640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19880ffb50 0x7f1988197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f19877fe640 1 -- 172.21.15.98:0/989541866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19881034e0 msgr2=0x7f198819f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f19877fe640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19881034e0 0x7f198819f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f19877fe640 1 -- 172.21.15.98:0/989541866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198819f810 con 0x7f1988102b20 2024-05-09T08:50:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f198ce59640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19881034e0 0x7f198819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f1987fff640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19880ffb50 0x7f1988197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f19877fe640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1988102b20 0x7f1988198130 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f197c031440 tx=0x7f197c032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f19857fa640 1 -- 172.21.15.98:0/989541866 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c03f030 con 0x7f1988102b20 2024-05-09T08:50:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f19857fa640 1 -- 172.21.15.98:0/989541866 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f197c003ce0 con 0x7f1988102b20 2024-05-09T08:50:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f198e8bc640 1 -- 172.21.15.98:0/989541866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f198819faa0 con 0x7f1988102b20 2024-05-09T08:50:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f19857fa640 1 -- 172.21.15.98:0/989541866 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c035070 con 0x7f1988102b20 2024-05-09T08:50:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.222+0000 7f198e8bc640 1 -- 172.21.15.98:0/989541866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f198819ff90 con 0x7f1988102b20 2024-05-09T08:50:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.223+0000 7f198e8bc640 1 -- 172.21.15.98:0/989541866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1988108c50 con 0x7f1988102b20 2024-05-09T08:50:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.224+0000 7f19857fa640 1 -- 172.21.15.98:0/989541866 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f197c042030 con 0x7f1988102b20 2024-05-09T08:50:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.225+0000 7f19857fa640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f19700a2b90 0x7f19700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.225+0000 7f1987fff640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f19700a2b90 0x7f19700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.225+0000 7f19857fa640 1 -- 172.21.15.98:0/989541866 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f197c0efa90 con 0x7f1988102b20 2024-05-09T08:50:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.226+0000 7f1987fff640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f19700a2b90 0x7f19700a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f19780058c0 tx=0x7f1978005d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.230+0000 7f19857fa640 1 -- 172.21.15.98:0/989541866 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f197c0bc690 con 0x7f1988102b20 2024-05-09T08:50:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.407+0000 7f198e8bc640 1 -- 172.21.15.98:0/989541866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1988062430 con 0x7f1988102b20 2024-05-09T08:50:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.409+0000 7f19857fa640 1 -- 172.21.15.98:0/989541866 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24622 (secure 0 0 0) 0x7f197c0bbf90 con 0x7f1988102b20 2024-05-09T08:50:01.408 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:01.409 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":92,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:49:18.355914+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:50:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.411+0000 7f198e8bc640 1 -- 172.21.15.98:0/989541866 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f19700a2b90 msgr2=0x7f19700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.411+0000 7f198e8bc640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f19700a2b90 0x7f19700a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f19780058c0 tx=0x7f1978005d50 comp rx=0 tx=0).stop 2024-05-09T08:50:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.411+0000 7f198e8bc640 1 -- 172.21.15.98:0/989541866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1988102b20 msgr2=0x7f1988198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.411+0000 7f198e8bc640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1988102b20 0x7f1988198130 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f197c031440 tx=0x7f197c032e20 comp rx=0 tx=0).stop 2024-05-09T08:50:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.411+0000 7f198e8bc640 1 -- 172.21.15.98:0/989541866 shutdown_connections 2024-05-09T08:50:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.411+0000 7f198e8bc640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19881034e0 0x7f198819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.411+0000 7f198e8bc640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f19700a2b90 0x7f19700a5050 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.411+0000 7f198e8bc640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1988102b20 0x7f1988198130 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.411+0000 7f198e8bc640 1 --2- 172.21.15.98:0/989541866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19880ffb50 0x7f1988197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.411+0000 7f198e8bc640 1 -- 172.21.15.98:0/989541866 >> 172.21.15.98:0/989541866 conn(0x7f19880fb4a0 msgr2=0x7f1988105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.411+0000 7f198e8bc640 1 -- 172.21.15.98:0/989541866 shutdown_connections 2024-05-09T08:50:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.411+0000 7f198e8bc640 1 -- 172.21.15.98:0/989541866 wait complete. 2024-05-09T08:50:01.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:01 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1616376387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:01.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:01 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3420900450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:01.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:01 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1081411408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:01.422 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:43:39.424443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:48:15.866450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:48:17.122162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:48:20.591475+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:50:01.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:50:01.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.528+0000 7f889cd70640 1 -- 172.21.15.98:0/3139629455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898102320 msgr2=0x7f8898108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.528+0000 7f889cd70640 1 --2- 172.21.15.98:0/3139629455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898102320 0x7f8898108bb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8890009ca0 tx=0x7f889002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.528+0000 7f889cd70640 1 -- 172.21.15.98:0/3139629455 shutdown_connections 2024-05-09T08:50:01.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.528+0000 7f889cd70640 1 --2- 172.21.15.98:0/3139629455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898102320 0x7f8898108bb0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.528+0000 7f889cd70640 1 --2- 172.21.15.98:0/3139629455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8898101960 0x7f8898101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.528+0000 7f889cd70640 1 --2- 172.21.15.98:0/3139629455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8898100770 0x7f8898100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.528+0000 7f889cd70640 1 -- 172.21.15.98:0/3139629455 >> 172.21.15.98:0/3139629455 conn(0x7f88980fbce0 msgr2=0x7f88980fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:01.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.528+0000 7f889cd70640 1 -- 172.21.15.98:0/3139629455 shutdown_connections 2024-05-09T08:50:01.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.528+0000 7f889cd70640 1 -- 172.21.15.98:0/3139629455 wait complete. 2024-05-09T08:50:01.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.528+0000 7f889cd70640 1 Processor -- start 2024-05-09T08:50:01.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.528+0000 7f889cd70640 1 -- start start 2024-05-09T08:50:01.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889cd70640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898100770 0x7f8898197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:01.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889cd70640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8898101960 0x7f8898198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:01.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889cd70640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8898102320 0x7f889819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:01.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889cd70640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f889810c3f0 con 0x7f8898100770 2024-05-09T08:50:01.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889cd70640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f889810c270 con 0x7f8898101960 2024-05-09T08:50:01.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889cd70640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f889810c570 con 0x7f8898102320 2024-05-09T08:50:01.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898100770 0x7f8898197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:01.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898100770 0x7f8898197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52088/0 (socket says 172.21.15.98:52088) 2024-05-09T08:50:01.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889659c640 1 -- 172.21.15.98:0/3402068820 learned_addr learned my addr 172.21.15.98:0/3402068820 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:01.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f8895d9b640 1 --2- 172.21.15.98:0/3402068820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8898101960 0x7f8898198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:01.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889659c640 1 -- 172.21.15.98:0/3402068820 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8898102320 msgr2=0x7f889819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889659c640 1 --2- 172.21.15.98:0/3402068820 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8898102320 0x7f889819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889659c640 1 -- 172.21.15.98:0/3402068820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8898101960 msgr2=0x7f8898198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889659c640 1 --2- 172.21.15.98:0/3402068820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8898101960 0x7f8898198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889659c640 1 -- 172.21.15.98:0/3402068820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889819f770 con 0x7f8898100770 2024-05-09T08:50:01.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889659c640 1 --2- 172.21.15.98:0/3402068820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898100770 0x7f8898197b50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f888800d950 tx=0x7f888800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:01.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f888f7fe640 1 -- 172.21.15.98:0/3402068820 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8888017070 con 0x7f8898100770 2024-05-09T08:50:01.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f888f7fe640 1 -- 172.21.15.98:0/3402068820 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8888009c30 con 0x7f8898100770 2024-05-09T08:50:01.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889cd70640 1 -- 172.21.15.98:0/3402068820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f889819fa60 con 0x7f8898100770 2024-05-09T08:50:01.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f888f7fe640 1 -- 172.21.15.98:0/3402068820 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888800f600 con 0x7f8898100770 2024-05-09T08:50:01.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.529+0000 7f889cd70640 1 -- 172.21.15.98:0/3402068820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f889819ffb0 con 0x7f8898100770 2024-05-09T08:50:01.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.530+0000 7f889cd70640 1 -- 172.21.15.98:0/3402068820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88981a01e0 con 0x7f8898100770 2024-05-09T08:50:01.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.531+0000 7f888f7fe640 1 -- 172.21.15.98:0/3402068820 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f888801a070 con 0x7f8898100770 2024-05-09T08:50:01.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.532+0000 7f888f7fe640 1 --2- 172.21.15.98:0/3402068820 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f88840a2c10 0x7f88840a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:01.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.533+0000 7f8895d9b640 1 --2- 172.21.15.98:0/3402068820 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f88840a2c10 0x7f88840a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:01.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.533+0000 7f888f7fe640 1 -- 172.21.15.98:0/3402068820 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f88880cee30 con 0x7f8898100770 2024-05-09T08:50:01.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.533+0000 7f8895d9b640 1 --2- 172.21.15.98:0/3402068820 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f88840a2c10 0x7f88840a50d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8880005fd0 tx=0x7f8880005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:01.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.537+0000 7f888f7fe640 1 -- 172.21.15.98:0/3402068820 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8888009de0 con 0x7f8898100770 2024-05-09T08:50:01.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:01 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1616376387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:01.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:01 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3420900450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:01.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:01 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1081411408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:01.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.715+0000 7f889cd70640 1 -- 172.21.15.98:0/3402068820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f88981a03b0 con 0x7f8898100770 2024-05-09T08:50:01.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.715+0000 7f888f7fe640 1 -- 172.21.15.98:0/3402068820 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f888809ba30 con 0x7f8898100770 2024-05-09T08:50:01.715 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-09T08:50:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.717+0000 7f889cd70640 1 -- 172.21.15.98:0/3402068820 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f88840a2c10 msgr2=0x7f88840a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.717+0000 7f889cd70640 1 --2- 172.21.15.98:0/3402068820 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f88840a2c10 0x7f88840a50d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8880005fd0 tx=0x7f8880005850 comp rx=0 tx=0).stop 2024-05-09T08:50:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.717+0000 7f889cd70640 1 -- 172.21.15.98:0/3402068820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898100770 msgr2=0x7f8898197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.717+0000 7f889cd70640 1 --2- 172.21.15.98:0/3402068820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898100770 0x7f8898197b50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f888800d950 tx=0x7f888800de10 comp rx=0 tx=0).stop 2024-05-09T08:50:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.717+0000 7f889cd70640 1 -- 172.21.15.98:0/3402068820 shutdown_connections 2024-05-09T08:50:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.717+0000 7f889cd70640 1 --2- 172.21.15.98:0/3402068820 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8898102320 0x7f889819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.717+0000 7f889cd70640 1 --2- 172.21.15.98:0/3402068820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8898101960 0x7f8898198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.717+0000 7f889cd70640 1 --2- 172.21.15.98:0/3402068820 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f88840a2c10 0x7f88840a50d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.717+0000 7f889cd70640 1 --2- 172.21.15.98:0/3402068820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898100770 0x7f8898197b50 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.717+0000 7f889cd70640 1 -- 172.21.15.98:0/3402068820 >> 172.21.15.98:0/3402068820 conn(0x7f88980fbce0 msgr2=0x7f88980fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.717+0000 7f889cd70640 1 -- 172.21.15.98:0/3402068820 shutdown_connections 2024-05-09T08:50:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.717+0000 7f889cd70640 1 -- 172.21.15.98:0/3402068820 wait complete. 2024-05-09T08:50:01.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:50:01.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:01 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1616376387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:01.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:01 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3420900450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:01.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:01 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1081411408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:01.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.833+0000 7f692e76a640 1 -- 172.21.15.98:0/2312706977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69281034d0 msgr2=0x7f6928109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.833+0000 7f692e76a640 1 --2- 172.21.15.98:0/2312706977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69281034d0 0x7f6928109d60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6920009cd0 tx=0x7f692002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.833+0000 7f692e76a640 1 -- 172.21.15.98:0/2312706977 shutdown_connections 2024-05-09T08:50:01.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.833+0000 7f692e76a640 1 --2- 172.21.15.98:0/2312706977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69281034d0 0x7f6928109d60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.833+0000 7f692e76a640 1 --2- 172.21.15.98:0/2312706977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6928102b10 0x7f6928102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.833+0000 7f692e76a640 1 --2- 172.21.15.98:0/2312706977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69280ffb40 0x7f69280fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.833+0000 7f692e76a640 1 -- 172.21.15.98:0/2312706977 >> 172.21.15.98:0/2312706977 conn(0x7f69280fb470 msgr2=0x7f69280fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:01.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.833+0000 7f692e76a640 1 -- 172.21.15.98:0/2312706977 shutdown_connections 2024-05-09T08:50:01.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.833+0000 7f692e76a640 1 -- 172.21.15.98:0/2312706977 wait complete. 2024-05-09T08:50:01.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f692e76a640 1 Processor -- start 2024-05-09T08:50:01.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f692e76a640 1 -- start start 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f692e76a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f69280ffb40 0x7f6928197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f692e76a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6928102b10 0x7f6928198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f692e76a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69281034d0 0x7f692819f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f692e76a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f692810d610 con 0x7f6928102b10 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f692e76a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f692810d490 con 0x7f69281034d0 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f692e76a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f692810d790 con 0x7f69280ffb40 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f69277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6928102b10 0x7f6928198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f69277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6928102b10 0x7f6928198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52108/0 (socket says 172.21.15.98:52108) 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f69277fe640 1 -- 172.21.15.98:0/1976354526 learned_addr learned my addr 172.21.15.98:0/1976354526 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f692cd07640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69281034d0 0x7f692819f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f6927fff640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f69280ffb40 0x7f6928197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f69277fe640 1 -- 172.21.15.98:0/1976354526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f69280ffb40 msgr2=0x7f6928197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f69277fe640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f69280ffb40 0x7f6928197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f69277fe640 1 -- 172.21.15.98:0/1976354526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69281034d0 msgr2=0x7f692819f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f69277fe640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69281034d0 0x7f692819f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:01.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f69277fe640 1 -- 172.21.15.98:0/1976354526 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692819f930 con 0x7f6928102b10 2024-05-09T08:50:01.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.834+0000 7f692cd07640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69281034d0 0x7f692819f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:01.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.835+0000 7f69277fe640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6928102b10 0x7f6928198260 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f691c00b4a0 tx=0x7f691c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:01.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.835+0000 7f69257fa640 1 -- 172.21.15.98:0/1976354526 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691c014070 con 0x7f6928102b10 2024-05-09T08:50:01.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.835+0000 7f69257fa640 1 -- 172.21.15.98:0/1976354526 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f691c010af0 con 0x7f6928102b10 2024-05-09T08:50:01.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.835+0000 7f69257fa640 1 -- 172.21.15.98:0/1976354526 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691c017d10 con 0x7f6928102b10 2024-05-09T08:50:01.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.835+0000 7f692e76a640 1 -- 172.21.15.98:0/1976354526 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f692819fc20 con 0x7f6928102b10 2024-05-09T08:50:01.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.835+0000 7f692e76a640 1 -- 172.21.15.98:0/1976354526 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69281a0120 con 0x7f6928102b10 2024-05-09T08:50:01.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.836+0000 7f692e76a640 1 -- 172.21.15.98:0/1976354526 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6928108d30 con 0x7f6928102b10 2024-05-09T08:50:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.837+0000 7f69257fa640 1 -- 172.21.15.98:0/1976354526 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f691c017070 con 0x7f6928102b10 2024-05-09T08:50:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.838+0000 7f69257fa640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f69100a2b90 0x7f69100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.838+0000 7f6927fff640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f69100a2b90 0x7f69100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.838+0000 7f69257fa640 1 -- 172.21.15.98:0/1976354526 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f691c0ce8b0 con 0x7f6928102b10 2024-05-09T08:50:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.838+0000 7f6927fff640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f69100a2b90 0x7f69100a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f69281991f0 tx=0x7f6918008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:01.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:01.843+0000 7f69257fa640 1 -- 172.21.15.98:0/1976354526 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f691c09b4b0 con 0x7f6928102b10 2024-05-09T08:50:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.019+0000 7f692e76a640 1 -- 172.21.15.98:0/1976354526 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f69280624d0 con 0x7f6928102b10 2024-05-09T08:50:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.020+0000 7f69257fa640 1 -- 172.21.15.98:0/1976354526 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f691c09adb0 con 0x7f6928102b10 2024-05-09T08:50:02.019 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:50:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.022+0000 7f692e76a640 1 -- 172.21.15.98:0/1976354526 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f69100a2b90 msgr2=0x7f69100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.022+0000 7f692e76a640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f69100a2b90 0x7f69100a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f69281991f0 tx=0x7f6918008040 comp rx=0 tx=0).stop 2024-05-09T08:50:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.022+0000 7f692e76a640 1 -- 172.21.15.98:0/1976354526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6928102b10 msgr2=0x7f6928198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.022+0000 7f692e76a640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6928102b10 0x7f6928198260 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f691c00b4a0 tx=0x7f691c00b960 comp rx=0 tx=0).stop 2024-05-09T08:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.022+0000 7f692e76a640 1 -- 172.21.15.98:0/1976354526 shutdown_connections 2024-05-09T08:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.022+0000 7f692e76a640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69281034d0 0x7f692819f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.022+0000 7f692e76a640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f69100a2b90 0x7f69100a5050 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.022+0000 7f692e76a640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6928102b10 0x7f6928198260 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.022+0000 7f692e76a640 1 --2- 172.21.15.98:0/1976354526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f69280ffb40 0x7f6928197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.022+0000 7f692e76a640 1 -- 172.21.15.98:0/1976354526 >> 172.21.15.98:0/1976354526 conn(0x7f69280fb470 msgr2=0x7f6928105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.022+0000 7f692e76a640 1 -- 172.21.15.98:0/1976354526 shutdown_connections 2024-05-09T08:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.022+0000 7f692e76a640 1 -- 172.21.15.98:0/1976354526 wait complete. 2024-05-09T08:50:02.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:50:02.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.138+0000 7f1772276640 1 -- 172.21.15.98:0/1500821820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f176c100a00 msgr2=0x7f176c100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.138+0000 7f1772276640 1 --2- 172.21.15.98:0/1500821820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f176c100a00 0x7f176c100e80 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1754009930 tx=0x7f175402a420 comp rx=0 tx=0).stop 2024-05-09T08:50:02.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.138+0000 7f1772276640 1 -- 172.21.15.98:0/1500821820 shutdown_connections 2024-05-09T08:50:02.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.138+0000 7f1772276640 1 --2- 172.21.15.98:0/1500821820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f176c1013c0 0x7f176c108c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.138+0000 7f1772276640 1 --2- 172.21.15.98:0/1500821820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f176c100a00 0x7f176c100e80 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.138+0000 7f1772276640 1 --2- 172.21.15.98:0/1500821820 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f176c101d60 0x7f176c102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.138+0000 7f1772276640 1 -- 172.21.15.98:0/1500821820 >> 172.21.15.98:0/1500821820 conn(0x7f176c0fb470 msgr2=0x7f176c0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:02.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.138+0000 7f1772276640 1 -- 172.21.15.98:0/1500821820 shutdown_connections 2024-05-09T08:50:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.138+0000 7f1772276640 1 -- 172.21.15.98:0/1500821820 wait complete. 2024-05-09T08:50:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.139+0000 7f1772276640 1 Processor -- start 2024-05-09T08:50:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.139+0000 7f1772276640 1 -- start start 2024-05-09T08:50:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.139+0000 7f1772276640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f176c100a00 0x7f176c1004c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.139+0000 7f1772276640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f176c1013c0 0x7f176c19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.139+0000 7f1772276640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f176c101d60 0x7f176c19eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.139+0000 7f1772276640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f176c10c400 con 0x7f176c101d60 2024-05-09T08:50:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.139+0000 7f1772276640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f176c10c280 con 0x7f176c100a00 2024-05-09T08:50:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.139+0000 7f1772276640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f176c10c580 con 0x7f176c1013c0 2024-05-09T08:50:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.139+0000 7f1770813640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f176c101d60 0x7f176c19eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.139+0000 7f1770813640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f176c101d60 0x7f176c19eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52128/0 (socket says 172.21.15.98:52128) 2024-05-09T08:50:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.139+0000 7f1770813640 1 -- 172.21.15.98:0/3417978351 learned_addr learned my addr 172.21.15.98:0/3417978351 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.139+0000 7f176bfff640 1 --2- 172.21.15.98:0/3417978351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f176c100a00 0x7f176c1004c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.139+0000 7f176b7fe640 1 --2- 172.21.15.98:0/3417978351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f176c1013c0 0x7f176c19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.140+0000 7f1770813640 1 -- 172.21.15.98:0/3417978351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f176c1013c0 msgr2=0x7f176c19c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.140+0000 7f1770813640 1 --2- 172.21.15.98:0/3417978351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f176c1013c0 0x7f176c19c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.140+0000 7f1770813640 1 -- 172.21.15.98:0/3417978351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f176c100a00 msgr2=0x7f176c1004c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.140+0000 7f1770813640 1 --2- 172.21.15.98:0/3417978351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f176c100a00 0x7f176c1004c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.140+0000 7f1770813640 1 -- 172.21.15.98:0/3417978351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f176c19f790 con 0x7f176c101d60 2024-05-09T08:50:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.140+0000 7f1770813640 1 --2- 172.21.15.98:0/3417978351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f176c101d60 0x7f176c19eed0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f176400cce0 tx=0x7f176400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.140+0000 7f17697fa640 1 -- 172.21.15.98:0/3417978351 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176400f450 con 0x7f176c101d60 2024-05-09T08:50:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.140+0000 7f1772276640 1 -- 172.21.15.98:0/3417978351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f176c19fa80 con 0x7f176c101d60 2024-05-09T08:50:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.140+0000 7f17697fa640 1 -- 172.21.15.98:0/3417978351 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f176400fad0 con 0x7f176c101d60 2024-05-09T08:50:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.140+0000 7f17697fa640 1 -- 172.21.15.98:0/3417978351 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1764018c10 con 0x7f176c101d60 2024-05-09T08:50:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.140+0000 7f1772276640 1 -- 172.21.15.98:0/3417978351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f176c19ff50 con 0x7f176c101d60 2024-05-09T08:50:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.141+0000 7f1772276640 1 -- 172.21.15.98:0/3417978351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f176c107b40 con 0x7f176c101d60 2024-05-09T08:50:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.142+0000 7f17697fa640 1 -- 172.21.15.98:0/3417978351 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f176400f5f0 con 0x7f176c101d60 2024-05-09T08:50:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.143+0000 7f17697fa640 1 --2- 172.21.15.98:0/3417978351 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f17580a2b90 0x7f17580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.143+0000 7f176bfff640 1 --2- 172.21.15.98:0/3417978351 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f17580a2b90 0x7f17580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.143+0000 7f17697fa640 1 -- 172.21.15.98:0/3417978351 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f17640cfa90 con 0x7f176c101d60 2024-05-09T08:50:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.143+0000 7f176bfff640 1 --2- 172.21.15.98:0/3417978351 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f17580a2b90 0x7f17580a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f176c069c10 tx=0x7f175c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:02.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.147+0000 7f17697fa640 1 -- 172.21.15.98:0/3417978351 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1764014030 con 0x7f176c101d60 2024-05-09T08:50:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.323+0000 7f1772276640 1 -- 172.21.15.98:0/3417978351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f176c062430 con 0x7f176c101d60 2024-05-09T08:50:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.323+0000 7f17697fa640 1 -- 172.21.15.98:0/3417978351 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f176409c690 con 0x7f176c101d60 2024-05-09T08:50:02.322 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-09T08:50:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.325+0000 7f1772276640 1 -- 172.21.15.98:0/3417978351 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f17580a2b90 msgr2=0x7f17580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.325+0000 7f1772276640 1 --2- 172.21.15.98:0/3417978351 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f17580a2b90 0x7f17580a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f176c069c10 tx=0x7f175c00b390 comp rx=0 tx=0).stop 2024-05-09T08:50:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.325+0000 7f1772276640 1 -- 172.21.15.98:0/3417978351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f176c101d60 msgr2=0x7f176c19eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.325+0000 7f1772276640 1 --2- 172.21.15.98:0/3417978351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f176c101d60 0x7f176c19eed0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f176400cce0 tx=0x7f176400a620 comp rx=0 tx=0).stop 2024-05-09T08:50:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.325+0000 7f1772276640 1 -- 172.21.15.98:0/3417978351 shutdown_connections 2024-05-09T08:50:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.325+0000 7f1772276640 1 --2- 172.21.15.98:0/3417978351 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f17580a2b90 0x7f17580a5050 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.325+0000 7f1772276640 1 --2- 172.21.15.98:0/3417978351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f176c101d60 0x7f176c19eed0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.325+0000 7f1772276640 1 --2- 172.21.15.98:0/3417978351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f176c1013c0 0x7f176c19c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.325+0000 7f1772276640 1 --2- 172.21.15.98:0/3417978351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f176c100a00 0x7f176c1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.325+0000 7f1772276640 1 -- 172.21.15.98:0/3417978351 >> 172.21.15.98:0/3417978351 conn(0x7f176c0fb470 msgr2=0x7f176c104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.325+0000 7f1772276640 1 -- 172.21.15.98:0/3417978351 shutdown_connections 2024-05-09T08:50:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.325+0000 7f1772276640 1 -- 172.21.15.98:0/3417978351 wait complete. 2024-05-09T08:50:02.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:50:02.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.443+0000 7f76fe99e640 1 -- 172.21.15.98:0/2747038599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f8102a70 msgr2=0x7f76f8102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.443+0000 7f76fe99e640 1 --2- 172.21.15.98:0/2747038599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f8102a70 0x7f76f8102ef0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009900 tx=0x7f76ec02a410 comp rx=0 tx=0).stop 2024-05-09T08:50:02.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.443+0000 7f76fe99e640 1 -- 172.21.15.98:0/2747038599 shutdown_connections 2024-05-09T08:50:02.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.443+0000 7f76fe99e640 1 --2- 172.21.15.98:0/2747038599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f8103430 0x7f76f8109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.443+0000 7f76fe99e640 1 --2- 172.21.15.98:0/2747038599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f8102a70 0x7f76f8102ef0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.443+0000 7f76fe99e640 1 --2- 172.21.15.98:0/2747038599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76f8101880 0x7f76f8101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.443+0000 7f76fe99e640 1 -- 172.21.15.98:0/2747038599 >> 172.21.15.98:0/2747038599 conn(0x7f76f80fce10 msgr2=0x7f76f80ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:02.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.443+0000 7f76fe99e640 1 -- 172.21.15.98:0/2747038599 shutdown_connections 2024-05-09T08:50:02.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.443+0000 7f76fe99e640 1 -- 172.21.15.98:0/2747038599 wait complete. 2024-05-09T08:50:02.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.443+0000 7f76fe99e640 1 Processor -- start 2024-05-09T08:50:02.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.443+0000 7f76fe99e640 1 -- start start 2024-05-09T08:50:02.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76fe99e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f8101880 0x7f76f8197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:02.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76fe99e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f8102a70 0x7f76f81980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:02.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76fe99e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76f8103430 0x7f76f819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:02.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76fe99e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f76f810d500 con 0x7f76f8101880 2024-05-09T08:50:02.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76fe99e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f76f810d380 con 0x7f76f8102a70 2024-05-09T08:50:02.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76fe99e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f76f810d680 con 0x7f76f8103430 2024-05-09T08:50:02.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76f7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f8101880 0x7f76f8197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:02.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76f7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f8101880 0x7f76f8197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52146/0 (socket says 172.21.15.98:52146) 2024-05-09T08:50:02.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76f7fff640 1 -- 172.21.15.98:0/3268997158 learned_addr learned my addr 172.21.15.98:0/3268997158 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:02.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76f77fe640 1 --2- 172.21.15.98:0/3268997158 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f8102a70 0x7f76f81980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:02.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76fcf3b640 1 --2- 172.21.15.98:0/3268997158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76f8103430 0x7f76f819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:02.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76f7fff640 1 -- 172.21.15.98:0/3268997158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76f8103430 msgr2=0x7f76f819f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76f7fff640 1 --2- 172.21.15.98:0/3268997158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76f8103430 0x7f76f819f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76f7fff640 1 -- 172.21.15.98:0/3268997158 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f8102a70 msgr2=0x7f76f81980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76f7fff640 1 --2- 172.21.15.98:0/3268997158 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f8102a70 0x7f76f81980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76f7fff640 1 -- 172.21.15.98:0/3268997158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f819f7b0 con 0x7f76f8101880 2024-05-09T08:50:02.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76f7fff640 1 --2- 172.21.15.98:0/3268997158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f8101880 0x7f76f8197b90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f76e800d950 tx=0x7f76e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:02.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76f57fa640 1 -- 172.21.15.98:0/3268997158 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e8017070 con 0x7f76f8101880 2024-05-09T08:50:02.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76f57fa640 1 -- 172.21.15.98:0/3268997158 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76e8009c30 con 0x7f76f8101880 2024-05-09T08:50:02.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76fe99e640 1 -- 172.21.15.98:0/3268997158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f819faa0 con 0x7f76f8101880 2024-05-09T08:50:02.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.444+0000 7f76f57fa640 1 -- 172.21.15.98:0/3268997158 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e800f600 con 0x7f76f8101880 2024-05-09T08:50:02.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.445+0000 7f76fe99e640 1 -- 172.21.15.98:0/3268997158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f819ff70 con 0x7f76f8101880 2024-05-09T08:50:02.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.445+0000 7f76fe99e640 1 -- 172.21.15.98:0/3268997158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76f8108c70 con 0x7f76f8101880 2024-05-09T08:50:02.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.447+0000 7f76f57fa640 1 -- 172.21.15.98:0/3268997158 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f76e801a070 con 0x7f76f8101880 2024-05-09T08:50:02.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.447+0000 7f76f57fa640 1 --2- 172.21.15.98:0/3268997158 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f76e00a2b90 0x7f76e00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:02.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.447+0000 7f76f77fe640 1 --2- 172.21.15.98:0/3268997158 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f76e00a2b90 0x7f76e00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:02.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.447+0000 7f76f57fa640 1 -- 172.21.15.98:0/3268997158 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f76e80ceee0 con 0x7f76f8101880 2024-05-09T08:50:02.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.448+0000 7f76f77fe640 1 --2- 172.21.15.98:0/3268997158 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f76e00a2b90 0x7f76e00a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f76ec000c00 tx=0x7f76ec0315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:02.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.452+0000 7f76f57fa640 1 -- 172.21.15.98:0/3268997158 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f76e8009de0 con 0x7f76f8101880 2024-05-09T08:50:02.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:02 smithi098 ceph-mon[30092]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-09T08:50:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:02 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/989541866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:02 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3402068820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:02 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1976354526' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.629+0000 7f76fe99e640 1 -- 172.21.15.98:0/3268997158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f76f8062430 con 0x7f76f8101880 2024-05-09T08:50:02.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.630+0000 7f76f57fa640 1 -- 172.21.15.98:0/3268997158 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f76e809bae0 con 0x7f76f8101880 2024-05-09T08:50:02.629 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:50:02.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.632+0000 7f76fe99e640 1 -- 172.21.15.98:0/3268997158 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f76e00a2b90 msgr2=0x7f76e00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.632+0000 7f76fe99e640 1 --2- 172.21.15.98:0/3268997158 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f76e00a2b90 0x7f76e00a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f76ec000c00 tx=0x7f76ec0315d0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.632+0000 7f76fe99e640 1 -- 172.21.15.98:0/3268997158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f8101880 msgr2=0x7f76f8197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.632+0000 7f76fe99e640 1 --2- 172.21.15.98:0/3268997158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f8101880 0x7f76f8197b90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f76e800d950 tx=0x7f76e800de10 comp rx=0 tx=0).stop 2024-05-09T08:50:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.632+0000 7f76fe99e640 1 -- 172.21.15.98:0/3268997158 shutdown_connections 2024-05-09T08:50:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.632+0000 7f76fe99e640 1 --2- 172.21.15.98:0/3268997158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76f8103430 0x7f76f819f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.632+0000 7f76fe99e640 1 --2- 172.21.15.98:0/3268997158 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f8102a70 0x7f76f81980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.632+0000 7f76fe99e640 1 --2- 172.21.15.98:0/3268997158 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f76e00a2b90 0x7f76e00a5050 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.632+0000 7f76fe99e640 1 --2- 172.21.15.98:0/3268997158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f8101880 0x7f76f8197b90 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.632+0000 7f76fe99e640 1 -- 172.21.15.98:0/3268997158 >> 172.21.15.98:0/3268997158 conn(0x7f76f80fce10 msgr2=0x7f76f8105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.632+0000 7f76fe99e640 1 -- 172.21.15.98:0/3268997158 shutdown_connections 2024-05-09T08:50:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.632+0000 7f76fe99e640 1 -- 172.21.15.98:0/3268997158 wait complete. 2024-05-09T08:50:02.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:50:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:02 smithi142 ceph-mon[34625]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-09T08:50:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:02 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/989541866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:02 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3402068820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:02 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1976354526' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:02.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:02 smithi105 ceph-mon[34498]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-09T08:50:02.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:02 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/989541866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:02.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:02 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3402068820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:02.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:02 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1976354526' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:02.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.744+0000 7f376ee95640 1 -- 172.21.15.98:0/2590630073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3768101890 msgr2=0x7f3768101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.744+0000 7f376ee95640 1 --2- 172.21.15.98:0/2590630073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3768101890 0x7f3768101cb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f375c009900 tx=0x7f375c02a410 comp rx=0 tx=0).stop 2024-05-09T08:50:02.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.744+0000 7f376ee95640 1 -- 172.21.15.98:0/2590630073 shutdown_connections 2024-05-09T08:50:02.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.744+0000 7f376ee95640 1 --2- 172.21.15.98:0/2590630073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3768103440 0x7f3768109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.744+0000 7f376ee95640 1 --2- 172.21.15.98:0/2590630073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3768102a80 0x7f3768102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.744+0000 7f376ee95640 1 --2- 172.21.15.98:0/2590630073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3768101890 0x7f3768101cb0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.744+0000 7f376ee95640 1 -- 172.21.15.98:0/2590630073 >> 172.21.15.98:0/2590630073 conn(0x7f37680fcde0 msgr2=0x7f37680ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.744+0000 7f376ee95640 1 -- 172.21.15.98:0/2590630073 shutdown_connections 2024-05-09T08:50:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.744+0000 7f376ee95640 1 -- 172.21.15.98:0/2590630073 wait complete. 2024-05-09T08:50:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.745+0000 7f376ee95640 1 Processor -- start 2024-05-09T08:50:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.745+0000 7f376ee95640 1 -- start start 2024-05-09T08:50:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.745+0000 7f376ee95640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3768101890 0x7f3768197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.745+0000 7f376ee95640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3768102a80 0x7f37681980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.745+0000 7f376ee95640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3768103440 0x7f376819f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.745+0000 7f376ee95640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f376810d530 con 0x7f3768101890 2024-05-09T08:50:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.745+0000 7f376ee95640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f376810d3b0 con 0x7f3768103440 2024-05-09T08:50:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.745+0000 7f376ee95640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f376810d6b0 con 0x7f3768102a80 2024-05-09T08:50:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.745+0000 7f376cc31640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3768101890 0x7f3768197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.745+0000 7f376cc31640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3768101890 0x7f3768197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52158/0 (socket says 172.21.15.98:52158) 2024-05-09T08:50:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.745+0000 7f376cc31640 1 -- 172.21.15.98:0/2093552097 learned_addr learned my addr 172.21.15.98:0/2093552097 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.745+0000 7f376d432640 1 --2- 172.21.15.98:0/2093552097 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3768103440 0x7f376819f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.745+0000 7f3767fff640 1 --2- 172.21.15.98:0/2093552097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3768102a80 0x7f37681980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.746+0000 7f376cc31640 1 -- 172.21.15.98:0/2093552097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3768102a80 msgr2=0x7f37681980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.746+0000 7f376cc31640 1 --2- 172.21.15.98:0/2093552097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3768102a80 0x7f37681980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.746+0000 7f376cc31640 1 -- 172.21.15.98:0/2093552097 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3768103440 msgr2=0x7f376819f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.746+0000 7f376cc31640 1 --2- 172.21.15.98:0/2093552097 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3768103440 0x7f376819f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.746+0000 7f376cc31640 1 -- 172.21.15.98:0/2093552097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376819f7c0 con 0x7f3768101890 2024-05-09T08:50:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.746+0000 7f376cc31640 1 --2- 172.21.15.98:0/2093552097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3768101890 0x7f3768197ba0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f375c031610 tx=0x7f375c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.746+0000 7f3765ffb640 1 -- 172.21.15.98:0/2093552097 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c03f030 con 0x7f3768101890 2024-05-09T08:50:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.746+0000 7f3765ffb640 1 -- 172.21.15.98:0/2093552097 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f375c003ce0 con 0x7f3768101890 2024-05-09T08:50:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.746+0000 7f376ee95640 1 -- 172.21.15.98:0/2093552097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376819fa50 con 0x7f3768101890 2024-05-09T08:50:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.746+0000 7f3765ffb640 1 -- 172.21.15.98:0/2093552097 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c035070 con 0x7f3768101890 2024-05-09T08:50:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.746+0000 7f376ee95640 1 -- 172.21.15.98:0/2093552097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376819ff90 con 0x7f3768101890 2024-05-09T08:50:02.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.747+0000 7f376ee95640 1 -- 172.21.15.98:0/2093552097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3768108bb0 con 0x7f3768101890 2024-05-09T08:50:02.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.748+0000 7f3765ffb640 1 -- 172.21.15.98:0/2093552097 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f375c042030 con 0x7f3768101890 2024-05-09T08:50:02.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.749+0000 7f3765ffb640 1 --2- 172.21.15.98:0/2093552097 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f37580a2c10 0x7f37580a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:02.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.749+0000 7f3767fff640 1 --2- 172.21.15.98:0/2093552097 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f37580a2c10 0x7f37580a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:02.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.749+0000 7f3765ffb640 1 -- 172.21.15.98:0/2093552097 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f375c0ef840 con 0x7f3768101890 2024-05-09T08:50:02.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.749+0000 7f3767fff640 1 --2- 172.21.15.98:0/2093552097 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f37580a2c10 0x7f37580a50d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f37681990b0 tx=0x7f375400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:02.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.753+0000 7f3765ffb640 1 -- 172.21.15.98:0/2093552097 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f375c0bbdf0 con 0x7f3768101890 2024-05-09T08:50:02.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.971+0000 7f376ee95640 1 -- 172.21.15.98:0/2093552097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3768062430 con 0x7f3768101890 2024-05-09T08:50:02.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.972+0000 7f3765ffb640 1 -- 172.21.15.98:0/2093552097 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11940 (secure 0 0 0) 0x7f375c0c00f0 con 0x7f3768101890 2024-05-09T08:50:02.971 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:02.972 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:48:39.758226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24457,24461,24388,24439,14667]},"id":1}]} 2024-05-09T08:50:02.972 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-05-09T08:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.974+0000 7f376ee95640 1 -- 172.21.15.98:0/2093552097 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f37580a2c10 msgr2=0x7f37580a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.974+0000 7f376ee95640 1 --2- 172.21.15.98:0/2093552097 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f37580a2c10 0x7f37580a50d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f37681990b0 tx=0x7f375400b390 comp rx=0 tx=0).stop 2024-05-09T08:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.974+0000 7f376ee95640 1 -- 172.21.15.98:0/2093552097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3768101890 msgr2=0x7f3768197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.974+0000 7f376ee95640 1 --2- 172.21.15.98:0/2093552097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3768101890 0x7f3768197ba0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f375c031610 tx=0x7f375c0043b0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.974+0000 7f376ee95640 1 -- 172.21.15.98:0/2093552097 shutdown_connections 2024-05-09T08:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.974+0000 7f376ee95640 1 --2- 172.21.15.98:0/2093552097 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3768103440 0x7f376819f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.974+0000 7f376ee95640 1 --2- 172.21.15.98:0/2093552097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3768102a80 0x7f37681980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.974+0000 7f376ee95640 1 --2- 172.21.15.98:0/2093552097 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f37580a2c10 0x7f37580a50d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.974+0000 7f376ee95640 1 --2- 172.21.15.98:0/2093552097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3768101890 0x7f3768197ba0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.974+0000 7f376ee95640 1 -- 172.21.15.98:0/2093552097 >> 172.21.15.98:0/2093552097 conn(0x7f37680fcde0 msgr2=0x7f3768105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.974+0000 7f376ee95640 1 -- 172.21.15.98:0/2093552097 shutdown_connections 2024-05-09T08:50:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:02.974+0000 7f376ee95640 1 -- 172.21.15.98:0/2093552097 wait complete. 2024-05-09T08:50:02.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:50:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.087+0000 7fdd6b914640 1 -- 172.21.15.98:0/3305950706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd64104ca0 msgr2=0x7fdd64107090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.087+0000 7fdd6b914640 1 --2- 172.21.15.98:0/3305950706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd64104ca0 0x7fdd64107090 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fdd60009900 tx=0x7fdd6002a410 comp rx=0 tx=0).stop 2024-05-09T08:50:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.087+0000 7fdd6b914640 1 -- 172.21.15.98:0/3305950706 shutdown_connections 2024-05-09T08:50:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.087+0000 7fdd6b914640 1 --2- 172.21.15.98:0/3305950706 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd641075d0 0x7fdd641099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.087+0000 7fdd6b914640 1 --2- 172.21.15.98:0/3305950706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd64104ca0 0x7fdd64107090 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.087+0000 7fdd6b914640 1 --2- 172.21.15.98:0/3305950706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd64102370 0x7fdd64104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.087+0000 7fdd6b914640 1 -- 172.21.15.98:0/3305950706 >> 172.21.15.98:0/3305950706 conn(0x7fdd640fbd20 msgr2=0x7fdd640fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.087+0000 7fdd6b914640 1 -- 172.21.15.98:0/3305950706 shutdown_connections 2024-05-09T08:50:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.087+0000 7fdd6b914640 1 -- 172.21.15.98:0/3305950706 wait complete. 2024-05-09T08:50:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.087+0000 7fdd6b914640 1 Processor -- start 2024-05-09T08:50:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.087+0000 7fdd6b914640 1 -- start start 2024-05-09T08:50:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd6b914640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd64102370 0x7fdd64199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd6b914640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd64104ca0 0x7fdd6419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd6b914640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd641075d0 0x7fdd641a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd6b914640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6410d4e0 con 0x7fdd641075d0 2024-05-09T08:50:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd6b914640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6410d360 con 0x7fdd64102370 2024-05-09T08:50:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd6b914640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6410d660 con 0x7fdd64104ca0 2024-05-09T08:50:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd69eb1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd641075d0 0x7fdd641a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd69eb1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd641075d0 0x7fdd641a1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52170/0 (socket says 172.21.15.98:52170) 2024-05-09T08:50:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd69eb1640 1 -- 172.21.15.98:0/424534175 learned_addr learned my addr 172.21.15.98:0/424534175 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd68eaf640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd64104ca0 0x7fdd6419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd696b0640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd64102370 0x7fdd64199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd69eb1640 1 -- 172.21.15.98:0/424534175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd64104ca0 msgr2=0x7fdd6419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd69eb1640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd64104ca0 0x7fdd6419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd69eb1640 1 -- 172.21.15.98:0/424534175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd64102370 msgr2=0x7fdd64199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd69eb1640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd64102370 0x7fdd64199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd69eb1640 1 -- 172.21.15.98:0/424534175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd641a19d0 con 0x7fdd641075d0 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd696b0640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd64102370 0x7fdd64199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd68eaf640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd64104ca0 0x7fdd6419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd69eb1640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd641075d0 0x7fdd641a1360 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c00cad0 tx=0x7fdd5c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd567fc640 1 -- 172.21.15.98:0/424534175 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c00ae20 con 0x7fdd641075d0 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd567fc640 1 -- 172.21.15.98:0/424534175 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd5c005020 con 0x7fdd641075d0 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd6b914640 1 -- 172.21.15.98:0/424534175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd641a1cc0 con 0x7fdd641075d0 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd567fc640 1 -- 172.21.15.98:0/424534175 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c00f6f0 con 0x7fdd641075d0 2024-05-09T08:50:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.088+0000 7fdd6b914640 1 -- 172.21.15.98:0/424534175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd641a2190 con 0x7fdd641075d0 2024-05-09T08:50:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.089+0000 7fdd6b914640 1 -- 172.21.15.98:0/424534175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd641090c0 con 0x7fdd641075d0 2024-05-09T08:50:03.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.090+0000 7fdd567fc640 1 -- 172.21.15.98:0/424534175 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fdd5c0051d0 con 0x7fdd641075d0 2024-05-09T08:50:03.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.091+0000 7fdd567fc640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdd500a2b90 0x7fdd500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.091+0000 7fdd696b0640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdd500a2b90 0x7fdd500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.091+0000 7fdd567fc640 1 -- 172.21.15.98:0/424534175 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fdd5c0cf510 con 0x7fdd641075d0 2024-05-09T08:50:03.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.092+0000 7fdd696b0640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdd500a2b90 0x7fdd500a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800baa0 tx=0x7fdd58005d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.095+0000 7fdd567fc640 1 -- 172.21.15.98:0/424534175 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdd5c014030 con 0x7fdd641075d0 2024-05-09T08:50:03.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.274+0000 7fdd6b914640 1 -- 172.21.15.98:0/424534175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd640624d0 con 0x7fdd641075d0 2024-05-09T08:50:03.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.276+0000 7fdd567fc640 1 -- 172.21.15.98:0/424534175 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24622 (secure 0 0 0) 0x7fdd5c09c110 con 0x7fdd641075d0 2024-05-09T08:50:03.275 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:03.275 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":92,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:49:18.355914+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:50:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.277+0000 7fdd6b914640 1 -- 172.21.15.98:0/424534175 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdd500a2b90 msgr2=0x7fdd500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.277+0000 7fdd6b914640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdd500a2b90 0x7fdd500a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800baa0 tx=0x7fdd58005d90 comp rx=0 tx=0).stop 2024-05-09T08:50:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.278+0000 7fdd6b914640 1 -- 172.21.15.98:0/424534175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd641075d0 msgr2=0x7fdd641a1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.278+0000 7fdd6b914640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd641075d0 0x7fdd641a1360 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c00cad0 tx=0x7fdd5c00cf90 comp rx=0 tx=0).stop 2024-05-09T08:50:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.278+0000 7fdd6b914640 1 -- 172.21.15.98:0/424534175 shutdown_connections 2024-05-09T08:50:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.278+0000 7fdd6b914640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fdd500a2b90 0x7fdd500a5050 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.278+0000 7fdd6b914640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd641075d0 0x7fdd641a1360 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.278+0000 7fdd6b914640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd64104ca0 0x7fdd6419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.278+0000 7fdd6b914640 1 --2- 172.21.15.98:0/424534175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd64102370 0x7fdd64199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.278+0000 7fdd6b914640 1 -- 172.21.15.98:0/424534175 >> 172.21.15.98:0/424534175 conn(0x7fdd640fbd20 msgr2=0x7fdd64101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.278+0000 7fdd6b914640 1 -- 172.21.15.98:0/424534175 shutdown_connections 2024-05-09T08:50:03.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.278+0000 7fdd6b914640 1 -- 172.21.15.98:0/424534175 wait complete. 2024-05-09T08:50:03.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:50:03.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.391+0000 7f7868410640 1 -- 172.21.15.98:0/1042560562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78601007f0 msgr2=0x7f7860100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.391+0000 7f7868410640 1 --2- 172.21.15.98:0/1042560562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78601007f0 0x7f7860100c10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7850009ca0 tx=0x7f785002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 -- 172.21.15.98:0/1042560562 shutdown_connections 2024-05-09T08:50:03.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 --2- 172.21.15.98:0/1042560562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78601023a0 0x7f7860108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 --2- 172.21.15.98:0/1042560562 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78601019e0 0x7f7860101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 --2- 172.21.15.98:0/1042560562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78601007f0 0x7f7860100c10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 -- 172.21.15.98:0/1042560562 >> 172.21.15.98:0/1042560562 conn(0x7f78600fbd80 msgr2=0x7f78600fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 -- 172.21.15.98:0/1042560562 shutdown_connections 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 -- 172.21.15.98:0/1042560562 wait complete. 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 Processor -- start 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 -- start start 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78601007f0 0x7f7860196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78601019e0 0x7f7860196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78601023a0 0x7f786019e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f786010c400 con 0x7f78601019e0 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f786010c280 con 0x7f78601023a0 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f7868410640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f786010c580 con 0x7f78601007f0 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f78659ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78601019e0 0x7f7860196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f78659ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78601019e0 0x7f7860196fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52198/0 (socket says 172.21.15.98:52198) 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.392+0000 7f78659ab640 1 -- 172.21.15.98:0/1245150341 learned_addr learned my addr 172.21.15.98:0/1245150341 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.393+0000 7f78661ac640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78601007f0 0x7f7860196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.393+0000 7f78669ad640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78601023a0 0x7f786019e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.393+0000 7f78659ab640 1 -- 172.21.15.98:0/1245150341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78601007f0 msgr2=0x7f7860196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.393+0000 7f78659ab640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78601007f0 0x7f7860196aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.393+0000 7f78659ab640 1 -- 172.21.15.98:0/1245150341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78601023a0 msgr2=0x7f786019e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.393+0000 7f78659ab640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78601023a0 0x7f786019e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.393+0000 7f78659ab640 1 -- 172.21.15.98:0/1245150341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786019e6c0 con 0x7f78601019e0 2024-05-09T08:50:03.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.393+0000 7f78661ac640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78601007f0 0x7f7860196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:03.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.393+0000 7f78659ab640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78601019e0 0x7f7860196fe0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f785400d950 tx=0x7f785400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.393+0000 7f785f7fe640 1 -- 172.21.15.98:0/1245150341 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7854017030 con 0x7f78601019e0 2024-05-09T08:50:03.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.393+0000 7f785f7fe640 1 -- 172.21.15.98:0/1245150341 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7854004020 con 0x7f78601019e0 2024-05-09T08:50:03.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.393+0000 7f7868410640 1 -- 172.21.15.98:0/1245150341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786019e9b0 con 0x7f78601019e0 2024-05-09T08:50:03.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.393+0000 7f785f7fe640 1 -- 172.21.15.98:0/1245150341 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78540047a0 con 0x7f78601019e0 2024-05-09T08:50:03.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.393+0000 7f7868410640 1 -- 172.21.15.98:0/1245150341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786019ef00 con 0x7f78601019e0 2024-05-09T08:50:03.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.394+0000 7f7868410640 1 -- 172.21.15.98:0/1245150341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7860107b10 con 0x7f78601019e0 2024-05-09T08:50:03.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.395+0000 7f785f7fe640 1 -- 172.21.15.98:0/1245150341 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f78540041d0 con 0x7f78601019e0 2024-05-09T08:50:03.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.396+0000 7f785f7fe640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f78480a2b90 0x7f78480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.396+0000 7f78661ac640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f78480a2b90 0x7f78480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.396+0000 7f785f7fe640 1 -- 172.21.15.98:0/1245150341 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f78540cf550 con 0x7f78601019e0 2024-05-09T08:50:03.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.396+0000 7f78661ac640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f78480a2b90 0x7f78480a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7850006010 tx=0x7f7850031bb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.399+0000 7f785f7fe640 1 -- 172.21.15.98:0/1245150341 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f785409c150 con 0x7f78601019e0 2024-05-09T08:50:03.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:03 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3417978351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:03 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3268997158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:03 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2093552097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:03.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:03 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/424534175' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:03.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.613+0000 7f7868410640 1 -- 172.21.15.98:0/1245150341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f786002d1c0 con 0x7f78601019e0 2024-05-09T08:50:03.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.613+0000 7f785f7fe640 1 -- 172.21.15.98:0/1245150341 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11940 (secure 0 0 0) 0x7f786002d1c0 con 0x7f78601019e0 2024-05-09T08:50:03.613 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:03.613 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:48:39.758226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24457,24461,24388,24439,14667]},"id":1}]} 2024-05-09T08:50:03.613 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-05-09T08:50:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.615+0000 7f7868410640 1 -- 172.21.15.98:0/1245150341 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f78480a2b90 msgr2=0x7f78480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.615+0000 7f7868410640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f78480a2b90 0x7f78480a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7850006010 tx=0x7f7850031bb0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.615+0000 7f7868410640 1 -- 172.21.15.98:0/1245150341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78601019e0 msgr2=0x7f7860196fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.615+0000 7f7868410640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78601019e0 0x7f7860196fe0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f785400d950 tx=0x7f785400de10 comp rx=0 tx=0).stop 2024-05-09T08:50:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.615+0000 7f7868410640 1 -- 172.21.15.98:0/1245150341 shutdown_connections 2024-05-09T08:50:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.615+0000 7f7868410640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78601023a0 0x7f786019e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.615+0000 7f7868410640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f78480a2b90 0x7f78480a5050 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.615+0000 7f7868410640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78601019e0 0x7f7860196fe0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.615+0000 7f7868410640 1 --2- 172.21.15.98:0/1245150341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78601007f0 0x7f7860196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.615+0000 7f7868410640 1 -- 172.21.15.98:0/1245150341 >> 172.21.15.98:0/1245150341 conn(0x7f78600fbd80 msgr2=0x7f7860104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.615+0000 7f7868410640 1 -- 172.21.15.98:0/1245150341 shutdown_connections 2024-05-09T08:50:03.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.616+0000 7f7868410640 1 -- 172.21.15.98:0/1245150341 wait complete. 2024-05-09T08:50:03.626 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-09T08:48:17.933763+0000', 'modified': '2024-05-09T08:48:39.758226+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24461, 'mds_1': 24457, 'mds_2': 24439, 'mds_3': 24388, 'mds_4': 14667}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14637': {'gid': 14637, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6839/3981776050', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 3981776050}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 3981776050}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14649': {'gid': 14649, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6841/2714016730', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6840', 'nonce': 2714016730}, {'type': 'v1', 'addr': '172.21.15.98:6841', 'nonce': 2714016730}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14667': {'gid': 14667, 'name': 'b', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.105:6839/1933710170', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6838', 'nonce': 1933710170}, {'type': 'v1', 'addr': '172.21.15.105:6839', 'nonce': 1933710170}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24388': {'gid': 24388, 'name': 'l', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.142:6833/606414638', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6832', 'nonce': 606414638}, {'type': 'v1', 'addr': '172.21.15.142:6833', 'nonce': 606414638}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24398': {'gid': 24398, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.105:6833/1472919356', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6832', 'nonce': 1472919356}, {'type': 'v1', 'addr': '172.21.15.105:6833', 'nonce': 1472919356}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24421': {'gid': 24421, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.142:6835/2791041858', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6834', 'nonce': 2791041858}, {'type': 'v1', 'addr': '172.21.15.142:6835', 'nonce': 2791041858}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24428': {'gid': 24428, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.105:6835/588012490', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6834', 'nonce': 588012490}, {'type': 'v1', 'addr': '172.21.15.105:6835', 'nonce': 588012490}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24439': {'gid': 24439, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.142:6837/1837318616', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6836', 'nonce': 1837318616}, {'type': 'v1', 'addr': '172.21.15.142:6837', 'nonce': 1837318616}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.142:6839/3036634959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6838', 'nonce': 3036634959}, {'type': 'v1', 'addr': '172.21.15.142:6839', 'nonce': 3036634959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'h', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.105:6837/992088454', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6836', 'nonce': 992088454}, {'type': 'v1', 'addr': '172.21.15.105:6837', 'nonce': 992088454}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24461, 'qdb_cluster': [24457, 24461, 24388, 24439, 14667]} 2024-05-09T08:50:03.627 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-05-09T08:50:03.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-09T08:50:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:03 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3417978351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:03 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3268997158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:03.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:03 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2093552097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:03.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:03 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/424534175' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:03.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.729+0000 7f7840ec1640 1 -- 172.21.15.98:0/2743069264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783c101880 msgr2=0x7f783c101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.729+0000 7f7840ec1640 1 --2- 172.21.15.98:0/2743069264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783c101880 0x7f783c101ca0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7834009900 tx=0x7f783402a410 comp rx=0 tx=0).stop 2024-05-09T08:50:03.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 -- 172.21.15.98:0/2743069264 shutdown_connections 2024-05-09T08:50:03.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 --2- 172.21.15.98:0/2743069264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f783c103430 0x7f783c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 --2- 172.21.15.98:0/2743069264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f783c102a70 0x7f783c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 --2- 172.21.15.98:0/2743069264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783c101880 0x7f783c101ca0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 -- 172.21.15.98:0/2743069264 >> 172.21.15.98:0/2743069264 conn(0x7f783c0fce10 msgr2=0x7f783c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 -- 172.21.15.98:0/2743069264 shutdown_connections 2024-05-09T08:50:03.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 -- 172.21.15.98:0/2743069264 wait complete. 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 Processor -- start 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 -- start start 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f783c101880 0x7f783c197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783c102a70 0x7f783c1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f783c103430 0x7f783c19f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f783c10d550 con 0x7f783c102a70 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f783c10d3d0 con 0x7f783c101880 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.730+0000 7f7840ec1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f783c10d6d0 con 0x7f783c103430 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f7839d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783c102a70 0x7f783c1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f7839d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783c102a70 0x7f783c1981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52212/0 (socket says 172.21.15.98:52212) 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f7839d9b640 1 -- 172.21.15.98:0/3257307538 learned_addr learned my addr 172.21.15.98:0/3257307538 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f783a59c640 1 --2- 172.21.15.98:0/3257307538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f783c101880 0x7f783c197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f7839d9b640 1 -- 172.21.15.98:0/3257307538 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f783c103430 msgr2=0x7f783c19f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f7839d9b640 1 --2- 172.21.15.98:0/3257307538 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f783c103430 0x7f783c19f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f7839d9b640 1 -- 172.21.15.98:0/3257307538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f783c101880 msgr2=0x7f783c197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f7839d9b640 1 --2- 172.21.15.98:0/3257307538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f783c101880 0x7f783c197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f7839d9b640 1 -- 172.21.15.98:0/3257307538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783c19f8c0 con 0x7f783c102a70 2024-05-09T08:50:03.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f7839d9b640 1 --2- 172.21.15.98:0/3257307538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783c102a70 0x7f783c1981c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f782400d950 tx=0x7f782400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f78337fe640 1 -- 172.21.15.98:0/3257307538 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7824017070 con 0x7f783c102a70 2024-05-09T08:50:03.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f7840ec1640 1 -- 172.21.15.98:0/3257307538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f783c19fbb0 con 0x7f783c102a70 2024-05-09T08:50:03.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f78337fe640 1 -- 172.21.15.98:0/3257307538 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7824009c30 con 0x7f783c102a70 2024-05-09T08:50:03.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f78337fe640 1 -- 172.21.15.98:0/3257307538 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782400f6b0 con 0x7f783c102a70 2024-05-09T08:50:03.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.731+0000 7f7840ec1640 1 -- 172.21.15.98:0/3257307538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f783c1a0150 con 0x7f783c102a70 2024-05-09T08:50:03.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:03 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3417978351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:03.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:03 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3268997158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:03.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:03 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2093552097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:03.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:03 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/424534175' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:03.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.733+0000 7f78337fe640 1 -- 172.21.15.98:0/3257307538 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f78240109d0 con 0x7f783c102a70 2024-05-09T08:50:03.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f7843bcd740 1 Processor -- start 2024-05-09T08:50:03.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f7843bcd740 1 -- start start 2024-05-09T08:50:03.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f7843bcd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f7843bcd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f7843bcd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1cf420 0x55b9eb1d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f7843bcd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b9eb12d4d0 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f7843bcd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b9eb0d53e0 con 0x55b9eb1cf420 2024-05-09T08:50:03.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f7843bcd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b9eb129490 con 0x55b9eb1c98c0 2024-05-09T08:50:03.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f78337fe640 1 --2- 172.21.15.98:0/3257307538 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f78280a2b90 0x7f78280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f783a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f783ad9d640 1 --2- 172.21.15.98:0/3257307538 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f78280a2b90 0x7f78280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f783a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52220/0 (socket says 172.21.15.98:52220) 2024-05-09T08:50:03.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f783a59c640 1 -- 172.21.15.98:0/1635673295 learned_addr learned my addr 172.21.15.98:0/1635673295 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:03.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f78337fe640 1 -- 172.21.15.98:0/3257307538 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f78240ceff0 con 0x7f783c102a70 2024-05-09T08:50:03.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.734+0000 7f7839d9b640 1 --2- 172.21.15.98:0/1635673295 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1cf420 0x55b9eb1d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.735+0000 7f783ad9d640 1 --2- 172.21.15.98:0/1635673295 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.735+0000 7f783a59c640 1 -- 172.21.15.98:0/1635673295 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1c98c0 msgr2=0x55b9eb1c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.735+0000 7f783a59c640 1 --2- 172.21.15.98:0/1635673295 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1c9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.735+0000 7f783a59c640 1 -- 172.21.15.98:0/1635673295 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1cf420 msgr2=0x55b9eb1d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.735+0000 7f783a59c640 1 --2- 172.21.15.98:0/1635673295 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1cf420 0x55b9eb1d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.735+0000 7f783a59c640 1 -- 172.21.15.98:0/1635673295 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9eae412d0 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.735+0000 7f783ad9d640 1 --2- 172.21.15.98:0/3257307538 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f78280a2b90 0x7f78280a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f782c0133e0 tx=0x7f782c0137e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.735+0000 7f783a59c640 1 --2- 172.21.15.98:0/1635673295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1cedb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7834031610 tx=0x7f7834032d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a34379e438e39c90 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.735+0000 7f7830ff9640 1 -- 172.21.15.98:0/1635673295 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7834040070 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.735+0000 7f7830ff9640 1 -- 172.21.15.98:0/1635673295 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78340318b0 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.735+0000 7f7830ff9640 1 -- 172.21.15.98:0/1635673295 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7834035040 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 -- 172.21.15.98:0/1635673295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 msgr2=0x55b9eb1cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 --2- 172.21.15.98:0/1635673295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1cedb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7834031610 tx=0x7f7834032d70 comp rx=0 tx=0).stop 2024-05-09T08:50:03.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 -- 172.21.15.98:0/1635673295 shutdown_connections 2024-05-09T08:50:03.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 --2- 172.21.15.98:0/1635673295 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1cf420 0x55b9eb1d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 --2- 172.21.15.98:0/1635673295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1cedb0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 --2- 172.21.15.98:0/1635673295 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 -- 172.21.15.98:0/1635673295 >> 172.21.15.98:0/1635673295 conn(0x55b9eb1c4f30 msgr2=0x55b9eb1c5330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 -- 172.21.15.98:0/1635673295 shutdown_connections 2024-05-09T08:50:03.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 -- 172.21.15.98:0/1635673295 wait complete. 2024-05-09T08:50:03.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 -- start start 2024-05-09T08:50:03.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 Processor -- start 2024-05-09T08:50:03.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1d3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1d4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1cf420 0x55b9eb1db250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b9eb1d2370 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b9eb0e88e0 con 0x55b9eb1c98c0 2024-05-09T08:50:03.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b9eb1d24f0 con 0x55b9eb1cf420 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.736+0000 7f7843bcd740 10 client.0 _ll_register_callbacks cb 0x55b9eb0b59c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7839d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1d4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7839d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1d4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52236/0 (socket says 172.21.15.98:52236) 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7839d9b640 1 -- 172.21.15.98:0/1064748436 learned_addr learned my addr 172.21.15.98:0/1064748436 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f783a59c640 1 --2- 172.21.15.98:0/1064748436 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1d3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7839d9b640 1 -- 172.21.15.98:0/1064748436 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1cf420 msgr2=0x55b9eb1db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7839d9b640 1 --2- 172.21.15.98:0/1064748436 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1cf420 0x55b9eb1db250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7839d9b640 1 -- 172.21.15.98:0/1064748436 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1c98c0 msgr2=0x55b9eb1d3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7839d9b640 1 --2- 172.21.15.98:0/1064748436 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1d3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7839d9b640 1 -- 172.21.15.98:0/1064748436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9eaf0b6b0 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7839d9b640 1 --2- 172.21.15.98:0/1064748436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1d4220 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f782409b440 tx=0x7f782409bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d8071e88196ffb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7821ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7821ffb640 1 -- 172.21.15.98:0/1064748436 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7824020040 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7821ffb640 1 -- 172.21.15.98:0/1064748436 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f782409a900 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7843bcd740 1 -- 172.21.15.98:0/1064748436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78340318b0 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7821ffb640 1 -- 172.21.15.98:0/1064748436 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f782409b4b0 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7843bcd740 10 client.15243 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:03.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7843bcd740 1 -- 172.21.15.98:0/1064748436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b9eb1de260 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.737+0000 7f7821ffb640 1 -- 172.21.15.98:0/1064748436 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78340318b0 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.738+0000 7f7821ffb640 1 -- 172.21.15.98:0/1064748436 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7824010460 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.738+0000 7f7843bcd740 10 client.15243 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:03.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.738+0000 7f7843bcd740 10 client.15243 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T08:50:03.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.738+0000 7f7843bcd740 20 client.15243 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:03.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.738+0000 7f7843bcd740 1 --2- 172.21.15.98:0/1064748436 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b9eb1e0850 0x55b9eb1e1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.738+0000 7f7843bcd740 4 client.15243 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:50:03.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.738+0000 7f7843bcd740 1 -- 172.21.15.98:0/1064748436 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9eb13ac00 con 0x55b9eb1e0850 2024-05-09T08:50:03.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.738+0000 7f783a59c640 1 --2- 172.21.15.98:0/1064748436 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b9eb1e0850 0x55b9eb1e1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.738+0000 7f7821ffb640 10 client.15243 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:50:03.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.738+0000 7f783a59c640 1 --2- 172.21.15.98:0/1064748436 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b9eb1e0850 0x55b9eb1e1cc0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b7271b30f43ab839 server_cookie=d307a7726ae9678d in_seq=0 out_seq=0 2024-05-09T08:50:03.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7821ffb640 1 -- 172.21.15.98:0/1064748436 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b9eb13ac00 con 0x55b9eb1e0850 2024-05-09T08:50:03.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7821ffb640 10 client.15243 handle_command_reply: tid=0 2024-05-09T08:50:03.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7843bcd740 1 client.15243 shutdown 2024-05-09T08:50:03.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7843bcd740 1 -- 172.21.15.98:0/1064748436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 msgr2=0x55b9eb1d4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7843bcd740 1 --2- 172.21.15.98:0/1064748436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1d4220 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f782409b440 tx=0x7f782409bbb0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7843bcd740 1 -- 172.21.15.98:0/1064748436 shutdown_connections 2024-05-09T08:50:03.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7843bcd740 1 --2- 172.21.15.98:0/1064748436 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b9eb1e0850 0x55b9eb1e1cc0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7843bcd740 1 --2- 172.21.15.98:0/1064748436 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1cf420 0x55b9eb1db250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7843bcd740 1 --2- 172.21.15.98:0/1064748436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1d4220 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7821ffb640 0 client.15243 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:50:03.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7843bcd740 1 --2- 172.21.15.98:0/1064748436 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1d3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7843bcd740 1 -- 172.21.15.98:0/1064748436 >> 172.21.15.98:0/1064748436 conn(0x55b9eb1c4f30 msgr2=0x55b9eb1c4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7843bcd740 1 -- 172.21.15.98:0/1064748436 shutdown_connections 2024-05-09T08:50:03.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7843bcd740 1 -- 172.21.15.98:0/1064748436 wait complete. 2024-05-09T08:50:03.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.739+0000 7f7843bcd740 20 client.15243 trim_cache size 0 max 16384 2024-05-09T08:50:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.757+0000 7f7843bcd740 1 Processor -- start 2024-05-09T08:50:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.757+0000 7f7843bcd740 1 -- start start 2024-05-09T08:50:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.757+0000 7f7843bcd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.757+0000 7f7843bcd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1db210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.757+0000 7f7843bcd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1cf420 0x55b9eb1c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.757+0000 7f7843bcd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b9eb1e0000 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.757+0000 7f7843bcd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b9eb1e06d0 con 0x55b9eb1cf420 2024-05-09T08:50:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.757+0000 7f7843bcd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b9eb1e0da0 con 0x55b9eb1c98c0 2024-05-09T08:50:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.758+0000 7f7839d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1db210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.758+0000 7f7839d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1db210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52244/0 (socket says 172.21.15.98:52244) 2024-05-09T08:50:03.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.758+0000 7f7839d9b640 1 -- 172.21.15.98:0/1707548655 learned_addr learned my addr 172.21.15.98:0/1707548655 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:03.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.758+0000 7f7839d9b640 1 -- 172.21.15.98:0/1707548655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1c98c0 msgr2=0x55b9eb1c7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.758+0000 7f783a59c640 1 --2- 172.21.15.98:0/1707548655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.758+0000 7f7839d9b640 1 --2- 172.21.15.98:0/1707548655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1c7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.758+0000 7f7839d9b640 1 -- 172.21.15.98:0/1707548655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1cf420 msgr2=0x55b9eb1c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.758+0000 7f7839d9b640 1 --2- 172.21.15.98:0/1707548655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1cf420 0x55b9eb1c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.758+0000 7f7839d9b640 1 -- 172.21.15.98:0/1707548655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9eb1cab40 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.758+0000 7f7839d9b640 1 --2- 172.21.15.98:0/1707548655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1db210 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7824000c00 tx=0x7f7824005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8abe0a355c1581 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.758+0000 7f7823fff640 1 -- 172.21.15.98:0/1707548655 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78240a4120 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.758+0000 7f7823fff640 1 -- 172.21.15.98:0/1707548655 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7824097b20 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.758+0000 7f7823fff640 1 -- 172.21.15.98:0/1707548655 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782409ab60 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 -- 172.21.15.98:0/1707548655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 msgr2=0x55b9eb1db210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 --2- 172.21.15.98:0/1707548655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1db210 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7824000c00 tx=0x7f7824005880 comp rx=0 tx=0).stop 2024-05-09T08:50:03.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 -- 172.21.15.98:0/1707548655 shutdown_connections 2024-05-09T08:50:03.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 --2- 172.21.15.98:0/1707548655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1cf420 0x55b9eb1c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 --2- 172.21.15.98:0/1707548655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1db210 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 --2- 172.21.15.98:0/1707548655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1c7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 -- 172.21.15.98:0/1707548655 >> 172.21.15.98:0/1707548655 conn(0x55b9eb1c4f30 msgr2=0x55b9eb1d28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 -- 172.21.15.98:0/1707548655 shutdown_connections 2024-05-09T08:50:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 -- 172.21.15.98:0/1707548655 wait complete. 2024-05-09T08:50:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 -- start start 2024-05-09T08:50:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 Processor -- start 2024-05-09T08:50:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1d7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1d7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1cf420 0x55b9eb1da6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b9eb1e1b40 con 0x55b9eb1c98c0 2024-05-09T08:50:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b9eb1e1470 con 0x55b9eb1ca2b0 2024-05-09T08:50:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b9eb1e2210 con 0x55b9eb1cf420 2024-05-09T08:50:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7843bcd740 10 client.0 _ll_register_callbacks cb 0x7f7824097750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f783a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1d7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f783a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1d7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52258/0 (socket says 172.21.15.98:52258) 2024-05-09T08:50:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f783a59c640 1 -- 172.21.15.98:0/2916614951 learned_addr learned my addr 172.21.15.98:0/2916614951 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f7839d9b640 1 --2- 172.21.15.98:0/2916614951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1d7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f783a59c640 1 -- 172.21.15.98:0/2916614951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1cf420 msgr2=0x55b9eb1da6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.759+0000 7f783a59c640 1 --2- 172.21.15.98:0/2916614951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1cf420 0x55b9eb1da6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f783a59c640 1 -- 172.21.15.98:0/2916614951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1ca2b0 msgr2=0x55b9eb1d7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f783a59c640 1 --2- 172.21.15.98:0/2916614951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1d7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f783a59c640 1 -- 172.21.15.98:0/2916614951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9eb1dacb0 con 0x55b9eb1c98c0 2024-05-09T08:50:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7839d9b640 1 --2- 172.21.15.98:0/2916614951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1d7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:03.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f783a59c640 1 --2- 172.21.15.98:0/2916614951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1d7350 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7834009900 tx=0x7f7834032e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb10c038253746e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7821ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:03.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7821ffb640 1 -- 172.21.15.98:0/2916614951 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7834053060 con 0x55b9eb1c98c0 2024-05-09T08:50:03.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7821ffb640 1 -- 172.21.15.98:0/2916614951 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f7834004240 con 0x55b9eb1c98c0 2024-05-09T08:50:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7843bcd740 1 -- 172.21.15.98:0/2916614951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7824097b20 con 0x55b9eb1c98c0 2024-05-09T08:50:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7821ffb640 1 -- 172.21.15.98:0/2916614951 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f783405a050 con 0x55b9eb1c98c0 2024-05-09T08:50:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7843bcd740 10 client.15249 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7843bcd740 1 -- 172.21.15.98:0/2916614951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b9eb247060 con 0x55b9eb1c98c0 2024-05-09T08:50:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7821ffb640 1 -- 172.21.15.98:0/2916614951 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7824097b20 con 0x55b9eb1c98c0 2024-05-09T08:50:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7821ffb640 1 -- 172.21.15.98:0/2916614951 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7834005440 con 0x55b9eb1c98c0 2024-05-09T08:50:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7843bcd740 10 client.15249 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7843bcd740 10 client.15249 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T08:50:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7843bcd740 20 client.15249 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7843bcd740 1 --2- 172.21.15.98:0/2916614951 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b9eb24b840 0x55b9eb24a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7843bcd740 4 client.15249 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:50:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.760+0000 7f7843bcd740 1 -- 172.21.15.98:0/2916614951 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "status"}) -- 0x55b9eb134f50 con 0x55b9eb24b840 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7839d9b640 1 --2- 172.21.15.98:0/2916614951 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b9eb24b840 0x55b9eb24a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7839d9b640 1 --2- 172.21.15.98:0/2916614951 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b9eb24b840 0x55b9eb24a550 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=409e7ec9c8b30ba0 server_cookie=5ee6ff9da395259e in_seq=0 out_seq=0 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7821ffb640 10 client.15249 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7821ffb640 1 -- 172.21.15.98:0/2916614951 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b9eb134f50 con 0x55b9eb24b840 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7821ffb640 10 client.15249 handle_command_reply: tid=0 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7843bcd740 1 client.15249 shutdown 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7843bcd740 1 -- 172.21.15.98:0/2916614951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1c98c0 msgr2=0x55b9eb1d7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7843bcd740 1 --2- 172.21.15.98:0/2916614951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1d7350 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7834009900 tx=0x7f7834032e60 comp rx=0 tx=0).stop 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7843bcd740 1 -- 172.21.15.98:0/2916614951 shutdown_connections 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7843bcd740 1 --2- 172.21.15.98:0/2916614951 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b9eb24b840 0x55b9eb24a550 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7843bcd740 1 --2- 172.21.15.98:0/2916614951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9eb1cf420 0x55b9eb1da6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7843bcd740 1 --2- 172.21.15.98:0/2916614951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9eb1ca2b0 0x55b9eb1d7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7843bcd740 1 --2- 172.21.15.98:0/2916614951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b9eb1c98c0 0x55b9eb1d7350 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7821ffb640 0 client.15249 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:50:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7843bcd740 1 -- 172.21.15.98:0/2916614951 >> 172.21.15.98:0/2916614951 conn(0x55b9eb1c4f30 msgr2=0x55b9eb1c7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7843bcd740 1 -- 172.21.15.98:0/2916614951 shutdown_connections 2024-05-09T08:50:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7843bcd740 1 -- 172.21.15.98:0/2916614951 wait complete. 2024-05-09T08:50:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.761+0000 7f7843bcd740 20 client.15249 trim_cache size 0 max 16384 2024-05-09T08:50:03.772 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:50:03.772 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a57d9712-0ddf-11ef-bc98-c7b262605968", 2024-05-09T08:50:03.772 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 4, 2024-05-09T08:50:03.772 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 14667, 2024-05-09T08:50:03.772 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-05-09T08:50:03.772 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-05-09T08:50:03.772 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-05-09T08:50:03.773 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 85.021959885000001, 2024-05-09T08:50:03.773 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-05-09T08:50:03.773 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 92, 2024-05-09T08:50:03.773 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 90, 2024-05-09T08:50:03.773 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 111.863762494 2024-05-09T08:50:03.773 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:50:03.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.762+0000 7f7840ec1640 1 -- 172.21.15.98:0/3257307538 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f78280a2b90 msgr2=0x7f78280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.762+0000 7f7840ec1640 1 --2- 172.21.15.98:0/3257307538 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f78280a2b90 0x7f78280a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f782c0133e0 tx=0x7f782c0137e0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.762+0000 7f7840ec1640 1 -- 172.21.15.98:0/3257307538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783c102a70 msgr2=0x7f783c1981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.762+0000 7f7840ec1640 1 --2- 172.21.15.98:0/3257307538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783c102a70 0x7f783c1981c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f782400d950 tx=0x7f782400de10 comp rx=0 tx=0).stop 2024-05-09T08:50:03.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.762+0000 7f7840ec1640 1 -- 172.21.15.98:0/3257307538 shutdown_connections 2024-05-09T08:50:03.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.762+0000 7f7840ec1640 1 --2- 172.21.15.98:0/3257307538 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f783c103430 0x7f783c19f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.762+0000 7f7840ec1640 1 --2- 172.21.15.98:0/3257307538 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f78280a2b90 0x7f78280a5050 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.762+0000 7f7840ec1640 1 --2- 172.21.15.98:0/3257307538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783c102a70 0x7f783c1981c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.762+0000 7f7840ec1640 1 --2- 172.21.15.98:0/3257307538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f783c101880 0x7f783c197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.762+0000 7f7840ec1640 1 -- 172.21.15.98:0/3257307538 >> 172.21.15.98:0/3257307538 conn(0x7f783c0fce10 msgr2=0x7f783c105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.762+0000 7f7840ec1640 1 -- 172.21.15.98:0/3257307538 shutdown_connections 2024-05-09T08:50:03.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.762+0000 7f7840ec1640 1 -- 172.21.15.98:0/3257307538 wait complete. 2024-05-09T08:50:03.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-05-09T08:50:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.878+0000 7fcf0d332640 1 -- 172.21.15.98:0/2098852790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf081023b0 msgr2=0x7fcf081047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.878+0000 7fcf0d332640 1 --2- 172.21.15.98:0/2098852790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf081023b0 0x7fcf081047a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fcef8009ca0 tx=0x7fcef802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.879+0000 7fcf0d332640 1 -- 172.21.15.98:0/2098852790 shutdown_connections 2024-05-09T08:50:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.879+0000 7fcf0d332640 1 --2- 172.21.15.98:0/2098852790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf08107610 0x7fcf08109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.879+0000 7fcf0d332640 1 --2- 172.21.15.98:0/2098852790 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf08104ce0 0x7fcf081070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.879+0000 7fcf0d332640 1 --2- 172.21.15.98:0/2098852790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf081023b0 0x7fcf081047a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.879+0000 7fcf0d332640 1 -- 172.21.15.98:0/2098852790 >> 172.21.15.98:0/2098852790 conn(0x7fcf080fbce0 msgr2=0x7fcf080fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.879+0000 7fcf0d332640 1 -- 172.21.15.98:0/2098852790 shutdown_connections 2024-05-09T08:50:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.879+0000 7fcf0d332640 1 -- 172.21.15.98:0/2098852790 wait complete. 2024-05-09T08:50:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.879+0000 7fcf0d332640 1 Processor -- start 2024-05-09T08:50:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.879+0000 7fcf0d332640 1 -- start start 2024-05-09T08:50:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf0d332640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf081023b0 0x7fcf08197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf0d332640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf08104ce0 0x7fcf08198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf0d332640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf08107610 0x7fcf0819f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf0d332640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0810d4f0 con 0x7fcf081023b0 2024-05-09T08:50:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf0d332640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0810d370 con 0x7fcf08107610 2024-05-09T08:50:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf0d332640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0810d670 con 0x7fcf08104ce0 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf06ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf081023b0 0x7fcf08197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf06ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf081023b0 0x7fcf08197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52276/0 (socket says 172.21.15.98:52276) 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf06ffd640 1 -- 172.21.15.98:0/438907920 learned_addr learned my addr 172.21.15.98:0/438907920 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf077fe640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf08107610 0x7fcf0819f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf067fc640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf08104ce0 0x7fcf08198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf06ffd640 1 -- 172.21.15.98:0/438907920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf08104ce0 msgr2=0x7fcf08198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf06ffd640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf08104ce0 0x7fcf08198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf06ffd640 1 -- 172.21.15.98:0/438907920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf08107610 msgr2=0x7fcf0819f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf06ffd640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf08107610 0x7fcf0819f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf06ffd640 1 -- 172.21.15.98:0/438907920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0819f810 con 0x7fcf081023b0 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf067fc640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf08104ce0 0x7fcf08198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf077fe640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf08107610 0x7fcf0819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf06ffd640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf081023b0 0x7fcf08197bf0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcef8031a40 tx=0x7fcef8031ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fceeffff640 1 -- 172.21.15.98:0/438907920 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef802faa0 con 0x7fcf081023b0 2024-05-09T08:50:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fceeffff640 1 -- 172.21.15.98:0/438907920 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcef802fc40 con 0x7fcf081023b0 2024-05-09T08:50:03.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf0d332640 1 -- 172.21.15.98:0/438907920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf0819faa0 con 0x7fcf081023b0 2024-05-09T08:50:03.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fceeffff640 1 -- 172.21.15.98:0/438907920 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef803a790 con 0x7fcf081023b0 2024-05-09T08:50:03.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.880+0000 7fcf0d332640 1 -- 172.21.15.98:0/438907920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf0819fff0 con 0x7fcf081023b0 2024-05-09T08:50:03.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.883+0000 7fceeffff640 1 -- 172.21.15.98:0/438907920 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fcef8030b30 con 0x7fcf081023b0 2024-05-09T08:50:03.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.883+0000 7fcf1003d740 1 Processor -- start 2024-05-09T08:50:03.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.883+0000 7fcf1003d740 1 -- start start 2024-05-09T08:50:03.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.883+0000 7fcf1003d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd438c0 0x55859bd43ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.883+0000 7fcf1003d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd442b0 0x55859bd48db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.883+0000 7fcf1003d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd49420 0x55859bd4b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf1003d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55859bca74d0 con 0x55859bd438c0 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf1003d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55859bc4f3e0 con 0x55859bd442b0 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf1003d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55859bca3490 con 0x55859bd49420 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf077fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd438c0 0x55859bd43ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf077fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd438c0 0x55859bd43ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52288/0 (socket says 172.21.15.98:52288) 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf077fe640 1 -- 172.21.15.98:0/1651142511 learned_addr learned my addr 172.21.15.98:0/1651142511 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fceeffff640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcef00a2b90 0x7fcef00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf067fc640 1 --2- 172.21.15.98:0/1651142511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd49420 0x55859bd4b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf06ffd640 1 --2- 172.21.15.98:0/1651142511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd442b0 0x55859bd48db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf077fe640 1 -- 172.21.15.98:0/1651142511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd49420 msgr2=0x55859bd4b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf077fe640 1 --2- 172.21.15.98:0/1651142511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd49420 0x55859bd4b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf077fe640 1 -- 172.21.15.98:0/1651142511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd442b0 msgr2=0x55859bd48db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf077fe640 1 --2- 172.21.15.98:0/1651142511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd442b0 0x55859bd48db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf077fe640 1 -- 172.21.15.98:0/1651142511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55859b9bb2d0 con 0x55859bd438c0 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf067fc640 1 --2- 172.21.15.98:0/1651142511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd49420 0x55859bd4b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:03.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf06ffd640 1 --2- 172.21.15.98:0/1651142511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd442b0 0x55859bd48db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fceeffff640 1 -- 172.21.15.98:0/438907920 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fcef80f0b80 con 0x7fcf081023b0 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf077fe640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcef00a2b90 0x7fcef00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fcf077fe640 1 --2- 172.21.15.98:0/1651142511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd438c0 0x55859bd43ce0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fcf0000be70 tx=0x7fcf0000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c7f985747f3433 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fceed7fa640 1 -- 172.21.15.98:0/1651142511 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf00012de0 con 0x55859bd438c0 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fceed7fa640 1 -- 172.21.15.98:0/1651142511 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf0001cdf0 con 0x55859bd438c0 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.884+0000 7fceed7fa640 1 -- 172.21.15.98:0/1651142511 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0001bd50 con 0x55859bd438c0 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.885+0000 7fcf1003d740 1 -- 172.21.15.98:0/1651142511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd438c0 msgr2=0x55859bd43ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.885+0000 7fcf077fe640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcef00a2b90 0x7fcef00a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcf00020000 tx=0x7fcf0001d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.885+0000 7fcf1003d740 1 --2- 172.21.15.98:0/1651142511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd438c0 0x55859bd43ce0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fcf0000be70 tx=0x7fcf0000f730 comp rx=0 tx=0).stop 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.885+0000 7fcf1003d740 1 -- 172.21.15.98:0/1651142511 shutdown_connections 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.885+0000 7fcf1003d740 1 --2- 172.21.15.98:0/1651142511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd49420 0x55859bd4b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.885+0000 7fcf1003d740 1 --2- 172.21.15.98:0/1651142511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd442b0 0x55859bd48db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.885+0000 7fcf1003d740 1 --2- 172.21.15.98:0/1651142511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd438c0 0x55859bd43ce0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.885+0000 7fcf1003d740 1 -- 172.21.15.98:0/1651142511 >> 172.21.15.98:0/1651142511 conn(0x55859bd3ef30 msgr2=0x55859bd3f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.885+0000 7fcf1003d740 1 -- 172.21.15.98:0/1651142511 shutdown_connections 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.885+0000 7fcf1003d740 1 -- 172.21.15.98:0/1651142511 wait complete. 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.885+0000 7fcf1003d740 1 -- start start 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf1003d740 1 Processor -- start 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf1003d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd438c0 0x55859bd4dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf1003d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd442b0 0x55859bd4e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf1003d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd49420 0x55859bd551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf1003d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55859bd4c3d0 con 0x55859bd442b0 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf1003d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55859bc628e0 con 0x55859bd49420 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf1003d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55859bd4c550 con 0x55859bd438c0 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf1003d740 10 client.0 _ll_register_callbacks cb 0x55859bc2f9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf067fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd442b0 0x55859bd4e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf067fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd442b0 0x55859bd4e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52304/0 (socket says 172.21.15.98:52304) 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf067fc640 1 -- 172.21.15.98:0/3704052688 learned_addr learned my addr 172.21.15.98:0/3704052688 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf067fc640 1 -- 172.21.15.98:0/3704052688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd438c0 msgr2=0x55859bd4dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf06ffd640 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd438c0 0x55859bd4dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf077fe640 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd49420 0x55859bd551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf067fc640 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd438c0 0x55859bd4dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf067fc640 1 -- 172.21.15.98:0/3704052688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd49420 msgr2=0x55859bd551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf067fc640 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd49420 0x55859bd551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf067fc640 1 -- 172.21.15.98:0/3704052688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55859ba856b0 con 0x55859bd442b0 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf06ffd640 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd438c0 0x55859bd4dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf077fe640 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd49420 0x55859bd551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf067fc640 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd442b0 0x55859bd4e1e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcefc00de80 tx=0x7fcefc00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e17ca657f8973e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fceeaff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fceeaff5640 1 -- 172.21.15.98:0/3704052688 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc004220 con 0x55859bd442b0 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fceeaff5640 1 -- 172.21.15.98:0/3704052688 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fcefc0043c0 con 0x55859bd442b0 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.886+0000 7fcf1003d740 1 -- 172.21.15.98:0/3704052688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf0001cdf0 con 0x55859bd442b0 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.887+0000 7fceeaff5640 1 -- 172.21.15.98:0/3704052688 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcefc022050 con 0x55859bd442b0 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.887+0000 7fcf1003d740 10 client.15261 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:03.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.887+0000 7fcf1003d740 1 -- 172.21.15.98:0/3704052688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55859bd58270 con 0x55859bd442b0 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.887+0000 7fceeaff5640 1 -- 172.21.15.98:0/3704052688 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf0001cdf0 con 0x55859bd442b0 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.887+0000 7fceeaff5640 1 -- 172.21.15.98:0/3704052688 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcefc003a10 con 0x55859bd442b0 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.887+0000 7fcf1003d740 10 client.15261 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.887+0000 7fcf1003d740 10 client.15261 resolve_mds: resolved name 'l' to daemon mds.l 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.887+0000 7fcf1003d740 20 client.15261 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.887+0000 7fcf1003d740 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55859bd5a930 0x55859bd5bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.887+0000 7fcf1003d740 4 client.15261 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.887+0000 7fcf1003d740 1 -- 172.21.15.98:0/3704052688 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55859bcb4c00 con 0x55859bd5a930 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.887+0000 7fcf06ffd640 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55859bd5a930 0x55859bd5bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.888+0000 7fceeaff5640 10 client.15261 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.888+0000 7fcf06ffd640 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55859bd5a930 0x55859bd5bda0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a892a113c2d2443d server_cookie=7095eaee428fb02c in_seq=0 out_seq=0 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fceeaff5640 1 -- 172.21.15.98:0/3704052688 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55859bcb4c00 con 0x55859bd5a930 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fceeaff5640 10 client.15261 handle_command_reply: tid=0 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fcf1003d740 1 client.15261 shutdown 2024-05-09T08:50:03.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fcf1003d740 1 -- 172.21.15.98:0/3704052688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd442b0 msgr2=0x55859bd4e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fcf1003d740 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd442b0 0x55859bd4e1e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcefc00de80 tx=0x7fcefc00f730 comp rx=0 tx=0).stop 2024-05-09T08:50:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fcf1003d740 1 -- 172.21.15.98:0/3704052688 shutdown_connections 2024-05-09T08:50:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fcf1003d740 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55859bd5a930 0x55859bd5bda0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fcf1003d740 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd49420 0x55859bd551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fceeaff5640 0 client.15261 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:50:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fcf1003d740 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd442b0 0x55859bd4e1e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fcf1003d740 1 --2- 172.21.15.98:0/3704052688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd438c0 0x55859bd4dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fcf1003d740 1 -- 172.21.15.98:0/3704052688 >> 172.21.15.98:0/3704052688 conn(0x55859bd3ef30 msgr2=0x55859bd3e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fcf1003d740 1 -- 172.21.15.98:0/3704052688 shutdown_connections 2024-05-09T08:50:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fcf1003d740 1 -- 172.21.15.98:0/3704052688 wait complete. 2024-05-09T08:50:03.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.889+0000 7fcf1003d740 20 client.15261 trim_cache size 0 max 16384 2024-05-09T08:50:03.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf1003d740 1 Processor -- start 2024-05-09T08:50:03.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf1003d740 1 -- start start 2024-05-09T08:50:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf1003d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd438c0 0x55859bd3d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf1003d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd442b0 0x55859bd550b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf1003d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd49420 0x55859bd3db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf1003d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55859bd5a0e0 con 0x55859bd442b0 2024-05-09T08:50:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf1003d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55859bd5a7b0 con 0x55859bd49420 2024-05-09T08:50:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf1003d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55859bd5ae80 con 0x55859bd438c0 2024-05-09T08:50:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf067fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd442b0 0x55859bd550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf067fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd442b0 0x55859bd550b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52320/0 (socket says 172.21.15.98:52320) 2024-05-09T08:50:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf067fc640 1 -- 172.21.15.98:0/1449013655 learned_addr learned my addr 172.21.15.98:0/1449013655 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf067fc640 1 -- 172.21.15.98:0/1449013655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd438c0 msgr2=0x55859bd3d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf067fc640 1 --2- 172.21.15.98:0/1449013655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd438c0 0x55859bd3d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf067fc640 1 -- 172.21.15.98:0/1449013655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd49420 msgr2=0x55859bd3db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf067fc640 1 --2- 172.21.15.98:0/1449013655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd49420 0x55859bd3db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf067fc640 1 -- 172.21.15.98:0/1449013655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55859bd58090 con 0x55859bd442b0 2024-05-09T08:50:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fcf067fc640 1 --2- 172.21.15.98:0/1449013655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd442b0 0x55859bd550b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcefc003e60 tx=0x7fcefc00ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7069a4210d881017 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fceecff9640 1 -- 172.21.15.98:0/1449013655 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc00fc30 con 0x55859bd442b0 2024-05-09T08:50:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fceecff9640 1 -- 172.21.15.98:0/1449013655 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcefc02a030 con 0x55859bd442b0 2024-05-09T08:50:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.907+0000 7fceecff9640 1 -- 172.21.15.98:0/1449013655 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc019c50 con 0x55859bd442b0 2024-05-09T08:50:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.908+0000 7fcf1003d740 1 -- 172.21.15.98:0/1449013655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd442b0 msgr2=0x55859bd550b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.908+0000 7fcf1003d740 1 --2- 172.21.15.98:0/1449013655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd442b0 0x55859bd550b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcefc003e60 tx=0x7fcefc00ffd0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.908+0000 7fcf1003d740 1 -- 172.21.15.98:0/1449013655 shutdown_connections 2024-05-09T08:50:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.908+0000 7fcf1003d740 1 --2- 172.21.15.98:0/1449013655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd49420 0x55859bd3db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.908+0000 7fcf1003d740 1 --2- 172.21.15.98:0/1449013655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd442b0 0x55859bd550b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.908+0000 7fcf1003d740 1 --2- 172.21.15.98:0/1449013655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd438c0 0x55859bd3d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.908+0000 7fcf1003d740 1 -- 172.21.15.98:0/1449013655 >> 172.21.15.98:0/1449013655 conn(0x55859bd3ef30 msgr2=0x55859bd455d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.908+0000 7fcf1003d740 1 -- 172.21.15.98:0/1449013655 shutdown_connections 2024-05-09T08:50:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.908+0000 7fcf1003d740 1 -- 172.21.15.98:0/1449013655 wait complete. 2024-05-09T08:50:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.908+0000 7fcf1003d740 1 -- start start 2024-05-09T08:50:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.908+0000 7fcf1003d740 1 Processor -- start 2024-05-09T08:50:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf1003d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd438c0 0x55859bd50c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf1003d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd442b0 0x55859bd51170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf1003d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd49420 0x55859bd54190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf1003d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55859bd5bc20 con 0x55859bd438c0 2024-05-09T08:50:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf1003d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55859bd5b550 con 0x55859bd442b0 2024-05-09T08:50:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf1003d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55859bd5c2f0 con 0x55859bd49420 2024-05-09T08:50:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf1003d740 10 client.0 _ll_register_callbacks cb 0x7fcefc00d380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf06ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd438c0 0x55859bd50c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf06ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd438c0 0x55859bd50c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52332/0 (socket says 172.21.15.98:52332) 2024-05-09T08:50:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf06ffd640 1 -- 172.21.15.98:0/2828660924 learned_addr learned my addr 172.21.15.98:0/2828660924 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf06ffd640 1 -- 172.21.15.98:0/2828660924 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd49420 msgr2=0x55859bd54190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf06ffd640 1 --2- 172.21.15.98:0/2828660924 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd49420 0x55859bd54190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf06ffd640 1 -- 172.21.15.98:0/2828660924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd442b0 msgr2=0x55859bd51170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf06ffd640 1 --2- 172.21.15.98:0/2828660924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd442b0 0x55859bd51170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf06ffd640 1 -- 172.21.15.98:0/2828660924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55859bd54780 con 0x55859bd438c0 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf06ffd640 1 --2- 172.21.15.98:0/2828660924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd438c0 0x55859bd50c30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcef80bcfc0 tx=0x7fcef808cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d31f79762256e13 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fceeaff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fceeaff5640 1 -- 172.21.15.98:0/2828660924 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef80bd080 con 0x55859bd438c0 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fceeaff5640 1 -- 172.21.15.98:0/2828660924 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fcef80dd0b0 con 0x55859bd438c0 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf1003d740 1 -- 172.21.15.98:0/2828660924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcefc02a030 con 0x55859bd438c0 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fceeaff5640 1 -- 172.21.15.98:0/2828660924 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcef80c1430 con 0x55859bd438c0 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf1003d740 10 client.15267 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fcf1003d740 1 -- 172.21.15.98:0/2828660924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55859bdc0bf0 con 0x55859bd438c0 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.909+0000 7fceeaff5640 1 -- 172.21.15.98:0/2828660924 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcefc02a030 con 0x55859bd438c0 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.910+0000 7fceeaff5640 1 -- 172.21.15.98:0/2828660924 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcef80bd2e0 con 0x55859bd438c0 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.910+0000 7fcf1003d740 10 client.15267 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.910+0000 7fcf1003d740 10 client.15267 resolve_mds: resolved name 'l' to daemon mds.l 2024-05-09T08:50:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.910+0000 7fcf1003d740 20 client.15267 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.910+0000 7fcf1003d740 1 --2- 172.21.15.98:0/2828660924 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55859bdc5920 0x55859bdc4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.910+0000 7fcf1003d740 4 client.15267 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.910+0000 7fcf1003d740 1 -- 172.21.15.98:0/2828660924 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "status"}) -- 0x55859bcaef50 con 0x55859bdc5920 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.910+0000 7fcf067fc640 1 --2- 172.21.15.98:0/2828660924 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55859bdc5920 0x55859bdc4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.910+0000 7fceeaff5640 10 client.15267 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.910+0000 7fcf067fc640 1 --2- 172.21.15.98:0/2828660924 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55859bdc5920 0x55859bdc4630 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eeaa034cfcd92234 server_cookie=cf2ff4f86e3b5e58 in_seq=0 out_seq=0 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fceeaff5640 1 -- 172.21.15.98:0/2828660924 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55859bcaef50 con 0x55859bdc5920 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fceeaff5640 10 client.15267 handle_command_reply: tid=0 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf1003d740 1 client.15267 shutdown 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf1003d740 1 -- 172.21.15.98:0/2828660924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd438c0 msgr2=0x55859bd50c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf1003d740 1 --2- 172.21.15.98:0/2828660924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd438c0 0x55859bd50c30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcef80bcfc0 tx=0x7fcef808cf30 comp rx=0 tx=0).stop 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf1003d740 1 -- 172.21.15.98:0/2828660924 shutdown_connections 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf1003d740 1 --2- 172.21.15.98:0/2828660924 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55859bdc5920 0x55859bdc4630 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf1003d740 1 --2- 172.21.15.98:0/2828660924 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55859bd49420 0x55859bd54190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fceeaff5640 0 client.15267 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:50:03.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf1003d740 1 --2- 172.21.15.98:0/2828660924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55859bd442b0 0x55859bd51170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf1003d740 1 --2- 172.21.15.98:0/2828660924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55859bd438c0 0x55859bd50c30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf1003d740 1 -- 172.21.15.98:0/2828660924 >> 172.21.15.98:0/2828660924 conn(0x55859bd3ef30 msgr2=0x55859bd41cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf1003d740 1 -- 172.21.15.98:0/2828660924 shutdown_connections 2024-05-09T08:50:03.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf1003d740 1 -- 172.21.15.98:0/2828660924 wait complete. 2024-05-09T08:50:03.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf1003d740 20 client.15267 trim_cache size 0 max 16384 2024-05-09T08:50:03.917 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:50:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a57d9712-0ddf-11ef-bc98-c7b262605968", 2024-05-09T08:50:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 3, 2024-05-09T08:50:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24388, 2024-05-09T08:50:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-05-09T08:50:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-05-09T08:50:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-05-09T08:50:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 86.178862507999995, 2024-05-09T08:50:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-05-09T08:50:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 92, 2024-05-09T08:50:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 90, 2024-05-09T08:50:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 143.51571508999999 2024-05-09T08:50:03.918 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:50:03.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf0d332640 1 -- 172.21.15.98:0/438907920 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcef00a2b90 msgr2=0x7fcef00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf0d332640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcef00a2b90 0x7fcef00a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcf00020000 tx=0x7fcf0001d000 comp rx=0 tx=0).stop 2024-05-09T08:50:03.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf0d332640 1 -- 172.21.15.98:0/438907920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf081023b0 msgr2=0x7fcf08197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:03.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf0d332640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf081023b0 0x7fcf08197bf0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcef8031a40 tx=0x7fcef8031ec0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf0d332640 1 -- 172.21.15.98:0/438907920 shutdown_connections 2024-05-09T08:50:03.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf0d332640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf08107610 0x7fcf0819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf0d332640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf08104ce0 0x7fcf08198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf0d332640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcef00a2b90 0x7fcef00a5050 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf0d332640 1 --2- 172.21.15.98:0/438907920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf081023b0 0x7fcf08197bf0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:03.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf0d332640 1 -- 172.21.15.98:0/438907920 >> 172.21.15.98:0/438907920 conn(0x7fcf080fbce0 msgr2=0x7fcf081031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:03.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf0d332640 1 -- 172.21.15.98:0/438907920 shutdown_connections 2024-05-09T08:50:03.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:03.911+0000 7fcf0d332640 1 -- 172.21.15.98:0/438907920 wait complete. 2024-05-09T08:50:03.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T08:50:04.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.025+0000 7fbd99ecd640 1 -- 172.21.15.98:0/3753542556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd94102320 msgr2=0x7fbd94108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.025+0000 7fbd99ecd640 1 --2- 172.21.15.98:0/3753542556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd94102320 0x7fbd94108bb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c009ca0 tx=0x7fbd8c02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.025+0000 7fbd99ecd640 1 -- 172.21.15.98:0/3753542556 shutdown_connections 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.025+0000 7fbd99ecd640 1 --2- 172.21.15.98:0/3753542556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd94102320 0x7fbd94108bb0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.025+0000 7fbd99ecd640 1 --2- 172.21.15.98:0/3753542556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbd94101960 0x7fbd94101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.025+0000 7fbd99ecd640 1 --2- 172.21.15.98:0/3753542556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd94100770 0x7fbd94100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.025+0000 7fbd99ecd640 1 -- 172.21.15.98:0/3753542556 >> 172.21.15.98:0/3753542556 conn(0x7fbd940fbce0 msgr2=0x7fbd940fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.025+0000 7fbd99ecd640 1 -- 172.21.15.98:0/3753542556 shutdown_connections 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.025+0000 7fbd99ecd640 1 -- 172.21.15.98:0/3753542556 wait complete. 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.025+0000 7fbd99ecd640 1 Processor -- start 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.025+0000 7fbd99ecd640 1 -- start start 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.025+0000 7fbd99ecd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd94100770 0x7fbd94069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd99ecd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd94101960 0x7fbd9419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd99ecd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbd94102320 0x7fbd9419eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd99ecd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9410c3f0 con 0x7fbd94101960 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd99ecd640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9410c270 con 0x7fbd94102320 2024-05-09T08:50:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd99ecd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9410c570 con 0x7fbd94100770 2024-05-09T08:50:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd8bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd94101960 0x7fbd9419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd8bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd94101960 0x7fbd9419c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52348/0 (socket says 172.21.15.98:52348) 2024-05-09T08:50:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd8bfff640 1 -- 172.21.15.98:0/1762384085 learned_addr learned my addr 172.21.15.98:0/1762384085 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd93fff640 1 --2- 172.21.15.98:0/1762384085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbd94102320 0x7fbd9419eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd8bfff640 1 -- 172.21.15.98:0/1762384085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd94100770 msgr2=0x7fbd94069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd8bfff640 1 --2- 172.21.15.98:0/1762384085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd94100770 0x7fbd94069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd8bfff640 1 -- 172.21.15.98:0/1762384085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbd94102320 msgr2=0x7fbd9419eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd8bfff640 1 --2- 172.21.15.98:0/1762384085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbd94102320 0x7fbd9419eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd8bfff640 1 -- 172.21.15.98:0/1762384085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd9419f7e0 con 0x7fbd94101960 2024-05-09T08:50:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd93fff640 1 --2- 172.21.15.98:0/1762384085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbd94102320 0x7fbd9419eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd8bfff640 1 --2- 172.21.15.98:0/1762384085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd94101960 0x7fbd9419c8b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c00e9f0 tx=0x7fbd7c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd917fa640 1 -- 172.21.15.98:0/1762384085 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c00cd80 con 0x7fbd94101960 2024-05-09T08:50:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd917fa640 1 -- 172.21.15.98:0/1762384085 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd7c010910 con 0x7fbd94101960 2024-05-09T08:50:04.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd917fa640 1 -- 172.21.15.98:0/1762384085 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c018990 con 0x7fbd94101960 2024-05-09T08:50:04.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd99ecd640 1 -- 172.21.15.98:0/1762384085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd9419fa70 con 0x7fbd94101960 2024-05-09T08:50:04.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.026+0000 7fbd99ecd640 1 -- 172.21.15.98:0/1762384085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd9419ffc0 con 0x7fbd94101960 2024-05-09T08:50:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.028+0000 7fbd917fa640 1 -- 172.21.15.98:0/1762384085 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbd7c010ac0 con 0x7fbd94101960 2024-05-09T08:50:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd9cbd9740 1 Processor -- start 2024-05-09T08:50:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd9cbd9740 1 -- start start 2024-05-09T08:50:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd9cbd9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd388c0 0x56421dd38ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd9cbd9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd392b0 0x56421dd3ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd9cbd9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd3e420 0x56421dd408b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd9cbd9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56421dc9c4d0 con 0x56421dd3e420 2024-05-09T08:50:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd9cbd9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56421dc443e0 con 0x56421dd388c0 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd9cbd9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56421dc98490 con 0x56421dd392b0 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd8bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd3e420 0x56421dd408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd8bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd3e420 0x56421dd408b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52354/0 (socket says 172.21.15.98:52354) 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd8bfff640 1 -- 172.21.15.98:0/3914102089 learned_addr learned my addr 172.21.15.98:0/3914102089 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd937fe640 1 --2- 172.21.15.98:0/3914102089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd392b0 0x56421dd3ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd917fa640 1 --2- 172.21.15.98:0/1762384085 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbd800a2b90 0x7fbd800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd93fff640 1 --2- 172.21.15.98:0/3914102089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd388c0 0x56421dd38ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd93fff640 1 --2- 172.21.15.98:0/1762384085 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbd800a2b90 0x7fbd800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd8bfff640 1 -- 172.21.15.98:0/3914102089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd392b0 msgr2=0x56421dd3ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd8bfff640 1 --2- 172.21.15.98:0/3914102089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd392b0 0x56421dd3ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd8bfff640 1 -- 172.21.15.98:0/3914102089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd388c0 msgr2=0x56421dd38ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd8bfff640 1 --2- 172.21.15.98:0/3914102089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd388c0 0x56421dd38ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd917fa640 1 -- 172.21.15.98:0/1762384085 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fbd7c0cf7f0 con 0x7fbd94101960 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.029+0000 7fbd8bfff640 1 -- 172.21.15.98:0/3914102089 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56421d9b02d0 con 0x56421dd3e420 2024-05-09T08:50:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.030+0000 7fbd8bfff640 1 --2- 172.21.15.98:0/3914102089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd3e420 0x56421dd408b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c0cced0 tx=0x7fbd7c018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd29bf9a4b5dac3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.030+0000 7fbd93fff640 1 --2- 172.21.15.98:0/1762384085 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbd800a2b90 0x7fbd800a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c005e30 tx=0x7fbd8c0095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.030+0000 7fbd89ffb640 1 -- 172.21.15.98:0/3914102089 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c09c3f0 con 0x56421dd3e420 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.030+0000 7fbd89ffb640 1 -- 172.21.15.98:0/3914102089 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd7c09bcf0 con 0x56421dd3e420 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.030+0000 7fbd89ffb640 1 -- 172.21.15.98:0/3914102089 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c05e350 con 0x56421dd3e420 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.030+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/3914102089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd3e420 msgr2=0x56421dd408b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.030+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/3914102089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd3e420 0x56421dd408b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c0cced0 tx=0x7fbd7c018420 comp rx=0 tx=0).stop 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.030+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/3914102089 shutdown_connections 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.030+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/3914102089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd3e420 0x56421dd408b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.030+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/3914102089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd392b0 0x56421dd3ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.030+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/3914102089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd388c0 0x56421dd38ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.030+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/3914102089 >> 172.21.15.98:0/3914102089 conn(0x56421dd33f30 msgr2=0x56421dd34330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.030+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/3914102089 shutdown_connections 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.030+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/3914102089 wait complete. 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.031+0000 7fbd9cbd9740 1 -- start start 2024-05-09T08:50:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.031+0000 7fbd9cbd9740 1 Processor -- start 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.031+0000 7fbd9cbd9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd388c0 0x56421dd42cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.031+0000 7fbd9cbd9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd392b0 0x56421dd43200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.031+0000 7fbd9cbd9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd3e420 0x56421dd4a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.031+0000 7fbd9cbd9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56421dd41370 con 0x56421dd392b0 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.031+0000 7fbd9cbd9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56421dc578e0 con 0x56421dd3e420 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.031+0000 7fbd9cbd9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56421dd414f0 con 0x56421dd388c0 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.031+0000 7fbd9cbd9740 10 client.0 _ll_register_callbacks cb 0x56421dc249c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.031+0000 7fbd8bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd392b0 0x56421dd43200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.031+0000 7fbd8bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd392b0 0x56421dd43200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52368/0 (socket says 172.21.15.98:52368) 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.031+0000 7fbd8bfff640 1 -- 172.21.15.98:0/1565553786 learned_addr learned my addr 172.21.15.98:0/1565553786 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.031+0000 7fbd937fe640 1 --2- 172.21.15.98:0/1565553786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd388c0 0x56421dd42cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.031+0000 7fbd93fff640 1 --2- 172.21.15.98:0/1565553786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd3e420 0x56421dd4a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd8bfff640 1 -- 172.21.15.98:0/1565553786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd388c0 msgr2=0x56421dd42cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd8bfff640 1 --2- 172.21.15.98:0/1565553786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd388c0 0x56421dd42cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd8bfff640 1 -- 172.21.15.98:0/1565553786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd3e420 msgr2=0x56421dd4a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd8bfff640 1 --2- 172.21.15.98:0/1565553786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd3e420 0x56421dd4a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd8bfff640 1 -- 172.21.15.98:0/1565553786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56421da7a6b0 con 0x56421dd392b0 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd93fff640 1 --2- 172.21.15.98:0/1565553786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd3e420 0x56421dd4a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd8bfff640 1 --2- 172.21.15.98:0/1565553786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd392b0 0x56421dd43200 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c000c00 tx=0x7fbd7c005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4ff26b2e2a27c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd72ffd640 1 -- 172.21.15.98:0/1565553786 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c0a46d0 con 0x56421dd392b0 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd72ffd640 1 -- 172.21.15.98:0/1565553786 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fbd7c0a4bb0 con 0x56421dd392b0 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/1565553786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd7c09bcf0 con 0x56421dd392b0 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd72ffd640 1 -- 172.21.15.98:0/1565553786 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbd7c0a4960 con 0x56421dd392b0 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd9cbd9740 10 client.15279 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/1565553786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56421dd4d210 con 0x56421dd392b0 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd72ffd640 1 -- 172.21.15.98:0/1565553786 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd7c09bcf0 con 0x56421dd392b0 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd72ffd640 1 -- 172.21.15.98:0/1565553786 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbd7c018b30 con 0x56421dd392b0 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd9cbd9740 10 client.15279 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd9cbd9740 10 client.15279 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T08:50:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd9cbd9740 20 client.15279 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/1565553786 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x56421dd4f3c0 0x56421dd50830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd9cbd9740 4 client.15279 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:50:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.032+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/1565553786 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56421dca9c00 con 0x56421dd4f3c0 2024-05-09T08:50:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.033+0000 7fbd937fe640 1 --2- 172.21.15.98:0/1565553786 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x56421dd4f3c0 0x56421dd50830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.033+0000 7fbd72ffd640 10 client.15279 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:50:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.033+0000 7fbd937fe640 1 --2- 172.21.15.98:0/1565553786 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x56421dd4f3c0 0x56421dd50830 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=95ee0eb0a8573780 server_cookie=61d738a0ea322c7f in_seq=0 out_seq=0 2024-05-09T08:50:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd72ffd640 1 -- 172.21.15.98:0/1565553786 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56421dca9c00 con 0x56421dd4f3c0 2024-05-09T08:50:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd72ffd640 10 client.15279 handle_command_reply: tid=0 2024-05-09T08:50:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd9cbd9740 1 client.15279 shutdown 2024-05-09T08:50:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/1565553786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd392b0 msgr2=0x56421dd43200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/1565553786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd392b0 0x56421dd43200 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c000c00 tx=0x7fbd7c005c90 comp rx=0 tx=0).stop 2024-05-09T08:50:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/1565553786 shutdown_connections 2024-05-09T08:50:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/1565553786 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x56421dd4f3c0 0x56421dd50830 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/1565553786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd3e420 0x56421dd4a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd72ffd640 0 client.15279 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:50:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/1565553786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd392b0 0x56421dd43200 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/1565553786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd388c0 0x56421dd42cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/1565553786 >> 172.21.15.98:0/1565553786 conn(0x56421dd33f30 msgr2=0x56421dd33000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/1565553786 shutdown_connections 2024-05-09T08:50:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/1565553786 wait complete. 2024-05-09T08:50:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.034+0000 7fbd9cbd9740 20 client.15279 trim_cache size 0 max 16384 2024-05-09T08:50:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd9cbd9740 1 Processor -- start 2024-05-09T08:50:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd9cbd9740 1 -- start start 2024-05-09T08:50:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd9cbd9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd388c0 0x56421dd34310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd9cbd9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd392b0 0x56421dd4a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd9cbd9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd3e420 0x56421dd37770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd9cbd9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56421dd4eb70 con 0x56421dd392b0 2024-05-09T08:50:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd9cbd9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56421dd4f240 con 0x56421dd388c0 2024-05-09T08:50:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd9cbd9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56421dd4f910 con 0x56421dd3e420 2024-05-09T08:50:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd8bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd392b0 0x56421dd4a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd8bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd392b0 0x56421dd4a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52372/0 (socket says 172.21.15.98:52372) 2024-05-09T08:50:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd8bfff640 1 -- 172.21.15.98:0/704655698 learned_addr learned my addr 172.21.15.98:0/704655698 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd937fe640 1 --2- 172.21.15.98:0/704655698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd388c0 0x56421dd34310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd8bfff640 1 -- 172.21.15.98:0/704655698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd3e420 msgr2=0x56421dd37770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd93fff640 1 --2- 172.21.15.98:0/704655698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd3e420 0x56421dd37770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd8bfff640 1 --2- 172.21.15.98:0/704655698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd3e420 0x56421dd37770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd8bfff640 1 -- 172.21.15.98:0/704655698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd388c0 msgr2=0x56421dd34310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd8bfff640 1 --2- 172.21.15.98:0/704655698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd388c0 0x56421dd34310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd8bfff640 1 -- 172.21.15.98:0/704655698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56421dd39b40 con 0x56421dd392b0 2024-05-09T08:50:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd937fe640 1 --2- 172.21.15.98:0/704655698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd388c0 0x56421dd34310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.052+0000 7fbd93fff640 1 --2- 172.21.15.98:0/704655698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd3e420 0x56421dd37770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.053+0000 7fbd8bfff640 1 --2- 172.21.15.98:0/704655698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd392b0 0x56421dd4a1f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c09b9f0 tx=0x7fbd7c0611b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f1bca28b6b1a6a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.053+0000 7fbd897fa640 1 -- 172.21.15.98:0/704655698 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c097f90 con 0x56421dd392b0 2024-05-09T08:50:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.053+0000 7fbd897fa640 1 -- 172.21.15.98:0/704655698 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd7c09c3f0 con 0x56421dd392b0 2024-05-09T08:50:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.053+0000 7fbd897fa640 1 -- 172.21.15.98:0/704655698 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c0a44d0 con 0x56421dd392b0 2024-05-09T08:50:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.053+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/704655698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd392b0 msgr2=0x56421dd4a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.053+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/704655698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd392b0 0x56421dd4a1f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c09b9f0 tx=0x7fbd7c0611b0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.053+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/704655698 shutdown_connections 2024-05-09T08:50:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.053+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/704655698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd3e420 0x56421dd37770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.053+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/704655698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd392b0 0x56421dd4a1f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.053+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/704655698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd388c0 0x56421dd34310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.053+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/704655698 >> 172.21.15.98:0/704655698 conn(0x56421dd33f30 msgr2=0x56421dd40850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.053+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/704655698 shutdown_connections 2024-05-09T08:50:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.053+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/704655698 wait complete. 2024-05-09T08:50:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd9cbd9740 1 -- start start 2024-05-09T08:50:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd9cbd9740 1 Processor -- start 2024-05-09T08:50:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd9cbd9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd388c0 0x56421dd45c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd9cbd9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd392b0 0x56421dd46140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd9cbd9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd3e420 0x56421dd49110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd9cbd9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56421dd506b0 con 0x56421dd388c0 2024-05-09T08:50:04.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd9cbd9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56421dd4ffe0 con 0x56421dd3e420 2024-05-09T08:50:04.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd9cbd9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56421dd50d80 con 0x56421dd392b0 2024-05-09T08:50:04.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd9cbd9740 10 client.0 _ll_register_callbacks cb 0x7fbd84005eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:04.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd388c0 0x56421dd45c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd388c0 0x56421dd45c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52380/0 (socket says 172.21.15.98:52380) 2024-05-09T08:50:04.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd937fe640 1 -- 172.21.15.98:0/3523028023 learned_addr learned my addr 172.21.15.98:0/3523028023 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd93fff640 1 --2- 172.21.15.98:0/3523028023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd3e420 0x56421dd49110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd937fe640 1 -- 172.21.15.98:0/3523028023 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd392b0 msgr2=0x56421dd46140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd937fe640 1 --2- 172.21.15.98:0/3523028023 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd392b0 0x56421dd46140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd937fe640 1 -- 172.21.15.98:0/3523028023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd3e420 msgr2=0x56421dd49110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd937fe640 1 --2- 172.21.15.98:0/3523028023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd3e420 0x56421dd49110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.054+0000 7fbd937fe640 1 -- 172.21.15.98:0/3523028023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56421dd49700 con 0x56421dd388c0 2024-05-09T08:50:04.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd937fe640 1 --2- 172.21.15.98:0/3523028023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd388c0 0x56421dd45c00 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbd84005e00 tx=0x7fbd84019f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f5a9111c87eb8e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:04.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd72ffd640 1 -- 172.21.15.98:0/3523028023 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84010c80 con 0x56421dd388c0 2024-05-09T08:50:04.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd72ffd640 1 -- 172.21.15.98:0/3523028023 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fbd84019290 con 0x56421dd388c0 2024-05-09T08:50:04.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/3523028023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd7c09c3f0 con 0x56421dd388c0 2024-05-09T08:50:04.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd72ffd640 1 -- 172.21.15.98:0/3523028023 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbd84010e20 con 0x56421dd388c0 2024-05-09T08:50:04.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd9cbd9740 10 client.15285 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:04.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/3523028023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56421ddb5680 con 0x56421dd388c0 2024-05-09T08:50:04.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd72ffd640 1 -- 172.21.15.98:0/3523028023 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd7c09c3f0 con 0x56421dd388c0 2024-05-09T08:50:04.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd72ffd640 1 -- 172.21.15.98:0/3523028023 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbd84022ca0 con 0x56421dd388c0 2024-05-09T08:50:04.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd9cbd9740 10 client.15285 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd9cbd9740 10 client.15285 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T08:50:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd9cbd9740 20 client.15285 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/3523028023 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x56421ddba310 0x56421ddb6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd9cbd9740 4 client.15285 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:50:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.055+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/3523028023 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "status"}) -- 0x56421dca3f50 con 0x56421ddba310 2024-05-09T08:50:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd8bfff640 1 --2- 172.21.15.98:0/3523028023 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x56421ddba310 0x56421ddb6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd8bfff640 1 --2- 172.21.15.98:0/3523028023 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x56421ddba310 0x56421ddb6750 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bea88f065dec7c23 server_cookie=201a9e84e6d04407 in_seq=0 out_seq=0 2024-05-09T08:50:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd72ffd640 10 client.15285 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:50:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd72ffd640 1 -- 172.21.15.98:0/3523028023 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56421dca3f50 con 0x56421ddba310 2024-05-09T08:50:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd72ffd640 10 client.15285 handle_command_reply: tid=0 2024-05-09T08:50:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd9cbd9740 1 client.15285 shutdown 2024-05-09T08:50:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/3523028023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd388c0 msgr2=0x56421dd45c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/3523028023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd388c0 0x56421dd45c00 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbd84005e00 tx=0x7fbd84019f90 comp rx=0 tx=0).stop 2024-05-09T08:50:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/3523028023 shutdown_connections 2024-05-09T08:50:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/3523028023 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x56421ddba310 0x56421ddb6750 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/3523028023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421dd3e420 0x56421dd49110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd72ffd640 0 client.15285 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:50:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/3523028023 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421dd392b0 0x56421dd46140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd9cbd9740 1 --2- 172.21.15.98:0/3523028023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421dd388c0 0x56421dd45c00 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.056+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/3523028023 >> 172.21.15.98:0/3523028023 conn(0x56421dd33f30 msgr2=0x56421dd34310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/3523028023 shutdown_connections 2024-05-09T08:50:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd9cbd9740 1 -- 172.21.15.98:0/3523028023 wait complete. 2024-05-09T08:50:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd9cbd9740 20 client.15285 trim_cache size 0 max 16384 2024-05-09T08:50:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:50:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a57d9712-0ddf-11ef-bc98-c7b262605968", 2024-05-09T08:50:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2024-05-09T08:50:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24439, 2024-05-09T08:50:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-05-09T08:50:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-05-09T08:50:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-05-09T08:50:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 86.775460824999996, 2024-05-09T08:50:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-05-09T08:50:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 92, 2024-05-09T08:50:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 90, 2024-05-09T08:50:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 126.358986261 2024-05-09T08:50:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:50:04.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd99ecd640 1 -- 172.21.15.98:0/1762384085 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbd800a2b90 msgr2=0x7fbd800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd99ecd640 1 --2- 172.21.15.98:0/1762384085 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbd800a2b90 0x7fbd800a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c005e30 tx=0x7fbd8c0095c0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd99ecd640 1 -- 172.21.15.98:0/1762384085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd94101960 msgr2=0x7fbd9419c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd99ecd640 1 --2- 172.21.15.98:0/1762384085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd94101960 0x7fbd9419c8b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c00e9f0 tx=0x7fbd7c00eeb0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd99ecd640 1 -- 172.21.15.98:0/1762384085 shutdown_connections 2024-05-09T08:50:04.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd99ecd640 1 --2- 172.21.15.98:0/1762384085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbd94102320 0x7fbd9419eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd99ecd640 1 --2- 172.21.15.98:0/1762384085 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbd800a2b90 0x7fbd800a5050 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd99ecd640 1 --2- 172.21.15.98:0/1762384085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd94101960 0x7fbd9419c8b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd99ecd640 1 --2- 172.21.15.98:0/1762384085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd94100770 0x7fbd94069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd99ecd640 1 -- 172.21.15.98:0/1762384085 >> 172.21.15.98:0/1762384085 conn(0x7fbd940fbce0 msgr2=0x7fbd94104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd99ecd640 1 -- 172.21.15.98:0/1762384085 shutdown_connections 2024-05-09T08:50:04.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.057+0000 7fbd99ecd640 1 -- 172.21.15.98:0/1762384085 wait complete. 2024-05-09T08:50:04.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-05-09T08:50:04.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.171+0000 7f33122a5640 1 -- 172.21.15.98:0/2499963386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f330c102310 msgr2=0x7f330c104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.171+0000 7f33122a5640 1 --2- 172.21.15.98:0/2499963386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f330c102310 0x7f330c104700 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f32fc009900 tx=0x7f32fc02a410 comp rx=0 tx=0).stop 2024-05-09T08:50:04.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.171+0000 7f33122a5640 1 -- 172.21.15.98:0/2499963386 shutdown_connections 2024-05-09T08:50:04.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.171+0000 7f33122a5640 1 --2- 172.21.15.98:0/2499963386 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f330c107570 0x7f330c1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.171+0000 7f33122a5640 1 --2- 172.21.15.98:0/2499963386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f330c104c40 0x7f330c107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.171+0000 7f33122a5640 1 --2- 172.21.15.98:0/2499963386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f330c102310 0x7f330c104700 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.171+0000 7f33122a5640 1 -- 172.21.15.98:0/2499963386 >> 172.21.15.98:0/2499963386 conn(0x7f330c0fbd20 msgr2=0x7f330c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.171+0000 7f33122a5640 1 -- 172.21.15.98:0/2499963386 shutdown_connections 2024-05-09T08:50:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.171+0000 7f33122a5640 1 -- 172.21.15.98:0/2499963386 wait complete. 2024-05-09T08:50:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.172+0000 7f33122a5640 1 Processor -- start 2024-05-09T08:50:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.172+0000 7f33122a5640 1 -- start start 2024-05-09T08:50:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.172+0000 7f33122a5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f330c102310 0x7f330c197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.172+0000 7f33122a5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f330c104c40 0x7f330c198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.172+0000 7f33122a5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f330c107570 0x7f330c19f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.172+0000 7f33122a5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f330c10d560 con 0x7f330c107570 2024-05-09T08:50:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.172+0000 7f33122a5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f330c10d3e0 con 0x7f330c104c40 2024-05-09T08:50:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.172+0000 7f33122a5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f330c10d6e0 con 0x7f330c102310 2024-05-09T08:50:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.172+0000 7f3310842640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f330c107570 0x7f330c19f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.172+0000 7f3310842640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f330c107570 0x7f330c19f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52402/0 (socket says 172.21.15.98:52402) 2024-05-09T08:50:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.172+0000 7f3310842640 1 -- 172.21.15.98:0/1997917496 learned_addr learned my addr 172.21.15.98:0/1997917496 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.172+0000 7f330b7fe640 1 --2- 172.21.15.98:0/1997917496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f330c104c40 0x7f330c198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.173+0000 7f3310842640 1 -- 172.21.15.98:0/1997917496 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f330c102310 msgr2=0x7f330c197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.173+0000 7f3310842640 1 --2- 172.21.15.98:0/1997917496 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f330c102310 0x7f330c197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.173+0000 7f3310842640 1 -- 172.21.15.98:0/1997917496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f330c104c40 msgr2=0x7f330c198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.173+0000 7f3310842640 1 --2- 172.21.15.98:0/1997917496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f330c104c40 0x7f330c198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.173+0000 7f3310842640 1 -- 172.21.15.98:0/1997917496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f330c19f830 con 0x7f330c107570 2024-05-09T08:50:04.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.173+0000 7f330b7fe640 1 --2- 172.21.15.98:0/1997917496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f330c104c40 0x7f330c198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:04.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.173+0000 7f3310842640 1 --2- 172.21.15.98:0/1997917496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f330c107570 0x7f330c19f190 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f330400cad0 tx=0x7f330400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.173+0000 7f33097fa640 1 -- 172.21.15.98:0/1997917496 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f330400ae20 con 0x7f330c107570 2024-05-09T08:50:04.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.173+0000 7f33097fa640 1 -- 172.21.15.98:0/1997917496 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3304005020 con 0x7f330c107570 2024-05-09T08:50:04.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.173+0000 7f33122a5640 1 -- 172.21.15.98:0/1997917496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f330c19fb20 con 0x7f330c107570 2024-05-09T08:50:04.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.173+0000 7f33097fa640 1 -- 172.21.15.98:0/1997917496 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f330400f6f0 con 0x7f330c107570 2024-05-09T08:50:04.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.173+0000 7f33122a5640 1 -- 172.21.15.98:0/1997917496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f330c1a0020 con 0x7f330c107570 2024-05-09T08:50:04.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.175+0000 7f33097fa640 1 -- 172.21.15.98:0/1997917496 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f33040051d0 con 0x7f330c107570 2024-05-09T08:50:04.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.175+0000 7f3314fb9740 1 Processor -- start 2024-05-09T08:50:04.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.175+0000 7f3314fb9740 1 -- start start 2024-05-09T08:50:04.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f3314fb9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6d78c0 0x56101a6d7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f3314fb9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6dcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f3314fb9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6dd420 0x56101a6df8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f3314fb9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56101a63b4d0 con 0x56101a6d82b0 2024-05-09T08:50:04.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f3314fb9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56101a5e33e0 con 0x56101a6d78c0 2024-05-09T08:50:04.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f3314fb9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56101a637490 con 0x56101a6dd420 2024-05-09T08:50:04.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f330bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6dcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f330bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6dcdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52406/0 (socket says 172.21.15.98:52406) 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f330bfff640 1 -- 172.21.15.98:0/4150561341 learned_addr learned my addr 172.21.15.98:0/4150561341 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f33097fa640 1 --2- 172.21.15.98:0/1997917496 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32f40a2b90 0x7f32f40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f330b7fe640 1 --2- 172.21.15.98:0/4150561341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6d78c0 0x56101a6d7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f330b7fe640 1 --2- 172.21.15.98:0/4150561341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6dd420 0x56101a6df8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f330bfff640 1 -- 172.21.15.98:0/4150561341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6dd420 msgr2=0x56101a6df8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f3310842640 1 --2- 172.21.15.98:0/1997917496 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32f40a2b90 0x7f32f40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f330bfff640 1 --2- 172.21.15.98:0/4150561341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6dd420 0x56101a6df8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f330bfff640 1 -- 172.21.15.98:0/4150561341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6d78c0 msgr2=0x56101a6d7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f330bfff640 1 --2- 172.21.15.98:0/4150561341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6d78c0 0x56101a6d7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f330bfff640 1 -- 172.21.15.98:0/4150561341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56101a34f2d0 con 0x56101a6d82b0 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f330b7fe640 1 --2- 172.21.15.98:0/4150561341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6dd420 0x56101a6df8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f33097fa640 1 -- 172.21.15.98:0/1997917496 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f33040cf4d0 con 0x7f330c107570 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f330bfff640 1 --2- 172.21.15.98:0/4150561341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6dcdb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0314e0 tx=0x7f32fc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99b1a26853a0e34 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f3310842640 1 --2- 172.21.15.98:0/1997917496 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32f40a2b90 0x7f32f40a5050 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f330409b6d0 tx=0x7f330406b7f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f32f27fc640 1 -- 172.21.15.98:0/4150561341 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc03e070 con 0x56101a6d82b0 2024-05-09T08:50:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f32f27fc640 1 -- 172.21.15.98:0/4150561341 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32fc0043c0 con 0x56101a6d82b0 2024-05-09T08:50:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.176+0000 7f32f27fc640 1 -- 172.21.15.98:0/4150561341 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc035040 con 0x56101a6d82b0 2024-05-09T08:50:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.177+0000 7f3314fb9740 1 -- 172.21.15.98:0/4150561341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 msgr2=0x56101a6dcdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.177+0000 7f3314fb9740 1 --2- 172.21.15.98:0/4150561341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6dcdb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0314e0 tx=0x7f32fc000bc0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.177+0000 7f3314fb9740 1 -- 172.21.15.98:0/4150561341 shutdown_connections 2024-05-09T08:50:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.177+0000 7f3314fb9740 1 --2- 172.21.15.98:0/4150561341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6dd420 0x56101a6df8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.177+0000 7f3314fb9740 1 --2- 172.21.15.98:0/4150561341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6dcdb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.177+0000 7f3314fb9740 1 --2- 172.21.15.98:0/4150561341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6d78c0 0x56101a6d7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.177+0000 7f3314fb9740 1 -- 172.21.15.98:0/4150561341 >> 172.21.15.98:0/4150561341 conn(0x56101a6d2f30 msgr2=0x56101a6d3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.177+0000 7f3314fb9740 1 -- 172.21.15.98:0/4150561341 shutdown_connections 2024-05-09T08:50:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.177+0000 7f3314fb9740 1 -- 172.21.15.98:0/4150561341 wait complete. 2024-05-09T08:50:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.177+0000 7f3314fb9740 1 -- start start 2024-05-09T08:50:04.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f3314fb9740 1 Processor -- start 2024-05-09T08:50:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f3314fb9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6d78c0 0x56101a6e1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f3314fb9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6e21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f3314fb9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6dd420 0x56101a6e91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f3314fb9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56101a6e0370 con 0x56101a6d82b0 2024-05-09T08:50:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f3314fb9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56101a5f68e0 con 0x56101a6dd420 2024-05-09T08:50:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f3314fb9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56101a6e04f0 con 0x56101a6d78c0 2024-05-09T08:50:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f3314fb9740 10 client.0 _ll_register_callbacks cb 0x56101a5c39c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f330bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6e21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f330bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6e21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52422/0 (socket says 172.21.15.98:52422) 2024-05-09T08:50:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f330bfff640 1 -- 172.21.15.98:0/2767642517 learned_addr learned my addr 172.21.15.98:0/2767642517 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f330b7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6d78c0 0x56101a6e1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f330b7fe640 1 --2- 172.21.15.98:0/2767642517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6dd420 0x56101a6e91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f330bfff640 1 -- 172.21.15.98:0/2767642517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6d78c0 msgr2=0x56101a6e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f330bfff640 1 --2- 172.21.15.98:0/2767642517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6d78c0 0x56101a6e1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f330bfff640 1 -- 172.21.15.98:0/2767642517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6dd420 msgr2=0x56101a6e91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f330bfff640 1 --2- 172.21.15.98:0/2767642517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6dd420 0x56101a6e91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f330bfff640 1 -- 172.21.15.98:0/2767642517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56101a4196b0 con 0x56101a6d82b0 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f330bfff640 1 --2- 172.21.15.98:0/2767642517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6e21c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f32fc009900 tx=0x7f32fc004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d448342e52d985 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.178+0000 7f32eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.179+0000 7f32eb7fe640 1 -- 172.21.15.98:0/2767642517 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc04b020 con 0x56101a6d82b0 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.179+0000 7f32eb7fe640 1 -- 172.21.15.98:0/2767642517 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f32fc005050 con 0x56101a6d82b0 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.179+0000 7f3314fb9740 1 -- 172.21.15.98:0/2767642517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32fc0043c0 con 0x56101a6d82b0 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.179+0000 7f32eb7fe640 1 -- 172.21.15.98:0/2767642517 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f32fc056050 con 0x56101a6d82b0 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.179+0000 7f3314fb9740 10 client.15297 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.179+0000 7f3314fb9740 1 -- 172.21.15.98:0/2767642517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56101a6ec130 con 0x56101a6d82b0 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.179+0000 7f32eb7fe640 1 -- 172.21.15.98:0/2767642517 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32fc0043c0 con 0x56101a6d82b0 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.179+0000 7f32eb7fe640 1 -- 172.21.15.98:0/2767642517 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f32fc0035f0 con 0x56101a6d82b0 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.179+0000 7f3314fb9740 10 client.15297 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.179+0000 7f3314fb9740 10 client.15297 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T08:50:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.179+0000 7f3314fb9740 20 client.15297 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.179+0000 7f3314fb9740 1 --2- 172.21.15.98:0/2767642517 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56101a6ee390 0x56101a6ef800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.179+0000 7f3314fb9740 4 client.15297 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:50:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.179+0000 7f3314fb9740 1 -- 172.21.15.98:0/2767642517 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56101a648c00 con 0x56101a6ee390 2024-05-09T08:50:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.180+0000 7f330b7fe640 1 --2- 172.21.15.98:0/2767642517 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56101a6ee390 0x56101a6ef800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.180+0000 7f32eb7fe640 10 client.15297 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:50:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.180+0000 7f330b7fe640 1 --2- 172.21.15.98:0/2767642517 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56101a6ee390 0x56101a6ef800 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c7768acf514f69ec server_cookie=b67f4b17e43dd29 in_seq=0 out_seq=0 2024-05-09T08:50:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f32eb7fe640 1 -- 172.21.15.98:0/2767642517 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56101a648c00 con 0x56101a6ee390 2024-05-09T08:50:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f32eb7fe640 10 client.15297 handle_command_reply: tid=0 2024-05-09T08:50:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f3314fb9740 1 client.15297 shutdown 2024-05-09T08:50:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f3314fb9740 1 -- 172.21.15.98:0/2767642517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 msgr2=0x56101a6e21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f3314fb9740 1 --2- 172.21.15.98:0/2767642517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6e21c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f32fc009900 tx=0x7f32fc004230 comp rx=0 tx=0).stop 2024-05-09T08:50:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f3314fb9740 1 -- 172.21.15.98:0/2767642517 shutdown_connections 2024-05-09T08:50:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f3314fb9740 1 --2- 172.21.15.98:0/2767642517 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56101a6ee390 0x56101a6ef800 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f3314fb9740 1 --2- 172.21.15.98:0/2767642517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6dd420 0x56101a6e91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f3314fb9740 1 --2- 172.21.15.98:0/2767642517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6e21c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f3314fb9740 1 --2- 172.21.15.98:0/2767642517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6d78c0 0x56101a6e1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f32eb7fe640 0 client.15297 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:50:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f3314fb9740 1 -- 172.21.15.98:0/2767642517 >> 172.21.15.98:0/2767642517 conn(0x56101a6d2f30 msgr2=0x56101a6d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f3314fb9740 1 -- 172.21.15.98:0/2767642517 shutdown_connections 2024-05-09T08:50:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f3314fb9740 1 -- 172.21.15.98:0/2767642517 wait complete. 2024-05-09T08:50:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.181+0000 7f3314fb9740 20 client.15297 trim_cache size 0 max 16384 2024-05-09T08:50:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.199+0000 7f3314fb9740 1 Processor -- start 2024-05-09T08:50:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.199+0000 7f3314fb9740 1 -- start start 2024-05-09T08:50:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.199+0000 7f3314fb9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6d78c0 0x56101a6df810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.199+0000 7f3314fb9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6e9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.199+0000 7f3314fb9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6dd420 0x56101a6dcc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.199+0000 7f3314fb9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56101a6edb40 con 0x56101a6d82b0 2024-05-09T08:50:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.199+0000 7f3314fb9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56101a6ee210 con 0x56101a6dd420 2024-05-09T08:50:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.199+0000 7f3314fb9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56101a6ee8e0 con 0x56101a6d78c0 2024-05-09T08:50:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.199+0000 7f330bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6e9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.199+0000 7f330bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6e9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52426/0 (socket says 172.21.15.98:52426) 2024-05-09T08:50:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.200+0000 7f330bfff640 1 -- 172.21.15.98:0/2673243036 learned_addr learned my addr 172.21.15.98:0/2673243036 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.200+0000 7f330bfff640 1 -- 172.21.15.98:0/2673243036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6d78c0 msgr2=0x56101a6df810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.200+0000 7f330bfff640 1 --2- 172.21.15.98:0/2673243036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6d78c0 0x56101a6df810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.200+0000 7f330bfff640 1 -- 172.21.15.98:0/2673243036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6dd420 msgr2=0x56101a6dcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.200+0000 7f330bfff640 1 --2- 172.21.15.98:0/2673243036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6dd420 0x56101a6dcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.200+0000 7f330bfff640 1 -- 172.21.15.98:0/2673243036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56101a6e95d0 con 0x56101a6d82b0 2024-05-09T08:50:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.200+0000 7f330bfff640 1 --2- 172.21.15.98:0/2673243036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6e9090 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f32fc004650 tx=0x7f32fc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a8278764d7d5a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.200+0000 7f32f1ffb640 1 -- 172.21.15.98:0/2673243036 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc04b070 con 0x56101a6d82b0 2024-05-09T08:50:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.200+0000 7f32f1ffb640 1 -- 172.21.15.98:0/2673243036 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32fc0035f0 con 0x56101a6d82b0 2024-05-09T08:50:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.200+0000 7f32f1ffb640 1 -- 172.21.15.98:0/2673243036 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc03e070 con 0x56101a6d82b0 2024-05-09T08:50:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 -- 172.21.15.98:0/2673243036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 msgr2=0x56101a6e9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 --2- 172.21.15.98:0/2673243036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6e9090 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f32fc004650 tx=0x7f32fc000bc0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 -- 172.21.15.98:0/2673243036 shutdown_connections 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 --2- 172.21.15.98:0/2673243036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6dd420 0x56101a6dcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 --2- 172.21.15.98:0/2673243036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6d82b0 0x56101a6e9090 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 --2- 172.21.15.98:0/2673243036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6d78c0 0x56101a6df810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 -- 172.21.15.98:0/2673243036 >> 172.21.15.98:0/2673243036 conn(0x56101a6d2f30 msgr2=0x56101a6d3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 -- 172.21.15.98:0/2673243036 shutdown_connections 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 -- 172.21.15.98:0/2673243036 wait complete. 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 -- start start 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 Processor -- start 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6d78c0 0x56101a6e4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6d82b0 0x56101a6e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6dd420 0x56101a6e7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56101a6ef680 con 0x56101a6dd420 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56101a6eefb0 con 0x56101a6d82b0 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56101a6efd50 con 0x56101a6d78c0 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f3314fb9740 10 client.0 _ll_register_callbacks cb 0x7f32fc005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f330b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6dd420 0x56101a6e7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f330b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6dd420 0x56101a6e7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52438/0 (socket says 172.21.15.98:52438) 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.201+0000 7f330b7fe640 1 -- 172.21.15.98:0/548881463 learned_addr learned my addr 172.21.15.98:0/548881463 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f330b7fe640 1 --2- 172.21.15.98:0/548881463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6d78c0 0x56101a6e4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f330bfff640 1 --2- 172.21.15.98:0/548881463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6d82b0 0x56101a6e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f330b7fe640 1 -- 172.21.15.98:0/548881463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6d78c0 msgr2=0x56101a6e4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f330b7fe640 1 --2- 172.21.15.98:0/548881463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6d78c0 0x56101a6e4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f330b7fe640 1 -- 172.21.15.98:0/548881463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6d82b0 msgr2=0x56101a6e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f330b7fe640 1 --2- 172.21.15.98:0/548881463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6d82b0 0x56101a6e4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f330b7fe640 1 -- 172.21.15.98:0/548881463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56101a6e8340 con 0x56101a6dd420 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f330b7fe640 1 --2- 172.21.15.98:0/548881463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6dd420 0x56101a6e7d50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x56101a6d8b10 tx=0x7f330000b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e3ddb73252e4d08 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f32eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f32eb7fe640 1 -- 172.21.15.98:0/548881463 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3300022790 con 0x56101a6dd420 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f32eb7fe640 1 -- 172.21.15.98:0/548881463 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f330001b040 con 0x56101a6dd420 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f3314fb9740 1 -- 172.21.15.98:0/548881463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32fc0035f0 con 0x56101a6dd420 2024-05-09T08:50:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f32eb7fe640 1 -- 172.21.15.98:0/548881463 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3300033050 con 0x56101a6dd420 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f3314fb9740 10 client.15303 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f3314fb9740 1 -- 172.21.15.98:0/548881463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56101a754470 con 0x56101a6dd420 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.202+0000 7f32eb7fe640 1 -- 172.21.15.98:0/548881463 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32fc0035f0 con 0x56101a6dd420 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.203+0000 7f32eb7fe640 1 -- 172.21.15.98:0/548881463 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3300011540 con 0x56101a6dd420 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.203+0000 7f3314fb9740 10 client.15303 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.203+0000 7f3314fb9740 10 client.15303 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.203+0000 7f3314fb9740 20 client.15303 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.203+0000 7f3314fb9740 1 --2- 172.21.15.98:0/548881463 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56101a759060 0x56101a7554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.203+0000 7f3314fb9740 4 client.15303 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.203+0000 7f3314fb9740 1 -- 172.21.15.98:0/548881463 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "status"}) -- 0x56101a642f50 con 0x56101a759060 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.203+0000 7f330bfff640 1 --2- 172.21.15.98:0/548881463 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56101a759060 0x56101a7554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.203+0000 7f32eb7fe640 10 client.15303 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.203+0000 7f330bfff640 1 --2- 172.21.15.98:0/548881463 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56101a759060 0x56101a7554a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5588b9a5b086fa6 server_cookie=bedb2882d1178fd4 in_seq=0 out_seq=0 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.203+0000 7f32eb7fe640 1 -- 172.21.15.98:0/548881463 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56101a642f50 con 0x56101a759060 2024-05-09T08:50:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.203+0000 7f32eb7fe640 10 client.15303 handle_command_reply: tid=0 2024-05-09T08:50:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f3314fb9740 1 client.15303 shutdown 2024-05-09T08:50:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f3314fb9740 1 -- 172.21.15.98:0/548881463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6dd420 msgr2=0x56101a6e7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f3314fb9740 1 --2- 172.21.15.98:0/548881463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6dd420 0x56101a6e7d50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x56101a6d8b10 tx=0x7f330000b2e0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f3314fb9740 1 -- 172.21.15.98:0/548881463 shutdown_connections 2024-05-09T08:50:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f3314fb9740 1 --2- 172.21.15.98:0/548881463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56101a6dd420 0x56101a6e7d50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f3314fb9740 1 --2- 172.21.15.98:0/548881463 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56101a759060 0x56101a7554a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f3314fb9740 1 --2- 172.21.15.98:0/548881463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56101a6d82b0 0x56101a6e4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f3314fb9740 1 --2- 172.21.15.98:0/548881463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56101a6d78c0 0x56101a6e4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f32eb7fe640 0 client.15303 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:50:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f3314fb9740 1 -- 172.21.15.98:0/548881463 >> 172.21.15.98:0/548881463 conn(0x56101a6d2f30 msgr2=0x56101a6d5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f3314fb9740 1 -- 172.21.15.98:0/548881463 shutdown_connections 2024-05-09T08:50:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f3314fb9740 1 -- 172.21.15.98:0/548881463 wait complete. 2024-05-09T08:50:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f3314fb9740 20 client.15303 trim_cache size 0 max 16384 2024-05-09T08:50:04.211 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:50:04.211 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a57d9712-0ddf-11ef-bc98-c7b262605968", 2024-05-09T08:50:04.211 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2024-05-09T08:50:04.211 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24457, 2024-05-09T08:50:04.211 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-05-09T08:50:04.211 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-05-09T08:50:04.211 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-05-09T08:50:04.211 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 87.935566976000004, 2024-05-09T08:50:04.211 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-05-09T08:50:04.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 92, 2024-05-09T08:50:04.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 90, 2024-05-09T08:50:04.211 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 120.77697212699999 2024-05-09T08:50:04.212 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:50:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f33122a5640 1 -- 172.21.15.98:0/1997917496 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32f40a2b90 msgr2=0x7f32f40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f33122a5640 1 --2- 172.21.15.98:0/1997917496 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32f40a2b90 0x7f32f40a5050 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f330409b6d0 tx=0x7f330406b7f0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f33122a5640 1 -- 172.21.15.98:0/1997917496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f330c107570 msgr2=0x7f330c19f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f33122a5640 1 --2- 172.21.15.98:0/1997917496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f330c107570 0x7f330c19f190 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f330400cad0 tx=0x7f330400cf90 comp rx=0 tx=0).stop 2024-05-09T08:50:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f33122a5640 1 -- 172.21.15.98:0/1997917496 shutdown_connections 2024-05-09T08:50:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f33122a5640 1 --2- 172.21.15.98:0/1997917496 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f32f40a2b90 0x7f32f40a5050 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f33122a5640 1 --2- 172.21.15.98:0/1997917496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f330c107570 0x7f330c19f190 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f33122a5640 1 --2- 172.21.15.98:0/1997917496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f330c104c40 0x7f330c198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f33122a5640 1 --2- 172.21.15.98:0/1997917496 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f330c102310 0x7f330c197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.204+0000 7f33122a5640 1 -- 172.21.15.98:0/1997917496 >> 172.21.15.98:0/1997917496 conn(0x7f330c0fbd20 msgr2=0x7f330c102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.205+0000 7f33122a5640 1 -- 172.21.15.98:0/1997917496 shutdown_connections 2024-05-09T08:50:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.205+0000 7f33122a5640 1 -- 172.21.15.98:0/1997917496 wait complete. 2024-05-09T08:50:04.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-05-09T08:50:04.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.320+0000 7f6e99ff9640 1 -- 172.21.15.98:0/1015150294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e94104c60 msgr2=0x7f6e94107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.320+0000 7f6e99ff9640 1 --2- 172.21.15.98:0/1015150294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e94104c60 0x7f6e94107050 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c009900 tx=0x7f6e7c02a410 comp rx=0 tx=0).stop 2024-05-09T08:50:04.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.320+0000 7f6e99ff9640 1 -- 172.21.15.98:0/1015150294 shutdown_connections 2024-05-09T08:50:04.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.320+0000 7f6e99ff9640 1 --2- 172.21.15.98:0/1015150294 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e94107590 0x7f6e941099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.320+0000 7f6e99ff9640 1 --2- 172.21.15.98:0/1015150294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e94104c60 0x7f6e94107050 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.320+0000 7f6e99ff9640 1 --2- 172.21.15.98:0/1015150294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e94102330 0x7f6e94104720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.320+0000 7f6e99ff9640 1 -- 172.21.15.98:0/1015150294 >> 172.21.15.98:0/1015150294 conn(0x7f6e940fbce0 msgr2=0x7f6e940fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.320+0000 7f6e99ff9640 1 -- 172.21.15.98:0/1015150294 shutdown_connections 2024-05-09T08:50:04.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.320+0000 7f6e99ff9640 1 -- 172.21.15.98:0/1015150294 wait complete. 2024-05-09T08:50:04.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.320+0000 7f6e99ff9640 1 Processor -- start 2024-05-09T08:50:04.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e99ff9640 1 -- start start 2024-05-09T08:50:04.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e99ff9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e94102330 0x7f6e94101ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e99ff9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e94104c60 0x7f6e941002e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e99ff9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e94107590 0x7f6e94100820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e99ff9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9410d4d0 con 0x7f6e94107590 2024-05-09T08:50:04.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e99ff9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9410d350 con 0x7f6e94102330 2024-05-09T08:50:04.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e99ff9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9410d650 con 0x7f6e94104c60 2024-05-09T08:50:04.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e93fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e94107590 0x7f6e94100820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e93fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e94107590 0x7f6e94100820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52464/0 (socket says 172.21.15.98:52464) 2024-05-09T08:50:04.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e93fff640 1 -- 172.21.15.98:0/3793652915 learned_addr learned my addr 172.21.15.98:0/3793652915 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e937fe640 1 --2- 172.21.15.98:0/3793652915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e94102330 0x7f6e94101ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e8bfff640 1 --2- 172.21.15.98:0/3793652915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e94104c60 0x7f6e941002e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e93fff640 1 -- 172.21.15.98:0/3793652915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e94104c60 msgr2=0x7f6e941002e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e93fff640 1 --2- 172.21.15.98:0/3793652915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e94104c60 0x7f6e941002e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e93fff640 1 -- 172.21.15.98:0/3793652915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e94102330 msgr2=0x7f6e94101ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e93fff640 1 --2- 172.21.15.98:0/3793652915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e94102330 0x7f6e94101ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e93fff640 1 -- 172.21.15.98:0/3793652915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e941010b0 con 0x7f6e94107590 2024-05-09T08:50:04.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e93fff640 1 --2- 172.21.15.98:0/3793652915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e94107590 0x7f6e94100820 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c00eaa0 tx=0x7f6e8c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e917fa640 1 -- 172.21.15.98:0/3793652915 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c00cdb0 con 0x7f6e94107590 2024-05-09T08:50:04.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e917fa640 1 -- 172.21.15.98:0/3793652915 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e8c005020 con 0x7f6e94107590 2024-05-09T08:50:04.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e99ff9640 1 -- 172.21.15.98:0/3793652915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e941a1d20 con 0x7f6e94107590 2024-05-09T08:50:04.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.321+0000 7f6e917fa640 1 -- 172.21.15.98:0/3793652915 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c0116b0 con 0x7f6e94107590 2024-05-09T08:50:04.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.322+0000 7f6e99ff9640 1 -- 172.21.15.98:0/3793652915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e941a2190 con 0x7f6e94107590 2024-05-09T08:50:04.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.324+0000 7f6e917fa640 1 -- 172.21.15.98:0/3793652915 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6e8c005560 con 0x7f6e94107590 2024-05-09T08:50:04.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.324+0000 7f6e9cde1740 1 Processor -- start 2024-05-09T08:50:04.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.324+0000 7f6e9cde1740 1 -- start start 2024-05-09T08:50:04.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.324+0000 7f6e9cde1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 0x560975025ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.324+0000 7f6e9cde1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x56097502adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.324+0000 7f6e9cde1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56097502b420 0x56097502d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.324+0000 7f6e9cde1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560974f894d0 con 0x5609750262b0 2024-05-09T08:50:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.324+0000 7f6e9cde1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560974f313e0 con 0x56097502b420 2024-05-09T08:50:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e9cde1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x560974f85490 con 0x5609750258c0 2024-05-09T08:50:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x56097502adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x56097502adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52478/0 (socket says 172.21.15.98:52478) 2024-05-09T08:50:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e937fe640 1 -- 172.21.15.98:0/3269517675 learned_addr learned my addr 172.21.15.98:0/3269517675 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e917fa640 1 --2- 172.21.15.98:0/3793652915 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6e800a2c10 0x7f6e800a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e8bfff640 1 --2- 172.21.15.98:0/3269517675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 0x560975025ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e937fe640 1 -- 172.21.15.98:0/3269517675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 msgr2=0x560975025ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e937fe640 1 --2- 172.21.15.98:0/3269517675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 0x560975025ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e937fe640 1 -- 172.21.15.98:0/3269517675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56097502b420 msgr2=0x56097502d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e937fe640 1 --2- 172.21.15.98:0/3269517675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56097502b420 0x56097502d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e93fff640 1 --2- 172.21.15.98:0/3793652915 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6e800a2c10 0x7f6e800a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e937fe640 1 -- 172.21.15.98:0/3269517675 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560974c9d2d0 con 0x5609750262b0 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e917fa640 1 -- 172.21.15.98:0/3793652915 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f6e8c0cf5c0 con 0x7f6e94107590 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e937fe640 1 --2- 172.21.15.98:0/3269517675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x56097502adb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6e8400bd10 tx=0x7f6e840107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31638fa8394b3485 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e93fff640 1 --2- 172.21.15.98:0/3793652915 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6e800a2c10 0x7f6e800a50d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c09baa0 tx=0x7f6e8c06ebd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e89ffb640 1 -- 172.21.15.98:0/3269517675 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e84004030 con 0x5609750262b0 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e89ffb640 1 -- 172.21.15.98:0/3269517675 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e840041d0 con 0x5609750262b0 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.325+0000 7f6e89ffb640 1 -- 172.21.15.98:0/3269517675 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e84011840 con 0x5609750262b0 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.326+0000 7f6e9cde1740 1 -- 172.21.15.98:0/3269517675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 msgr2=0x56097502adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.326+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/3269517675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x56097502adb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6e8400bd10 tx=0x7f6e840107a0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.326+0000 7f6e9cde1740 1 -- 172.21.15.98:0/3269517675 shutdown_connections 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.326+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/3269517675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56097502b420 0x56097502d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.326+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/3269517675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x56097502adb0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.326+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/3269517675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 0x560975025ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.326+0000 7f6e9cde1740 1 -- 172.21.15.98:0/3269517675 >> 172.21.15.98:0/3269517675 conn(0x560975020f30 msgr2=0x560975021330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.326+0000 7f6e9cde1740 1 -- 172.21.15.98:0/3269517675 shutdown_connections 2024-05-09T08:50:04.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.326+0000 7f6e9cde1740 1 -- 172.21.15.98:0/3269517675 wait complete. 2024-05-09T08:50:04.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.326+0000 7f6e9cde1740 1 -- start start 2024-05-09T08:50:04.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.326+0000 7f6e9cde1740 1 Processor -- start 2024-05-09T08:50:04.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e9cde1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750258c0 0x56097502fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e9cde1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5609750262b0 0x560975030270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e9cde1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56097502b420 0x560975037270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e9cde1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56097502e3d0 con 0x5609750258c0 2024-05-09T08:50:04.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e9cde1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560974f448e0 con 0x5609750262b0 2024-05-09T08:50:04.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e9cde1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56097502e550 con 0x56097502b420 2024-05-09T08:50:04.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e9cde1740 10 client.0 _ll_register_callbacks cb 0x560974f119c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:04.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e8bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750258c0 0x56097502fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e8bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750258c0 0x56097502fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52484/0 (socket says 172.21.15.98:52484) 2024-05-09T08:50:04.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e8bfff640 1 -- 172.21.15.98:0/2091466472 learned_addr learned my addr 172.21.15.98:0/2091466472 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e937fe640 1 --2- 172.21.15.98:0/2091466472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5609750262b0 0x560975030270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e8bfff640 1 --2- 172.21.15.98:0/2091466472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56097502b420 0x560975037270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e8bfff640 1 -- 172.21.15.98:0/2091466472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56097502b420 msgr2=0x560975037270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e8bfff640 1 --2- 172.21.15.98:0/2091466472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56097502b420 0x560975037270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e8bfff640 1 -- 172.21.15.98:0/2091466472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5609750262b0 msgr2=0x560975030270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e8bfff640 1 --2- 172.21.15.98:0/2091466472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5609750262b0 0x560975030270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e8bfff640 1 -- 172.21.15.98:0/2091466472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560974d676b0 con 0x5609750258c0 2024-05-09T08:50:04.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e937fe640 1 --2- 172.21.15.98:0/2091466472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5609750262b0 0x560975030270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:04.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e8bfff640 1 --2- 172.21.15.98:0/2091466472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750258c0 0x56097502fd30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c0314e0 tx=0x7f6e7c009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66aaf510a8d3f9d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:04.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e72ffd640 1 -- 172.21.15.98:0/2091466472 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c035040 con 0x5609750258c0 2024-05-09T08:50:04.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e72ffd640 1 -- 172.21.15.98:0/2091466472 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f6e7c004450 con 0x5609750258c0 2024-05-09T08:50:04.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.327+0000 7f6e9cde1740 1 -- 172.21.15.98:0/2091466472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e840041d0 con 0x5609750258c0 2024-05-09T08:50:04.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.328+0000 7f6e72ffd640 1 -- 172.21.15.98:0/2091466472 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6e7c050050 con 0x5609750258c0 2024-05-09T08:50:04.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.328+0000 7f6e9cde1740 10 client.15315 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:04.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.328+0000 7f6e9cde1740 1 -- 172.21.15.98:0/2091466472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56097503a2e0 con 0x5609750258c0 2024-05-09T08:50:04.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.328+0000 7f6e72ffd640 1 -- 172.21.15.98:0/2091466472 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e840041d0 con 0x5609750258c0 2024-05-09T08:50:04.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.328+0000 7f6e72ffd640 1 -- 172.21.15.98:0/2091466472 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6e7c049060 con 0x5609750258c0 2024-05-09T08:50:04.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.328+0000 7f6e9cde1740 10 client.15315 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:04.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.328+0000 7f6e9cde1740 10 client.15315 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-09T08:50:04.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.328+0000 7f6e9cde1740 20 client.15315 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:04.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.328+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/2091466472 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56097503c900 0x56097503dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.328+0000 7f6e9cde1740 4 client.15315 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:50:04.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.328+0000 7f6e9cde1740 1 -- 172.21.15.98:0/2091466472 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560974f96c00 con 0x56097503c900 2024-05-09T08:50:04.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.328+0000 7f6e937fe640 1 --2- 172.21.15.98:0/2091466472 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56097503c900 0x56097503dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.329+0000 7f6e937fe640 1 --2- 172.21.15.98:0/2091466472 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56097503c900 0x56097503dd70 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c50c45423d06d09 server_cookie=ffa5538c7f8ee47a in_seq=0 out_seq=0 2024-05-09T08:50:04.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.329+0000 7f6e72ffd640 10 client.15315 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.329+0000 7f6e72ffd640 1 -- 172.21.15.98:0/2091466472 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560974f96c00 con 0x56097503c900 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.329+0000 7f6e72ffd640 10 client.15315 handle_command_reply: tid=0 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.329+0000 7f6e9cde1740 1 client.15315 shutdown 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.329+0000 7f6e9cde1740 1 -- 172.21.15.98:0/2091466472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750258c0 msgr2=0x56097502fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.329+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/2091466472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750258c0 0x56097502fd30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c0314e0 tx=0x7f6e7c009520 comp rx=0 tx=0).stop 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.330+0000 7f6e9cde1740 1 -- 172.21.15.98:0/2091466472 shutdown_connections 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.330+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/2091466472 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56097503c900 0x56097503dd70 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.330+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/2091466472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56097502b420 0x560975037270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.330+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/2091466472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5609750262b0 0x560975030270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.330+0000 7f6e72ffd640 0 client.15315 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.330+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/2091466472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750258c0 0x56097502fd30 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.330+0000 7f6e9cde1740 1 -- 172.21.15.98:0/2091466472 >> 172.21.15.98:0/2091466472 conn(0x560975020f30 msgr2=0x560975020000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.330+0000 7f6e9cde1740 1 -- 172.21.15.98:0/2091466472 shutdown_connections 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.330+0000 7f6e9cde1740 1 -- 172.21.15.98:0/2091466472 wait complete. 2024-05-09T08:50:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.330+0000 7f6e9cde1740 20 client.15315 trim_cache size 0 max 16384 2024-05-09T08:50:04.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.347+0000 7f6e9cde1740 1 Processor -- start 2024-05-09T08:50:04.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e9cde1740 1 -- start start 2024-05-09T08:50:04.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e9cde1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 0x56097502d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e9cde1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x560975037230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e9cde1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56097502b420 0x560975023ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e9cde1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56097503c0b0 con 0x5609750262b0 2024-05-09T08:50:04.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e9cde1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56097503c780 con 0x56097502b420 2024-05-09T08:50:04.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e9cde1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56097503ce50 con 0x5609750258c0 2024-05-09T08:50:04.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x560975037230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x560975037230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52494/0 (socket says 172.21.15.98:52494) 2024-05-09T08:50:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e937fe640 1 -- 172.21.15.98:0/1499470382 learned_addr learned my addr 172.21.15.98:0/1499470382 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e8bfff640 1 --2- 172.21.15.98:0/1499470382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 0x56097502d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e937fe640 1 -- 172.21.15.98:0/1499470382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 msgr2=0x56097502d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e937fe640 1 --2- 172.21.15.98:0/1499470382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 0x56097502d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e937fe640 1 -- 172.21.15.98:0/1499470382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56097502b420 msgr2=0x560975023ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e937fe640 1 --2- 172.21.15.98:0/1499470382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56097502b420 0x560975023ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e937fe640 1 -- 172.21.15.98:0/1499470382 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560975020a70 con 0x5609750262b0 2024-05-09T08:50:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e937fe640 1 --2- 172.21.15.98:0/1499470382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x560975037230 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x560975026d20 tx=0x7f6e84005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a6406f83095a1d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e897fa640 1 -- 172.21.15.98:0/1499470382 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8402e020 con 0x5609750262b0 2024-05-09T08:50:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e897fa640 1 -- 172.21.15.98:0/1499470382 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e84003ad0 con 0x5609750262b0 2024-05-09T08:50:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.348+0000 7f6e897fa640 1 -- 172.21.15.98:0/1499470382 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8402a3c0 con 0x5609750262b0 2024-05-09T08:50:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.349+0000 7f6e9cde1740 1 -- 172.21.15.98:0/1499470382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 msgr2=0x560975037230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.349+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/1499470382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x560975037230 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x560975026d20 tx=0x7f6e84005cb0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.349+0000 7f6e9cde1740 1 -- 172.21.15.98:0/1499470382 shutdown_connections 2024-05-09T08:50:04.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.349+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/1499470382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56097502b420 0x560975023ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.349+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/1499470382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x560975037230 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.349+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/1499470382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 0x56097502d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.349+0000 7f6e9cde1740 1 -- 172.21.15.98:0/1499470382 >> 172.21.15.98:0/1499470382 conn(0x560975020f30 msgr2=0x5609750276b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.349+0000 7f6e9cde1740 1 -- 172.21.15.98:0/1499470382 shutdown_connections 2024-05-09T08:50:04.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.349+0000 7f6e9cde1740 1 -- 172.21.15.98:0/1499470382 wait complete. 2024-05-09T08:50:04.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.349+0000 7f6e9cde1740 1 -- start start 2024-05-09T08:50:04.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.349+0000 7f6e9cde1740 1 Processor -- start 2024-05-09T08:50:04.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e9cde1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 0x560975032df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e9cde1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x560975033330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e9cde1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56097502b420 0x560975036350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e9cde1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56097503dbf0 con 0x5609750262b0 2024-05-09T08:50:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e9cde1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56097503d520 con 0x56097502b420 2024-05-09T08:50:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e9cde1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56097503e2c0 con 0x5609750258c0 2024-05-09T08:50:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e9cde1740 10 client.0 _ll_register_callbacks cb 0x7f6e7c005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x560975033330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x560975033330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52496/0 (socket says 172.21.15.98:52496) 2024-05-09T08:50:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e937fe640 1 -- 172.21.15.98:0/1292374801 learned_addr learned my addr 172.21.15.98:0/1292374801 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e8bfff640 1 --2- 172.21.15.98:0/1292374801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 0x560975032df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e937fe640 1 -- 172.21.15.98:0/1292374801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 msgr2=0x560975032df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e937fe640 1 --2- 172.21.15.98:0/1292374801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 0x560975032df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e937fe640 1 -- 172.21.15.98:0/1292374801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56097502b420 msgr2=0x560975036350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e937fe640 1 --2- 172.21.15.98:0/1292374801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56097502b420 0x560975036350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e937fe640 1 -- 172.21.15.98:0/1292374801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560975036940 con 0x5609750262b0 2024-05-09T08:50:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e937fe640 1 --2- 172.21.15.98:0/1292374801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x560975033330 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6e84011010 tx=0x7f6e840053b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61927602270676 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e72ffd640 1 -- 172.21.15.98:0/1292374801 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e84003e00 con 0x5609750262b0 2024-05-09T08:50:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e72ffd640 1 -- 172.21.15.98:0/1292374801 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f6e840035f0 con 0x5609750262b0 2024-05-09T08:50:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e9cde1740 1 -- 172.21.15.98:0/1292374801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e84003ad0 con 0x5609750262b0 2024-05-09T08:50:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e72ffd640 1 -- 172.21.15.98:0/1292374801 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6e84003830 con 0x5609750262b0 2024-05-09T08:50:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e9cde1740 10 client.15321 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.350+0000 7f6e9cde1740 1 -- 172.21.15.98:0/1292374801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5609750a2ce0 con 0x5609750262b0 2024-05-09T08:50:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.351+0000 7f6e72ffd640 1 -- 172.21.15.98:0/1292374801 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e84003ad0 con 0x5609750262b0 2024-05-09T08:50:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.351+0000 7f6e72ffd640 1 -- 172.21.15.98:0/1292374801 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6e84011560 con 0x5609750262b0 2024-05-09T08:50:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.351+0000 7f6e9cde1740 10 client.15321 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.351+0000 7f6e9cde1740 10 client.15321 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-09T08:50:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.351+0000 7f6e9cde1740 20 client.15321 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.351+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/1292374801 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5609750a71d0 0x5609750a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.351+0000 7f6e9cde1740 4 client.15321 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:50:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.351+0000 7f6e9cde1740 1 -- 172.21.15.98:0/1292374801 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "status"}) -- 0x560974f90f50 con 0x5609750a71d0 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.351+0000 7f6e8bfff640 1 --2- 172.21.15.98:0/1292374801 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5609750a71d0 0x5609750a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e8bfff640 1 --2- 172.21.15.98:0/1292374801 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5609750a71d0 0x5609750a5ee0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8356bb76d9022604 server_cookie=4fd6ee54101738c6 in_seq=0 out_seq=0 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e72ffd640 10 client.15321 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e72ffd640 1 -- 172.21.15.98:0/1292374801 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x560974f90f50 con 0x5609750a71d0 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e72ffd640 10 client.15321 handle_command_reply: tid=0 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e9cde1740 1 client.15321 shutdown 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e9cde1740 1 -- 172.21.15.98:0/1292374801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 msgr2=0x560975033330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/1292374801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x560975033330 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6e84011010 tx=0x7f6e840053b0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e9cde1740 1 -- 172.21.15.98:0/1292374801 shutdown_connections 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/1292374801 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5609750a71d0 0x5609750a5ee0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/1292374801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56097502b420 0x560975036350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/1292374801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5609750262b0 0x560975033330 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e9cde1740 1 --2- 172.21.15.98:0/1292374801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5609750258c0 0x560975032df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e72ffd640 0 client.15321 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e9cde1740 1 -- 172.21.15.98:0/1292374801 >> 172.21.15.98:0/1292374801 conn(0x560975020f30 msgr2=0x560975023ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e9cde1740 1 -- 172.21.15.98:0/1292374801 shutdown_connections 2024-05-09T08:50:04.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e9cde1740 1 -- 172.21.15.98:0/1292374801 wait complete. 2024-05-09T08:50:04.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e9cde1740 20 client.15321 trim_cache size 0 max 16384 2024-05-09T08:50:04.359 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:50:04.359 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a57d9712-0ddf-11ef-bc98-c7b262605968", 2024-05-09T08:50:04.359 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-05-09T08:50:04.359 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24461, 2024-05-09T08:50:04.359 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-05-09T08:50:04.359 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-05-09T08:50:04.360 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-05-09T08:50:04.360 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 105.625329381, 2024-05-09T08:50:04.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-05-09T08:50:04.360 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 92, 2024-05-09T08:50:04.360 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 90, 2024-05-09T08:50:04.360 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 123.709221927 2024-05-09T08:50:04.360 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:50:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e99ff9640 1 -- 172.21.15.98:0/3793652915 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6e800a2c10 msgr2=0x7f6e800a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.352+0000 7f6e99ff9640 1 --2- 172.21.15.98:0/3793652915 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6e800a2c10 0x7f6e800a50d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c09baa0 tx=0x7f6e8c06ebd0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.353+0000 7f6e99ff9640 1 -- 172.21.15.98:0/3793652915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e94107590 msgr2=0x7f6e94100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.353+0000 7f6e99ff9640 1 --2- 172.21.15.98:0/3793652915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e94107590 0x7f6e94100820 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c00eaa0 tx=0x7f6e8c00ef60 comp rx=0 tx=0).stop 2024-05-09T08:50:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.353+0000 7f6e99ff9640 1 -- 172.21.15.98:0/3793652915 shutdown_connections 2024-05-09T08:50:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.353+0000 7f6e99ff9640 1 --2- 172.21.15.98:0/3793652915 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f6e800a2c10 0x7f6e800a50d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.353+0000 7f6e99ff9640 1 --2- 172.21.15.98:0/3793652915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e94107590 0x7f6e94100820 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.353+0000 7f6e99ff9640 1 --2- 172.21.15.98:0/3793652915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e94104c60 0x7f6e941002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.353+0000 7f6e99ff9640 1 --2- 172.21.15.98:0/3793652915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e94102330 0x7f6e94101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.353+0000 7f6e99ff9640 1 -- 172.21.15.98:0/3793652915 >> 172.21.15.98:0/3793652915 conn(0x7f6e940fbce0 msgr2=0x7f6e94102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.353+0000 7f6e99ff9640 1 -- 172.21.15.98:0/3793652915 shutdown_connections 2024-05-09T08:50:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:04.353+0000 7f6e99ff9640 1 -- 172.21.15.98:0/3793652915 wait complete. 2024-05-09T08:50:04.363 INFO:tasks.cephadm:Setup complete, yielding 2024-05-09T08:50:04.363 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-09T08:50:04.375 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-05-09T08:50:04.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- bash -c 'ceph orch status' 2024-05-09T08:50:04.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:04 smithi098 ceph-mon[30092]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-09T08:50:04.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:04 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1245150341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:50:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:04 smithi142 ceph-mon[34625]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-09T08:50:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:04 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1245150341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:04.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:04 smithi105 ceph-mon[34498]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-09T08:50:04.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:04 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1245150341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:06.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.443+0000 7fe47ca60700 1 -- 172.21.15.98:0/1114969365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe478101280 msgr2=0x7fe4781016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:06.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.443+0000 7fe47ca60700 1 --2- 172.21.15.98:0/1114969365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe478101280 0x7fe4781016a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe4680099c0 tx=0x7fe468009cd0 comp rx=0 tx=0).stop 2024-05-09T08:50:06.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.443+0000 7fe47ca60700 1 -- 172.21.15.98:0/1114969365 shutdown_connections 2024-05-09T08:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.443+0000 7fe47ca60700 1 --2- 172.21.15.98:0/1114969365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe478102e20 0x7fe4781096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.443+0000 7fe47ca60700 1 --2- 172.21.15.98:0/1114969365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe478101280 0x7fe4781016a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.443+0000 7fe47ca60700 1 --2- 172.21.15.98:0/1114969365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe478102480 0x7fe4781028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.443+0000 7fe47ca60700 1 -- 172.21.15.98:0/1114969365 >> 172.21.15.98:0/1114969365 conn(0x7fe4780fc820 msgr2=0x7fe4780fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.443+0000 7fe47ca60700 1 -- 172.21.15.98:0/1114969365 shutdown_connections 2024-05-09T08:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.443+0000 7fe47ca60700 1 -- 172.21.15.98:0/1114969365 wait complete. 2024-05-09T08:50:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe47ca60700 1 Processor -- start 2024-05-09T08:50:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe47ca60700 1 -- start start 2024-05-09T08:50:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe47ca60700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe478101280 0x7fe478197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe47ca60700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe478102480 0x7fe478197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe47ca60700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe478102e20 0x7fe47819eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe47ca60700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe47810cec0 con 0x7fe478102480 2024-05-09T08:50:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe47ca60700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe47810cd40 con 0x7fe478101280 2024-05-09T08:50:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe47ca60700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe47810d040 con 0x7fe478102e20 2024-05-09T08:50:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe475d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe478102480 0x7fe478197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe475d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe478102480 0x7fe478197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49716/0 (socket says 172.21.15.98:49716) 2024-05-09T08:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe475d9b700 1 -- 172.21.15.98:0/3808014788 learned_addr learned my addr 172.21.15.98:0/3808014788 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe476d9d700 1 --2- 172.21.15.98:0/3808014788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe478102e20 0x7fe47819eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe47659c700 1 --2- 172.21.15.98:0/3808014788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe478101280 0x7fe478197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe475d9b700 1 -- 172.21.15.98:0/3808014788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe478102e20 msgr2=0x7fe47819eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe475d9b700 1 --2- 172.21.15.98:0/3808014788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe478102e20 0x7fe47819eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe475d9b700 1 -- 172.21.15.98:0/3808014788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe478101280 msgr2=0x7fe478197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe475d9b700 1 --2- 172.21.15.98:0/3808014788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe478101280 0x7fe478197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.444+0000 7fe475d9b700 1 -- 172.21.15.98:0/3808014788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe47819f1c0 con 0x7fe478102480 2024-05-09T08:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.445+0000 7fe475d9b700 1 --2- 172.21.15.98:0/3808014788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe478102480 0x7fe478197aa0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe46c007b80 tx=0x7fe46c007e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.445+0000 7fe4677fe700 1 -- 172.21.15.98:0/3808014788 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c019020 con 0x7fe478102480 2024-05-09T08:50:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.445+0000 7fe4677fe700 1 -- 172.21.15.98:0/3808014788 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe46c004d10 con 0x7fe478102480 2024-05-09T08:50:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.445+0000 7fe47ca60700 1 -- 172.21.15.98:0/3808014788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe47806a060 con 0x7fe478102480 2024-05-09T08:50:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.445+0000 7fe4677fe700 1 -- 172.21.15.98:0/3808014788 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c00e540 con 0x7fe478102480 2024-05-09T08:50:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.445+0000 7fe47ca60700 1 -- 172.21.15.98:0/3808014788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe47806a620 con 0x7fe478102480 2024-05-09T08:50:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.446+0000 7fe47ca60700 1 -- 172.21.15.98:0/3808014788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe478069010 con 0x7fe478102480 2024-05-09T08:50:06.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.448+0000 7fe4677fe700 1 -- 172.21.15.98:0/3808014788 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe46c004ec0 con 0x7fe478102480 2024-05-09T08:50:06.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.448+0000 7fe4677fe700 1 --2- 172.21.15.98:0/3808014788 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe4600a2c20 0x7fe4600a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:06.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.448+0000 7fe47659c700 1 --2- 172.21.15.98:0/3808014788 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe4600a2c20 0x7fe4600a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:06.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.449+0000 7fe4677fe700 1 -- 172.21.15.98:0/3808014788 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fe46c0d0d70 con 0x7fe478102480 2024-05-09T08:50:06.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.449+0000 7fe47659c700 1 --2- 172.21.15.98:0/3808014788 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe4600a2c20 0x7fe4600a50e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe468000c00 tx=0x7fe468005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:06.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.452+0000 7fe4677fe700 1 -- 172.21.15.98:0/3808014788 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe46c023020 con 0x7fe478102480 2024-05-09T08:50:06.452 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:06 smithi098 ceph-mon[30092]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-09T08:50:06.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:06 smithi142 ceph-mon[34625]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-09T08:50:06.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.664+0000 7fe47ca60700 1 -- 172.21.15.98:0/3808014788 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fe47806a880 con 0x7fe4600a2c20 2024-05-09T08:50:06.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.665+0000 7fe4677fe700 1 -- 172.21.15.98:0/3808014788 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fe47806a880 con 0x7fe4600a2c20 2024-05-09T08:50:06.665 INFO:teuthology.orchestra.run.smithi098.stdout:Backend: cephadm 2024-05-09T08:50:06.665 INFO:teuthology.orchestra.run.smithi098.stdout:Available: Yes 2024-05-09T08:50:06.665 INFO:teuthology.orchestra.run.smithi098.stdout:Paused: No 2024-05-09T08:50:06.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.667+0000 7fe47ca60700 1 -- 172.21.15.98:0/3808014788 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe4600a2c20 msgr2=0x7fe4600a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.667+0000 7fe47ca60700 1 --2- 172.21.15.98:0/3808014788 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe4600a2c20 0x7fe4600a50e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe468000c00 tx=0x7fe468005fb0 comp rx=0 tx=0).stop 2024-05-09T08:50:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.667+0000 7fe47ca60700 1 -- 172.21.15.98:0/3808014788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe478102480 msgr2=0x7fe478197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.667+0000 7fe47ca60700 1 --2- 172.21.15.98:0/3808014788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe478102480 0x7fe478197aa0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe46c007b80 tx=0x7fe46c007e90 comp rx=0 tx=0).stop 2024-05-09T08:50:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.667+0000 7fe47ca60700 1 -- 172.21.15.98:0/3808014788 shutdown_connections 2024-05-09T08:50:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.667+0000 7fe47ca60700 1 --2- 172.21.15.98:0/3808014788 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe4600a2c20 0x7fe4600a50e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.667+0000 7fe47ca60700 1 --2- 172.21.15.98:0/3808014788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe478101280 0x7fe478197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.667+0000 7fe47ca60700 1 --2- 172.21.15.98:0/3808014788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe478102e20 0x7fe47819eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.667+0000 7fe47ca60700 1 --2- 172.21.15.98:0/3808014788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe478102480 0x7fe478197aa0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.667+0000 7fe47ca60700 1 -- 172.21.15.98:0/3808014788 >> 172.21.15.98:0/3808014788 conn(0x7fe4780fc820 msgr2=0x7fe478105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.668+0000 7fe47ca60700 1 -- 172.21.15.98:0/3808014788 shutdown_connections 2024-05-09T08:50:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:06.668+0000 7fe47ca60700 1 -- 172.21.15.98:0/3808014788 wait complete. 2024-05-09T08:50:06.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:06 smithi105 ceph-mon[34498]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-09T08:50:07.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-09T08:50:07.743 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:50:08.583 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:08 smithi098 ceph-mon[30092]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:08.583 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:08 smithi098 ceph-mon[30092]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:50:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:08 smithi142 ceph-mon[34625]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:08 smithi142 ceph-mon[34625]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:50:08.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:08 smithi105 ceph-mon[34498]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:08.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:08 smithi105 ceph-mon[34498]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:50:09.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.576+0000 7fc8daba1700 1 -- 172.21.15.98:0/235280494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d41024a0 msgr2=0x7fc8d4102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:09.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.576+0000 7fc8daba1700 1 --2- 172.21.15.98:0/235280494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d41024a0 0x7fc8d4102900 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc8c00099c0 tx=0x7fc8c0009cd0 comp rx=0 tx=0).stop 2024-05-09T08:50:09.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.576+0000 7fc8daba1700 1 -- 172.21.15.98:0/235280494 shutdown_connections 2024-05-09T08:50:09.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.576+0000 7fc8daba1700 1 --2- 172.21.15.98:0/235280494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d4102e40 0x7fc8d41096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:09.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.576+0000 7fc8daba1700 1 --2- 172.21.15.98:0/235280494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc8d41012a0 0x7fc8d41016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:09.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.576+0000 7fc8daba1700 1 --2- 172.21.15.98:0/235280494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d41024a0 0x7fc8d4102900 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:09.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.576+0000 7fc8daba1700 1 -- 172.21.15.98:0/235280494 >> 172.21.15.98:0/235280494 conn(0x7fc8d40fc880 msgr2=0x7fc8d40feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:09.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.576+0000 7fc8daba1700 1 -- 172.21.15.98:0/235280494 shutdown_connections 2024-05-09T08:50:09.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.576+0000 7fc8daba1700 1 -- 172.21.15.98:0/235280494 wait complete. 2024-05-09T08:50:09.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8daba1700 1 Processor -- start 2024-05-09T08:50:09.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8daba1700 1 -- start start 2024-05-09T08:50:09.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8daba1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d41012a0 0x7fc8d406b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:09.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8daba1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d41024a0 0x7fc8d419d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:09.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8daba1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc8d4102e40 0x7fc8d419fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:09.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8daba1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d410cfd0 con 0x7fc8d41024a0 2024-05-09T08:50:09.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8daba1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d410ce50 con 0x7fc8d41012a0 2024-05-09T08:50:09.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8daba1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d410d150 con 0x7fc8d4102e40 2024-05-09T08:50:09.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8d3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d41024a0 0x7fc8d419d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:09.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8d3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d41024a0 0x7fc8d419d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49730/0 (socket says 172.21.15.98:49730) 2024-05-09T08:50:09.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8d3fff700 1 -- 172.21.15.98:0/1925438292 learned_addr learned my addr 172.21.15.98:0/1925438292 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:09.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8d893d700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d41012a0 0x7fc8d406b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:09.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8d3fff700 1 -- 172.21.15.98:0/1925438292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc8d4102e40 msgr2=0x7fc8d419fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:09.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8d913e700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc8d4102e40 0x7fc8d419fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:09.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8d3fff700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc8d4102e40 0x7fc8d419fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:09.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8d3fff700 1 -- 172.21.15.98:0/1925438292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d41012a0 msgr2=0x7fc8d406b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:09.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8d3fff700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d41012a0 0x7fc8d406b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:09.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8d3fff700 1 -- 172.21.15.98:0/1925438292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d41a03d0 con 0x7fc8d41024a0 2024-05-09T08:50:09.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.577+0000 7fc8d893d700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d41012a0 0x7fc8d406b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:09.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.578+0000 7fc8d3fff700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d41024a0 0x7fc8d419d470 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000c010 tx=0x7fc8c00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:09.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.578+0000 7fc8d1ffb700 1 -- 172.21.15.98:0/1925438292 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0027070 con 0x7fc8d41024a0 2024-05-09T08:50:09.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.578+0000 7fc8d1ffb700 1 -- 172.21.15.98:0/1925438292 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8c0022470 con 0x7fc8d41024a0 2024-05-09T08:50:09.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.578+0000 7fc8daba1700 1 -- 172.21.15.98:0/1925438292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d41a0660 con 0x7fc8d41024a0 2024-05-09T08:50:09.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.578+0000 7fc8d1ffb700 1 -- 172.21.15.98:0/1925438292 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c000f6a0 con 0x7fc8d41024a0 2024-05-09T08:50:09.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.578+0000 7fc8daba1700 1 -- 172.21.15.98:0/1925438292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d41a0b30 con 0x7fc8d41024a0 2024-05-09T08:50:09.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.579+0000 7fc8daba1700 1 -- 172.21.15.98:0/1925438292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d4069010 con 0x7fc8d41024a0 2024-05-09T08:50:09.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.581+0000 7fc8d1ffb700 1 -- 172.21.15.98:0/1925438292 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc8c002a070 con 0x7fc8d41024a0 2024-05-09T08:50:09.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.582+0000 7fc8d1ffb700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc8c40a2c20 0x7fc8c40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:09.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.582+0000 7fc8d893d700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc8c40a2c20 0x7fc8c40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:09.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.582+0000 7fc8d1ffb700 1 -- 172.21.15.98:0/1925438292 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fc8c00d0810 con 0x7fc8d41024a0 2024-05-09T08:50:09.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.582+0000 7fc8d893d700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc8c40a2c20 0x7fc8c40a50e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4102300 tx=0x7fc8c8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:09.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.587+0000 7fc8d1ffb700 1 -- 172.21.15.98:0/1925438292 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc8c0013070 con 0x7fc8d41024a0 2024-05-09T08:50:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.793+0000 7fc8daba1700 1 -- 172.21.15.98:0/1925438292 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fc8d4062760 con 0x7fc8c40a2c20 2024-05-09T08:50:09.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.806+0000 7fc8d1ffb700 1 -- 172.21.15.98:0/1925438292 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7fc8d4062760 con 0x7fc8c40a2c20 2024-05-09T08:50:09.806 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-09T08:50:09.806 INFO:teuthology.orchestra.run.smithi098.stdout:mds.a smithi098 running (2m) 21s ago 2m 17.6M - 19.0.0-2515-g96805dcf d7acac7d5657 b659e2da800b 2024-05-09T08:50:09.806 INFO:teuthology.orchestra.run.smithi098.stdout:mds.b smithi105 running (118s) 51s ago 117s 19.2M - 19.0.0-2515-g96805dcf d7acac7d5657 1a83673e3f33 2024-05-09T08:50:09.806 INFO:teuthology.orchestra.run.smithi098.stdout:mds.c smithi142 running (2m) 51s ago 2m 17.3M - 19.0.0-2515-g96805dcf d7acac7d5657 a83c33cdbf6a 2024-05-09T08:50:09.806 INFO:teuthology.orchestra.run.smithi098.stdout:mds.d smithi098 running (2m) 21s ago 2m 16.1M - 19.0.0-2515-g96805dcf d7acac7d5657 c7c51b9ae468 2024-05-09T08:50:09.806 INFO:teuthology.orchestra.run.smithi098.stdout:mds.e smithi105 running (2m) 51s ago 2m 16.6M - 19.0.0-2515-g96805dcf d7acac7d5657 ce228e5ed330 2024-05-09T08:50:09.806 INFO:teuthology.orchestra.run.smithi098.stdout:mds.f smithi142 running (2m) 51s ago 2m 16.1M - 19.0.0-2515-g96805dcf d7acac7d5657 ea75d301ead0 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:mds.g smithi098 running (2m) 21s ago 2m 15.3M - 19.0.0-2515-g96805dcf d7acac7d5657 a66220490dfc 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:mds.h smithi105 running (2m) 51s ago 2m 25.8M - 19.0.0-2515-g96805dcf d7acac7d5657 d2eef274aff1 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:mds.i smithi142 running (2m) 51s ago 2m 20.2M - 19.0.0-2515-g96805dcf d7acac7d5657 8d37654b0fb0 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:mds.j smithi098 running (2m) 21s ago 2m 14.0M - 19.0.0-2515-g96805dcf d7acac7d5657 ecf3da05d375 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:mds.k smithi105 running (2m) 51s ago 2m 18.7M - 19.0.0-2515-g96805dcf d7acac7d5657 39d1fef78d28 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:mds.l smithi142 running (2m) 51s ago 2m 20.0M - 19.0.0-2515-g96805dcf d7acac7d5657 7a77746dd9b0 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi098 *:9283,8765 running (9m) 21s ago 9m 515M - 19.0.0-2515-g96805dcf d7acac7d5657 2da87a81222b 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi105 *:8443,8765 running (7m) 51s ago 7m 452M - 19.0.0-2515-g96805dcf d7acac7d5657 0ee24e7f39c9 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.z smithi142 *:8443,8765 running (7m) 51s ago 7m 454M - 19.0.0-2515-g96805dcf d7acac7d5657 725df350de80 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (9m) 21s ago 9m 51.3M 2048M 19.0.0-2515-g96805dcf d7acac7d5657 3ed83f274f2c 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi105 running (7m) 51s ago 7m 39.3M 2048M 19.0.0-2515-g96805dcf d7acac7d5657 085dc4d67b8c 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi142 running (8m) 51s ago 8m 39.5M 2048M 19.0.0-2515-g96805dcf d7acac7d5657 50e9afbdc9d3 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (7m) 21s ago 7m 46.2M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 62481a5cd800 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi105 running (7m) 51s ago 7m 57.5M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 e326c4105a1d 2024-05-09T08:50:09.807 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi142 running (6m) 51s ago 6m 47.6M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 e7ad21352328 2024-05-09T08:50:09.808 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (6m) 21s ago 6m 57.8M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 b1e2bbd93c77 2024-05-09T08:50:09.808 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi105 running (5m) 51s ago 5m 42.2M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 6a089ee8b997 2024-05-09T08:50:09.808 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi142 running (5m) 51s ago 5m 55.7M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 0d7e11883358 2024-05-09T08:50:09.808 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi098 running (4m) 21s ago 4m 42.7M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 8fe42d775f1c 2024-05-09T08:50:09.808 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi105 running (4m) 51s ago 4m 44.6M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 8be93cb580d5 2024-05-09T08:50:09.808 INFO:teuthology.orchestra.run.smithi098.stdout:osd.8 smithi142 running (4m) 51s ago 4m 42.0M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 9916b511a568 2024-05-09T08:50:09.808 INFO:teuthology.orchestra.run.smithi098.stdout:osd.9 smithi098 running (3m) 21s ago 3m 43.8M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 d74100668436 2024-05-09T08:50:09.808 INFO:teuthology.orchestra.run.smithi098.stdout:osd.10 smithi105 running (3m) 51s ago 3m 42.1M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 d46657d58b96 2024-05-09T08:50:09.808 INFO:teuthology.orchestra.run.smithi098.stdout:osd.11 smithi142 running (2m) 51s ago 2m 44.0M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 71ca3b94aed1 2024-05-09T08:50:09.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.809+0000 7fc8daba1700 1 -- 172.21.15.98:0/1925438292 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc8c40a2c20 msgr2=0x7fc8c40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:09.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.809+0000 7fc8daba1700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc8c40a2c20 0x7fc8c40a50e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4102300 tx=0x7fc8c8008040 comp rx=0 tx=0).stop 2024-05-09T08:50:09.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.809+0000 7fc8daba1700 1 -- 172.21.15.98:0/1925438292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d41024a0 msgr2=0x7fc8d419d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:09.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.809+0000 7fc8daba1700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d41024a0 0x7fc8d419d470 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000c010 tx=0x7fc8c00047b0 comp rx=0 tx=0).stop 2024-05-09T08:50:09.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.809+0000 7fc8daba1700 1 -- 172.21.15.98:0/1925438292 shutdown_connections 2024-05-09T08:50:09.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.809+0000 7fc8daba1700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fc8c40a2c20 0x7fc8c40a50e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:09.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.809+0000 7fc8daba1700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d41012a0 0x7fc8d406b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:09.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.809+0000 7fc8daba1700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc8d4102e40 0x7fc8d419fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:09.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.809+0000 7fc8daba1700 1 --2- 172.21.15.98:0/1925438292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d41024a0 0x7fc8d419d470 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:09.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.809+0000 7fc8daba1700 1 -- 172.21.15.98:0/1925438292 >> 172.21.15.98:0/1925438292 conn(0x7fc8d40fc880 msgr2=0x7fc8d4105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:09.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.809+0000 7fc8daba1700 1 -- 172.21.15.98:0/1925438292 shutdown_connections 2024-05-09T08:50:09.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:09.809+0000 7fc8daba1700 1 -- 172.21.15.98:0/1925438292 wait complete. 2024-05-09T08:50:10.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:10 smithi098 ceph-mon[30092]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-09T08:50:10.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-09T08:50:10.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:10 smithi142 ceph-mon[34625]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-09T08:50:10.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:10 smithi105 ceph-mon[34498]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-09T08:50:10.768 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:50:11.569 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:11 smithi098 ceph-mon[30092]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:11.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:11 smithi142 ceph-mon[34625]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:11.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:11 smithi105 ceph-mon[34498]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:12.566 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:12 smithi098 ceph-mon[30092]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:50:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:12 smithi142 ceph-mon[34625]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:50:12.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.669+0000 7fbd9590b700 1 -- 172.21.15.98:0/2556353368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd90100630 msgr2=0x7fbd90100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:12.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.669+0000 7fbd9590b700 1 --2- 172.21.15.98:0/2556353368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd90100630 0x7fbd90100a50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbd80009990 tx=0x7fbd80009ca0 comp rx=0 tx=0).stop 2024-05-09T08:50:12.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.669+0000 7fbd9590b700 1 -- 172.21.15.98:0/2556353368 shutdown_connections 2024-05-09T08:50:12.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.669+0000 7fbd9590b700 1 --2- 172.21.15.98:0/2556353368 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbd90103fb0 0x7fbd9010a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:12.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.669+0000 7fbd9590b700 1 --2- 172.21.15.98:0/2556353368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd90100630 0x7fbd90100a50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:12.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.669+0000 7fbd9590b700 1 --2- 172.21.15.98:0/2556353368 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd90103610 0x7fbd90103a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.669+0000 7fbd9590b700 1 -- 172.21.15.98:0/2556353368 >> 172.21.15.98:0/2556353368 conn(0x7fbd900fbfb0 msgr2=0x7fbd900fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.669+0000 7fbd9590b700 1 -- 172.21.15.98:0/2556353368 shutdown_connections 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.669+0000 7fbd9590b700 1 -- 172.21.15.98:0/2556353368 wait complete. 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd9590b700 1 Processor -- start 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd9590b700 1 -- start start 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd9590b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd90100630 0x7fbd901986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd9590b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd90103610 0x7fbd90198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd9590b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbd90103fb0 0x7fbd9019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd9590b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9010e020 con 0x7fbd90103610 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd9590b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9010dea0 con 0x7fbd90103fb0 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd9590b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9010e1a0 con 0x7fbd90100630 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd867fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd90103610 0x7fbd90198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd867fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd90103610 0x7fbd90198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49740/0 (socket says 172.21.15.98:49740) 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd867fc700 1 -- 172.21.15.98:0/439663943 learned_addr learned my addr 172.21.15.98:0/439663943 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd8effd700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd90100630 0x7fbd901986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd8f7fe700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbd90103fb0 0x7fbd9019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd867fc700 1 -- 172.21.15.98:0/439663943 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd90100630 msgr2=0x7fbd901986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd867fc700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd90100630 0x7fbd901986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd867fc700 1 -- 172.21.15.98:0/439663943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbd90103fb0 msgr2=0x7fbd9019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd867fc700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbd90103fb0 0x7fbd9019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd867fc700 1 -- 172.21.15.98:0/439663943 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd901a0330 con 0x7fbd90103610 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd8effd700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd90100630 0x7fbd901986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd8f7fe700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbd90103fb0 0x7fbd9019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.670+0000 7fbd867fc700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd90103610 0x7fbd90198c10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800b560 tx=0x7fbd7800b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.671+0000 7fbd94909700 1 -- 172.21.15.98:0/439663943 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd78015020 con 0x7fbd90103610 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.671+0000 7fbd94909700 1 -- 172.21.15.98:0/439663943 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd78011d10 con 0x7fbd90103610 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.671+0000 7fbd9590b700 1 -- 172.21.15.98:0/439663943 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd901a0620 con 0x7fbd90103610 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.671+0000 7fbd94909700 1 -- 172.21.15.98:0/439663943 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7800f2e0 con 0x7fbd90103610 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.671+0000 7fbd9590b700 1 -- 172.21.15.98:0/439663943 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd901a0be0 con 0x7fbd90103610 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.672+0000 7fbd9590b700 1 -- 172.21.15.98:0/439663943 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd90069010 con 0x7fbd90103610 2024-05-09T08:50:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.673+0000 7fbd94909700 1 -- 172.21.15.98:0/439663943 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbd78018020 con 0x7fbd90103610 2024-05-09T08:50:12.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.673+0000 7fbd94909700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbd7c0a2c20 0x7fbd7c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:12.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.673+0000 7fbd8effd700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbd7c0a2c20 0x7fbd7c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:12.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.674+0000 7fbd94909700 1 -- 172.21.15.98:0/439663943 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fbd780cec20 con 0x7fbd90103610 2024-05-09T08:50:12.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.674+0000 7fbd8effd700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbd7c0a2c20 0x7fbd7c0a50e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbd800039c0 tx=0x7fbd80005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:12.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.678+0000 7fbd94909700 1 -- 172.21.15.98:0/439663943 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbd7801f020 con 0x7fbd90103610 2024-05-09T08:50:12.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:12 smithi105 ceph-mon[34498]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:50:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.889+0000 7fbd9590b700 1 -- 172.21.15.98:0/439663943 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fbd90062760 con 0x7fbd7c0a2c20 2024-05-09T08:50:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.895+0000 7fbd94909700 1 -- 172.21.15.98:0/439663943 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fbd90062760 con 0x7fbd7c0a2c20 2024-05-09T08:50:12.894 INFO:teuthology.orchestra.run.smithi098.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-09T08:50:12.894 INFO:teuthology.orchestra.run.smithi098.stdout:mds.all 12/12 54s ago 2m smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l;count:12 2024-05-09T08:50:12.894 INFO:teuthology.orchestra.run.smithi098.stdout:mgr 3/3 54s ago 7m smithi098=x;smithi105=y;smithi142=z;count:3 2024-05-09T08:50:12.894 INFO:teuthology.orchestra.run.smithi098.stdout:mon 3/3 54s ago 8m smithi098:172.21.15.98=a;smithi105:172.21.15.105=b;smithi142:172.21.15.142=c;count:3 2024-05-09T08:50:12.894 INFO:teuthology.orchestra.run.smithi098.stdout:osd 12 54s ago - 2024-05-09T08:50:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.897+0000 7fbd9590b700 1 -- 172.21.15.98:0/439663943 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbd7c0a2c20 msgr2=0x7fbd7c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.897+0000 7fbd9590b700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbd7c0a2c20 0x7fbd7c0a50e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbd800039c0 tx=0x7fbd80005fb0 comp rx=0 tx=0).stop 2024-05-09T08:50:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.897+0000 7fbd9590b700 1 -- 172.21.15.98:0/439663943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd90103610 msgr2=0x7fbd90198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.897+0000 7fbd9590b700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd90103610 0x7fbd90198c10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800b560 tx=0x7fbd7800b920 comp rx=0 tx=0).stop 2024-05-09T08:50:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.897+0000 7fbd9590b700 1 -- 172.21.15.98:0/439663943 shutdown_connections 2024-05-09T08:50:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.897+0000 7fbd9590b700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fbd7c0a2c20 0x7fbd7c0a50e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.897+0000 7fbd9590b700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbd90103fb0 0x7fbd9019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.897+0000 7fbd9590b700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd90100630 0x7fbd901986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.897+0000 7fbd9590b700 1 --2- 172.21.15.98:0/439663943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd90103610 0x7fbd90198c10 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.897+0000 7fbd9590b700 1 -- 172.21.15.98:0/439663943 >> 172.21.15.98:0/439663943 conn(0x7fbd900fbfb0 msgr2=0x7fbd90106a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.897+0000 7fbd9590b700 1 -- 172.21.15.98:0/439663943 shutdown_connections 2024-05-09T08:50:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:12.897+0000 7fbd9590b700 1 -- 172.21.15.98:0/439663943 wait complete. 2024-05-09T08:50:13.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-09T08:50:13.939 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:50:14.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:13 smithi098 ceph-mon[30092]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:14.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:13 smithi098 ceph-mon[30092]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:50:14.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:13 smithi142 ceph-mon[34625]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:14.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:13 smithi142 ceph-mon[34625]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:50:14.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:13 smithi105 ceph-mon[34498]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:14.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:13 smithi105 ceph-mon[34498]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:50:15.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.632+0000 7fedcd9cf700 1 -- 172.21.15.98:0/640395161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedc81023c0 msgr2=0x7fedc81027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.632+0000 7fedcd9cf700 1 --2- 172.21.15.98:0/640395161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedc81023c0 0x7fedc81027e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fedb80099c0 tx=0x7fedb8009cd0 comp rx=0 tx=0).stop 2024-05-09T08:50:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.632+0000 7fedcd9cf700 1 -- 172.21.15.98:0/640395161 shutdown_connections 2024-05-09T08:50:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.632+0000 7fedcd9cf700 1 --2- 172.21.15.98:0/640395161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fedc8103f60 0x7fedc810a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.632+0000 7fedcd9cf700 1 --2- 172.21.15.98:0/640395161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedc81023c0 0x7fedc81027e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.632+0000 7fedcd9cf700 1 --2- 172.21.15.98:0/640395161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedc81035c0 0x7fedc8103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.632+0000 7fedcd9cf700 1 -- 172.21.15.98:0/640395161 >> 172.21.15.98:0/640395161 conn(0x7fedc80fd960 msgr2=0x7fedc80ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.632+0000 7fedcd9cf700 1 -- 172.21.15.98:0/640395161 shutdown_connections 2024-05-09T08:50:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.632+0000 7fedcd9cf700 1 -- 172.21.15.98:0/640395161 wait complete. 2024-05-09T08:50:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.633+0000 7fedcd9cf700 1 Processor -- start 2024-05-09T08:50:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.633+0000 7fedcd9cf700 1 -- start start 2024-05-09T08:50:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.633+0000 7fedcd9cf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedc81023c0 0x7fedc80760c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:15.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.633+0000 7fedcd9cf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedc81035c0 0x7fedc8074710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:15.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.633+0000 7fedcd9cf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fedc8103f60 0x7fedc8074c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:15.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.633+0000 7fedcd9cf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fedc810df80 con 0x7fedc81035c0 2024-05-09T08:50:15.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.633+0000 7fedcd9cf700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fedc810de00 con 0x7fedc8103f60 2024-05-09T08:50:15.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.633+0000 7fedcd9cf700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fedc810e100 con 0x7fedc81023c0 2024-05-09T08:50:15.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedc67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedc81035c0 0x7fedc8074710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:15.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedc67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedc81035c0 0x7fedc8074710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52602/0 (socket says 172.21.15.98:52602) 2024-05-09T08:50:15.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedc67fc700 1 -- 172.21.15.98:0/341658814 learned_addr learned my addr 172.21.15.98:0/341658814 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:15.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedc77fe700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fedc8103f60 0x7fedc8074c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:15.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedc6ffd700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedc81023c0 0x7fedc80760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:15.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedc67fc700 1 -- 172.21.15.98:0/341658814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedc81023c0 msgr2=0x7fedc80760c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:15.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedc67fc700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedc81023c0 0x7fedc80760c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:15.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedc67fc700 1 -- 172.21.15.98:0/341658814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fedc8103f60 msgr2=0x7fedc8074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:15.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedc67fc700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fedc8103f60 0x7fedc8074c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:15.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedc67fc700 1 -- 172.21.15.98:0/341658814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc8075500 con 0x7fedc81035c0 2024-05-09T08:50:15.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedc77fe700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fedc8103f60 0x7fedc8074c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:15.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedc67fc700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedc81035c0 0x7fedc8074710 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fedbc00c920 tx=0x7fedbc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:15.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedcc9cd700 1 -- 172.21.15.98:0/341658814 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedbc007b00 con 0x7fedc81035c0 2024-05-09T08:50:15.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedcc9cd700 1 -- 172.21.15.98:0/341658814 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedbc004d10 con 0x7fedc81035c0 2024-05-09T08:50:15.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedcc9cd700 1 -- 172.21.15.98:0/341658814 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedbc00f450 con 0x7fedc81035c0 2024-05-09T08:50:15.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedcd9cf700 1 -- 172.21.15.98:0/341658814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedc81a2700 con 0x7fedc81035c0 2024-05-09T08:50:15.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.634+0000 7fedcd9cf700 1 -- 172.21.15.98:0/341658814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedc81a2c10 con 0x7fedc81035c0 2024-05-09T08:50:15.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.635+0000 7fedcd9cf700 1 -- 172.21.15.98:0/341658814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedc8069010 con 0x7fedc81035c0 2024-05-09T08:50:15.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.637+0000 7fedcc9cd700 1 -- 172.21.15.98:0/341658814 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fedbc004750 con 0x7fedc81035c0 2024-05-09T08:50:15.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.637+0000 7fedcc9cd700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fedb00a2c20 0x7fedb00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:15.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.637+0000 7fedc6ffd700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fedb00a2c20 0x7fedb00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:15.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.637+0000 7fedcc9cd700 1 -- 172.21.15.98:0/341658814 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fedbc0cf400 con 0x7fedc81035c0 2024-05-09T08:50:15.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.638+0000 7fedc6ffd700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fedb00a2c20 0x7fedb00a50e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fedb8003890 tx=0x7fedb8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:15.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.641+0000 7fedcc9cd700 1 -- 172.21.15.98:0/341658814 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fedbc014030 con 0x7fedc81035c0 2024-05-09T08:50:15.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.844+0000 7fedcd9cf700 1 -- 172.21.15.98:0/341658814 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fedc8062760 con 0x7fedb00a2c20 2024-05-09T08:50:15.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.845+0000 7fedcc9cd700 1 -- 172.21.15.98:0/341658814 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fedc8062760 con 0x7fedb00a2c20 2024-05-09T08:50:15.844 INFO:teuthology.orchestra.run.smithi098.stdout:HOST ADDR LABELS STATUS 2024-05-09T08:50:15.845 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098 172.21.15.98 2024-05-09T08:50:15.845 INFO:teuthology.orchestra.run.smithi098.stdout:smithi105 172.21.15.105 2024-05-09T08:50:15.845 INFO:teuthology.orchestra.run.smithi098.stdout:smithi142 172.21.15.142 2024-05-09T08:50:15.845 INFO:teuthology.orchestra.run.smithi098.stdout:3 hosts in cluster 2024-05-09T08:50:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.847+0000 7fedcd9cf700 1 -- 172.21.15.98:0/341658814 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fedb00a2c20 msgr2=0x7fedb00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.847+0000 7fedcd9cf700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fedb00a2c20 0x7fedb00a50e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fedb8003890 tx=0x7fedb8005fb0 comp rx=0 tx=0).stop 2024-05-09T08:50:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.847+0000 7fedcd9cf700 1 -- 172.21.15.98:0/341658814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedc81035c0 msgr2=0x7fedc8074710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.847+0000 7fedcd9cf700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedc81035c0 0x7fedc8074710 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fedbc00c920 tx=0x7fedbc00cce0 comp rx=0 tx=0).stop 2024-05-09T08:50:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.847+0000 7fedcd9cf700 1 -- 172.21.15.98:0/341658814 shutdown_connections 2024-05-09T08:50:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.847+0000 7fedcd9cf700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fedb00a2c20 0x7fedb00a50e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.847+0000 7fedcd9cf700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fedc8103f60 0x7fedc8074c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.847+0000 7fedcd9cf700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedc81023c0 0x7fedc80760c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.847+0000 7fedcd9cf700 1 --2- 172.21.15.98:0/341658814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedc81035c0 0x7fedc8074710 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.847+0000 7fedcd9cf700 1 -- 172.21.15.98:0/341658814 >> 172.21.15.98:0/341658814 conn(0x7fedc80fd960 msgr2=0x7fedc81068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.848+0000 7fedcd9cf700 1 -- 172.21.15.98:0/341658814 shutdown_connections 2024-05-09T08:50:15.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:15.848+0000 7fedcd9cf700 1 -- 172.21.15.98:0/341658814 wait complete. 2024-05-09T08:50:16.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:16 smithi098 ceph-mon[30092]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-09T08:50:16.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-09T08:50:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:16 smithi142 ceph-mon[34625]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-09T08:50:16.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:16 smithi105 ceph-mon[34498]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-09T08:50:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:50:17.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:17 smithi098 ceph-mon[30092]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:17 smithi142 ceph-mon[34625]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:17.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:17 smithi105 ceph-mon[34498]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:18.445 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:18 smithi098 ceph-mon[30092]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:50:18.445 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 1]}]: dispatch 2024-05-09T08:50:18.445 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-05-09T08:50:18.445 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-05-09T08:50:18.445 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:18 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-05-09T08:50:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:18 smithi142 ceph-mon[34625]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:50:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 1]}]: dispatch 2024-05-09T08:50:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-05-09T08:50:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-05-09T08:50:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:18 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-05-09T08:50:18.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:18 smithi105 ceph-mon[34498]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:50:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 1]}]: dispatch 2024-05-09T08:50:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-05-09T08:50:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-05-09T08:50:18.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:18 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-05-09T08:50:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.855+0000 7f97a98ac700 1 -- 172.21.15.98:0/1059969108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97a41024b0 msgr2=0x7f97a4102910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.855+0000 7f97a98ac700 1 --2- 172.21.15.98:0/1059969108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97a41024b0 0x7f97a4102910 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f97980099c0 tx=0x7f9798009cd0 comp rx=0 tx=0).stop 2024-05-09T08:50:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.855+0000 7f97a98ac700 1 -- 172.21.15.98:0/1059969108 shutdown_connections 2024-05-09T08:50:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.855+0000 7f97a98ac700 1 --2- 172.21.15.98:0/1059969108 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f97a4102e50 0x7f97a4109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.855+0000 7f97a98ac700 1 --2- 172.21.15.98:0/1059969108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a41012b0 0x7f97a41016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.855+0000 7f97a98ac700 1 --2- 172.21.15.98:0/1059969108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97a41024b0 0x7f97a4102910 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.855+0000 7f97a98ac700 1 -- 172.21.15.98:0/1059969108 >> 172.21.15.98:0/1059969108 conn(0x7f97a40fc870 msgr2=0x7f97a40fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.856+0000 7f97a98ac700 1 -- 172.21.15.98:0/1059969108 shutdown_connections 2024-05-09T08:50:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.856+0000 7f97a98ac700 1 -- 172.21.15.98:0/1059969108 wait complete. 2024-05-09T08:50:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.856+0000 7f97a98ac700 1 Processor -- start 2024-05-09T08:50:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.856+0000 7f97a98ac700 1 -- start start 2024-05-09T08:50:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.856+0000 7f97a98ac700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f97a41012b0 0x7f97a4105240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.856+0000 7f97a98ac700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97a41024b0 0x7f97a4103890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.856+0000 7f97a98ac700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a4102e50 0x7f97a4103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.856+0000 7f97a98ac700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97a410cf10 con 0x7f97a41024b0 2024-05-09T08:50:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.856+0000 7f97a98ac700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f97a410cd90 con 0x7f97a41012b0 2024-05-09T08:50:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.856+0000 7f97a98ac700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f97a410d090 con 0x7f97a4102e50 2024-05-09T08:50:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a27fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97a41024b0 0x7f97a4103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a27fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97a41024b0 0x7f97a4103890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52620/0 (socket says 172.21.15.98:52620) 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a27fc700 1 -- 172.21.15.98:0/2362889167 learned_addr learned my addr 172.21.15.98:0/2362889167 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a2ffd700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f97a41012b0 0x7f97a4105240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a37fe700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a4102e50 0x7f97a4103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a27fc700 1 -- 172.21.15.98:0/2362889167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a4102e50 msgr2=0x7f97a4103dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a27fc700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a4102e50 0x7f97a4103dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a27fc700 1 -- 172.21.15.98:0/2362889167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f97a41012b0 msgr2=0x7f97a4105240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a27fc700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f97a41012b0 0x7f97a4105240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a27fc700 1 -- 172.21.15.98:0/2362889167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a4104680 con 0x7f97a41024b0 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a2ffd700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f97a41012b0 0x7f97a4105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a27fc700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97a41024b0 0x7f97a4103890 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9798003890 tx=0x7f9798003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a88aa700 1 -- 172.21.15.98:0/2362889167 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798027070 con 0x7f97a41024b0 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a88aa700 1 -- 172.21.15.98:0/2362889167 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9798022470 con 0x7f97a41024b0 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a98ac700 1 -- 172.21.15.98:0/2362889167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a41a2780 con 0x7f97a41024b0 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a88aa700 1 -- 172.21.15.98:0/2362889167 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979800f6c0 con 0x7f97a41024b0 2024-05-09T08:50:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.857+0000 7f97a98ac700 1 -- 172.21.15.98:0/2362889167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a41a2c50 con 0x7f97a41024b0 2024-05-09T08:50:18.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.858+0000 7f97a98ac700 1 -- 172.21.15.98:0/2362889167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a4069010 con 0x7f97a41024b0 2024-05-09T08:50:18.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.860+0000 7f97a88aa700 1 -- 172.21.15.98:0/2362889167 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f979802a070 con 0x7f97a41024b0 2024-05-09T08:50:18.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.861+0000 7f97a88aa700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f978c0a2c20 0x7f978c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:18.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.861+0000 7f97a2ffd700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f978c0a2c20 0x7f978c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:18.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.861+0000 7f97a88aa700 1 -- 172.21.15.98:0/2362889167 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f97980d0970 con 0x7f97a41024b0 2024-05-09T08:50:18.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.861+0000 7f97a2ffd700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f978c0a2c20 0x7f978c0a50e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f97a4102310 tx=0x7f9794008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:18.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:18.864+0000 7f97a88aa700 1 -- 172.21.15.98:0/2362889167 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9798031070 con 0x7f97a41024b0 2024-05-09T08:50:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:19.069+0000 7f97a98ac700 1 -- 172.21.15.98:0/2362889167 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f97a40fe9f0 con 0x7f978c0a2c20 2024-05-09T08:50:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:19.071+0000 7f97a88aa700 1 -- 172.21.15.98:0/2362889167 <== mgr.14150 v2:172.21.15.98:6800/1228297308 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f97a40fe9f0 con 0x7f978c0a2c20 2024-05-09T08:50:19.071 INFO:teuthology.orchestra.run.smithi098.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-09T08:50:19.071 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010H400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-09T08:50:19.071 INFO:teuthology.orchestra.run.smithi098.stdout:smithi105 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400TR400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-09T08:50:19.071 INFO:teuthology.orchestra.run.smithi098.stdout:smithi142 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-05-09T08:50:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:19.073+0000 7f97a98ac700 1 -- 172.21.15.98:0/2362889167 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f978c0a2c20 msgr2=0x7f978c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:19.073+0000 7f97a98ac700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f978c0a2c20 0x7f978c0a50e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f97a4102310 tx=0x7f9794008040 comp rx=0 tx=0).stop 2024-05-09T08:50:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:19.073+0000 7f97a98ac700 1 -- 172.21.15.98:0/2362889167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97a41024b0 msgr2=0x7f97a4103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:19.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:19.073+0000 7f97a98ac700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97a41024b0 0x7f97a4103890 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9798003890 tx=0x7f9798003730 comp rx=0 tx=0).stop 2024-05-09T08:50:19.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:19.073+0000 7f97a98ac700 1 -- 172.21.15.98:0/2362889167 shutdown_connections 2024-05-09T08:50:19.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:19.073+0000 7f97a98ac700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f978c0a2c20 0x7f978c0a50e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:19.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:19.073+0000 7f97a98ac700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f97a41012b0 0x7f97a4105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:19.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:19.073+0000 7f97a98ac700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a4102e50 0x7f97a4103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:19.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:19.073+0000 7f97a98ac700 1 --2- 172.21.15.98:0/2362889167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97a41024b0 0x7f97a4103890 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:19.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:19.073+0000 7f97a98ac700 1 -- 172.21.15.98:0/2362889167 >> 172.21.15.98:0/2362889167 conn(0x7f97a40fc870 msgr2=0x7f97a40fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:19.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:19.074+0000 7f97a98ac700 1 -- 172.21.15.98:0/2362889167 shutdown_connections 2024-05-09T08:50:19.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:19.074+0000 7f97a98ac700 1 -- 172.21.15.98:0/2362889167 wait complete. 2024-05-09T08:50:19.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:19 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 1]}]': finished 2024-05-09T08:50:19.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:19 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-05-09T08:50:19.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:19 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-05-09T08:50:19.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:19 smithi098 ceph-mon[30092]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-05-09T08:50:19.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:19 smithi098 ceph-mon[30092]: osdmap e93: 12 total, 12 up, 12 in 2024-05-09T08:50:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:19 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 1]}]': finished 2024-05-09T08:50:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:19 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-05-09T08:50:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:19 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-05-09T08:50:19.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:19 smithi142 ceph-mon[34625]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-05-09T08:50:19.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:19 smithi142 ceph-mon[34625]: osdmap e93: 12 total, 12 up, 12 in 2024-05-09T08:50:19.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:19 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 1]}]': finished 2024-05-09T08:50:19.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:19 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-05-09T08:50:19.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:19 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-05-09T08:50:19.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:19 smithi105 ceph-mon[34498]: from='mgr.14150 172.21.15.98:0/1172058648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-05-09T08:50:19.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:19 smithi105 ceph-mon[34498]: osdmap e93: 12 total, 12 up, 12 in 2024-05-09T08:50:19.909 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-09T08:50:19.917 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-05-09T08:50:19.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-09T08:50:20.166 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:50:20.571 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:20 smithi098 ceph-mon[30092]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:20.571 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:20 smithi098 ceph-mon[30092]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:50:20.571 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:20 smithi098 ceph-mon[30092]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-05-09T08:50:20.571 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:20 smithi098 ceph-mon[30092]: osdmap e94: 12 total, 12 up, 12 in 2024-05-09T08:50:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:20 smithi142 ceph-mon[34625]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:20 smithi142 ceph-mon[34625]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:50:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:20 smithi142 ceph-mon[34625]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-05-09T08:50:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:20 smithi142 ceph-mon[34625]: osdmap e94: 12 total, 12 up, 12 in 2024-05-09T08:50:20.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:20 smithi105 ceph-mon[34498]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:20.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:20 smithi105 ceph-mon[34498]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:50:20.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:20 smithi105 ceph-mon[34498]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-05-09T08:50:20.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:20 smithi105 ceph-mon[34498]: osdmap e94: 12 total, 12 up, 12 in 2024-05-09T08:50:22.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.003+0000 7f37682c5700 1 -- 172.21.15.98:0/3283712084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760102880 msgr2=0x7f3760102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:22.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.003+0000 7f37682c5700 1 --2- 172.21.15.98:0/3283712084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760102880 0x7f3760102ca0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f375c009990 tx=0x7f375c009ca0 comp rx=0 tx=0).stop 2024-05-09T08:50:22.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.004+0000 7f37682c5700 1 -- 172.21.15.98:0/3283712084 shutdown_connections 2024-05-09T08:50:22.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.004+0000 7f37682c5700 1 --2- 172.21.15.98:0/3283712084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3760101ea0 0x7f3760109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:22.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.004+0000 7f37682c5700 1 --2- 172.21.15.98:0/3283712084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760102880 0x7f3760102ca0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:22.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.004+0000 7f37682c5700 1 --2- 172.21.15.98:0/3283712084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37601014e0 0x7f3760101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:22.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.004+0000 7f37682c5700 1 -- 172.21.15.98:0/3283712084 >> 172.21.15.98:0/3283712084 conn(0x7f37600fbfd0 msgr2=0x7f37600fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:22.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.004+0000 7f37682c5700 1 -- 172.21.15.98:0/3283712084 shutdown_connections 2024-05-09T08:50:22.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.004+0000 7f37682c5700 1 -- 172.21.15.98:0/3283712084 wait complete. 2024-05-09T08:50:22.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.004+0000 7f37682c5700 1 Processor -- start 2024-05-09T08:50:22.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.004+0000 7f37682c5700 1 -- start start 2024-05-09T08:50:22.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f37682c5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37601014e0 0x7f3760100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:22.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f37682c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760101ea0 0x7f376019d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:22.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f37682c5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3760102880 0x7f376019f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:22.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f37682c5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f376010cf30 con 0x7f3760101ea0 2024-05-09T08:50:22.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f37682c5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f376010cdb0 con 0x7f37601014e0 2024-05-09T08:50:22.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f37682c5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f376010d0b0 con 0x7f3760102880 2024-05-09T08:50:22.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f3765860700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760101ea0 0x7f376019d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:22.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f3765860700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760101ea0 0x7f376019d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52650/0 (socket says 172.21.15.98:52650) 2024-05-09T08:50:22.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f3765860700 1 -- 172.21.15.98:0/399679000 learned_addr learned my addr 172.21.15.98:0/399679000 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:22.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f3766061700 1 --2- 172.21.15.98:0/399679000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37601014e0 0x7f3760100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:22.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f3766862700 1 --2- 172.21.15.98:0/399679000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3760102880 0x7f376019f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f3765860700 1 -- 172.21.15.98:0/399679000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3760102880 msgr2=0x7f376019f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f3765860700 1 --2- 172.21.15.98:0/399679000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3760102880 0x7f376019f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f3765860700 1 -- 172.21.15.98:0/399679000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37601014e0 msgr2=0x7f3760100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f3765860700 1 --2- 172.21.15.98:0/399679000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37601014e0 0x7f3760100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f3765860700 1 -- 172.21.15.98:0/399679000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37601a0280 con 0x7f3760101ea0 2024-05-09T08:50:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f3765860700 1 --2- 172.21.15.98:0/399679000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760101ea0 0x7f376019d370 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f375000b500 tx=0x7f375000b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f37577fe700 1 -- 172.21.15.98:0/399679000 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3750015020 con 0x7f3760101ea0 2024-05-09T08:50:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f37682c5700 1 -- 172.21.15.98:0/399679000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37601a0570 con 0x7f3760101ea0 2024-05-09T08:50:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f37577fe700 1 -- 172.21.15.98:0/399679000 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3750011d10 con 0x7f3760101ea0 2024-05-09T08:50:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f37577fe700 1 -- 172.21.15.98:0/399679000 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375000f2e0 con 0x7f3760101ea0 2024-05-09T08:50:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.005+0000 7f37682c5700 1 -- 172.21.15.98:0/399679000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37601a0b80 con 0x7f3760101ea0 2024-05-09T08:50:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.006+0000 7f37682c5700 1 -- 172.21.15.98:0/399679000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3760069010 con 0x7f3760101ea0 2024-05-09T08:50:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.008+0000 7f37577fe700 1 -- 172.21.15.98:0/399679000 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3750018020 con 0x7f3760101ea0 2024-05-09T08:50:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.008+0000 7f37577fe700 1 --2- 172.21.15.98:0/399679000 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f374c0a2ca0 0x7f374c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.008+0000 7f3766061700 1 --2- 172.21.15.98:0/399679000 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f374c0a2ca0 0x7f374c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.008+0000 7f37577fe700 1 -- 172.21.15.98:0/399679000 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f37500cf010 con 0x7f3760101ea0 2024-05-09T08:50:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.009+0000 7f3766061700 1 --2- 172.21.15.98:0/399679000 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f374c0a2ca0 0x7f374c0a5160 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f375c00b5c0 tx=0x7f375c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:22.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.012+0000 7f37577fe700 1 -- 172.21.15.98:0/399679000 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f375001f020 con 0x7f3760101ea0 2024-05-09T08:50:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.252+0000 7f37682c5700 1 -- 172.21.15.98:0/399679000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f376002dbb0 con 0x7f3760101ea0 2024-05-09T08:50:22.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.253+0000 7f37577fe700 1 -- 172.21.15.98:0/399679000 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3097 (secure 0 0 0) 0x7f375009bba0 con 0x7f3760101ea0 2024-05-09T08:50:22.252 INFO:teuthology.orchestra.run.smithi098.stdout:e28 2024-05-09T08:50:22.252 INFO:teuthology.orchestra.run.smithi098.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-09T08:50:22.252 INFO:teuthology.orchestra.run.smithi098.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T08:50:22.252 INFO:teuthology.orchestra.run.smithi098.stdout:legacy client fscid: 1 2024-05-09T08:50:22.252 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:22.252 INFO:teuthology.orchestra.run.smithi098.stdout:Filesystem 'cephfs' (1) 2024-05-09T08:50:22.252 INFO:teuthology.orchestra.run.smithi098.stdout:fs_name cephfs 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 28 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-05-09T08:48:17.933763+0000 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2024-05-09T08:48:39.758226+0000 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:tableserver 0 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:root 0 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:session_timeout 300 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:session_autoclose 300 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:max_file_size 1099511627776 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:max_xattr_size 65536 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:required_client_features {} 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:last_failure 0 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:last_failure_osd_epoch 0 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T08:50:22.253 INFO:teuthology.orchestra.run.smithi098.stdout:max_mds 5 2024-05-09T08:50:22.254 INFO:teuthology.orchestra.run.smithi098.stdout:in 0,1,2,3,4 2024-05-09T08:50:22.254 INFO:teuthology.orchestra.run.smithi098.stdout:up {0=24461,1=24457,2=24439,3=24388,4=14667} 2024-05-09T08:50:22.254 INFO:teuthology.orchestra.run.smithi098.stdout:failed 2024-05-09T08:50:22.254 INFO:teuthology.orchestra.run.smithi098.stdout:damaged 2024-05-09T08:50:22.254 INFO:teuthology.orchestra.run.smithi098.stdout:stopped 2024-05-09T08:50:22.254 INFO:teuthology.orchestra.run.smithi098.stdout:data_pools [3,4] 2024-05-09T08:50:22.254 INFO:teuthology.orchestra.run.smithi098.stdout:metadata_pool 2 2024-05-09T08:50:22.254 INFO:teuthology.orchestra.run.smithi098.stdout:inline_data disabled 2024-05-09T08:50:22.254 INFO:teuthology.orchestra.run.smithi098.stdout:balancer 2024-05-09T08:50:22.254 INFO:teuthology.orchestra.run.smithi098.stdout:bal_rank_mask -1 2024-05-09T08:50:22.254 INFO:teuthology.orchestra.run.smithi098.stdout:standby_count_wanted 0 2024-05-09T08:50:22.254 INFO:teuthology.orchestra.run.smithi098.stdout:qdb_cluster leader: 24461 members: 24457,24461,24388,24439,14667 2024-05-09T08:50:22.254 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.h{0:24461} state up:active seq 6 addr [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:50:22.254 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.e{0:24428} state up:standby-replay seq 1 addr [v2:172.21.15.105:6834/588012490,v1:172.21.15.105:6835/588012490] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:50:22.255 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.i{1:24457} state up:active seq 10 addr [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:50:22.255 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.f{1:24421} state up:standby-replay seq 1 addr [v2:172.21.15.142:6834/2791041858,v1:172.21.15.142:6835/2791041858] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:50:22.255 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c{2:24439} state up:active seq 11 addr [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:50:22.255 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.k{2:24398} state up:standby-replay seq 1 addr [v2:172.21.15.105:6832/1472919356,v1:172.21.15.105:6833/1472919356] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:50:22.255 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.l{3:24388} state up:active seq 16 addr [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:50:22.255 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a{3:14649} state up:standby-replay seq 1 addr [v2:172.21.15.98:6840/2714016730,v1:172.21.15.98:6841/2714016730] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:50:22.255 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b{4:14667} state up:active seq 8 addr [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:50:22.255 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.g{4:14637} state up:standby-replay seq 1 addr [v2:172.21.15.98:6838/3981776050,v1:172.21.15.98:6839/3981776050] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:50:22.255 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:22.255 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:22.255 INFO:teuthology.orchestra.run.smithi098.stdout:Standby daemons: 2024-05-09T08:50:22.255 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:22.255 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.j{-1:14556} state up:standby seq 1 addr [v2:172.21.15.98:6834/3172071169,v1:172.21.15.98:6835/3172071169] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:50:22.255 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d{-1:14568} state up:standby seq 1 addr [v2:172.21.15.98:6836/3134006534,v1:172.21.15.98:6837/3134006534] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:50:22.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.256+0000 7f37682c5700 1 -- 172.21.15.98:0/399679000 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f374c0a2ca0 msgr2=0x7f374c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:22.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.256+0000 7f37682c5700 1 --2- 172.21.15.98:0/399679000 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f374c0a2ca0 0x7f374c0a5160 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f375c00b5c0 tx=0x7f375c005fb0 comp rx=0 tx=0).stop 2024-05-09T08:50:22.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.256+0000 7f37682c5700 1 -- 172.21.15.98:0/399679000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760101ea0 msgr2=0x7f376019d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:22.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.256+0000 7f37682c5700 1 --2- 172.21.15.98:0/399679000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760101ea0 0x7f376019d370 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f375000b500 tx=0x7f375000b8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:22.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.256+0000 7f37682c5700 1 -- 172.21.15.98:0/399679000 shutdown_connections 2024-05-09T08:50:22.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.256+0000 7f37682c5700 1 --2- 172.21.15.98:0/399679000 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f374c0a2ca0 0x7f374c0a5160 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:22.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.256+0000 7f37682c5700 1 --2- 172.21.15.98:0/399679000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37601014e0 0x7f3760100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:22.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.256+0000 7f37682c5700 1 --2- 172.21.15.98:0/399679000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3760102880 0x7f376019f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:22.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.256+0000 7f37682c5700 1 --2- 172.21.15.98:0/399679000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760101ea0 0x7f376019d370 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:22.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.256+0000 7f37682c5700 1 -- 172.21.15.98:0/399679000 >> 172.21.15.98:0/399679000 conn(0x7f37600fbfd0 msgr2=0x7f37601057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:22.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.256+0000 7f37682c5700 1 -- 172.21.15.98:0/399679000 shutdown_connections 2024-05-09T08:50:22.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:22.256+0000 7f37682c5700 1 -- 172.21.15.98:0/399679000 wait complete. 2024-05-09T08:50:22.257 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-05-09T08:50:22.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:22 smithi098 ceph-mon[30092]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-05-09T08:50:22.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:22 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/399679000' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-09T08:50:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:22 smithi142 ceph-mon[34625]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-05-09T08:50:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:22 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/399679000' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-09T08:50:22.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:22 smithi105 ceph-mon[34498]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-05-09T08:50:22.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:22 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/399679000' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-09T08:50:23.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-09T08:50:23.251 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:50:24.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:24 smithi098 ceph-mon[30092]: pgmap v357: 129 pgs: 2 peering, 127 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-05-09T08:50:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:24 smithi142 ceph-mon[34625]: pgmap v357: 129 pgs: 2 peering, 127 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-05-09T08:50:24.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:24 smithi105 ceph-mon[34498]: pgmap v357: 129 pgs: 2 peering, 127 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-05-09T08:50:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.947+0000 7fb55eb60700 1 -- 172.21.15.98:0/3212891447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb558102380 msgr2=0x7fb5581027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.947+0000 7fb55eb60700 1 --2- 172.21.15.98:0/3212891447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb558102380 0x7fb5581027a0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0099c0 tx=0x7fb54c009cd0 comp rx=0 tx=0).stop 2024-05-09T08:50:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.948+0000 7fb55eb60700 1 -- 172.21.15.98:0/3212891447 shutdown_connections 2024-05-09T08:50:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.948+0000 7fb55eb60700 1 --2- 172.21.15.98:0/3212891447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb558103f20 0x7fb55810a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:24.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.948+0000 7fb55eb60700 1 --2- 172.21.15.98:0/3212891447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb558102380 0x7fb5581027a0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:24.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.948+0000 7fb55eb60700 1 --2- 172.21.15.98:0/3212891447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb558103580 0x7fb5581039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:24.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.948+0000 7fb55eb60700 1 -- 172.21.15.98:0/3212891447 >> 172.21.15.98:0/3212891447 conn(0x7fb5580fd920 msgr2=0x7fb5580ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:24.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.948+0000 7fb55eb60700 1 -- 172.21.15.98:0/3212891447 shutdown_connections 2024-05-09T08:50:24.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.948+0000 7fb55eb60700 1 -- 172.21.15.98:0/3212891447 wait complete. 2024-05-09T08:50:24.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.948+0000 7fb55eb60700 1 Processor -- start 2024-05-09T08:50:24.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.948+0000 7fb55eb60700 1 -- start start 2024-05-09T08:50:24.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.948+0000 7fb55eb60700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb558102380 0x7fb5580760c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:24.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55eb60700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb558103580 0x7fb558074710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:24.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55eb60700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb558103f20 0x7fb558074c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:24.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55eb60700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb55810df70 con 0x7fb558102380 2024-05-09T08:50:24.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55eb60700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb55810ddf0 con 0x7fb558103580 2024-05-09T08:50:24.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55eb60700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb55810e0f0 con 0x7fb558103f20 2024-05-09T08:50:24.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55c8fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb558102380 0x7fb5580760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:24.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55c8fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb558102380 0x7fb5580760c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52668/0 (socket says 172.21.15.98:52668) 2024-05-09T08:50:24.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55c8fc700 1 -- 172.21.15.98:0/1404500883 learned_addr learned my addr 172.21.15.98:0/1404500883 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:24.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55d0fd700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb558103f20 0x7fb558074c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:24.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb557fff700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb558103580 0x7fb558074710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:24.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55c8fc700 1 -- 172.21.15.98:0/1404500883 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb558103f20 msgr2=0x7fb558074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:24.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55c8fc700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb558103f20 0x7fb558074c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:24.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55c8fc700 1 -- 172.21.15.98:0/1404500883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb558103580 msgr2=0x7fb558074710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:24.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55c8fc700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb558103580 0x7fb558074710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:24.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55c8fc700 1 -- 172.21.15.98:0/1404500883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb558075500 con 0x7fb558102380 2024-05-09T08:50:24.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55d0fd700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb558103f20 0x7fb558074c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:24.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb557fff700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb558103580 0x7fb558074710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:24.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55c8fc700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb558102380 0x7fb5580760c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00c010 tx=0x7fb54c009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:24.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb555ffb700 1 -- 172.21.15.98:0/1404500883 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c027070 con 0x7fb558102380 2024-05-09T08:50:24.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb555ffb700 1 -- 172.21.15.98:0/1404500883 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb54c004150 con 0x7fb558102380 2024-05-09T08:50:24.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb55eb60700 1 -- 172.21.15.98:0/1404500883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5581a26e0 con 0x7fb558102380 2024-05-09T08:50:24.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.949+0000 7fb555ffb700 1 -- 172.21.15.98:0/1404500883 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c00f460 con 0x7fb558102380 2024-05-09T08:50:24.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.950+0000 7fb55eb60700 1 -- 172.21.15.98:0/1404500883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5581a2bc0 con 0x7fb558102380 2024-05-09T08:50:24.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.951+0000 7fb55eb60700 1 -- 172.21.15.98:0/1404500883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb558069010 con 0x7fb558102380 2024-05-09T08:50:24.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.952+0000 7fb555ffb700 1 -- 172.21.15.98:0/1404500883 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb54c02a070 con 0x7fb558102380 2024-05-09T08:50:24.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.952+0000 7fb555ffb700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb5480a2ca0 0x7fb5480a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:24.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.952+0000 7fb555ffb700 1 -- 172.21.15.98:0/1404500883 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fb54c0d0220 con 0x7fb558102380 2024-05-09T08:50:24.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.952+0000 7fb557fff700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb5480a2ca0 0x7fb5480a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:24.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.953+0000 7fb557fff700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb5480a2ca0 0x7fb5480a5160 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb544005950 tx=0x7fb544005f50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:24.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:24.957+0000 7fb555ffb700 1 -- 172.21.15.98:0/1404500883 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb54c05e740 con 0x7fb558102380 2024-05-09T08:50:25.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.154+0000 7fb55eb60700 1 -- 172.21.15.98:0/1404500883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fb558062f30 con 0x7fb558102380 2024-05-09T08:50:25.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.155+0000 7fb555ffb700 1 -- 172.21.15.98:0/1404500883 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v94) ==== 56+0+5718 (secure 0 0 0) 0x7fb54c013070 con 0x7fb558102380 2024-05-09T08:50:25.154 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 94 2024-05-09T08:50:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:fsid a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T08:50:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-05-09T08:40:17.456061+0000 2024-05-09T08:50:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2024-05-09T08:50:19.278344+0000 2024-05-09T08:50:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-09T08:50:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:crush_version 27 2024-05-09T08:50:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:full_ratio 0.95 2024-05-09T08:50:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:backfillfull_ratio 0.9 2024-05-09T08:50:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:nearfull_ratio 0.85 2024-05-09T08:50:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:require_min_compat_client luminous 2024-05-09T08:50:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:min_compat_client luminous 2024-05-09T08:50:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:require_osd_release squid 2024-05-09T08:50:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:stretch_mode_enabled false 2024-05-09T08:50:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-09T08:50:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-05-09T08:50:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-05-09T08:50:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 90 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-05-09T08:50:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:max_osd 12 2024-05-09T08:50:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 up in weight 1 up_from 8 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6802/2307851214,v1:172.21.15.98:6803/2307851214] [v2:172.21.15.98:6804/2307851214,v1:172.21.15.98:6805/2307851214] exists,up 9aaf45c6-e99a-4a52-88f0-b24f52befeef 2024-05-09T08:50:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 up in weight 1 up_from 13 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6800/2333305222,v1:172.21.15.105:6801/2333305222] [v2:172.21.15.105:6802/2333305222,v1:172.21.15.105:6803/2333305222] exists,up fec79393-4941-4307-8146-dfbf97c37be3 2024-05-09T08:50:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 up in weight 1 up_from 18 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6800/1020582557,v1:172.21.15.142:6801/1020582557] [v2:172.21.15.142:6802/1020582557,v1:172.21.15.142:6803/1020582557] exists,up 4b659273-578f-462c-8089-3cad73e6f8d2 2024-05-09T08:50:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 up in weight 1 up_from 26 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6810/2745237287,v1:172.21.15.98:6811/2745237287] [v2:172.21.15.98:6812/2745237287,v1:172.21.15.98:6813/2745237287] exists,up 4d229212-2643-4c3a-93c1-25adad98e37d 2024-05-09T08:50:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 up in weight 1 up_from 34 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6808/3797460474,v1:172.21.15.105:6809/3797460474] [v2:172.21.15.105:6810/3797460474,v1:172.21.15.105:6811/3797460474] exists,up 3cf860bd-1c29-45e5-ba31-5dd49f76913a 2024-05-09T08:50:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 up in weight 1 up_from 39 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6808/1058034789,v1:172.21.15.142:6809/1058034789] [v2:172.21.15.142:6810/1058034789,v1:172.21.15.142:6811/1058034789] exists,up 9d2c5554-188a-45eb-b33b-cf929f0b40fa 2024-05-09T08:50:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 up in weight 1 up_from 47 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6818/1723462836,v1:172.21.15.98:6819/1723462836] [v2:172.21.15.98:6820/1723462836,v1:172.21.15.98:6821/1723462836] exists,up 2aa58eef-6903-41c6-9cb2-5b80475736ea 2024-05-09T08:50:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 up in weight 1 up_from 52 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6816/352668764,v1:172.21.15.105:6817/352668764] [v2:172.21.15.105:6818/352668764,v1:172.21.15.105:6819/352668764] exists,up 72436c50-ea38-4783-a40f-2312deff10e2 2024-05-09T08:50:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:osd.8 up in weight 1 up_from 60 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6816/1607245901,v1:172.21.15.142:6817/1607245901] [v2:172.21.15.142:6818/1607245901,v1:172.21.15.142:6819/1607245901] exists,up a0716bad-82d8-4713-b945-f640664a0783 2024-05-09T08:50:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:osd.9 up in weight 1 up_from 68 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6826/1742411414,v1:172.21.15.98:6827/1742411414] [v2:172.21.15.98:6828/1742411414,v1:172.21.15.98:6829/1742411414] exists,up ae9746f9-c5b6-4574-8627-8cc50a41fa9e 2024-05-09T08:50:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:osd.10 up in weight 1 up_from 73 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6824/1999822218,v1:172.21.15.105:6825/1999822218] [v2:172.21.15.105:6826/1999822218,v1:172.21.15.105:6827/1999822218] exists,up 1963b555-2424-4d7e-b769-95433251224f 2024-05-09T08:50:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:osd.11 up in weight 1 up_from 81 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6824/4047948207,v1:172.21.15.142:6825/4047948207] [v2:172.21.15.142:6826/4047948207,v1:172.21.15.142:6827/4047948207] exists,up 0d685640-23a3-4c93-8a66-1db5b01c3252 2024-05-09T08:50:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.a [2,11,8,1] 2024-05-09T08:50:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.13 [2,11] 2024-05-09T08:50:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.13 [4,9] 2024-05-09T08:50:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.1b [4,7] 2024-05-09T08:50:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.1d [4,6] 2024-05-09T08:50:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.27 [3,9] 2024-05-09T08:50:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.31 [4,8] 2024-05-09T08:50:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.37 [5,6] 2024-05-09T08:50:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.38 [3,9] 2024-05-09T08:50:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.39 [5,8] 2024-05-09T08:50:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.5 [9,10] 2024-05-09T08:50:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.14 [9,8] 2024-05-09T08:50:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.16 [9,8] 2024-05-09T08:50:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.1c [5,8] 2024-05-09T08:50:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/2163305674 expires 2024-05-10T08:41:17.098271+0000 2024-05-09T08:50:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/1385598801 expires 2024-05-10T08:41:17.098271+0000 2024-05-09T08:50:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/498605085 expires 2024-05-10T08:41:17.098271+0000 2024-05-09T08:50:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/1313924889 expires 2024-05-10T08:40:50.793131+0000 2024-05-09T08:50:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/498605085 expires 2024-05-10T08:41:17.098271+0000 2024-05-09T08:50:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/812399881 expires 2024-05-10T08:40:50.793131+0000 2024-05-09T08:50:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/2540720935 expires 2024-05-10T08:40:50.793131+0000 2024-05-09T08:50:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/1313924889 expires 2024-05-10T08:40:50.793131+0000 2024-05-09T08:50:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3487867282 expires 2024-05-10T08:41:17.098271+0000 2024-05-09T08:50:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/645511714 expires 2024-05-10T08:40:50.793131+0000 2024-05-09T08:50:25.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.158+0000 7fb55eb60700 1 -- 172.21.15.98:0/1404500883 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb5480a2ca0 msgr2=0x7fb5480a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:25.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.158+0000 7fb55eb60700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb5480a2ca0 0x7fb5480a5160 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb544005950 tx=0x7fb544005f50 comp rx=0 tx=0).stop 2024-05-09T08:50:25.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.158+0000 7fb55eb60700 1 -- 172.21.15.98:0/1404500883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb558102380 msgr2=0x7fb5580760c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:25.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.158+0000 7fb55eb60700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb558102380 0x7fb5580760c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00c010 tx=0x7fb54c009f10 comp rx=0 tx=0).stop 2024-05-09T08:50:25.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.158+0000 7fb55eb60700 1 -- 172.21.15.98:0/1404500883 shutdown_connections 2024-05-09T08:50:25.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.158+0000 7fb55eb60700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb5480a2ca0 0x7fb5480a5160 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:25.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.158+0000 7fb55eb60700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb558103f20 0x7fb558074c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:25.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.158+0000 7fb55eb60700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb558102380 0x7fb5580760c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:25.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.158+0000 7fb55eb60700 1 --2- 172.21.15.98:0/1404500883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb558103580 0x7fb558074710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:25.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.158+0000 7fb55eb60700 1 -- 172.21.15.98:0/1404500883 >> 172.21.15.98:0/1404500883 conn(0x7fb5580fd920 msgr2=0x7fb558106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:25.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.158+0000 7fb55eb60700 1 -- 172.21.15.98:0/1404500883 shutdown_connections 2024-05-09T08:50:25.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.158+0000 7fb55eb60700 1 -- 172.21.15.98:0/1404500883 wait complete. 2024-05-09T08:50:25.530 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:25 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1404500883' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-09T08:50:25.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:25 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1404500883' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-09T08:50:25.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:25 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1404500883' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-09T08:50:25.844 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-09T08:50:25.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:50:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.962+0000 7f3a5e010640 1 -- 172.21.15.98:0/54706446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a58107610 msgr2=0x7f3a58109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.962+0000 7f3a5e010640 1 --2- 172.21.15.98:0/54706446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a58107610 0x7f3a58109a00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009900 tx=0x7f3a5002a410 comp rx=0 tx=0).stop 2024-05-09T08:50:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 -- 172.21.15.98:0/54706446 shutdown_connections 2024-05-09T08:50:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 --2- 172.21.15.98:0/54706446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a58107610 0x7f3a58109a00 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 --2- 172.21.15.98:0/54706446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a58104ce0 0x7f3a581070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 --2- 172.21.15.98:0/54706446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a581023b0 0x7f3a581047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 -- 172.21.15.98:0/54706446 >> 172.21.15.98:0/54706446 conn(0x7f3a580fbd40 msgr2=0x7f3a580fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 -- 172.21.15.98:0/54706446 shutdown_connections 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 -- 172.21.15.98:0/54706446 wait complete. 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 Processor -- start 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 -- start start 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a581023b0 0x7f3a58197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a58104ce0 0x7f3a58198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a58107610 0x7f3a5819f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5810d580 con 0x7f3a581023b0 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5810d400 con 0x7f3a58107610 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.963+0000 7f3a5e010640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5810d700 con 0x7f3a58104ce0 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a577fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a581023b0 0x7f3a58197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a577fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a581023b0 0x7f3a58197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49764/0 (socket says 172.21.15.98:49764) 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a577fe640 1 -- 172.21.15.98:0/2528663460 learned_addr learned my addr 172.21.15.98:0/2528663460 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a57fff640 1 --2- 172.21.15.98:0/2528663460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a58107610 0x7f3a5819f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a577fe640 1 -- 172.21.15.98:0/2528663460 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a58104ce0 msgr2=0x7f3a58198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a577fe640 1 --2- 172.21.15.98:0/2528663460 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a58104ce0 0x7f3a58198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a577fe640 1 -- 172.21.15.98:0/2528663460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a58107610 msgr2=0x7f3a5819f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a577fe640 1 --2- 172.21.15.98:0/2528663460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a58107610 0x7f3a5819f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a577fe640 1 -- 172.21.15.98:0/2528663460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5819f7e0 con 0x7f3a581023b0 2024-05-09T08:50:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a577fe640 1 --2- 172.21.15.98:0/2528663460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a581023b0 0x7f3a58197bc0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800ba50 tx=0x7f3a4800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a54ff9640 1 -- 172.21.15.98:0/2528663460 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a48019070 con 0x7f3a581023b0 2024-05-09T08:50:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a54ff9640 1 -- 172.21.15.98:0/2528663460 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a4800cbe0 con 0x7f3a581023b0 2024-05-09T08:50:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a5e010640 1 -- 172.21.15.98:0/2528663460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a5819fad0 con 0x7f3a581023b0 2024-05-09T08:50:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a54ff9640 1 -- 172.21.15.98:0/2528663460 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a480076e0 con 0x7f3a581023b0 2024-05-09T08:50:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.964+0000 7f3a5e010640 1 -- 172.21.15.98:0/2528663460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a581a0020 con 0x7f3a581023b0 2024-05-09T08:50:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.965+0000 7f3a5e010640 1 -- 172.21.15.98:0/2528663460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a580fd830 con 0x7f3a581023b0 2024-05-09T08:50:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.966+0000 7f3a54ff9640 1 -- 172.21.15.98:0/2528663460 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3a4801c070 con 0x7f3a581023b0 2024-05-09T08:50:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.967+0000 7f3a54ff9640 1 --2- 172.21.15.98:0/2528663460 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3a400a2c10 0x7f3a400a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.967+0000 7f3a56ffd640 1 --2- 172.21.15.98:0/2528663460 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3a400a2c10 0x7f3a400a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.968+0000 7f3a54ff9640 1 -- 172.21.15.98:0/2528663460 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f3a480cea40 con 0x7f3a581023b0 2024-05-09T08:50:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.968+0000 7f3a56ffd640 1 --2- 172.21.15.98:0/2528663460 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3a400a2c10 0x7f3a400a50d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c005fd0 tx=0x7f3a4c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:25.972+0000 7f3a54ff9640 1 -- 172.21.15.98:0/2528663460 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3a480d0e20 con 0x7f3a581023b0 2024-05-09T08:50:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.151+0000 7f3a5e010640 1 -- 172.21.15.98:0/2528663460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3a58062530 con 0x7f3a581023b0 2024-05-09T08:50:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.153+0000 7f3a54ff9640 1 -- 172.21.15.98:0/2528663460 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24829 (secure 0 0 0) 0x7f3a4809b5d0 con 0x7f3a581023b0 2024-05-09T08:50:26.153 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:26.153 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":94,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:50:19.278344+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":1}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:50:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.155+0000 7f3a5e010640 1 -- 172.21.15.98:0/2528663460 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3a400a2c10 msgr2=0x7f3a400a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.155+0000 7f3a5e010640 1 --2- 172.21.15.98:0/2528663460 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3a400a2c10 0x7f3a400a50d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c005fd0 tx=0x7f3a4c005850 comp rx=0 tx=0).stop 2024-05-09T08:50:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.155+0000 7f3a5e010640 1 -- 172.21.15.98:0/2528663460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a581023b0 msgr2=0x7f3a58197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.155+0000 7f3a5e010640 1 --2- 172.21.15.98:0/2528663460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a581023b0 0x7f3a58197bc0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800ba50 tx=0x7f3a4800bf10 comp rx=0 tx=0).stop 2024-05-09T08:50:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.155+0000 7f3a5e010640 1 -- 172.21.15.98:0/2528663460 shutdown_connections 2024-05-09T08:50:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.155+0000 7f3a5e010640 1 --2- 172.21.15.98:0/2528663460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a58107610 0x7f3a5819f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.155+0000 7f3a5e010640 1 --2- 172.21.15.98:0/2528663460 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a58104ce0 0x7f3a58198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.155+0000 7f3a5e010640 1 --2- 172.21.15.98:0/2528663460 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3a400a2c10 0x7f3a400a50d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.155+0000 7f3a5e010640 1 --2- 172.21.15.98:0/2528663460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a581023b0 0x7f3a58197bc0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.155+0000 7f3a5e010640 1 -- 172.21.15.98:0/2528663460 >> 172.21.15.98:0/2528663460 conn(0x7f3a580fbd40 msgr2=0x7f3a58102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.156+0000 7f3a5e010640 1 -- 172.21.15.98:0/2528663460 shutdown_connections 2024-05-09T08:50:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.156+0000 7f3a5e010640 1 -- 172.21.15.98:0/2528663460 wait complete. 2024-05-09T08:50:26.166 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:43:39.424443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:48:15.866450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:48:17.122162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:48:20.591475+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:50:26.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:50:26.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.272+0000 7f3a4642c640 1 -- 172.21.15.98:0/3716695296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40102340 msgr2=0x7f3a40108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.272+0000 7f3a4642c640 1 --2- 172.21.15.98:0/3716695296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40102340 0x7f3a40108bd0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009ca0 tx=0x7f3a3802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.272+0000 7f3a4642c640 1 -- 172.21.15.98:0/3716695296 shutdown_connections 2024-05-09T08:50:26.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.272+0000 7f3a4642c640 1 --2- 172.21.15.98:0/3716695296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40102340 0x7f3a40108bd0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.272+0000 7f3a4642c640 1 --2- 172.21.15.98:0/3716695296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a40101980 0x7f3a40101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.272+0000 7f3a4642c640 1 --2- 172.21.15.98:0/3716695296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a40100790 0x7f3a40100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.272+0000 7f3a4642c640 1 -- 172.21.15.98:0/3716695296 >> 172.21.15.98:0/3716695296 conn(0x7f3a400fbd20 msgr2=0x7f3a400fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:26.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.272+0000 7f3a4642c640 1 -- 172.21.15.98:0/3716695296 shutdown_connections 2024-05-09T08:50:26.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.272+0000 7f3a4642c640 1 -- 172.21.15.98:0/3716695296 wait complete. 2024-05-09T08:50:26.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a4642c640 1 Processor -- start 2024-05-09T08:50:26.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a4642c640 1 -- start start 2024-05-09T08:50:26.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a4642c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40100790 0x7f3a40197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:26.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a4642c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a40101980 0x7f3a40198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:26.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a4642c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a40102340 0x7f3a4019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:26.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a4642c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4010d530 con 0x7f3a40100790 2024-05-09T08:50:26.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a4642c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4010d3b0 con 0x7f3a40102340 2024-05-09T08:50:26.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a4642c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4010d6b0 con 0x7f3a40101980 2024-05-09T08:50:26.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a3ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40100790 0x7f3a40197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:26.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a3ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40100790 0x7f3a40197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49790/0 (socket says 172.21.15.98:49790) 2024-05-09T08:50:26.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a3ffff640 1 -- 172.21.15.98:0/694036329 learned_addr learned my addr 172.21.15.98:0/694036329 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:26.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a449c9640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a40102340 0x7f3a4019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:26.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a3f7fe640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a40101980 0x7f3a40198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:26.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a3ffff640 1 -- 172.21.15.98:0/694036329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a40101980 msgr2=0x7f3a40198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a3ffff640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a40101980 0x7f3a40198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a3ffff640 1 -- 172.21.15.98:0/694036329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a40102340 msgr2=0x7f3a4019f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a3ffff640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a40102340 0x7f3a4019f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a3ffff640 1 -- 172.21.15.98:0/694036329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4019f750 con 0x7f3a40100790 2024-05-09T08:50:26.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a449c9640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a40102340 0x7f3a4019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:26.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.273+0000 7f3a3ffff640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40100790 0x7f3a40197b30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000c970 tx=0x7f3a3000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:26.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.274+0000 7f3a3d7fa640 1 -- 172.21.15.98:0/694036329 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a30004220 con 0x7f3a40100790 2024-05-09T08:50:26.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.274+0000 7f3a3d7fa640 1 -- 172.21.15.98:0/694036329 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a300043c0 con 0x7f3a40100790 2024-05-09T08:50:26.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.274+0000 7f3a4642c640 1 -- 172.21.15.98:0/694036329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a4019fa40 con 0x7f3a40100790 2024-05-09T08:50:26.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.274+0000 7f3a3d7fa640 1 -- 172.21.15.98:0/694036329 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a30004a00 con 0x7f3a40100790 2024-05-09T08:50:26.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.274+0000 7f3a4642c640 1 -- 172.21.15.98:0/694036329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a4019ff40 con 0x7f3a40100790 2024-05-09T08:50:26.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.275+0000 7f3a4642c640 1 -- 172.21.15.98:0/694036329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a40107a50 con 0x7f3a40100790 2024-05-09T08:50:26.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.276+0000 7f3a3d7fa640 1 -- 172.21.15.98:0/694036329 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3a300079b0 con 0x7f3a40100790 2024-05-09T08:50:26.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.277+0000 7f3a3d7fa640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3a280a2b90 0x7f3a280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:26.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.277+0000 7f3a3f7fe640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3a280a2b90 0x7f3a280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:26.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.277+0000 7f3a3d7fa640 1 -- 172.21.15.98:0/694036329 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f3a300cf670 con 0x7f3a40100790 2024-05-09T08:50:26.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.278+0000 7f3a3f7fe640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3a280a2b90 0x7f3a280a5050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3a34005fd0 tx=0x7f3a34005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:26.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.280+0000 7f3a3d7fa640 1 -- 172.21.15.98:0/694036329 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3a30014030 con 0x7f3a40100790 2024-05-09T08:50:26.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.461+0000 7f3a4642c640 1 -- 172.21.15.98:0/694036329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3a4002d370 con 0x7f3a40100790 2024-05-09T08:50:26.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.461+0000 7f3a3d7fa640 1 -- 172.21.15.98:0/694036329 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f3a3009c200 con 0x7f3a40100790 2024-05-09T08:50:26.460 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-09T08:50:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.463+0000 7f3a4642c640 1 -- 172.21.15.98:0/694036329 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3a280a2b90 msgr2=0x7f3a280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.463+0000 7f3a4642c640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3a280a2b90 0x7f3a280a5050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3a34005fd0 tx=0x7f3a34005850 comp rx=0 tx=0).stop 2024-05-09T08:50:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.463+0000 7f3a4642c640 1 -- 172.21.15.98:0/694036329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40100790 msgr2=0x7f3a40197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.463+0000 7f3a4642c640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40100790 0x7f3a40197b30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000c970 tx=0x7f3a3000ce30 comp rx=0 tx=0).stop 2024-05-09T08:50:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.463+0000 7f3a4642c640 1 -- 172.21.15.98:0/694036329 shutdown_connections 2024-05-09T08:50:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.463+0000 7f3a4642c640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a40102340 0x7f3a4019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.463+0000 7f3a4642c640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a40101980 0x7f3a40198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.463+0000 7f3a4642c640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f3a280a2b90 0x7f3a280a5050 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.463+0000 7f3a4642c640 1 --2- 172.21.15.98:0/694036329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40100790 0x7f3a40197b30 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.463+0000 7f3a4642c640 1 -- 172.21.15.98:0/694036329 >> 172.21.15.98:0/694036329 conn(0x7f3a400fbd20 msgr2=0x7f3a40104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.463+0000 7f3a4642c640 1 -- 172.21.15.98:0/694036329 shutdown_connections 2024-05-09T08:50:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.463+0000 7f3a4642c640 1 -- 172.21.15.98:0/694036329 wait complete. 2024-05-09T08:50:26.472 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:26 smithi098 ceph-mon[30092]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 11 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T08:50:26.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:26 smithi098 ceph-mon[30092]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-05-09T08:50:26.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:26 smithi098 ceph-mon[30092]: Cluster is now healthy 2024-05-09T08:50:26.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:26 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2528663460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:26.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:50:26.559 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:26 smithi142 ceph-mon[34625]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 11 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T08:50:26.559 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:26 smithi142 ceph-mon[34625]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-05-09T08:50:26.559 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:26 smithi142 ceph-mon[34625]: Cluster is now healthy 2024-05-09T08:50:26.560 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:26 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2528663460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.577+0000 7f2846c38640 1 -- 172.21.15.98:0/4120376428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2840100770 msgr2=0x7f2840100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.577+0000 7f2846c38640 1 --2- 172.21.15.98:0/4120376428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2840100770 0x7f2840100b90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2830009ca0 tx=0x7f283002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.577+0000 7f2846c38640 1 -- 172.21.15.98:0/4120376428 shutdown_connections 2024-05-09T08:50:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.577+0000 7f2846c38640 1 --2- 172.21.15.98:0/4120376428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2840102320 0x7f2840108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.577+0000 7f2846c38640 1 --2- 172.21.15.98:0/4120376428 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2840101960 0x7f2840101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.577+0000 7f2846c38640 1 --2- 172.21.15.98:0/4120376428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2840100770 0x7f2840100b90 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.577+0000 7f2846c38640 1 -- 172.21.15.98:0/4120376428 >> 172.21.15.98:0/4120376428 conn(0x7f28400fbce0 msgr2=0x7f28400fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.578+0000 7f2846c38640 1 -- 172.21.15.98:0/4120376428 shutdown_connections 2024-05-09T08:50:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.578+0000 7f2846c38640 1 -- 172.21.15.98:0/4120376428 wait complete. 2024-05-09T08:50:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.578+0000 7f2846c38640 1 Processor -- start 2024-05-09T08:50:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.578+0000 7f2846c38640 1 -- start start 2024-05-09T08:50:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.578+0000 7f2846c38640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2840100770 0x7f2840069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.578+0000 7f2846c38640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2840101960 0x7f284019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.578+0000 7f2846c38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2840102320 0x7f284019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.578+0000 7f2846c38640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f284010c440 con 0x7f2840102320 2024-05-09T08:50:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.578+0000 7f2846c38640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f284010c2c0 con 0x7f2840100770 2024-05-09T08:50:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.578+0000 7f2846c38640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f284010c5c0 con 0x7f2840101960 2024-05-09T08:50:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.578+0000 7f28451d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2840102320 0x7f284019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.578+0000 7f28451d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2840102320 0x7f284019ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49812/0 (socket says 172.21.15.98:49812) 2024-05-09T08:50:26.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.578+0000 7f28451d5640 1 -- 172.21.15.98:0/2956003214 learned_addr learned my addr 172.21.15.98:0/2956003214 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:26.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.578+0000 7f28449d4640 1 --2- 172.21.15.98:0/2956003214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2840100770 0x7f2840069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:26.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.579+0000 7f28451d5640 1 -- 172.21.15.98:0/2956003214 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2840101960 msgr2=0x7f284019c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.579+0000 7f28451d5640 1 --2- 172.21.15.98:0/2956003214 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2840101960 0x7f284019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.579+0000 7f28451d5640 1 -- 172.21.15.98:0/2956003214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2840100770 msgr2=0x7f2840069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.579+0000 7f28451d5640 1 --2- 172.21.15.98:0/2956003214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2840100770 0x7f2840069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.579+0000 7f28451d5640 1 -- 172.21.15.98:0/2956003214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f284019f8a0 con 0x7f2840102320 2024-05-09T08:50:26.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.579+0000 7f28449d4640 1 --2- 172.21.15.98:0/2956003214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2840100770 0x7f2840069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:26.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.579+0000 7f28451d5640 1 --2- 172.21.15.98:0/2956003214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2840102320 0x7f284019ef90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f283800d750 tx=0x7f283800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:26.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.579+0000 7f283dffb640 1 -- 172.21.15.98:0/2956003214 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2838017070 con 0x7f2840102320 2024-05-09T08:50:26.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.579+0000 7f283dffb640 1 -- 172.21.15.98:0/2956003214 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2838005500 con 0x7f2840102320 2024-05-09T08:50:26.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.579+0000 7f2846c38640 1 -- 172.21.15.98:0/2956003214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f284019fb30 con 0x7f2840102320 2024-05-09T08:50:26.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.579+0000 7f283dffb640 1 -- 172.21.15.98:0/2956003214 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2838004d90 con 0x7f2840102320 2024-05-09T08:50:26.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.579+0000 7f2846c38640 1 -- 172.21.15.98:0/2956003214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28401a00b0 con 0x7f2840102320 2024-05-09T08:50:26.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.580+0000 7f2846c38640 1 -- 172.21.15.98:0/2956003214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2840107b10 con 0x7f2840102320 2024-05-09T08:50:26.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.581+0000 7f283dffb640 1 -- 172.21.15.98:0/2956003214 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2838004f30 con 0x7f2840102320 2024-05-09T08:50:26.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.582+0000 7f283dffb640 1 --2- 172.21.15.98:0/2956003214 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f282c0a2b90 0x7f282c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:26.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.582+0000 7f28449d4640 1 --2- 172.21.15.98:0/2956003214 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f282c0a2b90 0x7f282c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:26.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.582+0000 7f283dffb640 1 -- 172.21.15.98:0/2956003214 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f28380ceee0 con 0x7f2840102320 2024-05-09T08:50:26.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.582+0000 7f28449d4640 1 --2- 172.21.15.98:0/2956003214 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f282c0a2b90 0x7f282c0a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2830006010 tx=0x7f2830031bb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:26.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.586+0000 7f283dffb640 1 -- 172.21.15.98:0/2956003214 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f283809ba70 con 0x7f2840102320 2024-05-09T08:50:26.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:26 smithi105 ceph-mon[34498]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 11 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T08:50:26.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:26 smithi105 ceph-mon[34498]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-05-09T08:50:26.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:26 smithi105 ceph-mon[34498]: Cluster is now healthy 2024-05-09T08:50:26.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:26 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2528663460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:26.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.761+0000 7f2846c38640 1 -- 172.21.15.98:0/2956003214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f28400624d0 con 0x7f2840102320 2024-05-09T08:50:26.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.761+0000 7f283dffb640 1 -- 172.21.15.98:0/2956003214 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f283809b370 con 0x7f2840102320 2024-05-09T08:50:26.761 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:50:26.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.763+0000 7f2846c38640 1 -- 172.21.15.98:0/2956003214 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f282c0a2b90 msgr2=0x7f282c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.763+0000 7f2846c38640 1 --2- 172.21.15.98:0/2956003214 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f282c0a2b90 0x7f282c0a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2830006010 tx=0x7f2830031bb0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.763+0000 7f2846c38640 1 -- 172.21.15.98:0/2956003214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2840102320 msgr2=0x7f284019ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.763+0000 7f2846c38640 1 --2- 172.21.15.98:0/2956003214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2840102320 0x7f284019ef90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f283800d750 tx=0x7f283800dc10 comp rx=0 tx=0).stop 2024-05-09T08:50:26.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.763+0000 7f2846c38640 1 -- 172.21.15.98:0/2956003214 shutdown_connections 2024-05-09T08:50:26.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.763+0000 7f2846c38640 1 --2- 172.21.15.98:0/2956003214 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f282c0a2b90 0x7f282c0a5050 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.763+0000 7f2846c38640 1 --2- 172.21.15.98:0/2956003214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2840102320 0x7f284019ef90 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.763+0000 7f2846c38640 1 --2- 172.21.15.98:0/2956003214 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2840101960 0x7f284019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.763+0000 7f2846c38640 1 --2- 172.21.15.98:0/2956003214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2840100770 0x7f2840069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.763+0000 7f2846c38640 1 -- 172.21.15.98:0/2956003214 >> 172.21.15.98:0/2956003214 conn(0x7f28400fbce0 msgr2=0x7f2840104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:26.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.764+0000 7f2846c38640 1 -- 172.21.15.98:0/2956003214 shutdown_connections 2024-05-09T08:50:26.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.764+0000 7f2846c38640 1 -- 172.21.15.98:0/2956003214 wait complete. 2024-05-09T08:50:26.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:50:26.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.879+0000 7f52bd44d640 1 -- 172.21.15.98:0/1531786813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52b8100770 msgr2=0x7f52b8100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.879+0000 7f52bd44d640 1 --2- 172.21.15.98:0/1531786813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52b8100770 0x7f52b8100b90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f52a8009900 tx=0x7f52a802a410 comp rx=0 tx=0).stop 2024-05-09T08:50:26.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.879+0000 7f52bd44d640 1 -- 172.21.15.98:0/1531786813 shutdown_connections 2024-05-09T08:50:26.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.879+0000 7f52bd44d640 1 --2- 172.21.15.98:0/1531786813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b8102320 0x7f52b8108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.879+0000 7f52bd44d640 1 --2- 172.21.15.98:0/1531786813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52b8101960 0x7f52b8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.879+0000 7f52bd44d640 1 --2- 172.21.15.98:0/1531786813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52b8100770 0x7f52b8100b90 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.879+0000 7f52bd44d640 1 -- 172.21.15.98:0/1531786813 >> 172.21.15.98:0/1531786813 conn(0x7f52b80fbce0 msgr2=0x7f52b80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:26.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.880+0000 7f52bd44d640 1 -- 172.21.15.98:0/1531786813 shutdown_connections 2024-05-09T08:50:26.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.880+0000 7f52bd44d640 1 -- 172.21.15.98:0/1531786813 wait complete. 2024-05-09T08:50:26.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.880+0000 7f52bd44d640 1 Processor -- start 2024-05-09T08:50:26.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.880+0000 7f52bd44d640 1 -- start start 2024-05-09T08:50:26.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.880+0000 7f52bd44d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52b8100770 0x7f52b8197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:26.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.880+0000 7f52bd44d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52b8101960 0x7f52b81981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:26.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.880+0000 7f52bd44d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b8102320 0x7f52b819f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:26.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.880+0000 7f52bd44d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52b810d510 con 0x7f52b8101960 2024-05-09T08:50:26.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.880+0000 7f52bd44d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f52b810d390 con 0x7f52b8102320 2024-05-09T08:50:26.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.880+0000 7f52bd44d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f52b810d690 con 0x7f52b8100770 2024-05-09T08:50:26.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.880+0000 7f52ae7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52b8101960 0x7f52b81981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:26.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52ae7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52b8101960 0x7f52b81981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49828/0 (socket says 172.21.15.98:49828) 2024-05-09T08:50:26.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52ae7fc640 1 -- 172.21.15.98:0/1798165753 learned_addr learned my addr 172.21.15.98:0/1798165753 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:26.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52b77fe640 1 --2- 172.21.15.98:0/1798165753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b8102320 0x7f52b819f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:26.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52b6ffd640 1 --2- 172.21.15.98:0/1798165753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52b8100770 0x7f52b8197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:26.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52ae7fc640 1 -- 172.21.15.98:0/1798165753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52b8100770 msgr2=0x7f52b8197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52ae7fc640 1 --2- 172.21.15.98:0/1798165753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52b8100770 0x7f52b8197c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52ae7fc640 1 -- 172.21.15.98:0/1798165753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b8102320 msgr2=0x7f52b819f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:26.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52ae7fc640 1 --2- 172.21.15.98:0/1798165753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b8102320 0x7f52b819f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:26.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52ae7fc640 1 -- 172.21.15.98:0/1798165753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b819f880 con 0x7f52b8101960 2024-05-09T08:50:26.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52ae7fc640 1 --2- 172.21.15.98:0/1798165753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52b8101960 0x7f52b81981b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f52a000d770 tx=0x7f52a000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:26.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52b4ff9640 1 -- 172.21.15.98:0/1798165753 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a0017070 con 0x7f52b8101960 2024-05-09T08:50:26.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52bd44d640 1 -- 172.21.15.98:0/1798165753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52b819fb70 con 0x7f52b8101960 2024-05-09T08:50:26.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52b4ff9640 1 -- 172.21.15.98:0/1798165753 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52a0005500 con 0x7f52b8101960 2024-05-09T08:50:26.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52b4ff9640 1 -- 172.21.15.98:0/1798165753 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a0004e00 con 0x7f52b8101960 2024-05-09T08:50:26.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.881+0000 7f52bd44d640 1 -- 172.21.15.98:0/1798165753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52b81a00c0 con 0x7f52b8101960 2024-05-09T08:50:26.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.882+0000 7f52bd44d640 1 -- 172.21.15.98:0/1798165753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52b8107bb0 con 0x7f52b8101960 2024-05-09T08:50:26.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.883+0000 7f52b4ff9640 1 -- 172.21.15.98:0/1798165753 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f52a0004fa0 con 0x7f52b8101960 2024-05-09T08:50:26.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.884+0000 7f52b4ff9640 1 --2- 172.21.15.98:0/1798165753 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f52a40a2b90 0x7f52a40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.884+0000 7f52b6ffd640 1 --2- 172.21.15.98:0/1798165753 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f52a40a2b90 0x7f52a40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.884+0000 7f52b4ff9640 1 -- 172.21.15.98:0/1798165753 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f52a00cef20 con 0x7f52b8101960 2024-05-09T08:50:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.884+0000 7f52b6ffd640 1 --2- 172.21.15.98:0/1798165753 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f52a40a2b90 0x7f52a40a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f52a8006010 tx=0x7f52a8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:26.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:26.888+0000 7f52b4ff9640 1 -- 172.21.15.98:0/1798165753 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f52a009bab0 con 0x7f52b8101960 2024-05-09T08:50:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.063+0000 7f52bd44d640 1 -- 172.21.15.98:0/1798165753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f52b8062430 con 0x7f52b8101960 2024-05-09T08:50:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.063+0000 7f52b4ff9640 1 -- 172.21.15.98:0/1798165753 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f52a009b3b0 con 0x7f52b8101960 2024-05-09T08:50:27.062 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-09T08:50:27.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.065+0000 7f52bd44d640 1 -- 172.21.15.98:0/1798165753 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f52a40a2b90 msgr2=0x7f52a40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.065+0000 7f52bd44d640 1 --2- 172.21.15.98:0/1798165753 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f52a40a2b90 0x7f52a40a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f52a8006010 tx=0x7f52a8000bc0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.065+0000 7f52bd44d640 1 -- 172.21.15.98:0/1798165753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52b8101960 msgr2=0x7f52b81981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.065+0000 7f52bd44d640 1 --2- 172.21.15.98:0/1798165753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52b8101960 0x7f52b81981b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f52a000d770 tx=0x7f52a000dc30 comp rx=0 tx=0).stop 2024-05-09T08:50:27.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.065+0000 7f52bd44d640 1 -- 172.21.15.98:0/1798165753 shutdown_connections 2024-05-09T08:50:27.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.065+0000 7f52bd44d640 1 --2- 172.21.15.98:0/1798165753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b8102320 0x7f52b819f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.065+0000 7f52bd44d640 1 --2- 172.21.15.98:0/1798165753 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f52a40a2b90 0x7f52a40a5050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.065+0000 7f52bd44d640 1 --2- 172.21.15.98:0/1798165753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52b8101960 0x7f52b81981b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.065+0000 7f52bd44d640 1 --2- 172.21.15.98:0/1798165753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52b8100770 0x7f52b8197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.065+0000 7f52bd44d640 1 -- 172.21.15.98:0/1798165753 >> 172.21.15.98:0/1798165753 conn(0x7f52b80fbce0 msgr2=0x7f52b8104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:27.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.065+0000 7f52bd44d640 1 -- 172.21.15.98:0/1798165753 shutdown_connections 2024-05-09T08:50:27.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.065+0000 7f52bd44d640 1 -- 172.21.15.98:0/1798165753 wait complete. 2024-05-09T08:50:27.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:50:27.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.181+0000 7fcbacbc8640 1 -- 172.21.15.98:0/329527989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcba8104cc0 msgr2=0x7fcba81070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.181+0000 7fcbacbc8640 1 --2- 172.21.15.98:0/329527989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcba8104cc0 0x7fcba81070b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009900 tx=0x7fcb9002a410 comp rx=0 tx=0).stop 2024-05-09T08:50:27.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.181+0000 7fcbacbc8640 1 -- 172.21.15.98:0/329527989 shutdown_connections 2024-05-09T08:50:27.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.181+0000 7fcbacbc8640 1 --2- 172.21.15.98:0/329527989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcba81075f0 0x7fcba81099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.181+0000 7fcbacbc8640 1 --2- 172.21.15.98:0/329527989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcba8104cc0 0x7fcba81070b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.181+0000 7fcbacbc8640 1 --2- 172.21.15.98:0/329527989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcba8102390 0x7fcba8104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.181+0000 7fcbacbc8640 1 -- 172.21.15.98:0/329527989 >> 172.21.15.98:0/329527989 conn(0x7fcba80fbce0 msgr2=0x7fcba80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.181+0000 7fcbacbc8640 1 -- 172.21.15.98:0/329527989 shutdown_connections 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.181+0000 7fcbacbc8640 1 -- 172.21.15.98:0/329527989 wait complete. 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.181+0000 7fcbacbc8640 1 Processor -- start 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.181+0000 7fcbacbc8640 1 -- start start 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.181+0000 7fcbacbc8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcba8102390 0x7fcba8197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcbacbc8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcba8104cc0 0x7fcba8198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcbacbc8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcba81075f0 0x7fcba819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcbacbc8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcba810d540 con 0x7fcba8102390 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcbacbc8640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcba810d3c0 con 0x7fcba8104cc0 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcbacbc8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcba810d6c0 con 0x7fcba81075f0 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcba659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcba8102390 0x7fcba8197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcba659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcba8102390 0x7fcba8197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49846/0 (socket says 172.21.15.98:49846) 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcba659c640 1 -- 172.21.15.98:0/3698292960 learned_addr learned my addr 172.21.15.98:0/3698292960 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcb9ffff640 1 --2- 172.21.15.98:0/3698292960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcba8104cc0 0x7fcba8198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:27.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcba6d9d640 1 --2- 172.21.15.98:0/3698292960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcba81075f0 0x7fcba819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:27.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcba659c640 1 -- 172.21.15.98:0/3698292960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcba81075f0 msgr2=0x7fcba819f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcba659c640 1 --2- 172.21.15.98:0/3698292960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcba81075f0 0x7fcba819f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcba659c640 1 -- 172.21.15.98:0/3698292960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcba8104cc0 msgr2=0x7fcba8198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcba659c640 1 --2- 172.21.15.98:0/3698292960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcba8104cc0 0x7fcba8198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcba659c640 1 -- 172.21.15.98:0/3698292960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba819f800 con 0x7fcba8102390 2024-05-09T08:50:27.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcba659c640 1 --2- 172.21.15.98:0/3698292960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcba8102390 0x7fcba8197be0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800b4a0 tx=0x7fcb9800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:27.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcb9f7fe640 1 -- 172.21.15.98:0/3698292960 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb98014070 con 0x7fcba8102390 2024-05-09T08:50:27.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcb9f7fe640 1 -- 172.21.15.98:0/3698292960 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb98010af0 con 0x7fcba8102390 2024-05-09T08:50:27.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcbacbc8640 1 -- 172.21.15.98:0/3698292960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcba819faf0 con 0x7fcba8102390 2024-05-09T08:50:27.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcb9f7fe640 1 -- 172.21.15.98:0/3698292960 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb98017d10 con 0x7fcba8102390 2024-05-09T08:50:27.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.182+0000 7fcbacbc8640 1 -- 172.21.15.98:0/3698292960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcba819fff0 con 0x7fcba8102390 2024-05-09T08:50:27.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.183+0000 7fcbacbc8640 1 -- 172.21.15.98:0/3698292960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcba80fd7d0 con 0x7fcba8102390 2024-05-09T08:50:27.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.185+0000 7fcb9f7fe640 1 -- 172.21.15.98:0/3698292960 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fcb98017070 con 0x7fcba8102390 2024-05-09T08:50:27.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.185+0000 7fcb9f7fe640 1 --2- 172.21.15.98:0/3698292960 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcb940a2b90 0x7fcb940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:27.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.185+0000 7fcb9ffff640 1 --2- 172.21.15.98:0/3698292960 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcb940a2b90 0x7fcb940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:27.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.185+0000 7fcb9f7fe640 1 -- 172.21.15.98:0/3698292960 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fcb980ce900 con 0x7fcba8102390 2024-05-09T08:50:27.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.186+0000 7fcb9ffff640 1 --2- 172.21.15.98:0/3698292960 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcb940a2b90 0x7fcb940a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcb90006010 tx=0x7fcb90000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:27.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.190+0000 7fcb9f7fe640 1 -- 172.21.15.98:0/3698292960 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcb9809b490 con 0x7fcba8102390 2024-05-09T08:50:27.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:27 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/694036329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:27.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:27 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2956003214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:27.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:27 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1798165753' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:27.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.368+0000 7fcbacbc8640 1 -- 172.21.15.98:0/3698292960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcba8062430 con 0x7fcba8102390 2024-05-09T08:50:27.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.369+0000 7fcb9f7fe640 1 -- 172.21.15.98:0/3698292960 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fcb9809ad90 con 0x7fcba8102390 2024-05-09T08:50:27.368 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:50:27.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.371+0000 7fcbacbc8640 1 -- 172.21.15.98:0/3698292960 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcb940a2b90 msgr2=0x7fcb940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.371+0000 7fcbacbc8640 1 --2- 172.21.15.98:0/3698292960 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcb940a2b90 0x7fcb940a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcb90006010 tx=0x7fcb90000bc0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.371+0000 7fcbacbc8640 1 -- 172.21.15.98:0/3698292960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcba8102390 msgr2=0x7fcba8197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.371+0000 7fcbacbc8640 1 --2- 172.21.15.98:0/3698292960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcba8102390 0x7fcba8197be0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800b4a0 tx=0x7fcb9800b960 comp rx=0 tx=0).stop 2024-05-09T08:50:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.371+0000 7fcbacbc8640 1 -- 172.21.15.98:0/3698292960 shutdown_connections 2024-05-09T08:50:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.371+0000 7fcbacbc8640 1 --2- 172.21.15.98:0/3698292960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcba81075f0 0x7fcba819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.371+0000 7fcbacbc8640 1 --2- 172.21.15.98:0/3698292960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcba8104cc0 0x7fcba8198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.371+0000 7fcbacbc8640 1 --2- 172.21.15.98:0/3698292960 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcb940a2b90 0x7fcb940a5050 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.371+0000 7fcbacbc8640 1 --2- 172.21.15.98:0/3698292960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcba8102390 0x7fcba8197be0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.371+0000 7fcbacbc8640 1 -- 172.21.15.98:0/3698292960 >> 172.21.15.98:0/3698292960 conn(0x7fcba80fbce0 msgr2=0x7fcba8102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.371+0000 7fcbacbc8640 1 -- 172.21.15.98:0/3698292960 shutdown_connections 2024-05-09T08:50:27.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.371+0000 7fcbacbc8640 1 -- 172.21.15.98:0/3698292960 wait complete. 2024-05-09T08:50:27.381 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:50:27.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.483+0000 7f062ee31640 1 -- 172.21.15.98:0/2291629151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628103430 msgr2=0x7f0628109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.483+0000 7f062ee31640 1 --2- 172.21.15.98:0/2291629151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628103430 0x7f0628109cc0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0620009ca0 tx=0x7f062002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.483+0000 7f062ee31640 1 -- 172.21.15.98:0/2291629151 shutdown_connections 2024-05-09T08:50:27.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.483+0000 7f062ee31640 1 --2- 172.21.15.98:0/2291629151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628103430 0x7f0628109cc0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.483+0000 7f062ee31640 1 --2- 172.21.15.98:0/2291629151 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0628102a70 0x7f0628102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.483+0000 7f062ee31640 1 --2- 172.21.15.98:0/2291629151 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0628101880 0x7f0628101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.483+0000 7f062ee31640 1 -- 172.21.15.98:0/2291629151 >> 172.21.15.98:0/2291629151 conn(0x7f06280fce10 msgr2=0x7f06280ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.483+0000 7f062ee31640 1 -- 172.21.15.98:0/2291629151 shutdown_connections 2024-05-09T08:50:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.483+0000 7f062ee31640 1 -- 172.21.15.98:0/2291629151 wait complete. 2024-05-09T08:50:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f062ee31640 1 Processor -- start 2024-05-09T08:50:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f062ee31640 1 -- start start 2024-05-09T08:50:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f062ee31640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0628101880 0x7f0628197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f062ee31640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628102a70 0x7f0628197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f062ee31640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0628103430 0x7f062819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f062ee31640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f062810d4e0 con 0x7f0628102a70 2024-05-09T08:50:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f062ee31640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f062810d360 con 0x7f0628101880 2024-05-09T08:50:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f062ee31640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f062810d660 con 0x7f0628103430 2024-05-09T08:50:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f061ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628102a70 0x7f0628197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f061ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628102a70 0x7f0628197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49866/0 (socket says 172.21.15.98:49866) 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f061ffff640 1 -- 172.21.15.98:0/2061490302 learned_addr learned my addr 172.21.15.98:0/2061490302 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f062cbcd640 1 --2- 172.21.15.98:0/2061490302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0628101880 0x7f0628197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f062d3ce640 1 --2- 172.21.15.98:0/2061490302 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0628103430 0x7f062819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f061ffff640 1 -- 172.21.15.98:0/2061490302 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0628103430 msgr2=0x7f062819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f061ffff640 1 --2- 172.21.15.98:0/2061490302 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0628103430 0x7f062819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f061ffff640 1 -- 172.21.15.98:0/2061490302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0628101880 msgr2=0x7f0628197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f061ffff640 1 --2- 172.21.15.98:0/2061490302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0628101880 0x7f0628197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f061ffff640 1 -- 172.21.15.98:0/2061490302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f062819f6b0 con 0x7f0628102a70 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.484+0000 7f061ffff640 1 --2- 172.21.15.98:0/2061490302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628102a70 0x7f0628197fd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f061800c920 tx=0x7f061800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.485+0000 7f061dffb640 1 -- 172.21.15.98:0/2061490302 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618007c70 con 0x7f0628102a70 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.485+0000 7f061dffb640 1 -- 172.21.15.98:0/2061490302 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0618007e10 con 0x7f0628102a70 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.485+0000 7f062ee31640 1 -- 172.21.15.98:0/2061490302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f062819f9a0 con 0x7f0628102a70 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.485+0000 7f061dffb640 1 -- 172.21.15.98:0/2061490302 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618018890 con 0x7f0628102a70 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.485+0000 7f062ee31640 1 -- 172.21.15.98:0/2061490302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f062819fef0 con 0x7f0628102a70 2024-05-09T08:50:27.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.486+0000 7f062ee31640 1 -- 172.21.15.98:0/2061490302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0628108b40 con 0x7f0628102a70 2024-05-09T08:50:27.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.487+0000 7f061dffb640 1 -- 172.21.15.98:0/2061490302 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0618018a70 con 0x7f0628102a70 2024-05-09T08:50:27.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.487+0000 7f061dffb640 1 --2- 172.21.15.98:0/2061490302 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f06100a2b90 0x7f06100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:27.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.487+0000 7f062cbcd640 1 --2- 172.21.15.98:0/2061490302 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f06100a2b90 0x7f06100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:27.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.487+0000 7f061dffb640 1 -- 172.21.15.98:0/2061490302 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f06180cf4b0 con 0x7f0628102a70 2024-05-09T08:50:27.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.488+0000 7f062cbcd640 1 --2- 172.21.15.98:0/2061490302 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f06100a2b90 0x7f06100a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f06281028d0 tx=0x7f062400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:27.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.492+0000 7f061dffb640 1 -- 172.21.15.98:0/2061490302 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0618014030 con 0x7f0628102a70 2024-05-09T08:50:27.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:27 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/694036329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:27.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:27 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2956003214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:27.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:27 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1798165753' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:27.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.707+0000 7f062ee31640 1 -- 172.21.15.98:0/2061490302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f062802d1c0 con 0x7f0628102a70 2024-05-09T08:50:27.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.707+0000 7f061dffb640 1 -- 172.21.15.98:0/2061490302 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11940 (secure 0 0 0) 0x7f061809c040 con 0x7f0628102a70 2024-05-09T08:50:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:27.707 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:48:39.758226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24457,24461,24388,24439,14667]},"id":1}]} 2024-05-09T08:50:27.707 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-05-09T08:50:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.709+0000 7f062ee31640 1 -- 172.21.15.98:0/2061490302 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f06100a2b90 msgr2=0x7f06100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.709+0000 7f062ee31640 1 --2- 172.21.15.98:0/2061490302 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f06100a2b90 0x7f06100a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f06281028d0 tx=0x7f062400b390 comp rx=0 tx=0).stop 2024-05-09T08:50:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.709+0000 7f062ee31640 1 -- 172.21.15.98:0/2061490302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628102a70 msgr2=0x7f0628197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.709+0000 7f062ee31640 1 --2- 172.21.15.98:0/2061490302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628102a70 0x7f0628197fd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f061800c920 tx=0x7f061800cde0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.709+0000 7f062ee31640 1 -- 172.21.15.98:0/2061490302 shutdown_connections 2024-05-09T08:50:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.709+0000 7f062ee31640 1 --2- 172.21.15.98:0/2061490302 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0628103430 0x7f062819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.709+0000 7f062ee31640 1 --2- 172.21.15.98:0/2061490302 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f06100a2b90 0x7f06100a5050 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.709+0000 7f062ee31640 1 --2- 172.21.15.98:0/2061490302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628102a70 0x7f0628197fd0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.709+0000 7f062ee31640 1 --2- 172.21.15.98:0/2061490302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0628101880 0x7f0628197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.709+0000 7f062ee31640 1 -- 172.21.15.98:0/2061490302 >> 172.21.15.98:0/2061490302 conn(0x7f06280fce10 msgr2=0x7f0628105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.709+0000 7f062ee31640 1 -- 172.21.15.98:0/2061490302 shutdown_connections 2024-05-09T08:50:27.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.710+0000 7f062ee31640 1 -- 172.21.15.98:0/2061490302 wait complete. 2024-05-09T08:50:27.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:50:27.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:27 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/694036329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:27.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:27 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2956003214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:27.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:27 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1798165753' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.823+0000 7f49b2b9e640 1 -- 172.21.15.98:0/3901980680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac105240 msgr2=0x7f49ac1056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.823+0000 7f49b2b9e640 1 --2- 172.21.15.98:0/3901980680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac105240 0x7f49ac1056c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009ca0 tx=0x7f49a402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.823+0000 7f49b2b9e640 1 -- 172.21.15.98:0/3901980680 shutdown_connections 2024-05-09T08:50:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.823+0000 7f49b2b9e640 1 --2- 172.21.15.98:0/3901980680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac105240 0x7f49ac1056c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.823+0000 7f49b2b9e640 1 --2- 172.21.15.98:0/3901980680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ac102890 0x7f49ac104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.823+0000 7f49b2b9e640 1 --2- 172.21.15.98:0/3901980680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49ac0fff20 0x7f49ac102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.823+0000 7f49b2b9e640 1 -- 172.21.15.98:0/3901980680 >> 172.21.15.98:0/3901980680 conn(0x7f49ac0fbb40 msgr2=0x7f49ac0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.823+0000 7f49b2b9e640 1 -- 172.21.15.98:0/3901980680 shutdown_connections 2024-05-09T08:50:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.823+0000 7f49b2b9e640 1 -- 172.21.15.98:0/3901980680 wait complete. 2024-05-09T08:50:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.823+0000 7f49b2b9e640 1 Processor -- start 2024-05-09T08:50:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.823+0000 7f49b2b9e640 1 -- start start 2024-05-09T08:50:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b2b9e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac0fff20 0x7f49ac19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b2b9e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49ac102890 0x7f49ac19e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b2b9e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ac105240 0x7f49ac1a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b2b9e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac113900 con 0x7f49ac0fff20 2024-05-09T08:50:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b2b9e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac113780 con 0x7f49ac105240 2024-05-09T08:50:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b2b9e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac113a80 con 0x7f49ac102890 2024-05-09T08:50:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b1b9c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac0fff20 0x7f49ac19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b1b9c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac0fff20 0x7f49ac19df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49884/0 (socket says 172.21.15.98:49884) 2024-05-09T08:50:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b1b9c640 1 -- 172.21.15.98:0/1285346927 learned_addr learned my addr 172.21.15.98:0/1285346927 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b239d640 1 --2- 172.21.15.98:0/1285346927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ac105240 0x7f49ac1a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b139b640 1 --2- 172.21.15.98:0/1285346927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49ac102890 0x7f49ac19e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b1b9c640 1 -- 172.21.15.98:0/1285346927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49ac102890 msgr2=0x7f49ac19e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b1b9c640 1 --2- 172.21.15.98:0/1285346927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49ac102890 0x7f49ac19e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b1b9c640 1 -- 172.21.15.98:0/1285346927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ac105240 msgr2=0x7f49ac1a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b1b9c640 1 --2- 172.21.15.98:0/1285346927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ac105240 0x7f49ac1a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b1b9c640 1 -- 172.21.15.98:0/1285346927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ac1a5b70 con 0x7f49ac0fff20 2024-05-09T08:50:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b1b9c640 1 --2- 172.21.15.98:0/1285346927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac0fff20 0x7f49ac19df50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f49a800d950 tx=0x7f49a800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49a2ffd640 1 -- 172.21.15.98:0/1285346927 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a8017030 con 0x7f49ac0fff20 2024-05-09T08:50:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49a2ffd640 1 -- 172.21.15.98:0/1285346927 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49a8004020 con 0x7f49ac0fff20 2024-05-09T08:50:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49a2ffd640 1 -- 172.21.15.98:0/1285346927 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a80047a0 con 0x7f49ac0fff20 2024-05-09T08:50:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b2b9e640 1 -- 172.21.15.98:0/1285346927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ac1a5e60 con 0x7f49ac0fff20 2024-05-09T08:50:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.824+0000 7f49b2b9e640 1 -- 172.21.15.98:0/1285346927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ac1a62c0 con 0x7f49ac0fff20 2024-05-09T08:50:27.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.825+0000 7f49b2b9e640 1 -- 172.21.15.98:0/1285346927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ac0fd630 con 0x7f49ac0fff20 2024-05-09T08:50:27.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.826+0000 7f49a2ffd640 1 -- 172.21.15.98:0/1285346927 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f49a80041d0 con 0x7f49ac0fff20 2024-05-09T08:50:27.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.827+0000 7f49a2ffd640 1 --2- 172.21.15.98:0/1285346927 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f49980a2b90 0x7f49980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:27.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.827+0000 7f49b139b640 1 --2- 172.21.15.98:0/1285346927 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f49980a2b90 0x7f49980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:27.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.827+0000 7f49a2ffd640 1 -- 172.21.15.98:0/1285346927 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f49a80cef30 con 0x7f49ac0fff20 2024-05-09T08:50:27.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.828+0000 7f49b139b640 1 --2- 172.21.15.98:0/1285346927 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f49980a2b90 0x7f49980a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f49ac19f460 tx=0x7f499c005ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:27.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:27.832+0000 7f49a2ffd640 1 -- 172.21.15.98:0/1285346927 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f49a809bac0 con 0x7f49ac0fff20 2024-05-09T08:50:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.012+0000 7f49b2b9e640 1 -- 172.21.15.98:0/1285346927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49ac02d470 con 0x7f49ac0fff20 2024-05-09T08:50:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.014+0000 7f49a2ffd640 1 -- 172.21.15.98:0/1285346927 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24829 (secure 0 0 0) 0x7f49a809b3c0 con 0x7f49ac0fff20 2024-05-09T08:50:28.014 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:28.014 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":94,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:50:19.278344+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":1}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:50:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.016+0000 7f49b2b9e640 1 -- 172.21.15.98:0/1285346927 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f49980a2b90 msgr2=0x7f49980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.016+0000 7f49b2b9e640 1 --2- 172.21.15.98:0/1285346927 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f49980a2b90 0x7f49980a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f49ac19f460 tx=0x7f499c005ed0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.016+0000 7f49b2b9e640 1 -- 172.21.15.98:0/1285346927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac0fff20 msgr2=0x7f49ac19df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.016+0000 7f49b2b9e640 1 --2- 172.21.15.98:0/1285346927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac0fff20 0x7f49ac19df50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f49a800d950 tx=0x7f49a800de10 comp rx=0 tx=0).stop 2024-05-09T08:50:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.016+0000 7f49b2b9e640 1 -- 172.21.15.98:0/1285346927 shutdown_connections 2024-05-09T08:50:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.016+0000 7f49b2b9e640 1 --2- 172.21.15.98:0/1285346927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ac105240 0x7f49ac1a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.016+0000 7f49b2b9e640 1 --2- 172.21.15.98:0/1285346927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49ac102890 0x7f49ac19e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.016+0000 7f49b2b9e640 1 --2- 172.21.15.98:0/1285346927 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f49980a2b90 0x7f49980a5050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.016+0000 7f49b2b9e640 1 --2- 172.21.15.98:0/1285346927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac0fff20 0x7f49ac19df50 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.016+0000 7f49b2b9e640 1 -- 172.21.15.98:0/1285346927 >> 172.21.15.98:0/1285346927 conn(0x7f49ac0fbb40 msgr2=0x7f49ac100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.016+0000 7f49b2b9e640 1 -- 172.21.15.98:0/1285346927 shutdown_connections 2024-05-09T08:50:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.016+0000 7f49b2b9e640 1 -- 172.21.15.98:0/1285346927 wait complete. 2024-05-09T08:50:28.028 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:43:39.424443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:48:15.866450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:48:17.122162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:48:20.591475+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:50:28.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:50:28.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.132+0000 7f20a124c640 1 -- 172.21.15.98:0/4009130789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f209c101890 msgr2=0x7f209c101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.132+0000 7f20a124c640 1 --2- 172.21.15.98:0/4009130789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f209c101890 0x7f209c101cb0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f208c009900 tx=0x7f208c02a410 comp rx=0 tx=0).stop 2024-05-09T08:50:28.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.132+0000 7f20a124c640 1 -- 172.21.15.98:0/4009130789 shutdown_connections 2024-05-09T08:50:28.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.132+0000 7f20a124c640 1 --2- 172.21.15.98:0/4009130789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f209c103440 0x7f209c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.132+0000 7f20a124c640 1 --2- 172.21.15.98:0/4009130789 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f209c102a80 0x7f209c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.132+0000 7f20a124c640 1 --2- 172.21.15.98:0/4009130789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f209c101890 0x7f209c101cb0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.132+0000 7f20a124c640 1 -- 172.21.15.98:0/4009130789 >> 172.21.15.98:0/4009130789 conn(0x7f209c0fce00 msgr2=0x7f209c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:28.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.132+0000 7f20a124c640 1 -- 172.21.15.98:0/4009130789 shutdown_connections 2024-05-09T08:50:28.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.132+0000 7f20a124c640 1 -- 172.21.15.98:0/4009130789 wait complete. 2024-05-09T08:50:28.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f20a124c640 1 Processor -- start 2024-05-09T08:50:28.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f20a124c640 1 -- start start 2024-05-09T08:50:28.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f20a124c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f209c101890 0x7f209c197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:28.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f20a124c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f209c102a80 0x7f209c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f20a124c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f209c103440 0x7f209c19f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f20a124c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f209c10d510 con 0x7f209c103440 2024-05-09T08:50:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f20a124c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f209c10d390 con 0x7f209c101890 2024-05-09T08:50:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f20a124c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f209c10d690 con 0x7f209c102a80 2024-05-09T08:50:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f209b59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f209c103440 0x7f209c19f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f209b59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f209c103440 0x7f209c19f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49906/0 (socket says 172.21.15.98:49906) 2024-05-09T08:50:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f209b59e640 1 -- 172.21.15.98:0/547899364 learned_addr learned my addr 172.21.15.98:0/547899364 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f209ad9d640 1 --2- 172.21.15.98:0/547899364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f209c101890 0x7f209c197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f209a59c640 1 --2- 172.21.15.98:0/547899364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f209c102a80 0x7f209c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f209b59e640 1 -- 172.21.15.98:0/547899364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f209c102a80 msgr2=0x7f209c1980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.133+0000 7f209b59e640 1 --2- 172.21.15.98:0/547899364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f209c102a80 0x7f209c1980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.134+0000 7f209b59e640 1 -- 172.21.15.98:0/547899364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f209c101890 msgr2=0x7f209c197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.134+0000 7f209b59e640 1 --2- 172.21.15.98:0/547899364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f209c101890 0x7f209c197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.134+0000 7f209b59e640 1 -- 172.21.15.98:0/547899364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f209c19f7d0 con 0x7f209c103440 2024-05-09T08:50:28.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.134+0000 7f209b59e640 1 --2- 172.21.15.98:0/547899364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f209c103440 0x7f209c19f160 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f209400eb10 tx=0x7f209400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:28.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.134+0000 7f2083fff640 1 -- 172.21.15.98:0/547899364 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2094004b20 con 0x7f209c103440 2024-05-09T08:50:28.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.134+0000 7f20a124c640 1 -- 172.21.15.98:0/547899364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f209c19fac0 con 0x7f209c103440 2024-05-09T08:50:28.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.134+0000 7f2083fff640 1 -- 172.21.15.98:0/547899364 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2094004cc0 con 0x7f209c103440 2024-05-09T08:50:28.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.134+0000 7f2083fff640 1 -- 172.21.15.98:0/547899364 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2094010760 con 0x7f209c103440 2024-05-09T08:50:28.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.134+0000 7f20a124c640 1 -- 172.21.15.98:0/547899364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f209c1a0060 con 0x7f209c103440 2024-05-09T08:50:28.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.135+0000 7f20a124c640 1 -- 172.21.15.98:0/547899364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f209c108bb0 con 0x7f209c103440 2024-05-09T08:50:28.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.136+0000 7f2083fff640 1 -- 172.21.15.98:0/547899364 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2094005020 con 0x7f209c103440 2024-05-09T08:50:28.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.137+0000 7f2083fff640 1 --2- 172.21.15.98:0/547899364 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f20840a2b90 0x7f20840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:28.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.137+0000 7f209ad9d640 1 --2- 172.21.15.98:0/547899364 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f20840a2b90 0x7f20840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:28.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.137+0000 7f2083fff640 1 -- 172.21.15.98:0/547899364 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f20940cfa40 con 0x7f209c103440 2024-05-09T08:50:28.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.137+0000 7f209ad9d640 1 --2- 172.21.15.98:0/547899364 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f20840a2b90 0x7f20840a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f208c005ab0 tx=0x7f208c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:28.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.141+0000 7f2083fff640 1 -- 172.21.15.98:0/547899364 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f20940d1e20 con 0x7f209c103440 2024-05-09T08:50:28.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.318+0000 7f20a124c640 1 -- 172.21.15.98:0/547899364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f209c062430 con 0x7f209c103440 2024-05-09T08:50:28.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.318+0000 7f2083fff640 1 -- 172.21.15.98:0/547899364 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f209409c5d0 con 0x7f209c103440 2024-05-09T08:50:28.317 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-09T08:50:28.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.320+0000 7f20a124c640 1 -- 172.21.15.98:0/547899364 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f20840a2b90 msgr2=0x7f20840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.320+0000 7f20a124c640 1 --2- 172.21.15.98:0/547899364 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f20840a2b90 0x7f20840a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f208c005ab0 tx=0x7f208c000bc0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.320+0000 7f20a124c640 1 -- 172.21.15.98:0/547899364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f209c103440 msgr2=0x7f209c19f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.320+0000 7f20a124c640 1 --2- 172.21.15.98:0/547899364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f209c103440 0x7f209c19f160 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f209400eb10 tx=0x7f209400c2d0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.320+0000 7f20a124c640 1 -- 172.21.15.98:0/547899364 shutdown_connections 2024-05-09T08:50:28.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.320+0000 7f20a124c640 1 --2- 172.21.15.98:0/547899364 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f20840a2b90 0x7f20840a5050 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.320+0000 7f20a124c640 1 --2- 172.21.15.98:0/547899364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f209c103440 0x7f209c19f160 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.320+0000 7f20a124c640 1 --2- 172.21.15.98:0/547899364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f209c102a80 0x7f209c1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.320+0000 7f20a124c640 1 --2- 172.21.15.98:0/547899364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f209c101890 0x7f209c197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.320+0000 7f20a124c640 1 -- 172.21.15.98:0/547899364 >> 172.21.15.98:0/547899364 conn(0x7f209c0fce00 msgr2=0x7f209c105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:28.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.320+0000 7f20a124c640 1 -- 172.21.15.98:0/547899364 shutdown_connections 2024-05-09T08:50:28.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.320+0000 7f20a124c640 1 -- 172.21.15.98:0/547899364 wait complete. 2024-05-09T08:50:28.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:50:28.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.436+0000 7fe7c1961640 1 -- 172.21.15.98:0/3942289267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7bc1075c0 msgr2=0x7fe7bc1099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.436+0000 7fe7c1961640 1 --2- 172.21.15.98:0/3942289267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7bc1075c0 0x7fe7bc1099b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4009ca0 tx=0x7fe7b402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.436+0000 7fe7c1961640 1 -- 172.21.15.98:0/3942289267 shutdown_connections 2024-05-09T08:50:28.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.436+0000 7fe7c1961640 1 --2- 172.21.15.98:0/3942289267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7bc1075c0 0x7fe7bc1099b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.436+0000 7fe7c1961640 1 --2- 172.21.15.98:0/3942289267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7bc104c90 0x7fe7bc107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.436+0000 7fe7c1961640 1 --2- 172.21.15.98:0/3942289267 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7bc102360 0x7fe7bc104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.436+0000 7fe7c1961640 1 -- 172.21.15.98:0/3942289267 >> 172.21.15.98:0/3942289267 conn(0x7fe7bc0fbcf0 msgr2=0x7fe7bc0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:28.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.436+0000 7fe7c1961640 1 -- 172.21.15.98:0/3942289267 shutdown_connections 2024-05-09T08:50:28.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.436+0000 7fe7c1961640 1 -- 172.21.15.98:0/3942289267 wait complete. 2024-05-09T08:50:28.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.436+0000 7fe7c1961640 1 Processor -- start 2024-05-09T08:50:28.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7c1961640 1 -- start start 2024-05-09T08:50:28.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7c1961640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7bc102360 0x7fe7bc197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:28.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7c1961640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7bc104c90 0x7fe7bc198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:28.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7c1961640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7bc1075c0 0x7fe7bc19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:28.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7c1961640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc10d590 con 0x7fe7bc104c90 2024-05-09T08:50:28.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7c1961640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc10d410 con 0x7fe7bc1075c0 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7c1961640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc10d710 con 0x7fe7bc102360 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7ba7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7bc104c90 0x7fe7bc198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7ba7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7bc104c90 0x7fe7bc198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49930/0 (socket says 172.21.15.98:49930) 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7ba7fc640 1 -- 172.21.15.98:0/1525341547 learned_addr learned my addr 172.21.15.98:0/1525341547 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7baffd640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7bc102360 0x7fe7bc197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7ba7fc640 1 -- 172.21.15.98:0/1525341547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7bc102360 msgr2=0x7fe7bc197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7ba7fc640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7bc102360 0x7fe7bc197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7ba7fc640 1 -- 172.21.15.98:0/1525341547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7bc1075c0 msgr2=0x7fe7bc19f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7bb7fe640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7bc1075c0 0x7fe7bc19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7ba7fc640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7bc1075c0 0x7fe7bc19f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7ba7fc640 1 -- 172.21.15.98:0/1525341547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7bc19f840 con 0x7fe7bc104c90 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7baffd640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7bc102360 0x7fe7bc197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7ba7fc640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7bc104c90 0x7fe7bc198170 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000b450 tx=0x7fe7b000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7c095f640 1 -- 172.21.15.98:0/1525341547 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0015020 con 0x7fe7bc104c90 2024-05-09T08:50:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7c095f640 1 -- 172.21.15.98:0/1525341547 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7b0011d70 con 0x7fe7bc104c90 2024-05-09T08:50:28.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7c1961640 1 -- 172.21.15.98:0/1525341547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7bc19fb30 con 0x7fe7bc104c90 2024-05-09T08:50:28.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7c095f640 1 -- 172.21.15.98:0/1525341547 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b000f340 con 0x7fe7bc104c90 2024-05-09T08:50:28.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.437+0000 7fe7c1961640 1 -- 172.21.15.98:0/1525341547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7bc1a0080 con 0x7fe7bc104c90 2024-05-09T08:50:28.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.438+0000 7fe7c1961640 1 -- 172.21.15.98:0/1525341547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7bc0fd7e0 con 0x7fe7bc104c90 2024-05-09T08:50:28.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.439+0000 7fe7c095f640 1 -- 172.21.15.98:0/1525341547 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe7b0018020 con 0x7fe7bc104c90 2024-05-09T08:50:28.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.440+0000 7fe7c095f640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe7a40a2b90 0x7fe7a40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:28.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.440+0000 7fe7baffd640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe7a40a2b90 0x7fe7a40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:28.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.440+0000 7fe7c095f640 1 -- 172.21.15.98:0/1525341547 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fe7b00cefb0 con 0x7fe7bc104c90 2024-05-09T08:50:28.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.440+0000 7fe7baffd640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe7a40a2b90 0x7fe7a40a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac005fd0 tx=0x7fe7ac005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:28.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.444+0000 7fe7c095f640 1 -- 172.21.15.98:0/1525341547 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe7b001f020 con 0x7fe7bc104c90 2024-05-09T08:50:28.562 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:28 smithi142 ceph-mon[34625]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 7 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T08:50:28.563 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:28 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3698292960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:28.563 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:28 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2061490302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:28.563 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:28 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1285346927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:28.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:28 smithi098 ceph-mon[30092]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 7 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T08:50:28.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:28 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3698292960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:28.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:28 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2061490302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:28.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:28 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1285346927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.622+0000 7fe7c1961640 1 -- 172.21.15.98:0/1525341547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe7bc0624d0 con 0x7fe7bc104c90 2024-05-09T08:50:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.622+0000 7fe7c095f640 1 -- 172.21.15.98:0/1525341547 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fe7b000d040 con 0x7fe7bc104c90 2024-05-09T08:50:28.621 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:50:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.624+0000 7fe7c1961640 1 -- 172.21.15.98:0/1525341547 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe7a40a2b90 msgr2=0x7fe7a40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.624+0000 7fe7c1961640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe7a40a2b90 0x7fe7a40a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac005fd0 tx=0x7fe7ac005ea0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.624+0000 7fe7c1961640 1 -- 172.21.15.98:0/1525341547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7bc104c90 msgr2=0x7fe7bc198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.624+0000 7fe7c1961640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7bc104c90 0x7fe7bc198170 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000b450 tx=0x7fe7b000b910 comp rx=0 tx=0).stop 2024-05-09T08:50:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.624+0000 7fe7c1961640 1 -- 172.21.15.98:0/1525341547 shutdown_connections 2024-05-09T08:50:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.624+0000 7fe7c1961640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7bc1075c0 0x7fe7bc19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.624+0000 7fe7c1961640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fe7a40a2b90 0x7fe7a40a5050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.624+0000 7fe7c1961640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7bc104c90 0x7fe7bc198170 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.624+0000 7fe7c1961640 1 --2- 172.21.15.98:0/1525341547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7bc102360 0x7fe7bc197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.624+0000 7fe7c1961640 1 -- 172.21.15.98:0/1525341547 >> 172.21.15.98:0/1525341547 conn(0x7fe7bc0fbcf0 msgr2=0x7fe7bc102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.624+0000 7fe7c1961640 1 -- 172.21.15.98:0/1525341547 shutdown_connections 2024-05-09T08:50:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.624+0000 7fe7c1961640 1 -- 172.21.15.98:0/1525341547 wait complete. 2024-05-09T08:50:28.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:50:28.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:28 smithi105 ceph-mon[34498]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 7 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T08:50:28.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:28 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3698292960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:28.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:28 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2061490302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:28.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:28 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1285346927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 -- 172.21.15.98:0/3494344281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee0102370 msgr2=0x7fcee0104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 --2- 172.21.15.98:0/3494344281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee0102370 0x7fcee0104760 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fced0009900 tx=0x7fced002a410 comp rx=0 tx=0).stop 2024-05-09T08:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 -- 172.21.15.98:0/3494344281 shutdown_connections 2024-05-09T08:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 --2- 172.21.15.98:0/3494344281 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcee01075d0 0x7fcee01099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 --2- 172.21.15.98:0/3494344281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee0104ca0 0x7fcee0107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 --2- 172.21.15.98:0/3494344281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee0102370 0x7fcee0104760 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 -- 172.21.15.98:0/3494344281 >> 172.21.15.98:0/3494344281 conn(0x7fcee00fbd20 msgr2=0x7fcee00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 -- 172.21.15.98:0/3494344281 shutdown_connections 2024-05-09T08:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 -- 172.21.15.98:0/3494344281 wait complete. 2024-05-09T08:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 Processor -- start 2024-05-09T08:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 -- start start 2024-05-09T08:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcee0102370 0x7fcee0197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee0104ca0 0x7fcee0198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee01075d0 0x7fcee019f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcee010d550 con 0x7fcee01075d0 2024-05-09T08:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcee010d3d0 con 0x7fcee0104ca0 2024-05-09T08:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.740+0000 7fcee6b1d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcee010d6d0 con 0x7fcee0102370 2024-05-09T08:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fcee50ba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee01075d0 0x7fcee019f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fcee50ba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee01075d0 0x7fcee019f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49948/0 (socket says 172.21.15.98:49948) 2024-05-09T08:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fcee50ba640 1 -- 172.21.15.98:0/2755125832 learned_addr learned my addr 172.21.15.98:0/2755125832 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fcee48b9640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcee0102370 0x7fcee0197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fcee50ba640 1 -- 172.21.15.98:0/2755125832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcee0102370 msgr2=0x7fcee0197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fcee50ba640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcee0102370 0x7fcee0197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fcedffff640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee0104ca0 0x7fcee0198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:28.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fcee50ba640 1 -- 172.21.15.98:0/2755125832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee0104ca0 msgr2=0x7fcee0198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fcee50ba640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee0104ca0 0x7fcee0198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fcee50ba640 1 -- 172.21.15.98:0/2755125832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee019f830 con 0x7fcee01075d0 2024-05-09T08:50:28.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fcee48b9640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcee0102370 0x7fcee0197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:28.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fcee50ba640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee01075d0 0x7fcee019f190 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fced800c970 tx=0x7fced800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:28.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fceddffb640 1 -- 172.21.15.98:0/2755125832 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8007bb0 con 0x7fcee01075d0 2024-05-09T08:50:28.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fceddffb640 1 -- 172.21.15.98:0/2755125832 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced8007d50 con 0x7fcee01075d0 2024-05-09T08:50:28.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fcee6b1d640 1 -- 172.21.15.98:0/2755125832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcee019fb20 con 0x7fcee01075d0 2024-05-09T08:50:28.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fceddffb640 1 -- 172.21.15.98:0/2755125832 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced800f450 con 0x7fcee01075d0 2024-05-09T08:50:28.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.741+0000 7fcee6b1d640 1 -- 172.21.15.98:0/2755125832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcee01a0020 con 0x7fcee01075d0 2024-05-09T08:50:28.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.742+0000 7fcee6b1d640 1 -- 172.21.15.98:0/2755125832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcee00fd810 con 0x7fcee01075d0 2024-05-09T08:50:28.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.743+0000 7fceddffb640 1 -- 172.21.15.98:0/2755125832 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fced8004f60 con 0x7fcee01075d0 2024-05-09T08:50:28.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.744+0000 7fceddffb640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcecc0a2b90 0x7fcecc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:28.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.744+0000 7fcee48b9640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcecc0a2b90 0x7fcecc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:28.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.744+0000 7fceddffb640 1 -- 172.21.15.98:0/2755125832 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fced80cf2c0 con 0x7fcee01075d0 2024-05-09T08:50:28.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.745+0000 7fcee48b9640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcecc0a2b90 0x7fcecc0a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fced0006010 tx=0x7fced0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:28.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.749+0000 7fceddffb640 1 -- 172.21.15.98:0/2755125832 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fced8014030 con 0x7fcee01075d0 2024-05-09T08:50:28.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.922+0000 7fcee6b1d640 1 -- 172.21.15.98:0/2755125832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcee0062430 con 0x7fcee01075d0 2024-05-09T08:50:28.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.922+0000 7fceddffb640 1 -- 172.21.15.98:0/2755125832 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fced809be50 con 0x7fcee01075d0 2024-05-09T08:50:28.922 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-09T08:50:28.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.924+0000 7fcee6b1d640 1 -- 172.21.15.98:0/2755125832 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcecc0a2b90 msgr2=0x7fcecc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.924+0000 7fcee6b1d640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcecc0a2b90 0x7fcecc0a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fced0006010 tx=0x7fced0000bc0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.924+0000 7fcee6b1d640 1 -- 172.21.15.98:0/2755125832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee01075d0 msgr2=0x7fcee019f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:28.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.924+0000 7fcee6b1d640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee01075d0 0x7fcee019f190 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fced800c970 tx=0x7fced800ce30 comp rx=0 tx=0).stop 2024-05-09T08:50:28.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.924+0000 7fcee6b1d640 1 -- 172.21.15.98:0/2755125832 shutdown_connections 2024-05-09T08:50:28.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.924+0000 7fcee6b1d640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fcecc0a2b90 0x7fcecc0a5050 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.924+0000 7fcee6b1d640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcee01075d0 0x7fcee019f190 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.924+0000 7fcee6b1d640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee0104ca0 0x7fcee0198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.924+0000 7fcee6b1d640 1 --2- 172.21.15.98:0/2755125832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcee0102370 0x7fcee0197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:28.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.924+0000 7fcee6b1d640 1 -- 172.21.15.98:0/2755125832 >> 172.21.15.98:0/2755125832 conn(0x7fcee00fbd20 msgr2=0x7fcee0102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:28.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.925+0000 7fcee6b1d640 1 -- 172.21.15.98:0/2755125832 shutdown_connections 2024-05-09T08:50:28.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:28.925+0000 7fcee6b1d640 1 -- 172.21.15.98:0/2755125832 wait complete. 2024-05-09T08:50:28.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:50:29.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.039+0000 7f49772f7640 1 -- 172.21.15.98:0/3869846254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4970102340 msgr2=0x7f4970108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.039+0000 7f49772f7640 1 --2- 172.21.15.98:0/3869846254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4970102340 0x7f4970108bd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4968009ca0 tx=0x7f496802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.039+0000 7f49772f7640 1 -- 172.21.15.98:0/3869846254 shutdown_connections 2024-05-09T08:50:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.039+0000 7f49772f7640 1 --2- 172.21.15.98:0/3869846254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4970102340 0x7f4970108bd0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.039+0000 7f49772f7640 1 --2- 172.21.15.98:0/3869846254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4970101980 0x7f4970101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.039+0000 7f49772f7640 1 --2- 172.21.15.98:0/3869846254 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4970100790 0x7f4970100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.039+0000 7f49772f7640 1 -- 172.21.15.98:0/3869846254 >> 172.21.15.98:0/3869846254 conn(0x7f49700fbd20 msgr2=0x7f49700fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.039+0000 7f49772f7640 1 -- 172.21.15.98:0/3869846254 shutdown_connections 2024-05-09T08:50:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.039+0000 7f49772f7640 1 -- 172.21.15.98:0/3869846254 wait complete. 2024-05-09T08:50:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f49772f7640 1 Processor -- start 2024-05-09T08:50:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f49772f7640 1 -- start start 2024-05-09T08:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f49772f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4970100790 0x7f497006a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f49772f7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4970101980 0x7f497019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f49772f7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4970102340 0x7f497019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f49772f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f497010c410 con 0x7f4970100790 2024-05-09T08:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f49772f7640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f497010c290 con 0x7f4970102340 2024-05-09T08:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f49772f7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f497010c590 con 0x7f4970101980 2024-05-09T08:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f4975093640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4970100790 0x7f497006a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f4975093640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4970100790 0x7f497006a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49966/0 (socket says 172.21.15.98:49966) 2024-05-09T08:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f4975093640 1 -- 172.21.15.98:0/140188730 learned_addr learned my addr 172.21.15.98:0/140188730 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f4974892640 1 --2- 172.21.15.98:0/140188730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4970101980 0x7f497019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f4975093640 1 -- 172.21.15.98:0/140188730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4970101980 msgr2=0x7f497019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f4975093640 1 --2- 172.21.15.98:0/140188730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4970101980 0x7f497019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f4975093640 1 -- 172.21.15.98:0/140188730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4970102340 msgr2=0x7f497019eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f4975093640 1 --2- 172.21.15.98:0/140188730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4970102340 0x7f497019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f4975093640 1 -- 172.21.15.98:0/140188730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497019f7b0 con 0x7f4970100790 2024-05-09T08:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f4975093640 1 --2- 172.21.15.98:0/140188730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4970100790 0x7f497006a9e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f496c00d770 tx=0x7f496c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.040+0000 7f49667fc640 1 -- 172.21.15.98:0/140188730 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c017070 con 0x7f4970100790 2024-05-09T08:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.041+0000 7f49772f7640 1 -- 172.21.15.98:0/140188730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f497019fa40 con 0x7f4970100790 2024-05-09T08:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.041+0000 7f49667fc640 1 -- 172.21.15.98:0/140188730 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f496c009c30 con 0x7f4970100790 2024-05-09T08:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.041+0000 7f49667fc640 1 -- 172.21.15.98:0/140188730 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c003dd0 con 0x7f4970100790 2024-05-09T08:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.041+0000 7f49772f7640 1 -- 172.21.15.98:0/140188730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f497019ffe0 con 0x7f4970100790 2024-05-09T08:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.041+0000 7f49772f7640 1 -- 172.21.15.98:0/140188730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4970068aa0 con 0x7f4970100790 2024-05-09T08:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.043+0000 7f49667fc640 1 -- 172.21.15.98:0/140188730 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f496c003f70 con 0x7f4970100790 2024-05-09T08:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.043+0000 7f49667fc640 1 --2- 172.21.15.98:0/140188730 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f495c0a2b90 0x7f495c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.044+0000 7f4974892640 1 --2- 172.21.15.98:0/140188730 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f495c0a2b90 0x7f495c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.044+0000 7f49667fc640 1 -- 172.21.15.98:0/140188730 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f496c0cedd0 con 0x7f4970100790 2024-05-09T08:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.044+0000 7f4974892640 1 --2- 172.21.15.98:0/140188730 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f495c0a2b90 0x7f495c0a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f497006a750 tx=0x7f496000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.048+0000 7f49667fc640 1 -- 172.21.15.98:0/140188730 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f496c0d2030 con 0x7f4970100790 2024-05-09T08:50:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.229+0000 7f49772f7640 1 -- 172.21.15.98:0/140188730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4970062430 con 0x7f4970100790 2024-05-09T08:50:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.229+0000 7f49667fc640 1 -- 172.21.15.98:0/140188730 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f496c09b960 con 0x7f4970100790 2024-05-09T08:50:29.228 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.231+0000 7f49772f7640 1 -- 172.21.15.98:0/140188730 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f495c0a2b90 msgr2=0x7f495c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.231+0000 7f49772f7640 1 --2- 172.21.15.98:0/140188730 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f495c0a2b90 0x7f495c0a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f497006a750 tx=0x7f496000b390 comp rx=0 tx=0).stop 2024-05-09T08:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.231+0000 7f49772f7640 1 -- 172.21.15.98:0/140188730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4970100790 msgr2=0x7f497006a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.231+0000 7f49772f7640 1 --2- 172.21.15.98:0/140188730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4970100790 0x7f497006a9e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f496c00d770 tx=0x7f496c00dc30 comp rx=0 tx=0).stop 2024-05-09T08:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.231+0000 7f49772f7640 1 -- 172.21.15.98:0/140188730 shutdown_connections 2024-05-09T08:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.231+0000 7f49772f7640 1 --2- 172.21.15.98:0/140188730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4970102340 0x7f497019eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.231+0000 7f49772f7640 1 --2- 172.21.15.98:0/140188730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4970101980 0x7f497019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.231+0000 7f49772f7640 1 --2- 172.21.15.98:0/140188730 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f495c0a2b90 0x7f495c0a5050 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.231+0000 7f49772f7640 1 --2- 172.21.15.98:0/140188730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4970100790 0x7f497006a9e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.231+0000 7f49772f7640 1 -- 172.21.15.98:0/140188730 >> 172.21.15.98:0/140188730 conn(0x7f49700fbd20 msgr2=0x7f4970104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.231+0000 7f49772f7640 1 -- 172.21.15.98:0/140188730 shutdown_connections 2024-05-09T08:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.231+0000 7f49772f7640 1 -- 172.21.15.98:0/140188730 wait complete. 2024-05-09T08:50:29.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:50:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.346+0000 7f50d642c640 1 -- 172.21.15.98:0/4125530024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50d0102340 msgr2=0x7f50d0108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.346+0000 7f50d642c640 1 --2- 172.21.15.98:0/4125530024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50d0102340 0x7f50d0108bd0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f50c8009ca0 tx=0x7f50c802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.346+0000 7f50d642c640 1 -- 172.21.15.98:0/4125530024 shutdown_connections 2024-05-09T08:50:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.346+0000 7f50d642c640 1 --2- 172.21.15.98:0/4125530024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50d0102340 0x7f50d0108bd0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.346+0000 7f50d642c640 1 --2- 172.21.15.98:0/4125530024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50d0101980 0x7f50d0101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.346+0000 7f50d642c640 1 --2- 172.21.15.98:0/4125530024 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50d0100790 0x7f50d0100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.346+0000 7f50d642c640 1 -- 172.21.15.98:0/4125530024 >> 172.21.15.98:0/4125530024 conn(0x7f50d00fbd20 msgr2=0x7f50d00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.346+0000 7f50d642c640 1 -- 172.21.15.98:0/4125530024 shutdown_connections 2024-05-09T08:50:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.346+0000 7f50d642c640 1 -- 172.21.15.98:0/4125530024 wait complete. 2024-05-09T08:50:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.346+0000 7f50d642c640 1 Processor -- start 2024-05-09T08:50:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d642c640 1 -- start start 2024-05-09T08:50:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d642c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50d0100790 0x7f50d006aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d642c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50d0101980 0x7f50d019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d642c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50d0102340 0x7f50d019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d642c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50d010c460 con 0x7f50d0102340 2024-05-09T08:50:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d642c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f50d010c2e0 con 0x7f50d0101980 2024-05-09T08:50:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d642c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f50d010c5e0 con 0x7f50d0100790 2024-05-09T08:50:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d49c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50d0102340 0x7f50d019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d49c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50d0102340 0x7f50d019eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49978/0 (socket says 172.21.15.98:49978) 2024-05-09T08:50:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d49c9640 1 -- 172.21.15.98:0/3770834449 learned_addr learned my addr 172.21.15.98:0/3770834449 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50cffff640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50d0100790 0x7f50d006aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50c7fff640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50d0101980 0x7f50d019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d49c9640 1 -- 172.21.15.98:0/3770834449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50d0100790 msgr2=0x7f50d006aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d49c9640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50d0100790 0x7f50d006aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d49c9640 1 -- 172.21.15.98:0/3770834449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50d0101980 msgr2=0x7f50d019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d49c9640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50d0101980 0x7f50d019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d49c9640 1 -- 172.21.15.98:0/3770834449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d019f770 con 0x7f50d0102340 2024-05-09T08:50:29.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50c7fff640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50d0101980 0x7f50d019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:29.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d49c9640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50d0102340 0x7f50d019eeb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f50c80319f0 tx=0x7f50c802f450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:29.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50cdffb640 1 -- 172.21.15.98:0/3770834449 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c8003c70 con 0x7f50d0102340 2024-05-09T08:50:29.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50cdffb640 1 -- 172.21.15.98:0/3770834449 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50c8003e10 con 0x7f50d0102340 2024-05-09T08:50:29.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50d642c640 1 -- 172.21.15.98:0/3770834449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d019fa00 con 0x7f50d0102340 2024-05-09T08:50:29.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.347+0000 7f50cdffb640 1 -- 172.21.15.98:0/3770834449 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c802fd30 con 0x7f50d0102340 2024-05-09T08:50:29.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.348+0000 7f50d642c640 1 -- 172.21.15.98:0/3770834449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d019ff50 con 0x7f50d0102340 2024-05-09T08:50:29.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.348+0000 7f50d642c640 1 -- 172.21.15.98:0/3770834449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d0107a50 con 0x7f50d0102340 2024-05-09T08:50:29.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.350+0000 7f50cdffb640 1 -- 172.21.15.98:0/3770834449 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f50c8004b40 con 0x7f50d0102340 2024-05-09T08:50:29.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.350+0000 7f50cdffb640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f50bc0a2b90 0x7f50bc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.351+0000 7f50cffff640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f50bc0a2b90 0x7f50bc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:29.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.351+0000 7f50cdffb640 1 -- 172.21.15.98:0/3770834449 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f50c80f0bc0 con 0x7f50d0102340 2024-05-09T08:50:29.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.351+0000 7f50cffff640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f50bc0a2b90 0x7f50bc0a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f50c00058c0 tx=0x7f50c000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.354+0000 7f50cdffb640 1 -- 172.21.15.98:0/3770834449 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f50c80bd750 con 0x7f50d0102340 2024-05-09T08:50:29.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.567+0000 7f50d642c640 1 -- 172.21.15.98:0/3770834449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50d002d380 con 0x7f50d0102340 2024-05-09T08:50:29.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.568+0000 7f50cdffb640 1 -- 172.21.15.98:0/3770834449 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11940 (secure 0 0 0) 0x7f50c80bd050 con 0x7f50d0102340 2024-05-09T08:50:29.567 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:29.568 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:48:39.758226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24457,24461,24388,24439,14667]},"id":1}]} 2024-05-09T08:50:29.568 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-05-09T08:50:29.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.570+0000 7f50d642c640 1 -- 172.21.15.98:0/3770834449 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f50bc0a2b90 msgr2=0x7f50bc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.570+0000 7f50d642c640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f50bc0a2b90 0x7f50bc0a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f50c00058c0 tx=0x7f50c000a2d0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.570+0000 7f50d642c640 1 -- 172.21.15.98:0/3770834449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50d0102340 msgr2=0x7f50d019eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.570+0000 7f50d642c640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50d0102340 0x7f50d019eeb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f50c80319f0 tx=0x7f50c802f450 comp rx=0 tx=0).stop 2024-05-09T08:50:29.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.570+0000 7f50d642c640 1 -- 172.21.15.98:0/3770834449 shutdown_connections 2024-05-09T08:50:29.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.570+0000 7f50d642c640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f50bc0a2b90 0x7f50bc0a5050 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.570+0000 7f50d642c640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50d0102340 0x7f50d019eeb0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.570+0000 7f50d642c640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50d0101980 0x7f50d019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.570+0000 7f50d642c640 1 --2- 172.21.15.98:0/3770834449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50d0100790 0x7f50d006aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.570+0000 7f50d642c640 1 -- 172.21.15.98:0/3770834449 >> 172.21.15.98:0/3770834449 conn(0x7f50d00fbd20 msgr2=0x7f50d0104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:29.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.571+0000 7f50d642c640 1 -- 172.21.15.98:0/3770834449 shutdown_connections 2024-05-09T08:50:29.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.571+0000 7f50d642c640 1 -- 172.21.15.98:0/3770834449 wait complete. 2024-05-09T08:50:29.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:29 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/547899364' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:29.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:29 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1525341547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:29.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:29 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2755125832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:29.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:29 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/140188730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:29.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:50:29.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:29 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/547899364' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:29.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:29 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1525341547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:29.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:29 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2755125832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:29.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:29 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/140188730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:29.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.683+0000 7ff5cef14640 1 -- 172.21.15.98:0/2086362401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5c8102a90 msgr2=0x7ff5c8102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.683+0000 7ff5cef14640 1 --2- 172.21.15.98:0/2086362401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5c8102a90 0x7ff5c8102f10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009900 tx=0x7ff5bc02a410 comp rx=0 tx=0).stop 2024-05-09T08:50:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 -- 172.21.15.98:0/2086362401 shutdown_connections 2024-05-09T08:50:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 --2- 172.21.15.98:0/2086362401 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5c8103450 0x7ff5c8109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 --2- 172.21.15.98:0/2086362401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5c8102a90 0x7ff5c8102f10 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 --2- 172.21.15.98:0/2086362401 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5c81018a0 0x7ff5c8101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 -- 172.21.15.98:0/2086362401 >> 172.21.15.98:0/2086362401 conn(0x7ff5c80fce10 msgr2=0x7ff5c80ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 -- 172.21.15.98:0/2086362401 shutdown_connections 2024-05-09T08:50:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 -- 172.21.15.98:0/2086362401 wait complete. 2024-05-09T08:50:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 Processor -- start 2024-05-09T08:50:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 -- start start 2024-05-09T08:50:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5c81018a0 0x7ff5c8197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5c8102a90 0x7ff5c81980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5c8103450 0x7ff5c819f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c810d500 con 0x7ff5c8102a90 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c810d380 con 0x7ff5c81018a0 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5cef14640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c810d680 con 0x7ff5c8103450 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5c7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5c8102a90 0x7ff5c81980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5c7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5c8102a90 0x7ff5c81980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49984/0 (socket says 172.21.15.98:49984) 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.684+0000 7ff5c7fff640 1 -- 172.21.15.98:0/1193691523 learned_addr learned my addr 172.21.15.98:0/1193691523 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.685+0000 7ff5cd4b1640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5c8103450 0x7ff5c819f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.685+0000 7ff5cccb0640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5c81018a0 0x7ff5c8197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.685+0000 7ff5c7fff640 1 -- 172.21.15.98:0/1193691523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5c8103450 msgr2=0x7ff5c819f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.685+0000 7ff5c7fff640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5c8103450 0x7ff5c819f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.685+0000 7ff5c7fff640 1 -- 172.21.15.98:0/1193691523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5c81018a0 msgr2=0x7ff5c8197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.685+0000 7ff5c7fff640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5c81018a0 0x7ff5c8197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.685+0000 7ff5c7fff640 1 -- 172.21.15.98:0/1193691523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5c819f7c0 con 0x7ff5c8102a90 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.685+0000 7ff5cd4b1640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5c8103450 0x7ff5c819f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.685+0000 7ff5c7fff640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5c8102a90 0x7ff5c81980e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009230 tx=0x7ff5bc004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:29.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.685+0000 7ff5c5ffb640 1 -- 172.21.15.98:0/1193691523 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc03f030 con 0x7ff5c8102a90 2024-05-09T08:50:29.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.685+0000 7ff5cef14640 1 -- 172.21.15.98:0/1193691523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5c819fa50 con 0x7ff5c8102a90 2024-05-09T08:50:29.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.685+0000 7ff5c5ffb640 1 -- 172.21.15.98:0/1193691523 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5bc003ce0 con 0x7ff5c8102a90 2024-05-09T08:50:29.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.685+0000 7ff5c5ffb640 1 -- 172.21.15.98:0/1193691523 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc035070 con 0x7ff5c8102a90 2024-05-09T08:50:29.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.685+0000 7ff5cef14640 1 -- 172.21.15.98:0/1193691523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5c819ff40 con 0x7ff5c8102a90 2024-05-09T08:50:29.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.686+0000 7ff5cef14640 1 -- 172.21.15.98:0/1193691523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5c8108c40 con 0x7ff5c8102a90 2024-05-09T08:50:29.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.688+0000 7ff5c5ffb640 1 -- 172.21.15.98:0/1193691523 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff5bc042030 con 0x7ff5c8102a90 2024-05-09T08:50:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.689+0000 7ff5c5ffb640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff5b00a2b90 0x7ff5b00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.689+0000 7ff5cccb0640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff5b00a2b90 0x7ff5b00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.689+0000 7ff5c5ffb640 1 -- 172.21.15.98:0/1193691523 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7ff5bc0efe90 con 0x7ff5c8102a90 2024-05-09T08:50:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.689+0000 7ff5cccb0640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff5b00a2b90 0x7ff5b00a5050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8005fd0 tx=0x7ff5b8005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.692+0000 7ff5c5ffb640 1 -- 172.21.15.98:0/1193691523 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff5bc0bca20 con 0x7ff5c8102a90 2024-05-09T08:50:29.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:29 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/547899364' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:29.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:29 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1525341547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:29.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:29 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2755125832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:29.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:29 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/140188730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:29.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.873+0000 7ff5cef14640 1 -- 172.21.15.98:0/1193691523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5c802d390 con 0x7ff5c8102a90 2024-05-09T08:50:29.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.874+0000 7ff5c5ffb640 1 -- 172.21.15.98:0/1193691523 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24829 (secure 0 0 0) 0x7ff5bc0bc320 con 0x7ff5c8102a90 2024-05-09T08:50:29.874 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:29.874 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":94,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:50:19.278344+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":1}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:50:29.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.876+0000 7ff5cef14640 1 -- 172.21.15.98:0/1193691523 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff5b00a2b90 msgr2=0x7ff5b00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.876+0000 7ff5cef14640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff5b00a2b90 0x7ff5b00a5050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8005fd0 tx=0x7ff5b8005850 comp rx=0 tx=0).stop 2024-05-09T08:50:29.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.876+0000 7ff5cef14640 1 -- 172.21.15.98:0/1193691523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5c8102a90 msgr2=0x7ff5c81980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.876+0000 7ff5cef14640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5c8102a90 0x7ff5c81980e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009230 tx=0x7ff5bc004360 comp rx=0 tx=0).stop 2024-05-09T08:50:29.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.877+0000 7ff5cef14640 1 -- 172.21.15.98:0/1193691523 shutdown_connections 2024-05-09T08:50:29.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.877+0000 7ff5cef14640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5c8103450 0x7ff5c819f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.877+0000 7ff5cef14640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7ff5b00a2b90 0x7ff5b00a5050 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.877+0000 7ff5cef14640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5c8102a90 0x7ff5c81980e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.877+0000 7ff5cef14640 1 --2- 172.21.15.98:0/1193691523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5c81018a0 0x7ff5c8197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.877+0000 7ff5cef14640 1 -- 172.21.15.98:0/1193691523 >> 172.21.15.98:0/1193691523 conn(0x7ff5c80fce10 msgr2=0x7ff5c8105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:29.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.877+0000 7ff5cef14640 1 -- 172.21.15.98:0/1193691523 shutdown_connections 2024-05-09T08:50:29.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.877+0000 7ff5cef14640 1 -- 172.21.15.98:0/1193691523 wait complete. 2024-05-09T08:50:29.891 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-09T08:48:17.933763+0000', 'modified': '2024-05-09T08:48:39.758226+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24461, 'mds_1': 24457, 'mds_2': 24439, 'mds_3': 24388, 'mds_4': 14667}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14637': {'gid': 14637, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6839/3981776050', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 3981776050}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 3981776050}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14649': {'gid': 14649, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6841/2714016730', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6840', 'nonce': 2714016730}, {'type': 'v1', 'addr': '172.21.15.98:6841', 'nonce': 2714016730}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14667': {'gid': 14667, 'name': 'b', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.105:6839/1933710170', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6838', 'nonce': 1933710170}, {'type': 'v1', 'addr': '172.21.15.105:6839', 'nonce': 1933710170}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24388': {'gid': 24388, 'name': 'l', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.142:6833/606414638', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6832', 'nonce': 606414638}, {'type': 'v1', 'addr': '172.21.15.142:6833', 'nonce': 606414638}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24398': {'gid': 24398, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.105:6833/1472919356', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6832', 'nonce': 1472919356}, {'type': 'v1', 'addr': '172.21.15.105:6833', 'nonce': 1472919356}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24421': {'gid': 24421, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.142:6835/2791041858', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6834', 'nonce': 2791041858}, {'type': 'v1', 'addr': '172.21.15.142:6835', 'nonce': 2791041858}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24428': {'gid': 24428, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.105:6835/588012490', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6834', 'nonce': 588012490}, {'type': 'v1', 'addr': '172.21.15.105:6835', 'nonce': 588012490}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24439': {'gid': 24439, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.142:6837/1837318616', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6836', 'nonce': 1837318616}, {'type': 'v1', 'addr': '172.21.15.142:6837', 'nonce': 1837318616}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.142:6839/3036634959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6838', 'nonce': 3036634959}, {'type': 'v1', 'addr': '172.21.15.142:6839', 'nonce': 3036634959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'h', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.105:6837/992088454', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6836', 'nonce': 992088454}, {'type': 'v1', 'addr': '172.21.15.105:6837', 'nonce': 992088454}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24461, 'qdb_cluster': [24457, 24461, 24388, 24439, 14667]} 2024-05-09T08:50:29.891 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-05-09T08:50:29.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-09T08:50:29.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 -- 172.21.15.98:0/2367383149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2760104ce0 msgr2=0x7f27601070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 --2- 172.21.15.98:0/2367383149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2760104ce0 0x7f27601070d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2754009900 tx=0x7f275402a410 comp rx=0 tx=0).stop 2024-05-09T08:50:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 -- 172.21.15.98:0/2367383149 shutdown_connections 2024-05-09T08:50:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 --2- 172.21.15.98:0/2367383149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2760107610 0x7f2760109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 --2- 172.21.15.98:0/2367383149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2760104ce0 0x7f27601070d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 --2- 172.21.15.98:0/2367383149 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27601023b0 0x7f27601047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 -- 172.21.15.98:0/2367383149 >> 172.21.15.98:0/2367383149 conn(0x7f27600fbce0 msgr2=0x7f27600fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 -- 172.21.15.98:0/2367383149 shutdown_connections 2024-05-09T08:50:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 -- 172.21.15.98:0/2367383149 wait complete. 2024-05-09T08:50:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 Processor -- start 2024-05-09T08:50:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 -- start start 2024-05-09T08:50:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27601023b0 0x7f2760197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2760104ce0 0x7f27601980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2760107610 0x7f276019f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f276010d4d0 con 0x7f27601023b0 2024-05-09T08:50:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f276010d350 con 0x7f2760107610 2024-05-09T08:50:29.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.995+0000 7f27673c8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f276010d650 con 0x7f2760104ce0 2024-05-09T08:50:29.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f2765164640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27601023b0 0x7f2760197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:29.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f2765164640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27601023b0 0x7f2760197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50002/0 (socket says 172.21.15.98:50002) 2024-05-09T08:50:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f2765164640 1 -- 172.21.15.98:0/1605413278 learned_addr learned my addr 172.21.15.98:0/1605413278 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f2765965640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2760107610 0x7f276019f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f2764963640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2760104ce0 0x7f27601980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f2765164640 1 -- 172.21.15.98:0/1605413278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2760104ce0 msgr2=0x7f27601980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f2765164640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2760104ce0 0x7f27601980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f2765164640 1 -- 172.21.15.98:0/1605413278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2760107610 msgr2=0x7f276019f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f2765164640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2760107610 0x7f276019f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f2765164640 1 -- 172.21.15.98:0/1605413278 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f276019f7c0 con 0x7f27601023b0 2024-05-09T08:50:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f2764963640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2760104ce0 0x7f27601980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f2765164640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27601023b0 0x7f2760197ba0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f275800d950 tx=0x7f275800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f275e7fc640 1 -- 172.21.15.98:0/1605413278 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2758017070 con 0x7f27601023b0 2024-05-09T08:50:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f275e7fc640 1 -- 172.21.15.98:0/1605413278 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2758009c30 con 0x7f27601023b0 2024-05-09T08:50:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f275e7fc640 1 -- 172.21.15.98:0/1605413278 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275800f6b0 con 0x7f27601023b0 2024-05-09T08:50:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f27673c8640 1 -- 172.21.15.98:0/1605413278 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f276019fab0 con 0x7f27601023b0 2024-05-09T08:50:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.996+0000 7f27673c8640 1 -- 172.21.15.98:0/1605413278 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27601a0050 con 0x7f27601023b0 2024-05-09T08:50:30.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.998+0000 7f275e7fc640 1 -- 172.21.15.98:0/1605413278 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f27580109d0 con 0x7f27601023b0 2024-05-09T08:50:30.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f276a1b3740 1 Processor -- start 2024-05-09T08:50:30.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f276a1b3740 1 -- start start 2024-05-09T08:50:30.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f276a1b3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3227ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f276a1b3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a32282b0 0x55a9a322cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f276a1b3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a322d420 0x55a9a322f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f276a1b3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a9a318b4d0 con 0x55a9a32282b0 2024-05-09T08:50:30.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f276a1b3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a9a31333e0 con 0x55a9a32278c0 2024-05-09T08:50:30.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f276a1b3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a9a3187490 con 0x55a9a322d420 2024-05-09T08:50:30.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f2765164640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a32282b0 0x55a9a322cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f2765164640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a32282b0 0x55a9a322cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50014/0 (socket says 172.21.15.98:50014) 2024-05-09T08:50:30.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f2765164640 1 -- 172.21.15.98:0/3002636288 learned_addr learned my addr 172.21.15.98:0/3002636288 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f2765965640 1 --2- 172.21.15.98:0/3002636288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3227ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f275e7fc640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f27480a2c10 0x7f27480a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f2764963640 1 --2- 172.21.15.98:0/3002636288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a322d420 0x55a9a322f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f2765965640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f27480a2c10 0x7f27480a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f2765164640 1 -- 172.21.15.98:0/3002636288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a322d420 msgr2=0x55a9a322f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f2765164640 1 --2- 172.21.15.98:0/3002636288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a322d420 0x55a9a322f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f2765164640 1 -- 172.21.15.98:0/3002636288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 msgr2=0x55a9a3227ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:29.999+0000 7f2765164640 1 --2- 172.21.15.98:0/3002636288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3227ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.000+0000 7f275e7fc640 1 -- 172.21.15.98:0/1605413278 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f27580cf0b0 con 0x7f27601023b0 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.000+0000 7f2765164640 1 -- 172.21.15.98:0/3002636288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9a2e9f2d0 con 0x55a9a32282b0 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.000+0000 7f2765965640 1 --2- 172.21.15.98:0/3002636288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3227ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.000+0000 7f2765164640 1 --2- 172.21.15.98:0/3002636288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a32282b0 0x55a9a322cdb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f27580979a0 tx=0x7f2758005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3165ec94ef87580a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.000+0000 7f2765965640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f27480a2c10 0x7f27480a50d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55a9a3228b10 tx=0x7f2750004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.000+0000 7f27477fe640 1 -- 172.21.15.98:0/3002636288 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2758020030 con 0x55a9a32282b0 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.000+0000 7f27477fe640 1 -- 172.21.15.98:0/3002636288 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27580bb050 con 0x55a9a32282b0 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.000+0000 7f27477fe640 1 -- 172.21.15.98:0/3002636288 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275809b540 con 0x55a9a32282b0 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 -- 172.21.15.98:0/3002636288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a32282b0 msgr2=0x55a9a322cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 --2- 172.21.15.98:0/3002636288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a32282b0 0x55a9a322cdb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f27580979a0 tx=0x7f2758005880 comp rx=0 tx=0).stop 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 -- 172.21.15.98:0/3002636288 shutdown_connections 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 --2- 172.21.15.98:0/3002636288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a322d420 0x55a9a322f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 --2- 172.21.15.98:0/3002636288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a32282b0 0x55a9a322cdb0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 --2- 172.21.15.98:0/3002636288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3227ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 -- 172.21.15.98:0/3002636288 >> 172.21.15.98:0/3002636288 conn(0x55a9a3222f30 msgr2=0x55a9a3223330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 -- 172.21.15.98:0/3002636288 shutdown_connections 2024-05-09T08:50:30.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 -- 172.21.15.98:0/3002636288 wait complete. 2024-05-09T08:50:30.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 -- start start 2024-05-09T08:50:30.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 Processor -- start 2024-05-09T08:50:30.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3231bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32282b0 0x55a9a3232130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3239170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a9a3230310 con 0x55a9a322d420 2024-05-09T08:50:30.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a9a31468e0 con 0x55a9a32278c0 2024-05-09T08:50:30.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.001+0000 7f276a1b3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a9a3230490 con 0x55a9a32282b0 2024-05-09T08:50:30.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f276a1b3740 10 client.0 _ll_register_callbacks cb 0x55a9a31139c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:30.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2765965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3239170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2765965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3239170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50024/0 (socket says 172.21.15.98:50024) 2024-05-09T08:50:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2765965640 1 -- 172.21.15.98:0/2105009787 learned_addr learned my addr 172.21.15.98:0/2105009787 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2764963640 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32282b0 0x55a9a3232130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2765164640 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3231bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2765965640 1 -- 172.21.15.98:0/2105009787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32282b0 msgr2=0x55a9a3232130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2765965640 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32282b0 0x55a9a3232130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2765965640 1 -- 172.21.15.98:0/2105009787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 msgr2=0x55a9a3231bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2765965640 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3231bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2765965640 1 -- 172.21.15.98:0/2105009787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9a2f696b0 con 0x55a9a322d420 2024-05-09T08:50:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2764963640 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32282b0 0x55a9a3232130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2765164640 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3231bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2765965640 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3239170 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f275000f6c0 tx=0x7f2750004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1027d0920b3d526d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2744ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:30.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2744ff9640 1 -- 172.21.15.98:0/2105009787 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2750024020 con 0x55a9a322d420 2024-05-09T08:50:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2744ff9640 1 -- 172.21.15.98:0/2105009787 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f2750020610 con 0x55a9a322d420 2024-05-09T08:50:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f276a1b3740 1 -- 172.21.15.98:0/2105009787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27580bb050 con 0x55a9a322d420 2024-05-09T08:50:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2744ff9640 1 -- 172.21.15.98:0/2105009787 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f275002f050 con 0x55a9a322d420 2024-05-09T08:50:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f276a1b3740 10 client.15453 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f276a1b3740 1 -- 172.21.15.98:0/2105009787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a9a323c220 con 0x55a9a322d420 2024-05-09T08:50:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.002+0000 7f2744ff9640 1 -- 172.21.15.98:0/2105009787 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27580bb050 con 0x55a9a322d420 2024-05-09T08:50:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.003+0000 7f2744ff9640 1 -- 172.21.15.98:0/2105009787 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f27500115c0 con 0x55a9a322d420 2024-05-09T08:50:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.003+0000 7f276a1b3740 10 client.15453 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.003+0000 7f276a1b3740 10 client.15453 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T08:50:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.003+0000 7f276a1b3740 20 client.15453 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.003+0000 7f276a1b3740 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55a9a323e8d0 0x55a9a323fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.003+0000 7f276a1b3740 4 client.15453 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:50:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.003+0000 7f276a1b3740 1 -- 172.21.15.98:0/2105009787 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9a3198c00 con 0x55a9a323e8d0 2024-05-09T08:50:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.003+0000 7f2765164640 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55a9a323e8d0 0x55a9a323fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.004+0000 7f2744ff9640 10 client.15453 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.004+0000 7f2765164640 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55a9a323e8d0 0x55a9a323fd40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1ee7a702f96d3138 server_cookie=e8a39cb9f8823c88 in_seq=0 out_seq=0 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.004+0000 7f2744ff9640 1 -- 172.21.15.98:0/2105009787 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a9a3198c00 con 0x55a9a323e8d0 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.004+0000 7f2744ff9640 10 client.15453 handle_command_reply: tid=0 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.004+0000 7f276a1b3740 1 client.15453 shutdown 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.004+0000 7f276a1b3740 1 -- 172.21.15.98:0/2105009787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 msgr2=0x55a9a3239170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.004+0000 7f276a1b3740 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3239170 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f275000f6c0 tx=0x7f2750004a40 comp rx=0 tx=0).stop 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.004+0000 7f276a1b3740 1 -- 172.21.15.98:0/2105009787 shutdown_connections 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.004+0000 7f276a1b3740 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55a9a323e8d0 0x55a9a323fd40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.004+0000 7f276a1b3740 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3239170 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.004+0000 7f276a1b3740 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32282b0 0x55a9a3232130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.004+0000 7f2744ff9640 0 client.15453 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.004+0000 7f276a1b3740 1 --2- 172.21.15.98:0/2105009787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3231bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.004+0000 7f276a1b3740 1 -- 172.21.15.98:0/2105009787 >> 172.21.15.98:0/2105009787 conn(0x55a9a3222f30 msgr2=0x55a9a3222000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.005+0000 7f276a1b3740 1 -- 172.21.15.98:0/2105009787 shutdown_connections 2024-05-09T08:50:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.005+0000 7f276a1b3740 1 -- 172.21.15.98:0/2105009787 wait complete. 2024-05-09T08:50:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.005+0000 7f276a1b3740 20 client.15453 trim_cache size 0 max 16384 2024-05-09T08:50:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.022+0000 7f276a1b3740 1 Processor -- start 2024-05-09T08:50:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f276a1b3740 1 -- start start 2024-05-09T08:50:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f276a1b3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3231bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f276a1b3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32282b0 0x55a9a32320f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f276a1b3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3239090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f276a1b3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a9a323e080 con 0x55a9a322d420 2024-05-09T08:50:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f276a1b3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a9a323e750 con 0x55a9a32278c0 2024-05-09T08:50:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f276a1b3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a9a323ee20 con 0x55a9a32282b0 2024-05-09T08:50:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2765965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3239090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2765965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3239090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50026/0 (socket says 172.21.15.98:50026) 2024-05-09T08:50:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2765965640 1 -- 172.21.15.98:0/3857313005 learned_addr learned my addr 172.21.15.98:0/3857313005 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2765164640 1 --2- 172.21.15.98:0/3857313005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3231bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2765965640 1 -- 172.21.15.98:0/3857313005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32282b0 msgr2=0x55a9a32320f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2764963640 1 --2- 172.21.15.98:0/3857313005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32282b0 0x55a9a32320f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2765965640 1 --2- 172.21.15.98:0/3857313005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32282b0 0x55a9a32320f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2765965640 1 -- 172.21.15.98:0/3857313005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 msgr2=0x55a9a3231bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2765965640 1 --2- 172.21.15.98:0/3857313005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3231bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2765965640 1 -- 172.21.15.98:0/3857313005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9a322d100 con 0x55a9a322d420 2024-05-09T08:50:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2765164640 1 --2- 172.21.15.98:0/3857313005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3231bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2764963640 1 --2- 172.21.15.98:0/3857313005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32282b0 0x55a9a32320f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2765965640 1 --2- 172.21.15.98:0/3857313005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3239090 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f27500119e0 tx=0x7f2750010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af070ad532bc520 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2746ffd640 1 -- 172.21.15.98:0/3857313005 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2750037070 con 0x55a9a322d420 2024-05-09T08:50:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2746ffd640 1 -- 172.21.15.98:0/3857313005 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2750019040 con 0x55a9a322d420 2024-05-09T08:50:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.023+0000 7f2746ffd640 1 -- 172.21.15.98:0/3857313005 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2750024050 con 0x55a9a322d420 2024-05-09T08:50:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.024+0000 7f276a1b3740 1 -- 172.21.15.98:0/3857313005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 msgr2=0x55a9a3239090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.024+0000 7f276a1b3740 1 --2- 172.21.15.98:0/3857313005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3239090 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f27500119e0 tx=0x7f2750010ec0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.024+0000 7f276a1b3740 1 -- 172.21.15.98:0/3857313005 shutdown_connections 2024-05-09T08:50:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.024+0000 7f276a1b3740 1 --2- 172.21.15.98:0/3857313005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3239090 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.024+0000 7f276a1b3740 1 --2- 172.21.15.98:0/3857313005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32282b0 0x55a9a32320f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.024+0000 7f276a1b3740 1 --2- 172.21.15.98:0/3857313005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32278c0 0x55a9a3231bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.024+0000 7f276a1b3740 1 -- 172.21.15.98:0/3857313005 >> 172.21.15.98:0/3857313005 conn(0x55a9a3222f30 msgr2=0x55a9a323d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.024+0000 7f276a1b3740 1 -- 172.21.15.98:0/3857313005 shutdown_connections 2024-05-09T08:50:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.024+0000 7f276a1b3740 1 -- 172.21.15.98:0/3857313005 wait complete. 2024-05-09T08:50:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.024+0000 7f276a1b3740 1 -- start start 2024-05-09T08:50:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.024+0000 7f276a1b3740 1 Processor -- start 2024-05-09T08:50:30.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f276a1b3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32278c0 0x55a9a3235150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f276a1b3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32282b0 0x55a9a3235690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f276a1b3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3238250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f276a1b3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a9a323fbc0 con 0x55a9a322d420 2024-05-09T08:50:30.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f276a1b3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a9a323f4f0 con 0x55a9a32282b0 2024-05-09T08:50:30.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f276a1b3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a9a3240290 con 0x55a9a32278c0 2024-05-09T08:50:30.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f276a1b3740 10 client.0 _ll_register_callbacks cb 0x7f27500071c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:30.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2765965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3238250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2765965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3238250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50038/0 (socket says 172.21.15.98:50038) 2024-05-09T08:50:30.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2765965640 1 -- 172.21.15.98:0/1847762922 learned_addr learned my addr 172.21.15.98:0/1847762922 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2764963640 1 --2- 172.21.15.98:0/1847762922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32282b0 0x55a9a3235690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2765164640 1 --2- 172.21.15.98:0/1847762922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32278c0 0x55a9a3235150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2765965640 1 -- 172.21.15.98:0/1847762922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32278c0 msgr2=0x55a9a3235150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2765965640 1 --2- 172.21.15.98:0/1847762922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32278c0 0x55a9a3235150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2765965640 1 -- 172.21.15.98:0/1847762922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32282b0 msgr2=0x55a9a3235690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2765965640 1 --2- 172.21.15.98:0/1847762922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32282b0 0x55a9a3235690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2765965640 1 -- 172.21.15.98:0/1847762922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9a3238840 con 0x55a9a322d420 2024-05-09T08:50:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2765164640 1 --2- 172.21.15.98:0/1847762922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32278c0 0x55a9a3235150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2765965640 1 --2- 172.21.15.98:0/1847762922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3238250 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f27500107a0 tx=0x7f2750012460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254abb46856fdf6e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2744ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2744ff9640 1 -- 172.21.15.98:0/1847762922 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2750024070 con 0x55a9a322d420 2024-05-09T08:50:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f2744ff9640 1 -- 172.21.15.98:0/1847762922 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f2750004b20 con 0x55a9a322d420 2024-05-09T08:50:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.025+0000 7f276a1b3740 1 -- 172.21.15.98:0/1847762922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2750019040 con 0x55a9a322d420 2024-05-09T08:50:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.026+0000 7f2744ff9640 1 -- 172.21.15.98:0/1847762922 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f275004c050 con 0x55a9a322d420 2024-05-09T08:50:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.026+0000 7f276a1b3740 10 client.15459 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.026+0000 7f276a1b3740 1 -- 172.21.15.98:0/1847762922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a9a32a4bd0 con 0x55a9a322d420 2024-05-09T08:50:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.026+0000 7f2744ff9640 1 -- 172.21.15.98:0/1847762922 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2750019040 con 0x55a9a322d420 2024-05-09T08:50:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.026+0000 7f2744ff9640 1 -- 172.21.15.98:0/1847762922 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2750005500 con 0x55a9a322d420 2024-05-09T08:50:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.026+0000 7f276a1b3740 10 client.15459 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.026+0000 7f276a1b3740 10 client.15459 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T08:50:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.026+0000 7f276a1b3740 20 client.15459 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.026+0000 7f276a1b3740 1 --2- 172.21.15.98:0/1847762922 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55a9a32a93d0 0x55a9a32a80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.026+0000 7f276a1b3740 4 client.15459 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:50:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.026+0000 7f276a1b3740 1 -- 172.21.15.98:0/1847762922 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "status"}) -- 0x55a9a3230620 con 0x55a9a32a93d0 2024-05-09T08:50:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.026+0000 7f2765164640 1 --2- 172.21.15.98:0/1847762922 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55a9a32a93d0 0x55a9a32a80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f2744ff9640 10 client.15459 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:50:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f2765164640 1 --2- 172.21.15.98:0/1847762922 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55a9a32a93d0 0x55a9a32a80e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e16d821fb34174f9 server_cookie=ecad91edbbdc7c0f in_seq=0 out_seq=0 2024-05-09T08:50:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f2744ff9640 1 -- 172.21.15.98:0/1847762922 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55a9a3230620 con 0x55a9a32a93d0 2024-05-09T08:50:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f2744ff9640 10 client.15459 handle_command_reply: tid=0 2024-05-09T08:50:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f276a1b3740 1 client.15459 shutdown 2024-05-09T08:50:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f276a1b3740 1 -- 172.21.15.98:0/1847762922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 msgr2=0x55a9a3238250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f276a1b3740 1 --2- 172.21.15.98:0/1847762922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3238250 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f27500107a0 tx=0x7f2750012460 comp rx=0 tx=0).stop 2024-05-09T08:50:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f276a1b3740 1 -- 172.21.15.98:0/1847762922 shutdown_connections 2024-05-09T08:50:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f276a1b3740 1 --2- 172.21.15.98:0/1847762922 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55a9a32a93d0 0x55a9a32a80e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f276a1b3740 1 --2- 172.21.15.98:0/1847762922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a9a322d420 0x55a9a3238250 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f276a1b3740 1 --2- 172.21.15.98:0/1847762922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a32282b0 0x55a9a3235690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f2744ff9640 0 client.15459 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:50:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f276a1b3740 1 --2- 172.21.15.98:0/1847762922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9a32278c0 0x55a9a3235150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f276a1b3740 1 -- 172.21.15.98:0/1847762922 >> 172.21.15.98:0/1847762922 conn(0x55a9a3222f30 msgr2=0x55a9a3229670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f276a1b3740 1 -- 172.21.15.98:0/1847762922 shutdown_connections 2024-05-09T08:50:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f276a1b3740 1 -- 172.21.15.98:0/1847762922 wait complete. 2024-05-09T08:50:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.027+0000 7f276a1b3740 20 client.15459 trim_cache size 0 max 16384 2024-05-09T08:50:30.036 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:50:30.036 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a57d9712-0ddf-11ef-bc98-c7b262605968", 2024-05-09T08:50:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 4, 2024-05-09T08:50:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 14667, 2024-05-09T08:50:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-05-09T08:50:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-05-09T08:50:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-05-09T08:50:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 111.287566731, 2024-05-09T08:50:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-05-09T08:50:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 94, 2024-05-09T08:50:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 90, 2024-05-09T08:50:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 138.129369067 2024-05-09T08:50:30.038 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:50:30.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.028+0000 7f27673c8640 1 -- 172.21.15.98:0/1605413278 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f27480a2c10 msgr2=0x7f27480a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.028+0000 7f27673c8640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f27480a2c10 0x7f27480a50d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55a9a3228b10 tx=0x7f2750004ab0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.028+0000 7f27673c8640 1 -- 172.21.15.98:0/1605413278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27601023b0 msgr2=0x7f2760197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.028+0000 7f27673c8640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27601023b0 0x7f2760197ba0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f275800d950 tx=0x7f275800de10 comp rx=0 tx=0).stop 2024-05-09T08:50:30.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.028+0000 7f27673c8640 1 -- 172.21.15.98:0/1605413278 shutdown_connections 2024-05-09T08:50:30.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.028+0000 7f27673c8640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2760107610 0x7f276019f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.028+0000 7f27673c8640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2760104ce0 0x7f27601980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.028+0000 7f27673c8640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f27480a2c10 0x7f27480a50d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.028+0000 7f27673c8640 1 --2- 172.21.15.98:0/1605413278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27601023b0 0x7f2760197ba0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.028+0000 7f27673c8640 1 -- 172.21.15.98:0/1605413278 >> 172.21.15.98:0/1605413278 conn(0x7f27600fbce0 msgr2=0x7f2760102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.028+0000 7f27673c8640 1 -- 172.21.15.98:0/1605413278 shutdown_connections 2024-05-09T08:50:30.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.028+0000 7f27673c8640 1 -- 172.21.15.98:0/1605413278 wait complete. 2024-05-09T08:50:30.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-05-09T08:50:30.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.143+0000 7f92459ee640 1 -- 172.21.15.98:0/2404039372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9240102a70 msgr2=0x7f9240102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.143+0000 7f92459ee640 1 --2- 172.21.15.98:0/2404039372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9240102a70 0x7f9240102ef0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9234009900 tx=0x7f923402a410 comp rx=0 tx=0).stop 2024-05-09T08:50:30.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.143+0000 7f92459ee640 1 -- 172.21.15.98:0/2404039372 shutdown_connections 2024-05-09T08:50:30.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.143+0000 7f92459ee640 1 --2- 172.21.15.98:0/2404039372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9240103430 0x7f9240109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.143+0000 7f92459ee640 1 --2- 172.21.15.98:0/2404039372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9240102a70 0x7f9240102ef0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.143+0000 7f92459ee640 1 --2- 172.21.15.98:0/2404039372 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9240101880 0x7f9240101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.143+0000 7f92459ee640 1 -- 172.21.15.98:0/2404039372 >> 172.21.15.98:0/2404039372 conn(0x7f92400fce10 msgr2=0x7f92400ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.144+0000 7f92459ee640 1 -- 172.21.15.98:0/2404039372 shutdown_connections 2024-05-09T08:50:30.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.144+0000 7f92459ee640 1 -- 172.21.15.98:0/2404039372 wait complete. 2024-05-09T08:50:30.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.144+0000 7f92459ee640 1 Processor -- start 2024-05-09T08:50:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.144+0000 7f92459ee640 1 -- start start 2024-05-09T08:50:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.144+0000 7f92459ee640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9240101880 0x7f9240197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.144+0000 7f92459ee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9240102a70 0x7f9240198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.144+0000 7f92459ee640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9240103430 0x7f924019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.144+0000 7f92459ee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f924010d4e0 con 0x7f9240102a70 2024-05-09T08:50:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.144+0000 7f92459ee640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f924010d360 con 0x7f9240101880 2024-05-09T08:50:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.144+0000 7f92459ee640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f924010d660 con 0x7f9240103430 2024-05-09T08:50:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f923e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9240102a70 0x7f9240198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f923e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9240102a70 0x7f9240198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50052/0 (socket says 172.21.15.98:50052) 2024-05-09T08:50:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f923e7fc640 1 -- 172.21.15.98:0/2062374762 learned_addr learned my addr 172.21.15.98:0/2062374762 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f923effd640 1 --2- 172.21.15.98:0/2062374762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9240101880 0x7f9240197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f923e7fc640 1 -- 172.21.15.98:0/2062374762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9240103430 msgr2=0x7f924019f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f923f7fe640 1 --2- 172.21.15.98:0/2062374762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9240103430 0x7f924019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f923e7fc640 1 --2- 172.21.15.98:0/2062374762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9240103430 0x7f924019f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f923e7fc640 1 -- 172.21.15.98:0/2062374762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9240101880 msgr2=0x7f9240197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f923e7fc640 1 --2- 172.21.15.98:0/2062374762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9240101880 0x7f9240197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f923e7fc640 1 -- 172.21.15.98:0/2062374762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924019f840 con 0x7f9240102a70 2024-05-09T08:50:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f923e7fc640 1 --2- 172.21.15.98:0/2062374762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9240102a70 0x7f9240198170 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9234009230 tx=0x7f9234004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f92449ec640 1 -- 172.21.15.98:0/2062374762 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923403f030 con 0x7f9240102a70 2024-05-09T08:50:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f92449ec640 1 -- 172.21.15.98:0/2062374762 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9234003ce0 con 0x7f9240102a70 2024-05-09T08:50:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f92449ec640 1 -- 172.21.15.98:0/2062374762 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9234035070 con 0x7f9240102a70 2024-05-09T08:50:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f92459ee640 1 -- 172.21.15.98:0/2062374762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f924019fad0 con 0x7f9240102a70 2024-05-09T08:50:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.145+0000 7f92459ee640 1 -- 172.21.15.98:0/2062374762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f924019ffc0 con 0x7f9240102a70 2024-05-09T08:50:30.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.147+0000 7f92449ec640 1 -- 172.21.15.98:0/2062374762 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9234042030 con 0x7f9240102a70 2024-05-09T08:50:30.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f92487de740 1 Processor -- start 2024-05-09T08:50:30.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f92487de740 1 -- start start 2024-05-09T08:50:30.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f92449ec640 1 --2- 172.21.15.98:0/2062374762 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f92280a2b90 0x7f92280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f92487de740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691893cd0 0x5576918940f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f92487de740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576918946c0 0x5576918991c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f92487de740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691899830 0x55769189bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f92487de740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5576917f34d0 con 0x557691893cd0 2024-05-09T08:50:30.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f92487de740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55769179b3e0 con 0x5576918946c0 2024-05-09T08:50:30.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f92487de740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5576917ef490 con 0x557691899830 2024-05-09T08:50:30.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f923f7fe640 1 --2- 172.21.15.98:0/2062374762 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f92280a2b90 0x7f92280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f92449ec640 1 -- 172.21.15.98:0/2062374762 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f92340efe90 con 0x7f9240102a70 2024-05-09T08:50:30.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f923effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691893cd0 0x5576918940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f923effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691893cd0 0x5576918940f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50062/0 (socket says 172.21.15.98:50062) 2024-05-09T08:50:30.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f923effd640 1 -- 172.21.15.98:0/3774547739 learned_addr learned my addr 172.21.15.98:0/3774547739 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f923e7fc640 1 --2- 172.21.15.98:0/3774547739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576918946c0 0x5576918991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f923f7fe640 1 --2- 172.21.15.98:0/3774547739 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691899830 0x55769189bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f923effd640 1 -- 172.21.15.98:0/3774547739 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691899830 msgr2=0x55769189bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f923effd640 1 --2- 172.21.15.98:0/3774547739 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691899830 0x55769189bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f923effd640 1 -- 172.21.15.98:0/3774547739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576918946c0 msgr2=0x5576918991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f923effd640 1 --2- 172.21.15.98:0/3774547739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576918946c0 0x5576918991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.148+0000 7f923effd640 1 -- 172.21.15.98:0/3774547739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576915072d0 con 0x557691893cd0 2024-05-09T08:50:30.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.149+0000 7f923effd640 1 --2- 172.21.15.98:0/3774547739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691893cd0 0x5576918940f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f92300103a0 tx=0x7f9230010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d13c17ea3ba5b9a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.149+0000 7f923f7fe640 1 --2- 172.21.15.98:0/2062374762 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f92280a2b90 0x7f92280a5050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f923800be80 tx=0x7f923800b3f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.149+0000 7f9225ffb640 1 -- 172.21.15.98:0/3774547739 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9230016070 con 0x557691893cd0 2024-05-09T08:50:30.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.149+0000 7f9225ffb640 1 -- 172.21.15.98:0/3774547739 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9230004d20 con 0x557691893cd0 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.149+0000 7f9225ffb640 1 -- 172.21.15.98:0/3774547739 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92300054b0 con 0x557691893cd0 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.149+0000 7f92487de740 1 -- 172.21.15.98:0/3774547739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691893cd0 msgr2=0x5576918940f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.149+0000 7f92487de740 1 --2- 172.21.15.98:0/3774547739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691893cd0 0x5576918940f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f92300103a0 tx=0x7f9230010860 comp rx=0 tx=0).stop 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 -- 172.21.15.98:0/3774547739 shutdown_connections 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 --2- 172.21.15.98:0/3774547739 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691899830 0x55769189bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 --2- 172.21.15.98:0/3774547739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576918946c0 0x5576918991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 --2- 172.21.15.98:0/3774547739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691893cd0 0x5576918940f0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 -- 172.21.15.98:0/3774547739 >> 172.21.15.98:0/3774547739 conn(0x55769188af30 msgr2=0x55769188b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 -- 172.21.15.98:0/3774547739 shutdown_connections 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 -- 172.21.15.98:0/3774547739 wait complete. 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 -- start start 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 Processor -- start 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557691893cd0 0x55769189e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5576918946c0 0x55769189e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x5576918a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55769189c7e0 con 0x557691899830 2024-05-09T08:50:30.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5576917ae8e0 con 0x557691893cd0 2024-05-09T08:50:30.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55769189c960 con 0x5576918946c0 2024-05-09T08:50:30.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f92487de740 10 client.0 _ll_register_callbacks cb 0x55769177b9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:30.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f923f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x5576918a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f923effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557691893cd0 0x55769189e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.150+0000 7f923f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x5576918a56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50078/0 (socket says 172.21.15.98:50078) 2024-05-09T08:50:30.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f923f7fe640 1 -- 172.21.15.98:0/383093757 learned_addr learned my addr 172.21.15.98:0/383093757 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f923e7fc640 1 --2- 172.21.15.98:0/383093757 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5576918946c0 0x55769189e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f923f7fe640 1 -- 172.21.15.98:0/383093757 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5576918946c0 msgr2=0x55769189e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f923f7fe640 1 --2- 172.21.15.98:0/383093757 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5576918946c0 0x55769189e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f923f7fe640 1 -- 172.21.15.98:0/383093757 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557691893cd0 msgr2=0x55769189e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f923f7fe640 1 --2- 172.21.15.98:0/383093757 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557691893cd0 0x55769189e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f923f7fe640 1 -- 172.21.15.98:0/383093757 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576915d16b0 con 0x557691899830 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f923effd640 1 --2- 172.21.15.98:0/383093757 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557691893cd0 0x55769189e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f923f7fe640 1 --2- 172.21.15.98:0/383093757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x5576918a56b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9238012a80 tx=0x7f92380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97a754ea770d5d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f921effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f921effd640 1 -- 172.21.15.98:0/383093757 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238023030 con 0x557691899830 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f921effd640 1 -- 172.21.15.98:0/383093757 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f9238011c10 con 0x557691899830 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f92487de740 1 -- 172.21.15.98:0/383093757 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9230004d20 con 0x557691899830 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f921effd640 1 -- 172.21.15.98:0/383093757 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9238030050 con 0x557691899830 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f92487de740 10 client.15471 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f92487de740 1 -- 172.21.15.98:0/383093757 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5576918a8700 con 0x557691899830 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.151+0000 7f921effd640 1 -- 172.21.15.98:0/383093757 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9230004d20 con 0x557691899830 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.152+0000 7f921effd640 1 -- 172.21.15.98:0/383093757 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f923801c050 con 0x557691899830 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.152+0000 7f92487de740 10 client.15471 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.152+0000 7f92487de740 10 client.15471 resolve_mds: resolved name 'l' to daemon mds.l 2024-05-09T08:50:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.152+0000 7f92487de740 20 client.15471 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.152+0000 7f92487de740 1 --2- 172.21.15.98:0/383093757 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5576918aae40 0x5576918ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.152+0000 7f92487de740 4 client.15471 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:50:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.152+0000 7f92487de740 1 -- 172.21.15.98:0/383093757 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557691800c00 con 0x5576918aae40 2024-05-09T08:50:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.152+0000 7f923effd640 1 --2- 172.21.15.98:0/383093757 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5576918aae40 0x5576918ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.152+0000 7f923effd640 1 --2- 172.21.15.98:0/383093757 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5576918aae40 0x5576918ac2b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f117ae84cf2d1d1 server_cookie=e604c2f16d25ac63 in_seq=0 out_seq=0 2024-05-09T08:50:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.152+0000 7f921effd640 10 client.15471 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:50:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.153+0000 7f921effd640 1 -- 172.21.15.98:0/383093757 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557691800c00 con 0x5576918aae40 2024-05-09T08:50:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.153+0000 7f921effd640 10 client.15471 handle_command_reply: tid=0 2024-05-09T08:50:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.153+0000 7f92487de740 1 client.15471 shutdown 2024-05-09T08:50:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.154+0000 7f92487de740 1 -- 172.21.15.98:0/383093757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 msgr2=0x5576918a56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.154+0000 7f92487de740 1 --2- 172.21.15.98:0/383093757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x5576918a56b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9238012a80 tx=0x7f92380046c0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.154+0000 7f92487de740 1 -- 172.21.15.98:0/383093757 shutdown_connections 2024-05-09T08:50:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.154+0000 7f92487de740 1 --2- 172.21.15.98:0/383093757 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5576918aae40 0x5576918ac2b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.154+0000 7f92487de740 1 --2- 172.21.15.98:0/383093757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x5576918a56b0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.154+0000 7f92487de740 1 --2- 172.21.15.98:0/383093757 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5576918946c0 0x55769189e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.154+0000 7f921effd640 0 client.15471 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:50:30.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.154+0000 7f92487de740 1 --2- 172.21.15.98:0/383093757 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557691893cd0 0x55769189e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.154+0000 7f92487de740 1 -- 172.21.15.98:0/383093757 >> 172.21.15.98:0/383093757 conn(0x55769188af30 msgr2=0x55769188a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.154+0000 7f92487de740 1 -- 172.21.15.98:0/383093757 shutdown_connections 2024-05-09T08:50:30.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.154+0000 7f92487de740 1 -- 172.21.15.98:0/383093757 wait complete. 2024-05-09T08:50:30.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.154+0000 7f92487de740 20 client.15471 trim_cache size 0 max 16384 2024-05-09T08:50:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.171+0000 7f92487de740 1 Processor -- start 2024-05-09T08:50:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f92487de740 1 -- start start 2024-05-09T08:50:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f92487de740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691893cd0 0x55769188a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f92487de740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576918946c0 0x5576918a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f92487de740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x557691895b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f92487de740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5576918aa5f0 con 0x557691899830 2024-05-09T08:50:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f92487de740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5576918aacc0 con 0x5576918946c0 2024-05-09T08:50:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f92487de740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5576918ab390 con 0x557691893cd0 2024-05-09T08:50:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f923f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x557691895b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f923f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x557691895b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50082/0 (socket says 172.21.15.98:50082) 2024-05-09T08:50:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f923f7fe640 1 -- 172.21.15.98:0/333415814 learned_addr learned my addr 172.21.15.98:0/333415814 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f923f7fe640 1 -- 172.21.15.98:0/333415814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691893cd0 msgr2=0x55769188a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f923f7fe640 1 --2- 172.21.15.98:0/333415814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691893cd0 0x55769188a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f923f7fe640 1 -- 172.21.15.98:0/333415814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576918946c0 msgr2=0x5576918a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f923f7fe640 1 --2- 172.21.15.98:0/333415814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576918946c0 0x5576918a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f923f7fe640 1 -- 172.21.15.98:0/333415814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557691894aa0 con 0x557691899830 2024-05-09T08:50:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f923f7fe640 1 --2- 172.21.15.98:0/333415814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x557691895b30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9238012710 tx=0x7f9238012740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b5a016e36cc3fd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f92257fa640 1 -- 172.21.15.98:0/333415814 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238023070 con 0x557691899830 2024-05-09T08:50:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f92257fa640 1 -- 172.21.15.98:0/333415814 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f923801eda0 con 0x557691899830 2024-05-09T08:50:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.172+0000 7f92257fa640 1 -- 172.21.15.98:0/333415814 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92380053a0 con 0x557691899830 2024-05-09T08:50:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.173+0000 7f92487de740 1 -- 172.21.15.98:0/333415814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 msgr2=0x557691895b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.173+0000 7f92487de740 1 --2- 172.21.15.98:0/333415814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x557691895b30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9238012710 tx=0x7f9238012740 comp rx=0 tx=0).stop 2024-05-09T08:50:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.173+0000 7f92487de740 1 -- 172.21.15.98:0/333415814 shutdown_connections 2024-05-09T08:50:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.173+0000 7f92487de740 1 --2- 172.21.15.98:0/333415814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x557691895b30 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.173+0000 7f92487de740 1 --2- 172.21.15.98:0/333415814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576918946c0 0x5576918a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.173+0000 7f92487de740 1 --2- 172.21.15.98:0/333415814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691893cd0 0x55769188a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.173+0000 7f92487de740 1 -- 172.21.15.98:0/333415814 >> 172.21.15.98:0/333415814 conn(0x55769188af30 msgr2=0x55769189e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.173+0000 7f92487de740 1 -- 172.21.15.98:0/333415814 shutdown_connections 2024-05-09T08:50:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.173+0000 7f92487de740 1 -- 172.21.15.98:0/333415814 wait complete. 2024-05-09T08:50:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.173+0000 7f92487de740 1 -- start start 2024-05-09T08:50:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.173+0000 7f92487de740 1 Processor -- start 2024-05-09T08:50:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f92487de740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691893cd0 0x5576918a1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f92487de740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576918946c0 0x5576918a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f92487de740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x5576918a4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f92487de740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5576918ac130 con 0x557691899830 2024-05-09T08:50:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f92487de740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5576918aba60 con 0x5576918946c0 2024-05-09T08:50:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f92487de740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5576918ac800 con 0x557691893cd0 2024-05-09T08:50:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f92487de740 10 client.0 _ll_register_callbacks cb 0x7f9238005ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f923f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x5576918a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f923f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x5576918a4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50090/0 (socket says 172.21.15.98:50090) 2024-05-09T08:50:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f923f7fe640 1 -- 172.21.15.98:0/2161090217 learned_addr learned my addr 172.21.15.98:0/2161090217 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f923effd640 1 --2- 172.21.15.98:0/2161090217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691893cd0 0x5576918a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f923e7fc640 1 --2- 172.21.15.98:0/2161090217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576918946c0 0x5576918a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f923f7fe640 1 -- 172.21.15.98:0/2161090217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691893cd0 msgr2=0x5576918a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f923f7fe640 1 --2- 172.21.15.98:0/2161090217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691893cd0 0x5576918a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f923f7fe640 1 -- 172.21.15.98:0/2161090217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576918946c0 msgr2=0x5576918a1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f923f7fe640 1 --2- 172.21.15.98:0/2161090217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576918946c0 0x5576918a1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f923f7fe640 1 -- 172.21.15.98:0/2161090217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576918a4ea0 con 0x557691899830 2024-05-09T08:50:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f923effd640 1 --2- 172.21.15.98:0/2161090217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691893cd0 0x5576918a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f923f7fe640 1 --2- 172.21.15.98:0/2161090217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x5576918a4890 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f923801e650 tx=0x7f9238005a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59683e57300a2e1a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f921effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f921effd640 1 -- 172.21.15.98:0/2161090217 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238039070 con 0x557691899830 2024-05-09T08:50:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f921effd640 1 -- 172.21.15.98:0/2161090217 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f9238005320 con 0x557691899830 2024-05-09T08:50:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.174+0000 7f92487de740 1 -- 172.21.15.98:0/2161090217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f923801eda0 con 0x557691899830 2024-05-09T08:50:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.175+0000 7f921effd640 1 -- 172.21.15.98:0/2161090217 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9238045050 con 0x557691899830 2024-05-09T08:50:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.175+0000 7f92487de740 10 client.15477 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.175+0000 7f92487de740 1 -- 172.21.15.98:0/2161090217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557691911300 con 0x557691899830 2024-05-09T08:50:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.175+0000 7f921effd640 1 -- 172.21.15.98:0/2161090217 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f923801eda0 con 0x557691899830 2024-05-09T08:50:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.175+0000 7f921effd640 1 -- 172.21.15.98:0/2161090217 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9238033070 con 0x557691899830 2024-05-09T08:50:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.175+0000 7f92487de740 10 client.15477 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.175+0000 7f92487de740 10 client.15477 resolve_mds: resolved name 'l' to daemon mds.l 2024-05-09T08:50:30.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.175+0000 7f92487de740 20 client.15477 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:30.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.175+0000 7f92487de740 1 --2- 172.21.15.98:0/2161090217 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x557691915ac0 0x5576919147d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.175+0000 7f92487de740 4 client.15477 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:50:30.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.175+0000 7f92487de740 1 -- 172.21.15.98:0/2161090217 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "status"}) -- 0x5576917faf50 con 0x557691915ac0 2024-05-09T08:50:30.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.175+0000 7f923effd640 1 --2- 172.21.15.98:0/2161090217 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x557691915ac0 0x5576919147d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f921effd640 10 client.15477 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:50:30.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f923effd640 1 --2- 172.21.15.98:0/2161090217 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x557691915ac0 0x5576919147d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=185d1ce830b5af5a server_cookie=d965dafdf684b6e8 in_seq=0 out_seq=0 2024-05-09T08:50:30.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f921effd640 1 -- 172.21.15.98:0/2161090217 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5576917faf50 con 0x557691915ac0 2024-05-09T08:50:30.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f921effd640 10 client.15477 handle_command_reply: tid=0 2024-05-09T08:50:30.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f92487de740 1 client.15477 shutdown 2024-05-09T08:50:30.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f92487de740 1 -- 172.21.15.98:0/2161090217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 msgr2=0x5576918a4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f92487de740 1 --2- 172.21.15.98:0/2161090217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x5576918a4890 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f923801e650 tx=0x7f9238005a70 comp rx=0 tx=0).stop 2024-05-09T08:50:30.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f92487de740 1 -- 172.21.15.98:0/2161090217 shutdown_connections 2024-05-09T08:50:30.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f92487de740 1 --2- 172.21.15.98:0/2161090217 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x557691915ac0 0x5576919147d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f92487de740 1 --2- 172.21.15.98:0/2161090217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557691899830 0x5576918a4890 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f92487de740 1 --2- 172.21.15.98:0/2161090217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576918946c0 0x5576918a1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f921effd640 0 client.15477 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:50:30.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f92487de740 1 --2- 172.21.15.98:0/2161090217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557691893cd0 0x5576918a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f92487de740 1 -- 172.21.15.98:0/2161090217 >> 172.21.15.98:0/2161090217 conn(0x55769188af30 msgr2=0x55769189df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f92487de740 1 -- 172.21.15.98:0/2161090217 shutdown_connections 2024-05-09T08:50:30.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f92487de740 1 -- 172.21.15.98:0/2161090217 wait complete. 2024-05-09T08:50:30.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.176+0000 7f92487de740 20 client.15477 trim_cache size 0 max 16384 2024-05-09T08:50:30.184 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:50:30.185 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a57d9712-0ddf-11ef-bc98-c7b262605968", 2024-05-09T08:50:30.185 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 3, 2024-05-09T08:50:30.185 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24388, 2024-05-09T08:50:30.185 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-05-09T08:50:30.185 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-05-09T08:50:30.185 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-05-09T08:50:30.185 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 112.444115506, 2024-05-09T08:50:30.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-05-09T08:50:30.185 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 94, 2024-05-09T08:50:30.185 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 90, 2024-05-09T08:50:30.186 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 169.780967737 2024-05-09T08:50:30.186 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:50:30.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.177+0000 7f92459ee640 1 -- 172.21.15.98:0/2062374762 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f92280a2b90 msgr2=0x7f92280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.177+0000 7f92459ee640 1 --2- 172.21.15.98:0/2062374762 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f92280a2b90 0x7f92280a5050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f923800be80 tx=0x7f923800b3f0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.177+0000 7f92459ee640 1 -- 172.21.15.98:0/2062374762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9240102a70 msgr2=0x7f9240198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.177+0000 7f92459ee640 1 --2- 172.21.15.98:0/2062374762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9240102a70 0x7f9240198170 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9234009230 tx=0x7f9234004360 comp rx=0 tx=0).stop 2024-05-09T08:50:30.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.177+0000 7f92459ee640 1 -- 172.21.15.98:0/2062374762 shutdown_connections 2024-05-09T08:50:30.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.177+0000 7f92459ee640 1 --2- 172.21.15.98:0/2062374762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9240103430 0x7f924019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.177+0000 7f92459ee640 1 --2- 172.21.15.98:0/2062374762 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f92280a2b90 0x7f92280a5050 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.177+0000 7f92459ee640 1 --2- 172.21.15.98:0/2062374762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9240102a70 0x7f9240198170 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.177+0000 7f92459ee640 1 --2- 172.21.15.98:0/2062374762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9240101880 0x7f9240197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.177+0000 7f92459ee640 1 -- 172.21.15.98:0/2062374762 >> 172.21.15.98:0/2062374762 conn(0x7f92400fce10 msgr2=0x7f9240105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.177+0000 7f92459ee640 1 -- 172.21.15.98:0/2062374762 shutdown_connections 2024-05-09T08:50:30.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.177+0000 7f92459ee640 1 -- 172.21.15.98:0/2062374762 wait complete. 2024-05-09T08:50:30.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T08:50:30.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.291+0000 7fb2464bd640 1 -- 172.21.15.98:0/1152606244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb240103420 msgr2=0x7fb240109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.291+0000 7fb2464bd640 1 --2- 172.21.15.98:0/1152606244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb240103420 0x7fb240109cb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb238009ca0 tx=0x7fb23802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 -- 172.21.15.98:0/1152606244 shutdown_connections 2024-05-09T08:50:30.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 --2- 172.21.15.98:0/1152606244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb240103420 0x7fb240109cb0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 --2- 172.21.15.98:0/1152606244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb240102a60 0x7fb240102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 --2- 172.21.15.98:0/1152606244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb240101870 0x7fb240101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 -- 172.21.15.98:0/1152606244 >> 172.21.15.98:0/1152606244 conn(0x7fb2400fcde0 msgr2=0x7fb2400ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 -- 172.21.15.98:0/1152606244 shutdown_connections 2024-05-09T08:50:30.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 -- 172.21.15.98:0/1152606244 wait complete. 2024-05-09T08:50:30.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 Processor -- start 2024-05-09T08:50:30.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 -- start start 2024-05-09T08:50:30.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb240101870 0x7fb240197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb240102a60 0x7fb240198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb240103420 0x7fb24019f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb24010d5a0 con 0x7fb240101870 2024-05-09T08:50:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb24010d420 con 0x7fb240103420 2024-05-09T08:50:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb2464bd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb24010d720 con 0x7fb240102a60 2024-05-09T08:50:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb240101870 0x7fb240197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb240101870 0x7fb240197c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50100/0 (socket says 172.21.15.98:50100) 2024-05-09T08:50:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.292+0000 7fb23ffff640 1 -- 172.21.15.98:0/3673326588 learned_addr learned my addr 172.21.15.98:0/3673326588 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.293+0000 7fb237fff640 1 --2- 172.21.15.98:0/3673326588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb240102a60 0x7fb240198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.293+0000 7fb244a5a640 1 --2- 172.21.15.98:0/3673326588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb240103420 0x7fb24019f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.293+0000 7fb23ffff640 1 -- 172.21.15.98:0/3673326588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb240102a60 msgr2=0x7fb240198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.293+0000 7fb23ffff640 1 --2- 172.21.15.98:0/3673326588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb240102a60 0x7fb240198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.293+0000 7fb23ffff640 1 -- 172.21.15.98:0/3673326588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb240103420 msgr2=0x7fb24019f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.293+0000 7fb23ffff640 1 --2- 172.21.15.98:0/3673326588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb240103420 0x7fb24019f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.293+0000 7fb23ffff640 1 -- 172.21.15.98:0/3673326588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24019f830 con 0x7fb240101870 2024-05-09T08:50:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.293+0000 7fb23ffff640 1 --2- 172.21.15.98:0/3673326588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb240101870 0x7fb240197c20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb23000c970 tx=0x7fb23000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.293+0000 7fb23dffb640 1 -- 172.21.15.98:0/3673326588 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb230004220 con 0x7fb240101870 2024-05-09T08:50:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.293+0000 7fb2464bd640 1 -- 172.21.15.98:0/3673326588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb24019fb20 con 0x7fb240101870 2024-05-09T08:50:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.293+0000 7fb23dffb640 1 -- 172.21.15.98:0/3673326588 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2300043c0 con 0x7fb240101870 2024-05-09T08:50:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.293+0000 7fb23dffb640 1 -- 172.21.15.98:0/3673326588 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23000f450 con 0x7fb240101870 2024-05-09T08:50:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.293+0000 7fb2464bd640 1 -- 172.21.15.98:0/3673326588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2401a0070 con 0x7fb240101870 2024-05-09T08:50:30.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.295+0000 7fb23dffb640 1 -- 172.21.15.98:0/3673326588 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb2300079b0 con 0x7fb240101870 2024-05-09T08:50:30.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.295+0000 7fb2491d4740 1 Processor -- start 2024-05-09T08:50:30.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb2491d4740 1 -- start start 2024-05-09T08:50:30.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb2491d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c70150f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb2491d4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c70156c0 0x5582c701a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb2491d4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 0x5582c701ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb2491d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5582c6f744d0 con 0x5582c7014cd0 2024-05-09T08:50:30.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb2491d4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5582c6f1c3e0 con 0x5582c701a830 2024-05-09T08:50:30.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb2491d4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5582c6f70490 con 0x5582c70156c0 2024-05-09T08:50:30.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb244a5a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c70150f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb244a5a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c70150f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50106/0 (socket says 172.21.15.98:50106) 2024-05-09T08:50:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb244a5a640 1 -- 172.21.15.98:0/605651444 learned_addr learned my addr 172.21.15.98:0/605651444 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb23ffff640 1 --2- 172.21.15.98:0/605651444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c70156c0 0x5582c701a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb23dffb640 1 --2- 172.21.15.98:0/3673326588 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb22c0a2b90 0x7fb22c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb237fff640 1 --2- 172.21.15.98:0/605651444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 0x5582c701ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb244a5a640 1 -- 172.21.15.98:0/605651444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c70156c0 msgr2=0x5582c701a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb244a5a640 1 --2- 172.21.15.98:0/605651444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c70156c0 0x5582c701a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb244a5a640 1 -- 172.21.15.98:0/605651444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 msgr2=0x5582c701ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb244a5a640 1 --2- 172.21.15.98:0/605651444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 0x5582c701ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb244a5a640 1 -- 172.21.15.98:0/605651444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582c6c882d0 con 0x5582c7014cd0 2024-05-09T08:50:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb237fff640 1 --2- 172.21.15.98:0/605651444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 0x5582c701ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb23dffb640 1 -- 172.21.15.98:0/3673326588 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fb2300cf170 con 0x7fb240101870 2024-05-09T08:50:30.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb244a5a640 1 --2- 172.21.15.98:0/3673326588 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb22c0a2b90 0x7fb22c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.296+0000 7fb244a5a640 1 --2- 172.21.15.98:0/605651444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c70150f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb238035000 tx=0x7fb23802f500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe0b38b72d50da5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.297+0000 7fb2367fc640 1 -- 172.21.15.98:0/605651444 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb238042040 con 0x5582c7014cd0 2024-05-09T08:50:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.297+0000 7fb2367fc640 1 -- 172.21.15.98:0/605651444 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb238036070 con 0x5582c7014cd0 2024-05-09T08:50:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.297+0000 7fb2367fc640 1 -- 172.21.15.98:0/605651444 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23802fce0 con 0x5582c7014cd0 2024-05-09T08:50:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.297+0000 7fb2491d4740 1 -- 172.21.15.98:0/605651444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 msgr2=0x5582c70150f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.297+0000 7fb244a5a640 1 --2- 172.21.15.98:0/3673326588 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb22c0a2b90 0x7fb22c0a5050 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb23804d000 tx=0x7fb23804e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.297+0000 7fb2491d4740 1 --2- 172.21.15.98:0/605651444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c70150f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb238035000 tx=0x7fb23802f500 comp rx=0 tx=0).stop 2024-05-09T08:50:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.297+0000 7fb2491d4740 1 -- 172.21.15.98:0/605651444 shutdown_connections 2024-05-09T08:50:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.297+0000 7fb2491d4740 1 --2- 172.21.15.98:0/605651444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 0x5582c701ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.297+0000 7fb2491d4740 1 --2- 172.21.15.98:0/605651444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c70156c0 0x5582c701a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.297+0000 7fb2491d4740 1 --2- 172.21.15.98:0/605651444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c70150f0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.297+0000 7fb2491d4740 1 -- 172.21.15.98:0/605651444 >> 172.21.15.98:0/605651444 conn(0x5582c700bf30 msgr2=0x5582c700c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.297+0000 7fb2491d4740 1 -- 172.21.15.98:0/605651444 shutdown_connections 2024-05-09T08:50:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.297+0000 7fb2491d4740 1 -- 172.21.15.98:0/605651444 wait complete. 2024-05-09T08:50:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb2491d4740 1 -- start start 2024-05-09T08:50:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb2491d4740 1 Processor -- start 2024-05-09T08:50:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb2491d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c701f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb2491d4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c70156c0 0x5582c701f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb2491d4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 0x5582c7026640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb2491d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5582c701d7e0 con 0x5582c7014cd0 2024-05-09T08:50:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb2491d4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5582c6f2f8e0 con 0x5582c701a830 2024-05-09T08:50:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb2491d4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5582c701d960 con 0x5582c70156c0 2024-05-09T08:50:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb2491d4740 10 client.0 _ll_register_callbacks cb 0x5582c6efc9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c701f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c701f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50116/0 (socket says 172.21.15.98:50116) 2024-05-09T08:50:30.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb23ffff640 1 -- 172.21.15.98:0/1510712900 learned_addr learned my addr 172.21.15.98:0/1510712900 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb244a5a640 1 --2- 172.21.15.98:0/1510712900 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 0x5582c7026640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb23ffff640 1 -- 172.21.15.98:0/1510712900 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c70156c0 msgr2=0x5582c701f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb23ffff640 1 --2- 172.21.15.98:0/1510712900 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c70156c0 0x5582c701f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb23ffff640 1 -- 172.21.15.98:0/1510712900 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 msgr2=0x5582c7026640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb23ffff640 1 --2- 172.21.15.98:0/1510712900 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 0x5582c7026640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.298+0000 7fb23ffff640 1 -- 172.21.15.98:0/1510712900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582c6d526b0 con 0x5582c7014cd0 2024-05-09T08:50:30.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb23ffff640 1 --2- 172.21.15.98:0/1510712900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c701f0c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb230007590 tx=0x7fb230086050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a793b75b07894a63 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb21f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:30.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb21f7fe640 1 -- 172.21.15.98:0/1510712900 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23001d040 con 0x5582c7014cd0 2024-05-09T08:50:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb21f7fe640 1 -- 172.21.15.98:0/1510712900 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fb2300972c0 con 0x5582c7014cd0 2024-05-09T08:50:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb2491d4740 1 -- 172.21.15.98:0/1510712900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb238036070 con 0x5582c7014cd0 2024-05-09T08:50:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb21f7fe640 1 -- 172.21.15.98:0/1510712900 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb23009b3d0 con 0x5582c7014cd0 2024-05-09T08:50:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb2491d4740 10 client.15489 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb2491d4740 1 -- 172.21.15.98:0/1510712900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5582c7029690 con 0x5582c7014cd0 2024-05-09T08:50:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb21f7fe640 1 -- 172.21.15.98:0/1510712900 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb238036070 con 0x5582c7014cd0 2024-05-09T08:50:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb21f7fe640 1 -- 172.21.15.98:0/1510712900 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb2300d10f0 con 0x5582c7014cd0 2024-05-09T08:50:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb2491d4740 10 client.15489 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb2491d4740 10 client.15489 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T08:50:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb2491d4740 20 client.15489 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb2491d4740 1 --2- 172.21.15.98:0/1510712900 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5582c702bd20 0x5582c702d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb2491d4740 4 client.15489 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:50:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.299+0000 7fb2491d4740 1 -- 172.21.15.98:0/1510712900 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582c6f81c00 con 0x5582c702bd20 2024-05-09T08:50:30.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.300+0000 7fb237fff640 1 --2- 172.21.15.98:0/1510712900 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5582c702bd20 0x5582c702d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.300+0000 7fb21f7fe640 10 client.15489 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:50:30.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.300+0000 7fb237fff640 1 --2- 172.21.15.98:0/1510712900 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5582c702bd20 0x5582c702d190 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=85a11b0a916a8526 server_cookie=92c2997200c81309 in_seq=0 out_seq=0 2024-05-09T08:50:30.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb21f7fe640 1 -- 172.21.15.98:0/1510712900 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582c6f81c00 con 0x5582c702bd20 2024-05-09T08:50:30.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb21f7fe640 10 client.15489 handle_command_reply: tid=0 2024-05-09T08:50:30.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb2491d4740 1 client.15489 shutdown 2024-05-09T08:50:30.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb2491d4740 1 -- 172.21.15.98:0/1510712900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 msgr2=0x5582c701f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb2491d4740 1 --2- 172.21.15.98:0/1510712900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c701f0c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb230007590 tx=0x7fb230086050 comp rx=0 tx=0).stop 2024-05-09T08:50:30.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb2491d4740 1 -- 172.21.15.98:0/1510712900 shutdown_connections 2024-05-09T08:50:30.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb2491d4740 1 --2- 172.21.15.98:0/1510712900 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5582c702bd20 0x5582c702d190 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb2491d4740 1 --2- 172.21.15.98:0/1510712900 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 0x5582c7026640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb2491d4740 1 --2- 172.21.15.98:0/1510712900 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c70156c0 0x5582c701f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb21f7fe640 0 client.15489 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:50:30.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb2491d4740 1 --2- 172.21.15.98:0/1510712900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c701f0c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb2491d4740 1 -- 172.21.15.98:0/1510712900 >> 172.21.15.98:0/1510712900 conn(0x5582c700bf30 msgr2=0x5582c700b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb2491d4740 1 -- 172.21.15.98:0/1510712900 shutdown_connections 2024-05-09T08:50:30.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb2491d4740 1 -- 172.21.15.98:0/1510712900 wait complete. 2024-05-09T08:50:30.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.301+0000 7fb2491d4740 20 client.15489 trim_cache size 0 max 16384 2024-05-09T08:50:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb2491d4740 1 Processor -- start 2024-05-09T08:50:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb2491d4740 1 -- start start 2024-05-09T08:50:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb2491d4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c7014cd0 0x5582c701cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb2491d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c70156c0 0x5582c70265d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb2491d4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 0x5582c701ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb2491d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5582c702b4d0 con 0x5582c70156c0 2024-05-09T08:50:30.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb2491d4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5582c702bba0 con 0x5582c701a830 2024-05-09T08:50:30.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb2491d4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5582c702c270 con 0x5582c7014cd0 2024-05-09T08:50:30.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb237fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c70156c0 0x5582c70265d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb237fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c70156c0 0x5582c70265d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50122/0 (socket says 172.21.15.98:50122) 2024-05-09T08:50:30.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb237fff640 1 -- 172.21.15.98:0/1412238828 learned_addr learned my addr 172.21.15.98:0/1412238828 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb23ffff640 1 --2- 172.21.15.98:0/1412238828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c7014cd0 0x5582c701cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb244a5a640 1 --2- 172.21.15.98:0/1412238828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 0x5582c701ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb237fff640 1 -- 172.21.15.98:0/1412238828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c7014cd0 msgr2=0x5582c701cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb237fff640 1 --2- 172.21.15.98:0/1412238828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c7014cd0 0x5582c701cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb237fff640 1 -- 172.21.15.98:0/1412238828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 msgr2=0x5582c701ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb237fff640 1 --2- 172.21.15.98:0/1412238828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 0x5582c701ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb237fff640 1 -- 172.21.15.98:0/1412238828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582c700ba70 con 0x5582c70156c0 2024-05-09T08:50:30.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb244a5a640 1 --2- 172.21.15.98:0/1412238828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 0x5582c701ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:30.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb237fff640 1 --2- 172.21.15.98:0/1412238828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c70156c0 0x5582c70265d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb228009330 tx=0x7fb228016800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7d74c9030a11cf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.320+0000 7fb235ffb640 1 -- 172.21.15.98:0/1412238828 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb228009e10 con 0x5582c70156c0 2024-05-09T08:50:30.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.321+0000 7fb235ffb640 1 -- 172.21.15.98:0/1412238828 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb228016d00 con 0x5582c70156c0 2024-05-09T08:50:30.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.321+0000 7fb235ffb640 1 -- 172.21.15.98:0/1412238828 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb228014770 con 0x5582c70156c0 2024-05-09T08:50:30.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.321+0000 7fb2491d4740 1 -- 172.21.15.98:0/1412238828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c70156c0 msgr2=0x5582c70265d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.321+0000 7fb2491d4740 1 --2- 172.21.15.98:0/1412238828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c70156c0 0x5582c70265d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb228009330 tx=0x7fb228016800 comp rx=0 tx=0).stop 2024-05-09T08:50:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.321+0000 7fb2491d4740 1 -- 172.21.15.98:0/1412238828 shutdown_connections 2024-05-09T08:50:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.321+0000 7fb2491d4740 1 --2- 172.21.15.98:0/1412238828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c701a830 0x5582c701ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.321+0000 7fb2491d4740 1 --2- 172.21.15.98:0/1412238828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c70156c0 0x5582c70265d0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.321+0000 7fb2491d4740 1 --2- 172.21.15.98:0/1412238828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c7014cd0 0x5582c701cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.321+0000 7fb2491d4740 1 -- 172.21.15.98:0/1412238828 >> 172.21.15.98:0/1412238828 conn(0x5582c700bf30 msgr2=0x5582c7016a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.321+0000 7fb2491d4740 1 -- 172.21.15.98:0/1412238828 shutdown_connections 2024-05-09T08:50:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.321+0000 7fb2491d4740 1 -- 172.21.15.98:0/1412238828 wait complete. 2024-05-09T08:50:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb2491d4740 1 -- start start 2024-05-09T08:50:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb2491d4740 1 Processor -- start 2024-05-09T08:50:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb2491d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c7022210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb2491d4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c70156c0 0x5582c7022750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb2491d4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c701a830 0x5582c7025770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb2491d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5582c702d010 con 0x5582c7014cd0 2024-05-09T08:50:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb2491d4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5582c702c940 con 0x5582c70156c0 2024-05-09T08:50:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb2491d4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5582c702d6e0 con 0x5582c701a830 2024-05-09T08:50:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb2491d4740 10 client.0 _ll_register_callbacks cb 0x7fb23000c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c7022210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c7022210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50128/0 (socket says 172.21.15.98:50128) 2024-05-09T08:50:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb23ffff640 1 -- 172.21.15.98:0/195899674 learned_addr learned my addr 172.21.15.98:0/195899674 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb244a5a640 1 --2- 172.21.15.98:0/195899674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c701a830 0x5582c7025770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb23ffff640 1 -- 172.21.15.98:0/195899674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c701a830 msgr2=0x5582c7025770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb23ffff640 1 --2- 172.21.15.98:0/195899674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c701a830 0x5582c7025770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb23ffff640 1 -- 172.21.15.98:0/195899674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c70156c0 msgr2=0x5582c7022750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb23ffff640 1 --2- 172.21.15.98:0/195899674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c70156c0 0x5582c7022750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb23ffff640 1 -- 172.21.15.98:0/195899674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582c7025d60 con 0x5582c7014cd0 2024-05-09T08:50:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb244a5a640 1 --2- 172.21.15.98:0/195899674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c701a830 0x5582c7025770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb23ffff640 1 --2- 172.21.15.98:0/195899674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c7022210 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb23009aa50 tx=0x7fb230004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a7971aaac47745e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.322+0000 7fb21f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb21f7fe640 1 -- 172.21.15.98:0/195899674 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23001d040 con 0x5582c7014cd0 2024-05-09T08:50:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb21f7fe640 1 -- 172.21.15.98:0/195899674 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fb23009acd0 con 0x5582c7014cd0 2024-05-09T08:50:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb2491d4740 1 -- 172.21.15.98:0/195899674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb228016d00 con 0x5582c7014cd0 2024-05-09T08:50:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb21f7fe640 1 -- 172.21.15.98:0/195899674 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb230099740 con 0x5582c7014cd0 2024-05-09T08:50:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb2491d4740 10 client.15495 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb2491d4740 1 -- 172.21.15.98:0/195899674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5582c70921a0 con 0x5582c7014cd0 2024-05-09T08:50:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb21f7fe640 1 -- 172.21.15.98:0/195899674 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb228016d00 con 0x5582c7014cd0 2024-05-09T08:50:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb21f7fe640 1 -- 172.21.15.98:0/195899674 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb230084b00 con 0x5582c7014cd0 2024-05-09T08:50:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb2491d4740 10 client.15495 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb2491d4740 10 client.15495 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T08:50:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb2491d4740 20 client.15495 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:30.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb2491d4740 1 --2- 172.21.15.98:0/195899674 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5582c7096da0 0x5582c7095ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb2491d4740 4 client.15495 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:50:30.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb2491d4740 1 -- 172.21.15.98:0/195899674 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "status"}) -- 0x5582c6f7bf50 con 0x5582c7096da0 2024-05-09T08:50:30.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.323+0000 7fb237fff640 1 --2- 172.21.15.98:0/195899674 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5582c7096da0 0x5582c7095ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb237fff640 1 --2- 172.21.15.98:0/195899674 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5582c7096da0 0x5582c7095ab0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a31c607eab9a26c server_cookie=4632ce95ca678651 in_seq=0 out_seq=0 2024-05-09T08:50:30.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb21f7fe640 10 client.15495 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:50:30.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb21f7fe640 1 -- 172.21.15.98:0/195899674 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5582c6f7bf50 con 0x5582c7096da0 2024-05-09T08:50:30.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb21f7fe640 10 client.15495 handle_command_reply: tid=0 2024-05-09T08:50:30.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb2491d4740 1 client.15495 shutdown 2024-05-09T08:50:30.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb2491d4740 1 -- 172.21.15.98:0/195899674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 msgr2=0x5582c7022210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb2491d4740 1 --2- 172.21.15.98:0/195899674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c7022210 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb23009aa50 tx=0x7fb230004570 comp rx=0 tx=0).stop 2024-05-09T08:50:30.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb2491d4740 1 -- 172.21.15.98:0/195899674 shutdown_connections 2024-05-09T08:50:30.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb2491d4740 1 --2- 172.21.15.98:0/195899674 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5582c7096da0 0x5582c7095ab0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb2491d4740 1 --2- 172.21.15.98:0/195899674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582c701a830 0x5582c7025770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb2491d4740 1 --2- 172.21.15.98:0/195899674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582c70156c0 0x5582c7022750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb21f7fe640 0 client.15495 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:50:30.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb2491d4740 1 --2- 172.21.15.98:0/195899674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582c7014cd0 0x5582c7022210 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb2491d4740 1 -- 172.21.15.98:0/195899674 >> 172.21.15.98:0/195899674 conn(0x5582c700bf30 msgr2=0x5582c701ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb2491d4740 1 -- 172.21.15.98:0/195899674 shutdown_connections 2024-05-09T08:50:30.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb2491d4740 1 -- 172.21.15.98:0/195899674 wait complete. 2024-05-09T08:50:30.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.324+0000 7fb2491d4740 20 client.15495 trim_cache size 0 max 16384 2024-05-09T08:50:30.333 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:50:30.333 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a57d9712-0ddf-11ef-bc98-c7b262605968", 2024-05-09T08:50:30.334 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2024-05-09T08:50:30.334 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24439, 2024-05-09T08:50:30.334 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-05-09T08:50:30.334 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-05-09T08:50:30.334 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-05-09T08:50:30.334 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 113.043235099, 2024-05-09T08:50:30.334 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-05-09T08:50:30.334 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 94, 2024-05-09T08:50:30.334 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 90, 2024-05-09T08:50:30.334 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 152.62675794500001 2024-05-09T08:50:30.334 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:50:30.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.325+0000 7fb2464bd640 1 -- 172.21.15.98:0/3673326588 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb22c0a2b90 msgr2=0x7fb22c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.325+0000 7fb2464bd640 1 --2- 172.21.15.98:0/3673326588 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb22c0a2b90 0x7fb22c0a5050 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb23804d000 tx=0x7fb23804e000 comp rx=0 tx=0).stop 2024-05-09T08:50:30.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.325+0000 7fb2464bd640 1 -- 172.21.15.98:0/3673326588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb240101870 msgr2=0x7fb240197c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.325+0000 7fb2464bd640 1 --2- 172.21.15.98:0/3673326588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb240101870 0x7fb240197c20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb23000c970 tx=0x7fb23000cd70 comp rx=0 tx=0).stop 2024-05-09T08:50:30.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.325+0000 7fb2464bd640 1 -- 172.21.15.98:0/3673326588 shutdown_connections 2024-05-09T08:50:30.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.325+0000 7fb2464bd640 1 --2- 172.21.15.98:0/3673326588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb240103420 0x7fb24019f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.325+0000 7fb2464bd640 1 --2- 172.21.15.98:0/3673326588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb240102a60 0x7fb240198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.325+0000 7fb2464bd640 1 --2- 172.21.15.98:0/3673326588 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fb22c0a2b90 0x7fb22c0a5050 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.325+0000 7fb2464bd640 1 --2- 172.21.15.98:0/3673326588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb240101870 0x7fb240197c20 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.325+0000 7fb2464bd640 1 -- 172.21.15.98:0/3673326588 >> 172.21.15.98:0/3673326588 conn(0x7fb2400fcde0 msgr2=0x7fb240105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.325+0000 7fb2464bd640 1 -- 172.21.15.98:0/3673326588 shutdown_connections 2024-05-09T08:50:30.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.325+0000 7fb2464bd640 1 -- 172.21.15.98:0/3673326588 wait complete. 2024-05-09T08:50:30.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-05-09T08:50:30.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.441+0000 7f58fd7d5640 1 -- 172.21.15.98:0/3910233391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8101a50 msgr2=0x7f58f8108370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.441+0000 7f58fd7d5640 1 --2- 172.21.15.98:0/3910233391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8101a50 0x7f58f8108370 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009900 tx=0x7f58f002a410 comp rx=0 tx=0).stop 2024-05-09T08:50:30.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.441+0000 7f58fd7d5640 1 -- 172.21.15.98:0/3910233391 shutdown_connections 2024-05-09T08:50:30.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.441+0000 7f58fd7d5640 1 --2- 172.21.15.98:0/3910233391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8101a50 0x7f58f8108370 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.441+0000 7f58fd7d5640 1 --2- 172.21.15.98:0/3910233391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58f8101090 0x7f58f8101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.441+0000 7f58fd7d5640 1 --2- 172.21.15.98:0/3910233391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58f80ffeb0 0x7f58f81002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.441+0000 7f58fd7d5640 1 -- 172.21.15.98:0/3910233391 >> 172.21.15.98:0/3910233391 conn(0x7f58f80fb470 msgr2=0x7f58f80fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.442+0000 7f58fd7d5640 1 -- 172.21.15.98:0/3910233391 shutdown_connections 2024-05-09T08:50:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.442+0000 7f58fd7d5640 1 -- 172.21.15.98:0/3910233391 wait complete. 2024-05-09T08:50:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.442+0000 7f58fd7d5640 1 Processor -- start 2024-05-09T08:50:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.442+0000 7f58fd7d5640 1 -- start start 2024-05-09T08:50:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.442+0000 7f58fd7d5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58f80ffeb0 0x7f58f8197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.442+0000 7f58fd7d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58f8101090 0x7f58f8198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.442+0000 7f58fd7d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8101a50 0x7f58f819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.442+0000 7f58fd7d5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f58f810b380 con 0x7f58f8101a50 2024-05-09T08:50:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.442+0000 7f58fd7d5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f58f810b200 con 0x7f58f8101090 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.442+0000 7f58fd7d5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f58f810b500 con 0x7f58f80ffeb0 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58f77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8101a50 0x7f58f819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58f77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8101a50 0x7f58f819f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50142/0 (socket says 172.21.15.98:50142) 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58f77fe640 1 -- 172.21.15.98:0/209917852 learned_addr learned my addr 172.21.15.98:0/209917852 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58f67fc640 1 --2- 172.21.15.98:0/209917852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58f8101090 0x7f58f8198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58f6ffd640 1 --2- 172.21.15.98:0/209917852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58f80ffeb0 0x7f58f8197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58f77fe640 1 -- 172.21.15.98:0/209917852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58f80ffeb0 msgr2=0x7f58f8197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58f77fe640 1 --2- 172.21.15.98:0/209917852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58f80ffeb0 0x7f58f8197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58f77fe640 1 -- 172.21.15.98:0/209917852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58f8101090 msgr2=0x7f58f8198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58f77fe640 1 --2- 172.21.15.98:0/209917852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58f8101090 0x7f58f8198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58f77fe640 1 -- 172.21.15.98:0/209917852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f819f800 con 0x7f58f8101a50 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58f77fe640 1 --2- 172.21.15.98:0/209917852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8101a50 0x7f58f819f190 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f58f00314e0 tx=0x7f58f0004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58dffff640 1 -- 172.21.15.98:0/209917852 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f003f030 con 0x7f58f8101a50 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58fd7d5640 1 -- 172.21.15.98:0/209917852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f819fa90 con 0x7f58f8101a50 2024-05-09T08:50:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58dffff640 1 -- 172.21.15.98:0/209917852 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f0003ce0 con 0x7f58f8101a50 2024-05-09T08:50:30.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58dffff640 1 -- 172.21.15.98:0/209917852 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0035070 con 0x7f58f8101a50 2024-05-09T08:50:30.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.443+0000 7f58fd7d5640 1 -- 172.21.15.98:0/209917852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f81a0000 con 0x7f58f8101a50 2024-05-09T08:50:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.445+0000 7f58dffff640 1 -- 172.21.15.98:0/209917852 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f58f0042030 con 0x7f58f8101a50 2024-05-09T08:50:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.446+0000 7f59005c3740 1 Processor -- start 2024-05-09T08:50:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.446+0000 7f59005c3740 1 -- start start 2024-05-09T08:50:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.446+0000 7f59005c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbeff8c0 0x55ebdbeffce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.446+0000 7f59005c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf04db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.446+0000 7f59005c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbf05420 0x55ebdbf078b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.446+0000 7f59005c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ebdbe634d0 con 0x55ebdbf002b0 2024-05-09T08:50:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.446+0000 7f59005c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ebdbe0b3e0 con 0x55ebdbf05420 2024-05-09T08:50:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.446+0000 7f59005c3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ebdbe5f490 con 0x55ebdbeff8c0 2024-05-09T08:50:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.446+0000 7f58f6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf04db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.446+0000 7f58f6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf04db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50154/0 (socket says 172.21.15.98:50154) 2024-05-09T08:50:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.446+0000 7f58f6ffd640 1 -- 172.21.15.98:0/1133031730 learned_addr learned my addr 172.21.15.98:0/1133031730 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.446+0000 7f58dffff640 1 --2- 172.21.15.98:0/209917852 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f58e00a2b90 0x7f58e00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.446+0000 7f58f67fc640 1 --2- 172.21.15.98:0/1133031730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbeff8c0 0x55ebdbeffce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.447+0000 7f58f67fc640 1 --2- 172.21.15.98:0/1133031730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbf05420 0x55ebdbf078b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.447+0000 7f58f77fe640 1 --2- 172.21.15.98:0/209917852 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f58e00a2b90 0x7f58e00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.447+0000 7f58f6ffd640 1 -- 172.21.15.98:0/1133031730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbeff8c0 msgr2=0x55ebdbeffce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.447+0000 7f58f6ffd640 1 --2- 172.21.15.98:0/1133031730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbeff8c0 0x55ebdbeffce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.447+0000 7f58f6ffd640 1 -- 172.21.15.98:0/1133031730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbf05420 msgr2=0x55ebdbf078b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.447+0000 7f58f6ffd640 1 --2- 172.21.15.98:0/1133031730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbf05420 0x55ebdbf078b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.447+0000 7f58f6ffd640 1 -- 172.21.15.98:0/1133031730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebdbb772d0 con 0x55ebdbf002b0 2024-05-09T08:50:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.447+0000 7f58f67fc640 1 --2- 172.21.15.98:0/1133031730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbf05420 0x55ebdbf078b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.447+0000 7f58dffff640 1 -- 172.21.15.98:0/209917852 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f58f00efe00 con 0x7f58f8101a50 2024-05-09T08:50:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.447+0000 7f58f6ffd640 1 --2- 172.21.15.98:0/1133031730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf04db0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f58e8007de0 tx=0x7f58e800c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88a0fb0e5f035562 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.447+0000 7f58f77fe640 1 --2- 172.21.15.98:0/209917852 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f58e00a2b90 0x7f58e00a5050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f58f00b86b0 tx=0x7f58f008c100 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.447+0000 7f58dd7fa640 1 -- 172.21.15.98:0/1133031730 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e8015020 con 0x55ebdbf002b0 2024-05-09T08:50:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.447+0000 7f58dd7fa640 1 -- 172.21.15.98:0/1133031730 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58e8011b10 con 0x55ebdbf002b0 2024-05-09T08:50:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.447+0000 7f58dd7fa640 1 -- 172.21.15.98:0/1133031730 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e8010c50 con 0x55ebdbf002b0 2024-05-09T08:50:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 -- 172.21.15.98:0/1133031730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 msgr2=0x55ebdbf04db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 --2- 172.21.15.98:0/1133031730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf04db0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f58e8007de0 tx=0x7f58e800c560 comp rx=0 tx=0).stop 2024-05-09T08:50:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 -- 172.21.15.98:0/1133031730 shutdown_connections 2024-05-09T08:50:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 --2- 172.21.15.98:0/1133031730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbf05420 0x55ebdbf078b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 --2- 172.21.15.98:0/1133031730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf04db0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 --2- 172.21.15.98:0/1133031730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbeff8c0 0x55ebdbeffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 -- 172.21.15.98:0/1133031730 >> 172.21.15.98:0/1133031730 conn(0x55ebdbefaf30 msgr2=0x55ebdbefb330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 -- 172.21.15.98:0/1133031730 shutdown_connections 2024-05-09T08:50:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 -- 172.21.15.98:0/1133031730 wait complete. 2024-05-09T08:50:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 -- start start 2024-05-09T08:50:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 Processor -- start 2024-05-09T08:50:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 0x55ebdbf09ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf05420 0x55ebdbf11250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ebdbf083d0 con 0x55ebdbf002b0 2024-05-09T08:50:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ebdbe1e8e0 con 0x55ebdbeff8c0 2024-05-09T08:50:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ebdbf08550 con 0x55ebdbf05420 2024-05-09T08:50:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f59005c3740 10 client.0 _ll_register_callbacks cb 0x55ebdbdeb9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.448+0000 7f58f6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58f6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50160/0 (socket says 172.21.15.98:50160) 2024-05-09T08:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58f6ffd640 1 -- 172.21.15.98:0/197970829 learned_addr learned my addr 172.21.15.98:0/197970829 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58f67fc640 1 --2- 172.21.15.98:0/197970829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 0x55ebdbf09ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58f67fc640 1 --2- 172.21.15.98:0/197970829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf05420 0x55ebdbf11250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58f6ffd640 1 -- 172.21.15.98:0/197970829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf05420 msgr2=0x55ebdbf11250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58f6ffd640 1 --2- 172.21.15.98:0/197970829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf05420 0x55ebdbf11250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58f6ffd640 1 -- 172.21.15.98:0/197970829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 msgr2=0x55ebdbf09ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58f6ffd640 1 --2- 172.21.15.98:0/197970829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 0x55ebdbf09ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58f6ffd640 1 -- 172.21.15.98:0/197970829 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebdbc416b0 con 0x55ebdbf002b0 2024-05-09T08:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58f67fc640 1 --2- 172.21.15.98:0/197970829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf05420 0x55ebdbf11250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58f6ffd640 1 --2- 172.21.15.98:0/197970829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0a220 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f58e800ca60 tx=0x7f58e800cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be8020d2a1e9337 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58daff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58daff5640 1 -- 172.21.15.98:0/197970829 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e8029060 con 0x55ebdbf002b0 2024-05-09T08:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58daff5640 1 -- 172.21.15.98:0/197970829 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f58e8011640 con 0x55ebdbf002b0 2024-05-09T08:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f59005c3740 1 -- 172.21.15.98:0/197970829 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58e8011b10 con 0x55ebdbf002b0 2024-05-09T08:50:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58daff5640 1 -- 172.21.15.98:0/197970829 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f58e8030050 con 0x55ebdbf002b0 2024-05-09T08:50:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f59005c3740 10 client.15507 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f59005c3740 1 -- 172.21.15.98:0/197970829 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ebdbf14230 con 0x55ebdbf002b0 2024-05-09T08:50:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58daff5640 1 -- 172.21.15.98:0/197970829 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58e8011b10 con 0x55ebdbf002b0 2024-05-09T08:50:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.449+0000 7f58daff5640 1 -- 172.21.15.98:0/197970829 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f58e8023030 con 0x55ebdbf002b0 2024-05-09T08:50:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.450+0000 7f59005c3740 10 client.15507 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.450+0000 7f59005c3740 10 client.15507 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T08:50:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.450+0000 7f59005c3740 20 client.15507 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.450+0000 7f59005c3740 1 --2- 172.21.15.98:0/197970829 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55ebdbf166b0 0x55ebdbf17b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.450+0000 7f59005c3740 4 client.15507 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:50:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.450+0000 7f59005c3740 1 -- 172.21.15.98:0/197970829 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebdbe70c00 con 0x55ebdbf166b0 2024-05-09T08:50:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.450+0000 7f58f67fc640 1 --2- 172.21.15.98:0/197970829 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55ebdbf166b0 0x55ebdbf17b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.450+0000 7f58f67fc640 1 --2- 172.21.15.98:0/197970829 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55ebdbf166b0 0x55ebdbf17b20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7fa015adcf601974 server_cookie=d8c8a494563aff48 in_seq=0 out_seq=0 2024-05-09T08:50:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.450+0000 7f58daff5640 10 client.15507 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.451+0000 7f58daff5640 1 -- 172.21.15.98:0/197970829 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ebdbe70c00 con 0x55ebdbf166b0 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.451+0000 7f58daff5640 10 client.15507 handle_command_reply: tid=0 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.451+0000 7f59005c3740 1 client.15507 shutdown 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.451+0000 7f59005c3740 1 -- 172.21.15.98:0/197970829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 msgr2=0x55ebdbf0a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.451+0000 7f59005c3740 1 --2- 172.21.15.98:0/197970829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0a220 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f58e800ca60 tx=0x7f58e800cf00 comp rx=0 tx=0).stop 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.451+0000 7f59005c3740 1 -- 172.21.15.98:0/197970829 shutdown_connections 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.451+0000 7f59005c3740 1 --2- 172.21.15.98:0/197970829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf05420 0x55ebdbf11250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.451+0000 7f59005c3740 1 --2- 172.21.15.98:0/197970829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0a220 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.451+0000 7f59005c3740 1 --2- 172.21.15.98:0/197970829 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55ebdbf166b0 0x55ebdbf17b20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.451+0000 7f59005c3740 1 --2- 172.21.15.98:0/197970829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 0x55ebdbf09ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.451+0000 7f58daff5640 0 client.15507 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.451+0000 7f59005c3740 1 -- 172.21.15.98:0/197970829 >> 172.21.15.98:0/197970829 conn(0x55ebdbefaf30 msgr2=0x55ebdbefa000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.452+0000 7f59005c3740 1 -- 172.21.15.98:0/197970829 shutdown_connections 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.452+0000 7f59005c3740 1 -- 172.21.15.98:0/197970829 wait complete. 2024-05-09T08:50:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.452+0000 7f59005c3740 20 client.15507 trim_cache size 0 max 16384 2024-05-09T08:50:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.469+0000 7f59005c3740 1 Processor -- start 2024-05-09T08:50:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.469+0000 7f59005c3740 1 -- start start 2024-05-09T08:50:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.469+0000 7f59005c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 0x55ebdbf09ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.469+0000 7f59005c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.469+0000 7f59005c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf05420 0x55ebdbf11210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.469+0000 7f59005c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ebdbf15e60 con 0x55ebdbf05420 2024-05-09T08:50:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.469+0000 7f59005c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ebdbf16530 con 0x55ebdbeff8c0 2024-05-09T08:50:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.469+0000 7f59005c3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ebdbf16c00 con 0x55ebdbf002b0 2024-05-09T08:50:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.469+0000 7f58f67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf05420 0x55ebdbf11210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.469+0000 7f58f67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf05420 0x55ebdbf11210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50162/0 (socket says 172.21.15.98:50162) 2024-05-09T08:50:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.469+0000 7f58f67fc640 1 -- 172.21.15.98:0/261477012 learned_addr learned my addr 172.21.15.98:0/261477012 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.470+0000 7f58f67fc640 1 --2- 172.21.15.98:0/261477012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 0x55ebdbf09ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.470+0000 7f58f6ffd640 1 --2- 172.21.15.98:0/261477012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.470+0000 7f58f67fc640 1 -- 172.21.15.98:0/261477012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf002b0 msgr2=0x55ebdbf0a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.470+0000 7f58f67fc640 1 --2- 172.21.15.98:0/261477012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.470+0000 7f58f67fc640 1 -- 172.21.15.98:0/261477012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 msgr2=0x55ebdbf09ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.470+0000 7f58f67fc640 1 --2- 172.21.15.98:0/261477012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 0x55ebdbf09ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.470+0000 7f58f67fc640 1 -- 172.21.15.98:0/261477012 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebdbf00690 con 0x55ebdbf05420 2024-05-09T08:50:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.470+0000 7f58f67fc640 1 --2- 172.21.15.98:0/261477012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf05420 0x55ebdbf11210 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f58ec00c010 tx=0x7f58ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=339e194652a31a20 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.470+0000 7f58dcff9640 1 -- 172.21.15.98:0/261477012 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ec025020 con 0x55ebdbf05420 2024-05-09T08:50:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.470+0000 7f58dcff9640 1 -- 172.21.15.98:0/261477012 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58ec00fbc0 con 0x55ebdbf05420 2024-05-09T08:50:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.470+0000 7f58dcff9640 1 -- 172.21.15.98:0/261477012 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ec0053c0 con 0x55ebdbf05420 2024-05-09T08:50:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.470+0000 7f59005c3740 1 -- 172.21.15.98:0/261477012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf05420 msgr2=0x55ebdbf11210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.470+0000 7f59005c3740 1 --2- 172.21.15.98:0/261477012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf05420 0x55ebdbf11210 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f58ec00c010 tx=0x7f58ec004730 comp rx=0 tx=0).stop 2024-05-09T08:50:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 -- 172.21.15.98:0/261477012 shutdown_connections 2024-05-09T08:50:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 --2- 172.21.15.98:0/261477012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf05420 0x55ebdbf11210 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 --2- 172.21.15.98:0/261477012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 --2- 172.21.15.98:0/261477012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 0x55ebdbf09ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 -- 172.21.15.98:0/261477012 >> 172.21.15.98:0/261477012 conn(0x55ebdbefaf30 msgr2=0x55ebdbefb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 -- 172.21.15.98:0/261477012 shutdown_connections 2024-05-09T08:50:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 -- 172.21.15.98:0/261477012 wait complete. 2024-05-09T08:50:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 -- start start 2024-05-09T08:50:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 Processor -- start 2024-05-09T08:50:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 0x55ebdbf0d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf05420 0x55ebdbf10510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ebdbf179a0 con 0x55ebdbf002b0 2024-05-09T08:50:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ebdbf172d0 con 0x55ebdbeff8c0 2024-05-09T08:50:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ebdbf18070 con 0x55ebdbf05420 2024-05-09T08:50:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f59005c3740 10 client.0 _ll_register_callbacks cb 0x7f58ec00bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f58f6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f58f6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50166/0 (socket says 172.21.15.98:50166) 2024-05-09T08:50:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f58f6ffd640 1 -- 172.21.15.98:0/211326724 learned_addr learned my addr 172.21.15.98:0/211326724 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f58f67fc640 1 --2- 172.21.15.98:0/211326724 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 0x55ebdbf0d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f58f6ffd640 1 -- 172.21.15.98:0/211326724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf05420 msgr2=0x55ebdbf10510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f58f6ffd640 1 --2- 172.21.15.98:0/211326724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf05420 0x55ebdbf10510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f58f6ffd640 1 -- 172.21.15.98:0/211326724 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 msgr2=0x55ebdbf0d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f58f6ffd640 1 --2- 172.21.15.98:0/211326724 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 0x55ebdbf0d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f58f6ffd640 1 -- 172.21.15.98:0/211326724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebdbf10b00 con 0x55ebdbf002b0 2024-05-09T08:50:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.471+0000 7f58f67fc640 1 --2- 172.21.15.98:0/211326724 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 0x55ebdbf0d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f58f6ffd640 1 --2- 172.21.15.98:0/211326724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0d590 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x55ebdbf0b1b0 tx=0x7f58e8010f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d3746409aefac8a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f58daff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f58daff5640 1 -- 172.21.15.98:0/211326724 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e8015040 con 0x55ebdbf002b0 2024-05-09T08:50:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f58daff5640 1 -- 172.21.15.98:0/211326724 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f58e8011640 con 0x55ebdbf002b0 2024-05-09T08:50:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f59005c3740 1 -- 172.21.15.98:0/211326724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58ec00fbc0 con 0x55ebdbf002b0 2024-05-09T08:50:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f58daff5640 1 -- 172.21.15.98:0/211326724 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f58e8029040 con 0x55ebdbf002b0 2024-05-09T08:50:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f59005c3740 10 client.15513 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f59005c3740 1 -- 172.21.15.98:0/211326724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ebdbf7cc50 con 0x55ebdbf002b0 2024-05-09T08:50:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f58daff5640 1 -- 172.21.15.98:0/211326724 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58ec00fbc0 con 0x55ebdbf002b0 2024-05-09T08:50:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f58daff5640 1 -- 172.21.15.98:0/211326724 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f58e8023040 con 0x55ebdbf002b0 2024-05-09T08:50:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f59005c3740 10 client.15513 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f59005c3740 10 client.15513 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T08:50:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f59005c3740 20 client.15513 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f59005c3740 1 --2- 172.21.15.98:0/211326724 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55ebdbf81930 0x55ebdbf7dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f59005c3740 4 client.15513 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:50:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.472+0000 7f59005c3740 1 -- 172.21.15.98:0/211326724 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "status"}) -- 0x55ebdbe6af50 con 0x55ebdbf81930 2024-05-09T08:50:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f58f67fc640 1 --2- 172.21.15.98:0/211326724 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55ebdbf81930 0x55ebdbf7dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f58daff5640 10 client.15513 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:50:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f58f67fc640 1 --2- 172.21.15.98:0/211326724 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55ebdbf81930 0x55ebdbf7dd70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=31c9c4d0a40da1a server_cookie=6e32b726ce639f8e in_seq=0 out_seq=0 2024-05-09T08:50:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f58daff5640 1 -- 172.21.15.98:0/211326724 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55ebdbe6af50 con 0x55ebdbf81930 2024-05-09T08:50:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f58daff5640 10 client.15513 handle_command_reply: tid=0 2024-05-09T08:50:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f59005c3740 1 client.15513 shutdown 2024-05-09T08:50:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f59005c3740 1 -- 172.21.15.98:0/211326724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 msgr2=0x55ebdbf0d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f59005c3740 1 --2- 172.21.15.98:0/211326724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0d590 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x55ebdbf0b1b0 tx=0x7f58e8010f30 comp rx=0 tx=0).stop 2024-05-09T08:50:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f59005c3740 1 -- 172.21.15.98:0/211326724 shutdown_connections 2024-05-09T08:50:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f59005c3740 1 --2- 172.21.15.98:0/211326724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ebdbf05420 0x55ebdbf10510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f59005c3740 1 --2- 172.21.15.98:0/211326724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebdbf002b0 0x55ebdbf0d590 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f59005c3740 1 --2- 172.21.15.98:0/211326724 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55ebdbf81930 0x55ebdbf7dd70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f59005c3740 1 --2- 172.21.15.98:0/211326724 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ebdbeff8c0 0x55ebdbf0d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f58daff5640 0 client.15513 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:50:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f59005c3740 1 -- 172.21.15.98:0/211326724 >> 172.21.15.98:0/211326724 conn(0x55ebdbefaf30 msgr2=0x55ebdbf01630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f59005c3740 1 -- 172.21.15.98:0/211326724 shutdown_connections 2024-05-09T08:50:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f59005c3740 1 -- 172.21.15.98:0/211326724 wait complete. 2024-05-09T08:50:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.473+0000 7f59005c3740 20 client.15513 trim_cache size 0 max 16384 2024-05-09T08:50:30.480 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:50:30.480 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a57d9712-0ddf-11ef-bc98-c7b262605968", 2024-05-09T08:50:30.480 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2024-05-09T08:50:30.480 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24457, 2024-05-09T08:50:30.481 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-05-09T08:50:30.481 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-05-09T08:50:30.481 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-05-09T08:50:30.481 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 114.20521031, 2024-05-09T08:50:30.481 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-05-09T08:50:30.481 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 94, 2024-05-09T08:50:30.481 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 90, 2024-05-09T08:50:30.481 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 147.04661250999999 2024-05-09T08:50:30.481 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:50:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.474+0000 7f58fd7d5640 1 -- 172.21.15.98:0/209917852 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f58e00a2b90 msgr2=0x7f58e00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.474+0000 7f58fd7d5640 1 --2- 172.21.15.98:0/209917852 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f58e00a2b90 0x7f58e00a5050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f58f00b86b0 tx=0x7f58f008c100 comp rx=0 tx=0).stop 2024-05-09T08:50:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.474+0000 7f58fd7d5640 1 -- 172.21.15.98:0/209917852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8101a50 msgr2=0x7f58f819f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.474+0000 7f58fd7d5640 1 --2- 172.21.15.98:0/209917852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8101a50 0x7f58f819f190 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f58f00314e0 tx=0x7f58f0004360 comp rx=0 tx=0).stop 2024-05-09T08:50:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.474+0000 7f58fd7d5640 1 -- 172.21.15.98:0/209917852 shutdown_connections 2024-05-09T08:50:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.474+0000 7f58fd7d5640 1 --2- 172.21.15.98:0/209917852 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f58e00a2b90 0x7f58e00a5050 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.474+0000 7f58fd7d5640 1 --2- 172.21.15.98:0/209917852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f8101a50 0x7f58f819f190 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.474+0000 7f58fd7d5640 1 --2- 172.21.15.98:0/209917852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58f8101090 0x7f58f8198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.474+0000 7f58fd7d5640 1 --2- 172.21.15.98:0/209917852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58f80ffeb0 0x7f58f8197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.474+0000 7f58fd7d5640 1 -- 172.21.15.98:0/209917852 >> 172.21.15.98:0/209917852 conn(0x7f58f80fb470 msgr2=0x7f58f81065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.474+0000 7f58fd7d5640 1 -- 172.21.15.98:0/209917852 shutdown_connections 2024-05-09T08:50:30.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.474+0000 7f58fd7d5640 1 -- 172.21.15.98:0/209917852 wait complete. 2024-05-09T08:50:30.483 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:30 smithi098 ceph-mon[30092]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 6 B/s, 0 keys/s, 0 objects/s recovering 2024-05-09T08:50:30.483 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:30 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3770834449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:30.483 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:30 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1193691523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:30.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-05-09T08:50:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.586+0000 7fd4dce76640 1 -- 172.21.15.98:0/2562452012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d8103430 msgr2=0x7fd4d8109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.586+0000 7fd4dce76640 1 --2- 172.21.15.98:0/2562452012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d8103430 0x7fd4d8109cc0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0009ca0 tx=0x7fd4d002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.586+0000 7fd4dce76640 1 -- 172.21.15.98:0/2562452012 shutdown_connections 2024-05-09T08:50:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.586+0000 7fd4dce76640 1 --2- 172.21.15.98:0/2562452012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d8103430 0x7fd4d8109cc0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.586+0000 7fd4dce76640 1 --2- 172.21.15.98:0/2562452012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4d8102a70 0x7fd4d8102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.586+0000 7fd4dce76640 1 --2- 172.21.15.98:0/2562452012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4d8101880 0x7fd4d8101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.586+0000 7fd4dce76640 1 -- 172.21.15.98:0/2562452012 >> 172.21.15.98:0/2562452012 conn(0x7fd4d80fce10 msgr2=0x7fd4d80ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.586+0000 7fd4dce76640 1 -- 172.21.15.98:0/2562452012 shutdown_connections 2024-05-09T08:50:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.586+0000 7fd4dce76640 1 -- 172.21.15.98:0/2562452012 wait complete. 2024-05-09T08:50:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4dce76640 1 Processor -- start 2024-05-09T08:50:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4dce76640 1 -- start start 2024-05-09T08:50:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4dce76640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4d8101880 0x7fd4d8197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4dce76640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d8102a70 0x7fd4d8198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4dce76640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4d8103430 0x7fd4d819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4dce76640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d810d4e0 con 0x7fd4d8102a70 2024-05-09T08:50:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4dce76640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d810d360 con 0x7fd4d8103430 2024-05-09T08:50:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4dce76640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d810d660 con 0x7fd4d8101880 2024-05-09T08:50:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4cffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d8102a70 0x7fd4d8198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4cffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d8102a70 0x7fd4d8198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50192/0 (socket says 172.21.15.98:50192) 2024-05-09T08:50:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4cffff640 1 -- 172.21.15.98:0/3442285761 learned_addr learned my addr 172.21.15.98:0/3442285761 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4d659c640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4d8101880 0x7fd4d8197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4d6d9d640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4d8103430 0x7fd4d819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4cffff640 1 -- 172.21.15.98:0/3442285761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4d8101880 msgr2=0x7fd4d8197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4cffff640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4d8101880 0x7fd4d8197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4cffff640 1 -- 172.21.15.98:0/3442285761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4d8103430 msgr2=0x7fd4d819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4cffff640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4d8103430 0x7fd4d819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4cffff640 1 -- 172.21.15.98:0/3442285761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d819f740 con 0x7fd4d8102a70 2024-05-09T08:50:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.587+0000 7fd4d6d9d640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4d8103430 0x7fd4d819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.588+0000 7fd4cffff640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d8102a70 0x7fd4d8198060 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000d950 tx=0x7fd4c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.588+0000 7fd4cf7fe640 1 -- 172.21.15.98:0/3442285761 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0017070 con 0x7fd4d8102a70 2024-05-09T08:50:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.588+0000 7fd4dce76640 1 -- 172.21.15.98:0/3442285761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d819fa30 con 0x7fd4d8102a70 2024-05-09T08:50:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.588+0000 7fd4cf7fe640 1 -- 172.21.15.98:0/3442285761 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4c0009c30 con 0x7fd4d8102a70 2024-05-09T08:50:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.588+0000 7fd4cf7fe640 1 -- 172.21.15.98:0/3442285761 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c000f6b0 con 0x7fd4d8102a70 2024-05-09T08:50:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.588+0000 7fd4dce76640 1 -- 172.21.15.98:0/3442285761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d819ff80 con 0x7fd4d8102a70 2024-05-09T08:50:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.590+0000 7fd4cf7fe640 1 -- 172.21.15.98:0/3442285761 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd4c00109d0 con 0x7fd4d8102a70 2024-05-09T08:50:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.590+0000 7fd4dfb87740 1 Processor -- start 2024-05-09T08:50:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.590+0000 7fd4dfb87740 1 -- start start 2024-05-09T08:50:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4dfb87740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc30cd0 0x55986bc310f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4dfb87740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc316c0 0x55986bc361c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4dfb87740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc36830 0x55986bc38cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4dfb87740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55986bb904d0 con 0x55986bc30cd0 2024-05-09T08:50:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4dfb87740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55986bb383e0 con 0x55986bc316c0 2024-05-09T08:50:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4dfb87740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55986bb8c490 con 0x55986bc36830 2024-05-09T08:50:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4d6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc30cd0 0x55986bc310f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4d6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc30cd0 0x55986bc310f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50196/0 (socket says 172.21.15.98:50196) 2024-05-09T08:50:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4d6d9d640 1 -- 172.21.15.98:0/257827527 learned_addr learned my addr 172.21.15.98:0/257827527 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4d659c640 1 --2- 172.21.15.98:0/257827527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc316c0 0x55986bc361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4cffff640 1 --2- 172.21.15.98:0/257827527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc36830 0x55986bc38cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4cf7fe640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd4c40a2b90 0x7fd4c40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4d6d9d640 1 -- 172.21.15.98:0/257827527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc36830 msgr2=0x55986bc38cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4d6d9d640 1 --2- 172.21.15.98:0/257827527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc36830 0x55986bc38cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4d6d9d640 1 -- 172.21.15.98:0/257827527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc316c0 msgr2=0x55986bc361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4d6d9d640 1 --2- 172.21.15.98:0/257827527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc316c0 0x55986bc361c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4d6d9d640 1 -- 172.21.15.98:0/257827527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55986b8a42d0 con 0x55986bc30cd0 2024-05-09T08:50:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4cffff640 1 --2- 172.21.15.98:0/257827527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc36830 0x55986bc38cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4d6d9d640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd4c40a2b90 0x7fd4c40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4cf7fe640 1 -- 172.21.15.98:0/3442285761 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fd4c00cf0a0 con 0x7fd4d8102a70 2024-05-09T08:50:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4d6d9d640 1 --2- 172.21.15.98:0/257827527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc30cd0 0x55986bc310f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0005cc0 tx=0x7fd4d00099a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e2b3c87f89e75c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4ccff9640 1 -- 172.21.15.98:0/257827527 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0004040 con 0x55986bc30cd0 2024-05-09T08:50:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4ccff9640 1 -- 172.21.15.98:0/257827527 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4d0036070 con 0x55986bc30cd0 2024-05-09T08:50:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.591+0000 7fd4ccff9640 1 -- 172.21.15.98:0/257827527 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d002fbe0 con 0x55986bc30cd0 2024-05-09T08:50:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.592+0000 7fd4d6d9d640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd4c40a2b90 0x7fd4c40a5050 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd4d004b000 tx=0x7fd4d0048000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.592+0000 7fd4dfb87740 1 -- 172.21.15.98:0/257827527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc30cd0 msgr2=0x55986bc310f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.592+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/257827527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc30cd0 0x55986bc310f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0005cc0 tx=0x7fd4d00099a0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.592+0000 7fd4dfb87740 1 -- 172.21.15.98:0/257827527 shutdown_connections 2024-05-09T08:50:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.592+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/257827527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc36830 0x55986bc38cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.592+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/257827527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc316c0 0x55986bc361c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.592+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/257827527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc30cd0 0x55986bc310f0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.592+0000 7fd4dfb87740 1 -- 172.21.15.98:0/257827527 >> 172.21.15.98:0/257827527 conn(0x55986bc27f30 msgr2=0x55986bc28330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.592+0000 7fd4dfb87740 1 -- 172.21.15.98:0/257827527 shutdown_connections 2024-05-09T08:50:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.592+0000 7fd4dfb87740 1 -- 172.21.15.98:0/257827527 wait complete. 2024-05-09T08:50:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4dfb87740 1 -- start start 2024-05-09T08:50:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4dfb87740 1 Processor -- start 2024-05-09T08:50:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4dfb87740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc30cd0 0x55986bc3afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4dfb87740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc316c0 0x55986bc3b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4dfb87740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc36830 0x55986bc42530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4dfb87740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55986bc39660 con 0x55986bc36830 2024-05-09T08:50:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4dfb87740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55986bb4b8e0 con 0x55986bc30cd0 2024-05-09T08:50:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4dfb87740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55986bc397e0 con 0x55986bc316c0 2024-05-09T08:50:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4dfb87740 10 client.0 _ll_register_callbacks cb 0x55986bb189c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4d6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc36830 0x55986bc42530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4d6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc36830 0x55986bc42530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50210/0 (socket says 172.21.15.98:50210) 2024-05-09T08:50:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4d6d9d640 1 -- 172.21.15.98:0/1782111094 learned_addr learned my addr 172.21.15.98:0/1782111094 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4cffff640 1 --2- 172.21.15.98:0/1782111094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc316c0 0x55986bc3b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4d659c640 1 --2- 172.21.15.98:0/1782111094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc30cd0 0x55986bc3afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4d6d9d640 1 -- 172.21.15.98:0/1782111094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc316c0 msgr2=0x55986bc3b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4d6d9d640 1 --2- 172.21.15.98:0/1782111094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc316c0 0x55986bc3b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4d6d9d640 1 -- 172.21.15.98:0/1782111094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc30cd0 msgr2=0x55986bc3afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4d6d9d640 1 --2- 172.21.15.98:0/1782111094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc30cd0 0x55986bc3afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4d6d9d640 1 -- 172.21.15.98:0/1782111094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55986b96e6b0 con 0x55986bc36830 2024-05-09T08:50:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4d6d9d640 1 --2- 172.21.15.98:0/1782111094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc36830 0x55986bc42530 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd4d002a900 tx=0x7fd4d00041f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864e3d4edbff397f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.593+0000 7fd4bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4bdffb640 1 -- 172.21.15.98:0/1782111094 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d002fec0 con 0x55986bc36830 2024-05-09T08:50:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4bdffb640 1 -- 172.21.15.98:0/1782111094 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fd4d004d070 con 0x55986bc36830 2024-05-09T08:50:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1782111094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4d0036070 con 0x55986bc36830 2024-05-09T08:50:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4bdffb640 1 -- 172.21.15.98:0/1782111094 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd4d0004520 con 0x55986bc36830 2024-05-09T08:50:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4dfb87740 10 client.15525 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1782111094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55986bc39660 con 0x55986bc36830 2024-05-09T08:50:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4bdffb640 1 -- 172.21.15.98:0/1782111094 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4d0036070 con 0x55986bc36830 2024-05-09T08:50:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4bdffb640 1 -- 172.21.15.98:0/1782111094 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd4d0051480 con 0x55986bc36830 2024-05-09T08:50:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4dfb87740 10 client.15525 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4dfb87740 10 client.15525 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-09T08:50:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4dfb87740 20 client.15525 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1782111094 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55986bc47870 0x55986bc48ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4dfb87740 4 client.15525 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:50:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1782111094 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55986bb9dc00 con 0x55986bc47870 2024-05-09T08:50:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.594+0000 7fd4d659c640 1 --2- 172.21.15.98:0/1782111094 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55986bc47870 0x55986bc48ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.595+0000 7fd4d659c640 1 --2- 172.21.15.98:0/1782111094 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55986bc47870 0x55986bc48ce0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a04debb3b625ae02 server_cookie=311f92be5b389c41 in_seq=0 out_seq=0 2024-05-09T08:50:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.595+0000 7fd4bdffb640 10 client.15525 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.595+0000 7fd4bdffb640 1 -- 172.21.15.98:0/1782111094 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55986bb9dc00 con 0x55986bc47870 2024-05-09T08:50:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.595+0000 7fd4bdffb640 10 client.15525 handle_command_reply: tid=0 2024-05-09T08:50:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.596+0000 7fd4dfb87740 1 client.15525 shutdown 2024-05-09T08:50:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.596+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1782111094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc36830 msgr2=0x55986bc42530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.596+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1782111094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc36830 0x55986bc42530 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd4d002a900 tx=0x7fd4d00041f0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.596+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1782111094 shutdown_connections 2024-05-09T08:50:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.596+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1782111094 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55986bc47870 0x55986bc48ce0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.596+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1782111094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc36830 0x55986bc42530 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.596+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1782111094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc316c0 0x55986bc3b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.596+0000 7fd4bdffb640 0 client.15525 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.596+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1782111094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc30cd0 0x55986bc3afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.596+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1782111094 >> 172.21.15.98:0/1782111094 conn(0x55986bc27f30 msgr2=0x55986bc27000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.596+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1782111094 shutdown_connections 2024-05-09T08:50:30.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.596+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1782111094 wait complete. 2024-05-09T08:50:30.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.596+0000 7fd4dfb87740 20 client.15525 trim_cache size 0 max 16384 2024-05-09T08:50:30.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.613+0000 7fd4dfb87740 1 Processor -- start 2024-05-09T08:50:30.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.613+0000 7fd4dfb87740 1 -- start start 2024-05-09T08:50:30.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4dfb87740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc30cd0 0x55986bc32a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4dfb87740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc316c0 0x55986bc32fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4dfb87740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc36830 0x55986bc424f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4dfb87740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55986bc47020 con 0x55986bc36830 2024-05-09T08:50:30.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4dfb87740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55986bc476f0 con 0x55986bc316c0 2024-05-09T08:50:30.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4dfb87740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55986bc47dc0 con 0x55986bc30cd0 2024-05-09T08:50:30.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4d6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc36830 0x55986bc424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4d6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc36830 0x55986bc424f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50212/0 (socket says 172.21.15.98:50212) 2024-05-09T08:50:30.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4d6d9d640 1 -- 172.21.15.98:0/1317712772 learned_addr learned my addr 172.21.15.98:0/1317712772 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4d6d9d640 1 -- 172.21.15.98:0/1317712772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc30cd0 msgr2=0x55986bc32a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4d6d9d640 1 --2- 172.21.15.98:0/1317712772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc30cd0 0x55986bc32a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4d6d9d640 1 -- 172.21.15.98:0/1317712772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc316c0 msgr2=0x55986bc32fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4d6d9d640 1 --2- 172.21.15.98:0/1317712772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc316c0 0x55986bc32fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4d6d9d640 1 -- 172.21.15.98:0/1317712772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55986bc3be50 con 0x55986bc36830 2024-05-09T08:50:30.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4d6d9d640 1 --2- 172.21.15.98:0/1317712772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc36830 0x55986bc424f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0001220 tx=0x7fd4d0009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ac0aad6f35473d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4bffff640 1 -- 172.21.15.98:0/1317712772 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d002fec0 con 0x55986bc36830 2024-05-09T08:50:30.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4bffff640 1 -- 172.21.15.98:0/1317712772 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4d0036070 con 0x55986bc36830 2024-05-09T08:50:30.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.614+0000 7fd4bffff640 1 -- 172.21.15.98:0/1317712772 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0051b60 con 0x55986bc36830 2024-05-09T08:50:30.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1317712772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc36830 msgr2=0x55986bc424f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1317712772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc36830 0x55986bc424f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0001220 tx=0x7fd4d0009bf0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1317712772 shutdown_connections 2024-05-09T08:50:30.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1317712772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc36830 0x55986bc424f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1317712772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc316c0 0x55986bc32fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1317712772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc30cd0 0x55986bc32a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1317712772 >> 172.21.15.98:0/1317712772 conn(0x55986bc27f30 msgr2=0x55986bc28310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1317712772 shutdown_connections 2024-05-09T08:50:30.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1317712772 wait complete. 2024-05-09T08:50:30.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 -- start start 2024-05-09T08:50:30.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 Processor -- start 2024-05-09T08:50:30.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc30cd0 0x55986bc3e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc316c0 0x55986bc3e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc36830 0x55986bc415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55986bc48b60 con 0x55986bc30cd0 2024-05-09T08:50:30.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55986bc48490 con 0x55986bc316c0 2024-05-09T08:50:30.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55986bc49230 con 0x55986bc36830 2024-05-09T08:50:30.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.615+0000 7fd4dfb87740 10 client.0 _ll_register_callbacks cb 0x7fd4d004a020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4d659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc30cd0 0x55986bc3e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4d659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc30cd0 0x55986bc3e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50226/0 (socket says 172.21.15.98:50226) 2024-05-09T08:50:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4d659c640 1 -- 172.21.15.98:0/1867330984 learned_addr learned my addr 172.21.15.98:0/1867330984 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4d659c640 1 -- 172.21.15.98:0/1867330984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc36830 msgr2=0x55986bc415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4d659c640 1 --2- 172.21.15.98:0/1867330984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc36830 0x55986bc415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4cffff640 1 --2- 172.21.15.98:0/1867330984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc316c0 0x55986bc3e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4d659c640 1 -- 172.21.15.98:0/1867330984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc316c0 msgr2=0x55986bc3e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4d659c640 1 --2- 172.21.15.98:0/1867330984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc316c0 0x55986bc3e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4d659c640 1 -- 172.21.15.98:0/1867330984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55986bc41bb0 con 0x55986bc30cd0 2024-05-09T08:50:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4d659c640 1 --2- 172.21.15.98:0/1867330984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc30cd0 0x55986bc3e060 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd4c800ad40 tx=0x7fd4c8010af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13010b9207396359 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4bdffb640 1 -- 172.21.15.98:0/1867330984 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8004440 con 0x55986bc30cd0 2024-05-09T08:50:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4bdffb640 1 -- 172.21.15.98:0/1867330984 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fd4c80201b0 con 0x55986bc30cd0 2024-05-09T08:50:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1867330984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4d0036070 con 0x55986bc30cd0 2024-05-09T08:50:30.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4bdffb640 1 -- 172.21.15.98:0/1867330984 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd4c802f050 con 0x55986bc30cd0 2024-05-09T08:50:30.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4dfb87740 10 client.15531 fetch_fsmap learned FSMap version 28 2024-05-09T08:50:30.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1867330984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55986bcadd20 con 0x55986bc30cd0 2024-05-09T08:50:30.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.616+0000 7fd4bdffb640 1 -- 172.21.15.98:0/1867330984 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4d0036070 con 0x55986bc30cd0 2024-05-09T08:50:30.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.617+0000 7fd4bdffb640 1 -- 172.21.15.98:0/1867330984 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd4c8003bf0 con 0x55986bc30cd0 2024-05-09T08:50:30.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.617+0000 7fd4dfb87740 10 client.15531 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:50:30.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.617+0000 7fd4dfb87740 10 client.15531 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-09T08:50:30.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.617+0000 7fd4dfb87740 20 client.15531 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:30.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.617+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1867330984 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55986bcb2a00 0x55986bcaee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.617+0000 7fd4dfb87740 4 client.15531 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:50:30.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.617+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1867330984 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "status"}) -- 0x55986bb97f50 con 0x55986bcb2a00 2024-05-09T08:50:30.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.617+0000 7fd4cffff640 1 --2- 172.21.15.98:0/1867330984 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55986bcb2a00 0x55986bcaee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.617+0000 7fd4cffff640 1 --2- 172.21.15.98:0/1867330984 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55986bcb2a00 0x55986bcaee40 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f697212a9dc2b5f server_cookie=765eb9d9307dc709 in_seq=0 out_seq=0 2024-05-09T08:50:30.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.617+0000 7fd4bdffb640 10 client.15531 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.617+0000 7fd4bdffb640 1 -- 172.21.15.98:0/1867330984 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55986bb97f50 con 0x55986bcb2a00 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.617+0000 7fd4bdffb640 10 client.15531 handle_command_reply: tid=0 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.617+0000 7fd4dfb87740 1 client.15531 shutdown 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1867330984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc30cd0 msgr2=0x55986bc3e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1867330984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc30cd0 0x55986bc3e060 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd4c800ad40 tx=0x7fd4c8010af0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1867330984 shutdown_connections 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1867330984 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55986bcb2a00 0x55986bcaee40 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1867330984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55986bc36830 0x55986bc415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1867330984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55986bc316c0 0x55986bc3e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dfb87740 1 --2- 172.21.15.98:0/1867330984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55986bc30cd0 0x55986bc3e060 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4bdffb640 0 client.15531 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1867330984 >> 172.21.15.98:0/1867330984 conn(0x55986bc27f30 msgr2=0x55986bc32a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1867330984 shutdown_connections 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dfb87740 1 -- 172.21.15.98:0/1867330984 wait complete. 2024-05-09T08:50:30.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dfb87740 20 client.15531 trim_cache size 0 max 16384 2024-05-09T08:50:30.625 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:50:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a57d9712-0ddf-11ef-bc98-c7b262605968", 2024-05-09T08:50:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-05-09T08:50:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24461, 2024-05-09T08:50:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-05-09T08:50:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-05-09T08:50:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-05-09T08:50:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 131.89107454399999, 2024-05-09T08:50:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-05-09T08:50:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 94, 2024-05-09T08:50:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 90, 2024-05-09T08:50:30.626 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 149.97496828800001 2024-05-09T08:50:30.626 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:50:30.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dce76640 1 -- 172.21.15.98:0/3442285761 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd4c40a2b90 msgr2=0x7fd4c40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dce76640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd4c40a2b90 0x7fd4c40a5050 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd4d004b000 tx=0x7fd4d0048000 comp rx=0 tx=0).stop 2024-05-09T08:50:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dce76640 1 -- 172.21.15.98:0/3442285761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d8102a70 msgr2=0x7fd4d8198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dce76640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d8102a70 0x7fd4d8198060 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000d950 tx=0x7fd4c000de10 comp rx=0 tx=0).stop 2024-05-09T08:50:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dce76640 1 -- 172.21.15.98:0/3442285761 shutdown_connections 2024-05-09T08:50:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dce76640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4d8103430 0x7fd4d819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dce76640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fd4c40a2b90 0x7fd4c40a5050 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dce76640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d8102a70 0x7fd4d8198060 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dce76640 1 --2- 172.21.15.98:0/3442285761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4d8101880 0x7fd4d8197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dce76640 1 -- 172.21.15.98:0/3442285761 >> 172.21.15.98:0/3442285761 conn(0x7fd4d80fce10 msgr2=0x7fd4d8105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dce76640 1 -- 172.21.15.98:0/3442285761 shutdown_connections 2024-05-09T08:50:30.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.618+0000 7fd4dce76640 1 -- 172.21.15.98:0/3442285761 wait complete. 2024-05-09T08:50:30.629 INFO:teuthology.run_tasks:Running task sequential... 2024-05-09T08:50:30.639 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-09T08:50:30.639 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-09T08:50:30.639 INFO:teuthology.task.print:Enabling mgr modules 2024-05-09T08:50:30.639 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-05-09T08:50:30.640 INFO:teuthology.task.exec:Executing custom commands... 2024-05-09T08:50:30.640 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-05-09T08:50:30.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-05-09T08:50:30.641 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:30 smithi105 ceph-mon[34498]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 6 B/s, 0 keys/s, 0 objects/s recovering 2024-05-09T08:50:30.641 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:30 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3770834449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:30.641 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:30 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1193691523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:30 smithi142 ceph-mon[34625]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 6 B/s, 0 keys/s, 0 objects/s recovering 2024-05-09T08:50:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:30 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3770834449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:30 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1193691523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.742+0000 7f2f53190640 1 -- 172.21.15.98:0/3235191576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f4c101980 msgr2=0x7f2f4c101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.742+0000 7f2f53190640 1 --2- 172.21.15.98:0/3235191576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f4c101980 0x7f2f4c101e00 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009900 tx=0x7f2f4002a410 comp rx=0 tx=0).stop 2024-05-09T08:50:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.742+0000 7f2f53190640 1 -- 172.21.15.98:0/3235191576 shutdown_connections 2024-05-09T08:50:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.742+0000 7f2f53190640 1 --2- 172.21.15.98:0/3235191576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f4c102340 0x7f2f4c108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.742+0000 7f2f53190640 1 --2- 172.21.15.98:0/3235191576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f4c101980 0x7f2f4c101e00 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.742+0000 7f2f53190640 1 --2- 172.21.15.98:0/3235191576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c100790 0x7f2f4c100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.742+0000 7f2f53190640 1 -- 172.21.15.98:0/3235191576 >> 172.21.15.98:0/3235191576 conn(0x7f2f4c0fbd20 msgr2=0x7f2f4c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.742+0000 7f2f53190640 1 -- 172.21.15.98:0/3235191576 shutdown_connections 2024-05-09T08:50:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.742+0000 7f2f53190640 1 -- 172.21.15.98:0/3235191576 wait complete. 2024-05-09T08:50:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.743+0000 7f2f53190640 1 Processor -- start 2024-05-09T08:50:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.743+0000 7f2f53190640 1 -- start start 2024-05-09T08:50:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.743+0000 7f2f53190640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f4c100790 0x7f2f4c197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.743+0000 7f2f53190640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f4c101980 0x7f2f4c198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.743+0000 7f2f53190640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c102340 0x7f2f4c19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.743+0000 7f2f53190640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4c10c410 con 0x7f2f4c101980 2024-05-09T08:50:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.743+0000 7f2f53190640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4c10c290 con 0x7f2f4c100790 2024-05-09T08:50:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.743+0000 7f2f53190640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4c10c590 con 0x7f2f4c102340 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.743+0000 7f2f4bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f4c101980 0x7f2f4c198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.743+0000 7f2f4bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f4c101980 0x7f2f4c198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50238/0 (socket says 172.21.15.98:50238) 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.743+0000 7f2f4bfff640 1 -- 172.21.15.98:0/2365012473 learned_addr learned my addr 172.21.15.98:0/2365012473 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.743+0000 7f2f50f2c640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f4c100790 0x7f2f4c197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.743+0000 7f2f5172d640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c102340 0x7f2f4c19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.744+0000 7f2f4bfff640 1 -- 172.21.15.98:0/2365012473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c102340 msgr2=0x7f2f4c19f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.744+0000 7f2f4bfff640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c102340 0x7f2f4c19f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.744+0000 7f2f4bfff640 1 -- 172.21.15.98:0/2365012473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f4c100790 msgr2=0x7f2f4c197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.744+0000 7f2f4bfff640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f4c100790 0x7f2f4c197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.744+0000 7f2f4bfff640 1 -- 172.21.15.98:0/2365012473 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4c19f730 con 0x7f2f4c101980 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.744+0000 7f2f5172d640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c102340 0x7f2f4c19f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.744+0000 7f2f4bfff640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f4c101980 0x7f2f4c198050 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009230 tx=0x7f2f40004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.744+0000 7f2f49ffb640 1 -- 172.21.15.98:0/2365012473 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4003f030 con 0x7f2f4c101980 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.744+0000 7f2f49ffb640 1 -- 172.21.15.98:0/2365012473 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f40003ce0 con 0x7f2f4c101980 2024-05-09T08:50:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.744+0000 7f2f53190640 1 -- 172.21.15.98:0/2365012473 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f4c19f9c0 con 0x7f2f4c101980 2024-05-09T08:50:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.744+0000 7f2f49ffb640 1 -- 172.21.15.98:0/2365012473 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40035070 con 0x7f2f4c101980 2024-05-09T08:50:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.744+0000 7f2f53190640 1 -- 172.21.15.98:0/2365012473 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f4c19feb0 con 0x7f2f4c101980 2024-05-09T08:50:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.745+0000 7f2f53190640 1 -- 172.21.15.98:0/2365012473 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f4c1a00e0 con 0x7f2f4c101980 2024-05-09T08:50:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.746+0000 7f2f49ffb640 1 -- 172.21.15.98:0/2365012473 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2f40042030 con 0x7f2f4c101980 2024-05-09T08:50:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.747+0000 7f2f49ffb640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2f340a2c30 0x7f2f340a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.747+0000 7f2f49ffb640 1 -- 172.21.15.98:0/2365012473 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7f2f400efe30 con 0x7f2f4c101980 2024-05-09T08:50:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.747+0000 7f2f50f2c640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2f340a2c30 0x7f2f340a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.747+0000 7f2f50f2c640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2f340a2c30 0x7f2f340a50f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c005fd0 tx=0x7f2f3c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.751+0000 7f2f49ffb640 1 -- 172.21.15.98:0/2365012473 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2f400bc9c0 con 0x7f2f4c101980 2024-05-09T08:50:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:30.975+0000 7f2f53190640 1 -- 172.21.15.98:0/2365012473 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f2f4c1a02b0 con 0x7f2f4c101980 2024-05-09T08:50:31.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.316+0000 7f2f49ffb640 1 -- 172.21.15.98:0/2365012473 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f2f400bc2c0 con 0x7f2f4c101980 2024-05-09T08:50:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.319+0000 7f2f53190640 1 -- 172.21.15.98:0/2365012473 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2f340a2c30 msgr2=0x7f2f340a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.319+0000 7f2f53190640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2f340a2c30 0x7f2f340a50f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c005fd0 tx=0x7f2f3c00b390 comp rx=0 tx=0).stop 2024-05-09T08:50:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.319+0000 7f2f53190640 1 -- 172.21.15.98:0/2365012473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f4c101980 msgr2=0x7f2f4c198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.319+0000 7f2f53190640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f4c101980 0x7f2f4c198050 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009230 tx=0x7f2f40004360 comp rx=0 tx=0).stop 2024-05-09T08:50:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.319+0000 7f2f53190640 1 -- 172.21.15.98:0/2365012473 shutdown_connections 2024-05-09T08:50:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.319+0000 7f2f53190640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c102340 0x7f2f4c19f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.319+0000 7f2f53190640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7f2f340a2c30 0x7f2f340a50f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.319+0000 7f2f53190640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f4c101980 0x7f2f4c198050 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.319+0000 7f2f53190640 1 --2- 172.21.15.98:0/2365012473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f4c100790 0x7f2f4c197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.319+0000 7f2f53190640 1 -- 172.21.15.98:0/2365012473 >> 172.21.15.98:0/2365012473 conn(0x7f2f4c0fbd20 msgr2=0x7f2f4c06a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:31.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.320+0000 7f2f53190640 1 -- 172.21.15.98:0/2365012473 shutdown_connections 2024-05-09T08:50:31.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.320+0000 7f2f53190640 1 -- 172.21.15.98:0/2365012473 wait complete. 2024-05-09T08:50:31.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-05-09T08:50:31.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.443+0000 7fda80ad4640 1 -- 172.21.15.98:0/130238476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda7c101980 msgr2=0x7fda7c101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:31.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.443+0000 7fda80ad4640 1 --2- 172.21.15.98:0/130238476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda7c101980 0x7fda7c101e00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fda64009900 tx=0x7fda6402a410 comp rx=0 tx=0).stop 2024-05-09T08:50:31.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.443+0000 7fda80ad4640 1 -- 172.21.15.98:0/130238476 shutdown_connections 2024-05-09T08:50:31.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.443+0000 7fda80ad4640 1 --2- 172.21.15.98:0/130238476 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda7c102340 0x7fda7c108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.443+0000 7fda80ad4640 1 --2- 172.21.15.98:0/130238476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda7c101980 0x7fda7c101e00 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.443+0000 7fda80ad4640 1 --2- 172.21.15.98:0/130238476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda7c100790 0x7fda7c100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.443+0000 7fda80ad4640 1 -- 172.21.15.98:0/130238476 >> 172.21.15.98:0/130238476 conn(0x7fda7c0fbd00 msgr2=0x7fda7c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:31.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.443+0000 7fda80ad4640 1 -- 172.21.15.98:0/130238476 shutdown_connections 2024-05-09T08:50:31.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.443+0000 7fda80ad4640 1 -- 172.21.15.98:0/130238476 wait complete. 2024-05-09T08:50:31.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.444+0000 7fda80ad4640 1 Processor -- start 2024-05-09T08:50:31.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.444+0000 7fda80ad4640 1 -- start start 2024-05-09T08:50:31.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.444+0000 7fda80ad4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda7c100790 0x7fda7c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:31.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.444+0000 7fda80ad4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda7c101980 0x7fda7c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:31.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.444+0000 7fda80ad4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda7c102340 0x7fda7c19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:31.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.444+0000 7fda80ad4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c10c440 con 0x7fda7c102340 2024-05-09T08:50:31.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.444+0000 7fda80ad4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c10c2c0 con 0x7fda7c101980 2024-05-09T08:50:31.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.444+0000 7fda80ad4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c10c5c0 con 0x7fda7c100790 2024-05-09T08:50:31.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.444+0000 7fda7ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda7c102340 0x7fda7c19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:31.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.444+0000 7fda71d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda7c101980 0x7fda7c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:31.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.444+0000 7fda71d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda7c101980 0x7fda7c198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:36998/0 (socket says 172.21.15.98:36998) 2024-05-09T08:50:31.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.444+0000 7fda7ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda7c102340 0x7fda7c19f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50272/0 (socket says 172.21.15.98:50272) 2024-05-09T08:50:31.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.444+0000 7fda71d9b640 1 -- 172.21.15.98:0/2285173601 learned_addr learned my addr 172.21.15.98:0/2285173601 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:31.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.444+0000 7fda7a59c640 1 --2- 172.21.15.98:0/2285173601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda7c100790 0x7fda7c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:31.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.445+0000 7fda7ad9d640 1 -- 172.21.15.98:0/2285173601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda7c100790 msgr2=0x7fda7c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:31.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.445+0000 7fda7ad9d640 1 --2- 172.21.15.98:0/2285173601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda7c100790 0x7fda7c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.445+0000 7fda7ad9d640 1 -- 172.21.15.98:0/2285173601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda7c101980 msgr2=0x7fda7c198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:31.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.445+0000 7fda7ad9d640 1 --2- 172.21.15.98:0/2285173601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda7c101980 0x7fda7c198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.445+0000 7fda7ad9d640 1 -- 172.21.15.98:0/2285173601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda7c19f740 con 0x7fda7c102340 2024-05-09T08:50:31.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.445+0000 7fda71d9b640 1 --2- 172.21.15.98:0/2285173601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda7c101980 0x7fda7c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:50:31.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.445+0000 7fda7ad9d640 1 --2- 172.21.15.98:0/2285173601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda7c102340 0x7fda7c19f0d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fda7400da10 tx=0x7fda7400ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:31.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.445+0000 7fda73fff640 1 -- 172.21.15.98:0/2285173601 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7400bd70 con 0x7fda7c102340 2024-05-09T08:50:31.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.445+0000 7fda80ad4640 1 -- 172.21.15.98:0/2285173601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda7c102c00 con 0x7fda7c102340 2024-05-09T08:50:31.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.445+0000 7fda73fff640 1 -- 172.21.15.98:0/2285173601 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda74003e50 con 0x7fda7c102340 2024-05-09T08:50:31.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.445+0000 7fda80ad4640 1 -- 172.21.15.98:0/2285173601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda7c103150 con 0x7fda7c102340 2024-05-09T08:50:31.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.445+0000 7fda73fff640 1 -- 172.21.15.98:0/2285173601 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74010660 con 0x7fda7c102340 2024-05-09T08:50:31.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.446+0000 7fda80ad4640 1 -- 172.21.15.98:0/2285173601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda7c103410 con 0x7fda7c102340 2024-05-09T08:50:31.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.447+0000 7fda73fff640 1 -- 172.21.15.98:0/2285173601 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fda74003970 con 0x7fda7c102340 2024-05-09T08:50:31.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.448+0000 7fda73fff640 1 --2- 172.21.15.98:0/2285173601 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fda680a2c80 0x7fda680a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:31.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.448+0000 7fda7a59c640 1 -- 172.21.15.98:0/2285173601 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fda680a2c80 msgr2=0x7fda680a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1228297308 2024-05-09T08:50:31.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.448+0000 7fda73fff640 1 -- 172.21.15.98:0/2285173601 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fda740cf5e0 con 0x7fda7c102340 2024-05-09T08:50:31.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.448+0000 7fda7a59c640 1 --2- 172.21.15.98:0/2285173601 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fda680a2c80 0x7fda680a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:50:31.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.452+0000 7fda73fff640 1 -- 172.21.15.98:0/2285173601 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fda7409c170 con 0x7fda7c102340 2024-05-09T08:50:31.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:31 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2365012473' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-09T08:50:31.588 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:31 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: ignoring --setuser ceph since I am not root 2024-05-09T08:50:31.588 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:31 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: ignoring --setgroup ceph since I am not root 2024-05-09T08:50:31.588 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:31 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:31.502+0000 7f8d1d15a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:50:31.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.634+0000 7fda80ad4640 1 -- 172.21.15.98:0/2285173601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fda7c103bf0 con 0x7fda7c102340 2024-05-09T08:50:31.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.638+0000 7fda73fff640 1 -- 172.21.15.98:0/2285173601 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v26)=0 v26) ==== 138+0+0 (secure 0 0 0) 0x7fda7409ba70 con 0x7fda7c102340 2024-05-09T08:50:31.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.640+0000 7fda80ad4640 1 -- 172.21.15.98:0/2285173601 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fda680a2c80 msgr2=0x7fda680a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:50:31.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.640+0000 7fda80ad4640 1 --2- 172.21.15.98:0/2285173601 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fda680a2c80 0x7fda680a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.640+0000 7fda80ad4640 1 -- 172.21.15.98:0/2285173601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda7c102340 msgr2=0x7fda7c19f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:31.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.640+0000 7fda80ad4640 1 --2- 172.21.15.98:0/2285173601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda7c102340 0x7fda7c19f0d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fda7400da10 tx=0x7fda7400ded0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.640+0000 7fda80ad4640 1 -- 172.21.15.98:0/2285173601 shutdown_connections 2024-05-09T08:50:31.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.640+0000 7fda80ad4640 1 --2- 172.21.15.98:0/2285173601 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fda680a2c80 0x7fda680a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.640+0000 7fda80ad4640 1 --2- 172.21.15.98:0/2285173601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda7c102340 0x7fda7c19f0d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.640+0000 7fda80ad4640 1 --2- 172.21.15.98:0/2285173601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda7c101980 0x7fda7c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.640+0000 7fda80ad4640 1 --2- 172.21.15.98:0/2285173601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda7c100790 0x7fda7c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.640+0000 7fda80ad4640 1 -- 172.21.15.98:0/2285173601 >> 172.21.15.98:0/2285173601 conn(0x7fda7c0fbd00 msgr2=0x7fda7c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:31.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.640+0000 7fda80ad4640 1 -- 172.21.15.98:0/2285173601 shutdown_connections 2024-05-09T08:50:31.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.640+0000 7fda80ad4640 1 -- 172.21.15.98:0/2285173601 wait complete. 2024-05-09T08:50:31.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-05-09T08:50:31.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:31 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2365012473' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-09T08:50:31.658 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: ignoring --setuser ceph since I am not root 2024-05-09T08:50:31.658 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: ignoring --setgroup ceph since I am not root 2024-05-09T08:50:31.658 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:31 smithi142 ceph-mgr[36481]: -- 172.21.15.142:0/2636036696 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5584578525a0 con 0x558457826c00 2024-05-09T08:50:31.658 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:31.493+0000 7ff96f0cb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:50:31.679 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:31 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2365012473' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-09T08:50:31.679 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: ignoring --setuser ceph since I am not root 2024-05-09T08:50:31.679 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: ignoring --setgroup ceph since I am not root 2024-05-09T08:50:31.679 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:31 smithi105 ceph-mgr[35883]: -- 172.21.15.105:0/3030707103 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56360ac385a0 con 0x56360ac0d000 2024-05-09T08:50:31.680 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:31.496+0000 7fcc91034200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:50:31.680 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:31.679+0000 7fcc91034200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:50:31.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.755+0000 7fec51600640 1 -- 172.21.15.98:0/815406691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec4c101700 msgr2=0x7fec4c101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:31.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.756+0000 7fec51600640 1 --2- 172.21.15.98:0/815406691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec4c101700 0x7fec4c101b20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fec3c009900 tx=0x7fec3c02a410 comp rx=0 tx=0).stop 2024-05-09T08:50:31.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.756+0000 7fec51600640 1 -- 172.21.15.98:0/815406691 shutdown_connections 2024-05-09T08:50:31.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.756+0000 7fec51600640 1 --2- 172.21.15.98:0/815406691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec4c1032b0 0x7fec4c109b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.756+0000 7fec51600640 1 --2- 172.21.15.98:0/815406691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec4c1028f0 0x7fec4c102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.756+0000 7fec51600640 1 --2- 172.21.15.98:0/815406691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec4c101700 0x7fec4c101b20 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.756+0000 7fec51600640 1 -- 172.21.15.98:0/815406691 >> 172.21.15.98:0/815406691 conn(0x7fec4c0fcc90 msgr2=0x7fec4c0ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:31.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.756+0000 7fec51600640 1 -- 172.21.15.98:0/815406691 shutdown_connections 2024-05-09T08:50:31.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.756+0000 7fec51600640 1 -- 172.21.15.98:0/815406691 wait complete. 2024-05-09T08:50:31.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.756+0000 7fec51600640 1 Processor -- start 2024-05-09T08:50:31.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.756+0000 7fec51600640 1 -- start start 2024-05-09T08:50:31.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec51600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec4c101700 0x7fec4c0755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:31.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec51600640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec4c1028f0 0x7fec4c073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:31.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec51600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec4c1032b0 0x7fec4c074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:31.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec51600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c10d350 con 0x7fec4c1032b0 2024-05-09T08:50:31.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec51600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c10d1d0 con 0x7fec4c101700 2024-05-09T08:50:31.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec51600640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c10d4d0 con 0x7fec4c1028f0 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec4b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec4c1032b0 0x7fec4c074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec4b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec4c1032b0 0x7fec4c074160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50286/0 (socket says 172.21.15.98:50286) 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec4b7fe640 1 -- 172.21.15.98:0/4116699695 learned_addr learned my addr 172.21.15.98:0/4116699695 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec4affd640 1 --2- 172.21.15.98:0/4116699695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec4c101700 0x7fec4c0755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec43fff640 1 --2- 172.21.15.98:0/4116699695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec4c1028f0 0x7fec4c073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec4b7fe640 1 -- 172.21.15.98:0/4116699695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec4c1028f0 msgr2=0x7fec4c073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec4b7fe640 1 --2- 172.21.15.98:0/4116699695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec4c1028f0 0x7fec4c073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec4b7fe640 1 -- 172.21.15.98:0/4116699695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec4c101700 msgr2=0x7fec4c0755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec4b7fe640 1 --2- 172.21.15.98:0/4116699695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec4c101700 0x7fec4c0755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec4b7fe640 1 -- 172.21.15.98:0/4116699695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec4c0749f0 con 0x7fec4c1032b0 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec4affd640 1 --2- 172.21.15.98:0/4116699695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec4c101700 0x7fec4c0755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec4b7fe640 1 --2- 172.21.15.98:0/4116699695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec4c1032b0 0x7fec4c074160 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fec44009fd0 tx=0x7fec4400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec48ff9640 1 -- 172.21.15.98:0/4116699695 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec44017070 con 0x7fec4c1032b0 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec48ff9640 1 -- 172.21.15.98:0/4116699695 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec44003e50 con 0x7fec4c1032b0 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec51600640 1 -- 172.21.15.98:0/4116699695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec4c1a1b70 con 0x7fec4c1032b0 2024-05-09T08:50:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec48ff9640 1 -- 172.21.15.98:0/4116699695 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec440106c0 con 0x7fec4c1032b0 2024-05-09T08:50:31.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.757+0000 7fec51600640 1 -- 172.21.15.98:0/4116699695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec4c1a2070 con 0x7fec4c1032b0 2024-05-09T08:50:31.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.758+0000 7fec51600640 1 -- 172.21.15.98:0/4116699695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec4c04fe70 con 0x7fec4c1032b0 2024-05-09T08:50:31.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.760+0000 7fec48ff9640 1 -- 172.21.15.98:0/4116699695 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fec44003970 con 0x7fec4c1032b0 2024-05-09T08:50:31.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.760+0000 7fec48ff9640 1 --2- 172.21.15.98:0/4116699695 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fec380a2bd0 0x7fec380a5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:31.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.761+0000 7fec4affd640 1 -- 172.21.15.98:0/4116699695 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fec380a2bd0 msgr2=0x7fec380a5090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1228297308 2024-05-09T08:50:31.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.761+0000 7fec48ff9640 1 -- 172.21.15.98:0/4116699695 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fec440cedb0 con 0x7fec4c1032b0 2024-05-09T08:50:31.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.761+0000 7fec4affd640 1 --2- 172.21.15.98:0/4116699695 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fec380a2bd0 0x7fec380a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:50:31.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.765+0000 7fec48ff9640 1 -- 172.21.15.98:0/4116699695 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fec4c04fe70 con 0x7fec4c1032b0 2024-05-09T08:50:31.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.947+0000 7fec51600640 1 -- 172.21.15.98:0/4116699695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fec4c0624a0 con 0x7fec4c1032b0 2024-05-09T08:50:31.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.952+0000 7fec48ff9640 1 -- 172.21.15.98:0/4116699695 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v27)=0 v27) ==== 133+0+0 (secure 0 0 0) 0x7fec4409b2f0 con 0x7fec4c1032b0 2024-05-09T08:50:31.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.954+0000 7fec51600640 1 -- 172.21.15.98:0/4116699695 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fec380a2bd0 msgr2=0x7fec380a5090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:50:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.954+0000 7fec51600640 1 --2- 172.21.15.98:0/4116699695 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fec380a2bd0 0x7fec380a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.954+0000 7fec51600640 1 -- 172.21.15.98:0/4116699695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec4c1032b0 msgr2=0x7fec4c074160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.954+0000 7fec51600640 1 --2- 172.21.15.98:0/4116699695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec4c1032b0 0x7fec4c074160 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fec44009fd0 tx=0x7fec4400b260 comp rx=0 tx=0).stop 2024-05-09T08:50:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.954+0000 7fec51600640 1 -- 172.21.15.98:0/4116699695 shutdown_connections 2024-05-09T08:50:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.954+0000 7fec51600640 1 --2- 172.21.15.98:0/4116699695 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fec380a2bd0 0x7fec380a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.954+0000 7fec51600640 1 --2- 172.21.15.98:0/4116699695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec4c1032b0 0x7fec4c074160 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.954+0000 7fec51600640 1 --2- 172.21.15.98:0/4116699695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec4c1028f0 0x7fec4c073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.954+0000 7fec51600640 1 --2- 172.21.15.98:0/4116699695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec4c101700 0x7fec4c0755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.954+0000 7fec51600640 1 -- 172.21.15.98:0/4116699695 >> 172.21.15.98:0/4116699695 conn(0x7fec4c0fcc90 msgr2=0x7fec4c105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.954+0000 7fec51600640 1 -- 172.21.15.98:0/4116699695 shutdown_connections 2024-05-09T08:50:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:31.954+0000 7fec51600640 1 -- 172.21.15.98:0/4116699695 wait complete. 2024-05-09T08:50:31.964 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:31 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:31.694+0000 7f8d1d15a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:50:31.964 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:31 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:31.803+0000 7f8d1d15a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:50:31.965 INFO:teuthology.run_tasks:Running task kclient... 2024-05-09T08:50:31.978 INFO:tasks.kclient:Mounting kernel clients... 2024-05-09T08:50:31.978 INFO:tasks.kclient:config is {'client.0': {}} 2024-05-09T08:50:31.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:50:31.978 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-05-09T08:50:31.986 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:31.781+0000 7fcc91034200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:50:31.994 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-05-09T08:50:31.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:50:31.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:50:32.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:50:32.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-05-09T08:50:32.081 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:31.674+0000 7ff96f0cb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:50:32.082 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:31.777+0000 7ff96f0cb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:50:32.082 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:32 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:32.082+0000 7ff96f0cb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:50:32.183 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-05-09T08:50:32.183 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T08:50:32.183 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T08:50:32.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph fs ls 2024-05-09T08:50:32.223 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:32 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:32.131+0000 7f8d1d15a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:50:32.408 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:32 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:32.167+0000 7ff96f0cb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:50:32.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:32 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2365012473' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-09T08:50:32.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:32 smithi142 ceph-mon[34625]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-05-09T08:50:32.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:32 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2285173601' entity='client.admin' 2024-05-09T08:50:32.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:32 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/4116699695' entity='client.admin' 2024-05-09T08:50:32.437 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:50:32.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:32 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2365012473' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-09T08:50:32.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:32 smithi105 ceph-mon[34498]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-05-09T08:50:32.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:32 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2285173601' entity='client.admin' 2024-05-09T08:50:32.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:32 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/4116699695' entity='client.admin' 2024-05-09T08:50:32.487 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:32 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:32.089+0000 7fcc91034200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:50:32.487 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:32 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:32.175+0000 7fcc91034200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:50:32.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:32 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2365012473' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-09T08:50:32.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:32 smithi098 ceph-mon[30092]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-05-09T08:50:32.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:32 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2285173601' entity='client.admin' 2024-05-09T08:50:32.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:32 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4116699695' entity='client.admin' 2024-05-09T08:50:32.561 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:32 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:32.223+0000 7f8d1d15a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:50:33.658 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:33 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:33.334+0000 7ff96f0cb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:50:33.658 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:33 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:33.502+0000 7ff96f0cb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:50:33.683 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:33.344+0000 7fcc91034200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:50:33.683 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:33.513+0000 7fcc91034200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:50:33.683 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:33.683+0000 7fcc91034200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:50:33.769 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:33 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:33.455+0000 7f8d1d15a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:50:33.770 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:33 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:33.630+0000 7f8d1d15a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:50:33.960 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:33 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:33.673+0000 7ff96f0cb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:50:34.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:33 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:33.805+0000 7f8d1d15a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:50:34.236 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:33.974+0000 7fcc91034200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:50:34.236 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:34 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:34.067+0000 7fcc91034200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:50:34.251 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:33 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:33.960+0000 7ff96f0cb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:50:34.251 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:34 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:34.051+0000 7ff96f0cb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:50:34.338 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:34 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:34.114+0000 7f8d1d15a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:50:34.338 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:34 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:34.210+0000 7f8d1d15a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:50:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.341+0000 7fee1dd1e700 1 -- 172.21.15.98:0/1309162356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee181076c0 msgr2=0x7fee1810f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.341+0000 7fee1dd1e700 1 --2- 172.21.15.98:0/1309162356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee181076c0 0x7fee1810f290 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fee1000b0a0 tx=0x7fee1000b3b0 comp rx=0 tx=0).stop 2024-05-09T08:50:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.341+0000 7fee1dd1e700 1 -- 172.21.15.98:0/1309162356 shutdown_connections 2024-05-09T08:50:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.341+0000 7fee1dd1e700 1 --2- 172.21.15.98:0/1309162356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee18102600 0x7fee181029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.341+0000 7fee1dd1e700 1 --2- 172.21.15.98:0/1309162356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee181076c0 0x7fee1810f290 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:34.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.341+0000 7fee1dd1e700 1 --2- 172.21.15.98:0/1309162356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee18102fb0 0x7fee18106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:34.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.341+0000 7fee1dd1e700 1 -- 172.21.15.98:0/1309162356 >> 172.21.15.98:0/1309162356 conn(0x7fee180794d0 msgr2=0x7fee180798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:34.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.341+0000 7fee1dd1e700 1 -- 172.21.15.98:0/1309162356 shutdown_connections 2024-05-09T08:50:34.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.341+0000 7fee1dd1e700 1 -- 172.21.15.98:0/1309162356 wait complete. 2024-05-09T08:50:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.342+0000 7fee1dd1e700 1 Processor -- start 2024-05-09T08:50:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.342+0000 7fee1dd1e700 1 -- start start 2024-05-09T08:50:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.342+0000 7fee1dd1e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee18102600 0x7fee1819e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.342+0000 7fee1dd1e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee18102fb0 0x7fee1819ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.342+0000 7fee1dd1e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee181076c0 0x7fee181988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.342+0000 7fee1dd1e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fee181126f0 con 0x7fee18102600 2024-05-09T08:50:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.342+0000 7fee1dd1e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fee18112570 con 0x7fee181076c0 2024-05-09T08:50:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.342+0000 7fee1dd1e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fee18112870 con 0x7fee18102fb0 2024-05-09T08:50:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.342+0000 7fee177fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee18102600 0x7fee1819e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.342+0000 7fee177fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee18102600 0x7fee1819e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50312/0 (socket says 172.21.15.98:50312) 2024-05-09T08:50:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.342+0000 7fee16ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee18102fb0 0x7fee1819ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.342+0000 7fee177fe700 1 -- 172.21.15.98:0/499320617 learned_addr learned my addr 172.21.15.98:0/499320617 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.342+0000 7fee17fff700 1 --2- 172.21.15.98:0/499320617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee181076c0 0x7fee181988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.342+0000 7fee16ffd700 1 -- 172.21.15.98:0/499320617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee181076c0 msgr2=0x7fee181988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.343+0000 7fee16ffd700 1 --2- 172.21.15.98:0/499320617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee181076c0 0x7fee181988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.343+0000 7fee16ffd700 1 -- 172.21.15.98:0/499320617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee18102600 msgr2=0x7fee1819e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.343+0000 7fee16ffd700 1 --2- 172.21.15.98:0/499320617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee18102600 0x7fee1819e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.343+0000 7fee16ffd700 1 -- 172.21.15.98:0/499320617 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee18199050 con 0x7fee18102fb0 2024-05-09T08:50:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.343+0000 7fee177fe700 1 --2- 172.21.15.98:0/499320617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee18102600 0x7fee1819e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:50:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.343+0000 7fee16ffd700 1 --2- 172.21.15.98:0/499320617 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee18102fb0 0x7fee1819ec70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00e9c0 tx=0x7fee0c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.343+0000 7fee14ff9700 1 -- 172.21.15.98:0/499320617 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0c004220 con 0x7fee18102fb0 2024-05-09T08:50:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.343+0000 7fee1dd1e700 1 -- 172.21.15.98:0/499320617 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee18199340 con 0x7fee18102fb0 2024-05-09T08:50:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.343+0000 7fee14ff9700 1 -- 172.21.15.98:0/499320617 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee0c004d10 con 0x7fee18102fb0 2024-05-09T08:50:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.343+0000 7fee14ff9700 1 -- 172.21.15.98:0/499320617 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0c005020 con 0x7fee18102fb0 2024-05-09T08:50:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.343+0000 7fee1dd1e700 1 -- 172.21.15.98:0/499320617 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee181a5600 con 0x7fee18102fb0 2024-05-09T08:50:34.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.344+0000 7fee1dd1e700 1 -- 172.21.15.98:0/499320617 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee18069010 con 0x7fee18102fb0 2024-05-09T08:50:34.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.345+0000 7fee14ff9700 1 -- 172.21.15.98:0/499320617 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fee0c004830 con 0x7fee18102fb0 2024-05-09T08:50:34.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.346+0000 7fee14ff9700 1 --2- 172.21.15.98:0/499320617 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fee000a2cf0 0x7fee000a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:34.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.346+0000 7fee177fe700 1 -- 172.21.15.98:0/499320617 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fee000a2cf0 msgr2=0x7fee000a51b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1228297308 2024-05-09T08:50:34.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.346+0000 7fee14ff9700 1 -- 172.21.15.98:0/499320617 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fee0c0cf260 con 0x7fee18102fb0 2024-05-09T08:50:34.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.346+0000 7fee177fe700 1 --2- 172.21.15.98:0/499320617 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fee000a2cf0 0x7fee000a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:50:34.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.350+0000 7fee14ff9700 1 -- 172.21.15.98:0/499320617 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fee0c09bdb0 con 0x7fee18102fb0 2024-05-09T08:50:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.547+0000 7fee177fe700 1 -- 172.21.15.98:0/499320617 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fee000a2cf0 msgr2=0x7fee000a51b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1228297308 2024-05-09T08:50:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.547+0000 7fee177fe700 1 --2- 172.21.15.98:0/499320617 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fee000a2cf0 0x7fee000a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:50:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.599+0000 7fee1dd1e700 1 -- 172.21.15.98:0/499320617 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fee18199e20 con 0x7fee18102fb0 2024-05-09T08:50:34.599 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:34 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:34.426+0000 7f8d1d15a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:50:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.600+0000 7fee14ff9700 1 -- 172.21.15.98:0/499320617 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7fee0c09b6b0 con 0x7fee18102fb0 2024-05-09T08:50:34.600 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-09T08:50:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.602+0000 7fee1dd1e700 1 -- 172.21.15.98:0/499320617 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fee000a2cf0 msgr2=0x7fee000a51b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:50:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.602+0000 7fee1dd1e700 1 --2- 172.21.15.98:0/499320617 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fee000a2cf0 0x7fee000a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.602+0000 7fee1dd1e700 1 -- 172.21.15.98:0/499320617 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee18102fb0 msgr2=0x7fee1819ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.602+0000 7fee1dd1e700 1 --2- 172.21.15.98:0/499320617 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee18102fb0 0x7fee1819ec70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00e9c0 tx=0x7fee0c00ecd0 comp rx=0 tx=0).stop 2024-05-09T08:50:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.602+0000 7fee1dd1e700 1 -- 172.21.15.98:0/499320617 shutdown_connections 2024-05-09T08:50:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.602+0000 7fee1dd1e700 1 --2- 172.21.15.98:0/499320617 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fee000a2cf0 0x7fee000a51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.602+0000 7fee1dd1e700 1 --2- 172.21.15.98:0/499320617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee181076c0 0x7fee181988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.602+0000 7fee1dd1e700 1 --2- 172.21.15.98:0/499320617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee18102600 0x7fee1819e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.602+0000 7fee1dd1e700 1 --2- 172.21.15.98:0/499320617 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee18102fb0 0x7fee1819ec70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.602+0000 7fee1dd1e700 1 -- 172.21.15.98:0/499320617 >> 172.21.15.98:0/499320617 conn(0x7fee180794d0 msgr2=0x7fee181003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.602+0000 7fee1dd1e700 1 -- 172.21.15.98:0/499320617 shutdown_connections 2024-05-09T08:50:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:34.602+0000 7fee1dd1e700 1 -- 172.21.15.98:0/499320617 wait complete. 2024-05-09T08:50:34.651 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:34 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:34.269+0000 7fcc91034200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:50:34.651 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:34 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:34.465+0000 7fcc91034200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:50:34.651 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:34 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:34.252+0000 7ff96f0cb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:50:34.652 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:34 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:34.448+0000 7ff96f0cb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:50:34.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:34 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/499320617' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:50:34.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:34 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/499320617' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:50:34.972 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:34 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:34.630+0000 7f8d1d15a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:50:34.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:34 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/499320617' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:50:35.326 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T08:50:35.327 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T08:50:35.327 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-05-09T08:50:35.327 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T08:50:35.327 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T08:50:35.327 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T08:50:35.327 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T08:50:35.327 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T08:50:35.327 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T08:50:35.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:50:35.327 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-05-09T08:50:35.383 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T08:50:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-05-09T08:50:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-09T08:50:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42117sec preferred_lft 42117sec 2024-05-09T08:50:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-05-09T08:50:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:50:35.385 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T08:50:35.385 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:50:35.385 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-09T08:50:35.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2024-05-09T08:50:35.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2024-05-09T08:50:35.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2024-05-09T08:50:35.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T08:50:35.386 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-09T08:50:35.410 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:35 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:35.046+0000 7ff96f0cb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:50:35.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:50:35.482 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:35 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:35.065+0000 7fcc91034200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:50:35.507 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:35 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:35.249+0000 7f8d1d15a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:50:35.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:50:35.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:50:35.529 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T08:50:35.553 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-05-09T08:50:35.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:50:35.555 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-05-09T08:50:35.613 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-05-09T08:50:35.613 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T08:50:35.614 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T08:50:35.614 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T08:50:35.614 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T08:50:35.614 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:50:35.614 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-09T08:50:35.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T08:50:35.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T08:50:35.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T08:50:35.615 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-09T08:50:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:50:35.736 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:35 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:35.483+0000 7fcc91034200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:50:35.736 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:35 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:35.579+0000 7fcc91034200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:50:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:50:35.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:50:35.744 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-05-09T08:50:35.757 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:35 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:35.464+0000 7ff96f0cb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:50:35.758 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:35 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:35.559+0000 7ff96f0cb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:50:35.758 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:35 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:35.758+0000 7ff96f0cb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:50:35.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:50:35.760 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-05-09T08:50:35.812 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:35 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:35.707+0000 7f8d1d15a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:50:35.812 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:35 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:35.812+0000 7f8d1d15a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:50:35.819 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:50:35.819 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-09T08:50:35.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T08:50:35.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-09T08:50:35.820 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-09T08:50:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:50:35.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:50:35.938 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-09T08:50:35.938 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:50:35.938 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-09T08:50:35.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-09T08:50:35.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T08:50:35.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-09T08:50:35.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-09T08:50:35.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-09T08:50:35.939 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-09T08:50:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:50:36.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:50:36.055 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:50:36.056 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-09T08:50:36.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-05-09T08:50:36.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T08:50:36.056 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-09T08:50:36.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:36 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:36.038+0000 7f8d1d15a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:50:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:50:36.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:50:36.135 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T08:50:36.135 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T08:50:36.191 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T08:50:36.192 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T08:50:36.192 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T08:50:36.236 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:35 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:35.780+0000 7fcc91034200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:50:36.248 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-05-09T08:50:36.248 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-05-09T08:50:36.248 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/04326e37-0f86-4776-bf48-99735d3687da 2024-05-09T08:50:36.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:50:36.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/04326e37-0f86-4776-bf48-99735d3687da /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-09T08:50:36.332 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-09T08:50:36.333 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync". 2024-05-09T08:50:36.333 INFO:teuthology.orchestra.run.smithi098.stdout:invalid new device string format 2024-05-09T08:50:36.456 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-09T08:50:36.457 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync". 2024-05-09T08:50:36.457 INFO:teuthology.orchestra.run.smithi098.stdout:invalid new device string format 2024-05-09T08:50:36.457 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:6789,172.21.15.105:6789,172.21.15.142:6789" 2024-05-09T08:50:36.457 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.98:6789,172.21.15.105:6789,172.21.15.142:6789:/volumes/qa/sv_1/04326e37-0f86-4776-bf48-99735d3687da 2024-05-09T08:50:36.457 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync,key=0,fsid=a57d9712-0ddf-11ef-bc98-c7b262605968" will pass to kernel 2024-05-09T08:50:36.457 INFO:teuthology.orchestra.run.smithi098.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-09T08:50:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: You just mounted a file system that supports labels which does not 2024-05-09T08:50:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T08:50:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T08:50:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T08:50:36.459 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T08:50:36.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:50:36.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T08:50:36.489 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T08:50:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T08:50:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:50:36.588 INFO:teuthology.orchestra.run.smithi098.stdout:client25015 2024-05-09T08:50:36.589 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:50:36.589 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T08:50:36.605 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T08:50:36.609 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:36.275+0000 7ff96f0cb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:50:36.609 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:36.368+0000 7ff96f0cb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:50:36.630 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:36 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:36.296+0000 7fcc91034200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:50:36.630 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:36 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:36.387+0000 7fcc91034200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:50:36.694 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:36 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:36.594+0000 7f8d1d15a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:50:36.694 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:36 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:36.695+0000 7f8d1d15a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:50:36.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T08:50:36.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:50:36.702 INFO:teuthology.orchestra.run.smithi098.stdout:client25015 2024-05-09T08:50:36.703 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T08:50:36.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T08:50:36.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:50:36.762 INFO:teuthology.orchestra.run.smithi098.stdout:client25015 2024-05-09T08:50:36.762 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968.client25015/status') 2024-05-09T08:50:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:50:36.822 INFO:teuthology.orchestra.run.smithi098.stdout:instance: client.25015 (3)172.21.15.98:0/2251904167 2024-05-09T08:50:36.822 INFO:teuthology.orchestra.run.smithi098.stdout:blocklisted: false 2024-05-09T08:50:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records in 2024-05-09T08:50:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records out 2024-05-09T08:50:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:71 bytes copied, 2.2087e-05 s, 3.2 MB/s 2024-05-09T08:50:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:50:36.830 INFO:teuthology.run_tasks:Running task exec... 2024-05-09T08:50:36.842 INFO:teuthology.task.exec:Executing custom commands... 2024-05-09T08:50:36.843 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-05-09T08:50:36.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-05-09T08:50:36.908 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:36.609+0000 7ff96f0cb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:50:36.908 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:36.742+0000 7ff96f0cb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:50:36.908 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:36.838+0000 7ff96f0cb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:50:36.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.949+0000 7fad54d7e640 1 -- 172.21.15.98:0/1182363752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad50102340 msgr2=0x7fad50108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.949+0000 7fad54d7e640 1 --2- 172.21.15.98:0/1182363752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad50102340 0x7fad50108bd0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fad48009ca0 tx=0x7fad4802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.949+0000 7fad54d7e640 1 -- 172.21.15.98:0/1182363752 shutdown_connections 2024-05-09T08:50:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.949+0000 7fad54d7e640 1 --2- 172.21.15.98:0/1182363752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad50102340 0x7fad50108bd0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.949+0000 7fad54d7e640 1 --2- 172.21.15.98:0/1182363752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad50101980 0x7fad50101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.949+0000 7fad54d7e640 1 --2- 172.21.15.98:0/1182363752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad50100790 0x7fad50100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.949+0000 7fad54d7e640 1 -- 172.21.15.98:0/1182363752 >> 172.21.15.98:0/1182363752 conn(0x7fad500fbd00 msgr2=0x7fad500fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.949+0000 7fad54d7e640 1 -- 172.21.15.98:0/1182363752 shutdown_connections 2024-05-09T08:50:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.949+0000 7fad54d7e640 1 -- 172.21.15.98:0/1182363752 wait complete. 2024-05-09T08:50:36.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.950+0000 7fad54d7e640 1 Processor -- start 2024-05-09T08:50:36.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.950+0000 7fad54d7e640 1 -- start start 2024-05-09T08:50:36.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.950+0000 7fad54d7e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad50100790 0x7fad501969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:36.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.950+0000 7fad54d7e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad50101980 0x7fad50196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:36.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.950+0000 7fad54d7e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad50102340 0x7fad5019df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:36.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.950+0000 7fad54d7e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad5010c410 con 0x7fad50100790 2024-05-09T08:50:36.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.950+0000 7fad54d7e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fad5010c290 con 0x7fad50101980 2024-05-09T08:50:36.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.950+0000 7fad54d7e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fad5010c590 con 0x7fad50102340 2024-05-09T08:50:36.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.950+0000 7fad4e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad50100790 0x7fad501969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:36.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.950+0000 7fad4dd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad50101980 0x7fad50196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:36.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.950+0000 7fad4dd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad50101980 0x7fad50196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:56650/0 (socket says 172.21.15.98:56650) 2024-05-09T08:50:36.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.950+0000 7fad4e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad50100790 0x7fad501969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42776/0 (socket says 172.21.15.98:42776) 2024-05-09T08:50:36.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.950+0000 7fad4dd9b640 1 -- 172.21.15.98:0/3108715656 learned_addr learned my addr 172.21.15.98:0/3108715656 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:36.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.950+0000 7fad4ed9d640 1 --2- 172.21.15.98:0/3108715656 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad50102340 0x7fad5019df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:36.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.951+0000 7fad4dd9b640 1 -- 172.21.15.98:0/3108715656 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad50102340 msgr2=0x7fad5019df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:36.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.951+0000 7fad4dd9b640 1 --2- 172.21.15.98:0/3108715656 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad50102340 0x7fad5019df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:36.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.951+0000 7fad4dd9b640 1 -- 172.21.15.98:0/3108715656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad50100790 msgr2=0x7fad501969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:36.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.951+0000 7fad4dd9b640 1 --2- 172.21.15.98:0/3108715656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad50100790 0x7fad501969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:36.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.951+0000 7fad4dd9b640 1 -- 172.21.15.98:0/3108715656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5019e5e0 con 0x7fad50101980 2024-05-09T08:50:36.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.951+0000 7fad4dd9b640 1 --2- 172.21.15.98:0/3108715656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad50101980 0x7fad50196f00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fad4400e9f0 tx=0x7fad4400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:36.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.951+0000 7fad3f7fe640 1 -- 172.21.15.98:0/3108715656 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4400cd80 con 0x7fad50101980 2024-05-09T08:50:36.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.951+0000 7fad54d7e640 1 -- 172.21.15.98:0/3108715656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad5019e8d0 con 0x7fad50101980 2024-05-09T08:50:36.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.951+0000 7fad3f7fe640 1 -- 172.21.15.98:0/3108715656 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad44005500 con 0x7fad50101980 2024-05-09T08:50:36.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.951+0000 7fad3f7fe640 1 -- 172.21.15.98:0/3108715656 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44010690 con 0x7fad50101980 2024-05-09T08:50:36.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.951+0000 7fad54d7e640 1 -- 172.21.15.98:0/3108715656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad5019edd0 con 0x7fad50101980 2024-05-09T08:50:36.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.952+0000 7fad54d7e640 1 -- 172.21.15.98:0/3108715656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad50107a50 con 0x7fad50101980 2024-05-09T08:50:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.953+0000 7fad3f7fe640 1 -- 172.21.15.98:0/3108715656 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fad440046c0 con 0x7fad50101980 2024-05-09T08:50:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.954+0000 7fad3f7fe640 1 --2- 172.21.15.98:0/3108715656 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fad380a2c80 0x7fad380a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.954+0000 7fad4e59c640 1 -- 172.21.15.98:0/3108715656 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fad380a2c80 msgr2=0x7fad380a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1228297308 2024-05-09T08:50:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.954+0000 7fad3f7fe640 1 -- 172.21.15.98:0/3108715656 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fad440cf880 con 0x7fad50101980 2024-05-09T08:50:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.954+0000 7fad4e59c640 1 --2- 172.21.15.98:0/3108715656 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fad380a2c80 0x7fad380a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:50:36.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:36.957+0000 7fad3f7fe640 1 -- 172.21.15.98:0/3108715656 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fad4409c410 con 0x7fad50101980 2024-05-09T08:50:36.976 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:36 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:36.630+0000 7fcc91034200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:50:36.976 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:36 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:36.763+0000 7fcc91034200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:50:36.976 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:36 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:36.859+0000 7fcc91034200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:50:36.976 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:36 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:36.976+0000 7fcc91034200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:50:37.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.155+0000 7fad4e59c640 1 -- 172.21.15.98:0/3108715656 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fad380a2c80 msgr2=0x7fad380a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1228297308 2024-05-09T08:50:37.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.155+0000 7fad4e59c640 1 --2- 172.21.15.98:0/3108715656 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fad380a2c80 0x7fad380a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:50:37.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.183+0000 7fad54d7e640 1 -- 172.21.15.98:0/3108715656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fad5002d340 con 0x7fad50101980 2024-05-09T08:50:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.238+0000 7fad3f7fe640 1 -- 172.21.15.98:0/3108715656 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7fad4409bd10 con 0x7fad50101980 2024-05-09T08:50:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.240+0000 7fad54d7e640 1 -- 172.21.15.98:0/3108715656 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fad380a2c80 msgr2=0x7fad380a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:50:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.240+0000 7fad54d7e640 1 --2- 172.21.15.98:0/3108715656 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fad380a2c80 0x7fad380a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.240+0000 7fad54d7e640 1 -- 172.21.15.98:0/3108715656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad50101980 msgr2=0x7fad50196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.240+0000 7fad54d7e640 1 --2- 172.21.15.98:0/3108715656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad50101980 0x7fad50196f00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fad4400e9f0 tx=0x7fad4400edf0 comp rx=0 tx=0).stop 2024-05-09T08:50:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.240+0000 7fad54d7e640 1 -- 172.21.15.98:0/3108715656 shutdown_connections 2024-05-09T08:50:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.240+0000 7fad54d7e640 1 --2- 172.21.15.98:0/3108715656 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad50102340 0x7fad5019df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.240+0000 7fad54d7e640 1 --2- 172.21.15.98:0/3108715656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad50101980 0x7fad50196f00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.240+0000 7fad54d7e640 1 --2- 172.21.15.98:0/3108715656 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fad380a2c80 0x7fad380a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.240+0000 7fad54d7e640 1 --2- 172.21.15.98:0/3108715656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad50100790 0x7fad501969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.240+0000 7fad54d7e640 1 -- 172.21.15.98:0/3108715656 >> 172.21.15.98:0/3108715656 conn(0x7fad500fbd00 msgr2=0x7fad50104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.241+0000 7fad54d7e640 1 -- 172.21.15.98:0/3108715656 shutdown_connections 2024-05-09T08:50:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.241+0000 7fad54d7e640 1 -- 172.21.15.98:0/3108715656 wait complete. 2024-05-09T08:50:37.241 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:36 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:36.962+0000 7f8d1d15a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:50:37.241 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:37 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:37.105+0000 7f8d1d15a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:50:37.241 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:37 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:37.205+0000 7f8d1d15a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:50:37.241 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:36.953+0000 7ff96f0cb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:50:37.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-05-09T08:50:37.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.360+0000 7fca67b68640 1 -- 172.21.15.98:0/908627753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca60100790 msgr2=0x7fca60100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:37.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.360+0000 7fca67b68640 1 --2- 172.21.15.98:0/908627753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca60100790 0x7fca60100bb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009900 tx=0x7fca5c02a410 comp rx=0 tx=0).stop 2024-05-09T08:50:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.360+0000 7fca67b68640 1 -- 172.21.15.98:0/908627753 shutdown_connections 2024-05-09T08:50:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.360+0000 7fca67b68640 1 --2- 172.21.15.98:0/908627753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca60102340 0x7fca60108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.360+0000 7fca67b68640 1 --2- 172.21.15.98:0/908627753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca60101980 0x7fca60101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.360+0000 7fca67b68640 1 --2- 172.21.15.98:0/908627753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca60100790 0x7fca60100bb0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.360+0000 7fca67b68640 1 -- 172.21.15.98:0/908627753 >> 172.21.15.98:0/908627753 conn(0x7fca600fbd00 msgr2=0x7fca600fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.360+0000 7fca67b68640 1 -- 172.21.15.98:0/908627753 shutdown_connections 2024-05-09T08:50:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.360+0000 7fca67b68640 1 -- 172.21.15.98:0/908627753 wait complete. 2024-05-09T08:50:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.360+0000 7fca67b68640 1 Processor -- start 2024-05-09T08:50:37.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.360+0000 7fca67b68640 1 -- start start 2024-05-09T08:50:37.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca67b68640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca60100790 0x7fca60197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:37.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca67b68640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca60101980 0x7fca60198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:37.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca67b68640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca60102340 0x7fca6019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:37.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca67b68640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca6010c410 con 0x7fca60100790 2024-05-09T08:50:37.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca67b68640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fca6010c290 con 0x7fca60101980 2024-05-09T08:50:37.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca67b68640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fca6010c590 con 0x7fca60102340 2024-05-09T08:50:37.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca65904640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca60100790 0x7fca60197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:37.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca65904640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca60100790 0x7fca60197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42792/0 (socket says 172.21.15.98:42792) 2024-05-09T08:50:37.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca65904640 1 -- 172.21.15.98:0/577822607 learned_addr learned my addr 172.21.15.98:0/577822607 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca65103640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca60101980 0x7fca60198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca66105640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca60102340 0x7fca6019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca65904640 1 -- 172.21.15.98:0/577822607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca60102340 msgr2=0x7fca6019f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca65904640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca60102340 0x7fca6019f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca65904640 1 -- 172.21.15.98:0/577822607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca60101980 msgr2=0x7fca60198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca65904640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca60101980 0x7fca60198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca65904640 1 -- 172.21.15.98:0/577822607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca6019f770 con 0x7fca60100790 2024-05-09T08:50:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca65904640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca60100790 0x7fca60197b50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fca5c031610 tx=0x7fca5c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca56ffd640 1 -- 172.21.15.98:0/577822607 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c03f030 con 0x7fca60100790 2024-05-09T08:50:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca67b68640 1 -- 172.21.15.98:0/577822607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca6019fa00 con 0x7fca60100790 2024-05-09T08:50:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca56ffd640 1 -- 172.21.15.98:0/577822607 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca5c003ad0 con 0x7fca60100790 2024-05-09T08:50:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.361+0000 7fca56ffd640 1 -- 172.21.15.98:0/577822607 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c035070 con 0x7fca60100790 2024-05-09T08:50:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.362+0000 7fca67b68640 1 -- 172.21.15.98:0/577822607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca6019ff40 con 0x7fca60100790 2024-05-09T08:50:37.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.362+0000 7fca67b68640 1 -- 172.21.15.98:0/577822607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca601a0110 con 0x7fca60100790 2024-05-09T08:50:37.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.364+0000 7fca56ffd640 1 -- 172.21.15.98:0/577822607 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fca5c042030 con 0x7fca60100790 2024-05-09T08:50:37.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.364+0000 7fca56ffd640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fca4c0a2c80 0x7fca4c0a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:37.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.364+0000 7fca65103640 1 -- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fca4c0a2c80 msgr2=0x7fca4c0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1228297308 2024-05-09T08:50:37.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.364+0000 7fca56ffd640 1 -- 172.21.15.98:0/577822607 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9216+0+0 (secure 0 0 0) 0x7fca5c0efa00 con 0x7fca60100790 2024-05-09T08:50:37.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.364+0000 7fca65103640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fca4c0a2c80 0x7fca4c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:50:37.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.369+0000 7fca56ffd640 1 -- 172.21.15.98:0/577822607 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fca5c0bbf40 con 0x7fca60100790 2024-05-09T08:50:37.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.565+0000 7fca65103640 1 -- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fca4c0a2c80 msgr2=0x7fca4c0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1228297308 2024-05-09T08:50:37.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.565+0000 7fca65103640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fca4c0a2c80 0x7fca4c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:50:37.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.579+0000 7fca67b68640 1 -- 172.21.15.98:0/577822607 --> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fca600fde90 con 0x7fca4c0a2c80 2024-05-09T08:50:37.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:37 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3108715656' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-09T08:50:37.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:37 smithi098 ceph-mon[30092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-09T08:50:37.588 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:37 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:37.332+0000 7f8d1d15a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:50:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:37 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3108715656' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-09T08:50:37.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:37 smithi142 ceph-mon[34625]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-09T08:50:37.658 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:37 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:37.256+0000 7ff96f0cb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:50:37.736 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:37 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:37.282+0000 7fcc91034200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:50:37.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:37 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3108715656' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-09T08:50:37.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:37 smithi105 ceph-mon[34498]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-09T08:50:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.966+0000 7fca65103640 1 -- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fca4c0a2c80 msgr2=0x7fca4c0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1228297308 2024-05-09T08:50:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:37.966+0000 7fca65103640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fca4c0a2c80 0x7fca4c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T08:50:38.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:37 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:37.646+0000 7f8d1d15a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:50:38.158 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:37 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:37.852+0000 7ff96f0cb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:50:38.158 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 08:50:38 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z[36477]: 2024-05-09T08:50:38.066+0000 7ff96f0cb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:50:38.236 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:37 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:37.893+0000 7fcc91034200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:50:38.236 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 08:50:38 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y[35879]: 2024-05-09T08:50:38.112+0000 7fcc91034200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:50:38.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:38.254+0000 7fca56ffd640 1 -- 172.21.15.98:0/577822607 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 20) ==== 137850+0+0 (secure 0 0 0) 0x7fca5c003c80 con 0x7fca60100790 2024-05-09T08:50:38.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:38.502+0000 7fca56ffd640 1 -- 172.21.15.98:0/577822607 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 21) ==== 137518+0+0 (secure 0 0 0) 0x7fca5c03b460 con 0x7fca60100790 2024-05-09T08:50:38.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:38.502+0000 7fca56ffd640 1 -- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fca4c0a2c80 msgr2=0x7fca4c0a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:50:38.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:38.502+0000 7fca56ffd640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:6800/1228297308,v1:172.21.15.98:6801/1228297308] conn(0x7fca4c0a2c80 0x7fca4c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:38.588 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:38.266+0000 7f8d1d15a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:50:38.588 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:38.490+0000 7f8d1d15a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:50:38.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:38 smithi098 ceph-mon[30092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-09T08:50:38.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:38 smithi098 ceph-mon[30092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:50:38.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:38 smithi098 ceph-mon[30092]: Standby manager daemon z restarted 2024-05-09T08:50:38.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:38 smithi098 ceph-mon[30092]: Standby manager daemon z started 2024-05-09T08:50:38.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:38 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.142:0/3623561504' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:50:38.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:38 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.142:0/3623561504' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:50:38.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:38 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.142:0/3623561504' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:50:38.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:38 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.142:0/3623561504' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:50:38.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:38 smithi098 ceph-mon[30092]: Standby manager daemon y restarted 2024-05-09T08:50:38.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:38 smithi098 ceph-mon[30092]: Standby manager daemon y started 2024-05-09T08:50:38.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:38 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.105:0/1566443750' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:50:38.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:38 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.105:0/1566443750' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:50:38.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:38 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.105:0/1566443750' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:50:38.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:38 smithi098 ceph-mon[30092]: from='mgr.? 172.21.15.105:0/1566443750' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:50:38.649 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:38 smithi105 ceph-mon[34498]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-09T08:50:38.650 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:38 smithi105 ceph-mon[34498]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:50:38.650 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:38 smithi105 ceph-mon[34498]: Standby manager daemon z restarted 2024-05-09T08:50:38.650 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:38 smithi105 ceph-mon[34498]: Standby manager daemon z started 2024-05-09T08:50:38.650 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:38 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.142:0/3623561504' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:50:38.650 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:38 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.142:0/3623561504' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:50:38.650 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:38 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.142:0/3623561504' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:50:38.650 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:38 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.142:0/3623561504' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:50:38.650 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:38 smithi105 ceph-mon[34498]: Standby manager daemon y restarted 2024-05-09T08:50:38.650 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:38 smithi105 ceph-mon[34498]: Standby manager daemon y started 2024-05-09T08:50:38.650 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:38 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.105:0/1566443750' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:50:38.650 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:38 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.105:0/1566443750' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:50:38.651 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:38 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.105:0/1566443750' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:50:38.651 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:38 smithi105 ceph-mon[34498]: from='mgr.? 172.21.15.105:0/1566443750' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:50:38.651 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:38 smithi142 ceph-mon[34625]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-09T08:50:38.651 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:38 smithi142 ceph-mon[34625]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:50:38.651 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:38 smithi142 ceph-mon[34625]: Standby manager daemon z restarted 2024-05-09T08:50:38.651 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:38 smithi142 ceph-mon[34625]: Standby manager daemon z started 2024-05-09T08:50:38.651 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:38 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.142:0/3623561504' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:50:38.651 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:38 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.142:0/3623561504' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:50:38.651 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:38 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.142:0/3623561504' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:50:38.652 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:38 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.142:0/3623561504' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:50:38.652 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:38 smithi142 ceph-mon[34625]: Standby manager daemon y restarted 2024-05-09T08:50:38.652 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:38 smithi142 ceph-mon[34625]: Standby manager daemon y started 2024-05-09T08:50:38.652 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:38 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.105:0/1566443750' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:50:38.652 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:38 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.105:0/1566443750' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:50:38.652 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:38 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.105:0/1566443750' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:50:38.652 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:38 smithi142 ceph-mon[34625]: from='mgr.? 172.21.15.105:0/1566443750' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:50:39.088 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:38.731+0000 7f8c7ca7f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.088 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:38.731+0000 7f8c7ca7f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.088 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:38.731+0000 7f8c7ca7f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.088 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:38.731+0000 7f8c7ca7f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.088 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:38.731+0000 7f8c7ca7f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.088 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:38.732+0000 7f8c7fa85700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.088 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:38.732+0000 7f8c7fa85700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.088 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:38.732+0000 7f8c7fa85700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.088 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:38.732+0000 7f8c7fa85700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.088 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:38.732+0000 7f8c7fa85700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.510+0000 7fca56ffd640 1 -- 172.21.15.98:0/577822607 <== mon.0 v2:172.21.15.98:3300/0 9 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7fca5c097950 con 0x7fca60100790 2024-05-09T08:50:39.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.511+0000 7fca56ffd640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fca4c035390 0x7fca4c035770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:39.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.511+0000 7fca56ffd640 1 -- 172.21.15.98:0/577822607 --> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fca5c001d30 con 0x7fca4c035390 2024-05-09T08:50:39.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.512+0000 7fca65103640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fca4c035390 0x7fca4c035770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:39.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.515+0000 7fca65103640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fca4c035390 0x7fca4c035770 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fca60199060 tx=0x7fca50005f90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:39.522 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-05-09T08:50:39.522 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: Active manager daemon x restarted 2024-05-09T08:50:39.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: Activating manager daemon x 2024-05-09T08:50:39.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: osdmap e95: 12 total, 12 up, 12 in 2024-05-09T08:50:39.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: mgrmap e21: x(active, starting, since 0.0124302s), standbys: z, y 2024-05-09T08:50:39.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:50:39.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:50:39.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:50:39.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:50:39.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:50:39.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:50:39.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:50:39.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:50:39.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:50:39.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:50:39.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:50:39.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:50:39.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:50:39.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:50:39.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:50:39.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:50:39.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:50:39.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:50:39.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:50:39.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:50:39.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:50:39.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:50:39.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:50:39.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:50:39.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:50:39.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:50:39.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:50:39.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:50:39.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:50:39.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:50:39.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:50:39.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:50:39.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:50:39.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: Manager daemon x is now available 2024-05-09T08:50:39.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:50:39.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:50:39.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:39 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:50:39.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.537+0000 7fca56ffd640 1 -- 172.21.15.98:0/577822607 <== mgr.15555 v2:172.21.15.98:6800/4136638623 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fca5c001d30 con 0x7fca4c035390 2024-05-09T08:50:39.537 INFO:teuthology.orchestra.run.smithi098.stdout:{} 2024-05-09T08:50:39.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.539+0000 7fca67b68640 1 -- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fca4c035390 msgr2=0x7fca4c035770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:39.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.539+0000 7fca67b68640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fca4c035390 0x7fca4c035770 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fca60199060 tx=0x7fca50005f90 comp rx=0 tx=0).stop 2024-05-09T08:50:39.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.539+0000 7fca67b68640 1 -- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca60100790 msgr2=0x7fca60197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:39.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.539+0000 7fca67b68640 1 --2- 172.21.15.98:0/577822607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca60100790 0x7fca60197b50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fca5c031610 tx=0x7fca5c004270 comp rx=0 tx=0).stop 2024-05-09T08:50:39.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.539+0000 7fca66105640 1 -- 172.21.15.98:0/577822607 reap_dead start 2024-05-09T08:50:39.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.539+0000 7fca67b68640 1 -- 172.21.15.98:0/577822607 shutdown_connections 2024-05-09T08:50:39.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.539+0000 7fca67b68640 1 -- 172.21.15.98:0/577822607 >> 172.21.15.98:0/577822607 conn(0x7fca600fbd00 msgr2=0x7fca600fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:39.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.540+0000 7fca67b68640 1 -- 172.21.15.98:0/577822607 shutdown_connections 2024-05-09T08:50:39.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.540+0000 7fca67b68640 1 -- 172.21.15.98:0/577822607 wait complete. 2024-05-09T08:50:39.550 INFO:teuthology.run_tasks:Running task check-counter... 2024-05-09T08:50:39.567 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-05-09T08:50:39.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:50:39.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-05-09T08:50:39.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: Active manager daemon x restarted 2024-05-09T08:50:39.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: Activating manager daemon x 2024-05-09T08:50:39.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: osdmap e95: 12 total, 12 up, 12 in 2024-05-09T08:50:39.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: mgrmap e21: x(active, starting, since 0.0124302s), standbys: z, y 2024-05-09T08:50:39.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:50:39.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:50:39.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:50:39.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:50:39.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:50:39.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:50:39.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:50:39.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:50:39.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:50:39.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:50:39.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:50:39.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:50:39.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:50:39.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:50:39.663 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:50:39.663 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:50:39.663 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:50:39.663 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:50:39.663 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:50:39.663 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:50:39.664 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:50:39.664 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:50:39.664 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:50:39.664 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:50:39.664 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:50:39.664 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:50:39.664 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:50:39.665 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:50:39.665 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:50:39.665 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:50:39.665 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:50:39.665 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:50:39.665 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:50:39.665 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: Manager daemon x is now available 2024-05-09T08:50:39.666 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:50:39.666 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:50:39.666 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:39 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:50:39.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.682+0000 7f1672676640 1 -- 172.21.15.98:0/1953952048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c102b20 msgr2=0x7f166c102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:39.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.682+0000 7f1672676640 1 --2- 172.21.15.98:0/1953952048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c102b20 0x7f166c102fa0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1660009930 tx=0x7f166002a420 comp rx=0 tx=0).stop 2024-05-09T08:50:39.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.682+0000 7f1672676640 1 -- 172.21.15.98:0/1953952048 shutdown_connections 2024-05-09T08:50:39.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.682+0000 7f1672676640 1 --2- 172.21.15.98:0/1953952048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f166c1034e0 0x7f166c109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:39.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.682+0000 7f1672676640 1 --2- 172.21.15.98:0/1953952048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c102b20 0x7f166c102fa0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:39.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.682+0000 7f1672676640 1 --2- 172.21.15.98:0/1953952048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f166c0ffb50 0x7f166c0fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:39.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.682+0000 7f1672676640 1 -- 172.21.15.98:0/1953952048 >> 172.21.15.98:0/1953952048 conn(0x7f166c0fb4a0 msgr2=0x7f166c0fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:39.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.682+0000 7f1672676640 1 -- 172.21.15.98:0/1953952048 shutdown_connections 2024-05-09T08:50:39.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.682+0000 7f1672676640 1 -- 172.21.15.98:0/1953952048 wait complete. 2024-05-09T08:50:39.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.683+0000 7f1672676640 1 Processor -- start 2024-05-09T08:50:39.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.683+0000 7f1672676640 1 -- start start 2024-05-09T08:50:39.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.683+0000 7f1672676640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f166c0ffb50 0x7f166c197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:39.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.683+0000 7f1672676640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c102b20 0x7f166c198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:39.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.683+0000 7f1672676640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f166c1034e0 0x7f166c19f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:39.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.683+0000 7f1672676640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f166c10d530 con 0x7f166c102b20 2024-05-09T08:50:39.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.683+0000 7f1672676640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f166c10d3b0 con 0x7f166c0ffb50 2024-05-09T08:50:39.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.683+0000 7f1672676640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f166c10d6b0 con 0x7f166c1034e0 2024-05-09T08:50:39.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.683+0000 7f166b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c102b20 0x7f166c198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:39.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.683+0000 7f166bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f166c0ffb50 0x7f166c197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:39.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.683+0000 7f166b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c102b20 0x7f166c198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42974/0 (socket says 172.21.15.98:42974) 2024-05-09T08:50:39.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.683+0000 7f166bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f166c0ffb50 0x7f166c197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:56822/0 (socket says 172.21.15.98:56822) 2024-05-09T08:50:39.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.683+0000 7f166b7fe640 1 -- 172.21.15.98:0/1060856781 learned_addr learned my addr 172.21.15.98:0/1060856781 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:39.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.683+0000 7f1670c13640 1 --2- 172.21.15.98:0/1060856781 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f166c1034e0 0x7f166c19f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:39.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.684+0000 7f166b7fe640 1 -- 172.21.15.98:0/1060856781 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f166c1034e0 msgr2=0x7f166c19f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:39.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.684+0000 7f166b7fe640 1 --2- 172.21.15.98:0/1060856781 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f166c1034e0 0x7f166c19f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:39.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.684+0000 7f166b7fe640 1 -- 172.21.15.98:0/1060856781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f166c0ffb50 msgr2=0x7f166c197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:39.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.684+0000 7f166b7fe640 1 --2- 172.21.15.98:0/1060856781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f166c0ffb50 0x7f166c197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:39.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.684+0000 7f166b7fe640 1 -- 172.21.15.98:0/1060856781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166c19f7e0 con 0x7f166c102b20 2024-05-09T08:50:39.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.684+0000 7f166bfff640 1 --2- 172.21.15.98:0/1060856781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f166c0ffb50 0x7f166c197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:50:39.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.684+0000 7f166b7fe640 1 --2- 172.21.15.98:0/1060856781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c102b20 0x7f166c198100 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1660005ab0 tx=0x7f16600042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:39.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.684+0000 7f16697fa640 1 -- 172.21.15.98:0/1060856781 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166003f030 con 0x7f166c102b20 2024-05-09T08:50:39.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.684+0000 7f1672676640 1 -- 172.21.15.98:0/1060856781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166c19fa70 con 0x7f166c102b20 2024-05-09T08:50:39.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.684+0000 7f16697fa640 1 -- 172.21.15.98:0/1060856781 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16600044d0 con 0x7f166c102b20 2024-05-09T08:50:39.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.684+0000 7f16697fa640 1 -- 172.21.15.98:0/1060856781 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660035070 con 0x7f166c102b20 2024-05-09T08:50:39.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.684+0000 7f1672676640 1 -- 172.21.15.98:0/1060856781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f166c19ffb0 con 0x7f166c102b20 2024-05-09T08:50:39.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.685+0000 7f1672676640 1 -- 172.21.15.98:0/1060856781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f166c108cf0 con 0x7f166c102b20 2024-05-09T08:50:39.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.686+0000 7f16697fa640 1 -- 172.21.15.98:0/1060856781 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f1660042030 con 0x7f166c102b20 2024-05-09T08:50:39.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.687+0000 7f16697fa640 1 --2- 172.21.15.98:0/1060856781 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f16540a2990 0x7f16540a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:39.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.687+0000 7f166bfff640 1 --2- 172.21.15.98:0/1060856781 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f16540a2990 0x7f16540a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:39.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.687+0000 7f16697fa640 1 -- 172.21.15.98:0/1060856781 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f16600ef450 con 0x7f166c102b20 2024-05-09T08:50:39.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.687+0000 7f166bfff640 1 --2- 172.21.15.98:0/1060856781 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f16540a2990 0x7f16540a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f166c069cb0 tx=0x7f165c00b390 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:39.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.691+0000 7f16697fa640 1 -- 172.21.15.98:0/1060856781 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f16600bbc40 con 0x7f166c102b20 2024-05-09T08:50:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-05-09T08:50:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: Active manager daemon x restarted 2024-05-09T08:50:39.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: Activating manager daemon x 2024-05-09T08:50:39.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: osdmap e95: 12 total, 12 up, 12 in 2024-05-09T08:50:39.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: mgrmap e21: x(active, starting, since 0.0124302s), standbys: z, y 2024-05-09T08:50:39.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:50:39.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:50:39.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:50:39.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:50:39.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:50:39.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:50:39.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:50:39.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:50:39.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:50:39.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:50:39.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:50:39.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:50:39.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:50:39.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:50:39.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:50:39.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:50:39.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:50:39.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:50:39.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:50:39.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:50:39.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:50:39.740 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:50:39.740 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:50:39.740 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:50:39.740 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:50:39.740 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:50:39.740 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:50:39.740 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:50:39.740 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:50:39.740 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:50:39.740 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:50:39.740 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:50:39.741 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:50:39.741 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: Manager daemon x is now available 2024-05-09T08:50:39.741 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:50:39.741 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:50:39.741 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:39 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:50:39.838 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:39 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:39.522+0000 7f8c79a79700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.838 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:39 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:39.522+0000 7f8c79a79700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.838 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:39 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:39.522+0000 7f8c79a79700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.838 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:39 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:39.522+0000 7f8c79a79700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.838 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:50:39 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:50:39.522+0000 7f8c79a79700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:50:39.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.877+0000 7f1672676640 1 -- 172.21.15.98:0/1060856781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f166c062430 con 0x7f166c102b20 2024-05-09T08:50:39.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.878+0000 7f16697fa640 1 -- 172.21.15.98:0/1060856781 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25206 (secure 0 0 0) 0x7f16600bff40 con 0x7f166c102b20 2024-05-09T08:50:39.878 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:39.878 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":95,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:50:38.492526+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":1}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1013153207":"2024-05-10T08:50:38.492493+0000","172.21.15.98:6800/1228297308":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/808614683":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/1734294041":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/2113007677":"2024-05-10T08:50:38.492493+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1228297308":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:50:39.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.881+0000 7f1672676640 1 -- 172.21.15.98:0/1060856781 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f16540a2990 msgr2=0x7f16540a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:39.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.881+0000 7f1672676640 1 --2- 172.21.15.98:0/1060856781 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f16540a2990 0x7f16540a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f166c069cb0 tx=0x7f165c00b390 comp rx=0 tx=0).stop 2024-05-09T08:50:39.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.881+0000 7f1672676640 1 -- 172.21.15.98:0/1060856781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c102b20 msgr2=0x7f166c198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:39.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.881+0000 7f1672676640 1 --2- 172.21.15.98:0/1060856781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c102b20 0x7f166c198100 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1660005ab0 tx=0x7f16600042e0 comp rx=0 tx=0).stop 2024-05-09T08:50:39.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.881+0000 7f1672676640 1 -- 172.21.15.98:0/1060856781 shutdown_connections 2024-05-09T08:50:39.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.881+0000 7f1672676640 1 --2- 172.21.15.98:0/1060856781 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f16540a2990 0x7f16540a4e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:39.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.881+0000 7f1672676640 1 --2- 172.21.15.98:0/1060856781 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f166c1034e0 0x7f166c19f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:39.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.881+0000 7f1672676640 1 --2- 172.21.15.98:0/1060856781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c102b20 0x7f166c198100 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:39.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.881+0000 7f1672676640 1 --2- 172.21.15.98:0/1060856781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f166c0ffb50 0x7f166c197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:39.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.881+0000 7f1672676640 1 -- 172.21.15.98:0/1060856781 >> 172.21.15.98:0/1060856781 conn(0x7f166c0fb4a0 msgr2=0x7f166c105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:39.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.881+0000 7f1672676640 1 -- 172.21.15.98:0/1060856781 shutdown_connections 2024-05-09T08:50:39.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.881+0000 7f1672676640 1 -- 172.21.15.98:0/1060856781 wait complete. 2024-05-09T08:50:39.892 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:43:39.424443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:48:15.866450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:48:17.122162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:48:20.591475+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:50:39.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:50:39.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.998+0000 7efcd72ba640 1 -- 172.21.15.98:0/652781995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd0102350 msgr2=0x7efcd0104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.998+0000 7efcd72ba640 1 --2- 172.21.15.98:0/652781995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd0102350 0x7efcd0104740 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7efccc009900 tx=0x7efccc02a410 comp rx=0 tx=0).stop 2024-05-09T08:50:40.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.998+0000 7efcd72ba640 1 -- 172.21.15.98:0/652781995 shutdown_connections 2024-05-09T08:50:40.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.998+0000 7efcd72ba640 1 --2- 172.21.15.98:0/652781995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efcd01075b0 0x7efcd01099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.998+0000 7efcd72ba640 1 --2- 172.21.15.98:0/652781995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efcd0104c80 0x7efcd0107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.998+0000 7efcd72ba640 1 --2- 172.21.15.98:0/652781995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd0102350 0x7efcd0104740 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.998+0000 7efcd72ba640 1 -- 172.21.15.98:0/652781995 >> 172.21.15.98:0/652781995 conn(0x7efcd00fbce0 msgr2=0x7efcd00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:40.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.998+0000 7efcd72ba640 1 -- 172.21.15.98:0/652781995 shutdown_connections 2024-05-09T08:50:40.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.998+0000 7efcd72ba640 1 -- 172.21.15.98:0/652781995 wait complete. 2024-05-09T08:50:40.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd72ba640 1 Processor -- start 2024-05-09T08:50:40.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd72ba640 1 -- start start 2024-05-09T08:50:40.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd72ba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd0102350 0x7efcd0197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd72ba640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efcd0104c80 0x7efcd0198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd72ba640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efcd01075b0 0x7efcd019f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd72ba640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efcd010d510 con 0x7efcd0102350 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd72ba640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efcd010d390 con 0x7efcd01075b0 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd72ba640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efcd010d690 con 0x7efcd0104c80 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd5056640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd0102350 0x7efcd0197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd5056640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd0102350 0x7efcd0197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42986/0 (socket says 172.21.15.98:42986) 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd5056640 1 -- 172.21.15.98:0/164533881 learned_addr learned my addr 172.21.15.98:0/164533881 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd4855640 1 --2- 172.21.15.98:0/164533881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efcd0104c80 0x7efcd0198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd5857640 1 --2- 172.21.15.98:0/164533881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efcd01075b0 0x7efcd019f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd5056640 1 -- 172.21.15.98:0/164533881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efcd0104c80 msgr2=0x7efcd0198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd5056640 1 --2- 172.21.15.98:0/164533881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efcd0104c80 0x7efcd0198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd5056640 1 -- 172.21.15.98:0/164533881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efcd01075b0 msgr2=0x7efcd019f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd5056640 1 --2- 172.21.15.98:0/164533881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efcd01075b0 0x7efcd019f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd5056640 1 -- 172.21.15.98:0/164533881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd019f800 con 0x7efcd0102350 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:39.999+0000 7efcd5056640 1 --2- 172.21.15.98:0/164533881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd0102350 0x7efcd0197be0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7efccc009230 tx=0x7efccc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:40.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.000+0000 7efcc67fc640 1 -- 172.21.15.98:0/164533881 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efccc03f030 con 0x7efcd0102350 2024-05-09T08:50:40.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.000+0000 7efcc67fc640 1 -- 172.21.15.98:0/164533881 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efccc003ce0 con 0x7efcd0102350 2024-05-09T08:50:40.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.000+0000 7efcc67fc640 1 -- 172.21.15.98:0/164533881 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efccc035070 con 0x7efcd0102350 2024-05-09T08:50:40.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.000+0000 7efcd72ba640 1 -- 172.21.15.98:0/164533881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcd019fa90 con 0x7efcd0102350 2024-05-09T08:50:40.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.000+0000 7efcd72ba640 1 -- 172.21.15.98:0/164533881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcd019ffd0 con 0x7efcd0102350 2024-05-09T08:50:40.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.001+0000 7efcd72ba640 1 -- 172.21.15.98:0/164533881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcd00fd7d0 con 0x7efcd0102350 2024-05-09T08:50:40.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.002+0000 7efcc67fc640 1 -- 172.21.15.98:0/164533881 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7efccc042030 con 0x7efcd0102350 2024-05-09T08:50:40.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.002+0000 7efcc67fc640 1 --2- 172.21.15.98:0/164533881 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efcbc0a2a10 0x7efcbc0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.002+0000 7efcd4855640 1 --2- 172.21.15.98:0/164533881 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efcbc0a2a10 0x7efcbc0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.002+0000 7efcc67fc640 1 -- 172.21.15.98:0/164533881 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7efccc0ef470 con 0x7efcd0102350 2024-05-09T08:50:40.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.003+0000 7efcd4855640 1 --2- 172.21.15.98:0/164533881 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efcbc0a2a10 0x7efcbc0a4ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efcd01990f0 tx=0x7efcc0005f90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:40.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.007+0000 7efcc67fc640 1 -- 172.21.15.98:0/164533881 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efccc0bbc60 con 0x7efcd0102350 2024-05-09T08:50:40.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.186+0000 7efcd72ba640 1 -- 172.21.15.98:0/164533881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efcd0062430 con 0x7efcd0102350 2024-05-09T08:50:40.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.186+0000 7efcc67fc640 1 -- 172.21.15.98:0/164533881 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7efccc0bff60 con 0x7efcd0102350 2024-05-09T08:50:40.186 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-09T08:50:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.188+0000 7efcd72ba640 1 -- 172.21.15.98:0/164533881 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efcbc0a2a10 msgr2=0x7efcbc0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.188+0000 7efcd72ba640 1 --2- 172.21.15.98:0/164533881 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efcbc0a2a10 0x7efcbc0a4ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efcd01990f0 tx=0x7efcc0005f90 comp rx=0 tx=0).stop 2024-05-09T08:50:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.188+0000 7efcd72ba640 1 -- 172.21.15.98:0/164533881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd0102350 msgr2=0x7efcd0197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.188+0000 7efcd72ba640 1 --2- 172.21.15.98:0/164533881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd0102350 0x7efcd0197be0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7efccc009230 tx=0x7efccc0042f0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.188+0000 7efcd72ba640 1 -- 172.21.15.98:0/164533881 shutdown_connections 2024-05-09T08:50:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.188+0000 7efcd72ba640 1 --2- 172.21.15.98:0/164533881 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efcbc0a2a10 0x7efcbc0a4ed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.188+0000 7efcd72ba640 1 --2- 172.21.15.98:0/164533881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efcd01075b0 0x7efcd019f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.188+0000 7efcd72ba640 1 --2- 172.21.15.98:0/164533881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efcd0104c80 0x7efcd0198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.188+0000 7efcd72ba640 1 --2- 172.21.15.98:0/164533881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd0102350 0x7efcd0197be0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.188+0000 7efcd72ba640 1 -- 172.21.15.98:0/164533881 >> 172.21.15.98:0/164533881 conn(0x7efcd00fbce0 msgr2=0x7efcd0102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.188+0000 7efcd72ba640 1 -- 172.21.15.98:0/164533881 shutdown_connections 2024-05-09T08:50:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.188+0000 7efcd72ba640 1 -- 172.21.15.98:0/164533881 wait complete. 2024-05-09T08:50:40.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:50:40.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.321+0000 7f9259310640 1 -- 172.21.15.98:0/2479075724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9254100790 msgr2=0x7f9254100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.321+0000 7f9259310640 1 --2- 172.21.15.98:0/2479075724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9254100790 0x7f9254100bb0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9244009900 tx=0x7f924402a410 comp rx=0 tx=0).stop 2024-05-09T08:50:40.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.321+0000 7f9259310640 1 -- 172.21.15.98:0/2479075724 shutdown_connections 2024-05-09T08:50:40.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.321+0000 7f9259310640 1 --2- 172.21.15.98:0/2479075724 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9254102340 0x7f9254108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f9259310640 1 --2- 172.21.15.98:0/2479075724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9254101980 0x7f9254101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f9259310640 1 --2- 172.21.15.98:0/2479075724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9254100790 0x7f9254100bb0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f9259310640 1 -- 172.21.15.98:0/2479075724 >> 172.21.15.98:0/2479075724 conn(0x7f92540fbd20 msgr2=0x7f92540fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f9259310640 1 -- 172.21.15.98:0/2479075724 shutdown_connections 2024-05-09T08:50:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f9259310640 1 -- 172.21.15.98:0/2479075724 wait complete. 2024-05-09T08:50:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f9259310640 1 Processor -- start 2024-05-09T08:50:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f9259310640 1 -- start start 2024-05-09T08:50:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f9259310640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9254100790 0x7f92541046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f9259310640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9254101980 0x7f9254102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f9259310640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9254102340 0x7f92541032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f9259310640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f925410c400 con 0x7f9254102340 2024-05-09T08:50:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f9259310640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f925410c280 con 0x7f9254101980 2024-05-09T08:50:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f9259310640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f925410c580 con 0x7f9254100790 2024-05-09T08:50:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f92537fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9254102340 0x7f92541032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f92537fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9254102340 0x7f92541032a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42998/0 (socket says 172.21.15.98:42998) 2024-05-09T08:50:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f924bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9254101980 0x7f9254102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f92537fe640 1 -- 172.21.15.98:0/4058031169 learned_addr learned my addr 172.21.15.98:0/4058031169 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.322+0000 7f9252ffd640 1 --2- 172.21.15.98:0/4058031169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9254100790 0x7f92541046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.323+0000 7f924bfff640 1 -- 172.21.15.98:0/4058031169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9254100790 msgr2=0x7f92541046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.323+0000 7f924bfff640 1 --2- 172.21.15.98:0/4058031169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9254100790 0x7f92541046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.323+0000 7f924bfff640 1 -- 172.21.15.98:0/4058031169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9254102340 msgr2=0x7f92541032a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.323+0000 7f924bfff640 1 --2- 172.21.15.98:0/4058031169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9254102340 0x7f92541032a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.323+0000 7f924bfff640 1 -- 172.21.15.98:0/4058031169 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9254103b60 con 0x7f9254101980 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.323+0000 7f92537fe640 1 --2- 172.21.15.98:0/4058031169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9254102340 0x7f92541032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.323+0000 7f924bfff640 1 --2- 172.21.15.98:0/4058031169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9254101980 0x7f9254102d60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f923c00e9f0 tx=0x7f923c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.323+0000 7f9250ff9640 1 -- 172.21.15.98:0/4058031169 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c00ccc0 con 0x7f9254101980 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.323+0000 7f9259310640 1 -- 172.21.15.98:0/4058031169 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92541a1e00 con 0x7f9254101980 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.323+0000 7f9250ff9640 1 -- 172.21.15.98:0/4058031169 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f923c005020 con 0x7f9254101980 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.323+0000 7f9259310640 1 -- 172.21.15.98:0/4058031169 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92541a2270 con 0x7f9254101980 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.323+0000 7f9250ff9640 1 -- 172.21.15.98:0/4058031169 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c01c070 con 0x7f9254101980 2024-05-09T08:50:40.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.324+0000 7f9259310640 1 -- 172.21.15.98:0/4058031169 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92541a2550 con 0x7f9254101980 2024-05-09T08:50:40.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.325+0000 7f9250ff9640 1 -- 172.21.15.98:0/4058031169 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f923c043a70 con 0x7f9254101980 2024-05-09T08:50:40.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.326+0000 7f9250ff9640 1 --2- 172.21.15.98:0/4058031169 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f92400a2990 0x7f92400a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.326+0000 7f9250ff9640 1 -- 172.21.15.98:0/4058031169 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f923c0cf160 con 0x7f9254101980 2024-05-09T08:50:40.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.326+0000 7f9252ffd640 1 --2- 172.21.15.98:0/4058031169 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f92400a2990 0x7f92400a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.327+0000 7f9252ffd640 1 --2- 172.21.15.98:0/4058031169 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f92400a2990 0x7f92400a4e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9244009230 tx=0x7f92440315b0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:40.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.330+0000 7f9250ff9640 1 -- 172.21.15.98:0/4058031169 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f923c09be90 con 0x7f9254101980 2024-05-09T08:50:40.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.528+0000 7f9259310640 1 -- 172.21.15.98:0/4058031169 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f92541a2720 con 0x7f9254101980 2024-05-09T08:50:40.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.529+0000 7f9250ff9640 1 -- 172.21.15.98:0/4058031169 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f923c0051d0 con 0x7f9254101980 2024-05-09T08:50:40.528 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:50:40.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.531+0000 7f9259310640 1 -- 172.21.15.98:0/4058031169 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f92400a2990 msgr2=0x7f92400a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.531+0000 7f9259310640 1 --2- 172.21.15.98:0/4058031169 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f92400a2990 0x7f92400a4e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9244009230 tx=0x7f92440315b0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.531+0000 7f9259310640 1 -- 172.21.15.98:0/4058031169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9254101980 msgr2=0x7f9254102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.531+0000 7f9259310640 1 --2- 172.21.15.98:0/4058031169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9254101980 0x7f9254102d60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f923c00e9f0 tx=0x7f923c00eeb0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.531+0000 7f9259310640 1 -- 172.21.15.98:0/4058031169 shutdown_connections 2024-05-09T08:50:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.531+0000 7f9259310640 1 --2- 172.21.15.98:0/4058031169 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f92400a2990 0x7f92400a4e50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.531+0000 7f9259310640 1 --2- 172.21.15.98:0/4058031169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9254102340 0x7f92541032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.531+0000 7f9259310640 1 --2- 172.21.15.98:0/4058031169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9254101980 0x7f9254102d60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.531+0000 7f9259310640 1 --2- 172.21.15.98:0/4058031169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9254100790 0x7f92541046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.531+0000 7f9259310640 1 -- 172.21.15.98:0/4058031169 >> 172.21.15.98:0/4058031169 conn(0x7f92540fbd20 msgr2=0x7f9254069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.531+0000 7f9259310640 1 -- 172.21.15.98:0/4058031169 shutdown_connections 2024-05-09T08:50:40.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.531+0000 7f9259310640 1 -- 172.21.15.98:0/4058031169 wait complete. 2024-05-09T08:50:40.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:40 smithi098 ceph-mon[30092]: [09/May/2024:08:50:38] ENGINE Bus STARTING 2024-05-09T08:50:40.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:40 smithi098 ceph-mon[30092]: [09/May/2024:08:50:39] ENGINE Serving on http://172.21.15.98:8765 2024-05-09T08:50:40.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:40 smithi098 ceph-mon[30092]: [09/May/2024:08:50:39] ENGINE Serving on https://172.21.15.98:7150 2024-05-09T08:50:40.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:40 smithi098 ceph-mon[30092]: [09/May/2024:08:50:39] ENGINE Bus STARTED 2024-05-09T08:50:40.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:40 smithi098 ceph-mon[30092]: mgrmap e22: x(active, since 1.0186s), standbys: z, y 2024-05-09T08:50:40.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:40 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1060856781' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:40.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:40 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/164533881' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:40.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:50:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:40 smithi142 ceph-mon[34625]: [09/May/2024:08:50:38] ENGINE Bus STARTING 2024-05-09T08:50:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:40 smithi142 ceph-mon[34625]: [09/May/2024:08:50:39] ENGINE Serving on http://172.21.15.98:8765 2024-05-09T08:50:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:40 smithi142 ceph-mon[34625]: [09/May/2024:08:50:39] ENGINE Serving on https://172.21.15.98:7150 2024-05-09T08:50:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:40 smithi142 ceph-mon[34625]: [09/May/2024:08:50:39] ENGINE Bus STARTED 2024-05-09T08:50:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:40 smithi142 ceph-mon[34625]: mgrmap e22: x(active, since 1.0186s), standbys: z, y 2024-05-09T08:50:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:40 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1060856781' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:40 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/164533881' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.663+0000 7fd011112640 1 -- 172.21.15.98:0/1102147695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd00c0ffb30 msgr2=0x7fd00c0fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.663+0000 7fd011112640 1 --2- 172.21.15.98:0/1102147695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd00c0ffb30 0x7fd00c0fff50 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009930 tx=0x7fcffc02a420 comp rx=0 tx=0).stop 2024-05-09T08:50:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.663+0000 7fd011112640 1 -- 172.21.15.98:0/1102147695 shutdown_connections 2024-05-09T08:50:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.663+0000 7fd011112640 1 --2- 172.21.15.98:0/1102147695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd00c1034c0 0x7fd00c109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.663+0000 7fd011112640 1 --2- 172.21.15.98:0/1102147695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd00c102b00 0x7fd00c102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.663+0000 7fd011112640 1 --2- 172.21.15.98:0/1102147695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd00c0ffb30 0x7fd00c0fff50 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.663+0000 7fd011112640 1 -- 172.21.15.98:0/1102147695 >> 172.21.15.98:0/1102147695 conn(0x7fd00c0fb460 msgr2=0x7fd00c0fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:40.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.664+0000 7fd011112640 1 -- 172.21.15.98:0/1102147695 shutdown_connections 2024-05-09T08:50:40.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.664+0000 7fd011112640 1 -- 172.21.15.98:0/1102147695 wait complete. 2024-05-09T08:50:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.665+0000 7fd011112640 1 Processor -- start 2024-05-09T08:50:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.665+0000 7fd011112640 1 -- start start 2024-05-09T08:50:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.665+0000 7fd011112640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd00c0ffb30 0x7fd00c197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.665+0000 7fd011112640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd00c102b00 0x7fd00c198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.665+0000 7fd011112640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd00c1034c0 0x7fd00c19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.665+0000 7fd011112640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c10d4e0 con 0x7fd00c0ffb30 2024-05-09T08:50:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.665+0000 7fd011112640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c10d360 con 0x7fd00c102b00 2024-05-09T08:50:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.665+0000 7fd011112640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c10d660 con 0x7fd00c1034c0 2024-05-09T08:50:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.665+0000 7fd00ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd00c0ffb30 0x7fd00c197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.665+0000 7fd003fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd00c102b00 0x7fd00c198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.665+0000 7fd003fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd00c102b00 0x7fd00c198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:56878/0 (socket says 172.21.15.98:56878) 2024-05-09T08:50:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.665+0000 7fd00ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd00c0ffb30 0x7fd00c197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43030/0 (socket says 172.21.15.98:43030) 2024-05-09T08:50:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.665+0000 7fd003fff640 1 -- 172.21.15.98:0/1354922583 learned_addr learned my addr 172.21.15.98:0/1354922583 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.665+0000 7fd00b59e640 1 --2- 172.21.15.98:0/1354922583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd00c1034c0 0x7fd00c19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.666+0000 7fd003fff640 1 -- 172.21.15.98:0/1354922583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd00c1034c0 msgr2=0x7fd00c19f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.666+0000 7fd003fff640 1 --2- 172.21.15.98:0/1354922583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd00c1034c0 0x7fd00c19f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.666+0000 7fd003fff640 1 -- 172.21.15.98:0/1354922583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd00c0ffb30 msgr2=0x7fd00c197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.666+0000 7fd003fff640 1 --2- 172.21.15.98:0/1354922583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd00c0ffb30 0x7fd00c197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.666+0000 7fd003fff640 1 -- 172.21.15.98:0/1354922583 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd00c19f840 con 0x7fd00c102b00 2024-05-09T08:50:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.666+0000 7fd003fff640 1 --2- 172.21.15.98:0/1354922583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd00c102b00 0x7fd00c198170 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcff400e9f0 tx=0x7fcff400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.666+0000 7fd008d99640 1 -- 172.21.15.98:0/1354922583 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff400cd80 con 0x7fd00c102b00 2024-05-09T08:50:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.666+0000 7fd008d99640 1 -- 172.21.15.98:0/1354922583 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcff4010910 con 0x7fd00c102b00 2024-05-09T08:50:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.666+0000 7fd011112640 1 -- 172.21.15.98:0/1354922583 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd00c19fb30 con 0x7fd00c102b00 2024-05-09T08:50:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.666+0000 7fd008d99640 1 -- 172.21.15.98:0/1354922583 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4018990 con 0x7fd00c102b00 2024-05-09T08:50:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.666+0000 7fd011112640 1 -- 172.21.15.98:0/1354922583 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd00c1a0050 con 0x7fd00c102b00 2024-05-09T08:50:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.667+0000 7fd011112640 1 -- 172.21.15.98:0/1354922583 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd00c108d00 con 0x7fd00c102b00 2024-05-09T08:50:40.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.668+0000 7fd008d99640 1 -- 172.21.15.98:0/1354922583 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7fcff4010ac0 con 0x7fd00c102b00 2024-05-09T08:50:40.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.669+0000 7fd008d99640 1 --2- 172.21.15.98:0/1354922583 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcff80a2990 0x7fcff80a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.669+0000 7fd008d99640 1 -- 172.21.15.98:0/1354922583 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fcff40cf4c0 con 0x7fd00c102b00 2024-05-09T08:50:40.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.669+0000 7fd00ad9d640 1 --2- 172.21.15.98:0/1354922583 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcff80a2990 0x7fcff80a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.669+0000 7fd00ad9d640 1 --2- 172.21.15.98:0/1354922583 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcff80a2990 0x7fcff80a4e50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcffc031440 tx=0x7fcffc000bc0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.673+0000 7fd008d99640 1 -- 172.21.15.98:0/1354922583 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcff409bbe0 con 0x7fd00c102b00 2024-05-09T08:50:40.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:40 smithi105 ceph-mon[34498]: [09/May/2024:08:50:38] ENGINE Bus STARTING 2024-05-09T08:50:40.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:40 smithi105 ceph-mon[34498]: [09/May/2024:08:50:39] ENGINE Serving on http://172.21.15.98:8765 2024-05-09T08:50:40.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:40 smithi105 ceph-mon[34498]: [09/May/2024:08:50:39] ENGINE Serving on https://172.21.15.98:7150 2024-05-09T08:50:40.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:40 smithi105 ceph-mon[34498]: [09/May/2024:08:50:39] ENGINE Bus STARTED 2024-05-09T08:50:40.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:40 smithi105 ceph-mon[34498]: mgrmap e22: x(active, since 1.0186s), standbys: z, y 2024-05-09T08:50:40.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:40 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1060856781' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:40.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:40 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/164533881' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:40.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.861+0000 7fd011112640 1 -- 172.21.15.98:0/1354922583 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd00c062430 con 0x7fd00c102b00 2024-05-09T08:50:40.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.861+0000 7fd008d99640 1 -- 172.21.15.98:0/1354922583 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7fcff409fee0 con 0x7fd00c102b00 2024-05-09T08:50:40.861 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-09T08:50:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.863+0000 7fd011112640 1 -- 172.21.15.98:0/1354922583 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcff80a2990 msgr2=0x7fcff80a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.863+0000 7fd011112640 1 --2- 172.21.15.98:0/1354922583 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcff80a2990 0x7fcff80a4e50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcffc031440 tx=0x7fcffc000bc0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.863+0000 7fd011112640 1 -- 172.21.15.98:0/1354922583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd00c102b00 msgr2=0x7fd00c198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.863+0000 7fd011112640 1 --2- 172.21.15.98:0/1354922583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd00c102b00 0x7fd00c198170 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcff400e9f0 tx=0x7fcff400eeb0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.863+0000 7fd011112640 1 -- 172.21.15.98:0/1354922583 shutdown_connections 2024-05-09T08:50:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.863+0000 7fd011112640 1 --2- 172.21.15.98:0/1354922583 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcff80a2990 0x7fcff80a4e50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.863+0000 7fd011112640 1 --2- 172.21.15.98:0/1354922583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd00c1034c0 0x7fd00c19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.863+0000 7fd011112640 1 --2- 172.21.15.98:0/1354922583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd00c102b00 0x7fd00c198170 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.863+0000 7fd011112640 1 --2- 172.21.15.98:0/1354922583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd00c0ffb30 0x7fd00c197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.863+0000 7fd011112640 1 -- 172.21.15.98:0/1354922583 >> 172.21.15.98:0/1354922583 conn(0x7fd00c0fb460 msgr2=0x7fd00c105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:40.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.864+0000 7fd011112640 1 -- 172.21.15.98:0/1354922583 shutdown_connections 2024-05-09T08:50:40.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.864+0000 7fd011112640 1 -- 172.21.15.98:0/1354922583 wait complete. 2024-05-09T08:50:40.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:50:40.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.981+0000 7f50c33f9640 1 -- 172.21.15.98:0/2864214193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50bc1075b0 msgr2=0x7f50bc1099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.981+0000 7f50c33f9640 1 --2- 172.21.15.98:0/2864214193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50bc1075b0 0x7f50bc1099a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009ca0 tx=0x7f50b402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.981+0000 7f50c33f9640 1 -- 172.21.15.98:0/2864214193 shutdown_connections 2024-05-09T08:50:40.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.981+0000 7f50c33f9640 1 --2- 172.21.15.98:0/2864214193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50bc1075b0 0x7f50bc1099a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.981+0000 7f50c33f9640 1 --2- 172.21.15.98:0/2864214193 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50bc104c80 0x7f50bc107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.981+0000 7f50c33f9640 1 --2- 172.21.15.98:0/2864214193 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50bc102350 0x7f50bc104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.981+0000 7f50c33f9640 1 -- 172.21.15.98:0/2864214193 >> 172.21.15.98:0/2864214193 conn(0x7f50bc0fbce0 msgr2=0x7f50bc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:40.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.981+0000 7f50c33f9640 1 -- 172.21.15.98:0/2864214193 shutdown_connections 2024-05-09T08:50:40.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.981+0000 7f50c33f9640 1 -- 172.21.15.98:0/2864214193 wait complete. 2024-05-09T08:50:40.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.982+0000 7f50c33f9640 1 Processor -- start 2024-05-09T08:50:40.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.982+0000 7f50c33f9640 1 -- start start 2024-05-09T08:50:40.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.982+0000 7f50c33f9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50bc102350 0x7f50bc197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.982+0000 7f50c33f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50bc104c80 0x7f50bc1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.982+0000 7f50c33f9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50bc1075b0 0x7f50bc19f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.982+0000 7f50c33f9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc10d510 con 0x7f50bc104c80 2024-05-09T08:50:40.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.982+0000 7f50c33f9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc10d390 con 0x7f50bc102350 2024-05-09T08:50:40.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.982+0000 7f50c33f9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc10d690 con 0x7f50bc1075b0 2024-05-09T08:50:40.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.982+0000 7f50c1195640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50bc102350 0x7f50bc197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.982+0000 7f50c0994640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50bc104c80 0x7f50bc1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.982+0000 7f50c1195640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50bc102350 0x7f50bc197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:56886/0 (socket says 172.21.15.98:56886) 2024-05-09T08:50:40.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.982+0000 7f50c0994640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50bc104c80 0x7f50bc1980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43046/0 (socket says 172.21.15.98:43046) 2024-05-09T08:50:40.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.982+0000 7f50c1195640 1 -- 172.21.15.98:0/2394295664 learned_addr learned my addr 172.21.15.98:0/2394295664 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:40.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.983+0000 7f50c1996640 1 --2- 172.21.15.98:0/2394295664 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50bc1075b0 0x7f50bc19f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.983+0000 7f50c0994640 1 -- 172.21.15.98:0/2394295664 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50bc1075b0 msgr2=0x7f50bc19f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.983+0000 7f50c0994640 1 --2- 172.21.15.98:0/2394295664 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50bc1075b0 0x7f50bc19f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.983+0000 7f50c0994640 1 -- 172.21.15.98:0/2394295664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50bc102350 msgr2=0x7f50bc197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:40.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.983+0000 7f50c0994640 1 --2- 172.21.15.98:0/2394295664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50bc102350 0x7f50bc197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:40.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.983+0000 7f50c0994640 1 -- 172.21.15.98:0/2394295664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50bc19f7d0 con 0x7f50bc104c80 2024-05-09T08:50:40.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.983+0000 7f50c1195640 1 --2- 172.21.15.98:0/2394295664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50bc102350 0x7f50bc197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:50:40.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.983+0000 7f50c0994640 1 --2- 172.21.15.98:0/2394295664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50bc104c80 0x7f50bc1980f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00e9c0 tx=0x7f50ac00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:40.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.983+0000 7f50b27fc640 1 -- 172.21.15.98:0/2394295664 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ac00ccc0 con 0x7f50bc104c80 2024-05-09T08:50:40.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.983+0000 7f50b27fc640 1 -- 172.21.15.98:0/2394295664 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50ac005020 con 0x7f50bc104c80 2024-05-09T08:50:40.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.983+0000 7f50c33f9640 1 -- 172.21.15.98:0/2394295664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50bc19fac0 con 0x7f50bc104c80 2024-05-09T08:50:40.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.983+0000 7f50b27fc640 1 -- 172.21.15.98:0/2394295664 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ac004a00 con 0x7f50bc104c80 2024-05-09T08:50:40.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.983+0000 7f50c33f9640 1 -- 172.21.15.98:0/2394295664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50bc1a0010 con 0x7f50bc104c80 2024-05-09T08:50:40.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.984+0000 7f50c33f9640 1 -- 172.21.15.98:0/2394295664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50bc0fd7d0 con 0x7f50bc104c80 2024-05-09T08:50:40.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.985+0000 7f50b27fc640 1 -- 172.21.15.98:0/2394295664 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f50ac0051d0 con 0x7f50bc104c80 2024-05-09T08:50:40.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.986+0000 7f50b27fc640 1 --2- 172.21.15.98:0/2394295664 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50a80a2a10 0x7f50a80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:40.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.986+0000 7f50c1195640 1 --2- 172.21.15.98:0/2394295664 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50a80a2a10 0x7f50a80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:40.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.986+0000 7f50b27fc640 1 -- 172.21.15.98:0/2394295664 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f50ac0cf0d0 con 0x7f50bc104c80 2024-05-09T08:50:40.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.987+0000 7f50c1195640 1 --2- 172.21.15.98:0/2394295664 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50a80a2a10 0x7f50a80a4ed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f50b8011550 tx=0x7f50b8005f90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:40.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:40.989+0000 7f50b27fc640 1 -- 172.21.15.98:0/2394295664 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f50ac09be90 con 0x7f50bc104c80 2024-05-09T08:50:41.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.173+0000 7f50c33f9640 1 -- 172.21.15.98:0/2394295664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f50bc02d410 con 0x7f50bc104c80 2024-05-09T08:50:41.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.174+0000 7f50b27fc640 1 -- 172.21.15.98:0/2394295664 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f50ac09b790 con 0x7f50bc104c80 2024-05-09T08:50:41.173 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:50:41.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.176+0000 7f50c33f9640 1 -- 172.21.15.98:0/2394295664 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50a80a2a10 msgr2=0x7f50a80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.176+0000 7f50c33f9640 1 --2- 172.21.15.98:0/2394295664 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50a80a2a10 0x7f50a80a4ed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f50b8011550 tx=0x7f50b8005f90 comp rx=0 tx=0).stop 2024-05-09T08:50:41.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.176+0000 7f50c33f9640 1 -- 172.21.15.98:0/2394295664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50bc104c80 msgr2=0x7f50bc1980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.176+0000 7f50c33f9640 1 --2- 172.21.15.98:0/2394295664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50bc104c80 0x7f50bc1980f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00e9c0 tx=0x7f50ac00ee80 comp rx=0 tx=0).stop 2024-05-09T08:50:41.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.176+0000 7f50c33f9640 1 -- 172.21.15.98:0/2394295664 shutdown_connections 2024-05-09T08:50:41.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.176+0000 7f50c33f9640 1 --2- 172.21.15.98:0/2394295664 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50a80a2a10 0x7f50a80a4ed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.176+0000 7f50c33f9640 1 --2- 172.21.15.98:0/2394295664 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50bc1075b0 0x7f50bc19f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.176+0000 7f50c33f9640 1 --2- 172.21.15.98:0/2394295664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50bc104c80 0x7f50bc1980f0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.176+0000 7f50c33f9640 1 --2- 172.21.15.98:0/2394295664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50bc102350 0x7f50bc197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.176+0000 7f50c33f9640 1 -- 172.21.15.98:0/2394295664 >> 172.21.15.98:0/2394295664 conn(0x7f50bc0fbce0 msgr2=0x7f50bc102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:41.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.176+0000 7f50c33f9640 1 -- 172.21.15.98:0/2394295664 shutdown_connections 2024-05-09T08:50:41.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.176+0000 7f50c33f9640 1 -- 172.21.15.98:0/2394295664 wait complete. 2024-05-09T08:50:41.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:50:41.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.307+0000 7f758939e640 1 -- 172.21.15.98:0/1548856735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7584104ca0 msgr2=0x7f7584107090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.307+0000 7f758939e640 1 --2- 172.21.15.98:0/1548856735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7584104ca0 0x7f7584107090 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7578009900 tx=0x7f757802a410 comp rx=0 tx=0).stop 2024-05-09T08:50:41.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.308+0000 7f758939e640 1 -- 172.21.15.98:0/1548856735 shutdown_connections 2024-05-09T08:50:41.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.308+0000 7f758939e640 1 --2- 172.21.15.98:0/1548856735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75841075d0 0x7f7584109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.308+0000 7f758939e640 1 --2- 172.21.15.98:0/1548856735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7584104ca0 0x7f7584107090 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.308+0000 7f758939e640 1 --2- 172.21.15.98:0/1548856735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7584102370 0x7f7584104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.308+0000 7f758939e640 1 -- 172.21.15.98:0/1548856735 >> 172.21.15.98:0/1548856735 conn(0x7f75840fbd80 msgr2=0x7f75840fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:41.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.308+0000 7f758939e640 1 -- 172.21.15.98:0/1548856735 shutdown_connections 2024-05-09T08:50:41.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.308+0000 7f758939e640 1 -- 172.21.15.98:0/1548856735 wait complete. 2024-05-09T08:50:41.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.308+0000 7f758939e640 1 Processor -- start 2024-05-09T08:50:41.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.308+0000 7f758939e640 1 -- start start 2024-05-09T08:50:41.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f758939e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7584102370 0x7f7584101ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:41.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f758939e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7584104ca0 0x7f7584100320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:41.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f758939e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75841075d0 0x7f7584100860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:41.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f758939e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f758410d500 con 0x7f7584104ca0 2024-05-09T08:50:41.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f758939e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f758410d380 con 0x7f75841075d0 2024-05-09T08:50:41.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f758939e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f758410d680 con 0x7f7584102370 2024-05-09T08:50:41.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f75827fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7584104ca0 0x7f7584100320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:41.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f75827fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7584104ca0 0x7f7584100320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43076/0 (socket says 172.21.15.98:43076) 2024-05-09T08:50:41.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f75827fc640 1 -- 172.21.15.98:0/36091111 learned_addr learned my addr 172.21.15.98:0/36091111 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:41.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f7582ffd640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7584102370 0x7f7584101ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:41.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f75837fe640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75841075d0 0x7f7584100860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:41.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f75827fc640 1 -- 172.21.15.98:0/36091111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7584102370 msgr2=0x7f7584101ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f75827fc640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7584102370 0x7f7584101ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f75827fc640 1 -- 172.21.15.98:0/36091111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75841075d0 msgr2=0x7f7584100860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f75827fc640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75841075d0 0x7f7584100860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f75827fc640 1 -- 172.21.15.98:0/36091111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75841010f0 con 0x7f7584104ca0 2024-05-09T08:50:41.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f75837fe640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75841075d0 0x7f7584100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:50:41.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.309+0000 7f75827fc640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7584104ca0 0x7f7584100320 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7578031610 tx=0x7f7578004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:41.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.310+0000 7f756bfff640 1 -- 172.21.15.98:0/36091111 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757803f030 con 0x7f7584104ca0 2024-05-09T08:50:41.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.310+0000 7f758939e640 1 -- 172.21.15.98:0/36091111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75841a1c90 con 0x7f7584104ca0 2024-05-09T08:50:41.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.310+0000 7f756bfff640 1 -- 172.21.15.98:0/36091111 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7578003ad0 con 0x7f7584104ca0 2024-05-09T08:50:41.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.310+0000 7f756bfff640 1 -- 172.21.15.98:0/36091111 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7578035070 con 0x7f7584104ca0 2024-05-09T08:50:41.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.310+0000 7f758939e640 1 -- 172.21.15.98:0/36091111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75841a2190 con 0x7f7584104ca0 2024-05-09T08:50:41.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.311+0000 7f758939e640 1 -- 172.21.15.98:0/36091111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75840fd870 con 0x7f7584104ca0 2024-05-09T08:50:41.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.312+0000 7f756bfff640 1 -- 172.21.15.98:0/36091111 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f7578042030 con 0x7f7584104ca0 2024-05-09T08:50:41.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.313+0000 7f756bfff640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f756c0a2d90 0x7f756c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:41.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.313+0000 7f756bfff640 1 -- 172.21.15.98:0/36091111 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f75780f0200 con 0x7f7584104ca0 2024-05-09T08:50:41.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.313+0000 7f7582ffd640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f756c0a2d90 0x7f756c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:41.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.313+0000 7f7582ffd640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f756c0a2d90 0x7f756c0a5250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f75740095f0 tx=0x7f7574005f90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:41.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.320+0000 7f756bfff640 1 -- 172.21.15.98:0/36091111 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f75780bc2a0 con 0x7f7584104ca0 2024-05-09T08:50:41.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.509+0000 7f758939e640 1 -- 172.21.15.98:0/36091111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7584062430 con 0x7f7584104ca0 2024-05-09T08:50:41.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.511+0000 7f756bfff640 1 -- 172.21.15.98:0/36091111 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25206 (secure 0 0 0) 0x7f75780c05a0 con 0x7f7584104ca0 2024-05-09T08:50:41.510 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:41.511 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":95,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:50:38.492526+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":1}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1013153207":"2024-05-10T08:50:38.492493+0000","172.21.15.98:6800/1228297308":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/808614683":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/1734294041":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/2113007677":"2024-05-10T08:50:38.492493+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1228297308":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:50:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.513+0000 7f758939e640 1 -- 172.21.15.98:0/36091111 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f756c0a2d90 msgr2=0x7f756c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.513+0000 7f758939e640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f756c0a2d90 0x7f756c0a5250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f75740095f0 tx=0x7f7574005f90 comp rx=0 tx=0).stop 2024-05-09T08:50:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.513+0000 7f758939e640 1 -- 172.21.15.98:0/36091111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7584104ca0 msgr2=0x7f7584100320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.513+0000 7f758939e640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7584104ca0 0x7f7584100320 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7578031610 tx=0x7f7578004270 comp rx=0 tx=0).stop 2024-05-09T08:50:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.513+0000 7f758939e640 1 -- 172.21.15.98:0/36091111 shutdown_connections 2024-05-09T08:50:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.513+0000 7f758939e640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f756c0a2d90 0x7f756c0a5250 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.513+0000 7f758939e640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75841075d0 0x7f7584100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.513+0000 7f758939e640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7584104ca0 0x7f7584100320 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.513+0000 7f758939e640 1 --2- 172.21.15.98:0/36091111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7584102370 0x7f7584101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.513+0000 7f758939e640 1 -- 172.21.15.98:0/36091111 >> 172.21.15.98:0/36091111 conn(0x7f75840fbd80 msgr2=0x7f7584102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.513+0000 7f758939e640 1 -- 172.21.15.98:0/36091111 shutdown_connections 2024-05-09T08:50:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.513+0000 7f758939e640 1 -- 172.21.15.98:0/36091111 wait complete. 2024-05-09T08:50:41.522 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:41 smithi098 ceph-mon[30092]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:41.522 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:41 smithi098 ceph-mon[30092]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:50:41.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:41 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4058031169' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:41.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:41 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:41 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:41 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:41 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:41 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:41 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:41 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1354922583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:41.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:41 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2394295664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:41.524 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:43:39.424443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:48:15.866450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:48:17.122162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:48:20.591475+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:50:41.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:50:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.633+0000 7f598be88640 1 -- 172.21.15.98:0/1725324673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5984100770 msgr2=0x7f5984100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.633+0000 7f598be88640 1 --2- 172.21.15.98:0/1725324673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5984100770 0x7f5984100b90 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5980009900 tx=0x7f598002a410 comp rx=0 tx=0).stop 2024-05-09T08:50:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.633+0000 7f598be88640 1 -- 172.21.15.98:0/1725324673 shutdown_connections 2024-05-09T08:50:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.633+0000 7f598be88640 1 --2- 172.21.15.98:0/1725324673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5984102320 0x7f5984108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.633+0000 7f598be88640 1 --2- 172.21.15.98:0/1725324673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5984101960 0x7f5984101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.633+0000 7f598be88640 1 --2- 172.21.15.98:0/1725324673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5984100770 0x7f5984100b90 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.633+0000 7f598be88640 1 -- 172.21.15.98:0/1725324673 >> 172.21.15.98:0/1725324673 conn(0x7f59840fbce0 msgr2=0x7f59840fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.633+0000 7f598be88640 1 -- 172.21.15.98:0/1725324673 shutdown_connections 2024-05-09T08:50:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.633+0000 7f598be88640 1 -- 172.21.15.98:0/1725324673 wait complete. 2024-05-09T08:50:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.633+0000 7f598be88640 1 Processor -- start 2024-05-09T08:50:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.633+0000 7f598be88640 1 -- start start 2024-05-09T08:50:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.633+0000 7f598be88640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5984100770 0x7f5984197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f598be88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5984101960 0x7f5984198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f598be88640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5984102320 0x7f598419f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f598be88640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f598410c3e0 con 0x7f5984101960 2024-05-09T08:50:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f598be88640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f598410c260 con 0x7f5984100770 2024-05-09T08:50:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f598be88640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f598410c560 con 0x7f5984102320 2024-05-09T08:50:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f5989423640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5984101960 0x7f5984198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f5989423640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5984101960 0x7f5984198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43094/0 (socket says 172.21.15.98:43094) 2024-05-09T08:50:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f5989423640 1 -- 172.21.15.98:0/309169120 learned_addr learned my addr 172.21.15.98:0/309169120 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f598a425640 1 --2- 172.21.15.98:0/309169120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5984102320 0x7f598419f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f5989c24640 1 --2- 172.21.15.98:0/309169120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5984100770 0x7f5984197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f5989423640 1 -- 172.21.15.98:0/309169120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5984102320 msgr2=0x7f598419f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f5989423640 1 --2- 172.21.15.98:0/309169120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5984102320 0x7f598419f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f5989423640 1 -- 172.21.15.98:0/309169120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5984100770 msgr2=0x7f5984197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f5989423640 1 --2- 172.21.15.98:0/309169120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5984100770 0x7f5984197c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f5989423640 1 -- 172.21.15.98:0/309169120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f598419f860 con 0x7f5984101960 2024-05-09T08:50:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f5989423640 1 --2- 172.21.15.98:0/309169120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5984101960 0x7f5984198190 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f597400e9f0 tx=0x7f597400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f597affd640 1 -- 172.21.15.98:0/309169120 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597400cd80 con 0x7f5984101960 2024-05-09T08:50:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f598be88640 1 -- 172.21.15.98:0/309169120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f598419fb50 con 0x7f5984101960 2024-05-09T08:50:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f597affd640 1 -- 172.21.15.98:0/309169120 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5974005500 con 0x7f5984101960 2024-05-09T08:50:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f597affd640 1 -- 172.21.15.98:0/309169120 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5974010690 con 0x7f5984101960 2024-05-09T08:50:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.634+0000 7f598be88640 1 -- 172.21.15.98:0/309169120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59841a0070 con 0x7f5984101960 2024-05-09T08:50:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.635+0000 7f598be88640 1 -- 172.21.15.98:0/309169120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59841a02a0 con 0x7f5984101960 2024-05-09T08:50:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.637+0000 7f597affd640 1 -- 172.21.15.98:0/309169120 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f59740046c0 con 0x7f5984101960 2024-05-09T08:50:41.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.638+0000 7f597affd640 1 --2- 172.21.15.98:0/309169120 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f59700a2d90 0x7f59700a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:41.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.638+0000 7f597affd640 1 -- 172.21.15.98:0/309169120 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f59740d0050 con 0x7f5984101960 2024-05-09T08:50:41.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.638+0000 7f5989c24640 1 --2- 172.21.15.98:0/309169120 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f59700a2d90 0x7f59700a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:41.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.638+0000 7f5989c24640 1 --2- 172.21.15.98:0/309169120 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f59700a2d90 0x7f59700a5250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5980005ab0 tx=0x7f5980000bc0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:41.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.641+0000 7f597affd640 1 -- 172.21.15.98:0/309169120 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5974053740 con 0x7f5984101960 2024-05-09T08:50:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:41 smithi142 ceph-mon[34625]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:41 smithi142 ceph-mon[34625]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:50:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:41 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/4058031169' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:41 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:41 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:41 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:41 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:41 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:41 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:41 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1354922583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:41.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:41 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2394295664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:41.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:41 smithi105 ceph-mon[34498]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:41.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:41 smithi105 ceph-mon[34498]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:50:41.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:41 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/4058031169' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:41.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:41 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:41 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:41 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:41 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:41 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:41 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:41.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:41 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1354922583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:41.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:41 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2394295664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:41.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.834+0000 7f598be88640 1 -- 172.21.15.98:0/309169120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f59841a0470 con 0x7f5984101960 2024-05-09T08:50:41.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.834+0000 7f597affd640 1 -- 172.21.15.98:0/309169120 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f597409c740 con 0x7f5984101960 2024-05-09T08:50:41.833 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-09T08:50:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.836+0000 7f598be88640 1 -- 172.21.15.98:0/309169120 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f59700a2d90 msgr2=0x7f59700a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.836+0000 7f598be88640 1 --2- 172.21.15.98:0/309169120 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f59700a2d90 0x7f59700a5250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5980005ab0 tx=0x7f5980000bc0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.836+0000 7f598be88640 1 -- 172.21.15.98:0/309169120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5984101960 msgr2=0x7f5984198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.836+0000 7f598be88640 1 --2- 172.21.15.98:0/309169120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5984101960 0x7f5984198190 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f597400e9f0 tx=0x7f597400edf0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.836+0000 7f598be88640 1 -- 172.21.15.98:0/309169120 shutdown_connections 2024-05-09T08:50:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.836+0000 7f598be88640 1 --2- 172.21.15.98:0/309169120 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f59700a2d90 0x7f59700a5250 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.836+0000 7f598be88640 1 --2- 172.21.15.98:0/309169120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5984102320 0x7f598419f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.836+0000 7f598be88640 1 --2- 172.21.15.98:0/309169120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5984101960 0x7f5984198190 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.836+0000 7f598be88640 1 --2- 172.21.15.98:0/309169120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5984100770 0x7f5984197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.836+0000 7f598be88640 1 -- 172.21.15.98:0/309169120 >> 172.21.15.98:0/309169120 conn(0x7f59840fbce0 msgr2=0x7f59840fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.836+0000 7f598be88640 1 -- 172.21.15.98:0/309169120 shutdown_connections 2024-05-09T08:50:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.836+0000 7f598be88640 1 -- 172.21.15.98:0/309169120 wait complete. 2024-05-09T08:50:41.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:50:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.956+0000 7f81cd6c1640 1 -- 172.21.15.98:0/2810581404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c80ffb30 msgr2=0x7f81c80fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.956+0000 7f81cd6c1640 1 --2- 172.21.15.98:0/2810581404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c80ffb30 0x7f81c80fff50 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f81b8009930 tx=0x7f81b802a420 comp rx=0 tx=0).stop 2024-05-09T08:50:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.956+0000 7f81cd6c1640 1 -- 172.21.15.98:0/2810581404 shutdown_connections 2024-05-09T08:50:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.956+0000 7f81cd6c1640 1 --2- 172.21.15.98:0/2810581404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81c81034c0 0x7f81c8109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.956+0000 7f81cd6c1640 1 --2- 172.21.15.98:0/2810581404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f81c8102b00 0x7f81c8102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.956+0000 7f81cd6c1640 1 --2- 172.21.15.98:0/2810581404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c80ffb30 0x7f81c80fff50 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.956+0000 7f81cd6c1640 1 -- 172.21.15.98:0/2810581404 >> 172.21.15.98:0/2810581404 conn(0x7f81c80fb460 msgr2=0x7f81c80fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.956+0000 7f81cd6c1640 1 -- 172.21.15.98:0/2810581404 shutdown_connections 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.956+0000 7f81cd6c1640 1 -- 172.21.15.98:0/2810581404 wait complete. 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.957+0000 7f81cd6c1640 1 Processor -- start 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.957+0000 7f81cd6c1640 1 -- start start 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.957+0000 7f81cd6c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c80ffb30 0x7f81c8197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.957+0000 7f81cd6c1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81c8102b00 0x7f81c8198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.957+0000 7f81cd6c1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f81c81034c0 0x7f81c819f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.957+0000 7f81cd6c1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81c810d500 con 0x7f81c80ffb30 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.957+0000 7f81cd6c1640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f81c810d380 con 0x7f81c8102b00 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.957+0000 7f81cd6c1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f81c810d680 con 0x7f81c81034c0 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.957+0000 7f81c6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c80ffb30 0x7f81c8197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.957+0000 7f81c6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c80ffb30 0x7f81c8197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43114/0 (socket says 172.21.15.98:43114) 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.957+0000 7f81c6ffd640 1 -- 172.21.15.98:0/3894629545 learned_addr learned my addr 172.21.15.98:0/3894629545 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.957+0000 7f81be7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81c8102b00 0x7f81c8198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.957+0000 7f81c77fe640 1 --2- 172.21.15.98:0/3894629545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f81c81034c0 0x7f81c819f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.958+0000 7f81be7fc640 1 -- 172.21.15.98:0/3894629545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f81c81034c0 msgr2=0x7f81c819f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.958+0000 7f81be7fc640 1 --2- 172.21.15.98:0/3894629545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f81c81034c0 0x7f81c819f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.958+0000 7f81be7fc640 1 -- 172.21.15.98:0/3894629545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c80ffb30 msgr2=0x7f81c8197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.958+0000 7f81be7fc640 1 --2- 172.21.15.98:0/3894629545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c80ffb30 0x7f81c8197c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.958+0000 7f81be7fc640 1 -- 172.21.15.98:0/3894629545 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c819f860 con 0x7f81c8102b00 2024-05-09T08:50:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.958+0000 7f81be7fc640 1 --2- 172.21.15.98:0/3894629545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81c8102b00 0x7f81c8198190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f81b000e9f0 tx=0x7f81b000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.958+0000 7f81c4ff9640 1 -- 172.21.15.98:0/3894629545 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b000ccc0 con 0x7f81c8102b00 2024-05-09T08:50:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.958+0000 7f81cd6c1640 1 -- 172.21.15.98:0/3894629545 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81c819fb50 con 0x7f81c8102b00 2024-05-09T08:50:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.958+0000 7f81c4ff9640 1 -- 172.21.15.98:0/3894629545 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81b0005020 con 0x7f81c8102b00 2024-05-09T08:50:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.958+0000 7f81c4ff9640 1 -- 172.21.15.98:0/3894629545 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b00049e0 con 0x7f81c8102b00 2024-05-09T08:50:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.958+0000 7f81cd6c1640 1 -- 172.21.15.98:0/3894629545 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81c81a00a0 con 0x7f81c8102b00 2024-05-09T08:50:41.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.959+0000 7f81cd6c1640 1 -- 172.21.15.98:0/3894629545 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81c8108ca0 con 0x7f81c8102b00 2024-05-09T08:50:41.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.961+0000 7f81c4ff9640 1 -- 172.21.15.98:0/3894629545 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f81b0005560 con 0x7f81c8102b00 2024-05-09T08:50:41.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.961+0000 7f81c4ff9640 1 --2- 172.21.15.98:0/3894629545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f81b40a2d90 0x7f81b40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:41.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.961+0000 7f81c6ffd640 1 --2- 172.21.15.98:0/3894629545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f81b40a2d90 0x7f81b40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:41.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.961+0000 7f81c4ff9640 1 -- 172.21.15.98:0/3894629545 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f81b00cf980 con 0x7f81c8102b00 2024-05-09T08:50:41.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.962+0000 7f81c6ffd640 1 --2- 172.21.15.98:0/3894629545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f81b40a2d90 0x7f81b40a5250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f81b8006010 tx=0x7f81b8000bc0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:41.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:41.964+0000 7f81c4ff9640 1 -- 172.21.15.98:0/3894629545 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f81b00051d0 con 0x7f81c8102b00 2024-05-09T08:50:42.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.174+0000 7f81cd6c1640 1 -- 172.21.15.98:0/3894629545 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f81c802d3c0 con 0x7f81c8102b00 2024-05-09T08:50:42.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.175+0000 7f81c4ff9640 1 -- 172.21.15.98:0/3894629545 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f81b0084e20 con 0x7f81c8102b00 2024-05-09T08:50:42.174 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:50:42.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.177+0000 7f81cd6c1640 1 -- 172.21.15.98:0/3894629545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f81b40a2d90 msgr2=0x7f81b40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.177+0000 7f81cd6c1640 1 --2- 172.21.15.98:0/3894629545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f81b40a2d90 0x7f81b40a5250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f81b8006010 tx=0x7f81b8000bc0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.177+0000 7f81cd6c1640 1 -- 172.21.15.98:0/3894629545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81c8102b00 msgr2=0x7f81c8198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.177+0000 7f81cd6c1640 1 --2- 172.21.15.98:0/3894629545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81c8102b00 0x7f81c8198190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f81b000e9f0 tx=0x7f81b000eeb0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.177+0000 7f81cd6c1640 1 -- 172.21.15.98:0/3894629545 shutdown_connections 2024-05-09T08:50:42.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.177+0000 7f81cd6c1640 1 --2- 172.21.15.98:0/3894629545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f81b40a2d90 0x7f81b40a5250 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.177+0000 7f81cd6c1640 1 --2- 172.21.15.98:0/3894629545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f81c81034c0 0x7f81c819f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.177+0000 7f81cd6c1640 1 --2- 172.21.15.98:0/3894629545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81c8102b00 0x7f81c8198190 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.177+0000 7f81cd6c1640 1 --2- 172.21.15.98:0/3894629545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c80ffb30 0x7f81c8197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.177+0000 7f81cd6c1640 1 -- 172.21.15.98:0/3894629545 >> 172.21.15.98:0/3894629545 conn(0x7f81c80fb460 msgr2=0x7f81c8105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:42.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.177+0000 7f81cd6c1640 1 -- 172.21.15.98:0/3894629545 shutdown_connections 2024-05-09T08:50:42.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.177+0000 7f81cd6c1640 1 -- 172.21.15.98:0/3894629545 wait complete. 2024-05-09T08:50:42.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:50:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.297+0000 7f062fc12640 1 -- 172.21.15.98:0/88848407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628101960 msgr2=0x7f0628101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.297+0000 7f062fc12640 1 --2- 172.21.15.98:0/88848407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628101960 0x7f0628101de0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0618009900 tx=0x7f061802a410 comp rx=0 tx=0).stop 2024-05-09T08:50:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.297+0000 7f062fc12640 1 -- 172.21.15.98:0/88848407 shutdown_connections 2024-05-09T08:50:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.297+0000 7f062fc12640 1 --2- 172.21.15.98:0/88848407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0628102320 0x7f0628108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.297+0000 7f062fc12640 1 --2- 172.21.15.98:0/88848407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628101960 0x7f0628101de0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.297+0000 7f062fc12640 1 --2- 172.21.15.98:0/88848407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0628100770 0x7f0628100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.297+0000 7f062fc12640 1 -- 172.21.15.98:0/88848407 >> 172.21.15.98:0/88848407 conn(0x7f06280fbce0 msgr2=0x7f06280fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.297+0000 7f062fc12640 1 -- 172.21.15.98:0/88848407 shutdown_connections 2024-05-09T08:50:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.297+0000 7f062fc12640 1 -- 172.21.15.98:0/88848407 wait complete. 2024-05-09T08:50:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.298+0000 7f062fc12640 1 Processor -- start 2024-05-09T08:50:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.298+0000 7f062fc12640 1 -- start start 2024-05-09T08:50:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.298+0000 7f062fc12640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0628100770 0x7f0628104700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.298+0000 7f062fc12640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628101960 0x7f0628102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.298+0000 7f062fc12640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0628102320 0x7f0628103280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.298+0000 7f062fc12640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f062810c3d0 con 0x7f0628101960 2024-05-09T08:50:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.298+0000 7f062fc12640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f062810c250 con 0x7f0628102320 2024-05-09T08:50:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.298+0000 7f062fc12640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f062810c550 con 0x7f0628100770 2024-05-09T08:50:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.298+0000 7f062d1ad640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628101960 0x7f0628102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.298+0000 7f062d9ae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0628100770 0x7f0628104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.298+0000 7f062d1ad640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628101960 0x7f0628102d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43124/0 (socket says 172.21.15.98:43124) 2024-05-09T08:50:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.298+0000 7f062d1ad640 1 -- 172.21.15.98:0/1474333009 learned_addr learned my addr 172.21.15.98:0/1474333009 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.298+0000 7f062e1af640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0628102320 0x7f0628103280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.299+0000 7f062d9ae640 1 -- 172.21.15.98:0/1474333009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0628102320 msgr2=0x7f0628103280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.299+0000 7f062d9ae640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0628102320 0x7f0628103280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.299+0000 7f062d9ae640 1 -- 172.21.15.98:0/1474333009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628101960 msgr2=0x7f0628102d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.299+0000 7f062d9ae640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628101960 0x7f0628102d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.299+0000 7f062d9ae640 1 -- 172.21.15.98:0/1474333009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0628103b10 con 0x7f0628100770 2024-05-09T08:50:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.299+0000 7f062e1af640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0628102320 0x7f0628103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:50:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.299+0000 7f062d1ad640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628101960 0x7f0628102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:50:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.299+0000 7f062d9ae640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0628100770 0x7f0628104700 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f06281017c0 tx=0x7f062400dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.299+0000 7f061effd640 1 -- 172.21.15.98:0/1474333009 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0624017070 con 0x7f0628100770 2024-05-09T08:50:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.299+0000 7f062fc12640 1 -- 172.21.15.98:0/1474333009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06281a1cf0 con 0x7f0628100770 2024-05-09T08:50:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.299+0000 7f061effd640 1 -- 172.21.15.98:0/1474333009 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0624009c30 con 0x7f0628100770 2024-05-09T08:50:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.299+0000 7f061effd640 1 -- 172.21.15.98:0/1474333009 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062400f620 con 0x7f0628100770 2024-05-09T08:50:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.299+0000 7f062fc12640 1 -- 172.21.15.98:0/1474333009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06281a21b0 con 0x7f0628100770 2024-05-09T08:50:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.300+0000 7f062fc12640 1 -- 172.21.15.98:0/1474333009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06281a23e0 con 0x7f0628100770 2024-05-09T08:50:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.301+0000 7f061effd640 1 -- 172.21.15.98:0/1474333009 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f062401a070 con 0x7f0628100770 2024-05-09T08:50:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.302+0000 7f061effd640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f06140a2d90 0x7f06140a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.302+0000 7f062d1ad640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f06140a2d90 0x7f06140a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.302+0000 7f061effd640 1 -- 172.21.15.98:0/1474333009 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f06240cf370 con 0x7f0628100770 2024-05-09T08:50:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.303+0000 7f062d1ad640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f06140a2d90 0x7f06140a5250 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0618031440 tx=0x7f06180315b0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.307+0000 7f061effd640 1 -- 172.21.15.98:0/1474333009 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f062409b410 con 0x7f0628100770 2024-05-09T08:50:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.495+0000 7f062fc12640 1 -- 172.21.15.98:0/1474333009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f06281a2770 con 0x7f0628100770 2024-05-09T08:50:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.496+0000 7f061effd640 1 -- 172.21.15.98:0/1474333009 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f062409f710 con 0x7f0628100770 2024-05-09T08:50:42.495 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-09T08:50:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.498+0000 7f062fc12640 1 -- 172.21.15.98:0/1474333009 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f06140a2d90 msgr2=0x7f06140a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.498+0000 7f062fc12640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f06140a2d90 0x7f06140a5250 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0618031440 tx=0x7f06180315b0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.498+0000 7f062fc12640 1 -- 172.21.15.98:0/1474333009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0628100770 msgr2=0x7f0628104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.498+0000 7f062fc12640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0628100770 0x7f0628104700 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f06281017c0 tx=0x7f062400dde0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.498+0000 7f062fc12640 1 -- 172.21.15.98:0/1474333009 shutdown_connections 2024-05-09T08:50:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.498+0000 7f062fc12640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f06140a2d90 0x7f06140a5250 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.498+0000 7f062fc12640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0628102320 0x7f0628103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.498+0000 7f062fc12640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0628101960 0x7f0628102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.498+0000 7f062fc12640 1 --2- 172.21.15.98:0/1474333009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0628100770 0x7f0628104700 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.498+0000 7f062fc12640 1 -- 172.21.15.98:0/1474333009 >> 172.21.15.98:0/1474333009 conn(0x7f06280fbce0 msgr2=0x7f06280fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.499+0000 7f062fc12640 1 -- 172.21.15.98:0/1474333009 shutdown_connections 2024-05-09T08:50:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.499+0000 7f062fc12640 1 -- 172.21.15.98:0/1474333009 wait complete. 2024-05-09T08:50:42.508 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:50:42.508 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-05-09T08:50:42.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/36091111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:42.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.509 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/309169120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:42.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:50:42.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:50:42.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:42 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3894629545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:42.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:50:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.621+0000 7f68fa65d640 1 -- 172.21.15.98:0/2709549175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102a70 msgr2=0x7f68f4102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.621+0000 7f68fa65d640 1 --2- 172.21.15.98:0/2709549175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102a70 0x7f68f4102ef0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f68dc009900 tx=0x7f68dc02a410 comp rx=0 tx=0).stop 2024-05-09T08:50:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.621+0000 7f68fa65d640 1 -- 172.21.15.98:0/2709549175 shutdown_connections 2024-05-09T08:50:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.621+0000 7f68fa65d640 1 --2- 172.21.15.98:0/2709549175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68f4103430 0x7f68f4109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.621+0000 7f68fa65d640 1 --2- 172.21.15.98:0/2709549175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102a70 0x7f68f4102ef0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.621+0000 7f68fa65d640 1 --2- 172.21.15.98:0/2709549175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68f4101880 0x7f68f4101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.621+0000 7f68fa65d640 1 -- 172.21.15.98:0/2709549175 >> 172.21.15.98:0/2709549175 conn(0x7f68f40fce10 msgr2=0x7f68f40ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.621+0000 7f68fa65d640 1 -- 172.21.15.98:0/2709549175 shutdown_connections 2024-05-09T08:50:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.621+0000 7f68fa65d640 1 -- 172.21.15.98:0/2709549175 wait complete. 2024-05-09T08:50:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.621+0000 7f68fa65d640 1 Processor -- start 2024-05-09T08:50:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.621+0000 7f68fa65d640 1 -- start start 2024-05-09T08:50:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68fa65d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68f4101880 0x7f68f4197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68fa65d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102a70 0x7f68f41980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68fa65d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68f4103430 0x7f68f419f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68fa65d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68f410d500 con 0x7f68f4102a70 2024-05-09T08:50:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68fa65d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f68f410d380 con 0x7f68f4103430 2024-05-09T08:50:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68fa65d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f68f410d680 con 0x7f68f4101880 2024-05-09T08:50:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68f37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102a70 0x7f68f41980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68f3fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68f4101880 0x7f68f4197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:42.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68f37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102a70 0x7f68f41980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43156/0 (socket says 172.21.15.98:43156) 2024-05-09T08:50:42.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68f37fe640 1 -- 172.21.15.98:0/2244295735 learned_addr learned my addr 172.21.15.98:0/2244295735 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:42.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68f8bfa640 1 --2- 172.21.15.98:0/2244295735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68f4103430 0x7f68f419f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:42.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68f3fff640 1 -- 172.21.15.98:0/2244295735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68f4103430 msgr2=0x7f68f419f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68f3fff640 1 --2- 172.21.15.98:0/2244295735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68f4103430 0x7f68f419f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68f3fff640 1 -- 172.21.15.98:0/2244295735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102a70 msgr2=0x7f68f41980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68f3fff640 1 --2- 172.21.15.98:0/2244295735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102a70 0x7f68f41980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68f3fff640 1 -- 172.21.15.98:0/2244295735 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f419f7c0 con 0x7f68f4101880 2024-05-09T08:50:42.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.622+0000 7f68f3fff640 1 --2- 172.21.15.98:0/2244295735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68f4101880 0x7f68f4197ba0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f68e400e9c0 tx=0x7f68e400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:42.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.623+0000 7f68f17fa640 1 -- 172.21.15.98:0/2244295735 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e400ccc0 con 0x7f68f4101880 2024-05-09T08:50:42.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.623+0000 7f68f17fa640 1 -- 172.21.15.98:0/2244295735 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68e40035f0 con 0x7f68f4101880 2024-05-09T08:50:42.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.623+0000 7f68fa65d640 1 -- 172.21.15.98:0/2244295735 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f419fab0 con 0x7f68f4101880 2024-05-09T08:50:42.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.623+0000 7f68f17fa640 1 -- 172.21.15.98:0/2244295735 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e400ce60 con 0x7f68f4101880 2024-05-09T08:50:42.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.623+0000 7f68fa65d640 1 -- 172.21.15.98:0/2244295735 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f419ff80 con 0x7f68f4101880 2024-05-09T08:50:42.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.624+0000 7f68fa65d640 1 -- 172.21.15.98:0/2244295735 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f4108c90 con 0x7f68f4101880 2024-05-09T08:50:42.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.625+0000 7f68f17fa640 1 -- 172.21.15.98:0/2244295735 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f68e40037a0 con 0x7f68f4101880 2024-05-09T08:50:42.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.626+0000 7f68f17fa640 1 --2- 172.21.15.98:0/2244295735 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f68e00a2d90 0x7f68e00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:42.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.626+0000 7f68f37fe640 1 --2- 172.21.15.98:0/2244295735 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f68e00a2d90 0x7f68e00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:42.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.626+0000 7f68f17fa640 1 -- 172.21.15.98:0/2244295735 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f68e40d0230 con 0x7f68f4101880 2024-05-09T08:50:42.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.626+0000 7f68f37fe640 1 --2- 172.21.15.98:0/2244295735 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f68e00a2d90 0x7f68e00a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f68dc031440 tx=0x7f68dc0315b0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:42.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.629+0000 7f68f17fa640 1 -- 172.21.15.98:0/2244295735 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f68e409c920 con 0x7f68f4101880 2024-05-09T08:50:42.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:50:42.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-05-09T08:50:42.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/36091111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:42.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/309169120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:42.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:50:42.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:50:42.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:42 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3894629545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:42.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:50:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-05-09T08:50:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/36091111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/309169120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:42.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:42.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:50:42.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:50:42.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:50:42.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:42 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3894629545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:42.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.815+0000 7f68fa65d640 1 -- 172.21.15.98:0/2244295735 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f68f402d1c0 con 0x7f68f4101880 2024-05-09T08:50:42.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.815+0000 7f68f17fa640 1 -- 172.21.15.98:0/2244295735 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f68e409c220 con 0x7f68f4101880 2024-05-09T08:50:42.815 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:50:42.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.817+0000 7f68fa65d640 1 -- 172.21.15.98:0/2244295735 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f68e00a2d90 msgr2=0x7f68e00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.817+0000 7f68fa65d640 1 --2- 172.21.15.98:0/2244295735 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f68e00a2d90 0x7f68e00a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f68dc031440 tx=0x7f68dc0315b0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.817+0000 7f68fa65d640 1 -- 172.21.15.98:0/2244295735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68f4101880 msgr2=0x7f68f4197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.817+0000 7f68fa65d640 1 --2- 172.21.15.98:0/2244295735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68f4101880 0x7f68f4197ba0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f68e400e9c0 tx=0x7f68e400ee80 comp rx=0 tx=0).stop 2024-05-09T08:50:42.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.817+0000 7f68fa65d640 1 -- 172.21.15.98:0/2244295735 shutdown_connections 2024-05-09T08:50:42.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.817+0000 7f68fa65d640 1 --2- 172.21.15.98:0/2244295735 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f68e00a2d90 0x7f68e00a5250 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.817+0000 7f68fa65d640 1 --2- 172.21.15.98:0/2244295735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68f4103430 0x7f68f419f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.817+0000 7f68fa65d640 1 --2- 172.21.15.98:0/2244295735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f4102a70 0x7f68f41980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.817+0000 7f68fa65d640 1 --2- 172.21.15.98:0/2244295735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68f4101880 0x7f68f4197ba0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.817+0000 7f68fa65d640 1 -- 172.21.15.98:0/2244295735 >> 172.21.15.98:0/2244295735 conn(0x7f68f40fce10 msgr2=0x7f68f4105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:42.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.817+0000 7f68fa65d640 1 -- 172.21.15.98:0/2244295735 shutdown_connections 2024-05-09T08:50:42.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.817+0000 7f68fa65d640 1 -- 172.21.15.98:0/2244295735 wait complete. 2024-05-09T08:50:42.828 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-05-09T08:50:42.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:50:42.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.945+0000 7f28489a9640 1 -- 172.21.15.98:0/3616908086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2844102ac0 msgr2=0x7f2844102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.945+0000 7f28489a9640 1 --2- 172.21.15.98:0/3616908086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2844102ac0 0x7f2844102ea0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2834009ca0 tx=0x7f283402a9e0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.946+0000 7f28489a9640 1 -- 172.21.15.98:0/3616908086 shutdown_connections 2024-05-09T08:50:42.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.946+0000 7f28489a9640 1 --2- 172.21.15.98:0/3616908086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f284410be00 0x7f284410e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.946+0000 7f28489a9640 1 --2- 172.21.15.98:0/3616908086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28441033e0 0x7f284410b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.946+0000 7f28489a9640 1 --2- 172.21.15.98:0/3616908086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2844102ac0 0x7f2844102ea0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.946+0000 7f28489a9640 1 -- 172.21.15.98:0/3616908086 >> 172.21.15.98:0/3616908086 conn(0x7f2844077830 msgr2=0x7f2844077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:42.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.946+0000 7f28489a9640 1 -- 172.21.15.98:0/3616908086 shutdown_connections 2024-05-09T08:50:42.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.946+0000 7f28489a9640 1 -- 172.21.15.98:0/3616908086 wait complete. 2024-05-09T08:50:42.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.946+0000 7f28489a9640 1 Processor -- start 2024-05-09T08:50:42.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.946+0000 7f28489a9640 1 -- start start 2024-05-09T08:50:42.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.946+0000 7f28489a9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2844102ac0 0x7f2844100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:42.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.946+0000 7f28489a9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28441033e0 0x7f2844100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:42.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f28489a9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f284410be00 0x7f2844101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:42.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f28489a9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2844111c00 con 0x7f284410be00 2024-05-09T08:50:42.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f28489a9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2844111a80 con 0x7f28441033e0 2024-05-09T08:50:42.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f28489a9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2844111d80 con 0x7f2844102ac0 2024-05-09T08:50:42.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f2842d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f284410be00 0x7f2844101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:42.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f2842d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f284410be00 0x7f2844101190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43182/0 (socket says 172.21.15.98:43182) 2024-05-09T08:50:42.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f2842d9d640 1 -- 172.21.15.98:0/2625483248 learned_addr learned my addr 172.21.15.98:0/2625483248 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:42.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f284259c640 1 --2- 172.21.15.98:0/2625483248 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2844102ac0 0x7f2844100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:42.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f2841d9b640 1 --2- 172.21.15.98:0/2625483248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28441033e0 0x7f2844100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:42.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f2842d9d640 1 -- 172.21.15.98:0/2625483248 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2844102ac0 msgr2=0x7f2844100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f2842d9d640 1 --2- 172.21.15.98:0/2625483248 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2844102ac0 0x7f2844100710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f2842d9d640 1 -- 172.21.15.98:0/2625483248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28441033e0 msgr2=0x7f2844100c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:42.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f2842d9d640 1 --2- 172.21.15.98:0/2625483248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28441033e0 0x7f2844100c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:42.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f2842d9d640 1 -- 172.21.15.98:0/2625483248 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28441a6600 con 0x7f284410be00 2024-05-09T08:50:42.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f2842d9d640 1 --2- 172.21.15.98:0/2625483248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f284410be00 0x7f2844101190 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f283c00e970 tx=0x7f283c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:42.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f283b7fe640 1 -- 172.21.15.98:0/2625483248 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283c00ccc0 con 0x7f284410be00 2024-05-09T08:50:42.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f283b7fe640 1 -- 172.21.15.98:0/2625483248 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f283c00ce60 con 0x7f284410be00 2024-05-09T08:50:42.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f28489a9640 1 -- 172.21.15.98:0/2625483248 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28441a68f0 con 0x7f284410be00 2024-05-09T08:50:42.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f283b7fe640 1 -- 172.21.15.98:0/2625483248 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283c010490 con 0x7f284410be00 2024-05-09T08:50:42.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.947+0000 7f28489a9640 1 -- 172.21.15.98:0/2625483248 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28441a6df0 con 0x7f284410be00 2024-05-09T08:50:42.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.948+0000 7f28489a9640 1 -- 172.21.15.98:0/2625483248 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2844068650 con 0x7f284410be00 2024-05-09T08:50:42.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.949+0000 7f283b7fe640 1 -- 172.21.15.98:0/2625483248 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f283c005020 con 0x7f284410be00 2024-05-09T08:50:42.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.950+0000 7f283b7fe640 1 --2- 172.21.15.98:0/2625483248 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f28300a2d90 0x7f28300a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:42.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.950+0000 7f284259c640 1 --2- 172.21.15.98:0/2625483248 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f28300a2d90 0x7f28300a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:42.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.950+0000 7f283b7fe640 1 -- 172.21.15.98:0/2625483248 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f283c0cfe80 con 0x7f284410be00 2024-05-09T08:50:42.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.950+0000 7f284259c640 1 --2- 172.21.15.98:0/2625483248 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f28300a2d90 0x7f28300a5250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2834009430 tx=0x7f283403a040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:42.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:42.954+0000 7f283b7fe640 1 -- 172.21.15.98:0/2625483248 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f283c09c570 con 0x7f284410be00 2024-05-09T08:50:43.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.179+0000 7f28489a9640 1 -- 172.21.15.98:0/2625483248 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f284402d1c0 con 0x7f284410be00 2024-05-09T08:50:43.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.180+0000 7f283b7fe640 1 -- 172.21.15.98:0/2625483248 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11940 (secure 0 0 0) 0x7f283c09be70 con 0x7f284410be00 2024-05-09T08:50:43.179 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:43.180 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:50:37.185367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T08:50:43.180 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-05-09T08:50:43.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.182+0000 7f28489a9640 1 -- 172.21.15.98:0/2625483248 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f28300a2d90 msgr2=0x7f28300a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:43.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.182+0000 7f28489a9640 1 --2- 172.21.15.98:0/2625483248 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f28300a2d90 0x7f28300a5250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2834009430 tx=0x7f283403a040 comp rx=0 tx=0).stop 2024-05-09T08:50:43.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.182+0000 7f28489a9640 1 -- 172.21.15.98:0/2625483248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f284410be00 msgr2=0x7f2844101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:43.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.182+0000 7f28489a9640 1 --2- 172.21.15.98:0/2625483248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f284410be00 0x7f2844101190 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f283c00e970 tx=0x7f283c00ed70 comp rx=0 tx=0).stop 2024-05-09T08:50:43.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.182+0000 7f28489a9640 1 -- 172.21.15.98:0/2625483248 shutdown_connections 2024-05-09T08:50:43.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.182+0000 7f28489a9640 1 --2- 172.21.15.98:0/2625483248 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f28300a2d90 0x7f28300a5250 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.182+0000 7f28489a9640 1 --2- 172.21.15.98:0/2625483248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f284410be00 0x7f2844101190 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.182+0000 7f28489a9640 1 --2- 172.21.15.98:0/2625483248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28441033e0 0x7f2844100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.182+0000 7f28489a9640 1 --2- 172.21.15.98:0/2625483248 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2844102ac0 0x7f2844100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.182+0000 7f28489a9640 1 -- 172.21.15.98:0/2625483248 >> 172.21.15.98:0/2625483248 conn(0x7f2844077830 msgr2=0x7f28440fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:43.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.182+0000 7f28489a9640 1 -- 172.21.15.98:0/2625483248 shutdown_connections 2024-05-09T08:50:43.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.182+0000 7f28489a9640 1 -- 172.21.15.98:0/2625483248 wait complete. 2024-05-09T08:50:43.193 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-05-09T08:50:43.193 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-05-09T08:50:43.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:50:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.305+0000 7f3266341640 1 -- 172.21.15.98:0/96615954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3260106740 msgr2=0x7f3260106b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:43.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.305+0000 7f3266341640 1 --2- 172.21.15.98:0/96615954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3260106740 0x7f3260106b20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f3250009930 tx=0x7f325002a420 comp rx=0 tx=0).stop 2024-05-09T08:50:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.305+0000 7f3266341640 1 -- 172.21.15.98:0/96615954 shutdown_connections 2024-05-09T08:50:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.305+0000 7f3266341640 1 --2- 172.21.15.98:0/96615954 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3260101140 0x7f326010d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.305+0000 7f3266341640 1 --2- 172.21.15.98:0/96615954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3260100780 0x7f3260100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.305+0000 7f3266341640 1 --2- 172.21.15.98:0/96615954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3260106740 0x7f3260106b20 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.305+0000 7f3266341640 1 -- 172.21.15.98:0/96615954 >> 172.21.15.98:0/96615954 conn(0x7f3260077890 msgr2=0x7f3260077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.305+0000 7f3266341640 1 -- 172.21.15.98:0/96615954 shutdown_connections 2024-05-09T08:50:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.305+0000 7f3266341640 1 -- 172.21.15.98:0/96615954 wait complete. 2024-05-09T08:50:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.305+0000 7f3266341640 1 Processor -- start 2024-05-09T08:50:43.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.305+0000 7f3266341640 1 -- start start 2024-05-09T08:50:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f3266341640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3260100780 0x7f3260199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f3266341640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3260101140 0x7f3260199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f3266341640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3260106740 0x7f32601a4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f3266341640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3260110ba0 con 0x7f3260101140 2024-05-09T08:50:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f3266341640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3260110a20 con 0x7f3260106740 2024-05-09T08:50:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f3266341640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3260110d20 con 0x7f3260100780 2024-05-09T08:50:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f325f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3260101140 0x7f3260199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f325f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3260101140 0x7f3260199540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43196/0 (socket says 172.21.15.98:43196) 2024-05-09T08:50:43.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f325f7fe640 1 -- 172.21.15.98:0/2782591616 learned_addr learned my addr 172.21.15.98:0/2782591616 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f325ffff640 1 --2- 172.21.15.98:0/2782591616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3260100780 0x7f3260199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f32648de640 1 --2- 172.21.15.98:0/2782591616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3260106740 0x7f32601a4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f325f7fe640 1 -- 172.21.15.98:0/2782591616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3260100780 msgr2=0x7f3260199000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f325f7fe640 1 --2- 172.21.15.98:0/2782591616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3260100780 0x7f3260199000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f325f7fe640 1 -- 172.21.15.98:0/2782591616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3260106740 msgr2=0x7f32601a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f325f7fe640 1 --2- 172.21.15.98:0/2782591616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3260106740 0x7f32601a4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f325f7fe640 1 -- 172.21.15.98:0/2782591616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32601a45c0 con 0x7f3260101140 2024-05-09T08:50:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f325f7fe640 1 --2- 172.21.15.98:0/2782591616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3260101140 0x7f3260199540 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f325400e9f0 tx=0x7f325400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f325d7fa640 1 -- 172.21.15.98:0/2782591616 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325400cd80 con 0x7f3260101140 2024-05-09T08:50:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f325d7fa640 1 -- 172.21.15.98:0/2782591616 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3254005500 con 0x7f3260101140 2024-05-09T08:50:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.306+0000 7f3266341640 1 -- 172.21.15.98:0/2782591616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32601a4790 con 0x7f3260101140 2024-05-09T08:50:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.307+0000 7f325d7fa640 1 -- 172.21.15.98:0/2782591616 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254010690 con 0x7f3260101140 2024-05-09T08:50:43.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.307+0000 7f3266341640 1 -- 172.21.15.98:0/2782591616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32601a4cb0 con 0x7f3260101140 2024-05-09T08:50:43.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.307+0000 7f3266341640 1 -- 172.21.15.98:0/2782591616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3260068aa0 con 0x7f3260101140 2024-05-09T08:50:43.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.310+0000 7f325d7fa640 1 -- 172.21.15.98:0/2782591616 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f32540046c0 con 0x7f3260101140 2024-05-09T08:50:43.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.310+0000 7f325d7fa640 1 --2- 172.21.15.98:0/2782591616 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f32480a2ea0 0x7f32480a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:43.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.310+0000 7f325ffff640 1 --2- 172.21.15.98:0/2782591616 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f32480a2ea0 0x7f32480a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:43.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.310+0000 7f325d7fa640 1 -- 172.21.15.98:0/2782591616 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f32540d0000 con 0x7f3260101140 2024-05-09T08:50:43.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.311+0000 7f325ffff640 1 --2- 172.21.15.98:0/2782591616 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f32480a2ea0 0x7f32480a5360 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3250006010 tx=0x7f3250000bc0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:43.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.313+0000 7f325d7fa640 1 -- 172.21.15.98:0/2782591616 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3254053740 con 0x7f3260101140 2024-05-09T08:50:43.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.486+0000 7f3266341640 1 -- 172.21.15.98:0/2782591616 --> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3260108d10 con 0x7f32480a2ea0 2024-05-09T08:50:43.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.496+0000 7f325d7fa640 1 -- 172.21.15.98:0/2782591616 <== mgr.15555 v2:172.21.15.98:6800/4136638623 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339730 (secure 0 0 0) 0x7f3250004270 con 0x7f32480a2ea0 2024-05-09T08:50:43.499 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:43.501 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-09T08:50:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.499+0000 7f3266341640 1 -- 172.21.15.98:0/2782591616 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f32480a2ea0 msgr2=0x7f32480a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.499+0000 7f3266341640 1 --2- 172.21.15.98:0/2782591616 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f32480a2ea0 0x7f32480a5360 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3250006010 tx=0x7f3250000bc0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.499+0000 7f3266341640 1 -- 172.21.15.98:0/2782591616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3260101140 msgr2=0x7f3260199540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.499+0000 7f3266341640 1 --2- 172.21.15.98:0/2782591616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3260101140 0x7f3260199540 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f325400e9f0 tx=0x7f325400edf0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.500+0000 7f3266341640 1 -- 172.21.15.98:0/2782591616 shutdown_connections 2024-05-09T08:50:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.500+0000 7f3266341640 1 --2- 172.21.15.98:0/2782591616 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f32480a2ea0 0x7f32480a5360 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.500+0000 7f3266341640 1 --2- 172.21.15.98:0/2782591616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3260106740 0x7f32601a4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.500+0000 7f3266341640 1 --2- 172.21.15.98:0/2782591616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3260101140 0x7f3260199540 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.500+0000 7f3266341640 1 --2- 172.21.15.98:0/2782591616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3260100780 0x7f3260199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.500+0000 7f3266341640 1 -- 172.21.15.98:0/2782591616 >> 172.21.15.98:0/2782591616 conn(0x7f3260077890 msgr2=0x7f3260103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.500+0000 7f3266341640 1 -- 172.21.15.98:0/2782591616 shutdown_connections 2024-05-09T08:50:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.500+0000 7f3266341640 1 -- 172.21.15.98:0/2782591616 wait complete. 2024-05-09T08:50:43.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-05-09T08:50:43.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: Unable to set osd_memory_target on smithi142 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-05-09T08:50:43.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-09T08:50:43.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-09T08:50:43.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-05-09T08:50:43.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: Unable to set osd_memory_target on smithi098 to 184766668: error parsing value: Value '184766668' is below minimum 939524096 2024-05-09T08:50:43.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: Updating smithi098:/etc/ceph/ceph.conf 2024-05-09T08:50:43.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-09T08:50:43.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: Updating smithi142:/etc/ceph/ceph.conf 2024-05-09T08:50:43.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: Updating smithi105:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:50:43.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: Updating smithi098:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:50:43.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: Updating smithi142:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:50:43.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1474333009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:43.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2244295735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:43.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:50:43.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:50:43.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:50:43.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:43 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2625483248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:43.513 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-05-09T08:50:42.510505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190289,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1488,"num_read_kb":912,"num_write":557,"num_write_kb":4408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1180921,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":942,"ondisk_log_size":942,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353840,"kb_used_data":12968,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124587984,"statfs":{"total":1151940427776,"available":1151578095616,"internally_reserved":0,"allocated":13279232,"data_stored":9947273,"data_compressed":537339,"data_compressed_allocated":724992,"data_compressed_original":4300800,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.987472"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665191+0000","last_change":"2024-05-09T08:48:18.373685+0000","last_active":"2024-05-09T08:50:38.665191+0000","last_peered":"2024-05-09T08:50:38.665191+0000","last_clean":"2024-05-09T08:50:38.665191+0000","last_became_active":"2024-05-09T08:48:18.373305+0000","last_became_peered":"2024-05-09T08:48:18.373305+0000","last_unstale":"2024-05-09T08:50:38.665191+0000","last_undegraded":"2024-05-09T08:50:38.665191+0000","last_fullsized":"2024-05-09T08:50:38.665191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:30:45.261706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740969+0000","last_change":"2024-05-09T08:48:18.373152+0000","last_active":"2024-05-09T08:50:38.740969+0000","last_peered":"2024-05-09T08:50:38.740969+0000","last_clean":"2024-05-09T08:50:38.740969+0000","last_became_active":"2024-05-09T08:48:18.372015+0000","last_became_peered":"2024-05-09T08:48:18.372015+0000","last_unstale":"2024-05-09T08:50:38.740969+0000","last_undegraded":"2024-05-09T08:50:38.740969+0000","last_fullsized":"2024-05-09T08:50:38.740969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:47.512778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740875+0000","last_change":"2024-05-09T08:48:18.370177+0000","last_active":"2024-05-09T08:50:38.740875+0000","last_peered":"2024-05-09T08:50:38.740875+0000","last_clean":"2024-05-09T08:50:38.740875+0000","last_became_active":"2024-05-09T08:48:18.369842+0000","last_became_peered":"2024-05-09T08:48:18.369842+0000","last_unstale":"2024-05-09T08:50:38.740875+0000","last_undegraded":"2024-05-09T08:50:38.740875+0000","last_fullsized":"2024-05-09T08:50:38.740875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:00:24.015500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738316+0000","last_change":"2024-05-09T08:48:18.375364+0000","last_active":"2024-05-09T08:50:38.738316+0000","last_peered":"2024-05-09T08:50:38.738316+0000","last_clean":"2024-05-09T08:50:38.738316+0000","last_became_active":"2024-05-09T08:48:18.375030+0000","last_became_peered":"2024-05-09T08:48:18.375030+0000","last_unstale":"2024-05-09T08:50:38.738316+0000","last_undegraded":"2024-05-09T08:50:38.738316+0000","last_fullsized":"2024-05-09T08:50:38.738316+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:43:57.684123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498390+0000","last_change":"2024-05-09T08:48:18.374548+0000","last_active":"2024-05-09T08:50:38.498390+0000","last_peered":"2024-05-09T08:50:38.498390+0000","last_clean":"2024-05-09T08:50:38.498390+0000","last_became_active":"2024-05-09T08:48:18.374190+0000","last_became_peered":"2024-05-09T08:48:18.374190+0000","last_unstale":"2024-05-09T08:50:38.498390+0000","last_undegraded":"2024-05-09T08:50:38.498390+0000","last_fullsized":"2024-05-09T08:50:38.498390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:43:48.456839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.501744+0000","last_change":"2024-05-09T08:48:18.371234+0000","last_active":"2024-05-09T08:50:38.501744+0000","last_peered":"2024-05-09T08:50:38.501744+0000","last_clean":"2024-05-09T08:50:38.501744+0000","last_became_active":"2024-05-09T08:48:18.370218+0000","last_became_peered":"2024-05-09T08:48:18.370218+0000","last_unstale":"2024-05-09T08:50:38.501744+0000","last_undegraded":"2024-05-09T08:50:38.501744+0000","last_fullsized":"2024-05-09T08:50:38.501744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:02:09.923146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739889+0000","last_change":"2024-05-09T08:49:18.366083+0000","last_active":"2024-05-09T08:50:38.739889+0000","last_peered":"2024-05-09T08:50:38.739889+0000","last_clean":"2024-05-09T08:50:38.739889+0000","last_became_active":"2024-05-09T08:49:18.365755+0000","last_became_peered":"2024-05-09T08:49:18.365755+0000","last_unstale":"2024-05-09T08:50:38.739889+0000","last_undegraded":"2024-05-09T08:50:38.739889+0000","last_fullsized":"2024-05-09T08:50:38.739889+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:40:18.052955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.596412+0000","last_change":"2024-05-09T08:49:18.488640+0000","last_active":"2024-05-09T08:50:38.596412+0000","last_peered":"2024-05-09T08:50:38.596412+0000","last_clean":"2024-05-09T08:50:38.596412+0000","last_became_active":"2024-05-09T08:49:18.488404+0000","last_became_peered":"2024-05-09T08:49:18.488404+0000","last_unstale":"2024-05-09T08:50:38.596412+0000","last_undegraded":"2024-05-09T08:50:38.596412+0000","last_fullsized":"2024-05-09T08:50:38.596412+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:47.317453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743849+0000","last_change":"2024-05-09T08:49:18.368485+0000","last_active":"2024-05-09T08:50:38.743849+0000","last_peered":"2024-05-09T08:50:38.743849+0000","last_clean":"2024-05-09T08:50:38.743849+0000","last_became_active":"2024-05-09T08:49:18.368175+0000","last_became_peered":"2024-05-09T08:49:18.368175+0000","last_unstale":"2024-05-09T08:50:38.743849+0000","last_undegraded":"2024-05-09T08:50:38.743849+0000","last_fullsized":"2024-05-09T08:50:38.743849+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:39:32.399209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.502258+0000","last_change":"2024-05-09T08:48:18.363668+0000","last_active":"2024-05-09T08:50:38.502258+0000","last_peered":"2024-05-09T08:50:38.502258+0000","last_clean":"2024-05-09T08:50:38.502258+0000","last_became_active":"2024-05-09T08:48:18.363248+0000","last_became_peered":"2024-05-09T08:48:18.363248+0000","last_unstale":"2024-05-09T08:50:38.502258+0000","last_undegraded":"2024-05-09T08:50:38.502258+0000","last_fullsized":"2024-05-09T08:50:38.502258+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:19:33.802350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740440+0000","last_change":"2024-05-09T08:48:18.376401+0000","last_active":"2024-05-09T08:50:38.740440+0000","last_peered":"2024-05-09T08:50:38.740440+0000","last_clean":"2024-05-09T08:50:38.740440+0000","last_became_active":"2024-05-09T08:48:18.375520+0000","last_became_peered":"2024-05-09T08:48:18.375520+0000","last_unstale":"2024-05-09T08:50:38.740440+0000","last_undegraded":"2024-05-09T08:50:38.740440+0000","last_fullsized":"2024-05-09T08:50:38.740440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:40:36.145071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738307+0000","last_change":"2024-05-09T08:48:18.372847+0000","last_active":"2024-05-09T08:50:38.738307+0000","last_peered":"2024-05-09T08:50:38.738307+0000","last_clean":"2024-05-09T08:50:38.738307+0000","last_became_active":"2024-05-09T08:48:18.370671+0000","last_became_peered":"2024-05-09T08:48:18.370671+0000","last_unstale":"2024-05-09T08:50:38.738307+0000","last_undegraded":"2024-05-09T08:50:38.738307+0000","last_fullsized":"2024-05-09T08:50:38.738307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:06:22.438515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738426+0000","last_change":"2024-05-09T08:48:18.365810+0000","last_active":"2024-05-09T08:50:38.738426+0000","last_peered":"2024-05-09T08:50:38.738426+0000","last_clean":"2024-05-09T08:50:38.738426+0000","last_became_active":"2024-05-09T08:48:18.365214+0000","last_became_peered":"2024-05-09T08:48:18.365214+0000","last_unstale":"2024-05-09T08:50:38.738426+0000","last_undegraded":"2024-05-09T08:50:38.738426+0000","last_fullsized":"2024-05-09T08:50:38.738426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:00.037153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740431+0000","last_change":"2024-05-09T08:48:18.375279+0000","last_active":"2024-05-09T08:50:38.740431+0000","last_peered":"2024-05-09T08:50:38.740431+0000","last_clean":"2024-05-09T08:50:38.740431+0000","last_became_active":"2024-05-09T08:48:18.374918+0000","last_became_peered":"2024-05-09T08:48:18.374918+0000","last_unstale":"2024-05-09T08:50:38.740431+0000","last_undegraded":"2024-05-09T08:50:38.740431+0000","last_fullsized":"2024-05-09T08:50:38.740431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:23:12.889283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740477+0000","last_change":"2024-05-09T08:49:18.369768+0000","last_active":"2024-05-09T08:50:38.740477+0000","last_peered":"2024-05-09T08:50:38.740477+0000","last_clean":"2024-05-09T08:50:38.740477+0000","last_became_active":"2024-05-09T08:49:18.369520+0000","last_became_peered":"2024-05-09T08:49:18.369520+0000","last_unstale":"2024-05-09T08:50:38.740477+0000","last_undegraded":"2024-05-09T08:50:38.740477+0000","last_fullsized":"2024-05-09T08:50:38.740477+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:44:19.502358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743522+0000","last_change":"2024-05-09T08:48:18.373905+0000","last_active":"2024-05-09T08:50:38.743522+0000","last_peered":"2024-05-09T08:50:38.743522+0000","last_clean":"2024-05-09T08:50:38.743522+0000","last_became_active":"2024-05-09T08:48:18.373566+0000","last_became_peered":"2024-05-09T08:48:18.373566+0000","last_unstale":"2024-05-09T08:50:38.743522+0000","last_undegraded":"2024-05-09T08:50:38.743522+0000","last_fullsized":"2024-05-09T08:50:38.743522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:33.905686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665140+0000","last_change":"2024-05-09T08:48:18.373970+0000","last_active":"2024-05-09T08:50:38.665140+0000","last_peered":"2024-05-09T08:50:38.665140+0000","last_clean":"2024-05-09T08:50:38.665140+0000","last_became_active":"2024-05-09T08:48:18.372696+0000","last_became_peered":"2024-05-09T08:48:18.372696+0000","last_unstale":"2024-05-09T08:50:38.665140+0000","last_undegraded":"2024-05-09T08:50:38.665140+0000","last_fullsized":"2024-05-09T08:50:38.665140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:09:33.751950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597421+0000","last_change":"2024-05-09T08:48:18.376851+0000","last_active":"2024-05-09T08:50:38.597421+0000","last_peered":"2024-05-09T08:50:38.597421+0000","last_clean":"2024-05-09T08:50:38.597421+0000","last_became_active":"2024-05-09T08:48:18.376230+0000","last_became_peered":"2024-05-09T08:48:18.376230+0000","last_unstale":"2024-05-09T08:50:38.597421+0000","last_undegraded":"2024-05-09T08:50:38.597421+0000","last_fullsized":"2024-05-09T08:50:38.597421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:34.959179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738942+0000","last_change":"2024-05-09T08:48:18.370287+0000","last_active":"2024-05-09T08:50:38.738942+0000","last_peered":"2024-05-09T08:50:38.738942+0000","last_clean":"2024-05-09T08:50:38.738942+0000","last_became_active":"2024-05-09T08:48:18.369956+0000","last_became_peered":"2024-05-09T08:48:18.369956+0000","last_unstale":"2024-05-09T08:50:38.738942+0000","last_undegraded":"2024-05-09T08:50:38.738942+0000","last_fullsized":"2024-05-09T08:50:38.738942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:19:34.746216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738556+0000","last_change":"2024-05-09T08:48:22.377973+0000","last_active":"2024-05-09T08:50:38.738556+0000","last_peered":"2024-05-09T08:50:38.738556+0000","last_clean":"2024-05-09T08:50:38.738556+0000","last_became_active":"2024-05-09T08:48:22.377325+0000","last_became_peered":"2024-05-09T08:48:22.377325+0000","last_unstale":"2024-05-09T08:50:38.738556+0000","last_undegraded":"2024-05-09T08:50:38.738556+0000","last_fullsized":"2024-05-09T08:50:38.738556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:32:00.873082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"95'14","reported_seq":169,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.744593+0000","last_change":"2024-05-09T08:48:17.595784+0000","last_active":"2024-05-09T08:50:38.744593+0000","last_peered":"2024-05-09T08:50:38.744593+0000","last_clean":"2024-05-09T08:50:38.744593+0000","last_became_active":"2024-05-09T08:48:17.595395+0000","last_became_peered":"2024-05-09T08:48:17.595395+0000","last_unstale":"2024-05-09T08:50:38.744593+0000","last_undegraded":"2024-05-09T08:50:38.744593+0000","last_fullsized":"2024-05-09T08:50:38.744593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:47.475148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497681+0000","last_change":"2024-05-09T08:48:18.382332+0000","last_active":"2024-05-09T08:50:38.497681+0000","last_peered":"2024-05-09T08:50:38.497681+0000","last_clean":"2024-05-09T08:50:38.497681+0000","last_became_active":"2024-05-09T08:48:18.381536+0000","last_became_peered":"2024-05-09T08:48:18.381536+0000","last_unstale":"2024-05-09T08:50:38.497681+0000","last_undegraded":"2024-05-09T08:50:38.497681+0000","last_fullsized":"2024-05-09T08:50:38.497681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:52:48.327000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737688+0000","last_change":"2024-05-09T08:48:22.379541+0000","last_active":"2024-05-09T08:50:38.737688+0000","last_peered":"2024-05-09T08:50:38.737688+0000","last_clean":"2024-05-09T08:50:38.737688+0000","last_became_active":"2024-05-09T08:48:22.379135+0000","last_became_peered":"2024-05-09T08:48:22.379135+0000","last_unstale":"2024-05-09T08:50:38.737688+0000","last_undegraded":"2024-05-09T08:50:38.737688+0000","last_fullsized":"2024-05-09T08:50:38.737688+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:51.365075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497915+0000","last_change":"2024-05-09T08:48:17.592168+0000","last_active":"2024-05-09T08:50:38.497915+0000","last_peered":"2024-05-09T08:50:38.497915+0000","last_clean":"2024-05-09T08:50:38.497915+0000","last_became_active":"2024-05-09T08:48:17.591822+0000","last_became_peered":"2024-05-09T08:48:17.591822+0000","last_unstale":"2024-05-09T08:50:38.497915+0000","last_undegraded":"2024-05-09T08:50:38.497915+0000","last_fullsized":"2024-05-09T08:50:38.497915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:45:15.162240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.741025+0000","last_change":"2024-05-09T08:48:18.379396+0000","last_active":"2024-05-09T08:50:38.741025+0000","last_peered":"2024-05-09T08:50:38.741025+0000","last_clean":"2024-05-09T08:50:38.741025+0000","last_became_active":"2024-05-09T08:48:18.378770+0000","last_became_peered":"2024-05-09T08:48:18.378770+0000","last_unstale":"2024-05-09T08:50:38.741025+0000","last_undegraded":"2024-05-09T08:50:38.741025+0000","last_fullsized":"2024-05-09T08:50:38.741025+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:22:22.699186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737898+0000","last_change":"2024-05-09T08:48:22.387777+0000","last_active":"2024-05-09T08:50:38.737898+0000","last_peered":"2024-05-09T08:50:38.737898+0000","last_clean":"2024-05-09T08:50:38.737898+0000","last_became_active":"2024-05-09T08:48:22.387300+0000","last_became_peered":"2024-05-09T08:48:22.387300+0000","last_unstale":"2024-05-09T08:50:38.737898+0000","last_undegraded":"2024-05-09T08:50:38.737898+0000","last_fullsized":"2024-05-09T08:50:38.737898+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:08:35.646374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"95'13","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498917+0000","last_change":"2024-05-09T08:48:17.602493+0000","last_active":"2024-05-09T08:50:38.498917+0000","last_peered":"2024-05-09T08:50:38.498917+0000","last_clean":"2024-05-09T08:50:38.498917+0000","last_became_active":"2024-05-09T08:48:17.601625+0000","last_became_peered":"2024-05-09T08:48:17.601625+0000","last_unstale":"2024-05-09T08:50:38.498917+0000","last_undegraded":"2024-05-09T08:50:38.498917+0000","last_fullsized":"2024-05-09T08:50:38.498917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:33.044532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743329+0000","last_change":"2024-05-09T08:48:18.380615+0000","last_active":"2024-05-09T08:50:38.743329+0000","last_peered":"2024-05-09T08:50:38.743329+0000","last_clean":"2024-05-09T08:50:38.743329+0000","last_became_active":"2024-05-09T08:48:18.380196+0000","last_became_peered":"2024-05-09T08:48:18.380196+0000","last_unstale":"2024-05-09T08:50:38.743329+0000","last_undegraded":"2024-05-09T08:50:38.743329+0000","last_fullsized":"2024-05-09T08:50:38.743329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:27.299263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737797+0000","last_change":"2024-05-09T08:48:22.381954+0000","last_active":"2024-05-09T08:50:38.737797+0000","last_peered":"2024-05-09T08:50:38.737797+0000","last_clean":"2024-05-09T08:50:38.737797+0000","last_became_active":"2024-05-09T08:48:22.381527+0000","last_became_peered":"2024-05-09T08:48:22.381527+0000","last_unstale":"2024-05-09T08:50:38.737797+0000","last_undegraded":"2024-05-09T08:50:38.737797+0000","last_fullsized":"2024-05-09T08:50:38.737797+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:35.170753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498547+0000","last_change":"2024-05-09T08:48:17.592795+0000","last_active":"2024-05-09T08:50:38.498547+0000","last_peered":"2024-05-09T08:50:38.498547+0000","last_clean":"2024-05-09T08:50:38.498547+0000","last_became_active":"2024-05-09T08:48:17.592240+0000","last_became_peered":"2024-05-09T08:48:17.592240+0000","last_unstale":"2024-05-09T08:50:38.498547+0000","last_undegraded":"2024-05-09T08:50:38.498547+0000","last_fullsized":"2024-05-09T08:50:38.498547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:27:30.056805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605470+0000","last_change":"2024-05-09T08:50:19.350043+0000","last_active":"2024-05-09T08:50:38.605470+0000","last_peered":"2024-05-09T08:50:38.605470+0000","last_clean":"2024-05-09T08:50:38.605470+0000","last_became_active":"2024-05-09T08:50:19.349640+0000","last_became_peered":"2024-05-09T08:50:19.349640+0000","last_unstale":"2024-05-09T08:50:38.605470+0000","last_undegraded":"2024-05-09T08:50:38.605470+0000","last_fullsized":"2024-05-09T08:50:38.605470+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:00:47.277071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738696+0000","last_change":"2024-05-09T08:48:22.374267+0000","last_active":"2024-05-09T08:50:38.738696+0000","last_peered":"2024-05-09T08:50:38.738696+0000","last_clean":"2024-05-09T08:50:38.738696+0000","last_became_active":"2024-05-09T08:48:22.373824+0000","last_became_peered":"2024-05-09T08:48:22.373824+0000","last_unstale":"2024-05-09T08:50:38.738696+0000","last_undegraded":"2024-05-09T08:50:38.738696+0000","last_fullsized":"2024-05-09T08:50:38.738696+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:27.407543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":168,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605692+0000","last_change":"2024-05-09T08:48:17.596851+0000","last_active":"2024-05-09T08:50:38.605692+0000","last_peered":"2024-05-09T08:50:38.605692+0000","last_clean":"2024-05-09T08:50:38.605692+0000","last_became_active":"2024-05-09T08:48:17.596532+0000","last_became_peered":"2024-05-09T08:48:17.596532+0000","last_unstale":"2024-05-09T08:50:38.605692+0000","last_undegraded":"2024-05-09T08:50:38.605692+0000","last_fullsized":"2024-05-09T08:50:38.605692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:58:51.923188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498499+0000","last_change":"2024-05-09T08:48:18.372214+0000","last_active":"2024-05-09T08:50:38.498499+0000","last_peered":"2024-05-09T08:50:38.498499+0000","last_clean":"2024-05-09T08:50:38.498499+0000","last_became_active":"2024-05-09T08:48:18.371850+0000","last_became_peered":"2024-05-09T08:48:18.371850+0000","last_unstale":"2024-05-09T08:50:38.498499+0000","last_undegraded":"2024-05-09T08:50:38.498499+0000","last_fullsized":"2024-05-09T08:50:38.498499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:56.971735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738968+0000","last_change":"2024-05-09T08:49:18.371344+0000","last_active":"2024-05-09T08:50:38.738968+0000","last_peered":"2024-05-09T08:50:38.738968+0000","last_clean":"2024-05-09T08:50:38.738968+0000","last_became_active":"2024-05-09T08:49:18.370911+0000","last_became_peered":"2024-05-09T08:49:18.370911+0000","last_unstale":"2024-05-09T08:50:38.738968+0000","last_undegraded":"2024-05-09T08:50:38.738968+0000","last_fullsized":"2024-05-09T08:50:38.738968+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:53:49.122965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,8],"acting":[2,6,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.501701+0000","last_change":"2024-05-09T08:48:17.601726+0000","last_active":"2024-05-09T08:50:38.501701+0000","last_peered":"2024-05-09T08:50:38.501701+0000","last_clean":"2024-05-09T08:50:38.501701+0000","last_became_active":"2024-05-09T08:48:17.601372+0000","last_became_peered":"2024-05-09T08:48:17.601372+0000","last_unstale":"2024-05-09T08:50:38.501701+0000","last_undegraded":"2024-05-09T08:50:38.501701+0000","last_fullsized":"2024-05-09T08:50:38.501701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:27:56.683821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605526+0000","last_change":"2024-05-09T08:49:18.694713+0000","last_active":"2024-05-09T08:50:38.605526+0000","last_peered":"2024-05-09T08:50:38.605526+0000","last_clean":"2024-05-09T08:50:38.605526+0000","last_became_active":"2024-05-09T08:49:18.694246+0000","last_became_peered":"2024-05-09T08:49:18.694246+0000","last_unstale":"2024-05-09T08:50:38.605526+0000","last_undegraded":"2024-05-09T08:50:38.605526+0000","last_fullsized":"2024-05-09T08:50:38.605526+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:05:30.818965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665087+0000","last_change":"2024-05-09T08:48:22.390990+0000","last_active":"2024-05-09T08:50:38.665087+0000","last_peered":"2024-05-09T08:50:38.665087+0000","last_clean":"2024-05-09T08:50:38.665087+0000","last_became_active":"2024-05-09T08:48:22.390601+0000","last_became_peered":"2024-05-09T08:48:22.390601+0000","last_unstale":"2024-05-09T08:50:38.665087+0000","last_undegraded":"2024-05-09T08:50:38.665087+0000","last_fullsized":"2024-05-09T08:50:38.665087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:47:24.644408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738010+0000","last_change":"2024-05-09T08:48:17.598008+0000","last_active":"2024-05-09T08:50:38.738010+0000","last_peered":"2024-05-09T08:50:38.738010+0000","last_clean":"2024-05-09T08:50:38.738010+0000","last_became_active":"2024-05-09T08:48:17.597514+0000","last_became_peered":"2024-05-09T08:48:17.597514+0000","last_unstale":"2024-05-09T08:50:38.738010+0000","last_undegraded":"2024-05-09T08:50:38.738010+0000","last_fullsized":"2024-05-09T08:50:38.738010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:18:32.788536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497274+0000","last_change":"2024-05-09T08:48:18.378061+0000","last_active":"2024-05-09T08:50:38.497274+0000","last_peered":"2024-05-09T08:50:38.497274+0000","last_clean":"2024-05-09T08:50:38.497274+0000","last_became_active":"2024-05-09T08:48:18.377684+0000","last_became_peered":"2024-05-09T08:48:18.377684+0000","last_unstale":"2024-05-09T08:50:38.497274+0000","last_undegraded":"2024-05-09T08:50:38.497274+0000","last_fullsized":"2024-05-09T08:50:38.497274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:13:08.338307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740944+0000","last_change":"2024-05-09T08:48:22.385779+0000","last_active":"2024-05-09T08:50:38.740944+0000","last_peered":"2024-05-09T08:50:38.740944+0000","last_clean":"2024-05-09T08:50:38.740944+0000","last_became_active":"2024-05-09T08:48:22.385021+0000","last_became_peered":"2024-05-09T08:48:22.385021+0000","last_unstale":"2024-05-09T08:50:38.740944+0000","last_undegraded":"2024-05-09T08:50:38.740944+0000","last_fullsized":"2024-05-09T08:50:38.740944+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:51.350754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605038+0000","last_change":"2024-05-09T08:48:17.595750+0000","last_active":"2024-05-09T08:50:38.605038+0000","last_peered":"2024-05-09T08:50:38.605038+0000","last_clean":"2024-05-09T08:50:38.605038+0000","last_became_active":"2024-05-09T08:48:17.595394+0000","last_became_peered":"2024-05-09T08:48:17.595394+0000","last_unstale":"2024-05-09T08:50:38.605038+0000","last_undegraded":"2024-05-09T08:50:38.605038+0000","last_fullsized":"2024-05-09T08:50:38.605038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:39:21.975059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.596769+0000","last_change":"2024-05-09T08:48:18.370750+0000","last_active":"2024-05-09T08:50:38.596769+0000","last_peered":"2024-05-09T08:50:38.596769+0000","last_clean":"2024-05-09T08:50:38.596769+0000","last_became_active":"2024-05-09T08:48:18.370383+0000","last_became_peered":"2024-05-09T08:48:18.370383+0000","last_unstale":"2024-05-09T08:50:38.596769+0000","last_undegraded":"2024-05-09T08:50:38.596769+0000","last_fullsized":"2024-05-09T08:50:38.596769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:36.049350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497318+0000","last_change":"2024-05-09T08:48:22.377157+0000","last_active":"2024-05-09T08:50:38.497318+0000","last_peered":"2024-05-09T08:50:38.497318+0000","last_clean":"2024-05-09T08:50:38.497318+0000","last_became_active":"2024-05-09T08:48:22.376730+0000","last_became_peered":"2024-05-09T08:48:22.376730+0000","last_unstale":"2024-05-09T08:50:38.497318+0000","last_undegraded":"2024-05-09T08:50:38.497318+0000","last_fullsized":"2024-05-09T08:50:38.497318+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:41:52.080730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"95'7","reported_seq":282,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.894752+0000","last_change":"2024-05-09T08:48:17.598528+0000","last_active":"2024-05-09T08:50:38.894752+0000","last_peered":"2024-05-09T08:50:38.894752+0000","last_clean":"2024-05-09T08:50:38.894752+0000","last_became_active":"2024-05-09T08:48:17.598072+0000","last_became_peered":"2024-05-09T08:48:17.598072+0000","last_unstale":"2024-05-09T08:50:38.894752+0000","last_undegraded":"2024-05-09T08:50:38.894752+0000","last_fullsized":"2024-05-09T08:50:38.894752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:35.697154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743731+0000","last_change":"2024-05-09T08:48:18.380919+0000","last_active":"2024-05-09T08:50:38.743731+0000","last_peered":"2024-05-09T08:50:38.743731+0000","last_clean":"2024-05-09T08:50:38.743731+0000","last_became_active":"2024-05-09T08:48:18.380485+0000","last_became_peered":"2024-05-09T08:48:18.380485+0000","last_unstale":"2024-05-09T08:50:38.743731+0000","last_undegraded":"2024-05-09T08:50:38.743731+0000","last_fullsized":"2024-05-09T08:50:38.743731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:45.079173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.596707+0000","last_change":"2024-05-09T08:48:22.389667+0000","last_active":"2024-05-09T08:50:38.596707+0000","last_peered":"2024-05-09T08:50:38.596707+0000","last_clean":"2024-05-09T08:50:38.596707+0000","last_became_active":"2024-05-09T08:48:22.389268+0000","last_became_peered":"2024-05-09T08:48:22.389268+0000","last_unstale":"2024-05-09T08:50:38.596707+0000","last_undegraded":"2024-05-09T08:50:38.596707+0000","last_fullsized":"2024-05-09T08:50:38.596707+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:33:14.861855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740395+0000","last_change":"2024-05-09T08:48:17.596028+0000","last_active":"2024-05-09T08:50:38.740395+0000","last_peered":"2024-05-09T08:50:38.740395+0000","last_clean":"2024-05-09T08:50:38.740395+0000","last_became_active":"2024-05-09T08:48:17.595053+0000","last_became_peered":"2024-05-09T08:48:17.595053+0000","last_unstale":"2024-05-09T08:50:38.740395+0000","last_undegraded":"2024-05-09T08:50:38.740395+0000","last_fullsized":"2024-05-09T08:50:38.740395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:48:31.323595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740976+0000","last_change":"2024-05-09T08:48:18.373472+0000","last_active":"2024-05-09T08:50:38.740976+0000","last_peered":"2024-05-09T08:50:38.740976+0000","last_clean":"2024-05-09T08:50:38.740976+0000","last_became_active":"2024-05-09T08:48:18.373178+0000","last_became_peered":"2024-05-09T08:48:18.373178+0000","last_unstale":"2024-05-09T08:50:38.740976+0000","last_undegraded":"2024-05-09T08:50:38.740976+0000","last_fullsized":"2024-05-09T08:50:38.740976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:18.048722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738133+0000","last_change":"2024-05-09T08:48:22.386551+0000","last_active":"2024-05-09T08:50:38.738133+0000","last_peered":"2024-05-09T08:50:38.738133+0000","last_clean":"2024-05-09T08:50:38.738133+0000","last_became_active":"2024-05-09T08:48:22.386018+0000","last_became_peered":"2024-05-09T08:48:22.386018+0000","last_unstale":"2024-05-09T08:50:38.738133+0000","last_undegraded":"2024-05-09T08:50:38.738133+0000","last_fullsized":"2024-05-09T08:50:38.738133+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:59.812030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605522+0000","last_change":"2024-05-09T08:48:17.601207+0000","last_active":"2024-05-09T08:50:38.605522+0000","last_peered":"2024-05-09T08:50:38.605522+0000","last_clean":"2024-05-09T08:50:38.605522+0000","last_became_active":"2024-05-09T08:48:17.600878+0000","last_became_peered":"2024-05-09T08:48:17.600878+0000","last_unstale":"2024-05-09T08:50:38.605522+0000","last_undegraded":"2024-05-09T08:50:38.605522+0000","last_fullsized":"2024-05-09T08:50:38.605522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:30.040527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743577+0000","last_change":"2024-05-09T08:48:18.381577+0000","last_active":"2024-05-09T08:50:38.743577+0000","last_peered":"2024-05-09T08:50:38.743577+0000","last_clean":"2024-05-09T08:50:38.743577+0000","last_became_active":"2024-05-09T08:48:18.381223+0000","last_became_peered":"2024-05-09T08:48:18.381223+0000","last_unstale":"2024-05-09T08:50:38.743577+0000","last_undegraded":"2024-05-09T08:50:38.743577+0000","last_fullsized":"2024-05-09T08:50:38.743577+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:57.038392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.596698+0000","last_change":"2024-05-09T08:48:22.385885+0000","last_active":"2024-05-09T08:50:38.596698+0000","last_peered":"2024-05-09T08:50:38.596698+0000","last_clean":"2024-05-09T08:50:38.596698+0000","last_became_active":"2024-05-09T08:48:22.385403+0000","last_became_peered":"2024-05-09T08:48:22.385403+0000","last_unstale":"2024-05-09T08:50:38.596698+0000","last_undegraded":"2024-05-09T08:50:38.596698+0000","last_fullsized":"2024-05-09T08:50:38.596698+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:53:20.733523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'39","reported_seq":327,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.895223+0000","last_change":"2024-05-09T08:48:17.597848+0000","last_active":"2024-05-09T08:50:38.895223+0000","last_peered":"2024-05-09T08:50:38.895223+0000","last_clean":"2024-05-09T08:50:38.895223+0000","last_became_active":"2024-05-09T08:48:17.596989+0000","last_became_peered":"2024-05-09T08:48:17.596989+0000","last_unstale":"2024-05-09T08:50:38.895223+0000","last_undegraded":"2024-05-09T08:50:38.895223+0000","last_fullsized":"2024-05-09T08:50:38.895223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:07.256301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65904,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":254,"num_read_kb":192,"num_write":41,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498676+0000","last_change":"2024-05-09T08:48:18.365308+0000","last_active":"2024-05-09T08:50:38.498676+0000","last_peered":"2024-05-09T08:50:38.498676+0000","last_clean":"2024-05-09T08:50:38.498676+0000","last_became_active":"2024-05-09T08:48:18.364845+0000","last_became_peered":"2024-05-09T08:48:18.364845+0000","last_unstale":"2024-05-09T08:50:38.498676+0000","last_undegraded":"2024-05-09T08:50:38.498676+0000","last_fullsized":"2024-05-09T08:50:38.498676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:07.277730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605765+0000","last_change":"2024-05-09T08:48:22.377333+0000","last_active":"2024-05-09T08:50:38.605765+0000","last_peered":"2024-05-09T08:50:38.605765+0000","last_clean":"2024-05-09T08:50:38.605765+0000","last_became_active":"2024-05-09T08:48:22.376494+0000","last_became_peered":"2024-05-09T08:48:22.376494+0000","last_unstale":"2024-05-09T08:50:38.605765+0000","last_undegraded":"2024-05-09T08:50:38.605765+0000","last_fullsized":"2024-05-09T08:50:38.605765+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:49:17.693977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664772+0000","last_change":"2024-05-09T08:50:19.301723+0000","last_active":"2024-05-09T08:50:38.664772+0000","last_peered":"2024-05-09T08:50:38.664772+0000","last_clean":"2024-05-09T08:50:38.664772+0000","last_became_active":"2024-05-09T08:50:19.291600+0000","last_became_peered":"2024-05-09T08:50:19.291600+0000","last_unstale":"2024-05-09T08:50:38.664772+0000","last_undegraded":"2024-05-09T08:50:38.664772+0000","last_fullsized":"2024-05-09T08:50:38.664772+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:29.105282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738296+0000","last_change":"2024-05-09T08:48:18.373652+0000","last_active":"2024-05-09T08:50:38.738296+0000","last_peered":"2024-05-09T08:50:38.738296+0000","last_clean":"2024-05-09T08:50:38.738296+0000","last_became_active":"2024-05-09T08:48:18.373345+0000","last_became_peered":"2024-05-09T08:48:18.373345+0000","last_unstale":"2024-05-09T08:50:38.738296+0000","last_undegraded":"2024-05-09T08:50:38.738296+0000","last_fullsized":"2024-05-09T08:50:38.738296+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:14:34.843305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664855+0000","last_change":"2024-05-09T08:49:18.369432+0000","last_active":"2024-05-09T08:50:38.664855+0000","last_peered":"2024-05-09T08:50:38.664855+0000","last_clean":"2024-05-09T08:50:38.664855+0000","last_became_active":"2024-05-09T08:49:18.368978+0000","last_became_peered":"2024-05-09T08:49:18.368978+0000","last_unstale":"2024-05-09T08:50:38.664855+0000","last_undegraded":"2024-05-09T08:50:38.664855+0000","last_fullsized":"2024-05-09T08:50:38.664855+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:20.969162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605525+0000","last_change":"2024-05-09T08:48:17.592658+0000","last_active":"2024-05-09T08:50:38.605525+0000","last_peered":"2024-05-09T08:50:38.605525+0000","last_clean":"2024-05-09T08:50:38.605525+0000","last_became_active":"2024-05-09T08:48:17.592275+0000","last_became_peered":"2024-05-09T08:48:17.592275+0000","last_unstale":"2024-05-09T08:50:38.605525+0000","last_undegraded":"2024-05-09T08:50:38.605525+0000","last_fullsized":"2024-05-09T08:50:38.605525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:26:56.742079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740268+0000","last_change":"2024-05-09T08:49:18.364399+0000","last_active":"2024-05-09T08:50:38.740268+0000","last_peered":"2024-05-09T08:50:38.740268+0000","last_clean":"2024-05-09T08:50:38.740268+0000","last_became_active":"2024-05-09T08:49:18.364166+0000","last_became_peered":"2024-05-09T08:49:18.364166+0000","last_unstale":"2024-05-09T08:50:38.740268+0000","last_undegraded":"2024-05-09T08:50:38.740268+0000","last_fullsized":"2024-05-09T08:50:38.740268+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:49:36.480337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597700+0000","last_change":"2024-05-09T08:48:22.389568+0000","last_active":"2024-05-09T08:50:38.597700+0000","last_peered":"2024-05-09T08:50:38.597700+0000","last_clean":"2024-05-09T08:50:38.597700+0000","last_became_active":"2024-05-09T08:48:22.387192+0000","last_became_peered":"2024-05-09T08:48:22.387192+0000","last_unstale":"2024-05-09T08:50:38.597700+0000","last_undegraded":"2024-05-09T08:50:38.597700+0000","last_fullsized":"2024-05-09T08:50:38.597700+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:56.794208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"95'8","reported_seq":169,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.895068+0000","last_change":"2024-05-09T08:48:17.598760+0000","last_active":"2024-05-09T08:50:38.895068+0000","last_peered":"2024-05-09T08:50:38.895068+0000","last_clean":"2024-05-09T08:50:38.895068+0000","last_became_active":"2024-05-09T08:48:17.598192+0000","last_became_peered":"2024-05-09T08:48:17.598192+0000","last_unstale":"2024-05-09T08:50:38.895068+0000","last_undegraded":"2024-05-09T08:50:38.895068+0000","last_fullsized":"2024-05-09T08:50:38.895068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:29.007644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":128,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739920+0000","last_change":"2024-05-09T08:48:18.364570+0000","last_active":"2024-05-09T08:50:38.739920+0000","last_peered":"2024-05-09T08:50:38.739920+0000","last_clean":"2024-05-09T08:50:38.739920+0000","last_became_active":"2024-05-09T08:48:18.364191+0000","last_became_peered":"2024-05-09T08:48:18.364191+0000","last_unstale":"2024-05-09T08:50:38.739920+0000","last_undegraded":"2024-05-09T08:50:38.739920+0000","last_fullsized":"2024-05-09T08:50:38.739920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:49.267954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737704+0000","last_change":"2024-05-09T08:49:18.694742+0000","last_active":"2024-05-09T08:50:38.737704+0000","last_peered":"2024-05-09T08:50:38.737704+0000","last_clean":"2024-05-09T08:50:38.737704+0000","last_became_active":"2024-05-09T08:49:18.694241+0000","last_became_peered":"2024-05-09T08:49:18.694241+0000","last_unstale":"2024-05-09T08:50:38.737704+0000","last_undegraded":"2024-05-09T08:50:38.737704+0000","last_fullsized":"2024-05-09T08:50:38.737704+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:20.289651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498205+0000","last_change":"2024-05-09T08:48:17.599762+0000","last_active":"2024-05-09T08:50:38.498205+0000","last_peered":"2024-05-09T08:50:38.498205+0000","last_clean":"2024-05-09T08:50:38.498205+0000","last_became_active":"2024-05-09T08:48:17.599355+0000","last_became_peered":"2024-05-09T08:48:17.599355+0000","last_unstale":"2024-05-09T08:50:38.498205+0000","last_undegraded":"2024-05-09T08:50:38.498205+0000","last_fullsized":"2024-05-09T08:50:38.498205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:28:36.836403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605208+0000","last_change":"2024-05-09T08:48:18.372810+0000","last_active":"2024-05-09T08:50:38.605208+0000","last_peered":"2024-05-09T08:50:38.605208+0000","last_clean":"2024-05-09T08:50:38.605208+0000","last_became_active":"2024-05-09T08:48:18.372437+0000","last_became_peered":"2024-05-09T08:48:18.372437+0000","last_unstale":"2024-05-09T08:50:38.605208+0000","last_undegraded":"2024-05-09T08:50:38.605208+0000","last_fullsized":"2024-05-09T08:50:38.605208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:03:36.879719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605358+0000","last_change":"2024-05-09T08:48:22.379370+0000","last_active":"2024-05-09T08:50:38.605358+0000","last_peered":"2024-05-09T08:50:38.605358+0000","last_clean":"2024-05-09T08:50:38.605358+0000","last_became_active":"2024-05-09T08:48:22.378994+0000","last_became_peered":"2024-05-09T08:48:22.378994+0000","last_unstale":"2024-05-09T08:50:38.605358+0000","last_undegraded":"2024-05-09T08:50:38.605358+0000","last_fullsized":"2024-05-09T08:50:38.605358+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:08.005647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498896+0000","last_change":"2024-05-09T08:48:17.594236+0000","last_active":"2024-05-09T08:50:38.498896+0000","last_peered":"2024-05-09T08:50:38.498896+0000","last_clean":"2024-05-09T08:50:38.498896+0000","last_became_active":"2024-05-09T08:48:17.593418+0000","last_became_peered":"2024-05-09T08:48:17.593418+0000","last_unstale":"2024-05-09T08:50:38.498896+0000","last_undegraded":"2024-05-09T08:50:38.498896+0000","last_fullsized":"2024-05-09T08:50:38.498896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:23.930421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665072+0000","last_change":"2024-05-09T08:48:18.366767+0000","last_active":"2024-05-09T08:50:38.665072+0000","last_peered":"2024-05-09T08:50:38.665072+0000","last_clean":"2024-05-09T08:50:38.665072+0000","last_became_active":"2024-05-09T08:48:18.366435+0000","last_became_peered":"2024-05-09T08:48:18.366435+0000","last_unstale":"2024-05-09T08:50:38.665072+0000","last_undegraded":"2024-05-09T08:50:38.665072+0000","last_fullsized":"2024-05-09T08:50:38.665072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:45.111510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498398+0000","last_change":"2024-05-09T08:48:22.378642+0000","last_active":"2024-05-09T08:50:38.498398+0000","last_peered":"2024-05-09T08:50:38.498398+0000","last_clean":"2024-05-09T08:50:38.498398+0000","last_became_active":"2024-05-09T08:48:22.378161+0000","last_became_peered":"2024-05-09T08:48:22.378161+0000","last_unstale":"2024-05-09T08:50:38.498398+0000","last_undegraded":"2024-05-09T08:50:38.498398+0000","last_fullsized":"2024-05-09T08:50:38.498398+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:53.149728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597189+0000","last_change":"2024-05-09T08:48:17.599499+0000","last_active":"2024-05-09T08:50:38.597189+0000","last_peered":"2024-05-09T08:50:38.597189+0000","last_clean":"2024-05-09T08:50:38.597189+0000","last_became_active":"2024-05-09T08:48:17.599145+0000","last_became_peered":"2024-05-09T08:48:17.599145+0000","last_unstale":"2024-05-09T08:50:38.597189+0000","last_undegraded":"2024-05-09T08:50:38.597189+0000","last_fullsized":"2024-05-09T08:50:38.597189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:31:12.502004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.502012+0000","last_change":"2024-05-09T08:48:18.372355+0000","last_active":"2024-05-09T08:50:38.502012+0000","last_peered":"2024-05-09T08:50:38.502012+0000","last_clean":"2024-05-09T08:50:38.502012+0000","last_became_active":"2024-05-09T08:48:18.367743+0000","last_became_peered":"2024-05-09T08:48:18.367743+0000","last_unstale":"2024-05-09T08:50:38.502012+0000","last_undegraded":"2024-05-09T08:50:38.502012+0000","last_fullsized":"2024-05-09T08:50:38.502012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:31:12.174350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738469+0000","last_change":"2024-05-09T08:48:22.384064+0000","last_active":"2024-05-09T08:50:38.738469+0000","last_peered":"2024-05-09T08:50:38.738469+0000","last_clean":"2024-05-09T08:50:38.738469+0000","last_became_active":"2024-05-09T08:48:22.383670+0000","last_became_peered":"2024-05-09T08:48:22.383670+0000","last_unstale":"2024-05-09T08:50:38.738469+0000","last_undegraded":"2024-05-09T08:50:38.738469+0000","last_fullsized":"2024-05-09T08:50:38.738469+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:51:35.000827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"95'6","reported_seq":162,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.922953+0000","last_change":"2024-05-09T08:48:17.601596+0000","last_active":"2024-05-09T08:50:38.922953+0000","last_peered":"2024-05-09T08:50:38.922953+0000","last_clean":"2024-05-09T08:50:38.922953+0000","last_became_active":"2024-05-09T08:48:17.601250+0000","last_became_peered":"2024-05-09T08:48:17.601250+0000","last_unstale":"2024-05-09T08:50:38.922953+0000","last_undegraded":"2024-05-09T08:50:38.922953+0000","last_fullsized":"2024-05-09T08:50:38.922953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:48.237336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.501885+0000","last_change":"2024-05-09T08:48:18.368751+0000","last_active":"2024-05-09T08:50:38.501885+0000","last_peered":"2024-05-09T08:50:38.501885+0000","last_clean":"2024-05-09T08:50:38.501885+0000","last_became_active":"2024-05-09T08:48:18.367463+0000","last_became_peered":"2024-05-09T08:48:18.367463+0000","last_unstale":"2024-05-09T08:50:38.501885+0000","last_undegraded":"2024-05-09T08:50:38.501885+0000","last_fullsized":"2024-05-09T08:50:38.501885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:19:17.696602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738886+0000","last_change":"2024-05-09T08:48:22.385025+0000","last_active":"2024-05-09T08:50:38.738886+0000","last_peered":"2024-05-09T08:50:38.738886+0000","last_clean":"2024-05-09T08:50:38.738886+0000","last_became_active":"2024-05-09T08:48:22.384633+0000","last_became_peered":"2024-05-09T08:48:22.384633+0000","last_unstale":"2024-05-09T08:50:38.738886+0000","last_undegraded":"2024-05-09T08:50:38.738886+0000","last_fullsized":"2024-05-09T08:50:38.738886+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:17.804075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605162+0000","last_change":"2024-05-09T08:48:18.376741+0000","last_active":"2024-05-09T08:50:38.605162+0000","last_peered":"2024-05-09T08:50:38.605162+0000","last_clean":"2024-05-09T08:50:38.605162+0000","last_became_active":"2024-05-09T08:48:18.376376+0000","last_became_peered":"2024-05-09T08:48:18.376376+0000","last_unstale":"2024-05-09T08:50:38.605162+0000","last_undegraded":"2024-05-09T08:50:38.605162+0000","last_fullsized":"2024-05-09T08:50:38.605162+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:58:31.717555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"95'7","reported_seq":164,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740200+0000","last_change":"2024-05-09T08:48:17.598316+0000","last_active":"2024-05-09T08:50:38.740200+0000","last_peered":"2024-05-09T08:50:38.740200+0000","last_clean":"2024-05-09T08:50:38.740200+0000","last_became_active":"2024-05-09T08:48:17.598022+0000","last_became_peered":"2024-05-09T08:48:17.598022+0000","last_unstale":"2024-05-09T08:50:38.740200+0000","last_undegraded":"2024-05-09T08:50:38.740200+0000","last_fullsized":"2024-05-09T08:50:38.740200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:04:10.454894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740178+0000","last_change":"2024-05-09T08:49:18.492981+0000","last_active":"2024-05-09T08:50:38.740178+0000","last_peered":"2024-05-09T08:50:38.740178+0000","last_clean":"2024-05-09T08:50:38.740178+0000","last_became_active":"2024-05-09T08:49:18.489169+0000","last_became_peered":"2024-05-09T08:49:18.489169+0000","last_unstale":"2024-05-09T08:50:38.740178+0000","last_undegraded":"2024-05-09T08:50:38.740178+0000","last_fullsized":"2024-05-09T08:50:38.740178+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:02:53.955587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497378+0000","last_change":"2024-05-09T08:48:18.379203+0000","last_active":"2024-05-09T08:50:38.497378+0000","last_peered":"2024-05-09T08:50:38.497378+0000","last_clean":"2024-05-09T08:50:38.497378+0000","last_became_active":"2024-05-09T08:48:18.378885+0000","last_became_peered":"2024-05-09T08:48:18.378885+0000","last_unstale":"2024-05-09T08:50:38.497378+0000","last_undegraded":"2024-05-09T08:50:38.497378+0000","last_fullsized":"2024-05-09T08:50:38.497378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:40.588263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737925+0000","last_change":"2024-05-09T08:48:17.593520+0000","last_active":"2024-05-09T08:50:38.737925+0000","last_peered":"2024-05-09T08:50:38.737925+0000","last_clean":"2024-05-09T08:50:38.737925+0000","last_became_active":"2024-05-09T08:48:17.593127+0000","last_became_peered":"2024-05-09T08:48:17.593127+0000","last_unstale":"2024-05-09T08:50:38.737925+0000","last_undegraded":"2024-05-09T08:50:38.737925+0000","last_fullsized":"2024-05-09T08:50:38.737925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:34:29.567166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":367,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739767+0000","last_change":"2024-05-09T08:47:05.718649+0000","last_active":"2024-05-09T08:50:38.739767+0000","last_peered":"2024-05-09T08:50:38.739767+0000","last_clean":"2024-05-09T08:50:38.739767+0000","last_became_active":"2024-05-09T08:47:05.718088+0000","last_became_peered":"2024-05-09T08:47:05.718088+0000","last_unstale":"2024-05-09T08:50:38.739767+0000","last_undegraded":"2024-05-09T08:50:38.739767+0000","last_fullsized":"2024-05-09T08:50:38.739767+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:43:39.457880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:43:39.457880+0000","last_clean_scrub_stamp":"2024-05-09T08:43:39.457880+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:14.227590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605456+0000","last_change":"2024-05-09T08:48:22.376751+0000","last_active":"2024-05-09T08:50:38.605456+0000","last_peered":"2024-05-09T08:50:38.605456+0000","last_clean":"2024-05-09T08:50:38.605456+0000","last_became_active":"2024-05-09T08:48:22.376322+0000","last_became_peered":"2024-05-09T08:48:22.376322+0000","last_unstale":"2024-05-09T08:50:38.605456+0000","last_undegraded":"2024-05-09T08:50:38.605456+0000","last_fullsized":"2024-05-09T08:50:38.605456+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:08:07.859034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737852+0000","last_change":"2024-05-09T08:48:18.364063+0000","last_active":"2024-05-09T08:50:38.737852+0000","last_peered":"2024-05-09T08:50:38.737852+0000","last_clean":"2024-05-09T08:50:38.737852+0000","last_became_active":"2024-05-09T08:48:18.363694+0000","last_became_peered":"2024-05-09T08:48:18.363694+0000","last_unstale":"2024-05-09T08:50:38.737852+0000","last_undegraded":"2024-05-09T08:50:38.737852+0000","last_fullsized":"2024-05-09T08:50:38.737852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:42.536329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739870+0000","last_change":"2024-05-09T08:48:17.599787+0000","last_active":"2024-05-09T08:50:38.739870+0000","last_peered":"2024-05-09T08:50:38.739870+0000","last_clean":"2024-05-09T08:50:38.739870+0000","last_became_active":"2024-05-09T08:48:17.599248+0000","last_became_peered":"2024-05-09T08:48:17.599248+0000","last_unstale":"2024-05-09T08:50:38.739870+0000","last_undegraded":"2024-05-09T08:50:38.739870+0000","last_fullsized":"2024-05-09T08:50:38.739870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:17.448404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664823+0000","last_change":"2024-05-09T08:48:22.378913+0000","last_active":"2024-05-09T08:50:38.664823+0000","last_peered":"2024-05-09T08:50:38.664823+0000","last_clean":"2024-05-09T08:50:38.664823+0000","last_became_active":"2024-05-09T08:48:22.378441+0000","last_became_peered":"2024-05-09T08:48:22.378441+0000","last_unstale":"2024-05-09T08:50:38.664823+0000","last_undegraded":"2024-05-09T08:50:38.664823+0000","last_fullsized":"2024-05-09T08:50:38.664823+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:49.623778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498746+0000","last_change":"2024-05-09T08:48:17.600121+0000","last_active":"2024-05-09T08:50:38.498746+0000","last_peered":"2024-05-09T08:50:38.498746+0000","last_clean":"2024-05-09T08:50:38.498746+0000","last_became_active":"2024-05-09T08:48:17.599800+0000","last_became_peered":"2024-05-09T08:48:17.599800+0000","last_unstale":"2024-05-09T08:50:38.498746+0000","last_undegraded":"2024-05-09T08:50:38.498746+0000","last_fullsized":"2024-05-09T08:50:38.498746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:56.607393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738578+0000","last_change":"2024-05-09T08:48:18.366612+0000","last_active":"2024-05-09T08:50:38.738578+0000","last_peered":"2024-05-09T08:50:38.738578+0000","last_clean":"2024-05-09T08:50:38.738578+0000","last_became_active":"2024-05-09T08:48:18.366266+0000","last_became_peered":"2024-05-09T08:48:18.366266+0000","last_unstale":"2024-05-09T08:50:38.738578+0000","last_undegraded":"2024-05-09T08:50:38.738578+0000","last_fullsized":"2024-05-09T08:50:38.738578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:35:01.663456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497267+0000","last_change":"2024-05-09T08:48:22.384096+0000","last_active":"2024-05-09T08:50:38.497267+0000","last_peered":"2024-05-09T08:50:38.497267+0000","last_clean":"2024-05-09T08:50:38.497267+0000","last_became_active":"2024-05-09T08:48:22.383656+0000","last_became_peered":"2024-05-09T08:48:22.383656+0000","last_unstale":"2024-05-09T08:50:38.497267+0000","last_undegraded":"2024-05-09T08:50:38.497267+0000","last_fullsized":"2024-05-09T08:50:38.497267+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:13.622377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'48","reported_seq":105,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.707713+0000","last_change":"2024-05-09T08:48:17.593684+0000","last_active":"2024-05-09T08:50:38.707713+0000","last_peered":"2024-05-09T08:50:38.707713+0000","last_clean":"2024-05-09T08:50:38.707713+0000","last_became_active":"2024-05-09T08:48:17.593358+0000","last_became_peered":"2024-05-09T08:48:17.593358+0000","last_unstale":"2024-05-09T08:50:38.707713+0000","last_undegraded":"2024-05-09T08:50:38.707713+0000","last_fullsized":"2024-05-09T08:50:38.707713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:51:36.023421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497783+0000","last_change":"2024-05-09T08:48:18.374498+0000","last_active":"2024-05-09T08:50:38.497783+0000","last_peered":"2024-05-09T08:50:38.497783+0000","last_clean":"2024-05-09T08:50:38.497783+0000","last_became_active":"2024-05-09T08:48:18.374142+0000","last_became_peered":"2024-05-09T08:48:18.374142+0000","last_unstale":"2024-05-09T08:50:38.497783+0000","last_undegraded":"2024-05-09T08:50:38.497783+0000","last_fullsized":"2024-05-09T08:50:38.497783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:53.874419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738424+0000","last_change":"2024-05-09T08:48:22.378330+0000","last_active":"2024-05-09T08:50:38.738424+0000","last_peered":"2024-05-09T08:50:38.738424+0000","last_clean":"2024-05-09T08:50:38.738424+0000","last_became_active":"2024-05-09T08:48:22.377218+0000","last_became_peered":"2024-05-09T08:48:22.377218+0000","last_unstale":"2024-05-09T08:50:38.738424+0000","last_undegraded":"2024-05-09T08:50:38.738424+0000","last_fullsized":"2024-05-09T08:50:38.738424+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:41.650168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738328+0000","last_change":"2024-05-09T08:48:17.601410+0000","last_active":"2024-05-09T08:50:38.738328+0000","last_peered":"2024-05-09T08:50:38.738328+0000","last_clean":"2024-05-09T08:50:38.738328+0000","last_became_active":"2024-05-09T08:48:17.601075+0000","last_became_peered":"2024-05-09T08:48:17.601075+0000","last_unstale":"2024-05-09T08:50:38.738328+0000","last_undegraded":"2024-05-09T08:50:38.738328+0000","last_fullsized":"2024-05-09T08:50:38.738328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:45.760280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.741119+0000","last_change":"2024-05-09T08:48:18.379590+0000","last_active":"2024-05-09T08:50:38.741119+0000","last_peered":"2024-05-09T08:50:38.741119+0000","last_clean":"2024-05-09T08:50:38.741119+0000","last_became_active":"2024-05-09T08:48:18.379187+0000","last_became_peered":"2024-05-09T08:48:18.379187+0000","last_unstale":"2024-05-09T08:50:38.741119+0000","last_undegraded":"2024-05-09T08:50:38.741119+0000","last_fullsized":"2024-05-09T08:50:38.741119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:40:02.630449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737875+0000","last_change":"2024-05-09T08:48:22.386437+0000","last_active":"2024-05-09T08:50:38.737875+0000","last_peered":"2024-05-09T08:50:38.737875+0000","last_clean":"2024-05-09T08:50:38.737875+0000","last_became_active":"2024-05-09T08:48:22.385973+0000","last_became_peered":"2024-05-09T08:48:22.385973+0000","last_unstale":"2024-05-09T08:50:38.737875+0000","last_undegraded":"2024-05-09T08:50:38.737875+0000","last_fullsized":"2024-05-09T08:50:38.737875+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:11:37.338641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743068+0000","last_change":"2024-05-09T08:48:17.598586+0000","last_active":"2024-05-09T08:50:38.743068+0000","last_peered":"2024-05-09T08:50:38.743068+0000","last_clean":"2024-05-09T08:50:38.743068+0000","last_became_active":"2024-05-09T08:48:17.597851+0000","last_became_peered":"2024-05-09T08:48:17.597851+0000","last_unstale":"2024-05-09T08:50:38.743068+0000","last_undegraded":"2024-05-09T08:50:38.743068+0000","last_fullsized":"2024-05-09T08:50:38.743068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:28:22.996999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498169+0000","last_change":"2024-05-09T08:48:18.367319+0000","last_active":"2024-05-09T08:50:38.498169+0000","last_peered":"2024-05-09T08:50:38.498169+0000","last_clean":"2024-05-09T08:50:38.498169+0000","last_became_active":"2024-05-09T08:48:18.366034+0000","last_became_peered":"2024-05-09T08:48:18.366034+0000","last_unstale":"2024-05-09T08:50:38.498169+0000","last_undegraded":"2024-05-09T08:50:38.498169+0000","last_fullsized":"2024-05-09T08:50:38.498169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:22.998089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597628+0000","last_change":"2024-05-09T08:48:22.389129+0000","last_active":"2024-05-09T08:50:38.597628+0000","last_peered":"2024-05-09T08:50:38.597628+0000","last_clean":"2024-05-09T08:50:38.597628+0000","last_became_active":"2024-05-09T08:48:22.386639+0000","last_became_peered":"2024-05-09T08:48:22.386639+0000","last_unstale":"2024-05-09T08:50:38.597628+0000","last_undegraded":"2024-05-09T08:50:38.597628+0000","last_fullsized":"2024-05-09T08:50:38.597628+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:47:07.257464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497375+0000","last_change":"2024-05-09T08:48:17.595255+0000","last_active":"2024-05-09T08:50:38.497375+0000","last_peered":"2024-05-09T08:50:38.497375+0000","last_clean":"2024-05-09T08:50:38.497375+0000","last_became_active":"2024-05-09T08:48:17.594882+0000","last_became_peered":"2024-05-09T08:48:17.594882+0000","last_unstale":"2024-05-09T08:50:38.497375+0000","last_undegraded":"2024-05-09T08:50:38.497375+0000","last_fullsized":"2024-05-09T08:50:38.497375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:49.845326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664529+0000","last_change":"2024-05-09T08:48:18.378580+0000","last_active":"2024-05-09T08:50:38.664529+0000","last_peered":"2024-05-09T08:50:38.664529+0000","last_clean":"2024-05-09T08:50:38.664529+0000","last_became_active":"2024-05-09T08:48:18.378205+0000","last_became_peered":"2024-05-09T08:48:18.378205+0000","last_unstale":"2024-05-09T08:50:38.664529+0000","last_undegraded":"2024-05-09T08:50:38.664529+0000","last_fullsized":"2024-05-09T08:50:38.664529+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:50.103195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738066+0000","last_change":"2024-05-09T08:48:22.378494+0000","last_active":"2024-05-09T08:50:38.738066+0000","last_peered":"2024-05-09T08:50:38.738066+0000","last_clean":"2024-05-09T08:50:38.738066+0000","last_became_active":"2024-05-09T08:48:22.378076+0000","last_became_peered":"2024-05-09T08:48:22.378076+0000","last_unstale":"2024-05-09T08:50:38.738066+0000","last_undegraded":"2024-05-09T08:50:38.738066+0000","last_fullsized":"2024-05-09T08:50:38.738066+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:40:09.079968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'4","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664581+0000","last_change":"2024-05-09T08:50:19.307437+0000","last_active":"2024-05-09T08:50:38.664581+0000","last_peered":"2024-05-09T08:50:38.664581+0000","last_clean":"2024-05-09T08:50:38.664581+0000","last_became_active":"2024-05-09T08:50:19.293112+0000","last_became_peered":"2024-05-09T08:50:19.293112+0000","last_unstale":"2024-05-09T08:50:38.664581+0000","last_undegraded":"2024-05-09T08:50:38.664581+0000","last_fullsized":"2024-05-09T08:50:38.664581+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:20.776030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,1],"acting":[11,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497703+0000","last_change":"2024-05-09T08:48:18.373569+0000","last_active":"2024-05-09T08:50:38.497703+0000","last_peered":"2024-05-09T08:50:38.497703+0000","last_clean":"2024-05-09T08:50:38.497703+0000","last_became_active":"2024-05-09T08:48:18.373238+0000","last_became_peered":"2024-05-09T08:48:18.373238+0000","last_unstale":"2024-05-09T08:50:38.497703+0000","last_undegraded":"2024-05-09T08:50:38.497703+0000","last_fullsized":"2024-05-09T08:50:38.497703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:17:41.629419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738071+0000","last_change":"2024-05-09T08:48:22.381523+0000","last_active":"2024-05-09T08:50:38.738071+0000","last_peered":"2024-05-09T08:50:38.738071+0000","last_clean":"2024-05-09T08:50:38.738071+0000","last_became_active":"2024-05-09T08:48:22.380626+0000","last_became_peered":"2024-05-09T08:48:22.380626+0000","last_unstale":"2024-05-09T08:50:38.738071+0000","last_undegraded":"2024-05-09T08:50:38.738071+0000","last_fullsized":"2024-05-09T08:50:38.738071+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:47:45.092232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498421+0000","last_change":"2024-05-09T08:48:17.601479+0000","last_active":"2024-05-09T08:50:38.498421+0000","last_peered":"2024-05-09T08:50:38.498421+0000","last_clean":"2024-05-09T08:50:38.498421+0000","last_became_active":"2024-05-09T08:48:17.601140+0000","last_became_peered":"2024-05-09T08:48:17.601140+0000","last_unstale":"2024-05-09T08:50:38.498421+0000","last_undegraded":"2024-05-09T08:50:38.498421+0000","last_fullsized":"2024-05-09T08:50:38.498421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:38:24.011209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740038+0000","last_change":"2024-05-09T08:48:18.376890+0000","last_active":"2024-05-09T08:50:38.740038+0000","last_peered":"2024-05-09T08:50:38.740038+0000","last_clean":"2024-05-09T08:50:38.740038+0000","last_became_active":"2024-05-09T08:48:18.376472+0000","last_became_peered":"2024-05-09T08:48:18.376472+0000","last_unstale":"2024-05-09T08:50:38.740038+0000","last_undegraded":"2024-05-09T08:50:38.740038+0000","last_fullsized":"2024-05-09T08:50:38.740038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:40:16.238539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743128+0000","last_change":"2024-05-09T08:48:22.374743+0000","last_active":"2024-05-09T08:50:38.743128+0000","last_peered":"2024-05-09T08:50:38.743128+0000","last_clean":"2024-05-09T08:50:38.743128+0000","last_became_active":"2024-05-09T08:48:22.374324+0000","last_became_peered":"2024-05-09T08:48:22.374324+0000","last_unstale":"2024-05-09T08:50:38.743128+0000","last_undegraded":"2024-05-09T08:50:38.743128+0000","last_fullsized":"2024-05-09T08:50:38.743128+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:48.073647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"95'6","reported_seq":163,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.933811+0000","last_change":"2024-05-09T08:48:17.604700+0000","last_active":"2024-05-09T08:50:38.933811+0000","last_peered":"2024-05-09T08:50:38.933811+0000","last_clean":"2024-05-09T08:50:38.933811+0000","last_became_active":"2024-05-09T08:48:17.604334+0000","last_became_peered":"2024-05-09T08:48:17.604334+0000","last_unstale":"2024-05-09T08:50:38.933811+0000","last_undegraded":"2024-05-09T08:50:38.933811+0000","last_fullsized":"2024-05-09T08:50:38.933811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:15:37.346232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740574+0000","last_change":"2024-05-09T08:48:18.375769+0000","last_active":"2024-05-09T08:50:38.740574+0000","last_peered":"2024-05-09T08:50:38.740574+0000","last_clean":"2024-05-09T08:50:38.740574+0000","last_became_active":"2024-05-09T08:48:18.375059+0000","last_became_peered":"2024-05-09T08:48:18.375059+0000","last_unstale":"2024-05-09T08:50:38.740574+0000","last_undegraded":"2024-05-09T08:50:38.740574+0000","last_fullsized":"2024-05-09T08:50:38.740574+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:51.001240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740616+0000","last_change":"2024-05-09T08:48:22.385306+0000","last_active":"2024-05-09T08:50:38.740616+0000","last_peered":"2024-05-09T08:50:38.740616+0000","last_clean":"2024-05-09T08:50:38.740616+0000","last_became_active":"2024-05-09T08:48:22.383890+0000","last_became_peered":"2024-05-09T08:48:22.383890+0000","last_unstale":"2024-05-09T08:50:38.740616+0000","last_undegraded":"2024-05-09T08:50:38.740616+0000","last_fullsized":"2024-05-09T08:50:38.740616+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:50:52.369399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'41","reported_seq":83,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.706348+0000","last_change":"2024-05-09T08:48:17.591627+0000","last_active":"2024-05-09T08:50:38.706348+0000","last_peered":"2024-05-09T08:50:38.706348+0000","last_clean":"2024-05-09T08:50:38.706348+0000","last_became_active":"2024-05-09T08:48:17.591250+0000","last_became_peered":"2024-05-09T08:48:17.591250+0000","last_unstale":"2024-05-09T08:50:38.706348+0000","last_undegraded":"2024-05-09T08:50:38.706348+0000","last_fullsized":"2024-05-09T08:50:38.706348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:40:59.748892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740833+0000","last_change":"2024-05-09T08:48:18.372256+0000","last_active":"2024-05-09T08:50:38.740833+0000","last_peered":"2024-05-09T08:50:38.740833+0000","last_clean":"2024-05-09T08:50:38.740833+0000","last_became_active":"2024-05-09T08:48:18.371895+0000","last_became_peered":"2024-05-09T08:48:18.371895+0000","last_unstale":"2024-05-09T08:50:38.740833+0000","last_undegraded":"2024-05-09T08:50:38.740833+0000","last_fullsized":"2024-05-09T08:50:38.740833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:25.067587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605019+0000","last_change":"2024-05-09T08:48:22.373684+0000","last_active":"2024-05-09T08:50:38.605019+0000","last_peered":"2024-05-09T08:50:38.605019+0000","last_clean":"2024-05-09T08:50:38.605019+0000","last_became_active":"2024-05-09T08:48:22.373000+0000","last_became_peered":"2024-05-09T08:48:22.373000+0000","last_unstale":"2024-05-09T08:50:38.605019+0000","last_undegraded":"2024-05-09T08:50:38.605019+0000","last_fullsized":"2024-05-09T08:50:38.605019+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:53:14.416837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739078+0000","last_change":"2024-05-09T08:48:17.596910+0000","last_active":"2024-05-09T08:50:38.739078+0000","last_peered":"2024-05-09T08:50:38.739078+0000","last_clean":"2024-05-09T08:50:38.739078+0000","last_became_active":"2024-05-09T08:48:17.596605+0000","last_became_peered":"2024-05-09T08:48:17.596605+0000","last_unstale":"2024-05-09T08:50:38.739078+0000","last_undegraded":"2024-05-09T08:50:38.739078+0000","last_fullsized":"2024-05-09T08:50:38.739078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:05:25.880082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740734+0000","last_change":"2024-05-09T08:48:18.370354+0000","last_active":"2024-05-09T08:50:38.740734+0000","last_peered":"2024-05-09T08:50:38.740734+0000","last_clean":"2024-05-09T08:50:38.740734+0000","last_became_active":"2024-05-09T08:48:18.370043+0000","last_became_peered":"2024-05-09T08:48:18.370043+0000","last_unstale":"2024-05-09T08:50:38.740734+0000","last_undegraded":"2024-05-09T08:50:38.740734+0000","last_fullsized":"2024-05-09T08:50:38.740734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:21:34.066162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664588+0000","last_change":"2024-05-09T08:48:18.376431+0000","last_active":"2024-05-09T08:50:38.664588+0000","last_peered":"2024-05-09T08:50:38.664588+0000","last_clean":"2024-05-09T08:50:38.664588+0000","last_became_active":"2024-05-09T08:48:18.375598+0000","last_became_peered":"2024-05-09T08:48:18.375598+0000","last_unstale":"2024-05-09T08:50:38.664588+0000","last_undegraded":"2024-05-09T08:50:38.664588+0000","last_fullsized":"2024-05-09T08:50:38.664588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:51:13.481219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738676+0000","last_change":"2024-05-09T08:48:18.371460+0000","last_active":"2024-05-09T08:50:38.738676+0000","last_peered":"2024-05-09T08:50:38.738676+0000","last_clean":"2024-05-09T08:50:38.738676+0000","last_became_active":"2024-05-09T08:48:18.371091+0000","last_became_peered":"2024-05-09T08:48:18.371091+0000","last_unstale":"2024-05-09T08:50:38.738676+0000","last_undegraded":"2024-05-09T08:50:38.738676+0000","last_fullsized":"2024-05-09T08:50:38.738676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:02.829902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738487+0000","last_change":"2024-05-09T08:48:18.369453+0000","last_active":"2024-05-09T08:50:38.738487+0000","last_peered":"2024-05-09T08:50:38.738487+0000","last_clean":"2024-05-09T08:50:38.738487+0000","last_became_active":"2024-05-09T08:48:18.368966+0000","last_became_peered":"2024-05-09T08:48:18.368966+0000","last_unstale":"2024-05-09T08:50:38.738487+0000","last_undegraded":"2024-05-09T08:50:38.738487+0000","last_fullsized":"2024-05-09T08:50:38.738487+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:23:14.359222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740734+0000","last_change":"2024-05-09T08:48:18.378815+0000","last_active":"2024-05-09T08:50:38.740734+0000","last_peered":"2024-05-09T08:50:38.740734+0000","last_clean":"2024-05-09T08:50:38.740734+0000","last_became_active":"2024-05-09T08:48:18.378479+0000","last_became_peered":"2024-05-09T08:48:18.378479+0000","last_unstale":"2024-05-09T08:50:38.740734+0000","last_undegraded":"2024-05-09T08:50:38.740734+0000","last_fullsized":"2024-05-09T08:50:38.740734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:38:50.133239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738648+0000","last_change":"2024-05-09T08:48:18.365696+0000","last_active":"2024-05-09T08:50:38.738648+0000","last_peered":"2024-05-09T08:50:38.738648+0000","last_clean":"2024-05-09T08:50:38.738648+0000","last_became_active":"2024-05-09T08:48:18.365358+0000","last_became_peered":"2024-05-09T08:48:18.365358+0000","last_unstale":"2024-05-09T08:50:38.738648+0000","last_undegraded":"2024-05-09T08:50:38.738648+0000","last_fullsized":"2024-05-09T08:50:38.738648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:04.064150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665012+0000","last_change":"2024-05-09T08:48:18.368911+0000","last_active":"2024-05-09T08:50:38.665012+0000","last_peered":"2024-05-09T08:50:38.665012+0000","last_clean":"2024-05-09T08:50:38.665012+0000","last_became_active":"2024-05-09T08:48:18.368591+0000","last_became_peered":"2024-05-09T08:48:18.368591+0000","last_unstale":"2024-05-09T08:50:38.665012+0000","last_undegraded":"2024-05-09T08:50:38.665012+0000","last_fullsized":"2024-05-09T08:50:38.665012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:34:09.692168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738963+0000","last_change":"2024-05-09T08:48:18.373629+0000","last_active":"2024-05-09T08:50:38.738963+0000","last_peered":"2024-05-09T08:50:38.738963+0000","last_clean":"2024-05-09T08:50:38.738963+0000","last_became_active":"2024-05-09T08:48:18.372992+0000","last_became_peered":"2024-05-09T08:48:18.372992+0000","last_unstale":"2024-05-09T08:50:38.738963+0000","last_undegraded":"2024-05-09T08:50:38.738963+0000","last_fullsized":"2024-05-09T08:50:38.738963+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:42.532250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597288+0000","last_change":"2024-05-09T08:50:19.289044+0000","last_active":"2024-05-09T08:50:38.597288+0000","last_peered":"2024-05-09T08:50:38.597288+0000","last_clean":"2024-05-09T08:50:38.597288+0000","last_became_active":"2024-05-09T08:50:19.288657+0000","last_became_peered":"2024-05-09T08:50:19.288657+0000","last_unstale":"2024-05-09T08:50:38.597288+0000","last_undegraded":"2024-05-09T08:50:38.597288+0000","last_fullsized":"2024-05-09T08:50:38.597288+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:09:36.183086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605085+0000","last_change":"2024-05-09T08:48:18.370460+0000","last_active":"2024-05-09T08:50:38.605085+0000","last_peered":"2024-05-09T08:50:38.605085+0000","last_clean":"2024-05-09T08:50:38.605085+0000","last_became_active":"2024-05-09T08:48:18.370033+0000","last_became_peered":"2024-05-09T08:48:18.370033+0000","last_unstale":"2024-05-09T08:50:38.605085+0000","last_undegraded":"2024-05-09T08:50:38.605085+0000","last_fullsized":"2024-05-09T08:50:38.605085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:21:28.622511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497468+0000","last_change":"2024-05-09T08:48:18.371410+0000","last_active":"2024-05-09T08:50:38.497468+0000","last_peered":"2024-05-09T08:50:38.497468+0000","last_clean":"2024-05-09T08:50:38.497468+0000","last_became_active":"2024-05-09T08:48:18.371067+0000","last_became_peered":"2024-05-09T08:48:18.371067+0000","last_unstale":"2024-05-09T08:50:38.497468+0000","last_undegraded":"2024-05-09T08:50:38.497468+0000","last_fullsized":"2024-05-09T08:50:38.497468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:35:11.445992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743344+0000","last_change":"2024-05-09T08:48:18.382873+0000","last_active":"2024-05-09T08:50:38.743344+0000","last_peered":"2024-05-09T08:50:38.743344+0000","last_clean":"2024-05-09T08:50:38.743344+0000","last_became_active":"2024-05-09T08:48:18.382366+0000","last_became_peered":"2024-05-09T08:48:18.382366+0000","last_unstale":"2024-05-09T08:50:38.743344+0000","last_undegraded":"2024-05-09T08:50:38.743344+0000","last_fullsized":"2024-05-09T08:50:38.743344+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:23.343131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738362+0000","last_change":"2024-05-09T08:48:18.370821+0000","last_active":"2024-05-09T08:50:38.738362+0000","last_peered":"2024-05-09T08:50:38.738362+0000","last_clean":"2024-05-09T08:50:38.738362+0000","last_became_active":"2024-05-09T08:48:18.368208+0000","last_became_peered":"2024-05-09T08:48:18.368208+0000","last_unstale":"2024-05-09T08:50:38.738362+0000","last_undegraded":"2024-05-09T08:50:38.738362+0000","last_fullsized":"2024-05-09T08:50:38.738362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:14.633938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664977+0000","last_change":"2024-05-09T08:48:18.371626+0000","last_active":"2024-05-09T08:50:38.664977+0000","last_peered":"2024-05-09T08:50:38.664977+0000","last_clean":"2024-05-09T08:50:38.664977+0000","last_became_active":"2024-05-09T08:48:18.371081+0000","last_became_peered":"2024-05-09T08:48:18.371081+0000","last_unstale":"2024-05-09T08:50:38.664977+0000","last_undegraded":"2024-05-09T08:50:38.664977+0000","last_fullsized":"2024-05-09T08:50:38.664977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:27.372934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106929,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599687,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1352,"num_read_kb":793,"num_write":374,"num_write_kb":1468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":1799061,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351017,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":534442,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612654,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28216,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716936,"statfs":{"total":95995035648,"available":95966142464,"internally_reserved":0,"allocated":1224704,"data_stored":1323322,"data_compressed":71341,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776179,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":927129,"data_compressed":48688,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037817,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":927507,"data_compressed":48688,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299453,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":1132551,"data_compressed":65185,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462979,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":529808,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1118208,"data_stored":596742,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":531264,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1130496,"data_stored":597734,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1105920,"data_stored":1053941,"data_compressed":54844,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888141,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":534414,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724615,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1150976,"data_stored":1258419,"data_compressed":71341,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135260,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":200163,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:50:43.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:50:43.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.655+0000 7febef1a9640 1 -- 172.21.15.98:0/1867460311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe8106720 msgr2=0x7febe8106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:43.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.655+0000 7febef1a9640 1 --2- 172.21.15.98:0/1867460311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe8106720 0x7febe8106b00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7febe4009930 tx=0x7febe402a420 comp rx=0 tx=0).stop 2024-05-09T08:50:43.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.655+0000 7febef1a9640 1 -- 172.21.15.98:0/1867460311 shutdown_connections 2024-05-09T08:50:43.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.655+0000 7febef1a9640 1 --2- 172.21.15.98:0/1867460311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7febe81010d0 0x7febe810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.655+0000 7febef1a9640 1 --2- 172.21.15.98:0/1867460311 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febe8100710 0x7febe8100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.655+0000 7febef1a9640 1 --2- 172.21.15.98:0/1867460311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe8106720 0x7febe8106b00 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.655+0000 7febef1a9640 1 -- 172.21.15.98:0/1867460311 >> 172.21.15.98:0/1867460311 conn(0x7febe8077830 msgr2=0x7febe8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:43.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.655+0000 7febef1a9640 1 -- 172.21.15.98:0/1867460311 shutdown_connections 2024-05-09T08:50:43.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.655+0000 7febef1a9640 1 -- 172.21.15.98:0/1867460311 wait complete. 2024-05-09T08:50:43.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.655+0000 7febef1a9640 1 Processor -- start 2024-05-09T08:50:43.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.655+0000 7febef1a9640 1 -- start start 2024-05-09T08:50:43.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.655+0000 7febef1a9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febe8100710 0x7febe8198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:43.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.655+0000 7febef1a9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7febe81010d0 0x7febe8199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:43.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.655+0000 7febef1a9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe8106720 0x7febe81a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:43.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febef1a9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7febe8110b30 con 0x7febe8106720 2024-05-09T08:50:43.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febef1a9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7febe81109b0 con 0x7febe81010d0 2024-05-09T08:50:43.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febef1a9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7febe8110cb0 con 0x7febe8100710 2024-05-09T08:50:43.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febed746640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe8106720 0x7febe81a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:43.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febed746640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe8106720 0x7febe81a3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43202/0 (socket says 172.21.15.98:43202) 2024-05-09T08:50:43.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febed746640 1 -- 172.21.15.98:0/3057026157 learned_addr learned my addr 172.21.15.98:0/3057026157 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febecf45640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febe8100710 0x7febe8198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febdffff640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7febe81010d0 0x7febe8199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febed746640 1 -- 172.21.15.98:0/3057026157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febe8100710 msgr2=0x7febe8198f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febed746640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febe8100710 0x7febe8198f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febed746640 1 -- 172.21.15.98:0/3057026157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7febe81010d0 msgr2=0x7febe8199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febed746640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7febe81010d0 0x7febe8199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febed746640 1 -- 172.21.15.98:0/3057026157 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe81a44f0 con 0x7febe8106720 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febdffff640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7febe81010d0 0x7febe8199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febed746640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe8106720 0x7febe81a3fb0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7febe000cce0 tx=0x7febe000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febddffb640 1 -- 172.21.15.98:0/3057026157 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe000f450 con 0x7febe8106720 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febef1a9640 1 -- 172.21.15.98:0/3057026157 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe81a46c0 con 0x7febe8106720 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febddffb640 1 -- 172.21.15.98:0/3057026157 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febe000fad0 con 0x7febe8106720 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febddffb640 1 -- 172.21.15.98:0/3057026157 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe0018c10 con 0x7febe8106720 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.656+0000 7febef1a9640 1 -- 172.21.15.98:0/3057026157 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe81a4b90 con 0x7febe8106720 2024-05-09T08:50:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.658+0000 7febef1a9640 1 -- 172.21.15.98:0/3057026157 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febe8068650 con 0x7febe8106720 2024-05-09T08:50:43.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-05-09T08:50:43.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: Unable to set osd_memory_target on smithi142 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-05-09T08:50:43.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-09T08:50:43.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-09T08:50:43.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-05-09T08:50:43.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: Unable to set osd_memory_target on smithi098 to 184766668: error parsing value: Value '184766668' is below minimum 939524096 2024-05-09T08:50:43.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: Updating smithi098:/etc/ceph/ceph.conf 2024-05-09T08:50:43.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-09T08:50:43.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: Updating smithi142:/etc/ceph/ceph.conf 2024-05-09T08:50:43.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: Updating smithi105:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:50:43.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: Updating smithi098:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:50:43.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: Updating smithi142:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:50:43.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1474333009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:43.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2244295735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:43.661 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:50:43.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:50:43.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:50:43.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:43 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2625483248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:43.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.659+0000 7febddffb640 1 -- 172.21.15.98:0/3057026157 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7febe000f5f0 con 0x7febe8106720 2024-05-09T08:50:43.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.659+0000 7febddffb640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7febd00a2e20 0x7febd00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:43.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.659+0000 7febecf45640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7febd00a2e20 0x7febd00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:43.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.659+0000 7febddffb640 1 -- 172.21.15.98:0/3057026157 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7febe00d0140 con 0x7febe8106720 2024-05-09T08:50:43.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.660+0000 7febecf45640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7febd00a2e20 0x7febd00a52e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7febe4009900 tx=0x7febe40058d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:43.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.663+0000 7febddffb640 1 -- 172.21.15.98:0/3057026157 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7febe0014030 con 0x7febe8106720 2024-05-09T08:50:43.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-05-09T08:50:43.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: Unable to set osd_memory_target on smithi142 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-05-09T08:50:43.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-09T08:50:43.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-09T08:50:43.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-05-09T08:50:43.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: Unable to set osd_memory_target on smithi098 to 184766668: error parsing value: Value '184766668' is below minimum 939524096 2024-05-09T08:50:43.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: Updating smithi098:/etc/ceph/ceph.conf 2024-05-09T08:50:43.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-09T08:50:43.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: Updating smithi142:/etc/ceph/ceph.conf 2024-05-09T08:50:43.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: Updating smithi105:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:50:43.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: Updating smithi098:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:50:43.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: Updating smithi142:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:50:43.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1474333009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:43.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2244295735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:43.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:50:43.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:50:43.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:50:43.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:50:43.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:43 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2625483248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:43.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.836+0000 7febef1a9640 1 -- 172.21.15.98:0/3057026157 --> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7febe8061c70 con 0x7febd00a2e20 2024-05-09T08:50:43.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.846+0000 7febddffb640 1 -- 172.21.15.98:0/3057026157 <== mgr.15555 v2:172.21.15.98:6800/4136638623 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339730 (secure 0 0 0) 0x7febe8061c70 con 0x7febd00a2e20 2024-05-09T08:50:43.849 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:43.850 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-09T08:50:43.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.849+0000 7febef1a9640 1 -- 172.21.15.98:0/3057026157 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7febd00a2e20 msgr2=0x7febd00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:43.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.849+0000 7febef1a9640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7febd00a2e20 0x7febd00a52e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7febe4009900 tx=0x7febe40058d0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.849+0000 7febef1a9640 1 -- 172.21.15.98:0/3057026157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe8106720 msgr2=0x7febe81a3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:43.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.849+0000 7febef1a9640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe8106720 0x7febe81a3fb0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7febe000cce0 tx=0x7febe000a620 comp rx=0 tx=0).stop 2024-05-09T08:50:43.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.849+0000 7febef1a9640 1 -- 172.21.15.98:0/3057026157 shutdown_connections 2024-05-09T08:50:43.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.849+0000 7febef1a9640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7febd00a2e20 0x7febd00a52e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.849+0000 7febef1a9640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe8106720 0x7febe81a3fb0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.849+0000 7febef1a9640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7febe81010d0 0x7febe8199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.849+0000 7febef1a9640 1 --2- 172.21.15.98:0/3057026157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febe8100710 0x7febe8198f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:43.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.849+0000 7febef1a9640 1 -- 172.21.15.98:0/3057026157 >> 172.21.15.98:0/3057026157 conn(0x7febe8077830 msgr2=0x7febe8102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:43.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.850+0000 7febef1a9640 1 -- 172.21.15.98:0/3057026157 shutdown_connections 2024-05-09T08:50:43.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:43.850+0000 7febef1a9640 1 -- 172.21.15.98:0/3057026157 wait complete. 2024-05-09T08:50:43.860 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-05-09T08:50:42.510505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190289,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1488,"num_read_kb":912,"num_write":557,"num_write_kb":4408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1180921,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":942,"ondisk_log_size":942,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353840,"kb_used_data":12968,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124587984,"statfs":{"total":1151940427776,"available":1151578095616,"internally_reserved":0,"allocated":13279232,"data_stored":9947273,"data_compressed":537339,"data_compressed_allocated":724992,"data_compressed_original":4300800,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.987472"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665191+0000","last_change":"2024-05-09T08:48:18.373685+0000","last_active":"2024-05-09T08:50:38.665191+0000","last_peered":"2024-05-09T08:50:38.665191+0000","last_clean":"2024-05-09T08:50:38.665191+0000","last_became_active":"2024-05-09T08:48:18.373305+0000","last_became_peered":"2024-05-09T08:48:18.373305+0000","last_unstale":"2024-05-09T08:50:38.665191+0000","last_undegraded":"2024-05-09T08:50:38.665191+0000","last_fullsized":"2024-05-09T08:50:38.665191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:30:45.261706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740969+0000","last_change":"2024-05-09T08:48:18.373152+0000","last_active":"2024-05-09T08:50:38.740969+0000","last_peered":"2024-05-09T08:50:38.740969+0000","last_clean":"2024-05-09T08:50:38.740969+0000","last_became_active":"2024-05-09T08:48:18.372015+0000","last_became_peered":"2024-05-09T08:48:18.372015+0000","last_unstale":"2024-05-09T08:50:38.740969+0000","last_undegraded":"2024-05-09T08:50:38.740969+0000","last_fullsized":"2024-05-09T08:50:38.740969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:47.512778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740875+0000","last_change":"2024-05-09T08:48:18.370177+0000","last_active":"2024-05-09T08:50:38.740875+0000","last_peered":"2024-05-09T08:50:38.740875+0000","last_clean":"2024-05-09T08:50:38.740875+0000","last_became_active":"2024-05-09T08:48:18.369842+0000","last_became_peered":"2024-05-09T08:48:18.369842+0000","last_unstale":"2024-05-09T08:50:38.740875+0000","last_undegraded":"2024-05-09T08:50:38.740875+0000","last_fullsized":"2024-05-09T08:50:38.740875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:00:24.015500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738316+0000","last_change":"2024-05-09T08:48:18.375364+0000","last_active":"2024-05-09T08:50:38.738316+0000","last_peered":"2024-05-09T08:50:38.738316+0000","last_clean":"2024-05-09T08:50:38.738316+0000","last_became_active":"2024-05-09T08:48:18.375030+0000","last_became_peered":"2024-05-09T08:48:18.375030+0000","last_unstale":"2024-05-09T08:50:38.738316+0000","last_undegraded":"2024-05-09T08:50:38.738316+0000","last_fullsized":"2024-05-09T08:50:38.738316+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:43:57.684123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498390+0000","last_change":"2024-05-09T08:48:18.374548+0000","last_active":"2024-05-09T08:50:38.498390+0000","last_peered":"2024-05-09T08:50:38.498390+0000","last_clean":"2024-05-09T08:50:38.498390+0000","last_became_active":"2024-05-09T08:48:18.374190+0000","last_became_peered":"2024-05-09T08:48:18.374190+0000","last_unstale":"2024-05-09T08:50:38.498390+0000","last_undegraded":"2024-05-09T08:50:38.498390+0000","last_fullsized":"2024-05-09T08:50:38.498390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:43:48.456839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.501744+0000","last_change":"2024-05-09T08:48:18.371234+0000","last_active":"2024-05-09T08:50:38.501744+0000","last_peered":"2024-05-09T08:50:38.501744+0000","last_clean":"2024-05-09T08:50:38.501744+0000","last_became_active":"2024-05-09T08:48:18.370218+0000","last_became_peered":"2024-05-09T08:48:18.370218+0000","last_unstale":"2024-05-09T08:50:38.501744+0000","last_undegraded":"2024-05-09T08:50:38.501744+0000","last_fullsized":"2024-05-09T08:50:38.501744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:02:09.923146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739889+0000","last_change":"2024-05-09T08:49:18.366083+0000","last_active":"2024-05-09T08:50:38.739889+0000","last_peered":"2024-05-09T08:50:38.739889+0000","last_clean":"2024-05-09T08:50:38.739889+0000","last_became_active":"2024-05-09T08:49:18.365755+0000","last_became_peered":"2024-05-09T08:49:18.365755+0000","last_unstale":"2024-05-09T08:50:38.739889+0000","last_undegraded":"2024-05-09T08:50:38.739889+0000","last_fullsized":"2024-05-09T08:50:38.739889+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:40:18.052955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.596412+0000","last_change":"2024-05-09T08:49:18.488640+0000","last_active":"2024-05-09T08:50:38.596412+0000","last_peered":"2024-05-09T08:50:38.596412+0000","last_clean":"2024-05-09T08:50:38.596412+0000","last_became_active":"2024-05-09T08:49:18.488404+0000","last_became_peered":"2024-05-09T08:49:18.488404+0000","last_unstale":"2024-05-09T08:50:38.596412+0000","last_undegraded":"2024-05-09T08:50:38.596412+0000","last_fullsized":"2024-05-09T08:50:38.596412+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:47.317453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743849+0000","last_change":"2024-05-09T08:49:18.368485+0000","last_active":"2024-05-09T08:50:38.743849+0000","last_peered":"2024-05-09T08:50:38.743849+0000","last_clean":"2024-05-09T08:50:38.743849+0000","last_became_active":"2024-05-09T08:49:18.368175+0000","last_became_peered":"2024-05-09T08:49:18.368175+0000","last_unstale":"2024-05-09T08:50:38.743849+0000","last_undegraded":"2024-05-09T08:50:38.743849+0000","last_fullsized":"2024-05-09T08:50:38.743849+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:39:32.399209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.502258+0000","last_change":"2024-05-09T08:48:18.363668+0000","last_active":"2024-05-09T08:50:38.502258+0000","last_peered":"2024-05-09T08:50:38.502258+0000","last_clean":"2024-05-09T08:50:38.502258+0000","last_became_active":"2024-05-09T08:48:18.363248+0000","last_became_peered":"2024-05-09T08:48:18.363248+0000","last_unstale":"2024-05-09T08:50:38.502258+0000","last_undegraded":"2024-05-09T08:50:38.502258+0000","last_fullsized":"2024-05-09T08:50:38.502258+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:19:33.802350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740440+0000","last_change":"2024-05-09T08:48:18.376401+0000","last_active":"2024-05-09T08:50:38.740440+0000","last_peered":"2024-05-09T08:50:38.740440+0000","last_clean":"2024-05-09T08:50:38.740440+0000","last_became_active":"2024-05-09T08:48:18.375520+0000","last_became_peered":"2024-05-09T08:48:18.375520+0000","last_unstale":"2024-05-09T08:50:38.740440+0000","last_undegraded":"2024-05-09T08:50:38.740440+0000","last_fullsized":"2024-05-09T08:50:38.740440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:40:36.145071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738307+0000","last_change":"2024-05-09T08:48:18.372847+0000","last_active":"2024-05-09T08:50:38.738307+0000","last_peered":"2024-05-09T08:50:38.738307+0000","last_clean":"2024-05-09T08:50:38.738307+0000","last_became_active":"2024-05-09T08:48:18.370671+0000","last_became_peered":"2024-05-09T08:48:18.370671+0000","last_unstale":"2024-05-09T08:50:38.738307+0000","last_undegraded":"2024-05-09T08:50:38.738307+0000","last_fullsized":"2024-05-09T08:50:38.738307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:06:22.438515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738426+0000","last_change":"2024-05-09T08:48:18.365810+0000","last_active":"2024-05-09T08:50:38.738426+0000","last_peered":"2024-05-09T08:50:38.738426+0000","last_clean":"2024-05-09T08:50:38.738426+0000","last_became_active":"2024-05-09T08:48:18.365214+0000","last_became_peered":"2024-05-09T08:48:18.365214+0000","last_unstale":"2024-05-09T08:50:38.738426+0000","last_undegraded":"2024-05-09T08:50:38.738426+0000","last_fullsized":"2024-05-09T08:50:38.738426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:00.037153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740431+0000","last_change":"2024-05-09T08:48:18.375279+0000","last_active":"2024-05-09T08:50:38.740431+0000","last_peered":"2024-05-09T08:50:38.740431+0000","last_clean":"2024-05-09T08:50:38.740431+0000","last_became_active":"2024-05-09T08:48:18.374918+0000","last_became_peered":"2024-05-09T08:48:18.374918+0000","last_unstale":"2024-05-09T08:50:38.740431+0000","last_undegraded":"2024-05-09T08:50:38.740431+0000","last_fullsized":"2024-05-09T08:50:38.740431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:23:12.889283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740477+0000","last_change":"2024-05-09T08:49:18.369768+0000","last_active":"2024-05-09T08:50:38.740477+0000","last_peered":"2024-05-09T08:50:38.740477+0000","last_clean":"2024-05-09T08:50:38.740477+0000","last_became_active":"2024-05-09T08:49:18.369520+0000","last_became_peered":"2024-05-09T08:49:18.369520+0000","last_unstale":"2024-05-09T08:50:38.740477+0000","last_undegraded":"2024-05-09T08:50:38.740477+0000","last_fullsized":"2024-05-09T08:50:38.740477+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:44:19.502358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743522+0000","last_change":"2024-05-09T08:48:18.373905+0000","last_active":"2024-05-09T08:50:38.743522+0000","last_peered":"2024-05-09T08:50:38.743522+0000","last_clean":"2024-05-09T08:50:38.743522+0000","last_became_active":"2024-05-09T08:48:18.373566+0000","last_became_peered":"2024-05-09T08:48:18.373566+0000","last_unstale":"2024-05-09T08:50:38.743522+0000","last_undegraded":"2024-05-09T08:50:38.743522+0000","last_fullsized":"2024-05-09T08:50:38.743522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:33.905686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665140+0000","last_change":"2024-05-09T08:48:18.373970+0000","last_active":"2024-05-09T08:50:38.665140+0000","last_peered":"2024-05-09T08:50:38.665140+0000","last_clean":"2024-05-09T08:50:38.665140+0000","last_became_active":"2024-05-09T08:48:18.372696+0000","last_became_peered":"2024-05-09T08:48:18.372696+0000","last_unstale":"2024-05-09T08:50:38.665140+0000","last_undegraded":"2024-05-09T08:50:38.665140+0000","last_fullsized":"2024-05-09T08:50:38.665140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:09:33.751950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597421+0000","last_change":"2024-05-09T08:48:18.376851+0000","last_active":"2024-05-09T08:50:38.597421+0000","last_peered":"2024-05-09T08:50:38.597421+0000","last_clean":"2024-05-09T08:50:38.597421+0000","last_became_active":"2024-05-09T08:48:18.376230+0000","last_became_peered":"2024-05-09T08:48:18.376230+0000","last_unstale":"2024-05-09T08:50:38.597421+0000","last_undegraded":"2024-05-09T08:50:38.597421+0000","last_fullsized":"2024-05-09T08:50:38.597421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:34.959179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738942+0000","last_change":"2024-05-09T08:48:18.370287+0000","last_active":"2024-05-09T08:50:38.738942+0000","last_peered":"2024-05-09T08:50:38.738942+0000","last_clean":"2024-05-09T08:50:38.738942+0000","last_became_active":"2024-05-09T08:48:18.369956+0000","last_became_peered":"2024-05-09T08:48:18.369956+0000","last_unstale":"2024-05-09T08:50:38.738942+0000","last_undegraded":"2024-05-09T08:50:38.738942+0000","last_fullsized":"2024-05-09T08:50:38.738942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:19:34.746216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738556+0000","last_change":"2024-05-09T08:48:22.377973+0000","last_active":"2024-05-09T08:50:38.738556+0000","last_peered":"2024-05-09T08:50:38.738556+0000","last_clean":"2024-05-09T08:50:38.738556+0000","last_became_active":"2024-05-09T08:48:22.377325+0000","last_became_peered":"2024-05-09T08:48:22.377325+0000","last_unstale":"2024-05-09T08:50:38.738556+0000","last_undegraded":"2024-05-09T08:50:38.738556+0000","last_fullsized":"2024-05-09T08:50:38.738556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:32:00.873082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"95'14","reported_seq":169,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.744593+0000","last_change":"2024-05-09T08:48:17.595784+0000","last_active":"2024-05-09T08:50:38.744593+0000","last_peered":"2024-05-09T08:50:38.744593+0000","last_clean":"2024-05-09T08:50:38.744593+0000","last_became_active":"2024-05-09T08:48:17.595395+0000","last_became_peered":"2024-05-09T08:48:17.595395+0000","last_unstale":"2024-05-09T08:50:38.744593+0000","last_undegraded":"2024-05-09T08:50:38.744593+0000","last_fullsized":"2024-05-09T08:50:38.744593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:47.475148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497681+0000","last_change":"2024-05-09T08:48:18.382332+0000","last_active":"2024-05-09T08:50:38.497681+0000","last_peered":"2024-05-09T08:50:38.497681+0000","last_clean":"2024-05-09T08:50:38.497681+0000","last_became_active":"2024-05-09T08:48:18.381536+0000","last_became_peered":"2024-05-09T08:48:18.381536+0000","last_unstale":"2024-05-09T08:50:38.497681+0000","last_undegraded":"2024-05-09T08:50:38.497681+0000","last_fullsized":"2024-05-09T08:50:38.497681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:52:48.327000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737688+0000","last_change":"2024-05-09T08:48:22.379541+0000","last_active":"2024-05-09T08:50:38.737688+0000","last_peered":"2024-05-09T08:50:38.737688+0000","last_clean":"2024-05-09T08:50:38.737688+0000","last_became_active":"2024-05-09T08:48:22.379135+0000","last_became_peered":"2024-05-09T08:48:22.379135+0000","last_unstale":"2024-05-09T08:50:38.737688+0000","last_undegraded":"2024-05-09T08:50:38.737688+0000","last_fullsized":"2024-05-09T08:50:38.737688+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:51.365075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497915+0000","last_change":"2024-05-09T08:48:17.592168+0000","last_active":"2024-05-09T08:50:38.497915+0000","last_peered":"2024-05-09T08:50:38.497915+0000","last_clean":"2024-05-09T08:50:38.497915+0000","last_became_active":"2024-05-09T08:48:17.591822+0000","last_became_peered":"2024-05-09T08:48:17.591822+0000","last_unstale":"2024-05-09T08:50:38.497915+0000","last_undegraded":"2024-05-09T08:50:38.497915+0000","last_fullsized":"2024-05-09T08:50:38.497915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:45:15.162240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.741025+0000","last_change":"2024-05-09T08:48:18.379396+0000","last_active":"2024-05-09T08:50:38.741025+0000","last_peered":"2024-05-09T08:50:38.741025+0000","last_clean":"2024-05-09T08:50:38.741025+0000","last_became_active":"2024-05-09T08:48:18.378770+0000","last_became_peered":"2024-05-09T08:48:18.378770+0000","last_unstale":"2024-05-09T08:50:38.741025+0000","last_undegraded":"2024-05-09T08:50:38.741025+0000","last_fullsized":"2024-05-09T08:50:38.741025+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:22:22.699186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737898+0000","last_change":"2024-05-09T08:48:22.387777+0000","last_active":"2024-05-09T08:50:38.737898+0000","last_peered":"2024-05-09T08:50:38.737898+0000","last_clean":"2024-05-09T08:50:38.737898+0000","last_became_active":"2024-05-09T08:48:22.387300+0000","last_became_peered":"2024-05-09T08:48:22.387300+0000","last_unstale":"2024-05-09T08:50:38.737898+0000","last_undegraded":"2024-05-09T08:50:38.737898+0000","last_fullsized":"2024-05-09T08:50:38.737898+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:08:35.646374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"95'13","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498917+0000","last_change":"2024-05-09T08:48:17.602493+0000","last_active":"2024-05-09T08:50:38.498917+0000","last_peered":"2024-05-09T08:50:38.498917+0000","last_clean":"2024-05-09T08:50:38.498917+0000","last_became_active":"2024-05-09T08:48:17.601625+0000","last_became_peered":"2024-05-09T08:48:17.601625+0000","last_unstale":"2024-05-09T08:50:38.498917+0000","last_undegraded":"2024-05-09T08:50:38.498917+0000","last_fullsized":"2024-05-09T08:50:38.498917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:33.044532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743329+0000","last_change":"2024-05-09T08:48:18.380615+0000","last_active":"2024-05-09T08:50:38.743329+0000","last_peered":"2024-05-09T08:50:38.743329+0000","last_clean":"2024-05-09T08:50:38.743329+0000","last_became_active":"2024-05-09T08:48:18.380196+0000","last_became_peered":"2024-05-09T08:48:18.380196+0000","last_unstale":"2024-05-09T08:50:38.743329+0000","last_undegraded":"2024-05-09T08:50:38.743329+0000","last_fullsized":"2024-05-09T08:50:38.743329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:27.299263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737797+0000","last_change":"2024-05-09T08:48:22.381954+0000","last_active":"2024-05-09T08:50:38.737797+0000","last_peered":"2024-05-09T08:50:38.737797+0000","last_clean":"2024-05-09T08:50:38.737797+0000","last_became_active":"2024-05-09T08:48:22.381527+0000","last_became_peered":"2024-05-09T08:48:22.381527+0000","last_unstale":"2024-05-09T08:50:38.737797+0000","last_undegraded":"2024-05-09T08:50:38.737797+0000","last_fullsized":"2024-05-09T08:50:38.737797+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:35.170753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498547+0000","last_change":"2024-05-09T08:48:17.592795+0000","last_active":"2024-05-09T08:50:38.498547+0000","last_peered":"2024-05-09T08:50:38.498547+0000","last_clean":"2024-05-09T08:50:38.498547+0000","last_became_active":"2024-05-09T08:48:17.592240+0000","last_became_peered":"2024-05-09T08:48:17.592240+0000","last_unstale":"2024-05-09T08:50:38.498547+0000","last_undegraded":"2024-05-09T08:50:38.498547+0000","last_fullsized":"2024-05-09T08:50:38.498547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:27:30.056805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605470+0000","last_change":"2024-05-09T08:50:19.350043+0000","last_active":"2024-05-09T08:50:38.605470+0000","last_peered":"2024-05-09T08:50:38.605470+0000","last_clean":"2024-05-09T08:50:38.605470+0000","last_became_active":"2024-05-09T08:50:19.349640+0000","last_became_peered":"2024-05-09T08:50:19.349640+0000","last_unstale":"2024-05-09T08:50:38.605470+0000","last_undegraded":"2024-05-09T08:50:38.605470+0000","last_fullsized":"2024-05-09T08:50:38.605470+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:00:47.277071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738696+0000","last_change":"2024-05-09T08:48:22.374267+0000","last_active":"2024-05-09T08:50:38.738696+0000","last_peered":"2024-05-09T08:50:38.738696+0000","last_clean":"2024-05-09T08:50:38.738696+0000","last_became_active":"2024-05-09T08:48:22.373824+0000","last_became_peered":"2024-05-09T08:48:22.373824+0000","last_unstale":"2024-05-09T08:50:38.738696+0000","last_undegraded":"2024-05-09T08:50:38.738696+0000","last_fullsized":"2024-05-09T08:50:38.738696+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:27.407543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":168,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605692+0000","last_change":"2024-05-09T08:48:17.596851+0000","last_active":"2024-05-09T08:50:38.605692+0000","last_peered":"2024-05-09T08:50:38.605692+0000","last_clean":"2024-05-09T08:50:38.605692+0000","last_became_active":"2024-05-09T08:48:17.596532+0000","last_became_peered":"2024-05-09T08:48:17.596532+0000","last_unstale":"2024-05-09T08:50:38.605692+0000","last_undegraded":"2024-05-09T08:50:38.605692+0000","last_fullsized":"2024-05-09T08:50:38.605692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:58:51.923188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498499+0000","last_change":"2024-05-09T08:48:18.372214+0000","last_active":"2024-05-09T08:50:38.498499+0000","last_peered":"2024-05-09T08:50:38.498499+0000","last_clean":"2024-05-09T08:50:38.498499+0000","last_became_active":"2024-05-09T08:48:18.371850+0000","last_became_peered":"2024-05-09T08:48:18.371850+0000","last_unstale":"2024-05-09T08:50:38.498499+0000","last_undegraded":"2024-05-09T08:50:38.498499+0000","last_fullsized":"2024-05-09T08:50:38.498499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:56.971735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738968+0000","last_change":"2024-05-09T08:49:18.371344+0000","last_active":"2024-05-09T08:50:38.738968+0000","last_peered":"2024-05-09T08:50:38.738968+0000","last_clean":"2024-05-09T08:50:38.738968+0000","last_became_active":"2024-05-09T08:49:18.370911+0000","last_became_peered":"2024-05-09T08:49:18.370911+0000","last_unstale":"2024-05-09T08:50:38.738968+0000","last_undegraded":"2024-05-09T08:50:38.738968+0000","last_fullsized":"2024-05-09T08:50:38.738968+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:53:49.122965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,8],"acting":[2,6,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.501701+0000","last_change":"2024-05-09T08:48:17.601726+0000","last_active":"2024-05-09T08:50:38.501701+0000","last_peered":"2024-05-09T08:50:38.501701+0000","last_clean":"2024-05-09T08:50:38.501701+0000","last_became_active":"2024-05-09T08:48:17.601372+0000","last_became_peered":"2024-05-09T08:48:17.601372+0000","last_unstale":"2024-05-09T08:50:38.501701+0000","last_undegraded":"2024-05-09T08:50:38.501701+0000","last_fullsized":"2024-05-09T08:50:38.501701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:27:56.683821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605526+0000","last_change":"2024-05-09T08:49:18.694713+0000","last_active":"2024-05-09T08:50:38.605526+0000","last_peered":"2024-05-09T08:50:38.605526+0000","last_clean":"2024-05-09T08:50:38.605526+0000","last_became_active":"2024-05-09T08:49:18.694246+0000","last_became_peered":"2024-05-09T08:49:18.694246+0000","last_unstale":"2024-05-09T08:50:38.605526+0000","last_undegraded":"2024-05-09T08:50:38.605526+0000","last_fullsized":"2024-05-09T08:50:38.605526+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:05:30.818965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665087+0000","last_change":"2024-05-09T08:48:22.390990+0000","last_active":"2024-05-09T08:50:38.665087+0000","last_peered":"2024-05-09T08:50:38.665087+0000","last_clean":"2024-05-09T08:50:38.665087+0000","last_became_active":"2024-05-09T08:48:22.390601+0000","last_became_peered":"2024-05-09T08:48:22.390601+0000","last_unstale":"2024-05-09T08:50:38.665087+0000","last_undegraded":"2024-05-09T08:50:38.665087+0000","last_fullsized":"2024-05-09T08:50:38.665087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:47:24.644408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738010+0000","last_change":"2024-05-09T08:48:17.598008+0000","last_active":"2024-05-09T08:50:38.738010+0000","last_peered":"2024-05-09T08:50:38.738010+0000","last_clean":"2024-05-09T08:50:38.738010+0000","last_became_active":"2024-05-09T08:48:17.597514+0000","last_became_peered":"2024-05-09T08:48:17.597514+0000","last_unstale":"2024-05-09T08:50:38.738010+0000","last_undegraded":"2024-05-09T08:50:38.738010+0000","last_fullsized":"2024-05-09T08:50:38.738010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:18:32.788536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497274+0000","last_change":"2024-05-09T08:48:18.378061+0000","last_active":"2024-05-09T08:50:38.497274+0000","last_peered":"2024-05-09T08:50:38.497274+0000","last_clean":"2024-05-09T08:50:38.497274+0000","last_became_active":"2024-05-09T08:48:18.377684+0000","last_became_peered":"2024-05-09T08:48:18.377684+0000","last_unstale":"2024-05-09T08:50:38.497274+0000","last_undegraded":"2024-05-09T08:50:38.497274+0000","last_fullsized":"2024-05-09T08:50:38.497274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:13:08.338307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740944+0000","last_change":"2024-05-09T08:48:22.385779+0000","last_active":"2024-05-09T08:50:38.740944+0000","last_peered":"2024-05-09T08:50:38.740944+0000","last_clean":"2024-05-09T08:50:38.740944+0000","last_became_active":"2024-05-09T08:48:22.385021+0000","last_became_peered":"2024-05-09T08:48:22.385021+0000","last_unstale":"2024-05-09T08:50:38.740944+0000","last_undegraded":"2024-05-09T08:50:38.740944+0000","last_fullsized":"2024-05-09T08:50:38.740944+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:51.350754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605038+0000","last_change":"2024-05-09T08:48:17.595750+0000","last_active":"2024-05-09T08:50:38.605038+0000","last_peered":"2024-05-09T08:50:38.605038+0000","last_clean":"2024-05-09T08:50:38.605038+0000","last_became_active":"2024-05-09T08:48:17.595394+0000","last_became_peered":"2024-05-09T08:48:17.595394+0000","last_unstale":"2024-05-09T08:50:38.605038+0000","last_undegraded":"2024-05-09T08:50:38.605038+0000","last_fullsized":"2024-05-09T08:50:38.605038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:39:21.975059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.596769+0000","last_change":"2024-05-09T08:48:18.370750+0000","last_active":"2024-05-09T08:50:38.596769+0000","last_peered":"2024-05-09T08:50:38.596769+0000","last_clean":"2024-05-09T08:50:38.596769+0000","last_became_active":"2024-05-09T08:48:18.370383+0000","last_became_peered":"2024-05-09T08:48:18.370383+0000","last_unstale":"2024-05-09T08:50:38.596769+0000","last_undegraded":"2024-05-09T08:50:38.596769+0000","last_fullsized":"2024-05-09T08:50:38.596769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:36.049350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497318+0000","last_change":"2024-05-09T08:48:22.377157+0000","last_active":"2024-05-09T08:50:38.497318+0000","last_peered":"2024-05-09T08:50:38.497318+0000","last_clean":"2024-05-09T08:50:38.497318+0000","last_became_active":"2024-05-09T08:48:22.376730+0000","last_became_peered":"2024-05-09T08:48:22.376730+0000","last_unstale":"2024-05-09T08:50:38.497318+0000","last_undegraded":"2024-05-09T08:50:38.497318+0000","last_fullsized":"2024-05-09T08:50:38.497318+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:41:52.080730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"95'7","reported_seq":282,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.894752+0000","last_change":"2024-05-09T08:48:17.598528+0000","last_active":"2024-05-09T08:50:38.894752+0000","last_peered":"2024-05-09T08:50:38.894752+0000","last_clean":"2024-05-09T08:50:38.894752+0000","last_became_active":"2024-05-09T08:48:17.598072+0000","last_became_peered":"2024-05-09T08:48:17.598072+0000","last_unstale":"2024-05-09T08:50:38.894752+0000","last_undegraded":"2024-05-09T08:50:38.894752+0000","last_fullsized":"2024-05-09T08:50:38.894752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:35.697154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743731+0000","last_change":"2024-05-09T08:48:18.380919+0000","last_active":"2024-05-09T08:50:38.743731+0000","last_peered":"2024-05-09T08:50:38.743731+0000","last_clean":"2024-05-09T08:50:38.743731+0000","last_became_active":"2024-05-09T08:48:18.380485+0000","last_became_peered":"2024-05-09T08:48:18.380485+0000","last_unstale":"2024-05-09T08:50:38.743731+0000","last_undegraded":"2024-05-09T08:50:38.743731+0000","last_fullsized":"2024-05-09T08:50:38.743731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:45.079173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.596707+0000","last_change":"2024-05-09T08:48:22.389667+0000","last_active":"2024-05-09T08:50:38.596707+0000","last_peered":"2024-05-09T08:50:38.596707+0000","last_clean":"2024-05-09T08:50:38.596707+0000","last_became_active":"2024-05-09T08:48:22.389268+0000","last_became_peered":"2024-05-09T08:48:22.389268+0000","last_unstale":"2024-05-09T08:50:38.596707+0000","last_undegraded":"2024-05-09T08:50:38.596707+0000","last_fullsized":"2024-05-09T08:50:38.596707+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:33:14.861855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740395+0000","last_change":"2024-05-09T08:48:17.596028+0000","last_active":"2024-05-09T08:50:38.740395+0000","last_peered":"2024-05-09T08:50:38.740395+0000","last_clean":"2024-05-09T08:50:38.740395+0000","last_became_active":"2024-05-09T08:48:17.595053+0000","last_became_peered":"2024-05-09T08:48:17.595053+0000","last_unstale":"2024-05-09T08:50:38.740395+0000","last_undegraded":"2024-05-09T08:50:38.740395+0000","last_fullsized":"2024-05-09T08:50:38.740395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:48:31.323595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740976+0000","last_change":"2024-05-09T08:48:18.373472+0000","last_active":"2024-05-09T08:50:38.740976+0000","last_peered":"2024-05-09T08:50:38.740976+0000","last_clean":"2024-05-09T08:50:38.740976+0000","last_became_active":"2024-05-09T08:48:18.373178+0000","last_became_peered":"2024-05-09T08:48:18.373178+0000","last_unstale":"2024-05-09T08:50:38.740976+0000","last_undegraded":"2024-05-09T08:50:38.740976+0000","last_fullsized":"2024-05-09T08:50:38.740976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:18.048722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738133+0000","last_change":"2024-05-09T08:48:22.386551+0000","last_active":"2024-05-09T08:50:38.738133+0000","last_peered":"2024-05-09T08:50:38.738133+0000","last_clean":"2024-05-09T08:50:38.738133+0000","last_became_active":"2024-05-09T08:48:22.386018+0000","last_became_peered":"2024-05-09T08:48:22.386018+0000","last_unstale":"2024-05-09T08:50:38.738133+0000","last_undegraded":"2024-05-09T08:50:38.738133+0000","last_fullsized":"2024-05-09T08:50:38.738133+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:59.812030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605522+0000","last_change":"2024-05-09T08:48:17.601207+0000","last_active":"2024-05-09T08:50:38.605522+0000","last_peered":"2024-05-09T08:50:38.605522+0000","last_clean":"2024-05-09T08:50:38.605522+0000","last_became_active":"2024-05-09T08:48:17.600878+0000","last_became_peered":"2024-05-09T08:48:17.600878+0000","last_unstale":"2024-05-09T08:50:38.605522+0000","last_undegraded":"2024-05-09T08:50:38.605522+0000","last_fullsized":"2024-05-09T08:50:38.605522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:30.040527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743577+0000","last_change":"2024-05-09T08:48:18.381577+0000","last_active":"2024-05-09T08:50:38.743577+0000","last_peered":"2024-05-09T08:50:38.743577+0000","last_clean":"2024-05-09T08:50:38.743577+0000","last_became_active":"2024-05-09T08:48:18.381223+0000","last_became_peered":"2024-05-09T08:48:18.381223+0000","last_unstale":"2024-05-09T08:50:38.743577+0000","last_undegraded":"2024-05-09T08:50:38.743577+0000","last_fullsized":"2024-05-09T08:50:38.743577+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:57.038392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.596698+0000","last_change":"2024-05-09T08:48:22.385885+0000","last_active":"2024-05-09T08:50:38.596698+0000","last_peered":"2024-05-09T08:50:38.596698+0000","last_clean":"2024-05-09T08:50:38.596698+0000","last_became_active":"2024-05-09T08:48:22.385403+0000","last_became_peered":"2024-05-09T08:48:22.385403+0000","last_unstale":"2024-05-09T08:50:38.596698+0000","last_undegraded":"2024-05-09T08:50:38.596698+0000","last_fullsized":"2024-05-09T08:50:38.596698+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:53:20.733523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'39","reported_seq":327,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.895223+0000","last_change":"2024-05-09T08:48:17.597848+0000","last_active":"2024-05-09T08:50:38.895223+0000","last_peered":"2024-05-09T08:50:38.895223+0000","last_clean":"2024-05-09T08:50:38.895223+0000","last_became_active":"2024-05-09T08:48:17.596989+0000","last_became_peered":"2024-05-09T08:48:17.596989+0000","last_unstale":"2024-05-09T08:50:38.895223+0000","last_undegraded":"2024-05-09T08:50:38.895223+0000","last_fullsized":"2024-05-09T08:50:38.895223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:07.256301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65904,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":254,"num_read_kb":192,"num_write":41,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498676+0000","last_change":"2024-05-09T08:48:18.365308+0000","last_active":"2024-05-09T08:50:38.498676+0000","last_peered":"2024-05-09T08:50:38.498676+0000","last_clean":"2024-05-09T08:50:38.498676+0000","last_became_active":"2024-05-09T08:48:18.364845+0000","last_became_peered":"2024-05-09T08:48:18.364845+0000","last_unstale":"2024-05-09T08:50:38.498676+0000","last_undegraded":"2024-05-09T08:50:38.498676+0000","last_fullsized":"2024-05-09T08:50:38.498676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:07.277730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605765+0000","last_change":"2024-05-09T08:48:22.377333+0000","last_active":"2024-05-09T08:50:38.605765+0000","last_peered":"2024-05-09T08:50:38.605765+0000","last_clean":"2024-05-09T08:50:38.605765+0000","last_became_active":"2024-05-09T08:48:22.376494+0000","last_became_peered":"2024-05-09T08:48:22.376494+0000","last_unstale":"2024-05-09T08:50:38.605765+0000","last_undegraded":"2024-05-09T08:50:38.605765+0000","last_fullsized":"2024-05-09T08:50:38.605765+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:49:17.693977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664772+0000","last_change":"2024-05-09T08:50:19.301723+0000","last_active":"2024-05-09T08:50:38.664772+0000","last_peered":"2024-05-09T08:50:38.664772+0000","last_clean":"2024-05-09T08:50:38.664772+0000","last_became_active":"2024-05-09T08:50:19.291600+0000","last_became_peered":"2024-05-09T08:50:19.291600+0000","last_unstale":"2024-05-09T08:50:38.664772+0000","last_undegraded":"2024-05-09T08:50:38.664772+0000","last_fullsized":"2024-05-09T08:50:38.664772+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:29.105282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738296+0000","last_change":"2024-05-09T08:48:18.373652+0000","last_active":"2024-05-09T08:50:38.738296+0000","last_peered":"2024-05-09T08:50:38.738296+0000","last_clean":"2024-05-09T08:50:38.738296+0000","last_became_active":"2024-05-09T08:48:18.373345+0000","last_became_peered":"2024-05-09T08:48:18.373345+0000","last_unstale":"2024-05-09T08:50:38.738296+0000","last_undegraded":"2024-05-09T08:50:38.738296+0000","last_fullsized":"2024-05-09T08:50:38.738296+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:14:34.843305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664855+0000","last_change":"2024-05-09T08:49:18.369432+0000","last_active":"2024-05-09T08:50:38.664855+0000","last_peered":"2024-05-09T08:50:38.664855+0000","last_clean":"2024-05-09T08:50:38.664855+0000","last_became_active":"2024-05-09T08:49:18.368978+0000","last_became_peered":"2024-05-09T08:49:18.368978+0000","last_unstale":"2024-05-09T08:50:38.664855+0000","last_undegraded":"2024-05-09T08:50:38.664855+0000","last_fullsized":"2024-05-09T08:50:38.664855+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:20.969162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605525+0000","last_change":"2024-05-09T08:48:17.592658+0000","last_active":"2024-05-09T08:50:38.605525+0000","last_peered":"2024-05-09T08:50:38.605525+0000","last_clean":"2024-05-09T08:50:38.605525+0000","last_became_active":"2024-05-09T08:48:17.592275+0000","last_became_peered":"2024-05-09T08:48:17.592275+0000","last_unstale":"2024-05-09T08:50:38.605525+0000","last_undegraded":"2024-05-09T08:50:38.605525+0000","last_fullsized":"2024-05-09T08:50:38.605525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:26:56.742079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740268+0000","last_change":"2024-05-09T08:49:18.364399+0000","last_active":"2024-05-09T08:50:38.740268+0000","last_peered":"2024-05-09T08:50:38.740268+0000","last_clean":"2024-05-09T08:50:38.740268+0000","last_became_active":"2024-05-09T08:49:18.364166+0000","last_became_peered":"2024-05-09T08:49:18.364166+0000","last_unstale":"2024-05-09T08:50:38.740268+0000","last_undegraded":"2024-05-09T08:50:38.740268+0000","last_fullsized":"2024-05-09T08:50:38.740268+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:49:36.480337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597700+0000","last_change":"2024-05-09T08:48:22.389568+0000","last_active":"2024-05-09T08:50:38.597700+0000","last_peered":"2024-05-09T08:50:38.597700+0000","last_clean":"2024-05-09T08:50:38.597700+0000","last_became_active":"2024-05-09T08:48:22.387192+0000","last_became_peered":"2024-05-09T08:48:22.387192+0000","last_unstale":"2024-05-09T08:50:38.597700+0000","last_undegraded":"2024-05-09T08:50:38.597700+0000","last_fullsized":"2024-05-09T08:50:38.597700+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:56.794208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"95'8","reported_seq":169,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.895068+0000","last_change":"2024-05-09T08:48:17.598760+0000","last_active":"2024-05-09T08:50:38.895068+0000","last_peered":"2024-05-09T08:50:38.895068+0000","last_clean":"2024-05-09T08:50:38.895068+0000","last_became_active":"2024-05-09T08:48:17.598192+0000","last_became_peered":"2024-05-09T08:48:17.598192+0000","last_unstale":"2024-05-09T08:50:38.895068+0000","last_undegraded":"2024-05-09T08:50:38.895068+0000","last_fullsized":"2024-05-09T08:50:38.895068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:29.007644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":128,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739920+0000","last_change":"2024-05-09T08:48:18.364570+0000","last_active":"2024-05-09T08:50:38.739920+0000","last_peered":"2024-05-09T08:50:38.739920+0000","last_clean":"2024-05-09T08:50:38.739920+0000","last_became_active":"2024-05-09T08:48:18.364191+0000","last_became_peered":"2024-05-09T08:48:18.364191+0000","last_unstale":"2024-05-09T08:50:38.739920+0000","last_undegraded":"2024-05-09T08:50:38.739920+0000","last_fullsized":"2024-05-09T08:50:38.739920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:49.267954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737704+0000","last_change":"2024-05-09T08:49:18.694742+0000","last_active":"2024-05-09T08:50:38.737704+0000","last_peered":"2024-05-09T08:50:38.737704+0000","last_clean":"2024-05-09T08:50:38.737704+0000","last_became_active":"2024-05-09T08:49:18.694241+0000","last_became_peered":"2024-05-09T08:49:18.694241+0000","last_unstale":"2024-05-09T08:50:38.737704+0000","last_undegraded":"2024-05-09T08:50:38.737704+0000","last_fullsized":"2024-05-09T08:50:38.737704+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:20.289651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498205+0000","last_change":"2024-05-09T08:48:17.599762+0000","last_active":"2024-05-09T08:50:38.498205+0000","last_peered":"2024-05-09T08:50:38.498205+0000","last_clean":"2024-05-09T08:50:38.498205+0000","last_became_active":"2024-05-09T08:48:17.599355+0000","last_became_peered":"2024-05-09T08:48:17.599355+0000","last_unstale":"2024-05-09T08:50:38.498205+0000","last_undegraded":"2024-05-09T08:50:38.498205+0000","last_fullsized":"2024-05-09T08:50:38.498205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:28:36.836403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605208+0000","last_change":"2024-05-09T08:48:18.372810+0000","last_active":"2024-05-09T08:50:38.605208+0000","last_peered":"2024-05-09T08:50:38.605208+0000","last_clean":"2024-05-09T08:50:38.605208+0000","last_became_active":"2024-05-09T08:48:18.372437+0000","last_became_peered":"2024-05-09T08:48:18.372437+0000","last_unstale":"2024-05-09T08:50:38.605208+0000","last_undegraded":"2024-05-09T08:50:38.605208+0000","last_fullsized":"2024-05-09T08:50:38.605208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:03:36.879719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605358+0000","last_change":"2024-05-09T08:48:22.379370+0000","last_active":"2024-05-09T08:50:38.605358+0000","last_peered":"2024-05-09T08:50:38.605358+0000","last_clean":"2024-05-09T08:50:38.605358+0000","last_became_active":"2024-05-09T08:48:22.378994+0000","last_became_peered":"2024-05-09T08:48:22.378994+0000","last_unstale":"2024-05-09T08:50:38.605358+0000","last_undegraded":"2024-05-09T08:50:38.605358+0000","last_fullsized":"2024-05-09T08:50:38.605358+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:08.005647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498896+0000","last_change":"2024-05-09T08:48:17.594236+0000","last_active":"2024-05-09T08:50:38.498896+0000","last_peered":"2024-05-09T08:50:38.498896+0000","last_clean":"2024-05-09T08:50:38.498896+0000","last_became_active":"2024-05-09T08:48:17.593418+0000","last_became_peered":"2024-05-09T08:48:17.593418+0000","last_unstale":"2024-05-09T08:50:38.498896+0000","last_undegraded":"2024-05-09T08:50:38.498896+0000","last_fullsized":"2024-05-09T08:50:38.498896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:23.930421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665072+0000","last_change":"2024-05-09T08:48:18.366767+0000","last_active":"2024-05-09T08:50:38.665072+0000","last_peered":"2024-05-09T08:50:38.665072+0000","last_clean":"2024-05-09T08:50:38.665072+0000","last_became_active":"2024-05-09T08:48:18.366435+0000","last_became_peered":"2024-05-09T08:48:18.366435+0000","last_unstale":"2024-05-09T08:50:38.665072+0000","last_undegraded":"2024-05-09T08:50:38.665072+0000","last_fullsized":"2024-05-09T08:50:38.665072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:45.111510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498398+0000","last_change":"2024-05-09T08:48:22.378642+0000","last_active":"2024-05-09T08:50:38.498398+0000","last_peered":"2024-05-09T08:50:38.498398+0000","last_clean":"2024-05-09T08:50:38.498398+0000","last_became_active":"2024-05-09T08:48:22.378161+0000","last_became_peered":"2024-05-09T08:48:22.378161+0000","last_unstale":"2024-05-09T08:50:38.498398+0000","last_undegraded":"2024-05-09T08:50:38.498398+0000","last_fullsized":"2024-05-09T08:50:38.498398+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:53.149728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597189+0000","last_change":"2024-05-09T08:48:17.599499+0000","last_active":"2024-05-09T08:50:38.597189+0000","last_peered":"2024-05-09T08:50:38.597189+0000","last_clean":"2024-05-09T08:50:38.597189+0000","last_became_active":"2024-05-09T08:48:17.599145+0000","last_became_peered":"2024-05-09T08:48:17.599145+0000","last_unstale":"2024-05-09T08:50:38.597189+0000","last_undegraded":"2024-05-09T08:50:38.597189+0000","last_fullsized":"2024-05-09T08:50:38.597189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:31:12.502004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.502012+0000","last_change":"2024-05-09T08:48:18.372355+0000","last_active":"2024-05-09T08:50:38.502012+0000","last_peered":"2024-05-09T08:50:38.502012+0000","last_clean":"2024-05-09T08:50:38.502012+0000","last_became_active":"2024-05-09T08:48:18.367743+0000","last_became_peered":"2024-05-09T08:48:18.367743+0000","last_unstale":"2024-05-09T08:50:38.502012+0000","last_undegraded":"2024-05-09T08:50:38.502012+0000","last_fullsized":"2024-05-09T08:50:38.502012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:31:12.174350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738469+0000","last_change":"2024-05-09T08:48:22.384064+0000","last_active":"2024-05-09T08:50:38.738469+0000","last_peered":"2024-05-09T08:50:38.738469+0000","last_clean":"2024-05-09T08:50:38.738469+0000","last_became_active":"2024-05-09T08:48:22.383670+0000","last_became_peered":"2024-05-09T08:48:22.383670+0000","last_unstale":"2024-05-09T08:50:38.738469+0000","last_undegraded":"2024-05-09T08:50:38.738469+0000","last_fullsized":"2024-05-09T08:50:38.738469+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:51:35.000827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"95'6","reported_seq":162,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.922953+0000","last_change":"2024-05-09T08:48:17.601596+0000","last_active":"2024-05-09T08:50:38.922953+0000","last_peered":"2024-05-09T08:50:38.922953+0000","last_clean":"2024-05-09T08:50:38.922953+0000","last_became_active":"2024-05-09T08:48:17.601250+0000","last_became_peered":"2024-05-09T08:48:17.601250+0000","last_unstale":"2024-05-09T08:50:38.922953+0000","last_undegraded":"2024-05-09T08:50:38.922953+0000","last_fullsized":"2024-05-09T08:50:38.922953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:48.237336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.501885+0000","last_change":"2024-05-09T08:48:18.368751+0000","last_active":"2024-05-09T08:50:38.501885+0000","last_peered":"2024-05-09T08:50:38.501885+0000","last_clean":"2024-05-09T08:50:38.501885+0000","last_became_active":"2024-05-09T08:48:18.367463+0000","last_became_peered":"2024-05-09T08:48:18.367463+0000","last_unstale":"2024-05-09T08:50:38.501885+0000","last_undegraded":"2024-05-09T08:50:38.501885+0000","last_fullsized":"2024-05-09T08:50:38.501885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:19:17.696602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738886+0000","last_change":"2024-05-09T08:48:22.385025+0000","last_active":"2024-05-09T08:50:38.738886+0000","last_peered":"2024-05-09T08:50:38.738886+0000","last_clean":"2024-05-09T08:50:38.738886+0000","last_became_active":"2024-05-09T08:48:22.384633+0000","last_became_peered":"2024-05-09T08:48:22.384633+0000","last_unstale":"2024-05-09T08:50:38.738886+0000","last_undegraded":"2024-05-09T08:50:38.738886+0000","last_fullsized":"2024-05-09T08:50:38.738886+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:17.804075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605162+0000","last_change":"2024-05-09T08:48:18.376741+0000","last_active":"2024-05-09T08:50:38.605162+0000","last_peered":"2024-05-09T08:50:38.605162+0000","last_clean":"2024-05-09T08:50:38.605162+0000","last_became_active":"2024-05-09T08:48:18.376376+0000","last_became_peered":"2024-05-09T08:48:18.376376+0000","last_unstale":"2024-05-09T08:50:38.605162+0000","last_undegraded":"2024-05-09T08:50:38.605162+0000","last_fullsized":"2024-05-09T08:50:38.605162+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:58:31.717555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"95'7","reported_seq":164,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740200+0000","last_change":"2024-05-09T08:48:17.598316+0000","last_active":"2024-05-09T08:50:38.740200+0000","last_peered":"2024-05-09T08:50:38.740200+0000","last_clean":"2024-05-09T08:50:38.740200+0000","last_became_active":"2024-05-09T08:48:17.598022+0000","last_became_peered":"2024-05-09T08:48:17.598022+0000","last_unstale":"2024-05-09T08:50:38.740200+0000","last_undegraded":"2024-05-09T08:50:38.740200+0000","last_fullsized":"2024-05-09T08:50:38.740200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:04:10.454894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740178+0000","last_change":"2024-05-09T08:49:18.492981+0000","last_active":"2024-05-09T08:50:38.740178+0000","last_peered":"2024-05-09T08:50:38.740178+0000","last_clean":"2024-05-09T08:50:38.740178+0000","last_became_active":"2024-05-09T08:49:18.489169+0000","last_became_peered":"2024-05-09T08:49:18.489169+0000","last_unstale":"2024-05-09T08:50:38.740178+0000","last_undegraded":"2024-05-09T08:50:38.740178+0000","last_fullsized":"2024-05-09T08:50:38.740178+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:02:53.955587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497378+0000","last_change":"2024-05-09T08:48:18.379203+0000","last_active":"2024-05-09T08:50:38.497378+0000","last_peered":"2024-05-09T08:50:38.497378+0000","last_clean":"2024-05-09T08:50:38.497378+0000","last_became_active":"2024-05-09T08:48:18.378885+0000","last_became_peered":"2024-05-09T08:48:18.378885+0000","last_unstale":"2024-05-09T08:50:38.497378+0000","last_undegraded":"2024-05-09T08:50:38.497378+0000","last_fullsized":"2024-05-09T08:50:38.497378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:40.588263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737925+0000","last_change":"2024-05-09T08:48:17.593520+0000","last_active":"2024-05-09T08:50:38.737925+0000","last_peered":"2024-05-09T08:50:38.737925+0000","last_clean":"2024-05-09T08:50:38.737925+0000","last_became_active":"2024-05-09T08:48:17.593127+0000","last_became_peered":"2024-05-09T08:48:17.593127+0000","last_unstale":"2024-05-09T08:50:38.737925+0000","last_undegraded":"2024-05-09T08:50:38.737925+0000","last_fullsized":"2024-05-09T08:50:38.737925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:34:29.567166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":367,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739767+0000","last_change":"2024-05-09T08:47:05.718649+0000","last_active":"2024-05-09T08:50:38.739767+0000","last_peered":"2024-05-09T08:50:38.739767+0000","last_clean":"2024-05-09T08:50:38.739767+0000","last_became_active":"2024-05-09T08:47:05.718088+0000","last_became_peered":"2024-05-09T08:47:05.718088+0000","last_unstale":"2024-05-09T08:50:38.739767+0000","last_undegraded":"2024-05-09T08:50:38.739767+0000","last_fullsized":"2024-05-09T08:50:38.739767+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:43:39.457880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:43:39.457880+0000","last_clean_scrub_stamp":"2024-05-09T08:43:39.457880+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:14.227590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605456+0000","last_change":"2024-05-09T08:48:22.376751+0000","last_active":"2024-05-09T08:50:38.605456+0000","last_peered":"2024-05-09T08:50:38.605456+0000","last_clean":"2024-05-09T08:50:38.605456+0000","last_became_active":"2024-05-09T08:48:22.376322+0000","last_became_peered":"2024-05-09T08:48:22.376322+0000","last_unstale":"2024-05-09T08:50:38.605456+0000","last_undegraded":"2024-05-09T08:50:38.605456+0000","last_fullsized":"2024-05-09T08:50:38.605456+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:08:07.859034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737852+0000","last_change":"2024-05-09T08:48:18.364063+0000","last_active":"2024-05-09T08:50:38.737852+0000","last_peered":"2024-05-09T08:50:38.737852+0000","last_clean":"2024-05-09T08:50:38.737852+0000","last_became_active":"2024-05-09T08:48:18.363694+0000","last_became_peered":"2024-05-09T08:48:18.363694+0000","last_unstale":"2024-05-09T08:50:38.737852+0000","last_undegraded":"2024-05-09T08:50:38.737852+0000","last_fullsized":"2024-05-09T08:50:38.737852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:42.536329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739870+0000","last_change":"2024-05-09T08:48:17.599787+0000","last_active":"2024-05-09T08:50:38.739870+0000","last_peered":"2024-05-09T08:50:38.739870+0000","last_clean":"2024-05-09T08:50:38.739870+0000","last_became_active":"2024-05-09T08:48:17.599248+0000","last_became_peered":"2024-05-09T08:48:17.599248+0000","last_unstale":"2024-05-09T08:50:38.739870+0000","last_undegraded":"2024-05-09T08:50:38.739870+0000","last_fullsized":"2024-05-09T08:50:38.739870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:17.448404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664823+0000","last_change":"2024-05-09T08:48:22.378913+0000","last_active":"2024-05-09T08:50:38.664823+0000","last_peered":"2024-05-09T08:50:38.664823+0000","last_clean":"2024-05-09T08:50:38.664823+0000","last_became_active":"2024-05-09T08:48:22.378441+0000","last_became_peered":"2024-05-09T08:48:22.378441+0000","last_unstale":"2024-05-09T08:50:38.664823+0000","last_undegraded":"2024-05-09T08:50:38.664823+0000","last_fullsized":"2024-05-09T08:50:38.664823+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:49.623778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498746+0000","last_change":"2024-05-09T08:48:17.600121+0000","last_active":"2024-05-09T08:50:38.498746+0000","last_peered":"2024-05-09T08:50:38.498746+0000","last_clean":"2024-05-09T08:50:38.498746+0000","last_became_active":"2024-05-09T08:48:17.599800+0000","last_became_peered":"2024-05-09T08:48:17.599800+0000","last_unstale":"2024-05-09T08:50:38.498746+0000","last_undegraded":"2024-05-09T08:50:38.498746+0000","last_fullsized":"2024-05-09T08:50:38.498746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:56.607393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738578+0000","last_change":"2024-05-09T08:48:18.366612+0000","last_active":"2024-05-09T08:50:38.738578+0000","last_peered":"2024-05-09T08:50:38.738578+0000","last_clean":"2024-05-09T08:50:38.738578+0000","last_became_active":"2024-05-09T08:48:18.366266+0000","last_became_peered":"2024-05-09T08:48:18.366266+0000","last_unstale":"2024-05-09T08:50:38.738578+0000","last_undegraded":"2024-05-09T08:50:38.738578+0000","last_fullsized":"2024-05-09T08:50:38.738578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:35:01.663456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497267+0000","last_change":"2024-05-09T08:48:22.384096+0000","last_active":"2024-05-09T08:50:38.497267+0000","last_peered":"2024-05-09T08:50:38.497267+0000","last_clean":"2024-05-09T08:50:38.497267+0000","last_became_active":"2024-05-09T08:48:22.383656+0000","last_became_peered":"2024-05-09T08:48:22.383656+0000","last_unstale":"2024-05-09T08:50:38.497267+0000","last_undegraded":"2024-05-09T08:50:38.497267+0000","last_fullsized":"2024-05-09T08:50:38.497267+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:13.622377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'48","reported_seq":105,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.707713+0000","last_change":"2024-05-09T08:48:17.593684+0000","last_active":"2024-05-09T08:50:38.707713+0000","last_peered":"2024-05-09T08:50:38.707713+0000","last_clean":"2024-05-09T08:50:38.707713+0000","last_became_active":"2024-05-09T08:48:17.593358+0000","last_became_peered":"2024-05-09T08:48:17.593358+0000","last_unstale":"2024-05-09T08:50:38.707713+0000","last_undegraded":"2024-05-09T08:50:38.707713+0000","last_fullsized":"2024-05-09T08:50:38.707713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:51:36.023421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497783+0000","last_change":"2024-05-09T08:48:18.374498+0000","last_active":"2024-05-09T08:50:38.497783+0000","last_peered":"2024-05-09T08:50:38.497783+0000","last_clean":"2024-05-09T08:50:38.497783+0000","last_became_active":"2024-05-09T08:48:18.374142+0000","last_became_peered":"2024-05-09T08:48:18.374142+0000","last_unstale":"2024-05-09T08:50:38.497783+0000","last_undegraded":"2024-05-09T08:50:38.497783+0000","last_fullsized":"2024-05-09T08:50:38.497783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:53.874419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738424+0000","last_change":"2024-05-09T08:48:22.378330+0000","last_active":"2024-05-09T08:50:38.738424+0000","last_peered":"2024-05-09T08:50:38.738424+0000","last_clean":"2024-05-09T08:50:38.738424+0000","last_became_active":"2024-05-09T08:48:22.377218+0000","last_became_peered":"2024-05-09T08:48:22.377218+0000","last_unstale":"2024-05-09T08:50:38.738424+0000","last_undegraded":"2024-05-09T08:50:38.738424+0000","last_fullsized":"2024-05-09T08:50:38.738424+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:41.650168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738328+0000","last_change":"2024-05-09T08:48:17.601410+0000","last_active":"2024-05-09T08:50:38.738328+0000","last_peered":"2024-05-09T08:50:38.738328+0000","last_clean":"2024-05-09T08:50:38.738328+0000","last_became_active":"2024-05-09T08:48:17.601075+0000","last_became_peered":"2024-05-09T08:48:17.601075+0000","last_unstale":"2024-05-09T08:50:38.738328+0000","last_undegraded":"2024-05-09T08:50:38.738328+0000","last_fullsized":"2024-05-09T08:50:38.738328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:45.760280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.741119+0000","last_change":"2024-05-09T08:48:18.379590+0000","last_active":"2024-05-09T08:50:38.741119+0000","last_peered":"2024-05-09T08:50:38.741119+0000","last_clean":"2024-05-09T08:50:38.741119+0000","last_became_active":"2024-05-09T08:48:18.379187+0000","last_became_peered":"2024-05-09T08:48:18.379187+0000","last_unstale":"2024-05-09T08:50:38.741119+0000","last_undegraded":"2024-05-09T08:50:38.741119+0000","last_fullsized":"2024-05-09T08:50:38.741119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:40:02.630449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737875+0000","last_change":"2024-05-09T08:48:22.386437+0000","last_active":"2024-05-09T08:50:38.737875+0000","last_peered":"2024-05-09T08:50:38.737875+0000","last_clean":"2024-05-09T08:50:38.737875+0000","last_became_active":"2024-05-09T08:48:22.385973+0000","last_became_peered":"2024-05-09T08:48:22.385973+0000","last_unstale":"2024-05-09T08:50:38.737875+0000","last_undegraded":"2024-05-09T08:50:38.737875+0000","last_fullsized":"2024-05-09T08:50:38.737875+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:11:37.338641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743068+0000","last_change":"2024-05-09T08:48:17.598586+0000","last_active":"2024-05-09T08:50:38.743068+0000","last_peered":"2024-05-09T08:50:38.743068+0000","last_clean":"2024-05-09T08:50:38.743068+0000","last_became_active":"2024-05-09T08:48:17.597851+0000","last_became_peered":"2024-05-09T08:48:17.597851+0000","last_unstale":"2024-05-09T08:50:38.743068+0000","last_undegraded":"2024-05-09T08:50:38.743068+0000","last_fullsized":"2024-05-09T08:50:38.743068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:28:22.996999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498169+0000","last_change":"2024-05-09T08:48:18.367319+0000","last_active":"2024-05-09T08:50:38.498169+0000","last_peered":"2024-05-09T08:50:38.498169+0000","last_clean":"2024-05-09T08:50:38.498169+0000","last_became_active":"2024-05-09T08:48:18.366034+0000","last_became_peered":"2024-05-09T08:48:18.366034+0000","last_unstale":"2024-05-09T08:50:38.498169+0000","last_undegraded":"2024-05-09T08:50:38.498169+0000","last_fullsized":"2024-05-09T08:50:38.498169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:22.998089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597628+0000","last_change":"2024-05-09T08:48:22.389129+0000","last_active":"2024-05-09T08:50:38.597628+0000","last_peered":"2024-05-09T08:50:38.597628+0000","last_clean":"2024-05-09T08:50:38.597628+0000","last_became_active":"2024-05-09T08:48:22.386639+0000","last_became_peered":"2024-05-09T08:48:22.386639+0000","last_unstale":"2024-05-09T08:50:38.597628+0000","last_undegraded":"2024-05-09T08:50:38.597628+0000","last_fullsized":"2024-05-09T08:50:38.597628+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:47:07.257464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497375+0000","last_change":"2024-05-09T08:48:17.595255+0000","last_active":"2024-05-09T08:50:38.497375+0000","last_peered":"2024-05-09T08:50:38.497375+0000","last_clean":"2024-05-09T08:50:38.497375+0000","last_became_active":"2024-05-09T08:48:17.594882+0000","last_became_peered":"2024-05-09T08:48:17.594882+0000","last_unstale":"2024-05-09T08:50:38.497375+0000","last_undegraded":"2024-05-09T08:50:38.497375+0000","last_fullsized":"2024-05-09T08:50:38.497375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:49.845326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664529+0000","last_change":"2024-05-09T08:48:18.378580+0000","last_active":"2024-05-09T08:50:38.664529+0000","last_peered":"2024-05-09T08:50:38.664529+0000","last_clean":"2024-05-09T08:50:38.664529+0000","last_became_active":"2024-05-09T08:48:18.378205+0000","last_became_peered":"2024-05-09T08:48:18.378205+0000","last_unstale":"2024-05-09T08:50:38.664529+0000","last_undegraded":"2024-05-09T08:50:38.664529+0000","last_fullsized":"2024-05-09T08:50:38.664529+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:50.103195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738066+0000","last_change":"2024-05-09T08:48:22.378494+0000","last_active":"2024-05-09T08:50:38.738066+0000","last_peered":"2024-05-09T08:50:38.738066+0000","last_clean":"2024-05-09T08:50:38.738066+0000","last_became_active":"2024-05-09T08:48:22.378076+0000","last_became_peered":"2024-05-09T08:48:22.378076+0000","last_unstale":"2024-05-09T08:50:38.738066+0000","last_undegraded":"2024-05-09T08:50:38.738066+0000","last_fullsized":"2024-05-09T08:50:38.738066+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:40:09.079968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'4","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664581+0000","last_change":"2024-05-09T08:50:19.307437+0000","last_active":"2024-05-09T08:50:38.664581+0000","last_peered":"2024-05-09T08:50:38.664581+0000","last_clean":"2024-05-09T08:50:38.664581+0000","last_became_active":"2024-05-09T08:50:19.293112+0000","last_became_peered":"2024-05-09T08:50:19.293112+0000","last_unstale":"2024-05-09T08:50:38.664581+0000","last_undegraded":"2024-05-09T08:50:38.664581+0000","last_fullsized":"2024-05-09T08:50:38.664581+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:20.776030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,1],"acting":[11,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497703+0000","last_change":"2024-05-09T08:48:18.373569+0000","last_active":"2024-05-09T08:50:38.497703+0000","last_peered":"2024-05-09T08:50:38.497703+0000","last_clean":"2024-05-09T08:50:38.497703+0000","last_became_active":"2024-05-09T08:48:18.373238+0000","last_became_peered":"2024-05-09T08:48:18.373238+0000","last_unstale":"2024-05-09T08:50:38.497703+0000","last_undegraded":"2024-05-09T08:50:38.497703+0000","last_fullsized":"2024-05-09T08:50:38.497703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:17:41.629419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738071+0000","last_change":"2024-05-09T08:48:22.381523+0000","last_active":"2024-05-09T08:50:38.738071+0000","last_peered":"2024-05-09T08:50:38.738071+0000","last_clean":"2024-05-09T08:50:38.738071+0000","last_became_active":"2024-05-09T08:48:22.380626+0000","last_became_peered":"2024-05-09T08:48:22.380626+0000","last_unstale":"2024-05-09T08:50:38.738071+0000","last_undegraded":"2024-05-09T08:50:38.738071+0000","last_fullsized":"2024-05-09T08:50:38.738071+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:47:45.092232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498421+0000","last_change":"2024-05-09T08:48:17.601479+0000","last_active":"2024-05-09T08:50:38.498421+0000","last_peered":"2024-05-09T08:50:38.498421+0000","last_clean":"2024-05-09T08:50:38.498421+0000","last_became_active":"2024-05-09T08:48:17.601140+0000","last_became_peered":"2024-05-09T08:48:17.601140+0000","last_unstale":"2024-05-09T08:50:38.498421+0000","last_undegraded":"2024-05-09T08:50:38.498421+0000","last_fullsized":"2024-05-09T08:50:38.498421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:38:24.011209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740038+0000","last_change":"2024-05-09T08:48:18.376890+0000","last_active":"2024-05-09T08:50:38.740038+0000","last_peered":"2024-05-09T08:50:38.740038+0000","last_clean":"2024-05-09T08:50:38.740038+0000","last_became_active":"2024-05-09T08:48:18.376472+0000","last_became_peered":"2024-05-09T08:48:18.376472+0000","last_unstale":"2024-05-09T08:50:38.740038+0000","last_undegraded":"2024-05-09T08:50:38.740038+0000","last_fullsized":"2024-05-09T08:50:38.740038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:40:16.238539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743128+0000","last_change":"2024-05-09T08:48:22.374743+0000","last_active":"2024-05-09T08:50:38.743128+0000","last_peered":"2024-05-09T08:50:38.743128+0000","last_clean":"2024-05-09T08:50:38.743128+0000","last_became_active":"2024-05-09T08:48:22.374324+0000","last_became_peered":"2024-05-09T08:48:22.374324+0000","last_unstale":"2024-05-09T08:50:38.743128+0000","last_undegraded":"2024-05-09T08:50:38.743128+0000","last_fullsized":"2024-05-09T08:50:38.743128+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:48.073647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"95'6","reported_seq":163,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.933811+0000","last_change":"2024-05-09T08:48:17.604700+0000","last_active":"2024-05-09T08:50:38.933811+0000","last_peered":"2024-05-09T08:50:38.933811+0000","last_clean":"2024-05-09T08:50:38.933811+0000","last_became_active":"2024-05-09T08:48:17.604334+0000","last_became_peered":"2024-05-09T08:48:17.604334+0000","last_unstale":"2024-05-09T08:50:38.933811+0000","last_undegraded":"2024-05-09T08:50:38.933811+0000","last_fullsized":"2024-05-09T08:50:38.933811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:15:37.346232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740574+0000","last_change":"2024-05-09T08:48:18.375769+0000","last_active":"2024-05-09T08:50:38.740574+0000","last_peered":"2024-05-09T08:50:38.740574+0000","last_clean":"2024-05-09T08:50:38.740574+0000","last_became_active":"2024-05-09T08:48:18.375059+0000","last_became_peered":"2024-05-09T08:48:18.375059+0000","last_unstale":"2024-05-09T08:50:38.740574+0000","last_undegraded":"2024-05-09T08:50:38.740574+0000","last_fullsized":"2024-05-09T08:50:38.740574+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:51.001240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740616+0000","last_change":"2024-05-09T08:48:22.385306+0000","last_active":"2024-05-09T08:50:38.740616+0000","last_peered":"2024-05-09T08:50:38.740616+0000","last_clean":"2024-05-09T08:50:38.740616+0000","last_became_active":"2024-05-09T08:48:22.383890+0000","last_became_peered":"2024-05-09T08:48:22.383890+0000","last_unstale":"2024-05-09T08:50:38.740616+0000","last_undegraded":"2024-05-09T08:50:38.740616+0000","last_fullsized":"2024-05-09T08:50:38.740616+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:50:52.369399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'41","reported_seq":83,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.706348+0000","last_change":"2024-05-09T08:48:17.591627+0000","last_active":"2024-05-09T08:50:38.706348+0000","last_peered":"2024-05-09T08:50:38.706348+0000","last_clean":"2024-05-09T08:50:38.706348+0000","last_became_active":"2024-05-09T08:48:17.591250+0000","last_became_peered":"2024-05-09T08:48:17.591250+0000","last_unstale":"2024-05-09T08:50:38.706348+0000","last_undegraded":"2024-05-09T08:50:38.706348+0000","last_fullsized":"2024-05-09T08:50:38.706348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:40:59.748892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740833+0000","last_change":"2024-05-09T08:48:18.372256+0000","last_active":"2024-05-09T08:50:38.740833+0000","last_peered":"2024-05-09T08:50:38.740833+0000","last_clean":"2024-05-09T08:50:38.740833+0000","last_became_active":"2024-05-09T08:48:18.371895+0000","last_became_peered":"2024-05-09T08:48:18.371895+0000","last_unstale":"2024-05-09T08:50:38.740833+0000","last_undegraded":"2024-05-09T08:50:38.740833+0000","last_fullsized":"2024-05-09T08:50:38.740833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:25.067587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605019+0000","last_change":"2024-05-09T08:48:22.373684+0000","last_active":"2024-05-09T08:50:38.605019+0000","last_peered":"2024-05-09T08:50:38.605019+0000","last_clean":"2024-05-09T08:50:38.605019+0000","last_became_active":"2024-05-09T08:48:22.373000+0000","last_became_peered":"2024-05-09T08:48:22.373000+0000","last_unstale":"2024-05-09T08:50:38.605019+0000","last_undegraded":"2024-05-09T08:50:38.605019+0000","last_fullsized":"2024-05-09T08:50:38.605019+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:53:14.416837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739078+0000","last_change":"2024-05-09T08:48:17.596910+0000","last_active":"2024-05-09T08:50:38.739078+0000","last_peered":"2024-05-09T08:50:38.739078+0000","last_clean":"2024-05-09T08:50:38.739078+0000","last_became_active":"2024-05-09T08:48:17.596605+0000","last_became_peered":"2024-05-09T08:48:17.596605+0000","last_unstale":"2024-05-09T08:50:38.739078+0000","last_undegraded":"2024-05-09T08:50:38.739078+0000","last_fullsized":"2024-05-09T08:50:38.739078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:05:25.880082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740734+0000","last_change":"2024-05-09T08:48:18.370354+0000","last_active":"2024-05-09T08:50:38.740734+0000","last_peered":"2024-05-09T08:50:38.740734+0000","last_clean":"2024-05-09T08:50:38.740734+0000","last_became_active":"2024-05-09T08:48:18.370043+0000","last_became_peered":"2024-05-09T08:48:18.370043+0000","last_unstale":"2024-05-09T08:50:38.740734+0000","last_undegraded":"2024-05-09T08:50:38.740734+0000","last_fullsized":"2024-05-09T08:50:38.740734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:21:34.066162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664588+0000","last_change":"2024-05-09T08:48:18.376431+0000","last_active":"2024-05-09T08:50:38.664588+0000","last_peered":"2024-05-09T08:50:38.664588+0000","last_clean":"2024-05-09T08:50:38.664588+0000","last_became_active":"2024-05-09T08:48:18.375598+0000","last_became_peered":"2024-05-09T08:48:18.375598+0000","last_unstale":"2024-05-09T08:50:38.664588+0000","last_undegraded":"2024-05-09T08:50:38.664588+0000","last_fullsized":"2024-05-09T08:50:38.664588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:51:13.481219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738676+0000","last_change":"2024-05-09T08:48:18.371460+0000","last_active":"2024-05-09T08:50:38.738676+0000","last_peered":"2024-05-09T08:50:38.738676+0000","last_clean":"2024-05-09T08:50:38.738676+0000","last_became_active":"2024-05-09T08:48:18.371091+0000","last_became_peered":"2024-05-09T08:48:18.371091+0000","last_unstale":"2024-05-09T08:50:38.738676+0000","last_undegraded":"2024-05-09T08:50:38.738676+0000","last_fullsized":"2024-05-09T08:50:38.738676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:02.829902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738487+0000","last_change":"2024-05-09T08:48:18.369453+0000","last_active":"2024-05-09T08:50:38.738487+0000","last_peered":"2024-05-09T08:50:38.738487+0000","last_clean":"2024-05-09T08:50:38.738487+0000","last_became_active":"2024-05-09T08:48:18.368966+0000","last_became_peered":"2024-05-09T08:48:18.368966+0000","last_unstale":"2024-05-09T08:50:38.738487+0000","last_undegraded":"2024-05-09T08:50:38.738487+0000","last_fullsized":"2024-05-09T08:50:38.738487+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:23:14.359222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740734+0000","last_change":"2024-05-09T08:48:18.378815+0000","last_active":"2024-05-09T08:50:38.740734+0000","last_peered":"2024-05-09T08:50:38.740734+0000","last_clean":"2024-05-09T08:50:38.740734+0000","last_became_active":"2024-05-09T08:48:18.378479+0000","last_became_peered":"2024-05-09T08:48:18.378479+0000","last_unstale":"2024-05-09T08:50:38.740734+0000","last_undegraded":"2024-05-09T08:50:38.740734+0000","last_fullsized":"2024-05-09T08:50:38.740734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:38:50.133239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738648+0000","last_change":"2024-05-09T08:48:18.365696+0000","last_active":"2024-05-09T08:50:38.738648+0000","last_peered":"2024-05-09T08:50:38.738648+0000","last_clean":"2024-05-09T08:50:38.738648+0000","last_became_active":"2024-05-09T08:48:18.365358+0000","last_became_peered":"2024-05-09T08:48:18.365358+0000","last_unstale":"2024-05-09T08:50:38.738648+0000","last_undegraded":"2024-05-09T08:50:38.738648+0000","last_fullsized":"2024-05-09T08:50:38.738648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:04.064150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665012+0000","last_change":"2024-05-09T08:48:18.368911+0000","last_active":"2024-05-09T08:50:38.665012+0000","last_peered":"2024-05-09T08:50:38.665012+0000","last_clean":"2024-05-09T08:50:38.665012+0000","last_became_active":"2024-05-09T08:48:18.368591+0000","last_became_peered":"2024-05-09T08:48:18.368591+0000","last_unstale":"2024-05-09T08:50:38.665012+0000","last_undegraded":"2024-05-09T08:50:38.665012+0000","last_fullsized":"2024-05-09T08:50:38.665012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:34:09.692168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738963+0000","last_change":"2024-05-09T08:48:18.373629+0000","last_active":"2024-05-09T08:50:38.738963+0000","last_peered":"2024-05-09T08:50:38.738963+0000","last_clean":"2024-05-09T08:50:38.738963+0000","last_became_active":"2024-05-09T08:48:18.372992+0000","last_became_peered":"2024-05-09T08:48:18.372992+0000","last_unstale":"2024-05-09T08:50:38.738963+0000","last_undegraded":"2024-05-09T08:50:38.738963+0000","last_fullsized":"2024-05-09T08:50:38.738963+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:42.532250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597288+0000","last_change":"2024-05-09T08:50:19.289044+0000","last_active":"2024-05-09T08:50:38.597288+0000","last_peered":"2024-05-09T08:50:38.597288+0000","last_clean":"2024-05-09T08:50:38.597288+0000","last_became_active":"2024-05-09T08:50:19.288657+0000","last_became_peered":"2024-05-09T08:50:19.288657+0000","last_unstale":"2024-05-09T08:50:38.597288+0000","last_undegraded":"2024-05-09T08:50:38.597288+0000","last_fullsized":"2024-05-09T08:50:38.597288+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:09:36.183086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605085+0000","last_change":"2024-05-09T08:48:18.370460+0000","last_active":"2024-05-09T08:50:38.605085+0000","last_peered":"2024-05-09T08:50:38.605085+0000","last_clean":"2024-05-09T08:50:38.605085+0000","last_became_active":"2024-05-09T08:48:18.370033+0000","last_became_peered":"2024-05-09T08:48:18.370033+0000","last_unstale":"2024-05-09T08:50:38.605085+0000","last_undegraded":"2024-05-09T08:50:38.605085+0000","last_fullsized":"2024-05-09T08:50:38.605085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:21:28.622511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497468+0000","last_change":"2024-05-09T08:48:18.371410+0000","last_active":"2024-05-09T08:50:38.497468+0000","last_peered":"2024-05-09T08:50:38.497468+0000","last_clean":"2024-05-09T08:50:38.497468+0000","last_became_active":"2024-05-09T08:48:18.371067+0000","last_became_peered":"2024-05-09T08:48:18.371067+0000","last_unstale":"2024-05-09T08:50:38.497468+0000","last_undegraded":"2024-05-09T08:50:38.497468+0000","last_fullsized":"2024-05-09T08:50:38.497468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:35:11.445992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743344+0000","last_change":"2024-05-09T08:48:18.382873+0000","last_active":"2024-05-09T08:50:38.743344+0000","last_peered":"2024-05-09T08:50:38.743344+0000","last_clean":"2024-05-09T08:50:38.743344+0000","last_became_active":"2024-05-09T08:48:18.382366+0000","last_became_peered":"2024-05-09T08:48:18.382366+0000","last_unstale":"2024-05-09T08:50:38.743344+0000","last_undegraded":"2024-05-09T08:50:38.743344+0000","last_fullsized":"2024-05-09T08:50:38.743344+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:23.343131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738362+0000","last_change":"2024-05-09T08:48:18.370821+0000","last_active":"2024-05-09T08:50:38.738362+0000","last_peered":"2024-05-09T08:50:38.738362+0000","last_clean":"2024-05-09T08:50:38.738362+0000","last_became_active":"2024-05-09T08:48:18.368208+0000","last_became_peered":"2024-05-09T08:48:18.368208+0000","last_unstale":"2024-05-09T08:50:38.738362+0000","last_undegraded":"2024-05-09T08:50:38.738362+0000","last_fullsized":"2024-05-09T08:50:38.738362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:14.633938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664977+0000","last_change":"2024-05-09T08:48:18.371626+0000","last_active":"2024-05-09T08:50:38.664977+0000","last_peered":"2024-05-09T08:50:38.664977+0000","last_clean":"2024-05-09T08:50:38.664977+0000","last_became_active":"2024-05-09T08:48:18.371081+0000","last_became_peered":"2024-05-09T08:48:18.371081+0000","last_unstale":"2024-05-09T08:50:38.664977+0000","last_undegraded":"2024-05-09T08:50:38.664977+0000","last_fullsized":"2024-05-09T08:50:38.664977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:27.372934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106929,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599687,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1352,"num_read_kb":793,"num_write":374,"num_write_kb":1468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":1799061,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351017,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":534442,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612654,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28216,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716936,"statfs":{"total":95995035648,"available":95966142464,"internally_reserved":0,"allocated":1224704,"data_stored":1323322,"data_compressed":71341,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776179,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":927129,"data_compressed":48688,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037817,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":927507,"data_compressed":48688,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299453,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":1132551,"data_compressed":65185,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462979,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":529808,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1118208,"data_stored":596742,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":531264,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1130496,"data_stored":597734,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1105920,"data_stored":1053941,"data_compressed":54844,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888141,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":534414,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724615,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1150976,"data_stored":1258419,"data_compressed":71341,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135260,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":200163,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:50:43.865 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-05-09T08:50:43.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:50:44.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.003+0000 7fb9c0215640 1 -- 172.21.15.98:0/1967569662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9b81013d0 msgr2=0x7fb9b810d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.003+0000 7fb9c0215640 1 --2- 172.21.15.98:0/1967569662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9b81013d0 0x7fb9b810d5f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009ca0 tx=0x7fb9b002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.003+0000 7fb9c0215640 1 -- 172.21.15.98:0/1967569662 shutdown_connections 2024-05-09T08:50:44.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.003+0000 7fb9c0215640 1 --2- 172.21.15.98:0/1967569662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9b81013d0 0x7fb9b810d5f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.003+0000 7fb9c0215640 1 --2- 172.21.15.98:0/1967569662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8100a10 0x7fb9b8100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.003+0000 7fb9c0215640 1 --2- 172.21.15.98:0/1967569662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9b8106870 0x7fb9b8106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.003+0000 7fb9c0215640 1 -- 172.21.15.98:0/1967569662 >> 172.21.15.98:0/1967569662 conn(0x7fb9b8077710 msgr2=0x7fb9b8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.003+0000 7fb9c0215640 1 -- 172.21.15.98:0/1967569662 shutdown_connections 2024-05-09T08:50:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.003+0000 7fb9c0215640 1 -- 172.21.15.98:0/1967569662 wait complete. 2024-05-09T08:50:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.003+0000 7fb9c0215640 1 Processor -- start 2024-05-09T08:50:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.003+0000 7fb9c0215640 1 -- start start 2024-05-09T08:50:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9c0215640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8100a10 0x7fb9b8198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9c0215640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9b81013d0 0x7fb9b8199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9c0215640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9b8106870 0x7fb9b81a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9c0215640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b8110b60 con 0x7fb9b81013d0 2024-05-09T08:50:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9c0215640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b81109e0 con 0x7fb9b8106870 2024-05-09T08:50:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9c0215640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b8110ce0 con 0x7fb9b8100a10 2024-05-09T08:50:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9bd7b0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9b81013d0 0x7fb9b8199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9bd7b0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9b81013d0 0x7fb9b8199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43232/0 (socket says 172.21.15.98:43232) 2024-05-09T08:50:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9bd7b0640 1 -- 172.21.15.98:0/2378220824 learned_addr learned my addr 172.21.15.98:0/2378220824 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9bdfb1640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8100a10 0x7fb9b8198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9be7b2640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9b8106870 0x7fb9b81a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9bd7b0640 1 -- 172.21.15.98:0/2378220824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8100a10 msgr2=0x7fb9b8198e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9bd7b0640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8100a10 0x7fb9b8198e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9bd7b0640 1 -- 172.21.15.98:0/2378220824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9b8106870 msgr2=0x7fb9b81a3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9bd7b0640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9b8106870 0x7fb9b81a3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9bd7b0640 1 -- 172.21.15.98:0/2378220824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b81a4400 con 0x7fb9b81013d0 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9be7b2640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9b8106870 0x7fb9b81a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9bd7b0640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9b81013d0 0x7fb9b8199380 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800d770 tx=0x7fb9a800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9aeffd640 1 -- 172.21.15.98:0/2378220824 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8017070 con 0x7fb9b81013d0 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9aeffd640 1 -- 172.21.15.98:0/2378220824 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9a8005500 con 0x7fb9b81013d0 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9c0215640 1 -- 172.21.15.98:0/2378220824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9b81a45d0 con 0x7fb9b81013d0 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9aeffd640 1 -- 172.21.15.98:0/2378220824 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8004e00 con 0x7fb9b81013d0 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.004+0000 7fb9c0215640 1 -- 172.21.15.98:0/2378220824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9b81a4aa0 con 0x7fb9b81013d0 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.005+0000 7fb9c0215640 1 -- 172.21.15.98:0/2378220824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9b8068aa0 con 0x7fb9b81013d0 2024-05-09T08:50:44.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.007+0000 7fb9aeffd640 1 -- 172.21.15.98:0/2378220824 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fb9a8004fa0 con 0x7fb9b81013d0 2024-05-09T08:50:44.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.007+0000 7fb9aeffd640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb9a40a2e20 0x7fb9a40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.008+0000 7fb9bdfb1640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb9a40a2e20 0x7fb9a40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.008+0000 7fb9aeffd640 1 -- 172.21.15.98:0/2378220824 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fb9a80cf950 con 0x7fb9b81013d0 2024-05-09T08:50:44.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.008+0000 7fb9bdfb1640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb9a40a2e20 0x7fb9a40a52e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4005d00 tx=0x7fb9b400b410 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:44.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.011+0000 7fb9aeffd640 1 -- 172.21.15.98:0/2378220824 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb9a809bf00 con 0x7fb9b81013d0 2024-05-09T08:50:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.184+0000 7fb9c0215640 1 -- 172.21.15.98:0/2378220824 --> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb9b80ffb50 con 0x7fb9a40a2e20 2024-05-09T08:50:44.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.193+0000 7fb9aeffd640 1 -- 172.21.15.98:0/2378220824 <== mgr.15555 v2:172.21.15.98:6800/4136638623 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339730 (secure 0 0 0) 0x7fb9b80ffb50 con 0x7fb9a40a2e20 2024-05-09T08:50:44.196 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-09T08:50:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.197+0000 7fb9c0215640 1 -- 172.21.15.98:0/2378220824 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb9a40a2e20 msgr2=0x7fb9a40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.197+0000 7fb9c0215640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb9a40a2e20 0x7fb9a40a52e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4005d00 tx=0x7fb9b400b410 comp rx=0 tx=0).stop 2024-05-09T08:50:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.197+0000 7fb9c0215640 1 -- 172.21.15.98:0/2378220824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9b81013d0 msgr2=0x7fb9b8199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.197+0000 7fb9c0215640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9b81013d0 0x7fb9b8199380 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800d770 tx=0x7fb9a800dc30 comp rx=0 tx=0).stop 2024-05-09T08:50:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.197+0000 7fb9c0215640 1 -- 172.21.15.98:0/2378220824 shutdown_connections 2024-05-09T08:50:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.197+0000 7fb9c0215640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb9a40a2e20 0x7fb9a40a52e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.197+0000 7fb9c0215640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9b8106870 0x7fb9b81a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.197+0000 7fb9c0215640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9b81013d0 0x7fb9b8199380 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.197+0000 7fb9c0215640 1 --2- 172.21.15.98:0/2378220824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8100a10 0x7fb9b8198e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.197+0000 7fb9c0215640 1 -- 172.21.15.98:0/2378220824 >> 172.21.15.98:0/2378220824 conn(0x7fb9b8077710 msgr2=0x7fb9b80ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.197+0000 7fb9c0215640 1 -- 172.21.15.98:0/2378220824 shutdown_connections 2024-05-09T08:50:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.197+0000 7fb9c0215640 1 -- 172.21.15.98:0/2378220824 wait complete. 2024-05-09T08:50:44.207 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-05-09T08:50:42.510505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190289,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1488,"num_read_kb":912,"num_write":557,"num_write_kb":4408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1180921,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":942,"ondisk_log_size":942,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353840,"kb_used_data":12968,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124587984,"statfs":{"total":1151940427776,"available":1151578095616,"internally_reserved":0,"allocated":13279232,"data_stored":9947273,"data_compressed":537339,"data_compressed_allocated":724992,"data_compressed_original":4300800,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.987472"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665191+0000","last_change":"2024-05-09T08:48:18.373685+0000","last_active":"2024-05-09T08:50:38.665191+0000","last_peered":"2024-05-09T08:50:38.665191+0000","last_clean":"2024-05-09T08:50:38.665191+0000","last_became_active":"2024-05-09T08:48:18.373305+0000","last_became_peered":"2024-05-09T08:48:18.373305+0000","last_unstale":"2024-05-09T08:50:38.665191+0000","last_undegraded":"2024-05-09T08:50:38.665191+0000","last_fullsized":"2024-05-09T08:50:38.665191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:30:45.261706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740969+0000","last_change":"2024-05-09T08:48:18.373152+0000","last_active":"2024-05-09T08:50:38.740969+0000","last_peered":"2024-05-09T08:50:38.740969+0000","last_clean":"2024-05-09T08:50:38.740969+0000","last_became_active":"2024-05-09T08:48:18.372015+0000","last_became_peered":"2024-05-09T08:48:18.372015+0000","last_unstale":"2024-05-09T08:50:38.740969+0000","last_undegraded":"2024-05-09T08:50:38.740969+0000","last_fullsized":"2024-05-09T08:50:38.740969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:47.512778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740875+0000","last_change":"2024-05-09T08:48:18.370177+0000","last_active":"2024-05-09T08:50:38.740875+0000","last_peered":"2024-05-09T08:50:38.740875+0000","last_clean":"2024-05-09T08:50:38.740875+0000","last_became_active":"2024-05-09T08:48:18.369842+0000","last_became_peered":"2024-05-09T08:48:18.369842+0000","last_unstale":"2024-05-09T08:50:38.740875+0000","last_undegraded":"2024-05-09T08:50:38.740875+0000","last_fullsized":"2024-05-09T08:50:38.740875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:00:24.015500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738316+0000","last_change":"2024-05-09T08:48:18.375364+0000","last_active":"2024-05-09T08:50:38.738316+0000","last_peered":"2024-05-09T08:50:38.738316+0000","last_clean":"2024-05-09T08:50:38.738316+0000","last_became_active":"2024-05-09T08:48:18.375030+0000","last_became_peered":"2024-05-09T08:48:18.375030+0000","last_unstale":"2024-05-09T08:50:38.738316+0000","last_undegraded":"2024-05-09T08:50:38.738316+0000","last_fullsized":"2024-05-09T08:50:38.738316+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:43:57.684123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498390+0000","last_change":"2024-05-09T08:48:18.374548+0000","last_active":"2024-05-09T08:50:38.498390+0000","last_peered":"2024-05-09T08:50:38.498390+0000","last_clean":"2024-05-09T08:50:38.498390+0000","last_became_active":"2024-05-09T08:48:18.374190+0000","last_became_peered":"2024-05-09T08:48:18.374190+0000","last_unstale":"2024-05-09T08:50:38.498390+0000","last_undegraded":"2024-05-09T08:50:38.498390+0000","last_fullsized":"2024-05-09T08:50:38.498390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:43:48.456839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.501744+0000","last_change":"2024-05-09T08:48:18.371234+0000","last_active":"2024-05-09T08:50:38.501744+0000","last_peered":"2024-05-09T08:50:38.501744+0000","last_clean":"2024-05-09T08:50:38.501744+0000","last_became_active":"2024-05-09T08:48:18.370218+0000","last_became_peered":"2024-05-09T08:48:18.370218+0000","last_unstale":"2024-05-09T08:50:38.501744+0000","last_undegraded":"2024-05-09T08:50:38.501744+0000","last_fullsized":"2024-05-09T08:50:38.501744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:02:09.923146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739889+0000","last_change":"2024-05-09T08:49:18.366083+0000","last_active":"2024-05-09T08:50:38.739889+0000","last_peered":"2024-05-09T08:50:38.739889+0000","last_clean":"2024-05-09T08:50:38.739889+0000","last_became_active":"2024-05-09T08:49:18.365755+0000","last_became_peered":"2024-05-09T08:49:18.365755+0000","last_unstale":"2024-05-09T08:50:38.739889+0000","last_undegraded":"2024-05-09T08:50:38.739889+0000","last_fullsized":"2024-05-09T08:50:38.739889+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:40:18.052955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.596412+0000","last_change":"2024-05-09T08:49:18.488640+0000","last_active":"2024-05-09T08:50:38.596412+0000","last_peered":"2024-05-09T08:50:38.596412+0000","last_clean":"2024-05-09T08:50:38.596412+0000","last_became_active":"2024-05-09T08:49:18.488404+0000","last_became_peered":"2024-05-09T08:49:18.488404+0000","last_unstale":"2024-05-09T08:50:38.596412+0000","last_undegraded":"2024-05-09T08:50:38.596412+0000","last_fullsized":"2024-05-09T08:50:38.596412+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:47.317453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743849+0000","last_change":"2024-05-09T08:49:18.368485+0000","last_active":"2024-05-09T08:50:38.743849+0000","last_peered":"2024-05-09T08:50:38.743849+0000","last_clean":"2024-05-09T08:50:38.743849+0000","last_became_active":"2024-05-09T08:49:18.368175+0000","last_became_peered":"2024-05-09T08:49:18.368175+0000","last_unstale":"2024-05-09T08:50:38.743849+0000","last_undegraded":"2024-05-09T08:50:38.743849+0000","last_fullsized":"2024-05-09T08:50:38.743849+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:39:32.399209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.502258+0000","last_change":"2024-05-09T08:48:18.363668+0000","last_active":"2024-05-09T08:50:38.502258+0000","last_peered":"2024-05-09T08:50:38.502258+0000","last_clean":"2024-05-09T08:50:38.502258+0000","last_became_active":"2024-05-09T08:48:18.363248+0000","last_became_peered":"2024-05-09T08:48:18.363248+0000","last_unstale":"2024-05-09T08:50:38.502258+0000","last_undegraded":"2024-05-09T08:50:38.502258+0000","last_fullsized":"2024-05-09T08:50:38.502258+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:19:33.802350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740440+0000","last_change":"2024-05-09T08:48:18.376401+0000","last_active":"2024-05-09T08:50:38.740440+0000","last_peered":"2024-05-09T08:50:38.740440+0000","last_clean":"2024-05-09T08:50:38.740440+0000","last_became_active":"2024-05-09T08:48:18.375520+0000","last_became_peered":"2024-05-09T08:48:18.375520+0000","last_unstale":"2024-05-09T08:50:38.740440+0000","last_undegraded":"2024-05-09T08:50:38.740440+0000","last_fullsized":"2024-05-09T08:50:38.740440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:40:36.145071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738307+0000","last_change":"2024-05-09T08:48:18.372847+0000","last_active":"2024-05-09T08:50:38.738307+0000","last_peered":"2024-05-09T08:50:38.738307+0000","last_clean":"2024-05-09T08:50:38.738307+0000","last_became_active":"2024-05-09T08:48:18.370671+0000","last_became_peered":"2024-05-09T08:48:18.370671+0000","last_unstale":"2024-05-09T08:50:38.738307+0000","last_undegraded":"2024-05-09T08:50:38.738307+0000","last_fullsized":"2024-05-09T08:50:38.738307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:06:22.438515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738426+0000","last_change":"2024-05-09T08:48:18.365810+0000","last_active":"2024-05-09T08:50:38.738426+0000","last_peered":"2024-05-09T08:50:38.738426+0000","last_clean":"2024-05-09T08:50:38.738426+0000","last_became_active":"2024-05-09T08:48:18.365214+0000","last_became_peered":"2024-05-09T08:48:18.365214+0000","last_unstale":"2024-05-09T08:50:38.738426+0000","last_undegraded":"2024-05-09T08:50:38.738426+0000","last_fullsized":"2024-05-09T08:50:38.738426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:00.037153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740431+0000","last_change":"2024-05-09T08:48:18.375279+0000","last_active":"2024-05-09T08:50:38.740431+0000","last_peered":"2024-05-09T08:50:38.740431+0000","last_clean":"2024-05-09T08:50:38.740431+0000","last_became_active":"2024-05-09T08:48:18.374918+0000","last_became_peered":"2024-05-09T08:48:18.374918+0000","last_unstale":"2024-05-09T08:50:38.740431+0000","last_undegraded":"2024-05-09T08:50:38.740431+0000","last_fullsized":"2024-05-09T08:50:38.740431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:23:12.889283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740477+0000","last_change":"2024-05-09T08:49:18.369768+0000","last_active":"2024-05-09T08:50:38.740477+0000","last_peered":"2024-05-09T08:50:38.740477+0000","last_clean":"2024-05-09T08:50:38.740477+0000","last_became_active":"2024-05-09T08:49:18.369520+0000","last_became_peered":"2024-05-09T08:49:18.369520+0000","last_unstale":"2024-05-09T08:50:38.740477+0000","last_undegraded":"2024-05-09T08:50:38.740477+0000","last_fullsized":"2024-05-09T08:50:38.740477+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:44:19.502358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743522+0000","last_change":"2024-05-09T08:48:18.373905+0000","last_active":"2024-05-09T08:50:38.743522+0000","last_peered":"2024-05-09T08:50:38.743522+0000","last_clean":"2024-05-09T08:50:38.743522+0000","last_became_active":"2024-05-09T08:48:18.373566+0000","last_became_peered":"2024-05-09T08:48:18.373566+0000","last_unstale":"2024-05-09T08:50:38.743522+0000","last_undegraded":"2024-05-09T08:50:38.743522+0000","last_fullsized":"2024-05-09T08:50:38.743522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:33.905686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665140+0000","last_change":"2024-05-09T08:48:18.373970+0000","last_active":"2024-05-09T08:50:38.665140+0000","last_peered":"2024-05-09T08:50:38.665140+0000","last_clean":"2024-05-09T08:50:38.665140+0000","last_became_active":"2024-05-09T08:48:18.372696+0000","last_became_peered":"2024-05-09T08:48:18.372696+0000","last_unstale":"2024-05-09T08:50:38.665140+0000","last_undegraded":"2024-05-09T08:50:38.665140+0000","last_fullsized":"2024-05-09T08:50:38.665140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:09:33.751950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597421+0000","last_change":"2024-05-09T08:48:18.376851+0000","last_active":"2024-05-09T08:50:38.597421+0000","last_peered":"2024-05-09T08:50:38.597421+0000","last_clean":"2024-05-09T08:50:38.597421+0000","last_became_active":"2024-05-09T08:48:18.376230+0000","last_became_peered":"2024-05-09T08:48:18.376230+0000","last_unstale":"2024-05-09T08:50:38.597421+0000","last_undegraded":"2024-05-09T08:50:38.597421+0000","last_fullsized":"2024-05-09T08:50:38.597421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:34.959179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738942+0000","last_change":"2024-05-09T08:48:18.370287+0000","last_active":"2024-05-09T08:50:38.738942+0000","last_peered":"2024-05-09T08:50:38.738942+0000","last_clean":"2024-05-09T08:50:38.738942+0000","last_became_active":"2024-05-09T08:48:18.369956+0000","last_became_peered":"2024-05-09T08:48:18.369956+0000","last_unstale":"2024-05-09T08:50:38.738942+0000","last_undegraded":"2024-05-09T08:50:38.738942+0000","last_fullsized":"2024-05-09T08:50:38.738942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:19:34.746216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738556+0000","last_change":"2024-05-09T08:48:22.377973+0000","last_active":"2024-05-09T08:50:38.738556+0000","last_peered":"2024-05-09T08:50:38.738556+0000","last_clean":"2024-05-09T08:50:38.738556+0000","last_became_active":"2024-05-09T08:48:22.377325+0000","last_became_peered":"2024-05-09T08:48:22.377325+0000","last_unstale":"2024-05-09T08:50:38.738556+0000","last_undegraded":"2024-05-09T08:50:38.738556+0000","last_fullsized":"2024-05-09T08:50:38.738556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:32:00.873082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"95'14","reported_seq":169,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.744593+0000","last_change":"2024-05-09T08:48:17.595784+0000","last_active":"2024-05-09T08:50:38.744593+0000","last_peered":"2024-05-09T08:50:38.744593+0000","last_clean":"2024-05-09T08:50:38.744593+0000","last_became_active":"2024-05-09T08:48:17.595395+0000","last_became_peered":"2024-05-09T08:48:17.595395+0000","last_unstale":"2024-05-09T08:50:38.744593+0000","last_undegraded":"2024-05-09T08:50:38.744593+0000","last_fullsized":"2024-05-09T08:50:38.744593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:47.475148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497681+0000","last_change":"2024-05-09T08:48:18.382332+0000","last_active":"2024-05-09T08:50:38.497681+0000","last_peered":"2024-05-09T08:50:38.497681+0000","last_clean":"2024-05-09T08:50:38.497681+0000","last_became_active":"2024-05-09T08:48:18.381536+0000","last_became_peered":"2024-05-09T08:48:18.381536+0000","last_unstale":"2024-05-09T08:50:38.497681+0000","last_undegraded":"2024-05-09T08:50:38.497681+0000","last_fullsized":"2024-05-09T08:50:38.497681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:52:48.327000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737688+0000","last_change":"2024-05-09T08:48:22.379541+0000","last_active":"2024-05-09T08:50:38.737688+0000","last_peered":"2024-05-09T08:50:38.737688+0000","last_clean":"2024-05-09T08:50:38.737688+0000","last_became_active":"2024-05-09T08:48:22.379135+0000","last_became_peered":"2024-05-09T08:48:22.379135+0000","last_unstale":"2024-05-09T08:50:38.737688+0000","last_undegraded":"2024-05-09T08:50:38.737688+0000","last_fullsized":"2024-05-09T08:50:38.737688+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:51.365075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497915+0000","last_change":"2024-05-09T08:48:17.592168+0000","last_active":"2024-05-09T08:50:38.497915+0000","last_peered":"2024-05-09T08:50:38.497915+0000","last_clean":"2024-05-09T08:50:38.497915+0000","last_became_active":"2024-05-09T08:48:17.591822+0000","last_became_peered":"2024-05-09T08:48:17.591822+0000","last_unstale":"2024-05-09T08:50:38.497915+0000","last_undegraded":"2024-05-09T08:50:38.497915+0000","last_fullsized":"2024-05-09T08:50:38.497915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:45:15.162240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.741025+0000","last_change":"2024-05-09T08:48:18.379396+0000","last_active":"2024-05-09T08:50:38.741025+0000","last_peered":"2024-05-09T08:50:38.741025+0000","last_clean":"2024-05-09T08:50:38.741025+0000","last_became_active":"2024-05-09T08:48:18.378770+0000","last_became_peered":"2024-05-09T08:48:18.378770+0000","last_unstale":"2024-05-09T08:50:38.741025+0000","last_undegraded":"2024-05-09T08:50:38.741025+0000","last_fullsized":"2024-05-09T08:50:38.741025+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:22:22.699186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737898+0000","last_change":"2024-05-09T08:48:22.387777+0000","last_active":"2024-05-09T08:50:38.737898+0000","last_peered":"2024-05-09T08:50:38.737898+0000","last_clean":"2024-05-09T08:50:38.737898+0000","last_became_active":"2024-05-09T08:48:22.387300+0000","last_became_peered":"2024-05-09T08:48:22.387300+0000","last_unstale":"2024-05-09T08:50:38.737898+0000","last_undegraded":"2024-05-09T08:50:38.737898+0000","last_fullsized":"2024-05-09T08:50:38.737898+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:08:35.646374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"95'13","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498917+0000","last_change":"2024-05-09T08:48:17.602493+0000","last_active":"2024-05-09T08:50:38.498917+0000","last_peered":"2024-05-09T08:50:38.498917+0000","last_clean":"2024-05-09T08:50:38.498917+0000","last_became_active":"2024-05-09T08:48:17.601625+0000","last_became_peered":"2024-05-09T08:48:17.601625+0000","last_unstale":"2024-05-09T08:50:38.498917+0000","last_undegraded":"2024-05-09T08:50:38.498917+0000","last_fullsized":"2024-05-09T08:50:38.498917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:33.044532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743329+0000","last_change":"2024-05-09T08:48:18.380615+0000","last_active":"2024-05-09T08:50:38.743329+0000","last_peered":"2024-05-09T08:50:38.743329+0000","last_clean":"2024-05-09T08:50:38.743329+0000","last_became_active":"2024-05-09T08:48:18.380196+0000","last_became_peered":"2024-05-09T08:48:18.380196+0000","last_unstale":"2024-05-09T08:50:38.743329+0000","last_undegraded":"2024-05-09T08:50:38.743329+0000","last_fullsized":"2024-05-09T08:50:38.743329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:27.299263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737797+0000","last_change":"2024-05-09T08:48:22.381954+0000","last_active":"2024-05-09T08:50:38.737797+0000","last_peered":"2024-05-09T08:50:38.737797+0000","last_clean":"2024-05-09T08:50:38.737797+0000","last_became_active":"2024-05-09T08:48:22.381527+0000","last_became_peered":"2024-05-09T08:48:22.381527+0000","last_unstale":"2024-05-09T08:50:38.737797+0000","last_undegraded":"2024-05-09T08:50:38.737797+0000","last_fullsized":"2024-05-09T08:50:38.737797+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:35.170753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498547+0000","last_change":"2024-05-09T08:48:17.592795+0000","last_active":"2024-05-09T08:50:38.498547+0000","last_peered":"2024-05-09T08:50:38.498547+0000","last_clean":"2024-05-09T08:50:38.498547+0000","last_became_active":"2024-05-09T08:48:17.592240+0000","last_became_peered":"2024-05-09T08:48:17.592240+0000","last_unstale":"2024-05-09T08:50:38.498547+0000","last_undegraded":"2024-05-09T08:50:38.498547+0000","last_fullsized":"2024-05-09T08:50:38.498547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:27:30.056805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605470+0000","last_change":"2024-05-09T08:50:19.350043+0000","last_active":"2024-05-09T08:50:38.605470+0000","last_peered":"2024-05-09T08:50:38.605470+0000","last_clean":"2024-05-09T08:50:38.605470+0000","last_became_active":"2024-05-09T08:50:19.349640+0000","last_became_peered":"2024-05-09T08:50:19.349640+0000","last_unstale":"2024-05-09T08:50:38.605470+0000","last_undegraded":"2024-05-09T08:50:38.605470+0000","last_fullsized":"2024-05-09T08:50:38.605470+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:00:47.277071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738696+0000","last_change":"2024-05-09T08:48:22.374267+0000","last_active":"2024-05-09T08:50:38.738696+0000","last_peered":"2024-05-09T08:50:38.738696+0000","last_clean":"2024-05-09T08:50:38.738696+0000","last_became_active":"2024-05-09T08:48:22.373824+0000","last_became_peered":"2024-05-09T08:48:22.373824+0000","last_unstale":"2024-05-09T08:50:38.738696+0000","last_undegraded":"2024-05-09T08:50:38.738696+0000","last_fullsized":"2024-05-09T08:50:38.738696+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:27.407543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":168,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605692+0000","last_change":"2024-05-09T08:48:17.596851+0000","last_active":"2024-05-09T08:50:38.605692+0000","last_peered":"2024-05-09T08:50:38.605692+0000","last_clean":"2024-05-09T08:50:38.605692+0000","last_became_active":"2024-05-09T08:48:17.596532+0000","last_became_peered":"2024-05-09T08:48:17.596532+0000","last_unstale":"2024-05-09T08:50:38.605692+0000","last_undegraded":"2024-05-09T08:50:38.605692+0000","last_fullsized":"2024-05-09T08:50:38.605692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:58:51.923188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498499+0000","last_change":"2024-05-09T08:48:18.372214+0000","last_active":"2024-05-09T08:50:38.498499+0000","last_peered":"2024-05-09T08:50:38.498499+0000","last_clean":"2024-05-09T08:50:38.498499+0000","last_became_active":"2024-05-09T08:48:18.371850+0000","last_became_peered":"2024-05-09T08:48:18.371850+0000","last_unstale":"2024-05-09T08:50:38.498499+0000","last_undegraded":"2024-05-09T08:50:38.498499+0000","last_fullsized":"2024-05-09T08:50:38.498499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:56.971735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738968+0000","last_change":"2024-05-09T08:49:18.371344+0000","last_active":"2024-05-09T08:50:38.738968+0000","last_peered":"2024-05-09T08:50:38.738968+0000","last_clean":"2024-05-09T08:50:38.738968+0000","last_became_active":"2024-05-09T08:49:18.370911+0000","last_became_peered":"2024-05-09T08:49:18.370911+0000","last_unstale":"2024-05-09T08:50:38.738968+0000","last_undegraded":"2024-05-09T08:50:38.738968+0000","last_fullsized":"2024-05-09T08:50:38.738968+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:53:49.122965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,8],"acting":[2,6,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.501701+0000","last_change":"2024-05-09T08:48:17.601726+0000","last_active":"2024-05-09T08:50:38.501701+0000","last_peered":"2024-05-09T08:50:38.501701+0000","last_clean":"2024-05-09T08:50:38.501701+0000","last_became_active":"2024-05-09T08:48:17.601372+0000","last_became_peered":"2024-05-09T08:48:17.601372+0000","last_unstale":"2024-05-09T08:50:38.501701+0000","last_undegraded":"2024-05-09T08:50:38.501701+0000","last_fullsized":"2024-05-09T08:50:38.501701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:27:56.683821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605526+0000","last_change":"2024-05-09T08:49:18.694713+0000","last_active":"2024-05-09T08:50:38.605526+0000","last_peered":"2024-05-09T08:50:38.605526+0000","last_clean":"2024-05-09T08:50:38.605526+0000","last_became_active":"2024-05-09T08:49:18.694246+0000","last_became_peered":"2024-05-09T08:49:18.694246+0000","last_unstale":"2024-05-09T08:50:38.605526+0000","last_undegraded":"2024-05-09T08:50:38.605526+0000","last_fullsized":"2024-05-09T08:50:38.605526+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:05:30.818965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665087+0000","last_change":"2024-05-09T08:48:22.390990+0000","last_active":"2024-05-09T08:50:38.665087+0000","last_peered":"2024-05-09T08:50:38.665087+0000","last_clean":"2024-05-09T08:50:38.665087+0000","last_became_active":"2024-05-09T08:48:22.390601+0000","last_became_peered":"2024-05-09T08:48:22.390601+0000","last_unstale":"2024-05-09T08:50:38.665087+0000","last_undegraded":"2024-05-09T08:50:38.665087+0000","last_fullsized":"2024-05-09T08:50:38.665087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:47:24.644408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738010+0000","last_change":"2024-05-09T08:48:17.598008+0000","last_active":"2024-05-09T08:50:38.738010+0000","last_peered":"2024-05-09T08:50:38.738010+0000","last_clean":"2024-05-09T08:50:38.738010+0000","last_became_active":"2024-05-09T08:48:17.597514+0000","last_became_peered":"2024-05-09T08:48:17.597514+0000","last_unstale":"2024-05-09T08:50:38.738010+0000","last_undegraded":"2024-05-09T08:50:38.738010+0000","last_fullsized":"2024-05-09T08:50:38.738010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:18:32.788536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497274+0000","last_change":"2024-05-09T08:48:18.378061+0000","last_active":"2024-05-09T08:50:38.497274+0000","last_peered":"2024-05-09T08:50:38.497274+0000","last_clean":"2024-05-09T08:50:38.497274+0000","last_became_active":"2024-05-09T08:48:18.377684+0000","last_became_peered":"2024-05-09T08:48:18.377684+0000","last_unstale":"2024-05-09T08:50:38.497274+0000","last_undegraded":"2024-05-09T08:50:38.497274+0000","last_fullsized":"2024-05-09T08:50:38.497274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:13:08.338307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740944+0000","last_change":"2024-05-09T08:48:22.385779+0000","last_active":"2024-05-09T08:50:38.740944+0000","last_peered":"2024-05-09T08:50:38.740944+0000","last_clean":"2024-05-09T08:50:38.740944+0000","last_became_active":"2024-05-09T08:48:22.385021+0000","last_became_peered":"2024-05-09T08:48:22.385021+0000","last_unstale":"2024-05-09T08:50:38.740944+0000","last_undegraded":"2024-05-09T08:50:38.740944+0000","last_fullsized":"2024-05-09T08:50:38.740944+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:51.350754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605038+0000","last_change":"2024-05-09T08:48:17.595750+0000","last_active":"2024-05-09T08:50:38.605038+0000","last_peered":"2024-05-09T08:50:38.605038+0000","last_clean":"2024-05-09T08:50:38.605038+0000","last_became_active":"2024-05-09T08:48:17.595394+0000","last_became_peered":"2024-05-09T08:48:17.595394+0000","last_unstale":"2024-05-09T08:50:38.605038+0000","last_undegraded":"2024-05-09T08:50:38.605038+0000","last_fullsized":"2024-05-09T08:50:38.605038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:39:21.975059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.596769+0000","last_change":"2024-05-09T08:48:18.370750+0000","last_active":"2024-05-09T08:50:38.596769+0000","last_peered":"2024-05-09T08:50:38.596769+0000","last_clean":"2024-05-09T08:50:38.596769+0000","last_became_active":"2024-05-09T08:48:18.370383+0000","last_became_peered":"2024-05-09T08:48:18.370383+0000","last_unstale":"2024-05-09T08:50:38.596769+0000","last_undegraded":"2024-05-09T08:50:38.596769+0000","last_fullsized":"2024-05-09T08:50:38.596769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:36.049350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497318+0000","last_change":"2024-05-09T08:48:22.377157+0000","last_active":"2024-05-09T08:50:38.497318+0000","last_peered":"2024-05-09T08:50:38.497318+0000","last_clean":"2024-05-09T08:50:38.497318+0000","last_became_active":"2024-05-09T08:48:22.376730+0000","last_became_peered":"2024-05-09T08:48:22.376730+0000","last_unstale":"2024-05-09T08:50:38.497318+0000","last_undegraded":"2024-05-09T08:50:38.497318+0000","last_fullsized":"2024-05-09T08:50:38.497318+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:41:52.080730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"95'7","reported_seq":282,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.894752+0000","last_change":"2024-05-09T08:48:17.598528+0000","last_active":"2024-05-09T08:50:38.894752+0000","last_peered":"2024-05-09T08:50:38.894752+0000","last_clean":"2024-05-09T08:50:38.894752+0000","last_became_active":"2024-05-09T08:48:17.598072+0000","last_became_peered":"2024-05-09T08:48:17.598072+0000","last_unstale":"2024-05-09T08:50:38.894752+0000","last_undegraded":"2024-05-09T08:50:38.894752+0000","last_fullsized":"2024-05-09T08:50:38.894752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:35.697154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743731+0000","last_change":"2024-05-09T08:48:18.380919+0000","last_active":"2024-05-09T08:50:38.743731+0000","last_peered":"2024-05-09T08:50:38.743731+0000","last_clean":"2024-05-09T08:50:38.743731+0000","last_became_active":"2024-05-09T08:48:18.380485+0000","last_became_peered":"2024-05-09T08:48:18.380485+0000","last_unstale":"2024-05-09T08:50:38.743731+0000","last_undegraded":"2024-05-09T08:50:38.743731+0000","last_fullsized":"2024-05-09T08:50:38.743731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:45.079173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.596707+0000","last_change":"2024-05-09T08:48:22.389667+0000","last_active":"2024-05-09T08:50:38.596707+0000","last_peered":"2024-05-09T08:50:38.596707+0000","last_clean":"2024-05-09T08:50:38.596707+0000","last_became_active":"2024-05-09T08:48:22.389268+0000","last_became_peered":"2024-05-09T08:48:22.389268+0000","last_unstale":"2024-05-09T08:50:38.596707+0000","last_undegraded":"2024-05-09T08:50:38.596707+0000","last_fullsized":"2024-05-09T08:50:38.596707+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:33:14.861855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740395+0000","last_change":"2024-05-09T08:48:17.596028+0000","last_active":"2024-05-09T08:50:38.740395+0000","last_peered":"2024-05-09T08:50:38.740395+0000","last_clean":"2024-05-09T08:50:38.740395+0000","last_became_active":"2024-05-09T08:48:17.595053+0000","last_became_peered":"2024-05-09T08:48:17.595053+0000","last_unstale":"2024-05-09T08:50:38.740395+0000","last_undegraded":"2024-05-09T08:50:38.740395+0000","last_fullsized":"2024-05-09T08:50:38.740395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:48:31.323595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740976+0000","last_change":"2024-05-09T08:48:18.373472+0000","last_active":"2024-05-09T08:50:38.740976+0000","last_peered":"2024-05-09T08:50:38.740976+0000","last_clean":"2024-05-09T08:50:38.740976+0000","last_became_active":"2024-05-09T08:48:18.373178+0000","last_became_peered":"2024-05-09T08:48:18.373178+0000","last_unstale":"2024-05-09T08:50:38.740976+0000","last_undegraded":"2024-05-09T08:50:38.740976+0000","last_fullsized":"2024-05-09T08:50:38.740976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:18.048722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738133+0000","last_change":"2024-05-09T08:48:22.386551+0000","last_active":"2024-05-09T08:50:38.738133+0000","last_peered":"2024-05-09T08:50:38.738133+0000","last_clean":"2024-05-09T08:50:38.738133+0000","last_became_active":"2024-05-09T08:48:22.386018+0000","last_became_peered":"2024-05-09T08:48:22.386018+0000","last_unstale":"2024-05-09T08:50:38.738133+0000","last_undegraded":"2024-05-09T08:50:38.738133+0000","last_fullsized":"2024-05-09T08:50:38.738133+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:59.812030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605522+0000","last_change":"2024-05-09T08:48:17.601207+0000","last_active":"2024-05-09T08:50:38.605522+0000","last_peered":"2024-05-09T08:50:38.605522+0000","last_clean":"2024-05-09T08:50:38.605522+0000","last_became_active":"2024-05-09T08:48:17.600878+0000","last_became_peered":"2024-05-09T08:48:17.600878+0000","last_unstale":"2024-05-09T08:50:38.605522+0000","last_undegraded":"2024-05-09T08:50:38.605522+0000","last_fullsized":"2024-05-09T08:50:38.605522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:30.040527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743577+0000","last_change":"2024-05-09T08:48:18.381577+0000","last_active":"2024-05-09T08:50:38.743577+0000","last_peered":"2024-05-09T08:50:38.743577+0000","last_clean":"2024-05-09T08:50:38.743577+0000","last_became_active":"2024-05-09T08:48:18.381223+0000","last_became_peered":"2024-05-09T08:48:18.381223+0000","last_unstale":"2024-05-09T08:50:38.743577+0000","last_undegraded":"2024-05-09T08:50:38.743577+0000","last_fullsized":"2024-05-09T08:50:38.743577+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:57.038392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.596698+0000","last_change":"2024-05-09T08:48:22.385885+0000","last_active":"2024-05-09T08:50:38.596698+0000","last_peered":"2024-05-09T08:50:38.596698+0000","last_clean":"2024-05-09T08:50:38.596698+0000","last_became_active":"2024-05-09T08:48:22.385403+0000","last_became_peered":"2024-05-09T08:48:22.385403+0000","last_unstale":"2024-05-09T08:50:38.596698+0000","last_undegraded":"2024-05-09T08:50:38.596698+0000","last_fullsized":"2024-05-09T08:50:38.596698+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:53:20.733523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'39","reported_seq":327,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.895223+0000","last_change":"2024-05-09T08:48:17.597848+0000","last_active":"2024-05-09T08:50:38.895223+0000","last_peered":"2024-05-09T08:50:38.895223+0000","last_clean":"2024-05-09T08:50:38.895223+0000","last_became_active":"2024-05-09T08:48:17.596989+0000","last_became_peered":"2024-05-09T08:48:17.596989+0000","last_unstale":"2024-05-09T08:50:38.895223+0000","last_undegraded":"2024-05-09T08:50:38.895223+0000","last_fullsized":"2024-05-09T08:50:38.895223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:07.256301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65904,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":254,"num_read_kb":192,"num_write":41,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498676+0000","last_change":"2024-05-09T08:48:18.365308+0000","last_active":"2024-05-09T08:50:38.498676+0000","last_peered":"2024-05-09T08:50:38.498676+0000","last_clean":"2024-05-09T08:50:38.498676+0000","last_became_active":"2024-05-09T08:48:18.364845+0000","last_became_peered":"2024-05-09T08:48:18.364845+0000","last_unstale":"2024-05-09T08:50:38.498676+0000","last_undegraded":"2024-05-09T08:50:38.498676+0000","last_fullsized":"2024-05-09T08:50:38.498676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:07.277730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605765+0000","last_change":"2024-05-09T08:48:22.377333+0000","last_active":"2024-05-09T08:50:38.605765+0000","last_peered":"2024-05-09T08:50:38.605765+0000","last_clean":"2024-05-09T08:50:38.605765+0000","last_became_active":"2024-05-09T08:48:22.376494+0000","last_became_peered":"2024-05-09T08:48:22.376494+0000","last_unstale":"2024-05-09T08:50:38.605765+0000","last_undegraded":"2024-05-09T08:50:38.605765+0000","last_fullsized":"2024-05-09T08:50:38.605765+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:49:17.693977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664772+0000","last_change":"2024-05-09T08:50:19.301723+0000","last_active":"2024-05-09T08:50:38.664772+0000","last_peered":"2024-05-09T08:50:38.664772+0000","last_clean":"2024-05-09T08:50:38.664772+0000","last_became_active":"2024-05-09T08:50:19.291600+0000","last_became_peered":"2024-05-09T08:50:19.291600+0000","last_unstale":"2024-05-09T08:50:38.664772+0000","last_undegraded":"2024-05-09T08:50:38.664772+0000","last_fullsized":"2024-05-09T08:50:38.664772+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:29.105282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738296+0000","last_change":"2024-05-09T08:48:18.373652+0000","last_active":"2024-05-09T08:50:38.738296+0000","last_peered":"2024-05-09T08:50:38.738296+0000","last_clean":"2024-05-09T08:50:38.738296+0000","last_became_active":"2024-05-09T08:48:18.373345+0000","last_became_peered":"2024-05-09T08:48:18.373345+0000","last_unstale":"2024-05-09T08:50:38.738296+0000","last_undegraded":"2024-05-09T08:50:38.738296+0000","last_fullsized":"2024-05-09T08:50:38.738296+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:14:34.843305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664855+0000","last_change":"2024-05-09T08:49:18.369432+0000","last_active":"2024-05-09T08:50:38.664855+0000","last_peered":"2024-05-09T08:50:38.664855+0000","last_clean":"2024-05-09T08:50:38.664855+0000","last_became_active":"2024-05-09T08:49:18.368978+0000","last_became_peered":"2024-05-09T08:49:18.368978+0000","last_unstale":"2024-05-09T08:50:38.664855+0000","last_undegraded":"2024-05-09T08:50:38.664855+0000","last_fullsized":"2024-05-09T08:50:38.664855+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:20.969162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605525+0000","last_change":"2024-05-09T08:48:17.592658+0000","last_active":"2024-05-09T08:50:38.605525+0000","last_peered":"2024-05-09T08:50:38.605525+0000","last_clean":"2024-05-09T08:50:38.605525+0000","last_became_active":"2024-05-09T08:48:17.592275+0000","last_became_peered":"2024-05-09T08:48:17.592275+0000","last_unstale":"2024-05-09T08:50:38.605525+0000","last_undegraded":"2024-05-09T08:50:38.605525+0000","last_fullsized":"2024-05-09T08:50:38.605525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:26:56.742079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740268+0000","last_change":"2024-05-09T08:49:18.364399+0000","last_active":"2024-05-09T08:50:38.740268+0000","last_peered":"2024-05-09T08:50:38.740268+0000","last_clean":"2024-05-09T08:50:38.740268+0000","last_became_active":"2024-05-09T08:49:18.364166+0000","last_became_peered":"2024-05-09T08:49:18.364166+0000","last_unstale":"2024-05-09T08:50:38.740268+0000","last_undegraded":"2024-05-09T08:50:38.740268+0000","last_fullsized":"2024-05-09T08:50:38.740268+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:49:36.480337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597700+0000","last_change":"2024-05-09T08:48:22.389568+0000","last_active":"2024-05-09T08:50:38.597700+0000","last_peered":"2024-05-09T08:50:38.597700+0000","last_clean":"2024-05-09T08:50:38.597700+0000","last_became_active":"2024-05-09T08:48:22.387192+0000","last_became_peered":"2024-05-09T08:48:22.387192+0000","last_unstale":"2024-05-09T08:50:38.597700+0000","last_undegraded":"2024-05-09T08:50:38.597700+0000","last_fullsized":"2024-05-09T08:50:38.597700+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:56.794208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"95'8","reported_seq":169,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.895068+0000","last_change":"2024-05-09T08:48:17.598760+0000","last_active":"2024-05-09T08:50:38.895068+0000","last_peered":"2024-05-09T08:50:38.895068+0000","last_clean":"2024-05-09T08:50:38.895068+0000","last_became_active":"2024-05-09T08:48:17.598192+0000","last_became_peered":"2024-05-09T08:48:17.598192+0000","last_unstale":"2024-05-09T08:50:38.895068+0000","last_undegraded":"2024-05-09T08:50:38.895068+0000","last_fullsized":"2024-05-09T08:50:38.895068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:29.007644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":128,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739920+0000","last_change":"2024-05-09T08:48:18.364570+0000","last_active":"2024-05-09T08:50:38.739920+0000","last_peered":"2024-05-09T08:50:38.739920+0000","last_clean":"2024-05-09T08:50:38.739920+0000","last_became_active":"2024-05-09T08:48:18.364191+0000","last_became_peered":"2024-05-09T08:48:18.364191+0000","last_unstale":"2024-05-09T08:50:38.739920+0000","last_undegraded":"2024-05-09T08:50:38.739920+0000","last_fullsized":"2024-05-09T08:50:38.739920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:49.267954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737704+0000","last_change":"2024-05-09T08:49:18.694742+0000","last_active":"2024-05-09T08:50:38.737704+0000","last_peered":"2024-05-09T08:50:38.737704+0000","last_clean":"2024-05-09T08:50:38.737704+0000","last_became_active":"2024-05-09T08:49:18.694241+0000","last_became_peered":"2024-05-09T08:49:18.694241+0000","last_unstale":"2024-05-09T08:50:38.737704+0000","last_undegraded":"2024-05-09T08:50:38.737704+0000","last_fullsized":"2024-05-09T08:50:38.737704+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:20.289651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498205+0000","last_change":"2024-05-09T08:48:17.599762+0000","last_active":"2024-05-09T08:50:38.498205+0000","last_peered":"2024-05-09T08:50:38.498205+0000","last_clean":"2024-05-09T08:50:38.498205+0000","last_became_active":"2024-05-09T08:48:17.599355+0000","last_became_peered":"2024-05-09T08:48:17.599355+0000","last_unstale":"2024-05-09T08:50:38.498205+0000","last_undegraded":"2024-05-09T08:50:38.498205+0000","last_fullsized":"2024-05-09T08:50:38.498205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:28:36.836403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605208+0000","last_change":"2024-05-09T08:48:18.372810+0000","last_active":"2024-05-09T08:50:38.605208+0000","last_peered":"2024-05-09T08:50:38.605208+0000","last_clean":"2024-05-09T08:50:38.605208+0000","last_became_active":"2024-05-09T08:48:18.372437+0000","last_became_peered":"2024-05-09T08:48:18.372437+0000","last_unstale":"2024-05-09T08:50:38.605208+0000","last_undegraded":"2024-05-09T08:50:38.605208+0000","last_fullsized":"2024-05-09T08:50:38.605208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:03:36.879719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605358+0000","last_change":"2024-05-09T08:48:22.379370+0000","last_active":"2024-05-09T08:50:38.605358+0000","last_peered":"2024-05-09T08:50:38.605358+0000","last_clean":"2024-05-09T08:50:38.605358+0000","last_became_active":"2024-05-09T08:48:22.378994+0000","last_became_peered":"2024-05-09T08:48:22.378994+0000","last_unstale":"2024-05-09T08:50:38.605358+0000","last_undegraded":"2024-05-09T08:50:38.605358+0000","last_fullsized":"2024-05-09T08:50:38.605358+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:08.005647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498896+0000","last_change":"2024-05-09T08:48:17.594236+0000","last_active":"2024-05-09T08:50:38.498896+0000","last_peered":"2024-05-09T08:50:38.498896+0000","last_clean":"2024-05-09T08:50:38.498896+0000","last_became_active":"2024-05-09T08:48:17.593418+0000","last_became_peered":"2024-05-09T08:48:17.593418+0000","last_unstale":"2024-05-09T08:50:38.498896+0000","last_undegraded":"2024-05-09T08:50:38.498896+0000","last_fullsized":"2024-05-09T08:50:38.498896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:23.930421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665072+0000","last_change":"2024-05-09T08:48:18.366767+0000","last_active":"2024-05-09T08:50:38.665072+0000","last_peered":"2024-05-09T08:50:38.665072+0000","last_clean":"2024-05-09T08:50:38.665072+0000","last_became_active":"2024-05-09T08:48:18.366435+0000","last_became_peered":"2024-05-09T08:48:18.366435+0000","last_unstale":"2024-05-09T08:50:38.665072+0000","last_undegraded":"2024-05-09T08:50:38.665072+0000","last_fullsized":"2024-05-09T08:50:38.665072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:45.111510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498398+0000","last_change":"2024-05-09T08:48:22.378642+0000","last_active":"2024-05-09T08:50:38.498398+0000","last_peered":"2024-05-09T08:50:38.498398+0000","last_clean":"2024-05-09T08:50:38.498398+0000","last_became_active":"2024-05-09T08:48:22.378161+0000","last_became_peered":"2024-05-09T08:48:22.378161+0000","last_unstale":"2024-05-09T08:50:38.498398+0000","last_undegraded":"2024-05-09T08:50:38.498398+0000","last_fullsized":"2024-05-09T08:50:38.498398+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:53.149728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597189+0000","last_change":"2024-05-09T08:48:17.599499+0000","last_active":"2024-05-09T08:50:38.597189+0000","last_peered":"2024-05-09T08:50:38.597189+0000","last_clean":"2024-05-09T08:50:38.597189+0000","last_became_active":"2024-05-09T08:48:17.599145+0000","last_became_peered":"2024-05-09T08:48:17.599145+0000","last_unstale":"2024-05-09T08:50:38.597189+0000","last_undegraded":"2024-05-09T08:50:38.597189+0000","last_fullsized":"2024-05-09T08:50:38.597189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:31:12.502004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.502012+0000","last_change":"2024-05-09T08:48:18.372355+0000","last_active":"2024-05-09T08:50:38.502012+0000","last_peered":"2024-05-09T08:50:38.502012+0000","last_clean":"2024-05-09T08:50:38.502012+0000","last_became_active":"2024-05-09T08:48:18.367743+0000","last_became_peered":"2024-05-09T08:48:18.367743+0000","last_unstale":"2024-05-09T08:50:38.502012+0000","last_undegraded":"2024-05-09T08:50:38.502012+0000","last_fullsized":"2024-05-09T08:50:38.502012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:31:12.174350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738469+0000","last_change":"2024-05-09T08:48:22.384064+0000","last_active":"2024-05-09T08:50:38.738469+0000","last_peered":"2024-05-09T08:50:38.738469+0000","last_clean":"2024-05-09T08:50:38.738469+0000","last_became_active":"2024-05-09T08:48:22.383670+0000","last_became_peered":"2024-05-09T08:48:22.383670+0000","last_unstale":"2024-05-09T08:50:38.738469+0000","last_undegraded":"2024-05-09T08:50:38.738469+0000","last_fullsized":"2024-05-09T08:50:38.738469+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:51:35.000827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"95'6","reported_seq":162,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.922953+0000","last_change":"2024-05-09T08:48:17.601596+0000","last_active":"2024-05-09T08:50:38.922953+0000","last_peered":"2024-05-09T08:50:38.922953+0000","last_clean":"2024-05-09T08:50:38.922953+0000","last_became_active":"2024-05-09T08:48:17.601250+0000","last_became_peered":"2024-05-09T08:48:17.601250+0000","last_unstale":"2024-05-09T08:50:38.922953+0000","last_undegraded":"2024-05-09T08:50:38.922953+0000","last_fullsized":"2024-05-09T08:50:38.922953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:48.237336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.501885+0000","last_change":"2024-05-09T08:48:18.368751+0000","last_active":"2024-05-09T08:50:38.501885+0000","last_peered":"2024-05-09T08:50:38.501885+0000","last_clean":"2024-05-09T08:50:38.501885+0000","last_became_active":"2024-05-09T08:48:18.367463+0000","last_became_peered":"2024-05-09T08:48:18.367463+0000","last_unstale":"2024-05-09T08:50:38.501885+0000","last_undegraded":"2024-05-09T08:50:38.501885+0000","last_fullsized":"2024-05-09T08:50:38.501885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:19:17.696602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738886+0000","last_change":"2024-05-09T08:48:22.385025+0000","last_active":"2024-05-09T08:50:38.738886+0000","last_peered":"2024-05-09T08:50:38.738886+0000","last_clean":"2024-05-09T08:50:38.738886+0000","last_became_active":"2024-05-09T08:48:22.384633+0000","last_became_peered":"2024-05-09T08:48:22.384633+0000","last_unstale":"2024-05-09T08:50:38.738886+0000","last_undegraded":"2024-05-09T08:50:38.738886+0000","last_fullsized":"2024-05-09T08:50:38.738886+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:17.804075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605162+0000","last_change":"2024-05-09T08:48:18.376741+0000","last_active":"2024-05-09T08:50:38.605162+0000","last_peered":"2024-05-09T08:50:38.605162+0000","last_clean":"2024-05-09T08:50:38.605162+0000","last_became_active":"2024-05-09T08:48:18.376376+0000","last_became_peered":"2024-05-09T08:48:18.376376+0000","last_unstale":"2024-05-09T08:50:38.605162+0000","last_undegraded":"2024-05-09T08:50:38.605162+0000","last_fullsized":"2024-05-09T08:50:38.605162+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:58:31.717555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"95'7","reported_seq":164,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740200+0000","last_change":"2024-05-09T08:48:17.598316+0000","last_active":"2024-05-09T08:50:38.740200+0000","last_peered":"2024-05-09T08:50:38.740200+0000","last_clean":"2024-05-09T08:50:38.740200+0000","last_became_active":"2024-05-09T08:48:17.598022+0000","last_became_peered":"2024-05-09T08:48:17.598022+0000","last_unstale":"2024-05-09T08:50:38.740200+0000","last_undegraded":"2024-05-09T08:50:38.740200+0000","last_fullsized":"2024-05-09T08:50:38.740200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:04:10.454894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740178+0000","last_change":"2024-05-09T08:49:18.492981+0000","last_active":"2024-05-09T08:50:38.740178+0000","last_peered":"2024-05-09T08:50:38.740178+0000","last_clean":"2024-05-09T08:50:38.740178+0000","last_became_active":"2024-05-09T08:49:18.489169+0000","last_became_peered":"2024-05-09T08:49:18.489169+0000","last_unstale":"2024-05-09T08:50:38.740178+0000","last_undegraded":"2024-05-09T08:50:38.740178+0000","last_fullsized":"2024-05-09T08:50:38.740178+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:02:53.955587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497378+0000","last_change":"2024-05-09T08:48:18.379203+0000","last_active":"2024-05-09T08:50:38.497378+0000","last_peered":"2024-05-09T08:50:38.497378+0000","last_clean":"2024-05-09T08:50:38.497378+0000","last_became_active":"2024-05-09T08:48:18.378885+0000","last_became_peered":"2024-05-09T08:48:18.378885+0000","last_unstale":"2024-05-09T08:50:38.497378+0000","last_undegraded":"2024-05-09T08:50:38.497378+0000","last_fullsized":"2024-05-09T08:50:38.497378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:40.588263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737925+0000","last_change":"2024-05-09T08:48:17.593520+0000","last_active":"2024-05-09T08:50:38.737925+0000","last_peered":"2024-05-09T08:50:38.737925+0000","last_clean":"2024-05-09T08:50:38.737925+0000","last_became_active":"2024-05-09T08:48:17.593127+0000","last_became_peered":"2024-05-09T08:48:17.593127+0000","last_unstale":"2024-05-09T08:50:38.737925+0000","last_undegraded":"2024-05-09T08:50:38.737925+0000","last_fullsized":"2024-05-09T08:50:38.737925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:34:29.567166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":367,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739767+0000","last_change":"2024-05-09T08:47:05.718649+0000","last_active":"2024-05-09T08:50:38.739767+0000","last_peered":"2024-05-09T08:50:38.739767+0000","last_clean":"2024-05-09T08:50:38.739767+0000","last_became_active":"2024-05-09T08:47:05.718088+0000","last_became_peered":"2024-05-09T08:47:05.718088+0000","last_unstale":"2024-05-09T08:50:38.739767+0000","last_undegraded":"2024-05-09T08:50:38.739767+0000","last_fullsized":"2024-05-09T08:50:38.739767+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:43:39.457880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:43:39.457880+0000","last_clean_scrub_stamp":"2024-05-09T08:43:39.457880+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:14.227590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605456+0000","last_change":"2024-05-09T08:48:22.376751+0000","last_active":"2024-05-09T08:50:38.605456+0000","last_peered":"2024-05-09T08:50:38.605456+0000","last_clean":"2024-05-09T08:50:38.605456+0000","last_became_active":"2024-05-09T08:48:22.376322+0000","last_became_peered":"2024-05-09T08:48:22.376322+0000","last_unstale":"2024-05-09T08:50:38.605456+0000","last_undegraded":"2024-05-09T08:50:38.605456+0000","last_fullsized":"2024-05-09T08:50:38.605456+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:08:07.859034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737852+0000","last_change":"2024-05-09T08:48:18.364063+0000","last_active":"2024-05-09T08:50:38.737852+0000","last_peered":"2024-05-09T08:50:38.737852+0000","last_clean":"2024-05-09T08:50:38.737852+0000","last_became_active":"2024-05-09T08:48:18.363694+0000","last_became_peered":"2024-05-09T08:48:18.363694+0000","last_unstale":"2024-05-09T08:50:38.737852+0000","last_undegraded":"2024-05-09T08:50:38.737852+0000","last_fullsized":"2024-05-09T08:50:38.737852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:42.536329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739870+0000","last_change":"2024-05-09T08:48:17.599787+0000","last_active":"2024-05-09T08:50:38.739870+0000","last_peered":"2024-05-09T08:50:38.739870+0000","last_clean":"2024-05-09T08:50:38.739870+0000","last_became_active":"2024-05-09T08:48:17.599248+0000","last_became_peered":"2024-05-09T08:48:17.599248+0000","last_unstale":"2024-05-09T08:50:38.739870+0000","last_undegraded":"2024-05-09T08:50:38.739870+0000","last_fullsized":"2024-05-09T08:50:38.739870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:17.448404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664823+0000","last_change":"2024-05-09T08:48:22.378913+0000","last_active":"2024-05-09T08:50:38.664823+0000","last_peered":"2024-05-09T08:50:38.664823+0000","last_clean":"2024-05-09T08:50:38.664823+0000","last_became_active":"2024-05-09T08:48:22.378441+0000","last_became_peered":"2024-05-09T08:48:22.378441+0000","last_unstale":"2024-05-09T08:50:38.664823+0000","last_undegraded":"2024-05-09T08:50:38.664823+0000","last_fullsized":"2024-05-09T08:50:38.664823+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:49.623778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498746+0000","last_change":"2024-05-09T08:48:17.600121+0000","last_active":"2024-05-09T08:50:38.498746+0000","last_peered":"2024-05-09T08:50:38.498746+0000","last_clean":"2024-05-09T08:50:38.498746+0000","last_became_active":"2024-05-09T08:48:17.599800+0000","last_became_peered":"2024-05-09T08:48:17.599800+0000","last_unstale":"2024-05-09T08:50:38.498746+0000","last_undegraded":"2024-05-09T08:50:38.498746+0000","last_fullsized":"2024-05-09T08:50:38.498746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:56.607393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738578+0000","last_change":"2024-05-09T08:48:18.366612+0000","last_active":"2024-05-09T08:50:38.738578+0000","last_peered":"2024-05-09T08:50:38.738578+0000","last_clean":"2024-05-09T08:50:38.738578+0000","last_became_active":"2024-05-09T08:48:18.366266+0000","last_became_peered":"2024-05-09T08:48:18.366266+0000","last_unstale":"2024-05-09T08:50:38.738578+0000","last_undegraded":"2024-05-09T08:50:38.738578+0000","last_fullsized":"2024-05-09T08:50:38.738578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:35:01.663456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497267+0000","last_change":"2024-05-09T08:48:22.384096+0000","last_active":"2024-05-09T08:50:38.497267+0000","last_peered":"2024-05-09T08:50:38.497267+0000","last_clean":"2024-05-09T08:50:38.497267+0000","last_became_active":"2024-05-09T08:48:22.383656+0000","last_became_peered":"2024-05-09T08:48:22.383656+0000","last_unstale":"2024-05-09T08:50:38.497267+0000","last_undegraded":"2024-05-09T08:50:38.497267+0000","last_fullsized":"2024-05-09T08:50:38.497267+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:13.622377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'48","reported_seq":105,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.707713+0000","last_change":"2024-05-09T08:48:17.593684+0000","last_active":"2024-05-09T08:50:38.707713+0000","last_peered":"2024-05-09T08:50:38.707713+0000","last_clean":"2024-05-09T08:50:38.707713+0000","last_became_active":"2024-05-09T08:48:17.593358+0000","last_became_peered":"2024-05-09T08:48:17.593358+0000","last_unstale":"2024-05-09T08:50:38.707713+0000","last_undegraded":"2024-05-09T08:50:38.707713+0000","last_fullsized":"2024-05-09T08:50:38.707713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:51:36.023421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497783+0000","last_change":"2024-05-09T08:48:18.374498+0000","last_active":"2024-05-09T08:50:38.497783+0000","last_peered":"2024-05-09T08:50:38.497783+0000","last_clean":"2024-05-09T08:50:38.497783+0000","last_became_active":"2024-05-09T08:48:18.374142+0000","last_became_peered":"2024-05-09T08:48:18.374142+0000","last_unstale":"2024-05-09T08:50:38.497783+0000","last_undegraded":"2024-05-09T08:50:38.497783+0000","last_fullsized":"2024-05-09T08:50:38.497783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:53.874419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738424+0000","last_change":"2024-05-09T08:48:22.378330+0000","last_active":"2024-05-09T08:50:38.738424+0000","last_peered":"2024-05-09T08:50:38.738424+0000","last_clean":"2024-05-09T08:50:38.738424+0000","last_became_active":"2024-05-09T08:48:22.377218+0000","last_became_peered":"2024-05-09T08:48:22.377218+0000","last_unstale":"2024-05-09T08:50:38.738424+0000","last_undegraded":"2024-05-09T08:50:38.738424+0000","last_fullsized":"2024-05-09T08:50:38.738424+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:41.650168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738328+0000","last_change":"2024-05-09T08:48:17.601410+0000","last_active":"2024-05-09T08:50:38.738328+0000","last_peered":"2024-05-09T08:50:38.738328+0000","last_clean":"2024-05-09T08:50:38.738328+0000","last_became_active":"2024-05-09T08:48:17.601075+0000","last_became_peered":"2024-05-09T08:48:17.601075+0000","last_unstale":"2024-05-09T08:50:38.738328+0000","last_undegraded":"2024-05-09T08:50:38.738328+0000","last_fullsized":"2024-05-09T08:50:38.738328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:45.760280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.741119+0000","last_change":"2024-05-09T08:48:18.379590+0000","last_active":"2024-05-09T08:50:38.741119+0000","last_peered":"2024-05-09T08:50:38.741119+0000","last_clean":"2024-05-09T08:50:38.741119+0000","last_became_active":"2024-05-09T08:48:18.379187+0000","last_became_peered":"2024-05-09T08:48:18.379187+0000","last_unstale":"2024-05-09T08:50:38.741119+0000","last_undegraded":"2024-05-09T08:50:38.741119+0000","last_fullsized":"2024-05-09T08:50:38.741119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:40:02.630449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.737875+0000","last_change":"2024-05-09T08:48:22.386437+0000","last_active":"2024-05-09T08:50:38.737875+0000","last_peered":"2024-05-09T08:50:38.737875+0000","last_clean":"2024-05-09T08:50:38.737875+0000","last_became_active":"2024-05-09T08:48:22.385973+0000","last_became_peered":"2024-05-09T08:48:22.385973+0000","last_unstale":"2024-05-09T08:50:38.737875+0000","last_undegraded":"2024-05-09T08:50:38.737875+0000","last_fullsized":"2024-05-09T08:50:38.737875+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:11:37.338641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743068+0000","last_change":"2024-05-09T08:48:17.598586+0000","last_active":"2024-05-09T08:50:38.743068+0000","last_peered":"2024-05-09T08:50:38.743068+0000","last_clean":"2024-05-09T08:50:38.743068+0000","last_became_active":"2024-05-09T08:48:17.597851+0000","last_became_peered":"2024-05-09T08:48:17.597851+0000","last_unstale":"2024-05-09T08:50:38.743068+0000","last_undegraded":"2024-05-09T08:50:38.743068+0000","last_fullsized":"2024-05-09T08:50:38.743068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:28:22.996999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498169+0000","last_change":"2024-05-09T08:48:18.367319+0000","last_active":"2024-05-09T08:50:38.498169+0000","last_peered":"2024-05-09T08:50:38.498169+0000","last_clean":"2024-05-09T08:50:38.498169+0000","last_became_active":"2024-05-09T08:48:18.366034+0000","last_became_peered":"2024-05-09T08:48:18.366034+0000","last_unstale":"2024-05-09T08:50:38.498169+0000","last_undegraded":"2024-05-09T08:50:38.498169+0000","last_fullsized":"2024-05-09T08:50:38.498169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:22.998089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597628+0000","last_change":"2024-05-09T08:48:22.389129+0000","last_active":"2024-05-09T08:50:38.597628+0000","last_peered":"2024-05-09T08:50:38.597628+0000","last_clean":"2024-05-09T08:50:38.597628+0000","last_became_active":"2024-05-09T08:48:22.386639+0000","last_became_peered":"2024-05-09T08:48:22.386639+0000","last_unstale":"2024-05-09T08:50:38.597628+0000","last_undegraded":"2024-05-09T08:50:38.597628+0000","last_fullsized":"2024-05-09T08:50:38.597628+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:47:07.257464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497375+0000","last_change":"2024-05-09T08:48:17.595255+0000","last_active":"2024-05-09T08:50:38.497375+0000","last_peered":"2024-05-09T08:50:38.497375+0000","last_clean":"2024-05-09T08:50:38.497375+0000","last_became_active":"2024-05-09T08:48:17.594882+0000","last_became_peered":"2024-05-09T08:48:17.594882+0000","last_unstale":"2024-05-09T08:50:38.497375+0000","last_undegraded":"2024-05-09T08:50:38.497375+0000","last_fullsized":"2024-05-09T08:50:38.497375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:49.845326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664529+0000","last_change":"2024-05-09T08:48:18.378580+0000","last_active":"2024-05-09T08:50:38.664529+0000","last_peered":"2024-05-09T08:50:38.664529+0000","last_clean":"2024-05-09T08:50:38.664529+0000","last_became_active":"2024-05-09T08:48:18.378205+0000","last_became_peered":"2024-05-09T08:48:18.378205+0000","last_unstale":"2024-05-09T08:50:38.664529+0000","last_undegraded":"2024-05-09T08:50:38.664529+0000","last_fullsized":"2024-05-09T08:50:38.664529+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:50.103195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738066+0000","last_change":"2024-05-09T08:48:22.378494+0000","last_active":"2024-05-09T08:50:38.738066+0000","last_peered":"2024-05-09T08:50:38.738066+0000","last_clean":"2024-05-09T08:50:38.738066+0000","last_became_active":"2024-05-09T08:48:22.378076+0000","last_became_peered":"2024-05-09T08:48:22.378076+0000","last_unstale":"2024-05-09T08:50:38.738066+0000","last_undegraded":"2024-05-09T08:50:38.738066+0000","last_fullsized":"2024-05-09T08:50:38.738066+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:40:09.079968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'4","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664581+0000","last_change":"2024-05-09T08:50:19.307437+0000","last_active":"2024-05-09T08:50:38.664581+0000","last_peered":"2024-05-09T08:50:38.664581+0000","last_clean":"2024-05-09T08:50:38.664581+0000","last_became_active":"2024-05-09T08:50:19.293112+0000","last_became_peered":"2024-05-09T08:50:19.293112+0000","last_unstale":"2024-05-09T08:50:38.664581+0000","last_undegraded":"2024-05-09T08:50:38.664581+0000","last_fullsized":"2024-05-09T08:50:38.664581+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:20.776030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,1],"acting":[11,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497703+0000","last_change":"2024-05-09T08:48:18.373569+0000","last_active":"2024-05-09T08:50:38.497703+0000","last_peered":"2024-05-09T08:50:38.497703+0000","last_clean":"2024-05-09T08:50:38.497703+0000","last_became_active":"2024-05-09T08:48:18.373238+0000","last_became_peered":"2024-05-09T08:48:18.373238+0000","last_unstale":"2024-05-09T08:50:38.497703+0000","last_undegraded":"2024-05-09T08:50:38.497703+0000","last_fullsized":"2024-05-09T08:50:38.497703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:17:41.629419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738071+0000","last_change":"2024-05-09T08:48:22.381523+0000","last_active":"2024-05-09T08:50:38.738071+0000","last_peered":"2024-05-09T08:50:38.738071+0000","last_clean":"2024-05-09T08:50:38.738071+0000","last_became_active":"2024-05-09T08:48:22.380626+0000","last_became_peered":"2024-05-09T08:48:22.380626+0000","last_unstale":"2024-05-09T08:50:38.738071+0000","last_undegraded":"2024-05-09T08:50:38.738071+0000","last_fullsized":"2024-05-09T08:50:38.738071+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:47:45.092232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.498421+0000","last_change":"2024-05-09T08:48:17.601479+0000","last_active":"2024-05-09T08:50:38.498421+0000","last_peered":"2024-05-09T08:50:38.498421+0000","last_clean":"2024-05-09T08:50:38.498421+0000","last_became_active":"2024-05-09T08:48:17.601140+0000","last_became_peered":"2024-05-09T08:48:17.601140+0000","last_unstale":"2024-05-09T08:50:38.498421+0000","last_undegraded":"2024-05-09T08:50:38.498421+0000","last_fullsized":"2024-05-09T08:50:38.498421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:38:24.011209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740038+0000","last_change":"2024-05-09T08:48:18.376890+0000","last_active":"2024-05-09T08:50:38.740038+0000","last_peered":"2024-05-09T08:50:38.740038+0000","last_clean":"2024-05-09T08:50:38.740038+0000","last_became_active":"2024-05-09T08:48:18.376472+0000","last_became_peered":"2024-05-09T08:48:18.376472+0000","last_unstale":"2024-05-09T08:50:38.740038+0000","last_undegraded":"2024-05-09T08:50:38.740038+0000","last_fullsized":"2024-05-09T08:50:38.740038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:40:16.238539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743128+0000","last_change":"2024-05-09T08:48:22.374743+0000","last_active":"2024-05-09T08:50:38.743128+0000","last_peered":"2024-05-09T08:50:38.743128+0000","last_clean":"2024-05-09T08:50:38.743128+0000","last_became_active":"2024-05-09T08:48:22.374324+0000","last_became_peered":"2024-05-09T08:48:22.374324+0000","last_unstale":"2024-05-09T08:50:38.743128+0000","last_undegraded":"2024-05-09T08:50:38.743128+0000","last_fullsized":"2024-05-09T08:50:38.743128+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:48.073647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"95'6","reported_seq":163,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.933811+0000","last_change":"2024-05-09T08:48:17.604700+0000","last_active":"2024-05-09T08:50:38.933811+0000","last_peered":"2024-05-09T08:50:38.933811+0000","last_clean":"2024-05-09T08:50:38.933811+0000","last_became_active":"2024-05-09T08:48:17.604334+0000","last_became_peered":"2024-05-09T08:48:17.604334+0000","last_unstale":"2024-05-09T08:50:38.933811+0000","last_undegraded":"2024-05-09T08:50:38.933811+0000","last_fullsized":"2024-05-09T08:50:38.933811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:15:37.346232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740574+0000","last_change":"2024-05-09T08:48:18.375769+0000","last_active":"2024-05-09T08:50:38.740574+0000","last_peered":"2024-05-09T08:50:38.740574+0000","last_clean":"2024-05-09T08:50:38.740574+0000","last_became_active":"2024-05-09T08:48:18.375059+0000","last_became_peered":"2024-05-09T08:48:18.375059+0000","last_unstale":"2024-05-09T08:50:38.740574+0000","last_undegraded":"2024-05-09T08:50:38.740574+0000","last_fullsized":"2024-05-09T08:50:38.740574+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:51.001240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740616+0000","last_change":"2024-05-09T08:48:22.385306+0000","last_active":"2024-05-09T08:50:38.740616+0000","last_peered":"2024-05-09T08:50:38.740616+0000","last_clean":"2024-05-09T08:50:38.740616+0000","last_became_active":"2024-05-09T08:48:22.383890+0000","last_became_peered":"2024-05-09T08:48:22.383890+0000","last_unstale":"2024-05-09T08:50:38.740616+0000","last_undegraded":"2024-05-09T08:50:38.740616+0000","last_fullsized":"2024-05-09T08:50:38.740616+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:50:52.369399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'41","reported_seq":83,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.706348+0000","last_change":"2024-05-09T08:48:17.591627+0000","last_active":"2024-05-09T08:50:38.706348+0000","last_peered":"2024-05-09T08:50:38.706348+0000","last_clean":"2024-05-09T08:50:38.706348+0000","last_became_active":"2024-05-09T08:48:17.591250+0000","last_became_peered":"2024-05-09T08:48:17.591250+0000","last_unstale":"2024-05-09T08:50:38.706348+0000","last_undegraded":"2024-05-09T08:50:38.706348+0000","last_fullsized":"2024-05-09T08:50:38.706348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:40:59.748892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740833+0000","last_change":"2024-05-09T08:48:18.372256+0000","last_active":"2024-05-09T08:50:38.740833+0000","last_peered":"2024-05-09T08:50:38.740833+0000","last_clean":"2024-05-09T08:50:38.740833+0000","last_became_active":"2024-05-09T08:48:18.371895+0000","last_became_peered":"2024-05-09T08:48:18.371895+0000","last_unstale":"2024-05-09T08:50:38.740833+0000","last_undegraded":"2024-05-09T08:50:38.740833+0000","last_fullsized":"2024-05-09T08:50:38.740833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:25.067587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605019+0000","last_change":"2024-05-09T08:48:22.373684+0000","last_active":"2024-05-09T08:50:38.605019+0000","last_peered":"2024-05-09T08:50:38.605019+0000","last_clean":"2024-05-09T08:50:38.605019+0000","last_became_active":"2024-05-09T08:48:22.373000+0000","last_became_peered":"2024-05-09T08:48:22.373000+0000","last_unstale":"2024-05-09T08:50:38.605019+0000","last_undegraded":"2024-05-09T08:50:38.605019+0000","last_fullsized":"2024-05-09T08:50:38.605019+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:21.356323+0000","last_clean_scrub_stamp":"2024-05-09T08:48:21.356323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:53:14.416837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.739078+0000","last_change":"2024-05-09T08:48:17.596910+0000","last_active":"2024-05-09T08:50:38.739078+0000","last_peered":"2024-05-09T08:50:38.739078+0000","last_clean":"2024-05-09T08:50:38.739078+0000","last_became_active":"2024-05-09T08:48:17.596605+0000","last_became_peered":"2024-05-09T08:48:17.596605+0000","last_unstale":"2024-05-09T08:50:38.739078+0000","last_undegraded":"2024-05-09T08:50:38.739078+0000","last_fullsized":"2024-05-09T08:50:38.739078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:16.571794+0000","last_clean_scrub_stamp":"2024-05-09T08:48:16.571794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:05:25.880082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740734+0000","last_change":"2024-05-09T08:48:18.370354+0000","last_active":"2024-05-09T08:50:38.740734+0000","last_peered":"2024-05-09T08:50:38.740734+0000","last_clean":"2024-05-09T08:50:38.740734+0000","last_became_active":"2024-05-09T08:48:18.370043+0000","last_became_peered":"2024-05-09T08:48:18.370043+0000","last_unstale":"2024-05-09T08:50:38.740734+0000","last_undegraded":"2024-05-09T08:50:38.740734+0000","last_fullsized":"2024-05-09T08:50:38.740734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:21:34.066162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664588+0000","last_change":"2024-05-09T08:48:18.376431+0000","last_active":"2024-05-09T08:50:38.664588+0000","last_peered":"2024-05-09T08:50:38.664588+0000","last_clean":"2024-05-09T08:50:38.664588+0000","last_became_active":"2024-05-09T08:48:18.375598+0000","last_became_peered":"2024-05-09T08:48:18.375598+0000","last_unstale":"2024-05-09T08:50:38.664588+0000","last_undegraded":"2024-05-09T08:50:38.664588+0000","last_fullsized":"2024-05-09T08:50:38.664588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:51:13.481219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738676+0000","last_change":"2024-05-09T08:48:18.371460+0000","last_active":"2024-05-09T08:50:38.738676+0000","last_peered":"2024-05-09T08:50:38.738676+0000","last_clean":"2024-05-09T08:50:38.738676+0000","last_became_active":"2024-05-09T08:48:18.371091+0000","last_became_peered":"2024-05-09T08:48:18.371091+0000","last_unstale":"2024-05-09T08:50:38.738676+0000","last_undegraded":"2024-05-09T08:50:38.738676+0000","last_fullsized":"2024-05-09T08:50:38.738676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:02.829902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738487+0000","last_change":"2024-05-09T08:48:18.369453+0000","last_active":"2024-05-09T08:50:38.738487+0000","last_peered":"2024-05-09T08:50:38.738487+0000","last_clean":"2024-05-09T08:50:38.738487+0000","last_became_active":"2024-05-09T08:48:18.368966+0000","last_became_peered":"2024-05-09T08:48:18.368966+0000","last_unstale":"2024-05-09T08:50:38.738487+0000","last_undegraded":"2024-05-09T08:50:38.738487+0000","last_fullsized":"2024-05-09T08:50:38.738487+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:23:14.359222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.740734+0000","last_change":"2024-05-09T08:48:18.378815+0000","last_active":"2024-05-09T08:50:38.740734+0000","last_peered":"2024-05-09T08:50:38.740734+0000","last_clean":"2024-05-09T08:50:38.740734+0000","last_became_active":"2024-05-09T08:48:18.378479+0000","last_became_peered":"2024-05-09T08:48:18.378479+0000","last_unstale":"2024-05-09T08:50:38.740734+0000","last_undegraded":"2024-05-09T08:50:38.740734+0000","last_fullsized":"2024-05-09T08:50:38.740734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:38:50.133239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738648+0000","last_change":"2024-05-09T08:48:18.365696+0000","last_active":"2024-05-09T08:50:38.738648+0000","last_peered":"2024-05-09T08:50:38.738648+0000","last_clean":"2024-05-09T08:50:38.738648+0000","last_became_active":"2024-05-09T08:48:18.365358+0000","last_became_peered":"2024-05-09T08:48:18.365358+0000","last_unstale":"2024-05-09T08:50:38.738648+0000","last_undegraded":"2024-05-09T08:50:38.738648+0000","last_fullsized":"2024-05-09T08:50:38.738648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:04.064150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.665012+0000","last_change":"2024-05-09T08:48:18.368911+0000","last_active":"2024-05-09T08:50:38.665012+0000","last_peered":"2024-05-09T08:50:38.665012+0000","last_clean":"2024-05-09T08:50:38.665012+0000","last_became_active":"2024-05-09T08:48:18.368591+0000","last_became_peered":"2024-05-09T08:48:18.368591+0000","last_unstale":"2024-05-09T08:50:38.665012+0000","last_undegraded":"2024-05-09T08:50:38.665012+0000","last_fullsized":"2024-05-09T08:50:38.665012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:34:09.692168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738963+0000","last_change":"2024-05-09T08:48:18.373629+0000","last_active":"2024-05-09T08:50:38.738963+0000","last_peered":"2024-05-09T08:50:38.738963+0000","last_clean":"2024-05-09T08:50:38.738963+0000","last_became_active":"2024-05-09T08:48:18.372992+0000","last_became_peered":"2024-05-09T08:48:18.372992+0000","last_unstale":"2024-05-09T08:50:38.738963+0000","last_undegraded":"2024-05-09T08:50:38.738963+0000","last_fullsized":"2024-05-09T08:50:38.738963+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:42.532250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.597288+0000","last_change":"2024-05-09T08:50:19.289044+0000","last_active":"2024-05-09T08:50:38.597288+0000","last_peered":"2024-05-09T08:50:38.597288+0000","last_clean":"2024-05-09T08:50:38.597288+0000","last_became_active":"2024-05-09T08:50:19.288657+0000","last_became_peered":"2024-05-09T08:50:19.288657+0000","last_unstale":"2024-05-09T08:50:38.597288+0000","last_undegraded":"2024-05-09T08:50:38.597288+0000","last_fullsized":"2024-05-09T08:50:38.597288+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:09:36.183086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.605085+0000","last_change":"2024-05-09T08:48:18.370460+0000","last_active":"2024-05-09T08:50:38.605085+0000","last_peered":"2024-05-09T08:50:38.605085+0000","last_clean":"2024-05-09T08:50:38.605085+0000","last_became_active":"2024-05-09T08:48:18.370033+0000","last_became_peered":"2024-05-09T08:48:18.370033+0000","last_unstale":"2024-05-09T08:50:38.605085+0000","last_undegraded":"2024-05-09T08:50:38.605085+0000","last_fullsized":"2024-05-09T08:50:38.605085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:21:28.622511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.497468+0000","last_change":"2024-05-09T08:48:18.371410+0000","last_active":"2024-05-09T08:50:38.497468+0000","last_peered":"2024-05-09T08:50:38.497468+0000","last_clean":"2024-05-09T08:50:38.497468+0000","last_became_active":"2024-05-09T08:48:18.371067+0000","last_became_peered":"2024-05-09T08:48:18.371067+0000","last_unstale":"2024-05-09T08:50:38.497468+0000","last_undegraded":"2024-05-09T08:50:38.497468+0000","last_fullsized":"2024-05-09T08:50:38.497468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:35:11.445992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.743344+0000","last_change":"2024-05-09T08:48:18.382873+0000","last_active":"2024-05-09T08:50:38.743344+0000","last_peered":"2024-05-09T08:50:38.743344+0000","last_clean":"2024-05-09T08:50:38.743344+0000","last_became_active":"2024-05-09T08:48:18.382366+0000","last_became_peered":"2024-05-09T08:48:18.382366+0000","last_unstale":"2024-05-09T08:50:38.743344+0000","last_undegraded":"2024-05-09T08:50:38.743344+0000","last_fullsized":"2024-05-09T08:50:38.743344+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:23.343131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.738362+0000","last_change":"2024-05-09T08:48:18.370821+0000","last_active":"2024-05-09T08:50:38.738362+0000","last_peered":"2024-05-09T08:50:38.738362+0000","last_clean":"2024-05-09T08:50:38.738362+0000","last_became_active":"2024-05-09T08:48:18.368208+0000","last_became_peered":"2024-05-09T08:48:18.368208+0000","last_unstale":"2024-05-09T08:50:38.738362+0000","last_undegraded":"2024-05-09T08:50:38.738362+0000","last_fullsized":"2024-05-09T08:50:38.738362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:14.633938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-09T08:50:38.664977+0000","last_change":"2024-05-09T08:48:18.371626+0000","last_active":"2024-05-09T08:50:38.664977+0000","last_peered":"2024-05-09T08:50:38.664977+0000","last_clean":"2024-05-09T08:50:38.664977+0000","last_became_active":"2024-05-09T08:48:18.371081+0000","last_became_peered":"2024-05-09T08:48:18.371081+0000","last_unstale":"2024-05-09T08:50:38.664977+0000","last_undegraded":"2024-05-09T08:50:38.664977+0000","last_fullsized":"2024-05-09T08:50:38.664977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:48:17.573602+0000","last_clean_scrub_stamp":"2024-05-09T08:48:17.573602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:27.372934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106929,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599687,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1352,"num_read_kb":793,"num_write":374,"num_write_kb":1468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":1799061,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351017,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":534442,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612654,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28216,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716936,"statfs":{"total":95995035648,"available":95966142464,"internally_reserved":0,"allocated":1224704,"data_stored":1323322,"data_compressed":71341,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776179,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":927129,"data_compressed":48688,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037817,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":927507,"data_compressed":48688,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299453,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":1132551,"data_compressed":65185,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462979,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":529808,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1118208,"data_stored":596742,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":531264,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1130496,"data_stored":597734,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1105920,"data_stored":1053941,"data_compressed":54844,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888141,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":534414,"data_compressed":29542,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724615,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1150976,"data_stored":1258419,"data_compressed":71341,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35643,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135260,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":200163,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:50:44.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:50:44.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.350+0000 7fca2e710640 1 -- 172.21.15.98:0/3489546230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca281033b0 msgr2=0x7fca2810b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.350+0000 7fca2e710640 1 --2- 172.21.15.98:0/3489546230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca281033b0 0x7fca2810b890 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009900 tx=0x7fca1c02a480 comp rx=0 tx=0).stop 2024-05-09T08:50:44.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.350+0000 7fca2e710640 1 -- 172.21.15.98:0/3489546230 shutdown_connections 2024-05-09T08:50:44.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.350+0000 7fca2e710640 1 --2- 172.21.15.98:0/3489546230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca2810be60 0x7fca2810e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.350+0000 7fca2e710640 1 --2- 172.21.15.98:0/3489546230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca281033b0 0x7fca2810b890 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.350+0000 7fca2e710640 1 --2- 172.21.15.98:0/3489546230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca28102a90 0x7fca28102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.350+0000 7fca2e710640 1 -- 172.21.15.98:0/3489546230 >> 172.21.15.98:0/3489546230 conn(0x7fca28076ab0 msgr2=0x7fca28076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:44.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.350+0000 7fca2e710640 1 -- 172.21.15.98:0/3489546230 shutdown_connections 2024-05-09T08:50:44.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.350+0000 7fca2e710640 1 -- 172.21.15.98:0/3489546230 wait complete. 2024-05-09T08:50:44.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.350+0000 7fca2e710640 1 Processor -- start 2024-05-09T08:50:44.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca2e710640 1 -- start start 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca2e710640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca28102a90 0x7fca281006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca2e710640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca281033b0 0x7fca28100c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca2e710640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca2810be60 0x7fca28101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca2e710640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca28111bf0 con 0x7fca28102a90 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca2e710640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fca28111a70 con 0x7fca2810be60 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca2e710640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fca28111d70 con 0x7fca281033b0 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca27fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca28102a90 0x7fca281006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca27fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca28102a90 0x7fca281006e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43260/0 (socket says 172.21.15.98:43260) 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca27fff640 1 -- 172.21.15.98:0/3030512260 learned_addr learned my addr 172.21.15.98:0/3030512260 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca277fe640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca281033b0 0x7fca28100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca2ccad640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca2810be60 0x7fca28101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca27fff640 1 -- 172.21.15.98:0/3030512260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca281033b0 msgr2=0x7fca28100c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca27fff640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca281033b0 0x7fca28100c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca27fff640 1 -- 172.21.15.98:0/3030512260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca2810be60 msgr2=0x7fca28101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca27fff640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca2810be60 0x7fca28101160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca27fff640 1 -- 172.21.15.98:0/3030512260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca281a6550 con 0x7fca28102a90 2024-05-09T08:50:44.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca2ccad640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca2810be60 0x7fca28101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:44.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca277fe640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca281033b0 0x7fca28100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:44.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca27fff640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca28102a90 0x7fca281006e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fca1800c920 tx=0x7fca1800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:44.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.351+0000 7fca257fa640 1 -- 172.21.15.98:0/3030512260 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca18007c70 con 0x7fca28102a90 2024-05-09T08:50:44.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.352+0000 7fca257fa640 1 -- 172.21.15.98:0/3030512260 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca18007e10 con 0x7fca28102a90 2024-05-09T08:50:44.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.352+0000 7fca2e710640 1 -- 172.21.15.98:0/3030512260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca281a6840 con 0x7fca28102a90 2024-05-09T08:50:44.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.352+0000 7fca257fa640 1 -- 172.21.15.98:0/3030512260 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca18018890 con 0x7fca28102a90 2024-05-09T08:50:44.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.352+0000 7fca2e710640 1 -- 172.21.15.98:0/3030512260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca281a6d90 con 0x7fca28102a90 2024-05-09T08:50:44.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.352+0000 7fca2e710640 1 -- 172.21.15.98:0/3030512260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca2806a6b0 con 0x7fca28102a90 2024-05-09T08:50:44.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.354+0000 7fca257fa640 1 -- 172.21.15.98:0/3030512260 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fca18018a30 con 0x7fca28102a90 2024-05-09T08:50:44.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.354+0000 7fca257fa640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fca100a2e50 0x7fca100a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.354+0000 7fca277fe640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fca100a2e50 0x7fca100a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.354+0000 7fca257fa640 1 -- 172.21.15.98:0/3030512260 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fca180cfc80 con 0x7fca28102a90 2024-05-09T08:50:44.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.355+0000 7fca277fe640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fca100a2e50 0x7fca100a5310 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fca28102120 tx=0x7fca1c005a20 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:44.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.359+0000 7fca257fa640 1 -- 172.21.15.98:0/3030512260 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fca18014030 con 0x7fca28102a90 2024-05-09T08:50:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.539+0000 7fca2e710640 1 -- 172.21.15.98:0/3030512260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fca281a7090 con 0x7fca28102a90 2024-05-09T08:50:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.540+0000 7fca257fa640 1 -- 172.21.15.98:0/3030512260 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25206 (secure 0 0 0) 0x7fca1801f300 con 0x7fca28102a90 2024-05-09T08:50:44.540 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:44.540 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":95,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:50:38.492526+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":1}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1013153207":"2024-05-10T08:50:38.492493+0000","172.21.15.98:6800/1228297308":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/808614683":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/1734294041":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/2113007677":"2024-05-10T08:50:38.492493+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1228297308":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:50:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.542+0000 7fca2e710640 1 -- 172.21.15.98:0/3030512260 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fca100a2e50 msgr2=0x7fca100a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.542+0000 7fca2e710640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fca100a2e50 0x7fca100a5310 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fca28102120 tx=0x7fca1c005a20 comp rx=0 tx=0).stop 2024-05-09T08:50:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.542+0000 7fca2e710640 1 -- 172.21.15.98:0/3030512260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca28102a90 msgr2=0x7fca281006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.542+0000 7fca2e710640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca28102a90 0x7fca281006e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fca1800c920 tx=0x7fca1800cde0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.543+0000 7fca2e710640 1 -- 172.21.15.98:0/3030512260 shutdown_connections 2024-05-09T08:50:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.543+0000 7fca2e710640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fca100a2e50 0x7fca100a5310 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.543+0000 7fca2e710640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca2810be60 0x7fca28101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.543+0000 7fca2e710640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca281033b0 0x7fca28100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.543+0000 7fca2e710640 1 --2- 172.21.15.98:0/3030512260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca28102a90 0x7fca281006e0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.543+0000 7fca2e710640 1 -- 172.21.15.98:0/3030512260 >> 172.21.15.98:0/3030512260 conn(0x7fca28076ab0 msgr2=0x7fca280fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.543+0000 7fca2e710640 1 -- 172.21.15.98:0/3030512260 shutdown_connections 2024-05-09T08:50:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.543+0000 7fca2e710640 1 -- 172.21.15.98:0/3030512260 wait complete. 2024-05-09T08:50:44.552 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:44 smithi098 ceph-mon[30092]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:50:44.552 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:44 smithi098 ceph-mon[30092]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:50:44.552 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:44 smithi098 ceph-mon[30092]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:50:44.552 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:44 smithi098 ceph-mon[30092]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:50:44.552 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:44 smithi098 ceph-mon[30092]: Updating smithi105:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:50:44.552 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:44 smithi098 ceph-mon[30092]: Updating smithi098:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:50:44.552 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:44 smithi098 ceph-mon[30092]: Updating smithi142:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:50:44.552 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:44 smithi098 ceph-mon[30092]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-05-09T08:50:44.554 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:43:39.424443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:48:15.866450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:48:17.122162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:48:20.591475+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:50:44.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:50:44.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:44 smithi142 ceph-mon[34625]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:50:44.826 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:44 smithi142 ceph-mon[34625]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:50:44.826 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:44 smithi142 ceph-mon[34625]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:50:44.826 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:44 smithi142 ceph-mon[34625]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:50:44.827 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:44 smithi142 ceph-mon[34625]: Updating smithi105:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:50:44.827 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:44 smithi142 ceph-mon[34625]: Updating smithi098:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:50:44.827 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:44 smithi142 ceph-mon[34625]: Updating smithi142:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:50:44.827 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:44 smithi142 ceph-mon[34625]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-05-09T08:50:44.828 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:44 smithi105 ceph-mon[34498]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:50:44.828 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:44 smithi105 ceph-mon[34498]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:50:44.828 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:44 smithi105 ceph-mon[34498]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:50:44.828 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:44 smithi105 ceph-mon[34498]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:50:44.828 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:44 smithi105 ceph-mon[34498]: Updating smithi105:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:50:44.828 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:44 smithi105 ceph-mon[34498]: Updating smithi098:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:50:44.828 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:44 smithi105 ceph-mon[34498]: Updating smithi142:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:50:44.828 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:44 smithi105 ceph-mon[34498]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-05-09T08:50:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.659+0000 7fe18a909640 1 -- 172.21.15.98:0/2077597550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe184100a80 msgr2=0x7fe184100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.659+0000 7fe18a909640 1 --2- 172.21.15.98:0/2077597550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe184100a80 0x7fe184100ee0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009900 tx=0x7fe16c02a410 comp rx=0 tx=0).stop 2024-05-09T08:50:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.659+0000 7fe18a909640 1 -- 172.21.15.98:0/2077597550 shutdown_connections 2024-05-09T08:50:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.659+0000 7fe18a909640 1 --2- 172.21.15.98:0/2077597550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe184101420 0x7fe18410d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.659+0000 7fe18a909640 1 --2- 172.21.15.98:0/2077597550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe184100a80 0x7fe184100ee0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.659+0000 7fe18a909640 1 --2- 172.21.15.98:0/2077597550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe184106780 0x7fe184106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.659+0000 7fe18a909640 1 -- 172.21.15.98:0/2077597550 >> 172.21.15.98:0/2077597550 conn(0x7fe184076530 msgr2=0x7fe184076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.659+0000 7fe18a909640 1 -- 172.21.15.98:0/2077597550 shutdown_connections 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.659+0000 7fe18a909640 1 -- 172.21.15.98:0/2077597550 wait complete. 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe18a909640 1 Processor -- start 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe18a909640 1 -- start start 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe18a909640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe184100a80 0x7fe184196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe18a909640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe184101420 0x7fe184197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe18a909640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe184106780 0x7fe18419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe18a909640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe184110ad0 con 0x7fe184106780 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe18a909640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe184110950 con 0x7fe184101420 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe18a909640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe184110c50 con 0x7fe184100a80 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe188ea6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe184106780 0x7fe18419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe188ea6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe184106780 0x7fe18419b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43280/0 (socket says 172.21.15.98:43280) 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe188ea6640 1 -- 172.21.15.98:0/731995644 learned_addr learned my addr 172.21.15.98:0/731995644 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe183fff640 1 --2- 172.21.15.98:0/731995644 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe184100a80 0x7fe184196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe188ea6640 1 -- 172.21.15.98:0/731995644 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe184100a80 msgr2=0x7fe184196ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe188ea6640 1 --2- 172.21.15.98:0/731995644 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe184100a80 0x7fe184196ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe17bfff640 1 --2- 172.21.15.98:0/731995644 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe184101420 0x7fe184197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe188ea6640 1 -- 172.21.15.98:0/731995644 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe184101420 msgr2=0x7fe184197530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe188ea6640 1 --2- 172.21.15.98:0/731995644 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe184101420 0x7fe184197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe188ea6640 1 -- 172.21.15.98:0/731995644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18419bf90 con 0x7fe184106780 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.660+0000 7fe188ea6640 1 --2- 172.21.15.98:0/731995644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe184106780 0x7fe18419b860 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00eaa0 tx=0x7fe17c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.661+0000 7fe181ffb640 1 -- 172.21.15.98:0/731995644 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c00cdb0 con 0x7fe184106780 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.661+0000 7fe181ffb640 1 -- 172.21.15.98:0/731995644 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe17c005020 con 0x7fe184106780 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.661+0000 7fe18a909640 1 -- 172.21.15.98:0/731995644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe18419c280 con 0x7fe184106780 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.661+0000 7fe181ffb640 1 -- 172.21.15.98:0/731995644 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c0116b0 con 0x7fe184106780 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.661+0000 7fe18a909640 1 -- 172.21.15.98:0/731995644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1841a3b10 con 0x7fe184106780 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.662+0000 7fe18a909640 1 -- 172.21.15.98:0/731995644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe184101e70 con 0x7fe184106780 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.663+0000 7fe181ffb640 1 -- 172.21.15.98:0/731995644 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fe17c005560 con 0x7fe184106780 2024-05-09T08:50:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.663+0000 7fe181ffb640 1 --2- 172.21.15.98:0/731995644 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe1700a2e20 0x7fe1700a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.664+0000 7fe183fff640 1 --2- 172.21.15.98:0/731995644 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe1700a2e20 0x7fe1700a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.664+0000 7fe181ffb640 1 -- 172.21.15.98:0/731995644 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fe17c0d0160 con 0x7fe184106780 2024-05-09T08:50:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.664+0000 7fe183fff640 1 --2- 172.21.15.98:0/731995644 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe1700a2e20 0x7fe1700a52e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe174005fd0 tx=0x7fe174005850 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.667+0000 7fe181ffb640 1 -- 172.21.15.98:0/731995644 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe17c016030 con 0x7fe184106780 2024-05-09T08:50:44.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.845+0000 7fe18a909640 1 -- 172.21.15.98:0/731995644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe1841982f0 con 0x7fe184106780 2024-05-09T08:50:44.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.845+0000 7fe181ffb640 1 -- 172.21.15.98:0/731995644 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7fe17c09c770 con 0x7fe184106780 2024-05-09T08:50:44.845 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-09T08:50:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.847+0000 7fe18a909640 1 -- 172.21.15.98:0/731995644 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe1700a2e20 msgr2=0x7fe1700a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.847+0000 7fe18a909640 1 --2- 172.21.15.98:0/731995644 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe1700a2e20 0x7fe1700a52e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe174005fd0 tx=0x7fe174005850 comp rx=0 tx=0).stop 2024-05-09T08:50:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.847+0000 7fe18a909640 1 -- 172.21.15.98:0/731995644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe184106780 msgr2=0x7fe18419b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.847+0000 7fe18a909640 1 --2- 172.21.15.98:0/731995644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe184106780 0x7fe18419b860 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00eaa0 tx=0x7fe17c00ef60 comp rx=0 tx=0).stop 2024-05-09T08:50:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.847+0000 7fe18a909640 1 -- 172.21.15.98:0/731995644 shutdown_connections 2024-05-09T08:50:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.847+0000 7fe18a909640 1 --2- 172.21.15.98:0/731995644 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe1700a2e20 0x7fe1700a52e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.847+0000 7fe18a909640 1 --2- 172.21.15.98:0/731995644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe184106780 0x7fe18419b860 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.847+0000 7fe18a909640 1 --2- 172.21.15.98:0/731995644 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe184101420 0x7fe184197530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.847+0000 7fe18a909640 1 --2- 172.21.15.98:0/731995644 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe184100a80 0x7fe184196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.847+0000 7fe18a909640 1 -- 172.21.15.98:0/731995644 >> 172.21.15.98:0/731995644 conn(0x7fe184076530 msgr2=0x7fe1840ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:44.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.847+0000 7fe18a909640 1 -- 172.21.15.98:0/731995644 shutdown_connections 2024-05-09T08:50:44.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.847+0000 7fe18a909640 1 -- 172.21.15.98:0/731995644 wait complete. 2024-05-09T08:50:44.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:50:44.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.965+0000 7fe635909640 1 -- 172.21.15.98:0/781640942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630100d80 msgr2=0x7fe6301064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.965+0000 7fe635909640 1 --2- 172.21.15.98:0/781640942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630100d80 0x7fe6301064b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe618009900 tx=0x7fe61802a410 comp rx=0 tx=0).stop 2024-05-09T08:50:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.965+0000 7fe635909640 1 -- 172.21.15.98:0/781640942 shutdown_connections 2024-05-09T08:50:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.965+0000 7fe635909640 1 --2- 172.21.15.98:0/781640942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe630106b90 0x7fe63010e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.965+0000 7fe635909640 1 --2- 172.21.15.98:0/781640942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630100d80 0x7fe6301064b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.965+0000 7fe635909640 1 --2- 172.21.15.98:0/781640942 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6301003d0 0x7fe6301007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.965+0000 7fe635909640 1 -- 172.21.15.98:0/781640942 >> 172.21.15.98:0/781640942 conn(0x7fe6300759c0 msgr2=0x7fe630075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.965+0000 7fe635909640 1 -- 172.21.15.98:0/781640942 shutdown_connections 2024-05-09T08:50:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.965+0000 7fe635909640 1 -- 172.21.15.98:0/781640942 wait complete. 2024-05-09T08:50:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.965+0000 7fe635909640 1 Processor -- start 2024-05-09T08:50:44.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe635909640 1 -- start start 2024-05-09T08:50:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe635909640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6301003d0 0x7fe630198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe635909640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe630106b90 0x7fe630198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe635909640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe63019ca00 0x7fe63019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe635909640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe630111c80 con 0x7fe63019ca00 2024-05-09T08:50:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe635909640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe630111b00 con 0x7fe6301003d0 2024-05-09T08:50:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe635909640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe630111e00 con 0x7fe630106b90 2024-05-09T08:50:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe62f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe63019ca00 0x7fe63019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe62f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe63019ca00 0x7fe63019ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43298/0 (socket says 172.21.15.98:43298) 2024-05-09T08:50:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe62f7fe640 1 -- 172.21.15.98:0/2142909717 learned_addr learned my addr 172.21.15.98:0/2142909717 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe62e7fc640 1 --2- 172.21.15.98:0/2142909717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe630106b90 0x7fe630198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe62effd640 1 --2- 172.21.15.98:0/2142909717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6301003d0 0x7fe630198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe62f7fe640 1 -- 172.21.15.98:0/2142909717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe630106b90 msgr2=0x7fe630198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe62f7fe640 1 --2- 172.21.15.98:0/2142909717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe630106b90 0x7fe630198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe62f7fe640 1 -- 172.21.15.98:0/2142909717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6301003d0 msgr2=0x7fe630198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe62f7fe640 1 --2- 172.21.15.98:0/2142909717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6301003d0 0x7fe630198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe62f7fe640 1 -- 172.21.15.98:0/2142909717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe63019d4f0 con 0x7fe63019ca00 2024-05-09T08:50:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe62f7fe640 1 --2- 172.21.15.98:0/2142909717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe63019ca00 0x7fe63019ceb0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe62800cce0 tx=0x7fe62800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe634907640 1 -- 172.21.15.98:0/2142909717 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62800f450 con 0x7fe63019ca00 2024-05-09T08:50:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe635909640 1 -- 172.21.15.98:0/2142909717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe63019d7e0 con 0x7fe63019ca00 2024-05-09T08:50:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe634907640 1 -- 172.21.15.98:0/2142909717 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe62800fad0 con 0x7fe63019ca00 2024-05-09T08:50:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe634907640 1 -- 172.21.15.98:0/2142909717 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe628018c10 con 0x7fe63019ca00 2024-05-09T08:50:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.966+0000 7fe635909640 1 -- 172.21.15.98:0/2142909717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6301a4f50 con 0x7fe63019ca00 2024-05-09T08:50:44.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.967+0000 7fe635909640 1 -- 172.21.15.98:0/2142909717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe630068aa0 con 0x7fe63019ca00 2024-05-09T08:50:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.968+0000 7fe634907640 1 -- 172.21.15.98:0/2142909717 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fe62800f5f0 con 0x7fe63019ca00 2024-05-09T08:50:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.969+0000 7fe634907640 1 --2- 172.21.15.98:0/2142909717 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe61c0a2e20 0x7fe61c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.969+0000 7fe62effd640 1 --2- 172.21.15.98:0/2142909717 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe61c0a2e20 0x7fe61c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.969+0000 7fe634907640 1 -- 172.21.15.98:0/2142909717 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fe6280d0140 con 0x7fe63019ca00 2024-05-09T08:50:44.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.970+0000 7fe62effd640 1 --2- 172.21.15.98:0/2142909717 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe61c0a2e20 0x7fe61c0a52e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe620005fd0 tx=0x7fe620005850 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:44.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:44.973+0000 7fe634907640 1 -- 172.21.15.98:0/2142909717 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe628014030 con 0x7fe63019ca00 2024-05-09T08:50:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.151+0000 7fe635909640 1 -- 172.21.15.98:0/2142909717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe630100d80 con 0x7fe63019ca00 2024-05-09T08:50:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.152+0000 7fe634907640 1 -- 172.21.15.98:0/2142909717 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7fe62809c750 con 0x7fe63019ca00 2024-05-09T08:50:45.151 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:50:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.154+0000 7fe635909640 1 -- 172.21.15.98:0/2142909717 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe61c0a2e20 msgr2=0x7fe61c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.154+0000 7fe635909640 1 --2- 172.21.15.98:0/2142909717 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe61c0a2e20 0x7fe61c0a52e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe620005fd0 tx=0x7fe620005850 comp rx=0 tx=0).stop 2024-05-09T08:50:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.154+0000 7fe635909640 1 -- 172.21.15.98:0/2142909717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe63019ca00 msgr2=0x7fe63019ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.154+0000 7fe635909640 1 --2- 172.21.15.98:0/2142909717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe63019ca00 0x7fe63019ceb0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe62800cce0 tx=0x7fe62800a620 comp rx=0 tx=0).stop 2024-05-09T08:50:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.154+0000 7fe635909640 1 -- 172.21.15.98:0/2142909717 shutdown_connections 2024-05-09T08:50:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.154+0000 7fe635909640 1 --2- 172.21.15.98:0/2142909717 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe61c0a2e20 0x7fe61c0a52e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.154+0000 7fe635909640 1 --2- 172.21.15.98:0/2142909717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe63019ca00 0x7fe63019ceb0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.154+0000 7fe635909640 1 --2- 172.21.15.98:0/2142909717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe630106b90 0x7fe630198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.154+0000 7fe635909640 1 --2- 172.21.15.98:0/2142909717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6301003d0 0x7fe630198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.154+0000 7fe635909640 1 -- 172.21.15.98:0/2142909717 >> 172.21.15.98:0/2142909717 conn(0x7fe6300759c0 msgr2=0x7fe6300fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.154+0000 7fe635909640 1 -- 172.21.15.98:0/2142909717 shutdown_connections 2024-05-09T08:50:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.154+0000 7fe635909640 1 -- 172.21.15.98:0/2142909717 wait complete. 2024-05-09T08:50:45.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:50:45.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.272+0000 7fbe209b0640 1 -- 172.21.15.98:0/3288302044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c100700 msgr2=0x7fbe1c100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.272+0000 7fbe209b0640 1 --2- 172.21.15.98:0/3288302044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c100700 0x7fbe1c100b80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fbe04009930 tx=0x7fbe0402a420 comp rx=0 tx=0).stop 2024-05-09T08:50:45.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.272+0000 7fbe209b0640 1 -- 172.21.15.98:0/3288302044 shutdown_connections 2024-05-09T08:50:45.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.272+0000 7fbe209b0640 1 --2- 172.21.15.98:0/3288302044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe1c1010c0 0x7fbe1c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.272+0000 7fbe209b0640 1 --2- 172.21.15.98:0/3288302044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c100700 0x7fbe1c100b80 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.272+0000 7fbe209b0640 1 --2- 172.21.15.98:0/3288302044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe1c106710 0x7fbe1c106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.272+0000 7fbe209b0640 1 -- 172.21.15.98:0/3288302044 >> 172.21.15.98:0/3288302044 conn(0x7fbe1c077830 msgr2=0x7fbe1c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:45.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.272+0000 7fbe209b0640 1 -- 172.21.15.98:0/3288302044 shutdown_connections 2024-05-09T08:50:45.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.272+0000 7fbe209b0640 1 -- 172.21.15.98:0/3288302044 wait complete. 2024-05-09T08:50:45.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.273+0000 7fbe209b0640 1 Processor -- start 2024-05-09T08:50:45.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.273+0000 7fbe209b0640 1 -- start start 2024-05-09T08:50:45.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.273+0000 7fbe209b0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe1c100700 0x7fbe1c102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:45.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.273+0000 7fbe209b0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe1c1010c0 0x7fbe1c103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:45.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.273+0000 7fbe209b0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c106710 0x7fbe1c103740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:45.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.273+0000 7fbe209b0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1c110b70 con 0x7fbe1c106710 2024-05-09T08:50:45.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.273+0000 7fbe209b0640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1c1109f0 con 0x7fbe1c100700 2024-05-09T08:50:45.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.273+0000 7fbe209b0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1c110cf0 con 0x7fbe1c1010c0 2024-05-09T08:50:45.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.273+0000 7fbe1ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c106710 0x7fbe1c103740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:45.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.273+0000 7fbe1ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c106710 0x7fbe1c103740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43312/0 (socket says 172.21.15.98:43312) 2024-05-09T08:50:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.273+0000 7fbe1ad9d640 1 -- 172.21.15.98:0/2404210567 learned_addr learned my addr 172.21.15.98:0/2404210567 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.273+0000 7fbe1a59c640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe1c100700 0x7fbe1c102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.273+0000 7fbe11d9b640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe1c1010c0 0x7fbe1c103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.274+0000 7fbe1ad9d640 1 -- 172.21.15.98:0/2404210567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe1c1010c0 msgr2=0x7fbe1c103200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.274+0000 7fbe1ad9d640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe1c1010c0 0x7fbe1c103200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.274+0000 7fbe1ad9d640 1 -- 172.21.15.98:0/2404210567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe1c100700 msgr2=0x7fbe1c102cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.274+0000 7fbe1ad9d640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe1c100700 0x7fbe1c102cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.274+0000 7fbe1ad9d640 1 -- 172.21.15.98:0/2404210567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe1c1a6680 con 0x7fbe1c106710 2024-05-09T08:50:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.274+0000 7fbe1a59c640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe1c100700 0x7fbe1c102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:50:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.274+0000 7fbe1ad9d640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c106710 0x7fbe1c103740 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400efd0 tx=0x7fbe1400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.275+0000 7fbe13fff640 1 -- 172.21.15.98:0/2404210567 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe14003b70 con 0x7fbe1c106710 2024-05-09T08:50:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.275+0000 7fbe13fff640 1 -- 172.21.15.98:0/2404210567 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe14003d10 con 0x7fbe1c106710 2024-05-09T08:50:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.275+0000 7fbe209b0640 1 -- 172.21.15.98:0/2404210567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe1c1a6970 con 0x7fbe1c106710 2024-05-09T08:50:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.275+0000 7fbe13fff640 1 -- 172.21.15.98:0/2404210567 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe140109f0 con 0x7fbe1c106710 2024-05-09T08:50:45.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.275+0000 7fbe209b0640 1 -- 172.21.15.98:0/2404210567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe1c1a6e90 con 0x7fbe1c106710 2024-05-09T08:50:45.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.276+0000 7fbe209b0640 1 -- 172.21.15.98:0/2404210567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe1c068650 con 0x7fbe1c106710 2024-05-09T08:50:45.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.277+0000 7fbe13fff640 1 -- 172.21.15.98:0/2404210567 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fbe14004020 con 0x7fbe1c106710 2024-05-09T08:50:45.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.278+0000 7fbe13fff640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbe080a2ea0 0x7fbe080a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:45.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.278+0000 7fbe1a59c640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbe080a2ea0 0x7fbe080a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:45.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.278+0000 7fbe13fff640 1 -- 172.21.15.98:0/2404210567 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fbe140cfe50 con 0x7fbe1c106710 2024-05-09T08:50:45.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.279+0000 7fbe1a59c640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbe080a2ea0 0x7fbe080a5360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009f40 tx=0x7fbe0c005f90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:45.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.282+0000 7fbe13fff640 1 -- 172.21.15.98:0/2404210567 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbe1409c460 con 0x7fbe1c106710 2024-05-09T08:50:45.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.458+0000 7fbe209b0640 1 -- 172.21.15.98:0/2404210567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbe1c02d1c0 con 0x7fbe1c106710 2024-05-09T08:50:45.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.458+0000 7fbe13fff640 1 -- 172.21.15.98:0/2404210567 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7fbe1409bd60 con 0x7fbe1c106710 2024-05-09T08:50:45.458 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-09T08:50:45.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.460+0000 7fbe209b0640 1 -- 172.21.15.98:0/2404210567 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbe080a2ea0 msgr2=0x7fbe080a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.460+0000 7fbe209b0640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbe080a2ea0 0x7fbe080a5360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009f40 tx=0x7fbe0c005f90 comp rx=0 tx=0).stop 2024-05-09T08:50:45.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.460+0000 7fbe209b0640 1 -- 172.21.15.98:0/2404210567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c106710 msgr2=0x7fbe1c103740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.460+0000 7fbe209b0640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c106710 0x7fbe1c103740 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400efd0 tx=0x7fbe1400c6a0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.460+0000 7fbe209b0640 1 -- 172.21.15.98:0/2404210567 shutdown_connections 2024-05-09T08:50:45.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.460+0000 7fbe209b0640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbe080a2ea0 0x7fbe080a5360 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.460+0000 7fbe209b0640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c106710 0x7fbe1c103740 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.460+0000 7fbe209b0640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe1c1010c0 0x7fbe1c103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.460+0000 7fbe209b0640 1 --2- 172.21.15.98:0/2404210567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe1c100700 0x7fbe1c102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.461+0000 7fbe209b0640 1 -- 172.21.15.98:0/2404210567 >> 172.21.15.98:0/2404210567 conn(0x7fbe1c077830 msgr2=0x7fbe1c068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:45.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.461+0000 7fbe209b0640 1 -- 172.21.15.98:0/2404210567 shutdown_connections 2024-05-09T08:50:45.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.461+0000 7fbe209b0640 1 -- 172.21.15.98:0/2404210567 wait complete. 2024-05-09T08:50:45.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:45 smithi098 ceph-mon[30092]: from='client.15705 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:50:45.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:45 smithi098 ceph-mon[30092]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:50:45.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:45 smithi098 ceph-mon[30092]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:50:45.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:45 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3030512260' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:45.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:45 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/731995644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:45.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:45 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2142909717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:45.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:50:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.577+0000 7f1797ea5640 1 -- 172.21.15.98:0/740212368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17901021c0 msgr2=0x7f179010e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.577+0000 7f1797ea5640 1 --2- 172.21.15.98:0/740212368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17901021c0 0x7f179010e6d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1788009cd0 tx=0x7f178802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.577+0000 7f1797ea5640 1 -- 172.21.15.98:0/740212368 shutdown_connections 2024-05-09T08:50:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.577+0000 7f1797ea5640 1 --2- 172.21.15.98:0/740212368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17901021c0 0x7f179010e6d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.577+0000 7f1797ea5640 1 --2- 172.21.15.98:0/740212368 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1790101800 0x7f1790101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.577+0000 7f1797ea5640 1 --2- 172.21.15.98:0/740212368 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1790107810 0x7f1790107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.577+0000 7f1797ea5640 1 -- 172.21.15.98:0/740212368 >> 172.21.15.98:0/740212368 conn(0x7f1790078c80 msgr2=0x7f1790079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.577+0000 7f1797ea5640 1 -- 172.21.15.98:0/740212368 shutdown_connections 2024-05-09T08:50:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.577+0000 7f1797ea5640 1 -- 172.21.15.98:0/740212368 wait complete. 2024-05-09T08:50:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.578+0000 7f1797ea5640 1 Processor -- start 2024-05-09T08:50:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.578+0000 7f1797ea5640 1 -- start start 2024-05-09T08:50:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.578+0000 7f1797ea5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1790101800 0x7f1790198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.578+0000 7f1797ea5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17901021c0 0x7f1790198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.578+0000 7f1797ea5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1790107810 0x7f179019c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.578+0000 7f1797ea5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1790111c20 con 0x7f1790107810 2024-05-09T08:50:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.578+0000 7f1797ea5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1790111aa0 con 0x7f17901021c0 2024-05-09T08:50:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.578+0000 7f1797ea5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1790111da0 con 0x7f1790101800 2024-05-09T08:50:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.578+0000 7f1796442640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1790107810 0x7f179019c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.578+0000 7f1796442640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1790107810 0x7f179019c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56998/0 (socket says 172.21.15.98:56998) 2024-05-09T08:50:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.578+0000 7f1796442640 1 -- 172.21.15.98:0/80234058 learned_addr learned my addr 172.21.15.98:0/80234058 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.578+0000 7f1795440640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17901021c0 0x7f1790198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.578+0000 7f1795c41640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1790101800 0x7f1790198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.579+0000 7f1796442640 1 -- 172.21.15.98:0/80234058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1790101800 msgr2=0x7f1790198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.579+0000 7f1796442640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1790101800 0x7f1790198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.579+0000 7f1796442640 1 -- 172.21.15.98:0/80234058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17901021c0 msgr2=0x7f1790198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.579+0000 7f1796442640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17901021c0 0x7f1790198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.579+0000 7f1796442640 1 -- 172.21.15.98:0/80234058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179019d0e0 con 0x7f1790107810 2024-05-09T08:50:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.579+0000 7f1795c41640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1790101800 0x7f1790198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.579+0000 7f1796442640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1790107810 0x7f179019c9b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1788005cc0 tx=0x7f1788003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.579+0000 7f1786ffd640 1 -- 172.21.15.98:0/80234058 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788004040 con 0x7f1790107810 2024-05-09T08:50:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.579+0000 7f1797ea5640 1 -- 172.21.15.98:0/80234058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f179019d370 con 0x7f1790107810 2024-05-09T08:50:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.579+0000 7f1786ffd640 1 -- 172.21.15.98:0/80234058 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17880046c0 con 0x7f1790107810 2024-05-09T08:50:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.579+0000 7f1786ffd640 1 -- 172.21.15.98:0/80234058 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788046480 con 0x7f1790107810 2024-05-09T08:50:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.579+0000 7f1797ea5640 1 -- 172.21.15.98:0/80234058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17901a4c60 con 0x7f1790107810 2024-05-09T08:50:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.580+0000 7f1797ea5640 1 -- 172.21.15.98:0/80234058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1790068aa0 con 0x7f1790107810 2024-05-09T08:50:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.581+0000 7f1786ffd640 1 -- 172.21.15.98:0/80234058 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f17880305a0 con 0x7f1790107810 2024-05-09T08:50:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.582+0000 7f1786ffd640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f177c0a2e20 0x7f177c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.582+0000 7f1795c41640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f177c0a2e20 0x7f177c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.582+0000 7f1786ffd640 1 -- 172.21.15.98:0/80234058 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f178802e030 con 0x7f1790107810 2024-05-09T08:50:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.582+0000 7f1795c41640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f177c0a2e20 0x7f177c0a52e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f178c005fd0 tx=0x7f178c005850 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:45.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.586+0000 7f1786ffd640 1 -- 172.21.15.98:0/80234058 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f178804b060 con 0x7f1790107810 2024-05-09T08:50:45.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:45 smithi142 ceph-mon[34625]: from='client.15705 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:50:45.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:45 smithi142 ceph-mon[34625]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:50:45.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:45 smithi142 ceph-mon[34625]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:50:45.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:45 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3030512260' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:45.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:45 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/731995644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:45.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:45 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2142909717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:45.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:45 smithi105 ceph-mon[34498]: from='client.15705 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:50:45.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:45 smithi105 ceph-mon[34498]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:50:45.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:45 smithi105 ceph-mon[34498]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:50:45.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:45 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3030512260' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:45.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:45 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/731995644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:50:45.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:45 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2142909717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:50:45.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.763+0000 7f1797ea5640 1 -- 172.21.15.98:0/80234058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f179002d1c0 con 0x7f1790107810 2024-05-09T08:50:45.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.763+0000 7f1786ffd640 1 -- 172.21.15.98:0/80234058 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f17880bd6c0 con 0x7f1790107810 2024-05-09T08:50:45.762 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T08:50:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.765+0000 7f1797ea5640 1 -- 172.21.15.98:0/80234058 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f177c0a2e20 msgr2=0x7f177c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.765+0000 7f1797ea5640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f177c0a2e20 0x7f177c0a52e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f178c005fd0 tx=0x7f178c005850 comp rx=0 tx=0).stop 2024-05-09T08:50:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.765+0000 7f1797ea5640 1 -- 172.21.15.98:0/80234058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1790107810 msgr2=0x7f179019c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.765+0000 7f1797ea5640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1790107810 0x7f179019c9b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1788005cc0 tx=0x7f1788003970 comp rx=0 tx=0).stop 2024-05-09T08:50:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.765+0000 7f1797ea5640 1 -- 172.21.15.98:0/80234058 shutdown_connections 2024-05-09T08:50:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.765+0000 7f1797ea5640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f177c0a2e20 0x7f177c0a52e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.765+0000 7f1797ea5640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1790107810 0x7f179019c9b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.765+0000 7f1797ea5640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17901021c0 0x7f1790198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.765+0000 7f1797ea5640 1 --2- 172.21.15.98:0/80234058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1790101800 0x7f1790198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.765+0000 7f1797ea5640 1 -- 172.21.15.98:0/80234058 >> 172.21.15.98:0/80234058 conn(0x7f1790078c80 msgr2=0x7f17900ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:45.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.765+0000 7f1797ea5640 1 -- 172.21.15.98:0/80234058 shutdown_connections 2024-05-09T08:50:45.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.765+0000 7f1797ea5640 1 -- 172.21.15.98:0/80234058 wait complete. 2024-05-09T08:50:45.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:50:45.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.878+0000 7f0c3afce640 1 -- 172.21.15.98:0/1639822481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c341003f0 msgr2=0x7f0c341007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.878+0000 7f0c3afce640 1 --2- 172.21.15.98:0/1639822481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c341003f0 0x7f0c341007d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009900 tx=0x7f0c2802a410 comp rx=0 tx=0).stop 2024-05-09T08:50:45.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.878+0000 7f0c3afce640 1 -- 172.21.15.98:0/1639822481 shutdown_connections 2024-05-09T08:50:45.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.878+0000 7f0c3afce640 1 --2- 172.21.15.98:0/1639822481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c34106bb0 0x7f0c3410e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.878+0000 7f0c3afce640 1 --2- 172.21.15.98:0/1639822481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c34100da0 0x7f0c341064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.878+0000 7f0c3afce640 1 --2- 172.21.15.98:0/1639822481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c341003f0 0x7f0c341007d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.878+0000 7f0c3afce640 1 -- 172.21.15.98:0/1639822481 >> 172.21.15.98:0/1639822481 conn(0x7f0c34075a10 msgr2=0x7f0c34075e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:45.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.878+0000 7f0c3afce640 1 -- 172.21.15.98:0/1639822481 shutdown_connections 2024-05-09T08:50:45.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.878+0000 7f0c3afce640 1 -- 172.21.15.98:0/1639822481 wait complete. 2024-05-09T08:50:45.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c3afce640 1 Processor -- start 2024-05-09T08:50:45.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c3afce640 1 -- start start 2024-05-09T08:50:45.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c3afce640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c341003f0 0x7f0c341981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:45.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c3afce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c34106bb0 0x7f0c34198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:45.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c3afce640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c3419ca80 0x7f0c3419cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:45.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c3afce640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c34111ce0 con 0x7f0c34106bb0 2024-05-09T08:50:45.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c3afce640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0c34111b60 con 0x7f0c341003f0 2024-05-09T08:50:45.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c3afce640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0c34111e60 con 0x7f0c3419ca80 2024-05-09T08:50:45.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c34106bb0 0x7f0c34198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:45.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c34106bb0 0x7f0c34198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57014/0 (socket says 172.21.15.98:57014) 2024-05-09T08:50:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c33fff640 1 -- 172.21.15.98:0/3264581540 learned_addr learned my addr 172.21.15.98:0/3264581540 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c38d6a640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c341003f0 0x7f0c341981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c3956b640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c3419ca80 0x7f0c3419cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c33fff640 1 -- 172.21.15.98:0/3264581540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c3419ca80 msgr2=0x7f0c3419cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c33fff640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c3419ca80 0x7f0c3419cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c33fff640 1 -- 172.21.15.98:0/3264581540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c341003f0 msgr2=0x7f0c341981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c33fff640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c341003f0 0x7f0c341981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c33fff640 1 -- 172.21.15.98:0/3264581540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c3419d570 con 0x7f0c34106bb0 2024-05-09T08:50:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c38d6a640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c341003f0 0x7f0c341981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.879+0000 7f0c33fff640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c34106bb0 0x7f0c34198700 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000b450 tx=0x7f0c2000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.880+0000 7f0c31ffb640 1 -- 172.21.15.98:0/3264581540 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c20015020 con 0x7f0c34106bb0 2024-05-09T08:50:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.880+0000 7f0c31ffb640 1 -- 172.21.15.98:0/3264581540 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c20011d70 con 0x7f0c34106bb0 2024-05-09T08:50:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.880+0000 7f0c3afce640 1 -- 172.21.15.98:0/3264581540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c3419d860 con 0x7f0c34106bb0 2024-05-09T08:50:45.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.880+0000 7f0c31ffb640 1 -- 172.21.15.98:0/3264581540 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2000f340 con 0x7f0c34106bb0 2024-05-09T08:50:45.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.880+0000 7f0c3afce640 1 -- 172.21.15.98:0/3264581540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c341a5000 con 0x7f0c34106bb0 2024-05-09T08:50:45.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.881+0000 7f0c3afce640 1 -- 172.21.15.98:0/3264581540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c34068aa0 con 0x7f0c34106bb0 2024-05-09T08:50:45.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.882+0000 7f0c31ffb640 1 -- 172.21.15.98:0/3264581540 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f0c20018020 con 0x7f0c34106bb0 2024-05-09T08:50:45.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.882+0000 7f0c31ffb640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0c240a2e20 0x7f0c240a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:45.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.883+0000 7f0c38d6a640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0c240a2e20 0x7f0c240a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:45.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.883+0000 7f0c31ffb640 1 -- 172.21.15.98:0/3264581540 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f0c200cf4a0 con 0x7f0c34106bb0 2024-05-09T08:50:45.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.883+0000 7f0c38d6a640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0c240a2e20 0x7f0c240a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0c28005ab0 tx=0x7f0c28005a20 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:45.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:45.887+0000 7f0c31ffb640 1 -- 172.21.15.98:0/3264581540 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0c2001f020 con 0x7f0c34106bb0 2024-05-09T08:50:46.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.101+0000 7f0c3afce640 1 -- 172.21.15.98:0/3264581540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c34109dc0 con 0x7f0c34106bb0 2024-05-09T08:50:46.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.101+0000 7f0c31ffb640 1 -- 172.21.15.98:0/3264581540 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11940 (secure 0 0 0) 0x7f0c2000d040 con 0x7f0c34106bb0 2024-05-09T08:50:46.101 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:46.101 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:50:37.185367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T08:50:46.101 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-05-09T08:50:46.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.103+0000 7f0c3afce640 1 -- 172.21.15.98:0/3264581540 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0c240a2e20 msgr2=0x7f0c240a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.103+0000 7f0c3afce640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0c240a2e20 0x7f0c240a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0c28005ab0 tx=0x7f0c28005a20 comp rx=0 tx=0).stop 2024-05-09T08:50:46.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.103+0000 7f0c3afce640 1 -- 172.21.15.98:0/3264581540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c34106bb0 msgr2=0x7f0c34198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.103+0000 7f0c3afce640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c34106bb0 0x7f0c34198700 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000b450 tx=0x7f0c2000b910 comp rx=0 tx=0).stop 2024-05-09T08:50:46.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.103+0000 7f0c3afce640 1 -- 172.21.15.98:0/3264581540 shutdown_connections 2024-05-09T08:50:46.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.103+0000 7f0c3afce640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0c240a2e20 0x7f0c240a52e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.103+0000 7f0c3afce640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c3419ca80 0x7f0c3419cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.103+0000 7f0c3afce640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c34106bb0 0x7f0c34198700 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.103+0000 7f0c3afce640 1 --2- 172.21.15.98:0/3264581540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c341003f0 0x7f0c341981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.104+0000 7f0c3afce640 1 -- 172.21.15.98:0/3264581540 >> 172.21.15.98:0/3264581540 conn(0x7f0c34075a10 msgr2=0x7f0c340fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.104+0000 7f0c3afce640 1 -- 172.21.15.98:0/3264581540 shutdown_connections 2024-05-09T08:50:46.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.104+0000 7f0c3afce640 1 -- 172.21.15.98:0/3264581540 wait complete. 2024-05-09T08:50:46.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:50:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.216+0000 7f4389d77640 1 -- 172.21.15.98:0/2453676547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384101140 msgr2=0x7f438410d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.216+0000 7f4389d77640 1 --2- 172.21.15.98:0/2453676547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384101140 0x7f438410d600 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f437c009cd0 tx=0x7f437c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.216+0000 7f4389d77640 1 -- 172.21.15.98:0/2453676547 shutdown_connections 2024-05-09T08:50:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.216+0000 7f4389d77640 1 --2- 172.21.15.98:0/2453676547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384101140 0x7f438410d600 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.216+0000 7f4389d77640 1 --2- 172.21.15.98:0/2453676547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4384100780 0x7f4384100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.216+0000 7f4389d77640 1 --2- 172.21.15.98:0/2453676547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4384106740 0x7f4384106b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.216+0000 7f4389d77640 1 -- 172.21.15.98:0/2453676547 >> 172.21.15.98:0/2453676547 conn(0x7f4384077890 msgr2=0x7f4384077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.217+0000 7f4389d77640 1 -- 172.21.15.98:0/2453676547 shutdown_connections 2024-05-09T08:50:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.217+0000 7f4389d77640 1 -- 172.21.15.98:0/2453676547 wait complete. 2024-05-09T08:50:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.217+0000 7f4389d77640 1 Processor -- start 2024-05-09T08:50:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.217+0000 7f4389d77640 1 -- start start 2024-05-09T08:50:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.217+0000 7f4389d77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384100780 0x7f4384197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.217+0000 7f4389d77640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4384101140 0x7f43841975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.217+0000 7f4389d77640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4384106740 0x7f438419b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.217+0000 7f4389d77640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4384110b50 con 0x7f4384100780 2024-05-09T08:50:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.217+0000 7f4389d77640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f43841109d0 con 0x7f4384106740 2024-05-09T08:50:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.217+0000 7f4389d77640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4384110cd0 con 0x7f4384101140 2024-05-09T08:50:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.217+0000 7f43837fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384100780 0x7f4384197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.217+0000 7f43837fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384100780 0x7f4384197070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57030/0 (socket says 172.21.15.98:57030) 2024-05-09T08:50:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.217+0000 7f43837fe640 1 -- 172.21.15.98:0/697287211 learned_addr learned my addr 172.21.15.98:0/697287211 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.218+0000 7f4383fff640 1 --2- 172.21.15.98:0/697287211 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4384106740 0x7f438419b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.218+0000 7f43837fe640 1 -- 172.21.15.98:0/697287211 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4384101140 msgr2=0x7f43841975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.218+0000 7f43837fe640 1 --2- 172.21.15.98:0/697287211 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4384101140 0x7f43841975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.218+0000 7f43837fe640 1 -- 172.21.15.98:0/697287211 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4384106740 msgr2=0x7f438419b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.218+0000 7f43837fe640 1 --2- 172.21.15.98:0/697287211 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4384106740 0x7f438419b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.218+0000 7f43837fe640 1 -- 172.21.15.98:0/697287211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f438419c060 con 0x7f4384100780 2024-05-09T08:50:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.218+0000 7f4383fff640 1 --2- 172.21.15.98:0/697287211 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4384106740 0x7f438419b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.218+0000 7f43837fe640 1 --2- 172.21.15.98:0/697287211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384100780 0x7f4384197070 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f437400b4a0 tx=0x7f437400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.218+0000 7f43817fa640 1 -- 172.21.15.98:0/697287211 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374014070 con 0x7f4384100780 2024-05-09T08:50:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.218+0000 7f43817fa640 1 -- 172.21.15.98:0/697287211 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4374010af0 con 0x7f4384100780 2024-05-09T08:50:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.218+0000 7f4389d77640 1 -- 172.21.15.98:0/697287211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f438419c350 con 0x7f4384100780 2024-05-09T08:50:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.218+0000 7f43817fa640 1 -- 172.21.15.98:0/697287211 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374017d10 con 0x7f4384100780 2024-05-09T08:50:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.218+0000 7f4389d77640 1 -- 172.21.15.98:0/697287211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43841a3be0 con 0x7f4384100780 2024-05-09T08:50:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.219+0000 7f4389d77640 1 -- 172.21.15.98:0/697287211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4384068aa0 con 0x7f4384100780 2024-05-09T08:50:46.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.220+0000 7f43817fa640 1 -- 172.21.15.98:0/697287211 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f4374017070 con 0x7f4384100780 2024-05-09T08:50:46.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.221+0000 7f43817fa640 1 --2- 172.21.15.98:0/697287211 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f43700a2e20 0x7f43700a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.221+0000 7f437bfff640 1 --2- 172.21.15.98:0/697287211 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f43700a2e20 0x7f43700a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.221+0000 7f43817fa640 1 -- 172.21.15.98:0/697287211 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f43740cf1c0 con 0x7f4384100780 2024-05-09T08:50:46.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.221+0000 7f437bfff640 1 --2- 172.21.15.98:0/697287211 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f43700a2e20 0x7f43700a52e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4384077e20 tx=0x7f436c005c70 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.224+0000 7f43817fa640 1 -- 172.21.15.98:0/697287211 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f437409b8e0 con 0x7f4384100780 2024-05-09T08:50:46.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.406+0000 7f4389d77640 1 -- 172.21.15.98:0/697287211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f43841983c0 con 0x7f4384100780 2024-05-09T08:50:46.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.408+0000 7f43817fa640 1 -- 172.21.15.98:0/697287211 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25206 (secure 0 0 0) 0x7f43740b9e20 con 0x7f4384100780 2024-05-09T08:50:46.407 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:50:46.407 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":95,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T08:50:38.492526+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":1}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1013153207":"2024-05-10T08:50:38.492493+0000","172.21.15.98:6800/1228297308":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/808614683":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/1734294041":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/2113007677":"2024-05-10T08:50:38.492493+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1228297308":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:50:46.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.410+0000 7f4389d77640 1 -- 172.21.15.98:0/697287211 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f43700a2e20 msgr2=0x7f43700a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.410+0000 7f4389d77640 1 --2- 172.21.15.98:0/697287211 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f43700a2e20 0x7f43700a52e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4384077e20 tx=0x7f436c005c70 comp rx=0 tx=0).stop 2024-05-09T08:50:46.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.410+0000 7f4389d77640 1 -- 172.21.15.98:0/697287211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384100780 msgr2=0x7f4384197070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.410+0000 7f4389d77640 1 --2- 172.21.15.98:0/697287211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384100780 0x7f4384197070 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f437400b4a0 tx=0x7f437400b960 comp rx=0 tx=0).stop 2024-05-09T08:50:46.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.410+0000 7f4389d77640 1 -- 172.21.15.98:0/697287211 shutdown_connections 2024-05-09T08:50:46.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.410+0000 7f4389d77640 1 --2- 172.21.15.98:0/697287211 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f43700a2e20 0x7f43700a52e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.410+0000 7f4389d77640 1 --2- 172.21.15.98:0/697287211 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4384106740 0x7f438419b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.410+0000 7f4389d77640 1 --2- 172.21.15.98:0/697287211 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4384101140 0x7f43841975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.410+0000 7f4389d77640 1 --2- 172.21.15.98:0/697287211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384100780 0x7f4384197070 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.410+0000 7f4389d77640 1 -- 172.21.15.98:0/697287211 >> 172.21.15.98:0/697287211 conn(0x7f4384077890 msgr2=0x7f43841032c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.410+0000 7f4389d77640 1 -- 172.21.15.98:0/697287211 shutdown_connections 2024-05-09T08:50:46.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.410+0000 7f4389d77640 1 -- 172.21.15.98:0/697287211 wait complete. 2024-05-09T08:50:46.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:46 smithi098 ceph-mon[30092]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-09T08:50:46.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:46 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2404210567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:46.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:46 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/80234058' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:46.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:46 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3264581540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:46.421 DEBUG:tasks.fwd_scrub:Yielding 2024-05-09T08:50:46.421 INFO:teuthology.run_tasks:Running task exec... 2024-05-09T08:50:46.430 INFO:teuthology.task.exec:Executing custom commands... 2024-05-09T08:50:46.434 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-05-09T08:50:46.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-05-09T08:50:46.435 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-05-09T08:50:46.435 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:50:46.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:50:46.436 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:50:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.541+0000 7f6192dd5640 1 -- 172.21.15.98:0/3181906100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f618c10be00 msgr2=0x7f618c10e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.541+0000 7f6192dd5640 1 --2- 172.21.15.98:0/3181906100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f618c10be00 0x7f618c10e2a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f6184009ca0 tx=0x7f618402a9e0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.541+0000 7f6192dd5640 1 -- 172.21.15.98:0/3181906100 shutdown_connections 2024-05-09T08:50:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.541+0000 7f6192dd5640 1 --2- 172.21.15.98:0/3181906100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f618c10be00 0x7f618c10e2a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.541+0000 7f6192dd5640 1 --2- 172.21.15.98:0/3181906100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f618c1033e0 0x7f618c10b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.541+0000 7f6192dd5640 1 --2- 172.21.15.98:0/3181906100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f618c102ac0 0x7f618c102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.541+0000 7f6192dd5640 1 -- 172.21.15.98:0/3181906100 >> 172.21.15.98:0/3181906100 conn(0x7f618c0778b0 msgr2=0x7f618c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.541+0000 7f6192dd5640 1 -- 172.21.15.98:0/3181906100 shutdown_connections 2024-05-09T08:50:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.541+0000 7f6192dd5640 1 -- 172.21.15.98:0/3181906100 wait complete. 2024-05-09T08:50:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6192dd5640 1 Processor -- start 2024-05-09T08:50:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6192dd5640 1 -- start start 2024-05-09T08:50:46.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f13b50e7640 1 -- 172.21.15.98:0/2648071646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0102ab0 msgr2=0x7f13b0102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f13b50e7640 1 --2- 172.21.15.98:0/2648071646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0102ab0 0x7f13b0102e90 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009900 tx=0x7f13a002a510 comp rx=0 tx=0).stop 2024-05-09T08:50:46.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f13b50e7640 1 -- 172.21.15.98:0/2648071646 shutdown_connections 2024-05-09T08:50:46.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f13b50e7640 1 --2- 172.21.15.98:0/2648071646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13b010bdf0 0x7f13b010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f13b50e7640 1 --2- 172.21.15.98:0/2648071646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13b01033d0 0x7f13b010b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f13b50e7640 1 --2- 172.21.15.98:0/2648071646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0102ab0 0x7f13b0102e90 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f13b50e7640 1 -- 172.21.15.98:0/2648071646 >> 172.21.15.98:0/2648071646 conn(0x7f13b0077830 msgr2=0x7f13b0077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f13b50e7640 1 -- 172.21.15.98:0/2648071646 shutdown_connections 2024-05-09T08:50:46.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f13b50e7640 1 -- 172.21.15.98:0/2648071646 wait complete. 2024-05-09T08:50:46.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6192dd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f618c102ac0 0x7f618c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6192dd5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f618c1033e0 0x7f618c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6192dd5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f618c10be00 0x7f618c19c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6192dd5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f618c111c00 con 0x7f618c102ac0 2024-05-09T08:50:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6192dd5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f618c111a80 con 0x7f618c10be00 2024-05-09T08:50:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6192dd5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f618c111d80 con 0x7f618c1033e0 2024-05-09T08:50:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6190b71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f618c102ac0 0x7f618c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6190b71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f618c102ac0 0x7f618c198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57056/0 (socket says 172.21.15.98:57056) 2024-05-09T08:50:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6190b71640 1 -- 172.21.15.98:0/3434606053 learned_addr learned my addr 172.21.15.98:0/3434606053 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6191372640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f618c10be00 0x7f618c19c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f618bfff640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f618c1033e0 0x7f618c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6190b71640 1 -- 172.21.15.98:0/3434606053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f618c1033e0 msgr2=0x7f618c198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6190b71640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f618c1033e0 0x7f618c198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6190b71640 1 -- 172.21.15.98:0/3434606053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f618c10be00 msgr2=0x7f618c19c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6190b71640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f618c10be00 0x7f618c19c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6190b71640 1 -- 172.21.15.98:0/3434606053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618c19d030 con 0x7f618c102ac0 2024-05-09T08:50:46.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.542+0000 7f6191372640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f618c10be00 0x7f618c19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:46.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f6190b71640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f618c102ac0 0x7f618c198040 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f617c00d950 tx=0x7f617c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f13b50e7640 1 Processor -- start 2024-05-09T08:50:46.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f6189ffb640 1 -- 172.21.15.98:0/3434606053 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c017030 con 0x7f618c102ac0 2024-05-09T08:50:46.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f6192dd5640 1 -- 172.21.15.98:0/3434606053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618c19d320 con 0x7f618c102ac0 2024-05-09T08:50:46.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f6189ffb640 1 -- 172.21.15.98:0/3434606053 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f617c004020 con 0x7f618c102ac0 2024-05-09T08:50:46.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f6189ffb640 1 -- 172.21.15.98:0/3434606053 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c0047a0 con 0x7f618c102ac0 2024-05-09T08:50:46.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f13b50e7640 1 -- start start 2024-05-09T08:50:46.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f6192dd5640 1 -- 172.21.15.98:0/3434606053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618c100640 con 0x7f618c102ac0 2024-05-09T08:50:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f13b50e7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0102ab0 0x7f13b0100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f13b50e7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13b01033d0 0x7f13b0100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f13b50e7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13b010bdf0 0x7f13b0101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f13b50e7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13b0111c40 con 0x7f13b0102ab0 2024-05-09T08:50:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f13b50e7640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f13b0111ac0 con 0x7f13b010bdf0 2024-05-09T08:50:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f13b50e7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f13b0111dc0 con 0x7f13b01033d0 2024-05-09T08:50:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f13aed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0102ab0 0x7f13b0100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f13aed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0102ab0 0x7f13b0100700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57068/0 (socket says 172.21.15.98:57068) 2024-05-09T08:50:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.543+0000 7f13aed9d640 1 -- 172.21.15.98:0/2585780992 learned_addr learned my addr 172.21.15.98:0/2585780992 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.544+0000 7f13ae59c640 1 --2- 172.21.15.98:0/2585780992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13b01033d0 0x7f13b0100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.544+0000 7f13af59e640 1 --2- 172.21.15.98:0/2585780992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13b010bdf0 0x7f13b0101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.544+0000 7f13aed9d640 1 -- 172.21.15.98:0/2585780992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13b01033d0 msgr2=0x7f13b0100c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.544+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2585780992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13b01033d0 0x7f13b0100c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.544+0000 7f13aed9d640 1 -- 172.21.15.98:0/2585780992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13b010bdf0 msgr2=0x7f13b0101180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.544+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2585780992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13b010bdf0 0x7f13b0101180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.544+0000 7f13aed9d640 1 -- 172.21.15.98:0/2585780992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b01a65a0 con 0x7f13b0102ab0 2024-05-09T08:50:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.544+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2585780992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0102ab0 0x7f13b0100700 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f13a0004770 tx=0x7f13a0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.544+0000 7f1397fff640 1 -- 172.21.15.98:0/2585780992 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a003f070 con 0x7f13b0102ab0 2024-05-09T08:50:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.544+0000 7f1397fff640 1 -- 172.21.15.98:0/2585780992 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13a0031ca0 con 0x7f13b0102ab0 2024-05-09T08:50:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.544+0000 7f1397fff640 1 -- 172.21.15.98:0/2585780992 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0035030 con 0x7f13b0102ab0 2024-05-09T08:50:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.544+0000 7f13b50e7640 1 -- 172.21.15.98:0/2585780992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13b01a6830 con 0x7f13b0102ab0 2024-05-09T08:50:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.544+0000 7f6192dd5640 1 -- 172.21.15.98:0/3434606053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f618c04f5f0 con 0x7f618c102ac0 2024-05-09T08:50:46.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.544+0000 7f13b50e7640 1 -- 172.21.15.98:0/2585780992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13b01a6d70 con 0x7f13b0102ab0 2024-05-09T08:50:46.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.546+0000 7f6189ffb640 1 -- 172.21.15.98:0/3434606053 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f617c0041d0 con 0x7f618c102ac0 2024-05-09T08:50:46.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.546+0000 7f6189ffb640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f61780a72d0 0x7f61780a9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.546+0000 7f6189ffb640 1 -- 172.21.15.98:0/3434606053 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f617c0cf590 con 0x7f618c102ac0 2024-05-09T08:50:46.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.546+0000 7f618bfff640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f61780a72d0 0x7f61780a9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.547+0000 7f618bfff640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f61780a72d0 0x7f61780a9790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6174005d70 tx=0x7f6174005d00 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.547+0000 7f1397fff640 1 -- 172.21.15.98:0/2585780992 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f13a0042070 con 0x7f13b0102ab0 2024-05-09T08:50:46.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.547+0000 7f13b7df1740 1 Processor -- start 2024-05-09T08:50:46.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.547+0000 7f13b7df1740 1 -- start start 2024-05-09T08:50:46.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.547+0000 7f13b7df1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730eb0b0 0x558a730e93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.547+0000 7f13b7df1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730e99c0 0x558a730e9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.547+0000 7f13b7df1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730f4740 0x558a730f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.547+0000 7f13b7df1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558a7304e990 con 0x558a730e99c0 2024-05-09T08:50:46.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.547+0000 7f13b7df1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558a72ff65b0 con 0x558a730f4740 2024-05-09T08:50:46.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.547+0000 7f13b7df1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558a7304a950 con 0x558a730eb0b0 2024-05-09T08:50:46.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.547+0000 7f13aed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730e99c0 0x558a730e9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.548+0000 7f13aed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730e99c0 0x558a730e9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57082/0 (socket says 172.21.15.98:57082) 2024-05-09T08:50:46.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.548+0000 7f13aed9d640 1 -- 172.21.15.98:0/2048168995 learned_addr learned my addr 172.21.15.98:0/2048168995 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.548+0000 7f1397fff640 1 --2- 172.21.15.98:0/2585780992 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f13980a2ed0 0x7f13980a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.548+0000 7f13ae59c640 1 --2- 172.21.15.98:0/2048168995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730f4740 0x558a730f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.548+0000 7f13aed9d640 1 -- 172.21.15.98:0/2048168995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730eb0b0 msgr2=0x558a730e93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.548+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2048168995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730eb0b0 0x558a730e93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.548+0000 7f13af59e640 1 --2- 172.21.15.98:0/2585780992 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f13980a2ed0 0x7f13980a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.548+0000 7f13aed9d640 1 -- 172.21.15.98:0/2048168995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730f4740 msgr2=0x558a730f6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.548+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2048168995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730f4740 0x558a730f6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.548+0000 7f13aed9d640 1 -- 172.21.15.98:0/2048168995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a72e2a7e0 con 0x558a730e99c0 2024-05-09T08:50:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.548+0000 7f1397fff640 1 -- 172.21.15.98:0/2585780992 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f13a00effc0 con 0x7f13b0102ab0 2024-05-09T08:50:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.548+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2048168995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730e99c0 0x558a730e9e40 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f13a00ed2e0 tx=0x7f13a0086d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e18e673e92028d11 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.548+0000 7f13af59e640 1 --2- 172.21.15.98:0/2585780992 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f13980a2ed0 0x7f13980a5390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f13a80098b0 tx=0x7f13a8009840 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.551+0000 7f13957fa640 1 -- 172.21.15.98:0/2048168995 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a00bb9b0 con 0x558a730e99c0 2024-05-09T08:50:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.551+0000 7f13957fa640 1 -- 172.21.15.98:0/2048168995 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13a008ae50 con 0x558a730e99c0 2024-05-09T08:50:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.551+0000 7f13957fa640 1 -- 172.21.15.98:0/2048168995 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0005d50 con 0x558a730e99c0 2024-05-09T08:50:46.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.551+0000 7f6189ffb640 1 -- 172.21.15.98:0/3434606053 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f617c09bba0 con 0x7f618c102ac0 2024-05-09T08:50:46.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.552+0000 7f13b7df1740 1 -- 172.21.15.98:0/2048168995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730e99c0 msgr2=0x558a730e9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.552+0000 7f13b7df1740 1 --2- 172.21.15.98:0/2048168995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730e99c0 0x558a730e9e40 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f13a00ed2e0 tx=0x7f13a0086d40 comp rx=0 tx=0).stop 2024-05-09T08:50:46.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.552+0000 7f13b7df1740 1 -- 172.21.15.98:0/2048168995 shutdown_connections 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.552+0000 7f13b7df1740 1 --2- 172.21.15.98:0/2048168995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730f4740 0x558a730f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.552+0000 7f13b7df1740 1 --2- 172.21.15.98:0/2048168995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730e99c0 0x558a730e9e40 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.552+0000 7f13b7df1740 1 --2- 172.21.15.98:0/2048168995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730eb0b0 0x558a730e93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.552+0000 7f13b7df1740 1 -- 172.21.15.98:0/2048168995 >> 172.21.15.98:0/2048168995 conn(0x558a730b20a0 msgr2=0x558a730e5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.552+0000 7f13b7df1740 1 -- 172.21.15.98:0/2048168995 shutdown_connections 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.552+0000 7f13b7df1740 1 -- 172.21.15.98:0/2048168995 wait complete. 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.552+0000 7f13b7df1740 1 -- start start 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13b7df1740 1 Processor -- start 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13b7df1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730e99c0 0x558a730f1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13b7df1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730eb0b0 0x558a730f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13b7df1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730f4740 0x558a730f2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13b7df1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558a730f7390 con 0x558a730f4740 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13b7df1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558a73009b40 con 0x558a730e99c0 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13b7df1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558a730f7510 con 0x558a730eb0b0 2024-05-09T08:50:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13b7df1740 10 client.0 _ll_register_callbacks cb 0x558a72ffe0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:46.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13af59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730f4740 0x558a730f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13af59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730f4740 0x558a730f2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57086/0 (socket says 172.21.15.98:57086) 2024-05-09T08:50:46.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13af59e640 1 -- 172.21.15.98:0/2335635750 learned_addr learned my addr 172.21.15.98:0/2335635750 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:46.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13ae59c640 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730eb0b0 0x558a730f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730e99c0 0x558a730f1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13af59e640 1 -- 172.21.15.98:0/2335635750 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730eb0b0 msgr2=0x558a730f1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13af59e640 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730eb0b0 0x558a730f1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13af59e640 1 -- 172.21.15.98:0/2335635750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730e99c0 msgr2=0x558a730f1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13af59e640 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730e99c0 0x558a730f1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13af59e640 1 -- 172.21.15.98:0/2335635750 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a72e8af70 con 0x558a730f4740 2024-05-09T08:50:46.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13ae59c640 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730eb0b0 0x558a730f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:46.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730e99c0 0x558a730f1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:46.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f13af59e640 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730f4740 0x558a730f2080 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f13a8004a80 tx=0x7f13a8009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63cdef1d3b216ab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.553+0000 7f1392ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.554+0000 7f1392ff5640 1 -- 172.21.15.98:0/2335635750 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a80109b0 con 0x558a730f4740 2024-05-09T08:50:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.554+0000 7f1392ff5640 1 -- 172.21.15.98:0/2335635750 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f13a8010b50 con 0x558a730f4740 2024-05-09T08:50:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.554+0000 7f13b7df1740 1 -- 172.21.15.98:0/2335635750 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13a008ae50 con 0x558a730f4740 2024-05-09T08:50:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.554+0000 7f1392ff5640 1 -- 172.21.15.98:0/2335635750 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f13a8010d50 con 0x558a730f4740 2024-05-09T08:50:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.554+0000 7f13b7df1740 10 client.15777 fetch_fsmap learned FSMap version 29 2024-05-09T08:50:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.554+0000 7f13b7df1740 1 -- 172.21.15.98:0/2335635750 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558a72d22170 con 0x558a730f4740 2024-05-09T08:50:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.554+0000 7f1392ff5640 1 -- 172.21.15.98:0/2335635750 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13a008ae50 con 0x558a730f4740 2024-05-09T08:50:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.554+0000 7f1392ff5640 1 -- 172.21.15.98:0/2335635750 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f13a8019030 con 0x558a730f4740 2024-05-09T08:50:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.554+0000 7f13b7df1740 10 client.15777 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:50:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.554+0000 7f13b7df1740 10 client.15777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:50:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.554+0000 7f13b7df1740 20 client.15777 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.554+0000 7f13b7df1740 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a73104520 0x558a73106910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.554+0000 7f13b7df1740 4 client.15777 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:50:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.554+0000 7f13b7df1740 1 -- 172.21.15.98:0/2335635750 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a7305c0c0 con 0x558a73104520 2024-05-09T08:50:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.555+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a73104520 0x558a73106910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.555+0000 7f1392ff5640 10 client.15777 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.555+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a73104520 0x558a73106910 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b21b04494f0a06e server_cookie=69599b484e5e6f1e in_seq=0 out_seq=0 2024-05-09T08:50:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f1392ff5640 1 -- 172.21.15.98:0/2335635750 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558a7305c0c0 con 0x558a73104520 2024-05-09T08:50:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f1392ff5640 10 client.15777 handle_command_reply: tid=0 2024-05-09T08:50:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f13b7df1740 1 client.15777 shutdown 2024-05-09T08:50:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f13b7df1740 1 -- 172.21.15.98:0/2335635750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730f4740 msgr2=0x558a730f2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f13b7df1740 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730f4740 0x558a730f2080 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f13a8004a80 tx=0x7f13a8009600 comp rx=0 tx=0).stop 2024-05-09T08:50:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f13b7df1740 1 -- 172.21.15.98:0/2335635750 shutdown_connections 2024-05-09T08:50:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f13b7df1740 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a73104520 0x558a73106910 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f13b7df1740 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730f4740 0x558a730f2080 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f1392ff5640 0 client.15777 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f13b7df1740 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730eb0b0 0x558a730f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f13b7df1740 1 --2- 172.21.15.98:0/2335635750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730e99c0 0x558a730f1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f13b7df1740 1 -- 172.21.15.98:0/2335635750 >> 172.21.15.98:0/2335635750 conn(0x558a730b20a0 msgr2=0x558a730ebd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f13b7df1740 1 -- 172.21.15.98:0/2335635750 shutdown_connections 2024-05-09T08:50:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f13b7df1740 1 -- 172.21.15.98:0/2335635750 wait complete. 2024-05-09T08:50:46.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.556+0000 7f13b7df1740 20 client.15777 trim_cache size 0 max 16384 2024-05-09T08:50:46.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.592+0000 7f13b7df1740 1 Processor -- start 2024-05-09T08:50:46.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.592+0000 7f13b7df1740 1 -- start start 2024-05-09T08:50:46.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.592+0000 7f13b7df1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730e99c0 0x558a73100150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.592+0000 7f13b7df1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730eb0b0 0x558a7310c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.592+0000 7f13b7df1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730f4740 0x558a7310c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.592+0000 7f13b7df1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01b0fe0 con 0x558a730eb0b0 2024-05-09T08:50:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.592+0000 7f13b7df1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01b2b20 con 0x558a730f4740 2024-05-09T08:50:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.592+0000 7f13b7df1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01b16b0 con 0x558a730e99c0 2024-05-09T08:50:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.592+0000 7f13ae59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730eb0b0 0x558a7310c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.592+0000 7f13ae59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730eb0b0 0x558a7310c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57096/0 (socket says 172.21.15.98:57096) 2024-05-09T08:50:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.592+0000 7f13ae59c640 1 -- 172.21.15.98:0/2308347884 learned_addr learned my addr 172.21.15.98:0/2308347884 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.592+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2308347884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730e99c0 0x558a73100150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.593+0000 7f13ae59c640 1 -- 172.21.15.98:0/2308347884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730e99c0 msgr2=0x558a73100150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.593+0000 7f13ae59c640 1 --2- 172.21.15.98:0/2308347884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730e99c0 0x558a73100150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.593+0000 7f13ae59c640 1 -- 172.21.15.98:0/2308347884 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730f4740 msgr2=0x558a7310c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.593+0000 7f13ae59c640 1 --2- 172.21.15.98:0/2308347884 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730f4740 0x558a7310c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.593+0000 7f13ae59c640 1 -- 172.21.15.98:0/2308347884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a730e0fd0 con 0x558a730eb0b0 2024-05-09T08:50:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.593+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2308347884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730e99c0 0x558a73100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.593+0000 7f13ae59c640 1 --2- 172.21.15.98:0/2308347884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730eb0b0 0x558a7310c220 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f13a4009cb0 tx=0x7f13a4010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637154db3d30468d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.593+0000 7f1394ff9640 1 -- 172.21.15.98:0/2308347884 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4009e60 con 0x558a730eb0b0 2024-05-09T08:50:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.593+0000 7f1394ff9640 1 -- 172.21.15.98:0/2308347884 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13a4004250 con 0x558a730eb0b0 2024-05-09T08:50:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.593+0000 7f1394ff9640 1 -- 172.21.15.98:0/2308347884 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a401c860 con 0x558a730eb0b0 2024-05-09T08:50:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 -- 172.21.15.98:0/2308347884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730eb0b0 msgr2=0x558a7310c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 --2- 172.21.15.98:0/2308347884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730eb0b0 0x558a7310c220 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f13a4009cb0 tx=0x7f13a4010800 comp rx=0 tx=0).stop 2024-05-09T08:50:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 -- 172.21.15.98:0/2308347884 shutdown_connections 2024-05-09T08:50:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 --2- 172.21.15.98:0/2308347884 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730f4740 0x558a7310c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 --2- 172.21.15.98:0/2308347884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730eb0b0 0x558a7310c220 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 --2- 172.21.15.98:0/2308347884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730e99c0 0x558a73100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 -- 172.21.15.98:0/2308347884 >> 172.21.15.98:0/2308347884 conn(0x558a730b20a0 msgr2=0x558a730efed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 -- 172.21.15.98:0/2308347884 shutdown_connections 2024-05-09T08:50:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 -- 172.21.15.98:0/2308347884 wait complete. 2024-05-09T08:50:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 -- start start 2024-05-09T08:50:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 Processor -- start 2024-05-09T08:50:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730e99c0 0x558a730fc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730eb0b0 0x558a730fcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730f4740 0x558a730f75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01b2450 con 0x558a730e99c0 2024-05-09T08:50:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01b1d80 con 0x558a730eb0b0 2024-05-09T08:50:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01aefc0 con 0x558a730f4740 2024-05-09T08:50:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13b7df1740 10 client.0 _ll_register_callbacks cb 0x7f13a8009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13aed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730e99c0 0x558a730fc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.594+0000 7f13aed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730e99c0 0x558a730fc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57110/0 (socket says 172.21.15.98:57110) 2024-05-09T08:50:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f13aed9d640 1 -- 172.21.15.98:0/202746626 learned_addr learned my addr 172.21.15.98:0/202746626 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f13af59e640 1 --2- 172.21.15.98:0/202746626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730f4740 0x558a730f75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f13aed9d640 1 -- 172.21.15.98:0/202746626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730f4740 msgr2=0x558a730f75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f13aed9d640 1 --2- 172.21.15.98:0/202746626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730f4740 0x558a730f75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f13aed9d640 1 -- 172.21.15.98:0/202746626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730eb0b0 msgr2=0x558a730fcb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f13aed9d640 1 --2- 172.21.15.98:0/202746626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730eb0b0 0x558a730fcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f13aed9d640 1 -- 172.21.15.98:0/202746626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a730f7d30 con 0x558a730e99c0 2024-05-09T08:50:46.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f13af59e640 1 --2- 172.21.15.98:0/202746626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730f4740 0x558a730f75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:46.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f13aed9d640 1 --2- 172.21.15.98:0/202746626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730e99c0 0x558a730fc640 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f13a00ed2e0 tx=0x7f13a003b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3feb7ca3fb033cc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f1392ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:46.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f1392ff5640 1 -- 172.21.15.98:0/202746626 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a00bb9b0 con 0x558a730e99c0 2024-05-09T08:50:46.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f1392ff5640 1 -- 172.21.15.98:0/202746626 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f13a00ba350 con 0x558a730e99c0 2024-05-09T08:50:46.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f13b7df1740 1 -- 172.21.15.98:0/202746626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13a4004250 con 0x558a730e99c0 2024-05-09T08:50:46.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f1392ff5640 1 -- 172.21.15.98:0/202746626 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f13a00c0350 con 0x558a730e99c0 2024-05-09T08:50:46.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f13b7df1740 10 client.15783 fetch_fsmap learned FSMap version 29 2024-05-09T08:50:46.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f13b7df1740 1 -- 172.21.15.98:0/202746626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558a7316b440 con 0x558a730e99c0 2024-05-09T08:50:46.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.595+0000 7f1392ff5640 1 -- 172.21.15.98:0/202746626 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13a4004250 con 0x558a730e99c0 2024-05-09T08:50:46.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.596+0000 7f1392ff5640 1 -- 172.21.15.98:0/202746626 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f13a00b86c0 con 0x558a730e99c0 2024-05-09T08:50:46.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.596+0000 7f13b7df1740 10 client.15783 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:50:46.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.596+0000 7f13b7df1740 10 client.15783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:50:46.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.596+0000 7f13b7df1740 20 client.15783 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:46.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.596+0000 7f13b7df1740 1 --2- 172.21.15.98:0/202746626 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a7316b9e0 0x558a7316de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.596+0000 7f13b7df1740 4 client.15783 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:50:46.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.596+0000 7f13b7df1740 1 -- 172.21.15.98:0/202746626 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558a73056410 con 0x558a7316b9e0 2024-05-09T08:50:46.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.596+0000 7f13ae59c640 1 --2- 172.21.15.98:0/202746626 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a7316b9e0 0x558a7316de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.596+0000 7f13ae59c640 1 --2- 172.21.15.98:0/202746626 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a7316b9e0 0x558a7316de20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b58ff3b32553d22b server_cookie=32690fda557e6a9 in_seq=0 out_seq=0 2024-05-09T08:50:46.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.596+0000 7f1392ff5640 10 client.15783 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:46.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.597+0000 7f1392ff5640 1 -- 172.21.15.98:0/202746626 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558a73056410 con 0x558a7316b9e0 2024-05-09T08:50:46.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.597+0000 7f1392ff5640 10 client.15783 handle_command_reply: tid=0 2024-05-09T08:50:46.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.597+0000 7f13b7df1740 1 client.15783 shutdown 2024-05-09T08:50:46.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b7df1740 1 -- 172.21.15.98:0/202746626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730e99c0 msgr2=0x558a730fc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b7df1740 1 --2- 172.21.15.98:0/202746626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730e99c0 0x558a730fc640 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f13a00ed2e0 tx=0x7f13a003b890 comp rx=0 tx=0).stop 2024-05-09T08:50:46.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b7df1740 1 -- 172.21.15.98:0/202746626 shutdown_connections 2024-05-09T08:50:46.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b7df1740 1 --2- 172.21.15.98:0/202746626 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558a7316b9e0 0x558a7316de20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b7df1740 1 --2- 172.21.15.98:0/202746626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558a730f4740 0x558a730f75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b7df1740 1 --2- 172.21.15.98:0/202746626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a730eb0b0 0x558a730fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f1392ff5640 0 client.15783 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:46.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b7df1740 1 --2- 172.21.15.98:0/202746626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558a730e99c0 0x558a730fc640 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b7df1740 1 -- 172.21.15.98:0/202746626 >> 172.21.15.98:0/202746626 conn(0x558a730b20a0 msgr2=0x558a730e47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b7df1740 1 -- 172.21.15.98:0/202746626 shutdown_connections 2024-05-09T08:50:46.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b7df1740 1 -- 172.21.15.98:0/202746626 wait complete. 2024-05-09T08:50:46.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b7df1740 20 client.15783 trim_cache size 0 max 16384 2024-05-09T08:50:46.603 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:50:46.604 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:50:46.604 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "b4468cb3-d156-495d-b704-1cb28c8a7b7e", 2024-05-09T08:50:46.604 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:50:46.604 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:50:46.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b50e7640 1 -- 172.21.15.98:0/2585780992 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f13980a2ed0 msgr2=0x7f13980a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b50e7640 1 --2- 172.21.15.98:0/2585780992 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f13980a2ed0 0x7f13980a5390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f13a80098b0 tx=0x7f13a8009840 comp rx=0 tx=0).stop 2024-05-09T08:50:46.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b50e7640 1 -- 172.21.15.98:0/2585780992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0102ab0 msgr2=0x7f13b0100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b50e7640 1 --2- 172.21.15.98:0/2585780992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0102ab0 0x7f13b0100700 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f13a0004770 tx=0x7f13a0003c10 comp rx=0 tx=0).stop 2024-05-09T08:50:46.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b50e7640 1 -- 172.21.15.98:0/2585780992 shutdown_connections 2024-05-09T08:50:46.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b50e7640 1 --2- 172.21.15.98:0/2585780992 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f13980a2ed0 0x7f13980a5390 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b50e7640 1 --2- 172.21.15.98:0/2585780992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13b010bdf0 0x7f13b0101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b50e7640 1 --2- 172.21.15.98:0/2585780992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13b01033d0 0x7f13b0100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b50e7640 1 --2- 172.21.15.98:0/2585780992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0102ab0 0x7f13b0100700 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.598+0000 7f13b50e7640 1 -- 172.21.15.98:0/2585780992 >> 172.21.15.98:0/2585780992 conn(0x7f13b0077830 msgr2=0x7f13b00fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.599+0000 7f13b50e7640 1 -- 172.21.15.98:0/2585780992 shutdown_connections 2024-05-09T08:50:46.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.599+0000 7f13b50e7640 1 -- 172.21.15.98:0/2585780992 wait complete. 2024-05-09T08:50:46.609 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:50:46.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:50:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:46 smithi142 ceph-mon[34625]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-09T08:50:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:46 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2404210567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:46 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/80234058' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:46 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3264581540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:46.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.713+0000 7feabc37e640 1 -- 172.21.15.98:0/971206189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab41021d0 msgr2=0x7feab410e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.713+0000 7feabc37e640 1 --2- 172.21.15.98:0/971206189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab41021d0 0x7feab410e6e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7feab0009930 tx=0x7feab002a420 comp rx=0 tx=0).stop 2024-05-09T08:50:46.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 -- 172.21.15.98:0/971206189 shutdown_connections 2024-05-09T08:50:46.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 --2- 172.21.15.98:0/971206189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab41021d0 0x7feab410e6e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 --2- 172.21.15.98:0/971206189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feab4101810 0x7feab4101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 --2- 172.21.15.98:0/971206189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feab4107820 0x7feab4107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 -- 172.21.15.98:0/971206189 >> 172.21.15.98:0/971206189 conn(0x7feab4078c30 msgr2=0x7feab4079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 -- 172.21.15.98:0/971206189 shutdown_connections 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 -- 172.21.15.98:0/971206189 wait complete. 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 Processor -- start 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 -- start start 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feab4101810 0x7feab419a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab41021d0 0x7feab419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feab4107820 0x7feab419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feab4111c30 con 0x7feab41021d0 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feab4111ab0 con 0x7feab4107820 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feabc37e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7feab4111db0 con 0x7feab4101810 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.714+0000 7feab9919640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab41021d0 0x7feab419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feab9919640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab41021d0 0x7feab419a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57124/0 (socket says 172.21.15.98:57124) 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feab9919640 1 -- 172.21.15.98:0/3964136044 learned_addr learned my addr 172.21.15.98:0/3964136044 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feaba11a640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feab4101810 0x7feab419a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feaba91b640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feab4107820 0x7feab419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feab9919640 1 -- 172.21.15.98:0/3964136044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feab4101810 msgr2=0x7feab419a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feab9919640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feab4101810 0x7feab419a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feab9919640 1 -- 172.21.15.98:0/3964136044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feab4107820 msgr2=0x7feab419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feab9919640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feab4107820 0x7feab419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feab9919640 1 -- 172.21.15.98:0/3964136044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab419f2a0 con 0x7feab41021d0 2024-05-09T08:50:46.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feaba91b640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feab4107820 0x7feab419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:46.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feab9919640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab41021d0 0x7feab419a7f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7feaa800e9f0 tx=0x7feaa800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feaa77fe640 1 -- 172.21.15.98:0/3964136044 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa800cd80 con 0x7feab41021d0 2024-05-09T08:50:46.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feaa77fe640 1 -- 172.21.15.98:0/3964136044 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaa8010910 con 0x7feab41021d0 2024-05-09T08:50:46.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feabc37e640 1 -- 172.21.15.98:0/3964136044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab419f590 con 0x7feab41021d0 2024-05-09T08:50:46.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feaa77fe640 1 -- 172.21.15.98:0/3964136044 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa8018990 con 0x7feab41021d0 2024-05-09T08:50:46.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.715+0000 7feabc37e640 1 -- 172.21.15.98:0/3964136044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feab41a6e20 con 0x7feab41021d0 2024-05-09T08:50:46.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.717+0000 7feaa77fe640 1 -- 172.21.15.98:0/3964136044 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7feaa8010ac0 con 0x7feab41021d0 2024-05-09T08:50:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.718+0000 7feaa77fe640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feaa00a2e20 0x7feaa00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.718+0000 7feaba11a640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feaa00a2e20 0x7feaa00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.718+0000 7feaa77fe640 1 -- 172.21.15.98:0/3964136044 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7feaa80cfe60 con 0x7feab41021d0 2024-05-09T08:50:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.718+0000 7feaba11a640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feaa00a2e20 0x7feaa00a52e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feaac005fd0 tx=0x7feaac009680 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.719+0000 7feabf16e740 1 Processor -- start 2024-05-09T08:50:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.719+0000 7feabf16e740 1 -- start start 2024-05-09T08:50:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.719+0000 7feabf16e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb49280 0x55cb4fb49660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.719+0000 7feabf16e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb53d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.719+0000 7feabf16e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb54270 0x55cb4fb56660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.719+0000 7feabf16e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cb4faaed00 con 0x55cb4fb49280 2024-05-09T08:50:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.719+0000 7feabf16e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cb4fa564c0 con 0x55cb4fb54270 2024-05-09T08:50:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.719+0000 7feabf16e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cb4faaacc0 con 0x55cb4fb49ba0 2024-05-09T08:50:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.720+0000 7feaba11a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb49280 0x55cb4fb49660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.720+0000 7feaba11a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb49280 0x55cb4fb49660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57138/0 (socket says 172.21.15.98:57138) 2024-05-09T08:50:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.720+0000 7feaba11a640 1 -- 172.21.15.98:0/3860773858 learned_addr learned my addr 172.21.15.98:0/3860773858 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.720+0000 7feaba11a640 1 -- 172.21.15.98:0/3860773858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 msgr2=0x55cb4fb53d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.720+0000 7feaba11a640 1 --2- 172.21.15.98:0/3860773858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb53d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.720+0000 7feaba11a640 1 -- 172.21.15.98:0/3860773858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb54270 msgr2=0x55cb4fb56660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.720+0000 7feaba11a640 1 --2- 172.21.15.98:0/3860773858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb54270 0x55cb4fb56660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.720+0000 7feaba11a640 1 -- 172.21.15.98:0/3860773858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb4f88c7d0 con 0x55cb4fb49280 2024-05-09T08:50:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.720+0000 7feaba11a640 1 --2- 172.21.15.98:0/3860773858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb49280 0x55cb4fb49660 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feaac012770 tx=0x7feaac013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3799aaeeb4e4726b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.720+0000 7feaa4bfe640 1 -- 172.21.15.98:0/3860773858 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac023070 con 0x55cb4fb49280 2024-05-09T08:50:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.720+0000 7feaa4bfe640 1 -- 172.21.15.98:0/3860773858 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaac011c80 con 0x55cb4fb49280 2024-05-09T08:50:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.720+0000 7feaa4bfe640 1 -- 172.21.15.98:0/3860773858 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac01fde0 con 0x55cb4fb49280 2024-05-09T08:50:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.721+0000 7feabf16e740 1 -- 172.21.15.98:0/3860773858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb49280 msgr2=0x55cb4fb49660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.721+0000 7feabf16e740 1 --2- 172.21.15.98:0/3860773858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb49280 0x55cb4fb49660 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feaac012770 tx=0x7feaac013000 comp rx=0 tx=0).stop 2024-05-09T08:50:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.721+0000 7feabf16e740 1 -- 172.21.15.98:0/3860773858 shutdown_connections 2024-05-09T08:50:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.721+0000 7feabf16e740 1 --2- 172.21.15.98:0/3860773858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb54270 0x55cb4fb56660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.721+0000 7feabf16e740 1 --2- 172.21.15.98:0/3860773858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb53d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.721+0000 7feabf16e740 1 --2- 172.21.15.98:0/3860773858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb49280 0x55cb4fb49660 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.721+0000 7feabf16e740 1 -- 172.21.15.98:0/3860773858 >> 172.21.15.98:0/3860773858 conn(0x55cb4fb455a0 msgr2=0x55cb4fb45980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.721+0000 7feabf16e740 1 -- 172.21.15.98:0/3860773858 shutdown_connections 2024-05-09T08:50:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.721+0000 7feabf16e740 1 -- 172.21.15.98:0/3860773858 wait complete. 2024-05-09T08:50:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.721+0000 7feabf16e740 1 -- start start 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feabf16e740 1 Processor -- start 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feabf16e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb49280 0x55cb4fb51210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feabf16e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb51750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feabf16e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb54270 0x55cb4fb51c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feabf16e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cb4fb57060 con 0x55cb4fb54270 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feabf16e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cb4fa69b90 con 0x55cb4fb49280 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feabf16e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cb4fb571e0 con 0x55cb4fb49ba0 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feabf16e740 10 client.0 _ll_register_callbacks cb 0x55cb4faa5da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feaba91b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb54270 0x55cb4fb51c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feaba91b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb54270 0x55cb4fb51c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57152/0 (socket says 172.21.15.98:57152) 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feaba91b640 1 -- 172.21.15.98:0/682420813 learned_addr learned my addr 172.21.15.98:0/682420813 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feab9919640 1 --2- 172.21.15.98:0/682420813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb51750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feaba11a640 1 --2- 172.21.15.98:0/682420813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb49280 0x55cb4fb51210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feaba91b640 1 -- 172.21.15.98:0/682420813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 msgr2=0x55cb4fb51750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feaba91b640 1 --2- 172.21.15.98:0/682420813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb51750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feaba91b640 1 -- 172.21.15.98:0/682420813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb49280 msgr2=0x55cb4fb51210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feaba91b640 1 --2- 172.21.15.98:0/682420813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb49280 0x55cb4fb51210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feaba91b640 1 -- 172.21.15.98:0/682420813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb4f7c23f0 con 0x55cb4fb54270 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feab9919640 1 --2- 172.21.15.98:0/682420813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb51750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feaba91b640 1 --2- 172.21.15.98:0/682420813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb54270 0x55cb4fb51c90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7feab00092a0 tx=0x7feab0032e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588e16b74eb3d8fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7fea9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7fea9dffb640 1 -- 172.21.15.98:0/682420813 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab003f070 con 0x55cb4fb54270 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7fea9dffb640 1 -- 172.21.15.98:0/682420813 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7feab00319c0 con 0x55cb4fb54270 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.722+0000 7feabf16e740 1 -- 172.21.15.98:0/682420813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feaac011c80 con 0x55cb4fb54270 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.723+0000 7fea9dffb640 1 -- 172.21.15.98:0/682420813 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feab004b050 con 0x55cb4fb54270 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.723+0000 7feabf16e740 10 client.15795 fetch_fsmap learned FSMap version 29 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.723+0000 7feabf16e740 1 -- 172.21.15.98:0/682420813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cb4f7820f0 con 0x55cb4fb54270 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.723+0000 7fea9dffb640 1 -- 172.21.15.98:0/682420813 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaac011c80 con 0x55cb4fb54270 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.723+0000 7fea9dffb640 1 -- 172.21.15.98:0/682420813 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7feab0004750 con 0x55cb4fb54270 2024-05-09T08:50:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.723+0000 7feabf16e740 10 client.15795 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:50:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.723+0000 7feabf16e740 10 client.15795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:50:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.723+0000 7feabf16e740 20 client.15795 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.723+0000 7feabf16e740 1 --2- 172.21.15.98:0/682420813 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb4fb64650 0x55cb4fb66a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.723+0000 7feabf16e740 4 client.15795 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:50:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.723+0000 7feabf16e740 1 -- 172.21.15.98:0/682420813 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb4fabc430 con 0x55cb4fb64650 2024-05-09T08:50:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.723+0000 7feaba11a640 1 --2- 172.21.15.98:0/682420813 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb4fb64650 0x55cb4fb66a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.724+0000 7feaba11a640 1 --2- 172.21.15.98:0/682420813 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb4fb64650 0x55cb4fb66a70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bae2e67e21a692c server_cookie=3d0f016f54133aa9 in_seq=0 out_seq=0 2024-05-09T08:50:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.724+0000 7fea9dffb640 10 client.15795 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.724+0000 7fea9dffb640 1 -- 172.21.15.98:0/682420813 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cb4fabc430 con 0x55cb4fb64650 2024-05-09T08:50:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.724+0000 7fea9dffb640 10 client.15795 handle_command_reply: tid=0 2024-05-09T08:50:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.724+0000 7feabf16e740 1 client.15795 shutdown 2024-05-09T08:50:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.724+0000 7feabf16e740 1 -- 172.21.15.98:0/682420813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb54270 msgr2=0x55cb4fb51c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.724+0000 7feabf16e740 1 --2- 172.21.15.98:0/682420813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb54270 0x55cb4fb51c90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7feab00092a0 tx=0x7feab0032e60 comp rx=0 tx=0).stop 2024-05-09T08:50:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.724+0000 7feabf16e740 1 -- 172.21.15.98:0/682420813 shutdown_connections 2024-05-09T08:50:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.724+0000 7feabf16e740 1 --2- 172.21.15.98:0/682420813 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb4fb64650 0x55cb4fb66a70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.724+0000 7feabf16e740 1 --2- 172.21.15.98:0/682420813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb54270 0x55cb4fb51c90 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.724+0000 7feabf16e740 1 --2- 172.21.15.98:0/682420813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb51750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.724+0000 7fea9dffb640 0 client.15795 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.724+0000 7feabf16e740 1 --2- 172.21.15.98:0/682420813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb49280 0x55cb4fb51210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.724+0000 7feabf16e740 1 -- 172.21.15.98:0/682420813 >> 172.21.15.98:0/682420813 conn(0x55cb4fb455a0 msgr2=0x55cb4fb447e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.725+0000 7feabf16e740 1 -- 172.21.15.98:0/682420813 shutdown_connections 2024-05-09T08:50:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.725+0000 7feabf16e740 1 -- 172.21.15.98:0/682420813 wait complete. 2024-05-09T08:50:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.725+0000 7feabf16e740 20 client.15795 trim_cache size 0 max 16384 2024-05-09T08:50:46.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:46 smithi105 ceph-mon[34498]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-09T08:50:46.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:46 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2404210567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:50:46.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:46 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/80234058' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:50:46.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:46 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3264581540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:50:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feabf16e740 1 Processor -- start 2024-05-09T08:50:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feabf16e740 1 -- start start 2024-05-09T08:50:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feabf16e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb49280 0x55cb4fb4f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feabf16e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb5ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feabf16e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb54270 0x55cb4fb51ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feabf16e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feab41b1050 con 0x55cb4fb49280 2024-05-09T08:50:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feabf16e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feab41b2b90 con 0x55cb4fb54270 2024-05-09T08:50:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feabf16e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7feab41b1720 con 0x55cb4fb49ba0 2024-05-09T08:50:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feaba11a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb49280 0x55cb4fb4f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feaba11a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb49280 0x55cb4fb4f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57168/0 (socket says 172.21.15.98:57168) 2024-05-09T08:50:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feaba11a640 1 -- 172.21.15.98:0/3220519807 learned_addr learned my addr 172.21.15.98:0/3220519807 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feaba91b640 1 --2- 172.21.15.98:0/3220519807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb54270 0x55cb4fb51ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feab9919640 1 --2- 172.21.15.98:0/3220519807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb5ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feaba11a640 1 -- 172.21.15.98:0/3220519807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 msgr2=0x55cb4fb5ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feaba11a640 1 --2- 172.21.15.98:0/3220519807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb5ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feaba11a640 1 -- 172.21.15.98:0/3220519807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb54270 msgr2=0x55cb4fb51ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feaba11a640 1 --2- 172.21.15.98:0/3220519807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb54270 0x55cb4fb51ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feaba11a640 1 -- 172.21.15.98:0/3220519807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb4fb499d0 con 0x55cb4fb49280 2024-05-09T08:50:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feaba91b640 1 --2- 172.21.15.98:0/3220519807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb54270 0x55cb4fb51ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:50:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.760+0000 7feaba11a640 1 --2- 172.21.15.98:0/3220519807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb49280 0x55cb4fb4f9b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7feaac025000 tx=0x7feaac010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6aea29c2bb3df51 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.761+0000 7fea9ffff640 1 -- 172.21.15.98:0/3220519807 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac023040 con 0x55cb4fb49280 2024-05-09T08:50:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.761+0000 7fea9ffff640 1 -- 172.21.15.98:0/3220519807 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaac003970 con 0x55cb4fb49280 2024-05-09T08:50:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.761+0000 7fea9ffff640 1 -- 172.21.15.98:0/3220519807 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac011a70 con 0x55cb4fb49280 2024-05-09T08:50:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.761+0000 7feabf16e740 1 -- 172.21.15.98:0/3220519807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb49280 msgr2=0x55cb4fb4f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.761+0000 7feabf16e740 1 --2- 172.21.15.98:0/3220519807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb49280 0x55cb4fb4f9b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7feaac025000 tx=0x7feaac010da0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.761+0000 7feabf16e740 1 -- 172.21.15.98:0/3220519807 shutdown_connections 2024-05-09T08:50:46.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.761+0000 7feabf16e740 1 --2- 172.21.15.98:0/3220519807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb54270 0x55cb4fb51ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.761+0000 7feabf16e740 1 --2- 172.21.15.98:0/3220519807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb5ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.761+0000 7feabf16e740 1 --2- 172.21.15.98:0/3220519807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb49280 0x55cb4fb4f9b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.761+0000 7feabf16e740 1 -- 172.21.15.98:0/3220519807 >> 172.21.15.98:0/3220519807 conn(0x55cb4fb455a0 msgr2=0x55cb4fb4b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.761+0000 7feabf16e740 1 -- 172.21.15.98:0/3220519807 shutdown_connections 2024-05-09T08:50:46.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.761+0000 7feabf16e740 1 -- 172.21.15.98:0/3220519807 wait complete. 2024-05-09T08:50:46.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feabf16e740 1 -- start start 2024-05-09T08:50:46.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feabf16e740 1 Processor -- start 2024-05-09T08:50:46.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feabf16e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb49280 0x55cb4fb4dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feabf16e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb4e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feabf16e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb54270 0x55cb4fb4e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feabf16e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feab41b24c0 con 0x55cb4fb54270 2024-05-09T08:50:46.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feabf16e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feab41b1df0 con 0x55cb4fb49280 2024-05-09T08:50:46.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feabf16e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7feab41af030 con 0x55cb4fb49ba0 2024-05-09T08:50:46.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feabf16e740 10 client.0 _ll_register_callbacks cb 0x7feaa8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feaba91b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb54270 0x55cb4fb4e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feaba91b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb54270 0x55cb4fb4e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57176/0 (socket says 172.21.15.98:57176) 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feaba91b640 1 -- 172.21.15.98:0/4064135531 learned_addr learned my addr 172.21.15.98:0/4064135531 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feab9919640 1 --2- 172.21.15.98:0/4064135531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb4e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feaba11a640 1 --2- 172.21.15.98:0/4064135531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb49280 0x55cb4fb4dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feaba91b640 1 -- 172.21.15.98:0/4064135531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 msgr2=0x55cb4fb4e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feaba91b640 1 --2- 172.21.15.98:0/4064135531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb4e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feaba91b640 1 -- 172.21.15.98:0/4064135531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb49280 msgr2=0x55cb4fb4dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feaba91b640 1 --2- 172.21.15.98:0/4064135531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb49280 0x55cb4fb4dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feaba91b640 1 -- 172.21.15.98:0/4064135531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb4fbc9b90 con 0x55cb4fb54270 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7feaba91b640 1 --2- 172.21.15.98:0/4064135531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb54270 0x55cb4fb4e7d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7feab0009930 tx=0x7feab0031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54188ae75e2dfa47 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.762+0000 7fea9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7fea9dffb640 1 -- 172.21.15.98:0/4064135531 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab0053070 con 0x55cb4fb54270 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7feabf16e740 1 -- 172.21.15.98:0/4064135531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feaac003970 con 0x55cb4fb54270 2024-05-09T08:50:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7fea9dffb640 1 -- 172.21.15.98:0/4064135531 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7feab0004cf0 con 0x55cb4fb54270 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7fea9dffb640 1 -- 172.21.15.98:0/4064135531 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feab005f050 con 0x55cb4fb54270 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7feabf16e740 10 client.15801 fetch_fsmap learned FSMap version 29 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7feabf16e740 1 -- 172.21.15.98:0/4064135531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cb4fbcad80 con 0x55cb4fb54270 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7fea9dffb640 1 -- 172.21.15.98:0/4064135531 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaac003970 con 0x55cb4fb54270 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7fea9dffb640 1 -- 172.21.15.98:0/4064135531 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7feab003b7c0 con 0x55cb4fb54270 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7feabf16e740 10 client.15801 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7feabf16e740 10 client.15801 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7feabf16e740 20 client.15801 populate_metadata read hostname 'smithi098' 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7feabf16e740 1 --2- 172.21.15.98:0/4064135531 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb4fbcb4a0 0x55cb4fbcd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7feabf16e740 4 client.15801 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7feabf16e740 1 -- 172.21.15.98:0/4064135531 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cb4fab6780 con 0x55cb4fbcb4a0 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.763+0000 7feaba11a640 1 --2- 172.21.15.98:0/4064135531 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb4fbcb4a0 0x55cb4fbcd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7fea9dffb640 10 client.15801 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7feaba11a640 1 --2- 172.21.15.98:0/4064135531 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb4fbcb4a0 0x55cb4fbcd8c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa104e5d80d33b29 server_cookie=bc3e2d533b2da2eb in_seq=0 out_seq=0 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7fea9dffb640 1 -- 172.21.15.98:0/4064135531 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cb4fab6780 con 0x55cb4fbcb4a0 2024-05-09T08:50:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7fea9dffb640 10 client.15801 handle_command_reply: tid=0 2024-05-09T08:50:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7feabf16e740 1 client.15801 shutdown 2024-05-09T08:50:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7feabf16e740 1 -- 172.21.15.98:0/4064135531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb54270 msgr2=0x55cb4fb4e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7feabf16e740 1 --2- 172.21.15.98:0/4064135531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb54270 0x55cb4fb4e7d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7feab0009930 tx=0x7feab0031a50 comp rx=0 tx=0).stop 2024-05-09T08:50:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7feabf16e740 1 -- 172.21.15.98:0/4064135531 shutdown_connections 2024-05-09T08:50:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7feabf16e740 1 --2- 172.21.15.98:0/4064135531 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb4fbcb4a0 0x55cb4fbcd8c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7feabf16e740 1 --2- 172.21.15.98:0/4064135531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb4fb54270 0x55cb4fb4e7d0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7feabf16e740 1 --2- 172.21.15.98:0/4064135531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb4fb49ba0 0x55cb4fb4e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7fea9dffb640 0 client.15801 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:50:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7feabf16e740 1 --2- 172.21.15.98:0/4064135531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb4fb49280 0x55cb4fb4dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7feabf16e740 1 -- 172.21.15.98:0/4064135531 >> 172.21.15.98:0/4064135531 conn(0x55cb4fb455a0 msgr2=0x55cb4fb43f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7feabf16e740 1 -- 172.21.15.98:0/4064135531 shutdown_connections 2024-05-09T08:50:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.764+0000 7feabf16e740 1 -- 172.21.15.98:0/4064135531 wait complete. 2024-05-09T08:50:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.765+0000 7feabf16e740 20 client.15801 trim_cache size 0 max 16384 2024-05-09T08:50:46.771 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:50:46.771 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:50:46.771 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:50:46.771 INFO:teuthology.orchestra.run.smithi098.stdout: "b4468cb3-d156-495d-b704-1cb28c8a7b7e": { 2024-05-09T08:50:46.771 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:50:46.771 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "b4468cb3-d156-495d-b704-1cb28c8a7b7e", 2024-05-09T08:50:46.771 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:50:46.771 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:50:46.771 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:50:46.771 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:50:46.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.765+0000 7feabc37e640 1 -- 172.21.15.98:0/3964136044 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feaa00a2e20 msgr2=0x7feaa00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.765+0000 7feabc37e640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feaa00a2e20 0x7feaa00a52e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feaac005fd0 tx=0x7feaac009680 comp rx=0 tx=0).stop 2024-05-09T08:50:46.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.765+0000 7feabc37e640 1 -- 172.21.15.98:0/3964136044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab41021d0 msgr2=0x7feab419a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.765+0000 7feabc37e640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab41021d0 0x7feab419a7f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7feaa800e9f0 tx=0x7feaa800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.765+0000 7feabc37e640 1 -- 172.21.15.98:0/3964136044 shutdown_connections 2024-05-09T08:50:46.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.765+0000 7feabc37e640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feaa00a2e20 0x7feaa00a52e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.765+0000 7feabc37e640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feab4107820 0x7feab419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.765+0000 7feabc37e640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab41021d0 0x7feab419a7f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.765+0000 7feabc37e640 1 --2- 172.21.15.98:0/3964136044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feab4101810 0x7feab419a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.765+0000 7feabc37e640 1 -- 172.21.15.98:0/3964136044 >> 172.21.15.98:0/3964136044 conn(0x7feab4078c30 msgr2=0x7feab4103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.765+0000 7feabc37e640 1 -- 172.21.15.98:0/3964136044 shutdown_connections 2024-05-09T08:50:46.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.765+0000 7feabc37e640 1 -- 172.21.15.98:0/3964136044 wait complete. 2024-05-09T08:50:46.775 INFO:tasks.cephfs.filesystem:scrub status for tag:b4468cb3-d156-495d-b704-1cb28c8a7b7e - {'path': '/', 'tag': 'b4468cb3-d156-495d-b704-1cb28c8a7b7e', 'options': 'recursive,force'} 2024-05-09T08:50:46.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.801+0000 7f6192dd5640 1 -- 172.21.15.98:0/3434606053 --> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f618c0fde70 con 0x7f61780a72d0 2024-05-09T08:50:46.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.875+0000 7f6189ffb640 1 -- 172.21.15.98:0/3434606053 <== mgr.15555 v2:172.21.15.98:6800/4136638623 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f618c0fde70 con 0x7f61780a72d0 2024-05-09T08:50:46.875 INFO:teuthology.orchestra.run.smithi098.stdout:Schedule set for path / 2024-05-09T08:50:46.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.878+0000 7f6192dd5640 1 -- 172.21.15.98:0/3434606053 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f61780a72d0 msgr2=0x7f61780a9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.878+0000 7f6192dd5640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f61780a72d0 0x7f61780a9790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6174005d70 tx=0x7f6174005d00 comp rx=0 tx=0).stop 2024-05-09T08:50:46.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.878+0000 7f6192dd5640 1 -- 172.21.15.98:0/3434606053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f618c102ac0 msgr2=0x7f618c198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:46.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.878+0000 7f6192dd5640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f618c102ac0 0x7f618c198040 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f617c00d950 tx=0x7f617c00de10 comp rx=0 tx=0).stop 2024-05-09T08:50:46.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.878+0000 7f6192dd5640 1 -- 172.21.15.98:0/3434606053 shutdown_connections 2024-05-09T08:50:46.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.878+0000 7f6192dd5640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f61780a72d0 0x7f61780a9790 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.878+0000 7f6192dd5640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f618c10be00 0x7f618c19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.878+0000 7f6192dd5640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f618c1033e0 0x7f618c198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.878+0000 7f6192dd5640 1 --2- 172.21.15.98:0/3434606053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f618c102ac0 0x7f618c198040 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:46.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.878+0000 7f6192dd5640 1 -- 172.21.15.98:0/3434606053 >> 172.21.15.98:0/3434606053 conn(0x7f618c0778b0 msgr2=0x7f618c0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:46.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.878+0000 7f6192dd5640 1 -- 172.21.15.98:0/3434606053 shutdown_connections 2024-05-09T08:50:46.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:46.878+0000 7f6192dd5640 1 -- 172.21.15.98:0/3434606053 wait complete. 2024-05-09T08:50:46.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-05-09T08:50:47.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.032+0000 7f88dfaca640 1 -- 172.21.15.98:0/657568198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88d8100760 msgr2=0x7f88d8100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.032+0000 7f88dfaca640 1 --2- 172.21.15.98:0/657568198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88d8100760 0x7f88d8100be0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009930 tx=0x7f88cc02a420 comp rx=0 tx=0).stop 2024-05-09T08:50:47.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.032+0000 7f88dfaca640 1 -- 172.21.15.98:0/657568198 shutdown_connections 2024-05-09T08:50:47.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.032+0000 7f88dfaca640 1 --2- 172.21.15.98:0/657568198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88d8101120 0x7f88d810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.032+0000 7f88dfaca640 1 --2- 172.21.15.98:0/657568198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88d8100760 0x7f88d8100be0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.032+0000 7f88dfaca640 1 --2- 172.21.15.98:0/657568198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88d8106720 0x7f88d8106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.033+0000 7f88dfaca640 1 -- 172.21.15.98:0/657568198 >> 172.21.15.98:0/657568198 conn(0x7f88d80778b0 msgr2=0x7f88d8077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.033+0000 7f88dfaca640 1 -- 172.21.15.98:0/657568198 shutdown_connections 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.033+0000 7f88dfaca640 1 -- 172.21.15.98:0/657568198 wait complete. 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.033+0000 7f88dfaca640 1 Processor -- start 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.033+0000 7f88dfaca640 1 -- start start 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.033+0000 7f88dfaca640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88d8100760 0x7f88d8196fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.033+0000 7f88dfaca640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88d8101120 0x7f88d8197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.033+0000 7f88dfaca640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88d8106720 0x7f88d819b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.033+0000 7f88dfaca640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f88d8110b30 con 0x7f88d8100760 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.033+0000 7f88dfaca640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f88d81109b0 con 0x7f88d8101120 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.033+0000 7f88dfaca640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f88d8110cb0 con 0x7f88d8106720 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.033+0000 7f88dd866640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88d8100760 0x7f88d8196fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.033+0000 7f88dd866640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88d8100760 0x7f88d8196fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57194/0 (socket says 172.21.15.98:57194) 2024-05-09T08:50:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.033+0000 7f88dd866640 1 -- 172.21.15.98:0/2871122918 learned_addr learned my addr 172.21.15.98:0/2871122918 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.034+0000 7f88dd065640 1 --2- 172.21.15.98:0/2871122918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88d8101120 0x7f88d8197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.034+0000 7f88dd866640 1 -- 172.21.15.98:0/2871122918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88d8106720 msgr2=0x7f88d819b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.034+0000 7f88dd866640 1 --2- 172.21.15.98:0/2871122918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88d8106720 0x7f88d819b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.034+0000 7f88dd866640 1 -- 172.21.15.98:0/2871122918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88d8101120 msgr2=0x7f88d8197510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.034+0000 7f88dd866640 1 --2- 172.21.15.98:0/2871122918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88d8101120 0x7f88d8197510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.034+0000 7f88dd866640 1 -- 172.21.15.98:0/2871122918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d819bfc0 con 0x7f88d8100760 2024-05-09T08:50:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.034+0000 7f88dd065640 1 --2- 172.21.15.98:0/2871122918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88d8101120 0x7f88d8197510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.034+0000 7f88dd866640 1 --2- 172.21.15.98:0/2871122918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88d8100760 0x7f88d8196fd0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f88c800ea80 tx=0x7f88c800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.034+0000 7f88d6ffd640 1 -- 172.21.15.98:0/2871122918 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c800ce40 con 0x7f88d8100760 2024-05-09T08:50:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.034+0000 7f88d6ffd640 1 -- 172.21.15.98:0/2871122918 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88c8005020 con 0x7f88d8100760 2024-05-09T08:50:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.034+0000 7f88dfaca640 1 -- 172.21.15.98:0/2871122918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d819c2b0 con 0x7f88d8100760 2024-05-09T08:50:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.034+0000 7f88d6ffd640 1 -- 172.21.15.98:0/2871122918 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c8011760 con 0x7f88d8100760 2024-05-09T08:50:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.034+0000 7f88dfaca640 1 -- 172.21.15.98:0/2871122918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d81a3b40 con 0x7f88d8100760 2024-05-09T08:50:47.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.035+0000 7f88dfaca640 1 -- 172.21.15.98:0/2871122918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d804f5f0 con 0x7f88d8100760 2024-05-09T08:50:47.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.036+0000 7f88d6ffd640 1 -- 172.21.15.98:0/2871122918 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f88c80051d0 con 0x7f88d8100760 2024-05-09T08:50:47.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.037+0000 7f88d6ffd640 1 --2- 172.21.15.98:0/2871122918 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f88c00a2f40 0x7f88c00a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:47.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.037+0000 7f88dd065640 1 --2- 172.21.15.98:0/2871122918 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f88c00a2f40 0x7f88c00a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:47.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.037+0000 7f88d6ffd640 1 -- 172.21.15.98:0/2871122918 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f88c80d00c0 con 0x7f88d8100760 2024-05-09T08:50:47.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.037+0000 7f88dd065640 1 --2- 172.21.15.98:0/2871122918 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f88c00a2f40 0x7f88c00a5400 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f88cc000c00 tx=0x7f88cc032c70 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:47.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.040+0000 7f88d6ffd640 1 -- 172.21.15.98:0/2871122918 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f88c8016030 con 0x7f88d8100760 2024-05-09T08:50:47.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.240+0000 7f88dfaca640 1 -- 172.21.15.98:0/2871122918 --> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f88d8061c70 con 0x7f88c00a2f40 2024-05-09T08:50:47.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.300+0000 7f88d6ffd640 1 -- 172.21.15.98:0/2871122918 <== mgr.15555 v2:172.21.15.98:6800/4136638623 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f88d8061c70 con 0x7f88c00a2f40 2024-05-09T08:50:47.300 INFO:teuthology.orchestra.run.smithi098.stdout:Retention added to path / 2024-05-09T08:50:47.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.302+0000 7f88dfaca640 1 -- 172.21.15.98:0/2871122918 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f88c00a2f40 msgr2=0x7f88c00a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.302+0000 7f88dfaca640 1 --2- 172.21.15.98:0/2871122918 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f88c00a2f40 0x7f88c00a5400 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f88cc000c00 tx=0x7f88cc032c70 comp rx=0 tx=0).stop 2024-05-09T08:50:47.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.302+0000 7f88dfaca640 1 -- 172.21.15.98:0/2871122918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88d8100760 msgr2=0x7f88d8196fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.302+0000 7f88dfaca640 1 --2- 172.21.15.98:0/2871122918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88d8100760 0x7f88d8196fd0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f88c800ea80 tx=0x7f88c800ef40 comp rx=0 tx=0).stop 2024-05-09T08:50:47.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.302+0000 7f88dfaca640 1 -- 172.21.15.98:0/2871122918 shutdown_connections 2024-05-09T08:50:47.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.302+0000 7f88dfaca640 1 --2- 172.21.15.98:0/2871122918 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f88c00a2f40 0x7f88c00a5400 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.302+0000 7f88dfaca640 1 --2- 172.21.15.98:0/2871122918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88d8106720 0x7f88d819b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.302+0000 7f88dfaca640 1 --2- 172.21.15.98:0/2871122918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88d8101120 0x7f88d8197510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.302+0000 7f88dfaca640 1 --2- 172.21.15.98:0/2871122918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88d8100760 0x7f88d8196fd0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.302+0000 7f88dfaca640 1 -- 172.21.15.98:0/2871122918 >> 172.21.15.98:0/2871122918 conn(0x7f88d80778b0 msgr2=0x7f88d81032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:47.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.302+0000 7f88dfaca640 1 -- 172.21.15.98:0/2871122918 shutdown_connections 2024-05-09T08:50:47.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.302+0000 7f88dfaca640 1 -- 172.21.15.98:0/2871122918 wait complete. 2024-05-09T08:50:47.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-09T08:50:47.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.415+0000 7f8817348640 1 -- 172.21.15.98:0/2229922164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810106760 msgr2=0x7f8810106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.415+0000 7f8817348640 1 --2- 172.21.15.98:0/2229922164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810106760 0x7f8810106b40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8804009930 tx=0x7f880402a420 comp rx=0 tx=0).stop 2024-05-09T08:50:47.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.415+0000 7f8817348640 1 -- 172.21.15.98:0/2229922164 shutdown_connections 2024-05-09T08:50:47.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.415+0000 7f8817348640 1 --2- 172.21.15.98:0/2229922164 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8810101110 0x7f881010d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.415+0000 7f8817348640 1 --2- 172.21.15.98:0/2229922164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8810100750 0x7f8810100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.415+0000 7f8817348640 1 --2- 172.21.15.98:0/2229922164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810106760 0x7f8810106b40 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.415+0000 7f8817348640 1 -- 172.21.15.98:0/2229922164 >> 172.21.15.98:0/2229922164 conn(0x7f8810076ab0 msgr2=0x7f8810076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:47.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.415+0000 7f8817348640 1 -- 172.21.15.98:0/2229922164 shutdown_connections 2024-05-09T08:50:47.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.415+0000 7f8817348640 1 -- 172.21.15.98:0/2229922164 wait complete. 2024-05-09T08:50:47.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f8817348640 1 Processor -- start 2024-05-09T08:50:47.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f8817348640 1 -- start start 2024-05-09T08:50:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f8817348640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8810100750 0x7f88101980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f8817348640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8810101110 0x7f8810198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f8817348640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810106760 0x7f881019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f8817348640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8810111c70 con 0x7f8810106760 2024-05-09T08:50:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f8817348640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8810111af0 con 0x7f8810101110 2024-05-09T08:50:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f8817348640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8810111df0 con 0x7f8810100750 2024-05-09T08:50:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f88158e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810106760 0x7f881019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f88158e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810106760 0x7f881019c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57210/0 (socket says 172.21.15.98:57210) 2024-05-09T08:50:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f88158e5640 1 -- 172.21.15.98:0/4005026711 learned_addr learned my addr 172.21.15.98:0/4005026711 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f88150e4640 1 --2- 172.21.15.98:0/4005026711 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8810100750 0x7f88101980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f88148e3640 1 --2- 172.21.15.98:0/4005026711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8810101110 0x7f8810198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f88158e5640 1 -- 172.21.15.98:0/4005026711 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8810100750 msgr2=0x7f88101980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f88158e5640 1 --2- 172.21.15.98:0/4005026711 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8810100750 0x7f88101980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f88158e5640 1 -- 172.21.15.98:0/4005026711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8810101110 msgr2=0x7f8810198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f88158e5640 1 --2- 172.21.15.98:0/4005026711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8810101110 0x7f8810198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f88158e5640 1 -- 172.21.15.98:0/4005026711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881019d070 con 0x7f8810106760 2024-05-09T08:50:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.416+0000 7f88158e5640 1 --2- 172.21.15.98:0/4005026711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810106760 0x7f881019c940 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f880800cad0 tx=0x7f880800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.417+0000 7f88027fc640 1 -- 172.21.15.98:0/4005026711 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880800ae20 con 0x7f8810106760 2024-05-09T08:50:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.417+0000 7f88027fc640 1 -- 172.21.15.98:0/4005026711 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8808005020 con 0x7f8810106760 2024-05-09T08:50:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.417+0000 7f8817348640 1 -- 172.21.15.98:0/4005026711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f881019d360 con 0x7f8810106760 2024-05-09T08:50:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.417+0000 7f88027fc640 1 -- 172.21.15.98:0/4005026711 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880800f6f0 con 0x7f8810106760 2024-05-09T08:50:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.417+0000 7f8817348640 1 -- 172.21.15.98:0/4005026711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88101a4bf0 con 0x7f8810106760 2024-05-09T08:50:47.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.418+0000 7f8817348640 1 -- 172.21.15.98:0/4005026711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f881004f5f0 con 0x7f8810106760 2024-05-09T08:50:47.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.419+0000 7f88027fc640 1 -- 172.21.15.98:0/4005026711 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f88080051d0 con 0x7f8810106760 2024-05-09T08:50:47.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.419+0000 7f88027fc640 1 --2- 172.21.15.98:0/4005026711 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f87fc0a2ec0 0x7f87fc0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:47.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.419+0000 7f88150e4640 1 --2- 172.21.15.98:0/4005026711 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f87fc0a2ec0 0x7f87fc0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:47.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.419+0000 7f88027fc640 1 -- 172.21.15.98:0/4005026711 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f88080cfcf0 con 0x7f8810106760 2024-05-09T08:50:47.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.420+0000 7f88150e4640 1 --2- 172.21.15.98:0/4005026711 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f87fc0a2ec0 0x7f87fc0a5380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8804009900 tx=0x7f88040058d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:47.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.424+0000 7f88027fc640 1 -- 172.21.15.98:0/4005026711 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f88080d4070 con 0x7f8810106760 2024-05-09T08:50:47.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:47 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/697287211' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:47.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:47 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:50:47.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:47 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:50:47.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:47 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:50:47.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:47 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:50:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.619+0000 7f8817348640 1 -- 172.21.15.98:0/4005026711 --> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f8810061c70 con 0x7f87fc0a2ec0 2024-05-09T08:50:47.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.627+0000 7f88027fc640 1 -- 172.21.15.98:0/4005026711 <== mgr.15555 v2:172.21.15.98:6800/4136638623 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f8810061c70 con 0x7f87fc0a2ec0 2024-05-09T08:50:47.626 INFO:teuthology.orchestra.run.smithi098.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-09T00:00:00", "created": "2024-05-09T08:50:46", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-05-09T08:50:47.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.629+0000 7f8817348640 1 -- 172.21.15.98:0/4005026711 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f87fc0a2ec0 msgr2=0x7f87fc0a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.629+0000 7f8817348640 1 --2- 172.21.15.98:0/4005026711 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f87fc0a2ec0 0x7f87fc0a5380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8804009900 tx=0x7f88040058d0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.629+0000 7f8817348640 1 -- 172.21.15.98:0/4005026711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810106760 msgr2=0x7f881019c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.629+0000 7f8817348640 1 --2- 172.21.15.98:0/4005026711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810106760 0x7f881019c940 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f880800cad0 tx=0x7f880800cf90 comp rx=0 tx=0).stop 2024-05-09T08:50:47.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.629+0000 7f8817348640 1 -- 172.21.15.98:0/4005026711 shutdown_connections 2024-05-09T08:50:47.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.629+0000 7f8817348640 1 --2- 172.21.15.98:0/4005026711 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f87fc0a2ec0 0x7f87fc0a5380 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.629+0000 7f8817348640 1 --2- 172.21.15.98:0/4005026711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810106760 0x7f881019c940 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.629+0000 7f8817348640 1 --2- 172.21.15.98:0/4005026711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8810101110 0x7f8810198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.629+0000 7f8817348640 1 --2- 172.21.15.98:0/4005026711 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8810100750 0x7f88101980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.629+0000 7f8817348640 1 -- 172.21.15.98:0/4005026711 >> 172.21.15.98:0/4005026711 conn(0x7f8810076ab0 msgr2=0x7f88100fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:47.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.630+0000 7f8817348640 1 -- 172.21.15.98:0/4005026711 shutdown_connections 2024-05-09T08:50:47.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.630+0000 7f8817348640 1 -- 172.21.15.98:0/4005026711 wait complete. 2024-05-09T08:50:47.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-09T08:50:47.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:47 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/697287211' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:47.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:47 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:50:47.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:47 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:50:47.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:47 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:50:47.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:47 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:50:47.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:47 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/697287211' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:50:47.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:47 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:50:47.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:47 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:50:47.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:47 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:50:47.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:47 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:50:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.742+0000 7f32aeddc640 1 -- 172.21.15.98:0/4182691599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8106720 msgr2=0x7f32a8106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.742+0000 7f32aeddc640 1 --2- 172.21.15.98:0/4182691599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8106720 0x7f32a8106b00 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f3298009930 tx=0x7f329802a420 comp rx=0 tx=0).stop 2024-05-09T08:50:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.742+0000 7f32aeddc640 1 -- 172.21.15.98:0/4182691599 shutdown_connections 2024-05-09T08:50:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.742+0000 7f32aeddc640 1 --2- 172.21.15.98:0/4182691599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32a8101120 0x7f32a810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.742+0000 7f32aeddc640 1 --2- 172.21.15.98:0/4182691599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32a8100760 0x7f32a8100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.742+0000 7f32aeddc640 1 --2- 172.21.15.98:0/4182691599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8106720 0x7f32a8106b00 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.742+0000 7f32aeddc640 1 -- 172.21.15.98:0/4182691599 >> 172.21.15.98:0/4182691599 conn(0x7f32a80778b0 msgr2=0x7f32a8077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.742+0000 7f32aeddc640 1 -- 172.21.15.98:0/4182691599 shutdown_connections 2024-05-09T08:50:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.742+0000 7f32aeddc640 1 -- 172.21.15.98:0/4182691599 wait complete. 2024-05-09T08:50:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.743+0000 7f32aeddc640 1 Processor -- start 2024-05-09T08:50:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.743+0000 7f32aeddc640 1 -- start start 2024-05-09T08:50:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.743+0000 7f32aeddc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8100760 0x7f32a81980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.743+0000 7f32aeddc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32a8101120 0x7f32a8198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.743+0000 7f32aeddc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32a8106720 0x7f32a819c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.743+0000 7f32aeddc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f32a8110b30 con 0x7f32a8100760 2024-05-09T08:50:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.743+0000 7f32aeddc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f32a81109b0 con 0x7f32a8101120 2024-05-09T08:50:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.743+0000 7f32aeddc640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f32a8110cb0 con 0x7f32a8106720 2024-05-09T08:50:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.743+0000 7f32acb78640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8100760 0x7f32a81980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.743+0000 7f32acb78640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8100760 0x7f32a81980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57242/0 (socket says 172.21.15.98:57242) 2024-05-09T08:50:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.743+0000 7f32acb78640 1 -- 172.21.15.98:0/81685452 learned_addr learned my addr 172.21.15.98:0/81685452 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:50:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.743+0000 7f32a7fff640 1 --2- 172.21.15.98:0/81685452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32a8101120 0x7f32a8198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.744+0000 7f32acb78640 1 -- 172.21.15.98:0/81685452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32a8106720 msgr2=0x7f32a819c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.744+0000 7f32acb78640 1 --2- 172.21.15.98:0/81685452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32a8106720 0x7f32a819c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.744+0000 7f32acb78640 1 -- 172.21.15.98:0/81685452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32a8101120 msgr2=0x7f32a8198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.744+0000 7f32acb78640 1 --2- 172.21.15.98:0/81685452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32a8101120 0x7f32a8198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.744+0000 7f32acb78640 1 -- 172.21.15.98:0/81685452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a819d0e0 con 0x7f32a8100760 2024-05-09T08:50:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.744+0000 7f32a7fff640 1 --2- 172.21.15.98:0/81685452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32a8101120 0x7f32a8198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:50:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.744+0000 7f32acb78640 1 --2- 172.21.15.98:0/81685452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8100760 0x7f32a81980f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3298009230 tx=0x7f3298004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.744+0000 7f32a5ffb640 1 -- 172.21.15.98:0/81685452 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329803f030 con 0x7f32a8100760 2024-05-09T08:50:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.744+0000 7f32a5ffb640 1 -- 172.21.15.98:0/81685452 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3298003ad0 con 0x7f32a8100760 2024-05-09T08:50:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.744+0000 7f32aeddc640 1 -- 172.21.15.98:0/81685452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32a819d370 con 0x7f32a8100760 2024-05-09T08:50:47.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.744+0000 7f32a5ffb640 1 -- 172.21.15.98:0/81685452 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3298035070 con 0x7f32a8100760 2024-05-09T08:50:47.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.744+0000 7f32aeddc640 1 -- 172.21.15.98:0/81685452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32a81a4c60 con 0x7f32a8100760 2024-05-09T08:50:47.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.745+0000 7f32aeddc640 1 -- 172.21.15.98:0/81685452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32a804f5f0 con 0x7f32a8100760 2024-05-09T08:50:47.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.746+0000 7f32a5ffb640 1 -- 172.21.15.98:0/81685452 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f3298042030 con 0x7f32a8100760 2024-05-09T08:50:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.747+0000 7f32a5ffb640 1 --2- 172.21.15.98:0/81685452 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f32900a2ec0 0x7f32900a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:50:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.747+0000 7f32a7fff640 1 --2- 172.21.15.98:0/81685452 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f32900a2ec0 0x7f32900a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:50:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.747+0000 7f32a5ffb640 1 -- 172.21.15.98:0/81685452 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f32980effa0 con 0x7f32a8100760 2024-05-09T08:50:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.747+0000 7f32a7fff640 1 --2- 172.21.15.98:0/81685452 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f32900a2ec0 0x7f32900a5380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f32a8077d40 tx=0x7f329c005d90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:50:47.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.751+0000 7f32a5ffb640 1 -- 172.21.15.98:0/81685452 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f32980bc680 con 0x7f32a8100760 2024-05-09T08:50:47.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.949+0000 7f32aeddc640 1 -- 172.21.15.98:0/81685452 --> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f32a8061c70 con 0x7f32900a2ec0 2024-05-09T08:50:47.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.957+0000 7f32a5ffb640 1 -- 172.21.15.98:0/81685452 <== mgr.15555 v2:172.21.15.98:6800/4136638623 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f32a8061c70 con 0x7f32900a2ec0 2024-05-09T08:50:47.956 INFO:teuthology.orchestra.run.smithi098.stdout:/ 1m 6m3h 2024-05-09T08:50:47.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.959+0000 7f32aeddc640 1 -- 172.21.15.98:0/81685452 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f32900a2ec0 msgr2=0x7f32900a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.959+0000 7f32aeddc640 1 --2- 172.21.15.98:0/81685452 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f32900a2ec0 0x7f32900a5380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f32a8077d40 tx=0x7f329c005d90 comp rx=0 tx=0).stop 2024-05-09T08:50:47.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.959+0000 7f32aeddc640 1 -- 172.21.15.98:0/81685452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8100760 msgr2=0x7f32a81980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:50:47.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.959+0000 7f32aeddc640 1 --2- 172.21.15.98:0/81685452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8100760 0x7f32a81980f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3298009230 tx=0x7f3298004270 comp rx=0 tx=0).stop 2024-05-09T08:50:47.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.959+0000 7f32aeddc640 1 -- 172.21.15.98:0/81685452 shutdown_connections 2024-05-09T08:50:47.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.959+0000 7f32aeddc640 1 --2- 172.21.15.98:0/81685452 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f32900a2ec0 0x7f32900a5380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.959+0000 7f32aeddc640 1 --2- 172.21.15.98:0/81685452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32a8106720 0x7f32a819c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.959+0000 7f32aeddc640 1 --2- 172.21.15.98:0/81685452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32a8101120 0x7f32a8198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.959+0000 7f32aeddc640 1 --2- 172.21.15.98:0/81685452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8100760 0x7f32a81980f0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:50:47.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.959+0000 7f32aeddc640 1 -- 172.21.15.98:0/81685452 >> 172.21.15.98:0/81685452 conn(0x7f32a80778b0 msgr2=0x7f32a8069160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:50:47.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.959+0000 7f32aeddc640 1 -- 172.21.15.98:0/81685452 shutdown_connections 2024-05-09T08:50:47.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:50:47.959+0000 7f32aeddc640 1 -- 172.21.15.98:0/81685452 wait complete. 2024-05-09T08:50:47.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-05-09T08:50:47.995 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-05-09T08:50:48.006 INFO:teuthology.run_tasks:Running task background_exec... 2024-05-09T08:50:48.016 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-05-09T08:50:48.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-05-09T08:50:48.038 INFO:teuthology.run_tasks:Running task workunit... 2024-05-09T08:50:48.053 INFO:tasks.workunit:Pulling workunits from ref be85c202969d25af6e130678e7d73a20e9c4ca11 2024-05-09T08:50:48.054 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-09T08:50:48.054 INFO:tasks.workunit:timeout=3h 2024-05-09T08:50:48.054 INFO:tasks.workunit:cleanup=True 2024-05-09T08:50:48.055 DEBUG:teuthology.orchestra.run.smithi098:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-09T08:50:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-09T08:50:48.071 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-05-09T08:50:48.071 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 72h/114d Inode: 1099511628283 Links: 2 2024-05-09T08:50:48.071 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:50:48.071 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-05-09T08:50:48.071 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-05-09 08:48:39.804327853 +0000 2024-05-09T08:50:48.071 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-05-09 08:48:39.804327853 +0000 2024-05-09T08:50:48.071 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-05-09 08:50:36.484437812 +0000 2024-05-09T08:50:48.071 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-05-09 08:48:39.804327853 +0000 2024-05-09T08:50:48.072 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-09T08:50:48.072 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-09T08:50:48.145 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout be85c202969d25af6e130678e7d73a20e9c4ca11 2024-05-09T08:50:48.201 INFO:tasks.workunit.client.0.smithi098.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-09T08:50:48.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:48 smithi098 ceph-mon[30092]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.3 KiB/s wr, 26 op/s 2024-05-09T08:50:48.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:48 smithi098 ceph-mon[30092]: from='client.15768 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:48.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:48 smithi098 ceph-mon[30092]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:48.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:48 smithi142 ceph-mon[34625]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.3 KiB/s wr, 26 op/s 2024-05-09T08:50:48.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:48 smithi142 ceph-mon[34625]: from='client.15768 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:48.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:48 smithi142 ceph-mon[34625]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:48.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:48 smithi105 ceph-mon[34498]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.3 KiB/s wr, 26 op/s 2024-05-09T08:50:48.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:48 smithi105 ceph-mon[34498]: from='client.15768 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:48.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:48 smithi105 ceph-mon[34498]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:49.563 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:49 smithi098 ceph-mon[30092]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:49.564 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:49 smithi098 ceph-mon[30092]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:49.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:49 smithi142 ceph-mon[34625]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:49.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:49 smithi142 ceph-mon[34625]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:49.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:49 smithi105 ceph-mon[34498]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:49.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:49 smithi105 ceph-mon[34498]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:50:50.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:50 smithi098 ceph-mon[30092]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.0 KiB/s wr, 20 op/s 2024-05-09T08:50:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:50 smithi142 ceph-mon[34625]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.0 KiB/s wr, 20 op/s 2024-05-09T08:50:50.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:50 smithi105 ceph-mon[34498]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.0 KiB/s wr, 20 op/s 2024-05-09T08:50:52.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:52 smithi098 ceph-mon[30092]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 85 KiB/s wr, 56 op/s 2024-05-09T08:50:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:52 smithi142 ceph-mon[34625]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 85 KiB/s wr, 56 op/s 2024-05-09T08:50:52.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:52 smithi105 ceph-mon[34498]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 85 KiB/s wr, 56 op/s 2024-05-09T08:50:54.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:54 smithi098 ceph-mon[30092]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-05-09T08:50:54.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:54 smithi142 ceph-mon[34625]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-05-09T08:50:54.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:54 smithi105 ceph-mon[34498]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-05-09T08:50:56.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:56 smithi098 ceph-mon[30092]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-05-09T08:50:56.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:56 smithi142 ceph-mon[34625]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-05-09T08:50:56.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:56 smithi105 ceph-mon[34498]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-05-09T08:50:58.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:50:58 smithi098 ceph-mon[30092]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 55 op/s 2024-05-09T08:50:58.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:50:58 smithi142 ceph-mon[34625]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 55 op/s 2024-05-09T08:50:58.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:50:58 smithi105 ceph-mon[34498]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 55 op/s 2024-05-09T08:51:00.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:00 smithi105 ceph-mon[34498]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-09T08:51:00.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:00 smithi098 ceph-mon[30092]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-09T08:51:00.573 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:51:00 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:51:00.313+0000 7f8c54d30700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:51:00.574 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:51:00 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:51:00.313+0000 7f8c54d30700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:51:00.574 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:51:00 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:51:00.313+0000 7f8c54d30700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:51:00.574 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:51:00 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:51:00.313+0000 7f8c54d30700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:51:00.574 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 08:51:00 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x[30326]: 2024-05-09T08:51:00.313+0000 7f8c54d30700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:51:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:00 smithi142 ceph-mon[34625]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-09T08:51:01.173 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 -- 172.21.15.98:0/2180499973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09400fffa0 msgr2=0x7f0940100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:01.173 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 --2- 172.21.15.98:0/2180499973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09400fffa0 0x7f0940100380 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0930009900 tx=0x7f093002a410 comp rx=0 tx=0).stop 2024-05-09T08:51:01.173 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 -- 172.21.15.98:0/2180499973 shutdown_connections 2024-05-09T08:51:01.173 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 --2- 172.21.15.98:0/2180499973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0940106b60 0x7f094010e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.173 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 --2- 172.21.15.98:0/2180499973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0940100950 0x7f0940106480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 --2- 172.21.15.98:0/2180499973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09400fffa0 0x7f0940100380 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 -- 172.21.15.98:0/2180499973 >> 172.21.15.98:0/2180499973 conn(0x7f0940076cc0 msgr2=0x7f09400fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 -- 172.21.15.98:0/2180499973 shutdown_connections 2024-05-09T08:51:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 -- 172.21.15.98:0/2180499973 wait complete. 2024-05-09T08:51:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 Processor -- start 2024-05-09T08:51:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 -- start start 2024-05-09T08:51:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09400fffa0 0x7f09401980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0940106b60 0x7f0940198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f094019c960 0x7f094019ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0940111c70 con 0x7f0940106b60 2024-05-09T08:51:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0940111af0 con 0x7f094019c960 2024-05-09T08:51:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.172+0000 7f0945b5a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0940111df0 con 0x7f09400fffa0 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f0937fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0940106b60 0x7f0940198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f0937fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0940106b60 0x7f0940198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54380/0 (socket says 172.21.15.98:54380) 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f0937fff640 1 -- 172.21.15.98:0/1720158064 learned_addr learned my addr 172.21.15.98:0/1720158064 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f093ffff640 1 --2- 172.21.15.98:0/1720158064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f094019c960 0x7f094019ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f093f7fe640 1 --2- 172.21.15.98:0/1720158064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09400fffa0 0x7f09401980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f0937fff640 1 -- 172.21.15.98:0/1720158064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09400fffa0 msgr2=0x7f09401980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f0937fff640 1 --2- 172.21.15.98:0/1720158064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09400fffa0 0x7f09401980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f0937fff640 1 -- 172.21.15.98:0/1720158064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f094019c960 msgr2=0x7f094019ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f0937fff640 1 --2- 172.21.15.98:0/1720158064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f094019c960 0x7f094019ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f0937fff640 1 -- 172.21.15.98:0/1720158064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094019d450 con 0x7f0940106b60 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f0937fff640 1 --2- 172.21.15.98:0/1720158064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0940106b60 0x7f0940198630 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f092800d770 tx=0x7f092800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f093d7fa640 1 -- 172.21.15.98:0/1720158064 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0928017070 con 0x7f0940106b60 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f093d7fa640 1 -- 172.21.15.98:0/1720158064 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0928009c30 con 0x7f0940106b60 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f0945b5a640 1 -- 172.21.15.98:0/1720158064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f094019d740 con 0x7f0940106b60 2024-05-09T08:51:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f093d7fa640 1 -- 172.21.15.98:0/1720158064 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092800f620 con 0x7f0940106b60 2024-05-09T08:51:01.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.173+0000 7f0945b5a640 1 -- 172.21.15.98:0/1720158064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09401a4f30 con 0x7f0940106b60 2024-05-09T08:51:01.182 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.176+0000 7f093d7fa640 1 -- 172.21.15.98:0/1720158064 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f092801a070 con 0x7f0940106b60 2024-05-09T08:51:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.176+0000 7f0948867740 1 Processor -- start 2024-05-09T08:51:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.176+0000 7f0948867740 1 -- start start 2024-05-09T08:51:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.176+0000 7f0948867740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737350 0x563803737730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0948867740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803737c70 0x563803746500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0948867740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803746a40 0x563803748e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0948867740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56380369c340 con 0x563803746a40 2024-05-09T08:51:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0948867740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563803644290 con 0x563803737350 2024-05-09T08:51:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0948867740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563803698300 con 0x563803737c70 2024-05-09T08:51:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0937fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803746a40 0x563803748e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f093ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737350 0x563803737730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0937fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803746a40 0x563803748e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54392/0 (socket says 172.21.15.98:54392) 2024-05-09T08:51:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0937fff640 1 -- 172.21.15.98:0/1484266445 learned_addr learned my addr 172.21.15.98:0/1484266445 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f093d7fa640 1 --2- 172.21.15.98:0/1720158064 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f092c0a2ec0 0x7f092c0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f093f7fe640 1 --2- 172.21.15.98:0/1484266445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803737c70 0x563803746500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f093ffff640 1 --2- 172.21.15.98:0/1720158064 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f092c0a2ec0 0x7f092c0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f093d7fa640 1 -- 172.21.15.98:0/1720158064 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f09280cf5d0 con 0x7f0940106b60 2024-05-09T08:51:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0937fff640 1 -- 172.21.15.98:0/1484266445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803737c70 msgr2=0x563803746500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0937fff640 1 --2- 172.21.15.98:0/1484266445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803737c70 0x563803746500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0937fff640 1 -- 172.21.15.98:0/1484266445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737350 msgr2=0x563803737730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0937fff640 1 --2- 172.21.15.98:0/1484266445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737350 0x563803737730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0937fff640 1 -- 172.21.15.98:0/1484266445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638036688e0 con 0x563803746a40 2024-05-09T08:51:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f093ffff640 1 --2- 172.21.15.98:0/1484266445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737350 0x563803737730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0937fff640 1 --2- 172.21.15.98:0/1484266445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803746a40 0x563803748e30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f09280cc870 tx=0x7f0928010cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b65d25ba63c105 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f093ffff640 1 --2- 172.21.15.98:0/1720158064 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f092c0a2ec0 0x7f092c0a5380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0938009fd0 tx=0x7f0938004a40 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0935ffb640 1 -- 172.21.15.98:0/1484266445 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0928021070 con 0x563803746a40 2024-05-09T08:51:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0935ffb640 1 -- 172.21.15.98:0/1484266445 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0928009de0 con 0x563803746a40 2024-05-09T08:51:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.177+0000 7f0935ffb640 1 -- 172.21.15.98:0/1484266445 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092809b740 con 0x563803746a40 2024-05-09T08:51:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.178+0000 7f0948867740 1 -- 172.21.15.98:0/1484266445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803746a40 msgr2=0x563803748e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.178+0000 7f0948867740 1 --2- 172.21.15.98:0/1484266445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803746a40 0x563803748e30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f09280cc870 tx=0x7f0928010cd0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.178+0000 7f0948867740 1 -- 172.21.15.98:0/1484266445 shutdown_connections 2024-05-09T08:51:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.178+0000 7f0948867740 1 --2- 172.21.15.98:0/1484266445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803746a40 0x563803748e30 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.178+0000 7f0948867740 1 --2- 172.21.15.98:0/1484266445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803737c70 0x563803746500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.178+0000 7f0948867740 1 --2- 172.21.15.98:0/1484266445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737350 0x563803737730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.178+0000 7f0948867740 1 -- 172.21.15.98:0/1484266445 >> 172.21.15.98:0/1484266445 conn(0x563803733960 msgr2=0x563803733d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.178+0000 7f0948867740 1 -- 172.21.15.98:0/1484266445 shutdown_connections 2024-05-09T08:51:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.178+0000 7f0948867740 1 -- 172.21.15.98:0/1484266445 wait complete. 2024-05-09T08:51:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.178+0000 7f0948867740 1 -- start start 2024-05-09T08:51:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f0948867740 1 Processor -- start 2024-05-09T08:51:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f0948867740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737350 0x5638037438f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f0948867740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737c70 0x563803743e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f0948867740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803746a40 0x563803744370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f0948867740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563803749610 con 0x563803737350 2024-05-09T08:51:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f0948867740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563803657790 con 0x563803737c70 2024-05-09T08:51:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f0948867740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563803749790 con 0x563803746a40 2024-05-09T08:51:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f0948867740 10 client.0 _ll_register_callbacks cb 0x563803692a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f093f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737350 0x5638037438f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f093f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737350 0x5638037438f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54394/0 (socket says 172.21.15.98:54394) 2024-05-09T08:51:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f093f7fe640 1 -- 172.21.15.98:0/2283680786 learned_addr learned my addr 172.21.15.98:0/2283680786 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f0937fff640 1 --2- 172.21.15.98:0/2283680786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737c70 0x563803743e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f093ffff640 1 --2- 172.21.15.98:0/2283680786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803746a40 0x563803744370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f093f7fe640 1 -- 172.21.15.98:0/2283680786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803746a40 msgr2=0x563803744370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f093f7fe640 1 --2- 172.21.15.98:0/2283680786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803746a40 0x563803744370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f093f7fe640 1 -- 172.21.15.98:0/2283680786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737c70 msgr2=0x563803743e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f093f7fe640 1 --2- 172.21.15.98:0/2283680786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737c70 0x563803743e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f093f7fe640 1 -- 172.21.15.98:0/2283680786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638033b0210 con 0x563803737350 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f093f7fe640 1 --2- 172.21.15.98:0/2283680786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737350 0x5638037438f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0930009900 tx=0x7f09300319b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f6419ff2c62c2e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.179+0000 7f091effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f091effd640 1 -- 172.21.15.98:0/2283680786 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0930035040 con 0x563803737350 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f091effd640 1 -- 172.21.15.98:0/2283680786 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f0930004240 con 0x563803737350 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f0948867740 1 -- 172.21.15.98:0/2283680786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0928009de0 con 0x563803737350 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f091effd640 1 -- 172.21.15.98:0/2283680786 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0930004440 con 0x563803737350 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f0948867740 10 client.15837 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f0948867740 1 -- 172.21.15.98:0/2283680786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56380347c620 con 0x563803737350 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f091effd640 1 -- 172.21.15.98:0/2283680786 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0928009de0 con 0x563803737350 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f091effd640 1 -- 172.21.15.98:0/2283680786 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0930004b80 con 0x563803737350 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f0948867740 10 client.15837 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f0948867740 10 client.15837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f0948867740 20 client.15837 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f0948867740 1 --2- 172.21.15.98:0/2283680786 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638037566e0 0x563803758ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f0948867740 4 client.15837 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f0948867740 1 -- 172.21.15.98:0/2283680786 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638036a9a70 con 0x5638037566e0 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.180+0000 7f0937fff640 1 --2- 172.21.15.98:0/2283680786 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638037566e0 0x563803758ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.181+0000 7f091effd640 10 client.15837 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.181+0000 7f0937fff640 1 --2- 172.21.15.98:0/2283680786 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638037566e0 0x563803758ad0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24ef70fb3407c2bd server_cookie=b57391a416f1ccb8 in_seq=0 out_seq=0 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.181+0000 7f091effd640 1 -- 172.21.15.98:0/2283680786 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5638036a9a70 con 0x5638037566e0 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.181+0000 7f091effd640 10 client.15837 handle_command_reply: tid=0 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.181+0000 7f0948867740 1 client.15837 shutdown 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.182+0000 7f0948867740 1 -- 172.21.15.98:0/2283680786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737350 msgr2=0x5638037438f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.182+0000 7f0948867740 1 --2- 172.21.15.98:0/2283680786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737350 0x5638037438f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0930009900 tx=0x7f09300319b0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.182+0000 7f0948867740 1 -- 172.21.15.98:0/2283680786 shutdown_connections 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.182+0000 7f0948867740 1 --2- 172.21.15.98:0/2283680786 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638037566e0 0x563803758ad0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.182+0000 7f0948867740 1 --2- 172.21.15.98:0/2283680786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803746a40 0x563803744370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.182+0000 7f091effd640 0 client.15837 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.182+0000 7f0948867740 1 --2- 172.21.15.98:0/2283680786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737c70 0x563803743e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.182+0000 7f0948867740 1 --2- 172.21.15.98:0/2283680786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737350 0x5638037438f0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.182+0000 7f0948867740 1 -- 172.21.15.98:0/2283680786 >> 172.21.15.98:0/2283680786 conn(0x563803733960 msgr2=0x563803739730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.182+0000 7f0948867740 1 -- 172.21.15.98:0/2283680786 shutdown_connections 2024-05-09T08:51:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.182+0000 7f0948867740 1 -- 172.21.15.98:0/2283680786 wait complete. 2024-05-09T08:51:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.182+0000 7f0948867740 20 client.15837 trim_cache size 0 max 16384 2024-05-09T08:51:01.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.196+0000 7f0948867740 1 Processor -- start 2024-05-09T08:51:01.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.196+0000 7f0948867740 1 -- start start 2024-05-09T08:51:01.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.196+0000 7f0948867740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737350 0x563803746020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.196+0000 7f0948867740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737c70 0x5638037524d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.196+0000 7f0948867740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803746a40 0x563803742ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.196+0000 7f0948867740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09401acaf0 con 0x563803737c70 2024-05-09T08:51:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.196+0000 7f0948867740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f09401ae630 con 0x563803737350 2024-05-09T08:51:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.196+0000 7f0948867740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f09401ad1c0 con 0x563803746a40 2024-05-09T08:51:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.196+0000 7f0937fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737c70 0x5638037524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.196+0000 7f0937fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737c70 0x5638037524d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54404/0 (socket says 172.21.15.98:54404) 2024-05-09T08:51:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.196+0000 7f0937fff640 1 -- 172.21.15.98:0/1732803424 learned_addr learned my addr 172.21.15.98:0/1732803424 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f093f7fe640 1 --2- 172.21.15.98:0/1732803424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737350 0x563803746020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f0937fff640 1 -- 172.21.15.98:0/1732803424 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803746a40 msgr2=0x563803742ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f0937fff640 1 --2- 172.21.15.98:0/1732803424 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803746a40 0x563803742ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f0937fff640 1 -- 172.21.15.98:0/1732803424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737350 msgr2=0x563803746020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f0937fff640 1 --2- 172.21.15.98:0/1732803424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737350 0x563803746020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f0937fff640 1 -- 172.21.15.98:0/1732803424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563803737aa0 con 0x563803737c70 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f0937fff640 1 --2- 172.21.15.98:0/1732803424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737c70 0x5638037524d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0928097eb0 tx=0x7f092809b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6309ed30b5400f5a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f09357fa640 1 -- 172.21.15.98:0/1732803424 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0928014070 con 0x563803737c70 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f09357fa640 1 -- 172.21.15.98:0/1732803424 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f092809f890 con 0x563803737c70 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f09357fa640 1 -- 172.21.15.98:0/1732803424 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0928021070 con 0x563803737c70 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f0948867740 1 -- 172.21.15.98:0/1732803424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737c70 msgr2=0x5638037524d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f0948867740 1 --2- 172.21.15.98:0/1732803424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737c70 0x5638037524d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0928097eb0 tx=0x7f092809b620 comp rx=0 tx=0).stop 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f0948867740 1 -- 172.21.15.98:0/1732803424 shutdown_connections 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f0948867740 1 --2- 172.21.15.98:0/1732803424 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803746a40 0x563803742ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f0948867740 1 --2- 172.21.15.98:0/1732803424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737c70 0x5638037524d0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f0948867740 1 --2- 172.21.15.98:0/1732803424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803737350 0x563803746020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f0948867740 1 -- 172.21.15.98:0/1732803424 >> 172.21.15.98:0/1732803424 conn(0x563803733960 msgr2=0x563803732170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.197+0000 7f0948867740 1 -- 172.21.15.98:0/1732803424 shutdown_connections 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.198+0000 7f0948867740 1 -- 172.21.15.98:0/1732803424 wait complete. 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.198+0000 7f0948867740 1 -- start start 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.198+0000 7f0948867740 1 Processor -- start 2024-05-09T08:51:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.198+0000 7f0948867740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803737350 0x56380374ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.198+0000 7f0948867740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737c70 0x5638037404f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.198+0000 7f0948867740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803746a40 0x563803740c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.198+0000 7f0948867740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09401adf60 con 0x563803737c70 2024-05-09T08:51:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.198+0000 7f0948867740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f09401ad890 con 0x563803746a40 2024-05-09T08:51:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.198+0000 7f0948867740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f09401aaad0 con 0x563803737350 2024-05-09T08:51:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.198+0000 7f0948867740 10 client.0 _ll_register_callbacks cb 0x7f09300314d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.198+0000 7f0937fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737c70 0x5638037404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.198+0000 7f0937fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737c70 0x5638037404f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54420/0 (socket says 172.21.15.98:54420) 2024-05-09T08:51:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.198+0000 7f0937fff640 1 -- 172.21.15.98:0/973791640 learned_addr learned my addr 172.21.15.98:0/973791640 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.198+0000 7f093ffff640 1 --2- 172.21.15.98:0/973791640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803746a40 0x563803740c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0937fff640 1 -- 172.21.15.98:0/973791640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803737350 msgr2=0x56380374ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0937fff640 1 --2- 172.21.15.98:0/973791640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803737350 0x56380374ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0937fff640 1 -- 172.21.15.98:0/973791640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803746a40 msgr2=0x563803740c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0937fff640 1 --2- 172.21.15.98:0/973791640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803746a40 0x563803740c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0937fff640 1 -- 172.21.15.98:0/973791640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638037bbc50 con 0x563803737c70 2024-05-09T08:51:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f093ffff640 1 --2- 172.21.15.98:0/973791640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803746a40 0x563803740c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0937fff640 1 --2- 172.21.15.98:0/973791640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737c70 0x5638037404f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0928009610 tx=0x7f0928009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca03779a2b38439 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f091effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f091effd640 1 -- 172.21.15.98:0/973791640 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0928021030 con 0x563803737c70 2024-05-09T08:51:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f091effd640 1 -- 172.21.15.98:0/973791640 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f092809b740 con 0x563803737c70 2024-05-09T08:51:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0948867740 1 -- 172.21.15.98:0/973791640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5638036a3dc0 con 0x563803737c70 2024-05-09T08:51:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f091effd640 1 -- 172.21.15.98:0/973791640 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0928014070 con 0x563803737c70 2024-05-09T08:51:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0948867740 10 client.15843 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0948867740 1 -- 172.21.15.98:0/973791640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5638037bce00 con 0x563803737c70 2024-05-09T08:51:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f091effd640 1 -- 172.21.15.98:0/973791640 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5638036a3dc0 con 0x563803737c70 2024-05-09T08:51:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f091effd640 1 -- 172.21.15.98:0/973791640 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f09280a3cc0 con 0x563803737c70 2024-05-09T08:51:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0948867740 10 client.15843 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0948867740 10 client.15843 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0948867740 20 client.15843 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0948867740 1 --2- 172.21.15.98:0/973791640 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638037566e0 0x5638037bf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0948867740 4 client.15843 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:51:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f0948867740 1 -- 172.21.15.98:0/973791640 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5638036218e0 con 0x5638037566e0 2024-05-09T08:51:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.199+0000 7f093f7fe640 1 --2- 172.21.15.98:0/973791640 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638037566e0 0x5638037bf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.200+0000 7f093f7fe640 1 --2- 172.21.15.98:0/973791640 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638037566e0 0x5638037bf090 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=892f7a6c73614120 server_cookie=9d0664e3297c00e0 in_seq=0 out_seq=0 2024-05-09T08:51:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.200+0000 7f091effd640 10 client.15843 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:01.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.211+0000 7f091effd640 1 -- 172.21.15.98:0/973791640 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5638036218e0 con 0x5638037566e0 2024-05-09T08:51:01.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.211+0000 7f091effd640 10 client.15843 handle_command_reply: tid=0 2024-05-09T08:51:01.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.211+0000 7f0948867740 1 client.15843 shutdown 2024-05-09T08:51:01.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.211+0000 7f0948867740 1 -- 172.21.15.98:0/973791640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737c70 msgr2=0x5638037404f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:01.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.211+0000 7f0948867740 1 --2- 172.21.15.98:0/973791640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737c70 0x5638037404f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0928009610 tx=0x7f0928009e70 comp rx=0 tx=0).stop 2024-05-09T08:51:01.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0948867740 1 -- 172.21.15.98:0/973791640 shutdown_connections 2024-05-09T08:51:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0948867740 1 --2- 172.21.15.98:0/973791640 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638037566e0 0x5638037bf090 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0948867740 1 --2- 172.21.15.98:0/973791640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803746a40 0x563803740c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0948867740 1 --2- 172.21.15.98:0/973791640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563803737c70 0x5638037404f0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0948867740 1 --2- 172.21.15.98:0/973791640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563803737350 0x56380374ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f091effd640 0 client.15843 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0948867740 1 -- 172.21.15.98:0/973791640 >> 172.21.15.98:0/973791640 conn(0x563803733960 msgr2=0x563803732a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0948867740 1 -- 172.21.15.98:0/973791640 shutdown_connections 2024-05-09T08:51:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0948867740 1 -- 172.21.15.98:0/973791640 wait complete. 2024-05-09T08:51:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0948867740 20 client.15843 trim_cache size 0 max 16384 2024-05-09T08:51:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:51:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:51:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:51:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:51:01.213 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0945b5a640 1 -- 172.21.15.98:0/1720158064 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f092c0a2ec0 msgr2=0x7f092c0a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:01.213 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0945b5a640 1 --2- 172.21.15.98:0/1720158064 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f092c0a2ec0 0x7f092c0a5380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0938009fd0 tx=0x7f0938004a40 comp rx=0 tx=0).stop 2024-05-09T08:51:01.213 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0945b5a640 1 -- 172.21.15.98:0/1720158064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0940106b60 msgr2=0x7f0940198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:01.213 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0945b5a640 1 --2- 172.21.15.98:0/1720158064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0940106b60 0x7f0940198630 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f092800d770 tx=0x7f092800dc30 comp rx=0 tx=0).stop 2024-05-09T08:51:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0945b5a640 1 -- 172.21.15.98:0/1720158064 shutdown_connections 2024-05-09T08:51:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0945b5a640 1 --2- 172.21.15.98:0/1720158064 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f092c0a2ec0 0x7f092c0a5380 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0945b5a640 1 --2- 172.21.15.98:0/1720158064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f094019c960 0x7f094019ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0945b5a640 1 --2- 172.21.15.98:0/1720158064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0940106b60 0x7f0940198630 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0945b5a640 1 --2- 172.21.15.98:0/1720158064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09400fffa0 0x7f09401980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0945b5a640 1 -- 172.21.15.98:0/1720158064 >> 172.21.15.98:0/1720158064 conn(0x7f0940076cc0 msgr2=0x7f09400fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0945b5a640 1 -- 172.21.15.98:0/1720158064 shutdown_connections 2024-05-09T08:51:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:01.212+0000 7f0945b5a640 1 -- 172.21.15.98:0/1720158064 wait complete. 2024-05-09T08:51:01.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:01 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:51:01.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:01 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:51:01.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:01 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:51:02.590 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:02 smithi098 ceph-mon[30092]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-05-09T08:51:02.590 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:02 smithi098 ceph-mon[30092]: mgrmap e25: x(active, since 22s), standbys: z, y 2024-05-09T08:51:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:02 smithi142 ceph-mon[34625]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-05-09T08:51:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:02 smithi142 ceph-mon[34625]: mgrmap e25: x(active, since 22s), standbys: z, y 2024-05-09T08:51:02.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:02 smithi105 ceph-mon[34498]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-05-09T08:51:02.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:02 smithi105 ceph-mon[34498]: mgrmap e25: x(active, since 22s), standbys: z, y 2024-05-09T08:51:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:04 smithi098 ceph-mon[30092]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:51:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:04 smithi142 ceph-mon[34625]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:51:04.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:04 smithi105 ceph-mon[34498]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:51:06.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:06 smithi098 ceph-mon[30092]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-09T08:51:06.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:06 smithi142 ceph-mon[34625]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-09T08:51:06.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:06 smithi105 ceph-mon[34498]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-09T08:51:08.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:08 smithi098 ceph-mon[30092]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-05-09T08:51:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:08 smithi142 ceph-mon[34625]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-05-09T08:51:08.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:08 smithi105 ceph-mon[34498]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-05-09T08:51:10.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:10 smithi098 ceph-mon[30092]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-05-09T08:51:10.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:10 smithi142 ceph-mon[34625]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-05-09T08:51:10.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:10 smithi105 ceph-mon[34498]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-05-09T08:51:12.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:12 smithi098 ceph-mon[30092]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-05-09T08:51:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:12 smithi142 ceph-mon[34625]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-05-09T08:51:12.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:12 smithi105 ceph-mon[34498]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-05-09T08:51:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.308+0000 7f7148d09640 1 -- 172.21.15.98:0/1507784321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7144101840 msgr2=0x7f7144101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.308+0000 7f7148d09640 1 --2- 172.21.15.98:0/1507784321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7144101840 0x7f7144101cc0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f712c009930 tx=0x7f712c02a420 comp rx=0 tx=0).stop 2024-05-09T08:51:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.308+0000 7f7148d09640 1 -- 172.21.15.98:0/1507784321 shutdown_connections 2024-05-09T08:51:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.308+0000 7f7148d09640 1 --2- 172.21.15.98:0/1507784321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7144102200 0x7f714410e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.308+0000 7f7148d09640 1 --2- 172.21.15.98:0/1507784321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7144101840 0x7f7144101cc0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.308+0000 7f7148d09640 1 --2- 172.21.15.98:0/1507784321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7144107850 0x7f7144107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.308+0000 7f7148d09640 1 -- 172.21.15.98:0/1507784321 >> 172.21.15.98:0/1507784321 conn(0x7f7144078cc0 msgr2=0x7f71440790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.308+0000 7f7148d09640 1 -- 172.21.15.98:0/1507784321 shutdown_connections 2024-05-09T08:51:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.308+0000 7f7148d09640 1 -- 172.21.15.98:0/1507784321 wait complete. 2024-05-09T08:51:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.309+0000 7f7148d09640 1 Processor -- start 2024-05-09T08:51:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.309+0000 7f7148d09640 1 -- start start 2024-05-09T08:51:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.309+0000 7f7148d09640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7144101840 0x7f7144198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.309+0000 7f7148d09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7144102200 0x7f71441985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7148d09640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7144107850 0x7f714419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7148d09640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7144111c60 con 0x7f7144102200 2024-05-09T08:51:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7148d09640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7144111ae0 con 0x7f7144101840 2024-05-09T08:51:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7148d09640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7144111de0 con 0x7f7144107850 2024-05-09T08:51:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7139d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7144102200 0x7f71441985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7139d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7144102200 0x7f71441985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57192/0 (socket says 172.21.15.98:57192) 2024-05-09T08:51:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7139d9b640 1 -- 172.21.15.98:0/3259124759 learned_addr learned my addr 172.21.15.98:0/3259124759 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7142d9d640 1 --2- 172.21.15.98:0/3259124759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7144107850 0x7f714419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f714259c640 1 --2- 172.21.15.98:0/3259124759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7144101840 0x7f7144198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7139d9b640 1 -- 172.21.15.98:0/3259124759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7144107850 msgr2=0x7f714419c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7139d9b640 1 --2- 172.21.15.98:0/3259124759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7144107850 0x7f714419c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7139d9b640 1 -- 172.21.15.98:0/3259124759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7144101840 msgr2=0x7f7144198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7139d9b640 1 --2- 172.21.15.98:0/3259124759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7144101840 0x7f7144198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7139d9b640 1 -- 172.21.15.98:0/3259124759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f714419d070 con 0x7f7144102200 2024-05-09T08:51:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7139d9b640 1 --2- 172.21.15.98:0/3259124759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7144102200 0x7f71441985c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f712c009230 tx=0x7f712c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f713bfff640 1 -- 172.21.15.98:0/3259124759 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c03f030 con 0x7f7144102200 2024-05-09T08:51:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f713bfff640 1 -- 172.21.15.98:0/3259124759 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f712c003ad0 con 0x7f7144102200 2024-05-09T08:51:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7148d09640 1 -- 172.21.15.98:0/3259124759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f714419d300 con 0x7f7144102200 2024-05-09T08:51:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f713bfff640 1 -- 172.21.15.98:0/3259124759 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c035070 con 0x7f7144102200 2024-05-09T08:51:14.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.310+0000 7f7148d09640 1 -- 172.21.15.98:0/3259124759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71441a4bf0 con 0x7f7144102200 2024-05-09T08:51:14.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.313+0000 7f713bfff640 1 -- 172.21.15.98:0/3259124759 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f712c042030 con 0x7f7144102200 2024-05-09T08:51:14.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f714ba0c740 1 Processor -- start 2024-05-09T08:51:14.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f714ba0c740 1 -- start start 2024-05-09T08:51:14.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f714ba0c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcc1350 0x56443bcc1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f714ba0c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcd0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f714ba0c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcd0a40 0x56443bcd2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f714ba0c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56443bc26340 con 0x56443bcc1c70 2024-05-09T08:51:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f714ba0c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56443bbce290 con 0x56443bcd0a40 2024-05-09T08:51:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f714ba0c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56443bc22300 con 0x56443bcc1350 2024-05-09T08:51:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f714259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcd0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f714259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcd0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57208/0 (socket says 172.21.15.98:57208) 2024-05-09T08:51:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f714259c640 1 -- 172.21.15.98:0/4097451188 learned_addr learned my addr 172.21.15.98:0/4097451188 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f713bfff640 1 --2- 172.21.15.98:0/3259124759 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f71300a2f50 0x7f71300a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f7139d9b640 1 --2- 172.21.15.98:0/4097451188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcd0a40 0x56443bcd2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f7142d9d640 1 --2- 172.21.15.98:0/4097451188 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcc1350 0x56443bcc1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.314+0000 7f7142d9d640 1 --2- 172.21.15.98:0/3259124759 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f71300a2f50 0x7f71300a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.315+0000 7f714259c640 1 -- 172.21.15.98:0/4097451188 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcc1350 msgr2=0x56443bcc1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.315+0000 7f714259c640 1 --2- 172.21.15.98:0/4097451188 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcc1350 0x56443bcc1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.315+0000 7f714259c640 1 -- 172.21.15.98:0/4097451188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcd0a40 msgr2=0x56443bcd2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.315+0000 7f714259c640 1 --2- 172.21.15.98:0/4097451188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcd0a40 0x56443bcd2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.315+0000 7f714259c640 1 -- 172.21.15.98:0/4097451188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56443bbf28e0 con 0x56443bcc1c70 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.315+0000 7f7142d9d640 1 --2- 172.21.15.98:0/4097451188 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcc1350 0x56443bcc1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.315+0000 7f713bfff640 1 -- 172.21.15.98:0/3259124759 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f712c0f0070 con 0x7f7144102200 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.315+0000 7f714259c640 1 --2- 172.21.15.98:0/4097451188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcd0500 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f713400cb40 tx=0x7f713400e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8860fb1e0c8b481b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.315+0000 7f7142d9d640 1 --2- 172.21.15.98:0/3259124759 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f71300a2f50 0x7f71300a5410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f713c005c40 tx=0x7f713c005bd0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.315+0000 7f7138d99640 1 -- 172.21.15.98:0/4097451188 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7134010430 con 0x56443bcc1c70 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.315+0000 7f7138d99640 1 -- 172.21.15.98:0/4097451188 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7134010ab0 con 0x56443bcc1c70 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.315+0000 7f7138d99640 1 -- 172.21.15.98:0/4097451188 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713400fbb0 con 0x56443bcc1c70 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.316+0000 7f714ba0c740 1 -- 172.21.15.98:0/4097451188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 msgr2=0x56443bcd0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.316+0000 7f714ba0c740 1 --2- 172.21.15.98:0/4097451188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcd0500 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f713400cb40 tx=0x7f713400e350 comp rx=0 tx=0).stop 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.316+0000 7f714ba0c740 1 -- 172.21.15.98:0/4097451188 shutdown_connections 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.316+0000 7f714ba0c740 1 --2- 172.21.15.98:0/4097451188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcd0a40 0x56443bcd2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.316+0000 7f714ba0c740 1 --2- 172.21.15.98:0/4097451188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcd0500 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.316+0000 7f714ba0c740 1 --2- 172.21.15.98:0/4097451188 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcc1350 0x56443bcc1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.316+0000 7f714ba0c740 1 -- 172.21.15.98:0/4097451188 >> 172.21.15.98:0/4097451188 conn(0x56443bcbd960 msgr2=0x56443bcbdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:14.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.316+0000 7f714ba0c740 1 -- 172.21.15.98:0/4097451188 shutdown_connections 2024-05-09T08:51:14.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.316+0000 7f714ba0c740 1 -- 172.21.15.98:0/4097451188 wait complete. 2024-05-09T08:51:14.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.316+0000 7f714ba0c740 1 -- start start 2024-05-09T08:51:14.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.316+0000 7f714ba0c740 1 Processor -- start 2024-05-09T08:51:14.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f714ba0c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcc1350 0x56443bccd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f714ba0c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bccdea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f714ba0c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcd0a40 0x56443bcce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f714ba0c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56443bcd3670 con 0x56443bcc1c70 2024-05-09T08:51:14.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f714ba0c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56443bbe1790 con 0x56443bcc1350 2024-05-09T08:51:14.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f714ba0c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56443bcd37f0 con 0x56443bcd0a40 2024-05-09T08:51:14.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f714ba0c740 10 client.0 _ll_register_callbacks cb 0x56443bc1ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:14.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f7139d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bccdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f7139d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bccdea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57218/0 (socket says 172.21.15.98:57218) 2024-05-09T08:51:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f7139d9b640 1 -- 172.21.15.98:0/2190213854 learned_addr learned my addr 172.21.15.98:0/2190213854 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f714259c640 1 --2- 172.21.15.98:0/2190213854 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcc1350 0x56443bccd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f7139d9b640 1 -- 172.21.15.98:0/2190213854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcd0a40 msgr2=0x56443bcce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f7139d9b640 1 --2- 172.21.15.98:0/2190213854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcd0a40 0x56443bcce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f7139d9b640 1 -- 172.21.15.98:0/2190213854 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcc1350 msgr2=0x56443bccd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f7139d9b640 1 --2- 172.21.15.98:0/2190213854 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcc1350 0x56443bccd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f7139d9b640 1 -- 172.21.15.98:0/2190213854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56443b93a210 con 0x56443bcc1c70 2024-05-09T08:51:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f7139d9b640 1 --2- 172.21.15.98:0/2190213854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bccdea0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f712c0b8960 tx=0x7f712c040fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=996d9ccbf0ee1fe4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f7129ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f7129ffb640 1 -- 172.21.15.98:0/2190213854 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c049040 con 0x56443bcc1c70 2024-05-09T08:51:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f7129ffb640 1 -- 172.21.15.98:0/2190213854 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f712c0b8600 con 0x56443bcc1c70 2024-05-09T08:51:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f714ba0c740 1 -- 172.21.15.98:0/2190213854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7134010ab0 con 0x56443bcc1c70 2024-05-09T08:51:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.317+0000 7f7129ffb640 1 -- 172.21.15.98:0/2190213854 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f712c0bbf90 con 0x56443bcc1c70 2024-05-09T08:51:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.318+0000 7f714ba0c740 10 client.15855 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.318+0000 7f714ba0c740 1 -- 172.21.15.98:0/2190213854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56443ba06620 con 0x56443bcc1c70 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.318+0000 7f7129ffb640 1 -- 172.21.15.98:0/2190213854 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7134010ab0 con 0x56443bcc1c70 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.318+0000 7f7129ffb640 1 -- 172.21.15.98:0/2190213854 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f712c0f20f0 con 0x56443bcc1c70 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.318+0000 7f714ba0c740 10 client.15855 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.318+0000 7f714ba0c740 10 client.15855 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.318+0000 7f714ba0c740 20 client.15855 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.318+0000 7f714ba0c740 1 --2- 172.21.15.98:0/2190213854 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56443bce0790 0x56443bce2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.318+0000 7f714ba0c740 4 client.15855 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.318+0000 7f714ba0c740 1 -- 172.21.15.98:0/2190213854 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56443bc33a70 con 0x56443bce0790 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.318+0000 7f714259c640 1 --2- 172.21.15.98:0/2190213854 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56443bce0790 0x56443bce2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f7129ffb640 10 client.15855 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.318+0000 7f714259c640 1 --2- 172.21.15.98:0/2190213854 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56443bce0790 0x56443bce2c20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54d6778fc52e9d3f server_cookie=9dfddaa712475886 in_seq=0 out_seq=0 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f7129ffb640 1 -- 172.21.15.98:0/2190213854 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56443bc33a70 con 0x56443bce0790 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f7129ffb640 10 client.15855 handle_command_reply: tid=0 2024-05-09T08:51:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f714ba0c740 1 client.15855 shutdown 2024-05-09T08:51:14.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f714ba0c740 1 -- 172.21.15.98:0/2190213854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 msgr2=0x56443bccdea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:14.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f714ba0c740 1 --2- 172.21.15.98:0/2190213854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bccdea0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f712c0b8960 tx=0x7f712c040fb0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f714ba0c740 1 -- 172.21.15.98:0/2190213854 shutdown_connections 2024-05-09T08:51:14.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f714ba0c740 1 --2- 172.21.15.98:0/2190213854 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56443bce0790 0x56443bce2c20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f714ba0c740 1 --2- 172.21.15.98:0/2190213854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcd0a40 0x56443bcce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f714ba0c740 1 --2- 172.21.15.98:0/2190213854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bccdea0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f7129ffb640 0 client.15855 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:14.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f714ba0c740 1 --2- 172.21.15.98:0/2190213854 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcc1350 0x56443bccd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f714ba0c740 1 -- 172.21.15.98:0/2190213854 >> 172.21.15.98:0/2190213854 conn(0x56443bcbd960 msgr2=0x56443bcc37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:14.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f714ba0c740 1 -- 172.21.15.98:0/2190213854 shutdown_connections 2024-05-09T08:51:14.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.319+0000 7f714ba0c740 1 -- 172.21.15.98:0/2190213854 wait complete. 2024-05-09T08:51:14.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.320+0000 7f714ba0c740 20 client.15855 trim_cache size 0 max 16384 2024-05-09T08:51:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.335+0000 7f714ba0c740 1 Processor -- start 2024-05-09T08:51:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.335+0000 7f714ba0c740 1 -- start start 2024-05-09T08:51:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.335+0000 7f714ba0c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcc1350 0x56443bcd0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.335+0000 7f714ba0c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcdc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.335+0000 7f714ba0c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcd0a40 0x56443bcccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.335+0000 7f714ba0c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f71441acb30 con 0x56443bcc1c70 2024-05-09T08:51:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.335+0000 7f714ba0c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f71441ae670 con 0x56443bcd0a40 2024-05-09T08:51:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.335+0000 7f714ba0c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f71441ad200 con 0x56443bcc1350 2024-05-09T08:51:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.335+0000 7f7139d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcdc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.335+0000 7f7139d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcdc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57228/0 (socket says 172.21.15.98:57228) 2024-05-09T08:51:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.335+0000 7f7139d9b640 1 -- 172.21.15.98:0/1024350826 learned_addr learned my addr 172.21.15.98:0/1024350826 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.335+0000 7f7142d9d640 1 --2- 172.21.15.98:0/1024350826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcd0a40 0x56443bcccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.335+0000 7f714259c640 1 --2- 172.21.15.98:0/1024350826 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcc1350 0x56443bcd0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.336+0000 7f7139d9b640 1 -- 172.21.15.98:0/1024350826 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcc1350 msgr2=0x56443bcd0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.336+0000 7f7139d9b640 1 --2- 172.21.15.98:0/1024350826 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcc1350 0x56443bcd0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.336+0000 7f7139d9b640 1 -- 172.21.15.98:0/1024350826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcd0a40 msgr2=0x56443bcccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.336+0000 7f7139d9b640 1 --2- 172.21.15.98:0/1024350826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcd0a40 0x56443bcccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.336+0000 7f7139d9b640 1 -- 172.21.15.98:0/1024350826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56443bcc87d0 con 0x56443bcc1c70 2024-05-09T08:51:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.336+0000 7f7139d9b640 1 --2- 172.21.15.98:0/1024350826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcdc530 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f712c009a50 tx=0x7f712c0ba4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a559223d684bd36d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.336+0000 7f712bfff640 1 -- 172.21.15.98:0/1024350826 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c049050 con 0x56443bcc1c70 2024-05-09T08:51:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.336+0000 7f712bfff640 1 -- 172.21.15.98:0/1024350826 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f712c0d00b0 con 0x56443bcc1c70 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.336+0000 7f712bfff640 1 -- 172.21.15.98:0/1024350826 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c0b8600 con 0x56443bcc1c70 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 -- 172.21.15.98:0/1024350826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 msgr2=0x56443bcdc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 --2- 172.21.15.98:0/1024350826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcdc530 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f712c009a50 tx=0x7f712c0ba4a0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 -- 172.21.15.98:0/1024350826 shutdown_connections 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 --2- 172.21.15.98:0/1024350826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcd0a40 0x56443bcccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 --2- 172.21.15.98:0/1024350826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcdc530 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 --2- 172.21.15.98:0/1024350826 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcc1350 0x56443bcd0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 -- 172.21.15.98:0/1024350826 >> 172.21.15.98:0/1024350826 conn(0x56443bcbd960 msgr2=0x56443bcbdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 -- 172.21.15.98:0/1024350826 shutdown_connections 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 -- 172.21.15.98:0/1024350826 wait complete. 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 -- start start 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 Processor -- start 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcc1350 0x56443bcd8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcd0a40 0x56443bcca730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f71441adfa0 con 0x56443bcc1c70 2024-05-09T08:51:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f71441ad8d0 con 0x56443bcc1350 2024-05-09T08:51:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f71441aab10 con 0x56443bcd0a40 2024-05-09T08:51:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f714ba0c740 10 client.0 _ll_register_callbacks cb 0x7f712c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f7139d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f7139d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcca1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57230/0 (socket says 172.21.15.98:57230) 2024-05-09T08:51:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.337+0000 7f7139d9b640 1 -- 172.21.15.98:0/3026379513 learned_addr learned my addr 172.21.15.98:0/3026379513 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f714259c640 1 --2- 172.21.15.98:0/3026379513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcc1350 0x56443bcd8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f7139d9b640 1 -- 172.21.15.98:0/3026379513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcd0a40 msgr2=0x56443bcca730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f7139d9b640 1 --2- 172.21.15.98:0/3026379513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcd0a40 0x56443bcca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f7139d9b640 1 -- 172.21.15.98:0/3026379513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcc1350 msgr2=0x56443bcd8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f7139d9b640 1 --2- 172.21.15.98:0/3026379513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcc1350 0x56443bcd8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f7139d9b640 1 -- 172.21.15.98:0/3026379513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56443bccac70 con 0x56443bcc1c70 2024-05-09T08:51:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f7139d9b640 1 --2- 172.21.15.98:0/3026379513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcca1f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f712c0d5100 tx=0x7f712c005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6288770f5060c4a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f7129ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f7129ffb640 1 -- 172.21.15.98:0/3026379513 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c09e020 con 0x56443bcc1c70 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f7129ffb640 1 -- 172.21.15.98:0/3026379513 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f712c0dabd0 con 0x56443bcc1c70 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f714ba0c740 1 -- 172.21.15.98:0/3026379513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f712c0d00b0 con 0x56443bcc1c70 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f7129ffb640 1 -- 172.21.15.98:0/3026379513 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f712c049070 con 0x56443bcc1c70 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f714ba0c740 10 client.15861 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f714ba0c740 1 -- 172.21.15.98:0/3026379513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56443bccac70 con 0x56443bcc1c70 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f7129ffb640 1 -- 172.21.15.98:0/3026379513 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f712c0c4e50 con 0x56443bcc1c70 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f7129ffb640 1 -- 172.21.15.98:0/3026379513 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f712c0d9070 con 0x56443bcc1c70 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f714ba0c740 10 client.15861 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f714ba0c740 10 client.15861 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f714ba0c740 20 client.15861 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f714ba0c740 1 --2- 172.21.15.98:0/3026379513 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56443bd46e00 0x56443bd491f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f714ba0c740 4 client.15861 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:51:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.338+0000 7f714ba0c740 1 -- 172.21.15.98:0/3026379513 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56443bc2ddc0 con 0x56443bd46e00 2024-05-09T08:51:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.339+0000 7f714259c640 1 --2- 172.21.15.98:0/3026379513 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56443bd46e00 0x56443bd491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.339+0000 7f714259c640 1 --2- 172.21.15.98:0/3026379513 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56443bd46e00 0x56443bd491f0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7230ab8069d61205 server_cookie=62b75938dba96155 in_seq=0 out_seq=0 2024-05-09T08:51:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.339+0000 7f7129ffb640 10 client.15861 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f7129ffb640 1 -- 172.21.15.98:0/3026379513 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56443bc2ddc0 con 0x56443bd46e00 2024-05-09T08:51:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f7129ffb640 10 client.15861 handle_command_reply: tid=0 2024-05-09T08:51:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f714ba0c740 1 client.15861 shutdown 2024-05-09T08:51:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f714ba0c740 1 -- 172.21.15.98:0/3026379513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 msgr2=0x56443bcca1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f714ba0c740 1 --2- 172.21.15.98:0/3026379513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcca1f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f712c0d5100 tx=0x7f712c005c00 comp rx=0 tx=0).stop 2024-05-09T08:51:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f714ba0c740 1 -- 172.21.15.98:0/3026379513 shutdown_connections 2024-05-09T08:51:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f714ba0c740 1 --2- 172.21.15.98:0/3026379513 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56443bd46e00 0x56443bd491f0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f714ba0c740 1 --2- 172.21.15.98:0/3026379513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56443bcd0a40 0x56443bcca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f714ba0c740 1 --2- 172.21.15.98:0/3026379513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56443bcc1c70 0x56443bcca1f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f714ba0c740 1 --2- 172.21.15.98:0/3026379513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56443bcc1350 0x56443bcd8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f7129ffb640 0 client.15861 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f714ba0c740 1 -- 172.21.15.98:0/3026379513 >> 172.21.15.98:0/3026379513 conn(0x56443bcbd960 msgr2=0x56443bccd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f714ba0c740 1 -- 172.21.15.98:0/3026379513 shutdown_connections 2024-05-09T08:51:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f714ba0c740 1 -- 172.21.15.98:0/3026379513 wait complete. 2024-05-09T08:51:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f714ba0c740 20 client.15861 trim_cache size 0 max 16384 2024-05-09T08:51:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:51:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:51:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:51:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:51:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.343+0000 7f7148d09640 1 -- 172.21.15.98:0/3259124759 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f71300a2f50 msgr2=0x7f71300a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.344+0000 7f7148d09640 1 --2- 172.21.15.98:0/3259124759 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f71300a2f50 0x7f71300a5410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f713c005c40 tx=0x7f713c005bd0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.344+0000 7f7148d09640 1 -- 172.21.15.98:0/3259124759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7144102200 msgr2=0x7f71441985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.344+0000 7f7148d09640 1 --2- 172.21.15.98:0/3259124759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7144102200 0x7f71441985c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f712c009230 tx=0x7f712c004270 comp rx=0 tx=0).stop 2024-05-09T08:51:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.344+0000 7f7148d09640 1 -- 172.21.15.98:0/3259124759 shutdown_connections 2024-05-09T08:51:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.344+0000 7f7148d09640 1 --2- 172.21.15.98:0/3259124759 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f71300a2f50 0x7f71300a5410 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.344+0000 7f7148d09640 1 --2- 172.21.15.98:0/3259124759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7144107850 0x7f714419c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.344+0000 7f7148d09640 1 --2- 172.21.15.98:0/3259124759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7144102200 0x7f71441985c0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.344+0000 7f7148d09640 1 --2- 172.21.15.98:0/3259124759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7144101840 0x7f7144198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.344+0000 7f7148d09640 1 -- 172.21.15.98:0/3259124759 >> 172.21.15.98:0/3259124759 conn(0x7f7144078cc0 msgr2=0x7f71440ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.344+0000 7f7148d09640 1 -- 172.21.15.98:0/3259124759 shutdown_connections 2024-05-09T08:51:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:14.344+0000 7f7148d09640 1 -- 172.21.15.98:0/3259124759 wait complete. 2024-05-09T08:51:14.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:14 smithi098 ceph-mon[30092]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-05-09T08:51:14.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:14 smithi142 ceph-mon[34625]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-05-09T08:51:14.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:14 smithi105 ceph-mon[34498]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-05-09T08:51:16.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:16 smithi098 ceph-mon[30092]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-05-09T08:51:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:16 smithi142 ceph-mon[34625]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-05-09T08:51:16.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:16 smithi105 ceph-mon[34498]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-05-09T08:51:16.776 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:51:16.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:51:16.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.894+0000 7f639ae12640 1 -- 172.21.15.98:0/1785728970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394107820 msgr2=0x7f6394107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:16.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.894+0000 7f639ae12640 1 --2- 172.21.15.98:0/1785728970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394107820 0x7f6394107c00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6384009ca0 tx=0x7f638402a950 comp rx=0 tx=0).stop 2024-05-09T08:51:16.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.894+0000 7f639ae12640 1 -- 172.21.15.98:0/1785728970 shutdown_connections 2024-05-09T08:51:16.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.894+0000 7f639ae12640 1 --2- 172.21.15.98:0/1785728970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f639406a580 0x7f639410e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.894+0000 7f639ae12640 1 --2- 172.21.15.98:0/1785728970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6394108140 0x7f6394069e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.894+0000 7f639ae12640 1 --2- 172.21.15.98:0/1785728970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394107820 0x7f6394107c00 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.894+0000 7f639ae12640 1 -- 172.21.15.98:0/1785728970 >> 172.21.15.98:0/1785728970 conn(0x7f6394076530 msgr2=0x7f6394076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:16.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.894+0000 7f639ae12640 1 -- 172.21.15.98:0/1785728970 shutdown_connections 2024-05-09T08:51:16.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.894+0000 7f639ae12640 1 -- 172.21.15.98:0/1785728970 wait complete. 2024-05-09T08:51:16.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f639ae12640 1 Processor -- start 2024-05-09T08:51:16.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f639ae12640 1 -- start start 2024-05-09T08:51:16.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f639ae12640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f639406a580 0x7f6394100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f639ae12640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6394107820 0x7f6394101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f639ae12640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6394108140 0x7f63941015d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f639ae12640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6394111bf0 con 0x7f639406a580 2024-05-09T08:51:16.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f639ae12640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6394111a70 con 0x7f6394108140 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f639ae12640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6394111d70 con 0x7f6394107820 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f6398bae640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f639406a580 0x7f6394100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f6398bae640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f639406a580 0x7f6394100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57284/0 (socket says 172.21.15.98:57284) 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f6398bae640 1 -- 172.21.15.98:0/544040927 learned_addr learned my addr 172.21.15.98:0/544040927 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f63993af640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6394108140 0x7f63941015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f6393fff640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6394107820 0x7f6394101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f6398bae640 1 -- 172.21.15.98:0/544040927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6394107820 msgr2=0x7f6394101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f6398bae640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6394107820 0x7f6394101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f6398bae640 1 -- 172.21.15.98:0/544040927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6394108140 msgr2=0x7f63941015d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f6398bae640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6394108140 0x7f63941015d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f6398bae640 1 -- 172.21.15.98:0/544040927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63941a6510 con 0x7f639406a580 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f63993af640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6394108140 0x7f63941015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.895+0000 7f6398bae640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f639406a580 0x7f6394100b50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f6384005cc0 tx=0x7f638402f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.896+0000 7f6391ffb640 1 -- 172.21.15.98:0/544040927 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384034040 con 0x7f639406a580 2024-05-09T08:51:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.896+0000 7f639ae12640 1 -- 172.21.15.98:0/544040927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63941a67a0 con 0x7f639406a580 2024-05-09T08:51:16.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.896+0000 7f6391ffb640 1 -- 172.21.15.98:0/544040927 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6384003e40 con 0x7f639406a580 2024-05-09T08:51:16.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.896+0000 7f6391ffb640 1 -- 172.21.15.98:0/544040927 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384045480 con 0x7f639406a580 2024-05-09T08:51:16.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.896+0000 7f639ae12640 1 -- 172.21.15.98:0/544040927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63941a6c90 con 0x7f639406a580 2024-05-09T08:51:16.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.898+0000 7f6391ffb640 1 -- 172.21.15.98:0/544040927 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f6384037020 con 0x7f639406a580 2024-05-09T08:51:16.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f639dbf7740 1 Processor -- start 2024-05-09T08:51:16.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f639dbf7740 1 -- start start 2024-05-09T08:51:16.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f639dbf7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 0x558fdf477660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f639dbf7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf477ba0 0x558fdf481d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f639dbf7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf482270 0x558fdf484660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f639dbf7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558fdf3dcd00 con 0x558fdf482270 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f639dbf7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558fdf3844c0 con 0x558fdf477280 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f639dbf7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558fdf3d8cc0 con 0x558fdf477ba0 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f6391ffb640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63800a2ee0 0x7f63800a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f6393fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf482270 0x558fdf484660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f6393fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf482270 0x558fdf484660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57290/0 (socket says 172.21.15.98:57290) 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f6393fff640 1 -- 172.21.15.98:0/611894120 learned_addr learned my addr 172.21.15.98:0/611894120 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f6398bae640 1 --2- 172.21.15.98:0/611894120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf477ba0 0x558fdf481d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f63993af640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63800a2ee0 0x7f63800a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f63993af640 1 --2- 172.21.15.98:0/611894120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 0x558fdf477660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f6391ffb640 1 -- 172.21.15.98:0/544040927 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f63840f0c40 con 0x7f639406a580 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f6393fff640 1 -- 172.21.15.98:0/611894120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf477ba0 msgr2=0x558fdf481d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f6393fff640 1 --2- 172.21.15.98:0/611894120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf477ba0 0x558fdf481d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f6393fff640 1 -- 172.21.15.98:0/611894120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 msgr2=0x558fdf477660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f6393fff640 1 --2- 172.21.15.98:0/611894120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 0x558fdf477660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.899+0000 7f6393fff640 1 -- 172.21.15.98:0/611894120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fdf1ba7d0 con 0x558fdf482270 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.900+0000 7f6393fff640 1 --2- 172.21.15.98:0/611894120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf482270 0x558fdf484660 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f637c00cb90 tx=0x7f637c00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf286cba51827ec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.900+0000 7f63993af640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63800a2ee0 0x7f63800a53a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f638c013960 tx=0x7f638c005f70 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.900+0000 7f638affd640 1 -- 172.21.15.98:0/611894120 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637c005220 con 0x558fdf482270 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.900+0000 7f638affd640 1 -- 172.21.15.98:0/611894120 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f637c004c80 con 0x558fdf482270 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.900+0000 7f638affd640 1 -- 172.21.15.98:0/611894120 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637c0107f0 con 0x558fdf482270 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.900+0000 7f639dbf7740 1 -- 172.21.15.98:0/611894120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf482270 msgr2=0x558fdf484660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.900+0000 7f639dbf7740 1 --2- 172.21.15.98:0/611894120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf482270 0x558fdf484660 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f637c00cb90 tx=0x7f637c00e720 comp rx=0 tx=0).stop 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 -- 172.21.15.98:0/611894120 shutdown_connections 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 --2- 172.21.15.98:0/611894120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf482270 0x558fdf484660 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 --2- 172.21.15.98:0/611894120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf477ba0 0x558fdf481d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 --2- 172.21.15.98:0/611894120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 0x558fdf477660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 -- 172.21.15.98:0/611894120 >> 172.21.15.98:0/611894120 conn(0x558fdf4735a0 msgr2=0x558fdf473980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 -- 172.21.15.98:0/611894120 shutdown_connections 2024-05-09T08:51:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 -- 172.21.15.98:0/611894120 wait complete. 2024-05-09T08:51:16.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 -- start start 2024-05-09T08:51:16.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 Processor -- start 2024-05-09T08:51:16.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 0x558fdf47f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf477ba0 0x558fdf47f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf482270 0x558fdf47fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558fdf485060 con 0x558fdf477ba0 2024-05-09T08:51:16.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558fdf397b90 con 0x558fdf477280 2024-05-09T08:51:16.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558fdf4851e0 con 0x558fdf482270 2024-05-09T08:51:16.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.901+0000 7f639dbf7740 10 client.0 _ll_register_callbacks cb 0x558fdf3d3da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:16.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6393fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf477ba0 0x558fdf47f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6393fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf477ba0 0x558fdf47f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57300/0 (socket says 172.21.15.98:57300) 2024-05-09T08:51:16.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6393fff640 1 -- 172.21.15.98:0/243464131 learned_addr learned my addr 172.21.15.98:0/243464131 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:16.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f63993af640 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf482270 0x558fdf47fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6398bae640 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 0x558fdf47f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6393fff640 1 -- 172.21.15.98:0/243464131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf482270 msgr2=0x558fdf47fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6393fff640 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf482270 0x558fdf47fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6393fff640 1 -- 172.21.15.98:0/243464131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 msgr2=0x558fdf47f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6393fff640 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 0x558fdf47f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6393fff640 1 -- 172.21.15.98:0/243464131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fdf0f03f0 con 0x558fdf477ba0 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6398bae640 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 0x558fdf47f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f63993af640 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf482270 0x558fdf47fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6393fff640 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf477ba0 0x558fdf47f700 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f637c00f010 tx=0x7f637c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a58896e2bed0b2f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6373fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6373fff640 1 -- 172.21.15.98:0/243464131 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637c029060 con 0x558fdf477ba0 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6373fff640 1 -- 172.21.15.98:0/243464131 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f637c0048d0 con 0x558fdf477ba0 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f639dbf7740 1 -- 172.21.15.98:0/243464131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f637c004c80 con 0x558fdf477ba0 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f6373fff640 1 -- 172.21.15.98:0/243464131 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f637c030050 con 0x558fdf477ba0 2024-05-09T08:51:16.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f639dbf7740 10 client.15873 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.902+0000 7f639dbf7740 1 -- 172.21.15.98:0/243464131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558fdf0b00f0 con 0x558fdf477ba0 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.903+0000 7f6373fff640 1 -- 172.21.15.98:0/243464131 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f637c004c80 con 0x558fdf477ba0 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.903+0000 7f6373fff640 1 -- 172.21.15.98:0/243464131 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f637c01a410 con 0x558fdf477ba0 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.903+0000 7f639dbf7740 10 client.15873 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.903+0000 7f639dbf7740 10 client.15873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.903+0000 7f639dbf7740 20 client.15873 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.903+0000 7f639dbf7740 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558fdf492430 0x558fdf494820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.903+0000 7f639dbf7740 4 client.15873 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.903+0000 7f639dbf7740 1 -- 172.21.15.98:0/243464131 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fdf3ea430 con 0x558fdf492430 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.903+0000 7f6398bae640 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558fdf492430 0x558fdf494820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.904+0000 7f6373fff640 10 client.15873 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.904+0000 7f6398bae640 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558fdf492430 0x558fdf494820 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa78009ecf055e04 server_cookie=5122715c93f8740f in_seq=0 out_seq=0 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.904+0000 7f6373fff640 1 -- 172.21.15.98:0/243464131 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558fdf3ea430 con 0x558fdf492430 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.904+0000 7f6373fff640 10 client.15873 handle_command_reply: tid=0 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.904+0000 7f639dbf7740 1 client.15873 shutdown 2024-05-09T08:51:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.904+0000 7f639dbf7740 1 -- 172.21.15.98:0/243464131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf477ba0 msgr2=0x558fdf47f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:16.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.904+0000 7f639dbf7740 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf477ba0 0x558fdf47f700 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f637c00f010 tx=0x7f637c00c8d0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.904+0000 7f639dbf7740 1 -- 172.21.15.98:0/243464131 shutdown_connections 2024-05-09T08:51:16.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.904+0000 7f639dbf7740 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558fdf492430 0x558fdf494820 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.904+0000 7f639dbf7740 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf482270 0x558fdf47fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.904+0000 7f6373fff640 0 client.15873 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:16.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.904+0000 7f639dbf7740 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf477ba0 0x558fdf47f700 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.904+0000 7f639dbf7740 1 --2- 172.21.15.98:0/243464131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 0x558fdf47f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.904+0000 7f639dbf7740 1 -- 172.21.15.98:0/243464131 >> 172.21.15.98:0/243464131 conn(0x558fdf4735a0 msgr2=0x558fdf4793a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:16.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.905+0000 7f639dbf7740 1 -- 172.21.15.98:0/243464131 shutdown_connections 2024-05-09T08:51:16.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.905+0000 7f639dbf7740 1 -- 172.21.15.98:0/243464131 wait complete. 2024-05-09T08:51:16.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.905+0000 7f639dbf7740 20 client.15873 trim_cache size 0 max 16384 2024-05-09T08:51:16.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.922+0000 7f639dbf7740 1 Processor -- start 2024-05-09T08:51:16.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.922+0000 7f639dbf7740 1 -- start start 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.922+0000 7f639dbf7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf477280 0x558fdf48df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.922+0000 7f639dbf7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477ba0 0x558fdf46e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.922+0000 7f639dbf7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf482270 0x558fdf499e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.922+0000 7f639dbf7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63941b0e50 con 0x558fdf477280 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.922+0000 7f639dbf7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f63941b2990 con 0x558fdf477ba0 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.922+0000 7f639dbf7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f63941b1520 con 0x558fdf482270 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.922+0000 7f6398bae640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf477280 0x558fdf48df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.922+0000 7f6398bae640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf477280 0x558fdf48df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57302/0 (socket says 172.21.15.98:57302) 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.922+0000 7f6398bae640 1 -- 172.21.15.98:0/3368788242 learned_addr learned my addr 172.21.15.98:0/3368788242 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f6398bae640 1 -- 172.21.15.98:0/3368788242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf482270 msgr2=0x558fdf499e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f6398bae640 1 --2- 172.21.15.98:0/3368788242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf482270 0x558fdf499e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f6398bae640 1 -- 172.21.15.98:0/3368788242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477ba0 msgr2=0x558fdf46e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f6398bae640 1 --2- 172.21.15.98:0/3368788242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477ba0 0x558fdf46e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f6398bae640 1 -- 172.21.15.98:0/3368788242 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fdf47e170 con 0x558fdf477280 2024-05-09T08:51:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f6398bae640 1 --2- 172.21.15.98:0/3368788242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf477280 0x558fdf48df20 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f63840bcc00 tx=0x7f638408ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ff3f6143389685 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f638a7fc640 1 -- 172.21.15.98:0/3368788242 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638402c040 con 0x558fdf477280 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f638a7fc640 1 -- 172.21.15.98:0/3368788242 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63840bd370 con 0x558fdf477280 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f638a7fc640 1 -- 172.21.15.98:0/3368788242 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63840bb040 con 0x558fdf477280 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f639dbf7740 1 -- 172.21.15.98:0/3368788242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf477280 msgr2=0x558fdf48df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f639dbf7740 1 --2- 172.21.15.98:0/3368788242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf477280 0x558fdf48df20 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f63840bcc00 tx=0x7f638408ec30 comp rx=0 tx=0).stop 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f639dbf7740 1 -- 172.21.15.98:0/3368788242 shutdown_connections 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f639dbf7740 1 --2- 172.21.15.98:0/3368788242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf482270 0x558fdf499e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f639dbf7740 1 --2- 172.21.15.98:0/3368788242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477ba0 0x558fdf46e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f639dbf7740 1 --2- 172.21.15.98:0/3368788242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf477280 0x558fdf48df20 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f639dbf7740 1 -- 172.21.15.98:0/3368788242 >> 172.21.15.98:0/3368788242 conn(0x558fdf4735a0 msgr2=0x558fdf47fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f639dbf7740 1 -- 172.21.15.98:0/3368788242 shutdown_connections 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.923+0000 7f639dbf7740 1 -- 172.21.15.98:0/3368788242 wait complete. 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.924+0000 7f639dbf7740 1 -- start start 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.924+0000 7f639dbf7740 1 Processor -- start 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.924+0000 7f639dbf7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 0x558fdf4864d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.924+0000 7f639dbf7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf477ba0 0x558fdf486a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.924+0000 7f639dbf7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf482270 0x558fdf4f7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.924+0000 7f639dbf7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63941b22c0 con 0x558fdf482270 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.924+0000 7f639dbf7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f63941b1bf0 con 0x558fdf477280 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.924+0000 7f639dbf7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f63941aee30 con 0x558fdf477ba0 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.924+0000 7f639dbf7740 10 client.0 _ll_register_callbacks cb 0x7f637c014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.924+0000 7f63993af640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf482270 0x558fdf4f7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.924+0000 7f63993af640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf482270 0x558fdf4f7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57304/0 (socket says 172.21.15.98:57304) 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.924+0000 7f63993af640 1 -- 172.21.15.98:0/4205646986 learned_addr learned my addr 172.21.15.98:0/4205646986 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.924+0000 7f6393fff640 1 --2- 172.21.15.98:0/4205646986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf477ba0 0x558fdf486a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.924+0000 7f6398bae640 1 --2- 172.21.15.98:0/4205646986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 0x558fdf4864d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f63993af640 1 -- 172.21.15.98:0/4205646986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf477ba0 msgr2=0x558fdf486a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f63993af640 1 --2- 172.21.15.98:0/4205646986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf477ba0 0x558fdf486a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f63993af640 1 -- 172.21.15.98:0/4205646986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 msgr2=0x558fdf4864d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f63993af640 1 --2- 172.21.15.98:0/4205646986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 0x558fdf4864d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f63993af640 1 -- 172.21.15.98:0/4205646986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fdf4f7bd0 con 0x558fdf482270 2024-05-09T08:51:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f6398bae640 1 --2- 172.21.15.98:0/4205646986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 0x558fdf4864d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f63993af640 1 --2- 172.21.15.98:0/4205646986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf482270 0x558fdf4f7690 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f638c0042c0 tx=0x7f638c00bae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32388c377419a55b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f6373fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f6373fff640 1 -- 172.21.15.98:0/4205646986 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c026070 con 0x558fdf482270 2024-05-09T08:51:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f6373fff640 1 -- 172.21.15.98:0/4205646986 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f638c003af0 con 0x558fdf482270 2024-05-09T08:51:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f639dbf7740 1 -- 172.21.15.98:0/4205646986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63840bd370 con 0x558fdf482270 2024-05-09T08:51:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f6373fff640 1 -- 172.21.15.98:0/4205646986 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f638c032050 con 0x558fdf482270 2024-05-09T08:51:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f639dbf7740 10 client.15879 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f639dbf7740 1 -- 172.21.15.98:0/4205646986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558fdf4f8e40 con 0x558fdf482270 2024-05-09T08:51:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f6373fff640 1 -- 172.21.15.98:0/4205646986 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63840bd370 con 0x558fdf482270 2024-05-09T08:51:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f6373fff640 1 -- 172.21.15.98:0/4205646986 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f638c01f050 con 0x558fdf482270 2024-05-09T08:51:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f639dbf7740 10 client.15879 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f639dbf7740 10 client.15879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f639dbf7740 20 client.15879 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f639dbf7740 1 --2- 172.21.15.98:0/4205646986 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558fdf4f95c0 0x558fdf4fb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f639dbf7740 4 client.15879 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.925+0000 7f639dbf7740 1 -- 172.21.15.98:0/4205646986 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558fdf3e4780 con 0x558fdf4f95c0 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.926+0000 7f6398bae640 1 --2- 172.21.15.98:0/4205646986 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558fdf4f95c0 0x558fdf4fb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.926+0000 7f6398bae640 1 --2- 172.21.15.98:0/4205646986 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558fdf4f95c0 0x558fdf4fb9e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22533522803ea9fb server_cookie=e23958e8bd1666e4 in_seq=0 out_seq=0 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.926+0000 7f6373fff640 10 client.15879 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.926+0000 7f6373fff640 1 -- 172.21.15.98:0/4205646986 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f638408d050 con 0x558fdf4f95c0 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.926+0000 7f6373fff640 10 client.15879 handle_command_reply: tid=0 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.926+0000 7f639dbf7740 1 client.15879 shutdown 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.926+0000 7f639dbf7740 1 -- 172.21.15.98:0/4205646986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf482270 msgr2=0x558fdf4f7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.926+0000 7f639dbf7740 1 --2- 172.21.15.98:0/4205646986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf482270 0x558fdf4f7690 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f638c0042c0 tx=0x7f638c00bae0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639dbf7740 1 -- 172.21.15.98:0/4205646986 shutdown_connections 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639dbf7740 1 --2- 172.21.15.98:0/4205646986 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558fdf4f95c0 0x558fdf4fb9e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639dbf7740 1 --2- 172.21.15.98:0/4205646986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558fdf482270 0x558fdf4f7690 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639dbf7740 1 --2- 172.21.15.98:0/4205646986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fdf477ba0 0x558fdf486a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f6373fff640 0 client.15879 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639dbf7740 1 --2- 172.21.15.98:0/4205646986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558fdf477280 0x558fdf4864d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639dbf7740 1 -- 172.21.15.98:0/4205646986 >> 172.21.15.98:0/4205646986 conn(0x558fdf4735a0 msgr2=0x558fdf4789d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639dbf7740 1 -- 172.21.15.98:0/4205646986 shutdown_connections 2024-05-09T08:51:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639dbf7740 1 -- 172.21.15.98:0/4205646986 wait complete. 2024-05-09T08:51:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639dbf7740 20 client.15879 trim_cache size 0 max 16384 2024-05-09T08:51:16.932 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:51:16.932 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:51:16.932 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:51:16.932 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:51:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639ae12640 1 -- 172.21.15.98:0/544040927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63800a2ee0 msgr2=0x7f63800a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639ae12640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63800a2ee0 0x7f63800a53a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f638c013960 tx=0x7f638c005f70 comp rx=0 tx=0).stop 2024-05-09T08:51:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639ae12640 1 -- 172.21.15.98:0/544040927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f639406a580 msgr2=0x7f6394100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639ae12640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f639406a580 0x7f6394100b50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f6384005cc0 tx=0x7f638402f6c0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639ae12640 1 -- 172.21.15.98:0/544040927 shutdown_connections 2024-05-09T08:51:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639ae12640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63800a2ee0 0x7f63800a53a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639ae12640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6394108140 0x7f63941015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639ae12640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6394107820 0x7f6394101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639ae12640 1 --2- 172.21.15.98:0/544040927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f639406a580 0x7f6394100b50 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639ae12640 1 -- 172.21.15.98:0/544040927 >> 172.21.15.98:0/544040927 conn(0x7f6394076530 msgr2=0x7f63940ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639ae12640 1 -- 172.21.15.98:0/544040927 shutdown_connections 2024-05-09T08:51:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:16.927+0000 7f639ae12640 1 -- 172.21.15.98:0/544040927 wait complete. 2024-05-09T08:51:16.938 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:51:16.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:51:17.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.039+0000 7fd905d9e640 1 -- 172.21.15.98:0/1932919660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9001031a0 msgr2=0x7fd90010b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.039+0000 7fd905d9e640 1 --2- 172.21.15.98:0/1932919660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9001031a0 0x7fd90010b680 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0009900 tx=0x7fd8f002a410 comp rx=0 tx=0).stop 2024-05-09T08:51:17.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.039+0000 7fd905d9e640 1 -- 172.21.15.98:0/1932919660 shutdown_connections 2024-05-09T08:51:17.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.039+0000 7fd905d9e640 1 --2- 172.21.15.98:0/1932919660 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90010bc50 0x7fd90010e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.039+0000 7fd905d9e640 1 --2- 172.21.15.98:0/1932919660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9001031a0 0x7fd90010b680 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.039+0000 7fd905d9e640 1 --2- 172.21.15.98:0/1932919660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd900102880 0x7fd900102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.039+0000 7fd905d9e640 1 -- 172.21.15.98:0/1932919660 >> 172.21.15.98:0/1932919660 conn(0x7fd9000fc020 msgr2=0x7fd9000fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.039+0000 7fd905d9e640 1 -- 172.21.15.98:0/1932919660 shutdown_connections 2024-05-09T08:51:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.039+0000 7fd905d9e640 1 -- 172.21.15.98:0/1932919660 wait complete. 2024-05-09T08:51:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd905d9e640 1 Processor -- start 2024-05-09T08:51:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd905d9e640 1 -- start start 2024-05-09T08:51:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd905d9e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd900102880 0x7fd9001004d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd905d9e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd9001031a0 0x7fd900100a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd905d9e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd90010bc50 0x7fd900100f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd905d9e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd900111a00 con 0x7fd900102880 2024-05-09T08:51:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd905d9e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd900111880 con 0x7fd90010bc50 2024-05-09T08:51:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd905d9e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd900111b80 con 0x7fd9001031a0 2024-05-09T08:51:17.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd904d9c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd900102880 0x7fd9001004d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd904d9c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd900102880 0x7fd9001004d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57324/0 (socket says 172.21.15.98:57324) 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd904d9c640 1 -- 172.21.15.98:0/1489038263 learned_addr learned my addr 172.21.15.98:0/1489038263 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd8f7fff640 1 --2- 172.21.15.98:0/1489038263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd9001031a0 0x7fd900100a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd904d9c640 1 -- 172.21.15.98:0/1489038263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd9001031a0 msgr2=0x7fd900100a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd90559d640 1 --2- 172.21.15.98:0/1489038263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd90010bc50 0x7fd900100f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd904d9c640 1 --2- 172.21.15.98:0/1489038263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd9001031a0 0x7fd900100a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd904d9c640 1 -- 172.21.15.98:0/1489038263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd90010bc50 msgr2=0x7fd900100f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd904d9c640 1 --2- 172.21.15.98:0/1489038263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd90010bc50 0x7fd900100f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd904d9c640 1 -- 172.21.15.98:0/1489038263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9001a6440 con 0x7fd900102880 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.040+0000 7fd904d9c640 1 --2- 172.21.15.98:0/1489038263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd900102880 0x7fd9001004d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00db80 tx=0x7fd8fc00b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.041+0000 7fd8f5ffb640 1 -- 172.21.15.98:0/1489038263 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc017070 con 0x7fd900102880 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.041+0000 7fd8f5ffb640 1 -- 172.21.15.98:0/1489038263 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8fc003b70 con 0x7fd900102880 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.041+0000 7fd905d9e640 1 -- 172.21.15.98:0/1489038263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9001a6730 con 0x7fd900102880 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.041+0000 7fd8f5ffb640 1 -- 172.21.15.98:0/1489038263 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc0106c0 con 0x7fd900102880 2024-05-09T08:51:17.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.041+0000 7fd905d9e640 1 -- 172.21.15.98:0/1489038263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9001a6c80 con 0x7fd900102880 2024-05-09T08:51:17.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.042+0000 7fd905d9e640 1 -- 172.21.15.98:0/1489038263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9000689c0 con 0x7fd900102880 2024-05-09T08:51:17.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.043+0000 7fd8f5ffb640 1 -- 172.21.15.98:0/1489038263 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fd8fc004020 con 0x7fd900102880 2024-05-09T08:51:17.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.043+0000 7fd8f5ffb640 1 --2- 172.21.15.98:0/1489038263 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8e80a2eb0 0x7fd8e80a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.044+0000 7fd8f7fff640 1 --2- 172.21.15.98:0/1489038263 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8e80a2eb0 0x7fd8e80a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.044+0000 7fd8f5ffb640 1 -- 172.21.15.98:0/1489038263 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fd8fc0cfdc0 con 0x7fd900102880 2024-05-09T08:51:17.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.044+0000 7fd8f7fff640 1 --2- 172.21.15.98:0/1489038263 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8e80a2eb0 0x7fd8e80a5370 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00096c0 tx=0x7fd8f0000bc0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.047+0000 7fd8f5ffb640 1 -- 172.21.15.98:0/1489038263 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd8fc020070 con 0x7fd900102880 2024-05-09T08:51:17.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.261+0000 7fd905d9e640 1 -- 172.21.15.98:0/1489038263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd900106cc0 con 0x7fd900102880 2024-05-09T08:51:17.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.261+0000 7fd8f5ffb640 1 -- 172.21.15.98:0/1489038263 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11940 (secure 0 0 0) 0x7fd8fc09c300 con 0x7fd900102880 2024-05-09T08:51:17.261 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:51:17.261 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:50:37.185367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T08:51:17.261 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-05-09T08:51:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.263+0000 7fd905d9e640 1 -- 172.21.15.98:0/1489038263 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8e80a2eb0 msgr2=0x7fd8e80a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.263+0000 7fd905d9e640 1 --2- 172.21.15.98:0/1489038263 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8e80a2eb0 0x7fd8e80a5370 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00096c0 tx=0x7fd8f0000bc0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.263+0000 7fd905d9e640 1 -- 172.21.15.98:0/1489038263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd900102880 msgr2=0x7fd9001004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.263+0000 7fd905d9e640 1 --2- 172.21.15.98:0/1489038263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd900102880 0x7fd9001004d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00db80 tx=0x7fd8fc00b630 comp rx=0 tx=0).stop 2024-05-09T08:51:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.263+0000 7fd905d9e640 1 -- 172.21.15.98:0/1489038263 shutdown_connections 2024-05-09T08:51:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.263+0000 7fd905d9e640 1 --2- 172.21.15.98:0/1489038263 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8e80a2eb0 0x7fd8e80a5370 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.263+0000 7fd905d9e640 1 --2- 172.21.15.98:0/1489038263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd90010bc50 0x7fd900100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.263+0000 7fd905d9e640 1 --2- 172.21.15.98:0/1489038263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd9001031a0 0x7fd900100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.263+0000 7fd905d9e640 1 --2- 172.21.15.98:0/1489038263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd900102880 0x7fd9001004d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.263+0000 7fd905d9e640 1 -- 172.21.15.98:0/1489038263 >> 172.21.15.98:0/1489038263 conn(0x7fd9000fc020 msgr2=0x7fd9000fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.264+0000 7fd905d9e640 1 -- 172.21.15.98:0/1489038263 shutdown_connections 2024-05-09T08:51:17.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.264+0000 7fd905d9e640 1 -- 172.21.15.98:0/1489038263 wait complete. 2024-05-09T08:51:17.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:51:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.381+0000 7f509554f640 1 -- 172.21.15.98:0/2626414253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5090106710 msgr2=0x7f5090106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.381+0000 7f509554f640 1 --2- 172.21.15.98:0/2626414253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5090106710 0x7f5090106af0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5080009930 tx=0x7f508002a420 comp rx=0 tx=0).stop 2024-05-09T08:51:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.381+0000 7f509554f640 1 -- 172.21.15.98:0/2626414253 shutdown_connections 2024-05-09T08:51:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.381+0000 7f509554f640 1 --2- 172.21.15.98:0/2626414253 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50901010c0 0x7f509010d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.381+0000 7f509554f640 1 --2- 172.21.15.98:0/2626414253 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090100700 0x7f5090100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.381+0000 7f509554f640 1 --2- 172.21.15.98:0/2626414253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5090106710 0x7f5090106af0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.381+0000 7f509554f640 1 -- 172.21.15.98:0/2626414253 >> 172.21.15.98:0/2626414253 conn(0x7f5090077830 msgr2=0x7f5090077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.381+0000 7f509554f640 1 -- 172.21.15.98:0/2626414253 shutdown_connections 2024-05-09T08:51:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.381+0000 7f509554f640 1 -- 172.21.15.98:0/2626414253 wait complete. 2024-05-09T08:51:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f509554f640 1 Processor -- start 2024-05-09T08:51:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f509554f640 1 -- start start 2024-05-09T08:51:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f509554f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090100700 0x7f5090198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f509554f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50901010c0 0x7f5090199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f509554f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5090106710 0x7f50901a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f509554f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5090110ae0 con 0x7f50901010c0 2024-05-09T08:51:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f509554f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5090110960 con 0x7f5090100700 2024-05-09T08:51:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f509554f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5090110c60 con 0x7f5090106710 2024-05-09T08:51:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f508e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50901010c0 0x7f5090199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f508e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50901010c0 0x7f5090199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57334/0 (socket says 172.21.15.98:57334) 2024-05-09T08:51:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f508e7fc640 1 -- 172.21.15.98:0/3313747024 learned_addr learned my addr 172.21.15.98:0/3313747024 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f508effd640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090100700 0x7f5090198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f508f7fe640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5090106710 0x7f50901a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f508e7fc640 1 -- 172.21.15.98:0/3313747024 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5090106710 msgr2=0x7f50901a3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f508e7fc640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5090106710 0x7f50901a3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f508e7fc640 1 -- 172.21.15.98:0/3313747024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090100700 msgr2=0x7f5090198f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f508e7fc640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090100700 0x7f5090198f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f508e7fc640 1 -- 172.21.15.98:0/3313747024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50901a4500 con 0x7f50901010c0 2024-05-09T08:51:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.382+0000 7f508effd640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090100700 0x7f5090198f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.383+0000 7f508e7fc640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50901010c0 0x7f5090199480 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f508400e9f0 tx=0x7f508400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.383+0000 7f5077fff640 1 -- 172.21.15.98:0/3313747024 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508400cd80 con 0x7f50901010c0 2024-05-09T08:51:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.383+0000 7f5077fff640 1 -- 172.21.15.98:0/3313747024 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5084010910 con 0x7f50901010c0 2024-05-09T08:51:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.383+0000 7f509554f640 1 -- 172.21.15.98:0/3313747024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50901a46d0 con 0x7f50901010c0 2024-05-09T08:51:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.383+0000 7f5077fff640 1 -- 172.21.15.98:0/3313747024 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5084018990 con 0x7f50901010c0 2024-05-09T08:51:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.383+0000 7f509554f640 1 -- 172.21.15.98:0/3313747024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50901a4bf0 con 0x7f50901010c0 2024-05-09T08:51:17.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.385+0000 7f5077fff640 1 -- 172.21.15.98:0/3313747024 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f5084010ac0 con 0x7f50901010c0 2024-05-09T08:51:17.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.385+0000 7f5098256740 1 Processor -- start 2024-05-09T08:51:17.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.385+0000 7f5098256740 1 -- start start 2024-05-09T08:51:17.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.385+0000 7f5098256740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 0x558ce83ad710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.385+0000 7f5098256740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83adc50 0x558ce83b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.385+0000 7f5098256740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83b8320 0x558ce83ba710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.385+0000 7f5098256740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558ce8312c40 con 0x558ce83b8320 2024-05-09T08:51:17.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.385+0000 7f5098256740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558ce82ba580 con 0x558ce83ad330 2024-05-09T08:51:17.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.385+0000 7f5098256740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558ce830ec00 con 0x558ce83adc50 2024-05-09T08:51:17.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f5077fff640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50780a2eb0 0x7f50780a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f508e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83b8320 0x558ce83ba710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f508e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83b8320 0x558ce83ba710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57336/0 (socket says 172.21.15.98:57336) 2024-05-09T08:51:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f508e7fc640 1 -- 172.21.15.98:0/3366777761 learned_addr learned my addr 172.21.15.98:0/3366777761 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f508f7fe640 1 --2- 172.21.15.98:0/3366777761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 0x558ce83ad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f508f7fe640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50780a2eb0 0x7f50780a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f508effd640 1 --2- 172.21.15.98:0/3366777761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83adc50 0x558ce83b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f5077fff640 1 -- 172.21.15.98:0/3313747024 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f50840cff40 con 0x7f50901010c0 2024-05-09T08:51:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f508e7fc640 1 -- 172.21.15.98:0/3366777761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83adc50 msgr2=0x558ce83b7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f508e7fc640 1 --2- 172.21.15.98:0/3366777761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83adc50 0x558ce83b7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f508e7fc640 1 -- 172.21.15.98:0/3366777761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 msgr2=0x558ce83ad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f508e7fc640 1 --2- 172.21.15.98:0/3366777761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 0x558ce83ad710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f508e7fc640 1 -- 172.21.15.98:0/3366777761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ce80263f0 con 0x558ce83b8320 2024-05-09T08:51:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f508f7fe640 1 --2- 172.21.15.98:0/3366777761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 0x558ce83ad710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f508e7fc640 1 --2- 172.21.15.98:0/3366777761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83b8320 0x558ce83ba710 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f5084098830 tx=0x7f5084018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fbcf2581ca5714 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f508f7fe640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50780a2eb0 0x7f50780a5370 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5088005c40 tx=0x7f5088005bd0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f50757fa640 1 -- 172.21.15.98:0/3366777761 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50840a0210 con 0x558ce83b8320 2024-05-09T08:51:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f50757fa640 1 -- 172.21.15.98:0/3366777761 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50840a4ad0 con 0x558ce83b8320 2024-05-09T08:51:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.386+0000 7f50757fa640 1 -- 172.21.15.98:0/3366777761 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5084018b30 con 0x558ce83b8320 2024-05-09T08:51:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.387+0000 7f5098256740 1 -- 172.21.15.98:0/3366777761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83b8320 msgr2=0x558ce83ba710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.387+0000 7f5098256740 1 --2- 172.21.15.98:0/3366777761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83b8320 0x558ce83ba710 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f5084098830 tx=0x7f5084018420 comp rx=0 tx=0).stop 2024-05-09T08:51:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.387+0000 7f5098256740 1 -- 172.21.15.98:0/3366777761 shutdown_connections 2024-05-09T08:51:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.387+0000 7f5098256740 1 --2- 172.21.15.98:0/3366777761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83b8320 0x558ce83ba710 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.387+0000 7f5098256740 1 --2- 172.21.15.98:0/3366777761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83adc50 0x558ce83b7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.387+0000 7f5098256740 1 --2- 172.21.15.98:0/3366777761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 0x558ce83ad710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.387+0000 7f5098256740 1 -- 172.21.15.98:0/3366777761 >> 172.21.15.98:0/3366777761 conn(0x558ce83a9650 msgr2=0x558ce83a9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.387+0000 7f5098256740 1 -- 172.21.15.98:0/3366777761 shutdown_connections 2024-05-09T08:51:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.387+0000 7f5098256740 1 -- 172.21.15.98:0/3366777761 wait complete. 2024-05-09T08:51:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.387+0000 7f5098256740 1 -- start start 2024-05-09T08:51:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.387+0000 7f5098256740 1 Processor -- start 2024-05-09T08:51:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f5098256740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83ad330 0x558ce83b52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f5098256740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83adc50 0x558ce83b5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f5098256740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83b8320 0x558ce83b5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f5098256740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558ce83bb0b0 con 0x558ce83ad330 2024-05-09T08:51:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f5098256740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558ce82cdc40 con 0x558ce83adc50 2024-05-09T08:51:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f5098256740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558ce83bb230 con 0x558ce83b8320 2024-05-09T08:51:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f5098256740 10 client.0 _ll_register_callbacks cb 0x558ce8025be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f508effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83ad330 0x558ce83b52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f508effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83ad330 0x558ce83b52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57340/0 (socket says 172.21.15.98:57340) 2024-05-09T08:51:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f508effd640 1 -- 172.21.15.98:0/3275766719 learned_addr learned my addr 172.21.15.98:0/3275766719 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f508f7fe640 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83b8320 0x558ce83b5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f508e7fc640 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83adc50 0x558ce83b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f508effd640 1 -- 172.21.15.98:0/3275766719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83b8320 msgr2=0x558ce83b5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f508effd640 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83b8320 0x558ce83b5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f508effd640 1 -- 172.21.15.98:0/3275766719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83adc50 msgr2=0x558ce83b5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f508effd640 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83adc50 0x558ce83b5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f508effd640 1 -- 172.21.15.98:0/3275766719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ce80f07d0 con 0x558ce83ad330 2024-05-09T08:51:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f508e7fc640 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83adc50 0x558ce83b5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f508f7fe640 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83b8320 0x558ce83b5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f508effd640 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83ad330 0x558ce83b52e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f508002a940 tx=0x7f5080031860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b940e06fbf9cfb97 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f5072ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f5072ff5640 1 -- 172.21.15.98:0/3275766719 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5080035030 con 0x558ce83ad330 2024-05-09T08:51:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f5072ff5640 1 -- 172.21.15.98:0/3275766719 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f5080004460 con 0x558ce83ad330 2024-05-09T08:51:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f5098256740 1 -- 172.21.15.98:0/3275766719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50840a4ad0 con 0x558ce83ad330 2024-05-09T08:51:17.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f5072ff5640 1 -- 172.21.15.98:0/3275766719 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5080005030 con 0x558ce83ad330 2024-05-09T08:51:17.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f5098256740 10 client.15897 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:17.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.388+0000 7f5098256740 1 -- 172.21.15.98:0/3275766719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558ce83c72f0 con 0x558ce83ad330 2024-05-09T08:51:17.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.389+0000 7f5072ff5640 1 -- 172.21.15.98:0/3275766719 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50840a4ad0 con 0x558ce83ad330 2024-05-09T08:51:17.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.389+0000 7f5072ff5640 1 -- 172.21.15.98:0/3275766719 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f50800035f0 con 0x558ce83ad330 2024-05-09T08:51:17.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.389+0000 7f5098256740 10 client.15897 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:17.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.389+0000 7f5098256740 10 client.15897 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:51:17.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.389+0000 7f5098256740 20 client.15897 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:17.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.389+0000 7f5098256740 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x558ce83c8800 0x558ce83cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.389+0000 7f5098256740 4 client.15897 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:17.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.389+0000 7f5098256740 1 -- 172.21.15.98:0/3275766719 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ce8320370 con 0x558ce83c8800 2024-05-09T08:51:17.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.389+0000 7f508e7fc640 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x558ce83c8800 0x558ce83cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.390+0000 7f5072ff5640 10 client.15897 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:51:17.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.390+0000 7f508e7fc640 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x558ce83c8800 0x558ce83cac50 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=66ccb1f2763c394a server_cookie=5e099bae9972f4a5 in_seq=0 out_seq=0 2024-05-09T08:51:17.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.390+0000 7f5072ff5640 1 -- 172.21.15.98:0/3275766719 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558ce8320370 con 0x558ce83c8800 2024-05-09T08:51:17.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.390+0000 7f5072ff5640 10 client.15897 handle_command_reply: tid=0 2024-05-09T08:51:17.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.390+0000 7f5098256740 1 client.15897 shutdown 2024-05-09T08:51:17.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.390+0000 7f5098256740 1 -- 172.21.15.98:0/3275766719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83ad330 msgr2=0x558ce83b52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.390+0000 7f5098256740 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83ad330 0x558ce83b52e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f508002a940 tx=0x7f5080031860 comp rx=0 tx=0).stop 2024-05-09T08:51:17.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.390+0000 7f5098256740 1 -- 172.21.15.98:0/3275766719 shutdown_connections 2024-05-09T08:51:17.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.391+0000 7f5098256740 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x558ce83c8800 0x558ce83cac50 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.391+0000 7f5098256740 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83b8320 0x558ce83b5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.391+0000 7f5098256740 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83adc50 0x558ce83b5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.391+0000 7f5072ff5640 0 client.15897 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:51:17.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.391+0000 7f5098256740 1 --2- 172.21.15.98:0/3275766719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83ad330 0x558ce83b52e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.391+0000 7f5098256740 1 -- 172.21.15.98:0/3275766719 >> 172.21.15.98:0/3275766719 conn(0x558ce83a9650 msgr2=0x558ce83af450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.391+0000 7f5098256740 1 -- 172.21.15.98:0/3275766719 shutdown_connections 2024-05-09T08:51:17.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.391+0000 7f5098256740 1 -- 172.21.15.98:0/3275766719 wait complete. 2024-05-09T08:51:17.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.391+0000 7f5098256740 20 client.15897 trim_cache size 0 max 16384 2024-05-09T08:51:17.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f5098256740 1 Processor -- start 2024-05-09T08:51:17.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f5098256740 1 -- start start 2024-05-09T08:51:17.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f5098256740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 0x558ce83c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f5098256740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83adc50 0x558ce83b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f5098256740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83b8320 0x558ce83afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f5098256740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50901aed40 con 0x558ce83b8320 2024-05-09T08:51:17.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f5098256740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f50901b0880 con 0x558ce83ad330 2024-05-09T08:51:17.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f5098256740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f50901af410 con 0x558ce83adc50 2024-05-09T08:51:17.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f508f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83b8320 0x558ce83afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f508f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83b8320 0x558ce83afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57350/0 (socket says 172.21.15.98:57350) 2024-05-09T08:51:17.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f508f7fe640 1 -- 172.21.15.98:0/1743073908 learned_addr learned my addr 172.21.15.98:0/1743073908 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f508e7fc640 1 --2- 172.21.15.98:0/1743073908 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83adc50 0x558ce83b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f508f7fe640 1 -- 172.21.15.98:0/1743073908 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83adc50 msgr2=0x558ce83b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f508f7fe640 1 --2- 172.21.15.98:0/1743073908 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83adc50 0x558ce83b42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f508f7fe640 1 -- 172.21.15.98:0/1743073908 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 msgr2=0x558ce83c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f508f7fe640 1 --2- 172.21.15.98:0/1743073908 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 0x558ce83c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f508f7fe640 1 -- 172.21.15.98:0/1743073908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ce839ad20 con 0x558ce83b8320 2024-05-09T08:51:17.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f508f7fe640 1 --2- 172.21.15.98:0/1743073908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83b8320 0x558ce83afa90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5088009ec0 tx=0x7f50880096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1ac3d485f61f4c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f5074ff9640 1 -- 172.21.15.98:0/1743073908 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5088025070 con 0x558ce83b8320 2024-05-09T08:51:17.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f5074ff9640 1 -- 172.21.15.98:0/1743073908 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5088010c30 con 0x558ce83b8320 2024-05-09T08:51:17.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.404+0000 7f5074ff9640 1 -- 172.21.15.98:0/1743073908 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508802add0 con 0x558ce83b8320 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.405+0000 7f5098256740 1 -- 172.21.15.98:0/1743073908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83b8320 msgr2=0x558ce83afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.405+0000 7f5098256740 1 --2- 172.21.15.98:0/1743073908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83b8320 0x558ce83afa90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5088009ec0 tx=0x7f50880096c0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.405+0000 7f5098256740 1 -- 172.21.15.98:0/1743073908 shutdown_connections 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.405+0000 7f5098256740 1 --2- 172.21.15.98:0/1743073908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83b8320 0x558ce83afa90 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.405+0000 7f5098256740 1 --2- 172.21.15.98:0/1743073908 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83adc50 0x558ce83b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.405+0000 7f5098256740 1 --2- 172.21.15.98:0/1743073908 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 0x558ce83c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.405+0000 7f5098256740 1 -- 172.21.15.98:0/1743073908 >> 172.21.15.98:0/1743073908 conn(0x558ce83a9650 msgr2=0x558ce83b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.405+0000 7f5098256740 1 -- 172.21.15.98:0/1743073908 shutdown_connections 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.405+0000 7f5098256740 1 -- 172.21.15.98:0/1743073908 wait complete. 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.405+0000 7f5098256740 1 -- start start 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.405+0000 7f5098256740 1 Processor -- start 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5098256740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 0x558ce83c0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5098256740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83adc50 0x558ce83bbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5098256740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83b8320 0x558ce83bbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5098256740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50901b01b0 con 0x558ce83adc50 2024-05-09T08:51:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5098256740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f50901afae0 con 0x558ce83ad330 2024-05-09T08:51:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5098256740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f50901acd20 con 0x558ce83b8320 2024-05-09T08:51:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5098256740 10 client.0 _ll_register_callbacks cb 0x7f5084005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f508e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83adc50 0x558ce83bbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f508e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83adc50 0x558ce83bbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57360/0 (socket says 172.21.15.98:57360) 2024-05-09T08:51:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f508e7fc640 1 -- 172.21.15.98:0/2228285677 learned_addr learned my addr 172.21.15.98:0/2228285677 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f508f7fe640 1 --2- 172.21.15.98:0/2228285677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83b8320 0x558ce83bbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f508effd640 1 --2- 172.21.15.98:0/2228285677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 0x558ce83c0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f508e7fc640 1 -- 172.21.15.98:0/2228285677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83b8320 msgr2=0x558ce83bbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f508e7fc640 1 --2- 172.21.15.98:0/2228285677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83b8320 0x558ce83bbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f508e7fc640 1 -- 172.21.15.98:0/2228285677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 msgr2=0x558ce83c0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f508e7fc640 1 --2- 172.21.15.98:0/2228285677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 0x558ce83c0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f508e7fc640 1 -- 172.21.15.98:0/2228285677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ce83bc6d0 con 0x558ce83adc50 2024-05-09T08:51:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f508effd640 1 --2- 172.21.15.98:0/2228285677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 0x558ce83c0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f508e7fc640 1 --2- 172.21.15.98:0/2228285677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83adc50 0x558ce83bbaa0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f508409a110 tx=0x7f5084000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb5f19dd13e710c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5072ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5072ff5640 1 -- 172.21.15.98:0/2228285677 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50840b8070 con 0x558ce83adc50 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5072ff5640 1 -- 172.21.15.98:0/2228285677 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f508409c2c0 con 0x558ce83adc50 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5098256740 1 -- 172.21.15.98:0/2228285677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5088010c30 con 0x558ce83adc50 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5072ff5640 1 -- 172.21.15.98:0/2228285677 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f50840a4ad0 con 0x558ce83adc50 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5098256740 10 client.15903 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5098256740 1 -- 172.21.15.98:0/2228285677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558ce842ecf0 con 0x558ce83adc50 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.406+0000 7f5072ff5640 1 -- 172.21.15.98:0/2228285677 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5088010c30 con 0x558ce83adc50 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.407+0000 7f5072ff5640 1 -- 172.21.15.98:0/2228285677 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f508409b2b0 con 0x558ce83adc50 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.407+0000 7f5098256740 10 client.15903 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.407+0000 7f5098256740 10 client.15903 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.407+0000 7f5098256740 20 client.15903 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.407+0000 7f5098256740 1 --2- 172.21.15.98:0/2228285677 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x558ce842f050 0x558ce8431470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.407+0000 7f5098256740 4 client.15903 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.407+0000 7f5098256740 1 -- 172.21.15.98:0/2228285677 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558ce831a6c0 con 0x558ce842f050 2024-05-09T08:51:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.407+0000 7f508effd640 1 --2- 172.21.15.98:0/2228285677 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x558ce842f050 0x558ce8431470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.407+0000 7f5072ff5640 10 client.15903 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.407+0000 7f508effd640 1 --2- 172.21.15.98:0/2228285677 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x558ce842f050 0x558ce8431470 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5db2af938580a1d8 server_cookie=9efd5bdf9b429d61 in_seq=0 out_seq=0 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5072ff5640 1 -- 172.21.15.98:0/2228285677 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558ce831a6c0 con 0x558ce842f050 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5072ff5640 10 client.15903 handle_command_reply: tid=0 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5098256740 1 client.15903 shutdown 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5098256740 1 -- 172.21.15.98:0/2228285677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83adc50 msgr2=0x558ce83bbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5098256740 1 --2- 172.21.15.98:0/2228285677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83adc50 0x558ce83bbaa0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f508409a110 tx=0x7f5084000bc0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5098256740 1 -- 172.21.15.98:0/2228285677 shutdown_connections 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5098256740 1 --2- 172.21.15.98:0/2228285677 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x558ce842f050 0x558ce8431470 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5098256740 1 --2- 172.21.15.98:0/2228285677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558ce83b8320 0x558ce83bbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5072ff5640 0 client.15903 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5098256740 1 --2- 172.21.15.98:0/2228285677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ce83adc50 0x558ce83bbaa0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5098256740 1 --2- 172.21.15.98:0/2228285677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ce83ad330 0x558ce83c0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5098256740 1 -- 172.21.15.98:0/2228285677 >> 172.21.15.98:0/2228285677 conn(0x558ce83a9650 msgr2=0x558ce83b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5098256740 1 -- 172.21.15.98:0/2228285677 shutdown_connections 2024-05-09T08:51:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5098256740 1 -- 172.21.15.98:0/2228285677 wait complete. 2024-05-09T08:51:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f5098256740 20 client.15903 trim_cache size 0 max 16384 2024-05-09T08:51:17.421 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:51:17.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f509554f640 1 -- 172.21.15.98:0/3313747024 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50780a2eb0 msgr2=0x7f50780a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f509554f640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50780a2eb0 0x7f50780a5370 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5088005c40 tx=0x7f5088005bd0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f509554f640 1 -- 172.21.15.98:0/3313747024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50901010c0 msgr2=0x7f5090199480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f509554f640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50901010c0 0x7f5090199480 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f508400e9f0 tx=0x7f508400eeb0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f509554f640 1 -- 172.21.15.98:0/3313747024 shutdown_connections 2024-05-09T08:51:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f509554f640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50780a2eb0 0x7f50780a5370 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f509554f640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5090106710 0x7f50901a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f509554f640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50901010c0 0x7f5090199480 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f509554f640 1 --2- 172.21.15.98:0/3313747024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090100700 0x7f5090198f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.408+0000 7f509554f640 1 -- 172.21.15.98:0/3313747024 >> 172.21.15.98:0/3313747024 conn(0x7f5090077830 msgr2=0x7f5090102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.409+0000 7f509554f640 1 -- 172.21.15.98:0/3313747024 shutdown_connections 2024-05-09T08:51:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.409+0000 7f509554f640 1 -- 172.21.15.98:0/3313747024 wait complete. 2024-05-09T08:51:17.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:51:17.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.526+0000 7f2deb34f640 1 -- 172.21.15.98:0/1137010966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de410bdf0 msgr2=0x7f2de410e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.526+0000 7f2deb34f640 1 --2- 172.21.15.98:0/1137010966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de410bdf0 0x7f2de410e290 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc009900 tx=0x7f2ddc02a510 comp rx=0 tx=0).stop 2024-05-09T08:51:17.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.526+0000 7f2deb34f640 1 -- 172.21.15.98:0/1137010966 shutdown_connections 2024-05-09T08:51:17.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.526+0000 7f2deb34f640 1 --2- 172.21.15.98:0/1137010966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de410bdf0 0x7f2de410e290 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.526+0000 7f2deb34f640 1 --2- 172.21.15.98:0/1137010966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2de41033d0 0x7f2de410b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.526+0000 7f2deb34f640 1 --2- 172.21.15.98:0/1137010966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2de4102ab0 0x7f2de4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.526+0000 7f2deb34f640 1 -- 172.21.15.98:0/1137010966 >> 172.21.15.98:0/1137010966 conn(0x7f2de4077830 msgr2=0x7f2de4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2deb34f640 1 -- 172.21.15.98:0/1137010966 shutdown_connections 2024-05-09T08:51:17.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2deb34f640 1 -- 172.21.15.98:0/1137010966 wait complete. 2024-05-09T08:51:17.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2deb34f640 1 Processor -- start 2024-05-09T08:51:17.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2deb34f640 1 -- start start 2024-05-09T08:51:17.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2deb34f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2de4102ab0 0x7f2de41980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2deb34f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2de41033d0 0x7f2de4198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2deb34f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de410bdf0 0x7f2de419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2deb34f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2de4111bf0 con 0x7f2de410bdf0 2024-05-09T08:51:17.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2deb34f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2de4111a70 con 0x7f2de4102ab0 2024-05-09T08:51:17.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2deb34f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2de4111d70 con 0x7f2de41033d0 2024-05-09T08:51:17.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2de98ec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de410bdf0 0x7f2de419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2de98ec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de410bdf0 0x7f2de419c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57380/0 (socket says 172.21.15.98:57380) 2024-05-09T08:51:17.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2de98ec640 1 -- 172.21.15.98:0/1605593527 learned_addr learned my addr 172.21.15.98:0/1605593527 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2de88ea640 1 --2- 172.21.15.98:0/1605593527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2de41033d0 0x7f2de4198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.527+0000 7f2de90eb640 1 --2- 172.21.15.98:0/1605593527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2de4102ab0 0x7f2de41980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.528+0000 7f2de98ec640 1 -- 172.21.15.98:0/1605593527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2de41033d0 msgr2=0x7f2de4198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.528+0000 7f2de98ec640 1 --2- 172.21.15.98:0/1605593527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2de41033d0 0x7f2de4198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.528+0000 7f2de98ec640 1 -- 172.21.15.98:0/1605593527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2de4102ab0 msgr2=0x7f2de41980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.528+0000 7f2de98ec640 1 --2- 172.21.15.98:0/1605593527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2de4102ab0 0x7f2de41980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.528+0000 7f2de98ec640 1 -- 172.21.15.98:0/1605593527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de419d090 con 0x7f2de410bdf0 2024-05-09T08:51:17.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.528+0000 7f2de98ec640 1 --2- 172.21.15.98:0/1605593527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de410bdf0 0x7f2de419c960 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc005ab0 tx=0x7f2ddc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.528+0000 7f2de27fc640 1 -- 172.21.15.98:0/1605593527 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc03f030 con 0x7f2de410bdf0 2024-05-09T08:51:17.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.528+0000 7f2deb34f640 1 -- 172.21.15.98:0/1605593527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2de419d320 con 0x7f2de410bdf0 2024-05-09T08:51:17.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.528+0000 7f2de27fc640 1 -- 172.21.15.98:0/1605593527 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ddc003ce0 con 0x7f2de410bdf0 2024-05-09T08:51:17.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.528+0000 7f2de27fc640 1 -- 172.21.15.98:0/1605593527 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc035070 con 0x7f2de410bdf0 2024-05-09T08:51:17.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.528+0000 7f2deb34f640 1 -- 172.21.15.98:0/1605593527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2de41a4c10 con 0x7f2de410bdf0 2024-05-09T08:51:17.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2de27fc640 1 -- 172.21.15.98:0/1605593527 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f2ddc042030 con 0x7f2de410bdf0 2024-05-09T08:51:17.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2dee057740 1 Processor -- start 2024-05-09T08:51:17.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2dee057740 1 -- start start 2024-05-09T08:51:17.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2dee057740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27a8330 0x55f3e27a8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2dee057740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8c50 0x55f3e27b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2dee057740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27b3320 0x55f3e27b5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2dee057740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f3e270dc40 con 0x55f3e27a8c50 2024-05-09T08:51:17.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2dee057740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f3e26b5580 con 0x55f3e27b3320 2024-05-09T08:51:17.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2dee057740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f3e2709c00 con 0x55f3e27a8330 2024-05-09T08:51:17.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2de90eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8c50 0x55f3e27b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2de90eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8c50 0x55f3e27b2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57388/0 (socket says 172.21.15.98:57388) 2024-05-09T08:51:17.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2de90eb640 1 -- 172.21.15.98:0/2286734814 learned_addr learned my addr 172.21.15.98:0/2286734814 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2de88ea640 1 --2- 172.21.15.98:0/2286734814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27a8330 0x55f3e27a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2de27fc640 1 --2- 172.21.15.98:0/1605593527 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2dcc0a2eb0 0x7f2dcc0a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2de90eb640 1 -- 172.21.15.98:0/2286734814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27a8330 msgr2=0x55f3e27a8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2de90eb640 1 --2- 172.21.15.98:0/2286734814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27a8330 0x55f3e27a8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2de90eb640 1 -- 172.21.15.98:0/2286734814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27b3320 msgr2=0x55f3e27b5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2de90eb640 1 --2- 172.21.15.98:0/2286734814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27b3320 0x55f3e27b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2de90eb640 1 -- 172.21.15.98:0/2286734814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3e24213f0 con 0x55f3e27a8c50 2024-05-09T08:51:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.531+0000 7f2de98ec640 1 --2- 172.21.15.98:0/1605593527 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2dcc0a2eb0 0x7f2dcc0a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.532+0000 7f2de27fc640 1 -- 172.21.15.98:0/1605593527 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f2ddc0f0220 con 0x7f2de410bdf0 2024-05-09T08:51:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.532+0000 7f2de90eb640 1 --2- 172.21.15.98:0/2286734814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8c50 0x55f3e27b2de0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40039f0 tx=0x7f2dd4003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41a1edb4d59b1d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.532+0000 7f2dcb7fe640 1 -- 172.21.15.98:0/2286734814 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4004b20 con 0x55f3e27a8c50 2024-05-09T08:51:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.532+0000 7f2dcb7fe640 1 -- 172.21.15.98:0/2286734814 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dd4004cc0 con 0x55f3e27a8c50 2024-05-09T08:51:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.532+0000 7f2dcb7fe640 1 -- 172.21.15.98:0/2286734814 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4012790 con 0x55f3e27a8c50 2024-05-09T08:51:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.532+0000 7f2dee057740 1 -- 172.21.15.98:0/2286734814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8c50 msgr2=0x55f3e27b2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.532+0000 7f2de98ec640 1 --2- 172.21.15.98:0/1605593527 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2dcc0a2eb0 0x7f2dcc0a5370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc0ed3d0 tx=0x7f2ddc08e1b0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.532+0000 7f2dee057740 1 --2- 172.21.15.98:0/2286734814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8c50 0x55f3e27b2de0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40039f0 tx=0x7f2dd4003eb0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.532+0000 7f2dee057740 1 -- 172.21.15.98:0/2286734814 shutdown_connections 2024-05-09T08:51:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.532+0000 7f2dee057740 1 --2- 172.21.15.98:0/2286734814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27b3320 0x55f3e27b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.532+0000 7f2dee057740 1 --2- 172.21.15.98:0/2286734814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8c50 0x55f3e27b2de0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.532+0000 7f2dee057740 1 --2- 172.21.15.98:0/2286734814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27a8330 0x55f3e27a8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.532+0000 7f2dee057740 1 -- 172.21.15.98:0/2286734814 >> 172.21.15.98:0/2286734814 conn(0x55f3e27a4650 msgr2=0x55f3e27a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.532+0000 7f2dee057740 1 -- 172.21.15.98:0/2286734814 shutdown_connections 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.533+0000 7f2dee057740 1 -- 172.21.15.98:0/2286734814 wait complete. 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.533+0000 7f2dee057740 1 -- start start 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.533+0000 7f2dee057740 1 Processor -- start 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.533+0000 7f2dee057740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27a8330 0x55f3e27b0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.533+0000 7f2dee057740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27a8c50 0x55f3e27b0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.533+0000 7f2dee057740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27b3320 0x55f3e27b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.533+0000 7f2dee057740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f3e27b6110 con 0x55f3e27b3320 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.533+0000 7f2dee057740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f3e26c8c40 con 0x55f3e27a8330 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.533+0000 7f2dee057740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f3e27b6290 con 0x55f3e27a8c50 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.533+0000 7f2dee057740 10 client.0 _ll_register_callbacks cb 0x55f3e2420be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.533+0000 7f2de88ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27b3320 0x55f3e27b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.533+0000 7f2de88ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27b3320 0x55f3e27b0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57398/0 (socket says 172.21.15.98:57398) 2024-05-09T08:51:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.533+0000 7f2de90eb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27a8c50 0x55f3e27b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.533+0000 7f2de88ea640 1 -- 172.21.15.98:0/2949434465 learned_addr learned my addr 172.21.15.98:0/2949434465 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2de88ea640 1 --2- 172.21.15.98:0/2949434465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27a8330 0x55f3e27b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2de90eb640 1 -- 172.21.15.98:0/2949434465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27a8330 msgr2=0x55f3e27b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2de90eb640 1 --2- 172.21.15.98:0/2949434465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27a8330 0x55f3e27b0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2de90eb640 1 -- 172.21.15.98:0/2949434465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27b3320 msgr2=0x55f3e27b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2de90eb640 1 --2- 172.21.15.98:0/2949434465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27b3320 0x55f3e27b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2de90eb640 1 -- 172.21.15.98:0/2949434465 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3e24eb7d0 con 0x55f3e27a8c50 2024-05-09T08:51:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2de88ea640 1 --2- 172.21.15.98:0/2949434465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27b3320 0x55f3e27b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2de90eb640 1 --2- 172.21.15.98:0/2949434465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27a8c50 0x55f3e27b0790 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40039f0 tx=0x7f2dd4009c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73e2a887fbc40e6d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2dc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:51:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2dc8ff9640 1 -- 172.21.15.98:0/2949434465 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4012960 con 0x55f3e27a8c50 2024-05-09T08:51:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2dee057740 1 -- 172.21.15.98:0/2949434465 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dd4004cc0 con 0x55f3e27a8c50 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2dc8ff9640 1 -- 172.21.15.98:0/2949434465 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f2dd4012b00 con 0x55f3e27a8c50 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2dc8ff9640 1 -- 172.21.15.98:0/2949434465 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2dd4012d00 con 0x55f3e27a8c50 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2dee057740 10 client.25285 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.534+0000 7f2dee057740 1 -- 172.21.15.98:0/2949434465 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f3e27c22b0 con 0x55f3e27a8c50 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.535+0000 7f2dc8ff9640 1 -- 172.21.15.98:0/2949434465 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dd4004cc0 con 0x55f3e27a8c50 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.535+0000 7f2dc8ff9640 1 -- 172.21.15.98:0/2949434465 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2dd400f040 con 0x55f3e27a8c50 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.535+0000 7f2dee057740 10 client.25285 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.535+0000 7f2dee057740 10 client.25285 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.535+0000 7f2dee057740 20 client.25285 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.535+0000 7f2dee057740 1 --2- 172.21.15.98:0/2949434465 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e27c3660 0x55f3e27c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.535+0000 7f2dee057740 4 client.25285 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.535+0000 7f2dee057740 1 -- 172.21.15.98:0/2949434465 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3e271b370 con 0x55f3e27c3660 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.535+0000 7f2de88ea640 1 --2- 172.21.15.98:0/2949434465 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e27c3660 0x55f3e27c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.536+0000 7f2dc8ff9640 10 client.25285 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:51:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.536+0000 7f2de88ea640 1 --2- 172.21.15.98:0/2949434465 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e27c3660 0x55f3e27c5a50 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=45282ccd0c8b27cb server_cookie=89e602fa900564e7 in_seq=0 out_seq=0 2024-05-09T08:51:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.536+0000 7f2dc8ff9640 1 -- 172.21.15.98:0/2949434465 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f3e271b370 con 0x55f3e27c3660 2024-05-09T08:51:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.536+0000 7f2dc8ff9640 10 client.25285 handle_command_reply: tid=0 2024-05-09T08:51:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.536+0000 7f2dee057740 1 client.25285 shutdown 2024-05-09T08:51:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.537+0000 7f2dee057740 1 -- 172.21.15.98:0/2949434465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27a8c50 msgr2=0x55f3e27b0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.537+0000 7f2dee057740 1 --2- 172.21.15.98:0/2949434465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27a8c50 0x55f3e27b0790 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40039f0 tx=0x7f2dd4009c30 comp rx=0 tx=0).stop 2024-05-09T08:51:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.537+0000 7f2dee057740 1 -- 172.21.15.98:0/2949434465 shutdown_connections 2024-05-09T08:51:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.537+0000 7f2dee057740 1 --2- 172.21.15.98:0/2949434465 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e27c3660 0x55f3e27c5a50 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.537+0000 7f2dee057740 1 --2- 172.21.15.98:0/2949434465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27b3320 0x55f3e27b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.537+0000 7f2dc8ff9640 0 client.25285 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:51:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.537+0000 7f2dee057740 1 --2- 172.21.15.98:0/2949434465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27a8c50 0x55f3e27b0790 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.537+0000 7f2dee057740 1 --2- 172.21.15.98:0/2949434465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27a8330 0x55f3e27b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.537+0000 7f2dee057740 1 -- 172.21.15.98:0/2949434465 >> 172.21.15.98:0/2949434465 conn(0x55f3e27a4650 msgr2=0x55f3e27aa450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.537+0000 7f2dee057740 1 -- 172.21.15.98:0/2949434465 shutdown_connections 2024-05-09T08:51:17.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.537+0000 7f2dee057740 1 -- 172.21.15.98:0/2949434465 wait complete. 2024-05-09T08:51:17.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.537+0000 7f2dee057740 20 client.25285 trim_cache size 0 max 16384 2024-05-09T08:51:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2dee057740 1 Processor -- start 2024-05-09T08:51:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2dee057740 1 -- start start 2024-05-09T08:51:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2dee057740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27a8330 0x55f3e27befd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2dee057740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8c50 0x55f3e27af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2dee057740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27b3320 0x55f3e27cb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2dee057740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2de41aedd0 con 0x55f3e27a8c50 2024-05-09T08:51:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2dee057740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2de41b0910 con 0x55f3e27a8330 2024-05-09T08:51:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2dee057740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2de41af4a0 con 0x55f3e27b3320 2024-05-09T08:51:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2de90eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8c50 0x55f3e27af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2de90eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8c50 0x55f3e27af3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57402/0 (socket says 172.21.15.98:57402) 2024-05-09T08:51:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2de90eb640 1 -- 172.21.15.98:0/4096973407 learned_addr learned my addr 172.21.15.98:0/4096973407 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2de90eb640 1 -- 172.21.15.98:0/4096973407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27b3320 msgr2=0x55f3e27cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2de90eb640 1 --2- 172.21.15.98:0/4096973407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27b3320 0x55f3e27cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2de90eb640 1 -- 172.21.15.98:0/4096973407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27a8330 msgr2=0x55f3e27befd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2de90eb640 1 --2- 172.21.15.98:0/4096973407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27a8330 0x55f3e27befd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2de90eb640 1 -- 172.21.15.98:0/4096973407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3e27a8a80 con 0x55f3e27a8c50 2024-05-09T08:51:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2de90eb640 1 --2- 172.21.15.98:0/4096973407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8c50 0x55f3e27af3b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4009e30 tx=0x7f2dd4005ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff3866bda2e4c4da server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.550+0000 7f2dcaffd640 1 -- 172.21.15.98:0/4096973407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4004740 con 0x55f3e27a8c50 2024-05-09T08:51:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.551+0000 7f2dcaffd640 1 -- 172.21.15.98:0/4096973407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dd401e040 con 0x55f3e27a8c50 2024-05-09T08:51:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.551+0000 7f2dcaffd640 1 -- 172.21.15.98:0/4096973407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd40117e0 con 0x55f3e27a8c50 2024-05-09T08:51:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.551+0000 7f2dee057740 1 -- 172.21.15.98:0/4096973407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8c50 msgr2=0x55f3e27af3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.551+0000 7f2dee057740 1 --2- 172.21.15.98:0/4096973407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8c50 0x55f3e27af3b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4009e30 tx=0x7f2dd4005ae0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.551+0000 7f2dee057740 1 -- 172.21.15.98:0/4096973407 shutdown_connections 2024-05-09T08:51:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.551+0000 7f2dee057740 1 --2- 172.21.15.98:0/4096973407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27b3320 0x55f3e27cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.551+0000 7f2dee057740 1 --2- 172.21.15.98:0/4096973407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8c50 0x55f3e27af3b0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.551+0000 7f2dee057740 1 --2- 172.21.15.98:0/4096973407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27a8330 0x55f3e27befd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.551+0000 7f2dee057740 1 -- 172.21.15.98:0/4096973407 >> 172.21.15.98:0/4096973407 conn(0x55f3e27a4650 msgr2=0x55f3e27a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.551+0000 7f2dee057740 1 -- 172.21.15.98:0/4096973407 shutdown_connections 2024-05-09T08:51:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.551+0000 7f2dee057740 1 -- 172.21.15.98:0/4096973407 wait complete. 2024-05-09T08:51:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.551+0000 7f2dee057740 1 -- start start 2024-05-09T08:51:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2dee057740 1 Processor -- start 2024-05-09T08:51:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2dee057740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8330 0x55f3e27b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2dee057740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27a8c50 0x55f3e27b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2dee057740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27b3320 0x55f3e2827ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2dee057740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2de41b0240 con 0x55f3e27a8330 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2dee057740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2de41afb70 con 0x55f3e27a8c50 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2dee057740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2de41acdb0 con 0x55f3e27b3320 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2dee057740 10 client.0 _ll_register_callbacks cb 0x7f2dd4009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2de88ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8330 0x55f3e27b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2de88ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8330 0x55f3e27b64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57406/0 (socket says 172.21.15.98:57406) 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2de88ea640 1 -- 172.21.15.98:0/3755752240 learned_addr learned my addr 172.21.15.98:0/3755752240 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2de90eb640 1 --2- 172.21.15.98:0/3755752240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27a8c50 0x55f3e27b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2de88ea640 1 -- 172.21.15.98:0/3755752240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27b3320 msgr2=0x55f3e2827ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2de88ea640 1 --2- 172.21.15.98:0/3755752240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27b3320 0x55f3e2827ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2de88ea640 1 -- 172.21.15.98:0/3755752240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27a8c50 msgr2=0x55f3e27b69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2de88ea640 1 --2- 172.21.15.98:0/3755752240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27a8c50 0x55f3e27b69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2de88ea640 1 -- 172.21.15.98:0/3755752240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3e2827ff0 con 0x55f3e27a8330 2024-05-09T08:51:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2de88ea640 1 --2- 172.21.15.98:0/3755752240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8330 0x55f3e27b64a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8005940 tx=0x7f2dd800fb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d89c59a49f1f095 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.552+0000 7f2dc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2dc8ff9640 1 -- 172.21.15.98:0/3755752240 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8025060 con 0x55f3e27a8330 2024-05-09T08:51:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2dc8ff9640 1 -- 172.21.15.98:0/3755752240 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f2dd8020ae0 con 0x55f3e27a8330 2024-05-09T08:51:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2dee057740 1 -- 172.21.15.98:0/3755752240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dd401e040 con 0x55f3e27a8330 2024-05-09T08:51:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2dc8ff9640 1 -- 172.21.15.98:0/3755752240 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2dd802c050 con 0x55f3e27a8330 2024-05-09T08:51:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2dee057740 10 client.15921 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2dee057740 1 -- 172.21.15.98:0/3755752240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f3e2829240 con 0x55f3e27a8330 2024-05-09T08:51:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2dc8ff9640 1 -- 172.21.15.98:0/3755752240 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dd401e040 con 0x55f3e27a8330 2024-05-09T08:51:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2dc8ff9640 1 -- 172.21.15.98:0/3755752240 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2dd8020ce0 con 0x55f3e27a8330 2024-05-09T08:51:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2dee057740 10 client.15921 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2dee057740 10 client.15921 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:51:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2dee057740 20 client.15921 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2dee057740 1 --2- 172.21.15.98:0/3755752240 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e28299a0 0x55f3e282bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2dee057740 4 client.15921 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2dee057740 1 -- 172.21.15.98:0/3755752240 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3e27156c0 con 0x55f3e28299a0 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.553+0000 7f2de90eb640 1 --2- 172.21.15.98:0/3755752240 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e28299a0 0x55f3e282bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2de90eb640 1 --2- 172.21.15.98:0/3755752240 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e28299a0 0x55f3e282bdc0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2737e12cee62d03 server_cookie=b9bf9496ba6d045f in_seq=0 out_seq=0 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dc8ff9640 10 client.15921 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dc8ff9640 1 -- 172.21.15.98:0/3755752240 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3e27156c0 con 0x55f3e28299a0 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dc8ff9640 10 client.15921 handle_command_reply: tid=0 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dee057740 1 client.15921 shutdown 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dee057740 1 -- 172.21.15.98:0/3755752240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8330 msgr2=0x55f3e27b64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dee057740 1 --2- 172.21.15.98:0/3755752240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8330 0x55f3e27b64a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8005940 tx=0x7f2dd800fb90 comp rx=0 tx=0).stop 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dee057740 1 -- 172.21.15.98:0/3755752240 shutdown_connections 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dee057740 1 --2- 172.21.15.98:0/3755752240 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e28299a0 0x55f3e282bdc0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dee057740 1 --2- 172.21.15.98:0/3755752240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e27b3320 0x55f3e2827ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dc8ff9640 0 client.15921 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dee057740 1 --2- 172.21.15.98:0/3755752240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e27a8c50 0x55f3e27b69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dee057740 1 --2- 172.21.15.98:0/3755752240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e27a8330 0x55f3e27b64a0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dee057740 1 -- 172.21.15.98:0/3755752240 >> 172.21.15.98:0/3755752240 conn(0x55f3e27a4650 msgr2=0x55f3e27a37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dee057740 1 -- 172.21.15.98:0/3755752240 shutdown_connections 2024-05-09T08:51:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dee057740 1 -- 172.21.15.98:0/3755752240 wait complete. 2024-05-09T08:51:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.554+0000 7f2dee057740 20 client.15921 trim_cache size 0 max 16384 2024-05-09T08:51:17.566 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:51:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.555+0000 7f2deb34f640 1 -- 172.21.15.98:0/1605593527 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2dcc0a2eb0 msgr2=0x7f2dcc0a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.555+0000 7f2deb34f640 1 --2- 172.21.15.98:0/1605593527 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2dcc0a2eb0 0x7f2dcc0a5370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc0ed3d0 tx=0x7f2ddc08e1b0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.555+0000 7f2deb34f640 1 -- 172.21.15.98:0/1605593527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de410bdf0 msgr2=0x7f2de419c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.555+0000 7f2deb34f640 1 --2- 172.21.15.98:0/1605593527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de410bdf0 0x7f2de419c960 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc005ab0 tx=0x7f2ddc0042f0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.555+0000 7f2deb34f640 1 -- 172.21.15.98:0/1605593527 shutdown_connections 2024-05-09T08:51:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.555+0000 7f2deb34f640 1 --2- 172.21.15.98:0/1605593527 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2dcc0a2eb0 0x7f2dcc0a5370 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.555+0000 7f2deb34f640 1 --2- 172.21.15.98:0/1605593527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de410bdf0 0x7f2de419c960 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.555+0000 7f2deb34f640 1 --2- 172.21.15.98:0/1605593527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2de41033d0 0x7f2de4198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.555+0000 7f2deb34f640 1 --2- 172.21.15.98:0/1605593527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2de4102ab0 0x7f2de41980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.555+0000 7f2deb34f640 1 -- 172.21.15.98:0/1605593527 >> 172.21.15.98:0/1605593527 conn(0x7f2de4077830 msgr2=0x7f2de40fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.555+0000 7f2deb34f640 1 -- 172.21.15.98:0/1605593527 shutdown_connections 2024-05-09T08:51:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.555+0000 7f2deb34f640 1 -- 172.21.15.98:0/1605593527 wait complete. 2024-05-09T08:51:17.568 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:17 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1489038263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:51:17.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:51:17.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:17 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1489038263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:51:17.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.673+0000 7facb8fa2640 1 -- 172.21.15.98:0/3197412946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4101270 msgr2=0x7facb41016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.673+0000 7facb8fa2640 1 --2- 172.21.15.98:0/3197412946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4101270 0x7facb41016f0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009930 tx=0x7fac9c02a420 comp rx=0 tx=0).stop 2024-05-09T08:51:17.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.673+0000 7facb8fa2640 1 -- 172.21.15.98:0/3197412946 shutdown_connections 2024-05-09T08:51:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.673+0000 7facb8fa2640 1 --2- 172.21.15.98:0/3197412946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facb4101c30 0x7facb410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.673+0000 7facb8fa2640 1 --2- 172.21.15.98:0/3197412946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4101270 0x7facb41016f0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.673+0000 7facb8fa2640 1 --2- 172.21.15.98:0/3197412946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7facb4105700 0x7facb4105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.673+0000 7facb8fa2640 1 -- 172.21.15.98:0/3197412946 >> 172.21.15.98:0/3197412946 conn(0x7facb40759c0 msgr2=0x7facb4075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.673+0000 7facb8fa2640 1 -- 172.21.15.98:0/3197412946 shutdown_connections 2024-05-09T08:51:17.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.673+0000 7facb8fa2640 1 -- 172.21.15.98:0/3197412946 wait complete. 2024-05-09T08:51:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facb8fa2640 1 Processor -- start 2024-05-09T08:51:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facb8fa2640 1 -- start start 2024-05-09T08:51:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facb8fa2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7facb4101270 0x7facb4196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facb8fa2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4101c30 0x7facb41974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facb8fa2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facb4105700 0x7facb419b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facb8fa2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7facb4110b30 con 0x7facb4101c30 2024-05-09T08:51:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facb8fa2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7facb41109b0 con 0x7facb4101270 2024-05-09T08:51:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facb8fa2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7facb4110cb0 con 0x7facb4105700 2024-05-09T08:51:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4101c30 0x7facb41974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4101c30 0x7facb41974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57428/0 (socket says 172.21.15.98:57428) 2024-05-09T08:51:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facabfff640 1 -- 172.21.15.98:0/4120635753 learned_addr learned my addr 172.21.15.98:0/4120635753 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facb2d9d640 1 --2- 172.21.15.98:0/4120635753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facb4105700 0x7facb419b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facb259c640 1 --2- 172.21.15.98:0/4120635753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7facb4101270 0x7facb4196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facabfff640 1 -- 172.21.15.98:0/4120635753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facb4105700 msgr2=0x7facb419b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facabfff640 1 --2- 172.21.15.98:0/4120635753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facb4105700 0x7facb419b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facabfff640 1 -- 172.21.15.98:0/4120635753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7facb4101270 msgr2=0x7facb4196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facabfff640 1 --2- 172.21.15.98:0/4120635753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7facb4101270 0x7facb4196f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facabfff640 1 -- 172.21.15.98:0/4120635753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb419bf60 con 0x7facb4101c30 2024-05-09T08:51:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.674+0000 7facabfff640 1 --2- 172.21.15.98:0/4120635753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4101c30 0x7facb41974b0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fac9c004770 tx=0x7fac9c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.675+0000 7facab7fe640 1 -- 172.21.15.98:0/4120635753 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9c035070 con 0x7facb4101c30 2024-05-09T08:51:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.675+0000 7facab7fe640 1 -- 172.21.15.98:0/4120635753 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac9c031a20 con 0x7facb4101c30 2024-05-09T08:51:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.675+0000 7facb8fa2640 1 -- 172.21.15.98:0/4120635753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facb419c1f0 con 0x7facb4101c30 2024-05-09T08:51:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.675+0000 7facab7fe640 1 -- 172.21.15.98:0/4120635753 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9c038a80 con 0x7facb4101c30 2024-05-09T08:51:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.675+0000 7facb8fa2640 1 -- 172.21.15.98:0/4120635753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facb4100160 con 0x7facb4101c30 2024-05-09T08:51:17.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.677+0000 7facab7fe640 1 -- 172.21.15.98:0/4120635753 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fac9c031bd0 con 0x7facb4101c30 2024-05-09T08:51:17.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.677+0000 7facbbd90740 1 Processor -- start 2024-05-09T08:51:17.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.677+0000 7facbbd90740 1 -- start start 2024-05-09T08:51:17.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.677+0000 7facab7fe640 1 --2- 172.21.15.98:0/4120635753 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faca00a2f30 0x7faca00a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facbbd90740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6b330 0x55694ef6b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facbbd90740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6bc50 0x55694ef75de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facbbd90740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 0x55694ef78710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facbbd90740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55694eed0c40 con 0x55694ef6bc50 2024-05-09T08:51:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facbbd90740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55694ee78580 con 0x55694ef76320 2024-05-09T08:51:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facbbd90740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55694eeccc00 con 0x55694ef6b330 2024-05-09T08:51:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facb2d9d640 1 --2- 172.21.15.98:0/4120635753 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faca00a2f30 0x7faca00a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6bc50 0x55694ef75de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6bc50 0x55694ef75de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57438/0 (socket says 172.21.15.98:57438) 2024-05-09T08:51:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facabfff640 1 -- 172.21.15.98:0/3266030693 learned_addr learned my addr 172.21.15.98:0/3266030693 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facab7fe640 1 -- 172.21.15.98:0/4120635753 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fac9c0f1ad0 con 0x7facb4101c30 2024-05-09T08:51:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facb259c640 1 --2- 172.21.15.98:0/3266030693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6b330 0x55694ef6b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facb2d9d640 1 --2- 172.21.15.98:0/3266030693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 0x55694ef78710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facabfff640 1 -- 172.21.15.98:0/3266030693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6b330 msgr2=0x55694ef6b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facabfff640 1 --2- 172.21.15.98:0/3266030693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6b330 0x55694ef6b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facabfff640 1 -- 172.21.15.98:0/3266030693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 msgr2=0x55694ef78710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facabfff640 1 --2- 172.21.15.98:0/3266030693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 0x55694ef78710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facabfff640 1 -- 172.21.15.98:0/3266030693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55694ebe43f0 con 0x55694ef6bc50 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facb259c640 1 --2- 172.21.15.98:0/3266030693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6b330 0x55694ef6b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facb2d9d640 1 --2- 172.21.15.98:0/4120635753 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faca00a2f30 0x7faca00a53f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7facac009fd0 tx=0x7facac013be0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7facabfff640 1 --2- 172.21.15.98:0/3266030693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6bc50 0x55694ef75de0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fac9c000c00 tx=0x7fac9c035860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2064d14a743e01ea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7faca8ff9640 1 -- 172.21.15.98:0/3266030693 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9c0bd530 con 0x55694ef6bc50 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7faca8ff9640 1 -- 172.21.15.98:0/3266030693 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac9c0be0f0 con 0x55694ef6bc50 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.678+0000 7faca8ff9640 1 -- 172.21.15.98:0/3266030693 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9c0bd350 con 0x55694ef6bc50 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.679+0000 7facbbd90740 1 -- 172.21.15.98:0/3266030693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6bc50 msgr2=0x55694ef75de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.679+0000 7facbbd90740 1 --2- 172.21.15.98:0/3266030693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6bc50 0x55694ef75de0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fac9c000c00 tx=0x7fac9c035860 comp rx=0 tx=0).stop 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.679+0000 7facbbd90740 1 -- 172.21.15.98:0/3266030693 shutdown_connections 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.679+0000 7facbbd90740 1 --2- 172.21.15.98:0/3266030693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 0x55694ef78710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.679+0000 7facbbd90740 1 --2- 172.21.15.98:0/3266030693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6bc50 0x55694ef75de0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.679+0000 7facbbd90740 1 --2- 172.21.15.98:0/3266030693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6b330 0x55694ef6b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.679+0000 7facbbd90740 1 -- 172.21.15.98:0/3266030693 >> 172.21.15.98:0/3266030693 conn(0x55694ef67650 msgr2=0x55694ef67a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.679+0000 7facbbd90740 1 -- 172.21.15.98:0/3266030693 shutdown_connections 2024-05-09T08:51:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.679+0000 7facbbd90740 1 -- 172.21.15.98:0/3266030693 wait complete. 2024-05-09T08:51:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.679+0000 7facbbd90740 1 -- start start 2024-05-09T08:51:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facbbd90740 1 Processor -- start 2024-05-09T08:51:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facbbd90740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6b330 0x55694ef73300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facbbd90740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6bc50 0x55694ef73840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facbbd90740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 0x55694ef73d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facbbd90740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55694ef78ed0 con 0x55694ef6b330 2024-05-09T08:51:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facbbd90740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55694ee8bc40 con 0x55694ef76320 2024-05-09T08:51:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facbbd90740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55694ef79050 con 0x55694ef6bc50 2024-05-09T08:51:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facb259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6b330 0x55694ef73300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facbbd90740 10 client.0 _ll_register_callbacks cb 0x55694ebe3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facb259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6b330 0x55694ef73300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57448/0 (socket says 172.21.15.98:57448) 2024-05-09T08:51:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facb259c640 1 -- 172.21.15.98:0/3804211693 learned_addr learned my addr 172.21.15.98:0/3804211693 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facabfff640 1 --2- 172.21.15.98:0/3804211693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6bc50 0x55694ef73840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facb259c640 1 -- 172.21.15.98:0/3804211693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6bc50 msgr2=0x55694ef73840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facb259c640 1 --2- 172.21.15.98:0/3804211693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6bc50 0x55694ef73840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facb259c640 1 -- 172.21.15.98:0/3804211693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 msgr2=0x55694ef73d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facb259c640 1 --2- 172.21.15.98:0/3804211693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 0x55694ef73d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facb259c640 1 -- 172.21.15.98:0/3804211693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55694ecae7d0 con 0x55694ef6b330 2024-05-09T08:51:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facb259c640 1 --2- 172.21.15.98:0/3804211693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6b330 0x55694ef73300 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7faca400be20 tx=0x7faca40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a8f0ba4a05092a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7fac99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7fac99ffb640 1 -- 172.21.15.98:0/3804211693 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca4012650 con 0x55694ef6b330 2024-05-09T08:51:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7fac99ffb640 1 -- 172.21.15.98:0/3804211693 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7faca40127f0 con 0x55694ef6b330 2024-05-09T08:51:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.680+0000 7facbbd90740 1 -- 172.21.15.98:0/3804211693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac9c0be0f0 con 0x55694ef6b330 2024-05-09T08:51:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.681+0000 7fac99ffb640 1 -- 172.21.15.98:0/3804211693 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faca4012a30 con 0x55694ef6b330 2024-05-09T08:51:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.681+0000 7facbbd90740 10 client.15933 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.681+0000 7facbbd90740 1 -- 172.21.15.98:0/3804211693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55694ef85070 con 0x55694ef6b330 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.681+0000 7fac99ffb640 1 -- 172.21.15.98:0/3804211693 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac9c0be0f0 con 0x55694ef6b330 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.681+0000 7fac99ffb640 1 -- 172.21.15.98:0/3804211693 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7faca401a3d0 con 0x55694ef6b330 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.681+0000 7facbbd90740 10 client.15933 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.681+0000 7facbbd90740 10 client.15933 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.681+0000 7facbbd90740 20 client.15933 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.681+0000 7facbbd90740 1 --2- 172.21.15.98:0/3804211693 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55694ef86580 0x55694ef889d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.681+0000 7facbbd90740 4 client.15933 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.681+0000 7facbbd90740 1 -- 172.21.15.98:0/3804211693 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55694eede370 con 0x55694ef86580 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.681+0000 7facabfff640 1 --2- 172.21.15.98:0/3804211693 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55694ef86580 0x55694ef889d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.682+0000 7fac99ffb640 10 client.15933 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.682+0000 7facabfff640 1 --2- 172.21.15.98:0/3804211693 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55694ef86580 0x55694ef889d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=516c094c51fe2fbb server_cookie=17363ecb77d0a2c9 in_seq=0 out_seq=0 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.682+0000 7fac99ffb640 1 -- 172.21.15.98:0/3804211693 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55694eede370 con 0x55694ef86580 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.682+0000 7fac99ffb640 10 client.15933 handle_command_reply: tid=0 2024-05-09T08:51:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.682+0000 7facbbd90740 1 client.15933 shutdown 2024-05-09T08:51:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.682+0000 7facbbd90740 1 -- 172.21.15.98:0/3804211693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6b330 msgr2=0x55694ef73300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.682+0000 7facbbd90740 1 --2- 172.21.15.98:0/3804211693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6b330 0x55694ef73300 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7faca400be20 tx=0x7faca40117b0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.682+0000 7facbbd90740 1 -- 172.21.15.98:0/3804211693 shutdown_connections 2024-05-09T08:51:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.682+0000 7facbbd90740 1 --2- 172.21.15.98:0/3804211693 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55694ef86580 0x55694ef889d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.682+0000 7facbbd90740 1 --2- 172.21.15.98:0/3804211693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 0x55694ef73d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.682+0000 7fac99ffb640 0 client.15933 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:51:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.682+0000 7facbbd90740 1 --2- 172.21.15.98:0/3804211693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6bc50 0x55694ef73840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.682+0000 7facbbd90740 1 --2- 172.21.15.98:0/3804211693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6b330 0x55694ef73300 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.682+0000 7facbbd90740 1 -- 172.21.15.98:0/3804211693 >> 172.21.15.98:0/3804211693 conn(0x55694ef67650 msgr2=0x55694ef6d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.683+0000 7facbbd90740 1 -- 172.21.15.98:0/3804211693 shutdown_connections 2024-05-09T08:51:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.683+0000 7facbbd90740 1 -- 172.21.15.98:0/3804211693 wait complete. 2024-05-09T08:51:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.683+0000 7facbbd90740 20 client.15933 trim_cache size 0 max 16384 2024-05-09T08:51:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facbbd90740 1 Processor -- start 2024-05-09T08:51:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facbbd90740 1 -- start start 2024-05-09T08:51:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facbbd90740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6b330 0x55694ef81b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facbbd90740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef6bc50 0x55694ef67a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facbbd90740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef76320 0x55694ef6da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facbbd90740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7facb41acb10 con 0x55694ef76320 2024-05-09T08:51:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facbbd90740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7facb41ae650 con 0x55694ef6bc50 2024-05-09T08:51:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facbbd90740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7facb41ad1e0 con 0x55694ef6b330 2024-05-09T08:51:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facb2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef76320 0x55694ef6da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facb2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef76320 0x55694ef6da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57464/0 (socket says 172.21.15.98:57464) 2024-05-09T08:51:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facb2d9d640 1 -- 172.21.15.98:0/3550430423 learned_addr learned my addr 172.21.15.98:0/3550430423 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facb2d9d640 1 -- 172.21.15.98:0/3550430423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6b330 msgr2=0x55694ef81b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facb2d9d640 1 --2- 172.21.15.98:0/3550430423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6b330 0x55694ef81b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facb2d9d640 1 -- 172.21.15.98:0/3550430423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef6bc50 msgr2=0x55694ef67a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facb2d9d640 1 --2- 172.21.15.98:0/3550430423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef6bc50 0x55694ef67a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facb2d9d640 1 -- 172.21.15.98:0/3550430423 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55694ef722b0 con 0x55694ef76320 2024-05-09T08:51:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7facb2d9d640 1 --2- 172.21.15.98:0/3550430423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef76320 0x55694ef6da90 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7facac011c40 tx=0x7facac01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2c87575a2248e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.696+0000 7fac9bfff640 1 -- 172.21.15.98:0/3550430423 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac011c70 con 0x55694ef76320 2024-05-09T08:51:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.697+0000 7fac9bfff640 1 -- 172.21.15.98:0/3550430423 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facac011e10 con 0x55694ef76320 2024-05-09T08:51:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.697+0000 7fac9bfff640 1 -- 172.21.15.98:0/3550430423 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac01d7e0 con 0x55694ef76320 2024-05-09T08:51:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.697+0000 7facbbd90740 1 -- 172.21.15.98:0/3550430423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef76320 msgr2=0x55694ef6da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.697+0000 7facbbd90740 1 --2- 172.21.15.98:0/3550430423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef76320 0x55694ef6da90 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7facac011c40 tx=0x7facac01b000 comp rx=0 tx=0).stop 2024-05-09T08:51:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.697+0000 7facbbd90740 1 -- 172.21.15.98:0/3550430423 shutdown_connections 2024-05-09T08:51:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.697+0000 7facbbd90740 1 --2- 172.21.15.98:0/3550430423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef76320 0x55694ef6da90 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.697+0000 7facbbd90740 1 --2- 172.21.15.98:0/3550430423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef6bc50 0x55694ef67a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.697+0000 7facbbd90740 1 --2- 172.21.15.98:0/3550430423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6b330 0x55694ef81b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.697+0000 7facbbd90740 1 -- 172.21.15.98:0/3550430423 >> 172.21.15.98:0/3550430423 conn(0x55694ef67650 msgr2=0x55694ef667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.697+0000 7facbbd90740 1 -- 172.21.15.98:0/3550430423 shutdown_connections 2024-05-09T08:51:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.697+0000 7facbbd90740 1 -- 172.21.15.98:0/3550430423 wait complete. 2024-05-09T08:51:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.697+0000 7facbbd90740 1 -- start start 2024-05-09T08:51:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.697+0000 7facbbd90740 1 Processor -- start 2024-05-09T08:51:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facbbd90740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6b330 0x55694ef7e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facbbd90740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6bc50 0x55694ef6fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facbbd90740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 0x55694ef70580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facbbd90740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7facb41adf80 con 0x55694ef6bc50 2024-05-09T08:51:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facbbd90740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7facb41ad8b0 con 0x55694ef76320 2024-05-09T08:51:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facbbd90740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7facb41aaaf0 con 0x55694ef6b330 2024-05-09T08:51:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facbbd90740 10 client.0 _ll_register_callbacks cb 0x7faca4005f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6bc50 0x55694ef6fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6bc50 0x55694ef6fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57480/0 (socket says 172.21.15.98:57480) 2024-05-09T08:51:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facabfff640 1 -- 172.21.15.98:0/3510489907 learned_addr learned my addr 172.21.15.98:0/3510489907 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facabfff640 1 -- 172.21.15.98:0/3510489907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6b330 msgr2=0x55694ef7e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facabfff640 1 --2- 172.21.15.98:0/3510489907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6b330 0x55694ef7e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facabfff640 1 -- 172.21.15.98:0/3510489907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 msgr2=0x55694ef70580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facb2d9d640 1 --2- 172.21.15.98:0/3510489907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 0x55694ef70580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facabfff640 1 --2- 172.21.15.98:0/3510489907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 0x55694ef70580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facabfff640 1 -- 172.21.15.98:0/3510489907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55694efebb10 con 0x55694ef6bc50 2024-05-09T08:51:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facb2d9d640 1 --2- 172.21.15.98:0/3510489907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 0x55694ef70580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facabfff640 1 --2- 172.21.15.98:0/3510489907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6bc50 0x55694ef6fe40 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009a50 tx=0x7fac9c003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbff1c9363608b2a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7fac99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7fac99ffb640 1 -- 172.21.15.98:0/3510489907 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9c0bd9f0 con 0x55694ef6bc50 2024-05-09T08:51:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7fac99ffb640 1 -- 172.21.15.98:0/3510489907 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fac9c038c20 con 0x55694ef6bc50 2024-05-09T08:51:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facbbd90740 1 -- 172.21.15.98:0/3510489907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facac011e10 con 0x55694ef6bc50 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7fac99ffb640 1 -- 172.21.15.98:0/3510489907 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fac9c0be0f0 con 0x55694ef6bc50 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facbbd90740 10 client.15939 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.698+0000 7facbbd90740 1 -- 172.21.15.98:0/3510489907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55694efecc50 con 0x55694ef6bc50 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.699+0000 7fac99ffb640 1 -- 172.21.15.98:0/3510489907 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facac011e10 con 0x55694ef6bc50 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.699+0000 7fac99ffb640 1 -- 172.21.15.98:0/3510489907 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fac9c0c61d0 con 0x55694ef6bc50 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.699+0000 7facbbd90740 10 client.15939 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.699+0000 7facbbd90740 10 client.15939 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.699+0000 7facbbd90740 20 client.15939 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.699+0000 7facbbd90740 1 --2- 172.21.15.98:0/3510489907 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55694efed340 0x55694efef730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.699+0000 7facbbd90740 4 client.15939 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.699+0000 7facbbd90740 1 -- 172.21.15.98:0/3510489907 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55694eed86c0 con 0x55694efed340 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.699+0000 7facb259c640 1 --2- 172.21.15.98:0/3510489907 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55694efed340 0x55694efef730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.699+0000 7fac99ffb640 10 client.15939 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.699+0000 7facb259c640 1 --2- 172.21.15.98:0/3510489907 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55694efed340 0x55694efef730 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=406d11ba6f9a9672 server_cookie=948a48e633496a0a in_seq=0 out_seq=0 2024-05-09T08:51:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.699+0000 7fac99ffb640 1 -- 172.21.15.98:0/3510489907 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55694eed86c0 con 0x55694efed340 2024-05-09T08:51:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.699+0000 7fac99ffb640 10 client.15939 handle_command_reply: tid=0 2024-05-09T08:51:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.699+0000 7facbbd90740 1 client.15939 shutdown 2024-05-09T08:51:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facbbd90740 1 -- 172.21.15.98:0/3510489907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6bc50 msgr2=0x55694ef6fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facbbd90740 1 --2- 172.21.15.98:0/3510489907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6bc50 0x55694ef6fe40 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009a50 tx=0x7fac9c003c80 comp rx=0 tx=0).stop 2024-05-09T08:51:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facbbd90740 1 -- 172.21.15.98:0/3510489907 shutdown_connections 2024-05-09T08:51:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facbbd90740 1 --2- 172.21.15.98:0/3510489907 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55694efed340 0x55694efef730 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facbbd90740 1 --2- 172.21.15.98:0/3510489907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55694ef76320 0x55694ef70580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7fac99ffb640 0 client.15939 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:51:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facbbd90740 1 --2- 172.21.15.98:0/3510489907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55694ef6bc50 0x55694ef6fe40 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facbbd90740 1 --2- 172.21.15.98:0/3510489907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55694ef6b330 0x55694ef7e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facbbd90740 1 -- 172.21.15.98:0/3510489907 >> 172.21.15.98:0/3510489907 conn(0x55694ef67650 msgr2=0x55694ef667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facbbd90740 1 -- 172.21.15.98:0/3510489907 shutdown_connections 2024-05-09T08:51:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facbbd90740 1 -- 172.21.15.98:0/3510489907 wait complete. 2024-05-09T08:51:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facbbd90740 20 client.15939 trim_cache size 0 max 16384 2024-05-09T08:51:17.707 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:51:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facb8fa2640 1 -- 172.21.15.98:0/4120635753 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faca00a2f30 msgr2=0x7faca00a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facb8fa2640 1 --2- 172.21.15.98:0/4120635753 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faca00a2f30 0x7faca00a53f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7facac009fd0 tx=0x7facac013be0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facb8fa2640 1 -- 172.21.15.98:0/4120635753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4101c30 msgr2=0x7facb41974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facb8fa2640 1 --2- 172.21.15.98:0/4120635753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4101c30 0x7facb41974b0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fac9c004770 tx=0x7fac9c003b90 comp rx=0 tx=0).stop 2024-05-09T08:51:17.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facb8fa2640 1 -- 172.21.15.98:0/4120635753 shutdown_connections 2024-05-09T08:51:17.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facb8fa2640 1 --2- 172.21.15.98:0/4120635753 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faca00a2f30 0x7faca00a53f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facb8fa2640 1 --2- 172.21.15.98:0/4120635753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facb4105700 0x7facb419b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facb8fa2640 1 --2- 172.21.15.98:0/4120635753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4101c30 0x7facb41974b0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facb8fa2640 1 --2- 172.21.15.98:0/4120635753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7facb4101270 0x7facb4196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facb8fa2640 1 -- 172.21.15.98:0/4120635753 >> 172.21.15.98:0/4120635753 conn(0x7facb40759c0 msgr2=0x7facb40fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facb8fa2640 1 -- 172.21.15.98:0/4120635753 shutdown_connections 2024-05-09T08:51:17.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.700+0000 7facb8fa2640 1 -- 172.21.15.98:0/4120635753 wait complete. 2024-05-09T08:51:17.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:51:17.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:17 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1489038263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:51:17.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.814+0000 7fe8f7e72640 1 -- 172.21.15.98:0/3175447422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f01006f0 msgr2=0x7fe8f010e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.814+0000 7fe8f7e72640 1 --2- 172.21.15.98:0/3175447422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f01006f0 0x7fe8f010e6c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8009cd0 tx=0x7fe8e802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.814+0000 7fe8f7e72640 1 -- 172.21.15.98:0/3175447422 shutdown_connections 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.814+0000 7fe8f7e72640 1 --2- 172.21.15.98:0/3175447422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f01006f0 0x7fe8f010e6c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.814+0000 7fe8f7e72640 1 --2- 172.21.15.98:0/3175447422 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8f00ffd30 0x7fe8f01001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.814+0000 7fe8f7e72640 1 --2- 172.21.15.98:0/3175447422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8f0106810 0x7fe8f0106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.814+0000 7fe8f7e72640 1 -- 172.21.15.98:0/3175447422 >> 172.21.15.98:0/3175447422 conn(0x7fe8f0076cc0 msgr2=0x7fe8f00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.814+0000 7fe8f7e72640 1 -- 172.21.15.98:0/3175447422 shutdown_connections 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.814+0000 7fe8f7e72640 1 -- 172.21.15.98:0/3175447422 wait complete. 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.814+0000 7fe8f7e72640 1 Processor -- start 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.814+0000 7fe8f7e72640 1 -- start start 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f7e72640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8f00ffd30 0x7fe8f01981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f7e72640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8f01006f0 0x7fe8f0198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f7e72640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f0106810 0x7fe8f019cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f7e72640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f0111cf0 con 0x7fe8f0106810 2024-05-09T08:51:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f7e72640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f0111b70 con 0x7fe8f00ffd30 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f7e72640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f0111e70 con 0x7fe8f01006f0 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f640f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f0106810 0x7fe8f019cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f640f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f0106810 0x7fe8f019cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57494/0 (socket says 172.21.15.98:57494) 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f640f640 1 -- 172.21.15.98:0/3941111668 learned_addr learned my addr 172.21.15.98:0/3941111668 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f540d640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8f01006f0 0x7fe8f0198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f5c0e640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8f00ffd30 0x7fe8f01981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f640f640 1 -- 172.21.15.98:0/3941111668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8f01006f0 msgr2=0x7fe8f0198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f640f640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8f01006f0 0x7fe8f0198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f640f640 1 -- 172.21.15.98:0/3941111668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8f00ffd30 msgr2=0x7fe8f01981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f640f640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8f00ffd30 0x7fe8f01981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f640f640 1 -- 172.21.15.98:0/3941111668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f019d1e0 con 0x7fe8f0106810 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f5c0e640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8f00ffd30 0x7fe8f01981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f540d640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8f01006f0 0x7fe8f0198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f640f640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f0106810 0x7fe8f019cab0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8005cc0 tx=0x7fe8e8003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8e2ffd640 1 -- 172.21.15.98:0/3941111668 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e8004040 con 0x7fe8f0106810 2024-05-09T08:51:17.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8e2ffd640 1 -- 172.21.15.98:0/3941111668 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8e80046c0 con 0x7fe8f0106810 2024-05-09T08:51:17.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8f7e72640 1 -- 172.21.15.98:0/3941111668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8f019d470 con 0x7fe8f0106810 2024-05-09T08:51:17.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.815+0000 7fe8e2ffd640 1 -- 172.21.15.98:0/3941111668 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e8046480 con 0x7fe8f0106810 2024-05-09T08:51:17.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.816+0000 7fe8f7e72640 1 -- 172.21.15.98:0/3941111668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8f01a4d60 con 0x7fe8f0106810 2024-05-09T08:51:17.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.818+0000 7fe8e2ffd640 1 -- 172.21.15.98:0/3941111668 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fe8e80305a0 con 0x7fe8f0106810 2024-05-09T08:51:17.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.818+0000 7fe8fab7e740 1 Processor -- start 2024-05-09T08:51:17.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.818+0000 7fe8fab7e740 1 -- start start 2024-05-09T08:51:17.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.818+0000 7fe8fab7e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 0x560307bcf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.818+0000 7fe8fab7e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bcfc50 0x560307bd9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.818+0000 7fe8fab7e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bda320 0x560307bdc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.818+0000 7fe8fab7e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560307b34c40 con 0x560307bda320 2024-05-09T08:51:17.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.818+0000 7fe8fab7e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560307adc580 con 0x560307bcfc50 2024-05-09T08:51:17.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.818+0000 7fe8fab7e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x560307b30c00 con 0x560307bcf330 2024-05-09T08:51:17.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.818+0000 7fe8f540d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bda320 0x560307bdc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.818+0000 7fe8e2ffd640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8dc0a2eb0 0x7fe8dc0a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.818+0000 7fe8f540d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bda320 0x560307bdc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57510/0 (socket says 172.21.15.98:57510) 2024-05-09T08:51:17.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.818+0000 7fe8f540d640 1 -- 172.21.15.98:0/1785673776 learned_addr learned my addr 172.21.15.98:0/1785673776 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.819+0000 7fe8f5c0e640 1 --2- 172.21.15.98:0/1785673776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bcfc50 0x560307bd9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.819+0000 7fe8f540d640 1 --2- 172.21.15.98:0/1785673776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 0x560307bcf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.819+0000 7fe8f640f640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8dc0a2eb0 0x7fe8dc0a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.819+0000 7fe8e2ffd640 1 -- 172.21.15.98:0/3941111668 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fe8e802e030 con 0x7fe8f0106810 2024-05-09T08:51:17.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.819+0000 7fe8f540d640 1 -- 172.21.15.98:0/1785673776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 msgr2=0x560307bcf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.819+0000 7fe8f540d640 1 --2- 172.21.15.98:0/1785673776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 0x560307bcf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.819+0000 7fe8f540d640 1 -- 172.21.15.98:0/1785673776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bcfc50 msgr2=0x560307bd9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.819+0000 7fe8f540d640 1 --2- 172.21.15.98:0/1785673776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bcfc50 0x560307bd9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.819+0000 7fe8f540d640 1 -- 172.21.15.98:0/1785673776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603078483f0 con 0x560307bda320 2024-05-09T08:51:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.819+0000 7fe8f540d640 1 --2- 172.21.15.98:0/1785673776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bda320 0x560307bdc710 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40149b0 tx=0x7fe8e400b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7471dae7ebefbec0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.819+0000 7fe8f640f640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8dc0a2eb0 0x7fe8dc0a5370 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe8e80ee5b0 tx=0x7fe8e80ee040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.819+0000 7fe8dbfff640 1 -- 172.21.15.98:0/1785673776 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e4004030 con 0x560307bda320 2024-05-09T08:51:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.819+0000 7fe8dbfff640 1 -- 172.21.15.98:0/1785673776 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8e4005500 con 0x560307bda320 2024-05-09T08:51:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.819+0000 7fe8dbfff640 1 -- 172.21.15.98:0/1785673776 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e4004c00 con 0x560307bda320 2024-05-09T08:51:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.820+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1785673776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bda320 msgr2=0x560307bdc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.820+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1785673776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bda320 0x560307bdc710 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40149b0 tx=0x7fe8e400b540 comp rx=0 tx=0).stop 2024-05-09T08:51:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.820+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1785673776 shutdown_connections 2024-05-09T08:51:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.820+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1785673776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bda320 0x560307bdc710 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.820+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1785673776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bcfc50 0x560307bd9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.820+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1785673776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 0x560307bcf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.820+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1785673776 >> 172.21.15.98:0/1785673776 conn(0x560307bcb650 msgr2=0x560307bcba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.820+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1785673776 shutdown_connections 2024-05-09T08:51:17.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.820+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1785673776 wait complete. 2024-05-09T08:51:17.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.820+0000 7fe8fab7e740 1 -- start start 2024-05-09T08:51:17.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.820+0000 7fe8fab7e740 1 Processor -- start 2024-05-09T08:51:17.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8fab7e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 0x560307bd7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8fab7e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcfc50 0x560307bd7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8fab7e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bda320 0x560307bd7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8fab7e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560307bdd110 con 0x560307bcfc50 2024-05-09T08:51:17.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8fab7e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560307aefc40 con 0x560307bda320 2024-05-09T08:51:17.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8fab7e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x560307bdd290 con 0x560307bcf330 2024-05-09T08:51:17.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8fab7e740 10 client.0 _ll_register_callbacks cb 0x560307847be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:17.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8f5c0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcfc50 0x560307bd7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8f5c0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcfc50 0x560307bd7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57522/0 (socket says 172.21.15.98:57522) 2024-05-09T08:51:17.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8f5c0e640 1 -- 172.21.15.98:0/1155438647 learned_addr learned my addr 172.21.15.98:0/1155438647 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8f540d640 1 --2- 172.21.15.98:0/1155438647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bda320 0x560307bd7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8f540d640 1 --2- 172.21.15.98:0/1155438647 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 0x560307bd7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8f5c0e640 1 -- 172.21.15.98:0/1155438647 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 msgr2=0x560307bd7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8f5c0e640 1 --2- 172.21.15.98:0/1155438647 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 0x560307bd7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8f5c0e640 1 -- 172.21.15.98:0/1155438647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bda320 msgr2=0x560307bd7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8f5c0e640 1 --2- 172.21.15.98:0/1155438647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bda320 0x560307bd7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8f5c0e640 1 -- 172.21.15.98:0/1155438647 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603079127d0 con 0x560307bcfc50 2024-05-09T08:51:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8f540d640 1 --2- 172.21.15.98:0/1155438647 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 0x560307bd7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8f5c0e640 1 --2- 172.21.15.98:0/1155438647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcfc50 0x560307bd7790 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00d9d0 tx=0x7fe8ec00de90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e53b4499995f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8d97fa640 1 -- 172.21.15.98:0/1155438647 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec01b640 con 0x560307bcfc50 2024-05-09T08:51:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8d97fa640 1 -- 172.21.15.98:0/1155438647 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fe8ec011440 con 0x560307bcfc50 2024-05-09T08:51:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.821+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1155438647 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8e4005500 con 0x560307bcfc50 2024-05-09T08:51:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.822+0000 7fe8d97fa640 1 -- 172.21.15.98:0/1155438647 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe8ec01b7e0 con 0x560307bcfc50 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.822+0000 7fe8fab7e740 10 client.15951 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.822+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1155438647 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560307be93a0 con 0x560307bcfc50 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.822+0000 7fe8d97fa640 1 -- 172.21.15.98:0/1155438647 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8e4005500 con 0x560307bcfc50 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.822+0000 7fe8d97fa640 1 -- 172.21.15.98:0/1155438647 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe8ec019070 con 0x560307bcfc50 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.822+0000 7fe8fab7e740 10 client.15951 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.822+0000 7fe8fab7e740 10 client.15951 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.822+0000 7fe8fab7e740 20 client.15951 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.822+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1155438647 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x560307bea900 0x560307becd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.822+0000 7fe8fab7e740 4 client.15951 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.822+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1155438647 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560307b42370 con 0x560307bea900 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.822+0000 7fe8f540d640 1 --2- 172.21.15.98:0/1155438647 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x560307bea900 0x560307becd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.823+0000 7fe8d97fa640 10 client.15951 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.823+0000 7fe8f540d640 1 --2- 172.21.15.98:0/1155438647 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x560307bea900 0x560307becd40 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b9361af6b972fb92 server_cookie=62bb978308c38263 in_seq=0 out_seq=0 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.823+0000 7fe8d97fa640 1 -- 172.21.15.98:0/1155438647 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560307b42370 con 0x560307bea900 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.823+0000 7fe8d97fa640 10 client.15951 handle_command_reply: tid=0 2024-05-09T08:51:17.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.823+0000 7fe8fab7e740 1 client.15951 shutdown 2024-05-09T08:51:17.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.823+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1155438647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcfc50 msgr2=0x560307bd7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.823+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1155438647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcfc50 0x560307bd7790 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00d9d0 tx=0x7fe8ec00de90 comp rx=0 tx=0).stop 2024-05-09T08:51:17.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.823+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1155438647 shutdown_connections 2024-05-09T08:51:17.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.823+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1155438647 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x560307bea900 0x560307becd40 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.823+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1155438647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bda320 0x560307bd7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.823+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1155438647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcfc50 0x560307bd7790 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.823+0000 7fe8d97fa640 0 client.15951 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:51:17.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.823+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1155438647 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 0x560307bd7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.823+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1155438647 >> 172.21.15.98:0/1155438647 conn(0x560307bcb650 msgr2=0x560307bd1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.824+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1155438647 shutdown_connections 2024-05-09T08:51:17.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.824+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1155438647 wait complete. 2024-05-09T08:51:17.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.824+0000 7fe8fab7e740 20 client.15951 trim_cache size 0 max 16384 2024-05-09T08:51:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8fab7e740 1 Processor -- start 2024-05-09T08:51:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8fab7e740 1 -- start start 2024-05-09T08:51:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8fab7e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcf330 0x560307be6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8fab7e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bcfc50 0x560307bd1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8fab7e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bda320 0x560307bf2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8fab7e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f01aedc0 con 0x560307bcf330 2024-05-09T08:51:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8fab7e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f01b0900 con 0x560307bcfc50 2024-05-09T08:51:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8fab7e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f01af490 con 0x560307bda320 2024-05-09T08:51:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8f540d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcf330 0x560307be6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8f540d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcf330 0x560307be6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57532/0 (socket says 172.21.15.98:57532) 2024-05-09T08:51:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8f540d640 1 -- 172.21.15.98:0/3504301663 learned_addr learned my addr 172.21.15.98:0/3504301663 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8f5c0e640 1 --2- 172.21.15.98:0/3504301663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bcfc50 0x560307bd1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8f540d640 1 -- 172.21.15.98:0/3504301663 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bda320 msgr2=0x560307bf2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8f540d640 1 --2- 172.21.15.98:0/3504301663 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bda320 0x560307bf2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8f540d640 1 -- 172.21.15.98:0/3504301663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bcfc50 msgr2=0x560307bd1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8f540d640 1 --2- 172.21.15.98:0/3504301663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bcfc50 0x560307bd1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8f540d640 1 -- 172.21.15.98:0/3504301663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560307bc71a0 con 0x560307bcf330 2024-05-09T08:51:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8f5c0e640 1 --2- 172.21.15.98:0/3504301663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bcfc50 0x560307bd1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.837+0000 7fe8f540d640 1 --2- 172.21.15.98:0/3504301663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcf330 0x560307be6060 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40048b0 tx=0x7fe8e400b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709baa1e350e426c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.838+0000 7fe8db7fe640 1 -- 172.21.15.98:0/3504301663 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e4014550 con 0x560307bcf330 2024-05-09T08:51:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.838+0000 7fe8db7fe640 1 -- 172.21.15.98:0/3504301663 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8e402a880 con 0x560307bcf330 2024-05-09T08:51:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.838+0000 7fe8db7fe640 1 -- 172.21.15.98:0/3504301663 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e400fba0 con 0x560307bcf330 2024-05-09T08:51:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.838+0000 7fe8fab7e740 1 -- 172.21.15.98:0/3504301663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcf330 msgr2=0x560307be6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.838+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/3504301663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcf330 0x560307be6060 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40048b0 tx=0x7fe8e400b540 comp rx=0 tx=0).stop 2024-05-09T08:51:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.838+0000 7fe8fab7e740 1 -- 172.21.15.98:0/3504301663 shutdown_connections 2024-05-09T08:51:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.838+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/3504301663 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bda320 0x560307bf2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.838+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/3504301663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bcfc50 0x560307bd1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.838+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/3504301663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcf330 0x560307be6060 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.838+0000 7fe8fab7e740 1 -- 172.21.15.98:0/3504301663 >> 172.21.15.98:0/3504301663 conn(0x560307bcb650 msgr2=0x560307bca7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.838+0000 7fe8fab7e740 1 -- 172.21.15.98:0/3504301663 shutdown_connections 2024-05-09T08:51:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.838+0000 7fe8fab7e740 1 -- 172.21.15.98:0/3504301663 wait complete. 2024-05-09T08:51:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8fab7e740 1 -- start start 2024-05-09T08:51:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8fab7e740 1 Processor -- start 2024-05-09T08:51:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8fab7e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 0x560307be29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8fab7e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcfc50 0x560307bddd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8fab7e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bda320 0x560307bde2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8fab7e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f01b0230 con 0x560307bcfc50 2024-05-09T08:51:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8fab7e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f01afb60 con 0x560307bda320 2024-05-09T08:51:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8fab7e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f01acda0 con 0x560307bcf330 2024-05-09T08:51:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8fab7e740 10 client.0 _ll_register_callbacks cb 0x7fe8ec004100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8f5c0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcfc50 0x560307bddd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8f5c0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcfc50 0x560307bddd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57536/0 (socket says 172.21.15.98:57536) 2024-05-09T08:51:17.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8f5c0e640 1 -- 172.21.15.98:0/1205234916 learned_addr learned my addr 172.21.15.98:0/1205234916 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8f540d640 1 --2- 172.21.15.98:0/1205234916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 0x560307be29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8f5c0e640 1 -- 172.21.15.98:0/1205234916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 msgr2=0x560307be29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8f5c0e640 1 --2- 172.21.15.98:0/1205234916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 0x560307be29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8f5c0e640 1 -- 172.21.15.98:0/1205234916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bda320 msgr2=0x560307bde2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8f5c0e640 1 --2- 172.21.15.98:0/1205234916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bda320 0x560307bde2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8f5c0e640 1 -- 172.21.15.98:0/1205234916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560307bdeaf0 con 0x560307bcfc50 2024-05-09T08:51:17.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8f540d640 1 --2- 172.21.15.98:0/1205234916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 0x560307be29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:17.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8f5c0e640 1 --2- 172.21.15.98:0/1205234916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcfc50 0x560307bddd40 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec011f60 tx=0x7fe8ec004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108b1033f17d9da7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.839+0000 7fe8d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:17.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.840+0000 7fe8d97fa640 1 -- 172.21.15.98:0/1205234916 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec01b980 con 0x560307bcfc50 2024-05-09T08:51:17.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.840+0000 7fe8d97fa640 1 -- 172.21.15.98:0/1205234916 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fe8ec01bb20 con 0x560307bcfc50 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.840+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1205234916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8e402a880 con 0x560307bcfc50 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.840+0000 7fe8d97fa640 1 -- 172.21.15.98:0/1205234916 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe8ec036050 con 0x560307bcfc50 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.840+0000 7fe8fab7e740 10 client.15957 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.840+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1205234916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560307c50e90 con 0x560307bcfc50 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.840+0000 7fe8d97fa640 1 -- 172.21.15.98:0/1205234916 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8e402a880 con 0x560307bcfc50 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.840+0000 7fe8d97fa640 1 -- 172.21.15.98:0/1205234916 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe8ec031030 con 0x560307bcfc50 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.840+0000 7fe8fab7e740 10 client.15957 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.840+0000 7fe8fab7e740 10 client.15957 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.840+0000 7fe8fab7e740 20 client.15957 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.840+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1205234916 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x560307c51660 0x560307c53a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.840+0000 7fe8fab7e740 4 client.15957 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.840+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1205234916 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560307b3c6c0 con 0x560307c51660 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8f540d640 1 --2- 172.21.15.98:0/1205234916 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x560307c51660 0x560307c53a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8d97fa640 10 client.15957 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:51:17.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8f540d640 1 --2- 172.21.15.98:0/1205234916 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x560307c51660 0x560307c53a80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6c92e1bf8bfbf778 server_cookie=25da0152ffb2851e in_seq=0 out_seq=0 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8d97fa640 1 -- 172.21.15.98:0/1205234916 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560307b3c6c0 con 0x560307c51660 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8d97fa640 10 client.15957 handle_command_reply: tid=0 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8fab7e740 1 client.15957 shutdown 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1205234916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcfc50 msgr2=0x560307bddd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1205234916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcfc50 0x560307bddd40 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec011f60 tx=0x7fe8ec004230 comp rx=0 tx=0).stop 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1205234916 shutdown_connections 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1205234916 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x560307c51660 0x560307c53a80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1205234916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560307bda320 0x560307bde2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1205234916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560307bcfc50 0x560307bddd40 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8d97fa640 0 client.15957 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8fab7e740 1 --2- 172.21.15.98:0/1205234916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560307bcf330 0x560307be29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1205234916 >> 172.21.15.98:0/1205234916 conn(0x560307bcb650 msgr2=0x560307bd71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1205234916 shutdown_connections 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.841+0000 7fe8fab7e740 1 -- 172.21.15.98:0/1205234916 wait complete. 2024-05-09T08:51:17.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.842+0000 7fe8fab7e740 20 client.15957 trim_cache size 0 max 16384 2024-05-09T08:51:17.849 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:51:17.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.842+0000 7fe8f7e72640 1 -- 172.21.15.98:0/3941111668 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8dc0a2eb0 msgr2=0x7fe8dc0a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.842+0000 7fe8f7e72640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8dc0a2eb0 0x7fe8dc0a5370 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe8e80ee5b0 tx=0x7fe8e80ee040 comp rx=0 tx=0).stop 2024-05-09T08:51:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.842+0000 7fe8f7e72640 1 -- 172.21.15.98:0/3941111668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f0106810 msgr2=0x7fe8f019cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.842+0000 7fe8f7e72640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f0106810 0x7fe8f019cab0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8005cc0 tx=0x7fe8e8003970 comp rx=0 tx=0).stop 2024-05-09T08:51:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.842+0000 7fe8f7e72640 1 -- 172.21.15.98:0/3941111668 shutdown_connections 2024-05-09T08:51:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.842+0000 7fe8f7e72640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8dc0a2eb0 0x7fe8dc0a5370 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.842+0000 7fe8f7e72640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f0106810 0x7fe8f019cab0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.842+0000 7fe8f7e72640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8f01006f0 0x7fe8f0198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.842+0000 7fe8f7e72640 1 --2- 172.21.15.98:0/3941111668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8f00ffd30 0x7fe8f01981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.842+0000 7fe8f7e72640 1 -- 172.21.15.98:0/3941111668 >> 172.21.15.98:0/3941111668 conn(0x7fe8f0076cc0 msgr2=0x7fe8f0104930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.842+0000 7fe8f7e72640 1 -- 172.21.15.98:0/3941111668 shutdown_connections 2024-05-09T08:51:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.842+0000 7fe8f7e72640 1 -- 172.21.15.98:0/3941111668 wait complete. 2024-05-09T08:51:17.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:51:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.956+0000 7fa97357e640 1 -- 172.21.15.98:0/3529189737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c1010c0 msgr2=0x7fa96c10d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.956+0000 7fa97357e640 1 --2- 172.21.15.98:0/3529189737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c1010c0 0x7fa96c10d5d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa964009cd0 tx=0x7fa96402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.956+0000 7fa97357e640 1 -- 172.21.15.98:0/3529189737 shutdown_connections 2024-05-09T08:51:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.956+0000 7fa97357e640 1 --2- 172.21.15.98:0/3529189737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c1010c0 0x7fa96c10d5d0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.956+0000 7fa97357e640 1 --2- 172.21.15.98:0/3529189737 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96c100700 0x7fa96c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.956+0000 7fa97357e640 1 --2- 172.21.15.98:0/3529189737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa96c106710 0x7fa96c106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.956+0000 7fa97357e640 1 -- 172.21.15.98:0/3529189737 >> 172.21.15.98:0/3529189737 conn(0x7fa96c077830 msgr2=0x7fa96c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.956+0000 7fa97357e640 1 -- 172.21.15.98:0/3529189737 shutdown_connections 2024-05-09T08:51:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.956+0000 7fa97357e640 1 -- 172.21.15.98:0/3529189737 wait complete. 2024-05-09T08:51:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa97357e640 1 Processor -- start 2024-05-09T08:51:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa97357e640 1 -- start start 2024-05-09T08:51:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa97357e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa96c100700 0x7fa96c198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa97357e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96c1010c0 0x7fa96c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa97357e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c106710 0x7fa96c19ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa97357e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c110b20 con 0x7fa96c106710 2024-05-09T08:51:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa97357e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1109a0 con 0x7fa96c100700 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa97357e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c110ca0 con 0x7fa96c1010c0 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa971b1b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c106710 0x7fa96c19ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa971b1b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c106710 0x7fa96c19ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57558/0 (socket says 172.21.15.98:57558) 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa971b1b640 1 -- 172.21.15.98:0/2723969205 learned_addr learned my addr 172.21.15.98:0/2723969205 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa971b1b640 1 -- 172.21.15.98:0/2723969205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96c1010c0 msgr2=0x7fa96c1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa970b19640 1 --2- 172.21.15.98:0/2723969205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96c1010c0 0x7fa96c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa971b1b640 1 --2- 172.21.15.98:0/2723969205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96c1010c0 0x7fa96c1986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa971b1b640 1 -- 172.21.15.98:0/2723969205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa96c100700 msgr2=0x7fa96c198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa971b1b640 1 --2- 172.21.15.98:0/2723969205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa96c100700 0x7fa96c198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa971b1b640 1 -- 172.21.15.98:0/2723969205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa96c19d160 con 0x7fa96c106710 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa971b1b640 1 --2- 172.21.15.98:0/2723969205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c106710 0x7fa96c19ca30 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa964005cc0 tx=0x7fa964003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.957+0000 7fa95e7fc640 1 -- 172.21.15.98:0/2723969205 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa964004040 con 0x7fa96c106710 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.958+0000 7fa95e7fc640 1 -- 172.21.15.98:0/2723969205 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9640046c0 con 0x7fa96c106710 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.958+0000 7fa95e7fc640 1 -- 172.21.15.98:0/2723969205 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa964046480 con 0x7fa96c106710 2024-05-09T08:51:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.958+0000 7fa97357e640 1 -- 172.21.15.98:0/2723969205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa96c19d3f0 con 0x7fa96c106710 2024-05-09T08:51:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.958+0000 7fa97357e640 1 -- 172.21.15.98:0/2723969205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa96c1a4ce0 con 0x7fa96c106710 2024-05-09T08:51:17.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.960+0000 7fa95e7fc640 1 -- 172.21.15.98:0/2723969205 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fa9640305a0 con 0x7fa96c106710 2024-05-09T08:51:17.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.960+0000 7fa97636a740 1 Processor -- start 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.960+0000 7fa97636a740 1 -- start start 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.960+0000 7fa97636a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f5290330 0x5646f5290710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.960+0000 7fa97636a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290c50 0x5646f529ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa97636a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f529b320 0x5646f529d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa97636a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5646f51f5c40 con 0x5646f529b320 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa97636a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5646f519d580 con 0x5646f5290330 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa97636a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5646f51f1c00 con 0x5646f5290c50 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa95e7fc640 1 --2- 172.21.15.98:0/2723969205 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa9580a2eb0 0x7fa9580a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa970b19640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f529b320 0x5646f529d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa970b19640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f529b320 0x5646f529d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57562/0 (socket says 172.21.15.98:57562) 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa970b19640 1 -- 172.21.15.98:0/2266157430 learned_addr learned my addr 172.21.15.98:0/2266157430 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa971b1b640 1 --2- 172.21.15.98:0/2723969205 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa9580a2eb0 0x7fa9580a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa97131a640 1 --2- 172.21.15.98:0/2266157430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290c50 0x5646f529ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa970b19640 1 --2- 172.21.15.98:0/2266157430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f5290330 0x5646f5290710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa95e7fc640 1 -- 172.21.15.98:0/2723969205 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fa96402e030 con 0x7fa96c106710 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa970b19640 1 -- 172.21.15.98:0/2266157430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290c50 msgr2=0x5646f529ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa970b19640 1 --2- 172.21.15.98:0/2266157430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290c50 0x5646f529ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa970b19640 1 -- 172.21.15.98:0/2266157430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f5290330 msgr2=0x5646f5290710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa970b19640 1 --2- 172.21.15.98:0/2266157430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f5290330 0x5646f5290710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa970b19640 1 -- 172.21.15.98:0/2266157430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646f4f093f0 con 0x5646f529b320 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa970b19640 1 --2- 172.21.15.98:0/2266157430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f529b320 0x5646f529d710 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa960005fd0 tx=0x7fa96000e5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff23b64a869eea87 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa971b1b640 1 --2- 172.21.15.98:0/2723969205 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa9580a2eb0 0x7fa9580a5370 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa9640ee360 tx=0x7fa96403d040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa9577fe640 1 -- 172.21.15.98:0/2266157430 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960018030 con 0x5646f529b320 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa9577fe640 1 -- 172.21.15.98:0/2266157430 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa960005500 con 0x5646f529b320 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.961+0000 7fa9577fe640 1 -- 172.21.15.98:0/2266157430 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960010600 con 0x5646f529b320 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.962+0000 7fa97636a740 1 -- 172.21.15.98:0/2266157430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f529b320 msgr2=0x5646f529d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.962+0000 7fa97636a740 1 --2- 172.21.15.98:0/2266157430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f529b320 0x5646f529d710 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa960005fd0 tx=0x7fa96000e5b0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.962+0000 7fa97636a740 1 -- 172.21.15.98:0/2266157430 shutdown_connections 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.962+0000 7fa97636a740 1 --2- 172.21.15.98:0/2266157430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f529b320 0x5646f529d710 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.962+0000 7fa97636a740 1 --2- 172.21.15.98:0/2266157430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290c50 0x5646f529ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.962+0000 7fa97636a740 1 --2- 172.21.15.98:0/2266157430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f5290330 0x5646f5290710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.962+0000 7fa97636a740 1 -- 172.21.15.98:0/2266157430 >> 172.21.15.98:0/2266157430 conn(0x5646f528c650 msgr2=0x5646f528ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.962+0000 7fa97636a740 1 -- 172.21.15.98:0/2266157430 shutdown_connections 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.962+0000 7fa97636a740 1 -- 172.21.15.98:0/2266157430 wait complete. 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.962+0000 7fa97636a740 1 -- start start 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.962+0000 7fa97636a740 1 Processor -- start 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97636a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 0x5646f5298270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97636a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f5290c50 0x5646f52987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97636a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f529b320 0x5646f5298cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97636a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5646f529e110 con 0x5646f5290c50 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97636a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5646f51b0c40 con 0x5646f529b320 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97636a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5646f529e290 con 0x5646f5290330 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97636a740 10 client.0 _ll_register_callbacks cb 0x5646f4f08be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97131a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f5290c50 0x5646f52987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa970b19640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 0x5646f5298270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97131a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f5290c50 0x5646f52987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57574/0 (socket says 172.21.15.98:57574) 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97131a640 1 -- 172.21.15.98:0/136193268 learned_addr learned my addr 172.21.15.98:0/136193268 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa970b19640 1 --2- 172.21.15.98:0/136193268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f529b320 0x5646f5298cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97131a640 1 -- 172.21.15.98:0/136193268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 msgr2=0x5646f5298270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97131a640 1 --2- 172.21.15.98:0/136193268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 0x5646f5298270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97131a640 1 -- 172.21.15.98:0/136193268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f529b320 msgr2=0x5646f5298cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97131a640 1 --2- 172.21.15.98:0/136193268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f529b320 0x5646f5298cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97131a640 1 -- 172.21.15.98:0/136193268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646f4fd37d0 con 0x5646f5290c50 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa970b19640 1 --2- 172.21.15.98:0/136193268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 0x5646f5298270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa97131a640 1 --2- 172.21.15.98:0/136193268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f5290c50 0x5646f52987b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa968003900 tx=0x7fa968003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9528bfdaf81eaa6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.963+0000 7fa954ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.964+0000 7fa954ff9640 1 -- 172.21.15.98:0/136193268 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96801a470 con 0x5646f5290c50 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.964+0000 7fa954ff9640 1 -- 172.21.15.98:0/136193268 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fa96801a610 con 0x5646f5290c50 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.964+0000 7fa97636a740 1 -- 172.21.15.98:0/136193268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa960005500 con 0x5646f5290c50 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.964+0000 7fa954ff9640 1 -- 172.21.15.98:0/136193268 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa96801a810 con 0x5646f5290c50 2024-05-09T08:51:17.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.964+0000 7fa97636a740 10 client.15969 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.964+0000 7fa97636a740 1 -- 172.21.15.98:0/136193268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5646f52aa360 con 0x5646f5290c50 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.964+0000 7fa954ff9640 1 -- 172.21.15.98:0/136193268 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa960005500 con 0x5646f5290c50 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.964+0000 7fa954ff9640 1 -- 172.21.15.98:0/136193268 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa968011b50 con 0x5646f5290c50 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.964+0000 7fa97636a740 10 client.15969 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.964+0000 7fa97636a740 10 client.15969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.964+0000 7fa97636a740 20 client.15969 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.964+0000 7fa97636a740 1 --2- 172.21.15.98:0/136193268 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5646f52ab870 0x5646f52adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.964+0000 7fa97636a740 4 client.15969 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.964+0000 7fa97636a740 1 -- 172.21.15.98:0/136193268 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646f5203370 con 0x5646f52ab870 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.965+0000 7fa970b19640 1 --2- 172.21.15.98:0/136193268 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5646f52ab870 0x5646f52adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.965+0000 7fa970b19640 1 --2- 172.21.15.98:0/136193268 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5646f52ab870 0x5646f52adcb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8e77583840da114 server_cookie=76debeae57fbc34 in_seq=0 out_seq=0 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.965+0000 7fa954ff9640 10 client.15969 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa954ff9640 1 -- 172.21.15.98:0/136193268 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5646f5203370 con 0x5646f52ab870 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa954ff9640 10 client.15969 handle_command_reply: tid=0 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa97636a740 1 client.15969 shutdown 2024-05-09T08:51:17.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa97636a740 1 -- 172.21.15.98:0/136193268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f5290c50 msgr2=0x5646f52987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa97636a740 1 --2- 172.21.15.98:0/136193268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f5290c50 0x5646f52987b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa968003900 tx=0x7fa968003dc0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa97636a740 1 -- 172.21.15.98:0/136193268 shutdown_connections 2024-05-09T08:51:17.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa97636a740 1 --2- 172.21.15.98:0/136193268 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5646f52ab870 0x5646f52adcb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa97636a740 1 --2- 172.21.15.98:0/136193268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f529b320 0x5646f5298cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa954ff9640 0 client.15969 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:17.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa97636a740 1 --2- 172.21.15.98:0/136193268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f5290c50 0x5646f52987b0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa97636a740 1 --2- 172.21.15.98:0/136193268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 0x5646f5298270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa97636a740 1 -- 172.21.15.98:0/136193268 >> 172.21.15.98:0/136193268 conn(0x5646f528c650 msgr2=0x5646f5292450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa97636a740 1 -- 172.21.15.98:0/136193268 shutdown_connections 2024-05-09T08:51:17.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa97636a740 1 -- 172.21.15.98:0/136193268 wait complete. 2024-05-09T08:51:17.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.966+0000 7fa97636a740 20 client.15969 trim_cache size 0 max 16384 2024-05-09T08:51:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.979+0000 7fa97636a740 1 Processor -- start 2024-05-09T08:51:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.979+0000 7fa97636a740 1 -- start start 2024-05-09T08:51:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97636a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 0x5646f52a7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97636a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f5290c50 0x5646f5292a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97636a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f529b320 0x5646f5298b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97636a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1aed30 con 0x5646f5290c50 2024-05-09T08:51:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97636a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1b0870 con 0x5646f529b320 2024-05-09T08:51:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97636a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1af400 con 0x5646f5290330 2024-05-09T08:51:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97131a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f5290c50 0x5646f5292a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97131a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f5290c50 0x5646f5292a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57582/0 (socket says 172.21.15.98:57582) 2024-05-09T08:51:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97131a640 1 -- 172.21.15.98:0/3418539043 learned_addr learned my addr 172.21.15.98:0/3418539043 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa970b19640 1 --2- 172.21.15.98:0/3418539043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f529b320 0x5646f5298b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa970b19640 1 --2- 172.21.15.98:0/3418539043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 0x5646f52a7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97131a640 1 -- 172.21.15.98:0/3418539043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 msgr2=0x5646f52a7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97131a640 1 --2- 172.21.15.98:0/3418539043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 0x5646f52a7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97131a640 1 -- 172.21.15.98:0/3418539043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f529b320 msgr2=0x5646f5298b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97131a640 1 --2- 172.21.15.98:0/3418539043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f529b320 0x5646f5298b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97131a640 1 -- 172.21.15.98:0/3418539043 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646f5292fd0 con 0x5646f5290c50 2024-05-09T08:51:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa970b19640 1 --2- 172.21.15.98:0/3418539043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 0x5646f52a7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa97131a640 1 --2- 172.21.15.98:0/3418539043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f5290c50 0x5646f5292a90 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa968005c10 tx=0x7fa96800b350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad2c8133d503db7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa956ffd640 1 -- 172.21.15.98:0/3418539043 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96801a470 con 0x5646f5290c50 2024-05-09T08:51:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa956ffd640 1 -- 172.21.15.98:0/3418539043 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa968015070 con 0x5646f5290c50 2024-05-09T08:51:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.980+0000 7fa956ffd640 1 -- 172.21.15.98:0/3418539043 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968010c50 con 0x5646f5290c50 2024-05-09T08:51:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.981+0000 7fa97636a740 1 -- 172.21.15.98:0/3418539043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f5290c50 msgr2=0x5646f5292a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.981+0000 7fa97636a740 1 --2- 172.21.15.98:0/3418539043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f5290c50 0x5646f5292a90 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa968005c10 tx=0x7fa96800b350 comp rx=0 tx=0).stop 2024-05-09T08:51:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.981+0000 7fa97636a740 1 -- 172.21.15.98:0/3418539043 shutdown_connections 2024-05-09T08:51:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.981+0000 7fa97636a740 1 --2- 172.21.15.98:0/3418539043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f529b320 0x5646f5298b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.981+0000 7fa97636a740 1 --2- 172.21.15.98:0/3418539043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f5290c50 0x5646f5292a90 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.981+0000 7fa97636a740 1 --2- 172.21.15.98:0/3418539043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 0x5646f52a7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.981+0000 7fa97636a740 1 -- 172.21.15.98:0/3418539043 >> 172.21.15.98:0/3418539043 conn(0x5646f528c650 msgr2=0x5646f5296b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.981+0000 7fa97636a740 1 -- 172.21.15.98:0/3418539043 shutdown_connections 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.981+0000 7fa97636a740 1 -- 172.21.15.98:0/3418539043 wait complete. 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.981+0000 7fa97636a740 1 -- start start 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.981+0000 7fa97636a740 1 Processor -- start 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa97636a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 0x5646f52a3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa97636a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f5290c50 0x5646f52a3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa97636a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f529b320 0x5646f52946a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa97636a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1b01a0 con 0x5646f529b320 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa97636a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1afad0 con 0x5646f5290c50 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa97636a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1acd10 con 0x5646f5290330 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa97636a740 10 client.0 _ll_register_callbacks cb 0x7fa9680041c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa970b19640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f529b320 0x5646f52946a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa970b19640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f529b320 0x5646f52946a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57590/0 (socket says 172.21.15.98:57590) 2024-05-09T08:51:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa970b19640 1 -- 172.21.15.98:0/4198441530 learned_addr learned my addr 172.21.15.98:0/4198441530 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa970b19640 1 --2- 172.21.15.98:0/4198441530 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 0x5646f52a3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa97131a640 1 --2- 172.21.15.98:0/4198441530 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f5290c50 0x5646f52a3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa970b19640 1 -- 172.21.15.98:0/4198441530 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 msgr2=0x5646f52a3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa970b19640 1 --2- 172.21.15.98:0/4198441530 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 0x5646f52a3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa970b19640 1 -- 172.21.15.98:0/4198441530 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f5290c50 msgr2=0x5646f52a3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa970b19640 1 --2- 172.21.15.98:0/4198441530 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f5290c50 0x5646f52a3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa970b19640 1 -- 172.21.15.98:0/4198441530 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646f5294ea0 con 0x5646f529b320 2024-05-09T08:51:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa97131a640 1 --2- 172.21.15.98:0/4198441530 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f5290c50 0x5646f52a3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa970b19640 1 --2- 172.21.15.98:0/4198441530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f529b320 0x5646f52946a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa960004de0 tx=0x7fa96001f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e7de752fdc51a64 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa954ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.982+0000 7fa954ff9640 1 -- 172.21.15.98:0/4198441530 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960018040 con 0x5646f529b320 2024-05-09T08:51:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.983+0000 7fa954ff9640 1 -- 172.21.15.98:0/4198441530 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fa96002dae0 con 0x5646f529b320 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.983+0000 7fa97636a740 1 -- 172.21.15.98:0/4198441530 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa968015070 con 0x5646f529b320 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.983+0000 7fa954ff9640 1 -- 172.21.15.98:0/4198441530 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa96003e050 con 0x5646f529b320 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.983+0000 7fa97636a740 10 client.15975 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.983+0000 7fa97636a740 1 -- 172.21.15.98:0/4198441530 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5646f5311800 con 0x5646f529b320 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.983+0000 7fa954ff9640 1 -- 172.21.15.98:0/4198441530 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa968015070 con 0x5646f529b320 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.983+0000 7fa954ff9640 1 -- 172.21.15.98:0/4198441530 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa96000f040 con 0x5646f529b320 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.983+0000 7fa97636a740 10 client.15975 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.983+0000 7fa97636a740 10 client.15975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.983+0000 7fa97636a740 20 client.15975 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.983+0000 7fa97636a740 1 --2- 172.21.15.98:0/4198441530 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5646f5311e40 0x5646f5314260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.983+0000 7fa97636a740 4 client.15975 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.983+0000 7fa97636a740 1 -- 172.21.15.98:0/4198441530 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5646f51fd6c0 con 0x5646f5311e40 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.983+0000 7fa97131a640 1 --2- 172.21.15.98:0/4198441530 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5646f5311e40 0x5646f5314260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa97131a640 1 --2- 172.21.15.98:0/4198441530 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5646f5311e40 0x5646f5314260 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51c0922eac796765 server_cookie=61a4cebc8f6fe464 in_seq=0 out_seq=0 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa954ff9640 10 client.15975 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa954ff9640 1 -- 172.21.15.98:0/4198441530 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5646f51fd6c0 con 0x5646f5311e40 2024-05-09T08:51:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa954ff9640 10 client.15975 handle_command_reply: tid=0 2024-05-09T08:51:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa97636a740 1 client.15975 shutdown 2024-05-09T08:51:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa97636a740 1 -- 172.21.15.98:0/4198441530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f529b320 msgr2=0x5646f52946a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa97636a740 1 --2- 172.21.15.98:0/4198441530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f529b320 0x5646f52946a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa960004de0 tx=0x7fa96001f040 comp rx=0 tx=0).stop 2024-05-09T08:51:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa97636a740 1 -- 172.21.15.98:0/4198441530 shutdown_connections 2024-05-09T08:51:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa97636a740 1 --2- 172.21.15.98:0/4198441530 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5646f5311e40 0x5646f5314260 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa97636a740 1 --2- 172.21.15.98:0/4198441530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646f529b320 0x5646f52946a0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa97636a740 1 --2- 172.21.15.98:0/4198441530 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5646f5290c50 0x5646f52a3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa954ff9640 0 client.15975 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa97636a740 1 --2- 172.21.15.98:0/4198441530 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5646f5290330 0x5646f52a3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa97636a740 1 -- 172.21.15.98:0/4198441530 >> 172.21.15.98:0/4198441530 conn(0x5646f528c650 msgr2=0x5646f528af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa97636a740 1 -- 172.21.15.98:0/4198441530 shutdown_connections 2024-05-09T08:51:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa97636a740 1 -- 172.21.15.98:0/4198441530 wait complete. 2024-05-09T08:51:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.984+0000 7fa97636a740 20 client.15975 trim_cache size 0 max 16384 2024-05-09T08:51:17.991 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:51:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.985+0000 7fa97357e640 1 -- 172.21.15.98:0/2723969205 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa9580a2eb0 msgr2=0x7fa9580a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.985+0000 7fa97357e640 1 --2- 172.21.15.98:0/2723969205 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa9580a2eb0 0x7fa9580a5370 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa9640ee360 tx=0x7fa96403d040 comp rx=0 tx=0).stop 2024-05-09T08:51:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.985+0000 7fa97357e640 1 -- 172.21.15.98:0/2723969205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c106710 msgr2=0x7fa96c19ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.985+0000 7fa97357e640 1 --2- 172.21.15.98:0/2723969205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c106710 0x7fa96c19ca30 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa964005cc0 tx=0x7fa964003970 comp rx=0 tx=0).stop 2024-05-09T08:51:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.985+0000 7fa97357e640 1 -- 172.21.15.98:0/2723969205 shutdown_connections 2024-05-09T08:51:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.985+0000 7fa97357e640 1 --2- 172.21.15.98:0/2723969205 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa9580a2eb0 0x7fa9580a5370 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.985+0000 7fa97357e640 1 --2- 172.21.15.98:0/2723969205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c106710 0x7fa96c19ca30 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.985+0000 7fa97357e640 1 --2- 172.21.15.98:0/2723969205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96c1010c0 0x7fa96c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.985+0000 7fa97357e640 1 --2- 172.21.15.98:0/2723969205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa96c100700 0x7fa96c198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.985+0000 7fa97357e640 1 -- 172.21.15.98:0/2723969205 >> 172.21.15.98:0/2723969205 conn(0x7fa96c077830 msgr2=0x7fa96c068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.985+0000 7fa97357e640 1 -- 172.21.15.98:0/2723969205 shutdown_connections 2024-05-09T08:51:17.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:17.985+0000 7fa97357e640 1 -- 172.21.15.98:0/2723969205 wait complete. 2024-05-09T08:51:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:18 smithi142 ceph-mon[34625]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-05-09T08:51:18.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:18 smithi105 ceph-mon[34498]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-05-09T08:51:18.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:18 smithi098 ceph-mon[30092]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-05-09T08:51:18.995 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:51:18.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:51:18.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:51:19.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.113+0000 7f6a91f29640 1 -- 172.21.15.98:0/2827395764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c106810 msgr2=0x7f6a8c106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.113+0000 7f6a91f29640 1 --2- 172.21.15.98:0/2827395764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c106810 0x7f6a8c106bf0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c009cd0 tx=0x7f6a7c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.113+0000 7f6a91f29640 1 -- 172.21.15.98:0/2827395764 shutdown_connections 2024-05-09T08:51:19.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.113+0000 7f6a91f29640 1 --2- 172.21.15.98:0/2827395764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a8c1006f0 0x7f6a8c10e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.113+0000 7f6a91f29640 1 --2- 172.21.15.98:0/2827395764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a8c0ffd30 0x7f6a8c1001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.113+0000 7f6a91f29640 1 --2- 172.21.15.98:0/2827395764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c106810 0x7f6a8c106bf0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.113+0000 7f6a91f29640 1 -- 172.21.15.98:0/2827395764 >> 172.21.15.98:0/2827395764 conn(0x7f6a8c076cf0 msgr2=0x7f6a8c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:19.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.114+0000 7f6a91f29640 1 -- 172.21.15.98:0/2827395764 shutdown_connections 2024-05-09T08:51:19.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.114+0000 7f6a91f29640 1 -- 172.21.15.98:0/2827395764 wait complete. 2024-05-09T08:51:19.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.114+0000 7f6a91f29640 1 Processor -- start 2024-05-09T08:51:19.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.114+0000 7f6a91f29640 1 -- start start 2024-05-09T08:51:19.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a91f29640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a8c0ffd30 0x7f6a8c1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a91f29640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c1006f0 0x7f6a8c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a91f29640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a8c106810 0x7f6a8c19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a91f29640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c111d20 con 0x7f6a8c1006f0 2024-05-09T08:51:19.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a91f29640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c111ba0 con 0x7f6a8c106810 2024-05-09T08:51:19.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a91f29640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c111ea0 con 0x7f6a8c0ffd30 2024-05-09T08:51:19.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a8affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c1006f0 0x7f6a8c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a8affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c1006f0 0x7f6a8c198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57606/0 (socket says 172.21.15.98:57606) 2024-05-09T08:51:19.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a8b7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a8c0ffd30 0x7f6a8c1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a8affd640 1 -- 172.21.15.98:0/1924171754 learned_addr learned my addr 172.21.15.98:0/1924171754 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:19.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a8bfff640 1 --2- 172.21.15.98:0/1924171754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a8c106810 0x7f6a8c19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a8affd640 1 -- 172.21.15.98:0/1924171754 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a8c0ffd30 msgr2=0x7f6a8c1981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a8affd640 1 --2- 172.21.15.98:0/1924171754 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a8c0ffd30 0x7f6a8c1981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a8affd640 1 -- 172.21.15.98:0/1924171754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a8c106810 msgr2=0x7f6a8c19caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a8affd640 1 --2- 172.21.15.98:0/1924171754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a8c106810 0x7f6a8c19caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a8affd640 1 -- 172.21.15.98:0/1924171754 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a8c19d1d0 con 0x7f6a8c1006f0 2024-05-09T08:51:19.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a8bfff640 1 --2- 172.21.15.98:0/1924171754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a8c106810 0x7f6a8c19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:19.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a8affd640 1 --2- 172.21.15.98:0/1924171754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c1006f0 0x7f6a8c198720 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6a8000d770 tx=0x7f6a8000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:19.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a88ff9640 1 -- 172.21.15.98:0/1924171754 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a80017070 con 0x7f6a8c1006f0 2024-05-09T08:51:19.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a88ff9640 1 -- 172.21.15.98:0/1924171754 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a80005500 con 0x7f6a8c1006f0 2024-05-09T08:51:19.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a88ff9640 1 -- 172.21.15.98:0/1924171754 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a80004e00 con 0x7f6a8c1006f0 2024-05-09T08:51:19.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a91f29640 1 -- 172.21.15.98:0/1924171754 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a8c19d4c0 con 0x7f6a8c1006f0 2024-05-09T08:51:19.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.115+0000 7f6a91f29640 1 -- 172.21.15.98:0/1924171754 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a8c1a4d50 con 0x7f6a8c1006f0 2024-05-09T08:51:19.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a88ff9640 1 -- 172.21.15.98:0/1924171754 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f6a80004fa0 con 0x7f6a8c1006f0 2024-05-09T08:51:19.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a94c38740 1 Processor -- start 2024-05-09T08:51:19.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a94c38740 1 -- start start 2024-05-09T08:51:19.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a94c38740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577506120b0 0x5577506103f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a94c38740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 0x557750610e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a94c38740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55775061b740 0x55775061db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a94c38740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557750575990 con 0x55775061b740 2024-05-09T08:51:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a94c38740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55775051d5b0 con 0x5577506120b0 2024-05-09T08:51:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a94c38740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557750571950 con 0x5577506109c0 2024-05-09T08:51:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a8affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55775061b740 0x55775061db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a88ff9640 1 --2- 172.21.15.98:0/1924171754 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6a740a2f30 0x7f6a740a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a8affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55775061b740 0x55775061db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57612/0 (socket says 172.21.15.98:57612) 2024-05-09T08:51:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a8affd640 1 -- 172.21.15.98:0/476062979 learned_addr learned my addr 172.21.15.98:0/476062979 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a8bfff640 1 --2- 172.21.15.98:0/476062979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577506120b0 0x5577506103f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a8b7fe640 1 --2- 172.21.15.98:0/476062979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 0x557750610e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.118+0000 7f6a8bfff640 1 --2- 172.21.15.98:0/1924171754 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6a740a2f30 0x7f6a740a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.119+0000 7f6a88ff9640 1 -- 172.21.15.98:0/1924171754 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f6a800cf500 con 0x7f6a8c1006f0 2024-05-09T08:51:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.119+0000 7f6a8affd640 1 -- 172.21.15.98:0/476062979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 msgr2=0x557750610e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.119+0000 7f6a8affd640 1 --2- 172.21.15.98:0/476062979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 0x557750610e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.119+0000 7f6a8affd640 1 -- 172.21.15.98:0/476062979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577506120b0 msgr2=0x5577506103f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.119+0000 7f6a8affd640 1 --2- 172.21.15.98:0/476062979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577506120b0 0x5577506103f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.119+0000 7f6a8affd640 1 -- 172.21.15.98:0/476062979 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577503517e0 con 0x55775061b740 2024-05-09T08:51:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.119+0000 7f6a8bfff640 1 --2- 172.21.15.98:0/1924171754 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6a740a2f30 0x7f6a740a53f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6a84005fd0 tx=0x7f6a8400c7d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.119+0000 7f6a8affd640 1 --2- 172.21.15.98:0/476062979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55775061b740 0x55775061db30 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6a800a3b60 tx=0x7f6a80065b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=745f871b05c19e6b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.119+0000 7f6a71ffb640 1 -- 172.21.15.98:0/476062979 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a80020030 con 0x55775061b740 2024-05-09T08:51:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.119+0000 7f6a71ffb640 1 -- 172.21.15.98:0/476062979 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a800046c0 con 0x55775061b740 2024-05-09T08:51:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.119+0000 7f6a71ffb640 1 -- 172.21.15.98:0/476062979 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a80097a90 con 0x55775061b740 2024-05-09T08:51:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.120+0000 7f6a94c38740 1 -- 172.21.15.98:0/476062979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55775061b740 msgr2=0x55775061db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.120+0000 7f6a94c38740 1 --2- 172.21.15.98:0/476062979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55775061b740 0x55775061db30 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6a800a3b60 tx=0x7f6a80065b60 comp rx=0 tx=0).stop 2024-05-09T08:51:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.120+0000 7f6a94c38740 1 -- 172.21.15.98:0/476062979 shutdown_connections 2024-05-09T08:51:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.120+0000 7f6a94c38740 1 --2- 172.21.15.98:0/476062979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55775061b740 0x55775061db30 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.120+0000 7f6a94c38740 1 --2- 172.21.15.98:0/476062979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 0x557750610e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.120+0000 7f6a94c38740 1 --2- 172.21.15.98:0/476062979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577506120b0 0x5577506103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.120+0000 7f6a94c38740 1 -- 172.21.15.98:0/476062979 >> 172.21.15.98:0/476062979 conn(0x5577505d90a0 msgr2=0x55775060cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.120+0000 7f6a94c38740 1 -- 172.21.15.98:0/476062979 shutdown_connections 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.120+0000 7f6a94c38740 1 -- 172.21.15.98:0/476062979 wait complete. 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.120+0000 7f6a94c38740 1 -- start start 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.120+0000 7f6a94c38740 1 Processor -- start 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a94c38740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 0x557750618660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a94c38740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5577506120b0 0x557750618ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a94c38740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55775061b740 0x5577506190e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a94c38740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55775061e510 con 0x5577506120b0 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a94c38740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557750530b40 con 0x55775061b740 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a94c38740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55775061e690 con 0x5577506109c0 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a94c38740 10 client.0 _ll_register_callbacks cb 0x5577505250f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a8affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5577506120b0 0x557750618ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a8affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5577506120b0 0x557750618ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57618/0 (socket says 172.21.15.98:57618) 2024-05-09T08:51:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a8affd640 1 -- 172.21.15.98:0/115852084 learned_addr learned my addr 172.21.15.98:0/115852084 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a8b7fe640 1 --2- 172.21.15.98:0/115852084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 0x557750618660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a8bfff640 1 --2- 172.21.15.98:0/115852084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55775061b740 0x5577506190e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a8affd640 1 -- 172.21.15.98:0/115852084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 msgr2=0x557750618660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a8affd640 1 --2- 172.21.15.98:0/115852084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 0x557750618660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a8affd640 1 -- 172.21.15.98:0/115852084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55775061b740 msgr2=0x5577506190e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a8affd640 1 --2- 172.21.15.98:0/115852084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55775061b740 0x5577506190e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a8affd640 1 -- 172.21.15.98:0/115852084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577503b1f70 con 0x5577506120b0 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a8b7fe640 1 --2- 172.21.15.98:0/115852084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 0x557750618660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a8affd640 1 --2- 172.21.15.98:0/115852084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5577506120b0 0x557750618ba0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f6a8009b3b0 tx=0x7f6a800cc5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41dccefb07bf8fb6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a6affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a6affd640 1 -- 172.21.15.98:0/115852084 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a800b4060 con 0x5577506120b0 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a6affd640 1 -- 172.21.15.98:0/115852084 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f6a800cc210 con 0x5577506120b0 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a94c38740 1 -- 172.21.15.98:0/115852084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5577505830c0 con 0x5577506120b0 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a6affd640 1 -- 172.21.15.98:0/115852084 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6a80020040 con 0x5577506120b0 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a94c38740 10 client.15987 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.121+0000 7f6a94c38740 1 -- 172.21.15.98:0/115852084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557750249170 con 0x5577506120b0 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.122+0000 7f6a6affd640 1 -- 172.21.15.98:0/115852084 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5577505830c0 con 0x5577506120b0 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.122+0000 7f6a6affd640 1 -- 172.21.15.98:0/115852084 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6a800cf170 con 0x5577506120b0 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.122+0000 7f6a94c38740 10 client.15987 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.122+0000 7f6a94c38740 10 client.15987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.122+0000 7f6a94c38740 20 client.15987 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.122+0000 7f6a94c38740 1 --2- 172.21.15.98:0/115852084 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55775062b630 0x55775062da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.122+0000 7f6a94c38740 4 client.15987 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.122+0000 7f6a94c38740 1 -- 172.21.15.98:0/115852084 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55775057d410 con 0x55775062b630 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.122+0000 7f6a8b7fe640 1 --2- 172.21.15.98:0/115852084 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55775062b630 0x55775062da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.123+0000 7f6a6affd640 10 client.15987 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.123+0000 7f6a8b7fe640 1 --2- 172.21.15.98:0/115852084 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55775062b630 0x55775062da20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e91a4760542282fd server_cookie=73ec4c6ded6c20f3 in_seq=0 out_seq=0 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.123+0000 7f6a6affd640 1 -- 172.21.15.98:0/115852084 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55775057d410 con 0x55775062b630 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.123+0000 7f6a6affd640 10 client.15987 handle_command_reply: tid=0 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.123+0000 7f6a94c38740 1 client.15987 shutdown 2024-05-09T08:51:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.124+0000 7f6a94c38740 1 -- 172.21.15.98:0/115852084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5577506120b0 msgr2=0x557750618ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.124+0000 7f6a94c38740 1 --2- 172.21.15.98:0/115852084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5577506120b0 0x557750618ba0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f6a8009b3b0 tx=0x7f6a800cc5f0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.124+0000 7f6a94c38740 1 -- 172.21.15.98:0/115852084 shutdown_connections 2024-05-09T08:51:19.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.124+0000 7f6a94c38740 1 --2- 172.21.15.98:0/115852084 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55775062b630 0x55775062da20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.124+0000 7f6a94c38740 1 --2- 172.21.15.98:0/115852084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55775061b740 0x5577506190e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.124+0000 7f6a94c38740 1 --2- 172.21.15.98:0/115852084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5577506120b0 0x557750618ba0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.124+0000 7f6a6affd640 0 client.15987 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:19.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.124+0000 7f6a94c38740 1 --2- 172.21.15.98:0/115852084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 0x557750618660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.124+0000 7f6a94c38740 1 -- 172.21.15.98:0/115852084 >> 172.21.15.98:0/115852084 conn(0x5577505d90a0 msgr2=0x557750612d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:19.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.124+0000 7f6a94c38740 1 -- 172.21.15.98:0/115852084 shutdown_connections 2024-05-09T08:51:19.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.124+0000 7f6a94c38740 1 -- 172.21.15.98:0/115852084 wait complete. 2024-05-09T08:51:19.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.124+0000 7f6a94c38740 20 client.15987 trim_cache size 0 max 16384 2024-05-09T08:51:19.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.141+0000 7f6a94c38740 1 Processor -- start 2024-05-09T08:51:19.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.141+0000 7f6a94c38740 1 -- start start 2024-05-09T08:51:19.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a94c38740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577506109c0 0x55775060bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a94c38740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506120b0 0x55775061cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a94c38740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55775061b740 0x55775061d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a94c38740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c1aed20 con 0x55775061b740 2024-05-09T08:51:19.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a94c38740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c1b0860 con 0x5577506109c0 2024-05-09T08:51:19.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a94c38740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c1af3f0 con 0x5577506120b0 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a8bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55775061b740 0x55775061d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a8bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55775061b740 0x55775061d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57630/0 (socket says 172.21.15.98:57630) 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a8bfff640 1 -- 172.21.15.98:0/2031986473 learned_addr learned my addr 172.21.15.98:0/2031986473 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a8b7fe640 1 --2- 172.21.15.98:0/2031986473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577506109c0 0x55775060bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a8bfff640 1 -- 172.21.15.98:0/2031986473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506120b0 msgr2=0x55775061cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a8bfff640 1 --2- 172.21.15.98:0/2031986473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506120b0 0x55775061cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a8bfff640 1 -- 172.21.15.98:0/2031986473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577506109c0 msgr2=0x55775060bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a8bfff640 1 --2- 172.21.15.98:0/2031986473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577506109c0 0x55775060bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a8bfff640 1 -- 172.21.15.98:0/2031986473 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557750617610 con 0x55775061b740 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a8bfff640 1 --2- 172.21.15.98:0/2031986473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55775061b740 0x55775061d070 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6a8400cc30 tx=0x7f6a8401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c6d21e40fe5a25 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a717fa640 1 -- 172.21.15.98:0/2031986473 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8400fbd0 con 0x55775061b740 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a717fa640 1 -- 172.21.15.98:0/2031986473 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a8400fd70 con 0x55775061b740 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.142+0000 7f6a717fa640 1 -- 172.21.15.98:0/2031986473 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a84004de0 con 0x55775061b740 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.143+0000 7f6a94c38740 1 -- 172.21.15.98:0/2031986473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55775061b740 msgr2=0x55775061d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.143+0000 7f6a94c38740 1 --2- 172.21.15.98:0/2031986473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55775061b740 0x55775061d070 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6a8400cc30 tx=0x7f6a8401e000 comp rx=0 tx=0).stop 2024-05-09T08:51:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.143+0000 7f6a94c38740 1 -- 172.21.15.98:0/2031986473 shutdown_connections 2024-05-09T08:51:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.143+0000 7f6a94c38740 1 --2- 172.21.15.98:0/2031986473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55775061b740 0x55775061d070 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.143+0000 7f6a94c38740 1 --2- 172.21.15.98:0/2031986473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506120b0 0x55775061cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.143+0000 7f6a94c38740 1 --2- 172.21.15.98:0/2031986473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577506109c0 0x55775060bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.143+0000 7f6a94c38740 1 -- 172.21.15.98:0/2031986473 >> 172.21.15.98:0/2031986473 conn(0x5577505d90a0 msgr2=0x55775060b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.143+0000 7f6a94c38740 1 -- 172.21.15.98:0/2031986473 shutdown_connections 2024-05-09T08:51:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.143+0000 7f6a94c38740 1 -- 172.21.15.98:0/2031986473 wait complete. 2024-05-09T08:51:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.143+0000 7f6a94c38740 1 -- start start 2024-05-09T08:51:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.143+0000 7f6a94c38740 1 Processor -- start 2024-05-09T08:51:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a94c38740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 0x557750630df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a94c38740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5577506120b0 0x55775062da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a94c38740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55775061b740 0x55775062df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a94c38740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c1b0190 con 0x5577506120b0 2024-05-09T08:51:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a94c38740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c1afac0 con 0x55775061b740 2024-05-09T08:51:19.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a94c38740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c1acd00 con 0x5577506109c0 2024-05-09T08:51:19.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a94c38740 10 client.0 _ll_register_callbacks cb 0x7f6a8400cc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:19.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a8affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5577506120b0 0x55775062da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a8affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5577506120b0 0x55775062da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57646/0 (socket says 172.21.15.98:57646) 2024-05-09T08:51:19.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a8affd640 1 -- 172.21.15.98:0/200751610 learned_addr learned my addr 172.21.15.98:0/200751610 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:19.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a8bfff640 1 --2- 172.21.15.98:0/200751610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55775061b740 0x55775062df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a8affd640 1 -- 172.21.15.98:0/200751610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 msgr2=0x557750630df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a8affd640 1 --2- 172.21.15.98:0/200751610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 0x557750630df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a8affd640 1 -- 172.21.15.98:0/200751610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55775061b740 msgr2=0x55775062df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a8affd640 1 --2- 172.21.15.98:0/200751610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55775061b740 0x55775062df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a8affd640 1 -- 172.21.15.98:0/200751610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55775062e740 con 0x5577506120b0 2024-05-09T08:51:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a8bfff640 1 --2- 172.21.15.98:0/200751610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55775061b740 0x55775062df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a8affd640 1 --2- 172.21.15.98:0/200751610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5577506120b0 0x55775062da20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6a80097880 tx=0x7f6a800cf780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f63794599d0527f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a6affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a6affd640 1 -- 172.21.15.98:0/200751610 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a800b4040 con 0x5577506120b0 2024-05-09T08:51:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a6affd640 1 -- 172.21.15.98:0/200751610 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f6a800cc210 con 0x5577506120b0 2024-05-09T08:51:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a94c38740 1 -- 172.21.15.98:0/200751610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a8400fd70 con 0x5577506120b0 2024-05-09T08:51:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a6affd640 1 -- 172.21.15.98:0/200751610 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6a8009b1f0 con 0x5577506120b0 2024-05-09T08:51:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a94c38740 10 client.15993 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:19.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.144+0000 7f6a94c38740 1 -- 172.21.15.98:0/200751610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55775062e740 con 0x5577506120b0 2024-05-09T08:51:19.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.145+0000 7f6a6affd640 1 -- 172.21.15.98:0/200751610 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a8400fd70 con 0x5577506120b0 2024-05-09T08:51:19.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.145+0000 7f6a6affd640 1 -- 172.21.15.98:0/200751610 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6a800b42a0 con 0x5577506120b0 2024-05-09T08:51:19.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.145+0000 7f6a94c38740 10 client.15993 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:19.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.145+0000 7f6a94c38740 10 client.15993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:19.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.145+0000 7f6a94c38740 20 client.15993 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:19.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.145+0000 7f6a94c38740 1 --2- 172.21.15.98:0/200751610 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557750692a70 0x557750694e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.145+0000 7f6a94c38740 4 client.15993 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:51:19.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.145+0000 7f6a94c38740 1 -- 172.21.15.98:0/200751610 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5577504fada0 con 0x557750692a70 2024-05-09T08:51:19.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.145+0000 7f6a8b7fe640 1 --2- 172.21.15.98:0/200751610 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557750692a70 0x557750694e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.145+0000 7f6a6affd640 10 client.15993 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:19.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.145+0000 7f6a8b7fe640 1 --2- 172.21.15.98:0/200751610 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557750692a70 0x557750694e90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ef6aef18ed80e7 server_cookie=f2ab73096170e84d in_seq=0 out_seq=0 2024-05-09T08:51:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.146+0000 7f6a6affd640 1 -- 172.21.15.98:0/200751610 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5577504fada0 con 0x557750692a70 2024-05-09T08:51:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.146+0000 7f6a6affd640 10 client.15993 handle_command_reply: tid=0 2024-05-09T08:51:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.146+0000 7f6a94c38740 1 client.15993 shutdown 2024-05-09T08:51:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a94c38740 1 -- 172.21.15.98:0/200751610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5577506120b0 msgr2=0x55775062da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a94c38740 1 --2- 172.21.15.98:0/200751610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5577506120b0 0x55775062da20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6a80097880 tx=0x7f6a800cf780 comp rx=0 tx=0).stop 2024-05-09T08:51:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a94c38740 1 -- 172.21.15.98:0/200751610 shutdown_connections 2024-05-09T08:51:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a94c38740 1 --2- 172.21.15.98:0/200751610 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557750692a70 0x557750694e90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a94c38740 1 --2- 172.21.15.98:0/200751610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55775061b740 0x55775062df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a6affd640 0 client.15993 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a94c38740 1 --2- 172.21.15.98:0/200751610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5577506120b0 0x55775062da20 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a94c38740 1 --2- 172.21.15.98:0/200751610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5577506109c0 0x557750630df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a94c38740 1 -- 172.21.15.98:0/200751610 >> 172.21.15.98:0/200751610 conn(0x5577505d90a0 msgr2=0x55775060c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:19.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a94c38740 1 -- 172.21.15.98:0/200751610 shutdown_connections 2024-05-09T08:51:19.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a94c38740 1 -- 172.21.15.98:0/200751610 wait complete. 2024-05-09T08:51:19.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a94c38740 20 client.15993 trim_cache size 0 max 16384 2024-05-09T08:51:19.157 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:51:19.158 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:51:19.158 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "f9fcabfb-fb99-4bd1-96bf-8195a41ebecb", 2024-05-09T08:51:19.158 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:51:19.158 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:51:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a91f29640 1 -- 172.21.15.98:0/1924171754 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6a740a2f30 msgr2=0x7f6a740a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a91f29640 1 --2- 172.21.15.98:0/1924171754 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6a740a2f30 0x7f6a740a53f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6a84005fd0 tx=0x7f6a8400c7d0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a91f29640 1 -- 172.21.15.98:0/1924171754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c1006f0 msgr2=0x7f6a8c198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.147+0000 7f6a91f29640 1 --2- 172.21.15.98:0/1924171754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c1006f0 0x7f6a8c198720 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6a8000d770 tx=0x7f6a8000dc30 comp rx=0 tx=0).stop 2024-05-09T08:51:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.148+0000 7f6a91f29640 1 -- 172.21.15.98:0/1924171754 shutdown_connections 2024-05-09T08:51:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.148+0000 7f6a91f29640 1 --2- 172.21.15.98:0/1924171754 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6a740a2f30 0x7f6a740a53f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.148+0000 7f6a91f29640 1 --2- 172.21.15.98:0/1924171754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a8c106810 0x7f6a8c19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.148+0000 7f6a91f29640 1 --2- 172.21.15.98:0/1924171754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c1006f0 0x7f6a8c198720 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.148+0000 7f6a91f29640 1 --2- 172.21.15.98:0/1924171754 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a8c0ffd30 0x7f6a8c1981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.148+0000 7f6a91f29640 1 -- 172.21.15.98:0/1924171754 >> 172.21.15.98:0/1924171754 conn(0x7f6a8c076cf0 msgr2=0x7f6a8c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.148+0000 7f6a91f29640 1 -- 172.21.15.98:0/1924171754 shutdown_connections 2024-05-09T08:51:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.148+0000 7f6a91f29640 1 -- 172.21.15.98:0/1924171754 wait complete. 2024-05-09T08:51:19.163 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:51:19.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:51:19.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.267+0000 7fee0bd70640 1 -- 172.21.15.98:0/3664091803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee04101800 msgr2=0x7fee04101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.267+0000 7fee0bd70640 1 --2- 172.21.15.98:0/3664091803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee04101800 0x7fee04101c80 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fedf4009930 tx=0x7fedf402a420 comp rx=0 tx=0).stop 2024-05-09T08:51:19.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.267+0000 7fee0bd70640 1 -- 172.21.15.98:0/3664091803 shutdown_connections 2024-05-09T08:51:19.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.267+0000 7fee0bd70640 1 --2- 172.21.15.98:0/3664091803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee041021c0 0x7fee0410e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.267+0000 7fee0bd70640 1 --2- 172.21.15.98:0/3664091803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee04101800 0x7fee04101c80 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.267+0000 7fee0bd70640 1 --2- 172.21.15.98:0/3664091803 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee04107810 0x7fee04107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.267+0000 7fee0bd70640 1 -- 172.21.15.98:0/3664091803 >> 172.21.15.98:0/3664091803 conn(0x7fee04078c80 msgr2=0x7fee04079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:19.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.267+0000 7fee0bd70640 1 -- 172.21.15.98:0/3664091803 shutdown_connections 2024-05-09T08:51:19.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.267+0000 7fee0bd70640 1 -- 172.21.15.98:0/3664091803 wait complete. 2024-05-09T08:51:19.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.268+0000 7fee0bd70640 1 Processor -- start 2024-05-09T08:51:19.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.268+0000 7fee0bd70640 1 -- start start 2024-05-09T08:51:19.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.268+0000 7fee0bd70640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee04101800 0x7fee04198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.268+0000 7fee0bd70640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee041021c0 0x7fee041986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.268+0000 7fee0bd70640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee04107810 0x7fee0419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.268+0000 7fee0bd70640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fee04111c20 con 0x7fee04107810 2024-05-09T08:51:19.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.268+0000 7fee0bd70640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fee04111aa0 con 0x7fee041021c0 2024-05-09T08:51:19.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.268+0000 7fee0bd70640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fee04111da0 con 0x7fee04101800 2024-05-09T08:51:19.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.268+0000 7fee0a30d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee04107810 0x7fee0419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.268+0000 7fee0a30d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee04107810 0x7fee0419ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57668/0 (socket says 172.21.15.98:57668) 2024-05-09T08:51:19.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.268+0000 7fee0a30d640 1 -- 172.21.15.98:0/576191472 learned_addr learned my addr 172.21.15.98:0/576191472 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:19.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.268+0000 7fee09b0c640 1 --2- 172.21.15.98:0/576191472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee04101800 0x7fee04198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.269+0000 7fee0930b640 1 --2- 172.21.15.98:0/576191472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee041021c0 0x7fee041986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.269+0000 7fee0a30d640 1 -- 172.21.15.98:0/576191472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee04101800 msgr2=0x7fee04198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.269+0000 7fee0a30d640 1 --2- 172.21.15.98:0/576191472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee04101800 0x7fee04198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.269+0000 7fee0a30d640 1 -- 172.21.15.98:0/576191472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee041021c0 msgr2=0x7fee041986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.269+0000 7fee0a30d640 1 --2- 172.21.15.98:0/576191472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee041021c0 0x7fee041986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.269+0000 7fee0a30d640 1 -- 172.21.15.98:0/576191472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0419d170 con 0x7fee04107810 2024-05-09T08:51:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.269+0000 7fee0a30d640 1 --2- 172.21.15.98:0/576191472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee04107810 0x7fee0419ca40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00c970 tx=0x7fedfc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.269+0000 7fedfaffd640 1 -- 172.21.15.98:0/576191472 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc007bb0 con 0x7fee04107810 2024-05-09T08:51:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.269+0000 7fee0bd70640 1 -- 172.21.15.98:0/576191472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0419d460 con 0x7fee04107810 2024-05-09T08:51:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.269+0000 7fedfaffd640 1 -- 172.21.15.98:0/576191472 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedfc007d50 con 0x7fee04107810 2024-05-09T08:51:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.269+0000 7fedfaffd640 1 -- 172.21.15.98:0/576191472 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc00f450 con 0x7fee04107810 2024-05-09T08:51:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.269+0000 7fee0bd70640 1 -- 172.21.15.98:0/576191472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee041a4cf0 con 0x7fee04107810 2024-05-09T08:51:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.271+0000 7fedfaffd640 1 -- 172.21.15.98:0/576191472 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fedfc004f60 con 0x7fee04107810 2024-05-09T08:51:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fee0e722740 1 Processor -- start 2024-05-09T08:51:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fee0e722740 1 -- start start 2024-05-09T08:51:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fee0e722740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc5ffb280 0x558cc5ffb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fee0e722740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffbba0 0x558cc6005d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fee0e722740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc6006270 0x558cc6008660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fee0e722740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558cc5f60d00 con 0x558cc5ffbba0 2024-05-09T08:51:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fee0e722740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558cc5f084c0 con 0x558cc6006270 2024-05-09T08:51:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fee0e722740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558cc5f5ccc0 con 0x558cc5ffb280 2024-05-09T08:51:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fedfaffd640 1 --2- 172.21.15.98:0/576191472 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fedf00a2eb0 0x7fedf00a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fee09b0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffbba0 0x558cc6005d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fee09b0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffbba0 0x558cc6005d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57676/0 (socket says 172.21.15.98:57676) 2024-05-09T08:51:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fee09b0c640 1 -- 172.21.15.98:0/1912437137 learned_addr learned my addr 172.21.15.98:0/1912437137 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fee0930b640 1 --2- 172.21.15.98:0/1912437137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc6006270 0x558cc6008660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fee0a30d640 1 --2- 172.21.15.98:0/576191472 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fedf00a2eb0 0x7fedf00a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fee0930b640 1 --2- 172.21.15.98:0/1912437137 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc5ffb280 0x558cc5ffb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.272+0000 7fedfaffd640 1 -- 172.21.15.98:0/576191472 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fedfc0cf820 con 0x7fee04107810 2024-05-09T08:51:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.273+0000 7fee09b0c640 1 -- 172.21.15.98:0/1912437137 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc5ffb280 msgr2=0x558cc5ffb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.273+0000 7fee09b0c640 1 --2- 172.21.15.98:0/1912437137 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc5ffb280 0x558cc5ffb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.273+0000 7fee09b0c640 1 -- 172.21.15.98:0/1912437137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc6006270 msgr2=0x558cc6008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.273+0000 7fee09b0c640 1 --2- 172.21.15.98:0/1912437137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc6006270 0x558cc6008660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.273+0000 7fee09b0c640 1 -- 172.21.15.98:0/1912437137 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cc5d3e7d0 con 0x558cc5ffbba0 2024-05-09T08:51:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.273+0000 7fee0930b640 1 --2- 172.21.15.98:0/1912437137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc6006270 0x558cc6008660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.273+0000 7fee09b0c640 1 --2- 172.21.15.98:0/1912437137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffbba0 0x558cc6005d30 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fee00010370 tx=0x7fee00010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1918804f4abe20e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:19.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.273+0000 7fee0a30d640 1 --2- 172.21.15.98:0/576191472 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fedf00a2eb0 0x7fedf00a5370 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0cc9d0 tx=0x7fedfc086040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:19.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.273+0000 7fedeffff640 1 -- 172.21.15.98:0/1912437137 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00016070 con 0x558cc5ffbba0 2024-05-09T08:51:19.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.273+0000 7fedeffff640 1 -- 172.21.15.98:0/1912437137 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee00004d20 con 0x558cc5ffbba0 2024-05-09T08:51:19.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.273+0000 7fedeffff640 1 -- 172.21.15.98:0/1912437137 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00005480 con 0x558cc5ffbba0 2024-05-09T08:51:19.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.274+0000 7fee0e722740 1 -- 172.21.15.98:0/1912437137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffbba0 msgr2=0x558cc6005d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.274+0000 7fee0e722740 1 --2- 172.21.15.98:0/1912437137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffbba0 0x558cc6005d30 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fee00010370 tx=0x7fee00010830 comp rx=0 tx=0).stop 2024-05-09T08:51:19.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.274+0000 7fee0e722740 1 -- 172.21.15.98:0/1912437137 shutdown_connections 2024-05-09T08:51:19.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.274+0000 7fee0e722740 1 --2- 172.21.15.98:0/1912437137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc6006270 0x558cc6008660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.274+0000 7fee0e722740 1 --2- 172.21.15.98:0/1912437137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffbba0 0x558cc6005d30 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.274+0000 7fee0e722740 1 --2- 172.21.15.98:0/1912437137 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc5ffb280 0x558cc5ffb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.274+0000 7fee0e722740 1 -- 172.21.15.98:0/1912437137 >> 172.21.15.98:0/1912437137 conn(0x558cc5ff75a0 msgr2=0x558cc5ff7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.274+0000 7fee0e722740 1 -- 172.21.15.98:0/1912437137 shutdown_connections 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.274+0000 7fee0e722740 1 -- 172.21.15.98:0/1912437137 wait complete. 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.274+0000 7fee0e722740 1 -- start start 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.274+0000 7fee0e722740 1 Processor -- start 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee0e722740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc5ffb280 0x558cc60031f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee0e722740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffbba0 0x558cc6003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee0e722740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc6006270 0x558cc6003c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee0e722740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558cc6009060 con 0x558cc5ffbba0 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee0e722740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558cc5f1bb90 con 0x558cc5ffb280 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee0e722740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558cc60091e0 con 0x558cc6006270 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee0e722740 10 client.0 _ll_register_callbacks cb 0x558cc5f57da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee09b0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffbba0 0x558cc6003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee09b0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffbba0 0x558cc6003730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57682/0 (socket says 172.21.15.98:57682) 2024-05-09T08:51:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee09b0c640 1 -- 172.21.15.98:0/649839916 learned_addr learned my addr 172.21.15.98:0/649839916 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee0930b640 1 --2- 172.21.15.98:0/649839916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc5ffb280 0x558cc60031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee09b0c640 1 -- 172.21.15.98:0/649839916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc6006270 msgr2=0x558cc6003c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee09b0c640 1 --2- 172.21.15.98:0/649839916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc6006270 0x558cc6003c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee09b0c640 1 -- 172.21.15.98:0/649839916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc5ffb280 msgr2=0x558cc60031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee09b0c640 1 --2- 172.21.15.98:0/649839916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc5ffb280 0x558cc60031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee09b0c640 1 -- 172.21.15.98:0/649839916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cc5c743f0 con 0x558cc5ffbba0 2024-05-09T08:51:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee09b0c640 1 --2- 172.21.15.98:0/649839916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffbba0 0x558cc6003730 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fee00000f80 tx=0x7fee00004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6b7d592e3adc67 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7feded7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7feded7fa640 1 -- 172.21.15.98:0/649839916 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00029060 con 0x558cc5ffbba0 2024-05-09T08:51:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7feded7fa640 1 -- 172.21.15.98:0/649839916 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fee00003cc0 con 0x558cc5ffbba0 2024-05-09T08:51:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee0e722740 1 -- 172.21.15.98:0/649839916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee00004d20 con 0x558cc5ffbba0 2024-05-09T08:51:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7feded7fa640 1 -- 172.21.15.98:0/649839916 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fee00030050 con 0x558cc5ffbba0 2024-05-09T08:51:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee0e722740 10 client.16005 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.275+0000 7fee0e722740 1 -- 172.21.15.98:0/649839916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558cc5c340f0 con 0x558cc5ffbba0 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.276+0000 7feded7fa640 1 -- 172.21.15.98:0/649839916 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee00004d20 con 0x558cc5ffbba0 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.276+0000 7feded7fa640 1 -- 172.21.15.98:0/649839916 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fee0000f040 con 0x558cc5ffbba0 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.276+0000 7fee0e722740 10 client.16005 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.276+0000 7fee0e722740 10 client.16005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.276+0000 7fee0e722740 20 client.16005 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.276+0000 7fee0e722740 1 --2- 172.21.15.98:0/649839916 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558cc6016570 0x558cc6018960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.276+0000 7fee0e722740 4 client.16005 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.276+0000 7fee0e722740 1 -- 172.21.15.98:0/649839916 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cc5f6e430 con 0x558cc6016570 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.276+0000 7fee0930b640 1 --2- 172.21.15.98:0/649839916 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558cc6016570 0x558cc6018960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.277+0000 7feded7fa640 10 client.16005 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.277+0000 7fee0930b640 1 --2- 172.21.15.98:0/649839916 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558cc6016570 0x558cc6018960 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2af0112d422fad59 server_cookie=3bb86ad221f958e8 in_seq=0 out_seq=0 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.277+0000 7feded7fa640 1 -- 172.21.15.98:0/649839916 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558cc5f6e430 con 0x558cc6016570 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.277+0000 7feded7fa640 10 client.16005 handle_command_reply: tid=0 2024-05-09T08:51:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.277+0000 7fee0e722740 1 client.16005 shutdown 2024-05-09T08:51:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.277+0000 7fee0e722740 1 -- 172.21.15.98:0/649839916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffbba0 msgr2=0x558cc6003730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.277+0000 7fee0e722740 1 --2- 172.21.15.98:0/649839916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffbba0 0x558cc6003730 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fee00000f80 tx=0x7fee00004c70 comp rx=0 tx=0).stop 2024-05-09T08:51:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.278+0000 7fee0e722740 1 -- 172.21.15.98:0/649839916 shutdown_connections 2024-05-09T08:51:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.278+0000 7fee0e722740 1 --2- 172.21.15.98:0/649839916 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558cc6016570 0x558cc6018960 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.278+0000 7fee0e722740 1 --2- 172.21.15.98:0/649839916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc6006270 0x558cc6003c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.278+0000 7fee0e722740 1 --2- 172.21.15.98:0/649839916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffbba0 0x558cc6003730 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.278+0000 7feded7fa640 0 client.16005 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.278+0000 7fee0e722740 1 --2- 172.21.15.98:0/649839916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc5ffb280 0x558cc60031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.278+0000 7fee0e722740 1 -- 172.21.15.98:0/649839916 >> 172.21.15.98:0/649839916 conn(0x558cc5ff75a0 msgr2=0x558cc5ffd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.278+0000 7fee0e722740 1 -- 172.21.15.98:0/649839916 shutdown_connections 2024-05-09T08:51:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.278+0000 7fee0e722740 1 -- 172.21.15.98:0/649839916 wait complete. 2024-05-09T08:51:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.278+0000 7fee0e722740 20 client.16005 trim_cache size 0 max 16384 2024-05-09T08:51:19.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0e722740 1 Processor -- start 2024-05-09T08:51:19.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0e722740 1 -- start start 2024-05-09T08:51:19.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0e722740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffb280 0x558cc6011f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0e722740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc5ffbba0 0x558cc5ffda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0e722740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc6006270 0x558cc601df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0e722740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fee041aeec0 con 0x558cc5ffb280 2024-05-09T08:51:19.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0e722740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fee041b0a00 con 0x558cc6006270 2024-05-09T08:51:19.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0e722740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fee041af590 con 0x558cc5ffbba0 2024-05-09T08:51:19.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0930b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffb280 0x558cc6011f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0930b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffb280 0x558cc6011f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57694/0 (socket says 172.21.15.98:57694) 2024-05-09T08:51:19.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0930b640 1 -- 172.21.15.98:0/700824251 learned_addr learned my addr 172.21.15.98:0/700824251 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:19.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0930b640 1 -- 172.21.15.98:0/700824251 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc5ffbba0 msgr2=0x558cc5ffda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0930b640 1 --2- 172.21.15.98:0/700824251 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc5ffbba0 0x558cc5ffda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0930b640 1 -- 172.21.15.98:0/700824251 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc6006270 msgr2=0x558cc601df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0930b640 1 --2- 172.21.15.98:0/700824251 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc6006270 0x558cc601df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.295+0000 7fee0930b640 1 -- 172.21.15.98:0/700824251 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cc6008660 con 0x558cc5ffb280 2024-05-09T08:51:19.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.296+0000 7fee0930b640 1 --2- 172.21.15.98:0/700824251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffb280 0x558cc6011f20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fedf4000c00 tx=0x7fedf40058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a96855555ca07b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:19.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.296+0000 7fedef7fe640 1 -- 172.21.15.98:0/700824251 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4035040 con 0x558cc5ffb280 2024-05-09T08:51:19.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.296+0000 7fedef7fe640 1 -- 172.21.15.98:0/700824251 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedf404b070 con 0x558cc5ffb280 2024-05-09T08:51:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.296+0000 7fedef7fe640 1 -- 172.21.15.98:0/700824251 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4046e60 con 0x558cc5ffb280 2024-05-09T08:51:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.296+0000 7fee0e722740 1 -- 172.21.15.98:0/700824251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffb280 msgr2=0x558cc6011f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.296+0000 7fee0e722740 1 --2- 172.21.15.98:0/700824251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffb280 0x558cc6011f20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fedf4000c00 tx=0x7fedf40058b0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.296+0000 7fee0e722740 1 -- 172.21.15.98:0/700824251 shutdown_connections 2024-05-09T08:51:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.296+0000 7fee0e722740 1 --2- 172.21.15.98:0/700824251 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc6006270 0x558cc601df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.296+0000 7fee0e722740 1 --2- 172.21.15.98:0/700824251 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc5ffbba0 0x558cc5ffda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.296+0000 7fee0e722740 1 --2- 172.21.15.98:0/700824251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc5ffb280 0x558cc6011f20 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.296+0000 7fee0e722740 1 -- 172.21.15.98:0/700824251 >> 172.21.15.98:0/700824251 conn(0x558cc5ff75a0 msgr2=0x558cc6003a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.296+0000 7fee0e722740 1 -- 172.21.15.98:0/700824251 shutdown_connections 2024-05-09T08:51:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.296+0000 7fee0e722740 1 -- 172.21.15.98:0/700824251 wait complete. 2024-05-09T08:51:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0e722740 1 -- start start 2024-05-09T08:51:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0e722740 1 Processor -- start 2024-05-09T08:51:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0e722740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc5ffb280 0x558cc600e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0e722740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc5ffbba0 0x558cc5fffb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0e722740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc6006270 0x558cc6000060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0e722740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fee041b0330 con 0x558cc6006270 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0e722740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fee041afc60 con 0x558cc5ffbba0 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0e722740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fee041acea0 con 0x558cc5ffb280 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0e722740 10 client.0 _ll_register_callbacks cb 0x7fee00009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0930b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc6006270 0x558cc6000060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0930b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc6006270 0x558cc6000060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57700/0 (socket says 172.21.15.98:57700) 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0930b640 1 -- 172.21.15.98:0/335176359 learned_addr learned my addr 172.21.15.98:0/335176359 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee09b0c640 1 --2- 172.21.15.98:0/335176359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc5ffbba0 0x558cc5fffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0930b640 1 -- 172.21.15.98:0/335176359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc5ffb280 msgr2=0x558cc600e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0930b640 1 --2- 172.21.15.98:0/335176359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc5ffb280 0x558cc600e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0930b640 1 -- 172.21.15.98:0/335176359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc5ffbba0 msgr2=0x558cc5fffb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0930b640 1 --2- 172.21.15.98:0/335176359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc5ffbba0 0x558cc5fffb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee0930b640 1 -- 172.21.15.98:0/335176359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cc60007a0 con 0x558cc6006270 2024-05-09T08:51:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.297+0000 7fee09b0c640 1 --2- 172.21.15.98:0/335176359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc5ffbba0 0x558cc5fffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7fee0930b640 1 --2- 172.21.15.98:0/335176359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc6006270 0x558cc6000060 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fedf4009690 tx=0x7fedf4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eda25c38c639e8d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7feded7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7feded7fa640 1 -- 172.21.15.98:0/335176359 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4035040 con 0x558cc6006270 2024-05-09T08:51:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7feded7fa640 1 -- 172.21.15.98:0/335176359 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fedf4005050 con 0x558cc6006270 2024-05-09T08:51:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7fee0e722740 1 -- 172.21.15.98:0/335176359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedf404b070 con 0x558cc6006270 2024-05-09T08:51:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7feded7fa640 1 -- 172.21.15.98:0/335176359 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fedf4005250 con 0x558cc6006270 2024-05-09T08:51:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7fee0e722740 10 client.16011 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7fee0e722740 1 -- 172.21.15.98:0/335176359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558cc607c6b0 con 0x558cc6006270 2024-05-09T08:51:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7feded7fa640 1 -- 172.21.15.98:0/335176359 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedf404b070 con 0x558cc6006270 2024-05-09T08:51:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7feded7fa640 1 -- 172.21.15.98:0/335176359 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fedf403a040 con 0x558cc6006270 2024-05-09T08:51:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7fee0e722740 10 client.16011 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7fee0e722740 10 client.16011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7fee0e722740 20 client.16011 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7fee0e722740 1 --2- 172.21.15.98:0/335176359 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558cc607ccd0 0x558cc607f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7fee0e722740 4 client.16011 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7fee0e722740 1 -- 172.21.15.98:0/335176359 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558cc5f68780 con 0x558cc607ccd0 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.298+0000 7fee09b0c640 1 --2- 172.21.15.98:0/335176359 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558cc607ccd0 0x558cc607f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7fee09b0c640 1 --2- 172.21.15.98:0/335176359 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558cc607ccd0 0x558cc607f0c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d33b7eb9010eb5a3 server_cookie=616929b178bd4c53 in_seq=0 out_seq=0 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7feded7fa640 10 client.16011 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7feded7fa640 1 -- 172.21.15.98:0/335176359 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558cc5f68780 con 0x558cc607ccd0 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7feded7fa640 10 client.16011 handle_command_reply: tid=0 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7fee0e722740 1 client.16011 shutdown 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7fee0e722740 1 -- 172.21.15.98:0/335176359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc6006270 msgr2=0x558cc6000060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7fee0e722740 1 --2- 172.21.15.98:0/335176359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc6006270 0x558cc6000060 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fedf4009690 tx=0x7fedf4004040 comp rx=0 tx=0).stop 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7fee0e722740 1 -- 172.21.15.98:0/335176359 shutdown_connections 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7fee0e722740 1 --2- 172.21.15.98:0/335176359 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558cc607ccd0 0x558cc607f0c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7fee0e722740 1 --2- 172.21.15.98:0/335176359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cc6006270 0x558cc6000060 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7fee0e722740 1 --2- 172.21.15.98:0/335176359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558cc5ffbba0 0x558cc5fffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7fee0e722740 1 --2- 172.21.15.98:0/335176359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558cc5ffb280 0x558cc600e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7feded7fa640 0 client.16011 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:19.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7fee0e722740 1 -- 172.21.15.98:0/335176359 >> 172.21.15.98:0/335176359 conn(0x558cc5ff75a0 msgr2=0x558cc6003120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:19.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7fee0e722740 1 -- 172.21.15.98:0/335176359 shutdown_connections 2024-05-09T08:51:19.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7fee0e722740 1 -- 172.21.15.98:0/335176359 wait complete. 2024-05-09T08:51:19.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.299+0000 7fee0e722740 20 client.16011 trim_cache size 0 max 16384 2024-05-09T08:51:19.306 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:51:19.306 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:51:19.306 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:51:19.306 INFO:teuthology.orchestra.run.smithi098.stdout: "f9fcabfb-fb99-4bd1-96bf-8195a41ebecb": { 2024-05-09T08:51:19.307 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:51:19.307 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "f9fcabfb-fb99-4bd1-96bf-8195a41ebecb", 2024-05-09T08:51:19.307 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:51:19.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:51:19.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:51:19.307 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:51:19.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.300+0000 7fee0bd70640 1 -- 172.21.15.98:0/576191472 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fedf00a2eb0 msgr2=0x7fedf00a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.300+0000 7fee0bd70640 1 --2- 172.21.15.98:0/576191472 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fedf00a2eb0 0x7fedf00a5370 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0cc9d0 tx=0x7fedfc086040 comp rx=0 tx=0).stop 2024-05-09T08:51:19.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.300+0000 7fee0bd70640 1 -- 172.21.15.98:0/576191472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee04107810 msgr2=0x7fee0419ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:19.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.300+0000 7fee0bd70640 1 --2- 172.21.15.98:0/576191472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee04107810 0x7fee0419ca40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00c970 tx=0x7fedfc00ce30 comp rx=0 tx=0).stop 2024-05-09T08:51:19.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.300+0000 7fee0bd70640 1 -- 172.21.15.98:0/576191472 shutdown_connections 2024-05-09T08:51:19.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.300+0000 7fee0bd70640 1 --2- 172.21.15.98:0/576191472 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fedf00a2eb0 0x7fedf00a5370 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.300+0000 7fee0bd70640 1 --2- 172.21.15.98:0/576191472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee04107810 0x7fee0419ca40 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.300+0000 7fee0bd70640 1 --2- 172.21.15.98:0/576191472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee041021c0 0x7fee041986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.300+0000 7fee0bd70640 1 --2- 172.21.15.98:0/576191472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee04101800 0x7fee04198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:19.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.300+0000 7fee0bd70640 1 -- 172.21.15.98:0/576191472 >> 172.21.15.98:0/576191472 conn(0x7fee04078c80 msgr2=0x7fee040ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:19.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.300+0000 7fee0bd70640 1 -- 172.21.15.98:0/576191472 shutdown_connections 2024-05-09T08:51:19.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:19.300+0000 7fee0bd70640 1 -- 172.21.15.98:0/576191472 wait complete. 2024-05-09T08:51:19.310 INFO:tasks.cephfs.filesystem:scrub status for tag:f9fcabfb-fb99-4bd1-96bf-8195a41ebecb - {'path': '/', 'tag': 'f9fcabfb-fb99-4bd1-96bf-8195a41ebecb', 'options': 'recursive,force'} 2024-05-09T08:51:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:20 smithi142 ceph-mon[34625]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:51:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:20 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:51:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:20 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:51:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:20 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:51:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:20 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:51:20.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:20 smithi105 ceph-mon[34498]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:51:20.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:20 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:51:20.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:20 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:51:20.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:20 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:51:20.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:20 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:51:20.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:20 smithi098 ceph-mon[30092]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:51:20.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:20 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:51:20.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:20 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:51:20.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:20 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:51:20.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:20 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:51:21.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:21 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:51:21.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:21 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:51:21.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:21 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:51:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:22 smithi142 ceph-mon[34625]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:51:22.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:22 smithi105 ceph-mon[34498]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:51:22.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:22 smithi098 ceph-mon[30092]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:51:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:24 smithi142 ceph-mon[34625]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:51:24.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:24 smithi105 ceph-mon[34498]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:51:24.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:24 smithi098 ceph-mon[30092]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:51:26.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:26 smithi142 ceph-mon[34625]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-05-09T08:51:26.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:26 smithi105 ceph-mon[34498]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-05-09T08:51:26.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:26 smithi098 ceph-mon[30092]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-05-09T08:51:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.433+0000 7f0c0eb61640 1 -- 172.21.15.98:0/3074603630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c08107830 msgr2=0x7f0c08107c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.433+0000 7f0c0eb61640 1 --2- 172.21.15.98:0/3074603630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c08107830 0x7f0c08107c10 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009930 tx=0x7f0bfc02a420 comp rx=0 tx=0).stop 2024-05-09T08:51:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.433+0000 7f0c0eb61640 1 -- 172.21.15.98:0/3074603630 shutdown_connections 2024-05-09T08:51:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.433+0000 7f0c0eb61640 1 --2- 172.21.15.98:0/3074603630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c081021e0 0x7f0c0810e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.433+0000 7f0c0eb61640 1 --2- 172.21.15.98:0/3074603630 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c08101820 0x7f0c08101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.433+0000 7f0c0eb61640 1 --2- 172.21.15.98:0/3074603630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c08107830 0x7f0c08107c10 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.433+0000 7f0c0eb61640 1 -- 172.21.15.98:0/3074603630 >> 172.21.15.98:0/3074603630 conn(0x7f0c080fd2f0 msgr2=0x7f0c080ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.433+0000 7f0c0eb61640 1 -- 172.21.15.98:0/3074603630 shutdown_connections 2024-05-09T08:51:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.433+0000 7f0c0eb61640 1 -- 172.21.15.98:0/3074603630 wait complete. 2024-05-09T08:51:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0eb61640 1 Processor -- start 2024-05-09T08:51:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0eb61640 1 -- start start 2024-05-09T08:51:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0eb61640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c08101820 0x7f0c08073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0eb61640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c081021e0 0x7f0c08074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0eb61640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c08107830 0x7f0c080745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0eb61640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c08111c60 con 0x7f0c08107830 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0eb61640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0c08111ae0 con 0x7f0c081021e0 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0eb61640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0c08111de0 con 0x7f0c08101820 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0d0fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c08107830 0x7f0c080745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0d0fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c08107830 0x7f0c080745b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52430/0 (socket says 172.21.15.98:52430) 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0d0fe640 1 -- 172.21.15.98:0/2579885416 learned_addr learned my addr 172.21.15.98:0/2579885416 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0c8fd640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c08101820 0x7f0c08073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c07fff640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c081021e0 0x7f0c08074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0d0fe640 1 -- 172.21.15.98:0/2579885416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c08101820 msgr2=0x7f0c08073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0d0fe640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c08101820 0x7f0c08073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0d0fe640 1 -- 172.21.15.98:0/2579885416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c081021e0 msgr2=0x7f0c08074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0d0fe640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c081021e0 0x7f0c08074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0d0fe640 1 -- 172.21.15.98:0/2579885416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c081a6660 con 0x7f0c08107830 2024-05-09T08:51:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c07fff640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c081021e0 0x7f0c08074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.434+0000 7f0c0d0fe640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c08107830 0x7f0c080745b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0c0000eb10 tx=0x7f0c0000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.435+0000 7f0c05ffb640 1 -- 172.21.15.98:0/2579885416 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c00004b20 con 0x7f0c08107830 2024-05-09T08:51:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.435+0000 7f0c05ffb640 1 -- 172.21.15.98:0/2579885416 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c00004cc0 con 0x7f0c08107830 2024-05-09T08:51:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.435+0000 7f0c0eb61640 1 -- 172.21.15.98:0/2579885416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c081a6950 con 0x7f0c08107830 2024-05-09T08:51:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.435+0000 7f0c05ffb640 1 -- 172.21.15.98:0/2579885416 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c00010760 con 0x7f0c08107830 2024-05-09T08:51:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.435+0000 7f0c0eb61640 1 -- 172.21.15.98:0/2579885416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c081a6ea0 con 0x7f0c08107830 2024-05-09T08:51:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.437+0000 7f0c05ffb640 1 -- 172.21.15.98:0/2579885416 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f0c00005020 con 0x7f0c08107830 2024-05-09T08:51:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.438+0000 7f0c05ffb640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0bf80a2f50 0x7f0bf80a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.438+0000 7f0c0c8fd640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0bf80a2f50 0x7f0bf80a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.438+0000 7f0c05ffb640 1 -- 172.21.15.98:0/2579885416 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f0c000d0240 con 0x7f0c08107830 2024-05-09T08:51:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.438+0000 7f0c0c8fd640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0bf80a2f50 0x7f0bf80a5410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009900 tx=0x7f0bfc0058d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.438+0000 7f0c1186e740 1 Processor -- start 2024-05-09T08:51:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.438+0000 7f0c1186e740 1 -- start start 2024-05-09T08:51:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c1186e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 0x558212d17830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c1186e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d220f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c1186e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d24a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c1186e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558212c7c340 con 0x558212d17d70 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c1186e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558212c24290 con 0x558212d22630 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c1186e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558212c78300 con 0x558212d17450 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c07fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d220f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c07fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d220f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52440/0 (socket says 172.21.15.98:52440) 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c07fff640 1 -- 172.21.15.98:0/3403029866 learned_addr learned my addr 172.21.15.98:0/3403029866 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c0c8fd640 1 --2- 172.21.15.98:0/3403029866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 0x558212d17830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c0d0fe640 1 --2- 172.21.15.98:0/3403029866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d24a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c07fff640 1 -- 172.21.15.98:0/3403029866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 msgr2=0x558212d17830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c07fff640 1 --2- 172.21.15.98:0/3403029866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 0x558212d17830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c07fff640 1 -- 172.21.15.98:0/3403029866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 msgr2=0x558212d24a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c07fff640 1 --2- 172.21.15.98:0/3403029866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d24a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c07fff640 1 -- 172.21.15.98:0/3403029866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558212c488e0 con 0x558212d17d70 2024-05-09T08:51:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0c07fff640 1 --2- 172.21.15.98:0/3403029866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d220f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400a9d0 tx=0x7f0bf400ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a7f538ad0971a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.439+0000 7f0bf2ffd640 1 -- 172.21.15.98:0/3403029866 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4015070 con 0x558212d17d70 2024-05-09T08:51:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.440+0000 7f0bf2ffd640 1 -- 172.21.15.98:0/3403029866 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bf400cc60 con 0x558212d17d70 2024-05-09T08:51:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.440+0000 7f0bf2ffd640 1 -- 172.21.15.98:0/3403029866 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf40105b0 con 0x558212d17d70 2024-05-09T08:51:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.440+0000 7f0c1186e740 1 -- 172.21.15.98:0/3403029866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 msgr2=0x558212d220f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.440+0000 7f0c1186e740 1 --2- 172.21.15.98:0/3403029866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d220f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400a9d0 tx=0x7f0bf400ae90 comp rx=0 tx=0).stop 2024-05-09T08:51:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.440+0000 7f0c1186e740 1 -- 172.21.15.98:0/3403029866 shutdown_connections 2024-05-09T08:51:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.440+0000 7f0c1186e740 1 --2- 172.21.15.98:0/3403029866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d24a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.440+0000 7f0c1186e740 1 --2- 172.21.15.98:0/3403029866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d220f0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.440+0000 7f0c1186e740 1 --2- 172.21.15.98:0/3403029866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 0x558212d17830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.440+0000 7f0c1186e740 1 -- 172.21.15.98:0/3403029866 >> 172.21.15.98:0/3403029866 conn(0x558212d13960 msgr2=0x558212d13d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.440+0000 7f0c1186e740 1 -- 172.21.15.98:0/3403029866 shutdown_connections 2024-05-09T08:51:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.440+0000 7f0c1186e740 1 -- 172.21.15.98:0/3403029866 wait complete. 2024-05-09T08:51:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.441+0000 7f0c1186e740 1 -- start start 2024-05-09T08:51:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.441+0000 7f0c1186e740 1 Processor -- start 2024-05-09T08:51:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.441+0000 7f0c1186e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 0x558212d1f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.441+0000 7f0c1186e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d1fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.441+0000 7f0c1186e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d1fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.441+0000 7f0c1186e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558212d25260 con 0x558212d17d70 2024-05-09T08:51:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.441+0000 7f0c1186e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558212c37790 con 0x558212d22630 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.441+0000 7f0c1186e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558212d253e0 con 0x558212d17450 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.441+0000 7f0c1186e740 10 client.0 _ll_register_callbacks cb 0x558212c72a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.441+0000 7f0c07fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d1fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.441+0000 7f0c07fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d1fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52452/0 (socket says 172.21.15.98:52452) 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.441+0000 7f0c07fff640 1 -- 172.21.15.98:0/3311329571 learned_addr learned my addr 172.21.15.98:0/3311329571 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.441+0000 7f0c0c8fd640 1 --2- 172.21.15.98:0/3311329571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 0x558212d1f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0c0d0fe640 1 --2- 172.21.15.98:0/3311329571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d1fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0c07fff640 1 -- 172.21.15.98:0/3311329571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 msgr2=0x558212d1f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0c07fff640 1 --2- 172.21.15.98:0/3311329571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 0x558212d1f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0c07fff640 1 -- 172.21.15.98:0/3311329571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 msgr2=0x558212d1fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0c07fff640 1 --2- 172.21.15.98:0/3311329571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d1fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0c07fff640 1 -- 172.21.15.98:0/3311329571 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558212990210 con 0x558212d17d70 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0c07fff640 1 --2- 172.21.15.98:0/3311329571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d1fab0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4008010 tx=0x7f0bf4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c216efdddaea76 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0be7fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0be7fff640 1 -- 172.21.15.98:0/3311329571 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4020070 con 0x558212d17d70 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0be7fff640 1 -- 172.21.15.98:0/3311329571 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f0bf401a450 con 0x558212d17d70 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0c1186e740 1 -- 172.21.15.98:0/3311329571 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0bf400cc60 con 0x558212d17d70 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0be7fff640 1 -- 172.21.15.98:0/3311329571 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0bf4015040 con 0x558212d17d70 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0c1186e740 10 client.16023 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0c1186e740 1 -- 172.21.15.98:0/3311329571 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558212a5c620 con 0x558212d17d70 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.442+0000 7f0be7fff640 1 -- 172.21.15.98:0/3311329571 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bf400cc60 con 0x558212d17d70 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.443+0000 7f0be7fff640 1 -- 172.21.15.98:0/3311329571 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0bf4024700 con 0x558212d17d70 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.443+0000 7f0c1186e740 10 client.16023 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.443+0000 7f0c1186e740 10 client.16023 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.443+0000 7f0c1186e740 20 client.16023 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.443+0000 7f0c1186e740 1 --2- 172.21.15.98:0/3311329571 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558212d321d0 0x558212d345c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.443+0000 7f0c1186e740 4 client.16023 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.443+0000 7f0c1186e740 1 -- 172.21.15.98:0/3311329571 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558212c89a70 con 0x558212d321d0 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.443+0000 7f0c0c8fd640 1 --2- 172.21.15.98:0/3311329571 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558212d321d0 0x558212d345c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.443+0000 7f0c0c8fd640 1 --2- 172.21.15.98:0/3311329571 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558212d321d0 0x558212d345c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26dd05f455f03fb1 server_cookie=e3141a4bed47f762 in_seq=0 out_seq=0 2024-05-09T08:51:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.443+0000 7f0be7fff640 10 client.16023 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0be7fff640 1 -- 172.21.15.98:0/3311329571 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558212c89a70 con 0x558212d321d0 2024-05-09T08:51:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0be7fff640 10 client.16023 handle_command_reply: tid=0 2024-05-09T08:51:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0c1186e740 1 client.16023 shutdown 2024-05-09T08:51:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0c1186e740 1 -- 172.21.15.98:0/3311329571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 msgr2=0x558212d1fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0c1186e740 1 --2- 172.21.15.98:0/3311329571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d1fab0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4008010 tx=0x7f0bf4011ab0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0c1186e740 1 -- 172.21.15.98:0/3311329571 shutdown_connections 2024-05-09T08:51:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0c1186e740 1 --2- 172.21.15.98:0/3311329571 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558212d321d0 0x558212d345c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0c1186e740 1 --2- 172.21.15.98:0/3311329571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d1fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0c1186e740 1 --2- 172.21.15.98:0/3311329571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d1fab0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0c1186e740 1 --2- 172.21.15.98:0/3311329571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 0x558212d1f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0be7fff640 0 client.16023 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0c1186e740 1 -- 172.21.15.98:0/3311329571 >> 172.21.15.98:0/3311329571 conn(0x558212d13960 msgr2=0x558212d19690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0c1186e740 1 -- 172.21.15.98:0/3311329571 shutdown_connections 2024-05-09T08:51:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0c1186e740 1 -- 172.21.15.98:0/3311329571 wait complete. 2024-05-09T08:51:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.444+0000 7f0c1186e740 20 client.16023 trim_cache size 0 max 16384 2024-05-09T08:51:27.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.459+0000 7f0c1186e740 1 Processor -- start 2024-05-09T08:51:27.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.459+0000 7f0c1186e740 1 -- start start 2024-05-09T08:51:27.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.459+0000 7f0c1186e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 0x558212d2e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.459+0000 7f0c1186e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d19e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.459+0000 7f0c1186e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d39bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.459+0000 7f0c1186e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c081b1000 con 0x558212d17d70 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.459+0000 7f0c1186e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0c081b2b40 con 0x558212d22630 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.459+0000 7f0c1186e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0c081b16d0 con 0x558212d17450 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.459+0000 7f0c07fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d19e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.459+0000 7f0c07fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d19e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52464/0 (socket says 172.21.15.98:52464) 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.459+0000 7f0c07fff640 1 -- 172.21.15.98:0/166940249 learned_addr learned my addr 172.21.15.98:0/166940249 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.459+0000 7f0c0c8fd640 1 --2- 172.21.15.98:0/166940249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 0x558212d2e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.460+0000 7f0c0d0fe640 1 --2- 172.21.15.98:0/166940249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d39bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.460+0000 7f0c07fff640 1 -- 172.21.15.98:0/166940249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 msgr2=0x558212d2e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.460+0000 7f0c07fff640 1 --2- 172.21.15.98:0/166940249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 0x558212d2e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.460+0000 7f0c07fff640 1 -- 172.21.15.98:0/166940249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 msgr2=0x558212d39bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.460+0000 7f0c07fff640 1 --2- 172.21.15.98:0/166940249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d39bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.460+0000 7f0c07fff640 1 -- 172.21.15.98:0/166940249 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558212d24d90 con 0x558212d17d70 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.460+0000 7f0c0d0fe640 1 --2- 172.21.15.98:0/166940249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d39bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.460+0000 7f0c07fff640 1 --2- 172.21.15.98:0/166940249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d19e10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4023440 tx=0x7f0bf400ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76932b189599c074 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.460+0000 7f0bf27fc640 1 -- 172.21.15.98:0/166940249 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4020040 con 0x558212d17d70 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.460+0000 7f0bf27fc640 1 -- 172.21.15.98:0/166940249 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bf40248e0 con 0x558212d17d70 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.460+0000 7f0bf27fc640 1 -- 172.21.15.98:0/166940249 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf40239c0 con 0x558212d17d70 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.461+0000 7f0c1186e740 1 -- 172.21.15.98:0/166940249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 msgr2=0x558212d19e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.461+0000 7f0c1186e740 1 --2- 172.21.15.98:0/166940249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d19e10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4023440 tx=0x7f0bf400ae90 comp rx=0 tx=0).stop 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.461+0000 7f0c1186e740 1 -- 172.21.15.98:0/166940249 shutdown_connections 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.461+0000 7f0c1186e740 1 --2- 172.21.15.98:0/166940249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d39bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.461+0000 7f0c1186e740 1 --2- 172.21.15.98:0/166940249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17d70 0x558212d19e10 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.461+0000 7f0c1186e740 1 --2- 172.21.15.98:0/166940249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17450 0x558212d2e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.461+0000 7f0c1186e740 1 -- 172.21.15.98:0/166940249 >> 172.21.15.98:0/166940249 conn(0x558212d13960 msgr2=0x558212d12a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.461+0000 7f0c1186e740 1 -- 172.21.15.98:0/166940249 shutdown_connections 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.461+0000 7f0c1186e740 1 -- 172.21.15.98:0/166940249 wait complete. 2024-05-09T08:51:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.461+0000 7f0c1186e740 1 -- start start 2024-05-09T08:51:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.461+0000 7f0c1186e740 1 Processor -- start 2024-05-09T08:51:27.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c1186e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17450 0x558212d1c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c1186e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17d70 0x558212d1c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c1186e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d1cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c1186e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c081b2470 con 0x558212d17450 2024-05-09T08:51:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c1186e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0c081b1da0 con 0x558212d22630 2024-05-09T08:51:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c1186e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0c081aefe0 con 0x558212d17d70 2024-05-09T08:51:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c1186e740 10 client.0 _ll_register_callbacks cb 0x7f0bfc040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c0c8fd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17450 0x558212d1c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c0c8fd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17450 0x558212d1c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52466/0 (socket says 172.21.15.98:52466) 2024-05-09T08:51:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c0c8fd640 1 -- 172.21.15.98:0/261463511 learned_addr learned my addr 172.21.15.98:0/261463511 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c07fff640 1 --2- 172.21.15.98:0/261463511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17d70 0x558212d1c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c0c8fd640 1 -- 172.21.15.98:0/261463511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17d70 msgr2=0x558212d1c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c0c8fd640 1 --2- 172.21.15.98:0/261463511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17d70 0x558212d1c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c0c8fd640 1 -- 172.21.15.98:0/261463511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 msgr2=0x558212d1cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c0c8fd640 1 --2- 172.21.15.98:0/261463511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d1cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c0c8fd640 1 -- 172.21.15.98:0/261463511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558212d97460 con 0x558212d17450 2024-05-09T08:51:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0c0c8fd640 1 --2- 172.21.15.98:0/261463511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17450 0x558212d1c280 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc03f040 tx=0x7f0bfc02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4796c720d97c80ec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.462+0000 7f0be7fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.463+0000 7f0be7fff640 1 -- 172.21.15.98:0/261463511 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc035040 con 0x558212d17450 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.463+0000 7f0be7fff640 1 -- 172.21.15.98:0/261463511 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f0bfc053320 con 0x558212d17450 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.463+0000 7f0c1186e740 1 -- 172.21.15.98:0/261463511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0bf40248e0 con 0x558212d17450 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.463+0000 7f0be7fff640 1 -- 172.21.15.98:0/261463511 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0bfc065050 con 0x558212d17450 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.463+0000 7f0c1186e740 10 client.16029 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.463+0000 7f0c1186e740 1 -- 172.21.15.98:0/261463511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558212d987b0 con 0x558212d17450 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.463+0000 7f0be7fff640 1 -- 172.21.15.98:0/261463511 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bf40248e0 con 0x558212d17450 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.463+0000 7f0be7fff640 1 -- 172.21.15.98:0/261463511 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0bfc003bf0 con 0x558212d17450 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.463+0000 7f0c1186e740 10 client.16029 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.463+0000 7f0c1186e740 10 client.16029 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.463+0000 7f0c1186e740 20 client.16029 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.463+0000 7f0c1186e740 1 --2- 172.21.15.98:0/261463511 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558212d98ee0 0x558212d9b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.463+0000 7f0c1186e740 4 client.16029 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.463+0000 7f0c1186e740 1 -- 172.21.15.98:0/261463511 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558212c83dc0 con 0x558212d98ee0 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.464+0000 7f0c07fff640 1 --2- 172.21.15.98:0/261463511 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558212d98ee0 0x558212d9b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.464+0000 7f0c07fff640 1 --2- 172.21.15.98:0/261463511 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558212d98ee0 0x558212d9b320 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46473615c143f166 server_cookie=1934a7c7bb143352 in_seq=0 out_seq=0 2024-05-09T08:51:27.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.464+0000 7f0be7fff640 10 client.16029 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0be7fff640 1 -- 172.21.15.98:0/261463511 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558212c83dc0 con 0x558212d98ee0 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0be7fff640 10 client.16029 handle_command_reply: tid=0 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0c1186e740 1 client.16029 shutdown 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0c1186e740 1 -- 172.21.15.98:0/261463511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17450 msgr2=0x558212d1c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0c1186e740 1 --2- 172.21.15.98:0/261463511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17450 0x558212d1c280 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc03f040 tx=0x7f0bfc02a420 comp rx=0 tx=0).stop 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0c1186e740 1 -- 172.21.15.98:0/261463511 shutdown_connections 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0c1186e740 1 --2- 172.21.15.98:0/261463511 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558212d98ee0 0x558212d9b320 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0c1186e740 1 --2- 172.21.15.98:0/261463511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558212d22630 0x558212d1cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0c1186e740 1 --2- 172.21.15.98:0/261463511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558212d17d70 0x558212d1c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0c1186e740 1 --2- 172.21.15.98:0/261463511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558212d17450 0x558212d1c280 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0be7fff640 0 client.16029 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0c1186e740 1 -- 172.21.15.98:0/261463511 >> 172.21.15.98:0/261463511 conn(0x558212d13960 msgr2=0x558212d1dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0c1186e740 1 -- 172.21.15.98:0/261463511 shutdown_connections 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0c1186e740 1 -- 172.21.15.98:0/261463511 wait complete. 2024-05-09T08:51:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.471+0000 7f0c1186e740 20 client.16029 trim_cache size 0 max 16384 2024-05-09T08:51:27.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:51:27.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:51:27.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:51:27.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:51:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.472+0000 7f0c0eb61640 1 -- 172.21.15.98:0/2579885416 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0bf80a2f50 msgr2=0x7f0bf80a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.472+0000 7f0c0eb61640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0bf80a2f50 0x7f0bf80a5410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009900 tx=0x7f0bfc0058d0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.472+0000 7f0c0eb61640 1 -- 172.21.15.98:0/2579885416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c08107830 msgr2=0x7f0c080745b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.472+0000 7f0c0eb61640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c08107830 0x7f0c080745b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0c0000eb10 tx=0x7f0c0000c2d0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.472+0000 7f0c0eb61640 1 -- 172.21.15.98:0/2579885416 shutdown_connections 2024-05-09T08:51:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.472+0000 7f0c0eb61640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0bf80a2f50 0x7f0bf80a5410 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.472+0000 7f0c0eb61640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c08107830 0x7f0c080745b0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.472+0000 7f0c0eb61640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c081021e0 0x7f0c08074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.472+0000 7f0c0eb61640 1 --2- 172.21.15.98:0/2579885416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c08101820 0x7f0c08073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.472+0000 7f0c0eb61640 1 -- 172.21.15.98:0/2579885416 >> 172.21.15.98:0/2579885416 conn(0x7f0c080fd2f0 msgr2=0x7f0c080fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.472+0000 7f0c0eb61640 1 -- 172.21.15.98:0/2579885416 shutdown_connections 2024-05-09T08:51:27.474 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:27.472+0000 7f0c0eb61640 1 -- 172.21.15.98:0/2579885416 wait complete. 2024-05-09T08:51:28.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:28 smithi142 ceph-mon[34625]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-05-09T08:51:28.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:28 smithi105 ceph-mon[34498]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-05-09T08:51:28.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:28 smithi098 ceph-mon[30092]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-05-09T08:51:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:30 smithi142 ceph-mon[34625]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-05-09T08:51:30.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:30 smithi105 ceph-mon[34498]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-05-09T08:51:30.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:30 smithi098 ceph-mon[30092]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-05-09T08:51:32.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:32 smithi142 ceph-mon[34625]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.5 KiB/s wr, 15 op/s 2024-05-09T08:51:32.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:32 smithi105 ceph-mon[34498]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.5 KiB/s wr, 15 op/s 2024-05-09T08:51:32.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:32 smithi098 ceph-mon[30092]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.5 KiB/s wr, 15 op/s 2024-05-09T08:51:34.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:34 smithi142 ceph-mon[34625]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-05-09T08:51:34.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:34 smithi105 ceph-mon[34498]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-05-09T08:51:34.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:34 smithi098 ceph-mon[30092]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-05-09T08:51:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:36 smithi142 ceph-mon[34625]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.5 KiB/s wr, 14 op/s 2024-05-09T08:51:36.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:36 smithi105 ceph-mon[34498]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.5 KiB/s wr, 14 op/s 2024-05-09T08:51:36.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:36 smithi098 ceph-mon[30092]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.5 KiB/s wr, 14 op/s 2024-05-09T08:51:38.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:38 smithi142 ceph-mon[34625]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T08:51:38.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:38 smithi105 ceph-mon[34498]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T08:51:38.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:38 smithi098 ceph-mon[30092]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T08:51:40.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.609+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1110785232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a18100760 msgr2=0x7f6a18100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:40.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.609+0000 7f6a1d1ce640 1 --2- 172.21.15.98:0/1110785232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a18100760 0x7f6a18100be0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009930 tx=0x7f6a0c02a420 comp rx=0 tx=0).stop 2024-05-09T08:51:40.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.610+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1110785232 shutdown_connections 2024-05-09T08:51:40.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.610+0000 7f6a1d1ce640 1 --2- 172.21.15.98:0/1110785232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a18101120 0x7f6a1810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.610+0000 7f6a1d1ce640 1 --2- 172.21.15.98:0/1110785232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a18100760 0x7f6a18100be0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.610+0000 7f6a1d1ce640 1 --2- 172.21.15.98:0/1110785232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a18106720 0x7f6a18106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.610+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1110785232 >> 172.21.15.98:0/1110785232 conn(0x7f6a180778b0 msgr2=0x7f6a18077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:40.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.610+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1110785232 shutdown_connections 2024-05-09T08:51:40.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.610+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1110785232 wait complete. 2024-05-09T08:51:40.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.611+0000 7f6a1d1ce640 1 Processor -- start 2024-05-09T08:51:40.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.611+0000 7f6a1d1ce640 1 -- start start 2024-05-09T08:51:40.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.612+0000 7f6a1d1ce640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a18100760 0x7f6a18196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.612+0000 7f6a1d1ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a18101120 0x7f6a18197470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.612+0000 7f6a1d1ce640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a18106720 0x7f6a1819b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.612+0000 7f6a1d1ce640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a18110b30 con 0x7f6a18101120 2024-05-09T08:51:40.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.612+0000 7f6a16d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a18100760 0x7f6a18196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.612+0000 7f6a1659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a18101120 0x7f6a18197470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.612+0000 7f6a16d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a18100760 0x7f6a18196f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:51208/0 (socket says 172.21.15.98:51208) 2024-05-09T08:51:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.612+0000 7f6a16d9d640 1 -- 172.21.15.98:0/1707303374 learned_addr learned my addr 172.21.15.98:0/1707303374 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.612+0000 7f6a1759e640 1 --2- 172.21.15.98:0/1707303374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a18106720 0x7f6a1819b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.612+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1707303374 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181109b0 con 0x7f6a18106720 2024-05-09T08:51:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.612+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1707303374 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6a18110cb0 con 0x7f6a18100760 2024-05-09T08:51:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.613+0000 7f6a1659c640 1 -- 172.21.15.98:0/1707303374 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a18100760 msgr2=0x7f6a18196f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.613+0000 7f6a1659c640 1 --2- 172.21.15.98:0/1707303374 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a18100760 0x7f6a18196f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.613+0000 7f6a1659c640 1 -- 172.21.15.98:0/1707303374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a18106720 msgr2=0x7f6a1819b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.613+0000 7f6a1659c640 1 --2- 172.21.15.98:0/1707303374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a18106720 0x7f6a1819b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.613+0000 7f6a1659c640 1 -- 172.21.15.98:0/1707303374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a1819bf20 con 0x7f6a18101120 2024-05-09T08:51:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.613+0000 7f6a1659c640 1 --2- 172.21.15.98:0/1707303374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a18101120 0x7f6a18197470 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c000c00 tx=0x7f6a0c0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:40.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.613+0000 7f69fffff640 1 -- 172.21.15.98:0/1707303374 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c035020 con 0x7f6a18101120 2024-05-09T08:51:40.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.613+0000 7f69fffff640 1 -- 172.21.15.98:0/1707303374 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a0c040070 con 0x7f6a18101120 2024-05-09T08:51:40.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.613+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1707303374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a1819c1b0 con 0x7f6a18101120 2024-05-09T08:51:40.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.614+0000 7f69fffff640 1 -- 172.21.15.98:0/1707303374 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c0039c0 con 0x7f6a18101120 2024-05-09T08:51:40.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.614+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1707303374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a18068ee0 con 0x7f6a18101120 2024-05-09T08:51:40.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.617+0000 7f69fffff640 1 -- 172.21.15.98:0/1707303374 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f6a0c003b60 con 0x7f6a18101120 2024-05-09T08:51:40.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.617+0000 7f6a1ffb8740 1 Processor -- start 2024-05-09T08:51:40.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.617+0000 7f6a1ffb8740 1 -- start start 2024-05-09T08:51:40.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a1ffb8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 0x557be772d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a1ffb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7728f60 0x557be77293e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a1ffb8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be7735440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a1ffb8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557be7684340 con 0x557be7728f60 2024-05-09T08:51:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a1ffb8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557be762c290 con 0x557be772d4f0 2024-05-09T08:51:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a1ffb8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557be7680300 con 0x557be7729920 2024-05-09T08:51:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a16d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7728f60 0x557be77293e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a16d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7728f60 0x557be77293e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53598/0 (socket says 172.21.15.98:53598) 2024-05-09T08:51:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a16d9d640 1 -- 172.21.15.98:0/2227602471 learned_addr learned my addr 172.21.15.98:0/2227602471 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a1659c640 1 --2- 172.21.15.98:0/2227602471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be7735440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a1759e640 1 --2- 172.21.15.98:0/2227602471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 0x557be772d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a16d9d640 1 -- 172.21.15.98:0/2227602471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 msgr2=0x557be7735440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a16d9d640 1 --2- 172.21.15.98:0/2227602471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be7735440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a16d9d640 1 -- 172.21.15.98:0/2227602471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 msgr2=0x557be772d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a16d9d640 1 --2- 172.21.15.98:0/2227602471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 0x557be772d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a16d9d640 1 -- 172.21.15.98:0/2227602471 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557be76508e0 con 0x557be7728f60 2024-05-09T08:51:40.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.618+0000 7f6a1659c640 1 --2- 172.21.15.98:0/2227602471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be7735440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:40.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.619+0000 7f6a16d9d640 1 --2- 172.21.15.98:0/2227602471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7728f60 0x557be77293e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800bd40 tx=0x7f6a080107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7fb8d99482ef2b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:40.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.619+0000 7f69fffff640 1 --2- 172.21.15.98:0/1707303374 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6a000a2fd0 0x7f6a000a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.619+0000 7f6a1759e640 1 --2- 172.21.15.98:0/1707303374 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6a000a2fd0 0x7f6a000a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.619+0000 7f69fffff640 1 -- 172.21.15.98:0/1707303374 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f6a0c03abb0 con 0x7f6a18101120 2024-05-09T08:51:40.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.619+0000 7f69fd7fa640 1 -- 172.21.15.98:0/2227602471 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0801c480 con 0x557be7728f60 2024-05-09T08:51:40.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.619+0000 7f69fd7fa640 1 -- 172.21.15.98:0/2227602471 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a08011920 con 0x557be7728f60 2024-05-09T08:51:40.623 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.620+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/2227602471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7728f60 msgr2=0x557be77293e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.620+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/2227602471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7728f60 0x557be77293e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800bd40 tx=0x7f6a080107a0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.620+0000 7f6a1759e640 1 --2- 172.21.15.98:0/1707303374 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6a000a2fd0 0x7f6a000a5490 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6a10005fd0 tx=0x7f6a10009600 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.620+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/2227602471 shutdown_connections 2024-05-09T08:51:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.620+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/2227602471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be7735440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.620+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/2227602471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7728f60 0x557be77293e0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.620+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/2227602471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 0x557be772d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.620+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/2227602471 >> 172.21.15.98:0/2227602471 conn(0x557be771b410 msgr2=0x557be771b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.620+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/2227602471 shutdown_connections 2024-05-09T08:51:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.620+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/2227602471 wait complete. 2024-05-09T08:51:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.621+0000 7f6a1ffb8740 1 -- start start 2024-05-09T08:51:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.621+0000 7f6a1ffb8740 1 Processor -- start 2024-05-09T08:51:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1ffb8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be7728f60 0x557be7735410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1ffb8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be77327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1ffb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be772d4f0 0x557be7732e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be772d4f0 0x557be7732e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be772d4f0 0x557be7732e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53600/0 (socket says 172.21.15.98:53600) 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1759e640 1 -- 172.21.15.98:0/3860791505 learned_addr learned my addr 172.21.15.98:0/3860791505 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a16d9d640 1 --2- 172.21.15.98:0/3860791505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be7728f60 0x557be7735410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3860791505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557be7735aa0 con 0x557be772d4f0 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1659c640 1 --2- 172.21.15.98:0/3860791505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be77327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3860791505 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557be763f790 con 0x557be7728f60 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3860791505 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557be7735c20 con 0x557be7729920 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1ffb8740 10 client.0 _ll_register_callbacks cb 0x557be767aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1659c640 1 -- 172.21.15.98:0/3860791505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be7728f60 msgr2=0x557be7735410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1659c640 1 --2- 172.21.15.98:0/3860791505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be7728f60 0x557be7735410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1659c640 1 -- 172.21.15.98:0/3860791505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be772d4f0 msgr2=0x557be7732e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1659c640 1 --2- 172.21.15.98:0/3860791505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be772d4f0 0x557be7732e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a1659c640 1 -- 172.21.15.98:0/3860791505 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557be7398210 con 0x557be7729920 2024-05-09T08:51:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.622+0000 7f6a16d9d640 1 --2- 172.21.15.98:0/3860791505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be7728f60 0x557be7735410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.623+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3860791505 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a08011920 con 0x557be7729920 2024-05-09T08:51:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.623+0000 7f6a1659c640 1 --2- 172.21.15.98:0/3860791505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be77327a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0ef710 tx=0x7f6a0c031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2449ad21b0e4d8b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.623+0000 7f69faff5640 10 client.25366 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:51:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.623+0000 7f69faff5640 1 -- 172.21.15.98:0/3860791505 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c0bde40 con 0x557be7729920 2024-05-09T08:51:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.623+0000 7f69faff5640 1 -- 172.21.15.98:0/3860791505 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f6a0c0c6cc0 con 0x557be7729920 2024-05-09T08:51:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.623+0000 7f69faff5640 1 -- 172.21.15.98:0/3860791505 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6a0c0c6ec0 con 0x557be7729920 2024-05-09T08:51:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.623+0000 7f6a1ffb8740 10 client.25366 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.623+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3860791505 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({fsmap=29}) -- 0x557be7464620 con 0x557be7729920 2024-05-09T08:51:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.624+0000 7f69faff5640 1 -- 172.21.15.98:0/3860791505 <== mon.1 v2:172.21.15.142:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6a0c04d030 con 0x557be7729920 2024-05-09T08:51:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.624+0000 7f6a1ffb8740 10 client.25366 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.624+0000 7f6a1ffb8740 10 client.25366 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.624+0000 7f6a1ffb8740 20 client.25366 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.624+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/3860791505 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557be77429f0 0x557be7744e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.624+0000 7f6a1ffb8740 4 client.25366 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.624+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3860791505 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6a08011920 con 0x557be77429f0 2024-05-09T08:51:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.624+0000 7f6a16d9d640 1 --2- 172.21.15.98:0/3860791505 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557be77429f0 0x557be7744e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.625+0000 7f69faff5640 10 client.25366 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.625+0000 7f6a16d9d640 1 --2- 172.21.15.98:0/3860791505 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557be77429f0 0x557be7744e00 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1893cc0abc56827 server_cookie=f498d8316ad39fd7 in_seq=0 out_seq=0 2024-05-09T08:51:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.625+0000 7f69faff5640 1 -- 172.21.15.98:0/3860791505 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f6a08011920 con 0x557be77429f0 2024-05-09T08:51:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.625+0000 7f69faff5640 10 client.25366 handle_command_reply: tid=0 2024-05-09T08:51:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.625+0000 7f6a1ffb8740 1 client.25366 shutdown 2024-05-09T08:51:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.625+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3860791505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 msgr2=0x557be77327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.626+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/3860791505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be77327a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0ef710 tx=0x7f6a0c031400 comp rx=0 tx=0).stop 2024-05-09T08:51:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.626+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3860791505 shutdown_connections 2024-05-09T08:51:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.626+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/3860791505 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557be77429f0 0x557be7744e00 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.626+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/3860791505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be772d4f0 0x557be7732e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.626+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/3860791505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be77327a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.626+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/3860791505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be7728f60 0x557be7735410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.626+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3860791505 >> 172.21.15.98:0/3860791505 conn(0x557be771b410 msgr2=0x557be77287e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.626+0000 7f69faff5640 0 client.25366 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.626+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3860791505 shutdown_connections 2024-05-09T08:51:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.626+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3860791505 wait complete. 2024-05-09T08:51:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.626+0000 7f6a1ffb8740 20 client.25366 trim_cache size 0 max 16384 2024-05-09T08:51:40.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.655+0000 7f6a1ffb8740 1 Processor -- start 2024-05-09T08:51:40.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.655+0000 7f6a1ffb8740 1 -- start start 2024-05-09T08:51:40.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.655+0000 7f6a1ffb8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7728f60 0x557be7744a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.655+0000 7f6a1ffb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7729920 0x557be774a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7729920 0x557be774a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7729920 0x557be774a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53612/0 (socket says 172.21.15.98:53612) 2024-05-09T08:51:40.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1659c640 1 -- 172.21.15.98:0/2187275798 learned_addr learned my addr 172.21.15.98:0/2187275798 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:40.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/2187275798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 0x557be774ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/2187275798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181ac9b0 con 0x557be7729920 2024-05-09T08:51:40.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/2187275798 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181ae4f0 con 0x557be772d4f0 2024-05-09T08:51:40.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/2187275798 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181ad080 con 0x557be7728f60 2024-05-09T08:51:40.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a16d9d640 1 --2- 172.21.15.98:0/2187275798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7728f60 0x557be7744a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1759e640 1 --2- 172.21.15.98:0/2187275798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 0x557be774ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1659c640 1 -- 172.21.15.98:0/2187275798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7728f60 msgr2=0x557be7744a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1659c640 1 --2- 172.21.15.98:0/2187275798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7728f60 0x557be7744a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1659c640 1 -- 172.21.15.98:0/2187275798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 msgr2=0x557be774ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1659c640 1 --2- 172.21.15.98:0/2187275798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 0x557be774ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1659c640 1 -- 172.21.15.98:0/2187275798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557be7733200 con 0x557be7729920 2024-05-09T08:51:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1759e640 1 --2- 172.21.15.98:0/2187275798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 0x557be774ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f6a1659c640 1 --2- 172.21.15.98:0/2187275798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7729920 0x557be774a710 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c03a470 tx=0x7f6a0c0c1000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7aeedabfc337781 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.656+0000 7f69fcff9640 1 -- 172.21.15.98:0/2187275798 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c0dc060 con 0x557be7729920 2024-05-09T08:51:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.657+0000 7f69fcff9640 1 -- 172.21.15.98:0/2187275798 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a0c004780 con 0x557be7729920 2024-05-09T08:51:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.657+0000 7f69fcff9640 1 -- 172.21.15.98:0/2187275798 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c0d7230 con 0x557be7729920 2024-05-09T08:51:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.657+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/2187275798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7729920 msgr2=0x557be774a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.657+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/2187275798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7729920 0x557be774a710 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c03a470 tx=0x7f6a0c0c1000 comp rx=0 tx=0).stop 2024-05-09T08:51:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.658+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/2187275798 shutdown_connections 2024-05-09T08:51:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.658+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/2187275798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 0x557be774ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.658+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/2187275798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7729920 0x557be774a710 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.658+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/2187275798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7728f60 0x557be7744a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.658+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/2187275798 >> 172.21.15.98:0/2187275798 conn(0x557be771b410 msgr2=0x557be7731070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.658+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/2187275798 shutdown_connections 2024-05-09T08:51:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.658+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/2187275798 wait complete. 2024-05-09T08:51:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.658+0000 7f6a1ffb8740 1 -- start start 2024-05-09T08:51:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.658+0000 7f6a1ffb8740 1 Processor -- start 2024-05-09T08:51:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.658+0000 7f6a1ffb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7728f60 0x557be772af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.658+0000 7f6a1ffb8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be772b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.658+0000 7f6a1ffb8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 0x557be77395d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a1ffb8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181ade20 con 0x557be7728f60 2024-05-09T08:51:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a1ffb8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181ad750 con 0x557be772d4f0 2024-05-09T08:51:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a1ffb8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181aa990 con 0x557be7729920 2024-05-09T08:51:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a16d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7728f60 0x557be772af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a1ffb8740 10 client.0 _ll_register_callbacks cb 0x7f6a0c005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a16d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7728f60 0x557be772af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53618/0 (socket says 172.21.15.98:53618) 2024-05-09T08:51:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a16d9d640 1 -- 172.21.15.98:0/3791446531 learned_addr learned my addr 172.21.15.98:0/3791446531 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a1759e640 1 --2- 172.21.15.98:0/3791446531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 0x557be77395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a1659c640 1 --2- 172.21.15.98:0/3791446531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be772b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a16d9d640 1 -- 172.21.15.98:0/3791446531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 msgr2=0x557be772b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a16d9d640 1 --2- 172.21.15.98:0/3791446531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be772b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a16d9d640 1 -- 172.21.15.98:0/3791446531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 msgr2=0x557be77395d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a16d9d640 1 --2- 172.21.15.98:0/3791446531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 0x557be77395d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a16d9d640 1 -- 172.21.15.98:0/3791446531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557be7739b40 con 0x557be7728f60 2024-05-09T08:51:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a1659c640 1 --2- 172.21.15.98:0/3791446531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be772b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f6a16d9d640 1 --2- 172.21.15.98:0/3791446531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7728f60 0x557be772af70 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x557be771ef50 tx=0x7f6a0801a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65039102063da070 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f69faff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f69faff5640 1 -- 172.21.15.98:0/3791446531 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0801adc0 con 0x557be7728f60 2024-05-09T08:51:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.659+0000 7f69faff5640 1 -- 172.21.15.98:0/3791446531 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f6a080050b0 con 0x557be7728f60 2024-05-09T08:51:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.660+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3791446531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a0c004780 con 0x557be7728f60 2024-05-09T08:51:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.660+0000 7f69faff5640 1 -- 172.21.15.98:0/3791446531 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6a080052b0 con 0x557be7728f60 2024-05-09T08:51:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.660+0000 7f6a1ffb8740 10 client.16047 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.660+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3791446531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557be77a9760 con 0x557be7728f60 2024-05-09T08:51:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.660+0000 7f69faff5640 1 -- 172.21.15.98:0/3791446531 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a0c004780 con 0x557be7728f60 2024-05-09T08:51:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.661+0000 7f69faff5640 1 -- 172.21.15.98:0/3791446531 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6a08029090 con 0x557be7728f60 2024-05-09T08:51:40.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.661+0000 7f6a1ffb8740 10 client.16047 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:40.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.661+0000 7f6a1ffb8740 10 client.16047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:40.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.661+0000 7f6a1ffb8740 20 client.16047 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:40.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.661+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/3791446531 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557be77a9ba0 0x557be77abf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:40.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.661+0000 7f6a1ffb8740 4 client.16047 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:51:40.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.661+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3791446531 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557be7691a70 con 0x557be77a9ba0 2024-05-09T08:51:40.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.661+0000 7f6a1659c640 1 --2- 172.21.15.98:0/3791446531 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557be77a9ba0 0x557be77abf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:40.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.662+0000 7f69faff5640 10 client.16047 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:40.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.662+0000 7f6a1659c640 1 --2- 172.21.15.98:0/3791446531 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557be77a9ba0 0x557be77abf90 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93ece4aafb7ffdb5 server_cookie=eef6d99364b52664 in_seq=0 out_seq=0 2024-05-09T08:51:40.666 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:40 smithi142 ceph-mon[34625]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T08:51:40.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.665+0000 7f69faff5640 1 -- 172.21.15.98:0/3791446531 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557be7691a70 con 0x557be77a9ba0 2024-05-09T08:51:40.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.665+0000 7f69faff5640 10 client.16047 handle_command_reply: tid=0 2024-05-09T08:51:40.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.665+0000 7f6a1ffb8740 1 client.16047 shutdown 2024-05-09T08:51:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.666+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3791446531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7728f60 msgr2=0x557be772af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.666+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/3791446531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7728f60 0x557be772af70 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x557be771ef50 tx=0x7f6a0801a620 comp rx=0 tx=0).stop 2024-05-09T08:51:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.666+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3791446531 shutdown_connections 2024-05-09T08:51:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.666+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/3791446531 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557be77a9ba0 0x557be77abf90 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.666+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/3791446531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557be772d4f0 0x557be77395d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.666+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/3791446531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557be7729920 0x557be772b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.666+0000 7f6a1ffb8740 1 --2- 172.21.15.98:0/3791446531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557be7728f60 0x557be772af70 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.666+0000 7f69faff5640 0 client.16047 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.666+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3791446531 >> 172.21.15.98:0/3791446531 conn(0x557be771b410 msgr2=0x557be772c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.666+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3791446531 shutdown_connections 2024-05-09T08:51:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.666+0000 7f6a1ffb8740 1 -- 172.21.15.98:0/3791446531 wait complete. 2024-05-09T08:51:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.666+0000 7f6a1ffb8740 20 client.16047 trim_cache size 0 max 16384 2024-05-09T08:51:40.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:51:40.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:51:40.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:51:40.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:51:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.667+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1707303374 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6a000a2fd0 msgr2=0x7f6a000a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.667+0000 7f6a1d1ce640 1 --2- 172.21.15.98:0/1707303374 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6a000a2fd0 0x7f6a000a5490 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6a10005fd0 tx=0x7f6a10009600 comp rx=0 tx=0).stop 2024-05-09T08:51:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.667+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1707303374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a18101120 msgr2=0x7f6a18197470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.667+0000 7f6a1d1ce640 1 --2- 172.21.15.98:0/1707303374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a18101120 0x7f6a18197470 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c000c00 tx=0x7f6a0c0056d0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.667+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1707303374 shutdown_connections 2024-05-09T08:51:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.667+0000 7f6a1d1ce640 1 --2- 172.21.15.98:0/1707303374 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6a000a2fd0 0x7f6a000a5490 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.667+0000 7f6a1d1ce640 1 --2- 172.21.15.98:0/1707303374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a18106720 0x7f6a1819b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.667+0000 7f6a1d1ce640 1 --2- 172.21.15.98:0/1707303374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a18101120 0x7f6a18197470 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.667+0000 7f6a1d1ce640 1 --2- 172.21.15.98:0/1707303374 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a18100760 0x7f6a18196f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:40.671 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.667+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1707303374 >> 172.21.15.98:0/1707303374 conn(0x7f6a180778b0 msgr2=0x7f6a18103280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:40.671 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.667+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1707303374 shutdown_connections 2024-05-09T08:51:40.671 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:40.667+0000 7f6a1d1ce640 1 -- 172.21.15.98:0/1707303374 wait complete. 2024-05-09T08:51:40.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:40 smithi105 ceph-mon[34498]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T08:51:40.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:40 smithi098 ceph-mon[30092]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T08:51:42.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:42 smithi142 ceph-mon[34625]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-05-09T08:51:42.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:42 smithi105 ceph-mon[34498]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-05-09T08:51:42.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:42 smithi098 ceph-mon[30092]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-05-09T08:51:43.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:43 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:51:43.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:43 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:51:43.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:43 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:51:44.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:44 smithi142 ceph-mon[34625]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:51:44.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:44 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:44.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:44 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:44.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:44 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:44.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:44 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:44.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:44 smithi105 ceph-mon[34498]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:51:44.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:44 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:44.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:44 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:44.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:44 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:44.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:44 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:44.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:44 smithi098 ceph-mon[30092]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:51:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:44 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:44 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:44 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:44 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:45.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:45 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:51:45.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:45 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:51:45.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:45 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:45.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:45 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:51:45.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:45 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:51:45.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:45 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:45.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:45 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:51:45.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:45 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:51:45.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:45 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:51:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:46 smithi142 ceph-mon[34625]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:51:46.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:46 smithi105 ceph-mon[34498]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:51:46.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:46 smithi098 ceph-mon[30092]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:51:47.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:47 smithi142 ceph-mon[34625]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:51:47.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:47 smithi105 ceph-mon[34498]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:51:47.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:47 smithi098 ceph-mon[30092]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:51:49.311 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:51:49.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:51:49.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.431+0000 7f6cc759e640 1 -- 172.21.15.98:0/1523500060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc810e020 msgr2=0x7f6cc8110410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.431+0000 7f6cc759e640 1 --2- 172.21.15.98:0/1523500060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc810e020 0x7f6cc8110410 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009ca0 tx=0x7f6cc002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.431+0000 7f6cc759e640 1 -- 172.21.15.98:0/1523500060 shutdown_connections 2024-05-09T08:51:49.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.431+0000 7f6cc759e640 1 --2- 172.21.15.98:0/1523500060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc810e020 0x7f6cc8110410 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.431+0000 7f6cc759e640 1 --2- 172.21.15.98:0/1523500060 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc8100e20 0x7f6cc810dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.431+0000 7f6cc759e640 1 --2- 172.21.15.98:0/1523500060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cc8100470 0x7f6cc8100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.431+0000 7f6cc759e640 1 -- 172.21.15.98:0/1523500060 >> 172.21.15.98:0/1523500060 conn(0x7f6cc80fc0b0 msgr2=0x7f6cc80fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:49.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.431+0000 7f6cc759e640 1 -- 172.21.15.98:0/1523500060 shutdown_connections 2024-05-09T08:51:49.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.431+0000 7f6cc759e640 1 -- 172.21.15.98:0/1523500060 wait complete. 2024-05-09T08:51:49.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc759e640 1 Processor -- start 2024-05-09T08:51:49.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc759e640 1 -- start start 2024-05-09T08:51:49.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc8100470 0x7f6cc819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc759e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cc8100e20 0x7f6cc819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc759e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc810e020 0x7f6cc819ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc759e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc8113cd0 con 0x7f6cc8100470 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc759e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc8113b50 con 0x7f6cc8100e20 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc759e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc8113e50 con 0x7f6cc810e020 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc8100470 0x7f6cc819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc8100470 0x7f6cc819a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47648/0 (socket says 172.21.15.98:47648) 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc659c640 1 -- 172.21.15.98:0/2982412221 learned_addr learned my addr 172.21.15.98:0/2982412221 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc6d9d640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc810e020 0x7f6cc819ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc5d9b640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cc8100e20 0x7f6cc819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc659c640 1 -- 172.21.15.98:0/2982412221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc810e020 msgr2=0x7f6cc819ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc659c640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc810e020 0x7f6cc819ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc659c640 1 -- 172.21.15.98:0/2982412221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cc8100e20 msgr2=0x7f6cc819a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc659c640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cc8100e20 0x7f6cc819a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc659c640 1 -- 172.21.15.98:0/2982412221 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc819f160 con 0x7f6cc8100470 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.432+0000 7f6cc5d9b640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cc8100e20 0x7f6cc819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:49.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.433+0000 7f6cc659c640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc8100470 0x7f6cc819a170 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800e9f0 tx=0x7f6cb800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:49.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.433+0000 7f6cb77fe640 1 -- 172.21.15.98:0/2982412221 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb800cd80 con 0x7f6cc8100470 2024-05-09T08:51:49.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.433+0000 7f6cb77fe640 1 -- 172.21.15.98:0/2982412221 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cb8010910 con 0x7f6cc8100470 2024-05-09T08:51:49.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.433+0000 7f6cc759e640 1 -- 172.21.15.98:0/2982412221 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cc819f450 con 0x7f6cc8100470 2024-05-09T08:51:49.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.433+0000 7f6cb77fe640 1 -- 172.21.15.98:0/2982412221 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8018990 con 0x7f6cc8100470 2024-05-09T08:51:49.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.434+0000 7f6cc759e640 1 -- 172.21.15.98:0/2982412221 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cc81a6ce0 con 0x7f6cc8100470 2024-05-09T08:51:49.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.436+0000 7f6cb77fe640 1 -- 172.21.15.98:0/2982412221 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f6cb8010ac0 con 0x7f6cc8100470 2024-05-09T08:51:49.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.436+0000 7f6ccf9eb740 1 Processor -- start 2024-05-09T08:51:49.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.437+0000 7f6ccf9eb740 1 -- start start 2024-05-09T08:51:49.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.437+0000 7f6ccf9eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de0e6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.437+0000 7f6ccf9eb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 0x5614de0f96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.437+0000 7f6ccf9eb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de0fc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.437+0000 7f6ccf9eb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5614de04bd00 con 0x5614de0f9c30 2024-05-09T08:51:49.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.437+0000 7f6ccf9eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5614ddff34c0 con 0x5614de0e6180 2024-05-09T08:51:49.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.437+0000 7f6ccf9eb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5614de047cc0 con 0x5614de0e6b30 2024-05-09T08:51:49.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.437+0000 7f6cc5d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de0fc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.437+0000 7f6cc6d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de0e6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.437+0000 7f6cc5d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de0fc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47656/0 (socket says 172.21.15.98:47656) 2024-05-09T08:51:49.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.437+0000 7f6cc5d9b640 1 -- 172.21.15.98:0/998973523 learned_addr learned my addr 172.21.15.98:0/998973523 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:49.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.437+0000 7f6cc659c640 1 --2- 172.21.15.98:0/998973523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 0x5614de0f96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.437+0000 7f6cc5d9b640 1 -- 172.21.15.98:0/998973523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 msgr2=0x5614de0f96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.437+0000 7f6cc5d9b640 1 --2- 172.21.15.98:0/998973523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 0x5614de0f96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.438+0000 7f6cc5d9b640 1 -- 172.21.15.98:0/998973523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 msgr2=0x5614de0e6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.438+0000 7f6cc5d9b640 1 --2- 172.21.15.98:0/998973523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de0e6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.438+0000 7f6cc5d9b640 1 -- 172.21.15.98:0/998973523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614dde297d0 con 0x5614de0f9c30 2024-05-09T08:51:49.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.438+0000 7f6cc6d9d640 1 --2- 172.21.15.98:0/998973523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de0e6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:49.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.438+0000 7f6cb77fe640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6cb00a2eb0 0x7f6cb00a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.438+0000 7f6cc5d9b640 1 --2- 172.21.15.98:0/998973523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de0fc020 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc009ba0 tx=0x7f6cbc00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d047c26cf11507 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.438+0000 7f6cc6d9d640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6cb00a2eb0 0x7f6cb00a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.438+0000 7f6cb77fe640 1 -- 172.21.15.98:0/2982412221 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f6cb80cff40 con 0x7f6cc8100470 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.438+0000 7f6cb4ff9640 1 -- 172.21.15.98:0/998973523 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc016070 con 0x5614de0f9c30 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.438+0000 7f6cb4ff9640 1 -- 172.21.15.98:0/998973523 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cbc011950 con 0x5614de0f9c30 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.438+0000 7f6cb4ff9640 1 -- 172.21.15.98:0/998973523 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc010870 con 0x5614de0f9c30 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.439+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/998973523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 msgr2=0x5614de0fc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.439+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/998973523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de0fc020 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc009ba0 tx=0x7f6cbc00e6c0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.439+0000 7f6cc6d9d640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6cb00a2eb0 0x7f6cb00a5370 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0005cc0 tx=0x7f6cc0005c30 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.439+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/998973523 shutdown_connections 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.439+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/998973523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de0fc020 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.439+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/998973523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 0x5614de0f96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.439+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/998973523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de0e6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.439+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/998973523 >> 172.21.15.98:0/998973523 conn(0x5614de0e2750 msgr2=0x5614de0e2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:49.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.439+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/998973523 shutdown_connections 2024-05-09T08:51:49.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.439+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/998973523 wait complete. 2024-05-09T08:51:49.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.439+0000 7f6ccf9eb740 1 -- start start 2024-05-09T08:51:49.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6ccf9eb740 1 Processor -- start 2024-05-09T08:51:49.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6ccf9eb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6180 0x5614de0f6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6ccf9eb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0e6b30 0x5614de0f7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6ccf9eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0f9c30 0x5614de0f7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6ccf9eb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5614de0fc9f0 con 0x5614de0e6b30 2024-05-09T08:51:49.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6ccf9eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5614de006b90 con 0x5614de0f9c30 2024-05-09T08:51:49.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6ccf9eb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5614de0fcb70 con 0x5614de0e6180 2024-05-09T08:51:49.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6ccf9eb740 10 client.0 _ll_register_callbacks cb 0x5614de042da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:49.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6cc5d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0e6b30 0x5614de0f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6cc5d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0e6b30 0x5614de0f7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47664/0 (socket says 172.21.15.98:47664) 2024-05-09T08:51:49.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6cc5d9b640 1 -- 172.21.15.98:0/1954799696 learned_addr learned my addr 172.21.15.98:0/1954799696 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:49.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6cc659c640 1 --2- 172.21.15.98:0/1954799696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6180 0x5614de0f6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6cc6d9d640 1 --2- 172.21.15.98:0/1954799696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0f9c30 0x5614de0f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6cc5d9b640 1 -- 172.21.15.98:0/1954799696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6180 msgr2=0x5614de0f6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6cc5d9b640 1 --2- 172.21.15.98:0/1954799696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6180 0x5614de0f6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6cc5d9b640 1 -- 172.21.15.98:0/1954799696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0f9c30 msgr2=0x5614de0f7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.440+0000 7f6cc5d9b640 1 --2- 172.21.15.98:0/1954799696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0f9c30 0x5614de0f7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.441+0000 7f6cc5d9b640 1 -- 172.21.15.98:0/1954799696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614ddd5f3f0 con 0x5614de0e6b30 2024-05-09T08:51:49.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.441+0000 7f6cc5d9b640 1 --2- 172.21.15.98:0/1954799696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0e6b30 0x5614de0f7140 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc000c00 tx=0x7f6cbc011680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4932b8e81d7491b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.441+0000 7f6cadffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.441+0000 7f6cadffb640 1 -- 172.21.15.98:0/1954799696 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc027020 con 0x5614de0e6b30 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.441+0000 7f6cadffb640 1 -- 172.21.15.98:0/1954799696 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f6cbc004020 con 0x5614de0e6b30 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.441+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1954799696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cbc011950 con 0x5614de0e6b30 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.441+0000 7f6cadffb640 1 -- 172.21.15.98:0/1954799696 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6cbc016050 con 0x5614de0e6b30 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.441+0000 7f6ccf9eb740 10 client.16059 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.441+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1954799696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5614ddd1f0f0 con 0x5614de0e6b30 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.442+0000 7f6cadffb640 1 -- 172.21.15.98:0/1954799696 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cbc011770 con 0x5614de0e6b30 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.442+0000 7f6cadffb640 1 -- 172.21.15.98:0/1954799696 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6cbc00f040 con 0x5614de0e6b30 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.442+0000 7f6ccf9eb740 10 client.16059 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.442+0000 7f6ccf9eb740 10 client.16059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.442+0000 7f6ccf9eb740 20 client.16059 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.442+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1954799696 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5614de10a3d0 0x5614de10c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.442+0000 7f6ccf9eb740 4 client.16059 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.442+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1954799696 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6cbc011950 con 0x5614de10a3d0 2024-05-09T08:51:49.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.442+0000 7f6cc659c640 1 --2- 172.21.15.98:0/1954799696 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5614de10a3d0 0x5614de10c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.443+0000 7f6cadffb640 10 client.16059 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:49.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.443+0000 7f6cc659c640 1 --2- 172.21.15.98:0/1954799696 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5614de10a3d0 0x5614de10c810 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52345a4f8d5a2466 server_cookie=ee356995b07bcc31 in_seq=0 out_seq=0 2024-05-09T08:51:49.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.443+0000 7f6cadffb640 1 -- 172.21.15.98:0/1954799696 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f6cbc011950 con 0x5614de10a3d0 2024-05-09T08:51:49.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.443+0000 7f6cadffb640 10 client.16059 handle_command_reply: tid=0 2024-05-09T08:51:49.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.443+0000 7f6ccf9eb740 1 client.16059 shutdown 2024-05-09T08:51:49.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.443+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1954799696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0e6b30 msgr2=0x5614de0f7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.444+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1954799696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0e6b30 0x5614de0f7140 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc000c00 tx=0x7f6cbc011680 comp rx=0 tx=0).stop 2024-05-09T08:51:49.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.444+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1954799696 shutdown_connections 2024-05-09T08:51:49.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.444+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1954799696 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5614de10a3d0 0x5614de10c810 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.444+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1954799696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0f9c30 0x5614de0f7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.444+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1954799696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0e6b30 0x5614de0f7140 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.444+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1954799696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6180 0x5614de0f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.444+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1954799696 >> 172.21.15.98:0/1954799696 conn(0x5614de0e2750 msgr2=0x5614de0e8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:49.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.444+0000 7f6cadffb640 0 client.16059 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:49.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.444+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1954799696 shutdown_connections 2024-05-09T08:51:49.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.444+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1954799696 wait complete. 2024-05-09T08:51:49.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.444+0000 7f6ccf9eb740 20 client.16059 trim_cache size 0 max 16384 2024-05-09T08:51:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.461+0000 7f6ccf9eb740 1 Processor -- start 2024-05-09T08:51:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.461+0000 7f6ccf9eb740 1 -- start start 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.461+0000 7f6ccf9eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de105920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.461+0000 7f6ccf9eb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 0x5614de0f72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.461+0000 7f6ccf9eb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de112120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6ccf9eb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc81acae0 con 0x5614de0f9c30 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6ccf9eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc81ae620 con 0x5614de0e6180 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6ccf9eb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc81ad1b0 con 0x5614de0e6b30 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6cc6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de112120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6cc6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de112120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47666/0 (socket says 172.21.15.98:47666) 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6cc6d9d640 1 -- 172.21.15.98:0/1693834549 learned_addr learned my addr 172.21.15.98:0/1693834549 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6cc5d9b640 1 --2- 172.21.15.98:0/1693834549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 0x5614de0f72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6cc659c640 1 --2- 172.21.15.98:0/1693834549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de105920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6cc6d9d640 1 -- 172.21.15.98:0/1693834549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 msgr2=0x5614de0f72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6cc6d9d640 1 --2- 172.21.15.98:0/1693834549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 0x5614de0f72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6cc6d9d640 1 -- 172.21.15.98:0/1693834549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 msgr2=0x5614de105920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6cc6d9d640 1 --2- 172.21.15.98:0/1693834549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de105920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6cc6d9d640 1 -- 172.21.15.98:0/1693834549 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614de0e30d0 con 0x5614de0f9c30 2024-05-09T08:51:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6cc5d9b640 1 --2- 172.21.15.98:0/1693834549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 0x5614de0f72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6cc659c640 1 --2- 172.21.15.98:0/1693834549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6cc6d9d640 1 --2- 172.21.15.98:0/1693834549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de112120 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8102650 tx=0x7f6cc0040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ce55d56194d127 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6caffff640 1 -- 172.21.15.98:0/1693834549 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0031920 con 0x5614de0f9c30 2024-05-09T08:51:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6caffff640 1 -- 172.21.15.98:0/1693834549 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cc0036040 con 0x5614de0f9c30 2024-05-09T08:51:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.462+0000 7f6caffff640 1 -- 172.21.15.98:0/1693834549 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc002f950 con 0x5614de0f9c30 2024-05-09T08:51:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.463+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1693834549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 msgr2=0x5614de112120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.463+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1693834549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de112120 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8102650 tx=0x7f6cc0040000 comp rx=0 tx=0).stop 2024-05-09T08:51:49.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.463+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1693834549 shutdown_connections 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.463+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1693834549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de112120 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.463+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1693834549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 0x5614de0f72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.463+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1693834549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.463+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1693834549 >> 172.21.15.98:0/1693834549 conn(0x5614de0e2750 msgr2=0x5614de0f6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.463+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1693834549 shutdown_connections 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.463+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1693834549 wait complete. 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6ccf9eb740 1 -- start start 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6ccf9eb740 1 Processor -- start 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6ccf9eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de101e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6ccf9eb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 0x5614de102370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6ccf9eb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de0f3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6ccf9eb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc81adf50 con 0x5614de0f9c30 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6ccf9eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc81ad880 con 0x5614de0e6180 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6ccf9eb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc81aaac0 con 0x5614de0e6b30 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6ccf9eb740 10 client.0 _ll_register_callbacks cb 0x7f6cb8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6cc6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de0f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6cc6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de0f3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47680/0 (socket says 172.21.15.98:47680) 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6cc6d9d640 1 -- 172.21.15.98:0/1258791733 learned_addr learned my addr 172.21.15.98:0/1258791733 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6cc659c640 1 --2- 172.21.15.98:0/1258791733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de101e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6cc5d9b640 1 --2- 172.21.15.98:0/1258791733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 0x5614de102370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6cc6d9d640 1 -- 172.21.15.98:0/1258791733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 msgr2=0x5614de102370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6cc6d9d640 1 --2- 172.21.15.98:0/1258791733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 0x5614de102370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6cc6d9d640 1 -- 172.21.15.98:0/1258791733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 msgr2=0x5614de101e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6cc6d9d640 1 --2- 172.21.15.98:0/1258791733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de101e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6cc6d9d640 1 -- 172.21.15.98:0/1258791733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614de0f3b50 con 0x5614de0f9c30 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.464+0000 7f6cc659c640 1 --2- 172.21.15.98:0/1258791733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.465+0000 7f6cc6d9d640 1 --2- 172.21.15.98:0/1258791733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de0f3320 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009ca0 tx=0x7f6cc002f6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada22f962f3e42a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.465+0000 7f6cadffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.465+0000 7f6cadffb640 1 -- 172.21.15.98:0/1258791733 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0003fc0 con 0x5614de0f9c30 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.465+0000 7f6cadffb640 1 -- 172.21.15.98:0/1258791733 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f6cc0041040 con 0x5614de0f9c30 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.465+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1258791733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cc0036040 con 0x5614de0f9c30 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.465+0000 7f6cadffb640 1 -- 172.21.15.98:0/1258791733 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6cc0064050 con 0x5614de0f9c30 2024-05-09T08:51:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.465+0000 7f6ccf9eb740 10 client.16065 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.465+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1258791733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5614de1704b0 con 0x5614de0f9c30 2024-05-09T08:51:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.465+0000 7f6cadffb640 1 -- 172.21.15.98:0/1258791733 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cc0004640 con 0x5614de0f9c30 2024-05-09T08:51:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.465+0000 7f6cadffb640 1 -- 172.21.15.98:0/1258791733 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6cc0030940 con 0x5614de0f9c30 2024-05-09T08:51:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.466+0000 7f6ccf9eb740 10 client.16065 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.466+0000 7f6ccf9eb740 10 client.16065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.466+0000 7f6ccf9eb740 20 client.16065 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.466+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1258791733 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5614de170b00 0x5614de172f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.466+0000 7f6ccf9eb740 4 client.16065 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:51:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.466+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1258791733 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f6cc0036040 con 0x5614de170b00 2024-05-09T08:51:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.466+0000 7f6cc659c640 1 --2- 172.21.15.98:0/1258791733 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5614de170b00 0x5614de172f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.466+0000 7f6cc659c640 1 --2- 172.21.15.98:0/1258791733 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5614de170b00 0x5614de172f20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b96f0aa86c90199b server_cookie=e005899a17074c71 in_seq=0 out_seq=0 2024-05-09T08:51:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.466+0000 7f6cadffb640 10 client.16065 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6cadffb640 1 -- 172.21.15.98:0/1258791733 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f6cc0036040 con 0x5614de170b00 2024-05-09T08:51:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6cadffb640 10 client.16065 handle_command_reply: tid=0 2024-05-09T08:51:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6ccf9eb740 1 client.16065 shutdown 2024-05-09T08:51:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1258791733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 msgr2=0x5614de0f3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1258791733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de0f3320 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009ca0 tx=0x7f6cc002f6b0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1258791733 shutdown_connections 2024-05-09T08:51:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1258791733 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5614de170b00 0x5614de172f20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1258791733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614de0f9c30 0x5614de0f3320 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1258791733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5614de0e6b30 0x5614de102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6cadffb640 0 client.16065 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6ccf9eb740 1 --2- 172.21.15.98:0/1258791733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5614de0e6180 0x5614de101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1258791733 >> 172.21.15.98:0/1258791733 conn(0x5614de0e2750 msgr2=0x5614de0e8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1258791733 shutdown_connections 2024-05-09T08:51:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6ccf9eb740 1 -- 172.21.15.98:0/1258791733 wait complete. 2024-05-09T08:51:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6ccf9eb740 20 client.16065 trim_cache size 0 max 16384 2024-05-09T08:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:51:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6cc759e640 1 -- 172.21.15.98:0/2982412221 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6cb00a2eb0 msgr2=0x7f6cb00a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6cc759e640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6cb00a2eb0 0x7f6cb00a5370 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0005cc0 tx=0x7f6cc0005c30 comp rx=0 tx=0).stop 2024-05-09T08:51:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6cc759e640 1 -- 172.21.15.98:0/2982412221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc8100470 msgr2=0x7f6cc819a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.467+0000 7f6cc759e640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc8100470 0x7f6cc819a170 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800e9f0 tx=0x7f6cb800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.468+0000 7f6cc759e640 1 -- 172.21.15.98:0/2982412221 shutdown_connections 2024-05-09T08:51:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.468+0000 7f6cc759e640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6cb00a2eb0 0x7f6cb00a5370 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.468+0000 7f6cc759e640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc810e020 0x7f6cc819ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.468+0000 7f6cc759e640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cc8100e20 0x7f6cc819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.468+0000 7f6cc759e640 1 --2- 172.21.15.98:0/2982412221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc8100470 0x7f6cc819a170 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.468+0000 7f6cc759e640 1 -- 172.21.15.98:0/2982412221 >> 172.21.15.98:0/2982412221 conn(0x7f6cc80fc0b0 msgr2=0x7f6cc810f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.468+0000 7f6cc759e640 1 -- 172.21.15.98:0/2982412221 shutdown_connections 2024-05-09T08:51:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.468+0000 7f6cc759e640 1 -- 172.21.15.98:0/2982412221 wait complete. 2024-05-09T08:51:49.478 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:51:49.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:51:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.588+0000 7f72c1e58640 1 -- 172.21.15.98:0/2787066597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72bc101820 msgr2=0x7f72bc101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.588+0000 7f72c1e58640 1 --2- 172.21.15.98:0/2787066597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72bc101820 0x7f72bc101ca0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009890 tx=0x7f72b002a360 comp rx=0 tx=0).stop 2024-05-09T08:51:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.588+0000 7f72c1e58640 1 -- 172.21.15.98:0/2787066597 shutdown_connections 2024-05-09T08:51:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.588+0000 7f72c1e58640 1 --2- 172.21.15.98:0/2787066597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72bc1021e0 0x7f72bc10e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.588+0000 7f72c1e58640 1 --2- 172.21.15.98:0/2787066597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72bc101820 0x7f72bc101ca0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.588+0000 7f72c1e58640 1 --2- 172.21.15.98:0/2787066597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72bc107830 0x7f72bc107c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.588+0000 7f72c1e58640 1 -- 172.21.15.98:0/2787066597 >> 172.21.15.98:0/2787066597 conn(0x7f72bc0fd2d0 msgr2=0x7f72bc0ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.588+0000 7f72c1e58640 1 -- 172.21.15.98:0/2787066597 shutdown_connections 2024-05-09T08:51:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.588+0000 7f72c1e58640 1 -- 172.21.15.98:0/2787066597 wait complete. 2024-05-09T08:51:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.589+0000 7f72c1e58640 1 Processor -- start 2024-05-09T08:51:49.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.589+0000 7f72c1e58640 1 -- start start 2024-05-09T08:51:49.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.589+0000 7f72c1e58640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72bc101820 0x7f72bc198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.589+0000 7f72c1e58640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72bc1021e0 0x7f72bc1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.589+0000 7f72c1e58640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72bc107830 0x7f72bc19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.589+0000 7f72c1e58640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f72bc111c00 con 0x7f72bc1021e0 2024-05-09T08:51:49.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.589+0000 7f72c1e58640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f72bc111a80 con 0x7f72bc101820 2024-05-09T08:51:49.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.589+0000 7f72c1e58640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f72bc111d80 con 0x7f72bc107830 2024-05-09T08:51:49.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.589+0000 7f72baffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72bc1021e0 0x7f72bc1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.589+0000 7f72bb7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72bc101820 0x7f72bc198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.589+0000 7f72baffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72bc1021e0 0x7f72bc1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47700/0 (socket says 172.21.15.98:47700) 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.589+0000 7f72baffd640 1 -- 172.21.15.98:0/1585667272 learned_addr learned my addr 172.21.15.98:0/1585667272 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.589+0000 7f72bb7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72bc101820 0x7f72bc198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:56010/0 (socket says 172.21.15.98:56010) 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.590+0000 7f72bbfff640 1 --2- 172.21.15.98:0/1585667272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72bc107830 0x7f72bc19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.590+0000 7f72bb7fe640 1 -- 172.21.15.98:0/1585667272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72bc107830 msgr2=0x7f72bc19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.590+0000 7f72bb7fe640 1 --2- 172.21.15.98:0/1585667272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72bc107830 0x7f72bc19ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.590+0000 7f72bb7fe640 1 -- 172.21.15.98:0/1585667272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72bc1021e0 msgr2=0x7f72bc1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.590+0000 7f72bb7fe640 1 --2- 172.21.15.98:0/1585667272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72bc1021e0 0x7f72bc1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.590+0000 7f72bb7fe640 1 -- 172.21.15.98:0/1585667272 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72bc19d170 con 0x7f72bc101820 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.590+0000 7f72baffd640 1 --2- 172.21.15.98:0/1585667272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72bc1021e0 0x7f72bc1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.590+0000 7f72bb7fe640 1 --2- 172.21.15.98:0/1585667272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72bc101820 0x7f72bc198180 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f72ac005fd0 tx=0x7f72ac00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.590+0000 7f72b8ff9640 1 -- 172.21.15.98:0/1585667272 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac00cd00 con 0x7f72bc101820 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.590+0000 7f72c1e58640 1 -- 172.21.15.98:0/1585667272 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72bc19d460 con 0x7f72bc101820 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.590+0000 7f72c1e58640 1 -- 172.21.15.98:0/1585667272 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72bc073a60 con 0x7f72bc101820 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.590+0000 7f72b8ff9640 1 -- 172.21.15.98:0/1585667272 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72ac0035f0 con 0x7f72bc101820 2024-05-09T08:51:49.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.590+0000 7f72b8ff9640 1 -- 172.21.15.98:0/1585667272 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac010670 con 0x7f72bc101820 2024-05-09T08:51:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.591+0000 7f72c1e58640 1 -- 172.21.15.98:0/1585667272 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72bc068650 con 0x7f72bc101820 2024-05-09T08:51:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.593+0000 7f72b8ff9640 1 -- 172.21.15.98:0/1585667272 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f72ac0037a0 con 0x7f72bc101820 2024-05-09T08:51:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.593+0000 7f72b8ff9640 1 --2- 172.21.15.98:0/1585667272 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f72a40a2f30 0x7f72a40a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.593+0000 7f72b8ff9640 1 -- 172.21.15.98:0/1585667272 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f72ac0cffe0 con 0x7f72bc101820 2024-05-09T08:51:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.594+0000 7f72baffd640 1 --2- 172.21.15.98:0/1585667272 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f72a40a2f30 0x7f72a40a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.594+0000 7f72baffd640 1 --2- 172.21.15.98:0/1585667272 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f72a40a2f30 0x7f72a40a53f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f72bc100ad0 tx=0x7f72b00056d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:49.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.597+0000 7f72b8ff9640 1 -- 172.21.15.98:0/1585667272 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f72ac009510 con 0x7f72bc101820 2024-05-09T08:51:49.640 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:49 smithi098 ceph-mon[30092]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:51:49.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.822+0000 7f72c1e58640 1 -- 172.21.15.98:0/1585667272 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72bc109de0 con 0x7f72bc101820 2024-05-09T08:51:49.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.823+0000 7f72b8ff9640 1 -- 172.21.15.98:0/1585667272 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11940 (secure 0 0 0) 0x7f72ac09c600 con 0x7f72bc101820 2024-05-09T08:51:49.822 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:51:49.823 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:50:37.185367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T08:51:49.823 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-05-09T08:51:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.825+0000 7f72c1e58640 1 -- 172.21.15.98:0/1585667272 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f72a40a2f30 msgr2=0x7f72a40a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.825+0000 7f72c1e58640 1 --2- 172.21.15.98:0/1585667272 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f72a40a2f30 0x7f72a40a53f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f72bc100ad0 tx=0x7f72b00056d0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.825+0000 7f72c1e58640 1 -- 172.21.15.98:0/1585667272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72bc101820 msgr2=0x7f72bc198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.825+0000 7f72c1e58640 1 --2- 172.21.15.98:0/1585667272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72bc101820 0x7f72bc198180 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f72ac005fd0 tx=0x7f72ac00ee50 comp rx=0 tx=0).stop 2024-05-09T08:51:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.825+0000 7f72c1e58640 1 -- 172.21.15.98:0/1585667272 shutdown_connections 2024-05-09T08:51:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.825+0000 7f72c1e58640 1 --2- 172.21.15.98:0/1585667272 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f72a40a2f30 0x7f72a40a53f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.825+0000 7f72c1e58640 1 --2- 172.21.15.98:0/1585667272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72bc107830 0x7f72bc19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.825+0000 7f72c1e58640 1 --2- 172.21.15.98:0/1585667272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72bc1021e0 0x7f72bc1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.825+0000 7f72c1e58640 1 --2- 172.21.15.98:0/1585667272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72bc101820 0x7f72bc198180 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.825+0000 7f72c1e58640 1 -- 172.21.15.98:0/1585667272 >> 172.21.15.98:0/1585667272 conn(0x7f72bc0fd2d0 msgr2=0x7f72bc0fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.826+0000 7f72c1e58640 1 -- 172.21.15.98:0/1585667272 shutdown_connections 2024-05-09T08:51:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.826+0000 7f72c1e58640 1 -- 172.21.15.98:0/1585667272 wait complete. 2024-05-09T08:51:49.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:51:49.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:49 smithi142 ceph-mon[34625]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:51:49.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.948+0000 7fd648238640 1 -- 172.21.15.98:0/2874269970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd640106710 msgr2=0x7fd640106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.948+0000 7fd648238640 1 --2- 172.21.15.98:0/2874269970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd640106710 0x7fd640106af0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009930 tx=0x7fd63c02a420 comp rx=0 tx=0).stop 2024-05-09T08:51:49.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.948+0000 7fd648238640 1 -- 172.21.15.98:0/2874269970 shutdown_connections 2024-05-09T08:51:49.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.948+0000 7fd648238640 1 --2- 172.21.15.98:0/2874269970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6401010c0 0x7fd64010d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.948+0000 7fd648238640 1 --2- 172.21.15.98:0/2874269970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd640100700 0x7fd640100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.948+0000 7fd648238640 1 --2- 172.21.15.98:0/2874269970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd640106710 0x7fd640106af0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.948+0000 7fd648238640 1 -- 172.21.15.98:0/2874269970 >> 172.21.15.98:0/2874269970 conn(0x7fd640077830 msgr2=0x7fd640077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.949+0000 7fd648238640 1 -- 172.21.15.98:0/2874269970 shutdown_connections 2024-05-09T08:51:49.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.949+0000 7fd648238640 1 -- 172.21.15.98:0/2874269970 wait complete. 2024-05-09T08:51:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.949+0000 7fd648238640 1 Processor -- start 2024-05-09T08:51:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.949+0000 7fd648238640 1 -- start start 2024-05-09T08:51:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd648238640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd640100700 0x7fd640198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd648238640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6401010c0 0x7fd640199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd6457d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6401010c0 0x7fd640199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd6457d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6401010c0 0x7fd640199490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47714/0 (socket says 172.21.15.98:47714) 2024-05-09T08:51:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd648238640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd640106710 0x7fd6401a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd6457d3640 1 -- 172.21.15.98:0/2531634074 learned_addr learned my addr 172.21.15.98:0/2531634074 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd648238640 1 -- 172.21.15.98:0/2531634074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd640110b70 con 0x7fd6401010c0 2024-05-09T08:51:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd645fd4640 1 --2- 172.21.15.98:0/2531634074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd640100700 0x7fd640198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd648238640 1 -- 172.21.15.98:0/2531634074 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd6401109f0 con 0x7fd640106710 2024-05-09T08:51:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd648238640 1 -- 172.21.15.98:0/2531634074 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd640110cf0 con 0x7fd640100700 2024-05-09T08:51:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd6467d5640 1 --2- 172.21.15.98:0/2531634074 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd640106710 0x7fd6401a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd6457d3640 1 -- 172.21.15.98:0/2531634074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd640100700 msgr2=0x7fd640198f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd6457d3640 1 --2- 172.21.15.98:0/2531634074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd640100700 0x7fd640198f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd6457d3640 1 -- 172.21.15.98:0/2531634074 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd640106710 msgr2=0x7fd6401a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd6457d3640 1 --2- 172.21.15.98:0/2531634074 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd640106710 0x7fd6401a3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.950+0000 7fd6457d3640 1 -- 172.21.15.98:0/2531634074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6401a4510 con 0x7fd6401010c0 2024-05-09T08:51:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.951+0000 7fd6457d3640 1 --2- 172.21.15.98:0/2531634074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6401010c0 0x7fd640199490 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd63000e9f0 tx=0x7fd63000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.951+0000 7fd636ffd640 1 -- 172.21.15.98:0/2531634074 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63000cd80 con 0x7fd6401010c0 2024-05-09T08:51:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.951+0000 7fd648238640 1 -- 172.21.15.98:0/2531634074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6401a46e0 con 0x7fd6401010c0 2024-05-09T08:51:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.951+0000 7fd648238640 1 -- 172.21.15.98:0/2531634074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6401a4c80 con 0x7fd6401010c0 2024-05-09T08:51:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.951+0000 7fd636ffd640 1 -- 172.21.15.98:0/2531634074 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd630010910 con 0x7fd6401010c0 2024-05-09T08:51:49.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.951+0000 7fd636ffd640 1 -- 172.21.15.98:0/2531634074 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6300188f0 con 0x7fd6401010c0 2024-05-09T08:51:49.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.953+0000 7fd636ffd640 1 -- 172.21.15.98:0/2531634074 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fd630018b70 con 0x7fd6401010c0 2024-05-09T08:51:49.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.954+0000 7fd636ffd640 1 --2- 172.21.15.98:0/2531634074 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd62c0a2f30 0x7fd62c0a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.954+0000 7fd636ffd640 1 -- 172.21.15.98:0/2531634074 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fd6300cfe50 con 0x7fd6401010c0 2024-05-09T08:51:49.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.954+0000 7fd645fd4640 1 --2- 172.21.15.98:0/2531634074 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd62c0a2f30 0x7fd62c0a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.954+0000 7fd64b021740 1 Processor -- start 2024-05-09T08:51:49.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd64b021740 1 -- start start 2024-05-09T08:51:49.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd645fd4640 1 --2- 172.21.15.98:0/2531634074 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd62c0a2f30 0x7fd62c0a53f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd63c006010 tx=0x7fd63c000bc0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:49.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd64b021740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7eb230 0x55991e7eb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd64b021740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7ebb50 0x55991e7fa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd64b021740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7fa730 0x55991e7fcb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd64b021740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55991e750c40 con 0x55991e7eb230 2024-05-09T08:51:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd64b021740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55991e6f8580 con 0x55991e7fa730 2024-05-09T08:51:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd64b021740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55991e74cc00 con 0x55991e7ebb50 2024-05-09T08:51:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd645fd4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7eb230 0x55991e7eb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd645fd4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7eb230 0x55991e7eb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47716/0 (socket says 172.21.15.98:47716) 2024-05-09T08:51:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd645fd4640 1 -- 172.21.15.98:0/664463551 learned_addr learned my addr 172.21.15.98:0/664463551 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd6457d3640 1 --2- 172.21.15.98:0/664463551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7ebb50 0x55991e7fa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd6467d5640 1 --2- 172.21.15.98:0/664463551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7fa730 0x55991e7fcb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd645fd4640 1 -- 172.21.15.98:0/664463551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7ebb50 msgr2=0x55991e7fa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd645fd4640 1 --2- 172.21.15.98:0/664463551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7ebb50 0x55991e7fa1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd645fd4640 1 -- 172.21.15.98:0/664463551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7fa730 msgr2=0x55991e7fcb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd645fd4640 1 --2- 172.21.15.98:0/664463551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7fa730 0x55991e7fcb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.955+0000 7fd645fd4640 1 -- 172.21.15.98:0/664463551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55991e4643f0 con 0x55991e7eb230 2024-05-09T08:51:49.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.956+0000 7fd645fd4640 1 --2- 172.21.15.98:0/664463551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7eb230 0x55991e7eb610 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009930 tx=0x7fd63c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d8a750463f4533 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:49.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.956+0000 7fd62bfff640 1 -- 172.21.15.98:0/664463551 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63c035040 con 0x55991e7eb230 2024-05-09T08:51:49.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.956+0000 7fd62bfff640 1 -- 172.21.15.98:0/664463551 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd63c04e070 con 0x55991e7eb230 2024-05-09T08:51:49.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.956+0000 7fd62bfff640 1 -- 172.21.15.98:0/664463551 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63c032d00 con 0x55991e7eb230 2024-05-09T08:51:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.957+0000 7fd64b021740 1 -- 172.21.15.98:0/664463551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7eb230 msgr2=0x55991e7eb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.957+0000 7fd64b021740 1 --2- 172.21.15.98:0/664463551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7eb230 0x55991e7eb610 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009930 tx=0x7fd63c03d000 comp rx=0 tx=0).stop 2024-05-09T08:51:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.957+0000 7fd64b021740 1 -- 172.21.15.98:0/664463551 shutdown_connections 2024-05-09T08:51:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.957+0000 7fd64b021740 1 --2- 172.21.15.98:0/664463551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7fa730 0x55991e7fcb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.957+0000 7fd64b021740 1 --2- 172.21.15.98:0/664463551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7ebb50 0x55991e7fa1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.957+0000 7fd64b021740 1 --2- 172.21.15.98:0/664463551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7eb230 0x55991e7eb610 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.957+0000 7fd64b021740 1 -- 172.21.15.98:0/664463551 >> 172.21.15.98:0/664463551 conn(0x55991e7e7650 msgr2=0x55991e7e7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.957+0000 7fd64b021740 1 -- 172.21.15.98:0/664463551 shutdown_connections 2024-05-09T08:51:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.957+0000 7fd64b021740 1 -- 172.21.15.98:0/664463551 wait complete. 2024-05-09T08:51:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.957+0000 7fd64b021740 1 -- start start 2024-05-09T08:51:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.958+0000 7fd64b021740 1 Processor -- start 2024-05-09T08:51:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.958+0000 7fd64b021740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7eb230 0x55991e7f7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.958+0000 7fd64b021740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7ebb50 0x55991e7f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.958+0000 7fd64b021740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7fa730 0x55991e7f80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.958+0000 7fd64b021740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55991e7fd490 con 0x55991e7fa730 2024-05-09T08:51:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.958+0000 7fd64b021740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55991e70bc40 con 0x55991e7eb230 2024-05-09T08:51:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.958+0000 7fd64b021740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55991e7fd610 con 0x55991e7ebb50 2024-05-09T08:51:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.958+0000 7fd6467d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7fa730 0x55991e7f80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.958+0000 7fd6467d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7fa730 0x55991e7f80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47718/0 (socket says 172.21.15.98:47718) 2024-05-09T08:51:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.958+0000 7fd645fd4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7eb230 0x55991e7f7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.958+0000 7fd6467d5640 1 -- 172.21.15.98:0/3687963234 learned_addr learned my addr 172.21.15.98:0/3687963234 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.958+0000 7fd6457d3640 1 --2- 172.21.15.98:0/3687963234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7ebb50 0x55991e7f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.958+0000 7fd64b021740 10 client.0 _ll_register_callbacks cb 0x55991e463be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.959+0000 7fd6467d5640 1 -- 172.21.15.98:0/3687963234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7ebb50 msgr2=0x55991e7f7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.959+0000 7fd6467d5640 1 --2- 172.21.15.98:0/3687963234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7ebb50 0x55991e7f7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.959+0000 7fd6467d5640 1 -- 172.21.15.98:0/3687963234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7eb230 msgr2=0x55991e7f7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.959+0000 7fd6467d5640 1 --2- 172.21.15.98:0/3687963234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7eb230 0x55991e7f7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.959+0000 7fd6467d5640 1 -- 172.21.15.98:0/3687963234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55991e52e7d0 con 0x55991e7fa730 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.959+0000 7fd645fd4640 1 --2- 172.21.15.98:0/3687963234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7eb230 0x55991e7f7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.959+0000 7fd6467d5640 1 --2- 172.21.15.98:0/3687963234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7fa730 0x55991e7f80b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd638011410 tx=0x7fd6380118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f65dd05f3bebb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.959+0000 7fd6297fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.959+0000 7fd6297fa640 1 -- 172.21.15.98:0/3687963234 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638012920 con 0x55991e7fa730 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.959+0000 7fd6297fa640 1 -- 172.21.15.98:0/3687963234 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fd638012ac0 con 0x55991e7fa730 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.959+0000 7fd64b021740 1 -- 172.21.15.98:0/3687963234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd63c04e070 con 0x55991e7fa730 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.959+0000 7fd6297fa640 1 -- 172.21.15.98:0/3687963234 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd638026050 con 0x55991e7fa730 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.959+0000 7fd64b021740 10 client.16083 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.959+0000 7fd64b021740 1 -- 172.21.15.98:0/3687963234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55991e8096b0 con 0x55991e7fa730 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.960+0000 7fd6297fa640 1 -- 172.21.15.98:0/3687963234 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd63c04e070 con 0x55991e7fa730 2024-05-09T08:51:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.960+0000 7fd6297fa640 1 -- 172.21.15.98:0/3687963234 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd63800f3d0 con 0x55991e7fa730 2024-05-09T08:51:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.960+0000 7fd64b021740 10 client.16083 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.960+0000 7fd64b021740 10 client.16083 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:51:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.960+0000 7fd64b021740 20 client.16083 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.960+0000 7fd64b021740 1 --2- 172.21.15.98:0/3687963234 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55991e80ac10 0x55991e80d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.960+0000 7fd64b021740 4 client.16083 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.960+0000 7fd64b021740 1 -- 172.21.15.98:0/3687963234 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55991e75e370 con 0x55991e80ac10 2024-05-09T08:51:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.960+0000 7fd645fd4640 1 --2- 172.21.15.98:0/3687963234 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55991e80ac10 0x55991e80d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.961+0000 7fd6297fa640 10 client.16083 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:51:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.961+0000 7fd645fd4640 1 --2- 172.21.15.98:0/3687963234 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55991e80ac10 0x55991e80d0a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c196224148f080d9 server_cookie=f64267a6aa403a03 in_seq=0 out_seq=0 2024-05-09T08:51:49.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.961+0000 7fd6297fa640 1 -- 172.21.15.98:0/3687963234 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55991e75e370 con 0x55991e80ac10 2024-05-09T08:51:49.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.961+0000 7fd6297fa640 10 client.16083 handle_command_reply: tid=0 2024-05-09T08:51:49.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.961+0000 7fd64b021740 1 client.16083 shutdown 2024-05-09T08:51:49.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.962+0000 7fd64b021740 1 -- 172.21.15.98:0/3687963234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7fa730 msgr2=0x55991e7f80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.962+0000 7fd64b021740 1 --2- 172.21.15.98:0/3687963234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7fa730 0x55991e7f80b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd638011410 tx=0x7fd6380118d0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.962+0000 7fd64b021740 1 -- 172.21.15.98:0/3687963234 shutdown_connections 2024-05-09T08:51:49.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.962+0000 7fd64b021740 1 --2- 172.21.15.98:0/3687963234 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55991e80ac10 0x55991e80d0a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.962+0000 7fd64b021740 1 --2- 172.21.15.98:0/3687963234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7fa730 0x55991e7f80b0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.962+0000 7fd64b021740 1 --2- 172.21.15.98:0/3687963234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7ebb50 0x55991e7f7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.962+0000 7fd64b021740 1 --2- 172.21.15.98:0/3687963234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7eb230 0x55991e7f7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.962+0000 7fd64b021740 1 -- 172.21.15.98:0/3687963234 >> 172.21.15.98:0/3687963234 conn(0x55991e7e7650 msgr2=0x55991e7ed3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:49.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.962+0000 7fd6297fa640 0 client.16083 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:51:49.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.962+0000 7fd64b021740 1 -- 172.21.15.98:0/3687963234 shutdown_connections 2024-05-09T08:51:49.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.962+0000 7fd64b021740 1 -- 172.21.15.98:0/3687963234 wait complete. 2024-05-09T08:51:49.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.962+0000 7fd64b021740 20 client.16083 trim_cache size 0 max 16384 2024-05-09T08:51:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.975+0000 7fd64b021740 1 Processor -- start 2024-05-09T08:51:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.975+0000 7fd64b021740 1 -- start start 2024-05-09T08:51:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd64b021740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7eb230 0x55991e8063f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd64b021740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7ebb50 0x55991e7f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd64b021740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7fa730 0x55991e7e26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd64b021740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd62400ff50 con 0x55991e7fa730 2024-05-09T08:51:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd64b021740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd624011a90 con 0x55991e7ebb50 2024-05-09T08:51:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd64b021740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd624010620 con 0x55991e7eb230 2024-05-09T08:51:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd6467d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7fa730 0x55991e7e26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd6467d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7fa730 0x55991e7e26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47732/0 (socket says 172.21.15.98:47732) 2024-05-09T08:51:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd6467d5640 1 -- 172.21.15.98:0/3490013434 learned_addr learned my addr 172.21.15.98:0/3490013434 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd645fd4640 1 --2- 172.21.15.98:0/3490013434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7eb230 0x55991e8063f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd6457d3640 1 --2- 172.21.15.98:0/3490013434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7ebb50 0x55991e7f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd6467d5640 1 -- 172.21.15.98:0/3490013434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7eb230 msgr2=0x55991e8063f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd6467d5640 1 --2- 172.21.15.98:0/3490013434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7eb230 0x55991e8063f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd6467d5640 1 -- 172.21.15.98:0/3490013434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7ebb50 msgr2=0x55991e7f6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd6467d5640 1 --2- 172.21.15.98:0/3490013434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7ebb50 0x55991e7f6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd6467d5640 1 -- 172.21.15.98:0/3490013434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55991e7eb980 con 0x55991e7fa730 2024-05-09T08:51:49.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd6467d5640 1 --2- 172.21.15.98:0/3490013434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7fa730 0x55991e7e26c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd63800f820 tx=0x7fd638009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af274a3e3c5bf0c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:49.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd62b7fe640 1 -- 172.21.15.98:0/3490013434 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63800fde0 con 0x55991e7fa730 2024-05-09T08:51:49.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd62b7fe640 1 -- 172.21.15.98:0/3490013434 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd638003e50 con 0x55991e7fa730 2024-05-09T08:51:49.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.976+0000 7fd62b7fe640 1 -- 172.21.15.98:0/3490013434 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638005240 con 0x55991e7fa730 2024-05-09T08:51:49.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.977+0000 7fd64b021740 1 -- 172.21.15.98:0/3490013434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7fa730 msgr2=0x55991e7e26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.977+0000 7fd64b021740 1 --2- 172.21.15.98:0/3490013434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7fa730 0x55991e7e26c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd63800f820 tx=0x7fd638009c20 comp rx=0 tx=0).stop 2024-05-09T08:51:49.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.977+0000 7fd64b021740 1 -- 172.21.15.98:0/3490013434 shutdown_connections 2024-05-09T08:51:49.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.977+0000 7fd64b021740 1 --2- 172.21.15.98:0/3490013434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7fa730 0x55991e7e26c0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.977+0000 7fd64b021740 1 --2- 172.21.15.98:0/3490013434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7ebb50 0x55991e7f6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.977+0000 7fd64b021740 1 --2- 172.21.15.98:0/3490013434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7eb230 0x55991e8063f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.977+0000 7fd64b021740 1 -- 172.21.15.98:0/3490013434 >> 172.21.15.98:0/3490013434 conn(0x55991e7e7650 msgr2=0x55991e7e67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.977+0000 7fd64b021740 1 -- 172.21.15.98:0/3490013434 shutdown_connections 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.977+0000 7fd64b021740 1 -- 172.21.15.98:0/3490013434 wait complete. 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd64b021740 1 -- start start 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd64b021740 1 Processor -- start 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd64b021740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7eb230 0x55991e8025c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd64b021740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7ebb50 0x55991e802b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd64b021740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7fa730 0x55991e7fdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd64b021740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6240113c0 con 0x55991e7ebb50 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd64b021740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd624010cf0 con 0x55991e7fa730 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd64b021740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd62400df30 con 0x55991e7eb230 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd64b021740 10 client.0 _ll_register_callbacks cb 0x7fd6380098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd6457d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7ebb50 0x55991e802b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd6457d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7ebb50 0x55991e802b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47746/0 (socket says 172.21.15.98:47746) 2024-05-09T08:51:49.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd6457d3640 1 -- 172.21.15.98:0/513811626 learned_addr learned my addr 172.21.15.98:0/513811626 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:49.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd6467d5640 1 --2- 172.21.15.98:0/513811626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7fa730 0x55991e7fdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd645fd4640 1 --2- 172.21.15.98:0/513811626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7eb230 0x55991e8025c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd6457d3640 1 -- 172.21.15.98:0/513811626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7eb230 msgr2=0x55991e8025c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd6457d3640 1 --2- 172.21.15.98:0/513811626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7eb230 0x55991e8025c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.978+0000 7fd6457d3640 1 -- 172.21.15.98:0/513811626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7fa730 msgr2=0x55991e7fdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:49.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd6457d3640 1 --2- 172.21.15.98:0/513811626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7fa730 0x55991e7fdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd6457d3640 1 -- 172.21.15.98:0/513811626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55991e7fe2a0 con 0x55991e7ebb50 2024-05-09T08:51:49.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd6467d5640 1 --2- 172.21.15.98:0/513811626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7fa730 0x55991e7fdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:49.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd6457d3640 1 --2- 172.21.15.98:0/513811626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7ebb50 0x55991e802b00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd63001afd0 tx=0x7fd63009c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793e4bbfecc6ea1c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:49.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd6297fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:49.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd6297fa640 1 -- 172.21.15.98:0/513811626 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6300b8020 con 0x55991e7ebb50 2024-05-09T08:51:49.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd64b021740 1 -- 172.21.15.98:0/513811626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd638003e50 con 0x55991e7ebb50 2024-05-09T08:51:49.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd6297fa640 1 -- 172.21.15.98:0/513811626 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fd63009b3c0 con 0x55991e7ebb50 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd6297fa640 1 -- 172.21.15.98:0/513811626 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd630018e60 con 0x55991e7ebb50 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd64b021740 10 client.16089 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd64b021740 1 -- 172.21.15.98:0/513811626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55991e870d70 con 0x55991e7ebb50 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd6297fa640 1 -- 172.21.15.98:0/513811626 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd638003e50 con 0x55991e7ebb50 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd6297fa640 1 -- 172.21.15.98:0/513811626 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd630097ac0 con 0x55991e7ebb50 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd64b021740 10 client.16089 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd64b021740 10 client.16089 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd64b021740 20 client.16089 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd64b021740 1 --2- 172.21.15.98:0/513811626 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55991e8714a0 0x55991e8738c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd64b021740 4 client.16089 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.979+0000 7fd64b021740 1 -- 172.21.15.98:0/513811626 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55991e7586c0 con 0x55991e8714a0 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.980+0000 7fd645fd4640 1 --2- 172.21.15.98:0/513811626 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55991e8714a0 0x55991e8738c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.980+0000 7fd6297fa640 10 client.16089 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.980+0000 7fd645fd4640 1 --2- 172.21.15.98:0/513811626 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55991e8714a0 0x55991e8738c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=899a7916529f9b0c server_cookie=e254746f24022f61 in_seq=0 out_seq=0 2024-05-09T08:51:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.980+0000 7fd6297fa640 1 -- 172.21.15.98:0/513811626 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55991e7586c0 con 0x55991e8714a0 2024-05-09T08:51:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.980+0000 7fd6297fa640 10 client.16089 handle_command_reply: tid=0 2024-05-09T08:51:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.980+0000 7fd64b021740 1 client.16089 shutdown 2024-05-09T08:51:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.980+0000 7fd64b021740 1 -- 172.21.15.98:0/513811626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7ebb50 msgr2=0x55991e802b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.980+0000 7fd64b021740 1 --2- 172.21.15.98:0/513811626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7ebb50 0x55991e802b00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd63001afd0 tx=0x7fd63009c470 comp rx=0 tx=0).stop 2024-05-09T08:51:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd64b021740 1 -- 172.21.15.98:0/513811626 shutdown_connections 2024-05-09T08:51:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd64b021740 1 --2- 172.21.15.98:0/513811626 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55991e8714a0 0x55991e8738c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd64b021740 1 --2- 172.21.15.98:0/513811626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55991e7fa730 0x55991e7fdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd64b021740 1 --2- 172.21.15.98:0/513811626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55991e7ebb50 0x55991e802b00 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd64b021740 1 --2- 172.21.15.98:0/513811626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55991e7eb230 0x55991e8025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd64b021740 1 -- 172.21.15.98:0/513811626 >> 172.21.15.98:0/513811626 conn(0x55991e7e7650 msgr2=0x55991e7e67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd6297fa640 0 client.16089 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:51:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd64b021740 1 -- 172.21.15.98:0/513811626 shutdown_connections 2024-05-09T08:51:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd64b021740 1 -- 172.21.15.98:0/513811626 wait complete. 2024-05-09T08:51:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd64b021740 20 client.16089 trim_cache size 0 max 16384 2024-05-09T08:51:49.985 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:51:49.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd648238640 1 -- 172.21.15.98:0/2531634074 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd62c0a2f30 msgr2=0x7fd62c0a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd648238640 1 --2- 172.21.15.98:0/2531634074 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd62c0a2f30 0x7fd62c0a53f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd63c006010 tx=0x7fd63c000bc0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd648238640 1 -- 172.21.15.98:0/2531634074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6401010c0 msgr2=0x7fd640199490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:49.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd648238640 1 --2- 172.21.15.98:0/2531634074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6401010c0 0x7fd640199490 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd63000e9f0 tx=0x7fd63000eeb0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd648238640 1 -- 172.21.15.98:0/2531634074 shutdown_connections 2024-05-09T08:51:49.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd648238640 1 --2- 172.21.15.98:0/2531634074 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd62c0a2f30 0x7fd62c0a53f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd648238640 1 --2- 172.21.15.98:0/2531634074 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd640106710 0x7fd6401a3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd648238640 1 --2- 172.21.15.98:0/2531634074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6401010c0 0x7fd640199490 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd648238640 1 --2- 172.21.15.98:0/2531634074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd640100700 0x7fd640198f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:49.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.981+0000 7fd648238640 1 -- 172.21.15.98:0/2531634074 >> 172.21.15.98:0/2531634074 conn(0x7fd640077830 msgr2=0x7fd640102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:49.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.982+0000 7fd648238640 1 -- 172.21.15.98:0/2531634074 shutdown_connections 2024-05-09T08:51:49.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:49.982+0000 7fd648238640 1 -- 172.21.15.98:0/2531634074 wait complete. 2024-05-09T08:51:49.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:49 smithi105 ceph-mon[34498]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:51:49.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:51:50.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.102+0000 7f3644b90640 1 -- 172.21.15.98:0/382903267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36401010d0 msgr2=0x7f364010d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.102+0000 7f3644b90640 1 --2- 172.21.15.98:0/382903267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36401010d0 0x7f364010d5e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3638009cd0 tx=0x7f363802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.103+0000 7f3644b90640 1 -- 172.21.15.98:0/382903267 shutdown_connections 2024-05-09T08:51:50.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.103+0000 7f3644b90640 1 --2- 172.21.15.98:0/382903267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36401010d0 0x7f364010d5e0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.103+0000 7f3644b90640 1 --2- 172.21.15.98:0/382903267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3640100710 0x7f3640100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.103+0000 7f3644b90640 1 --2- 172.21.15.98:0/382903267 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3640106720 0x7f3640106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.103+0000 7f3644b90640 1 -- 172.21.15.98:0/382903267 >> 172.21.15.98:0/382903267 conn(0x7f3640077860 msgr2=0x7f3640077c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.103+0000 7f3644b90640 1 -- 172.21.15.98:0/382903267 shutdown_connections 2024-05-09T08:51:50.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.103+0000 7f3644b90640 1 -- 172.21.15.98:0/382903267 wait complete. 2024-05-09T08:51:50.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.104+0000 7f3644b90640 1 Processor -- start 2024-05-09T08:51:50.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.104+0000 7f3644b90640 1 -- start start 2024-05-09T08:51:50.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.104+0000 7f3644b90640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3640100710 0x7f3640198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.104+0000 7f3644b90640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36401010d0 0x7f3640199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.104+0000 7f3644b90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3640106720 0x7f36401a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.104+0000 7f3644b90640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3640110b30 con 0x7f3640106720 2024-05-09T08:51:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.104+0000 7f3644b90640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f36401109b0 con 0x7f36401010d0 2024-05-09T08:51:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.104+0000 7f3644b90640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3640110cb0 con 0x7f3640100710 2024-05-09T08:51:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.104+0000 7f363e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3640100710 0x7f3640198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.104+0000 7f363e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3640100710 0x7f3640198f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:60250/0 (socket says 172.21.15.98:60250) 2024-05-09T08:51:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.104+0000 7f363e59c640 1 -- 172.21.15.98:0/1402391054 learned_addr learned my addr 172.21.15.98:0/1402391054 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.104+0000 7f363dd9b640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36401010d0 0x7f3640199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.104+0000 7f363ed9d640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3640106720 0x7f36401a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.105+0000 7f363e59c640 1 -- 172.21.15.98:0/1402391054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36401010d0 msgr2=0x7f3640199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.105+0000 7f363e59c640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36401010d0 0x7f3640199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.105+0000 7f363e59c640 1 -- 172.21.15.98:0/1402391054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3640106720 msgr2=0x7f36401a3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.105+0000 7f363e59c640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3640106720 0x7f36401a3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.105+0000 7f363e59c640 1 -- 172.21.15.98:0/1402391054 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36401a44d0 con 0x7f3640100710 2024-05-09T08:51:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.105+0000 7f363ed9d640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3640106720 0x7f36401a3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.105+0000 7f363e59c640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3640100710 0x7f3640198f10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f363000e9f0 tx=0x7f363000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.105+0000 7f362f7fe640 1 -- 172.21.15.98:0/1402391054 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363000cd80 con 0x7f3640100710 2024-05-09T08:51:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.105+0000 7f3644b90640 1 -- 172.21.15.98:0/1402391054 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36401a46a0 con 0x7f3640100710 2024-05-09T08:51:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.105+0000 7f362f7fe640 1 -- 172.21.15.98:0/1402391054 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3630005500 con 0x7f3640100710 2024-05-09T08:51:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.105+0000 7f362f7fe640 1 -- 172.21.15.98:0/1402391054 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3630010690 con 0x7f3640100710 2024-05-09T08:51:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.105+0000 7f3644b90640 1 -- 172.21.15.98:0/1402391054 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36401a4bc0 con 0x7f3640100710 2024-05-09T08:51:50.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.108+0000 7f364797c740 1 Processor -- start 2024-05-09T08:51:50.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.109+0000 7f364797c740 1 -- start start 2024-05-09T08:51:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.109+0000 7f364797c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d00333c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.109+0000 7f364797c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003337c50 0x55d0033380d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.109+0000 7f364797c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003338610 0x55d003344340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.109+0000 7f364797c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d003293c40 con 0x55d00333c3f0 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.109+0000 7f364797c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d00323b580 con 0x55d003338610 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.109+0000 7f364797c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d00328fc00 con 0x55d003337c50 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.109+0000 7f362f7fe640 1 -- 172.21.15.98:0/1402391054 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f36300046c0 con 0x7f3640100710 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.109+0000 7f363ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d00333c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.109+0000 7f363ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d00333c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47764/0 (socket says 172.21.15.98:47764) 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.109+0000 7f363ed9d640 1 -- 172.21.15.98:0/86127620 learned_addr learned my addr 172.21.15.98:0/86127620 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.109+0000 7f363e59c640 1 --2- 172.21.15.98:0/86127620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003337c50 0x55d0033380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.109+0000 7f363dd9b640 1 --2- 172.21.15.98:0/86127620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003338610 0x55d003344340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.110+0000 7f363e59c640 1 -- 172.21.15.98:0/86127620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003338610 msgr2=0x55d003344340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.110+0000 7f363e59c640 1 --2- 172.21.15.98:0/86127620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003338610 0x55d003344340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.110+0000 7f363e59c640 1 -- 172.21.15.98:0/86127620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 msgr2=0x55d00333c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.110+0000 7f363e59c640 1 --2- 172.21.15.98:0/86127620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d00333c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.110+0000 7f363e59c640 1 -- 172.21.15.98:0/86127620 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d002fa73f0 con 0x55d003337c50 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.110+0000 7f363ed9d640 1 --2- 172.21.15.98:0/86127620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d00333c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.110+0000 7f363e59c640 1 --2- 172.21.15.98:0/86127620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003337c50 0x55d0033380d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f36300cd0c0 tx=0x7f36300d3f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ac7d155bde24474 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.110+0000 7f362f7fe640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f36280a2fb0 0x7f36280a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.110+0000 7f362cff9640 1 -- 172.21.15.98:0/86127620 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3630098130 con 0x55d003337c50 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.110+0000 7f362cff9640 1 -- 172.21.15.98:0/86127620 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f363009c590 con 0x55d003337c50 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.110+0000 7f362f7fe640 1 -- 172.21.15.98:0/1402391054 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f36300cff70 con 0x7f3640100710 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.110+0000 7f363ed9d640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f36280a2fb0 0x7f36280a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.111+0000 7f364797c740 1 -- 172.21.15.98:0/86127620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003337c50 msgr2=0x55d0033380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.111+0000 7f364797c740 1 --2- 172.21.15.98:0/86127620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003337c50 0x55d0033380d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f36300cd0c0 tx=0x7f36300d3f10 comp rx=0 tx=0).stop 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.111+0000 7f363ed9d640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f36280a2fb0 0x7f36280a5470 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3638009430 tx=0x7f3638005d80 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.111+0000 7f364797c740 1 -- 172.21.15.98:0/86127620 shutdown_connections 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.111+0000 7f364797c740 1 --2- 172.21.15.98:0/86127620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003338610 0x55d003344340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.111+0000 7f364797c740 1 --2- 172.21.15.98:0/86127620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003337c50 0x55d0033380d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.111+0000 7f364797c740 1 --2- 172.21.15.98:0/86127620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d00333c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.111+0000 7f364797c740 1 -- 172.21.15.98:0/86127620 >> 172.21.15.98:0/86127620 conn(0x55d00332a100 msgr2=0x55d00332a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.111+0000 7f364797c740 1 -- 172.21.15.98:0/86127620 shutdown_connections 2024-05-09T08:51:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.111+0000 7f364797c740 1 -- 172.21.15.98:0/86127620 wait complete. 2024-05-09T08:51:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.112+0000 7f364797c740 1 -- start start 2024-05-09T08:51:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.112+0000 7f364797c740 1 Processor -- start 2024-05-09T08:51:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.112+0000 7f364797c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d003337c50 0x55d003344250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.112+0000 7f364797c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 0x55d0033415e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.112+0000 7f364797c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d00333c3f0 0x55d003341bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.112+0000 7f364797c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d003344940 con 0x55d003337c50 2024-05-09T08:51:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.112+0000 7f364797c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d00324ec40 con 0x55d00333c3f0 2024-05-09T08:51:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.112+0000 7f364797c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d003344ac0 con 0x55d003338610 2024-05-09T08:51:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.112+0000 7f364797c740 10 client.0 _ll_register_callbacks cb 0x55d002fa6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.112+0000 7f363e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d003337c50 0x55d003344250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.112+0000 7f363e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d003337c50 0x55d003344250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47776/0 (socket says 172.21.15.98:47776) 2024-05-09T08:51:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.112+0000 7f363e59c640 1 -- 172.21.15.98:0/2752998405 learned_addr learned my addr 172.21.15.98:0/2752998405 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.112+0000 7f363dd9b640 1 --2- 172.21.15.98:0/2752998405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 0x55d0033415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.112+0000 7f363ed9d640 1 --2- 172.21.15.98:0/2752998405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d00333c3f0 0x55d003341bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.113+0000 7f363e59c640 1 -- 172.21.15.98:0/2752998405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 msgr2=0x55d0033415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.113+0000 7f363e59c640 1 --2- 172.21.15.98:0/2752998405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 0x55d0033415e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.113+0000 7f363e59c640 1 -- 172.21.15.98:0/2752998405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d00333c3f0 msgr2=0x55d003341bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.113+0000 7f363e59c640 1 --2- 172.21.15.98:0/2752998405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d00333c3f0 0x55d003341bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.113+0000 7f363e59c640 1 -- 172.21.15.98:0/2752998405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0030717d0 con 0x55d003337c50 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.113+0000 7f363e59c640 1 --2- 172.21.15.98:0/2752998405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d003337c50 0x55d003344250 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f36300982f0 tx=0x7f3630009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5832085d1a98c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.113+0000 7f3625ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.113+0000 7f3625ffb640 1 -- 172.21.15.98:0/2752998405 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363009b7f0 con 0x55d003337c50 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.113+0000 7f3625ffb640 1 -- 172.21.15.98:0/2752998405 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f363000fc30 con 0x55d003337c50 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.113+0000 7f364797c740 1 -- 172.21.15.98:0/2752998405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f363009c590 con 0x55d003337c50 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.113+0000 7f3625ffb640 1 -- 172.21.15.98:0/2752998405 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f363008b050 con 0x55d003337c50 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.113+0000 7f364797c740 10 client.16101 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.113+0000 7f364797c740 1 -- 172.21.15.98:0/2752998405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d003344940 con 0x55d003337c50 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.114+0000 7f3625ffb640 1 -- 172.21.15.98:0/2752998405 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36300b8080 con 0x55d003337c50 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.114+0000 7f3625ffb640 1 -- 172.21.15.98:0/2752998405 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f363001f340 con 0x55d003337c50 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.114+0000 7f364797c740 10 client.16101 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.114+0000 7f364797c740 10 client.16101 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.114+0000 7f364797c740 20 client.16101 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.114+0000 7f364797c740 1 --2- 172.21.15.98:0/2752998405 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55d003351b20 0x55d003353f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.114+0000 7f364797c740 4 client.16101 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.114+0000 7f364797c740 1 -- 172.21.15.98:0/2752998405 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0032a1370 con 0x55d003351b20 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.115+0000 7f363dd9b640 1 --2- 172.21.15.98:0/2752998405 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55d003351b20 0x55d003353f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.115+0000 7f3625ffb640 10 client.16101 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.115+0000 7f363dd9b640 1 --2- 172.21.15.98:0/2752998405 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55d003351b20 0x55d003353f10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5af6f5f32b4016cb server_cookie=12645592a8cb085d in_seq=0 out_seq=0 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f3625ffb640 1 -- 172.21.15.98:0/2752998405 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d0032a1370 con 0x55d003351b20 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f3625ffb640 10 client.16101 handle_command_reply: tid=0 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f364797c740 1 client.16101 shutdown 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f364797c740 1 -- 172.21.15.98:0/2752998405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d003337c50 msgr2=0x55d003344250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f364797c740 1 --2- 172.21.15.98:0/2752998405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d003337c50 0x55d003344250 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f36300982f0 tx=0x7f3630009310 comp rx=0 tx=0).stop 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f364797c740 1 -- 172.21.15.98:0/2752998405 shutdown_connections 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f364797c740 1 --2- 172.21.15.98:0/2752998405 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55d003351b20 0x55d003353f10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f364797c740 1 --2- 172.21.15.98:0/2752998405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d00333c3f0 0x55d003341bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f364797c740 1 --2- 172.21.15.98:0/2752998405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 0x55d0033415e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f3625ffb640 0 client.16101 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:51:50.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f364797c740 1 --2- 172.21.15.98:0/2752998405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d003337c50 0x55d003344250 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f364797c740 1 -- 172.21.15.98:0/2752998405 >> 172.21.15.98:0/2752998405 conn(0x55d00332a100 msgr2=0x55d0033374d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f364797c740 1 -- 172.21.15.98:0/2752998405 shutdown_connections 2024-05-09T08:51:50.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f364797c740 1 -- 172.21.15.98:0/2752998405 wait complete. 2024-05-09T08:51:50.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.116+0000 7f364797c740 20 client.16101 trim_cache size 0 max 16384 2024-05-09T08:51:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.142+0000 7f364797c740 1 Processor -- start 2024-05-09T08:51:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.142+0000 7f364797c740 1 -- start start 2024-05-09T08:51:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.142+0000 7f364797c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003337c50 0x55d00334d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.142+0000 7f364797c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 0x55d003340860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f364797c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d00333a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f364797c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f36401acbf0 con 0x55d00333c3f0 2024-05-09T08:51:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f364797c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f36401ae730 con 0x55d003337c50 2024-05-09T08:51:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f364797c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f36401ad2c0 con 0x55d003338610 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f363ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d00333a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f363ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d00333a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47782/0 (socket says 172.21.15.98:47782) 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f363ed9d640 1 -- 172.21.15.98:0/2086320278 learned_addr learned my addr 172.21.15.98:0/2086320278 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f363e59c640 1 --2- 172.21.15.98:0/2086320278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003337c50 0x55d00334d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f363dd9b640 1 --2- 172.21.15.98:0/2086320278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 0x55d003340860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f363ed9d640 1 -- 172.21.15.98:0/2086320278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 msgr2=0x55d003340860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f363ed9d640 1 --2- 172.21.15.98:0/2086320278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 0x55d003340860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f363ed9d640 1 -- 172.21.15.98:0/2086320278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003337c50 msgr2=0x55d00334d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f363ed9d640 1 --2- 172.21.15.98:0/2086320278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003337c50 0x55d00334d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f363ed9d640 1 -- 172.21.15.98:0/2086320278 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d003354280 con 0x55d00333c3f0 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f363e59c640 1 --2- 172.21.15.98:0/2086320278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003337c50 0x55d00334d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f363ed9d640 1 --2- 172.21.15.98:0/2086320278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d00333a7e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f36380319d0 tx=0x7f3638040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ed149cea15b874 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.143+0000 7f3627fff640 1 -- 172.21.15.98:0/2086320278 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3638056070 con 0x55d00333c3f0 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.144+0000 7f3627fff640 1 -- 172.21.15.98:0/2086320278 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3638036040 con 0x55d00333c3f0 2024-05-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.144+0000 7f3627fff640 1 -- 172.21.15.98:0/2086320278 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3638047990 con 0x55d00333c3f0 2024-05-09T08:51:50.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.144+0000 7f364797c740 1 -- 172.21.15.98:0/2086320278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 msgr2=0x55d00333a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.144+0000 7f364797c740 1 --2- 172.21.15.98:0/2086320278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d00333a7e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f36380319d0 tx=0x7f3638040000 comp rx=0 tx=0).stop 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.144+0000 7f364797c740 1 -- 172.21.15.98:0/2086320278 shutdown_connections 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.144+0000 7f364797c740 1 --2- 172.21.15.98:0/2086320278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d00333a7e0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.144+0000 7f364797c740 1 --2- 172.21.15.98:0/2086320278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 0x55d003340860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.144+0000 7f364797c740 1 --2- 172.21.15.98:0/2086320278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003337c50 0x55d00334d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.144+0000 7f364797c740 1 -- 172.21.15.98:0/2086320278 >> 172.21.15.98:0/2086320278 conn(0x55d00332a100 msgr2=0x55d00333b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f364797c740 1 -- 172.21.15.98:0/2086320278 shutdown_connections 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f364797c740 1 -- 172.21.15.98:0/2086320278 wait complete. 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f364797c740 1 -- start start 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f364797c740 1 Processor -- start 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f364797c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003337c50 0x55d003349da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f364797c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 0x55d00334a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f364797c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d003346d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f364797c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f36401ae060 con 0x55d00333c3f0 2024-05-09T08:51:50.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f364797c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f36401ad990 con 0x55d003337c50 2024-05-09T08:51:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f364797c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f36401aabd0 con 0x55d003338610 2024-05-09T08:51:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f364797c740 10 client.0 _ll_register_callbacks cb 0x7f3634005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f363ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d003346d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f363ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d003346d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47786/0 (socket says 172.21.15.98:47786) 2024-05-09T08:51:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f363ed9d640 1 -- 172.21.15.98:0/2752530268 learned_addr learned my addr 172.21.15.98:0/2752530268 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f363e59c640 1 --2- 172.21.15.98:0/2752530268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003337c50 0x55d003349da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.145+0000 7f363dd9b640 1 --2- 172.21.15.98:0/2752530268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 0x55d00334a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.146+0000 7f363ed9d640 1 -- 172.21.15.98:0/2752530268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 msgr2=0x55d00334a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.146+0000 7f363ed9d640 1 --2- 172.21.15.98:0/2752530268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 0x55d00334a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.146+0000 7f363ed9d640 1 -- 172.21.15.98:0/2752530268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003337c50 msgr2=0x55d003349da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.146+0000 7f363ed9d640 1 --2- 172.21.15.98:0/2752530268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003337c50 0x55d003349da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.146+0000 7f363ed9d640 1 -- 172.21.15.98:0/2752530268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d003347580 con 0x55d00333c3f0 2024-05-09T08:51:50.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.146+0000 7f363ed9d640 1 --2- 172.21.15.98:0/2752530268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d003346d70 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3638031eb0 tx=0x7f363802a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622aaaeb35cd55de server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.146+0000 7f3625ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.146+0000 7f3625ffb640 1 -- 172.21.15.98:0/2752530268 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363805f060 con 0x55d00333c3f0 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.146+0000 7f3625ffb640 1 -- 172.21.15.98:0/2752530268 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f363804e040 con 0x55d00333c3f0 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.146+0000 7f364797c740 1 -- 172.21.15.98:0/2752530268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3638036040 con 0x55d00333c3f0 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.146+0000 7f3625ffb640 1 -- 172.21.15.98:0/2752530268 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3638056030 con 0x55d00333c3f0 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.146+0000 7f364797c740 10 client.16107 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.146+0000 7f364797c740 1 -- 172.21.15.98:0/2752530268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d0033b8a50 con 0x55d00333c3f0 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.147+0000 7f3625ffb640 1 -- 172.21.15.98:0/2752530268 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3638036040 con 0x55d00333c3f0 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.147+0000 7f3625ffb640 1 -- 172.21.15.98:0/2752530268 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3638030070 con 0x55d00333c3f0 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.147+0000 7f364797c740 10 client.16107 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.147+0000 7f364797c740 10 client.16107 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.147+0000 7f364797c740 20 client.16107 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.147+0000 7f364797c740 1 --2- 172.21.15.98:0/2752530268 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55d0033b9180 0x55d0033bb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.147+0000 7f364797c740 4 client.16107 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:51:50.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.147+0000 7f364797c740 1 -- 172.21.15.98:0/2752530268 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d00329b6c0 con 0x55d0033b9180 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.147+0000 7f363e59c640 1 --2- 172.21.15.98:0/2752530268 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55d0033b9180 0x55d0033bb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f3625ffb640 10 client.16107 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f363e59c640 1 --2- 172.21.15.98:0/2752530268 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55d0033b9180 0x55d0033bb5a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a3e751aebfac9d41 server_cookie=1a85f210b59708c6 in_seq=0 out_seq=0 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f3625ffb640 1 -- 172.21.15.98:0/2752530268 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d00329b6c0 con 0x55d0033b9180 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f3625ffb640 10 client.16107 handle_command_reply: tid=0 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f364797c740 1 client.16107 shutdown 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f364797c740 1 -- 172.21.15.98:0/2752530268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 msgr2=0x55d003346d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f364797c740 1 --2- 172.21.15.98:0/2752530268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d003346d70 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3638031eb0 tx=0x7f363802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f364797c740 1 -- 172.21.15.98:0/2752530268 shutdown_connections 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f364797c740 1 --2- 172.21.15.98:0/2752530268 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55d0033b9180 0x55d0033bb5a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f3625ffb640 0 client.16107 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f364797c740 1 --2- 172.21.15.98:0/2752530268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d00333c3f0 0x55d003346d70 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f364797c740 1 --2- 172.21.15.98:0/2752530268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d003338610 0x55d00334a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f364797c740 1 --2- 172.21.15.98:0/2752530268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d003337c50 0x55d003349da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f364797c740 1 -- 172.21.15.98:0/2752530268 >> 172.21.15.98:0/2752530268 conn(0x55d00332a100 msgr2=0x55d00333b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f364797c740 1 -- 172.21.15.98:0/2752530268 shutdown_connections 2024-05-09T08:51:50.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f364797c740 1 -- 172.21.15.98:0/2752530268 wait complete. 2024-05-09T08:51:50.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.148+0000 7f364797c740 20 client.16107 trim_cache size 0 max 16384 2024-05-09T08:51:50.153 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:51:50.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.149+0000 7f3644b90640 1 -- 172.21.15.98:0/1402391054 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f36280a2fb0 msgr2=0x7f36280a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.149+0000 7f3644b90640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f36280a2fb0 0x7f36280a5470 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3638009430 tx=0x7f3638005d80 comp rx=0 tx=0).stop 2024-05-09T08:51:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.149+0000 7f3644b90640 1 -- 172.21.15.98:0/1402391054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3640100710 msgr2=0x7f3640198f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.149+0000 7f3644b90640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3640100710 0x7f3640198f10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f363000e9f0 tx=0x7f363000edf0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.149+0000 7f3644b90640 1 -- 172.21.15.98:0/1402391054 shutdown_connections 2024-05-09T08:51:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.149+0000 7f3644b90640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f36280a2fb0 0x7f36280a5470 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.149+0000 7f3644b90640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3640106720 0x7f36401a3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.149+0000 7f3644b90640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36401010d0 0x7f3640199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.149+0000 7f3644b90640 1 --2- 172.21.15.98:0/1402391054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3640100710 0x7f3640198f10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.149+0000 7f3644b90640 1 -- 172.21.15.98:0/1402391054 >> 172.21.15.98:0/1402391054 conn(0x7f3640077860 msgr2=0x7f3640102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.149+0000 7f3644b90640 1 -- 172.21.15.98:0/1402391054 shutdown_connections 2024-05-09T08:51:50.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.149+0000 7f3644b90640 1 -- 172.21.15.98:0/1402391054 wait complete. 2024-05-09T08:51:50.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:51:50.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.265+0000 7f04df6c3640 1 -- 172.21.15.98:0/2689908059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04d8100960 msgr2=0x7f04d8106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.265+0000 7f04df6c3640 1 --2- 172.21.15.98:0/2689908059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04d8100960 0x7f04d8106490 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f04c8009900 tx=0x7f04c802a410 comp rx=0 tx=0).stop 2024-05-09T08:51:50.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.265+0000 7f04df6c3640 1 -- 172.21.15.98:0/2689908059 shutdown_connections 2024-05-09T08:51:50.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.265+0000 7f04df6c3640 1 --2- 172.21.15.98:0/2689908059 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d8106bc0 0x7f04d810e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.265+0000 7f04df6c3640 1 --2- 172.21.15.98:0/2689908059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04d8100960 0x7f04d8106490 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.265+0000 7f04df6c3640 1 --2- 172.21.15.98:0/2689908059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04d80fffb0 0x7f04d8100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.265+0000 7f04df6c3640 1 -- 172.21.15.98:0/2689908059 >> 172.21.15.98:0/2689908059 conn(0x7f04d8076cd0 msgr2=0x7f04d80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.265+0000 7f04df6c3640 1 -- 172.21.15.98:0/2689908059 shutdown_connections 2024-05-09T08:51:50.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.266+0000 7f04df6c3640 1 -- 172.21.15.98:0/2689908059 wait complete. 2024-05-09T08:51:50.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.266+0000 7f04df6c3640 1 Processor -- start 2024-05-09T08:51:50.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.266+0000 7f04df6c3640 1 -- start start 2024-05-09T08:51:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04df6c3640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04d80fffb0 0x7f04d8198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04df6c3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d8106bc0 0x7f04d8198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04df6c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04d819c9e0 0x7f04d819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04df6c3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04d8111c40 con 0x7f04d819c9e0 2024-05-09T08:51:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04df6c3640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f04d8111ac0 con 0x7f04d8106bc0 2024-05-09T08:51:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04df6c3640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f04d8111dc0 con 0x7f04d80fffb0 2024-05-09T08:51:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04ddc60640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04d819c9e0 0x7f04d819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04ddc60640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04d819c9e0 0x7f04d819ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47802/0 (socket says 172.21.15.98:47802) 2024-05-09T08:51:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04ddc60640 1 -- 172.21.15.98:0/3378614898 learned_addr learned my addr 172.21.15.98:0/3378614898 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04dd45f640 1 --2- 172.21.15.98:0/3378614898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04d80fffb0 0x7f04d8198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04dcc5e640 1 --2- 172.21.15.98:0/3378614898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d8106bc0 0x7f04d8198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04ddc60640 1 -- 172.21.15.98:0/3378614898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04d80fffb0 msgr2=0x7f04d8198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04ddc60640 1 --2- 172.21.15.98:0/3378614898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04d80fffb0 0x7f04d8198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04ddc60640 1 -- 172.21.15.98:0/3378614898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d8106bc0 msgr2=0x7f04d8198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04ddc60640 1 --2- 172.21.15.98:0/3378614898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d8106bc0 0x7f04d8198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.267+0000 7f04ddc60640 1 -- 172.21.15.98:0/3378614898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d819d4d0 con 0x7f04d819c9e0 2024-05-09T08:51:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.268+0000 7f04ddc60640 1 --2- 172.21.15.98:0/3378614898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04d819c9e0 0x7f04d819ce90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f04d000cce0 tx=0x7f04d000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.268+0000 7f04ce7fc640 1 -- 172.21.15.98:0/3378614898 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d000f450 con 0x7f04d819c9e0 2024-05-09T08:51:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.268+0000 7f04ce7fc640 1 -- 172.21.15.98:0/3378614898 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d000fad0 con 0x7f04d819c9e0 2024-05-09T08:51:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.268+0000 7f04ce7fc640 1 -- 172.21.15.98:0/3378614898 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0018b60 con 0x7f04d819c9e0 2024-05-09T08:51:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.268+0000 7f04df6c3640 1 -- 172.21.15.98:0/3378614898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d819d7c0 con 0x7f04d819c9e0 2024-05-09T08:51:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.268+0000 7f04df6c3640 1 -- 172.21.15.98:0/3378614898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d81a4f30 con 0x7f04d819c9e0 2024-05-09T08:51:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.271+0000 7f04e23cc740 1 Processor -- start 2024-05-09T08:51:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.272+0000 7f04ce7fc640 1 -- 172.21.15.98:0/3378614898 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f04d000fc80 con 0x7f04d819c9e0 2024-05-09T08:51:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.272+0000 7f04e23cc740 1 -- start start 2024-05-09T08:51:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.272+0000 7f04e23cc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637e330 0x55886637e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.272+0000 7f04e23cc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55886637ec50 0x558866388de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.272+0000 7f04e23cc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558866389320 0x55886638b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.272+0000 7f04e23cc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5588662e3c40 con 0x55886637ec50 2024-05-09T08:51:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.272+0000 7f04e23cc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55886628b580 con 0x55886637e330 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.272+0000 7f04e23cc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5588662dfc00 con 0x558866389320 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.272+0000 7f04dd45f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55886637ec50 0x558866388de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.272+0000 7f04dcc5e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637e330 0x55886637e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.273+0000 7f04dd45f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55886637ec50 0x558866388de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47808/0 (socket says 172.21.15.98:47808) 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.273+0000 7f04dd45f640 1 -- 172.21.15.98:0/1452803460 learned_addr learned my addr 172.21.15.98:0/1452803460 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.273+0000 7f04dcc5e640 1 --2- 172.21.15.98:0/1452803460 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558866389320 0x55886638b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.273+0000 7f04dd45f640 1 -- 172.21.15.98:0/1452803460 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558866389320 msgr2=0x55886638b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.273+0000 7f04dd45f640 1 --2- 172.21.15.98:0/1452803460 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558866389320 0x55886638b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.273+0000 7f04dd45f640 1 -- 172.21.15.98:0/1452803460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637e330 msgr2=0x55886637e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.273+0000 7f04dd45f640 1 --2- 172.21.15.98:0/1452803460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637e330 0x55886637e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.273+0000 7f04dd45f640 1 -- 172.21.15.98:0/1452803460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558865ff73f0 con 0x55886637ec50 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.273+0000 7f04dcc5e640 1 --2- 172.21.15.98:0/1452803460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637e330 0x55886637e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.273+0000 7f04ce7fc640 1 --2- 172.21.15.98:0/3378614898 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f04c40a2f30 0x7f04c40a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.273+0000 7f04dcc5e640 1 --2- 172.21.15.98:0/3378614898 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f04c40a2f30 0x7f04c40a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.273+0000 7f04ce7fc640 1 -- 172.21.15.98:0/3378614898 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f04d00cffb0 con 0x7f04d819c9e0 2024-05-09T08:51:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.273+0000 7f04dd45f640 1 --2- 172.21.15.98:0/1452803460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55886637ec50 0x558866388de0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f04d4009cd0 tx=0x7f04d400e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6505d6dab0a39a87 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.274+0000 7f04c37fe640 1 -- 172.21.15.98:0/1452803460 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d4017070 con 0x55886637ec50 2024-05-09T08:51:50.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.274+0000 7f04c37fe640 1 -- 172.21.15.98:0/1452803460 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d400ecb0 con 0x55886637ec50 2024-05-09T08:51:50.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.274+0000 7f04c37fe640 1 -- 172.21.15.98:0/1452803460 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d40125d0 con 0x55886637ec50 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.274+0000 7f04e23cc740 1 -- 172.21.15.98:0/1452803460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55886637ec50 msgr2=0x558866388de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.274+0000 7f04e23cc740 1 --2- 172.21.15.98:0/1452803460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55886637ec50 0x558866388de0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f04d4009cd0 tx=0x7f04d400e5a0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.274+0000 7f04dcc5e640 1 --2- 172.21.15.98:0/3378614898 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f04c40a2f30 0x7f04c40a53f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f04c8005940 tx=0x7f04c8000bc0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.274+0000 7f04e23cc740 1 -- 172.21.15.98:0/1452803460 shutdown_connections 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.275+0000 7f04e23cc740 1 --2- 172.21.15.98:0/1452803460 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558866389320 0x55886638b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.275+0000 7f04e23cc740 1 --2- 172.21.15.98:0/1452803460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55886637ec50 0x558866388de0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.275+0000 7f04e23cc740 1 --2- 172.21.15.98:0/1452803460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637e330 0x55886637e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.275+0000 7f04e23cc740 1 -- 172.21.15.98:0/1452803460 >> 172.21.15.98:0/1452803460 conn(0x55886637a650 msgr2=0x55886637aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.275+0000 7f04e23cc740 1 -- 172.21.15.98:0/1452803460 shutdown_connections 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.275+0000 7f04e23cc740 1 -- 172.21.15.98:0/1452803460 wait complete. 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.275+0000 7f04e23cc740 1 -- start start 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.275+0000 7f04e23cc740 1 Processor -- start 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.275+0000 7f04e23cc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637e330 0x5588663862b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.275+0000 7f04e23cc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55886637ec50 0x5588663867f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04e23cc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558866389320 0x558866386d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04e23cc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55886638c110 con 0x558866389320 2024-05-09T08:51:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04e23cc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55886629ec40 con 0x55886637e330 2024-05-09T08:51:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04e23cc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55886638c290 con 0x55886637ec50 2024-05-09T08:51:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04e23cc740 10 client.0 _ll_register_callbacks cb 0x558865ff6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04ddc60640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558866389320 0x558866386d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04ddc60640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558866389320 0x558866386d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47814/0 (socket says 172.21.15.98:47814) 2024-05-09T08:51:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04ddc60640 1 -- 172.21.15.98:0/2063875259 learned_addr learned my addr 172.21.15.98:0/2063875259 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04dd45f640 1 --2- 172.21.15.98:0/2063875259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637e330 0x5588663862b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04dcc5e640 1 --2- 172.21.15.98:0/2063875259 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55886637ec50 0x5588663867f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04ddc60640 1 -- 172.21.15.98:0/2063875259 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55886637ec50 msgr2=0x5588663867f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04ddc60640 1 --2- 172.21.15.98:0/2063875259 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55886637ec50 0x5588663867f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04ddc60640 1 -- 172.21.15.98:0/2063875259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637e330 msgr2=0x5588663862b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04ddc60640 1 --2- 172.21.15.98:0/2063875259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637e330 0x5588663862b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04ddc60640 1 -- 172.21.15.98:0/2063875259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588660c17d0 con 0x558866389320 2024-05-09T08:51:50.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.276+0000 7f04dd45f640 1 --2- 172.21.15.98:0/2063875259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637e330 0x5588663862b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:50.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.277+0000 7f04c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:50.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.277+0000 7f04ddc60640 1 --2- 172.21.15.98:0/2063875259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558866389320 0x558866386d30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f04d00988a0 tx=0x7f04d009bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b743a77898efee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.277+0000 7f04c0ff9640 1 -- 172.21.15.98:0/2063875259 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d00b4020 con 0x558866389320 2024-05-09T08:51:50.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.277+0000 7f04c0ff9640 1 -- 172.21.15.98:0/2063875259 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f04d00a48c0 con 0x558866389320 2024-05-09T08:51:50.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.277+0000 7f04e23cc740 1 -- 172.21.15.98:0/2063875259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04d400ecb0 con 0x558866389320 2024-05-09T08:51:50.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.277+0000 7f04c0ff9640 1 -- 172.21.15.98:0/2063875259 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f04d00a4b40 con 0x558866389320 2024-05-09T08:51:50.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.277+0000 7f04e23cc740 10 client.16119 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:50.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.277+0000 7f04e23cc740 1 -- 172.21.15.98:0/2063875259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5588663983c0 con 0x558866389320 2024-05-09T08:51:50.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.278+0000 7f04c0ff9640 1 -- 172.21.15.98:0/2063875259 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d400ecb0 con 0x558866389320 2024-05-09T08:51:50.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.278+0000 7f04c0ff9640 1 -- 172.21.15.98:0/2063875259 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f04d00bab10 con 0x558866389320 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.278+0000 7f04e23cc740 10 client.16119 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.278+0000 7f04e23cc740 10 client.16119 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.278+0000 7f04e23cc740 20 client.16119 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.278+0000 7f04e23cc740 1 --2- 172.21.15.98:0/2063875259 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5588663997d0 0x55886639bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.278+0000 7f04e23cc740 4 client.16119 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.278+0000 7f04e23cc740 1 -- 172.21.15.98:0/2063875259 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588662f1370 con 0x5588663997d0 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.278+0000 7f04dd45f640 1 --2- 172.21.15.98:0/2063875259 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5588663997d0 0x55886639bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.279+0000 7f04c0ff9640 10 client.16119 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.279+0000 7f04dd45f640 1 --2- 172.21.15.98:0/2063875259 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5588663997d0 0x55886639bc70 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=30d4be98fb08189d server_cookie=61b7e351587bd4e4 in_seq=0 out_seq=0 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.279+0000 7f04c0ff9640 1 -- 172.21.15.98:0/2063875259 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588662f1370 con 0x5588663997d0 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.279+0000 7f04c0ff9640 10 client.16119 handle_command_reply: tid=0 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.279+0000 7f04e23cc740 1 client.16119 shutdown 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.279+0000 7f04e23cc740 1 -- 172.21.15.98:0/2063875259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558866389320 msgr2=0x558866386d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.279+0000 7f04e23cc740 1 --2- 172.21.15.98:0/2063875259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558866389320 0x558866386d30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f04d00988a0 tx=0x7f04d009bf90 comp rx=0 tx=0).stop 2024-05-09T08:51:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.280+0000 7f04e23cc740 1 -- 172.21.15.98:0/2063875259 shutdown_connections 2024-05-09T08:51:50.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.280+0000 7f04e23cc740 1 --2- 172.21.15.98:0/2063875259 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5588663997d0 0x55886639bc70 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.280+0000 7f04e23cc740 1 --2- 172.21.15.98:0/2063875259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558866389320 0x558866386d30 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.280+0000 7f04e23cc740 1 --2- 172.21.15.98:0/2063875259 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55886637ec50 0x5588663867f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.280+0000 7f04c0ff9640 0 client.16119 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:51:50.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.280+0000 7f04e23cc740 1 --2- 172.21.15.98:0/2063875259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637e330 0x5588663862b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.280+0000 7f04e23cc740 1 -- 172.21.15.98:0/2063875259 >> 172.21.15.98:0/2063875259 conn(0x55886637a650 msgr2=0x558866380450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.280+0000 7f04e23cc740 1 -- 172.21.15.98:0/2063875259 shutdown_connections 2024-05-09T08:51:50.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.280+0000 7f04e23cc740 1 -- 172.21.15.98:0/2063875259 wait complete. 2024-05-09T08:51:50.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.280+0000 7f04e23cc740 20 client.16119 trim_cache size 0 max 16384 2024-05-09T08:51:50.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.293+0000 7f04e23cc740 1 Processor -- start 2024-05-09T08:51:50.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.293+0000 7f04e23cc740 1 -- start start 2024-05-09T08:51:50.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.293+0000 7f04e23cc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55886637e330 0x558866395020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.293+0000 7f04e23cc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637ec50 0x558866385430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.293+0000 7f04e23cc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558866389320 0x558866385970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.293+0000 7f04e23cc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04d81b0f20 con 0x55886637e330 2024-05-09T08:51:50.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.293+0000 7f04e23cc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f04d81b2a60 con 0x55886637ec50 2024-05-09T08:51:50.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.293+0000 7f04e23cc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f04d81b15f0 con 0x558866389320 2024-05-09T08:51:50.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.293+0000 7f04dd45f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55886637e330 0x558866395020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.293+0000 7f04dd45f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55886637e330 0x558866395020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47816/0 (socket says 172.21.15.98:47816) 2024-05-09T08:51:50.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.293+0000 7f04dd45f640 1 -- 172.21.15.98:0/1531097471 learned_addr learned my addr 172.21.15.98:0/1531097471 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.294+0000 7f04dcc5e640 1 --2- 172.21.15.98:0/1531097471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637ec50 0x558866385430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.294+0000 7f04ddc60640 1 --2- 172.21.15.98:0/1531097471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558866389320 0x558866385970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.294+0000 7f04dd45f640 1 -- 172.21.15.98:0/1531097471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558866389320 msgr2=0x558866385970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.294+0000 7f04dd45f640 1 --2- 172.21.15.98:0/1531097471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558866389320 0x558866385970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.294+0000 7f04dd45f640 1 -- 172.21.15.98:0/1531097471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637ec50 msgr2=0x558866385430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.294+0000 7f04dd45f640 1 --2- 172.21.15.98:0/1531097471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637ec50 0x558866385430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.294+0000 7f04dd45f640 1 -- 172.21.15.98:0/1531097471 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55886637ea80 con 0x55886637e330 2024-05-09T08:51:50.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.294+0000 7f04dd45f640 1 --2- 172.21.15.98:0/1531097471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55886637e330 0x558866395020 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f04d400eaa0 tx=0x7f04d4009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad1336bedcbdda0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.294+0000 7f04c2ffd640 1 -- 172.21.15.98:0/1531097471 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d402a070 con 0x55886637e330 2024-05-09T08:51:50.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.294+0000 7f04c2ffd640 1 -- 172.21.15.98:0/1531097471 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d4026890 con 0x55886637e330 2024-05-09T08:51:50.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.294+0000 7f04c2ffd640 1 -- 172.21.15.98:0/1531097471 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d4017040 con 0x55886637e330 2024-05-09T08:51:50.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.295+0000 7f04e23cc740 1 -- 172.21.15.98:0/1531097471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55886637e330 msgr2=0x558866395020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.295+0000 7f04e23cc740 1 --2- 172.21.15.98:0/1531097471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55886637e330 0x558866395020 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f04d400eaa0 tx=0x7f04d4009110 comp rx=0 tx=0).stop 2024-05-09T08:51:50.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.295+0000 7f04e23cc740 1 -- 172.21.15.98:0/1531097471 shutdown_connections 2024-05-09T08:51:50.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.295+0000 7f04e23cc740 1 --2- 172.21.15.98:0/1531097471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558866389320 0x558866385970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.295+0000 7f04e23cc740 1 --2- 172.21.15.98:0/1531097471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55886637ec50 0x558866385430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.295+0000 7f04e23cc740 1 --2- 172.21.15.98:0/1531097471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55886637e330 0x558866395020 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.295+0000 7f04e23cc740 1 -- 172.21.15.98:0/1531097471 >> 172.21.15.98:0/1531097471 conn(0x55886637a650 msgr2=0x55886637aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.295+0000 7f04e23cc740 1 -- 172.21.15.98:0/1531097471 shutdown_connections 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.295+0000 7f04e23cc740 1 -- 172.21.15.98:0/1531097471 wait complete. 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.295+0000 7f04e23cc740 1 -- start start 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.295+0000 7f04e23cc740 1 Processor -- start 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.296+0000 7f04e23cc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55886637ec50 0x558866391720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.296+0000 7f04e23cc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558866389320 0x558866382b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.296+0000 7f04e23cc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558866383080 0x558866383500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.296+0000 7f04e23cc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04d81b2390 con 0x558866383080 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.296+0000 7f04e23cc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f04d81b1cc0 con 0x558866389320 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.296+0000 7f04e23cc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f04d81aef00 con 0x55886637ec50 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.296+0000 7f04e23cc740 10 client.0 _ll_register_callbacks cb 0x7f04d0098170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.296+0000 7f04ddc60640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558866383080 0x558866383500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.296+0000 7f04ddc60640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558866383080 0x558866383500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47832/0 (socket says 172.21.15.98:47832) 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.296+0000 7f04ddc60640 1 -- 172.21.15.98:0/2381406762 learned_addr learned my addr 172.21.15.98:0/2381406762 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.296+0000 7f04dd45f640 1 --2- 172.21.15.98:0/2381406762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55886637ec50 0x558866391720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.296+0000 7f04dcc5e640 1 --2- 172.21.15.98:0/2381406762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558866389320 0x558866382b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04ddc60640 1 -- 172.21.15.98:0/2381406762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55886637ec50 msgr2=0x558866391720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04ddc60640 1 --2- 172.21.15.98:0/2381406762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55886637ec50 0x558866391720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04ddc60640 1 -- 172.21.15.98:0/2381406762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558866389320 msgr2=0x558866382b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04ddc60640 1 --2- 172.21.15.98:0/2381406762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558866389320 0x558866382b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04ddc60640 1 -- 172.21.15.98:0/2381406762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588663fee00 con 0x558866383080 2024-05-09T08:51:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04dd45f640 1 --2- 172.21.15.98:0/2381406762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55886637ec50 0x558866391720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04ddc60640 1 --2- 172.21.15.98:0/2381406762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558866383080 0x558866383500 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f04d0098330 tx=0x7f04d009bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5638f606925dd06b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04c0ff9640 1 -- 172.21.15.98:0/2381406762 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d00b4070 con 0x558866383080 2024-05-09T08:51:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04c0ff9640 1 -- 172.21.15.98:0/2381406762 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f04d000f5f0 con 0x558866383080 2024-05-09T08:51:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04e23cc740 1 -- 172.21.15.98:0/2381406762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04d4026890 con 0x558866383080 2024-05-09T08:51:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04c0ff9640 1 -- 172.21.15.98:0/2381406762 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f04d009c5d0 con 0x558866383080 2024-05-09T08:51:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04e23cc740 10 client.16125 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04e23cc740 1 -- 172.21.15.98:0/2381406762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558866400010 con 0x558866383080 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04c0ff9640 1 -- 172.21.15.98:0/2381406762 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d0097e30 con 0x558866383080 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.297+0000 7f04c0ff9640 1 -- 172.21.15.98:0/2381406762 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f04d00cd410 con 0x558866383080 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.298+0000 7f04e23cc740 10 client.16125 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.298+0000 7f04e23cc740 10 client.16125 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.298+0000 7f04e23cc740 20 client.16125 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.298+0000 7f04e23cc740 1 --2- 172.21.15.98:0/2381406762 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x558866400730 0x558866402b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.298+0000 7f04e23cc740 4 client.16125 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.298+0000 7f04e23cc740 1 -- 172.21.15.98:0/2381406762 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f04d4026890 con 0x558866400730 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.298+0000 7f04dd45f640 1 --2- 172.21.15.98:0/2381406762 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x558866400730 0x558866402b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.298+0000 7f04c0ff9640 10 client.16125 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.298+0000 7f04dd45f640 1 --2- 172.21.15.98:0/2381406762 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x558866400730 0x558866402b20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2005de464270b661 server_cookie=6d808741ec5f97a5 in_seq=0 out_seq=0 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04c0ff9640 1 -- 172.21.15.98:0/2381406762 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f04d4026890 con 0x558866400730 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04c0ff9640 10 client.16125 handle_command_reply: tid=0 2024-05-09T08:51:50.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04e23cc740 1 client.16125 shutdown 2024-05-09T08:51:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04e23cc740 1 -- 172.21.15.98:0/2381406762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558866383080 msgr2=0x558866383500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04e23cc740 1 --2- 172.21.15.98:0/2381406762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558866383080 0x558866383500 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f04d0098330 tx=0x7f04d009bd20 comp rx=0 tx=0).stop 2024-05-09T08:51:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04e23cc740 1 -- 172.21.15.98:0/2381406762 shutdown_connections 2024-05-09T08:51:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04e23cc740 1 --2- 172.21.15.98:0/2381406762 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x558866400730 0x558866402b20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04e23cc740 1 --2- 172.21.15.98:0/2381406762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558866383080 0x558866383500 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04c0ff9640 0 client.16125 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:51:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04e23cc740 1 --2- 172.21.15.98:0/2381406762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558866389320 0x558866382b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04e23cc740 1 --2- 172.21.15.98:0/2381406762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55886637ec50 0x558866391720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04e23cc740 1 -- 172.21.15.98:0/2381406762 >> 172.21.15.98:0/2381406762 conn(0x55886637a650 msgr2=0x5588663797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04e23cc740 1 -- 172.21.15.98:0/2381406762 shutdown_connections 2024-05-09T08:51:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04e23cc740 1 -- 172.21.15.98:0/2381406762 wait complete. 2024-05-09T08:51:50.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.299+0000 7f04e23cc740 20 client.16125 trim_cache size 0 max 16384 2024-05-09T08:51:50.304 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:51:50.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.300+0000 7f04df6c3640 1 -- 172.21.15.98:0/3378614898 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f04c40a2f30 msgr2=0x7f04c40a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.300+0000 7f04df6c3640 1 --2- 172.21.15.98:0/3378614898 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f04c40a2f30 0x7f04c40a53f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f04c8005940 tx=0x7f04c8000bc0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.300+0000 7f04df6c3640 1 -- 172.21.15.98:0/3378614898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04d819c9e0 msgr2=0x7f04d819ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.300+0000 7f04df6c3640 1 --2- 172.21.15.98:0/3378614898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04d819c9e0 0x7f04d819ce90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f04d000cce0 tx=0x7f04d000a620 comp rx=0 tx=0).stop 2024-05-09T08:51:50.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.300+0000 7f04df6c3640 1 -- 172.21.15.98:0/3378614898 shutdown_connections 2024-05-09T08:51:50.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.300+0000 7f04df6c3640 1 --2- 172.21.15.98:0/3378614898 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f04c40a2f30 0x7f04c40a53f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.300+0000 7f04df6c3640 1 --2- 172.21.15.98:0/3378614898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04d819c9e0 0x7f04d819ce90 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.300+0000 7f04df6c3640 1 --2- 172.21.15.98:0/3378614898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d8106bc0 0x7f04d8198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.300+0000 7f04df6c3640 1 --2- 172.21.15.98:0/3378614898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04d80fffb0 0x7f04d8198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.300+0000 7f04df6c3640 1 -- 172.21.15.98:0/3378614898 >> 172.21.15.98:0/3378614898 conn(0x7f04d8076cd0 msgr2=0x7f04d8104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.300+0000 7f04df6c3640 1 -- 172.21.15.98:0/3378614898 shutdown_connections 2024-05-09T08:51:50.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.300+0000 7f04df6c3640 1 -- 172.21.15.98:0/3378614898 wait complete. 2024-05-09T08:51:50.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:51:50.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.428+0000 7f9c0add5640 1 -- 172.21.15.98:0/417336610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c04102ab0 msgr2=0x7f9c04102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.428+0000 7f9c0add5640 1 --2- 172.21.15.98:0/417336610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c04102ab0 0x7f9c04102e90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009900 tx=0x7f9bf802a510 comp rx=0 tx=0).stop 2024-05-09T08:51:50.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.429+0000 7f9c0add5640 1 -- 172.21.15.98:0/417336610 shutdown_connections 2024-05-09T08:51:50.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.429+0000 7f9c0add5640 1 --2- 172.21.15.98:0/417336610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c0410bdf0 0x7f9c0410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.429+0000 7f9c0add5640 1 --2- 172.21.15.98:0/417336610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c041033d0 0x7f9c0410b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.429+0000 7f9c0add5640 1 --2- 172.21.15.98:0/417336610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c04102ab0 0x7f9c04102e90 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.429+0000 7f9c0add5640 1 -- 172.21.15.98:0/417336610 >> 172.21.15.98:0/417336610 conn(0x7f9c04077830 msgr2=0x7f9c04077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.429+0000 7f9c0add5640 1 -- 172.21.15.98:0/417336610 shutdown_connections 2024-05-09T08:51:50.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.429+0000 7f9c0add5640 1 -- 172.21.15.98:0/417336610 wait complete. 2024-05-09T08:51:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.429+0000 7f9c0add5640 1 Processor -- start 2024-05-09T08:51:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.429+0000 7f9c0add5640 1 -- start start 2024-05-09T08:51:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c0add5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c04102ab0 0x7f9c04198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c0add5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c041033d0 0x7f9c04198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c0add5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c0410bdf0 0x7f9c0419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c0add5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9c04111bf0 con 0x7f9c041033d0 2024-05-09T08:51:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c03fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c041033d0 0x7f9c04198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c03fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c041033d0 0x7f9c04198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47852/0 (socket says 172.21.15.98:47852) 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c03fff640 1 -- 172.21.15.98:0/325606379 learned_addr learned my addr 172.21.15.98:0/325606379 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c0add5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9c04111a70 con 0x7f9c04102ab0 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c0add5640 1 -- 172.21.15.98:0/325606379 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9c04111d70 con 0x7f9c0410bdf0 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c09372640 1 --2- 172.21.15.98:0/325606379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c0410bdf0 0x7f9c0419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c08b71640 1 --2- 172.21.15.98:0/325606379 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c04102ab0 0x7f9c04198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c03fff640 1 -- 172.21.15.98:0/325606379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c0410bdf0 msgr2=0x7f9c0419c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c03fff640 1 --2- 172.21.15.98:0/325606379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c0410bdf0 0x7f9c0419c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c03fff640 1 -- 172.21.15.98:0/325606379 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c04102ab0 msgr2=0x7f9c04198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c03fff640 1 --2- 172.21.15.98:0/325606379 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c04102ab0 0x7f9c04198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.430+0000 7f9c03fff640 1 -- 172.21.15.98:0/325606379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c0419d040 con 0x7f9c041033d0 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.431+0000 7f9c03fff640 1 --2- 172.21.15.98:0/325606379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c041033d0 0x7f9c04198590 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000c920 tx=0x7f9bf000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.431+0000 7f9c01ffb640 1 -- 172.21.15.98:0/325606379 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0007c70 con 0x7f9c041033d0 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.431+0000 7f9c01ffb640 1 -- 172.21.15.98:0/325606379 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bf0007e10 con 0x7f9c041033d0 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.431+0000 7f9c0add5640 1 -- 172.21.15.98:0/325606379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c0419d330 con 0x7f9c041033d0 2024-05-09T08:51:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.431+0000 7f9c01ffb640 1 -- 172.21.15.98:0/325606379 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0018890 con 0x7f9c041033d0 2024-05-09T08:51:50.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.431+0000 7f9c0add5640 1 -- 172.21.15.98:0/325606379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c041a4bc0 con 0x7f9c041033d0 2024-05-09T08:51:50.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.434+0000 7f9c01ffb640 1 -- 172.21.15.98:0/325606379 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f9bf0018a50 con 0x7f9c041033d0 2024-05-09T08:51:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.435+0000 7f9c0dbbf740 1 Processor -- start 2024-05-09T08:51:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.435+0000 7f9c0dbbf740 1 -- start start 2024-05-09T08:51:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.435+0000 7f9c01ffb640 1 --2- 172.21.15.98:0/325606379 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9bf40a2f30 0x7f9bf40a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.436+0000 7f9c09372640 1 --2- 172.21.15.98:0/325606379 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9bf40a2f30 0x7f9bf40a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.436+0000 7f9c01ffb640 1 -- 172.21.15.98:0/325606379 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f9bf00cfb90 con 0x7f9c041033d0 2024-05-09T08:51:50.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.436+0000 7f9c0dbbf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df1f330 0x55c97df1f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.436+0000 7f9c09372640 1 --2- 172.21.15.98:0/325606379 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9bf40a2f30 0x7f9bf40a53f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc005fd0 tx=0x7f9bfc009600 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.436+0000 7f9c08b71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df1f330 0x55c97df1f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.436+0000 7f9c08b71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df1f330 0x55c97df1f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47862/0 (socket says 172.21.15.98:47862) 2024-05-09T08:51:50.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.436+0000 7f9c08b71640 1 -- 172.21.15.98:0/3573515049 learned_addr learned my addr 172.21.15.98:0/3573515049 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.436+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3573515049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df1fc50 0x55c97df29de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.437+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3573515049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df2a320 0x55c97df2c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.437+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3573515049 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c97de84c40 con 0x55c97df1f330 2024-05-09T08:51:50.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.437+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3573515049 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c97de2c580 con 0x55c97df2a320 2024-05-09T08:51:50.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.437+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3573515049 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c97de80c00 con 0x55c97df1fc50 2024-05-09T08:51:50.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.437+0000 7f9c09372640 1 --2- 172.21.15.98:0/3573515049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df2a320 0x55c97df2c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.437+0000 7f9c03fff640 1 --2- 172.21.15.98:0/3573515049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df1fc50 0x55c97df29de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.437+0000 7f9c08b71640 1 -- 172.21.15.98:0/3573515049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df1fc50 msgr2=0x55c97df29de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.437+0000 7f9c08b71640 1 --2- 172.21.15.98:0/3573515049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df1fc50 0x55c97df29de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.437+0000 7f9c08b71640 1 -- 172.21.15.98:0/3573515049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df2a320 msgr2=0x55c97df2c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.437+0000 7f9c08b71640 1 --2- 172.21.15.98:0/3573515049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df2a320 0x55c97df2c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.437+0000 7f9c08b71640 1 -- 172.21.15.98:0/3573515049 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c97db983f0 con 0x55c97df1f330 2024-05-09T08:51:50.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.438+0000 7f9c08b71640 1 --2- 172.21.15.98:0/3573515049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df1f330 0x55c97df1f710 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009900 tx=0x7f9bf8031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8628a46c93e56205 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.438+0000 7f9beeffd640 1 -- 172.21.15.98:0/3573515049 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8035040 con 0x55c97df1f330 2024-05-09T08:51:50.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.438+0000 7f9beeffd640 1 -- 172.21.15.98:0/3573515049 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bf8005020 con 0x55c97df1f330 2024-05-09T08:51:50.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.438+0000 7f9beeffd640 1 -- 172.21.15.98:0/3573515049 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf803fad0 con 0x55c97df1f330 2024-05-09T08:51:50.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.439+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3573515049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df1f330 msgr2=0x55c97df1f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.439+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3573515049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df1f330 0x55c97df1f710 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009900 tx=0x7f9bf8031630 comp rx=0 tx=0).stop 2024-05-09T08:51:50.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.439+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3573515049 shutdown_connections 2024-05-09T08:51:50.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.439+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3573515049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df2a320 0x55c97df2c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.439+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3573515049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df1fc50 0x55c97df29de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.439+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3573515049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df1f330 0x55c97df1f710 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.439+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3573515049 >> 172.21.15.98:0/3573515049 conn(0x55c97df1b650 msgr2=0x55c97df1ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.440+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3573515049 shutdown_connections 2024-05-09T08:51:50.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.440+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3573515049 wait complete. 2024-05-09T08:51:50.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.440+0000 7f9c0dbbf740 1 -- start start 2024-05-09T08:51:50.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.441+0000 7f9c0dbbf740 1 Processor -- start 2024-05-09T08:51:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.441+0000 7f9c0dbbf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df1fc50 0x55c97df273e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.441+0000 7f9c0dbbf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df2a320 0x55c97df27920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.441+0000 7f9c0dbbf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df27e60 0x55c97df36550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.441+0000 7f9c09372640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df27e60 0x55c97df36550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.441+0000 7f9c0dbbf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c97df2d0b0 con 0x55c97df27e60 2024-05-09T08:51:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.441+0000 7f9c03fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df2a320 0x55c97df27920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.441+0000 7f9c09372640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df27e60 0x55c97df36550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47866/0 (socket says 172.21.15.98:47866) 2024-05-09T08:51:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.441+0000 7f9c09372640 1 -- 172.21.15.98:0/3033088147 learned_addr learned my addr 172.21.15.98:0/3033088147 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.441+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3033088147 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c97de3fc40 con 0x55c97df2a320 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.441+0000 7f9c08b71640 1 --2- 172.21.15.98:0/3033088147 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df1fc50 0x55c97df273e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.441+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3033088147 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c97df2d230 con 0x55c97df1fc50 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.442+0000 7f9c0dbbf740 10 client.0 _ll_register_callbacks cb 0x55c97db97be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.442+0000 7f9c09372640 1 -- 172.21.15.98:0/3033088147 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df1fc50 msgr2=0x55c97df273e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.442+0000 7f9c09372640 1 --2- 172.21.15.98:0/3033088147 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df1fc50 0x55c97df273e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.442+0000 7f9c09372640 1 -- 172.21.15.98:0/3033088147 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df2a320 msgr2=0x55c97df27920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.442+0000 7f9c09372640 1 --2- 172.21.15.98:0/3033088147 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df2a320 0x55c97df27920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.442+0000 7f9c09372640 1 -- 172.21.15.98:0/3033088147 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c97dc627d0 con 0x55c97df27e60 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.442+0000 7f9c03fff640 1 --2- 172.21.15.98:0/3033088147 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df2a320 0x55c97df27920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.442+0000 7f9c09372640 1 --2- 172.21.15.98:0/3033088147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df27e60 0x55c97df36550 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc012cc0 tx=0x7f9bfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f880b650b283f774 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.442+0000 7f9be3fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.442+0000 7f9be3fff640 1 -- 172.21.15.98:0/3033088147 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc029670 con 0x55c97df27e60 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.442+0000 7f9be3fff640 1 -- 172.21.15.98:0/3033088147 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f9bfc029810 con 0x55c97df27e60 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.443+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3033088147 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bf8005020 con 0x55c97df27e60 2024-05-09T08:51:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.443+0000 7f9be3fff640 1 -- 172.21.15.98:0/3033088147 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9bfc029a10 con 0x55c97df27e60 2024-05-09T08:51:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.443+0000 7f9c0dbbf740 10 client.16137 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.443+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3033088147 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c97df398e0 con 0x55c97df27e60 2024-05-09T08:51:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.443+0000 7f9be3fff640 1 -- 172.21.15.98:0/3033088147 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bfc024070 con 0x55c97df27e60 2024-05-09T08:51:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.443+0000 7f9be3fff640 1 -- 172.21.15.98:0/3033088147 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9bfc00f040 con 0x55c97df27e60 2024-05-09T08:51:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.443+0000 7f9c0dbbf740 10 client.16137 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.443+0000 7f9c0dbbf740 10 client.16137 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:51:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.443+0000 7f9c0dbbf740 20 client.16137 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.443+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3033088147 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c97df3adf0 0x55c97df3d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.443+0000 7f9c0dbbf740 4 client.16137 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.443+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3033088147 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9bf8005020 con 0x55c97df3adf0 2024-05-09T08:51:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.444+0000 7f9c08b71640 1 --2- 172.21.15.98:0/3033088147 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c97df3adf0 0x55c97df3d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.444+0000 7f9c08b71640 1 --2- 172.21.15.98:0/3033088147 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c97df3adf0 0x55c97df3d250 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e3c7eaa824e18858 server_cookie=de5561f775d36c97 in_seq=0 out_seq=0 2024-05-09T08:51:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.444+0000 7f9be3fff640 10 client.16137 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:51:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.445+0000 7f9be3fff640 1 -- 172.21.15.98:0/3033088147 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f9bf8005020 con 0x55c97df3adf0 2024-05-09T08:51:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.445+0000 7f9be3fff640 10 client.16137 handle_command_reply: tid=0 2024-05-09T08:51:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.445+0000 7f9c0dbbf740 1 client.16137 shutdown 2024-05-09T08:51:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.445+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3033088147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df27e60 msgr2=0x55c97df36550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.445+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3033088147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df27e60 0x55c97df36550 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc012cc0 tx=0x7f9bfc004a40 comp rx=0 tx=0).stop 2024-05-09T08:51:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.445+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3033088147 shutdown_connections 2024-05-09T08:51:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.446+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3033088147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df27e60 0x55c97df36550 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.446+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3033088147 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c97df3adf0 0x55c97df3d250 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.446+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3033088147 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df2a320 0x55c97df27920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.446+0000 7f9be3fff640 0 client.16137 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:51:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.446+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3033088147 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df1fc50 0x55c97df273e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.446+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3033088147 >> 172.21.15.98:0/3033088147 conn(0x55c97df1b650 msgr2=0x55c97df21350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.446+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3033088147 shutdown_connections 2024-05-09T08:51:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.446+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3033088147 wait complete. 2024-05-09T08:51:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.446+0000 7f9c0dbbf740 20 client.16137 trim_cache size 0 max 16384 2024-05-09T08:51:50.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.472+0000 7f9c0dbbf740 1 Processor -- start 2024-05-09T08:51:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.472+0000 7f9c0dbbf740 1 -- start start 2024-05-09T08:51:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.472+0000 7f9c0dbbf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df1fc50 0x55c97df21a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.472+0000 7f9c0dbbf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df27e60 0x55c97df42b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.472+0000 7f9c0dbbf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df2a320 0x55c97df43060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.472+0000 7f9c0dbbf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9c041acb20 con 0x55c97df2a320 2024-05-09T08:51:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.472+0000 7f9c0dbbf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9c041ae660 con 0x55c97df1fc50 2024-05-09T08:51:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.472+0000 7f9c0dbbf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9c041ad1f0 con 0x55c97df27e60 2024-05-09T08:51:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.472+0000 7f9c09372640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df2a320 0x55c97df43060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.472+0000 7f9c09372640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df2a320 0x55c97df43060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47868/0 (socket says 172.21.15.98:47868) 2024-05-09T08:51:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.472+0000 7f9c09372640 1 -- 172.21.15.98:0/2524183970 learned_addr learned my addr 172.21.15.98:0/2524183970 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.472+0000 7f9c03fff640 1 --2- 172.21.15.98:0/2524183970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df27e60 0x55c97df42b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.473+0000 7f9c09372640 1 -- 172.21.15.98:0/2524183970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df27e60 msgr2=0x55c97df42b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.473+0000 7f9c09372640 1 --2- 172.21.15.98:0/2524183970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df27e60 0x55c97df42b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.473+0000 7f9c09372640 1 -- 172.21.15.98:0/2524183970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df1fc50 msgr2=0x55c97df21a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.472+0000 7f9c08b71640 1 --2- 172.21.15.98:0/2524183970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df1fc50 0x55c97df21a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.473+0000 7f9c09372640 1 --2- 172.21.15.98:0/2524183970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df1fc50 0x55c97df21a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.473+0000 7f9c09372640 1 -- 172.21.15.98:0/2524183970 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c97df25930 con 0x55c97df2a320 2024-05-09T08:51:50.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.473+0000 7f9c03fff640 1 --2- 172.21.15.98:0/2524183970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df27e60 0x55c97df42b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:50.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.473+0000 7f9c08b71640 1 --2- 172.21.15.98:0/2524183970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df1fc50 0x55c97df21a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:50.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.473+0000 7f9c09372640 1 --2- 172.21.15.98:0/2524183970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df2a320 0x55c97df43060 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc010bd0 tx=0x7f9bfc011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7244933c1f2c00cc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.473+0000 7f9bee7fc640 1 -- 172.21.15.98:0/2524183970 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc03a020 con 0x55c97df2a320 2024-05-09T08:51:50.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.473+0000 7f9bee7fc640 1 -- 172.21.15.98:0/2524183970 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bfc024070 con 0x55c97df2a320 2024-05-09T08:51:50.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.473+0000 7f9bee7fc640 1 -- 172.21.15.98:0/2524183970 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc01fc70 con 0x55c97df2a320 2024-05-09T08:51:50.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.474+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/2524183970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df2a320 msgr2=0x55c97df43060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.474+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/2524183970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df2a320 0x55c97df43060 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc010bd0 tx=0x7f9bfc011eb0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.474+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/2524183970 shutdown_connections 2024-05-09T08:51:50.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.474+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/2524183970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df2a320 0x55c97df43060 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.474+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/2524183970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df27e60 0x55c97df42b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.474+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/2524183970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df1fc50 0x55c97df21a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.474+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/2524183970 >> 172.21.15.98:0/2524183970 conn(0x55c97df1b650 msgr2=0x55c97df25cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.474+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/2524183970 shutdown_connections 2024-05-09T08:51:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.474+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/2524183970 wait complete. 2024-05-09T08:51:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.475+0000 7f9c0dbbf740 1 -- start start 2024-05-09T08:51:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.475+0000 7f9c0dbbf740 1 Processor -- start 2024-05-09T08:51:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c0dbbf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df1fc50 0x55c97df328a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c0dbbf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df27e60 0x55c97df2dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c0dbbf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df2a320 0x55c97df2e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c0dbbf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9c041adf90 con 0x55c97df2a320 2024-05-09T08:51:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c0dbbf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9c041ad8c0 con 0x55c97df1fc50 2024-05-09T08:51:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c0dbbf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9c041aab00 con 0x55c97df27e60 2024-05-09T08:51:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c09372640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df2a320 0x55c97df2e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c09372640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df2a320 0x55c97df2e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47882/0 (socket says 172.21.15.98:47882) 2024-05-09T08:51:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c09372640 1 -- 172.21.15.98:0/3001658486 learned_addr learned my addr 172.21.15.98:0/3001658486 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c03fff640 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df27e60 0x55c97df2dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c08b71640 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df1fc50 0x55c97df328a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c0dbbf740 10 client.0 _ll_register_callbacks cb 0x7f9bfc012940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c09372640 1 -- 172.21.15.98:0/3001658486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df27e60 msgr2=0x55c97df2dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c09372640 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df27e60 0x55c97df2dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c09372640 1 -- 172.21.15.98:0/3001658486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df1fc50 msgr2=0x55c97df328a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c09372640 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df1fc50 0x55c97df328a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.476+0000 7f9c09372640 1 -- 172.21.15.98:0/3001658486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c97df2e950 con 0x55c97df2a320 2024-05-09T08:51:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.477+0000 7f9c03fff640 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df27e60 0x55c97df2dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.477+0000 7f9c08b71640 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df1fc50 0x55c97df328a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.477+0000 7f9c09372640 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df2a320 0x55c97df2e150 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc010c70 tx=0x7f9bfc029720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d72608d6a79a771 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.477+0000 7f9be3fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.477+0000 7f9be3fff640 1 -- 172.21.15.98:0/3001658486 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc03a030 con 0x55c97df2a320 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.477+0000 7f9be3fff640 1 -- 172.21.15.98:0/3001658486 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f9bfc01f8f0 con 0x55c97df2a320 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.477+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3001658486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bfc024070 con 0x55c97df2a320 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.477+0000 7f9be3fff640 1 -- 172.21.15.98:0/3001658486 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9bfc01f640 con 0x55c97df2a320 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.477+0000 7f9c0dbbf740 10 client.16143 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.478+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3001658486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c97dfa1310 con 0x55c97df2a320 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.478+0000 7f9be3fff640 1 -- 172.21.15.98:0/3001658486 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bfc024070 con 0x55c97df2a320 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.478+0000 7f9be3fff640 1 -- 172.21.15.98:0/3001658486 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9bfc00f040 con 0x55c97df2a320 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.478+0000 7f9c0dbbf740 10 client.16143 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.478+0000 7f9c0dbbf740 10 client.16143 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.478+0000 7f9c0dbbf740 20 client.16143 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.478+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c97df3adf0 0x55c97dfa39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.478+0000 7f9c0dbbf740 4 client.16143 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.478+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3001658486 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c97de92370 con 0x55c97df3adf0 2024-05-09T08:51:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.479+0000 7f9c08b71640 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c97df3adf0 0x55c97dfa39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.479+0000 7f9be3fff640 10 client.16143 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.479+0000 7f9c08b71640 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c97df3adf0 0x55c97dfa39b0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a445ca7832caa461 server_cookie=ba2f79319cc06aa0 in_seq=0 out_seq=0 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.479+0000 7f9be3fff640 1 -- 172.21.15.98:0/3001658486 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c97de92370 con 0x55c97df3adf0 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.479+0000 7f9be3fff640 10 client.16143 handle_command_reply: tid=0 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.479+0000 7f9c0dbbf740 1 client.16143 shutdown 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.480+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3001658486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df2a320 msgr2=0x55c97df2e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.480+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df2a320 0x55c97df2e150 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc010c70 tx=0x7f9bfc029720 comp rx=0 tx=0).stop 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.480+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3001658486 shutdown_connections 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.480+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c97df2a320 0x55c97df2e150 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.480+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c97df27e60 0x55c97df2dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.480+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c97df3adf0 0x55c97dfa39b0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.480+0000 7f9c0dbbf740 1 --2- 172.21.15.98:0/3001658486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c97df1fc50 0x55c97df328a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.480+0000 7f9be3fff640 0 client.16143 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.480+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3001658486 >> 172.21.15.98:0/3001658486 conn(0x55c97df1b650 msgr2=0x55c97df19f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.480+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3001658486 shutdown_connections 2024-05-09T08:51:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.480+0000 7f9c0dbbf740 1 -- 172.21.15.98:0/3001658486 wait complete. 2024-05-09T08:51:50.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.480+0000 7f9c0dbbf740 20 client.16143 trim_cache size 0 max 16384 2024-05-09T08:51:50.485 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:51:50.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.481+0000 7f9c0add5640 1 -- 172.21.15.98:0/325606379 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9bf40a2f30 msgr2=0x7f9bf40a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.481+0000 7f9c0add5640 1 --2- 172.21.15.98:0/325606379 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9bf40a2f30 0x7f9bf40a53f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc005fd0 tx=0x7f9bfc009600 comp rx=0 tx=0).stop 2024-05-09T08:51:50.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.481+0000 7f9c0add5640 1 -- 172.21.15.98:0/325606379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c041033d0 msgr2=0x7f9c04198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.481+0000 7f9c0add5640 1 --2- 172.21.15.98:0/325606379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c041033d0 0x7f9c04198590 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000c920 tx=0x7f9bf000cde0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.481+0000 7f9c0add5640 1 -- 172.21.15.98:0/325606379 shutdown_connections 2024-05-09T08:51:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.481+0000 7f9c0add5640 1 --2- 172.21.15.98:0/325606379 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9bf40a2f30 0x7f9bf40a53f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.481+0000 7f9c0add5640 1 --2- 172.21.15.98:0/325606379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c0410bdf0 0x7f9c0419c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.481+0000 7f9c0add5640 1 --2- 172.21.15.98:0/325606379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c041033d0 0x7f9c04198590 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.481+0000 7f9c0add5640 1 --2- 172.21.15.98:0/325606379 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c04102ab0 0x7f9c04198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.481+0000 7f9c0add5640 1 -- 172.21.15.98:0/325606379 >> 172.21.15.98:0/325606379 conn(0x7f9c04077830 msgr2=0x7f9c040fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.481+0000 7f9c0add5640 1 -- 172.21.15.98:0/325606379 shutdown_connections 2024-05-09T08:51:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.481+0000 7f9c0add5640 1 -- 172.21.15.98:0/325606379 wait complete. 2024-05-09T08:51:50.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:51:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.642+0000 7f11f2bdc640 1 -- 172.21.15.98:0/2836061796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec100f50 msgr2=0x7f11ec10dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.642+0000 7f11f2bdc640 1 --2- 172.21.15.98:0/2836061796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec100f50 0x7f11ec10dca0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f11d8009900 tx=0x7f11d802a410 comp rx=0 tx=0).stop 2024-05-09T08:51:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.642+0000 7f11f2bdc640 1 -- 172.21.15.98:0/2836061796 shutdown_connections 2024-05-09T08:51:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.642+0000 7f11f2bdc640 1 --2- 172.21.15.98:0/2836061796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11ec10e1e0 0x7f11ec1105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.642+0000 7f11f2bdc640 1 --2- 172.21.15.98:0/2836061796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec100f50 0x7f11ec10dca0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.642+0000 7f11f2bdc640 1 --2- 172.21.15.98:0/2836061796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11ec100630 0x7f11ec100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.642+0000 7f11f2bdc640 1 -- 172.21.15.98:0/2836061796 >> 172.21.15.98:0/2836061796 conn(0x7f11ec077830 msgr2=0x7f11ec077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.642+0000 7f11f2bdc640 1 -- 172.21.15.98:0/2836061796 shutdown_connections 2024-05-09T08:51:50.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.642+0000 7f11f2bdc640 1 -- 172.21.15.98:0/2836061796 wait complete. 2024-05-09T08:51:50.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.643+0000 7f11f2bdc640 1 Processor -- start 2024-05-09T08:51:50.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.643+0000 7f11f2bdc640 1 -- start start 2024-05-09T08:51:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.643+0000 7f11f2bdc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11ec100630 0x7f11ec19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.643+0000 7f11f2bdc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11ec100f50 0x7f11ec19a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.643+0000 7f11f2bdc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec10e1e0 0x7f11ec19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.643+0000 7f11f2bdc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec113e10 con 0x7f11ec10e1e0 2024-05-09T08:51:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.643+0000 7f11f2bdc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec113c90 con 0x7f11ec100f50 2024-05-09T08:51:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.643+0000 7f11f2bdc640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec113f90 con 0x7f11ec100630 2024-05-09T08:51:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.643+0000 7f11f1179640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec10e1e0 0x7f11ec19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.643+0000 7f11ebfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11ec100f50 0x7f11ec19a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.643+0000 7f11ebfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11ec100f50 0x7f11ec19a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:56232/0 (socket says 172.21.15.98:56232) 2024-05-09T08:51:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.643+0000 7f11ebfff640 1 -- 172.21.15.98:0/697235169 learned_addr learned my addr 172.21.15.98:0/697235169 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.644+0000 7f11f0978640 1 --2- 172.21.15.98:0/697235169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11ec100630 0x7f11ec19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.644+0000 7f11f1179640 1 -- 172.21.15.98:0/697235169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11ec100630 msgr2=0x7f11ec19a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.644+0000 7f11f1179640 1 --2- 172.21.15.98:0/697235169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11ec100630 0x7f11ec19a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.644+0000 7f11f1179640 1 -- 172.21.15.98:0/697235169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11ec100f50 msgr2=0x7f11ec19a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.644+0000 7f11f1179640 1 --2- 172.21.15.98:0/697235169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11ec100f50 0x7f11ec19a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.644+0000 7f11f1179640 1 -- 172.21.15.98:0/697235169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec19f300 con 0x7f11ec10e1e0 2024-05-09T08:51:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.644+0000 7f11ebfff640 1 --2- 172.21.15.98:0/697235169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11ec100f50 0x7f11ec19a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.644+0000 7f11f1179640 1 --2- 172.21.15.98:0/697235169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec10e1e0 0x7f11ec19ebd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f11e4005fd0 tx=0x7f11e400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.644+0000 7f11e9ffb640 1 -- 172.21.15.98:0/697235169 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4004b20 con 0x7f11ec10e1e0 2024-05-09T08:51:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.644+0000 7f11e9ffb640 1 -- 172.21.15.98:0/697235169 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e4004cc0 con 0x7f11ec10e1e0 2024-05-09T08:51:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.644+0000 7f11f2bdc640 1 -- 172.21.15.98:0/697235169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11ec19f5f0 con 0x7f11ec10e1e0 2024-05-09T08:51:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.645+0000 7f11e9ffb640 1 -- 172.21.15.98:0/697235169 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e40187a0 con 0x7f11ec10e1e0 2024-05-09T08:51:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.645+0000 7f11f2bdc640 1 -- 172.21.15.98:0/697235169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11ec1a6e80 con 0x7f11ec10e1e0 2024-05-09T08:51:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.647+0000 7f11e9ffb640 1 -- 172.21.15.98:0/697235169 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f11e4018940 con 0x7f11ec10e1e0 2024-05-09T08:51:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f59c8740 1 Processor -- start 2024-05-09T08:51:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11e9ffb640 1 --2- 172.21.15.98:0/697235169 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11dc0a2f30 0x7f11dc0a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f59c8740 1 -- start start 2024-05-09T08:51:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11e9ffb640 1 -- 172.21.15.98:0/697235169 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f11e40d0070 con 0x7f11ec10e1e0 2024-05-09T08:51:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11ebfff640 1 --2- 172.21.15.98:0/697235169 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11dc0a2f30 0x7f11dc0a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f59c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea38710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f59c8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea38c50 0x55caeea42de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f59c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea43320 0x55caeea45710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f59c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55caee99dc40 con 0x55caeea38330 2024-05-09T08:51:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f59c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55caee945580 con 0x55caeea43320 2024-05-09T08:51:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f59c8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55caee999c00 con 0x55caeea38c50 2024-05-09T08:51:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f0978640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea38710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f0978640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea38710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47918/0 (socket says 172.21.15.98:47918) 2024-05-09T08:51:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f0978640 1 -- 172.21.15.98:0/4202792433 learned_addr learned my addr 172.21.15.98:0/4202792433 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f1179640 1 --2- 172.21.15.98:0/4202792433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea43320 0x55caeea45710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f0978640 1 -- 172.21.15.98:0/4202792433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea38c50 msgr2=0x55caeea42de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:51:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f0978640 1 --2- 172.21.15.98:0/4202792433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea38c50 0x55caeea42de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f0978640 1 -- 172.21.15.98:0/4202792433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea43320 msgr2=0x55caeea45710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f0978640 1 --2- 172.21.15.98:0/4202792433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea43320 0x55caeea45710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.648+0000 7f11f0978640 1 -- 172.21.15.98:0/4202792433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caee6b13f0 con 0x55caeea38330 2024-05-09T08:51:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.649+0000 7f11ebfff640 1 --2- 172.21.15.98:0/697235169 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11dc0a2f30 0x7f11dc0a53f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f11d8006010 tx=0x7f11d8005a20 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.649+0000 7f11f0978640 1 --2- 172.21.15.98:0/4202792433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea38710 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009cf0 tx=0x7f11e0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4859ae3ace48a55 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.649+0000 7f11d6ffd640 1 -- 172.21.15.98:0/4202792433 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0004230 con 0x55caeea38330 2024-05-09T08:51:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.649+0000 7f11d6ffd640 1 -- 172.21.15.98:0/4202792433 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e00043d0 con 0x55caeea38330 2024-05-09T08:51:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.649+0000 7f11d6ffd640 1 -- 172.21.15.98:0/4202792433 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0019960 con 0x55caeea38330 2024-05-09T08:51:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.650+0000 7f11f59c8740 1 -- 172.21.15.98:0/4202792433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 msgr2=0x55caeea38710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.650+0000 7f11f59c8740 1 --2- 172.21.15.98:0/4202792433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea38710 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009cf0 tx=0x7f11e0011810 comp rx=0 tx=0).stop 2024-05-09T08:51:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.651+0000 7f11f59c8740 1 -- 172.21.15.98:0/4202792433 shutdown_connections 2024-05-09T08:51:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.651+0000 7f11f59c8740 1 --2- 172.21.15.98:0/4202792433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea43320 0x55caeea45710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.651+0000 7f11f59c8740 1 --2- 172.21.15.98:0/4202792433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea38c50 0x55caeea42de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.651+0000 7f11f59c8740 1 --2- 172.21.15.98:0/4202792433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea38710 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.651+0000 7f11f59c8740 1 -- 172.21.15.98:0/4202792433 >> 172.21.15.98:0/4202792433 conn(0x55caeea34650 msgr2=0x55caeea34a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.651+0000 7f11f59c8740 1 -- 172.21.15.98:0/4202792433 shutdown_connections 2024-05-09T08:51:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.651+0000 7f11f59c8740 1 -- 172.21.15.98:0/4202792433 wait complete. 2024-05-09T08:51:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.651+0000 7f11f59c8740 1 -- start start 2024-05-09T08:51:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.651+0000 7f11f59c8740 1 Processor -- start 2024-05-09T08:51:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f59c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea402e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f59c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38c50 0x55caeea40820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f59c8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea43320 0x55caeea40d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f59c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55caeea46110 con 0x55caeea38330 2024-05-09T08:51:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f59c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55caee958c40 con 0x55caeea38c50 2024-05-09T08:51:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f59c8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55caeea46290 con 0x55caeea43320 2024-05-09T08:51:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f59c8740 10 client.0 _ll_register_callbacks cb 0x55caee6b0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f0978640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f0978640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea402e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47922/0 (socket says 172.21.15.98:47922) 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f0978640 1 -- 172.21.15.98:0/834402048 learned_addr learned my addr 172.21.15.98:0/834402048 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f1179640 1 --2- 172.21.15.98:0/834402048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea43320 0x55caeea40d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11ebfff640 1 --2- 172.21.15.98:0/834402048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38c50 0x55caeea40820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f0978640 1 -- 172.21.15.98:0/834402048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea43320 msgr2=0x55caeea40d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f0978640 1 --2- 172.21.15.98:0/834402048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea43320 0x55caeea40d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f0978640 1 -- 172.21.15.98:0/834402048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38c50 msgr2=0x55caeea40820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f0978640 1 --2- 172.21.15.98:0/834402048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38c50 0x55caeea40820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f0978640 1 -- 172.21.15.98:0/834402048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55caee77b7d0 con 0x55caeea38330 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11f0978640 1 --2- 172.21.15.98:0/834402048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea402e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009560 tx=0x7f11e0005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c5c2acbbcae333 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.652+0000 7f11cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.653+0000 7f11cbfff640 1 -- 172.21.15.98:0/834402048 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0027070 con 0x55caeea38330 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.653+0000 7f11cbfff640 1 -- 172.21.15.98:0/834402048 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f11e00037f0 con 0x55caeea38330 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.653+0000 7f11f59c8740 1 -- 172.21.15.98:0/834402048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11e00043d0 con 0x55caeea38330 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.653+0000 7f11cbfff640 1 -- 172.21.15.98:0/834402048 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f11e0009ea0 con 0x55caeea38330 2024-05-09T08:51:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.653+0000 7f11f59c8740 10 client.16155 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.653+0000 7f11f59c8740 1 -- 172.21.15.98:0/834402048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55caeea52350 con 0x55caeea38330 2024-05-09T08:51:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.653+0000 7f11cbfff640 1 -- 172.21.15.98:0/834402048 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e00043d0 con 0x55caeea38330 2024-05-09T08:51:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.653+0000 7f11cbfff640 1 -- 172.21.15.98:0/834402048 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f11e0004ba0 con 0x55caeea38330 2024-05-09T08:51:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.653+0000 7f11f59c8740 10 client.16155 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.653+0000 7f11f59c8740 10 client.16155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.653+0000 7f11f59c8740 20 client.16155 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.653+0000 7f11f59c8740 1 --2- 172.21.15.98:0/834402048 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55caeea537b0 0x55caeea55ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.653+0000 7f11f59c8740 4 client.16155 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.653+0000 7f11f59c8740 1 -- 172.21.15.98:0/834402048 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55caee9ab370 con 0x55caeea537b0 2024-05-09T08:51:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.654+0000 7f11ebfff640 1 --2- 172.21.15.98:0/834402048 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55caeea537b0 0x55caeea55ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.654+0000 7f11cbfff640 10 client.16155 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.654+0000 7f11ebfff640 1 --2- 172.21.15.98:0/834402048 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55caeea537b0 0x55caeea55ba0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4987bd91b779458 server_cookie=3772efd8d7f61be7 in_seq=0 out_seq=0 2024-05-09T08:51:50.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11cbfff640 1 -- 172.21.15.98:0/834402048 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55caee9ab370 con 0x55caeea537b0 2024-05-09T08:51:50.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11cbfff640 10 client.16155 handle_command_reply: tid=0 2024-05-09T08:51:50.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11f59c8740 1 client.16155 shutdown 2024-05-09T08:51:50.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11f59c8740 1 -- 172.21.15.98:0/834402048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 msgr2=0x55caeea402e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11f59c8740 1 --2- 172.21.15.98:0/834402048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea402e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009560 tx=0x7f11e0005270 comp rx=0 tx=0).stop 2024-05-09T08:51:50.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11f59c8740 1 -- 172.21.15.98:0/834402048 shutdown_connections 2024-05-09T08:51:50.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11f59c8740 1 --2- 172.21.15.98:0/834402048 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55caeea537b0 0x55caeea55ba0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11cbfff640 0 client.16155 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:50.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11f59c8740 1 --2- 172.21.15.98:0/834402048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea43320 0x55caeea40d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11f59c8740 1 --2- 172.21.15.98:0/834402048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38c50 0x55caeea40820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11f59c8740 1 --2- 172.21.15.98:0/834402048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea402e0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11f59c8740 1 -- 172.21.15.98:0/834402048 >> 172.21.15.98:0/834402048 conn(0x55caeea34650 msgr2=0x55caeea3a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11f59c8740 1 -- 172.21.15.98:0/834402048 shutdown_connections 2024-05-09T08:51:50.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11f59c8740 1 -- 172.21.15.98:0/834402048 wait complete. 2024-05-09T08:51:50.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.655+0000 7f11f59c8740 20 client.16155 trim_cache size 0 max 16384 2024-05-09T08:51:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.676+0000 7f11f59c8740 1 Processor -- start 2024-05-09T08:51:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.676+0000 7f11f59c8740 1 -- start start 2024-05-09T08:51:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.676+0000 7f11f59c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea4efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f59c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38c50 0x55caeea5b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f59c8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea43320 0x55caeea5b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f59c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b10a0 con 0x55caeea38330 2024-05-09T08:51:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f59c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b2be0 con 0x55caeea38c50 2024-05-09T08:51:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f59c8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b1770 con 0x55caeea43320 2024-05-09T08:51:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f0978640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea4efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f0978640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea4efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47932/0 (socket says 172.21.15.98:47932) 2024-05-09T08:51:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f0978640 1 -- 172.21.15.98:0/1735220346 learned_addr learned my addr 172.21.15.98:0/1735220346 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11ebfff640 1 --2- 172.21.15.98:0/1735220346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38c50 0x55caeea5b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f1179640 1 --2- 172.21.15.98:0/1735220346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea43320 0x55caeea5b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f0978640 1 -- 172.21.15.98:0/1735220346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea43320 msgr2=0x55caeea5b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f0978640 1 --2- 172.21.15.98:0/1735220346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea43320 0x55caeea5b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f0978640 1 -- 172.21.15.98:0/1735220346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38c50 msgr2=0x55caeea5b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f0978640 1 --2- 172.21.15.98:0/1735220346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38c50 0x55caeea5b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f0978640 1 -- 172.21.15.98:0/1735220346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caeea39050 con 0x55caeea38330 2024-05-09T08:51:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f1179640 1 --2- 172.21.15.98:0/1735220346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea43320 0x55caeea5b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11ebfff640 1 --2- 172.21.15.98:0/1735220346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38c50 0x55caeea5b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11f0978640 1 --2- 172.21.15.98:0/1735220346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea4efd0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f11e0011d90 tx=0x7f11e0009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05c8f1cd7fff77a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.677+0000 7f11d67fc640 1 -- 172.21.15.98:0/1735220346 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0030070 con 0x55caeea38330 2024-05-09T08:51:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.678+0000 7f11d67fc640 1 -- 172.21.15.98:0/1735220346 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e0009910 con 0x55caeea38330 2024-05-09T08:51:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.678+0000 7f11d67fc640 1 -- 172.21.15.98:0/1735220346 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0027040 con 0x55caeea38330 2024-05-09T08:51:50.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.678+0000 7f11f59c8740 1 -- 172.21.15.98:0/1735220346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 msgr2=0x55caeea4efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.678+0000 7f11f59c8740 1 --2- 172.21.15.98:0/1735220346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea4efd0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f11e0011d90 tx=0x7f11e0009bf0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.678+0000 7f11f59c8740 1 -- 172.21.15.98:0/1735220346 shutdown_connections 2024-05-09T08:51:50.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.678+0000 7f11f59c8740 1 --2- 172.21.15.98:0/1735220346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea43320 0x55caeea5b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.678+0000 7f11f59c8740 1 --2- 172.21.15.98:0/1735220346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38c50 0x55caeea5b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.678+0000 7f11f59c8740 1 --2- 172.21.15.98:0/1735220346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea38330 0x55caeea4efd0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.678+0000 7f11f59c8740 1 -- 172.21.15.98:0/1735220346 >> 172.21.15.98:0/1735220346 conn(0x55caeea34650 msgr2=0x55caeea34a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.679+0000 7f11f59c8740 1 -- 172.21.15.98:0/1735220346 shutdown_connections 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.679+0000 7f11f59c8740 1 -- 172.21.15.98:0/1735220346 wait complete. 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.679+0000 7f11f59c8740 1 -- start start 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.679+0000 7f11f59c8740 1 Processor -- start 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f59c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38330 0x55caeea46690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f59c8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea38c50 0x55caeea46bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f59c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea43320 0x55caeeab7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f59c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b2510 con 0x55caeea43320 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f59c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b1e40 con 0x55caeea38330 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f1179640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea43320 0x55caeeab7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f1179640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea43320 0x55caeeab7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47944/0 (socket says 172.21.15.98:47944) 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f1179640 1 -- 172.21.15.98:0/134548042 learned_addr learned my addr 172.21.15.98:0/134548042 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f0978640 1 --2- 172.21.15.98:0/134548042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38330 0x55caeea46690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11ebfff640 1 --2- 172.21.15.98:0/134548042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea38c50 0x55caeea46bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f59c8740 1 -- 172.21.15.98:0/134548042 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1af080 con 0x55caeea38c50 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f1179640 1 -- 172.21.15.98:0/134548042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea38c50 msgr2=0x55caeea46bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f1179640 1 --2- 172.21.15.98:0/134548042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea38c50 0x55caeea46bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f1179640 1 -- 172.21.15.98:0/134548042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38330 msgr2=0x55caeea46690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f1179640 1 --2- 172.21.15.98:0/134548042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38330 0x55caeea46690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.680+0000 7f11f1179640 1 -- 172.21.15.98:0/134548042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55caeeab8190 con 0x55caeea43320 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.681+0000 7f11f59c8740 10 client.0 _ll_register_callbacks cb 0x7f11e400ef70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.681+0000 7f11f59c8740 1 -- 172.21.15.98:0/134548042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11e0009910 con 0x55caeea43320 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.681+0000 7f11f1179640 1 --2- 172.21.15.98:0/134548042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea43320 0x55caeeab7c50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f11e409bc90 tx=0x7f11e400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8870d2712fb6b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.681+0000 7f11cbfff640 10 client.16161 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.681+0000 7f11cbfff640 1 -- 172.21.15.98:0/134548042 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e409bf90 con 0x55caeea43320 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.681+0000 7f11cbfff640 1 -- 172.21.15.98:0/134548042 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f11e40a4770 con 0x55caeea43320 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.681+0000 7f11cbfff640 1 -- 172.21.15.98:0/134548042 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f11e40a4970 con 0x55caeea43320 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.681+0000 7f11f59c8740 10 client.16161 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.681+0000 7f11f59c8740 1 -- 172.21.15.98:0/134548042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55caeeab9330 con 0x55caeea43320 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.682+0000 7f11cbfff640 1 -- 172.21.15.98:0/134548042 <== mon.0 v2:172.21.15.98:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f11e4020030 con 0x55caeea43320 2024-05-09T08:51:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.682+0000 7f11f59c8740 10 client.16161 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.682+0000 7f11f59c8740 10 client.16161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.682+0000 7f11f59c8740 20 client.16161 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.682+0000 7f11f59c8740 1 --2- 172.21.15.98:0/134548042 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55caeeab99a0 0x55caeeabbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.682+0000 7f11f59c8740 4 client.16161 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:51:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.682+0000 7f11f59c8740 1 -- 172.21.15.98:0/134548042 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55caee9a56c0 con 0x55caeeab99a0 2024-05-09T08:51:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.682+0000 7f11f0978640 1 --2- 172.21.15.98:0/134548042 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55caeeab99a0 0x55caeeabbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11f0978640 1 --2- 172.21.15.98:0/134548042 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55caeeab99a0 0x55caeeabbe00 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37959cde2085ae7c server_cookie=984f8cdc3c15f504 in_seq=0 out_seq=0 2024-05-09T08:51:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11cbfff640 10 client.16161 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11cbfff640 1 -- 172.21.15.98:0/134548042 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55caee9a56c0 con 0x55caeeab99a0 2024-05-09T08:51:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11cbfff640 10 client.16161 handle_command_reply: tid=0 2024-05-09T08:51:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11f59c8740 1 client.16161 shutdown 2024-05-09T08:51:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11f59c8740 1 -- 172.21.15.98:0/134548042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea43320 msgr2=0x55caeeab7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11f59c8740 1 --2- 172.21.15.98:0/134548042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea43320 0x55caeeab7c50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f11e409bc90 tx=0x7f11e400ced0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11f59c8740 1 -- 172.21.15.98:0/134548042 shutdown_connections 2024-05-09T08:51:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11f59c8740 1 --2- 172.21.15.98:0/134548042 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55caeeab99a0 0x55caeeabbe00 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11cbfff640 0 client.16161 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11f59c8740 1 --2- 172.21.15.98:0/134548042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caeea43320 0x55caeeab7c50 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11f59c8740 1 --2- 172.21.15.98:0/134548042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caeea38c50 0x55caeea46bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11f59c8740 1 --2- 172.21.15.98:0/134548042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caeea38330 0x55caeea46690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11f59c8740 1 -- 172.21.15.98:0/134548042 >> 172.21.15.98:0/134548042 conn(0x55caeea34650 msgr2=0x55caeea32f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11f59c8740 1 -- 172.21.15.98:0/134548042 shutdown_connections 2024-05-09T08:51:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11f59c8740 1 -- 172.21.15.98:0/134548042 wait complete. 2024-05-09T08:51:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.683+0000 7f11f59c8740 20 client.16161 trim_cache size 0 max 16384 2024-05-09T08:51:50.688 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:51:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.684+0000 7f11f2bdc640 1 -- 172.21.15.98:0/697235169 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11dc0a2f30 msgr2=0x7f11dc0a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.684+0000 7f11f2bdc640 1 --2- 172.21.15.98:0/697235169 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11dc0a2f30 0x7f11dc0a53f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f11d8006010 tx=0x7f11d8005a20 comp rx=0 tx=0).stop 2024-05-09T08:51:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.684+0000 7f11f2bdc640 1 -- 172.21.15.98:0/697235169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec10e1e0 msgr2=0x7f11ec19ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.684+0000 7f11f2bdc640 1 --2- 172.21.15.98:0/697235169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec10e1e0 0x7f11ec19ebd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f11e4005fd0 tx=0x7f11e400c6a0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.684+0000 7f11f2bdc640 1 -- 172.21.15.98:0/697235169 shutdown_connections 2024-05-09T08:51:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.684+0000 7f11f2bdc640 1 --2- 172.21.15.98:0/697235169 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11dc0a2f30 0x7f11dc0a53f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.684+0000 7f11f2bdc640 1 --2- 172.21.15.98:0/697235169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec10e1e0 0x7f11ec19ebd0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.684+0000 7f11f2bdc640 1 --2- 172.21.15.98:0/697235169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11ec100f50 0x7f11ec19a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.684+0000 7f11f2bdc640 1 --2- 172.21.15.98:0/697235169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11ec100630 0x7f11ec19a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.684+0000 7f11f2bdc640 1 -- 172.21.15.98:0/697235169 >> 172.21.15.98:0/697235169 conn(0x7f11ec077830 msgr2=0x7f11ec10f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.684+0000 7f11f2bdc640 1 -- 172.21.15.98:0/697235169 shutdown_connections 2024-05-09T08:51:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:50.684+0000 7f11f2bdc640 1 -- 172.21.15.98:0/697235169 wait complete. 2024-05-09T08:51:50.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:50 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1585667272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:51:50.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:50 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1585667272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:51:50.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:50 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1585667272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:51:51.699 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:51:51.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:51:51.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:51:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.808+0000 7fd782058640 1 -- 172.21.15.98:0/1386650858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd77c1010c0 msgr2=0x7fd77c10d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.808+0000 7fd782058640 1 --2- 172.21.15.98:0/1386650858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd77c1010c0 0x7fd77c10d5d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd774009cd0 tx=0x7fd77402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.808+0000 7fd782058640 1 -- 172.21.15.98:0/1386650858 shutdown_connections 2024-05-09T08:51:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.808+0000 7fd782058640 1 --2- 172.21.15.98:0/1386650858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd77c1010c0 0x7fd77c10d5d0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.808+0000 7fd782058640 1 --2- 172.21.15.98:0/1386650858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c100700 0x7fd77c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.808+0000 7fd782058640 1 --2- 172.21.15.98:0/1386650858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd77c106710 0x7fd77c106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.808+0000 7fd782058640 1 -- 172.21.15.98:0/1386650858 >> 172.21.15.98:0/1386650858 conn(0x7fd77c077830 msgr2=0x7fd77c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.808+0000 7fd782058640 1 -- 172.21.15.98:0/1386650858 shutdown_connections 2024-05-09T08:51:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.808+0000 7fd782058640 1 -- 172.21.15.98:0/1386650858 wait complete. 2024-05-09T08:51:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.809+0000 7fd782058640 1 Processor -- start 2024-05-09T08:51:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.809+0000 7fd782058640 1 -- start start 2024-05-09T08:51:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.809+0000 7fd782058640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd77c100700 0x7fd77c198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.809+0000 7fd782058640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd77c1010c0 0x7fd77c199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.809+0000 7fd782058640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c106710 0x7fd77c1a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.809+0000 7fd782058640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c110b20 con 0x7fd77c100700 2024-05-09T08:51:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.809+0000 7fd782058640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1109a0 con 0x7fd77c106710 2024-05-09T08:51:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.809+0000 7fd782058640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c110ca0 con 0x7fd77c1010c0 2024-05-09T08:51:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd77c100700 0x7fd77c198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd77c100700 0x7fd77c198f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47956/0 (socket says 172.21.15.98:47956) 2024-05-09T08:51:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd77b7fe640 1 -- 172.21.15.98:0/2141104409 learned_addr learned my addr 172.21.15.98:0/2141104409 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd77bfff640 1 --2- 172.21.15.98:0/2141104409 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c106710 0x7fd77c1a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd77affd640 1 --2- 172.21.15.98:0/2141104409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd77c1010c0 0x7fd77c199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd77b7fe640 1 -- 172.21.15.98:0/2141104409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd77c1010c0 msgr2=0x7fd77c199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd77b7fe640 1 --2- 172.21.15.98:0/2141104409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd77c1010c0 0x7fd77c199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd77b7fe640 1 -- 172.21.15.98:0/2141104409 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c106710 msgr2=0x7fd77c1a3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd77b7fe640 1 --2- 172.21.15.98:0/2141104409 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c106710 0x7fd77c1a3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd77b7fe640 1 -- 172.21.15.98:0/2141104409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77c1a4500 con 0x7fd77c100700 2024-05-09T08:51:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd77b7fe640 1 --2- 172.21.15.98:0/2141104409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd77c100700 0x7fd77c198f40 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00d770 tx=0x7fd76c00db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd778ff9640 1 -- 172.21.15.98:0/2141104409 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c017070 con 0x7fd77c100700 2024-05-09T08:51:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd782058640 1 -- 172.21.15.98:0/2141104409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd77c196d30 con 0x7fd77c100700 2024-05-09T08:51:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd778ff9640 1 -- 172.21.15.98:0/2141104409 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd76c009820 con 0x7fd77c100700 2024-05-09T08:51:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd778ff9640 1 -- 172.21.15.98:0/2141104409 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c004cd0 con 0x7fd77c100700 2024-05-09T08:51:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.810+0000 7fd782058640 1 -- 172.21.15.98:0/2141104409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd77c197280 con 0x7fd77c100700 2024-05-09T08:51:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.813+0000 7fd778ff9640 1 -- 172.21.15.98:0/2141104409 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fd76c0099d0 con 0x7fd77c100700 2024-05-09T08:51:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.814+0000 7fd784d67740 1 Processor -- start 2024-05-09T08:51:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.814+0000 7fd784d67740 1 -- start start 2024-05-09T08:51:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.814+0000 7fd784d67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624194010b0 0x5624193ff3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.814+0000 7fd784d67740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624193ff9c0 0x5624193ffe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.814+0000 7fd778ff9640 1 --2- 172.21.15.98:0/2141104409 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd7640a2eb0 0x7fd7640a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.814+0000 7fd784d67740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56241940a740 0x56241940cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.814+0000 7fd784d67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562419364990 con 0x5624194010b0 2024-05-09T08:51:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.814+0000 7fd784d67740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56241930c5b0 con 0x56241940a740 2024-05-09T08:51:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.814+0000 7fd784d67740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562419360950 con 0x5624193ff9c0 2024-05-09T08:51:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.814+0000 7fd77bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624194010b0 0x5624193ff3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.814+0000 7fd77bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624194010b0 0x5624193ff3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47972/0 (socket says 172.21.15.98:47972) 2024-05-09T08:51:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.814+0000 7fd77bfff640 1 -- 172.21.15.98:0/4091093468 learned_addr learned my addr 172.21.15.98:0/4091093468 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.814+0000 7fd77affd640 1 --2- 172.21.15.98:0/2141104409 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd7640a2eb0 0x7fd7640a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.814+0000 7fd77b7fe640 1 --2- 172.21.15.98:0/4091093468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624193ff9c0 0x5624193ffe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.815+0000 7fd77bfff640 1 --2- 172.21.15.98:0/4091093468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56241940a740 0x56241940cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.815+0000 7fd77bfff640 1 -- 172.21.15.98:0/4091093468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624193ff9c0 msgr2=0x5624193ffe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.815+0000 7fd77bfff640 1 --2- 172.21.15.98:0/4091093468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624193ff9c0 0x5624193ffe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.815+0000 7fd77bfff640 1 -- 172.21.15.98:0/4091093468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56241940a740 msgr2=0x56241940cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.815+0000 7fd77bfff640 1 --2- 172.21.15.98:0/4091093468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56241940a740 0x56241940cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.815+0000 7fd77bfff640 1 -- 172.21.15.98:0/4091093468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624191407e0 con 0x5624194010b0 2024-05-09T08:51:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.815+0000 7fd778ff9640 1 -- 172.21.15.98:0/2141104409 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fd76c0cf410 con 0x7fd77c100700 2024-05-09T08:51:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.815+0000 7fd77affd640 1 --2- 172.21.15.98:0/2141104409 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd7640a2eb0 0x7fd7640a5370 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0694e0 tx=0x7fd77000a2d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.815+0000 7fd77bfff640 1 --2- 172.21.15.98:0/4091093468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624194010b0 0x5624193ff3f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd774009cd0 tx=0x7fd774005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65514b31e324c8c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.815+0000 7fd761ffb640 1 -- 172.21.15.98:0/4091093468 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd774031c90 con 0x5624194010b0 2024-05-09T08:51:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.815+0000 7fd761ffb640 1 -- 172.21.15.98:0/4091093468 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd774036040 con 0x5624194010b0 2024-05-09T08:51:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.815+0000 7fd761ffb640 1 -- 172.21.15.98:0/4091093468 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7740305a0 con 0x5624194010b0 2024-05-09T08:51:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.816+0000 7fd784d67740 1 -- 172.21.15.98:0/4091093468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624194010b0 msgr2=0x5624193ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.816+0000 7fd784d67740 1 --2- 172.21.15.98:0/4091093468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624194010b0 0x5624193ff3f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd774009cd0 tx=0x7fd774005fa0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.816+0000 7fd784d67740 1 -- 172.21.15.98:0/4091093468 shutdown_connections 2024-05-09T08:51:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.816+0000 7fd784d67740 1 --2- 172.21.15.98:0/4091093468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56241940a740 0x56241940cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.816+0000 7fd784d67740 1 --2- 172.21.15.98:0/4091093468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624193ff9c0 0x5624193ffe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.816+0000 7fd784d67740 1 --2- 172.21.15.98:0/4091093468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624194010b0 0x5624193ff3f0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.816+0000 7fd784d67740 1 -- 172.21.15.98:0/4091093468 >> 172.21.15.98:0/4091093468 conn(0x5624193c80a0 msgr2=0x5624193fbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.816+0000 7fd784d67740 1 -- 172.21.15.98:0/4091093468 shutdown_connections 2024-05-09T08:51:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.816+0000 7fd784d67740 1 -- 172.21.15.98:0/4091093468 wait complete. 2024-05-09T08:51:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.817+0000 7fd784d67740 1 -- start start 2024-05-09T08:51:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.817+0000 7fd784d67740 1 Processor -- start 2024-05-09T08:51:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.817+0000 7fd784d67740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624193ff9c0 0x562419407690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.817+0000 7fd784d67740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624194010b0 0x562419407bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.817+0000 7fd784d67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241940a740 0x562419408110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.817+0000 7fd784d67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56241940d570 con 0x56241940a740 2024-05-09T08:51:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.817+0000 7fd784d67740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56241931fb40 con 0x5624193ff9c0 2024-05-09T08:51:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.817+0000 7fd784d67740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56241940d6f0 con 0x5624194010b0 2024-05-09T08:51:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.817+0000 7fd77bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241940a740 0x562419408110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.817+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624193ff9c0 0x562419407690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.817+0000 7fd784d67740 10 client.0 _ll_register_callbacks cb 0x5624193140f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.817+0000 7fd77bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241940a740 0x562419408110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47986/0 (socket says 172.21.15.98:47986) 2024-05-09T08:51:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.817+0000 7fd77bfff640 1 -- 172.21.15.98:0/3712781491 learned_addr learned my addr 172.21.15.98:0/3712781491 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.818+0000 7fd77affd640 1 --2- 172.21.15.98:0/3712781491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624194010b0 0x562419407bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.818+0000 7fd77b7fe640 1 -- 172.21.15.98:0/3712781491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624194010b0 msgr2=0x562419407bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.818+0000 7fd77b7fe640 1 --2- 172.21.15.98:0/3712781491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624194010b0 0x562419407bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.818+0000 7fd77b7fe640 1 -- 172.21.15.98:0/3712781491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241940a740 msgr2=0x562419408110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.818+0000 7fd77b7fe640 1 --2- 172.21.15.98:0/3712781491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241940a740 0x562419408110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.818+0000 7fd77b7fe640 1 -- 172.21.15.98:0/3712781491 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624191a0f70 con 0x5624193ff9c0 2024-05-09T08:51:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.818+0000 7fd77b7fe640 1 --2- 172.21.15.98:0/3712781491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624193ff9c0 0x562419407690 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0cc530 tx=0x7fd76c09afa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=546b5ab9d676c4f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.818+0000 7fd75affd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:51:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.818+0000 7fd75affd640 1 -- 172.21.15.98:0/3712781491 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c020070 con 0x5624193ff9c0 2024-05-09T08:51:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.818+0000 7fd784d67740 1 -- 172.21.15.98:0/3712781491 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd774036040 con 0x5624193ff9c0 2024-05-09T08:51:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.818+0000 7fd75affd640 1 -- 172.21.15.98:0/3712781491 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fd76c0845d0 con 0x5624193ff9c0 2024-05-09T08:51:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.818+0000 7fd75affd640 1 -- 172.21.15.98:0/3712781491 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd76c09b330 con 0x5624193ff9c0 2024-05-09T08:51:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.818+0000 7fd784d67740 10 client.25541 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.818+0000 7fd784d67740 1 -- 172.21.15.98:0/3712781491 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562419038170 con 0x5624193ff9c0 2024-05-09T08:51:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.819+0000 7fd75affd640 1 -- 172.21.15.98:0/3712781491 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd774036040 con 0x5624193ff9c0 2024-05-09T08:51:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.819+0000 7fd75affd640 1 -- 172.21.15.98:0/3712781491 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd76c0a0ab0 con 0x5624193ff9c0 2024-05-09T08:51:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.819+0000 7fd784d67740 10 client.25541 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.819+0000 7fd784d67740 10 client.25541 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.819+0000 7fd784d67740 20 client.25541 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.819+0000 7fd784d67740 1 --2- 172.21.15.98:0/3712781491 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56241941aa90 0x56241941cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.819+0000 7fd784d67740 4 client.25541 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.819+0000 7fd784d67740 1 -- 172.21.15.98:0/3712781491 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624193720c0 con 0x56241941aa90 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.819+0000 7fd77affd640 1 --2- 172.21.15.98:0/3712781491 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56241941aa90 0x56241941cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.820+0000 7fd77affd640 1 --2- 172.21.15.98:0/3712781491 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56241941aa90 0x56241941cee0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3caa07600aed0346 server_cookie=354b22efda99a357 in_seq=0 out_seq=0 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.820+0000 7fd75affd640 10 client.25541 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.820+0000 7fd75affd640 1 -- 172.21.15.98:0/3712781491 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5624193720c0 con 0x56241941aa90 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.820+0000 7fd75affd640 10 client.25541 handle_command_reply: tid=0 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.820+0000 7fd784d67740 1 client.25541 shutdown 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.820+0000 7fd784d67740 1 -- 172.21.15.98:0/3712781491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624193ff9c0 msgr2=0x562419407690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.820+0000 7fd784d67740 1 --2- 172.21.15.98:0/3712781491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624193ff9c0 0x562419407690 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0cc530 tx=0x7fd76c09afa0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.820+0000 7fd784d67740 1 -- 172.21.15.98:0/3712781491 shutdown_connections 2024-05-09T08:51:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.820+0000 7fd784d67740 1 --2- 172.21.15.98:0/3712781491 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56241941aa90 0x56241941cee0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.820+0000 7fd784d67740 1 --2- 172.21.15.98:0/3712781491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241940a740 0x562419408110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.820+0000 7fd784d67740 1 --2- 172.21.15.98:0/3712781491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624194010b0 0x562419407bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.820+0000 7fd784d67740 1 --2- 172.21.15.98:0/3712781491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624193ff9c0 0x562419407690 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.820+0000 7fd784d67740 1 -- 172.21.15.98:0/3712781491 >> 172.21.15.98:0/3712781491 conn(0x5624193c80a0 msgr2=0x562419401d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:51.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.820+0000 7fd75affd640 0 client.25541 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:51.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.821+0000 7fd784d67740 1 -- 172.21.15.98:0/3712781491 shutdown_connections 2024-05-09T08:51:51.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.821+0000 7fd784d67740 1 -- 172.21.15.98:0/3712781491 wait complete. 2024-05-09T08:51:51.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.821+0000 7fd784d67740 20 client.25541 trim_cache size 0 max 16384 2024-05-09T08:51:51.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:51 smithi098 ceph-mon[30092]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:51:51.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.838+0000 7fd784d67740 1 Processor -- start 2024-05-09T08:51:51.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.838+0000 7fd784d67740 1 -- start start 2024-05-09T08:51:51.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd784d67740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624193ff9c0 0x5624194224f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd784d67740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624194010b0 0x562419422a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd784d67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241940a740 0x562419404910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd784d67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1b0e10 con 0x56241940a740 2024-05-09T08:51:51.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd784d67740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1b2950 con 0x5624193ff9c0 2024-05-09T08:51:51.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd784d67740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1b14e0 con 0x5624194010b0 2024-05-09T08:51:51.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd77bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241940a740 0x562419404910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd77bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241940a740 0x562419404910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47994/0 (socket says 172.21.15.98:47994) 2024-05-09T08:51:51.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd77bfff640 1 -- 172.21.15.98:0/2799356534 learned_addr learned my addr 172.21.15.98:0/2799356534 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:51.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd77affd640 1 --2- 172.21.15.98:0/2799356534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624194010b0 0x562419422a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd77b7fe640 1 --2- 172.21.15.98:0/2799356534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624193ff9c0 0x5624194224f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd77bfff640 1 -- 172.21.15.98:0/2799356534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624194010b0 msgr2=0x562419422a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd77bfff640 1 --2- 172.21.15.98:0/2799356534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624194010b0 0x562419422a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd77bfff640 1 -- 172.21.15.98:0/2799356534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624193ff9c0 msgr2=0x5624194224f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd77bfff640 1 --2- 172.21.15.98:0/2799356534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624193ff9c0 0x5624194224f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd77bfff640 1 -- 172.21.15.98:0/2799356534 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562419400c50 con 0x56241940a740 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.839+0000 7fd77bfff640 1 --2- 172.21.15.98:0/2799356534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241940a740 0x562419404910 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd77402aed0 tx=0x7fd774000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5554e37e1af49a8c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.840+0000 7fd7617fa640 1 -- 172.21.15.98:0/2799356534 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd774004840 con 0x56241940a740 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.840+0000 7fd7617fa640 1 -- 172.21.15.98:0/2799356534 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd774036030 con 0x56241940a740 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.840+0000 7fd7617fa640 1 -- 172.21.15.98:0/2799356534 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd774030a80 con 0x56241940a740 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.840+0000 7fd784d67740 1 -- 172.21.15.98:0/2799356534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241940a740 msgr2=0x562419404910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.840+0000 7fd784d67740 1 --2- 172.21.15.98:0/2799356534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241940a740 0x562419404910 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd77402aed0 tx=0x7fd774000f40 comp rx=0 tx=0).stop 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.840+0000 7fd784d67740 1 -- 172.21.15.98:0/2799356534 shutdown_connections 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.840+0000 7fd784d67740 1 --2- 172.21.15.98:0/2799356534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56241940a740 0x562419404910 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.840+0000 7fd784d67740 1 --2- 172.21.15.98:0/2799356534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5624194010b0 0x562419422a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.840+0000 7fd784d67740 1 --2- 172.21.15.98:0/2799356534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624193ff9c0 0x5624194224f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.840+0000 7fd784d67740 1 -- 172.21.15.98:0/2799356534 >> 172.21.15.98:0/2799356534 conn(0x5624193c80a0 msgr2=0x5624193fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:51.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.841+0000 7fd784d67740 1 -- 172.21.15.98:0/2799356534 shutdown_connections 2024-05-09T08:51:51.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.841+0000 7fd784d67740 1 -- 172.21.15.98:0/2799356534 wait complete. 2024-05-09T08:51:51.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.841+0000 7fd784d67740 1 -- start start 2024-05-09T08:51:51.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.841+0000 7fd784d67740 1 Processor -- start 2024-05-09T08:51:51.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.841+0000 7fd784d67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624193ff9c0 0x562419422170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.841+0000 7fd784d67740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624194010b0 0x5624194226b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.841+0000 7fd784d67740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56241940a740 0x562419420610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.841+0000 7fd784d67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1b2280 con 0x5624193ff9c0 2024-05-09T08:51:51.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.841+0000 7fd784d67740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1b1bb0 con 0x5624194010b0 2024-05-09T08:51:51.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.841+0000 7fd784d67740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1aedf0 con 0x56241940a740 2024-05-09T08:51:51.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624193ff9c0 0x562419422170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624193ff9c0 0x562419422170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48006/0 (socket says 172.21.15.98:48006) 2024-05-09T08:51:51.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd77b7fe640 1 -- 172.21.15.98:0/1170433399 learned_addr learned my addr 172.21.15.98:0/1170433399 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:51.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd77affd640 1 --2- 172.21.15.98:0/1170433399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624194010b0 0x5624194226b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd784d67740 10 client.0 _ll_register_callbacks cb 0x7fd76c0046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:51.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd77bfff640 1 --2- 172.21.15.98:0/1170433399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56241940a740 0x562419420610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd77b7fe640 1 -- 172.21.15.98:0/1170433399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56241940a740 msgr2=0x562419420610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd77b7fe640 1 --2- 172.21.15.98:0/1170433399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56241940a740 0x562419420610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd77b7fe640 1 -- 172.21.15.98:0/1170433399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624194010b0 msgr2=0x5624194226b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd77b7fe640 1 --2- 172.21.15.98:0/1170433399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624194010b0 0x5624194226b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd77b7fe640 1 -- 172.21.15.98:0/1170433399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562419422f70 con 0x5624193ff9c0 2024-05-09T08:51:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd784d67740 1 -- 172.21.15.98:0/1170433399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd774036030 con 0x5624193ff9c0 2024-05-09T08:51:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd77b7fe640 1 --2- 172.21.15.98:0/1170433399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624193ff9c0 0x562419422170 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0a4160 tx=0x7fd76c009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f8dc7220be34e07 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd75affd640 10 client.16179 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd75affd640 1 -- 172.21.15.98:0/1170433399 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c0b8030 con 0x5624193ff9c0 2024-05-09T08:51:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd75affd640 1 -- 172.21.15.98:0/1170433399 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fd76c09b7a0 con 0x5624193ff9c0 2024-05-09T08:51:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd75affd640 1 -- 172.21.15.98:0/1170433399 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd76c020070 con 0x5624193ff9c0 2024-05-09T08:51:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd784d67740 10 client.16179 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.842+0000 7fd784d67740 1 -- 172.21.15.98:0/1170433399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=29}) -- 0x562419422f70 con 0x5624193ff9c0 2024-05-09T08:51:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.843+0000 7fd75affd640 1 -- 172.21.15.98:0/1170433399 <== mon.0 v2:172.21.15.98:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd76c0a7060 con 0x5624193ff9c0 2024-05-09T08:51:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.843+0000 7fd784d67740 10 client.16179 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.843+0000 7fd784d67740 10 client.16179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.843+0000 7fd784d67740 20 client.16179 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.843+0000 7fd784d67740 1 --2- 172.21.15.98:0/1170433399 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5624194819c0 0x562419483db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.843+0000 7fd784d67740 4 client.16179 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:51:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.843+0000 7fd784d67740 1 -- 172.21.15.98:0/1170433399 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56241936c410 con 0x5624194819c0 2024-05-09T08:51:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.843+0000 7fd77affd640 1 --2- 172.21.15.98:0/1170433399 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5624194819c0 0x562419483db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.843+0000 7fd75affd640 10 client.16179 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.843+0000 7fd77affd640 1 --2- 172.21.15.98:0/1170433399 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5624194819c0 0x562419483db0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18233b8801a627b9 server_cookie=d6a7a31654ed14ad in_seq=0 out_seq=0 2024-05-09T08:51:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.844+0000 7fd75affd640 1 -- 172.21.15.98:0/1170433399 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56241936c410 con 0x5624194819c0 2024-05-09T08:51:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.844+0000 7fd75affd640 10 client.16179 handle_command_reply: tid=0 2024-05-09T08:51:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.844+0000 7fd784d67740 1 client.16179 shutdown 2024-05-09T08:51:51.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.845+0000 7fd784d67740 1 -- 172.21.15.98:0/1170433399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624193ff9c0 msgr2=0x562419422170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:51.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.845+0000 7fd784d67740 1 --2- 172.21.15.98:0/1170433399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624193ff9c0 0x562419422170 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0a4160 tx=0x7fd76c009f70 comp rx=0 tx=0).stop 2024-05-09T08:51:51.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.845+0000 7fd784d67740 1 -- 172.21.15.98:0/1170433399 shutdown_connections 2024-05-09T08:51:51.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.845+0000 7fd784d67740 1 --2- 172.21.15.98:0/1170433399 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5624194819c0 0x562419483db0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.845+0000 7fd75affd640 0 client.16179 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:51.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.845+0000 7fd784d67740 1 --2- 172.21.15.98:0/1170433399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56241940a740 0x562419420610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.845+0000 7fd784d67740 1 --2- 172.21.15.98:0/1170433399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624194010b0 0x5624194226b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.845+0000 7fd784d67740 1 --2- 172.21.15.98:0/1170433399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624193ff9c0 0x562419422170 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.845+0000 7fd784d67740 1 -- 172.21.15.98:0/1170433399 >> 172.21.15.98:0/1170433399 conn(0x5624193c80a0 msgr2=0x562419408ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:51.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.845+0000 7fd784d67740 1 -- 172.21.15.98:0/1170433399 shutdown_connections 2024-05-09T08:51:51.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.845+0000 7fd784d67740 1 -- 172.21.15.98:0/1170433399 wait complete. 2024-05-09T08:51:51.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.845+0000 7fd784d67740 20 client.16179 trim_cache size 0 max 16384 2024-05-09T08:51:51.850 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:51:51.850 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:51:51.850 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "1541f91c-e415-4366-b9e8-5748352f5e4b", 2024-05-09T08:51:51.850 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:51:51.850 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:51:51.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.846+0000 7fd782058640 1 -- 172.21.15.98:0/2141104409 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd7640a2eb0 msgr2=0x7fd7640a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:51.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.846+0000 7fd782058640 1 --2- 172.21.15.98:0/2141104409 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd7640a2eb0 0x7fd7640a5370 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0694e0 tx=0x7fd77000a2d0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.846+0000 7fd782058640 1 -- 172.21.15.98:0/2141104409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd77c100700 msgr2=0x7fd77c198f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:51.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.846+0000 7fd782058640 1 --2- 172.21.15.98:0/2141104409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd77c100700 0x7fd77c198f40 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00d770 tx=0x7fd76c00db70 comp rx=0 tx=0).stop 2024-05-09T08:51:51.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.846+0000 7fd782058640 1 -- 172.21.15.98:0/2141104409 shutdown_connections 2024-05-09T08:51:51.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.846+0000 7fd782058640 1 --2- 172.21.15.98:0/2141104409 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd7640a2eb0 0x7fd7640a5370 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.846+0000 7fd782058640 1 --2- 172.21.15.98:0/2141104409 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c106710 0x7fd77c1a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.846+0000 7fd782058640 1 --2- 172.21.15.98:0/2141104409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd77c1010c0 0x7fd77c199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.846+0000 7fd782058640 1 --2- 172.21.15.98:0/2141104409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd77c100700 0x7fd77c198f40 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.846+0000 7fd782058640 1 -- 172.21.15.98:0/2141104409 >> 172.21.15.98:0/2141104409 conn(0x7fd77c077830 msgr2=0x7fd77c102ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:51.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.846+0000 7fd782058640 1 -- 172.21.15.98:0/2141104409 shutdown_connections 2024-05-09T08:51:51.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.846+0000 7fd782058640 1 -- 172.21.15.98:0/2141104409 wait complete. 2024-05-09T08:51:51.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:51:51.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:51:51.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:51 smithi142 ceph-mon[34625]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:51:51.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.963+0000 7f34b60b3640 1 -- 172.21.15.98:0/257942425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b010bdf0 msgr2=0x7f34b010e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:51.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.963+0000 7f34b60b3640 1 --2- 172.21.15.98:0/257942425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b010bdf0 0x7f34b010e290 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f34a8009ca0 tx=0x7f34a802a9e0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.963+0000 7f34b60b3640 1 -- 172.21.15.98:0/257942425 shutdown_connections 2024-05-09T08:51:51.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.963+0000 7f34b60b3640 1 --2- 172.21.15.98:0/257942425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b010bdf0 0x7f34b010e290 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.963+0000 7f34b60b3640 1 --2- 172.21.15.98:0/257942425 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34b01033d0 0x7f34b010b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.963+0000 7f34b60b3640 1 --2- 172.21.15.98:0/257942425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34b0102ab0 0x7f34b0102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.963+0000 7f34b60b3640 1 -- 172.21.15.98:0/257942425 >> 172.21.15.98:0/257942425 conn(0x7f34b0077830 msgr2=0x7f34b0077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:51.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.963+0000 7f34b60b3640 1 -- 172.21.15.98:0/257942425 shutdown_connections 2024-05-09T08:51:51.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.963+0000 7f34b60b3640 1 -- 172.21.15.98:0/257942425 wait complete. 2024-05-09T08:51:51.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.964+0000 7f34b60b3640 1 Processor -- start 2024-05-09T08:51:51.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.964+0000 7f34b60b3640 1 -- start start 2024-05-09T08:51:51.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34b60b3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b0102ab0 0x7f34b01980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34b60b3640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34b01033d0 0x7f34b0198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34b60b3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34b010bdf0 0x7f34b019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34b60b3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34b0111bf0 con 0x7f34b0102ab0 2024-05-09T08:51:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34b60b3640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f34b0111a70 con 0x7f34b010bdf0 2024-05-09T08:51:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34b60b3640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f34b0111d70 con 0x7f34b01033d0 2024-05-09T08:51:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34af7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b0102ab0 0x7f34b01980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34af7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b0102ab0 0x7f34b01980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48028/0 (socket says 172.21.15.98:48028) 2024-05-09T08:51:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34af7fe640 1 -- 172.21.15.98:0/750428065 learned_addr learned my addr 172.21.15.98:0/750428065 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34aeffd640 1 --2- 172.21.15.98:0/750428065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34b01033d0 0x7f34b0198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34affff640 1 --2- 172.21.15.98:0/750428065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34b010bdf0 0x7f34b019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34af7fe640 1 -- 172.21.15.98:0/750428065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34b01033d0 msgr2=0x7f34b0198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34af7fe640 1 --2- 172.21.15.98:0/750428065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34b01033d0 0x7f34b0198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34af7fe640 1 -- 172.21.15.98:0/750428065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34b010bdf0 msgr2=0x7f34b019c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34af7fe640 1 --2- 172.21.15.98:0/750428065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34b010bdf0 0x7f34b019c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34af7fe640 1 -- 172.21.15.98:0/750428065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b019d090 con 0x7f34b0102ab0 2024-05-09T08:51:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.965+0000 7f34af7fe640 1 --2- 172.21.15.98:0/750428065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b0102ab0 0x7f34b01980f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f34a000d950 tx=0x7f34a000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.966+0000 7f34acff9640 1 -- 172.21.15.98:0/750428065 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a0017030 con 0x7f34b0102ab0 2024-05-09T08:51:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.966+0000 7f34b60b3640 1 -- 172.21.15.98:0/750428065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b019d380 con 0x7f34b0102ab0 2024-05-09T08:51:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.966+0000 7f34acff9640 1 -- 172.21.15.98:0/750428065 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34a0004020 con 0x7f34b0102ab0 2024-05-09T08:51:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.966+0000 7f34acff9640 1 -- 172.21.15.98:0/750428065 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a00047a0 con 0x7f34b0102ab0 2024-05-09T08:51:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.966+0000 7f34b60b3640 1 -- 172.21.15.98:0/750428065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b01a4c10 con 0x7f34b0102ab0 2024-05-09T08:51:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.968+0000 7f34b8dc2740 1 Processor -- start 2024-05-09T08:51:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.968+0000 7f34b8dc2740 1 -- start start 2024-05-09T08:51:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34b8dc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964280 0x5587a7964660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34b8dc2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964ba0 0x5587a796ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34b8dc2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a796f270 0x5587a7971660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34b8dc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5587a78c9d00 con 0x5587a7964280 2024-05-09T08:51:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34b8dc2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5587a78714c0 con 0x5587a7964ba0 2024-05-09T08:51:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34b8dc2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5587a78c5cc0 con 0x5587a796f270 2024-05-09T08:51:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964280 0x5587a7964660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964280 0x5587a7964660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48038/0 (socket says 172.21.15.98:48038) 2024-05-09T08:51:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34affff640 1 -- 172.21.15.98:0/3567519156 learned_addr learned my addr 172.21.15.98:0/3567519156 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34aeffd640 1 --2- 172.21.15.98:0/3567519156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a796f270 0x5587a7971660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34acff9640 1 -- 172.21.15.98:0/750428065 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f34a00041d0 con 0x7f34b0102ab0 2024-05-09T08:51:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34affff640 1 -- 172.21.15.98:0/3567519156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a796f270 msgr2=0x5587a7971660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34affff640 1 --2- 172.21.15.98:0/3567519156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a796f270 0x5587a7971660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34affff640 1 -- 172.21.15.98:0/3567519156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964ba0 msgr2=0x5587a796ed30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:51:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34affff640 1 --2- 172.21.15.98:0/3567519156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964ba0 0x5587a796ed30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.969+0000 7f34affff640 1 -- 172.21.15.98:0/3567519156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587a76a77d0 con 0x5587a7964280 2024-05-09T08:51:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.970+0000 7f34affff640 1 --2- 172.21.15.98:0/3567519156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964280 0x5587a7964660 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f34a802afd0 tx=0x7f34a8005fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b8025d1269ae84 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.970+0000 7f3495ffb640 1 -- 172.21.15.98:0/3567519156 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8046070 con 0x5587a7964280 2024-05-09T08:51:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.970+0000 7f3495ffb640 1 -- 172.21.15.98:0/3567519156 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34a8003970 con 0x5587a7964280 2024-05-09T08:51:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.970+0000 7f34acff9640 1 --2- 172.21.15.98:0/750428065 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f34980a2eb0 0x7f34980a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.970+0000 7f34af7fe640 1 --2- 172.21.15.98:0/750428065 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f34980a2eb0 0x7f34980a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.970+0000 7f34acff9640 1 -- 172.21.15.98:0/750428065 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f34a00cf4c0 con 0x7f34b0102ab0 2024-05-09T08:51:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.971+0000 7f3495ffb640 1 -- 172.21.15.98:0/3567519156 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8033050 con 0x5587a7964280 2024-05-09T08:51:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.971+0000 7f34b8dc2740 1 -- 172.21.15.98:0/3567519156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964280 msgr2=0x5587a7964660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.971+0000 7f34af7fe640 1 --2- 172.21.15.98:0/750428065 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f34980a2eb0 0x7f34980a5370 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f34a00cc670 tx=0x7f34a0090870 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.971+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/3567519156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964280 0x5587a7964660 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f34a802afd0 tx=0x7f34a8005fc0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.971+0000 7f34b8dc2740 1 -- 172.21.15.98:0/3567519156 shutdown_connections 2024-05-09T08:51:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.971+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/3567519156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a796f270 0x5587a7971660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.971+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/3567519156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964ba0 0x5587a796ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.971+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/3567519156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964280 0x5587a7964660 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.971+0000 7f34b8dc2740 1 -- 172.21.15.98:0/3567519156 >> 172.21.15.98:0/3567519156 conn(0x5587a79605a0 msgr2=0x5587a7960980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.971+0000 7f34b8dc2740 1 -- 172.21.15.98:0/3567519156 shutdown_connections 2024-05-09T08:51:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.971+0000 7f34b8dc2740 1 -- 172.21.15.98:0/3567519156 wait complete. 2024-05-09T08:51:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.971+0000 7f34b8dc2740 1 -- start start 2024-05-09T08:51:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.972+0000 7f34b8dc2740 1 Processor -- start 2024-05-09T08:51:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.972+0000 7f34b8dc2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a7964280 0x5587a796c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.972+0000 7f34b8dc2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964ba0 0x5587a796c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.972+0000 7f34b8dc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a796f270 0x5587a796cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.972+0000 7f34b8dc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5587a79720a0 con 0x5587a796f270 2024-05-09T08:51:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.972+0000 7f34b8dc2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5587a7884b90 con 0x5587a7964ba0 2024-05-09T08:51:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.972+0000 7f34b8dc2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5587a7972220 con 0x5587a7964280 2024-05-09T08:51:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.972+0000 7f34b8dc2740 10 client.0 _ll_register_callbacks cb 0x5587a78c0da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.972+0000 7f34affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a796f270 0x5587a796cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.972+0000 7f34affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a796f270 0x5587a796cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48042/0 (socket says 172.21.15.98:48042) 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.972+0000 7f34affff640 1 -- 172.21.15.98:0/2184196506 learned_addr learned my addr 172.21.15.98:0/2184196506 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.973+0000 7f34aeffd640 1 --2- 172.21.15.98:0/2184196506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964ba0 0x5587a796c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.973+0000 7f34affff640 1 -- 172.21.15.98:0/2184196506 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a7964280 msgr2=0x5587a796c140 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.973+0000 7f34affff640 1 --2- 172.21.15.98:0/2184196506 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a7964280 0x5587a796c140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.973+0000 7f34affff640 1 -- 172.21.15.98:0/2184196506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964ba0 msgr2=0x5587a796c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.973+0000 7f34affff640 1 --2- 172.21.15.98:0/2184196506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964ba0 0x5587a796c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.973+0000 7f34affff640 1 -- 172.21.15.98:0/2184196506 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587a75dd3f0 con 0x5587a796f270 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.973+0000 7f34affff640 1 --2- 172.21.15.98:0/2184196506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a796f270 0x5587a796cbc0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f34a802afa0 tx=0x7f34a802ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db1f54835f5129c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.973+0000 7f348effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.973+0000 7f348effd640 1 -- 172.21.15.98:0/2184196506 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8046040 con 0x5587a796f270 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.973+0000 7f348effd640 1 -- 172.21.15.98:0/2184196506 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f34a802ee50 con 0x5587a796f270 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.973+0000 7f34b8dc2740 1 -- 172.21.15.98:0/2184196506 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34a8003970 con 0x5587a796f270 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.974+0000 7f348effd640 1 -- 172.21.15.98:0/2184196506 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f34a8003dc0 con 0x5587a796f270 2024-05-09T08:51:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.974+0000 7f34b8dc2740 10 client.16191 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.974+0000 7f34b8dc2740 1 -- 172.21.15.98:0/2184196506 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5587a759d0f0 con 0x5587a796f270 2024-05-09T08:51:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.974+0000 7f348effd640 1 -- 172.21.15.98:0/2184196506 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34a8003970 con 0x5587a796f270 2024-05-09T08:51:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.974+0000 7f348effd640 1 -- 172.21.15.98:0/2184196506 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f34a8036790 con 0x5587a796f270 2024-05-09T08:51:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.974+0000 7f34b8dc2740 10 client.16191 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.974+0000 7f34b8dc2740 10 client.16191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.974+0000 7f34b8dc2740 20 client.16191 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.974+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/2184196506 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5587a797f440 0x5587a7981830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.974+0000 7f34b8dc2740 4 client.16191 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.974+0000 7f34b8dc2740 1 -- 172.21.15.98:0/2184196506 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587a78d7430 con 0x5587a797f440 2024-05-09T08:51:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.974+0000 7f34aeffd640 1 --2- 172.21.15.98:0/2184196506 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5587a797f440 0x5587a7981830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.975+0000 7f348effd640 10 client.16191 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.975+0000 7f34aeffd640 1 --2- 172.21.15.98:0/2184196506 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5587a797f440 0x5587a7981830 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=113f8cda8172da01 server_cookie=f596b08319e59b5 in_seq=0 out_seq=0 2024-05-09T08:51:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.975+0000 7f348effd640 1 -- 172.21.15.98:0/2184196506 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5587a78d7430 con 0x5587a797f440 2024-05-09T08:51:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.975+0000 7f348effd640 10 client.16191 handle_command_reply: tid=0 2024-05-09T08:51:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.975+0000 7f34b8dc2740 1 client.16191 shutdown 2024-05-09T08:51:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.976+0000 7f34b8dc2740 1 -- 172.21.15.98:0/2184196506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a796f270 msgr2=0x5587a796cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.976+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/2184196506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a796f270 0x5587a796cbc0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f34a802afa0 tx=0x7f34a802ed60 comp rx=0 tx=0).stop 2024-05-09T08:51:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.976+0000 7f34b8dc2740 1 -- 172.21.15.98:0/2184196506 shutdown_connections 2024-05-09T08:51:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.976+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/2184196506 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5587a797f440 0x5587a7981830 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.976+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/2184196506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a796f270 0x5587a796cbc0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.976+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/2184196506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964ba0 0x5587a796c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.976+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/2184196506 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a7964280 0x5587a796c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.976+0000 7f34b8dc2740 1 -- 172.21.15.98:0/2184196506 >> 172.21.15.98:0/2184196506 conn(0x5587a79605a0 msgr2=0x5587a795f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.976+0000 7f348effd640 0 client.16191 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.976+0000 7f34b8dc2740 1 -- 172.21.15.98:0/2184196506 shutdown_connections 2024-05-09T08:51:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.976+0000 7f34b8dc2740 1 -- 172.21.15.98:0/2184196506 wait complete. 2024-05-09T08:51:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:51.976+0000 7f34b8dc2740 20 client.16191 trim_cache size 0 max 16384 2024-05-09T08:51:51.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:51 smithi105 ceph-mon[34498]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:51:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.008+0000 7f34b8dc2740 1 Processor -- start 2024-05-09T08:51:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.008+0000 7f34b8dc2740 1 -- start start 2024-05-09T08:51:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34b8dc2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964280 0x5587a796b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34b8dc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964ba0 0x5587a797abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34b8dc2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a796f270 0x5587a7984470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34b8dc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01aed90 con 0x5587a7964ba0 2024-05-09T08:51:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34b8dc2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01b08d0 con 0x5587a7964280 2024-05-09T08:51:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34b8dc2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01af460 con 0x5587a796f270 2024-05-09T08:51:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34aeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964ba0 0x5587a797abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34aeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964ba0 0x5587a797abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48056/0 (socket says 172.21.15.98:48056) 2024-05-09T08:51:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34aeffd640 1 -- 172.21.15.98:0/160892312 learned_addr learned my addr 172.21.15.98:0/160892312 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34affff640 1 --2- 172.21.15.98:0/160892312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964280 0x5587a796b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34aeffd640 1 -- 172.21.15.98:0/160892312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a796f270 msgr2=0x5587a7984470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34aeffd640 1 --2- 172.21.15.98:0/160892312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a796f270 0x5587a7984470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34aeffd640 1 -- 172.21.15.98:0/160892312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964280 msgr2=0x5587a796b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34aeffd640 1 --2- 172.21.15.98:0/160892312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964280 0x5587a796b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34aeffd640 1 -- 172.21.15.98:0/160892312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587a796a690 con 0x5587a7964ba0 2024-05-09T08:51:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34affff640 1 --2- 172.21.15.98:0/160892312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964280 0x5587a796b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:51:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.009+0000 7f34aeffd640 1 --2- 172.21.15.98:0/160892312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964ba0 0x5587a797abd0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x5587a7961730 tx=0x7f34a400b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c815c58efa8a8ce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.010+0000 7f34957fa640 1 -- 172.21.15.98:0/160892312 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a4003d20 con 0x5587a7964ba0 2024-05-09T08:51:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.010+0000 7f34957fa640 1 -- 172.21.15.98:0/160892312 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34a4024030 con 0x5587a7964ba0 2024-05-09T08:51:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.010+0000 7f34957fa640 1 -- 172.21.15.98:0/160892312 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a4004e50 con 0x5587a7964ba0 2024-05-09T08:51:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.010+0000 7f34b8dc2740 1 -- 172.21.15.98:0/160892312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964ba0 msgr2=0x5587a797abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.011+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/160892312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964ba0 0x5587a797abd0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x5587a7961730 tx=0x7f34a400b390 comp rx=0 tx=0).stop 2024-05-09T08:51:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.011+0000 7f34b8dc2740 1 -- 172.21.15.98:0/160892312 shutdown_connections 2024-05-09T08:51:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.011+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/160892312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a796f270 0x5587a7984470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.011+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/160892312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964ba0 0x5587a797abd0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.011+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/160892312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a7964280 0x5587a796b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.011+0000 7f34b8dc2740 1 -- 172.21.15.98:0/160892312 >> 172.21.15.98:0/160892312 conn(0x5587a79605a0 msgr2=0x5587a797a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.011+0000 7f34b8dc2740 1 -- 172.21.15.98:0/160892312 shutdown_connections 2024-05-09T08:51:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.011+0000 7f34b8dc2740 1 -- 172.21.15.98:0/160892312 wait complete. 2024-05-09T08:51:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.011+0000 7f34b8dc2740 1 -- start start 2024-05-09T08:51:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.011+0000 7f34b8dc2740 1 Processor -- start 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34b8dc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964280 0x5587a79846f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34b8dc2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a7964ba0 0x5587a79ddc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964280 0x5587a79846f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964280 0x5587a79846f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48068/0 (socket says 172.21.15.98:48068) 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34affff640 1 -- 172.21.15.98:0/883210061 learned_addr learned my addr 172.21.15.98:0/883210061 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34aeffd640 1 --2- 172.21.15.98:0/883210061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a7964ba0 0x5587a79ddc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/883210061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a796f270 0x5587a79de170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34b8dc2740 1 -- 172.21.15.98:0/883210061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01b0200 con 0x5587a7964280 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34b8dc2740 1 -- 172.21.15.98:0/883210061 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01afb30 con 0x5587a796f270 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34b8dc2740 1 -- 172.21.15.98:0/883210061 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01acd70 con 0x5587a7964ba0 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34b8dc2740 10 client.0 _ll_register_callbacks cb 0x7f34a8044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34aeffd640 1 -- 172.21.15.98:0/883210061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a796f270 msgr2=0x5587a79de170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34aeffd640 1 --2- 172.21.15.98:0/883210061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a796f270 0x5587a79de170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34aeffd640 1 -- 172.21.15.98:0/883210061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964280 msgr2=0x5587a79846f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34aeffd640 1 --2- 172.21.15.98:0/883210061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964280 0x5587a79846f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34aeffd640 1 -- 172.21.15.98:0/883210061 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587a79de810 con 0x5587a7964ba0 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34b8dc2740 1 -- 172.21.15.98:0/883210061 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34a4024030 con 0x5587a7964ba0 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.012+0000 7f34affff640 1 --2- 172.21.15.98:0/883210061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964280 0x5587a79846f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.013+0000 7f34aeffd640 1 --2- 172.21.15.98:0/883210061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a7964ba0 0x5587a79ddc30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x5587a796a920 tx=0x7f34a4004890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=709659f97b826a5a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.013+0000 7f348effd640 10 client.25498 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.013+0000 7f348effd640 1 -- 172.21.15.98:0/883210061 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a4003b20 con 0x5587a7964ba0 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.013+0000 7f348effd640 1 -- 172.21.15.98:0/883210061 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f34a4003cc0 con 0x5587a7964ba0 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.013+0000 7f348effd640 1 -- 172.21.15.98:0/883210061 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f34a4003f20 con 0x5587a7964ba0 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.013+0000 7f34b8dc2740 10 client.25498 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.013+0000 7f34b8dc2740 1 -- 172.21.15.98:0/883210061 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5587a79841d0 con 0x5587a7964ba0 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.013+0000 7f348effd640 1 -- 172.21.15.98:0/883210061 <== mon.1 v2:172.21.15.142:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f34a4005020 con 0x5587a7964ba0 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.013+0000 7f34b8dc2740 10 client.25498 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.014+0000 7f34b8dc2740 10 client.25498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.014+0000 7f34b8dc2740 20 client.25498 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.014+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/883210061 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5587a79e9d10 0x5587a79ec100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.014+0000 7f34b8dc2740 4 client.25498 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:51:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.014+0000 7f34b8dc2740 1 -- 172.21.15.98:0/883210061 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5587a78d1780 con 0x5587a79e9d10 2024-05-09T08:51:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.014+0000 7f34affff640 1 --2- 172.21.15.98:0/883210061 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5587a79e9d10 0x5587a79ec100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.014+0000 7f34affff640 1 --2- 172.21.15.98:0/883210061 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5587a79e9d10 0x5587a79ec100 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77a3de1e70099348 server_cookie=7db964e6363ae756 in_seq=0 out_seq=0 2024-05-09T08:51:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.014+0000 7f348effd640 10 client.25498 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.014+0000 7f348effd640 1 -- 172.21.15.98:0/883210061 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5587a78d1780 con 0x5587a79e9d10 2024-05-09T08:51:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.014+0000 7f348effd640 10 client.25498 handle_command_reply: tid=0 2024-05-09T08:51:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.014+0000 7f34b8dc2740 1 client.25498 shutdown 2024-05-09T08:51:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.015+0000 7f34b8dc2740 1 -- 172.21.15.98:0/883210061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a7964ba0 msgr2=0x5587a79ddc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.015+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/883210061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a7964ba0 0x5587a79ddc30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x5587a796a920 tx=0x7f34a4004890 comp rx=0 tx=0).stop 2024-05-09T08:51:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.015+0000 7f34b8dc2740 1 -- 172.21.15.98:0/883210061 shutdown_connections 2024-05-09T08:51:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.015+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/883210061 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5587a79e9d10 0x5587a79ec100 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.015+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/883210061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5587a796f270 0x5587a79de170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.015+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/883210061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5587a7964ba0 0x5587a79ddc30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.015+0000 7f34b8dc2740 1 --2- 172.21.15.98:0/883210061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587a7964280 0x5587a79846f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.015+0000 7f34b8dc2740 1 -- 172.21.15.98:0/883210061 >> 172.21.15.98:0/883210061 conn(0x5587a79605a0 msgr2=0x5587a797a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.015+0000 7f348effd640 0 client.25498 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.015+0000 7f34b8dc2740 1 -- 172.21.15.98:0/883210061 shutdown_connections 2024-05-09T08:51:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.015+0000 7f34b8dc2740 1 -- 172.21.15.98:0/883210061 wait complete. 2024-05-09T08:51:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.015+0000 7f34b8dc2740 20 client.25498 trim_cache size 0 max 16384 2024-05-09T08:51:52.019 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:51:52.019 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:51:52.019 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:51:52.019 INFO:teuthology.orchestra.run.smithi098.stdout: "1541f91c-e415-4366-b9e8-5748352f5e4b": { 2024-05-09T08:51:52.019 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:51:52.020 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "1541f91c-e415-4366-b9e8-5748352f5e4b", 2024-05-09T08:51:52.020 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:51:52.020 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:51:52.020 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:51:52.020 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:51:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.016+0000 7f34b60b3640 1 -- 172.21.15.98:0/750428065 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f34980a2eb0 msgr2=0x7f34980a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.016+0000 7f34b60b3640 1 --2- 172.21.15.98:0/750428065 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f34980a2eb0 0x7f34980a5370 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f34a00cc670 tx=0x7f34a0090870 comp rx=0 tx=0).stop 2024-05-09T08:51:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.016+0000 7f34b60b3640 1 -- 172.21.15.98:0/750428065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b0102ab0 msgr2=0x7f34b01980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.016+0000 7f34b60b3640 1 --2- 172.21.15.98:0/750428065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b0102ab0 0x7f34b01980f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f34a000d950 tx=0x7f34a000de10 comp rx=0 tx=0).stop 2024-05-09T08:51:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.016+0000 7f34b60b3640 1 -- 172.21.15.98:0/750428065 shutdown_connections 2024-05-09T08:51:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.016+0000 7f34b60b3640 1 --2- 172.21.15.98:0/750428065 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f34980a2eb0 0x7f34980a5370 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.016+0000 7f34b60b3640 1 --2- 172.21.15.98:0/750428065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34b010bdf0 0x7f34b019c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.016+0000 7f34b60b3640 1 --2- 172.21.15.98:0/750428065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34b01033d0 0x7f34b0198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.016+0000 7f34b60b3640 1 --2- 172.21.15.98:0/750428065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b0102ab0 0x7f34b01980f0 secure :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f34a000d950 tx=0x7f34a000de10 comp rx=0 tx=0).stop 2024-05-09T08:51:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.016+0000 7f34b60b3640 1 -- 172.21.15.98:0/750428065 >> 172.21.15.98:0/750428065 conn(0x7f34b0077830 msgr2=0x7f34b00fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.016+0000 7f34b60b3640 1 -- 172.21.15.98:0/750428065 shutdown_connections 2024-05-09T08:51:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:51:52.016+0000 7f34b60b3640 1 -- 172.21.15.98:0/750428065 wait complete. 2024-05-09T08:51:52.030 INFO:tasks.cephfs.filesystem:scrub status for tag:1541f91c-e415-4366-b9e8-5748352f5e4b - {'path': '/', 'tag': '1541f91c-e415-4366-b9e8-5748352f5e4b', 'options': 'recursive,force'} 2024-05-09T08:51:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.806+0000 7f8034cc1640 1 -- 172.21.15.98:0/2052302418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8030101120 msgr2=0x7f803010d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.806+0000 7f8034cc1640 1 --2- 172.21.15.98:0/2052302418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8030101120 0x7f803010d5e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8028009cd0 tx=0x7f802802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.806+0000 7f8034cc1640 1 -- 172.21.15.98:0/2052302418 shutdown_connections 2024-05-09T08:51:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.806+0000 7f8034cc1640 1 --2- 172.21.15.98:0/2052302418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8030101120 0x7f803010d5e0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.806+0000 7f8034cc1640 1 --2- 172.21.15.98:0/2052302418 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8030100760 0x7f8030100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.806+0000 7f8034cc1640 1 --2- 172.21.15.98:0/2052302418 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8030106720 0x7f8030106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.806+0000 7f8034cc1640 1 -- 172.21.15.98:0/2052302418 >> 172.21.15.98:0/2052302418 conn(0x7f80300778b0 msgr2=0x7f8030077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.806+0000 7f8034cc1640 1 -- 172.21.15.98:0/2052302418 shutdown_connections 2024-05-09T08:51:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.806+0000 7f8034cc1640 1 -- 172.21.15.98:0/2052302418 wait complete. 2024-05-09T08:51:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.807+0000 7f8034cc1640 1 Processor -- start 2024-05-09T08:51:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.807+0000 7f8034cc1640 1 -- start start 2024-05-09T08:51:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f8034cc1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8030100760 0x7f8030196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f8034cc1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8030101120 0x7f8030197400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f8034cc1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8030106720 0x7f803019b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f8034cc1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8030110b80 con 0x7f8030106720 2024-05-09T08:51:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f8034cc1640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8030110a00 con 0x7f8030100760 2024-05-09T08:51:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f8034cc1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8030110d00 con 0x7f8030101120 2024-05-09T08:51:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f802ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8030106720 0x7f803019b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f802e59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8030100760 0x7f8030196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f802ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8030106720 0x7f803019b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48084/0 (socket says 172.21.15.98:48084) 2024-05-09T08:51:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f802e59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8030100760 0x7f8030196ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:56372/0 (socket says 172.21.15.98:56372) 2024-05-09T08:51:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f802ed9d640 1 -- 172.21.15.98:0/608436054 learned_addr learned my addr 172.21.15.98:0/608436054 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f8025d9b640 1 --2- 172.21.15.98:0/608436054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8030101120 0x7f8030197400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f802e59c640 1 -- 172.21.15.98:0/608436054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8030101120 msgr2=0x7f8030197400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f802e59c640 1 --2- 172.21.15.98:0/608436054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8030101120 0x7f8030197400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f802e59c640 1 -- 172.21.15.98:0/608436054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8030106720 msgr2=0x7f803019b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f802e59c640 1 --2- 172.21.15.98:0/608436054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8030106720 0x7f803019b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f802e59c640 1 -- 172.21.15.98:0/608436054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f803019beb0 con 0x7f8030100760 2024-05-09T08:51:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.808+0000 7f802ed9d640 1 --2- 172.21.15.98:0/608436054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8030106720 0x7f803019b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.809+0000 7f802e59c640 1 --2- 172.21.15.98:0/608436054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8030100760 0x7f8030196ec0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8020005fd0 tx=0x7f802000dd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.809+0000 7f8027fff640 1 -- 172.21.15.98:0/608436054 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8020018070 con 0x7f8030100760 2024-05-09T08:51:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.809+0000 7f8027fff640 1 -- 172.21.15.98:0/608436054 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f802000be30 con 0x7f8030100760 2024-05-09T08:51:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.809+0000 7f8034cc1640 1 -- 172.21.15.98:0/608436054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f803019c1a0 con 0x7f8030100760 2024-05-09T08:51:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.809+0000 7f8034cc1640 1 -- 172.21.15.98:0/608436054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8030068ee0 con 0x7f8030100760 2024-05-09T08:51:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.810+0000 7f8027fff640 1 -- 172.21.15.98:0/608436054 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8020012d90 con 0x7f8030100760 2024-05-09T08:51:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.812+0000 7f8027fff640 1 -- 172.21.15.98:0/608436054 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f8020025440 con 0x7f8030100760 2024-05-09T08:51:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.812+0000 7f80379cd740 1 Processor -- start 2024-05-09T08:51:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.813+0000 7f80379cd740 1 -- start start 2024-05-09T08:51:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.813+0000 7f80379cd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 0x55813749f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.813+0000 7f80379cd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55813749f580 0x5581374a9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.813+0000 7f80379cd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581374a9e40 0x5581374ac230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.813+0000 7f80379cd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5581373fb340 con 0x5581374a9e40 2024-05-09T08:51:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.813+0000 7f80379cd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5581373a3290 con 0x55813749f580 2024-05-09T08:51:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.813+0000 7f80379cd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5581373f7300 con 0x55813749ec60 2024-05-09T08:51:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.813+0000 7f8025d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581374a9e40 0x5581374ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.813+0000 7f8025d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581374a9e40 0x5581374ac230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48090/0 (socket says 172.21.15.98:48090) 2024-05-09T08:51:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.813+0000 7f8025d9b640 1 -- 172.21.15.98:0/3905147386 learned_addr learned my addr 172.21.15.98:0/3905147386 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.813+0000 7f802e59c640 1 --2- 172.21.15.98:0/3905147386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55813749f580 0x5581374a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.813+0000 7f802ed9d640 1 --2- 172.21.15.98:0/3905147386 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 0x55813749f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.814+0000 7f802e59c640 1 -- 172.21.15.98:0/3905147386 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 msgr2=0x55813749f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.814+0000 7f802e59c640 1 --2- 172.21.15.98:0/3905147386 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 0x55813749f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.814+0000 7f802e59c640 1 -- 172.21.15.98:0/3905147386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581374a9e40 msgr2=0x5581374ac230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.814+0000 7f802e59c640 1 --2- 172.21.15.98:0/3905147386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581374a9e40 0x5581374ac230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.814+0000 7f802e59c640 1 -- 172.21.15.98:0/3905147386 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581373c78e0 con 0x55813749f580 2024-05-09T08:51:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.814+0000 7f8025d9b640 1 --2- 172.21.15.98:0/3905147386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581374a9e40 0x5581374ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.814+0000 7f802e59c640 1 --2- 172.21.15.98:0/3905147386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55813749f580 0x5581374a9900 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f80200d0710 tx=0x7f80200d5a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25da79403d3376a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.814+0000 7f8024d99640 1 -- 172.21.15.98:0/3905147386 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80200d50d0 con 0x55813749f580 2024-05-09T08:51:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.814+0000 7f8024d99640 1 -- 172.21.15.98:0/3905147386 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80200d37c0 con 0x55813749f580 2024-05-09T08:51:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.814+0000 7f8024d99640 1 -- 172.21.15.98:0/3905147386 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80200117e0 con 0x55813749f580 2024-05-09T08:51:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.814+0000 7f8027fff640 1 --2- 172.21.15.98:0/608436054 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f801c0a2fd0 0x7f801c0a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.815+0000 7f802ed9d640 1 --2- 172.21.15.98:0/608436054 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f801c0a2fd0 0x7f801c0a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.815+0000 7f8027fff640 1 -- 172.21.15.98:0/608436054 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f80200d34b0 con 0x7f8030100760 2024-05-09T08:51:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.815+0000 7f80379cd740 1 -- 172.21.15.98:0/3905147386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55813749f580 msgr2=0x5581374a9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.815+0000 7f80379cd740 1 --2- 172.21.15.98:0/3905147386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55813749f580 0x5581374a9900 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f80200d0710 tx=0x7f80200d5a50 comp rx=0 tx=0).stop 2024-05-09T08:51:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.815+0000 7f802ed9d640 1 --2- 172.21.15.98:0/608436054 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f801c0a2fd0 0x7f801c0a5490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8028031860 tx=0x7f802803c040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.815+0000 7f80379cd740 1 -- 172.21.15.98:0/3905147386 shutdown_connections 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.815+0000 7f80379cd740 1 --2- 172.21.15.98:0/3905147386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581374a9e40 0x5581374ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.815+0000 7f80379cd740 1 --2- 172.21.15.98:0/3905147386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55813749f580 0x5581374a9900 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.815+0000 7f80379cd740 1 --2- 172.21.15.98:0/3905147386 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 0x55813749f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.815+0000 7f80379cd740 1 -- 172.21.15.98:0/3905147386 >> 172.21.15.98:0/3905147386 conn(0x55813749b170 msgr2=0x55813749b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.816+0000 7f80379cd740 1 -- 172.21.15.98:0/3905147386 shutdown_connections 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.816+0000 7f80379cd740 1 -- 172.21.15.98:0/3905147386 wait complete. 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.816+0000 7f80379cd740 1 -- start start 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.816+0000 7f80379cd740 1 Processor -- start 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.816+0000 7f80379cd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 0x5581374a6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.816+0000 7f80379cd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55813749f580 0x5581374a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.816+0000 7f80379cd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581374a9e40 0x5581374a78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f80379cd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5581374aca70 con 0x5581374a9e40 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f80379cd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5581373b6790 con 0x55813749f580 2024-05-09T08:51:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f80379cd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5581374acbf0 con 0x55813749ec60 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f80379cd740 10 client.0 _ll_register_callbacks cb 0x5581373f1a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f802ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581374a9e40 0x5581374a78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f802ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581374a9e40 0x5581374a78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48094/0 (socket says 172.21.15.98:48094) 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f802ed9d640 1 -- 172.21.15.98:0/1563232025 learned_addr learned my addr 172.21.15.98:0/1563232025 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f802e59c640 1 --2- 172.21.15.98:0/1563232025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 0x5581374a6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f8025d9b640 1 --2- 172.21.15.98:0/1563232025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55813749f580 0x5581374a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f8025d9b640 1 -- 172.21.15.98:0/1563232025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 msgr2=0x5581374a6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f8025d9b640 1 --2- 172.21.15.98:0/1563232025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 0x5581374a6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f8025d9b640 1 -- 172.21.15.98:0/1563232025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581374a9e40 msgr2=0x5581374a78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f8025d9b640 1 --2- 172.21.15.98:0/1563232025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581374a9e40 0x5581374a78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f8025d9b640 1 -- 172.21.15.98:0/1563232025 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55813710f210 con 0x55813749f580 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f802ed9d640 1 --2- 172.21.15.98:0/1563232025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581374a9e40 0x5581374a78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f8025d9b640 1 --2- 172.21.15.98:0/1563232025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55813749f580 0x5581374a73a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x55813749c590 tx=0x7f8018010eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be93114d58607993 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.817+0000 7f8015ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:51:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.818+0000 7f8015ffb640 1 -- 172.21.15.98:0/1563232025 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801801e060 con 0x55813749f580 2024-05-09T08:51:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.818+0000 7f80379cd740 1 -- 172.21.15.98:0/1563232025 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80200d37c0 con 0x55813749f580 2024-05-09T08:51:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.818+0000 7f8015ffb640 1 -- 172.21.15.98:0/1563232025 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f801800fb60 con 0x55813749f580 2024-05-09T08:51:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.818+0000 7f8015ffb640 1 -- 172.21.15.98:0/1563232025 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8018025050 con 0x55813749f580 2024-05-09T08:51:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.818+0000 7f80379cd740 10 client.25562 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.818+0000 7f80379cd740 1 -- 172.21.15.98:0/1563232025 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5581371db620 con 0x55813749f580 2024-05-09T08:51:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.818+0000 7f8015ffb640 1 -- 172.21.15.98:0/1563232025 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80200d37c0 con 0x55813749f580 2024-05-09T08:51:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.818+0000 7f8015ffb640 1 -- 172.21.15.98:0/1563232025 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f801800e3b0 con 0x55813749f580 2024-05-09T08:51:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.818+0000 7f80379cd740 10 client.25562 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.818+0000 7f80379cd740 10 client.25562 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.818+0000 7f80379cd740 20 client.25562 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.819+0000 7f80379cd740 1 --2- 172.21.15.98:0/1563232025 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5581374b9fa0 0x5581374bc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.819+0000 7f80379cd740 4 client.25562 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:51:53.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.819+0000 7f80379cd740 1 -- 172.21.15.98:0/1563232025 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558137408a70 con 0x5581374b9fa0 2024-05-09T08:51:53.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.819+0000 7f802e59c640 1 --2- 172.21.15.98:0/1563232025 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5581374b9fa0 0x5581374bc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.819+0000 7f8015ffb640 10 client.25562 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.819+0000 7f802e59c640 1 --2- 172.21.15.98:0/1563232025 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5581374b9fa0 0x5581374bc390 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49b9a07eccca7249 server_cookie=43818a02cd127810 in_seq=0 out_seq=0 2024-05-09T08:51:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.820+0000 7f8015ffb640 1 -- 172.21.15.98:0/1563232025 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558137408a70 con 0x5581374b9fa0 2024-05-09T08:51:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.820+0000 7f8015ffb640 10 client.25562 handle_command_reply: tid=0 2024-05-09T08:51:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.820+0000 7f80379cd740 1 client.25562 shutdown 2024-05-09T08:51:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.820+0000 7f80379cd740 1 -- 172.21.15.98:0/1563232025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55813749f580 msgr2=0x5581374a73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.820+0000 7f80379cd740 1 --2- 172.21.15.98:0/1563232025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55813749f580 0x5581374a73a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x55813749c590 tx=0x7f8018010eb0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.820+0000 7f80379cd740 1 -- 172.21.15.98:0/1563232025 shutdown_connections 2024-05-09T08:51:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.820+0000 7f80379cd740 1 --2- 172.21.15.98:0/1563232025 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5581374b9fa0 0x5581374bc390 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.820+0000 7f80379cd740 1 --2- 172.21.15.98:0/1563232025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5581374a9e40 0x5581374a78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.820+0000 7f8015ffb640 0 client.25562 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.820+0000 7f80379cd740 1 --2- 172.21.15.98:0/1563232025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55813749f580 0x5581374a73a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.820+0000 7f80379cd740 1 --2- 172.21.15.98:0/1563232025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 0x5581374a6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.820+0000 7f80379cd740 1 -- 172.21.15.98:0/1563232025 >> 172.21.15.98:0/1563232025 conn(0x55813749b170 msgr2=0x558137491a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.820+0000 7f80379cd740 1 -- 172.21.15.98:0/1563232025 shutdown_connections 2024-05-09T08:51:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.820+0000 7f80379cd740 1 -- 172.21.15.98:0/1563232025 wait complete. 2024-05-09T08:51:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.821+0000 7f80379cd740 20 client.25562 trim_cache size 0 max 16384 2024-05-09T08:51:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.850+0000 7f80379cd740 1 Processor -- start 2024-05-09T08:51:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.850+0000 7f80379cd740 1 -- start start 2024-05-09T08:51:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.850+0000 7f80379cd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55813749ec60 0x5581374b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.850+0000 7f80379cd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749f580 0x5581374a1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.850+0000 7f80379cd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5581374a9e40 0x5581374a7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.850+0000 7f80379cd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80301acb70 con 0x55813749ec60 2024-05-09T08:51:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.850+0000 7f80379cd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f80301ae6b0 con 0x5581374a9e40 2024-05-09T08:51:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.850+0000 7f80379cd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f80301ad240 con 0x55813749f580 2024-05-09T08:51:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.850+0000 7f802e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55813749ec60 0x5581374b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.850+0000 7f802e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55813749ec60 0x5581374b5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48106/0 (socket says 172.21.15.98:48106) 2024-05-09T08:51:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.850+0000 7f802e59c640 1 -- 172.21.15.98:0/3492843469 learned_addr learned my addr 172.21.15.98:0/3492843469 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.850+0000 7f802ed9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5581374a9e40 0x5581374a7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.850+0000 7f8025d9b640 1 --2- 172.21.15.98:0/3492843469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749f580 0x5581374a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.851+0000 7f802ed9d640 1 -- 172.21.15.98:0/3492843469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749f580 msgr2=0x5581374a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.851+0000 7f802ed9d640 1 --2- 172.21.15.98:0/3492843469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749f580 0x5581374a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.851+0000 7f802ed9d640 1 -- 172.21.15.98:0/3492843469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55813749ec60 msgr2=0x5581374b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.851+0000 7f802ed9d640 1 --2- 172.21.15.98:0/3492843469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55813749ec60 0x5581374b5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.851+0000 7f802ed9d640 1 -- 172.21.15.98:0/3492843469 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581374ac460 con 0x5581374a9e40 2024-05-09T08:51:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.851+0000 7f802e59c640 1 --2- 172.21.15.98:0/3492843469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55813749ec60 0x5581374b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:51:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.851+0000 7f802ed9d640 1 --2- 172.21.15.98:0/3492843469 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5581374a9e40 0x5581374a7530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8028009a60 tx=0x7f8028042000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2448a1d6a7d03405 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.851+0000 7f8017fff640 1 -- 172.21.15.98:0/3492843469 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802804c040 con 0x5581374a9e40 2024-05-09T08:51:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.851+0000 7f8017fff640 1 -- 172.21.15.98:0/3492843469 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8028057070 con 0x5581374a9e40 2024-05-09T08:51:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.851+0000 7f8017fff640 1 -- 172.21.15.98:0/3492843469 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8028052ae0 con 0x5581374a9e40 2024-05-09T08:51:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.852+0000 7f80379cd740 1 -- 172.21.15.98:0/3492843469 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5581374a9e40 msgr2=0x5581374a7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.852+0000 7f80379cd740 1 --2- 172.21.15.98:0/3492843469 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5581374a9e40 0x5581374a7530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8028009a60 tx=0x7f8028042000 comp rx=0 tx=0).stop 2024-05-09T08:51:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.852+0000 7f80379cd740 1 -- 172.21.15.98:0/3492843469 shutdown_connections 2024-05-09T08:51:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.852+0000 7f80379cd740 1 --2- 172.21.15.98:0/3492843469 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5581374a9e40 0x5581374a7530 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.852+0000 7f80379cd740 1 --2- 172.21.15.98:0/3492843469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749f580 0x5581374a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.852+0000 7f80379cd740 1 --2- 172.21.15.98:0/3492843469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55813749ec60 0x5581374b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.852+0000 7f80379cd740 1 -- 172.21.15.98:0/3492843469 >> 172.21.15.98:0/3492843469 conn(0x55813749b170 msgr2=0x5581374a6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.852+0000 7f80379cd740 1 -- 172.21.15.98:0/3492843469 shutdown_connections 2024-05-09T08:51:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.852+0000 7f80379cd740 1 -- 172.21.15.98:0/3492843469 wait complete. 2024-05-09T08:51:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.853+0000 7f80379cd740 1 -- start start 2024-05-09T08:51:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.853+0000 7f80379cd740 1 Processor -- start 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.853+0000 7f80379cd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 0x5581374a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.853+0000 7f80379cd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55813749f580 0x5581374a4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.853+0000 7f80379cd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5581374a9e40 0x55813751f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.853+0000 7f8025d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55813749f580 0x5581374a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.853+0000 7f8025d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55813749f580 0x5581374a4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48110/0 (socket says 172.21.15.98:48110) 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.853+0000 7f8025d9b640 1 -- 172.21.15.98:0/2001119705 learned_addr learned my addr 172.21.15.98:0/2001119705 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.853+0000 7f80379cd740 1 -- 172.21.15.98:0/2001119705 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80301adfe0 con 0x55813749f580 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.853+0000 7f80379cd740 1 -- 172.21.15.98:0/2001119705 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f80301ad910 con 0x5581374a9e40 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.853+0000 7f80379cd740 1 -- 172.21.15.98:0/2001119705 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f80301aab50 con 0x55813749ec60 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.853+0000 7f802e59c640 1 --2- 172.21.15.98:0/2001119705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 0x5581374a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.854+0000 7f80379cd740 10 client.0 _ll_register_callbacks cb 0x7f801800a8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.854+0000 7f802ed9d640 1 --2- 172.21.15.98:0/2001119705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5581374a9e40 0x55813751f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.854+0000 7f8025d9b640 1 -- 172.21.15.98:0/2001119705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 msgr2=0x5581374a3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.854+0000 7f8025d9b640 1 --2- 172.21.15.98:0/2001119705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 0x5581374a3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.854+0000 7f8025d9b640 1 -- 172.21.15.98:0/2001119705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5581374a9e40 msgr2=0x55813751f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:51:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.854+0000 7f8025d9b640 1 --2- 172.21.15.98:0/2001119705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5581374a9e40 0x55813751f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.854+0000 7f8025d9b640 1 -- 172.21.15.98:0/2001119705 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581374a4cf0 con 0x55813749f580 2024-05-09T08:51:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.854+0000 7f802ed9d640 1 --2- 172.21.15.98:0/2001119705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5581374a9e40 0x55813751f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:51:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.854+0000 7f80379cd740 1 -- 172.21.15.98:0/2001119705 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8028057070 con 0x55813749f580 2024-05-09T08:51:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.854+0000 7f8025d9b640 1 --2- 172.21.15.98:0/2001119705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55813749f580 0x5581374a4420 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x5581374a8430 tx=0x7f801800e640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37ce00348bd50d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:51:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.854+0000 7f8015ffb640 10 client.16215 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:51:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.854+0000 7f8015ffb640 1 -- 172.21.15.98:0/2001119705 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8018037050 con 0x55813749f580 2024-05-09T08:51:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.855+0000 7f8015ffb640 1 -- 172.21.15.98:0/2001119705 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f801800fd70 con 0x55813749f580 2024-05-09T08:51:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.855+0000 7f8015ffb640 1 -- 172.21.15.98:0/2001119705 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f801802d070 con 0x55813749f580 2024-05-09T08:51:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.855+0000 7f80379cd740 10 client.16215 fetch_fsmap learned FSMap version 29 2024-05-09T08:51:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.855+0000 7f80379cd740 1 -- 172.21.15.98:0/2001119705 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=29}) -- 0x558137520720 con 0x55813749f580 2024-05-09T08:51:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.855+0000 7f8015ffb640 1 -- 172.21.15.98:0/2001119705 <== mon.0 v2:172.21.15.98:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f801800e9c0 con 0x55813749f580 2024-05-09T08:51:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.855+0000 7f80379cd740 10 client.16215 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:51:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.855+0000 7f80379cd740 10 client.16215 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:51:53.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.855+0000 7f80379cd740 20 client.16215 populate_metadata read hostname 'smithi098' 2024-05-09T08:51:53.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.855+0000 7f80379cd740 1 --2- 172.21.15.98:0/2001119705 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5581374b9fa0 0x558137522da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:51:53.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.855+0000 7f80379cd740 4 client.16215 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:51:53.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.855+0000 7f80379cd740 1 -- 172.21.15.98:0/2001119705 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558137402dc0 con 0x5581374b9fa0 2024-05-09T08:51:53.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.856+0000 7f802e59c640 1 --2- 172.21.15.98:0/2001119705 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5581374b9fa0 0x558137522da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:51:53.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.856+0000 7f8015ffb640 10 client.16215 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:53.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.856+0000 7f802e59c640 1 --2- 172.21.15.98:0/2001119705 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5581374b9fa0 0x558137522da0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd54ab830e40ade9 server_cookie=4dc289ad86b0eb03 in_seq=0 out_seq=0 2024-05-09T08:51:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.860+0000 7f8015ffb640 1 -- 172.21.15.98:0/2001119705 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558137402dc0 con 0x5581374b9fa0 2024-05-09T08:51:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.860+0000 7f8015ffb640 10 client.16215 handle_command_reply: tid=0 2024-05-09T08:51:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.860+0000 7f80379cd740 1 client.16215 shutdown 2024-05-09T08:51:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.860+0000 7f80379cd740 1 -- 172.21.15.98:0/2001119705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55813749f580 msgr2=0x5581374a4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.860+0000 7f80379cd740 1 --2- 172.21.15.98:0/2001119705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55813749f580 0x5581374a4420 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x5581374a8430 tx=0x7f801800e640 comp rx=0 tx=0).stop 2024-05-09T08:51:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.860+0000 7f80379cd740 1 -- 172.21.15.98:0/2001119705 shutdown_connections 2024-05-09T08:51:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.860+0000 7f80379cd740 1 --2- 172.21.15.98:0/2001119705 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5581374b9fa0 0x558137522da0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.860+0000 7f80379cd740 1 --2- 172.21.15.98:0/2001119705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5581374a9e40 0x55813751f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.860+0000 7f80379cd740 1 --2- 172.21.15.98:0/2001119705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55813749f580 0x5581374a4420 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.860+0000 7f80379cd740 1 --2- 172.21.15.98:0/2001119705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55813749ec60 0x5581374a3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.860+0000 7f8015ffb640 0 client.16215 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:51:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.860+0000 7f80379cd740 1 -- 172.21.15.98:0/2001119705 >> 172.21.15.98:0/2001119705 conn(0x55813749b170 msgr2=0x558137491a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.861+0000 7f80379cd740 1 -- 172.21.15.98:0/2001119705 shutdown_connections 2024-05-09T08:51:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.861+0000 7f80379cd740 1 -- 172.21.15.98:0/2001119705 wait complete. 2024-05-09T08:51:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.861+0000 7f80379cd740 20 client.16215 trim_cache size 0 max 16384 2024-05-09T08:51:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:51:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:51:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:51:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:51:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.861+0000 7f8034cc1640 1 -- 172.21.15.98:0/608436054 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f801c0a2fd0 msgr2=0x7f801c0a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.861+0000 7f8034cc1640 1 --2- 172.21.15.98:0/608436054 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f801c0a2fd0 0x7f801c0a5490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8028031860 tx=0x7f802803c040 comp rx=0 tx=0).stop 2024-05-09T08:51:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.861+0000 7f8034cc1640 1 -- 172.21.15.98:0/608436054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8030100760 msgr2=0x7f8030196ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:51:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.861+0000 7f8034cc1640 1 --2- 172.21.15.98:0/608436054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8030100760 0x7f8030196ec0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8020005fd0 tx=0x7f802000dd20 comp rx=0 tx=0).stop 2024-05-09T08:51:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.862+0000 7f8034cc1640 1 -- 172.21.15.98:0/608436054 shutdown_connections 2024-05-09T08:51:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.862+0000 7f8034cc1640 1 --2- 172.21.15.98:0/608436054 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f801c0a2fd0 0x7f801c0a5490 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.862+0000 7f8034cc1640 1 --2- 172.21.15.98:0/608436054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8030106720 0x7f803019b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.862+0000 7f8034cc1640 1 --2- 172.21.15.98:0/608436054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8030101120 0x7f8030197400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.862+0000 7f8034cc1640 1 --2- 172.21.15.98:0/608436054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8030100760 0x7f8030196ec0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:51:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.862+0000 7f8034cc1640 1 -- 172.21.15.98:0/608436054 >> 172.21.15.98:0/608436054 conn(0x7f80300778b0 msgr2=0x7f8030103100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:51:53.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.862+0000 7f8034cc1640 1 -- 172.21.15.98:0/608436054 shutdown_connections 2024-05-09T08:51:53.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:51:53.862+0000 7f8034cc1640 1 -- 172.21.15.98:0/608436054 wait complete. 2024-05-09T08:51:53.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:53 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:51:53.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:53 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:51:53.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:53 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:51:53.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:53 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:51:53.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:53 smithi142 ceph-mon[34625]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:51:53.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:53 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:51:53.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:53 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:51:53.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:53 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:51:53.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:53 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:51:53.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:53 smithi105 ceph-mon[34498]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:51:54.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:53 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:51:54.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:53 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:51:54.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:53 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:51:54.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:53 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:51:54.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:53 smithi098 ceph-mon[30092]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:51:55.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:55 smithi142 ceph-mon[34625]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 511 B/s wr, 14 op/s 2024-05-09T08:51:55.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:55 smithi105 ceph-mon[34498]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 511 B/s wr, 14 op/s 2024-05-09T08:51:56.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:55 smithi098 ceph-mon[30092]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 511 B/s wr, 14 op/s 2024-05-09T08:51:56.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:56 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:51:56.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:56 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:51:57.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:56 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:51:57.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:57 smithi142 ceph-mon[34625]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:51:57.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:57 smithi105 ceph-mon[34498]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:51:58.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:57 smithi098 ceph-mon[30092]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:51:59.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:51:59 smithi142 ceph-mon[34625]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:51:59.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:51:59 smithi105 ceph-mon[34498]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:52:00.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:51:59 smithi098 ceph-mon[30092]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:52:01.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:01 smithi142 ceph-mon[34625]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:52:01.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:01 smithi105 ceph-mon[34498]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:52:02.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:01 smithi098 ceph-mon[30092]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:52:03.782 INFO:tasks.workunit.client.0.smithi098.stderr:Note: switching to 'be85c202969d25af6e130678e7d73a20e9c4ca11'. 2024-05-09T08:52:03.782 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-09T08:52:03.782 INFO:tasks.workunit.client.0.smithi098.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-09T08:52:03.782 INFO:tasks.workunit.client.0.smithi098.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-09T08:52:03.782 INFO:tasks.workunit.client.0.smithi098.stderr:state without impacting any branches by switching back to a branch. 2024-05-09T08:52:03.782 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-09T08:52:03.782 INFO:tasks.workunit.client.0.smithi098.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-09T08:52:03.782 INFO:tasks.workunit.client.0.smithi098.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-09T08:52:03.782 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-09T08:52:03.782 INFO:tasks.workunit.client.0.smithi098.stderr: git switch -c 2024-05-09T08:52:03.783 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-09T08:52:03.783 INFO:tasks.workunit.client.0.smithi098.stderr:Or undo this operation with: 2024-05-09T08:52:03.783 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-09T08:52:03.783 INFO:tasks.workunit.client.0.smithi098.stderr: git switch - 2024-05-09T08:52:03.783 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-09T08:52:03.783 INFO:tasks.workunit.client.0.smithi098.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-09T08:52:03.783 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-09T08:52:03.783 INFO:tasks.workunit.client.0.smithi098.stderr:HEAD is now at be85c202969 Merge pull request #57242 from cbodley/wip-65767-squid 2024-05-09T08:52:03.786 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-09T08:52:03.848 INFO:tasks.workunit.client.0.smithi098.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-09T08:52:03.850 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-09T08:52:03.851 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-09T08:52:03.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:03 smithi142 ceph-mon[34625]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:52:03.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:03 smithi105 ceph-mon[34498]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:52:04.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:03 smithi098 ceph-mon[30092]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:52:04.514 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-09T08:52:04.570 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-09T08:52:04.613 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-09T08:52:04.614 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-09T08:52:04.614 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-09T08:52:04.653 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-09T08:52:04.656 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T08:52:04.656 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-09T08:52:04.713 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-05-09T08:52:04.714 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-05-09T08:52:04.715 DEBUG:teuthology.orchestra.run.smithi098:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=be85c202969d25af6e130678e7d73a20e9c4ca11 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-05-09T08:52:04.775 INFO:tasks.workunit.client.0.smithi098.stdout:getting iogen 2024-05-09T08:52:04.775 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 'getting iogen' 2024-05-09T08:52:04.776 INFO:tasks.workunit.client.0.smithi098.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-05-09T08:52:04.800 INFO:tasks.workunit.client.0.smithi098.stderr:--2024-05-09 08:52:04-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-05-09T08:52:04.801 INFO:tasks.workunit.client.0.smithi098.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-05-09T08:52:04.857 INFO:tasks.workunit.client.0.smithi098.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-05-09T08:52:04.912 INFO:tasks.workunit.client.0.smithi098.stderr:HTTP request sent, awaiting response... 200 OK 2024-05-09T08:52:04.912 INFO:tasks.workunit.client.0.smithi098.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-05-09T08:52:04.913 INFO:tasks.workunit.client.0.smithi098.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-05-09T08:52:04.917 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-09T08:52:04.917 INFO:tasks.workunit.client.0.smithi098.stderr: 0K ....... 100% 1.24G=0s 2024-05-09T08:52:04.917 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-09T08:52:04.917 INFO:tasks.workunit.client.0.smithi098.stderr:2024-05-09 08:52:04 (1.24 GB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-05-09T08:52:04.917 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-09T08:52:04.917 INFO:tasks.workunit.client.0.smithi098.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-05-09T08:52:04.941 INFO:tasks.workunit.client.0.smithi098.stdout:iogen_3.1p0 2024-05-09T08:52:04.942 INFO:tasks.workunit.client.0.smithi098.stdout:iogen_3.1p0/src 2024-05-09T08:52:04.944 INFO:tasks.workunit.client.0.smithi098.stdout:iogen_3.1p0/src/iogen.8 2024-05-09T08:52:04.945 INFO:tasks.workunit.client.0.smithi098.stdout:iogen_3.1p0/src/iogen.c 2024-05-09T08:52:04.946 INFO:tasks.workunit.client.0.smithi098.stdout:iogen_3.1p0/README 2024-05-09T08:52:04.947 INFO:tasks.workunit.client.0.smithi098.stdout:iogen_3.1p0/Makefile.bsd 2024-05-09T08:52:04.948 INFO:tasks.workunit.client.0.smithi098.stdout:iogen_3.1p0/Makefile 2024-05-09T08:52:04.949 INFO:tasks.workunit.client.0.smithi098.stderr:+ cd iogen_3.1p0 2024-05-09T08:52:04.950 INFO:tasks.workunit.client.0.smithi098.stdout:making iogen 2024-05-09T08:52:04.950 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 'making iogen' 2024-05-09T08:52:04.950 INFO:tasks.workunit.client.0.smithi098.stderr:+ make 2024-05-09T08:52:04.958 INFO:tasks.workunit.client.0.smithi098.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-05-09T08:52:05.174 INFO:tasks.workunit.client.0.smithi098.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-05-09T08:52:05.281 INFO:tasks.workunit.client.0.smithi098.stdout:running iogen 2024-05-09T08:52:05.282 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 'running iogen' 2024-05-09T08:52:05.282 INFO:tasks.workunit.client.0.smithi098.stderr:+ ./iogen -n 5 -s 2g 2024-05-09T08:52:05.282 INFO:tasks.workunit.client.0.smithi098.stderr:child spawned 89366 2024-05-09T08:52:05.282 INFO:tasks.workunit.client.0.smithi098.stderr:child spawned 89367 2024-05-09T08:52:05.282 INFO:tasks.workunit.client.0.smithi098.stderr:child spawned 89368 2024-05-09T08:52:05.282 INFO:tasks.workunit.client.0.smithi098.stderr:child spawned 89369 2024-05-09T08:52:05.282 INFO:tasks.workunit.client.0.smithi098.stderr:child spawned 89370 2024-05-09T08:52:05.282 INFO:tasks.workunit.client.0.smithi098.stdout:sleep for 10 min 2024-05-09T08:52:05.283 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 'sleep for 10 min' 2024-05-09T08:52:05.283 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 600 2024-05-09T08:52:05.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:05 smithi142 ceph-mon[34625]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-05-09T08:52:05.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:05 smithi105 ceph-mon[34498]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-05-09T08:52:06.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:05 smithi098 ceph-mon[30092]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-05-09T08:52:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.029+0000 7f50e5afa640 1 -- 172.21.15.98:0/2232511445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50e00ffb30 msgr2=0x7f50e00fff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.029+0000 7f50e5afa640 1 --2- 172.21.15.98:0/2232511445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50e00ffb30 0x7f50e00fff90 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009900 tx=0x7f50cc02a490 comp rx=0 tx=0).stop 2024-05-09T08:52:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.030+0000 7f50e5afa640 1 -- 172.21.15.98:0/2232511445 shutdown_connections 2024-05-09T08:52:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.030+0000 7f50e5afa640 1 --2- 172.21.15.98:0/2232511445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50e01004d0 0x7f50e0105280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.030+0000 7f50e5afa640 1 --2- 172.21.15.98:0/2232511445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50e00ffb30 0x7f50e00fff90 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.030+0000 7f50e5afa640 1 --2- 172.21.15.98:0/2232511445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50e0106590 0x7f50e0106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.030+0000 7f50e5afa640 1 -- 172.21.15.98:0/2232511445 >> 172.21.15.98:0/2232511445 conn(0x7f50e0076ab0 msgr2=0x7f50e00fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.031+0000 7f50e5afa640 1 -- 172.21.15.98:0/2232511445 shutdown_connections 2024-05-09T08:52:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.031+0000 7f50e5afa640 1 -- 172.21.15.98:0/2232511445 wait complete. 2024-05-09T08:52:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.032+0000 7f50e5afa640 1 Processor -- start 2024-05-09T08:52:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.032+0000 7f50e5afa640 1 -- start start 2024-05-09T08:52:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.032+0000 7f50e5afa640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50e00ffb30 0x7f50e010eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.032+0000 7f50e5afa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50e01004d0 0x7f50e010f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.032+0000 7f50e5afa640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50e0106590 0x7f50e0113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.032+0000 7f50e5afa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50e01075f0 con 0x7f50e01004d0 2024-05-09T08:52:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.033+0000 7f50e5afa640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f50e0107470 con 0x7f50e00ffb30 2024-05-09T08:52:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.033+0000 7f50e5afa640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f50e0107770 con 0x7f50e0106590 2024-05-09T08:52:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.033+0000 7f50dffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50e01004d0 0x7f50e010f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.033+0000 7f50dffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50e01004d0 0x7f50e010f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37006/0 (socket says 172.21.15.98:37006) 2024-05-09T08:52:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.033+0000 7f50dffff640 1 -- 172.21.15.98:0/3084877947 learned_addr learned my addr 172.21.15.98:0/3084877947 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.033+0000 7f50e4af8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50e00ffb30 0x7f50e010eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.033+0000 7f50e52f9640 1 --2- 172.21.15.98:0/3084877947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50e0106590 0x7f50e0113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.034+0000 7f50dffff640 1 -- 172.21.15.98:0/3084877947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50e0106590 msgr2=0x7f50e0113560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.034+0000 7f50dffff640 1 --2- 172.21.15.98:0/3084877947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50e0106590 0x7f50e0113560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.034+0000 7f50dffff640 1 -- 172.21.15.98:0/3084877947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50e00ffb30 msgr2=0x7f50e010eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.034+0000 7f50dffff640 1 --2- 172.21.15.98:0/3084877947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50e00ffb30 0x7f50e010eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.034+0000 7f50dffff640 1 -- 172.21.15.98:0/3084877947 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e0113aa0 con 0x7f50e01004d0 2024-05-09T08:52:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.034+0000 7f50dffff640 1 --2- 172.21.15.98:0/3084877947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50e01004d0 0x7f50e010f050 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009850 tx=0x7f50cc0309c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.034+0000 7f50ddffb640 1 -- 172.21.15.98:0/3084877947 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc03f070 con 0x7f50e01004d0 2024-05-09T08:52:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.034+0000 7f50e5afa640 1 -- 172.21.15.98:0/3084877947 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50e0113d30 con 0x7f50e01004d0 2024-05-09T08:52:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.034+0000 7f50ddffb640 1 -- 172.21.15.98:0/3084877947 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50cc0035f0 con 0x7f50e01004d0 2024-05-09T08:52:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.034+0000 7f50ddffb640 1 -- 172.21.15.98:0/3084877947 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc035030 con 0x7f50e01004d0 2024-05-09T08:52:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.034+0000 7f50e5afa640 1 -- 172.21.15.98:0/3084877947 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50e0114210 con 0x7f50e01004d0 2024-05-09T08:52:07.036 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.037+0000 7f50ddffb640 1 -- 172.21.15.98:0/3084877947 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f50cc042070 con 0x7f50e01004d0 2024-05-09T08:52:07.036 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.037+0000 7f50e9a60740 1 Processor -- start 2024-05-09T08:52:07.037 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.037+0000 7f50e9a60740 1 -- start start 2024-05-09T08:52:07.037 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.038+0000 7f50e9a60740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4600 0x55eda06b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.037 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.038+0000 7f50e9a60740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06b4f20 0x55eda06bf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.037 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.038+0000 7f50e9a60740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06bf7e0 0x55eda06c1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.038+0000 7f50e9a60740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eda0619340 con 0x55eda06b4600 2024-05-09T08:52:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.038+0000 7f50e9a60740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55eda05c1290 con 0x55eda06b4f20 2024-05-09T08:52:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.038+0000 7f50e9a60740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55eda0615300 con 0x55eda06bf7e0 2024-05-09T08:52:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.038+0000 7f50e52f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4600 0x55eda06b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.038+0000 7f50ddffb640 1 --2- 172.21.15.98:0/3084877947 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50d00a2ea0 0x7f50d00a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.038+0000 7f50e52f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4600 0x55eda06b49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37022/0 (socket says 172.21.15.98:37022) 2024-05-09T08:52:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.038+0000 7f50e52f9640 1 -- 172.21.15.98:0/576028701 learned_addr learned my addr 172.21.15.98:0/576028701 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.038+0000 7f50ddffb640 1 -- 172.21.15.98:0/3084877947 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f50cc0f08c0 con 0x7f50e01004d0 2024-05-09T08:52:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.038+0000 7f50e4af8640 1 --2- 172.21.15.98:0/576028701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06b4f20 0x55eda06bf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.038+0000 7f50e52f9640 1 --2- 172.21.15.98:0/3084877947 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50d00a2ea0 0x7f50d00a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.039+0000 7f50dffff640 1 --2- 172.21.15.98:0/576028701 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06bf7e0 0x55eda06c1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.039+0000 7f50e52f9640 1 -- 172.21.15.98:0/576028701 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06bf7e0 msgr2=0x55eda06c1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.039+0000 7f50e52f9640 1 --2- 172.21.15.98:0/576028701 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06bf7e0 0x55eda06c1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.039+0000 7f50e52f9640 1 -- 172.21.15.98:0/576028701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06b4f20 msgr2=0x55eda06bf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.039+0000 7f50e52f9640 1 --2- 172.21.15.98:0/576028701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06b4f20 0x55eda06bf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.039+0000 7f50e52f9640 1 -- 172.21.15.98:0/576028701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eda05e58e0 con 0x55eda06b4600 2024-05-09T08:52:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.039+0000 7f50e52f9640 1 --2- 172.21.15.98:0/576028701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4600 0x55eda06b49e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f50d800cb20 tx=0x7f50d80072b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89c4dbab8f7a18d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.040+0000 7f50e52f9640 1 --2- 172.21.15.98:0/3084877947 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50d00a2ea0 0x7f50d00a5360 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f50d80077b0 tx=0x7f50d8019000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.041+0000 7f50caffd640 1 -- 172.21.15.98:0/576028701 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d8015070 con 0x55eda06b4600 2024-05-09T08:52:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.042+0000 7f50caffd640 1 -- 172.21.15.98:0/576028701 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50d8016040 con 0x55eda06b4600 2024-05-09T08:52:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.042+0000 7f50caffd640 1 -- 172.21.15.98:0/576028701 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d801ab10 con 0x55eda06b4600 2024-05-09T08:52:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.042+0000 7f50e9a60740 1 -- 172.21.15.98:0/576028701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4600 msgr2=0x55eda06b49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.042+0000 7f50e9a60740 1 --2- 172.21.15.98:0/576028701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4600 0x55eda06b49e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f50d800cb20 tx=0x7f50d80072b0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.043+0000 7f50e9a60740 1 -- 172.21.15.98:0/576028701 shutdown_connections 2024-05-09T08:52:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.043+0000 7f50e9a60740 1 --2- 172.21.15.98:0/576028701 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06bf7e0 0x55eda06c1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.043+0000 7f50e9a60740 1 --2- 172.21.15.98:0/576028701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06b4f20 0x55eda06bf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.043+0000 7f50e9a60740 1 --2- 172.21.15.98:0/576028701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4600 0x55eda06b49e0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.043+0000 7f50e9a60740 1 -- 172.21.15.98:0/576028701 >> 172.21.15.98:0/576028701 conn(0x55eda06b0b10 msgr2=0x55eda06b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.043+0000 7f50e9a60740 1 -- 172.21.15.98:0/576028701 shutdown_connections 2024-05-09T08:52:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.043+0000 7f50e9a60740 1 -- 172.21.15.98:0/576028701 wait complete. 2024-05-09T08:52:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.043+0000 7f50e9a60740 1 -- start start 2024-05-09T08:52:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.043+0000 7f50e9a60740 1 Processor -- start 2024-05-09T08:52:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.044+0000 7f50e9a60740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06b4600 0x55eda06bc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.044+0000 7f50e9a60740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06b4f20 0x55eda06bcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.044+0000 7f50e9a60740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06bf7e0 0x55eda06bd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.044+0000 7f50e9a60740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eda06c23b0 con 0x55eda06bf7e0 2024-05-09T08:52:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.044+0000 7f50e9a60740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55eda05d4790 con 0x55eda06b4f20 2024-05-09T08:52:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.044+0000 7f50e9a60740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55eda06c2530 con 0x55eda06b4600 2024-05-09T08:52:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.045+0000 7f50e9a60740 10 client.0 _ll_register_callbacks cb 0x55eda060fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.045+0000 7f50e52f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06bf7e0 0x55eda06bd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.045+0000 7f50e52f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06bf7e0 0x55eda06bd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37036/0 (socket says 172.21.15.98:37036) 2024-05-09T08:52:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.045+0000 7f50e52f9640 1 -- 172.21.15.98:0/3653902236 learned_addr learned my addr 172.21.15.98:0/3653902236 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.045+0000 7f50e4af8640 1 --2- 172.21.15.98:0/3653902236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06b4600 0x55eda06bc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.045+0000 7f50dffff640 1 --2- 172.21.15.98:0/3653902236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06b4f20 0x55eda06bcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.046+0000 7f50e52f9640 1 -- 172.21.15.98:0/3653902236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06b4600 msgr2=0x55eda06bc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.046+0000 7f50e52f9640 1 --2- 172.21.15.98:0/3653902236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06b4600 0x55eda06bc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.046+0000 7f50e52f9640 1 -- 172.21.15.98:0/3653902236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06b4f20 msgr2=0x55eda06bcd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.046+0000 7f50e52f9640 1 --2- 172.21.15.98:0/3653902236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06b4f20 0x55eda06bcd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.046+0000 7f50e52f9640 1 -- 172.21.15.98:0/3653902236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eda032d210 con 0x55eda06bf7e0 2024-05-09T08:52:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.047+0000 7f50e52f9640 1 --2- 172.21.15.98:0/3653902236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06bf7e0 0x55eda06bd2c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f50d801ac10 tx=0x7f50d8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b32e546e8419ed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.047+0000 7f50bffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:52:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.047+0000 7f50bffff640 1 -- 172.21.15.98:0/3653902236 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d802f060 con 0x55eda06bf7e0 2024-05-09T08:52:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.047+0000 7f50bffff640 1 -- 172.21.15.98:0/3653902236 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f50d800aa90 con 0x55eda06bf7e0 2024-05-09T08:52:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.047+0000 7f50e9a60740 1 -- 172.21.15.98:0/3653902236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50d8016040 con 0x55eda06bf7e0 2024-05-09T08:52:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.047+0000 7f50bffff640 1 -- 172.21.15.98:0/3653902236 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f50d8015050 con 0x55eda06bf7e0 2024-05-09T08:52:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.047+0000 7f50e9a60740 10 client.16227 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.048+0000 7f50e9a60740 1 -- 172.21.15.98:0/3653902236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55eda03f9620 con 0x55eda06bf7e0 2024-05-09T08:52:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.048+0000 7f50bffff640 1 -- 172.21.15.98:0/3653902236 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50d8016040 con 0x55eda06bf7e0 2024-05-09T08:52:07.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.048+0000 7f50bffff640 1 -- 172.21.15.98:0/3653902236 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f50d8029030 con 0x55eda06bf7e0 2024-05-09T08:52:07.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.049+0000 7f50e9a60740 10 client.16227 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:07.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.049+0000 7f50e9a60740 10 client.16227 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.049+0000 7f50e9a60740 20 client.16227 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.049+0000 7f50e9a60740 1 --2- 172.21.15.98:0/3653902236 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eda06cf4b0 0x55eda06d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.049+0000 7f50e9a60740 4 client.16227 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:52:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.049+0000 7f50e9a60740 1 -- 172.21.15.98:0/3653902236 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eda0626a70 con 0x55eda06cf4b0 2024-05-09T08:52:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.050+0000 7f50e4af8640 1 --2- 172.21.15.98:0/3653902236 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eda06cf4b0 0x55eda06d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.059+0000 7f50bffff640 10 client.16227 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.059+0000 7f50e4af8640 1 --2- 172.21.15.98:0/3653902236 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eda06cf4b0 0x55eda06d18a0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e21b187bb9600043 server_cookie=959e9c24e9c276b6 in_seq=0 out_seq=0 2024-05-09T08:52:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.070+0000 7f50bffff640 1 -- 172.21.15.98:0/3653902236 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eda0626a70 con 0x55eda06cf4b0 2024-05-09T08:52:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.070+0000 7f50bffff640 10 client.16227 handle_command_reply: tid=0 2024-05-09T08:52:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.070+0000 7f50e9a60740 1 client.16227 shutdown 2024-05-09T08:52:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.077+0000 7f50e9a60740 1 -- 172.21.15.98:0/3653902236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06bf7e0 msgr2=0x55eda06bd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.077+0000 7f50e9a60740 1 --2- 172.21.15.98:0/3653902236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06bf7e0 0x55eda06bd2c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f50d801ac10 tx=0x7f50d8009750 comp rx=0 tx=0).stop 2024-05-09T08:52:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.080+0000 7f50e9a60740 1 -- 172.21.15.98:0/3653902236 shutdown_connections 2024-05-09T08:52:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.080+0000 7f50e9a60740 1 --2- 172.21.15.98:0/3653902236 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eda06cf4b0 0x55eda06d18a0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.080+0000 7f50bffff640 0 client.16227 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.080+0000 7f50e9a60740 1 --2- 172.21.15.98:0/3653902236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06bf7e0 0x55eda06bd2c0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.080+0000 7f50e9a60740 1 --2- 172.21.15.98:0/3653902236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06b4f20 0x55eda06bcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.080+0000 7f50e9a60740 1 --2- 172.21.15.98:0/3653902236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06b4600 0x55eda06bc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.080+0000 7f50e9a60740 1 -- 172.21.15.98:0/3653902236 >> 172.21.15.98:0/3653902236 conn(0x55eda06b0b10 msgr2=0x55eda06b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:07.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.085+0000 7f50e9a60740 1 -- 172.21.15.98:0/3653902236 shutdown_connections 2024-05-09T08:52:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.085+0000 7f50e9a60740 1 -- 172.21.15.98:0/3653902236 wait complete. 2024-05-09T08:52:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.085+0000 7f50e9a60740 20 client.16227 trim_cache size 0 max 16384 2024-05-09T08:52:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.119+0000 7f50e9a60740 1 Processor -- start 2024-05-09T08:52:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.119+0000 7f50e9a60740 1 -- start start 2024-05-09T08:52:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50e9a60740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06b4600 0x55eda06cb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50e9a60740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4f20 0x55eda06bb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50e9a60740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06bf7e0 0x55eda06b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50e9a60740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50e01bd7f0 con 0x55eda06b4f20 2024-05-09T08:52:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50e9a60740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f50e01bf330 con 0x55eda06bf7e0 2024-05-09T08:52:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50e9a60740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f50e01bdec0 con 0x55eda06b4600 2024-05-09T08:52:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50dffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4f20 0x55eda06bb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50dffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4f20 0x55eda06bb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37046/0 (socket says 172.21.15.98:37046) 2024-05-09T08:52:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50dffff640 1 -- 172.21.15.98:0/2651430361 learned_addr learned my addr 172.21.15.98:0/2651430361 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50dffff640 1 -- 172.21.15.98:0/2651430361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06b4600 msgr2=0x55eda06cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50dffff640 1 --2- 172.21.15.98:0/2651430361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06b4600 0x55eda06cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50dffff640 1 -- 172.21.15.98:0/2651430361 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06bf7e0 msgr2=0x55eda06b6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50dffff640 1 --2- 172.21.15.98:0/2651430361 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06bf7e0 0x55eda06b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50dffff640 1 -- 172.21.15.98:0/2651430361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eda06b7500 con 0x55eda06b4f20 2024-05-09T08:52:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.120+0000 7f50dffff640 1 --2- 172.21.15.98:0/2651430361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4f20 0x55eda06bb9a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0bc770 tx=0x7f50cc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab205e1f07b159c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.121+0000 7f50ca7fc640 1 -- 172.21.15.98:0/2651430361 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc049040 con 0x55eda06b4f20 2024-05-09T08:52:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.121+0000 7f50ca7fc640 1 -- 172.21.15.98:0/2651430361 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50cc0bcee0 con 0x55eda06b4f20 2024-05-09T08:52:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.121+0000 7f50ca7fc640 1 -- 172.21.15.98:0/2651430361 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc0b8e50 con 0x55eda06b4f20 2024-05-09T08:52:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.121+0000 7f50e9a60740 1 -- 172.21.15.98:0/2651430361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4f20 msgr2=0x55eda06bb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.121+0000 7f50e9a60740 1 --2- 172.21.15.98:0/2651430361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4f20 0x55eda06bb9a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0bc770 tx=0x7f50cc005fd0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.121+0000 7f50e9a60740 1 -- 172.21.15.98:0/2651430361 shutdown_connections 2024-05-09T08:52:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.122+0000 7f50e9a60740 1 --2- 172.21.15.98:0/2651430361 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06bf7e0 0x55eda06b6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.122+0000 7f50e9a60740 1 --2- 172.21.15.98:0/2651430361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4f20 0x55eda06bb9a0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.122+0000 7f50e9a60740 1 --2- 172.21.15.98:0/2651430361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06b4600 0x55eda06cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.122+0000 7f50e9a60740 1 -- 172.21.15.98:0/2651430361 >> 172.21.15.98:0/2651430361 conn(0x55eda06b0b10 msgr2=0x55eda06af320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.122+0000 7f50e9a60740 1 -- 172.21.15.98:0/2651430361 shutdown_connections 2024-05-09T08:52:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.122+0000 7f50e9a60740 1 -- 172.21.15.98:0/2651430361 wait complete. 2024-05-09T08:52:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.122+0000 7f50e9a60740 1 -- start start 2024-05-09T08:52:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.122+0000 7f50e9a60740 1 Processor -- start 2024-05-09T08:52:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e9a60740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4600 0x55eda06c7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e9a60740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06b4f20 0x55eda06c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e4af8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4600 0x55eda06c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e9a60740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06bf7e0 0x55eda06c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e9a60740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50e01bec60 con 0x55eda06b4600 2024-05-09T08:52:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e4af8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4600 0x55eda06c7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37054/0 (socket says 172.21.15.98:37054) 2024-05-09T08:52:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e4af8640 1 -- 172.21.15.98:0/1910057255 learned_addr learned my addr 172.21.15.98:0/1910057255 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e9a60740 1 -- 172.21.15.98:0/1910057255 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f50e01be590 con 0x55eda06bf7e0 2024-05-09T08:52:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e9a60740 1 -- 172.21.15.98:0/1910057255 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f50e01bb7d0 con 0x55eda06b4f20 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e9a60740 10 client.0 _ll_register_callbacks cb 0x7f50d800c8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e4af8640 1 -- 172.21.15.98:0/1910057255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06b4f20 msgr2=0x55eda06c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e4af8640 1 --2- 172.21.15.98:0/1910057255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06b4f20 0x55eda06c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e4af8640 1 -- 172.21.15.98:0/1910057255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06bf7e0 msgr2=0x55eda06c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e4af8640 1 --2- 172.21.15.98:0/1910057255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06bf7e0 0x55eda06c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.123+0000 7f50e4af8640 1 -- 172.21.15.98:0/1910057255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eda06c3470 con 0x55eda06b4600 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.124+0000 7f50e4af8640 1 --2- 172.21.15.98:0/1910057255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4600 0x55eda06c7710 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009330 tx=0x7f50d4017810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989107dc50f48347 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.124+0000 7f50bffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.124+0000 7f50bffff640 1 -- 172.21.15.98:0/1910057255 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d400f040 con 0x55eda06b4600 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.124+0000 7f50bffff640 1 -- 172.21.15.98:0/1910057255 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f50d4025630 con 0x55eda06b4600 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.124+0000 7f50e9a60740 1 -- 172.21.15.98:0/1910057255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50cc0bcee0 con 0x55eda06b4600 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.124+0000 7f50bffff640 1 -- 172.21.15.98:0/1910057255 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f50d4025830 con 0x55eda06b4600 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.124+0000 7f50e9a60740 10 client.16233 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.124+0000 7f50e9a60740 1 -- 172.21.15.98:0/1910057255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55eda07354a0 con 0x55eda06b4600 2024-05-09T08:52:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.125+0000 7f50bffff640 1 -- 172.21.15.98:0/1910057255 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50cc0bcee0 con 0x55eda06b4600 2024-05-09T08:52:07.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.125+0000 7f50bffff640 1 -- 172.21.15.98:0/1910057255 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f50d402fbb0 con 0x55eda06b4600 2024-05-09T08:52:07.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.125+0000 7f50e9a60740 10 client.16233 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:07.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.125+0000 7f50e9a60740 10 client.16233 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:07.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.125+0000 7f50e9a60740 20 client.16233 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:07.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.125+0000 7f50e9a60740 1 --2- 172.21.15.98:0/1910057255 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eda06cf4b0 0x55eda0737c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:07.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.125+0000 7f50e9a60740 4 client.16233 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:52:07.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.125+0000 7f50e9a60740 1 -- 172.21.15.98:0/1910057255 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eda0620dc0 con 0x55eda06cf4b0 2024-05-09T08:52:07.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.126+0000 7f50dffff640 1 --2- 172.21.15.98:0/1910057255 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eda06cf4b0 0x55eda0737c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:07.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.127+0000 7f50bffff640 10 client.16233 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:07.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.127+0000 7f50dffff640 1 --2- 172.21.15.98:0/1910057255 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eda06cf4b0 0x55eda0737c60 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c541fd6249bfbf27 server_cookie=e65ebc3a91b816d9 in_seq=0 out_seq=0 2024-05-09T08:52:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.312+0000 7f50bffff640 1 -- 172.21.15.98:0/1910057255 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eda0620dc0 con 0x55eda06cf4b0 2024-05-09T08:52:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.312+0000 7f50bffff640 10 client.16233 handle_command_reply: tid=0 2024-05-09T08:52:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.312+0000 7f50e9a60740 1 client.16233 shutdown 2024-05-09T08:52:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.314+0000 7f50e9a60740 1 -- 172.21.15.98:0/1910057255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4600 msgr2=0x55eda06c7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.314+0000 7f50e9a60740 1 --2- 172.21.15.98:0/1910057255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4600 0x55eda06c7710 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009330 tx=0x7f50d4017810 comp rx=0 tx=0).stop 2024-05-09T08:52:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.314+0000 7f50e9a60740 1 -- 172.21.15.98:0/1910057255 shutdown_connections 2024-05-09T08:52:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.314+0000 7f50e9a60740 1 --2- 172.21.15.98:0/1910057255 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eda06cf4b0 0x55eda0737c60 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.314+0000 7f50e9a60740 1 --2- 172.21.15.98:0/1910057255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eda06bf7e0 0x55eda06c2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.314+0000 7f50e9a60740 1 --2- 172.21.15.98:0/1910057255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eda06b4f20 0x55eda06c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.314+0000 7f50e9a60740 1 --2- 172.21.15.98:0/1910057255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eda06b4600 0x55eda06c7710 secure :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009330 tx=0x7f50d4017810 comp rx=0 tx=0).stop 2024-05-09T08:52:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.314+0000 7f50e9a60740 1 -- 172.21.15.98:0/1910057255 >> 172.21.15.98:0/1910057255 conn(0x55eda06b0b10 msgr2=0x55eda06bc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.314+0000 7f50bffff640 0 client.16233 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.314+0000 7f50e9a60740 1 -- 172.21.15.98:0/1910057255 shutdown_connections 2024-05-09T08:52:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.315+0000 7f50e9a60740 1 -- 172.21.15.98:0/1910057255 wait complete. 2024-05-09T08:52:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.315+0000 7f50e9a60740 20 client.16233 trim_cache size 0 max 16384 2024-05-09T08:52:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:52:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:52:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:52:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:52:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.316+0000 7f50e5afa640 1 -- 172.21.15.98:0/3084877947 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50d00a2ea0 msgr2=0x7f50d00a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.316+0000 7f50e5afa640 1 --2- 172.21.15.98:0/3084877947 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50d00a2ea0 0x7f50d00a5360 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f50d80077b0 tx=0x7f50d8019000 comp rx=0 tx=0).stop 2024-05-09T08:52:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.317+0000 7f50e5afa640 1 -- 172.21.15.98:0/3084877947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50e01004d0 msgr2=0x7f50e010f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.317+0000 7f50e5afa640 1 --2- 172.21.15.98:0/3084877947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50e01004d0 0x7f50e010f050 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009850 tx=0x7f50cc0309c0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.317+0000 7f50e5afa640 1 -- 172.21.15.98:0/3084877947 shutdown_connections 2024-05-09T08:52:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.317+0000 7f50e5afa640 1 --2- 172.21.15.98:0/3084877947 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f50d00a2ea0 0x7f50d00a5360 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.317+0000 7f50e5afa640 1 --2- 172.21.15.98:0/3084877947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50e0106590 0x7f50e0113560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.317+0000 7f50e5afa640 1 --2- 172.21.15.98:0/3084877947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50e01004d0 0x7f50e010f050 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.317+0000 7f50e5afa640 1 --2- 172.21.15.98:0/3084877947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50e00ffb30 0x7f50e010eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:07.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.317+0000 7f50e5afa640 1 -- 172.21.15.98:0/3084877947 >> 172.21.15.98:0/3084877947 conn(0x7f50e0076ab0 msgr2=0x7f50e0103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:07.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.319+0000 7f50e5afa640 1 -- 172.21.15.98:0/3084877947 shutdown_connections 2024-05-09T08:52:07.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:07.319+0000 7f50e5afa640 1 -- 172.21.15.98:0/3084877947 wait complete. 2024-05-09T08:52:07.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:07 smithi142 ceph-mon[34625]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-09T08:52:07.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:07 smithi105 ceph-mon[34498]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-09T08:52:08.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:07 smithi098 ceph-mon[30092]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-09T08:52:09.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:09 smithi142 ceph-mon[34625]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-09T08:52:09.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:09 smithi105 ceph-mon[34498]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-09T08:52:10.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:09 smithi098 ceph-mon[30092]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-09T08:52:11.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:11 smithi142 ceph-mon[34625]: pgmap v49: 129 pgs: 129 active+clean; 3.6 GiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 309 MiB/s wr, 253 op/s 2024-05-09T08:52:11.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:11 smithi142 ceph-mon[34625]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-05-09T08:52:11.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:11 smithi105 ceph-mon[34498]: pgmap v49: 129 pgs: 129 active+clean; 3.6 GiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 309 MiB/s wr, 253 op/s 2024-05-09T08:52:11.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:11 smithi105 ceph-mon[34498]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-05-09T08:52:11.988 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:11 smithi098 ceph-mon[30092]: pgmap v49: 129 pgs: 129 active+clean; 3.6 GiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 309 MiB/s wr, 253 op/s 2024-05-09T08:52:11.988 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:11 smithi098 ceph-mon[30092]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-05-09T08:52:13.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:13 smithi142 ceph-mon[34625]: pgmap v50: 129 pgs: 129 active+clean; 3.6 GiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 309 MiB/s wr, 248 op/s 2024-05-09T08:52:13.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:13 smithi105 ceph-mon[34498]: pgmap v50: 129 pgs: 129 active+clean; 3.6 GiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 309 MiB/s wr, 248 op/s 2024-05-09T08:52:14.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:13 smithi098 ceph-mon[30092]: pgmap v50: 129 pgs: 129 active+clean; 3.6 GiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 309 MiB/s wr, 248 op/s 2024-05-09T08:52:15.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:15 smithi142 ceph-mon[34625]: pgmap v51: 129 pgs: 129 active+clean; 5.4 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 469 MiB/s wr, 560 op/s 2024-05-09T08:52:15.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:15 smithi105 ceph-mon[34498]: pgmap v51: 129 pgs: 129 active+clean; 5.4 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 469 MiB/s wr, 560 op/s 2024-05-09T08:52:16.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:15 smithi098 ceph-mon[30092]: pgmap v51: 129 pgs: 129 active+clean; 5.4 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 469 MiB/s wr, 560 op/s 2024-05-09T08:52:17.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:17 smithi105 ceph-mon[34498]: pgmap v52: 129 pgs: 129 active+clean; 8.2 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 730 MiB/s wr, 826 op/s 2024-05-09T08:52:18.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:17 smithi098 ceph-mon[30092]: pgmap v52: 129 pgs: 129 active+clean; 8.2 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 730 MiB/s wr, 826 op/s 2024-05-09T08:52:18.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:17 smithi142 ceph-mon[34625]: pgmap v52: 129 pgs: 129 active+clean; 8.2 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 730 MiB/s wr, 826 op/s 2024-05-09T08:52:20.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:19 smithi098 ceph-mon[30092]: pgmap v53: 129 pgs: 129 active+clean; 8.2 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 730 MiB/s wr, 826 op/s 2024-05-09T08:52:20.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:19 smithi142 ceph-mon[34625]: pgmap v53: 129 pgs: 129 active+clean; 8.2 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 730 MiB/s wr, 826 op/s 2024-05-09T08:52:20.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:19 smithi105 ceph-mon[34498]: pgmap v53: 129 pgs: 129 active+clean; 8.2 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 730 MiB/s wr, 826 op/s 2024-05-09T08:52:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.592+0000 7f2b0aea0640 1 -- 172.21.15.98:0/923652076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b041102d0 msgr2=0x7f2b0410dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.592+0000 7f2b0aea0640 1 --2- 172.21.15.98:0/923652076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b041102d0 0x7f2b0410dee0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009900 tx=0x7f2b0002a4a0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.592+0000 7f2b0aea0640 1 -- 172.21.15.98:0/923652076 shutdown_connections 2024-05-09T08:52:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.592+0000 7f2b0aea0640 1 --2- 172.21.15.98:0/923652076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b0410e500 0x7f2b0411d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.592+0000 7f2b0aea0640 1 --2- 172.21.15.98:0/923652076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b041102d0 0x7f2b0410dee0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.592+0000 7f2b0aea0640 1 --2- 172.21.15.98:0/923652076 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b0410fef0 0x7f2b0410d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.592+0000 7f2b0aea0640 1 -- 172.21.15.98:0/923652076 >> 172.21.15.98:0/923652076 conn(0x7f2b0406c770 msgr2=0x7f2b0406cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.593+0000 7f2b0aea0640 1 -- 172.21.15.98:0/923652076 shutdown_connections 2024-05-09T08:52:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.595+0000 7f2b0aea0640 1 -- 172.21.15.98:0/923652076 wait complete. 2024-05-09T08:52:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.596+0000 7f2b0aea0640 1 Processor -- start 2024-05-09T08:52:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.597+0000 7f2b0aea0640 1 -- start start 2024-05-09T08:52:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.597+0000 7f2b0aea0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b0410e500 0x7f2b0411cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.597+0000 7f2b0aea0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b0410fef0 0x7f2b04116d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.597+0000 7f2b0aea0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b041102d0 0x7f2b041172b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.597+0000 7f2b0aea0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b04111220 con 0x7f2b0410fef0 2024-05-09T08:52:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.597+0000 7f2b0aea0640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2b041110a0 con 0x7f2b041102d0 2024-05-09T08:52:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.597+0000 7f2b0aea0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2b041113a0 con 0x7f2b0410e500 2024-05-09T08:52:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.600+0000 7f2b09e9e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b0410e500 0x7f2b0411cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.600+0000 7f2b0969d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b0410fef0 0x7f2b04116d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.600+0000 7f2b0969d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b0410fef0 0x7f2b04116d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50620/0 (socket says 172.21.15.98:50620) 2024-05-09T08:52:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.600+0000 7f2b0969d640 1 -- 172.21.15.98:0/1463818106 learned_addr learned my addr 172.21.15.98:0/1463818106 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:20.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.601+0000 7f2b0a69f640 1 --2- 172.21.15.98:0/1463818106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b041102d0 0x7f2b041172b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.602+0000 7f2b0969d640 1 -- 172.21.15.98:0/1463818106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b0410e500 msgr2=0x7f2b0411cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.602+0000 7f2b0969d640 1 --2- 172.21.15.98:0/1463818106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b0410e500 0x7f2b0411cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.602+0000 7f2b0969d640 1 -- 172.21.15.98:0/1463818106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b041102d0 msgr2=0x7f2b041172b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.602+0000 7f2b0969d640 1 --2- 172.21.15.98:0/1463818106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b041102d0 0x7f2b041172b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.602+0000 7f2b0969d640 1 -- 172.21.15.98:0/1463818106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b04117a30 con 0x7f2b0410fef0 2024-05-09T08:52:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.607+0000 7f2b0969d640 1 --2- 172.21.15.98:0/1463818106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b0410fef0 0x7f2b04116d70 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f2b00004650 tx=0x7f2b000052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.609+0000 7f2afaffd640 1 -- 172.21.15.98:0/1463818106 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b00040030 con 0x7f2b0410fef0 2024-05-09T08:52:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.609+0000 7f2b0aea0640 1 -- 172.21.15.98:0/1463818106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b041bbbf0 con 0x7f2b0410fef0 2024-05-09T08:52:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.609+0000 7f2b0aea0640 1 -- 172.21.15.98:0/1463818106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b041bc0b0 con 0x7f2b0410fef0 2024-05-09T08:52:20.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.613+0000 7f2b0ee05740 1 Processor -- start 2024-05-09T08:52:20.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.613+0000 7f2afaffd640 1 -- 172.21.15.98:0/1463818106 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b0003c920 con 0x7f2b0410fef0 2024-05-09T08:52:20.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.614+0000 7f2afaffd640 1 -- 172.21.15.98:0/1463818106 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b00040030 con 0x7f2b0410fef0 2024-05-09T08:52:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.617+0000 7f2b0ee05740 1 -- start start 2024-05-09T08:52:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.618+0000 7f2b0ee05740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd2459e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.618+0000 7f2b0ee05740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd245f20 0x5625bd2502a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.618+0000 7f2b0ee05740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd2507e0 0x5625bd252bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.618+0000 7f2b0ee05740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5625bd1aa340 con 0x5625bd245600 2024-05-09T08:52:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.618+0000 7f2b0ee05740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5625bd152290 con 0x5625bd2507e0 2024-05-09T08:52:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.618+0000 7f2b0ee05740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5625bd1a6300 con 0x5625bd245f20 2024-05-09T08:52:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.619+0000 7f2b0a69f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd2459e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.619+0000 7f2b0a69f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd2459e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50636/0 (socket says 172.21.15.98:50636) 2024-05-09T08:52:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.619+0000 7f2b0a69f640 1 -- 172.21.15.98:0/1807064774 learned_addr learned my addr 172.21.15.98:0/1807064774 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.619+0000 7f2b0a69f640 1 -- 172.21.15.98:0/1807064774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd245f20 msgr2=0x5625bd2502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.619+0000 7f2b0a69f640 1 --2- 172.21.15.98:0/1807064774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd245f20 0x5625bd2502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.619+0000 7f2b0a69f640 1 -- 172.21.15.98:0/1807064774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd2507e0 msgr2=0x5625bd252bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.619+0000 7f2b0a69f640 1 --2- 172.21.15.98:0/1807064774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd2507e0 0x5625bd252bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.619+0000 7f2b0a69f640 1 -- 172.21.15.98:0/1807064774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625bd1768e0 con 0x5625bd245600 2024-05-09T08:52:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.620+0000 7f2b0a69f640 1 --2- 172.21.15.98:0/1807064774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd2459e0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2afc009fa0 tx=0x7f2afc010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b640c271523b1172 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.625+0000 7f2afaffd640 1 -- 172.21.15.98:0/1463818106 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2b00043030 con 0x7f2b0410fef0 2024-05-09T08:52:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.626+0000 7f2afaffd640 1 --2- 172.21.15.98:0/1463818106 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2af00a30f0 0x7f2af00a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.627+0000 7f2afaffd640 1 -- 172.21.15.98:0/1463818106 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f2b00041070 con 0x7f2b0410fef0 2024-05-09T08:52:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.628+0000 7f2aeffff640 1 -- 172.21.15.98:0/1807064774 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc003b70 con 0x5625bd245600 2024-05-09T08:52:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.628+0000 7f2aeffff640 1 -- 172.21.15.98:0/1807064774 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2afc003d10 con 0x5625bd245600 2024-05-09T08:52:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.630+0000 7f2b0ee05740 1 -- 172.21.15.98:0/1807064774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 msgr2=0x5625bd2459e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.630+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/1807064774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd2459e0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2afc009fa0 tx=0x7f2afc010860 comp rx=0 tx=0).stop 2024-05-09T08:52:20.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.632+0000 7f2b09e9e640 1 --2- 172.21.15.98:0/1463818106 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2af00a30f0 0x7f2af00a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:20.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.632+0000 7f2b0ee05740 1 -- 172.21.15.98:0/1807064774 shutdown_connections 2024-05-09T08:52:20.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.632+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/1807064774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd2507e0 0x5625bd252bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.632+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/1807064774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd245f20 0x5625bd2502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.632+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/1807064774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd2459e0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.632+0000 7f2b0ee05740 1 -- 172.21.15.98:0/1807064774 >> 172.21.15.98:0/1807064774 conn(0x5625bd241b10 msgr2=0x5625bd241ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:20.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.632+0000 7f2b0ee05740 1 -- 172.21.15.98:0/1807064774 shutdown_connections 2024-05-09T08:52:20.636 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.635+0000 7f2b0ee05740 1 -- 172.21.15.98:0/1807064774 wait complete. 2024-05-09T08:52:20.636 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.635+0000 7f2b0ee05740 1 -- start start 2024-05-09T08:52:20.636 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.636+0000 7f2b0ee05740 1 Processor -- start 2024-05-09T08:52:20.636 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.637+0000 7f2b0ee05740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd24d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.636 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.637+0000 7f2b0ee05740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd245f20 0x5625bd24dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.637+0000 7f2b0ee05740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd2507e0 0x5625bd24e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.637+0000 7f2b0ee05740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5625bd253410 con 0x5625bd245600 2024-05-09T08:52:20.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.637+0000 7f2b0ee05740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5625bd165790 con 0x5625bd245f20 2024-05-09T08:52:20.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.637+0000 7f2b0ee05740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5625bd253590 con 0x5625bd2507e0 2024-05-09T08:52:20.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.637+0000 7f2b0ee05740 10 client.0 _ll_register_callbacks cb 0x5625bd1a0a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:20.638 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.637+0000 7f2b0969d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd245f20 0x5625bd24dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:20.638 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.638+0000 7f2b0969d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd245f20 0x5625bd24dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:43886/0 (socket says 172.21.15.98:43886) 2024-05-09T08:52:20.638 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.638+0000 7f2b0969d640 1 -- 172.21.15.98:0/303124928 learned_addr learned my addr 172.21.15.98:0/303124928 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.639+0000 7f2b0969d640 1 -- 172.21.15.98:0/303124928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd2507e0 msgr2=0x5625bd24e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.639+0000 7f2b0969d640 1 --2- 172.21.15.98:0/303124928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd2507e0 0x5625bd24e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.639+0000 7f2b0969d640 1 -- 172.21.15.98:0/303124928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 msgr2=0x5625bd24d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.639+0000 7f2b0969d640 1 --2- 172.21.15.98:0/303124928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd24d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.639+0000 7f2b0969d640 1 -- 172.21.15.98:0/303124928 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625bcebe210 con 0x5625bd245f20 2024-05-09T08:52:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.640+0000 7f2b0969d640 1 --2- 172.21.15.98:0/303124928 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd245f20 0x5625bd24dc60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2b00004bd0 tx=0x7f2b000c3a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72bf2e527ff9594b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.642+0000 7f2b09e9e640 1 --2- 172.21.15.98:0/1463818106 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2af00a30f0 0x7f2af00a55b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x5625bd246ba0 tx=0x7f2af40046c0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.642+0000 7f2aed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:52:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.642+0000 7f2aed7fa640 1 -- 172.21.15.98:0/303124928 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b000b7de0 con 0x5625bd245f20 2024-05-09T08:52:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.642+0000 7f2aed7fa640 1 -- 172.21.15.98:0/303124928 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f2b000bf5d0 con 0x5625bd245f20 2024-05-09T08:52:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.644+0000 7f2b0ee05740 1 -- 172.21.15.98:0/303124928 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2afc003d10 con 0x5625bd245f20 2024-05-09T08:52:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.649+0000 7f2aed7fa640 1 -- 172.21.15.98:0/303124928 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2b000b7f80 con 0x5625bd245f20 2024-05-09T08:52:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.649+0000 7f2b0ee05740 10 client.25583 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.649+0000 7f2b0ee05740 1 -- 172.21.15.98:0/303124928 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625bcf8a620 con 0x5625bd245f20 2024-05-09T08:52:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.650+0000 7f2aed7fa640 1 -- 172.21.15.98:0/303124928 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2afc003d10 con 0x5625bd245f20 2024-05-09T08:52:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.651+0000 7f2aed7fa640 1 -- 172.21.15.98:0/303124928 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2b000bb610 con 0x5625bd245f20 2024-05-09T08:52:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.652+0000 7f2b0ee05740 10 client.25583 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.652+0000 7f2b0ee05740 10 client.25583 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.652+0000 7f2b0ee05740 20 client.25583 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.652+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/303124928 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625bd260690 0x5625bd262af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.652+0000 7f2b0ee05740 4 client.25583 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:52:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.652+0000 7f2b0ee05740 1 -- 172.21.15.98:0/303124928 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625bd1b7a70 con 0x5625bd260690 2024-05-09T08:52:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.658+0000 7f2b09e9e640 1 --2- 172.21.15.98:0/303124928 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625bd260690 0x5625bd262af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.678+0000 7f2aed7fa640 10 client.25583 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.678+0000 7f2b09e9e640 1 --2- 172.21.15.98:0/303124928 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625bd260690 0x5625bd262af0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f22ed51f789c8429 server_cookie=a0e23728c3a05d6c in_seq=0 out_seq=0 2024-05-09T08:52:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.685+0000 7f2aed7fa640 1 -- 172.21.15.98:0/303124928 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5625bd1b7a70 con 0x5625bd260690 2024-05-09T08:52:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.685+0000 7f2aed7fa640 10 client.25583 handle_command_reply: tid=0 2024-05-09T08:52:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.685+0000 7f2b0ee05740 1 client.25583 shutdown 2024-05-09T08:52:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.691+0000 7f2b0ee05740 1 -- 172.21.15.98:0/303124928 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd245f20 msgr2=0x5625bd24dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.691+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/303124928 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd245f20 0x5625bd24dc60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2b00004bd0 tx=0x7f2b000c3a90 comp rx=0 tx=0).stop 2024-05-09T08:52:20.692 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.691+0000 7f2b0ee05740 1 -- 172.21.15.98:0/303124928 shutdown_connections 2024-05-09T08:52:20.692 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.691+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/303124928 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625bd260690 0x5625bd262af0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.692 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.691+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/303124928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd2507e0 0x5625bd24e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.691+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/303124928 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd245f20 0x5625bd24dc60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.691+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/303124928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd24d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.691+0000 7f2b0ee05740 1 -- 172.21.15.98:0/303124928 >> 172.21.15.98:0/303124928 conn(0x5625bd241b10 msgr2=0x5625bd247890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.691+0000 7f2aed7fa640 0 client.25583 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.691+0000 7f2b0ee05740 1 -- 172.21.15.98:0/303124928 shutdown_connections 2024-05-09T08:52:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.691+0000 7f2b0ee05740 1 -- 172.21.15.98:0/303124928 wait complete. 2024-05-09T08:52:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.693+0000 7f2b0ee05740 20 client.25583 trim_cache size 0 max 16384 2024-05-09T08:52:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.751+0000 7f2b0ee05740 1 Processor -- start 2024-05-09T08:52:20.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.751+0000 7f2b0ee05740 1 -- start start 2024-05-09T08:52:20.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.752+0000 7f2b0ee05740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd24fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.752+0000 7f2b0ee05740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd245f20 0x5625bd247fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.752+0000 7f2b0ee05740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd2507e0 0x5625bd240320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.752+0000 7f2b0ee05740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae800fe30 con 0x5625bd245600 2024-05-09T08:52:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.752+0000 7f2b0ee05740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae8011970 con 0x5625bd2507e0 2024-05-09T08:52:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.752+0000 7f2b0ee05740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae8010500 con 0x5625bd245f20 2024-05-09T08:52:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.753+0000 7f2b0969d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd245f20 0x5625bd247fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.753+0000 7f2b0969d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd245f20 0x5625bd247fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:35178/0 (socket says 172.21.15.98:35178) 2024-05-09T08:52:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.753+0000 7f2b0969d640 1 -- 172.21.15.98:0/806542471 learned_addr learned my addr 172.21.15.98:0/806542471 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.753+0000 7f2b0a69f640 1 --2- 172.21.15.98:0/806542471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd2507e0 0x5625bd240320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.754+0000 7f2b0969d640 1 -- 172.21.15.98:0/806542471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd2507e0 msgr2=0x5625bd240320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.754+0000 7f2b0969d640 1 --2- 172.21.15.98:0/806542471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd2507e0 0x5625bd240320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.754+0000 7f2b0969d640 1 -- 172.21.15.98:0/806542471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 msgr2=0x5625bd24fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.754+0000 7f2b0969d640 1 --2- 172.21.15.98:0/806542471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd24fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.754+0000 7f2b0969d640 1 -- 172.21.15.98:0/806542471 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625bd24bc70 con 0x5625bd245f20 2024-05-09T08:52:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.756+0000 7f2b0969d640 1 --2- 172.21.15.98:0/806542471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd245f20 0x5625bd247fc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2b000b8630 tx=0x7f2b000b7410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adee8b9c58683057 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:20.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.767+0000 7f2aef7fe640 1 -- 172.21.15.98:0/806542471 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b00040070 con 0x5625bd245f20 2024-05-09T08:52:20.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.767+0000 7f2aef7fe640 1 -- 172.21.15.98:0/806542471 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b0003c920 con 0x5625bd245f20 2024-05-09T08:52:20.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.767+0000 7f2aef7fe640 1 -- 172.21.15.98:0/806542471 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b000b8120 con 0x5625bd245f20 2024-05-09T08:52:20.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.771+0000 7f2b0ee05740 1 -- 172.21.15.98:0/806542471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd245f20 msgr2=0x5625bd247fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:20.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.771+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/806542471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd245f20 0x5625bd247fc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2b000b8630 tx=0x7f2b000b7410 comp rx=0 tx=0).stop 2024-05-09T08:52:20.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.774+0000 7f2b0ee05740 1 -- 172.21.15.98:0/806542471 shutdown_connections 2024-05-09T08:52:20.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.774+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/806542471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd2507e0 0x5625bd240320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.774+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/806542471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd245f20 0x5625bd247fc0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.774+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/806542471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd24fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.774+0000 7f2b0ee05740 1 -- 172.21.15.98:0/806542471 >> 172.21.15.98:0/806542471 conn(0x5625bd241b10 msgr2=0x5625bd24c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.774+0000 7f2b0ee05740 1 -- 172.21.15.98:0/806542471 shutdown_connections 2024-05-09T08:52:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.777+0000 7f2b0ee05740 1 -- 172.21.15.98:0/806542471 wait complete. 2024-05-09T08:52:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.778+0000 7f2b0ee05740 1 -- start start 2024-05-09T08:52:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.778+0000 7f2b0ee05740 1 Processor -- start 2024-05-09T08:52:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.778+0000 7f2b0ee05740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd2543e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.779+0000 7f2b0ee05740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd245f20 0x5625bd254920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.779+0000 7f2b0ee05740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd2507e0 0x5625bd254e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.779+0000 7f2b0ee05740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae80112a0 con 0x5625bd245600 2024-05-09T08:52:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.779+0000 7f2b0ee05740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae8010bd0 con 0x5625bd245f20 2024-05-09T08:52:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.779+0000 7f2b0ee05740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae800dec0 con 0x5625bd2507e0 2024-05-09T08:52:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.779+0000 7f2b0ee05740 10 client.0 _ll_register_callbacks cb 0x7f2b00039510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.779+0000 7f2b0a69f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd2507e0 0x5625bd254e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.779+0000 7f2b0a69f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd2507e0 0x5625bd254e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:35194/0 (socket says 172.21.15.98:35194) 2024-05-09T08:52:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.779+0000 7f2b0a69f640 1 -- 172.21.15.98:0/327969293 learned_addr learned my addr 172.21.15.98:0/327969293 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.780+0000 7f2b0a69f640 1 -- 172.21.15.98:0/327969293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd245f20 msgr2=0x5625bd254920 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:52:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.780+0000 7f2b0a69f640 1 --2- 172.21.15.98:0/327969293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd245f20 0x5625bd254920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.780+0000 7f2b0a69f640 1 -- 172.21.15.98:0/327969293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 msgr2=0x5625bd2543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.780+0000 7f2b0a69f640 1 --2- 172.21.15.98:0/327969293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd2543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.780+0000 7f2b0a69f640 1 -- 172.21.15.98:0/327969293 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625bd2c5d90 con 0x5625bd2507e0 2024-05-09T08:52:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.781+0000 7f2b0a69f640 1 --2- 172.21.15.98:0/327969293 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd2507e0 0x5625bd254e60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2afc011800 tx=0x7f2afc011f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ec32d2b648acff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.783+0000 7f2aed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:52:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.783+0000 7f2aed7fa640 1 -- 172.21.15.98:0/327969293 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc003b80 con 0x5625bd2507e0 2024-05-09T08:52:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.783+0000 7f2b0ee05740 1 -- 172.21.15.98:0/327969293 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b0003c920 con 0x5625bd2507e0 2024-05-09T08:52:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.783+0000 7f2aed7fa640 1 -- 172.21.15.98:0/327969293 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f2afc003d20 con 0x5625bd2507e0 2024-05-09T08:52:20.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.800+0000 7f2aed7fa640 1 -- 172.21.15.98:0/327969293 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2afc0050c0 con 0x5625bd2507e0 2024-05-09T08:52:20.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.801+0000 7f2b0ee05740 10 client.25534 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:20.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.801+0000 7f2b0ee05740 1 -- 172.21.15.98:0/327969293 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625bd2e9090 con 0x5625bd2507e0 2024-05-09T08:52:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.805+0000 7f2aed7fa640 1 -- 172.21.15.98:0/327969293 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b0003c920 con 0x5625bd2507e0 2024-05-09T08:52:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.805+0000 7f2aed7fa640 1 -- 172.21.15.98:0/327969293 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2afc026900 con 0x5625bd2507e0 2024-05-09T08:52:20.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.805+0000 7f2b0ee05740 10 client.25534 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:20.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.805+0000 7f2b0ee05740 10 client.25534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:20.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.805+0000 7f2b0ee05740 20 client.25534 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:20.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.805+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/327969293 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625bd2e9620 0x5625bd2eba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:20.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.805+0000 7f2b0ee05740 4 client.25534 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:52:20.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.805+0000 7f2b0ee05740 1 -- 172.21.15.98:0/327969293 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5625bd1b1dc0 con 0x5625bd2e9620 2024-05-09T08:52:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.806+0000 7f2b09e9e640 1 --2- 172.21.15.98:0/327969293 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625bd2e9620 0x5625bd2eba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.811+0000 7f2b09e9e640 1 --2- 172.21.15.98:0/327969293 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625bd2e9620 0x5625bd2eba40 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d741a2224ef3f2c3 server_cookie=f732b0bee51ca3e9 in_seq=0 out_seq=0 2024-05-09T08:52:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:20.811+0000 7f2aed7fa640 10 client.25534 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:22.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:52:22.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:52:22.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:21 smithi098 ceph-mon[30092]: pgmap v54: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 GiB/s wr, 2.07k op/s 2024-05-09T08:52:22.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:21 smithi142 ceph-mon[34625]: pgmap v54: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 GiB/s wr, 2.07k op/s 2024-05-09T08:52:22.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:21 smithi105 ceph-mon[34498]: pgmap v54: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 GiB/s wr, 2.07k op/s 2024-05-09T08:52:22.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.332+0000 7f84f8efa640 1 -- 172.21.15.98:0/656613118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84f410fed0 msgr2=0x7f84f41102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:22.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.332+0000 7f84f8efa640 1 --2- 172.21.15.98:0/656613118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84f410fed0 0x7f84f41102b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f84ec009900 tx=0x7f84ec02a4d0 comp rx=0 tx=0).stop 2024-05-09T08:52:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.339+0000 7f84f8efa640 1 -- 172.21.15.98:0/656613118 shutdown_connections 2024-05-09T08:52:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.339+0000 7f84f8efa640 1 --2- 172.21.15.98:0/656613118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f41092b0 0x7f84f411d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.339+0000 7f84f8efa640 1 --2- 172.21.15.98:0/656613118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84f4108830 0x7f84f4108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.339+0000 7f84f8efa640 1 --2- 172.21.15.98:0/656613118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84f410fed0 0x7f84f41102b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.339+0000 7f84f8efa640 1 -- 172.21.15.98:0/656613118 >> 172.21.15.98:0/656613118 conn(0x7f84f406c7b0 msgr2=0x7f84f406cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.339+0000 7f84f8efa640 1 -- 172.21.15.98:0/656613118 shutdown_connections 2024-05-09T08:52:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.339+0000 7f84f8efa640 1 -- 172.21.15.98:0/656613118 wait complete. 2024-05-09T08:52:22.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.340+0000 7f84f8efa640 1 Processor -- start 2024-05-09T08:52:22.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.341+0000 7f84f8efa640 1 -- start start 2024-05-09T08:52:22.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.341+0000 7f84f8efa640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84f4108830 0x7f84f41a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:22.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.341+0000 7f84f8efa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f41092b0 0x7f84f41a7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:22.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.342+0000 7f84f8efa640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84f410fed0 0x7f84f41ab640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:22.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.342+0000 7f84f8efa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f84f4120950 con 0x7f84f41092b0 2024-05-09T08:52:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.342+0000 7f84f37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84f4108830 0x7f84f41a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.342+0000 7f84f37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84f4108830 0x7f84f41a6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:43916/0 (socket says 172.21.15.98:43916) 2024-05-09T08:52:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.342+0000 7f84f37fe640 1 -- 172.21.15.98:0/59747621 learned_addr learned my addr 172.21.15.98:0/59747621 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.342+0000 7f84f2ffd640 1 --2- 172.21.15.98:0/59747621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f41092b0 0x7f84f41a7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.342+0000 7f84f3fff640 1 --2- 172.21.15.98:0/59747621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84f410fed0 0x7f84f41ab640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.342+0000 7f84f8efa640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f84f41207d0 con 0x7f84f4108830 2024-05-09T08:52:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.342+0000 7f84f8efa640 1 -- 172.21.15.98:0/59747621 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f84f4120ad0 con 0x7f84f410fed0 2024-05-09T08:52:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.343+0000 7f84f2ffd640 1 -- 172.21.15.98:0/59747621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84f410fed0 msgr2=0x7f84f41ab640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.343+0000 7f84f2ffd640 1 --2- 172.21.15.98:0/59747621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84f410fed0 0x7f84f41ab640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.343+0000 7f84f2ffd640 1 -- 172.21.15.98:0/59747621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84f4108830 msgr2=0x7f84f41a6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.343+0000 7f84f2ffd640 1 --2- 172.21.15.98:0/59747621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84f4108830 0x7f84f41a6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.343+0000 7f84f2ffd640 1 -- 172.21.15.98:0/59747621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f41abd70 con 0x7f84f41092b0 2024-05-09T08:52:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.343+0000 7f84f2ffd640 1 --2- 172.21.15.98:0/59747621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f41092b0 0x7f84f41a7310 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f84e400bd50 tx=0x7f84e400be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:22.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.344+0000 7f84f0ff9640 1 -- 172.21.15.98:0/59747621 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e4019070 con 0x7f84f41092b0 2024-05-09T08:52:22.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.344+0000 7f84f0ff9640 1 -- 172.21.15.98:0/59747621 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84e4014920 con 0x7f84f41092b0 2024-05-09T08:52:22.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.344+0000 7f84f0ff9640 1 -- 172.21.15.98:0/59747621 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e4012990 con 0x7f84f41092b0 2024-05-09T08:52:22.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.345+0000 7f84f8efa640 1 -- 172.21.15.98:0/59747621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84f41ac060 con 0x7f84f41092b0 2024-05-09T08:52:22.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.345+0000 7f84f8efa640 1 -- 172.21.15.98:0/59747621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84f41b38f0 con 0x7f84f41092b0 2024-05-09T08:52:22.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.349+0000 7f84fce60740 1 Processor -- start 2024-05-09T08:52:22.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.350+0000 7f84fce60740 1 -- start start 2024-05-09T08:52:22.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.350+0000 7f84fce60740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f89430 0x555e68f89810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:22.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.350+0000 7f84fce60740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89d50 0x555e68f93ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:22.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.350+0000 7f84fce60740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f94420 0x555e68f96810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:22.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.351+0000 7f84f0ff9640 1 -- 172.21.15.98:0/59747621 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f84e401c070 con 0x7f84f41092b0 2024-05-09T08:52:22.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.352+0000 7f84f37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89d50 0x555e68f93ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:22.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.352+0000 7f84f37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89d50 0x555e68f93ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50714/0 (socket says 172.21.15.98:50714) 2024-05-09T08:52:22.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.352+0000 7f84f3fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f89430 0x555e68f89810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:22.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.352+0000 7f84f3fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f89430 0x555e68f89810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:35218/0 (socket says 172.21.15.98:35218) 2024-05-09T08:52:22.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.352+0000 7f84f0ff9640 1 --2- 172.21.15.98:0/59747621 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f84e00a2dc0 0x7f84e00a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:22.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.353+0000 7f84f37fe640 1 -- 172.21.15.98:0/1135020881 learned_addr learned my addr 172.21.15.98:0/1135020881 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:22.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.353+0000 7f84fce60740 1 -- 172.21.15.98:0/1135020881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555e68eeed00 con 0x555e68f89d50 2024-05-09T08:52:22.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.353+0000 7f84fce60740 1 -- 172.21.15.98:0/1135020881 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555e68e964c0 con 0x555e68f94420 2024-05-09T08:52:22.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.353+0000 7f84fce60740 1 -- 172.21.15.98:0/1135020881 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555e68eeacc0 con 0x555e68f89430 2024-05-09T08:52:22.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.353+0000 7f84f0ff9640 1 -- 172.21.15.98:0/59747621 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f84e40991f0 con 0x7f84f41092b0 2024-05-09T08:52:22.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.354+0000 7f84f2ffd640 1 --2- 172.21.15.98:0/1135020881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f94420 0x555e68f96810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.354+0000 7f84f3fff640 1 -- 172.21.15.98:0/1135020881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f94420 msgr2=0x555e68f96810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.354+0000 7f84f3fff640 1 --2- 172.21.15.98:0/1135020881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f94420 0x555e68f96810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.354+0000 7f84f3fff640 1 -- 172.21.15.98:0/1135020881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89d50 msgr2=0x555e68f93ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.354+0000 7f84f3fff640 1 --2- 172.21.15.98:0/1135020881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89d50 0x555e68f93ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.354+0000 7f84f3fff640 1 -- 172.21.15.98:0/1135020881 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e68ccc7d0 con 0x555e68f89430 2024-05-09T08:52:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.355+0000 7f84f3fff640 1 --2- 172.21.15.98:0/1135020881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f89430 0x555e68f89810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00ac70 tx=0x7f84dc0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73354b761306f087 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.355+0000 7f84e9ffb640 1 -- 172.21.15.98:0/1135020881 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc005220 con 0x555e68f89430 2024-05-09T08:52:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.355+0000 7f84e9ffb640 1 -- 172.21.15.98:0/1135020881 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84dc004c80 con 0x555e68f89430 2024-05-09T08:52:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.355+0000 7f84e9ffb640 1 -- 172.21.15.98:0/1135020881 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc012990 con 0x555e68f89430 2024-05-09T08:52:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.356+0000 7f84f3fff640 1 --2- 172.21.15.98:0/59747621 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f84e00a2dc0 0x7f84e00a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:22.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.359+0000 7f84fce60740 1 -- 172.21.15.98:0/1135020881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f89430 msgr2=0x555e68f89810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:22.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.359+0000 7f84fce60740 1 --2- 172.21.15.98:0/1135020881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f89430 0x555e68f89810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00ac70 tx=0x7f84dc0117b0 comp rx=0 tx=0).stop 2024-05-09T08:52:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.366+0000 7f84fce60740 1 -- 172.21.15.98:0/1135020881 shutdown_connections 2024-05-09T08:52:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.366+0000 7f84fce60740 1 --2- 172.21.15.98:0/1135020881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f94420 0x555e68f96810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.366+0000 7f84fce60740 1 --2- 172.21.15.98:0/1135020881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89d50 0x555e68f93ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.366+0000 7f84fce60740 1 --2- 172.21.15.98:0/1135020881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f89430 0x555e68f89810 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.366+0000 7f84fce60740 1 -- 172.21.15.98:0/1135020881 >> 172.21.15.98:0/1135020881 conn(0x555e68f85750 msgr2=0x555e68f85b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:22.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.368+0000 7f84fce60740 1 -- 172.21.15.98:0/1135020881 shutdown_connections 2024-05-09T08:52:22.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.368+0000 7f84fce60740 1 -- 172.21.15.98:0/1135020881 wait complete. 2024-05-09T08:52:22.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.368+0000 7f84f3fff640 1 --2- 172.21.15.98:0/59747621 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f84e00a2dc0 0x7f84e00a5280 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00ac70 tx=0x7f84dc0117b0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:22.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.369+0000 7f84fce60740 1 -- start start 2024-05-09T08:52:22.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.369+0000 7f84fce60740 1 Processor -- start 2024-05-09T08:52:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.369+0000 7f84fce60740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89430 0x555e68f913f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84fce60740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f89d50 0x555e68f91930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84fce60740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f94420 0x555e68f91e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84fce60740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555e68f97210 con 0x555e68f89430 2024-05-09T08:52:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84fce60740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555e68ea9b90 con 0x555e68f89d50 2024-05-09T08:52:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84fce60740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555e68f97390 con 0x555e68f94420 2024-05-09T08:52:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84fce60740 10 client.0 _ll_register_callbacks cb 0x555e68ee5da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84f2ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f89d50 0x555e68f91930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84f2ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f89d50 0x555e68f91930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:43926/0 (socket says 172.21.15.98:43926) 2024-05-09T08:52:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84f2ffd640 1 -- 172.21.15.98:0/1949101169 learned_addr learned my addr 172.21.15.98:0/1949101169 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84f3fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f94420 0x555e68f91e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84f3fff640 1 -- 172.21.15.98:0/1949101169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f89d50 msgr2=0x555e68f91930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84f3fff640 1 --2- 172.21.15.98:0/1949101169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f89d50 0x555e68f91930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84f3fff640 1 -- 172.21.15.98:0/1949101169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89430 msgr2=0x555e68f913f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84f3fff640 1 --2- 172.21.15.98:0/1949101169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89430 0x555e68f913f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84f3fff640 1 -- 172.21.15.98:0/1949101169 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e68c023f0 con 0x555e68f94420 2024-05-09T08:52:22.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84f3fff640 1 --2- 172.21.15.98:0/1949101169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f94420 0x555e68f91e70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00aa00 tx=0x7f84dc00a6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a8893184ec1d484 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:22.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.370+0000 7f84d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:52:22.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.371+0000 7f84d2ffd640 1 -- 172.21.15.98:0/1949101169 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc034060 con 0x555e68f94420 2024-05-09T08:52:22.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.371+0000 7f84d2ffd640 1 -- 172.21.15.98:0/1949101169 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f84dc01a5e0 con 0x555e68f94420 2024-05-09T08:52:22.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.373+0000 7f84fce60740 1 -- 172.21.15.98:0/1949101169 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84dc004c80 con 0x555e68f94420 2024-05-09T08:52:22.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.377+0000 7f84d2ffd640 1 -- 172.21.15.98:0/1949101169 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f84dc012940 con 0x555e68f94420 2024-05-09T08:52:22.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.381+0000 7f84fce60740 10 client.25546 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:22.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.381+0000 7f84fce60740 1 -- 172.21.15.98:0/1949101169 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555e68bc20f0 con 0x555e68f94420 2024-05-09T08:52:22.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.382+0000 7f84d2ffd640 1 -- 172.21.15.98:0/1949101169 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84dc004c80 con 0x555e68f94420 2024-05-09T08:52:22.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.382+0000 7f84d2ffd640 1 -- 172.21.15.98:0/1949101169 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f84dc023430 con 0x555e68f94420 2024-05-09T08:52:22.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.382+0000 7f84fce60740 10 client.25546 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:22.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.382+0000 7f84fce60740 10 client.25546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:22.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.382+0000 7f84fce60740 20 client.25546 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:22.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.382+0000 7f84fce60740 1 --2- 172.21.15.98:0/1949101169 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555e68fa4b30 0x555e68fa6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:22.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.382+0000 7f84fce60740 4 client.25546 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:52:22.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.382+0000 7f84fce60740 1 -- 172.21.15.98:0/1949101169 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e68efc430 con 0x555e68fa4b30 2024-05-09T08:52:22.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.386+0000 7f84f37fe640 1 --2- 172.21.15.98:0/1949101169 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555e68fa4b30 0x555e68fa6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:22.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.393+0000 7f84f37fe640 1 --2- 172.21.15.98:0/1949101169 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555e68fa4b30 0x555e68fa6fb0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=448c14227eccc662 server_cookie=8e749decda884b2d in_seq=0 out_seq=0 2024-05-09T08:52:22.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:22.393+0000 7f84d2ffd640 10 client.25546 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:24.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:23 smithi098 ceph-mon[30092]: pgmap v55: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 786 MiB/s wr, 1.84k op/s 2024-05-09T08:52:24.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:23 smithi142 ceph-mon[34625]: pgmap v55: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 786 MiB/s wr, 1.84k op/s 2024-05-09T08:52:24.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:23 smithi105 ceph-mon[34498]: pgmap v55: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 786 MiB/s wr, 1.84k op/s 2024-05-09T08:52:26.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:25 smithi098 ceph-mon[30092]: pgmap v56: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 940 MiB/s wr, 2.10k op/s 2024-05-09T08:52:26.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:25 smithi142 ceph-mon[34625]: pgmap v56: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 940 MiB/s wr, 2.10k op/s 2024-05-09T08:52:26.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:25 smithi105 ceph-mon[34498]: pgmap v56: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 940 MiB/s wr, 2.10k op/s 2024-05-09T08:52:27.830 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.831+0000 7f2aed7fa640 1 -- 172.21.15.98:0/327969293 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5625bd1b1dc0 con 0x5625bd2e9620 2024-05-09T08:52:27.830 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.831+0000 7f2aed7fa640 10 client.25534 handle_command_reply: tid=0 2024-05-09T08:52:27.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.831+0000 7f2b0ee05740 1 client.25534 shutdown 2024-05-09T08:52:27.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.831+0000 7f2b0ee05740 1 -- 172.21.15.98:0/327969293 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd2507e0 msgr2=0x5625bd254e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:27.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.831+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/327969293 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd2507e0 0x5625bd254e60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2afc011800 tx=0x7f2afc011f90 comp rx=0 tx=0).stop 2024-05-09T08:52:27.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.831+0000 7f84d2ffd640 1 -- 172.21.15.98:0/1949101169 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e68efc430 con 0x555e68fa4b30 2024-05-09T08:52:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.831+0000 7f84d2ffd640 10 client.25546 handle_command_reply: tid=0 2024-05-09T08:52:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.831+0000 7f84fce60740 1 client.25546 shutdown 2024-05-09T08:52:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f84fce60740 1 -- 172.21.15.98:0/1949101169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f94420 msgr2=0x555e68f91e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f84fce60740 1 --2- 172.21.15.98:0/1949101169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f94420 0x555e68f91e70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00aa00 tx=0x7f84dc00a6b0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f84fce60740 1 -- 172.21.15.98:0/1949101169 shutdown_connections 2024-05-09T08:52:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f84fce60740 1 --2- 172.21.15.98:0/1949101169 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555e68fa4b30 0x555e68fa6fb0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f84fce60740 1 --2- 172.21.15.98:0/1949101169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f94420 0x555e68f91e70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f84fce60740 1 --2- 172.21.15.98:0/1949101169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f89d50 0x555e68f91930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f84fce60740 1 --2- 172.21.15.98:0/1949101169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89430 0x555e68f913f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f84fce60740 1 -- 172.21.15.98:0/1949101169 >> 172.21.15.98:0/1949101169 conn(0x555e68f85750 msgr2=0x555e68f8b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f84d2ffd640 0 client.25546 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f2b0ee05740 1 -- 172.21.15.98:0/327969293 shutdown_connections 2024-05-09T08:52:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/327969293 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625bd2e9620 0x5625bd2eba40 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/327969293 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625bd2507e0 0x5625bd254e60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/327969293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625bd245f20 0x5625bd254920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f2b0ee05740 1 --2- 172.21.15.98:0/327969293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625bd245600 0x5625bd2543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f2b0ee05740 1 -- 172.21.15.98:0/327969293 >> 172.21.15.98:0/327969293 conn(0x5625bd241b10 msgr2=0x5625bd246fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f2aed7fa640 0 client.25534 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f2b0ee05740 1 -- 172.21.15.98:0/327969293 shutdown_connections 2024-05-09T08:52:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.832+0000 7f84fce60740 1 -- 172.21.15.98:0/1949101169 shutdown_connections 2024-05-09T08:52:27.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.833+0000 7f2b0ee05740 1 -- 172.21.15.98:0/327969293 wait complete. 2024-05-09T08:52:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.833+0000 7f84fce60740 1 -- 172.21.15.98:0/1949101169 wait complete. 2024-05-09T08:52:27.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.833+0000 7f2b0ee05740 20 client.25534 trim_cache size 0 max 16384 2024-05-09T08:52:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.833+0000 7f84fce60740 20 client.25546 trim_cache size 0 max 16384 2024-05-09T08:52:27.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:52:27.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:52:27.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:52:27.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:52:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.834+0000 7f2b0aea0640 1 -- 172.21.15.98:0/1463818106 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2af00a30f0 msgr2=0x7f2af00a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.834+0000 7f2b0aea0640 1 --2- 172.21.15.98:0/1463818106 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2af00a30f0 0x7f2af00a55b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x5625bd246ba0 tx=0x7f2af40046c0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.834+0000 7f2b0aea0640 1 -- 172.21.15.98:0/1463818106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b0410fef0 msgr2=0x7f2b04116d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.834+0000 7f2b0aea0640 1 --2- 172.21.15.98:0/1463818106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b0410fef0 0x7f2b04116d70 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f2b00004650 tx=0x7f2b000052b0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.838+0000 7f2b0aea0640 1 -- 172.21.15.98:0/1463818106 shutdown_connections 2024-05-09T08:52:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.838+0000 7f2b0aea0640 1 --2- 172.21.15.98:0/1463818106 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2af00a30f0 0x7f2af00a55b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.838+0000 7f2b0aea0640 1 --2- 172.21.15.98:0/1463818106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b041102d0 0x7f2b041172b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.838+0000 7f2b0aea0640 1 --2- 172.21.15.98:0/1463818106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b0410fef0 0x7f2b04116d70 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.838+0000 7f2b0aea0640 1 --2- 172.21.15.98:0/1463818106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b0410e500 0x7f2b0411cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.838+0000 7f2b0aea0640 1 -- 172.21.15.98:0/1463818106 >> 172.21.15.98:0/1463818106 conn(0x7f2b0406c770 msgr2=0x7f2b0410ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.838+0000 7f2b0aea0640 1 -- 172.21.15.98:0/1463818106 shutdown_connections 2024-05-09T08:52:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:27.840+0000 7f2b0aea0640 1 -- 172.21.15.98:0/1463818106 wait complete. 2024-05-09T08:52:27.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.872+0000 7f84fce60740 1 Processor -- start 2024-05-09T08:52:27.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.872+0000 7f84fce60740 1 -- start start 2024-05-09T08:52:27.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.873+0000 7f84fce60740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f89430 0x555e68fa00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:27.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.873+0000 7f84fce60740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89d50 0x555e68f93330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:27.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.873+0000 7f84fce60740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f94420 0x555e68fac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:27.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.873+0000 7f84fce60740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f84f41bda10 con 0x555e68f89d50 2024-05-09T08:52:27.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.873+0000 7f84fce60740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f84f41bf550 con 0x555e68f89430 2024-05-09T08:52:27.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.873+0000 7f84fce60740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f84f41be0e0 con 0x555e68f94420 2024-05-09T08:52:27.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.874+0000 7f84f2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89d50 0x555e68f93330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:27.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.874+0000 7f84f37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f89430 0x555e68fa00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:27.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.874+0000 7f84f2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89d50 0x555e68f93330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47074/0 (socket says 172.21.15.98:47074) 2024-05-09T08:52:27.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.874+0000 7f84f2ffd640 1 -- 172.21.15.98:0/3254702390 learned_addr learned my addr 172.21.15.98:0/3254702390 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:27.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.875+0000 7f84f3fff640 1 --2- 172.21.15.98:0/3254702390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f94420 0x555e68fac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:27.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.876+0000 7f84f2ffd640 1 -- 172.21.15.98:0/3254702390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f94420 msgr2=0x555e68fac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:27.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.876+0000 7f84f2ffd640 1 --2- 172.21.15.98:0/3254702390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f94420 0x555e68fac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.876+0000 7f84f2ffd640 1 -- 172.21.15.98:0/3254702390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f89430 msgr2=0x555e68fa00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:27.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.876+0000 7f84f2ffd640 1 --2- 172.21.15.98:0/3254702390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f89430 0x555e68fa00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.876+0000 7f84f2ffd640 1 -- 172.21.15.98:0/3254702390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e68f89b80 con 0x555e68f89d50 2024-05-09T08:52:27.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.876+0000 7f84f2ffd640 1 --2- 172.21.15.98:0/3254702390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89d50 0x555e68f93330 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f84e4097b20 tx=0x7f84e40bb050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fed7916df9d80b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:27.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.882+0000 7f84e97fa640 1 -- 172.21.15.98:0/3254702390 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e40d3060 con 0x555e68f89d50 2024-05-09T08:52:27.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.882+0000 7f84e97fa640 1 -- 172.21.15.98:0/3254702390 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84e40a3d10 con 0x555e68f89d50 2024-05-09T08:52:27.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.884+0000 7f84fce60740 1 -- 172.21.15.98:0/3254702390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89d50 msgr2=0x555e68f93330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:27.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.884+0000 7f84fce60740 1 --2- 172.21.15.98:0/3254702390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89d50 0x555e68f93330 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f84e4097b20 tx=0x7f84e40bb050 comp rx=0 tx=0).stop 2024-05-09T08:52:27.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.885+0000 7f84fce60740 1 -- 172.21.15.98:0/3254702390 shutdown_connections 2024-05-09T08:52:27.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.885+0000 7f84fce60740 1 --2- 172.21.15.98:0/3254702390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f94420 0x555e68fac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.885+0000 7f84fce60740 1 --2- 172.21.15.98:0/3254702390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89d50 0x555e68f93330 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.885+0000 7f84fce60740 1 --2- 172.21.15.98:0/3254702390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f89430 0x555e68fa00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.885+0000 7f84fce60740 1 -- 172.21.15.98:0/3254702390 >> 172.21.15.98:0/3254702390 conn(0x555e68f85750 msgr2=0x555e68f92c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:27.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.889+0000 7f84fce60740 1 -- 172.21.15.98:0/3254702390 shutdown_connections 2024-05-09T08:52:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.890+0000 7f84fce60740 1 -- 172.21.15.98:0/3254702390 wait complete. 2024-05-09T08:52:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.890+0000 7f84fce60740 1 -- start start 2024-05-09T08:52:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.891+0000 7f84fce60740 1 Processor -- start 2024-05-09T08:52:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.891+0000 7f84fce60740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89430 0x555e68f9c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.891+0000 7f84fce60740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f89d50 0x555e68f9cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.891+0000 7f84fce60740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f94420 0x555e68f8dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.891+0000 7f84fce60740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f84f41bee80 con 0x555e68f89430 2024-05-09T08:52:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.891+0000 7f84fce60740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f84f41be7b0 con 0x555e68f94420 2024-05-09T08:52:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.891+0000 7f84fce60740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f84f41bb9f0 con 0x555e68f89d50 2024-05-09T08:52:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.891+0000 7f84fce60740 10 client.0 _ll_register_callbacks cb 0x7f84dc026040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:27.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.893+0000 7f84f2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f89d50 0x555e68f9cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:27.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.893+0000 7f84f2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f89d50 0x555e68f9cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55282/0 (socket says 172.21.15.98:55282) 2024-05-09T08:52:27.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.893+0000 7f84f2ffd640 1 -- 172.21.15.98:0/3197262722 learned_addr learned my addr 172.21.15.98:0/3197262722 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:27.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.894+0000 7f84f37fe640 1 --2- 172.21.15.98:0/3197262722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89430 0x555e68f9c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:27.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.894+0000 7f84f3fff640 1 --2- 172.21.15.98:0/3197262722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f94420 0x555e68f8dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:27.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.895+0000 7f84f2ffd640 1 -- 172.21.15.98:0/3197262722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f94420 msgr2=0x555e68f8dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:27.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.895+0000 7f84f2ffd640 1 --2- 172.21.15.98:0/3197262722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f94420 0x555e68f8dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.895+0000 7f84f2ffd640 1 -- 172.21.15.98:0/3197262722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89430 msgr2=0x555e68f9c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:27.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.895+0000 7f84f2ffd640 1 --2- 172.21.15.98:0/3197262722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89430 0x555e68f9c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.895+0000 7f84f2ffd640 1 -- 172.21.15.98:0/3197262722 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e68f8e3a0 con 0x555e68f89d50 2024-05-09T08:52:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.898+0000 7f84f2ffd640 1 --2- 172.21.15.98:0/3197262722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f89d50 0x555e68f9cb20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f84e409b0e0 tx=0x7f84e400bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23c143d9bb969158 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.898+0000 7f84d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:52:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.898+0000 7f84d2ffd640 1 -- 172.21.15.98:0/3197262722 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e409a5a0 con 0x555e68f89d50 2024-05-09T08:52:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.898+0000 7f84d2ffd640 1 -- 172.21.15.98:0/3197262722 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f84e409a740 con 0x555e68f89d50 2024-05-09T08:52:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.898+0000 7f84fce60740 1 -- 172.21.15.98:0/3197262722 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84e40a3d10 con 0x555e68f89d50 2024-05-09T08:52:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.900+0000 7f84d2ffd640 1 -- 172.21.15.98:0/3197262722 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f84e4014440 con 0x555e68f89d50 2024-05-09T08:52:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.902+0000 7f84fce60740 10 client.25552 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.902+0000 7f84fce60740 1 -- 172.21.15.98:0/3197262722 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555e6901bcd0 con 0x555e68f89d50 2024-05-09T08:52:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.905+0000 7f84d2ffd640 1 -- 172.21.15.98:0/3197262722 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84e40a3d10 con 0x555e68f89d50 2024-05-09T08:52:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.906+0000 7f84d2ffd640 1 -- 172.21.15.98:0/3197262722 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f84e40b9ce0 con 0x555e68f89d50 2024-05-09T08:52:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.906+0000 7f84fce60740 10 client.25552 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.906+0000 7f84fce60740 10 client.25552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.906+0000 7f84fce60740 20 client.25552 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.906+0000 7f84fce60740 1 --2- 172.21.15.98:0/3197262722 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555e6901c280 0x555e6901e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.906+0000 7f84fce60740 4 client.25552 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:52:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.906+0000 7f84fce60740 1 -- 172.21.15.98:0/3197262722 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555e68ef6780 con 0x555e6901c280 2024-05-09T08:52:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.906+0000 7f84f37fe640 1 --2- 172.21.15.98:0/3197262722 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555e6901c280 0x555e6901e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.907+0000 7f84d2ffd640 10 client.25552 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.908+0000 7f84f37fe640 1 --2- 172.21.15.98:0/3197262722 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555e6901c280 0x555e6901e6a0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c22884db5bb5cff3 server_cookie=9ffc19683fe2415b in_seq=0 out_seq=0 2024-05-09T08:52:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.908+0000 7f84d2ffd640 1 -- 172.21.15.98:0/3197262722 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555e68ef6780 con 0x555e6901c280 2024-05-09T08:52:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.909+0000 7f84d2ffd640 10 client.25552 handle_command_reply: tid=0 2024-05-09T08:52:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.909+0000 7f84fce60740 1 client.25552 shutdown 2024-05-09T08:52:27.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.917+0000 7f84fce60740 1 -- 172.21.15.98:0/3197262722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f89d50 msgr2=0x555e68f9cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:27.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.917+0000 7f84fce60740 1 --2- 172.21.15.98:0/3197262722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f89d50 0x555e68f9cb20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f84e409b0e0 tx=0x7f84e400bf70 comp rx=0 tx=0).stop 2024-05-09T08:52:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.921+0000 7f84fce60740 1 -- 172.21.15.98:0/3197262722 shutdown_connections 2024-05-09T08:52:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.921+0000 7f84fce60740 1 --2- 172.21.15.98:0/3197262722 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555e6901c280 0x555e6901e6a0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.921+0000 7f84fce60740 1 --2- 172.21.15.98:0/3197262722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e68f94420 0x555e68f8dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.921+0000 7f84fce60740 1 --2- 172.21.15.98:0/3197262722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e68f89d50 0x555e68f9cb20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.921+0000 7f84fce60740 1 --2- 172.21.15.98:0/3197262722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e68f89430 0x555e68f9c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.921+0000 7f84d2ffd640 0 client.25552 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.921+0000 7f84fce60740 1 -- 172.21.15.98:0/3197262722 >> 172.21.15.98:0/3197262722 conn(0x555e68f85750 msgr2=0x555e68f934b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:27.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.924+0000 7f84fce60740 1 -- 172.21.15.98:0/3197262722 shutdown_connections 2024-05-09T08:52:27.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.925+0000 7f84fce60740 1 -- 172.21.15.98:0/3197262722 wait complete. 2024-05-09T08:52:27.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.925+0000 7f84fce60740 20 client.25552 trim_cache size 0 max 16384 2024-05-09T08:52:27.926 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:52:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:52:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:52:27.926 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:52:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.929+0000 7f84f8efa640 1 -- 172.21.15.98:0/59747621 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f84e00a2dc0 msgr2=0x7f84e00a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.929+0000 7f84f8efa640 1 --2- 172.21.15.98:0/59747621 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f84e00a2dc0 0x7f84e00a5280 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00ac70 tx=0x7f84dc0117b0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.929+0000 7f84f8efa640 1 -- 172.21.15.98:0/59747621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f41092b0 msgr2=0x7f84f41a7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.929+0000 7f84f8efa640 1 --2- 172.21.15.98:0/59747621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f41092b0 0x7f84f41a7310 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f84e400bd50 tx=0x7f84e400be40 comp rx=0 tx=0).stop 2024-05-09T08:52:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.931+0000 7f84f8efa640 1 -- 172.21.15.98:0/59747621 shutdown_connections 2024-05-09T08:52:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.931+0000 7f84f8efa640 1 --2- 172.21.15.98:0/59747621 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f84e00a2dc0 0x7f84e00a5280 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.931+0000 7f84f8efa640 1 --2- 172.21.15.98:0/59747621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84f410fed0 0x7f84f41ab640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.931+0000 7f84f8efa640 1 --2- 172.21.15.98:0/59747621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f41092b0 0x7f84f41a7310 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.931+0000 7f84f8efa640 1 --2- 172.21.15.98:0/59747621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84f4108830 0x7f84f41a6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.931+0000 7f84f8efa640 1 -- 172.21.15.98:0/59747621 >> 172.21.15.98:0/59747621 conn(0x7f84f406c7b0 msgr2=0x7f84f4070390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.933+0000 7f84f8efa640 1 -- 172.21.15.98:0/59747621 shutdown_connections 2024-05-09T08:52:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:27.933+0000 7f84f8efa640 1 -- 172.21.15.98:0/59747621 wait complete. 2024-05-09T08:52:27.959 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:52:27.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:52:28.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:27 smithi098 ceph-mon[30092]: pgmap v57: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1020 MiB/s wr, 2.22k op/s 2024-05-09T08:52:28.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:27 smithi142 ceph-mon[34625]: pgmap v57: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1020 MiB/s wr, 2.22k op/s 2024-05-09T08:52:28.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:27 smithi105 ceph-mon[34498]: pgmap v57: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1020 MiB/s wr, 2.22k op/s 2024-05-09T08:52:28.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.299+0000 7fef0dcca640 1 -- 172.21.15.98:0/3584199178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef081092d0 msgr2=0x7fef0811d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:28.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.299+0000 7fef0dcca640 1 --2- 172.21.15.98:0/3584199178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef081092d0 0x7fef0811d020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fef00009ca0 tx=0x7fef0002a950 comp rx=0 tx=0).stop 2024-05-09T08:52:28.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.300+0000 7fef0dcca640 1 -- 172.21.15.98:0/3584199178 shutdown_connections 2024-05-09T08:52:28.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.300+0000 7fef0dcca640 1 --2- 172.21.15.98:0/3584199178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef081092d0 0x7fef0811d020 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:28.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.300+0000 7fef0dcca640 1 --2- 172.21.15.98:0/3584199178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef08108830 0x7fef08108cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:28.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.300+0000 7fef0dcca640 1 --2- 172.21.15.98:0/3584199178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef0810fe30 0x7fef08110210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:28.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.300+0000 7fef0dcca640 1 -- 172.21.15.98:0/3584199178 >> 172.21.15.98:0/3584199178 conn(0x7fef0806c770 msgr2=0x7fef0806cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:28.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.301+0000 7fef0dcca640 1 -- 172.21.15.98:0/3584199178 shutdown_connections 2024-05-09T08:52:28.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.301+0000 7fef0dcca640 1 -- 172.21.15.98:0/3584199178 wait complete. 2024-05-09T08:52:28.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.302+0000 7fef0dcca640 1 Processor -- start 2024-05-09T08:52:28.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.302+0000 7fef0dcca640 1 -- start start 2024-05-09T08:52:28.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.303+0000 7fef0dcca640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef08108830 0x7fef0811cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:28.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.303+0000 7fef0dcca640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef081092d0 0x7fef0811d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:28.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.303+0000 7fef0dcca640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef0810fe30 0x7fef08115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:28.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.303+0000 7fef0dcca640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef08120850 con 0x7fef0810fe30 2024-05-09T08:52:28.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.303+0000 7fef0dcca640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fef081206d0 con 0x7fef081092d0 2024-05-09T08:52:28.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.303+0000 7fef0dcca640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fef081209d0 con 0x7fef08108830 2024-05-09T08:52:28.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.306+0000 7fef06ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef081092d0 0x7fef0811d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:28.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.306+0000 7fef06ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef081092d0 0x7fef0811d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:53084/0 (socket says 172.21.15.98:53084) 2024-05-09T08:52:28.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.306+0000 7fef06ffd640 1 -- 172.21.15.98:0/3037886198 learned_addr learned my addr 172.21.15.98:0/3037886198 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:28.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.307+0000 7fef077fe640 1 --2- 172.21.15.98:0/3037886198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef08108830 0x7fef0811cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:28.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.308+0000 7fef07fff640 1 --2- 172.21.15.98:0/3037886198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef0810fe30 0x7fef08115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:28.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.310+0000 7fef06ffd640 1 -- 172.21.15.98:0/3037886198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef08108830 msgr2=0x7fef0811cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:28.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.310+0000 7fef06ffd640 1 --2- 172.21.15.98:0/3037886198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef08108830 0x7fef0811cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:28.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.310+0000 7fef06ffd640 1 -- 172.21.15.98:0/3037886198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef0810fe30 msgr2=0x7fef08115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:28.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.310+0000 7fef06ffd640 1 --2- 172.21.15.98:0/3037886198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef0810fe30 0x7fef08115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:28.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.310+0000 7fef06ffd640 1 -- 172.21.15.98:0/3037886198 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef081165f0 con 0x7fef081092d0 2024-05-09T08:52:28.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.312+0000 7fef06ffd640 1 --2- 172.21.15.98:0/3037886198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef081092d0 0x7fef0811d010 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feefc00e9c0 tx=0x7feefc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:28.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.314+0000 7fef04ff9640 1 -- 172.21.15.98:0/3037886198 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc00cd80 con 0x7fef081092d0 2024-05-09T08:52:28.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.314+0000 7fef0dcca640 1 -- 172.21.15.98:0/3037886198 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef081168e0 con 0x7fef081092d0 2024-05-09T08:52:28.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.314+0000 7fef0dcca640 1 -- 172.21.15.98:0/3037886198 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef0811a820 con 0x7fef081092d0 2024-05-09T08:52:28.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.316+0000 7fef04ff9640 1 -- 172.21.15.98:0/3037886198 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feefc0035f0 con 0x7fef081092d0 2024-05-09T08:52:28.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.317+0000 7fef0dcca640 1 -- 172.21.15.98:0/3037886198 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef0806da40 con 0x7fef081092d0 2024-05-09T08:52:28.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.317+0000 7fef04ff9640 1 -- 172.21.15.98:0/3037886198 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc010690 con 0x7fef081092d0 2024-05-09T08:52:28.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.324+0000 7fef04ff9640 1 -- 172.21.15.98:0/3037886198 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7feefc010830 con 0x7fef081092d0 2024-05-09T08:52:28.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.325+0000 7fef04ff9640 1 --2- 172.21.15.98:0/3037886198 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feef00a2ed0 0x7feef00a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:28.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.326+0000 7fef077fe640 1 --2- 172.21.15.98:0/3037886198 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feef00a2ed0 0x7feef00a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:28.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.326+0000 7fef077fe640 1 --2- 172.21.15.98:0/3037886198 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feef00a2ed0 0x7feef00a5390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feef8009b50 tx=0x7feef8005f90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:28.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.326+0000 7fef04ff9640 1 -- 172.21.15.98:0/3037886198 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7feefc0cfd90 con 0x7fef081092d0 2024-05-09T08:52:28.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.331+0000 7fef04ff9640 1 -- 172.21.15.98:0/3037886198 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feefc09c6a0 con 0x7fef081092d0 2024-05-09T08:52:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.883+0000 7fef0dcca640 1 -- 172.21.15.98:0/3037886198 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef08068570 con 0x7fef081092d0 2024-05-09T08:52:28.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.886+0000 7fef04ff9640 1 -- 172.21.15.98:0/3037886198 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11940 (secure 0 0 0) 0x7feefc09bfa0 con 0x7fef081092d0 2024-05-09T08:52:28.889 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:52:28.889 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:50:37.185367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/588012490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":588012490},{"type":"v1","addr":"172.21.15.105:6835","nonce":588012490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T08:52:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-05-09T08:52:28.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.906+0000 7fef0dcca640 1 -- 172.21.15.98:0/3037886198 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feef00a2ed0 msgr2=0x7feef00a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:28.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.906+0000 7fef0dcca640 1 --2- 172.21.15.98:0/3037886198 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feef00a2ed0 0x7feef00a5390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feef8009b50 tx=0x7feef8005f90 comp rx=0 tx=0).stop 2024-05-09T08:52:28.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.906+0000 7fef0dcca640 1 -- 172.21.15.98:0/3037886198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef081092d0 msgr2=0x7fef0811d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:28.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.906+0000 7fef0dcca640 1 --2- 172.21.15.98:0/3037886198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef081092d0 0x7fef0811d010 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feefc00e9c0 tx=0x7feefc00ee80 comp rx=0 tx=0).stop 2024-05-09T08:52:28.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.909+0000 7fef0dcca640 1 -- 172.21.15.98:0/3037886198 shutdown_connections 2024-05-09T08:52:28.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.909+0000 7fef0dcca640 1 --2- 172.21.15.98:0/3037886198 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feef00a2ed0 0x7feef00a5390 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:28.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.909+0000 7fef0dcca640 1 --2- 172.21.15.98:0/3037886198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef0810fe30 0x7fef08115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:28.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.909+0000 7fef0dcca640 1 --2- 172.21.15.98:0/3037886198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef081092d0 0x7fef0811d010 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:28.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.909+0000 7fef0dcca640 1 --2- 172.21.15.98:0/3037886198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef08108830 0x7fef0811cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:28.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.909+0000 7fef0dcca640 1 -- 172.21.15.98:0/3037886198 >> 172.21.15.98:0/3037886198 conn(0x7fef0806c770 msgr2=0x7fef080702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:28.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.909+0000 7fef0dcca640 1 -- 172.21.15.98:0/3037886198 shutdown_connections 2024-05-09T08:52:28.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:28.909+0000 7fef0dcca640 1 -- 172.21.15.98:0/3037886198 wait complete. 2024-05-09T08:52:28.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:52:29.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.081+0000 7f5cfceb3640 1 -- 172.21.15.98:0/3931780542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf8106710 msgr2=0x7f5cf8106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.081+0000 7f5cfceb3640 1 --2- 172.21.15.98:0/3931780542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf8106710 0x7f5cf8106af0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8009930 tx=0x7f5ce802a420 comp rx=0 tx=0).stop 2024-05-09T08:52:29.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.081+0000 7f5cfceb3640 1 -- 172.21.15.98:0/3931780542 shutdown_connections 2024-05-09T08:52:29.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.081+0000 7f5cfceb3640 1 --2- 172.21.15.98:0/3931780542 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cf81010c0 0x7f5cf810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.081+0000 7f5cfceb3640 1 --2- 172.21.15.98:0/3931780542 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cf8100700 0x7f5cf8100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.081+0000 7f5cfceb3640 1 --2- 172.21.15.98:0/3931780542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf8106710 0x7f5cf8106af0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.081+0000 7f5cfceb3640 1 -- 172.21.15.98:0/3931780542 >> 172.21.15.98:0/3931780542 conn(0x7f5cf8077830 msgr2=0x7f5cf8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.081+0000 7f5cfceb3640 1 -- 172.21.15.98:0/3931780542 shutdown_connections 2024-05-09T08:52:29.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.081+0000 7f5cfceb3640 1 -- 172.21.15.98:0/3931780542 wait complete. 2024-05-09T08:52:29.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.082+0000 7f5cfceb3640 1 Processor -- start 2024-05-09T08:52:29.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.082+0000 7f5cfceb3640 1 -- start start 2024-05-09T08:52:29.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.082+0000 7f5cfceb3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf8100700 0x7f5cf8196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.082+0000 7f5cfceb3640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cf81010c0 0x7f5cf81972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.082+0000 7f5cfceb3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cf8106710 0x7f5cf81977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.082+0000 7f5cfceb3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf8110b70 con 0x7f5cf8100700 2024-05-09T08:52:29.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.082+0000 7f5cfceb3640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf81109f0 con 0x7f5cf8106710 2024-05-09T08:52:29.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.082+0000 7f5cfceb3640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf8110cf0 con 0x7f5cf81010c0 2024-05-09T08:52:29.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.082+0000 7f5cf659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf8100700 0x7f5cf8196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.082+0000 7f5cf6d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cf8106710 0x7f5cf81977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.082+0000 7f5cf659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf8100700 0x7f5cf8196d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47128/0 (socket says 172.21.15.98:47128) 2024-05-09T08:52:29.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.082+0000 7f5cf659c640 1 -- 172.21.15.98:0/2562441481 learned_addr learned my addr 172.21.15.98:0/2562441481 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.083+0000 7f5ceffff640 1 --2- 172.21.15.98:0/2562441481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cf81010c0 0x7f5cf81972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.083+0000 7f5cf6d9d640 1 -- 172.21.15.98:0/2562441481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cf81010c0 msgr2=0x7f5cf81972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.083+0000 7f5cf6d9d640 1 --2- 172.21.15.98:0/2562441481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cf81010c0 0x7f5cf81972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.083+0000 7f5cf6d9d640 1 -- 172.21.15.98:0/2562441481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf8100700 msgr2=0x7f5cf8196d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.083+0000 7f5cf6d9d640 1 --2- 172.21.15.98:0/2562441481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf8100700 0x7f5cf8196d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.083+0000 7f5cf6d9d640 1 -- 172.21.15.98:0/2562441481 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf81a65d0 con 0x7f5cf8106710 2024-05-09T08:52:29.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.083+0000 7f5cf6d9d640 1 --2- 172.21.15.98:0/2562441481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cf8106710 0x7f5cf81977e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009fd0 tx=0x7f5cf000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.083+0000 7f5cef7fe640 1 -- 172.21.15.98:0/2562441481 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf000ce20 con 0x7f5cf8106710 2024-05-09T08:52:29.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.083+0000 7f5cfceb3640 1 -- 172.21.15.98:0/2562441481 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf81a68c0 con 0x7f5cf8106710 2024-05-09T08:52:29.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.083+0000 7f5cef7fe640 1 -- 172.21.15.98:0/2562441481 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cf0005020 con 0x7f5cf8106710 2024-05-09T08:52:29.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.083+0000 7f5cef7fe640 1 -- 172.21.15.98:0/2562441481 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0010720 con 0x7f5cf8106710 2024-05-09T08:52:29.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.083+0000 7f5cfceb3640 1 -- 172.21.15.98:0/2562441481 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf81a6e60 con 0x7f5cf8106710 2024-05-09T08:52:29.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.086+0000 7f5cffbbf740 1 Processor -- start 2024-05-09T08:52:29.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.086+0000 7f5cef7fe640 1 -- 172.21.15.98:0/2562441481 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5cf0004b20 con 0x7f5cf8106710 2024-05-09T08:52:29.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5cffbbf740 1 -- start start 2024-05-09T08:52:29.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5cffbbf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d655397710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5cffbbf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397c50 0x55d6553a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5cffbbf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6553a2320 0x55d6553a4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5cffbbf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d6552fcc40 con 0x55d6553a2320 2024-05-09T08:52:29.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5cffbbf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d6552a4580 con 0x55d655397330 2024-05-09T08:52:29.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5cffbbf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d6552f8c00 con 0x55d655397c50 2024-05-09T08:52:29.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5ceffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d655397710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5ceffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d655397710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:53120/0 (socket says 172.21.15.98:53120) 2024-05-09T08:52:29.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5ceffff640 1 -- 172.21.15.98:0/786062654 learned_addr learned my addr 172.21.15.98:0/786062654 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5ceffff640 1 --2- 172.21.15.98:0/786062654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6553a2320 0x55d6553a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5cf659c640 1 --2- 172.21.15.98:0/786062654 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397c50 0x55d6553a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5cef7fe640 1 --2- 172.21.15.98:0/2562441481 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5ce40a2d00 0x7f5ce40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5ceffff640 1 -- 172.21.15.98:0/786062654 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397c50 msgr2=0x55d6553a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.087+0000 7f5ceffff640 1 --2- 172.21.15.98:0/786062654 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397c50 0x55d6553a1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.088+0000 7f5ceffff640 1 -- 172.21.15.98:0/786062654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6553a2320 msgr2=0x55d6553a4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.088+0000 7f5cf6d9d640 1 --2- 172.21.15.98:0/2562441481 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5ce40a2d00 0x7f5ce40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.088+0000 7f5ceffff640 1 --2- 172.21.15.98:0/786062654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6553a2320 0x55d6553a4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.088+0000 7f5ceffff640 1 -- 172.21.15.98:0/786062654 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6550103f0 con 0x55d655397330 2024-05-09T08:52:29.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.088+0000 7f5ceffff640 1 --2- 172.21.15.98:0/786062654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d655397710 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0012790 tx=0x7f5ce0009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc56da32e2c5fa1f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.088+0000 7f5cecff9640 1 -- 172.21.15.98:0/786062654 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0012d10 con 0x55d655397330 2024-05-09T08:52:29.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.088+0000 7f5cecff9640 1 -- 172.21.15.98:0/786062654 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ce0010cb0 con 0x55d655397330 2024-05-09T08:52:29.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.088+0000 7f5cecff9640 1 -- 172.21.15.98:0/786062654 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce001ea50 con 0x55d655397330 2024-05-09T08:52:29.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.088+0000 7f5cef7fe640 1 -- 172.21.15.98:0/2562441481 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f5cf00cfe80 con 0x7f5cf8106710 2024-05-09T08:52:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.089+0000 7f5cffbbf740 1 -- 172.21.15.98:0/786062654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 msgr2=0x55d655397710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.089+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/786062654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d655397710 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0012790 tx=0x7f5ce0009700 comp rx=0 tx=0).stop 2024-05-09T08:52:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.089+0000 7f5cf6d9d640 1 --2- 172.21.15.98:0/2562441481 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5ce40a2d00 0x7f5ce40a51c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00cd2c0 tx=0x7f5cf0097c60 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.089+0000 7f5cffbbf740 1 -- 172.21.15.98:0/786062654 shutdown_connections 2024-05-09T08:52:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.089+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/786062654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6553a2320 0x55d6553a4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.089+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/786062654 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397c50 0x55d6553a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.089+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/786062654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d655397710 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.089+0000 7f5cffbbf740 1 -- 172.21.15.98:0/786062654 >> 172.21.15.98:0/786062654 conn(0x55d655393650 msgr2=0x55d655393a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.089+0000 7f5cffbbf740 1 -- 172.21.15.98:0/786062654 shutdown_connections 2024-05-09T08:52:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.089+0000 7f5cffbbf740 1 -- 172.21.15.98:0/786062654 wait complete. 2024-05-09T08:52:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.089+0000 7f5cffbbf740 1 -- start start 2024-05-09T08:52:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.089+0000 7f5cffbbf740 1 Processor -- start 2024-05-09T08:52:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cffbbf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d65539f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cffbbf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d655397c50 0x55d65539f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cffbbf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6553a2320 0x55d65539fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cffbbf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d6553a5110 con 0x55d655397c50 2024-05-09T08:52:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cffbbf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d6552b7c40 con 0x55d655397330 2024-05-09T08:52:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cffbbf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d6553a5290 con 0x55d6553a2320 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cffbbf740 10 client.0 _ll_register_callbacks cb 0x55d65500fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5ceffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d65539f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cf659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d655397c50 0x55d65539f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5ceffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d65539f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:53134/0 (socket says 172.21.15.98:53134) 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cf659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d655397c50 0x55d65539f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47144/0 (socket says 172.21.15.98:47144) 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5ceffff640 1 -- 172.21.15.98:0/2545944094 learned_addr learned my addr 172.21.15.98:0/2545944094 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5ceffff640 1 --2- 172.21.15.98:0/2545944094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6553a2320 0x55d65539fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cf659c640 1 -- 172.21.15.98:0/2545944094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6553a2320 msgr2=0x55d65539fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cf659c640 1 --2- 172.21.15.98:0/2545944094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6553a2320 0x55d65539fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cf659c640 1 -- 172.21.15.98:0/2545944094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 msgr2=0x55d65539f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cf659c640 1 --2- 172.21.15.98:0/2545944094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d65539f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cf659c640 1 -- 172.21.15.98:0/2545944094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6550da7d0 con 0x55d655397c50 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5ceffff640 1 --2- 172.21.15.98:0/2545944094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d65539f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cf659c640 1 --2- 172.21.15.98:0/2545944094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d655397c50 0x55d65539f800 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x55d6553988f0 tx=0x7f5ce80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7036a8626836de4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.090+0000 7f5cddffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:52:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.091+0000 7f5cddffb640 1 -- 172.21.15.98:0/2545944094 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8035040 con 0x55d655397c50 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.091+0000 7f5cddffb640 1 -- 172.21.15.98:0/2545944094 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f5ce8005450 con 0x55d655397c50 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.091+0000 7f5cffbbf740 1 -- 172.21.15.98:0/2545944094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ce0010cb0 con 0x55d655397c50 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.091+0000 7f5cddffb640 1 -- 172.21.15.98:0/2545944094 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5ce804f050 con 0x55d655397c50 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.091+0000 7f5cffbbf740 10 client.16272 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.091+0000 7f5cffbbf740 1 -- 172.21.15.98:0/2545944094 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d6553b1340 con 0x55d655397c50 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.091+0000 7f5cddffb640 1 -- 172.21.15.98:0/2545944094 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ce0010cb0 con 0x55d655397c50 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.091+0000 7f5cddffb640 1 -- 172.21.15.98:0/2545944094 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5ce8003780 con 0x55d655397c50 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.091+0000 7f5cffbbf740 10 client.16272 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.091+0000 7f5cffbbf740 10 client.16272 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.091+0000 7f5cffbbf740 20 client.16272 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.091+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/2545944094 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d6553b2800 0x55d6553b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.091+0000 7f5cffbbf740 4 client.16272 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.091+0000 7f5cffbbf740 1 -- 172.21.15.98:0/2545944094 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d65530a370 con 0x55d6553b2800 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.092+0000 7f5ceffff640 1 --2- 172.21.15.98:0/2545944094 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d6553b2800 0x55d6553b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.092+0000 7f5ceffff640 1 --2- 172.21.15.98:0/2545944094 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d6553b2800 0x55d6553b4bf0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7516b7a2788afcd2 server_cookie=f2209a71ac7883ce in_seq=0 out_seq=0 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.092+0000 7f5cddffb640 10 client.16272 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cddffb640 1 -- 172.21.15.98:0/2545944094 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d65530a370 con 0x55d6553b2800 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cddffb640 10 client.16272 handle_command_reply: tid=0 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cffbbf740 1 client.16272 shutdown 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cffbbf740 1 -- 172.21.15.98:0/2545944094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d655397c50 msgr2=0x55d65539f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/2545944094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d655397c50 0x55d65539f800 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x55d6553988f0 tx=0x7f5ce80043b0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cffbbf740 1 -- 172.21.15.98:0/2545944094 shutdown_connections 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/2545944094 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d6553b2800 0x55d6553b4bf0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/2545944094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6553a2320 0x55d65539fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/2545944094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d655397c50 0x55d65539f800 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cddffb640 0 client.16272 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/2545944094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d65539f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cffbbf740 1 -- 172.21.15.98:0/2545944094 >> 172.21.15.98:0/2545944094 conn(0x55d655393650 msgr2=0x55d655399450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cffbbf740 1 -- 172.21.15.98:0/2545944094 shutdown_connections 2024-05-09T08:52:29.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cffbbf740 1 -- 172.21.15.98:0/2545944094 wait complete. 2024-05-09T08:52:29.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.093+0000 7f5cffbbf740 20 client.16272 trim_cache size 0 max 16384 2024-05-09T08:52:29.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.112+0000 7f5cffbbf740 1 Processor -- start 2024-05-09T08:52:29.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.112+0000 7f5cffbbf740 1 -- start start 2024-05-09T08:52:29.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.112+0000 7f5cffbbf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397330 0x55d6553a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.112+0000 7f5cffbbf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d655397c50 0x55d6553ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.112+0000 7f5cffbbf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6553a2320 0x55d6553ba4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.112+0000 7f5cffbbf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf81b0f50 con 0x55d655397c50 2024-05-09T08:52:29.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.112+0000 7f5cffbbf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf81b2a90 con 0x55d6553a2320 2024-05-09T08:52:29.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.112+0000 7f5cffbbf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf81b1620 con 0x55d655397330 2024-05-09T08:52:29.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.112+0000 7f5cf659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d655397c50 0x55d6553ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.112+0000 7f5cf659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d655397c50 0x55d6553ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47150/0 (socket says 172.21.15.98:47150) 2024-05-09T08:52:29.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.112+0000 7f5cf659c640 1 -- 172.21.15.98:0/4088336485 learned_addr learned my addr 172.21.15.98:0/4088336485 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.112+0000 7f5ceffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397330 0x55d6553a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.113+0000 7f5ceffff640 1 --2- 172.21.15.98:0/4088336485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6553a2320 0x55d6553ba4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.113+0000 7f5cf659c640 1 -- 172.21.15.98:0/4088336485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397330 msgr2=0x55d6553a19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.113+0000 7f5cf659c640 1 --2- 172.21.15.98:0/4088336485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397330 0x55d6553a19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.113+0000 7f5cf659c640 1 -- 172.21.15.98:0/4088336485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6553a2320 msgr2=0x55d6553ba4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.113+0000 7f5cf659c640 1 --2- 172.21.15.98:0/4088336485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6553a2320 0x55d6553ba4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.113+0000 7f5cf659c640 1 -- 172.21.15.98:0/4088336485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6553b4f60 con 0x55d655397c50 2024-05-09T08:52:29.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.113+0000 7f5ceffff640 1 --2- 172.21.15.98:0/4088336485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397330 0x55d6553a19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:52:29.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.113+0000 7f5cf659c640 1 --2- 172.21.15.98:0/4088336485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d655397c50 0x55d6553ae000 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5ce802a940 tx=0x7f5ce8003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5642edd62f0ee296 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.113+0000 7f5cdffff640 1 -- 172.21.15.98:0/4088336485 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8035030 con 0x55d655397c50 2024-05-09T08:52:29.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.113+0000 7f5cdffff640 1 -- 172.21.15.98:0/4088336485 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ce8041070 con 0x55d655397c50 2024-05-09T08:52:29.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.113+0000 7f5cdffff640 1 -- 172.21.15.98:0/4088336485 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8005640 con 0x55d655397c50 2024-05-09T08:52:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.114+0000 7f5cffbbf740 1 -- 172.21.15.98:0/4088336485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d655397c50 msgr2=0x55d6553ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.114+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/4088336485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d655397c50 0x55d6553ae000 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5ce802a940 tx=0x7f5ce8003f00 comp rx=0 tx=0).stop 2024-05-09T08:52:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.114+0000 7f5cffbbf740 1 -- 172.21.15.98:0/4088336485 shutdown_connections 2024-05-09T08:52:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.114+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/4088336485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6553a2320 0x55d6553ba4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.114+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/4088336485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d655397c50 0x55d6553ae000 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.114+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/4088336485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397330 0x55d6553a19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.114+0000 7f5cffbbf740 1 -- 172.21.15.98:0/4088336485 >> 172.21.15.98:0/4088336485 conn(0x55d655393650 msgr2=0x55d655393a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.115+0000 7f5cffbbf740 1 -- 172.21.15.98:0/4088336485 shutdown_connections 2024-05-09T08:52:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.115+0000 7f5cffbbf740 1 -- 172.21.15.98:0/4088336485 wait complete. 2024-05-09T08:52:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.115+0000 7f5cffbbf740 1 -- start start 2024-05-09T08:52:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.115+0000 7f5cffbbf740 1 Processor -- start 2024-05-09T08:52:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.115+0000 7f5cffbbf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d6553a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.115+0000 7f5cffbbf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397c50 0x55d6553a6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.115+0000 7f5cffbbf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6553a2320 0x55d6553a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.115+0000 7f5cffbbf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf81b23c0 con 0x55d6553a2320 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.115+0000 7f5cffbbf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf81b1cf0 con 0x55d655397330 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.115+0000 7f5cffbbf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf81aef30 con 0x55d655397c50 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5cffbbf740 10 client.0 _ll_register_callbacks cb 0x7f5ce8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5ceffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d6553a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5cf659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397c50 0x55d6553a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5cf659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397c50 0x55d6553a6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55378/0 (socket says 172.21.15.98:55378) 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5cf659c640 1 -- 172.21.15.98:0/4177482872 learned_addr learned my addr 172.21.15.98:0/4177482872 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5ceffff640 1 --2- 172.21.15.98:0/4177482872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6553a2320 0x55d6553a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5cf659c640 1 -- 172.21.15.98:0/4177482872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 msgr2=0x55d6553a5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5cf659c640 1 --2- 172.21.15.98:0/4177482872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d6553a5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5cf659c640 1 -- 172.21.15.98:0/4177482872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6553a2320 msgr2=0x55d6553a68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5cf659c640 1 --2- 172.21.15.98:0/4177482872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6553a2320 0x55d6553a68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5cf659c640 1 -- 172.21.15.98:0/4177482872 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d655420540 con 0x55d655397c50 2024-05-09T08:52:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5cf659c640 1 --2- 172.21.15.98:0/4177482872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397c50 0x55d6553a6390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8005b90 tx=0x7f5ce80055f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6ca70b9ee368c74 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5cddffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:52:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5cddffb640 1 -- 172.21.15.98:0/4177482872 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8003800 con 0x55d655397c50 2024-05-09T08:52:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5cffbbf740 1 -- 172.21.15.98:0/4177482872 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ce8041070 con 0x55d655397c50 2024-05-09T08:52:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.116+0000 7f5cddffb640 1 -- 172.21.15.98:0/4177482872 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f5ce8005760 con 0x55d655397c50 2024-05-09T08:52:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.117+0000 7f5cddffb640 1 -- 172.21.15.98:0/4177482872 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5ce8031750 con 0x55d655397c50 2024-05-09T08:52:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.117+0000 7f5cffbbf740 10 client.25576 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.117+0000 7f5cffbbf740 1 -- 172.21.15.98:0/4177482872 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d655421750 con 0x55d655397c50 2024-05-09T08:52:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.117+0000 7f5cddffb640 1 -- 172.21.15.98:0/4177482872 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ce8041070 con 0x55d655397c50 2024-05-09T08:52:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.117+0000 7f5cddffb640 1 -- 172.21.15.98:0/4177482872 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5ce80046c0 con 0x55d655397c50 2024-05-09T08:52:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.117+0000 7f5cffbbf740 10 client.25576 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.117+0000 7f5cffbbf740 10 client.25576 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:52:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.117+0000 7f5cffbbf740 20 client.25576 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.117+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/4177482872 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d6553b2800 0x55d655423e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.117+0000 7f5cffbbf740 4 client.25576 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.117+0000 7f5cffbbf740 1 -- 172.21.15.98:0/4177482872 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d6553046c0 con 0x55d6553b2800 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.117+0000 7f5ceffff640 1 --2- 172.21.15.98:0/4177482872 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d6553b2800 0x55d655423e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cddffb640 10 client.25576 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5ceffff640 1 --2- 172.21.15.98:0/4177482872 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d6553b2800 0x55d655423e20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=590bc02a40bcadb9 server_cookie=276fc84b99cf3413 in_seq=0 out_seq=0 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cddffb640 1 -- 172.21.15.98:0/4177482872 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d6553046c0 con 0x55d6553b2800 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cddffb640 10 client.25576 handle_command_reply: tid=0 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cffbbf740 1 client.25576 shutdown 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cffbbf740 1 -- 172.21.15.98:0/4177482872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397c50 msgr2=0x55d6553a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/4177482872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397c50 0x55d6553a6390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8005b90 tx=0x7f5ce80055f0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cffbbf740 1 -- 172.21.15.98:0/4177482872 shutdown_connections 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/4177482872 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d6553b2800 0x55d655423e20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/4177482872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6553a2320 0x55d6553a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cddffb640 0 client.25576 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:52:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/4177482872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d655397c50 0x55d6553a6390 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cffbbf740 1 --2- 172.21.15.98:0/4177482872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d655397330 0x55d6553a5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cffbbf740 1 -- 172.21.15.98:0/4177482872 >> 172.21.15.98:0/4177482872 conn(0x55d655393650 msgr2=0x55d655391f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cffbbf740 1 -- 172.21.15.98:0/4177482872 shutdown_connections 2024-05-09T08:52:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cffbbf740 1 -- 172.21.15.98:0/4177482872 wait complete. 2024-05-09T08:52:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.118+0000 7f5cffbbf740 20 client.25576 trim_cache size 0 max 16384 2024-05-09T08:52:29.124 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:52:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.119+0000 7f5cfceb3640 1 -- 172.21.15.98:0/2562441481 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5ce40a2d00 msgr2=0x7f5ce40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.119+0000 7f5cfceb3640 1 --2- 172.21.15.98:0/2562441481 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5ce40a2d00 0x7f5ce40a51c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00cd2c0 tx=0x7f5cf0097c60 comp rx=0 tx=0).stop 2024-05-09T08:52:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.119+0000 7f5cfceb3640 1 -- 172.21.15.98:0/2562441481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cf8106710 msgr2=0x7f5cf81977e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.119+0000 7f5cfceb3640 1 --2- 172.21.15.98:0/2562441481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cf8106710 0x7f5cf81977e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009fd0 tx=0x7f5cf000ef10 comp rx=0 tx=0).stop 2024-05-09T08:52:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.119+0000 7f5cfceb3640 1 -- 172.21.15.98:0/2562441481 shutdown_connections 2024-05-09T08:52:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.119+0000 7f5cfceb3640 1 --2- 172.21.15.98:0/2562441481 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5ce40a2d00 0x7f5ce40a51c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.119+0000 7f5cfceb3640 1 --2- 172.21.15.98:0/2562441481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cf8106710 0x7f5cf81977e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.119+0000 7f5cfceb3640 1 --2- 172.21.15.98:0/2562441481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cf81010c0 0x7f5cf81972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.119+0000 7f5cfceb3640 1 --2- 172.21.15.98:0/2562441481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf8100700 0x7f5cf8196d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.119+0000 7f5cfceb3640 1 -- 172.21.15.98:0/2562441481 >> 172.21.15.98:0/2562441481 conn(0x7f5cf8077830 msgr2=0x7f5cf8102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.119+0000 7f5cfceb3640 1 -- 172.21.15.98:0/2562441481 shutdown_connections 2024-05-09T08:52:29.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.119+0000 7f5cfceb3640 1 -- 172.21.15.98:0/2562441481 wait complete. 2024-05-09T08:52:29.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:52:29.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.373+0000 7f8727fff640 1 -- 172.21.15.98:0/3671582503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872810fed0 msgr2=0x7f87281102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.373+0000 7f8727fff640 1 --2- 172.21.15.98:0/3671582503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872810fed0 0x7f87281102b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f871c009860 tx=0x7f871c02a3f0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.374+0000 7f8727fff640 1 -- 172.21.15.98:0/3671582503 shutdown_connections 2024-05-09T08:52:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.374+0000 7f8727fff640 1 --2- 172.21.15.98:0/3671582503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87281092b0 0x7f872811d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.374+0000 7f8727fff640 1 --2- 172.21.15.98:0/3671582503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8728108830 0x7f8728108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.374+0000 7f8727fff640 1 --2- 172.21.15.98:0/3671582503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872810fed0 0x7f87281102b0 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f871c009860 tx=0x7f871c02a3f0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.374+0000 7f8727fff640 1 -- 172.21.15.98:0/3671582503 >> 172.21.15.98:0/3671582503 conn(0x7f872806c7b0 msgr2=0x7f872806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.379+0000 7f8727fff640 1 -- 172.21.15.98:0/3671582503 shutdown_connections 2024-05-09T08:52:29.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.382+0000 7f8727fff640 1 -- 172.21.15.98:0/3671582503 wait complete. 2024-05-09T08:52:29.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.383+0000 7f8727fff640 1 Processor -- start 2024-05-09T08:52:29.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.383+0000 7f8727fff640 1 -- start start 2024-05-09T08:52:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.383+0000 7f8727fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8728108830 0x7f872811cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.383+0000 7f8727fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87281092b0 0x7f8728115ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.383+0000 7f8727fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8728116400 0x7f87281168b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.383+0000 7f8727fff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8728120930 con 0x7f87281092b0 2024-05-09T08:52:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.383+0000 7f8727fff640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f87281207b0 con 0x7f8728108830 2024-05-09T08:52:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.383+0000 7f8727fff640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8728120ab0 con 0x7f8728116400 2024-05-09T08:52:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.384+0000 7f8726ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8728108830 0x7f872811cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.384+0000 7f8726ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8728108830 0x7f872811cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:53164/0 (socket says 172.21.15.98:53164) 2024-05-09T08:52:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.384+0000 7f8726ffd640 1 -- 172.21.15.98:0/4033286935 learned_addr learned my addr 172.21.15.98:0/4033286935 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.385+0000 7f8726ffd640 1 -- 172.21.15.98:0/4033286935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8728116400 msgr2=0x7f87281168b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:52:29.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.385+0000 7f8726ffd640 1 --2- 172.21.15.98:0/4033286935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8728116400 0x7f87281168b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.385+0000 7f8726ffd640 1 -- 172.21.15.98:0/4033286935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87281092b0 msgr2=0x7f8728115ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:52:29.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.385+0000 7f8726ffd640 1 --2- 172.21.15.98:0/4033286935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87281092b0 0x7f8728115ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.385+0000 7f8726ffd640 1 -- 172.21.15.98:0/4033286935 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872811a9e0 con 0x7f8728108830 2024-05-09T08:52:29.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.386+0000 7f8726ffd640 1 --2- 172.21.15.98:0/4033286935 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8728108830 0x7f872811cdf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f871c009780 tx=0x7f871c0309b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.386+0000 7f872c8b8640 1 -- 172.21.15.98:0/4033286935 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c005550 con 0x7f8728108830 2024-05-09T08:52:29.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.386+0000 7f8727fff640 1 -- 172.21.15.98:0/4033286935 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872811ac70 con 0x7f8728108830 2024-05-09T08:52:29.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.387+0000 7f8727fff640 1 -- 172.21.15.98:0/4033286935 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f872811b1b0 con 0x7f8728108830 2024-05-09T08:52:29.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.392+0000 7f87305ca740 1 Processor -- start 2024-05-09T08:52:29.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.394+0000 7f87305ca740 1 -- start start 2024-05-09T08:52:29.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.400+0000 7f87305ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2d470 0x55a894a2d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.400+0000 7f87305ca740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2dd90 0x55a894a382b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.400+0000 7f87305ca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a3abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.400+0000 7f87305ca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a894992c40 con 0x55a894a387f0 2024-05-09T08:52:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.400+0000 7f87305ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a89493a580 con 0x55a894a2d470 2024-05-09T08:52:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.401+0000 7f87305ca740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a89498ec00 con 0x55a894a2dd90 2024-05-09T08:52:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.401+0000 7f872c8b8640 1 -- 172.21.15.98:0/4033286935 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f871c0318e0 con 0x7f8728108830 2024-05-09T08:52:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.401+0000 7f872c8b8640 1 -- 172.21.15.98:0/4033286935 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c03c420 con 0x7f8728108830 2024-05-09T08:52:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.401+0000 7f87277fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2d470 0x55a894a2d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.401+0000 7f87277fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2d470 0x55a894a2d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:53176/0 (socket says 172.21.15.98:53176) 2024-05-09T08:52:29.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.401+0000 7f87277fe640 1 -- 172.21.15.98:0/3432637018 learned_addr learned my addr 172.21.15.98:0/3432637018 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.401+0000 7f8726ffd640 1 --2- 172.21.15.98:0/3432637018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2dd90 0x55a894a382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.401+0000 7f872c8b8640 1 -- 172.21.15.98:0/4033286935 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f871c039040 con 0x7f8728108830 2024-05-09T08:52:29.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.402+0000 7f87277fe640 1 -- 172.21.15.98:0/3432637018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2dd90 msgr2=0x55a894a382b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.402+0000 7f87277fe640 1 --2- 172.21.15.98:0/3432637018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2dd90 0x55a894a382b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.402+0000 7f87267fc640 1 --2- 172.21.15.98:0/3432637018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a3abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.402+0000 7f87277fe640 1 -- 172.21.15.98:0/3432637018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 msgr2=0x55a894a3abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.402+0000 7f87277fe640 1 --2- 172.21.15.98:0/3432637018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a3abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.402+0000 7f87277fe640 1 -- 172.21.15.98:0/3432637018 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8946a63f0 con 0x55a894a2d470 2024-05-09T08:52:29.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.402+0000 7f87267fc640 1 --2- 172.21.15.98:0/3432637018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a3abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:52:29.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.404+0000 7f872c8b8640 1 --2- 172.21.15.98:0/4033286935 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f87100a2e70 0x7f87100a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.404+0000 7f87277fe640 1 --2- 172.21.15.98:0/3432637018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2d470 0x55a894a2d850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f871800ba00 tx=0x7f871800bec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e439e42f98840078 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.404+0000 7f872c8b8640 1 -- 172.21.15.98:0/4033286935 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f871c0f1860 con 0x7f8728108830 2024-05-09T08:52:29.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.405+0000 7f8726ffd640 1 --2- 172.21.15.98:0/4033286935 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f87100a2e70 0x7f87100a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.407+0000 7f8726ffd640 1 --2- 172.21.15.98:0/4033286935 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f87100a2e70 0x7f87100a5330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x55a894953f70 tx=0x7f871c0f7530 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.408+0000 7f8727fff640 1 -- 172.21.15.98:0/3432637018 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718016070 con 0x55a894a2d470 2024-05-09T08:52:29.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.408+0000 7f8727fff640 1 -- 172.21.15.98:0/3432637018 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8718005500 con 0x55a894a2d470 2024-05-09T08:52:29.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.408+0000 7f8727fff640 1 -- 172.21.15.98:0/3432637018 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718010460 con 0x55a894a2d470 2024-05-09T08:52:29.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.411+0000 7f87305ca740 1 -- 172.21.15.98:0/3432637018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2d470 msgr2=0x55a894a2d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.411+0000 7f87305ca740 1 --2- 172.21.15.98:0/3432637018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2d470 0x55a894a2d850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f871800ba00 tx=0x7f871800bec0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.411+0000 7f87305ca740 1 -- 172.21.15.98:0/3432637018 shutdown_connections 2024-05-09T08:52:29.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.411+0000 7f87305ca740 1 --2- 172.21.15.98:0/3432637018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a3abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.411+0000 7f87305ca740 1 --2- 172.21.15.98:0/3432637018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2dd90 0x55a894a382b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.411+0000 7f87305ca740 1 --2- 172.21.15.98:0/3432637018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2d470 0x55a894a2d850 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.411+0000 7f87305ca740 1 -- 172.21.15.98:0/3432637018 >> 172.21.15.98:0/3432637018 conn(0x55a894a29b20 msgr2=0x55a894a29f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.412+0000 7f87305ca740 1 -- 172.21.15.98:0/3432637018 shutdown_connections 2024-05-09T08:52:29.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.413+0000 7f87305ca740 1 -- 172.21.15.98:0/3432637018 wait complete. 2024-05-09T08:52:29.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.414+0000 7f87305ca740 1 -- start start 2024-05-09T08:52:29.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.414+0000 7f87305ca740 1 Processor -- start 2024-05-09T08:52:29.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.415+0000 7f87305ca740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2d470 0x55a894a357d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.415+0000 7f87305ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2dd90 0x55a894a35d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.415+0000 7f87305ca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a36250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.415+0000 7f87305ca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a894a3b300 con 0x55a894a387f0 2024-05-09T08:52:29.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.415+0000 7f87305ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a89494dc40 con 0x55a894a2dd90 2024-05-09T08:52:29.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.415+0000 7f87305ca740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a894a3b480 con 0x55a894a2d470 2024-05-09T08:52:29.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.415+0000 7f87305ca740 10 client.0 _ll_register_callbacks cb 0x55a8946a5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:29.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.420+0000 7f87277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a36250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.420+0000 7f87267fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2dd90 0x55a894a35d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.420+0000 7f87277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a36250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47184/0 (socket says 172.21.15.98:47184) 2024-05-09T08:52:29.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.420+0000 7f87277fe640 1 -- 172.21.15.98:0/272940941 learned_addr learned my addr 172.21.15.98:0/272940941 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.420+0000 7f87277fe640 1 --2- 172.21.15.98:0/272940941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2d470 0x55a894a357d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.422+0000 7f87277fe640 1 -- 172.21.15.98:0/272940941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2d470 msgr2=0x55a894a357d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.422+0000 7f87277fe640 1 --2- 172.21.15.98:0/272940941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2d470 0x55a894a357d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.422+0000 7f87277fe640 1 -- 172.21.15.98:0/272940941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2dd90 msgr2=0x55a894a35d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.422+0000 7f87277fe640 1 --2- 172.21.15.98:0/272940941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2dd90 0x55a894a35d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.422+0000 7f87277fe640 1 -- 172.21.15.98:0/272940941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8947707d0 con 0x55a894a387f0 2024-05-09T08:52:29.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.423+0000 7f87277fe640 1 --2- 172.21.15.98:0/272940941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a36250 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8718010580 tx=0x7f8718005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2363f45ee0f1a66a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.424+0000 7f8706ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:52:29.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.428+0000 7f8706ffd640 1 -- 172.21.15.98:0/272940941 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718016040 con 0x55a894a387f0 2024-05-09T08:52:29.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.428+0000 7f8706ffd640 1 -- 172.21.15.98:0/272940941 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f8718005130 con 0x55a894a387f0 2024-05-09T08:52:29.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.429+0000 7f87305ca740 1 -- 172.21.15.98:0/272940941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8718005500 con 0x55a894a387f0 2024-05-09T08:52:29.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.430+0000 7f8706ffd640 1 -- 172.21.15.98:0/272940941 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8718005330 con 0x55a894a387f0 2024-05-09T08:52:29.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.430+0000 7f87305ca740 10 client.16281 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:29.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.430+0000 7f87305ca740 1 -- 172.21.15.98:0/272940941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a894a47330 con 0x55a894a387f0 2024-05-09T08:52:29.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.431+0000 7f8706ffd640 1 -- 172.21.15.98:0/272940941 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8718005500 con 0x55a894a387f0 2024-05-09T08:52:29.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.432+0000 7f8706ffd640 1 -- 172.21.15.98:0/272940941 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8718010bf0 con 0x55a894a387f0 2024-05-09T08:52:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.432+0000 7f87305ca740 10 client.16281 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.432+0000 7f87305ca740 10 client.16281 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:52:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.432+0000 7f87305ca740 20 client.16281 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.432+0000 7f87305ca740 1 --2- 172.21.15.98:0/272940941 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a894a48c80 0x55a894a4b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.432+0000 7f87305ca740 4 client.16281 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:52:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.432+0000 7f87305ca740 1 -- 172.21.15.98:0/272940941 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8949a0370 con 0x55a894a48c80 2024-05-09T08:52:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.433+0000 7f87267fc640 1 --2- 172.21.15.98:0/272940941 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a894a48c80 0x55a894a4b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.436+0000 7f87267fc640 1 --2- 172.21.15.98:0/272940941 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a894a48c80 0x55a894a4b0a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2ccaf0f3cbf6f221 server_cookie=d57cdbc41a903e21 in_seq=0 out_seq=0 2024-05-09T08:52:29.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.436+0000 7f8706ffd640 10 client.16281 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:52:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.439+0000 7f8706ffd640 1 -- 172.21.15.98:0/272940941 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8949a0370 con 0x55a894a48c80 2024-05-09T08:52:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.439+0000 7f8706ffd640 10 client.16281 handle_command_reply: tid=0 2024-05-09T08:52:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.439+0000 7f87305ca740 1 client.16281 shutdown 2024-05-09T08:52:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.442+0000 7f87305ca740 1 -- 172.21.15.98:0/272940941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 msgr2=0x55a894a36250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.442+0000 7f87305ca740 1 --2- 172.21.15.98:0/272940941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a36250 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8718010580 tx=0x7f8718005a20 comp rx=0 tx=0).stop 2024-05-09T08:52:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.443+0000 7f87305ca740 1 -- 172.21.15.98:0/272940941 shutdown_connections 2024-05-09T08:52:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.443+0000 7f87305ca740 1 --2- 172.21.15.98:0/272940941 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a894a48c80 0x55a894a4b0a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.443+0000 7f87305ca740 1 --2- 172.21.15.98:0/272940941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a36250 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.443+0000 7f87305ca740 1 --2- 172.21.15.98:0/272940941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2dd90 0x55a894a35d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.443+0000 7f87305ca740 1 --2- 172.21.15.98:0/272940941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2d470 0x55a894a357d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.443+0000 7f87305ca740 1 -- 172.21.15.98:0/272940941 >> 172.21.15.98:0/272940941 conn(0x55a894a29b20 msgr2=0x55a894a2f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.443+0000 7f8706ffd640 0 client.16281 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:52:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.444+0000 7f87305ca740 1 -- 172.21.15.98:0/272940941 shutdown_connections 2024-05-09T08:52:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.444+0000 7f87305ca740 1 -- 172.21.15.98:0/272940941 wait complete. 2024-05-09T08:52:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.446+0000 7f87305ca740 20 client.16281 trim_cache size 0 max 16384 2024-05-09T08:52:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.474+0000 7f87305ca740 1 Processor -- start 2024-05-09T08:52:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.474+0000 7f87305ca740 1 -- start start 2024-05-09T08:52:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.474+0000 7f87305ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2d470 0x55a894a44250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.474+0000 7f87305ca740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2dd90 0x55a894a50970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.474+0000 7f87305ca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a50eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.474+0000 7f87305ca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f871400ff00 con 0x55a894a387f0 2024-05-09T08:52:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.474+0000 7f87305ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8714011a40 con 0x55a894a2d470 2024-05-09T08:52:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.474+0000 7f87305ca740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f87140105d0 con 0x55a894a2dd90 2024-05-09T08:52:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.475+0000 7f87277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a50eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.475+0000 7f87277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a50eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47198/0 (socket says 172.21.15.98:47198) 2024-05-09T08:52:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.475+0000 7f87277fe640 1 -- 172.21.15.98:0/2476714187 learned_addr learned my addr 172.21.15.98:0/2476714187 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.475+0000 7f87277fe640 1 --2- 172.21.15.98:0/2476714187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2d470 0x55a894a44250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.475+0000 7f87277fe640 1 -- 172.21.15.98:0/2476714187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2dd90 msgr2=0x55a894a50970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.475+0000 7f87277fe640 1 --2- 172.21.15.98:0/2476714187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2dd90 0x55a894a50970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.475+0000 7f87277fe640 1 -- 172.21.15.98:0/2476714187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2d470 msgr2=0x55a894a44250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.475+0000 7f87277fe640 1 --2- 172.21.15.98:0/2476714187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2d470 0x55a894a44250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.475+0000 7f87277fe640 1 -- 172.21.15.98:0/2476714187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a894a2e170 con 0x55a894a387f0 2024-05-09T08:52:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.476+0000 7f87277fe640 1 --2- 172.21.15.98:0/2476714187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a50eb0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f8718001eb0 tx=0x7f8718000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405bd0ec167470d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.477+0000 7f870d7fa640 1 -- 172.21.15.98:0/2476714187 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718016070 con 0x55a894a387f0 2024-05-09T08:52:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.477+0000 7f870d7fa640 1 -- 172.21.15.98:0/2476714187 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f871800d6b0 con 0x55a894a387f0 2024-05-09T08:52:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.477+0000 7f870d7fa640 1 -- 172.21.15.98:0/2476714187 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871800dcd0 con 0x55a894a387f0 2024-05-09T08:52:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.478+0000 7f87305ca740 1 -- 172.21.15.98:0/2476714187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 msgr2=0x55a894a50eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.478+0000 7f87305ca740 1 --2- 172.21.15.98:0/2476714187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a50eb0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f8718001eb0 tx=0x7f8718000bc0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.478+0000 7f87305ca740 1 -- 172.21.15.98:0/2476714187 shutdown_connections 2024-05-09T08:52:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.478+0000 7f87305ca740 1 --2- 172.21.15.98:0/2476714187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a387f0 0x55a894a50eb0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.478+0000 7f87305ca740 1 --2- 172.21.15.98:0/2476714187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2dd90 0x55a894a50970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.478+0000 7f87305ca740 1 --2- 172.21.15.98:0/2476714187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a2d470 0x55a894a44250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.478+0000 7f87305ca740 1 -- 172.21.15.98:0/2476714187 >> 172.21.15.98:0/2476714187 conn(0x55a894a29b20 msgr2=0x55a894a30a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.479+0000 7f87305ca740 1 -- 172.21.15.98:0/2476714187 shutdown_connections 2024-05-09T08:52:29.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.479+0000 7f87305ca740 1 -- 172.21.15.98:0/2476714187 wait complete. 2024-05-09T08:52:29.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.479+0000 7f87305ca740 1 -- start start 2024-05-09T08:52:29.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.480+0000 7f87305ca740 1 Processor -- start 2024-05-09T08:52:29.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.481+0000 7f87305ca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a2d470 0x55a894a40760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.481+0000 7f87305ca740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2dd90 0x55a894a40ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.481+0000 7f87305ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a387f0 0x55a894a31a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.481+0000 7f87305ca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8714011370 con 0x55a894a2d470 2024-05-09T08:52:29.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.481+0000 7f87305ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8714010ca0 con 0x55a894a387f0 2024-05-09T08:52:29.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.481+0000 7f87305ca740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f871400df90 con 0x55a894a2dd90 2024-05-09T08:52:29.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.481+0000 7f87305ca740 10 client.0 _ll_register_callbacks cb 0x7f871c0f7480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:29.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.481+0000 7f87277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a2d470 0x55a894a40760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.481+0000 7f87277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a2d470 0x55a894a40760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47206/0 (socket says 172.21.15.98:47206) 2024-05-09T08:52:29.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.481+0000 7f87277fe640 1 -- 172.21.15.98:0/4121160588 learned_addr learned my addr 172.21.15.98:0/4121160588 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.481+0000 7f87277fe640 1 --2- 172.21.15.98:0/4121160588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a387f0 0x55a894a31a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.482+0000 7f87277fe640 1 -- 172.21.15.98:0/4121160588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2dd90 msgr2=0x55a894a40ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.482+0000 7f87277fe640 1 --2- 172.21.15.98:0/4121160588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2dd90 0x55a894a40ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.482+0000 7f87277fe640 1 -- 172.21.15.98:0/4121160588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a387f0 msgr2=0x55a894a31a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.482+0000 7f87277fe640 1 --2- 172.21.15.98:0/4121160588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a387f0 0x55a894a31a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.482+0000 7f87277fe640 1 -- 172.21.15.98:0/4121160588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a894a32000 con 0x55a894a2d470 2024-05-09T08:52:29.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.482+0000 7f87277fe640 1 --2- 172.21.15.98:0/4121160588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a2d470 0x55a894a40760 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f871800b2e0 tx=0x7f87180051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3516deb80cd144b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.482+0000 7f8706ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:52:29.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.483+0000 7f8706ffd640 1 -- 172.21.15.98:0/4121160588 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718038040 con 0x55a894a2d470 2024-05-09T08:52:29.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.483+0000 7f8706ffd640 1 -- 172.21.15.98:0/4121160588 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f8718010510 con 0x55a894a2d470 2024-05-09T08:52:29.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.484+0000 7f87305ca740 1 -- 172.21.15.98:0/4121160588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f871800d6b0 con 0x55a894a2d470 2024-05-09T08:52:29.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.484+0000 7f8706ffd640 1 -- 172.21.15.98:0/4121160588 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8718024040 con 0x55a894a2d470 2024-05-09T08:52:29.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.484+0000 7f87305ca740 10 client.16287 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:29.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.484+0000 7f87305ca740 1 -- 172.21.15.98:0/4121160588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a894aaeaa0 con 0x55a894a2d470 2024-05-09T08:52:29.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.485+0000 7f8706ffd640 1 -- 172.21.15.98:0/4121160588 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f871800d6b0 con 0x55a894a2d470 2024-05-09T08:52:29.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.485+0000 7f8706ffd640 1 -- 172.21.15.98:0/4121160588 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f871803e030 con 0x55a894a2d470 2024-05-09T08:52:29.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.485+0000 7f87305ca740 10 client.16287 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:29.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.486+0000 7f87305ca740 10 client.16287 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:52:29.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.486+0000 7f87305ca740 20 client.16287 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:29.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.486+0000 7f87305ca740 1 --2- 172.21.15.98:0/4121160588 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a894aaf030 0x55a894ab1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.486+0000 7f87305ca740 4 client.16287 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:52:29.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.486+0000 7f87305ca740 1 -- 172.21.15.98:0/4121160588 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a89499a6c0 con 0x55a894aaf030 2024-05-09T08:52:29.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.486+0000 7f87267fc640 1 --2- 172.21.15.98:0/4121160588 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a894aaf030 0x55a894ab1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.488+0000 7f87267fc640 1 --2- 172.21.15.98:0/4121160588 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a894aaf030 0x55a894ab1450 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a28a80945bd79ab2 server_cookie=f88056fd81a0aed0 in_seq=0 out_seq=0 2024-05-09T08:52:29.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.488+0000 7f8706ffd640 10 client.16287 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:52:29.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.488+0000 7f8706ffd640 1 -- 172.21.15.98:0/4121160588 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a89499a6c0 con 0x55a894aaf030 2024-05-09T08:52:29.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.488+0000 7f8706ffd640 10 client.16287 handle_command_reply: tid=0 2024-05-09T08:52:29.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.488+0000 7f87305ca740 1 client.16287 shutdown 2024-05-09T08:52:29.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.488+0000 7f87305ca740 1 -- 172.21.15.98:0/4121160588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a2d470 msgr2=0x55a894a40760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.488+0000 7f87305ca740 1 --2- 172.21.15.98:0/4121160588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a2d470 0x55a894a40760 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f871800b2e0 tx=0x7f87180051d0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.489+0000 7f87305ca740 1 -- 172.21.15.98:0/4121160588 shutdown_connections 2024-05-09T08:52:29.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.489+0000 7f87305ca740 1 --2- 172.21.15.98:0/4121160588 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a894aaf030 0x55a894ab1450 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.489+0000 7f87305ca740 1 --2- 172.21.15.98:0/4121160588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a894a387f0 0x55a894a31a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.489+0000 7f87305ca740 1 --2- 172.21.15.98:0/4121160588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a894a2dd90 0x55a894a40ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.489+0000 7f87305ca740 1 --2- 172.21.15.98:0/4121160588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a894a2d470 0x55a894a40760 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.489+0000 7f8706ffd640 0 client.16287 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:52:29.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.489+0000 7f87305ca740 1 -- 172.21.15.98:0/4121160588 >> 172.21.15.98:0/4121160588 conn(0x55a894a29b20 msgr2=0x55a894a28450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.489+0000 7f87305ca740 1 -- 172.21.15.98:0/4121160588 shutdown_connections 2024-05-09T08:52:29.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.489+0000 7f87305ca740 1 -- 172.21.15.98:0/4121160588 wait complete. 2024-05-09T08:52:29.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.489+0000 7f87305ca740 20 client.16287 trim_cache size 0 max 16384 2024-05-09T08:52:29.489 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:52:29.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.491+0000 7f870e7fc640 1 -- 172.21.15.98:0/4033286935 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f87100a2e70 msgr2=0x7f87100a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.491+0000 7f870e7fc640 1 --2- 172.21.15.98:0/4033286935 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f87100a2e70 0x7f87100a5330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x55a894953f70 tx=0x7f871c0f7530 comp rx=0 tx=0).stop 2024-05-09T08:52:29.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.491+0000 7f870e7fc640 1 -- 172.21.15.98:0/4033286935 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8728108830 msgr2=0x7f872811cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.491+0000 7f870e7fc640 1 --2- 172.21.15.98:0/4033286935 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8728108830 0x7f872811cdf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f871c009780 tx=0x7f871c0309b0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.491+0000 7f870e7fc640 1 -- 172.21.15.98:0/4033286935 shutdown_connections 2024-05-09T08:52:29.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.491+0000 7f870e7fc640 1 --2- 172.21.15.98:0/4033286935 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f87100a2e70 0x7f87100a5330 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.491+0000 7f870e7fc640 1 --2- 172.21.15.98:0/4033286935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8728116400 0x7f87281168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.492+0000 7f870e7fc640 1 --2- 172.21.15.98:0/4033286935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87281092b0 0x7f8728115ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.492+0000 7f870e7fc640 1 --2- 172.21.15.98:0/4033286935 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8728108830 0x7f872811cdf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.492+0000 7f870e7fc640 1 -- 172.21.15.98:0/4033286935 >> 172.21.15.98:0/4033286935 conn(0x7f872806c7b0 msgr2=0x7f87280723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.492+0000 7f870e7fc640 1 -- 172.21.15.98:0/4033286935 shutdown_connections 2024-05-09T08:52:29.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.492+0000 7f870e7fc640 1 -- 172.21.15.98:0/4033286935 wait complete. 2024-05-09T08:52:29.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:52:29.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.723+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/1626841826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2c00957f0 msgr2=0x7fe2c00a1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.723+0000 7fe2cd3b3640 1 --2- 172.21.15.98:0/1626841826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2c00957f0 0x7fe2c00a1ac0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8067530 tx=0x7fe2c8050590 comp rx=0 tx=0).stop 2024-05-09T08:52:29.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.723+0000 7fe2c6ffd640 1 -- 172.21.15.98:0/1626841826 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8077970 con 0x7fe2c00957f0 2024-05-09T08:52:29.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.723+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/1626841826 shutdown_connections 2024-05-09T08:52:29.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.723+0000 7fe2cd3b3640 1 --2- 172.21.15.98:0/1626841826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2c00957f0 0x7fe2c00a1ac0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.723+0000 7fe2cd3b3640 1 --2- 172.21.15.98:0/1626841826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2c0094e50 0x7fe2c00952b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.723+0000 7fe2cd3b3640 1 --2- 172.21.15.98:0/1626841826 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe2c0099bc0 0x7fe2c0099fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.723+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/1626841826 >> 172.21.15.98:0/1626841826 conn(0x7fe2c000a740 msgr2=0x7fe2c000ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.725+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/1626841826 shutdown_connections 2024-05-09T08:52:29.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.725+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/1626841826 wait complete. 2024-05-09T08:52:29.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.726+0000 7fe2cd3b3640 1 Processor -- start 2024-05-09T08:52:29.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.726+0000 7fe2cd3b3640 1 -- start start 2024-05-09T08:52:29.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.727+0000 7fe2cd3b3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2c0094e50 0x7fe2c0129ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.727+0000 7fe2cd3b3640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe2c0099bc0 0x7fe2c012a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.727+0000 7fe2cd3b3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2c012e750 0x7fe2c012ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.727+0000 7fe2c77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe2c0099bc0 0x7fe2c012a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.727+0000 7fe2c77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe2c0099bc0 0x7fe2c012a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55434/0 (socket says 172.21.15.98:55434) 2024-05-09T08:52:29.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.727+0000 7fe2c77fe640 1 -- 172.21.15.98:0/4049826820 learned_addr learned my addr 172.21.15.98:0/4049826820 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.727+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/4049826820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c00a3ea0 con 0x7fe2c0094e50 2024-05-09T08:52:29.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.727+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/4049826820 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c00a3d20 con 0x7fe2c012e750 2024-05-09T08:52:29.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.727+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/4049826820 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c00a4020 con 0x7fe2c0099bc0 2024-05-09T08:52:29.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.728+0000 7fe2ccbb2640 1 --2- 172.21.15.98:0/4049826820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2c012e750 0x7fe2c012ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.728+0000 7fe2c77fe640 1 -- 172.21.15.98:0/4049826820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2c012e750 msgr2=0x7fe2c012ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.728+0000 7fe2c77fe640 1 --2- 172.21.15.98:0/4049826820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2c012e750 0x7fe2c012ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.728+0000 7fe2c77fe640 1 -- 172.21.15.98:0/4049826820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2c0094e50 msgr2=0x7fe2c0129ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.728+0000 7fe2c77fe640 1 --2- 172.21.15.98:0/4049826820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2c0094e50 0x7fe2c0129ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.728+0000 7fe2c77fe640 1 -- 172.21.15.98:0/4049826820 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2c012f240 con 0x7fe2c0099bc0 2024-05-09T08:52:29.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.729+0000 7fe2c77fe640 1 --2- 172.21.15.98:0/4049826820 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe2c0099bc0 0x7fe2c012a420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400b4a0 tx=0x7fe2b400b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.729+0000 7fe2c57fa640 1 -- 172.21.15.98:0/4049826820 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4014070 con 0x7fe2c0099bc0 2024-05-09T08:52:29.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.729+0000 7fe2c57fa640 1 -- 172.21.15.98:0/4049826820 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2b4010af0 con 0x7fe2c0099bc0 2024-05-09T08:52:29.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.729+0000 7fe2c57fa640 1 -- 172.21.15.98:0/4049826820 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4017d10 con 0x7fe2c0099bc0 2024-05-09T08:52:29.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.729+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/4049826820 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2c012f530 con 0x7fe2c0099bc0 2024-05-09T08:52:29.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.729+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/4049826820 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2c0136d50 con 0x7fe2c0099bc0 2024-05-09T08:52:29.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.733+0000 7fe2c57fa640 1 -- 172.21.15.98:0/4049826820 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe2b4017070 con 0x7fe2c0099bc0 2024-05-09T08:52:29.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.735+0000 7fe2d1403740 1 Processor -- start 2024-05-09T08:52:29.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.735+0000 7fe2d1403740 1 -- start start 2024-05-09T08:52:29.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2d1403740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca104f0 0x562fbca108d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2d1403740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca10e10 0x562fbca1afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2d1403740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca1b4e0 0x562fbca1d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2d1403740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562fbc975c40 con 0x562fbca10e10 2024-05-09T08:52:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2d1403740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562fbc91d580 con 0x562fbca1b4e0 2024-05-09T08:52:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2d1403740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562fbc971c00 con 0x562fbca104f0 2024-05-09T08:52:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2c7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca10e10 0x562fbca1afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2c7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca10e10 0x562fbca1afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47230/0 (socket says 172.21.15.98:47230) 2024-05-09T08:52:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2c7fff640 1 -- 172.21.15.98:0/1503418482 learned_addr learned my addr 172.21.15.98:0/1503418482 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2ccbb2640 1 --2- 172.21.15.98:0/1503418482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca104f0 0x562fbca108d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2c7fff640 1 -- 172.21.15.98:0/1503418482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca104f0 msgr2=0x562fbca108d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2c7fff640 1 --2- 172.21.15.98:0/1503418482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca104f0 0x562fbca108d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2c7fff640 1 -- 172.21.15.98:0/1503418482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca1b4e0 msgr2=0x562fbca1d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2c7fff640 1 --2- 172.21.15.98:0/1503418482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca1b4e0 0x562fbca1d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2c7fff640 1 -- 172.21.15.98:0/1503418482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fbc6893f0 con 0x562fbca10e10 2024-05-09T08:52:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.736+0000 7fe2ccbb2640 1 --2- 172.21.15.98:0/1503418482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca104f0 0x562fbca108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:52:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.737+0000 7fe2c7fff640 1 --2- 172.21.15.98:0/1503418482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca10e10 0x562fbca1afa0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc012140 tx=0x7fe2bc012600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee25cb512dfafed4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.737+0000 7fe2c57fa640 1 --2- 172.21.15.98:0/4049826820 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe2b80a2dc0 0x7fe2b80a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.737+0000 7fe2c57fa640 1 -- 172.21.15.98:0/4049826820 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fe2b40ce970 con 0x7fe2c0099bc0 2024-05-09T08:52:29.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.738+0000 7fe2ccbb2640 1 --2- 172.21.15.98:0/4049826820 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe2b80a2dc0 0x7fe2b80a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.738+0000 7fe2b27fc640 1 -- 172.21.15.98:0/1503418482 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc00d0d0 con 0x562fbca10e10 2024-05-09T08:52:29.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.738+0000 7fe2b27fc640 1 -- 172.21.15.98:0/1503418482 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2bc005500 con 0x562fbca10e10 2024-05-09T08:52:29.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.738+0000 7fe2b27fc640 1 -- 172.21.15.98:0/1503418482 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc004df0 con 0x562fbca10e10 2024-05-09T08:52:29.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.739+0000 7fe2ccbb2640 1 --2- 172.21.15.98:0/4049826820 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe2b80a2dc0 0x7fe2b80a5280 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8067120 tx=0x7fe2c8067700 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.739+0000 7fe2d1403740 1 -- 172.21.15.98:0/1503418482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca10e10 msgr2=0x562fbca1afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.739+0000 7fe2d1403740 1 --2- 172.21.15.98:0/1503418482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca10e10 0x562fbca1afa0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc012140 tx=0x7fe2bc012600 comp rx=0 tx=0).stop 2024-05-09T08:52:29.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.740+0000 7fe2d1403740 1 -- 172.21.15.98:0/1503418482 shutdown_connections 2024-05-09T08:52:29.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.740+0000 7fe2d1403740 1 --2- 172.21.15.98:0/1503418482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca1b4e0 0x562fbca1d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.740+0000 7fe2d1403740 1 --2- 172.21.15.98:0/1503418482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca10e10 0x562fbca1afa0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.740+0000 7fe2d1403740 1 --2- 172.21.15.98:0/1503418482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca104f0 0x562fbca108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.740+0000 7fe2d1403740 1 -- 172.21.15.98:0/1503418482 >> 172.21.15.98:0/1503418482 conn(0x562fbca0c810 msgr2=0x562fbca0cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.746+0000 7fe2d1403740 1 -- 172.21.15.98:0/1503418482 shutdown_connections 2024-05-09T08:52:29.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.748+0000 7fe2d1403740 1 -- 172.21.15.98:0/1503418482 wait complete. 2024-05-09T08:52:29.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.748+0000 7fe2d1403740 1 -- start start 2024-05-09T08:52:29.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.749+0000 7fe2d1403740 1 Processor -- start 2024-05-09T08:52:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.751+0000 7fe2d1403740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca104f0 0x562fbca184f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.751+0000 7fe2d1403740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca10e10 0x562fbca18a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.751+0000 7fe2d1403740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca1b4e0 0x562fbca18f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.751+0000 7fe2d1403740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562fbca2f280 con 0x562fbca1b4e0 2024-05-09T08:52:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.751+0000 7fe2d1403740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562fbc930c40 con 0x562fbca104f0 2024-05-09T08:52:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.751+0000 7fe2d1403740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562fbca2f400 con 0x562fbca10e10 2024-05-09T08:52:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.751+0000 7fe2d1403740 10 client.0 _ll_register_callbacks cb 0x562fbc688be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.751+0000 7fe2c77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca10e10 0x562fbca18a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.751+0000 7fe2c77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca10e10 0x562fbca18a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55458/0 (socket says 172.21.15.98:55458) 2024-05-09T08:52:29.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.751+0000 7fe2c77fe640 1 -- 172.21.15.98:0/2092437462 learned_addr learned my addr 172.21.15.98:0/2092437462 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.751+0000 7fe2c77fe640 1 -- 172.21.15.98:0/2092437462 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca104f0 msgr2=0x562fbca184f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.751+0000 7fe2c77fe640 1 --2- 172.21.15.98:0/2092437462 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca104f0 0x562fbca184f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.751+0000 7fe2c77fe640 1 -- 172.21.15.98:0/2092437462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca1b4e0 msgr2=0x562fbca18f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:52:29.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.751+0000 7fe2c77fe640 1 --2- 172.21.15.98:0/2092437462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca1b4e0 0x562fbca18f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.752+0000 7fe2c77fe640 1 -- 172.21.15.98:0/2092437462 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fbc7537d0 con 0x562fbca10e10 2024-05-09T08:52:29.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.752+0000 7fe2c77fe640 1 --2- 172.21.15.98:0/2092437462 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca10e10 0x562fbca18a30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe2b40cbec0 tx=0x7fe2b409b750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b65e527efb802e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.752+0000 7fe2a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:52:29.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.752+0000 7fe2a77fe640 1 -- 172.21.15.98:0/2092437462 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b401e050 con 0x562fbca10e10 2024-05-09T08:52:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.752+0000 7fe2d1403740 1 -- 172.21.15.98:0/2092437462 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2bc005500 con 0x562fbca10e10 2024-05-09T08:52:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.752+0000 7fe2a77fe640 1 -- 172.21.15.98:0/2092437462 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fe2b40cbf80 con 0x562fbca10e10 2024-05-09T08:52:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.753+0000 7fe2a77fe640 1 -- 172.21.15.98:0/2092437462 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe2b409bd30 con 0x562fbca10e10 2024-05-09T08:52:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.753+0000 7fe2d1403740 10 client.25594 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.753+0000 7fe2d1403740 1 -- 172.21.15.98:0/2092437462 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562fbca38800 con 0x562fbca10e10 2024-05-09T08:52:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.753+0000 7fe2a77fe640 1 -- 172.21.15.98:0/2092437462 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2bc005500 con 0x562fbca10e10 2024-05-09T08:52:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.754+0000 7fe2a77fe640 1 -- 172.21.15.98:0/2092437462 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe2b409b180 con 0x562fbca10e10 2024-05-09T08:52:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.754+0000 7fe2d1403740 10 client.25594 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.754+0000 7fe2d1403740 10 client.25594 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:52:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.754+0000 7fe2d1403740 20 client.25594 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:29.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.754+0000 7fe2d1403740 1 --2- 172.21.15.98:0/2092437462 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562fbca3a150 0x562fbca3c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.754+0000 7fe2d1403740 4 client.25594 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:52:29.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.754+0000 7fe2d1403740 1 -- 172.21.15.98:0/2092437462 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fbc983370 con 0x562fbca3a150 2024-05-09T08:52:29.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.755+0000 7fe2c7fff640 1 --2- 172.21.15.98:0/2092437462 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562fbca3a150 0x562fbca3c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.758+0000 7fe2c7fff640 1 --2- 172.21.15.98:0/2092437462 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562fbca3a150 0x562fbca3c5d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=16f0e73be7f8b02 server_cookie=1e963cefb89cce67 in_seq=0 out_seq=0 2024-05-09T08:52:29.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.758+0000 7fe2a77fe640 10 client.25594 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:52:29.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.760+0000 7fe2a77fe640 1 -- 172.21.15.98:0/2092437462 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562fbc983370 con 0x562fbca3a150 2024-05-09T08:52:29.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.760+0000 7fe2a77fe640 10 client.25594 handle_command_reply: tid=0 2024-05-09T08:52:29.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.760+0000 7fe2d1403740 1 client.25594 shutdown 2024-05-09T08:52:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.761+0000 7fe2d1403740 1 -- 172.21.15.98:0/2092437462 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca10e10 msgr2=0x562fbca18a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.761+0000 7fe2d1403740 1 --2- 172.21.15.98:0/2092437462 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca10e10 0x562fbca18a30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe2b40cbec0 tx=0x7fe2b409b750 comp rx=0 tx=0).stop 2024-05-09T08:52:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.761+0000 7fe2d1403740 1 -- 172.21.15.98:0/2092437462 shutdown_connections 2024-05-09T08:52:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.761+0000 7fe2d1403740 1 --2- 172.21.15.98:0/2092437462 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562fbca3a150 0x562fbca3c5d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.761+0000 7fe2d1403740 1 --2- 172.21.15.98:0/2092437462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca1b4e0 0x562fbca18f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.761+0000 7fe2d1403740 1 --2- 172.21.15.98:0/2092437462 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca10e10 0x562fbca18a30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.761+0000 7fe2d1403740 1 --2- 172.21.15.98:0/2092437462 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca104f0 0x562fbca184f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.761+0000 7fe2d1403740 1 -- 172.21.15.98:0/2092437462 >> 172.21.15.98:0/2092437462 conn(0x562fbca0c810 msgr2=0x562fbca0b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.761+0000 7fe2a77fe640 0 client.25594 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:52:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.761+0000 7fe2d1403740 1 -- 172.21.15.98:0/2092437462 shutdown_connections 2024-05-09T08:52:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.761+0000 7fe2d1403740 1 -- 172.21.15.98:0/2092437462 wait complete. 2024-05-09T08:52:29.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.761+0000 7fe2d1403740 20 client.25594 trim_cache size 0 max 16384 2024-05-09T08:52:29.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.789+0000 7fe2d1403740 1 Processor -- start 2024-05-09T08:52:29.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.789+0000 7fe2d1403740 1 -- start start 2024-05-09T08:52:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.789+0000 7fe2d1403740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca104f0 0x562fbca1ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.789+0000 7fe2d1403740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca10e10 0x562fbca18d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.789+0000 7fe2d1403740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca1b4e0 0x562fbca48a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.789+0000 7fe2d1403740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac00ff10 con 0x562fbca1b4e0 2024-05-09T08:52:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.790+0000 7fe2d1403740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac011a50 con 0x562fbca10e10 2024-05-09T08:52:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.790+0000 7fe2d1403740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac0105e0 con 0x562fbca104f0 2024-05-09T08:52:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.790+0000 7fe2c7fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca104f0 0x562fbca1ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.790+0000 7fe2c7fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca104f0 0x562fbca1ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55474/0 (socket says 172.21.15.98:55474) 2024-05-09T08:52:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.790+0000 7fe2c7fff640 1 -- 172.21.15.98:0/196310880 learned_addr learned my addr 172.21.15.98:0/196310880 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.790+0000 7fe2ccbb2640 1 --2- 172.21.15.98:0/196310880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca1b4e0 0x562fbca48a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.790+0000 7fe2c77fe640 1 --2- 172.21.15.98:0/196310880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca10e10 0x562fbca18d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.791+0000 7fe2c7fff640 1 -- 172.21.15.98:0/196310880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca10e10 msgr2=0x562fbca18d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.791+0000 7fe2c7fff640 1 --2- 172.21.15.98:0/196310880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca10e10 0x562fbca18d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.791+0000 7fe2c7fff640 1 -- 172.21.15.98:0/196310880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca1b4e0 msgr2=0x562fbca48a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.791+0000 7fe2c7fff640 1 --2- 172.21.15.98:0/196310880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca1b4e0 0x562fbca48a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.791+0000 7fe2c7fff640 1 -- 172.21.15.98:0/196310880 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fbca0da40 con 0x562fbca104f0 2024-05-09T08:52:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.791+0000 7fe2ccbb2640 1 --2- 172.21.15.98:0/196310880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca1b4e0 0x562fbca48a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:52:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.791+0000 7fe2c7fff640 1 --2- 172.21.15.98:0/196310880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca104f0 0x562fbca1ac20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc012140 tx=0x7fe2bc00d310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f81a7169bfb1d745 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.791+0000 7fe2b1ffb640 1 -- 172.21.15.98:0/196310880 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc02b070 con 0x562fbca104f0 2024-05-09T08:52:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.791+0000 7fe2b1ffb640 1 -- 172.21.15.98:0/196310880 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2bc004510 con 0x562fbca104f0 2024-05-09T08:52:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.792+0000 7fe2d1403740 1 -- 172.21.15.98:0/196310880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca104f0 msgr2=0x562fbca1ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.792+0000 7fe2d1403740 1 --2- 172.21.15.98:0/196310880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca104f0 0x562fbca1ac20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc012140 tx=0x7fe2bc00d310 comp rx=0 tx=0).stop 2024-05-09T08:52:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.793+0000 7fe2d1403740 1 -- 172.21.15.98:0/196310880 shutdown_connections 2024-05-09T08:52:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.793+0000 7fe2d1403740 1 --2- 172.21.15.98:0/196310880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca1b4e0 0x562fbca48a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.793+0000 7fe2d1403740 1 --2- 172.21.15.98:0/196310880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca10e10 0x562fbca18d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.793+0000 7fe2d1403740 1 --2- 172.21.15.98:0/196310880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca104f0 0x562fbca1ac20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.793+0000 7fe2d1403740 1 -- 172.21.15.98:0/196310880 >> 172.21.15.98:0/196310880 conn(0x562fbca0c810 msgr2=0x562fbca0b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.793+0000 7fe2d1403740 1 -- 172.21.15.98:0/196310880 shutdown_connections 2024-05-09T08:52:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.793+0000 7fe2d1403740 1 -- 172.21.15.98:0/196310880 wait complete. 2024-05-09T08:52:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.794+0000 7fe2d1403740 1 -- start start 2024-05-09T08:52:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.794+0000 7fe2d1403740 1 Processor -- start 2024-05-09T08:52:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.797+0000 7fe2d1403740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca104f0 0x562fbca48d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.797+0000 7fe2d1403740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca10e10 0x562fbca46120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.797+0000 7fe2d1403740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca1b4e0 0x562fbca46660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.797+0000 7fe2d1403740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac011380 con 0x562fbca104f0 2024-05-09T08:52:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.797+0000 7fe2d1403740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac010cb0 con 0x562fbca10e10 2024-05-09T08:52:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.797+0000 7fe2d1403740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac00def0 con 0x562fbca1b4e0 2024-05-09T08:52:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.797+0000 7fe2d1403740 10 client.0 _ll_register_callbacks cb 0x7fe2b4017eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.797+0000 7fe2c77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca10e10 0x562fbca46120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.797+0000 7fe2c77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca10e10 0x562fbca46120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:53268/0 (socket says 172.21.15.98:53268) 2024-05-09T08:52:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.797+0000 7fe2c77fe640 1 -- 172.21.15.98:0/2002190508 learned_addr learned my addr 172.21.15.98:0/2002190508 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.798+0000 7fe2c77fe640 1 -- 172.21.15.98:0/2002190508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca1b4e0 msgr2=0x562fbca46660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:52:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.798+0000 7fe2c77fe640 1 --2- 172.21.15.98:0/2002190508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca1b4e0 0x562fbca46660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.798+0000 7fe2c77fe640 1 -- 172.21.15.98:0/2002190508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca104f0 msgr2=0x562fbca48d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.798+0000 7fe2c77fe640 1 --2- 172.21.15.98:0/2002190508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca104f0 0x562fbca48d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.798+0000 7fe2c77fe640 1 -- 172.21.15.98:0/2002190508 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fbca46d00 con 0x562fbca10e10 2024-05-09T08:52:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.798+0000 7fe2c77fe640 1 --2- 172.21.15.98:0/2002190508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca10e10 0x562fbca46120 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe2b409b620 tx=0x7fe2b4017570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18ed06b67ef3af79 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.798+0000 7fe2a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:52:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.798+0000 7fe2a77fe640 1 -- 172.21.15.98:0/2002190508 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b401e040 con 0x562fbca10e10 2024-05-09T08:52:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.798+0000 7fe2a77fe640 1 -- 172.21.15.98:0/2002190508 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fe2b409b180 con 0x562fbca10e10 2024-05-09T08:52:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.799+0000 7fe2d1403740 1 -- 172.21.15.98:0/2002190508 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2bc004510 con 0x562fbca10e10 2024-05-09T08:52:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.799+0000 7fe2a77fe640 1 -- 172.21.15.98:0/2002190508 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe2b40a00e0 con 0x562fbca10e10 2024-05-09T08:52:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.799+0000 7fe2d1403740 10 client.25649 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.799+0000 7fe2d1403740 1 -- 172.21.15.98:0/2002190508 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562fbcaa3240 con 0x562fbca10e10 2024-05-09T08:52:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.800+0000 7fe2a77fe640 1 -- 172.21.15.98:0/2002190508 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2bc004510 con 0x562fbca10e10 2024-05-09T08:52:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.800+0000 7fe2a77fe640 1 -- 172.21.15.98:0/2002190508 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe2b409abf0 con 0x562fbca10e10 2024-05-09T08:52:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.800+0000 7fe2d1403740 10 client.25649 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.800+0000 7fe2d1403740 10 client.25649 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:52:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.800+0000 7fe2d1403740 20 client.25649 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.800+0000 7fe2d1403740 1 --2- 172.21.15.98:0/2002190508 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562fbcaa3920 0x562fbcaa5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.800+0000 7fe2d1403740 4 client.25649 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:52:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.800+0000 7fe2d1403740 1 -- 172.21.15.98:0/2002190508 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562fbc97d6c0 con 0x562fbcaa3920 2024-05-09T08:52:29.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.802+0000 7fe2c7fff640 1 --2- 172.21.15.98:0/2002190508 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562fbcaa3920 0x562fbcaa5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:29.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.803+0000 7fe2a77fe640 10 client.25649 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:52:29.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.803+0000 7fe2c7fff640 1 --2- 172.21.15.98:0/2002190508 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562fbcaa3920 0x562fbcaa5d80 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bd0a392ee03a63af server_cookie=75534fa68bd8fa38 in_seq=0 out_seq=0 2024-05-09T08:52:29.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.804+0000 7fe2a77fe640 1 -- 172.21.15.98:0/2002190508 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562fbc97d6c0 con 0x562fbcaa3920 2024-05-09T08:52:29.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.804+0000 7fe2a77fe640 10 client.25649 handle_command_reply: tid=0 2024-05-09T08:52:29.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.804+0000 7fe2d1403740 1 client.25649 shutdown 2024-05-09T08:52:29.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.805+0000 7fe2d1403740 1 -- 172.21.15.98:0/2002190508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca10e10 msgr2=0x562fbca46120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.805+0000 7fe2d1403740 1 --2- 172.21.15.98:0/2002190508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca10e10 0x562fbca46120 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe2b409b620 tx=0x7fe2b4017570 comp rx=0 tx=0).stop 2024-05-09T08:52:29.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.805+0000 7fe2d1403740 1 -- 172.21.15.98:0/2002190508 shutdown_connections 2024-05-09T08:52:29.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.805+0000 7fe2d1403740 1 --2- 172.21.15.98:0/2002190508 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562fbcaa3920 0x562fbcaa5d80 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.805+0000 7fe2d1403740 1 --2- 172.21.15.98:0/2002190508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562fbca1b4e0 0x562fbca46660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.805+0000 7fe2d1403740 1 --2- 172.21.15.98:0/2002190508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562fbca10e10 0x562fbca46120 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.805+0000 7fe2d1403740 1 --2- 172.21.15.98:0/2002190508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fbca104f0 0x562fbca48d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.805+0000 7fe2d1403740 1 -- 172.21.15.98:0/2002190508 >> 172.21.15.98:0/2002190508 conn(0x562fbca0c810 msgr2=0x562fbca376e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.805+0000 7fe2a77fe640 0 client.25649 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:52:29.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.805+0000 7fe2d1403740 1 -- 172.21.15.98:0/2002190508 shutdown_connections 2024-05-09T08:52:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.808+0000 7fe2d1403740 1 -- 172.21.15.98:0/2002190508 wait complete. 2024-05-09T08:52:29.808 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:52:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.809+0000 7fe2d1403740 20 client.25649 trim_cache size 0 max 16384 2024-05-09T08:52:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.809+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/4049826820 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe2b80a2dc0 msgr2=0x7fe2b80a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.809+0000 7fe2cd3b3640 1 --2- 172.21.15.98:0/4049826820 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe2b80a2dc0 0x7fe2b80a5280 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8067120 tx=0x7fe2c8067700 comp rx=0 tx=0).stop 2024-05-09T08:52:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.810+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/4049826820 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe2c0099bc0 msgr2=0x7fe2c012a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.810+0000 7fe2cd3b3640 1 --2- 172.21.15.98:0/4049826820 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe2c0099bc0 0x7fe2c012a420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400b4a0 tx=0x7fe2b400b960 comp rx=0 tx=0).stop 2024-05-09T08:52:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.811+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/4049826820 shutdown_connections 2024-05-09T08:52:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.811+0000 7fe2cd3b3640 1 --2- 172.21.15.98:0/4049826820 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe2b80a2dc0 0x7fe2b80a5280 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.811+0000 7fe2cd3b3640 1 --2- 172.21.15.98:0/4049826820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2c012e750 0x7fe2c012ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.811+0000 7fe2cd3b3640 1 --2- 172.21.15.98:0/4049826820 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe2c0099bc0 0x7fe2c012a420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.811+0000 7fe2cd3b3640 1 --2- 172.21.15.98:0/4049826820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2c0094e50 0x7fe2c0129ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.811+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/4049826820 >> 172.21.15.98:0/4049826820 conn(0x7fe2c000a740 msgr2=0x7fe2c0093d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.811+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/4049826820 shutdown_connections 2024-05-09T08:52:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:29.812+0000 7fe2cd3b3640 1 -- 172.21.15.98:0/4049826820 wait complete. 2024-05-09T08:52:29.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:52:30.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:29 smithi098 ceph-mon[30092]: pgmap v58: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 759 MiB/s wr, 1.93k op/s 2024-05-09T08:52:30.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:29 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3037886198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:52:30.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:29 smithi142 ceph-mon[34625]: pgmap v58: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 759 MiB/s wr, 1.93k op/s 2024-05-09T08:52:30.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:29 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3037886198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:52:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.199+0000 7f5db759e640 1 -- 172.21.15.98:0/691812007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8100690 msgr2=0x7f5db8100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.199+0000 7f5db759e640 1 --2- 172.21.15.98:0/691812007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8100690 0x7f5db8100a70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5db001a910 tx=0x7f5db003b4f0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.199+0000 7f5db759e640 1 -- 172.21.15.98:0/691812007 shutdown_connections 2024-05-09T08:52:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.199+0000 7f5db759e640 1 --2- 172.21.15.98:0/691812007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5db8105a90 0x7f5db8105f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.199+0000 7f5db759e640 1 --2- 172.21.15.98:0/691812007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5db8100fb0 0x7f5db8105470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.199+0000 7f5db759e640 1 --2- 172.21.15.98:0/691812007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8100690 0x7f5db8100a70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.199+0000 7f5db759e640 1 -- 172.21.15.98:0/691812007 >> 172.21.15.98:0/691812007 conn(0x7f5db80fc0a0 msgr2=0x7f5db80fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.203+0000 7f5db759e640 1 -- 172.21.15.98:0/691812007 shutdown_connections 2024-05-09T08:52:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.207+0000 7f5db759e640 1 -- 172.21.15.98:0/691812007 wait complete. 2024-05-09T08:52:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.209+0000 7f5db759e640 1 Processor -- start 2024-05-09T08:52:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.209+0000 7f5db759e640 1 -- start start 2024-05-09T08:52:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.209+0000 7f5db759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5db8100690 0x7f5db81149e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.209+0000 7f5db759e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5db8100fb0 0x7f5db8110aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.209+0000 7f5db759e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8105a90 0x7f5db8110fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.209+0000 7f5db759e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5db8109480 con 0x7f5db8100690 2024-05-09T08:52:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.209+0000 7f5db759e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5db8109300 con 0x7f5db8100fb0 2024-05-09T08:52:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.209+0000 7f5db759e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5db8109600 con 0x7f5db8105a90 2024-05-09T08:52:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.211+0000 7f5db6d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8105a90 0x7f5db8110fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.211+0000 7f5db6d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8105a90 0x7f5db8110fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55496/0 (socket says 172.21.15.98:55496) 2024-05-09T08:52:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.211+0000 7f5db6d9d640 1 -- 172.21.15.98:0/2010940064 learned_addr learned my addr 172.21.15.98:0/2010940064 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.211+0000 7f5db659c640 1 --2- 172.21.15.98:0/2010940064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5db8100690 0x7f5db81149e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.211+0000 7f5db659c640 1 -- 172.21.15.98:0/2010940064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8105a90 msgr2=0x7f5db8110fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.211+0000 7f5db659c640 1 --2- 172.21.15.98:0/2010940064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8105a90 0x7f5db8110fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.212+0000 7f5db659c640 1 -- 172.21.15.98:0/2010940064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5db8100fb0 msgr2=0x7f5db8110aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:52:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.212+0000 7f5db659c640 1 --2- 172.21.15.98:0/2010940064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5db8100fb0 0x7f5db8110aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.212+0000 7f5db659c640 1 -- 172.21.15.98:0/2010940064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db8111710 con 0x7f5db8100690 2024-05-09T08:52:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.212+0000 7f5db6d9d640 1 --2- 172.21.15.98:0/2010940064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8105a90 0x7f5db8110fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:52:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.212+0000 7f5db659c640 1 --2- 172.21.15.98:0/2010940064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5db8100690 0x7f5db81149e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f5db003bae0 tx=0x7f5db0003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.223+0000 7f5daf7fe640 1 -- 172.21.15.98:0/2010940064 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db0050070 con 0x7f5db8100690 2024-05-09T08:52:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.223+0000 7f5daf7fe640 1 -- 172.21.15.98:0/2010940064 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5db0003c20 con 0x7f5db8100690 2024-05-09T08:52:30.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.224+0000 7f5daf7fe640 1 -- 172.21.15.98:0/2010940064 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db0040890 con 0x7f5db8100690 2024-05-09T08:52:30.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.225+0000 7f5db759e640 1 -- 172.21.15.98:0/2010940064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db81a24a0 con 0x7f5db8100690 2024-05-09T08:52:30.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.225+0000 7f5db759e640 1 -- 172.21.15.98:0/2010940064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db81a2850 con 0x7f5db8100690 2024-05-09T08:52:30.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.229+0000 7f5dbf9c3740 1 Processor -- start 2024-05-09T08:52:30.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.231+0000 7f5dbf9c3740 1 -- start start 2024-05-09T08:52:30.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.234+0000 7f5dbf9c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f605520 0x55d05f605900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.234+0000 7f5dbf9c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605e40 0x55d05f60ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.235+0000 7f5dbf9c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f610510 0x55d05f612900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.235+0000 7f5dbf9c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d05f56ac40 con 0x55d05f610510 2024-05-09T08:52:30.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.235+0000 7f5dbf9c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d05f512580 con 0x55d05f605520 2024-05-09T08:52:30.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.235+0000 7f5db6d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f605520 0x55d05f605900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.235+0000 7f5db6d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f605520 0x55d05f605900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:53312/0 (socket says 172.21.15.98:53312) 2024-05-09T08:52:30.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.235+0000 7f5db6d9d640 1 -- 172.21.15.98:0/2399532088 learned_addr learned my addr 172.21.15.98:0/2399532088 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:30.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.235+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2399532088 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d05f566c00 con 0x55d05f605e40 2024-05-09T08:52:30.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:29 smithi105 ceph-mon[34498]: pgmap v58: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 759 MiB/s wr, 1.93k op/s 2024-05-09T08:52:30.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:29 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3037886198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:52:30.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.235+0000 7f5daf7fe640 1 -- 172.21.15.98:0/2010940064 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5db0053070 con 0x7f5db8100690 2024-05-09T08:52:30.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.237+0000 7f5db5d9b640 1 --2- 172.21.15.98:0/2399532088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f610510 0x55d05f612900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.237+0000 7f5db5d9b640 1 -- 172.21.15.98:0/2399532088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605e40 msgr2=0x55d05f60ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:52:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.237+0000 7f5db5d9b640 1 --2- 172.21.15.98:0/2399532088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605e40 0x55d05f60ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.237+0000 7f5db5d9b640 1 -- 172.21.15.98:0/2399532088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f605520 msgr2=0x55d05f605900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.237+0000 7f5db5d9b640 1 --2- 172.21.15.98:0/2399532088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f605520 0x55d05f605900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.238+0000 7f5db5d9b640 1 -- 172.21.15.98:0/2399532088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d05f27e3f0 con 0x55d05f610510 2024-05-09T08:52:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.238+0000 7f5db6d9d640 1 --2- 172.21.15.98:0/2399532088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f605520 0x55d05f605900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:52:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.238+0000 7f5db5d9b640 1 --2- 172.21.15.98:0/2399532088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f610510 0x55d05f612900 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f5db81122c0 tx=0x7f5da0009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43bdd392b8246bee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.238+0000 7f5daf7fe640 1 --2- 172.21.15.98:0/2010940064 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5da40a2d60 0x7f5da40a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.239+0000 7f5daf7fe640 1 -- 172.21.15.98:0/2010940064 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f5db0100cd0 con 0x7f5db8100690 2024-05-09T08:52:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.240+0000 7f5db6d9d640 1 --2- 172.21.15.98:0/2010940064 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5da40a2d60 0x7f5da40a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.240+0000 7f5dacff9640 1 -- 172.21.15.98:0/2399532088 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da0015020 con 0x55d05f610510 2024-05-09T08:52:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.240+0000 7f5dacff9640 1 -- 172.21.15.98:0/2399532088 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5da0011d10 con 0x55d05f610510 2024-05-09T08:52:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.241+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2399532088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f610510 msgr2=0x55d05f612900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.241+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2399532088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f610510 0x55d05f612900 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f5db81122c0 tx=0x7f5da0009b00 comp rx=0 tx=0).stop 2024-05-09T08:52:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.243+0000 7f5db6d9d640 1 --2- 172.21.15.98:0/2010940064 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5da40a2d60 0x7f5da40a5220 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5da8011490 tx=0x7f5da8009660 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.243+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2399532088 shutdown_connections 2024-05-09T08:52:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.243+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2399532088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f610510 0x55d05f612900 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.243+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2399532088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605e40 0x55d05f60ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.243+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2399532088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f605520 0x55d05f605900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.243+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2399532088 >> 172.21.15.98:0/2399532088 conn(0x55d05f601840 msgr2=0x55d05f601c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.246+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2399532088 shutdown_connections 2024-05-09T08:52:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.251+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2399532088 wait complete. 2024-05-09T08:52:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.251+0000 7f5dbf9c3740 1 -- start start 2024-05-09T08:52:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.252+0000 7f5dbf9c3740 1 Processor -- start 2024-05-09T08:52:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.252+0000 7f5dbf9c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605520 0x55d05f60ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.252+0000 7f5dbf9c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605e40 0x55d05f60d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.252+0000 7f5dbf9c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f610510 0x55d05f60da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.252+0000 7f5dbf9c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d05f613350 con 0x55d05f605e40 2024-05-09T08:52:30.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.252+0000 7f5dbf9c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d05f525c40 con 0x55d05f610510 2024-05-09T08:52:30.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.252+0000 7f5dbf9c3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d05f6134d0 con 0x55d05f605520 2024-05-09T08:52:30.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.252+0000 7f5dbf9c3740 10 client.0 _ll_register_callbacks cb 0x55d05f27dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:30.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.253+0000 7f5db659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605520 0x55d05f60ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.253+0000 7f5db659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605520 0x55d05f60ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55510/0 (socket says 172.21.15.98:55510) 2024-05-09T08:52:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.253+0000 7f5db659c640 1 -- 172.21.15.98:0/4207672483 learned_addr learned my addr 172.21.15.98:0/4207672483 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.254+0000 7f5db6d9d640 1 --2- 172.21.15.98:0/4207672483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f610510 0x55d05f60da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.254+0000 7f5db659c640 1 -- 172.21.15.98:0/4207672483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f610510 msgr2=0x55d05f60da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.254+0000 7f5db659c640 1 --2- 172.21.15.98:0/4207672483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f610510 0x55d05f60da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.254+0000 7f5db659c640 1 -- 172.21.15.98:0/4207672483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605e40 msgr2=0x55d05f60d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.254+0000 7f5db659c640 1 --2- 172.21.15.98:0/4207672483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605e40 0x55d05f60d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.254+0000 7f5db659c640 1 -- 172.21.15.98:0/4207672483 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d05f3487d0 con 0x55d05f605520 2024-05-09T08:52:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.255+0000 7f5db659c640 1 --2- 172.21.15.98:0/4207672483 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605520 0x55d05f60ffa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5db00ccad0 tx=0x7f5db00ccc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=186ce7bf4a125690 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:30.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.263+0000 7f5d9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:52:30.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.263+0000 7f5d9dffb640 1 -- 172.21.15.98:0/4207672483 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db00d57b0 con 0x55d05f605520 2024-05-09T08:52:30.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.264+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/4207672483 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5da0011d10 con 0x55d05f605520 2024-05-09T08:52:30.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.264+0000 7f5d9dffb640 1 -- 172.21.15.98:0/4207672483 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f5db00cc220 con 0x55d05f605520 2024-05-09T08:52:30.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.267+0000 7f5d9dffb640 1 -- 172.21.15.98:0/4207672483 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5db0005500 con 0x55d05f605520 2024-05-09T08:52:30.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.267+0000 7f5dbf9c3740 10 client.25606 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:30.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.267+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/4207672483 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d05f613400 con 0x55d05f605520 2024-05-09T08:52:30.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.268+0000 7f5d9dffb640 1 -- 172.21.15.98:0/4207672483 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5da0011d10 con 0x55d05f605520 2024-05-09T08:52:30.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.268+0000 7f5d9dffb640 1 -- 172.21.15.98:0/4207672483 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5db00e2030 con 0x55d05f605520 2024-05-09T08:52:30.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.268+0000 7f5dbf9c3740 10 client.25606 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:30.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.268+0000 7f5dbf9c3740 10 client.25606 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:52:30.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.268+0000 7f5dbf9c3740 20 client.25606 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:30.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.268+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/4207672483 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d05f620760 0x55d05f622bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.268+0000 7f5dbf9c3740 4 client.25606 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:52:30.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.268+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/4207672483 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d05f578370 con 0x55d05f620760 2024-05-09T08:52:30.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.272+0000 7f5db5d9b640 1 --2- 172.21.15.98:0/4207672483 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d05f620760 0x55d05f622bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.276+0000 7f5db5d9b640 1 --2- 172.21.15.98:0/4207672483 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d05f620760 0x55d05f622bc0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ae284113f0f2cb9d server_cookie=69f6ccf6ec3b5956 in_seq=0 out_seq=0 2024-05-09T08:52:30.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.277+0000 7f5d9dffb640 10 client.25606 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:52:30.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.278+0000 7f5d9dffb640 1 -- 172.21.15.98:0/4207672483 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d05f578370 con 0x55d05f620760 2024-05-09T08:52:30.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.278+0000 7f5d9dffb640 10 client.25606 handle_command_reply: tid=0 2024-05-09T08:52:30.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.278+0000 7f5dbf9c3740 1 client.25606 shutdown 2024-05-09T08:52:30.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.278+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/4207672483 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605520 msgr2=0x55d05f60ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.279+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/4207672483 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605520 0x55d05f60ffa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5db00ccad0 tx=0x7f5db00ccc60 comp rx=0 tx=0).stop 2024-05-09T08:52:30.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.281+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/4207672483 shutdown_connections 2024-05-09T08:52:30.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.281+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/4207672483 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d05f620760 0x55d05f622bc0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.281+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/4207672483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f610510 0x55d05f60da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.281+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/4207672483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605e40 0x55d05f60d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.281+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/4207672483 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605520 0x55d05f60ffa0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.281+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/4207672483 >> 172.21.15.98:0/4207672483 conn(0x55d05f601840 msgr2=0x55d05f607640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:30.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.282+0000 7f5d9dffb640 0 client.25606 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:52:30.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.283+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/4207672483 shutdown_connections 2024-05-09T08:52:30.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.284+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/4207672483 wait complete. 2024-05-09T08:52:30.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.284+0000 7f5dbf9c3740 20 client.25606 trim_cache size 0 max 16384 2024-05-09T08:52:30.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.323+0000 7f5dbf9c3740 1 Processor -- start 2024-05-09T08:52:30.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.323+0000 7f5dbf9c3740 1 -- start start 2024-05-09T08:52:30.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.323+0000 7f5dbf9c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605520 0x55d05f61c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.324+0000 7f5dbf9c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605e40 0x55d05f607c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.324+0000 7f5dbf9c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f610510 0x55d05f6086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.324+0000 7f5dbf9c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5db81ac730 con 0x55d05f605520 2024-05-09T08:52:30.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.324+0000 7f5dbf9c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5db81ae270 con 0x55d05f610510 2024-05-09T08:52:30.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.324+0000 7f5dbf9c3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5db81ace00 con 0x55d05f605e40 2024-05-09T08:52:30.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.324+0000 7f5db659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605520 0x55d05f61c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.324+0000 7f5db659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605520 0x55d05f61c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47306/0 (socket says 172.21.15.98:47306) 2024-05-09T08:52:30.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.324+0000 7f5db659c640 1 -- 172.21.15.98:0/2402401382 learned_addr learned my addr 172.21.15.98:0/2402401382 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.324+0000 7f5db659c640 1 -- 172.21.15.98:0/2402401382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605e40 msgr2=0x55d05f607c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:52:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.324+0000 7f5db659c640 1 --2- 172.21.15.98:0/2402401382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605e40 0x55d05f607c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.324+0000 7f5db659c640 1 -- 172.21.15.98:0/2402401382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f610510 msgr2=0x55d05f6086b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:52:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.324+0000 7f5db659c640 1 --2- 172.21.15.98:0/2402401382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f610510 0x55d05f6086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.324+0000 7f5db659c640 1 -- 172.21.15.98:0/2402401382 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d05f605c70 con 0x55d05f605520 2024-05-09T08:52:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.325+0000 7f5db659c640 1 --2- 172.21.15.98:0/2402401382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605520 0x55d05f61c260 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5db00d7040 tx=0x7f5db00cccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c647af85ff8daa9c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:30.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.327+0000 7f5d9ffff640 1 -- 172.21.15.98:0/2402401382 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db00d1180 con 0x55d05f605520 2024-05-09T08:52:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.328+0000 7f5d9ffff640 1 -- 172.21.15.98:0/2402401382 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5db0005500 con 0x55d05f605520 2024-05-09T08:52:30.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.329+0000 7f5d9ffff640 1 -- 172.21.15.98:0/2402401382 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db00cc220 con 0x55d05f605520 2024-05-09T08:52:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.330+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2402401382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605520 msgr2=0x55d05f61c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:30.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.330+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2402401382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605520 0x55d05f61c260 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5db00d7040 tx=0x7f5db00cccd0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.335+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2402401382 shutdown_connections 2024-05-09T08:52:30.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.335+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2402401382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f610510 0x55d05f6086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.335+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2402401382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605e40 0x55d05f607c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.335+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2402401382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605520 0x55d05f61c260 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.335+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2402401382 >> 172.21.15.98:0/2402401382 conn(0x55d05f601840 msgr2=0x55d05f601c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:30.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.338+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2402401382 shutdown_connections 2024-05-09T08:52:30.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.340+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2402401382 wait complete. 2024-05-09T08:52:30.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.341+0000 7f5dbf9c3740 1 -- start start 2024-05-09T08:52:30.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.341+0000 7f5dbf9c3740 1 Processor -- start 2024-05-09T08:52:30.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.342+0000 7f5dbf9c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605520 0x55d05f613920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.342+0000 7f5dbf9c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605e40 0x55d05f613e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.342+0000 7f5db659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605520 0x55d05f613920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.342+0000 7f5db659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605520 0x55d05f613920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55520/0 (socket says 172.21.15.98:55520) 2024-05-09T08:52:30.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.343+0000 7f5db659c640 1 -- 172.21.15.98:0/2114361118 learned_addr learned my addr 172.21.15.98:0/2114361118 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:30.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.343+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2114361118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f610510 0x55d05f684d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.343+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2114361118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5db81adba0 con 0x55d05f605e40 2024-05-09T08:52:30.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.343+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2114361118 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5db81ad4d0 con 0x55d05f610510 2024-05-09T08:52:30.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.343+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2114361118 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5db81aa710 con 0x55d05f605520 2024-05-09T08:52:30.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.343+0000 7f5dbf9c3740 10 client.0 _ll_register_callbacks cb 0x7f5db00c8fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:30.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.343+0000 7f5db6d9d640 1 --2- 172.21.15.98:0/2114361118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f610510 0x55d05f684d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.343+0000 7f5db5d9b640 1 --2- 172.21.15.98:0/2114361118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605e40 0x55d05f613e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.344+0000 7f5db659c640 1 -- 172.21.15.98:0/2114361118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f610510 msgr2=0x55d05f684d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:30.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.344+0000 7f5db659c640 1 --2- 172.21.15.98:0/2114361118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f610510 0x55d05f684d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.344+0000 7f5db659c640 1 -- 172.21.15.98:0/2114361118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605e40 msgr2=0x55d05f613e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:30.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.344+0000 7f5db659c640 1 --2- 172.21.15.98:0/2114361118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605e40 0x55d05f613e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.344+0000 7f5db659c640 1 -- 172.21.15.98:0/2114361118 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d05f685270 con 0x55d05f605520 2024-05-09T08:52:30.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.346+0000 7f5db659c640 1 --2- 172.21.15.98:0/2114361118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605520 0x55d05f613920 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5db00ccad0 tx=0x7f5db003b4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b029cf3d5125222b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:30.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.346+0000 7f5d9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:52:30.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.347+0000 7f5d9dffb640 1 -- 172.21.15.98:0/2114361118 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db00d1180 con 0x55d05f605520 2024-05-09T08:52:30.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.347+0000 7f5d9dffb640 1 -- 172.21.15.98:0/2114361118 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f5db00d2520 con 0x55d05f605520 2024-05-09T08:52:30.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.347+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2114361118 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5db0005500 con 0x55d05f605520 2024-05-09T08:52:30.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.350+0000 7f5d9dffb640 1 -- 172.21.15.98:0/2114361118 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5db00cc730 con 0x55d05f605520 2024-05-09T08:52:30.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.351+0000 7f5dbf9c3740 10 client.25609 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:30.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.352+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2114361118 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d05f685270 con 0x55d05f605520 2024-05-09T08:52:30.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.352+0000 7f5d9dffb640 1 -- 172.21.15.98:0/2114361118 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5db00ccdd0 con 0x55d05f605520 2024-05-09T08:52:30.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.353+0000 7f5d9dffb640 1 -- 172.21.15.98:0/2114361118 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5db00e2070 con 0x55d05f605520 2024-05-09T08:52:30.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.353+0000 7f5dbf9c3740 10 client.25609 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:30.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.353+0000 7f5dbf9c3740 10 client.25609 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:52:30.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.353+0000 7f5dbf9c3740 20 client.25609 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:30.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.353+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2114361118 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d05f686750 0x55d05f688b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.353+0000 7f5dbf9c3740 4 client.25609 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:52:30.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.353+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2114361118 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f5db0005500 con 0x55d05f686750 2024-05-09T08:52:30.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.355+0000 7f5db5d9b640 1 --2- 172.21.15.98:0/2114361118 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d05f686750 0x55d05f688b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.358+0000 7f5db5d9b640 1 --2- 172.21.15.98:0/2114361118 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d05f686750 0x55d05f688b40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e3936c863929ac4e server_cookie=ca57092306d8ed28 in_seq=0 out_seq=0 2024-05-09T08:52:30.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.358+0000 7f5d9dffb640 10 client.25609 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:52:30.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.358+0000 7f5d9dffb640 1 -- 172.21.15.98:0/2114361118 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f5db0005500 con 0x55d05f686750 2024-05-09T08:52:30.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.358+0000 7f5d9dffb640 10 client.25609 handle_command_reply: tid=0 2024-05-09T08:52:30.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.359+0000 7f5dbf9c3740 1 client.25609 shutdown 2024-05-09T08:52:30.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.366+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2114361118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605520 msgr2=0x55d05f613920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:30.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.366+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2114361118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605520 0x55d05f613920 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5db00ccad0 tx=0x7f5db003b4f0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.366+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2114361118 shutdown_connections 2024-05-09T08:52:30.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.366+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2114361118 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d05f686750 0x55d05f688b40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.366+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2114361118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d05f610510 0x55d05f684d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.366+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2114361118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d05f605e40 0x55d05f613e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.366+0000 7f5dbf9c3740 1 --2- 172.21.15.98:0/2114361118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d05f605520 0x55d05f613920 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5db00ccad0 tx=0x7f5db003b4f0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.366+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2114361118 >> 172.21.15.98:0/2114361118 conn(0x55d05f601840 msgr2=0x55d05f6086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:30.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.367+0000 7f5d9dffb640 0 client.25609 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:52:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.374+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2114361118 shutdown_connections 2024-05-09T08:52:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.374+0000 7f5dbf9c3740 1 -- 172.21.15.98:0/2114361118 wait complete. 2024-05-09T08:52:30.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.375+0000 7f5dbf9c3740 20 client.25609 trim_cache size 0 max 16384 2024-05-09T08:52:30.375 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:52:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.377+0000 7f5db759e640 1 -- 172.21.15.98:0/2010940064 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5da40a2d60 msgr2=0x7f5da40a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.377+0000 7f5db759e640 1 --2- 172.21.15.98:0/2010940064 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5da40a2d60 0x7f5da40a5220 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5da8011490 tx=0x7f5da8009660 comp rx=0 tx=0).stop 2024-05-09T08:52:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.378+0000 7f5db759e640 1 -- 172.21.15.98:0/2010940064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5db8100690 msgr2=0x7f5db81149e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.378+0000 7f5db759e640 1 --2- 172.21.15.98:0/2010940064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5db8100690 0x7f5db81149e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f5db003bae0 tx=0x7f5db0003820 comp rx=0 tx=0).stop 2024-05-09T08:52:30.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.380+0000 7f5db759e640 1 -- 172.21.15.98:0/2010940064 shutdown_connections 2024-05-09T08:52:30.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.380+0000 7f5db759e640 1 --2- 172.21.15.98:0/2010940064 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5da40a2d60 0x7f5da40a5220 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.380+0000 7f5db759e640 1 --2- 172.21.15.98:0/2010940064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8105a90 0x7f5db8110fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.380+0000 7f5db759e640 1 --2- 172.21.15.98:0/2010940064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5db8100fb0 0x7f5db8110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.380+0000 7f5db759e640 1 --2- 172.21.15.98:0/2010940064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5db8100690 0x7f5db81149e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.380+0000 7f5db759e640 1 -- 172.21.15.98:0/2010940064 >> 172.21.15.98:0/2010940064 conn(0x7f5db80fc0a0 msgr2=0x7f5db80fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:30.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.383+0000 7f5db759e640 1 -- 172.21.15.98:0/2010940064 shutdown_connections 2024-05-09T08:52:30.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.383+0000 7f5db759e640 1 -- 172.21.15.98:0/2010940064 wait complete. 2024-05-09T08:52:30.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:52:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.820+0000 7fd21133b640 1 -- 172.21.15.98:0/2568239863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd20c11b010 msgr2=0x7fd20c11d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.820+0000 7fd21133b640 1 --2- 172.21.15.98:0/2568239863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd20c11b010 0x7fd20c11d400 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd200009930 tx=0x7fd20002a480 comp rx=0 tx=0).stop 2024-05-09T08:52:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.822+0000 7fd21133b640 1 -- 172.21.15.98:0/2568239863 shutdown_connections 2024-05-09T08:52:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.822+0000 7fd21133b640 1 --2- 172.21.15.98:0/2568239863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd20c11b010 0x7fd20c11d400 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.822+0000 7fd21133b640 1 --2- 172.21.15.98:0/2568239863 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd20c10e0b0 0x7fd20c10e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.822+0000 7fd21133b640 1 --2- 172.21.15.98:0/2568239863 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd20c110080 0x7fd20c10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.822+0000 7fd21133b640 1 -- 172.21.15.98:0/2568239863 >> 172.21.15.98:0/2568239863 conn(0x7fd20c06c940 msgr2=0x7fd20c06cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:30.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.826+0000 7fd21133b640 1 -- 172.21.15.98:0/2568239863 shutdown_connections 2024-05-09T08:52:30.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.829+0000 7fd21133b640 1 -- 172.21.15.98:0/2568239863 wait complete. 2024-05-09T08:52:30.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.830+0000 7fd21133b640 1 Processor -- start 2024-05-09T08:52:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.843+0000 7fd21133b640 1 -- start start 2024-05-09T08:52:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.843+0000 7fd21133b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd20c10e0b0 0x7fd20c1a7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.843+0000 7fd21133b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd20c110080 0x7fd20c1a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.843+0000 7fd21133b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd20c11b010 0x7fd20c1ab970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.843+0000 7fd21133b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c120ab0 con 0x7fd20c11b010 2024-05-09T08:52:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.843+0000 7fd21133b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c120930 con 0x7fd20c10e0b0 2024-05-09T08:52:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.843+0000 7fd21133b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c120c30 con 0x7fd20c110080 2024-05-09T08:52:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.844+0000 7fd20affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd20c10e0b0 0x7fd20c1a7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.844+0000 7fd20affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd20c10e0b0 0x7fd20c1a7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:53336/0 (socket says 172.21.15.98:53336) 2024-05-09T08:52:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.844+0000 7fd20affd640 1 -- 172.21.15.98:0/1861880656 learned_addr learned my addr 172.21.15.98:0/1861880656 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:30.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.845+0000 7fd20a7fc640 1 --2- 172.21.15.98:0/1861880656 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd20c110080 0x7fd20c1a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.847+0000 7fd20affd640 1 -- 172.21.15.98:0/1861880656 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd20c110080 msgr2=0x7fd20c1a7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:30.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.847+0000 7fd20affd640 1 --2- 172.21.15.98:0/1861880656 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd20c110080 0x7fd20c1a7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.847+0000 7fd20affd640 1 -- 172.21.15.98:0/1861880656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd20c11b010 msgr2=0x7fd20c1ab970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:30.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.847+0000 7fd20affd640 1 --2- 172.21.15.98:0/1861880656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd20c11b010 0x7fd20c1ab970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.847+0000 7fd20affd640 1 -- 172.21.15.98:0/1861880656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd20c1ac0a0 con 0x7fd20c10e0b0 2024-05-09T08:52:30.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.850+0000 7fd20affd640 1 --2- 172.21.15.98:0/1861880656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd20c10e0b0 0x7fd20c1a7100 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00b450 tx=0x7fd1fc00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:30.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.852+0000 7fd1f3fff640 1 -- 172.21.15.98:0/1861880656 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc015020 con 0x7fd20c10e0b0 2024-05-09T08:52:30.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.852+0000 7fd21133b640 1 -- 172.21.15.98:0/1861880656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd20c1ac390 con 0x7fd20c10e0b0 2024-05-09T08:52:30.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.852+0000 7fd21133b640 1 -- 172.21.15.98:0/1861880656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd20c1b3c20 con 0x7fd20c10e0b0 2024-05-09T08:52:30.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.853+0000 7fd1f3fff640 1 -- 172.21.15.98:0/1861880656 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1fc011d70 con 0x7fd20c10e0b0 2024-05-09T08:52:30.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.854+0000 7fd1f3fff640 1 -- 172.21.15.98:0/1861880656 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc00f340 con 0x7fd20c10e0b0 2024-05-09T08:52:30.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.862+0000 7fd21404a740 1 Processor -- start 2024-05-09T08:52:30.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.862+0000 7fd21404a740 1 -- start start 2024-05-09T08:52:30.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.863+0000 7fd21404a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244330 0x562c69244710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.863+0000 7fd21404a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c69244c50 0x562c6924ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.863+0000 7fd21404a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c6924f320 0x562c69251710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.863+0000 7fd21404a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562c691a9c40 con 0x562c69244330 2024-05-09T08:52:30.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.863+0000 7fd21404a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562c69151580 con 0x562c6924f320 2024-05-09T08:52:30.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.864+0000 7fd20affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c69244c50 0x562c6924ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.864+0000 7fd20affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c69244c50 0x562c6924ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55564/0 (socket says 172.21.15.98:55564) 2024-05-09T08:52:30.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.864+0000 7fd20affd640 1 -- 172.21.15.98:0/2216615138 learned_addr learned my addr 172.21.15.98:0/2216615138 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:30.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.864+0000 7fd21404a740 1 -- 172.21.15.98:0/2216615138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562c691a5c00 con 0x562c69244c50 2024-05-09T08:52:30.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.864+0000 7fd1f3fff640 1 -- 172.21.15.98:0/1861880656 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd1fc018020 con 0x7fd20c10e0b0 2024-05-09T08:52:30.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.864+0000 7fd20affd640 1 -- 172.21.15.98:0/2216615138 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c6924f320 msgr2=0x562c69251710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:52:30.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.864+0000 7fd20affd640 1 --2- 172.21.15.98:0/2216615138 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c6924f320 0x562c69251710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.864+0000 7fd20affd640 1 -- 172.21.15.98:0/2216615138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244330 msgr2=0x562c69244710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:30.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.864+0000 7fd20affd640 1 --2- 172.21.15.98:0/2216615138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244330 0x562c69244710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.864+0000 7fd20affd640 1 -- 172.21.15.98:0/2216615138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c68ebd3f0 con 0x562c69244c50 2024-05-09T08:52:30.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.866+0000 7fd20affd640 1 --2- 172.21.15.98:0/2216615138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c69244c50 0x562c6924ede0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc0cc450 tx=0x7fd1fc0d76a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1edc271913a5bb2e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:30.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.866+0000 7fd1f3fff640 1 --2- 172.21.15.98:0/1861880656 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd1f40a2d60 0x7fd1f40a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.867+0000 7fd1f17fa640 1 -- 172.21.15.98:0/2216615138 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0e2070 con 0x562c69244c50 2024-05-09T08:52:30.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.867+0000 7fd1f3fff640 1 -- 172.21.15.98:0/1861880656 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fd1fc0cf0d0 con 0x7fd20c10e0b0 2024-05-09T08:52:30.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.867+0000 7fd1f17fa640 1 -- 172.21.15.98:0/2216615138 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1fc0cf9e0 con 0x562c69244c50 2024-05-09T08:52:30.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.867+0000 7fd20b7fe640 1 --2- 172.21.15.98:0/1861880656 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd1f40a2d60 0x7fd1f40a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.868+0000 7fd21404a740 1 -- 172.21.15.98:0/2216615138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c69244c50 msgr2=0x562c6924ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:30.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.868+0000 7fd21404a740 1 --2- 172.21.15.98:0/2216615138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c69244c50 0x562c6924ede0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc0cc450 tx=0x7fd1fc0d76a0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.869+0000 7fd20b7fe640 1 --2- 172.21.15.98:0/1861880656 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd1f40a2d60 0x7fd1f40a5220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd200009230 tx=0x7fd20002a980 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:30.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.869+0000 7fd21404a740 1 -- 172.21.15.98:0/2216615138 shutdown_connections 2024-05-09T08:52:30.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.869+0000 7fd21404a740 1 --2- 172.21.15.98:0/2216615138 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c6924f320 0x562c69251710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.869+0000 7fd21404a740 1 --2- 172.21.15.98:0/2216615138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c69244c50 0x562c6924ede0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.869+0000 7fd21404a740 1 --2- 172.21.15.98:0/2216615138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244330 0x562c69244710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.869+0000 7fd21404a740 1 -- 172.21.15.98:0/2216615138 >> 172.21.15.98:0/2216615138 conn(0x562c69240650 msgr2=0x562c69240a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:30.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.871+0000 7fd21404a740 1 -- 172.21.15.98:0/2216615138 shutdown_connections 2024-05-09T08:52:30.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.872+0000 7fd21404a740 1 -- 172.21.15.98:0/2216615138 wait complete. 2024-05-09T08:52:30.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.873+0000 7fd21404a740 1 -- start start 2024-05-09T08:52:30.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.874+0000 7fd21404a740 1 Processor -- start 2024-05-09T08:52:30.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.879+0000 7fd21404a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c69244330 0x562c6924ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.879+0000 7fd21404a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244c50 0x562c6924c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.879+0000 7fd21404a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c6924f320 0x562c6924c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.879+0000 7fd21404a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562c69252160 con 0x562c69244c50 2024-05-09T08:52:30.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.879+0000 7fd21404a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562c69164c40 con 0x562c6924f320 2024-05-09T08:52:30.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.879+0000 7fd21404a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562c692522e0 con 0x562c69244330 2024-05-09T08:52:30.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.879+0000 7fd21404a740 10 client.0 _ll_register_callbacks cb 0x562c68ebcbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:30.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.880+0000 7fd20b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c6924f320 0x562c6924c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.880+0000 7fd20b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c6924f320 0x562c6924c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:53344/0 (socket says 172.21.15.98:53344) 2024-05-09T08:52:30.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.880+0000 7fd20b7fe640 1 -- 172.21.15.98:0/2141569708 learned_addr learned my addr 172.21.15.98:0/2141569708 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:30.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.880+0000 7fd20affd640 1 --2- 172.21.15.98:0/2141569708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c69244330 0x562c6924ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.881+0000 7fd20a7fc640 1 --2- 172.21.15.98:0/2141569708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244c50 0x562c6924c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.881+0000 7fd20b7fe640 1 -- 172.21.15.98:0/2141569708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c69244330 msgr2=0x562c6924ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:30.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.881+0000 7fd20b7fe640 1 --2- 172.21.15.98:0/2141569708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c69244330 0x562c6924ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.881+0000 7fd20b7fe640 1 -- 172.21.15.98:0/2141569708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244c50 msgr2=0x562c6924c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:30.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.881+0000 7fd20b7fe640 1 --2- 172.21.15.98:0/2141569708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244c50 0x562c6924c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.881+0000 7fd20b7fe640 1 -- 172.21.15.98:0/2141569708 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c68f877d0 con 0x562c6924f320 2024-05-09T08:52:30.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.884+0000 7fd20b7fe640 1 --2- 172.21.15.98:0/2141569708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c6924f320 0x562c6924c8b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd200033000 tx=0x7fd200004590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc1012c9d6463e9c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:30.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.887+0000 7fd1eeff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:52:30.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.888+0000 7fd1eeff5640 1 -- 172.21.15.98:0/2141569708 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20004b020 con 0x562c6924f320 2024-05-09T08:52:30.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.888+0000 7fd21404a740 1 -- 172.21.15.98:0/2141569708 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1fc0cf9e0 con 0x562c6924f320 2024-05-09T08:52:30.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.888+0000 7fd1eeff5640 1 -- 172.21.15.98:0/2141569708 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fd2000315b0 con 0x562c6924f320 2024-05-09T08:52:30.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.888+0000 7fd1eeff5640 1 -- 172.21.15.98:0/2141569708 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd200056050 con 0x562c6924f320 2024-05-09T08:52:30.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.889+0000 7fd21404a740 10 client.25664 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:30.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.889+0000 7fd21404a740 1 -- 172.21.15.98:0/2141569708 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562c6925e320 con 0x562c6924f320 2024-05-09T08:52:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.890+0000 7fd1eeff5640 1 -- 172.21.15.98:0/2141569708 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1fc0cf9e0 con 0x562c6924f320 2024-05-09T08:52:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.890+0000 7fd1eeff5640 1 -- 172.21.15.98:0/2141569708 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd20003a5b0 con 0x562c6924f320 2024-05-09T08:52:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.893+0000 7fd21404a740 10 client.25664 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.893+0000 7fd21404a740 10 client.25664 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.893+0000 7fd21404a740 20 client.25664 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.893+0000 7fd21404a740 1 --2- 172.21.15.98:0/2141569708 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562c6925f830 0x562c69261c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.893+0000 7fd21404a740 4 client.25664 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:52:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.893+0000 7fd21404a740 1 -- 172.21.15.98:0/2141569708 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c691b7370 con 0x562c6925f830 2024-05-09T08:52:30.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.897+0000 7fd20affd640 1 --2- 172.21.15.98:0/2141569708 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562c6925f830 0x562c69261c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.902+0000 7fd20affd640 1 --2- 172.21.15.98:0/2141569708 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562c6925f830 0x562c69261c50 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ea87fed962c25e6 server_cookie=62dcc2110b207991 in_seq=0 out_seq=0 2024-05-09T08:52:30.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.905+0000 7fd1eeff5640 10 client.25664 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:30.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.905+0000 7fd1eeff5640 1 -- 172.21.15.98:0/2141569708 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562c691b7370 con 0x562c6925f830 2024-05-09T08:52:30.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.905+0000 7fd1eeff5640 10 client.25664 handle_command_reply: tid=0 2024-05-09T08:52:30.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.907+0000 7fd21404a740 1 client.25664 shutdown 2024-05-09T08:52:30.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.907+0000 7fd21404a740 1 -- 172.21.15.98:0/2141569708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c6924f320 msgr2=0x562c6924c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:30.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.907+0000 7fd21404a740 1 --2- 172.21.15.98:0/2141569708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c6924f320 0x562c6924c8b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd200033000 tx=0x7fd200004590 comp rx=0 tx=0).stop 2024-05-09T08:52:30.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.908+0000 7fd21404a740 1 -- 172.21.15.98:0/2141569708 shutdown_connections 2024-05-09T08:52:30.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.908+0000 7fd21404a740 1 --2- 172.21.15.98:0/2141569708 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562c6925f830 0x562c69261c50 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.908+0000 7fd21404a740 1 --2- 172.21.15.98:0/2141569708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c6924f320 0x562c6924c8b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.908+0000 7fd21404a740 1 --2- 172.21.15.98:0/2141569708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244c50 0x562c6924c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.908+0000 7fd21404a740 1 --2- 172.21.15.98:0/2141569708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c69244330 0x562c6924ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.908+0000 7fd21404a740 1 -- 172.21.15.98:0/2141569708 >> 172.21.15.98:0/2141569708 conn(0x562c69240650 msgr2=0x562c6923f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.913+0000 7fd1eeff5640 0 client.25664 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.913+0000 7fd21404a740 1 -- 172.21.15.98:0/2141569708 shutdown_connections 2024-05-09T08:52:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.913+0000 7fd21404a740 1 -- 172.21.15.98:0/2141569708 wait complete. 2024-05-09T08:52:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.913+0000 7fd21404a740 20 client.25664 trim_cache size 0 max 16384 2024-05-09T08:52:30.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.948+0000 7fd21404a740 1 Processor -- start 2024-05-09T08:52:30.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.950+0000 7fd21404a740 1 -- start start 2024-05-09T08:52:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.951+0000 7fd21404a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244330 0x562c6925b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.951+0000 7fd21404a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c69244c50 0x562c69267500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.951+0000 7fd21404a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c6924f320 0x562c69267a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.951+0000 7fd21404a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f80100e0 con 0x562c69244330 2024-05-09T08:52:30.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.951+0000 7fd21404a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f8011c20 con 0x562c69244c50 2024-05-09T08:52:30.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.951+0000 7fd21404a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f80107b0 con 0x562c6924f320 2024-05-09T08:52:30.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.952+0000 7fd20affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244330 0x562c6925b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.952+0000 7fd20a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c69244c50 0x562c69267500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.952+0000 7fd20affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244330 0x562c6925b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47370/0 (socket says 172.21.15.98:47370) 2024-05-09T08:52:30.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.952+0000 7fd20a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c69244c50 0x562c69267500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:53356/0 (socket says 172.21.15.98:53356) 2024-05-09T08:52:30.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.952+0000 7fd20affd640 1 -- 172.21.15.98:0/1514518404 learned_addr learned my addr 172.21.15.98:0/1514518404 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:30.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.953+0000 7fd20affd640 1 -- 172.21.15.98:0/1514518404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c6924f320 msgr2=0x562c69267a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:52:30.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.953+0000 7fd20affd640 1 --2- 172.21.15.98:0/1514518404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c6924f320 0x562c69267a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.953+0000 7fd20affd640 1 -- 172.21.15.98:0/1514518404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c69244c50 msgr2=0x562c69267500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:30.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.953+0000 7fd20affd640 1 --2- 172.21.15.98:0/1514518404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c69244c50 0x562c69267500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.953+0000 7fd20affd640 1 -- 172.21.15.98:0/1514518404 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c6924a6c0 con 0x562c69244330 2024-05-09T08:52:30.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.962+0000 7fd20affd640 1 --2- 172.21.15.98:0/1514518404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244330 0x562c6925b100 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc097cb0 tx=0x7fd1fc013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce28b96a6edbfa7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:30.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.966+0000 7fd1f0ff9640 1 -- 172.21.15.98:0/1514518404 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc09b0b0 con 0x562c69244330 2024-05-09T08:52:30.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.966+0000 7fd1f0ff9640 1 -- 172.21.15.98:0/1514518404 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1fc095e50 con 0x562c69244330 2024-05-09T08:52:30.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.967+0000 7fd21404a740 1 -- 172.21.15.98:0/1514518404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244330 msgr2=0x562c6925b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:30.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.967+0000 7fd21404a740 1 --2- 172.21.15.98:0/1514518404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244330 0x562c6925b100 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc097cb0 tx=0x7fd1fc013040 comp rx=0 tx=0).stop 2024-05-09T08:52:30.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.968+0000 7fd21404a740 1 -- 172.21.15.98:0/1514518404 shutdown_connections 2024-05-09T08:52:30.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.968+0000 7fd21404a740 1 --2- 172.21.15.98:0/1514518404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c6924f320 0x562c69267a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.968+0000 7fd21404a740 1 --2- 172.21.15.98:0/1514518404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c69244c50 0x562c69267500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.968+0000 7fd21404a740 1 --2- 172.21.15.98:0/1514518404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244330 0x562c6925b100 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.968+0000 7fd21404a740 1 -- 172.21.15.98:0/1514518404 >> 172.21.15.98:0/1514518404 conn(0x562c69240650 msgr2=0x562c69240a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.976+0000 7fd21404a740 1 -- 172.21.15.98:0/1514518404 shutdown_connections 2024-05-09T08:52:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.976+0000 7fd21404a740 1 -- 172.21.15.98:0/1514518404 wait complete. 2024-05-09T08:52:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.976+0000 7fd21404a740 1 -- start start 2024-05-09T08:52:30.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.976+0000 7fd21404a740 1 Processor -- start 2024-05-09T08:52:30.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.977+0000 7fd21404a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c69244330 0x562c692500c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.977+0000 7fd21404a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244c50 0x562c69250600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.977+0000 7fd21404a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c6924f320 0x562c69250b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:30.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.977+0000 7fd21404a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f8011550 con 0x562c69244c50 2024-05-09T08:52:30.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.977+0000 7fd21404a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f8010e80 con 0x562c69244330 2024-05-09T08:52:30.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.977+0000 7fd21404a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f800e0c0 con 0x562c6924f320 2024-05-09T08:52:30.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.977+0000 7fd21404a740 10 client.0 _ll_register_callbacks cb 0x7fd200009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:30.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.978+0000 7fd20affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c69244330 0x562c692500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:30.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.978+0000 7fd20affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c69244330 0x562c692500c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:53358/0 (socket says 172.21.15.98:53358) 2024-05-09T08:52:30.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.978+0000 7fd20affd640 1 -- 172.21.15.98:0/624161125 learned_addr learned my addr 172.21.15.98:0/624161125 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:30.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.979+0000 7fd20affd640 1 -- 172.21.15.98:0/624161125 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c6924f320 msgr2=0x562c69250b40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:52:30.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.979+0000 7fd20affd640 1 --2- 172.21.15.98:0/624161125 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c6924f320 0x562c69250b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.979+0000 7fd20affd640 1 -- 172.21.15.98:0/624161125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244c50 msgr2=0x562c69250600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:30.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.979+0000 7fd20affd640 1 --2- 172.21.15.98:0/624161125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244c50 0x562c69250600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:30.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.979+0000 7fd20affd640 1 -- 172.21.15.98:0/624161125 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c69255750 con 0x562c69244330 2024-05-09T08:52:30.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.980+0000 7fd20affd640 1 --2- 172.21.15.98:0/624161125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c69244330 0x562c692500c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009fd0 tx=0x7fd1fc09b140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9df4edb5feadffb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.980+0000 7fd1eeff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:52:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.981+0000 7fd1eeff5640 1 -- 172.21.15.98:0/624161125 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0af060 con 0x562c69244330 2024-05-09T08:52:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.981+0000 7fd1eeff5640 1 -- 172.21.15.98:0/624161125 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fd1fc0995e0 con 0x562c69244330 2024-05-09T08:52:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.981+0000 7fd21404a740 1 -- 172.21.15.98:0/624161125 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1fc095e50 con 0x562c69244330 2024-05-09T08:52:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:30.991+0000 7fd1eeff5640 1 -- 172.21.15.98:0/624161125 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd1fc09ae20 con 0x562c69244330 2024-05-09T08:52:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.003+0000 7fd21404a740 10 client.25670 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.003+0000 7fd21404a740 1 -- 172.21.15.98:0/624161125 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562c692d6f60 con 0x562c69244330 2024-05-09T08:52:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.008+0000 7fd1eeff5640 1 -- 172.21.15.98:0/624161125 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1fc095e50 con 0x562c69244330 2024-05-09T08:52:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.008+0000 7fd1eeff5640 1 -- 172.21.15.98:0/624161125 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd1fc0cc680 con 0x562c69244330 2024-05-09T08:52:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.008+0000 7fd21404a740 10 client.25670 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.008+0000 7fd21404a740 10 client.25670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.008+0000 7fd21404a740 20 client.25670 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.008+0000 7fd21404a740 1 --2- 172.21.15.98:0/624161125 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562c692d7350 0x562c692d9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.008+0000 7fd21404a740 4 client.25670 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:52:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.008+0000 7fd21404a740 1 -- 172.21.15.98:0/624161125 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562c691b16c0 con 0x562c692d7350 2024-05-09T08:52:31.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.017+0000 7fd20a7fc640 1 --2- 172.21.15.98:0/624161125 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562c692d7350 0x562c692d9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:31.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.020+0000 7fd20a7fc640 1 --2- 172.21.15.98:0/624161125 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562c692d7350 0x562c692d9740 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6257cb885a891a6e server_cookie=76298b30ea3e109d in_seq=0 out_seq=0 2024-05-09T08:52:31.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.020+0000 7fd1eeff5640 10 client.25670 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:31.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.027+0000 7fd1eeff5640 1 -- 172.21.15.98:0/624161125 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562c691b16c0 con 0x562c692d7350 2024-05-09T08:52:31.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.027+0000 7fd1eeff5640 10 client.25670 handle_command_reply: tid=0 2024-05-09T08:52:31.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.027+0000 7fd21404a740 1 client.25670 shutdown 2024-05-09T08:52:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.033+0000 7fd21404a740 1 -- 172.21.15.98:0/624161125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c69244330 msgr2=0x562c692500c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.033+0000 7fd21404a740 1 --2- 172.21.15.98:0/624161125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c69244330 0x562c692500c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009fd0 tx=0x7fd1fc09b140 comp rx=0 tx=0).stop 2024-05-09T08:52:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.034+0000 7fd21404a740 1 -- 172.21.15.98:0/624161125 shutdown_connections 2024-05-09T08:52:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.034+0000 7fd21404a740 1 --2- 172.21.15.98:0/624161125 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562c692d7350 0x562c692d9740 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.034+0000 7fd21404a740 1 --2- 172.21.15.98:0/624161125 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562c6924f320 0x562c69250b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.034+0000 7fd21404a740 1 --2- 172.21.15.98:0/624161125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562c69244c50 0x562c69250600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.034+0000 7fd21404a740 1 --2- 172.21.15.98:0/624161125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c69244330 0x562c692500c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.034+0000 7fd21404a740 1 -- 172.21.15.98:0/624161125 >> 172.21.15.98:0/624161125 conn(0x562c69240650 msgr2=0x562c69247460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.035+0000 7fd1eeff5640 0 client.25670 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.036+0000 7fd21404a740 1 -- 172.21.15.98:0/624161125 shutdown_connections 2024-05-09T08:52:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.039+0000 7fd21404a740 1 -- 172.21.15.98:0/624161125 wait complete. 2024-05-09T08:52:31.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.041+0000 7fd21404a740 20 client.25670 trim_cache size 0 max 16384 2024-05-09T08:52:31.042 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:52:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.049+0000 7fd21133b640 1 -- 172.21.15.98:0/1861880656 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd1f40a2d60 msgr2=0x7fd1f40a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.049+0000 7fd21133b640 1 --2- 172.21.15.98:0/1861880656 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd1f40a2d60 0x7fd1f40a5220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd200009230 tx=0x7fd20002a980 comp rx=0 tx=0).stop 2024-05-09T08:52:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.049+0000 7fd21133b640 1 -- 172.21.15.98:0/1861880656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd20c10e0b0 msgr2=0x7fd20c1a7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.049+0000 7fd21133b640 1 --2- 172.21.15.98:0/1861880656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd20c10e0b0 0x7fd20c1a7100 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00b450 tx=0x7fd1fc00b910 comp rx=0 tx=0).stop 2024-05-09T08:52:31.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.055+0000 7fd21133b640 1 -- 172.21.15.98:0/1861880656 shutdown_connections 2024-05-09T08:52:31.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.055+0000 7fd21133b640 1 --2- 172.21.15.98:0/1861880656 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd1f40a2d60 0x7fd1f40a5220 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:31.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.055+0000 7fd21133b640 1 --2- 172.21.15.98:0/1861880656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd20c11b010 0x7fd20c1ab970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:31.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.055+0000 7fd21133b640 1 --2- 172.21.15.98:0/1861880656 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd20c110080 0x7fd20c1a7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:31.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.055+0000 7fd21133b640 1 --2- 172.21.15.98:0/1861880656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd20c10e0b0 0x7fd20c1a7100 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:31.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.055+0000 7fd21133b640 1 -- 172.21.15.98:0/1861880656 >> 172.21.15.98:0/1861880656 conn(0x7fd20c06c940 msgr2=0x7fd20c11c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:31.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.059+0000 7fd21133b640 1 -- 172.21.15.98:0/1861880656 shutdown_connections 2024-05-09T08:52:31.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:31.059+0000 7fd21133b640 1 -- 172.21.15.98:0/1861880656 wait complete. 2024-05-09T08:52:32.100 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:52:32.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:52:32.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:52:32.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:31 smithi142 ceph-mon[34625]: pgmap v59: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.1 GiB/s wr, 2.66k op/s 2024-05-09T08:52:32.209 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:31 smithi098 ceph-mon[30092]: pgmap v59: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.1 GiB/s wr, 2.66k op/s 2024-05-09T08:52:32.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:31 smithi105 ceph-mon[34498]: pgmap v59: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.1 GiB/s wr, 2.66k op/s 2024-05-09T08:52:32.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.426+0000 7f0407fff640 1 -- 172.21.15.98:0/3439430825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f040810fed0 msgr2=0x7f04081102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:32.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.426+0000 7f0407fff640 1 --2- 172.21.15.98:0/3439430825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f040810fed0 0x7f04081102b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f03f8009860 tx=0x7f03f802a410 comp rx=0 tx=0).stop 2024-05-09T08:52:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.432+0000 7f0407fff640 1 -- 172.21.15.98:0/3439430825 shutdown_connections 2024-05-09T08:52:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.432+0000 7f0407fff640 1 --2- 172.21.15.98:0/3439430825 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04081092b0 0x7f040811d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.432+0000 7f0407fff640 1 --2- 172.21.15.98:0/3439430825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0408108830 0x7f0408108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.432+0000 7f0407fff640 1 --2- 172.21.15.98:0/3439430825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f040810fed0 0x7f04081102b0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.432+0000 7f0407fff640 1 -- 172.21.15.98:0/3439430825 >> 172.21.15.98:0/3439430825 conn(0x7f040806c7b0 msgr2=0x7f040806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:32.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.435+0000 7f0407fff640 1 -- 172.21.15.98:0/3439430825 shutdown_connections 2024-05-09T08:52:32.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.438+0000 7f0407fff640 1 -- 172.21.15.98:0/3439430825 wait complete. 2024-05-09T08:52:32.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.439+0000 7f0407fff640 1 Processor -- start 2024-05-09T08:52:32.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.440+0000 7f0407fff640 1 -- start start 2024-05-09T08:52:32.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.440+0000 7f0407fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0408108830 0x7f040811cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.440+0000 7f0407fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04081092b0 0x7f0408115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.440+0000 7f0407fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f040810fed0 0x7f04081161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.440+0000 7f0407fff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04081208b0 con 0x7f04081092b0 2024-05-09T08:52:32.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.440+0000 7f0407fff640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0408120730 con 0x7f0408108830 2024-05-09T08:52:32.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.440+0000 7f0407fff640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0408120a30 con 0x7f040810fed0 2024-05-09T08:52:32.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.440+0000 7f04077fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f040810fed0 0x7f04081161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:32.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.440+0000 7f04077fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f040810fed0 0x7f04081161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55610/0 (socket says 172.21.15.98:55610) 2024-05-09T08:52:32.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.440+0000 7f04077fe640 1 -- 172.21.15.98:0/3059816269 learned_addr learned my addr 172.21.15.98:0/3059816269 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:32.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.441+0000 7f04077fe640 1 -- 172.21.15.98:0/3059816269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0408108830 msgr2=0x7f040811cbf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:52:32.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.441+0000 7f04077fe640 1 --2- 172.21.15.98:0/3059816269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0408108830 0x7f040811cbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.441+0000 7f04077fe640 1 -- 172.21.15.98:0/3059816269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04081092b0 msgr2=0x7f0408115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:32.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.441+0000 7f04077fe640 1 --2- 172.21.15.98:0/3059816269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04081092b0 0x7f0408115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.441+0000 7f04077fe640 1 -- 172.21.15.98:0/3059816269 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0408116770 con 0x7f040810fed0 2024-05-09T08:52:32.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.441+0000 7f04077fe640 1 --2- 172.21.15.98:0/3059816269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f040810fed0 0x7f04081161b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00e970 tx=0x7f03fc00ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:32.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.446+0000 7f03effff640 1 -- 172.21.15.98:0/3059816269 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc00ccc0 con 0x7f040810fed0 2024-05-09T08:52:32.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.446+0000 7f03effff640 1 -- 172.21.15.98:0/3059816269 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03fc005500 con 0x7f040810fed0 2024-05-09T08:52:32.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.447+0000 7f03effff640 1 -- 172.21.15.98:0/3059816269 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc010430 con 0x7f040810fed0 2024-05-09T08:52:32.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.449+0000 7f0407fff640 1 -- 172.21.15.98:0/3059816269 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0408116a60 con 0x7f040810fed0 2024-05-09T08:52:32.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.449+0000 7f0407fff640 1 -- 172.21.15.98:0/3059816269 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f040811aa10 con 0x7f040810fed0 2024-05-09T08:52:32.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.456+0000 7f041024a740 1 Processor -- start 2024-05-09T08:52:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.456+0000 7f041024a740 1 -- start start 2024-05-09T08:52:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.457+0000 7f041024a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.457+0000 7f041024a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.457+0000 7f041024a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1d3930 0x55b0dc1d5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.457+0000 7f041024a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b0dc12d990 con 0x55b0dc1c8bb0 2024-05-09T08:52:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.457+0000 7f041024a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b0dc0d55b0 con 0x55b0dc1d3930 2024-05-09T08:52:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.457+0000 7f041024a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b0dc129950 con 0x55b0dc1ca2a0 2024-05-09T08:52:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.457+0000 7f03effff640 1 -- 172.21.15.98:0/3059816269 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f03fc004680 con 0x7f040810fed0 2024-05-09T08:52:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.457+0000 7f04067fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1d3930 0x55b0dc1d5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.457+0000 7f04067fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1d3930 0x55b0dc1d5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:53374/0 (socket says 172.21.15.98:53374) 2024-05-09T08:52:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.457+0000 7f04067fc640 1 -- 172.21.15.98:0/2454727994 learned_addr learned my addr 172.21.15.98:0/2454727994 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.457+0000 7f0406ffd640 1 --2- 172.21.15.98:0/2454727994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.457+0000 7f04067fc640 1 --2- 172.21.15.98:0/2454727994 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.463+0000 7f03effff640 1 --2- 172.21.15.98:0/3059816269 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f03f00a2d60 0x7f03f00a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.464+0000 7f03effff640 1 -- 172.21.15.98:0/3059816269 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f03fc0cfb30 con 0x7f040810fed0 2024-05-09T08:52:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.464+0000 7f04067fc640 1 -- 172.21.15.98:0/2454727994 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 msgr2=0x55b0dc1c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.464+0000 7f04067fc640 1 --2- 172.21.15.98:0/2454727994 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.464+0000 7f04067fc640 1 -- 172.21.15.98:0/2454727994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 msgr2=0x55b0dc1c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.464+0000 7f04067fc640 1 --2- 172.21.15.98:0/2454727994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1c9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.464+0000 7f04067fc640 1 -- 172.21.15.98:0/2454727994 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0dbf097e0 con 0x55b0dc1d3930 2024-05-09T08:52:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.465+0000 7f04067fc640 1 --2- 172.21.15.98:0/2454727994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1d3930 0x55b0dc1d5d20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0400034a70 tx=0x7f0400009660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1755c52522ad86ee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:32.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.466+0000 7f03ed7fa640 1 -- 172.21.15.98:0/2454727994 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040002f9b0 con 0x55b0dc1d3930 2024-05-09T08:52:32.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.466+0000 7f03ed7fa640 1 -- 172.21.15.98:0/2454727994 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f040002fb50 con 0x55b0dc1d3930 2024-05-09T08:52:32.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.467+0000 7f03ed7fa640 1 -- 172.21.15.98:0/2454727994 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04000313d0 con 0x55b0dc1d3930 2024-05-09T08:52:32.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.467+0000 7f04077fe640 1 --2- 172.21.15.98:0/3059816269 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f03f00a2d60 0x7f03f00a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:32.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.468+0000 7f041024a740 1 -- 172.21.15.98:0/2454727994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1d3930 msgr2=0x55b0dc1d5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:32.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.468+0000 7f041024a740 1 --2- 172.21.15.98:0/2454727994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1d3930 0x55b0dc1d5d20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0400034a70 tx=0x7f0400009660 comp rx=0 tx=0).stop 2024-05-09T08:52:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.474+0000 7f041024a740 1 -- 172.21.15.98:0/2454727994 shutdown_connections 2024-05-09T08:52:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.474+0000 7f041024a740 1 --2- 172.21.15.98:0/2454727994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1d3930 0x55b0dc1d5d20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.474+0000 7f041024a740 1 --2- 172.21.15.98:0/2454727994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1c9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.474+0000 7f041024a740 1 --2- 172.21.15.98:0/2454727994 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.474+0000 7f041024a740 1 -- 172.21.15.98:0/2454727994 >> 172.21.15.98:0/2454727994 conn(0x55b0dc1910a0 msgr2=0x55b0dc1c5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:32.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.476+0000 7f041024a740 1 -- 172.21.15.98:0/2454727994 shutdown_connections 2024-05-09T08:52:32.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.477+0000 7f04077fe640 1 --2- 172.21.15.98:0/3059816269 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f03f00a2d60 0x7f03f00a5220 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f03fc0a4370 tx=0x7f03fc0094a0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:32.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.478+0000 7f041024a740 1 -- 172.21.15.98:0/2454727994 wait complete. 2024-05-09T08:52:32.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.479+0000 7f041024a740 1 -- start start 2024-05-09T08:52:32.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.479+0000 7f041024a740 1 Processor -- start 2024-05-09T08:52:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.480+0000 7f041024a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.480+0000 7f041024a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.480+0000 7f041024a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1d3930 0x55b0dc1d1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.480+0000 7f041024a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b0dc1d67c0 con 0x55b0dc1d3930 2024-05-09T08:52:32.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.480+0000 7f041024a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b0dc0e8b40 con 0x55b0dc1c8bb0 2024-05-09T08:52:32.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.480+0000 7f041024a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b0dc1d6940 con 0x55b0dc1ca2a0 2024-05-09T08:52:32.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.480+0000 7f041024a740 10 client.0 _ll_register_callbacks cb 0x55b0dc0dd0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:32.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.480+0000 7f0406ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:32.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.481+0000 7f04067fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1d3930 0x55b0dc1d1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.481+0000 7f04067fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1d3930 0x55b0dc1d1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47406/0 (socket says 172.21.15.98:47406) 2024-05-09T08:52:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.481+0000 7f04067fc640 1 -- 172.21.15.98:0/2783840929 learned_addr learned my addr 172.21.15.98:0/2783840929 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.481+0000 7f04067fc640 1 --2- 172.21.15.98:0/2783840929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.482+0000 7f04067fc640 1 -- 172.21.15.98:0/2783840929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 msgr2=0x55b0dc1d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.482+0000 7f04067fc640 1 --2- 172.21.15.98:0/2783840929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1d0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.482+0000 7f04067fc640 1 -- 172.21.15.98:0/2783840929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1d3930 msgr2=0x55b0dc1d1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.482+0000 7f04067fc640 1 --2- 172.21.15.98:0/2783840929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1d3930 0x55b0dc1d1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.482+0000 7f04067fc640 1 -- 172.21.15.98:0/2783840929 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0dbf69f70 con 0x55b0dc1c8bb0 2024-05-09T08:52:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.483+0000 7f04067fc640 1 --2- 172.21.15.98:0/2783840929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1d0900 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0400005fd0 tx=0x7f0400004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10e464b7418f5686 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.483+0000 7f03eaff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:52:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.483+0000 7f03eaff5640 1 -- 172.21.15.98:0/2783840929 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0400049040 con 0x55b0dc1c8bb0 2024-05-09T08:52:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.484+0000 7f041024a740 1 -- 172.21.15.98:0/2783840929 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f040002fb50 con 0x55b0dc1c8bb0 2024-05-09T08:52:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.484+0000 7f03eaff5640 1 -- 172.21.15.98:0/2783840929 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f040002f870 con 0x55b0dc1c8bb0 2024-05-09T08:52:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.484+0000 7f03eaff5640 1 -- 172.21.15.98:0/2783840929 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0400057050 con 0x55b0dc1c8bb0 2024-05-09T08:52:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.485+0000 7f041024a740 10 client.25679 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.485+0000 7f041024a740 1 -- 172.21.15.98:0/2783840929 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b0dbe01170 con 0x55b0dc1c8bb0 2024-05-09T08:52:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.486+0000 7f03eaff5640 1 -- 172.21.15.98:0/2783840929 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f040002fb50 con 0x55b0dc1c8bb0 2024-05-09T08:52:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.486+0000 7f03eaff5640 1 -- 172.21.15.98:0/2783840929 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f040003e070 con 0x55b0dc1c8bb0 2024-05-09T08:52:32.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.487+0000 7f041024a740 10 client.25679 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:32.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.488+0000 7f041024a740 10 client.25679 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:32.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.488+0000 7f041024a740 20 client.25679 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:32.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.488+0000 7f041024a740 1 --2- 172.21.15.98:0/2783840929 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b0dc1e3c10 0x55b0dc1e6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.488+0000 7f041024a740 4 client.25679 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:52:32.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.488+0000 7f041024a740 1 -- 172.21.15.98:0/2783840929 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0dc13b0c0 con 0x55b0dc1e3c10 2024-05-09T08:52:32.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.488+0000 7f0406ffd640 1 --2- 172.21.15.98:0/2783840929 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b0dc1e3c10 0x55b0dc1e6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:32.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.490+0000 7f0406ffd640 1 --2- 172.21.15.98:0/2783840929 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b0dc1e3c10 0x55b0dc1e6040 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ab2878712f7ac3f server_cookie=3dd6d0024c8a2af4 in_seq=0 out_seq=0 2024-05-09T08:52:32.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.491+0000 7f03eaff5640 10 client.25679 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.495+0000 7f03eaff5640 1 -- 172.21.15.98:0/2783840929 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b0dc13b0c0 con 0x55b0dc1e3c10 2024-05-09T08:52:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.495+0000 7f03eaff5640 10 client.25679 handle_command_reply: tid=0 2024-05-09T08:52:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.495+0000 7f041024a740 1 client.25679 shutdown 2024-05-09T08:52:32.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.500+0000 7f041024a740 1 -- 172.21.15.98:0/2783840929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1c8bb0 msgr2=0x55b0dc1d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:32.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.500+0000 7f041024a740 1 --2- 172.21.15.98:0/2783840929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1d0900 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0400005fd0 tx=0x7f0400004a40 comp rx=0 tx=0).stop 2024-05-09T08:52:32.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.502+0000 7f041024a740 1 -- 172.21.15.98:0/2783840929 shutdown_connections 2024-05-09T08:52:32.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.502+0000 7f041024a740 1 --2- 172.21.15.98:0/2783840929 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b0dc1e3c10 0x55b0dc1e6040 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.502+0000 7f041024a740 1 --2- 172.21.15.98:0/2783840929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1d3930 0x55b0dc1d1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.502+0000 7f041024a740 1 --2- 172.21.15.98:0/2783840929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1d0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.502+0000 7f041024a740 1 --2- 172.21.15.98:0/2783840929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1d0900 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.502+0000 7f041024a740 1 -- 172.21.15.98:0/2783840929 >> 172.21.15.98:0/2783840929 conn(0x55b0dc1910a0 msgr2=0x55b0dc1caf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:32.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.502+0000 7f03eaff5640 0 client.25679 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.505+0000 7f041024a740 1 -- 172.21.15.98:0/2783840929 shutdown_connections 2024-05-09T08:52:32.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.507+0000 7f041024a740 1 -- 172.21.15.98:0/2783840929 wait complete. 2024-05-09T08:52:32.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.508+0000 7f041024a740 20 client.25679 trim_cache size 0 max 16384 2024-05-09T08:52:32.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.552+0000 7f041024a740 1 Processor -- start 2024-05-09T08:52:32.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.552+0000 7f041024a740 1 -- start start 2024-05-09T08:52:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.553+0000 7f041024a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1df6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.553+0000 7f041024a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.553+0000 7f041024a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1d3930 0x55b0dc1eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.553+0000 7f041024a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0408082c70 con 0x55b0dc1c8bb0 2024-05-09T08:52:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.553+0000 7f0406ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.553+0000 7f0406ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1d0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:53402/0 (socket says 172.21.15.98:53402) 2024-05-09T08:52:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.554+0000 7f0406ffd640 1 -- 172.21.15.98:0/1686004617 learned_addr learned my addr 172.21.15.98:0/1686004617 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.554+0000 7f041024a740 1 -- 172.21.15.98:0/1686004617 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f04080847b0 con 0x55b0dc1ca2a0 2024-05-09T08:52:32.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.554+0000 7f041024a740 1 -- 172.21.15.98:0/1686004617 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0408083340 con 0x55b0dc1d3930 2024-05-09T08:52:32.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.557+0000 7f0406ffd640 1 -- 172.21.15.98:0/1686004617 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1d3930 msgr2=0x55b0dc1eb8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:52:32.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.557+0000 7f0406ffd640 1 --2- 172.21.15.98:0/1686004617 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1d3930 0x55b0dc1eb8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.557+0000 7f0406ffd640 1 -- 172.21.15.98:0/1686004617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 msgr2=0x55b0dc1df6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:32.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.557+0000 7f0406ffd640 1 --2- 172.21.15.98:0/1686004617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1df6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.557+0000 7f0406ffd640 1 -- 172.21.15.98:0/1686004617 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0dc1d6350 con 0x55b0dc1ca2a0 2024-05-09T08:52:32.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.558+0000 7f0406ffd640 1 --2- 172.21.15.98:0/1686004617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1d0200 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f03f80095f0 tx=0x7f03f802a910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98a3229819de300c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:32.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.558+0000 7f03ecff9640 1 -- 172.21.15.98:0/1686004617 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f804a070 con 0x55b0dc1ca2a0 2024-05-09T08:52:32.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.558+0000 7f03ecff9640 1 -- 172.21.15.98:0/1686004617 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03f8045dc0 con 0x55b0dc1ca2a0 2024-05-09T08:52:32.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.559+0000 7f03ecff9640 1 -- 172.21.15.98:0/1686004617 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f80052a0 con 0x55b0dc1ca2a0 2024-05-09T08:52:32.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.560+0000 7f041024a740 1 -- 172.21.15.98:0/1686004617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1ca2a0 msgr2=0x55b0dc1d0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:32.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.560+0000 7f041024a740 1 --2- 172.21.15.98:0/1686004617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1d0200 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f03f80095f0 tx=0x7f03f802a910 comp rx=0 tx=0).stop 2024-05-09T08:52:32.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.560+0000 7f041024a740 1 -- 172.21.15.98:0/1686004617 shutdown_connections 2024-05-09T08:52:32.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.560+0000 7f041024a740 1 --2- 172.21.15.98:0/1686004617 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1d3930 0x55b0dc1eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.560+0000 7f041024a740 1 --2- 172.21.15.98:0/1686004617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1d0200 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.560+0000 7f041024a740 1 --2- 172.21.15.98:0/1686004617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1df6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.560+0000 7f041024a740 1 -- 172.21.15.98:0/1686004617 >> 172.21.15.98:0/1686004617 conn(0x55b0dc1910a0 msgr2=0x55b0dc1c4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:32.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.560+0000 7f041024a740 1 -- 172.21.15.98:0/1686004617 shutdown_connections 2024-05-09T08:52:32.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.560+0000 7f041024a740 1 -- 172.21.15.98:0/1686004617 wait complete. 2024-05-09T08:52:32.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.561+0000 7f041024a740 1 -- start start 2024-05-09T08:52:32.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.561+0000 7f041024a740 1 Processor -- start 2024-05-09T08:52:32.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.562+0000 7f041024a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1dbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.562+0000 7f041024a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.562+0000 7f041024a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1d3930 0x55b0dc1cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.562+0000 7f041024a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04080840e0 con 0x55b0dc1c8bb0 2024-05-09T08:52:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.562+0000 7f04067fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1dbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.562+0000 7f041024a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0408083a10 con 0x55b0dc1d3930 2024-05-09T08:52:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.562+0000 7f04067fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1dbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47422/0 (socket says 172.21.15.98:47422) 2024-05-09T08:52:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.562+0000 7f041024a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f040811b540 con 0x55b0dc1ca2a0 2024-05-09T08:52:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.562+0000 7f04067fc640 1 -- 172.21.15.98:0/584382029 learned_addr learned my addr 172.21.15.98:0/584382029 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.562+0000 7f041024a740 10 client.0 _ll_register_callbacks cb 0x7f0400034420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.562+0000 7f04067fc640 1 --2- 172.21.15.98:0/584382029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1d3930 0x55b0dc1cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.563+0000 7f0406ffd640 1 --2- 172.21.15.98:0/584382029 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.563+0000 7f04067fc640 1 -- 172.21.15.98:0/584382029 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 msgr2=0x55b0dc1cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.563+0000 7f04067fc640 1 --2- 172.21.15.98:0/584382029 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1cd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.563+0000 7f04067fc640 1 -- 172.21.15.98:0/584382029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 msgr2=0x55b0dc1dbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.563+0000 7f04067fc640 1 --2- 172.21.15.98:0/584382029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1dbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.563+0000 7f04067fc640 1 -- 172.21.15.98:0/584382029 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0dc1cdce0 con 0x55b0dc1d3930 2024-05-09T08:52:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.563+0000 7f04067fc640 1 --2- 172.21.15.98:0/584382029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1d3930 0x55b0dc1cd6f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f04000096a0 tx=0x7f0400000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b37732a3df97b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.563+0000 7f03eaff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:52:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.563+0000 7f03eaff5640 1 -- 172.21.15.98:0/584382029 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0400009c60 con 0x55b0dc1d3930 2024-05-09T08:52:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.563+0000 7f03eaff5640 1 -- 172.21.15.98:0/584382029 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f0400003970 con 0x55b0dc1d3930 2024-05-09T08:52:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.564+0000 7f041024a740 1 -- 172.21.15.98:0/584382029 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03f8045dc0 con 0x55b0dc1d3930 2024-05-09T08:52:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.564+0000 7f03eaff5640 1 -- 172.21.15.98:0/584382029 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0400003b70 con 0x55b0dc1d3930 2024-05-09T08:52:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.564+0000 7f041024a740 10 client.25685 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.564+0000 7f041024a740 1 -- 172.21.15.98:0/584382029 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b0dc249f40 con 0x55b0dc1d3930 2024-05-09T08:52:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.564+0000 7f03eaff5640 1 -- 172.21.15.98:0/584382029 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03f8045dc0 con 0x55b0dc1d3930 2024-05-09T08:52:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.565+0000 7f03eaff5640 1 -- 172.21.15.98:0/584382029 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f040005a060 con 0x55b0dc1d3930 2024-05-09T08:52:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.565+0000 7f041024a740 10 client.25685 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.565+0000 7f041024a740 10 client.25685 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.565+0000 7f041024a740 20 client.25685 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.565+0000 7f041024a740 1 --2- 172.21.15.98:0/584382029 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b0dc24a5d0 0x55b0dc24c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.565+0000 7f041024a740 4 client.25685 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:52:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.565+0000 7f041024a740 1 -- 172.21.15.98:0/584382029 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b0dc135410 con 0x55b0dc24a5d0 2024-05-09T08:52:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.565+0000 7f0406ffd640 1 --2- 172.21.15.98:0/584382029 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b0dc24a5d0 0x55b0dc24c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.566+0000 7f03eaff5640 10 client.25685 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.566+0000 7f0406ffd640 1 --2- 172.21.15.98:0/584382029 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b0dc24a5d0 0x55b0dc24c9f0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be06b118da91cc56 server_cookie=30130598215d1a8c in_seq=0 out_seq=0 2024-05-09T08:52:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.664+0000 7f03eaff5640 1 -- 172.21.15.98:0/584382029 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b0dc135410 con 0x55b0dc24a5d0 2024-05-09T08:52:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.664+0000 7f03eaff5640 10 client.25685 handle_command_reply: tid=0 2024-05-09T08:52:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.664+0000 7f041024a740 1 client.25685 shutdown 2024-05-09T08:52:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.668+0000 7f041024a740 1 -- 172.21.15.98:0/584382029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1d3930 msgr2=0x55b0dc1cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.668+0000 7f041024a740 1 --2- 172.21.15.98:0/584382029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1d3930 0x55b0dc1cd6f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f04000096a0 tx=0x7f0400000f40 comp rx=0 tx=0).stop 2024-05-09T08:52:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.676+0000 7f041024a740 1 -- 172.21.15.98:0/584382029 shutdown_connections 2024-05-09T08:52:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.676+0000 7f041024a740 1 --2- 172.21.15.98:0/584382029 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b0dc24a5d0 0x55b0dc24c9f0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.676+0000 7f041024a740 1 --2- 172.21.15.98:0/584382029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0dc1d3930 0x55b0dc1cd6f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.677+0000 7f041024a740 1 --2- 172.21.15.98:0/584382029 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b0dc1ca2a0 0x55b0dc1cd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.677+0000 7f041024a740 1 --2- 172.21.15.98:0/584382029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0dc1c8bb0 0x55b0dc1dbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.677+0000 7f041024a740 1 -- 172.21.15.98:0/584382029 >> 172.21.15.98:0/584382029 conn(0x55b0dc1910a0 msgr2=0x55b0dc1ca680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.677+0000 7f03eaff5640 0 client.25685 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.678+0000 7f041024a740 1 -- 172.21.15.98:0/584382029 shutdown_connections 2024-05-09T08:52:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.682+0000 7f041024a740 1 -- 172.21.15.98:0/584382029 wait complete. 2024-05-09T08:52:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.683+0000 7f041024a740 20 client.25685 trim_cache size 0 max 16384 2024-05-09T08:52:32.684 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:52:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:52:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "b5a48e69-e2e2-4416-97f5-fb3a51b01ffa", 2024-05-09T08:52:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:52:32.684 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:52:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.685+0000 7f0407fff640 1 -- 172.21.15.98:0/3059816269 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f03f00a2d60 msgr2=0x7f03f00a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.685+0000 7f0407fff640 1 --2- 172.21.15.98:0/3059816269 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f03f00a2d60 0x7f03f00a5220 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f03fc0a4370 tx=0x7f03fc0094a0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.685+0000 7f0407fff640 1 -- 172.21.15.98:0/3059816269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f040810fed0 msgr2=0x7f04081161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.685+0000 7f0407fff640 1 --2- 172.21.15.98:0/3059816269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f040810fed0 0x7f04081161b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00e970 tx=0x7f03fc00ee30 comp rx=0 tx=0).stop 2024-05-09T08:52:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.692+0000 7f0407fff640 1 -- 172.21.15.98:0/3059816269 shutdown_connections 2024-05-09T08:52:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.692+0000 7f0407fff640 1 --2- 172.21.15.98:0/3059816269 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f03f00a2d60 0x7f03f00a5220 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.692+0000 7f0407fff640 1 --2- 172.21.15.98:0/3059816269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f040810fed0 0x7f04081161b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.692+0000 7f0407fff640 1 --2- 172.21.15.98:0/3059816269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04081092b0 0x7f0408115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.692+0000 7f0407fff640 1 --2- 172.21.15.98:0/3059816269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0408108830 0x7f040811cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.692+0000 7f0407fff640 1 -- 172.21.15.98:0/3059816269 >> 172.21.15.98:0/3059816269 conn(0x7f040806c7b0 msgr2=0x7f04080709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.693+0000 7f0407fff640 1 -- 172.21.15.98:0/3059816269 shutdown_connections 2024-05-09T08:52:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:32.693+0000 7f0407fff640 1 -- 172.21.15.98:0/3059816269 wait complete. 2024-05-09T08:52:32.731 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:52:32.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:52:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.018+0000 7f8995149640 1 -- 172.21.15.98:0/2722913872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990108830 msgr2=0x7f8990108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.018+0000 7f8995149640 1 --2- 172.21.15.98:0/2722913872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990108830 0x7f8990108c90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8988009ca0 tx=0x7f898802a970 comp rx=0 tx=0).stop 2024-05-09T08:52:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.019+0000 7f8995149640 1 -- 172.21.15.98:0/2722913872 shutdown_connections 2024-05-09T08:52:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.019+0000 7f8995149640 1 --2- 172.21.15.98:0/2722913872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89901092b0 0x7f899011d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.019+0000 7f8995149640 1 --2- 172.21.15.98:0/2722913872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990108830 0x7f8990108c90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.019+0000 7f8995149640 1 --2- 172.21.15.98:0/2722913872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f899010fed0 0x7f89901102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.019+0000 7f8995149640 1 -- 172.21.15.98:0/2722913872 >> 172.21.15.98:0/2722913872 conn(0x7f899006c7b0 msgr2=0x7f899006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.020+0000 7f8995149640 1 -- 172.21.15.98:0/2722913872 shutdown_connections 2024-05-09T08:52:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.022+0000 7f8995149640 1 -- 172.21.15.98:0/2722913872 wait complete. 2024-05-09T08:52:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.023+0000 7f8995149640 1 Processor -- start 2024-05-09T08:52:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.023+0000 7f8995149640 1 -- start start 2024-05-09T08:52:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.023+0000 7f8995149640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990108830 0x7f899011cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.023+0000 7f8995149640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89901092b0 0x7f899011d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.023+0000 7f8995149640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f899010fed0 0x7f8990115dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.023+0000 7f8995149640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89901208b0 con 0x7f899010fed0 2024-05-09T08:52:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.023+0000 7f8995149640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8990120730 con 0x7f8990108830 2024-05-09T08:52:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.023+0000 7f8995149640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8990120a30 con 0x7f89901092b0 2024-05-09T08:52:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.024+0000 7f898ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990108830 0x7f899011cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.024+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89901092b0 0x7f899011d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.024+0000 7f898ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990108830 0x7f899011cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:53430/0 (socket says 172.21.15.98:53430) 2024-05-09T08:52:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.024+0000 7f898ffff640 1 -- 172.21.15.98:0/2636692528 learned_addr learned my addr 172.21.15.98:0/2636692528 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.024+0000 7f8994948640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f899010fed0 0x7f8990115dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.024+0000 7f898ffff640 1 -- 172.21.15.98:0/2636692528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89901092b0 msgr2=0x7f899011d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.024+0000 7f898ffff640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89901092b0 0x7f899011d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.024+0000 7f898ffff640 1 -- 172.21.15.98:0/2636692528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f899010fed0 msgr2=0x7f8990115dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.024+0000 7f898ffff640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f899010fed0 0x7f8990115dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.024+0000 7f898ffff640 1 -- 172.21.15.98:0/2636692528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8990116690 con 0x7f8990108830 2024-05-09T08:52:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.024+0000 7f898f7fe640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89901092b0 0x7f899011d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:52:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.024+0000 7f8994948640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f899010fed0 0x7f8990115dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:52:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.024+0000 7f898ffff640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990108830 0x7f899011cb90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f898000e9f0 tx=0x7f898000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.024+0000 7f898d7fa640 1 -- 172.21.15.98:0/2636692528 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898000cd80 con 0x7f8990108830 2024-05-09T08:52:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.025+0000 7f898d7fa640 1 -- 172.21.15.98:0/2636692528 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8980010910 con 0x7f8990108830 2024-05-09T08:52:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.025+0000 7f898d7fa640 1 -- 172.21.15.98:0/2636692528 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8980018990 con 0x7f8990108830 2024-05-09T08:52:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.025+0000 7f8995149640 1 -- 172.21.15.98:0/2636692528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8990116980 con 0x7f8990108830 2024-05-09T08:52:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.025+0000 7f8995149640 1 -- 172.21.15.98:0/2636692528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f899011a8f0 con 0x7f8990108830 2024-05-09T08:52:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.033+0000 7f899918a740 1 Processor -- start 2024-05-09T08:52:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.033+0000 7f898d7fa640 1 -- 172.21.15.98:0/2636692528 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8980010ac0 con 0x7f8990108830 2024-05-09T08:52:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.034+0000 7f899918a740 1 -- start start 2024-05-09T08:52:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.035+0000 7f899918a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d681d430 0x5627d681d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.035+0000 7f899918a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681dd50 0x5627d6827ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.035+0000 7f899918a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6828420 0x5627d682a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.035+0000 7f899918a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5627d6782d00 con 0x5627d6828420 2024-05-09T08:52:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.035+0000 7f899918a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5627d672a4c0 con 0x5627d681d430 2024-05-09T08:52:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.035+0000 7f899918a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5627d677ecc0 con 0x5627d681dd50 2024-05-09T08:52:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.035+0000 7f898ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681dd50 0x5627d6827ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.035+0000 7f898ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681dd50 0x5627d6827ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55670/0 (socket says 172.21.15.98:55670) 2024-05-09T08:52:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.035+0000 7f898ffff640 1 -- 172.21.15.98:0/3351011818 learned_addr learned my addr 172.21.15.98:0/3351011818 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.036+0000 7f8994948640 1 --2- 172.21.15.98:0/3351011818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d681d430 0x5627d681d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.037+0000 7f898ffff640 1 -- 172.21.15.98:0/3351011818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d681d430 msgr2=0x5627d681d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.037+0000 7f898ffff640 1 --2- 172.21.15.98:0/3351011818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d681d430 0x5627d681d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.037+0000 7f898ffff640 1 -- 172.21.15.98:0/3351011818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6828420 msgr2=0x5627d682a810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:52:33.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.037+0000 7f898ffff640 1 --2- 172.21.15.98:0/3351011818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6828420 0x5627d682a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.037+0000 7f898ffff640 1 -- 172.21.15.98:0/3351011818 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627d65607d0 con 0x5627d681dd50 2024-05-09T08:52:33.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.041+0000 7f898d7fa640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89780a2ed0 0x7f89780a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.041+0000 7f898ffff640 1 --2- 172.21.15.98:0/3351011818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681dd50 0x5627d6827ee0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f89800ccd30 tx=0x7f89800d8130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0bbd27312ff66e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:33.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.041+0000 7f8994948640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89780a2ed0 0x7f89780a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:33.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.042+0000 7f89767fc640 1 -- 172.21.15.98:0/3351011818 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8980097e60 con 0x5627d681dd50 2024-05-09T08:52:33.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.042+0000 7f89767fc640 1 -- 172.21.15.98:0/3351011818 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f898009c2c0 con 0x5627d681dd50 2024-05-09T08:52:33.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.042+0000 7f898d7fa640 1 -- 172.21.15.98:0/2636692528 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f89800cfac0 con 0x7f8990108830 2024-05-09T08:52:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.045+0000 7f899918a740 1 -- 172.21.15.98:0/3351011818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681dd50 msgr2=0x5627d6827ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.045+0000 7f899918a740 1 --2- 172.21.15.98:0/3351011818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681dd50 0x5627d6827ee0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f89800ccd30 tx=0x7f89800d8130 comp rx=0 tx=0).stop 2024-05-09T08:52:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.045+0000 7f8994948640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89780a2ed0 0x7f89780a5390 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f898400bc20 tx=0x7f8984005e90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.045+0000 7f899918a740 1 -- 172.21.15.98:0/3351011818 shutdown_connections 2024-05-09T08:52:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.045+0000 7f899918a740 1 --2- 172.21.15.98:0/3351011818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6828420 0x5627d682a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.045+0000 7f899918a740 1 --2- 172.21.15.98:0/3351011818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681dd50 0x5627d6827ee0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.045+0000 7f899918a740 1 --2- 172.21.15.98:0/3351011818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d681d430 0x5627d681d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.045+0000 7f899918a740 1 -- 172.21.15.98:0/3351011818 >> 172.21.15.98:0/3351011818 conn(0x5627d6819750 msgr2=0x5627d6819b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.047+0000 7f899918a740 1 -- 172.21.15.98:0/3351011818 shutdown_connections 2024-05-09T08:52:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.048+0000 7f899918a740 1 -- 172.21.15.98:0/3351011818 wait complete. 2024-05-09T08:52:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.048+0000 7f899918a740 1 -- start start 2024-05-09T08:52:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.049+0000 7f899918a740 1 Processor -- start 2024-05-09T08:52:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.049+0000 7f899918a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681d430 0x5627d6827e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.049+0000 7f899918a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d6828420 0x5627d6825100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.049+0000 7f899918a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6825690 0x5627d6825af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.049+0000 7f899918a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5627d682b2b0 con 0x5627d6825690 2024-05-09T08:52:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.049+0000 7f899918a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5627d673db90 con 0x5627d6828420 2024-05-09T08:52:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.049+0000 7f899918a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5627d682b430 con 0x5627d681d430 2024-05-09T08:52:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.049+0000 7f899918a740 10 client.0 _ll_register_callbacks cb 0x5627d6779da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.050+0000 7f898ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681d430 0x5627d6827e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.050+0000 7f898ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681d430 0x5627d6827e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55684/0 (socket says 172.21.15.98:55684) 2024-05-09T08:52:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.050+0000 7f898ffff640 1 -- 172.21.15.98:0/231059592 learned_addr learned my addr 172.21.15.98:0/231059592 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.050+0000 7f898f7fe640 1 --2- 172.21.15.98:0/231059592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d6828420 0x5627d6825100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.050+0000 7f898ffff640 1 -- 172.21.15.98:0/231059592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d6828420 msgr2=0x5627d6825100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.050+0000 7f898ffff640 1 --2- 172.21.15.98:0/231059592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d6828420 0x5627d6825100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.050+0000 7f898ffff640 1 -- 172.21.15.98:0/231059592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6825690 msgr2=0x5627d6825af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.050+0000 7f898ffff640 1 --2- 172.21.15.98:0/231059592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6825690 0x5627d6825af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.050+0000 7f898ffff640 1 -- 172.21.15.98:0/231059592 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627d64963f0 con 0x5627d681d430 2024-05-09T08:52:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.051+0000 7f898ffff640 1 --2- 172.21.15.98:0/231059592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681d430 0x5627d6827e10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f898000e9c0 tx=0x7f898001f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e1be309993e2fd7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.051+0000 7f896f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:52:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.052+0000 7f896f7fe640 1 -- 172.21.15.98:0/231059592 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898009b700 con 0x5627d681d430 2024-05-09T08:52:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.052+0000 7f899918a740 1 -- 172.21.15.98:0/231059592 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5627d6790430 con 0x5627d681d430 2024-05-09T08:52:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.052+0000 7f896f7fe640 1 -- 172.21.15.98:0/231059592 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f8980099c60 con 0x5627d681d430 2024-05-09T08:52:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.052+0000 7f896f7fe640 1 -- 172.21.15.98:0/231059592 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f898009b990 con 0x5627d681d430 2024-05-09T08:52:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.052+0000 7f899918a740 10 client.25642 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.052+0000 7f899918a740 1 -- 172.21.15.98:0/231059592 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5627d64560f0 con 0x5627d681d430 2024-05-09T08:52:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.053+0000 7f896f7fe640 1 -- 172.21.15.98:0/231059592 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5627d6790430 con 0x5627d681d430 2024-05-09T08:52:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.053+0000 7f896f7fe640 1 -- 172.21.15.98:0/231059592 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f89800a43a0 con 0x5627d681d430 2024-05-09T08:52:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.054+0000 7f899918a740 10 client.25642 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.054+0000 7f899918a740 10 client.25642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.054+0000 7f899918a740 20 client.25642 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.054+0000 7f899918a740 1 --2- 172.21.15.98:0/231059592 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5627d6838cc0 0x5627d683b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.054+0000 7f899918a740 4 client.25642 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:52:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.054+0000 7f899918a740 1 -- 172.21.15.98:0/231059592 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627d678a780 con 0x5627d6838cc0 2024-05-09T08:52:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.055+0000 7f898f7fe640 1 --2- 172.21.15.98:0/231059592 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5627d6838cc0 0x5627d683b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.055+0000 7f898f7fe640 1 --2- 172.21.15.98:0/231059592 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5627d6838cc0 0x5627d683b110 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da00fdfd0645523d server_cookie=4c94cafec6e43e37 in_seq=0 out_seq=0 2024-05-09T08:52:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.056+0000 7f896f7fe640 10 client.25642 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.057+0000 7f896f7fe640 1 -- 172.21.15.98:0/231059592 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5627d678a780 con 0x5627d6838cc0 2024-05-09T08:52:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.057+0000 7f896f7fe640 10 client.25642 handle_command_reply: tid=0 2024-05-09T08:52:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.057+0000 7f899918a740 1 client.25642 shutdown 2024-05-09T08:52:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.058+0000 7f899918a740 1 -- 172.21.15.98:0/231059592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681d430 msgr2=0x5627d6827e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.058+0000 7f899918a740 1 --2- 172.21.15.98:0/231059592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681d430 0x5627d6827e10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f898000e9c0 tx=0x7f898001f000 comp rx=0 tx=0).stop 2024-05-09T08:52:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.058+0000 7f899918a740 1 -- 172.21.15.98:0/231059592 shutdown_connections 2024-05-09T08:52:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.058+0000 7f899918a740 1 --2- 172.21.15.98:0/231059592 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5627d6838cc0 0x5627d683b110 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.058+0000 7f899918a740 1 --2- 172.21.15.98:0/231059592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6825690 0x5627d6825af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.058+0000 7f899918a740 1 --2- 172.21.15.98:0/231059592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d6828420 0x5627d6825100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.058+0000 7f899918a740 1 --2- 172.21.15.98:0/231059592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681d430 0x5627d6827e10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.058+0000 7f899918a740 1 -- 172.21.15.98:0/231059592 >> 172.21.15.98:0/231059592 conn(0x5627d6819750 msgr2=0x5627d681ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.058+0000 7f896f7fe640 0 client.25642 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.059+0000 7f899918a740 1 -- 172.21.15.98:0/231059592 shutdown_connections 2024-05-09T08:52:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.059+0000 7f899918a740 1 -- 172.21.15.98:0/231059592 wait complete. 2024-05-09T08:52:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.059+0000 7f899918a740 20 client.25642 trim_cache size 0 max 16384 2024-05-09T08:52:33.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.096+0000 7f899918a740 1 Processor -- start 2024-05-09T08:52:33.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.096+0000 7f899918a740 1 -- start start 2024-05-09T08:52:33.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.096+0000 7f899918a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d681d430 0x5627d6827940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.097+0000 7f899918a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d6825690 0x5627d68409c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.097+0000 7f899918a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d6828420 0x5627d6840f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.097+0000 7f899918a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89901b51d0 con 0x5627d681d430 2024-05-09T08:52:33.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.097+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d6825690 0x5627d68409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:33.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.097+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d6825690 0x5627d68409c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55696/0 (socket says 172.21.15.98:55696) 2024-05-09T08:52:33.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.097+0000 7f898f7fe640 1 -- 172.21.15.98:0/3514450908 learned_addr learned my addr 172.21.15.98:0/3514450908 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:33.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.097+0000 7f8994948640 1 --2- 172.21.15.98:0/3514450908 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d6828420 0x5627d6840f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:33.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.098+0000 7f899918a740 1 -- 172.21.15.98:0/3514450908 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89901d8da0 con 0x5627d6828420 2024-05-09T08:52:33.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.098+0000 7f899918a740 1 -- 172.21.15.98:0/3514450908 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f89901b58a0 con 0x5627d6825690 2024-05-09T08:52:33.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.098+0000 7f898ffff640 1 --2- 172.21.15.98:0/3514450908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d681d430 0x5627d6827940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:33.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:32 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:52:33.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:32 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:52:33.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:32 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:52:33.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.099+0000 7f898ffff640 1 -- 172.21.15.98:0/3514450908 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d6825690 msgr2=0x5627d68409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:33.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.099+0000 7f898ffff640 1 --2- 172.21.15.98:0/3514450908 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d6825690 0x5627d68409c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.099+0000 7f898ffff640 1 -- 172.21.15.98:0/3514450908 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d6828420 msgr2=0x5627d6840f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:33.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.099+0000 7f898ffff640 1 --2- 172.21.15.98:0/3514450908 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d6828420 0x5627d6840f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.099+0000 7f898ffff640 1 -- 172.21.15.98:0/3514450908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627d6814f30 con 0x5627d681d430 2024-05-09T08:52:33.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.099+0000 7f898ffff640 1 --2- 172.21.15.98:0/3514450908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d681d430 0x5627d6827940 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f89800a0010 tx=0x7f898009a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c777e2f601ceda34 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.100+0000 7f8975ffb640 1 -- 172.21.15.98:0/3514450908 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898009b700 con 0x5627d681d430 2024-05-09T08:52:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.100+0000 7f8975ffb640 1 -- 172.21.15.98:0/3514450908 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89800d8630 con 0x5627d681d430 2024-05-09T08:52:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.100+0000 7f8975ffb640 1 -- 172.21.15.98:0/3514450908 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8980097e60 con 0x5627d681d430 2024-05-09T08:52:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.101+0000 7f899918a740 1 -- 172.21.15.98:0/3514450908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d681d430 msgr2=0x5627d6827940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.101+0000 7f899918a740 1 --2- 172.21.15.98:0/3514450908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d681d430 0x5627d6827940 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f89800a0010 tx=0x7f898009a0d0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.101+0000 7f899918a740 1 -- 172.21.15.98:0/3514450908 shutdown_connections 2024-05-09T08:52:33.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.101+0000 7f899918a740 1 --2- 172.21.15.98:0/3514450908 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d6828420 0x5627d6840f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.101+0000 7f899918a740 1 --2- 172.21.15.98:0/3514450908 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d6825690 0x5627d68409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.101+0000 7f899918a740 1 --2- 172.21.15.98:0/3514450908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d681d430 0x5627d6827940 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.101+0000 7f899918a740 1 -- 172.21.15.98:0/3514450908 >> 172.21.15.98:0/3514450908 conn(0x5627d6819750 msgr2=0x5627d6824660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:33.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.102+0000 7f899918a740 1 -- 172.21.15.98:0/3514450908 shutdown_connections 2024-05-09T08:52:33.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.102+0000 7f899918a740 1 -- 172.21.15.98:0/3514450908 wait complete. 2024-05-09T08:52:33.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.102+0000 7f899918a740 1 -- start start 2024-05-09T08:52:33.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.103+0000 7f899918a740 1 Processor -- start 2024-05-09T08:52:33.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.104+0000 7f899918a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681d430 0x5627d6830b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.104+0000 7f899918a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6825690 0x5627d6829aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.104+0000 7f899918a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d6828420 0x5627d682a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.104+0000 7f899918a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89901b6640 con 0x5627d6825690 2024-05-09T08:52:33.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.104+0000 7f899918a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89901b5f70 con 0x5627d6828420 2024-05-09T08:52:33.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.104+0000 7f899918a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f89901b31b0 con 0x5627d681d430 2024-05-09T08:52:33.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.104+0000 7f899918a740 10 client.0 _ll_register_callbacks cb 0x7f898009f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:33.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.105+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6825690 0x5627d6829aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:33.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.106+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6825690 0x5627d6829aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47468/0 (socket says 172.21.15.98:47468) 2024-05-09T08:52:33.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.106+0000 7f898f7fe640 1 -- 172.21.15.98:0/2212720860 learned_addr learned my addr 172.21.15.98:0/2212720860 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:33.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.107+0000 7f898f7fe640 1 -- 172.21.15.98:0/2212720860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681d430 msgr2=0x5627d6830b20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:52:33.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.107+0000 7f898f7fe640 1 --2- 172.21.15.98:0/2212720860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681d430 0x5627d6830b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.107+0000 7f898f7fe640 1 -- 172.21.15.98:0/2212720860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d6828420 msgr2=0x5627d682a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:33.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.107+0000 7f898f7fe640 1 --2- 172.21.15.98:0/2212720860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d6828420 0x5627d682a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.107+0000 7f898f7fe640 1 -- 172.21.15.98:0/2212720860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627d6822520 con 0x5627d6825690 2024-05-09T08:52:33.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.109+0000 7f898f7fe640 1 --2- 172.21.15.98:0/2212720860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6825690 0x5627d6829aa0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8988005e30 tx=0x7f8988003a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc09ca809dbaac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:33.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.110+0000 7f896f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:52:33.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.111+0000 7f896f7fe640 1 -- 172.21.15.98:0/2212720860 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898804d020 con 0x5627d6825690 2024-05-09T08:52:33.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.112+0000 7f899918a740 1 -- 172.21.15.98:0/2212720860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89800d8630 con 0x5627d6825690 2024-05-09T08:52:33.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.112+0000 7f896f7fe640 1 -- 172.21.15.98:0/2212720860 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f898802f3c0 con 0x5627d6825690 2024-05-09T08:52:33.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.113+0000 7f896f7fe640 1 -- 172.21.15.98:0/2212720860 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8988058050 con 0x5627d6825690 2024-05-09T08:52:33.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.113+0000 7f899918a740 10 client.16332 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:33.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.113+0000 7f899918a740 1 -- 172.21.15.98:0/2212720860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5627d6828ca0 con 0x5627d6825690 2024-05-09T08:52:33.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.115+0000 7f896f7fe640 1 -- 172.21.15.98:0/2212720860 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89800d8630 con 0x5627d6825690 2024-05-09T08:52:33.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.115+0000 7f896f7fe640 1 -- 172.21.15.98:0/2212720860 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8988004130 con 0x5627d6825690 2024-05-09T08:52:33.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.115+0000 7f899918a740 10 client.16332 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.115+0000 7f899918a740 10 client.16332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.115+0000 7f899918a740 20 client.16332 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.115+0000 7f899918a740 1 --2- 172.21.15.98:0/2212720860 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5627d689f230 0x5627d68a1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.115+0000 7f899918a740 4 client.16332 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:52:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.115+0000 7f899918a740 1 -- 172.21.15.98:0/2212720860 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5627d67076f0 con 0x5627d689f230 2024-05-09T08:52:33.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.120+0000 7f898ffff640 1 --2- 172.21.15.98:0/2212720860 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5627d689f230 0x5627d68a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:33.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.132+0000 7f898ffff640 1 --2- 172.21.15.98:0/2212720860 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5627d689f230 0x5627d68a1620 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4672dae6a33888fd server_cookie=69bd65cdfbd98c41 in_seq=0 out_seq=0 2024-05-09T08:52:33.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.132+0000 7f896f7fe640 10 client.16332 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:33.135 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:32 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:52:33.135 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:32 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:52:33.135 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:32 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:52:33.136 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7f157da76700 time 2024-05-09T08:52:33.087700+0000 2024-05-09T08:52:33.136 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-05-09T08:52:33.136 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:52:33.137 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f15900af3cd] 2024-05-09T08:52:33.137 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7f15900af593] 2024-05-09T08:52:33.137 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 3: (MDLog::_replay_thread()+0xd8) [0x563cd13b30d8] 2024-05-09T08:52:33.137 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 4: (MDLog::ReplayThread::entry()+0x11) [0x563cd13bcd01] 2024-05-09T08:52:33.137 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 5: /lib64/libpthread.so.0(+0x81ca) [0x7f158ee2e1ca] 2024-05-09T08:52:33.137 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 6: clone() 2024-05-09T08:52:33.137 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2024-05-09T08:52:33.088+0000 7f157da76700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7f157da76700 time 2024-05-09T08:52:33.087700+0000 2024-05-09T08:52:33.137 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-05-09T08:52:33.137 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2024-05-09T08:52:33.137 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:52:33.137 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f15900af3cd] 2024-05-09T08:52:33.138 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7f15900af593] 2024-05-09T08:52:33.138 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 3: (MDLog::_replay_thread()+0xd8) [0x563cd13b30d8] 2024-05-09T08:52:33.138 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 4: (MDLog::ReplayThread::entry()+0x11) [0x563cd13bcd01] 2024-05-09T08:52:33.138 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 5: /lib64/libpthread.so.0(+0x81ca) [0x7f158ee2e1ca] 2024-05-09T08:52:33.138 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 6: clone() 2024-05-09T08:52:33.138 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2024-05-09T08:52:33.138 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: *** Caught signal (Aborted) ** 2024-05-09T08:52:33.138 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: in thread 7f157da76700 thread_name:md_log_replay 2024-05-09T08:52:33.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.137+0000 7f896f7fe640 1 -- 172.21.15.98:0/2212720860 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x5627d67076f0 con 0x5627d689f230 2024-05-09T08:52:33.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.137+0000 7f896f7fe640 10 client.16332 handle_command_reply: tid=0 2024-05-09T08:52:33.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.137+0000 7f899918a740 1 client.16332 shutdown 2024-05-09T08:52:33.139 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:52:33.140 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f158ee38d20] 2024-05-09T08:52:33.140 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2: gsignal() 2024-05-09T08:52:33.140 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 3: abort() 2024-05-09T08:52:33.140 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f15900af427] 2024-05-09T08:52:33.140 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7f15900af593] 2024-05-09T08:52:33.140 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 6: (MDLog::_replay_thread()+0xd8) [0x563cd13b30d8] 2024-05-09T08:52:33.140 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 7: (MDLog::ReplayThread::entry()+0x11) [0x563cd13bcd01] 2024-05-09T08:52:33.140 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 8: /lib64/libpthread.so.0(+0x81ca) [0x7f158ee2e1ca] 2024-05-09T08:52:33.140 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 9: clone() 2024-05-09T08:52:33.140 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2024-05-09T08:52:33.089+0000 7f157da76700 -1 *** Caught signal (Aborted) ** 2024-05-09T08:52:33.140 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: in thread 7f157da76700 thread_name:md_log_replay 2024-05-09T08:52:33.140 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2024-05-09T08:52:33.140 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f158ee38d20] 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2: gsignal() 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 3: abort() 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f15900af427] 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7f15900af593] 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 6: (MDLog::_replay_thread()+0xd8) [0x563cd13b30d8] 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 7: (MDLog::ReplayThread::entry()+0x11) [0x563cd13bcd01] 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 8: /lib64/libpthread.so.0(+0x81ca) [0x7f158ee2e1ca] 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 9: clone() 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: -1> 2024-05-09T08:52:33.088+0000 7f157da76700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7f157da76700 time 2024-05-09T08:52:33.087700+0000 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2024-05-09T08:52:33.141 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:52:33.142 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f15900af3cd] 2024-05-09T08:52:33.142 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7f15900af593] 2024-05-09T08:52:33.142 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 3: (MDLog::_replay_thread()+0xd8) [0x563cd13b30d8] 2024-05-09T08:52:33.142 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 4: (MDLog::ReplayThread::entry()+0x11) [0x563cd13bcd01] 2024-05-09T08:52:33.142 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 5: /lib64/libpthread.so.0(+0x81ca) [0x7f158ee2e1ca] 2024-05-09T08:52:33.142 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 6: clone() 2024-05-09T08:52:33.142 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2024-05-09T08:52:33.142 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 0> 2024-05-09T08:52:33.089+0000 7f157da76700 -1 *** Caught signal (Aborted) ** 2024-05-09T08:52:33.142 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: in thread 7f157da76700 thread_name:md_log_replay 2024-05-09T08:52:33.142 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2024-05-09T08:52:33.142 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:52:33.142 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f158ee38d20] 2024-05-09T08:52:33.142 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2: gsignal() 2024-05-09T08:52:33.142 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 3: abort() 2024-05-09T08:52:33.143 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f15900af427] 2024-05-09T08:52:33.143 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7f15900af593] 2024-05-09T08:52:33.143 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 6: (MDLog::_replay_thread()+0xd8) [0x563cd13b30d8] 2024-05-09T08:52:33.143 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 7: (MDLog::ReplayThread::entry()+0x11) [0x563cd13bcd01] 2024-05-09T08:52:33.143 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 8: /lib64/libpthread.so.0(+0x81ca) [0x7f158ee2e1ca] 2024-05-09T08:52:33.143 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 9: clone() 2024-05-09T08:52:33.143 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T08:52:33.143 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2024-05-09T08:52:33.143 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: -1> 2024-05-09T08:52:33.088+0000 7f157da76700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7f157da76700 time 2024-05-09T08:52:33.087700+0000 2024-05-09T08:52:33.143 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-05-09T08:52:33.143 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2024-05-09T08:52:33.143 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:52:33.144 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f15900af3cd] 2024-05-09T08:52:33.144 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7f15900af593] 2024-05-09T08:52:33.144 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 3: (MDLog::_replay_thread()+0xd8) [0x563cd13b30d8] 2024-05-09T08:52:33.144 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 4: (MDLog::ReplayThread::entry()+0x11) [0x563cd13bcd01] 2024-05-09T08:52:33.144 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 5: /lib64/libpthread.so.0(+0x81ca) [0x7f158ee2e1ca] 2024-05-09T08:52:33.144 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 6: clone() 2024-05-09T08:52:33.144 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2024-05-09T08:52:33.144 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 0> 2024-05-09T08:52:33.089+0000 7f157da76700 -1 *** Caught signal (Aborted) ** 2024-05-09T08:52:33.144 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: in thread 7f157da76700 thread_name:md_log_replay 2024-05-09T08:52:33.145 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2024-05-09T08:52:33.145 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:52:33.145 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f158ee38d20] 2024-05-09T08:52:33.145 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2: gsignal() 2024-05-09T08:52:33.145 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 3: abort() 2024-05-09T08:52:33.145 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f15900af427] 2024-05-09T08:52:33.145 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7f15900af593] 2024-05-09T08:52:33.145 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 6: (MDLog::_replay_thread()+0xd8) [0x563cd13b30d8] 2024-05-09T08:52:33.145 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 7: (MDLog::ReplayThread::entry()+0x11) [0x563cd13bcd01] 2024-05-09T08:52:33.145 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 8: /lib64/libpthread.so.0(+0x81ca) [0x7f158ee2e1ca] 2024-05-09T08:52:33.145 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 9: clone() 2024-05-09T08:52:33.145 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T08:52:33.145 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[64217]: 2024-05-09T08:52:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.144+0000 7f899918a740 1 -- 172.21.15.98:0/2212720860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6825690 msgr2=0x5627d6829aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.144+0000 7f899918a740 1 --2- 172.21.15.98:0/2212720860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6825690 0x5627d6829aa0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8988005e30 tx=0x7f8988003a60 comp rx=0 tx=0).stop 2024-05-09T08:52:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.144+0000 7f899918a740 1 -- 172.21.15.98:0/2212720860 shutdown_connections 2024-05-09T08:52:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.144+0000 7f899918a740 1 --2- 172.21.15.98:0/2212720860 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5627d689f230 0x5627d68a1620 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.144+0000 7f899918a740 1 --2- 172.21.15.98:0/2212720860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5627d6828420 0x5627d682a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.144+0000 7f899918a740 1 --2- 172.21.15.98:0/2212720860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627d6825690 0x5627d6829aa0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.144+0000 7f899918a740 1 --2- 172.21.15.98:0/2212720860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627d681d430 0x5627d6830b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.144+0000 7f899918a740 1 -- 172.21.15.98:0/2212720860 >> 172.21.15.98:0/2212720860 conn(0x5627d6819750 msgr2=0x5627d6823950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:33.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.144+0000 7f896f7fe640 0 client.16332 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:33.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.144+0000 7f899918a740 1 -- 172.21.15.98:0/2212720860 shutdown_connections 2024-05-09T08:52:33.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.145+0000 7f899918a740 1 -- 172.21.15.98:0/2212720860 wait complete. 2024-05-09T08:52:33.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.145+0000 7f899918a740 20 client.16332 trim_cache size 0 max 16384 2024-05-09T08:52:33.149 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:52:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "scrub active (1 inodes in the stack)", 2024-05-09T08:52:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:52:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "b5a48e69-e2e2-4416-97f5-fb3a51b01ffa": { 2024-05-09T08:52:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:52:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "b5a48e69-e2e2-4416-97f5-fb3a51b01ffa", 2024-05-09T08:52:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:52:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:52:33.150 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:52:33.150 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:52:33.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:32 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:52:33.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:32 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:52:33.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:32 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:52:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.162+0000 7f8995149640 1 -- 172.21.15.98:0/2636692528 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89780a2ed0 msgr2=0x7f89780a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.162+0000 7f8995149640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89780a2ed0 0x7f89780a5390 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f898400bc20 tx=0x7f8984005e90 comp rx=0 tx=0).stop 2024-05-09T08:52:33.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.162+0000 7f8995149640 1 -- 172.21.15.98:0/2636692528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990108830 msgr2=0x7f899011cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:33.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.163+0000 7f8995149640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990108830 0x7f899011cb90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f898000e9f0 tx=0x7f898000eeb0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.165+0000 7f8995149640 1 -- 172.21.15.98:0/2636692528 shutdown_connections 2024-05-09T08:52:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.165+0000 7f8995149640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89780a2ed0 0x7f89780a5390 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.165+0000 7f8995149640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f899010fed0 0x7f8990115dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.165+0000 7f8995149640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89901092b0 0x7f899011d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.165+0000 7f8995149640 1 --2- 172.21.15.98:0/2636692528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990108830 0x7f899011cb90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.165+0000 7f8995149640 1 -- 172.21.15.98:0/2636692528 >> 172.21.15.98:0/2636692528 conn(0x7f899006c7b0 msgr2=0x7f89900703f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:33.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.167+0000 7f8995149640 1 -- 172.21.15.98:0/2636692528 shutdown_connections 2024-05-09T08:52:33.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:52:33.168+0000 7f8995149640 1 -- 172.21.15.98:0/2636692528 wait complete. 2024-05-09T08:52:33.235 INFO:tasks.cephfs.filesystem:scrub status for tag:b5a48e69-e2e2-4416-97f5-fb3a51b01ffa - {'path': '/', 'tag': 'b5a48e69-e2e2-4416-97f5-fb3a51b01ffa', 'options': 'recursive,force'} 2024-05-09T08:52:33.487 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:33 smithi105 podman[67998]: 2024-05-09 08:52:33.192466766 +0000 UTC m=+0.033842458 container died ce228e5ed3306a98be0e32cc00de77728e586e2cbea1c77e4c15d6c8df7e56b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=) 2024-05-09T08:52:34.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:33 smithi142 ceph-mon[34625]: pgmap v60: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 752 MiB/s wr, 1.41k op/s 2024-05-09T08:52:34.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:33 smithi105 ceph-mon[34498]: pgmap v60: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 752 MiB/s wr, 1.41k op/s 2024-05-09T08:52:34.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:33 smithi098 ceph-mon[30092]: pgmap v60: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 752 MiB/s wr, 1.41k op/s 2024-05-09T08:52:35.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:35 smithi105 ceph-mon[34498]: pgmap v61: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 885 MiB/s wr, 1.66k op/s 2024-05-09T08:52:36.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:35 smithi142 ceph-mon[34625]: pgmap v61: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 885 MiB/s wr, 1.66k op/s 2024-05-09T08:52:36.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:35 smithi098 ceph-mon[30092]: pgmap v61: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 885 MiB/s wr, 1.66k op/s 2024-05-09T08:52:37.203 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:36 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:52:37.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:36 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:52:37.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:36 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:52:38.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:37 smithi105 ceph-mon[34498]: pgmap v62: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 964 MiB/s wr, 1.81k op/s 2024-05-09T08:52:38.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:37 smithi098 ceph-mon[30092]: pgmap v62: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 964 MiB/s wr, 1.81k op/s 2024-05-09T08:52:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:37 smithi142 ceph-mon[34625]: pgmap v62: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 964 MiB/s wr, 1.81k op/s 2024-05-09T08:52:40.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:39 smithi105 ceph-mon[34498]: pgmap v63: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 723 MiB/s wr, 1.38k op/s 2024-05-09T08:52:40.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:39 smithi098 ceph-mon[30092]: pgmap v63: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 723 MiB/s wr, 1.38k op/s 2024-05-09T08:52:40.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:39 smithi142 ceph-mon[34625]: pgmap v63: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 723 MiB/s wr, 1.38k op/s 2024-05-09T08:52:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.041+0000 7f78f5438640 1 -- 172.21.15.98:0/3897731281 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78f0076680 msgr2=0x7f78f01121b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.041+0000 7f78f5438640 1 --2- 172.21.15.98:0/3897731281 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78f0076680 0x7f78f01121b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f78e0009930 tx=0x7f78e002a480 comp rx=0 tx=0).stop 2024-05-09T08:52:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.047+0000 7f78f5438640 1 -- 172.21.15.98:0/3897731281 shutdown_connections 2024-05-09T08:52:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.047+0000 7f78f5438640 1 --2- 172.21.15.98:0/3897731281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f01126f0 0x7f78f0114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.047+0000 7f78f5438640 1 --2- 172.21.15.98:0/3897731281 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78f0076680 0x7f78f01121b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.047+0000 7f78f5438640 1 --2- 172.21.15.98:0/3897731281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78f0075d60 0x7f78f0076140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.047+0000 7f78f5438640 1 -- 172.21.15.98:0/3897731281 >> 172.21.15.98:0/3897731281 conn(0x7f78f00ff540 msgr2=0x7f78f0101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.049+0000 7f78f5438640 1 -- 172.21.15.98:0/3897731281 shutdown_connections 2024-05-09T08:52:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.049+0000 7f78f5438640 1 -- 172.21.15.98:0/3897731281 wait complete. 2024-05-09T08:52:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.050+0000 7f78f5438640 1 Processor -- start 2024-05-09T08:52:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.053+0000 7f78f5438640 1 -- start start 2024-05-09T08:52:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.053+0000 7f78f5438640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f0075d60 0x7f78f0111c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.053+0000 7f78f5438640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78f0076680 0x7f78f01121a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.053+0000 7f78f5438640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78f01126f0 0x7f78f010bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.053+0000 7f78f5438640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78f01043b0 con 0x7f78f0075d60 2024-05-09T08:52:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.053+0000 7f78f5438640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f78f0104230 con 0x7f78f01126f0 2024-05-09T08:52:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.053+0000 7f78f5438640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f78f0104530 con 0x7f78f0076680 2024-05-09T08:52:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.054+0000 7f78ee7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78f0076680 0x7f78f01121a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.054+0000 7f78ee7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78f0076680 0x7f78f01121a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45192/0 (socket says 172.21.15.98:45192) 2024-05-09T08:52:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.054+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2357950048 learned_addr learned my addr 172.21.15.98:0/2357950048 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.054+0000 7f78ef7fe640 1 --2- 172.21.15.98:0/2357950048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78f01126f0 0x7f78f010bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.054+0000 7f78eeffd640 1 --2- 172.21.15.98:0/2357950048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f0075d60 0x7f78f0111c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.055+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2357950048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78f01126f0 msgr2=0x7f78f010bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.055+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/2357950048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78f01126f0 0x7f78f010bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.055+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2357950048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f0075d60 msgr2=0x7f78f0111c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.055+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/2357950048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f0075d60 0x7f78f0111c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.055+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2357950048 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f010c810 con 0x7f78f0076680 2024-05-09T08:52:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.055+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/2357950048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78f0076680 0x7f78f01121a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f78e0009850 tx=0x7f78e0032710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.056+0000 7f78d7fff640 1 -- 172.21.15.98:0/2357950048 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e003f070 con 0x7f78f0076680 2024-05-09T08:52:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.056+0000 7f78d7fff640 1 -- 172.21.15.98:0/2357950048 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e0032e10 con 0x7f78f0076680 2024-05-09T08:52:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.056+0000 7f78d7fff640 1 -- 172.21.15.98:0/2357950048 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0035030 con 0x7f78f0076680 2024-05-09T08:52:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.056+0000 7f78f5438640 1 -- 172.21.15.98:0/2357950048 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f010caa0 con 0x7f78f0076680 2024-05-09T08:52:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.056+0000 7f78f5438640 1 -- 172.21.15.98:0/2357950048 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f01b37f0 con 0x7f78f0076680 2024-05-09T08:52:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.061+0000 7f78d7fff640 1 -- 172.21.15.98:0/2357950048 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f78e0042070 con 0x7f78f0076680 2024-05-09T08:52:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.062+0000 7f78d7fff640 1 --2- 172.21.15.98:0/2357950048 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f78d80a2f50 0x7f78d80a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.062+0000 7f78d7fff640 1 -- 172.21.15.98:0/2357950048 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f78e00f0190 con 0x7f78f0076680 2024-05-09T08:52:41.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.064+0000 7f78eeffd640 1 --2- 172.21.15.98:0/2357950048 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f78d80a2f50 0x7f78d80a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.065+0000 7f78f8226740 1 Processor -- start 2024-05-09T08:52:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.066+0000 7f78f8226740 1 -- start start 2024-05-09T08:52:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.066+0000 7f78f8226740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6450 0x5603cdcf6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.066+0000 7f78f8226740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdcf6d70 0x5603cdd010f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.066+0000 7f78f8226740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdd01630 0x5603cdd03a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.066+0000 7f78f8226740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5603cdc5b340 con 0x5603cdcf6450 2024-05-09T08:52:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.066+0000 7f78f8226740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5603cdc03290 con 0x5603cdd01630 2024-05-09T08:52:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.066+0000 7f78f8226740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5603cdc57300 con 0x5603cdcf6d70 2024-05-09T08:52:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.067+0000 7f78ee7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdcf6d70 0x5603cdd010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.067+0000 7f78ee7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdcf6d70 0x5603cdd010f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45202/0 (socket says 172.21.15.98:45202) 2024-05-09T08:52:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.067+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2599115325 learned_addr learned my addr 172.21.15.98:0/2599115325 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.067+0000 7f78ef7fe640 1 --2- 172.21.15.98:0/2599115325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdd01630 0x5603cdd03a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.067+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2599115325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdd01630 msgr2=0x5603cdd03a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.067+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/2599115325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdd01630 0x5603cdd03a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.067+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2599115325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6450 msgr2=0x5603cdcf6830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:52:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.067+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/2599115325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6450 0x5603cdcf6830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.067+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2599115325 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603cdc278e0 con 0x5603cdcf6d70 2024-05-09T08:52:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.067+0000 7f78ef7fe640 1 --2- 172.21.15.98:0/2599115325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdd01630 0x5603cdd03a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:52:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.068+0000 7f78eeffd640 1 --2- 172.21.15.98:0/2357950048 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f78d80a2f50 0x7f78d80a5410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009890 tx=0x7f78e4008040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.068+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/2599115325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdcf6d70 0x5603cdd010f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f78e00bbff0 tx=0x7f78e00bbd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17940ddb13e36c24 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.068+0000 7f78d57fa640 1 -- 172.21.15.98:0/2599115325 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0049030 con 0x5603cdcf6d70 2024-05-09T08:52:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.068+0000 7f78d57fa640 1 -- 172.21.15.98:0/2599115325 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e00c06a0 con 0x5603cdcf6d70 2024-05-09T08:52:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.068+0000 7f78d57fa640 1 -- 172.21.15.98:0/2599115325 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e00b6e60 con 0x5603cdcf6d70 2024-05-09T08:52:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.069+0000 7f78f8226740 1 -- 172.21.15.98:0/2599115325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdcf6d70 msgr2=0x5603cdd010f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.069+0000 7f78f8226740 1 --2- 172.21.15.98:0/2599115325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdcf6d70 0x5603cdd010f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f78e00bbff0 tx=0x7f78e00bbd10 comp rx=0 tx=0).stop 2024-05-09T08:52:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.070+0000 7f78f8226740 1 -- 172.21.15.98:0/2599115325 shutdown_connections 2024-05-09T08:52:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.070+0000 7f78f8226740 1 --2- 172.21.15.98:0/2599115325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdd01630 0x5603cdd03a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.070+0000 7f78f8226740 1 --2- 172.21.15.98:0/2599115325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdcf6d70 0x5603cdd010f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.070+0000 7f78f8226740 1 --2- 172.21.15.98:0/2599115325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6450 0x5603cdcf6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.070+0000 7f78f8226740 1 -- 172.21.15.98:0/2599115325 >> 172.21.15.98:0/2599115325 conn(0x5603cdcf2960 msgr2=0x5603cdcf2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.070+0000 7f78f8226740 1 -- 172.21.15.98:0/2599115325 shutdown_connections 2024-05-09T08:52:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.070+0000 7f78f8226740 1 -- 172.21.15.98:0/2599115325 wait complete. 2024-05-09T08:52:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.070+0000 7f78f8226740 1 -- start start 2024-05-09T08:52:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.070+0000 7f78f8226740 1 Processor -- start 2024-05-09T08:52:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.071+0000 7f78f8226740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6450 0x5603cdcfe550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.071+0000 7f78f8226740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdcf6d70 0x5603cdcfea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.071+0000 7f78f8226740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdd01630 0x5603cdcfefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.071+0000 7f78f8226740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5603cdd15030 con 0x5603cdd01630 2024-05-09T08:52:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.071+0000 7f78f8226740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5603cdc16790 con 0x5603cdcf6450 2024-05-09T08:52:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.071+0000 7f78f8226740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5603cdd151b0 con 0x5603cdcf6d70 2024-05-09T08:52:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.071+0000 7f78f8226740 10 client.0 _ll_register_callbacks cb 0x5603cdc51a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.071+0000 7f78ee7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdcf6d70 0x5603cdcfea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.071+0000 7f78ee7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdcf6d70 0x5603cdcfea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45214/0 (socket says 172.21.15.98:45214) 2024-05-09T08:52:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.071+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2055444731 learned_addr learned my addr 172.21.15.98:0/2055444731 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.072+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2055444731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6450 msgr2=0x5603cdcfe550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.072+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/2055444731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6450 0x5603cdcfe550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.072+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2055444731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdd01630 msgr2=0x5603cdcfefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.072+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/2055444731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdd01630 0x5603cdcfefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.072+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2055444731 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603cd96f210 con 0x5603cdcf6d70 2024-05-09T08:52:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.073+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/2055444731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdcf6d70 0x5603cdcfea90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f78e00b8750 tx=0x7f78e002a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59c664bef8e83ee5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.073+0000 7f78d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:52:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.073+0000 7f78d2ff5640 1 -- 172.21.15.98:0/2055444731 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e00da060 con 0x5603cdcf6d70 2024-05-09T08:52:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.073+0000 7f78d2ff5640 1 -- 172.21.15.98:0/2055444731 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f78e00d5260 con 0x5603cdcf6d70 2024-05-09T08:52:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.073+0000 7f78f8226740 1 -- 172.21.15.98:0/2055444731 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78e00c06a0 con 0x5603cdcf6d70 2024-05-09T08:52:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.073+0000 7f78d2ff5640 1 -- 172.21.15.98:0/2055444731 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f78e0049070 con 0x5603cdcf6d70 2024-05-09T08:52:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.073+0000 7f78f8226740 10 client.25657 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.073+0000 7f78f8226740 1 -- 172.21.15.98:0/2055444731 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5603cda3b620 con 0x5603cdcf6d70 2024-05-09T08:52:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.074+0000 7f78d2ff5640 1 -- 172.21.15.98:0/2055444731 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e00c06a0 con 0x5603cdcf6d70 2024-05-09T08:52:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.075+0000 7f78d2ff5640 1 -- 172.21.15.98:0/2055444731 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f78e00c4ad0 con 0x5603cdcf6d70 2024-05-09T08:52:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.075+0000 7f78f8226740 10 client.25657 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.075+0000 7f78f8226740 10 client.25657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.075+0000 7f78f8226740 20 client.25657 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.075+0000 7f78f8226740 1 --2- 172.21.15.98:0/2055444731 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5603cdd21f20 0x5603cdd24310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.075+0000 7f78f8226740 4 client.25657 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:52:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.075+0000 7f78f8226740 1 -- 172.21.15.98:0/2055444731 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603cdc68a70 con 0x5603cdd21f20 2024-05-09T08:52:41.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.076+0000 7f78eeffd640 1 --2- 172.21.15.98:0/2055444731 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5603cdd21f20 0x5603cdd24310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.080+0000 7f78eeffd640 1 --2- 172.21.15.98:0/2055444731 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5603cdd21f20 0x5603cdd24310 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a933d2d86de276d5 server_cookie=e9e4aba39ff209d0 in_seq=0 out_seq=0 2024-05-09T08:52:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.080+0000 7f78d2ff5640 10 client.25657 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.085+0000 7f78d2ff5640 1 -- 172.21.15.98:0/2055444731 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5603cdc68a70 con 0x5603cdd21f20 2024-05-09T08:52:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.085+0000 7f78d2ff5640 10 client.25657 handle_command_reply: tid=0 2024-05-09T08:52:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.085+0000 7f78f8226740 1 client.25657 shutdown 2024-05-09T08:52:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.085+0000 7f78f8226740 1 -- 172.21.15.98:0/2055444731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdcf6d70 msgr2=0x5603cdcfea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.085+0000 7f78f8226740 1 --2- 172.21.15.98:0/2055444731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdcf6d70 0x5603cdcfea90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f78e00b8750 tx=0x7f78e002a480 comp rx=0 tx=0).stop 2024-05-09T08:52:41.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.090+0000 7f78f8226740 1 -- 172.21.15.98:0/2055444731 shutdown_connections 2024-05-09T08:52:41.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.090+0000 7f78f8226740 1 --2- 172.21.15.98:0/2055444731 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5603cdd21f20 0x5603cdd24310 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.090+0000 7f78f8226740 1 --2- 172.21.15.98:0/2055444731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdd01630 0x5603cdcfefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.090+0000 7f78f8226740 1 --2- 172.21.15.98:0/2055444731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdcf6d70 0x5603cdcfea90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.090+0000 7f78f8226740 1 --2- 172.21.15.98:0/2055444731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6450 0x5603cdcfe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.090+0000 7f78f8226740 1 -- 172.21.15.98:0/2055444731 >> 172.21.15.98:0/2055444731 conn(0x5603cdcf2960 msgr2=0x5603cdcf86e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:41.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.091+0000 7f78d2ff5640 0 client.25657 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.091+0000 7f78f8226740 1 -- 172.21.15.98:0/2055444731 shutdown_connections 2024-05-09T08:52:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.091+0000 7f78f8226740 1 -- 172.21.15.98:0/2055444731 wait complete. 2024-05-09T08:52:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.091+0000 7f78f8226740 20 client.25657 trim_cache size 0 max 16384 2024-05-09T08:52:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.129+0000 7f78f8226740 1 Processor -- start 2024-05-09T08:52:41.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.141+0000 7f78f8226740 1 -- start start 2024-05-09T08:52:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.141+0000 7f78f8226740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6450 0x5603cdd1dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.142+0000 7f78f8226740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6d70 0x5603cdcf1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.142+0000 7f78f8226740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdd01630 0x5603cdcf8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.142+0000 7f78f8226740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc00ff20 con 0x5603cdcf6d70 2024-05-09T08:52:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.142+0000 7f78f8226740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc011a60 con 0x5603cdcf6450 2024-05-09T08:52:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.142+0000 7f78f8226740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc0105f0 con 0x5603cdd01630 2024-05-09T08:52:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.143+0000 7f78ee7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6d70 0x5603cdcf1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.143+0000 7f78ee7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6d70 0x5603cdcf1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40874/0 (socket says 172.21.15.98:40874) 2024-05-09T08:52:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.143+0000 7f78ee7fc640 1 -- 172.21.15.98:0/3672471407 learned_addr learned my addr 172.21.15.98:0/3672471407 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.143+0000 7f78ef7fe640 1 --2- 172.21.15.98:0/3672471407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdd01630 0x5603cdcf8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.143+0000 7f78eeffd640 1 --2- 172.21.15.98:0/3672471407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6450 0x5603cdd1dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.144+0000 7f78ee7fc640 1 -- 172.21.15.98:0/3672471407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdd01630 msgr2=0x5603cdcf8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.144+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/3672471407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdd01630 0x5603cdcf8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.144+0000 7f78ee7fc640 1 -- 172.21.15.98:0/3672471407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6450 msgr2=0x5603cdd1dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.144+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/3672471407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6450 0x5603cdd1dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.144+0000 7f78ee7fc640 1 -- 172.21.15.98:0/3672471407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603cdcf9350 con 0x5603cdcf6d70 2024-05-09T08:52:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.144+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/3672471407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6d70 0x5603cdcf1170 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f78e00b8cc0 tx=0x7f78e00bbd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f9458c16a770c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.147+0000 7f78d4ff9640 1 -- 172.21.15.98:0/3672471407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e00da070 con 0x5603cdcf6d70 2024-05-09T08:52:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.147+0000 7f78d4ff9640 1 -- 172.21.15.98:0/3672471407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e00ba390 con 0x5603cdcf6d70 2024-05-09T08:52:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.147+0000 7f78d4ff9640 1 -- 172.21.15.98:0/3672471407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e00b6e60 con 0x5603cdcf6d70 2024-05-09T08:52:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.149+0000 7f78f8226740 1 -- 172.21.15.98:0/3672471407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6d70 msgr2=0x5603cdcf1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.149+0000 7f78f8226740 1 --2- 172.21.15.98:0/3672471407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6d70 0x5603cdcf1170 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f78e00b8cc0 tx=0x7f78e00bbd10 comp rx=0 tx=0).stop 2024-05-09T08:52:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.152+0000 7f78f8226740 1 -- 172.21.15.98:0/3672471407 shutdown_connections 2024-05-09T08:52:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.152+0000 7f78f8226740 1 --2- 172.21.15.98:0/3672471407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdd01630 0x5603cdcf8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.152+0000 7f78f8226740 1 --2- 172.21.15.98:0/3672471407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6d70 0x5603cdcf1170 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.152+0000 7f78f8226740 1 --2- 172.21.15.98:0/3672471407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6450 0x5603cdd1dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.152+0000 7f78f8226740 1 -- 172.21.15.98:0/3672471407 >> 172.21.15.98:0/3672471407 conn(0x5603cdcf2960 msgr2=0x5603cdcfd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:41.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.154+0000 7f78f8226740 1 -- 172.21.15.98:0/3672471407 shutdown_connections 2024-05-09T08:52:41.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.154+0000 7f78f8226740 1 -- 172.21.15.98:0/3672471407 wait complete. 2024-05-09T08:52:41.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.155+0000 7f78f8226740 1 -- start start 2024-05-09T08:52:41.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.155+0000 7f78f8226740 1 Processor -- start 2024-05-09T08:52:41.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.155+0000 7f78f8226740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6450 0x5603cdcfb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.156+0000 7f78f8226740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6d70 0x5603cdcfbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.156+0000 7f78f8226740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdd01630 0x5603cdd87270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.156+0000 7f78f8226740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc011390 con 0x5603cdcf6450 2024-05-09T08:52:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.156+0000 7f78f8226740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc010cc0 con 0x5603cdcf6d70 2024-05-09T08:52:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.156+0000 7f78ee7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6d70 0x5603cdcfbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.156+0000 7f78ee7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6d70 0x5603cdcfbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:40414/0 (socket says 172.21.15.98:40414) 2024-05-09T08:52:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.156+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2128292832 learned_addr learned my addr 172.21.15.98:0/2128292832 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.156+0000 7f78f8226740 1 -- 172.21.15.98:0/2128292832 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc00df00 con 0x5603cdd01630 2024-05-09T08:52:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.157+0000 7f78ef7fe640 1 --2- 172.21.15.98:0/2128292832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdd01630 0x5603cdd87270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.157+0000 7f78f8226740 10 client.0 _ll_register_callbacks cb 0x7f78e002a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:41.157 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.158+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2128292832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdd01630 msgr2=0x5603cdd87270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:41.157 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.158+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/2128292832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdd01630 0x5603cdd87270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.158+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2128292832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6450 msgr2=0x5603cdcfb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:41.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.158+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/2128292832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6450 0x5603cdcfb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:41.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.158+0000 7f78ee7fc640 1 -- 172.21.15.98:0/2128292832 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603cdcfc410 con 0x5603cdcf6d70 2024-05-09T08:52:41.159 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.160+0000 7f78ee7fc640 1 --2- 172.21.15.98:0/2128292832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6d70 0x5603cdcfbb40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f78e00c4a30 tx=0x7f78e0005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81dc9a20eca57fa4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:41.160 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.160+0000 7f78d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:52:41.160 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.160+0000 7f78d2ff5640 1 -- 172.21.15.98:0/2128292832 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e00da070 con 0x5603cdcf6d70 2024-05-09T08:52:41.160 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.160+0000 7f78f8226740 1 -- 172.21.15.98:0/2128292832 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5603cdc62dc0 con 0x5603cdcf6d70 2024-05-09T08:52:41.160 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.160+0000 7f78d2ff5640 1 -- 172.21.15.98:0/2128292832 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f78e00c4ad0 con 0x5603cdcf6d70 2024-05-09T08:52:41.161 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.161+0000 7f78d2ff5640 1 -- 172.21.15.98:0/2128292832 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f78e00b6e60 con 0x5603cdcf6d70 2024-05-09T08:52:41.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.161+0000 7f78f8226740 10 client.25712 fetch_fsmap learned FSMap version 29 2024-05-09T08:52:41.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.161+0000 7f78f8226740 1 -- 172.21.15.98:0/2128292832 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5603cdcfc410 con 0x5603cdcf6d70 2024-05-09T08:52:41.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.162+0000 7f78d2ff5640 1 -- 172.21.15.98:0/2128292832 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e00c06a0 con 0x5603cdcf6d70 2024-05-09T08:52:41.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.162+0000 7f78d2ff5640 1 -- 172.21.15.98:0/2128292832 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f78e00bb740 con 0x5603cdcf6d70 2024-05-09T08:52:41.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.162+0000 7f78f8226740 10 client.25712 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:52:41.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.162+0000 7f78f8226740 10 client.25712 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:41.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.162+0000 7f78f8226740 20 client.25712 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:41.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.162+0000 7f78f8226740 1 --2- 172.21.15.98:0/2128292832 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5603cdd88b00 0x5603cdd8af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:41.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.162+0000 7f78f8226740 4 client.25712 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:52:41.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.162+0000 7f78f8226740 1 -- 172.21.15.98:0/2128292832 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5603cdc62dc0 con 0x5603cdd88b00 2024-05-09T08:52:41.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:40 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:52:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.299+0000 7f78eeffd640 1 --2- 172.21.15.98:0/2128292832 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5603cdd88b00 0x5603cdd8af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.303+0000 7f78eeffd640 1 --2- 172.21.15.98:0/2128292832 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5603cdd88b00 0x5603cdd8af20 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea130d9b697c43b0 server_cookie=ecb52fbf23d24f26 in_seq=0 out_seq=0 2024-05-09T08:52:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:41.303+0000 7f78d2ff5640 10 client.25712 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:41.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:40 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:52:41.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:40 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:52:42.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:41 smithi105 ceph-mon[34498]: pgmap v64: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1011 MiB/s wr, 1.98k op/s 2024-05-09T08:52:42.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:41 smithi098 ceph-mon[30092]: pgmap v64: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1011 MiB/s wr, 1.98k op/s 2024-05-09T08:52:42.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:41 smithi142 ceph-mon[34625]: pgmap v64: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1011 MiB/s wr, 1.98k op/s 2024-05-09T08:52:44.323 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:43 smithi105 ceph-mon[34498]: pgmap v65: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 743 MiB/s wr, 1.51k op/s 2024-05-09T08:52:44.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:43 smithi098 ceph-mon[30092]: pgmap v65: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 743 MiB/s wr, 1.51k op/s 2024-05-09T08:52:44.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:44 smithi142 ceph-mon[34625]: pgmap v65: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 743 MiB/s wr, 1.51k op/s 2024-05-09T08:52:44.737 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:44 smithi105 podman[67998]: 2024-05-09 08:52:44.32490267 +0000 UTC m=+11.166278347 container remove ce228e5ed3306a98be0e32cc00de77728e586e2cbea1c77e4c15d6c8df7e56b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, RELEASE=squid-96805dc, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:52:44.737 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:44 smithi105 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-05-09T08:52:44.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.936+0000 7f78d2ff5640 1 -- 172.21.15.98:0/2128292832 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5603cdc62dc0 con 0x5603cdd88b00 2024-05-09T08:52:44.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.936+0000 7f78d2ff5640 10 client.25712 handle_command_reply: tid=0 2024-05-09T08:52:44.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.936+0000 7f78f8226740 1 client.25712 shutdown 2024-05-09T08:52:44.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.940+0000 7f78f8226740 1 -- 172.21.15.98:0/2128292832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6d70 msgr2=0x5603cdcfbb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:44.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.940+0000 7f78f8226740 1 --2- 172.21.15.98:0/2128292832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6d70 0x5603cdcfbb40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f78e00c4a30 tx=0x7f78e0005fd0 comp rx=0 tx=0).stop 2024-05-09T08:52:44.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.940+0000 7f78f8226740 1 -- 172.21.15.98:0/2128292832 shutdown_connections 2024-05-09T08:52:44.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.940+0000 7f78f8226740 1 --2- 172.21.15.98:0/2128292832 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5603cdd88b00 0x5603cdd8af20 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:44.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.940+0000 7f78f8226740 1 --2- 172.21.15.98:0/2128292832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603cdd01630 0x5603cdd87270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:44.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.940+0000 7f78f8226740 1 --2- 172.21.15.98:0/2128292832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603cdcf6d70 0x5603cdcfbb40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:44.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.940+0000 7f78f8226740 1 --2- 172.21.15.98:0/2128292832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603cdcf6450 0x5603cdcfb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:44.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.940+0000 7f78f8226740 1 -- 172.21.15.98:0/2128292832 >> 172.21.15.98:0/2128292832 conn(0x5603cdcf2960 msgr2=0x5603cdcfe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:44.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.940+0000 7f78d2ff5640 0 client.25712 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:44.941 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.940+0000 7f78f8226740 1 -- 172.21.15.98:0/2128292832 shutdown_connections 2024-05-09T08:52:44.941 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.941+0000 7f78f8226740 1 -- 172.21.15.98:0/2128292832 wait complete. 2024-05-09T08:52:44.941 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.941+0000 7f78f8226740 20 client.25712 trim_cache size 0 max 16384 2024-05-09T08:52:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:52:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:52:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:52:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:52:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.945+0000 7f78f5438640 1 -- 172.21.15.98:0/2357950048 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f78d80a2f50 msgr2=0x7f78d80a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.945+0000 7f78f5438640 1 --2- 172.21.15.98:0/2357950048 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f78d80a2f50 0x7f78d80a5410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009890 tx=0x7f78e4008040 comp rx=0 tx=0).stop 2024-05-09T08:52:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.946+0000 7f78f5438640 1 -- 172.21.15.98:0/2357950048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78f0076680 msgr2=0x7f78f01121a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.946+0000 7f78f5438640 1 --2- 172.21.15.98:0/2357950048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78f0076680 0x7f78f01121a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f78e0009850 tx=0x7f78e0032710 comp rx=0 tx=0).stop 2024-05-09T08:52:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.946+0000 7f78f5438640 1 -- 172.21.15.98:0/2357950048 shutdown_connections 2024-05-09T08:52:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.946+0000 7f78f5438640 1 --2- 172.21.15.98:0/2357950048 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f78d80a2f50 0x7f78d80a5410 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.946+0000 7f78f5438640 1 --2- 172.21.15.98:0/2357950048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78f01126f0 0x7f78f010bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.946+0000 7f78f5438640 1 --2- 172.21.15.98:0/2357950048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78f0076680 0x7f78f01121a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.946+0000 7f78f5438640 1 --2- 172.21.15.98:0/2357950048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f0075d60 0x7f78f0111c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.946+0000 7f78f5438640 1 -- 172.21.15.98:0/2357950048 >> 172.21.15.98:0/2357950048 conn(0x7f78f00ff540 msgr2=0x7f78f0101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:44.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.947+0000 7f78f5438640 1 -- 172.21.15.98:0/2357950048 shutdown_connections 2024-05-09T08:52:44.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:44.947+0000 7f78f5438640 1 -- 172.21.15.98:0/2357950048 wait complete. 2024-05-09T08:52:45.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:45 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:52:45.237 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:44 smithi105 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-05-09T08:52:45.237 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:44 smithi105 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.e.service: Consumed 3.322s CPU time. 2024-05-09T08:52:45.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:45 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:52:45.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:45 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:52:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:46 smithi098 ceph-mon[30092]: pgmap v66: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 784 MiB/s wr, 1.62k op/s 2024-05-09T08:52:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:46 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:52:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:46 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:52:46.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:46 smithi142 ceph-mon[34625]: pgmap v66: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 784 MiB/s wr, 1.62k op/s 2024-05-09T08:52:46.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:46 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:52:46.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:46 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:52:46.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:46 smithi105 ceph-mon[34498]: pgmap v66: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 784 MiB/s wr, 1.62k op/s 2024-05-09T08:52:46.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:46 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:52:46.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:46 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:52:47.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:47 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:52:47.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:47 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:52:47.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:47 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:52:47.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:47 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:52:47.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:47 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:52:47.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:47 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:52:47.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:47 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:52:47.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:47 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:52:47.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:47 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:52:48.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:48 smithi098 ceph-mon[30092]: pgmap v67: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 966 MiB/s wr, 2.07k op/s 2024-05-09T08:52:48.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:48 smithi098 ceph-mon[30092]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-09T08:52:48.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:48 smithi098 ceph-mon[30092]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-09T08:52:48.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:48 smithi142 ceph-mon[34625]: pgmap v67: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 966 MiB/s wr, 2.07k op/s 2024-05-09T08:52:48.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:48 smithi142 ceph-mon[34625]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-09T08:52:48.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:48 smithi142 ceph-mon[34625]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-09T08:52:48.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:48 smithi105 ceph-mon[34498]: pgmap v67: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 966 MiB/s wr, 2.07k op/s 2024-05-09T08:52:48.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:48 smithi105 ceph-mon[34498]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-09T08:52:48.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:48 smithi105 ceph-mon[34498]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-09T08:52:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:50 smithi098 ceph-mon[30092]: pgmap v68: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 733 MiB/s wr, 1.66k op/s 2024-05-09T08:52:50.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:50 smithi142 ceph-mon[34625]: pgmap v68: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 733 MiB/s wr, 1.66k op/s 2024-05-09T08:52:50.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:50 smithi105 ceph-mon[34498]: pgmap v68: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 733 MiB/s wr, 1.66k op/s 2024-05-09T08:52:52.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:52 smithi098 ceph-mon[30092]: pgmap v69: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1011 MiB/s wr, 2.23k op/s 2024-05-09T08:52:52.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:52 smithi142 ceph-mon[34625]: pgmap v69: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1011 MiB/s wr, 2.23k op/s 2024-05-09T08:52:52.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:52 smithi105 ceph-mon[34498]: pgmap v69: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1011 MiB/s wr, 2.23k op/s 2024-05-09T08:52:54.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:54 smithi098 ceph-mon[30092]: pgmap v70: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 808 MiB/s wr, 1.83k op/s 2024-05-09T08:52:54.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:54 smithi142 ceph-mon[34625]: pgmap v70: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 808 MiB/s wr, 1.83k op/s 2024-05-09T08:52:54.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:54 smithi105 ceph-mon[34498]: pgmap v70: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 808 MiB/s wr, 1.83k op/s 2024-05-09T08:52:55.462 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:54 smithi105 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 1. 2024-05-09T08:52:55.462 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:54 smithi105 systemd[1]: Stopped Ceph mds.e for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:52:55.462 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:54 smithi105 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.e.service: Consumed 3.322s CPU time. 2024-05-09T08:52:55.462 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:55 smithi105 systemd[1]: Starting Ceph mds.e for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:52:55.736 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:55 smithi105 podman[68209]: 2024-05-09 08:52:55.369785083 +0000 UTC m=+0.148775862 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:52:56.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:56 smithi098 ceph-mon[30092]: pgmap v71: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 807 MiB/s wr, 1.73k op/s 2024-05-09T08:52:56.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:56 smithi142 ceph-mon[34625]: pgmap v71: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 807 MiB/s wr, 1.73k op/s 2024-05-09T08:52:56.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:56 smithi105 ceph-mon[34498]: pgmap v71: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 807 MiB/s wr, 1.73k op/s 2024-05-09T08:52:58.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:52:58 smithi105 ceph-mon[34498]: pgmap v72: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 949 MiB/s wr, 1.92k op/s 2024-05-09T08:52:58.237 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:52:57 smithi105 podman[68209]: 2024-05-09 08:52:57.927399786 +0000 UTC m=+2.706390542 container create aade97aca6951d5ec2fe8cf6455e2b33b26f8b775875e9b68219d81f99d42ce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc) 2024-05-09T08:52:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.249+0000 7fdfdf5d5640 1 -- 172.21.15.98:0/1089143577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfd811b010 msgr2=0x7fdfd811d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.249+0000 7fdfdf5d5640 1 --2- 172.21.15.98:0/1089143577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfd811b010 0x7fdfd811d400 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009cd0 tx=0x7fdfd002a9a0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.249+0000 7fdfdf5d5640 1 -- 172.21.15.98:0/1089143577 shutdown_connections 2024-05-09T08:52:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.249+0000 7fdfdf5d5640 1 --2- 172.21.15.98:0/1089143577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfd811b010 0x7fdfd811d400 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.249+0000 7fdfdf5d5640 1 --2- 172.21.15.98:0/1089143577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfd8110400 0x7fdfd8108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.249+0000 7fdfdf5d5640 1 --2- 172.21.15.98:0/1089143577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd8110020 0x7fdfd8108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.249+0000 7fdfdf5d5640 1 -- 172.21.15.98:0/1089143577 >> 172.21.15.98:0/1089143577 conn(0x7fdfd806c900 msgr2=0x7fdfd806cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:58.259 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.259+0000 7fdfdf5d5640 1 -- 172.21.15.98:0/1089143577 shutdown_connections 2024-05-09T08:52:58.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.260+0000 7fdfdf5d5640 1 -- 172.21.15.98:0/1089143577 wait complete. 2024-05-09T08:52:58.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.262+0000 7fdfdf5d5640 1 Processor -- start 2024-05-09T08:52:58.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.262+0000 7fdfdf5d5640 1 -- start start 2024-05-09T08:52:58.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.262+0000 7fdfdf5d5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfd8110020 0x7fdfd811a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.262+0000 7fdfdf5d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfd811b010 0x7fdfd811aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.262+0000 7fdfdf5d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd8113710 0x7fdfd8113bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.262+0000 7fdfdf5d5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd8120a90 con 0x7fdfd8113710 2024-05-09T08:52:58.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.262+0000 7fdfdf5d5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd8120910 con 0x7fdfd811b010 2024-05-09T08:52:58.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.262+0000 7fdfdf5d5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd8120c10 con 0x7fdfd8110020 2024-05-09T08:52:58.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.263+0000 7fdfdcb70640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfd811b010 0x7fdfd811aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.263+0000 7fdfdcb70640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfd811b010 0x7fdfd811aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:42500/0 (socket says 172.21.15.98:42500) 2024-05-09T08:52:58.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.263+0000 7fdfdcb70640 1 -- 172.21.15.98:0/3945488973 learned_addr learned my addr 172.21.15.98:0/3945488973 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:58.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.263+0000 7fdfdd371640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfd8110020 0x7fdfd811a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.264+0000 7fdfddb72640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd8113710 0x7fdfd8113bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.264+0000 7fdfdcb70640 1 -- 172.21.15.98:0/3945488973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfd8110020 msgr2=0x7fdfd811a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:58.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.264+0000 7fdfdcb70640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfd8110020 0x7fdfd811a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.264+0000 7fdfdcb70640 1 -- 172.21.15.98:0/3945488973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd8113710 msgr2=0x7fdfd8113bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:58.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.264+0000 7fdfdcb70640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd8113710 0x7fdfd8113bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.264+0000 7fdfdcb70640 1 -- 172.21.15.98:0/3945488973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd81180a0 con 0x7fdfd811b010 2024-05-09T08:52:58.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.264+0000 7fdfdd371640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfd8110020 0x7fdfd811a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:52:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.264+0000 7fdfdcb70640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfd811b010 0x7fdfd811aac0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800b470 tx=0x7fdfc800b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.265+0000 7fdfce7fc640 1 -- 172.21.15.98:0/3945488973 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8015020 con 0x7fdfd811b010 2024-05-09T08:52:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.265+0000 7fdfdf5d5640 1 -- 172.21.15.98:0/3945488973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd8118390 con 0x7fdfd811b010 2024-05-09T08:52:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.266+0000 7fdfdf5d5640 1 -- 172.21.15.98:0/3945488973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd81188e0 con 0x7fdfd811b010 2024-05-09T08:52:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.266+0000 7fdfce7fc640 1 -- 172.21.15.98:0/3945488973 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc8011d70 con 0x7fdfd811b010 2024-05-09T08:52:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.266+0000 7fdfce7fc640 1 -- 172.21.15.98:0/3945488973 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc800f340 con 0x7fdfd811b010 2024-05-09T08:52:58.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.275+0000 7fdfce7fc640 1 -- 172.21.15.98:0/3945488973 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fdfc8018020 con 0x7fdfd811b010 2024-05-09T08:52:58.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.276+0000 7fdfce7fc640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdfc40a2fd0 0x7fdfc40a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.276+0000 7fdfce7fc640 1 -- 172.21.15.98:0/3945488973 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fdfc80d0270 con 0x7fdfd811b010 2024-05-09T08:52:58.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.280+0000 7fdfe23c2740 1 Processor -- start 2024-05-09T08:52:58.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.280+0000 7fdfe23c2740 1 -- start start 2024-05-09T08:52:58.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.280+0000 7fdfe23c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74f3450 0x558be74f3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.280+0000 7fdfe23c2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3d70 0x558be74fe0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.280+0000 7fdfe23c2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74fe630 0x558be7500a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.280+0000 7fdfe23c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558be7458340 con 0x558be74f3450 2024-05-09T08:52:58.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.280+0000 7fdfe23c2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558be7400290 con 0x558be74f3d70 2024-05-09T08:52:58.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.280+0000 7fdfe23c2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558be7454300 con 0x558be74fe630 2024-05-09T08:52:58.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.283+0000 7fdfdd371640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdfc40a2fd0 0x7fdfc40a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.283+0000 7fdfdcb70640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3d70 0x558be74fe0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.283+0000 7fdfddb72640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74fe630 0x558be7500a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.285+0000 7fdfdd371640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74f3450 0x558be74f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.285+0000 7fdfddb72640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74fe630 0x558be7500a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:46248/0 (socket says 172.21.15.98:46248) 2024-05-09T08:52:58.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.285+0000 7fdfddb72640 1 -- 172.21.15.98:0/3910624204 learned_addr learned my addr 172.21.15.98:0/3910624204 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:58.286 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.287+0000 7fdfddb72640 1 -- 172.21.15.98:0/3910624204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3d70 msgr2=0x558be74fe0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:58.286 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.287+0000 7fdfddb72640 1 --2- 172.21.15.98:0/3910624204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3d70 0x558be74fe0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.286 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.287+0000 7fdfddb72640 1 -- 172.21.15.98:0/3910624204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74f3450 msgr2=0x558be74f3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:58.286 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.287+0000 7fdfddb72640 1 --2- 172.21.15.98:0/3910624204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74f3450 0x558be74f3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.287 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.287+0000 7fdfddb72640 1 -- 172.21.15.98:0/3910624204 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558be74248e0 con 0x558be74fe630 2024-05-09T08:52:58.287 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.287+0000 7fdfdcb70640 1 --2- 172.21.15.98:0/3910624204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3d70 0x558be74fe0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:52:58.287 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.288+0000 7fdfdd371640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdfc40a2fd0 0x7fdfc40a5490 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4015420 tx=0x7fdfd4015820 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:58.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.288+0000 7fdfddb72640 1 --2- 172.21.15.98:0/3910624204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74fe630 0x558be7500a20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009cd0 tx=0x7fdfd002f6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1898bb9a4426a50b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:58.299 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.297+0000 7fdfc37fe640 1 -- 172.21.15.98:0/3910624204 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0047070 con 0x558be74fe630 2024-05-09T08:52:58.299 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.297+0000 7fdfc37fe640 1 -- 172.21.15.98:0/3910624204 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfd0003e50 con 0x558be74fe630 2024-05-09T08:52:58.299 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.297+0000 7fdfc37fe640 1 -- 172.21.15.98:0/3910624204 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0034050 con 0x558be74fe630 2024-05-09T08:52:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.299+0000 7fdfe23c2740 1 -- 172.21.15.98:0/3910624204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74fe630 msgr2=0x558be7500a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.299+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/3910624204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74fe630 0x558be7500a20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009cd0 tx=0x7fdfd002f6f0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.299+0000 7fdfe23c2740 1 -- 172.21.15.98:0/3910624204 shutdown_connections 2024-05-09T08:52:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.299+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/3910624204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74fe630 0x558be7500a20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.299+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/3910624204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3d70 0x558be74fe0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.299+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/3910624204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74f3450 0x558be74f3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.299+0000 7fdfe23c2740 1 -- 172.21.15.98:0/3910624204 >> 172.21.15.98:0/3910624204 conn(0x558be74ef960 msgr2=0x558be74efd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.303+0000 7fdfe23c2740 1 -- 172.21.15.98:0/3910624204 shutdown_connections 2024-05-09T08:52:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.303+0000 7fdfe23c2740 1 -- 172.21.15.98:0/3910624204 wait complete. 2024-05-09T08:52:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.304+0000 7fdfe23c2740 1 -- start start 2024-05-09T08:52:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.304+0000 7fdfe23c2740 1 Processor -- start 2024-05-09T08:52:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.305+0000 7fdfe23c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74f3450 0x558be74fb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.305+0000 7fdfe23c2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3d70 0x558be74fbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.305+0000 7fdfe23c2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74fe630 0x558be74fc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.305+0000 7fdfe23c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558be7501200 con 0x558be74f3450 2024-05-09T08:52:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.305+0000 7fdfe23c2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558be7413790 con 0x558be74fe630 2024-05-09T08:52:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.305+0000 7fdfe23c2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558be7501380 con 0x558be74f3d70 2024-05-09T08:52:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.305+0000 7fdfe23c2740 10 client.0 _ll_register_callbacks cb 0x558be744ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.307+0000 7fdfdd371640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74f3450 0x558be74fb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.307+0000 7fdfdd371640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74f3450 0x558be74fb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51936/0 (socket says 172.21.15.98:51936) 2024-05-09T08:52:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.307+0000 7fdfdd371640 1 -- 172.21.15.98:0/2867871651 learned_addr learned my addr 172.21.15.98:0/2867871651 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.307+0000 7fdfddb72640 1 --2- 172.21.15.98:0/2867871651 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74fe630 0x558be74fc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.308+0000 7fdfdcb70640 1 --2- 172.21.15.98:0/2867871651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3d70 0x558be74fbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.308+0000 7fdfdd371640 1 -- 172.21.15.98:0/2867871651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3d70 msgr2=0x558be74fbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.308+0000 7fdfdd371640 1 --2- 172.21.15.98:0/2867871651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3d70 0x558be74fbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.308+0000 7fdfdd371640 1 -- 172.21.15.98:0/2867871651 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74fe630 msgr2=0x558be74fc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.308+0000 7fdfdd371640 1 --2- 172.21.15.98:0/2867871651 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74fe630 0x558be74fc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.308+0000 7fdfdd371640 1 -- 172.21.15.98:0/2867871651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558be716c210 con 0x558be74f3450 2024-05-09T08:52:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.309+0000 7fdfdd371640 1 --2- 172.21.15.98:0/2867871651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74f3450 0x558be74fb590 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fdfd40126e0 tx=0x7fdfd4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b034ef70094769 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.309+0000 7fdfc0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:52:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.313+0000 7fdfc0ff9640 1 -- 172.21.15.98:0/2867871651 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4020900 con 0x558be74f3450 2024-05-09T08:52:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.313+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2867871651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfd0003e50 con 0x558be74f3450 2024-05-09T08:52:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.313+0000 7fdfc0ff9640 1 -- 172.21.15.98:0/2867871651 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fdfd4020aa0 con 0x558be74f3450 2024-05-09T08:52:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.318+0000 7fdfc0ff9640 1 -- 172.21.15.98:0/2867871651 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdfd4020ca0 con 0x558be74f3450 2024-05-09T08:52:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.319+0000 7fdfe23c2740 10 client.16353 fetch_fsmap learned FSMap version 30 2024-05-09T08:52:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.319+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2867871651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558be7238620 con 0x558be74f3450 2024-05-09T08:52:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.321+0000 7fdfc0ff9640 1 -- 172.21.15.98:0/2867871651 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfd0003e50 con 0x558be74f3450 2024-05-09T08:52:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.321+0000 7fdfc0ff9640 1 -- 172.21.15.98:0/2867871651 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 7757+0+0 (secure 0 0 0) 0x7fdfd402a520 con 0x558be74f3450 2024-05-09T08:52:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.323+0000 7fdfe23c2740 10 client.16353 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:52:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.323+0000 7fdfe23c2740 10 client.16353 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:58.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.323+0000 7fdfe23c2740 20 client.16353 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:58.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.323+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/2867871651 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558be750e730 0x558be7510bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.323+0000 7fdfe23c2740 4 client.16353 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:52:58.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.323+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2867871651 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558be7465a70 con 0x558be750e730 2024-05-09T08:52:58.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.324+0000 7fdfdcb70640 1 --2- 172.21.15.98:0/2867871651 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558be750e730 0x558be7510bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.326+0000 7fdfdcb70640 1 --2- 172.21.15.98:0/2867871651 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558be750e730 0x558be7510bb0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5eb3af9bd2cefed5 server_cookie=de06b6159d23e92 in_seq=0 out_seq=0 2024-05-09T08:52:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.326+0000 7fdfc0ff9640 10 client.16353 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:58.332 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.333+0000 7fdfc0ff9640 1 -- 172.21.15.98:0/2867871651 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558be7465a70 con 0x558be750e730 2024-05-09T08:52:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.333+0000 7fdfc0ff9640 10 client.16353 handle_command_reply: tid=0 2024-05-09T08:52:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.333+0000 7fdfe23c2740 1 client.16353 shutdown 2024-05-09T08:52:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.335+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2867871651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74f3450 msgr2=0x558be74fb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.335+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/2867871651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74f3450 0x558be74fb590 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fdfd40126e0 tx=0x7fdfd4004730 comp rx=0 tx=0).stop 2024-05-09T08:52:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.335+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2867871651 shutdown_connections 2024-05-09T08:52:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.335+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/2867871651 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558be750e730 0x558be7510bb0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.335+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/2867871651 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74fe630 0x558be74fc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.335+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/2867871651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3d70 0x558be74fbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.335+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/2867871651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74f3450 0x558be74fb590 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.335+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2867871651 >> 172.21.15.98:0/2867871651 conn(0x558be74ef960 msgr2=0x558be74f56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.335+0000 7fdfc0ff9640 0 client.16353 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:52:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.337+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2867871651 shutdown_connections 2024-05-09T08:52:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.337+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2867871651 wait complete. 2024-05-09T08:52:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.338+0000 7fdfe23c2740 20 client.16353 trim_cache size 0 max 16384 2024-05-09T08:52:58.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:52:58 smithi098 ceph-mon[30092]: pgmap v72: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 949 MiB/s wr, 1.92k op/s 2024-05-09T08:52:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.375+0000 7fdfe23c2740 1 Processor -- start 2024-05-09T08:52:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.375+0000 7fdfe23c2740 1 -- start start 2024-05-09T08:52:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.375+0000 7fdfe23c2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3450 0x558be750a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.377+0000 7fdfe23c2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3d70 0x558be74f5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.377+0000 7fdfe23c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74fe630 0x558be74f9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.377+0000 7fdfdd371640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3450 0x558be750a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.378+0000 7fdfdd371640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3450 0x558be750a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:46268/0 (socket says 172.21.15.98:46268) 2024-05-09T08:52:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.378+0000 7fdfdd371640 1 -- 172.21.15.98:0/3348664221 learned_addr learned my addr 172.21.15.98:0/3348664221 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.378+0000 7fdfdcb70640 1 --2- 172.21.15.98:0/3348664221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3d70 0x558be74f5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.378+0000 7fdfe23c2740 1 -- 172.21.15.98:0/3348664221 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfbc011b50 con 0x558be74fe630 2024-05-09T08:52:58.378 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.378+0000 7fdfe23c2740 1 -- 172.21.15.98:0/3348664221 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdfbc00f940 con 0x558be74f3d70 2024-05-09T08:52:58.378 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.378+0000 7fdfe23c2740 1 -- 172.21.15.98:0/3348664221 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdfbc011480 con 0x558be74f3450 2024-05-09T08:52:58.378 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.378+0000 7fdfddb72640 1 --2- 172.21.15.98:0/3348664221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74fe630 0x558be74f9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.378 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.379+0000 7fdfdcb70640 1 -- 172.21.15.98:0/3348664221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3450 msgr2=0x558be750a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:58.378 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.379+0000 7fdfdcb70640 1 --2- 172.21.15.98:0/3348664221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3450 0x558be750a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.379+0000 7fdfdcb70640 1 -- 172.21.15.98:0/3348664221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74fe630 msgr2=0x558be74f9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.379+0000 7fdfdcb70640 1 --2- 172.21.15.98:0/3348664221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74fe630 0x558be74f9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.379+0000 7fdfdcb70640 1 -- 172.21.15.98:0/3348664221 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558be74fa540 con 0x558be74f3d70 2024-05-09T08:52:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.380+0000 7fdfdcb70640 1 --2- 172.21.15.98:0/3348664221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3d70 0x558be74f5e60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdfc80cd6b0 tx=0x7fdfc801c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a5e202189a833f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.390+0000 7fdfc2ffd640 1 -- 172.21.15.98:0/3348664221 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc80ade60 con 0x558be74f3d70 2024-05-09T08:52:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.390+0000 7fdfc2ffd640 1 -- 172.21.15.98:0/3348664221 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc80abe50 con 0x558be74f3d70 2024-05-09T08:52:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.390+0000 7fdfc2ffd640 1 -- 172.21.15.98:0/3348664221 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc809b8d0 con 0x558be74f3d70 2024-05-09T08:52:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.393+0000 7fdfe23c2740 1 -- 172.21.15.98:0/3348664221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3d70 msgr2=0x558be74f5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:52:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.393+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/3348664221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3d70 0x558be74f5e60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdfc80cd6b0 tx=0x7fdfc801c040 comp rx=0 tx=0).stop 2024-05-09T08:52:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.393+0000 7fdfe23c2740 1 -- 172.21.15.98:0/3348664221 shutdown_connections 2024-05-09T08:52:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.393+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/3348664221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74fe630 0x558be74f9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.393+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/3348664221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3d70 0x558be74f5e60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.393+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/3348664221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3450 0x558be750a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.393+0000 7fdfe23c2740 1 -- 172.21.15.98:0/3348664221 >> 172.21.15.98:0/3348664221 conn(0x558be74ef960 msgr2=0x558be74eea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:52:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.397+0000 7fdfe23c2740 1 -- 172.21.15.98:0/3348664221 shutdown_connections 2024-05-09T08:52:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.397+0000 7fdfe23c2740 1 -- 172.21.15.98:0/3348664221 wait complete. 2024-05-09T08:52:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.397+0000 7fdfe23c2740 1 -- start start 2024-05-09T08:52:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.398+0000 7fdfe23c2740 1 Processor -- start 2024-05-09T08:52:58.398 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.398+0000 7fdfe23c2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3450 0x558be7506b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.399+0000 7fdfe23c2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3d70 0x558be7501f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.399+0000 7fdfe23c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74fe630 0x558be7502640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.399+0000 7fdfdd371640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3450 0x558be7506b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.399+0000 7fdfdd371640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3450 0x558be7506b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:42530/0 (socket says 172.21.15.98:42530) 2024-05-09T08:52:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.399+0000 7fdfdd371640 1 -- 172.21.15.98:0/2786585787 learned_addr learned my addr 172.21.15.98:0/2786585787 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:52:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.399+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2786585787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfbc0106e0 con 0x558be74fe630 2024-05-09T08:52:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.400+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2786585787 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdfbc010010 con 0x558be74f3450 2024-05-09T08:52:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.400+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2786585787 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdfbc010db0 con 0x558be74f3d70 2024-05-09T08:52:58.400 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.400+0000 7fdfdcb70640 1 --2- 172.21.15.98:0/2786585787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3d70 0x558be7501f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.400 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.400+0000 7fdfe23c2740 10 client.0 _ll_register_callbacks cb 0x7fdfd4005e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:52:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.403+0000 7fdfdd371640 1 -- 172.21.15.98:0/2786585787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3d70 msgr2=0x558be7501f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.403+0000 7fdfdd371640 1 --2- 172.21.15.98:0/2786585787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3d70 0x558be7501f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.403+0000 7fdfdd371640 1 -- 172.21.15.98:0/2786585787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74fe630 msgr2=0x558be7502640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:52:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.403+0000 7fdfdd371640 1 --2- 172.21.15.98:0/2786585787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74fe630 0x558be7502640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:52:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.403+0000 7fdfdd371640 1 -- 172.21.15.98:0/2786585787 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558be7573d30 con 0x558be74f3450 2024-05-09T08:52:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.404+0000 7fdfdd371640 1 --2- 172.21.15.98:0/2786585787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3450 0x558be7506b90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4009350 tx=0x7fdfd402a610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab86ca750e9321f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:52:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.404+0000 7fdfc0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:52:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.404+0000 7fdfc0ff9640 1 -- 172.21.15.98:0/2786585787 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd40099e0 con 0x558be74f3450 2024-05-09T08:52:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.404+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2786585787 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfc80abe50 con 0x558be74f3450 2024-05-09T08:52:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.404+0000 7fdfc0ff9640 1 -- 172.21.15.98:0/2786585787 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fdfd4004020 con 0x558be74f3450 2024-05-09T08:52:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.405+0000 7fdfc0ff9640 1 -- 172.21.15.98:0/2786585787 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdfd4043050 con 0x558be74f3450 2024-05-09T08:52:58.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:52:58 smithi142 ceph-mon[34625]: pgmap v72: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 949 MiB/s wr, 1.92k op/s 2024-05-09T08:52:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.406+0000 7fdfe23c2740 10 client.25730 fetch_fsmap learned FSMap version 30 2024-05-09T08:52:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.406+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2786585787 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558be7574ee0 con 0x558be74f3450 2024-05-09T08:52:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.410+0000 7fdfc0ff9640 1 -- 172.21.15.98:0/2786585787 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc80abe50 con 0x558be74f3450 2024-05-09T08:52:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.410+0000 7fdfc0ff9640 1 -- 172.21.15.98:0/2786585787 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 30) ==== 7757+0+0 (secure 0 0 0) 0x7fdfd4003ad0 con 0x558be74f3450 2024-05-09T08:52:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.410+0000 7fdfe23c2740 10 client.25730 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:52:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.410+0000 7fdfe23c2740 10 client.25730 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:52:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.410+0000 7fdfe23c2740 20 client.25730 populate_metadata read hostname 'smithi098' 2024-05-09T08:52:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.410+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/2786585787 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558be7575630 0x558be7577a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:52:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.410+0000 7fdfe23c2740 4 client.25730 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:52:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.410+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2786585787 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558be745fdc0 con 0x558be7575630 2024-05-09T08:52:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.564+0000 7fdfdcb70640 1 --2- 172.21.15.98:0/2786585787 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558be7575630 0x558be7577a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:52:58.567 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.567+0000 7fdfdcb70640 1 --2- 172.21.15.98:0/2786585787 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558be7575630 0x558be7577a50 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64db96cf85a7762e server_cookie=58eb182f8156ae09 in_seq=0 out_seq=0 2024-05-09T08:52:58.567 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:52:58.568+0000 7fdfc0ff9640 10 client.25730 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:00.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:00 smithi142 ceph-mon[34625]: pgmap v73: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 717 MiB/s wr, 1.42k op/s 2024-05-09T08:53:00.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:00 smithi105 ceph-mon[34498]: pgmap v73: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 717 MiB/s wr, 1.42k op/s 2024-05-09T08:53:00.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:00 smithi098 ceph-mon[30092]: pgmap v73: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 717 MiB/s wr, 1.42k op/s 2024-05-09T08:53:01.486 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:53:01 smithi105 podman[68209]: 2024-05-09 08:53:01.048684977 +0000 UTC m=+5.827675734 container init aade97aca6951d5ec2fe8cf6455e2b33b26f8b775875e9b68219d81f99d42ce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507) 2024-05-09T08:53:01.487 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:53:01 smithi105 podman[68209]: 2024-05-09 08:53:01.053842826 +0000 UTC m=+5.832833589 container start aade97aca6951d5ec2fe8cf6455e2b33b26f8b775875e9b68219d81f99d42ce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.build-date=20240507) 2024-05-09T08:53:01.487 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:53:01 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: starting mds.e at 2024-05-09T08:53:02.387 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:02 smithi105 ceph-mon[34498]: pgmap v74: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 964 MiB/s wr, 2.07k op/s 2024-05-09T08:53:02.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:02 smithi142 ceph-mon[34625]: pgmap v74: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 964 MiB/s wr, 2.07k op/s 2024-05-09T08:53:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:02 smithi098 ceph-mon[30092]: pgmap v74: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 964 MiB/s wr, 2.07k op/s 2024-05-09T08:53:03.236 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:53:03.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:53:03.237 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:53:02 smithi105 bash[68209]: aade97aca6951d5ec2fe8cf6455e2b33b26f8b775875e9b68219d81f99d42ce1 2024-05-09T08:53:03.237 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:53:02 smithi105 systemd[1]: Started Ceph mds.e for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:53:03.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.565+0000 7f895f942640 1 -- 172.21.15.98:0/2412402647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8958106bc0 msgr2=0x7f895810e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:03.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.565+0000 7f895f942640 1 --2- 172.21.15.98:0/2412402647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8958106bc0 0x7f895810e740 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f8950009ca0 tx=0x7f895002a8d0 comp rx=0 tx=0).stop 2024-05-09T08:53:03.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.570+0000 7f895f942640 1 -- 172.21.15.98:0/2412402647 shutdown_connections 2024-05-09T08:53:03.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.570+0000 7f895f942640 1 --2- 172.21.15.98:0/2412402647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8958106bc0 0x7f895810e740 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:03.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.570+0000 7f895f942640 1 --2- 172.21.15.98:0/2412402647 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8958100960 0x7f8958106490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:03.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.570+0000 7f895f942640 1 --2- 172.21.15.98:0/2412402647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89580fffb0 0x7f8958100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:03.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.570+0000 7f895f942640 1 -- 172.21.15.98:0/2412402647 >> 172.21.15.98:0/2412402647 conn(0x7f8958076cd0 msgr2=0x7f89580fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:03.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.575+0000 7f895f942640 1 -- 172.21.15.98:0/2412402647 shutdown_connections 2024-05-09T08:53:03.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.576+0000 7f895f942640 1 -- 172.21.15.98:0/2412402647 wait complete. 2024-05-09T08:53:03.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.582+0000 7f895f942640 1 Processor -- start 2024-05-09T08:53:03.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.582+0000 7f895f942640 1 -- start start 2024-05-09T08:53:03.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.583+0000 7f895f942640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89580fffb0 0x7f8958117090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:03.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.583+0000 7f895f942640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8958106bc0 0x7f89581111a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:03.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.583+0000 7f895f942640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89581116e0 0x7f8958111b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:03.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.583+0000 7f895f942640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f895810fad0 con 0x7f89581116e0 2024-05-09T08:53:03.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.585+0000 7f895dedf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89581116e0 0x7f8958111b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:03.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.585+0000 7f895dedf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89581116e0 0x7f8958111b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51976/0 (socket says 172.21.15.98:51976) 2024-05-09T08:53:03.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.585+0000 7f895dedf640 1 -- 172.21.15.98:0/2327378446 learned_addr learned my addr 172.21.15.98:0/2327378446 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:03.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.585+0000 7f895d6de640 1 --2- 172.21.15.98:0/2327378446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89580fffb0 0x7f8958117090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:03.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.585+0000 7f895f942640 1 -- 172.21.15.98:0/2327378446 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f895810f950 con 0x7f8958106bc0 2024-05-09T08:53:03.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.585+0000 7f895f942640 1 -- 172.21.15.98:0/2327378446 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f895810fc50 con 0x7f89580fffb0 2024-05-09T08:53:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.589+0000 7f895dedf640 1 -- 172.21.15.98:0/2327378446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89580fffb0 msgr2=0x7f8958117090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.589+0000 7f895dedf640 1 --2- 172.21.15.98:0/2327378446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89580fffb0 0x7f8958117090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.589+0000 7f895dedf640 1 -- 172.21.15.98:0/2327378446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8958106bc0 msgr2=0x7f89581111a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:53:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.589+0000 7f895dedf640 1 --2- 172.21.15.98:0/2327378446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8958106bc0 0x7f89581111a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.589+0000 7f895dedf640 1 -- 172.21.15.98:0/2327378446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89581b37f0 con 0x7f89581116e0 2024-05-09T08:53:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.589+0000 7f895d6de640 1 --2- 172.21.15.98:0/2327378446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89580fffb0 0x7f8958117090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:53:03.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.589+0000 7f895dedf640 1 --2- 172.21.15.98:0/2327378446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89581116e0 0x7f8958111b90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8950031690 tx=0x7f89500316c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:03.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:03 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.105:6834/3228987570,v1:172.21.15.105:6835/3228987570] up:boot 2024-05-09T08:53:03.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:03 smithi098 ceph-mon[30092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:53:03.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:03 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:53:03.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.593+0000 7f894e7fc640 1 -- 172.21.15.98:0/2327378446 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950003e20 con 0x7f89581116e0 2024-05-09T08:53:03.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.593+0000 7f894e7fc640 1 -- 172.21.15.98:0/2327378446 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8950004520 con 0x7f89581116e0 2024-05-09T08:53:03.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.594+0000 7f894e7fc640 1 -- 172.21.15.98:0/2327378446 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895002fd40 con 0x7f89581116e0 2024-05-09T08:53:03.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.594+0000 7f895f942640 1 -- 172.21.15.98:0/2327378446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89581b39c0 con 0x7f89581116e0 2024-05-09T08:53:03.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.595+0000 7f895f942640 1 -- 172.21.15.98:0/2327378446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89581b3e90 con 0x7f89581116e0 2024-05-09T08:53:03.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.599+0000 7f896264f740 1 Processor -- start 2024-05-09T08:53:03.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.600+0000 7f896264f740 1 -- start start 2024-05-09T08:53:03.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.602+0000 7f896264f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9260 0x5632ba9f9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:03.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.602+0000 7f896264f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9b80 0x5632baa03f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:03.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.602+0000 7f896264f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632baa04440 0x5632baa06830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:03.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.602+0000 7f896264f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5632ba95ed00 con 0x5632ba9f9260 2024-05-09T08:53:03.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.602+0000 7f896264f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5632ba9064c0 con 0x5632ba9f9b80 2024-05-09T08:53:03.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.602+0000 7f896264f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5632ba95acc0 con 0x5632baa04440 2024-05-09T08:53:03.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.604+0000 7f895cedd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632baa04440 0x5632baa06830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:03.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.604+0000 7f895cedd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632baa04440 0x5632baa06830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:46308/0 (socket says 172.21.15.98:46308) 2024-05-09T08:53:03.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.604+0000 7f895cedd640 1 -- 172.21.15.98:0/3149129997 learned_addr learned my addr 172.21.15.98:0/3149129997 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:03.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.605+0000 7f895cedd640 1 -- 172.21.15.98:0/3149129997 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9b80 msgr2=0x5632baa03f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:03.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.605+0000 7f895cedd640 1 --2- 172.21.15.98:0/3149129997 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9b80 0x5632baa03f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:03.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.605+0000 7f895cedd640 1 -- 172.21.15.98:0/3149129997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9260 msgr2=0x5632ba9f9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:03.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.605+0000 7f895cedd640 1 --2- 172.21.15.98:0/3149129997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9260 0x5632ba9f9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:03.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.605+0000 7f895cedd640 1 -- 172.21.15.98:0/3149129997 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632ba73c7d0 con 0x5632baa04440 2024-05-09T08:53:03.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.606+0000 7f895cedd640 1 --2- 172.21.15.98:0/3149129997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632baa04440 0x5632baa06830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f895400b7d0 tx=0x7f895400bc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb348be900cb7991 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:03.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.606+0000 7f894e7fc640 1 -- 172.21.15.98:0/2327378446 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8950004040 con 0x7f89581116e0 2024-05-09T08:53:03.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.607+0000 7f8942ffd640 1 -- 172.21.15.98:0/3149129997 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954018070 con 0x5632baa04440 2024-05-09T08:53:03.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.607+0000 7f8942ffd640 1 -- 172.21.15.98:0/3149129997 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8954006500 con 0x5632baa04440 2024-05-09T08:53:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.608+0000 7f894e7fc640 1 --2- 172.21.15.98:0/2327378446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89440a2f10 0x7f89440a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.608+0000 7f894e7fc640 1 -- 172.21.15.98:0/2327378446 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f89500f0860 con 0x7f89581116e0 2024-05-09T08:53:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.608+0000 7f896264f740 1 -- 172.21.15.98:0/3149129997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632baa04440 msgr2=0x5632baa06830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.608+0000 7f896264f740 1 --2- 172.21.15.98:0/3149129997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632baa04440 0x5632baa06830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f895400b7d0 tx=0x7f895400bc90 comp rx=0 tx=0).stop 2024-05-09T08:53:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.611+0000 7f896264f740 1 -- 172.21.15.98:0/3149129997 shutdown_connections 2024-05-09T08:53:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.611+0000 7f896264f740 1 --2- 172.21.15.98:0/3149129997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632baa04440 0x5632baa06830 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.611+0000 7f896264f740 1 --2- 172.21.15.98:0/3149129997 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9b80 0x5632baa03f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.611+0000 7f896264f740 1 --2- 172.21.15.98:0/3149129997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9260 0x5632ba9f9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.611+0000 7f896264f740 1 -- 172.21.15.98:0/3149129997 >> 172.21.15.98:0/3149129997 conn(0x5632ba9f5770 msgr2=0x5632ba9f5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:03.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.612+0000 7f896264f740 1 -- 172.21.15.98:0/3149129997 shutdown_connections 2024-05-09T08:53:03.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.612+0000 7f896264f740 1 -- 172.21.15.98:0/3149129997 wait complete. 2024-05-09T08:53:03.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.612+0000 7f895d6de640 1 --2- 172.21.15.98:0/2327378446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89440a2f10 0x7f89440a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:03.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.613+0000 7f896264f740 1 -- start start 2024-05-09T08:53:03.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.613+0000 7f896264f740 1 Processor -- start 2024-05-09T08:53:03.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.613+0000 7f895d6de640 1 --2- 172.21.15.98:0/2327378446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89440a2f10 0x7f89440a53d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x5632ba9faa10 tx=0x7f89480047d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:03.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.614+0000 7f896264f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ba9f9260 0x5632baa01330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:03.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.614+0000 7f896264f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9b80 0x5632baa01870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:03.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.614+0000 7f896264f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632baa04440 0x5632baa01db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:03.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.614+0000 7f896264f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5632baa07110 con 0x5632baa04440 2024-05-09T08:53:03.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.614+0000 7f896264f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5632ba919b90 con 0x5632ba9f9b80 2024-05-09T08:53:03.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.614+0000 7f896264f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5632baa07290 con 0x5632ba9f9260 2024-05-09T08:53:03.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.614+0000 7f896264f740 10 client.0 _ll_register_callbacks cb 0x5632ba955da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.615+0000 7f895dedf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632baa04440 0x5632baa01db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.615+0000 7f895cedd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9b80 0x5632baa01870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.615+0000 7f895cedd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9b80 0x5632baa01870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:42568/0 (socket says 172.21.15.98:42568) 2024-05-09T08:53:03.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.615+0000 7f895cedd640 1 -- 172.21.15.98:0/1515201060 learned_addr learned my addr 172.21.15.98:0/1515201060 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:03.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.616+0000 7f895dedf640 1 -- 172.21.15.98:0/1515201060 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ba9f9260 msgr2=0x5632baa01330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:03.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.616+0000 7f895d6de640 1 --2- 172.21.15.98:0/1515201060 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ba9f9260 0x5632baa01330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:03.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.617+0000 7f895dedf640 1 --2- 172.21.15.98:0/1515201060 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ba9f9260 0x5632baa01330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:03.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.617+0000 7f895dedf640 1 -- 172.21.15.98:0/1515201060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9b80 msgr2=0x5632baa01870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:03.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.617+0000 7f895dedf640 1 --2- 172.21.15.98:0/1515201060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9b80 0x5632baa01870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:03.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.617+0000 7f895dedf640 1 -- 172.21.15.98:0/1515201060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632ba6723f0 con 0x5632baa04440 2024-05-09T08:53:03.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.617+0000 7f895d6de640 1 --2- 172.21.15.98:0/1515201060 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ba9f9260 0x5632baa01330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:53:03.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.617+0000 7f895cedd640 1 --2- 172.21.15.98:0/1515201060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9b80 0x5632baa01870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:53:03.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.617+0000 7f895dedf640 1 --2- 172.21.15.98:0/1515201060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632baa04440 0x5632baa01db0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f89500edca0 tx=0x7f89500bc900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44cb06c24726873d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:03.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.617+0000 7f8940ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:03.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.618+0000 7f8940ff9640 1 -- 172.21.15.98:0/1515201060 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89500bd170 con 0x5632baa04440 2024-05-09T08:53:03.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.618+0000 7f896264f740 1 -- 172.21.15.98:0/1515201060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8954006500 con 0x5632baa04440 2024-05-09T08:53:03.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.618+0000 7f8940ff9640 1 -- 172.21.15.98:0/1515201060 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f89500d5060 con 0x5632baa04440 2024-05-09T08:53:03.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.619+0000 7f8940ff9640 1 -- 172.21.15.98:0/1515201060 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f89500da050 con 0x5632baa04440 2024-05-09T08:53:03.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.620+0000 7f896264f740 10 client.16368 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:03.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.620+0000 7f896264f740 1 -- 172.21.15.98:0/1515201060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5632ba6320f0 con 0x5632baa04440 2024-05-09T08:53:03.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.622+0000 7f8940ff9640 1 -- 172.21.15.98:0/1515201060 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8954006500 con 0x5632baa04440 2024-05-09T08:53:03.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.622+0000 7f8940ff9640 1 -- 172.21.15.98:0/1515201060 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f895004c030 con 0x5632baa04440 2024-05-09T08:53:03.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.622+0000 7f896264f740 10 client.16368 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:03.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.622+0000 7f896264f740 10 client.16368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:03.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.622+0000 7f896264f740 20 client.16368 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:03.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.622+0000 7f896264f740 1 --2- 172.21.15.98:0/1515201060 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5632baa14480 0x5632baa168b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:03.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.622+0000 7f896264f740 4 client.16368 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:03.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.622+0000 7f896264f740 1 -- 172.21.15.98:0/1515201060 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632ba96c430 con 0x5632baa14480 2024-05-09T08:53:03.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.625+0000 7f895d6de640 1 --2- 172.21.15.98:0/1515201060 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5632baa14480 0x5632baa168b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:03.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.632+0000 7f895d6de640 1 --2- 172.21.15.98:0/1515201060 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5632baa14480 0x5632baa168b0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb2cabbcd6cf1f00 server_cookie=dd86d86309b8a956 in_seq=0 out_seq=0 2024-05-09T08:53:03.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:03.632+0000 7f8940ff9640 10 client.16368 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:03 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.105:6834/3228987570,v1:172.21.15.105:6835/3228987570] up:boot 2024-05-09T08:53:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:03 smithi142 ceph-mon[34625]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:53:03.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:03 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:53:03.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:03 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.105:6834/3228987570,v1:172.21.15.105:6835/3228987570] up:boot 2024-05-09T08:53:03.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:03 smithi105 ceph-mon[34498]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:53:03.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:03 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:53:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:04 smithi098 ceph-mon[30092]: pgmap v75: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 770 MiB/s wr, 1.62k op/s 2024-05-09T08:53:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:04 smithi142 ceph-mon[34625]: pgmap v75: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 770 MiB/s wr, 1.62k op/s 2024-05-09T08:53:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.690+0000 7fdfc0ff9640 1 -- 172.21.15.98:0/2786585787 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fdfc80b4050 con 0x558be7575630 2024-05-09T08:53:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.690+0000 7fdfc0ff9640 10 client.25730 handle_command_reply: tid=0 2024-05-09T08:53:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.690+0000 7fdfe23c2740 1 client.25730 shutdown 2024-05-09T08:53:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.690+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2786585787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3450 msgr2=0x558be7506b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.690+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/2786585787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3450 0x558be7506b90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4009350 tx=0x7fdfd402a610 comp rx=0 tx=0).stop 2024-05-09T08:53:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.691+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2786585787 shutdown_connections 2024-05-09T08:53:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.691+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/2786585787 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558be7575630 0x558be7577a50 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.691+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/2786585787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558be74fe630 0x558be7502640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.691+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/2786585787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558be74f3d70 0x558be7501f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.691+0000 7fdfe23c2740 1 --2- 172.21.15.98:0/2786585787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558be74f3450 0x558be7506b90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.691+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2786585787 >> 172.21.15.98:0/2786585787 conn(0x558be74ef960 msgr2=0x558be74fb530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.691+0000 7fdfc0ff9640 0 client.25730 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.691+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2786585787 shutdown_connections 2024-05-09T08:53:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.691+0000 7fdfe23c2740 1 -- 172.21.15.98:0/2786585787 wait complete. 2024-05-09T08:53:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.691+0000 7fdfe23c2740 20 client.25730 trim_cache size 0 max 16384 2024-05-09T08:53:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:53:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:53:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:53:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:53:04.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.692+0000 7f8940ff9640 1 -- 172.21.15.98:0/1515201060 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632ba96c430 con 0x5632baa14480 2024-05-09T08:53:04.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.692+0000 7f8940ff9640 10 client.16368 handle_command_reply: tid=0 2024-05-09T08:53:04.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.692+0000 7f896264f740 1 client.16368 shutdown 2024-05-09T08:53:04.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.694+0000 7f896264f740 1 -- 172.21.15.98:0/1515201060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632baa04440 msgr2=0x5632baa01db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:04.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.694+0000 7f896264f740 1 --2- 172.21.15.98:0/1515201060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632baa04440 0x5632baa01db0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f89500edca0 tx=0x7f89500bc900 comp rx=0 tx=0).stop 2024-05-09T08:53:04.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.694+0000 7fdfdf5d5640 1 -- 172.21.15.98:0/3945488973 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdfc40a2fd0 msgr2=0x7fdfc40a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:04.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.694+0000 7fdfdf5d5640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdfc40a2fd0 0x7fdfc40a5490 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4015420 tx=0x7fdfd4015820 comp rx=0 tx=0).stop 2024-05-09T08:53:04.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.694+0000 7fdfdf5d5640 1 -- 172.21.15.98:0/3945488973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfd811b010 msgr2=0x7fdfd811aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:04.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.694+0000 7fdfdf5d5640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfd811b010 0x7fdfd811aac0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800b470 tx=0x7fdfc800b930 comp rx=0 tx=0).stop 2024-05-09T08:53:04.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.694+0000 7fdfdf5d5640 1 -- 172.21.15.98:0/3945488973 shutdown_connections 2024-05-09T08:53:04.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.694+0000 7fdfdf5d5640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdfc40a2fd0 0x7fdfc40a5490 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.694+0000 7fdfdf5d5640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd8113710 0x7fdfd8113bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.694+0000 7fdfdf5d5640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfd811b010 0x7fdfd811aac0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.694+0000 7fdfdf5d5640 1 --2- 172.21.15.98:0/3945488973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfd8110020 0x7fdfd811a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.694+0000 7fdfdf5d5640 1 -- 172.21.15.98:0/3945488973 >> 172.21.15.98:0/3945488973 conn(0x7fdfd806c900 msgr2=0x7fdfd811b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:04.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.694+0000 7f896264f740 1 -- 172.21.15.98:0/1515201060 shutdown_connections 2024-05-09T08:53:04.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.694+0000 7f896264f740 1 --2- 172.21.15.98:0/1515201060 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5632baa14480 0x5632baa168b0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.694+0000 7f896264f740 1 --2- 172.21.15.98:0/1515201060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632baa04440 0x5632baa01db0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.694+0000 7f896264f740 1 --2- 172.21.15.98:0/1515201060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9b80 0x5632baa01870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.694+0000 7f896264f740 1 --2- 172.21.15.98:0/1515201060 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ba9f9260 0x5632baa01330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.694+0000 7f896264f740 1 -- 172.21.15.98:0/1515201060 >> 172.21.15.98:0/1515201060 conn(0x5632ba9f5770 msgr2=0x5632ba9f48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:04.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.694+0000 7f8940ff9640 0 client.16368 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.695+0000 7fdfdf5d5640 1 -- 172.21.15.98:0/3945488973 shutdown_connections 2024-05-09T08:53:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:04.695+0000 7fdfdf5d5640 1 -- 172.21.15.98:0/3945488973 wait complete. 2024-05-09T08:53:04.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.695+0000 7f896264f740 1 -- 172.21.15.98:0/1515201060 shutdown_connections 2024-05-09T08:53:04.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.695+0000 7f896264f740 1 -- 172.21.15.98:0/1515201060 wait complete. 2024-05-09T08:53:04.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.695+0000 7f896264f740 20 client.16368 trim_cache size 0 max 16384 2024-05-09T08:53:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:04 smithi105 ceph-mon[34498]: pgmap v75: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 770 MiB/s wr, 1.62k op/s 2024-05-09T08:53:04.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.748+0000 7f896264f740 1 Processor -- start 2024-05-09T08:53:04.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.748+0000 7f896264f740 1 -- start start 2024-05-09T08:53:04.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.748+0000 7f896264f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9260 0x5632baa10020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:04.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.748+0000 7f896264f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9b80 0x5632baa01bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:04.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.749+0000 7f896264f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632baa04440 0x5632baa1c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:04.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.749+0000 7f896264f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89581bddf0 con 0x5632ba9f9b80 2024-05-09T08:53:04.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.749+0000 7f896264f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89581bf930 con 0x5632ba9f9260 2024-05-09T08:53:04.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.749+0000 7f896264f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f89581be4c0 con 0x5632baa04440 2024-05-09T08:53:04.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.750+0000 7f895cedd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9b80 0x5632baa01bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:04.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.750+0000 7f895cedd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9b80 0x5632baa01bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52012/0 (socket says 172.21.15.98:52012) 2024-05-09T08:53:04.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.750+0000 7f895cedd640 1 -- 172.21.15.98:0/807347660 learned_addr learned my addr 172.21.15.98:0/807347660 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:04.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.750+0000 7f895cedd640 1 -- 172.21.15.98:0/807347660 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632baa04440 msgr2=0x5632baa1c180 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:53:04.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.750+0000 7f895cedd640 1 --2- 172.21.15.98:0/807347660 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632baa04440 0x5632baa1c180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.750+0000 7f895cedd640 1 -- 172.21.15.98:0/807347660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9260 msgr2=0x5632baa10020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:04.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.750+0000 7f895cedd640 1 --2- 172.21.15.98:0/807347660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9260 0x5632baa10020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.750+0000 7f895cedd640 1 -- 172.21.15.98:0/807347660 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632baa06830 con 0x5632ba9f9b80 2024-05-09T08:53:04.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.751+0000 7f895cedd640 1 --2- 172.21.15.98:0/807347660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9b80 0x5632baa01bd0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f8954000ea0 tx=0x7f8954004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93931ae76f4ca056 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:04.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.752+0000 7f89427fc640 1 -- 172.21.15.98:0/807347660 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954018040 con 0x5632ba9f9b80 2024-05-09T08:53:04.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.752+0000 7f89427fc640 1 -- 172.21.15.98:0/807347660 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8954004af0 con 0x5632ba9f9b80 2024-05-09T08:53:04.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.752+0000 7f89427fc640 1 -- 172.21.15.98:0/807347660 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954004540 con 0x5632ba9f9b80 2024-05-09T08:53:04.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.754+0000 7f896264f740 1 -- 172.21.15.98:0/807347660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9b80 msgr2=0x5632baa01bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:04.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.754+0000 7f896264f740 1 --2- 172.21.15.98:0/807347660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9b80 0x5632baa01bd0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f8954000ea0 tx=0x7f8954004f90 comp rx=0 tx=0).stop 2024-05-09T08:53:04.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.755+0000 7f896264f740 1 -- 172.21.15.98:0/807347660 shutdown_connections 2024-05-09T08:53:04.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.755+0000 7f896264f740 1 --2- 172.21.15.98:0/807347660 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632baa04440 0x5632baa1c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.755+0000 7f896264f740 1 --2- 172.21.15.98:0/807347660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9b80 0x5632baa01bd0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.755+0000 7f896264f740 1 --2- 172.21.15.98:0/807347660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ba9f9260 0x5632baa10020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.755+0000 7f896264f740 1 -- 172.21.15.98:0/807347660 >> 172.21.15.98:0/807347660 conn(0x5632ba9f5770 msgr2=0x5632ba9fb570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:04.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.760+0000 7f896264f740 1 -- 172.21.15.98:0/807347660 shutdown_connections 2024-05-09T08:53:04.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.760+0000 7f896264f740 1 -- 172.21.15.98:0/807347660 wait complete. 2024-05-09T08:53:04.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.761+0000 7f896264f740 1 -- start start 2024-05-09T08:53:04.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.761+0000 7f896264f740 1 Processor -- start 2024-05-09T08:53:04.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.762+0000 7f896264f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9260 0x5632baa76430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:04.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.762+0000 7f896264f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ba9f9b80 0x5632baa0c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:04.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.762+0000 7f896264f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632baa04440 0x5632ba9fe420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:04.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.762+0000 7f896264f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89581bf260 con 0x5632ba9f9260 2024-05-09T08:53:04.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.762+0000 7f896264f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89581beb90 con 0x5632baa04440 2024-05-09T08:53:04.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.762+0000 7f896264f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f89581bbd70 con 0x5632ba9f9b80 2024-05-09T08:53:04.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.762+0000 7f896264f740 10 client.0 _ll_register_callbacks cb 0x7f8950005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:04.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.764+0000 7f895d6de640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9260 0x5632baa76430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:04.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.764+0000 7f895d6de640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9260 0x5632baa76430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52026/0 (socket says 172.21.15.98:52026) 2024-05-09T08:53:04.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.764+0000 7f895d6de640 1 -- 172.21.15.98:0/4511477 learned_addr learned my addr 172.21.15.98:0/4511477 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:04.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.765+0000 7f895dedf640 1 --2- 172.21.15.98:0/4511477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632baa04440 0x5632ba9fe420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:04.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.766+0000 7f895d6de640 1 -- 172.21.15.98:0/4511477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ba9f9b80 msgr2=0x5632baa0c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:04.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.766+0000 7f895d6de640 1 --2- 172.21.15.98:0/4511477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ba9f9b80 0x5632baa0c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.766+0000 7f895d6de640 1 -- 172.21.15.98:0/4511477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632baa04440 msgr2=0x5632ba9fe420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:04.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.766+0000 7f895d6de640 1 --2- 172.21.15.98:0/4511477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632baa04440 0x5632ba9fe420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.766+0000 7f895d6de640 1 -- 172.21.15.98:0/4511477 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632ba9fe960 con 0x5632ba9f9260 2024-05-09T08:53:04.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.766+0000 7f895d6de640 1 --2- 172.21.15.98:0/4511477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9260 0x5632baa76430 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f894800ee10 tx=0x7f89480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85372e923eca8064 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:04.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.767+0000 7f8937fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:04.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.768+0000 7f8937fff640 1 -- 172.21.15.98:0/4511477 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8948029070 con 0x5632ba9f9260 2024-05-09T08:53:04.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.768+0000 7f896264f740 1 -- 172.21.15.98:0/4511477 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8954004af0 con 0x5632ba9f9260 2024-05-09T08:53:04.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.768+0000 7f8937fff640 1 -- 172.21.15.98:0/4511477 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f894800f040 con 0x5632ba9f9260 2024-05-09T08:53:04.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.770+0000 7f8937fff640 1 -- 172.21.15.98:0/4511477 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8948037050 con 0x5632ba9f9260 2024-05-09T08:53:04.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.771+0000 7f896264f740 10 client.16374 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:04.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.771+0000 7f896264f740 1 -- 172.21.15.98:0/4511477 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5632baa8c180 con 0x5632ba9f9260 2024-05-09T08:53:04.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.773+0000 7f8937fff640 1 -- 172.21.15.98:0/4511477 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8954004af0 con 0x5632ba9f9260 2024-05-09T08:53:04.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.773+0000 7f8937fff640 1 -- 172.21.15.98:0/4511477 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8948015c70 con 0x5632ba9f9260 2024-05-09T08:53:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.773+0000 7f896264f740 10 client.16374 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.773+0000 7f896264f740 10 client.16374 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.773+0000 7f896264f740 20 client.16374 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.773+0000 7f896264f740 1 --2- 172.21.15.98:0/4511477 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5632baa8c8e0 0x5632baa8ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.773+0000 7f896264f740 4 client.16374 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:53:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.773+0000 7f896264f740 1 -- 172.21.15.98:0/4511477 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5632ba966780 con 0x5632baa8c8e0 2024-05-09T08:53:04.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.775+0000 7f895cedd640 1 --2- 172.21.15.98:0/4511477 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5632baa8c8e0 0x5632baa8ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.783+0000 7f895cedd640 1 --2- 172.21.15.98:0/4511477 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5632baa8c8e0 0x5632baa8ed00 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b46ccfadd5c71a8 server_cookie=721b2e46e48b72d0 in_seq=0 out_seq=0 2024-05-09T08:53:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.783+0000 7f8937fff640 10 client.16374 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:04.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.785+0000 7f8937fff640 1 -- 172.21.15.98:0/4511477 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5632ba966780 con 0x5632baa8c8e0 2024-05-09T08:53:04.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.785+0000 7f8937fff640 10 client.16374 handle_command_reply: tid=0 2024-05-09T08:53:04.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.786+0000 7f896264f740 1 client.16374 shutdown 2024-05-09T08:53:04.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.792+0000 7f896264f740 1 -- 172.21.15.98:0/4511477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9260 msgr2=0x5632baa76430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:04.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.792+0000 7f896264f740 1 --2- 172.21.15.98:0/4511477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9260 0x5632baa76430 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f894800ee10 tx=0x7f89480046c0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.798+0000 7f896264f740 1 -- 172.21.15.98:0/4511477 shutdown_connections 2024-05-09T08:53:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.798+0000 7f896264f740 1 --2- 172.21.15.98:0/4511477 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5632baa8c8e0 0x5632baa8ed00 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.798+0000 7f896264f740 1 --2- 172.21.15.98:0/4511477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632baa04440 0x5632ba9fe420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.798+0000 7f896264f740 1 --2- 172.21.15.98:0/4511477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ba9f9b80 0x5632baa0c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.798+0000 7f896264f740 1 --2- 172.21.15.98:0/4511477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ba9f9260 0x5632baa76430 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.798+0000 7f896264f740 1 -- 172.21.15.98:0/4511477 >> 172.21.15.98:0/4511477 conn(0x5632ba9f5770 msgr2=0x5632ba9f4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.799+0000 7f8937fff640 0 client.16374 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:04.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.803+0000 7f896264f740 1 -- 172.21.15.98:0/4511477 shutdown_connections 2024-05-09T08:53:04.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.803+0000 7f896264f740 1 -- 172.21.15.98:0/4511477 wait complete. 2024-05-09T08:53:04.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.803+0000 7f896264f740 20 client.16374 trim_cache size 0 max 16384 2024-05-09T08:53:04.806 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:53:04.806 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:53:04.806 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:53:04.806 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:53:04.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.807+0000 7f8943fff640 1 -- 172.21.15.98:0/2327378446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89440a2f10 msgr2=0x7f89440a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:04.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.807+0000 7f8943fff640 1 --2- 172.21.15.98:0/2327378446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89440a2f10 0x7f89440a53d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x5632ba9faa10 tx=0x7f89480047d0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.807+0000 7f8943fff640 1 -- 172.21.15.98:0/2327378446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89581116e0 msgr2=0x7f8958111b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:04.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.807+0000 7f8943fff640 1 --2- 172.21.15.98:0/2327378446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89581116e0 0x7f8958111b90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8950031690 tx=0x7f89500316c0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.808+0000 7f8943fff640 1 -- 172.21.15.98:0/2327378446 shutdown_connections 2024-05-09T08:53:04.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.808+0000 7f8943fff640 1 --2- 172.21.15.98:0/2327378446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89440a2f10 0x7f89440a53d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.808+0000 7f8943fff640 1 --2- 172.21.15.98:0/2327378446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89581116e0 0x7f8958111b90 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.808+0000 7f8943fff640 1 --2- 172.21.15.98:0/2327378446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8958106bc0 0x7f89581111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.808+0000 7f8943fff640 1 --2- 172.21.15.98:0/2327378446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89580fffb0 0x7f8958117090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:04.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.808+0000 7f8943fff640 1 -- 172.21.15.98:0/2327378446 >> 172.21.15.98:0/2327378446 conn(0x7f8958076cd0 msgr2=0x7f8958104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:04.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.808+0000 7f8943fff640 1 -- 172.21.15.98:0/2327378446 shutdown_connections 2024-05-09T08:53:04.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:04.808+0000 7f8943fff640 1 -- 172.21.15.98:0/2327378446 wait complete. 2024-05-09T08:53:04.831 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:53:04.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:53:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.059+0000 7f865cd92640 1 -- 172.21.15.98:0/2577227550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86580fff50 msgr2=0x7f86581003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.059+0000 7f865cd92640 1 --2- 172.21.15.98:0/2577227550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86580fff50 0x7f86581003d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f864c009900 tx=0x7f864c02a410 comp rx=0 tx=0).stop 2024-05-09T08:53:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.059+0000 7f865cd92640 1 -- 172.21.15.98:0/2577227550 shutdown_connections 2024-05-09T08:53:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.059+0000 7f865cd92640 1 --2- 172.21.15.98:0/2577227550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8658100910 0x7f865810e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.059+0000 7f865cd92640 1 --2- 172.21.15.98:0/2577227550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86580fff50 0x7f86581003d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.059+0000 7f865cd92640 1 --2- 172.21.15.98:0/2577227550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f86581067b0 0x7f8658106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.059+0000 7f865cd92640 1 -- 172.21.15.98:0/2577227550 >> 172.21.15.98:0/2577227550 conn(0x7f86580758a0 msgr2=0x7f8658075cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.062+0000 7f865cd92640 1 -- 172.21.15.98:0/2577227550 shutdown_connections 2024-05-09T08:53:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.063+0000 7f865cd92640 1 -- 172.21.15.98:0/2577227550 wait complete. 2024-05-09T08:53:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.064+0000 7f865cd92640 1 Processor -- start 2024-05-09T08:53:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.064+0000 7f865cd92640 1 -- start start 2024-05-09T08:53:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.064+0000 7f865cd92640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f86580fff50 0x7f8658072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.064+0000 7f865cd92640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8658100910 0x7f86580734b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.064+0000 7f865cd92640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86581067b0 0x7f865806c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.064+0000 7f865cd92640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8658111a50 con 0x7f8658100910 2024-05-09T08:53:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.064+0000 7f865cd92640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f86581118d0 con 0x7f86580fff50 2024-05-09T08:53:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.064+0000 7f865cd92640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8658111bd0 con 0x7f86581067b0 2024-05-09T08:53:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.065+0000 7f8656ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8658100910 0x7f86580734b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.065+0000 7f8656ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8658100910 0x7f86580734b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52054/0 (socket says 172.21.15.98:52054) 2024-05-09T08:53:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.065+0000 7f8656ffd640 1 -- 172.21.15.98:0/4195294583 learned_addr learned my addr 172.21.15.98:0/4195294583 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.066+0000 7f8656ffd640 1 -- 172.21.15.98:0/4195294583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86581067b0 msgr2=0x7f865806c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.066+0000 7f8656ffd640 1 --2- 172.21.15.98:0/4195294583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86581067b0 0x7f865806c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.066+0000 7f8656ffd640 1 -- 172.21.15.98:0/4195294583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f86580fff50 msgr2=0x7f8658072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.066+0000 7f8656ffd640 1 --2- 172.21.15.98:0/4195294583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f86580fff50 0x7f8658072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.066+0000 7f8656ffd640 1 -- 172.21.15.98:0/4195294583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865806ca90 con 0x7f8658100910 2024-05-09T08:53:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.067+0000 7f8656ffd640 1 --2- 172.21.15.98:0/4195294583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8658100910 0x7f86580734b0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f864c006010 tx=0x7f864c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.067+0000 7f8654ff9640 1 -- 172.21.15.98:0/4195294583 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c040070 con 0x7f8658100910 2024-05-09T08:53:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.068+0000 7f8654ff9640 1 -- 172.21.15.98:0/4195294583 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f864c031c80 con 0x7f8658100910 2024-05-09T08:53:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.069+0000 7f865cd92640 1 -- 172.21.15.98:0/4195294583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f865806cd20 con 0x7f8658100910 2024-05-09T08:53:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.069+0000 7f8654ff9640 1 -- 172.21.15.98:0/4195294583 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c040070 con 0x7f8658100910 2024-05-09T08:53:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.069+0000 7f865cd92640 1 -- 172.21.15.98:0/4195294583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8658070c70 con 0x7f8658100910 2024-05-09T08:53:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.070+0000 7f865cd92640 1 -- 172.21.15.98:0/4195294583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8658069970 con 0x7f8658100910 2024-05-09T08:53:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.075+0000 7f8654ff9640 1 -- 172.21.15.98:0/4195294583 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f864c043070 con 0x7f8658100910 2024-05-09T08:53:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.077+0000 7f8654ff9640 1 --2- 172.21.15.98:0/4195294583 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f86400a2f00 0x7f86400a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.077+0000 7f8654ff9640 1 -- 172.21.15.98:0/4195294583 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f864c04de30 con 0x7f8658100910 2024-05-09T08:53:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.080+0000 7f86577fe640 1 --2- 172.21.15.98:0/4195294583 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f86400a2f00 0x7f86400a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.082+0000 7f86577fe640 1 --2- 172.21.15.98:0/4195294583 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f86400a2f00 0x7f86400a53c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f86480058c0 tx=0x7f8648005850 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.086+0000 7f8654ff9640 1 -- 172.21.15.98:0/4195294583 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f864c0bd1e0 con 0x7f8658100910 2024-05-09T08:53:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.554+0000 7f865cd92640 1 -- 172.21.15.98:0/4195294583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f865806d520 con 0x7f8658100910 2024-05-09T08:53:05.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.556+0000 7f8654ff9640 1 -- 172.21.15.98:0/4195294583 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11944 (secure 0 0 0) 0x7f864c0bd1e0 con 0x7f8658100910 2024-05-09T08:53:05.556 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:53:05.556 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":25736,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3228987570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3228987570},{"type":"v1","addr":"172.21.15.105:6835","nonce":3228987570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:52:47.790415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14568":{"gid":14568,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24439,14667,24388,24461,24457]},"id":1}]} 2024-05-09T08:53:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 31 2024-05-09T08:53:05.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.560+0000 7f865cd92640 1 -- 172.21.15.98:0/4195294583 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f86400a2f00 msgr2=0x7f86400a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.560+0000 7f865cd92640 1 --2- 172.21.15.98:0/4195294583 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f86400a2f00 0x7f86400a53c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f86480058c0 tx=0x7f8648005850 comp rx=0 tx=0).stop 2024-05-09T08:53:05.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.560+0000 7f865cd92640 1 -- 172.21.15.98:0/4195294583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8658100910 msgr2=0x7f86580734b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.560+0000 7f865cd92640 1 --2- 172.21.15.98:0/4195294583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8658100910 0x7f86580734b0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f864c006010 tx=0x7f864c0045e0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.561+0000 7f865cd92640 1 -- 172.21.15.98:0/4195294583 shutdown_connections 2024-05-09T08:53:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.561+0000 7f865cd92640 1 --2- 172.21.15.98:0/4195294583 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f86400a2f00 0x7f86400a53c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.561+0000 7f865cd92640 1 --2- 172.21.15.98:0/4195294583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86581067b0 0x7f865806c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.561+0000 7f865cd92640 1 --2- 172.21.15.98:0/4195294583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8658100910 0x7f86580734b0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.561+0000 7f865cd92640 1 --2- 172.21.15.98:0/4195294583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f86580fff50 0x7f8658072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.561+0000 7f865cd92640 1 -- 172.21.15.98:0/4195294583 >> 172.21.15.98:0/4195294583 conn(0x7f86580758a0 msgr2=0x7f86580fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.562+0000 7f865cd92640 1 -- 172.21.15.98:0/4195294583 shutdown_connections 2024-05-09T08:53:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.562+0000 7f865cd92640 1 -- 172.21.15.98:0/4195294583 wait complete. 2024-05-09T08:53:05.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:53:05.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.761+0000 7f552e909640 1 -- 172.21.15.98:0/3942856009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55281011c0 msgr2=0x7f5528105680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.761+0000 7f552e909640 1 --2- 172.21.15.98:0/3942856009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55281011c0 0x7f5528105680 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f551c009900 tx=0x7f551c02a510 comp rx=0 tx=0).stop 2024-05-09T08:53:05.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.761+0000 7f552e909640 1 -- 172.21.15.98:0/3942856009 shutdown_connections 2024-05-09T08:53:05.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.761+0000 7f552e909640 1 --2- 172.21.15.98:0/3942856009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528105bc0 0x7f5528106050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.761+0000 7f552e909640 1 --2- 172.21.15.98:0/3942856009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55281011c0 0x7f5528105680 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.761+0000 7f552e909640 1 --2- 172.21.15.98:0/3942856009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55281008a0 0x7f5528100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.761+0000 7f552e909640 1 -- 172.21.15.98:0/3942856009 >> 172.21.15.98:0/3942856009 conn(0x7f5528077830 msgr2=0x7f5528077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:05.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.761+0000 7f552e909640 1 -- 172.21.15.98:0/3942856009 shutdown_connections 2024-05-09T08:53:05.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.761+0000 7f552e909640 1 -- 172.21.15.98:0/3942856009 wait complete. 2024-05-09T08:53:05.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.762+0000 7f552e909640 1 Processor -- start 2024-05-09T08:53:05.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.762+0000 7f552e909640 1 -- start start 2024-05-09T08:53:05.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.762+0000 7f552e909640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55281008a0 0x7f552819e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.762+0000 7f552e909640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55281011c0 0x7f552819eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.762+0000 7f552e909640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5528105bc0 0x7f55281a3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.762+0000 7f552e909640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5528118280 con 0x7f55281008a0 2024-05-09T08:53:05.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.762+0000 7f552e909640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5528118100 con 0x7f55281011c0 2024-05-09T08:53:05.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.762+0000 7f552e909640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5528118400 con 0x7f5528105bc0 2024-05-09T08:53:05.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.763+0000 7f552cea6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5528105bc0 0x7f55281a3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.763+0000 7f552cea6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5528105bc0 0x7f55281a3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:59420/0 (socket says 172.21.15.98:59420) 2024-05-09T08:53:05.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.763+0000 7f552cea6640 1 -- 172.21.15.98:0/3682908689 learned_addr learned my addr 172.21.15.98:0/3682908689 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:05.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.763+0000 7f5527fff640 1 --2- 172.21.15.98:0/3682908689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55281008a0 0x7f552819e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.763+0000 7f552cea6640 1 -- 172.21.15.98:0/3682908689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55281011c0 msgr2=0x7f552819eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.763+0000 7f552cea6640 1 --2- 172.21.15.98:0/3682908689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55281011c0 0x7f552819eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.763+0000 7f552cea6640 1 -- 172.21.15.98:0/3682908689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55281008a0 msgr2=0x7f552819e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.763+0000 7f552cea6640 1 --2- 172.21.15.98:0/3682908689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55281008a0 0x7f552819e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.763+0000 7f552cea6640 1 -- 172.21.15.98:0/3682908689 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55281a3750 con 0x7f5528105bc0 2024-05-09T08:53:05.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.763+0000 7f552cea6640 1 --2- 172.21.15.98:0/3682908689 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5528105bc0 0x7f55281a3020 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5518009fd0 tx=0x7f551800b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:05.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.764+0000 7f55257fa640 1 -- 172.21.15.98:0/3682908689 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5518011830 con 0x7f5528105bc0 2024-05-09T08:53:05.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.764+0000 7f55257fa640 1 -- 172.21.15.98:0/3682908689 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5518016070 con 0x7f5528105bc0 2024-05-09T08:53:05.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.764+0000 7f55257fa640 1 -- 172.21.15.98:0/3682908689 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551800f2e0 con 0x7f5528105bc0 2024-05-09T08:53:05.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.764+0000 7f552e909640 1 -- 172.21.15.98:0/3682908689 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55281a3a40 con 0x7f5528105bc0 2024-05-09T08:53:05.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.764+0000 7f552e909640 1 -- 172.21.15.98:0/3682908689 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55281ab2d0 con 0x7f5528105bc0 2024-05-09T08:53:05.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.767+0000 7f553160f740 1 Processor -- start 2024-05-09T08:53:05.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.768+0000 7f55257fa640 1 -- 172.21.15.98:0/3682908689 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f55180119d0 con 0x7f5528105bc0 2024-05-09T08:53:05.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.768+0000 7f553160f740 1 -- start start 2024-05-09T08:53:05.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.769+0000 7f553160f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adaf5330 0x55d9adaf5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.769+0000 7f553160f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adaffde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.769+0000 7f553160f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adb00320 0x55d9adb02710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f553160f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d9ada5ac40 con 0x55d9adb00320 2024-05-09T08:53:05.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f553160f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d9ada02580 con 0x55d9adaf5c50 2024-05-09T08:53:05.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f553160f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d9ada56c00 con 0x55d9adaf5330 2024-05-09T08:53:05.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f55257fa640 1 --2- 172.21.15.98:0/3682908689 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f55100a2e50 0x7f55100a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f55277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adb00320 0x55d9adb02710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f55277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adb00320 0x55d9adb02710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53258/0 (socket says 172.21.15.98:53258) 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f55277fe640 1 -- 172.21.15.98:0/1844622700 learned_addr learned my addr 172.21.15.98:0/1844622700 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f5527fff640 1 --2- 172.21.15.98:0/1844622700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adaffde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f55277fe640 1 --2- 172.21.15.98:0/1844622700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adaf5330 0x55d9adaf5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f55257fa640 1 -- 172.21.15.98:0/3682908689 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f55180cee10 con 0x7f5528105bc0 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f552cea6640 1 --2- 172.21.15.98:0/3682908689 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f55100a2e50 0x7f55100a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f55277fe640 1 -- 172.21.15.98:0/1844622700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adaf5330 msgr2=0x55d9adaf5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f55277fe640 1 --2- 172.21.15.98:0/1844622700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adaf5330 0x55d9adaf5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f55277fe640 1 -- 172.21.15.98:0/1844622700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 msgr2=0x55d9adaffde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f55277fe640 1 --2- 172.21.15.98:0/1844622700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adaffde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.770+0000 7f55277fe640 1 -- 172.21.15.98:0/1844622700 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9ad76e3f0 con 0x55d9adb00320 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.771+0000 7f55277fe640 1 --2- 172.21.15.98:0/1844622700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adb00320 0x55d9adb02710 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f551c005c20 tx=0x7f551c005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994b7bd8f0344083 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.771+0000 7f550e7fc640 1 -- 172.21.15.98:0/1844622700 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c035070 con 0x55d9adb00320 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.772+0000 7f550e7fc640 1 -- 172.21.15.98:0/1844622700 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f551c031cd0 con 0x55d9adb00320 2024-05-09T08:53:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.772+0000 7f550e7fc640 1 -- 172.21.15.98:0/1844622700 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c003d80 con 0x55d9adb00320 2024-05-09T08:53:05.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.772+0000 7f553160f740 1 -- 172.21.15.98:0/1844622700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adb00320 msgr2=0x55d9adb02710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.772+0000 7f552cea6640 1 --2- 172.21.15.98:0/3682908689 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f55100a2e50 0x7f55100a5310 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5518097480 tx=0x7f551800bf90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:05.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.772+0000 7f553160f740 1 --2- 172.21.15.98:0/1844622700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adb00320 0x55d9adb02710 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f551c005c20 tx=0x7f551c005b70 comp rx=0 tx=0).stop 2024-05-09T08:53:05.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.773+0000 7f553160f740 1 -- 172.21.15.98:0/1844622700 shutdown_connections 2024-05-09T08:53:05.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.773+0000 7f553160f740 1 --2- 172.21.15.98:0/1844622700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adb00320 0x55d9adb02710 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.773+0000 7f553160f740 1 --2- 172.21.15.98:0/1844622700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adaffde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.773+0000 7f553160f740 1 --2- 172.21.15.98:0/1844622700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adaf5330 0x55d9adaf5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.773+0000 7f553160f740 1 -- 172.21.15.98:0/1844622700 >> 172.21.15.98:0/1844622700 conn(0x55d9adaf1650 msgr2=0x55d9adaf1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:05.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.773+0000 7f553160f740 1 -- 172.21.15.98:0/1844622700 shutdown_connections 2024-05-09T08:53:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.773+0000 7f553160f740 1 -- 172.21.15.98:0/1844622700 wait complete. 2024-05-09T08:53:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.773+0000 7f553160f740 1 -- start start 2024-05-09T08:53:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.773+0000 7f553160f740 1 Processor -- start 2024-05-09T08:53:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f553160f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adafd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f553160f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adafd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f553160f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adb00320 0x55d9adafddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f553160f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d9adb030b0 con 0x55d9adaf5330 2024-05-09T08:53:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f553160f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d9ada15c40 con 0x55d9adaf5c50 2024-05-09T08:53:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f553160f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d9adb03230 con 0x55d9adb00320 2024-05-09T08:53:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f55277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adafd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f55277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adafd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53272/0 (socket says 172.21.15.98:53272) 2024-05-09T08:53:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f55277fe640 1 -- 172.21.15.98:0/12281241 learned_addr learned my addr 172.21.15.98:0/12281241 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f553160f740 10 client.0 _ll_register_callbacks cb 0x55d9ad76dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f5527fff640 1 --2- 172.21.15.98:0/12281241 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adafd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f55277fe640 1 --2- 172.21.15.98:0/12281241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adb00320 0x55d9adafddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f55277fe640 1 -- 172.21.15.98:0/12281241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adb00320 msgr2=0x55d9adafddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f55277fe640 1 --2- 172.21.15.98:0/12281241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adb00320 0x55d9adafddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f55277fe640 1 -- 172.21.15.98:0/12281241 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 msgr2=0x55d9adafd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f55277fe640 1 --2- 172.21.15.98:0/12281241 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adafd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.774+0000 7f55277fe640 1 -- 172.21.15.98:0/12281241 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9ad8387d0 con 0x55d9adaf5330 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.775+0000 7f55277fe640 1 --2- 172.21.15.98:0/12281241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adafd370 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f551c02aa30 tx=0x7f551c003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12e41e2e319fe25 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.775+0000 7f55077fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.775+0000 7f55077fe640 1 -- 172.21.15.98:0/12281241 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c051060 con 0x55d9adaf5330 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.775+0000 7f553160f740 1 -- 172.21.15.98:0/12281241 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f551c031cd0 con 0x55d9adaf5330 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.775+0000 7f55077fe640 1 -- 172.21.15.98:0/12281241 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f551c003d10 con 0x55d9adaf5330 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.775+0000 7f55077fe640 1 -- 172.21.15.98:0/12281241 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f551c058050 con 0x55d9adaf5330 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.775+0000 7f553160f740 10 client.16392 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.775+0000 7f553160f740 1 -- 172.21.15.98:0/12281241 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d9adb0f350 con 0x55d9adaf5330 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.775+0000 7f55077fe640 1 -- 172.21.15.98:0/12281241 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f551c031cd0 con 0x55d9adaf5330 2024-05-09T08:53:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.776+0000 7f55077fe640 1 -- 172.21.15.98:0/12281241 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f551c046070 con 0x55d9adaf5330 2024-05-09T08:53:05.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.776+0000 7f553160f740 10 client.16392 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:05.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.776+0000 7f553160f740 10 client.16392 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:53:05.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.776+0000 7f553160f740 20 client.16392 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:05.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.776+0000 7f553160f740 1 --2- 172.21.15.98:0/12281241 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d9adb10850 0x55d9adb12cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.776+0000 7f553160f740 4 client.16392 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:05.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.776+0000 7f553160f740 1 -- 172.21.15.98:0/12281241 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9ada68370 con 0x55d9adb10850 2024-05-09T08:53:05.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.776+0000 7f5527fff640 1 --2- 172.21.15.98:0/12281241 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d9adb10850 0x55d9adb12cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.777+0000 7f5527fff640 1 --2- 172.21.15.98:0/12281241 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d9adb10850 0x55d9adb12cb0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f48e5fb38c245160 server_cookie=c4d01c1d628e51c0 in_seq=0 out_seq=0 2024-05-09T08:53:05.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.777+0000 7f55077fe640 10 client.16392 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:53:05.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.777+0000 7f55077fe640 1 -- 172.21.15.98:0/12281241 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d9ada68370 con 0x55d9adb10850 2024-05-09T08:53:05.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.777+0000 7f55077fe640 10 client.16392 handle_command_reply: tid=0 2024-05-09T08:53:05.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.778+0000 7f553160f740 1 client.16392 shutdown 2024-05-09T08:53:05.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.778+0000 7f553160f740 1 -- 172.21.15.98:0/12281241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 msgr2=0x55d9adafd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.778+0000 7f553160f740 1 --2- 172.21.15.98:0/12281241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adafd370 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f551c02aa30 tx=0x7f551c003960 comp rx=0 tx=0).stop 2024-05-09T08:53:05.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.779+0000 7f553160f740 1 -- 172.21.15.98:0/12281241 shutdown_connections 2024-05-09T08:53:05.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.779+0000 7f553160f740 1 --2- 172.21.15.98:0/12281241 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d9adb10850 0x55d9adb12cb0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.779+0000 7f553160f740 1 --2- 172.21.15.98:0/12281241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adb00320 0x55d9adafddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.779+0000 7f553160f740 1 --2- 172.21.15.98:0/12281241 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adafd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.779+0000 7f55077fe640 0 client.16392 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:53:05.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.779+0000 7f553160f740 1 --2- 172.21.15.98:0/12281241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adafd370 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.779+0000 7f553160f740 1 -- 172.21.15.98:0/12281241 >> 172.21.15.98:0/12281241 conn(0x55d9adaf1650 msgr2=0x55d9adaf7450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:05.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.779+0000 7f553160f740 1 -- 172.21.15.98:0/12281241 shutdown_connections 2024-05-09T08:53:05.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.779+0000 7f553160f740 1 -- 172.21.15.98:0/12281241 wait complete. 2024-05-09T08:53:05.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.779+0000 7f553160f740 20 client.16392 trim_cache size 0 max 16384 2024-05-09T08:53:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.800+0000 7f553160f740 1 Processor -- start 2024-05-09T08:53:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.800+0000 7f553160f740 1 -- start start 2024-05-09T08:53:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.800+0000 7f553160f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adaffaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.800+0000 7f553160f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adb0bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.800+0000 7f553160f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adb00320 0x55d9adaf7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.800+0000 7f553160f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55281b53a0 con 0x55d9adaf5330 2024-05-09T08:53:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.800+0000 7f553160f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f55281b6ee0 con 0x55d9adaf5c50 2024-05-09T08:53:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.800+0000 7f553160f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f55281b5a70 con 0x55d9adb00320 2024-05-09T08:53:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.800+0000 7f55277fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adb00320 0x55d9adaf7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.801+0000 7f55277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adaffaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.801+0000 7f55277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adaffaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53282/0 (socket says 172.21.15.98:53282) 2024-05-09T08:53:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.801+0000 7f55277fe640 1 -- 172.21.15.98:0/2758943630 learned_addr learned my addr 172.21.15.98:0/2758943630 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:05.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.801+0000 7f5527fff640 1 --2- 172.21.15.98:0/2758943630 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adb0bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.801+0000 7f55277fe640 1 -- 172.21.15.98:0/2758943630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adb00320 msgr2=0x55d9adaf7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:05.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.801+0000 7f55277fe640 1 --2- 172.21.15.98:0/2758943630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adb00320 0x55d9adaf7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.801+0000 7f55277fe640 1 -- 172.21.15.98:0/2758943630 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 msgr2=0x55d9adb0bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:05.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.801+0000 7f55277fe640 1 --2- 172.21.15.98:0/2758943630 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adb0bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.801+0000 7f55277fe640 1 -- 172.21.15.98:0/2758943630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9adaecf30 con 0x55d9adaf5330 2024-05-09T08:53:05.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.802+0000 7f55277fe640 1 --2- 172.21.15.98:0/2758943630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adaffaa0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f551c009230 tx=0x7f551c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abbbba7c34b41930 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:05.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.802+0000 7f550dffb640 1 -- 172.21.15.98:0/2758943630 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c051070 con 0x55d9adaf5330 2024-05-09T08:53:05.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.802+0000 7f550dffb640 1 -- 172.21.15.98:0/2758943630 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f551c049e50 con 0x55d9adaf5330 2024-05-09T08:53:05.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.802+0000 7f550dffb640 1 -- 172.21.15.98:0/2758943630 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c035040 con 0x55d9adaf5330 2024-05-09T08:53:05.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.803+0000 7f553160f740 1 -- 172.21.15.98:0/2758943630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 msgr2=0x55d9adaffaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.803+0000 7f553160f740 1 --2- 172.21.15.98:0/2758943630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adaffaa0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f551c009230 tx=0x7f551c000bc0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.804+0000 7f553160f740 1 -- 172.21.15.98:0/2758943630 shutdown_connections 2024-05-09T08:53:05.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.804+0000 7f553160f740 1 --2- 172.21.15.98:0/2758943630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adb00320 0x55d9adaf7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.804+0000 7f553160f740 1 --2- 172.21.15.98:0/2758943630 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adb0bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.804+0000 7f553160f740 1 --2- 172.21.15.98:0/2758943630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adaffaa0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.804+0000 7f553160f740 1 -- 172.21.15.98:0/2758943630 >> 172.21.15.98:0/2758943630 conn(0x55d9adaf1650 msgr2=0x55d9adaf1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:05.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.805+0000 7f553160f740 1 -- 172.21.15.98:0/2758943630 shutdown_connections 2024-05-09T08:53:05.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.806+0000 7f553160f740 1 -- 172.21.15.98:0/2758943630 wait complete. 2024-05-09T08:53:05.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.806+0000 7f553160f740 1 -- start start 2024-05-09T08:53:05.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.806+0000 7f553160f740 1 Processor -- start 2024-05-09T08:53:05.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.807+0000 7f553160f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adb08480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.807+0000 7f553160f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adb089c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.807+0000 7f553160f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adb00320 0x55d9adaf9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.807+0000 7f553160f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55281b6810 con 0x55d9adaf5330 2024-05-09T08:53:05.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.807+0000 7f553160f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f55281b6140 con 0x55d9adaf5c50 2024-05-09T08:53:05.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.807+0000 7f553160f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f55281b3380 con 0x55d9adb00320 2024-05-09T08:53:05.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.807+0000 7f553160f740 10 client.0 _ll_register_callbacks cb 0x7f55200099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:05.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.807+0000 7f5527fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adb089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.807+0000 7f5527fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adb089c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:49272/0 (socket says 172.21.15.98:49272) 2024-05-09T08:53:05.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.807+0000 7f5527fff640 1 -- 172.21.15.98:0/3865326698 learned_addr learned my addr 172.21.15.98:0/3865326698 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.807+0000 7f5527fff640 1 -- 172.21.15.98:0/3865326698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adb00320 msgr2=0x55d9adaf9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.808+0000 7f5527fff640 1 --2- 172.21.15.98:0/3865326698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adb00320 0x55d9adaf9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.808+0000 7f5527fff640 1 -- 172.21.15.98:0/3865326698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 msgr2=0x55d9adb08480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.808+0000 7f5527fff640 1 --2- 172.21.15.98:0/3865326698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adb08480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.808+0000 7f5527fff640 1 -- 172.21.15.98:0/3865326698 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9adafa070 con 0x55d9adaf5c50 2024-05-09T08:53:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.808+0000 7f5527fff640 1 --2- 172.21.15.98:0/3865326698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adb089c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55d9adaf68f0 tx=0x7f55200138f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fca0901e302d1996 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.808+0000 7f55077fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:53:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.808+0000 7f55077fe640 1 -- 172.21.15.98:0/3865326698 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5520020410 con 0x55d9adaf5c50 2024-05-09T08:53:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.808+0000 7f553160f740 1 -- 172.21.15.98:0/3865326698 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f551c049e50 con 0x55d9adaf5c50 2024-05-09T08:53:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.808+0000 7f55077fe640 1 -- 172.21.15.98:0/3865326698 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f55200205b0 con 0x55d9adaf5c50 2024-05-09T08:53:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.808+0000 7f55077fe640 1 -- 172.21.15.98:0/3865326698 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5520030050 con 0x55d9adaf5c50 2024-05-09T08:53:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.808+0000 7f553160f740 10 client.25757 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.808+0000 7f553160f740 1 -- 172.21.15.98:0/3865326698 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d9adb769a0 con 0x55d9adaf5c50 2024-05-09T08:53:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.809+0000 7f55077fe640 1 -- 172.21.15.98:0/3865326698 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f551c049e50 con 0x55d9adaf5c50 2024-05-09T08:53:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.809+0000 7f55077fe640 1 -- 172.21.15.98:0/3865326698 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5520011440 con 0x55d9adaf5c50 2024-05-09T08:53:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.809+0000 7f553160f740 10 client.25757 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.809+0000 7f553160f740 10 client.25757 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:53:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.809+0000 7f553160f740 20 client.25757 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.810+0000 7f553160f740 1 --2- 172.21.15.98:0/3865326698 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d9adb10850 0x55d9adb791e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.810+0000 7f553160f740 4 client.25757 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:53:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.810+0000 7f553160f740 1 -- 172.21.15.98:0/3865326698 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d9ada626c0 con 0x55d9adb10850 2024-05-09T08:53:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.810+0000 7f55277fe640 1 --2- 172.21.15.98:0/3865326698 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d9adb10850 0x55d9adb791e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.811+0000 7f55277fe640 1 --2- 172.21.15.98:0/3865326698 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d9adb10850 0x55d9adb791e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c6b250a6aed3dfee server_cookie=d85331339a3d01c5 in_seq=0 out_seq=0 2024-05-09T08:53:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.811+0000 7f55077fe640 10 client.25757 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:53:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.811+0000 7f55077fe640 1 -- 172.21.15.98:0/3865326698 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d9ada626c0 con 0x55d9adb10850 2024-05-09T08:53:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.811+0000 7f55077fe640 10 client.25757 handle_command_reply: tid=0 2024-05-09T08:53:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.811+0000 7f553160f740 1 client.25757 shutdown 2024-05-09T08:53:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.812+0000 7f553160f740 1 -- 172.21.15.98:0/3865326698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 msgr2=0x55d9adb089c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.812+0000 7f553160f740 1 --2- 172.21.15.98:0/3865326698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adb089c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55d9adaf68f0 tx=0x7f55200138f0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.812+0000 7f553160f740 1 -- 172.21.15.98:0/3865326698 shutdown_connections 2024-05-09T08:53:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.812+0000 7f553160f740 1 --2- 172.21.15.98:0/3865326698 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55d9adb10850 0x55d9adb791e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.812+0000 7f553160f740 1 --2- 172.21.15.98:0/3865326698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d9adb00320 0x55d9adaf9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.812+0000 7f553160f740 1 --2- 172.21.15.98:0/3865326698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9adaf5c50 0x55d9adb089c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.812+0000 7f553160f740 1 --2- 172.21.15.98:0/3865326698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d9adaf5330 0x55d9adb08480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.812+0000 7f553160f740 1 -- 172.21.15.98:0/3865326698 >> 172.21.15.98:0/3865326698 conn(0x55d9adaf1650 msgr2=0x55d9adaeff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.812+0000 7f55077fe640 0 client.25757 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:53:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.813+0000 7f553160f740 1 -- 172.21.15.98:0/3865326698 shutdown_connections 2024-05-09T08:53:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.813+0000 7f553160f740 1 -- 172.21.15.98:0/3865326698 wait complete. 2024-05-09T08:53:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.813+0000 7f553160f740 20 client.25757 trim_cache size 0 max 16384 2024-05-09T08:53:05.813 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:53:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.815+0000 7f552e909640 1 -- 172.21.15.98:0/3682908689 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f55100a2e50 msgr2=0x7f55100a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.815+0000 7f552e909640 1 --2- 172.21.15.98:0/3682908689 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f55100a2e50 0x7f55100a5310 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5518097480 tx=0x7f551800bf90 comp rx=0 tx=0).stop 2024-05-09T08:53:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.815+0000 7f552e909640 1 -- 172.21.15.98:0/3682908689 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5528105bc0 msgr2=0x7f55281a3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.815+0000 7f552e909640 1 --2- 172.21.15.98:0/3682908689 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5528105bc0 0x7f55281a3020 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5518009fd0 tx=0x7f551800b8e0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.815+0000 7f552e909640 1 -- 172.21.15.98:0/3682908689 shutdown_connections 2024-05-09T08:53:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.815+0000 7f552e909640 1 --2- 172.21.15.98:0/3682908689 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f55100a2e50 0x7f55100a5310 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.815+0000 7f552e909640 1 --2- 172.21.15.98:0/3682908689 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5528105bc0 0x7f55281a3020 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.815+0000 7f552e909640 1 --2- 172.21.15.98:0/3682908689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55281011c0 0x7f552819eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.815+0000 7f552e909640 1 --2- 172.21.15.98:0/3682908689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55281008a0 0x7f552819e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.816+0000 7f552e909640 1 -- 172.21.15.98:0/3682908689 >> 172.21.15.98:0/3682908689 conn(0x7f5528077830 msgr2=0x7f5528103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.816+0000 7f552e909640 1 -- 172.21.15.98:0/3682908689 shutdown_connections 2024-05-09T08:53:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.817+0000 7f552e909640 1 -- 172.21.15.98:0/3682908689 wait complete. 2024-05-09T08:53:05.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:53:05.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.988+0000 7f37e8cd4640 1 -- 172.21.15.98:0/955133636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e410bdf0 msgr2=0x7f37e410e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.988+0000 7f37e8cd4640 1 --2- 172.21.15.98:0/955133636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e410bdf0 0x7f37e410e290 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009ca0 tx=0x7f37dc02a9e0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.988+0000 7f37e8cd4640 1 -- 172.21.15.98:0/955133636 shutdown_connections 2024-05-09T08:53:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.988+0000 7f37e8cd4640 1 --2- 172.21.15.98:0/955133636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e410bdf0 0x7f37e410e290 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.988+0000 7f37e8cd4640 1 --2- 172.21.15.98:0/955133636 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37e41033d0 0x7f37e410b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.988+0000 7f37e8cd4640 1 --2- 172.21.15.98:0/955133636 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e4102ab0 0x7f37e4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.988+0000 7f37e8cd4640 1 -- 172.21.15.98:0/955133636 >> 172.21.15.98:0/955133636 conn(0x7f37e4077830 msgr2=0x7f37e4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.988+0000 7f37e8cd4640 1 -- 172.21.15.98:0/955133636 shutdown_connections 2024-05-09T08:53:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.988+0000 7f37e8cd4640 1 -- 172.21.15.98:0/955133636 wait complete. 2024-05-09T08:53:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e8cd4640 1 Processor -- start 2024-05-09T08:53:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e8cd4640 1 -- start start 2024-05-09T08:53:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e8cd4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e4102ab0 0x7f37e4198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e8cd4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37e41033d0 0x7f37e4198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e8cd4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e410bdf0 0x7f37e419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e8cd4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37e4111bf0 con 0x7f37e410bdf0 2024-05-09T08:53:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e8cd4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f37e4111a70 con 0x7f37e41033d0 2024-05-09T08:53:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e8cd4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f37e4111d70 con 0x7f37e4102ab0 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e410bdf0 0x7f37e419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e410bdf0 0x7f37e419c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53310/0 (socket says 172.21.15.98:53310) 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e2d9d640 1 -- 172.21.15.98:0/505650053 learned_addr learned my addr 172.21.15.98:0/505650053 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e259c640 1 --2- 172.21.15.98:0/505650053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e4102ab0 0x7f37e4198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e1d9b640 1 --2- 172.21.15.98:0/505650053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37e41033d0 0x7f37e4198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e2d9d640 1 -- 172.21.15.98:0/505650053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e4102ab0 msgr2=0x7f37e4198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e2d9d640 1 --2- 172.21.15.98:0/505650053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e4102ab0 0x7f37e4198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e2d9d640 1 -- 172.21.15.98:0/505650053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37e41033d0 msgr2=0x7f37e4198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e2d9d640 1 --2- 172.21.15.98:0/505650053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37e41033d0 0x7f37e4198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e2d9d640 1 -- 172.21.15.98:0/505650053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e419d0b0 con 0x7f37e410bdf0 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.989+0000 7f37e2d9d640 1 --2- 172.21.15.98:0/505650053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e410bdf0 0x7f37e419c980 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f37dc006010 tx=0x7f37dc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.990+0000 7f37d37fe640 1 -- 172.21.15.98:0/505650053 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc047070 con 0x7f37e410bdf0 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.990+0000 7f37d37fe640 1 -- 172.21.15.98:0/505650053 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37dc02fb20 con 0x7f37e410bdf0 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.990+0000 7f37e8cd4640 1 -- 172.21.15.98:0/505650053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e419d340 con 0x7f37e410bdf0 2024-05-09T08:53:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.990+0000 7f37d37fe640 1 -- 172.21.15.98:0/505650053 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc033050 con 0x7f37e410bdf0 2024-05-09T08:53:05.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.990+0000 7f37e8cd4640 1 -- 172.21.15.98:0/505650053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e41a4c30 con 0x7f37e410bdf0 2024-05-09T08:53:05.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.992+0000 7f37d37fe640 1 -- 172.21.15.98:0/505650053 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f37dc04a070 con 0x7f37e410bdf0 2024-05-09T08:53:05.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.992+0000 7f37eb9e4740 1 Processor -- start 2024-05-09T08:53:05.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.992+0000 7f37eb9e4740 1 -- start start 2024-05-09T08:53:05.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37eb9e4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54ed000 0x55f3e54ed3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37eb9e4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed920 0x55f3e55009c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37eb9e4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e5500f00 0x55f3e55032f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:05.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37eb9e4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f3e5452c40 con 0x55f3e54ed920 2024-05-09T08:53:05.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37eb9e4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f3e53fa580 con 0x55f3e54ed000 2024-05-09T08:53:05.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37eb9e4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f3e544ec00 con 0x55f3e5500f00 2024-05-09T08:53:05.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37e259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed920 0x55f3e55009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37e259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed920 0x55f3e55009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53314/0 (socket says 172.21.15.98:53314) 2024-05-09T08:53:05.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37e259c640 1 -- 172.21.15.98:0/3905135495 learned_addr learned my addr 172.21.15.98:0/3905135495 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:05.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37e1d9b640 1 --2- 172.21.15.98:0/3905135495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54ed000 0x55f3e54ed3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:05.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37e1d9b640 1 --2- 172.21.15.98:0/3905135495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e5500f00 0x55f3e55032f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37e259c640 1 -- 172.21.15.98:0/3905135495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e5500f00 msgr2=0x55f3e55032f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37e259c640 1 --2- 172.21.15.98:0/3905135495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e5500f00 0x55f3e55032f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37e259c640 1 -- 172.21.15.98:0/3905135495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54ed000 msgr2=0x55f3e54ed3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37e259c640 1 --2- 172.21.15.98:0/3905135495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54ed000 0x55f3e54ed3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37e259c640 1 -- 172.21.15.98:0/3905135495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3e51663f0 con 0x55f3e54ed920 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37d37fe640 1 --2- 172.21.15.98:0/505650053 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f37cc0a2d00 0x7f37cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37e1d9b640 1 --2- 172.21.15.98:0/505650053 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f37cc0a2d00 0x7f37cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37e259c640 1 --2- 172.21.15.98:0/3905135495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed920 0x55f3e55009c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f37d4009c90 tx=0x7f37d40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588a7a60f4a86475 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.993+0000 7f37d37fe640 1 -- 172.21.15.98:0/505650053 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f37dc0f0500 con 0x7f37e410bdf0 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.994+0000 7f37d0ff9640 1 -- 172.21.15.98:0/3905135495 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d4012450 con 0x55f3e54ed920 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.994+0000 7f37d0ff9640 1 -- 172.21.15.98:0/3905135495 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37d4012ad0 con 0x55f3e54ed920 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.994+0000 7f37d0ff9640 1 -- 172.21.15.98:0/3905135495 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d400fb20 con 0x55f3e54ed920 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.994+0000 7f37eb9e4740 1 -- 172.21.15.98:0/3905135495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed920 msgr2=0x55f3e55009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.994+0000 7f37e1d9b640 1 --2- 172.21.15.98:0/505650053 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f37cc0a2d00 0x7f37cc0a51c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55f3e54eaa70 tx=0x7f37d8015040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.994+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/3905135495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed920 0x55f3e55009c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f37d4009c90 tx=0x7f37d40117b0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.994+0000 7f37eb9e4740 1 -- 172.21.15.98:0/3905135495 shutdown_connections 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.994+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/3905135495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e5500f00 0x55f3e55032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.994+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/3905135495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed920 0x55f3e55009c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.994+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/3905135495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54ed000 0x55f3e54ed3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.994+0000 7f37eb9e4740 1 -- 172.21.15.98:0/3905135495 >> 172.21.15.98:0/3905135495 conn(0x55f3e54e9650 msgr2=0x55f3e54e9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.994+0000 7f37eb9e4740 1 -- 172.21.15.98:0/3905135495 shutdown_connections 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.994+0000 7f37eb9e4740 1 -- 172.21.15.98:0/3905135495 wait complete. 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.995+0000 7f37eb9e4740 1 -- start start 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.995+0000 7f37eb9e4740 1 Processor -- start 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.995+0000 7f37eb9e4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed000 0x55f3e54fde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.995+0000 7f37eb9e4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54ed920 0x55f3e54fe340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.995+0000 7f37eb9e4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e5500f00 0x55f3e54fe880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.995+0000 7f37eb9e4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f3e5503920 con 0x55f3e54ed000 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.995+0000 7f37eb9e4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f3e540dc40 con 0x55f3e54ed920 2024-05-09T08:53:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.995+0000 7f37eb9e4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f3e5503aa0 con 0x55f3e5500f00 2024-05-09T08:53:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.995+0000 7f37eb9e4740 10 client.0 _ll_register_callbacks cb 0x55f3e5165be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.995+0000 7f37e259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed000 0x55f3e54fde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.995+0000 7f37e259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed000 0x55f3e54fde00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53330/0 (socket says 172.21.15.98:53330) 2024-05-09T08:53:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.995+0000 7f37e259c640 1 -- 172.21.15.98:0/699692035 learned_addr learned my addr 172.21.15.98:0/699692035 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.995+0000 7f37e1d9b640 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54ed920 0x55f3e54fe340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.995+0000 7f37e2d9d640 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e5500f00 0x55f3e54fe880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37e259c640 1 -- 172.21.15.98:0/699692035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e5500f00 msgr2=0x55f3e54fe880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37e259c640 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e5500f00 0x55f3e54fe880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37e259c640 1 -- 172.21.15.98:0/699692035 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54ed920 msgr2=0x55f3e54fe340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37e259c640 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54ed920 0x55f3e54fe340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37e259c640 1 -- 172.21.15.98:0/699692035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3e52307d0 con 0x55f3e54ed000 2024-05-09T08:53:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37e1d9b640 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54ed920 0x55f3e54fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:53:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37e2d9d640 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e5500f00 0x55f3e54fe880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:53:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37e259c640 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed000 0x55f3e54fde00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f37d400f760 tx=0x7f37d40129c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1cf3ed0533373c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37c9ffb640 1 -- 172.21.15.98:0/699692035 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d4029020 con 0x55f3e54ed000 2024-05-09T08:53:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37c9ffb640 1 -- 172.21.15.98:0/699692035 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f37d4009d80 con 0x55f3e54ed000 2024-05-09T08:53:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37eb9e4740 1 -- 172.21.15.98:0/699692035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37d4012ad0 con 0x55f3e54ed000 2024-05-09T08:53:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37c9ffb640 1 -- 172.21.15.98:0/699692035 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f37d4034050 con 0x55f3e54ed000 2024-05-09T08:53:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37eb9e4740 10 client.16407 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37eb9e4740 1 -- 172.21.15.98:0/699692035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f3e550fb00 con 0x55f3e54ed000 2024-05-09T08:53:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37c9ffb640 1 -- 172.21.15.98:0/699692035 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37d4012ad0 con 0x55f3e54ed000 2024-05-09T08:53:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37c9ffb640 1 -- 172.21.15.98:0/699692035 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f37d40046c0 con 0x55f3e54ed000 2024-05-09T08:53:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37eb9e4740 10 client.16407 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37eb9e4740 10 client.16407 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:53:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37eb9e4740 20 client.16407 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e5510fc0 0x55f3e55133b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37eb9e4740 4 client.16407 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.996+0000 7f37eb9e4740 1 -- 172.21.15.98:0/699692035 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3e5460370 con 0x55f3e5510fc0 2024-05-09T08:53:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.997+0000 7f37e1d9b640 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e5510fc0 0x55f3e55133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.997+0000 7f37c9ffb640 10 client.16407 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:53:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.997+0000 7f37e1d9b640 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e5510fc0 0x55f3e55133b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e9d2475cb150b837 server_cookie=6e299dd9a3cfa69d in_seq=0 out_seq=0 2024-05-09T08:53:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37c9ffb640 1 -- 172.21.15.98:0/699692035 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f3e5460370 con 0x55f3e5510fc0 2024-05-09T08:53:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37c9ffb640 10 client.16407 handle_command_reply: tid=0 2024-05-09T08:53:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37eb9e4740 1 client.16407 shutdown 2024-05-09T08:53:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37eb9e4740 1 -- 172.21.15.98:0/699692035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed000 msgr2=0x55f3e54fde00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed000 0x55f3e54fde00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f37d400f760 tx=0x7f37d40129c0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37eb9e4740 1 -- 172.21.15.98:0/699692035 shutdown_connections 2024-05-09T08:53:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e5510fc0 0x55f3e55133b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e5500f00 0x55f3e54fe880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37c9ffb640 0 client.16407 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:53:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54ed920 0x55f3e54fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/699692035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed000 0x55f3e54fde00 secure :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f37d400f760 tx=0x7f37d40129c0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37eb9e4740 1 -- 172.21.15.98:0/699692035 >> 172.21.15.98:0/699692035 conn(0x55f3e54e9650 msgr2=0x55f3e54ef4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37eb9e4740 1 -- 172.21.15.98:0/699692035 shutdown_connections 2024-05-09T08:53:06.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37eb9e4740 1 -- 172.21.15.98:0/699692035 wait complete. 2024-05-09T08:53:06.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:05.998+0000 7f37eb9e4740 20 client.16407 trim_cache size 0 max 16384 2024-05-09T08:53:06.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.011+0000 7f37eb9e4740 1 Processor -- start 2024-05-09T08:53:06.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.011+0000 7f37eb9e4740 1 -- start start 2024-05-09T08:53:06.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.011+0000 7f37eb9e4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed920 0x55f3e550c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.011+0000 7f37eb9e4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e5500f00 0x55f3e54efbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.011+0000 7f37eb9e4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e54fd230 0x55f3e54e8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.011+0000 7f37eb9e4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41acb30 con 0x55f3e54ed920 2024-05-09T08:53:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.011+0000 7f37eb9e4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41ae670 con 0x55f3e5500f00 2024-05-09T08:53:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.011+0000 7f37eb9e4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41ad200 con 0x55f3e54fd230 2024-05-09T08:53:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.011+0000 7f37e259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed920 0x55f3e550c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.011+0000 7f37e259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed920 0x55f3e550c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53336/0 (socket says 172.21.15.98:53336) 2024-05-09T08:53:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.011+0000 7f37e259c640 1 -- 172.21.15.98:0/1691933956 learned_addr learned my addr 172.21.15.98:0/1691933956 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.011+0000 7f37e2d9d640 1 --2- 172.21.15.98:0/1691933956 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e54fd230 0x55f3e54e8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.011+0000 7f37e1d9b640 1 --2- 172.21.15.98:0/1691933956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e5500f00 0x55f3e54efbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37e259c640 1 -- 172.21.15.98:0/1691933956 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e54fd230 msgr2=0x55f3e54e8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37e259c640 1 --2- 172.21.15.98:0/1691933956 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e54fd230 0x55f3e54e8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37e259c640 1 -- 172.21.15.98:0/1691933956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e5500f00 msgr2=0x55f3e54efbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37e259c640 1 --2- 172.21.15.98:0/1691933956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e5500f00 0x55f3e54efbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37e259c640 1 -- 172.21.15.98:0/1691933956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3e54dad20 con 0x55f3e54ed920 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37e259c640 1 --2- 172.21.15.98:0/1691933956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed920 0x55f3e550c850 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f37d4006010 tx=0x7f37d4012860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=969b2dd7f1532e77 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37cbfff640 1 -- 172.21.15.98:0/1691933956 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d403d070 con 0x55f3e54ed920 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37cbfff640 1 -- 172.21.15.98:0/1691933956 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37d4009ae0 con 0x55f3e54ed920 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37cbfff640 1 -- 172.21.15.98:0/1691933956 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d4029040 con 0x55f3e54ed920 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37eb9e4740 1 -- 172.21.15.98:0/1691933956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed920 msgr2=0x55f3e550c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/1691933956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed920 0x55f3e550c850 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f37d4006010 tx=0x7f37d4012860 comp rx=0 tx=0).stop 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37eb9e4740 1 -- 172.21.15.98:0/1691933956 shutdown_connections 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/1691933956 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e54fd230 0x55f3e54e8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/1691933956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e5500f00 0x55f3e54efbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/1691933956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e54ed920 0x55f3e550c850 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.012+0000 7f37eb9e4740 1 -- 172.21.15.98:0/1691933956 >> 172.21.15.98:0/1691933956 conn(0x55f3e54e9650 msgr2=0x55f3e54e7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37eb9e4740 1 -- 172.21.15.98:0/1691933956 shutdown_connections 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37eb9e4740 1 -- 172.21.15.98:0/1691933956 wait complete. 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37eb9e4740 1 -- start start 2024-05-09T08:53:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37eb9e4740 1 Processor -- start 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37eb9e4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e54ed920 0x55f3e5504980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37eb9e4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54fd230 0x55f3e5504ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37eb9e4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e5500f00 0x55f3e5505400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37eb9e4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41adfa0 con 0x55f3e5500f00 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37eb9e4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41ad8d0 con 0x55f3e54fd230 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37eb9e4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41aab10 con 0x55f3e54ed920 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37eb9e4740 10 client.0 _ll_register_callbacks cb 0x7f37d800e870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37e2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e5500f00 0x55f3e5505400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37e259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e54ed920 0x55f3e5504980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37e2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e5500f00 0x55f3e5505400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53340/0 (socket says 172.21.15.98:53340) 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37e2d9d640 1 -- 172.21.15.98:0/955386551 learned_addr learned my addr 172.21.15.98:0/955386551 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37e1d9b640 1 --2- 172.21.15.98:0/955386551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54fd230 0x55f3e5504ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37e2d9d640 1 -- 172.21.15.98:0/955386551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e54ed920 msgr2=0x55f3e5504980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37e2d9d640 1 --2- 172.21.15.98:0/955386551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e54ed920 0x55f3e5504980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37e2d9d640 1 -- 172.21.15.98:0/955386551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54fd230 msgr2=0x55f3e5504ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37e2d9d640 1 --2- 172.21.15.98:0/955386551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54fd230 0x55f3e5504ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.013+0000 7f37e2d9d640 1 -- 172.21.15.98:0/955386551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3e5576590 con 0x55f3e5500f00 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37e2d9d640 1 --2- 172.21.15.98:0/955386551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e5500f00 0x55f3e5505400 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0b8fd0 tx=0x7f37dc0bc200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc88fb0eb8145ffb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37c9ffb640 1 -- 172.21.15.98:0/955386551 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0d5060 con 0x55f3e5500f00 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37c9ffb640 1 -- 172.21.15.98:0/955386551 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f37dc0c4fe0 con 0x55f3e5500f00 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37eb9e4740 1 -- 172.21.15.98:0/955386551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37d4009ae0 con 0x55f3e5500f00 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37c9ffb640 1 -- 172.21.15.98:0/955386551 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f37dc0b8040 con 0x55f3e5500f00 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37eb9e4740 10 client.16413 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37eb9e4740 1 -- 172.21.15.98:0/955386551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f3e5577890 con 0x55f3e5500f00 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37c9ffb640 1 -- 172.21.15.98:0/955386551 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37d4009ae0 con 0x55f3e5500f00 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37c9ffb640 1 -- 172.21.15.98:0/955386551 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f37dc0bba50 con 0x55f3e5500f00 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37eb9e4740 10 client.16413 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37eb9e4740 10 client.16413 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:53:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37eb9e4740 20 client.16413 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/955386551 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e5578030 0x55f3e557a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37eb9e4740 4 client.16413 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.014+0000 7f37eb9e4740 1 -- 172.21.15.98:0/955386551 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3e545a6c0 con 0x55f3e5578030 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.015+0000 7f37e259c640 1 --2- 172.21.15.98:0/955386551 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e5578030 0x55f3e557a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.015+0000 7f37e259c640 1 --2- 172.21.15.98:0/955386551 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e5578030 0x55f3e557a450 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4467c133dddc8026 server_cookie=c2d7ea1dcf3a1d60 in_seq=0 out_seq=0 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.015+0000 7f37c9ffb640 10 client.16413 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37c9ffb640 1 -- 172.21.15.98:0/955386551 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3e545a6c0 con 0x55f3e5578030 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37c9ffb640 10 client.16413 handle_command_reply: tid=0 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37eb9e4740 1 client.16413 shutdown 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37eb9e4740 1 -- 172.21.15.98:0/955386551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e5500f00 msgr2=0x55f3e5505400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/955386551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e5500f00 0x55f3e5505400 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0b8fd0 tx=0x7f37dc0bc200 comp rx=0 tx=0).stop 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37eb9e4740 1 -- 172.21.15.98:0/955386551 shutdown_connections 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/955386551 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f3e5578030 0x55f3e557a450 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/955386551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3e5500f00 0x55f3e5505400 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37c9ffb640 0 client.16413 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:53:06.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/955386551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3e54fd230 0x55f3e5504ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37eb9e4740 1 --2- 172.21.15.98:0/955386551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f3e54ed920 0x55f3e5504980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37eb9e4740 1 -- 172.21.15.98:0/955386551 >> 172.21.15.98:0/955386551 conn(0x55f3e54e9650 msgr2=0x55f3e54e87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37eb9e4740 1 -- 172.21.15.98:0/955386551 shutdown_connections 2024-05-09T08:53:06.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37eb9e4740 1 -- 172.21.15.98:0/955386551 wait complete. 2024-05-09T08:53:06.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37eb9e4740 20 client.16413 trim_cache size 0 max 16384 2024-05-09T08:53:06.022 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:53:06.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37e8cd4640 1 -- 172.21.15.98:0/505650053 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f37cc0a2d00 msgr2=0x7f37cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37e8cd4640 1 --2- 172.21.15.98:0/505650053 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f37cc0a2d00 0x7f37cc0a51c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55f3e54eaa70 tx=0x7f37d8015040 comp rx=0 tx=0).stop 2024-05-09T08:53:06.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37e8cd4640 1 -- 172.21.15.98:0/505650053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e410bdf0 msgr2=0x7f37e419c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37e8cd4640 1 --2- 172.21.15.98:0/505650053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e410bdf0 0x7f37e419c980 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f37dc006010 tx=0x7f37dc003ba0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37e8cd4640 1 -- 172.21.15.98:0/505650053 shutdown_connections 2024-05-09T08:53:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37e8cd4640 1 --2- 172.21.15.98:0/505650053 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f37cc0a2d00 0x7f37cc0a51c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37e8cd4640 1 --2- 172.21.15.98:0/505650053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e410bdf0 0x7f37e419c980 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37e8cd4640 1 --2- 172.21.15.98:0/505650053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37e41033d0 0x7f37e4198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37e8cd4640 1 --2- 172.21.15.98:0/505650053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e4102ab0 0x7f37e4198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37e8cd4640 1 -- 172.21.15.98:0/505650053 >> 172.21.15.98:0/505650053 conn(0x7f37e4077830 msgr2=0x7f37e40fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37e8cd4640 1 -- 172.21.15.98:0/505650053 shutdown_connections 2024-05-09T08:53:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.016+0000 7f37e8cd4640 1 -- 172.21.15.98:0/505650053 wait complete. 2024-05-09T08:53:06.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:53:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.210+0000 7f34d1238640 1 -- 172.21.15.98:0/3337907508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34cc10e1e0 msgr2=0x7f34cc1105d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.210+0000 7f34d1238640 1 --2- 172.21.15.98:0/3337907508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34cc10e1e0 0x7f34cc1105d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009ca0 tx=0x7f34c402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.210+0000 7f34d1238640 1 -- 172.21.15.98:0/3337907508 shutdown_connections 2024-05-09T08:53:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.210+0000 7f34d1238640 1 --2- 172.21.15.98:0/3337907508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34cc10e1e0 0x7f34cc1105d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.210+0000 7f34d1238640 1 --2- 172.21.15.98:0/3337907508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc100f50 0x7f34cc10dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.210+0000 7f34d1238640 1 --2- 172.21.15.98:0/3337907508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34cc100630 0x7f34cc100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.210+0000 7f34d1238640 1 -- 172.21.15.98:0/3337907508 >> 172.21.15.98:0/3337907508 conn(0x7f34cc077830 msgr2=0x7f34cc077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.210+0000 7f34d1238640 1 -- 172.21.15.98:0/3337907508 shutdown_connections 2024-05-09T08:53:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.210+0000 7f34d1238640 1 -- 172.21.15.98:0/3337907508 wait complete. 2024-05-09T08:53:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.211+0000 7f34d1238640 1 Processor -- start 2024-05-09T08:53:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.211+0000 7f34d1238640 1 -- start start 2024-05-09T08:53:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.212+0000 7f34d1238640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc100630 0x7f34cc19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.212+0000 7f34d1238640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34cc100f50 0x7f34cc19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.212+0000 7f34d1238640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34cc10e1e0 0x7f34cc19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.212+0000 7f34d1238640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc113e10 con 0x7f34cc100630 2024-05-09T08:53:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.212+0000 7f34ca59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34cc100f50 0x7f34cc19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.212+0000 7f34ca59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34cc100f50 0x7f34cc19a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:49362/0 (socket says 172.21.15.98:49362) 2024-05-09T08:53:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.212+0000 7f34cad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc100630 0x7f34cc19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.212+0000 7f34ca59c640 1 -- 172.21.15.98:0/1659456150 learned_addr learned my addr 172.21.15.98:0/1659456150 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.212+0000 7f34cb59e640 1 --2- 172.21.15.98:0/1659456150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34cc10e1e0 0x7f34cc19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.212+0000 7f34d1238640 1 -- 172.21.15.98:0/1659456150 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc113c90 con 0x7f34cc100f50 2024-05-09T08:53:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.212+0000 7f34d1238640 1 -- 172.21.15.98:0/1659456150 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc113f90 con 0x7f34cc10e1e0 2024-05-09T08:53:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.213+0000 7f34ca59c640 1 -- 172.21.15.98:0/1659456150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34cc10e1e0 msgr2=0x7f34cc19eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.213+0000 7f34ca59c640 1 --2- 172.21.15.98:0/1659456150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34cc10e1e0 0x7f34cc19eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.213+0000 7f34ca59c640 1 -- 172.21.15.98:0/1659456150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc100630 msgr2=0x7f34cc19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.213+0000 7f34ca59c640 1 --2- 172.21.15.98:0/1659456150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc100630 0x7f34cc19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.213+0000 7f34ca59c640 1 -- 172.21.15.98:0/1659456150 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34cc19f280 con 0x7f34cc100f50 2024-05-09T08:53:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.213+0000 7f34ca59c640 1 --2- 172.21.15.98:0/1659456150 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34cc100f50 0x7f34cc19a7d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00d950 tx=0x7f34bc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.213+0000 7f34b3fff640 1 -- 172.21.15.98:0/1659456150 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc017030 con 0x7f34cc100f50 2024-05-09T08:53:06.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.213+0000 7f34b3fff640 1 -- 172.21.15.98:0/1659456150 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34bc004020 con 0x7f34cc100f50 2024-05-09T08:53:06.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.213+0000 7f34b3fff640 1 -- 172.21.15.98:0/1659456150 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc0047a0 con 0x7f34cc100f50 2024-05-09T08:53:06.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.214+0000 7f34d1238640 1 -- 172.21.15.98:0/1659456150 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34cc19f570 con 0x7f34cc100f50 2024-05-09T08:53:06.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.214+0000 7f34d1238640 1 -- 172.21.15.98:0/1659456150 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34cc1a6e00 con 0x7f34cc100f50 2024-05-09T08:53:06.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.217+0000 7f34b3fff640 1 -- 172.21.15.98:0/1659456150 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f34bc0041d0 con 0x7f34cc100f50 2024-05-09T08:53:06.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.218+0000 7f34b3fff640 1 --2- 172.21.15.98:0/1659456150 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f34b40a2e00 0x7f34b40a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.218+0000 7f34b3fff640 1 -- 172.21.15.98:0/1659456150 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f34bc0cf090 con 0x7f34cc100f50 2024-05-09T08:53:06.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.218+0000 7f34d4029740 1 Processor -- start 2024-05-09T08:53:06.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.219+0000 7f34d4029740 1 -- start start 2024-05-09T08:53:06.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.219+0000 7f34d4029740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a0f7d0 0x5615c9a0fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.219+0000 7f34d4029740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a10180 0x5615c9a0b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.219+0000 7f34d4029740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 0x5615c9a25990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.219+0000 7f34cad9d640 1 --2- 172.21.15.98:0/1659456150 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f34b40a2e00 0x7f34b40a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.219+0000 7f34d4029740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5615c996cc40 con 0x5615c9a10180 2024-05-09T08:53:06.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.219+0000 7f34d4029740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5615c9914580 con 0x5615c9a0f7d0 2024-05-09T08:53:06.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.219+0000 7f34d4029740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5615c9968c00 con 0x5615c9a235a0 2024-05-09T08:53:06.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.219+0000 7f34ca59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a10180 0x5615c9a0b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.219+0000 7f34ca59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a10180 0x5615c9a0b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53370/0 (socket says 172.21.15.98:53370) 2024-05-09T08:53:06.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.219+0000 7f34ca59c640 1 -- 172.21.15.98:0/1024509813 learned_addr learned my addr 172.21.15.98:0/1024509813 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.219+0000 7f34cb59e640 1 --2- 172.21.15.98:0/1024509813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 0x5615c9a25990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.219+0000 7f34ca59c640 1 -- 172.21.15.98:0/1024509813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 msgr2=0x5615c9a25990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.219+0000 7f34ca59c640 1 --2- 172.21.15.98:0/1024509813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 0x5615c9a25990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.220+0000 7f34ca59c640 1 -- 172.21.15.98:0/1024509813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a0f7d0 msgr2=0x5615c9a0fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.220+0000 7f34ca59c640 1 --2- 172.21.15.98:0/1024509813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a0f7d0 0x5615c9a0fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.220+0000 7f34ca59c640 1 -- 172.21.15.98:0/1024509813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615c96803f0 con 0x5615c9a10180 2024-05-09T08:53:06.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.220+0000 7f34ca59c640 1 --2- 172.21.15.98:0/1024509813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a10180 0x5615c9a0b020 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f34bc097b60 tx=0x7f34bc097560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cb79d68a2cc63f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.220+0000 7f34b17fa640 1 -- 172.21.15.98:0/1024509813 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc09b190 con 0x5615c9a10180 2024-05-09T08:53:06.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.220+0000 7f34b17fa640 1 -- 172.21.15.98:0/1024509813 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34bc09f540 con 0x5615c9a10180 2024-05-09T08:53:06.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.221+0000 7f34b17fa640 1 -- 172.21.15.98:0/1024509813 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc0bb070 con 0x5615c9a10180 2024-05-09T08:53:06.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.221+0000 7f34d4029740 1 -- 172.21.15.98:0/1024509813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a10180 msgr2=0x5615c9a0b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.221+0000 7f34d4029740 1 --2- 172.21.15.98:0/1024509813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a10180 0x5615c9a0b020 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f34bc097b60 tx=0x7f34bc097560 comp rx=0 tx=0).stop 2024-05-09T08:53:06.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.222+0000 7f34d4029740 1 -- 172.21.15.98:0/1024509813 shutdown_connections 2024-05-09T08:53:06.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.222+0000 7f34d4029740 1 --2- 172.21.15.98:0/1024509813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 0x5615c9a25990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.222+0000 7f34d4029740 1 --2- 172.21.15.98:0/1024509813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a10180 0x5615c9a0b020 secure :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f34bc097b60 tx=0x7f34bc097560 comp rx=0 tx=0).stop 2024-05-09T08:53:06.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.222+0000 7f34d4029740 1 --2- 172.21.15.98:0/1024509813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a0f7d0 0x5615c9a0fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.222+0000 7f34d4029740 1 -- 172.21.15.98:0/1024509813 >> 172.21.15.98:0/1024509813 conn(0x5615c9a0be60 msgr2=0x5615c9a0c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.223+0000 7f34cad9d640 1 --2- 172.21.15.98:0/1659456150 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f34b40a2e00 0x7f34b40a52c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f34c0005fd0 tx=0x7f34c00041e0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.223+0000 7f34d4029740 1 -- 172.21.15.98:0/1024509813 shutdown_connections 2024-05-09T08:53:06.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.223+0000 7f34d4029740 1 -- 172.21.15.98:0/1024509813 wait complete. 2024-05-09T08:53:06.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.224+0000 7f34d4029740 1 -- start start 2024-05-09T08:53:06.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.224+0000 7f34d4029740 1 Processor -- start 2024-05-09T08:53:06.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.225+0000 7f34d4029740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a08670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.225+0000 7f34d4029740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a235a0 0x5615c9a08bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.225+0000 7f34d4029740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a090f0 0x5615c9a2f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.225+0000 7f34d4029740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5615c9a26170 con 0x5615c9a0f7d0 2024-05-09T08:53:06.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.225+0000 7f34d4029740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5615c9927c40 con 0x5615c9a235a0 2024-05-09T08:53:06.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.225+0000 7f34d4029740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5615c9a262f0 con 0x5615c9a090f0 2024-05-09T08:53:06.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.225+0000 7f34d4029740 10 client.0 _ll_register_callbacks cb 0x5615c967fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:06.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.225+0000 7f34ca59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a235a0 0x5615c9a08bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.225+0000 7f34ca59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a235a0 0x5615c9a08bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:49386/0 (socket says 172.21.15.98:49386) 2024-05-09T08:53:06.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.225+0000 7f34ca59c640 1 -- 172.21.15.98:0/165404084 learned_addr learned my addr 172.21.15.98:0/165404084 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.226+0000 7f34cb59e640 1 --2- 172.21.15.98:0/165404084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a090f0 0x5615c9a2f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.226+0000 7f34cad9d640 1 --2- 172.21.15.98:0/165404084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a08670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.226+0000 7f34ca59c640 1 -- 172.21.15.98:0/165404084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a090f0 msgr2=0x5615c9a2f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.226+0000 7f34ca59c640 1 --2- 172.21.15.98:0/165404084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a090f0 0x5615c9a2f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.226+0000 7f34ca59c640 1 -- 172.21.15.98:0/165404084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 msgr2=0x5615c9a08670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.226+0000 7f34ca59c640 1 --2- 172.21.15.98:0/165404084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a08670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.226+0000 7f34ca59c640 1 -- 172.21.15.98:0/165404084 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615c974a7d0 con 0x5615c9a235a0 2024-05-09T08:53:06.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.226+0000 7f34cad9d640 1 --2- 172.21.15.98:0/165404084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a08670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:53:06.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.227+0000 7f34ca59c640 1 --2- 172.21.15.98:0/165404084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a235a0 0x5615c9a08bb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f34bc09ae90 tx=0x7f34bc004700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e50abc06b35777b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.227+0000 7f34aeff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:53:06.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.227+0000 7f34aeff5640 1 -- 172.21.15.98:0/165404084 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc09b190 con 0x5615c9a235a0 2024-05-09T08:53:06.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.228+0000 7f34d4029740 1 -- 172.21.15.98:0/165404084 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34bc09f540 con 0x5615c9a235a0 2024-05-09T08:53:06.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.228+0000 7f34aeff5640 1 -- 172.21.15.98:0/165404084 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f34bc0bb0c0 con 0x5615c9a235a0 2024-05-09T08:53:06.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.229+0000 7f34aeff5640 1 -- 172.21.15.98:0/165404084 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f34bc0cc640 con 0x5615c9a235a0 2024-05-09T08:53:06.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.229+0000 7f34d4029740 10 client.25781 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:06.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.229+0000 7f34d4029740 1 -- 172.21.15.98:0/165404084 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5615c9a329a0 con 0x5615c9a235a0 2024-05-09T08:53:06.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.229+0000 7f34aeff5640 1 -- 172.21.15.98:0/165404084 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34bc09f540 con 0x5615c9a235a0 2024-05-09T08:53:06.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.229+0000 7f34aeff5640 1 -- 172.21.15.98:0/165404084 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f34bc096ee0 con 0x5615c9a235a0 2024-05-09T08:53:06.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.229+0000 7f34d4029740 10 client.25781 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:06.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.229+0000 7f34d4029740 10 client.25781 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:53:06.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.229+0000 7f34d4029740 20 client.25781 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:06.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.229+0000 7f34d4029740 1 --2- 172.21.15.98:0/165404084 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5615c9a33e50 0x5615c9a36290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.229+0000 7f34d4029740 4 client.25781 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:06.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.229+0000 7f34d4029740 1 -- 172.21.15.98:0/165404084 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615c997a370 con 0x5615c9a33e50 2024-05-09T08:53:06.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.232+0000 7f34cad9d640 1 --2- 172.21.15.98:0/165404084 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5615c9a33e50 0x5615c9a36290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.233+0000 7f34aeff5640 10 client.25781 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:53:06.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.233+0000 7f34cad9d640 1 --2- 172.21.15.98:0/165404084 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5615c9a33e50 0x5615c9a36290 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=549c00bfc97bd4e3 server_cookie=1799e98a94ffbb65 in_seq=0 out_seq=0 2024-05-09T08:53:06.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.234+0000 7f34aeff5640 1 -- 172.21.15.98:0/165404084 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5615c997a370 con 0x5615c9a33e50 2024-05-09T08:53:06.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.234+0000 7f34aeff5640 10 client.25781 handle_command_reply: tid=0 2024-05-09T08:53:06.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.234+0000 7f34d4029740 1 client.25781 shutdown 2024-05-09T08:53:06.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.235+0000 7f34d4029740 1 -- 172.21.15.98:0/165404084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a235a0 msgr2=0x5615c9a08bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.235+0000 7f34d4029740 1 --2- 172.21.15.98:0/165404084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a235a0 0x5615c9a08bb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f34bc09ae90 tx=0x7f34bc004700 comp rx=0 tx=0).stop 2024-05-09T08:53:06.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.235+0000 7f34d4029740 1 -- 172.21.15.98:0/165404084 shutdown_connections 2024-05-09T08:53:06.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.235+0000 7f34d4029740 1 --2- 172.21.15.98:0/165404084 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5615c9a33e50 0x5615c9a36290 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.235+0000 7f34d4029740 1 --2- 172.21.15.98:0/165404084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a090f0 0x5615c9a2f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.235+0000 7f34d4029740 1 --2- 172.21.15.98:0/165404084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a235a0 0x5615c9a08bb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.235+0000 7f34d4029740 1 --2- 172.21.15.98:0/165404084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a08670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.235+0000 7f34d4029740 1 -- 172.21.15.98:0/165404084 >> 172.21.15.98:0/165404084 conn(0x5615c9a0be60 msgr2=0x5615c9a11cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.235+0000 7f34aeff5640 0 client.25781 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:53:06.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.235+0000 7f34d4029740 1 -- 172.21.15.98:0/165404084 shutdown_connections 2024-05-09T08:53:06.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.235+0000 7f34d4029740 1 -- 172.21.15.98:0/165404084 wait complete. 2024-05-09T08:53:06.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.235+0000 7f34d4029740 20 client.25781 trim_cache size 0 max 16384 2024-05-09T08:53:06.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.261+0000 7f34d4029740 1 Processor -- start 2024-05-09T08:53:06.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.262+0000 7f34d4029740 1 -- start start 2024-05-09T08:53:06.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.262+0000 7f34d4029740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a090f0 0x5615c9a0b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.263+0000 7f34d4029740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a3bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.263+0000 7f34d4029740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 0x5615c9a3c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.263+0000 7f34d4029740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc1acc10 con 0x5615c9a0f7d0 2024-05-09T08:53:06.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.263+0000 7f34d4029740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc1ae750 con 0x5615c9a090f0 2024-05-09T08:53:06.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.263+0000 7f34d4029740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc1ad2e0 con 0x5615c9a235a0 2024-05-09T08:53:06.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.264+0000 7f34ca59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a3bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.264+0000 7f34ca59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a3bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53386/0 (socket says 172.21.15.98:53386) 2024-05-09T08:53:06.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.264+0000 7f34ca59c640 1 -- 172.21.15.98:0/2350346227 learned_addr learned my addr 172.21.15.98:0/2350346227 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.265+0000 7f34cb59e640 1 --2- 172.21.15.98:0/2350346227 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 0x5615c9a3c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.265+0000 7f34cad9d640 1 --2- 172.21.15.98:0/2350346227 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a090f0 0x5615c9a0b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.266+0000 7f34ca59c640 1 -- 172.21.15.98:0/2350346227 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 msgr2=0x5615c9a3c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.266+0000 7f34ca59c640 1 --2- 172.21.15.98:0/2350346227 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 0x5615c9a3c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.266+0000 7f34ca59c640 1 -- 172.21.15.98:0/2350346227 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a090f0 msgr2=0x5615c9a0b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.266+0000 7f34ca59c640 1 --2- 172.21.15.98:0/2350346227 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a090f0 0x5615c9a0b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.266+0000 7f34ca59c640 1 -- 172.21.15.98:0/2350346227 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615c9a08ee0 con 0x5615c9a0f7d0 2024-05-09T08:53:06.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.267+0000 7f34ca59c640 1 --2- 172.21.15.98:0/2350346227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a3bb60 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f34bc097b60 tx=0x7f34bc0974c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f0f295cdcaf5ac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.267+0000 7f34b0ff9640 1 -- 172.21.15.98:0/2350346227 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc014070 con 0x5615c9a0f7d0 2024-05-09T08:53:06.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.268+0000 7f34b0ff9640 1 -- 172.21.15.98:0/2350346227 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34bc09f540 con 0x5615c9a0f7d0 2024-05-09T08:53:06.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.268+0000 7f34b0ff9640 1 -- 172.21.15.98:0/2350346227 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc096ee0 con 0x5615c9a0f7d0 2024-05-09T08:53:06.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.269+0000 7f34d4029740 1 -- 172.21.15.98:0/2350346227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 msgr2=0x5615c9a3bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.269+0000 7f34d4029740 1 --2- 172.21.15.98:0/2350346227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a3bb60 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f34bc097b60 tx=0x7f34bc0974c0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.269+0000 7f34d4029740 1 -- 172.21.15.98:0/2350346227 shutdown_connections 2024-05-09T08:53:06.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.269+0000 7f34d4029740 1 --2- 172.21.15.98:0/2350346227 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 0x5615c9a3c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.269+0000 7f34d4029740 1 --2- 172.21.15.98:0/2350346227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a3bb60 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.269+0000 7f34d4029740 1 --2- 172.21.15.98:0/2350346227 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a090f0 0x5615c9a0b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.269+0000 7f34d4029740 1 -- 172.21.15.98:0/2350346227 >> 172.21.15.98:0/2350346227 conn(0x5615c9a0be60 msgr2=0x5615c9a0c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.269+0000 7f34d4029740 1 -- 172.21.15.98:0/2350346227 shutdown_connections 2024-05-09T08:53:06.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.270+0000 7f34d4029740 1 -- 172.21.15.98:0/2350346227 wait complete. 2024-05-09T08:53:06.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.270+0000 7f34d4029740 1 -- start start 2024-05-09T08:53:06.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.270+0000 7f34d4029740 1 Processor -- start 2024-05-09T08:53:06.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.271+0000 7f34d4029740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a090f0 0x5615c9a2b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.271+0000 7f34d4029740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a2bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.271+0000 7f34d4029740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 0x5615c9a04ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34cad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a090f0 0x5615c9a2b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34cad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a090f0 0x5615c9a2b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:49414/0 (socket says 172.21.15.98:49414) 2024-05-09T08:53:06.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34cad9d640 1 -- 172.21.15.98:0/3315158983 learned_addr learned my addr 172.21.15.98:0/3315158983 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34d4029740 1 -- 172.21.15.98:0/3315158983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc1ae080 con 0x5615c9a0f7d0 2024-05-09T08:53:06.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34ca59c640 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a2bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34d4029740 1 -- 172.21.15.98:0/3315158983 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc1ad9b0 con 0x5615c9a090f0 2024-05-09T08:53:06.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34d4029740 1 -- 172.21.15.98:0/3315158983 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc1aac10 con 0x5615c9a235a0 2024-05-09T08:53:06.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34cb59e640 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 0x5615c9a04ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34d4029740 10 client.0 _ll_register_callbacks cb 0x7f34bc09aff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:06.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34ca59c640 1 -- 172.21.15.98:0/3315158983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 msgr2=0x5615c9a04ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34ca59c640 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 0x5615c9a04ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34ca59c640 1 -- 172.21.15.98:0/3315158983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a090f0 msgr2=0x5615c9a2b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34ca59c640 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a090f0 0x5615c9a2b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34ca59c640 1 -- 172.21.15.98:0/3315158983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615c9a052b0 con 0x5615c9a0f7d0 2024-05-09T08:53:06.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34cb59e640 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 0x5615c9a04ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:53:06.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.272+0000 7f34cad9d640 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a090f0 0x5615c9a2b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:53:06.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.273+0000 7f34ca59c640 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a2bac0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x5615c996cd90 tx=0x7f34bc09a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f5c08526906ad1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.273+0000 7f34aeff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:06.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.273+0000 7f34aeff5640 1 -- 172.21.15.98:0/3315158983 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc014070 con 0x5615c9a0f7d0 2024-05-09T08:53:06.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.273+0000 7f34aeff5640 1 -- 172.21.15.98:0/3315158983 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f34bc09a5e0 con 0x5615c9a0f7d0 2024-05-09T08:53:06.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.273+0000 7f34d4029740 1 -- 172.21.15.98:0/3315158983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34bc09f540 con 0x5615c9a0f7d0 2024-05-09T08:53:06.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.273+0000 7f34aeff5640 1 -- 172.21.15.98:0/3315158983 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f34bc09b190 con 0x5615c9a0f7d0 2024-05-09T08:53:06.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.273+0000 7f34d4029740 10 client.16431 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:06.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.273+0000 7f34d4029740 1 -- 172.21.15.98:0/3315158983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5615c9a052b0 con 0x5615c9a0f7d0 2024-05-09T08:53:06.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.274+0000 7f34aeff5640 1 -- 172.21.15.98:0/3315158983 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34bc09f540 con 0x5615c9a0f7d0 2024-05-09T08:53:06.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.274+0000 7f34aeff5640 1 -- 172.21.15.98:0/3315158983 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f34bc0a07e0 con 0x5615c9a0f7d0 2024-05-09T08:53:06.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.274+0000 7f34d4029740 10 client.16431 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:06.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.274+0000 7f34d4029740 10 client.16431 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:53:06.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.274+0000 7f34d4029740 20 client.16431 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:06.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.274+0000 7f34d4029740 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5615c9aab0e0 0x5615c9aad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.274+0000 7f34d4029740 4 client.16431 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:53:06.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.274+0000 7f34d4029740 1 -- 172.21.15.98:0/3315158983 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5615c99746c0 con 0x5615c9aab0e0 2024-05-09T08:53:06.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.274+0000 7f34cad9d640 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5615c9aab0e0 0x5615c9aad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.276+0000 7f34cad9d640 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5615c9aab0e0 0x5615c9aad4d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=269760ae9e3be8d3 server_cookie=e92a7e5c24f4970 in_seq=0 out_seq=0 2024-05-09T08:53:06.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.276+0000 7f34aeff5640 10 client.16431 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:53:06.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.276+0000 7f34aeff5640 1 -- 172.21.15.98:0/3315158983 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5615c99746c0 con 0x5615c9aab0e0 2024-05-09T08:53:06.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.276+0000 7f34aeff5640 10 client.16431 handle_command_reply: tid=0 2024-05-09T08:53:06.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.276+0000 7f34d4029740 1 client.16431 shutdown 2024-05-09T08:53:06.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.277+0000 7f34d4029740 1 -- 172.21.15.98:0/3315158983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 msgr2=0x5615c9a2bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.277+0000 7f34d4029740 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a2bac0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x5615c996cd90 tx=0x7f34bc09a420 comp rx=0 tx=0).stop 2024-05-09T08:53:06.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.277+0000 7f34d4029740 1 -- 172.21.15.98:0/3315158983 shutdown_connections 2024-05-09T08:53:06.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.277+0000 7f34d4029740 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5615c9aab0e0 0x5615c9aad4d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.277+0000 7f34d4029740 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5615c9a235a0 0x5615c9a04ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.277+0000 7f34d4029740 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5615c9a0f7d0 0x5615c9a2bac0 secure :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0x5615c996cd90 tx=0x7f34bc09a420 comp rx=0 tx=0).stop 2024-05-09T08:53:06.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.277+0000 7f34d4029740 1 --2- 172.21.15.98:0/3315158983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5615c9a090f0 0x5615c9a2b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.277+0000 7f34d4029740 1 -- 172.21.15.98:0/3315158983 >> 172.21.15.98:0/3315158983 conn(0x5615c9a0be60 msgr2=0x5615c9a0b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.277+0000 7f34aeff5640 0 client.16431 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:53:06.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.277+0000 7f34d4029740 1 -- 172.21.15.98:0/3315158983 shutdown_connections 2024-05-09T08:53:06.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.278+0000 7f34d4029740 1 -- 172.21.15.98:0/3315158983 wait complete. 2024-05-09T08:53:06.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.278+0000 7f34d4029740 20 client.16431 trim_cache size 0 max 16384 2024-05-09T08:53:06.278 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:53:06.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.279+0000 7f34d1238640 1 -- 172.21.15.98:0/1659456150 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f34b40a2e00 msgr2=0x7f34b40a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.279+0000 7f34d1238640 1 --2- 172.21.15.98:0/1659456150 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f34b40a2e00 0x7f34b40a52c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f34c0005fd0 tx=0x7f34c00041e0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.279+0000 7f34d1238640 1 -- 172.21.15.98:0/1659456150 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34cc100f50 msgr2=0x7f34cc19a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.279+0000 7f34d1238640 1 --2- 172.21.15.98:0/1659456150 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34cc100f50 0x7f34cc19a7d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00d950 tx=0x7f34bc00de10 comp rx=0 tx=0).stop 2024-05-09T08:53:06.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.279+0000 7f34d1238640 1 -- 172.21.15.98:0/1659456150 shutdown_connections 2024-05-09T08:53:06.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.279+0000 7f34d1238640 1 --2- 172.21.15.98:0/1659456150 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f34b40a2e00 0x7f34b40a52c0 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f34c0005fd0 tx=0x7f34c00041e0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.279+0000 7f34d1238640 1 --2- 172.21.15.98:0/1659456150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34cc10e1e0 0x7f34cc19eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.279+0000 7f34d1238640 1 --2- 172.21.15.98:0/1659456150 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34cc100f50 0x7f34cc19a7d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.279+0000 7f34d1238640 1 --2- 172.21.15.98:0/1659456150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc100630 0x7f34cc19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.279+0000 7f34d1238640 1 -- 172.21.15.98:0/1659456150 >> 172.21.15.98:0/1659456150 conn(0x7f34cc077830 msgr2=0x7f34cc10f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.280+0000 7f34d1238640 1 -- 172.21.15.98:0/1659456150 shutdown_connections 2024-05-09T08:53:06.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.280+0000 7f34d1238640 1 -- 172.21.15.98:0/1659456150 wait complete. 2024-05-09T08:53:06.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:53:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.487+0000 7feee633b640 1 -- 172.21.15.98:0/2666446522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee0112890 msgr2=0x7feee0114c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.487+0000 7feee633b640 1 --2- 172.21.15.98:0/2666446522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee0112890 0x7feee0114c80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7feed801acb0 tx=0x7feed803b9b0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.490+0000 7feee633b640 1 -- 172.21.15.98:0/2666446522 shutdown_connections 2024-05-09T08:53:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.490+0000 7feee633b640 1 --2- 172.21.15.98:0/2666446522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee0112890 0x7feee0114c80 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.490+0000 7feee633b640 1 --2- 172.21.15.98:0/2666446522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feee00744f0 0x7feee0112350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.490+0000 7feee633b640 1 --2- 172.21.15.98:0/2666446522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee0073bd0 0x7feee0073fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.490+0000 7feee633b640 1 -- 172.21.15.98:0/2666446522 >> 172.21.15.98:0/2666446522 conn(0x7feee00fd390 msgr2=0x7feee00ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.491+0000 7feee633b640 1 -- 172.21.15.98:0/2666446522 shutdown_connections 2024-05-09T08:53:06.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.493+0000 7feee633b640 1 -- 172.21.15.98:0/2666446522 wait complete. 2024-05-09T08:53:06.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.494+0000 7feee633b640 1 Processor -- start 2024-05-09T08:53:06.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.495+0000 7feee633b640 1 -- start start 2024-05-09T08:53:06.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.496+0000 7feee633b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee0073bd0 0x7feee00707f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.496+0000 7feee633b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feee00744f0 0x7feee0070d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.496+0000 7feee633b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee0112890 0x7feee0071270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.496+0000 7feee633b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feee0108590 con 0x7feee0112890 2024-05-09T08:53:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.496+0000 7feee633b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feee0108410 con 0x7feee00744f0 2024-05-09T08:53:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.496+0000 7feee633b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7feee0108710 con 0x7feee0073bd0 2024-05-09T08:53:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.499+0000 7feee48d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee0112890 0x7feee0071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.499+0000 7feee48d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee0112890 0x7feee0071270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53408/0 (socket says 172.21.15.98:53408) 2024-05-09T08:53:06.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.499+0000 7feee48d8640 1 -- 172.21.15.98:0/176949442 learned_addr learned my addr 172.21.15.98:0/176949442 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.500+0000 7feee48d8640 1 -- 172.21.15.98:0/176949442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee0073bd0 msgr2=0x7feee00707f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:53:06.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.500+0000 7feee48d8640 1 --2- 172.21.15.98:0/176949442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee0073bd0 0x7feee00707f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.500+0000 7feee48d8640 1 -- 172.21.15.98:0/176949442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feee00744f0 msgr2=0x7feee0070d30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:53:06.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.500+0000 7feee48d8640 1 --2- 172.21.15.98:0/176949442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feee00744f0 0x7feee0070d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.500+0000 7feee48d8640 1 -- 172.21.15.98:0/176949442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee0111eb0 con 0x7feee0112890 2024-05-09T08:53:06.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.501+0000 7feee48d8640 1 --2- 172.21.15.98:0/176949442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee0112890 0x7feee0071270 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7feed8040ac0 tx=0x7feed8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.502+0000 7feedd7fa640 1 -- 172.21.15.98:0/176949442 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed8056070 con 0x7feee0112890 2024-05-09T08:53:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.502+0000 7feee633b640 1 -- 172.21.15.98:0/176949442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feee0112140 con 0x7feee0112890 2024-05-09T08:53:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.502+0000 7feee633b640 1 -- 172.21.15.98:0/176949442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feee0112630 con 0x7feee0112890 2024-05-09T08:53:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.502+0000 7feedd7fa640 1 -- 172.21.15.98:0/176949442 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feed80476d0 con 0x7feee0112890 2024-05-09T08:53:06.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.504+0000 7feedd7fa640 1 -- 172.21.15.98:0/176949442 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed8044040 con 0x7feee0112890 2024-05-09T08:53:06.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.508+0000 7feedd7fa640 1 -- 172.21.15.98:0/176949442 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7feed8059070 con 0x7feee0112890 2024-05-09T08:53:06.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.510+0000 7feedd7fa640 1 --2- 172.21.15.98:0/176949442 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feec80a2ce0 0x7feec80a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.510+0000 7feee9049740 1 Processor -- start 2024-05-09T08:53:06.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.511+0000 7feee9049740 1 -- start start 2024-05-09T08:53:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.512+0000 7feee9049740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095ed330 0x55eb095ed710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.512+0000 7feee9049740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095edc50 0x55eb095f7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.512+0000 7feee9049740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095f8320 0x55eb095fa710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.512+0000 7feee9049740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eb09552c40 con 0x55eb095ed330 2024-05-09T08:53:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.512+0000 7feee9049740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55eb094fa580 con 0x55eb095f8320 2024-05-09T08:53:06.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.512+0000 7feee9049740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55eb0954ec00 con 0x55eb095edc50 2024-05-09T08:53:06.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.512+0000 7feedd7fa640 1 -- 172.21.15.98:0/176949442 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7feed801a8a0 con 0x7feee0112890 2024-05-09T08:53:06.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.515+0000 7feee48d8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095f8320 0x55eb095fa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.517+0000 7feedffff640 1 --2- 172.21.15.98:0/176949442 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feec80a2ce0 0x7feec80a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.517+0000 7feee48d8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095f8320 0x55eb095fa710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:49442/0 (socket says 172.21.15.98:49442) 2024-05-09T08:53:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.517+0000 7feee48d8640 1 -- 172.21.15.98:0/847516934 learned_addr learned my addr 172.21.15.98:0/847516934 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.518+0000 7feee48d8640 1 -- 172.21.15.98:0/847516934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095edc50 msgr2=0x55eb095f7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.518+0000 7feee48d8640 1 --2- 172.21.15.98:0/847516934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095edc50 0x55eb095f7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.518+0000 7feee48d8640 1 -- 172.21.15.98:0/847516934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095ed330 msgr2=0x55eb095ed710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.518+0000 7feee48d8640 1 --2- 172.21.15.98:0/847516934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095ed330 0x55eb095ed710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.518+0000 7feee48d8640 1 -- 172.21.15.98:0/847516934 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb092663f0 con 0x55eb095f8320 2024-05-09T08:53:06.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.520+0000 7feedffff640 1 --2- 172.21.15.98:0/176949442 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feec80a2ce0 0x7feec80a51a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feed0009860 tx=0x7feed0009170 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.522+0000 7feee48d8640 1 --2- 172.21.15.98:0/847516934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095f8320 0x55eb095fa710 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feed80d5aa0 tx=0x7feed80cd250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74394880e8aab7c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.523+0000 7feec67fc640 1 -- 172.21.15.98:0/847516934 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed80e6070 con 0x55eb095f8320 2024-05-09T08:53:06.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.523+0000 7feec67fc640 1 -- 172.21.15.98:0/847516934 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feed80d5fd0 con 0x55eb095f8320 2024-05-09T08:53:06.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.523+0000 7feec67fc640 1 -- 172.21.15.98:0/847516934 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed8004b40 con 0x55eb095f8320 2024-05-09T08:53:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.528+0000 7feee9049740 1 -- 172.21.15.98:0/847516934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095f8320 msgr2=0x55eb095fa710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.528+0000 7feee9049740 1 --2- 172.21.15.98:0/847516934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095f8320 0x55eb095fa710 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feed80d5aa0 tx=0x7feed80cd250 comp rx=0 tx=0).stop 2024-05-09T08:53:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.528+0000 7feee9049740 1 -- 172.21.15.98:0/847516934 shutdown_connections 2024-05-09T08:53:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.528+0000 7feee9049740 1 --2- 172.21.15.98:0/847516934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095f8320 0x55eb095fa710 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.528+0000 7feee9049740 1 --2- 172.21.15.98:0/847516934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095edc50 0x55eb095f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.528+0000 7feee9049740 1 --2- 172.21.15.98:0/847516934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095ed330 0x55eb095ed710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.528+0000 7feee9049740 1 -- 172.21.15.98:0/847516934 >> 172.21.15.98:0/847516934 conn(0x55eb095e9650 msgr2=0x55eb095e9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.528+0000 7feee9049740 1 -- 172.21.15.98:0/847516934 shutdown_connections 2024-05-09T08:53:06.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.529+0000 7feee9049740 1 -- 172.21.15.98:0/847516934 wait complete. 2024-05-09T08:53:06.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.529+0000 7feee9049740 1 -- start start 2024-05-09T08:53:06.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.530+0000 7feee9049740 1 Processor -- start 2024-05-09T08:53:06.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.530+0000 7feee9049740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095ed330 0x55eb095f5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.530+0000 7feee9049740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095edc50 0x55eb095f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.530+0000 7feee9049740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095f8320 0x55eb095f5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.530+0000 7feee9049740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eb094fa580 con 0x55eb095f8320 2024-05-09T08:53:06.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.530+0000 7feee9049740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55eb0950dc40 con 0x55eb095edc50 2024-05-09T08:53:06.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.530+0000 7feee9049740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55eb095faed0 con 0x55eb095ed330 2024-05-09T08:53:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.530+0000 7feee9049740 10 client.0 _ll_register_callbacks cb 0x55eb09265be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.530+0000 7feee48d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095f8320 0x55eb095f5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.531+0000 7feee48d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095f8320 0x55eb095f5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53414/0 (socket says 172.21.15.98:53414) 2024-05-09T08:53:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.531+0000 7feee48d8640 1 -- 172.21.15.98:0/2486673370 learned_addr learned my addr 172.21.15.98:0/2486673370 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.531+0000 7feedffff640 1 --2- 172.21.15.98:0/2486673370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095ed330 0x55eb095f5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.531+0000 7feee48d8640 1 -- 172.21.15.98:0/2486673370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095ed330 msgr2=0x55eb095f5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.531+0000 7feee48d8640 1 --2- 172.21.15.98:0/2486673370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095ed330 0x55eb095f5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.531+0000 7feee48d8640 1 -- 172.21.15.98:0/2486673370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095edc50 msgr2=0x55eb095f57b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:53:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.531+0000 7feee48d8640 1 --2- 172.21.15.98:0/2486673370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095edc50 0x55eb095f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.532+0000 7feee48d8640 1 -- 172.21.15.98:0/2486673370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb093307d0 con 0x55eb095f8320 2024-05-09T08:53:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.532+0000 7feebf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.532+0000 7feee48d8640 1 --2- 172.21.15.98:0/2486673370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095f8320 0x55eb095f5cf0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7feed80cce30 tx=0x7feed80fe5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30862ef2b0d2ad9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.533+0000 7feebf7fe640 1 -- 172.21.15.98:0/2486673370 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed80e6040 con 0x55eb095f8320 2024-05-09T08:53:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.533+0000 7feebf7fe640 1 -- 172.21.15.98:0/2486673370 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7feed8004b40 con 0x55eb095f8320 2024-05-09T08:53:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.533+0000 7feee9049740 1 -- 172.21.15.98:0/2486673370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feed80d5fd0 con 0x55eb095f8320 2024-05-09T08:53:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.534+0000 7feebf7fe640 1 -- 172.21.15.98:0/2486673370 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7feed80e62e0 con 0x55eb095f8320 2024-05-09T08:53:06.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.534+0000 7feee9049740 10 client.16440 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:06.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.534+0000 7feee9049740 1 -- 172.21.15.98:0/2486673370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55eb09606e80 con 0x55eb095f8320 2024-05-09T08:53:06.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.535+0000 7feebf7fe640 1 -- 172.21.15.98:0/2486673370 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feed80d5fd0 con 0x55eb095f8320 2024-05-09T08:53:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.535+0000 7feebf7fe640 1 -- 172.21.15.98:0/2486673370 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7feed80c9140 con 0x55eb095f8320 2024-05-09T08:53:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.536+0000 7feee9049740 10 client.16440 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.536+0000 7feee9049740 10 client.16440 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:53:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.536+0000 7feee9049740 20 client.16440 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.536+0000 7feee9049740 1 --2- 172.21.15.98:0/2486673370 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb09608220 0x55eb0960a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.536+0000 7feee9049740 4 client.16440 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.536+0000 7feee9049740 1 -- 172.21.15.98:0/2486673370 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb09560370 con 0x55eb09608220 2024-05-09T08:53:06.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.538+0000 7feedffff640 1 --2- 172.21.15.98:0/2486673370 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb09608220 0x55eb0960a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.550+0000 7feedffff640 1 --2- 172.21.15.98:0/2486673370 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb09608220 0x55eb0960a680 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b494cfadd94cd8b8 server_cookie=c433128f5945e413 in_seq=0 out_seq=0 2024-05-09T08:53:06.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.550+0000 7feebf7fe640 10 client.16440 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:53:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.554+0000 7feebf7fe640 1 -- 172.21.15.98:0/2486673370 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb09560370 con 0x55eb09608220 2024-05-09T08:53:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.554+0000 7feebf7fe640 10 client.16440 handle_command_reply: tid=0 2024-05-09T08:53:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.555+0000 7feee9049740 1 client.16440 shutdown 2024-05-09T08:53:06.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.555+0000 7feee9049740 1 -- 172.21.15.98:0/2486673370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095f8320 msgr2=0x55eb095f5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.555+0000 7feee9049740 1 --2- 172.21.15.98:0/2486673370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095f8320 0x55eb095f5cf0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7feed80cce30 tx=0x7feed80fe5a0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.556+0000 7feee9049740 1 -- 172.21.15.98:0/2486673370 shutdown_connections 2024-05-09T08:53:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.556+0000 7feee9049740 1 --2- 172.21.15.98:0/2486673370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095f8320 0x55eb095f5cf0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.556+0000 7feee9049740 1 --2- 172.21.15.98:0/2486673370 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb09608220 0x55eb0960a680 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.556+0000 7feee9049740 1 --2- 172.21.15.98:0/2486673370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095edc50 0x55eb095f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.556+0000 7feee9049740 1 --2- 172.21.15.98:0/2486673370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095ed330 0x55eb095f5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.556+0000 7feee9049740 1 -- 172.21.15.98:0/2486673370 >> 172.21.15.98:0/2486673370 conn(0x55eb095e9650 msgr2=0x55eb095ef350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.556+0000 7feebf7fe640 0 client.16440 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:53:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.556+0000 7feee9049740 1 -- 172.21.15.98:0/2486673370 shutdown_connections 2024-05-09T08:53:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.556+0000 7feee9049740 1 -- 172.21.15.98:0/2486673370 wait complete. 2024-05-09T08:53:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.556+0000 7feee9049740 20 client.16440 trim_cache size 0 max 16384 2024-05-09T08:53:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.576+0000 7feee9049740 1 Processor -- start 2024-05-09T08:53:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.577+0000 7feee9049740 1 -- start start 2024-05-09T08:53:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.577+0000 7feee9049740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095ed330 0x55eb09603b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.577+0000 7feee9049740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095edc50 0x55eb095f43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.577+0000 7feee9049740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095f8320 0x55eb0960ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.578+0000 7feee48d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095f8320 0x55eb0960ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.578+0000 7feee48d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095f8320 0x55eb0960ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53416/0 (socket says 172.21.15.98:53416) 2024-05-09T08:53:06.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.578+0000 7feedffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095ed330 0x55eb09603b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.578+0000 7feedffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095ed330 0x55eb09603b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:49466/0 (socket says 172.21.15.98:49466) 2024-05-09T08:53:06.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.578+0000 7feedf7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095edc50 0x55eb095f43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.578+0000 7feedf7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095edc50 0x55eb095f43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:59616/0 (socket says 172.21.15.98:59616) 2024-05-09T08:53:06.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.578+0000 7feee48d8640 1 -- 172.21.15.98:0/3469849198 learned_addr learned my addr 172.21.15.98:0/3469849198 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.578+0000 7feee9049740 1 -- 172.21.15.98:0/3469849198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feecc010060 con 0x55eb095f8320 2024-05-09T08:53:06.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.578+0000 7feee9049740 1 -- 172.21.15.98:0/3469849198 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feecc011ba0 con 0x55eb095ed330 2024-05-09T08:53:06.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.578+0000 7feee9049740 1 -- 172.21.15.98:0/3469849198 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7feecc010730 con 0x55eb095edc50 2024-05-09T08:53:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.579+0000 7feee48d8640 1 -- 172.21.15.98:0/3469849198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095edc50 msgr2=0x55eb095f43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.579+0000 7feee48d8640 1 --2- 172.21.15.98:0/3469849198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095edc50 0x55eb095f43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.579+0000 7feee48d8640 1 -- 172.21.15.98:0/3469849198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095ed330 msgr2=0x55eb09603b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.579+0000 7feee48d8640 1 --2- 172.21.15.98:0/3469849198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095ed330 0x55eb09603b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.579+0000 7feee48d8640 1 -- 172.21.15.98:0/3469849198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb0960ad00 con 0x55eb095f8320 2024-05-09T08:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.579+0000 7feee48d8640 1 --2- 172.21.15.98:0/3469849198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095f8320 0x55eb0960ff90 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7feed803b9f0 tx=0x7feed80feb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47143d51673bbd91 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.579+0000 7feec5ffb640 1 -- 172.21.15.98:0/3469849198 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed80e62e0 con 0x55eb095f8320 2024-05-09T08:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.579+0000 7feec5ffb640 1 -- 172.21.15.98:0/3469849198 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feed80d5fd0 con 0x55eb095f8320 2024-05-09T08:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.579+0000 7feec5ffb640 1 -- 172.21.15.98:0/3469849198 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed80c9310 con 0x55eb095f8320 2024-05-09T08:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.580+0000 7feee9049740 1 -- 172.21.15.98:0/3469849198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095f8320 msgr2=0x55eb0960ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.580+0000 7feee9049740 1 --2- 172.21.15.98:0/3469849198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095f8320 0x55eb0960ff90 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7feed803b9f0 tx=0x7feed80feb20 comp rx=0 tx=0).stop 2024-05-09T08:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.580+0000 7feee9049740 1 -- 172.21.15.98:0/3469849198 shutdown_connections 2024-05-09T08:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.580+0000 7feee9049740 1 --2- 172.21.15.98:0/3469849198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095f8320 0x55eb0960ff90 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.580+0000 7feee9049740 1 --2- 172.21.15.98:0/3469849198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095edc50 0x55eb095f43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.580+0000 7feee9049740 1 --2- 172.21.15.98:0/3469849198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095ed330 0x55eb09603b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.580+0000 7feee9049740 1 -- 172.21.15.98:0/3469849198 >> 172.21.15.98:0/3469849198 conn(0x55eb095e9650 msgr2=0x55eb095e46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.580+0000 7feee9049740 1 -- 172.21.15.98:0/3469849198 shutdown_connections 2024-05-09T08:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.580+0000 7feee9049740 1 -- 172.21.15.98:0/3469849198 wait complete. 2024-05-09T08:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.581+0000 7feee9049740 1 -- start start 2024-05-09T08:53:06.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.581+0000 7feee9049740 1 Processor -- start 2024-05-09T08:53:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.581+0000 7feee9049740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095ed330 0x55eb09600510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.582+0000 7feee9049740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095edc50 0x55eb095f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.582+0000 7feee9049740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095f8320 0x55eb095f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.582+0000 7feee9049740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feecc0114d0 con 0x55eb095edc50 2024-05-09T08:53:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.582+0000 7feee9049740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feecc010e00 con 0x55eb095ed330 2024-05-09T08:53:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.582+0000 7feee9049740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7feecc00e000 con 0x55eb095f8320 2024-05-09T08:53:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.582+0000 7feee9049740 10 client.0 _ll_register_callbacks cb 0x7feed80e4070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.582+0000 7feedf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095edc50 0x55eb095f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.582+0000 7feedf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095edc50 0x55eb095f1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53430/0 (socket says 172.21.15.98:53430) 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.582+0000 7feedf7fe640 1 -- 172.21.15.98:0/1957021392 learned_addr learned my addr 172.21.15.98:0/1957021392 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.582+0000 7feedf7fe640 1 -- 172.21.15.98:0/1957021392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095f8320 msgr2=0x55eb095f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.582+0000 7feedf7fe640 1 --2- 172.21.15.98:0/1957021392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095f8320 0x55eb095f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.582+0000 7feedf7fe640 1 -- 172.21.15.98:0/1957021392 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095ed330 msgr2=0x55eb09600510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.582+0000 7feedf7fe640 1 --2- 172.21.15.98:0/1957021392 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095ed330 0x55eb09600510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.582+0000 7feedf7fe640 1 -- 172.21.15.98:0/1957021392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb095fa710 con 0x55eb095edc50 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.583+0000 7feedf7fe640 1 --2- 172.21.15.98:0/1957021392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095edc50 0x55eb095f1dc0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7feed4004720 tx=0x7feed400d530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e018a5df641804 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.583+0000 7feebf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.583+0000 7feebf7fe640 1 -- 172.21.15.98:0/1957021392 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed400dda0 con 0x55eb095edc50 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.583+0000 7feebf7fe640 1 -- 172.21.15.98:0/1957021392 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7feed4014430 con 0x55eb095edc50 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.583+0000 7feee9049740 1 -- 172.21.15.98:0/1957021392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feed80d5fd0 con 0x55eb095edc50 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.583+0000 7feebf7fe640 1 -- 172.21.15.98:0/1957021392 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7feed4022050 con 0x55eb095edc50 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.584+0000 7feee9049740 10 client.16446 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.584+0000 7feee9049740 1 -- 172.21.15.98:0/1957021392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55eb0966e800 con 0x55eb095edc50 2024-05-09T08:53:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.584+0000 7feebf7fe640 1 -- 172.21.15.98:0/1957021392 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feed80d5fd0 con 0x55eb095edc50 2024-05-09T08:53:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.584+0000 7feebf7fe640 1 -- 172.21.15.98:0/1957021392 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7feed4013070 con 0x55eb095edc50 2024-05-09T08:53:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.584+0000 7feee9049740 10 client.16446 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.584+0000 7feee9049740 10 client.16446 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:53:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.584+0000 7feee9049740 20 client.16446 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.584+0000 7feee9049740 1 --2- 172.21.15.98:0/1957021392 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb0966f000 0x55eb09671440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.584+0000 7feee9049740 4 client.16446 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:53:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.584+0000 7feee9049740 1 -- 172.21.15.98:0/1957021392 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eb0955a6c0 con 0x55eb0966f000 2024-05-09T08:53:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.585+0000 7feedffff640 1 --2- 172.21.15.98:0/1957021392 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb0966f000 0x55eb09671440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.587+0000 7feedffff640 1 --2- 172.21.15.98:0/1957021392 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb0966f000 0x55eb09671440 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4e606240fa2500c8 server_cookie=fbd20752447929ca in_seq=0 out_seq=0 2024-05-09T08:53:06.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.587+0000 7feebf7fe640 10 client.16446 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:53:06.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:06 smithi098 ceph-mon[30092]: pgmap v76: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 732 MiB/s wr, 1.49k op/s 2024-05-09T08:53:06.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:06 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4195294583' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:53:06.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.588+0000 7feebf7fe640 1 -- 172.21.15.98:0/1957021392 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eb0955a6c0 con 0x55eb0966f000 2024-05-09T08:53:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.588+0000 7feebf7fe640 10 client.16446 handle_command_reply: tid=0 2024-05-09T08:53:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.588+0000 7feee9049740 1 client.16446 shutdown 2024-05-09T08:53:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.589+0000 7feee9049740 1 -- 172.21.15.98:0/1957021392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095edc50 msgr2=0x55eb095f1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.589+0000 7feee9049740 1 --2- 172.21.15.98:0/1957021392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095edc50 0x55eb095f1dc0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7feed4004720 tx=0x7feed400d530 comp rx=0 tx=0).stop 2024-05-09T08:53:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.589+0000 7feee9049740 1 -- 172.21.15.98:0/1957021392 shutdown_connections 2024-05-09T08:53:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.589+0000 7feee9049740 1 --2- 172.21.15.98:0/1957021392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb095f8320 0x55eb095f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.589+0000 7feee9049740 1 --2- 172.21.15.98:0/1957021392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb095edc50 0x55eb095f1dc0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.589+0000 7feee9049740 1 --2- 172.21.15.98:0/1957021392 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb0966f000 0x55eb09671440 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.589+0000 7feee9049740 1 --2- 172.21.15.98:0/1957021392 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb095ed330 0x55eb09600510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.589+0000 7feee9049740 1 -- 172.21.15.98:0/1957021392 >> 172.21.15.98:0/1957021392 conn(0x55eb095e9650 msgr2=0x55eb095e7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.589+0000 7feebf7fe640 0 client.16446 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:53:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.589+0000 7feee9049740 1 -- 172.21.15.98:0/1957021392 shutdown_connections 2024-05-09T08:53:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.589+0000 7feee9049740 1 -- 172.21.15.98:0/1957021392 wait complete. 2024-05-09T08:53:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.589+0000 7feee9049740 20 client.16446 trim_cache size 0 max 16384 2024-05-09T08:53:06.590 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:53:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.590+0000 7feee633b640 1 -- 172.21.15.98:0/176949442 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feec80a2ce0 msgr2=0x7feec80a51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.590+0000 7feee633b640 1 --2- 172.21.15.98:0/176949442 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feec80a2ce0 0x7feec80a51a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feed0009860 tx=0x7feed0009170 comp rx=0 tx=0).stop 2024-05-09T08:53:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.590+0000 7feee633b640 1 -- 172.21.15.98:0/176949442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee0112890 msgr2=0x7feee0071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.590+0000 7feee633b640 1 --2- 172.21.15.98:0/176949442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee0112890 0x7feee0071270 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7feed8040ac0 tx=0x7feed8003ba0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.590+0000 7feee633b640 1 -- 172.21.15.98:0/176949442 shutdown_connections 2024-05-09T08:53:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.590+0000 7feee633b640 1 --2- 172.21.15.98:0/176949442 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7feec80a2ce0 0x7feec80a51a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.590+0000 7feee633b640 1 --2- 172.21.15.98:0/176949442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee0112890 0x7feee0071270 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.590+0000 7feee633b640 1 --2- 172.21.15.98:0/176949442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feee00744f0 0x7feee0070d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.590+0000 7feee633b640 1 --2- 172.21.15.98:0/176949442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee0073bd0 0x7feee00707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.590+0000 7feee633b640 1 -- 172.21.15.98:0/176949442 >> 172.21.15.98:0/176949442 conn(0x7feee00fd390 msgr2=0x7feee00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.590+0000 7feee633b640 1 -- 172.21.15.98:0/176949442 shutdown_connections 2024-05-09T08:53:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.590+0000 7feee633b640 1 -- 172.21.15.98:0/176949442 wait complete. 2024-05-09T08:53:06.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:53:06.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:06 smithi142 ceph-mon[34625]: pgmap v76: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 732 MiB/s wr, 1.49k op/s 2024-05-09T08:53:06.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:06 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/4195294583' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:53:06.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:06 smithi105 ceph-mon[34498]: pgmap v76: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 732 MiB/s wr, 1.49k op/s 2024-05-09T08:53:06.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:06 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/4195294583' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:53:06.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.809+0000 7f65be7fc640 1 -- 172.21.15.98:0/2225829866 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8030e30 con 0x7f65c010e1f0 2024-05-09T08:53:06.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.810+0000 7f65c5cfc640 1 -- 172.21.15.98:0/2225829866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65c010e1f0 msgr2=0x7f65c01105e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.810+0000 7f65c5cfc640 1 --2- 172.21.15.98:0/2225829866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65c010e1f0 0x7f65c01105e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f65b8009be0 tx=0x7f65b802a6e0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.810+0000 7f65c5cfc640 1 -- 172.21.15.98:0/2225829866 shutdown_connections 2024-05-09T08:53:06.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.810+0000 7f65c5cfc640 1 --2- 172.21.15.98:0/2225829866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65c010e1f0 0x7f65c01105e0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.810+0000 7f65c5cfc640 1 --2- 172.21.15.98:0/2225829866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c0100f60 0x7f65c010dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.810+0000 7f65c5cfc640 1 --2- 172.21.15.98:0/2225829866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65c0100640 0x7f65c0100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.810+0000 7f65c5cfc640 1 -- 172.21.15.98:0/2225829866 >> 172.21.15.98:0/2225829866 conn(0x7f65c0076ab0 msgr2=0x7f65c0076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.810+0000 7f65c5cfc640 1 -- 172.21.15.98:0/2225829866 shutdown_connections 2024-05-09T08:53:06.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.810+0000 7f65c5cfc640 1 -- 172.21.15.98:0/2225829866 wait complete. 2024-05-09T08:53:06.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.811+0000 7f65c5cfc640 1 Processor -- start 2024-05-09T08:53:06.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.811+0000 7f65c5cfc640 1 -- start start 2024-05-09T08:53:06.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.811+0000 7f65c5cfc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65c0100640 0x7f65c01099f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.811+0000 7f65c5cfc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65c0100f60 0x7f65c0109f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.811+0000 7f65c5cfc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c010e1f0 0x7f65c010a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.811+0000 7f65c5cfc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65c0103a50 con 0x7f65c0100f60 2024-05-09T08:53:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.811+0000 7f65c5cfc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f65c01038d0 con 0x7f65c0100640 2024-05-09T08:53:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.811+0000 7f65c5cfc640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f65c0103bd0 con 0x7f65c010e1f0 2024-05-09T08:53:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.811+0000 7f65bf7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65c0100640 0x7f65c01099f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.811+0000 7f65bffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c010e1f0 0x7f65c010a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.811+0000 7f65bf7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65c0100640 0x7f65c01099f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:49506/0 (socket says 172.21.15.98:49506) 2024-05-09T08:53:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.811+0000 7f65bffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c010e1f0 0x7f65c010a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:59630/0 (socket says 172.21.15.98:59630) 2024-05-09T08:53:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.811+0000 7f65bf7fe640 1 -- 172.21.15.98:0/3160630192 learned_addr learned my addr 172.21.15.98:0/3160630192 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.812+0000 7f65bffff640 1 -- 172.21.15.98:0/3160630192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65c0100640 msgr2=0x7f65c01099f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.812+0000 7f65bffff640 1 --2- 172.21.15.98:0/3160630192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65c0100640 0x7f65c01099f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.812+0000 7f65bffff640 1 -- 172.21.15.98:0/3160630192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65c0100f60 msgr2=0x7f65c0109f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.812+0000 7f65bffff640 1 --2- 172.21.15.98:0/3160630192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65c0100f60 0x7f65c0109f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.812+0000 7f65bffff640 1 -- 172.21.15.98:0/3160630192 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c01042c0 con 0x7f65c010e1f0 2024-05-09T08:53:06.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.812+0000 7f65bf7fe640 1 --2- 172.21.15.98:0/3160630192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65c0100640 0x7f65c01099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:53:06.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.812+0000 7f65bffff640 1 --2- 172.21.15.98:0/3160630192 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c010e1f0 0x7f65c010a470 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f65b8034040 tx=0x7f65b80305d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.813+0000 7f65bcff9640 1 -- 172.21.15.98:0/3160630192 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8041020 con 0x7f65c010e1f0 2024-05-09T08:53:06.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.813+0000 7f65bcff9640 1 -- 172.21.15.98:0/3160630192 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65b8037040 con 0x7f65c010e1f0 2024-05-09T08:53:06.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.814+0000 7f65c5cfc640 1 -- 172.21.15.98:0/3160630192 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65c01044f0 con 0x7f65c010e1f0 2024-05-09T08:53:06.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.814+0000 7f65c5cfc640 1 -- 172.21.15.98:0/3160630192 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65c01049e0 con 0x7f65c010e1f0 2024-05-09T08:53:06.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.814+0000 7f65bcff9640 1 -- 172.21.15.98:0/3160630192 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8036070 con 0x7f65c010e1f0 2024-05-09T08:53:06.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.818+0000 7f65c8a06740 1 Processor -- start 2024-05-09T08:53:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.818+0000 7f65c8a06740 1 -- start start 2024-05-09T08:53:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.818+0000 7f65bcff9640 1 -- 172.21.15.98:0/3160630192 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f65b8044020 con 0x7f65c010e1f0 2024-05-09T08:53:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.818+0000 7f65c8a06740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 0x556168601710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.819+0000 7f65c8a06740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x55616860bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.819+0000 7f65c8a06740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x55616860e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.819+0000 7f65beffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 0x556168601710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.819+0000 7f65beffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 0x556168601710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:59640/0 (socket says 172.21.15.98:59640) 2024-05-09T08:53:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.819+0000 7f65beffd640 1 -- 172.21.15.98:0/2278602778 learned_addr learned my addr 172.21.15.98:0/2278602778 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.819+0000 7f65c8a06740 1 -- 172.21.15.98:0/2278602778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556168566c40 con 0x55616860c320 2024-05-09T08:53:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.819+0000 7f65c8a06740 1 -- 172.21.15.98:0/2278602778 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55616850e580 con 0x556168601c50 2024-05-09T08:53:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.819+0000 7f65c8a06740 1 -- 172.21.15.98:0/2278602778 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556168562c00 con 0x556168601330 2024-05-09T08:53:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.819+0000 7f65beffd640 1 --2- 172.21.15.98:0/2278602778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x55616860e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.819+0000 7f65beffd640 1 -- 172.21.15.98:0/2278602778 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 msgr2=0x55616860bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.819+0000 7f65beffd640 1 --2- 172.21.15.98:0/2278602778 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x55616860bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.819+0000 7f65beffd640 1 -- 172.21.15.98:0/2278602778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 msgr2=0x55616860e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.819+0000 7f65beffd640 1 --2- 172.21.15.98:0/2278602778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x55616860e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.819+0000 7f65beffd640 1 -- 172.21.15.98:0/2278602778 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55616827a3f0 con 0x556168601330 2024-05-09T08:53:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.820+0000 7f65bcff9640 1 --2- 172.21.15.98:0/3160630192 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f65a80a2d30 0x7f65a80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.820+0000 7f65bcff9640 1 -- 172.21.15.98:0/3160630192 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f65b80ef730 con 0x7f65c010e1f0 2024-05-09T08:53:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.820+0000 7f65beffd640 1 --2- 172.21.15.98:0/2278602778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 0x556168601710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f65b40137d0 tx=0x7f65b4009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46b0a258d2238630 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.821+0000 7f65bf7fe640 1 --2- 172.21.15.98:0/3160630192 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f65a80a2d30 0x7f65a80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.821+0000 7f65a5ffb640 1 -- 172.21.15.98:0/2278602778 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b40132d0 con 0x556168601330 2024-05-09T08:53:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.821+0000 7f65a5ffb640 1 -- 172.21.15.98:0/2278602778 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65b4013e10 con 0x556168601330 2024-05-09T08:53:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.821+0000 7f65a5ffb640 1 -- 172.21.15.98:0/2278602778 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b40035f0 con 0x556168601330 2024-05-09T08:53:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.822+0000 7f65c8a06740 1 -- 172.21.15.98:0/2278602778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 msgr2=0x556168601710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.822+0000 7f65c8a06740 1 --2- 172.21.15.98:0/2278602778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 0x556168601710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f65b40137d0 tx=0x7f65b4009620 comp rx=0 tx=0).stop 2024-05-09T08:53:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.823+0000 7f65c8a06740 1 -- 172.21.15.98:0/2278602778 shutdown_connections 2024-05-09T08:53:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.823+0000 7f65c8a06740 1 --2- 172.21.15.98:0/2278602778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x55616860e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.823+0000 7f65c8a06740 1 --2- 172.21.15.98:0/2278602778 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x55616860bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.823+0000 7f65c8a06740 1 --2- 172.21.15.98:0/2278602778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 0x556168601710 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.823+0000 7f65c8a06740 1 -- 172.21.15.98:0/2278602778 >> 172.21.15.98:0/2278602778 conn(0x5561685fd650 msgr2=0x5561685fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.823+0000 7f65bf7fe640 1 --2- 172.21.15.98:0/3160630192 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f65a80a2d30 0x7f65a80a51f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f65b00093b0 tx=0x7f65b0009340 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.824+0000 7f65c8a06740 1 -- 172.21.15.98:0/2278602778 shutdown_connections 2024-05-09T08:53:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.824+0000 7f65c8a06740 1 -- 172.21.15.98:0/2278602778 wait complete. 2024-05-09T08:53:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.824+0000 7f65c8a06740 1 -- start start 2024-05-09T08:53:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.824+0000 7f65c8a06740 1 Processor -- start 2024-05-09T08:53:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65c8a06740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556168601330 0x556168609210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65c8a06740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x556168609750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65c8a06740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55616860c320 0x556168609c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65bf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556168601330 0x556168609210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65bf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556168601330 0x556168609210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53456/0 (socket says 172.21.15.98:53456) 2024-05-09T08:53:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65bf7fe640 1 -- 172.21.15.98:0/1388806150 learned_addr learned my addr 172.21.15.98:0/1388806150 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65bffff640 1 --2- 172.21.15.98:0/1388806150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55616860c320 0x556168609c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65c8a06740 1 -- 172.21.15.98:0/1388806150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55616860f0b0 con 0x556168601330 2024-05-09T08:53:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65c8a06740 1 -- 172.21.15.98:0/1388806150 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556168521c40 con 0x556168601c50 2024-05-09T08:53:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65c8a06740 1 -- 172.21.15.98:0/1388806150 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55616860f230 con 0x55616860c320 2024-05-09T08:53:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65beffd640 1 --2- 172.21.15.98:0/1388806150 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x556168609750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65c8a06740 10 client.0 _ll_register_callbacks cb 0x556168279be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65bffff640 1 -- 172.21.15.98:0/1388806150 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 msgr2=0x556168609750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65bffff640 1 --2- 172.21.15.98:0/1388806150 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x556168609750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65bffff640 1 -- 172.21.15.98:0/1388806150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556168601330 msgr2=0x556168609210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65bffff640 1 --2- 172.21.15.98:0/1388806150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556168601330 0x556168609210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.825+0000 7f65bffff640 1 -- 172.21.15.98:0/1388806150 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561683447d0 con 0x55616860c320 2024-05-09T08:53:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.826+0000 7f65bf7fe640 1 --2- 172.21.15.98:0/1388806150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556168601330 0x556168609210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:53:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.827+0000 7f65bffff640 1 --2- 172.21.15.98:0/1388806150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55616860c320 0x556168609c90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f65b80eca60 tx=0x7f65b80bbfc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dae5fd28f70a99e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.827+0000 7f659effd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:53:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.827+0000 7f659effd640 1 -- 172.21.15.98:0/1388806150 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b80c44a0 con 0x55616860c320 2024-05-09T08:53:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.827+0000 7f659effd640 1 -- 172.21.15.98:0/1388806150 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f65b80b7f30 con 0x55616860c320 2024-05-09T08:53:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.827+0000 7f65c8a06740 1 -- 172.21.15.98:0/1388806150 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65b4013e10 con 0x55616860c320 2024-05-09T08:53:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.829+0000 7f659effd640 1 -- 172.21.15.98:0/1388806150 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f65b80ecce0 con 0x55616860c320 2024-05-09T08:53:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.829+0000 7f65c8a06740 10 client.25759 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.829+0000 7f65c8a06740 1 -- 172.21.15.98:0/1388806150 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55616860f100 con 0x55616860c320 2024-05-09T08:53:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.830+0000 7f659effd640 1 -- 172.21.15.98:0/1388806150 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65b4013e10 con 0x55616860c320 2024-05-09T08:53:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.831+0000 7f659effd640 1 -- 172.21.15.98:0/1388806150 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f65b80bbbb0 con 0x55616860c320 2024-05-09T08:53:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.831+0000 7f65c8a06740 10 client.25759 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.831+0000 7f65c8a06740 10 client.25759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.831+0000 7f65c8a06740 20 client.25759 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.831+0000 7f65c8a06740 1 --2- 172.21.15.98:0/1388806150 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55616861c6e0 0x55616861eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.831+0000 7f65c8a06740 4 client.25759 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.831+0000 7f65c8a06740 1 -- 172.21.15.98:0/1388806150 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556168574370 con 0x55616861c6e0 2024-05-09T08:53:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.832+0000 7f65bf7fe640 1 --2- 172.21.15.98:0/1388806150 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55616861c6e0 0x55616861eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.833+0000 7f659effd640 10 client.25759 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.833+0000 7f65bf7fe640 1 --2- 172.21.15.98:0/1388806150 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55616861c6e0 0x55616861eb10 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=758292af48cff603 server_cookie=f8e63defc44388d2 in_seq=0 out_seq=0 2024-05-09T08:53:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.834+0000 7f659effd640 1 -- 172.21.15.98:0/1388806150 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556168574370 con 0x55616861c6e0 2024-05-09T08:53:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.834+0000 7f659effd640 10 client.25759 handle_command_reply: tid=0 2024-05-09T08:53:06.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.834+0000 7f65c8a06740 1 client.25759 shutdown 2024-05-09T08:53:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.835+0000 7f65c8a06740 1 -- 172.21.15.98:0/1388806150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55616860c320 msgr2=0x556168609c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.835+0000 7f65c8a06740 1 --2- 172.21.15.98:0/1388806150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55616860c320 0x556168609c90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f65b80eca60 tx=0x7f65b80bbfc0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.835+0000 7f65c8a06740 1 -- 172.21.15.98:0/1388806150 shutdown_connections 2024-05-09T08:53:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.835+0000 7f65c8a06740 1 --2- 172.21.15.98:0/1388806150 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55616861c6e0 0x55616861eb10 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.835+0000 7f65c8a06740 1 --2- 172.21.15.98:0/1388806150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55616860c320 0x556168609c90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.835+0000 7f65c8a06740 1 --2- 172.21.15.98:0/1388806150 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x556168609750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.835+0000 7f65c8a06740 1 --2- 172.21.15.98:0/1388806150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556168601330 0x556168609210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.835+0000 7f65c8a06740 1 -- 172.21.15.98:0/1388806150 >> 172.21.15.98:0/1388806150 conn(0x5561685fd650 msgr2=0x556168603450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.835+0000 7f659effd640 0 client.25759 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.835+0000 7f65c8a06740 1 -- 172.21.15.98:0/1388806150 shutdown_connections 2024-05-09T08:53:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.836+0000 7f65c8a06740 1 -- 172.21.15.98:0/1388806150 wait complete. 2024-05-09T08:53:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.836+0000 7f65c8a06740 20 client.25759 trim_cache size 0 max 16384 2024-05-09T08:53:06.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.867+0000 7f65c8a06740 1 Processor -- start 2024-05-09T08:53:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.871+0000 7f65c8a06740 1 -- start start 2024-05-09T08:53:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.872+0000 7f65c8a06740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 0x556168607b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.872+0000 7f65c8a06740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x556168603a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.872+0000 7f65c8a06740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x556168609ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.872+0000 7f65c8a06740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65c011b2f0 con 0x55616860c320 2024-05-09T08:53:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.872+0000 7f65c8a06740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f65c01c7b90 con 0x556168601c50 2024-05-09T08:53:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.872+0000 7f65c8a06740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f65c01c6720 con 0x556168601330 2024-05-09T08:53:06.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.874+0000 7f65beffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x556168603a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.874+0000 7f65bffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x556168609ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.874+0000 7f65bffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x556168609ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53470/0 (socket says 172.21.15.98:53470) 2024-05-09T08:53:06.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.874+0000 7f65bffff640 1 -- 172.21.15.98:0/365883292 learned_addr learned my addr 172.21.15.98:0/365883292 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.876+0000 7f65bf7fe640 1 --2- 172.21.15.98:0/365883292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 0x556168607b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.877+0000 7f65bffff640 1 -- 172.21.15.98:0/365883292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 msgr2=0x556168607b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.877+0000 7f65bffff640 1 --2- 172.21.15.98:0/365883292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 0x556168607b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.877+0000 7f65bffff640 1 -- 172.21.15.98:0/365883292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 msgr2=0x556168603a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.877+0000 7f65bffff640 1 --2- 172.21.15.98:0/365883292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x556168603a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.877+0000 7f65bffff640 1 -- 172.21.15.98:0/365883292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561686081e0 con 0x55616860c320 2024-05-09T08:53:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.881+0000 7f65bffff640 1 --2- 172.21.15.98:0/365883292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x556168609ab0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f65b80b8290 tx=0x7f65b802a6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d8084b1ab217ff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.884+0000 7f65a57fa640 1 -- 172.21.15.98:0/365883292 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b80c44a0 con 0x55616860c320 2024-05-09T08:53:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.884+0000 7f65a57fa640 1 -- 172.21.15.98:0/365883292 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65b80c40b0 con 0x55616860c320 2024-05-09T08:53:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.884+0000 7f65a57fa640 1 -- 172.21.15.98:0/365883292 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b80ef2f0 con 0x55616860c320 2024-05-09T08:53:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.885+0000 7f65c8a06740 1 -- 172.21.15.98:0/365883292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 msgr2=0x556168609ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.885+0000 7f65c8a06740 1 --2- 172.21.15.98:0/365883292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x556168609ab0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f65b80b8290 tx=0x7f65b802a6e0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.887+0000 7f65c8a06740 1 -- 172.21.15.98:0/365883292 shutdown_connections 2024-05-09T08:53:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.887+0000 7f65c8a06740 1 --2- 172.21.15.98:0/365883292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x556168609ab0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.887+0000 7f65c8a06740 1 --2- 172.21.15.98:0/365883292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x556168603a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.887+0000 7f65c8a06740 1 --2- 172.21.15.98:0/365883292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 0x556168607b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.887+0000 7f65c8a06740 1 -- 172.21.15.98:0/365883292 >> 172.21.15.98:0/365883292 conn(0x5561685fd650 msgr2=0x55616860b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.890+0000 7f65c8a06740 1 -- 172.21.15.98:0/365883292 shutdown_connections 2024-05-09T08:53:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.893+0000 7f65c8a06740 1 -- 172.21.15.98:0/365883292 wait complete. 2024-05-09T08:53:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.894+0000 7f65c8a06740 1 -- start start 2024-05-09T08:53:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.895+0000 7f65c8a06740 1 Processor -- start 2024-05-09T08:53:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.895+0000 7f65c8a06740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 0x556168606190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.896+0000 7f65c8a06740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x5561686066d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.896+0000 7f65c8a06740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x556168681c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.896+0000 7f65c8a06740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65c01c74c0 con 0x55616860c320 2024-05-09T08:53:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.896+0000 7f65c8a06740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f65c01c6df0 con 0x556168601c50 2024-05-09T08:53:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.896+0000 7f65c8a06740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f65c01192d0 con 0x556168601330 2024-05-09T08:53:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.896+0000 7f65beffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x5561686066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.896+0000 7f65beffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x5561686066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:49546/0 (socket says 172.21.15.98:49546) 2024-05-09T08:53:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.896+0000 7f65beffd640 1 -- 172.21.15.98:0/1964024625 learned_addr learned my addr 172.21.15.98:0/1964024625 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.897+0000 7f65c8a06740 10 client.0 _ll_register_callbacks cb 0x7f65b8009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.898+0000 7f65bffff640 1 --2- 172.21.15.98:0/1964024625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x556168681c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.900+0000 7f65bffff640 1 -- 172.21.15.98:0/1964024625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 msgr2=0x556168606190 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:53:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.900+0000 7f65bffff640 1 --2- 172.21.15.98:0/1964024625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 0x556168606190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.900+0000 7f65bffff640 1 -- 172.21.15.98:0/1964024625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 msgr2=0x5561686066d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.900+0000 7f65bffff640 1 --2- 172.21.15.98:0/1964024625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x5561686066d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.900+0000 7f65bffff640 1 -- 172.21.15.98:0/1964024625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556168601a80 con 0x55616860c320 2024-05-09T08:53:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.909+0000 7f65bffff640 1 --2- 172.21.15.98:0/1964024625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x556168681c90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x5561685fe140 tx=0x7f65b80ef520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d993696553b27cb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:06.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.910+0000 7f659effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:06.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.914+0000 7f659effd640 1 -- 172.21.15.98:0/1964024625 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b80c44a0 con 0x55616860c320 2024-05-09T08:53:06.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.914+0000 7f65c8a06740 1 -- 172.21.15.98:0/1964024625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65b80c40b0 con 0x55616860c320 2024-05-09T08:53:06.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.914+0000 7f659effd640 1 -- 172.21.15.98:0/1964024625 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f65b8003d80 con 0x55616860c320 2024-05-09T08:53:06.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.915+0000 7f659effd640 1 -- 172.21.15.98:0/1964024625 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f65b8004b40 con 0x55616860c320 2024-05-09T08:53:06.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.917+0000 7f65c8a06740 10 client.16461 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:06.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.917+0000 7f65c8a06740 1 -- 172.21.15.98:0/1964024625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556168601a80 con 0x55616860c320 2024-05-09T08:53:06.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.920+0000 7f659effd640 1 -- 172.21.15.98:0/1964024625 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65b8004ce0 con 0x55616860c320 2024-05-09T08:53:06.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.921+0000 7f659effd640 1 -- 172.21.15.98:0/1964024625 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f65b80cc030 con 0x55616860c320 2024-05-09T08:53:06.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.923+0000 7f65c8a06740 10 client.16461 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:06.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.923+0000 7f65c8a06740 10 client.16461 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:06.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.923+0000 7f65c8a06740 20 client.16461 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:06.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.923+0000 7f65c8a06740 1 --2- 172.21.15.98:0/1964024625 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556168683310 0x556168685700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:06.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.923+0000 7f65c8a06740 4 client.16461 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:53:06.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.923+0000 7f65c8a06740 1 -- 172.21.15.98:0/1964024625 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f65b80c40b0 con 0x556168683310 2024-05-09T08:53:06.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.923+0000 7f65bf7fe640 1 --2- 172.21.15.98:0/1964024625 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556168683310 0x556168685700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.927+0000 7f65bf7fe640 1 --2- 172.21.15.98:0/1964024625 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556168683310 0x556168685700 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c467e441efd99e6 server_cookie=11ce730f4a8e84e6 in_seq=0 out_seq=0 2024-05-09T08:53:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.927+0000 7f659effd640 10 client.16461 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:06.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.932+0000 7f659effd640 1 -- 172.21.15.98:0/1964024625 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f65b80c40b0 con 0x556168683310 2024-05-09T08:53:06.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.932+0000 7f659effd640 10 client.16461 handle_command_reply: tid=0 2024-05-09T08:53:06.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.935+0000 7f65c8a06740 1 client.16461 shutdown 2024-05-09T08:53:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.936+0000 7f65c8a06740 1 -- 172.21.15.98:0/1964024625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 msgr2=0x556168681c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.936+0000 7f65c8a06740 1 --2- 172.21.15.98:0/1964024625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x556168681c90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x5561685fe140 tx=0x7f65b80ef520 comp rx=0 tx=0).stop 2024-05-09T08:53:06.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.941+0000 7f65c8a06740 1 -- 172.21.15.98:0/1964024625 shutdown_connections 2024-05-09T08:53:06.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.941+0000 7f65c8a06740 1 --2- 172.21.15.98:0/1964024625 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556168683310 0x556168685700 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.941+0000 7f65c8a06740 1 --2- 172.21.15.98:0/1964024625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55616860c320 0x556168681c90 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.941+0000 7f65c8a06740 1 --2- 172.21.15.98:0/1964024625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556168601c50 0x5561686066d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.941+0000 7f65c8a06740 1 --2- 172.21.15.98:0/1964024625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556168601330 0x556168606190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.941+0000 7f65c8a06740 1 -- 172.21.15.98:0/1964024625 >> 172.21.15.98:0/1964024625 conn(0x5561685fd650 msgr2=0x55616860a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.941+0000 7f659effd640 0 client.16461 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:06.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.943+0000 7f65c8a06740 1 -- 172.21.15.98:0/1964024625 shutdown_connections 2024-05-09T08:53:06.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.945+0000 7f65c8a06740 1 -- 172.21.15.98:0/1964024625 wait complete. 2024-05-09T08:53:06.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.950+0000 7f65c8a06740 20 client.16461 trim_cache size 0 max 16384 2024-05-09T08:53:06.953 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:53:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.956+0000 7f65c5cfc640 1 -- 172.21.15.98:0/3160630192 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f65a80a2d30 msgr2=0x7f65a80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.956+0000 7f65c5cfc640 1 --2- 172.21.15.98:0/3160630192 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f65a80a2d30 0x7f65a80a51f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f65b00093b0 tx=0x7f65b0009340 comp rx=0 tx=0).stop 2024-05-09T08:53:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.956+0000 7f65c5cfc640 1 -- 172.21.15.98:0/3160630192 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c010e1f0 msgr2=0x7f65c010a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.956+0000 7f65c5cfc640 1 --2- 172.21.15.98:0/3160630192 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c010e1f0 0x7f65c010a470 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f65b8034040 tx=0x7f65b80305d0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.957+0000 7f65c5cfc640 1 -- 172.21.15.98:0/3160630192 shutdown_connections 2024-05-09T08:53:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.957+0000 7f65c5cfc640 1 --2- 172.21.15.98:0/3160630192 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f65a80a2d30 0x7f65a80a51f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.957+0000 7f65c5cfc640 1 --2- 172.21.15.98:0/3160630192 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c010e1f0 0x7f65c010a470 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.957+0000 7f65c5cfc640 1 --2- 172.21.15.98:0/3160630192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65c0100f60 0x7f65c0109f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.957+0000 7f65c5cfc640 1 --2- 172.21.15.98:0/3160630192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65c0100640 0x7f65c01099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.957+0000 7f65c5cfc640 1 -- 172.21.15.98:0/3160630192 >> 172.21.15.98:0/3160630192 conn(0x7f65c0076ab0 msgr2=0x7f65c010f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.957+0000 7f65c5cfc640 1 -- 172.21.15.98:0/3160630192 shutdown_connections 2024-05-09T08:53:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:06.957+0000 7f65c5cfc640 1 -- 172.21.15.98:0/3160630192 wait complete. 2024-05-09T08:53:07.986 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:53:07.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:53:07.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:53:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.209+0000 7fbca1eae640 1 -- 172.21.15.98:0/256525367 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc9c108830 msgr2=0x7fbc9c108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.209+0000 7fbca1eae640 1 --2- 172.21.15.98:0/256525367 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc9c108830 0x7fbc9c108c90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c009860 tx=0x7fbc8c02a3f0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.209+0000 7fbca1eae640 1 -- 172.21.15.98:0/256525367 shutdown_connections 2024-05-09T08:53:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.209+0000 7fbca1eae640 1 --2- 172.21.15.98:0/256525367 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c1092b0 0x7fbc9c11d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.209+0000 7fbca1eae640 1 --2- 172.21.15.98:0/256525367 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc9c108830 0x7fbc9c108c90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.209+0000 7fbca1eae640 1 --2- 172.21.15.98:0/256525367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc9c10fed0 0x7fbc9c1102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.209+0000 7fbca1eae640 1 -- 172.21.15.98:0/256525367 >> 172.21.15.98:0/256525367 conn(0x7fbc9c06c7b0 msgr2=0x7fbc9c06cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.209+0000 7fbca1eae640 1 -- 172.21.15.98:0/256525367 shutdown_connections 2024-05-09T08:53:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.209+0000 7fbca1eae640 1 -- 172.21.15.98:0/256525367 wait complete. 2024-05-09T08:53:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.210+0000 7fbca1eae640 1 Processor -- start 2024-05-09T08:53:08.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.210+0000 7fbca1eae640 1 -- start start 2024-05-09T08:53:08.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.211+0000 7fbca1eae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c108830 0x7fbc9c1a6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.211+0000 7fbca1eae640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc9c1092b0 0x7fbc9c1a7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.211+0000 7fbca0eac640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c108830 0x7fbc9c1a6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.211+0000 7fbca0eac640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c108830 0x7fbc9c1a6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:59674/0 (socket says 172.21.15.98:59674) 2024-05-09T08:53:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.211+0000 7fbca0eac640 1 -- 172.21.15.98:0/3230741259 learned_addr learned my addr 172.21.15.98:0/3230741259 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.212+0000 7fbca1eae640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc9c10fed0 0x7fbc9c1ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.212+0000 7fbca1eae640 1 -- 172.21.15.98:0/3230741259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c120950 con 0x7fbc9c1092b0 2024-05-09T08:53:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.212+0000 7fbca16ad640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc9c10fed0 0x7fbc9c1ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.212+0000 7fbca1eae640 1 -- 172.21.15.98:0/3230741259 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c1207d0 con 0x7fbc9c10fed0 2024-05-09T08:53:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.212+0000 7fbca1eae640 1 -- 172.21.15.98:0/3230741259 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c120ad0 con 0x7fbc9c108830 2024-05-09T08:53:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.212+0000 7fbc9bfff640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc9c1092b0 0x7fbc9c1a7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.213+0000 7fbca0eac640 1 -- 172.21.15.98:0/3230741259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc9c10fed0 msgr2=0x7fbc9c1ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.213+0000 7fbca0eac640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc9c10fed0 0x7fbc9c1ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.213+0000 7fbca0eac640 1 -- 172.21.15.98:0/3230741259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc9c1092b0 msgr2=0x7fbc9c1a7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.213+0000 7fbca0eac640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc9c1092b0 0x7fbc9c1a7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.213+0000 7fbca0eac640 1 -- 172.21.15.98:0/3230741259 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9c1abe20 con 0x7fbc9c108830 2024-05-09T08:53:08.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.213+0000 7fbca0eac640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c108830 0x7fbc9c1a6e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400ed80 tx=0x7fbc9400c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:08.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.213+0000 7fbc99ffb640 1 -- 172.21.15.98:0/3230741259 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9401c070 con 0x7fbc9c108830 2024-05-09T08:53:08.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.214+0000 7fbc99ffb640 1 -- 172.21.15.98:0/3230741259 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc94017b20 con 0x7fbc9c108830 2024-05-09T08:53:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.214+0000 7fbca1eae640 1 -- 172.21.15.98:0/3230741259 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc9c1ac110 con 0x7fbc9c108830 2024-05-09T08:53:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.214+0000 7fbc99ffb640 1 -- 172.21.15.98:0/3230741259 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9400fcf0 con 0x7fbc9c108830 2024-05-09T08:53:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.214+0000 7fbca1eae640 1 -- 172.21.15.98:0/3230741259 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc9c1b39a0 con 0x7fbc9c108830 2024-05-09T08:53:08.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.217+0000 7fbc99ffb640 1 -- 172.21.15.98:0/3230741259 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fbc9401f070 con 0x7fbc9c108830 2024-05-09T08:53:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.219+0000 7fbca5e1e740 1 Processor -- start 2024-05-09T08:53:08.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.219+0000 7fbc99ffb640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbc840a2dc0 0x7fbc840a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.220+0000 7fbc99ffb640 1 -- 172.21.15.98:0/3230741259 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fbc940cf6c0 con 0x7fbc9c108830 2024-05-09T08:53:08.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.220+0000 7fbca16ad640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbc840a2dc0 0x7fbc840a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.221+0000 7fbca5e1e740 1 -- start start 2024-05-09T08:53:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.221+0000 7fbca5e1e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e5705e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.221+0000 7fbca5e1e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e571030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.221+0000 7fbca5e1e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 0x55a43e57dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.221+0000 7fbca5e1e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a43e4d5990 con 0x55a43e5722a0 2024-05-09T08:53:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.221+0000 7fbca5e1e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a43e47d5b0 con 0x55a43e57b930 2024-05-09T08:53:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.221+0000 7fbca5e1e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a43e4d1950 con 0x55a43e570bb0 2024-05-09T08:53:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.221+0000 7fbca16ad640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 0x55a43e57dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.221+0000 7fbca16ad640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 0x55a43e57dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:49568/0 (socket says 172.21.15.98:49568) 2024-05-09T08:53:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.221+0000 7fbca16ad640 1 -- 172.21.15.98:0/4074608902 learned_addr learned my addr 172.21.15.98:0/4074608902 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.222+0000 7fbc9bfff640 1 --2- 172.21.15.98:0/4074608902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e571030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.222+0000 7fbca0eac640 1 --2- 172.21.15.98:0/4074608902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e5705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.222+0000 7fbc9bfff640 1 -- 172.21.15.98:0/4074608902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 msgr2=0x55a43e57dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.222+0000 7fbc9bfff640 1 --2- 172.21.15.98:0/4074608902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 0x55a43e57dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.222+0000 7fbc9bfff640 1 -- 172.21.15.98:0/4074608902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 msgr2=0x55a43e5705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.222+0000 7fbc9bfff640 1 --2- 172.21.15.98:0/4074608902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e5705e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.222+0000 7fbc9bfff640 1 -- 172.21.15.98:0/4074608902 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a43e2b17e0 con 0x55a43e570bb0 2024-05-09T08:53:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.222+0000 7fbca16ad640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbc840a2dc0 0x7fbc840a5280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbc90006fd0 tx=0x7fbc90009060 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.223+0000 7fbc9bfff640 1 --2- 172.21.15.98:0/4074608902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e571030 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c009830 tx=0x7fbc8c030df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43da8c2b72e81346 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:08.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.223+0000 7fbc82ffd640 1 -- 172.21.15.98:0/4074608902 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c0305a0 con 0x55a43e570bb0 2024-05-09T08:53:08.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.223+0000 7fbc82ffd640 1 -- 172.21.15.98:0/4074608902 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc8c040070 con 0x55a43e570bb0 2024-05-09T08:53:08.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.224+0000 7fbc82ffd640 1 -- 172.21.15.98:0/4074608902 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c004b80 con 0x55a43e570bb0 2024-05-09T08:53:08.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.224+0000 7fbca5e1e740 1 -- 172.21.15.98:0/4074608902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 msgr2=0x55a43e571030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.224+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/4074608902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e571030 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c009830 tx=0x7fbc8c030df0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.225+0000 7fbca5e1e740 1 -- 172.21.15.98:0/4074608902 shutdown_connections 2024-05-09T08:53:08.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.225+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/4074608902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 0x55a43e57dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.225+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/4074608902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e571030 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.225+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/4074608902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e5705e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.225+0000 7fbca5e1e740 1 -- 172.21.15.98:0/4074608902 >> 172.21.15.98:0/4074608902 conn(0x55a43e5390a0 msgr2=0x55a43e56d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:08.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.226+0000 7fbca5e1e740 1 -- 172.21.15.98:0/4074608902 shutdown_connections 2024-05-09T08:53:08.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.226+0000 7fbca5e1e740 1 -- 172.21.15.98:0/4074608902 wait complete. 2024-05-09T08:53:08.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.227+0000 7fbca5e1e740 1 -- start start 2024-05-09T08:53:08.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.227+0000 7fbca5e1e740 1 Processor -- start 2024-05-09T08:53:08.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.228+0000 7fbca5e1e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e570bb0 0x55a43e5787c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.228+0000 7fbca5e1e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e578d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.228+0000 7fbca5e1e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e57b930 0x55a43e579240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.228+0000 7fbca5e1e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a43e58f810 con 0x55a43e5722a0 2024-05-09T08:53:08.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.228+0000 7fbca5e1e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a43e490b40 con 0x55a43e570bb0 2024-05-09T08:53:08.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.228+0000 7fbca5e1e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a43e58f990 con 0x55a43e57b930 2024-05-09T08:53:08.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.228+0000 7fbca5e1e740 10 client.0 _ll_register_callbacks cb 0x55a43e4850f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.228+0000 7fbc9bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e578d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.228+0000 7fbc9bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e578d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53512/0 (socket says 172.21.15.98:53512) 2024-05-09T08:53:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.228+0000 7fbc9bfff640 1 -- 172.21.15.98:0/364251867 learned_addr learned my addr 172.21.15.98:0/364251867 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.229+0000 7fbca0eac640 1 --2- 172.21.15.98:0/364251867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e570bb0 0x55a43e5787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.231+0000 7fbca0eac640 1 -- 172.21.15.98:0/364251867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e57b930 msgr2=0x55a43e579240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.231+0000 7fbca0eac640 1 --2- 172.21.15.98:0/364251867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e57b930 0x55a43e579240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.231+0000 7fbca0eac640 1 -- 172.21.15.98:0/364251867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 msgr2=0x55a43e578d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.231+0000 7fbca0eac640 1 --2- 172.21.15.98:0/364251867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e578d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.231+0000 7fbca0eac640 1 -- 172.21.15.98:0/364251867 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a43e311f70 con 0x55a43e570bb0 2024-05-09T08:53:08.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.232+0000 7fbca0eac640 1 --2- 172.21.15.98:0/364251867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e570bb0 0x55a43e5787c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbc940ccb60 tx=0x7fbc9409baa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3059c88deb9557f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:08.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.232+0000 7fbc7bfff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:53:08.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.233+0000 7fbc7bfff640 1 -- 172.21.15.98:0/364251867 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc94098050 con 0x55a43e570bb0 2024-05-09T08:53:08.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.233+0000 7fbc7bfff640 1 -- 172.21.15.98:0/364251867 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fbc940cf2b0 con 0x55a43e570bb0 2024-05-09T08:53:08.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.233+0000 7fbca5e1e740 1 -- 172.21.15.98:0/364251867 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc8c040070 con 0x55a43e570bb0 2024-05-09T08:53:08.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.233+0000 7fbc7bfff640 1 -- 172.21.15.98:0/364251867 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbc940981f0 con 0x55a43e570bb0 2024-05-09T08:53:08.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.234+0000 7fbca5e1e740 10 client.25817 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:08.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.234+0000 7fbca5e1e740 1 -- 172.21.15.98:0/364251867 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a43e1a9170 con 0x55a43e570bb0 2024-05-09T08:53:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.235+0000 7fbc7bfff640 1 -- 172.21.15.98:0/364251867 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc8c040070 con 0x55a43e570bb0 2024-05-09T08:53:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.235+0000 7fbc7bfff640 1 -- 172.21.15.98:0/364251867 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fbc9400b030 con 0x55a43e570bb0 2024-05-09T08:53:08.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.235+0000 7fbca5e1e740 10 client.25817 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:08.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.235+0000 7fbca5e1e740 10 client.25817 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:08.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.235+0000 7fbca5e1e740 20 client.25817 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:08.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.235+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/364251867 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a43e59c9d0 0x55a43e59ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.235+0000 7fbca5e1e740 4 client.25817 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:08.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.235+0000 7fbca5e1e740 1 -- 172.21.15.98:0/364251867 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a43e4e30c0 con 0x55a43e59c9d0 2024-05-09T08:53:08.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.236+0000 7fbc9bfff640 1 --2- 172.21.15.98:0/364251867 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a43e59c9d0 0x55a43e59ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.237+0000 7fbc9bfff640 1 --2- 172.21.15.98:0/364251867 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a43e59c9d0 0x55a43e59ee20 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d886f1dda4cd89e server_cookie=aea375caa0896c5c in_seq=0 out_seq=0 2024-05-09T08:53:08.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.237+0000 7fbc7bfff640 10 client.25817 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:08.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.239+0000 7fbc7bfff640 1 -- 172.21.15.98:0/364251867 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a43e4e30c0 con 0x55a43e59c9d0 2024-05-09T08:53:08.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.239+0000 7fbc7bfff640 10 client.25817 handle_command_reply: tid=0 2024-05-09T08:53:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.239+0000 7fbca5e1e740 1 client.25817 shutdown 2024-05-09T08:53:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.240+0000 7fbca5e1e740 1 -- 172.21.15.98:0/364251867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e570bb0 msgr2=0x55a43e5787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.240+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/364251867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e570bb0 0x55a43e5787c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbc940ccb60 tx=0x7fbc9409baa0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.240+0000 7fbca5e1e740 1 -- 172.21.15.98:0/364251867 shutdown_connections 2024-05-09T08:53:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.240+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/364251867 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a43e59c9d0 0x55a43e59ee20 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.240+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/364251867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e57b930 0x55a43e579240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.240+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/364251867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e578d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.240+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/364251867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e570bb0 0x55a43e5787c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.240+0000 7fbca5e1e740 1 -- 172.21.15.98:0/364251867 >> 172.21.15.98:0/364251867 conn(0x55a43e5390a0 msgr2=0x55a43e572f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.240+0000 7fbc7bfff640 0 client.25817 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.241+0000 7fbca5e1e740 1 -- 172.21.15.98:0/364251867 shutdown_connections 2024-05-09T08:53:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.241+0000 7fbca5e1e740 1 -- 172.21.15.98:0/364251867 wait complete. 2024-05-09T08:53:08.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.242+0000 7fbca5e1e740 20 client.25817 trim_cache size 0 max 16384 2024-05-09T08:53:08.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.285+0000 7fbca5e1e740 1 Processor -- start 2024-05-09T08:53:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.286+0000 7fbca5e1e740 1 -- start start 2024-05-09T08:53:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.286+0000 7fbca5e1e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e598760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.286+0000 7fbca5e1e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e56c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.286+0000 7fbca5e1e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 0x55a43e5a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.286+0000 7fbca5e1e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c1b5240 con 0x55a43e5722a0 2024-05-09T08:53:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.286+0000 7fbca5e1e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c1b6d80 con 0x55a43e57b930 2024-05-09T08:53:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.286+0000 7fbca5e1e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c1b5910 con 0x55a43e570bb0 2024-05-09T08:53:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.287+0000 7fbca0eac640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e598760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.287+0000 7fbca0eac640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e598760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:59692/0 (socket says 172.21.15.98:59692) 2024-05-09T08:53:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.287+0000 7fbca0eac640 1 -- 172.21.15.98:0/2059678918 learned_addr learned my addr 172.21.15.98:0/2059678918 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.287+0000 7fbc9bfff640 1 --2- 172.21.15.98:0/2059678918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e56c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.287+0000 7fbc9bfff640 1 -- 172.21.15.98:0/2059678918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 msgr2=0x55a43e598760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.287+0000 7fbc9bfff640 1 --2- 172.21.15.98:0/2059678918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e598760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.287+0000 7fbc9bfff640 1 -- 172.21.15.98:0/2059678918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 msgr2=0x55a43e5a46c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:53:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.287+0000 7fbc9bfff640 1 --2- 172.21.15.98:0/2059678918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 0x55a43e5a46c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.287+0000 7fbc9bfff640 1 -- 172.21.15.98:0/2059678918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a43e577700 con 0x55a43e5722a0 2024-05-09T08:53:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.288+0000 7fbc9bfff640 1 --2- 172.21.15.98:0/2059678918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e56c0f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c004700 tx=0x7fbc8c030bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c6a95440132b985 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.288+0000 7fbc827fc640 1 -- 172.21.15.98:0/2059678918 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c004040 con 0x55a43e5722a0 2024-05-09T08:53:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.288+0000 7fbc827fc640 1 -- 172.21.15.98:0/2059678918 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc8c040040 con 0x55a43e5722a0 2024-05-09T08:53:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.289+0000 7fbca5e1e740 1 -- 172.21.15.98:0/2059678918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 msgr2=0x55a43e56c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.289+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/2059678918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e56c0f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c004700 tx=0x7fbc8c030bd0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.291+0000 7fbca5e1e740 1 -- 172.21.15.98:0/2059678918 shutdown_connections 2024-05-09T08:53:08.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.291+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/2059678918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 0x55a43e5a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.291+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/2059678918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e56c0f0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.291+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/2059678918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e598760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.291+0000 7fbca5e1e740 1 -- 172.21.15.98:0/2059678918 >> 172.21.15.98:0/2059678918 conn(0x55a43e5390a0 msgr2=0x55a43e56b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.292+0000 7fbca5e1e740 1 -- 172.21.15.98:0/2059678918 shutdown_connections 2024-05-09T08:53:08.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.293+0000 7fbca5e1e740 1 -- 172.21.15.98:0/2059678918 wait complete. 2024-05-09T08:53:08.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.293+0000 7fbca5e1e740 1 -- start start 2024-05-09T08:53:08.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.294+0000 7fbca5e1e740 1 Processor -- start 2024-05-09T08:53:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.296+0000 7fbca5e1e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e5784f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.296+0000 7fbca5e1e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e578a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.296+0000 7fbca5e1e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 0x55a43e578f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.296+0000 7fbca5e1e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c1b66b0 con 0x55a43e5722a0 2024-05-09T08:53:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.296+0000 7fbca5e1e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c1b5fe0 con 0x55a43e57b930 2024-05-09T08:53:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.296+0000 7fbca5e1e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c118640 con 0x55a43e570bb0 2024-05-09T08:53:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.296+0000 7fbca5e1e740 10 client.0 _ll_register_callbacks cb 0x7fbc94097c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.296+0000 7fbca0eac640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e5784f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.297+0000 7fbca0eac640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e5784f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:59704/0 (socket says 172.21.15.98:59704) 2024-05-09T08:53:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.297+0000 7fbca0eac640 1 -- 172.21.15.98:0/4027032028 learned_addr learned my addr 172.21.15.98:0/4027032028 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.297+0000 7fbca16ad640 1 --2- 172.21.15.98:0/4027032028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 0x55a43e578f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.297+0000 7fbc9bfff640 1 --2- 172.21.15.98:0/4027032028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e578a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.297+0000 7fbca0eac640 1 -- 172.21.15.98:0/4027032028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 msgr2=0x55a43e578f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.297+0000 7fbca0eac640 1 --2- 172.21.15.98:0/4027032028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 0x55a43e578f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.297+0000 7fbca0eac640 1 -- 172.21.15.98:0/4027032028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 msgr2=0x55a43e578a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.297+0000 7fbca0eac640 1 --2- 172.21.15.98:0/4027032028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e578a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.297+0000 7fbca0eac640 1 -- 172.21.15.98:0/4027032028 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a43e592ff0 con 0x55a43e570bb0 2024-05-09T08:53:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.298+0000 7fbca0eac640 1 --2- 172.21.15.98:0/4027032028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e5784f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbc940983b0 tx=0x7fbc94099ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7238e4be21811654 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.299+0000 7fbc7bfff640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:53:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.299+0000 7fbc7bfff640 1 -- 172.21.15.98:0/4027032028 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9407e060 con 0x55a43e570bb0 2024-05-09T08:53:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.299+0000 7fbc7bfff640 1 -- 172.21.15.98:0/4027032028 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fbc94017cd0 con 0x55a43e570bb0 2024-05-09T08:53:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.299+0000 7fbca5e1e740 1 -- 172.21.15.98:0/4027032028 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc8c040040 con 0x55a43e570bb0 2024-05-09T08:53:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.300+0000 7fbc7bfff640 1 -- 172.21.15.98:0/4027032028 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbc9409b340 con 0x55a43e570bb0 2024-05-09T08:53:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.300+0000 7fbca5e1e740 10 client.25771 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.300+0000 7fbca5e1e740 1 -- 172.21.15.98:0/4027032028 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a43e592ff0 con 0x55a43e570bb0 2024-05-09T08:53:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.300+0000 7fbc7bfff640 1 -- 172.21.15.98:0/4027032028 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc8c040040 con 0x55a43e570bb0 2024-05-09T08:53:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.300+0000 7fbc7bfff640 1 -- 172.21.15.98:0/4027032028 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fbc9409ae30 con 0x55a43e570bb0 2024-05-09T08:53:08.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.300+0000 7fbca5e1e740 10 client.25771 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:08.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.300+0000 7fbca5e1e740 10 client.25771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:08.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.300+0000 7fbca5e1e740 20 client.25771 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:08.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.300+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/4027032028 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a43e603710 0x55a43e605b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.300+0000 7fbca5e1e740 4 client.25771 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:53:08.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.300+0000 7fbca5e1e740 1 -- 172.21.15.98:0/4027032028 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a43e4dd410 con 0x55a43e603710 2024-05-09T08:53:08.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.301+0000 7fbc9bfff640 1 --2- 172.21.15.98:0/4027032028 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a43e603710 0x55a43e605b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.301+0000 7fbc9bfff640 1 --2- 172.21.15.98:0/4027032028 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a43e603710 0x55a43e605b30 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c747a25c42cd5f7e server_cookie=c6484be64fbffe5c in_seq=0 out_seq=0 2024-05-09T08:53:08.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.302+0000 7fbc7bfff640 10 client.25771 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:08.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.303+0000 7fbc7bfff640 1 -- 172.21.15.98:0/4027032028 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a43e4dd410 con 0x55a43e603710 2024-05-09T08:53:08.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.303+0000 7fbc7bfff640 10 client.25771 handle_command_reply: tid=0 2024-05-09T08:53:08.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.303+0000 7fbca5e1e740 1 client.25771 shutdown 2024-05-09T08:53:08.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.304+0000 7fbca5e1e740 1 -- 172.21.15.98:0/4027032028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 msgr2=0x55a43e5784f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.304+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/4027032028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e5784f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbc940983b0 tx=0x7fbc94099ef0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.304+0000 7fbca5e1e740 1 -- 172.21.15.98:0/4027032028 shutdown_connections 2024-05-09T08:53:08.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.304+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/4027032028 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a43e603710 0x55a43e605b30 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.304+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/4027032028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a43e57b930 0x55a43e578f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.304+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/4027032028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a43e5722a0 0x55a43e578a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.304+0000 7fbca5e1e740 1 --2- 172.21.15.98:0/4027032028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a43e570bb0 0x55a43e5784f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.304+0000 7fbca5e1e740 1 -- 172.21.15.98:0/4027032028 >> 172.21.15.98:0/4027032028 conn(0x55a43e5390a0 msgr2=0x55a43e56c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:08.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.304+0000 7fbc7bfff640 0 client.25771 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:08.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.305+0000 7fbca5e1e740 1 -- 172.21.15.98:0/4027032028 shutdown_connections 2024-05-09T08:53:08.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.305+0000 7fbca5e1e740 1 -- 172.21.15.98:0/4027032028 wait complete. 2024-05-09T08:53:08.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.305+0000 7fbca5e1e740 20 client.25771 trim_cache size 0 max 16384 2024-05-09T08:53:08.310 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:53:08.310 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:53:08.311 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "405565c6-1940-477a-a3e6-4ffddce98511", 2024-05-09T08:53:08.311 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:53:08.311 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:53:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.311+0000 7fbca1eae640 1 -- 172.21.15.98:0/3230741259 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbc840a2dc0 msgr2=0x7fbc840a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.311+0000 7fbca1eae640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbc840a2dc0 0x7fbc840a5280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbc90006fd0 tx=0x7fbc90009060 comp rx=0 tx=0).stop 2024-05-09T08:53:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.311+0000 7fbca1eae640 1 -- 172.21.15.98:0/3230741259 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c108830 msgr2=0x7fbc9c1a6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.311+0000 7fbca1eae640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c108830 0x7fbc9c1a6e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400ed80 tx=0x7fbc9400c6e0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.313+0000 7fbca1eae640 1 -- 172.21.15.98:0/3230741259 shutdown_connections 2024-05-09T08:53:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.313+0000 7fbca1eae640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbc840a2dc0 0x7fbc840a5280 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.313+0000 7fbca1eae640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc9c10fed0 0x7fbc9c1ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.313+0000 7fbca1eae640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc9c1092b0 0x7fbc9c1a7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.314+0000 7fbca1eae640 1 --2- 172.21.15.98:0/3230741259 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c108830 0x7fbc9c1a6e30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.314+0000 7fbca1eae640 1 -- 172.21.15.98:0/3230741259 >> 172.21.15.98:0/3230741259 conn(0x7fbc9c06c7b0 msgr2=0x7fbc9c070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:08.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.314+0000 7fbca1eae640 1 -- 172.21.15.98:0/3230741259 shutdown_connections 2024-05-09T08:53:08.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.314+0000 7fbca1eae640 1 -- 172.21.15.98:0/3230741259 wait complete. 2024-05-09T08:53:08.366 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:53:08.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:53:08.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:08 smithi098 ceph-mon[30092]: pgmap v77: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 865 MiB/s wr, 1.65k op/s 2024-05-09T08:53:08.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.619+0000 7f64aa6d9640 1 -- 172.21.15.98:0/3654801822 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a41092b0 msgr2=0x7f64a411d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.619+0000 7f64aa6d9640 1 --2- 172.21.15.98:0/3654801822 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a41092b0 0x7f64a411d020 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6494009860 tx=0x7f649402a410 comp rx=0 tx=0).stop 2024-05-09T08:53:08.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.620+0000 7f64aa6d9640 1 -- 172.21.15.98:0/3654801822 shutdown_connections 2024-05-09T08:53:08.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.620+0000 7f64aa6d9640 1 --2- 172.21.15.98:0/3654801822 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a41092b0 0x7f64a411d020 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.620+0000 7f64aa6d9640 1 --2- 172.21.15.98:0/3654801822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a4108830 0x7f64a4108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.620+0000 7f64aa6d9640 1 --2- 172.21.15.98:0/3654801822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a410fed0 0x7f64a41102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.620+0000 7f64aa6d9640 1 -- 172.21.15.98:0/3654801822 >> 172.21.15.98:0/3654801822 conn(0x7f64a406c7b0 msgr2=0x7f64a406cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:08.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.623+0000 7f64aa6d9640 1 -- 172.21.15.98:0/3654801822 shutdown_connections 2024-05-09T08:53:08.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.625+0000 7f64aa6d9640 1 -- 172.21.15.98:0/3654801822 wait complete. 2024-05-09T08:53:08.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.628+0000 7f64aa6d9640 1 Processor -- start 2024-05-09T08:53:08.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.633+0000 7f64aa6d9640 1 -- start start 2024-05-09T08:53:08.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.633+0000 7f64aa6d9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a4108830 0x7f64a41a6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.633+0000 7f64aa6d9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a41092b0 0x7f64a41a7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.633+0000 7f64aa6d9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a410fed0 0x7f64a41ab610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.633+0000 7f64aa6d9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f64a41208b0 con 0x7f64a41092b0 2024-05-09T08:53:08.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.633+0000 7f64aa6d9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f64a4120730 con 0x7f64a4108830 2024-05-09T08:53:08.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.633+0000 7f64aa6d9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f64a4120a30 con 0x7f64a410fed0 2024-05-09T08:53:08.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.634+0000 7f64a9ed8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a410fed0 0x7f64a41ab610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.634+0000 7f64a9ed8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a410fed0 0x7f64a41ab610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:59726/0 (socket says 172.21.15.98:59726) 2024-05-09T08:53:08.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.634+0000 7f64a9ed8640 1 -- 172.21.15.98:0/9639011 learned_addr learned my addr 172.21.15.98:0/9639011 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:08.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.636+0000 7f64a8ed6640 1 --2- 172.21.15.98:0/9639011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a41092b0 0x7f64a41a7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.637+0000 7f64a9ed8640 1 -- 172.21.15.98:0/9639011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a4108830 msgr2=0x7f64a41a6d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:53:08.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.637+0000 7f64a9ed8640 1 --2- 172.21.15.98:0/9639011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a4108830 0x7f64a41a6d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.637+0000 7f64a9ed8640 1 -- 172.21.15.98:0/9639011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a41092b0 msgr2=0x7f64a41a7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.637+0000 7f64a9ed8640 1 --2- 172.21.15.98:0/9639011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a41092b0 0x7f64a41a7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.637+0000 7f64a9ed8640 1 -- 172.21.15.98:0/9639011 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a41abd40 con 0x7f64a410fed0 2024-05-09T08:53:08.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.639+0000 7f64a9ed8640 1 --2- 172.21.15.98:0/9639011 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a410fed0 0x7f64a41ab610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f649402a980 tx=0x7f6494030960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:08.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.641+0000 7f649a7fc640 1 -- 172.21.15.98:0/9639011 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6494040020 con 0x7f64a410fed0 2024-05-09T08:53:08.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.641+0000 7f64aa6d9640 1 -- 172.21.15.98:0/9639011 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64a41abfd0 con 0x7f64a410fed0 2024-05-09T08:53:08.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.641+0000 7f64aa6d9640 1 -- 172.21.15.98:0/9639011 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64a41b38c0 con 0x7f64a410fed0 2024-05-09T08:53:08.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.646+0000 7f64ae647740 1 Processor -- start 2024-05-09T08:53:08.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.646+0000 7f649a7fc640 1 -- 172.21.15.98:0/9639011 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6494005550 con 0x7f64a410fed0 2024-05-09T08:53:08.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.648+0000 7f649a7fc640 1 -- 172.21.15.98:0/9639011 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6494004920 con 0x7f64a410fed0 2024-05-09T08:53:08.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.649+0000 7f64ae647740 1 -- start start 2024-05-09T08:53:08.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.649+0000 7f64ae647740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900430 0x55de27900810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.649+0000 7f64ae647740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900d50 0x55de2790aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.649+0000 7f64ae647740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de2790d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.649+0000 7f64ae647740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55de27865d00 con 0x55de27900d50 2024-05-09T08:53:08.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.649+0000 7f64ae647740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55de2780d4c0 con 0x55de2790b420 2024-05-09T08:53:08.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.649+0000 7f64ae647740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55de27861cc0 con 0x55de27900430 2024-05-09T08:53:08.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.651+0000 7f64a8ed6640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de2790d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.651+0000 7f64a8ed6640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de2790d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:49626/0 (socket says 172.21.15.98:49626) 2024-05-09T08:53:08.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.651+0000 7f64a8ed6640 1 -- 172.21.15.98:0/2820199619 learned_addr learned my addr 172.21.15.98:0/2820199619 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:08.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.651+0000 7f64a8ed6640 1 --2- 172.21.15.98:0/2820199619 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900430 0x55de27900810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.651+0000 7f64a96d7640 1 --2- 172.21.15.98:0/2820199619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900d50 0x55de2790aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.652+0000 7f64a8ed6640 1 -- 172.21.15.98:0/2820199619 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900430 msgr2=0x55de27900810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.652+0000 7f64a8ed6640 1 --2- 172.21.15.98:0/2820199619 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900430 0x55de27900810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.652+0000 7f64a8ed6640 1 -- 172.21.15.98:0/2820199619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900d50 msgr2=0x55de2790aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.652+0000 7f64a8ed6640 1 --2- 172.21.15.98:0/2820199619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900d50 0x55de2790aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.652+0000 7f64a8ed6640 1 -- 172.21.15.98:0/2820199619 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de276437d0 con 0x55de2790b420 2024-05-09T08:53:08.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.652+0000 7f64a8ed6640 1 --2- 172.21.15.98:0/2820199619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de2790d810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f649c0091e0 tx=0x7f649c010040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca1280ece155eb46 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:08.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.653+0000 7f648f7fe640 1 -- 172.21.15.98:0/2820199619 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c00cc00 con 0x55de2790b420 2024-05-09T08:53:08.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.653+0000 7f648f7fe640 1 -- 172.21.15.98:0/2820199619 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f649c00cda0 con 0x55de2790b420 2024-05-09T08:53:08.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.653+0000 7f648f7fe640 1 -- 172.21.15.98:0/2820199619 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c004420 con 0x55de2790b420 2024-05-09T08:53:08.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.653+0000 7f649a7fc640 1 -- 172.21.15.98:0/9639011 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6494043020 con 0x7f64a410fed0 2024-05-09T08:53:08.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.654+0000 7f649a7fc640 1 --2- 172.21.15.98:0/9639011 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f64900a2ce0 0x7f64900a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.655+0000 7f649a7fc640 1 -- 172.21.15.98:0/9639011 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f64940f06b0 con 0x7f64a410fed0 2024-05-09T08:53:08.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.655+0000 7f64ae647740 1 -- 172.21.15.98:0/2820199619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 msgr2=0x55de2790d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.655+0000 7f64ae647740 1 --2- 172.21.15.98:0/2820199619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de2790d810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f649c0091e0 tx=0x7f649c010040 comp rx=0 tx=0).stop 2024-05-09T08:53:08.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.656+0000 7f64a96d7640 1 --2- 172.21.15.98:0/9639011 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f64900a2ce0 0x7f64900a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:08 smithi142 ceph-mon[34625]: pgmap v77: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 865 MiB/s wr, 1.65k op/s 2024-05-09T08:53:08.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.657+0000 7f64ae647740 1 -- 172.21.15.98:0/2820199619 shutdown_connections 2024-05-09T08:53:08.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.658+0000 7f64ae647740 1 --2- 172.21.15.98:0/2820199619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de2790d810 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.658+0000 7f64ae647740 1 --2- 172.21.15.98:0/2820199619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900d50 0x55de2790aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.658+0000 7f64ae647740 1 --2- 172.21.15.98:0/2820199619 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900430 0x55de27900810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.658+0000 7f64ae647740 1 -- 172.21.15.98:0/2820199619 >> 172.21.15.98:0/2820199619 conn(0x55de278fc750 msgr2=0x55de278fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:08.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.659+0000 7f64ae647740 1 -- 172.21.15.98:0/2820199619 shutdown_connections 2024-05-09T08:53:08.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.659+0000 7f64ae647740 1 -- 172.21.15.98:0/2820199619 wait complete. 2024-05-09T08:53:08.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.660+0000 7f64ae647740 1 -- start start 2024-05-09T08:53:08.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.660+0000 7f64ae647740 1 Processor -- start 2024-05-09T08:53:08.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.661+0000 7f64a96d7640 1 --2- 172.21.15.98:0/9639011 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f64900a2ce0 0x7f64900a51a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55de279019f0 tx=0x7f64a0005ce0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:08.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.661+0000 7f64ae647740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900430 0x55de279083f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.661+0000 7f64ae647740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de27900d50 0x55de27908930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.661+0000 7f64ae647740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de2790b420 0x55de27908e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.661+0000 7f64ae647740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55de2790e1b0 con 0x55de27900430 2024-05-09T08:53:08.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.661+0000 7f64ae647740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55de27820b90 con 0x55de27900d50 2024-05-09T08:53:08.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.661+0000 7f64ae647740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55de2790e330 con 0x55de2790b420 2024-05-09T08:53:08.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.661+0000 7f64ae647740 10 client.0 _ll_register_callbacks cb 0x55de2785cda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:08.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.661+0000 7f64a9ed8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de2790b420 0x55de27908e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.661+0000 7f64a9ed8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de2790b420 0x55de27908e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:59746/0 (socket says 172.21.15.98:59746) 2024-05-09T08:53:08.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.661+0000 7f64a9ed8640 1 -- 172.21.15.98:0/1029863798 learned_addr learned my addr 172.21.15.98:0/1029863798 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:08.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.661+0000 7f64a96d7640 1 --2- 172.21.15.98:0/1029863798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900430 0x55de279083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.662+0000 7f64a96d7640 1 -- 172.21.15.98:0/1029863798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de2790b420 msgr2=0x55de27908e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.662+0000 7f64a96d7640 1 --2- 172.21.15.98:0/1029863798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de2790b420 0x55de27908e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.662+0000 7f64a96d7640 1 -- 172.21.15.98:0/1029863798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de27900d50 msgr2=0x55de27908930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.662+0000 7f64a96d7640 1 --2- 172.21.15.98:0/1029863798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de27900d50 0x55de27908930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.662+0000 7f64a96d7640 1 -- 172.21.15.98:0/1029863798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de275793f0 con 0x55de27900430 2024-05-09T08:53:08.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.662+0000 7f64a96d7640 1 --2- 172.21.15.98:0/1029863798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900430 0x55de279083f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f64a000c340 tx=0x7f64a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899765dc3ee55e45 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:08.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.663+0000 7f648cff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:08.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.666+0000 7f648cff9640 1 -- 172.21.15.98:0/1029863798 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0021070 con 0x55de27900430 2024-05-09T08:53:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.666+0000 7f64ae647740 1 -- 172.21.15.98:0/1029863798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f649c00cda0 con 0x55de27900430 2024-05-09T08:53:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.667+0000 7f648cff9640 1 -- 172.21.15.98:0/1029863798 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f64a000f990 con 0x55de27900430 2024-05-09T08:53:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.667+0000 7f648cff9640 1 -- 172.21.15.98:0/1029863798 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f64a002d050 con 0x55de27900430 2024-05-09T08:53:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.667+0000 7f64ae647740 10 client.16491 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.667+0000 7f64ae647740 1 -- 172.21.15.98:0/1029863798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55de275390f0 con 0x55de27900430 2024-05-09T08:53:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.669+0000 7f648cff9640 1 -- 172.21.15.98:0/1029863798 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f649c00cda0 con 0x55de27900430 2024-05-09T08:53:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.670+0000 7f648cff9640 1 -- 172.21.15.98:0/1029863798 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f64a00253b0 con 0x55de27900430 2024-05-09T08:53:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.670+0000 7f64ae647740 10 client.16491 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.670+0000 7f64ae647740 10 client.16491 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.670+0000 7f64ae647740 20 client.16491 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.670+0000 7f64ae647740 1 --2- 172.21.15.98:0/1029863798 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55de2791bb10 0x55de2791dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.670+0000 7f64ae647740 4 client.16491 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.670+0000 7f64ae647740 1 -- 172.21.15.98:0/1029863798 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de27873430 con 0x55de2791bb10 2024-05-09T08:53:08.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.670+0000 7f64a8ed6640 1 --2- 172.21.15.98:0/1029863798 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55de2791bb10 0x55de2791dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.674+0000 7f64a8ed6640 1 --2- 172.21.15.98:0/1029863798 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55de2791bb10 0x55de2791dfd0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdffbd24c9ba725f server_cookie=2119fe415e9f670a in_seq=0 out_seq=0 2024-05-09T08:53:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.674+0000 7f648cff9640 10 client.16491 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.676+0000 7f648cff9640 1 -- 172.21.15.98:0/1029863798 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55de27873430 con 0x55de2791bb10 2024-05-09T08:53:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.676+0000 7f648cff9640 10 client.16491 handle_command_reply: tid=0 2024-05-09T08:53:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.676+0000 7f64ae647740 1 client.16491 shutdown 2024-05-09T08:53:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.677+0000 7f64ae647740 1 -- 172.21.15.98:0/1029863798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900430 msgr2=0x55de279083f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.677+0000 7f64ae647740 1 --2- 172.21.15.98:0/1029863798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900430 0x55de279083f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f64a000c340 tx=0x7f64a00046c0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.678+0000 7f64ae647740 1 -- 172.21.15.98:0/1029863798 shutdown_connections 2024-05-09T08:53:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.678+0000 7f64ae647740 1 --2- 172.21.15.98:0/1029863798 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55de2791bb10 0x55de2791dfd0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.678+0000 7f64ae647740 1 --2- 172.21.15.98:0/1029863798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de2790b420 0x55de27908e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.678+0000 7f64ae647740 1 --2- 172.21.15.98:0/1029863798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de27900d50 0x55de27908930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.678+0000 7f64ae647740 1 --2- 172.21.15.98:0/1029863798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900430 0x55de279083f0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.678+0000 7f64ae647740 1 -- 172.21.15.98:0/1029863798 >> 172.21.15.98:0/1029863798 conn(0x55de278fc750 msgr2=0x55de278fba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:08.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.678+0000 7f648cff9640 0 client.16491 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:08.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.678+0000 7f64ae647740 1 -- 172.21.15.98:0/1029863798 shutdown_connections 2024-05-09T08:53:08.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.678+0000 7f64ae647740 1 -- 172.21.15.98:0/1029863798 wait complete. 2024-05-09T08:53:08.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.679+0000 7f64ae647740 20 client.16491 trim_cache size 0 max 16384 2024-05-09T08:53:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.724+0000 7f64ae647740 1 Processor -- start 2024-05-09T08:53:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.725+0000 7f64ae647740 1 -- start start 2024-05-09T08:53:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.725+0000 7f64ae647740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900430 0x55de27906ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.725+0000 7f64ae647740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900d50 0x55de27917100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.725+0000 7f64ae647740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de27902b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.725+0000 7f64ae647740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6488010090 con 0x55de27900430 2024-05-09T08:53:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.725+0000 7f64ae647740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6488011bd0 con 0x55de2790b420 2024-05-09T08:53:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.725+0000 7f64ae647740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6488010760 con 0x55de27900d50 2024-05-09T08:53:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.726+0000 7f64a9ed8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de27902b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.726+0000 7f64a9ed8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de27902b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:49636/0 (socket says 172.21.15.98:49636) 2024-05-09T08:53:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.726+0000 7f64a9ed8640 1 -- 172.21.15.98:0/700758502 learned_addr learned my addr 172.21.15.98:0/700758502 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.726+0000 7f64a8ed6640 1 --2- 172.21.15.98:0/700758502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900d50 0x55de27917100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.727+0000 7f64a8ed6640 1 -- 172.21.15.98:0/700758502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 msgr2=0x55de27902b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.727+0000 7f64a8ed6640 1 --2- 172.21.15.98:0/700758502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de27902b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.727+0000 7f64a8ed6640 1 -- 172.21.15.98:0/700758502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900430 msgr2=0x55de27906ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.727+0000 7f64a8ed6640 1 --2- 172.21.15.98:0/700758502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900430 0x55de27906ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.727+0000 7f64a8ed6640 1 -- 172.21.15.98:0/700758502 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de2790da40 con 0x55de27900d50 2024-05-09T08:53:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.729+0000 7f64a8ed6640 1 --2- 172.21.15.98:0/700758502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900d50 0x55de27917100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f649c002230 tx=0x7f649c00cf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c303c9d4cf636d73 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.729+0000 7f648effd640 1 -- 172.21.15.98:0/700758502 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c04de60 con 0x55de27900d50 2024-05-09T08:53:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.729+0000 7f648effd640 1 -- 172.21.15.98:0/700758502 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f649c051030 con 0x55de27900d50 2024-05-09T08:53:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.729+0000 7f648effd640 1 -- 172.21.15.98:0/700758502 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c042040 con 0x55de27900d50 2024-05-09T08:53:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.731+0000 7f64ae647740 1 -- 172.21.15.98:0/700758502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900d50 msgr2=0x55de27917100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.731+0000 7f64ae647740 1 --2- 172.21.15.98:0/700758502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900d50 0x55de27917100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f649c002230 tx=0x7f649c00cf50 comp rx=0 tx=0).stop 2024-05-09T08:53:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.733+0000 7f64ae647740 1 -- 172.21.15.98:0/700758502 shutdown_connections 2024-05-09T08:53:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.733+0000 7f64ae647740 1 --2- 172.21.15.98:0/700758502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de27902b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.733+0000 7f64ae647740 1 --2- 172.21.15.98:0/700758502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900d50 0x55de27917100 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.733+0000 7f64ae647740 1 --2- 172.21.15.98:0/700758502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900430 0x55de27906ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.733+0000 7f64ae647740 1 -- 172.21.15.98:0/700758502 >> 172.21.15.98:0/700758502 conn(0x55de278fc750 msgr2=0x55de27908c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:08.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:08 smithi105 ceph-mon[34498]: pgmap v77: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 865 MiB/s wr, 1.65k op/s 2024-05-09T08:53:08.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.737+0000 7f64ae647740 1 -- 172.21.15.98:0/700758502 shutdown_connections 2024-05-09T08:53:08.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.738+0000 7f64ae647740 1 -- 172.21.15.98:0/700758502 wait complete. 2024-05-09T08:53:08.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.738+0000 7f64ae647740 1 -- start start 2024-05-09T08:53:08.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.738+0000 7f64ae647740 1 Processor -- start 2024-05-09T08:53:08.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.739+0000 7f64ae647740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900430 0x55de27913590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.740+0000 7f64ae647740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900d50 0x55de27913ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.740+0000 7f64a96d7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900430 0x55de27913590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.740+0000 7f64a96d7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900430 0x55de27913590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:59766/0 (socket says 172.21.15.98:59766) 2024-05-09T08:53:08.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.740+0000 7f64a96d7640 1 -- 172.21.15.98:0/2519874913 learned_addr learned my addr 172.21.15.98:0/2519874913 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:08.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.740+0000 7f64ae647740 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de2790ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.740+0000 7f64ae647740 1 -- 172.21.15.98:0/2519874913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6488011500 con 0x55de27900d50 2024-05-09T08:53:08.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.740+0000 7f64a9ed8640 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de2790ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.740+0000 7f64ae647740 1 -- 172.21.15.98:0/2519874913 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6488010e30 con 0x55de2790b420 2024-05-09T08:53:08.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.740+0000 7f64ae647740 1 -- 172.21.15.98:0/2519874913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f648800e070 con 0x55de27900430 2024-05-09T08:53:08.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.741+0000 7f64ae647740 10 client.0 _ll_register_callbacks cb 0x7f64a0014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:08.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.741+0000 7f64a8ed6640 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900d50 0x55de27913ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.741+0000 7f64a9ed8640 1 -- 172.21.15.98:0/2519874913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900430 msgr2=0x55de27913590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.741+0000 7f64a9ed8640 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900430 0x55de27913590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.741+0000 7f64a9ed8640 1 -- 172.21.15.98:0/2519874913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900d50 msgr2=0x55de27913ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:08.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.741+0000 7f64a9ed8640 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900d50 0x55de27913ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.741+0000 7f64a9ed8640 1 -- 172.21.15.98:0/2519874913 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de2790f290 con 0x55de2790b420 2024-05-09T08:53:08.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.741+0000 7f64a8ed6640 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900d50 0x55de27913ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:53:08.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.744+0000 7f64a9ed8640 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de2790ea90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6494031960 tx=0x7f64940bc870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40103dc7544ff8d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:08.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.745+0000 7f648cff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:53:08.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.745+0000 7f648cff9640 1 -- 172.21.15.98:0/2519874913 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6494031400 con 0x55de2790b420 2024-05-09T08:53:08.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.745+0000 7f64ae647740 1 -- 172.21.15.98:0/2519874913 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f649c051030 con 0x55de2790b420 2024-05-09T08:53:08.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.745+0000 7f648cff9640 1 -- 172.21.15.98:0/2519874913 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f64940edc60 con 0x55de2790b420 2024-05-09T08:53:08.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.745+0000 7f648cff9640 1 -- 172.21.15.98:0/2519874913 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f64940ede60 con 0x55de2790b420 2024-05-09T08:53:08.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.746+0000 7f64ae647740 10 client.25829 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:08.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.746+0000 7f64ae647740 1 -- 172.21.15.98:0/2519874913 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55de27981d60 con 0x55de2790b420 2024-05-09T08:53:08.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.747+0000 7f648cff9640 1 -- 172.21.15.98:0/2519874913 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f649c051030 con 0x55de2790b420 2024-05-09T08:53:08.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.748+0000 7f648cff9640 1 -- 172.21.15.98:0/2519874913 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f64940c5060 con 0x55de2790b420 2024-05-09T08:53:08.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.748+0000 7f64ae647740 10 client.25829 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:08.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.748+0000 7f64ae647740 10 client.25829 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:08.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.748+0000 7f64ae647740 20 client.25829 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:08.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.748+0000 7f64ae647740 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55de279822a0 0x55de27984690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:08.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.748+0000 7f64ae647740 4 client.25829 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:53:08.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.748+0000 7f64ae647740 1 -- 172.21.15.98:0/2519874913 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55de2786d780 con 0x55de279822a0 2024-05-09T08:53:08.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.749+0000 7f64a96d7640 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55de279822a0 0x55de27984690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:08.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.750+0000 7f64a96d7640 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55de279822a0 0x55de27984690 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a215b4abf5d3824 server_cookie=e838f5f84658c206 in_seq=0 out_seq=0 2024-05-09T08:53:08.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.750+0000 7f648cff9640 10 client.25829 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:08.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.752+0000 7f648cff9640 1 -- 172.21.15.98:0/2519874913 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55de2786d780 con 0x55de279822a0 2024-05-09T08:53:08.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.752+0000 7f648cff9640 10 client.25829 handle_command_reply: tid=0 2024-05-09T08:53:08.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.753+0000 7f64ae647740 1 client.25829 shutdown 2024-05-09T08:53:08.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.757+0000 7f64ae647740 1 -- 172.21.15.98:0/2519874913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 msgr2=0x55de2790ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.757+0000 7f64ae647740 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de2790ea90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6494031960 tx=0x7f64940bc870 comp rx=0 tx=0).stop 2024-05-09T08:53:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.759+0000 7f64ae647740 1 -- 172.21.15.98:0/2519874913 shutdown_connections 2024-05-09T08:53:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.759+0000 7f64ae647740 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55de279822a0 0x55de27984690 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.760+0000 7f64ae647740 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de2790b420 0x55de2790ea90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.760+0000 7f64ae647740 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de27900d50 0x55de27913ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.760+0000 7f64ae647740 1 --2- 172.21.15.98:0/2519874913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55de27900430 0x55de27913590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.760+0000 7f64ae647740 1 -- 172.21.15.98:0/2519874913 >> 172.21.15.98:0/2519874913 conn(0x55de278fc750 msgr2=0x55de27908c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.760+0000 7f648cff9640 0 client.25829 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:08.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.764+0000 7f64ae647740 1 -- 172.21.15.98:0/2519874913 shutdown_connections 2024-05-09T08:53:08.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.765+0000 7f64ae647740 1 -- 172.21.15.98:0/2519874913 wait complete. 2024-05-09T08:53:08.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.767+0000 7f64ae647740 20 client.25829 trim_cache size 0 max 16384 2024-05-09T08:53:08.769 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:53:08.769 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "scrub active (0 inodes in the stack)", 2024-05-09T08:53:08.770 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:53:08.770 INFO:teuthology.orchestra.run.smithi098.stdout: "405565c6-1940-477a-a3e6-4ffddce98511": { 2024-05-09T08:53:08.770 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:53:08.770 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "405565c6-1940-477a-a3e6-4ffddce98511", 2024-05-09T08:53:08.770 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:53:08.770 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:53:08.770 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:53:08.770 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:53:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.772+0000 7f64aa6d9640 1 -- 172.21.15.98:0/9639011 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f64900a2ce0 msgr2=0x7f64900a51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.772+0000 7f64aa6d9640 1 --2- 172.21.15.98:0/9639011 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f64900a2ce0 0x7f64900a51a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55de279019f0 tx=0x7f64a0005ce0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.772+0000 7f64aa6d9640 1 -- 172.21.15.98:0/9639011 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a410fed0 msgr2=0x7f64a41ab610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.772+0000 7f64aa6d9640 1 --2- 172.21.15.98:0/9639011 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a410fed0 0x7f64a41ab610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f649402a980 tx=0x7f6494030960 comp rx=0 tx=0).stop 2024-05-09T08:53:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.773+0000 7f64aa6d9640 1 -- 172.21.15.98:0/9639011 shutdown_connections 2024-05-09T08:53:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.773+0000 7f64aa6d9640 1 --2- 172.21.15.98:0/9639011 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f64900a2ce0 0x7f64900a51a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.773+0000 7f64aa6d9640 1 --2- 172.21.15.98:0/9639011 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a410fed0 0x7f64a41ab610 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.773+0000 7f64aa6d9640 1 --2- 172.21.15.98:0/9639011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a41092b0 0x7f64a41a7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.773+0000 7f64aa6d9640 1 --2- 172.21.15.98:0/9639011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a4108830 0x7f64a41a6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.773+0000 7f64aa6d9640 1 -- 172.21.15.98:0/9639011 >> 172.21.15.98:0/9639011 conn(0x7f64a406c7b0 msgr2=0x7f64a40703f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:08.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.775+0000 7f64aa6d9640 1 -- 172.21.15.98:0/9639011 shutdown_connections 2024-05-09T08:53:08.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:08.775+0000 7f64aa6d9640 1 -- 172.21.15.98:0/9639011 wait complete. 2024-05-09T08:53:08.804 INFO:tasks.cephfs.filesystem:scrub status for tag:405565c6-1940-477a-a3e6-4ffddce98511 - {'path': '/', 'tag': '405565c6-1940-477a-a3e6-4ffddce98511', 'options': 'recursive,force'} 2024-05-09T08:53:09.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:09 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:53:09.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:09 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:53:09.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:09 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:53:09.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:09 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:53:09.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:09 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:53:09.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:09 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:53:09.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:09 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:53:09.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:09 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:53:09.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:09 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:53:10.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:10 smithi142 ceph-mon[34625]: pgmap v78: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 756 MiB/s wr, 1.65k op/s 2024-05-09T08:53:10.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:10 smithi105 ceph-mon[34498]: pgmap v78: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 756 MiB/s wr, 1.65k op/s 2024-05-09T08:53:10.840 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:10 smithi098 ceph-mon[30092]: pgmap v78: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 756 MiB/s wr, 1.65k op/s 2024-05-09T08:53:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:12 smithi142 ceph-mon[34625]: pgmap v79: 129 pgs: 129 active+clean; 11 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 858 MiB/s wr, 2.75k op/s 2024-05-09T08:53:12.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:12 smithi105 ceph-mon[34498]: pgmap v79: 129 pgs: 129 active+clean; 11 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 858 MiB/s wr, 2.75k op/s 2024-05-09T08:53:12.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:12 smithi098 ceph-mon[30092]: pgmap v79: 129 pgs: 129 active+clean; 11 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 858 MiB/s wr, 2.75k op/s 2024-05-09T08:53:14.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:14 smithi142 ceph-mon[34625]: pgmap v80: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 653 MiB/s wr, 2.37k op/s 2024-05-09T08:53:14.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:14 smithi105 ceph-mon[34498]: pgmap v80: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 653 MiB/s wr, 2.37k op/s 2024-05-09T08:53:14.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:14 smithi098 ceph-mon[30092]: pgmap v80: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 653 MiB/s wr, 2.37k op/s 2024-05-09T08:53:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:16 smithi142 ceph-mon[34625]: pgmap v81: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 630 MiB/s wr, 2.66k op/s 2024-05-09T08:53:16.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:16 smithi105 ceph-mon[34498]: pgmap v81: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 630 MiB/s wr, 2.66k op/s 2024-05-09T08:53:16.839 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:16 smithi098 ceph-mon[30092]: pgmap v81: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 630 MiB/s wr, 2.66k op/s 2024-05-09T08:53:17.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.913+0000 7f8995e1c640 1 -- 172.21.15.98:0/1105683569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f899011b010 msgr2=0x7f899011d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:17.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.913+0000 7f8995e1c640 1 --2- 172.21.15.98:0/1105683569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f899011b010 0x7f899011d400 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8988009c10 tx=0x7f898802a7a0 comp rx=0 tx=0).stop 2024-05-09T08:53:17.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.913+0000 7f8995e1c640 1 -- 172.21.15.98:0/1105683569 shutdown_connections 2024-05-09T08:53:17.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.913+0000 7f8995e1c640 1 --2- 172.21.15.98:0/1105683569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f899011b010 0x7f899011d400 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:17.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.913+0000 7f8995e1c640 1 --2- 172.21.15.98:0/1105683569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8990110400 0x7f8990108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:17.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.913+0000 7f8995e1c640 1 --2- 172.21.15.98:0/1105683569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990110020 0x7f8990108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:17.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.913+0000 7f8995e1c640 1 -- 172.21.15.98:0/1105683569 >> 172.21.15.98:0/1105683569 conn(0x7f899006c900 msgr2=0x7f899006cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:17.914 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.915+0000 7f8995e1c640 1 -- 172.21.15.98:0/1105683569 shutdown_connections 2024-05-09T08:53:17.914 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.915+0000 7f8995e1c640 1 -- 172.21.15.98:0/1105683569 wait complete. 2024-05-09T08:53:17.915 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.916+0000 7f8995e1c640 1 Processor -- start 2024-05-09T08:53:17.915 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.916+0000 7f8995e1c640 1 -- start start 2024-05-09T08:53:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.916+0000 7f8995e1c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8990110020 0x7f899011a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.916+0000 7f8995e1c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f899011b010 0x7f8990116880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.917+0000 7f8995e1c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990116dc0 0x7f8990117270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.917+0000 7f8995e1c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89901131e0 con 0x7f8990110020 2024-05-09T08:53:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.917+0000 7f8995e1c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8990113060 con 0x7f8990116dc0 2024-05-09T08:53:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.917+0000 7f8995e1c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8990113360 con 0x7f899011b010 2024-05-09T08:53:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.919+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8990110020 0x7f899011a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.919+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8990110020 0x7f899011a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38194/0 (socket says 172.21.15.98:38194) 2024-05-09T08:53:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.919+0000 7f898f7fe640 1 -- 172.21.15.98:0/3911694604 learned_addr learned my addr 172.21.15.98:0/3911694604 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.919+0000 7f898ffff640 1 --2- 172.21.15.98:0/3911694604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990116dc0 0x7f8990117270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.919+0000 7f898effd640 1 --2- 172.21.15.98:0/3911694604 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f899011b010 0x7f8990116880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:17.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.924+0000 7f898f7fe640 1 -- 172.21.15.98:0/3911694604 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f899011b010 msgr2=0x7f8990116880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:17.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.924+0000 7f898f7fe640 1 --2- 172.21.15.98:0/3911694604 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f899011b010 0x7f8990116880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:17.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.924+0000 7f898f7fe640 1 -- 172.21.15.98:0/3911694604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990116dc0 msgr2=0x7f8990117270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:17.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.924+0000 7f898f7fe640 1 --2- 172.21.15.98:0/3911694604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990116dc0 0x7f8990117270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:17.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.924+0000 7f898f7fe640 1 -- 172.21.15.98:0/3911694604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89901aae50 con 0x7f8990110020 2024-05-09T08:53:17.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.925+0000 7f898f7fe640 1 --2- 172.21.15.98:0/3911694604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8990110020 0x7f899011a7a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f898001ea80 tx=0x7f898001ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:17.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.927+0000 7f898cff9640 1 -- 172.21.15.98:0/3911694604 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8980028070 con 0x7f8990110020 2024-05-09T08:53:17.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.927+0000 7f898cff9640 1 -- 172.21.15.98:0/3911694604 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f898001ac20 con 0x7f8990110020 2024-05-09T08:53:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.928+0000 7f898cff9640 1 -- 172.21.15.98:0/3911694604 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8980020750 con 0x7f8990110020 2024-05-09T08:53:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.930+0000 7f8995e1c640 1 -- 172.21.15.98:0/3911694604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89901ab140 con 0x7f8990110020 2024-05-09T08:53:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.930+0000 7f8995e1c640 1 -- 172.21.15.98:0/3911694604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89901ab610 con 0x7f8990110020 2024-05-09T08:53:17.945 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.945+0000 7f8998c07740 1 Processor -- start 2024-05-09T08:53:17.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.948+0000 7f8998c07740 1 -- start start 2024-05-09T08:53:17.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.949+0000 7f8998c07740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8a450 0x55a2fac8a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:17.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.949+0000 7f898cff9640 1 -- 172.21.15.98:0/3911694604 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f89800208f0 con 0x7f8990110020 2024-05-09T08:53:17.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.949+0000 7f8998c07740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8ad70 0x55a2fac950f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:17.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.949+0000 7f8998c07740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 0x55a2fac97a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:17.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.949+0000 7f8998c07740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a2fabef340 con 0x55a2fac8a450 2024-05-09T08:53:17.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.949+0000 7f8998c07740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a2fab97290 con 0x55a2fac8ad70 2024-05-09T08:53:17.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.949+0000 7f8998c07740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a2fabeb300 con 0x55a2fac95630 2024-05-09T08:53:17.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.951+0000 7f898ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8a450 0x55a2fac8a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:17.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.951+0000 7f898ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8a450 0x55a2fac8a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38196/0 (socket says 172.21.15.98:38196) 2024-05-09T08:53:17.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.951+0000 7f898ffff640 1 -- 172.21.15.98:0/1539259531 learned_addr learned my addr 172.21.15.98:0/1539259531 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:17.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.951+0000 7f898f7fe640 1 --2- 172.21.15.98:0/1539259531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8ad70 0x55a2fac950f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:17.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.951+0000 7f898cff9640 1 --2- 172.21.15.98:0/3911694604 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89780a2ea0 0x7f89780a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:17.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.952+0000 7f898ffff640 1 --2- 172.21.15.98:0/3911694604 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89780a2ea0 0x7f89780a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:17.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.952+0000 7f898cff9640 1 -- 172.21.15.98:0/3911694604 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f89800e0240 con 0x7f8990110020 2024-05-09T08:53:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.954+0000 7f898ffff640 1 -- 172.21.15.98:0/1539259531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 msgr2=0x55a2fac97a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.954+0000 7f898ffff640 1 --2- 172.21.15.98:0/1539259531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 0x55a2fac97a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.954+0000 7f898ffff640 1 -- 172.21.15.98:0/1539259531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8ad70 msgr2=0x55a2fac950f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.954+0000 7f898ffff640 1 --2- 172.21.15.98:0/1539259531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8ad70 0x55a2fac950f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.954+0000 7f898ffff640 1 -- 172.21.15.98:0/1539259531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2fabbb8e0 con 0x55a2fac8a450 2024-05-09T08:53:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.955+0000 7f898ffff640 1 --2- 172.21.15.98:0/1539259531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8a450 0x55a2fac8a830 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f8988006010 tx=0x7f8988045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0177c01eb42a6fd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.955+0000 7f898ffff640 1 --2- 172.21.15.98:0/3911694604 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89780a2ea0 0x7f89780a5360 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8988005f00 tx=0x7f8988009880 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.955+0000 7f8975ffb640 1 -- 172.21.15.98:0/1539259531 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8988036040 con 0x55a2fac8a450 2024-05-09T08:53:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.955+0000 7f8975ffb640 1 -- 172.21.15.98:0/1539259531 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8988004d40 con 0x55a2fac8a450 2024-05-09T08:53:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.955+0000 7f8975ffb640 1 -- 172.21.15.98:0/1539259531 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8988004830 con 0x55a2fac8a450 2024-05-09T08:53:17.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.960+0000 7f8998c07740 1 -- 172.21.15.98:0/1539259531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8a450 msgr2=0x55a2fac8a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:17.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.960+0000 7f8998c07740 1 --2- 172.21.15.98:0/1539259531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8a450 0x55a2fac8a830 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f8988006010 tx=0x7f8988045000 comp rx=0 tx=0).stop 2024-05-09T08:53:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.960+0000 7f8998c07740 1 -- 172.21.15.98:0/1539259531 shutdown_connections 2024-05-09T08:53:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.960+0000 7f8998c07740 1 --2- 172.21.15.98:0/1539259531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 0x55a2fac97a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.961+0000 7f8998c07740 1 --2- 172.21.15.98:0/1539259531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8ad70 0x55a2fac950f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.961+0000 7f8998c07740 1 --2- 172.21.15.98:0/1539259531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8a450 0x55a2fac8a830 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.961+0000 7f8998c07740 1 -- 172.21.15.98:0/1539259531 >> 172.21.15.98:0/1539259531 conn(0x55a2fac86960 msgr2=0x55a2fac86d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:17.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.966+0000 7f8998c07740 1 -- 172.21.15.98:0/1539259531 shutdown_connections 2024-05-09T08:53:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.970+0000 7f8998c07740 1 -- 172.21.15.98:0/1539259531 wait complete. 2024-05-09T08:53:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.971+0000 7f8998c07740 1 -- start start 2024-05-09T08:53:17.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.971+0000 7f8998c07740 1 Processor -- start 2024-05-09T08:53:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.972+0000 7f8998c07740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8a450 0x55a2fac92610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.972+0000 7f8998c07740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8ad70 0x55a2fac92b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.972+0000 7f8998c07740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 0x55a2fac93090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.973+0000 7f8998c07740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a2fac98200 con 0x55a2fac8a450 2024-05-09T08:53:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.973+0000 7f8998c07740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a2fabaa790 con 0x55a2fac8ad70 2024-05-09T08:53:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.973+0000 7f8998c07740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a2fac98380 con 0x55a2fac95630 2024-05-09T08:53:17.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.973+0000 7f8998c07740 10 client.0 _ll_register_callbacks cb 0x55a2fabe5a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:17.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.975+0000 7f898ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 0x55a2fac93090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:17.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.975+0000 7f898ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 0x55a2fac93090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45592/0 (socket says 172.21.15.98:45592) 2024-05-09T08:53:17.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.975+0000 7f898ffff640 1 -- 172.21.15.98:0/2854862204 learned_addr learned my addr 172.21.15.98:0/2854862204 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:17.978 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.979+0000 7f898ffff640 1 -- 172.21.15.98:0/2854862204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8ad70 msgr2=0x55a2fac92b50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:53:17.978 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.979+0000 7f898ffff640 1 --2- 172.21.15.98:0/2854862204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8ad70 0x55a2fac92b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:17.978 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.979+0000 7f898ffff640 1 -- 172.21.15.98:0/2854862204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8a450 msgr2=0x55a2fac92610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:17.978 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.979+0000 7f898ffff640 1 --2- 172.21.15.98:0/2854862204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8a450 0x55a2fac92610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:17.978 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.979+0000 7f898ffff640 1 -- 172.21.15.98:0/2854862204 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2fa903210 con 0x55a2fac95630 2024-05-09T08:53:17.980 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.980+0000 7f898ffff640 1 --2- 172.21.15.98:0/2854862204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 0x55a2fac93090 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8988004fc0 tx=0x7f898802aca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0b64770fb428e2e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:17.980 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.980+0000 7f896effd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:53:17.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.983+0000 7f896effd640 1 -- 172.21.15.98:0/2854862204 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8988053060 con 0x55a2fac95630 2024-05-09T08:53:17.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.983+0000 7f8998c07740 1 -- 172.21.15.98:0/2854862204 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8988004d40 con 0x55a2fac95630 2024-05-09T08:53:17.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.983+0000 7f896effd640 1 -- 172.21.15.98:0/2854862204 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f8988030d40 con 0x55a2fac95630 2024-05-09T08:53:17.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.984+0000 7f896effd640 1 -- 172.21.15.98:0/2854862204 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f898805a050 con 0x55a2fac95630 2024-05-09T08:53:17.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.986+0000 7f8998c07740 10 client.25795 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:17.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.986+0000 7f8998c07740 1 -- 172.21.15.98:0/2854862204 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a2fa9cf620 con 0x55a2fac95630 2024-05-09T08:53:17.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.990+0000 7f896effd640 1 -- 172.21.15.98:0/2854862204 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8988004d40 con 0x55a2fac95630 2024-05-09T08:53:17.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.992+0000 7f896effd640 1 -- 172.21.15.98:0/2854862204 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8988005710 con 0x55a2fac95630 2024-05-09T08:53:17.997 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.994+0000 7f8998c07740 10 client.25795 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:17.997 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.994+0000 7f8998c07740 10 client.25795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:17.997 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.994+0000 7f8998c07740 20 client.25795 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:17.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.994+0000 7f8998c07740 1 --2- 172.21.15.98:0/2854862204 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a2faca4ff0 0x55a2faca73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:17.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.994+0000 7f8998c07740 4 client.25795 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:17.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:17.994+0000 7f8998c07740 1 -- 172.21.15.98:0/2854862204 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2fabfca70 con 0x55a2faca4ff0 2024-05-09T08:53:18.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.005+0000 7f898f7fe640 1 --2- 172.21.15.98:0/2854862204 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a2faca4ff0 0x55a2faca73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:18.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.019+0000 7f898f7fe640 1 --2- 172.21.15.98:0/2854862204 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a2faca4ff0 0x55a2faca73e0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f624b4b5ef1c82b server_cookie=b53d49614046e1c8 in_seq=0 out_seq=0 2024-05-09T08:53:18.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.022+0000 7f896effd640 10 client.25795 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:18.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.025+0000 7f896effd640 1 -- 172.21.15.98:0/2854862204 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a2fabfca70 con 0x55a2faca4ff0 2024-05-09T08:53:18.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.025+0000 7f896effd640 10 client.25795 handle_command_reply: tid=0 2024-05-09T08:53:18.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.028+0000 7f8998c07740 1 client.25795 shutdown 2024-05-09T08:53:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.033+0000 7f8998c07740 1 -- 172.21.15.98:0/2854862204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 msgr2=0x55a2fac93090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.033+0000 7f8998c07740 1 --2- 172.21.15.98:0/2854862204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 0x55a2fac93090 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8988004fc0 tx=0x7f898802aca0 comp rx=0 tx=0).stop 2024-05-09T08:53:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.036+0000 7f8998c07740 1 -- 172.21.15.98:0/2854862204 shutdown_connections 2024-05-09T08:53:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.036+0000 7f8998c07740 1 --2- 172.21.15.98:0/2854862204 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a2faca4ff0 0x55a2faca73e0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.036+0000 7f8998c07740 1 --2- 172.21.15.98:0/2854862204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 0x55a2fac93090 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.036+0000 7f8998c07740 1 --2- 172.21.15.98:0/2854862204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8ad70 0x55a2fac92b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.036+0000 7f8998c07740 1 --2- 172.21.15.98:0/2854862204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8a450 0x55a2fac92610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.036+0000 7f8998c07740 1 -- 172.21.15.98:0/2854862204 >> 172.21.15.98:0/2854862204 conn(0x55a2fac86960 msgr2=0x55a2fac8c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.037+0000 7f896effd640 0 client.25795 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:18.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.044+0000 7f8998c07740 1 -- 172.21.15.98:0/2854862204 shutdown_connections 2024-05-09T08:53:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.046+0000 7f8998c07740 1 -- 172.21.15.98:0/2854862204 wait complete. 2024-05-09T08:53:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.047+0000 7f8998c07740 20 client.25795 trim_cache size 0 max 16384 2024-05-09T08:53:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.078+0000 7f8998c07740 1 Processor -- start 2024-05-09T08:53:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.078+0000 7f8998c07740 1 -- start start 2024-05-09T08:53:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.078+0000 7f8998c07740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac8a450 0x55a2faca10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.079+0000 7f8998c07740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8ad70 0x55a2fac92ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.079+0000 7f8998c07740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac95630 0x55a2fac85880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.079+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac8a450 0x55a2faca10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.079+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac8a450 0x55a2faca10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45598/0 (socket says 172.21.15.98:45598) 2024-05-09T08:53:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.079+0000 7f898f7fe640 1 -- 172.21.15.98:0/304433394 learned_addr learned my addr 172.21.15.98:0/304433394 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.079+0000 7f8998c07740 1 -- 172.21.15.98:0/304433394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89901b5150 con 0x55a2fac8ad70 2024-05-09T08:53:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.079+0000 7f898ffff640 1 --2- 172.21.15.98:0/304433394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac95630 0x55a2fac85880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.079+0000 7f8998c07740 1 -- 172.21.15.98:0/304433394 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89901b6c90 con 0x55a2fac95630 2024-05-09T08:53:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.079+0000 7f8998c07740 1 -- 172.21.15.98:0/304433394 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f89901b5820 con 0x55a2fac8a450 2024-05-09T08:53:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.080+0000 7f898effd640 1 --2- 172.21.15.98:0/304433394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8ad70 0x55a2fac92ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.080+0000 7f898effd640 1 -- 172.21.15.98:0/304433394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac8a450 msgr2=0x55a2faca10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.080+0000 7f898effd640 1 --2- 172.21.15.98:0/304433394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac8a450 0x55a2faca10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.080+0000 7f898effd640 1 -- 172.21.15.98:0/304433394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac95630 msgr2=0x55a2fac85880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.080+0000 7f898effd640 1 --2- 172.21.15.98:0/304433394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac95630 0x55a2fac85880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.081+0000 7f898effd640 1 -- 172.21.15.98:0/304433394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2fac91620 con 0x55a2fac8ad70 2024-05-09T08:53:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.082+0000 7f898effd640 1 --2- 172.21.15.98:0/304433394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8ad70 0x55a2fac92ed0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x55a2fac87d80 tx=0x7f898400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8289683c0c47c5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.085+0000 7f89757fa640 1 -- 172.21.15.98:0/304433394 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89840125f0 con 0x55a2fac8ad70 2024-05-09T08:53:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.085+0000 7f89757fa640 1 -- 172.21.15.98:0/304433394 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8984018070 con 0x55a2fac8ad70 2024-05-09T08:53:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.085+0000 7f89757fa640 1 -- 172.21.15.98:0/304433394 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8984013be0 con 0x55a2fac8ad70 2024-05-09T08:53:18.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.086+0000 7f8998c07740 1 -- 172.21.15.98:0/304433394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8ad70 msgr2=0x55a2fac92ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.086+0000 7f8998c07740 1 --2- 172.21.15.98:0/304433394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8ad70 0x55a2fac92ed0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x55a2fac87d80 tx=0x7f898400b8d0 comp rx=0 tx=0).stop 2024-05-09T08:53:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.086+0000 7f8998c07740 1 -- 172.21.15.98:0/304433394 shutdown_connections 2024-05-09T08:53:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.086+0000 7f8998c07740 1 --2- 172.21.15.98:0/304433394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac95630 0x55a2fac85880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.086+0000 7f8998c07740 1 --2- 172.21.15.98:0/304433394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8ad70 0x55a2fac92ed0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.086+0000 7f8998c07740 1 --2- 172.21.15.98:0/304433394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac8a450 0x55a2faca10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.087+0000 7f8998c07740 1 -- 172.21.15.98:0/304433394 >> 172.21.15.98:0/304433394 conn(0x55a2fac86960 msgr2=0x55a2fac85170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:18.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.088+0000 7f8998c07740 1 -- 172.21.15.98:0/304433394 shutdown_connections 2024-05-09T08:53:18.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.088+0000 7f8998c07740 1 -- 172.21.15.98:0/304433394 wait complete. 2024-05-09T08:53:18.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.089+0000 7f8998c07740 1 -- start start 2024-05-09T08:53:18.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.089+0000 7f8998c07740 1 Processor -- start 2024-05-09T08:53:18.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.090+0000 7f8998c07740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8a450 0x55a2fac8f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:18.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.090+0000 7f8998c07740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8ad70 0x55a2fac8fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.090+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8a450 0x55a2fac8f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.090+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8a450 0x55a2fac8f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:43058/0 (socket says 172.21.15.98:43058) 2024-05-09T08:53:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.091+0000 7f898f7fe640 1 -- 172.21.15.98:0/3508031074 learned_addr learned my addr 172.21.15.98:0/3508031074 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.091+0000 7f8998c07740 1 --2- 172.21.15.98:0/3508031074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 0x55a2fac8ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.091+0000 7f8998c07740 1 -- 172.21.15.98:0/3508031074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89901b65c0 con 0x55a2fac8ad70 2024-05-09T08:53:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.091+0000 7f898ffff640 1 --2- 172.21.15.98:0/3508031074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 0x55a2fac8ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.091+0000 7f8998c07740 1 -- 172.21.15.98:0/3508031074 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89901b5ef0 con 0x55a2fac8a450 2024-05-09T08:53:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.091+0000 7f8998c07740 1 -- 172.21.15.98:0/3508031074 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f89901b31e0 con 0x55a2fac95630 2024-05-09T08:53:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.091+0000 7f898f7fe640 1 -- 172.21.15.98:0/3508031074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 msgr2=0x55a2fac8ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.091+0000 7f898f7fe640 1 --2- 172.21.15.98:0/3508031074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 0x55a2fac8ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.091+0000 7f898f7fe640 1 -- 172.21.15.98:0/3508031074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8ad70 msgr2=0x55a2fac8fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.091+0000 7f898f7fe640 1 --2- 172.21.15.98:0/3508031074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8ad70 0x55a2fac8fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.092+0000 7f898f7fe640 1 -- 172.21.15.98:0/3508031074 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2fad0a420 con 0x55a2fac8a450 2024-05-09T08:53:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.092+0000 7f898f7fe640 1 --2- 172.21.15.98:0/3508031074 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8a450 0x55a2fac8f510 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8980021fd0 tx=0x7f89800e2610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=590253a9eb7d8f0a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:18.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.092+0000 7f8998c07740 10 client.0 _ll_register_callbacks cb 0x7f8988009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:18.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.092+0000 7f896effd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:53:18.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.092+0000 7f896effd640 1 -- 172.21.15.98:0/3508031074 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89800c5070 con 0x55a2fac8a450 2024-05-09T08:53:18.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.092+0000 7f896effd640 1 -- 172.21.15.98:0/3508031074 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f89800dd7f0 con 0x55a2fac8a450 2024-05-09T08:53:18.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.093+0000 7f8998c07740 1 -- 172.21.15.98:0/3508031074 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8984018070 con 0x55a2fac8a450 2024-05-09T08:53:18.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.093+0000 7f896effd640 1 -- 172.21.15.98:0/3508031074 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f89800b0770 con 0x55a2fac8a450 2024-05-09T08:53:18.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.093+0000 7f8998c07740 10 client.25841 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:18.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.093+0000 7f8998c07740 1 -- 172.21.15.98:0/3508031074 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55a2fad0b5c0 con 0x55a2fac8a450 2024-05-09T08:53:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.095+0000 7f896effd640 1 -- 172.21.15.98:0/3508031074 <== mon.2 v2:172.21.15.105:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f898001f040 con 0x55a2fac8a450 2024-05-09T08:53:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.095+0000 7f8998c07740 10 client.25841 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.095+0000 7f8998c07740 10 client.25841 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.095+0000 7f8998c07740 20 client.25841 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.095+0000 7f8998c07740 1 --2- 172.21.15.98:0/3508031074 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a2fad0bc40 0x55a2fad0e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.095+0000 7f8998c07740 4 client.25841 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:53:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.096+0000 7f8998c07740 1 -- 172.21.15.98:0/3508031074 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f8984018070 con 0x55a2fad0bc40 2024-05-09T08:53:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.097+0000 7f898effd640 1 --2- 172.21.15.98:0/3508031074 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a2fad0bc40 0x55a2fad0e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:18.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.098+0000 7f898effd640 1 --2- 172.21.15.98:0/3508031074 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a2fad0bc40 0x55a2fad0e0c0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96974b7173594952 server_cookie=1ec3f27229b50dcf in_seq=0 out_seq=0 2024-05-09T08:53:18.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:18.098+0000 7f896effd640 10 client.25841 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:18 smithi142 ceph-mon[34625]: pgmap v82: 129 pgs: 129 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 765 MiB/s wr, 3.92k op/s 2024-05-09T08:53:18.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:18 smithi105 ceph-mon[34498]: pgmap v82: 129 pgs: 129 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 765 MiB/s wr, 3.92k op/s 2024-05-09T08:53:18.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:18 smithi098 ceph-mon[30092]: pgmap v82: 129 pgs: 129 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 765 MiB/s wr, 3.92k op/s 2024-05-09T08:53:20.641 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:20 smithi098 ceph-mon[30092]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 617 MiB/s wr, 3.92k op/s 2024-05-09T08:53:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:20 smithi142 ceph-mon[34625]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 617 MiB/s wr, 3.92k op/s 2024-05-09T08:53:20.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:20 smithi105 ceph-mon[34498]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 617 MiB/s wr, 3.92k op/s 2024-05-09T08:53:21.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.520+0000 7f896effd640 1 -- 172.21.15.98:0/3508031074 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f8984018070 con 0x55a2fad0bc40 2024-05-09T08:53:21.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.520+0000 7f896effd640 10 client.25841 handle_command_reply: tid=0 2024-05-09T08:53:21.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.520+0000 7f8998c07740 1 client.25841 shutdown 2024-05-09T08:53:21.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.520+0000 7f8998c07740 1 -- 172.21.15.98:0/3508031074 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8a450 msgr2=0x55a2fac8f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:21.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.520+0000 7f8998c07740 1 --2- 172.21.15.98:0/3508031074 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8a450 0x55a2fac8f510 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8980021fd0 tx=0x7f89800e2610 comp rx=0 tx=0).stop 2024-05-09T08:53:21.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.520+0000 7f8998c07740 1 -- 172.21.15.98:0/3508031074 shutdown_connections 2024-05-09T08:53:21.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.520+0000 7f8998c07740 1 --2- 172.21.15.98:0/3508031074 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a2fad0bc40 0x55a2fad0e0c0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:21.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.520+0000 7f8998c07740 1 --2- 172.21.15.98:0/3508031074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2fac95630 0x55a2fac8ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:21.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.520+0000 7f8998c07740 1 --2- 172.21.15.98:0/3508031074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2fac8ad70 0x55a2fac8fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:21.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.520+0000 7f8998c07740 1 --2- 172.21.15.98:0/3508031074 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a2fac8a450 0x55a2fac8f510 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:21.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.520+0000 7f8998c07740 1 -- 172.21.15.98:0/3508031074 >> 172.21.15.98:0/3508031074 conn(0x55a2fac86960 msgr2=0x55a2fac85a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:21.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.520+0000 7f896effd640 0 client.25841 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:21.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.521+0000 7f8998c07740 1 -- 172.21.15.98:0/3508031074 shutdown_connections 2024-05-09T08:53:21.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.521+0000 7f8998c07740 1 -- 172.21.15.98:0/3508031074 wait complete. 2024-05-09T08:53:21.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.521+0000 7f8998c07740 20 client.25841 trim_cache size 0 max 16384 2024-05-09T08:53:21.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:53:21.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:53:21.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:53:21.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:53:21.526 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.527+0000 7f8995e1c640 1 -- 172.21.15.98:0/3911694604 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89780a2ea0 msgr2=0x7f89780a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:21.526 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.527+0000 7f8995e1c640 1 --2- 172.21.15.98:0/3911694604 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89780a2ea0 0x7f89780a5360 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8988005f00 tx=0x7f8988009880 comp rx=0 tx=0).stop 2024-05-09T08:53:21.526 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.527+0000 7f8995e1c640 1 -- 172.21.15.98:0/3911694604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8990110020 msgr2=0x7f899011a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:21.526 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.527+0000 7f8995e1c640 1 --2- 172.21.15.98:0/3911694604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8990110020 0x7f899011a7a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f898001ea80 tx=0x7f898001ef40 comp rx=0 tx=0).stop 2024-05-09T08:53:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.528+0000 7f8995e1c640 1 -- 172.21.15.98:0/3911694604 shutdown_connections 2024-05-09T08:53:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.528+0000 7f8995e1c640 1 --2- 172.21.15.98:0/3911694604 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89780a2ea0 0x7f89780a5360 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.528+0000 7f8995e1c640 1 --2- 172.21.15.98:0/3911694604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8990116dc0 0x7f8990117270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.528+0000 7f8995e1c640 1 --2- 172.21.15.98:0/3911694604 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f899011b010 0x7f8990116880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.528+0000 7f8995e1c640 1 --2- 172.21.15.98:0/3911694604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8990110020 0x7f899011a7a0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.528+0000 7f8995e1c640 1 -- 172.21.15.98:0/3911694604 >> 172.21.15.98:0/3911694604 conn(0x7f899006c900 msgr2=0x7f899011b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.530+0000 7f8995e1c640 1 -- 172.21.15.98:0/3911694604 shutdown_connections 2024-05-09T08:53:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:21.530+0000 7f8995e1c640 1 -- 172.21.15.98:0/3911694604 wait complete. 2024-05-09T08:53:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:22 smithi142 ceph-mon[34625]: pgmap v84: 129 pgs: 129 active+clean; 15 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 751 MiB/s wr, 3.98k op/s 2024-05-09T08:53:22.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:22 smithi105 ceph-mon[34498]: pgmap v84: 129 pgs: 129 active+clean; 15 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 751 MiB/s wr, 3.98k op/s 2024-05-09T08:53:22.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:22 smithi098 ceph-mon[30092]: pgmap v84: 129 pgs: 129 active+clean; 15 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 751 MiB/s wr, 3.98k op/s 2024-05-09T08:53:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:24 smithi142 ceph-mon[34625]: pgmap v85: 129 pgs: 129 active+clean; 15 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 632 MiB/s wr, 2.89k op/s 2024-05-09T08:53:24.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:24 smithi105 ceph-mon[34498]: pgmap v85: 129 pgs: 129 active+clean; 15 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 632 MiB/s wr, 2.89k op/s 2024-05-09T08:53:24.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:24 smithi098 ceph-mon[30092]: pgmap v85: 129 pgs: 129 active+clean; 15 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 632 MiB/s wr, 2.89k op/s 2024-05-09T08:53:26.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:26 smithi142 ceph-mon[34625]: pgmap v86: 129 pgs: 129 active+clean; 16 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 666 MiB/s wr, 2.81k op/s 2024-05-09T08:53:26.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:26 smithi105 ceph-mon[34498]: pgmap v86: 129 pgs: 129 active+clean; 16 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 666 MiB/s wr, 2.81k op/s 2024-05-09T08:53:26.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:26 smithi098 ceph-mon[30092]: pgmap v86: 129 pgs: 129 active+clean; 16 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 666 MiB/s wr, 2.81k op/s 2024-05-09T08:53:27.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:27 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:53:27.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:27 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:53:27.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:27 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:53:28.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:28 smithi105 ceph-mon[34498]: pgmap v87: 129 pgs: 129 active+clean; 17 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 817 MiB/s wr, 2.91k op/s 2024-05-09T08:53:28.808 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:28 smithi142 ceph-mon[34625]: pgmap v87: 129 pgs: 129 active+clean; 17 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 817 MiB/s wr, 2.91k op/s 2024-05-09T08:53:28.840 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:28 smithi098 ceph-mon[30092]: pgmap v87: 129 pgs: 129 active+clean; 17 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 817 MiB/s wr, 2.91k op/s 2024-05-09T08:53:30.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:30 smithi105 ceph-mon[34498]: pgmap v88: 129 pgs: 129 active+clean; 17 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 668 MiB/s wr, 1.64k op/s 2024-05-09T08:53:30.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:30 smithi098 ceph-mon[30092]: pgmap v88: 129 pgs: 129 active+clean; 17 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 668 MiB/s wr, 1.64k op/s 2024-05-09T08:53:30.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:30 smithi142 ceph-mon[34625]: pgmap v88: 129 pgs: 129 active+clean; 17 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 668 MiB/s wr, 1.64k op/s 2024-05-09T08:53:31.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:31 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:53:31.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:31 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:53:31.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:31 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:53:32.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:32 smithi105 ceph-mon[34498]: pgmap v89: 129 pgs: 129 active+clean; 18 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 865 MiB/s wr, 2.00k op/s 2024-05-09T08:53:32.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:32 smithi098 ceph-mon[30092]: pgmap v89: 129 pgs: 129 active+clean; 18 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 865 MiB/s wr, 2.00k op/s 2024-05-09T08:53:32.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:32 smithi142 ceph-mon[34625]: pgmap v89: 129 pgs: 129 active+clean; 18 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 865 MiB/s wr, 2.00k op/s 2024-05-09T08:53:33.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:33 smithi105 ceph-mon[34498]: pgmap v90: 129 pgs: 129 active+clean; 18 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 699 MiB/s wr, 1.76k op/s 2024-05-09T08:53:33.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:33 smithi098 ceph-mon[30092]: pgmap v90: 129 pgs: 129 active+clean; 18 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 699 MiB/s wr, 1.76k op/s 2024-05-09T08:53:33.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:33 smithi142 ceph-mon[34625]: pgmap v90: 129 pgs: 129 active+clean; 18 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 699 MiB/s wr, 1.76k op/s 2024-05-09T08:53:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.855+0000 7efd6a973640 1 -- 172.21.15.98:0/3433542612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd6410ccd0 msgr2=0x7efd6410d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.855+0000 7efd6a973640 1 --2- 172.21.15.98:0/3433542612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd6410ccd0 0x7efd6410d0b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7efd54009900 tx=0x7efd5402a440 comp rx=0 tx=0).stop 2024-05-09T08:53:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.855+0000 7efd6a973640 1 -- 172.21.15.98:0/3433542612 shutdown_connections 2024-05-09T08:53:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.855+0000 7efd6a973640 1 --2- 172.21.15.98:0/3433542612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd6410e300 0x7efd64079a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.855+0000 7efd6a973640 1 --2- 172.21.15.98:0/3433542612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd6410d960 0x7efd6410ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.855+0000 7efd6a973640 1 --2- 172.21.15.98:0/3433542612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd6410ccd0 0x7efd6410d0b0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.855+0000 7efd6a973640 1 -- 172.21.15.98:0/3433542612 >> 172.21.15.98:0/3433542612 conn(0x7efd6406c770 msgr2=0x7efd6406cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.857+0000 7efd6a973640 1 -- 172.21.15.98:0/3433542612 shutdown_connections 2024-05-09T08:53:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.857+0000 7efd6a973640 1 -- 172.21.15.98:0/3433542612 wait complete. 2024-05-09T08:53:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.858+0000 7efd6a973640 1 Processor -- start 2024-05-09T08:53:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.858+0000 7efd6a973640 1 -- start start 2024-05-09T08:53:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.858+0000 7efd6a973640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd6410ccd0 0x7efd64073ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.858+0000 7efd6a973640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd6410d960 0x7efd64074410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.858+0000 7efd6a973640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd6410e300 0x7efd641bb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.858+0000 7efd6a973640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efd6407da30 con 0x7efd6410e300 2024-05-09T08:53:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.858+0000 7efd6a973640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efd6407d8b0 con 0x7efd6410ccd0 2024-05-09T08:53:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.858+0000 7efd6a973640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efd6407dbb0 con 0x7efd6410d960 2024-05-09T08:53:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.859+0000 7efd69170640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd6410d960 0x7efd64074410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.859+0000 7efd69170640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd6410d960 0x7efd64074410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:33416/0 (socket says 172.21.15.98:33416) 2024-05-09T08:53:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.859+0000 7efd69170640 1 -- 172.21.15.98:0/117215552 learned_addr learned my addr 172.21.15.98:0/117215552 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.860+0000 7efd69170640 1 -- 172.21.15.98:0/117215552 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd6410ccd0 msgr2=0x7efd64073ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.860+0000 7efd69971640 1 --2- 172.21.15.98:0/117215552 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd6410ccd0 0x7efd64073ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.860+0000 7efd69170640 1 --2- 172.21.15.98:0/117215552 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd6410ccd0 0x7efd64073ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.860+0000 7efd69170640 1 -- 172.21.15.98:0/117215552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd6410e300 msgr2=0x7efd641bb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.860+0000 7efd69170640 1 --2- 172.21.15.98:0/117215552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd6410e300 0x7efd641bb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.860+0000 7efd69170640 1 -- 172.21.15.98:0/117215552 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd641bbd40 con 0x7efd6410d960 2024-05-09T08:53:34.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.862+0000 7efd69170640 1 --2- 172.21.15.98:0/117215552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd6410d960 0x7efd64074410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00ba50 tx=0x7efd5c00bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:34.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.869+0000 7efd5affd640 1 -- 172.21.15.98:0/117215552 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c019070 con 0x7efd6410d960 2024-05-09T08:53:34.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.869+0000 7efd5affd640 1 -- 172.21.15.98:0/117215552 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd5c00cc10 con 0x7efd6410d960 2024-05-09T08:53:34.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.869+0000 7efd5affd640 1 -- 172.21.15.98:0/117215552 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c0076a0 con 0x7efd6410d960 2024-05-09T08:53:34.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.871+0000 7efd6a973640 1 -- 172.21.15.98:0/117215552 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd641bbfd0 con 0x7efd6410d960 2024-05-09T08:53:34.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.871+0000 7efd6a973640 1 -- 172.21.15.98:0/117215552 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd641bc4d0 con 0x7efd6410d960 2024-05-09T08:53:34.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.875+0000 7efd6e9be740 1 Processor -- start 2024-05-09T08:53:34.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.876+0000 7efd6e9be740 1 -- start start 2024-05-09T08:53:34.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.876+0000 7efd6e9be740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb025d0 0x55faabb029b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:34.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.876+0000 7efd6e9be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb02ef0 0x55faabb0d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:34.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.876+0000 7efd6e9be740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0d7b0 0x55faabb0fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:34.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.876+0000 7efd6e9be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55faaba67340 con 0x55faabb02ef0 2024-05-09T08:53:34.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.876+0000 7efd6e9be740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55faaba0f290 con 0x55faabb0d7b0 2024-05-09T08:53:34.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.876+0000 7efd6e9be740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55faaba63300 con 0x55faabb025d0 2024-05-09T08:53:34.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.878+0000 7efd69170640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0d7b0 0x55faabb0fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:34.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.878+0000 7efd6a172640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb025d0 0x55faabb029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:34.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.878+0000 7efd6a172640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb025d0 0x55faabb029b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:33422/0 (socket says 172.21.15.98:33422) 2024-05-09T08:53:34.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.878+0000 7efd6a172640 1 -- 172.21.15.98:0/893348011 learned_addr learned my addr 172.21.15.98:0/893348011 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:34.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.879+0000 7efd69971640 1 --2- 172.21.15.98:0/893348011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb02ef0 0x55faabb0d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:34.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.880+0000 7efd69971640 1 -- 172.21.15.98:0/893348011 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb025d0 msgr2=0x55faabb029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:34.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.880+0000 7efd69971640 1 --2- 172.21.15.98:0/893348011 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb025d0 0x55faabb029b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.880+0000 7efd69971640 1 -- 172.21.15.98:0/893348011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0d7b0 msgr2=0x55faabb0fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:34.881 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.880+0000 7efd69971640 1 --2- 172.21.15.98:0/893348011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0d7b0 0x55faabb0fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.881 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.880+0000 7efd69971640 1 -- 172.21.15.98:0/893348011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faaba338e0 con 0x55faabb02ef0 2024-05-09T08:53:34.881 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.881+0000 7efd69971640 1 --2- 172.21.15.98:0/893348011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb02ef0 0x55faabb0d270 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7efd54006010 tx=0x7efd540041a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe52f53df7a4b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.884+0000 7efd5affd640 1 -- 172.21.15.98:0/117215552 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7efd5c044d10 con 0x7efd6410d960 2024-05-09T08:53:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.885+0000 7efd4ffff640 1 -- 172.21.15.98:0/893348011 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5403f070 con 0x55faabb02ef0 2024-05-09T08:53:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.885+0000 7efd4ffff640 1 -- 172.21.15.98:0/893348011 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd54005520 con 0x55faabb02ef0 2024-05-09T08:53:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.885+0000 7efd4ffff640 1 -- 172.21.15.98:0/893348011 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd54035030 con 0x55faabb02ef0 2024-05-09T08:53:34.886 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.886+0000 7efd5affd640 1 --2- 172.21.15.98:0/117215552 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efd500a2f00 0x7efd500a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:34.886 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.886+0000 7efd5affd640 1 -- 172.21.15.98:0/117215552 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7efd5c0d4310 con 0x7efd6410d960 2024-05-09T08:53:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.889+0000 7efd6a172640 1 --2- 172.21.15.98:0/117215552 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efd500a2f00 0x7efd500a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.890+0000 7efd6e9be740 1 -- 172.21.15.98:0/893348011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb02ef0 msgr2=0x55faabb0d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.890+0000 7efd6e9be740 1 --2- 172.21.15.98:0/893348011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb02ef0 0x55faabb0d270 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7efd54006010 tx=0x7efd540041a0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.890+0000 7efd6e9be740 1 -- 172.21.15.98:0/893348011 shutdown_connections 2024-05-09T08:53:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.890+0000 7efd6e9be740 1 --2- 172.21.15.98:0/893348011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0d7b0 0x55faabb0fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.890+0000 7efd6e9be740 1 --2- 172.21.15.98:0/893348011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb02ef0 0x55faabb0d270 secure :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7efd54006010 tx=0x7efd540041a0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.890+0000 7efd6e9be740 1 --2- 172.21.15.98:0/893348011 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb025d0 0x55faabb029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.890+0000 7efd6e9be740 1 -- 172.21.15.98:0/893348011 >> 172.21.15.98:0/893348011 conn(0x55faabafeae0 msgr2=0x55faabafeec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.891+0000 7efd6a172640 1 --2- 172.21.15.98:0/117215552 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efd500a2f00 0x7efd500a53c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efd600098b0 tx=0x7efd60008040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.894+0000 7efd6e9be740 1 -- 172.21.15.98:0/893348011 shutdown_connections 2024-05-09T08:53:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.897+0000 7efd6e9be740 1 -- 172.21.15.98:0/893348011 wait complete. 2024-05-09T08:53:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.898+0000 7efd6e9be740 1 -- start start 2024-05-09T08:53:34.901 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.901+0000 7efd6e9be740 1 Processor -- start 2024-05-09T08:53:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.905+0000 7efd6e9be740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb025d0 0x55faabb0a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.905+0000 7efd6e9be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb0d7b0 0x55faabb0ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.905+0000 7efd6e9be740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0b320 0x55faabb197f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.905+0000 7efd6e9be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55faabb10380 con 0x55faabb0d7b0 2024-05-09T08:53:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.905+0000 7efd6e9be740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55faaba22790 con 0x55faabb025d0 2024-05-09T08:53:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.905+0000 7efd6e9be740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55faabb10500 con 0x55faabb0b320 2024-05-09T08:53:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.905+0000 7efd6e9be740 10 client.0 _ll_register_callbacks cb 0x55faaba5da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:34.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.909+0000 7efd69971640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb025d0 0x55faabb0a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:34.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.910+0000 7efd6a172640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0b320 0x55faabb197f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.912+0000 7efd69971640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb025d0 0x55faabb0a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:57292/0 (socket says 172.21.15.98:57292) 2024-05-09T08:53:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.912+0000 7efd69971640 1 -- 172.21.15.98:0/226500643 learned_addr learned my addr 172.21.15.98:0/226500643 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:34.916 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.916+0000 7efd69170640 1 --2- 172.21.15.98:0/226500643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb0d7b0 0x55faabb0ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:34.916 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.917+0000 7efd6a172640 1 -- 172.21.15.98:0/226500643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb025d0 msgr2=0x55faabb0a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:34.916 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.917+0000 7efd6a172640 1 --2- 172.21.15.98:0/226500643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb025d0 0x55faabb0a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.916 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.917+0000 7efd6a172640 1 -- 172.21.15.98:0/226500643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb0d7b0 msgr2=0x55faabb0ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:34.916 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.917+0000 7efd6a172640 1 --2- 172.21.15.98:0/226500643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb0d7b0 0x55faabb0ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.917+0000 7efd6a172640 1 -- 172.21.15.98:0/226500643 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faab77b210 con 0x55faabb0b320 2024-05-09T08:53:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.922+0000 7efd6a172640 1 --2- 172.21.15.98:0/226500643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0b320 0x55faabb197f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efd6000dd70 tx=0x7efd600047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e67cd75282547c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.923+0000 7efd4d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:53:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.923+0000 7efd4d7fa640 1 -- 172.21.15.98:0/226500643 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60024060 con 0x55faabb0b320 2024-05-09T08:53:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.924+0000 7efd6e9be740 1 -- 172.21.15.98:0/226500643 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd54005520 con 0x55faabb0b320 2024-05-09T08:53:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.924+0000 7efd4d7fa640 1 -- 172.21.15.98:0/226500643 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7efd6001f400 con 0x55faabb0b320 2024-05-09T08:53:34.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.930+0000 7efd4d7fa640 1 -- 172.21.15.98:0/226500643 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efd60014e40 con 0x55faabb0b320 2024-05-09T08:53:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.930+0000 7efd6e9be740 10 client.25810 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.930+0000 7efd6e9be740 1 -- 172.21.15.98:0/226500643 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55faab847620 con 0x55faabb0b320 2024-05-09T08:53:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.931+0000 7efd4d7fa640 1 -- 172.21.15.98:0/226500643 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd54005520 con 0x55faabb0b320 2024-05-09T08:53:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.931+0000 7efd4d7fa640 1 -- 172.21.15.98:0/226500643 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7efd60013070 con 0x55faabb0b320 2024-05-09T08:53:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.933+0000 7efd6e9be740 10 client.25810 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.933+0000 7efd6e9be740 10 client.25810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.933+0000 7efd6e9be740 20 client.25810 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.933+0000 7efd6e9be740 1 --2- 172.21.15.98:0/226500643 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55faabb1daf0 0x55faabb1ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.933+0000 7efd6e9be740 4 client.25810 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.933+0000 7efd6e9be740 1 -- 172.21.15.98:0/226500643 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55faaba74a70 con 0x55faabb1daf0 2024-05-09T08:53:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.933+0000 7efd69971640 1 --2- 172.21.15.98:0/226500643 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55faabb1daf0 0x55faabb1ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.944+0000 7efd69971640 1 --2- 172.21.15.98:0/226500643 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55faabb1daf0 0x55faabb1ff00 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11f768854c00db2c server_cookie=36385f500512380a in_seq=0 out_seq=0 2024-05-09T08:53:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.947+0000 7efd4d7fa640 10 client.25810 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.947+0000 7efd4d7fa640 1 -- 172.21.15.98:0/226500643 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55faaba74a70 con 0x55faabb1daf0 2024-05-09T08:53:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.947+0000 7efd4d7fa640 10 client.25810 handle_command_reply: tid=0 2024-05-09T08:53:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.947+0000 7efd6e9be740 1 client.25810 shutdown 2024-05-09T08:53:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.950+0000 7efd6e9be740 1 -- 172.21.15.98:0/226500643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0b320 msgr2=0x55faabb197f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.950+0000 7efd6e9be740 1 --2- 172.21.15.98:0/226500643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0b320 0x55faabb197f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efd6000dd70 tx=0x7efd600047c0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.953+0000 7efd6e9be740 1 -- 172.21.15.98:0/226500643 shutdown_connections 2024-05-09T08:53:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.953+0000 7efd6e9be740 1 --2- 172.21.15.98:0/226500643 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55faabb1daf0 0x55faabb1ff00 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.953+0000 7efd6e9be740 1 --2- 172.21.15.98:0/226500643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0b320 0x55faabb197f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.953+0000 7efd6e9be740 1 --2- 172.21.15.98:0/226500643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb0d7b0 0x55faabb0ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.953+0000 7efd6e9be740 1 --2- 172.21.15.98:0/226500643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb025d0 0x55faabb0a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.953+0000 7efd6e9be740 1 -- 172.21.15.98:0/226500643 >> 172.21.15.98:0/226500643 conn(0x55faabafeae0 msgr2=0x55faabb04860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.955+0000 7efd4d7fa640 0 client.25810 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.958+0000 7efd6e9be740 1 -- 172.21.15.98:0/226500643 shutdown_connections 2024-05-09T08:53:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.959+0000 7efd6e9be740 1 -- 172.21.15.98:0/226500643 wait complete. 2024-05-09T08:53:34.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:34.960+0000 7efd6e9be740 20 client.25810 trim_cache size 0 max 16384 2024-05-09T08:53:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.001+0000 7efd6e9be740 1 Processor -- start 2024-05-09T08:53:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.001+0000 7efd6e9be740 1 -- start start 2024-05-09T08:53:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.001+0000 7efd6e9be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb025d0 0x55faabafe420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.001+0000 7efd6e9be740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0b320 0x55faabb25810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.001+0000 7efd6e9be740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0d7b0 0x55faabb25d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.001+0000 7efd6e9be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efd48010040 con 0x55faabb025d0 2024-05-09T08:53:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.001+0000 7efd6e9be740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efd48011b80 con 0x55faabb0b320 2024-05-09T08:53:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.001+0000 7efd6e9be740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efd48010710 con 0x55faabb0d7b0 2024-05-09T08:53:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.002+0000 7efd69971640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb025d0 0x55faabafe420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.002+0000 7efd69971640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb025d0 0x55faabafe420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59214/0 (socket says 172.21.15.98:59214) 2024-05-09T08:53:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.002+0000 7efd69971640 1 -- 172.21.15.98:0/3287688592 learned_addr learned my addr 172.21.15.98:0/3287688592 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.002+0000 7efd69170640 1 --2- 172.21.15.98:0/3287688592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0b320 0x55faabb25810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.002+0000 7efd6a172640 1 --2- 172.21.15.98:0/3287688592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0d7b0 0x55faabb25d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.002+0000 7efd69170640 1 -- 172.21.15.98:0/3287688592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0d7b0 msgr2=0x55faabb25d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.002+0000 7efd69170640 1 --2- 172.21.15.98:0/3287688592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0d7b0 0x55faabb25d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.002+0000 7efd69170640 1 -- 172.21.15.98:0/3287688592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb025d0 msgr2=0x55faabafe420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.002+0000 7efd69170640 1 --2- 172.21.15.98:0/3287688592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb025d0 0x55faabafe420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.002+0000 7efd69170640 1 -- 172.21.15.98:0/3287688592 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faabb08df0 con 0x55faabb0b320 2024-05-09T08:53:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.002+0000 7efd69170640 1 --2- 172.21.15.98:0/3287688592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0b320 0x55faabb25810 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efd5c0d1750 tx=0x7efd5c09c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5003cd371495d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.003+0000 7efd4f7fe640 1 -- 172.21.15.98:0/3287688592 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c0a0c20 con 0x55faabb0b320 2024-05-09T08:53:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.003+0000 7efd4f7fe640 1 -- 172.21.15.98:0/3287688592 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd5c0a0520 con 0x55faabb0b320 2024-05-09T08:53:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.004+0000 7efd6e9be740 1 -- 172.21.15.98:0/3287688592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0b320 msgr2=0x55faabb25810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.004+0000 7efd6e9be740 1 --2- 172.21.15.98:0/3287688592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0b320 0x55faabb25810 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efd5c0d1750 tx=0x7efd5c09c040 comp rx=0 tx=0).stop 2024-05-09T08:53:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.009+0000 7efd6e9be740 1 -- 172.21.15.98:0/3287688592 shutdown_connections 2024-05-09T08:53:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.009+0000 7efd6e9be740 1 --2- 172.21.15.98:0/3287688592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0d7b0 0x55faabb25d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.009+0000 7efd6e9be740 1 --2- 172.21.15.98:0/3287688592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0b320 0x55faabb25810 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.009+0000 7efd6e9be740 1 --2- 172.21.15.98:0/3287688592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb025d0 0x55faabafe420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.009+0000 7efd6e9be740 1 -- 172.21.15.98:0/3287688592 >> 172.21.15.98:0/3287688592 conn(0x55faabafeae0 msgr2=0x55faabb0a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.011+0000 7efd6e9be740 1 -- 172.21.15.98:0/3287688592 shutdown_connections 2024-05-09T08:53:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.011+0000 7efd6e9be740 1 -- 172.21.15.98:0/3287688592 wait complete. 2024-05-09T08:53:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.011+0000 7efd6e9be740 1 -- start start 2024-05-09T08:53:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.011+0000 7efd6e9be740 1 Processor -- start 2024-05-09T08:53:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.012+0000 7efd6e9be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb025d0 0x55faabb075f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.012+0000 7efd6e9be740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0b320 0x55faabb07b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.012+0000 7efd6e9be740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0d7b0 0x55faabb08070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.012+0000 7efd6e9be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efd480114b0 con 0x55faabb025d0 2024-05-09T08:53:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.012+0000 7efd6e9be740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efd48010de0 con 0x55faabb0b320 2024-05-09T08:53:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.012+0000 7efd6e9be740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efd4800e020 con 0x55faabb0d7b0 2024-05-09T08:53:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.012+0000 7efd6e9be740 10 client.0 _ll_register_callbacks cb 0x7efd60006ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.012+0000 7efd69170640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0b320 0x55faabb07b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.012+0000 7efd69170640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0b320 0x55faabb07b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:57322/0 (socket says 172.21.15.98:57322) 2024-05-09T08:53:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.012+0000 7efd69170640 1 -- 172.21.15.98:0/1124071544 learned_addr learned my addr 172.21.15.98:0/1124071544 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.012+0000 7efd6a172640 1 --2- 172.21.15.98:0/1124071544 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0d7b0 0x55faabb08070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.013+0000 7efd69170640 1 -- 172.21.15.98:0/1124071544 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0d7b0 msgr2=0x55faabb08070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.013+0000 7efd69170640 1 --2- 172.21.15.98:0/1124071544 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0d7b0 0x55faabb08070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.013+0000 7efd69170640 1 -- 172.21.15.98:0/1124071544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb025d0 msgr2=0x55faabb075f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:53:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.014+0000 7efd69971640 1 --2- 172.21.15.98:0/1124071544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb025d0 0x55faabb075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.014+0000 7efd69170640 1 --2- 172.21.15.98:0/1124071544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb025d0 0x55faabb075f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.014+0000 7efd69170640 1 -- 172.21.15.98:0/1124071544 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faabb830b0 con 0x55faabb0b320 2024-05-09T08:53:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.015+0000 7efd69170640 1 --2- 172.21.15.98:0/1124071544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0b320 0x55faabb07b30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00c7d0 tx=0x7efd5c0d4160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5a441892f0767d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.015+0000 7efd4d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:53:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.015+0000 7efd4d7fa640 1 -- 172.21.15.98:0/1124071544 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c0bd070 con 0x55faabb0b320 2024-05-09T08:53:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.015+0000 7efd6e9be740 1 -- 172.21.15.98:0/1124071544 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd5c0a0520 con 0x55faabb0b320 2024-05-09T08:53:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.015+0000 7efd4d7fa640 1 -- 172.21.15.98:0/1124071544 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7efd5c0b9260 con 0x55faabb0b320 2024-05-09T08:53:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.016+0000 7efd4d7fa640 1 -- 172.21.15.98:0/1124071544 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efd5c089050 con 0x55faabb0b320 2024-05-09T08:53:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.016+0000 7efd6e9be740 10 client.25853 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.016+0000 7efd6e9be740 1 -- 172.21.15.98:0/1124071544 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55faabb843c0 con 0x55faabb0b320 2024-05-09T08:53:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.017+0000 7efd4d7fa640 1 -- 172.21.15.98:0/1124071544 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd5c0a0520 con 0x55faabb0b320 2024-05-09T08:53:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.018+0000 7efd4d7fa640 1 -- 172.21.15.98:0/1124071544 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7efd5c09f3e0 con 0x55faabb0b320 2024-05-09T08:53:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.018+0000 7efd6e9be740 10 client.25853 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.018+0000 7efd6e9be740 10 client.25853 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.018+0000 7efd6e9be740 20 client.25853 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.018+0000 7efd6e9be740 1 --2- 172.21.15.98:0/1124071544 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55faabb84a10 0x55faabb86e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.018+0000 7efd6e9be740 4 client.25853 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:53:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.018+0000 7efd6e9be740 1 -- 172.21.15.98:0/1124071544 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55faaba6edc0 con 0x55faabb84a10 2024-05-09T08:53:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.018+0000 7efd69971640 1 --2- 172.21.15.98:0/1124071544 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55faabb84a10 0x55faabb86e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:35.025 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.020+0000 7efd69971640 1 --2- 172.21.15.98:0/1124071544 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55faabb84a10 0x55faabb86e30 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e93480834709391 server_cookie=41b24cf3baec22df in_seq=0 out_seq=0 2024-05-09T08:53:35.025 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:35.020+0000 7efd4d7fa640 10 client.25853 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:35.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:35 smithi142 ceph-mon[34625]: pgmap v91: 129 pgs: 129 active+clean; 19 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 739 MiB/s wr, 1.80k op/s 2024-05-09T08:53:35.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:35 smithi105 ceph-mon[34498]: pgmap v91: 129 pgs: 129 active+clean; 19 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 739 MiB/s wr, 1.80k op/s 2024-05-09T08:53:36.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:35 smithi098 ceph-mon[30092]: pgmap v91: 129 pgs: 129 active+clean; 19 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 739 MiB/s wr, 1.80k op/s 2024-05-09T08:53:37.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:37 smithi142 ceph-mon[34625]: pgmap v92: 129 pgs: 129 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 889 MiB/s wr, 2.22k op/s 2024-05-09T08:53:37.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:37 smithi105 ceph-mon[34498]: pgmap v92: 129 pgs: 129 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 889 MiB/s wr, 2.22k op/s 2024-05-09T08:53:38.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:37 smithi098 ceph-mon[30092]: pgmap v92: 129 pgs: 129 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 889 MiB/s wr, 2.22k op/s 2024-05-09T08:53:38.805 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:53:38.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:53:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.123+0000 7f84ba51d640 1 -- 172.21.15.98:0/435780985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b4114980 msgr2=0x7f84b4116d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.123+0000 7f84ba51d640 1 --2- 172.21.15.98:0/435780985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b4114980 0x7f84b4116d70 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f84ac009ca0 tx=0x7f84ac02a8f0 comp rx=0 tx=0).stop 2024-05-09T08:53:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.123+0000 7f84ba51d640 1 -- 172.21.15.98:0/435780985 shutdown_connections 2024-05-09T08:53:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.123+0000 7f84ba51d640 1 --2- 172.21.15.98:0/435780985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b4114980 0x7f84b4116d70 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.123+0000 7f84ba51d640 1 --2- 172.21.15.98:0/435780985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84b4109360 0x7f84b4114440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.123+0000 7f84ba51d640 1 --2- 172.21.15.98:0/435780985 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84b4108a40 0x7f84b4108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.123+0000 7f84ba51d640 1 -- 172.21.15.98:0/435780985 >> 172.21.15.98:0/435780985 conn(0x7f84b406c940 msgr2=0x7f84b406cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.127+0000 7f84ba51d640 1 -- 172.21.15.98:0/435780985 shutdown_connections 2024-05-09T08:53:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.130+0000 7f84ba51d640 1 -- 172.21.15.98:0/435780985 wait complete. 2024-05-09T08:53:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.130+0000 7f84ba51d640 1 Processor -- start 2024-05-09T08:53:39.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.135+0000 7f84ba51d640 1 -- start start 2024-05-09T08:53:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.135+0000 7f84ba51d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84b4108a40 0x7f84b4113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.135+0000 7f84ba51d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b4109360 0x7f84b4114360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.135+0000 7f84ba51d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84b4114980 0x7f84b410af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.135+0000 7f84ba51d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f84b411a530 con 0x7f84b4109360 2024-05-09T08:53:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.135+0000 7f84ba51d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f84b411a3b0 con 0x7f84b4108a40 2024-05-09T08:53:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.135+0000 7f84ba51d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f84b411a6b0 con 0x7f84b4114980 2024-05-09T08:53:39.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.137+0000 7f84b8aba640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84b4114980 0x7f84b410af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:39.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.137+0000 7f84b8aba640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84b4114980 0x7f84b410af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:57056/0 (socket says 172.21.15.98:57056) 2024-05-09T08:53:39.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.137+0000 7f84b8aba640 1 -- 172.21.15.98:0/1764777633 learned_addr learned my addr 172.21.15.98:0/1764777633 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:39.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.138+0000 7f84b3fff640 1 --2- 172.21.15.98:0/1764777633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84b4108a40 0x7f84b4113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:39.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.141+0000 7f84b8aba640 1 -- 172.21.15.98:0/1764777633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84b4108a40 msgr2=0x7f84b4113e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:39.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.141+0000 7f84b8aba640 1 --2- 172.21.15.98:0/1764777633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84b4108a40 0x7f84b4113e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:39.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.141+0000 7f84b8aba640 1 -- 172.21.15.98:0/1764777633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b4109360 msgr2=0x7f84b4114360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:39.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.141+0000 7f84b8aba640 1 --2- 172.21.15.98:0/1764777633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b4109360 0x7f84b4114360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:39.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.141+0000 7f84b8aba640 1 -- 172.21.15.98:0/1764777633 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84b410b7c0 con 0x7f84b4114980 2024-05-09T08:53:39.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.141+0000 7f84b3fff640 1 --2- 172.21.15.98:0/1764777633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84b4108a40 0x7f84b4113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:53:39.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.142+0000 7f84b8aba640 1 --2- 172.21.15.98:0/1764777633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84b4114980 0x7f84b410af00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f84ac0316e0 tx=0x7f84ac031710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.146+0000 7f84b17fa640 1 -- 172.21.15.98:0/1764777633 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac004040 con 0x7f84b4114980 2024-05-09T08:53:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.146+0000 7f84b17fa640 1 -- 172.21.15.98:0/1764777633 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84ac02d070 con 0x7f84b4114980 2024-05-09T08:53:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.146+0000 7f84b17fa640 1 -- 172.21.15.98:0/1764777633 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac02fba0 con 0x7f84b4114980 2024-05-09T08:53:39.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.147+0000 7f84ba51d640 1 -- 172.21.15.98:0/1764777633 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84b410ba50 con 0x7f84b4114980 2024-05-09T08:53:39.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.149+0000 7f84ba51d640 1 -- 172.21.15.98:0/1764777633 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84b410fa40 con 0x7f84b4114980 2024-05-09T08:53:39.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.156+0000 7f84bd222740 1 Processor -- start 2024-05-09T08:53:39.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.156+0000 7f84bd222740 1 -- start start 2024-05-09T08:53:39.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.156+0000 7f84bd222740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6280 0x558bf28c6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.156+0000 7f84bd222740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28c6ba0 0x558bf28d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.156+0000 7f84bd222740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28d1270 0x558bf28d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.156+0000 7f84bd222740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558bf282bd00 con 0x558bf28d1270 2024-05-09T08:53:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.156+0000 7f84bd222740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558bf27d34c0 con 0x558bf28c6ba0 2024-05-09T08:53:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.156+0000 7f84bd222740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558bf2827cc0 con 0x558bf28c6280 2024-05-09T08:53:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.158+0000 7f84b3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28c6ba0 0x558bf28d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.159+0000 7f84b3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28c6ba0 0x558bf28d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55184/0 (socket says 172.21.15.98:55184) 2024-05-09T08:53:39.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.159+0000 7f84b3fff640 1 -- 172.21.15.98:0/967081019 learned_addr learned my addr 172.21.15.98:0/967081019 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:39.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.160+0000 7f84b37fe640 1 --2- 172.21.15.98:0/967081019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6280 0x558bf28c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:39.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.165+0000 7f84b37fe640 1 --2- 172.21.15.98:0/967081019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28d1270 0x558bf28d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:39.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.167+0000 7f84b17fa640 1 -- 172.21.15.98:0/1764777633 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f84ac02fd40 con 0x7f84b4114980 2024-05-09T08:53:39.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.173+0000 7f84b17fa640 1 --2- 172.21.15.98:0/1764777633 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f849c0a2f00 0x7f849c0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:39.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.173+0000 7f84b17fa640 1 -- 172.21.15.98:0/1764777633 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f84ac0f0d60 con 0x7f84b4114980 2024-05-09T08:53:39.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.174+0000 7f84b37fe640 1 -- 172.21.15.98:0/967081019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6280 msgr2=0x558bf28c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:39.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.174+0000 7f84b37fe640 1 --2- 172.21.15.98:0/967081019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6280 0x558bf28c6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:39.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.174+0000 7f84b37fe640 1 -- 172.21.15.98:0/967081019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28c6ba0 msgr2=0x558bf28d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:39.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.174+0000 7f84b37fe640 1 --2- 172.21.15.98:0/967081019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28c6ba0 0x558bf28d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:39.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.174+0000 7f84b37fe640 1 -- 172.21.15.98:0/967081019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bf26097d0 con 0x558bf28d1270 2024-05-09T08:53:39.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.174+0000 7f84b37fe640 1 --2- 172.21.15.98:0/967081019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28d1270 0x558bf28d3660 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f84b410c590 tx=0x7f84a4001140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48114a032de03e63 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:39.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.175+0000 7f84b8aba640 1 --2- 172.21.15.98:0/1764777633 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f849c0a2f00 0x7f849c0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:39.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.175+0000 7f849a7fc640 1 -- 172.21.15.98:0/967081019 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a4012040 con 0x558bf28d1270 2024-05-09T08:53:39.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.175+0000 7f849a7fc640 1 -- 172.21.15.98:0/967081019 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84a4008e00 con 0x558bf28d1270 2024-05-09T08:53:39.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.176+0000 7f84bd222740 1 -- 172.21.15.98:0/967081019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28d1270 msgr2=0x558bf28d3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:39.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.176+0000 7f84bd222740 1 --2- 172.21.15.98:0/967081019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28d1270 0x558bf28d3660 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f84b410c590 tx=0x7f84a4001140 comp rx=0 tx=0).stop 2024-05-09T08:53:39.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.180+0000 7f84bd222740 1 -- 172.21.15.98:0/967081019 shutdown_connections 2024-05-09T08:53:39.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.180+0000 7f84bd222740 1 --2- 172.21.15.98:0/967081019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28d1270 0x558bf28d3660 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:39.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.180+0000 7f84bd222740 1 --2- 172.21.15.98:0/967081019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28c6ba0 0x558bf28d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:39.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.180+0000 7f84bd222740 1 --2- 172.21.15.98:0/967081019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6280 0x558bf28c6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:39.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.180+0000 7f84bd222740 1 -- 172.21.15.98:0/967081019 >> 172.21.15.98:0/967081019 conn(0x558bf28c25a0 msgr2=0x558bf28c2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:39.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.183+0000 7f84bd222740 1 -- 172.21.15.98:0/967081019 shutdown_connections 2024-05-09T08:53:39.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.183+0000 7f84bd222740 1 -- 172.21.15.98:0/967081019 wait complete. 2024-05-09T08:53:39.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.183+0000 7f84bd222740 1 -- start start 2024-05-09T08:53:39.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.184+0000 7f84bd222740 1 Processor -- start 2024-05-09T08:53:39.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.185+0000 7f84bd222740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6280 0x558bf28ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:39.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.185+0000 7f84bd222740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28c6ba0 0x558bf28ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:39.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.185+0000 7f84bd222740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28d1270 0x558bf28ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:39.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.185+0000 7f84bd222740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558bf28d40b0 con 0x558bf28c6280 2024-05-09T08:53:39.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.185+0000 7f84bd222740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558bf27e6b90 con 0x558bf28c6ba0 2024-05-09T08:53:39.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.185+0000 7f84bd222740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558bf28d4230 con 0x558bf28d1270 2024-05-09T08:53:39.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.185+0000 7f84bd222740 10 client.0 _ll_register_callbacks cb 0x558bf2822da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:39.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.185+0000 7f84b37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6280 0x558bf28ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:39.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.186+0000 7f84b37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6280 0x558bf28ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44050/0 (socket says 172.21.15.98:44050) 2024-05-09T08:53:39.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.186+0000 7f84b37fe640 1 -- 172.21.15.98:0/1467547560 learned_addr learned my addr 172.21.15.98:0/1467547560 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:39.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.186+0000 7f84b8aba640 1 --2- 172.21.15.98:0/1764777633 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f849c0a2f00 0x7f849c0a53c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f84ac0ee200 tx=0x7f84ac03e040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:39.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.187+0000 7f84b37fe640 1 -- 172.21.15.98:0/1467547560 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28d1270 msgr2=0x558bf28ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:39.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.187+0000 7f84b37fe640 1 --2- 172.21.15.98:0/1467547560 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28d1270 0x558bf28ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:39.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.187+0000 7f84b37fe640 1 -- 172.21.15.98:0/1467547560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28c6ba0 msgr2=0x558bf28ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:39.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.187+0000 7f84b37fe640 1 --2- 172.21.15.98:0/1467547560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28c6ba0 0x558bf28ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:39.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.187+0000 7f84b37fe640 1 -- 172.21.15.98:0/1467547560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bf253f3f0 con 0x558bf28c6280 2024-05-09T08:53:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.192+0000 7f84b37fe640 1 --2- 172.21.15.98:0/1467547560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6280 0x558bf28ce300 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f84a40013b0 tx=0x7f84a40010b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ebc2dda6e178dd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:39.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.193+0000 7f84937fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.195+0000 7f84937fe640 1 -- 172.21.15.98:0/1467547560 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a40070f0 con 0x558bf28c6280 2024-05-09T08:53:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.195+0000 7f84bd222740 1 -- 172.21.15.98:0/1467547560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558bf2839430 con 0x558bf28c6280 2024-05-09T08:53:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.195+0000 7f84937fe640 1 -- 172.21.15.98:0/1467547560 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f84a4007290 con 0x558bf28c6280 2024-05-09T08:53:39.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.199+0000 7f84937fe640 1 -- 172.21.15.98:0/1467547560 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f84a4028050 con 0x558bf28c6280 2024-05-09T08:53:39.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.199+0000 7f84bd222740 10 client.16524 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:39.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.199+0000 7f84bd222740 1 -- 172.21.15.98:0/1467547560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558bf24ff0f0 con 0x558bf28c6280 2024-05-09T08:53:39.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.202+0000 7f84937fe640 1 -- 172.21.15.98:0/1467547560 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558bf2839430 con 0x558bf28c6280 2024-05-09T08:53:39.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.202+0000 7f84937fe640 1 -- 172.21.15.98:0/1467547560 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f84a4009490 con 0x558bf28c6280 2024-05-09T08:53:39.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.202+0000 7f84bd222740 10 client.16524 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:39.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.202+0000 7f84bd222740 10 client.16524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:39.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.202+0000 7f84bd222740 20 client.16524 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:39.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.202+0000 7f84bd222740 1 --2- 172.21.15.98:0/1467547560 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558bf28e1a30 0x558bf28e3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:39.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.202+0000 7f84bd222740 4 client.16524 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:39.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.202+0000 7f84bd222740 1 -- 172.21.15.98:0/1467547560 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558bf2833780 con 0x558bf28e1a30 2024-05-09T08:53:39.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.203+0000 7f84b3fff640 1 --2- 172.21.15.98:0/1467547560 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558bf28e1a30 0x558bf28e3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:39.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.207+0000 7f84937fe640 10 client.16524 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:39.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:39.207+0000 7f84b3fff640 1 --2- 172.21.15.98:0/1467547560 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558bf28e1a30 0x558bf28e3eb0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3241738585096bf0 server_cookie=ba6e545fd914502c in_seq=0 out_seq=0 2024-05-09T08:53:39.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:39 smithi105 ceph-mon[34498]: pgmap v93: 129 pgs: 129 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 710 MiB/s wr, 1.82k op/s 2024-05-09T08:53:40.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:39 smithi098 ceph-mon[30092]: pgmap v93: 129 pgs: 129 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 710 MiB/s wr, 1.82k op/s 2024-05-09T08:53:40.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:39 smithi142 ceph-mon[34625]: pgmap v93: 129 pgs: 129 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 710 MiB/s wr, 1.82k op/s 2024-05-09T08:53:41.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:41 smithi105 ceph-mon[34498]: pgmap v94: 129 pgs: 129 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 958 MiB/s wr, 2.50k op/s 2024-05-09T08:53:42.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:41 smithi098 ceph-mon[30092]: pgmap v94: 129 pgs: 129 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 958 MiB/s wr, 2.50k op/s 2024-05-09T08:53:42.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:41 smithi142 ceph-mon[34625]: pgmap v94: 129 pgs: 129 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 958 MiB/s wr, 2.50k op/s 2024-05-09T08:53:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.357+0000 7efd4d7fa640 1 -- 172.21.15.98:0/1124071544 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55faaba6edc0 con 0x55faabb84a10 2024-05-09T08:53:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.357+0000 7efd4d7fa640 10 client.25853 handle_command_reply: tid=0 2024-05-09T08:53:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.357+0000 7efd6e9be740 1 client.25853 shutdown 2024-05-09T08:53:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.357+0000 7efd6e9be740 1 -- 172.21.15.98:0/1124071544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0b320 msgr2=0x55faabb07b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.357+0000 7efd6e9be740 1 --2- 172.21.15.98:0/1124071544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0b320 0x55faabb07b30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00c7d0 tx=0x7efd5c0d4160 comp rx=0 tx=0).stop 2024-05-09T08:53:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.358+0000 7efd6e9be740 1 -- 172.21.15.98:0/1124071544 shutdown_connections 2024-05-09T08:53:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.358+0000 7efd6e9be740 1 --2- 172.21.15.98:0/1124071544 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55faabb84a10 0x55faabb86e30 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.358+0000 7efd6e9be740 1 --2- 172.21.15.98:0/1124071544 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55faabb0d7b0 0x55faabb08070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.358+0000 7efd6e9be740 1 --2- 172.21.15.98:0/1124071544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55faabb0b320 0x55faabb07b30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.358+0000 7efd6e9be740 1 --2- 172.21.15.98:0/1124071544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55faabb025d0 0x55faabb075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.358+0000 7efd6e9be740 1 -- 172.21.15.98:0/1124071544 >> 172.21.15.98:0/1124071544 conn(0x55faabafeae0 msgr2=0x55faabb03f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.358+0000 7efd4d7fa640 0 client.25853 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.358+0000 7efd6e9be740 1 -- 172.21.15.98:0/1124071544 shutdown_connections 2024-05-09T08:53:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.359+0000 7efd6e9be740 1 -- 172.21.15.98:0/1124071544 wait complete. 2024-05-09T08:53:42.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.358+0000 7f84937fe640 1 -- 172.21.15.98:0/1467547560 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558bf2833780 con 0x558bf28e1a30 2024-05-09T08:53:42.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.358+0000 7f84937fe640 10 client.16524 handle_command_reply: tid=0 2024-05-09T08:53:42.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.358+0000 7f84bd222740 1 client.16524 shutdown 2024-05-09T08:53:42.359 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.359+0000 7efd6e9be740 20 client.25853 trim_cache size 0 max 16384 2024-05-09T08:53:42.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.360+0000 7f84bd222740 1 -- 172.21.15.98:0/1467547560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6280 msgr2=0x558bf28ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:42.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.360+0000 7f84bd222740 1 --2- 172.21.15.98:0/1467547560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6280 0x558bf28ce300 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f84a40013b0 tx=0x7f84a40010b0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.360+0000 7f84bd222740 1 -- 172.21.15.98:0/1467547560 shutdown_connections 2024-05-09T08:53:42.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.360+0000 7f84bd222740 1 --2- 172.21.15.98:0/1467547560 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558bf28e1a30 0x558bf28e3eb0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.360+0000 7f84bd222740 1 --2- 172.21.15.98:0/1467547560 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28d1270 0x558bf28ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.360+0000 7f84bd222740 1 --2- 172.21.15.98:0/1467547560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28c6ba0 0x558bf28ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.360+0000 7f84bd222740 1 --2- 172.21.15.98:0/1467547560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6280 0x558bf28ce300 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.360+0000 7f84bd222740 1 -- 172.21.15.98:0/1467547560 >> 172.21.15.98:0/1467547560 conn(0x558bf28c25a0 msgr2=0x558bf28c83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:42.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.360+0000 7f84937fe640 0 client.16524 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:53:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:53:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:53:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:53:42.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.360+0000 7f84bd222740 1 -- 172.21.15.98:0/1467547560 shutdown_connections 2024-05-09T08:53:42.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.360+0000 7f84bd222740 1 -- 172.21.15.98:0/1467547560 wait complete. 2024-05-09T08:53:42.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.360+0000 7f84bd222740 20 client.16524 trim_cache size 0 max 16384 2024-05-09T08:53:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.382+0000 7efd6a973640 1 -- 172.21.15.98:0/117215552 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efd500a2f00 msgr2=0x7efd500a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.382+0000 7efd6a973640 1 --2- 172.21.15.98:0/117215552 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efd500a2f00 0x7efd500a53c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efd600098b0 tx=0x7efd60008040 comp rx=0 tx=0).stop 2024-05-09T08:53:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.382+0000 7efd6a973640 1 -- 172.21.15.98:0/117215552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd6410d960 msgr2=0x7efd64074410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.382+0000 7efd6a973640 1 --2- 172.21.15.98:0/117215552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd6410d960 0x7efd64074410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00ba50 tx=0x7efd5c00bf10 comp rx=0 tx=0).stop 2024-05-09T08:53:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.384+0000 7efd6a973640 1 -- 172.21.15.98:0/117215552 shutdown_connections 2024-05-09T08:53:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.384+0000 7efd6a973640 1 --2- 172.21.15.98:0/117215552 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efd500a2f00 0x7efd500a53c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.384+0000 7efd6a973640 1 --2- 172.21.15.98:0/117215552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd6410e300 0x7efd641bb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.384+0000 7efd6a973640 1 --2- 172.21.15.98:0/117215552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd6410d960 0x7efd64074410 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.384+0000 7efd6a973640 1 --2- 172.21.15.98:0/117215552 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd6410ccd0 0x7efd64073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.384+0000 7efd6a973640 1 -- 172.21.15.98:0/117215552 >> 172.21.15.98:0/117215552 conn(0x7efd6406c770 msgr2=0x7efd64071e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.388+0000 7efd6a973640 1 -- 172.21.15.98:0/117215552 shutdown_connections 2024-05-09T08:53:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:42.388+0000 7efd6a973640 1 -- 172.21.15.98:0/117215552 wait complete. 2024-05-09T08:53:42.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.410+0000 7f84bd222740 1 Processor -- start 2024-05-09T08:53:42.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.410+0000 7f84bd222740 1 -- start start 2024-05-09T08:53:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.410+0000 7f84bd222740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6280 0x558bf28dcfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.411+0000 7f84bd222740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6ba0 0x558bf28c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.411+0000 7f84bd222740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28d1270 0x558bf28ceba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.411+0000 7f84bd222740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f84b41c64b0 con 0x558bf28c6280 2024-05-09T08:53:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.411+0000 7f84bd222740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f84b41c7ff0 con 0x558bf28d1270 2024-05-09T08:53:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.411+0000 7f84bd222740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f84b41c6b80 con 0x558bf28c6ba0 2024-05-09T08:53:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.415+0000 7f84b3fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6ba0 0x558bf28c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.415+0000 7f84b37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28d1270 0x558bf28ceba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.415+0000 7f84b37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6280 0x558bf28dcfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.415+0000 7f84b37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6280 0x558bf28dcfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44054/0 (socket says 172.21.15.98:44054) 2024-05-09T08:53:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.415+0000 7f84b37fe640 1 -- 172.21.15.98:0/2956447022 learned_addr learned my addr 172.21.15.98:0/2956447022 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.416+0000 7f84b37fe640 1 -- 172.21.15.98:0/2956447022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6ba0 msgr2=0x558bf28c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.416+0000 7f84b37fe640 1 --2- 172.21.15.98:0/2956447022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6ba0 0x558bf28c95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.416+0000 7f84b37fe640 1 -- 172.21.15.98:0/2956447022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6280 msgr2=0x558bf28dcfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.416+0000 7f84b37fe640 1 --2- 172.21.15.98:0/2956447022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6280 0x558bf28dcfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.416+0000 7f84b37fe640 1 -- 172.21.15.98:0/2956447022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bf28c69d0 con 0x558bf28d1270 2024-05-09T08:53:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.416+0000 7f84b37fe640 1 --2- 172.21.15.98:0/2956447022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28d1270 0x558bf28ceba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f84a40016c0 tx=0x7f84a401b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ee031f0465d3467 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:42.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.432+0000 7f8499ffb640 1 -- 172.21.15.98:0/2956447022 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a4004020 con 0x558bf28d1270 2024-05-09T08:53:42.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.432+0000 7f8499ffb640 1 -- 172.21.15.98:0/2956447022 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84a4009c00 con 0x558bf28d1270 2024-05-09T08:53:42.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.432+0000 7f8499ffb640 1 -- 172.21.15.98:0/2956447022 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a40198c0 con 0x558bf28d1270 2024-05-09T08:53:42.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.434+0000 7f84bd222740 1 -- 172.21.15.98:0/2956447022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28d1270 msgr2=0x558bf28ceba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:42.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.434+0000 7f84bd222740 1 --2- 172.21.15.98:0/2956447022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28d1270 0x558bf28ceba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f84a40016c0 tx=0x7f84a401b040 comp rx=0 tx=0).stop 2024-05-09T08:53:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.440+0000 7f84bd222740 1 -- 172.21.15.98:0/2956447022 shutdown_connections 2024-05-09T08:53:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.440+0000 7f84bd222740 1 --2- 172.21.15.98:0/2956447022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28d1270 0x558bf28ceba0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.440+0000 7f84bd222740 1 --2- 172.21.15.98:0/2956447022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6ba0 0x558bf28c95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.440+0000 7f84bd222740 1 --2- 172.21.15.98:0/2956447022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6280 0x558bf28dcfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.441+0000 7f84bd222740 1 -- 172.21.15.98:0/2956447022 >> 172.21.15.98:0/2956447022 conn(0x558bf28c25a0 msgr2=0x558bf28cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:42.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.446+0000 7f84bd222740 1 -- 172.21.15.98:0/2956447022 shutdown_connections 2024-05-09T08:53:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.446+0000 7f84bd222740 1 -- 172.21.15.98:0/2956447022 wait complete. 2024-05-09T08:53:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.447+0000 7f84bd222740 1 -- start start 2024-05-09T08:53:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.447+0000 7f84bd222740 1 Processor -- start 2024-05-09T08:53:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.448+0000 7f84bd222740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6280 0x558bf28cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.448+0000 7f84bd222740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6ba0 0x558bf28d9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.448+0000 7f84bd222740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28d1270 0x558bf28d9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.448+0000 7f84bd222740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f84b41c7920 con 0x558bf28c6ba0 2024-05-09T08:53:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.448+0000 7f84bd222740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f84b41c7250 con 0x558bf28d1270 2024-05-09T08:53:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.448+0000 7f84bd222740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f84b41c4490 con 0x558bf28c6280 2024-05-09T08:53:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.448+0000 7f84bd222740 10 client.0 _ll_register_callbacks cb 0x7f84a800a1d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.448+0000 7f84b37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28d1270 0x558bf28d9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.448+0000 7f84b37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28d1270 0x558bf28d9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55214/0 (socket says 172.21.15.98:55214) 2024-05-09T08:53:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.448+0000 7f84b37fe640 1 -- 172.21.15.98:0/690672048 learned_addr learned my addr 172.21.15.98:0/690672048 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.448+0000 7f84b37fe640 1 --2- 172.21.15.98:0/690672048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6280 0x558bf28cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.449+0000 7f84b37fe640 1 -- 172.21.15.98:0/690672048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6280 msgr2=0x558bf28cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.449+0000 7f84b37fe640 1 --2- 172.21.15.98:0/690672048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6280 0x558bf28cbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.449+0000 7f84b37fe640 1 -- 172.21.15.98:0/690672048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6ba0 msgr2=0x558bf28d9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.449+0000 7f84b37fe640 1 --2- 172.21.15.98:0/690672048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6ba0 0x558bf28d9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.449+0000 7f84b37fe640 1 -- 172.21.15.98:0/690672048 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bf28ca6c0 con 0x558bf28d1270 2024-05-09T08:53:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.449+0000 7f84b37fe640 1 --2- 172.21.15.98:0/690672048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28d1270 0x558bf28d9960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x558bf28cbb40 tx=0x7f84a4006040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56ec992db2c8310f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.450+0000 7f84937fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:53:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.450+0000 7f84937fe640 1 -- 172.21.15.98:0/690672048 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a4004220 con 0x558bf28d1270 2024-05-09T08:53:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.450+0000 7f84bd222740 1 -- 172.21.15.98:0/690672048 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84a4009c00 con 0x558bf28d1270 2024-05-09T08:53:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.450+0000 7f84937fe640 1 -- 172.21.15.98:0/690672048 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f84a40043c0 con 0x558bf28d1270 2024-05-09T08:53:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.451+0000 7f84937fe640 1 -- 172.21.15.98:0/690672048 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f84a4019870 con 0x558bf28d1270 2024-05-09T08:53:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.456+0000 7f84bd222740 10 client.25862 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.456+0000 7f84bd222740 1 -- 172.21.15.98:0/690672048 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558bf2947450 con 0x558bf28d1270 2024-05-09T08:53:42.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.457+0000 7f84937fe640 1 -- 172.21.15.98:0/690672048 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84a4009c00 con 0x558bf28d1270 2024-05-09T08:53:42.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.457+0000 7f84937fe640 1 -- 172.21.15.98:0/690672048 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f84a4003c00 con 0x558bf28d1270 2024-05-09T08:53:42.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.457+0000 7f84bd222740 10 client.25862 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:42.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.457+0000 7f84bd222740 10 client.25862 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:42.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.457+0000 7f84bd222740 20 client.25862 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.457+0000 7f84bd222740 1 --2- 172.21.15.98:0/690672048 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558bf2947aa0 0x558bf2949ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.457+0000 7f84bd222740 4 client.25862 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:53:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.457+0000 7f84bd222740 1 -- 172.21.15.98:0/690672048 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558bf27b06f0 con 0x558bf2947aa0 2024-05-09T08:53:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.458+0000 7f84b3fff640 1 --2- 172.21.15.98:0/690672048 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558bf2947aa0 0x558bf2949ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:42.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.467+0000 7f84b3fff640 1 --2- 172.21.15.98:0/690672048 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558bf2947aa0 0x558bf2949ec0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35aacc4f69ec77d4 server_cookie=258c507da7c02697 in_seq=0 out_seq=0 2024-05-09T08:53:42.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.467+0000 7f84937fe640 10 client.25862 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:42.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.471+0000 7f84937fe640 1 -- 172.21.15.98:0/690672048 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558bf27b06f0 con 0x558bf2947aa0 2024-05-09T08:53:42.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.471+0000 7f84937fe640 10 client.25862 handle_command_reply: tid=0 2024-05-09T08:53:42.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.471+0000 7f84bd222740 1 client.25862 shutdown 2024-05-09T08:53:42.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.474+0000 7f84bd222740 1 -- 172.21.15.98:0/690672048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28d1270 msgr2=0x558bf28d9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:42.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.474+0000 7f84bd222740 1 --2- 172.21.15.98:0/690672048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28d1270 0x558bf28d9960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x558bf28cbb40 tx=0x7f84a4006040 comp rx=0 tx=0).stop 2024-05-09T08:53:42.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.475+0000 7f84bd222740 1 -- 172.21.15.98:0/690672048 shutdown_connections 2024-05-09T08:53:42.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.475+0000 7f84bd222740 1 --2- 172.21.15.98:0/690672048 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558bf2947aa0 0x558bf2949ec0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.475+0000 7f84bd222740 1 --2- 172.21.15.98:0/690672048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558bf28d1270 0x558bf28d9960 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.475+0000 7f84bd222740 1 --2- 172.21.15.98:0/690672048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558bf28c6ba0 0x558bf28d9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.475+0000 7f84bd222740 1 --2- 172.21.15.98:0/690672048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558bf28c6280 0x558bf28cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.475+0000 7f84bd222740 1 -- 172.21.15.98:0/690672048 >> 172.21.15.98:0/690672048 conn(0x558bf28c25a0 msgr2=0x558bf28d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:42.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.475+0000 7f84937fe640 0 client.25862 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:42.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.475+0000 7f84bd222740 1 -- 172.21.15.98:0/690672048 shutdown_connections 2024-05-09T08:53:42.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.475+0000 7f84bd222740 1 -- 172.21.15.98:0/690672048 wait complete. 2024-05-09T08:53:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.479+0000 7f84bd222740 20 client.25862 trim_cache size 0 max 16384 2024-05-09T08:53:42.482 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:53:42.482 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:53:42.482 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:53:42.482 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:53:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.488+0000 7f84ba51d640 1 -- 172.21.15.98:0/1764777633 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f849c0a2f00 msgr2=0x7f849c0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.488+0000 7f84ba51d640 1 --2- 172.21.15.98:0/1764777633 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f849c0a2f00 0x7f849c0a53c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f84ac0ee200 tx=0x7f84ac03e040 comp rx=0 tx=0).stop 2024-05-09T08:53:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.488+0000 7f84ba51d640 1 -- 172.21.15.98:0/1764777633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84b4114980 msgr2=0x7f84b410af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.488+0000 7f84ba51d640 1 --2- 172.21.15.98:0/1764777633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84b4114980 0x7f84b410af00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f84ac0316e0 tx=0x7f84ac031710 comp rx=0 tx=0).stop 2024-05-09T08:53:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.488+0000 7f84ba51d640 1 -- 172.21.15.98:0/1764777633 shutdown_connections 2024-05-09T08:53:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.488+0000 7f84ba51d640 1 --2- 172.21.15.98:0/1764777633 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f849c0a2f00 0x7f849c0a53c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.488+0000 7f84ba51d640 1 --2- 172.21.15.98:0/1764777633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84b4114980 0x7f84b410af00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.488+0000 7f84ba51d640 1 --2- 172.21.15.98:0/1764777633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84b4109360 0x7f84b4114360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.488+0000 7f84ba51d640 1 --2- 172.21.15.98:0/1764777633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84b4108a40 0x7f84b4113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.488+0000 7f84ba51d640 1 -- 172.21.15.98:0/1764777633 >> 172.21.15.98:0/1764777633 conn(0x7f84b406c940 msgr2=0x7f84b4070500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.489+0000 7f84ba51d640 1 -- 172.21.15.98:0/1764777633 shutdown_connections 2024-05-09T08:53:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.490+0000 7f84ba51d640 1 -- 172.21.15.98:0/1764777633 wait complete. 2024-05-09T08:53:42.533 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:53:42.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:53:42.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.815+0000 7f91d0b48640 1 -- 172.21.15.98:0/931643643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91cc1010d0 msgr2=0x7f91cc10d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:42.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.815+0000 7f91d0b48640 1 --2- 172.21.15.98:0/931643643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91cc1010d0 0x7f91cc10d5e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f91c4009cd0 tx=0x7f91c402a930 comp rx=0 tx=0).stop 2024-05-09T08:53:42.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.818+0000 7f91d0b48640 1 -- 172.21.15.98:0/931643643 shutdown_connections 2024-05-09T08:53:42.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.818+0000 7f91d0b48640 1 --2- 172.21.15.98:0/931643643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91cc1010d0 0x7f91cc10d5e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.818+0000 7f91d0b48640 1 --2- 172.21.15.98:0/931643643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91cc100710 0x7f91cc100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.818+0000 7f91d0b48640 1 --2- 172.21.15.98:0/931643643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91cc106720 0x7f91cc106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.818+0000 7f91d0b48640 1 -- 172.21.15.98:0/931643643 >> 172.21.15.98:0/931643643 conn(0x7f91cc077830 msgr2=0x7f91cc077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:42.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.821+0000 7f91d0b48640 1 -- 172.21.15.98:0/931643643 shutdown_connections 2024-05-09T08:53:42.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.823+0000 7f91d0b48640 1 -- 172.21.15.98:0/931643643 wait complete. 2024-05-09T08:53:42.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.824+0000 7f91d0b48640 1 Processor -- start 2024-05-09T08:53:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.826+0000 7f91d0b48640 1 -- start start 2024-05-09T08:53:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.826+0000 7f91d0b48640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91cc100710 0x7f91cc109850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.826+0000 7f91d0b48640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91cc1010d0 0x7f91cc109d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.826+0000 7f91d0b48640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91cc106720 0x7f91cc10a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.826+0000 7f91d0b48640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f91cc10fb70 con 0x7f91cc100710 2024-05-09T08:53:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.827+0000 7f91d0b48640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f91cc10f9f0 con 0x7f91cc106720 2024-05-09T08:53:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.827+0000 7f91d0b48640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f91cc10fcf0 con 0x7f91cc1010d0 2024-05-09T08:53:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.827+0000 7f91cad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91cc106720 0x7f91cc10a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.827+0000 7f91cad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91cc106720 0x7f91cc10a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55232/0 (socket says 172.21.15.98:55232) 2024-05-09T08:53:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.827+0000 7f91cad9d640 1 -- 172.21.15.98:0/1135501033 learned_addr learned my addr 172.21.15.98:0/1135501033 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.828+0000 7f91cad9d640 1 -- 172.21.15.98:0/1135501033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91cc1010d0 msgr2=0x7f91cc109d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.828+0000 7f91cad9d640 1 --2- 172.21.15.98:0/1135501033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91cc1010d0 0x7f91cc109d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.828+0000 7f91cad9d640 1 -- 172.21.15.98:0/1135501033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91cc100710 msgr2=0x7f91cc109850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:53:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.828+0000 7f91cad9d640 1 --2- 172.21.15.98:0/1135501033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91cc100710 0x7f91cc109850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.828+0000 7f91cad9d640 1 -- 172.21.15.98:0/1135501033 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91cc073410 con 0x7f91cc106720 2024-05-09T08:53:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.829+0000 7f91cad9d640 1 --2- 172.21.15.98:0/1135501033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91cc106720 0x7f91cc10a2d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f91c402fbd0 tx=0x7f91c402fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.829+0000 7f91bb7fe640 1 -- 172.21.15.98:0/1135501033 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c4003970 con 0x7f91cc106720 2024-05-09T08:53:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.829+0000 7f91d0b48640 1 -- 172.21.15.98:0/1135501033 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91cc0736a0 con 0x7f91cc106720 2024-05-09T08:53:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.830+0000 7f91d0b48640 1 -- 172.21.15.98:0/1135501033 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91cc115520 con 0x7f91cc106720 2024-05-09T08:53:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.831+0000 7f91bb7fe640 1 -- 172.21.15.98:0/1135501033 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91c402d070 con 0x7f91cc106720 2024-05-09T08:53:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.831+0000 7f91bb7fe640 1 -- 172.21.15.98:0/1135501033 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c4030850 con 0x7f91cc106720 2024-05-09T08:53:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.832+0000 7f91d0b48640 1 -- 172.21.15.98:0/1135501033 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91cc101e50 con 0x7f91cc106720 2024-05-09T08:53:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.836+0000 7f91bb7fe640 1 -- 172.21.15.98:0/1135501033 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f91c40309f0 con 0x7f91cc106720 2024-05-09T08:53:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.837+0000 7f91bb7fe640 1 --2- 172.21.15.98:0/1135501033 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f91b40a2e40 0x7f91b40a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.837+0000 7f91ca59c640 1 --2- 172.21.15.98:0/1135501033 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f91b40a2e40 0x7f91b40a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.838+0000 7f91bb7fe640 1 -- 172.21.15.98:0/1135501033 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f91c40f0db0 con 0x7f91cc106720 2024-05-09T08:53:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.842+0000 7f91ca59c640 1 --2- 172.21.15.98:0/1135501033 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f91b40a2e40 0x7f91b40a5300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f91bc009670 tx=0x7f91bc006bd0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:42.842+0000 7f91bb7fe640 1 -- 172.21.15.98:0/1135501033 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f91c40c5c30 con 0x7f91cc106720 2024-05-09T08:53:43.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.477+0000 7f91d0b48640 1 -- 172.21.15.98:0/1135501033 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91cc068650 con 0x7f91cc106720 2024-05-09T08:53:43.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.479+0000 7f91bb7fe640 1 -- 172.21.15.98:0/1135501033 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11944 (secure 0 0 0) 0x7f91c40bb390 con 0x7f91cc106720 2024-05-09T08:53:43.481 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:53:43.481 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":25736,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3228987570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3228987570},{"type":"v1","addr":"172.21.15.105:6835","nonce":3228987570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:52:47.790415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14568":{"gid":14568,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6837/3134006534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3134006534},{"type":"v1","addr":"172.21.15.98:6837","nonce":3134006534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24439,14667,24388,24461,24457]},"id":1}]} 2024-05-09T08:53:43.481 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 31 2024-05-09T08:53:43.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.492+0000 7f91d0b48640 1 -- 172.21.15.98:0/1135501033 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f91b40a2e40 msgr2=0x7f91b40a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:43.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.492+0000 7f91d0b48640 1 --2- 172.21.15.98:0/1135501033 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f91b40a2e40 0x7f91b40a5300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f91bc009670 tx=0x7f91bc006bd0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.492+0000 7f91d0b48640 1 -- 172.21.15.98:0/1135501033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91cc106720 msgr2=0x7f91cc10a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:43.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.492+0000 7f91d0b48640 1 --2- 172.21.15.98:0/1135501033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91cc106720 0x7f91cc10a2d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f91c402fbd0 tx=0x7f91c402fc00 comp rx=0 tx=0).stop 2024-05-09T08:53:43.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.495+0000 7f91d0b48640 1 -- 172.21.15.98:0/1135501033 shutdown_connections 2024-05-09T08:53:43.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.495+0000 7f91d0b48640 1 --2- 172.21.15.98:0/1135501033 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f91b40a2e40 0x7f91b40a5300 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.495+0000 7f91d0b48640 1 --2- 172.21.15.98:0/1135501033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91cc106720 0x7f91cc10a2d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.495+0000 7f91d0b48640 1 --2- 172.21.15.98:0/1135501033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91cc1010d0 0x7f91cc109d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.495+0000 7f91d0b48640 1 --2- 172.21.15.98:0/1135501033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91cc100710 0x7f91cc109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.495+0000 7f91d0b48640 1 -- 172.21.15.98:0/1135501033 >> 172.21.15.98:0/1135501033 conn(0x7f91cc077830 msgr2=0x7f91cc102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:43.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.500+0000 7f91d0b48640 1 -- 172.21.15.98:0/1135501033 shutdown_connections 2024-05-09T08:53:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.503+0000 7f91d0b48640 1 -- 172.21.15.98:0/1135501033 wait complete. 2024-05-09T08:53:43.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:53:43.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.743+0000 7f9fb2a29640 1 -- 172.21.15.98:0/2534975550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fac10e0b0 msgr2=0x7f9fac10e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:43.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.744+0000 7f9fb2a29640 1 --2- 172.21.15.98:0/2534975550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fac10e0b0 0x7f9fac10e530 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9fa001a940 tx=0x7f9fa003b490 comp rx=0 tx=0).stop 2024-05-09T08:53:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.749+0000 7f9fb2a29640 1 -- 172.21.15.98:0/2534975550 shutdown_connections 2024-05-09T08:53:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.749+0000 7f9fb2a29640 1 --2- 172.21.15.98:0/2534975550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fac11b010 0x7f9fac11d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.749+0000 7f9fb2a29640 1 --2- 172.21.15.98:0/2534975550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fac10e0b0 0x7f9fac10e530 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.749+0000 7f9fb2a29640 1 --2- 172.21.15.98:0/2534975550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fac110080 0x7f9fac10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.749+0000 7f9fb2a29640 1 -- 172.21.15.98:0/2534975550 >> 172.21.15.98:0/2534975550 conn(0x7f9fac06c940 msgr2=0x7f9fac06cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:43.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.753+0000 7f9fb2a29640 1 -- 172.21.15.98:0/2534975550 shutdown_connections 2024-05-09T08:53:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.753+0000 7f9fb2a29640 1 -- 172.21.15.98:0/2534975550 wait complete. 2024-05-09T08:53:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.754+0000 7f9fb2a29640 1 Processor -- start 2024-05-09T08:53:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.754+0000 7f9fb2a29640 1 -- start start 2024-05-09T08:53:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.755+0000 7f9fb2a29640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fac10e0b0 0x7f9fac11a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.755+0000 7f9fb2a29640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fac110080 0x7f9fac1169e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.755+0000 7f9fb2a29640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fac11b010 0x7f9fac116f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.755+0000 7f9fb2a29640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9fac1132b0 con 0x7f9fac110080 2024-05-09T08:53:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.755+0000 7f9fb2a29640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9fac113130 con 0x7f9fac10e0b0 2024-05-09T08:53:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.755+0000 7f9fb2a29640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9fac113430 con 0x7f9fac11b010 2024-05-09T08:53:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.755+0000 7f9fab7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fac110080 0x7f9fac1169e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.755+0000 7f9fab7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fac110080 0x7f9fac1169e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44078/0 (socket says 172.21.15.98:44078) 2024-05-09T08:53:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.755+0000 7f9fab7fe640 1 -- 172.21.15.98:0/2324849774 learned_addr learned my addr 172.21.15.98:0/2324849774 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.756+0000 7f9fabfff640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fac10e0b0 0x7f9fac11a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.756+0000 7f9fb0fc6640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fac11b010 0x7f9fac116f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.756+0000 7f9fab7fe640 1 -- 172.21.15.98:0/2324849774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fac11b010 msgr2=0x7f9fac116f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.756+0000 7f9fab7fe640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fac11b010 0x7f9fac116f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.756+0000 7f9fab7fe640 1 -- 172.21.15.98:0/2324849774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fac10e0b0 msgr2=0x7f9fac11a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.756+0000 7f9fab7fe640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fac10e0b0 0x7f9fac11a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.756+0000 7f9fab7fe640 1 -- 172.21.15.98:0/2324849774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fac1177b0 con 0x7f9fac110080 2024-05-09T08:53:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.756+0000 7f9fb0fc6640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fac11b010 0x7f9fac116f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:53:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.756+0000 7f9fabfff640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fac10e0b0 0x7f9fac11a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:53:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.756+0000 7f9fab7fe640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fac110080 0x7f9fac1169e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0005ab0 tx=0x7f9fa0004c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.758+0000 7f9fa97fa640 1 -- 172.21.15.98:0/2324849774 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa004f070 con 0x7f9fac110080 2024-05-09T08:53:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.759+0000 7f9fb2a29640 1 -- 172.21.15.98:0/2324849774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fac079af0 con 0x7f9fac110080 2024-05-09T08:53:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.759+0000 7f9fb2a29640 1 -- 172.21.15.98:0/2324849774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fac07a040 con 0x7f9fac110080 2024-05-09T08:53:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.759+0000 7f9fa97fa640 1 -- 172.21.15.98:0/2324849774 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fa0040c00 con 0x7f9fac110080 2024-05-09T08:53:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.759+0000 7f9fa97fa640 1 -- 172.21.15.98:0/2324849774 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0046030 con 0x7f9fac110080 2024-05-09T08:53:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.767+0000 7f9fb580c740 1 Processor -- start 2024-05-09T08:53:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.767+0000 7f9fa97fa640 1 -- 172.21.15.98:0/2324849774 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9fa0052070 con 0x7f9fac110080 2024-05-09T08:53:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.768+0000 7f9fb580c740 1 -- start start 2024-05-09T08:53:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.768+0000 7f9fb580c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9e330 0x55579da9e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.768+0000 7f9fb580c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9ec50 0x55579daa8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.768+0000 7f9fb580c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 0x55579daab710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.768+0000 7f9fb580c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55579da03c40 con 0x55579daa9320 2024-05-09T08:53:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.768+0000 7f9fb580c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55579d9ab580 con 0x55579da9ec50 2024-05-09T08:53:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.769+0000 7f9fb580c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55579d9ffc00 con 0x55579da9e330 2024-05-09T08:53:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.769+0000 7f9fabfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9ec50 0x55579daa8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.769+0000 7f9fabfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9ec50 0x55579daa8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55262/0 (socket says 172.21.15.98:55262) 2024-05-09T08:53:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.769+0000 7f9fabfff640 1 -- 172.21.15.98:0/3219249869 learned_addr learned my addr 172.21.15.98:0/3219249869 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.769+0000 7f9fabfff640 1 -- 172.21.15.98:0/3219249869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9e330 msgr2=0x55579da9e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.769+0000 7f9fabfff640 1 --2- 172.21.15.98:0/3219249869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9e330 0x55579da9e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.769+0000 7f9fabfff640 1 -- 172.21.15.98:0/3219249869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 msgr2=0x55579daab710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.769+0000 7f9fabfff640 1 --2- 172.21.15.98:0/3219249869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 0x55579daab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.769+0000 7f9fabfff640 1 -- 172.21.15.98:0/3219249869 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55579d7173f0 con 0x55579da9ec50 2024-05-09T08:53:43.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.770+0000 7f9fabfff640 1 --2- 172.21.15.98:0/3219249869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9ec50 0x55579daa8de0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00a9c0 tx=0x7f9f9c00ae80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcc19def23cd0b22 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:43.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.771+0000 7f9fa97fa640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9f940a2f00 0x7f9f940a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.771+0000 7f9fa97fa640 1 -- 172.21.15.98:0/2324849774 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f9fa0100a60 con 0x7f9fac110080 2024-05-09T08:53:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.773+0000 7f9fb0fc6640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9f940a2f00 0x7f9f940a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.773+0000 7f9f927fc640 1 -- 172.21.15.98:0/3219249869 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c015070 con 0x55579da9ec50 2024-05-09T08:53:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.773+0000 7f9f927fc640 1 -- 172.21.15.98:0/3219249869 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f9c00cc60 con 0x55579da9ec50 2024-05-09T08:53:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.774+0000 7f9f927fc640 1 -- 172.21.15.98:0/3219249869 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c0186d0 con 0x55579da9ec50 2024-05-09T08:53:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.774+0000 7f9fb580c740 1 -- 172.21.15.98:0/3219249869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9ec50 msgr2=0x55579daa8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.774+0000 7f9fb580c740 1 --2- 172.21.15.98:0/3219249869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9ec50 0x55579daa8de0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00a9c0 tx=0x7f9f9c00ae80 comp rx=0 tx=0).stop 2024-05-09T08:53:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.774+0000 7f9fb0fc6640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9f940a2f00 0x7f9f940a53c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4008da0 tx=0x7f9fa4008330 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.774+0000 7f9fb580c740 1 -- 172.21.15.98:0/3219249869 shutdown_connections 2024-05-09T08:53:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.774+0000 7f9fb580c740 1 --2- 172.21.15.98:0/3219249869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 0x55579daab710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.774+0000 7f9fb580c740 1 --2- 172.21.15.98:0/3219249869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9ec50 0x55579daa8de0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.774+0000 7f9fb580c740 1 --2- 172.21.15.98:0/3219249869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9e330 0x55579da9e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.774+0000 7f9fb580c740 1 -- 172.21.15.98:0/3219249869 >> 172.21.15.98:0/3219249869 conn(0x55579da9a650 msgr2=0x55579da9aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.775+0000 7f9fb580c740 1 -- 172.21.15.98:0/3219249869 shutdown_connections 2024-05-09T08:53:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.775+0000 7f9fb580c740 1 -- 172.21.15.98:0/3219249869 wait complete. 2024-05-09T08:53:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.775+0000 7f9fb580c740 1 -- start start 2024-05-09T08:53:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.775+0000 7f9fb580c740 1 Processor -- start 2024-05-09T08:53:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.776+0000 7f9fb580c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9e330 0x55579daa6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.776+0000 7f9fb580c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9ec50 0x55579daa67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.776+0000 7f9fb580c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 0x55579daa6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.776+0000 7f9fb580c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55579daac160 con 0x55579daa9320 2024-05-09T08:53:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.776+0000 7f9fb580c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55579d9bec40 con 0x55579da9e330 2024-05-09T08:53:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.776+0000 7f9fb580c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55579daac2e0 con 0x55579da9ec50 2024-05-09T08:53:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.776+0000 7f9fb580c740 10 client.0 _ll_register_callbacks cb 0x55579d716be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.777+0000 7f9fb0fc6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 0x55579daa6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.777+0000 7f9fb0fc6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 0x55579daa6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44088/0 (socket says 172.21.15.98:44088) 2024-05-09T08:53:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.777+0000 7f9fb0fc6640 1 -- 172.21.15.98:0/1832685769 learned_addr learned my addr 172.21.15.98:0/1832685769 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.777+0000 7f9fb0fc6640 1 -- 172.21.15.98:0/1832685769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9ec50 msgr2=0x55579daa67b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:53:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.777+0000 7f9fb0fc6640 1 --2- 172.21.15.98:0/1832685769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9ec50 0x55579daa67b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.777+0000 7f9fb0fc6640 1 -- 172.21.15.98:0/1832685769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9e330 msgr2=0x55579daa6270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:53:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.777+0000 7f9fb0fc6640 1 --2- 172.21.15.98:0/1832685769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9e330 0x55579daa6270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.777+0000 7f9fb0fc6640 1 -- 172.21.15.98:0/1832685769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55579d7e17d0 con 0x55579daa9320 2024-05-09T08:53:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.778+0000 7f9fb0fc6640 1 --2- 172.21.15.98:0/1832685769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 0x55579daa6cf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4010e50 tx=0x7f9fa4004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4acc3c2022098c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.778+0000 7f9f8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:43.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.778+0000 7f9f8b7fe640 1 -- 172.21.15.98:0/1832685769 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4021020 con 0x55579daa9320 2024-05-09T08:53:43.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.778+0000 7f9fb580c740 1 -- 172.21.15.98:0/1832685769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f9c00cc60 con 0x55579daa9320 2024-05-09T08:53:43.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.779+0000 7f9f8b7fe640 1 -- 172.21.15.98:0/1832685769 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f9fa400fd40 con 0x55579daa9320 2024-05-09T08:53:43.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.779+0000 7f9f8b7fe640 1 -- 172.21.15.98:0/1832685769 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9fa402c050 con 0x55579daa9320 2024-05-09T08:53:43.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.779+0000 7f9fb580c740 10 client.16536 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:43.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.779+0000 7f9fb580c740 1 -- 172.21.15.98:0/1832685769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55579dab84d0 con 0x55579daa9320 2024-05-09T08:53:43.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.780+0000 7f9f8b7fe640 1 -- 172.21.15.98:0/1832685769 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f9c00cc60 con 0x55579daa9320 2024-05-09T08:53:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.781+0000 7f9f8b7fe640 1 -- 172.21.15.98:0/1832685769 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9fa4028510 con 0x55579daa9320 2024-05-09T08:53:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.781+0000 7f9fb580c740 10 client.16536 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.781+0000 7f9fb580c740 10 client.16536 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:53:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.781+0000 7f9fb580c740 20 client.16536 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:43.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.781+0000 7f9fb580c740 1 --2- 172.21.15.98:0/1832685769 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55579dab9dd0 0x55579dabc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.781+0000 7f9fb580c740 4 client.16536 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:43.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.781+0000 7f9fb580c740 1 -- 172.21.15.98:0/1832685769 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55579da11370 con 0x55579dab9dd0 2024-05-09T08:53:43.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.782+0000 7f9fabfff640 1 --2- 172.21.15.98:0/1832685769 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55579dab9dd0 0x55579dabc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:43.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.786+0000 7f9fabfff640 1 --2- 172.21.15.98:0/1832685769 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55579dab9dd0 0x55579dabc250 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1f5e6945e129a210 server_cookie=66efedea881d011 in_seq=0 out_seq=0 2024-05-09T08:53:43.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.786+0000 7f9f8b7fe640 10 client.16536 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:53:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.788+0000 7f9f8b7fe640 1 -- 172.21.15.98:0/1832685769 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55579da11370 con 0x55579dab9dd0 2024-05-09T08:53:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.788+0000 7f9f8b7fe640 10 client.16536 handle_command_reply: tid=0 2024-05-09T08:53:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.789+0000 7f9fb580c740 1 client.16536 shutdown 2024-05-09T08:53:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.792+0000 7f9fb580c740 1 -- 172.21.15.98:0/1832685769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 msgr2=0x55579daa6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:43.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.792+0000 7f9fb580c740 1 --2- 172.21.15.98:0/1832685769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 0x55579daa6cf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4010e50 tx=0x7f9fa4004a60 comp rx=0 tx=0).stop 2024-05-09T08:53:43.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.794+0000 7f9fb580c740 1 -- 172.21.15.98:0/1832685769 shutdown_connections 2024-05-09T08:53:43.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.794+0000 7f9fb580c740 1 --2- 172.21.15.98:0/1832685769 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55579dab9dd0 0x55579dabc250 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.794+0000 7f9fb580c740 1 --2- 172.21.15.98:0/1832685769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 0x55579daa6cf0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.794+0000 7f9fb580c740 1 --2- 172.21.15.98:0/1832685769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9ec50 0x55579daa67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.794+0000 7f9fb580c740 1 --2- 172.21.15.98:0/1832685769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9e330 0x55579daa6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.794+0000 7f9fb580c740 1 -- 172.21.15.98:0/1832685769 >> 172.21.15.98:0/1832685769 conn(0x55579da9a650 msgr2=0x55579daa0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:43.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.795+0000 7f9f8b7fe640 0 client.16536 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:53:43.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.796+0000 7f9fb580c740 1 -- 172.21.15.98:0/1832685769 shutdown_connections 2024-05-09T08:53:43.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.797+0000 7f9fb580c740 1 -- 172.21.15.98:0/1832685769 wait complete. 2024-05-09T08:53:43.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.797+0000 7f9fb580c740 20 client.16536 trim_cache size 0 max 16384 2024-05-09T08:53:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.822+0000 7f9fb580c740 1 Processor -- start 2024-05-09T08:53:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.822+0000 7f9fb580c740 1 -- start start 2024-05-09T08:53:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.822+0000 7f9fb580c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579da9e330 0x55579dab5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.822+0000 7f9fb580c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9ec50 0x55579daa0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.822+0000 7f9fb580c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579daa9320 0x55579dac1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.822+0000 7f9fb580c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9800fef0 con 0x55579da9e330 2024-05-09T08:53:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.822+0000 7f9fb580c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9f98011a30 con 0x55579daa9320 2024-05-09T08:53:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.822+0000 7f9fabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579da9e330 0x55579dab5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.822+0000 7f9fabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579da9e330 0x55579dab5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44090/0 (socket says 172.21.15.98:44090) 2024-05-09T08:53:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.822+0000 7f9fabfff640 1 -- 172.21.15.98:0/3652185231 learned_addr learned my addr 172.21.15.98:0/3652185231 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.822+0000 7f9fb580c740 1 -- 172.21.15.98:0/3652185231 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9f980105c0 con 0x55579da9ec50 2024-05-09T08:53:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.823+0000 7f9fb0fc6640 1 --2- 172.21.15.98:0/3652185231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579daa9320 0x55579dac1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.823+0000 7f9fabfff640 1 -- 172.21.15.98:0/3652185231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9ec50 msgr2=0x55579daa0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.823+0000 7f9fabfff640 1 --2- 172.21.15.98:0/3652185231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9ec50 0x55579daa0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.823+0000 7f9fabfff640 1 -- 172.21.15.98:0/3652185231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579daa9320 msgr2=0x55579dac1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.823+0000 7f9fabfff640 1 --2- 172.21.15.98:0/3652185231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579daa9320 0x55579dac1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.823+0000 7f9fabfff640 1 -- 172.21.15.98:0/3652185231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55579da8bd20 con 0x55579da9e330 2024-05-09T08:53:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.823+0000 7f9fabfff640 1 --2- 172.21.15.98:0/3652185231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579da9e330 0x55579dab5100 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00a8e0 tx=0x7f9f9c011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bed9d212a19fbac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.823+0000 7f9f91ffb640 1 -- 172.21.15.98:0/3652185231 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c015040 con 0x55579da9e330 2024-05-09T08:53:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.823+0000 7f9f91ffb640 1 -- 172.21.15.98:0/3652185231 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f9c004530 con 0x55579da9e330 2024-05-09T08:53:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.823+0000 7f9f91ffb640 1 -- 172.21.15.98:0/3652185231 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c003690 con 0x55579da9e330 2024-05-09T08:53:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.825+0000 7f9fb580c740 1 -- 172.21.15.98:0/3652185231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579da9e330 msgr2=0x55579dab5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.825+0000 7f9fb580c740 1 --2- 172.21.15.98:0/3652185231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579da9e330 0x55579dab5100 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00a8e0 tx=0x7f9f9c011ec0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.825+0000 7f9fb580c740 1 -- 172.21.15.98:0/3652185231 shutdown_connections 2024-05-09T08:53:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.825+0000 7f9fb580c740 1 --2- 172.21.15.98:0/3652185231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579daa9320 0x55579dac1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.825+0000 7f9fb580c740 1 --2- 172.21.15.98:0/3652185231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9ec50 0x55579daa0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.825+0000 7f9fb580c740 1 --2- 172.21.15.98:0/3652185231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579da9e330 0x55579dab5100 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.825+0000 7f9fb580c740 1 -- 172.21.15.98:0/3652185231 >> 172.21.15.98:0/3652185231 conn(0x55579da9a650 msgr2=0x55579daa8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.826+0000 7f9fb580c740 1 -- 172.21.15.98:0/3652185231 shutdown_connections 2024-05-09T08:53:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.826+0000 7f9fb580c740 1 -- 172.21.15.98:0/3652185231 wait complete. 2024-05-09T08:53:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.826+0000 7f9fb580c740 1 -- start start 2024-05-09T08:53:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.827+0000 7f9fb580c740 1 Processor -- start 2024-05-09T08:53:43.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.827+0000 7f9fb580c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9e330 0x55579dab1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.828+0000 7f9fabfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9e330 0x55579dab1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.828+0000 7f9fabfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9e330 0x55579dab1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55286/0 (socket says 172.21.15.98:55286) 2024-05-09T08:53:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.828+0000 7f9fb580c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9ec50 0x55579daa2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.828+0000 7f9fabfff640 1 -- 172.21.15.98:0/254332906 learned_addr learned my addr 172.21.15.98:0/254332906 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.828+0000 7f9fb580c740 1 --2- 172.21.15.98:0/254332906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 0x55579daa3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.828+0000 7f9fab7fe640 1 --2- 172.21.15.98:0/254332906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9ec50 0x55579daa2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.828+0000 7f9fb580c740 1 -- 172.21.15.98:0/254332906 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9f98011360 con 0x55579daa9320 2024-05-09T08:53:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.828+0000 7f9fb580c740 1 -- 172.21.15.98:0/254332906 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9f98010c90 con 0x55579da9e330 2024-05-09T08:53:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.828+0000 7f9fb580c740 1 -- 172.21.15.98:0/254332906 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9800df80 con 0x55579da9ec50 2024-05-09T08:53:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.828+0000 7f9fb580c740 10 client.0 _ll_register_callbacks cb 0x7f9fa40085c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.828+0000 7f9fb0fc6640 1 --2- 172.21.15.98:0/254332906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 0x55579daa3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.829+0000 7f9fb0fc6640 1 -- 172.21.15.98:0/254332906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9ec50 msgr2=0x55579daa2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.829+0000 7f9fb0fc6640 1 --2- 172.21.15.98:0/254332906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9ec50 0x55579daa2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.829+0000 7f9fb0fc6640 1 -- 172.21.15.98:0/254332906 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9e330 msgr2=0x55579dab1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.829+0000 7f9fb0fc6640 1 --2- 172.21.15.98:0/254332906 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9e330 0x55579dab1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.829+0000 7f9fb0fc6640 1 -- 172.21.15.98:0/254332906 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55579daa3590 con 0x55579daa9320 2024-05-09T08:53:43.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.830+0000 7f9fb0fc6640 1 --2- 172.21.15.98:0/254332906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 0x55579daa3020 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4010e20 tx=0x7f9fa400fef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606fe0a1a04b3e59 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:43.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.830+0000 7f9f8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:43.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.831+0000 7f9f8b7fe640 1 -- 172.21.15.98:0/254332906 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4021040 con 0x55579daa9320 2024-05-09T08:53:43.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.831+0000 7f9f8b7fe640 1 -- 172.21.15.98:0/254332906 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f9fa4028040 con 0x55579daa9320 2024-05-09T08:53:43.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.831+0000 7f9fb580c740 1 -- 172.21.15.98:0/254332906 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f9c004530 con 0x55579daa9320 2024-05-09T08:53:43.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.831+0000 7f9f8b7fe640 1 -- 172.21.15.98:0/254332906 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9fa4028260 con 0x55579daa9320 2024-05-09T08:53:43.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.831+0000 7f9fb580c740 10 client.16542 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:43.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.831+0000 7f9fb580c740 1 -- 172.21.15.98:0/254332906 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55579db20150 con 0x55579daa9320 2024-05-09T08:53:43.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.832+0000 7f9f8b7fe640 1 -- 172.21.15.98:0/254332906 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f9c004530 con 0x55579daa9320 2024-05-09T08:53:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.832+0000 7f9f8b7fe640 1 -- 172.21.15.98:0/254332906 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9fa401dc20 con 0x55579daa9320 2024-05-09T08:53:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.832+0000 7f9fb580c740 10 client.16542 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.832+0000 7f9fb580c740 10 client.16542 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:53:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.832+0000 7f9fb580c740 20 client.16542 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.832+0000 7f9fb580c740 1 --2- 172.21.15.98:0/254332906 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55579db20810 0x55579db22c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.832+0000 7f9fb580c740 4 client.16542 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:53:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.832+0000 7f9fb580c740 1 -- 172.21.15.98:0/254332906 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55579da0b6c0 con 0x55579db20810 2024-05-09T08:53:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.833+0000 7f9fabfff640 1 --2- 172.21.15.98:0/254332906 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55579db20810 0x55579db22c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.834+0000 7f9f8b7fe640 10 client.16542 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:53:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.834+0000 7f9fabfff640 1 --2- 172.21.15.98:0/254332906 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55579db20810 0x55579db22c00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6769aef8f91a003 server_cookie=f3af39971b22b6be in_seq=0 out_seq=0 2024-05-09T08:53:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.834+0000 7f9f8b7fe640 1 -- 172.21.15.98:0/254332906 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55579da0b6c0 con 0x55579db20810 2024-05-09T08:53:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.834+0000 7f9f8b7fe640 10 client.16542 handle_command_reply: tid=0 2024-05-09T08:53:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.834+0000 7f9fb580c740 1 client.16542 shutdown 2024-05-09T08:53:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.835+0000 7f9fb580c740 1 -- 172.21.15.98:0/254332906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 msgr2=0x55579daa3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.835+0000 7f9fb580c740 1 --2- 172.21.15.98:0/254332906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 0x55579daa3020 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4010e20 tx=0x7f9fa400fef0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.835+0000 7f9fb580c740 1 -- 172.21.15.98:0/254332906 shutdown_connections 2024-05-09T08:53:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.835+0000 7f9fb580c740 1 --2- 172.21.15.98:0/254332906 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55579db20810 0x55579db22c00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.835+0000 7f9fb580c740 1 --2- 172.21.15.98:0/254332906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55579daa9320 0x55579daa3020 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.835+0000 7f9fb580c740 1 --2- 172.21.15.98:0/254332906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55579da9ec50 0x55579daa2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.835+0000 7f9fb580c740 1 --2- 172.21.15.98:0/254332906 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55579da9e330 0x55579dab1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.835+0000 7f9fb580c740 1 -- 172.21.15.98:0/254332906 >> 172.21.15.98:0/254332906 conn(0x55579da9a650 msgr2=0x55579daa7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:43.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.835+0000 7f9f8b7fe640 0 client.16542 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:53:43.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.835+0000 7f9fb580c740 1 -- 172.21.15.98:0/254332906 shutdown_connections 2024-05-09T08:53:43.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.835+0000 7f9fb580c740 1 -- 172.21.15.98:0/254332906 wait complete. 2024-05-09T08:53:43.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.835+0000 7f9fb580c740 20 client.16542 trim_cache size 0 max 16384 2024-05-09T08:53:43.836 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:53:43.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.836+0000 7f9fb2a29640 1 -- 172.21.15.98:0/2324849774 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9f940a2f00 msgr2=0x7f9f940a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:43.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.836+0000 7f9fb2a29640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9f940a2f00 0x7f9f940a53c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4008da0 tx=0x7f9fa4008330 comp rx=0 tx=0).stop 2024-05-09T08:53:43.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.836+0000 7f9fb2a29640 1 -- 172.21.15.98:0/2324849774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fac110080 msgr2=0x7f9fac1169e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:43.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.836+0000 7f9fb2a29640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fac110080 0x7f9fac1169e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0005ab0 tx=0x7f9fa0004c20 comp rx=0 tx=0).stop 2024-05-09T08:53:43.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.836+0000 7f9fb2a29640 1 -- 172.21.15.98:0/2324849774 shutdown_connections 2024-05-09T08:53:43.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.836+0000 7f9fb2a29640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9f940a2f00 0x7f9f940a53c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.836+0000 7f9fb2a29640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fac11b010 0x7f9fac116f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.836+0000 7f9fb2a29640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fac110080 0x7f9fac1169e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.836+0000 7f9fb2a29640 1 --2- 172.21.15.98:0/2324849774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fac10e0b0 0x7f9fac11a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:43.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.836+0000 7f9fb2a29640 1 -- 172.21.15.98:0/2324849774 >> 172.21.15.98:0/2324849774 conn(0x7f9fac06c940 msgr2=0x7f9fac11bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:43.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.837+0000 7f9fb2a29640 1 -- 172.21.15.98:0/2324849774 shutdown_connections 2024-05-09T08:53:43.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:43.837+0000 7f9fb2a29640 1 -- 172.21.15.98:0/2324849774 wait complete. 2024-05-09T08:53:43.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:43 smithi098 ceph-mon[30092]: pgmap v95: 129 pgs: 129 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 733 MiB/s wr, 1.93k op/s 2024-05-09T08:53:43.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:43 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1135501033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:53:43.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:53:43.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:43 smithi105 ceph-mon[34498]: pgmap v95: 129 pgs: 129 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 733 MiB/s wr, 1.93k op/s 2024-05-09T08:53:43.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:43 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1135501033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:53:44.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:43 smithi142 ceph-mon[34625]: pgmap v95: 129 pgs: 129 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 733 MiB/s wr, 1.93k op/s 2024-05-09T08:53:44.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:43 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1135501033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:53:44.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.196+0000 7f926f37a640 1 -- 172.21.15.98:0/4273125406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92681098f0 msgr2=0x7f9268109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.196+0000 7f926f37a640 1 --2- 172.21.15.98:0/4273125406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92681098f0 0x7f9268109cd0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9258009cd0 tx=0x7f925802a930 comp rx=0 tx=0).stop 2024-05-09T08:53:44.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.198+0000 7f926f37a640 1 -- 172.21.15.98:0/4273125406 shutdown_connections 2024-05-09T08:53:44.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.198+0000 7f926f37a640 1 --2- 172.21.15.98:0/4273125406 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92681042a0 0x7f92681107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.198+0000 7f926f37a640 1 --2- 172.21.15.98:0/4273125406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92681038e0 0x7f9268103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.198+0000 7f926f37a640 1 --2- 172.21.15.98:0/4273125406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92681098f0 0x7f9268109cd0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.198+0000 7f926f37a640 1 -- 172.21.15.98:0/4273125406 >> 172.21.15.98:0/4273125406 conn(0x7f92680ff370 msgr2=0x7f9268101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.200+0000 7f926f37a640 1 -- 172.21.15.98:0/4273125406 shutdown_connections 2024-05-09T08:53:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.201+0000 7f926f37a640 1 -- 172.21.15.98:0/4273125406 wait complete. 2024-05-09T08:53:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.202+0000 7f926f37a640 1 Processor -- start 2024-05-09T08:53:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.205+0000 7f926f37a640 1 -- start start 2024-05-09T08:53:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.205+0000 7f926f37a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92681038e0 0x7f926810f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.205+0000 7f926f37a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92681042a0 0x7f926810fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.205+0000 7f926f37a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9268110160 0x7f9268114a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.205+0000 7f926f37a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9268076fd0 con 0x7f92681042a0 2024-05-09T08:53:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.205+0000 7f926f37a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9268076e50 con 0x7f92681038e0 2024-05-09T08:53:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.205+0000 7f926f37a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9268077150 con 0x7f9268110160 2024-05-09T08:53:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.207+0000 7f926e378640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92681038e0 0x7f926810f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.207+0000 7f926e378640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92681038e0 0x7f926810f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55290/0 (socket says 172.21.15.98:55290) 2024-05-09T08:53:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.207+0000 7f926e378640 1 -- 172.21.15.98:0/291133497 learned_addr learned my addr 172.21.15.98:0/291133497 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.215+0000 7f926eb79640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9268110160 0x7f9268114a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.215+0000 7f926db77640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92681042a0 0x7f926810fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.215+0000 7f926e378640 1 -- 172.21.15.98:0/291133497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9268110160 msgr2=0x7f9268114a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.215+0000 7f926e378640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9268110160 0x7f9268114a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.215+0000 7f926e378640 1 -- 172.21.15.98:0/291133497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92681042a0 msgr2=0x7f926810fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.215+0000 7f926e378640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92681042a0 0x7f926810fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.215+0000 7f926e378640 1 -- 172.21.15.98:0/291133497 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9268115300 con 0x7f92681038e0 2024-05-09T08:53:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.216+0000 7f926e378640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92681038e0 0x7f926810f5a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9258009ca0 tx=0x7f9258030fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.216+0000 7f926db77640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92681042a0 0x7f926810fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:53:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.216+0000 7f926eb79640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9268110160 0x7f9268114a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:53:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.216+0000 7f92677fe640 1 -- 172.21.15.98:0/291133497 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258003c70 con 0x7f92681038e0 2024-05-09T08:53:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.216+0000 7f926f37a640 1 -- 172.21.15.98:0/291133497 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9268115590 con 0x7f92681038e0 2024-05-09T08:53:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.216+0000 7f926f37a640 1 -- 172.21.15.98:0/291133497 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9268115b30 con 0x7f92681038e0 2024-05-09T08:53:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.217+0000 7f92677fe640 1 -- 172.21.15.98:0/291133497 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f925802d070 con 0x7f92681038e0 2024-05-09T08:53:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.217+0000 7f92677fe640 1 -- 172.21.15.98:0/291133497 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258004540 con 0x7f92681038e0 2024-05-09T08:53:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.221+0000 7f92733ca740 1 Processor -- start 2024-05-09T08:53:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.221+0000 7f92677fe640 1 -- 172.21.15.98:0/291133497 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f92580046e0 con 0x7f92681038e0 2024-05-09T08:53:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.221+0000 7f92733ca740 1 -- start start 2024-05-09T08:53:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.221+0000 7f92733ca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a234f0 0x555f48a238d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.221+0000 7f92733ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a23e10 0x555f48a2dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.221+0000 7f92733ca740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a2e4e0 0x555f48a308d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f92733ca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555f48988c40 con 0x555f48a234f0 2024-05-09T08:53:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f92733ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555f48930580 con 0x555f48a23e10 2024-05-09T08:53:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f92733ca740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555f48984c00 con 0x555f48a2e4e0 2024-05-09T08:53:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f926eb79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a234f0 0x555f48a238d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f926e378640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a23e10 0x555f48a2dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f926eb79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a234f0 0x555f48a238d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44116/0 (socket says 172.21.15.98:44116) 2024-05-09T08:53:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f926db77640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a2e4e0 0x555f48a308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f926eb79640 1 -- 172.21.15.98:0/1609109872 learned_addr learned my addr 172.21.15.98:0/1609109872 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f926e378640 1 -- 172.21.15.98:0/1609109872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a2e4e0 msgr2=0x555f48a308d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f926e378640 1 --2- 172.21.15.98:0/1609109872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a2e4e0 0x555f48a308d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f926e378640 1 -- 172.21.15.98:0/1609109872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a234f0 msgr2=0x555f48a238d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f926e378640 1 --2- 172.21.15.98:0/1609109872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a234f0 0x555f48a238d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f926e378640 1 -- 172.21.15.98:0/1609109872 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f4869c3f0 con 0x555f48a23e10 2024-05-09T08:53:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f926db77640 1 --2- 172.21.15.98:0/1609109872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a2e4e0 0x555f48a308d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:53:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.222+0000 7f926eb79640 1 --2- 172.21.15.98:0/1609109872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a234f0 0x555f48a238d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:53:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.223+0000 7f926e378640 1 --2- 172.21.15.98:0/1609109872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a23e10 0x555f48a2dfa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f92580ee080 tx=0x7f92580f4d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82acf4ebc755a028 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.223+0000 7f92677fe640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f92500a2dc0 0x7f92500a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.223+0000 7f926eb79640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f92500a2dc0 0x7f92500a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.223+0000 7f9264bfe640 1 -- 172.21.15.98:0/1609109872 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258031940 con 0x555f48a23e10 2024-05-09T08:53:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.223+0000 7f9264bfe640 1 -- 172.21.15.98:0/1609109872 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9258036030 con 0x555f48a23e10 2024-05-09T08:53:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.223+0000 7f9264bfe640 1 -- 172.21.15.98:0/1609109872 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580bc690 con 0x555f48a23e10 2024-05-09T08:53:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.224+0000 7f92677fe640 1 -- 172.21.15.98:0/291133497 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f92580f0be0 con 0x7f92681038e0 2024-05-09T08:53:44.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.225+0000 7f926eb79640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f92500a2dc0 0x7f92500a5280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9260006fd0 tx=0x7f9260008040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.225+0000 7f92733ca740 1 -- 172.21.15.98:0/1609109872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a23e10 msgr2=0x555f48a2dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.225+0000 7f92733ca740 1 --2- 172.21.15.98:0/1609109872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a23e10 0x555f48a2dfa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f92580ee080 tx=0x7f92580f4d20 comp rx=0 tx=0).stop 2024-05-09T08:53:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.225+0000 7f92733ca740 1 -- 172.21.15.98:0/1609109872 shutdown_connections 2024-05-09T08:53:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.225+0000 7f92733ca740 1 --2- 172.21.15.98:0/1609109872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a2e4e0 0x555f48a308d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.225+0000 7f92733ca740 1 --2- 172.21.15.98:0/1609109872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a23e10 0x555f48a2dfa0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.225+0000 7f92733ca740 1 --2- 172.21.15.98:0/1609109872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a234f0 0x555f48a238d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.225+0000 7f92733ca740 1 -- 172.21.15.98:0/1609109872 >> 172.21.15.98:0/1609109872 conn(0x555f48a1f810 msgr2=0x555f48a1fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.225+0000 7f92733ca740 1 -- 172.21.15.98:0/1609109872 shutdown_connections 2024-05-09T08:53:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.225+0000 7f92733ca740 1 -- 172.21.15.98:0/1609109872 wait complete. 2024-05-09T08:53:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.226+0000 7f92733ca740 1 -- start start 2024-05-09T08:53:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.226+0000 7f92733ca740 1 Processor -- start 2024-05-09T08:53:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.226+0000 7f92733ca740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a234f0 0x555f48a2b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.226+0000 7f92733ca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a23e10 0x555f48a2b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f92733ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a2e4e0 0x555f48a2bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f92733ca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555f48a31090 con 0x555f48a23e10 2024-05-09T08:53:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f92733ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555f48943c40 con 0x555f48a2e4e0 2024-05-09T08:53:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f92733ca740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555f48a31210 con 0x555f48a234f0 2024-05-09T08:53:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f926e378640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a234f0 0x555f48a2b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f92733ca740 10 client.0 _ll_register_callbacks cb 0x555f4869bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f926e378640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a234f0 0x555f48a2b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:57210/0 (socket says 172.21.15.98:57210) 2024-05-09T08:53:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f926e378640 1 -- 172.21.15.98:0/3128667827 learned_addr learned my addr 172.21.15.98:0/3128667827 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f926eb79640 1 --2- 172.21.15.98:0/3128667827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a2e4e0 0x555f48a2bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f926db77640 1 --2- 172.21.15.98:0/3128667827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a23e10 0x555f48a2b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f926eb79640 1 -- 172.21.15.98:0/3128667827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a234f0 msgr2=0x555f48a2b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f926eb79640 1 --2- 172.21.15.98:0/3128667827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a234f0 0x555f48a2b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f926eb79640 1 -- 172.21.15.98:0/3128667827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a23e10 msgr2=0x555f48a2b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f926eb79640 1 --2- 172.21.15.98:0/3128667827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a23e10 0x555f48a2b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f926eb79640 1 -- 172.21.15.98:0/3128667827 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f487667d0 con 0x555f48a2e4e0 2024-05-09T08:53:44.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f926db77640 1 --2- 172.21.15.98:0/3128667827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a23e10 0x555f48a2b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:53:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f926eb79640 1 --2- 172.21.15.98:0/3128667827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a2e4e0 0x555f48a2bec0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f926000a820 tx=0x7f9260004700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=115a6e38ca8b5aee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.227+0000 7f924dffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:53:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.228+0000 7f924dffb640 1 -- 172.21.15.98:0/3128667827 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926001b050 con 0x555f48a2e4e0 2024-05-09T08:53:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.228+0000 7f92733ca740 1 -- 172.21.15.98:0/3128667827 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9258036030 con 0x555f48a2e4e0 2024-05-09T08:53:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.228+0000 7f924dffb640 1 -- 172.21.15.98:0/3128667827 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f9260015820 con 0x555f48a2e4e0 2024-05-09T08:53:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.228+0000 7f924dffb640 1 -- 172.21.15.98:0/3128667827 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9260015a20 con 0x555f48a2e4e0 2024-05-09T08:53:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.228+0000 7f92733ca740 10 client.25883 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.228+0000 7f92733ca740 1 -- 172.21.15.98:0/3128667827 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555f48a3d220 con 0x555f48a2e4e0 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.228+0000 7f924dffb640 1 -- 172.21.15.98:0/3128667827 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9258036030 con 0x555f48a2e4e0 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.229+0000 7f924dffb640 1 -- 172.21.15.98:0/3128667827 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9260006320 con 0x555f48a2e4e0 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.229+0000 7f92733ca740 10 client.25883 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.229+0000 7f92733ca740 10 client.25883 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.229+0000 7f92733ca740 20 client.25883 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.229+0000 7f92733ca740 1 --2- 172.21.15.98:0/3128667827 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x555f48a3e7a0 0x555f48a40c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.229+0000 7f92733ca740 4 client.25883 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.229+0000 7f92733ca740 1 -- 172.21.15.98:0/3128667827 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f48996370 con 0x555f48a3e7a0 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.229+0000 7f926e378640 1 --2- 172.21.15.98:0/3128667827 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x555f48a3e7a0 0x555f48a40c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.230+0000 7f924dffb640 10 client.25883 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.230+0000 7f926e378640 1 --2- 172.21.15.98:0/3128667827 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x555f48a3e7a0 0x555f48a40c10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b8b8744f48d91826 server_cookie=5c37aca3a5713899 in_seq=0 out_seq=0 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.231+0000 7f924dffb640 1 -- 172.21.15.98:0/3128667827 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f48996370 con 0x555f48a3e7a0 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.231+0000 7f924dffb640 10 client.25883 handle_command_reply: tid=0 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.231+0000 7f92733ca740 1 client.25883 shutdown 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.231+0000 7f92733ca740 1 -- 172.21.15.98:0/3128667827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a2e4e0 msgr2=0x555f48a2bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.231+0000 7f92733ca740 1 --2- 172.21.15.98:0/3128667827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a2e4e0 0x555f48a2bec0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f926000a820 tx=0x7f9260004700 comp rx=0 tx=0).stop 2024-05-09T08:53:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.231+0000 7f92733ca740 1 -- 172.21.15.98:0/3128667827 shutdown_connections 2024-05-09T08:53:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.231+0000 7f92733ca740 1 --2- 172.21.15.98:0/3128667827 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x555f48a3e7a0 0x555f48a40c10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.231+0000 7f92733ca740 1 --2- 172.21.15.98:0/3128667827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a2e4e0 0x555f48a2bec0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.231+0000 7f92733ca740 1 --2- 172.21.15.98:0/3128667827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a23e10 0x555f48a2b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.231+0000 7f924dffb640 0 client.25883 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:53:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.231+0000 7f92733ca740 1 --2- 172.21.15.98:0/3128667827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a234f0 0x555f48a2b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.231+0000 7f92733ca740 1 -- 172.21.15.98:0/3128667827 >> 172.21.15.98:0/3128667827 conn(0x555f48a1f810 msgr2=0x555f48a1e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.231+0000 7f92733ca740 1 -- 172.21.15.98:0/3128667827 shutdown_connections 2024-05-09T08:53:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.231+0000 7f92733ca740 1 -- 172.21.15.98:0/3128667827 wait complete. 2024-05-09T08:53:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.232+0000 7f92733ca740 20 client.25883 trim_cache size 0 max 16384 2024-05-09T08:53:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.248+0000 7f92733ca740 1 Processor -- start 2024-05-09T08:53:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.248+0000 7f92733ca740 1 -- start start 2024-05-09T08:53:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.248+0000 7f92733ca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a234f0 0x555f48a39f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f92733ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a23e10 0x555f48a25c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f92733ca740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a2e4e0 0x555f48a46520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f92733ca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f925400ff90 con 0x555f48a234f0 2024-05-09T08:53:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f92733ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9254011ad0 con 0x555f48a23e10 2024-05-09T08:53:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f92733ca740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9254010660 con 0x555f48a2e4e0 2024-05-09T08:53:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f926e378640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a234f0 0x555f48a39f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f926e378640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a234f0 0x555f48a39f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44122/0 (socket says 172.21.15.98:44122) 2024-05-09T08:53:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f926e378640 1 -- 172.21.15.98:0/943063568 learned_addr learned my addr 172.21.15.98:0/943063568 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f926db77640 1 --2- 172.21.15.98:0/943063568 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a23e10 0x555f48a25c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f926eb79640 1 --2- 172.21.15.98:0/943063568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a2e4e0 0x555f48a46520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f926e378640 1 -- 172.21.15.98:0/943063568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a2e4e0 msgr2=0x555f48a46520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f926e378640 1 --2- 172.21.15.98:0/943063568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a2e4e0 0x555f48a46520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f926e378640 1 -- 172.21.15.98:0/943063568 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a23e10 msgr2=0x555f48a25c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f926e378640 1 --2- 172.21.15.98:0/943063568 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a23e10 0x555f48a25c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f926e378640 1 -- 172.21.15.98:0/943063568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f48a1b170 con 0x555f48a234f0 2024-05-09T08:53:44.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.249+0000 7f926e378640 1 --2- 172.21.15.98:0/943063568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a234f0 0x555f48a39f50 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f9258031040 tx=0x7f92580095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3250e6e5f50a21 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.250+0000 7f924ffff640 1 -- 172.21.15.98:0/943063568 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580bc870 con 0x555f48a234f0 2024-05-09T08:53:44.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.250+0000 7f924ffff640 1 -- 172.21.15.98:0/943063568 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9258036040 con 0x555f48a234f0 2024-05-09T08:53:44.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.250+0000 7f924ffff640 1 -- 172.21.15.98:0/943063568 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580bca50 con 0x555f48a234f0 2024-05-09T08:53:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.250+0000 7f92733ca740 1 -- 172.21.15.98:0/943063568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a234f0 msgr2=0x555f48a39f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.250+0000 7f92733ca740 1 --2- 172.21.15.98:0/943063568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a234f0 0x555f48a39f50 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f9258031040 tx=0x7f92580095c0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.250+0000 7f92733ca740 1 -- 172.21.15.98:0/943063568 shutdown_connections 2024-05-09T08:53:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.250+0000 7f92733ca740 1 --2- 172.21.15.98:0/943063568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a2e4e0 0x555f48a46520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.250+0000 7f92733ca740 1 --2- 172.21.15.98:0/943063568 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a23e10 0x555f48a25c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.250+0000 7f92733ca740 1 --2- 172.21.15.98:0/943063568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a234f0 0x555f48a39f50 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.250+0000 7f92733ca740 1 -- 172.21.15.98:0/943063568 >> 172.21.15.98:0/943063568 conn(0x555f48a1f810 msgr2=0x555f48a1e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.251+0000 7f92733ca740 1 -- 172.21.15.98:0/943063568 shutdown_connections 2024-05-09T08:53:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.251+0000 7f92733ca740 1 -- 172.21.15.98:0/943063568 wait complete. 2024-05-09T08:53:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.251+0000 7f92733ca740 1 -- start start 2024-05-09T08:53:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.251+0000 7f92733ca740 1 Processor -- start 2024-05-09T08:53:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.251+0000 7f92733ca740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a234f0 0x555f48a36860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.251+0000 7f92733ca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a23e10 0x555f48a31bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.251+0000 7f92733ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a2e4e0 0x555f48a32110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f926db77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a23e10 0x555f48a31bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f926e378640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a234f0 0x555f48a36860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f926e378640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a234f0 0x555f48a36860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:57234/0 (socket says 172.21.15.98:57234) 2024-05-09T08:53:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f926db77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a23e10 0x555f48a31bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44130/0 (socket says 172.21.15.98:44130) 2024-05-09T08:53:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f926eb79640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a2e4e0 0x555f48a32110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f926eb79640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a2e4e0 0x555f48a32110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55336/0 (socket says 172.21.15.98:55336) 2024-05-09T08:53:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f926e378640 1 -- 172.21.15.98:0/3942597083 learned_addr learned my addr 172.21.15.98:0/3942597083 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f92733ca740 1 -- 172.21.15.98:0/3942597083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9254011400 con 0x555f48a23e10 2024-05-09T08:53:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f92733ca740 1 -- 172.21.15.98:0/3942597083 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9254010d30 con 0x555f48a2e4e0 2024-05-09T08:53:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f92733ca740 1 -- 172.21.15.98:0/3942597083 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f925400df70 con 0x555f48a234f0 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f92733ca740 10 client.0 _ll_register_callbacks cb 0x7f9260006c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f926e378640 1 -- 172.21.15.98:0/3942597083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a2e4e0 msgr2=0x555f48a32110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f926e378640 1 --2- 172.21.15.98:0/3942597083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a2e4e0 0x555f48a32110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f926e378640 1 -- 172.21.15.98:0/3942597083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a23e10 msgr2=0x555f48a31bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f926e378640 1 --2- 172.21.15.98:0/3942597083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a23e10 0x555f48a31bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f926e378640 1 -- 172.21.15.98:0/3942597083 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f48a32940 con 0x555f48a234f0 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f926eb79640 1 --2- 172.21.15.98:0/3942597083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a2e4e0 0x555f48a32110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f926e378640 1 --2- 172.21.15.98:0/3942597083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a234f0 0x555f48a36860 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f92580049d0 tx=0x7f92580f4d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf5c67b719cbb306 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f924dffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f924dffb640 1 -- 172.21.15.98:0/3942597083 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580bccc0 con 0x555f48a234f0 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.252+0000 7f92733ca740 1 -- 172.21.15.98:0/3942597083 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9258036040 con 0x555f48a234f0 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.253+0000 7f924dffb640 1 -- 172.21.15.98:0/3942597083 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f925804d390 con 0x555f48a234f0 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.253+0000 7f924dffb640 1 -- 172.21.15.98:0/3942597083 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9258031480 con 0x555f48a234f0 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.253+0000 7f92733ca740 10 client.25843 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.253+0000 7f92733ca740 1 -- 172.21.15.98:0/3942597083 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555f48aa4dd0 con 0x555f48a234f0 2024-05-09T08:53:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.253+0000 7f924dffb640 1 -- 172.21.15.98:0/3942597083 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92580bf070 con 0x555f48a234f0 2024-05-09T08:53:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.253+0000 7f924dffb640 1 -- 172.21.15.98:0/3942597083 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f92580ee160 con 0x555f48a234f0 2024-05-09T08:53:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.253+0000 7f92733ca740 10 client.25843 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.253+0000 7f92733ca740 10 client.25843 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:53:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.253+0000 7f92733ca740 20 client.25843 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.253+0000 7f92733ca740 1 --2- 172.21.15.98:0/3942597083 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x555f48aa5350 0x555f48aa7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.253+0000 7f92733ca740 4 client.25843 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:53:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.253+0000 7f92733ca740 1 -- 172.21.15.98:0/3942597083 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555f489906c0 con 0x555f48aa5350 2024-05-09T08:53:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.253+0000 7f926db77640 1 --2- 172.21.15.98:0/3942597083 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x555f48aa5350 0x555f48aa7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f926db77640 1 --2- 172.21.15.98:0/3942597083 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x555f48aa5350 0x555f48aa7740 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7f1fa1eeef03e6ba server_cookie=c388f64c9ba3b0ac in_seq=0 out_seq=0 2024-05-09T08:53:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f924dffb640 10 client.25843 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:53:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f924dffb640 1 -- 172.21.15.98:0/3942597083 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555f489906c0 con 0x555f48aa5350 2024-05-09T08:53:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f924dffb640 10 client.25843 handle_command_reply: tid=0 2024-05-09T08:53:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f92733ca740 1 client.25843 shutdown 2024-05-09T08:53:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f92733ca740 1 -- 172.21.15.98:0/3942597083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a234f0 msgr2=0x555f48a36860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f92733ca740 1 --2- 172.21.15.98:0/3942597083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a234f0 0x555f48a36860 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f92580049d0 tx=0x7f92580f4d20 comp rx=0 tx=0).stop 2024-05-09T08:53:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f92733ca740 1 -- 172.21.15.98:0/3942597083 shutdown_connections 2024-05-09T08:53:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f92733ca740 1 --2- 172.21.15.98:0/3942597083 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x555f48aa5350 0x555f48aa7740 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f92733ca740 1 --2- 172.21.15.98:0/3942597083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f48a2e4e0 0x555f48a32110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f92733ca740 1 --2- 172.21.15.98:0/3942597083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f48a23e10 0x555f48a31bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f924dffb640 0 client.25843 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:53:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f92733ca740 1 --2- 172.21.15.98:0/3942597083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f48a234f0 0x555f48a36860 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f92733ca740 1 -- 172.21.15.98:0/3942597083 >> 172.21.15.98:0/3942597083 conn(0x555f48a1f810 msgr2=0x555f48a2b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f92733ca740 1 -- 172.21.15.98:0/3942597083 shutdown_connections 2024-05-09T08:53:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f92733ca740 1 -- 172.21.15.98:0/3942597083 wait complete. 2024-05-09T08:53:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.254+0000 7f92733ca740 20 client.25843 trim_cache size 0 max 16384 2024-05-09T08:53:44.260 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:53:44.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.255+0000 7f926f37a640 1 -- 172.21.15.98:0/291133497 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f92500a2dc0 msgr2=0x7f92500a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.255+0000 7f926f37a640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f92500a2dc0 0x7f92500a5280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9260006fd0 tx=0x7f9260008040 comp rx=0 tx=0).stop 2024-05-09T08:53:44.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.255+0000 7f926f37a640 1 -- 172.21.15.98:0/291133497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92681038e0 msgr2=0x7f926810f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.255+0000 7f926f37a640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92681038e0 0x7f926810f5a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9258009ca0 tx=0x7f9258030fd0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.255+0000 7f926f37a640 1 -- 172.21.15.98:0/291133497 shutdown_connections 2024-05-09T08:53:44.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.255+0000 7f926f37a640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f92500a2dc0 0x7f92500a5280 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.255+0000 7f926f37a640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9268110160 0x7f9268114a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.255+0000 7f926f37a640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92681042a0 0x7f926810fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.255+0000 7f926f37a640 1 --2- 172.21.15.98:0/291133497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92681038e0 0x7f926810f5a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.255+0000 7f926f37a640 1 -- 172.21.15.98:0/291133497 >> 172.21.15.98:0/291133497 conn(0x7f92680ff370 msgr2=0x7f92680ffad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.255+0000 7f926f37a640 1 -- 172.21.15.98:0/291133497 shutdown_connections 2024-05-09T08:53:44.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.255+0000 7f926f37a640 1 -- 172.21.15.98:0/291133497 wait complete. 2024-05-09T08:53:44.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:53:44.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.561+0000 7f66b4f8b640 1 -- 172.21.15.98:0/259178876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66b010d440 msgr2=0x7f66b0115010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.561+0000 7f66b4f8b640 1 --2- 172.21.15.98:0/259178876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66b010d440 0x7f66b0115010 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f66a8008410 tx=0x7f66a802a160 comp rx=0 tx=0).stop 2024-05-09T08:53:44.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.568+0000 7f66b4f8b640 1 -- 172.21.15.98:0/259178876 shutdown_connections 2024-05-09T08:53:44.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.568+0000 7f66b4f8b640 1 --2- 172.21.15.98:0/259178876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66b010d440 0x7f66b0115010 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.568+0000 7f66b4f8b640 1 --2- 172.21.15.98:0/259178876 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66b0109450 0x7f66b010cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.568+0000 7f66b4f8b640 1 --2- 172.21.15.98:0/259178876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b0108aa0 0x7f66b0108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.568+0000 7f66b4f8b640 1 -- 172.21.15.98:0/259178876 >> 172.21.15.98:0/259178876 conn(0x7f66b006c7b0 msgr2=0x7f66b006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.573+0000 7f66b4f8b640 1 -- 172.21.15.98:0/259178876 shutdown_connections 2024-05-09T08:53:44.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.574+0000 7f66b4f8b640 1 -- 172.21.15.98:0/259178876 wait complete. 2024-05-09T08:53:44.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.575+0000 7f66b4f8b640 1 Processor -- start 2024-05-09T08:53:44.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.576+0000 7f66b4f8b640 1 -- start start 2024-05-09T08:53:44.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.577+0000 7f66b4f8b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66b0108aa0 0x7f66b01ac900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.577+0000 7f66b4f8b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66b010d440 0x7f66b01ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.577+0000 7f66b4f8b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b01a6ab0 0x7f66b01a6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.577+0000 7f66b4f8b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66b0118040 con 0x7f66b0108aa0 2024-05-09T08:53:44.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.577+0000 7f66b4f8b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f66b0117ec0 con 0x7f66b01a6ab0 2024-05-09T08:53:44.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.577+0000 7f66b4f8b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f66b01181c0 con 0x7f66b010d440 2024-05-09T08:53:44.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.578+0000 7f66af7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66b0108aa0 0x7f66b01ac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.578+0000 7f66af7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66b0108aa0 0x7f66b01ac900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44144/0 (socket says 172.21.15.98:44144) 2024-05-09T08:53:44.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.578+0000 7f66af7fe640 1 -- 172.21.15.98:0/608290590 learned_addr learned my addr 172.21.15.98:0/608290590 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.579+0000 7f66af7fe640 1 -- 172.21.15.98:0/608290590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66b010d440 msgr2=0x7f66b01ace40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:53:44.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.579+0000 7f66af7fe640 1 --2- 172.21.15.98:0/608290590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66b010d440 0x7f66b01ace40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.579+0000 7f66af7fe640 1 -- 172.21.15.98:0/608290590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b01a6ab0 msgr2=0x7f66b01a6f60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:53:44.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.579+0000 7f66af7fe640 1 --2- 172.21.15.98:0/608290590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b01a6ab0 0x7f66b01a6f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.579+0000 7f66af7fe640 1 -- 172.21.15.98:0/608290590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b01a75d0 con 0x7f66b0108aa0 2024-05-09T08:53:44.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.579+0000 7f66af7fe640 1 --2- 172.21.15.98:0/608290590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66b0108aa0 0x7f66b01ac900 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f66a000cce0 tx=0x7f66a000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.580+0000 7f66acff9640 1 -- 172.21.15.98:0/608290590 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a000f650 con 0x7f66b0108aa0 2024-05-09T08:53:44.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.580+0000 7f66acff9640 1 -- 172.21.15.98:0/608290590 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a000fcd0 con 0x7f66b0108aa0 2024-05-09T08:53:44.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.580+0000 7f66b4f8b640 1 -- 172.21.15.98:0/608290590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b01a78c0 con 0x7f66b0108aa0 2024-05-09T08:53:44.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.580+0000 7f66b4f8b640 1 -- 172.21.15.98:0/608290590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b019e520 con 0x7f66b0108aa0 2024-05-09T08:53:44.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.581+0000 7f66acff9640 1 -- 172.21.15.98:0/608290590 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a0018b90 con 0x7f66b0108aa0 2024-05-09T08:53:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.592+0000 7f66b8fcf740 1 Processor -- start 2024-05-09T08:53:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.592+0000 7f66b8fcf740 1 -- start start 2024-05-09T08:53:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.593+0000 7f66b8fcf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777520 0x55c214777900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.593+0000 7f66b8fcf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777e40 0x55c214781fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.593+0000 7f66b8fcf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214782510 0x55c214784900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.593+0000 7f66b8fcf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c2146dcc40 con 0x55c214777520 2024-05-09T08:53:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.594+0000 7f66b8fcf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c214684580 con 0x55c214782510 2024-05-09T08:53:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.594+0000 7f66b8fcf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c2146d8c00 con 0x55c214777e40 2024-05-09T08:53:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.594+0000 7f66acff9640 1 -- 172.21.15.98:0/608290590 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f66a000f7f0 con 0x7f66b0108aa0 2024-05-09T08:53:44.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.595+0000 7f66affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777520 0x55c214777900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.595+0000 7f66affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777520 0x55c214777900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44146/0 (socket says 172.21.15.98:44146) 2024-05-09T08:53:44.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.595+0000 7f66affff640 1 -- 172.21.15.98:0/2186054093 learned_addr learned my addr 172.21.15.98:0/2186054093 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.595+0000 7f66aeffd640 1 --2- 172.21.15.98:0/2186054093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214782510 0x55c214784900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.595+0000 7f66af7fe640 1 --2- 172.21.15.98:0/2186054093 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777e40 0x55c214781fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.596+0000 7f66acff9640 1 --2- 172.21.15.98:0/608290590 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f66980a2f00 0x7f66980a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.597+0000 7f66affff640 1 --2- 172.21.15.98:0/608290590 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f66980a2f00 0x7f66980a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.597+0000 7f66affff640 1 -- 172.21.15.98:0/2186054093 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777e40 msgr2=0x55c214781fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.597+0000 7f66affff640 1 --2- 172.21.15.98:0/2186054093 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777e40 0x55c214781fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.597+0000 7f66affff640 1 -- 172.21.15.98:0/2186054093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214782510 msgr2=0x55c214784900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.597+0000 7f66affff640 1 --2- 172.21.15.98:0/2186054093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214782510 0x55c214784900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.597+0000 7f66affff640 1 -- 172.21.15.98:0/2186054093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2143f03f0 con 0x55c214777520 2024-05-09T08:53:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.597+0000 7f66acff9640 1 -- 172.21.15.98:0/608290590 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f66a00cfc60 con 0x7f66b0108aa0 2024-05-09T08:53:44.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.599+0000 7f66affff640 1 --2- 172.21.15.98:0/2186054093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777520 0x55c214777900 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f66a8005d00 tx=0x7f66a8009dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9b0a9f79442675 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.600+0000 7f66affff640 1 --2- 172.21.15.98:0/608290590 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f66980a2f00 0x7f66980a53c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f66a800af90 tx=0x7f66a8042000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.601+0000 7f669dffb640 1 -- 172.21.15.98:0/2186054093 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8004040 con 0x55c214777520 2024-05-09T08:53:44.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.601+0000 7f669dffb640 1 -- 172.21.15.98:0/2186054093 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a80041e0 con 0x55c214777520 2024-05-09T08:53:44.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.601+0000 7f669dffb640 1 -- 172.21.15.98:0/2186054093 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8003c30 con 0x55c214777520 2024-05-09T08:53:44.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.602+0000 7f66b8fcf740 1 -- 172.21.15.98:0/2186054093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777520 msgr2=0x55c214777900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.602+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/2186054093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777520 0x55c214777900 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f66a8005d00 tx=0x7f66a8009dd0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.605+0000 7f66b8fcf740 1 -- 172.21.15.98:0/2186054093 shutdown_connections 2024-05-09T08:53:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.606+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/2186054093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214782510 0x55c214784900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.606+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/2186054093 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777e40 0x55c214781fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.606+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/2186054093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777520 0x55c214777900 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.606+0000 7f66b8fcf740 1 -- 172.21.15.98:0/2186054093 >> 172.21.15.98:0/2186054093 conn(0x55c214773840 msgr2=0x55c214773c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.607+0000 7f66b8fcf740 1 -- 172.21.15.98:0/2186054093 shutdown_connections 2024-05-09T08:53:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.607+0000 7f66b8fcf740 1 -- 172.21.15.98:0/2186054093 wait complete. 2024-05-09T08:53:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.607+0000 7f66b8fcf740 1 -- start start 2024-05-09T08:53:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.608+0000 7f66b8fcf740 1 Processor -- start 2024-05-09T08:53:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.609+0000 7f66b8fcf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777520 0x55c21477f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.609+0000 7f66b8fcf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777e40 0x55c21477f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.609+0000 7f66b8fcf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214782510 0x55c21477fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.609+0000 7f66b8fcf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c2147853a0 con 0x55c214777e40 2024-05-09T08:53:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.609+0000 7f66b8fcf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c214697c40 con 0x55c214782510 2024-05-09T08:53:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.609+0000 7f66b8fcf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c214785520 con 0x55c214777520 2024-05-09T08:53:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.609+0000 7f66b8fcf740 10 client.0 _ll_register_callbacks cb 0x55c2143efbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.609+0000 7f66aeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777e40 0x55c21477f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.609+0000 7f66aeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777e40 0x55c21477f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44158/0 (socket says 172.21.15.98:44158) 2024-05-09T08:53:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.610+0000 7f66aeffd640 1 -- 172.21.15.98:0/21947648 learned_addr learned my addr 172.21.15.98:0/21947648 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.610+0000 7f66af7fe640 1 --2- 172.21.15.98:0/21947648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777520 0x55c21477f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.610+0000 7f66aeffd640 1 -- 172.21.15.98:0/21947648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777520 msgr2=0x55c21477f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.610+0000 7f66aeffd640 1 --2- 172.21.15.98:0/21947648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777520 0x55c21477f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.610+0000 7f66aeffd640 1 -- 172.21.15.98:0/21947648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214782510 msgr2=0x55c21477fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.610+0000 7f66aeffd640 1 --2- 172.21.15.98:0/21947648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214782510 0x55c21477fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.610+0000 7f66aeffd640 1 -- 172.21.15.98:0/21947648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2144ba7d0 con 0x55c214777e40 2024-05-09T08:53:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.610+0000 7f66aeffd640 1 --2- 172.21.15.98:0/21947648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777e40 0x55c21477f8d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f66a400b330 tx=0x7f66a400b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a193c4cf528bba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.610+0000 7f6692ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:44.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.611+0000 7f6692ffd640 1 -- 172.21.15.98:0/21947648 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a40035f0 con 0x55c214777e40 2024-05-09T08:53:44.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.612+0000 7f66b8fcf740 1 -- 172.21.15.98:0/21947648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a80041e0 con 0x55c214777e40 2024-05-09T08:53:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.612+0000 7f6692ffd640 1 -- 172.21.15.98:0/21947648 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f66a4003790 con 0x55c214777e40 2024-05-09T08:53:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.613+0000 7f6692ffd640 1 -- 172.21.15.98:0/21947648 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f66a4024050 con 0x55c214777e40 2024-05-09T08:53:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.613+0000 7f66b8fcf740 10 client.16566 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.613+0000 7f66b8fcf740 1 -- 172.21.15.98:0/21947648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c2147915b0 con 0x55c214777e40 2024-05-09T08:53:44.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.614+0000 7f6692ffd640 1 -- 172.21.15.98:0/21947648 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a401b070 con 0x55c214777e40 2024-05-09T08:53:44.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.614+0000 7f6692ffd640 1 -- 172.21.15.98:0/21947648 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f66a40169a0 con 0x55c214777e40 2024-05-09T08:53:44.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.614+0000 7f66b8fcf740 10 client.16566 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:44.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.614+0000 7f66b8fcf740 10 client.16566 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:53:44.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.614+0000 7f66b8fcf740 20 client.16566 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.614+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/21947648 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c214792b10 0x55c214794f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.615+0000 7f66b8fcf740 4 client.16566 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.615+0000 7f66b8fcf740 1 -- 172.21.15.98:0/21947648 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f66a80041e0 con 0x55c214792b10 2024-05-09T08:53:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.616+0000 7f66af7fe640 1 --2- 172.21.15.98:0/21947648 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c214792b10 0x55c214794f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.618+0000 7f66af7fe640 1 --2- 172.21.15.98:0/21947648 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c214792b10 0x55c214794f50 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=27db79505485ad4e server_cookie=f38bc134ea47c446 in_seq=0 out_seq=0 2024-05-09T08:53:44.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.618+0000 7f6692ffd640 10 client.16566 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:53:44.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.620+0000 7f6692ffd640 1 -- 172.21.15.98:0/21947648 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f66a80041e0 con 0x55c214792b10 2024-05-09T08:53:44.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.621+0000 7f6692ffd640 10 client.16566 handle_command_reply: tid=0 2024-05-09T08:53:44.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.623+0000 7f66b8fcf740 1 client.16566 shutdown 2024-05-09T08:53:44.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.625+0000 7f66b8fcf740 1 -- 172.21.15.98:0/21947648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777e40 msgr2=0x55c21477f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.625+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/21947648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777e40 0x55c21477f8d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f66a400b330 tx=0x7f66a400b7f0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.626+0000 7f66b8fcf740 1 -- 172.21.15.98:0/21947648 shutdown_connections 2024-05-09T08:53:44.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.626+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/21947648 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c214792b10 0x55c214794f50 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.626+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/21947648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214782510 0x55c21477fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.626+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/21947648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777e40 0x55c21477f8d0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.626+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/21947648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777520 0x55c21477f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.626+0000 7f66b8fcf740 1 -- 172.21.15.98:0/21947648 >> 172.21.15.98:0/21947648 conn(0x55c214773840 msgr2=0x55c214779520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.626+0000 7f6692ffd640 0 client.16566 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:53:44.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.629+0000 7f66b8fcf740 1 -- 172.21.15.98:0/21947648 shutdown_connections 2024-05-09T08:53:44.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.629+0000 7f66b8fcf740 1 -- 172.21.15.98:0/21947648 wait complete. 2024-05-09T08:53:44.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.630+0000 7f66b8fcf740 20 client.16566 trim_cache size 0 max 16384 2024-05-09T08:53:44.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.667+0000 7f66b8fcf740 1 Processor -- start 2024-05-09T08:53:44.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.668+0000 7f66b8fcf740 1 -- start start 2024-05-09T08:53:44.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.668+0000 7f66b8fcf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777520 0x55c214781ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.668+0000 7f66b8fcf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214777e40 0x55c21478e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.669+0000 7f66b8fcf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214782510 0x55c214779c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.669+0000 7f66b8fcf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6694010090 con 0x55c214782510 2024-05-09T08:53:44.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.669+0000 7f66affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214782510 0x55c214779c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.669+0000 7f66affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214782510 0x55c214779c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44164/0 (socket says 172.21.15.98:44164) 2024-05-09T08:53:44.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.669+0000 7f66affff640 1 -- 172.21.15.98:0/3898958403 learned_addr learned my addr 172.21.15.98:0/3898958403 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.670+0000 7f66af7fe640 1 --2- 172.21.15.98:0/3898958403 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777520 0x55c214781ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.671+0000 7f66aeffd640 1 --2- 172.21.15.98:0/3898958403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214777e40 0x55c21478e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.671+0000 7f66b8fcf740 1 -- 172.21.15.98:0/3898958403 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6694011bd0 con 0x55c214777e40 2024-05-09T08:53:44.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.671+0000 7f66b8fcf740 1 -- 172.21.15.98:0/3898958403 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6694010760 con 0x55c214777520 2024-05-09T08:53:44.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.672+0000 7f66affff640 1 -- 172.21.15.98:0/3898958403 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777520 msgr2=0x55c214781ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.672+0000 7f66affff640 1 --2- 172.21.15.98:0/3898958403 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777520 0x55c214781ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.672+0000 7f66affff640 1 -- 172.21.15.98:0/3898958403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214777e40 msgr2=0x55c21478e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.672+0000 7f66affff640 1 --2- 172.21.15.98:0/3898958403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214777e40 0x55c21478e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.672+0000 7f66affff640 1 -- 172.21.15.98:0/3898958403 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c214784c70 con 0x55c214782510 2024-05-09T08:53:44.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.672+0000 7f66affff640 1 --2- 172.21.15.98:0/3898958403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214782510 0x55c214779c80 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f66a802a750 tx=0x7f66a80056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6701494aeee24295 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.672+0000 7f669d7fa640 1 -- 172.21.15.98:0/3898958403 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a80323f0 con 0x55c214782510 2024-05-09T08:53:44.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.672+0000 7f669d7fa640 1 -- 172.21.15.98:0/3898958403 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a8032590 con 0x55c214782510 2024-05-09T08:53:44.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.672+0000 7f669d7fa640 1 -- 172.21.15.98:0/3898958403 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8003cb0 con 0x55c214782510 2024-05-09T08:53:44.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.673+0000 7f66b8fcf740 1 -- 172.21.15.98:0/3898958403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214782510 msgr2=0x55c214779c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.673+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/3898958403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214782510 0x55c214779c80 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f66a802a750 tx=0x7f66a80056d0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.675+0000 7f66b8fcf740 1 -- 172.21.15.98:0/3898958403 shutdown_connections 2024-05-09T08:53:44.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.675+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/3898958403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214782510 0x55c214779c80 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.675+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/3898958403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214777e40 0x55c21478e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.675+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/3898958403 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214777520 0x55c214781ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.675+0000 7f66b8fcf740 1 -- 172.21.15.98:0/3898958403 >> 172.21.15.98:0/3898958403 conn(0x55c214773840 msgr2=0x55c21477e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.676+0000 7f66b8fcf740 1 -- 172.21.15.98:0/3898958403 shutdown_connections 2024-05-09T08:53:44.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.676+0000 7f66b8fcf740 1 -- 172.21.15.98:0/3898958403 wait complete. 2024-05-09T08:53:44.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.677+0000 7f66b8fcf740 1 -- start start 2024-05-09T08:53:44.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.678+0000 7f66b8fcf740 1 Processor -- start 2024-05-09T08:53:44.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.679+0000 7f66b8fcf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214777520 0x55c21477c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.679+0000 7f66b8fcf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777e40 0x55c21477c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.680+0000 7f66b8fcf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214782510 0x55c21477cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.680+0000 7f66b8fcf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6694011500 con 0x55c214777e40 2024-05-09T08:53:44.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.680+0000 7f66b8fcf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6694010e30 con 0x55c214777520 2024-05-09T08:53:44.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.680+0000 7f66b8fcf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f669400e070 con 0x55c214782510 2024-05-09T08:53:44.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.680+0000 7f66af7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214777520 0x55c21477c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.680+0000 7f66af7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214777520 0x55c21477c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55390/0 (socket says 172.21.15.98:55390) 2024-05-09T08:53:44.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.680+0000 7f66af7fe640 1 -- 172.21.15.98:0/647307229 learned_addr learned my addr 172.21.15.98:0/647307229 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.680+0000 7f66affff640 1 --2- 172.21.15.98:0/647307229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214782510 0x55c21477cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.681+0000 7f66b8fcf740 10 client.0 _ll_register_callbacks cb 0x7f66a400dca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:44.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.681+0000 7f66af7fe640 1 -- 172.21.15.98:0/647307229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214782510 msgr2=0x55c21477cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.681+0000 7f66af7fe640 1 --2- 172.21.15.98:0/647307229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214782510 0x55c21477cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.681+0000 7f66af7fe640 1 -- 172.21.15.98:0/647307229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777e40 msgr2=0x55c21477c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.681+0000 7f66af7fe640 1 --2- 172.21.15.98:0/647307229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777e40 0x55c21477c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.681+0000 7f66af7fe640 1 -- 172.21.15.98:0/647307229 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2147f8190 con 0x55c214777520 2024-05-09T08:53:44.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.682+0000 7f66af7fe640 1 --2- 172.21.15.98:0/647307229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214777520 0x55c21477c160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f66a009ba60 tx=0x7f66a0007940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9989fe9f7f31a1a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.682+0000 7f6692ffd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:53:44.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.688+0000 7f6692ffd640 1 -- 172.21.15.98:0/647307229 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a00b8070 con 0x55c214777520 2024-05-09T08:53:44.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.688+0000 7f6692ffd640 1 -- 172.21.15.98:0/647307229 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f66a00983d0 con 0x55c214777520 2024-05-09T08:53:44.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.688+0000 7f66b8fcf740 1 -- 172.21.15.98:0/647307229 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a8032590 con 0x55c214777520 2024-05-09T08:53:44.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.689+0000 7f6692ffd640 1 -- 172.21.15.98:0/647307229 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f66a00184b0 con 0x55c214777520 2024-05-09T08:53:44.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.689+0000 7f66b8fcf740 10 client.25895 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:44.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.689+0000 7f66b8fcf740 1 -- 172.21.15.98:0/647307229 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c2147f94d0 con 0x55c214777520 2024-05-09T08:53:44.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.690+0000 7f6692ffd640 1 -- 172.21.15.98:0/647307229 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a8032590 con 0x55c214777520 2024-05-09T08:53:44.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.690+0000 7f6692ffd640 1 -- 172.21.15.98:0/647307229 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f66a00b2040 con 0x55c214777520 2024-05-09T08:53:44.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.691+0000 7f66b8fcf740 10 client.25895 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:44.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.691+0000 7f66b8fcf740 10 client.25895 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:53:44.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.691+0000 7f66b8fcf740 20 client.25895 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:44.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.691+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/647307229 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c2147f9c20 0x55c2147fc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.691+0000 7f66b8fcf740 4 client.25895 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:53:44.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.691+0000 7f66b8fcf740 1 -- 172.21.15.98:0/647307229 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c2146ea370 con 0x55c2147f9c20 2024-05-09T08:53:44.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.691+0000 7f66aeffd640 1 --2- 172.21.15.98:0/647307229 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c2147f9c20 0x55c2147fc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.692+0000 7f66aeffd640 1 --2- 172.21.15.98:0/647307229 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c2147f9c20 0x55c2147fc040 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1b52e16bcf4fcae6 server_cookie=5c0c488364a9a587 in_seq=0 out_seq=0 2024-05-09T08:53:44.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.692+0000 7f6692ffd640 10 client.25895 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:53:44.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.692+0000 7f6692ffd640 1 -- 172.21.15.98:0/647307229 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c2146ea370 con 0x55c2147f9c20 2024-05-09T08:53:44.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.692+0000 7f6692ffd640 10 client.25895 handle_command_reply: tid=0 2024-05-09T08:53:44.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.692+0000 7f66b8fcf740 1 client.25895 shutdown 2024-05-09T08:53:44.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.692+0000 7f66b8fcf740 1 -- 172.21.15.98:0/647307229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214777520 msgr2=0x55c21477c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.692+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/647307229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214777520 0x55c21477c160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f66a009ba60 tx=0x7f66a0007940 comp rx=0 tx=0).stop 2024-05-09T08:53:44.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.692+0000 7f66b8fcf740 1 -- 172.21.15.98:0/647307229 shutdown_connections 2024-05-09T08:53:44.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.692+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/647307229 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c2147f9c20 0x55c2147fc040 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.692+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/647307229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c214782510 0x55c21477cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.692+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/647307229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c214777e40 0x55c21477c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.692+0000 7f66b8fcf740 1 --2- 172.21.15.98:0/647307229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c214777520 0x55c21477c160 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.692+0000 7f66b8fcf740 1 -- 172.21.15.98:0/647307229 >> 172.21.15.98:0/647307229 conn(0x55c214773840 msgr2=0x55c214772130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.692+0000 7f6692ffd640 0 client.25895 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:53:44.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.693+0000 7f66b8fcf740 1 -- 172.21.15.98:0/647307229 shutdown_connections 2024-05-09T08:53:44.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.693+0000 7f66b8fcf740 1 -- 172.21.15.98:0/647307229 wait complete. 2024-05-09T08:53:44.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.693+0000 7f66b8fcf740 20 client.25895 trim_cache size 0 max 16384 2024-05-09T08:53:44.694 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:53:44.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.694+0000 7f66b4f8b640 1 -- 172.21.15.98:0/608290590 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f66980a2f00 msgr2=0x7f66980a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.694+0000 7f66b4f8b640 1 --2- 172.21.15.98:0/608290590 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f66980a2f00 0x7f66980a53c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f66a800af90 tx=0x7f66a8042000 comp rx=0 tx=0).stop 2024-05-09T08:53:44.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.694+0000 7f66b4f8b640 1 -- 172.21.15.98:0/608290590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66b0108aa0 msgr2=0x7f66b01ac900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.694+0000 7f66b4f8b640 1 --2- 172.21.15.98:0/608290590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66b0108aa0 0x7f66b01ac900 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f66a000cce0 tx=0x7f66a000a620 comp rx=0 tx=0).stop 2024-05-09T08:53:44.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.694+0000 7f66b4f8b640 1 -- 172.21.15.98:0/608290590 shutdown_connections 2024-05-09T08:53:44.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.694+0000 7f66b4f8b640 1 --2- 172.21.15.98:0/608290590 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f66980a2f00 0x7f66980a53c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.694+0000 7f66b4f8b640 1 --2- 172.21.15.98:0/608290590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b01a6ab0 0x7f66b01a6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.694+0000 7f66b4f8b640 1 --2- 172.21.15.98:0/608290590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66b010d440 0x7f66b01ace40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.694+0000 7f66b4f8b640 1 --2- 172.21.15.98:0/608290590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66b0108aa0 0x7f66b01ac900 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.694+0000 7f66b4f8b640 1 -- 172.21.15.98:0/608290590 >> 172.21.15.98:0/608290590 conn(0x7f66b006c7b0 msgr2=0x7f66b00711c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.694+0000 7f66b4f8b640 1 -- 172.21.15.98:0/608290590 shutdown_connections 2024-05-09T08:53:44.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.694+0000 7f66b4f8b640 1 -- 172.21.15.98:0/608290590 wait complete. 2024-05-09T08:53:44.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:53:44.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.840+0000 7fe8c265d640 1 -- 172.21.15.98:0/308733566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8bc107890 msgr2=0x7fe8bc107c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.840+0000 7fe8c265d640 1 --2- 172.21.15.98:0/308733566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8bc107890 0x7fe8bc107c70 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4009930 tx=0x7fe8a402a440 comp rx=0 tx=0).stop 2024-05-09T08:53:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.840+0000 7fe8c265d640 1 -- 172.21.15.98:0/308733566 shutdown_connections 2024-05-09T08:53:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.840+0000 7fe8c265d640 1 --2- 172.21.15.98:0/308733566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8bc102240 0x7fe8bc10e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.840+0000 7fe8c265d640 1 --2- 172.21.15.98:0/308733566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8bc101880 0x7fe8bc101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.840+0000 7fe8c265d640 1 --2- 172.21.15.98:0/308733566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8bc107890 0x7fe8bc107c70 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.840+0000 7fe8c265d640 1 -- 172.21.15.98:0/308733566 >> 172.21.15.98:0/308733566 conn(0x7fe8bc0fd390 msgr2=0x7fe8bc0ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.841+0000 7fe8c265d640 1 -- 172.21.15.98:0/308733566 shutdown_connections 2024-05-09T08:53:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.841+0000 7fe8c265d640 1 -- 172.21.15.98:0/308733566 wait complete. 2024-05-09T08:53:44.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.842+0000 7fe8c265d640 1 Processor -- start 2024-05-09T08:53:44.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.842+0000 7fe8c265d640 1 -- start start 2024-05-09T08:53:44.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8c265d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8bc101880 0x7fe8bc198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8c265d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8bc102240 0x7fe8bc198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8c265d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8bc107890 0x7fe8bc19cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8c265d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc111cb0 con 0x7fe8bc101880 2024-05-09T08:53:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8c265d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc111b30 con 0x7fe8bc102240 2024-05-09T08:53:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8c265d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc111e30 con 0x7fe8bc107890 2024-05-09T08:53:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8bbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8bc101880 0x7fe8bc198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8b3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8bc102240 0x7fe8bc198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8bbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8bc101880 0x7fe8bc198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44192/0 (socket says 172.21.15.98:44192) 2024-05-09T08:53:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8b3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8bc102240 0x7fe8bc198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55414/0 (socket says 172.21.15.98:55414) 2024-05-09T08:53:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8bbfff640 1 -- 172.21.15.98:0/2359313393 learned_addr learned my addr 172.21.15.98:0/2359313393 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8c0bfa640 1 --2- 172.21.15.98:0/2359313393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8bc107890 0x7fe8bc19cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8c0bfa640 1 -- 172.21.15.98:0/2359313393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8bc102240 msgr2=0x7fe8bc198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8c0bfa640 1 --2- 172.21.15.98:0/2359313393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8bc102240 0x7fe8bc198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8c0bfa640 1 -- 172.21.15.98:0/2359313393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8bc101880 msgr2=0x7fe8bc198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8c0bfa640 1 --2- 172.21.15.98:0/2359313393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8bc101880 0x7fe8bc198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.843+0000 7fe8c0bfa640 1 -- 172.21.15.98:0/2359313393 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8bc19d1f0 con 0x7fe8bc107890 2024-05-09T08:53:44.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.844+0000 7fe8c0bfa640 1 --2- 172.21.15.98:0/2359313393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8bc107890 0x7fe8bc19cac0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009fd0 tx=0x7fe8b400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.844+0000 7fe8b9ffb640 1 -- 172.21.15.98:0/2359313393 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b400ce20 con 0x7fe8bc107890 2024-05-09T08:53:44.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.844+0000 7fe8c265d640 1 -- 172.21.15.98:0/2359313393 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8bc19d4e0 con 0x7fe8bc107890 2024-05-09T08:53:44.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.844+0000 7fe8c265d640 1 -- 172.21.15.98:0/2359313393 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8bc073a60 con 0x7fe8bc107890 2024-05-09T08:53:44.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.844+0000 7fe8b9ffb640 1 -- 172.21.15.98:0/2359313393 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8b4005020 con 0x7fe8bc107890 2024-05-09T08:53:44.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.844+0000 7fe8b9ffb640 1 -- 172.21.15.98:0/2359313393 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4010690 con 0x7fe8bc107890 2024-05-09T08:53:44.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.847+0000 7fe8b9ffb640 1 -- 172.21.15.98:0/2359313393 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe8b4010970 con 0x7fe8bc107890 2024-05-09T08:53:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.848+0000 7fe8c536a740 1 Processor -- start 2024-05-09T08:53:44.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.848+0000 7fe8b9ffb640 1 --2- 172.21.15.98:0/2359313393 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8a80a2d80 0x7fe8a80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.848+0000 7fe8c536a740 1 -- start start 2024-05-09T08:53:44.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.848+0000 7fe8c536a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 0x55d6ba8f8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.848+0000 7fe8c536a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8c50 0x55d6ba902de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.848+0000 7fe8b9ffb640 1 -- 172.21.15.98:0/2359313393 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fe8b40cfea0 con 0x7fe8bc107890 2024-05-09T08:53:44.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.848+0000 7fe8c536a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba903320 0x55d6ba905710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.848+0000 7fe8c536a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d6ba85dc40 con 0x55d6ba8f8c50 2024-05-09T08:53:44.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.848+0000 7fe8c536a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d6ba805580 con 0x55d6ba8f8330 2024-05-09T08:53:44.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.848+0000 7fe8c536a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d6ba859c00 con 0x55d6ba903320 2024-05-09T08:53:44.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.849+0000 7fe8b3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8c50 0x55d6ba902de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.849+0000 7fe8b3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8c50 0x55d6ba902de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44208/0 (socket says 172.21.15.98:44208) 2024-05-09T08:53:44.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.849+0000 7fe8b3fff640 1 -- 172.21.15.98:0/4109082313 learned_addr learned my addr 172.21.15.98:0/4109082313 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.849+0000 7fe8b3fff640 1 --2- 172.21.15.98:0/2359313393 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8a80a2d80 0x7fe8a80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.849+0000 7fe8c0bfa640 1 --2- 172.21.15.98:0/4109082313 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba903320 0x55d6ba905710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.850+0000 7fe8bbfff640 1 --2- 172.21.15.98:0/4109082313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 0x55d6ba8f8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.850+0000 7fe8b3fff640 1 -- 172.21.15.98:0/4109082313 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba903320 msgr2=0x55d6ba905710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.850+0000 7fe8b3fff640 1 --2- 172.21.15.98:0/4109082313 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba903320 0x55d6ba905710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.850+0000 7fe8b3fff640 1 -- 172.21.15.98:0/4109082313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 msgr2=0x55d6ba8f8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.850+0000 7fe8b3fff640 1 --2- 172.21.15.98:0/4109082313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 0x55d6ba8f8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.850+0000 7fe8b3fff640 1 -- 172.21.15.98:0/4109082313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6ba5713f0 con 0x55d6ba8f8c50 2024-05-09T08:53:44.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.850+0000 7fe8b3fff640 1 --2- 172.21.15.98:0/2359313393 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8a80a2d80 0x7fe8a80a5240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac0119d0 tx=0x7fe8ac009870 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.850+0000 7fe8b3fff640 1 --2- 172.21.15.98:0/4109082313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8c50 0x55d6ba902de0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00f790 tx=0x7fe8ac01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd79fb38d0d42cbc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.853+0000 7fe8b27fc640 1 -- 172.21.15.98:0/4109082313 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac02b070 con 0x55d6ba8f8c50 2024-05-09T08:53:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.853+0000 7fe8b27fc640 1 -- 172.21.15.98:0/4109082313 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8ac00ec30 con 0x55d6ba8f8c50 2024-05-09T08:53:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.854+0000 7fe8b27fc640 1 -- 172.21.15.98:0/4109082313 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac01f6b0 con 0x55d6ba8f8c50 2024-05-09T08:53:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.854+0000 7fe8c536a740 1 -- 172.21.15.98:0/4109082313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8c50 msgr2=0x55d6ba902de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.854+0000 7fe8c536a740 1 --2- 172.21.15.98:0/4109082313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8c50 0x55d6ba902de0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00f790 tx=0x7fe8ac01d000 comp rx=0 tx=0).stop 2024-05-09T08:53:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.854+0000 7fe8c536a740 1 -- 172.21.15.98:0/4109082313 shutdown_connections 2024-05-09T08:53:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.854+0000 7fe8c536a740 1 --2- 172.21.15.98:0/4109082313 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba903320 0x55d6ba905710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.854+0000 7fe8c536a740 1 --2- 172.21.15.98:0/4109082313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8c50 0x55d6ba902de0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.854+0000 7fe8c536a740 1 --2- 172.21.15.98:0/4109082313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 0x55d6ba8f8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.854+0000 7fe8c536a740 1 -- 172.21.15.98:0/4109082313 >> 172.21.15.98:0/4109082313 conn(0x55d6ba8f4650 msgr2=0x55d6ba8f4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.855+0000 7fe8c536a740 1 -- 172.21.15.98:0/4109082313 shutdown_connections 2024-05-09T08:53:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.855+0000 7fe8c536a740 1 -- 172.21.15.98:0/4109082313 wait complete. 2024-05-09T08:53:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.855+0000 7fe8c536a740 1 -- start start 2024-05-09T08:53:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.856+0000 7fe8c536a740 1 Processor -- start 2024-05-09T08:53:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.856+0000 7fe8c536a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8330 0x55d6ba900260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.857+0000 7fe8c536a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba8f8c50 0x55d6ba9007a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.857+0000 7fe8bbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8330 0x55d6ba900260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.857+0000 7fe8bbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8330 0x55d6ba900260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44210/0 (socket says 172.21.15.98:44210) 2024-05-09T08:53:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.857+0000 7fe8bbfff640 1 -- 172.21.15.98:0/3664031317 learned_addr learned my addr 172.21.15.98:0/3664031317 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.857+0000 7fe8c536a740 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba903320 0x55d6ba900ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.857+0000 7fe8c536a740 1 -- 172.21.15.98:0/3664031317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d6ba917110 con 0x55d6ba8f8330 2024-05-09T08:53:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.857+0000 7fe8c0bfa640 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba903320 0x55d6ba900ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.857+0000 7fe8b3fff640 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba8f8c50 0x55d6ba9007a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.857+0000 7fe8c536a740 1 -- 172.21.15.98:0/3664031317 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d6ba818c40 con 0x55d6ba903320 2024-05-09T08:53:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.857+0000 7fe8c536a740 1 -- 172.21.15.98:0/3664031317 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d6ba917290 con 0x55d6ba8f8c50 2024-05-09T08:53:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.858+0000 7fe8b3fff640 1 -- 172.21.15.98:0/3664031317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba903320 msgr2=0x55d6ba900ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.858+0000 7fe8b3fff640 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba903320 0x55d6ba900ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.858+0000 7fe8b3fff640 1 -- 172.21.15.98:0/3664031317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8330 msgr2=0x55d6ba900260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.858+0000 7fe8b3fff640 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8330 0x55d6ba900260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.858+0000 7fe8c536a740 10 client.0 _ll_register_callbacks cb 0x55d6ba570be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.858+0000 7fe8b3fff640 1 -- 172.21.15.98:0/3664031317 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6ba63b7d0 con 0x55d6ba8f8c50 2024-05-09T08:53:44.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.858+0000 7fe8c536a740 1 -- 172.21.15.98:0/3664031317 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8ac00ec30 con 0x55d6ba8f8c50 2024-05-09T08:53:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.858+0000 7fe8c0bfa640 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba903320 0x55d6ba900ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:53:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.859+0000 7fe8b3fff640 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba8f8c50 0x55d6ba9007a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00f790 tx=0x7fe8ac00ff10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fb0c450dd8ef70b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.859+0000 7fe89b7fe640 10 client.25864 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:53:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.859+0000 7fe89b7fe640 1 -- 172.21.15.98:0/3664031317 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac034060 con 0x55d6ba8f8c50 2024-05-09T08:53:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.859+0000 7fe89b7fe640 1 -- 172.21.15.98:0/3664031317 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fe8ac0050c0 con 0x55d6ba8f8c50 2024-05-09T08:53:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.859+0000 7fe89b7fe640 1 -- 172.21.15.98:0/3664031317 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe8ac02b070 con 0x55d6ba8f8c50 2024-05-09T08:53:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.859+0000 7fe8c536a740 10 client.25864 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.860+0000 7fe8c536a740 1 -- 172.21.15.98:0/3664031317 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55d6ba923310 con 0x55d6ba8f8c50 2024-05-09T08:53:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.860+0000 7fe89b7fe640 1 -- 172.21.15.98:0/3664031317 <== mon.1 v2:172.21.15.142:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe8ac037050 con 0x55d6ba8f8c50 2024-05-09T08:53:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.860+0000 7fe8c536a740 10 client.25864 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.860+0000 7fe8c536a740 10 client.25864 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:53:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.860+0000 7fe8c536a740 20 client.25864 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.860+0000 7fe8c536a740 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d6ba924660 0x55d6ba926a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.860+0000 7fe8c536a740 4 client.25864 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:44.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.860+0000 7fe8c536a740 1 -- 172.21.15.98:0/3664031317 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6ba86b370 con 0x55d6ba924660 2024-05-09T08:53:44.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.860+0000 7fe8bbfff640 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d6ba924660 0x55d6ba926a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.862+0000 7fe8bbfff640 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d6ba924660 0x55d6ba926a50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f61f61923403fa72 server_cookie=d0263968ba7077b4 in_seq=0 out_seq=0 2024-05-09T08:53:44.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.862+0000 7fe89b7fe640 10 client.25864 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:53:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.863+0000 7fe89b7fe640 1 -- 172.21.15.98:0/3664031317 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d6ba86b370 con 0x55d6ba924660 2024-05-09T08:53:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.863+0000 7fe89b7fe640 10 client.25864 handle_command_reply: tid=0 2024-05-09T08:53:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.863+0000 7fe8c536a740 1 client.25864 shutdown 2024-05-09T08:53:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.863+0000 7fe8c536a740 1 -- 172.21.15.98:0/3664031317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba8f8c50 msgr2=0x55d6ba9007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.863+0000 7fe8c536a740 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba8f8c50 0x55d6ba9007a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00f790 tx=0x7fe8ac00ff10 comp rx=0 tx=0).stop 2024-05-09T08:53:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.863+0000 7fe8c536a740 1 -- 172.21.15.98:0/3664031317 shutdown_connections 2024-05-09T08:53:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.863+0000 7fe8c536a740 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d6ba924660 0x55d6ba926a50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.863+0000 7fe8c536a740 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba903320 0x55d6ba900ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.863+0000 7fe8c536a740 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba8f8c50 0x55d6ba9007a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.863+0000 7fe8c536a740 1 --2- 172.21.15.98:0/3664031317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8330 0x55d6ba900260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.863+0000 7fe8c536a740 1 -- 172.21.15.98:0/3664031317 >> 172.21.15.98:0/3664031317 conn(0x55d6ba8f4650 msgr2=0x55d6ba8fa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.864+0000 7fe89b7fe640 0 client.25864 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:53:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.864+0000 7fe8c536a740 1 -- 172.21.15.98:0/3664031317 shutdown_connections 2024-05-09T08:53:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.864+0000 7fe8c536a740 1 -- 172.21.15.98:0/3664031317 wait complete. 2024-05-09T08:53:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.866+0000 7fe8c536a740 20 client.25864 trim_cache size 0 max 16384 2024-05-09T08:53:44.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.893+0000 7fe8c536a740 1 Processor -- start 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.894+0000 7fe8c536a740 1 -- start start 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.895+0000 7fe8c536a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 0x55d6ba902990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.895+0000 7fe8c536a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba8f8c50 0x55d6ba8faa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.895+0000 7fe8c536a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba903320 0x55d6ba900b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.895+0000 7fe8c536a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c010020 con 0x55d6ba903320 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.895+0000 7fe8c536a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c011b60 con 0x55d6ba8f8330 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.895+0000 7fe8c536a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c0106f0 con 0x55d6ba8f8c50 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.895+0000 7fe8c0bfa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba903320 0x55d6ba900b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.895+0000 7fe8c0bfa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba903320 0x55d6ba900b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44218/0 (socket says 172.21.15.98:44218) 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.895+0000 7fe8c0bfa640 1 -- 172.21.15.98:0/1127943606 learned_addr learned my addr 172.21.15.98:0/1127943606 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.896+0000 7fe8c0bfa640 1 -- 172.21.15.98:0/1127943606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba8f8c50 msgr2=0x55d6ba8faa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.896+0000 7fe8c0bfa640 1 --2- 172.21.15.98:0/1127943606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba8f8c50 0x55d6ba8faa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.896+0000 7fe8c0bfa640 1 -- 172.21.15.98:0/1127943606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 msgr2=0x55d6ba902990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.896+0000 7fe8c0bfa640 1 --2- 172.21.15.98:0/1127943606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 0x55d6ba902990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.896+0000 7fe8c0bfa640 1 -- 172.21.15.98:0/1127943606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6ba917000 con 0x55d6ba903320 2024-05-09T08:53:44.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.896+0000 7fe8c0bfa640 1 --2- 172.21.15.98:0/1127943606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba903320 0x55d6ba900b00 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe8b409cf60 tx=0x7fe8b4009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7d8991a3b29bd4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.897+0000 7fe8b1ffb640 1 -- 172.21.15.98:0/1127943606 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4005540 con 0x55d6ba903320 2024-05-09T08:53:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.897+0000 7fe8b1ffb640 1 -- 172.21.15.98:0/1127943606 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8b4099640 con 0x55d6ba903320 2024-05-09T08:53:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.897+0000 7fe8b1ffb640 1 -- 172.21.15.98:0/1127943606 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b409c680 con 0x55d6ba903320 2024-05-09T08:53:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.898+0000 7fe8c536a740 1 -- 172.21.15.98:0/1127943606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba903320 msgr2=0x55d6ba900b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.898+0000 7fe8c536a740 1 --2- 172.21.15.98:0/1127943606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba903320 0x55d6ba900b00 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe8b409cf60 tx=0x7fe8b4009590 comp rx=0 tx=0).stop 2024-05-09T08:53:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.898+0000 7fe8c536a740 1 -- 172.21.15.98:0/1127943606 shutdown_connections 2024-05-09T08:53:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.898+0000 7fe8c536a740 1 --2- 172.21.15.98:0/1127943606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba903320 0x55d6ba900b00 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.898+0000 7fe8c536a740 1 --2- 172.21.15.98:0/1127943606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba8f8c50 0x55d6ba8faa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.899+0000 7fe8c536a740 1 --2- 172.21.15.98:0/1127943606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 0x55d6ba902990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.899+0000 7fe8c536a740 1 -- 172.21.15.98:0/1127943606 >> 172.21.15.98:0/1127943606 conn(0x55d6ba8f4650 msgr2=0x55d6ba8f37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.899+0000 7fe8c536a740 1 -- 172.21.15.98:0/1127943606 shutdown_connections 2024-05-09T08:53:44.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.899+0000 7fe8c536a740 1 -- 172.21.15.98:0/1127943606 wait complete. 2024-05-09T08:53:44.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.899+0000 7fe8c536a740 1 -- start start 2024-05-09T08:53:44.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.900+0000 7fe8c536a740 1 Processor -- start 2024-05-09T08:53:44.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.900+0000 7fe8c536a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 0x55d6ba91c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.900+0000 7fe8c536a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8c50 0x55d6ba8fca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.900+0000 7fe8c536a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba903320 0x55d6ba8fcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.900+0000 7fe8c536a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c011490 con 0x55d6ba8f8c50 2024-05-09T08:53:44.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.900+0000 7fe8c536a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c010dc0 con 0x55d6ba8f8330 2024-05-09T08:53:44.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.900+0000 7fe8c536a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c00e000 con 0x55d6ba903320 2024-05-09T08:53:44.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.900+0000 7fe8c536a740 10 client.0 _ll_register_callbacks cb 0x7fe8ac029070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.900+0000 7fe8b3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8c50 0x55d6ba8fca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.900+0000 7fe8b3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8c50 0x55d6ba8fca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44230/0 (socket says 172.21.15.98:44230) 2024-05-09T08:53:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.900+0000 7fe8b3fff640 1 -- 172.21.15.98:0/1846418872 learned_addr learned my addr 172.21.15.98:0/1846418872 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.901+0000 7fe8bbfff640 1 --2- 172.21.15.98:0/1846418872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 0x55d6ba91c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.901+0000 7fe8b3fff640 1 -- 172.21.15.98:0/1846418872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba903320 msgr2=0x55d6ba8fcfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.901+0000 7fe8b3fff640 1 --2- 172.21.15.98:0/1846418872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba903320 0x55d6ba8fcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.901+0000 7fe8b3fff640 1 -- 172.21.15.98:0/1846418872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 msgr2=0x55d6ba91c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.901+0000 7fe8b3fff640 1 --2- 172.21.15.98:0/1846418872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 0x55d6ba91c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.901+0000 7fe8b3fff640 1 -- 172.21.15.98:0/1846418872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6ba8fd590 con 0x55d6ba8f8c50 2024-05-09T08:53:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.901+0000 7fe8bbfff640 1 --2- 172.21.15.98:0/1846418872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 0x55d6ba91c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:53:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.901+0000 7fe8b3fff640 1 --2- 172.21.15.98:0/1846418872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8c50 0x55d6ba8fca90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x55d6ba901830 tx=0x7fe8ac01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e925482078b33c2e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.901+0000 7fe89b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:44.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.901+0000 7fe89b7fe640 1 -- 172.21.15.98:0/1846418872 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac011120 con 0x55d6ba8f8c50 2024-05-09T08:53:44.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.901+0000 7fe89b7fe640 1 -- 172.21.15.98:0/1846418872 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fe8ac005510 con 0x55d6ba8f8c50 2024-05-09T08:53:44.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.902+0000 7fe8c536a740 1 -- 172.21.15.98:0/1846418872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8b4099640 con 0x55d6ba8f8c50 2024-05-09T08:53:44.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.903+0000 7fe89b7fe640 1 -- 172.21.15.98:0/1846418872 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe8ac02e3b0 con 0x55d6ba8f8c50 2024-05-09T08:53:44.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.903+0000 7fe8c536a740 10 client.16587 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:44.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.903+0000 7fe8c536a740 1 -- 172.21.15.98:0/1846418872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d6ba98a760 con 0x55d6ba8f8c50 2024-05-09T08:53:44.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.904+0000 7fe89b7fe640 1 -- 172.21.15.98:0/1846418872 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8ac00ec30 con 0x55d6ba8f8c50 2024-05-09T08:53:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.905+0000 7fe89b7fe640 1 -- 172.21.15.98:0/1846418872 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe8ac037040 con 0x55d6ba8f8c50 2024-05-09T08:53:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.905+0000 7fe8c536a740 10 client.16587 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.905+0000 7fe8c536a740 10 client.16587 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:53:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.905+0000 7fe8c536a740 20 client.16587 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.905+0000 7fe8c536a740 1 --2- 172.21.15.98:0/1846418872 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d6ba98aeb0 0x55d6ba98d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.905+0000 7fe8c536a740 4 client.16587 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:53:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.905+0000 7fe8c536a740 1 -- 172.21.15.98:0/1846418872 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fe8b4099640 con 0x55d6ba98aeb0 2024-05-09T08:53:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.905+0000 7fe8bbfff640 1 --2- 172.21.15.98:0/1846418872 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d6ba98aeb0 0x55d6ba98d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:44.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe89b7fe640 10 client.16587 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:53:44.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe8bbfff640 1 --2- 172.21.15.98:0/1846418872 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d6ba98aeb0 0x55d6ba98d2a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=33cc904326f4b4cc server_cookie=598a1a9c8d8b292e in_seq=0 out_seq=0 2024-05-09T08:53:44.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe89b7fe640 1 -- 172.21.15.98:0/1846418872 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fe8b4099640 con 0x55d6ba98aeb0 2024-05-09T08:53:44.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe89b7fe640 10 client.16587 handle_command_reply: tid=0 2024-05-09T08:53:44.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe8c536a740 1 client.16587 shutdown 2024-05-09T08:53:44.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe8c536a740 1 -- 172.21.15.98:0/1846418872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8c50 msgr2=0x55d6ba8fca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe8c536a740 1 --2- 172.21.15.98:0/1846418872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8c50 0x55d6ba8fca90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x55d6ba901830 tx=0x7fe8ac01d000 comp rx=0 tx=0).stop 2024-05-09T08:53:44.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe8c536a740 1 -- 172.21.15.98:0/1846418872 shutdown_connections 2024-05-09T08:53:44.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe8c536a740 1 --2- 172.21.15.98:0/1846418872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6ba903320 0x55d6ba8fcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe8c536a740 1 --2- 172.21.15.98:0/1846418872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d6ba8f8c50 0x55d6ba8fca90 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe8c536a740 1 --2- 172.21.15.98:0/1846418872 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d6ba98aeb0 0x55d6ba98d2a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe89b7fe640 0 client.16587 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:53:44.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe8c536a740 1 --2- 172.21.15.98:0/1846418872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d6ba8f8330 0x55d6ba91c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe8c536a740 1 -- 172.21.15.98:0/1846418872 >> 172.21.15.98:0/1846418872 conn(0x55d6ba8f4650 msgr2=0x55d6ba91f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe8c536a740 1 -- 172.21.15.98:0/1846418872 shutdown_connections 2024-05-09T08:53:44.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.907+0000 7fe8c536a740 1 -- 172.21.15.98:0/1846418872 wait complete. 2024-05-09T08:53:44.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.908+0000 7fe8c536a740 20 client.16587 trim_cache size 0 max 16384 2024-05-09T08:53:44.908 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:53:44.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.908+0000 7fe8c265d640 1 -- 172.21.15.98:0/2359313393 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8a80a2d80 msgr2=0x7fe8a80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.908+0000 7fe8c265d640 1 --2- 172.21.15.98:0/2359313393 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8a80a2d80 0x7fe8a80a5240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac0119d0 tx=0x7fe8ac009870 comp rx=0 tx=0).stop 2024-05-09T08:53:44.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.908+0000 7fe8c265d640 1 -- 172.21.15.98:0/2359313393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8bc107890 msgr2=0x7fe8bc19cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:44.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.908+0000 7fe8c265d640 1 --2- 172.21.15.98:0/2359313393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8bc107890 0x7fe8bc19cac0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009fd0 tx=0x7fe8b400ef10 comp rx=0 tx=0).stop 2024-05-09T08:53:44.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.908+0000 7fe8c265d640 1 -- 172.21.15.98:0/2359313393 shutdown_connections 2024-05-09T08:53:44.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.908+0000 7fe8c265d640 1 --2- 172.21.15.98:0/2359313393 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8a80a2d80 0x7fe8a80a5240 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.908+0000 7fe8c265d640 1 --2- 172.21.15.98:0/2359313393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8bc107890 0x7fe8bc19cac0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.908+0000 7fe8c265d640 1 --2- 172.21.15.98:0/2359313393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8bc102240 0x7fe8bc198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.908+0000 7fe8c265d640 1 --2- 172.21.15.98:0/2359313393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8bc101880 0x7fe8bc198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:44.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.908+0000 7fe8c265d640 1 -- 172.21.15.98:0/2359313393 >> 172.21.15.98:0/2359313393 conn(0x7fe8bc0fd390 msgr2=0x7fe8bc0fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:44.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.908+0000 7fe8c265d640 1 -- 172.21.15.98:0/2359313393 shutdown_connections 2024-05-09T08:53:44.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:44.908+0000 7fe8c265d640 1 -- 172.21.15.98:0/2359313393 wait complete. 2024-05-09T08:53:44.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:53:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.170+0000 7fb84d6ed640 1 -- 172.21.15.98:0/3807685709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8481092b0 msgr2=0x7fb84811d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.170+0000 7fb84d6ed640 1 --2- 172.21.15.98:0/3807685709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8481092b0 0x7fb84811d020 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb83c009860 tx=0x7fb83c02a410 comp rx=0 tx=0).stop 2024-05-09T08:53:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.170+0000 7fb84d6ed640 1 -- 172.21.15.98:0/3807685709 shutdown_connections 2024-05-09T08:53:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.170+0000 7fb84d6ed640 1 --2- 172.21.15.98:0/3807685709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8481092b0 0x7fb84811d020 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.170+0000 7fb84d6ed640 1 --2- 172.21.15.98:0/3807685709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb848108830 0x7fb848108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.170+0000 7fb84d6ed640 1 --2- 172.21.15.98:0/3807685709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb84810fed0 0x7fb8481102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.170+0000 7fb84d6ed640 1 -- 172.21.15.98:0/3807685709 >> 172.21.15.98:0/3807685709 conn(0x7fb84806c7b0 msgr2=0x7fb84806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:45.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.175+0000 7fb84d6ed640 1 -- 172.21.15.98:0/3807685709 shutdown_connections 2024-05-09T08:53:45.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.176+0000 7fb84d6ed640 1 -- 172.21.15.98:0/3807685709 wait complete. 2024-05-09T08:53:45.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.178+0000 7fb84d6ed640 1 Processor -- start 2024-05-09T08:53:45.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.178+0000 7fb84d6ed640 1 -- start start 2024-05-09T08:53:45.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.179+0000 7fb84d6ed640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb848108830 0x7fb8481a6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.179+0000 7fb84d6ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8481092b0 0x7fb8481a7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.179+0000 7fb84d6ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb84810fed0 0x7fb8481ab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.179+0000 7fb84d6ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb848120900 con 0x7fb84810fed0 2024-05-09T08:53:45.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.179+0000 7fb84d6ed640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb848120780 con 0x7fb8481092b0 2024-05-09T08:53:45.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.179+0000 7fb84d6ed640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb848120a80 con 0x7fb848108830 2024-05-09T08:53:45.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.181+0000 7fb8477fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8481092b0 0x7fb8481a7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:45.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.185+0000 7fb8477fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8481092b0 0x7fb8481a7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55488/0 (socket says 172.21.15.98:55488) 2024-05-09T08:53:45.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.185+0000 7fb8477fe640 1 -- 172.21.15.98:0/1454128346 learned_addr learned my addr 172.21.15.98:0/1454128346 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:45.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.185+0000 7fb84ceec640 1 --2- 172.21.15.98:0/1454128346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb84810fed0 0x7fb8481ab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:45.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.185+0000 7fb847fff640 1 --2- 172.21.15.98:0/1454128346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb848108830 0x7fb8481a6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.192+0000 7fb84ceec640 1 -- 172.21.15.98:0/1454128346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb848108830 msgr2=0x7fb8481a6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.192+0000 7fb84ceec640 1 --2- 172.21.15.98:0/1454128346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb848108830 0x7fb8481a6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.192+0000 7fb84ceec640 1 -- 172.21.15.98:0/1454128346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8481092b0 msgr2=0x7fb8481a7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.192+0000 7fb84ceec640 1 --2- 172.21.15.98:0/1454128346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8481092b0 0x7fb8481a7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.192+0000 7fb84ceec640 1 -- 172.21.15.98:0/1454128346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8481abe10 con 0x7fb84810fed0 2024-05-09T08:53:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.194+0000 7fb84ceec640 1 --2- 172.21.15.98:0/1454128346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb84810fed0 0x7fb8481ab6e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb83c009780 tx=0x7fb83c0309b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.195+0000 7fb8457fa640 1 -- 172.21.15.98:0/1454128346 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c005590 con 0x7fb84810fed0 2024-05-09T08:53:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.195+0000 7fb8457fa640 1 -- 172.21.15.98:0/1454128346 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb83c0318e0 con 0x7fb84810fed0 2024-05-09T08:53:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.195+0000 7fb8457fa640 1 -- 172.21.15.98:0/1454128346 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c03c420 con 0x7fb84810fed0 2024-05-09T08:53:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.196+0000 7fb84d6ed640 1 -- 172.21.15.98:0/1454128346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8481ac0a0 con 0x7fb84810fed0 2024-05-09T08:53:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.196+0000 7fb84d6ed640 1 -- 172.21.15.98:0/1454128346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8481ac510 con 0x7fb84810fed0 2024-05-09T08:53:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.203+0000 7fb8457fa640 1 -- 172.21.15.98:0/1454128346 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb83c039040 con 0x7fb84810fed0 2024-05-09T08:53:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.205+0000 7fb8457fa640 1 --2- 172.21.15.98:0/1454128346 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8300a2f00 0x7fb8300a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.206+0000 7fb8457fa640 1 -- 172.21.15.98:0/1454128346 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fb83c0f0440 con 0x7fb84810fed0 2024-05-09T08:53:45.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.208+0000 7fb851656740 1 Processor -- start 2024-05-09T08:53:45.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.208+0000 7fb851656740 1 -- start start 2024-05-09T08:53:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.208+0000 7fb851656740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca0b4f0 0x55b65ca0b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.209+0000 7fb851656740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca0be10 0x55b65ca15fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.209+0000 7fb851656740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca164e0 0x55b65ca188d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.209+0000 7fb851656740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b65c970c40 con 0x55b65ca0b4f0 2024-05-09T08:53:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.209+0000 7fb851656740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b65c918580 con 0x55b65ca0be10 2024-05-09T08:53:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.209+0000 7fb851656740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b65c96cc00 con 0x55b65ca164e0 2024-05-09T08:53:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.209+0000 7fb8477fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca0be10 0x55b65ca15fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.209+0000 7fb8477fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca0be10 0x55b65ca15fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55492/0 (socket says 172.21.15.98:55492) 2024-05-09T08:53:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.209+0000 7fb8477fe640 1 -- 172.21.15.98:0/245341327 learned_addr learned my addr 172.21.15.98:0/245341327 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.210+0000 7fb8477fe640 1 -- 172.21.15.98:0/245341327 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca164e0 msgr2=0x55b65ca188d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:53:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.210+0000 7fb8477fe640 1 --2- 172.21.15.98:0/245341327 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca164e0 0x55b65ca188d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.210+0000 7fb8477fe640 1 -- 172.21.15.98:0/245341327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca0b4f0 msgr2=0x55b65ca0b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:45.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.210+0000 7fb8477fe640 1 --2- 172.21.15.98:0/245341327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca0b4f0 0x55b65ca0b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.210+0000 7fb8477fe640 1 -- 172.21.15.98:0/245341327 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b65c6843f0 con 0x55b65ca0be10 2024-05-09T08:53:45.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.213+0000 7fb847fff640 1 --2- 172.21.15.98:0/1454128346 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8300a2f00 0x7fb8300a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:45.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.217+0000 7fb8477fe640 1 --2- 172.21.15.98:0/245341327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca0be10 0x55b65ca15fa0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb84000e270 tx=0x7fb840009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfe0e26a4ae1686b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:45.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.221+0000 7fb82e7fc640 1 -- 172.21.15.98:0/245341327 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb840018020 con 0x55b65ca0be10 2024-05-09T08:53:45.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.221+0000 7fb82e7fc640 1 -- 172.21.15.98:0/245341327 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb840014d70 con 0x55b65ca0be10 2024-05-09T08:53:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.222+0000 7fb851656740 1 -- 172.21.15.98:0/245341327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca0be10 msgr2=0x55b65ca15fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.222+0000 7fb851656740 1 --2- 172.21.15.98:0/245341327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca0be10 0x55b65ca15fa0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb84000e270 tx=0x7fb840009790 comp rx=0 tx=0).stop 2024-05-09T08:53:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.223+0000 7fb851656740 1 -- 172.21.15.98:0/245341327 shutdown_connections 2024-05-09T08:53:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.223+0000 7fb851656740 1 --2- 172.21.15.98:0/245341327 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca164e0 0x55b65ca188d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.223+0000 7fb851656740 1 --2- 172.21.15.98:0/245341327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca0be10 0x55b65ca15fa0 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb84000e270 tx=0x7fb840009790 comp rx=0 tx=0).stop 2024-05-09T08:53:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.223+0000 7fb851656740 1 --2- 172.21.15.98:0/245341327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca0b4f0 0x55b65ca0b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.223+0000 7fb851656740 1 -- 172.21.15.98:0/245341327 >> 172.21.15.98:0/245341327 conn(0x55b65ca07810 msgr2=0x55b65ca07bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.225+0000 7fb847fff640 1 --2- 172.21.15.98:0/1454128346 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8300a2f00 0x7fb8300a53c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb8380058c0 tx=0x7fb838005850 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:45.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.229+0000 7fb851656740 1 -- 172.21.15.98:0/245341327 shutdown_connections 2024-05-09T08:53:45.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.229+0000 7fb851656740 1 -- 172.21.15.98:0/245341327 wait complete. 2024-05-09T08:53:45.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.230+0000 7fb851656740 1 -- start start 2024-05-09T08:53:45.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.230+0000 7fb851656740 1 Processor -- start 2024-05-09T08:53:45.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.231+0000 7fb851656740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca0b4f0 0x55b65ca134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.231+0000 7fb851656740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca164e0 0x55b65ca13a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.231+0000 7fb851656740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca13f70 0x55b65ca22810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.231+0000 7fb851656740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b65ca19320 con 0x55b65ca0b4f0 2024-05-09T08:53:45.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.231+0000 7fb851656740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b65c92bc40 con 0x55b65ca164e0 2024-05-09T08:53:45.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.231+0000 7fb851656740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b65ca194a0 con 0x55b65ca13f70 2024-05-09T08:53:45.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.231+0000 7fb851656740 10 client.0 _ll_register_callbacks cb 0x55b65c683be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:45.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.232+0000 7fb84ceec640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca13f70 0x55b65ca22810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:45.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.232+0000 7fb84ceec640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca13f70 0x55b65ca22810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:57390/0 (socket says 172.21.15.98:57390) 2024-05-09T08:53:45.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.232+0000 7fb84ceec640 1 -- 172.21.15.98:0/1460260291 learned_addr learned my addr 172.21.15.98:0/1460260291 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:45.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.233+0000 7fb84ceec640 1 -- 172.21.15.98:0/1460260291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca164e0 msgr2=0x55b65ca13a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:45.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.233+0000 7fb84ceec640 1 --2- 172.21.15.98:0/1460260291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca164e0 0x55b65ca13a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.233+0000 7fb84ceec640 1 -- 172.21.15.98:0/1460260291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca0b4f0 msgr2=0x55b65ca134f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:45.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.233+0000 7fb84ceec640 1 --2- 172.21.15.98:0/1460260291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca0b4f0 0x55b65ca134f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.233+0000 7fb84ceec640 1 -- 172.21.15.98:0/1460260291 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b65c74e7d0 con 0x55b65ca13f70 2024-05-09T08:53:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.233+0000 7fb84ceec640 1 --2- 172.21.15.98:0/1460260291 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca13f70 0x55b65ca22810 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb83c031000 tx=0x7fb83c009460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=450b52d840abc7db server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.237+0000 7fb8277fe640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:53:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.237+0000 7fb8277fe640 1 -- 172.21.15.98:0/1460260291 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c0bc540 con 0x55b65ca13f70 2024-05-09T08:53:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.237+0000 7fb851656740 1 -- 172.21.15.98:0/1460260291 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb840014d70 con 0x55b65ca13f70 2024-05-09T08:53:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.237+0000 7fb8277fe640 1 -- 172.21.15.98:0/1460260291 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fb83c0046c0 con 0x55b65ca13f70 2024-05-09T08:53:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.240+0000 7fb8277fe640 1 -- 172.21.15.98:0/1460260291 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb83c0c08f0 con 0x55b65ca13f70 2024-05-09T08:53:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.240+0000 7fb851656740 10 client.25870 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.240+0000 7fb851656740 1 -- 172.21.15.98:0/1460260291 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b65ca19370 con 0x55b65ca13f70 2024-05-09T08:53:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.244+0000 7fb8277fe640 1 -- 172.21.15.98:0/1460260291 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb840014d70 con 0x55b65ca13f70 2024-05-09T08:53:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.244+0000 7fb8277fe640 1 -- 172.21.15.98:0/1460260291 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb83c04b020 con 0x55b65ca13f70 2024-05-09T08:53:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.244+0000 7fb851656740 10 client.25870 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.244+0000 7fb851656740 10 client.25870 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.244+0000 7fb851656740 20 client.25870 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.244+0000 7fb851656740 1 --2- 172.21.15.98:0/1460260291 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b65ca271d0 0x55b65ca29620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.244+0000 7fb851656740 4 client.25870 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.244+0000 7fb851656740 1 -- 172.21.15.98:0/1460260291 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb840013e50 con 0x55b65ca271d0 2024-05-09T08:53:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.244+0000 7fb847fff640 1 --2- 172.21.15.98:0/1460260291 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b65ca271d0 0x55b65ca29620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.245+0000 7fb847fff640 1 --2- 172.21.15.98:0/1460260291 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b65ca271d0 0x55b65ca29620 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90f58b410d3ee406 server_cookie=b0d1d6e37da14ad3 in_seq=0 out_seq=0 2024-05-09T08:53:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.248+0000 7fb8277fe640 10 client.25870 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.248+0000 7fb8277fe640 1 -- 172.21.15.98:0/1460260291 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fb840013e50 con 0x55b65ca271d0 2024-05-09T08:53:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.248+0000 7fb8277fe640 10 client.25870 handle_command_reply: tid=0 2024-05-09T08:53:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.248+0000 7fb851656740 1 client.25870 shutdown 2024-05-09T08:53:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.250+0000 7fb851656740 1 -- 172.21.15.98:0/1460260291 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca13f70 msgr2=0x55b65ca22810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.250+0000 7fb851656740 1 --2- 172.21.15.98:0/1460260291 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca13f70 0x55b65ca22810 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb83c031000 tx=0x7fb83c009460 comp rx=0 tx=0).stop 2024-05-09T08:53:45.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.255+0000 7fb851656740 1 -- 172.21.15.98:0/1460260291 shutdown_connections 2024-05-09T08:53:45.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.255+0000 7fb851656740 1 --2- 172.21.15.98:0/1460260291 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b65ca271d0 0x55b65ca29620 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.255+0000 7fb851656740 1 --2- 172.21.15.98:0/1460260291 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca13f70 0x55b65ca22810 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.255+0000 7fb851656740 1 --2- 172.21.15.98:0/1460260291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca164e0 0x55b65ca13a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.255+0000 7fb851656740 1 --2- 172.21.15.98:0/1460260291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca0b4f0 0x55b65ca134f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.255+0000 7fb851656740 1 -- 172.21.15.98:0/1460260291 >> 172.21.15.98:0/1460260291 conn(0x55b65ca07810 msgr2=0x55b65ca0d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:45.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.255+0000 7fb8277fe640 0 client.25870 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:45.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.255+0000 7fb851656740 1 -- 172.21.15.98:0/1460260291 shutdown_connections 2024-05-09T08:53:45.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.255+0000 7fb851656740 1 -- 172.21.15.98:0/1460260291 wait complete. 2024-05-09T08:53:45.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.256+0000 7fb851656740 20 client.25870 trim_cache size 0 max 16384 2024-05-09T08:53:45.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.301+0000 7fb851656740 1 Processor -- start 2024-05-09T08:53:45.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.301+0000 7fb851656740 1 -- start start 2024-05-09T08:53:45.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.301+0000 7fb851656740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca0b4f0 0x55b65ca0d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.301+0000 7fb851656740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca13f70 0x55b65ca0d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.301+0000 7fb851656740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca164e0 0x55b65ca2ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.301+0000 7fb851656740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8481bd930 con 0x55b65ca13f70 2024-05-09T08:53:45.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.301+0000 7fb851656740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb8481bf470 con 0x55b65ca164e0 2024-05-09T08:53:45.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.301+0000 7fb851656740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb8481be000 con 0x55b65ca0b4f0 2024-05-09T08:53:45.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.303+0000 7fb84ceec640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca164e0 0x55b65ca2ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:45.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.303+0000 7fb847fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca0b4f0 0x55b65ca0d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:45.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.303+0000 7fb847fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca0b4f0 0x55b65ca0d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:57402/0 (socket says 172.21.15.98:57402) 2024-05-09T08:53:45.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.303+0000 7fb847fff640 1 -- 172.21.15.98:0/243985835 learned_addr learned my addr 172.21.15.98:0/243985835 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:45.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.305+0000 7fb847fff640 1 -- 172.21.15.98:0/243985835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca164e0 msgr2=0x55b65ca2ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:45.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.305+0000 7fb847fff640 1 --2- 172.21.15.98:0/243985835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca164e0 0x55b65ca2ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.305+0000 7fb847fff640 1 -- 172.21.15.98:0/243985835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca13f70 msgr2=0x55b65ca0d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:45.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.305+0000 7fb847fff640 1 --2- 172.21.15.98:0/243985835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca13f70 0x55b65ca0d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.305+0000 7fb847fff640 1 -- 172.21.15.98:0/243985835 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b65ca08a20 con 0x55b65ca0b4f0 2024-05-09T08:53:45.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.306+0000 7fb847fff640 1 --2- 172.21.15.98:0/243985835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca0b4f0 0x55b65ca0d390 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb838011c40 tx=0x7fb838014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39c59c40d2152c59 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:45.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.306+0000 7fb82dffb640 1 -- 172.21.15.98:0/243985835 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83802c070 con 0x55b65ca0b4f0 2024-05-09T08:53:45.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.307+0000 7fb82dffb640 1 -- 172.21.15.98:0/243985835 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb838026880 con 0x55b65ca0b4f0 2024-05-09T08:53:45.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.308+0000 7fb851656740 1 -- 172.21.15.98:0/243985835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca0b4f0 msgr2=0x55b65ca0d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:45.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.308+0000 7fb851656740 1 --2- 172.21.15.98:0/243985835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca0b4f0 0x55b65ca0d390 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb838011c40 tx=0x7fb838014000 comp rx=0 tx=0).stop 2024-05-09T08:53:45.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.317+0000 7fb851656740 1 -- 172.21.15.98:0/243985835 shutdown_connections 2024-05-09T08:53:45.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.317+0000 7fb851656740 1 --2- 172.21.15.98:0/243985835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca164e0 0x55b65ca2ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.317+0000 7fb851656740 1 --2- 172.21.15.98:0/243985835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca13f70 0x55b65ca0d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.317+0000 7fb851656740 1 --2- 172.21.15.98:0/243985835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca0b4f0 0x55b65ca0d390 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.317+0000 7fb851656740 1 -- 172.21.15.98:0/243985835 >> 172.21.15.98:0/243985835 conn(0x55b65ca07810 msgr2=0x55b65ca0cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:45.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.321+0000 7fb851656740 1 -- 172.21.15.98:0/243985835 shutdown_connections 2024-05-09T08:53:45.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.321+0000 7fb851656740 1 -- 172.21.15.98:0/243985835 wait complete. 2024-05-09T08:53:45.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.322+0000 7fb851656740 1 -- start start 2024-05-09T08:53:45.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.322+0000 7fb851656740 1 Processor -- start 2024-05-09T08:53:45.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.323+0000 7fb851656740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca0b4f0 0x55b65ca1e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.323+0000 7fb851656740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca13f70 0x55b65ca1ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.323+0000 7fb851656740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca164e0 0x55b65ca0f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.323+0000 7fb851656740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8481beda0 con 0x55b65ca13f70 2024-05-09T08:53:45.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.323+0000 7fb851656740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb8481be6d0 con 0x55b65ca0b4f0 2024-05-09T08:53:45.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.323+0000 7fb851656740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb8481bb910 con 0x55b65ca164e0 2024-05-09T08:53:45.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.323+0000 7fb851656740 10 client.0 _ll_register_callbacks cb 0x7fb83c0b8120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:45.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.323+0000 7fb847fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca0b4f0 0x55b65ca1e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:45.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.323+0000 7fb8477fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca13f70 0x55b65ca1ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:45.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.323+0000 7fb8477fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca13f70 0x55b65ca1ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44290/0 (socket says 172.21.15.98:44290) 2024-05-09T08:53:45.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.323+0000 7fb847fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca0b4f0 0x55b65ca1e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55526/0 (socket says 172.21.15.98:55526) 2024-05-09T08:53:45.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.323+0000 7fb8477fe640 1 -- 172.21.15.98:0/461959966 learned_addr learned my addr 172.21.15.98:0/461959966 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:45.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.324+0000 7fb84ceec640 1 --2- 172.21.15.98:0/461959966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca164e0 0x55b65ca0f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:45.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.324+0000 7fb8477fe640 1 -- 172.21.15.98:0/461959966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca164e0 msgr2=0x55b65ca0f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:45.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.324+0000 7fb8477fe640 1 --2- 172.21.15.98:0/461959966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca164e0 0x55b65ca0f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.324+0000 7fb8477fe640 1 -- 172.21.15.98:0/461959966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca0b4f0 msgr2=0x55b65ca1e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:45.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.324+0000 7fb8477fe640 1 --2- 172.21.15.98:0/461959966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca0b4f0 0x55b65ca1e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.324+0000 7fb8477fe640 1 -- 172.21.15.98:0/461959966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b65ca0ff30 con 0x55b65ca13f70 2024-05-09T08:53:45.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.328+0000 7fb8477fe640 1 --2- 172.21.15.98:0/461959966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca13f70 0x55b65ca1ecb0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x55b65ca0cab0 tx=0x7fb840003c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2608d38585314c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:45.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.335+0000 7fb8277fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:45.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.335+0000 7fb8277fe640 1 -- 172.21.15.98:0/461959966 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8400135e0 con 0x55b65ca13f70 2024-05-09T08:53:45.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.335+0000 7fb8277fe640 1 -- 172.21.15.98:0/461959966 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7fb840013780 con 0x55b65ca13f70 2024-05-09T08:53:45.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.335+0000 7fb851656740 1 -- 172.21.15.98:0/461959966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb838026880 con 0x55b65ca13f70 2024-05-09T08:53:45.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.339+0000 7fb8277fe640 1 -- 172.21.15.98:0/461959966 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb840013980 con 0x55b65ca13f70 2024-05-09T08:53:45.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.339+0000 7fb851656740 10 client.16596 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:45.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.339+0000 7fb851656740 1 -- 172.21.15.98:0/461959966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b65ca8d1b0 con 0x55b65ca13f70 2024-05-09T08:53:45.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.340+0000 7fb8277fe640 1 -- 172.21.15.98:0/461959966 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb838026880 con 0x55b65ca13f70 2024-05-09T08:53:45.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.342+0000 7fb8277fe640 1 -- 172.21.15.98:0/461959966 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb840004360 con 0x55b65ca13f70 2024-05-09T08:53:45.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.342+0000 7fb851656740 10 client.16596 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:45.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.342+0000 7fb851656740 10 client.16596 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:45.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.342+0000 7fb851656740 20 client.16596 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:45.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.342+0000 7fb851656740 1 --2- 172.21.15.98:0/461959966 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b65ca8d7e0 0x55b65ca8fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:45.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.342+0000 7fb851656740 4 client.16596 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:53:45.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.342+0000 7fb851656740 1 -- 172.21.15.98:0/461959966 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b65c97e370 con 0x55b65ca8d7e0 2024-05-09T08:53:45.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.343+0000 7fb847fff640 1 --2- 172.21.15.98:0/461959966 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b65ca8d7e0 0x55b65ca8fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:45.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.345+0000 7fb847fff640 1 --2- 172.21.15.98:0/461959966 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b65ca8d7e0 0x55b65ca8fc00 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7a499ba10420b73 server_cookie=49b2460c0761b5a9 in_seq=0 out_seq=0 2024-05-09T08:53:45.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.345+0000 7fb8277fe640 10 client.16596 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:45.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.347+0000 7fb8277fe640 1 -- 172.21.15.98:0/461959966 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b65c97e370 con 0x55b65ca8d7e0 2024-05-09T08:53:45.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.347+0000 7fb8277fe640 10 client.16596 handle_command_reply: tid=0 2024-05-09T08:53:45.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.348+0000 7fb851656740 1 client.16596 shutdown 2024-05-09T08:53:45.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.350+0000 7fb851656740 1 -- 172.21.15.98:0/461959966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca13f70 msgr2=0x55b65ca1ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:45.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.350+0000 7fb851656740 1 --2- 172.21.15.98:0/461959966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca13f70 0x55b65ca1ecb0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x55b65ca0cab0 tx=0x7fb840003c40 comp rx=0 tx=0).stop 2024-05-09T08:53:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.353+0000 7fb851656740 1 -- 172.21.15.98:0/461959966 shutdown_connections 2024-05-09T08:53:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.353+0000 7fb851656740 1 --2- 172.21.15.98:0/461959966 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b65ca8d7e0 0x55b65ca8fc00 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.353+0000 7fb851656740 1 --2- 172.21.15.98:0/461959966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b65ca164e0 0x55b65ca0f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.354+0000 7fb851656740 1 --2- 172.21.15.98:0/461959966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b65ca13f70 0x55b65ca1ecb0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.354+0000 7fb851656740 1 --2- 172.21.15.98:0/461959966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65ca0b4f0 0x55b65ca1e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.354+0000 7fb851656740 1 -- 172.21.15.98:0/461959966 >> 172.21.15.98:0/461959966 conn(0x55b65ca07810 msgr2=0x55b65ca0cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.354+0000 7fb8277fe640 0 client.16596 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:45.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.354+0000 7fb851656740 1 -- 172.21.15.98:0/461959966 shutdown_connections 2024-05-09T08:53:45.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.354+0000 7fb851656740 1 -- 172.21.15.98:0/461959966 wait complete. 2024-05-09T08:53:45.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.354+0000 7fb851656740 20 client.16596 trim_cache size 0 max 16384 2024-05-09T08:53:45.355 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:53:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.355+0000 7fb84d6ed640 1 -- 172.21.15.98:0/1454128346 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8300a2f00 msgr2=0x7fb8300a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.355+0000 7fb84d6ed640 1 --2- 172.21.15.98:0/1454128346 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8300a2f00 0x7fb8300a53c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb8380058c0 tx=0x7fb838005850 comp rx=0 tx=0).stop 2024-05-09T08:53:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.356+0000 7fb84d6ed640 1 -- 172.21.15.98:0/1454128346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb84810fed0 msgr2=0x7fb8481ab6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.356+0000 7fb84d6ed640 1 --2- 172.21.15.98:0/1454128346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb84810fed0 0x7fb8481ab6e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb83c009780 tx=0x7fb83c0309b0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.356+0000 7fb84d6ed640 1 -- 172.21.15.98:0/1454128346 shutdown_connections 2024-05-09T08:53:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.356+0000 7fb84d6ed640 1 --2- 172.21.15.98:0/1454128346 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8300a2f00 0x7fb8300a53c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.356+0000 7fb84d6ed640 1 --2- 172.21.15.98:0/1454128346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb84810fed0 0x7fb8481ab6e0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.357+0000 7fb84d6ed640 1 --2- 172.21.15.98:0/1454128346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8481092b0 0x7fb8481a7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.357+0000 7fb84d6ed640 1 --2- 172.21.15.98:0/1454128346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb848108830 0x7fb8481a6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.357+0000 7fb84d6ed640 1 -- 172.21.15.98:0/1454128346 >> 172.21.15.98:0/1454128346 conn(0x7fb84806c7b0 msgr2=0x7fb848070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.357+0000 7fb84d6ed640 1 -- 172.21.15.98:0/1454128346 shutdown_connections 2024-05-09T08:53:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:45.357+0000 7fb84d6ed640 1 -- 172.21.15.98:0/1454128346 wait complete. 2024-05-09T08:53:45.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:45 smithi105 ceph-mon[34498]: pgmap v96: 129 pgs: 129 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 750 MiB/s wr, 1.94k op/s 2024-05-09T08:53:46.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:45 smithi098 ceph-mon[30092]: pgmap v96: 129 pgs: 129 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 750 MiB/s wr, 1.94k op/s 2024-05-09T08:53:46.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:45 smithi142 ceph-mon[34625]: pgmap v96: 129 pgs: 129 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 750 MiB/s wr, 1.94k op/s 2024-05-09T08:53:46.389 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:53:46.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:53:46.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:53:46.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.654+0000 7f89fd75f640 1 -- 172.21.15.98:0/3886124549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f810d910 msgr2=0x7f89f810dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:46.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.654+0000 7f89fd75f640 1 --2- 172.21.15.98:0/3886124549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f810d910 0x7f89f810dd90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f89f0009ca0 tx=0x7f89f002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.655+0000 7f89fd75f640 1 -- 172.21.15.98:0/3886124549 shutdown_connections 2024-05-09T08:53:46.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.655+0000 7f89fd75f640 1 --2- 172.21.15.98:0/3886124549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89f810e4c0 0x7f89f811d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.655+0000 7f89fd75f640 1 --2- 172.21.15.98:0/3886124549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f810d910 0x7f89f810dd90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.655+0000 7f89fd75f640 1 --2- 172.21.15.98:0/3886124549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89f810fe40 0x7f89f8110220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.656+0000 7f89fd75f640 1 -- 172.21.15.98:0/3886124549 >> 172.21.15.98:0/3886124549 conn(0x7f89f806c710 msgr2=0x7f89f806cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:46.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.661+0000 7f89fd75f640 1 -- 172.21.15.98:0/3886124549 shutdown_connections 2024-05-09T08:53:46.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.663+0000 7f89fd75f640 1 -- 172.21.15.98:0/3886124549 wait complete. 2024-05-09T08:53:46.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.664+0000 7f89fd75f640 1 Processor -- start 2024-05-09T08:53:46.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.664+0000 7f89fd75f640 1 -- start start 2024-05-09T08:53:46.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.664+0000 7f89fd75f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89f810d910 0x7f89f81ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.664+0000 7f89fd75f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f810e4c0 0x7f89f81acde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.664+0000 7f89fd75f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89f810fe40 0x7f89f81a6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.664+0000 7f89fd75f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89f8120870 con 0x7f89f810fe40 2024-05-09T08:53:46.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.664+0000 7f89fd75f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89f81206f0 con 0x7f89f810e4c0 2024-05-09T08:53:46.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.664+0000 7f89fd75f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f89f81209f0 con 0x7f89f810d910 2024-05-09T08:53:46.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.665+0000 7f89f77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f810e4c0 0x7f89f81acde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:46.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.665+0000 7f89f77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f810e4c0 0x7f89f81acde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54844/0 (socket says 172.21.15.98:54844) 2024-05-09T08:53:46.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.665+0000 7f89f77fe640 1 -- 172.21.15.98:0/4111898408 learned_addr learned my addr 172.21.15.98:0/4111898408 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:46.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.666+0000 7f89fcf5e640 1 --2- 172.21.15.98:0/4111898408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89f810fe40 0x7f89f81a6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:46.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.666+0000 7f89f7fff640 1 --2- 172.21.15.98:0/4111898408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89f810d910 0x7f89f81ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:46.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.667+0000 7f89f77fe640 1 -- 172.21.15.98:0/4111898408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89f810d910 msgr2=0x7f89f81ac8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:46.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.667+0000 7f89f77fe640 1 --2- 172.21.15.98:0/4111898408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89f810d910 0x7f89f81ac8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.667+0000 7f89f77fe640 1 -- 172.21.15.98:0/4111898408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89f810fe40 msgr2=0x7f89f81a6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:46.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.667+0000 7f89f77fe640 1 --2- 172.21.15.98:0/4111898408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89f810fe40 0x7f89f81a6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.667+0000 7f89f77fe640 1 -- 172.21.15.98:0/4111898408 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f81a71a0 con 0x7f89f810e4c0 2024-05-09T08:53:46.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.671+0000 7f89f77fe640 1 --2- 172.21.15.98:0/4111898408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f810e4c0 0x7f89f81acde0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f89f0031d50 tx=0x7f89f0030b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:46.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.673+0000 7f89f57fa640 1 -- 172.21.15.98:0/4111898408 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f003f070 con 0x7f89f810e4c0 2024-05-09T08:53:46.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.673+0000 7f89fd75f640 1 -- 172.21.15.98:0/4111898408 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89f81a7430 con 0x7f89f810e4c0 2024-05-09T08:53:46.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.673+0000 7f89fd75f640 1 -- 172.21.15.98:0/4111898408 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89f81107f0 con 0x7f89f810e4c0 2024-05-09T08:53:46.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.674+0000 7f89f57fa640 1 -- 172.21.15.98:0/4111898408 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f0003b70 con 0x7f89f810e4c0 2024-05-09T08:53:46.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.674+0000 7f89f57fa640 1 -- 172.21.15.98:0/4111898408 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f0044850 con 0x7f89f810e4c0 2024-05-09T08:53:46.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.693+0000 7f89f57fa640 1 -- 172.21.15.98:0/4111898408 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f89f00449f0 con 0x7f89f810e4c0 2024-05-09T08:53:46.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.693+0000 7f8a017a9740 1 Processor -- start 2024-05-09T08:53:46.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.696+0000 7f89f57fa640 1 --2- 172.21.15.98:0/4111898408 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89e00a2f00 0x7f89e00a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.697+0000 7f8a017a9740 1 -- start start 2024-05-09T08:53:46.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.697+0000 7f89f57fa640 1 -- 172.21.15.98:0/4111898408 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f89f00f1020 con 0x7f89f810e4c0 2024-05-09T08:53:46.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.697+0000 7f8a017a9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d1932c0 0x55bd8d191600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.697+0000 7f8a017a9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d191bd0 0x55bd8d192050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.697+0000 7f8a017a9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d19c950 0x55bd8d19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.697+0000 7f8a017a9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d0f6990 con 0x55bd8d191bd0 2024-05-09T08:53:46.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.697+0000 7f8a017a9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d09e5b0 con 0x55bd8d19c950 2024-05-09T08:53:46.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.697+0000 7f8a017a9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d0f2950 con 0x55bd8d1932c0 2024-05-09T08:53:46.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.700+0000 7f89f7fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d1932c0 0x55bd8d191600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:46.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.700+0000 7f89f7fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d1932c0 0x55bd8d191600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:50474/0 (socket says 172.21.15.98:50474) 2024-05-09T08:53:46.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.700+0000 7f89f7fff640 1 -- 172.21.15.98:0/2836279278 learned_addr learned my addr 172.21.15.98:0/2836279278 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:46.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.701+0000 7f89f7fff640 1 -- 172.21.15.98:0/2836279278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d19c950 msgr2=0x55bd8d19ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:53:46.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.701+0000 7f89f7fff640 1 --2- 172.21.15.98:0/2836279278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d19c950 0x55bd8d19ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.701+0000 7f89f7fff640 1 -- 172.21.15.98:0/2836279278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d191bd0 msgr2=0x55bd8d192050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:46.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.701+0000 7f89f7fff640 1 --2- 172.21.15.98:0/2836279278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d191bd0 0x55bd8d192050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.701+0000 7f89f7fff640 1 -- 172.21.15.98:0/2836279278 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd8ced27e0 con 0x55bd8d1932c0 2024-05-09T08:53:46.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.702+0000 7f89f7fff640 1 --2- 172.21.15.98:0/2836279278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d1932c0 0x55bd8d191600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f89e800bc90 tx=0x7f89e800f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ab7879fc37481b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:46.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.703+0000 7f89fcf5e640 1 --2- 172.21.15.98:0/4111898408 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89e00a2f00 0x7f89e00a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.705+0000 7f89de7fc640 1 -- 172.21.15.98:0/2836279278 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e8015070 con 0x55bd8d1932c0 2024-05-09T08:53:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.705+0000 7f89de7fc640 1 -- 172.21.15.98:0/2836279278 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89e801a940 con 0x55bd8d1932c0 2024-05-09T08:53:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.705+0000 7f89de7fc640 1 -- 172.21.15.98:0/2836279278 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e8011aa0 con 0x55bd8d1932c0 2024-05-09T08:53:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.707+0000 7f8a017a9740 1 -- 172.21.15.98:0/2836279278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d1932c0 msgr2=0x55bd8d191600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.707+0000 7f8a017a9740 1 --2- 172.21.15.98:0/2836279278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d1932c0 0x55bd8d191600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f89e800bc90 tx=0x7f89e800f730 comp rx=0 tx=0).stop 2024-05-09T08:53:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.708+0000 7f89fcf5e640 1 --2- 172.21.15.98:0/4111898408 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89e00a2f00 0x7f89e00a53c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x55bd8d18f2c0 tx=0x7f89ec0090e0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:46.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.712+0000 7f8a017a9740 1 -- 172.21.15.98:0/2836279278 shutdown_connections 2024-05-09T08:53:46.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.712+0000 7f8a017a9740 1 --2- 172.21.15.98:0/2836279278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d19c950 0x55bd8d19ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.712+0000 7f8a017a9740 1 --2- 172.21.15.98:0/2836279278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d191bd0 0x55bd8d192050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.712+0000 7f8a017a9740 1 --2- 172.21.15.98:0/2836279278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d1932c0 0x55bd8d191600 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.712+0000 7f8a017a9740 1 -- 172.21.15.98:0/2836279278 >> 172.21.15.98:0/2836279278 conn(0x55bd8d15a0a0 msgr2=0x55bd8d18e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:46.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.716+0000 7f8a017a9740 1 -- 172.21.15.98:0/2836279278 shutdown_connections 2024-05-09T08:53:46.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.718+0000 7f8a017a9740 1 -- 172.21.15.98:0/2836279278 wait complete. 2024-05-09T08:53:46.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.720+0000 7f8a017a9740 1 -- start start 2024-05-09T08:53:46.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.720+0000 7f8a017a9740 1 Processor -- start 2024-05-09T08:53:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.721+0000 7f8a017a9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d191bd0 0x55bd8d1998c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.721+0000 7f8a017a9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d1932c0 0x55bd8d199e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.721+0000 7f8a017a9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d19c950 0x55bd8d19a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.721+0000 7f89f77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d1932c0 0x55bd8d199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.721+0000 7f89f77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d1932c0 0x55bd8d199e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35302/0 (socket says 172.21.15.98:35302) 2024-05-09T08:53:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.721+0000 7f89f77fe640 1 -- 172.21.15.98:0/3455306126 learned_addr learned my addr 172.21.15.98:0/3455306126 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.722+0000 7f8a017a9740 1 -- 172.21.15.98:0/3455306126 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d19f7e0 con 0x55bd8d1932c0 2024-05-09T08:53:46.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.722+0000 7f8a017a9740 1 -- 172.21.15.98:0/3455306126 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d0b1b40 con 0x55bd8d191bd0 2024-05-09T08:53:46.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.722+0000 7f8a017a9740 1 -- 172.21.15.98:0/3455306126 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d19f960 con 0x55bd8d19c950 2024-05-09T08:53:46.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.722+0000 7f8a017a9740 10 client.0 _ll_register_callbacks cb 0x55bd8d0a60f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:46.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.722+0000 7f89fcf5e640 1 --2- 172.21.15.98:0/3455306126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d19c950 0x55bd8d19a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:46.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.723+0000 7f89fcf5e640 1 -- 172.21.15.98:0/3455306126 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d191bd0 msgr2=0x55bd8d1998c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:53:46.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.723+0000 7f89fcf5e640 1 --2- 172.21.15.98:0/3455306126 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d191bd0 0x55bd8d1998c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.723+0000 7f89fcf5e640 1 -- 172.21.15.98:0/3455306126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d1932c0 msgr2=0x55bd8d199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:46.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.723+0000 7f89fcf5e640 1 --2- 172.21.15.98:0/3455306126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d1932c0 0x55bd8d199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.723+0000 7f89fcf5e640 1 -- 172.21.15.98:0/3455306126 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd8cf32f70 con 0x55bd8d19c950 2024-05-09T08:53:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.727+0000 7f89fcf5e640 1 --2- 172.21.15.98:0/3455306126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d19c950 0x55bd8d19a340 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00e5a0 tx=0x7f89ec0047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=561e7d337ee2ea52 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.727+0000 7f89d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:53:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.728+0000 7f89d77fe640 1 -- 172.21.15.98:0/3455306126 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec023020 con 0x55bd8d19c950 2024-05-09T08:53:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.728+0000 7f8a017a9740 1 -- 172.21.15.98:0/3455306126 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89e801a940 con 0x55bd8d19c950 2024-05-09T08:53:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.729+0000 7f89d77fe640 1 -- 172.21.15.98:0/3455306126 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f89ec013510 con 0x55bd8d19c950 2024-05-09T08:53:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.729+0000 7f89d77fe640 1 -- 172.21.15.98:0/3455306126 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f89ec02e050 con 0x55bd8d19c950 2024-05-09T08:53:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.729+0000 7f8a017a9740 10 client.25888 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.730+0000 7f8a017a9740 1 -- 172.21.15.98:0/3455306126 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55bd8cdca170 con 0x55bd8d19c950 2024-05-09T08:53:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.731+0000 7f89d77fe640 1 -- 172.21.15.98:0/3455306126 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89e801a940 con 0x55bd8d19c950 2024-05-09T08:53:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.731+0000 7f89d77fe640 1 -- 172.21.15.98:0/3455306126 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f89ec01e030 con 0x55bd8d19c950 2024-05-09T08:53:46.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.731+0000 7f8a017a9740 10 client.25888 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:46.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.731+0000 7f8a017a9740 10 client.25888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:46.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.731+0000 7f8a017a9740 20 client.25888 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:46.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.731+0000 7f8a017a9740 1 --2- 172.21.15.98:0/3455306126 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bd8d1ace00 0x55bd8d1af2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.731+0000 7f8a017a9740 4 client.25888 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:46.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.731+0000 7f8a017a9740 1 -- 172.21.15.98:0/3455306126 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd8d1040c0 con 0x55bd8d1ace00 2024-05-09T08:53:46.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.736+0000 7f89f7fff640 1 --2- 172.21.15.98:0/3455306126 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bd8d1ace00 0x55bd8d1af2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.744+0000 7f89d77fe640 10 client.25888 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.744+0000 7f89f7fff640 1 --2- 172.21.15.98:0/3455306126 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bd8d1ace00 0x55bd8d1af2a0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1a132879577592f server_cookie=248cea392c978844 in_seq=0 out_seq=0 2024-05-09T08:53:46.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.747+0000 7f89d77fe640 1 -- 172.21.15.98:0/3455306126 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bd8d1040c0 con 0x55bd8d1ace00 2024-05-09T08:53:46.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.747+0000 7f89d77fe640 10 client.25888 handle_command_reply: tid=0 2024-05-09T08:53:46.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.747+0000 7f8a017a9740 1 client.25888 shutdown 2024-05-09T08:53:46.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.748+0000 7f8a017a9740 1 -- 172.21.15.98:0/3455306126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d19c950 msgr2=0x55bd8d19a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:46.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.748+0000 7f8a017a9740 1 --2- 172.21.15.98:0/3455306126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d19c950 0x55bd8d19a340 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00e5a0 tx=0x7f89ec0047d0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.755+0000 7f8a017a9740 1 -- 172.21.15.98:0/3455306126 shutdown_connections 2024-05-09T08:53:46.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.755+0000 7f8a017a9740 1 --2- 172.21.15.98:0/3455306126 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bd8d1ace00 0x55bd8d1af2a0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.755+0000 7f8a017a9740 1 --2- 172.21.15.98:0/3455306126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d19c950 0x55bd8d19a340 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.755+0000 7f8a017a9740 1 --2- 172.21.15.98:0/3455306126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d1932c0 0x55bd8d199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.755+0000 7f8a017a9740 1 --2- 172.21.15.98:0/3455306126 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d191bd0 0x55bd8d1998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.755+0000 7f8a017a9740 1 -- 172.21.15.98:0/3455306126 >> 172.21.15.98:0/3455306126 conn(0x55bd8d15a0a0 msgr2=0x55bd8d193fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:46.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.756+0000 7f89d77fe640 0 client.25888 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:46.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.763+0000 7f8a017a9740 1 -- 172.21.15.98:0/3455306126 shutdown_connections 2024-05-09T08:53:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.766+0000 7f8a017a9740 1 -- 172.21.15.98:0/3455306126 wait complete. 2024-05-09T08:53:46.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.766+0000 7f8a017a9740 20 client.25888 trim_cache size 0 max 16384 2024-05-09T08:53:46.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.812+0000 7f8a017a9740 1 Processor -- start 2024-05-09T08:53:46.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.812+0000 7f8a017a9740 1 -- start start 2024-05-09T08:53:46.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.812+0000 7f8a017a9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d191bd0 0x55bd8d1a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.813+0000 7f8a017a9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d1932c0 0x55bd8d19a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.813+0000 7f8a017a9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d19c950 0x55bd8d18d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.813+0000 7f8a017a9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89e4010030 con 0x55bd8d191bd0 2024-05-09T08:53:46.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.813+0000 7f8a017a9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89e4011b70 con 0x55bd8d19c950 2024-05-09T08:53:46.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.813+0000 7f8a017a9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f89e4010700 con 0x55bd8d1932c0 2024-05-09T08:53:46.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.813+0000 7f89f77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d1932c0 0x55bd8d19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:46.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.813+0000 7f89f77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d1932c0 0x55bd8d19a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:50492/0 (socket says 172.21.15.98:50492) 2024-05-09T08:53:46.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.813+0000 7f89f77fe640 1 -- 172.21.15.98:0/3331688189 learned_addr learned my addr 172.21.15.98:0/3331688189 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:46.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.813+0000 7f89fcf5e640 1 --2- 172.21.15.98:0/3331688189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d19c950 0x55bd8d18d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:46.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.814+0000 7f89f77fe640 1 -- 172.21.15.98:0/3331688189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d19c950 msgr2=0x55bd8d18d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:46.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.814+0000 7f89f77fe640 1 --2- 172.21.15.98:0/3331688189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d19c950 0x55bd8d18d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.814+0000 7f89f77fe640 1 -- 172.21.15.98:0/3331688189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d191bd0 msgr2=0x55bd8d1a8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:46.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.814+0000 7f89f77fe640 1 --2- 172.21.15.98:0/3331688189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d191bd0 0x55bd8d1a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.814+0000 7f89f77fe640 1 -- 172.21.15.98:0/3331688189 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd8d198870 con 0x55bd8d1932c0 2024-05-09T08:53:46.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.817+0000 7f89f77fe640 1 --2- 172.21.15.98:0/3331688189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d1932c0 0x55bd8d19a160 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f89f00b9580 tx=0x7f89f00095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc413ea4ededecb3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:46.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.818+0000 7f89ddffb640 1 -- 172.21.15.98:0/3331688189 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f00d6070 con 0x55bd8d1932c0 2024-05-09T08:53:46.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.818+0000 7f89ddffb640 1 -- 172.21.15.98:0/3331688189 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f0036030 con 0x55bd8d1932c0 2024-05-09T08:53:46.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.819+0000 7f8a017a9740 1 -- 172.21.15.98:0/3331688189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d1932c0 msgr2=0x55bd8d19a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:46.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.819+0000 7f8a017a9740 1 --2- 172.21.15.98:0/3331688189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d1932c0 0x55bd8d19a160 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f89f00b9580 tx=0x7f89f00095c0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.823+0000 7f8a017a9740 1 -- 172.21.15.98:0/3331688189 shutdown_connections 2024-05-09T08:53:46.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.823+0000 7f8a017a9740 1 --2- 172.21.15.98:0/3331688189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d19c950 0x55bd8d18d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.823+0000 7f8a017a9740 1 --2- 172.21.15.98:0/3331688189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d1932c0 0x55bd8d19a160 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.823+0000 7f8a017a9740 1 --2- 172.21.15.98:0/3331688189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d191bd0 0x55bd8d1a8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.823+0000 7f8a017a9740 1 -- 172.21.15.98:0/3331688189 >> 172.21.15.98:0/3331688189 conn(0x55bd8d15a0a0 msgr2=0x55bd8d18ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:46.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.823+0000 7f8a017a9740 1 -- 172.21.15.98:0/3331688189 shutdown_connections 2024-05-09T08:53:46.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.823+0000 7f8a017a9740 1 -- 172.21.15.98:0/3331688189 wait complete. 2024-05-09T08:53:46.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.823+0000 7f8a017a9740 1 -- start start 2024-05-09T08:53:46.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.824+0000 7f8a017a9740 1 Processor -- start 2024-05-09T08:53:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.826+0000 7f8a017a9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d191bd0 0x55bd8d1a0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.826+0000 7f8a017a9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d1932c0 0x55bd8d1a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.826+0000 7f8a017a9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d19c950 0x55bd8d1a12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.826+0000 7f8a017a9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89e40114a0 con 0x55bd8d1932c0 2024-05-09T08:53:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.826+0000 7f8a017a9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89e4010dd0 con 0x55bd8d191bd0 2024-05-09T08:53:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.826+0000 7f8a017a9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f89e400e010 con 0x55bd8d19c950 2024-05-09T08:53:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.826+0000 7f8a017a9740 10 client.0 _ll_register_callbacks cb 0x7f89ec009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.827+0000 7f89f7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d191bd0 0x55bd8d1a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.827+0000 7f89f7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d191bd0 0x55bd8d1a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54884/0 (socket says 172.21.15.98:54884) 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.827+0000 7f89f7fff640 1 -- 172.21.15.98:0/2654108961 learned_addr learned my addr 172.21.15.98:0/2654108961 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.827+0000 7f89fcf5e640 1 --2- 172.21.15.98:0/2654108961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d19c950 0x55bd8d1a12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.827+0000 7f89f7fff640 1 -- 172.21.15.98:0/2654108961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d19c950 msgr2=0x55bd8d1a12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.827+0000 7f89f7fff640 1 --2- 172.21.15.98:0/2654108961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d19c950 0x55bd8d1a12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.827+0000 7f89f7fff640 1 -- 172.21.15.98:0/2654108961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d1932c0 msgr2=0x55bd8d1a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.827+0000 7f89f7fff640 1 --2- 172.21.15.98:0/2654108961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d1932c0 0x55bd8d1a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.827+0000 7f89f7fff640 1 -- 172.21.15.98:0/2654108961 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd8d212620 con 0x55bd8d191bd0 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.828+0000 7f89f7fff640 1 --2- 172.21.15.98:0/2654108961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d191bd0 0x55bd8d1a0860 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f89e80115b0 tx=0x7f89e8005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=862c9fab1e9cfd9a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.828+0000 7f89d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.828+0000 7f89d77fe640 1 -- 172.21.15.98:0/2654108961 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e8015040 con 0x55bd8d191bd0 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.828+0000 7f8a017a9740 1 -- 172.21.15.98:0/2654108961 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89f0036030 con 0x55bd8d191bd0 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.828+0000 7f89d77fe640 1 -- 172.21.15.98:0/2654108961 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f89e8004030 con 0x55bd8d191bd0 2024-05-09T08:53:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.828+0000 7f89d77fe640 1 -- 172.21.15.98:0/2654108961 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f89e8004250 con 0x55bd8d191bd0 2024-05-09T08:53:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.828+0000 7f8a017a9740 10 client.25931 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.828+0000 7f8a017a9740 1 -- 172.21.15.98:0/2654108961 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55bd8d213920 con 0x55bd8d191bd0 2024-05-09T08:53:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.829+0000 7f89d77fe640 1 -- 172.21.15.98:0/2654108961 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f0036030 con 0x55bd8d191bd0 2024-05-09T08:53:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.829+0000 7f89d77fe640 1 -- 172.21.15.98:0/2654108961 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f89e8027170 con 0x55bd8d191bd0 2024-05-09T08:53:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.829+0000 7f8a017a9740 10 client.25931 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.829+0000 7f8a017a9740 10 client.25931 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.829+0000 7f8a017a9740 20 client.25931 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.829+0000 7f8a017a9740 1 --2- 172.21.15.98:0/2654108961 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bd8d213fa0 0x55bd8d216390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.829+0000 7f8a017a9740 4 client.25931 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:53:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.829+0000 7f8a017a9740 1 -- 172.21.15.98:0/2654108961 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bd8d0fe410 con 0x55bd8d213fa0 2024-05-09T08:53:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.829+0000 7f89f77fe640 1 --2- 172.21.15.98:0/2654108961 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bd8d213fa0 0x55bd8d216390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.830+0000 7f89f77fe640 1 --2- 172.21.15.98:0/2654108961 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bd8d213fa0 0x55bd8d216390 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df445e2d05f7f729 server_cookie=33e5a5a68426ce29 in_seq=0 out_seq=0 2024-05-09T08:53:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.830+0000 7f89d77fe640 10 client.25931 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:46.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.831+0000 7f89d77fe640 1 -- 172.21.15.98:0/2654108961 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bd8d0fe410 con 0x55bd8d213fa0 2024-05-09T08:53:46.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.831+0000 7f89d77fe640 10 client.25931 handle_command_reply: tid=0 2024-05-09T08:53:46.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.831+0000 7f8a017a9740 1 client.25931 shutdown 2024-05-09T08:53:46.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.834+0000 7f8a017a9740 1 -- 172.21.15.98:0/2654108961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d191bd0 msgr2=0x55bd8d1a0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:46.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.834+0000 7f8a017a9740 1 --2- 172.21.15.98:0/2654108961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d191bd0 0x55bd8d1a0860 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f89e80115b0 tx=0x7f89e8005270 comp rx=0 tx=0).stop 2024-05-09T08:53:46.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.836+0000 7f8a017a9740 1 -- 172.21.15.98:0/2654108961 shutdown_connections 2024-05-09T08:53:46.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.836+0000 7f8a017a9740 1 --2- 172.21.15.98:0/2654108961 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bd8d213fa0 0x55bd8d216390 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.836+0000 7f8a017a9740 1 --2- 172.21.15.98:0/2654108961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bd8d19c950 0x55bd8d1a12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.836+0000 7f8a017a9740 1 --2- 172.21.15.98:0/2654108961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bd8d1932c0 0x55bd8d1a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.836+0000 7f8a017a9740 1 --2- 172.21.15.98:0/2654108961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bd8d191bd0 0x55bd8d1a0860 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.836+0000 7f8a017a9740 1 -- 172.21.15.98:0/2654108961 >> 172.21.15.98:0/2654108961 conn(0x55bd8d15a0a0 msgr2=0x55bd8d18d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:46.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.836+0000 7f89d77fe640 0 client.25931 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:46.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.839+0000 7f8a017a9740 1 -- 172.21.15.98:0/2654108961 shutdown_connections 2024-05-09T08:53:46.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.839+0000 7f8a017a9740 1 -- 172.21.15.98:0/2654108961 wait complete. 2024-05-09T08:53:46.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.839+0000 7f8a017a9740 20 client.25931 trim_cache size 0 max 16384 2024-05-09T08:53:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:53:46.839 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:53:46.839 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "f0c1805e-17ba-45be-b798-711a853148cc", 2024-05-09T08:53:46.839 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:53:46.840 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:53:46.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.841+0000 7f89fd75f640 1 -- 172.21.15.98:0/4111898408 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89e00a2f00 msgr2=0x7f89e00a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:46.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.841+0000 7f89fd75f640 1 --2- 172.21.15.98:0/4111898408 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89e00a2f00 0x7f89e00a53c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x55bd8d18f2c0 tx=0x7f89ec0090e0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.841+0000 7f89fd75f640 1 -- 172.21.15.98:0/4111898408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f810e4c0 msgr2=0x7f89f81acde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:46.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.841+0000 7f89fd75f640 1 --2- 172.21.15.98:0/4111898408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f810e4c0 0x7f89f81acde0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f89f0031d50 tx=0x7f89f0030b80 comp rx=0 tx=0).stop 2024-05-09T08:53:46.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.846+0000 7f89fd75f640 1 -- 172.21.15.98:0/4111898408 shutdown_connections 2024-05-09T08:53:46.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.846+0000 7f89fd75f640 1 --2- 172.21.15.98:0/4111898408 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89e00a2f00 0x7f89e00a53c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.846+0000 7f89fd75f640 1 --2- 172.21.15.98:0/4111898408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89f810fe40 0x7f89f81a6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.846+0000 7f89fd75f640 1 --2- 172.21.15.98:0/4111898408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f810e4c0 0x7f89f81acde0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.846+0000 7f89fd75f640 1 --2- 172.21.15.98:0/4111898408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89f810d910 0x7f89f81ac8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:46.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.846+0000 7f89fd75f640 1 -- 172.21.15.98:0/4111898408 >> 172.21.15.98:0/4111898408 conn(0x7f89f806c710 msgr2=0x7f89f8070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:46.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.848+0000 7f89fd75f640 1 -- 172.21.15.98:0/4111898408 shutdown_connections 2024-05-09T08:53:46.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:46.848+0000 7f89fd75f640 1 -- 172.21.15.98:0/4111898408 wait complete. 2024-05-09T08:53:46.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:53:46.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:53:47.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.302+0000 7f2bd9d96640 1 -- 172.21.15.98:0/2857358905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bd4110080 msgr2=0x7f2bd410db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:47.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.302+0000 7f2bd9d96640 1 --2- 172.21.15.98:0/2857358905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bd4110080 0x7f2bd410db70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4009900 tx=0x7f2bc402a440 comp rx=0 tx=0).stop 2024-05-09T08:53:47.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.304+0000 7f2bd9d96640 1 -- 172.21.15.98:0/2857358905 shutdown_connections 2024-05-09T08:53:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.304+0000 7f2bd9d96640 1 --2- 172.21.15.98:0/2857358905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bd411b010 0x7f2bd411d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.304+0000 7f2bd9d96640 1 --2- 172.21.15.98:0/2857358905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bd410e0b0 0x7f2bd410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.304+0000 7f2bd9d96640 1 --2- 172.21.15.98:0/2857358905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bd4110080 0x7f2bd410db70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.304+0000 7f2bd9d96640 1 -- 172.21.15.98:0/2857358905 >> 172.21.15.98:0/2857358905 conn(0x7f2bd406c940 msgr2=0x7f2bd406cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.305+0000 7f2bd9d96640 1 -- 172.21.15.98:0/2857358905 shutdown_connections 2024-05-09T08:53:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.306+0000 7f2bd9d96640 1 -- 172.21.15.98:0/2857358905 wait complete. 2024-05-09T08:53:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.306+0000 7f2bd9d96640 1 Processor -- start 2024-05-09T08:53:47.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.307+0000 7f2bd9d96640 1 -- start start 2024-05-09T08:53:47.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.307+0000 7f2bd9d96640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bd410e0b0 0x7f2bd41a6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.307+0000 7f2bd9d96640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bd4110080 0x7f2bd41a7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.307+0000 7f2bd9d96640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bd411b010 0x7f2bd41ab8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.307+0000 7f2bd9d96640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd4120ae0 con 0x7f2bd411b010 2024-05-09T08:53:47.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.307+0000 7f2bd9d96640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd4120960 con 0x7f2bd410e0b0 2024-05-09T08:53:47.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.307+0000 7f2bd9d96640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd4120c60 con 0x7f2bd4110080 2024-05-09T08:53:47.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.307+0000 7f2bd37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bd410e0b0 0x7f2bd41a6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.307+0000 7f2bd37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bd410e0b0 0x7f2bd41a6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54900/0 (socket says 172.21.15.98:54900) 2024-05-09T08:53:47.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.307+0000 7f2bd37fe640 1 -- 172.21.15.98:0/1824873586 learned_addr learned my addr 172.21.15.98:0/1824873586 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:47.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.307+0000 7f2bd3fff640 1 --2- 172.21.15.98:0/1824873586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bd411b010 0x7f2bd41ab8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.308+0000 7f2bd2ffd640 1 --2- 172.21.15.98:0/1824873586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bd4110080 0x7f2bd41a7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.308+0000 7f2bd3fff640 1 -- 172.21.15.98:0/1824873586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bd4110080 msgr2=0x7f2bd41a7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:47.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.308+0000 7f2bd3fff640 1 --2- 172.21.15.98:0/1824873586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bd4110080 0x7f2bd41a7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.308+0000 7f2bd3fff640 1 -- 172.21.15.98:0/1824873586 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bd410e0b0 msgr2=0x7f2bd41a6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:47.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.308+0000 7f2bd3fff640 1 --2- 172.21.15.98:0/1824873586 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bd410e0b0 0x7f2bd41a6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.308+0000 7f2bd3fff640 1 -- 172.21.15.98:0/1824873586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bd41abfa0 con 0x7f2bd411b010 2024-05-09T08:53:47.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.309+0000 7f2bd3fff640 1 --2- 172.21.15.98:0/1824873586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bd411b010 0x7f2bd41ab8a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc00be20 tx=0x7f2bcc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:47.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.310+0000 7f2bd0ff9640 1 -- 172.21.15.98:0/1824873586 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc019070 con 0x7f2bd411b010 2024-05-09T08:53:47.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.311+0000 7f2bd9d96640 1 -- 172.21.15.98:0/1824873586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bd41ac290 con 0x7f2bd411b010 2024-05-09T08:53:47.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.311+0000 7f2bd9d96640 1 -- 172.21.15.98:0/1824873586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bd41107f0 con 0x7f2bd411b010 2024-05-09T08:53:47.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.311+0000 7f2bd0ff9640 1 -- 172.21.15.98:0/1824873586 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bcc014920 con 0x7f2bd411b010 2024-05-09T08:53:47.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.311+0000 7f2bd0ff9640 1 -- 172.21.15.98:0/1824873586 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc012930 con 0x7f2bd411b010 2024-05-09T08:53:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.315+0000 7f2bdcb77740 1 Processor -- start 2024-05-09T08:53:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.315+0000 7f2bd0ff9640 1 -- 172.21.15.98:0/1824873586 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2bcc01c070 con 0x7f2bd411b010 2024-05-09T08:53:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.316+0000 7f2bdcb77740 1 -- start start 2024-05-09T08:53:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.316+0000 7f2bdcb77740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac4e280 0x55794ac4e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.316+0000 7f2bdcb77740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4eba0 0x55794ac58d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.317+0000 7f2bd37fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4eba0 0x55794ac58d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.317+0000 7f2bd37fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4eba0 0x55794ac58d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:50512/0 (socket says 172.21.15.98:50512) 2024-05-09T08:53:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.317+0000 7f2bd0ff9640 1 --2- 172.21.15.98:0/1824873586 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2bbc0a2e80 0x7f2bbc0a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.317+0000 7f2bd0ff9640 1 -- 172.21.15.98:0/1824873586 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f2bcc0ceec0 con 0x7f2bd411b010 2024-05-09T08:53:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.318+0000 7f2bdcb77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac59270 0x55794ac5b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.318+0000 7f2bd37fe640 1 -- 172.21.15.98:0/3975018867 learned_addr learned my addr 172.21.15.98:0/3975018867 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.318+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3975018867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55794abb3d00 con 0x55794ac59270 2024-05-09T08:53:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.318+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3975018867 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55794ab5b4c0 con 0x55794ac4e280 2024-05-09T08:53:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.318+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3975018867 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55794abafcc0 con 0x55794ac4eba0 2024-05-09T08:53:47.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.318+0000 7f2bd2ffd640 1 --2- 172.21.15.98:0/3975018867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac4e280 0x55794ac4e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.319+0000 7f2bd2ffd640 1 --2- 172.21.15.98:0/3975018867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac59270 0x55794ac5b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.319+0000 7f2bd37fe640 1 -- 172.21.15.98:0/3975018867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac4e280 msgr2=0x55794ac4e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:47.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.319+0000 7f2bd37fe640 1 --2- 172.21.15.98:0/3975018867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac4e280 0x55794ac4e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.319+0000 7f2bd37fe640 1 -- 172.21.15.98:0/3975018867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac59270 msgr2=0x55794ac5b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.319+0000 7f2bd37fe640 1 --2- 172.21.15.98:0/3975018867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac59270 0x55794ac5b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.319+0000 7f2bd37fe640 1 -- 172.21.15.98:0/3975018867 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55794a9917d0 con 0x55794ac4eba0 2024-05-09T08:53:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.319+0000 7f2bd37fe640 1 --2- 172.21.15.98:0/3975018867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4eba0 0x55794ac58d30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4004120 tx=0x7f2bc4004150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cee4aaf2d1b7fb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.320+0000 7f2bb9ffb640 1 -- 172.21.15.98:0/3975018867 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4035030 con 0x55794ac4eba0 2024-05-09T08:53:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.320+0000 7f2bb9ffb640 1 -- 172.21.15.98:0/3975018867 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bc4003ad0 con 0x55794ac4eba0 2024-05-09T08:53:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.320+0000 7f2bb9ffb640 1 -- 172.21.15.98:0/3975018867 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4031de0 con 0x55794ac4eba0 2024-05-09T08:53:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.320+0000 7f2bd3fff640 1 --2- 172.21.15.98:0/1824873586 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2bbc0a2e80 0x7f2bbc0a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.321+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3975018867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4eba0 msgr2=0x55794ac58d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.321+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/3975018867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4eba0 0x55794ac58d30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4004120 tx=0x7f2bc4004150 comp rx=0 tx=0).stop 2024-05-09T08:53:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.321+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3975018867 shutdown_connections 2024-05-09T08:53:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.321+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/3975018867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac59270 0x55794ac5b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.321+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/3975018867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4eba0 0x55794ac58d30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.321+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/3975018867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac4e280 0x55794ac4e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.321+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3975018867 >> 172.21.15.98:0/3975018867 conn(0x55794ac4a5a0 msgr2=0x55794ac4a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.322+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3975018867 shutdown_connections 2024-05-09T08:53:47.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.322+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3975018867 wait complete. 2024-05-09T08:53:47.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.322+0000 7f2bd3fff640 1 --2- 172.21.15.98:0/1824873586 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2bbc0a2e80 0x7f2bbc0a5340 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc098b70 tx=0x7f2bcc009f90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:47.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.322+0000 7f2bdcb77740 1 -- start start 2024-05-09T08:53:47.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.323+0000 7f2bdcb77740 1 Processor -- start 2024-05-09T08:53:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.323+0000 7f2bdcb77740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac4e280 0x55794ac56190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.323+0000 7f2bdcb77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4eba0 0x55794ac566d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.323+0000 7f2bdcb77740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac59270 0x55794ac56c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.323+0000 7f2bdcb77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55794ac5c000 con 0x55794ac4eba0 2024-05-09T08:53:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.324+0000 7f2bdcb77740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55794ab6eb90 con 0x55794ac4e280 2024-05-09T08:53:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.324+0000 7f2bdcb77740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55794ac5c180 con 0x55794ac59270 2024-05-09T08:53:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.324+0000 7f2bdcb77740 10 client.0 _ll_register_callbacks cb 0x55794abaada0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.324+0000 7f2bd37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4eba0 0x55794ac566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.324+0000 7f2bd37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4eba0 0x55794ac566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35348/0 (socket says 172.21.15.98:35348) 2024-05-09T08:53:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.324+0000 7f2bd37fe640 1 -- 172.21.15.98:0/3668867685 learned_addr learned my addr 172.21.15.98:0/3668867685 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.324+0000 7f2bd2ffd640 1 --2- 172.21.15.98:0/3668867685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac4e280 0x55794ac56190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.324+0000 7f2bd2ffd640 1 --2- 172.21.15.98:0/3668867685 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac59270 0x55794ac56c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.324+0000 7f2bd2ffd640 1 -- 172.21.15.98:0/3668867685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac4e280 msgr2=0x55794ac56190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:47.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.324+0000 7f2bd2ffd640 1 --2- 172.21.15.98:0/3668867685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac4e280 0x55794ac56190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.324+0000 7f2bd2ffd640 1 -- 172.21.15.98:0/3668867685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4eba0 msgr2=0x55794ac566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:47.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.324+0000 7f2bd2ffd640 1 --2- 172.21.15.98:0/3668867685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4eba0 0x55794ac566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.324+0000 7f2bd2ffd640 1 -- 172.21.15.98:0/3668867685 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55794a8c73f0 con 0x55794ac59270 2024-05-09T08:53:47.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.325+0000 7f2bd2ffd640 1 --2- 172.21.15.98:0/3668867685 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac59270 0x55794ac56c10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2bd411d450 tx=0x7f2bc8004700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e990ac7d0c1994d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:47.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.325+0000 7f2bb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:53:47.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.325+0000 7f2bb2ffd640 1 -- 172.21.15.98:0/3668867685 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8015360 con 0x55794ac59270 2024-05-09T08:53:47.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.325+0000 7f2bb2ffd640 1 -- 172.21.15.98:0/3668867685 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f2bc8015500 con 0x55794ac59270 2024-05-09T08:53:47.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.325+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3668867685 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bc4003ad0 con 0x55794ac59270 2024-05-09T08:53:47.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.327+0000 7f2bb2ffd640 1 -- 172.21.15.98:0/3668867685 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2bc8015700 con 0x55794ac59270 2024-05-09T08:53:47.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.327+0000 7f2bdcb77740 10 client.25906 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:47.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.327+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3668867685 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55794a8870f0 con 0x55794ac59270 2024-05-09T08:53:47.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.327+0000 7f2bb2ffd640 1 -- 172.21.15.98:0/3668867685 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bc4003ad0 con 0x55794ac59270 2024-05-09T08:53:47.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.327+0000 7f2bb2ffd640 1 -- 172.21.15.98:0/3668867685 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f2bc8015040 con 0x55794ac59270 2024-05-09T08:53:47.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.327+0000 7f2bdcb77740 10 client.25906 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:47.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.327+0000 7f2bdcb77740 10 client.25906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:47.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.327+0000 7f2bdcb77740 20 client.25906 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:47.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.327+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/3668867685 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55794ac69680 0x55794ac6bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.327+0000 7f2bdcb77740 4 client.25906 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:47.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.327+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3668867685 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55794abc1430 con 0x55794ac69680 2024-05-09T08:53:47.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.327+0000 7f2bd37fe640 1 --2- 172.21.15.98:0/3668867685 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55794ac69680 0x55794ac6bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.331+0000 7f2bd37fe640 1 --2- 172.21.15.98:0/3668867685 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55794ac69680 0x55794ac6bab0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ead108d157f3840f server_cookie=aae9299bb7b355f3 in_seq=0 out_seq=0 2024-05-09T08:53:47.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.331+0000 7f2bb2ffd640 10 client.25906 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.335+0000 7f2bb2ffd640 1 -- 172.21.15.98:0/3668867685 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55794abc1430 con 0x55794ac69680 2024-05-09T08:53:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.335+0000 7f2bb2ffd640 10 client.25906 handle_command_reply: tid=0 2024-05-09T08:53:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.335+0000 7f2bdcb77740 1 client.25906 shutdown 2024-05-09T08:53:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.335+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3668867685 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac59270 msgr2=0x55794ac56c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.335+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/3668867685 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac59270 0x55794ac56c10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2bd411d450 tx=0x7f2bc8004700 comp rx=0 tx=0).stop 2024-05-09T08:53:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.335+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3668867685 shutdown_connections 2024-05-09T08:53:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.335+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/3668867685 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55794ac69680 0x55794ac6bab0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.335+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/3668867685 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac59270 0x55794ac56c10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.335+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/3668867685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4eba0 0x55794ac566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.335+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/3668867685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac4e280 0x55794ac56190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.335+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3668867685 >> 172.21.15.98:0/3668867685 conn(0x55794ac4a5a0 msgr2=0x55794ac497e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.335+0000 7f2bb2ffd640 0 client.25906 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.335+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3668867685 shutdown_connections 2024-05-09T08:53:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.336+0000 7f2bdcb77740 1 -- 172.21.15.98:0/3668867685 wait complete. 2024-05-09T08:53:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.336+0000 7f2bdcb77740 20 client.25906 trim_cache size 0 max 16384 2024-05-09T08:53:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.359+0000 7f2bdcb77740 1 Processor -- start 2024-05-09T08:53:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.360+0000 7f2bdcb77740 1 -- start start 2024-05-09T08:53:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.360+0000 7f2bdcb77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4e280 0x55794ac64f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.360+0000 7f2bdcb77740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4eba0 0x55794ac55a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.360+0000 7f2bdcb77740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac59270 0x55794ac713c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.360+0000 7f2bdcb77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc000ff70 con 0x55794ac4e280 2024-05-09T08:53:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.360+0000 7f2bdcb77740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc0011ab0 con 0x55794ac59270 2024-05-09T08:53:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.360+0000 7f2bdcb77740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc0010640 con 0x55794ac4eba0 2024-05-09T08:53:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.360+0000 7f2bd2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4e280 0x55794ac64f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.360+0000 7f2bd2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4e280 0x55794ac64f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35356/0 (socket says 172.21.15.98:35356) 2024-05-09T08:53:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.360+0000 7f2bd2ffd640 1 -- 172.21.15.98:0/4040041329 learned_addr learned my addr 172.21.15.98:0/4040041329 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.361+0000 7f2bd37fe640 1 --2- 172.21.15.98:0/4040041329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4eba0 0x55794ac55a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.361+0000 7f2bd2ffd640 1 --2- 172.21.15.98:0/4040041329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac59270 0x55794ac713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.367+0000 7f2bd37fe640 1 -- 172.21.15.98:0/4040041329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac59270 msgr2=0x55794ac713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:47.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.367+0000 7f2bd37fe640 1 --2- 172.21.15.98:0/4040041329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac59270 0x55794ac713c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.367+0000 7f2bd37fe640 1 -- 172.21.15.98:0/4040041329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4e280 msgr2=0x55794ac64f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:47.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.367+0000 7f2bd37fe640 1 --2- 172.21.15.98:0/4040041329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4e280 0x55794ac64f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.367+0000 7f2bd37fe640 1 -- 172.21.15.98:0/4040041329 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55794ac458a0 con 0x55794ac4eba0 2024-05-09T08:53:47.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.367+0000 7f2bd2ffd640 1 --2- 172.21.15.98:0/4040041329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4e280 0x55794ac64f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:53:47.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.368+0000 7f2bd37fe640 1 --2- 172.21.15.98:0/4040041329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4eba0 0x55794ac55a70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2bc402a480 tx=0x7f2bc4031940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5671f28f4c10eb40 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:47.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.368+0000 7f2bb97fa640 1 -- 172.21.15.98:0/4040041329 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4058070 con 0x55794ac4eba0 2024-05-09T08:53:47.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.368+0000 7f2bb97fa640 1 -- 172.21.15.98:0/4040041329 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bc40316f0 con 0x55794ac4eba0 2024-05-09T08:53:47.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.368+0000 7f2bb97fa640 1 -- 172.21.15.98:0/4040041329 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4035040 con 0x55794ac4eba0 2024-05-09T08:53:47.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.369+0000 7f2bdcb77740 1 -- 172.21.15.98:0/4040041329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4eba0 msgr2=0x55794ac55a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:47.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.370+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/4040041329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4eba0 0x55794ac55a70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2bc402a480 tx=0x7f2bc4031940 comp rx=0 tx=0).stop 2024-05-09T08:53:47.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.370+0000 7f2bdcb77740 1 -- 172.21.15.98:0/4040041329 shutdown_connections 2024-05-09T08:53:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.370+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/4040041329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac59270 0x55794ac713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.370+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/4040041329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4eba0 0x55794ac55a70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.370+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/4040041329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4e280 0x55794ac64f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.370+0000 7f2bdcb77740 1 -- 172.21.15.98:0/4040041329 >> 172.21.15.98:0/4040041329 conn(0x55794ac4a5a0 msgr2=0x55794ac502a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.370+0000 7f2bdcb77740 1 -- 172.21.15.98:0/4040041329 shutdown_connections 2024-05-09T08:53:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.370+0000 7f2bdcb77740 1 -- 172.21.15.98:0/4040041329 wait complete. 2024-05-09T08:53:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.371+0000 7f2bdcb77740 1 -- start start 2024-05-09T08:53:47.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.371+0000 7f2bdcb77740 1 Processor -- start 2024-05-09T08:53:47.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.372+0000 7f2bdcb77740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4e280 0x55794acc7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.372+0000 7f2bdcb77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4eba0 0x55794acc8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.373+0000 7f2bd2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4e280 0x55794acc7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.373+0000 7f2bd37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4eba0 0x55794acc8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.373+0000 7f2bd2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4e280 0x55794acc7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:50550/0 (socket says 172.21.15.98:50550) 2024-05-09T08:53:47.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.373+0000 7f2bdcb77740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac59270 0x55794ac5c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.373+0000 7f2bdcb77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc00113e0 con 0x55794ac4eba0 2024-05-09T08:53:47.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.373+0000 7f2bdcb77740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc0010d10 con 0x55794ac59270 2024-05-09T08:53:47.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.373+0000 7f2bdcb77740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc000df50 con 0x55794ac4e280 2024-05-09T08:53:47.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.373+0000 7f2bdcb77740 10 client.0 _ll_register_callbacks cb 0x7f2bc800df10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:47.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.373+0000 7f2bd2ffd640 1 -- 172.21.15.98:0/1650634728 learned_addr learned my addr 172.21.15.98:0/1650634728 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:47.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.374+0000 7f2bd2ffd640 1 --2- 172.21.15.98:0/1650634728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac59270 0x55794ac5c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.375+0000 7f2bd37fe640 1 -- 172.21.15.98:0/1650634728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4e280 msgr2=0x55794acc7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:47.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.375+0000 7f2bd37fe640 1 --2- 172.21.15.98:0/1650634728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4e280 0x55794acc7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.375+0000 7f2bd37fe640 1 -- 172.21.15.98:0/1650634728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac59270 msgr2=0x55794ac5c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:47.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.375+0000 7f2bd37fe640 1 --2- 172.21.15.98:0/1650634728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac59270 0x55794ac5c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.375+0000 7f2bd37fe640 1 -- 172.21.15.98:0/1650634728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55794ac5cc80 con 0x55794ac4eba0 2024-05-09T08:53:47.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.377+0000 7f2bd37fe640 1 --2- 172.21.15.98:0/1650634728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4eba0 0x55794acc8470 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x55794ac5bd30 tx=0x7f2bc4031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdebf6cea58414d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:47.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.377+0000 7f2bb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:47.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.378+0000 7f2bb2ffd640 1 -- 172.21.15.98:0/1650634728 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4058040 con 0x55794ac4eba0 2024-05-09T08:53:47.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.379+0000 7f2bdcb77740 1 -- 172.21.15.98:0/1650634728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bc40316f0 con 0x55794ac4eba0 2024-05-09T08:53:47.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.379+0000 7f2bb2ffd640 1 -- 172.21.15.98:0/1650634728 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f2bc4052170 con 0x55794ac4eba0 2024-05-09T08:53:47.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.380+0000 7f2bb2ffd640 1 -- 172.21.15.98:0/1650634728 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2bc4035040 con 0x55794ac4eba0 2024-05-09T08:53:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.385+0000 7f2bdcb77740 10 client.16617 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.385+0000 7f2bdcb77740 1 -- 172.21.15.98:0/1650634728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55794accf670 con 0x55794ac4eba0 2024-05-09T08:53:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.386+0000 7f2bb2ffd640 1 -- 172.21.15.98:0/1650634728 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bc40316f0 con 0x55794ac4eba0 2024-05-09T08:53:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.386+0000 7f2bb2ffd640 1 -- 172.21.15.98:0/1650634728 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f2bc4003710 con 0x55794ac4eba0 2024-05-09T08:53:47.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.387+0000 7f2bdcb77740 10 client.16617 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:47.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.387+0000 7f2bdcb77740 10 client.16617 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:47.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.387+0000 7f2bdcb77740 20 client.16617 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:47.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.387+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/1650634728 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55794accf950 0x55794acd1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:47.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.387+0000 7f2bdcb77740 4 client.16617 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:53:47.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.387+0000 7f2bdcb77740 1 -- 172.21.15.98:0/1650634728 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55794abbb780 con 0x55794accf950 2024-05-09T08:53:47.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.390+0000 7f2bd2ffd640 1 --2- 172.21.15.98:0/1650634728 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55794accf950 0x55794acd1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:47.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.393+0000 7f2bd2ffd640 1 --2- 172.21.15.98:0/1650634728 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55794accf950 0x55794acd1d40 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e6d175ffe813f63 server_cookie=59c4118dc7aa1fb9 in_seq=0 out_seq=0 2024-05-09T08:53:47.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.393+0000 7f2bb2ffd640 10 client.16617 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:47.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.395+0000 7f2bb2ffd640 1 -- 172.21.15.98:0/1650634728 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55794abbb780 con 0x55794accf950 2024-05-09T08:53:47.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.395+0000 7f2bb2ffd640 10 client.16617 handle_command_reply: tid=0 2024-05-09T08:53:47.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.396+0000 7f2bdcb77740 1 client.16617 shutdown 2024-05-09T08:53:47.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.398+0000 7f2bdcb77740 1 -- 172.21.15.98:0/1650634728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4eba0 msgr2=0x55794acc8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:47.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.398+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/1650634728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4eba0 0x55794acc8470 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x55794ac5bd30 tx=0x7f2bc4031400 comp rx=0 tx=0).stop 2024-05-09T08:53:47.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.398+0000 7f2bdcb77740 1 -- 172.21.15.98:0/1650634728 shutdown_connections 2024-05-09T08:53:47.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.398+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/1650634728 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55794accf950 0x55794acd1d40 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.398+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/1650634728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55794ac59270 0x55794ac5c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.398+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/1650634728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55794ac4eba0 0x55794acc8470 secure :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0x55794ac5bd30 tx=0x7f2bc4031400 comp rx=0 tx=0).stop 2024-05-09T08:53:47.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.398+0000 7f2bdcb77740 1 --2- 172.21.15.98:0/1650634728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55794ac4e280 0x55794acc7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.398+0000 7f2bdcb77740 1 -- 172.21.15.98:0/1650634728 >> 172.21.15.98:0/1650634728 conn(0x55794ac4a5a0 msgr2=0x55794ac56150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:47.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.399+0000 7f2bb2ffd640 0 client.16617 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:47.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.400+0000 7f2bdcb77740 1 -- 172.21.15.98:0/1650634728 shutdown_connections 2024-05-09T08:53:47.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.403+0000 7f2bdcb77740 1 -- 172.21.15.98:0/1650634728 wait complete. 2024-05-09T08:53:47.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.404+0000 7f2bdcb77740 20 client.16617 trim_cache size 0 max 16384 2024-05-09T08:53:47.412 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:53:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "scrub active (0 inodes in the stack)", 2024-05-09T08:53:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:53:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "f0c1805e-17ba-45be-b798-711a853148cc": { 2024-05-09T08:53:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:53:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "f0c1805e-17ba-45be-b798-711a853148cc", 2024-05-09T08:53:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:53:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:53:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:53:47.413 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:53:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.419+0000 7f2bd9d96640 1 -- 172.21.15.98:0/1824873586 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2bbc0a2e80 msgr2=0x7f2bbc0a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.419+0000 7f2bd9d96640 1 --2- 172.21.15.98:0/1824873586 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2bbc0a2e80 0x7f2bbc0a5340 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc098b70 tx=0x7f2bcc009f90 comp rx=0 tx=0).stop 2024-05-09T08:53:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.419+0000 7f2bd9d96640 1 -- 172.21.15.98:0/1824873586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bd411b010 msgr2=0x7f2bd41ab8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.419+0000 7f2bd9d96640 1 --2- 172.21.15.98:0/1824873586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bd411b010 0x7f2bd41ab8a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc00be20 tx=0x7f2bcc00bf10 comp rx=0 tx=0).stop 2024-05-09T08:53:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.419+0000 7f2bd9d96640 1 -- 172.21.15.98:0/1824873586 shutdown_connections 2024-05-09T08:53:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.419+0000 7f2bd9d96640 1 --2- 172.21.15.98:0/1824873586 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2bbc0a2e80 0x7f2bbc0a5340 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.419+0000 7f2bd9d96640 1 --2- 172.21.15.98:0/1824873586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bd411b010 0x7f2bd41ab8a0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.419+0000 7f2bd9d96640 1 --2- 172.21.15.98:0/1824873586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bd4110080 0x7f2bd41a7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.419+0000 7f2bd9d96640 1 --2- 172.21.15.98:0/1824873586 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bd410e0b0 0x7f2bd41a6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.419+0000 7f2bd9d96640 1 -- 172.21.15.98:0/1824873586 >> 172.21.15.98:0/1824873586 conn(0x7f2bd406c940 msgr2=0x7f2bd4072330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:47.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.423+0000 7f2bd9d96640 1 -- 172.21.15.98:0/1824873586 shutdown_connections 2024-05-09T08:53:47.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:53:47.423+0000 7f2bd9d96640 1 -- 172.21.15.98:0/1824873586 wait complete. 2024-05-09T08:53:47.479 INFO:tasks.cephfs.filesystem:scrub status for tag:f0c1805e-17ba-45be-b798-711a853148cc - {'path': '/', 'tag': 'f0c1805e-17ba-45be-b798-711a853148cc', 'options': 'recursive,force'} 2024-05-09T08:53:47.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:47 smithi098 ceph-mon[30092]: pgmap v97: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 870 MiB/s wr, 2.13k op/s 2024-05-09T08:53:47.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:47 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:53:47.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:47 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:53:47.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:47 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:53:47.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:47 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:53:47.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:47 smithi105 ceph-mon[34498]: pgmap v97: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 870 MiB/s wr, 2.13k op/s 2024-05-09T08:53:47.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:47 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:53:47.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:47 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:53:47.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:47 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:53:47.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:47 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:53:48.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:47 smithi142 ceph-mon[34625]: pgmap v97: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 870 MiB/s wr, 2.13k op/s 2024-05-09T08:53:48.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:47 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:53:48.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:47 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:53:48.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:47 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:53:48.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:47 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:53:49.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:49 smithi105 ceph-mon[34498]: pgmap v98: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 722 MiB/s wr, 1.76k op/s 2024-05-09T08:53:49.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:49 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:53:49.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:49 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:53:49.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:49 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:53:50.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:49 smithi098 ceph-mon[30092]: pgmap v98: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 722 MiB/s wr, 1.76k op/s 2024-05-09T08:53:50.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:49 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:53:50.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:49 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:53:50.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:49 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:53:50.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:49 smithi142 ceph-mon[34625]: pgmap v98: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 722 MiB/s wr, 1.76k op/s 2024-05-09T08:53:50.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:49 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:53:50.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:49 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:53:50.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:49 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:53:51.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:51 smithi105 ceph-mon[34498]: pgmap v99: 129 pgs: 129 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 907 MiB/s wr, 2.50k op/s 2024-05-09T08:53:52.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:51 smithi098 ceph-mon[30092]: pgmap v99: 129 pgs: 129 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 907 MiB/s wr, 2.50k op/s 2024-05-09T08:53:52.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:51 smithi142 ceph-mon[34625]: pgmap v99: 129 pgs: 129 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 907 MiB/s wr, 2.50k op/s 2024-05-09T08:53:53.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:53 smithi105 ceph-mon[34498]: pgmap v100: 129 pgs: 129 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 663 MiB/s wr, 1.84k op/s 2024-05-09T08:53:54.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:53 smithi098 ceph-mon[30092]: pgmap v100: 129 pgs: 129 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 663 MiB/s wr, 1.84k op/s 2024-05-09T08:53:54.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:53 smithi142 ceph-mon[34625]: pgmap v100: 129 pgs: 129 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 663 MiB/s wr, 1.84k op/s 2024-05-09T08:53:55.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.629+0000 7f093d74c640 1 -- 172.21.15.98:0/1927626140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093811b010 msgr2=0x7f093811d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:55.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.629+0000 7f093d74c640 1 --2- 172.21.15.98:0/1927626140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093811b010 0x7f093811d400 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f092c01a940 tx=0x7f092c03b520 comp rx=0 tx=0).stop 2024-05-09T08:53:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.629+0000 7f093d74c640 1 -- 172.21.15.98:0/1927626140 shutdown_connections 2024-05-09T08:53:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.629+0000 7f093d74c640 1 --2- 172.21.15.98:0/1927626140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093811b010 0x7f093811d400 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.629+0000 7f093d74c640 1 --2- 172.21.15.98:0/1927626140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0938110400 0x7f0938108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.629+0000 7f093d74c640 1 --2- 172.21.15.98:0/1927626140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938110020 0x7f0938108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.629+0000 7f093d74c640 1 -- 172.21.15.98:0/1927626140 >> 172.21.15.98:0/1927626140 conn(0x7f093806c900 msgr2=0x7f093806cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:55.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.640+0000 7f093d74c640 1 -- 172.21.15.98:0/1927626140 shutdown_connections 2024-05-09T08:53:55.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.640+0000 7f093d74c640 1 -- 172.21.15.98:0/1927626140 wait complete. 2024-05-09T08:53:55.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.641+0000 7f093d74c640 1 Processor -- start 2024-05-09T08:53:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.641+0000 7f093d74c640 1 -- start start 2024-05-09T08:53:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.642+0000 7f093d74c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938110020 0x7f093811a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.642+0000 7f093d74c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093811b010 0x7f09381166e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.642+0000 7f093d74c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0938116c20 0x7f09381170d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.642+0000 7f093d74c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09381131d0 con 0x7f093811b010 2024-05-09T08:53:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.642+0000 7f093d74c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0938113050 con 0x7f0938110020 2024-05-09T08:53:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.642+0000 7f093d74c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0938113350 con 0x7f0938116c20 2024-05-09T08:53:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.643+0000 7f09377fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0938116c20 0x7f09381170d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.643+0000 7f09377fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0938116c20 0x7f09381170d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:43886/0 (socket says 172.21.15.98:43886) 2024-05-09T08:53:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.643+0000 7f09377fe640 1 -- 172.21.15.98:0/637869159 learned_addr learned my addr 172.21.15.98:0/637869159 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.644+0000 7f0936ffd640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938110020 0x7f093811a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.644+0000 7f09367fc640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093811b010 0x7f09381166e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.645 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.646+0000 7f09367fc640 1 -- 172.21.15.98:0/637869159 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0938116c20 msgr2=0x7f09381170d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:55.645 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.646+0000 7f09367fc640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0938116c20 0x7f09381170d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.646+0000 7f09367fc640 1 -- 172.21.15.98:0/637869159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938110020 msgr2=0x7f093811a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:55.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.647+0000 7f09367fc640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938110020 0x7f093811a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.647+0000 7f09367fc640 1 -- 172.21.15.98:0/637869159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09381aae00 con 0x7f093811b010 2024-05-09T08:53:55.647 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.647+0000 7f0936ffd640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938110020 0x7f093811a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:53:55.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.652+0000 7f09367fc640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093811b010 0x7f09381166e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f0930005fd0 tx=0x7f093000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:55.659 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.660+0000 7f091ffff640 1 -- 172.21.15.98:0/637869159 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0930010430 con 0x7f093811b010 2024-05-09T08:53:55.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.660+0000 7f093d74c640 1 -- 172.21.15.98:0/637869159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09381aafd0 con 0x7f093811b010 2024-05-09T08:53:55.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.660+0000 7f093d74c640 1 -- 172.21.15.98:0/637869159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09381ab4f0 con 0x7f093811b010 2024-05-09T08:53:55.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.660+0000 7f091ffff640 1 -- 172.21.15.98:0/637869159 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0930010ab0 con 0x7f093811b010 2024-05-09T08:53:55.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.661+0000 7f091ffff640 1 -- 172.21.15.98:0/637869159 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0930017b70 con 0x7f093811b010 2024-05-09T08:53:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.669+0000 7f0940458740 1 Processor -- start 2024-05-09T08:53:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.670+0000 7f0940458740 1 -- start start 2024-05-09T08:53:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.670+0000 7f0940458740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fcc450 0x55e330fcc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.670+0000 7f0940458740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 0x55e330fd70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.670+0000 7f0940458740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fd7630 0x55e330fd9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.670+0000 7f0940458740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e330f31340 con 0x55e330fd7630 2024-05-09T08:53:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.670+0000 7f0940458740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e330ed9290 con 0x55e330fccd70 2024-05-09T08:53:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.670+0000 7f0940458740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e330f2d300 con 0x55e330fcc450 2024-05-09T08:53:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.670+0000 7f09377fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fcc450 0x55e330fcc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.670+0000 7f09377fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fcc450 0x55e330fcc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:43892/0 (socket says 172.21.15.98:43892) 2024-05-09T08:53:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.671+0000 7f09377fe640 1 -- 172.21.15.98:0/2965750963 learned_addr learned my addr 172.21.15.98:0/2965750963 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.671+0000 7f09367fc640 1 --2- 172.21.15.98:0/2965750963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fd7630 0x55e330fd9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.671+0000 7f0936ffd640 1 --2- 172.21.15.98:0/2965750963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 0x55e330fd70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.672+0000 7f091ffff640 1 -- 172.21.15.98:0/637869159 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0930017d10 con 0x7f093811b010 2024-05-09T08:53:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.672+0000 7f09377fe640 1 -- 172.21.15.98:0/2965750963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 msgr2=0x55e330fd70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.672+0000 7f09377fe640 1 --2- 172.21.15.98:0/2965750963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 0x55e330fd70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.672+0000 7f09377fe640 1 -- 172.21.15.98:0/2965750963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fd7630 msgr2=0x55e330fd9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.672+0000 7f09377fe640 1 --2- 172.21.15.98:0/2965750963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fd7630 0x55e330fd9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.672+0000 7f09377fe640 1 -- 172.21.15.98:0/2965750963 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e330efd8e0 con 0x55e330fcc450 2024-05-09T08:53:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.673+0000 7f09377fe640 1 --2- 172.21.15.98:0/2965750963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fcc450 0x55e330fcc830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f092c005e70 tx=0x7f092c005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3d6829960d96498 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.673+0000 7f091d7fa640 1 -- 172.21.15.98:0/2965750963 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c0046c0 con 0x55e330fcc450 2024-05-09T08:53:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.673+0000 7f091d7fa640 1 -- 172.21.15.98:0/2965750963 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f092c011070 con 0x55e330fcc450 2024-05-09T08:53:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.673+0000 7f091d7fa640 1 -- 172.21.15.98:0/2965750963 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c00c560 con 0x55e330fcc450 2024-05-09T08:53:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.674+0000 7f0940458740 1 -- 172.21.15.98:0/2965750963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fcc450 msgr2=0x55e330fcc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.675+0000 7f0940458740 1 --2- 172.21.15.98:0/2965750963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fcc450 0x55e330fcc830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f092c005e70 tx=0x7f092c005be0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.675+0000 7f0940458740 1 -- 172.21.15.98:0/2965750963 shutdown_connections 2024-05-09T08:53:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.675+0000 7f0940458740 1 --2- 172.21.15.98:0/2965750963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fd7630 0x55e330fd9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.675+0000 7f0940458740 1 --2- 172.21.15.98:0/2965750963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 0x55e330fd70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.675+0000 7f0940458740 1 --2- 172.21.15.98:0/2965750963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fcc450 0x55e330fcc830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.675+0000 7f0940458740 1 -- 172.21.15.98:0/2965750963 >> 172.21.15.98:0/2965750963 conn(0x55e330fc8960 msgr2=0x55e330fc8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.677+0000 7f091ffff640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09200a2ed0 0x7f09200a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.677+0000 7f091ffff640 1 -- 172.21.15.98:0/637869159 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f09300cff70 con 0x7f093811b010 2024-05-09T08:53:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.682+0000 7f0936ffd640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09200a2ed0 0x7f09200a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.683+0000 7f0936ffd640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09200a2ed0 0x7f09200a5390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55e330fcd9f0 tx=0x7f0928005f90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.685+0000 7f0940458740 1 -- 172.21.15.98:0/2965750963 shutdown_connections 2024-05-09T08:53:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.685+0000 7f0940458740 1 -- 172.21.15.98:0/2965750963 wait complete. 2024-05-09T08:53:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.685+0000 7f0940458740 1 -- start start 2024-05-09T08:53:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.685+0000 7f0940458740 1 Processor -- start 2024-05-09T08:53:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.686+0000 7f0940458740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fcc450 0x55e330fd45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.686+0000 7f0940458740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 0x55e330fd4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.686+0000 7f0940458740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fd7630 0x55e330fd5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.686+0000 7f0940458740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e330fda200 con 0x55e330fd7630 2024-05-09T08:53:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.686+0000 7f0940458740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e330eec790 con 0x55e330fccd70 2024-05-09T08:53:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.686+0000 7f0940458740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e330fda380 con 0x55e330fcc450 2024-05-09T08:53:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.686+0000 7f0940458740 10 client.0 _ll_register_callbacks cb 0x55e330f27a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.690+0000 7f09367fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 0x55e330fd4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.690+0000 7f0936ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fcc450 0x55e330fd45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.690+0000 7f09377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fd7630 0x55e330fd5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.690+0000 7f09377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fd7630 0x55e330fd5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49468/0 (socket says 172.21.15.98:49468) 2024-05-09T08:53:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.690+0000 7f09377fe640 1 -- 172.21.15.98:0/644483287 learned_addr learned my addr 172.21.15.98:0/644483287 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:55.692 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.692+0000 7f09377fe640 1 -- 172.21.15.98:0/644483287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fcc450 msgr2=0x55e330fd45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.692+0000 7f09377fe640 1 --2- 172.21.15.98:0/644483287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fcc450 0x55e330fd45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.692+0000 7f09377fe640 1 -- 172.21.15.98:0/644483287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 msgr2=0x55e330fd4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.692+0000 7f09377fe640 1 --2- 172.21.15.98:0/644483287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 0x55e330fd4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.693+0000 7f09377fe640 1 -- 172.21.15.98:0/644483287 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e330c45210 con 0x55e330fd7630 2024-05-09T08:53:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.696+0000 7f09377fe640 1 --2- 172.21.15.98:0/644483287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fd7630 0x55e330fd5070 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f092c005ea0 tx=0x7f092c041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed6600c4d1e4cd98 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.696+0000 7f091aff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.699+0000 7f091aff5640 1 -- 172.21.15.98:0/644483287 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c075060 con 0x55e330fd7630 2024-05-09T08:53:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.699+0000 7f0940458740 1 -- 172.21.15.98:0/644483287 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f092c011070 con 0x55e330fd7630 2024-05-09T08:53:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.699+0000 7f091aff5640 1 -- 172.21.15.98:0/644483287 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f092c00c880 con 0x55e330fd7630 2024-05-09T08:53:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.702+0000 7f091aff5640 1 -- 172.21.15.98:0/644483287 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f092c07c050 con 0x55e330fd7630 2024-05-09T08:53:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.703+0000 7f0940458740 10 client.16629 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.703+0000 7f0940458740 1 -- 172.21.15.98:0/644483287 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e330d11620 con 0x55e330fd7630 2024-05-09T08:53:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.705+0000 7f091aff5640 1 -- 172.21.15.98:0/644483287 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f092c011070 con 0x55e330fd7630 2024-05-09T08:53:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.707+0000 7f091aff5640 1 -- 172.21.15.98:0/644483287 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f092c042b30 con 0x55e330fd7630 2024-05-09T08:53:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.707+0000 7f0940458740 10 client.16629 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.707+0000 7f0940458740 10 client.16629 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.707+0000 7f0940458740 20 client.16629 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.707+0000 7f0940458740 1 --2- 172.21.15.98:0/644483287 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e330fe72e0 0x55e330fe9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.707+0000 7f0940458740 4 client.16629 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:53:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.707+0000 7f0940458740 1 -- 172.21.15.98:0/644483287 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e330f3ea70 con 0x55e330fe72e0 2024-05-09T08:53:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.708+0000 7f0936ffd640 1 --2- 172.21.15.98:0/644483287 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e330fe72e0 0x55e330fe9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.710+0000 7f0936ffd640 1 --2- 172.21.15.98:0/644483287 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e330fe72e0 0x55e330fe9720 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18a1e2fd75f3b0b2 server_cookie=e4c37fbe2ae4e3b6 in_seq=0 out_seq=0 2024-05-09T08:53:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.710+0000 7f091aff5640 10 client.16629 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:55.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.713+0000 7f091aff5640 1 -- 172.21.15.98:0/644483287 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e330f3ea70 con 0x55e330fe72e0 2024-05-09T08:53:55.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.713+0000 7f091aff5640 10 client.16629 handle_command_reply: tid=0 2024-05-09T08:53:55.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.714+0000 7f0940458740 1 client.16629 shutdown 2024-05-09T08:53:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.714+0000 7f0940458740 1 -- 172.21.15.98:0/644483287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fd7630 msgr2=0x55e330fd5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.714+0000 7f0940458740 1 --2- 172.21.15.98:0/644483287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fd7630 0x55e330fd5070 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f092c005ea0 tx=0x7f092c041000 comp rx=0 tx=0).stop 2024-05-09T08:53:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.714+0000 7f0940458740 1 -- 172.21.15.98:0/644483287 shutdown_connections 2024-05-09T08:53:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.714+0000 7f0940458740 1 --2- 172.21.15.98:0/644483287 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e330fe72e0 0x55e330fe9720 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.714+0000 7f0940458740 1 --2- 172.21.15.98:0/644483287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fd7630 0x55e330fd5070 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.714+0000 7f0940458740 1 --2- 172.21.15.98:0/644483287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 0x55e330fd4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.714+0000 7f0940458740 1 --2- 172.21.15.98:0/644483287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fcc450 0x55e330fd45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.714+0000 7f0940458740 1 -- 172.21.15.98:0/644483287 >> 172.21.15.98:0/644483287 conn(0x55e330fc8960 msgr2=0x55e330fce690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.714+0000 7f091aff5640 0 client.16629 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.714+0000 7f0940458740 1 -- 172.21.15.98:0/644483287 shutdown_connections 2024-05-09T08:53:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.715+0000 7f0940458740 1 -- 172.21.15.98:0/644483287 wait complete. 2024-05-09T08:53:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.715+0000 7f0940458740 20 client.16629 trim_cache size 0 max 16384 2024-05-09T08:53:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.753+0000 7f0940458740 1 Processor -- start 2024-05-09T08:53:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.753+0000 7f0940458740 1 -- start start 2024-05-09T08:53:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.754+0000 7f0940458740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fcc450 0x55e330fe30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.754+0000 7f0940458740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fccd70 0x55e330fc7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.754+0000 7f0940458740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fd7630 0x55e330feeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.754+0000 7f09367fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fccd70 0x55e330fc7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.754+0000 7f09367fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fccd70 0x55e330fc7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49480/0 (socket says 172.21.15.98:49480) 2024-05-09T08:53:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.754+0000 7f09367fc640 1 -- 172.21.15.98:0/4176689190 learned_addr learned my addr 172.21.15.98:0/4176689190 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.754+0000 7f0940458740 1 -- 172.21.15.98:0/4176689190 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f092400ff00 con 0x55e330fccd70 2024-05-09T08:53:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.754+0000 7f0940458740 1 -- 172.21.15.98:0/4176689190 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0924011a40 con 0x55e330fcc450 2024-05-09T08:53:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.754+0000 7f0940458740 1 -- 172.21.15.98:0/4176689190 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f09240105d0 con 0x55e330fd7630 2024-05-09T08:53:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.754+0000 7f0936ffd640 1 --2- 172.21.15.98:0/4176689190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fcc450 0x55e330fe30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.755+0000 7f09367fc640 1 -- 172.21.15.98:0/4176689190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fd7630 msgr2=0x55e330feeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.755+0000 7f09367fc640 1 --2- 172.21.15.98:0/4176689190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fd7630 0x55e330feeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.755+0000 7f09367fc640 1 -- 172.21.15.98:0/4176689190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fcc450 msgr2=0x55e330fe30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.755+0000 7f09367fc640 1 --2- 172.21.15.98:0/4176689190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fcc450 0x55e330fe30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.755+0000 7f09367fc640 1 -- 172.21.15.98:0/4176689190 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e330fcc210 con 0x55e330fccd70 2024-05-09T08:53:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.755+0000 7f09367fc640 1 --2- 172.21.15.98:0/4176689190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fccd70 0x55e330fc7880 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f09300984d0 tx=0x7f09300d3790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a66af37655827a00 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.756+0000 7f091cff9640 1 -- 172.21.15.98:0/4176689190 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093009b9d0 con 0x55e330fccd70 2024-05-09T08:53:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.756+0000 7f091cff9640 1 -- 172.21.15.98:0/4176689190 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f093009c770 con 0x55e330fccd70 2024-05-09T08:53:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.756+0000 7f091cff9640 1 -- 172.21.15.98:0/4176689190 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0930010c60 con 0x55e330fccd70 2024-05-09T08:53:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.757+0000 7f0940458740 1 -- 172.21.15.98:0/4176689190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fccd70 msgr2=0x55e330fc7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:53:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.757+0000 7f0940458740 1 --2- 172.21.15.98:0/4176689190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fccd70 0x55e330fc7880 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f09300984d0 tx=0x7f09300d3790 comp rx=0 tx=0).stop 2024-05-09T08:53:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.759+0000 7f0940458740 1 -- 172.21.15.98:0/4176689190 shutdown_connections 2024-05-09T08:53:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.759+0000 7f0940458740 1 --2- 172.21.15.98:0/4176689190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fd7630 0x55e330feeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.759+0000 7f0940458740 1 --2- 172.21.15.98:0/4176689190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fccd70 0x55e330fc7880 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.759+0000 7f0940458740 1 --2- 172.21.15.98:0/4176689190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fcc450 0x55e330fe30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.759+0000 7f0940458740 1 -- 172.21.15.98:0/4176689190 >> 172.21.15.98:0/4176689190 conn(0x55e330fc8960 msgr2=0x55e330fc7170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:53:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.760+0000 7f0940458740 1 -- 172.21.15.98:0/4176689190 shutdown_connections 2024-05-09T08:53:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.760+0000 7f0940458740 1 -- 172.21.15.98:0/4176689190 wait complete. 2024-05-09T08:53:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.760+0000 7f0940458740 1 -- start start 2024-05-09T08:53:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.761+0000 7f0940458740 1 Processor -- start 2024-05-09T08:53:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.761+0000 7f0940458740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fcc450 0x55e330fda7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.761+0000 7f0940458740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 0x55e330fdacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.761+0000 7f0940458740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fd7630 0x55e33104b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.761+0000 7f0940458740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0924011370 con 0x55e330fcc450 2024-05-09T08:53:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.761+0000 7f0940458740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0924010ca0 con 0x55e330fccd70 2024-05-09T08:53:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.761+0000 7f0940458740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f092400df90 con 0x55e330fd7630 2024-05-09T08:53:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.761+0000 7f0940458740 10 client.0 _ll_register_callbacks cb 0x7f09280095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:53:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.763+0000 7f09367fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 0x55e330fdacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.763+0000 7f0936ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fcc450 0x55e330fda7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.763+0000 7f0936ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fcc450 0x55e330fda7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49484/0 (socket says 172.21.15.98:49484) 2024-05-09T08:53:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.763+0000 7f0936ffd640 1 -- 172.21.15.98:0/61798569 learned_addr learned my addr 172.21.15.98:0/61798569 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:53:55.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.764+0000 7f09377fe640 1 --2- 172.21.15.98:0/61798569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fd7630 0x55e33104b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.765+0000 7f0936ffd640 1 -- 172.21.15.98:0/61798569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fd7630 msgr2=0x55e33104b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:55.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.765+0000 7f0936ffd640 1 --2- 172.21.15.98:0/61798569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fd7630 0x55e33104b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.765+0000 7f0936ffd640 1 -- 172.21.15.98:0/61798569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 msgr2=0x55e330fdacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:53:55.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.765+0000 7f0936ffd640 1 --2- 172.21.15.98:0/61798569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 0x55e330fdacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:53:55.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.765+0000 7f0936ffd640 1 -- 172.21.15.98:0/61798569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e33104bd90 con 0x55e330fcc450 2024-05-09T08:53:55.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.766+0000 7f0936ffd640 1 --2- 172.21.15.98:0/61798569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fcc450 0x55e330fda7b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f0928009f90 tx=0x7f09280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913bf80eab478035 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:53:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.769+0000 7f091aff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:53:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.769+0000 7f091aff5640 1 -- 172.21.15.98:0/61798569 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0928032470 con 0x55e330fcc450 2024-05-09T08:53:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.769+0000 7f091aff5640 1 -- 172.21.15.98:0/61798569 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9474+0+0 (secure 0 0 0) 0x7f0928032610 con 0x55e330fcc450 2024-05-09T08:53:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.769+0000 7f0940458740 1 -- 172.21.15.98:0/61798569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f093009c770 con 0x55e330fcc450 2024-05-09T08:53:55.771 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.772+0000 7f091aff5640 1 -- 172.21.15.98:0/61798569 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f092803b050 con 0x55e330fcc450 2024-05-09T08:53:55.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.773+0000 7f0940458740 10 client.16635 fetch_fsmap learned FSMap version 31 2024-05-09T08:53:55.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.773+0000 7f0940458740 1 -- 172.21.15.98:0/61798569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e33104d030 con 0x55e330fcc450 2024-05-09T08:53:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.774+0000 7f091aff5640 1 -- 172.21.15.98:0/61798569 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f092802d070 con 0x55e330fcc450 2024-05-09T08:53:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.775+0000 7f091aff5640 1 -- 172.21.15.98:0/61798569 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f092801a040 con 0x55e330fcc450 2024-05-09T08:53:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.776+0000 7f0940458740 10 client.16635 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:53:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.776+0000 7f0940458740 10 client.16635 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:53:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.776+0000 7f0940458740 20 client.16635 populate_metadata read hostname 'smithi098' 2024-05-09T08:53:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.776+0000 7f0940458740 1 --2- 172.21.15.98:0/61798569 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e33104d730 0x55e33104fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:53:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.776+0000 7f0940458740 4 client.16635 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:53:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.776+0000 7f0940458740 1 -- 172.21.15.98:0/61798569 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f093009c770 con 0x55e33104d730 2024-05-09T08:53:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.777+0000 7f09367fc640 1 --2- 172.21.15.98:0/61798569 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e33104d730 0x55e33104fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:53:55.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.780+0000 7f09367fc640 1 --2- 172.21.15.98:0/61798569 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e33104d730 0x55e33104fb50 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6335ea2f4498c33 server_cookie=152a0af621c36478 in_seq=0 out_seq=0 2024-05-09T08:53:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:53:55.780+0000 7f091aff5640 10 client.16635 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:53:56.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:55 smithi098 ceph-mon[30092]: pgmap v101: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 735 MiB/s wr, 1.84k op/s 2024-05-09T08:53:56.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:55 smithi142 ceph-mon[34625]: pgmap v101: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 735 MiB/s wr, 1.84k op/s 2024-05-09T08:53:56.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:55 smithi105 ceph-mon[34498]: pgmap v101: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 735 MiB/s wr, 1.84k op/s 2024-05-09T08:53:58.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:57 smithi098 ceph-mon[30092]: pgmap v102: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 865 MiB/s wr, 1.85k op/s 2024-05-09T08:53:58.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:57 smithi142 ceph-mon[34625]: pgmap v102: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 865 MiB/s wr, 1.85k op/s 2024-05-09T08:53:58.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:57 smithi105 ceph-mon[34498]: pgmap v102: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 865 MiB/s wr, 1.85k op/s 2024-05-09T08:54:00.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:53:59 smithi098 ceph-mon[30092]: pgmap v103: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 725 MiB/s wr, 1.51k op/s 2024-05-09T08:54:00.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:53:59 smithi142 ceph-mon[34625]: pgmap v103: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 725 MiB/s wr, 1.51k op/s 2024-05-09T08:54:00.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:53:59 smithi105 ceph-mon[34498]: pgmap v103: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 725 MiB/s wr, 1.51k op/s 2024-05-09T08:54:02.039 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:01 smithi105 ceph-mon[34498]: pgmap v104: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 880 MiB/s wr, 1.67k op/s 2024-05-09T08:54:02.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:01 smithi098 ceph-mon[30092]: pgmap v104: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 880 MiB/s wr, 1.67k op/s 2024-05-09T08:54:02.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:01 smithi142 ceph-mon[34625]: pgmap v104: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 880 MiB/s wr, 1.67k op/s 2024-05-09T08:54:02.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.879+0000 7f091aff5640 1 -- 172.21.15.98:0/61798569 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f093009c770 con 0x55e33104d730 2024-05-09T08:54:02.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.879+0000 7f091aff5640 10 client.16635 handle_command_reply: tid=0 2024-05-09T08:54:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.879+0000 7f0940458740 1 client.16635 shutdown 2024-05-09T08:54:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.879+0000 7f0940458740 1 -- 172.21.15.98:0/61798569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fcc450 msgr2=0x55e330fda7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.879+0000 7f0940458740 1 --2- 172.21.15.98:0/61798569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fcc450 0x55e330fda7b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f0928009f90 tx=0x7f09280046c0 comp rx=0 tx=0).stop 2024-05-09T08:54:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.879+0000 7f0940458740 1 -- 172.21.15.98:0/61798569 shutdown_connections 2024-05-09T08:54:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.879+0000 7f0940458740 1 --2- 172.21.15.98:0/61798569 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e33104d730 0x55e33104fb50 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.879+0000 7f0940458740 1 --2- 172.21.15.98:0/61798569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e330fd7630 0x55e33104b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.879+0000 7f0940458740 1 --2- 172.21.15.98:0/61798569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e330fccd70 0x55e330fdacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.879+0000 7f091aff5640 0 client.16635 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.879+0000 7f0940458740 1 --2- 172.21.15.98:0/61798569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e330fcc450 0x55e330fda7b0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.879+0000 7f0940458740 1 -- 172.21.15.98:0/61798569 >> 172.21.15.98:0/61798569 conn(0x55e330fc8960 msgr2=0x55e330fcde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.879+0000 7f0940458740 1 -- 172.21.15.98:0/61798569 shutdown_connections 2024-05-09T08:54:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.880+0000 7f0940458740 1 -- 172.21.15.98:0/61798569 wait complete. 2024-05-09T08:54:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.880+0000 7f0940458740 20 client.16635 trim_cache size 0 max 16384 2024-05-09T08:54:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:54:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:54:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:54:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:54:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.880+0000 7f093d74c640 1 -- 172.21.15.98:0/637869159 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09200a2ed0 msgr2=0x7f09200a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.880+0000 7f093d74c640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09200a2ed0 0x7f09200a5390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55e330fcd9f0 tx=0x7f0928005f90 comp rx=0 tx=0).stop 2024-05-09T08:54:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.880+0000 7f093d74c640 1 -- 172.21.15.98:0/637869159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093811b010 msgr2=0x7f09381166e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.880+0000 7f093d74c640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093811b010 0x7f09381166e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f0930005fd0 tx=0x7f093000ef90 comp rx=0 tx=0).stop 2024-05-09T08:54:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.880+0000 7f093d74c640 1 -- 172.21.15.98:0/637869159 shutdown_connections 2024-05-09T08:54:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.880+0000 7f093d74c640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09200a2ed0 0x7f09200a5390 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.880+0000 7f093d74c640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0938116c20 0x7f09381170d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.880+0000 7f093d74c640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093811b010 0x7f09381166e0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.880+0000 7f093d74c640 1 --2- 172.21.15.98:0/637869159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938110020 0x7f093811a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.880+0000 7f093d74c640 1 -- 172.21.15.98:0/637869159 >> 172.21.15.98:0/637869159 conn(0x7f093806c900 msgr2=0x7f093811b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.880+0000 7f093d74c640 1 -- 172.21.15.98:0/637869159 shutdown_connections 2024-05-09T08:54:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:02.880+0000 7f093d74c640 1 -- 172.21.15.98:0/637869159 wait complete. 2024-05-09T08:54:03.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:03 smithi105 ceph-mon[34498]: pgmap v105: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 750 MiB/s wr, 967 op/s 2024-05-09T08:54:04.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:03 smithi098 ceph-mon[30092]: pgmap v105: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 750 MiB/s wr, 967 op/s 2024-05-09T08:54:04.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:03 smithi142 ceph-mon[34625]: pgmap v105: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 750 MiB/s wr, 967 op/s 2024-05-09T08:54:05.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:04 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:54:05.118 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:04 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:54:05.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:04 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:54:06.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:05 smithi098 ceph-mon[30092]: pgmap v106: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 786 MiB/s wr, 987 op/s 2024-05-09T08:54:06.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:05 smithi142 ceph-mon[34625]: pgmap v106: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 786 MiB/s wr, 987 op/s 2024-05-09T08:54:06.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:05 smithi105 ceph-mon[34498]: pgmap v106: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 786 MiB/s wr, 987 op/s 2024-05-09T08:54:06.588 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 08:54:06 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d[71020]: ignoring --setuser ceph since I am not root 2024-05-09T08:54:06.588 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 08:54:06 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d[71020]: ignoring --setgroup ceph since I am not root 2024-05-09T08:54:06.589 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 08:54:06 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d[71020]: starting mds.d at 2024-05-09T08:54:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:06 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:54:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:06 smithi098 ceph-mon[30092]: daemon mds.d restarted 2024-05-09T08:54:07.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:06 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:54:07.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:06 smithi142 ceph-mon[34625]: daemon mds.d restarted 2024-05-09T08:54:07.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:06 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:54:07.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:06 smithi105 ceph-mon[34498]: daemon mds.d restarted 2024-05-09T08:54:08.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:07 smithi098 ceph-mon[30092]: pgmap v107: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 908 MiB/s wr, 1.22k op/s 2024-05-09T08:54:08.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:07 smithi098 ceph-mon[30092]: osdmap e96: 12 total, 12 up, 12 in 2024-05-09T08:54:08.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:07 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.98:6836/3349671892,v1:172.21.15.98:6837/3349671892] up:boot 2024-05-09T08:54:08.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:07 smithi098 ceph-mon[30092]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:54:08.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:07 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:54:08.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:07 smithi098 ceph-mon[30092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:54:08.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:07 smithi142 ceph-mon[34625]: pgmap v107: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 908 MiB/s wr, 1.22k op/s 2024-05-09T08:54:08.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:07 smithi142 ceph-mon[34625]: osdmap e96: 12 total, 12 up, 12 in 2024-05-09T08:54:08.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:07 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.98:6836/3349671892,v1:172.21.15.98:6837/3349671892] up:boot 2024-05-09T08:54:08.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:07 smithi142 ceph-mon[34625]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:54:08.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:07 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:54:08.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:07 smithi142 ceph-mon[34625]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:54:08.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:07 smithi105 ceph-mon[34498]: pgmap v107: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 908 MiB/s wr, 1.22k op/s 2024-05-09T08:54:08.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:07 smithi105 ceph-mon[34498]: osdmap e96: 12 total, 12 up, 12 in 2024-05-09T08:54:08.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:07 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.98:6836/3349671892,v1:172.21.15.98:6837/3349671892] up:boot 2024-05-09T08:54:08.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:07 smithi105 ceph-mon[34498]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:54:08.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:07 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:54:08.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:07 smithi105 ceph-mon[34498]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:54:10.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:09 smithi098 ceph-mon[30092]: pgmap v109: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 809 MiB/s wr, 1.31k op/s 2024-05-09T08:54:10.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:09 smithi142 ceph-mon[34625]: pgmap v109: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 809 MiB/s wr, 1.31k op/s 2024-05-09T08:54:10.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:09 smithi105 ceph-mon[34498]: pgmap v109: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 809 MiB/s wr, 1.31k op/s 2024-05-09T08:54:12.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:11 smithi098 ceph-mon[30092]: pgmap v110: 129 pgs: 129 active+clean; 21 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 791 MiB/s wr, 2.34k op/s 2024-05-09T08:54:12.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:11 smithi142 ceph-mon[34625]: pgmap v110: 129 pgs: 129 active+clean; 21 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 791 MiB/s wr, 2.34k op/s 2024-05-09T08:54:12.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:11 smithi105 ceph-mon[34498]: pgmap v110: 129 pgs: 129 active+clean; 21 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 791 MiB/s wr, 2.34k op/s 2024-05-09T08:54:14.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:13 smithi098 ceph-mon[30092]: pgmap v111: 129 pgs: 129 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 746 MiB/s wr, 2.62k op/s 2024-05-09T08:54:14.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:13 smithi142 ceph-mon[34625]: pgmap v111: 129 pgs: 129 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 746 MiB/s wr, 2.62k op/s 2024-05-09T08:54:14.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:13 smithi105 ceph-mon[34498]: pgmap v111: 129 pgs: 129 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 746 MiB/s wr, 2.62k op/s 2024-05-09T08:54:16.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:15 smithi142 ceph-mon[34625]: pgmap v112: 129 pgs: 129 active+clean; 22 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 748 MiB/s wr, 3.04k op/s 2024-05-09T08:54:16.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:15 smithi105 ceph-mon[34498]: pgmap v112: 129 pgs: 129 active+clean; 22 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 748 MiB/s wr, 3.04k op/s 2024-05-09T08:54:16.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:15 smithi098 ceph-mon[30092]: pgmap v112: 129 pgs: 129 active+clean; 22 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 748 MiB/s wr, 3.04k op/s 2024-05-09T08:54:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.656+0000 7fa7cc010640 1 -- 172.21.15.98:0/2679359222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c411b010 msgr2=0x7fa7c411d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.656+0000 7fa7cc010640 1 --2- 172.21.15.98:0/2679359222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c411b010 0x7fa7c411d400 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa7c001ace0 tx=0x7fa7c003ba20 comp rx=0 tx=0).stop 2024-05-09T08:54:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.656+0000 7fa7cc010640 1 -- 172.21.15.98:0/2679359222 shutdown_connections 2024-05-09T08:54:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.656+0000 7fa7cc010640 1 --2- 172.21.15.98:0/2679359222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c411b010 0x7fa7c411d400 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.656+0000 7fa7cc010640 1 --2- 172.21.15.98:0/2679359222 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c4110400 0x7fa7c4108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.656+0000 7fa7cc010640 1 --2- 172.21.15.98:0/2679359222 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7c4110020 0x7fa7c4108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.656+0000 7fa7cc010640 1 -- 172.21.15.98:0/2679359222 >> 172.21.15.98:0/2679359222 conn(0x7fa7c406c900 msgr2=0x7fa7c406cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.662+0000 7fa7cc010640 1 -- 172.21.15.98:0/2679359222 shutdown_connections 2024-05-09T08:54:16.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.665+0000 7fa7cc010640 1 -- 172.21.15.98:0/2679359222 wait complete. 2024-05-09T08:54:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.666+0000 7fa7cc010640 1 Processor -- start 2024-05-09T08:54:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.666+0000 7fa7cc010640 1 -- start start 2024-05-09T08:54:16.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.667+0000 7fa7cc010640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c4110020 0x7fa7c411a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.667+0000 7fa7cc010640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c411b010 0x7fa7c4116940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.667+0000 7fa7cc010640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7c4116e80 0x7fa7c4117330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.667+0000 7fa7cc010640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c4113280 con 0x7fa7c411b010 2024-05-09T08:54:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.667+0000 7fa7cc010640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c4113100 con 0x7fa7c4116e80 2024-05-09T08:54:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.667+0000 7fa7cc010640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c4113400 con 0x7fa7c4110020 2024-05-09T08:54:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.668+0000 7fa7c95ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c411b010 0x7fa7c4116940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.668+0000 7fa7ca5ad640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7c4116e80 0x7fa7c4117330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.669+0000 7fa7ca5ad640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7c4116e80 0x7fa7c4117330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45276/0 (socket says 172.21.15.98:45276) 2024-05-09T08:54:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.669+0000 7fa7ca5ad640 1 -- 172.21.15.98:0/1989227643 learned_addr learned my addr 172.21.15.98:0/1989227643 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.669+0000 7fa7ca5ad640 1 -- 172.21.15.98:0/1989227643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c4110020 msgr2=0x7fa7c411a8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:54:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.669+0000 7fa7ca5ad640 1 --2- 172.21.15.98:0/1989227643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c4110020 0x7fa7c411a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.669+0000 7fa7ca5ad640 1 -- 172.21.15.98:0/1989227643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c411b010 msgr2=0x7fa7c4116940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.669+0000 7fa7ca5ad640 1 --2- 172.21.15.98:0/1989227643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c411b010 0x7fa7c4116940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.670+0000 7fa7ca5ad640 1 -- 172.21.15.98:0/1989227643 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c41aaf30 con 0x7fa7c4116e80 2024-05-09T08:54:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.670+0000 7fa7c95ab640 1 --2- 172.21.15.98:0/1989227643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c411b010 0x7fa7c4116940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:54:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.670+0000 7fa7ca5ad640 1 --2- 172.21.15.98:0/1989227643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7c4116e80 0x7fa7c4117330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa7c003c010 tx=0x7fa7c0003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.672+0000 7fa7baffd640 1 -- 172.21.15.98:0/1989227643 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0044030 con 0x7fa7c4116e80 2024-05-09T08:54:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.672+0000 7fa7baffd640 1 -- 172.21.15.98:0/1989227643 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7c003fdd0 con 0x7fa7c4116e80 2024-05-09T08:54:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.672+0000 7fa7baffd640 1 -- 172.21.15.98:0/1989227643 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c00047f0 con 0x7fa7c4116e80 2024-05-09T08:54:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.672+0000 7fa7cc010640 1 -- 172.21.15.98:0/1989227643 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7c41ab1c0 con 0x7fa7c4116e80 2024-05-09T08:54:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.673+0000 7fa7cc010640 1 -- 172.21.15.98:0/1989227643 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7c41ab660 con 0x7fa7c4116e80 2024-05-09T08:54:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.679+0000 7fa7baffd640 1 -- 172.21.15.98:0/1989227643 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa7c0047070 con 0x7fa7c4116e80 2024-05-09T08:54:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.680+0000 7fa7baffd640 1 --2- 172.21.15.98:0/1989227643 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa7b00a3010 0x7fa7b00a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.681+0000 7fa7baffd640 1 -- 172.21.15.98:0/1989227643 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7fa7c0102690 con 0x7fa7c4116e80 2024-05-09T08:54:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.682+0000 7fa7c9dac640 1 --2- 172.21.15.98:0/1989227643 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa7b00a3010 0x7fa7b00a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.770+0000 7fa7cedf3740 1 Processor -- start 2024-05-09T08:54:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.770+0000 7fa7cedf3740 1 -- start start 2024-05-09T08:54:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.771+0000 7fa7cedf3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 0x56359732f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.771+0000 7fa7cedf3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732fd70 0x56359733a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.771+0000 7fa7cedf3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359733a630 0x56359733ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.771+0000 7fa7cedf3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563597294340 con 0x56359732fd70 2024-05-09T08:54:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.772+0000 7fa7cedf3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56359723c290 con 0x56359733a630 2024-05-09T08:54:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.772+0000 7fa7cedf3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563597290300 con 0x56359732f450 2024-05-09T08:54:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.773+0000 7fa7c95ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732fd70 0x56359733a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.773+0000 7fa7c95ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732fd70 0x56359733a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43034/0 (socket says 172.21.15.98:43034) 2024-05-09T08:54:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.773+0000 7fa7c95ab640 1 -- 172.21.15.98:0/2135172489 learned_addr learned my addr 172.21.15.98:0/2135172489 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.776+0000 7fa7c9dac640 1 --2- 172.21.15.98:0/2135172489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 0x56359732f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.777+0000 7fa7c95ab640 1 -- 172.21.15.98:0/2135172489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 msgr2=0x56359732f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.777+0000 7fa7c95ab640 1 --2- 172.21.15.98:0/2135172489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 0x56359732f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.777+0000 7fa7c95ab640 1 -- 172.21.15.98:0/2135172489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359733a630 msgr2=0x56359733ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.777+0000 7fa7c95ab640 1 --2- 172.21.15.98:0/2135172489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359733a630 0x56359733ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.777+0000 7fa7c95ab640 1 -- 172.21.15.98:0/2135172489 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635972608e0 con 0x56359732fd70 2024-05-09T08:54:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.777+0000 7fa7c9dac640 1 --2- 172.21.15.98:0/2135172489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 0x56359732f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:54:16.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.779+0000 7fa7c9dac640 1 --2- 172.21.15.98:0/1989227643 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa7b00a3010 0x7fa7b00a54d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009670 tx=0x7fa7b4006bd0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:16.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.780+0000 7fa7c95ab640 1 --2- 172.21.15.98:0/2135172489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732fd70 0x56359733a0f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009da0 tx=0x7fa7bc00e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=311d202f19e41043 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:16.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.780+0000 7fa7affff640 1 -- 172.21.15.98:0/2135172489 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc017070 con 0x56359732fd70 2024-05-09T08:54:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.780+0000 7fa7affff640 1 -- 172.21.15.98:0/2135172489 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7bc013950 con 0x56359732fd70 2024-05-09T08:54:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.784+0000 7fa7affff640 1 -- 172.21.15.98:0/2135172489 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc0129b0 con 0x56359732fd70 2024-05-09T08:54:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.785+0000 7fa7cedf3740 1 -- 172.21.15.98:0/2135172489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732fd70 msgr2=0x56359733a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.785+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/2135172489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732fd70 0x56359733a0f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009da0 tx=0x7fa7bc00e5a0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.785+0000 7fa7cedf3740 1 -- 172.21.15.98:0/2135172489 shutdown_connections 2024-05-09T08:54:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.785+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/2135172489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359733a630 0x56359733ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.785+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/2135172489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732fd70 0x56359733a0f0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.785+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/2135172489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 0x56359732f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.785+0000 7fa7cedf3740 1 -- 172.21.15.98:0/2135172489 >> 172.21.15.98:0/2135172489 conn(0x56359732b960 msgr2=0x56359732bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.786+0000 7fa7cedf3740 1 -- 172.21.15.98:0/2135172489 shutdown_connections 2024-05-09T08:54:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.786+0000 7fa7cedf3740 1 -- 172.21.15.98:0/2135172489 wait complete. 2024-05-09T08:54:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.786+0000 7fa7cedf3740 1 -- start start 2024-05-09T08:54:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.787+0000 7fa7cedf3740 1 Processor -- start 2024-05-09T08:54:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.788+0000 7fa7cedf3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732f450 0x5635973374f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.788+0000 7fa7cedf3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359732fd70 0x563597337a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.788+0000 7fa7cedf3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359733a630 0x563597337f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.788+0000 7fa7cedf3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56359733d260 con 0x56359732f450 2024-05-09T08:54:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.788+0000 7fa7cedf3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56359724f790 con 0x56359732fd70 2024-05-09T08:54:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.788+0000 7fa7cedf3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56359733d3e0 con 0x56359733a630 2024-05-09T08:54:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.788+0000 7fa7cedf3740 10 client.0 _ll_register_callbacks cb 0x56359728aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.788+0000 7fa7ca5ad640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359733a630 0x563597337f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.789+0000 7fa7ca5ad640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359733a630 0x563597337f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45368/0 (socket says 172.21.15.98:45368) 2024-05-09T08:54:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.789+0000 7fa7ca5ad640 1 -- 172.21.15.98:0/485331466 learned_addr learned my addr 172.21.15.98:0/485331466 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.789+0000 7fa7c95ab640 1 --2- 172.21.15.98:0/485331466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359732fd70 0x563597337a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.790+0000 7fa7c9dac640 1 --2- 172.21.15.98:0/485331466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732f450 0x5635973374f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.791+0000 7fa7ca5ad640 1 -- 172.21.15.98:0/485331466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359732fd70 msgr2=0x563597337a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.791+0000 7fa7ca5ad640 1 --2- 172.21.15.98:0/485331466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359732fd70 0x563597337a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.791+0000 7fa7ca5ad640 1 -- 172.21.15.98:0/485331466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732f450 msgr2=0x5635973374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.791+0000 7fa7ca5ad640 1 --2- 172.21.15.98:0/485331466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732f450 0x5635973374f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.791+0000 7fa7ca5ad640 1 -- 172.21.15.98:0/485331466 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563596fa8210 con 0x56359733a630 2024-05-09T08:54:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.791+0000 7fa7c9dac640 1 --2- 172.21.15.98:0/485331466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732f450 0x5635973374f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:54:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.792+0000 7fa7ca5ad640 1 --2- 172.21.15.98:0/485331466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359733a630 0x563597337f70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa7c00ffa20 tx=0x7fa7c00ce2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a967afd486f54be2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:16.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.795+0000 7fa7ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:54:16.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.795+0000 7fa7ad7fa640 1 -- 172.21.15.98:0/485331466 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c00e6070 con 0x56359733a630 2024-05-09T08:54:16.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.795+0000 7fa7ad7fa640 1 -- 172.21.15.98:0/485331466 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7fa7c00ffc40 con 0x56359733a630 2024-05-09T08:54:16.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.797+0000 7fa7cedf3740 1 -- 172.21.15.98:0/485331466 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7bc013950 con 0x56359733a630 2024-05-09T08:54:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.798+0000 7fa7ad7fa640 1 -- 172.21.15.98:0/485331466 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa7c00ed050 con 0x56359733a630 2024-05-09T08:54:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.798+0000 7fa7cedf3740 10 client.25930 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.798+0000 7fa7cedf3740 1 -- 172.21.15.98:0/485331466 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563597074620 con 0x56359733a630 2024-05-09T08:54:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.799+0000 7fa7ad7fa640 1 -- 172.21.15.98:0/485331466 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7bc013950 con 0x56359733a630 2024-05-09T08:54:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.800+0000 7fa7ad7fa640 1 -- 172.21.15.98:0/485331466 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fa7c0051070 con 0x56359733a630 2024-05-09T08:54:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.800+0000 7fa7cedf3740 10 client.25930 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.800+0000 7fa7cedf3740 10 client.25930 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.800+0000 7fa7cedf3740 20 client.25930 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.800+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/485331466 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56359734a310 0x56359734c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.800+0000 7fa7cedf3740 4 client.25930 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.800+0000 7fa7cedf3740 1 -- 172.21.15.98:0/485331466 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635972a1a70 con 0x56359734a310 2024-05-09T08:54:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.801+0000 7fa7c9dac640 1 --2- 172.21.15.98:0/485331466 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56359734a310 0x56359734c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.804+0000 7fa7c9dac640 1 --2- 172.21.15.98:0/485331466 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56359734a310 0x56359734c770 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30503f6d87a81302 server_cookie=6344fdb0ffc6e52 in_seq=0 out_seq=0 2024-05-09T08:54:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.805+0000 7fa7ad7fa640 10 client.25930 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.807+0000 7fa7ad7fa640 1 -- 172.21.15.98:0/485331466 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5635972a1a70 con 0x56359734a310 2024-05-09T08:54:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.807+0000 7fa7ad7fa640 10 client.25930 handle_command_reply: tid=0 2024-05-09T08:54:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.807+0000 7fa7cedf3740 1 client.25930 shutdown 2024-05-09T08:54:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.808+0000 7fa7cedf3740 1 -- 172.21.15.98:0/485331466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359733a630 msgr2=0x563597337f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.808+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/485331466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359733a630 0x563597337f70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa7c00ffa20 tx=0x7fa7c00ce2d0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.808+0000 7fa7cedf3740 1 -- 172.21.15.98:0/485331466 shutdown_connections 2024-05-09T08:54:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.808+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/485331466 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56359734a310 0x56359734c770 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.808+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/485331466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359733a630 0x563597337f70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.808+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/485331466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359732fd70 0x563597337a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.808+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/485331466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732f450 0x5635973374f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.808+0000 7fa7cedf3740 1 -- 172.21.15.98:0/485331466 >> 172.21.15.98:0/485331466 conn(0x56359732b960 msgr2=0x563597331690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.809+0000 7fa7ad7fa640 0 client.25930 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.813+0000 7fa7cedf3740 1 -- 172.21.15.98:0/485331466 shutdown_connections 2024-05-09T08:54:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.814+0000 7fa7cedf3740 1 -- 172.21.15.98:0/485331466 wait complete. 2024-05-09T08:54:16.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.816+0000 7fa7cedf3740 20 client.25930 trim_cache size 0 max 16384 2024-05-09T08:54:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.850+0000 7fa7cedf3740 1 Processor -- start 2024-05-09T08:54:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.850+0000 7fa7cedf3740 1 -- start start 2024-05-09T08:54:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.850+0000 7fa7cedf3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 0x5635973461d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.850+0000 7fa7cedf3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732fd70 0x563597331e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.851+0000 7fa7c95ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732fd70 0x563597331e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.851+0000 7fa7c95ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732fd70 0x563597331e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43054/0 (socket says 172.21.15.98:43054) 2024-05-09T08:54:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.851+0000 7fa7c95ab640 1 -- 172.21.15.98:0/439559584 learned_addr learned my addr 172.21.15.98:0/439559584 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.851+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/439559584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359733a630 0x563597335dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.851+0000 7fa7cedf3740 1 -- 172.21.15.98:0/439559584 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a800feb0 con 0x56359732fd70 2024-05-09T08:54:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.851+0000 7fa7cedf3740 1 -- 172.21.15.98:0/439559584 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a80119f0 con 0x56359733a630 2024-05-09T08:54:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.851+0000 7fa7cedf3740 1 -- 172.21.15.98:0/439559584 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a8010580 con 0x56359732f450 2024-05-09T08:54:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.851+0000 7fa7c9dac640 1 --2- 172.21.15.98:0/439559584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 0x5635973461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.852+0000 7fa7ca5ad640 1 --2- 172.21.15.98:0/439559584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359733a630 0x563597335dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.852+0000 7fa7c95ab640 1 -- 172.21.15.98:0/439559584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 msgr2=0x5635973461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.852+0000 7fa7c95ab640 1 --2- 172.21.15.98:0/439559584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 0x5635973461d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.852+0000 7fa7c95ab640 1 -- 172.21.15.98:0/439559584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359733a630 msgr2=0x563597335dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.852+0000 7fa7c95ab640 1 --2- 172.21.15.98:0/439559584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359733a630 0x563597335dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.852+0000 7fa7c95ab640 1 -- 172.21.15.98:0/439559584 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635973364a0 con 0x56359732fd70 2024-05-09T08:54:16.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.852+0000 7fa7c9dac640 1 --2- 172.21.15.98:0/439559584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 0x5635973461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:54:16.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.853+0000 7fa7c95ab640 1 --2- 172.21.15.98:0/439559584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732fd70 0x563597331e10 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc012580 tx=0x7fa7bc012470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42751d9ba4026c37 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:16.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.854+0000 7fa7af7fe640 1 -- 172.21.15.98:0/439559584 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc017070 con 0x56359732fd70 2024-05-09T08:54:16.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.854+0000 7fa7af7fe640 1 -- 172.21.15.98:0/439559584 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7bc0044b0 con 0x56359732fd70 2024-05-09T08:54:16.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.854+0000 7fa7af7fe640 1 -- 172.21.15.98:0/439559584 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc012cd0 con 0x56359732fd70 2024-05-09T08:54:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.855+0000 7fa7cedf3740 1 -- 172.21.15.98:0/439559584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732fd70 msgr2=0x563597331e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.855+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/439559584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732fd70 0x563597331e10 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc012580 tx=0x7fa7bc012470 comp rx=0 tx=0).stop 2024-05-09T08:54:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.856+0000 7fa7cedf3740 1 -- 172.21.15.98:0/439559584 shutdown_connections 2024-05-09T08:54:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.856+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/439559584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359733a630 0x563597335dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.856+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/439559584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359732fd70 0x563597331e10 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.856+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/439559584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 0x5635973461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.856+0000 7fa7cedf3740 1 -- 172.21.15.98:0/439559584 >> 172.21.15.98:0/439559584 conn(0x56359732b960 msgr2=0x563597337d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.856+0000 7fa7cedf3740 1 -- 172.21.15.98:0/439559584 shutdown_connections 2024-05-09T08:54:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.856+0000 7fa7cedf3740 1 -- 172.21.15.98:0/439559584 wait complete. 2024-05-09T08:54:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.857+0000 7fa7cedf3740 1 -- start start 2024-05-09T08:54:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.857+0000 7fa7cedf3740 1 Processor -- start 2024-05-09T08:54:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7cedf3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 0x56359733e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7cedf3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359732fd70 0x56359733e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7cedf3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359733a630 0x56359733ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7cedf3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a8011320 con 0x56359733a630 2024-05-09T08:54:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7cedf3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a8010c50 con 0x56359732fd70 2024-05-09T08:54:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7cedf3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a800df40 con 0x56359732f450 2024-05-09T08:54:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7cedf3740 10 client.0 _ll_register_callbacks cb 0x7fa7c0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7c9dac640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 0x56359733e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7c95ab640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359732fd70 0x56359733e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7c95ab640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359732fd70 0x56359733e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45306/0 (socket says 172.21.15.98:45306) 2024-05-09T08:54:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7c95ab640 1 -- 172.21.15.98:0/386950501 learned_addr learned my addr 172.21.15.98:0/386950501 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7ca5ad640 1 --2- 172.21.15.98:0/386950501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359733a630 0x56359733ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7c95ab640 1 -- 172.21.15.98:0/386950501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 msgr2=0x56359733e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7c95ab640 1 --2- 172.21.15.98:0/386950501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 0x56359733e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7c95ab640 1 -- 172.21.15.98:0/386950501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359733a630 msgr2=0x56359733ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.858+0000 7fa7c95ab640 1 --2- 172.21.15.98:0/386950501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359733a630 0x56359733ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.859+0000 7fa7c95ab640 1 -- 172.21.15.98:0/386950501 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635973af9b0 con 0x56359732fd70 2024-05-09T08:54:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.859+0000 7fa7c95ab640 1 --2- 172.21.15.98:0/386950501 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359732fd70 0x56359733e750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009da0 tx=0x7fa7bc0090e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=623f065f267c217b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.860+0000 7fa7ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:54:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.860+0000 7fa7ad7fa640 1 -- 172.21.15.98:0/386950501 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc017040 con 0x56359732fd70 2024-05-09T08:54:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.860+0000 7fa7ad7fa640 1 -- 172.21.15.98:0/386950501 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7fa7bc005b80 con 0x56359732fd70 2024-05-09T08:54:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.860+0000 7fa7cedf3740 1 -- 172.21.15.98:0/386950501 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7bc0044b0 con 0x56359732fd70 2024-05-09T08:54:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.860+0000 7fa7ad7fa640 1 -- 172.21.15.98:0/386950501 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa7bc037050 con 0x56359732fd70 2024-05-09T08:54:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.860+0000 7fa7cedf3740 10 client.25973 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.860+0000 7fa7cedf3740 1 -- 172.21.15.98:0/386950501 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5635973b0c30 con 0x56359732fd70 2024-05-09T08:54:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.863+0000 7fa7ad7fa640 1 -- 172.21.15.98:0/386950501 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7bc0125b0 con 0x56359732fd70 2024-05-09T08:54:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.863+0000 7fa7ad7fa640 1 -- 172.21.15.98:0/386950501 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fa7bc023070 con 0x56359732fd70 2024-05-09T08:54:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.863+0000 7fa7cedf3740 10 client.25973 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.863+0000 7fa7cedf3740 10 client.25973 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.864+0000 7fa7cedf3740 20 client.25973 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.864+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/386950501 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5635973b0e50 0x5635973b3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.864+0000 7fa7cedf3740 4 client.25973 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:54:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.864+0000 7fa7c9dac640 1 --2- 172.21.15.98:0/386950501 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5635973b0e50 0x5635973b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.864+0000 7fa7cedf3740 1 -- 172.21.15.98:0/386950501 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fa7bc0044b0 con 0x5635973b0e50 2024-05-09T08:54:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.865+0000 7fa7c9dac640 1 --2- 172.21.15.98:0/386950501 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5635973b0e50 0x5635973b3240 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b230631a44d7912 server_cookie=d297ce188e5b8516 in_seq=0 out_seq=0 2024-05-09T08:54:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:16.865+0000 7fa7ad7fa640 10 client.25973 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:17.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:54:17.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:54:18.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:17 smithi142 ceph-mon[34625]: pgmap v113: 129 pgs: 129 active+clean; 23 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 630 MiB/s wr, 3.44k op/s 2024-05-09T08:54:18.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:17 smithi105 ceph-mon[34498]: pgmap v113: 129 pgs: 129 active+clean; 23 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 630 MiB/s wr, 3.44k op/s 2024-05-09T08:54:18.238 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:17 smithi098 ceph-mon[30092]: pgmap v113: 129 pgs: 129 active+clean; 23 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 630 MiB/s wr, 3.44k op/s 2024-05-09T08:54:18.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.385+0000 7f7a240ae640 1 -- 172.21.15.98:0/3510986668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a1c106710 msgr2=0x7f7a1c106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:18.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.385+0000 7f7a240ae640 1 --2- 172.21.15.98:0/3510986668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a1c106710 0x7f7a1c106af0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009930 tx=0x7f7a1802a420 comp rx=0 tx=0).stop 2024-05-09T08:54:18.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.385+0000 7f7a240ae640 1 -- 172.21.15.98:0/3510986668 shutdown_connections 2024-05-09T08:54:18.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.385+0000 7f7a240ae640 1 --2- 172.21.15.98:0/3510986668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a1c1010c0 0x7f7a1c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:18.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.385+0000 7f7a240ae640 1 --2- 172.21.15.98:0/3510986668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a1c100700 0x7f7a1c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:18.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.385+0000 7f7a240ae640 1 --2- 172.21.15.98:0/3510986668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a1c106710 0x7f7a1c106af0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:18.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.385+0000 7f7a240ae640 1 -- 172.21.15.98:0/3510986668 >> 172.21.15.98:0/3510986668 conn(0x7f7a1c077830 msgr2=0x7f7a1c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:18.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.385+0000 7f7a240ae640 1 -- 172.21.15.98:0/3510986668 shutdown_connections 2024-05-09T08:54:18.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.385+0000 7f7a240ae640 1 -- 172.21.15.98:0/3510986668 wait complete. 2024-05-09T08:54:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.386+0000 7f7a240ae640 1 Processor -- start 2024-05-09T08:54:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.386+0000 7f7a240ae640 1 -- start start 2024-05-09T08:54:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a240ae640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a1c100700 0x7f7a1c198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a240ae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a1c1010c0 0x7f7a1c199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a240ae640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a1c106710 0x7f7a1c1a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a240ae640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c110ae0 con 0x7f7a1c100700 2024-05-09T08:54:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a240ae640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c110960 con 0x7f7a1c106710 2024-05-09T08:54:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a240ae640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c110c60 con 0x7f7a1c1010c0 2024-05-09T08:54:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a21649640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a1c1010c0 0x7f7a1c199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a2264b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a1c106710 0x7f7a1c1a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a21e4a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a1c100700 0x7f7a1c198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a21649640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a1c1010c0 0x7f7a1c199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45402/0 (socket says 172.21.15.98:45402) 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a21649640 1 -- 172.21.15.98:0/493470676 learned_addr learned my addr 172.21.15.98:0/493470676 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a21649640 1 -- 172.21.15.98:0/493470676 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a1c106710 msgr2=0x7f7a1c1a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a21649640 1 --2- 172.21.15.98:0/493470676 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a1c106710 0x7f7a1c1a3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a21649640 1 -- 172.21.15.98:0/493470676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a1c100700 msgr2=0x7f7a1c198f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a21649640 1 --2- 172.21.15.98:0/493470676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a1c100700 0x7f7a1c198f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a21649640 1 -- 172.21.15.98:0/493470676 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1c1a44e0 con 0x7f7a1c1010c0 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.387+0000 7f7a21649640 1 --2- 172.21.15.98:0/493470676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a1c1010c0 0x7f7a1c199460 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00ea60 tx=0x7f7a0c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.388+0000 7f7a12ffd640 1 -- 172.21.15.98:0/493470676 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c00cde0 con 0x7f7a1c1010c0 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.388+0000 7f7a12ffd640 1 -- 172.21.15.98:0/493470676 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a0c0035f0 con 0x7f7a1c1010c0 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.388+0000 7f7a12ffd640 1 -- 172.21.15.98:0/493470676 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c010670 con 0x7f7a1c1010c0 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.388+0000 7f7a240ae640 1 -- 172.21.15.98:0/493470676 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a1c1a46b0 con 0x7f7a1c1010c0 2024-05-09T08:54:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.388+0000 7f7a240ae640 1 -- 172.21.15.98:0/493470676 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a1c1a4c50 con 0x7f7a1c1010c0 2024-05-09T08:54:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.390+0000 7f7a12ffd640 1 -- 172.21.15.98:0/493470676 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7a0c0037a0 con 0x7f7a1c1010c0 2024-05-09T08:54:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.391+0000 7f7a26db6740 1 Processor -- start 2024-05-09T08:54:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.391+0000 7f7a26db6740 1 -- start start 2024-05-09T08:54:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.391+0000 7f7a26db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a071e280 0x55c8a071e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.391+0000 7f7a26db6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071eba0 0x55c8a0728d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.391+0000 7f7a26db6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a0729270 0x55c8a072b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.391+0000 7f7a26db6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c8a0683d00 con 0x55c8a071e280 2024-05-09T08:54:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.391+0000 7f7a26db6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c8a062b4c0 con 0x55c8a0729270 2024-05-09T08:54:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.391+0000 7f7a26db6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c8a067fcc0 con 0x55c8a071eba0 2024-05-09T08:54:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.391+0000 7f7a2264b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a071e280 0x55c8a071e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.391+0000 7f7a21649640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a0729270 0x55c8a072b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.391+0000 7f7a2264b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a071e280 0x55c8a071e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43072/0 (socket says 172.21.15.98:43072) 2024-05-09T08:54:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.391+0000 7f7a2264b640 1 -- 172.21.15.98:0/4138477697 learned_addr learned my addr 172.21.15.98:0/4138477697 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.392+0000 7f7a21e4a640 1 --2- 172.21.15.98:0/4138477697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071eba0 0x55c8a0728d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.392+0000 7f7a12ffd640 1 --2- 172.21.15.98:0/493470676 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f7a080a2d80 0x7f7a080a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.392+0000 7f7a21649640 1 -- 172.21.15.98:0/4138477697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071eba0 msgr2=0x55c8a0728d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.392+0000 7f7a21649640 1 --2- 172.21.15.98:0/4138477697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071eba0 0x55c8a0728d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.392+0000 7f7a21649640 1 -- 172.21.15.98:0/4138477697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a071e280 msgr2=0x55c8a071e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.392+0000 7f7a21649640 1 --2- 172.21.15.98:0/4138477697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a071e280 0x55c8a071e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.392+0000 7f7a21649640 1 -- 172.21.15.98:0/4138477697 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8a04617d0 con 0x55c8a0729270 2024-05-09T08:54:18.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.392+0000 7f7a2264b640 1 --2- 172.21.15.98:0/4138477697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a071e280 0x55c8a071e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:54:18.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.392+0000 7f7a2264b640 1 --2- 172.21.15.98:0/493470676 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f7a080a2d80 0x7f7a080a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:18.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.392+0000 7f7a12ffd640 1 -- 172.21.15.98:0/493470676 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7f7a0c0cfaa0 con 0x7f7a1c1010c0 2024-05-09T08:54:18.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.392+0000 7f7a21649640 1 --2- 172.21.15.98:0/4138477697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a0729270 0x55c8a072b660 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0cce30 tx=0x7f7a0c020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=136b1a82eb870870 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:18.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.392+0000 7f7a07fff640 1 -- 172.21.15.98:0/4138477697 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c09b560 con 0x55c8a0729270 2024-05-09T08:54:18.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.392+0000 7f7a07fff640 1 -- 172.21.15.98:0/4138477697 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a0c09c300 con 0x55c8a0729270 2024-05-09T08:54:18.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.392+0000 7f7a07fff640 1 -- 172.21.15.98:0/4138477697 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c090ef0 con 0x55c8a0729270 2024-05-09T08:54:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.393+0000 7f7a26db6740 1 -- 172.21.15.98:0/4138477697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a0729270 msgr2=0x55c8a072b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.393+0000 7f7a26db6740 1 --2- 172.21.15.98:0/4138477697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a0729270 0x55c8a072b660 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0cce30 tx=0x7f7a0c020000 comp rx=0 tx=0).stop 2024-05-09T08:54:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.393+0000 7f7a2264b640 1 --2- 172.21.15.98:0/493470676 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f7a080a2d80 0x7f7a080a5240 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7a14009fd0 tx=0x7f7a14004a40 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.393+0000 7f7a26db6740 1 -- 172.21.15.98:0/4138477697 shutdown_connections 2024-05-09T08:54:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.393+0000 7f7a26db6740 1 --2- 172.21.15.98:0/4138477697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a0729270 0x55c8a072b660 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.393+0000 7f7a26db6740 1 --2- 172.21.15.98:0/4138477697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071eba0 0x55c8a0728d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.393+0000 7f7a26db6740 1 --2- 172.21.15.98:0/4138477697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a071e280 0x55c8a071e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.393+0000 7f7a26db6740 1 -- 172.21.15.98:0/4138477697 >> 172.21.15.98:0/4138477697 conn(0x55c8a071a5a0 msgr2=0x55c8a071a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.393+0000 7f7a26db6740 1 -- 172.21.15.98:0/4138477697 shutdown_connections 2024-05-09T08:54:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.393+0000 7f7a26db6740 1 -- 172.21.15.98:0/4138477697 wait complete. 2024-05-09T08:54:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.394+0000 7f7a26db6740 1 -- start start 2024-05-09T08:54:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.394+0000 7f7a26db6740 1 Processor -- start 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.394+0000 7f7a26db6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071e280 0x55c8a07261f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.394+0000 7f7a26db6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071eba0 0x55c8a0726730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.394+0000 7f7a26db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a0729270 0x55c8a0726c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.394+0000 7f7a26db6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c8a072c000 con 0x55c8a0729270 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.394+0000 7f7a26db6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c8a063eb90 con 0x55c8a071eba0 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.394+0000 7f7a26db6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c8a072c180 con 0x55c8a071e280 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.394+0000 7f7a26db6740 10 client.0 _ll_register_callbacks cb 0x55c8a067ada0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.394+0000 7f7a21e4a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071e280 0x55c8a07261f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.394+0000 7f7a21649640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071eba0 0x55c8a0726730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a2264b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a0729270 0x55c8a0726c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a21e4a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071e280 0x55c8a07261f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45416/0 (socket says 172.21.15.98:45416) 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a21649640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071eba0 0x55c8a0726730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45354/0 (socket says 172.21.15.98:45354) 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a21e4a640 1 -- 172.21.15.98:0/654046774 learned_addr learned my addr 172.21.15.98:0/654046774 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a21e4a640 1 -- 172.21.15.98:0/654046774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071eba0 msgr2=0x55c8a0726730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a21e4a640 1 --2- 172.21.15.98:0/654046774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071eba0 0x55c8a0726730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a21e4a640 1 -- 172.21.15.98:0/654046774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a0729270 msgr2=0x55c8a0726c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a21e4a640 1 --2- 172.21.15.98:0/654046774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a0729270 0x55c8a0726c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a21e4a640 1 -- 172.21.15.98:0/654046774 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8a03973f0 con 0x55c8a071e280 2024-05-09T08:54:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a21649640 1 --2- 172.21.15.98:0/654046774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071eba0 0x55c8a0726730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:54:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a2264b640 1 --2- 172.21.15.98:0/654046774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a0729270 0x55c8a0726c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:54:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a21e4a640 1 --2- 172.21.15.98:0/654046774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071e280 0x55c8a07261f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7a18000c00 tx=0x7f7a18031920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc8762e406c0b49a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a057fa640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:54:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a057fa640 1 -- 172.21.15.98:0/654046774 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a18035040 con 0x55c8a071e280 2024-05-09T08:54:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a057fa640 1 -- 172.21.15.98:0/654046774 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7f7a18004240 con 0x55c8a071e280 2024-05-09T08:54:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.395+0000 7f7a26db6740 1 -- 172.21.15.98:0/654046774 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a0c09c300 con 0x55c8a071e280 2024-05-09T08:54:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.396+0000 7f7a057fa640 1 -- 172.21.15.98:0/654046774 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7a1804f050 con 0x55c8a071e280 2024-05-09T08:54:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.396+0000 7f7a26db6740 10 client.25942 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.396+0000 7f7a26db6740 1 -- 172.21.15.98:0/654046774 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c8a03570f0 con 0x55c8a071e280 2024-05-09T08:54:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.396+0000 7f7a057fa640 1 -- 172.21.15.98:0/654046774 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a0c09c300 con 0x55c8a071e280 2024-05-09T08:54:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.396+0000 7f7a057fa640 1 -- 172.21.15.98:0/654046774 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f7a18004ba0 con 0x55c8a071e280 2024-05-09T08:54:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.396+0000 7f7a26db6740 10 client.25942 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.396+0000 7f7a26db6740 10 client.25942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.396+0000 7f7a26db6740 20 client.25942 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.396+0000 7f7a26db6740 1 --2- 172.21.15.98:0/654046774 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c8a0739570 0x55c8a073b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.396+0000 7f7a26db6740 4 client.25942 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.396+0000 7f7a26db6740 1 -- 172.21.15.98:0/654046774 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8a0691430 con 0x55c8a0739570 2024-05-09T08:54:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.396+0000 7f7a21649640 1 --2- 172.21.15.98:0/654046774 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c8a0739570 0x55c8a073b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.397+0000 7f7a057fa640 10 client.25942 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:18.397+0000 7f7a21649640 1 --2- 172.21.15.98:0/654046774 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c8a0739570 0x55c8a073b9a0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba0322e55fc652d5 server_cookie=8892fd88b23c7a97 in_seq=0 out_seq=0 2024-05-09T08:54:20.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:19 smithi142 ceph-mon[34625]: pgmap v114: 129 pgs: 129 active+clean; 23 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 594 MiB/s wr, 3.15k op/s 2024-05-09T08:54:20.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:19 smithi105 ceph-mon[34498]: pgmap v114: 129 pgs: 129 active+clean; 23 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 594 MiB/s wr, 3.15k op/s 2024-05-09T08:54:20.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:19 smithi098 ceph-mon[30092]: pgmap v114: 129 pgs: 129 active+clean; 23 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 594 MiB/s wr, 3.15k op/s 2024-05-09T08:54:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.499+0000 7fa7ad7fa640 1 -- 172.21.15.98:0/386950501 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fa7bc0044b0 con 0x5635973b0e50 2024-05-09T08:54:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.499+0000 7fa7ad7fa640 10 client.25973 handle_command_reply: tid=0 2024-05-09T08:54:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.499+0000 7fa7cedf3740 1 client.25973 shutdown 2024-05-09T08:54:20.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.499+0000 7f7a057fa640 1 -- 172.21.15.98:0/654046774 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8a0691430 con 0x55c8a0739570 2024-05-09T08:54:20.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.499+0000 7f7a057fa640 10 client.25942 handle_command_reply: tid=0 2024-05-09T08:54:20.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.499+0000 7f7a26db6740 1 client.25942 shutdown 2024-05-09T08:54:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.500+0000 7fa7cedf3740 1 -- 172.21.15.98:0/386950501 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359732fd70 msgr2=0x56359733e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.500+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/386950501 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359732fd70 0x56359733e750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009da0 tx=0x7fa7bc0090e0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.500+0000 7f7a26db6740 1 -- 172.21.15.98:0/654046774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071e280 msgr2=0x55c8a07261f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:20.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.500+0000 7f7a26db6740 1 --2- 172.21.15.98:0/654046774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071e280 0x55c8a07261f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7a18000c00 tx=0x7f7a18031920 comp rx=0 tx=0).stop 2024-05-09T08:54:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.500+0000 7fa7cedf3740 1 -- 172.21.15.98:0/386950501 shutdown_connections 2024-05-09T08:54:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.500+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/386950501 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5635973b0e50 0x5635973b3240 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.500+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/386950501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56359733a630 0x56359733ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.500+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/386950501 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56359732fd70 0x56359733e750 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.500+0000 7fa7cedf3740 1 --2- 172.21.15.98:0/386950501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56359732f450 0x56359733e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.500+0000 7fa7cedf3740 1 -- 172.21.15.98:0/386950501 >> 172.21.15.98:0/386950501 conn(0x56359732b960 msgr2=0x563597329f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.500+0000 7fa7ad7fa640 0 client.25973 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.501+0000 7f7a26db6740 1 -- 172.21.15.98:0/654046774 shutdown_connections 2024-05-09T08:54:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.501+0000 7f7a26db6740 1 --2- 172.21.15.98:0/654046774 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c8a0739570 0x55c8a073b9a0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.501+0000 7f7a26db6740 1 --2- 172.21.15.98:0/654046774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a0729270 0x55c8a0726c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.501+0000 7f7a26db6740 1 --2- 172.21.15.98:0/654046774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071eba0 0x55c8a0726730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.501+0000 7f7a26db6740 1 --2- 172.21.15.98:0/654046774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071e280 0x55c8a07261f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.501+0000 7f7a26db6740 1 -- 172.21.15.98:0/654046774 >> 172.21.15.98:0/654046774 conn(0x55c8a071a5a0 msgr2=0x55c8a07203a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.501+0000 7f7a057fa640 0 client.25942 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.501+0000 7fa7cedf3740 1 -- 172.21.15.98:0/386950501 shutdown_connections 2024-05-09T08:54:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.501+0000 7fa7cedf3740 1 -- 172.21.15.98:0/386950501 wait complete. 2024-05-09T08:54:20.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.501+0000 7f7a26db6740 1 -- 172.21.15.98:0/654046774 shutdown_connections 2024-05-09T08:54:20.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.501+0000 7f7a26db6740 1 -- 172.21.15.98:0/654046774 wait complete. 2024-05-09T08:54:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.501+0000 7fa7cedf3740 20 client.25973 trim_cache size 0 max 16384 2024-05-09T08:54:20.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.502+0000 7f7a26db6740 20 client.25942 trim_cache size 0 max 16384 2024-05-09T08:54:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:54:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:54:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:54:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:54:20.510 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.511+0000 7fa7cc010640 1 -- 172.21.15.98:0/1989227643 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa7b00a3010 msgr2=0x7fa7b00a54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:20.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.511+0000 7fa7cc010640 1 --2- 172.21.15.98:0/1989227643 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa7b00a3010 0x7fa7b00a54d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009670 tx=0x7fa7b4006bd0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.511+0000 7fa7cc010640 1 -- 172.21.15.98:0/1989227643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7c4116e80 msgr2=0x7fa7c4117330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:20.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.511+0000 7fa7cc010640 1 --2- 172.21.15.98:0/1989227643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7c4116e80 0x7fa7c4117330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa7c003c010 tx=0x7fa7c0003ba0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.514 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.514+0000 7fa7cc010640 1 -- 172.21.15.98:0/1989227643 shutdown_connections 2024-05-09T08:54:20.514 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.514+0000 7fa7cc010640 1 --2- 172.21.15.98:0/1989227643 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa7b00a3010 0x7fa7b00a54d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.514 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.514+0000 7fa7cc010640 1 --2- 172.21.15.98:0/1989227643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7c4116e80 0x7fa7c4117330 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.514 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.514+0000 7fa7cc010640 1 --2- 172.21.15.98:0/1989227643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c411b010 0x7fa7c4116940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.514 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.514+0000 7fa7cc010640 1 --2- 172.21.15.98:0/1989227643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c4110020 0x7fa7c411a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.514 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.514+0000 7fa7cc010640 1 -- 172.21.15.98:0/1989227643 >> 172.21.15.98:0/1989227643 conn(0x7fa7c406c900 msgr2=0x7fa7c410cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.521+0000 7fa7cc010640 1 -- 172.21.15.98:0/1989227643 shutdown_connections 2024-05-09T08:54:20.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:20.521+0000 7fa7cc010640 1 -- 172.21.15.98:0/1989227643 wait complete. 2024-05-09T08:54:20.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.552+0000 7f7a26db6740 1 Processor -- start 2024-05-09T08:54:20.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.552+0000 7f7a26db6740 1 -- start start 2024-05-09T08:54:20.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.552+0000 7f7a26db6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071e280 0x55c8a0725ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:20.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.552+0000 7f7a26db6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071eba0 0x55c8a0720a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:20.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.553+0000 7f7a26db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a0729270 0x55c8a0741270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:20.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.553+0000 7f7a26db6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c1aed80 con 0x55c8a0729270 2024-05-09T08:54:20.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.553+0000 7f7a26db6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c1b08c0 con 0x55c8a071e280 2024-05-09T08:54:20.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.553+0000 7f7a26db6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c1af450 con 0x55c8a071eba0 2024-05-09T08:54:20.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.557+0000 7f7a21e4a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071e280 0x55c8a0725ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:20.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.557+0000 7f7a21e4a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071e280 0x55c8a0725ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45358/0 (socket says 172.21.15.98:45358) 2024-05-09T08:54:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.557+0000 7f7a21e4a640 1 -- 172.21.15.98:0/1839935885 learned_addr learned my addr 172.21.15.98:0/1839935885 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.557+0000 7f7a21649640 1 --2- 172.21.15.98:0/1839935885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071eba0 0x55c8a0720a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.557+0000 7f7a2264b640 1 --2- 172.21.15.98:0/1839935885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a0729270 0x55c8a0741270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.559+0000 7f7a21e4a640 1 -- 172.21.15.98:0/1839935885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071eba0 msgr2=0x55c8a0720a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.559+0000 7f7a21e4a640 1 --2- 172.21.15.98:0/1839935885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071eba0 0x55c8a0720a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.559+0000 7f7a21e4a640 1 -- 172.21.15.98:0/1839935885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a0729270 msgr2=0x55c8a0741270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.559+0000 7f7a21e4a640 1 --2- 172.21.15.98:0/1839935885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a0729270 0x55c8a0741270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.559+0000 7f7a21e4a640 1 -- 172.21.15.98:0/1839935885 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8a071ef80 con 0x55c8a071e280 2024-05-09T08:54:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.560+0000 7f7a21e4a640 1 --2- 172.21.15.98:0/1839935885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071e280 0x55c8a0725ad0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009230 tx=0x7f7a18009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11bf1cf26d0ec974 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:20.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.562+0000 7f7a077fe640 1 -- 172.21.15.98:0/1839935885 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a18035030 con 0x55c8a071e280 2024-05-09T08:54:20.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.562+0000 7f7a077fe640 1 -- 172.21.15.98:0/1839935885 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a180054b0 con 0x55c8a071e280 2024-05-09T08:54:20.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.562+0000 7f7a077fe640 1 -- 172.21.15.98:0/1839935885 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a18004cd0 con 0x55c8a071e280 2024-05-09T08:54:20.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.563+0000 7f7a26db6740 1 -- 172.21.15.98:0/1839935885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071e280 msgr2=0x55c8a0725ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:20.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.563+0000 7f7a26db6740 1 --2- 172.21.15.98:0/1839935885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071e280 0x55c8a0725ad0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009230 tx=0x7f7a18009260 comp rx=0 tx=0).stop 2024-05-09T08:54:20.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.563+0000 7f7a26db6740 1 -- 172.21.15.98:0/1839935885 shutdown_connections 2024-05-09T08:54:20.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.563+0000 7f7a26db6740 1 --2- 172.21.15.98:0/1839935885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a0729270 0x55c8a0741270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.563+0000 7f7a26db6740 1 --2- 172.21.15.98:0/1839935885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a071eba0 0x55c8a0720a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.563+0000 7f7a26db6740 1 --2- 172.21.15.98:0/1839935885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071e280 0x55c8a0725ad0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.563+0000 7f7a26db6740 1 -- 172.21.15.98:0/1839935885 >> 172.21.15.98:0/1839935885 conn(0x55c8a071a5a0 msgr2=0x55c8a0718f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.570+0000 7f7a26db6740 1 -- 172.21.15.98:0/1839935885 shutdown_connections 2024-05-09T08:54:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.575+0000 7f7a26db6740 1 -- 172.21.15.98:0/1839935885 wait complete. 2024-05-09T08:54:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.575+0000 7f7a26db6740 1 -- start start 2024-05-09T08:54:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.575+0000 7f7a26db6740 1 Processor -- start 2024-05-09T08:54:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.576+0000 7f7a26db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a071e280 0x55c8a072c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.576+0000 7f7a26db6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071eba0 0x55c8a072c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.576+0000 7f7a26db6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a0729270 0x55c8a072ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.576+0000 7f7a26db6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c1b01f0 con 0x55c8a071e280 2024-05-09T08:54:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.576+0000 7f7a26db6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c1afb20 con 0x55c8a071eba0 2024-05-09T08:54:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.576+0000 7f7a26db6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c1acd60 con 0x55c8a0729270 2024-05-09T08:54:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.576+0000 7f7a26db6740 10 client.0 _ll_register_callbacks cb 0x7f7a18009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.577+0000 7f7a21e4a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a071e280 0x55c8a072c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.577+0000 7f7a21e4a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a071e280 0x55c8a072c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43102/0 (socket says 172.21.15.98:43102) 2024-05-09T08:54:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.577+0000 7f7a21e4a640 1 -- 172.21.15.98:0/3634439907 learned_addr learned my addr 172.21.15.98:0/3634439907 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.579+0000 7f7a21e4a640 1 -- 172.21.15.98:0/3634439907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a0729270 msgr2=0x55c8a072ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.579+0000 7f7a21e4a640 1 --2- 172.21.15.98:0/3634439907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a0729270 0x55c8a072ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.579+0000 7f7a21e4a640 1 -- 172.21.15.98:0/3634439907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071eba0 msgr2=0x55c8a072c8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:54:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.579+0000 7f7a21e4a640 1 --2- 172.21.15.98:0/3634439907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071eba0 0x55c8a072c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.579+0000 7f7a21e4a640 1 -- 172.21.15.98:0/3634439907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8a07b74c0 con 0x55c8a071e280 2024-05-09T08:54:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.579+0000 7f7a21e4a640 1 --2- 172.21.15.98:0/3634439907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a071e280 0x55c8a072c390 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009850 tx=0x7f7a18004810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218a4d76f79ead74 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.587+0000 7f7a057fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:54:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.587+0000 7f7a057fa640 1 -- 172.21.15.98:0/3634439907 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a18035040 con 0x55c8a071e280 2024-05-09T08:54:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.587+0000 7f7a26db6740 1 -- 172.21.15.98:0/3634439907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a180054b0 con 0x55c8a071e280 2024-05-09T08:54:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.587+0000 7f7a057fa640 1 -- 172.21.15.98:0/3634439907 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7f7a180051e0 con 0x55c8a071e280 2024-05-09T08:54:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.592+0000 7f7a057fa640 1 -- 172.21.15.98:0/3634439907 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7a18004950 con 0x55c8a071e280 2024-05-09T08:54:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.592+0000 7f7a26db6740 10 client.16674 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.592+0000 7f7a26db6740 1 -- 172.21.15.98:0/3634439907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c8a07b87d0 con 0x55c8a071e280 2024-05-09T08:54:20.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.595+0000 7f7a057fa640 1 -- 172.21.15.98:0/3634439907 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a180054b0 con 0x55c8a071e280 2024-05-09T08:54:20.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.595+0000 7f7a057fa640 1 -- 172.21.15.98:0/3634439907 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f7a18046c80 con 0x55c8a071e280 2024-05-09T08:54:20.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.595+0000 7f7a26db6740 10 client.16674 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:20.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.595+0000 7f7a26db6740 10 client.16674 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:20.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.595+0000 7f7a26db6740 20 client.16674 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:20.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.595+0000 7f7a26db6740 1 --2- 172.21.15.98:0/3634439907 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c8a07b8de0 0x55c8a07bb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:20.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.595+0000 7f7a26db6740 4 client.16674 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:54:20.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.595+0000 7f7a26db6740 1 -- 172.21.15.98:0/3634439907 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c8a068b780 con 0x55c8a07b8de0 2024-05-09T08:54:20.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.598+0000 7f7a21649640 1 --2- 172.21.15.98:0/3634439907 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c8a07b8de0 0x55c8a07bb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.609+0000 7f7a21649640 1 --2- 172.21.15.98:0/3634439907 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c8a07b8de0 0x55c8a07bb260 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=261c324b71e5a25 server_cookie=876014fe4c8661d3 in_seq=0 out_seq=0 2024-05-09T08:54:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.610+0000 7f7a057fa640 10 client.16674 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:20.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.615+0000 7f7a057fa640 1 -- 172.21.15.98:0/3634439907 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c8a068b780 con 0x55c8a07b8de0 2024-05-09T08:54:20.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.615+0000 7f7a057fa640 10 client.16674 handle_command_reply: tid=0 2024-05-09T08:54:20.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.615+0000 7f7a26db6740 1 client.16674 shutdown 2024-05-09T08:54:20.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.615+0000 7f7a26db6740 1 -- 172.21.15.98:0/3634439907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a071e280 msgr2=0x55c8a072c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:20.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.615+0000 7f7a26db6740 1 --2- 172.21.15.98:0/3634439907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a071e280 0x55c8a072c390 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009850 tx=0x7f7a18004810 comp rx=0 tx=0).stop 2024-05-09T08:54:20.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.617+0000 7f7a26db6740 1 -- 172.21.15.98:0/3634439907 shutdown_connections 2024-05-09T08:54:20.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.617+0000 7f7a26db6740 1 --2- 172.21.15.98:0/3634439907 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c8a07b8de0 0x55c8a07bb260 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.617+0000 7f7a26db6740 1 --2- 172.21.15.98:0/3634439907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c8a0729270 0x55c8a072ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.617+0000 7f7a26db6740 1 --2- 172.21.15.98:0/3634439907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c8a071eba0 0x55c8a072c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.617+0000 7f7a26db6740 1 --2- 172.21.15.98:0/3634439907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c8a071e280 0x55c8a072c390 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.617+0000 7f7a26db6740 1 -- 172.21.15.98:0/3634439907 >> 172.21.15.98:0/3634439907 conn(0x55c8a071a5a0 msgr2=0x55c8a07197e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:20.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.617+0000 7f7a057fa640 0 client.16674 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:20.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.617+0000 7f7a26db6740 1 -- 172.21.15.98:0/3634439907 shutdown_connections 2024-05-09T08:54:20.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.618+0000 7f7a26db6740 1 -- 172.21.15.98:0/3634439907 wait complete. 2024-05-09T08:54:20.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.618+0000 7f7a26db6740 20 client.16674 trim_cache size 0 max 16384 2024-05-09T08:54:20.618 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:54:20.618 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:54:20.618 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:54:20.618 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:54:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.623+0000 7f7a240ae640 1 -- 172.21.15.98:0/493470676 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f7a080a2d80 msgr2=0x7f7a080a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.623+0000 7f7a240ae640 1 --2- 172.21.15.98:0/493470676 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f7a080a2d80 0x7f7a080a5240 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7a14009fd0 tx=0x7f7a14004a40 comp rx=0 tx=0).stop 2024-05-09T08:54:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.623+0000 7f7a240ae640 1 -- 172.21.15.98:0/493470676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a1c1010c0 msgr2=0x7f7a1c199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.623+0000 7f7a240ae640 1 --2- 172.21.15.98:0/493470676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a1c1010c0 0x7f7a1c199460 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00ea60 tx=0x7f7a0c00ef20 comp rx=0 tx=0).stop 2024-05-09T08:54:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.623+0000 7f7a240ae640 1 -- 172.21.15.98:0/493470676 shutdown_connections 2024-05-09T08:54:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.623+0000 7f7a240ae640 1 --2- 172.21.15.98:0/493470676 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f7a080a2d80 0x7f7a080a5240 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.623+0000 7f7a240ae640 1 --2- 172.21.15.98:0/493470676 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a1c106710 0x7f7a1c1a3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.623+0000 7f7a240ae640 1 --2- 172.21.15.98:0/493470676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a1c1010c0 0x7f7a1c199460 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.623+0000 7f7a240ae640 1 --2- 172.21.15.98:0/493470676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a1c100700 0x7f7a1c198f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.624+0000 7f7a240ae640 1 -- 172.21.15.98:0/493470676 >> 172.21.15.98:0/493470676 conn(0x7f7a1c077830 msgr2=0x7f7a1c102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:20.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.625+0000 7f7a240ae640 1 -- 172.21.15.98:0/493470676 shutdown_connections 2024-05-09T08:54:20.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.625+0000 7f7a240ae640 1 -- 172.21.15.98:0/493470676 wait complete. 2024-05-09T08:54:20.655 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:54:20.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:54:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.901+0000 7ffb025c7640 1 -- 172.21.15.98:0/2432916373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc1144d0 msgr2=0x7ffafc1168e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.901+0000 7ffb025c7640 1 --2- 172.21.15.98:0/2432916373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc1144d0 0x7ffafc1168e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffaec009860 tx=0x7ffaec02a340 comp rx=0 tx=0).stop 2024-05-09T08:54:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.901+0000 7ffb025c7640 1 -- 172.21.15.98:0/2432916373 shutdown_connections 2024-05-09T08:54:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.901+0000 7ffb025c7640 1 --2- 172.21.15.98:0/2432916373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc1144d0 0x7ffafc1168e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.901+0000 7ffb025c7640 1 --2- 172.21.15.98:0/2432916373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc108e00 0x7ffafc109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.901+0000 7ffb025c7640 1 --2- 172.21.15.98:0/2432916373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc06dac0 0x7ffafc108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.902+0000 7ffb025c7640 1 -- 172.21.15.98:0/2432916373 >> 172.21.15.98:0/2432916373 conn(0x7ffafc06c740 msgr2=0x7ffafc06cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.902+0000 7ffb025c7640 1 -- 172.21.15.98:0/2432916373 shutdown_connections 2024-05-09T08:54:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.902+0000 7ffb025c7640 1 -- 172.21.15.98:0/2432916373 wait complete. 2024-05-09T08:54:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.903+0000 7ffb025c7640 1 Processor -- start 2024-05-09T08:54:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.903+0000 7ffb025c7640 1 -- start start 2024-05-09T08:54:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.903+0000 7ffb025c7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc06dac0 0x7ffafc10a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.903+0000 7ffb025c7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc108e00 0x7ffafc10a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.903+0000 7ffb025c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc1144d0 0x7ffafc10ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.903+0000 7ffb025c7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc11a1a0 con 0x7ffafc1144d0 2024-05-09T08:54:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.903+0000 7ffb025c7640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc11a020 con 0x7ffafc108e00 2024-05-09T08:54:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.903+0000 7ffb025c7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc11a320 con 0x7ffafc06dac0 2024-05-09T08:54:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.903+0000 7ffb015c5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc06dac0 0x7ffafc10a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.903+0000 7ffb01dc6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc1144d0 0x7ffafc10ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.903+0000 7ffb015c5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc06dac0 0x7ffafc10a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45446/0 (socket says 172.21.15.98:45446) 2024-05-09T08:54:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.903+0000 7ffb01dc6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc1144d0 0x7ffafc10ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43112/0 (socket says 172.21.15.98:43112) 2024-05-09T08:54:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.903+0000 7ffb015c5640 1 -- 172.21.15.98:0/2970025707 learned_addr learned my addr 172.21.15.98:0/2970025707 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.904+0000 7ffb015c5640 1 -- 172.21.15.98:0/2970025707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc108e00 msgr2=0x7ffafc10a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.904+0000 7ffb015c5640 1 --2- 172.21.15.98:0/2970025707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc108e00 0x7ffafc10a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.904+0000 7ffb015c5640 1 -- 172.21.15.98:0/2970025707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc1144d0 msgr2=0x7ffafc10ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.904+0000 7ffb015c5640 1 --2- 172.21.15.98:0/2970025707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc1144d0 0x7ffafc10ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.904+0000 7ffb015c5640 1 -- 172.21.15.98:0/2970025707 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffafc1aea40 con 0x7ffafc06dac0 2024-05-09T08:54:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.904+0000 7ffb015c5640 1 --2- 172.21.15.98:0/2970025707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc06dac0 0x7ffafc10a2a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800e9c0 tx=0x7ffaf800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.904+0000 7ffaf27fc640 1 -- 172.21.15.98:0/2970025707 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf800ccc0 con 0x7ffafc06dac0 2024-05-09T08:54:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.905+0000 7ffaf27fc640 1 -- 172.21.15.98:0/2970025707 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaf8005500 con 0x7ffafc06dac0 2024-05-09T08:54:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.905+0000 7ffb025c7640 1 -- 172.21.15.98:0/2970025707 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffafc1a02f0 con 0x7ffafc06dac0 2024-05-09T08:54:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.905+0000 7ffaf27fc640 1 -- 172.21.15.98:0/2970025707 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8010430 con 0x7ffafc06dac0 2024-05-09T08:54:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.905+0000 7ffb025c7640 1 -- 172.21.15.98:0/2970025707 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffafc1a0830 con 0x7ffafc06dac0 2024-05-09T08:54:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.906+0000 7ffb025c7640 1 -- 172.21.15.98:0/2970025707 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffafc068570 con 0x7ffafc06dac0 2024-05-09T08:54:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.908+0000 7ffaf27fc640 1 -- 172.21.15.98:0/2970025707 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ffaf80046c0 con 0x7ffafc06dac0 2024-05-09T08:54:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.909+0000 7ffaf27fc640 1 --2- 172.21.15.98:0/2970025707 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffae80a2e00 0x7ffae80a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.910+0000 7ffaf27fc640 1 -- 172.21.15.98:0/2970025707 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7ffaf80cfbd0 con 0x7ffafc06dac0 2024-05-09T08:54:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.910+0000 7ffb00dc4640 1 --2- 172.21.15.98:0/2970025707 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffae80a2e00 0x7ffae80a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.912+0000 7ffb00dc4640 1 --2- 172.21.15.98:0/2970025707 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffae80a2e00 0x7ffae80a52c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffafc10bd30 tx=0x7ffaf4009960 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:20.920+0000 7ffaf27fc640 1 -- 172.21.15.98:0/2970025707 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffaf809c3d0 con 0x7ffafc06dac0 2024-05-09T08:54:21.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.454+0000 7ffb025c7640 1 -- 172.21.15.98:0/2970025707 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffafc1a0af0 con 0x7ffafc06dac0 2024-05-09T08:54:21.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.455+0000 7ffaf27fc640 1 -- 172.21.15.98:0/2970025707 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11944 (secure 0 0 0) 0x7ffaf809bcd0 con 0x7ffafc06dac0 2024-05-09T08:54:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:54:21.455 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":16641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:54:06.818680+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25736":{"gid":25736,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/3228987570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3228987570},{"type":"v1","addr":"172.21.15.105:6835","nonce":3228987570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T08:54:21.455 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 33 2024-05-09T08:54:21.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.459+0000 7ffb025c7640 1 -- 172.21.15.98:0/2970025707 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffae80a2e00 msgr2=0x7ffae80a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:21.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.459+0000 7ffb025c7640 1 --2- 172.21.15.98:0/2970025707 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffae80a2e00 0x7ffae80a52c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffafc10bd30 tx=0x7ffaf4009960 comp rx=0 tx=0).stop 2024-05-09T08:54:21.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.459+0000 7ffb025c7640 1 -- 172.21.15.98:0/2970025707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc06dac0 msgr2=0x7ffafc10a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:21.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.459+0000 7ffb025c7640 1 --2- 172.21.15.98:0/2970025707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc06dac0 0x7ffafc10a2a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800e9c0 tx=0x7ffaf800ee80 comp rx=0 tx=0).stop 2024-05-09T08:54:21.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.460+0000 7ffb025c7640 1 -- 172.21.15.98:0/2970025707 shutdown_connections 2024-05-09T08:54:21.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.460+0000 7ffb025c7640 1 --2- 172.21.15.98:0/2970025707 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffae80a2e00 0x7ffae80a52c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.460+0000 7ffb025c7640 1 --2- 172.21.15.98:0/2970025707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc1144d0 0x7ffafc10ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.460+0000 7ffb025c7640 1 --2- 172.21.15.98:0/2970025707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffafc108e00 0x7ffafc10a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.460+0000 7ffb025c7640 1 --2- 172.21.15.98:0/2970025707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffafc06dac0 0x7ffafc10a2a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.460+0000 7ffb025c7640 1 -- 172.21.15.98:0/2970025707 >> 172.21.15.98:0/2970025707 conn(0x7ffafc06c740 msgr2=0x7ffafc0700a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:21.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.460+0000 7ffb025c7640 1 -- 172.21.15.98:0/2970025707 shutdown_connections 2024-05-09T08:54:21.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.460+0000 7ffb025c7640 1 -- 172.21.15.98:0/2970025707 wait complete. 2024-05-09T08:54:21.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:54:21.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.880+0000 7f7949809640 1 -- 172.21.15.98:0/4149886410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f794411b010 msgr2=0x7f794411d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.880+0000 7f7949809640 1 --2- 172.21.15.98:0/4149886410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f794411b010 0x7f794411d400 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f793c009c10 tx=0x7f793c02a6e0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.885+0000 7f7949809640 1 -- 172.21.15.98:0/4149886410 shutdown_connections 2024-05-09T08:54:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.885+0000 7f7949809640 1 --2- 172.21.15.98:0/4149886410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f794411b010 0x7f794411d400 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.885+0000 7f7949809640 1 --2- 172.21.15.98:0/4149886410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f794410e0b0 0x7f794410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.885+0000 7f7949809640 1 --2- 172.21.15.98:0/4149886410 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7944110080 0x7f794410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.885+0000 7f7949809640 1 -- 172.21.15.98:0/4149886410 >> 172.21.15.98:0/4149886410 conn(0x7f794406c940 msgr2=0x7f794406cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:21.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.887+0000 7f7949809640 1 -- 172.21.15.98:0/4149886410 shutdown_connections 2024-05-09T08:54:21.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.887+0000 7f7949809640 1 -- 172.21.15.98:0/4149886410 wait complete. 2024-05-09T08:54:21.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.888+0000 7f7949809640 1 Processor -- start 2024-05-09T08:54:21.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.888+0000 7f7949809640 1 -- start start 2024-05-09T08:54:21.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.888+0000 7f7949809640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f794410e0b0 0x7f79441a6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.888+0000 7f7949809640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7944110080 0x7f79441a7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.888+0000 7f7949809640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f794411b010 0x7f79441ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.888+0000 7f7949809640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79441209d0 con 0x7f794411b010 2024-05-09T08:54:21.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.888+0000 7f7949809640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7944120850 con 0x7f7944110080 2024-05-09T08:54:21.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.888+0000 7f7949809640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7944120b50 con 0x7f794410e0b0 2024-05-09T08:54:21.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.890+0000 7f79437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f794411b010 0x7f79441ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:21.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.890+0000 7f79437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f794411b010 0x7f79441ab7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43132/0 (socket says 172.21.15.98:43132) 2024-05-09T08:54:21.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.890+0000 7f79437fe640 1 -- 172.21.15.98:0/1886389301 learned_addr learned my addr 172.21.15.98:0/1886389301 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:21.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.891+0000 7f79427fc640 1 --2- 172.21.15.98:0/1886389301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7944110080 0x7f79441a7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:21.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.892+0000 7f7942ffd640 1 --2- 172.21.15.98:0/1886389301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f794410e0b0 0x7f79441a6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:21.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.892+0000 7f79437fe640 1 -- 172.21.15.98:0/1886389301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f794410e0b0 msgr2=0x7f79441a6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:21.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.892+0000 7f79437fe640 1 --2- 172.21.15.98:0/1886389301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f794410e0b0 0x7f79441a6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.892+0000 7f79437fe640 1 -- 172.21.15.98:0/1886389301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7944110080 msgr2=0x7f79441a7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:21.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.892+0000 7f79437fe640 1 --2- 172.21.15.98:0/1886389301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7944110080 0x7f79441a7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.892+0000 7f79437fe640 1 -- 172.21.15.98:0/1886389301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79441abec0 con 0x7f794411b010 2024-05-09T08:54:21.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.893+0000 7f79437fe640 1 --2- 172.21.15.98:0/1886389301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f794411b010 0x7f79441ab7c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f793c034040 tx=0x7f793c030640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:21.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.893+0000 7f7948807640 1 -- 172.21.15.98:0/1886389301 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c041030 con 0x7f794411b010 2024-05-09T08:54:21.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.894+0000 7f7949809640 1 -- 172.21.15.98:0/1886389301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79441ac150 con 0x7f794411b010 2024-05-09T08:54:21.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.894+0000 7f7948807640 1 -- 172.21.15.98:0/1886389301 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f793c003e50 con 0x7f794411b010 2024-05-09T08:54:21.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.894+0000 7f7948807640 1 -- 172.21.15.98:0/1886389301 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c036040 con 0x7f794411b010 2024-05-09T08:54:21.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.895+0000 7f7949809640 1 -- 172.21.15.98:0/1886389301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79441107f0 con 0x7f794411b010 2024-05-09T08:54:21.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.901+0000 7f794c5f1740 1 Processor -- start 2024-05-09T08:54:21.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.904+0000 7f794c5f1740 1 -- start start 2024-05-09T08:54:21.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.904+0000 7f794c5f1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561506671330 0x561506671710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.905+0000 7f794c5f1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671c50 0x56150667bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.905+0000 7f79427fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561506671330 0x561506671710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:21.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.905+0000 7f79427fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561506671330 0x561506671710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45462/0 (socket says 172.21.15.98:45462) 2024-05-09T08:54:21.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.905+0000 7f79427fc640 1 -- 172.21.15.98:0/1358354771 learned_addr learned my addr 172.21.15.98:0/1358354771 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:21.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.905+0000 7f794c5f1740 1 --2- 172.21.15.98:0/1358354771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56150667c320 0x56150667e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.905+0000 7f794c5f1740 1 -- 172.21.15.98:0/1358354771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5615065d6c40 con 0x56150667c320 2024-05-09T08:54:21.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.905+0000 7f794c5f1740 1 -- 172.21.15.98:0/1358354771 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56150657e580 con 0x561506671c50 2024-05-09T08:54:21.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.905+0000 7f794c5f1740 1 -- 172.21.15.98:0/1358354771 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5615065d2c00 con 0x561506671330 2024-05-09T08:54:21.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.905+0000 7f79427fc640 1 -- 172.21.15.98:0/1358354771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671c50 msgr2=0x56150667bde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:54:21.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.905+0000 7f79427fc640 1 --2- 172.21.15.98:0/1358354771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671c50 0x56150667bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.905+0000 7f79427fc640 1 -- 172.21.15.98:0/1358354771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56150667c320 msgr2=0x56150667e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:21.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.905+0000 7f79427fc640 1 --2- 172.21.15.98:0/1358354771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56150667c320 0x56150667e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.905+0000 7f79427fc640 1 -- 172.21.15.98:0/1358354771 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615062ea3f0 con 0x561506671330 2024-05-09T08:54:21.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.906+0000 7f79427fc640 1 --2- 172.21.15.98:0/1358354771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561506671330 0x561506671710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7938009da0 tx=0x7f7938013830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7844f41e8cdbefed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:21.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.906+0000 7f7948807640 1 -- 172.21.15.98:0/1886389301 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f793c044070 con 0x7f794411b010 2024-05-09T08:54:21.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.908+0000 7f7948807640 1 --2- 172.21.15.98:0/1886389301 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f792c0a2f30 0x7f792c0a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.908+0000 7f7948807640 1 -- 172.21.15.98:0/1886389301 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7f793c0efef0 con 0x7f794411b010 2024-05-09T08:54:21.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.909+0000 7f7942ffd640 1 --2- 172.21.15.98:0/1886389301 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f792c0a2f30 0x7f792c0a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:21.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.908+0000 7f7929ffb640 1 -- 172.21.15.98:0/1358354771 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7938004050 con 0x561506671330 2024-05-09T08:54:21.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.910+0000 7f7929ffb640 1 -- 172.21.15.98:0/1358354771 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7938017030 con 0x561506671330 2024-05-09T08:54:21.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.910+0000 7f7929ffb640 1 -- 172.21.15.98:0/1358354771 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7938010690 con 0x561506671330 2024-05-09T08:54:21.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.912+0000 7f794c5f1740 1 -- 172.21.15.98:0/1358354771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561506671330 msgr2=0x561506671710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:21.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.912+0000 7f794c5f1740 1 --2- 172.21.15.98:0/1358354771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561506671330 0x561506671710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7938009da0 tx=0x7f7938013830 comp rx=0 tx=0).stop 2024-05-09T08:54:21.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.912+0000 7f794c5f1740 1 -- 172.21.15.98:0/1358354771 shutdown_connections 2024-05-09T08:54:21.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.912+0000 7f794c5f1740 1 --2- 172.21.15.98:0/1358354771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56150667c320 0x56150667e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.912+0000 7f794c5f1740 1 --2- 172.21.15.98:0/1358354771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671c50 0x56150667bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.912+0000 7f794c5f1740 1 --2- 172.21.15.98:0/1358354771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561506671330 0x561506671710 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.912+0000 7f794c5f1740 1 -- 172.21.15.98:0/1358354771 >> 172.21.15.98:0/1358354771 conn(0x56150666d650 msgr2=0x56150666da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:21.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.912+0000 7f794c5f1740 1 -- 172.21.15.98:0/1358354771 shutdown_connections 2024-05-09T08:54:21.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.913+0000 7f794c5f1740 1 -- 172.21.15.98:0/1358354771 wait complete. 2024-05-09T08:54:21.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.913+0000 7f794c5f1740 1 -- start start 2024-05-09T08:54:21.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.913+0000 7f794c5f1740 1 Processor -- start 2024-05-09T08:54:21.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.914+0000 7f794c5f1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671330 0x5615066792f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.914+0000 7f794c5f1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561506671c50 0x561506679830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.914+0000 7f794c5f1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x561506679d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.915+0000 7f794c5f1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56150667f110 con 0x561506671c50 2024-05-09T08:54:21.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.915+0000 7f79437fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x561506679d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:21.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.915+0000 7f79437fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x561506679d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45466/0 (socket says 172.21.15.98:45466) 2024-05-09T08:54:21.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.915+0000 7f79437fe640 1 -- 172.21.15.98:0/3427743998 learned_addr learned my addr 172.21.15.98:0/3427743998 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:21.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.915+0000 7f794c5f1740 1 -- 172.21.15.98:0/3427743998 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x561506591c40 con 0x561506671330 2024-05-09T08:54:21.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.915+0000 7f794c5f1740 1 -- 172.21.15.98:0/3427743998 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56150667f290 con 0x56150667c320 2024-05-09T08:54:21.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.915+0000 7f794c5f1740 10 client.0 _ll_register_callbacks cb 0x5615062e9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:21.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.915+0000 7f79437fe640 1 -- 172.21.15.98:0/3427743998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671330 msgr2=0x5615066792f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:54:21.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.915+0000 7f79437fe640 1 --2- 172.21.15.98:0/3427743998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671330 0x5615066792f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.915+0000 7f79437fe640 1 -- 172.21.15.98:0/3427743998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561506671c50 msgr2=0x561506679830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:21.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.915+0000 7f79437fe640 1 --2- 172.21.15.98:0/3427743998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561506671c50 0x561506679830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.915+0000 7f79437fe640 1 -- 172.21.15.98:0/3427743998 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615063b47d0 con 0x56150667c320 2024-05-09T08:54:21.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.916+0000 7f79437fe640 1 --2- 172.21.15.98:0/3427743998 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x561506679d70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f793c0c4730 tx=0x7f793c0bbdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ae4e6c1c8410bda server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:21.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.916+0000 7f794c5f1740 1 -- 172.21.15.98:0/3427743998 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7938017030 con 0x56150667c320 2024-05-09T08:54:21.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.916+0000 7f7922ffd640 10 client.25960 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:54:21.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.917+0000 7f7922ffd640 1 -- 172.21.15.98:0/3427743998 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c0dc0c0 con 0x56150667c320 2024-05-09T08:54:21.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.917+0000 7f7922ffd640 1 -- 172.21.15.98:0/3427743998 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7f793c0da890 con 0x56150667c320 2024-05-09T08:54:21.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.917+0000 7f7922ffd640 1 -- 172.21.15.98:0/3427743998 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f793c0c9050 con 0x56150667c320 2024-05-09T08:54:21.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.917+0000 7f794c5f1740 10 client.25960 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:21.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.917+0000 7f794c5f1740 1 -- 172.21.15.98:0/3427743998 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({fsmap=33}) -- 0x56150668b1f0 con 0x56150667c320 2024-05-09T08:54:21.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.918+0000 7f7942ffd640 1 --2- 172.21.15.98:0/1886389301 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f792c0a2f30 0x7f792c0a53f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7934005d80 tx=0x7f7934005c90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:21.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.919+0000 7f7922ffd640 1 -- 172.21.15.98:0/3427743998 <== mon.1 v2:172.21.15.142:3300/0 4 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f793c0ed4a0 con 0x56150667c320 2024-05-09T08:54:21.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.919+0000 7f794c5f1740 10 client.25960 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:21.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.919+0000 7f794c5f1740 10 client.25960 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:54:21.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.919+0000 7f794c5f1740 20 client.25960 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:21.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.919+0000 7f794c5f1740 1 --2- 172.21.15.98:0/3427743998 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x56150668c650 0x56150668eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.919+0000 7f794c5f1740 4 client.25960 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:21.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.919+0000 7f794c5f1740 1 -- 172.21.15.98:0/3427743998 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7938017030 con 0x56150668c650 2024-05-09T08:54:21.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.919+0000 7f7942ffd640 1 --2- 172.21.15.98:0/3427743998 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x56150668c650 0x56150668eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:21.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.920+0000 7f7942ffd640 1 --2- 172.21.15.98:0/3427743998 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x56150668c650 0x56150668eb10 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1c88ad6c304105c9 server_cookie=e13d946d1ec467af in_seq=0 out_seq=0 2024-05-09T08:54:21.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.920+0000 7f7922ffd640 10 client.25960 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:54:21.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.922+0000 7f7922ffd640 1 -- 172.21.15.98:0/3427743998 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f7938017030 con 0x56150668c650 2024-05-09T08:54:21.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.922+0000 7f7922ffd640 10 client.25960 handle_command_reply: tid=0 2024-05-09T08:54:21.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.922+0000 7f794c5f1740 1 client.25960 shutdown 2024-05-09T08:54:21.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.929+0000 7f794c5f1740 1 -- 172.21.15.98:0/3427743998 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 msgr2=0x561506679d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:21.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.929+0000 7f794c5f1740 1 --2- 172.21.15.98:0/3427743998 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x561506679d70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f793c0c4730 tx=0x7f793c0bbdc0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.929+0000 7f794c5f1740 1 -- 172.21.15.98:0/3427743998 shutdown_connections 2024-05-09T08:54:21.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.929+0000 7f794c5f1740 1 --2- 172.21.15.98:0/3427743998 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x56150668c650 0x56150668eb10 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.929+0000 7f794c5f1740 1 --2- 172.21.15.98:0/3427743998 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x561506679d70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.929+0000 7f794c5f1740 1 --2- 172.21.15.98:0/3427743998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561506671c50 0x561506679830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.929+0000 7f794c5f1740 1 --2- 172.21.15.98:0/3427743998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671330 0x5615066792f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.929+0000 7f794c5f1740 1 -- 172.21.15.98:0/3427743998 >> 172.21.15.98:0/3427743998 conn(0x56150666d650 msgr2=0x561506673450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:21.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.929+0000 7f7922ffd640 0 client.25960 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:54:21.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.929+0000 7f794c5f1740 1 -- 172.21.15.98:0/3427743998 shutdown_connections 2024-05-09T08:54:21.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.929+0000 7f794c5f1740 1 -- 172.21.15.98:0/3427743998 wait complete. 2024-05-09T08:54:21.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.930+0000 7f794c5f1740 20 client.25960 trim_cache size 0 max 16384 2024-05-09T08:54:21.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.962+0000 7f794c5f1740 1 Processor -- start 2024-05-09T08:54:21.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.962+0000 7f794c5f1740 1 -- start start 2024-05-09T08:54:21.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.963+0000 7f794c5f1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671330 0x561506687fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.963+0000 7f794c5f1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561506671c50 0x561506679b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.964+0000 7f794c5f1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x5615066943e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.964+0000 7f7942ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671330 0x561506687fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.964+0000 7f7942ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671330 0x561506687fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45396/0 (socket says 172.21.15.98:45396) 2024-05-09T08:54:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.964+0000 7f7942ffd640 1 -- 172.21.15.98:0/928586586 learned_addr learned my addr 172.21.15.98:0/928586586 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.964+0000 7f794c5f1740 1 -- 172.21.15.98:0/928586586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79441c8660 con 0x561506671c50 2024-05-09T08:54:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.964+0000 7f794c5f1740 1 -- 172.21.15.98:0/928586586 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f79441c6450 con 0x561506671330 2024-05-09T08:54:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.964+0000 7f794c5f1740 1 -- 172.21.15.98:0/928586586 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f79441c7f90 con 0x56150667c320 2024-05-09T08:54:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.964+0000 7f79437fe640 1 --2- 172.21.15.98:0/928586586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x5615066943e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.965+0000 7f79427fc640 1 --2- 172.21.15.98:0/928586586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561506671c50 0x561506679b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.965+0000 7f7942ffd640 1 -- 172.21.15.98:0/928586586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 msgr2=0x5615066943e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.965+0000 7f7942ffd640 1 --2- 172.21.15.98:0/928586586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x5615066943e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.965+0000 7f7942ffd640 1 -- 172.21.15.98:0/928586586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561506671c50 msgr2=0x561506679b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.965+0000 7f7942ffd640 1 --2- 172.21.15.98:0/928586586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561506671c50 0x561506679b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.965+0000 7f7942ffd640 1 -- 172.21.15.98:0/928586586 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615066782a0 con 0x561506671330 2024-05-09T08:54:21.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.968+0000 7f7942ffd640 1 --2- 172.21.15.98:0/928586586 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671330 0x561506687fd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7934005fd0 tx=0x7f7934011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f143bc3de6ebed89 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:21.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.969+0000 7f79297fa640 1 -- 172.21.15.98:0/928586586 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7934005020 con 0x561506671330 2024-05-09T08:54:21.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.970+0000 7f79297fa640 1 -- 172.21.15.98:0/928586586 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7934017070 con 0x561506671330 2024-05-09T08:54:21.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.970+0000 7f79297fa640 1 -- 172.21.15.98:0/928586586 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79340241d0 con 0x561506671330 2024-05-09T08:54:21.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.971+0000 7f794c5f1740 1 -- 172.21.15.98:0/928586586 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671330 msgr2=0x561506687fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:21.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.971+0000 7f794c5f1740 1 --2- 172.21.15.98:0/928586586 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671330 0x561506687fd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7934005fd0 tx=0x7f7934011000 comp rx=0 tx=0).stop 2024-05-09T08:54:21.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.971+0000 7f794c5f1740 1 -- 172.21.15.98:0/928586586 shutdown_connections 2024-05-09T08:54:21.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.971+0000 7f794c5f1740 1 --2- 172.21.15.98:0/928586586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x5615066943e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.971+0000 7f794c5f1740 1 --2- 172.21.15.98:0/928586586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561506671c50 0x561506679b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.971+0000 7f794c5f1740 1 --2- 172.21.15.98:0/928586586 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671330 0x561506687fd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.971+0000 7f794c5f1740 1 -- 172.21.15.98:0/928586586 >> 172.21.15.98:0/928586586 conn(0x56150666d650 msgr2=0x56150666da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:21.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.971+0000 7f794c5f1740 1 -- 172.21.15.98:0/928586586 shutdown_connections 2024-05-09T08:54:21.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.971+0000 7f794c5f1740 1 -- 172.21.15.98:0/928586586 wait complete. 2024-05-09T08:54:21.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.972+0000 7f794c5f1740 1 -- start start 2024-05-09T08:54:21.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.972+0000 7f794c5f1740 1 Processor -- start 2024-05-09T08:54:21.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.973+0000 7f794c5f1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561506671330 0x561506684870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.973+0000 7f794c5f1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671c50 0x56150667fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.973+0000 7f794c5f1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x5615066800d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.973+0000 7f794c5f1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79441c71f0 con 0x561506671330 2024-05-09T08:54:21.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.973+0000 7f794c5f1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f79441c6b20 con 0x561506671c50 2024-05-09T08:54:21.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.973+0000 7f794c5f1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f79441c78c0 con 0x56150667c320 2024-05-09T08:54:21.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.973+0000 7f794c5f1740 10 client.0 _ll_register_callbacks cb 0x7f793c030860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:21.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.974+0000 7f79437fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x5615066800d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:21.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.974+0000 7f79437fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x5615066800d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45486/0 (socket says 172.21.15.98:45486) 2024-05-09T08:54:21.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.974+0000 7f79437fe640 1 -- 172.21.15.98:0/3602718951 learned_addr learned my addr 172.21.15.98:0/3602718951 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:21.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.974+0000 7f79427fc640 1 --2- 172.21.15.98:0/3602718951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671c50 0x56150667fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:21.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.974+0000 7f7942ffd640 1 --2- 172.21.15.98:0/3602718951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561506671330 0x561506684870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:21.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.974+0000 7f79437fe640 1 -- 172.21.15.98:0/3602718951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671c50 msgr2=0x56150667fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:21.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.974+0000 7f79437fe640 1 --2- 172.21.15.98:0/3602718951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671c50 0x56150667fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.974+0000 7f79437fe640 1 -- 172.21.15.98:0/3602718951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561506671330 msgr2=0x561506684870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:21.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.974+0000 7f79437fe640 1 --2- 172.21.15.98:0/3602718951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561506671330 0x561506684870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:21.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.974+0000 7f79437fe640 1 -- 172.21.15.98:0/3602718951 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561506671a80 con 0x56150667c320 2024-05-09T08:54:21.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.976+0000 7f79437fe640 1 --2- 172.21.15.98:0/3602718951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x5615066800d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f793c0bbf30 tx=0x7f793c030960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbb8dee4fd82fea7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:21.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.976+0000 7f7922ffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:54:21.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.976+0000 7f7922ffd640 1 -- 172.21.15.98:0/3602718951 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c0dc0c0 con 0x56150667c320 2024-05-09T08:54:21.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.976+0000 7f794c5f1740 1 -- 172.21.15.98:0/3602718951 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7934017070 con 0x56150667c320 2024-05-09T08:54:21.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.976+0000 7f7922ffd640 1 -- 172.21.15.98:0/3602718951 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7f793c0c49e0 con 0x56150667c320 2024-05-09T08:54:21.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.977+0000 7f7922ffd640 1 -- 172.21.15.98:0/3602718951 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f793c0bc6f0 con 0x56150667c320 2024-05-09T08:54:21.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.977+0000 7f794c5f1740 10 client.25966 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:21.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.977+0000 7f794c5f1740 1 -- 172.21.15.98:0/3602718951 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5615066f2a40 con 0x56150667c320 2024-05-09T08:54:21.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.979+0000 7f7922ffd640 1 -- 172.21.15.98:0/3602718951 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7934017070 con 0x56150667c320 2024-05-09T08:54:21.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.980+0000 7f7922ffd640 1 -- 172.21.15.98:0/3602718951 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f793c0bc250 con 0x56150667c320 2024-05-09T08:54:21.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.981+0000 7f794c5f1740 10 client.25966 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:21.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.981+0000 7f794c5f1740 10 client.25966 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:54:21.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.981+0000 7f794c5f1740 20 client.25966 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:21.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.981+0000 7f794c5f1740 1 --2- 172.21.15.98:0/3602718951 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5615066f3000 0x5615066f5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:21.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.981+0000 7f794c5f1740 4 client.25966 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:54:21.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.981+0000 7f794c5f1740 1 -- 172.21.15.98:0/3602718951 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5615065e4370 con 0x5615066f3000 2024-05-09T08:54:21.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.982+0000 7f7942ffd640 1 --2- 172.21.15.98:0/3602718951 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5615066f3000 0x5615066f5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:21.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.992+0000 7f7942ffd640 1 --2- 172.21.15.98:0/3602718951 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5615066f3000 0x5615066f5420 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=df757107b0cd3a04 server_cookie=84faa89434bacaba in_seq=0 out_seq=0 2024-05-09T08:54:21.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.992+0000 7f7922ffd640 10 client.25966 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:54:21.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.993+0000 7f7922ffd640 1 -- 172.21.15.98:0/3602718951 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5615065e4370 con 0x5615066f3000 2024-05-09T08:54:21.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.993+0000 7f7922ffd640 10 client.25966 handle_command_reply: tid=0 2024-05-09T08:54:21.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:21.994+0000 7f794c5f1740 1 client.25966 shutdown 2024-05-09T08:54:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.002+0000 7f794c5f1740 1 -- 172.21.15.98:0/3602718951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 msgr2=0x5615066800d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.002+0000 7f794c5f1740 1 --2- 172.21.15.98:0/3602718951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x5615066800d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f793c0bbf30 tx=0x7f793c030960 comp rx=0 tx=0).stop 2024-05-09T08:54:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.002+0000 7f794c5f1740 1 -- 172.21.15.98:0/3602718951 shutdown_connections 2024-05-09T08:54:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.002+0000 7f794c5f1740 1 --2- 172.21.15.98:0/3602718951 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5615066f3000 0x5615066f5420 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.002+0000 7f794c5f1740 1 --2- 172.21.15.98:0/3602718951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56150667c320 0x5615066800d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.002+0000 7f794c5f1740 1 --2- 172.21.15.98:0/3602718951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561506671c50 0x56150667fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.002+0000 7f794c5f1740 1 --2- 172.21.15.98:0/3602718951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561506671330 0x561506684870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.002+0000 7f794c5f1740 1 -- 172.21.15.98:0/3602718951 >> 172.21.15.98:0/3602718951 conn(0x56150666d650 msgr2=0x56150667a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.002+0000 7f7922ffd640 0 client.25966 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:54:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.002+0000 7f794c5f1740 1 -- 172.21.15.98:0/3602718951 shutdown_connections 2024-05-09T08:54:22.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.002+0000 7f794c5f1740 1 -- 172.21.15.98:0/3602718951 wait complete. 2024-05-09T08:54:22.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.004+0000 7f794c5f1740 20 client.25966 trim_cache size 0 max 16384 2024-05-09T08:54:22.009 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:54:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.021+0000 7f7949809640 1 -- 172.21.15.98:0/1886389301 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f792c0a2f30 msgr2=0x7f792c0a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.021+0000 7f7949809640 1 --2- 172.21.15.98:0/1886389301 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f792c0a2f30 0x7f792c0a53f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7934005d80 tx=0x7f7934005c90 comp rx=0 tx=0).stop 2024-05-09T08:54:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.021+0000 7f7949809640 1 -- 172.21.15.98:0/1886389301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f794411b010 msgr2=0x7f79441ab7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.021+0000 7f7949809640 1 --2- 172.21.15.98:0/1886389301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f794411b010 0x7f79441ab7c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f793c034040 tx=0x7f793c030640 comp rx=0 tx=0).stop 2024-05-09T08:54:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.022+0000 7f7949809640 1 -- 172.21.15.98:0/1886389301 shutdown_connections 2024-05-09T08:54:22.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.022+0000 7f7949809640 1 --2- 172.21.15.98:0/1886389301 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f792c0a2f30 0x7f792c0a53f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.022+0000 7f7949809640 1 --2- 172.21.15.98:0/1886389301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f794411b010 0x7f79441ab7c0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.022+0000 7f7949809640 1 --2- 172.21.15.98:0/1886389301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7944110080 0x7f79441a7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.022+0000 7f7949809640 1 --2- 172.21.15.98:0/1886389301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f794410e0b0 0x7f79441a6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.022+0000 7f7949809640 1 -- 172.21.15.98:0/1886389301 >> 172.21.15.98:0/1886389301 conn(0x7f794406c940 msgr2=0x7f7944072330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:22.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.023+0000 7f7949809640 1 -- 172.21.15.98:0/1886389301 shutdown_connections 2024-05-09T08:54:22.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.023+0000 7f7949809640 1 -- 172.21.15.98:0/1886389301 wait complete. 2024-05-09T08:54:22.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:54:22.143 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:21 smithi098 ceph-mon[30092]: pgmap v115: 129 pgs: 129 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 759 MiB/s wr, 4.00k op/s 2024-05-09T08:54:22.143 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:21 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2970025707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:54:22.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:21 smithi142 ceph-mon[34625]: pgmap v115: 129 pgs: 129 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 759 MiB/s wr, 4.00k op/s 2024-05-09T08:54:22.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:21 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2970025707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:54:22.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:21 smithi105 ceph-mon[34498]: pgmap v115: 129 pgs: 129 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 759 MiB/s wr, 4.00k op/s 2024-05-09T08:54:22.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:21 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2970025707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:54:22.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.337+0000 7ff6a7276640 1 -- 172.21.15.98:0/296824154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6a0100710 msgr2=0x7ff6a0100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.337+0000 7ff6a7276640 1 --2- 172.21.15.98:0/296824154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6a0100710 0x7ff6a0100b90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff69001a940 tx=0x7ff69003b450 comp rx=0 tx=0).stop 2024-05-09T08:54:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.338+0000 7ff6a7276640 1 -- 172.21.15.98:0/296824154 shutdown_connections 2024-05-09T08:54:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.338+0000 7ff6a7276640 1 --2- 172.21.15.98:0/296824154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6a01010d0 0x7ff6a010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.338+0000 7ff6a7276640 1 --2- 172.21.15.98:0/296824154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6a0100710 0x7ff6a0100b90 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.338+0000 7ff6a7276640 1 --2- 172.21.15.98:0/296824154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6a0106720 0x7ff6a0106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.338+0000 7ff6a7276640 1 -- 172.21.15.98:0/296824154 >> 172.21.15.98:0/296824154 conn(0x7ff6a0077790 msgr2=0x7ff6a0077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:22.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.339+0000 7ff6a7276640 1 -- 172.21.15.98:0/296824154 shutdown_connections 2024-05-09T08:54:22.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.340+0000 7ff6a7276640 1 -- 172.21.15.98:0/296824154 wait complete. 2024-05-09T08:54:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.341+0000 7ff6a7276640 1 Processor -- start 2024-05-09T08:54:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.341+0000 7ff6a7276640 1 -- start start 2024-05-09T08:54:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.342+0000 7ff6a7276640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6a0100710 0x7ff6a0072ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.342+0000 7ff6a7276640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6a01010d0 0x7ff6a0073400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.342+0000 7ff6a7276640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6a0106720 0x7ff6a006e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.342+0000 7ff6a7276640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a006caf0 con 0x7ff6a0100710 2024-05-09T08:54:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.342+0000 7ff6a7276640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a006c970 con 0x7ff6a0106720 2024-05-09T08:54:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.342+0000 7ff6a7276640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a006cc70 con 0x7ff6a01010d0 2024-05-09T08:54:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.342+0000 7ff6a5813640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6a0106720 0x7ff6a006e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.342+0000 7ff6a5813640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6a0106720 0x7ff6a006e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45436/0 (socket says 172.21.15.98:45436) 2024-05-09T08:54:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.342+0000 7ff6a5813640 1 -- 172.21.15.98:0/3641206655 learned_addr learned my addr 172.21.15.98:0/3641206655 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:22.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.344+0000 7ff6a5813640 1 -- 172.21.15.98:0/3641206655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6a01010d0 msgr2=0x7ff6a0073400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:54:22.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.344+0000 7ff6a5813640 1 --2- 172.21.15.98:0/3641206655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6a01010d0 0x7ff6a0073400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.344+0000 7ff6a5813640 1 -- 172.21.15.98:0/3641206655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6a0100710 msgr2=0x7ff6a0072ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.344+0000 7ff6a5813640 1 --2- 172.21.15.98:0/3641206655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6a0100710 0x7ff6a0072ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.344+0000 7ff6a5813640 1 -- 172.21.15.98:0/3641206655 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a006e920 con 0x7ff6a0106720 2024-05-09T08:54:22.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.344+0000 7ff6a5813640 1 --2- 172.21.15.98:0/3641206655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6a0106720 0x7ff6a006e060 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff69800eb10 tx=0x7ff69800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:22.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.345+0000 7ff6967fc640 1 -- 172.21.15.98:0/3641206655 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff698004b20 con 0x7ff6a0106720 2024-05-09T08:54:22.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.345+0000 7ff6967fc640 1 -- 172.21.15.98:0/3641206655 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff698004cc0 con 0x7ff6a0106720 2024-05-09T08:54:22.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.345+0000 7ff6967fc640 1 -- 172.21.15.98:0/3641206655 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff698010790 con 0x7ff6a0106720 2024-05-09T08:54:22.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.345+0000 7ff6a7276640 1 -- 172.21.15.98:0/3641206655 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a006ec10 con 0x7ff6a0106720 2024-05-09T08:54:22.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.345+0000 7ff6a7276640 1 -- 172.21.15.98:0/3641206655 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a010e3b0 con 0x7ff6a0106720 2024-05-09T08:54:22.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.350+0000 7ff6967fc640 1 -- 172.21.15.98:0/3641206655 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff698005000 con 0x7ff6a0106720 2024-05-09T08:54:22.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.350+0000 7ff6a9f88740 1 Processor -- start 2024-05-09T08:54:22.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.351+0000 7ff6a9f88740 1 -- start start 2024-05-09T08:54:22.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.353+0000 7ff6a9f88740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a15330 0x55f4a0a15710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.353+0000 7ff6a9f88740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 0x55f4a0a1fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.353+0000 7ff6a9f88740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a20320 0x55f4a0a22710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.353+0000 7ff6a9f88740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f4a097ac40 con 0x55f4a0a20320 2024-05-09T08:54:22.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.353+0000 7ff6a9f88740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f4a0922580 con 0x55f4a0a15330 2024-05-09T08:54:22.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.353+0000 7ff6a9f88740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f4a0976c00 con 0x55f4a0a15c50 2024-05-09T08:54:22.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.353+0000 7ff6a5012640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 0x55f4a0a1fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.353+0000 7ff6a5012640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 0x55f4a0a1fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45500/0 (socket says 172.21.15.98:45500) 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.353+0000 7ff6a5012640 1 -- 172.21.15.98:0/735925865 learned_addr learned my addr 172.21.15.98:0/735925865 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.353+0000 7ff6a4811640 1 --2- 172.21.15.98:0/735925865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a15330 0x55f4a0a15710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.353+0000 7ff6a4811640 1 --2- 172.21.15.98:0/735925865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a20320 0x55f4a0a22710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.354+0000 7ff6a5012640 1 -- 172.21.15.98:0/735925865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a15330 msgr2=0x55f4a0a15710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.354+0000 7ff6a5012640 1 --2- 172.21.15.98:0/735925865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a15330 0x55f4a0a15710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.354+0000 7ff6a5012640 1 -- 172.21.15.98:0/735925865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a20320 msgr2=0x55f4a0a22710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.354+0000 7ff6a5012640 1 --2- 172.21.15.98:0/735925865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a20320 0x55f4a0a22710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.354+0000 7ff6a5012640 1 -- 172.21.15.98:0/735925865 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4a068e3f0 con 0x55f4a0a15c50 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.354+0000 7ff6a4811640 1 --2- 172.21.15.98:0/735925865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a20320 0x55f4a0a22710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.354+0000 7ff6967fc640 1 --2- 172.21.15.98:0/3641206655 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff68c0a2f10 0x7ff68c0a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.354+0000 7ff6a5012640 1 --2- 172.21.15.98:0/735925865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 0x55f4a0a1fde0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00aa10 tx=0x7ff69c00aed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fea57caf4aa7b092 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.354+0000 7ff68b7fe640 1 -- 172.21.15.98:0/735925865 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c00cde0 con 0x55f4a0a15c50 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.355+0000 7ff68b7fe640 1 -- 172.21.15.98:0/735925865 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff69c0035f0 con 0x55f4a0a15c50 2024-05-09T08:54:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.355+0000 7ff68b7fe640 1 -- 172.21.15.98:0/735925865 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c00f690 con 0x55f4a0a15c50 2024-05-09T08:54:22.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.355+0000 7ff6967fc640 1 -- 172.21.15.98:0/3641206655 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7ff6980cfef0 con 0x7ff6a0106720 2024-05-09T08:54:22.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.356+0000 7ff6a4811640 1 --2- 172.21.15.98:0/3641206655 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff68c0a2f10 0x7ff68c0a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.356+0000 7ff6a9f88740 1 -- 172.21.15.98:0/735925865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 msgr2=0x55f4a0a1fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.356+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/735925865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 0x55f4a0a1fde0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00aa10 tx=0x7ff69c00aed0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.357+0000 7ff6a4811640 1 --2- 172.21.15.98:0/3641206655 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff68c0a2f10 0x7ff68c0a53d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff69001a910 tx=0x7ff690000bc0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:22.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.358+0000 7ff6a9f88740 1 -- 172.21.15.98:0/735925865 shutdown_connections 2024-05-09T08:54:22.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.358+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/735925865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a20320 0x55f4a0a22710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.358+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/735925865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 0x55f4a0a1fde0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.358+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/735925865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a15330 0x55f4a0a15710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.358+0000 7ff6a9f88740 1 -- 172.21.15.98:0/735925865 >> 172.21.15.98:0/735925865 conn(0x55f4a0a11650 msgr2=0x55f4a0a11a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:22.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.359+0000 7ff6a9f88740 1 -- 172.21.15.98:0/735925865 shutdown_connections 2024-05-09T08:54:22.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.359+0000 7ff6a9f88740 1 -- 172.21.15.98:0/735925865 wait complete. 2024-05-09T08:54:22.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.360+0000 7ff6a9f88740 1 -- start start 2024-05-09T08:54:22.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.360+0000 7ff6a9f88740 1 Processor -- start 2024-05-09T08:54:22.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.361+0000 7ff6a9f88740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a1d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.362+0000 7ff6a9f88740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 0x55f4a0a1d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.362+0000 7ff6a9f88740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a20320 0x55f4a0a1dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.362+0000 7ff6a9f88740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f4a0a23110 con 0x55f4a0a15330 2024-05-09T08:54:22.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.362+0000 7ff6a9f88740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f4a0935c40 con 0x55f4a0a20320 2024-05-09T08:54:22.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.362+0000 7ff6a9f88740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f4a0a23290 con 0x55f4a0a15c50 2024-05-09T08:54:22.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.362+0000 7ff6a9f88740 10 client.0 _ll_register_callbacks cb 0x55f4a068dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:22.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.365+0000 7ff6a5813640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a20320 0x55f4a0a1dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.365+0000 7ff6a5012640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a1d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.365+0000 7ff6a5012640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a1d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43220/0 (socket says 172.21.15.98:43220) 2024-05-09T08:54:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.365+0000 7ff6a5012640 1 -- 172.21.15.98:0/1196055874 learned_addr learned my addr 172.21.15.98:0/1196055874 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.365+0000 7ff6a4811640 1 --2- 172.21.15.98:0/1196055874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 0x55f4a0a1d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.366+0000 7ff6a5012640 1 -- 172.21.15.98:0/1196055874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 msgr2=0x55f4a0a1d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.366+0000 7ff6a5012640 1 --2- 172.21.15.98:0/1196055874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 0x55f4a0a1d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.366+0000 7ff6a5012640 1 -- 172.21.15.98:0/1196055874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a20320 msgr2=0x55f4a0a1dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:22.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.366+0000 7ff6a5012640 1 --2- 172.21.15.98:0/1196055874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a20320 0x55f4a0a1dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.366+0000 7ff6a5012640 1 -- 172.21.15.98:0/1196055874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4a07587d0 con 0x55f4a0a15330 2024-05-09T08:54:22.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.367+0000 7ff6a5012640 1 --2- 172.21.15.98:0/1196055874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a1d270 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff69c004700 tx=0x7ff69c0039d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7b65af0ab547a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.368+0000 7ff688ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:54:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.368+0000 7ff688ff9640 1 -- 172.21.15.98:0/1196055874 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c028060 con 0x55f4a0a15330 2024-05-09T08:54:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.368+0000 7ff6a9f88740 1 -- 172.21.15.98:0/1196055874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff69c0035f0 con 0x55f4a0a15330 2024-05-09T08:54:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.369+0000 7ff688ff9640 1 -- 172.21.15.98:0/1196055874 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7ff69c004230 con 0x55f4a0a15330 2024-05-09T08:54:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.370+0000 7ff688ff9640 1 -- 172.21.15.98:0/1196055874 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff69c02f050 con 0x55f4a0a15330 2024-05-09T08:54:22.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.371+0000 7ff6a9f88740 10 client.16701 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:22.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.371+0000 7ff6a9f88740 1 -- 172.21.15.98:0/1196055874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f4a0a2f320 con 0x55f4a0a15330 2024-05-09T08:54:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.372+0000 7ff688ff9640 1 -- 172.21.15.98:0/1196055874 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff69c00a630 con 0x55f4a0a15330 2024-05-09T08:54:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.372+0000 7ff688ff9640 1 -- 172.21.15.98:0/1196055874 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ff69c023410 con 0x55f4a0a15330 2024-05-09T08:54:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.373+0000 7ff6a9f88740 10 client.16701 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.373+0000 7ff6a9f88740 10 client.16701 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:54:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.373+0000 7ff6a9f88740 20 client.16701 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.374+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/1196055874 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f4a0a30b70 0x55f4a0a32ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.374+0000 7ff6a9f88740 4 client.16701 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.374+0000 7ff6a9f88740 1 -- 172.21.15.98:0/1196055874 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff69c0035f0 con 0x55f4a0a30b70 2024-05-09T08:54:22.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.377+0000 7ff6a4811640 1 --2- 172.21.15.98:0/1196055874 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f4a0a30b70 0x55f4a0a32ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.393+0000 7ff6a4811640 1 --2- 172.21.15.98:0/1196055874 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f4a0a30b70 0x55f4a0a32ff0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e90a8b83c2745fb0 server_cookie=474171881604b900 in_seq=0 out_seq=0 2024-05-09T08:54:22.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.393+0000 7ff688ff9640 10 client.16701 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:54:22.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.396+0000 7ff688ff9640 1 -- 172.21.15.98:0/1196055874 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7ff69c0035f0 con 0x55f4a0a30b70 2024-05-09T08:54:22.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.396+0000 7ff688ff9640 10 client.16701 handle_command_reply: tid=0 2024-05-09T08:54:22.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.396+0000 7ff6a9f88740 1 client.16701 shutdown 2024-05-09T08:54:22.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.397+0000 7ff6a9f88740 1 -- 172.21.15.98:0/1196055874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 msgr2=0x55f4a0a1d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.397+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/1196055874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a1d270 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff69c004700 tx=0x7ff69c0039d0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.400+0000 7ff6a9f88740 1 -- 172.21.15.98:0/1196055874 shutdown_connections 2024-05-09T08:54:22.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.401+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/1196055874 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f4a0a30b70 0x55f4a0a32ff0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.401+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/1196055874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a20320 0x55f4a0a1dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.401+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/1196055874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 0x55f4a0a1d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.401+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/1196055874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a1d270 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.401+0000 7ff6a9f88740 1 -- 172.21.15.98:0/1196055874 >> 172.21.15.98:0/1196055874 conn(0x55f4a0a11650 msgr2=0x55f4a0a17450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.401+0000 7ff688ff9640 0 client.16701 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:54:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.404+0000 7ff6a9f88740 1 -- 172.21.15.98:0/1196055874 shutdown_connections 2024-05-09T08:54:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.404+0000 7ff6a9f88740 1 -- 172.21.15.98:0/1196055874 wait complete. 2024-05-09T08:54:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.406+0000 7ff6a9f88740 20 client.16701 trim_cache size 0 max 16384 2024-05-09T08:54:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.461+0000 7ff6a9f88740 1 Processor -- start 2024-05-09T08:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.462+0000 7ff6a9f88740 1 -- start start 2024-05-09T08:54:22.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.462+0000 7ff6a9f88740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a2bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.462+0000 7ff6a9f88740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 0x55f4a0a38900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.462+0000 7ff6a9f88740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a20320 0x55f4a0a38e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.462+0000 7ff6a9f88740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a01d7320 con 0x55f4a0a15330 2024-05-09T08:54:22.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.462+0000 7ff6a9f88740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a01d8e60 con 0x55f4a0a20320 2024-05-09T08:54:22.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.462+0000 7ff6a9f88740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a01d79f0 con 0x55f4a0a15c50 2024-05-09T08:54:22.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.463+0000 7ff6a5012640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a2bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.463+0000 7ff6a5012640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a2bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43228/0 (socket says 172.21.15.98:43228) 2024-05-09T08:54:22.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.463+0000 7ff6a5012640 1 -- 172.21.15.98:0/3403706030 learned_addr learned my addr 172.21.15.98:0/3403706030 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:22.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.464+0000 7ff6a5012640 1 -- 172.21.15.98:0/3403706030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 msgr2=0x55f4a0a38900 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:54:22.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.464+0000 7ff6a5012640 1 --2- 172.21.15.98:0/3403706030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 0x55f4a0a38900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.464+0000 7ff6a5012640 1 -- 172.21.15.98:0/3403706030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a20320 msgr2=0x55f4a0a38e40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:54:22.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.464+0000 7ff6a5012640 1 --2- 172.21.15.98:0/3403706030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a20320 0x55f4a0a38e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.464+0000 7ff6a5012640 1 -- 172.21.15.98:0/3403706030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4a0a02d20 con 0x55f4a0a15330 2024-05-09T08:54:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.465+0000 7ff6a5012640 1 --2- 172.21.15.98:0/3403706030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a2bfd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff69c026040 tx=0x7ff69c00aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1d50b7c0caffdf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.466+0000 7ff68affd640 1 -- 172.21.15.98:0/3403706030 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c01c020 con 0x55f4a0a15330 2024-05-09T08:54:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.466+0000 7ff68affd640 1 -- 172.21.15.98:0/3403706030 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff69c0035f0 con 0x55f4a0a15330 2024-05-09T08:54:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.467+0000 7ff6a9f88740 1 -- 172.21.15.98:0/3403706030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 msgr2=0x55f4a0a2bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.467+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/3403706030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a2bfd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff69c026040 tx=0x7ff69c00aed0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.469+0000 7ff6a9f88740 1 -- 172.21.15.98:0/3403706030 shutdown_connections 2024-05-09T08:54:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.470+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/3403706030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a20320 0x55f4a0a38e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.470+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/3403706030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a15c50 0x55f4a0a38900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.470+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/3403706030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a2bfd0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.470+0000 7ff6a9f88740 1 -- 172.21.15.98:0/3403706030 >> 172.21.15.98:0/3403706030 conn(0x55f4a0a11650 msgr2=0x55f4a0a1e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.471+0000 7ff6a9f88740 1 -- 172.21.15.98:0/3403706030 shutdown_connections 2024-05-09T08:54:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.471+0000 7ff6a9f88740 1 -- 172.21.15.98:0/3403706030 wait complete. 2024-05-09T08:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.472+0000 7ff6a9f88740 1 -- start start 2024-05-09T08:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.472+0000 7ff6a9f88740 1 Processor -- start 2024-05-09T08:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.473+0000 7ff6a9f88740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a256a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.473+0000 7ff6a9f88740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a15c50 0x55f4a0a25be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.473+0000 7ff6a9f88740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a20320 0x55f4a0a95170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.473+0000 7ff6a9f88740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a01d8790 con 0x55f4a0a15330 2024-05-09T08:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.473+0000 7ff6a9f88740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a01d80c0 con 0x55f4a0a15c50 2024-05-09T08:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.473+0000 7ff6a9f88740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a0068aa0 con 0x55f4a0a20320 2024-05-09T08:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.473+0000 7ff6a9f88740 10 client.0 _ll_register_callbacks cb 0x7ff69001a790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.474+0000 7ff6a5012640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a256a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.474+0000 7ff6a5012640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a256a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43244/0 (socket says 172.21.15.98:43244) 2024-05-09T08:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.474+0000 7ff6a5012640 1 -- 172.21.15.98:0/2552308945 learned_addr learned my addr 172.21.15.98:0/2552308945 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.475+0000 7ff6a5012640 1 -- 172.21.15.98:0/2552308945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a20320 msgr2=0x55f4a0a95170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.475+0000 7ff6a5012640 1 --2- 172.21.15.98:0/2552308945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a20320 0x55f4a0a95170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.475+0000 7ff6a5012640 1 -- 172.21.15.98:0/2552308945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a15c50 msgr2=0x55f4a0a25be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.475+0000 7ff6a5012640 1 --2- 172.21.15.98:0/2552308945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a15c50 0x55f4a0a25be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.475+0000 7ff6a5012640 1 -- 172.21.15.98:0/2552308945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4a0a956b0 con 0x55f4a0a15330 2024-05-09T08:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.476+0000 7ff6a5012640 1 --2- 172.21.15.98:0/2552308945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a256a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff69c004700 tx=0x7ff69c023d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c91442c97edabc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.476+0000 7ff688ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.476+0000 7ff688ff9640 1 -- 172.21.15.98:0/2552308945 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c01c070 con 0x55f4a0a15330 2024-05-09T08:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.476+0000 7ff688ff9640 1 -- 172.21.15.98:0/2552308945 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7ff69c004440 con 0x55f4a0a15330 2024-05-09T08:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.477+0000 7ff6a9f88740 1 -- 172.21.15.98:0/2552308945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff69c0035f0 con 0x55f4a0a15330 2024-05-09T08:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.477+0000 7ff688ff9640 1 -- 172.21.15.98:0/2552308945 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff69c0039d0 con 0x55f4a0a15330 2024-05-09T08:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.478+0000 7ff6a9f88740 10 client.16707 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.478+0000 7ff6a9f88740 1 -- 172.21.15.98:0/2552308945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f4a0a968b0 con 0x55f4a0a15330 2024-05-09T08:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.480+0000 7ff688ff9640 1 -- 172.21.15.98:0/2552308945 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff69c0035f0 con 0x55f4a0a15330 2024-05-09T08:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.480+0000 7ff688ff9640 1 -- 172.21.15.98:0/2552308945 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ff69c00fad0 con 0x55f4a0a15330 2024-05-09T08:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.482+0000 7ff6a9f88740 10 client.16707 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.482+0000 7ff6a9f88740 10 client.16707 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.482+0000 7ff6a9f88740 20 client.16707 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.482+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/2552308945 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f4a0a96ea0 0x55f4a0a992c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.482+0000 7ff6a9f88740 4 client.16707 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.482+0000 7ff6a9f88740 1 -- 172.21.15.98:0/2552308945 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f4a0988370 con 0x55f4a0a96ea0 2024-05-09T08:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.485+0000 7ff6a4811640 1 --2- 172.21.15.98:0/2552308945 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f4a0a96ea0 0x55f4a0a992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.505+0000 7ff6a4811640 1 --2- 172.21.15.98:0/2552308945 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f4a0a96ea0 0x55f4a0a992c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=aeb5a3887332df5b server_cookie=c0c5bf94ea35b4a6 in_seq=0 out_seq=0 2024-05-09T08:54:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.505+0000 7ff688ff9640 10 client.16707 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:54:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.508+0000 7ff688ff9640 1 -- 172.21.15.98:0/2552308945 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f4a0988370 con 0x55f4a0a96ea0 2024-05-09T08:54:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.508+0000 7ff688ff9640 10 client.16707 handle_command_reply: tid=0 2024-05-09T08:54:22.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.509+0000 7ff6a9f88740 1 client.16707 shutdown 2024-05-09T08:54:22.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.510+0000 7ff6a9f88740 1 -- 172.21.15.98:0/2552308945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 msgr2=0x55f4a0a256a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.510+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/2552308945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a256a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff69c004700 tx=0x7ff69c023d80 comp rx=0 tx=0).stop 2024-05-09T08:54:22.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.512+0000 7ff6a9f88740 1 -- 172.21.15.98:0/2552308945 shutdown_connections 2024-05-09T08:54:22.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.512+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/2552308945 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55f4a0a96ea0 0x55f4a0a992c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.512+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/2552308945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f4a0a20320 0x55f4a0a95170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.512+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/2552308945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4a0a15c50 0x55f4a0a25be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.512+0000 7ff6a9f88740 1 --2- 172.21.15.98:0/2552308945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f4a0a15330 0x55f4a0a256a0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.512+0000 7ff6a9f88740 1 -- 172.21.15.98:0/2552308945 >> 172.21.15.98:0/2552308945 conn(0x55f4a0a11650 msgr2=0x55f4a0a2b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:22.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.512+0000 7ff688ff9640 0 client.16707 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:54:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.516+0000 7ff6a9f88740 1 -- 172.21.15.98:0/2552308945 shutdown_connections 2024-05-09T08:54:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.520+0000 7ff6a9f88740 1 -- 172.21.15.98:0/2552308945 wait complete. 2024-05-09T08:54:22.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.523+0000 7ff6a9f88740 20 client.16707 trim_cache size 0 max 16384 2024-05-09T08:54:22.526 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:54:22.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.530+0000 7ff6a7276640 1 -- 172.21.15.98:0/3641206655 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff68c0a2f10 msgr2=0x7ff68c0a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.530+0000 7ff6a7276640 1 --2- 172.21.15.98:0/3641206655 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff68c0a2f10 0x7ff68c0a53d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff69001a910 tx=0x7ff690000bc0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.530+0000 7ff6a7276640 1 -- 172.21.15.98:0/3641206655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6a0106720 msgr2=0x7ff6a006e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.530+0000 7ff6a7276640 1 --2- 172.21.15.98:0/3641206655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6a0106720 0x7ff6a006e060 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff69800eb10 tx=0x7ff69800c2d0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.537+0000 7ff6a7276640 1 -- 172.21.15.98:0/3641206655 shutdown_connections 2024-05-09T08:54:22.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.537+0000 7ff6a7276640 1 --2- 172.21.15.98:0/3641206655 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff68c0a2f10 0x7ff68c0a53d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.537+0000 7ff6a7276640 1 --2- 172.21.15.98:0/3641206655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6a0106720 0x7ff6a006e060 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.537+0000 7ff6a7276640 1 --2- 172.21.15.98:0/3641206655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6a01010d0 0x7ff6a0073400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.537+0000 7ff6a7276640 1 --2- 172.21.15.98:0/3641206655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6a0100710 0x7ff6a0072ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.537+0000 7ff6a7276640 1 -- 172.21.15.98:0/3641206655 >> 172.21.15.98:0/3641206655 conn(0x7ff6a0077790 msgr2=0x7ff6a0103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:22.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.540+0000 7ff6a7276640 1 -- 172.21.15.98:0/3641206655 shutdown_connections 2024-05-09T08:54:22.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.543+0000 7ff6a7276640 1 -- 172.21.15.98:0/3641206655 wait complete. 2024-05-09T08:54:22.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:54:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.910+0000 7eff7329e640 1 -- 172.21.15.98:0/4222209486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c11b010 msgr2=0x7eff6c11d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.910+0000 7eff7329e640 1 --2- 172.21.15.98:0/4222209486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c11b010 0x7eff6c11d400 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7eff6801a940 tx=0x7eff6803b470 comp rx=0 tx=0).stop 2024-05-09T08:54:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.913+0000 7eff7329e640 1 -- 172.21.15.98:0/4222209486 shutdown_connections 2024-05-09T08:54:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.913+0000 7eff7329e640 1 --2- 172.21.15.98:0/4222209486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c11b010 0x7eff6c11d400 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.913+0000 7eff7329e640 1 --2- 172.21.15.98:0/4222209486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c10e0b0 0x7eff6c10e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.913+0000 7eff7329e640 1 --2- 172.21.15.98:0/4222209486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff6c110080 0x7eff6c10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.913+0000 7eff7329e640 1 -- 172.21.15.98:0/4222209486 >> 172.21.15.98:0/4222209486 conn(0x7eff6c06c940 msgr2=0x7eff6c06cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:22.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.918+0000 7eff7329e640 1 -- 172.21.15.98:0/4222209486 shutdown_connections 2024-05-09T08:54:22.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.919+0000 7eff7329e640 1 -- 172.21.15.98:0/4222209486 wait complete. 2024-05-09T08:54:22.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.920+0000 7eff7329e640 1 Processor -- start 2024-05-09T08:54:22.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.920+0000 7eff7329e640 1 -- start start 2024-05-09T08:54:22.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.920+0000 7eff7329e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c10e0b0 0x7eff6c11a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.920+0000 7eff7329e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c110080 0x7eff6c116740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.920+0000 7eff7329e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff6c11b010 0x7eff6c116c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.920+0000 7eff7329e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c1131d0 con 0x7eff6c10e0b0 2024-05-09T08:54:22.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.920+0000 7eff7329e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c113050 con 0x7eff6c11b010 2024-05-09T08:54:22.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.920+0000 7eff7329e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c113350 con 0x7eff6c110080 2024-05-09T08:54:22.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.922+0000 7eff70839640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c110080 0x7eff6c116740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.922+0000 7eff70839640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c110080 0x7eff6c116740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45552/0 (socket says 172.21.15.98:45552) 2024-05-09T08:54:22.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.922+0000 7eff70839640 1 -- 172.21.15.98:0/3768229026 learned_addr learned my addr 172.21.15.98:0/3768229026 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:22.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.922+0000 7eff7183b640 1 --2- 172.21.15.98:0/3768229026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff6c11b010 0x7eff6c116c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.924+0000 7eff70839640 1 -- 172.21.15.98:0/3768229026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff6c11b010 msgr2=0x7eff6c116c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.924+0000 7eff70839640 1 --2- 172.21.15.98:0/3768229026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff6c11b010 0x7eff6c116c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.924+0000 7eff70839640 1 -- 172.21.15.98:0/3768229026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c10e0b0 msgr2=0x7eff6c11a660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:54:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.924+0000 7eff70839640 1 --2- 172.21.15.98:0/3768229026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c10e0b0 0x7eff6c11a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.924+0000 7eff70839640 1 -- 172.21.15.98:0/3768229026 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6c117270 con 0x7eff6c110080 2024-05-09T08:54:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.925+0000 7eff70839640 1 --2- 172.21.15.98:0/3768229026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c110080 0x7eff6c116740 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00d980 tx=0x7eff5c00de40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.931+0000 7eff627fc640 1 -- 172.21.15.98:0/3768229026 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c017070 con 0x7eff6c110080 2024-05-09T08:54:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.932+0000 7eff627fc640 1 -- 172.21.15.98:0/3768229026 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff5c009c30 con 0x7eff6c110080 2024-05-09T08:54:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.932+0000 7eff7329e640 1 -- 172.21.15.98:0/3768229026 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff6c117560 con 0x7eff6c110080 2024-05-09T08:54:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.932+0000 7eff7329e640 1 -- 172.21.15.98:0/3768229026 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff6c1cd130 con 0x7eff6c110080 2024-05-09T08:54:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.932+0000 7eff627fc640 1 -- 172.21.15.98:0/3768229026 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c00f620 con 0x7eff6c110080 2024-05-09T08:54:22.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.938+0000 7eff627fc640 1 -- 172.21.15.98:0/3768229026 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7eff5c00f7c0 con 0x7eff6c110080 2024-05-09T08:54:22.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.940+0000 7eff627fc640 1 --2- 172.21.15.98:0/3768229026 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7eff580a2f00 0x7eff580a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.941+0000 7eff627fc640 1 -- 172.21.15.98:0/3768229026 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7eff5c0d04c0 con 0x7eff6c110080 2024-05-09T08:54:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.942+0000 7eff75faf740 1 Processor -- start 2024-05-09T08:54:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.942+0000 7eff75faf740 1 -- start start 2024-05-09T08:54:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.942+0000 7eff75faf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc12d330 0x5564fc12d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.942+0000 7eff75faf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.942+0000 7eff75faf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc138320 0x5564fc13a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.942+0000 7eff75faf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5564fc092c40 con 0x5564fc12d330 2024-05-09T08:54:22.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.942+0000 7eff75faf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5564fc03a580 con 0x5564fc138320 2024-05-09T08:54:22.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.942+0000 7eff75faf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5564fc08ec00 con 0x5564fc12dc50 2024-05-09T08:54:22.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.943+0000 7eff7183b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc138320 0x5564fc13a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.943+0000 7eff7183b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc138320 0x5564fc13a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45514/0 (socket says 172.21.15.98:45514) 2024-05-09T08:54:22.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.943+0000 7eff7183b640 1 -- 172.21.15.98:0/3315219151 learned_addr learned my addr 172.21.15.98:0/3315219151 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:22.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.943+0000 7eff70839640 1 --2- 172.21.15.98:0/3315219151 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.944+0000 7eff70839640 1 -- 172.21.15.98:0/3315219151 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc138320 msgr2=0x5564fc13a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:22.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.944+0000 7eff70839640 1 --2- 172.21.15.98:0/3315219151 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc138320 0x5564fc13a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.944+0000 7eff70839640 1 -- 172.21.15.98:0/3315219151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc12d330 msgr2=0x5564fc12d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:22.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.944+0000 7eff70839640 1 --2- 172.21.15.98:0/3315219151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc12d330 0x5564fc12d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.944+0000 7eff70839640 1 -- 172.21.15.98:0/3315219151 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564fbda63f0 con 0x5564fc12dc50 2024-05-09T08:54:22.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.944+0000 7eff7183b640 1 --2- 172.21.15.98:0/3315219151 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc138320 0x5564fc13a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:54:22.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.944+0000 7eff70839640 1 --2- 172.21.15.98:0/3315219151 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc137de0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0cd7f0 tx=0x7eff5c09cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b54bbf8daaedd034 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:22.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.946+0000 7eff7103a640 1 --2- 172.21.15.98:0/3768229026 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7eff580a2f00 0x7eff580a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.956+0000 7eff577fe640 1 -- 172.21.15.98:0/3315219151 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c020040 con 0x5564fc12dc50 2024-05-09T08:54:22.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.956+0000 7eff577fe640 1 -- 172.21.15.98:0/3315219151 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff5c0a5230 con 0x5564fc12dc50 2024-05-09T08:54:22.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.957+0000 7eff75faf740 1 -- 172.21.15.98:0/3315219151 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 msgr2=0x5564fc137de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.957+0000 7eff75faf740 1 --2- 172.21.15.98:0/3315219151 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc137de0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0cd7f0 tx=0x7eff5c09cd80 comp rx=0 tx=0).stop 2024-05-09T08:54:22.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.958+0000 7eff75faf740 1 -- 172.21.15.98:0/3315219151 shutdown_connections 2024-05-09T08:54:22.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.958+0000 7eff75faf740 1 --2- 172.21.15.98:0/3315219151 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc138320 0x5564fc13a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.958+0000 7eff75faf740 1 --2- 172.21.15.98:0/3315219151 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc137de0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.958+0000 7eff75faf740 1 --2- 172.21.15.98:0/3315219151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc12d330 0x5564fc12d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.958+0000 7eff75faf740 1 -- 172.21.15.98:0/3315219151 >> 172.21.15.98:0/3315219151 conn(0x5564fc129650 msgr2=0x5564fc129a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.961+0000 7eff75faf740 1 -- 172.21.15.98:0/3315219151 shutdown_connections 2024-05-09T08:54:22.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.961+0000 7eff75faf740 1 -- 172.21.15.98:0/3315219151 wait complete. 2024-05-09T08:54:22.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.962+0000 7eff75faf740 1 -- start start 2024-05-09T08:54:22.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.962+0000 7eff75faf740 1 Processor -- start 2024-05-09T08:54:22.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.963+0000 7eff75faf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12d330 0x5564fc1351e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.963+0000 7eff75faf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc12dc50 0x5564fc135720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.963+0000 7eff75faf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc138320 0x5564fc135c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.963+0000 7eff75faf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5564fc13b100 con 0x5564fc138320 2024-05-09T08:54:22.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.963+0000 7eff75faf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5564fc04dc40 con 0x5564fc12dc50 2024-05-09T08:54:22.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.963+0000 7eff75faf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5564fc13b280 con 0x5564fc12d330 2024-05-09T08:54:22.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.963+0000 7eff75faf740 10 client.0 _ll_register_callbacks cb 0x5564fbda5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:22.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.963+0000 7eff7103a640 1 --2- 172.21.15.98:0/3768229026 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7eff580a2f00 0x7eff580a53c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7eff64009c80 tx=0x7eff64009220 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:22.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.963+0000 7eff7183b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc138320 0x5564fc135c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.963+0000 7eff7183b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc138320 0x5564fc135c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43276/0 (socket says 172.21.15.98:43276) 2024-05-09T08:54:22.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.963+0000 7eff7183b640 1 -- 172.21.15.98:0/868484649 learned_addr learned my addr 172.21.15.98:0/868484649 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:22.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.964+0000 7eff7183b640 1 -- 172.21.15.98:0/868484649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12d330 msgr2=0x5564fc1351e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:54:22.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.964+0000 7eff7183b640 1 --2- 172.21.15.98:0/868484649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12d330 0x5564fc1351e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.964+0000 7eff7183b640 1 -- 172.21.15.98:0/868484649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc12dc50 msgr2=0x5564fc135720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:22.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.964+0000 7eff7183b640 1 --2- 172.21.15.98:0/868484649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc12dc50 0x5564fc135720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.964+0000 7eff7183b640 1 -- 172.21.15.98:0/868484649 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564fbe707d0 con 0x5564fc138320 2024-05-09T08:54:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.971+0000 7eff7183b640 1 --2- 172.21.15.98:0/868484649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc138320 0x5564fc135c60 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7eff68005ec0 tx=0x7eff68040e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603be3bbf6da71a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.971+0000 7eff54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:54:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.972+0000 7eff54ff9640 1 -- 172.21.15.98:0/868484649 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff68046040 con 0x5564fc138320 2024-05-09T08:54:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.972+0000 7eff54ff9640 1 -- 172.21.15.98:0/868484649 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7eff680048d0 con 0x5564fc138320 2024-05-09T08:54:22.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.976+0000 7eff75faf740 1 -- 172.21.15.98:0/868484649 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff5c0a5230 con 0x5564fc138320 2024-05-09T08:54:22.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.979+0000 7eff54ff9640 1 -- 172.21.15.98:0/868484649 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7eff6801a0b0 con 0x5564fc138320 2024-05-09T08:54:22.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.980+0000 7eff75faf740 10 client.16713 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:22.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.980+0000 7eff75faf740 1 -- 172.21.15.98:0/868484649 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5564fc1472b0 con 0x5564fc138320 2024-05-09T08:54:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.984+0000 7eff54ff9640 1 -- 172.21.15.98:0/868484649 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff5c0a5230 con 0x5564fc138320 2024-05-09T08:54:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.985+0000 7eff54ff9640 1 -- 172.21.15.98:0/868484649 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7eff680105d0 con 0x5564fc138320 2024-05-09T08:54:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.985+0000 7eff75faf740 10 client.16713 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.985+0000 7eff75faf740 10 client.16713 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:54:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.985+0000 7eff75faf740 20 client.16713 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.985+0000 7eff75faf740 1 --2- 172.21.15.98:0/868484649 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5564fc148bc0 0x5564fc14b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.985+0000 7eff75faf740 4 client.16713 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.985+0000 7eff75faf740 1 -- 172.21.15.98:0/868484649 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564fc0a0370 con 0x5564fc148bc0 2024-05-09T08:54:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.986+0000 7eff7103a640 1 --2- 172.21.15.98:0/868484649 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5564fc148bc0 0x5564fc14b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.989+0000 7eff7103a640 1 --2- 172.21.15.98:0/868484649 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5564fc148bc0 0x5564fc14b060 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cf8ceae41af1f74a server_cookie=8eac09784695c439 in_seq=0 out_seq=0 2024-05-09T08:54:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.990+0000 7eff54ff9640 10 client.16713 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:54:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.991+0000 7eff54ff9640 1 -- 172.21.15.98:0/868484649 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5564fc0a0370 con 0x5564fc148bc0 2024-05-09T08:54:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.991+0000 7eff54ff9640 10 client.16713 handle_command_reply: tid=0 2024-05-09T08:54:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.991+0000 7eff75faf740 1 client.16713 shutdown 2024-05-09T08:54:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.992+0000 7eff75faf740 1 -- 172.21.15.98:0/868484649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc138320 msgr2=0x5564fc135c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.992+0000 7eff75faf740 1 --2- 172.21.15.98:0/868484649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc138320 0x5564fc135c60 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7eff68005ec0 tx=0x7eff68040e30 comp rx=0 tx=0).stop 2024-05-09T08:54:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.994+0000 7eff75faf740 1 -- 172.21.15.98:0/868484649 shutdown_connections 2024-05-09T08:54:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.994+0000 7eff75faf740 1 --2- 172.21.15.98:0/868484649 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5564fc148bc0 0x5564fc14b060 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.994+0000 7eff75faf740 1 --2- 172.21.15.98:0/868484649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc138320 0x5564fc135c60 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.994+0000 7eff75faf740 1 --2- 172.21.15.98:0/868484649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc12dc50 0x5564fc135720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.994+0000 7eff75faf740 1 --2- 172.21.15.98:0/868484649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12d330 0x5564fc1351e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.994+0000 7eff75faf740 1 -- 172.21.15.98:0/868484649 >> 172.21.15.98:0/868484649 conn(0x5564fc129650 msgr2=0x5564fc12f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:22.995+0000 7eff54ff9640 0 client.16713 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:54:23.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.001+0000 7eff75faf740 1 -- 172.21.15.98:0/868484649 shutdown_connections 2024-05-09T08:54:23.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.001+0000 7eff75faf740 1 -- 172.21.15.98:0/868484649 wait complete. 2024-05-09T08:54:23.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.001+0000 7eff75faf740 20 client.16713 trim_cache size 0 max 16384 2024-05-09T08:54:23.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.050+0000 7eff75faf740 1 Processor -- start 2024-05-09T08:54:23.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.053+0000 7eff75faf740 1 -- start start 2024-05-09T08:54:23.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.053+0000 7eff75faf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc12d330 0x5564fc137910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.054+0000 7eff75faf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc144040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.054+0000 7eff75faf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc138320 0x5564fc150970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.054+0000 7eff7103a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc12d330 0x5564fc137910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.054+0000 7eff7103a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc12d330 0x5564fc137910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45540/0 (socket says 172.21.15.98:45540) 2024-05-09T08:54:23.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.055+0000 7eff70839640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc144040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.055+0000 7eff70839640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc144040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45576/0 (socket says 172.21.15.98:45576) 2024-05-09T08:54:23.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.055+0000 7eff7103a640 1 -- 172.21.15.98:0/1946239656 learned_addr learned my addr 172.21.15.98:0/1946239656 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:23.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.055+0000 7eff75faf740 1 -- 172.21.15.98:0/1946239656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff500100d0 con 0x5564fc138320 2024-05-09T08:54:23.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.055+0000 7eff75faf740 1 -- 172.21.15.98:0/1946239656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7eff50011c10 con 0x5564fc12d330 2024-05-09T08:54:23.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.055+0000 7eff75faf740 1 -- 172.21.15.98:0/1946239656 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7eff500107a0 con 0x5564fc12dc50 2024-05-09T08:54:23.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.056+0000 7eff7183b640 1 --2- 172.21.15.98:0/1946239656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc138320 0x5564fc150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.058+0000 7eff7103a640 1 -- 172.21.15.98:0/1946239656 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 msgr2=0x5564fc144040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.058+0000 7eff7103a640 1 --2- 172.21.15.98:0/1946239656 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc144040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.058+0000 7eff7103a640 1 -- 172.21.15.98:0/1946239656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc138320 msgr2=0x5564fc150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.058+0000 7eff7103a640 1 --2- 172.21.15.98:0/1946239656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc138320 0x5564fc150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.058+0000 7eff7103a640 1 -- 172.21.15.98:0/1946239656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564fc133750 con 0x5564fc12d330 2024-05-09T08:54:23.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.059+0000 7eff7103a640 1 --2- 172.21.15.98:0/1946239656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc12d330 0x5564fc137910 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7eff64005cc0 tx=0x7eff64010000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d109e0bffae09e69 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:23.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.062+0000 7eff56ffd640 1 -- 172.21.15.98:0/1946239656 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff64027030 con 0x5564fc12d330 2024-05-09T08:54:23.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.062+0000 7eff56ffd640 1 -- 172.21.15.98:0/1946239656 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff64016040 con 0x5564fc12d330 2024-05-09T08:54:23.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.062+0000 7eff56ffd640 1 -- 172.21.15.98:0/1946239656 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6401d9d0 con 0x5564fc12d330 2024-05-09T08:54:23.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.064+0000 7eff75faf740 1 -- 172.21.15.98:0/1946239656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc12d330 msgr2=0x5564fc137910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.064+0000 7eff75faf740 1 --2- 172.21.15.98:0/1946239656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc12d330 0x5564fc137910 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7eff64005cc0 tx=0x7eff64010000 comp rx=0 tx=0).stop 2024-05-09T08:54:23.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.066+0000 7eff75faf740 1 -- 172.21.15.98:0/1946239656 shutdown_connections 2024-05-09T08:54:23.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.066+0000 7eff75faf740 1 --2- 172.21.15.98:0/1946239656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc138320 0x5564fc150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.066+0000 7eff75faf740 1 --2- 172.21.15.98:0/1946239656 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc144040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.066+0000 7eff75faf740 1 --2- 172.21.15.98:0/1946239656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc12d330 0x5564fc137910 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.066+0000 7eff75faf740 1 -- 172.21.15.98:0/1946239656 >> 172.21.15.98:0/1946239656 conn(0x5564fc129650 msgr2=0x5564fc12f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.069+0000 7eff75faf740 1 -- 172.21.15.98:0/1946239656 shutdown_connections 2024-05-09T08:54:23.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.069+0000 7eff75faf740 1 -- 172.21.15.98:0/1946239656 wait complete. 2024-05-09T08:54:23.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.069+0000 7eff75faf740 1 -- start start 2024-05-09T08:54:23.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.070+0000 7eff75faf740 1 Processor -- start 2024-05-09T08:54:23.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.071+0000 7eff75faf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc12d330 0x5564fc1408d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.071+0000 7eff75faf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc131b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.071+0000 7eff75faf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc138320 0x5564fc1320c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.071+0000 7eff75faf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff50011540 con 0x5564fc12d330 2024-05-09T08:54:23.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.072+0000 7eff75faf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7eff50010e70 con 0x5564fc138320 2024-05-09T08:54:23.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.072+0000 7eff75faf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7eff5000e0b0 con 0x5564fc12dc50 2024-05-09T08:54:23.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.072+0000 7eff75faf740 10 client.0 _ll_register_callbacks cb 0x7eff68040ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:23.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.072+0000 7eff70839640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc131b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.072+0000 7eff70839640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc131b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45580/0 (socket says 172.21.15.98:45580) 2024-05-09T08:54:23.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.072+0000 7eff70839640 1 -- 172.21.15.98:0/3832350822 learned_addr learned my addr 172.21.15.98:0/3832350822 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:23.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.072+0000 7eff7183b640 1 --2- 172.21.15.98:0/3832350822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc138320 0x5564fc1320c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.073+0000 7eff7103a640 1 --2- 172.21.15.98:0/3832350822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc12d330 0x5564fc1408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.073+0000 7eff70839640 1 -- 172.21.15.98:0/3832350822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc138320 msgr2=0x5564fc1320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.073+0000 7eff70839640 1 --2- 172.21.15.98:0/3832350822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc138320 0x5564fc1320c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.073+0000 7eff70839640 1 -- 172.21.15.98:0/3832350822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc12d330 msgr2=0x5564fc1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.073+0000 7eff70839640 1 --2- 172.21.15.98:0/3832350822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc12d330 0x5564fc1408d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.073+0000 7eff70839640 1 -- 172.21.15.98:0/3832350822 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564fc1328f0 con 0x5564fc12dc50 2024-05-09T08:54:23.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.076+0000 7eff70839640 1 --2- 172.21.15.98:0/3832350822 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc131b80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7eff5c098a20 tx=0x7eff5c00df70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9bc46a4089629b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:23.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.076+0000 7eff54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:54:23.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.076+0000 7eff54ff9640 1 -- 172.21.15.98:0/3832350822 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c020040 con 0x5564fc12dc50 2024-05-09T08:54:23.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.076+0000 7eff75faf740 1 -- 172.21.15.98:0/3832350822 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff64016040 con 0x5564fc12dc50 2024-05-09T08:54:23.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.077+0000 7eff54ff9640 1 -- 172.21.15.98:0/3832350822 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7eff5c098c30 con 0x5564fc12dc50 2024-05-09T08:54:23.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.077+0000 7eff54ff9640 1 -- 172.21.15.98:0/3832350822 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7eff5c0cdc70 con 0x5564fc12dc50 2024-05-09T08:54:23.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.077+0000 7eff75faf740 10 client.25984 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:23.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.077+0000 7eff75faf740 1 -- 172.21.15.98:0/3832350822 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5564fc1af1c0 con 0x5564fc12dc50 2024-05-09T08:54:23.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.077+0000 7eff54ff9640 1 -- 172.21.15.98:0/3832350822 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff64016040 con 0x5564fc12dc50 2024-05-09T08:54:23.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.078+0000 7eff54ff9640 1 -- 172.21.15.98:0/3832350822 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7eff5c09b480 con 0x5564fc12dc50 2024-05-09T08:54:23.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.078+0000 7eff75faf740 10 client.25984 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:23.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.078+0000 7eff75faf740 10 client.25984 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:54:23.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.078+0000 7eff75faf740 20 client.25984 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:23.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.078+0000 7eff75faf740 1 --2- 172.21.15.98:0/3832350822 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5564fc1af4a0 0x5564fc1b1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.078+0000 7eff75faf740 4 client.25984 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:54:23.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.078+0000 7eff75faf740 1 -- 172.21.15.98:0/3832350822 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5564fc09a6c0 con 0x5564fc1af4a0 2024-05-09T08:54:23.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.078+0000 7eff7103a640 1 --2- 172.21.15.98:0/3832350822 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5564fc1af4a0 0x5564fc1b1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.079+0000 7eff7103a640 1 --2- 172.21.15.98:0/3832350822 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5564fc1af4a0 0x5564fc1b1890 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7f2a41f561f97c26 server_cookie=39a51bd6a6f9cf94 in_seq=0 out_seq=0 2024-05-09T08:54:23.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.081+0000 7eff54ff9640 10 client.25984 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:54:23.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.081+0000 7eff54ff9640 1 -- 172.21.15.98:0/3832350822 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5564fc09a6c0 con 0x5564fc1af4a0 2024-05-09T08:54:23.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.081+0000 7eff54ff9640 10 client.25984 handle_command_reply: tid=0 2024-05-09T08:54:23.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.081+0000 7eff75faf740 1 client.25984 shutdown 2024-05-09T08:54:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.091+0000 7eff75faf740 1 -- 172.21.15.98:0/3832350822 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 msgr2=0x5564fc131b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.091+0000 7eff75faf740 1 --2- 172.21.15.98:0/3832350822 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc131b80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7eff5c098a20 tx=0x7eff5c00df70 comp rx=0 tx=0).stop 2024-05-09T08:54:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.092+0000 7eff75faf740 1 -- 172.21.15.98:0/3832350822 shutdown_connections 2024-05-09T08:54:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.092+0000 7eff75faf740 1 --2- 172.21.15.98:0/3832350822 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5564fc1af4a0 0x5564fc1b1890 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.092+0000 7eff75faf740 1 --2- 172.21.15.98:0/3832350822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5564fc138320 0x5564fc1320c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.092+0000 7eff75faf740 1 --2- 172.21.15.98:0/3832350822 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5564fc12dc50 0x5564fc131b80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.092+0000 7eff75faf740 1 --2- 172.21.15.98:0/3832350822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fc12d330 0x5564fc1408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.092+0000 7eff54ff9640 0 client.25984 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:54:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.092+0000 7eff75faf740 1 -- 172.21.15.98:0/3832350822 >> 172.21.15.98:0/3832350822 conn(0x5564fc129650 msgr2=0x5564fc1351a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.093+0000 7eff75faf740 1 -- 172.21.15.98:0/3832350822 shutdown_connections 2024-05-09T08:54:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.093+0000 7eff75faf740 1 -- 172.21.15.98:0/3832350822 wait complete. 2024-05-09T08:54:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.094+0000 7eff75faf740 20 client.25984 trim_cache size 0 max 16384 2024-05-09T08:54:23.093 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:54:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.100+0000 7eff7329e640 1 -- 172.21.15.98:0/3768229026 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7eff580a2f00 msgr2=0x7eff580a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.100+0000 7eff7329e640 1 --2- 172.21.15.98:0/3768229026 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7eff580a2f00 0x7eff580a53c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7eff64009c80 tx=0x7eff64009220 comp rx=0 tx=0).stop 2024-05-09T08:54:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.100+0000 7eff7329e640 1 -- 172.21.15.98:0/3768229026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c110080 msgr2=0x7eff6c116740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.100+0000 7eff7329e640 1 --2- 172.21.15.98:0/3768229026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c110080 0x7eff6c116740 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00d980 tx=0x7eff5c00de40 comp rx=0 tx=0).stop 2024-05-09T08:54:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.100+0000 7eff7329e640 1 -- 172.21.15.98:0/3768229026 shutdown_connections 2024-05-09T08:54:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.100+0000 7eff7329e640 1 --2- 172.21.15.98:0/3768229026 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7eff580a2f00 0x7eff580a53c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.100+0000 7eff7329e640 1 --2- 172.21.15.98:0/3768229026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff6c11b010 0x7eff6c116c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.100+0000 7eff7329e640 1 --2- 172.21.15.98:0/3768229026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c110080 0x7eff6c116740 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.100+0000 7eff7329e640 1 --2- 172.21.15.98:0/3768229026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c10e0b0 0x7eff6c11a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.100+0000 7eff7329e640 1 -- 172.21.15.98:0/3768229026 >> 172.21.15.98:0/3768229026 conn(0x7eff6c06c940 msgr2=0x7eff6c072330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.104+0000 7eff7329e640 1 -- 172.21.15.98:0/3768229026 shutdown_connections 2024-05-09T08:54:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.104+0000 7eff7329e640 1 -- 172.21.15.98:0/3768229026 wait complete. 2024-05-09T08:54:23.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:54:23.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.434+0000 7f1f484f5640 1 -- 172.21.15.98:0/2818904883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f40106700 msgr2=0x7f1f40106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.434+0000 7f1f484f5640 1 --2- 172.21.15.98:0/2818904883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f40106700 0x7f1f40106ae0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c009930 tx=0x7f1f3c02a420 comp rx=0 tx=0).stop 2024-05-09T08:54:23.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.434+0000 7f1f484f5640 1 -- 172.21.15.98:0/2818904883 shutdown_connections 2024-05-09T08:54:23.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.434+0000 7f1f484f5640 1 --2- 172.21.15.98:0/2818904883 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f401015d0 0x7f1f4010d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.434+0000 7f1f484f5640 1 --2- 172.21.15.98:0/2818904883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40100c10 0x7f1f40101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.434+0000 7f1f484f5640 1 --2- 172.21.15.98:0/2818904883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f40106700 0x7f1f40106ae0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.434+0000 7f1f484f5640 1 -- 172.21.15.98:0/2818904883 >> 172.21.15.98:0/2818904883 conn(0x7f1f400762b0 msgr2=0x7f1f400766c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.434+0000 7f1f484f5640 1 -- 172.21.15.98:0/2818904883 shutdown_connections 2024-05-09T08:54:23.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.434+0000 7f1f484f5640 1 -- 172.21.15.98:0/2818904883 wait complete. 2024-05-09T08:54:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.435+0000 7f1f484f5640 1 Processor -- start 2024-05-09T08:54:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.435+0000 7f1f484f5640 1 -- start start 2024-05-09T08:54:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.435+0000 7f1f484f5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f40100c10 0x7f1f40197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.435+0000 7f1f484f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f401015d0 0x7f1f401976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.435+0000 7f1f484f5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40106700 0x7f1f4019ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.435+0000 7f1f484f5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1f40110be0 con 0x7f1f401015d0 2024-05-09T08:54:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.435+0000 7f1f484f5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1f40110a60 con 0x7f1f40106700 2024-05-09T08:54:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.435+0000 7f1f484f5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1f40110d60 con 0x7f1f40100c10 2024-05-09T08:54:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.435+0000 7f1f45a90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f401015d0 0x7f1f401976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.435+0000 7f1f45a90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f401015d0 0x7f1f401976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43324/0 (socket says 172.21.15.98:43324) 2024-05-09T08:54:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.435+0000 7f1f45a90640 1 -- 172.21.15.98:0/144160988 learned_addr learned my addr 172.21.15.98:0/144160988 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.436+0000 7f1f45a90640 1 -- 172.21.15.98:0/144160988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f40100c10 msgr2=0x7f1f40197160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:54:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.436+0000 7f1f45a90640 1 --2- 172.21.15.98:0/144160988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f40100c10 0x7f1f40197160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.436+0000 7f1f45a90640 1 -- 172.21.15.98:0/144160988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40106700 msgr2=0x7f1f4019ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.436+0000 7f1f45a90640 1 --2- 172.21.15.98:0/144160988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40106700 0x7f1f4019ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.436+0000 7f1f45a90640 1 -- 172.21.15.98:0/144160988 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f4019c150 con 0x7f1f401015d0 2024-05-09T08:54:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.436+0000 7f1f45a90640 1 --2- 172.21.15.98:0/144160988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f401015d0 0x7f1f401976a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f1f3000bfc0 tx=0x7f1f3000de20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.436+0000 7f1f377fe640 1 -- 172.21.15.98:0/144160988 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30012b80 con 0x7f1f401015d0 2024-05-09T08:54:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.436+0000 7f1f377fe640 1 -- 172.21.15.98:0/144160988 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f30012d20 con 0x7f1f401015d0 2024-05-09T08:54:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.436+0000 7f1f377fe640 1 -- 172.21.15.98:0/144160988 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3001aa10 con 0x7f1f401015d0 2024-05-09T08:54:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.436+0000 7f1f484f5640 1 -- 172.21.15.98:0/144160988 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f4019c440 con 0x7f1f401015d0 2024-05-09T08:54:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.436+0000 7f1f484f5640 1 -- 172.21.15.98:0/144160988 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f40069b60 con 0x7f1f401015d0 2024-05-09T08:54:23.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.439+0000 7f1f4b1fe740 1 Processor -- start 2024-05-09T08:54:23.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.440+0000 7f1f4b1fe740 1 -- start start 2024-05-09T08:54:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.440+0000 7f1f4b1fe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0330 0x55fb335c0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.440+0000 7f1f4b1fe740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0c50 0x55fb335cade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.440+0000 7f1f4b1fe740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335cb320 0x55fb335cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.440+0000 7f1f4b1fe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fb33525c40 con 0x55fb335c0330 2024-05-09T08:54:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.440+0000 7f1f4b1fe740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fb334cd580 con 0x55fb335cb320 2024-05-09T08:54:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.440+0000 7f1f4b1fe740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fb33521c00 con 0x55fb335c0c50 2024-05-09T08:54:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.440+0000 7f1f46a92640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0330 0x55fb335c0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.440+0000 7f1f46291640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0c50 0x55fb335cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.440+0000 7f1f46a92640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0330 0x55fb335c0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43340/0 (socket says 172.21.15.98:43340) 2024-05-09T08:54:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.440+0000 7f1f46a92640 1 -- 172.21.15.98:0/1382199931 learned_addr learned my addr 172.21.15.98:0/1382199931 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.441+0000 7f1f46291640 1 -- 172.21.15.98:0/1382199931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335cb320 msgr2=0x55fb335cd710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:54:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.441+0000 7f1f46291640 1 --2- 172.21.15.98:0/1382199931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335cb320 0x55fb335cd710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.441+0000 7f1f46291640 1 -- 172.21.15.98:0/1382199931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0330 msgr2=0x55fb335c0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.441+0000 7f1f46291640 1 --2- 172.21.15.98:0/1382199931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0330 0x55fb335c0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.441+0000 7f1f46291640 1 -- 172.21.15.98:0/1382199931 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb332393f0 con 0x55fb335c0c50 2024-05-09T08:54:23.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.441+0000 7f1f46a92640 1 --2- 172.21.15.98:0/1382199931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0330 0x55fb335c0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:54:23.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.441+0000 7f1f46291640 1 --2- 172.21.15.98:0/1382199931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0c50 0x55fb335cade0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c02aa10 tx=0x7f1f3c031500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e3ba0ea20969f14 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:23.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.441+0000 7f1f377fe640 1 -- 172.21.15.98:0/144160988 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1f3001abb0 con 0x7f1f401015d0 2024-05-09T08:54:23.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.441+0000 7f1f34bfe640 1 -- 172.21.15.98:0/1382199931 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c03e070 con 0x55fb335c0c50 2024-05-09T08:54:23.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.441+0000 7f1f34bfe640 1 -- 172.21.15.98:0/1382199931 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f3c004340 con 0x55fb335c0c50 2024-05-09T08:54:23.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.441+0000 7f1f34bfe640 1 -- 172.21.15.98:0/1382199931 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c035040 con 0x55fb335c0c50 2024-05-09T08:54:23.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.442+0000 7f1f377fe640 1 --2- 172.21.15.98:0/144160988 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1f2c0a2d80 0x7f1f2c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.442+0000 7f1f46a92640 1 --2- 172.21.15.98:0/144160988 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1f2c0a2d80 0x7f1f2c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.442+0000 7f1f377fe640 1 -- 172.21.15.98:0/144160988 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7f1f300d34f0 con 0x7f1f401015d0 2024-05-09T08:54:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.442+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/1382199931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0c50 msgr2=0x55fb335cade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.442+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/1382199931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0c50 0x55fb335cade0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c02aa10 tx=0x7f1f3c031500 comp rx=0 tx=0).stop 2024-05-09T08:54:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.443+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/1382199931 shutdown_connections 2024-05-09T08:54:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.443+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/1382199931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335cb320 0x55fb335cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.443+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/1382199931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0c50 0x55fb335cade0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.443+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/1382199931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0330 0x55fb335c0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.443+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/1382199931 >> 172.21.15.98:0/1382199931 conn(0x55fb335bc650 msgr2=0x55fb335bca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.443+0000 7f1f46a92640 1 --2- 172.21.15.98:0/144160988 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1f2c0a2d80 0x7f1f2c0a5240 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1f3800e330 tx=0x7f1f3800b040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.443+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/1382199931 shutdown_connections 2024-05-09T08:54:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.443+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/1382199931 wait complete. 2024-05-09T08:54:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.443+0000 7f1f4b1fe740 1 -- start start 2024-05-09T08:54:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.443+0000 7f1f4b1fe740 1 Processor -- start 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.444+0000 7f1f4b1fe740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0330 0x55fb335c8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.444+0000 7f1f4b1fe740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335c0c50 0x55fb335c8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.444+0000 7f1f4b1fe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335cb320 0x55fb335c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.444+0000 7f1f4b1fe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fb335ce0b0 con 0x55fb335cb320 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.444+0000 7f1f4b1fe740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fb334e0c40 con 0x55fb335c0c50 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.444+0000 7f1f4b1fe740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fb335ce230 con 0x55fb335c0330 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.444+0000 7f1f4b1fe740 10 client.0 _ll_register_callbacks cb 0x55fb33238be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.444+0000 7f1f46a92640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335cb320 0x55fb335c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.444+0000 7f1f46a92640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335cb320 0x55fb335c8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43350/0 (socket says 172.21.15.98:43350) 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.444+0000 7f1f46a92640 1 -- 172.21.15.98:0/3484792984 learned_addr learned my addr 172.21.15.98:0/3484792984 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.444+0000 7f1f45a90640 1 --2- 172.21.15.98:0/3484792984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335c0c50 0x55fb335c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.444+0000 7f1f46291640 1 --2- 172.21.15.98:0/3484792984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0330 0x55fb335c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f46a92640 1 -- 172.21.15.98:0/3484792984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0330 msgr2=0x55fb335c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f46a92640 1 --2- 172.21.15.98:0/3484792984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0330 0x55fb335c8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f46a92640 1 -- 172.21.15.98:0/3484792984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335c0c50 msgr2=0x55fb335c8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f46a92640 1 --2- 172.21.15.98:0/3484792984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335c0c50 0x55fb335c8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f46a92640 1 -- 172.21.15.98:0/3484792984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb333037d0 con 0x55fb335cb320 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f45a90640 1 --2- 172.21.15.98:0/3484792984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335c0c50 0x55fb335c8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f46a92640 1 --2- 172.21.15.98:0/3484792984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335cb320 0x55fb335c8d90 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f1f38015bb0 tx=0x7f1f38004bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41185f684a1a93e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f29ffb640 1 -- 172.21.15.98:0/3484792984 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f38023020 con 0x55fb335cb320 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f29ffb640 1 -- 172.21.15.98:0/3484792984 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7f1f38013b40 con 0x55fb335cb320 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3484792984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f3c004340 con 0x55fb335cb320 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f29ffb640 1 -- 172.21.15.98:0/3484792984 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1f3802e050 con 0x55fb335cb320 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f4b1fe740 10 client.16728 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3484792984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fb335da360 con 0x55fb335cb320 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.445+0000 7f1f29ffb640 1 -- 172.21.15.98:0/3484792984 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f3c004340 con 0x55fb335cb320 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.446+0000 7f1f29ffb640 1 -- 172.21.15.98:0/3484792984 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f1f3801e030 con 0x55fb335cb320 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.446+0000 7f1f4b1fe740 10 client.16728 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.446+0000 7f1f4b1fe740 10 client.16728 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:54:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.446+0000 7f1f4b1fe740 20 client.16728 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:23.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.446+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/3484792984 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55fb335db870 0x55fb335ddc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.446+0000 7f1f4b1fe740 4 client.16728 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:23.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.446+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3484792984 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb33533370 con 0x55fb335db870 2024-05-09T08:54:23.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.446+0000 7f1f46291640 1 --2- 172.21.15.98:0/3484792984 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55fb335db870 0x55fb335ddc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.447+0000 7f1f29ffb640 10 client.16728 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.447+0000 7f1f46291640 1 --2- 172.21.15.98:0/3484792984 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55fb335db870 0x55fb335ddc60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=455a1a71279755f server_cookie=2ae89eabf0709108 in_seq=0 out_seq=0 2024-05-09T08:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.447+0000 7f1f29ffb640 1 -- 172.21.15.98:0/3484792984 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fb33533370 con 0x55fb335db870 2024-05-09T08:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.447+0000 7f1f29ffb640 10 client.16728 handle_command_reply: tid=0 2024-05-09T08:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.447+0000 7f1f4b1fe740 1 client.16728 shutdown 2024-05-09T08:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.448+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3484792984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335cb320 msgr2=0x55fb335c8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.448+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/3484792984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335cb320 0x55fb335c8d90 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f1f38015bb0 tx=0x7f1f38004bc0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.448+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3484792984 shutdown_connections 2024-05-09T08:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.448+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/3484792984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335cb320 0x55fb335c8d90 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.448+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/3484792984 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55fb335db870 0x55fb335ddc60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.448+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/3484792984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335c0c50 0x55fb335c8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.448+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/3484792984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0330 0x55fb335c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.448+0000 7f1f29ffb640 0 client.16728 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.448+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3484792984 >> 172.21.15.98:0/3484792984 conn(0x55fb335bc650 msgr2=0x55fb335c2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.448+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3484792984 shutdown_connections 2024-05-09T08:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.448+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3484792984 wait complete. 2024-05-09T08:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.448+0000 7f1f4b1fe740 20 client.16728 trim_cache size 0 max 16384 2024-05-09T08:54:23.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.470+0000 7f1f4b1fe740 1 Processor -- start 2024-05-09T08:54:23.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.470+0000 7f1f4b1fe740 1 -- start start 2024-05-09T08:54:23.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.470+0000 7f1f4b1fe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0330 0x55fb335d6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.470+0000 7f1f4b1fe740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0c50 0x55fb335e3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.470+0000 7f1f4b1fe740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335cb320 0x55fb335e3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.470+0000 7f1f4b1fe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1f401acc80 con 0x55fb335c0330 2024-05-09T08:54:23.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.470+0000 7f1f4b1fe740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1f401ae7c0 con 0x55fb335cb320 2024-05-09T08:54:23.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.470+0000 7f1f4b1fe740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1f401ad350 con 0x55fb335c0c50 2024-05-09T08:54:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.471+0000 7f1f46291640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0330 0x55fb335d6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.471+0000 7f1f46291640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0330 0x55fb335d6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43354/0 (socket says 172.21.15.98:43354) 2024-05-09T08:54:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.471+0000 7f1f46291640 1 -- 172.21.15.98:0/2510567331 learned_addr learned my addr 172.21.15.98:0/2510567331 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.471+0000 7f1f45a90640 1 --2- 172.21.15.98:0/2510567331 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0c50 0x55fb335e3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.471+0000 7f1f46a92640 1 --2- 172.21.15.98:0/2510567331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335cb320 0x55fb335e3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.472+0000 7f1f46291640 1 -- 172.21.15.98:0/2510567331 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0c50 msgr2=0x55fb335e3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.472+0000 7f1f46291640 1 --2- 172.21.15.98:0/2510567331 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0c50 0x55fb335e3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.472+0000 7f1f46291640 1 -- 172.21.15.98:0/2510567331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335cb320 msgr2=0x55fb335e3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.472+0000 7f1f46291640 1 --2- 172.21.15.98:0/2510567331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335cb320 0x55fb335e3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.472+0000 7f1f46291640 1 -- 172.21.15.98:0/2510567331 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb335ddfd0 con 0x55fb335c0330 2024-05-09T08:54:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.472+0000 7f1f46291640 1 --2- 172.21.15.98:0/2510567331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0330 0x55fb335d6f70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c005ab0 tx=0x7f1f3c02ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6ee49fcd0e71a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.473+0000 7f1f2bfff640 1 -- 172.21.15.98:0/2510567331 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c053020 con 0x55fb335c0330 2024-05-09T08:54:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.473+0000 7f1f2bfff640 1 -- 172.21.15.98:0/2510567331 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f3c031d60 con 0x55fb335c0330 2024-05-09T08:54:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.473+0000 7f1f2bfff640 1 -- 172.21.15.98:0/2510567331 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c035050 con 0x55fb335c0330 2024-05-09T08:54:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.474+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/2510567331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0330 msgr2=0x55fb335d6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.474+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/2510567331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0330 0x55fb335d6f70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c005ab0 tx=0x7f1f3c02ab60 comp rx=0 tx=0).stop 2024-05-09T08:54:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.475+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/2510567331 shutdown_connections 2024-05-09T08:54:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.475+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/2510567331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335cb320 0x55fb335e3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.475+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/2510567331 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0c50 0x55fb335e3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.475+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/2510567331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0330 0x55fb335d6f70 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.475+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/2510567331 >> 172.21.15.98:0/2510567331 conn(0x55fb335bc650 msgr2=0x55fb335c35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.476+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/2510567331 shutdown_connections 2024-05-09T08:54:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.476+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/2510567331 wait complete. 2024-05-09T08:54:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.476+0000 7f1f4b1fe740 1 -- start start 2024-05-09T08:54:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.476+0000 7f1f4b1fe740 1 Processor -- start 2024-05-09T08:54:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.477+0000 7f1f4b1fe740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0330 0x55fb335d3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.477+0000 7f1f4b1fe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0c50 0x55fb335cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.477+0000 7f1f46291640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0330 0x55fb335d3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.477+0000 7f1f46291640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0330 0x55fb335d3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45640/0 (socket says 172.21.15.98:45640) 2024-05-09T08:54:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.477+0000 7f1f46291640 1 -- 172.21.15.98:0/3791948599 learned_addr learned my addr 172.21.15.98:0/3791948599 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.477+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/3791948599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335cb320 0x55fb335cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.477+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3791948599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1f401ae0f0 con 0x55fb335c0c50 2024-05-09T08:54:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.477+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3791948599 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1f401ada20 con 0x55fb335cb320 2024-05-09T08:54:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.477+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3791948599 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1f401aac60 con 0x55fb335c0330 2024-05-09T08:54:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.477+0000 7f1f46a92640 1 --2- 172.21.15.98:0/3791948599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335cb320 0x55fb335cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.478+0000 7f1f4b1fe740 10 client.0 _ll_register_callbacks cb 0x7f1f3800ee90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.478+0000 7f1f45a90640 1 --2- 172.21.15.98:0/3791948599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0c50 0x55fb335cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.478+0000 7f1f46291640 1 -- 172.21.15.98:0/3791948599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335cb320 msgr2=0x55fb335cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.478+0000 7f1f46291640 1 --2- 172.21.15.98:0/3791948599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335cb320 0x55fb335cefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.478+0000 7f1f46291640 1 -- 172.21.15.98:0/3791948599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0c50 msgr2=0x55fb335cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.478+0000 7f1f46291640 1 --2- 172.21.15.98:0/3791948599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0c50 0x55fb335cea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.478+0000 7f1f46291640 1 -- 172.21.15.98:0/3791948599 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb335cf6b0 con 0x55fb335c0330 2024-05-09T08:54:23.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.478+0000 7f1f46291640 1 --2- 172.21.15.98:0/3791948599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0330 0x55fb335d3710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c0043f0 tx=0x7f1f3c02a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80b44af3767a52cd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:23.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.479+0000 7f1f29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:54:23.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.479+0000 7f1f29ffb640 1 -- 172.21.15.98:0/3791948599 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c05e020 con 0x55fb335c0330 2024-05-09T08:54:23.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.479+0000 7f1f29ffb640 1 -- 172.21.15.98:0/3791948599 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7f1f3c04f470 con 0x55fb335c0330 2024-05-09T08:54:23.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.480+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3791948599 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f3c031d60 con 0x55fb335c0330 2024-05-09T08:54:23.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.481+0000 7f1f29ffb640 1 -- 172.21.15.98:0/3791948599 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1f3c03e040 con 0x55fb335c0330 2024-05-09T08:54:23.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.482+0000 7f1f4b1fe740 10 client.25996 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:23.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.482+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3791948599 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fb33641c90 con 0x55fb335c0330 2024-05-09T08:54:23.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.485+0000 7f1f29ffb640 1 -- 172.21.15.98:0/3791948599 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f3c04f6b0 con 0x55fb335c0330 2024-05-09T08:54:23.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.486+0000 7f1f29ffb640 1 -- 172.21.15.98:0/3791948599 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f1f3c04c040 con 0x55fb335c0330 2024-05-09T08:54:23.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.486+0000 7f1f4b1fe740 10 client.25996 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:23.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.486+0000 7f1f4b1fe740 10 client.25996 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:54:23.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.486+0000 7f1f4b1fe740 20 client.25996 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:23.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.486+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/3791948599 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55fb33641f80 0x55fb33655590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.486+0000 7f1f4b1fe740 4 client.25996 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:54:23.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.486+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3791948599 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f1f3c031d60 con 0x55fb33641f80 2024-05-09T08:54:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.490+0000 7f1f45a90640 1 --2- 172.21.15.98:0/3791948599 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55fb33641f80 0x55fb33655590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.493+0000 7f1f45a90640 1 --2- 172.21.15.98:0/3791948599 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55fb33641f80 0x55fb33655590 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fd97ef760d78b204 server_cookie=57e1c6286fab61f3 in_seq=0 out_seq=0 2024-05-09T08:54:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.493+0000 7f1f29ffb640 10 client.25996 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:54:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.493+0000 7f1f29ffb640 1 -- 172.21.15.98:0/3791948599 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f1f3c031d60 con 0x55fb33641f80 2024-05-09T08:54:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.493+0000 7f1f29ffb640 10 client.25996 handle_command_reply: tid=0 2024-05-09T08:54:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.494+0000 7f1f4b1fe740 1 client.25996 shutdown 2024-05-09T08:54:23.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.495+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3791948599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0330 msgr2=0x55fb335d3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.495+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/3791948599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0330 0x55fb335d3710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c0043f0 tx=0x7f1f3c02a420 comp rx=0 tx=0).stop 2024-05-09T08:54:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.501+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3791948599 shutdown_connections 2024-05-09T08:54:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.501+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/3791948599 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55fb33641f80 0x55fb33655590 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.501+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/3791948599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb335cb320 0x55fb335cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.501+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/3791948599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fb335c0c50 0x55fb335cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.501+0000 7f1f4b1fe740 1 --2- 172.21.15.98:0/3791948599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fb335c0330 0x55fb335d3710 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.501+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3791948599 >> 172.21.15.98:0/3791948599 conn(0x55fb335bc650 msgr2=0x55fb335baf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.502+0000 7f1f29ffb640 0 client.25996 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:54:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.504+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3791948599 shutdown_connections 2024-05-09T08:54:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.507+0000 7f1f4b1fe740 1 -- 172.21.15.98:0/3791948599 wait complete. 2024-05-09T08:54:23.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.510+0000 7f1f4b1fe740 20 client.25996 trim_cache size 0 max 16384 2024-05-09T08:54:23.512 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:54:23.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.515+0000 7f1f484f5640 1 -- 172.21.15.98:0/144160988 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1f2c0a2d80 msgr2=0x7f1f2c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.515+0000 7f1f484f5640 1 --2- 172.21.15.98:0/144160988 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1f2c0a2d80 0x7f1f2c0a5240 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1f3800e330 tx=0x7f1f3800b040 comp rx=0 tx=0).stop 2024-05-09T08:54:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.515+0000 7f1f484f5640 1 -- 172.21.15.98:0/144160988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f401015d0 msgr2=0x7f1f401976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.515+0000 7f1f484f5640 1 --2- 172.21.15.98:0/144160988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f401015d0 0x7f1f401976a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f1f3000bfc0 tx=0x7f1f3000de20 comp rx=0 tx=0).stop 2024-05-09T08:54:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.515+0000 7f1f484f5640 1 -- 172.21.15.98:0/144160988 shutdown_connections 2024-05-09T08:54:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.515+0000 7f1f484f5640 1 --2- 172.21.15.98:0/144160988 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1f2c0a2d80 0x7f1f2c0a5240 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.515+0000 7f1f484f5640 1 --2- 172.21.15.98:0/144160988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40106700 0x7f1f4019ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.515+0000 7f1f484f5640 1 --2- 172.21.15.98:0/144160988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f401015d0 0x7f1f401976a0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.515+0000 7f1f484f5640 1 --2- 172.21.15.98:0/144160988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f40100c10 0x7f1f40197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.515+0000 7f1f484f5640 1 -- 172.21.15.98:0/144160988 >> 172.21.15.98:0/144160988 conn(0x7f1f400762b0 msgr2=0x7f1f400fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.515+0000 7f1f484f5640 1 -- 172.21.15.98:0/144160988 shutdown_connections 2024-05-09T08:54:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.515+0000 7f1f484f5640 1 -- 172.21.15.98:0/144160988 wait complete. 2024-05-09T08:54:23.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:54:23.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.894+0000 7f4c19638640 1 -- 172.21.15.98:0/839904697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1410fed0 msgr2=0x7f4c141102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.894+0000 7f4c19638640 1 --2- 172.21.15.98:0/839904697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1410fed0 0x7f4c141102b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4c04009900 tx=0x7f4c0402a490 comp rx=0 tx=0).stop 2024-05-09T08:54:23.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.894+0000 7f4c19638640 1 -- 172.21.15.98:0/839904697 shutdown_connections 2024-05-09T08:54:23.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.894+0000 7f4c19638640 1 --2- 172.21.15.98:0/839904697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c141092b0 0x7f4c1411d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.894+0000 7f4c19638640 1 --2- 172.21.15.98:0/839904697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c14108830 0x7f4c14108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.894+0000 7f4c19638640 1 --2- 172.21.15.98:0/839904697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1410fed0 0x7f4c141102b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.894+0000 7f4c19638640 1 -- 172.21.15.98:0/839904697 >> 172.21.15.98:0/839904697 conn(0x7f4c1406c7b0 msgr2=0x7f4c1406cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.897+0000 7f4c19638640 1 -- 172.21.15.98:0/839904697 shutdown_connections 2024-05-09T08:54:23.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.897+0000 7f4c19638640 1 -- 172.21.15.98:0/839904697 wait complete. 2024-05-09T08:54:23.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.898+0000 7f4c19638640 1 Processor -- start 2024-05-09T08:54:23.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.898+0000 7f4c19638640 1 -- start start 2024-05-09T08:54:23.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.899+0000 7f4c19638640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c14108830 0x7f4c1411ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.899+0000 7f4c19638640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c141092b0 0x7f4c14116e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.899+0000 7f4c19638640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1410fed0 0x7f4c141173d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.899+0000 7f4c19638640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4c14111240 con 0x7f4c14108830 2024-05-09T08:54:23.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.899+0000 7f4c19638640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4c141110c0 con 0x7f4c1410fed0 2024-05-09T08:54:23.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.899+0000 7f4c19638640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4c141113c0 con 0x7f4c141092b0 2024-05-09T08:54:23.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.899+0000 7f4c137fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c141092b0 0x7f4c14116e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.899+0000 7f4c137fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c141092b0 0x7f4c14116e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45660/0 (socket says 172.21.15.98:45660) 2024-05-09T08:54:23.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.899+0000 7f4c137fe640 1 -- 172.21.15.98:0/1457274067 learned_addr learned my addr 172.21.15.98:0/1457274067 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:23.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.900+0000 7f4c137fe640 1 -- 172.21.15.98:0/1457274067 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1410fed0 msgr2=0x7f4c141173d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:54:23.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.900+0000 7f4c137fe640 1 --2- 172.21.15.98:0/1457274067 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1410fed0 0x7f4c141173d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.900+0000 7f4c137fe640 1 -- 172.21.15.98:0/1457274067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c14108830 msgr2=0x7f4c1411ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.900+0000 7f4c137fe640 1 --2- 172.21.15.98:0/1457274067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c14108830 0x7f4c1411ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.900+0000 7f4c137fe640 1 -- 172.21.15.98:0/1457274067 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c14117ba0 con 0x7f4c141092b0 2024-05-09T08:54:23.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.900+0000 7f4c137fe640 1 --2- 172.21.15.98:0/1457274067 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c141092b0 0x7f4c14116e90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4c0800d950 tx=0x7f4c0800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:23.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.900+0000 7f4c117fa640 1 -- 172.21.15.98:0/1457274067 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c08017070 con 0x7f4c141092b0 2024-05-09T08:54:23.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.901+0000 7f4c19638640 1 -- 172.21.15.98:0/1457274067 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c141bbbb0 con 0x7f4c141092b0 2024-05-09T08:54:23.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.901+0000 7f4c19638640 1 -- 172.21.15.98:0/1457274067 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c141bc150 con 0x7f4c141092b0 2024-05-09T08:54:23.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.901+0000 7f4c117fa640 1 -- 172.21.15.98:0/1457274067 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c08009c30 con 0x7f4c141092b0 2024-05-09T08:54:23.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.901+0000 7f4c117fa640 1 -- 172.21.15.98:0/1457274067 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0800f6b0 con 0x7f4c141092b0 2024-05-09T08:54:23.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.907+0000 7f4c117fa640 1 -- 172.21.15.98:0/1457274067 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4c0800f850 con 0x7f4c141092b0 2024-05-09T08:54:23.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.908+0000 7f4c117fa640 1 --2- 172.21.15.98:0/1457274067 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4bfc0a2e00 0x7f4bfc0a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.908+0000 7f4c117fa640 1 -- 172.21.15.98:0/1457274067 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7f4c080cf7b0 con 0x7f4c141092b0 2024-05-09T08:54:23.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.909+0000 7f4c13fff640 1 --2- 172.21.15.98:0/1457274067 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4bfc0a2e00 0x7f4bfc0a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.909+0000 7f4c1d5a3740 1 Processor -- start 2024-05-09T08:54:23.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.910+0000 7f4c1d5a3740 1 -- start start 2024-05-09T08:54:23.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.910+0000 7f4c1d5a3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2520 0x5625ffac2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.910+0000 7f4c1d5a3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2e40 0x5625ffaccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.910+0000 7f4c1d5a3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 0x5625ffacf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.910+0000 7f4c1d5a3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5625ffa27c40 con 0x5625ffac2e40 2024-05-09T08:54:23.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.911+0000 7f4c1d5a3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5625ff9cf580 con 0x5625ffac2520 2024-05-09T08:54:23.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.911+0000 7f4c1d5a3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5625ffa23c00 con 0x5625ffacd510 2024-05-09T08:54:23.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.911+0000 7f4c13fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2520 0x5625ffac2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.911+0000 7f4c13fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2520 0x5625ffac2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45626/0 (socket says 172.21.15.98:45626) 2024-05-09T08:54:23.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.911+0000 7f4c13fff640 1 -- 172.21.15.98:0/1349570609 learned_addr learned my addr 172.21.15.98:0/1349570609 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:23.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.911+0000 7f4c18e37640 1 --2- 172.21.15.98:0/1349570609 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 0x5625ffacf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.912+0000 7f4c13fff640 1 -- 172.21.15.98:0/1349570609 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 msgr2=0x5625ffacf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.912+0000 7f4c13fff640 1 --2- 172.21.15.98:0/1349570609 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 0x5625ffacf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.912+0000 7f4c13fff640 1 -- 172.21.15.98:0/1349570609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2e40 msgr2=0x5625ffaccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.912+0000 7f4c13fff640 1 --2- 172.21.15.98:0/1349570609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2e40 0x5625ffaccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.912+0000 7f4c13fff640 1 -- 172.21.15.98:0/1349570609 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625ff73b3f0 con 0x5625ffac2520 2024-05-09T08:54:23.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.912+0000 7f4c13fff640 1 --2- 172.21.15.98:0/1349570609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2520 0x5625ffac2900 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4c04031550 tx=0x7f4c04003940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e60de7391d5abf97 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:23.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.913+0000 7f4bfa7fc640 1 -- 172.21.15.98:0/1349570609 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c04035040 con 0x5625ffac2520 2024-05-09T08:54:23.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.913+0000 7f4bfa7fc640 1 -- 172.21.15.98:0/1349570609 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c04004180 con 0x5625ffac2520 2024-05-09T08:54:23.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.914+0000 7f4bfa7fc640 1 -- 172.21.15.98:0/1349570609 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c040396a0 con 0x5625ffac2520 2024-05-09T08:54:23.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.914+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/1349570609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2520 msgr2=0x5625ffac2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.914+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/1349570609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2520 0x5625ffac2900 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4c04031550 tx=0x7f4c04003940 comp rx=0 tx=0).stop 2024-05-09T08:54:23.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.914+0000 7f4c13fff640 1 --2- 172.21.15.98:0/1457274067 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4bfc0a2e00 0x7f4bfc0a52c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4c040039b0 tx=0x7f4c04039000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:23.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.915+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/1349570609 shutdown_connections 2024-05-09T08:54:23.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.915+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/1349570609 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 0x5625ffacf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.915+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/1349570609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2e40 0x5625ffaccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.915+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/1349570609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2520 0x5625ffac2900 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.915+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/1349570609 >> 172.21.15.98:0/1349570609 conn(0x5625ffabe840 msgr2=0x5625ffabec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.915+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/1349570609 shutdown_connections 2024-05-09T08:54:23.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.915+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/1349570609 wait complete. 2024-05-09T08:54:23.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.915+0000 7f4c1d5a3740 1 -- start start 2024-05-09T08:54:23.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.916+0000 7f4c1d5a3740 1 Processor -- start 2024-05-09T08:54:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.916+0000 7f4c1d5a3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2520 0x5625ffaca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.916+0000 7f4c1d5a3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2e40 0x5625ffaca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.916+0000 7f4c1d5a3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 0x5625ffacae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.916+0000 7f4c1d5a3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5625ffad02f0 con 0x5625ffac2520 2024-05-09T08:54:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.916+0000 7f4c1d5a3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5625ff9e2c40 con 0x5625ffac2e40 2024-05-09T08:54:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.916+0000 7f4c1d5a3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5625ffad0470 con 0x5625ffacd510 2024-05-09T08:54:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.916+0000 7f4c1d5a3740 10 client.0 _ll_register_callbacks cb 0x5625ff73abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.916+0000 7f4c137fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2e40 0x5625ffaca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.916+0000 7f4c137fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2e40 0x5625ffaca8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45638/0 (socket says 172.21.15.98:45638) 2024-05-09T08:54:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.917+0000 7f4c137fe640 1 -- 172.21.15.98:0/2850100710 learned_addr learned my addr 172.21.15.98:0/2850100710 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.917+0000 7f4c18e37640 1 --2- 172.21.15.98:0/2850100710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 0x5625ffacae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.917+0000 7f4c137fe640 1 -- 172.21.15.98:0/2850100710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 msgr2=0x5625ffacae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.917+0000 7f4c137fe640 1 --2- 172.21.15.98:0/2850100710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 0x5625ffacae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.917+0000 7f4c137fe640 1 -- 172.21.15.98:0/2850100710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2520 msgr2=0x5625ffaca3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:54:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.917+0000 7f4c137fe640 1 --2- 172.21.15.98:0/2850100710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2520 0x5625ffaca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.917+0000 7f4c137fe640 1 -- 172.21.15.98:0/2850100710 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625ff8057d0 con 0x5625ffac2e40 2024-05-09T08:54:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.917+0000 7f4c137fe640 1 --2- 172.21.15.98:0/2850100710 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2e40 0x5625ffaca8e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4c08010d20 tx=0x7f4c0809b830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11a3dc8373355c42 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:23.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.922+0000 7f4bf37fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:54:23.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.922+0000 7f4bf37fe640 1 -- 172.21.15.98:0/2850100710 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c08020070 con 0x5625ffac2e40 2024-05-09T08:54:23.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.923+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/2850100710 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c04004180 con 0x5625ffac2e40 2024-05-09T08:54:23.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.923+0000 7f4bf37fe640 1 -- 172.21.15.98:0/2850100710 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7f4c080cf330 con 0x5625ffac2e40 2024-05-09T08:54:23.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.923+0000 7f4bf37fe640 1 -- 172.21.15.98:0/2850100710 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4c080cf550 con 0x5625ffac2e40 2024-05-09T08:54:23.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.924+0000 7f4c1d5a3740 10 client.26045 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:23.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.924+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/2850100710 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5625ffadc560 con 0x5625ffac2e40 2024-05-09T08:54:23.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.928+0000 7f4bf37fe640 1 -- 172.21.15.98:0/2850100710 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c04004180 con 0x5625ffac2e40 2024-05-09T08:54:23.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.928+0000 7f4bf37fe640 1 -- 172.21.15.98:0/2850100710 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f4c08010460 con 0x5625ffac2e40 2024-05-09T08:54:23.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.928+0000 7f4c1d5a3740 10 client.26045 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:23.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.928+0000 7f4c1d5a3740 10 client.26045 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:23.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.928+0000 7f4c1d5a3740 20 client.26045 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:23.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.928+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/2850100710 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625ffadde20 0x5625ffae0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.928+0000 7f4c1d5a3740 4 client.26045 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:23.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.928+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/2850100710 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625ffa35370 con 0x5625ffadde20 2024-05-09T08:54:23.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.929+0000 7f4c13fff640 1 --2- 172.21.15.98:0/2850100710 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625ffadde20 0x5625ffae0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.934+0000 7f4bf37fe640 10 client.26045 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:23.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.934+0000 7f4c13fff640 1 --2- 172.21.15.98:0/2850100710 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625ffadde20 0x5625ffae0280 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44a90df1db01e955 server_cookie=c8eba22da81d883c in_seq=0 out_seq=0 2024-05-09T08:54:23.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.936+0000 7f4bf37fe640 1 -- 172.21.15.98:0/2850100710 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5625ffa35370 con 0x5625ffadde20 2024-05-09T08:54:23.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.936+0000 7f4bf37fe640 10 client.26045 handle_command_reply: tid=0 2024-05-09T08:54:23.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.936+0000 7f4c1d5a3740 1 client.26045 shutdown 2024-05-09T08:54:23.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.937+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/2850100710 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2e40 msgr2=0x5625ffaca8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.937+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/2850100710 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2e40 0x5625ffaca8e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4c08010d20 tx=0x7f4c0809b830 comp rx=0 tx=0).stop 2024-05-09T08:54:23.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.937+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/2850100710 shutdown_connections 2024-05-09T08:54:23.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.937+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/2850100710 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625ffadde20 0x5625ffae0280 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.937+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/2850100710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 0x5625ffacae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.937+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/2850100710 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2e40 0x5625ffaca8e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.937+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/2850100710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2520 0x5625ffaca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.937+0000 7f4bf37fe640 0 client.26045 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:23.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.937+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/2850100710 >> 172.21.15.98:0/2850100710 conn(0x5625ffabe840 msgr2=0x5625ffac4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.940+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/2850100710 shutdown_connections 2024-05-09T08:54:23.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.940+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/2850100710 wait complete. 2024-05-09T08:54:23.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.940+0000 7f4c1d5a3740 20 client.26045 trim_cache size 0 max 16384 2024-05-09T08:54:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.967+0000 7f4c1d5a3740 1 Processor -- start 2024-05-09T08:54:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.967+0000 7f4c1d5a3740 1 -- start start 2024-05-09T08:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.967+0000 7f4c1d5a3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2520 0x5625ffaccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c1d5a3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffac2e40 0x5625ffad9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c1d5a3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffacd510 0x5625ffabd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c1d5a3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0000fff0 con 0x5625ffac2520 2024-05-09T08:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c1d5a3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4c00011b30 con 0x5625ffacd510 2024-05-09T08:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c1d5a3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4c000106c0 con 0x5625ffac2e40 2024-05-09T08:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c13fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2520 0x5625ffaccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c13fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2520 0x5625ffaccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43412/0 (socket says 172.21.15.98:43412) 2024-05-09T08:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c13fff640 1 -- 172.21.15.98:0/2149689099 learned_addr learned my addr 172.21.15.98:0/2149689099 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c18e37640 1 --2- 172.21.15.98:0/2149689099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffacd510 0x5625ffabd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c137fe640 1 --2- 172.21.15.98:0/2149689099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffac2e40 0x5625ffad9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c13fff640 1 -- 172.21.15.98:0/2149689099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffac2e40 msgr2=0x5625ffad9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c13fff640 1 --2- 172.21.15.98:0/2149689099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffac2e40 0x5625ffad9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c13fff640 1 -- 172.21.15.98:0/2149689099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffacd510 msgr2=0x5625ffabd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c13fff640 1 --2- 172.21.15.98:0/2149689099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffacd510 0x5625ffabd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c13fff640 1 -- 172.21.15.98:0/2149689099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625ffac39e0 con 0x5625ffac2520 2024-05-09T08:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c137fe640 1 --2- 172.21.15.98:0/2149689099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffac2e40 0x5625ffad9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c18e37640 1 --2- 172.21.15.98:0/2149689099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffacd510 0x5625ffabd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.968+0000 7f4c13fff640 1 --2- 172.21.15.98:0/2149689099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2520 0x5625ffaccad0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4c04031550 tx=0x7f4c04003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a75841d88c64b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.969+0000 7f4bf9ffb640 1 -- 172.21.15.98:0/2149689099 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c04035040 con 0x5625ffac2520 2024-05-09T08:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.969+0000 7f4bf9ffb640 1 -- 172.21.15.98:0/2149689099 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c04004160 con 0x5625ffac2520 2024-05-09T08:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.969+0000 7f4bf9ffb640 1 -- 172.21.15.98:0/2149689099 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c040582d0 con 0x5625ffac2520 2024-05-09T08:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.969+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/2149689099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2520 msgr2=0x5625ffaccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.970+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/2149689099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2520 0x5625ffaccad0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4c04031550 tx=0x7f4c04003ea0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.970+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/2149689099 shutdown_connections 2024-05-09T08:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.970+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/2149689099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffacd510 0x5625ffabd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.970+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/2149689099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffac2e40 0x5625ffad9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.970+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/2149689099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffac2520 0x5625ffaccad0 secure :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4c04031550 tx=0x7f4c04003ea0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.970+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/2149689099 >> 172.21.15.98:0/2149689099 conn(0x5625ffabe840 msgr2=0x5625ffabd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.970+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/2149689099 shutdown_connections 2024-05-09T08:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.970+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/2149689099 wait complete. 2024-05-09T08:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.971+0000 7f4c1d5a3740 1 -- start start 2024-05-09T08:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.971+0000 7f4c1d5a3740 1 Processor -- start 2024-05-09T08:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.972+0000 7f4c1d5a3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2e40 0x5625ffad1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.972+0000 7f4c1d5a3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 0x5625ffad17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.972+0000 7f4c1d5a3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffad1d10 0x5625ffb43400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.972+0000 7f4c1d5a3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4c00011460 con 0x5625ffad1d10 2024-05-09T08:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.972+0000 7f4c1d5a3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4c00010d90 con 0x5625ffac2e40 2024-05-09T08:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.972+0000 7f4c1d5a3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0000dfd0 con 0x5625ffacd510 2024-05-09T08:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.972+0000 7f4c13fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2e40 0x5625ffad1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.972+0000 7f4c1d5a3740 10 client.0 _ll_register_callbacks cb 0x7f4c08010bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.972+0000 7f4c13fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2e40 0x5625ffad1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45646/0 (socket says 172.21.15.98:45646) 2024-05-09T08:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.972+0000 7f4c13fff640 1 -- 172.21.15.98:0/3774922006 learned_addr learned my addr 172.21.15.98:0/3774922006 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.972+0000 7f4c137fe640 1 --2- 172.21.15.98:0/3774922006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 0x5625ffad17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.972+0000 7f4c18e37640 1 --2- 172.21.15.98:0/3774922006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffad1d10 0x5625ffb43400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.973+0000 7f4c13fff640 1 -- 172.21.15.98:0/3774922006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 msgr2=0x5625ffad17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.973+0000 7f4c13fff640 1 --2- 172.21.15.98:0/3774922006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 0x5625ffad17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.973+0000 7f4c13fff640 1 -- 172.21.15.98:0/3774922006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffad1d10 msgr2=0x5625ffb43400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.973+0000 7f4c13fff640 1 --2- 172.21.15.98:0/3774922006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffad1d10 0x5625ffb43400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.973+0000 7f4c13fff640 1 -- 172.21.15.98:0/3774922006 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625ffb43970 con 0x5625ffac2e40 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.973+0000 7f4c13fff640 1 --2- 172.21.15.98:0/3774922006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2e40 0x5625ffad1290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4c04006010 tx=0x7f4c04005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f474c79a6c00202 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.973+0000 7f4bf37fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.973+0000 7f4bf37fe640 1 -- 172.21.15.98:0/3774922006 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c04035070 con 0x5625ffac2e40 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.973+0000 7f4bf37fe640 1 -- 172.21.15.98:0/3774922006 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7f4c040317c0 con 0x5625ffac2e40 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.974+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/3774922006 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c04004160 con 0x5625ffac2e40 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.974+0000 7f4bf37fe640 1 -- 172.21.15.98:0/3774922006 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4c040319c0 con 0x5625ffac2e40 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.974+0000 7f4c1d5a3740 10 client.26051 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.974+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/3774922006 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5625ffb44c30 con 0x5625ffac2e40 2024-05-09T08:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.974+0000 7f4bf37fe640 1 -- 172.21.15.98:0/3774922006 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c04004160 con 0x5625ffac2e40 2024-05-09T08:54:23.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.974+0000 7f4bf37fe640 1 -- 172.21.15.98:0/3774922006 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f4c04049030 con 0x5625ffac2e40 2024-05-09T08:54:23.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.974+0000 7f4c1d5a3740 10 client.26051 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:23.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.974+0000 7f4c1d5a3740 10 client.26051 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.974+0000 7f4c1d5a3740 20 client.26051 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.974+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/3774922006 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625ffb44e60 0x5625ffb47250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.974+0000 7f4c1d5a3740 4 client.26051 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.974+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/3774922006 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625ffac51c0 con 0x5625ffb44e60 2024-05-09T08:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.975+0000 7f4c137fe640 1 --2- 172.21.15.98:0/3774922006 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625ffb44e60 0x5625ffb47250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.975+0000 7f4bf37fe640 10 client.26051 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.975+0000 7f4c137fe640 1 --2- 172.21.15.98:0/3774922006 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625ffb44e60 0x5625ffb47250 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0c1792a98f18ee0 server_cookie=9aa0f4a40fb90d8f in_seq=0 out_seq=0 2024-05-09T08:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.977+0000 7f4bf37fe640 1 -- 172.21.15.98:0/3774922006 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625ffac51c0 con 0x5625ffb44e60 2024-05-09T08:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.977+0000 7f4bf37fe640 10 client.26051 handle_command_reply: tid=0 2024-05-09T08:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.977+0000 7f4c1d5a3740 1 client.26051 shutdown 2024-05-09T08:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.978+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/3774922006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2e40 msgr2=0x5625ffad1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.978+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/3774922006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2e40 0x5625ffad1290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4c04006010 tx=0x7f4c04005c90 comp rx=0 tx=0).stop 2024-05-09T08:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.978+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/3774922006 shutdown_connections 2024-05-09T08:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.978+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/3774922006 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5625ffb44e60 0x5625ffb47250 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.978+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/3774922006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625ffad1d10 0x5625ffb43400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.978+0000 7f4bf37fe640 0 client.26051 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.978+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/3774922006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5625ffacd510 0x5625ffad17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.978+0000 7f4c1d5a3740 1 --2- 172.21.15.98:0/3774922006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625ffac2e40 0x5625ffad1290 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.978+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/3774922006 >> 172.21.15.98:0/3774922006 conn(0x5625ffabe840 msgr2=0x5625ffabd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.978+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/3774922006 shutdown_connections 2024-05-09T08:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.978+0000 7f4c1d5a3740 1 -- 172.21.15.98:0/3774922006 wait complete. 2024-05-09T08:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.978+0000 7f4c1d5a3740 20 client.26051 trim_cache size 0 max 16384 2024-05-09T08:54:23.979 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.979+0000 7f4c19638640 1 -- 172.21.15.98:0/1457274067 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4bfc0a2e00 msgr2=0x7f4bfc0a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.979+0000 7f4c19638640 1 --2- 172.21.15.98:0/1457274067 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4bfc0a2e00 0x7f4bfc0a52c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4c040039b0 tx=0x7f4c04039000 comp rx=0 tx=0).stop 2024-05-09T08:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.979+0000 7f4c19638640 1 -- 172.21.15.98:0/1457274067 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c141092b0 msgr2=0x7f4c14116e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.979+0000 7f4c19638640 1 --2- 172.21.15.98:0/1457274067 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c141092b0 0x7f4c14116e90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4c0800d950 tx=0x7f4c0800de10 comp rx=0 tx=0).stop 2024-05-09T08:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.980+0000 7f4c19638640 1 -- 172.21.15.98:0/1457274067 shutdown_connections 2024-05-09T08:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.980+0000 7f4c19638640 1 --2- 172.21.15.98:0/1457274067 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4bfc0a2e00 0x7f4bfc0a52c0 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4c040039b0 tx=0x7f4c04039000 comp rx=0 tx=0).stop 2024-05-09T08:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.980+0000 7f4c19638640 1 --2- 172.21.15.98:0/1457274067 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1410fed0 0x7f4c141173d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.980+0000 7f4c19638640 1 --2- 172.21.15.98:0/1457274067 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c141092b0 0x7f4c14116e90 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4c0800d950 tx=0x7f4c0800de10 comp rx=0 tx=0).stop 2024-05-09T08:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.980+0000 7f4c19638640 1 --2- 172.21.15.98:0/1457274067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c14108830 0x7f4c1411ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.980+0000 7f4c19638640 1 -- 172.21.15.98:0/1457274067 >> 172.21.15.98:0/1457274067 conn(0x7f4c1406c7b0 msgr2=0x7f4c14070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.980+0000 7f4c19638640 1 -- 172.21.15.98:0/1457274067 shutdown_connections 2024-05-09T08:54:23.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:23.980+0000 7f4c19638640 1 -- 172.21.15.98:0/1457274067 wait complete. 2024-05-09T08:54:24.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:23 smithi142 ceph-mon[34625]: pgmap v116: 129 pgs: 129 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 554 MiB/s wr, 2.82k op/s 2024-05-09T08:54:24.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:23 smithi105 ceph-mon[34498]: pgmap v116: 129 pgs: 129 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 554 MiB/s wr, 2.82k op/s 2024-05-09T08:54:24.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:23 smithi098 ceph-mon[30092]: pgmap v116: 129 pgs: 129 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 554 MiB/s wr, 2.82k op/s 2024-05-09T08:54:25.008 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:54:25.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:54:25.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:54:25.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.403+0000 7ffa9606b640 1 -- 172.21.15.98:0/746812557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa9010d4b0 msgr2=0x7ffa9010d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:25.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.403+0000 7ffa9606b640 1 --2- 172.21.15.98:0/746812557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa9010d4b0 0x7ffa9010d890 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009930 tx=0x7ffa8c02a440 comp rx=0 tx=0).stop 2024-05-09T08:54:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.408+0000 7ffa9606b640 1 -- 172.21.15.98:0/746812557 shutdown_connections 2024-05-09T08:54:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.408+0000 7ffa9606b640 1 --2- 172.21.15.98:0/746812557 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa9010eb60 0x7ffa90115340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.408+0000 7ffa9606b640 1 --2- 172.21.15.98:0/746812557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9010e1a0 0x7ffa9010e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.408+0000 7ffa9606b640 1 --2- 172.21.15.98:0/746812557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa9010d4b0 0x7ffa9010d890 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.408+0000 7ffa9606b640 1 -- 172.21.15.98:0/746812557 >> 172.21.15.98:0/746812557 conn(0x7ffa9006c710 msgr2=0x7ffa9006cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:25.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.408+0000 7ffa9606b640 1 -- 172.21.15.98:0/746812557 shutdown_connections 2024-05-09T08:54:25.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.408+0000 7ffa9606b640 1 -- 172.21.15.98:0/746812557 wait complete. 2024-05-09T08:54:25.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.409+0000 7ffa9606b640 1 Processor -- start 2024-05-09T08:54:25.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.411+0000 7ffa9606b640 1 -- start start 2024-05-09T08:54:25.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.411+0000 7ffa9606b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9010d4b0 0x7ffa9019e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.411+0000 7ffa9606b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa9010e1a0 0x7ffa9019e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.411+0000 7ffa9606b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa9010eb60 0x7ffa9019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.411+0000 7ffa9606b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa90118920 con 0x7ffa9010d4b0 2024-05-09T08:54:25.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.411+0000 7ffa9606b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901187a0 con 0x7ffa9010eb60 2024-05-09T08:54:25.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.411+0000 7ffa9606b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffa90118aa0 con 0x7ffa9010e1a0 2024-05-09T08:54:25.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.411+0000 7ffa94868640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa9010e1a0 0x7ffa9019e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:25.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.412+0000 7ffa94868640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa9010e1a0 0x7ffa9019e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:45698/0 (socket says 172.21.15.98:45698) 2024-05-09T08:54:25.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.412+0000 7ffa94868640 1 -- 172.21.15.98:0/371100446 learned_addr learned my addr 172.21.15.98:0/371100446 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:25.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.412+0000 7ffa9586a640 1 --2- 172.21.15.98:0/371100446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa9010eb60 0x7ffa9019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:25.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.412+0000 7ffa94868640 1 -- 172.21.15.98:0/371100446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa9010eb60 msgr2=0x7ffa9019edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:25.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.412+0000 7ffa94868640 1 --2- 172.21.15.98:0/371100446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa9010eb60 0x7ffa9019edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.412+0000 7ffa94868640 1 -- 172.21.15.98:0/371100446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9010d4b0 msgr2=0x7ffa9019e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:25.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.412+0000 7ffa94868640 1 --2- 172.21.15.98:0/371100446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9010d4b0 0x7ffa9019e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.412+0000 7ffa94868640 1 -- 172.21.15.98:0/371100446 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa901bb820 con 0x7ffa9010e1a0 2024-05-09T08:54:25.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.413+0000 7ffa94868640 1 --2- 172.21.15.98:0/371100446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa9010e1a0 0x7ffa9019e890 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800ea80 tx=0x7ffa8800ef40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:25.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.413+0000 7ffa867fc640 1 -- 172.21.15.98:0/371100446 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8800ce40 con 0x7ffa9010e1a0 2024-05-09T08:54:25.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.414+0000 7ffa867fc640 1 -- 172.21.15.98:0/371100446 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa88005020 con 0x7ffa9010e1a0 2024-05-09T08:54:25.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.414+0000 7ffa867fc640 1 -- 172.21.15.98:0/371100446 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa880116d0 con 0x7ffa9010e1a0 2024-05-09T08:54:25.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.416+0000 7ffa9606b640 1 -- 172.21.15.98:0/371100446 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa901bbb10 con 0x7ffa9010e1a0 2024-05-09T08:54:25.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.416+0000 7ffa9606b640 1 -- 172.21.15.98:0/371100446 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa901bc010 con 0x7ffa9010e1a0 2024-05-09T08:54:25.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.424+0000 7ffa99fdc740 1 Processor -- start 2024-05-09T08:54:25.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.426+0000 7ffa867fc640 1 -- 172.21.15.98:0/371100446 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ffa88005520 con 0x7ffa9010e1a0 2024-05-09T08:54:25.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.428+0000 7ffa867fc640 1 --2- 172.21.15.98:0/371100446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffa7c0a2f00 0x7ffa7c0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.428+0000 7ffa867fc640 1 -- 172.21.15.98:0/371100446 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7ffa880cfb30 con 0x7ffa9010e1a0 2024-05-09T08:54:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.429+0000 7ffa99fdc740 1 -- start start 2024-05-09T08:54:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.429+0000 7ffa99fdc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c877a2c0 0x55c0c8778600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.429+0000 7ffa99fdc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8778bd0 0x55c0c8779050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.429+0000 7ffa99fdc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c8783950 0x55c0c8785d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.429+0000 7ffa99fdc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c0c86dd990 con 0x55c0c8778bd0 2024-05-09T08:54:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.429+0000 7ffa99fdc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c0c86855b0 con 0x55c0c877a2c0 2024-05-09T08:54:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.429+0000 7ffa99fdc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c0c86d9950 con 0x55c0c8783950 2024-05-09T08:54:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.430+0000 7ffa95069640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c877a2c0 0x55c0c8778600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.430+0000 7ffa95069640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c877a2c0 0x55c0c8778600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45666/0 (socket says 172.21.15.98:45666) 2024-05-09T08:54:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.430+0000 7ffa95069640 1 -- 172.21.15.98:0/3754362769 learned_addr learned my addr 172.21.15.98:0/3754362769 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.430+0000 7ffa94868640 1 --2- 172.21.15.98:0/3754362769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8778bd0 0x55c0c8779050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.430+0000 7ffa9586a640 1 --2- 172.21.15.98:0/3754362769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c8783950 0x55c0c8785d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.430+0000 7ffa9586a640 1 --2- 172.21.15.98:0/371100446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffa7c0a2f00 0x7ffa7c0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.431+0000 7ffa95069640 1 -- 172.21.15.98:0/3754362769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c8783950 msgr2=0x55c0c8785d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.431+0000 7ffa95069640 1 --2- 172.21.15.98:0/3754362769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c8783950 0x55c0c8785d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.431+0000 7ffa95069640 1 -- 172.21.15.98:0/3754362769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8778bd0 msgr2=0x55c0c8779050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.431+0000 7ffa95069640 1 --2- 172.21.15.98:0/3754362769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8778bd0 0x55c0c8779050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.431+0000 7ffa95069640 1 -- 172.21.15.98:0/3754362769 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0c84b97e0 con 0x55c0c877a2c0 2024-05-09T08:54:25.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.432+0000 7ffa95069640 1 --2- 172.21.15.98:0/3754362769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c877a2c0 0x55c0c8778600 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c02aa30 tx=0x7ffa8c0314b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=764e4b5d43346320 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:25.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.433+0000 7ffa9586a640 1 --2- 172.21.15.98:0/371100446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffa7c0a2f00 0x7ffa7c0a53c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55c0c87762c0 tx=0x7ffa80006d50 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:25.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.434+0000 7ffa7b7fe640 1 -- 172.21.15.98:0/3754362769 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c03e030 con 0x55c0c877a2c0 2024-05-09T08:54:25.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.434+0000 7ffa7b7fe640 1 -- 172.21.15.98:0/3754362769 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa8c0035f0 con 0x55c0c877a2c0 2024-05-09T08:54:25.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.436+0000 7ffa99fdc740 1 -- 172.21.15.98:0/3754362769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c877a2c0 msgr2=0x55c0c8778600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:25.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.436+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/3754362769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c877a2c0 0x55c0c8778600 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c02aa30 tx=0x7ffa8c0314b0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.444+0000 7ffa99fdc740 1 -- 172.21.15.98:0/3754362769 shutdown_connections 2024-05-09T08:54:25.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.444+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/3754362769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c8783950 0x55c0c8785d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.444+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/3754362769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8778bd0 0x55c0c8779050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.444+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/3754362769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c877a2c0 0x55c0c8778600 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.444+0000 7ffa99fdc740 1 -- 172.21.15.98:0/3754362769 >> 172.21.15.98:0/3754362769 conn(0x55c0c87410a0 msgr2=0x55c0c8775130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:25.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.449+0000 7ffa99fdc740 1 -- 172.21.15.98:0/3754362769 shutdown_connections 2024-05-09T08:54:25.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.453+0000 7ffa99fdc740 1 -- 172.21.15.98:0/3754362769 wait complete. 2024-05-09T08:54:25.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.453+0000 7ffa99fdc740 1 -- start start 2024-05-09T08:54:25.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.453+0000 7ffa99fdc740 1 Processor -- start 2024-05-09T08:54:25.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.455+0000 7ffa99fdc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8778bd0 0x55c0c8780790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.455+0000 7ffa99fdc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c877a2c0 0x55c0c8780cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.455+0000 7ffa99fdc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8783950 0x55c0c8781210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.455+0000 7ffa99fdc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c0c87867d0 con 0x55c0c8778bd0 2024-05-09T08:54:25.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.455+0000 7ffa99fdc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c0c8698b40 con 0x55c0c8783950 2024-05-09T08:54:25.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.455+0000 7ffa99fdc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c0c8786950 con 0x55c0c877a2c0 2024-05-09T08:54:25.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.455+0000 7ffa99fdc740 10 client.0 _ll_register_callbacks cb 0x55c0c868d0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:25.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.455+0000 7ffa95069640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8778bd0 0x55c0c8780790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:25.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.455+0000 7ffa95069640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8778bd0 0x55c0c8780790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43464/0 (socket says 172.21.15.98:43464) 2024-05-09T08:54:25.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.455+0000 7ffa95069640 1 -- 172.21.15.98:0/1326279353 learned_addr learned my addr 172.21.15.98:0/1326279353 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:25.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.456+0000 7ffa95069640 1 -- 172.21.15.98:0/1326279353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c877a2c0 msgr2=0x55c0c8780cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:25.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.456+0000 7ffa95069640 1 --2- 172.21.15.98:0/1326279353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c877a2c0 0x55c0c8780cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.456+0000 7ffa95069640 1 -- 172.21.15.98:0/1326279353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8783950 msgr2=0x55c0c8781210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:54:25.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.456+0000 7ffa9586a640 1 --2- 172.21.15.98:0/1326279353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8783950 0x55c0c8781210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:25.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.456+0000 7ffa95069640 1 --2- 172.21.15.98:0/1326279353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8783950 0x55c0c8781210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.457+0000 7ffa95069640 1 -- 172.21.15.98:0/1326279353 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0c8519f70 con 0x55c0c8778bd0 2024-05-09T08:54:25.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.457+0000 7ffa9586a640 1 --2- 172.21.15.98:0/1326279353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8783950 0x55c0c8781210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:54:25.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.457+0000 7ffa95069640 1 --2- 172.21.15.98:0/1326279353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8778bd0 0x55c0c8780790 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009830 tx=0x7ffa8c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ce8e36ea7effc9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:25.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.458+0000 7ffa78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:54:25.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.458+0000 7ffa78ff9640 1 -- 172.21.15.98:0/1326279353 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c03e040 con 0x55c0c8778bd0 2024-05-09T08:54:25.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.458+0000 7ffa78ff9640 1 -- 172.21.15.98:0/1326279353 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7ffa8c031ad0 con 0x55c0c8778bd0 2024-05-09T08:54:25.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.459+0000 7ffa99fdc740 1 -- 172.21.15.98:0/1326279353 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa8c0035f0 con 0x55c0c8778bd0 2024-05-09T08:54:25.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.459+0000 7ffa78ff9640 1 -- 172.21.15.98:0/1326279353 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ffa8c031cd0 con 0x55c0c8778bd0 2024-05-09T08:54:25.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.459+0000 7ffa99fdc740 10 client.16758 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:25.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.459+0000 7ffa99fdc740 1 -- 172.21.15.98:0/1326279353 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c0c83b1170 con 0x55c0c8778bd0 2024-05-09T08:54:25.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.459+0000 7ffa78ff9640 1 -- 172.21.15.98:0/1326279353 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa8c0035f0 con 0x55c0c8778bd0 2024-05-09T08:54:25.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.459+0000 7ffa78ff9640 1 -- 172.21.15.98:0/1326279353 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ffa8c043730 con 0x55c0c8778bd0 2024-05-09T08:54:25.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.462+0000 7ffa99fdc740 10 client.16758 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:25.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.462+0000 7ffa99fdc740 10 client.16758 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:25.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.462+0000 7ffa99fdc740 20 client.16758 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:25.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.462+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/1326279353 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c0c8793820 0x55c0c8795c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.462+0000 7ffa99fdc740 4 client.16758 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:25.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.462+0000 7ffa99fdc740 1 -- 172.21.15.98:0/1326279353 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0c86eb0c0 con 0x55c0c8793820 2024-05-09T08:54:25.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.463+0000 7ffa94868640 1 --2- 172.21.15.98:0/1326279353 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c0c8793820 0x55c0c8795c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:25.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.463+0000 7ffa78ff9640 10 client.16758 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:25.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.463+0000 7ffa94868640 1 --2- 172.21.15.98:0/1326279353 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c0c8793820 0x55c0c8795c10 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50f338595c257001 server_cookie=5f6306633ee62bb4 in_seq=0 out_seq=0 2024-05-09T08:54:25.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.465+0000 7ffa78ff9640 1 -- 172.21.15.98:0/1326279353 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0c86eb0c0 con 0x55c0c8793820 2024-05-09T08:54:25.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.465+0000 7ffa78ff9640 10 client.16758 handle_command_reply: tid=0 2024-05-09T08:54:25.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.466+0000 7ffa99fdc740 1 client.16758 shutdown 2024-05-09T08:54:25.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.466+0000 7ffa99fdc740 1 -- 172.21.15.98:0/1326279353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8778bd0 msgr2=0x55c0c8780790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:25.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.466+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/1326279353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8778bd0 0x55c0c8780790 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009830 tx=0x7ffa8c0043b0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.467+0000 7ffa99fdc740 1 -- 172.21.15.98:0/1326279353 shutdown_connections 2024-05-09T08:54:25.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.467+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/1326279353 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c0c8793820 0x55c0c8795c10 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.467+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/1326279353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8783950 0x55c0c8781210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.467+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/1326279353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c877a2c0 0x55c0c8780cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.467+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/1326279353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8778bd0 0x55c0c8780790 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.467+0000 7ffa99fdc740 1 -- 172.21.15.98:0/1326279353 >> 172.21.15.98:0/1326279353 conn(0x55c0c87410a0 msgr2=0x55c0c877afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.467+0000 7ffa78ff9640 0 client.16758 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:25.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.472+0000 7ffa99fdc740 1 -- 172.21.15.98:0/1326279353 shutdown_connections 2024-05-09T08:54:25.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.472+0000 7ffa99fdc740 1 -- 172.21.15.98:0/1326279353 wait complete. 2024-05-09T08:54:25.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.472+0000 7ffa99fdc740 20 client.16758 trim_cache size 0 max 16384 2024-05-09T08:54:25.486 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:54:25 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: ignoring --setuser ceph since I am not root 2024-05-09T08:54:25.486 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:54:25 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: ignoring --setgroup ceph since I am not root 2024-05-09T08:54:25.486 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:54:25 smithi105 ceph-mds[68227]: -- 172.21.15.105:0/279174585 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5601c68b7a40 con 0x5601c688b000 2024-05-09T08:54:25.486 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:54:25 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: starting mds.e at 2024-05-09T08:54:25.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.530+0000 7ffa99fdc740 1 Processor -- start 2024-05-09T08:54:25.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.531+0000 7ffa99fdc740 1 -- start start 2024-05-09T08:54:25.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.531+0000 7ffa99fdc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8778bd0 0x55c0c878f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.531+0000 7ffa99fdc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c877a2c0 0x55c0c8774070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.531+0000 7ffa99fdc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8783950 0x55c0c877adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.532+0000 7ffa99fdc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901c6300 con 0x55c0c8783950 2024-05-09T08:54:25.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.532+0000 7ffa99fdc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901c7e40 con 0x55c0c8778bd0 2024-05-09T08:54:25.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.532+0000 7ffa99fdc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901c69d0 con 0x55c0c877a2c0 2024-05-09T08:54:25.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.532+0000 7ffa9586a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8783950 0x55c0c877adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:25.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.532+0000 7ffa9586a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8783950 0x55c0c877adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43476/0 (socket says 172.21.15.98:43476) 2024-05-09T08:54:25.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.532+0000 7ffa9586a640 1 -- 172.21.15.98:0/1239891007 learned_addr learned my addr 172.21.15.98:0/1239891007 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:25.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.533+0000 7ffa9586a640 1 -- 172.21.15.98:0/1239891007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c877a2c0 msgr2=0x55c0c8774070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:54:25.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.533+0000 7ffa9586a640 1 --2- 172.21.15.98:0/1239891007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c877a2c0 0x55c0c8774070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.533+0000 7ffa9586a640 1 -- 172.21.15.98:0/1239891007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8778bd0 msgr2=0x55c0c878f720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:54:25.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.533+0000 7ffa9586a640 1 --2- 172.21.15.98:0/1239891007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8778bd0 0x55c0c878f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.533+0000 7ffa9586a640 1 -- 172.21.15.98:0/1239891007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0c877ece0 con 0x55c0c8783950 2024-05-09T08:54:25.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.534+0000 7ffa9586a640 1 --2- 172.21.15.98:0/1239891007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8783950 0x55c0c877adb0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ffa80011d60 tx=0x7ffa80008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8eab4a4ce5657a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:25.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.548+0000 7ffa7affd640 1 -- 172.21.15.98:0/1239891007 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa80011d90 con 0x55c0c8783950 2024-05-09T08:54:25.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.549+0000 7ffa7affd640 1 -- 172.21.15.98:0/1239891007 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa80026070 con 0x55c0c8783950 2024-05-09T08:54:25.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.549+0000 7ffa7affd640 1 -- 172.21.15.98:0/1239891007 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa800146a0 con 0x55c0c8783950 2024-05-09T08:54:25.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.552+0000 7ffa99fdc740 1 -- 172.21.15.98:0/1239891007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8783950 msgr2=0x55c0c877adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:25.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.552+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/1239891007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8783950 0x55c0c877adb0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ffa80011d60 tx=0x7ffa80008040 comp rx=0 tx=0).stop 2024-05-09T08:54:25.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.552+0000 7ffa99fdc740 1 -- 172.21.15.98:0/1239891007 shutdown_connections 2024-05-09T08:54:25.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.552+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/1239891007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c8783950 0x55c0c877adb0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.552+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/1239891007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c877a2c0 0x55c0c8774070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.552+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/1239891007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8778bd0 0x55c0c878f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.552+0000 7ffa99fdc740 1 -- 172.21.15.98:0/1239891007 >> 172.21.15.98:0/1239891007 conn(0x55c0c87410a0 msgr2=0x55c0c877a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:25.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.556+0000 7ffa99fdc740 1 -- 172.21.15.98:0/1239891007 shutdown_connections 2024-05-09T08:54:25.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.556+0000 7ffa99fdc740 1 -- 172.21.15.98:0/1239891007 wait complete. 2024-05-09T08:54:25.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.556+0000 7ffa99fdc740 1 -- start start 2024-05-09T08:54:25.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.556+0000 7ffa99fdc740 1 Processor -- start 2024-05-09T08:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.557+0000 7ffa99fdc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8778bd0 0x55c0c8788710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.557+0000 7ffa99fdc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c877a2c0 0x55c0c8788c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.557+0000 7ffa99fdc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c8783950 0x55c0c8789190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.557+0000 7ffa99fdc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901c7770 con 0x55c0c877a2c0 2024-05-09T08:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.557+0000 7ffa99fdc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901c70a0 con 0x55c0c8778bd0 2024-05-09T08:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.557+0000 7ffa99fdc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901c42e0 con 0x55c0c8783950 2024-05-09T08:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.557+0000 7ffa99fdc740 10 client.0 _ll_register_callbacks cb 0x7ffa8c031400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.558+0000 7ffa94868640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c877a2c0 0x55c0c8788c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.558+0000 7ffa94868640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c877a2c0 0x55c0c8788c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43488/0 (socket says 172.21.15.98:43488) 2024-05-09T08:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.558+0000 7ffa94868640 1 -- 172.21.15.98:0/2834138441 learned_addr learned my addr 172.21.15.98:0/2834138441 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.559+0000 7ffa95069640 1 --2- 172.21.15.98:0/2834138441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8778bd0 0x55c0c8788710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.559+0000 7ffa9586a640 1 --2- 172.21.15.98:0/2834138441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c8783950 0x55c0c8789190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.560+0000 7ffa94868640 1 -- 172.21.15.98:0/2834138441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c8783950 msgr2=0x55c0c8789190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.560+0000 7ffa94868640 1 --2- 172.21.15.98:0/2834138441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c8783950 0x55c0c8789190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.560+0000 7ffa94868640 1 -- 172.21.15.98:0/2834138441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8778bd0 msgr2=0x55c0c8788710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.560+0000 7ffa94868640 1 --2- 172.21.15.98:0/2834138441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8778bd0 0x55c0c8788710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.560+0000 7ffa94868640 1 -- 172.21.15.98:0/2834138441 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0c87f9ae0 con 0x55c0c877a2c0 2024-05-09T08:54:25.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.569+0000 7ffa94868640 1 --2- 172.21.15.98:0/2834138441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c877a2c0 0x55c0c8788c50 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ffa880cce00 tx=0x7ffa8800ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d13a43154b775f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:25.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.581+0000 7ffa78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:54:25.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.582+0000 7ffa78ff9640 1 -- 172.21.15.98:0/2834138441 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa880b9060 con 0x55c0c877a2c0 2024-05-09T08:54:25.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.582+0000 7ffa78ff9640 1 -- 172.21.15.98:0/2834138441 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9474+0+0 (secure 0 0 0) 0x7ffa880982a0 con 0x55c0c877a2c0 2024-05-09T08:54:25.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.582+0000 7ffa99fdc740 1 -- 172.21.15.98:0/2834138441 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa80026070 con 0x55c0c877a2c0 2024-05-09T08:54:25.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.586+0000 7ffa78ff9640 1 -- 172.21.15.98:0/2834138441 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ffa8801f040 con 0x55c0c877a2c0 2024-05-09T08:54:25.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.587+0000 7ffa99fdc740 10 client.16764 fetch_fsmap learned FSMap version 33 2024-05-09T08:54:25.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.587+0000 7ffa99fdc740 1 -- 172.21.15.98:0/2834138441 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c0c87faeb0 con 0x55c0c877a2c0 2024-05-09T08:54:25.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.588+0000 7ffa78ff9640 1 -- 172.21.15.98:0/2834138441 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa80026070 con 0x55c0c877a2c0 2024-05-09T08:54:25.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.588+0000 7ffa78ff9640 1 -- 172.21.15.98:0/2834138441 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ffa880b4070 con 0x55c0c877a2c0 2024-05-09T08:54:25.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.589+0000 7ffa99fdc740 10 client.16764 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T08:54:25.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.589+0000 7ffa99fdc740 10 client.16764 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:25.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.589+0000 7ffa99fdc740 20 client.16764 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:25.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.589+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/2834138441 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c0c87fb540 0x55c0c87fd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:25.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.589+0000 7ffa99fdc740 4 client.16764 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:54:25.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.589+0000 7ffa99fdc740 1 -- 172.21.15.98:0/2834138441 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c0c86e5410 con 0x55c0c87fb540 2024-05-09T08:54:25.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.591+0000 7ffa95069640 1 --2- 172.21.15.98:0/2834138441 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c0c87fb540 0x55c0c87fd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.606+0000 7ffa78ff9640 10 client.16764 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:25.606+0000 7ffa95069640 1 --2- 172.21.15.98:0/2834138441 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c0c87fb540 0x55c0c87fd960 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c732f8baaa2ce7e server_cookie=ee91fb3aa9c55542 in_seq=0 out_seq=0 2024-05-09T08:54:26.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:25 smithi142 ceph-mon[34625]: pgmap v117: 129 pgs: 129 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 618 MiB/s wr, 2.79k op/s 2024-05-09T08:54:26.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:25 smithi142 ceph-mon[34625]: daemon mds.e restarted 2024-05-09T08:54:26.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:25 smithi105 ceph-mon[34498]: pgmap v117: 129 pgs: 129 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 618 MiB/s wr, 2.79k op/s 2024-05-09T08:54:26.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:25 smithi105 ceph-mon[34498]: daemon mds.e restarted 2024-05-09T08:54:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:25 smithi098 ceph-mon[30092]: pgmap v117: 129 pgs: 129 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 618 MiB/s wr, 2.79k op/s 2024-05-09T08:54:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:25 smithi098 ceph-mon[30092]: daemon mds.e restarted 2024-05-09T08:54:26.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.477+0000 7ffa78ff9640 1 -- 172.21.15.98:0/2834138441 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c0c86e5410 con 0x55c0c87fb540 2024-05-09T08:54:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.477+0000 7ffa78ff9640 10 client.16764 handle_command_reply: tid=0 2024-05-09T08:54:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.477+0000 7ffa99fdc740 1 client.16764 shutdown 2024-05-09T08:54:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.477+0000 7ffa99fdc740 1 -- 172.21.15.98:0/2834138441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c877a2c0 msgr2=0x55c0c8788c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.477+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/2834138441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c877a2c0 0x55c0c8788c50 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ffa880cce00 tx=0x7ffa8800ea00 comp rx=0 tx=0).stop 2024-05-09T08:54:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.477+0000 7ffa99fdc740 1 -- 172.21.15.98:0/2834138441 shutdown_connections 2024-05-09T08:54:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.477+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/2834138441 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c0c87fb540 0x55c0c87fd960 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.477+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/2834138441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0c8783950 0x55c0c8789190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.477+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/2834138441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0c877a2c0 0x55c0c8788c50 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.477+0000 7ffa99fdc740 1 --2- 172.21.15.98:0/2834138441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0c8778bd0 0x55c0c8788710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.477+0000 7ffa99fdc740 1 -- 172.21.15.98:0/2834138441 >> 172.21.15.98:0/2834138441 conn(0x55c0c87410a0 msgr2=0x55c0c8780fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:26.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.477+0000 7ffa78ff9640 0 client.16764 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:26.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.478+0000 7ffa99fdc740 1 -- 172.21.15.98:0/2834138441 shutdown_connections 2024-05-09T08:54:26.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.478+0000 7ffa99fdc740 1 -- 172.21.15.98:0/2834138441 wait complete. 2024-05-09T08:54:26.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.478+0000 7ffa99fdc740 20 client.16764 trim_cache size 0 max 16384 2024-05-09T08:54:26.481 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:54:26.481 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:54:26.481 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "c6b6ef5f-f2f0-4cb5-a3b5-4dd749b10c3e", 2024-05-09T08:54:26.481 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:54:26.481 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:54:26.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.491+0000 7ffa9606b640 1 -- 172.21.15.98:0/371100446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffa7c0a2f00 msgr2=0x7ffa7c0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:26.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.491+0000 7ffa9606b640 1 --2- 172.21.15.98:0/371100446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffa7c0a2f00 0x7ffa7c0a53c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55c0c87762c0 tx=0x7ffa80006d50 comp rx=0 tx=0).stop 2024-05-09T08:54:26.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.491+0000 7ffa9606b640 1 -- 172.21.15.98:0/371100446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa9010e1a0 msgr2=0x7ffa9019e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:26.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.491+0000 7ffa9606b640 1 --2- 172.21.15.98:0/371100446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa9010e1a0 0x7ffa9019e890 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800ea80 tx=0x7ffa8800ef40 comp rx=0 tx=0).stop 2024-05-09T08:54:26.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.495+0000 7ffa9606b640 1 -- 172.21.15.98:0/371100446 shutdown_connections 2024-05-09T08:54:26.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.495+0000 7ffa9606b640 1 --2- 172.21.15.98:0/371100446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffa7c0a2f00 0x7ffa7c0a53c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.495+0000 7ffa9606b640 1 --2- 172.21.15.98:0/371100446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa9010eb60 0x7ffa9019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.495+0000 7ffa9606b640 1 --2- 172.21.15.98:0/371100446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa9010e1a0 0x7ffa9019e890 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.495+0000 7ffa9606b640 1 --2- 172.21.15.98:0/371100446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9010d4b0 0x7ffa9019e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.495+0000 7ffa9606b640 1 -- 172.21.15.98:0/371100446 >> 172.21.15.98:0/371100446 conn(0x7ffa9006c710 msgr2=0x7ffa9010c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:26.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.496+0000 7ffa9606b640 1 -- 172.21.15.98:0/371100446 shutdown_connections 2024-05-09T08:54:26.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.500+0000 7ffa9606b640 1 -- 172.21.15.98:0/371100446 wait complete. 2024-05-09T08:54:26.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:54:26.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:54:26.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.803+0000 7f3e5f59e640 1 -- 172.21.15.98:0/3126954990 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e601092b0 msgr2=0x7f3e6011d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:26.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.803+0000 7f3e5f59e640 1 --2- 172.21.15.98:0/3126954990 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e601092b0 0x7f3e6011d020 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009900 tx=0x7f3e5002a4d0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.805+0000 7f3e5f59e640 1 -- 172.21.15.98:0/3126954990 shutdown_connections 2024-05-09T08:54:26.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.805+0000 7f3e5f59e640 1 --2- 172.21.15.98:0/3126954990 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e601092b0 0x7f3e6011d020 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.805+0000 7f3e5f59e640 1 --2- 172.21.15.98:0/3126954990 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e60108830 0x7f3e60108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.805+0000 7f3e5f59e640 1 --2- 172.21.15.98:0/3126954990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e6010fed0 0x7f3e601102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.805+0000 7f3e5f59e640 1 -- 172.21.15.98:0/3126954990 >> 172.21.15.98:0/3126954990 conn(0x7f3e6006c7b0 msgr2=0x7f3e6006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:26.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.808+0000 7f3e5f59e640 1 -- 172.21.15.98:0/3126954990 shutdown_connections 2024-05-09T08:54:26.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.812+0000 7f3e5f59e640 1 -- 172.21.15.98:0/3126954990 wait complete. 2024-05-09T08:54:26.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.813+0000 7f3e5f59e640 1 Processor -- start 2024-05-09T08:54:26.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.813+0000 7f3e5f59e640 1 -- start start 2024-05-09T08:54:26.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.814+0000 7f3e5f59e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e60108830 0x7f3e6011cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.814+0000 7f3e5f59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e601092b0 0x7f3e6011d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.814+0000 7f3e5f59e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e6010fed0 0x7f3e60115db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.814+0000 7f3e5f59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e60120950 con 0x7f3e601092b0 2024-05-09T08:54:26.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.814+0000 7f3e5f59e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3e601207d0 con 0x7f3e6010fed0 2024-05-09T08:54:26.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.814+0000 7f3e5f59e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3e60120ad0 con 0x7f3e60108830 2024-05-09T08:54:26.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.815+0000 7f3e5dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e601092b0 0x7f3e6011d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.815+0000 7f3e5dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e601092b0 0x7f3e6011d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50828/0 (socket says 172.21.15.98:50828) 2024-05-09T08:54:26.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.815+0000 7f3e5dd9b640 1 -- 172.21.15.98:0/899966691 learned_addr learned my addr 172.21.15.98:0/899966691 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:26.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.816+0000 7f3e5ed9d640 1 --2- 172.21.15.98:0/899966691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e6010fed0 0x7f3e60115db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.820+0000 7f3e5e59c640 1 --2- 172.21.15.98:0/899966691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e60108830 0x7f3e6011cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.820+0000 7f3e5dd9b640 1 -- 172.21.15.98:0/899966691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e60108830 msgr2=0x7f3e6011cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.820+0000 7f3e5dd9b640 1 --2- 172.21.15.98:0/899966691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e60108830 0x7f3e6011cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.820+0000 7f3e5dd9b640 1 -- 172.21.15.98:0/899966691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e6010fed0 msgr2=0x7f3e60115db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.820+0000 7f3e5dd9b640 1 --2- 172.21.15.98:0/899966691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e6010fed0 0x7f3e60115db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.820+0000 7f3e5dd9b640 1 -- 172.21.15.98:0/899966691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e60116670 con 0x7f3e601092b0 2024-05-09T08:54:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.823+0000 7f3e5dd9b640 1 --2- 172.21.15.98:0/899966691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e601092b0 0x7f3e6011d0c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400e9f0 tx=0x7f3e5400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:26.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.825+0000 7f3e4f7fe640 1 -- 172.21.15.98:0/899966691 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5400cd80 con 0x7f3e601092b0 2024-05-09T08:54:26.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.825+0000 7f3e4f7fe640 1 -- 172.21.15.98:0/899966691 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e54010910 con 0x7f3e601092b0 2024-05-09T08:54:26.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.825+0000 7f3e4f7fe640 1 -- 172.21.15.98:0/899966691 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e540188f0 con 0x7f3e601092b0 2024-05-09T08:54:26.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.825+0000 7f3e5f59e640 1 -- 172.21.15.98:0/899966691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e60116960 con 0x7f3e601092b0 2024-05-09T08:54:26.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.825+0000 7f3e5f59e640 1 -- 172.21.15.98:0/899966691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e6011a940 con 0x7f3e601092b0 2024-05-09T08:54:26.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.833+0000 7f3e677c3740 1 Processor -- start 2024-05-09T08:54:26.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.833+0000 7f3e4f7fe640 1 -- 172.21.15.98:0/899966691 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3e54010ac0 con 0x7f3e601092b0 2024-05-09T08:54:26.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.834+0000 7f3e4f7fe640 1 --2- 172.21.15.98:0/899966691 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e480a2ed0 0x7f3e480a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.834+0000 7f3e4f7fe640 1 -- 172.21.15.98:0/899966691 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f3e540cfb60 con 0x7f3e601092b0 2024-05-09T08:54:26.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.835+0000 7f3e677c3740 1 -- start start 2024-05-09T08:54:26.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.836+0000 7f3e677c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166470 0x560082166850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.836+0000 7f3e677c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082166d90 0x560082170f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.836+0000 7f3e677c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082171460 0x560082173850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.836+0000 7f3e677c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5600820cbd00 con 0x560082171460 2024-05-09T08:54:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.836+0000 7f3e677c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5600820734c0 con 0x560082166470 2024-05-09T08:54:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.836+0000 7f3e677c3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5600820c7cc0 con 0x560082166d90 2024-05-09T08:54:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.837+0000 7f3e5ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082171460 0x560082173850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.837+0000 7f3e5ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082171460 0x560082173850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50832/0 (socket says 172.21.15.98:50832) 2024-05-09T08:54:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.837+0000 7f3e5ed9d640 1 -- 172.21.15.98:0/2190649982 learned_addr learned my addr 172.21.15.98:0/2190649982 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:26.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.837+0000 7f3e5dd9b640 1 --2- 172.21.15.98:0/2190649982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082166d90 0x560082170f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.837+0000 7f3e5e59c640 1 --2- 172.21.15.98:0/2190649982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166470 0x560082166850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.841+0000 7f3e5ed9d640 1 --2- 172.21.15.98:0/899966691 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e480a2ed0 0x7f3e480a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.842+0000 7f3e5e59c640 1 -- 172.21.15.98:0/2190649982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082166d90 msgr2=0x560082170f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:26.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.842+0000 7f3e5e59c640 1 --2- 172.21.15.98:0/2190649982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082166d90 0x560082170f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.842+0000 7f3e5e59c640 1 -- 172.21.15.98:0/2190649982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082171460 msgr2=0x560082173850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:26.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.842+0000 7f3e5e59c640 1 --2- 172.21.15.98:0/2190649982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082171460 0x560082173850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.842+0000 7f3e5e59c640 1 -- 172.21.15.98:0/2190649982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560081ea97d0 con 0x560082166470 2024-05-09T08:54:26.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.843+0000 7f3e5e59c640 1 --2- 172.21.15.98:0/2190649982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166470 0x560082166850 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3e58007d50 tx=0x7f3e5800c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1400b2b123a49055 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:26.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.845+0000 7f3e4cff9640 1 -- 172.21.15.98:0/2190649982 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e58015020 con 0x560082166470 2024-05-09T08:54:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.845+0000 7f3e4cff9640 1 -- 172.21.15.98:0/2190649982 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e5800ce50 con 0x560082166470 2024-05-09T08:54:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.845+0000 7f3e4cff9640 1 -- 172.21.15.98:0/2190649982 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5800d040 con 0x560082166470 2024-05-09T08:54:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.846+0000 7f3e677c3740 1 -- 172.21.15.98:0/2190649982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166470 msgr2=0x560082166850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.846+0000 7f3e677c3740 1 --2- 172.21.15.98:0/2190649982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166470 0x560082166850 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3e58007d50 tx=0x7f3e5800c540 comp rx=0 tx=0).stop 2024-05-09T08:54:26.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.849+0000 7f3e677c3740 1 -- 172.21.15.98:0/2190649982 shutdown_connections 2024-05-09T08:54:26.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.849+0000 7f3e677c3740 1 --2- 172.21.15.98:0/2190649982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082171460 0x560082173850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.849+0000 7f3e677c3740 1 --2- 172.21.15.98:0/2190649982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082166d90 0x560082170f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.849+0000 7f3e677c3740 1 --2- 172.21.15.98:0/2190649982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166470 0x560082166850 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.849+0000 7f3e677c3740 1 -- 172.21.15.98:0/2190649982 >> 172.21.15.98:0/2190649982 conn(0x560082162790 msgr2=0x560082162b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:26.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.851+0000 7f3e5ed9d640 1 --2- 172.21.15.98:0/899966691 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e480a2ed0 0x7f3e480a5390 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3e50005ab0 tx=0x7f3e50005a20 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:26.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.854+0000 7f3e677c3740 1 -- 172.21.15.98:0/2190649982 shutdown_connections 2024-05-09T08:54:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.858+0000 7f3e677c3740 1 -- 172.21.15.98:0/2190649982 wait complete. 2024-05-09T08:54:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.859+0000 7f3e677c3740 1 -- start start 2024-05-09T08:54:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.859+0000 7f3e677c3740 1 Processor -- start 2024-05-09T08:54:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.860+0000 7f3e677c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082166470 0x56008216e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.860+0000 7f3e677c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 0x56008216e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.860+0000 7f3e677c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082171460 0x56008216eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.860+0000 7f3e677c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560082174250 con 0x560082166470 2024-05-09T08:54:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.860+0000 7f3e677c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560082086b90 con 0x560082166d90 2024-05-09T08:54:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.860+0000 7f3e677c3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5600821743d0 con 0x560082171460 2024-05-09T08:54:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.860+0000 7f3e677c3740 10 client.0 _ll_register_callbacks cb 0x5600820c2da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.860+0000 7f3e5dd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 0x56008216e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.862+0000 7f3e5e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082166470 0x56008216e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.862+0000 7f3e5e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082166470 0x56008216e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50848/0 (socket says 172.21.15.98:50848) 2024-05-09T08:54:26.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.862+0000 7f3e5e59c640 1 -- 172.21.15.98:0/378851529 learned_addr learned my addr 172.21.15.98:0/378851529 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:26.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.862+0000 7f3e5ed9d640 1 --2- 172.21.15.98:0/378851529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082171460 0x56008216eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.863+0000 7f3e5e59c640 1 -- 172.21.15.98:0/378851529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082171460 msgr2=0x56008216eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:26.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.863+0000 7f3e5e59c640 1 --2- 172.21.15.98:0/378851529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082171460 0x56008216eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.863+0000 7f3e5e59c640 1 -- 172.21.15.98:0/378851529 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 msgr2=0x56008216e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:26.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.863+0000 7f3e5e59c640 1 --2- 172.21.15.98:0/378851529 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 0x56008216e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.863+0000 7f3e5e59c640 1 -- 172.21.15.98:0/378851529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560081ddf3f0 con 0x560082166470 2024-05-09T08:54:26.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.864+0000 7f3e5e59c640 1 --2- 172.21.15.98:0/378851529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082166470 0x56008216e420 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800ca40 tx=0x7f3e58011bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6554c74bacb7593 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:26.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.864+0000 7f3e45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:54:26.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.866+0000 7f3e45ffb640 1 -- 172.21.15.98:0/378851529 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e58010510 con 0x560082166470 2024-05-09T08:54:26.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.867+0000 7f3e677c3740 1 -- 172.21.15.98:0/378851529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e5800ce50 con 0x560082166470 2024-05-09T08:54:26.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.867+0000 7f3e45ffb640 1 -- 172.21.15.98:0/378851529 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f3e580106b0 con 0x560082166470 2024-05-09T08:54:26.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.868+0000 7f3e45ffb640 1 -- 172.21.15.98:0/378851529 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3e5802f050 con 0x560082166470 2024-05-09T08:54:26.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.869+0000 7f3e677c3740 10 client.16773 fetch_fsmap learned FSMap version 35 2024-05-09T08:54:26.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.869+0000 7f3e677c3740 1 -- 172.21.15.98:0/378851529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560081d9f0f0 con 0x560082166470 2024-05-09T08:54:26.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.881+0000 7f3e45ffb640 1 -- 172.21.15.98:0/378851529 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e5800ce50 con 0x560082166470 2024-05-09T08:54:26.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.881+0000 7f3e45ffb640 1 -- 172.21.15.98:0/378851529 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3e58010050 con 0x560082166470 2024-05-09T08:54:26.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.881+0000 7f3e677c3740 10 client.16773 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:54:26.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.881+0000 7f3e677c3740 10 client.16773 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:26.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.881+0000 7f3e677c3740 20 client.16773 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:26.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.881+0000 7f3e677c3740 1 --2- 172.21.15.98:0/378851529 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x560082181a90 0x560082183ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.881+0000 7f3e677c3740 4 client.16773 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:26.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.881+0000 7f3e677c3740 1 -- 172.21.15.98:0/378851529 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600820d9430 con 0x560082181a90 2024-05-09T08:54:26.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.884+0000 7f3e5dd9b640 1 --2- 172.21.15.98:0/378851529 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x560082181a90 0x560082183ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.888+0000 7f3e45ffb640 10 client.16773 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:26.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.888+0000 7f3e5dd9b640 1 --2- 172.21.15.98:0/378851529 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x560082181a90 0x560082183ef0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1311df7fa8892e91 server_cookie=5d90cc8c24886cb0 in_seq=0 out_seq=0 2024-05-09T08:54:26.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.903+0000 7f3e45ffb640 1 -- 172.21.15.98:0/378851529 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5600820d9430 con 0x560082181a90 2024-05-09T08:54:26.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.903+0000 7f3e45ffb640 10 client.16773 handle_command_reply: tid=0 2024-05-09T08:54:26.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.903+0000 7f3e677c3740 1 client.16773 shutdown 2024-05-09T08:54:26.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.907+0000 7f3e677c3740 1 -- 172.21.15.98:0/378851529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082166470 msgr2=0x56008216e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:26.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.907+0000 7f3e677c3740 1 --2- 172.21.15.98:0/378851529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082166470 0x56008216e420 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800ca40 tx=0x7f3e58011bd0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.907+0000 7f3e677c3740 1 -- 172.21.15.98:0/378851529 shutdown_connections 2024-05-09T08:54:26.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.907+0000 7f3e677c3740 1 --2- 172.21.15.98:0/378851529 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x560082181a90 0x560082183ef0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.907+0000 7f3e677c3740 1 --2- 172.21.15.98:0/378851529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082171460 0x56008216eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.907+0000 7f3e677c3740 1 --2- 172.21.15.98:0/378851529 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 0x56008216e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.907+0000 7f3e677c3740 1 --2- 172.21.15.98:0/378851529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082166470 0x56008216e420 secure :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800ca40 tx=0x7f3e58011bd0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.907+0000 7f3e677c3740 1 -- 172.21.15.98:0/378851529 >> 172.21.15.98:0/378851529 conn(0x560082162790 msgr2=0x560082168590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:26.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.908+0000 7f3e45ffb640 0 client.16773 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.912+0000 7f3e677c3740 1 -- 172.21.15.98:0/378851529 shutdown_connections 2024-05-09T08:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.912+0000 7f3e677c3740 1 -- 172.21.15.98:0/378851529 wait complete. 2024-05-09T08:54:26.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.916+0000 7f3e677c3740 20 client.16773 trim_cache size 0 max 16384 2024-05-09T08:54:26.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.956+0000 7f3e677c3740 1 Processor -- start 2024-05-09T08:54:26.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.956+0000 7f3e677c3740 1 -- start start 2024-05-09T08:54:26.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.956+0000 7f3e677c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 0x56008217d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.956+0000 7f3e677c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082171460 0x56008216ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.956+0000 7f3e677c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082189800 0x560082189c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.956+0000 7f3e677c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e60130760 con 0x560082189800 2024-05-09T08:54:26.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.956+0000 7f3e677c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3e601d8e90 con 0x560082166d90 2024-05-09T08:54:26.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.956+0000 7f3e677c3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3e60130e30 con 0x560082171460 2024-05-09T08:54:26.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.956+0000 7f3e5ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082189800 0x560082189c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.957+0000 7f3e5ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082189800 0x560082189c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50864/0 (socket says 172.21.15.98:50864) 2024-05-09T08:54:26.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.957+0000 7f3e5ed9d640 1 -- 172.21.15.98:0/119510622 learned_addr learned my addr 172.21.15.98:0/119510622 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:26.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.957+0000 7f3e5dd9b640 1 --2- 172.21.15.98:0/119510622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082171460 0x56008216ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.959+0000 7f3e5e59c640 1 --2- 172.21.15.98:0/119510622 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 0x56008217d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.960+0000 7f3e5dd9b640 1 -- 172.21.15.98:0/119510622 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 msgr2=0x56008217d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:26.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.960+0000 7f3e5dd9b640 1 --2- 172.21.15.98:0/119510622 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 0x56008217d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.960+0000 7f3e5dd9b640 1 -- 172.21.15.98:0/119510622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082189800 msgr2=0x560082189c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:26.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.960+0000 7f3e5dd9b640 1 --2- 172.21.15.98:0/119510622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082189800 0x560082189c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.960+0000 7f3e5dd9b640 1 -- 172.21.15.98:0/119510622 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56008215ce40 con 0x560082171460 2024-05-09T08:54:26.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.961+0000 7f3e5dd9b640 1 --2- 172.21.15.98:0/119510622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082171460 0x56008216ecf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3e540cce90 tx=0x7f3e5409cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f768a51c10c53a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:26.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.967+0000 7f3e47fff640 1 -- 172.21.15.98:0/119510622 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e540b4e60 con 0x560082171460 2024-05-09T08:54:26.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.967+0000 7f3e47fff640 1 -- 172.21.15.98:0/119510622 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e540b1040 con 0x560082171460 2024-05-09T08:54:26.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.968+0000 7f3e677c3740 1 -- 172.21.15.98:0/119510622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082171460 msgr2=0x56008216ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:26.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.968+0000 7f3e677c3740 1 --2- 172.21.15.98:0/119510622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082171460 0x56008216ecf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3e540cce90 tx=0x7f3e5409cf90 comp rx=0 tx=0).stop 2024-05-09T08:54:26.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.968+0000 7f3e677c3740 1 -- 172.21.15.98:0/119510622 shutdown_connections 2024-05-09T08:54:26.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.968+0000 7f3e677c3740 1 --2- 172.21.15.98:0/119510622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082189800 0x560082189c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.968+0000 7f3e677c3740 1 --2- 172.21.15.98:0/119510622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082171460 0x56008216ecf0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.968+0000 7f3e677c3740 1 --2- 172.21.15.98:0/119510622 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 0x56008217d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.968+0000 7f3e677c3740 1 -- 172.21.15.98:0/119510622 >> 172.21.15.98:0/119510622 conn(0x560082162790 msgr2=0x560082167bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.970+0000 7f3e677c3740 1 -- 172.21.15.98:0/119510622 shutdown_connections 2024-05-09T08:54:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.970+0000 7f3e677c3740 1 -- 172.21.15.98:0/119510622 wait complete. 2024-05-09T08:54:26.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.971+0000 7f3e677c3740 1 -- start start 2024-05-09T08:54:26.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.971+0000 7f3e677c3740 1 Processor -- start 2024-05-09T08:54:26.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.971+0000 7f3e677c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 0x560082179980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.971+0000 7f3e677c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082171460 0x56008216adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.971+0000 7f3e677c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082189800 0x56008216b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.971+0000 7f3e677c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e60131bd0 con 0x560082171460 2024-05-09T08:54:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.971+0000 7f3e677c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3e60131500 con 0x560082166d90 2024-05-09T08:54:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.971+0000 7f3e677c3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3e60068570 con 0x560082189800 2024-05-09T08:54:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.971+0000 7f3e677c3740 10 client.0 _ll_register_callbacks cb 0x7f3e540977e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.972+0000 7f3e5dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082171460 0x56008216adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.972+0000 7f3e5dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082171460 0x56008216adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50880/0 (socket says 172.21.15.98:50880) 2024-05-09T08:54:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.972+0000 7f3e5dd9b640 1 -- 172.21.15.98:0/995496801 learned_addr learned my addr 172.21.15.98:0/995496801 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:26.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.973+0000 7f3e5ed9d640 1 --2- 172.21.15.98:0/995496801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082189800 0x56008216b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.973+0000 7f3e5e59c640 1 --2- 172.21.15.98:0/995496801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 0x560082179980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:26.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.974+0000 7f3e5ed9d640 1 -- 172.21.15.98:0/995496801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 msgr2=0x560082179980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:26.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.974+0000 7f3e5ed9d640 1 --2- 172.21.15.98:0/995496801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 0x560082179980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.974+0000 7f3e5ed9d640 1 -- 172.21.15.98:0/995496801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082171460 msgr2=0x56008216adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:26.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.974+0000 7f3e5ed9d640 1 --2- 172.21.15.98:0/995496801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082171460 0x56008216adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:26.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.974+0000 7f3e5ed9d640 1 -- 172.21.15.98:0/995496801 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56008216bac0 con 0x560082189800 2024-05-09T08:54:26.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.974+0000 7f3e5dd9b640 1 --2- 172.21.15.98:0/995496801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082171460 0x56008216adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:54:26.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.974+0000 7f3e5ed9d640 1 --2- 172.21.15.98:0/995496801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082189800 0x56008216b2f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3e50033040 tx=0x7f3e5002a4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74805368d7c89719 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:26.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.975+0000 7f3e45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:54:26.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.975+0000 7f3e45ffb640 1 -- 172.21.15.98:0/995496801 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5004d920 con 0x560082189800 2024-05-09T08:54:26.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.975+0000 7f3e45ffb640 1 -- 172.21.15.98:0/995496801 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f3e5004dac0 con 0x560082189800 2024-05-09T08:54:26.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.975+0000 7f3e677c3740 1 -- 172.21.15.98:0/995496801 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e540b1040 con 0x560082189800 2024-05-09T08:54:26.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.983+0000 7f3e45ffb640 1 -- 172.21.15.98:0/995496801 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3e5004dcc0 con 0x560082189800 2024-05-09T08:54:26.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.984+0000 7f3e677c3740 10 client.26041 fetch_fsmap learned FSMap version 35 2024-05-09T08:54:26.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.984+0000 7f3e677c3740 1 -- 172.21.15.98:0/995496801 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5600821e8080 con 0x560082189800 2024-05-09T08:54:26.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.989+0000 7f3e45ffb640 1 -- 172.21.15.98:0/995496801 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e540b1040 con 0x560082189800 2024-05-09T08:54:26.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.989+0000 7f3e45ffb640 1 -- 172.21.15.98:0/995496801 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3e50055070 con 0x560082189800 2024-05-09T08:54:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.990+0000 7f3e677c3740 10 client.26041 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:54:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.990+0000 7f3e677c3740 10 client.26041 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.990+0000 7f3e677c3740 20 client.26041 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.990+0000 7f3e677c3740 1 --2- 172.21.15.98:0/995496801 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5600821e87f0 0x5600821eac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.990+0000 7f3e677c3740 4 client.26041 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:54:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.990+0000 7f3e677c3740 1 -- 172.21.15.98:0/995496801 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5600820d3780 con 0x5600821e87f0 2024-05-09T08:54:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:26.994+0000 7f3e5e59c640 1 --2- 172.21.15.98:0/995496801 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5600821e87f0 0x5600821eac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:27.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.007+0000 7f3e5e59c640 1 --2- 172.21.15.98:0/995496801 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5600821e87f0 0x5600821eac10 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49d1a1029bfe248a server_cookie=49bd7f91490f6526 in_seq=0 out_seq=0 2024-05-09T08:54:27.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.007+0000 7f3e45ffb640 10 client.26041 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:27.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.009+0000 7f3e45ffb640 1 -- 172.21.15.98:0/995496801 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x5600820d3780 con 0x5600821e87f0 2024-05-09T08:54:27.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.009+0000 7f3e45ffb640 10 client.26041 handle_command_reply: tid=0 2024-05-09T08:54:27.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.009+0000 7f3e677c3740 1 client.26041 shutdown 2024-05-09T08:54:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.014+0000 7f3e677c3740 1 -- 172.21.15.98:0/995496801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082189800 msgr2=0x56008216b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.014+0000 7f3e677c3740 1 --2- 172.21.15.98:0/995496801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082189800 0x56008216b2f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3e50033040 tx=0x7f3e5002a4d0 comp rx=0 tx=0).stop 2024-05-09T08:54:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.014+0000 7f3e677c3740 1 -- 172.21.15.98:0/995496801 shutdown_connections 2024-05-09T08:54:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.014+0000 7f3e677c3740 1 --2- 172.21.15.98:0/995496801 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5600821e87f0 0x5600821eac10 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.014+0000 7f3e677c3740 1 --2- 172.21.15.98:0/995496801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560082189800 0x56008216b2f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.014+0000 7f3e677c3740 1 --2- 172.21.15.98:0/995496801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560082171460 0x56008216adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.014+0000 7f3e677c3740 1 --2- 172.21.15.98:0/995496801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560082166d90 0x560082179980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.014+0000 7f3e677c3740 1 -- 172.21.15.98:0/995496801 >> 172.21.15.98:0/995496801 conn(0x560082162790 msgr2=0x560082189080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.014+0000 7f3e45ffb640 0 client.26041 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:27.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.017+0000 7f3e677c3740 1 -- 172.21.15.98:0/995496801 shutdown_connections 2024-05-09T08:54:27.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.020+0000 7f3e677c3740 1 -- 172.21.15.98:0/995496801 wait complete. 2024-05-09T08:54:27.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.021+0000 7f3e677c3740 20 client.26041 trim_cache size 0 max 16384 2024-05-09T08:54:27.022 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:54:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "scrub active (1 inodes in the stack)", 2024-05-09T08:54:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:54:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "c6b6ef5f-f2f0-4cb5-a3b5-4dd749b10c3e": { 2024-05-09T08:54:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:54:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "c6b6ef5f-f2f0-4cb5-a3b5-4dd749b10c3e", 2024-05-09T08:54:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:54:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:54:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:54:27.022 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:54:27.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.027+0000 7f3e5f59e640 1 -- 172.21.15.98:0/899966691 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e480a2ed0 msgr2=0x7f3e480a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:27.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.027+0000 7f3e5f59e640 1 --2- 172.21.15.98:0/899966691 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e480a2ed0 0x7f3e480a5390 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3e50005ab0 tx=0x7f3e50005a20 comp rx=0 tx=0).stop 2024-05-09T08:54:27.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.027+0000 7f3e5f59e640 1 -- 172.21.15.98:0/899966691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e601092b0 msgr2=0x7f3e6011d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:27.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.027+0000 7f3e5f59e640 1 --2- 172.21.15.98:0/899966691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e601092b0 0x7f3e6011d0c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400e9f0 tx=0x7f3e5400eeb0 comp rx=0 tx=0).stop 2024-05-09T08:54:27.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.028+0000 7f3e5f59e640 1 -- 172.21.15.98:0/899966691 shutdown_connections 2024-05-09T08:54:27.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.028+0000 7f3e5f59e640 1 --2- 172.21.15.98:0/899966691 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e480a2ed0 0x7f3e480a5390 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:27.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.028+0000 7f3e5f59e640 1 --2- 172.21.15.98:0/899966691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e6010fed0 0x7f3e60115db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:27.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.028+0000 7f3e5f59e640 1 --2- 172.21.15.98:0/899966691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e601092b0 0x7f3e6011d0c0 secure :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400e9f0 tx=0x7f3e5400eeb0 comp rx=0 tx=0).stop 2024-05-09T08:54:27.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.028+0000 7f3e5f59e640 1 --2- 172.21.15.98:0/899966691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e60108830 0x7f3e6011cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:27.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.028+0000 7f3e5f59e640 1 -- 172.21.15.98:0/899966691 >> 172.21.15.98:0/899966691 conn(0x7f3e6006c7b0 msgr2=0x7f3e6010d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:27.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.032+0000 7f3e5f59e640 1 -- 172.21.15.98:0/899966691 shutdown_connections 2024-05-09T08:54:27.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:27.034+0000 7f3e5f59e640 1 -- 172.21.15.98:0/899966691 wait complete. 2024-05-09T08:54:27.072 INFO:tasks.cephfs.filesystem:scrub status for tag:c6b6ef5f-f2f0-4cb5-a3b5-4dd749b10c3e - {'path': '/', 'tag': 'c6b6ef5f-f2f0-4cb5-a3b5-4dd749b10c3e', 'options': 'recursive,force'} 2024-05-09T08:54:27.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:26 smithi142 ceph-mon[34625]: osdmap e97: 12 total, 12 up, 12 in 2024-05-09T08:54:27.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:26 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.105:6834/3204012039,v1:172.21.15.105:6835/3204012039] up:boot 2024-05-09T08:54:27.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:26 smithi142 ceph-mon[34625]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:54:27.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:26 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:54:27.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:26 smithi142 ceph-mon[34625]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:54:27.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:26 smithi105 ceph-mon[34498]: osdmap e97: 12 total, 12 up, 12 in 2024-05-09T08:54:27.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:26 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.105:6834/3204012039,v1:172.21.15.105:6835/3204012039] up:boot 2024-05-09T08:54:27.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:26 smithi105 ceph-mon[34498]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:54:27.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:26 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:54:27.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:26 smithi105 ceph-mon[34498]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:54:27.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:26 smithi098 ceph-mon[30092]: osdmap e97: 12 total, 12 up, 12 in 2024-05-09T08:54:27.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:26 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.105:6834/3204012039,v1:172.21.15.105:6835/3204012039] up:boot 2024-05-09T08:54:27.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:26 smithi098 ceph-mon[30092]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:54:27.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:26 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:54:27.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:26 smithi098 ceph-mon[30092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:54:28.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:27 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:54:28.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:27 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:54:28.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:27 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:54:28.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:27 smithi142 ceph-mon[34625]: pgmap v119: 129 pgs: 129 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 669 MiB/s wr, 2.34k op/s 2024-05-09T08:54:28.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:27 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:54:28.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:27 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:54:28.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:27 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:54:28.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:27 smithi105 ceph-mon[34498]: pgmap v119: 129 pgs: 129 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 669 MiB/s wr, 2.34k op/s 2024-05-09T08:54:28.358 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:27 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:54:28.358 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:27 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:54:28.359 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:27 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:54:28.359 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:27 smithi098 ceph-mon[30092]: pgmap v119: 129 pgs: 129 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 669 MiB/s wr, 2.34k op/s 2024-05-09T08:54:29.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:28 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:54:29.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:28 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:54:29.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:28 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:54:30.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:29 smithi105 ceph-mon[34498]: pgmap v120: 129 pgs: 129 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 685 MiB/s wr, 2.08k op/s 2024-05-09T08:54:30.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:29 smithi098 ceph-mon[30092]: pgmap v120: 129 pgs: 129 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 685 MiB/s wr, 2.08k op/s 2024-05-09T08:54:30.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:29 smithi142 ceph-mon[34625]: pgmap v120: 129 pgs: 129 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 685 MiB/s wr, 2.08k op/s 2024-05-09T08:54:32.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:31 smithi105 ceph-mon[34498]: pgmap v121: 129 pgs: 129 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 756 MiB/s wr, 1.25k op/s 2024-05-09T08:54:32.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:31 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:54:32.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:31 smithi098 ceph-mon[30092]: pgmap v121: 129 pgs: 129 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 756 MiB/s wr, 1.25k op/s 2024-05-09T08:54:32.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:31 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:54:32.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:31 smithi142 ceph-mon[34625]: pgmap v121: 129 pgs: 129 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 756 MiB/s wr, 1.25k op/s 2024-05-09T08:54:32.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:31 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:54:33.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.808+0000 7f787c82a640 1 -- 172.21.15.98:0/4094770260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f787810fef0 msgr2=0x7f787810d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:33.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.808+0000 7f787c82a640 1 --2- 172.21.15.98:0/4094770260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f787810fef0 0x7f787810d9a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f786c009900 tx=0x7f786c02a4d0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.810 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.811+0000 7f787c82a640 1 -- 172.21.15.98:0/4094770260 shutdown_connections 2024-05-09T08:54:33.810 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.811+0000 7f787c82a640 1 --2- 172.21.15.98:0/4094770260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787810e500 0x7f787811d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.810 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.811+0000 7f787c82a640 1 --2- 172.21.15.98:0/4094770260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78781102d0 0x7f787810dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.810 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.811+0000 7f787c82a640 1 --2- 172.21.15.98:0/4094770260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f787810fef0 0x7f787810d9a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.811+0000 7f787c82a640 1 -- 172.21.15.98:0/4094770260 >> 172.21.15.98:0/4094770260 conn(0x7f787806c770 msgr2=0x7f787806cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.814+0000 7f787c82a640 1 -- 172.21.15.98:0/4094770260 shutdown_connections 2024-05-09T08:54:33.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.815+0000 7f787c82a640 1 -- 172.21.15.98:0/4094770260 wait complete. 2024-05-09T08:54:33.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.816+0000 7f787c82a640 1 Processor -- start 2024-05-09T08:54:33.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.816+0000 7f787c82a640 1 -- start start 2024-05-09T08:54:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.817+0000 7f787c82a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787810e500 0x7f78781a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.817+0000 7f787c82a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f787810fef0 0x7f78781a7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.817+0000 7f787c82a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78781102d0 0x7f78781ab790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.817+0000 7f787c82a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7878120970 con 0x7f787810e500 2024-05-09T08:54:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.817+0000 7f787c82a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f78781207f0 con 0x7f787810fef0 2024-05-09T08:54:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.817+0000 7f787c82a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7878120af0 con 0x7f78781102d0 2024-05-09T08:54:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.818+0000 7f78777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787810e500 0x7f78781a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.818+0000 7f78777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787810e500 0x7f78781a6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50892/0 (socket says 172.21.15.98:50892) 2024-05-09T08:54:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.818+0000 7f78777fe640 1 -- 172.21.15.98:0/666324311 learned_addr learned my addr 172.21.15.98:0/666324311 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.819+0000 7f78777fe640 1 -- 172.21.15.98:0/666324311 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78781102d0 msgr2=0x7f78781ab790 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:54:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.819+0000 7f78777fe640 1 --2- 172.21.15.98:0/666324311 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78781102d0 0x7f78781ab790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.819+0000 7f78777fe640 1 -- 172.21.15.98:0/666324311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f787810fef0 msgr2=0x7f78781a7410 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:54:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.819+0000 7f78777fe640 1 --2- 172.21.15.98:0/666324311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f787810fef0 0x7f78781a7410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.819+0000 7f78777fe640 1 -- 172.21.15.98:0/666324311 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78781abec0 con 0x7f787810e500 2024-05-09T08:54:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.820+0000 7f78777fe640 1 --2- 172.21.15.98:0/666324311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787810e500 0x7f78781a6ed0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f786c02aa40 tx=0x7f786c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.820+0000 7f7874ff9640 1 -- 172.21.15.98:0/666324311 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c03f070 con 0x7f787810e500 2024-05-09T08:54:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.820+0000 7f7874ff9640 1 -- 172.21.15.98:0/666324311 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f786c031c20 con 0x7f787810e500 2024-05-09T08:54:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.820+0000 7f7874ff9640 1 -- 172.21.15.98:0/666324311 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c035030 con 0x7f787810e500 2024-05-09T08:54:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.821+0000 7f787c82a640 1 -- 172.21.15.98:0/666324311 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78781ac150 con 0x7f787810e500 2024-05-09T08:54:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.821+0000 7f787c82a640 1 -- 172.21.15.98:0/666324311 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78781b3a40 con 0x7f787810e500 2024-05-09T08:54:33.830 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.830+0000 7f7874ff9640 1 -- 172.21.15.98:0/666324311 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f786c042070 con 0x7f787810e500 2024-05-09T08:54:33.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.832+0000 7f7874ff9640 1 --2- 172.21.15.98:0/666324311 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f78600a2f00 0x7f78600a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.832+0000 7f7874ff9640 1 -- 172.21.15.98:0/666324311 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f786c0efff0 con 0x7f787810e500 2024-05-09T08:54:33.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.832+0000 7f7876ffd640 1 --2- 172.21.15.98:0/666324311 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f78600a2f00 0x7f78600a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:33.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.832+0000 7f7880797740 1 Processor -- start 2024-05-09T08:54:33.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.833+0000 7f7880797740 1 -- start start 2024-05-09T08:54:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.833+0000 7f7880797740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39600 0x561352f399e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.833+0000 7f7876ffd640 1 --2- 172.21.15.98:0/666324311 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f78600a2f00 0x7f78600a53c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7878072860 tx=0x7f7868006bd0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.833+0000 7f7880797740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39f20 0x561352f442a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.834+0000 7f78777fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39600 0x561352f399e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.834+0000 7f78777fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39600 0x561352f399e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:34182/0 (socket says 172.21.15.98:34182) 2024-05-09T08:54:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.834+0000 7f78777fe640 1 -- 172.21.15.98:0/2969213873 learned_addr learned my addr 172.21.15.98:0/2969213873 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.834+0000 7f7876ffd640 1 --2- 172.21.15.98:0/2969213873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39f20 0x561352f442a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.834+0000 7f7880797740 1 --2- 172.21.15.98:0/2969213873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 0x561352f46bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.834+0000 7f7880797740 1 -- 172.21.15.98:0/2969213873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561352e9e340 con 0x561352f447e0 2024-05-09T08:54:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.834+0000 7f7880797740 1 -- 172.21.15.98:0/2969213873 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x561352e46290 con 0x561352f39600 2024-05-09T08:54:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.834+0000 7f7880797740 1 -- 172.21.15.98:0/2969213873 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561352e9a300 con 0x561352f39f20 2024-05-09T08:54:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.835+0000 7f78777fe640 1 -- 172.21.15.98:0/2969213873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39f20 msgr2=0x561352f442a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.835+0000 7f78777fe640 1 --2- 172.21.15.98:0/2969213873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39f20 0x561352f442a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.835+0000 7f78777fe640 1 -- 172.21.15.98:0/2969213873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 msgr2=0x561352f46bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.835+0000 7f78777fe640 1 --2- 172.21.15.98:0/2969213873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 0x561352f46bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.835+0000 7f78777fe640 1 -- 172.21.15.98:0/2969213873 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561352e6a8e0 con 0x561352f39600 2024-05-09T08:54:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.836+0000 7f78777fe640 1 --2- 172.21.15.98:0/2969213873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39600 0x561352f399e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f786c0b8550 tx=0x7f786c03b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27861073872c87ba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.836+0000 7f785dffb640 1 -- 172.21.15.98:0/2969213873 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c0c4d60 con 0x561352f39600 2024-05-09T08:54:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.837+0000 7f785dffb640 1 -- 172.21.15.98:0/2969213873 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f786c0ba3f0 con 0x561352f39600 2024-05-09T08:54:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.837+0000 7f785dffb640 1 -- 172.21.15.98:0/2969213873 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c03b4f0 con 0x561352f39600 2024-05-09T08:54:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.838+0000 7f7880797740 1 -- 172.21.15.98:0/2969213873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39600 msgr2=0x561352f399e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.838+0000 7f7880797740 1 --2- 172.21.15.98:0/2969213873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39600 0x561352f399e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f786c0b8550 tx=0x7f786c03b890 comp rx=0 tx=0).stop 2024-05-09T08:54:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.838+0000 7f7880797740 1 -- 172.21.15.98:0/2969213873 shutdown_connections 2024-05-09T08:54:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.838+0000 7f7880797740 1 --2- 172.21.15.98:0/2969213873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 0x561352f46bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.838+0000 7f7880797740 1 --2- 172.21.15.98:0/2969213873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39f20 0x561352f442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.838+0000 7f7880797740 1 --2- 172.21.15.98:0/2969213873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39600 0x561352f399e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.838+0000 7f7880797740 1 -- 172.21.15.98:0/2969213873 >> 172.21.15.98:0/2969213873 conn(0x561352f35b10 msgr2=0x561352f35ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:33.839 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.840+0000 7f7880797740 1 -- 172.21.15.98:0/2969213873 shutdown_connections 2024-05-09T08:54:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.840+0000 7f7880797740 1 -- 172.21.15.98:0/2969213873 wait complete. 2024-05-09T08:54:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.841+0000 7f7880797740 1 -- start start 2024-05-09T08:54:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.841+0000 7f7880797740 1 Processor -- start 2024-05-09T08:54:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.842+0000 7f7880797740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39600 0x561352f418c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.842+0000 7f7880797740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39f20 0x561352f41e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.842+0000 7f7880797740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 0x561352f42340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.842+0000 7f7880797740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561352f471d0 con 0x561352f447e0 2024-05-09T08:54:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.842+0000 7f7880797740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x561352e59790 con 0x561352f39f20 2024-05-09T08:54:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.842+0000 7f7880797740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561352f47350 con 0x561352f39600 2024-05-09T08:54:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.842+0000 7f7880797740 10 client.0 _ll_register_callbacks cb 0x561352e94a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.842+0000 7f78777fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39600 0x561352f418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.842+0000 7f78777fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39600 0x561352f418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:59270/0 (socket says 172.21.15.98:59270) 2024-05-09T08:54:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.842+0000 7f78777fe640 1 -- 172.21.15.98:0/3522937350 learned_addr learned my addr 172.21.15.98:0/3522937350 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.843+0000 7f7877fff640 1 --2- 172.21.15.98:0/3522937350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 0x561352f42340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.843+0000 7f7877fff640 1 -- 172.21.15.98:0/3522937350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39600 msgr2=0x561352f418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.843+0000 7f7877fff640 1 --2- 172.21.15.98:0/3522937350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39600 0x561352f418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.843+0000 7f7877fff640 1 -- 172.21.15.98:0/3522937350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39f20 msgr2=0x561352f41e00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:54:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.843+0000 7f7877fff640 1 --2- 172.21.15.98:0/3522937350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39f20 0x561352f41e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.843+0000 7f7877fff640 1 -- 172.21.15.98:0/3522937350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561352bb2210 con 0x561352f447e0 2024-05-09T08:54:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.844+0000 7f7877fff640 1 --2- 172.21.15.98:0/3522937350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 0x561352f42340 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x561352f36f30 tx=0x7f787000d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341d33180196c5b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.844+0000 7f7856ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:54:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.845+0000 7f7856ffd640 1 -- 172.21.15.98:0/3522937350 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7870016070 con 0x561352f447e0 2024-05-09T08:54:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.845+0000 7f7856ffd640 1 -- 172.21.15.98:0/3522937350 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f7870003b70 con 0x561352f447e0 2024-05-09T08:54:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.845+0000 7f7880797740 1 -- 172.21.15.98:0/3522937350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f786c0ba3f0 con 0x561352f447e0 2024-05-09T08:54:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.845+0000 7f7856ffd640 1 -- 172.21.15.98:0/3522937350 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f787001f050 con 0x561352f447e0 2024-05-09T08:54:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.849+0000 7f7880797740 10 client.16788 fetch_fsmap learned FSMap version 35 2024-05-09T08:54:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.849+0000 7f7880797740 1 -- 172.21.15.98:0/3522937350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561352c7e620 con 0x561352f447e0 2024-05-09T08:54:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.853+0000 7f7856ffd640 1 -- 172.21.15.98:0/3522937350 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f786c0ba3f0 con 0x561352f447e0 2024-05-09T08:54:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.853+0000 7f7856ffd640 1 -- 172.21.15.98:0/3522937350 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f787001a5d0 con 0x561352f447e0 2024-05-09T08:54:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.854+0000 7f7880797740 10 client.16788 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:54:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.854+0000 7f7880797740 10 client.16788 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.854+0000 7f7880797740 20 client.16788 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.854+0000 7f7880797740 1 --2- 172.21.15.98:0/3522937350 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561352f54420 0x561352f56870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.854+0000 7f7880797740 4 client.16788 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.854+0000 7f7880797740 1 -- 172.21.15.98:0/3522937350 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561352eaba70 con 0x561352f54420 2024-05-09T08:54:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.854+0000 7f78777fe640 1 --2- 172.21.15.98:0/3522937350 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561352f54420 0x561352f56870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.858+0000 7f7856ffd640 10 client.16788 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.858+0000 7f78777fe640 1 --2- 172.21.15.98:0/3522937350 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561352f54420 0x561352f56870 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4d6fc1da92aaf4d server_cookie=86d92daa9d27c8a6 in_seq=0 out_seq=0 2024-05-09T08:54:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.861+0000 7f7856ffd640 1 -- 172.21.15.98:0/3522937350 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561352eaba70 con 0x561352f54420 2024-05-09T08:54:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.861+0000 7f7856ffd640 10 client.16788 handle_command_reply: tid=0 2024-05-09T08:54:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.861+0000 7f7880797740 1 client.16788 shutdown 2024-05-09T08:54:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.862+0000 7f7880797740 1 -- 172.21.15.98:0/3522937350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 msgr2=0x561352f42340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.862+0000 7f7880797740 1 --2- 172.21.15.98:0/3522937350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 0x561352f42340 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x561352f36f30 tx=0x7f787000d6b0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.864+0000 7f7880797740 1 -- 172.21.15.98:0/3522937350 shutdown_connections 2024-05-09T08:54:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.864+0000 7f7880797740 1 --2- 172.21.15.98:0/3522937350 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561352f54420 0x561352f56870 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.864+0000 7f7880797740 1 --2- 172.21.15.98:0/3522937350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 0x561352f42340 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.864+0000 7f7880797740 1 --2- 172.21.15.98:0/3522937350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39f20 0x561352f41e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.864+0000 7f7880797740 1 --2- 172.21.15.98:0/3522937350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39600 0x561352f418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.864+0000 7f7856ffd640 0 client.16788 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.864+0000 7f7880797740 1 -- 172.21.15.98:0/3522937350 >> 172.21.15.98:0/3522937350 conn(0x561352f35b10 msgr2=0x561352f3b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.865+0000 7f7880797740 1 -- 172.21.15.98:0/3522937350 shutdown_connections 2024-05-09T08:54:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.865+0000 7f7880797740 1 -- 172.21.15.98:0/3522937350 wait complete. 2024-05-09T08:54:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.865+0000 7f7880797740 20 client.16788 trim_cache size 0 max 16384 2024-05-09T08:54:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.919+0000 7f7880797740 1 Processor -- start 2024-05-09T08:54:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.919+0000 7f7880797740 1 -- start start 2024-05-09T08:54:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.919+0000 7f7880797740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39600 0x561352f43ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.919+0000 7f7880797740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39f20 0x561352f50000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.920+0000 7f7880797740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 0x561352f35ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.920+0000 7f7880797740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78781bdbc0 con 0x561352f447e0 2024-05-09T08:54:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.920+0000 7f7880797740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f78781bf700 con 0x561352f39600 2024-05-09T08:54:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.920+0000 7f7880797740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f78781be290 con 0x561352f39f20 2024-05-09T08:54:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.924+0000 7f78777fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39600 0x561352f43ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.924+0000 7f78777fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39600 0x561352f43ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:34202/0 (socket says 172.21.15.98:34202) 2024-05-09T08:54:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.924+0000 7f78777fe640 1 -- 172.21.15.98:0/305890671 learned_addr learned my addr 172.21.15.98:0/305890671 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.924+0000 7f7876ffd640 1 --2- 172.21.15.98:0/305890671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39f20 0x561352f50000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.924+0000 7f7877fff640 1 --2- 172.21.15.98:0/305890671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 0x561352f35ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:33.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.925+0000 7f7877fff640 1 -- 172.21.15.98:0/305890671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39f20 msgr2=0x561352f50000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:33.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.925+0000 7f7877fff640 1 --2- 172.21.15.98:0/305890671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39f20 0x561352f50000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.925+0000 7f7877fff640 1 -- 172.21.15.98:0/305890671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39600 msgr2=0x561352f43ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:33.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.925+0000 7f7877fff640 1 --2- 172.21.15.98:0/305890671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39600 0x561352f43ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.925+0000 7f7877fff640 1 -- 172.21.15.98:0/305890671 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561352f40870 con 0x561352f447e0 2024-05-09T08:54:33.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.926+0000 7f7877fff640 1 --2- 172.21.15.98:0/305890671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 0x561352f35ef0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x561352f36f30 tx=0x7f7870009440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a8272702a456556 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.934+0000 7f785d7fa640 1 -- 172.21.15.98:0/305890671 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7870016040 con 0x561352f447e0 2024-05-09T08:54:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.935+0000 7f785d7fa640 1 -- 172.21.15.98:0/305890671 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7870003970 con 0x561352f447e0 2024-05-09T08:54:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.935+0000 7f785d7fa640 1 -- 172.21.15.98:0/305890671 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7870010a70 con 0x561352f447e0 2024-05-09T08:54:33.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.936+0000 7f7880797740 1 -- 172.21.15.98:0/305890671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 msgr2=0x561352f35ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:33.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.936+0000 7f7880797740 1 --2- 172.21.15.98:0/305890671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 0x561352f35ef0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x561352f36f30 tx=0x7f7870009440 comp rx=0 tx=0).stop 2024-05-09T08:54:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.938+0000 7f7880797740 1 -- 172.21.15.98:0/305890671 shutdown_connections 2024-05-09T08:54:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.938+0000 7f7880797740 1 --2- 172.21.15.98:0/305890671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f447e0 0x561352f35ef0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.938+0000 7f7880797740 1 --2- 172.21.15.98:0/305890671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39f20 0x561352f50000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.938+0000 7f7880797740 1 --2- 172.21.15.98:0/305890671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f39600 0x561352f43ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.939+0000 7f7880797740 1 -- 172.21.15.98:0/305890671 >> 172.21.15.98:0/305890671 conn(0x561352f35b10 msgr2=0x561352f34bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.948+0000 7f7880797740 1 -- 172.21.15.98:0/305890671 shutdown_connections 2024-05-09T08:54:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.948+0000 7f7880797740 1 -- 172.21.15.98:0/305890671 wait complete. 2024-05-09T08:54:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.949+0000 7f7880797740 1 -- start start 2024-05-09T08:54:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.950+0000 7f7880797740 1 Processor -- start 2024-05-09T08:54:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.950+0000 7f7880797740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39600 0x561352f48330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.950+0000 7f7880797740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f39f20 0x561352f48870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.950+0000 7f7880797740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f447e0 0x561352fb99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.950+0000 7f7880797740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78781bf030 con 0x561352f39f20 2024-05-09T08:54:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.950+0000 7f7880797740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f78781be960 con 0x561352f447e0 2024-05-09T08:54:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.950+0000 7f7880797740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f78781bbba0 con 0x561352f39600 2024-05-09T08:54:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.950+0000 7f7880797740 10 client.0 _ll_register_callbacks cb 0x7f7870009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:33.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.951+0000 7f7876ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f39f20 0x561352f48870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:33.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.951+0000 7f7876ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f39f20 0x561352f48870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50950/0 (socket says 172.21.15.98:50950) 2024-05-09T08:54:33.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.951+0000 7f7876ffd640 1 -- 172.21.15.98:0/272080619 learned_addr learned my addr 172.21.15.98:0/272080619 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:33.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.952+0000 7f7877fff640 1 --2- 172.21.15.98:0/272080619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f447e0 0x561352fb99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:33.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.952+0000 7f78777fe640 1 --2- 172.21.15.98:0/272080619 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39600 0x561352f48330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.952+0000 7f7876ffd640 1 -- 172.21.15.98:0/272080619 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39600 msgr2=0x561352f48330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.953+0000 7f7876ffd640 1 --2- 172.21.15.98:0/272080619 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39600 0x561352f48330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.953+0000 7f7876ffd640 1 -- 172.21.15.98:0/272080619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f447e0 msgr2=0x561352fb99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.953+0000 7f7876ffd640 1 --2- 172.21.15.98:0/272080619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f447e0 0x561352fb99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:33.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.953+0000 7f7876ffd640 1 -- 172.21.15.98:0/272080619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561352f46f40 con 0x561352f39f20 2024-05-09T08:54:33.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.954+0000 7f7876ffd640 1 --2- 172.21.15.98:0/272080619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f39f20 0x561352f48870 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f7868012390 tx=0x7f78680046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fdb1d7bf56dae3b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:33.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.954+0000 7f7856ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:54:33.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.955+0000 7f7856ffd640 1 -- 172.21.15.98:0/272080619 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7868024030 con 0x561352f39f20 2024-05-09T08:54:33.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.955+0000 7f7880797740 1 -- 172.21.15.98:0/272080619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7870003970 con 0x561352f39f20 2024-05-09T08:54:33.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.955+0000 7f7856ffd640 1 -- 172.21.15.98:0/272080619 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f7868020640 con 0x561352f39f20 2024-05-09T08:54:33.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.956+0000 7f7856ffd640 1 -- 172.21.15.98:0/272080619 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7868031050 con 0x561352f39f20 2024-05-09T08:54:33.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.963+0000 7f7880797740 10 client.16794 fetch_fsmap learned FSMap version 35 2024-05-09T08:54:33.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.963+0000 7f7880797740 1 -- 172.21.15.98:0/272080619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561352fcbfb0 con 0x561352f39f20 2024-05-09T08:54:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.967+0000 7f7856ffd640 1 -- 172.21.15.98:0/272080619 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7870003970 con 0x561352f39f20 2024-05-09T08:54:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.967+0000 7f7856ffd640 1 -- 172.21.15.98:0/272080619 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f7868012070 con 0x561352f39f20 2024-05-09T08:54:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.968+0000 7f7880797740 10 client.16794 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:54:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.968+0000 7f7880797740 10 client.16794 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.968+0000 7f7880797740 20 client.16794 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.968+0000 7f7880797740 1 --2- 172.21.15.98:0/272080619 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561352fcc790 0x561352fcebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.968+0000 7f7880797740 4 client.16794 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:54:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.968+0000 7f7880797740 1 -- 172.21.15.98:0/272080619 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561352ea5dc0 con 0x561352fcc790 2024-05-09T08:54:33.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.970+0000 7f78777fe640 1 --2- 172.21.15.98:0/272080619 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561352fcc790 0x561352fcebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:33.978 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.978+0000 7f78777fe640 1 --2- 172.21.15.98:0/272080619 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561352fcc790 0x561352fcebb0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=615441a8bff5e049 server_cookie=60f03586f0d88f3 in_seq=0 out_seq=0 2024-05-09T08:54:33.978 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:33.978+0000 7f7856ffd640 10 client.16794 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:34.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:33 smithi105 ceph-mon[34498]: pgmap v122: 129 pgs: 129 active+clean; 29 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 763 MiB/s wr, 1.20k op/s 2024-05-09T08:54:34.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:33 smithi098 ceph-mon[30092]: pgmap v122: 129 pgs: 129 active+clean; 29 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 763 MiB/s wr, 1.20k op/s 2024-05-09T08:54:34.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:33 smithi142 ceph-mon[34625]: pgmap v122: 129 pgs: 129 active+clean; 29 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 763 MiB/s wr, 1.20k op/s 2024-05-09T08:54:36.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:35 smithi105 ceph-mon[34498]: pgmap v123: 129 pgs: 129 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 782 MiB/s wr, 1.01k op/s 2024-05-09T08:54:36.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:35 smithi098 ceph-mon[30092]: pgmap v123: 129 pgs: 129 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 782 MiB/s wr, 1.01k op/s 2024-05-09T08:54:36.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:35 smithi142 ceph-mon[34625]: pgmap v123: 129 pgs: 129 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 782 MiB/s wr, 1.01k op/s 2024-05-09T08:54:36.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.843+0000 7f7856ffd640 1 -- 172.21.15.98:0/272080619 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561352ea5dc0 con 0x561352fcc790 2024-05-09T08:54:36.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.843+0000 7f7856ffd640 10 client.16794 handle_command_reply: tid=0 2024-05-09T08:54:36.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.843+0000 7f7880797740 1 client.16794 shutdown 2024-05-09T08:54:36.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.844+0000 7f7880797740 1 -- 172.21.15.98:0/272080619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f39f20 msgr2=0x561352f48870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:36.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.844+0000 7f7880797740 1 --2- 172.21.15.98:0/272080619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f39f20 0x561352f48870 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f7868012390 tx=0x7f78680046e0 comp rx=0 tx=0).stop 2024-05-09T08:54:36.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.844+0000 7f7880797740 1 -- 172.21.15.98:0/272080619 shutdown_connections 2024-05-09T08:54:36.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.844+0000 7f7880797740 1 --2- 172.21.15.98:0/272080619 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561352fcc790 0x561352fcebb0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:36.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.845+0000 7f7880797740 1 --2- 172.21.15.98:0/272080619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561352f447e0 0x561352fb99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:36.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.845+0000 7f7880797740 1 --2- 172.21.15.98:0/272080619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561352f39f20 0x561352f48870 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:36.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.845+0000 7f7880797740 1 --2- 172.21.15.98:0/272080619 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561352f39600 0x561352f48330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:36.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.845+0000 7f7880797740 1 -- 172.21.15.98:0/272080619 >> 172.21.15.98:0/272080619 conn(0x561352f35b10 msgr2=0x561352f34bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:36.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.845+0000 7f7856ffd640 0 client.16794 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:36.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.845+0000 7f7880797740 1 -- 172.21.15.98:0/272080619 shutdown_connections 2024-05-09T08:54:36.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.845+0000 7f7880797740 1 -- 172.21.15.98:0/272080619 wait complete. 2024-05-09T08:54:36.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.845+0000 7f7880797740 20 client.16794 trim_cache size 0 max 16384 2024-05-09T08:54:36.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:54:36.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:54:36.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:54:36.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:54:36.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.846+0000 7f787c82a640 1 -- 172.21.15.98:0/666324311 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f78600a2f00 msgr2=0x7f78600a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:36.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.846+0000 7f787c82a640 1 --2- 172.21.15.98:0/666324311 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f78600a2f00 0x7f78600a53c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7878072860 tx=0x7f7868006bd0 comp rx=0 tx=0).stop 2024-05-09T08:54:36.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.846+0000 7f787c82a640 1 -- 172.21.15.98:0/666324311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787810e500 msgr2=0x7f78781a6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:36.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.847+0000 7f787c82a640 1 --2- 172.21.15.98:0/666324311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787810e500 0x7f78781a6ed0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f786c02aa40 tx=0x7f786c003b90 comp rx=0 tx=0).stop 2024-05-09T08:54:36.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.847+0000 7f787c82a640 1 -- 172.21.15.98:0/666324311 shutdown_connections 2024-05-09T08:54:36.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.847+0000 7f787c82a640 1 --2- 172.21.15.98:0/666324311 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f78600a2f00 0x7f78600a53c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:36.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.847+0000 7f787c82a640 1 --2- 172.21.15.98:0/666324311 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78781102d0 0x7f78781ab790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:36.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.847+0000 7f787c82a640 1 --2- 172.21.15.98:0/666324311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f787810fef0 0x7f78781a7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:36.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.847+0000 7f787c82a640 1 --2- 172.21.15.98:0/666324311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787810e500 0x7f78781a6ed0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:36.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.847+0000 7f787c82a640 1 -- 172.21.15.98:0/666324311 >> 172.21.15.98:0/666324311 conn(0x7f787806c770 msgr2=0x7f7878070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.849+0000 7f787c82a640 1 -- 172.21.15.98:0/666324311 shutdown_connections 2024-05-09T08:54:36.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:36.849+0000 7f787c82a640 1 -- 172.21.15.98:0/666324311 wait complete. 2024-05-09T08:54:38.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:37 smithi105 ceph-mon[34498]: pgmap v124: 129 pgs: 129 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 745 MiB/s wr, 905 op/s 2024-05-09T08:54:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:37 smithi098 ceph-mon[30092]: pgmap v124: 129 pgs: 129 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 745 MiB/s wr, 905 op/s 2024-05-09T08:54:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:37 smithi142 ceph-mon[34625]: pgmap v124: 129 pgs: 129 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 745 MiB/s wr, 905 op/s 2024-05-09T08:54:40.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:39 smithi105 ceph-mon[34498]: pgmap v125: 129 pgs: 129 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 713 MiB/s wr, 868 op/s 2024-05-09T08:54:40.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:39 smithi098 ceph-mon[30092]: pgmap v125: 129 pgs: 129 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 713 MiB/s wr, 868 op/s 2024-05-09T08:54:40.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:39 smithi142 ceph-mon[34625]: pgmap v125: 129 pgs: 129 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 713 MiB/s wr, 868 op/s 2024-05-09T08:54:42.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:41 smithi105 ceph-mon[34498]: pgmap v126: 129 pgs: 129 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 947 MiB/s wr, 1.25k op/s 2024-05-09T08:54:42.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:41 smithi098 ceph-mon[30092]: pgmap v126: 129 pgs: 129 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 947 MiB/s wr, 1.25k op/s 2024-05-09T08:54:42.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:41 smithi142 ceph-mon[34625]: pgmap v126: 129 pgs: 129 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 947 MiB/s wr, 1.25k op/s 2024-05-09T08:54:44.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:43 smithi105 ceph-mon[34498]: pgmap v127: 129 pgs: 129 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 715 MiB/s wr, 1.02k op/s 2024-05-09T08:54:44.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:43 smithi098 ceph-mon[30092]: pgmap v127: 129 pgs: 129 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 715 MiB/s wr, 1.02k op/s 2024-05-09T08:54:44.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:43 smithi142 ceph-mon[34625]: pgmap v127: 129 pgs: 129 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 715 MiB/s wr, 1.02k op/s 2024-05-09T08:54:46.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:45 smithi105 ceph-mon[34498]: pgmap v128: 129 pgs: 129 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 806 MiB/s wr, 1.17k op/s 2024-05-09T08:54:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:45 smithi098 ceph-mon[30092]: pgmap v128: 129 pgs: 129 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 806 MiB/s wr, 1.17k op/s 2024-05-09T08:54:46.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:45 smithi142 ceph-mon[34625]: pgmap v128: 129 pgs: 129 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 806 MiB/s wr, 1.17k op/s 2024-05-09T08:54:48.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:47 smithi098 ceph-mon[30092]: pgmap v129: 129 pgs: 129 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 904 MiB/s wr, 1.35k op/s 2024-05-09T08:54:48.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:47 smithi142 ceph-mon[34625]: pgmap v129: 129 pgs: 129 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 904 MiB/s wr, 1.35k op/s 2024-05-09T08:54:48.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:47 smithi105 ceph-mon[34498]: pgmap v129: 129 pgs: 129 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 904 MiB/s wr, 1.35k op/s 2024-05-09T08:54:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.001+0000 7f8dfd35a640 1 -- 172.21.15.98:0/2716963165 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8df8100790 msgr2=0x7f8df8100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.001+0000 7f8dfd35a640 1 --2- 172.21.15.98:0/2716963165 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8df8100790 0x7f8df8100c10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009860 tx=0x7f8de002a340 comp rx=0 tx=0).stop 2024-05-09T08:54:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.001+0000 7f8dfd35a640 1 -- 172.21.15.98:0/2716963165 shutdown_connections 2024-05-09T08:54:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.001+0000 7f8dfd35a640 1 --2- 172.21.15.98:0/2716963165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8df8101150 0x7f8df810d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.001+0000 7f8dfd35a640 1 --2- 172.21.15.98:0/2716963165 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8df8100790 0x7f8df8100c10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.001+0000 7f8dfd35a640 1 --2- 172.21.15.98:0/2716963165 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df8106750 0x7f8df8106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.001+0000 7f8dfd35a640 1 -- 172.21.15.98:0/2716963165 >> 172.21.15.98:0/2716963165 conn(0x7f8df8077880 msgr2=0x7f8df8077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.001+0000 7f8dfd35a640 1 -- 172.21.15.98:0/2716963165 shutdown_connections 2024-05-09T08:54:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.001+0000 7f8dfd35a640 1 -- 172.21.15.98:0/2716963165 wait complete. 2024-05-09T08:54:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.002+0000 7f8dfd35a640 1 Processor -- start 2024-05-09T08:54:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.002+0000 7f8dfd35a640 1 -- start start 2024-05-09T08:54:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.002+0000 7f8dfd35a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df8100790 0x7f8df8195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.002+0000 7f8dfd35a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8df8101150 0x7f8df8196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.002+0000 7f8dfd35a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8df8106750 0x7f8df819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.002+0000 7f8dfd35a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8df810fa50 con 0x7f8df8101150 2024-05-09T08:54:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.002+0000 7f8dfd35a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8df810f8d0 con 0x7f8df8106750 2024-05-09T08:54:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.002+0000 7f8dfd35a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8df810fbd0 con 0x7f8df8100790 2024-05-09T08:54:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.002+0000 7f8deffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8df8101150 0x7f8df8196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.002+0000 7f8deffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8df8101150 0x7f8df8196350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36338/0 (socket says 172.21.15.98:36338) 2024-05-09T08:54:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8deffff640 1 -- 172.21.15.98:0/318322140 learned_addr learned my addr 172.21.15.98:0/318322140 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8df77fe640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8df8106750 0x7f8df819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8df6ffd640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df8100790 0x7f8df8195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8deffff640 1 -- 172.21.15.98:0/318322140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df8100790 msgr2=0x7f8df8195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8deffff640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df8100790 0x7f8df8195e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8deffff640 1 -- 172.21.15.98:0/318322140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8df8106750 msgr2=0x7f8df819a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8deffff640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8df8106750 0x7f8df819a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8deffff640 1 -- 172.21.15.98:0/318322140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df819ae00 con 0x7f8df8101150 2024-05-09T08:54:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8df77fe640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8df8106750 0x7f8df819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:54:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8deffff640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8df8101150 0x7f8df8196350 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f8de0035040 tx=0x7f8de0033b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8df4ff9640 1 -- 172.21.15.98:0/318322140 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0037040 con 0x7f8df8101150 2024-05-09T08:54:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8df4ff9640 1 -- 172.21.15.98:0/318322140 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de0040070 con 0x7f8df8101150 2024-05-09T08:54:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8df4ff9640 1 -- 172.21.15.98:0/318322140 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0003a10 con 0x7f8df8101150 2024-05-09T08:54:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8dfd35a640 1 -- 172.21.15.98:0/318322140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df819b090 con 0x7f8df8101150 2024-05-09T08:54:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.003+0000 7f8dfd35a640 1 -- 172.21.15.98:0/318322140 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df81a2980 con 0x7f8df8101150 2024-05-09T08:54:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.007+0000 7f8df4ff9640 1 -- 172.21.15.98:0/318322140 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8de0004150 con 0x7f8df8101150 2024-05-09T08:54:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.008+0000 7f8e00066740 1 Processor -- start 2024-05-09T08:54:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.008+0000 7f8e00066740 1 -- start start 2024-05-09T08:54:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.008+0000 7f8e00066740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc1ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.008+0000 7f8e00066740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc25500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.008+0000 7f8e00066740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc27e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.008+0000 7f8e00066740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cb0fb7b340 con 0x55cb0fc25a40 2024-05-09T08:54:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.008+0000 7f8e00066740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cb0fb23290 con 0x55cb0fc1a860 2024-05-09T08:54:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.008+0000 7f8e00066740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cb0fb77300 con 0x55cb0fc1b180 2024-05-09T08:54:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.008+0000 7f8df77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc1ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.008+0000 7f8df4ff9640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f8de40a2da0 0x7f8de40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.008+0000 7f8df77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc1ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:44674/0 (socket says 172.21.15.98:44674) 2024-05-09T08:54:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.008+0000 7f8df77fe640 1 -- 172.21.15.98:0/622636873 learned_addr learned my addr 172.21.15.98:0/622636873 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.008+0000 7f8df6ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc25500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.009+0000 7f8df77fe640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f8de40a2da0 0x7f8de40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.009+0000 7f8df6ffd640 1 -- 172.21.15.98:0/622636873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 msgr2=0x55cb0fc1ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.009+0000 7f8df6ffd640 1 --2- 172.21.15.98:0/622636873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc1ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.009+0000 7f8df6ffd640 1 -- 172.21.15.98:0/622636873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 msgr2=0x55cb0fc27e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.009+0000 7f8df4ff9640 1 -- 172.21.15.98:0/318322140 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f8de00ef450 con 0x7f8df8101150 2024-05-09T08:54:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.009+0000 7f8deffff640 1 --2- 172.21.15.98:0/622636873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc27e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.009+0000 7f8df6ffd640 1 --2- 172.21.15.98:0/622636873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc27e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.009+0000 7f8df6ffd640 1 -- 172.21.15.98:0/622636873 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb0fb478e0 con 0x55cb0fc1b180 2024-05-09T08:54:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.009+0000 7f8df77fe640 1 --2- 172.21.15.98:0/622636873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc1ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:54:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.009+0000 7f8deffff640 1 --2- 172.21.15.98:0/622636873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc27e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:54:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.010+0000 7f8df77fe640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f8de40a2da0 0x7f8de40a5260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8df0005c40 tx=0x7f8df0010040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.013+0000 7f8df6ffd640 1 --2- 172.21.15.98:0/622636873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc25500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009c00 tx=0x7f8de8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b3fd73acb992eb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.013+0000 7f8ded7fa640 1 -- 172.21.15.98:0/622636873 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8004050 con 0x55cb0fc1b180 2024-05-09T08:54:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.013+0000 7f8ded7fa640 1 -- 172.21.15.98:0/622636873 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de8005500 con 0x55cb0fc1b180 2024-05-09T08:54:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.013+0000 7f8ded7fa640 1 -- 172.21.15.98:0/622636873 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8004e00 con 0x55cb0fc1b180 2024-05-09T08:54:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.015+0000 7f8e00066740 1 -- 172.21.15.98:0/622636873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 msgr2=0x55cb0fc25500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.015+0000 7f8e00066740 1 --2- 172.21.15.98:0/622636873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc25500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009c00 tx=0x7f8de8011810 comp rx=0 tx=0).stop 2024-05-09T08:54:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.015+0000 7f8e00066740 1 -- 172.21.15.98:0/622636873 shutdown_connections 2024-05-09T08:54:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.015+0000 7f8e00066740 1 --2- 172.21.15.98:0/622636873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc27e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.015+0000 7f8e00066740 1 --2- 172.21.15.98:0/622636873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc25500 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.015+0000 7f8e00066740 1 --2- 172.21.15.98:0/622636873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc1ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.015+0000 7f8e00066740 1 -- 172.21.15.98:0/622636873 >> 172.21.15.98:0/622636873 conn(0x55cb0fc16d70 msgr2=0x55cb0fc17150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.016+0000 7f8e00066740 1 -- 172.21.15.98:0/622636873 shutdown_connections 2024-05-09T08:54:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.016+0000 7f8e00066740 1 -- 172.21.15.98:0/622636873 wait complete. 2024-05-09T08:54:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.016+0000 7f8e00066740 1 -- start start 2024-05-09T08:54:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.016+0000 7f8e00066740 1 Processor -- start 2024-05-09T08:54:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8e00066740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc22a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8e00066740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc22fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8e00066740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc234e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8e00066740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cb0fc28670 con 0x55cb0fc25a40 2024-05-09T08:54:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8e00066740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cb0fb36790 con 0x55cb0fc1a860 2024-05-09T08:54:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8e00066740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cb0fc287f0 con 0x55cb0fc1b180 2024-05-09T08:54:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8df6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc22a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8df6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc22a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:44688/0 (socket says 172.21.15.98:44688) 2024-05-09T08:54:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8df6ffd640 1 -- 172.21.15.98:0/3697997857 learned_addr learned my addr 172.21.15.98:0/3697997857 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8e00066740 10 client.0 _ll_register_callbacks cb 0x55cb0fb71a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8df77fe640 1 --2- 172.21.15.98:0/3697997857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc234e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8deffff640 1 --2- 172.21.15.98:0/3697997857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc22fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8df6ffd640 1 -- 172.21.15.98:0/3697997857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 msgr2=0x55cb0fc22fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8df6ffd640 1 --2- 172.21.15.98:0/3697997857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc22fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8df6ffd640 1 -- 172.21.15.98:0/3697997857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 msgr2=0x55cb0fc234e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8df6ffd640 1 --2- 172.21.15.98:0/3697997857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc234e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.017+0000 7f8df6ffd640 1 -- 172.21.15.98:0/3697997857 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb0f88f210 con 0x55cb0fc1a860 2024-05-09T08:54:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.018+0000 7f8df6ffd640 1 --2- 172.21.15.98:0/3697997857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc22a60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8de801b040 tx=0x7f8de8009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79817c88622c2e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.018+0000 7f8dde7fc640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:54:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.018+0000 7f8dde7fc640 1 -- 172.21.15.98:0/3697997857 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8029070 con 0x55cb0fc1a860 2024-05-09T08:54:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.018+0000 7f8e00066740 1 -- 172.21.15.98:0/3697997857 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8de8005500 con 0x55cb0fc1a860 2024-05-09T08:54:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.018+0000 7f8dde7fc640 1 -- 172.21.15.98:0/3697997857 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f8de8003a00 con 0x55cb0fc1a860 2024-05-09T08:54:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.018+0000 7f8dde7fc640 1 -- 172.21.15.98:0/3697997857 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8de8035050 con 0x55cb0fc1a860 2024-05-09T08:54:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.018+0000 7f8e00066740 10 client.26111 fetch_fsmap learned FSMap version 35 2024-05-09T08:54:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.018+0000 7f8e00066740 1 -- 172.21.15.98:0/3697997857 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55cb0f95b620 con 0x55cb0fc1a860 2024-05-09T08:54:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.021+0000 7f8dde7fc640 1 -- 172.21.15.98:0/3697997857 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de8003c40 con 0x55cb0fc1a860 2024-05-09T08:54:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.022+0000 7f8dde7fc640 1 -- 172.21.15.98:0/3697997857 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f8de80234d0 con 0x55cb0fc1a860 2024-05-09T08:54:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.022+0000 7f8e00066740 10 client.26111 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:54:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.022+0000 7f8e00066740 10 client.26111 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.022+0000 7f8e00066740 20 client.26111 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.022+0000 7f8e00066740 1 --2- 172.21.15.98:0/3697997857 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb0fc35440 0x55cb0fc37830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.022+0000 7f8e00066740 4 client.26111 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.022+0000 7f8e00066740 1 -- 172.21.15.98:0/3697997857 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8de8005500 con 0x55cb0fc35440 2024-05-09T08:54:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.022+0000 7f8deffff640 1 --2- 172.21.15.98:0/3697997857 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb0fc35440 0x55cb0fc37830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.024+0000 7f8deffff640 1 --2- 172.21.15.98:0/3697997857 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb0fc35440 0x55cb0fc37830 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bf3d9b07ce2eee7 server_cookie=578c2896874a1f67 in_seq=0 out_seq=0 2024-05-09T08:54:50.024 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.024+0000 7f8dde7fc640 10 client.26111 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:50.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.028+0000 7f8dde7fc640 1 -- 172.21.15.98:0/3697997857 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f8de8005500 con 0x55cb0fc35440 2024-05-09T08:54:50.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.028+0000 7f8dde7fc640 10 client.26111 handle_command_reply: tid=0 2024-05-09T08:54:50.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.028+0000 7f8e00066740 1 client.26111 shutdown 2024-05-09T08:54:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.028+0000 7f8e00066740 1 -- 172.21.15.98:0/3697997857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 msgr2=0x55cb0fc22a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.028+0000 7f8e00066740 1 --2- 172.21.15.98:0/3697997857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc22a60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8de801b040 tx=0x7f8de8009900 comp rx=0 tx=0).stop 2024-05-09T08:54:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.028+0000 7f8e00066740 1 -- 172.21.15.98:0/3697997857 shutdown_connections 2024-05-09T08:54:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.028+0000 7f8e00066740 1 --2- 172.21.15.98:0/3697997857 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb0fc35440 0x55cb0fc37830 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.028+0000 7f8e00066740 1 --2- 172.21.15.98:0/3697997857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc234e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.028+0000 7f8dde7fc640 0 client.26111 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.028+0000 7f8e00066740 1 --2- 172.21.15.98:0/3697997857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc22fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.028+0000 7f8e00066740 1 --2- 172.21.15.98:0/3697997857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc22a60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.028+0000 7f8e00066740 1 -- 172.21.15.98:0/3697997857 >> 172.21.15.98:0/3697997857 conn(0x55cb0fc16d70 msgr2=0x55cb0fc1caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.031+0000 7f8e00066740 1 -- 172.21.15.98:0/3697997857 shutdown_connections 2024-05-09T08:54:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.031+0000 7f8e00066740 1 -- 172.21.15.98:0/3697997857 wait complete. 2024-05-09T08:54:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.031+0000 7f8e00066740 20 client.26111 trim_cache size 0 max 16384 2024-05-09T08:54:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.061+0000 7f8e00066740 1 Processor -- start 2024-05-09T08:54:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.062+0000 7f8e00066740 1 -- start start 2024-05-09T08:54:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.062+0000 7f8e00066740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc25190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.062+0000 7f8e00066740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc219a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.063+0000 7f8df6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc25190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.063+0000 7f8df6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc25190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:44694/0 (socket says 172.21.15.98:44694) 2024-05-09T08:54:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.063+0000 7f8e00066740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc31530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.063+0000 7f8df6ffd640 1 -- 172.21.15.98:0/1463285658 learned_addr learned my addr 172.21.15.98:0/1463285658 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.063+0000 7f8e00066740 1 -- 172.21.15.98:0/1463285658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8df81aca40 con 0x55cb0fc25a40 2024-05-09T08:54:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.063+0000 7f8e00066740 1 -- 172.21.15.98:0/1463285658 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8df81ae580 con 0x55cb0fc1a860 2024-05-09T08:54:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.063+0000 7f8e00066740 1 -- 172.21.15.98:0/1463285658 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8df81ad110 con 0x55cb0fc1b180 2024-05-09T08:54:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.063+0000 7f8df77fe640 1 --2- 172.21.15.98:0/1463285658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc31530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.063+0000 7f8df6ffd640 1 -- 172.21.15.98:0/1463285658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 msgr2=0x55cb0fc219a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:54:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.063+0000 7f8df6ffd640 1 --2- 172.21.15.98:0/1463285658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc219a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.063+0000 7f8df6ffd640 1 -- 172.21.15.98:0/1463285658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 msgr2=0x55cb0fc31530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.063+0000 7f8df6ffd640 1 --2- 172.21.15.98:0/1463285658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc31530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.063+0000 7f8df6ffd640 1 -- 172.21.15.98:0/1463285658 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb0fc10f00 con 0x55cb0fc1a860 2024-05-09T08:54:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.065+0000 7f8df6ffd640 1 --2- 172.21.15.98:0/1463285658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc25190 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009ff0 tx=0x7f8de80059b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f736c5667ce58e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.065+0000 7f8decff9640 1 -- 172.21.15.98:0/1463285658 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8029070 con 0x55cb0fc1a860 2024-05-09T08:54:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.065+0000 7f8decff9640 1 -- 172.21.15.98:0/1463285658 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de8023cc0 con 0x55cb0fc1a860 2024-05-09T08:54:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.065+0000 7f8decff9640 1 -- 172.21.15.98:0/1463285658 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de802e440 con 0x55cb0fc1a860 2024-05-09T08:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.067+0000 7f8e00066740 1 -- 172.21.15.98:0/1463285658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 msgr2=0x55cb0fc25190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.067+0000 7f8e00066740 1 --2- 172.21.15.98:0/1463285658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc25190 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009ff0 tx=0x7f8de80059b0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.067+0000 7f8e00066740 1 -- 172.21.15.98:0/1463285658 shutdown_connections 2024-05-09T08:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.067+0000 7f8e00066740 1 --2- 172.21.15.98:0/1463285658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc31530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.067+0000 7f8e00066740 1 --2- 172.21.15.98:0/1463285658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc219a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.067+0000 7f8e00066740 1 --2- 172.21.15.98:0/1463285658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc25190 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.067+0000 7f8e00066740 1 -- 172.21.15.98:0/1463285658 >> 172.21.15.98:0/1463285658 conn(0x55cb0fc16d70 msgr2=0x55cb0fc232b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.074+0000 7f8e00066740 1 -- 172.21.15.98:0/1463285658 shutdown_connections 2024-05-09T08:54:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.074+0000 7f8e00066740 1 -- 172.21.15.98:0/1463285658 wait complete. 2024-05-09T08:54:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.074+0000 7f8e00066740 1 -- start start 2024-05-09T08:54:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.075+0000 7f8e00066740 1 Processor -- start 2024-05-09T08:54:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.075+0000 7f8e00066740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc1f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.075+0000 7f8e00066740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc1fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.075+0000 7f8e00066740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc200a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.075+0000 7f8e00066740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8df81adeb0 con 0x55cb0fc25a40 2024-05-09T08:54:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.075+0000 7f8e00066740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8df81ad7e0 con 0x55cb0fc1a860 2024-05-09T08:54:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.075+0000 7f8e00066740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8df81aaa20 con 0x55cb0fc1b180 2024-05-09T08:54:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.075+0000 7f8e00066740 10 client.0 _ll_register_callbacks cb 0x7f8df0009950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.077+0000 7f8deffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc1fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.077+0000 7f8deffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc1fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:51248/0 (socket says 172.21.15.98:51248) 2024-05-09T08:54:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.077+0000 7f8deffff640 1 -- 172.21.15.98:0/1069418286 learned_addr learned my addr 172.21.15.98:0/1069418286 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.077+0000 7f8df6ffd640 1 --2- 172.21.15.98:0/1069418286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc1f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.077+0000 7f8df77fe640 1 --2- 172.21.15.98:0/1069418286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc200a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.077+0000 7f8deffff640 1 -- 172.21.15.98:0/1069418286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 msgr2=0x55cb0fc1f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.077+0000 7f8deffff640 1 --2- 172.21.15.98:0/1069418286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc1f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.077+0000 7f8deffff640 1 -- 172.21.15.98:0/1069418286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 msgr2=0x55cb0fc200a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.077+0000 7f8deffff640 1 --2- 172.21.15.98:0/1069418286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc200a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.077+0000 7f8deffff640 1 -- 172.21.15.98:0/1069418286 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb0fcab770 con 0x55cb0fc1b180 2024-05-09T08:54:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.077+0000 7f8deffff640 1 --2- 172.21.15.98:0/1069418286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc1fb60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8de00bb3c0 tx=0x7f8de002a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=293737912b65f07d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.077+0000 7f8dde7fc640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:54:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.078+0000 7f8dde7fc640 1 -- 172.21.15.98:0/1069418286 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de00d4060 con 0x55cb0fc1b180 2024-05-09T08:54:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.078+0000 7f8dde7fc640 1 -- 172.21.15.98:0/1069418286 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f8de00b7d30 con 0x55cb0fc1b180 2024-05-09T08:54:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.078+0000 7f8e00066740 1 -- 172.21.15.98:0/1069418286 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8de8023cc0 con 0x55cb0fc1b180 2024-05-09T08:54:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.078+0000 7f8dde7fc640 1 -- 172.21.15.98:0/1069418286 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8de00bb6c0 con 0x55cb0fc1b180 2024-05-09T08:54:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.079+0000 7f8e00066740 10 client.26068 fetch_fsmap learned FSMap version 35 2024-05-09T08:54:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.079+0000 7f8e00066740 1 -- 172.21.15.98:0/1069418286 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55cb0fc938c0 con 0x55cb0fc1b180 2024-05-09T08:54:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.079+0000 7f8dde7fc640 1 -- 172.21.15.98:0/1069418286 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de8023cc0 con 0x55cb0fc1b180 2024-05-09T08:54:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.079+0000 7f8dde7fc640 1 -- 172.21.15.98:0/1069418286 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f8de003a070 con 0x55cb0fc1b180 2024-05-09T08:54:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.079+0000 7f8e00066740 10 client.26068 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:54:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.079+0000 7f8e00066740 10 client.26068 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.079+0000 7f8e00066740 20 client.26068 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.079+0000 7f8e00066740 1 --2- 172.21.15.98:0/1069418286 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb0fc35440 0x55cb0fc96080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.079+0000 7f8e00066740 4 client.26068 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:54:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.079+0000 7f8e00066740 1 -- 172.21.15.98:0/1069418286 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cb0fb88a70 con 0x55cb0fc35440 2024-05-09T08:54:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.080+0000 7f8df6ffd640 1 --2- 172.21.15.98:0/1069418286 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb0fc35440 0x55cb0fc96080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.080+0000 7f8dde7fc640 10 client.26068 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:50.081+0000 7f8df6ffd640 1 --2- 172.21.15.98:0/1069418286 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb0fc35440 0x55cb0fc96080 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=979d079fae4d7b50 server_cookie=d83f1e4b4ec2d2a1 in_seq=0 out_seq=0 2024-05-09T08:54:50.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:49 smithi098 ceph-mon[30092]: pgmap v130: 129 pgs: 129 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 773 MiB/s wr, 1.20k op/s 2024-05-09T08:54:50.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:49 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:54:50.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:49 smithi142 ceph-mon[34625]: pgmap v130: 129 pgs: 129 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 773 MiB/s wr, 1.20k op/s 2024-05-09T08:54:50.501 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:49 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:54:50.502 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:49 smithi105 ceph-mon[34498]: pgmap v130: 129 pgs: 129 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 773 MiB/s wr, 1.20k op/s 2024-05-09T08:54:50.502 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:49 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:54:52.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:52 smithi098 ceph-mon[30092]: pgmap v131: 129 pgs: 129 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1013 MiB/s wr, 1.86k op/s 2024-05-09T08:54:52.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:52 smithi142 ceph-mon[34625]: pgmap v131: 129 pgs: 129 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1013 MiB/s wr, 1.86k op/s 2024-05-09T08:54:52.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:52 smithi105 ceph-mon[34498]: pgmap v131: 129 pgs: 129 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1013 MiB/s wr, 1.86k op/s 2024-05-09T08:54:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.325+0000 7f8dde7fc640 1 -- 172.21.15.98:0/1069418286 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cb0fb88a70 con 0x55cb0fc35440 2024-05-09T08:54:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.325+0000 7f8dde7fc640 10 client.26068 handle_command_reply: tid=0 2024-05-09T08:54:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.325+0000 7f8e00066740 1 client.26068 shutdown 2024-05-09T08:54:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.326+0000 7f8e00066740 1 -- 172.21.15.98:0/1069418286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 msgr2=0x55cb0fc1fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.326+0000 7f8e00066740 1 --2- 172.21.15.98:0/1069418286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc1fb60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8de00bb3c0 tx=0x7f8de002a340 comp rx=0 tx=0).stop 2024-05-09T08:54:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.326+0000 7f8e00066740 1 -- 172.21.15.98:0/1069418286 shutdown_connections 2024-05-09T08:54:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.326+0000 7f8e00066740 1 --2- 172.21.15.98:0/1069418286 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cb0fc35440 0x55cb0fc96080 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.326+0000 7f8e00066740 1 --2- 172.21.15.98:0/1069418286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cb0fc25a40 0x55cb0fc200a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.326+0000 7f8e00066740 1 --2- 172.21.15.98:0/1069418286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cb0fc1b180 0x55cb0fc1fb60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.326+0000 7f8e00066740 1 --2- 172.21.15.98:0/1069418286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cb0fc1a860 0x55cb0fc1f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.326+0000 7f8e00066740 1 -- 172.21.15.98:0/1069418286 >> 172.21.15.98:0/1069418286 conn(0x55cb0fc16d70 msgr2=0x55cb0fc22a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.326+0000 7f8dde7fc640 0 client.26068 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:53.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:54:53.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:54:53.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:54:53.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:54:53.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.327+0000 7f8e00066740 1 -- 172.21.15.98:0/1069418286 shutdown_connections 2024-05-09T08:54:53.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.327+0000 7f8e00066740 1 -- 172.21.15.98:0/1069418286 wait complete. 2024-05-09T08:54:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.327+0000 7f8e00066740 20 client.26068 trim_cache size 0 max 16384 2024-05-09T08:54:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.331+0000 7f8dfd35a640 1 -- 172.21.15.98:0/318322140 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f8de40a2da0 msgr2=0x7f8de40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.331+0000 7f8dfd35a640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f8de40a2da0 0x7f8de40a5260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8df0005c40 tx=0x7f8df0010040 comp rx=0 tx=0).stop 2024-05-09T08:54:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.331+0000 7f8dfd35a640 1 -- 172.21.15.98:0/318322140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8df8101150 msgr2=0x7f8df8196350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.331+0000 7f8dfd35a640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8df8101150 0x7f8df8196350 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f8de0035040 tx=0x7f8de0033b60 comp rx=0 tx=0).stop 2024-05-09T08:54:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.333+0000 7f8dfd35a640 1 -- 172.21.15.98:0/318322140 shutdown_connections 2024-05-09T08:54:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.333+0000 7f8dfd35a640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f8de40a2da0 0x7f8de40a5260 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.333+0000 7f8dfd35a640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8df8106750 0x7f8df819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.333+0000 7f8dfd35a640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8df8101150 0x7f8df8196350 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.333+0000 7f8dfd35a640 1 --2- 172.21.15.98:0/318322140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df8100790 0x7f8df8195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.333+0000 7f8dfd35a640 1 -- 172.21.15.98:0/318322140 >> 172.21.15.98:0/318322140 conn(0x7f8df8077880 msgr2=0x7f8df8103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.333+0000 7f8dfd35a640 1 -- 172.21.15.98:0/318322140 shutdown_connections 2024-05-09T08:54:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:54:53.333+0000 7f8dfd35a640 1 -- 172.21.15.98:0/318322140 wait complete. 2024-05-09T08:54:54.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:54 smithi098 ceph-mon[30092]: pgmap v132: 129 pgs: 129 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 702 MiB/s wr, 1.42k op/s 2024-05-09T08:54:54.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:54 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:54:54.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:54 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:54:54.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:54 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:54:54.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:54 smithi142 ceph-mon[34625]: pgmap v132: 129 pgs: 129 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 702 MiB/s wr, 1.42k op/s 2024-05-09T08:54:54.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:54 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:54:54.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:54 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:54:54.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:54 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:54:54.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:54 smithi105 ceph-mon[34498]: pgmap v132: 129 pgs: 129 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 702 MiB/s wr, 1.42k op/s 2024-05-09T08:54:54.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:54 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:54:54.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:54 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:54:54.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:54 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:54:56.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:56 smithi098 ceph-mon[30092]: pgmap v133: 129 pgs: 129 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 744 MiB/s wr, 1.46k op/s 2024-05-09T08:54:56.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:56 smithi142 ceph-mon[34625]: pgmap v133: 129 pgs: 129 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 744 MiB/s wr, 1.46k op/s 2024-05-09T08:54:56.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:56 smithi105 ceph-mon[34498]: pgmap v133: 129 pgs: 129 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 744 MiB/s wr, 1.46k op/s 2024-05-09T08:54:57.073 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:54:57.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:54:57.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.408+0000 7efdff110640 1 -- 172.21.15.98:0/2435053241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdf81090c0 msgr2=0x7efdf8109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.408+0000 7efdff110640 1 --2- 172.21.15.98:0/2435053241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdf81090c0 0x7efdf8109540 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efdec009930 tx=0x7efdec02a440 comp rx=0 tx=0).stop 2024-05-09T08:54:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.408+0000 7efdff110640 1 -- 172.21.15.98:0/2435053241 shutdown_connections 2024-05-09T08:54:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.408+0000 7efdff110640 1 --2- 172.21.15.98:0/2435053241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf8109a80 0x7efdf8115530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.408+0000 7efdff110640 1 --2- 172.21.15.98:0/2435053241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdf81090c0 0x7efdf8109540 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.408+0000 7efdff110640 1 --2- 172.21.15.98:0/2435053241 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf810d6a0 0x7efdf810da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.408+0000 7efdff110640 1 -- 172.21.15.98:0/2435053241 >> 172.21.15.98:0/2435053241 conn(0x7efdf806c940 msgr2=0x7efdf806cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.409+0000 7efdff110640 1 -- 172.21.15.98:0/2435053241 shutdown_connections 2024-05-09T08:54:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.409+0000 7efdff110640 1 -- 172.21.15.98:0/2435053241 wait complete. 2024-05-09T08:54:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.409+0000 7efdff110640 1 Processor -- start 2024-05-09T08:54:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.409+0000 7efdff110640 1 -- start start 2024-05-09T08:54:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.410+0000 7efdff110640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf81090c0 0x7efdf81a66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.410+0000 7efdff110640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdf8109a80 0x7efdf81a6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.410+0000 7efdff110640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf810d6a0 0x7efdf81a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.410+0000 7efdff110640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdf8118b70 con 0x7efdf810d6a0 2024-05-09T08:54:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.410+0000 7efdff110640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efdf81189f0 con 0x7efdf81090c0 2024-05-09T08:54:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.410+0000 7efdff110640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efdf8118cf0 con 0x7efdf8109a80 2024-05-09T08:54:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.410+0000 7efdfd6ad640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf810d6a0 0x7efdf81a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.410+0000 7efdfd6ad640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf810d6a0 0x7efdf81a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37158/0 (socket says 172.21.15.98:37158) 2024-05-09T08:54:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.410+0000 7efdfd6ad640 1 -- 172.21.15.98:0/941945696 learned_addr learned my addr 172.21.15.98:0/941945696 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.410+0000 7efdf7fff640 1 --2- 172.21.15.98:0/941945696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdf8109a80 0x7efdf81a6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.410+0000 7efdfceac640 1 --2- 172.21.15.98:0/941945696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf81090c0 0x7efdf81a66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.411+0000 7efdfd6ad640 1 -- 172.21.15.98:0/941945696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdf8109a80 msgr2=0x7efdf81a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.411+0000 7efdfd6ad640 1 --2- 172.21.15.98:0/941945696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdf8109a80 0x7efdf81a6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.411+0000 7efdfd6ad640 1 -- 172.21.15.98:0/941945696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf81090c0 msgr2=0x7efdf81a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.411+0000 7efdfd6ad640 1 --2- 172.21.15.98:0/941945696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf81090c0 0x7efdf81a66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.411+0000 7efdfd6ad640 1 -- 172.21.15.98:0/941945696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf81a1010 con 0x7efdf810d6a0 2024-05-09T08:54:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.411+0000 7efdfd6ad640 1 --2- 172.21.15.98:0/941945696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf810d6a0 0x7efdf81a0840 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7efde8009fd0 tx=0x7efde800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.411+0000 7efdf5ffb640 1 -- 172.21.15.98:0/941945696 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde8011830 con 0x7efdf810d6a0 2024-05-09T08:54:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.411+0000 7efdff110640 1 -- 172.21.15.98:0/941945696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdf81a1300 con 0x7efdf810d6a0 2024-05-09T08:54:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.411+0000 7efdf5ffb640 1 -- 172.21.15.98:0/941945696 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efde8016070 con 0x7efdf810d6a0 2024-05-09T08:54:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.411+0000 7efdf5ffb640 1 -- 172.21.15.98:0/941945696 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde800f2e0 con 0x7efdf810d6a0 2024-05-09T08:54:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.413+0000 7efdff110640 1 -- 172.21.15.98:0/941945696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdf810a2c0 con 0x7efdf810d6a0 2024-05-09T08:54:57.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.416+0000 7efdf5ffb640 1 -- 172.21.15.98:0/941945696 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7efde80119d0 con 0x7efdf810d6a0 2024-05-09T08:54:57.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.417+0000 7efdf5ffb640 1 --2- 172.21.15.98:0/941945696 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efde00a2d00 0x7efde00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.417+0000 7efdf5ffb640 1 -- 172.21.15.98:0/941945696 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7efde80ce9a0 con 0x7efdf810d6a0 2024-05-09T08:54:57.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.417+0000 7efdfceac640 1 --2- 172.21.15.98:0/941945696 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efde00a2d00 0x7efde00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.418+0000 7efdfceac640 1 --2- 172.21.15.98:0/941945696 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efde00a2d00 0x7efde00a51c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009f60 tx=0x7efdf00094f0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.419+0000 7efe01e18740 1 Processor -- start 2024-05-09T08:54:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.419+0000 7efe01e18740 1 -- start start 2024-05-09T08:54:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.419+0000 7efe01e18740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 0x555a73fe6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.419+0000 7efe01e18740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6ba0 0x555a73ff0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.419+0000 7efe01e18740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73ff1270 0x555a73ff3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.419+0000 7efe01e18740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555a73f4bd00 con 0x555a73ff1270 2024-05-09T08:54:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.419+0000 7efe01e18740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555a73ef34c0 con 0x555a73fe6ba0 2024-05-09T08:54:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.419+0000 7efe01e18740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555a73f47cc0 con 0x555a73fe6280 2024-05-09T08:54:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.419+0000 7efdf7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6ba0 0x555a73ff0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.419+0000 7efdfd6ad640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73ff1270 0x555a73ff3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.419+0000 7efdf7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6ba0 0x555a73ff0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54914/0 (socket says 172.21.15.98:54914) 2024-05-09T08:54:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.420+0000 7efdf7fff640 1 -- 172.21.15.98:0/86643272 learned_addr learned my addr 172.21.15.98:0/86643272 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.420+0000 7efdfceac640 1 --2- 172.21.15.98:0/86643272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 0x555a73fe6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.420+0000 7efdfd6ad640 1 -- 172.21.15.98:0/86643272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 msgr2=0x555a73fe6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.420+0000 7efdfd6ad640 1 --2- 172.21.15.98:0/86643272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 0x555a73fe6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.420+0000 7efdfd6ad640 1 -- 172.21.15.98:0/86643272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6ba0 msgr2=0x555a73ff0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.420+0000 7efdfd6ad640 1 --2- 172.21.15.98:0/86643272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6ba0 0x555a73ff0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.420+0000 7efdfd6ad640 1 -- 172.21.15.98:0/86643272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a73d297d0 con 0x555a73ff1270 2024-05-09T08:54:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.420+0000 7efdfd6ad640 1 --2- 172.21.15.98:0/86643272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73ff1270 0x555a73ff3660 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7efde80cbe40 tx=0x7efde80047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ab84815e62f933 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.420+0000 7efddeffd640 1 -- 172.21.15.98:0/86643272 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde809efc0 con 0x555a73ff1270 2024-05-09T08:54:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.420+0000 7efddeffd640 1 -- 172.21.15.98:0/86643272 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efde80b1070 con 0x555a73ff1270 2024-05-09T08:54:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.421+0000 7efddeffd640 1 -- 172.21.15.98:0/86643272 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde8097280 con 0x555a73ff1270 2024-05-09T08:54:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.422+0000 7efe01e18740 1 -- 172.21.15.98:0/86643272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73ff1270 msgr2=0x555a73ff3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.422+0000 7efe01e18740 1 --2- 172.21.15.98:0/86643272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73ff1270 0x555a73ff3660 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7efde80cbe40 tx=0x7efde80047d0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.422+0000 7efe01e18740 1 -- 172.21.15.98:0/86643272 shutdown_connections 2024-05-09T08:54:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.422+0000 7efe01e18740 1 --2- 172.21.15.98:0/86643272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73ff1270 0x555a73ff3660 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.422+0000 7efe01e18740 1 --2- 172.21.15.98:0/86643272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6ba0 0x555a73ff0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.422+0000 7efe01e18740 1 --2- 172.21.15.98:0/86643272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 0x555a73fe6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.422+0000 7efe01e18740 1 -- 172.21.15.98:0/86643272 >> 172.21.15.98:0/86643272 conn(0x555a73fe25a0 msgr2=0x555a73fe2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.422+0000 7efe01e18740 1 -- 172.21.15.98:0/86643272 shutdown_connections 2024-05-09T08:54:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.422+0000 7efe01e18740 1 -- 172.21.15.98:0/86643272 wait complete. 2024-05-09T08:54:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.423+0000 7efe01e18740 1 -- start start 2024-05-09T08:54:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.423+0000 7efe01e18740 1 Processor -- start 2024-05-09T08:54:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.425+0000 7efe01e18740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 0x555a73fee290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.425+0000 7efe01e18740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6ba0 0x555a73fee7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.425+0000 7efe01e18740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73ff1270 0x555a73feed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.426+0000 7efe01e18740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555a73ff3e80 con 0x555a73ff1270 2024-05-09T08:54:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.426+0000 7efe01e18740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555a73f06b90 con 0x555a73fe6ba0 2024-05-09T08:54:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.426+0000 7efe01e18740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555a73ff4000 con 0x555a73fe6280 2024-05-09T08:54:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.426+0000 7efe01e18740 10 client.0 _ll_register_callbacks cb 0x555a73f42da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.427+0000 7efdfd6ad640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73ff1270 0x555a73feed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.427+0000 7efdfd6ad640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73ff1270 0x555a73feed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37176/0 (socket says 172.21.15.98:37176) 2024-05-09T08:54:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.427+0000 7efdfd6ad640 1 -- 172.21.15.98:0/1358761684 learned_addr learned my addr 172.21.15.98:0/1358761684 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.427+0000 7efdfceac640 1 --2- 172.21.15.98:0/1358761684 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 0x555a73fee290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.427+0000 7efdfd6ad640 1 -- 172.21.15.98:0/1358761684 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 msgr2=0x555a73fee290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.427+0000 7efdfd6ad640 1 --2- 172.21.15.98:0/1358761684 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 0x555a73fee290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.427+0000 7efdfd6ad640 1 -- 172.21.15.98:0/1358761684 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6ba0 msgr2=0x555a73fee7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:54:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.427+0000 7efdfd6ad640 1 --2- 172.21.15.98:0/1358761684 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6ba0 0x555a73fee7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.428+0000 7efdfd6ad640 1 -- 172.21.15.98:0/1358761684 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a73c5f3f0 con 0x555a73ff1270 2024-05-09T08:54:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.428+0000 7efdfd6ad640 1 --2- 172.21.15.98:0/1358761684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73ff1270 0x555a73feed10 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7efde800bde0 tx=0x7efde809f090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93755433aa2a921 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.428+0000 7efdd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:54:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.429+0000 7efdd7fff640 1 -- 172.21.15.98:0/1358761684 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde8062070 con 0x555a73ff1270 2024-05-09T08:54:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.429+0000 7efdd7fff640 1 -- 172.21.15.98:0/1358761684 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7efde8097280 con 0x555a73ff1270 2024-05-09T08:54:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.429+0000 7efe01e18740 1 -- 172.21.15.98:0/1358761684 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efde80b1070 con 0x555a73ff1270 2024-05-09T08:54:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.430+0000 7efdd7fff640 1 -- 172.21.15.98:0/1358761684 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efde80b8070 con 0x555a73ff1270 2024-05-09T08:54:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.430+0000 7efe01e18740 10 client.16818 fetch_fsmap learned FSMap version 35 2024-05-09T08:54:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.430+0000 7efe01e18740 1 -- 172.21.15.98:0/1358761684 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x555a73c1f0f0 con 0x555a73ff1270 2024-05-09T08:54:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.431+0000 7efdd7fff640 1 -- 172.21.15.98:0/1358761684 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efde809a060 con 0x555a73ff1270 2024-05-09T08:54:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.431+0000 7efdd7fff640 1 -- 172.21.15.98:0/1358761684 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7efde8065020 con 0x555a73ff1270 2024-05-09T08:54:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.432+0000 7efe01e18740 10 client.16818 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:54:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.432+0000 7efe01e18740 10 client.16818 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.432+0000 7efe01e18740 20 client.16818 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.432+0000 7efe01e18740 1 --2- 172.21.15.98:0/1358761684 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a74001120 0x555a74003510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.432+0000 7efe01e18740 4 client.16818 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.432+0000 7efe01e18740 1 -- 172.21.15.98:0/1358761684 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7efde80b1070 con 0x555a74001120 2024-05-09T08:54:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.434+0000 7efdfceac640 1 --2- 172.21.15.98:0/1358761684 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a74001120 0x555a74003510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.438+0000 7efdfceac640 1 --2- 172.21.15.98:0/1358761684 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a74001120 0x555a74003510 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14dfbc1085e92555 server_cookie=b4cfed35aa7fb89f in_seq=0 out_seq=0 2024-05-09T08:54:57.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.438+0000 7efdd7fff640 10 client.16818 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:57.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.440+0000 7efdd7fff640 1 -- 172.21.15.98:0/1358761684 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7efde80b1070 con 0x555a74001120 2024-05-09T08:54:57.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.440+0000 7efdd7fff640 10 client.16818 handle_command_reply: tid=0 2024-05-09T08:54:57.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.440+0000 7efe01e18740 1 client.16818 shutdown 2024-05-09T08:54:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.441+0000 7efe01e18740 1 -- 172.21.15.98:0/1358761684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73ff1270 msgr2=0x555a73feed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.441+0000 7efe01e18740 1 --2- 172.21.15.98:0/1358761684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73ff1270 0x555a73feed10 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7efde800bde0 tx=0x7efde809f090 comp rx=0 tx=0).stop 2024-05-09T08:54:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.442+0000 7efe01e18740 1 -- 172.21.15.98:0/1358761684 shutdown_connections 2024-05-09T08:54:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.442+0000 7efe01e18740 1 --2- 172.21.15.98:0/1358761684 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a74001120 0x555a74003510 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.442+0000 7efe01e18740 1 --2- 172.21.15.98:0/1358761684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73ff1270 0x555a73feed10 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.442+0000 7efe01e18740 1 --2- 172.21.15.98:0/1358761684 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6ba0 0x555a73fee7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.442+0000 7efe01e18740 1 --2- 172.21.15.98:0/1358761684 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 0x555a73fee290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.442+0000 7efe01e18740 1 -- 172.21.15.98:0/1358761684 >> 172.21.15.98:0/1358761684 conn(0x555a73fe25a0 msgr2=0x555a73fe83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.442+0000 7efdd7fff640 0 client.16818 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.442+0000 7efe01e18740 1 -- 172.21.15.98:0/1358761684 shutdown_connections 2024-05-09T08:54:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.442+0000 7efe01e18740 1 -- 172.21.15.98:0/1358761684 wait complete. 2024-05-09T08:54:57.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.442+0000 7efe01e18740 20 client.16818 trim_cache size 0 max 16384 2024-05-09T08:54:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.471+0000 7efe01e18740 1 Processor -- start 2024-05-09T08:54:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.471+0000 7efe01e18740 1 -- start start 2024-05-09T08:54:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.471+0000 7efe01e18740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 0x555a73feca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.472+0000 7efe01e18740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73fe6ba0 0x555a73feeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.472+0000 7efe01e18740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73ff1270 0x555a73fe8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.472+0000 7efdfceac640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 0x555a73feca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.472+0000 7efdfceac640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 0x555a73feca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:50916/0 (socket says 172.21.15.98:50916) 2024-05-09T08:54:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.472+0000 7efdfceac640 1 -- 172.21.15.98:0/1055389422 learned_addr learned my addr 172.21.15.98:0/1055389422 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.472+0000 7efdfd6ad640 1 --2- 172.21.15.98:0/1055389422 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73ff1270 0x555a73fe8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.472+0000 7efe01e18740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdf81b9960 con 0x555a73fe6ba0 2024-05-09T08:54:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.472+0000 7efe01e18740 1 -- 172.21.15.98:0/1055389422 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efdf81bb4a0 con 0x555a73ff1270 2024-05-09T08:54:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.472+0000 7efe01e18740 1 -- 172.21.15.98:0/1055389422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efdf81ba030 con 0x555a73fe6280 2024-05-09T08:54:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.473+0000 7efdf7fff640 1 --2- 172.21.15.98:0/1055389422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73fe6ba0 0x555a73feeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.473+0000 7efdfceac640 1 -- 172.21.15.98:0/1055389422 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73ff1270 msgr2=0x555a73fe8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.473+0000 7efdfceac640 1 --2- 172.21.15.98:0/1055389422 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73ff1270 0x555a73fe8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.473+0000 7efdfceac640 1 -- 172.21.15.98:0/1055389422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73fe6ba0 msgr2=0x555a73feeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.473+0000 7efdfceac640 1 --2- 172.21.15.98:0/1055389422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73fe6ba0 0x555a73feeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.473+0000 7efdfceac640 1 -- 172.21.15.98:0/1055389422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a74003880 con 0x555a73fe6280 2024-05-09T08:54:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.474+0000 7efdfceac640 1 --2- 172.21.15.98:0/1055389422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 0x555a73feca30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efdf00176e0 tx=0x7efdf0015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5de976931d2b2811 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.474+0000 7efdde7fc640 1 -- 172.21.15.98:0/1055389422 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf002c030 con 0x555a73fe6280 2024-05-09T08:54:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.474+0000 7efdde7fc640 1 -- 172.21.15.98:0/1055389422 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdf0027640 con 0x555a73fe6280 2024-05-09T08:54:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.474+0000 7efdde7fc640 1 -- 172.21.15.98:0/1055389422 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf00249c0 con 0x555a73fe6280 2024-05-09T08:54:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.475+0000 7efe01e18740 1 -- 172.21.15.98:0/1055389422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 msgr2=0x555a73feca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.475+0000 7efe01e18740 1 --2- 172.21.15.98:0/1055389422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 0x555a73feca30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efdf00176e0 tx=0x7efdf0015000 comp rx=0 tx=0).stop 2024-05-09T08:54:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.476+0000 7efe01e18740 1 -- 172.21.15.98:0/1055389422 shutdown_connections 2024-05-09T08:54:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.476+0000 7efe01e18740 1 --2- 172.21.15.98:0/1055389422 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73ff1270 0x555a73fe8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.476+0000 7efe01e18740 1 --2- 172.21.15.98:0/1055389422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73fe6ba0 0x555a73feeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.476+0000 7efe01e18740 1 --2- 172.21.15.98:0/1055389422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73fe6280 0x555a73feca30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.476+0000 7efe01e18740 1 -- 172.21.15.98:0/1055389422 >> 172.21.15.98:0/1055389422 conn(0x555a73fe25a0 msgr2=0x555a73fe17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.476+0000 7efe01e18740 1 -- 172.21.15.98:0/1055389422 shutdown_connections 2024-05-09T08:54:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.476+0000 7efe01e18740 1 -- 172.21.15.98:0/1055389422 wait complete. 2024-05-09T08:54:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.476+0000 7efe01e18740 1 -- start start 2024-05-09T08:54:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.476+0000 7efe01e18740 1 Processor -- start 2024-05-09T08:54:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.477+0000 7efe01e18740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6280 0x555a73feb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.477+0000 7efe01e18740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73fe6ba0 0x555a73feb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.477+0000 7efe01e18740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73ff1270 0x555a73febc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.477+0000 7efe01e18740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdf81badd0 con 0x555a73fe6ba0 2024-05-09T08:54:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.477+0000 7efe01e18740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efdf81ba700 con 0x555a73fe6280 2024-05-09T08:54:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.477+0000 7efdfceac640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6280 0x555a73feb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.477+0000 7efdfceac640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6280 0x555a73feb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54948/0 (socket says 172.21.15.98:54948) 2024-05-09T08:54:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.477+0000 7efdfceac640 1 -- 172.21.15.98:0/920563329 learned_addr learned my addr 172.21.15.98:0/920563329 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.477+0000 7efdfd6ad640 1 --2- 172.21.15.98:0/920563329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73ff1270 0x555a73febc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.477+0000 7efe01e18740 1 -- 172.21.15.98:0/920563329 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efdf81b7940 con 0x555a73ff1270 2024-05-09T08:54:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.477+0000 7efe01e18740 10 client.0 _ll_register_callbacks cb 0x7efdec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.478+0000 7efdf7fff640 1 --2- 172.21.15.98:0/920563329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73fe6ba0 0x555a73feb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.478+0000 7efdfceac640 1 -- 172.21.15.98:0/920563329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73ff1270 msgr2=0x555a73febc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.478+0000 7efdfceac640 1 --2- 172.21.15.98:0/920563329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73ff1270 0x555a73febc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.478+0000 7efdfceac640 1 -- 172.21.15.98:0/920563329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73fe6ba0 msgr2=0x555a73feb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.478+0000 7efdfceac640 1 --2- 172.21.15.98:0/920563329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73fe6ba0 0x555a73feb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.478+0000 7efdfceac640 1 -- 172.21.15.98:0/920563329 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a73fe69d0 con 0x555a73fe6280 2024-05-09T08:54:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.478+0000 7efdfceac640 1 --2- 172.21.15.98:0/920563329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6280 0x555a73feb200 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efdf0000f80 tx=0x7efdf0017530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3710d2b986ecc82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.478+0000 7efdd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:54:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.478+0000 7efdd7fff640 1 -- 172.21.15.98:0/920563329 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf002c070 con 0x555a73fe6280 2024-05-09T08:54:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.478+0000 7efdd7fff640 1 -- 172.21.15.98:0/920563329 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7efdf0027360 con 0x555a73fe6280 2024-05-09T08:54:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.479+0000 7efe01e18740 1 -- 172.21.15.98:0/920563329 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdf0027640 con 0x555a73fe6280 2024-05-09T08:54:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.479+0000 7efdd7fff640 1 -- 172.21.15.98:0/920563329 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efdf003c050 con 0x555a73fe6280 2024-05-09T08:54:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.479+0000 7efe01e18740 10 client.26126 fetch_fsmap learned FSMap version 35 2024-05-09T08:54:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.479+0000 7efe01e18740 1 -- 172.21.15.98:0/920563329 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x555a74067730 con 0x555a73fe6280 2024-05-09T08:54:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.479+0000 7efdd7fff640 1 -- 172.21.15.98:0/920563329 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdf0027640 con 0x555a73fe6280 2024-05-09T08:54:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.479+0000 7efdd7fff640 1 -- 172.21.15.98:0/920563329 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7efdf0020040 con 0x555a73fe6280 2024-05-09T08:54:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.479+0000 7efe01e18740 10 client.26126 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:54:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.479+0000 7efe01e18740 10 client.26126 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:54:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.479+0000 7efe01e18740 20 client.26126 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.479+0000 7efe01e18740 1 --2- 172.21.15.98:0/920563329 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a74067da0 0x555a7406a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.479+0000 7efe01e18740 4 client.26126 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:54:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.479+0000 7efe01e18740 1 -- 172.21.15.98:0/920563329 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555a73f59430 con 0x555a74067da0 2024-05-09T08:54:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.480+0000 7efdf7fff640 1 --2- 172.21.15.98:0/920563329 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a74067da0 0x555a7406a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.480+0000 7efdd7fff640 10 client.26126 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.480+0000 7efdf7fff640 1 --2- 172.21.15.98:0/920563329 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a74067da0 0x555a7406a1c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=653fff56042a80f6 server_cookie=645c9f15e82fabe7 in_seq=0 out_seq=0 2024-05-09T08:54:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.482+0000 7efdd7fff640 1 -- 172.21.15.98:0/920563329 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555a73f59430 con 0x555a74067da0 2024-05-09T08:54:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.482+0000 7efdd7fff640 10 client.26126 handle_command_reply: tid=0 2024-05-09T08:54:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.482+0000 7efe01e18740 1 client.26126 shutdown 2024-05-09T08:54:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.483+0000 7efe01e18740 1 -- 172.21.15.98:0/920563329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6280 msgr2=0x555a73feb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.483+0000 7efe01e18740 1 --2- 172.21.15.98:0/920563329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6280 0x555a73feb200 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efdf0000f80 tx=0x7efdf0017530 comp rx=0 tx=0).stop 2024-05-09T08:54:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.483+0000 7efe01e18740 1 -- 172.21.15.98:0/920563329 shutdown_connections 2024-05-09T08:54:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.483+0000 7efe01e18740 1 --2- 172.21.15.98:0/920563329 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a74067da0 0x555a7406a1c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.483+0000 7efe01e18740 1 --2- 172.21.15.98:0/920563329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a73ff1270 0x555a73febc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.483+0000 7efe01e18740 1 --2- 172.21.15.98:0/920563329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a73fe6ba0 0x555a73feb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.483+0000 7efe01e18740 1 --2- 172.21.15.98:0/920563329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a73fe6280 0x555a73feb200 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.483+0000 7efe01e18740 1 -- 172.21.15.98:0/920563329 >> 172.21.15.98:0/920563329 conn(0x555a73fe25a0 msgr2=0x555a73fefac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.483+0000 7efdd7fff640 0 client.26126 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:54:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.483+0000 7efe01e18740 1 -- 172.21.15.98:0/920563329 shutdown_connections 2024-05-09T08:54:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.483+0000 7efe01e18740 1 -- 172.21.15.98:0/920563329 wait complete. 2024-05-09T08:54:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.483+0000 7efe01e18740 20 client.26126 trim_cache size 0 max 16384 2024-05-09T08:54:57.484 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:54:57.485 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:54:57.485 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:54:57.485 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:54:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.484+0000 7efdff110640 1 -- 172.21.15.98:0/941945696 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efde00a2d00 msgr2=0x7efde00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.484+0000 7efdff110640 1 --2- 172.21.15.98:0/941945696 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efde00a2d00 0x7efde00a51c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009f60 tx=0x7efdf00094f0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.484+0000 7efdff110640 1 -- 172.21.15.98:0/941945696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf810d6a0 msgr2=0x7efdf81a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.484+0000 7efdff110640 1 --2- 172.21.15.98:0/941945696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf810d6a0 0x7efdf81a0840 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7efde8009fd0 tx=0x7efde800b8e0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.484+0000 7efdff110640 1 -- 172.21.15.98:0/941945696 shutdown_connections 2024-05-09T08:54:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.484+0000 7efdff110640 1 --2- 172.21.15.98:0/941945696 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efde00a2d00 0x7efde00a51c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.484+0000 7efdff110640 1 --2- 172.21.15.98:0/941945696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf810d6a0 0x7efdf81a0840 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.484+0000 7efdff110640 1 --2- 172.21.15.98:0/941945696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdf8109a80 0x7efdf81a6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.484+0000 7efdff110640 1 --2- 172.21.15.98:0/941945696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf81090c0 0x7efdf81a66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.484+0000 7efdff110640 1 -- 172.21.15.98:0/941945696 >> 172.21.15.98:0/941945696 conn(0x7efdf806c940 msgr2=0x7efdf810f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.484+0000 7efdff110640 1 -- 172.21.15.98:0/941945696 shutdown_connections 2024-05-09T08:54:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.484+0000 7efdff110640 1 -- 172.21.15.98:0/941945696 wait complete. 2024-05-09T08:54:57.497 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:54:57.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:54:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.667+0000 7f95edf54640 1 -- 172.21.15.98:0/1942027828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95e806dac0 msgr2=0x7f95e8108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.667+0000 7f95edf54640 1 --2- 172.21.15.98:0/1942027828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95e806dac0 0x7f95e8108830 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f95dc009900 tx=0x7f95dc02a420 comp rx=0 tx=0).stop 2024-05-09T08:54:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.667+0000 7f95edf54640 1 -- 172.21.15.98:0/1942027828 shutdown_connections 2024-05-09T08:54:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.667+0000 7f95edf54640 1 --2- 172.21.15.98:0/1942027828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95e81144d0 0x7f95e81168e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.667+0000 7f95edf54640 1 --2- 172.21.15.98:0/1942027828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e8108e00 0x7f95e8109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.667+0000 7f95edf54640 1 --2- 172.21.15.98:0/1942027828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95e806dac0 0x7f95e8108830 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f95dc009900 tx=0x7f95dc02a420 comp rx=0 tx=0).stop 2024-05-09T08:54:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.667+0000 7f95edf54640 1 -- 172.21.15.98:0/1942027828 >> 172.21.15.98:0/1942027828 conn(0x7f95e806c740 msgr2=0x7f95e806cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:57.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.672+0000 7f95edf54640 1 -- 172.21.15.98:0/1942027828 shutdown_connections 2024-05-09T08:54:57.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.672+0000 7f95edf54640 1 -- 172.21.15.98:0/1942027828 wait complete. 2024-05-09T08:54:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.673+0000 7f95edf54640 1 Processor -- start 2024-05-09T08:54:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.673+0000 7f95edf54640 1 -- start start 2024-05-09T08:54:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.673+0000 7f95edf54640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95e8108e00 0x7f95e81a67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.673+0000 7f95edf54640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95e81144d0 0x7f95e81a6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.673+0000 7f95edf54640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e81a0990 0x7f95e81a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.674+0000 7f95e7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95e81144d0 0x7f95e81a6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.674+0000 7f95edf54640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f95e811a330 con 0x7f95e81a0990 2024-05-09T08:54:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.674+0000 7f95edf54640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f95e811a1b0 con 0x7f95e81144d0 2024-05-09T08:54:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.674+0000 7f95edf54640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f95e811a4b0 con 0x7f95e8108e00 2024-05-09T08:54:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.674+0000 7f95e7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95e81144d0 0x7f95e81a6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54966/0 (socket says 172.21.15.98:54966) 2024-05-09T08:54:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.674+0000 7f95e7fff640 1 -- 172.21.15.98:0/2381958932 learned_addr learned my addr 172.21.15.98:0/2381958932 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.674+0000 7f95ecf52640 1 --2- 172.21.15.98:0/2381958932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95e8108e00 0x7f95e81a67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.675+0000 7f95ed753640 1 --2- 172.21.15.98:0/2381958932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e81a0990 0x7f95e81a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.676+0000 7f95ecf52640 1 -- 172.21.15.98:0/2381958932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95e81144d0 msgr2=0x7f95e81a6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.676+0000 7f95ecf52640 1 --2- 172.21.15.98:0/2381958932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95e81144d0 0x7f95e81a6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.676+0000 7f95ecf52640 1 -- 172.21.15.98:0/2381958932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e81a0990 msgr2=0x7f95e81a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.676+0000 7f95ecf52640 1 --2- 172.21.15.98:0/2381958932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e81a0990 0x7f95e81a0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.676+0000 7f95ecf52640 1 -- 172.21.15.98:0/2381958932 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e81a14b0 con 0x7f95e8108e00 2024-05-09T08:54:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.677+0000 7f95ecf52640 1 --2- 172.21.15.98:0/2381958932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95e8108e00 0x7f95e81a67e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f95dc008000 tx=0x7f95dc003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.678+0000 7f95e5ffb640 1 -- 172.21.15.98:0/2381958932 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc03f020 con 0x7f95e8108e00 2024-05-09T08:54:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.678+0000 7f95e5ffb640 1 -- 172.21.15.98:0/2381958932 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95dc003e40 con 0x7f95e8108e00 2024-05-09T08:54:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.678+0000 7f95e5ffb640 1 -- 172.21.15.98:0/2381958932 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc03f020 con 0x7f95e8108e00 2024-05-09T08:54:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.679+0000 7f95edf54640 1 -- 172.21.15.98:0/2381958932 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e81a1740 con 0x7f95e8108e00 2024-05-09T08:54:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.679+0000 7f95edf54640 1 -- 172.21.15.98:0/2381958932 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e81ad8e0 con 0x7f95e8108e00 2024-05-09T08:54:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.686+0000 7f95cf7fe640 1 -- 172.21.15.98:0/2381958932 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e8068570 con 0x7f95e8108e00 2024-05-09T08:54:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.689+0000 7f95e5ffb640 1 -- 172.21.15.98:0/2381958932 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f95dc042020 con 0x7f95e8108e00 2024-05-09T08:54:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.691+0000 7f95e5ffb640 1 --2- 172.21.15.98:0/2381958932 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f95d00a31c0 0x7f95d00a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.691+0000 7f95e5ffb640 1 -- 172.21.15.98:0/2381958932 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f95dc0317c0 con 0x7f95e8108e00 2024-05-09T08:54:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.691+0000 7f95e7fff640 1 --2- 172.21.15.98:0/2381958932 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f95d00a31c0 0x7f95d00a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.692+0000 7f95e7fff640 1 --2- 172.21.15.98:0/2381958932 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f95d00a31c0 0x7f95d00a5680 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f95e80727d0 tx=0x7f95e000b040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:57.693+0000 7f95e5ffb640 1 -- 172.21.15.98:0/2381958932 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f95dc049020 con 0x7f95e8108e00 2024-05-09T08:54:58.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:58 smithi098 ceph-mon[30092]: pgmap v134: 129 pgs: 129 active+clean; 30 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 889 MiB/s wr, 1.65k op/s 2024-05-09T08:54:58.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.342+0000 7f95cf7fe640 1 -- 172.21.15.98:0/2381958932 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95e8062430 con 0x7f95e8108e00 2024-05-09T08:54:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.347+0000 7f95e5ffb640 1 -- 172.21.15.98:0/2381958932 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11944 (secure 0 0 0) 0x7f95dc0bc7c0 con 0x7f95e8108e00 2024-05-09T08:54:58.349 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:54:58.349 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":16641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:54:25.881641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16749":{"gid":16749,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/3204012039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3204012039},{"type":"v1","addr":"172.21.15.105:6835","nonce":3204012039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24439,14667,24388,24461,24457]},"id":1}]} 2024-05-09T08:54:58.350 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 35 2024-05-09T08:54:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.356+0000 7f95cf7fe640 1 -- 172.21.15.98:0/2381958932 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f95d00a31c0 msgr2=0x7f95d00a5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.356+0000 7f95cf7fe640 1 --2- 172.21.15.98:0/2381958932 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f95d00a31c0 0x7f95d00a5680 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f95e80727d0 tx=0x7f95e000b040 comp rx=0 tx=0).stop 2024-05-09T08:54:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.356+0000 7f95cf7fe640 1 -- 172.21.15.98:0/2381958932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95e8108e00 msgr2=0x7f95e81a67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.356+0000 7f95cf7fe640 1 --2- 172.21.15.98:0/2381958932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95e8108e00 0x7f95e81a67e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f95dc008000 tx=0x7f95dc003cd0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.357+0000 7f95cf7fe640 1 -- 172.21.15.98:0/2381958932 shutdown_connections 2024-05-09T08:54:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.357+0000 7f95cf7fe640 1 --2- 172.21.15.98:0/2381958932 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f95d00a31c0 0x7f95d00a5680 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.357+0000 7f95cf7fe640 1 --2- 172.21.15.98:0/2381958932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e81a0990 0x7f95e81a0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.357+0000 7f95cf7fe640 1 --2- 172.21.15.98:0/2381958932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95e81144d0 0x7f95e81a6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.357+0000 7f95cf7fe640 1 --2- 172.21.15.98:0/2381958932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95e8108e00 0x7f95e81a67e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.357+0000 7f95cf7fe640 1 -- 172.21.15.98:0/2381958932 >> 172.21.15.98:0/2381958932 conn(0x7f95e806c740 msgr2=0x7f95e8071780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.357+0000 7f95cf7fe640 1 -- 172.21.15.98:0/2381958932 shutdown_connections 2024-05-09T08:54:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.362+0000 7f95cf7fe640 1 -- 172.21.15.98:0/2381958932 wait complete. 2024-05-09T08:54:58.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:54:58.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:58 smithi142 ceph-mon[34625]: pgmap v134: 129 pgs: 129 active+clean; 30 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 889 MiB/s wr, 1.65k op/s 2024-05-09T08:54:58.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:58 smithi105 ceph-mon[34498]: pgmap v134: 129 pgs: 129 active+clean; 30 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 889 MiB/s wr, 1.65k op/s 2024-05-09T08:54:58.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.771+0000 7fd156116640 1 -- 172.21.15.98:0/1643680008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd150102c60 msgr2=0x7fd150100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:58.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.771+0000 7fd156116640 1 --2- 172.21.15.98:0/1643680008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd150102c60 0x7fd150100280 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd138009900 tx=0x7fd13802a400 comp rx=0 tx=0).stop 2024-05-09T08:54:58.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.771+0000 7fd156116640 1 -- 172.21.15.98:0/1643680008 shutdown_connections 2024-05-09T08:54:58.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.771+0000 7fd156116640 1 --2- 172.21.15.98:0/1643680008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd15010e4c0 0x7fd1501108b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.771+0000 7fd156116640 1 --2- 172.21.15.98:0/1643680008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd150102c60 0x7fd150100280 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.771+0000 7fd156116640 1 --2- 172.21.15.98:0/1643680008 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1501027f0 0x7fd1500ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.771+0000 7fd156116640 1 -- 172.21.15.98:0/1643680008 >> 172.21.15.98:0/1643680008 conn(0x7fd150076cd0 msgr2=0x7fd1500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:58.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.771+0000 7fd156116640 1 -- 172.21.15.98:0/1643680008 shutdown_connections 2024-05-09T08:54:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.771+0000 7fd156116640 1 -- 172.21.15.98:0/1643680008 wait complete. 2024-05-09T08:54:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.772+0000 7fd156116640 1 Processor -- start 2024-05-09T08:54:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.772+0000 7fd156116640 1 -- start start 2024-05-09T08:54:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.773+0000 7fd156116640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1501027f0 0x7fd150104b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.773+0000 7fd156116640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd150102c60 0x7fd1501050c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.773+0000 7fd156116640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15010e4c0 0x7fd150105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.773+0000 7fd156116640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1501043c0 con 0x7fd15010e4c0 2024-05-09T08:54:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.773+0000 7fd156116640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd150104240 con 0x7fd1501027f0 2024-05-09T08:54:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.773+0000 7fd156116640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd150104540 con 0x7fd150102c60 2024-05-09T08:54:58.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.773+0000 7fd14f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1501027f0 0x7fd150104b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.773+0000 7fd14f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1501027f0 0x7fd150104b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54980/0 (socket says 172.21.15.98:54980) 2024-05-09T08:54:58.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.773+0000 7fd14f7fe640 1 -- 172.21.15.98:0/3166081469 learned_addr learned my addr 172.21.15.98:0/3166081469 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:58.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.773+0000 7fd14effd640 1 --2- 172.21.15.98:0/3166081469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd150102c60 0x7fd1501050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.774+0000 7fd14f7fe640 1 -- 172.21.15.98:0/3166081469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd150102c60 msgr2=0x7fd1501050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:58.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.774+0000 7fd14f7fe640 1 --2- 172.21.15.98:0/3166081469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd150102c60 0x7fd1501050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.774+0000 7fd14f7fe640 1 -- 172.21.15.98:0/3166081469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15010e4c0 msgr2=0x7fd150105600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:58.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.774+0000 7fd14f7fe640 1 --2- 172.21.15.98:0/3166081469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15010e4c0 0x7fd150105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.774+0000 7fd14f7fe640 1 -- 172.21.15.98:0/3166081469 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1501a2270 con 0x7fd1501027f0 2024-05-09T08:54:58.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.774+0000 7fd14f7fe640 1 --2- 172.21.15.98:0/3166081469 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1501027f0 0x7fd150104b80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd14000ba50 tx=0x7fd14000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:58.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.775+0000 7fd14cff9640 1 -- 172.21.15.98:0/3166081469 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140019070 con 0x7fd1501027f0 2024-05-09T08:54:58.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.775+0000 7fd14cff9640 1 -- 172.21.15.98:0/3166081469 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd14000cbe0 con 0x7fd1501027f0 2024-05-09T08:54:58.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.775+0000 7fd14cff9640 1 -- 172.21.15.98:0/3166081469 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140007770 con 0x7fd1501027f0 2024-05-09T08:54:58.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.777+0000 7fd156116640 1 -- 172.21.15.98:0/3166081469 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1501a2560 con 0x7fd1501027f0 2024-05-09T08:54:58.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.777+0000 7fd156116640 1 -- 172.21.15.98:0/3166081469 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1501a2a30 con 0x7fd1501027f0 2024-05-09T08:54:58.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.780+0000 7fd158e1b740 1 Processor -- start 2024-05-09T08:54:58.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.795+0000 7fd158e1b740 1 -- start start 2024-05-09T08:54:58.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.795+0000 7fd14cff9640 1 -- 172.21.15.98:0/3166081469 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd140014440 con 0x7fd1501027f0 2024-05-09T08:54:58.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.795+0000 7fd158e1b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2330 0x5555c35f2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.795+0000 7fd158e1b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2c50 0x5555c35fcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.795+0000 7fd158e1b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 0x5555c35ff710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.795+0000 7fd158e1b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5555c3557c40 con 0x5555c35fd320 2024-05-09T08:54:58.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.795+0000 7fd158e1b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5555c34ff580 con 0x5555c35f2330 2024-05-09T08:54:58.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.795+0000 7fd158e1b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5555c3553c00 con 0x5555c35f2c50 2024-05-09T08:54:58.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.796+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2330 0x5555c35f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.796+0000 7fd14cff9640 1 --2- 172.21.15.98:0/3166081469 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd13c0a2ed0 0x7fd13c0a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.797+0000 7fd14cff9640 1 -- 172.21.15.98:0/3166081469 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fd1400cecc0 con 0x7fd1501027f0 2024-05-09T08:54:58.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.797+0000 7fd14f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2c50 0x5555c35fcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.797+0000 7fd14f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2c50 0x5555c35fcde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:50948/0 (socket says 172.21.15.98:50948) 2024-05-09T08:54:58.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.797+0000 7fd14f7fe640 1 -- 172.21.15.98:0/1661013576 learned_addr learned my addr 172.21.15.98:0/1661013576 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.797+0000 7fd14effd640 1 --2- 172.21.15.98:0/1661013576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 0x5555c35ff710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.800+0000 7fd14f7fe640 1 -- 172.21.15.98:0/1661013576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2330 msgr2=0x5555c35f2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.800+0000 7fd14f7fe640 1 --2- 172.21.15.98:0/1661013576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2330 0x5555c35f2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.800+0000 7fd14f7fe640 1 -- 172.21.15.98:0/1661013576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 msgr2=0x5555c35ff710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.800+0000 7fd14f7fe640 1 --2- 172.21.15.98:0/1661013576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 0x5555c35ff710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.800+0000 7fd14f7fe640 1 -- 172.21.15.98:0/1661013576 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555c326b3f0 con 0x5555c35f2c50 2024-05-09T08:54:58.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.801+0000 7fd14f7fe640 1 --2- 172.21.15.98:0/1661013576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2c50 0x5555c35fcde0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd1400978a0 tx=0x7fd140004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2f4b6841425a69f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:58.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.802+0000 7fd145ffb640 1 -- 172.21.15.98:0/1661013576 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1400b3070 con 0x5555c35f2c50 2024-05-09T08:54:58.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.803+0000 7fd14ffff640 1 --2- 172.21.15.98:0/3166081469 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd13c0a2ed0 0x7fd13c0a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.808+0000 7fd145ffb640 1 -- 172.21.15.98:0/1661013576 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd14009aed0 con 0x5555c35f2c50 2024-05-09T08:54:58.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.808+0000 7fd145ffb640 1 -- 172.21.15.98:0/1661013576 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140097540 con 0x5555c35f2c50 2024-05-09T08:54:58.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.809+0000 7fd158e1b740 1 -- 172.21.15.98:0/1661013576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2c50 msgr2=0x5555c35fcde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:58.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.809+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1661013576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2c50 0x5555c35fcde0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd1400978a0 tx=0x7fd140004a40 comp rx=0 tx=0).stop 2024-05-09T08:54:58.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.812+0000 7fd14ffff640 1 --2- 172.21.15.98:0/3166081469 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd13c0a2ed0 0x7fd13c0a5390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd150073fc0 tx=0x7fd1480046c0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:58.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.812+0000 7fd158e1b740 1 -- 172.21.15.98:0/1661013576 shutdown_connections 2024-05-09T08:54:58.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.812+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1661013576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 0x5555c35ff710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.812+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1661013576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2c50 0x5555c35fcde0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.812+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1661013576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2330 0x5555c35f2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.812+0000 7fd158e1b740 1 -- 172.21.15.98:0/1661013576 >> 172.21.15.98:0/1661013576 conn(0x5555c35ee650 msgr2=0x5555c35eea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:58.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.815+0000 7fd158e1b740 1 -- 172.21.15.98:0/1661013576 shutdown_connections 2024-05-09T08:54:58.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.819+0000 7fd158e1b740 1 -- 172.21.15.98:0/1661013576 wait complete. 2024-05-09T08:54:58.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.819+0000 7fd158e1b740 1 -- start start 2024-05-09T08:54:58.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.819+0000 7fd158e1b740 1 Processor -- start 2024-05-09T08:54:58.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.821+0000 7fd158e1b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2330 0x5555c35fa310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.821+0000 7fd158e1b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2c50 0x5555c35fa850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.821+0000 7fd158e1b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 0x5555c35fad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.821+0000 7fd158e1b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5555c3600110 con 0x5555c35fd320 2024-05-09T08:54:58.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.821+0000 7fd158e1b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5555c3512c40 con 0x5555c35f2c50 2024-05-09T08:54:58.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.821+0000 7fd158e1b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5555c3600290 con 0x5555c35f2330 2024-05-09T08:54:58.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.821+0000 7fd158e1b740 10 client.0 _ll_register_callbacks cb 0x5555c326abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:58.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.824+0000 7fd14f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2330 0x5555c35fa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.824+0000 7fd14f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2330 0x5555c35fa310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:50952/0 (socket says 172.21.15.98:50952) 2024-05-09T08:54:58.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.824+0000 7fd14f7fe640 1 -- 172.21.15.98:0/1616182600 learned_addr learned my addr 172.21.15.98:0/1616182600 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:58.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.824+0000 7fd14effd640 1 --2- 172.21.15.98:0/1616182600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2c50 0x5555c35fa850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.824+0000 7fd14ffff640 1 --2- 172.21.15.98:0/1616182600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 0x5555c35fad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.825+0000 7fd14f7fe640 1 -- 172.21.15.98:0/1616182600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2c50 msgr2=0x5555c35fa850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:58.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.825+0000 7fd14f7fe640 1 --2- 172.21.15.98:0/1616182600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2c50 0x5555c35fa850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.825+0000 7fd14f7fe640 1 -- 172.21.15.98:0/1616182600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 msgr2=0x5555c35fad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:58.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.825+0000 7fd14f7fe640 1 --2- 172.21.15.98:0/1616182600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 0x5555c35fad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.825+0000 7fd14f7fe640 1 -- 172.21.15.98:0/1616182600 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555c33357d0 con 0x5555c35f2330 2024-05-09T08:54:58.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.827+0000 7fd14f7fe640 1 --2- 172.21.15.98:0/1616182600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2330 0x5555c35fa310 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd140000f80 tx=0x7fd14009f370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c1e30989398b5b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:58.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.830+0000 7fd12effd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:54:58.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.830+0000 7fd12effd640 1 -- 172.21.15.98:0/1616182600 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1400b3040 con 0x5555c35f2330 2024-05-09T08:54:58.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.830+0000 7fd12effd640 1 -- 172.21.15.98:0/1616182600 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fd1400a36b0 con 0x5555c35f2330 2024-05-09T08:54:58.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.831+0000 7fd158e1b740 1 -- 172.21.15.98:0/1616182600 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd14009aed0 con 0x5555c35f2330 2024-05-09T08:54:58.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.833+0000 7fd12effd640 1 -- 172.21.15.98:0/1616182600 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd1400cc470 con 0x5555c35f2330 2024-05-09T08:54:58.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.833+0000 7fd158e1b740 10 client.26095 fetch_fsmap learned FSMap version 35 2024-05-09T08:54:58.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.833+0000 7fd158e1b740 1 -- 172.21.15.98:0/1616182600 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5555c3600160 con 0x5555c35f2330 2024-05-09T08:54:58.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.835+0000 7fd12effd640 1 -- 172.21.15.98:0/1616182600 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd14009aed0 con 0x5555c35f2330 2024-05-09T08:54:58.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.835+0000 7fd12effd640 1 -- 172.21.15.98:0/1616182600 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd140096c20 con 0x5555c35f2330 2024-05-09T08:54:58.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.835+0000 7fd158e1b740 10 client.26095 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:54:58.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.835+0000 7fd158e1b740 10 client.26095 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:54:58.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.835+0000 7fd158e1b740 20 client.26095 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:58.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.835+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1616182600 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5555c361e920 0x5555c3620d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.835+0000 7fd158e1b740 4 client.26095 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:58.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.835+0000 7fd158e1b740 1 -- 172.21.15.98:0/1616182600 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555c3565370 con 0x5555c361e920 2024-05-09T08:54:58.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.839+0000 7fd14effd640 1 --2- 172.21.15.98:0/1616182600 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5555c361e920 0x5555c3620d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.854+0000 7fd12effd640 10 client.26095 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:54:58.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.854+0000 7fd14effd640 1 --2- 172.21.15.98:0/1616182600 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5555c361e920 0x5555c3620d70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=90c9869cfa25a071 server_cookie=da8602c63e2bd185 in_seq=0 out_seq=0 2024-05-09T08:54:58.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.855+0000 7fd12effd640 1 -- 172.21.15.98:0/1616182600 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5555c3565370 con 0x5555c361e920 2024-05-09T08:54:58.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.855+0000 7fd12effd640 10 client.26095 handle_command_reply: tid=0 2024-05-09T08:54:58.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.855+0000 7fd158e1b740 1 client.26095 shutdown 2024-05-09T08:54:58.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.862+0000 7fd158e1b740 1 -- 172.21.15.98:0/1616182600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2330 msgr2=0x5555c35fa310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:58.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.862+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1616182600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2330 0x5555c35fa310 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd140000f80 tx=0x7fd14009f370 comp rx=0 tx=0).stop 2024-05-09T08:54:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.863+0000 7fd158e1b740 1 -- 172.21.15.98:0/1616182600 shutdown_connections 2024-05-09T08:54:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.863+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1616182600 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5555c361e920 0x5555c3620d70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.864+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1616182600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 0x5555c35fad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.864+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1616182600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2c50 0x5555c35fa850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.864+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1616182600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2330 0x5555c35fa310 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.864+0000 7fd12effd640 0 client.26095 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:54:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.864+0000 7fd158e1b740 1 -- 172.21.15.98:0/1616182600 >> 172.21.15.98:0/1616182600 conn(0x5555c35ee650 msgr2=0x5555c35f4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:58.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.869+0000 7fd158e1b740 1 -- 172.21.15.98:0/1616182600 shutdown_connections 2024-05-09T08:54:58.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.869+0000 7fd158e1b740 1 -- 172.21.15.98:0/1616182600 wait complete. 2024-05-09T08:54:58.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.869+0000 7fd158e1b740 20 client.26095 trim_cache size 0 max 16384 2024-05-09T08:54:58.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.898+0000 7fd158e1b740 1 Processor -- start 2024-05-09T08:54:58.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.898+0000 7fd158e1b740 1 -- start start 2024-05-09T08:54:58.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.898+0000 7fd158e1b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2330 0x5555c35fca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.899+0000 7fd158e1b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35f2c50 0x5555c35fabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.899+0000 7fd158e1b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35fd320 0x5555c35f4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.899+0000 7fd158e1b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1501a6720 con 0x5555c35f2c50 2024-05-09T08:54:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.899+0000 7fd14f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2330 0x5555c35fca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.899+0000 7fd14f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2330 0x5555c35fca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55014/0 (socket says 172.21.15.98:55014) 2024-05-09T08:54:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.899+0000 7fd14f7fe640 1 -- 172.21.15.98:0/1380731675 learned_addr learned my addr 172.21.15.98:0/1380731675 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.899+0000 7fd14ffff640 1 --2- 172.21.15.98:0/1380731675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35fd320 0x5555c35f4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.899+0000 7fd158e1b740 1 -- 172.21.15.98:0/1380731675 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd1501a4510 con 0x5555c35f2330 2024-05-09T08:54:58.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.899+0000 7fd158e1b740 1 -- 172.21.15.98:0/1380731675 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd1501a6050 con 0x5555c35fd320 2024-05-09T08:54:58.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.902+0000 7fd14ffff640 1 -- 172.21.15.98:0/1380731675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2330 msgr2=0x5555c35fca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:58.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.902+0000 7fd14ffff640 1 --2- 172.21.15.98:0/1380731675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2330 0x5555c35fca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.902+0000 7fd14ffff640 1 -- 172.21.15.98:0/1380731675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35f2c50 msgr2=0x5555c35fabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:58.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.902+0000 7fd14ffff640 1 --2- 172.21.15.98:0/1380731675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35f2c50 0x5555c35fabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.902+0000 7fd14ffff640 1 -- 172.21.15.98:0/1380731675 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555c35dfd20 con 0x5555c35fd320 2024-05-09T08:54:58.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.903+0000 7fd14ffff640 1 --2- 172.21.15.98:0/1380731675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35fd320 0x5555c35f4a90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd148011ca0 tx=0x7fd148014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a25e68f73f07074 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:58.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.904+0000 7fd1457fa640 1 -- 172.21.15.98:0/1380731675 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148013be0 con 0x5555c35fd320 2024-05-09T08:54:58.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.904+0000 7fd1457fa640 1 -- 172.21.15.98:0/1380731675 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd148011e50 con 0x5555c35fd320 2024-05-09T08:54:58.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.904+0000 7fd1457fa640 1 -- 172.21.15.98:0/1380731675 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14801d680 con 0x5555c35fd320 2024-05-09T08:54:58.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.905+0000 7fd158e1b740 1 -- 172.21.15.98:0/1380731675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35fd320 msgr2=0x5555c35f4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:58.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.905+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1380731675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35fd320 0x5555c35f4a90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd148011ca0 tx=0x7fd148014000 comp rx=0 tx=0).stop 2024-05-09T08:54:58.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.915+0000 7fd158e1b740 1 -- 172.21.15.98:0/1380731675 shutdown_connections 2024-05-09T08:54:58.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.915+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1380731675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35fd320 0x5555c35f4a90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.915+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1380731675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35f2c50 0x5555c35fabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.915+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1380731675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2330 0x5555c35fca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.915+0000 7fd158e1b740 1 -- 172.21.15.98:0/1380731675 >> 172.21.15.98:0/1380731675 conn(0x5555c35ee650 msgr2=0x5555c35ed7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:58.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.915+0000 7fd158e1b740 1 -- 172.21.15.98:0/1380731675 shutdown_connections 2024-05-09T08:54:58.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.916+0000 7fd158e1b740 1 -- 172.21.15.98:0/1380731675 wait complete. 2024-05-09T08:54:58.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.916+0000 7fd158e1b740 1 -- start start 2024-05-09T08:54:58.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.917+0000 7fd158e1b740 1 Processor -- start 2024-05-09T08:54:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.917+0000 7fd158e1b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2330 0x5555c36056d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.918+0000 7fd158e1b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2c50 0x5555c35f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.918+0000 7fd14f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2330 0x5555c36056d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.918+0000 7fd14f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2330 0x5555c36056d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:50972/0 (socket says 172.21.15.98:50972) 2024-05-09T08:54:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.918+0000 7fd14f7fe640 1 -- 172.21.15.98:0/1855480962 learned_addr learned my addr 172.21.15.98:0/1855480962 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.918+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1855480962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 0x5555c35f7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.918+0000 7fd158e1b740 1 -- 172.21.15.98:0/1855480962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1501a52b0 con 0x5555c35fd320 2024-05-09T08:54:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.918+0000 7fd158e1b740 1 -- 172.21.15.98:0/1855480962 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd1501a4be0 con 0x5555c35f2c50 2024-05-09T08:54:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.918+0000 7fd158e1b740 1 -- 172.21.15.98:0/1855480962 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd1501a5980 con 0x5555c35f2330 2024-05-09T08:54:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.919+0000 7fd158e1b740 10 client.0 _ll_register_callbacks cb 0x7fd140097170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.919+0000 7fd14effd640 1 --2- 172.21.15.98:0/1855480962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2c50 0x5555c35f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.919+0000 7fd14ffff640 1 --2- 172.21.15.98:0/1855480962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 0x5555c35f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.919+0000 7fd14ffff640 1 -- 172.21.15.98:0/1855480962 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2330 msgr2=0x5555c36056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:58.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.919+0000 7fd14ffff640 1 --2- 172.21.15.98:0/1855480962 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2330 0x5555c36056d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.919+0000 7fd14ffff640 1 -- 172.21.15.98:0/1855480962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2c50 msgr2=0x5555c35f6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:58.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.919+0000 7fd14ffff640 1 --2- 172.21.15.98:0/1855480962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2c50 0x5555c35f6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.919+0000 7fd14ffff640 1 -- 172.21.15.98:0/1855480962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555c35f7640 con 0x5555c35fd320 2024-05-09T08:54:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.920+0000 7fd14ffff640 1 --2- 172.21.15.98:0/1855480962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 0x5555c35f7100 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x5555c35ea2a0 tx=0x7fd148027670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b157151cd5ae87 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.920+0000 7fd12effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:54:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.920+0000 7fd12effd640 1 -- 172.21.15.98:0/1855480962 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14801d620 con 0x5555c35fd320 2024-05-09T08:54:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.920+0000 7fd158e1b740 1 -- 172.21.15.98:0/1855480962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd148011e50 con 0x5555c35fd320 2024-05-09T08:54:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.921+0000 7fd12effd640 1 -- 172.21.15.98:0/1855480962 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fd14801d7c0 con 0x5555c35fd320 2024-05-09T08:54:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.921+0000 7fd12effd640 1 -- 172.21.15.98:0/1855480962 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd14803f050 con 0x5555c35fd320 2024-05-09T08:54:58.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.922+0000 7fd158e1b740 10 client.16839 fetch_fsmap learned FSMap version 35 2024-05-09T08:54:58.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.922+0000 7fd158e1b740 1 -- 172.21.15.98:0/1855480962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5555c3684ba0 con 0x5555c35fd320 2024-05-09T08:54:58.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.926+0000 7fd12effd640 1 -- 172.21.15.98:0/1855480962 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd148011e50 con 0x5555c35fd320 2024-05-09T08:54:58.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.926+0000 7fd12effd640 1 -- 172.21.15.98:0/1855480962 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd148025020 con 0x5555c35fd320 2024-05-09T08:54:58.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.929+0000 7fd158e1b740 10 client.16839 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:54:58.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.929+0000 7fd158e1b740 10 client.16839 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:54:58.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.929+0000 7fd158e1b740 20 client.16839 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:58.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.929+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1855480962 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5555c3685190 0x5555c36875b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:58.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.929+0000 7fd158e1b740 4 client.16839 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:54:58.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.929+0000 7fd158e1b740 1 -- 172.21.15.98:0/1855480962 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5555c355f6c0 con 0x5555c3685190 2024-05-09T08:54:58.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.930+0000 7fd14f7fe640 1 --2- 172.21.15.98:0/1855480962 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5555c3685190 0x5555c36875b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:58.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.930+0000 7fd14f7fe640 1 --2- 172.21.15.98:0/1855480962 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5555c3685190 0x5555c36875b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=38418472c9b7df3f server_cookie=8f6c75e5f5b12525 in_seq=0 out_seq=0 2024-05-09T08:54:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.931+0000 7fd12effd640 10 client.16839 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:54:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.931+0000 7fd12effd640 1 -- 172.21.15.98:0/1855480962 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5555c355f6c0 con 0x5555c3685190 2024-05-09T08:54:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.931+0000 7fd12effd640 10 client.16839 handle_command_reply: tid=0 2024-05-09T08:54:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.932+0000 7fd158e1b740 1 client.16839 shutdown 2024-05-09T08:54:58.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.932+0000 7fd158e1b740 1 -- 172.21.15.98:0/1855480962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 msgr2=0x5555c35f7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.932+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1855480962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 0x5555c35f7100 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x5555c35ea2a0 tx=0x7fd148027670 comp rx=0 tx=0).stop 2024-05-09T08:54:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.932+0000 7fd158e1b740 1 -- 172.21.15.98:0/1855480962 shutdown_connections 2024-05-09T08:54:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.932+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1855480962 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5555c3685190 0x5555c36875b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.932+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1855480962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5555c35fd320 0x5555c35f7100 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.932+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1855480962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5555c35f2c50 0x5555c35f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.932+0000 7fd158e1b740 1 --2- 172.21.15.98:0/1855480962 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555c35f2330 0x5555c36056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.932+0000 7fd158e1b740 1 -- 172.21.15.98:0/1855480962 >> 172.21.15.98:0/1855480962 conn(0x5555c35ee650 msgr2=0x5555c3607fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.933+0000 7fd12effd640 0 client.16839 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:54:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.933+0000 7fd158e1b740 1 -- 172.21.15.98:0/1855480962 shutdown_connections 2024-05-09T08:54:58.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.934+0000 7fd158e1b740 1 -- 172.21.15.98:0/1855480962 wait complete. 2024-05-09T08:54:58.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.934+0000 7fd158e1b740 20 client.16839 trim_cache size 0 max 16384 2024-05-09T08:54:58.936 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:54:58.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.940+0000 7fd156116640 1 -- 172.21.15.98:0/3166081469 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd13c0a2ed0 msgr2=0x7fd13c0a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:58.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.941+0000 7fd156116640 1 --2- 172.21.15.98:0/3166081469 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd13c0a2ed0 0x7fd13c0a5390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd150073fc0 tx=0x7fd1480046c0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.941+0000 7fd156116640 1 -- 172.21.15.98:0/3166081469 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1501027f0 msgr2=0x7fd150104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:58.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.941+0000 7fd156116640 1 --2- 172.21.15.98:0/3166081469 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1501027f0 0x7fd150104b80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd14000ba50 tx=0x7fd14000bf10 comp rx=0 tx=0).stop 2024-05-09T08:54:58.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.942+0000 7fd156116640 1 -- 172.21.15.98:0/3166081469 shutdown_connections 2024-05-09T08:54:58.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.942+0000 7fd156116640 1 --2- 172.21.15.98:0/3166081469 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd13c0a2ed0 0x7fd13c0a5390 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.942+0000 7fd156116640 1 --2- 172.21.15.98:0/3166081469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15010e4c0 0x7fd150105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.942+0000 7fd156116640 1 --2- 172.21.15.98:0/3166081469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd150102c60 0x7fd1501050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.942+0000 7fd156116640 1 --2- 172.21.15.98:0/3166081469 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1501027f0 0x7fd150104b80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:58.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.942+0000 7fd156116640 1 -- 172.21.15.98:0/3166081469 >> 172.21.15.98:0/3166081469 conn(0x7fd150076cd0 msgr2=0x7fd15010f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:58.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.943+0000 7fd156116640 1 -- 172.21.15.98:0/3166081469 shutdown_connections 2024-05-09T08:54:58.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:58.943+0000 7fd156116640 1 -- 172.21.15.98:0/3166081469 wait complete. 2024-05-09T08:54:58.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:54:59.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.288+0000 7fcb8cbc4640 1 -- 172.21.15.98:0/1897024240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb88110080 msgr2=0x7fcb8810db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:59.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.288+0000 7fcb8cbc4640 1 --2- 172.21.15.98:0/1897024240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb88110080 0x7fcb8810db70 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fcb7801a940 tx=0x7fcb7803b490 comp rx=0 tx=0).stop 2024-05-09T08:54:59.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.291+0000 7fcb8cbc4640 1 -- 172.21.15.98:0/1897024240 shutdown_connections 2024-05-09T08:54:59.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.291+0000 7fcb8cbc4640 1 --2- 172.21.15.98:0/1897024240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb8811b010 0x7fcb8811d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.291+0000 7fcb8cbc4640 1 --2- 172.21.15.98:0/1897024240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb8810e0b0 0x7fcb8810e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.291+0000 7fcb8cbc4640 1 --2- 172.21.15.98:0/1897024240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb88110080 0x7fcb8810db70 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.291+0000 7fcb8cbc4640 1 -- 172.21.15.98:0/1897024240 >> 172.21.15.98:0/1897024240 conn(0x7fcb8806c940 msgr2=0x7fcb8806cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:59.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.297+0000 7fcb8cbc4640 1 -- 172.21.15.98:0/1897024240 shutdown_connections 2024-05-09T08:54:59.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.297+0000 7fcb8cbc4640 1 -- 172.21.15.98:0/1897024240 wait complete. 2024-05-09T08:54:59.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.298+0000 7fcb8cbc4640 1 Processor -- start 2024-05-09T08:54:59.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.298+0000 7fcb8cbc4640 1 -- start start 2024-05-09T08:54:59.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.298+0000 7fcb8cbc4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb8810e0b0 0x7fcb8811a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.298+0000 7fcb8cbc4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb88110080 0x7fcb881168c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.298+0000 7fcb8cbc4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb8811b010 0x7fcb88116e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.298+0000 7fcb8cbc4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb88113210 con 0x7fcb88110080 2024-05-09T08:54:59.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.298+0000 7fcb8cbc4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcb88113090 con 0x7fcb8810e0b0 2024-05-09T08:54:59.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.298+0000 7fcb8cbc4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcb88113390 con 0x7fcb8811b010 2024-05-09T08:54:59.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.301+0000 7fcb85d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb88110080 0x7fcb881168c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.301+0000 7fcb85d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb88110080 0x7fcb881168c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37292/0 (socket says 172.21.15.98:37292) 2024-05-09T08:54:59.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.301+0000 7fcb85d9b640 1 -- 172.21.15.98:0/3314332002 learned_addr learned my addr 172.21.15.98:0/3314332002 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:59.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.301+0000 7fcb8659c640 1 --2- 172.21.15.98:0/3314332002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb8810e0b0 0x7fcb8811a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.301+0000 7fcb86d9d640 1 --2- 172.21.15.98:0/3314332002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb8811b010 0x7fcb88116e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.302+0000 7fcb85d9b640 1 -- 172.21.15.98:0/3314332002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb8811b010 msgr2=0x7fcb88116e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:59.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.302+0000 7fcb85d9b640 1 --2- 172.21.15.98:0/3314332002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb8811b010 0x7fcb88116e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.302+0000 7fcb85d9b640 1 -- 172.21.15.98:0/3314332002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb8810e0b0 msgr2=0x7fcb8811a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:59.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.302+0000 7fcb85d9b640 1 --2- 172.21.15.98:0/3314332002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb8810e0b0 0x7fcb8811a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.302+0000 7fcb85d9b640 1 -- 172.21.15.98:0/3314332002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb881176c0 con 0x7fcb88110080 2024-05-09T08:54:59.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.303+0000 7fcb85d9b640 1 --2- 172.21.15.98:0/3314332002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb88110080 0x7fcb881168c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000eab0 tx=0x7fcb8000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:59.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.303+0000 7fcb777fe640 1 -- 172.21.15.98:0/3314332002 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8000ce40 con 0x7fcb88110080 2024-05-09T08:54:59.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.303+0000 7fcb8cbc4640 1 -- 172.21.15.98:0/3314332002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb881aada0 con 0x7fcb88110080 2024-05-09T08:54:59.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.303+0000 7fcb777fe640 1 -- 172.21.15.98:0/3314332002 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb80011950 con 0x7fcb88110080 2024-05-09T08:54:59.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.303+0000 7fcb777fe640 1 -- 172.21.15.98:0/3314332002 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80010b40 con 0x7fcb88110080 2024-05-09T08:54:59.309 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:54:59 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2381958932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:54:59.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.314+0000 7fcb8cbc4640 1 -- 172.21.15.98:0/3314332002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb881ab1b0 con 0x7fcb88110080 2024-05-09T08:54:59.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.327+0000 7fcb8f9a8740 1 Processor -- start 2024-05-09T08:54:59.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.331+0000 7fcb777fe640 1 -- 172.21.15.98:0/3314332002 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcb80010ce0 con 0x7fcb88110080 2024-05-09T08:54:59.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.332+0000 7fcb777fe640 1 --2- 172.21.15.98:0/3314332002 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcb700a2ed0 0x7fcb700a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.332+0000 7fcb777fe640 1 -- 172.21.15.98:0/3314332002 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fcb800d3170 con 0x7fcb88110080 2024-05-09T08:54:59.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.333+0000 7fcb8f9a8740 1 -- start start 2024-05-09T08:54:59.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.333+0000 7fcb8f9a8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883330 0x557ec1883710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.333+0000 7fcb8f9a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883c50 0x557ec188dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.333+0000 7fcb8f9a8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec188e320 0x557ec1890710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.333+0000 7fcb8f9a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557ec17e8c40 con 0x557ec1883c50 2024-05-09T08:54:59.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.333+0000 7fcb8f9a8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557ec1790580 con 0x557ec188e320 2024-05-09T08:54:59.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.333+0000 7fcb8f9a8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557ec17e4c00 con 0x557ec1883330 2024-05-09T08:54:59.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.334+0000 7fcb86d9d640 1 --2- 172.21.15.98:0/3314332002 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcb700a2ed0 0x7fcb700a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.335+0000 7fcb85d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883c50 0x557ec188dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.335+0000 7fcb85d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883c50 0x557ec188dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37300/0 (socket says 172.21.15.98:37300) 2024-05-09T08:54:59.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.335+0000 7fcb85d9b640 1 -- 172.21.15.98:0/439362367 learned_addr learned my addr 172.21.15.98:0/439362367 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:59.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.336+0000 7fcb85d9b640 1 -- 172.21.15.98:0/439362367 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883330 msgr2=0x557ec1883710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:59.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.336+0000 7fcb85d9b640 1 --2- 172.21.15.98:0/439362367 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883330 0x557ec1883710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.336+0000 7fcb85d9b640 1 -- 172.21.15.98:0/439362367 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec188e320 msgr2=0x557ec1890710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:59.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.336+0000 7fcb85d9b640 1 --2- 172.21.15.98:0/439362367 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec188e320 0x557ec1890710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.336+0000 7fcb85d9b640 1 -- 172.21.15.98:0/439362367 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ec14fc3f0 con 0x557ec1883c50 2024-05-09T08:54:59.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.337+0000 7fcb85d9b640 1 --2- 172.21.15.98:0/439362367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883c50 0x557ec188dde0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fcb8009c170 tx=0x7fcb8009c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74c67f5f54ba7e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:59.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.337+0000 7fcb74ff9640 1 -- 172.21.15.98:0/439362367 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb800d4060 con 0x557ec1883c50 2024-05-09T08:54:59.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.337+0000 7fcb74ff9640 1 -- 172.21.15.98:0/439362367 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb8009c1e0 con 0x557ec1883c50 2024-05-09T08:54:59.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:54:59 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2381958932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:54:59.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.339+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/439362367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883c50 msgr2=0x557ec188dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:59.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.339+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/439362367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883c50 0x557ec188dde0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fcb8009c170 tx=0x7fcb8009c070 comp rx=0 tx=0).stop 2024-05-09T08:54:59.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.341+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/439362367 shutdown_connections 2024-05-09T08:54:59.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.341+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/439362367 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec188e320 0x557ec1890710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.341+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/439362367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883c50 0x557ec188dde0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.341+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/439362367 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883330 0x557ec1883710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.341+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/439362367 >> 172.21.15.98:0/439362367 conn(0x557ec187f650 msgr2=0x557ec187fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:59.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.341+0000 7fcb86d9d640 1 --2- 172.21.15.98:0/3314332002 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcb700a2ed0 0x7fcb700a5390 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x557ec18807e0 tx=0x7fcb7c0094b0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:59.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.343+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/439362367 shutdown_connections 2024-05-09T08:54:59.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.345+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/439362367 wait complete. 2024-05-09T08:54:59.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.345+0000 7fcb8f9a8740 1 -- start start 2024-05-09T08:54:59.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.345+0000 7fcb8f9a8740 1 Processor -- start 2024-05-09T08:54:59.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.346+0000 7fcb8f9a8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883330 0x557ec188b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.346+0000 7fcb8f9a8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec1883c50 0x557ec188b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.346+0000 7fcb8f9a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec188e320 0x557ec188bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.346+0000 7fcb8f9a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557ec1891100 con 0x557ec188e320 2024-05-09T08:54:59.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.346+0000 7fcb8f9a8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557ec17a3c40 con 0x557ec1883c50 2024-05-09T08:54:59.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.346+0000 7fcb8f9a8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557ec1891280 con 0x557ec1883330 2024-05-09T08:54:59.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.346+0000 7fcb8f9a8740 10 client.0 _ll_register_callbacks cb 0x557ec14fbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:59.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.348+0000 7fcb8659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883330 0x557ec188b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.348+0000 7fcb86d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec188e320 0x557ec188bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.348+0000 7fcb86d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec188e320 0x557ec188bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37310/0 (socket says 172.21.15.98:37310) 2024-05-09T08:54:59.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.348+0000 7fcb86d9d640 1 -- 172.21.15.98:0/117119993 learned_addr learned my addr 172.21.15.98:0/117119993 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:59.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.350+0000 7fcb86d9d640 1 -- 172.21.15.98:0/117119993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883330 msgr2=0x557ec188b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:59.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.350+0000 7fcb86d9d640 1 --2- 172.21.15.98:0/117119993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883330 0x557ec188b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.350+0000 7fcb86d9d640 1 -- 172.21.15.98:0/117119993 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec1883c50 msgr2=0x557ec188b720 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:54:59.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.350+0000 7fcb86d9d640 1 --2- 172.21.15.98:0/117119993 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec1883c50 0x557ec188b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.350+0000 7fcb86d9d640 1 -- 172.21.15.98:0/117119993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ec15c67d0 con 0x557ec188e320 2024-05-09T08:54:59.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.353+0000 7fcb86d9d640 1 --2- 172.21.15.98:0/117119993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec188e320 0x557ec188bc60 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0166b0 tx=0x7fcb7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9378f949888b0e8f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:59.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.353+0000 7fcb6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:54:59.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.354+0000 7fcb6dffb640 1 -- 172.21.15.98:0/117119993 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c0148d0 con 0x557ec188e320 2024-05-09T08:54:59.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.354+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/117119993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb8009c1e0 con 0x557ec188e320 2024-05-09T08:54:59.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.354+0000 7fcb6dffb640 1 -- 172.21.15.98:0/117119993 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fcb7c014a70 con 0x557ec188e320 2024-05-09T08:54:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.356+0000 7fcb6dffb640 1 -- 172.21.15.98:0/117119993 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcb7c02e050 con 0x557ec188e320 2024-05-09T08:54:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.357+0000 7fcb8f9a8740 10 client.16851 fetch_fsmap learned FSMap version 35 2024-05-09T08:54:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.357+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/117119993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557ec18bf340 con 0x557ec188e320 2024-05-09T08:54:59.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.366+0000 7fcb6dffb640 1 -- 172.21.15.98:0/117119993 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb8009c1e0 con 0x557ec188e320 2024-05-09T08:54:59.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.366+0000 7fcb6dffb640 1 -- 172.21.15.98:0/117119993 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fcb7c029030 con 0x557ec188e320 2024-05-09T08:54:59.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.366+0000 7fcb8f9a8740 10 client.16851 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:54:59.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.366+0000 7fcb8f9a8740 10 client.16851 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:54:59.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.366+0000 7fcb8f9a8740 20 client.16851 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:59.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.366+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/117119993 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x557ec18c0c70 0x557ec18c30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.366+0000 7fcb8f9a8740 4 client.16851 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:59.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.366+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/117119993 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ec17f6370 con 0x557ec18c0c70 2024-05-09T08:54:59.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.367+0000 7fcb8659c640 1 --2- 172.21.15.98:0/117119993 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x557ec18c0c70 0x557ec18c30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.373+0000 7fcb8659c640 1 --2- 172.21.15.98:0/117119993 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x557ec18c0c70 0x557ec18c30d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4fe8ef8d4bebfff8 server_cookie=9cbdf9d585c4e489 in_seq=0 out_seq=0 2024-05-09T08:54:59.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.373+0000 7fcb6dffb640 10 client.16851 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:54:59.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.375+0000 7fcb6dffb640 1 -- 172.21.15.98:0/117119993 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557ec17f6370 con 0x557ec18c0c70 2024-05-09T08:54:59.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.375+0000 7fcb6dffb640 10 client.16851 handle_command_reply: tid=0 2024-05-09T08:54:59.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.375+0000 7fcb8f9a8740 1 client.16851 shutdown 2024-05-09T08:54:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.380+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/117119993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec188e320 msgr2=0x557ec188bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.380+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/117119993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec188e320 0x557ec188bc60 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0166b0 tx=0x7fcb7c004750 comp rx=0 tx=0).stop 2024-05-09T08:54:59.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.393+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/117119993 shutdown_connections 2024-05-09T08:54:59.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.393+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/117119993 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x557ec18c0c70 0x557ec18c30d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.393+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/117119993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec188e320 0x557ec188bc60 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.393+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/117119993 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec1883c50 0x557ec188b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.393+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/117119993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883330 0x557ec188b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.393+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/117119993 >> 172.21.15.98:0/117119993 conn(0x557ec187f650 msgr2=0x557ec1885450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:59.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.393+0000 7fcb6dffb640 0 client.16851 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:54:59.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.402+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/117119993 shutdown_connections 2024-05-09T08:54:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.406+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/117119993 wait complete. 2024-05-09T08:54:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.406+0000 7fcb8f9a8740 20 client.16851 trim_cache size 0 max 16384 2024-05-09T08:54:59.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.454+0000 7fcb8f9a8740 1 Processor -- start 2024-05-09T08:54:59.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.457+0000 7fcb8f9a8740 1 -- start start 2024-05-09T08:54:59.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.458+0000 7fcb8f9a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883330 0x557ec1886560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.458+0000 7fcb8f9a8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec1883c50 0x557ec1885a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.458+0000 7fcb8f9a8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec188e320 0x557ec188ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.458+0000 7fcb8f9a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8807dbe0 con 0x557ec1883330 2024-05-09T08:54:59.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.458+0000 7fcb8f9a8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8807f720 con 0x557ec1883c50 2024-05-09T08:54:59.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.458+0000 7fcb8f9a8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8807e2b0 con 0x557ec188e320 2024-05-09T08:54:59.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.459+0000 7fcb86d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec188e320 0x557ec188ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.459+0000 7fcb85d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec1883c50 0x557ec1885a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.459+0000 7fcb85d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec1883c50 0x557ec1885a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55068/0 (socket says 172.21.15.98:55068) 2024-05-09T08:54:59.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.459+0000 7fcb85d9b640 1 -- 172.21.15.98:0/3296461270 learned_addr learned my addr 172.21.15.98:0/3296461270 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:59.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.461+0000 7fcb86d9d640 1 -- 172.21.15.98:0/3296461270 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec1883c50 msgr2=0x557ec1885a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:59.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.461+0000 7fcb86d9d640 1 --2- 172.21.15.98:0/3296461270 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec1883c50 0x557ec1885a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.461+0000 7fcb86d9d640 1 -- 172.21.15.98:0/3296461270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883330 msgr2=0x557ec1886560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:59.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.461+0000 7fcb86d9d640 1 --2- 172.21.15.98:0/3296461270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883330 0x557ec1886560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.461+0000 7fcb86d9d640 1 -- 172.21.15.98:0/3296461270 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ec1883a80 con 0x557ec188e320 2024-05-09T08:54:59.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.467+0000 7fcb86d9d640 1 --2- 172.21.15.98:0/3296461270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec188e320 0x557ec188ba80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c020510 tx=0x7fcb7c009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ee6b26a7bfd7a3a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.471+0000 7fcb6ffff640 1 -- 172.21.15.98:0/3296461270 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c036070 con 0x557ec188e320 2024-05-09T08:54:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.471+0000 7fcb6ffff640 1 -- 172.21.15.98:0/3296461270 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb7c01d040 con 0x557ec188e320 2024-05-09T08:54:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.471+0000 7fcb6ffff640 1 -- 172.21.15.98:0/3296461270 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c021430 con 0x557ec188e320 2024-05-09T08:54:59.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.472+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/3296461270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec188e320 msgr2=0x557ec188ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:59.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.472+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/3296461270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec188e320 0x557ec188ba80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c020510 tx=0x7fcb7c009f80 comp rx=0 tx=0).stop 2024-05-09T08:54:59.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.476+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/3296461270 shutdown_connections 2024-05-09T08:54:59.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.476+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/3296461270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec188e320 0x557ec188ba80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.476+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/3296461270 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec1883c50 0x557ec1885a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.476+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/3296461270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883330 0x557ec1886560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.476+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/3296461270 >> 172.21.15.98:0/3296461270 conn(0x557ec187f650 msgr2=0x557ec188d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:59.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.481+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/3296461270 shutdown_connections 2024-05-09T08:54:59.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.481+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/3296461270 wait complete. 2024-05-09T08:54:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.482+0000 7fcb8f9a8740 1 -- start start 2024-05-09T08:54:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.482+0000 7fcb8f9a8740 1 Processor -- start 2024-05-09T08:54:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.483+0000 7fcb8f9a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883330 0x557ec1896710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.483+0000 7fcb8f9a8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883c50 0x557ec1887a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.483+0000 7fcb8f9a8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec188e320 0x557ec1887fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.483+0000 7fcb8f9a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8807f050 con 0x557ec1883330 2024-05-09T08:54:59.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.483+0000 7fcb8f9a8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8807e980 con 0x557ec188e320 2024-05-09T08:54:59.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.483+0000 7fcb8f9a8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcb88117e30 con 0x557ec1883c50 2024-05-09T08:54:59.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.483+0000 7fcb8f9a8740 10 client.0 _ll_register_callbacks cb 0x7fcb7c0096a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:59.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.485+0000 7fcb85d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883c50 0x557ec1887a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.485+0000 7fcb85d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883c50 0x557ec1887a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:51044/0 (socket says 172.21.15.98:51044) 2024-05-09T08:54:59.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.485+0000 7fcb85d9b640 1 -- 172.21.15.98:0/4149181648 learned_addr learned my addr 172.21.15.98:0/4149181648 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:59.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.486+0000 7fcb8659c640 1 --2- 172.21.15.98:0/4149181648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883330 0x557ec1896710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:54:59 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2381958932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:54:59.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.487+0000 7fcb85d9b640 1 -- 172.21.15.98:0/4149181648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec188e320 msgr2=0x557ec1887fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:54:59.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.487+0000 7fcb85d9b640 1 --2- 172.21.15.98:0/4149181648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec188e320 0x557ec1887fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.487+0000 7fcb85d9b640 1 -- 172.21.15.98:0/4149181648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883330 msgr2=0x557ec1896710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:59.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.487+0000 7fcb85d9b640 1 --2- 172.21.15.98:0/4149181648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883330 0x557ec1896710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.487+0000 7fcb85d9b640 1 -- 172.21.15.98:0/4149181648 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ec1888590 con 0x557ec1883c50 2024-05-09T08:54:59.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.490+0000 7fcb85d9b640 1 --2- 172.21.15.98:0/4149181648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883c50 0x557ec1887a90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000c940 tx=0x7fcb8000ea00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1ef5c8feb913555 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:59.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.490+0000 7fcb6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:54:59.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.491+0000 7fcb6dffb640 1 -- 172.21.15.98:0/4149181648 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8009c1e0 con 0x557ec1883c50 2024-05-09T08:54:59.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.491+0000 7fcb6dffb640 1 -- 172.21.15.98:0/4149181648 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fcb800a0120 con 0x557ec1883c50 2024-05-09T08:54:59.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.491+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/4149181648 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb7c01d040 con 0x557ec1883c50 2024-05-09T08:54:59.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.493+0000 7fcb6dffb640 1 -- 172.21.15.98:0/4149181648 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcb800a0590 con 0x557ec1883c50 2024-05-09T08:54:59.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.495+0000 7fcb8f9a8740 10 client.26116 fetch_fsmap learned FSMap version 35 2024-05-09T08:54:59.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.495+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/4149181648 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557ec1927070 con 0x557ec1883c50 2024-05-09T08:54:59.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.499+0000 7fcb6dffb640 1 -- 172.21.15.98:0/4149181648 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb7c01d040 con 0x557ec1883c50 2024-05-09T08:54:59.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.499+0000 7fcb6dffb640 1 -- 172.21.15.98:0/4149181648 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fcb8000f040 con 0x557ec1883c50 2024-05-09T08:54:59.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.499+0000 7fcb8f9a8740 10 client.26116 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:54:59.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.499+0000 7fcb8f9a8740 10 client.26116 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:54:59.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.499+0000 7fcb8f9a8740 20 client.26116 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:59.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.499+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/4149181648 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x557ec1927350 0x557ec1929740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.499+0000 7fcb8f9a8740 4 client.26116 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:54:59.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.499+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/4149181648 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557ec17f06c0 con 0x557ec1927350 2024-05-09T08:54:59.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.502+0000 7fcb8659c640 1 --2- 172.21.15.98:0/4149181648 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x557ec1927350 0x557ec1929740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.529+0000 7fcb8659c640 1 --2- 172.21.15.98:0/4149181648 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x557ec1927350 0x557ec1929740 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=95d593df9a6d12e server_cookie=7174fb95a88ff656 in_seq=0 out_seq=0 2024-05-09T08:54:59.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.529+0000 7fcb6dffb640 10 client.26116 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:54:59.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.536+0000 7fcb6dffb640 1 -- 172.21.15.98:0/4149181648 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557ec17f06c0 con 0x557ec1927350 2024-05-09T08:54:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.536+0000 7fcb6dffb640 10 client.26116 handle_command_reply: tid=0 2024-05-09T08:54:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.536+0000 7fcb8f9a8740 1 client.26116 shutdown 2024-05-09T08:54:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.536+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/4149181648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883c50 msgr2=0x557ec1887a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.536+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/4149181648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883c50 0x557ec1887a90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000c940 tx=0x7fcb8000ea00 comp rx=0 tx=0).stop 2024-05-09T08:54:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.536+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/4149181648 shutdown_connections 2024-05-09T08:54:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.536+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/4149181648 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x557ec1927350 0x557ec1929740 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.536+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/4149181648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ec188e320 0x557ec1887fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.536+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/4149181648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557ec1883c50 0x557ec1887a90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.536+0000 7fcb6dffb640 0 client.26116 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:54:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.536+0000 7fcb8f9a8740 1 --2- 172.21.15.98:0/4149181648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557ec1883330 0x557ec1896710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.536+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/4149181648 >> 172.21.15.98:0/4149181648 conn(0x557ec187f650 msgr2=0x557ec188c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:59.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.536+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/4149181648 shutdown_connections 2024-05-09T08:54:59.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.537+0000 7fcb8f9a8740 1 -- 172.21.15.98:0/4149181648 wait complete. 2024-05-09T08:54:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.540+0000 7fcb8f9a8740 20 client.26116 trim_cache size 0 max 16384 2024-05-09T08:54:59.544 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:54:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.550+0000 7fcb8cbc4640 1 -- 172.21.15.98:0/3314332002 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcb700a2ed0 msgr2=0x7fcb700a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.550+0000 7fcb8cbc4640 1 --2- 172.21.15.98:0/3314332002 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcb700a2ed0 0x7fcb700a5390 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x557ec18807e0 tx=0x7fcb7c0094b0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.550+0000 7fcb8cbc4640 1 -- 172.21.15.98:0/3314332002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb88110080 msgr2=0x7fcb881168c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.550+0000 7fcb8cbc4640 1 --2- 172.21.15.98:0/3314332002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb88110080 0x7fcb881168c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000eab0 tx=0x7fcb8000ef70 comp rx=0 tx=0).stop 2024-05-09T08:54:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.551+0000 7fcb8cbc4640 1 -- 172.21.15.98:0/3314332002 shutdown_connections 2024-05-09T08:54:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.551+0000 7fcb8cbc4640 1 --2- 172.21.15.98:0/3314332002 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcb700a2ed0 0x7fcb700a5390 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.551+0000 7fcb8cbc4640 1 --2- 172.21.15.98:0/3314332002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb8811b010 0x7fcb88116e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.551+0000 7fcb8cbc4640 1 --2- 172.21.15.98:0/3314332002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb88110080 0x7fcb881168c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.551+0000 7fcb8cbc4640 1 --2- 172.21.15.98:0/3314332002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb8810e0b0 0x7fcb8811a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.551+0000 7fcb8cbc4640 1 -- 172.21.15.98:0/3314332002 >> 172.21.15.98:0/3314332002 conn(0x7fcb8806c940 msgr2=0x7fcb8811bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.553+0000 7fcb8cbc4640 1 -- 172.21.15.98:0/3314332002 shutdown_connections 2024-05-09T08:54:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.553+0000 7fcb8cbc4640 1 -- 172.21.15.98:0/3314332002 wait complete. 2024-05-09T08:54:59.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:54:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.847+0000 7fd0b8493640 1 -- 172.21.15.98:0/1578265977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0b01080b0 msgr2=0x7fd0b0108490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.847+0000 7fd0b8493640 1 --2- 172.21.15.98:0/1578265977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0b01080b0 0x7fd0b0108490 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac009860 tx=0x7fd0ac02a360 comp rx=0 tx=0).stop 2024-05-09T08:54:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.855+0000 7fd0b8493640 1 -- 172.21.15.98:0/1578265977 shutdown_connections 2024-05-09T08:54:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.855+0000 7fd0b8493640 1 --2- 172.21.15.98:0/1578265977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0b010d130 0x7fd0b0114d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.855+0000 7fd0b8493640 1 --2- 172.21.15.98:0/1578265977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b0108a60 0x7fd0b010ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.855+0000 7fd0b8493640 1 --2- 172.21.15.98:0/1578265977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0b01080b0 0x7fd0b0108490 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.855+0000 7fd0b8493640 1 -- 172.21.15.98:0/1578265977 >> 172.21.15.98:0/1578265977 conn(0x7fd0b0075d70 msgr2=0x7fd0b0078190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.858+0000 7fd0b8493640 1 -- 172.21.15.98:0/1578265977 shutdown_connections 2024-05-09T08:54:59.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.860+0000 7fd0b8493640 1 -- 172.21.15.98:0/1578265977 wait complete. 2024-05-09T08:54:59.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.861+0000 7fd0b8493640 1 Processor -- start 2024-05-09T08:54:59.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.861+0000 7fd0b8493640 1 -- start start 2024-05-09T08:54:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.862+0000 7fd0b8493640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b01080b0 0x7fd0b0081ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.862+0000 7fd0b8493640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0b010d130 0x7fd0b007af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.862+0000 7fd0b8493640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0b007b4b0 0x7fd0b007b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.862+0000 7fd0b8493640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b0118210 con 0x7fd0b01080b0 2024-05-09T08:54:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.862+0000 7fd0b8493640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b0118090 con 0x7fd0b007b4b0 2024-05-09T08:54:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.862+0000 7fd0b8493640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b0118390 con 0x7fd0b010d130 2024-05-09T08:54:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.865+0000 7fd0b622f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b01080b0 0x7fd0b0081ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.865+0000 7fd0b622f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b01080b0 0x7fd0b0081ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37334/0 (socket says 172.21.15.98:37334) 2024-05-09T08:54:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.865+0000 7fd0b622f640 1 -- 172.21.15.98:0/1656368419 learned_addr learned my addr 172.21.15.98:0/1656368419 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.867+0000 7fd0b6a30640 1 --2- 172.21.15.98:0/1656368419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0b007b4b0 0x7fd0b007b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.867+0000 7fd0b5a2e640 1 --2- 172.21.15.98:0/1656368419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0b010d130 0x7fd0b007af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.869+0000 7fd0b622f640 1 -- 172.21.15.98:0/1656368419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0b010d130 msgr2=0x7fd0b007af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:59.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.869+0000 7fd0b622f640 1 --2- 172.21.15.98:0/1656368419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0b010d130 0x7fd0b007af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.869+0000 7fd0b622f640 1 -- 172.21.15.98:0/1656368419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0b007b4b0 msgr2=0x7fd0b007b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:59.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.870+0000 7fd0b622f640 1 --2- 172.21.15.98:0/1656368419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0b007b4b0 0x7fd0b007b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.870+0000 7fd0b622f640 1 -- 172.21.15.98:0/1656368419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b007fa10 con 0x7fd0b01080b0 2024-05-09T08:54:59.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.872+0000 7fd0b622f640 1 --2- 172.21.15.98:0/1656368419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b01080b0 0x7fd0b0081ef0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0058c0 tx=0x7fd0ac032650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:59.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.879+0000 7fd0a77fe640 1 -- 172.21.15.98:0/1656368419 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac037040 con 0x7fd0b01080b0 2024-05-09T08:54:59.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.879+0000 7fd0b8493640 1 -- 172.21.15.98:0/1656368419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b007fca0 con 0x7fd0b01080b0 2024-05-09T08:54:59.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.879+0000 7fd0a77fe640 1 -- 172.21.15.98:0/1656368419 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0ac040070 con 0x7fd0b01080b0 2024-05-09T08:54:59.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.879+0000 7fd0a77fe640 1 -- 172.21.15.98:0/1656368419 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac0052b0 con 0x7fd0b01080b0 2024-05-09T08:54:59.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.879+0000 7fd0b8493640 1 -- 172.21.15.98:0/1656368419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b0080190 con 0x7fd0b01080b0 2024-05-09T08:54:59.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.884+0000 7fd0bb19d740 1 Processor -- start 2024-05-09T08:54:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.888+0000 7fd0bb19d740 1 -- start start 2024-05-09T08:54:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.888+0000 7fd0bb19d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 0x5644ef526710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.888+0000 7fd0bb19d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef526c50 0x5644ef530de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.888+0000 7fd0bb19d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef531320 0x5644ef533710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.888+0000 7fd0bb19d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5644ef48bc40 con 0x5644ef526330 2024-05-09T08:54:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.888+0000 7fd0bb19d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5644ef433580 con 0x5644ef526c50 2024-05-09T08:54:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.888+0000 7fd0bb19d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5644ef487c00 con 0x5644ef531320 2024-05-09T08:54:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.890+0000 7fd0a77fe640 1 -- 172.21.15.98:0/1656368419 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd0ac005450 con 0x7fd0b01080b0 2024-05-09T08:54:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.890+0000 7fd0b622f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef526c50 0x5644ef530de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.890+0000 7fd0b622f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef526c50 0x5644ef530de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55098/0 (socket says 172.21.15.98:55098) 2024-05-09T08:54:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.890+0000 7fd0b622f640 1 -- 172.21.15.98:0/1782331540 learned_addr learned my addr 172.21.15.98:0/1782331540 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.891+0000 7fd0a77fe640 1 --2- 172.21.15.98:0/1656368419 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd09c0a2e80 0x7fd09c0a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.891+0000 7fd0a77fe640 1 -- 172.21.15.98:0/1656368419 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fd0ac0ef1c0 con 0x7fd0b01080b0 2024-05-09T08:54:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.892+0000 7fd0b5a2e640 1 --2- 172.21.15.98:0/1782331540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef531320 0x5644ef533710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.894+0000 7fd0b622f640 1 -- 172.21.15.98:0/1782331540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef531320 msgr2=0x5644ef533710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:59.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.894+0000 7fd0b622f640 1 --2- 172.21.15.98:0/1782331540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef531320 0x5644ef533710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.894+0000 7fd0b622f640 1 -- 172.21.15.98:0/1782331540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 msgr2=0x5644ef526710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:54:59.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.894+0000 7fd0b622f640 1 --2- 172.21.15.98:0/1782331540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 0x5644ef526710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.894+0000 7fd0b622f640 1 -- 172.21.15.98:0/1782331540 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644ef19f3f0 con 0x5644ef526c50 2024-05-09T08:54:59.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.895+0000 7fd0b622f640 1 --2- 172.21.15.98:0/1782331540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef526c50 0x5644ef530de0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac009830 tx=0x7fd0ac02a360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b87e8f5e1808d2f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:59.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.896+0000 7fd0b6a30640 1 --2- 172.21.15.98:0/1656368419 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd09c0a2e80 0x7fd09c0a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.896+0000 7fd0a4bfe640 1 -- 172.21.15.98:0/1782331540 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac0baf70 con 0x5644ef526c50 2024-05-09T08:54:59.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.896+0000 7fd0a4bfe640 1 -- 172.21.15.98:0/1782331540 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0ac0bbb30 con 0x5644ef526c50 2024-05-09T08:54:59.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.896+0000 7fd0a4bfe640 1 -- 172.21.15.98:0/1782331540 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac0bad90 con 0x5644ef526c50 2024-05-09T08:54:59.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.898+0000 7fd0bb19d740 1 -- 172.21.15.98:0/1782331540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef526c50 msgr2=0x5644ef530de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:59.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.898+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/1782331540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef526c50 0x5644ef530de0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac009830 tx=0x7fd0ac02a360 comp rx=0 tx=0).stop 2024-05-09T08:54:59.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.902+0000 7fd0bb19d740 1 -- 172.21.15.98:0/1782331540 shutdown_connections 2024-05-09T08:54:59.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.902+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/1782331540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef531320 0x5644ef533710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.902+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/1782331540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef526c50 0x5644ef530de0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.902+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/1782331540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 0x5644ef526710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.902+0000 7fd0bb19d740 1 -- 172.21.15.98:0/1782331540 >> 172.21.15.98:0/1782331540 conn(0x5644ef522650 msgr2=0x5644ef522a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:59.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.906+0000 7fd0b6a30640 1 --2- 172.21.15.98:0/1656368419 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd09c0a2e80 0x7fd09c0a5340 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8005cc0 tx=0x7fd0a8005c10 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:59.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.912+0000 7fd0bb19d740 1 -- 172.21.15.98:0/1782331540 shutdown_connections 2024-05-09T08:54:59.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.912+0000 7fd0bb19d740 1 -- 172.21.15.98:0/1782331540 wait complete. 2024-05-09T08:54:59.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.912+0000 7fd0bb19d740 1 -- start start 2024-05-09T08:54:59.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.912+0000 7fd0bb19d740 1 Processor -- start 2024-05-09T08:54:59.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.913+0000 7fd0bb19d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 0x5644ef52e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.913+0000 7fd0bb19d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526c50 0x5644ef52e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.913+0000 7fd0bb19d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 0x5644ef52ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.913+0000 7fd0bb19d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5644ef533ed0 con 0x5644ef526330 2024-05-09T08:54:59.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.913+0000 7fd0bb19d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5644ef446c40 con 0x5644ef531320 2024-05-09T08:54:59.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.913+0000 7fd0bb19d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5644ef534050 con 0x5644ef526c50 2024-05-09T08:54:59.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.913+0000 7fd0bb19d740 10 client.0 _ll_register_callbacks cb 0x5644ef19ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:54:59.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.914+0000 7fd0b6a30640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 0x5644ef52ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.914+0000 7fd0b622f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 0x5644ef52e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.914+0000 7fd0b622f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 0x5644ef52e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37350/0 (socket says 172.21.15.98:37350) 2024-05-09T08:54:59.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.914+0000 7fd0b622f640 1 -- 172.21.15.98:0/2986515835 learned_addr learned my addr 172.21.15.98:0/2986515835 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:59.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.915+0000 7fd0b6a30640 1 -- 172.21.15.98:0/2986515835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526c50 msgr2=0x5644ef52e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:59.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.915+0000 7fd0b6a30640 1 --2- 172.21.15.98:0/2986515835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526c50 0x5644ef52e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.915+0000 7fd0b6a30640 1 -- 172.21.15.98:0/2986515835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 msgr2=0x5644ef52e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:59.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.915+0000 7fd0b6a30640 1 --2- 172.21.15.98:0/2986515835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 0x5644ef52e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.915+0000 7fd0b6a30640 1 -- 172.21.15.98:0/2986515835 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644ef2697d0 con 0x5644ef531320 2024-05-09T08:54:59.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.916+0000 7fd0b6a30640 1 --2- 172.21.15.98:0/2986515835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 0x5644ef52ed40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800e7e0 tx=0x7fd0a8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74dbce5ac223f76b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:59.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.920+0000 7fd099ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:54:59.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.920+0000 7fd099ffb640 1 -- 172.21.15.98:0/2986515835 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8021070 con 0x5644ef531320 2024-05-09T08:54:59.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.920+0000 7fd0bb19d740 1 -- 172.21.15.98:0/2986515835 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0ac0bbb30 con 0x5644ef531320 2024-05-09T08:54:59.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.921+0000 7fd099ffb640 1 -- 172.21.15.98:0/2986515835 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fd0a800fcd0 con 0x5644ef531320 2024-05-09T08:54:59.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.922+0000 7fd099ffb640 1 -- 172.21.15.98:0/2986515835 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd0a802d050 con 0x5644ef531320 2024-05-09T08:54:59.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.926+0000 7fd0bb19d740 10 client.26165 fetch_fsmap learned FSMap version 35 2024-05-09T08:54:59.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.926+0000 7fd0bb19d740 1 -- 172.21.15.98:0/2986515835 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5644ef5400d0 con 0x5644ef531320 2024-05-09T08:54:59.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.927+0000 7fd099ffb640 1 -- 172.21.15.98:0/2986515835 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0ac0bbb30 con 0x5644ef531320 2024-05-09T08:54:59.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.927+0000 7fd099ffb640 1 -- 172.21.15.98:0/2986515835 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd0a8025530 con 0x5644ef531320 2024-05-09T08:54:59.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.927+0000 7fd0bb19d740 10 client.26165 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:54:59.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.927+0000 7fd0bb19d740 10 client.26165 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:54:59.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.927+0000 7fd0bb19d740 20 client.26165 populate_metadata read hostname 'smithi098' 2024-05-09T08:54:59.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.927+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/2986515835 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5644ef541a00 0x5644ef543e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.927+0000 7fd0bb19d740 4 client.26165 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:54:59.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.927+0000 7fd0bb19d740 1 -- 172.21.15.98:0/2986515835 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644ef499370 con 0x5644ef541a00 2024-05-09T08:54:59.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.929+0000 7fd0b622f640 1 --2- 172.21.15.98:0/2986515835 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5644ef541a00 0x5644ef543e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.932+0000 7fd0b622f640 1 --2- 172.21.15.98:0/2986515835 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5644ef541a00 0x5644ef543e80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a4eaf829e8ebaed9 server_cookie=dfcda64d6d8f30f5 in_seq=0 out_seq=0 2024-05-09T08:54:59.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.932+0000 7fd099ffb640 10 client.26165 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:54:59.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.934+0000 7fd099ffb640 1 -- 172.21.15.98:0/2986515835 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5644ef499370 con 0x5644ef541a00 2024-05-09T08:54:59.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.934+0000 7fd099ffb640 10 client.26165 handle_command_reply: tid=0 2024-05-09T08:54:59.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.934+0000 7fd0bb19d740 1 client.26165 shutdown 2024-05-09T08:54:59.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.935+0000 7fd0bb19d740 1 -- 172.21.15.98:0/2986515835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 msgr2=0x5644ef52ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:54:59.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.935+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/2986515835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 0x5644ef52ed40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800e7e0 tx=0x7fd0a8004a40 comp rx=0 tx=0).stop 2024-05-09T08:54:59.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.941+0000 7fd0bb19d740 1 -- 172.21.15.98:0/2986515835 shutdown_connections 2024-05-09T08:54:59.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.941+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/2986515835 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5644ef541a00 0x5644ef543e80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.941+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/2986515835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 0x5644ef52ed40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.941+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/2986515835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526c50 0x5644ef52e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.941+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/2986515835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 0x5644ef52e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.941+0000 7fd0bb19d740 1 -- 172.21.15.98:0/2986515835 >> 172.21.15.98:0/2986515835 conn(0x5644ef522650 msgr2=0x5644ef5217b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:54:59.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.941+0000 7fd099ffb640 0 client.26165 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:54:59.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.946+0000 7fd0bb19d740 1 -- 172.21.15.98:0/2986515835 shutdown_connections 2024-05-09T08:54:59.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.953+0000 7fd0bb19d740 1 -- 172.21.15.98:0/2986515835 wait complete. 2024-05-09T08:54:59.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.953+0000 7fd0bb19d740 20 client.26165 trim_cache size 0 max 16384 2024-05-09T08:54:59.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.996+0000 7fd0bb19d740 1 Processor -- start 2024-05-09T08:54:59.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.996+0000 7fd0bb19d740 1 -- start start 2024-05-09T08:54:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.997+0000 7fd0bb19d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526330 0x5644ef52cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.997+0000 7fd0bb19d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526c50 0x5644ef52eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.997+0000 7fd0bb19d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 0x5644ef53c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:54:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.997+0000 7fd0bb19d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd094010080 con 0x5644ef526c50 2024-05-09T08:54:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.997+0000 7fd0bb19d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd094011bc0 con 0x5644ef531320 2024-05-09T08:54:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.997+0000 7fd0bb19d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd094010750 con 0x5644ef526330 2024-05-09T08:54:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.998+0000 7fd0b622f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526330 0x5644ef52cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.998+0000 7fd0b622f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526330 0x5644ef52cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:51092/0 (socket says 172.21.15.98:51092) 2024-05-09T08:54:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.998+0000 7fd0b622f640 1 -- 172.21.15.98:0/2796766935 learned_addr learned my addr 172.21.15.98:0/2796766935 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:54:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.998+0000 7fd0b6a30640 1 --2- 172.21.15.98:0/2796766935 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 0x5644ef53c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:54:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.999+0000 7fd0b622f640 1 -- 172.21.15.98:0/2796766935 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 msgr2=0x5644ef53c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:54:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.999+0000 7fd0b622f640 1 --2- 172.21.15.98:0/2796766935 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 0x5644ef53c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.999+0000 7fd0b622f640 1 -- 172.21.15.98:0/2796766935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526c50 msgr2=0x5644ef52eb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:54:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.999+0000 7fd0b622f640 1 --2- 172.21.15.98:0/2796766935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526c50 0x5644ef52eb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:54:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.999+0000 7fd0b622f640 1 -- 172.21.15.98:0/2796766935 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644ef52d270 con 0x5644ef526330 2024-05-09T08:54:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:54:59.999+0000 7fd0b622f640 1 --2- 172.21.15.98:0/2796766935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526330 0x5644ef52cb90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0b7e00 tx=0x7fd0ac0b7040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e15ddb6e2c92999 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:54:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.000+0000 7fd09bfff640 1 -- 172.21.15.98:0/2796766935 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac0bad90 con 0x5644ef526330 2024-05-09T08:54:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.000+0000 7fd09bfff640 1 -- 172.21.15.98:0/2796766935 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0ac0bbb30 con 0x5644ef526330 2024-05-09T08:54:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.000+0000 7fd09bfff640 1 -- 172.21.15.98:0/2796766935 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac0baf70 con 0x5644ef526330 2024-05-09T08:55:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.003+0000 7fd0bb19d740 1 -- 172.21.15.98:0/2796766935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526330 msgr2=0x5644ef52cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.003+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/2796766935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526330 0x5644ef52cb90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0b7e00 tx=0x7fd0ac0b7040 comp rx=0 tx=0).stop 2024-05-09T08:55:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.009+0000 7fd0bb19d740 1 -- 172.21.15.98:0/2796766935 shutdown_connections 2024-05-09T08:55:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.009+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/2796766935 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 0x5644ef53c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.009+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/2796766935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526c50 0x5644ef52eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.009+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/2796766935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526330 0x5644ef52cb90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.009+0000 7fd0bb19d740 1 -- 172.21.15.98:0/2796766935 >> 172.21.15.98:0/2796766935 conn(0x5644ef522650 msgr2=0x5644ef5217b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.009+0000 7fd0bb19d740 1 -- 172.21.15.98:0/2796766935 shutdown_connections 2024-05-09T08:55:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.009+0000 7fd0bb19d740 1 -- 172.21.15.98:0/2796766935 wait complete. 2024-05-09T08:55:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.010+0000 7fd0bb19d740 1 -- start start 2024-05-09T08:55:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.010+0000 7fd0bb19d740 1 Processor -- start 2024-05-09T08:55:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.011+0000 7fd0bb19d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 0x5644ef534a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.011+0000 7fd0bb19d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526c50 0x5644ef534fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.011+0000 7fd0bb19d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 0x5644ef5354f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.011+0000 7fd0bb19d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd0940114f0 con 0x5644ef526330 2024-05-09T08:55:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.011+0000 7fd0bb19d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd094010e20 con 0x5644ef531320 2024-05-09T08:55:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.011+0000 7fd0bb19d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd09400e060 con 0x5644ef526c50 2024-05-09T08:55:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.012+0000 7fd0bb19d740 10 client.0 _ll_register_callbacks cb 0x7fd0a8014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.012+0000 7fd0b622f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 0x5644ef534a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.012+0000 7fd0b622f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 0x5644ef534a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37360/0 (socket says 172.21.15.98:37360) 2024-05-09T08:55:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.012+0000 7fd0b622f640 1 -- 172.21.15.98:0/3833709785 learned_addr learned my addr 172.21.15.98:0/3833709785 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.012+0000 7fd0b5a2e640 1 --2- 172.21.15.98:0/3833709785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526c50 0x5644ef534fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.013+0000 7fd0b622f640 1 -- 172.21.15.98:0/3833709785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526c50 msgr2=0x5644ef534fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.013+0000 7fd0b622f640 1 --2- 172.21.15.98:0/3833709785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526c50 0x5644ef534fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.013+0000 7fd0b622f640 1 -- 172.21.15.98:0/3833709785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 msgr2=0x5644ef5354f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.013+0000 7fd0b622f640 1 --2- 172.21.15.98:0/3833709785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 0x5644ef5354f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.013+0000 7fd0b622f640 1 -- 172.21.15.98:0/3833709785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644ef5a71d0 con 0x5644ef526330 2024-05-09T08:55:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.013+0000 7fd0b622f640 1 --2- 172.21.15.98:0/3833709785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 0x5644ef534a70 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0b7010 tx=0x7fd0ac0327c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c7808958755177 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.019+0000 7fd099ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:55:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.019+0000 7fd099ffb640 1 -- 172.21.15.98:0/3833709785 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac0bad90 con 0x5644ef526330 2024-05-09T08:55:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.019+0000 7fd099ffb640 1 -- 172.21.15.98:0/3833709785 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fd0ac0b9730 con 0x5644ef526330 2024-05-09T08:55:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.020+0000 7fd0bb19d740 1 -- 172.21.15.98:0/3833709785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0ac0bbb30 con 0x5644ef526330 2024-05-09T08:55:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.028+0000 7fd099ffb640 1 -- 172.21.15.98:0/3833709785 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd0ac0bb430 con 0x5644ef526330 2024-05-09T08:55:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.029+0000 7fd0bb19d740 10 client.16866 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.029+0000 7fd0bb19d740 1 -- 172.21.15.98:0/3833709785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5644ef5a8490 con 0x5644ef526330 2024-05-09T08:55:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.030+0000 7fd099ffb640 1 -- 172.21.15.98:0/3833709785 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0ac0bf7e0 con 0x5644ef526330 2024-05-09T08:55:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.030+0000 7fd099ffb640 1 -- 172.21.15.98:0/3833709785 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd0ac0caae0 con 0x5644ef526330 2024-05-09T08:55:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.030+0000 7fd0bb19d740 10 client.16866 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.030+0000 7fd0bb19d740 10 client.16866 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:55:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.030+0000 7fd0bb19d740 20 client.16866 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.030+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/3833709785 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5644ef5a8720 0x5644ef5aab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.030+0000 7fd0bb19d740 4 client.16866 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:55:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.030+0000 7fd0bb19d740 1 -- 172.21.15.98:0/3833709785 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fd0ac0bbb30 con 0x5644ef5a8720 2024-05-09T08:55:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.031+0000 7fd0b5a2e640 1 --2- 172.21.15.98:0/3833709785 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5644ef5a8720 0x5644ef5aab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.038+0000 7fd0b5a2e640 1 --2- 172.21.15.98:0/3833709785 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5644ef5a8720 0x5644ef5aab10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3cd73a953932316c server_cookie=5f264ce61a6d9654 in_seq=0 out_seq=0 2024-05-09T08:55:00.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.040+0000 7fd099ffb640 10 client.16866 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:55:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.043+0000 7fd099ffb640 1 -- 172.21.15.98:0/3833709785 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fd0ac0bbb30 con 0x5644ef5a8720 2024-05-09T08:55:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.043+0000 7fd099ffb640 10 client.16866 handle_command_reply: tid=0 2024-05-09T08:55:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.043+0000 7fd0bb19d740 1 client.16866 shutdown 2024-05-09T08:55:00.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.044+0000 7fd0bb19d740 1 -- 172.21.15.98:0/3833709785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 msgr2=0x5644ef534a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.044+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/3833709785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 0x5644ef534a70 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0b7010 tx=0x7fd0ac0327c0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.044+0000 7fd0bb19d740 1 -- 172.21.15.98:0/3833709785 shutdown_connections 2024-05-09T08:55:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.044+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/3833709785 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5644ef5a8720 0x5644ef5aab10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.044+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/3833709785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5644ef531320 0x5644ef5354f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.044+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/3833709785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644ef526c50 0x5644ef534fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.044+0000 7fd0bb19d740 1 --2- 172.21.15.98:0/3833709785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5644ef526330 0x5644ef534a70 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.044+0000 7fd0bb19d740 1 -- 172.21.15.98:0/3833709785 >> 172.21.15.98:0/3833709785 conn(0x5644ef522650 msgr2=0x5644ef5217b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.044+0000 7fd099ffb640 0 client.16866 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:55:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.046+0000 7fd0bb19d740 1 -- 172.21.15.98:0/3833709785 shutdown_connections 2024-05-09T08:55:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.046+0000 7fd0bb19d740 1 -- 172.21.15.98:0/3833709785 wait complete. 2024-05-09T08:55:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.047+0000 7fd0bb19d740 20 client.16866 trim_cache size 0 max 16384 2024-05-09T08:55:00.052 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:55:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.053+0000 7fd0b8493640 1 -- 172.21.15.98:0/1656368419 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd09c0a2e80 msgr2=0x7fd09c0a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.053+0000 7fd0b8493640 1 --2- 172.21.15.98:0/1656368419 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd09c0a2e80 0x7fd09c0a5340 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8005cc0 tx=0x7fd0a8005c10 comp rx=0 tx=0).stop 2024-05-09T08:55:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.053+0000 7fd0b8493640 1 -- 172.21.15.98:0/1656368419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b01080b0 msgr2=0x7fd0b0081ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.053+0000 7fd0b8493640 1 --2- 172.21.15.98:0/1656368419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b01080b0 0x7fd0b0081ef0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0058c0 tx=0x7fd0ac032650 comp rx=0 tx=0).stop 2024-05-09T08:55:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.053+0000 7fd0b8493640 1 -- 172.21.15.98:0/1656368419 shutdown_connections 2024-05-09T08:55:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.053+0000 7fd0b8493640 1 --2- 172.21.15.98:0/1656368419 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd09c0a2e80 0x7fd09c0a5340 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.053+0000 7fd0b8493640 1 --2- 172.21.15.98:0/1656368419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0b007b4b0 0x7fd0b007b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.053+0000 7fd0b8493640 1 --2- 172.21.15.98:0/1656368419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0b010d130 0x7fd0b007af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.053+0000 7fd0b8493640 1 --2- 172.21.15.98:0/1656368419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b01080b0 0x7fd0b0081ef0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.053+0000 7fd0b8493640 1 -- 172.21.15.98:0/1656368419 >> 172.21.15.98:0/1656368419 conn(0x7fd0b0075d70 msgr2=0x7fd0b00770d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.054+0000 7fd0b8493640 1 -- 172.21.15.98:0/1656368419 shutdown_connections 2024-05-09T08:55:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.054+0000 7fd0b8493640 1 -- 172.21.15.98:0/1656368419 wait complete. 2024-05-09T08:55:00.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:55:00.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:00 smithi098 ceph-mon[30092]: pgmap v135: 129 pgs: 129 active+clean; 30 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 704 MiB/s wr, 1.39k op/s 2024-05-09T08:55:00.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:00 smithi142 ceph-mon[34625]: pgmap v135: 129 pgs: 129 active+clean; 30 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 704 MiB/s wr, 1.39k op/s 2024-05-09T08:55:00.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:00 smithi105 ceph-mon[34498]: pgmap v135: 129 pgs: 129 active+clean; 30 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 704 MiB/s wr, 1.39k op/s 2024-05-09T08:55:00.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.517+0000 7fa5a63c1640 1 -- 172.21.15.98:0/3004855341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5a0110080 msgr2=0x7fa5a010db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:00.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.517+0000 7fa5a63c1640 1 --2- 172.21.15.98:0/3004855341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5a0110080 0x7fa5a010db70 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa598009cd0 tx=0x7fa59802a910 comp rx=0 tx=0).stop 2024-05-09T08:55:00.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.518+0000 7fa5a63c1640 1 -- 172.21.15.98:0/3004855341 shutdown_connections 2024-05-09T08:55:00.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.518+0000 7fa5a63c1640 1 --2- 172.21.15.98:0/3004855341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5a011b010 0x7fa5a011d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.518+0000 7fa5a63c1640 1 --2- 172.21.15.98:0/3004855341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5a010e0b0 0x7fa5a010e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.518+0000 7fa5a63c1640 1 --2- 172.21.15.98:0/3004855341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5a0110080 0x7fa5a010db70 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.518+0000 7fa5a63c1640 1 -- 172.21.15.98:0/3004855341 >> 172.21.15.98:0/3004855341 conn(0x7fa5a006c940 msgr2=0x7fa5a006cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:00.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.520+0000 7fa5a63c1640 1 -- 172.21.15.98:0/3004855341 shutdown_connections 2024-05-09T08:55:00.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.522+0000 7fa5a63c1640 1 -- 172.21.15.98:0/3004855341 wait complete. 2024-05-09T08:55:00.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.523+0000 7fa5a63c1640 1 Processor -- start 2024-05-09T08:55:00.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.523+0000 7fa5a63c1640 1 -- start start 2024-05-09T08:55:00.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.524+0000 7fa5a63c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5a010e0b0 0x7fa5a01a6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.524+0000 7fa5a63c1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5a0110080 0x7fa5a01a74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.524+0000 7fa5a63c1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5a011b010 0x7fa5a01ab820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.524+0000 7fa5a63c1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a01209e0 con 0x7fa5a010e0b0 2024-05-09T08:55:00.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.524+0000 7fa5a63c1640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a0120860 con 0x7fa5a011b010 2024-05-09T08:55:00.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.524+0000 7fa5a63c1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a0120b60 con 0x7fa5a0110080 2024-05-09T08:55:00.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.526+0000 7fa59f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5a0110080 0x7fa5a01a74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.526+0000 7fa5a495e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5a011b010 0x7fa5a01ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.527+0000 7fa59ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5a010e0b0 0x7fa5a01a6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.527+0000 7fa59ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5a010e0b0 0x7fa5a01a6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37368/0 (socket says 172.21.15.98:37368) 2024-05-09T08:55:00.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.527+0000 7fa59ffff640 1 -- 172.21.15.98:0/2120279519 learned_addr learned my addr 172.21.15.98:0/2120279519 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:00.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.532+0000 7fa5a495e640 1 -- 172.21.15.98:0/2120279519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5a0110080 msgr2=0x7fa5a01a74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:00.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.532+0000 7fa5a495e640 1 --2- 172.21.15.98:0/2120279519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5a0110080 0x7fa5a01a74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.532+0000 7fa5a495e640 1 -- 172.21.15.98:0/2120279519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5a010e0b0 msgr2=0x7fa5a01a6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:00.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.532+0000 7fa5a495e640 1 --2- 172.21.15.98:0/2120279519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5a010e0b0 0x7fa5a01a6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.532+0000 7fa5a495e640 1 -- 172.21.15.98:0/2120279519 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a01abf20 con 0x7fa5a011b010 2024-05-09T08:55:00.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.533+0000 7fa5a495e640 1 --2- 172.21.15.98:0/2120279519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5a011b010 0x7fa5a01ab820 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa59400e9a0 tx=0x7fa59400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:00.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.538+0000 7fa59d7fa640 1 -- 172.21.15.98:0/2120279519 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59400ccc0 con 0x7fa5a011b010 2024-05-09T08:55:00.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.538+0000 7fa59d7fa640 1 -- 172.21.15.98:0/2120279519 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa59400ce60 con 0x7fa5a011b010 2024-05-09T08:55:00.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.538+0000 7fa59d7fa640 1 -- 172.21.15.98:0/2120279519 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa594004a90 con 0x7fa5a011b010 2024-05-09T08:55:00.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.540+0000 7fa5a63c1640 1 -- 172.21.15.98:0/2120279519 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5a01ac210 con 0x7fa5a011b010 2024-05-09T08:55:00.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.540+0000 7fa5a63c1640 1 -- 172.21.15.98:0/2120279519 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5a01107f0 con 0x7fa5a011b010 2024-05-09T08:55:00.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.548+0000 7fa5a91ab740 1 Processor -- start 2024-05-09T08:55:00.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.553+0000 7fa5a91ab740 1 -- start start 2024-05-09T08:55:00.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.553+0000 7fa5a91ab740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017330 0x55eb96017710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.553+0000 7fa5a91ab740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017c50 0x55eb96021de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.554+0000 7fa5a91ab740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 0x55eb96024710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.554+0000 7fa5a91ab740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eb95f7cc40 con 0x55eb96022320 2024-05-09T08:55:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.554+0000 7fa59f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017330 0x55eb96017710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.554+0000 7fa59f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017330 0x55eb96017710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:51124/0 (socket says 172.21.15.98:51124) 2024-05-09T08:55:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.554+0000 7fa59f7fe640 1 -- 172.21.15.98:0/480301839 learned_addr learned my addr 172.21.15.98:0/480301839 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.554+0000 7fa59ffff640 1 --2- 172.21.15.98:0/480301839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017c50 0x55eb96021de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.554+0000 7fa5a91ab740 1 -- 172.21.15.98:0/480301839 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55eb95f24580 con 0x55eb96017c50 2024-05-09T08:55:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.554+0000 7fa5a91ab740 1 -- 172.21.15.98:0/480301839 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55eb95f78c00 con 0x55eb96017330 2024-05-09T08:55:00.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.559+0000 7fa59f7fe640 1 --2- 172.21.15.98:0/480301839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 0x55eb96024710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.560+0000 7fa59f7fe640 1 -- 172.21.15.98:0/480301839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017330 msgr2=0x55eb96017710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:00.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.560+0000 7fa59f7fe640 1 --2- 172.21.15.98:0/480301839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017330 0x55eb96017710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.560+0000 7fa59f7fe640 1 -- 172.21.15.98:0/480301839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017c50 msgr2=0x55eb96021de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:00.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.560+0000 7fa59f7fe640 1 --2- 172.21.15.98:0/480301839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017c50 0x55eb96021de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.560+0000 7fa59f7fe640 1 -- 172.21.15.98:0/480301839 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb95c903f0 con 0x55eb96022320 2024-05-09T08:55:00.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.562+0000 7fa59d7fa640 1 -- 172.21.15.98:0/2120279519 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa594004c30 con 0x7fa5a011b010 2024-05-09T08:55:00.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.562+0000 7fa59f7fe640 1 --2- 172.21.15.98:0/480301839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 0x55eb96024710 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa590012780 tx=0x7fa590005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad230856f980ce8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:00.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.563+0000 7fa59d7fa640 1 --2- 172.21.15.98:0/2120279519 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa5880a2d80 0x7fa5880a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.564+0000 7fa59d7fa640 1 -- 172.21.15.98:0/2120279519 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fa5940cf830 con 0x7fa5a011b010 2024-05-09T08:55:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.567+0000 7fa59ffff640 1 --2- 172.21.15.98:0/2120279519 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa5880a2d80 0x7fa5880a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.567+0000 7fa5867fc640 1 -- 172.21.15.98:0/480301839 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59000d740 con 0x55eb96022320 2024-05-09T08:55:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.567+0000 7fa5867fc640 1 -- 172.21.15.98:0/480301839 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa59000ddc0 con 0x55eb96022320 2024-05-09T08:55:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.568+0000 7fa5a91ab740 1 -- 172.21.15.98:0/480301839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 msgr2=0x55eb96024710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.568+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/480301839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 0x55eb96024710 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa590012780 tx=0x7fa590005f90 comp rx=0 tx=0).stop 2024-05-09T08:55:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.570+0000 7fa5a91ab740 1 -- 172.21.15.98:0/480301839 shutdown_connections 2024-05-09T08:55:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.570+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/480301839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 0x55eb96024710 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.570+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/480301839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017c50 0x55eb96021de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.570+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/480301839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017330 0x55eb96017710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.570+0000 7fa5a91ab740 1 -- 172.21.15.98:0/480301839 >> 172.21.15.98:0/480301839 conn(0x55eb96013650 msgr2=0x55eb96013a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.573+0000 7fa59ffff640 1 --2- 172.21.15.98:0/2120279519 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa5880a2d80 0x7fa5880a5240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55eb960188f0 tx=0x7fa598005d80 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.576+0000 7fa5a91ab740 1 -- 172.21.15.98:0/480301839 shutdown_connections 2024-05-09T08:55:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.579+0000 7fa5a91ab740 1 -- 172.21.15.98:0/480301839 wait complete. 2024-05-09T08:55:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.580+0000 7fa5a91ab740 1 -- start start 2024-05-09T08:55:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.580+0000 7fa5a91ab740 1 Processor -- start 2024-05-09T08:55:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.581+0000 7fa5a91ab740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017330 0x55eb96021dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.581+0000 7fa5a91ab740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017c50 0x55eb9601f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.581+0000 7fa5a91ab740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 0x55eb9601f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.581+0000 7fa5a91ab740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eb960251b0 con 0x55eb96022320 2024-05-09T08:55:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.581+0000 7fa5a91ab740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55eb95f37c40 con 0x55eb96017c50 2024-05-09T08:55:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.581+0000 7fa5a91ab740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55eb96025330 con 0x55eb96017330 2024-05-09T08:55:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.581+0000 7fa5a91ab740 10 client.0 _ll_register_callbacks cb 0x55eb95c8fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.581+0000 7fa59f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017c50 0x55eb9601f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.582+0000 7fa59f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017c50 0x55eb9601f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55160/0 (socket says 172.21.15.98:55160) 2024-05-09T08:55:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.582+0000 7fa59f7fe640 1 -- 172.21.15.98:0/1203325518 learned_addr learned my addr 172.21.15.98:0/1203325518 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.582+0000 7fa59ffff640 1 --2- 172.21.15.98:0/1203325518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017330 0x55eb96021dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.583+0000 7fa59ffff640 1 -- 172.21.15.98:0/1203325518 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017c50 msgr2=0x55eb9601f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.583+0000 7fa59ffff640 1 --2- 172.21.15.98:0/1203325518 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017c50 0x55eb9601f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.583+0000 7fa59ffff640 1 -- 172.21.15.98:0/1203325518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 msgr2=0x55eb9601f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.583+0000 7fa59ffff640 1 --2- 172.21.15.98:0/1203325518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 0x55eb9601f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.583+0000 7fa59ffff640 1 -- 172.21.15.98:0/1203325518 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb95d5a7d0 con 0x55eb96017330 2024-05-09T08:55:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.583+0000 7fa59ffff640 1 --2- 172.21.15.98:0/1203325518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017330 0x55eb96021dd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa598000f80 tx=0x7fa59802a910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33b1bbd8e481100d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.589+0000 7fa57f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:55:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.589+0000 7fa57f7fe640 1 -- 172.21.15.98:0/1203325518 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59802f620 con 0x55eb96017330 2024-05-09T08:55:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.589+0000 7fa57f7fe640 1 -- 172.21.15.98:0/1203325518 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fa59802f7c0 con 0x55eb96017330 2024-05-09T08:55:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.590+0000 7fa5a91ab740 1 -- 172.21.15.98:0/1203325518 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa59000ddc0 con 0x55eb96017330 2024-05-09T08:55:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.590+0000 7fa57f7fe640 1 -- 172.21.15.98:0/1203325518 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa59802f9c0 con 0x55eb96017330 2024-05-09T08:55:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.592+0000 7fa5a91ab740 10 client.26140 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.592+0000 7fa5a91ab740 1 -- 172.21.15.98:0/1203325518 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55eb96031300 con 0x55eb96017330 2024-05-09T08:55:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.593+0000 7fa57f7fe640 1 -- 172.21.15.98:0/1203325518 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa59000ddc0 con 0x55eb96017330 2024-05-09T08:55:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.593+0000 7fa57f7fe640 1 -- 172.21.15.98:0/1203325518 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fa598051070 con 0x55eb96017330 2024-05-09T08:55:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.593+0000 7fa5a91ab740 10 client.26140 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.593+0000 7fa5a91ab740 10 client.26140 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:55:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.593+0000 7fa5a91ab740 20 client.26140 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.593+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/1203325518 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb96032c10 0x55eb960350d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.593+0000 7fa5a91ab740 4 client.26140 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.593+0000 7fa5a91ab740 1 -- 172.21.15.98:0/1203325518 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb95f8a370 con 0x55eb96032c10 2024-05-09T08:55:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.594+0000 7fa59f7fe640 1 --2- 172.21.15.98:0/1203325518 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb96032c10 0x55eb960350d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.602+0000 7fa59f7fe640 1 --2- 172.21.15.98:0/1203325518 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb96032c10 0x55eb960350d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8c1d6d488f700bb9 server_cookie=ac4396c261b1d99d in_seq=0 out_seq=0 2024-05-09T08:55:00.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.604+0000 7fa57f7fe640 10 client.26140 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:55:00.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.608+0000 7fa57f7fe640 1 -- 172.21.15.98:0/1203325518 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb95f8a370 con 0x55eb96032c10 2024-05-09T08:55:00.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.608+0000 7fa57f7fe640 10 client.26140 handle_command_reply: tid=0 2024-05-09T08:55:00.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.608+0000 7fa5a91ab740 1 client.26140 shutdown 2024-05-09T08:55:00.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.615+0000 7fa5a91ab740 1 -- 172.21.15.98:0/1203325518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017330 msgr2=0x55eb96021dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:00.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.615+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/1203325518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017330 0x55eb96021dd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa598000f80 tx=0x7fa59802a910 comp rx=0 tx=0).stop 2024-05-09T08:55:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.618+0000 7fa5a91ab740 1 -- 172.21.15.98:0/1203325518 shutdown_connections 2024-05-09T08:55:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.618+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/1203325518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 0x55eb9601f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.618+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/1203325518 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb96032c10 0x55eb960350d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.618+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/1203325518 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017c50 0x55eb9601f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.618+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/1203325518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017330 0x55eb96021dd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.618+0000 7fa5a91ab740 1 -- 172.21.15.98:0/1203325518 >> 172.21.15.98:0/1203325518 conn(0x55eb96013650 msgr2=0x55eb96019450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.620+0000 7fa57f7fe640 0 client.26140 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:55:00.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.628+0000 7fa5a91ab740 1 -- 172.21.15.98:0/1203325518 shutdown_connections 2024-05-09T08:55:00.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.630+0000 7fa5a91ab740 1 -- 172.21.15.98:0/1203325518 wait complete. 2024-05-09T08:55:00.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.631+0000 7fa5a91ab740 20 client.26140 trim_cache size 0 max 16384 2024-05-09T08:55:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.664+0000 7fa5a91ab740 1 Processor -- start 2024-05-09T08:55:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.664+0000 7fa5a91ab740 1 -- start start 2024-05-09T08:55:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.665+0000 7fa5a91ab740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017330 0x55eb9602e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.665+0000 7fa5a91ab740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017c50 0x55eb96019a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.665+0000 7fa59ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017330 0x55eb9602e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.665+0000 7fa59ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017330 0x55eb9602e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55166/0 (socket says 172.21.15.98:55166) 2024-05-09T08:55:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.665+0000 7fa59ffff640 1 -- 172.21.15.98:0/2481247157 learned_addr learned my addr 172.21.15.98:0/2481247157 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.665+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/2481247157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 0x55eb96034c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.666+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2481247157 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a01c8770 con 0x55eb96022320 2024-05-09T08:55:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.666+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2481247157 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a01c6560 con 0x55eb96017330 2024-05-09T08:55:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.666+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2481247157 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a01c80a0 con 0x55eb96017c50 2024-05-09T08:55:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.666+0000 7fa59f7fe640 1 --2- 172.21.15.98:0/2481247157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017c50 0x55eb96019a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.666+0000 7fa5a495e640 1 --2- 172.21.15.98:0/2481247157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 0x55eb96034c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.667+0000 7fa5a495e640 1 -- 172.21.15.98:0/2481247157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017c50 msgr2=0x55eb96019a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.667+0000 7fa5a495e640 1 --2- 172.21.15.98:0/2481247157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017c50 0x55eb96019a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.667+0000 7fa5a495e640 1 -- 172.21.15.98:0/2481247157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017330 msgr2=0x55eb9602e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.667+0000 7fa5a495e640 1 --2- 172.21.15.98:0/2481247157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017330 0x55eb9602e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.667+0000 7fa5a495e640 1 -- 172.21.15.98:0/2481247157 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb9601fc10 con 0x55eb96022320 2024-05-09T08:55:00.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.667+0000 7fa5a495e640 1 --2- 172.21.15.98:0/2481247157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 0x55eb96034c20 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fa5940ccbc0 tx=0x7fa59409b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437c306c0eae8f0a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.673+0000 7fa585ffb640 1 -- 172.21.15.98:0/2481247157 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59409b990 con 0x55eb96022320 2024-05-09T08:55:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.673+0000 7fa585ffb640 1 -- 172.21.15.98:0/2481247157 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa59409fd40 con 0x55eb96022320 2024-05-09T08:55:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.673+0000 7fa585ffb640 1 -- 172.21.15.98:0/2481247157 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5940cf430 con 0x55eb96022320 2024-05-09T08:55:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.674+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2481247157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 msgr2=0x55eb96034c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.675+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/2481247157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 0x55eb96034c20 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fa5940ccbc0 tx=0x7fa59409b880 comp rx=0 tx=0).stop 2024-05-09T08:55:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.675+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2481247157 shutdown_connections 2024-05-09T08:55:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.675+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/2481247157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96022320 0x55eb96034c20 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.675+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/2481247157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017c50 0x55eb96019a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.675+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/2481247157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96017330 0x55eb9602e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.675+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2481247157 >> 172.21.15.98:0/2481247157 conn(0x55eb96013650 msgr2=0x55eb96013a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.675+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2481247157 shutdown_connections 2024-05-09T08:55:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.675+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2481247157 wait complete. 2024-05-09T08:55:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.676+0000 7fa5a91ab740 1 -- start start 2024-05-09T08:55:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.677+0000 7fa5a91ab740 1 Processor -- start 2024-05-09T08:55:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.677+0000 7fa5a91ab740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96017330 0x55eb9602ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.678+0000 7fa5a91ab740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017c50 0x55eb9601c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.678+0000 7fa5a91ab740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96022320 0x55eb9601b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.678+0000 7fa5a91ab740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a01c7300 con 0x55eb96017330 2024-05-09T08:55:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.678+0000 7fa5a91ab740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a01c6c30 con 0x55eb96022320 2024-05-09T08:55:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.678+0000 7fa5a91ab740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a01c79d0 con 0x55eb96017c50 2024-05-09T08:55:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.678+0000 7fa5a91ab740 10 client.0 _ll_register_callbacks cb 0x7fa598009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.679+0000 7fa59ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96017330 0x55eb9602ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.679+0000 7fa59ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96017330 0x55eb9602ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37416/0 (socket says 172.21.15.98:37416) 2024-05-09T08:55:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.679+0000 7fa59ffff640 1 -- 172.21.15.98:0/2670780819 learned_addr learned my addr 172.21.15.98:0/2670780819 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.679+0000 7fa59ffff640 1 -- 172.21.15.98:0/2670780819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017c50 msgr2=0x55eb9601c890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:55:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.679+0000 7fa59f7fe640 1 --2- 172.21.15.98:0/2670780819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017c50 0x55eb9601c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.679+0000 7fa5a495e640 1 --2- 172.21.15.98:0/2670780819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96022320 0x55eb9601b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.680+0000 7fa59ffff640 1 --2- 172.21.15.98:0/2670780819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017c50 0x55eb9601c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.680+0000 7fa59ffff640 1 -- 172.21.15.98:0/2670780819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96022320 msgr2=0x55eb9601b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.680+0000 7fa59ffff640 1 --2- 172.21.15.98:0/2670780819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96022320 0x55eb9601b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.680+0000 7fa59ffff640 1 -- 172.21.15.98:0/2670780819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb9601b910 con 0x55eb96017330 2024-05-09T08:55:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.680+0000 7fa5a495e640 1 --2- 172.21.15.98:0/2670780819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96022320 0x55eb9601b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:55:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.681+0000 7fa59ffff640 1 --2- 172.21.15.98:0/2670780819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96017330 0x55eb9602ab10 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa598030f90 tx=0x7fa598009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d80cd0bf02898250 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.681+0000 7fa57f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:55:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.681+0000 7fa57f7fe640 1 -- 172.21.15.98:0/2670780819 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa598053a00 con 0x55eb96017330 2024-05-09T08:55:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.682+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2670780819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa59409fd40 con 0x55eb96017330 2024-05-09T08:55:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.682+0000 7fa57f7fe640 1 -- 172.21.15.98:0/2670780819 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fa598053ba0 con 0x55eb96017330 2024-05-09T08:55:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.682+0000 7fa57f7fe640 1 -- 172.21.15.98:0/2670780819 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa59806d050 con 0x55eb96017330 2024-05-09T08:55:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.682+0000 7fa5a91ab740 10 client.16881 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.682+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2670780819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55eb96098840 con 0x55eb96017330 2024-05-09T08:55:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.683+0000 7fa57f7fe640 1 -- 172.21.15.98:0/2670780819 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa59409fd40 con 0x55eb96017330 2024-05-09T08:55:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.683+0000 7fa57f7fe640 1 -- 172.21.15.98:0/2670780819 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fa598051040 con 0x55eb96017330 2024-05-09T08:55:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.683+0000 7fa5a91ab740 10 client.16881 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.683+0000 7fa5a91ab740 10 client.16881 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:55:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.684+0000 7fa5a91ab740 20 client.16881 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.684+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/2670780819 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb96098f90 0x55eb9609b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.684+0000 7fa5a91ab740 4 client.16881 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:55:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.684+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2670780819 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eb95f846c0 con 0x55eb96098f90 2024-05-09T08:55:00.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.687+0000 7fa59f7fe640 1 --2- 172.21.15.98:0/2670780819 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb96098f90 0x55eb9609b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.692+0000 7fa59f7fe640 1 --2- 172.21.15.98:0/2670780819 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb96098f90 0x55eb9609b3b0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=44860c24e6f77cef server_cookie=28a78e977e676b4b in_seq=0 out_seq=0 2024-05-09T08:55:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.692+0000 7fa57f7fe640 10 client.16881 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:55:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.692+0000 7fa57f7fe640 1 -- 172.21.15.98:0/2670780819 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eb95f846c0 con 0x55eb96098f90 2024-05-09T08:55:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.692+0000 7fa57f7fe640 10 client.16881 handle_command_reply: tid=0 2024-05-09T08:55:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.692+0000 7fa5a91ab740 1 client.16881 shutdown 2024-05-09T08:55:00.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.694+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2670780819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96017330 msgr2=0x55eb9602ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:00.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.694+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/2670780819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96017330 0x55eb9602ab10 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa598030f90 tx=0x7fa598009780 comp rx=0 tx=0).stop 2024-05-09T08:55:00.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.694+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2670780819 shutdown_connections 2024-05-09T08:55:00.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.694+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/2670780819 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55eb96098f90 0x55eb9609b3b0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.694+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/2670780819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eb96022320 0x55eb9601b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.694+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/2670780819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eb96017c50 0x55eb9601c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.694+0000 7fa5a91ab740 1 --2- 172.21.15.98:0/2670780819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb96017330 0x55eb9602ab10 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.694+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2670780819 >> 172.21.15.98:0/2670780819 conn(0x55eb96013650 msgr2=0x55eb96034c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:00.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.694+0000 7fa57f7fe640 0 client.16881 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:55:00.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.695+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2670780819 shutdown_connections 2024-05-09T08:55:00.695 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:55:00.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.695+0000 7fa5a91ab740 1 -- 172.21.15.98:0/2670780819 wait complete. 2024-05-09T08:55:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.695+0000 7fa5a91ab740 20 client.16881 trim_cache size 0 max 16384 2024-05-09T08:55:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.700+0000 7fa5a63c1640 1 -- 172.21.15.98:0/2120279519 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa5880a2d80 msgr2=0x7fa5880a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.700+0000 7fa5a63c1640 1 --2- 172.21.15.98:0/2120279519 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa5880a2d80 0x7fa5880a5240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55eb960188f0 tx=0x7fa598005d80 comp rx=0 tx=0).stop 2024-05-09T08:55:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.700+0000 7fa5a63c1640 1 -- 172.21.15.98:0/2120279519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5a011b010 msgr2=0x7fa5a01ab820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:00.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.700+0000 7fa5a63c1640 1 --2- 172.21.15.98:0/2120279519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5a011b010 0x7fa5a01ab820 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa59400e9a0 tx=0x7fa59400ee60 comp rx=0 tx=0).stop 2024-05-09T08:55:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.702+0000 7fa5a63c1640 1 -- 172.21.15.98:0/2120279519 shutdown_connections 2024-05-09T08:55:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.702+0000 7fa5a63c1640 1 --2- 172.21.15.98:0/2120279519 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa5880a2d80 0x7fa5880a5240 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.702+0000 7fa5a63c1640 1 --2- 172.21.15.98:0/2120279519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5a011b010 0x7fa5a01ab820 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.702+0000 7fa5a63c1640 1 --2- 172.21.15.98:0/2120279519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5a0110080 0x7fa5a01a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.702+0000 7fa5a63c1640 1 --2- 172.21.15.98:0/2120279519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5a010e0b0 0x7fa5a01a6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.702+0000 7fa5a63c1640 1 -- 172.21.15.98:0/2120279519 >> 172.21.15.98:0/2120279519 conn(0x7fa5a006c940 msgr2=0x7fa5a011c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:00.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.704+0000 7fa5a63c1640 1 -- 172.21.15.98:0/2120279519 shutdown_connections 2024-05-09T08:55:00.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:00.704+0000 7fa5a63c1640 1 -- 172.21.15.98:0/2120279519 wait complete. 2024-05-09T08:55:00.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:55:01.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.120+0000 7fa58209e640 1 -- 172.21.15.98:0/3483789437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa57c108aa0 msgr2=0x7fa57c108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:01.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.120+0000 7fa58209e640 1 --2- 172.21.15.98:0/3483789437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa57c108aa0 0x7fa57c108e80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa56c009860 tx=0x7fa56c02a360 comp rx=0 tx=0).stop 2024-05-09T08:55:01.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.121+0000 7fa58209e640 1 -- 172.21.15.98:0/3483789437 shutdown_connections 2024-05-09T08:55:01.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.121+0000 7fa58209e640 1 --2- 172.21.15.98:0/3483789437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa57c10d090 0x7fa57c079b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.121+0000 7fa58209e640 1 --2- 172.21.15.98:0/3483789437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c109450 0x7fa57c10c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.121+0000 7fa58209e640 1 --2- 172.21.15.98:0/3483789437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa57c108aa0 0x7fa57c108e80 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa56c009860 tx=0x7fa56c02a360 comp rx=0 tx=0).stop 2024-05-09T08:55:01.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.121+0000 7fa58209e640 1 -- 172.21.15.98:0/3483789437 >> 172.21.15.98:0/3483789437 conn(0x7fa57c06c810 msgr2=0x7fa57c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:01.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.123+0000 7fa58209e640 1 -- 172.21.15.98:0/3483789437 shutdown_connections 2024-05-09T08:55:01.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.128+0000 7fa58209e640 1 -- 172.21.15.98:0/3483789437 wait complete. 2024-05-09T08:55:01.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.128+0000 7fa58209e640 1 Processor -- start 2024-05-09T08:55:01.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.130+0000 7fa58209e640 1 -- start start 2024-05-09T08:55:01.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.130+0000 7fa58209e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa57c10d090 0x7fa57c07f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.130+0000 7fa58209e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa57c07fd10 0x7fa57c080170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.130+0000 7fa58209e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c1bdce0 0x7fa57c1c00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.130+0000 7fa58209e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c07cfc0 con 0x7fa57c1bdce0 2024-05-09T08:55:01.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.130+0000 7fa58209e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c07ce40 con 0x7fa57c10d090 2024-05-09T08:55:01.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.130+0000 7fa58209e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c07d140 con 0x7fa57c07fd10 2024-05-09T08:55:01.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.130+0000 7fa57bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c1bdce0 0x7fa57c1c00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:01.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.130+0000 7fa57bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c1bdce0 0x7fa57c1c00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37432/0 (socket says 172.21.15.98:37432) 2024-05-09T08:55:01.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.130+0000 7fa57bfff640 1 -- 172.21.15.98:0/1535023798 learned_addr learned my addr 172.21.15.98:0/1535023798 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:01.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.131+0000 7fa57affd640 1 --2- 172.21.15.98:0/1535023798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa57c07fd10 0x7fa57c080170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:01.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.131+0000 7fa57bfff640 1 -- 172.21.15.98:0/1535023798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa57c07fd10 msgr2=0x7fa57c080170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:01.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.131+0000 7fa57bfff640 1 --2- 172.21.15.98:0/1535023798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa57c07fd10 0x7fa57c080170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.131+0000 7fa57bfff640 1 -- 172.21.15.98:0/1535023798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa57c10d090 msgr2=0x7fa57c07f7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:55:01.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.131+0000 7fa57bfff640 1 --2- 172.21.15.98:0/1535023798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa57c10d090 0x7fa57c07f7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.131+0000 7fa57bfff640 1 -- 172.21.15.98:0/1535023798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57c1c0610 con 0x7fa57c1bdce0 2024-05-09T08:55:01.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.131+0000 7fa57bfff640 1 --2- 172.21.15.98:0/1535023798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c1bdce0 0x7fa57c1c00d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa57000d750 tx=0x7fa57000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:01.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.133+0000 7fa578ff9640 1 -- 172.21.15.98:0/1535023798 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa570017070 con 0x7fa57c1bdce0 2024-05-09T08:55:01.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.133+0000 7fa578ff9640 1 -- 172.21.15.98:0/1535023798 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa570005500 con 0x7fa57c1bdce0 2024-05-09T08:55:01.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.133+0000 7fa578ff9640 1 -- 172.21.15.98:0/1535023798 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa570004d60 con 0x7fa57c1bdce0 2024-05-09T08:55:01.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.133+0000 7fa58209e640 1 -- 172.21.15.98:0/1535023798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa57c1c0900 con 0x7fa57c1bdce0 2024-05-09T08:55:01.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.133+0000 7fa58209e640 1 -- 172.21.15.98:0/1535023798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa57c1b58b0 con 0x7fa57c1bdce0 2024-05-09T08:55:01.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.138+0000 7fa584dac740 1 Processor -- start 2024-05-09T08:55:01.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.138+0000 7fa584dac740 1 -- start start 2024-05-09T08:55:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.139+0000 7fa584dac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90f520 0x557b6a90f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.139+0000 7fa584dac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a90fe40 0x557b6a919fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.139+0000 7fa584dac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 0x557b6a91c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.139+0000 7fa584dac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557b6a874c40 con 0x557b6a91a510 2024-05-09T08:55:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.139+0000 7fa584dac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557b6a81c580 con 0x557b6a90f520 2024-05-09T08:55:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.139+0000 7fa584dac740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557b6a870c00 con 0x557b6a90fe40 2024-05-09T08:55:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.139+0000 7fa57affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90f520 0x557b6a90f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:01.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.139+0000 7fa57affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90f520 0x557b6a90f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55204/0 (socket says 172.21.15.98:55204) 2024-05-09T08:55:01.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.139+0000 7fa57affd640 1 -- 172.21.15.98:0/2386297561 learned_addr learned my addr 172.21.15.98:0/2386297561 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:01.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.139+0000 7fa57affd640 1 -- 172.21.15.98:0/2386297561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a90fe40 msgr2=0x557b6a919fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:55:01.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.139+0000 7fa57affd640 1 --2- 172.21.15.98:0/2386297561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a90fe40 0x557b6a919fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.139+0000 7fa57affd640 1 -- 172.21.15.98:0/2386297561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 msgr2=0x557b6a91c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:01.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.139+0000 7fa57affd640 1 --2- 172.21.15.98:0/2386297561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 0x557b6a91c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.139+0000 7fa57affd640 1 -- 172.21.15.98:0/2386297561 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b6a5883f0 con 0x557b6a90f520 2024-05-09T08:55:01.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.144+0000 7fa578ff9640 1 -- 172.21.15.98:0/1535023798 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa570004f00 con 0x7fa57c1bdce0 2024-05-09T08:55:01.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.145+0000 7fa578ff9640 1 --2- 172.21.15.98:0/1535023798 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa5640a2f00 0x7fa5640a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.145+0000 7fa578ff9640 1 -- 172.21.15.98:0/1535023798 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fa5700cee10 con 0x7fa57c1bdce0 2024-05-09T08:55:01.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.147+0000 7fa57affd640 1 --2- 172.21.15.98:0/2386297561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90f520 0x557b6a90f900 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa574009b60 tx=0x7fa57400b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aadcce91e18b28a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:01.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.150+0000 7fa57b7fe640 1 --2- 172.21.15.98:0/1535023798 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa5640a2f00 0x7fa5640a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:01.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.152+0000 7fa561ffb640 1 -- 172.21.15.98:0/2386297561 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa574009e50 con 0x557b6a90f520 2024-05-09T08:55:01.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.152+0000 7fa561ffb640 1 -- 172.21.15.98:0/2386297561 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa57400adb0 con 0x557b6a90f520 2024-05-09T08:55:01.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.153+0000 7fa584dac740 1 -- 172.21.15.98:0/2386297561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90f520 msgr2=0x557b6a90f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:01.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.153+0000 7fa584dac740 1 --2- 172.21.15.98:0/2386297561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90f520 0x557b6a90f900 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa574009b60 tx=0x7fa57400b730 comp rx=0 tx=0).stop 2024-05-09T08:55:01.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.155+0000 7fa584dac740 1 -- 172.21.15.98:0/2386297561 shutdown_connections 2024-05-09T08:55:01.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.155+0000 7fa584dac740 1 --2- 172.21.15.98:0/2386297561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 0x557b6a91c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.155+0000 7fa584dac740 1 --2- 172.21.15.98:0/2386297561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a90fe40 0x557b6a919fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.155+0000 7fa584dac740 1 --2- 172.21.15.98:0/2386297561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90f520 0x557b6a90f900 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.155+0000 7fa584dac740 1 -- 172.21.15.98:0/2386297561 >> 172.21.15.98:0/2386297561 conn(0x557b6a90b840 msgr2=0x557b6a90bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:01.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.156+0000 7fa584dac740 1 -- 172.21.15.98:0/2386297561 shutdown_connections 2024-05-09T08:55:01.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.156+0000 7fa57b7fe640 1 --2- 172.21.15.98:0/1535023798 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa5640a2f00 0x7fa5640a53c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa56c02a8d0 tx=0x7fa56c000bc0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:01.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.156+0000 7fa584dac740 1 -- 172.21.15.98:0/2386297561 wait complete. 2024-05-09T08:55:01.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.157+0000 7fa584dac740 1 -- start start 2024-05-09T08:55:01.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.157+0000 7fa584dac740 1 Processor -- start 2024-05-09T08:55:01.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.160+0000 7fa584dac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a90f520 0x557b6a9174d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.160+0000 7fa584dac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90fe40 0x557b6a917a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.160+0000 7fa584dac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 0x557b6a917f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.160+0000 7fa584dac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557b6a91d350 con 0x557b6a91a510 2024-05-09T08:55:01.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.161+0000 7fa584dac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557b6a82fc40 con 0x557b6a90fe40 2024-05-09T08:55:01.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.161+0000 7fa584dac740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557b6a91d4d0 con 0x557b6a90f520 2024-05-09T08:55:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.161+0000 7fa584dac740 10 client.0 _ll_register_callbacks cb 0x557b6a587be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.161+0000 7fa57affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90fe40 0x557b6a917a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.161+0000 7fa57affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90fe40 0x557b6a917a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55210/0 (socket says 172.21.15.98:55210) 2024-05-09T08:55:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.161+0000 7fa57affd640 1 -- 172.21.15.98:0/2359111990 learned_addr learned my addr 172.21.15.98:0/2359111990 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.161+0000 7fa57bfff640 1 --2- 172.21.15.98:0/2359111990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 0x557b6a917f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.161+0000 7fa57b7fe640 1 --2- 172.21.15.98:0/2359111990 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a90f520 0x557b6a9174d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.161+0000 7fa57bfff640 1 -- 172.21.15.98:0/2359111990 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a90f520 msgr2=0x557b6a9174d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.161+0000 7fa57bfff640 1 --2- 172.21.15.98:0/2359111990 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a90f520 0x557b6a9174d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.161+0000 7fa57bfff640 1 -- 172.21.15.98:0/2359111990 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90fe40 msgr2=0x557b6a917a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:01.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.161+0000 7fa57bfff640 1 --2- 172.21.15.98:0/2359111990 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90fe40 0x557b6a917a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.161+0000 7fa57bfff640 1 -- 172.21.15.98:0/2359111990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b6a6527d0 con 0x557b6a91a510 2024-05-09T08:55:01.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.162+0000 7fa57bfff640 1 --2- 172.21.15.98:0/2359111990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 0x557b6a917f50 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa5700cc140 tx=0x7fa57009b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73e07e555fbd40e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.165+0000 7fa55affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:55:01.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.165+0000 7fa55affd640 1 -- 172.21.15.98:0/2359111990 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa570020040 con 0x557b6a91a510 2024-05-09T08:55:01.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.165+0000 7fa55affd640 1 -- 172.21.15.98:0/2359111990 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fa570097600 con 0x557b6a91a510 2024-05-09T08:55:01.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.167+0000 7fa584dac740 1 -- 172.21.15.98:0/2359111990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557b6a882370 con 0x557b6a91a510 2024-05-09T08:55:01.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.171+0000 7fa55affd640 1 -- 172.21.15.98:0/2359111990 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa5700a3c00 con 0x557b6a91a510 2024-05-09T08:55:01.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.172+0000 7fa584dac740 10 client.16890 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:01.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.172+0000 7fa584dac740 1 -- 172.21.15.98:0/2359111990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557b6a929720 con 0x557b6a91a510 2024-05-09T08:55:01.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.174+0000 7fa55affd640 1 -- 172.21.15.98:0/2359111990 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5700a3e50 con 0x557b6a91a510 2024-05-09T08:55:01.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.174+0000 7fa55affd640 1 -- 172.21.15.98:0/2359111990 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fa570096ce0 con 0x557b6a91a510 2024-05-09T08:55:01.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.174+0000 7fa584dac740 10 client.16890 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:01.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.174+0000 7fa584dac740 10 client.16890 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:01.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.174+0000 7fa584dac740 20 client.16890 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:01.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.174+0000 7fa584dac740 1 --2- 172.21.15.98:0/2359111990 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557b6a92b040 0x557b6a92d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.174+0000 7fa584dac740 4 client.16890 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:01.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.174+0000 7fa584dac740 1 -- 172.21.15.98:0/2359111990 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b6a87c6c0 con 0x557b6a92b040 2024-05-09T08:55:01.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.175+0000 7fa57b7fe640 1 --2- 172.21.15.98:0/2359111990 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557b6a92b040 0x557b6a92d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:01.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.177+0000 7fa57b7fe640 1 --2- 172.21.15.98:0/2359111990 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557b6a92b040 0x557b6a92d470 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2451a8fc493cf9e server_cookie=1364769c03bc72a3 in_seq=0 out_seq=0 2024-05-09T08:55:01.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.177+0000 7fa55affd640 10 client.16890 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:01.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.179+0000 7fa55affd640 1 -- 172.21.15.98:0/2359111990 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557b6a87c6c0 con 0x557b6a92b040 2024-05-09T08:55:01.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.179+0000 7fa55affd640 10 client.16890 handle_command_reply: tid=0 2024-05-09T08:55:01.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.179+0000 7fa584dac740 1 client.16890 shutdown 2024-05-09T08:55:01.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.179+0000 7fa584dac740 1 -- 172.21.15.98:0/2359111990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 msgr2=0x557b6a917f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:01.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.179+0000 7fa584dac740 1 --2- 172.21.15.98:0/2359111990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 0x557b6a917f50 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa5700cc140 tx=0x7fa57009b750 comp rx=0 tx=0).stop 2024-05-09T08:55:01.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.180+0000 7fa584dac740 1 -- 172.21.15.98:0/2359111990 shutdown_connections 2024-05-09T08:55:01.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.180+0000 7fa584dac740 1 --2- 172.21.15.98:0/2359111990 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557b6a92b040 0x557b6a92d470 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.180+0000 7fa584dac740 1 --2- 172.21.15.98:0/2359111990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 0x557b6a917f50 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.180+0000 7fa584dac740 1 --2- 172.21.15.98:0/2359111990 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90fe40 0x557b6a917a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.180+0000 7fa584dac740 1 --2- 172.21.15.98:0/2359111990 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a90f520 0x557b6a9174d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.180+0000 7fa584dac740 1 -- 172.21.15.98:0/2359111990 >> 172.21.15.98:0/2359111990 conn(0x557b6a90b840 msgr2=0x557b6a911640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:01.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.180+0000 7fa55affd640 0 client.16890 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:01.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.180+0000 7fa584dac740 1 -- 172.21.15.98:0/2359111990 shutdown_connections 2024-05-09T08:55:01.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.180+0000 7fa584dac740 1 -- 172.21.15.98:0/2359111990 wait complete. 2024-05-09T08:55:01.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.180+0000 7fa584dac740 20 client.16890 trim_cache size 0 max 16384 2024-05-09T08:55:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.210+0000 7fa584dac740 1 Processor -- start 2024-05-09T08:55:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.211+0000 7fa584dac740 1 -- start start 2024-05-09T08:55:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.211+0000 7fa584dac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a90f520 0x557b6a926260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.211+0000 7fa584dac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90fe40 0x557b6a911d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.211+0000 7fa584dac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 0x557b6a917d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.212+0000 7fa584dac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c1cf020 con 0x557b6a91a510 2024-05-09T08:55:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.212+0000 7fa584dac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c1d0b60 con 0x557b6a90fe40 2024-05-09T08:55:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.212+0000 7fa57affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90fe40 0x557b6a911d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.212+0000 7fa57affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90fe40 0x557b6a911d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55220/0 (socket says 172.21.15.98:55220) 2024-05-09T08:55:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.212+0000 7fa57affd640 1 -- 172.21.15.98:0/4142183114 learned_addr learned my addr 172.21.15.98:0/4142183114 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.212+0000 7fa584dac740 1 -- 172.21.15.98:0/4142183114 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c1cf6f0 con 0x557b6a90f520 2024-05-09T08:55:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.213+0000 7fa57bfff640 1 --2- 172.21.15.98:0/4142183114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 0x557b6a917d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.215+0000 7fa57affd640 1 -- 172.21.15.98:0/4142183114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a90f520 msgr2=0x557b6a926260 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:55:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.215+0000 7fa57affd640 1 --2- 172.21.15.98:0/4142183114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a90f520 0x557b6a926260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.215+0000 7fa57affd640 1 -- 172.21.15.98:0/4142183114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 msgr2=0x557b6a917d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.215+0000 7fa57affd640 1 --2- 172.21.15.98:0/4142183114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 0x557b6a917d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.215+0000 7fa57affd640 1 -- 172.21.15.98:0/4142183114 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b6a90fc70 con 0x557b6a90fe40 2024-05-09T08:55:01.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.216+0000 7fa57affd640 1 --2- 172.21.15.98:0/4142183114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90fe40 0x557b6a911d90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa574004a10 tx=0x7fa574009d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f22a944eca371660 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:01.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.216+0000 7fa5617fa640 1 -- 172.21.15.98:0/4142183114 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa574027070 con 0x557b6a90fe40 2024-05-09T08:55:01.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.216+0000 7fa5617fa640 1 -- 172.21.15.98:0/4142183114 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa574003970 con 0x557b6a90fe40 2024-05-09T08:55:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.217+0000 7fa584dac740 1 -- 172.21.15.98:0/4142183114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90fe40 msgr2=0x557b6a911d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.217+0000 7fa584dac740 1 --2- 172.21.15.98:0/4142183114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90fe40 0x557b6a911d90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa574004a10 tx=0x7fa574009d00 comp rx=0 tx=0).stop 2024-05-09T08:55:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.218+0000 7fa584dac740 1 -- 172.21.15.98:0/4142183114 shutdown_connections 2024-05-09T08:55:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.218+0000 7fa584dac740 1 --2- 172.21.15.98:0/4142183114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a91a510 0x557b6a917d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.218+0000 7fa584dac740 1 --2- 172.21.15.98:0/4142183114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a90fe40 0x557b6a911d90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.218+0000 7fa584dac740 1 --2- 172.21.15.98:0/4142183114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a90f520 0x557b6a926260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.218+0000 7fa584dac740 1 -- 172.21.15.98:0/4142183114 >> 172.21.15.98:0/4142183114 conn(0x557b6a90b840 msgr2=0x557b6a912870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.223+0000 7fa584dac740 1 -- 172.21.15.98:0/4142183114 shutdown_connections 2024-05-09T08:55:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.223+0000 7fa584dac740 1 -- 172.21.15.98:0/4142183114 wait complete. 2024-05-09T08:55:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.223+0000 7fa584dac740 1 -- start start 2024-05-09T08:55:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.224+0000 7fa584dac740 1 Processor -- start 2024-05-09T08:55:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.225+0000 7fa584dac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a91a510 0x557b6a922c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.225+0000 7fa584dac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a9147f0 0x557b6a914c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.225+0000 7fa584dac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a915310 0x557b6a913740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.225+0000 7fa584dac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c1d0490 con 0x557b6a915310 2024-05-09T08:55:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.225+0000 7fa584dac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c1cfdc0 con 0x557b6a9147f0 2024-05-09T08:55:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.225+0000 7fa584dac740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c1cd020 con 0x557b6a91a510 2024-05-09T08:55:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.225+0000 7fa584dac740 10 client.0 _ll_register_callbacks cb 0x7fa5700972b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.226+0000 7fa57bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a915310 0x557b6a913740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.226+0000 7fa57bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a915310 0x557b6a913740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37460/0 (socket says 172.21.15.98:37460) 2024-05-09T08:55:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.226+0000 7fa57bfff640 1 -- 172.21.15.98:0/513202443 learned_addr learned my addr 172.21.15.98:0/513202443 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.227+0000 7fa57bfff640 1 -- 172.21.15.98:0/513202443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a91a510 msgr2=0x557b6a922c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.227+0000 7fa57bfff640 1 --2- 172.21.15.98:0/513202443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a91a510 0x557b6a922c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.227+0000 7fa57bfff640 1 -- 172.21.15.98:0/513202443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a9147f0 msgr2=0x557b6a914c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.227+0000 7fa57bfff640 1 --2- 172.21.15.98:0/513202443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a9147f0 0x557b6a914c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.227+0000 7fa57bfff640 1 -- 172.21.15.98:0/513202443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b6a913e70 con 0x557b6a915310 2024-05-09T08:55:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.228+0000 7fa57bfff640 1 --2- 172.21.15.98:0/513202443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a915310 0x557b6a913740 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa570097050 tx=0x7fa570009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0d067606130101 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.228+0000 7fa55affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:55:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.229+0000 7fa55affd640 1 -- 172.21.15.98:0/513202443 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa570014070 con 0x557b6a915310 2024-05-09T08:55:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.229+0000 7fa584dac740 1 -- 172.21.15.98:0/513202443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557b6a7f9820 con 0x557b6a915310 2024-05-09T08:55:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.229+0000 7fa55affd640 1 -- 172.21.15.98:0/513202443 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fa5700cc3c0 con 0x557b6a915310 2024-05-09T08:55:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.230+0000 7fa55affd640 1 -- 172.21.15.98:0/513202443 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa57007e050 con 0x557b6a915310 2024-05-09T08:55:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.240+0000 7fa584dac740 10 client.16896 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.240+0000 7fa584dac740 1 -- 172.21.15.98:0/513202443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557b6a9a1ff0 con 0x557b6a915310 2024-05-09T08:55:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.242+0000 7fa55affd640 1 -- 172.21.15.98:0/513202443 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557b6a7f9820 con 0x557b6a915310 2024-05-09T08:55:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.242+0000 7fa55affd640 1 -- 172.21.15.98:0/513202443 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fa57009a3e0 con 0x557b6a915310 2024-05-09T08:55:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.243+0000 7fa584dac740 10 client.16896 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.243+0000 7fa584dac740 10 client.16896 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.243+0000 7fa584dac740 20 client.16896 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.243+0000 7fa584dac740 1 --2- 172.21.15.98:0/513202443 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557b6a9a2570 0x557b6a9a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.243+0000 7fa584dac740 4 client.16896 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:55:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.243+0000 7fa584dac740 1 -- 172.21.15.98:0/513202443 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b6a845f80 con 0x557b6a9a2570 2024-05-09T08:55:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.244+0000 7fa57b7fe640 1 --2- 172.21.15.98:0/513202443 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557b6a9a2570 0x557b6a9a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.255+0000 7fa57b7fe640 1 --2- 172.21.15.98:0/513202443 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557b6a9a2570 0x557b6a9a4a30 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c47bcedca6a7fb45 server_cookie=9386f02731e804ca in_seq=0 out_seq=0 2024-05-09T08:55:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.255+0000 7fa55affd640 10 client.16896 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.256+0000 7fa55affd640 1 -- 172.21.15.98:0/513202443 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b6a845f80 con 0x557b6a9a2570 2024-05-09T08:55:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.256+0000 7fa55affd640 10 client.16896 handle_command_reply: tid=0 2024-05-09T08:55:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.257+0000 7fa584dac740 1 client.16896 shutdown 2024-05-09T08:55:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.258+0000 7fa584dac740 1 -- 172.21.15.98:0/513202443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a915310 msgr2=0x557b6a913740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.258+0000 7fa584dac740 1 --2- 172.21.15.98:0/513202443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a915310 0x557b6a913740 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa570097050 tx=0x7fa570009f60 comp rx=0 tx=0).stop 2024-05-09T08:55:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.265+0000 7fa584dac740 1 -- 172.21.15.98:0/513202443 shutdown_connections 2024-05-09T08:55:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.265+0000 7fa584dac740 1 --2- 172.21.15.98:0/513202443 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557b6a9a2570 0x557b6a9a4a30 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.265+0000 7fa584dac740 1 --2- 172.21.15.98:0/513202443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b6a915310 0x557b6a913740 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.265+0000 7fa584dac740 1 --2- 172.21.15.98:0/513202443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557b6a9147f0 0x557b6a914c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.265+0000 7fa584dac740 1 --2- 172.21.15.98:0/513202443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557b6a91a510 0x557b6a922c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.265+0000 7fa584dac740 1 -- 172.21.15.98:0/513202443 >> 172.21.15.98:0/513202443 conn(0x557b6a90b840 msgr2=0x557b6a90a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.265+0000 7fa55affd640 0 client.16896 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.266+0000 7fa584dac740 1 -- 172.21.15.98:0/513202443 shutdown_connections 2024-05-09T08:55:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.266+0000 7fa584dac740 1 -- 172.21.15.98:0/513202443 wait complete. 2024-05-09T08:55:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.266+0000 7fa584dac740 20 client.16896 trim_cache size 0 max 16384 2024-05-09T08:55:01.268 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:55:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.271+0000 7fa58209e640 1 -- 172.21.15.98:0/1535023798 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa5640a2f00 msgr2=0x7fa5640a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.271+0000 7fa58209e640 1 --2- 172.21.15.98:0/1535023798 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa5640a2f00 0x7fa5640a53c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa56c02a8d0 tx=0x7fa56c000bc0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.271+0000 7fa58209e640 1 -- 172.21.15.98:0/1535023798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c1bdce0 msgr2=0x7fa57c1c00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.271+0000 7fa58209e640 1 --2- 172.21.15.98:0/1535023798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c1bdce0 0x7fa57c1c00d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa57000d750 tx=0x7fa57000dc10 comp rx=0 tx=0).stop 2024-05-09T08:55:01.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.276+0000 7fa58209e640 1 -- 172.21.15.98:0/1535023798 shutdown_connections 2024-05-09T08:55:01.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.276+0000 7fa58209e640 1 --2- 172.21.15.98:0/1535023798 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa5640a2f00 0x7fa5640a53c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.276+0000 7fa58209e640 1 --2- 172.21.15.98:0/1535023798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c1bdce0 0x7fa57c1c00d0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.276+0000 7fa58209e640 1 --2- 172.21.15.98:0/1535023798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa57c07fd10 0x7fa57c080170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.276+0000 7fa58209e640 1 --2- 172.21.15.98:0/1535023798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa57c10d090 0x7fa57c07f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:01.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.276+0000 7fa58209e640 1 -- 172.21.15.98:0/1535023798 >> 172.21.15.98:0/1535023798 conn(0x7fa57c06c810 msgr2=0x7fa57c0715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:01.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.284+0000 7fa58209e640 1 -- 172.21.15.98:0/1535023798 shutdown_connections 2024-05-09T08:55:01.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:01.287+0000 7fa58209e640 1 -- 172.21.15.98:0/1535023798 wait complete. 2024-05-09T08:55:02.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:02 smithi098 ceph-mon[30092]: pgmap v136: 129 pgs: 129 active+clean; 30 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 928 MiB/s wr, 2.13k op/s 2024-05-09T08:55:02.347 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:55:02.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:55:02.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:55:02.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:02 smithi142 ceph-mon[34625]: pgmap v136: 129 pgs: 129 active+clean; 30 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 928 MiB/s wr, 2.13k op/s 2024-05-09T08:55:02.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:02 smithi105 ceph-mon[34498]: pgmap v136: 129 pgs: 129 active+clean; 30 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 928 MiB/s wr, 2.13k op/s 2024-05-09T08:55:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.624+0000 7fec769b6640 1 -- 172.21.15.98:0/2509385927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec7010d130 msgr2=0x7fec70114d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.624+0000 7fec769b6640 1 --2- 172.21.15.98:0/2509385927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec7010d130 0x7fec70114d00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fec58009900 tx=0x7fec5802a400 comp rx=0 tx=0).stop 2024-05-09T08:55:02.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.629+0000 7fec769b6640 1 -- 172.21.15.98:0/2509385927 shutdown_connections 2024-05-09T08:55:02.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.629+0000 7fec769b6640 1 --2- 172.21.15.98:0/2509385927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec7010d130 0x7fec70114d00 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.629+0000 7fec769b6640 1 --2- 172.21.15.98:0/2509385927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec70108a60 0x7fec7010ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.629+0000 7fec769b6640 1 --2- 172.21.15.98:0/2509385927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec701080b0 0x7fec70108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.629+0000 7fec769b6640 1 -- 172.21.15.98:0/2509385927 >> 172.21.15.98:0/2509385927 conn(0x7fec70075d70 msgr2=0x7fec70078190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:02.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.634+0000 7fec769b6640 1 -- 172.21.15.98:0/2509385927 shutdown_connections 2024-05-09T08:55:02.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.636+0000 7fec769b6640 1 -- 172.21.15.98:0/2509385927 wait complete. 2024-05-09T08:55:02.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.637+0000 7fec769b6640 1 Processor -- start 2024-05-09T08:55:02.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.638+0000 7fec769b6640 1 -- start start 2024-05-09T08:55:02.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.638+0000 7fec769b6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec701080b0 0x7fec70073000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.638+0000 7fec769b6640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec7010d130 0x7fec7006c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.638+0000 7fec769b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec70073650 0x7fec7006c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.638+0000 7fec769b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fec701181d0 con 0x7fec70073650 2024-05-09T08:55:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.638+0000 7fec769b6640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fec70118050 con 0x7fec7010d130 2024-05-09T08:55:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.638+0000 7fec769b6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fec70118350 con 0x7fec701080b0 2024-05-09T08:55:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.639+0000 7fec74f53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec70073650 0x7fec7006c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.639+0000 7fec74f53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec70073650 0x7fec7006c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37480/0 (socket says 172.21.15.98:37480) 2024-05-09T08:55:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.639+0000 7fec74f53640 1 -- 172.21.15.98:0/3156852545 learned_addr learned my addr 172.21.15.98:0/3156852545 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.640+0000 7fec6ffff640 1 --2- 172.21.15.98:0/3156852545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec701080b0 0x7fec70073000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.641+0000 7fec74f53640 1 -- 172.21.15.98:0/3156852545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec701080b0 msgr2=0x7fec70073000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.641+0000 7fec74f53640 1 --2- 172.21.15.98:0/3156852545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec701080b0 0x7fec70073000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.641+0000 7fec74f53640 1 -- 172.21.15.98:0/3156852545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec7010d130 msgr2=0x7fec7006c080 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:55:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.641+0000 7fec74f53640 1 --2- 172.21.15.98:0/3156852545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec7010d130 0x7fec7006c080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.641+0000 7fec74f53640 1 -- 172.21.15.98:0/3156852545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec70070b20 con 0x7fec70073650 2024-05-09T08:55:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.642+0000 7fec74f53640 1 --2- 172.21.15.98:0/3156852545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec70073650 0x7fec7006c660 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fec5802a970 tx=0x7fec58003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:02.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.644+0000 7fec6d7fa640 1 -- 172.21.15.98:0/3156852545 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5803f070 con 0x7fec70073650 2024-05-09T08:55:02.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.644+0000 7fec6d7fa640 1 -- 172.21.15.98:0/3156852545 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec58031c90 con 0x7fec70073650 2024-05-09T08:55:02.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.644+0000 7fec6d7fa640 1 -- 172.21.15.98:0/3156852545 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec58035030 con 0x7fec70073650 2024-05-09T08:55:02.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.645+0000 7fec769b6640 1 -- 172.21.15.98:0/3156852545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec70070db0 con 0x7fec70073650 2024-05-09T08:55:02.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.645+0000 7fec769b6640 1 -- 172.21.15.98:0/3156852545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec70071250 con 0x7fec70073650 2024-05-09T08:55:02.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.652+0000 7fec6d7fa640 1 -- 172.21.15.98:0/3156852545 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fec58042070 con 0x7fec70073650 2024-05-09T08:55:02.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.653+0000 7fec7979b740 1 Processor -- start 2024-05-09T08:55:02.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.657+0000 7fec7979b740 1 -- start start 2024-05-09T08:55:02.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.657+0000 7fec7979b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca666260a0 0x55ca666243e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.657+0000 7fec7979b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 0x55ca66624e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.657+0000 7fec7979b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 0x55ca66631b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.657+0000 7fec7979b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ca66589990 con 0x55ca666260a0 2024-05-09T08:55:02.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.657+0000 7fec7979b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ca665315b0 con 0x55ca6662f730 2024-05-09T08:55:02.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.657+0000 7fec7979b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ca66585950 con 0x55ca666249b0 2024-05-09T08:55:02.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.659+0000 7fec6ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 0x55ca66624e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.660+0000 7fec6d7fa640 1 --2- 172.21.15.98:0/3156852545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fec5c0a2f00 0x7fec5c0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.660+0000 7fec6ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 0x55ca66624e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:51218/0 (socket says 172.21.15.98:51218) 2024-05-09T08:55:02.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.660+0000 7fec6ffff640 1 -- 172.21.15.98:0/3524061022 learned_addr learned my addr 172.21.15.98:0/3524061022 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:02.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.662+0000 7fec6f7fe640 1 --2- 172.21.15.98:0/3524061022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 0x55ca66631b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.662+0000 7fec6f7fe640 1 --2- 172.21.15.98:0/3524061022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca666260a0 0x55ca666243e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.662+0000 7fec74f53640 1 --2- 172.21.15.98:0/3156852545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fec5c0a2f00 0x7fec5c0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.663+0000 7fec6d7fa640 1 -- 172.21.15.98:0/3156852545 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fec580f1210 con 0x7fec70073650 2024-05-09T08:55:02.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.663+0000 7fec6ffff640 1 -- 172.21.15.98:0/3524061022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 msgr2=0x55ca66631b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:02.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.663+0000 7fec6ffff640 1 --2- 172.21.15.98:0/3524061022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 0x55ca66631b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.663+0000 7fec6ffff640 1 -- 172.21.15.98:0/3524061022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca666260a0 msgr2=0x55ca666243e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:02.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.663+0000 7fec6ffff640 1 --2- 172.21.15.98:0/3524061022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca666260a0 0x55ca666243e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.663+0000 7fec6ffff640 1 -- 172.21.15.98:0/3524061022 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca663657e0 con 0x55ca666249b0 2024-05-09T08:55:02.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.666+0000 7fec6ffff640 1 --2- 172.21.15.98:0/3524061022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 0x55ca66624e30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec68011470 tx=0x7fec68011930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33816f711f71c1cb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.670+0000 7fec74f53640 1 --2- 172.21.15.98:0/3156852545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fec5c0a2f00 0x7fec5c0a53c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fec580bd010 tx=0x7fec5802a400 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.670+0000 7fec667fc640 1 -- 172.21.15.98:0/3524061022 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6801e070 con 0x55ca666249b0 2024-05-09T08:55:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.670+0000 7fec667fc640 1 -- 172.21.15.98:0/3524061022 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec68004030 con 0x55ca666249b0 2024-05-09T08:55:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.670+0000 7fec667fc640 1 -- 172.21.15.98:0/3524061022 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6800f7e0 con 0x55ca666249b0 2024-05-09T08:55:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.671+0000 7fec7979b740 1 -- 172.21.15.98:0/3524061022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 msgr2=0x55ca66624e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.671+0000 7fec7979b740 1 --2- 172.21.15.98:0/3524061022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 0x55ca66624e30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec68011470 tx=0x7fec68011930 comp rx=0 tx=0).stop 2024-05-09T08:55:02.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.679+0000 7fec7979b740 1 -- 172.21.15.98:0/3524061022 shutdown_connections 2024-05-09T08:55:02.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.679+0000 7fec7979b740 1 --2- 172.21.15.98:0/3524061022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 0x55ca66631b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.679+0000 7fec7979b740 1 --2- 172.21.15.98:0/3524061022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 0x55ca66624e30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.679+0000 7fec7979b740 1 --2- 172.21.15.98:0/3524061022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca666260a0 0x55ca666243e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.679+0000 7fec7979b740 1 -- 172.21.15.98:0/3524061022 >> 172.21.15.98:0/3524061022 conn(0x55ca665ed0a0 msgr2=0x55ca66620f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:02.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.680+0000 7fec7979b740 1 -- 172.21.15.98:0/3524061022 shutdown_connections 2024-05-09T08:55:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.685+0000 7fec7979b740 1 -- 172.21.15.98:0/3524061022 wait complete. 2024-05-09T08:55:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.686+0000 7fec7979b740 1 -- start start 2024-05-09T08:55:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.686+0000 7fec7979b740 1 Processor -- start 2024-05-09T08:55:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.687+0000 7fec7979b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 0x55ca6662c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.687+0000 7fec7979b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca666260a0 0x55ca6662cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.687+0000 7fec7979b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca6662f730 0x55ca6662d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.687+0000 7fec7979b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ca666325c0 con 0x55ca6662f730 2024-05-09T08:55:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.687+0000 7fec7979b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ca66544b40 con 0x55ca666260a0 2024-05-09T08:55:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.687+0000 7fec7979b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ca66632740 con 0x55ca666249b0 2024-05-09T08:55:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.687+0000 7fec7979b740 10 client.0 _ll_register_callbacks cb 0x55ca665390f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:02.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.692+0000 7fec6ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca666260a0 0x55ca6662cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.692+0000 7fec6ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca666260a0 0x55ca6662cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55254/0 (socket says 172.21.15.98:55254) 2024-05-09T08:55:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.692+0000 7fec6ffff640 1 -- 172.21.15.98:0/334167902 learned_addr learned my addr 172.21.15.98:0/334167902 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.694+0000 7fec6f7fe640 1 --2- 172.21.15.98:0/334167902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 0x55ca6662c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.694+0000 7fec6f7fe640 1 --2- 172.21.15.98:0/334167902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca6662f730 0x55ca6662d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.695+0000 7fec6ffff640 1 -- 172.21.15.98:0/334167902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 msgr2=0x55ca6662c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.695+0000 7fec6ffff640 1 --2- 172.21.15.98:0/334167902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 0x55ca6662c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.695+0000 7fec6ffff640 1 -- 172.21.15.98:0/334167902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca6662f730 msgr2=0x55ca6662d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.695+0000 7fec6ffff640 1 --2- 172.21.15.98:0/334167902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca6662f730 0x55ca6662d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.695+0000 7fec6ffff640 1 -- 172.21.15.98:0/334167902 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca663c5f70 con 0x55ca666260a0 2024-05-09T08:55:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.696+0000 7fec6ffff640 1 --2- 172.21.15.98:0/334167902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca666260a0 0x55ca6662cbb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fec68004ce0 tx=0x7fec68004d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5f93784617645c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.697+0000 7fec4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:55:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.699+0000 7fec4f7fe640 1 -- 172.21.15.98:0/334167902 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6801e050 con 0x55ca666260a0 2024-05-09T08:55:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.699+0000 7fec7979b740 1 -- 172.21.15.98:0/334167902 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec68004030 con 0x55ca666260a0 2024-05-09T08:55:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.699+0000 7fec4f7fe640 1 -- 172.21.15.98:0/334167902 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fec680043f0 con 0x55ca666260a0 2024-05-09T08:55:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.704+0000 7fec4f7fe640 1 -- 172.21.15.98:0/334167902 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fec68033050 con 0x55ca666260a0 2024-05-09T08:55:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.704+0000 7fec7979b740 10 client.26198 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:02.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.704+0000 7fec7979b740 1 -- 172.21.15.98:0/334167902 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ca6625d170 con 0x55ca666260a0 2024-05-09T08:55:02.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.708+0000 7fec4f7fe640 1 -- 172.21.15.98:0/334167902 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec68004030 con 0x55ca666260a0 2024-05-09T08:55:02.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.709+0000 7fec4f7fe640 1 -- 172.21.15.98:0/334167902 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fec68025720 con 0x55ca666260a0 2024-05-09T08:55:02.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.709+0000 7fec7979b740 10 client.26198 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:02.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.709+0000 7fec7979b740 10 client.26198 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:02.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.709+0000 7fec7979b740 20 client.26198 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:02.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.710+0000 7fec7979b740 1 --2- 172.21.15.98:0/334167902 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ca6663f8f0 0x55ca66641ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.710+0000 7fec7979b740 4 client.26198 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:02.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.710+0000 7fec7979b740 1 -- 172.21.15.98:0/334167902 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca665970c0 con 0x55ca6663f8f0 2024-05-09T08:55:02.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.717+0000 7fec6f7fe640 1 --2- 172.21.15.98:0/334167902 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ca6663f8f0 0x55ca66641ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.736+0000 7fec6f7fe640 1 --2- 172.21.15.98:0/334167902 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ca6663f8f0 0x55ca66641ce0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea0430b14a246857 server_cookie=5308af93725c96e8 in_seq=0 out_seq=0 2024-05-09T08:55:02.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.737+0000 7fec4f7fe640 10 client.26198 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:02.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.742+0000 7fec4f7fe640 1 -- 172.21.15.98:0/334167902 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ca665970c0 con 0x55ca6663f8f0 2024-05-09T08:55:02.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.742+0000 7fec4f7fe640 10 client.26198 handle_command_reply: tid=0 2024-05-09T08:55:02.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.742+0000 7fec7979b740 1 client.26198 shutdown 2024-05-09T08:55:02.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.743+0000 7fec7979b740 1 -- 172.21.15.98:0/334167902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca666260a0 msgr2=0x55ca6662cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:02.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.743+0000 7fec7979b740 1 --2- 172.21.15.98:0/334167902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca666260a0 0x55ca6662cbb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fec68004ce0 tx=0x7fec68004d10 comp rx=0 tx=0).stop 2024-05-09T08:55:02.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.743+0000 7fec7979b740 1 -- 172.21.15.98:0/334167902 shutdown_connections 2024-05-09T08:55:02.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.743+0000 7fec7979b740 1 --2- 172.21.15.98:0/334167902 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ca6663f8f0 0x55ca66641ce0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.743+0000 7fec7979b740 1 --2- 172.21.15.98:0/334167902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca6662f730 0x55ca6662d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.743+0000 7fec7979b740 1 --2- 172.21.15.98:0/334167902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca666260a0 0x55ca6662cbb0 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fec68004ce0 tx=0x7fec68004d10 comp rx=0 tx=0).stop 2024-05-09T08:55:02.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.743+0000 7fec7979b740 1 --2- 172.21.15.98:0/334167902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 0x55ca6662c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.743+0000 7fec7979b740 1 -- 172.21.15.98:0/334167902 >> 172.21.15.98:0/334167902 conn(0x55ca665ed0a0 msgr2=0x55ca66626d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:02.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.744+0000 7fec4f7fe640 0 client.26198 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:02.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.751+0000 7fec7979b740 1 -- 172.21.15.98:0/334167902 shutdown_connections 2024-05-09T08:55:02.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.751+0000 7fec7979b740 1 -- 172.21.15.98:0/334167902 wait complete. 2024-05-09T08:55:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.752+0000 7fec7979b740 20 client.26198 trim_cache size 0 max 16384 2024-05-09T08:55:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.798+0000 7fec7979b740 1 Processor -- start 2024-05-09T08:55:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.799+0000 7fec7979b740 1 -- start start 2024-05-09T08:55:02.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.799+0000 7fec7979b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 0x55ca6661fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.799+0000 7fec7979b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 0x55ca6662cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.799+0000 7fec7979b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca6662eeb0 0x55ca66626b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.800+0000 7fec7979b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fec701bda20 con 0x55ca6662eeb0 2024-05-09T08:55:02.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.800+0000 7fec7979b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fec701bf560 con 0x55ca6662f730 2024-05-09T08:55:02.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.800+0000 7fec7979b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fec701be0f0 con 0x55ca666249b0 2024-05-09T08:55:02.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.800+0000 7fec6ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 0x55ca6662cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.800+0000 7fec6ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 0x55ca6662cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55270/0 (socket says 172.21.15.98:55270) 2024-05-09T08:55:02.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.800+0000 7fec6ffff640 1 -- 172.21.15.98:0/3081808914 learned_addr learned my addr 172.21.15.98:0/3081808914 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:02.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.801+0000 7fec6f7fe640 1 --2- 172.21.15.98:0/3081808914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca6662eeb0 0x55ca66626b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.802+0000 7fec6ffff640 1 -- 172.21.15.98:0/3081808914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 msgr2=0x55ca6661fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:02.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.802+0000 7fec6ffff640 1 --2- 172.21.15.98:0/3081808914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 0x55ca6661fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.802+0000 7fec6ffff640 1 -- 172.21.15.98:0/3081808914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca6662eeb0 msgr2=0x55ca66626b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:02.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.802+0000 7fec6ffff640 1 --2- 172.21.15.98:0/3081808914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca6662eeb0 0x55ca66626b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.802+0000 7fec6ffff640 1 -- 172.21.15.98:0/3081808914 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca66621120 con 0x55ca6662f730 2024-05-09T08:55:02.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.805+0000 7fec6ffff640 1 --2- 172.21.15.98:0/3081808914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 0x55ca6662cf10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fec68025ce0 tx=0x7fec68011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea1173f497656468 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:02.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.806+0000 7fec65ffb640 1 -- 172.21.15.98:0/3081808914 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6801e040 con 0x55ca6662f730 2024-05-09T08:55:02.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.806+0000 7fec65ffb640 1 -- 172.21.15.98:0/3081808914 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec6802a070 con 0x55ca6662f730 2024-05-09T08:55:02.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.808+0000 7fec7979b740 1 -- 172.21.15.98:0/3081808914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 msgr2=0x55ca6662cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:02.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.808+0000 7fec7979b740 1 --2- 172.21.15.98:0/3081808914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 0x55ca6662cf10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fec68025ce0 tx=0x7fec68011930 comp rx=0 tx=0).stop 2024-05-09T08:55:02.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.808+0000 7fec7979b740 1 -- 172.21.15.98:0/3081808914 shutdown_connections 2024-05-09T08:55:02.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.808+0000 7fec7979b740 1 --2- 172.21.15.98:0/3081808914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca6662eeb0 0x55ca66626b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.808+0000 7fec7979b740 1 --2- 172.21.15.98:0/3081808914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 0x55ca6662cf10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.808+0000 7fec7979b740 1 --2- 172.21.15.98:0/3081808914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca666249b0 0x55ca6661fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.808+0000 7fec7979b740 1 -- 172.21.15.98:0/3081808914 >> 172.21.15.98:0/3081808914 conn(0x55ca665ed0a0 msgr2=0x55ca66626480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:02.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.813+0000 7fec7979b740 1 -- 172.21.15.98:0/3081808914 shutdown_connections 2024-05-09T08:55:02.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.814+0000 7fec7979b740 1 -- 172.21.15.98:0/3081808914 wait complete. 2024-05-09T08:55:02.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.814+0000 7fec7979b740 1 -- start start 2024-05-09T08:55:02.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.815+0000 7fec7979b740 1 Processor -- start 2024-05-09T08:55:02.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.815+0000 7fec7979b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca666249b0 0x55ca66633820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.815+0000 7fec7979b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca6662eeb0 0x55ca66633d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.815+0000 7fec7979b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 0x55ca666a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.815+0000 7fec7979b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fec701bee90 con 0x55ca666249b0 2024-05-09T08:55:02.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.815+0000 7fec7979b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fec701be7c0 con 0x55ca6662f730 2024-05-09T08:55:02.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.815+0000 7fec7979b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fec701bbab0 con 0x55ca6662eeb0 2024-05-09T08:55:02.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.815+0000 7fec7979b740 10 client.0 _ll_register_callbacks cb 0x7fec6801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:02.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.816+0000 7fec6ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca6662eeb0 0x55ca66633d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.816+0000 7fec6ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca6662eeb0 0x55ca66633d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:51234/0 (socket says 172.21.15.98:51234) 2024-05-09T08:55:02.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.816+0000 7fec6ffff640 1 -- 172.21.15.98:0/2302995003 learned_addr learned my addr 172.21.15.98:0/2302995003 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:02.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.816+0000 7fec6f7fe640 1 --2- 172.21.15.98:0/2302995003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 0x55ca666a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.821+0000 7fec6ffff640 1 -- 172.21.15.98:0/2302995003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 msgr2=0x55ca666a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:02.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.821+0000 7fec6ffff640 1 --2- 172.21.15.98:0/2302995003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 0x55ca666a4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.821+0000 7fec6ffff640 1 -- 172.21.15.98:0/2302995003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca666249b0 msgr2=0x55ca66633820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:02.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.821+0000 7fec6ffff640 1 --2- 172.21.15.98:0/2302995003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca666249b0 0x55ca66633820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.821+0000 7fec6ffff640 1 -- 172.21.15.98:0/2302995003 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca666a53a0 con 0x55ca6662eeb0 2024-05-09T08:55:02.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.822+0000 7fec4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:55:02.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.822+0000 7fec6ffff640 1 --2- 172.21.15.98:0/2302995003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca6662eeb0 0x55ca66633d60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fec68027040 tx=0x7fec680194d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd81d98c39df9e49 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:02.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.823+0000 7fec4f7fe640 1 -- 172.21.15.98:0/2302995003 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6801e040 con 0x55ca6662eeb0 2024-05-09T08:55:02.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.823+0000 7fec4f7fe640 1 -- 172.21.15.98:0/2302995003 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fec680043f0 con 0x55ca6662eeb0 2024-05-09T08:55:02.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.823+0000 7fec7979b740 1 -- 172.21.15.98:0/2302995003 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec6802a070 con 0x55ca6662eeb0 2024-05-09T08:55:02.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.824+0000 7fec4f7fe640 1 -- 172.21.15.98:0/2302995003 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fec68045050 con 0x55ca6662eeb0 2024-05-09T08:55:02.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.824+0000 7fec7979b740 10 client.26161 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:02.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.824+0000 7fec7979b740 1 -- 172.21.15.98:0/2302995003 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ca666a6580 con 0x55ca6662eeb0 2024-05-09T08:55:02.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.825+0000 7fec4f7fe640 1 -- 172.21.15.98:0/2302995003 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec6802a070 con 0x55ca6662eeb0 2024-05-09T08:55:02.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.825+0000 7fec4f7fe640 1 -- 172.21.15.98:0/2302995003 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fec68019ce0 con 0x55ca6662eeb0 2024-05-09T08:55:02.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.825+0000 7fec7979b740 10 client.26161 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:02.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.825+0000 7fec7979b740 10 client.26161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:02.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.825+0000 7fec7979b740 20 client.26161 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:02.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.825+0000 7fec7979b740 1 --2- 172.21.15.98:0/2302995003 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ca666a6bb0 0x55ca666a8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:02.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.825+0000 7fec7979b740 4 client.26161 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:55:02.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.825+0000 7fec7979b740 1 -- 172.21.15.98:0/2302995003 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ca66591410 con 0x55ca666a6bb0 2024-05-09T08:55:02.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.826+0000 7fec6f7fe640 1 --2- 172.21.15.98:0/2302995003 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ca666a6bb0 0x55ca666a8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:02.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.829+0000 7fec6f7fe640 1 --2- 172.21.15.98:0/2302995003 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ca666a6bb0 0x55ca666a8fd0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21905b3b926a776d server_cookie=4af171f3371b9cb6 in_seq=0 out_seq=0 2024-05-09T08:55:02.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.829+0000 7fec4f7fe640 10 client.26161 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:02.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.831+0000 7fec4f7fe640 1 -- 172.21.15.98:0/2302995003 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ca66591410 con 0x55ca666a6bb0 2024-05-09T08:55:02.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.831+0000 7fec4f7fe640 10 client.26161 handle_command_reply: tid=0 2024-05-09T08:55:02.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.831+0000 7fec7979b740 1 client.26161 shutdown 2024-05-09T08:55:02.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.836+0000 7fec7979b740 1 -- 172.21.15.98:0/2302995003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca6662eeb0 msgr2=0x55ca66633d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:02.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.836+0000 7fec7979b740 1 --2- 172.21.15.98:0/2302995003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca6662eeb0 0x55ca66633d60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fec68027040 tx=0x7fec680194d0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.840+0000 7fec7979b740 1 -- 172.21.15.98:0/2302995003 shutdown_connections 2024-05-09T08:55:02.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.840+0000 7fec7979b740 1 --2- 172.21.15.98:0/2302995003 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ca666a6bb0 0x55ca666a8fd0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.840+0000 7fec7979b740 1 --2- 172.21.15.98:0/2302995003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca6662f730 0x55ca666a4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.840+0000 7fec7979b740 1 --2- 172.21.15.98:0/2302995003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ca6662eeb0 0x55ca66633d60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.840+0000 7fec7979b740 1 --2- 172.21.15.98:0/2302995003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca666249b0 0x55ca66633820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.840+0000 7fec7979b740 1 -- 172.21.15.98:0/2302995003 >> 172.21.15.98:0/2302995003 conn(0x55ca665ed0a0 msgr2=0x55ca6662e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:02.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.840+0000 7fec4f7fe640 0 client.26161 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:02.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.840+0000 7fec7979b740 1 -- 172.21.15.98:0/2302995003 shutdown_connections 2024-05-09T08:55:02.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.844+0000 7fec7979b740 1 -- 172.21.15.98:0/2302995003 wait complete. 2024-05-09T08:55:02.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.844+0000 7fec7979b740 20 client.26161 trim_cache size 0 max 16384 2024-05-09T08:55:02.853 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:55:02.854 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:55:02.854 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "6db6271a-97b9-4e89-a3eb-9bf525bdbe34", 2024-05-09T08:55:02.854 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:55:02.854 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:55:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.862+0000 7fec769b6640 1 -- 172.21.15.98:0/3156852545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fec5c0a2f00 msgr2=0x7fec5c0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.862+0000 7fec769b6640 1 --2- 172.21.15.98:0/3156852545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fec5c0a2f00 0x7fec5c0a53c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fec580bd010 tx=0x7fec5802a400 comp rx=0 tx=0).stop 2024-05-09T08:55:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.862+0000 7fec769b6640 1 -- 172.21.15.98:0/3156852545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec70073650 msgr2=0x7fec7006c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.862+0000 7fec769b6640 1 --2- 172.21.15.98:0/3156852545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec70073650 0x7fec7006c660 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fec5802a970 tx=0x7fec58003b90 comp rx=0 tx=0).stop 2024-05-09T08:55:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.863+0000 7fec769b6640 1 -- 172.21.15.98:0/3156852545 shutdown_connections 2024-05-09T08:55:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.863+0000 7fec769b6640 1 --2- 172.21.15.98:0/3156852545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fec5c0a2f00 0x7fec5c0a53c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.863+0000 7fec769b6640 1 --2- 172.21.15.98:0/3156852545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec70073650 0x7fec7006c660 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.863+0000 7fec769b6640 1 --2- 172.21.15.98:0/3156852545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec7010d130 0x7fec7006c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.863+0000 7fec769b6640 1 --2- 172.21.15.98:0/3156852545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec701080b0 0x7fec70073000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.863+0000 7fec769b6640 1 -- 172.21.15.98:0/3156852545 >> 172.21.15.98:0/3156852545 conn(0x7fec70075d70 msgr2=0x7fec70078070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.866+0000 7fec769b6640 1 -- 172.21.15.98:0/3156852545 shutdown_connections 2024-05-09T08:55:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:02.866+0000 7fec769b6640 1 -- 172.21.15.98:0/3156852545 wait complete. 2024-05-09T08:55:02.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:55:02.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:55:03.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.150+0000 7f7604e2a640 1 -- 172.21.15.98:0/4151801816 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f760010e0b0 msgr2=0x7f760010e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:03.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.150+0000 7f7604e2a640 1 --2- 172.21.15.98:0/4151801816 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f760010e0b0 0x7f760010e530 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f75f8009c10 tx=0x7f75f802a720 comp rx=0 tx=0).stop 2024-05-09T08:55:03.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.152+0000 7f7604e2a640 1 -- 172.21.15.98:0/4151801816 shutdown_connections 2024-05-09T08:55:03.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.152+0000 7f7604e2a640 1 --2- 172.21.15.98:0/4151801816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f760011b010 0x7f760011d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.152+0000 7f7604e2a640 1 --2- 172.21.15.98:0/4151801816 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f760010e0b0 0x7f760010e530 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.152+0000 7f7604e2a640 1 --2- 172.21.15.98:0/4151801816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7600110080 0x7f760010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.152+0000 7f7604e2a640 1 -- 172.21.15.98:0/4151801816 >> 172.21.15.98:0/4151801816 conn(0x7f760006c940 msgr2=0x7f760006cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.156+0000 7f7604e2a640 1 -- 172.21.15.98:0/4151801816 shutdown_connections 2024-05-09T08:55:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.156+0000 7f7604e2a640 1 -- 172.21.15.98:0/4151801816 wait complete. 2024-05-09T08:55:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.157+0000 7f7604e2a640 1 Processor -- start 2024-05-09T08:55:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.158+0000 7f7604e2a640 1 -- start start 2024-05-09T08:55:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.158+0000 7f7604e2a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760010e0b0 0x7f760011a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.158+0000 7f7604e2a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7600110080 0x7f76001148e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.158+0000 7f7604e2a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f760011b010 0x7f7600114e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.158+0000 7f7604e2a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7600111320 con 0x7f760010e0b0 2024-05-09T08:55:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.158+0000 7f7604e2a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f76001111a0 con 0x7f760011b010 2024-05-09T08:55:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.158+0000 7f7604e2a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f76001114a0 con 0x7f7600110080 2024-05-09T08:55:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.159+0000 7f75fdd9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7600110080 0x7f76001148e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.159+0000 7f75fdd9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7600110080 0x7f76001148e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:51254/0 (socket says 172.21.15.98:51254) 2024-05-09T08:55:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.159+0000 7f75fdd9b640 1 -- 172.21.15.98:0/3439813745 learned_addr learned my addr 172.21.15.98:0/3439813745 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.159+0000 7f75fed9d640 1 --2- 172.21.15.98:0/3439813745 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f760011b010 0x7f7600114e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.160+0000 7f75fed9d640 1 -- 172.21.15.98:0/3439813745 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7600110080 msgr2=0x7f76001148e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.160+0000 7f75fed9d640 1 --2- 172.21.15.98:0/3439813745 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7600110080 0x7f76001148e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.160+0000 7f75fed9d640 1 -- 172.21.15.98:0/3439813745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760010e0b0 msgr2=0x7f760011a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.160+0000 7f75fed9d640 1 --2- 172.21.15.98:0/3439813745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760010e0b0 0x7f760011a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.160+0000 7f75fed9d640 1 -- 172.21.15.98:0/3439813745 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76001156e0 con 0x7f760011b010 2024-05-09T08:55:03.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.161+0000 7f75fed9d640 1 --2- 172.21.15.98:0/3439813745 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f760011b010 0x7f7600114e20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009fd0 tx=0x7f75f000eba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.161+0000 7f75ef7fe640 1 -- 172.21.15.98:0/3439813745 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f0004220 con 0x7f760011b010 2024-05-09T08:55:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.161+0000 7f7604e2a640 1 -- 172.21.15.98:0/3439813745 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76001bbc10 con 0x7f760011b010 2024-05-09T08:55:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.162+0000 7f7604e2a640 1 -- 172.21.15.98:0/3439813745 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76001bc130 con 0x7f760011b010 2024-05-09T08:55:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.162+0000 7f75ef7fe640 1 -- 172.21.15.98:0/3439813745 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75f000cd90 con 0x7f760011b010 2024-05-09T08:55:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.162+0000 7f75ef7fe640 1 -- 172.21.15.98:0/3439813745 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f0004a00 con 0x7f760011b010 2024-05-09T08:55:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.165+0000 7f7607c14740 1 Processor -- start 2024-05-09T08:55:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.168+0000 7f7607c14740 1 -- start start 2024-05-09T08:55:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.168+0000 7f7607c14740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62d9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.168+0000 7f7607c14740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62d9ba0 0x564bc62e3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.168+0000 7f7607c14740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62e6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.168+0000 7f7607c14740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564bc623ed00 con 0x564bc62e4270 2024-05-09T08:55:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.168+0000 7f7607c14740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564bc61e64c0 con 0x564bc62d9ba0 2024-05-09T08:55:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.168+0000 7f7607c14740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564bc623acc0 con 0x564bc62d9280 2024-05-09T08:55:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.168+0000 7f75ef7fe640 1 -- 172.21.15.98:0/3439813745 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f75f0004ba0 con 0x7f760011b010 2024-05-09T08:55:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.169+0000 7f75fdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62e6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.169+0000 7f75fdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62e6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37532/0 (socket says 172.21.15.98:37532) 2024-05-09T08:55:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.169+0000 7f75fdd9b640 1 -- 172.21.15.98:0/2792550351 learned_addr learned my addr 172.21.15.98:0/2792550351 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.169+0000 7f75fdd9b640 1 -- 172.21.15.98:0/2792550351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 msgr2=0x564bc62d9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.170+0000 7f75fdd9b640 1 --2- 172.21.15.98:0/2792550351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62d9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.170+0000 7f75fdd9b640 1 -- 172.21.15.98:0/2792550351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62d9ba0 msgr2=0x564bc62e3d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:55:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.170+0000 7f75fdd9b640 1 --2- 172.21.15.98:0/2792550351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62d9ba0 0x564bc62e3d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.170+0000 7f75fdd9b640 1 -- 172.21.15.98:0/2792550351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bc601c7d0 con 0x564bc62e4270 2024-05-09T08:55:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.170+0000 7f75fdd9b640 1 --2- 172.21.15.98:0/2792550351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62e6660 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f75f8009b30 tx=0x7f75f8030ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c64fd2ce2c1606 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.171+0000 7f75ef7fe640 1 --2- 172.21.15.98:0/3439813745 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f75e80a2e00 0x7f75e80a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.171+0000 7f75ef7fe640 1 -- 172.21.15.98:0/3439813745 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f75f00cf520 con 0x7f760011b010 2024-05-09T08:55:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.171+0000 7f75ecff9640 1 -- 172.21.15.98:0/2792550351 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f8042020 con 0x564bc62e4270 2024-05-09T08:55:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.172+0000 7f75ecff9640 1 -- 172.21.15.98:0/2792550351 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75f8003e50 con 0x564bc62e4270 2024-05-09T08:55:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.172+0000 7f75ecff9640 1 -- 172.21.15.98:0/2792550351 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f8036040 con 0x564bc62e4270 2024-05-09T08:55:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.173+0000 7f7607c14740 1 -- 172.21.15.98:0/2792550351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 msgr2=0x564bc62e6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.173+0000 7f7607c14740 1 --2- 172.21.15.98:0/2792550351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62e6660 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f75f8009b30 tx=0x7f75f8030ac0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.173+0000 7f7607c14740 1 -- 172.21.15.98:0/2792550351 shutdown_connections 2024-05-09T08:55:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.173+0000 7f7607c14740 1 --2- 172.21.15.98:0/2792550351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62e6660 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.173+0000 7f7607c14740 1 --2- 172.21.15.98:0/2792550351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62d9ba0 0x564bc62e3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.173+0000 7f7607c14740 1 --2- 172.21.15.98:0/2792550351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62d9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.173+0000 7f7607c14740 1 -- 172.21.15.98:0/2792550351 >> 172.21.15.98:0/2792550351 conn(0x564bc62d55a0 msgr2=0x564bc62d5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.174+0000 7f7607c14740 1 -- 172.21.15.98:0/2792550351 shutdown_connections 2024-05-09T08:55:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.174+0000 7f7607c14740 1 -- 172.21.15.98:0/2792550351 wait complete. 2024-05-09T08:55:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.174+0000 7f7607c14740 1 -- start start 2024-05-09T08:55:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.175+0000 7f75fe59c640 1 --2- 172.21.15.98:0/3439813745 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f75e80a2e00 0x7f75e80a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.175+0000 7f7607c14740 1 Processor -- start 2024-05-09T08:55:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.175+0000 7f7607c14740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62e1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.175+0000 7f7607c14740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62d9ba0 0x564bc62e1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.176+0000 7f7607c14740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62e4270 0x564bc62e1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.176+0000 7f7607c14740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564bc62e7050 con 0x564bc62d9ba0 2024-05-09T08:55:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.176+0000 7f7607c14740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564bc61f9b90 con 0x564bc62e4270 2024-05-09T08:55:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.176+0000 7f7607c14740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564bc62e71d0 con 0x564bc62d9280 2024-05-09T08:55:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.176+0000 7f7607c14740 10 client.0 _ll_register_callbacks cb 0x564bc6235da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.176+0000 7f75fed9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62e4270 0x564bc62e1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.176+0000 7f75fed9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62e4270 0x564bc62e1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55288/0 (socket says 172.21.15.98:55288) 2024-05-09T08:55:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.176+0000 7f75fed9d640 1 -- 172.21.15.98:0/910470528 learned_addr learned my addr 172.21.15.98:0/910470528 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.178+0000 7f75fdd9b640 1 --2- 172.21.15.98:0/910470528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62d9ba0 0x564bc62e1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.178+0000 7f75fed9d640 1 -- 172.21.15.98:0/910470528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 msgr2=0x564bc62e1240 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:55:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.178+0000 7f75fed9d640 1 --2- 172.21.15.98:0/910470528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62e1240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.178+0000 7f75fed9d640 1 -- 172.21.15.98:0/910470528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62d9ba0 msgr2=0x564bc62e1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.178+0000 7f75fed9d640 1 --2- 172.21.15.98:0/910470528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62d9ba0 0x564bc62e1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.178+0000 7f75fed9d640 1 -- 172.21.15.98:0/910470528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bc5f523f0 con 0x564bc62e4270 2024-05-09T08:55:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.179+0000 7f75fed9d640 1 --2- 172.21.15.98:0/910470528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62e4270 0x564bc62e1cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f75f000cfd0 tx=0x7f75f009b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee9d35c06bf6d20a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.179+0000 7f75e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:55:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.179+0000 7f75fe59c640 1 --2- 172.21.15.98:0/3439813745 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f75e80a2e00 0x7f75e80a52c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x564bc62da840 tx=0x7f75f4005c10 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.180+0000 7f75e5ffb640 1 -- 172.21.15.98:0/910470528 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f00b4060 con 0x564bc62e4270 2024-05-09T08:55:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.181+0000 7f7607c14740 1 -- 172.21.15.98:0/910470528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75f8003e50 con 0x564bc62e4270 2024-05-09T08:55:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.181+0000 7f75e5ffb640 1 -- 172.21.15.98:0/910470528 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f75f00ccad0 con 0x564bc62e4270 2024-05-09T08:55:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.184+0000 7f75e5ffb640 1 -- 172.21.15.98:0/910470528 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f75f001c040 con 0x564bc62e4270 2024-05-09T08:55:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.186+0000 7f7607c14740 10 client.26213 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.186+0000 7f7607c14740 1 -- 172.21.15.98:0/910470528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564bc5f120f0 con 0x564bc62e4270 2024-05-09T08:55:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.187+0000 7f75e5ffb640 1 -- 172.21.15.98:0/910470528 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75f8003e50 con 0x564bc62e4270 2024-05-09T08:55:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.187+0000 7f75e5ffb640 1 -- 172.21.15.98:0/910470528 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f75f001f030 con 0x564bc62e4270 2024-05-09T08:55:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.187+0000 7f7607c14740 10 client.26213 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.187+0000 7f7607c14740 10 client.26213 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.187+0000 7f7607c14740 20 client.26213 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.188+0000 7f7607c14740 1 --2- 172.21.15.98:0/910470528 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564bc62f4b50 0x564bc62f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.188+0000 7f7607c14740 4 client.26213 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.188+0000 7f7607c14740 1 -- 172.21.15.98:0/910470528 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564bc624c430 con 0x564bc62f4b50 2024-05-09T08:55:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.190+0000 7f75fe59c640 1 --2- 172.21.15.98:0/910470528 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564bc62f4b50 0x564bc62f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.195+0000 7f75fe59c640 1 --2- 172.21.15.98:0/910470528 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564bc62f4b50 0x564bc62f6f70 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2ee85f46781ecf8 server_cookie=8dfdd484ef21daa3 in_seq=0 out_seq=0 2024-05-09T08:55:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.195+0000 7f75e5ffb640 10 client.26213 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.198+0000 7f75e5ffb640 1 -- 172.21.15.98:0/910470528 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564bc624c430 con 0x564bc62f4b50 2024-05-09T08:55:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.198+0000 7f75e5ffb640 10 client.26213 handle_command_reply: tid=0 2024-05-09T08:55:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.198+0000 7f7607c14740 1 client.26213 shutdown 2024-05-09T08:55:03.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.203+0000 7f7607c14740 1 -- 172.21.15.98:0/910470528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62e4270 msgr2=0x564bc62e1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:03.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.203+0000 7f7607c14740 1 --2- 172.21.15.98:0/910470528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62e4270 0x564bc62e1cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f75f000cfd0 tx=0x7f75f009b040 comp rx=0 tx=0).stop 2024-05-09T08:55:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.206+0000 7f7607c14740 1 -- 172.21.15.98:0/910470528 shutdown_connections 2024-05-09T08:55:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.207+0000 7f7607c14740 1 --2- 172.21.15.98:0/910470528 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564bc62f4b50 0x564bc62f6f70 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.207+0000 7f7607c14740 1 --2- 172.21.15.98:0/910470528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62e4270 0x564bc62e1cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.207+0000 7f7607c14740 1 --2- 172.21.15.98:0/910470528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62d9ba0 0x564bc62e1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.207+0000 7f7607c14740 1 --2- 172.21.15.98:0/910470528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62e1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.207+0000 7f7607c14740 1 -- 172.21.15.98:0/910470528 >> 172.21.15.98:0/910470528 conn(0x564bc62d55a0 msgr2=0x564bc62d47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.207+0000 7f75e5ffb640 0 client.26213 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:03.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.207+0000 7f7607c14740 1 -- 172.21.15.98:0/910470528 shutdown_connections 2024-05-09T08:55:03.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.209+0000 7f7607c14740 1 -- 172.21.15.98:0/910470528 wait complete. 2024-05-09T08:55:03.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.211+0000 7f7607c14740 20 client.26213 trim_cache size 0 max 16384 2024-05-09T08:55:03.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.260+0000 7f7607c14740 1 Processor -- start 2024-05-09T08:55:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.261+0000 7f7607c14740 1 -- start start 2024-05-09T08:55:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.262+0000 7f7607c14740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62d4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.262+0000 7f7607c14740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62d9ba0 0x564bc62e1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.262+0000 7f7607c14740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62fc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.262+0000 7f7607c14740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75e0010090 con 0x564bc62e4270 2024-05-09T08:55:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.262+0000 7f7607c14740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f75e0011bd0 con 0x564bc62d9ba0 2024-05-09T08:55:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.262+0000 7f7607c14740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f75e0010760 con 0x564bc62d9280 2024-05-09T08:55:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.262+0000 7f75fdd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62d9ba0 0x564bc62e1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.262+0000 7f75fdd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62d9ba0 0x564bc62e1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:55304/0 (socket says 172.21.15.98:55304) 2024-05-09T08:55:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.262+0000 7f75fdd9b640 1 -- 172.21.15.98:0/1331626040 learned_addr learned my addr 172.21.15.98:0/1331626040 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.262+0000 7f75fe59c640 1 --2- 172.21.15.98:0/1331626040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62d4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.262+0000 7f75fed9d640 1 --2- 172.21.15.98:0/1331626040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62fc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.263+0000 7f75fe59c640 1 -- 172.21.15.98:0/1331626040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62d9ba0 msgr2=0x564bc62e1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.263+0000 7f75fe59c640 1 --2- 172.21.15.98:0/1331626040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62d9ba0 0x564bc62e1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.263+0000 7f75fe59c640 1 -- 172.21.15.98:0/1331626040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 msgr2=0x564bc62fc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.263+0000 7f75fe59c640 1 --2- 172.21.15.98:0/1331626040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62fc880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.263+0000 7f75fe59c640 1 -- 172.21.15.98:0/1331626040 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bc62f75f0 con 0x564bc62d9280 2024-05-09T08:55:03.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.263+0000 7f75fe59c640 1 --2- 172.21.15.98:0/1331626040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62d4f60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x564bc62f4f30 tx=0x7f75f400f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d70be0513d3cbe2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:03.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.263+0000 7f75e7fff640 1 -- 172.21.15.98:0/1331626040 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f4027030 con 0x564bc62d9280 2024-05-09T08:55:03.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.264+0000 7f75e7fff640 1 -- 172.21.15.98:0/1331626040 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75f4016040 con 0x564bc62d9280 2024-05-09T08:55:03.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.264+0000 7f75e7fff640 1 -- 172.21.15.98:0/1331626040 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f4022820 con 0x564bc62d9280 2024-05-09T08:55:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.265+0000 7f7607c14740 1 -- 172.21.15.98:0/1331626040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 msgr2=0x564bc62d4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.265+0000 7f7607c14740 1 --2- 172.21.15.98:0/1331626040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62d4f60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x564bc62f4f30 tx=0x7f75f400f000 comp rx=0 tx=0).stop 2024-05-09T08:55:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.265+0000 7f7607c14740 1 -- 172.21.15.98:0/1331626040 shutdown_connections 2024-05-09T08:55:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.265+0000 7f7607c14740 1 --2- 172.21.15.98:0/1331626040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62fc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.265+0000 7f7607c14740 1 --2- 172.21.15.98:0/1331626040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62d9ba0 0x564bc62e1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.265+0000 7f7607c14740 1 --2- 172.21.15.98:0/1331626040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62d4f60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.265+0000 7f7607c14740 1 -- 172.21.15.98:0/1331626040 >> 172.21.15.98:0/1331626040 conn(0x564bc62d55a0 msgr2=0x564bc62db260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.267+0000 7f7607c14740 1 -- 172.21.15.98:0/1331626040 shutdown_connections 2024-05-09T08:55:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.267+0000 7f7607c14740 1 -- 172.21.15.98:0/1331626040 wait complete. 2024-05-09T08:55:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.267+0000 7f7607c14740 1 -- start start 2024-05-09T08:55:03.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.267+0000 7f7607c14740 1 Processor -- start 2024-05-09T08:55:03.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.268+0000 7f7607c14740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62ec900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.268+0000 7f7607c14740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62d9ba0 0x564bc62ddd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.268+0000 7f7607c14740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62de280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.268+0000 7f7607c14740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75e0011500 con 0x564bc62e4270 2024-05-09T08:55:03.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.268+0000 7f7607c14740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f75e0010e30 con 0x564bc62d9ba0 2024-05-09T08:55:03.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.268+0000 7f7607c14740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f75e000e070 con 0x564bc62d9280 2024-05-09T08:55:03.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.268+0000 7f7607c14740 10 client.0 _ll_register_callbacks cb 0x7f75f0005760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:03.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.268+0000 7f75fe59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62ec900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:03.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.268+0000 7f75fed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62de280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:03.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.268+0000 7f75fed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62de280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37564/0 (socket says 172.21.15.98:37564) 2024-05-09T08:55:03.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.268+0000 7f75fed9d640 1 -- 172.21.15.98:0/296707989 learned_addr learned my addr 172.21.15.98:0/296707989 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:03.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.269+0000 7f75fed9d640 1 -- 172.21.15.98:0/296707989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 msgr2=0x564bc62ec900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:03.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.269+0000 7f75fed9d640 1 --2- 172.21.15.98:0/296707989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62ec900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.269+0000 7f75fed9d640 1 -- 172.21.15.98:0/296707989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62d9ba0 msgr2=0x564bc62ddd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:03.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.269+0000 7f75fed9d640 1 --2- 172.21.15.98:0/296707989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62d9ba0 0x564bc62ddd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.269+0000 7f75fed9d640 1 -- 172.21.15.98:0/296707989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bc62deab0 con 0x564bc62e4270 2024-05-09T08:55:03.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.270+0000 7f75fed9d640 1 --2- 172.21.15.98:0/296707989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62de280 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f75f000c6a0 tx=0x7f75f0005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97fc9a36322b569 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:03.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.270+0000 7f75e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:55:03.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.271+0000 7f75e5ffb640 1 -- 172.21.15.98:0/296707989 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f00b4040 con 0x564bc62e4270 2024-05-09T08:55:03.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.271+0000 7f7607c14740 1 -- 172.21.15.98:0/296707989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75f4016040 con 0x564bc62e4270 2024-05-09T08:55:03.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.271+0000 7f75e5ffb640 1 -- 172.21.15.98:0/296707989 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f75f00afd40 con 0x564bc62e4270 2024-05-09T08:55:03.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.272+0000 7f75e5ffb640 1 -- 172.21.15.98:0/296707989 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f75f009b3f0 con 0x564bc62e4270 2024-05-09T08:55:03.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.273+0000 7f7607c14740 10 client.16923 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:03.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.273+0000 7f7607c14740 1 -- 172.21.15.98:0/296707989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564bc635b190 con 0x564bc62e4270 2024-05-09T08:55:03.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.274+0000 7f75e5ffb640 1 -- 172.21.15.98:0/296707989 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75f4016040 con 0x564bc62e4270 2024-05-09T08:55:03.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.274+0000 7f75e5ffb640 1 -- 172.21.15.98:0/296707989 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f75f001f040 con 0x564bc62e4270 2024-05-09T08:55:03.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.274+0000 7f7607c14740 10 client.16923 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:03.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.274+0000 7f7607c14740 10 client.16923 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:03.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.274+0000 7f7607c14740 20 client.16923 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:03.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.274+0000 7f7607c14740 1 --2- 172.21.15.98:0/296707989 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564bc635b820 0x564bc635dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:03.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.274+0000 7f7607c14740 4 client.16923 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:55:03.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.274+0000 7f7607c14740 1 -- 172.21.15.98:0/296707989 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564bc6246780 con 0x564bc635b820 2024-05-09T08:55:03.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.276+0000 7f75fe59c640 1 --2- 172.21.15.98:0/296707989 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564bc635b820 0x564bc635dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:03.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.281+0000 7f75fe59c640 1 --2- 172.21.15.98:0/296707989 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564bc635b820 0x564bc635dc40 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12f7851ea4ab6b6c server_cookie=efd27db809314752 in_seq=0 out_seq=0 2024-05-09T08:55:03.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.281+0000 7f75e5ffb640 10 client.16923 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:03.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.282+0000 7f75e5ffb640 1 -- 172.21.15.98:0/296707989 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564bc6246780 con 0x564bc635b820 2024-05-09T08:55:03.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.282+0000 7f75e5ffb640 10 client.16923 handle_command_reply: tid=0 2024-05-09T08:55:03.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.282+0000 7f7607c14740 1 client.16923 shutdown 2024-05-09T08:55:03.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.282+0000 7f7607c14740 1 -- 172.21.15.98:0/296707989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 msgr2=0x564bc62de280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:03.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.282+0000 7f7607c14740 1 --2- 172.21.15.98:0/296707989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62de280 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f75f000c6a0 tx=0x7f75f0005390 comp rx=0 tx=0).stop 2024-05-09T08:55:03.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.284+0000 7f7607c14740 1 -- 172.21.15.98:0/296707989 shutdown_connections 2024-05-09T08:55:03.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.284+0000 7f7607c14740 1 --2- 172.21.15.98:0/296707989 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564bc635b820 0x564bc635dc40 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.284+0000 7f7607c14740 1 --2- 172.21.15.98:0/296707989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bc62e4270 0x564bc62de280 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.284+0000 7f7607c14740 1 --2- 172.21.15.98:0/296707989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564bc62d9ba0 0x564bc62ddd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.284+0000 7f7607c14740 1 --2- 172.21.15.98:0/296707989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564bc62d9280 0x564bc62ec900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.284+0000 7f7607c14740 1 -- 172.21.15.98:0/296707989 >> 172.21.15.98:0/296707989 conn(0x564bc62d55a0 msgr2=0x564bc62e2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:03.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.284+0000 7f75e5ffb640 0 client.16923 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:03.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.284+0000 7f7607c14740 1 -- 172.21.15.98:0/296707989 shutdown_connections 2024-05-09T08:55:03.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.284+0000 7f7607c14740 1 -- 172.21.15.98:0/296707989 wait complete. 2024-05-09T08:55:03.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.285+0000 7f7607c14740 20 client.16923 trim_cache size 0 max 16384 2024-05-09T08:55:03.284 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:55:03.284 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:55:03.285 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:55:03.285 INFO:teuthology.orchestra.run.smithi098.stdout: "6db6271a-97b9-4e89-a3eb-9bf525bdbe34": { 2024-05-09T08:55:03.285 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:55:03.285 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "6db6271a-97b9-4e89-a3eb-9bf525bdbe34", 2024-05-09T08:55:03.285 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:55:03.285 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:55:03.285 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:55:03.285 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:55:03.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.286+0000 7f7604e2a640 1 -- 172.21.15.98:0/3439813745 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f75e80a2e00 msgr2=0x7f75e80a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:03.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.286+0000 7f7604e2a640 1 --2- 172.21.15.98:0/3439813745 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f75e80a2e00 0x7f75e80a52c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x564bc62da840 tx=0x7f75f4005c10 comp rx=0 tx=0).stop 2024-05-09T08:55:03.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.286+0000 7f7604e2a640 1 -- 172.21.15.98:0/3439813745 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f760011b010 msgr2=0x7f7600114e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:03.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.286+0000 7f7604e2a640 1 --2- 172.21.15.98:0/3439813745 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f760011b010 0x7f7600114e20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009fd0 tx=0x7f75f000eba0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.289+0000 7f7604e2a640 1 -- 172.21.15.98:0/3439813745 shutdown_connections 2024-05-09T08:55:03.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.289+0000 7f7604e2a640 1 --2- 172.21.15.98:0/3439813745 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f75e80a2e00 0x7f75e80a52c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.289+0000 7f7604e2a640 1 --2- 172.21.15.98:0/3439813745 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f760011b010 0x7f7600114e20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.289+0000 7f7604e2a640 1 --2- 172.21.15.98:0/3439813745 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7600110080 0x7f76001148e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.289+0000 7f7604e2a640 1 --2- 172.21.15.98:0/3439813745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760010e0b0 0x7f760011a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:03.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.289+0000 7f7604e2a640 1 -- 172.21.15.98:0/3439813745 >> 172.21.15.98:0/3439813745 conn(0x7f760006c940 msgr2=0x7f760011c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:03.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.289+0000 7f7604e2a640 1 -- 172.21.15.98:0/3439813745 shutdown_connections 2024-05-09T08:55:03.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:03.289+0000 7f7604e2a640 1 -- 172.21.15.98:0/3439813745 wait complete. 2024-05-09T08:55:03.307 INFO:tasks.cephfs.filesystem:scrub status for tag:6db6271a-97b9-4e89-a3eb-9bf525bdbe34 - {'path': '/', 'tag': '6db6271a-97b9-4e89-a3eb-9bf525bdbe34', 'options': 'recursive,force'} 2024-05-09T08:55:04.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:04 smithi142 ceph-mon[34625]: pgmap v137: 129 pgs: 129 active+clean; 30 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 644 MiB/s wr, 1.41k op/s 2024-05-09T08:55:04.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:04 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:55:04.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:04 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:55:04.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:04 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:55:04.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:04 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:55:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:04 smithi105 ceph-mon[34498]: pgmap v137: 129 pgs: 129 active+clean; 30 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 644 MiB/s wr, 1.41k op/s 2024-05-09T08:55:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:04 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:55:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:04 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:55:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:04 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:55:04.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:04 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:55:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:04 smithi098 ceph-mon[30092]: pgmap v137: 129 pgs: 129 active+clean; 30 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 644 MiB/s wr, 1.41k op/s 2024-05-09T08:55:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:04 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:55:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:04 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:55:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:04 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:55:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:04 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:55:06.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:06 smithi142 ceph-mon[34625]: pgmap v138: 129 pgs: 129 active+clean; 30 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 711 MiB/s wr, 1.54k op/s 2024-05-09T08:55:06.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:06 smithi105 ceph-mon[34498]: pgmap v138: 129 pgs: 129 active+clean; 30 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 711 MiB/s wr, 1.54k op/s 2024-05-09T08:55:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.563+0000 7f8060837640 1 -- 172.21.15.98:0/708457290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f805c10fef0 msgr2=0x7f805c10d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.563+0000 7f8060837640 1 --2- 172.21.15.98:0/708457290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f805c10fef0 0x7f805c10d9a0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f804c009900 tx=0x7f804c02a4d0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.564+0000 7f8060837640 1 -- 172.21.15.98:0/708457290 shutdown_connections 2024-05-09T08:55:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.564+0000 7f8060837640 1 --2- 172.21.15.98:0/708457290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f805c10e500 0x7f805c11d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.564+0000 7f8060837640 1 --2- 172.21.15.98:0/708457290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f805c1102d0 0x7f805c10dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.564+0000 7f8060837640 1 --2- 172.21.15.98:0/708457290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f805c10fef0 0x7f805c10d9a0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.564+0000 7f8060837640 1 -- 172.21.15.98:0/708457290 >> 172.21.15.98:0/708457290 conn(0x7f805c06c770 msgr2=0x7f805c06cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:06.566 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.567+0000 7f8060837640 1 -- 172.21.15.98:0/708457290 shutdown_connections 2024-05-09T08:55:06.570 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.570+0000 7f8060837640 1 -- 172.21.15.98:0/708457290 wait complete. 2024-05-09T08:55:06.570 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.571+0000 7f8060837640 1 Processor -- start 2024-05-09T08:55:06.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.571+0000 7f8060837640 1 -- start start 2024-05-09T08:55:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.572+0000 7f8060837640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f805c10e500 0x7f805c11c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.572+0000 7f8060837640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f805c10fef0 0x7f805c1156f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.572+0000 7f8060837640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f805c1102d0 0x7f805c115c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.572+0000 7f8060837640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f805c120a10 con 0x7f805c10fef0 2024-05-09T08:55:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.572+0000 7f8060837640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f805c120890 con 0x7f805c1102d0 2024-05-09T08:55:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.572+0000 7f8060837640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f805c120b90 con 0x7f805c10e500 2024-05-09T08:55:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.573+0000 7f805a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f805c10fef0 0x7f805c1156f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.573+0000 7f805a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f805c10fef0 0x7f805c1156f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57528/0 (socket says 172.21.15.98:57528) 2024-05-09T08:55:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.573+0000 7f805a59c640 1 -- 172.21.15.98:0/2252712300 learned_addr learned my addr 172.21.15.98:0/2252712300 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.573+0000 7f805b59e640 1 --2- 172.21.15.98:0/2252712300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f805c1102d0 0x7f805c115c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.573+0000 7f805ad9d640 1 --2- 172.21.15.98:0/2252712300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f805c10e500 0x7f805c11c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.574+0000 7f805a59c640 1 -- 172.21.15.98:0/2252712300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f805c10e500 msgr2=0x7f805c11c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.574+0000 7f805a59c640 1 --2- 172.21.15.98:0/2252712300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f805c10e500 0x7f805c11c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.574+0000 7f805a59c640 1 -- 172.21.15.98:0/2252712300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f805c1102d0 msgr2=0x7f805c115c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.574+0000 7f805a59c640 1 --2- 172.21.15.98:0/2252712300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f805c1102d0 0x7f805c115c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.574+0000 7f805a59c640 1 -- 172.21.15.98:0/2252712300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805c116270 con 0x7f805c10fef0 2024-05-09T08:55:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.575+0000 7f805a59c640 1 --2- 172.21.15.98:0/2252712300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f805c10fef0 0x7f805c1156f0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f805000b450 tx=0x7f805000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.576+0000 7f8043fff640 1 -- 172.21.15.98:0/2252712300 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050015020 con 0x7f805c10fef0 2024-05-09T08:55:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.576+0000 7f8060837640 1 -- 172.21.15.98:0/2252712300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f805c11a190 con 0x7f805c10fef0 2024-05-09T08:55:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.576+0000 7f8060837640 1 -- 172.21.15.98:0/2252712300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f805c11a6e0 con 0x7f805c10fef0 2024-05-09T08:55:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.578+0000 7f8043fff640 1 -- 172.21.15.98:0/2252712300 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8050011d70 con 0x7f805c10fef0 2024-05-09T08:55:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.578+0000 7f8043fff640 1 -- 172.21.15.98:0/2252712300 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805000f340 con 0x7f805c10fef0 2024-05-09T08:55:06.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.581+0000 7f8063e24740 1 Processor -- start 2024-05-09T08:55:06.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.582+0000 7f8063e24740 1 -- start start 2024-05-09T08:55:06.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.582+0000 7f8063e24740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae811600 0x5567ae8119e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.582+0000 7f8063e24740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 0x5567ae81c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.582+0000 7f8063e24740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae81c7e0 0x5567ae81ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.582+0000 7f8063e24740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5567ae776340 con 0x5567ae811f20 2024-05-09T08:55:06.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.582+0000 7f8063e24740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5567ae71e290 con 0x5567ae81c7e0 2024-05-09T08:55:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.582+0000 7f8063e24740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5567ae772300 con 0x5567ae811600 2024-05-09T08:55:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.583+0000 7f805ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 0x5567ae81c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.583+0000 7f805ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 0x5567ae81c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57544/0 (socket says 172.21.15.98:57544) 2024-05-09T08:55:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.583+0000 7f805ad9d640 1 -- 172.21.15.98:0/3235718413 learned_addr learned my addr 172.21.15.98:0/3235718413 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.583+0000 7f805a59c640 1 --2- 172.21.15.98:0/3235718413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae81c7e0 0x5567ae81ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.583+0000 7f805ad9d640 1 -- 172.21.15.98:0/3235718413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae811600 msgr2=0x5567ae8119e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.583+0000 7f805b59e640 1 --2- 172.21.15.98:0/3235718413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae811600 0x5567ae8119e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.584+0000 7f805ad9d640 1 --2- 172.21.15.98:0/3235718413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae811600 0x5567ae8119e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.584+0000 7f805ad9d640 1 -- 172.21.15.98:0/3235718413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae81c7e0 msgr2=0x5567ae81ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.584+0000 7f805ad9d640 1 --2- 172.21.15.98:0/3235718413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae81c7e0 0x5567ae81ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.584+0000 7f805ad9d640 1 -- 172.21.15.98:0/3235718413 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567ae7428e0 con 0x5567ae811f20 2024-05-09T08:55:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.584+0000 7f805b59e640 1 --2- 172.21.15.98:0/3235718413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae811600 0x5567ae8119e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:55:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.584+0000 7f805ad9d640 1 --2- 172.21.15.98:0/3235718413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 0x5567ae81c2a0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f804c009900 tx=0x7f804c031ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89fff15bdcbcf86 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:06.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.586+0000 7f80417fa640 1 -- 172.21.15.98:0/3235718413 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c040070 con 0x5567ae811f20 2024-05-09T08:55:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.586+0000 7f80417fa640 1 -- 172.21.15.98:0/3235718413 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f804c031780 con 0x5567ae811f20 2024-05-09T08:55:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.586+0000 7f80417fa640 1 -- 172.21.15.98:0/3235718413 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c035070 con 0x5567ae811f20 2024-05-09T08:55:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.587+0000 7f8063e24740 1 -- 172.21.15.98:0/3235718413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 msgr2=0x5567ae81c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.587+0000 7f8063e24740 1 --2- 172.21.15.98:0/3235718413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 0x5567ae81c2a0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f804c009900 tx=0x7f804c031ad0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.589+0000 7f8063e24740 1 -- 172.21.15.98:0/3235718413 shutdown_connections 2024-05-09T08:55:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.589+0000 7f8063e24740 1 --2- 172.21.15.98:0/3235718413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae81c7e0 0x5567ae81ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.589+0000 7f8063e24740 1 --2- 172.21.15.98:0/3235718413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 0x5567ae81c2a0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.589+0000 7f8063e24740 1 --2- 172.21.15.98:0/3235718413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae811600 0x5567ae8119e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.589+0000 7f8063e24740 1 -- 172.21.15.98:0/3235718413 >> 172.21.15.98:0/3235718413 conn(0x5567ae80db10 msgr2=0x5567ae80def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.589+0000 7f8043fff640 1 -- 172.21.15.98:0/2252712300 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8050018020 con 0x7f805c10fef0 2024-05-09T08:55:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.591+0000 7f8043fff640 1 --2- 172.21.15.98:0/2252712300 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f80440a2e10 0x7f80440a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.591+0000 7f8043fff640 1 -- 172.21.15.98:0/2252712300 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f80500cf4a0 con 0x7f805c10fef0 2024-05-09T08:55:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.592+0000 7f805ad9d640 1 --2- 172.21.15.98:0/2252712300 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f80440a2e10 0x7f80440a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:06 smithi098 ceph-mon[30092]: pgmap v138: 129 pgs: 129 active+clean; 30 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 711 MiB/s wr, 1.54k op/s 2024-05-09T08:55:06.592 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.592+0000 7f8063e24740 1 -- 172.21.15.98:0/3235718413 shutdown_connections 2024-05-09T08:55:06.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.592+0000 7f8063e24740 1 -- 172.21.15.98:0/3235718413 wait complete. 2024-05-09T08:55:06.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.593+0000 7f8063e24740 1 -- start start 2024-05-09T08:55:06.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.593+0000 7f8063e24740 1 Processor -- start 2024-05-09T08:55:06.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.594+0000 7f8063e24740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811600 0x5567ae819780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.594+0000 7f8063e24740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae811f20 0x5567ae819cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.594+0000 7f8063e24740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae81c7e0 0x5567ae81a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.594+0000 7f8063e24740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5567ae81f3b0 con 0x5567ae811600 2024-05-09T08:55:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.594+0000 7f8063e24740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5567ae731790 con 0x5567ae811f20 2024-05-09T08:55:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.594+0000 7f8063e24740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5567ae81f530 con 0x5567ae81c7e0 2024-05-09T08:55:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.594+0000 7f8063e24740 10 client.0 _ll_register_callbacks cb 0x5567ae76ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.594+0000 7f805ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811600 0x5567ae819780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.594+0000 7f805ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811600 0x5567ae819780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57548/0 (socket says 172.21.15.98:57548) 2024-05-09T08:55:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.594+0000 7f805ad9d640 1 -- 172.21.15.98:0/877874315 learned_addr learned my addr 172.21.15.98:0/877874315 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.594+0000 7f805ad9d640 1 --2- 172.21.15.98:0/2252712300 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f80440a2e10 0x7f80440a52d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f804c02aac0 tx=0x7f804c005710 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.594+0000 7f805a59c640 1 --2- 172.21.15.98:0/877874315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae811f20 0x5567ae819cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.595+0000 7f805b59e640 1 --2- 172.21.15.98:0/877874315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae81c7e0 0x5567ae81a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.595+0000 7f805ad9d640 1 -- 172.21.15.98:0/877874315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae81c7e0 msgr2=0x5567ae81a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.595+0000 7f805ad9d640 1 --2- 172.21.15.98:0/877874315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae81c7e0 0x5567ae81a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.595+0000 7f805ad9d640 1 -- 172.21.15.98:0/877874315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae811f20 msgr2=0x5567ae819cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.595+0000 7f805ad9d640 1 --2- 172.21.15.98:0/877874315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae811f20 0x5567ae819cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.595+0000 7f805ad9d640 1 -- 172.21.15.98:0/877874315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567ae48a210 con 0x5567ae811600 2024-05-09T08:55:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.595+0000 7f803eff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:55:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.595+0000 7f805ad9d640 1 --2- 172.21.15.98:0/877874315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811600 0x5567ae819780 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f804c005610 tx=0x7f804c02a9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5208be283074c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.596+0000 7f803eff5640 1 -- 172.21.15.98:0/877874315 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c040030 con 0x5567ae811600 2024-05-09T08:55:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.596+0000 7f8063e24740 1 -- 172.21.15.98:0/877874315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f804c031780 con 0x5567ae811600 2024-05-09T08:55:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.596+0000 7f803eff5640 1 -- 172.21.15.98:0/877874315 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f804c0047c0 con 0x5567ae811600 2024-05-09T08:55:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.597+0000 7f803eff5640 1 -- 172.21.15.98:0/877874315 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f804c040030 con 0x5567ae811600 2024-05-09T08:55:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.597+0000 7f8063e24740 10 client.16935 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.597+0000 7f8063e24740 1 -- 172.21.15.98:0/877874315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5567ae556620 con 0x5567ae811600 2024-05-09T08:55:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.597+0000 7f803eff5640 1 -- 172.21.15.98:0/877874315 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f804c031780 con 0x5567ae811600 2024-05-09T08:55:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.598+0000 7f803eff5640 1 -- 172.21.15.98:0/877874315 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f804c003b90 con 0x5567ae811600 2024-05-09T08:55:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.598+0000 7f8063e24740 10 client.16935 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.598+0000 7f8063e24740 10 client.16935 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.598+0000 7f8063e24740 20 client.16935 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.598+0000 7f8063e24740 1 --2- 172.21.15.98:0/877874315 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5567ae84e4c0 0x5567ae8508b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.598+0000 7f8063e24740 4 client.16935 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.598+0000 7f8063e24740 1 -- 172.21.15.98:0/877874315 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567ae783a70 con 0x5567ae84e4c0 2024-05-09T08:55:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.598+0000 7f805a59c640 1 --2- 172.21.15.98:0/877874315 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5567ae84e4c0 0x5567ae8508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.606+0000 7f805a59c640 1 --2- 172.21.15.98:0/877874315 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5567ae84e4c0 0x5567ae8508b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d364db553767908b server_cookie=1d9468b97e7d921d in_seq=0 out_seq=0 2024-05-09T08:55:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.606+0000 7f803eff5640 10 client.16935 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.614+0000 7f803eff5640 1 -- 172.21.15.98:0/877874315 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5567ae783a70 con 0x5567ae84e4c0 2024-05-09T08:55:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.614+0000 7f803eff5640 10 client.16935 handle_command_reply: tid=0 2024-05-09T08:55:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.614+0000 7f8063e24740 1 client.16935 shutdown 2024-05-09T08:55:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.615+0000 7f8063e24740 1 -- 172.21.15.98:0/877874315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811600 msgr2=0x5567ae819780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.615+0000 7f8063e24740 1 --2- 172.21.15.98:0/877874315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811600 0x5567ae819780 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f804c005610 tx=0x7f804c02a9d0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.615+0000 7f8063e24740 1 -- 172.21.15.98:0/877874315 shutdown_connections 2024-05-09T08:55:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.615+0000 7f8063e24740 1 --2- 172.21.15.98:0/877874315 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5567ae84e4c0 0x5567ae8508b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.615+0000 7f8063e24740 1 --2- 172.21.15.98:0/877874315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae81c7e0 0x5567ae81a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.615+0000 7f8063e24740 1 --2- 172.21.15.98:0/877874315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae811f20 0x5567ae819cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.615+0000 7f8063e24740 1 --2- 172.21.15.98:0/877874315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811600 0x5567ae819780 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.615+0000 7f8063e24740 1 -- 172.21.15.98:0/877874315 >> 172.21.15.98:0/877874315 conn(0x5567ae80db10 msgr2=0x5567ae813890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.615+0000 7f803eff5640 0 client.16935 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.615+0000 7f8063e24740 1 -- 172.21.15.98:0/877874315 shutdown_connections 2024-05-09T08:55:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.615+0000 7f8063e24740 1 -- 172.21.15.98:0/877874315 wait complete. 2024-05-09T08:55:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.615+0000 7f8063e24740 20 client.16935 trim_cache size 0 max 16384 2024-05-09T08:55:06.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.648+0000 7f8063e24740 1 Processor -- start 2024-05-09T08:55:06.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.648+0000 7f8063e24740 1 -- start start 2024-05-09T08:55:06.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.649+0000 7f8063e24740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae811600 0x5567ae81beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.649+0000 7f8063e24740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 0x5567ae814a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.649+0000 7f8063e24740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae81c7e0 0x5567ae813f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.649+0000 7f805a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 0x5567ae814a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.650+0000 7f805a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 0x5567ae814a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57556/0 (socket says 172.21.15.98:57556) 2024-05-09T08:55:06.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.650+0000 7f805a59c640 1 -- 172.21.15.98:0/1723882399 learned_addr learned my addr 172.21.15.98:0/1723882399 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.650+0000 7f805ad9d640 1 --2- 172.21.15.98:0/1723882399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae811600 0x5567ae81beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.650+0000 7f805b59e640 1 --2- 172.21.15.98:0/1723882399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae81c7e0 0x5567ae813f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.650+0000 7f8063e24740 1 -- 172.21.15.98:0/1723882399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8048010080 con 0x5567ae811f20 2024-05-09T08:55:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.650+0000 7f8063e24740 1 -- 172.21.15.98:0/1723882399 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8048011bc0 con 0x5567ae811600 2024-05-09T08:55:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.650+0000 7f8063e24740 1 -- 172.21.15.98:0/1723882399 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8048010750 con 0x5567ae81c7e0 2024-05-09T08:55:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.652+0000 7f805b59e640 1 -- 172.21.15.98:0/1723882399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae811600 msgr2=0x5567ae81beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.652+0000 7f805b59e640 1 --2- 172.21.15.98:0/1723882399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae811600 0x5567ae81beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.652+0000 7f805b59e640 1 -- 172.21.15.98:0/1723882399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 msgr2=0x5567ae814a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.652+0000 7f805b59e640 1 --2- 172.21.15.98:0/1723882399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 0x5567ae814a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.652+0000 7f805b59e640 1 -- 172.21.15.98:0/1723882399 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567ae811d10 con 0x5567ae81c7e0 2024-05-09T08:55:06.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.653+0000 7f805b59e640 1 --2- 172.21.15.98:0/1723882399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae81c7e0 0x5567ae813f30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f805400b480 tx=0x7f805400b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b33029bc714a4afb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.654+0000 7f8040ff9640 1 -- 172.21.15.98:0/1723882399 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805401b070 con 0x5567ae81c7e0 2024-05-09T08:55:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.654+0000 7f8040ff9640 1 -- 172.21.15.98:0/1723882399 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8054016420 con 0x5567ae81c7e0 2024-05-09T08:55:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.654+0000 7f8040ff9640 1 -- 172.21.15.98:0/1723882399 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054009e60 con 0x5567ae81c7e0 2024-05-09T08:55:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.655+0000 7f8063e24740 1 -- 172.21.15.98:0/1723882399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae81c7e0 msgr2=0x5567ae813f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.655+0000 7f8063e24740 1 --2- 172.21.15.98:0/1723882399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae81c7e0 0x5567ae813f30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f805400b480 tx=0x7f805400b940 comp rx=0 tx=0).stop 2024-05-09T08:55:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.656+0000 7f8063e24740 1 -- 172.21.15.98:0/1723882399 shutdown_connections 2024-05-09T08:55:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.656+0000 7f8063e24740 1 --2- 172.21.15.98:0/1723882399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae81c7e0 0x5567ae813f30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.656+0000 7f8063e24740 1 --2- 172.21.15.98:0/1723882399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 0x5567ae814a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.656+0000 7f8063e24740 1 --2- 172.21.15.98:0/1723882399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae811600 0x5567ae81beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.656+0000 7f8063e24740 1 -- 172.21.15.98:0/1723882399 >> 172.21.15.98:0/1723882399 conn(0x5567ae80db10 msgr2=0x5567ae80cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.662+0000 7f8063e24740 1 -- 172.21.15.98:0/1723882399 shutdown_connections 2024-05-09T08:55:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.662+0000 7f8063e24740 1 -- 172.21.15.98:0/1723882399 wait complete. 2024-05-09T08:55:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.662+0000 7f8063e24740 1 -- start start 2024-05-09T08:55:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.662+0000 7f8063e24740 1 Processor -- start 2024-05-09T08:55:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.663+0000 7f8063e24740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae811600 0x5567ae850e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.663+0000 7f8063e24740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 0x5567ae851390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.663+0000 7f8063e24740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae81c7e0 0x5567ae8518d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.663+0000 7f8063e24740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80480114f0 con 0x5567ae811f20 2024-05-09T08:55:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.663+0000 7f8063e24740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8048010e20 con 0x5567ae81c7e0 2024-05-09T08:55:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.663+0000 7f8063e24740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f804800e000 con 0x5567ae811600 2024-05-09T08:55:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.663+0000 7f8063e24740 10 client.0 _ll_register_callbacks cb 0x7f804c031a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.664+0000 7f805ad9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae811600 0x5567ae850e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.664+0000 7f805ad9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae811600 0x5567ae850e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:34002/0 (socket says 172.21.15.98:34002) 2024-05-09T08:55:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.664+0000 7f805ad9d640 1 -- 172.21.15.98:0/405148557 learned_addr learned my addr 172.21.15.98:0/405148557 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.665+0000 7f805ad9d640 1 -- 172.21.15.98:0/405148557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae81c7e0 msgr2=0x5567ae8518d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.665+0000 7f805ad9d640 1 --2- 172.21.15.98:0/405148557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae81c7e0 0x5567ae8518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.665+0000 7f805ad9d640 1 -- 172.21.15.98:0/405148557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 msgr2=0x5567ae851390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.665+0000 7f805ad9d640 1 --2- 172.21.15.98:0/405148557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 0x5567ae851390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.665+0000 7f805ad9d640 1 -- 172.21.15.98:0/405148557 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567ae8b36a0 con 0x5567ae811600 2024-05-09T08:55:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.667+0000 7f805ad9d640 1 --2- 172.21.15.98:0/405148557 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae811600 0x5567ae850e50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f804c02aa40 tx=0x7f804c02a4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c985600c80b78850 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.668+0000 7f803eff5640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:55:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.670+0000 7f803eff5640 1 -- 172.21.15.98:0/405148557 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c035040 con 0x5567ae811600 2024-05-09T08:55:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.670+0000 7f803eff5640 1 -- 172.21.15.98:0/405148557 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f804c03b790 con 0x5567ae811600 2024-05-09T08:55:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.671+0000 7f8063e24740 1 -- 172.21.15.98:0/405148557 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8054016420 con 0x5567ae811600 2024-05-09T08:55:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.674+0000 7f803eff5640 1 -- 172.21.15.98:0/405148557 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f804c0044a0 con 0x5567ae811600 2024-05-09T08:55:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.675+0000 7f8063e24740 10 client.26185 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.675+0000 7f8063e24740 1 -- 172.21.15.98:0/405148557 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5567ae8b4910 con 0x5567ae811600 2024-05-09T08:55:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.679+0000 7f803eff5640 1 -- 172.21.15.98:0/405148557 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8054016420 con 0x5567ae811600 2024-05-09T08:55:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.679+0000 7f803eff5640 1 -- 172.21.15.98:0/405148557 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f804c003b90 con 0x5567ae811600 2024-05-09T08:55:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.679+0000 7f8063e24740 10 client.26185 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.679+0000 7f8063e24740 10 client.26185 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.679+0000 7f8063e24740 20 client.26185 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.679+0000 7f8063e24740 1 --2- 172.21.15.98:0/405148557 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5567ae8b5010 0x5567ae8b74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.679+0000 7f8063e24740 4 client.26185 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:55:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.679+0000 7f8063e24740 1 -- 172.21.15.98:0/405148557 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5567ae77ddc0 con 0x5567ae8b5010 2024-05-09T08:55:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.681+0000 7f805a59c640 1 --2- 172.21.15.98:0/405148557 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5567ae8b5010 0x5567ae8b74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.690+0000 7f805a59c640 1 --2- 172.21.15.98:0/405148557 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5567ae8b5010 0x5567ae8b74b0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8f745a560208cd9 server_cookie=fc797e4d26c40004 in_seq=0 out_seq=0 2024-05-09T08:55:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:06.691+0000 7f803eff5640 10 client.26185 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:07.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:07 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:55:07.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:07 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:55:07.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:07 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:55:08.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:08 smithi142 ceph-mon[34625]: pgmap v139: 129 pgs: 129 active+clean; 30 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 935 MiB/s wr, 1.89k op/s 2024-05-09T08:55:08.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:08 smithi105 ceph-mon[34498]: pgmap v139: 129 pgs: 129 active+clean; 30 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 935 MiB/s wr, 1.89k op/s 2024-05-09T08:55:08.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:08 smithi098 ceph-mon[30092]: pgmap v139: 129 pgs: 129 active+clean; 30 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 935 MiB/s wr, 1.89k op/s 2024-05-09T08:55:10.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:10 smithi142 ceph-mon[34625]: pgmap v140: 129 pgs: 129 active+clean; 30 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 714 MiB/s wr, 1.67k op/s 2024-05-09T08:55:10.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:10 smithi105 ceph-mon[34498]: pgmap v140: 129 pgs: 129 active+clean; 30 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 714 MiB/s wr, 1.67k op/s 2024-05-09T08:55:10.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:10 smithi098 ceph-mon[30092]: pgmap v140: 129 pgs: 129 active+clean; 30 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 714 MiB/s wr, 1.67k op/s 2024-05-09T08:55:11.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.223+0000 7f803eff5640 1 -- 172.21.15.98:0/405148557 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5567ae77ddc0 con 0x5567ae8b5010 2024-05-09T08:55:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.223+0000 7f803eff5640 10 client.26185 handle_command_reply: tid=0 2024-05-09T08:55:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.223+0000 7f8063e24740 1 client.26185 shutdown 2024-05-09T08:55:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.223+0000 7f8063e24740 1 -- 172.21.15.98:0/405148557 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae811600 msgr2=0x5567ae850e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.223+0000 7f8063e24740 1 --2- 172.21.15.98:0/405148557 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae811600 0x5567ae850e50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f804c02aa40 tx=0x7f804c02a4d0 comp rx=0 tx=0).stop 2024-05-09T08:55:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.224+0000 7f8063e24740 1 -- 172.21.15.98:0/405148557 shutdown_connections 2024-05-09T08:55:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.224+0000 7f8063e24740 1 --2- 172.21.15.98:0/405148557 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5567ae8b5010 0x5567ae8b74b0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.224+0000 7f8063e24740 1 --2- 172.21.15.98:0/405148557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5567ae81c7e0 0x5567ae8518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.224+0000 7f8063e24740 1 --2- 172.21.15.98:0/405148557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5567ae811f20 0x5567ae851390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.224+0000 7f8063e24740 1 --2- 172.21.15.98:0/405148557 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567ae811600 0x5567ae850e50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.224+0000 7f8063e24740 1 -- 172.21.15.98:0/405148557 >> 172.21.15.98:0/405148557 conn(0x5567ae80db10 msgr2=0x5567ae827780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.224+0000 7f803eff5640 0 client.26185 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.224+0000 7f8063e24740 1 -- 172.21.15.98:0/405148557 shutdown_connections 2024-05-09T08:55:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.224+0000 7f8063e24740 1 -- 172.21.15.98:0/405148557 wait complete. 2024-05-09T08:55:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.224+0000 7f8063e24740 20 client.26185 trim_cache size 0 max 16384 2024-05-09T08:55:11.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:55:11.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:55:11.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:55:11.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:55:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.228+0000 7f8060837640 1 -- 172.21.15.98:0/2252712300 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f80440a2e10 msgr2=0x7f80440a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.228+0000 7f8060837640 1 --2- 172.21.15.98:0/2252712300 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f80440a2e10 0x7f80440a52d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f804c02aac0 tx=0x7f804c005710 comp rx=0 tx=0).stop 2024-05-09T08:55:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.228+0000 7f8060837640 1 -- 172.21.15.98:0/2252712300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f805c10fef0 msgr2=0x7f805c1156f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:11.229 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.228+0000 7f8060837640 1 --2- 172.21.15.98:0/2252712300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f805c10fef0 0x7f805c1156f0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f805000b450 tx=0x7f805000b910 comp rx=0 tx=0).stop 2024-05-09T08:55:11.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.229+0000 7f8060837640 1 -- 172.21.15.98:0/2252712300 shutdown_connections 2024-05-09T08:55:11.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.229+0000 7f8060837640 1 --2- 172.21.15.98:0/2252712300 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f80440a2e10 0x7f80440a52d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:11.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.229+0000 7f8060837640 1 --2- 172.21.15.98:0/2252712300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f805c1102d0 0x7f805c115c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:11.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.229+0000 7f8060837640 1 --2- 172.21.15.98:0/2252712300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f805c10fef0 0x7f805c1156f0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:11.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.229+0000 7f8060837640 1 --2- 172.21.15.98:0/2252712300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f805c10e500 0x7f805c11c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:11.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.229+0000 7f8060837640 1 -- 172.21.15.98:0/2252712300 >> 172.21.15.98:0/2252712300 conn(0x7f805c06c770 msgr2=0x7f805c070d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:11.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.230+0000 7f8060837640 1 -- 172.21.15.98:0/2252712300 shutdown_connections 2024-05-09T08:55:11.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:11.231+0000 7f8060837640 1 -- 172.21.15.98:0/2252712300 wait complete. 2024-05-09T08:55:12.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:12 smithi142 ceph-mon[34625]: pgmap v141: 129 pgs: 129 active+clean; 31 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 852 MiB/s wr, 2.97k op/s 2024-05-09T08:55:12.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:12 smithi105 ceph-mon[34498]: pgmap v141: 129 pgs: 129 active+clean; 31 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 852 MiB/s wr, 2.97k op/s 2024-05-09T08:55:12.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:12 smithi098 ceph-mon[30092]: pgmap v141: 129 pgs: 129 active+clean; 31 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 852 MiB/s wr, 2.97k op/s 2024-05-09T08:55:14.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:14 smithi142 ceph-mon[34625]: pgmap v142: 129 pgs: 129 active+clean; 31 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 614 MiB/s wr, 2.31k op/s 2024-05-09T08:55:14.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:14 smithi105 ceph-mon[34498]: pgmap v142: 129 pgs: 129 active+clean; 31 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 614 MiB/s wr, 2.31k op/s 2024-05-09T08:55:14.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:14 smithi098 ceph-mon[30092]: pgmap v142: 129 pgs: 129 active+clean; 31 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 614 MiB/s wr, 2.31k op/s 2024-05-09T08:55:16.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:16 smithi142 ceph-mon[34625]: pgmap v143: 129 pgs: 129 active+clean; 32 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 679 MiB/s wr, 2.59k op/s 2024-05-09T08:55:16.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:16 smithi105 ceph-mon[34498]: pgmap v143: 129 pgs: 129 active+clean; 32 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 679 MiB/s wr, 2.59k op/s 2024-05-09T08:55:16.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:16 smithi098 ceph-mon[30092]: pgmap v143: 129 pgs: 129 active+clean; 32 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 679 MiB/s wr, 2.59k op/s 2024-05-09T08:55:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:18 smithi142 ceph-mon[34625]: pgmap v144: 129 pgs: 129 active+clean; 34 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 786 MiB/s wr, 3.25k op/s 2024-05-09T08:55:18.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:18 smithi105 ceph-mon[34498]: pgmap v144: 129 pgs: 129 active+clean; 34 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 786 MiB/s wr, 3.25k op/s 2024-05-09T08:55:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:18 smithi098 ceph-mon[30092]: pgmap v144: 129 pgs: 129 active+clean; 34 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 786 MiB/s wr, 3.25k op/s 2024-05-09T08:55:20.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:20 smithi142 ceph-mon[34625]: pgmap v145: 129 pgs: 129 active+clean; 34 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 521 MiB/s wr, 2.94k op/s 2024-05-09T08:55:20.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:20 smithi105 ceph-mon[34498]: pgmap v145: 129 pgs: 129 active+clean; 34 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 521 MiB/s wr, 2.94k op/s 2024-05-09T08:55:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:20 smithi098 ceph-mon[30092]: pgmap v145: 129 pgs: 129 active+clean; 34 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 521 MiB/s wr, 2.94k op/s 2024-05-09T08:55:22.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:22 smithi142 ceph-mon[34625]: pgmap v146: 129 pgs: 129 active+clean; 36 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 653 MiB/s wr, 3.15k op/s 2024-05-09T08:55:22.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:22 smithi105 ceph-mon[34498]: pgmap v146: 129 pgs: 129 active+clean; 36 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 653 MiB/s wr, 3.15k op/s 2024-05-09T08:55:22.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:22 smithi098 ceph-mon[30092]: pgmap v146: 129 pgs: 129 active+clean; 36 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 653 MiB/s wr, 3.15k op/s 2024-05-09T08:55:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.477+0000 7f9ff075c640 1 -- 172.21.15.98:0/3802945836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe811b010 msgr2=0x7f9fe811d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.477+0000 7f9ff075c640 1 --2- 172.21.15.98:0/3802945836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe811b010 0x7f9fe811d400 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009930 tx=0x7f9fd802a510 comp rx=0 tx=0).stop 2024-05-09T08:55:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.480+0000 7f9ff075c640 1 -- 172.21.15.98:0/3802945836 shutdown_connections 2024-05-09T08:55:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.480+0000 7f9ff075c640 1 --2- 172.21.15.98:0/3802945836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe811b010 0x7f9fe811d400 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.481+0000 7f9ff075c640 1 --2- 172.21.15.98:0/3802945836 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fe8110400 0x7f9fe8108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.481+0000 7f9ff075c640 1 --2- 172.21.15.98:0/3802945836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8110020 0x7f9fe8108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.481+0000 7f9ff075c640 1 -- 172.21.15.98:0/3802945836 >> 172.21.15.98:0/3802945836 conn(0x7f9fe806c900 msgr2=0x7f9fe806cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.481+0000 7f9ff075c640 1 -- 172.21.15.98:0/3802945836 shutdown_connections 2024-05-09T08:55:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.481+0000 7f9ff075c640 1 -- 172.21.15.98:0/3802945836 wait complete. 2024-05-09T08:55:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.482+0000 7f9ff075c640 1 Processor -- start 2024-05-09T08:55:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.482+0000 7f9ff075c640 1 -- start start 2024-05-09T08:55:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.483+0000 7f9ff075c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8110020 0x7f9fe811a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.483+0000 7f9ff075c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fe811b010 0x7f9fe811aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.483+0000 7f9ff075c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe8113710 0x7f9fe8113bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.483+0000 7f9ff075c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe8120a90 con 0x7f9fe8110020 2024-05-09T08:55:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.483+0000 7f9ff075c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe8120910 con 0x7f9fe8113710 2024-05-09T08:55:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.483+0000 7f9ff075c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe8120c10 con 0x7f9fe811b010 2024-05-09T08:55:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.483+0000 7f9fedcf7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fe811b010 0x7f9fe811aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.483+0000 7f9feecf9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe8113710 0x7f9fe8113bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.483+0000 7f9fedcf7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fe811b010 0x7f9fe811aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:35374/0 (socket says 172.21.15.98:35374) 2024-05-09T08:55:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.483+0000 7f9fedcf7640 1 -- 172.21.15.98:0/3031823303 learned_addr learned my addr 172.21.15.98:0/3031823303 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.483+0000 7f9fee4f8640 1 --2- 172.21.15.98:0/3031823303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8110020 0x7f9fe811a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.484+0000 7f9fee4f8640 1 -- 172.21.15.98:0/3031823303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fe811b010 msgr2=0x7f9fe811aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.484+0000 7f9fee4f8640 1 --2- 172.21.15.98:0/3031823303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fe811b010 0x7f9fe811aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.484+0000 7f9fee4f8640 1 -- 172.21.15.98:0/3031823303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe8113710 msgr2=0x7f9fe8113bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.484+0000 7f9fee4f8640 1 --2- 172.21.15.98:0/3031823303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe8113710 0x7f9fe8113bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.484+0000 7f9fee4f8640 1 -- 172.21.15.98:0/3031823303 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe81180a0 con 0x7f9fe8110020 2024-05-09T08:55:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.484+0000 7f9fedcf7640 1 --2- 172.21.15.98:0/3031823303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fe811b010 0x7f9fe811aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:55:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.484+0000 7f9fee4f8640 1 --2- 172.21.15.98:0/3031823303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8110020 0x7f9fe811a580 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000e9f0 tx=0x7f9fe000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:24.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:24 smithi105 ceph-mon[34498]: pgmap v147: 129 pgs: 129 active+clean; 37 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 562 MiB/s wr, 1.84k op/s 2024-05-09T08:55:24.490 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.490+0000 7f9fdf7fe640 1 -- 172.21.15.98:0/3031823303 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe000cd80 con 0x7f9fe8110020 2024-05-09T08:55:24.490 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.490+0000 7f9fdf7fe640 1 -- 172.21.15.98:0/3031823303 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe0011950 con 0x7f9fe8110020 2024-05-09T08:55:24.490 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.490+0000 7f9fdf7fe640 1 -- 172.21.15.98:0/3031823303 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0010a40 con 0x7f9fe8110020 2024-05-09T08:55:24.490 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.490+0000 7f9ff075c640 1 -- 172.21.15.98:0/3031823303 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe8118390 con 0x7f9fe8110020 2024-05-09T08:55:24.490 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.490+0000 7f9ff075c640 1 -- 172.21.15.98:0/3031823303 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe8118890 con 0x7f9fe8110020 2024-05-09T08:55:24.496 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.497+0000 7f9ff345f740 1 Processor -- start 2024-05-09T08:55:24.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.499+0000 7f9ff345f740 1 -- start start 2024-05-09T08:55:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.499+0000 7f9ff345f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0450 0x5561f12d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.499+0000 7f9ff345f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12d0d70 0x5561f12db0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.499+0000 7f9ff345f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12db630 0x5561f12dda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.500+0000 7f9ff345f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5561f1235340 con 0x5561f12d0d70 2024-05-09T08:55:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.500+0000 7f9ff345f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5561f11dd290 con 0x5561f12db630 2024-05-09T08:55:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.500+0000 7f9ff345f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5561f1231300 con 0x5561f12d0450 2024-05-09T08:55:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.500+0000 7f9feecf9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0450 0x5561f12d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.500+0000 7f9feecf9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0450 0x5561f12d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:35390/0 (socket says 172.21.15.98:35390) 2024-05-09T08:55:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.500+0000 7f9feecf9640 1 -- 172.21.15.98:0/561223823 learned_addr learned my addr 172.21.15.98:0/561223823 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.500+0000 7f9feecf9640 1 -- 172.21.15.98:0/561223823 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12db630 msgr2=0x5561f12dda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.500+0000 7f9feecf9640 1 --2- 172.21.15.98:0/561223823 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12db630 0x5561f12dda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.500+0000 7f9feecf9640 1 -- 172.21.15.98:0/561223823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12d0d70 msgr2=0x5561f12db0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.500+0000 7f9feecf9640 1 --2- 172.21.15.98:0/561223823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12d0d70 0x5561f12db0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.500+0000 7f9feecf9640 1 -- 172.21.15.98:0/561223823 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561f12018e0 con 0x5561f12d0450 2024-05-09T08:55:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.501+0000 7f9feecf9640 1 --2- 172.21.15.98:0/561223823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0450 0x5561f12d0830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009230 tx=0x7f9fd8004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=716621780aa7d140 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.502+0000 7f9fdcff9640 1 -- 172.21.15.98:0/561223823 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8040070 con 0x5561f12d0450 2024-05-09T08:55:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.502+0000 7f9fdcff9640 1 -- 172.21.15.98:0/561223823 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd8004470 con 0x5561f12d0450 2024-05-09T08:55:24.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.502+0000 7f9fdf7fe640 1 -- 172.21.15.98:0/3031823303 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9fe0011b00 con 0x7f9fe8110020 2024-05-09T08:55:24.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.503+0000 7f9ff345f740 1 -- 172.21.15.98:0/561223823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0450 msgr2=0x5561f12d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:24.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.503+0000 7f9ff345f740 1 --2- 172.21.15.98:0/561223823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0450 0x5561f12d0830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009230 tx=0x7f9fd8004040 comp rx=0 tx=0).stop 2024-05-09T08:55:24.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.503+0000 7f9fdf7fe640 1 --2- 172.21.15.98:0/3031823303 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9fd40a2fd0 0x7f9fd40a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.504+0000 7f9fdf7fe640 1 -- 172.21.15.98:0/3031823303 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f9fe00cfbd0 con 0x7f9fe8110020 2024-05-09T08:55:24.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.504+0000 7f9fee4f8640 1 --2- 172.21.15.98:0/3031823303 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9fd40a2fd0 0x7f9fd40a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:24.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.505+0000 7f9fee4f8640 1 --2- 172.21.15.98:0/3031823303 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9fd40a2fd0 0x7f9fd40a5490 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x5561f12d19f0 tx=0x7f9fe0005b50 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.505+0000 7f9ff345f740 1 -- 172.21.15.98:0/561223823 shutdown_connections 2024-05-09T08:55:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.505+0000 7f9ff345f740 1 --2- 172.21.15.98:0/561223823 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12db630 0x5561f12dda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.505+0000 7f9ff345f740 1 --2- 172.21.15.98:0/561223823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12d0d70 0x5561f12db0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.505+0000 7f9ff345f740 1 --2- 172.21.15.98:0/561223823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0450 0x5561f12d0830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.505+0000 7f9ff345f740 1 -- 172.21.15.98:0/561223823 >> 172.21.15.98:0/561223823 conn(0x5561f12cc960 msgr2=0x5561f12ccd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.505+0000 7f9ff345f740 1 -- 172.21.15.98:0/561223823 shutdown_connections 2024-05-09T08:55:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.505+0000 7f9ff345f740 1 -- 172.21.15.98:0/561223823 wait complete. 2024-05-09T08:55:24.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.506+0000 7f9ff345f740 1 -- start start 2024-05-09T08:55:24.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.506+0000 7f9ff345f740 1 Processor -- start 2024-05-09T08:55:24.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.506+0000 7f9ff345f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12d0450 0x5561f12d84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.507+0000 7f9ff345f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0d70 0x5561f12d89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.507+0000 7f9ff345f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12db630 0x5561f12d8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.507+0000 7f9ff345f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5561f12de200 con 0x5561f12db630 2024-05-09T08:55:24.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.507+0000 7f9ff345f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5561f11f0790 con 0x5561f12d0450 2024-05-09T08:55:24.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.507+0000 7f9ff345f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5561f12de380 con 0x5561f12d0d70 2024-05-09T08:55:24.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.507+0000 7f9ff345f740 10 client.0 _ll_register_callbacks cb 0x5561f122ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:24.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.507+0000 7f9feecf9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12db630 0x5561f12d8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:24.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.507+0000 7f9feecf9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12d0450 0x5561f12d84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:24.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.507+0000 7f9fedcf7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0d70 0x5561f12d89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:24.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.508+0000 7f9feecf9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12d0450 0x5561f12d84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:57778/0 (socket says 172.21.15.98:57778) 2024-05-09T08:55:24.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.508+0000 7f9feecf9640 1 -- 172.21.15.98:0/2341256805 learned_addr learned my addr 172.21.15.98:0/2341256805 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:24.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.509+0000 7f9feecf9640 1 -- 172.21.15.98:0/2341256805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0d70 msgr2=0x5561f12d89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:24.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.509+0000 7f9feecf9640 1 --2- 172.21.15.98:0/2341256805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0d70 0x5561f12d89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.509+0000 7f9feecf9640 1 -- 172.21.15.98:0/2341256805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12db630 msgr2=0x5561f12d8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:24.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.509+0000 7f9feecf9640 1 --2- 172.21.15.98:0/2341256805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12db630 0x5561f12d8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.509+0000 7f9feecf9640 1 -- 172.21.15.98:0/2341256805 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561f0f49210 con 0x5561f12d0450 2024-05-09T08:55:24.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.509+0000 7f9fedcf7640 1 --2- 172.21.15.98:0/2341256805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0d70 0x5561f12d89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:55:24.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.509+0000 7f9feecf9640 1 --2- 172.21.15.98:0/2341256805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12d0450 0x5561f12d84b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9fd80096c0 tx=0x7f9fd802aa10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=292fd0cf5fad3807 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:24.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.510+0000 7f9fd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:55:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.510+0000 7f9fd1ffb640 1 -- 172.21.15.98:0/2341256805 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8040070 con 0x5561f12d0450 2024-05-09T08:55:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.510+0000 7f9fd1ffb640 1 -- 172.21.15.98:0/2341256805 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f9fd80035f0 con 0x5561f12d0450 2024-05-09T08:55:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.510+0000 7f9ff345f740 1 -- 172.21.15.98:0/2341256805 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fd8004470 con 0x5561f12d0450 2024-05-09T08:55:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.511+0000 7f9fd1ffb640 1 -- 172.21.15.98:0/2341256805 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9fd80037f0 con 0x5561f12d0450 2024-05-09T08:55:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.511+0000 7f9ff345f740 10 client.26237 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:24.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.511+0000 7f9ff345f740 1 -- 172.21.15.98:0/2341256805 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5561f1015620 con 0x5561f12d0450 2024-05-09T08:55:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.513+0000 7f9fd1ffb640 1 -- 172.21.15.98:0/2341256805 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd8004470 con 0x5561f12d0450 2024-05-09T08:55:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.513+0000 7f9fd1ffb640 1 -- 172.21.15.98:0/2341256805 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f9fd8031c70 con 0x5561f12d0450 2024-05-09T08:55:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.514+0000 7f9ff345f740 10 client.26237 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.514+0000 7f9ff345f740 10 client.26237 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.514+0000 7f9ff345f740 20 client.26237 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.514+0000 7f9ff345f740 1 --2- 172.21.15.98:0/2341256805 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5561f12eb180 0x5561f12ed570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.514+0000 7f9ff345f740 4 client.26237 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.514+0000 7f9ff345f740 1 -- 172.21.15.98:0/2341256805 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561f1242a70 con 0x5561f12eb180 2024-05-09T08:55:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.515+0000 7f9fedcf7640 1 --2- 172.21.15.98:0/2341256805 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5561f12eb180 0x5561f12ed570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:24.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.518+0000 7f9fedcf7640 1 --2- 172.21.15.98:0/2341256805 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5561f12eb180 0x5561f12ed570 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9eeb151b3bc39b01 server_cookie=8ff641b820d28666 in_seq=0 out_seq=0 2024-05-09T08:55:24.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.519+0000 7f9fd1ffb640 10 client.26237 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.528+0000 7f9fd1ffb640 1 -- 172.21.15.98:0/2341256805 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561f1242a70 con 0x5561f12eb180 2024-05-09T08:55:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.528+0000 7f9fd1ffb640 10 client.26237 handle_command_reply: tid=0 2024-05-09T08:55:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.528+0000 7f9ff345f740 1 client.26237 shutdown 2024-05-09T08:55:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.533+0000 7f9ff345f740 1 -- 172.21.15.98:0/2341256805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12d0450 msgr2=0x5561f12d84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.533+0000 7f9ff345f740 1 --2- 172.21.15.98:0/2341256805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12d0450 0x5561f12d84b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9fd80096c0 tx=0x7f9fd802aa10 comp rx=0 tx=0).stop 2024-05-09T08:55:24.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.535+0000 7f9ff345f740 1 -- 172.21.15.98:0/2341256805 shutdown_connections 2024-05-09T08:55:24.535 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.535+0000 7f9ff345f740 1 --2- 172.21.15.98:0/2341256805 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5561f12eb180 0x5561f12ed570 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.535 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.535+0000 7f9ff345f740 1 --2- 172.21.15.98:0/2341256805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12db630 0x5561f12d8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.535 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.535+0000 7f9ff345f740 1 --2- 172.21.15.98:0/2341256805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0d70 0x5561f12d89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.535 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.535+0000 7f9ff345f740 1 --2- 172.21.15.98:0/2341256805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12d0450 0x5561f12d84b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.535 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.535+0000 7f9ff345f740 1 -- 172.21.15.98:0/2341256805 >> 172.21.15.98:0/2341256805 conn(0x5561f12cc960 msgr2=0x5561f12d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:24.535 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.535+0000 7f9fd1ffb640 0 client.26237 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.538+0000 7f9ff345f740 1 -- 172.21.15.98:0/2341256805 shutdown_connections 2024-05-09T08:55:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.538+0000 7f9ff345f740 1 -- 172.21.15.98:0/2341256805 wait complete. 2024-05-09T08:55:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.538+0000 7f9ff345f740 20 client.26237 trim_cache size 0 max 16384 2024-05-09T08:55:24.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.581+0000 7f9ff345f740 1 Processor -- start 2024-05-09T08:55:24.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.581+0000 7f9ff345f740 1 -- start start 2024-05-09T08:55:24.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.582+0000 7f9ff345f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0450 0x5561f12dabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.582+0000 7f9ff345f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12d0d70 0x5561f12e7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.582+0000 7f9feecf9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0450 0x5561f12dabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:24.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.582+0000 7f9feecf9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0450 0x5561f12dabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:35396/0 (socket says 172.21.15.98:35396) 2024-05-09T08:55:24.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.582+0000 7f9feecf9640 1 -- 172.21.15.98:0/416830500 learned_addr learned my addr 172.21.15.98:0/416830500 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:24.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.582+0000 7f9ff345f740 1 --2- 172.21.15.98:0/416830500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12db630 0x5561f12d7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.582+0000 7f9fedcf7640 1 --2- 172.21.15.98:0/416830500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12d0d70 0x5561f12e7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:24.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.583+0000 7f9ff345f740 1 -- 172.21.15.98:0/416830500 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81c63d0 con 0x5561f12db630 2024-05-09T08:55:24.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.583+0000 7f9ff345f740 1 -- 172.21.15.98:0/416830500 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81c7f10 con 0x5561f12d0d70 2024-05-09T08:55:24.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.583+0000 7f9ff345f740 1 -- 172.21.15.98:0/416830500 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81c6aa0 con 0x5561f12d0450 2024-05-09T08:55:24.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.583+0000 7f9feecf9640 1 -- 172.21.15.98:0/416830500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12d0d70 msgr2=0x5561f12e7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:24.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.583+0000 7f9feecf9640 1 --2- 172.21.15.98:0/416830500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12d0d70 0x5561f12e7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.583+0000 7f9feecf9640 1 -- 172.21.15.98:0/416830500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12db630 msgr2=0x5561f12d7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:24.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.583+0000 7f9feecf9640 1 --2- 172.21.15.98:0/416830500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12db630 0x5561f12d7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.583+0000 7f9feecf9640 1 -- 172.21.15.98:0/416830500 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561f12d0210 con 0x5561f12d0450 2024-05-09T08:55:24.591 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:24 smithi098 ceph-mon[30092]: pgmap v147: 129 pgs: 129 active+clean; 37 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 562 MiB/s wr, 1.84k op/s 2024-05-09T08:55:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.593+0000 7f9feecf9640 1 --2- 172.21.15.98:0/416830500 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0450 0x5561f12dabe0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009250 tx=0x7f9fd802aa80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa364b769c538831 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.594+0000 7f9fd3fff640 1 -- 172.21.15.98:0/416830500 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8040040 con 0x5561f12d0450 2024-05-09T08:55:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.594+0000 7f9fd3fff640 1 -- 172.21.15.98:0/416830500 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd80049b0 con 0x5561f12d0450 2024-05-09T08:55:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.594+0000 7f9fd3fff640 1 -- 172.21.15.98:0/416830500 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8003990 con 0x5561f12d0450 2024-05-09T08:55:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.595+0000 7f9ff345f740 1 -- 172.21.15.98:0/416830500 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0450 msgr2=0x5561f12dabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.595+0000 7f9ff345f740 1 --2- 172.21.15.98:0/416830500 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0450 0x5561f12dabe0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009250 tx=0x7f9fd802aa80 comp rx=0 tx=0).stop 2024-05-09T08:55:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.598+0000 7f9ff345f740 1 -- 172.21.15.98:0/416830500 shutdown_connections 2024-05-09T08:55:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.598+0000 7f9ff345f740 1 --2- 172.21.15.98:0/416830500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12db630 0x5561f12d7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.598+0000 7f9ff345f740 1 --2- 172.21.15.98:0/416830500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12d0d70 0x5561f12e7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.598+0000 7f9ff345f740 1 --2- 172.21.15.98:0/416830500 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0450 0x5561f12dabe0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.598+0000 7f9ff345f740 1 -- 172.21.15.98:0/416830500 >> 172.21.15.98:0/416830500 conn(0x5561f12cc960 msgr2=0x5561f12d8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.598+0000 7f9ff345f740 1 -- 172.21.15.98:0/416830500 shutdown_connections 2024-05-09T08:55:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.599+0000 7f9ff345f740 1 -- 172.21.15.98:0/416830500 wait complete. 2024-05-09T08:55:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.599+0000 7f9ff345f740 1 -- start start 2024-05-09T08:55:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.599+0000 7f9ff345f740 1 Processor -- start 2024-05-09T08:55:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.600+0000 7f9ff345f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12d0450 0x5561f12efe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.600+0000 7f9ff345f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0d70 0x5561f12f03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.601+0000 7f9ff345f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12db630 0x5561f12ed3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.601+0000 7f9fedcf7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0d70 0x5561f12f03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:24.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.601+0000 7f9fedcf7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0d70 0x5561f12f03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:35412/0 (socket says 172.21.15.98:35412) 2024-05-09T08:55:24.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.601+0000 7f9fedcf7640 1 -- 172.21.15.98:0/623408107 learned_addr learned my addr 172.21.15.98:0/623408107 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:24.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.601+0000 7f9feecf9640 1 --2- 172.21.15.98:0/623408107 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12db630 0x5561f12ed3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:24.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.601+0000 7f9ff345f740 1 -- 172.21.15.98:0/623408107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81c7840 con 0x5561f12d0450 2024-05-09T08:55:24.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.601+0000 7f9ff345f740 1 -- 172.21.15.98:0/623408107 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81c7170 con 0x5561f12db630 2024-05-09T08:55:24.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.601+0000 7f9ff345f740 1 -- 172.21.15.98:0/623408107 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81c43b0 con 0x5561f12d0d70 2024-05-09T08:55:24.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.602+0000 7f9ff345f740 10 client.0 _ll_register_callbacks cb 0x7f9fd8005150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:24.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.602+0000 7f9fedcf7640 1 -- 172.21.15.98:0/623408107 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12db630 msgr2=0x5561f12ed3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:24.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.602+0000 7f9fedcf7640 1 --2- 172.21.15.98:0/623408107 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12db630 0x5561f12ed3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.602+0000 7f9fedcf7640 1 -- 172.21.15.98:0/623408107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12d0450 msgr2=0x5561f12efe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:24.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.602+0000 7f9fedcf7640 1 --2- 172.21.15.98:0/623408107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12d0450 0x5561f12efe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:24.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.602+0000 7f9fedcf7640 1 -- 172.21.15.98:0/623408107 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561f12edaa0 con 0x5561f12d0d70 2024-05-09T08:55:24.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.605+0000 7f9fedcf7640 1 --2- 172.21.15.98:0/623408107 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0d70 0x5561f12f03d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8114e50 tx=0x7f9fe400f930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d2ad577d9a08972 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:24.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.605+0000 7f9fd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:55:24.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.605+0000 7f9fd1ffb640 1 -- 172.21.15.98:0/623408107 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe40166f0 con 0x5561f12d0d70 2024-05-09T08:55:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.605+0000 7f9ff345f740 1 -- 172.21.15.98:0/623408107 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5561f123cdc0 con 0x5561f12d0d70 2024-05-09T08:55:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.606+0000 7f9fd1ffb640 1 -- 172.21.15.98:0/623408107 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f9fe4016890 con 0x5561f12d0d70 2024-05-09T08:55:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.606+0000 7f9fd1ffb640 1 -- 172.21.15.98:0/623408107 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9fe402d050 con 0x5561f12d0d70 2024-05-09T08:55:24.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.608+0000 7f9ff345f740 10 client.26200 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:24.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.608+0000 7f9ff345f740 1 -- 172.21.15.98:0/623408107 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5561f1350e40 con 0x5561f12d0d70 2024-05-09T08:55:24.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.610+0000 7f9fd1ffb640 1 -- 172.21.15.98:0/623408107 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe4020070 con 0x5561f12d0d70 2024-05-09T08:55:24.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.610+0000 7f9fd1ffb640 1 -- 172.21.15.98:0/623408107 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f9fe4014410 con 0x5561f12d0d70 2024-05-09T08:55:24.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.610+0000 7f9ff345f740 10 client.26200 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:24.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.610+0000 7f9ff345f740 10 client.26200 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.610+0000 7f9ff345f740 20 client.26200 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.610+0000 7f9ff345f740 1 --2- 172.21.15.98:0/623408107 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5561f13515d0 0x5561f13539f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.610+0000 7f9ff345f740 4 client.26200 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:55:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.610+0000 7f9feecf9640 1 --2- 172.21.15.98:0/623408107 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5561f13515d0 0x5561f13539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.610+0000 7f9ff345f740 1 -- 172.21.15.98:0/623408107 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5561f123cdc0 con 0x5561f13515d0 2024-05-09T08:55:24.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.613+0000 7f9feecf9640 1 --2- 172.21.15.98:0/623408107 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5561f13515d0 0x5561f13539f0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d06744f816283df2 server_cookie=75b777b2694ab217 in_seq=0 out_seq=0 2024-05-09T08:55:24.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:24.613+0000 7f9fd1ffb640 10 client.26200 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:24 smithi142 ceph-mon[34625]: pgmap v147: 129 pgs: 129 active+clean; 37 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 562 MiB/s wr, 1.84k op/s 2024-05-09T08:55:26.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:26 smithi105 ceph-mon[34498]: pgmap v148: 129 pgs: 129 active+clean; 37 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 623 MiB/s wr, 1.77k op/s 2024-05-09T08:55:26.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:26 smithi098 ceph-mon[30092]: pgmap v148: 129 pgs: 129 active+clean; 37 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 623 MiB/s wr, 1.77k op/s 2024-05-09T08:55:26.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:26 smithi142 ceph-mon[34625]: pgmap v148: 129 pgs: 129 active+clean; 37 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 623 MiB/s wr, 1.77k op/s 2024-05-09T08:55:28.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:28 smithi105 ceph-mon[34498]: pgmap v149: 129 pgs: 129 active+clean; 37 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 731 MiB/s wr, 1.90k op/s 2024-05-09T08:55:28.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:28 smithi098 ceph-mon[30092]: pgmap v149: 129 pgs: 129 active+clean; 37 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 731 MiB/s wr, 1.90k op/s 2024-05-09T08:55:28.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:28 smithi142 ceph-mon[34625]: pgmap v149: 129 pgs: 129 active+clean; 37 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 731 MiB/s wr, 1.90k op/s 2024-05-09T08:55:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.164+0000 7f9fd1ffb640 1 -- 172.21.15.98:0/623408107 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5561f123cdc0 con 0x5561f13515d0 2024-05-09T08:55:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.164+0000 7f9fd1ffb640 10 client.26200 handle_command_reply: tid=0 2024-05-09T08:55:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.164+0000 7f9ff345f740 1 client.26200 shutdown 2024-05-09T08:55:29.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:55:29.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:55:29.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:55:29.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:55:29.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.166+0000 7f9ff345f740 1 -- 172.21.15.98:0/623408107 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0d70 msgr2=0x5561f12f03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:29.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.166+0000 7f9ff345f740 1 --2- 172.21.15.98:0/623408107 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0d70 0x5561f12f03d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8114e50 tx=0x7f9fe400f930 comp rx=0 tx=0).stop 2024-05-09T08:55:29.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.166+0000 7f9ff345f740 1 -- 172.21.15.98:0/623408107 shutdown_connections 2024-05-09T08:55:29.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.166+0000 7f9ff345f740 1 --2- 172.21.15.98:0/623408107 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5561f13515d0 0x5561f13539f0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.166+0000 7f9ff345f740 1 --2- 172.21.15.98:0/623408107 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561f12db630 0x5561f12ed3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.166+0000 7f9ff345f740 1 --2- 172.21.15.98:0/623408107 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5561f12d0d70 0x5561f12f03d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.166+0000 7f9ff345f740 1 --2- 172.21.15.98:0/623408107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5561f12d0450 0x5561f12efe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.166+0000 7f9ff345f740 1 -- 172.21.15.98:0/623408107 >> 172.21.15.98:0/623408107 conn(0x5561f12cc960 msgr2=0x5561f12e6670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.166+0000 7f9fd1ffb640 0 client.26200 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.166+0000 7f9ff345f740 1 -- 172.21.15.98:0/623408107 shutdown_connections 2024-05-09T08:55:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.166+0000 7f9ff345f740 1 -- 172.21.15.98:0/623408107 wait complete. 2024-05-09T08:55:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.166+0000 7f9ff345f740 20 client.26200 trim_cache size 0 max 16384 2024-05-09T08:55:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.168+0000 7f9ff075c640 1 -- 172.21.15.98:0/3031823303 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9fd40a2fd0 msgr2=0x7f9fd40a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.168+0000 7f9ff075c640 1 --2- 172.21.15.98:0/3031823303 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9fd40a2fd0 0x7f9fd40a5490 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x5561f12d19f0 tx=0x7f9fe0005b50 comp rx=0 tx=0).stop 2024-05-09T08:55:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.168+0000 7f9ff075c640 1 -- 172.21.15.98:0/3031823303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8110020 msgr2=0x7f9fe811a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.168+0000 7f9ff075c640 1 --2- 172.21.15.98:0/3031823303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8110020 0x7f9fe811a580 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000e9f0 tx=0x7f9fe000eeb0 comp rx=0 tx=0).stop 2024-05-09T08:55:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.169+0000 7f9ff075c640 1 -- 172.21.15.98:0/3031823303 shutdown_connections 2024-05-09T08:55:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.169+0000 7f9ff075c640 1 --2- 172.21.15.98:0/3031823303 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9fd40a2fd0 0x7f9fd40a5490 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.169+0000 7f9ff075c640 1 --2- 172.21.15.98:0/3031823303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe8113710 0x7f9fe8113bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.169+0000 7f9ff075c640 1 --2- 172.21.15.98:0/3031823303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fe811b010 0x7f9fe811aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.169+0000 7f9ff075c640 1 --2- 172.21.15.98:0/3031823303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe8110020 0x7f9fe811a580 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.169+0000 7f9ff075c640 1 -- 172.21.15.98:0/3031823303 >> 172.21.15.98:0/3031823303 conn(0x7f9fe806c900 msgr2=0x7f9fe811b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:29.170 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.170+0000 7f9ff075c640 1 -- 172.21.15.98:0/3031823303 shutdown_connections 2024-05-09T08:55:29.170 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:29.170+0000 7f9ff075c640 1 -- 172.21.15.98:0/3031823303 wait complete. 2024-05-09T08:55:30.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:30 smithi142 ceph-mon[34625]: pgmap v150: 129 pgs: 129 active+clean; 38 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 635 MiB/s wr, 1.34k op/s 2024-05-09T08:55:30.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:30 smithi105 ceph-mon[34498]: pgmap v150: 129 pgs: 129 active+clean; 38 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 635 MiB/s wr, 1.34k op/s 2024-05-09T08:55:30.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:30 smithi098 ceph-mon[30092]: pgmap v150: 129 pgs: 129 active+clean; 38 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 635 MiB/s wr, 1.34k op/s 2024-05-09T08:55:32.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:32 smithi105 ceph-mon[34498]: pgmap v151: 129 pgs: 129 active+clean; 39 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 832 MiB/s wr, 1.47k op/s 2024-05-09T08:55:32.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:32 smithi098 ceph-mon[30092]: pgmap v151: 129 pgs: 129 active+clean; 39 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 832 MiB/s wr, 1.47k op/s 2024-05-09T08:55:32.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:32 smithi142 ceph-mon[34625]: pgmap v151: 129 pgs: 129 active+clean; 39 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 832 MiB/s wr, 1.47k op/s 2024-05-09T08:55:33.308 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:55:33.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:55:33.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.450+0000 7fb8daffd640 1 -- 172.21.15.98:0/2474972388 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc0319b0 con 0x7fb8dc10fed0 2024-05-09T08:55:33.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.450+0000 7fb8e1468640 1 -- 172.21.15.98:0/2474972388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8dc10fed0 msgr2=0x7fb8dc1102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:33.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.450+0000 7fb8e1468640 1 --2- 172.21.15.98:0/2474972388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8dc10fed0 0x7fb8dc1102b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc009900 tx=0x7fb8cc02a480 comp rx=0 tx=0).stop 2024-05-09T08:55:33.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.450+0000 7fb8e1468640 1 -- 172.21.15.98:0/2474972388 shutdown_connections 2024-05-09T08:55:33.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.450+0000 7fb8e1468640 1 --2- 172.21.15.98:0/2474972388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8dc1092b0 0x7fb8dc11d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.450+0000 7fb8e1468640 1 --2- 172.21.15.98:0/2474972388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8dc108830 0x7fb8dc108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.450+0000 7fb8e1468640 1 --2- 172.21.15.98:0/2474972388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8dc10fed0 0x7fb8dc1102b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.450+0000 7fb8e1468640 1 -- 172.21.15.98:0/2474972388 >> 172.21.15.98:0/2474972388 conn(0x7fb8dc06c7b0 msgr2=0x7fb8dc06cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:33.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.450+0000 7fb8e1468640 1 -- 172.21.15.98:0/2474972388 shutdown_connections 2024-05-09T08:55:33.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.451+0000 7fb8e1468640 1 -- 172.21.15.98:0/2474972388 wait complete. 2024-05-09T08:55:33.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.451+0000 7fb8e1468640 1 Processor -- start 2024-05-09T08:55:33.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.451+0000 7fb8e1468640 1 -- start start 2024-05-09T08:55:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.452+0000 7fb8e1468640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8dc108830 0x7fb8dc1a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.452+0000 7fb8e1468640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8dc1092b0 0x7fb8dc1a7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.452+0000 7fb8e1468640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8dc10fed0 0x7fb8dc1ab690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.452+0000 7fb8e1468640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc1208b0 con 0x7fb8dc108830 2024-05-09T08:55:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.452+0000 7fb8e1468640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc120730 con 0x7fb8dc1092b0 2024-05-09T08:55:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.452+0000 7fb8e1468640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc120a30 con 0x7fb8dc10fed0 2024-05-09T08:55:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.453+0000 7fb8e0c67640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8dc10fed0 0x7fb8dc1ab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.453+0000 7fb8e0c67640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8dc10fed0 0x7fb8dc1ab690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:41036/0 (socket says 172.21.15.98:41036) 2024-05-09T08:55:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.453+0000 7fb8e0c67640 1 -- 172.21.15.98:0/793094782 learned_addr learned my addr 172.21.15.98:0/793094782 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.454+0000 7fb8e0c67640 1 -- 172.21.15.98:0/793094782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8dc1092b0 msgr2=0x7fb8dc1a7310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:55:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.454+0000 7fb8e0c67640 1 --2- 172.21.15.98:0/793094782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8dc1092b0 0x7fb8dc1a7310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.454+0000 7fb8e0c67640 1 -- 172.21.15.98:0/793094782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8dc108830 msgr2=0x7fb8dc1a6dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:55:33.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.454+0000 7fb8e0c67640 1 --2- 172.21.15.98:0/793094782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8dc108830 0x7fb8dc1a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.454+0000 7fb8e0c67640 1 -- 172.21.15.98:0/793094782 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8dc1abdc0 con 0x7fb8dc10fed0 2024-05-09T08:55:33.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.454+0000 7fb8e0c67640 1 --2- 172.21.15.98:0/793094782 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8dc10fed0 0x7fb8dc1ab690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400efd0 tx=0x7fb8d400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:33.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.454+0000 7fb8d97fa640 1 -- 172.21.15.98:0/793094782 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4003b70 con 0x7fb8dc10fed0 2024-05-09T08:55:33.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.454+0000 7fb8d97fa640 1 -- 172.21.15.98:0/793094782 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8d4003d10 con 0x7fb8dc10fed0 2024-05-09T08:55:33.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.456+0000 7fb8d97fa640 1 -- 172.21.15.98:0/793094782 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4003b70 con 0x7fb8dc10fed0 2024-05-09T08:55:33.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.456+0000 7fb8e1468640 1 -- 172.21.15.98:0/793094782 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8dc1ac0b0 con 0x7fb8dc10fed0 2024-05-09T08:55:33.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.456+0000 7fb8e1468640 1 -- 172.21.15.98:0/793094782 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8dc1b3940 con 0x7fb8dc10fed0 2024-05-09T08:55:33.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.460+0000 7fb8e53d6740 1 Processor -- start 2024-05-09T08:55:33.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.461+0000 7fb8d97fa640 1 -- 172.21.15.98:0/793094782 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb8d4003ec0 con 0x7fb8dc10fed0 2024-05-09T08:55:33.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.462+0000 7fb8d97fa640 1 --2- 172.21.15.98:0/793094782 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8c40a3140 0x7fb8c40a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.463+0000 7fb8d97fa640 1 -- 172.21.15.98:0/793094782 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fb8d4003970 con 0x7fb8dc10fed0 2024-05-09T08:55:33.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.463+0000 7fb8e53d6740 1 -- start start 2024-05-09T08:55:33.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.464+0000 7fb8e53d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b985b470 0x5633b985b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.464+0000 7fb8e53d6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b985bd90 0x5633b9865f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.464+0000 7fb8e53d6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b9866460 0x5633b9868850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.464+0000 7fb8e53d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5633b97c0d00 con 0x5633b985b470 2024-05-09T08:55:33.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.464+0000 7fb8e53d6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5633b97684c0 con 0x5633b985bd90 2024-05-09T08:55:33.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.464+0000 7fb8e53d6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5633b97bccc0 con 0x5633b9866460 2024-05-09T08:55:33.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.464+0000 7fb8db7fe640 1 --2- 172.21.15.98:0/793094782 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8c40a3140 0x7fb8c40a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.464+0000 7fb8dbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b985b470 0x5633b985b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.464+0000 7fb8dbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b985b470 0x5633b985b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47514/0 (socket says 172.21.15.98:47514) 2024-05-09T08:55:33.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.464+0000 7fb8dbfff640 1 -- 172.21.15.98:0/3805113584 learned_addr learned my addr 172.21.15.98:0/3805113584 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.464+0000 7fb8db7fe640 1 --2- 172.21.15.98:0/3805113584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b985bd90 0x5633b9865f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.465+0000 7fb8e0c67640 1 --2- 172.21.15.98:0/3805113584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b9866460 0x5633b9868850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.466+0000 7fb8e0c67640 1 -- 172.21.15.98:0/3805113584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b985bd90 msgr2=0x5633b9865f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.466+0000 7fb8e0c67640 1 --2- 172.21.15.98:0/3805113584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b985bd90 0x5633b9865f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.466+0000 7fb8e0c67640 1 -- 172.21.15.98:0/3805113584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b985b470 msgr2=0x5633b985b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.466+0000 7fb8e0c67640 1 --2- 172.21.15.98:0/3805113584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b985b470 0x5633b985b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.466+0000 7fb8e0c67640 1 -- 172.21.15.98:0/3805113584 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633b959e7d0 con 0x5633b9866460 2024-05-09T08:55:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.466+0000 7fb8e0c67640 1 --2- 172.21.15.98:0/3805113584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b9866460 0x5633b9868850 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb8d40cbf70 tx=0x7fb8d40a02a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fa3e54083b2f716 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.467+0000 7fb8c27fc640 1 -- 172.21.15.98:0/3805113584 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4097800 con 0x5633b9866460 2024-05-09T08:55:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.468+0000 7fb8c27fc640 1 -- 172.21.15.98:0/3805113584 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8d4003d10 con 0x5633b9866460 2024-05-09T08:55:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.468+0000 7fb8c27fc640 1 -- 172.21.15.98:0/3805113584 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d40979a0 con 0x5633b9866460 2024-05-09T08:55:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.469+0000 7fb8e53d6740 1 -- 172.21.15.98:0/3805113584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b9866460 msgr2=0x5633b9868850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.469+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/3805113584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b9866460 0x5633b9868850 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb8d40cbf70 tx=0x7fb8d40a02a0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.474+0000 7fb8e53d6740 1 -- 172.21.15.98:0/3805113584 shutdown_connections 2024-05-09T08:55:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.474+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/3805113584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b9866460 0x5633b9868850 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.474+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/3805113584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b985bd90 0x5633b9865f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.474+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/3805113584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b985b470 0x5633b985b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.474+0000 7fb8e53d6740 1 -- 172.21.15.98:0/3805113584 >> 172.21.15.98:0/3805113584 conn(0x5633b9857790 msgr2=0x5633b9857b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.475+0000 7fb8db7fe640 1 --2- 172.21.15.98:0/793094782 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8c40a3140 0x7fb8c40a5600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc10f1e0 tx=0x7fb8d0009650 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.486+0000 7fb8e53d6740 1 -- 172.21.15.98:0/3805113584 shutdown_connections 2024-05-09T08:55:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.486+0000 7fb8e53d6740 1 -- 172.21.15.98:0/3805113584 wait complete. 2024-05-09T08:55:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.488+0000 7fb8e53d6740 1 -- start start 2024-05-09T08:55:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.489+0000 7fb8e53d6740 1 Processor -- start 2024-05-09T08:55:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.493+0000 7fb8e53d6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b985b470 0x5633b98634d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.493+0000 7fb8e53d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9866460 0x5633b9863a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.493+0000 7fb8e53d6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b9863f50 0x5633b9872660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.493+0000 7fb8e53d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5633b98691f0 con 0x5633b9866460 2024-05-09T08:55:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.493+0000 7fb8e53d6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5633b977bb90 con 0x5633b985b470 2024-05-09T08:55:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.493+0000 7fb8e53d6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5633b9869370 con 0x5633b9863f50 2024-05-09T08:55:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.493+0000 7fb8e53d6740 10 client.0 _ll_register_callbacks cb 0x5633b97b7da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.496+0000 7fb8dbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b985b470 0x5633b98634d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.497+0000 7fb8dbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b985b470 0x5633b98634d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:47308/0 (socket says 172.21.15.98:47308) 2024-05-09T08:55:33.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.497+0000 7fb8dbfff640 1 -- 172.21.15.98:0/2776562779 learned_addr learned my addr 172.21.15.98:0/2776562779 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:33.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.500+0000 7fb8db7fe640 1 --2- 172.21.15.98:0/2776562779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9866460 0x5633b9863a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.500+0000 7fb8e0c67640 1 --2- 172.21.15.98:0/2776562779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b9863f50 0x5633b9872660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.500+0000 7fb8dbfff640 1 -- 172.21.15.98:0/2776562779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b9863f50 msgr2=0x5633b9872660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:33.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.500+0000 7fb8dbfff640 1 --2- 172.21.15.98:0/2776562779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b9863f50 0x5633b9872660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.501+0000 7fb8dbfff640 1 -- 172.21.15.98:0/2776562779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9866460 msgr2=0x5633b9863a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.501+0000 7fb8dbfff640 1 --2- 172.21.15.98:0/2776562779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9866460 0x5633b9863a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.501+0000 7fb8dbfff640 1 -- 172.21.15.98:0/2776562779 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633b94d43f0 con 0x5633b985b470 2024-05-09T08:55:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.504+0000 7fb8dbfff640 1 --2- 172.21.15.98:0/2776562779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b985b470 0x5633b98634d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc0099f0 tx=0x7fb8cc004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9a765ffb397f591 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:33.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.512+0000 7fb8bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:55:33.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.512+0000 7fb8bb7fe640 1 -- 172.21.15.98:0/2776562779 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc040070 con 0x5633b985b470 2024-05-09T08:55:33.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.512+0000 7fb8bb7fe640 1 -- 172.21.15.98:0/2776562779 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fb8cc005250 con 0x5633b985b470 2024-05-09T08:55:33.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.512+0000 7fb8e53d6740 1 -- 172.21.15.98:0/2776562779 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8d4003d10 con 0x5633b985b470 2024-05-09T08:55:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.515+0000 7fb8bb7fe640 1 -- 172.21.15.98:0/2776562779 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb8cc04c050 con 0x5633b985b470 2024-05-09T08:55:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.515+0000 7fb8e53d6740 10 client.26252 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.515+0000 7fb8e53d6740 1 -- 172.21.15.98:0/2776562779 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5633b94940f0 con 0x5633b985b470 2024-05-09T08:55:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.519+0000 7fb8bb7fe640 1 -- 172.21.15.98:0/2776562779 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8d4003d10 con 0x5633b985b470 2024-05-09T08:55:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.519+0000 7fb8bb7fe640 1 -- 172.21.15.98:0/2776562779 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fb8cc003790 con 0x5633b985b470 2024-05-09T08:55:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.519+0000 7fb8e53d6740 10 client.26252 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.519+0000 7fb8e53d6740 10 client.26252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.519+0000 7fb8e53d6740 20 client.26252 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.519+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/2776562779 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5633b98770a0 0x5633b9879500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.519+0000 7fb8e53d6740 4 client.26252 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.519+0000 7fb8e53d6740 1 -- 172.21.15.98:0/2776562779 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633b97ce430 con 0x5633b98770a0 2024-05-09T08:55:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.524+0000 7fb8db7fe640 1 --2- 172.21.15.98:0/2776562779 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5633b98770a0 0x5633b9879500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.535+0000 7fb8bb7fe640 10 client.26252 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.535+0000 7fb8db7fe640 1 --2- 172.21.15.98:0/2776562779 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5633b98770a0 0x5633b9879500 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2d1f42520a1c31 server_cookie=58479975a83b1f43 in_seq=0 out_seq=0 2024-05-09T08:55:33.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.540+0000 7fb8bb7fe640 1 -- 172.21.15.98:0/2776562779 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5633b97ce430 con 0x5633b98770a0 2024-05-09T08:55:33.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.540+0000 7fb8bb7fe640 10 client.26252 handle_command_reply: tid=0 2024-05-09T08:55:33.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.540+0000 7fb8e53d6740 1 client.26252 shutdown 2024-05-09T08:55:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.541+0000 7fb8e53d6740 1 -- 172.21.15.98:0/2776562779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b985b470 msgr2=0x5633b98634d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.541+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/2776562779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b985b470 0x5633b98634d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc0099f0 tx=0x7fb8cc004270 comp rx=0 tx=0).stop 2024-05-09T08:55:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.542+0000 7fb8e53d6740 1 -- 172.21.15.98:0/2776562779 shutdown_connections 2024-05-09T08:55:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.542+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/2776562779 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5633b98770a0 0x5633b9879500 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.542+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/2776562779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b9863f50 0x5633b9872660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.542+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/2776562779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9866460 0x5633b9863a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.542+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/2776562779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b985b470 0x5633b98634d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.542+0000 7fb8e53d6740 1 -- 172.21.15.98:0/2776562779 >> 172.21.15.98:0/2776562779 conn(0x5633b9857790 msgr2=0x5633b9856a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.543+0000 7fb8bb7fe640 0 client.26252 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.543+0000 7fb8e53d6740 1 -- 172.21.15.98:0/2776562779 shutdown_connections 2024-05-09T08:55:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.543+0000 7fb8e53d6740 1 -- 172.21.15.98:0/2776562779 wait complete. 2024-05-09T08:55:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.544+0000 7fb8e53d6740 20 client.26252 trim_cache size 0 max 16384 2024-05-09T08:55:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.609+0000 7fb8e53d6740 1 Processor -- start 2024-05-09T08:55:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.614+0000 7fb8e53d6740 1 -- start start 2024-05-09T08:55:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.614+0000 7fb8e53d6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b985b470 0x5633b985d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.614+0000 7fb8e53d6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b9863f50 0x5633b987ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.614+0000 7fb8e53d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9866460 0x5633b987f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.615+0000 7fb8e53d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc07da70 con 0x5633b9866460 2024-05-09T08:55:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.615+0000 7fb8e53d6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc07f5b0 con 0x5633b9863f50 2024-05-09T08:55:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.615+0000 7fb8e53d6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc07e140 con 0x5633b985b470 2024-05-09T08:55:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.615+0000 7fb8db7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b9863f50 0x5633b987ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.615+0000 7fb8db7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b9863f50 0x5633b987ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:47312/0 (socket says 172.21.15.98:47312) 2024-05-09T08:55:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.615+0000 7fb8db7fe640 1 -- 172.21.15.98:0/2049162980 learned_addr learned my addr 172.21.15.98:0/2049162980 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.615+0000 7fb8e0c67640 1 --2- 172.21.15.98:0/2049162980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9866460 0x5633b987f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.616+0000 7fb8db7fe640 1 -- 172.21.15.98:0/2049162980 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b985b470 msgr2=0x5633b985d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.616+0000 7fb8db7fe640 1 --2- 172.21.15.98:0/2049162980 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b985b470 0x5633b985d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.616+0000 7fb8db7fe640 1 -- 172.21.15.98:0/2049162980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9866460 msgr2=0x5633b987f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:33.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.616+0000 7fb8db7fe640 1 --2- 172.21.15.98:0/2049162980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9866460 0x5633b987f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.616+0000 7fb8db7fe640 1 -- 172.21.15.98:0/2049162980 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633b9863d70 con 0x5633b9863f50 2024-05-09T08:55:33.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.617+0000 7fb8db7fe640 1 --2- 172.21.15.98:0/2049162980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b9863f50 0x5633b987ee10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0019000 tx=0x7fb8d001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abe084fd12f6a5a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:33.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.620+0000 7fb8c1ffb640 1 -- 172.21.15.98:0/2049162980 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0010d50 con 0x5633b9863f50 2024-05-09T08:55:33.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.621+0000 7fb8c1ffb640 1 -- 172.21.15.98:0/2049162980 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8d0028890 con 0x5633b9863f50 2024-05-09T08:55:33.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.621+0000 7fb8c1ffb640 1 -- 172.21.15.98:0/2049162980 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d00047a0 con 0x5633b9863f50 2024-05-09T08:55:33.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.622+0000 7fb8e53d6740 1 -- 172.21.15.98:0/2049162980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b9863f50 msgr2=0x5633b987ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:33.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.622+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/2049162980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b9863f50 0x5633b987ee10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0019000 tx=0x7fb8d001a000 comp rx=0 tx=0).stop 2024-05-09T08:55:33.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.622+0000 7fb8e53d6740 1 -- 172.21.15.98:0/2049162980 shutdown_connections 2024-05-09T08:55:33.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.622+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/2049162980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9866460 0x5633b987f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.622+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/2049162980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b9863f50 0x5633b987ee10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.622+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/2049162980 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b985b470 0x5633b985d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.622+0000 7fb8e53d6740 1 -- 172.21.15.98:0/2049162980 >> 172.21.15.98:0/2049162980 conn(0x5633b9857790 msgr2=0x5633b985cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:33.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.631+0000 7fb8e53d6740 1 -- 172.21.15.98:0/2049162980 shutdown_connections 2024-05-09T08:55:33.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.631+0000 7fb8e53d6740 1 -- 172.21.15.98:0/2049162980 wait complete. 2024-05-09T08:55:33.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.632+0000 7fb8e53d6740 1 -- start start 2024-05-09T08:55:33.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.632+0000 7fb8e53d6740 1 Processor -- start 2024-05-09T08:55:33.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.633+0000 7fb8e53d6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b985b470 0x5633b9869870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.633+0000 7fb8e53d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9863f50 0x5633b9869db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.633+0000 7fb8e53d6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b9866460 0x5633b98d56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.633+0000 7fb8e53d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc07eee0 con 0x5633b9863f50 2024-05-09T08:55:33.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.633+0000 7fb8e53d6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc07e810 con 0x5633b9866460 2024-05-09T08:55:33.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.633+0000 7fb8e53d6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc07ba50 con 0x5633b985b470 2024-05-09T08:55:33.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.633+0000 7fb8e53d6740 10 client.0 _ll_register_callbacks cb 0x7fb8d0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:33.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.634+0000 7fb8db7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9863f50 0x5633b9869db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.634+0000 7fb8db7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9863f50 0x5633b9869db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47538/0 (socket says 172.21.15.98:47538) 2024-05-09T08:55:33.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.634+0000 7fb8db7fe640 1 -- 172.21.15.98:0/3996345699 learned_addr learned my addr 172.21.15.98:0/3996345699 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:33.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.634+0000 7fb8e0c67640 1 --2- 172.21.15.98:0/3996345699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b9866460 0x5633b98d56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.635+0000 7fb8dbfff640 1 --2- 172.21.15.98:0/3996345699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b985b470 0x5633b9869870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.635+0000 7fb8db7fe640 1 -- 172.21.15.98:0/3996345699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b985b470 msgr2=0x5633b9869870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:33.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.635+0000 7fb8db7fe640 1 --2- 172.21.15.98:0/3996345699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b985b470 0x5633b9869870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.636+0000 7fb8db7fe640 1 -- 172.21.15.98:0/3996345699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b9866460 msgr2=0x5633b98d56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:33.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.636+0000 7fb8db7fe640 1 --2- 172.21.15.98:0/3996345699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b9866460 0x5633b98d56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.636+0000 7fb8db7fe640 1 -- 172.21.15.98:0/3996345699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633b98d5cf0 con 0x5633b9863f50 2024-05-09T08:55:33.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.637+0000 7fb8db7fe640 1 --2- 172.21.15.98:0/3996345699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9863f50 0x5633b9869db0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fb8d002c000 tx=0x7fb8d0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985314290b5736a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:33.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.637+0000 7fb8bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:55:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.637+0000 7fb8bb7fe640 1 -- 172.21.15.98:0/3996345699 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0010d50 con 0x5633b9863f50 2024-05-09T08:55:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.638+0000 7fb8e53d6740 1 -- 172.21.15.98:0/3996345699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8d0028890 con 0x5633b9863f50 2024-05-09T08:55:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.639+0000 7fb8bb7fe640 1 -- 172.21.15.98:0/3996345699 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fb8d00054d0 con 0x5633b9863f50 2024-05-09T08:55:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.640+0000 7fb8bb7fe640 1 -- 172.21.15.98:0/3996345699 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb8d00283b0 con 0x5633b9863f50 2024-05-09T08:55:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.640+0000 7fb8e53d6740 10 client.16962 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.640+0000 7fb8e53d6740 1 -- 172.21.15.98:0/3996345699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5633b98ee0c0 con 0x5633b9863f50 2024-05-09T08:55:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.641+0000 7fb8bb7fe640 1 -- 172.21.15.98:0/3996345699 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8d0028890 con 0x5633b9863f50 2024-05-09T08:55:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.641+0000 7fb8bb7fe640 1 -- 172.21.15.98:0/3996345699 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fb8d0032bb0 con 0x5633b9863f50 2024-05-09T08:55:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.642+0000 7fb8e53d6740 10 client.16962 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.642+0000 7fb8e53d6740 10 client.16962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.642+0000 7fb8e53d6740 20 client.16962 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.642+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/3996345699 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5633b98ee8d0 0x5633b98f0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.642+0000 7fb8e53d6740 4 client.16962 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:55:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.642+0000 7fb8e53d6740 1 -- 172.21.15.98:0/3996345699 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5633b97c8780 con 0x5633b98ee8d0 2024-05-09T08:55:33.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.645+0000 7fb8dbfff640 1 --2- 172.21.15.98:0/3996345699 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5633b98ee8d0 0x5633b98f0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.650+0000 7fb8dbfff640 1 --2- 172.21.15.98:0/3996345699 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5633b98ee8d0 0x5633b98f0cf0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97fae23a504bbbd9 server_cookie=e74a175ec563dc32 in_seq=0 out_seq=0 2024-05-09T08:55:33.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.651+0000 7fb8bb7fe640 10 client.16962 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:33.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.652+0000 7fb8bb7fe640 1 -- 172.21.15.98:0/3996345699 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5633b97c8780 con 0x5633b98ee8d0 2024-05-09T08:55:33.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.653+0000 7fb8bb7fe640 10 client.16962 handle_command_reply: tid=0 2024-05-09T08:55:33.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.653+0000 7fb8e53d6740 1 client.16962 shutdown 2024-05-09T08:55:33.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.653+0000 7fb8e53d6740 1 -- 172.21.15.98:0/3996345699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9863f50 msgr2=0x5633b9869db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:33.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.653+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/3996345699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9863f50 0x5633b9869db0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fb8d002c000 tx=0x7fb8d0000bc0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.657+0000 7fb8e53d6740 1 -- 172.21.15.98:0/3996345699 shutdown_connections 2024-05-09T08:55:33.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.657+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/3996345699 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5633b98ee8d0 0x5633b98f0cf0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.657+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/3996345699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633b9866460 0x5633b98d56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.657+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/3996345699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9863f50 0x5633b9869db0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.657+0000 7fb8e53d6740 1 --2- 172.21.15.98:0/3996345699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5633b985b470 0x5633b9869870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.657+0000 7fb8e53d6740 1 -- 172.21.15.98:0/3996345699 >> 172.21.15.98:0/3996345699 conn(0x5633b9857790 msgr2=0x5633b985d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:33.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.657+0000 7fb8bb7fe640 0 client.16962 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:33.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.661+0000 7fb8e53d6740 1 -- 172.21.15.98:0/3996345699 shutdown_connections 2024-05-09T08:55:33.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.661+0000 7fb8e53d6740 1 -- 172.21.15.98:0/3996345699 wait complete. 2024-05-09T08:55:33.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.662+0000 7fb8e53d6740 20 client.16962 trim_cache size 0 max 16384 2024-05-09T08:55:33.668 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:55:33.668 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:55:33.668 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:55:33.668 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:55:33.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.670+0000 7fb8e1468640 1 -- 172.21.15.98:0/793094782 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8c40a3140 msgr2=0x7fb8c40a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:33.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.670+0000 7fb8e1468640 1 --2- 172.21.15.98:0/793094782 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8c40a3140 0x7fb8c40a5600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc10f1e0 tx=0x7fb8d0009650 comp rx=0 tx=0).stop 2024-05-09T08:55:33.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.670+0000 7fb8e1468640 1 -- 172.21.15.98:0/793094782 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8dc10fed0 msgr2=0x7fb8dc1ab690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:33.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.670+0000 7fb8e1468640 1 --2- 172.21.15.98:0/793094782 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8dc10fed0 0x7fb8dc1ab690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400efd0 tx=0x7fb8d400c6a0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.673+0000 7fb8e1468640 1 -- 172.21.15.98:0/793094782 shutdown_connections 2024-05-09T08:55:33.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.673+0000 7fb8e1468640 1 --2- 172.21.15.98:0/793094782 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8c40a3140 0x7fb8c40a5600 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.673+0000 7fb8e1468640 1 --2- 172.21.15.98:0/793094782 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8dc10fed0 0x7fb8dc1ab690 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.673+0000 7fb8e1468640 1 --2- 172.21.15.98:0/793094782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8dc1092b0 0x7fb8dc1a7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.673+0000 7fb8e1468640 1 --2- 172.21.15.98:0/793094782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8dc108830 0x7fb8dc1a6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.673+0000 7fb8e1468640 1 -- 172.21.15.98:0/793094782 >> 172.21.15.98:0/793094782 conn(0x7fb8dc06c7b0 msgr2=0x7fb8dc070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:33.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.673+0000 7fb8e1468640 1 -- 172.21.15.98:0/793094782 shutdown_connections 2024-05-09T08:55:33.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.673+0000 7fb8e1468640 1 -- 172.21.15.98:0/793094782 wait complete. 2024-05-09T08:55:33.706 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:55:33.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:55:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.969+0000 7f03fa9e2640 1 -- 172.21.15.98:0/1775207511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03f4106710 msgr2=0x7f03f4106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.969+0000 7f03fa9e2640 1 --2- 172.21.15.98:0/1775207511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03f4106710 0x7f03f4106af0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009cd0 tx=0x7f03ec02a8f0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.969+0000 7f03fa9e2640 1 -- 172.21.15.98:0/1775207511 shutdown_connections 2024-05-09T08:55:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.969+0000 7f03fa9e2640 1 --2- 172.21.15.98:0/1775207511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03f41010c0 0x7f03f410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.969+0000 7f03fa9e2640 1 --2- 172.21.15.98:0/1775207511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03f4100700 0x7f03f4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.969+0000 7f03fa9e2640 1 --2- 172.21.15.98:0/1775207511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03f4106710 0x7f03f4106af0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.969+0000 7f03fa9e2640 1 -- 172.21.15.98:0/1775207511 >> 172.21.15.98:0/1775207511 conn(0x7f03f4076ab0 msgr2=0x7f03f4076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.970+0000 7f03fa9e2640 1 -- 172.21.15.98:0/1775207511 shutdown_connections 2024-05-09T08:55:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.970+0000 7f03fa9e2640 1 -- 172.21.15.98:0/1775207511 wait complete. 2024-05-09T08:55:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.971+0000 7f03fa9e2640 1 Processor -- start 2024-05-09T08:55:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.971+0000 7f03fa9e2640 1 -- start start 2024-05-09T08:55:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.972+0000 7f03fa9e2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03f4100700 0x7f03f4073030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.972+0000 7f03fa9e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03f41010c0 0x7f03f406c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.972+0000 7f03fa9e2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03f4106710 0x7f03f406c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.972+0000 7f03fa9e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03f410fa60 con 0x7f03f41010c0 2024-05-09T08:55:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.972+0000 7f03fa9e2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f03f410f8e0 con 0x7f03f4106710 2024-05-09T08:55:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.972+0000 7f03fa9e2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f03f410fbe0 con 0x7f03f4100700 2024-05-09T08:55:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.972+0000 7f03f37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03f41010c0 0x7f03f406c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.972+0000 7f03f37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03f41010c0 0x7f03f406c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47566/0 (socket says 172.21.15.98:47566) 2024-05-09T08:55:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.972+0000 7f03f37fe640 1 -- 172.21.15.98:0/4247976413 learned_addr learned my addr 172.21.15.98:0/4247976413 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.972+0000 7f03f3fff640 1 --2- 172.21.15.98:0/4247976413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03f4100700 0x7f03f4073030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.972+0000 7f03f8f7f640 1 --2- 172.21.15.98:0/4247976413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03f4106710 0x7f03f406c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.973+0000 7f03f3fff640 1 -- 172.21.15.98:0/4247976413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03f4106710 msgr2=0x7f03f406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.973+0000 7f03f3fff640 1 --2- 172.21.15.98:0/4247976413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03f4106710 0x7f03f406c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.973+0000 7f03f3fff640 1 -- 172.21.15.98:0/4247976413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03f41010c0 msgr2=0x7f03f406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.973+0000 7f03f3fff640 1 --2- 172.21.15.98:0/4247976413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03f41010c0 0x7f03f406c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.973+0000 7f03f3fff640 1 -- 172.21.15.98:0/4247976413 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f406cc00 con 0x7f03f4100700 2024-05-09T08:55:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.973+0000 7f03f3fff640 1 --2- 172.21.15.98:0/4247976413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03f4100700 0x7f03f4073030 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009ca0 tx=0x7f03ec003f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.974+0000 7f03f17fa640 1 -- 172.21.15.98:0/4247976413 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec030ec0 con 0x7f03f4100700 2024-05-09T08:55:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.974+0000 7f03fa9e2640 1 -- 172.21.15.98:0/4247976413 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f406ce90 con 0x7f03f4100700 2024-05-09T08:55:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.974+0000 7f03fa9e2640 1 -- 172.21.15.98:0/4247976413 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f4070e40 con 0x7f03f4100700 2024-05-09T08:55:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.974+0000 7f03f17fa640 1 -- 172.21.15.98:0/4247976413 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03ec040070 con 0x7f03f4100700 2024-05-09T08:55:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.974+0000 7f03f17fa640 1 -- 172.21.15.98:0/4247976413 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec031b30 con 0x7f03f4100700 2024-05-09T08:55:33.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.977+0000 7f03fa9e2640 1 -- 172.21.15.98:0/4247976413 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f4101e40 con 0x7f03f4100700 2024-05-09T08:55:33.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.979+0000 7f03f17fa640 1 -- 172.21.15.98:0/4247976413 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f03ec031cd0 con 0x7f03f4100700 2024-05-09T08:55:33.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.980+0000 7f03f17fa640 1 --2- 172.21.15.98:0/4247976413 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f03dc0a3070 0x7f03dc0a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:33.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.981+0000 7f03f17fa640 1 -- 172.21.15.98:0/4247976413 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f03ec0f06d0 con 0x7f03f4100700 2024-05-09T08:55:33.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.981+0000 7f03f37fe640 1 --2- 172.21.15.98:0/4247976413 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f03dc0a3070 0x7f03dc0a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:33.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.985+0000 7f03f17fa640 1 -- 172.21.15.98:0/4247976413 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f03ec0bcfe0 con 0x7f03f4100700 2024-05-09T08:55:33.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:33.995+0000 7f03f37fe640 1 --2- 172.21.15.98:0/4247976413 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f03dc0a3070 0x7f03dc0a5530 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f03f406d870 tx=0x7f03e800b630 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:34.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:34 smithi105 ceph-mon[34498]: pgmap v152: 129 pgs: 129 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 708 MiB/s wr, 1.15k op/s 2024-05-09T08:55:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.533+0000 7f03fa9e2640 1 -- 172.21.15.98:0/4247976413 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03f4071920 con 0x7f03f4100700 2024-05-09T08:55:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.535+0000 7f03f17fa640 1 -- 172.21.15.98:0/4247976413 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11944 (secure 0 0 0) 0x7f03ec0bc8e0 con 0x7f03f4100700 2024-05-09T08:55:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:55:34.537 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":16641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:54:25.881641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16749":{"gid":16749,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/3204012039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3204012039},{"type":"v1","addr":"172.21.15.105:6835","nonce":3204012039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24439,14667,24388,24461,24457]},"id":1}]} 2024-05-09T08:55:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 35 2024-05-09T08:55:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.560+0000 7f03fa9e2640 1 -- 172.21.15.98:0/4247976413 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f03dc0a3070 msgr2=0x7f03dc0a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.560+0000 7f03fa9e2640 1 --2- 172.21.15.98:0/4247976413 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f03dc0a3070 0x7f03dc0a5530 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f03f406d870 tx=0x7f03e800b630 comp rx=0 tx=0).stop 2024-05-09T08:55:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.560+0000 7f03fa9e2640 1 -- 172.21.15.98:0/4247976413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03f4100700 msgr2=0x7f03f4073030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.560+0000 7f03fa9e2640 1 --2- 172.21.15.98:0/4247976413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03f4100700 0x7f03f4073030 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009ca0 tx=0x7f03ec003f80 comp rx=0 tx=0).stop 2024-05-09T08:55:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.560+0000 7f03fa9e2640 1 -- 172.21.15.98:0/4247976413 shutdown_connections 2024-05-09T08:55:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.560+0000 7f03fa9e2640 1 --2- 172.21.15.98:0/4247976413 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f03dc0a3070 0x7f03dc0a5530 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f03f406d870 tx=0x7f03e800b630 comp rx=0 tx=0).stop 2024-05-09T08:55:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.560+0000 7f03fa9e2640 1 --2- 172.21.15.98:0/4247976413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03f4106710 0x7f03f406c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.560+0000 7f03fa9e2640 1 --2- 172.21.15.98:0/4247976413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03f41010c0 0x7f03f406c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.560+0000 7f03fa9e2640 1 --2- 172.21.15.98:0/4247976413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03f4100700 0x7f03f4073030 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.560+0000 7f03fa9e2640 1 -- 172.21.15.98:0/4247976413 >> 172.21.15.98:0/4247976413 conn(0x7f03f4076ab0 msgr2=0x7f03f40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.564+0000 7f03fa9e2640 1 -- 172.21.15.98:0/4247976413 shutdown_connections 2024-05-09T08:55:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.568+0000 7f03fa9e2640 1 -- 172.21.15.98:0/4247976413 wait complete. 2024-05-09T08:55:34.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:34 smithi098 ceph-mon[30092]: pgmap v152: 129 pgs: 129 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 708 MiB/s wr, 1.15k op/s 2024-05-09T08:55:34.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:55:34.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:34 smithi142 ceph-mon[34625]: pgmap v152: 129 pgs: 129 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 708 MiB/s wr, 1.15k op/s 2024-05-09T08:55:34.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.889+0000 7f957359e640 1 -- 172.21.15.98:0/2528715099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95741092b0 msgr2=0x7f957411d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:34.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.889+0000 7f957359e640 1 --2- 172.21.15.98:0/2528715099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95741092b0 0x7f957411d020 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f956c009ca0 tx=0x7f956c02a940 comp rx=0 tx=0).stop 2024-05-09T08:55:34.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.891+0000 7f957359e640 1 -- 172.21.15.98:0/2528715099 shutdown_connections 2024-05-09T08:55:34.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.891+0000 7f957359e640 1 --2- 172.21.15.98:0/2528715099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95741092b0 0x7f957411d020 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.891+0000 7f957359e640 1 --2- 172.21.15.98:0/2528715099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9574108830 0x7f9574108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.891+0000 7f957359e640 1 --2- 172.21.15.98:0/2528715099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f957410fed0 0x7f95741102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.891+0000 7f957359e640 1 -- 172.21.15.98:0/2528715099 >> 172.21.15.98:0/2528715099 conn(0x7f957406c7b0 msgr2=0x7f957406cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:34.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.898+0000 7f957359e640 1 -- 172.21.15.98:0/2528715099 shutdown_connections 2024-05-09T08:55:34.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.901+0000 7f957359e640 1 -- 172.21.15.98:0/2528715099 wait complete. 2024-05-09T08:55:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.902+0000 7f957359e640 1 Processor -- start 2024-05-09T08:55:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.902+0000 7f957359e640 1 -- start start 2024-05-09T08:55:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.903+0000 7f957359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9574108830 0x7f95741a6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.903+0000 7f957359e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95741092b0 0x7f95741a72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.903+0000 7f957359e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f957410fed0 0x7f95741ab620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.903+0000 7f957359e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9574120860 con 0x7f9574108830 2024-05-09T08:55:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.903+0000 7f957359e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f95741206e0 con 0x7f95741092b0 2024-05-09T08:55:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.903+0000 7f957359e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f95741209e0 con 0x7f957410fed0 2024-05-09T08:55:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.905+0000 7f9572d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f957410fed0 0x7f95741ab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.905+0000 7f957259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9574108830 0x7f95741a6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.906+0000 7f957259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9574108830 0x7f95741a6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47586/0 (socket says 172.21.15.98:47586) 2024-05-09T08:55:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.906+0000 7f957259c640 1 -- 172.21.15.98:0/3357133545 learned_addr learned my addr 172.21.15.98:0/3357133545 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.906+0000 7f9571d9b640 1 --2- 172.21.15.98:0/3357133545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95741092b0 0x7f95741a72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:34.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.908+0000 7f9572d9d640 1 -- 172.21.15.98:0/3357133545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95741092b0 msgr2=0x7f95741a72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:34.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.908+0000 7f9572d9d640 1 --2- 172.21.15.98:0/3357133545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95741092b0 0x7f95741a72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.908+0000 7f9572d9d640 1 -- 172.21.15.98:0/3357133545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9574108830 msgr2=0x7f95741a6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.908+0000 7f9572d9d640 1 --2- 172.21.15.98:0/3357133545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9574108830 0x7f95741a6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.909+0000 7f9572d9d640 1 -- 172.21.15.98:0/3357133545 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95741abd50 con 0x7f957410fed0 2024-05-09T08:55:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.909+0000 7f9572d9d640 1 --2- 172.21.15.98:0/3357133545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f957410fed0 0x7f95741ab620 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f956c02af30 tx=0x7f956c02f700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.910+0000 7f95637fe640 1 -- 172.21.15.98:0/3357133545 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c034040 con 0x7f957410fed0 2024-05-09T08:55:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.910+0000 7f957359e640 1 -- 172.21.15.98:0/3357133545 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95741abfe0 con 0x7f957410fed0 2024-05-09T08:55:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.910+0000 7f957359e640 1 -- 172.21.15.98:0/3357133545 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95741b38d0 con 0x7f957410fed0 2024-05-09T08:55:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.914+0000 7f957bbac740 1 Processor -- start 2024-05-09T08:55:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.914+0000 7f957bbac740 1 -- start start 2024-05-09T08:55:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.914+0000 7f957bbac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b180520 0x563a1b180900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.914+0000 7f957bbac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b18afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.915+0000 7f957bbac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b18b510 0x563a1b18d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:34.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.915+0000 7f957bbac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563a1b0e5c40 con 0x563a1b18b510 2024-05-09T08:55:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.915+0000 7f957bbac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563a1b08d580 con 0x563a1b180520 2024-05-09T08:55:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.915+0000 7f957bbac740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563a1b0e1c00 con 0x563a1b180e40 2024-05-09T08:55:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.915+0000 7f95637fe640 1 -- 172.21.15.98:0/3357133545 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f956c02fe10 con 0x7f957410fed0 2024-05-09T08:55:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.915+0000 7f95637fe640 1 -- 172.21.15.98:0/3357133545 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c004570 con 0x7f957410fed0 2024-05-09T08:55:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.915+0000 7f9571d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b180520 0x563a1b180900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.915+0000 7f9571d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b180520 0x563a1b180900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:47374/0 (socket says 172.21.15.98:47374) 2024-05-09T08:55:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.915+0000 7f9571d9b640 1 -- 172.21.15.98:0/2067073591 learned_addr learned my addr 172.21.15.98:0/2067073591 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.920+0000 7f9571d9b640 1 -- 172.21.15.98:0/2067073591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 msgr2=0x563a1b18afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.920+0000 7f9571d9b640 1 --2- 172.21.15.98:0/2067073591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b18afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.920+0000 7f9571d9b640 1 -- 172.21.15.98:0/2067073591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b18b510 msgr2=0x563a1b18d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:34.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.920+0000 7f9571d9b640 1 --2- 172.21.15.98:0/2067073591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b18b510 0x563a1b18d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.920+0000 7f9571d9b640 1 -- 172.21.15.98:0/2067073591 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a1adf93f0 con 0x563a1b180520 2024-05-09T08:55:34.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.924+0000 7f9571d9b640 1 --2- 172.21.15.98:0/2067073591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b180520 0x563a1b180900 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f956800ec10 tx=0x7f9568010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b0ba55a06fca079 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:34.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.924+0000 7f95637fe640 1 -- 172.21.15.98:0/3357133545 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f956c037020 con 0x7f957410fed0 2024-05-09T08:55:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.926+0000 7f9560ff9640 1 -- 172.21.15.98:0/2067073591 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9568012450 con 0x563a1b180520 2024-05-09T08:55:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.926+0000 7f95637fe640 1 --2- 172.21.15.98:0/3357133545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f955c0a2f00 0x7f955c0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.926+0000 7f9560ff9640 1 -- 172.21.15.98:0/2067073591 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9568012ad0 con 0x563a1b180520 2024-05-09T08:55:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.926+0000 7f9560ff9640 1 -- 172.21.15.98:0/2067073591 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956801aab0 con 0x563a1b180520 2024-05-09T08:55:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.927+0000 7f95637fe640 1 -- 172.21.15.98:0/3357133545 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f956c0f0550 con 0x7f957410fed0 2024-05-09T08:55:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.928+0000 7f957bbac740 1 -- 172.21.15.98:0/2067073591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b180520 msgr2=0x563a1b180900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.928+0000 7f957bbac740 1 --2- 172.21.15.98:0/2067073591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b180520 0x563a1b180900 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f956800ec10 tx=0x7f9568010740 comp rx=0 tx=0).stop 2024-05-09T08:55:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.928+0000 7f957259c640 1 --2- 172.21.15.98:0/3357133545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f955c0a2f00 0x7f955c0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.929+0000 7f957bbac740 1 -- 172.21.15.98:0/2067073591 shutdown_connections 2024-05-09T08:55:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.929+0000 7f957bbac740 1 --2- 172.21.15.98:0/2067073591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b18b510 0x563a1b18d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.929+0000 7f957bbac740 1 --2- 172.21.15.98:0/2067073591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b18afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.929+0000 7f957bbac740 1 --2- 172.21.15.98:0/2067073591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b180520 0x563a1b180900 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.929+0000 7f957bbac740 1 -- 172.21.15.98:0/2067073591 >> 172.21.15.98:0/2067073591 conn(0x563a1b17c840 msgr2=0x563a1b17cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.930+0000 7f957259c640 1 --2- 172.21.15.98:0/3357133545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f955c0a2f00 0x7f955c0a53c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x563a1b181ae0 tx=0x7f9564009880 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:34.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.934+0000 7f957bbac740 1 -- 172.21.15.98:0/2067073591 shutdown_connections 2024-05-09T08:55:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.938+0000 7f957bbac740 1 -- 172.21.15.98:0/2067073591 wait complete. 2024-05-09T08:55:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.938+0000 7f957bbac740 1 -- start start 2024-05-09T08:55:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.938+0000 7f957bbac740 1 Processor -- start 2024-05-09T08:55:34.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.941+0000 7f957bbac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b180520 0x563a1b1883d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:34.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.941+0000 7f957bbac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b188910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:34.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.941+0000 7f957bbac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b18b510 0x563a1b188e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.941+0000 7f957bbac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563a1b18e300 con 0x563a1b180520 2024-05-09T08:55:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.941+0000 7f957bbac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563a1b0a0c40 con 0x563a1b18b510 2024-05-09T08:55:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.941+0000 7f957bbac740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563a1b18e480 con 0x563a1b180e40 2024-05-09T08:55:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.941+0000 7f957bbac740 10 client.0 _ll_register_callbacks cb 0x563a1adf8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.942+0000 7f9572d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b18b510 0x563a1b188e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.942+0000 7f9572d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b18b510 0x563a1b188e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:47384/0 (socket says 172.21.15.98:47384) 2024-05-09T08:55:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.942+0000 7f9572d9d640 1 -- 172.21.15.98:0/3023912007 learned_addr learned my addr 172.21.15.98:0/3023912007 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.943+0000 7f9571d9b640 1 --2- 172.21.15.98:0/3023912007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b188910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.943+0000 7f9572d9d640 1 -- 172.21.15.98:0/3023912007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 msgr2=0x563a1b188910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.943+0000 7f9572d9d640 1 --2- 172.21.15.98:0/3023912007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b188910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.943+0000 7f9572d9d640 1 -- 172.21.15.98:0/3023912007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b180520 msgr2=0x563a1b1883d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.943+0000 7f9572d9d640 1 --2- 172.21.15.98:0/3023912007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b180520 0x563a1b1883d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.943+0000 7f9572d9d640 1 -- 172.21.15.98:0/3023912007 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a1aec37d0 con 0x563a1b18b510 2024-05-09T08:55:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.945+0000 7f9572d9d640 1 --2- 172.21.15.98:0/3023912007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b18b510 0x563a1b188e50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f956c0b9130 tx=0x7f956c0bc0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=923b745831bb0afa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.946+0000 7f9559ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:55:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.946+0000 7f9559ffb640 1 -- 172.21.15.98:0/3023912007 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c0bc760 con 0x563a1b18b510 2024-05-09T08:55:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.946+0000 7f957bbac740 1 -- 172.21.15.98:0/3023912007 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9568012ad0 con 0x563a1b18b510 2024-05-09T08:55:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.946+0000 7f9559ffb640 1 -- 172.21.15.98:0/3023912007 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f956c0f0160 con 0x563a1b18b510 2024-05-09T08:55:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.948+0000 7f9559ffb640 1 -- 172.21.15.98:0/3023912007 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f956c0c0b10 con 0x563a1b18b510 2024-05-09T08:55:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.957+0000 7f957bbac740 10 client.26270 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:34.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.957+0000 7f957bbac740 1 -- 172.21.15.98:0/3023912007 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563a1b19a4c0 con 0x563a1b18b510 2024-05-09T08:55:34.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.964+0000 7f9559ffb640 1 -- 172.21.15.98:0/3023912007 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9568012ad0 con 0x563a1b18b510 2024-05-09T08:55:34.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.964+0000 7f9559ffb640 1 -- 172.21.15.98:0/3023912007 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f956c02f070 con 0x563a1b18b510 2024-05-09T08:55:34.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.968+0000 7f957bbac740 10 client.26270 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:34.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.968+0000 7f957bbac740 10 client.26270 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:55:34.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.968+0000 7f957bbac740 20 client.26270 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:34.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.968+0000 7f957bbac740 1 --2- 172.21.15.98:0/3023912007 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563a1b19b8b0 0x563a1b19dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:34.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.968+0000 7f957bbac740 4 client.26270 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:34.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.968+0000 7f957bbac740 1 -- 172.21.15.98:0/3023912007 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a1b0f3370 con 0x563a1b19b8b0 2024-05-09T08:55:34.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.972+0000 7f957259c640 1 --2- 172.21.15.98:0/3023912007 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563a1b19b8b0 0x563a1b19dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:34.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.988+0000 7f957259c640 1 --2- 172.21.15.98:0/3023912007 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563a1b19b8b0 0x563a1b19dce0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1510f161c26b4397 server_cookie=78727c176699a916 in_seq=0 out_seq=0 2024-05-09T08:55:34.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.990+0000 7f9559ffb640 10 client.26270 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:55:34.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.993+0000 7f9559ffb640 1 -- 172.21.15.98:0/3023912007 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a1b0f3370 con 0x563a1b19b8b0 2024-05-09T08:55:34.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.993+0000 7f9559ffb640 10 client.26270 handle_command_reply: tid=0 2024-05-09T08:55:34.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.994+0000 7f957bbac740 1 client.26270 shutdown 2024-05-09T08:55:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.994+0000 7f957bbac740 1 -- 172.21.15.98:0/3023912007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b18b510 msgr2=0x563a1b188e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.994+0000 7f957bbac740 1 --2- 172.21.15.98:0/3023912007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b18b510 0x563a1b188e50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f956c0b9130 tx=0x7f956c0bc0c0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.995+0000 7f957bbac740 1 -- 172.21.15.98:0/3023912007 shutdown_connections 2024-05-09T08:55:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.995+0000 7f957bbac740 1 --2- 172.21.15.98:0/3023912007 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563a1b19b8b0 0x563a1b19dce0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.995+0000 7f957bbac740 1 --2- 172.21.15.98:0/3023912007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b18b510 0x563a1b188e50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.995+0000 7f957bbac740 1 --2- 172.21.15.98:0/3023912007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b188910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.995+0000 7f957bbac740 1 --2- 172.21.15.98:0/3023912007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b180520 0x563a1b1883d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.995+0000 7f957bbac740 1 -- 172.21.15.98:0/3023912007 >> 172.21.15.98:0/3023912007 conn(0x563a1b17c840 msgr2=0x563a1b182540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.995+0000 7f9559ffb640 0 client.26270 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:55:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.995+0000 7f957bbac740 1 -- 172.21.15.98:0/3023912007 shutdown_connections 2024-05-09T08:55:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:34.999+0000 7f957bbac740 1 -- 172.21.15.98:0/3023912007 wait complete. 2024-05-09T08:55:35.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.000+0000 7f957bbac740 20 client.26270 trim_cache size 0 max 16384 2024-05-09T08:55:35.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.047+0000 7f957bbac740 1 Processor -- start 2024-05-09T08:55:35.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.052+0000 7f957bbac740 1 -- start start 2024-05-09T08:55:35.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.052+0000 7f957bbac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b180520 0x563a1b18ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.052+0000 7f957bbac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b182c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.053+0000 7f957bbac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b18b510 0x563a1b17c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.053+0000 7f9571d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b182c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.053+0000 7f9571d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b182c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:41154/0 (socket says 172.21.15.98:41154) 2024-05-09T08:55:35.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.053+0000 7f9571d9b640 1 -- 172.21.15.98:0/2205478655 learned_addr learned my addr 172.21.15.98:0/2205478655 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:35.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.053+0000 7f957bbac740 1 -- 172.21.15.98:0/2205478655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f955400fff0 con 0x563a1b18b510 2024-05-09T08:55:35.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.053+0000 7f957bbac740 1 -- 172.21.15.98:0/2205478655 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9554011b30 con 0x563a1b180520 2024-05-09T08:55:35.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.053+0000 7f957bbac740 1 -- 172.21.15.98:0/2205478655 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f95540106c0 con 0x563a1b180e40 2024-05-09T08:55:35.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.054+0000 7f957259c640 1 --2- 172.21.15.98:0/2205478655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b180520 0x563a1b18ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.054+0000 7f9572d9d640 1 --2- 172.21.15.98:0/2205478655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b18b510 0x563a1b17c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.054+0000 7f9572d9d640 1 -- 172.21.15.98:0/2205478655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 msgr2=0x563a1b182c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:35.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.054+0000 7f9572d9d640 1 --2- 172.21.15.98:0/2205478655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b182c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.054+0000 7f9572d9d640 1 -- 172.21.15.98:0/2205478655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b180520 msgr2=0x563a1b18ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:35.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.054+0000 7f9572d9d640 1 --2- 172.21.15.98:0/2205478655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b180520 0x563a1b18ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.054+0000 7f9572d9d640 1 -- 172.21.15.98:0/2205478655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a1b19e360 con 0x563a1b18b510 2024-05-09T08:55:35.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.055+0000 7f9572d9d640 1 --2- 172.21.15.98:0/2205478655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b18b510 0x563a1b17c140 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f956c0c4ea0 tx=0x7f956c0bc1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb2a8a5adf47fb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:35.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.058+0000 7f955bfff640 1 -- 172.21.15.98:0/2205478655 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c0bc760 con 0x563a1b18b510 2024-05-09T08:55:35.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.058+0000 7f955bfff640 1 -- 172.21.15.98:0/2205478655 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f956c0bcc00 con 0x563a1b18b510 2024-05-09T08:55:35.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.058+0000 7f955bfff640 1 -- 172.21.15.98:0/2205478655 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c0c51a0 con 0x563a1b18b510 2024-05-09T08:55:35.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.060+0000 7f957bbac740 1 -- 172.21.15.98:0/2205478655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b18b510 msgr2=0x563a1b17c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.060+0000 7f957bbac740 1 --2- 172.21.15.98:0/2205478655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b18b510 0x563a1b17c140 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f956c0c4ea0 tx=0x7f956c0bc1f0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.060+0000 7f957bbac740 1 -- 172.21.15.98:0/2205478655 shutdown_connections 2024-05-09T08:55:35.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.060+0000 7f957bbac740 1 --2- 172.21.15.98:0/2205478655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b18b510 0x563a1b17c140 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.060+0000 7f957bbac740 1 --2- 172.21.15.98:0/2205478655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b182c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.060+0000 7f957bbac740 1 --2- 172.21.15.98:0/2205478655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b180520 0x563a1b18ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.060+0000 7f957bbac740 1 -- 172.21.15.98:0/2205478655 >> 172.21.15.98:0/2205478655 conn(0x563a1b17c840 msgr2=0x563a1b188c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:35.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.065+0000 7f957bbac740 1 -- 172.21.15.98:0/2205478655 shutdown_connections 2024-05-09T08:55:35.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.066+0000 7f957bbac740 1 -- 172.21.15.98:0/2205478655 wait complete. 2024-05-09T08:55:35.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.066+0000 7f957bbac740 1 -- start start 2024-05-09T08:55:35.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.066+0000 7f957bbac740 1 Processor -- start 2024-05-09T08:55:35.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.075+0000 7f957bbac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b180520 0x563a1b193760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.075+0000 7f957bbac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b193ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.075+0000 7f957bbac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b18b510 0x563a1b1908e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.075+0000 7f957bbac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9554011460 con 0x563a1b180520 2024-05-09T08:55:35.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.075+0000 7f957bbac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9554010d90 con 0x563a1b18b510 2024-05-09T08:55:35.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.075+0000 7f957bbac740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f955400dfd0 con 0x563a1b180e40 2024-05-09T08:55:35.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.075+0000 7f957bbac740 10 client.0 _ll_register_callbacks cb 0x7f956c02f810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:35.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.078+0000 7f9572d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b18b510 0x563a1b1908e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.078+0000 7f9572d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b18b510 0x563a1b1908e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:47410/0 (socket says 172.21.15.98:47410) 2024-05-09T08:55:35.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.078+0000 7f9572d9d640 1 -- 172.21.15.98:0/2145879975 learned_addr learned my addr 172.21.15.98:0/2145879975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:35.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.078+0000 7f9571d9b640 1 --2- 172.21.15.98:0/2145879975 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b193ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.078+0000 7f9572d9d640 1 -- 172.21.15.98:0/2145879975 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 msgr2=0x563a1b193ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.078+0000 7f9572d9d640 1 --2- 172.21.15.98:0/2145879975 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b193ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.078+0000 7f9572d9d640 1 -- 172.21.15.98:0/2145879975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b180520 msgr2=0x563a1b193760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:55:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.078+0000 7f9572d9d640 1 --2- 172.21.15.98:0/2145879975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b180520 0x563a1b193760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.078+0000 7f9572d9d640 1 -- 172.21.15.98:0/2145879975 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a1b190fe0 con 0x563a1b18b510 2024-05-09T08:55:35.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.082+0000 7f9572d9d640 1 --2- 172.21.15.98:0/2145879975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b18b510 0x563a1b1908e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f956c0bc6f0 tx=0x7f956c0f0390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc49aae11fbf9803 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:35.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.082+0000 7f9559ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:55:35.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.083+0000 7f9559ffb640 1 -- 172.21.15.98:0/2145879975 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c0bc760 con 0x563a1b18b510 2024-05-09T08:55:35.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.083+0000 7f957bbac740 1 -- 172.21.15.98:0/2145879975 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f956c0bcc00 con 0x563a1b18b510 2024-05-09T08:55:35.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.083+0000 7f9559ffb640 1 -- 172.21.15.98:0/2145879975 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f956c004b40 con 0x563a1b18b510 2024-05-09T08:55:35.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.084+0000 7f9559ffb640 1 -- 172.21.15.98:0/2145879975 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f956c0c53d0 con 0x563a1b18b510 2024-05-09T08:55:35.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.084+0000 7f957bbac740 10 client.26273 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:35.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.084+0000 7f957bbac740 1 -- 172.21.15.98:0/2145879975 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563a1b190fe0 con 0x563a1b18b510 2024-05-09T08:55:35.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.088+0000 7f9559ffb640 1 -- 172.21.15.98:0/2145879975 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f956c0bcc00 con 0x563a1b18b510 2024-05-09T08:55:35.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.088+0000 7f9559ffb640 1 -- 172.21.15.98:0/2145879975 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f956c02f070 con 0x563a1b18b510 2024-05-09T08:55:35.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.089+0000 7f957bbac740 10 client.26273 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:35.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.089+0000 7f957bbac740 10 client.26273 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:55:35.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.089+0000 7f957bbac740 20 client.26273 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:35.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.089+0000 7f957bbac740 1 --2- 172.21.15.98:0/2145879975 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563a1b213cd0 0x563a1b2160c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.089+0000 7f957bbac740 4 client.26273 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:55:35.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.089+0000 7f957bbac740 1 -- 172.21.15.98:0/2145879975 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a1b0ed6c0 con 0x563a1b213cd0 2024-05-09T08:55:35.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.093+0000 7f957259c640 1 --2- 172.21.15.98:0/2145879975 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563a1b213cd0 0x563a1b2160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.098+0000 7f957259c640 1 --2- 172.21.15.98:0/2145879975 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563a1b213cd0 0x563a1b2160c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8adf3a2f5f4f7a39 server_cookie=16055d711c5e2de7 in_seq=0 out_seq=0 2024-05-09T08:55:35.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.099+0000 7f9559ffb640 10 client.26273 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:55:35.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.099+0000 7f9559ffb640 1 -- 172.21.15.98:0/2145879975 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a1b0ed6c0 con 0x563a1b213cd0 2024-05-09T08:55:35.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.099+0000 7f9559ffb640 10 client.26273 handle_command_reply: tid=0 2024-05-09T08:55:35.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.099+0000 7f957bbac740 1 client.26273 shutdown 2024-05-09T08:55:35.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.100+0000 7f957bbac740 1 -- 172.21.15.98:0/2145879975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b18b510 msgr2=0x563a1b1908e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.100+0000 7f957bbac740 1 --2- 172.21.15.98:0/2145879975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b18b510 0x563a1b1908e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f956c0bc6f0 tx=0x7f956c0f0390 comp rx=0 tx=0).stop 2024-05-09T08:55:35.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.100+0000 7f957bbac740 1 -- 172.21.15.98:0/2145879975 shutdown_connections 2024-05-09T08:55:35.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.100+0000 7f957bbac740 1 --2- 172.21.15.98:0/2145879975 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563a1b213cd0 0x563a1b2160c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.100+0000 7f957bbac740 1 --2- 172.21.15.98:0/2145879975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a1b18b510 0x563a1b1908e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.100+0000 7f957bbac740 1 --2- 172.21.15.98:0/2145879975 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a1b180e40 0x563a1b193ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.100+0000 7f957bbac740 1 --2- 172.21.15.98:0/2145879975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a1b180520 0x563a1b193760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.100+0000 7f957bbac740 1 -- 172.21.15.98:0/2145879975 >> 172.21.15.98:0/2145879975 conn(0x563a1b17c840 msgr2=0x563a1b181c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:35.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.100+0000 7f9559ffb640 0 client.26273 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:55:35.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.101+0000 7f957bbac740 1 -- 172.21.15.98:0/2145879975 shutdown_connections 2024-05-09T08:55:35.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.101+0000 7f957bbac740 1 -- 172.21.15.98:0/2145879975 wait complete. 2024-05-09T08:55:35.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.101+0000 7f957bbac740 20 client.26273 trim_cache size 0 max 16384 2024-05-09T08:55:35.101 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:55:35.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.103+0000 7f957359e640 1 -- 172.21.15.98:0/3357133545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f955c0a2f00 msgr2=0x7f955c0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.103+0000 7f957359e640 1 --2- 172.21.15.98:0/3357133545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f955c0a2f00 0x7f955c0a53c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x563a1b181ae0 tx=0x7f9564009880 comp rx=0 tx=0).stop 2024-05-09T08:55:35.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.103+0000 7f957359e640 1 -- 172.21.15.98:0/3357133545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f957410fed0 msgr2=0x7f95741ab620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.103+0000 7f957359e640 1 --2- 172.21.15.98:0/3357133545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f957410fed0 0x7f95741ab620 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f956c02af30 tx=0x7f956c02f700 comp rx=0 tx=0).stop 2024-05-09T08:55:35.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.104+0000 7f957359e640 1 -- 172.21.15.98:0/3357133545 shutdown_connections 2024-05-09T08:55:35.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.104+0000 7f957359e640 1 --2- 172.21.15.98:0/3357133545 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f955c0a2f00 0x7f955c0a53c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.104+0000 7f957359e640 1 --2- 172.21.15.98:0/3357133545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f957410fed0 0x7f95741ab620 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.104+0000 7f957359e640 1 --2- 172.21.15.98:0/3357133545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95741092b0 0x7f95741a72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.104+0000 7f957359e640 1 --2- 172.21.15.98:0/3357133545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9574108830 0x7f95741a6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.104+0000 7f957359e640 1 -- 172.21.15.98:0/3357133545 >> 172.21.15.98:0/3357133545 conn(0x7f957406c7b0 msgr2=0x7f9574070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:35.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.104+0000 7f957359e640 1 -- 172.21.15.98:0/3357133545 shutdown_connections 2024-05-09T08:55:35.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.104+0000 7f957359e640 1 -- 172.21.15.98:0/3357133545 wait complete. 2024-05-09T08:55:35.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:55:35.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.409+0000 7fd8c5a21640 1 -- 172.21.15.98:0/3605562353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8c0108830 msgr2=0x7fd8c0108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.409+0000 7fd8c5a21640 1 --2- 172.21.15.98:0/3605562353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8c0108830 0x7fd8c0108c90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009be0 tx=0x7fd8b802a770 comp rx=0 tx=0).stop 2024-05-09T08:55:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.412+0000 7fd8c5a21640 1 -- 172.21.15.98:0/3605562353 shutdown_connections 2024-05-09T08:55:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.412+0000 7fd8c5a21640 1 --2- 172.21.15.98:0/3605562353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c01092b0 0x7fd8c011d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.412+0000 7fd8c5a21640 1 --2- 172.21.15.98:0/3605562353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8c0108830 0x7fd8c0108c90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.412+0000 7fd8c5a21640 1 --2- 172.21.15.98:0/3605562353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c010fed0 0x7fd8c01102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.413+0000 7fd8c5a21640 1 -- 172.21.15.98:0/3605562353 >> 172.21.15.98:0/3605562353 conn(0x7fd8c006c7b0 msgr2=0x7fd8c006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.414+0000 7fd8c5a21640 1 -- 172.21.15.98:0/3605562353 shutdown_connections 2024-05-09T08:55:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.414+0000 7fd8c5a21640 1 -- 172.21.15.98:0/3605562353 wait complete. 2024-05-09T08:55:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.415+0000 7fd8c5a21640 1 Processor -- start 2024-05-09T08:55:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.415+0000 7fd8c5a21640 1 -- start start 2024-05-09T08:55:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.415+0000 7fd8c5a21640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c0108830 0x7fd8c011cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.415+0000 7fd8c5a21640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c01092b0 0x7fd8c0118de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.415+0000 7fd8c5a21640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8c010fed0 0x7fd8c0119320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.415+0000 7fd8c5a21640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0113190 con 0x7fd8c01092b0 2024-05-09T08:55:35.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.415+0000 7fd8c5a21640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0113010 con 0x7fd8c010fed0 2024-05-09T08:55:35.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.415+0000 7fd8c5a21640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0113310 con 0x7fd8c0108830 2024-05-09T08:55:35.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.418+0000 7fd8c5220640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8c010fed0 0x7fd8c0119320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.418+0000 7fd8c5220640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8c010fed0 0x7fd8c0119320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:47434/0 (socket says 172.21.15.98:47434) 2024-05-09T08:55:35.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.418+0000 7fd8c5220640 1 -- 172.21.15.98:0/1571794051 learned_addr learned my addr 172.21.15.98:0/1571794051 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:35.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.419+0000 7fd8c4a1f640 1 --2- 172.21.15.98:0/1571794051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c0108830 0x7fd8c011cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.420+0000 7fd8bffff640 1 --2- 172.21.15.98:0/1571794051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c01092b0 0x7fd8c0118de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.421+0000 7fd8c5220640 1 -- 172.21.15.98:0/1571794051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c0108830 msgr2=0x7fd8c011cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.421+0000 7fd8c5220640 1 --2- 172.21.15.98:0/1571794051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c0108830 0x7fd8c011cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.421+0000 7fd8c5220640 1 -- 172.21.15.98:0/1571794051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c01092b0 msgr2=0x7fd8c0118de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.421+0000 7fd8c5220640 1 --2- 172.21.15.98:0/1571794051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c01092b0 0x7fd8c0118de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.421+0000 7fd8c5220640 1 -- 172.21.15.98:0/1571794051 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c0119a20 con 0x7fd8c010fed0 2024-05-09T08:55:35.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.421+0000 7fd8c5220640 1 --2- 172.21.15.98:0/1571794051 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8c010fed0 0x7fd8c0119320 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c970 tx=0x7fd8b400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:35.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.423+0000 7fd8bdffb640 1 -- 172.21.15.98:0/1571794051 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4007c70 con 0x7fd8c010fed0 2024-05-09T08:55:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.423+0000 7fd8bdffb640 1 -- 172.21.15.98:0/1571794051 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8b4007e10 con 0x7fd8c010fed0 2024-05-09T08:55:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.423+0000 7fd8c5a21640 1 -- 172.21.15.98:0/1571794051 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c01ccca0 con 0x7fd8c010fed0 2024-05-09T08:55:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.423+0000 7fd8bdffb640 1 -- 172.21.15.98:0/1571794051 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b400f6b0 con 0x7fd8c010fed0 2024-05-09T08:55:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.424+0000 7fd8c5a21640 1 -- 172.21.15.98:0/1571794051 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c01cd0b0 con 0x7fd8c010fed0 2024-05-09T08:55:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.428+0000 7fd8c998a740 1 Processor -- start 2024-05-09T08:55:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.428+0000 7fd8c998a740 1 -- start start 2024-05-09T08:55:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.428+0000 7fd8c998a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ce2520 0x560b77ce2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.428+0000 7fd8c998a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2e40 0x560b77cecfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.428+0000 7fd8c998a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ced510 0x560b77cef900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.428+0000 7fd8c998a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560b77c47c40 con 0x560b77ce2e40 2024-05-09T08:55:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.428+0000 7fd8c998a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560b77bef580 con 0x560b77ced510 2024-05-09T08:55:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.428+0000 7fd8c998a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x560b77c43c00 con 0x560b77ce2520 2024-05-09T08:55:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.429+0000 7fd8bffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ce2520 0x560b77ce2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.429+0000 7fd8bffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ce2520 0x560b77ce2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:41192/0 (socket says 172.21.15.98:41192) 2024-05-09T08:55:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.429+0000 7fd8bffff640 1 -- 172.21.15.98:0/1656046110 learned_addr learned my addr 172.21.15.98:0/1656046110 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.430+0000 7fd8bffff640 1 --2- 172.21.15.98:0/1656046110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ced510 0x560b77cef900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.430+0000 7fd8bffff640 1 -- 172.21.15.98:0/1656046110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ced510 msgr2=0x560b77cef900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.430+0000 7fd8bffff640 1 --2- 172.21.15.98:0/1656046110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ced510 0x560b77cef900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.430+0000 7fd8bffff640 1 -- 172.21.15.98:0/1656046110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2e40 msgr2=0x560b77cecfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.430+0000 7fd8bffff640 1 --2- 172.21.15.98:0/1656046110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2e40 0x560b77cecfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.430+0000 7fd8bffff640 1 -- 172.21.15.98:0/1656046110 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b7795b3f0 con 0x560b77ce2520 2024-05-09T08:55:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.433+0000 7fd8bffff640 1 --2- 172.21.15.98:0/1656046110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ce2520 0x560b77ce2900 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8006010 tx=0x7fd8b8009300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acd9fb31828faf4d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.433+0000 7fd8bdffb640 1 -- 172.21.15.98:0/1571794051 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd8b40046c0 con 0x7fd8c010fed0 2024-05-09T08:55:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.442+0000 7fd8bdffb640 1 --2- 172.21.15.98:0/1571794051 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8a80a2ed0 0x7fd8a80a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.442+0000 7fd8bdffb640 1 -- 172.21.15.98:0/1571794051 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fd8b40cf910 con 0x7fd8c010fed0 2024-05-09T08:55:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.445+0000 7fd8a6ffd640 1 -- 172.21.15.98:0/1656046110 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8048070 con 0x560b77ce2520 2024-05-09T08:55:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.445+0000 7fd8a6ffd640 1 -- 172.21.15.98:0/1656046110 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8b8003970 con 0x560b77ce2520 2024-05-09T08:55:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.445+0000 7fd8a6ffd640 1 -- 172.21.15.98:0/1656046110 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8036030 con 0x560b77ce2520 2024-05-09T08:55:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.446+0000 7fd8c998a740 1 -- 172.21.15.98:0/1656046110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ce2520 msgr2=0x560b77ce2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.446+0000 7fd8c998a740 1 --2- 172.21.15.98:0/1656046110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ce2520 0x560b77ce2900 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8006010 tx=0x7fd8b8009300 comp rx=0 tx=0).stop 2024-05-09T08:55:35.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.446+0000 7fd8c998a740 1 -- 172.21.15.98:0/1656046110 shutdown_connections 2024-05-09T08:55:35.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.446+0000 7fd8c998a740 1 --2- 172.21.15.98:0/1656046110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ced510 0x560b77cef900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.446+0000 7fd8c998a740 1 --2- 172.21.15.98:0/1656046110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2e40 0x560b77cecfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.446+0000 7fd8c998a740 1 --2- 172.21.15.98:0/1656046110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ce2520 0x560b77ce2900 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.446+0000 7fd8c998a740 1 -- 172.21.15.98:0/1656046110 >> 172.21.15.98:0/1656046110 conn(0x560b77cde840 msgr2=0x560b77cdec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:35.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.447+0000 7fd8c4a1f640 1 --2- 172.21.15.98:0/1571794051 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8a80a2ed0 0x7fd8a80a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.449+0000 7fd8c998a740 1 -- 172.21.15.98:0/1656046110 shutdown_connections 2024-05-09T08:55:35.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.451+0000 7fd8c998a740 1 -- 172.21.15.98:0/1656046110 wait complete. 2024-05-09T08:55:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.451+0000 7fd8c998a740 1 -- start start 2024-05-09T08:55:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.451+0000 7fd8c998a740 1 Processor -- start 2024-05-09T08:55:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.452+0000 7fd8c998a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2520 0x560b77cea4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.452+0000 7fd8c998a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2e40 0x560b77ceaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.452+0000 7fd8c998a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ced510 0x560b77ceaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.452+0000 7fd8c998a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560b77cf02a0 con 0x560b77ce2e40 2024-05-09T08:55:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.452+0000 7fd8c998a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560b77c02c40 con 0x560b77ce2520 2024-05-09T08:55:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.452+0000 7fd8c998a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x560b77cf0420 con 0x560b77ced510 2024-05-09T08:55:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.452+0000 7fd8c998a740 10 client.0 _ll_register_callbacks cb 0x560b7795abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.455+0000 7fd8c5220640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ced510 0x560b77ceaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.455+0000 7fd8c5220640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ced510 0x560b77ceaf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:41194/0 (socket says 172.21.15.98:41194) 2024-05-09T08:55:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.455+0000 7fd8c5220640 1 -- 172.21.15.98:0/2538973364 learned_addr learned my addr 172.21.15.98:0/2538973364 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.456+0000 7fd8bffff640 1 --2- 172.21.15.98:0/2538973364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2e40 0x560b77ceaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.457+0000 7fd8c5220640 1 -- 172.21.15.98:0/2538973364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2520 msgr2=0x560b77cea4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:55:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.457+0000 7fd8c5220640 1 --2- 172.21.15.98:0/2538973364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2520 0x560b77cea4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.457+0000 7fd8c5220640 1 -- 172.21.15.98:0/2538973364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2e40 msgr2=0x560b77ceaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.457+0000 7fd8c5220640 1 --2- 172.21.15.98:0/2538973364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2e40 0x560b77ceaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.457+0000 7fd8c5220640 1 -- 172.21.15.98:0/2538973364 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b77a257d0 con 0x560b77ced510 2024-05-09T08:55:35.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.458+0000 7fd8c5220640 1 --2- 172.21.15.98:0/2538973364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ced510 0x560b77ceaf60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400afd0 tx=0x7fd8b409c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f14b5b254f7a11a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:35.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.459+0000 7fd89ffff640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:55:35.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.459+0000 7fd89ffff640 1 -- 172.21.15.98:0/2538973364 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40cd0c0 con 0x560b77ced510 2024-05-09T08:55:35.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.460+0000 7fd8c998a740 1 -- 172.21.15.98:0/2538973364 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8b8003970 con 0x560b77ced510 2024-05-09T08:55:35.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.460+0000 7fd89ffff640 1 -- 172.21.15.98:0/2538973364 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fd8b40a4270 con 0x560b77ced510 2024-05-09T08:55:35.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.461+0000 7fd89ffff640 1 -- 172.21.15.98:0/2538973364 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd8b40a4700 con 0x560b77ced510 2024-05-09T08:55:35.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.461+0000 7fd8c998a740 10 client.26242 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:35.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.461+0000 7fd8c998a740 1 -- 172.21.15.98:0/2538973364 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560b77cfc560 con 0x560b77ced510 2024-05-09T08:55:35.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.461+0000 7fd8c4a1f640 1 --2- 172.21.15.98:0/1571794051 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8a80a2ed0 0x7fd8a80a5390 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0004700 tx=0x7fd8b0009530 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:35.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.461+0000 7fd89ffff640 1 -- 172.21.15.98:0/2538973364 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8b8003970 con 0x560b77ced510 2024-05-09T08:55:35.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.462+0000 7fd89ffff640 1 -- 172.21.15.98:0/2538973364 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd8b409afe0 con 0x560b77ced510 2024-05-09T08:55:35.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.462+0000 7fd8c998a740 10 client.26242 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:35.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.462+0000 7fd8c998a740 10 client.26242 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:55:35.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.463+0000 7fd8c998a740 20 client.26242 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:35.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.463+0000 7fd8c998a740 1 --2- 172.21.15.98:0/2538973364 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560b77cfda20 0x560b77cffe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.463+0000 7fd8c998a740 4 client.26242 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:35.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.463+0000 7fd8c998a740 1 -- 172.21.15.98:0/2538973364 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b77c55370 con 0x560b77cfda20 2024-05-09T08:55:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.467+0000 7fd8c4a1f640 1 --2- 172.21.15.98:0/2538973364 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560b77cfda20 0x560b77cffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.487+0000 7fd8c4a1f640 1 --2- 172.21.15.98:0/2538973364 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560b77cfda20 0x560b77cffe10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e28e688e5cae9d92 server_cookie=f495eef86ca40a5a in_seq=0 out_seq=0 2024-05-09T08:55:35.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.488+0000 7fd89ffff640 10 client.26242 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:55:35.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.494+0000 7fd89ffff640 1 -- 172.21.15.98:0/2538973364 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560b77c55370 con 0x560b77cfda20 2024-05-09T08:55:35.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.494+0000 7fd89ffff640 10 client.26242 handle_command_reply: tid=0 2024-05-09T08:55:35.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.494+0000 7fd8c998a740 1 client.26242 shutdown 2024-05-09T08:55:35.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.499+0000 7fd8c998a740 1 -- 172.21.15.98:0/2538973364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ced510 msgr2=0x560b77ceaf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.499+0000 7fd8c998a740 1 --2- 172.21.15.98:0/2538973364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ced510 0x560b77ceaf60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400afd0 tx=0x7fd8b409c2d0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.499+0000 7fd8c998a740 1 -- 172.21.15.98:0/2538973364 shutdown_connections 2024-05-09T08:55:35.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.499+0000 7fd8c998a740 1 --2- 172.21.15.98:0/2538973364 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560b77cfda20 0x560b77cffe10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.499+0000 7fd8c998a740 1 --2- 172.21.15.98:0/2538973364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ced510 0x560b77ceaf60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.499+0000 7fd8c998a740 1 --2- 172.21.15.98:0/2538973364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2e40 0x560b77ceaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.499+0000 7fd8c998a740 1 --2- 172.21.15.98:0/2538973364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2520 0x560b77cea4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.499+0000 7fd8c998a740 1 -- 172.21.15.98:0/2538973364 >> 172.21.15.98:0/2538973364 conn(0x560b77cde840 msgr2=0x560b77ce4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:35.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.499+0000 7fd89ffff640 0 client.26242 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:55:35.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.506+0000 7fd8c998a740 1 -- 172.21.15.98:0/2538973364 shutdown_connections 2024-05-09T08:55:35.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.511+0000 7fd8c998a740 1 -- 172.21.15.98:0/2538973364 wait complete. 2024-05-09T08:55:35.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.511+0000 7fd8c998a740 20 client.26242 trim_cache size 0 max 16384 2024-05-09T08:55:35.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.546+0000 7fd8c998a740 1 Processor -- start 2024-05-09T08:55:35.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.546+0000 7fd8c998a740 1 -- start start 2024-05-09T08:55:35.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.547+0000 7fd8c998a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2520 0x560b77cf9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.547+0000 7fd8c998a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2e40 0x560b77d05670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.547+0000 7fd8c998a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ced510 0x560b77d05bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.548+0000 7fd8c998a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c007d760 con 0x560b77ce2520 2024-05-09T08:55:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.548+0000 7fd8c998a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c007f2a0 con 0x560b77ce2e40 2024-05-09T08:55:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.548+0000 7fd8bffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2e40 0x560b77d05670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.548+0000 7fd8bffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2e40 0x560b77d05670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:47468/0 (socket says 172.21.15.98:47468) 2024-05-09T08:55:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.548+0000 7fd8bffff640 1 -- 172.21.15.98:0/3184742755 learned_addr learned my addr 172.21.15.98:0/3184742755 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.548+0000 7fd8c998a740 1 -- 172.21.15.98:0/3184742755 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c007de30 con 0x560b77ced510 2024-05-09T08:55:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.548+0000 7fd8c4a1f640 1 --2- 172.21.15.98:0/3184742755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2520 0x560b77cf9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.548+0000 7fd8c5220640 1 --2- 172.21.15.98:0/3184742755 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ced510 0x560b77d05bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.549+0000 7fd8c4a1f640 1 -- 172.21.15.98:0/3184742755 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ced510 msgr2=0x560b77d05bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.549+0000 7fd8c4a1f640 1 --2- 172.21.15.98:0/3184742755 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ced510 0x560b77d05bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.549+0000 7fd8c4a1f640 1 -- 172.21.15.98:0/3184742755 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2e40 msgr2=0x560b77d05670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.549+0000 7fd8c4a1f640 1 --2- 172.21.15.98:0/3184742755 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2e40 0x560b77d05670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.549+0000 7fd8c4a1f640 1 -- 172.21.15.98:0/3184742755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b77ce4c80 con 0x560b77ce2520 2024-05-09T08:55:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.550+0000 7fd8c4a1f640 1 --2- 172.21.15.98:0/3184742755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2520 0x560b77cf9190 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x560b77cf0530 tx=0x7fd8b0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f9705b3e9c568 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.550+0000 7fd8a67fc640 1 -- 172.21.15.98:0/3184742755 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b00263d0 con 0x560b77ce2520 2024-05-09T08:55:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.550+0000 7fd8a67fc640 1 -- 172.21.15.98:0/3184742755 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8b0012510 con 0x560b77ce2520 2024-05-09T08:55:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.550+0000 7fd8a67fc640 1 -- 172.21.15.98:0/3184742755 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b001f420 con 0x560b77ce2520 2024-05-09T08:55:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.552+0000 7fd8c998a740 1 -- 172.21.15.98:0/3184742755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2520 msgr2=0x560b77cf9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.552+0000 7fd8c998a740 1 --2- 172.21.15.98:0/3184742755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2520 0x560b77cf9190 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x560b77cf0530 tx=0x7fd8b0016000 comp rx=0 tx=0).stop 2024-05-09T08:55:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.552+0000 7fd8c998a740 1 -- 172.21.15.98:0/3184742755 shutdown_connections 2024-05-09T08:55:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.552+0000 7fd8c998a740 1 --2- 172.21.15.98:0/3184742755 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ced510 0x560b77d05bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.552+0000 7fd8c998a740 1 --2- 172.21.15.98:0/3184742755 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2e40 0x560b77d05670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.552+0000 7fd8c998a740 1 --2- 172.21.15.98:0/3184742755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ce2520 0x560b77cf9190 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.553+0000 7fd8c998a740 1 -- 172.21.15.98:0/3184742755 >> 172.21.15.98:0/3184742755 conn(0x560b77cde840 msgr2=0x560b77cdec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.557+0000 7fd8c998a740 1 -- 172.21.15.98:0/3184742755 shutdown_connections 2024-05-09T08:55:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.557+0000 7fd8c998a740 1 -- 172.21.15.98:0/3184742755 wait complete. 2024-05-09T08:55:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.558+0000 7fd8c998a740 1 -- start start 2024-05-09T08:55:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.559+0000 7fd8c998a740 1 Processor -- start 2024-05-09T08:55:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.560+0000 7fd8c998a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ce2520 0x560b77cf56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.560+0000 7fd8c998a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2e40 0x560b77cf5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.560+0000 7fd8c4a1f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ce2520 0x560b77cf56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.560+0000 7fd8c4a1f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ce2520 0x560b77cf56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:46714/0 (socket says 172.21.15.98:46714) 2024-05-09T08:55:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.560+0000 7fd8c4a1f640 1 -- 172.21.15.98:0/2648398818 learned_addr learned my addr 172.21.15.98:0/2648398818 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.560+0000 7fd8bffff640 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2e40 0x560b77cf5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.560+0000 7fd8c998a740 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ced510 0x560b77cf2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.561+0000 7fd8c998a740 1 -- 172.21.15.98:0/2648398818 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c007ebd0 con 0x560b77ced510 2024-05-09T08:55:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.561+0000 7fd8c998a740 1 -- 172.21.15.98:0/2648398818 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c007e500 con 0x560b77ce2e40 2024-05-09T08:55:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.561+0000 7fd8c998a740 1 -- 172.21.15.98:0/2648398818 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c007b7f0 con 0x560b77ce2520 2024-05-09T08:55:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.561+0000 7fd8c998a740 10 client.0 _ll_register_callbacks cb 0x7fd8b4004a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.561+0000 7fd8c5220640 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ced510 0x560b77cf2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.562+0000 7fd8c5220640 1 -- 172.21.15.98:0/2648398818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ce2520 msgr2=0x560b77cf56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.562+0000 7fd8c5220640 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ce2520 0x560b77cf56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.562+0000 7fd8c5220640 1 -- 172.21.15.98:0/2648398818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2e40 msgr2=0x560b77cf5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.562+0000 7fd8c5220640 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2e40 0x560b77cf5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.562+0000 7fd8c5220640 1 -- 172.21.15.98:0/2648398818 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b77cf2ea0 con 0x560b77ced510 2024-05-09T08:55:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.562+0000 7fd8c5220640 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ced510 0x560b77cf2670 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c940 tx=0x7fd8b40cd0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9413889dcc90bb1b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.563+0000 7fd89ffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:55:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.563+0000 7fd89ffff640 1 -- 172.21.15.98:0/2648398818 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b409b860 con 0x560b77ced510 2024-05-09T08:55:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.563+0000 7fd8c4a1f640 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ce2520 0x560b77cf56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:55:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.563+0000 7fd8bffff640 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2e40 0x560b77cf5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:55:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.563+0000 7fd8c998a740 1 -- 172.21.15.98:0/2648398818 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8b0012510 con 0x560b77ced510 2024-05-09T08:55:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.563+0000 7fd89ffff640 1 -- 172.21.15.98:0/2648398818 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fd8b409aee0 con 0x560b77ced510 2024-05-09T08:55:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.564+0000 7fd89ffff640 1 -- 172.21.15.98:0/2648398818 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd8b409b0e0 con 0x560b77ced510 2024-05-09T08:55:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.568+0000 7fd8c998a740 10 client.16992 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.568+0000 7fd8c998a740 1 -- 172.21.15.98:0/2648398818 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560b77d64a10 con 0x560b77ced510 2024-05-09T08:55:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.568+0000 7fd89ffff640 1 -- 172.21.15.98:0/2648398818 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8b0012510 con 0x560b77ced510 2024-05-09T08:55:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.568+0000 7fd89ffff640 1 -- 172.21.15.98:0/2648398818 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd8b40ac060 con 0x560b77ced510 2024-05-09T08:55:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.572+0000 7fd8c998a740 10 client.16992 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.572+0000 7fd8c998a740 10 client.16992 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:55:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.572+0000 7fd8c998a740 20 client.16992 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.572+0000 7fd8c998a740 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560b77d64f30 0x560b77d67350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.572+0000 7fd8c998a740 4 client.16992 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:55:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.572+0000 7fd8c998a740 1 -- 172.21.15.98:0/2648398818 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560b77cde690 con 0x560b77d64f30 2024-05-09T08:55:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.573+0000 7fd8c4a1f640 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560b77d64f30 0x560b77d67350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.579+0000 7fd8c4a1f640 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560b77d64f30 0x560b77d67350 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=21b52e2c75008297 server_cookie=200ba930baf35db0 in_seq=0 out_seq=0 2024-05-09T08:55:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.580+0000 7fd89ffff640 10 client.16992 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:55:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.581+0000 7fd89ffff640 1 -- 172.21.15.98:0/2648398818 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560b77cde690 con 0x560b77d64f30 2024-05-09T08:55:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.581+0000 7fd89ffff640 10 client.16992 handle_command_reply: tid=0 2024-05-09T08:55:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.582+0000 7fd8c998a740 1 client.16992 shutdown 2024-05-09T08:55:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.587+0000 7fd8c998a740 1 -- 172.21.15.98:0/2648398818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ced510 msgr2=0x560b77cf2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.587+0000 7fd8c998a740 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ced510 0x560b77cf2670 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c940 tx=0x7fd8b40cd0f0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.587+0000 7fd8c998a740 1 -- 172.21.15.98:0/2648398818 shutdown_connections 2024-05-09T08:55:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.587+0000 7fd8c998a740 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560b77d64f30 0x560b77d67350 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.587+0000 7fd8c998a740 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560b77ced510 0x560b77cf2670 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.587+0000 7fd8c998a740 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560b77ce2e40 0x560b77cf5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.587+0000 7fd8c998a740 1 --2- 172.21.15.98:0/2648398818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560b77ce2520 0x560b77cf56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.587+0000 7fd8c998a740 1 -- 172.21.15.98:0/2648398818 >> 172.21.15.98:0/2648398818 conn(0x560b77cde840 msgr2=0x560b77cdd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:35.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.587+0000 7fd89ffff640 0 client.16992 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:55:35.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:35 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4247976413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:55:35.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.589+0000 7fd8c998a740 1 -- 172.21.15.98:0/2648398818 shutdown_connections 2024-05-09T08:55:35.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.589+0000 7fd8c998a740 1 -- 172.21.15.98:0/2648398818 wait complete. 2024-05-09T08:55:35.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.589+0000 7fd8c998a740 20 client.16992 trim_cache size 0 max 16384 2024-05-09T08:55:35.591 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:55:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.591+0000 7fd8c5a21640 1 -- 172.21.15.98:0/1571794051 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8a80a2ed0 msgr2=0x7fd8a80a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.591+0000 7fd8c5a21640 1 --2- 172.21.15.98:0/1571794051 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8a80a2ed0 0x7fd8a80a5390 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0004700 tx=0x7fd8b0009530 comp rx=0 tx=0).stop 2024-05-09T08:55:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.592+0000 7fd8c5a21640 1 -- 172.21.15.98:0/1571794051 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8c010fed0 msgr2=0x7fd8c0119320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.592+0000 7fd8c5a21640 1 --2- 172.21.15.98:0/1571794051 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8c010fed0 0x7fd8c0119320 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c970 tx=0x7fd8b400ce30 comp rx=0 tx=0).stop 2024-05-09T08:55:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.595+0000 7fd8c5a21640 1 -- 172.21.15.98:0/1571794051 shutdown_connections 2024-05-09T08:55:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.595+0000 7fd8c5a21640 1 --2- 172.21.15.98:0/1571794051 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8a80a2ed0 0x7fd8a80a5390 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.595+0000 7fd8c5a21640 1 --2- 172.21.15.98:0/1571794051 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8c010fed0 0x7fd8c0119320 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.595+0000 7fd8c5a21640 1 --2- 172.21.15.98:0/1571794051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c01092b0 0x7fd8c0118de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.596+0000 7fd8c5a21640 1 --2- 172.21.15.98:0/1571794051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c0108830 0x7fd8c011cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.596+0000 7fd8c5a21640 1 -- 172.21.15.98:0/1571794051 >> 172.21.15.98:0/1571794051 conn(0x7fd8c006c7b0 msgr2=0x7fd8c0070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.599+0000 7fd8c5a21640 1 -- 172.21.15.98:0/1571794051 shutdown_connections 2024-05-09T08:55:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.599+0000 7fd8c5a21640 1 -- 172.21.15.98:0/1571794051 wait complete. 2024-05-09T08:55:35.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:55:35.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:35 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/4247976413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:55:35.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:35 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/4247976413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:55:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.950+0000 7f893c790640 1 -- 172.21.15.98:0/2911600241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f893411b010 msgr2=0x7f893411d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.950+0000 7f893c790640 1 --2- 172.21.15.98:0/2911600241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f893411b010 0x7f893411d400 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8924009860 tx=0x7f892402a360 comp rx=0 tx=0).stop 2024-05-09T08:55:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.950+0000 7f893c790640 1 -- 172.21.15.98:0/2911600241 shutdown_connections 2024-05-09T08:55:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.950+0000 7f893c790640 1 --2- 172.21.15.98:0/2911600241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f893411b010 0x7f893411d400 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.950+0000 7f893c790640 1 --2- 172.21.15.98:0/2911600241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893410e0b0 0x7f893410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.950+0000 7f893c790640 1 --2- 172.21.15.98:0/2911600241 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8934110080 0x7f893410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.950+0000 7f893c790640 1 -- 172.21.15.98:0/2911600241 >> 172.21.15.98:0/2911600241 conn(0x7f893406c940 msgr2=0x7f893406cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.950+0000 7f893c790640 1 -- 172.21.15.98:0/2911600241 shutdown_connections 2024-05-09T08:55:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.950+0000 7f893c790640 1 -- 172.21.15.98:0/2911600241 wait complete. 2024-05-09T08:55:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.951+0000 7f893c790640 1 Processor -- start 2024-05-09T08:55:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.956+0000 7f893c790640 1 -- start start 2024-05-09T08:55:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.956+0000 7f893c790640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893410e0b0 0x7f893411a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.956+0000 7f893c790640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8934110080 0x7f89341167b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.956+0000 7f893c790640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f893411b010 0x7f8934116cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.956+0000 7f893c790640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89341131a0 con 0x7f893410e0b0 2024-05-09T08:55:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.956+0000 7f893c790640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8934113020 con 0x7f8934110080 2024-05-09T08:55:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.956+0000 7f893c790640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8934113320 con 0x7f893411b010 2024-05-09T08:55:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.959+0000 7f8939d2b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8934110080 0x7f89341167b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.959+0000 7f8939d2b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8934110080 0x7f89341167b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45554/0 (socket says 172.21.15.98:45554) 2024-05-09T08:55:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.959+0000 7f8939d2b640 1 -- 172.21.15.98:0/3490663553 learned_addr learned my addr 172.21.15.98:0/3490663553 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.959+0000 7f893a52c640 1 --2- 172.21.15.98:0/3490663553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893410e0b0 0x7f893411a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.960+0000 7f8939d2b640 1 -- 172.21.15.98:0/3490663553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f893411b010 msgr2=0x7f8934116cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:55:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.960+0000 7f8939d2b640 1 --2- 172.21.15.98:0/3490663553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f893411b010 0x7f8934116cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.961+0000 7f8939d2b640 1 -- 172.21.15.98:0/3490663553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893410e0b0 msgr2=0x7f893411a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.961+0000 7f8939d2b640 1 --2- 172.21.15.98:0/3490663553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893410e0b0 0x7f893411a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.961+0000 7f8939d2b640 1 -- 172.21.15.98:0/3490663553 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8934117420 con 0x7f8934110080 2024-05-09T08:55:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.961+0000 7f8939d2b640 1 --2- 172.21.15.98:0/3490663553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8934110080 0x7f89341167b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f892c01fbc0 tx=0x7f892c01d380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.962+0000 7f892b7fe640 1 -- 172.21.15.98:0/3490663553 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c02c070 con 0x7f8934110080 2024-05-09T08:55:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.962+0000 7f892b7fe640 1 -- 172.21.15.98:0/3490663553 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f892c01ddc0 con 0x7f8934110080 2024-05-09T08:55:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.962+0000 7f892b7fe640 1 -- 172.21.15.98:0/3490663553 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c026640 con 0x7f8934110080 2024-05-09T08:55:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.963+0000 7f893c790640 1 -- 172.21.15.98:0/3490663553 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f893407bab0 con 0x7f8934110080 2024-05-09T08:55:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.963+0000 7f893c790640 1 -- 172.21.15.98:0/3490663553 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f893407bfb0 con 0x7f8934110080 2024-05-09T08:55:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.967+0000 7f893f57c740 1 Processor -- start 2024-05-09T08:55:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.971+0000 7f892b7fe640 1 -- 172.21.15.98:0/3490663553 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f892c02f070 con 0x7f8934110080 2024-05-09T08:55:35.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.972+0000 7f892b7fe640 1 --2- 172.21.15.98:0/3490663553 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89200a2f00 0x7f89200a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.973+0000 7f892b7fe640 1 -- 172.21.15.98:0/3490663553 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f892c0e0190 con 0x7f8934110080 2024-05-09T08:55:35.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.973+0000 7f893f57c740 1 -- start start 2024-05-09T08:55:35.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.973+0000 7f893f57c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6e9330 0x55d86f6e9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.973+0000 7f893f57c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9c50 0x55d86f6f3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.973+0000 7f893f57c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6f4320 0x55d86f6f6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.973+0000 7f893f57c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d86f64ec40 con 0x55d86f6e9c50 2024-05-09T08:55:35.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.973+0000 7f893f57c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d86f5f6580 con 0x55d86f6f4320 2024-05-09T08:55:35.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.973+0000 7f893f57c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d86f64ac00 con 0x55d86f6e9330 2024-05-09T08:55:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.977+0000 7f893ad2d640 1 --2- 172.21.15.98:0/3490663553 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89200a2f00 0x7f89200a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.977+0000 7f8939d2b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9c50 0x55d86f6f3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.977+0000 7f8939d2b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9c50 0x55d86f6f3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37194/0 (socket says 172.21.15.98:37194) 2024-05-09T08:55:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.977+0000 7f8939d2b640 1 -- 172.21.15.98:0/1989318210 learned_addr learned my addr 172.21.15.98:0/1989318210 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.977+0000 7f893ad2d640 1 --2- 172.21.15.98:0/1989318210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6f4320 0x55d86f6f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.977+0000 7f893a52c640 1 --2- 172.21.15.98:0/1989318210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6e9330 0x55d86f6e9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.978+0000 7f8939d2b640 1 -- 172.21.15.98:0/1989318210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6e9330 msgr2=0x55d86f6e9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.978+0000 7f8939d2b640 1 --2- 172.21.15.98:0/1989318210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6e9330 0x55d86f6e9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.978+0000 7f8939d2b640 1 -- 172.21.15.98:0/1989318210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6f4320 msgr2=0x55d86f6f6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.978+0000 7f8939d2b640 1 --2- 172.21.15.98:0/1989318210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6f4320 0x55d86f6f6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.978+0000 7f8939d2b640 1 -- 172.21.15.98:0/1989318210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d86f3623f0 con 0x55d86f6e9c50 2024-05-09T08:55:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.978+0000 7f8939d2b640 1 --2- 172.21.15.98:0/1989318210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9c50 0x55d86f6f3de0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f892c020fd0 tx=0x7f892c0ac290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ece5b7e60bcf8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:35.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.980+0000 7f893ad2d640 1 --2- 172.21.15.98:0/3490663553 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89200a2f00 0x7f89200a53c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8924009780 tx=0x7f892402c040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:35.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.980+0000 7f8928ff9640 1 -- 172.21.15.98:0/1989318210 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c0c5070 con 0x55d86f6e9c50 2024-05-09T08:55:35.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.980+0000 7f8928ff9640 1 -- 172.21.15.98:0/1989318210 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f892c0b4f00 con 0x55d86f6e9c50 2024-05-09T08:55:35.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.982+0000 7f893f57c740 1 -- 172.21.15.98:0/1989318210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9c50 msgr2=0x55d86f6f3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:35.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.982+0000 7f893f57c740 1 --2- 172.21.15.98:0/1989318210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9c50 0x55d86f6f3de0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f892c020fd0 tx=0x7f892c0ac290 comp rx=0 tx=0).stop 2024-05-09T08:55:35.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.984+0000 7f893f57c740 1 -- 172.21.15.98:0/1989318210 shutdown_connections 2024-05-09T08:55:35.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.984+0000 7f893f57c740 1 --2- 172.21.15.98:0/1989318210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6f4320 0x55d86f6f6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.984+0000 7f893f57c740 1 --2- 172.21.15.98:0/1989318210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9c50 0x55d86f6f3de0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.984+0000 7f893f57c740 1 --2- 172.21.15.98:0/1989318210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6e9330 0x55d86f6e9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.984+0000 7f893f57c740 1 -- 172.21.15.98:0/1989318210 >> 172.21.15.98:0/1989318210 conn(0x55d86f6e5650 msgr2=0x55d86f6e5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:35.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.985+0000 7f893f57c740 1 -- 172.21.15.98:0/1989318210 shutdown_connections 2024-05-09T08:55:35.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.985+0000 7f893f57c740 1 -- 172.21.15.98:0/1989318210 wait complete. 2024-05-09T08:55:35.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.986+0000 7f893f57c740 1 -- start start 2024-05-09T08:55:35.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.986+0000 7f893f57c740 1 Processor -- start 2024-05-09T08:55:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.987+0000 7f893f57c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9330 0x55d86f6f1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.987+0000 7f893f57c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 0x55d86f6f17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.987+0000 7f893f57c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6f4320 0x55d86f6f1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.987+0000 7f893f57c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d86f6f7160 con 0x55d86f6e9330 2024-05-09T08:55:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.987+0000 7f893f57c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d86f609c40 con 0x55d86f6e9c50 2024-05-09T08:55:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.987+0000 7f893f57c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d86f6f72e0 con 0x55d86f6f4320 2024-05-09T08:55:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.987+0000 7f893f57c740 10 client.0 _ll_register_callbacks cb 0x55d86f361be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.987+0000 7f8939d2b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 0x55d86f6f17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.988+0000 7f8939d2b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 0x55d86f6f17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45580/0 (socket says 172.21.15.98:45580) 2024-05-09T08:55:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.988+0000 7f8939d2b640 1 -- 172.21.15.98:0/938498166 learned_addr learned my addr 172.21.15.98:0/938498166 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.988+0000 7f893ad2d640 1 --2- 172.21.15.98:0/938498166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6f4320 0x55d86f6f1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.988+0000 7f893a52c640 1 --2- 172.21.15.98:0/938498166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9330 0x55d86f6f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:35.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.988+0000 7f8939d2b640 1 -- 172.21.15.98:0/938498166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6f4320 msgr2=0x55d86f6f1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:35.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.988+0000 7f8939d2b640 1 --2- 172.21.15.98:0/938498166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6f4320 0x55d86f6f1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.988+0000 7f8939d2b640 1 -- 172.21.15.98:0/938498166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9330 msgr2=0x55d86f6f1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:35.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.988+0000 7f8939d2b640 1 --2- 172.21.15.98:0/938498166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9330 0x55d86f6f1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:35.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.988+0000 7f8939d2b640 1 -- 172.21.15.98:0/938498166 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d86f42c7d0 con 0x55d86f6e9c50 2024-05-09T08:55:35.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.988+0000 7f8939d2b640 1 --2- 172.21.15.98:0/938498166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 0x55d86f6f17d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f892c0b49d0 tx=0x7f892c0a8530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cff4eca0f1e62f57 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.990+0000 7f891dffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:55:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.990+0000 7f891dffb640 1 -- 172.21.15.98:0/938498166 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c0c5040 con 0x55d86f6e9c50 2024-05-09T08:55:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.990+0000 7f891dffb640 1 -- 172.21.15.98:0/938498166 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f892c027b80 con 0x55d86f6e9c50 2024-05-09T08:55:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.990+0000 7f893f57c740 1 -- 172.21.15.98:0/938498166 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f892c0b4f00 con 0x55d86f6e9c50 2024-05-09T08:55:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:35.991+0000 7f891dffb640 1 -- 172.21.15.98:0/938498166 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f892c0b0410 con 0x55d86f6e9c50 2024-05-09T08:55:35.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.000+0000 7f893f57c740 10 client.26294 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:35.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.000+0000 7f893f57c740 1 -- 172.21.15.98:0/938498166 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55d86f6f7160 con 0x55d86f6e9c50 2024-05-09T08:55:36.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.003+0000 7f891dffb640 1 -- 172.21.15.98:0/938498166 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f892c0b4f00 con 0x55d86f6e9c50 2024-05-09T08:55:36.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.003+0000 7f891dffb640 1 -- 172.21.15.98:0/938498166 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f892c0b17d0 con 0x55d86f6e9c50 2024-05-09T08:55:36.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.003+0000 7f893f57c740 10 client.26294 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:36.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.003+0000 7f893f57c740 10 client.26294 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:55:36.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.003+0000 7f893f57c740 20 client.26294 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:36.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.003+0000 7f893f57c740 1 --2- 172.21.15.98:0/938498166 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d86f704500 0x55d86f7068f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.003+0000 7f893f57c740 4 client.26294 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:36.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.004+0000 7f893f57c740 1 -- 172.21.15.98:0/938498166 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d86f65c370 con 0x55d86f704500 2024-05-09T08:55:36.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.004+0000 7f893a52c640 1 --2- 172.21.15.98:0/938498166 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d86f704500 0x55d86f7068f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.004+0000 7f891dffb640 10 client.26294 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:55:36.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.005+0000 7f893a52c640 1 --2- 172.21.15.98:0/938498166 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d86f704500 0x55d86f7068f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5fdd272860f87423 server_cookie=953c29fd6ce34066 in_seq=0 out_seq=0 2024-05-09T08:55:36.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.007+0000 7f891dffb640 1 -- 172.21.15.98:0/938498166 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d86f65c370 con 0x55d86f704500 2024-05-09T08:55:36.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.007+0000 7f891dffb640 10 client.26294 handle_command_reply: tid=0 2024-05-09T08:55:36.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.007+0000 7f893f57c740 1 client.26294 shutdown 2024-05-09T08:55:36.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.007+0000 7f893f57c740 1 -- 172.21.15.98:0/938498166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 msgr2=0x55d86f6f17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:36.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.007+0000 7f893f57c740 1 --2- 172.21.15.98:0/938498166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 0x55d86f6f17d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f892c0b49d0 tx=0x7f892c0a8530 comp rx=0 tx=0).stop 2024-05-09T08:55:36.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.007+0000 7f893f57c740 1 -- 172.21.15.98:0/938498166 shutdown_connections 2024-05-09T08:55:36.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.007+0000 7f893f57c740 1 --2- 172.21.15.98:0/938498166 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d86f704500 0x55d86f7068f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.007+0000 7f893f57c740 1 --2- 172.21.15.98:0/938498166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6f4320 0x55d86f6f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.007+0000 7f893f57c740 1 --2- 172.21.15.98:0/938498166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 0x55d86f6f17d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.007+0000 7f893f57c740 1 --2- 172.21.15.98:0/938498166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9330 0x55d86f6f1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.007+0000 7f893f57c740 1 -- 172.21.15.98:0/938498166 >> 172.21.15.98:0/938498166 conn(0x55d86f6e5650 msgr2=0x55d86f6eb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:36.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.007+0000 7f891dffb640 0 client.26294 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:55:36.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.008+0000 7f893f57c740 1 -- 172.21.15.98:0/938498166 shutdown_connections 2024-05-09T08:55:36.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.011+0000 7f893f57c740 1 -- 172.21.15.98:0/938498166 wait complete. 2024-05-09T08:55:36.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.011+0000 7f893f57c740 20 client.26294 trim_cache size 0 max 16384 2024-05-09T08:55:36.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.050+0000 7f893f57c740 1 Processor -- start 2024-05-09T08:55:36.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.050+0000 7f893f57c740 1 -- start start 2024-05-09T08:55:36.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.051+0000 7f893f57c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6e9330 0x55d86f6f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.051+0000 7f893f57c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 0x55d86f6eba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.051+0000 7f893f57c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6f4320 0x55d86f709530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.051+0000 7f893f57c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89341c6450 con 0x55d86f6f4320 2024-05-09T08:55:36.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.051+0000 7f893f57c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89341c7f90 con 0x55d86f6e9c50 2024-05-09T08:55:36.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.051+0000 7f893f57c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f89341c6b20 con 0x55d86f6e9330 2024-05-09T08:55:36.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.051+0000 7f893ad2d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6f4320 0x55d86f709530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.051+0000 7f893ad2d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6f4320 0x55d86f709530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37218/0 (socket says 172.21.15.98:37218) 2024-05-09T08:55:36.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.051+0000 7f893ad2d640 1 -- 172.21.15.98:0/1167854354 learned_addr learned my addr 172.21.15.98:0/1167854354 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:36.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.054+0000 7f893a52c640 1 --2- 172.21.15.98:0/1167854354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6e9330 0x55d86f6f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.056+0000 7f893ad2d640 1 -- 172.21.15.98:0/1167854354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6e9330 msgr2=0x55d86f6f1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:36.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.056+0000 7f893ad2d640 1 --2- 172.21.15.98:0/1167854354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6e9330 0x55d86f6f1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.056+0000 7f893ad2d640 1 -- 172.21.15.98:0/1167854354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 msgr2=0x55d86f6eba90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:55:36.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.056+0000 7f893ad2d640 1 --2- 172.21.15.98:0/1167854354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 0x55d86f6eba90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.056+0000 7f893ad2d640 1 -- 172.21.15.98:0/1167854354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d86f6ece70 con 0x55d86f6f4320 2024-05-09T08:55:36.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.057+0000 7f893ad2d640 1 --2- 172.21.15.98:0/1167854354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6f4320 0x55d86f709530 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f892402a8d0 tx=0x7f8924005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e534db4e4d97cac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:36.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.061+0000 7f891ffff640 1 -- 172.21.15.98:0/1167854354 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924051070 con 0x55d86f6f4320 2024-05-09T08:55:36.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.061+0000 7f891ffff640 1 -- 172.21.15.98:0/1167854354 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8924030e30 con 0x55d86f6f4320 2024-05-09T08:55:36.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.062+0000 7f893f57c740 1 -- 172.21.15.98:0/1167854354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6f4320 msgr2=0x55d86f709530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:36.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.062+0000 7f893f57c740 1 --2- 172.21.15.98:0/1167854354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6f4320 0x55d86f709530 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f892402a8d0 tx=0x7f8924005f90 comp rx=0 tx=0).stop 2024-05-09T08:55:36.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.063+0000 7f893f57c740 1 -- 172.21.15.98:0/1167854354 shutdown_connections 2024-05-09T08:55:36.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.063+0000 7f893f57c740 1 --2- 172.21.15.98:0/1167854354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6f4320 0x55d86f709530 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.063+0000 7f893f57c740 1 --2- 172.21.15.98:0/1167854354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 0x55d86f6eba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.063+0000 7f893f57c740 1 --2- 172.21.15.98:0/1167854354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f6e9330 0x55d86f6f1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.063+0000 7f893f57c740 1 -- 172.21.15.98:0/1167854354 >> 172.21.15.98:0/1167854354 conn(0x55d86f6e5650 msgr2=0x55d86f6f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:36.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.071+0000 7f893f57c740 1 -- 172.21.15.98:0/1167854354 shutdown_connections 2024-05-09T08:55:36.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.071+0000 7f893f57c740 1 -- 172.21.15.98:0/1167854354 wait complete. 2024-05-09T08:55:36.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.071+0000 7f893f57c740 1 -- start start 2024-05-09T08:55:36.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.072+0000 7f893f57c740 1 Processor -- start 2024-05-09T08:55:36.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.072+0000 7f893f57c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9330 0x55d86f707080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.072+0000 7f893f57c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 0x55d86f7075c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.072+0000 7f893f57c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f707b00 0x55d86f7696e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.072+0000 7f893f57c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89341c78c0 con 0x55d86f6e9330 2024-05-09T08:55:36.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.072+0000 7f893f57c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89341c71f0 con 0x55d86f6e9c50 2024-05-09T08:55:36.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.072+0000 7f893f57c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8934084140 con 0x55d86f707b00 2024-05-09T08:55:36.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.072+0000 7f893f57c740 10 client.0 _ll_register_callbacks cb 0x7f892c0c3070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:36.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.074+0000 7f893ad2d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f707b00 0x55d86f7696e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.074+0000 7f893a52c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9330 0x55d86f707080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.074+0000 7f893a52c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9330 0x55d86f707080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37232/0 (socket says 172.21.15.98:37232) 2024-05-09T08:55:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.074+0000 7f893a52c640 1 -- 172.21.15.98:0/3453525670 learned_addr learned my addr 172.21.15.98:0/3453525670 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.074+0000 7f8939d2b640 1 --2- 172.21.15.98:0/3453525670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 0x55d86f7075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.075+0000 7f893a52c640 1 -- 172.21.15.98:0/3453525670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f707b00 msgr2=0x55d86f7696e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.075+0000 7f893a52c640 1 --2- 172.21.15.98:0/3453525670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f707b00 0x55d86f7696e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.075+0000 7f893a52c640 1 -- 172.21.15.98:0/3453525670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 msgr2=0x55d86f7075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.075+0000 7f893a52c640 1 --2- 172.21.15.98:0/3453525670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 0x55d86f7075c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.075+0000 7f893a52c640 1 -- 172.21.15.98:0/3453525670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d86f6f0240 con 0x55d86f6e9330 2024-05-09T08:55:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.076+0000 7f893a52c640 1 --2- 172.21.15.98:0/3453525670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9330 0x55d86f707080 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f8930009c40 tx=0x7f893001df20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1d4cfcb2de18ad server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.076+0000 7f891dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:55:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.077+0000 7f891dffb640 1 -- 172.21.15.98:0/3453525670 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893000f040 con 0x55d86f6e9330 2024-05-09T08:55:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.077+0000 7f891dffb640 1 -- 172.21.15.98:0/3453525670 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f8930016d20 con 0x55d86f6e9330 2024-05-09T08:55:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.077+0000 7f893f57c740 1 -- 172.21.15.98:0/3453525670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d86f6566c0 con 0x55d86f6e9330 2024-05-09T08:55:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.078+0000 7f891dffb640 1 -- 172.21.15.98:0/3453525670 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f893002e050 con 0x55d86f6e9330 2024-05-09T08:55:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.078+0000 7f893f57c740 10 client.17010 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.078+0000 7f893f57c740 1 -- 172.21.15.98:0/3453525670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55d86f76ae20 con 0x55d86f6e9330 2024-05-09T08:55:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.079+0000 7f891dffb640 1 -- 172.21.15.98:0/3453525670 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d86f6566c0 con 0x55d86f6e9330 2024-05-09T08:55:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.079+0000 7f891dffb640 1 -- 172.21.15.98:0/3453525670 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f89300268c0 con 0x55d86f6e9330 2024-05-09T08:55:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.080+0000 7f893f57c740 10 client.17010 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.080+0000 7f893f57c740 10 client.17010 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:55:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.080+0000 7f893f57c740 20 client.17010 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.080+0000 7f893f57c740 1 --2- 172.21.15.98:0/3453525670 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d86f76b550 0x55d86f76d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.080+0000 7f893f57c740 4 client.17010 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:55:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.080+0000 7f893f57c740 1 -- 172.21.15.98:0/3453525670 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d86f5d3820 con 0x55d86f76b550 2024-05-09T08:55:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.080+0000 7f8939d2b640 1 --2- 172.21.15.98:0/3453525670 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d86f76b550 0x55d86f76d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.081+0000 7f891dffb640 10 client.17010 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:55:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.082+0000 7f8939d2b640 1 --2- 172.21.15.98:0/3453525670 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d86f76b550 0x55d86f76d970 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5d683b2cd66a886b server_cookie=83b28be06f326161 in_seq=0 out_seq=0 2024-05-09T08:55:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.082+0000 7f891dffb640 1 -- 172.21.15.98:0/3453525670 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d86f5d3820 con 0x55d86f76b550 2024-05-09T08:55:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.082+0000 7f891dffb640 10 client.17010 handle_command_reply: tid=0 2024-05-09T08:55:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.082+0000 7f893f57c740 1 client.17010 shutdown 2024-05-09T08:55:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.090+0000 7f893f57c740 1 -- 172.21.15.98:0/3453525670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9330 msgr2=0x55d86f707080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.090+0000 7f893f57c740 1 --2- 172.21.15.98:0/3453525670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9330 0x55d86f707080 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f8930009c40 tx=0x7f893001df20 comp rx=0 tx=0).stop 2024-05-09T08:55:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.090+0000 7f893f57c740 1 -- 172.21.15.98:0/3453525670 shutdown_connections 2024-05-09T08:55:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.090+0000 7f893f57c740 1 --2- 172.21.15.98:0/3453525670 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d86f76b550 0x55d86f76d970 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.090+0000 7f893f57c740 1 --2- 172.21.15.98:0/3453525670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d86f707b00 0x55d86f7696e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.090+0000 7f893f57c740 1 --2- 172.21.15.98:0/3453525670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d86f6e9c50 0x55d86f7075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.090+0000 7f893f57c740 1 --2- 172.21.15.98:0/3453525670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d86f6e9330 0x55d86f707080 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.090+0000 7f891dffb640 0 client.17010 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:55:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.090+0000 7f893f57c740 1 -- 172.21.15.98:0/3453525670 >> 172.21.15.98:0/3453525670 conn(0x55d86f6e5650 msgr2=0x55d86f6f2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.093+0000 7f893f57c740 1 -- 172.21.15.98:0/3453525670 shutdown_connections 2024-05-09T08:55:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.093+0000 7f893f57c740 1 -- 172.21.15.98:0/3453525670 wait complete. 2024-05-09T08:55:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.095+0000 7f893f57c740 20 client.17010 trim_cache size 0 max 16384 2024-05-09T08:55:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:55:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.106+0000 7f893c790640 1 -- 172.21.15.98:0/3490663553 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89200a2f00 msgr2=0x7f89200a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.106+0000 7f893c790640 1 --2- 172.21.15.98:0/3490663553 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89200a2f00 0x7f89200a53c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8924009780 tx=0x7f892402c040 comp rx=0 tx=0).stop 2024-05-09T08:55:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.106+0000 7f893c790640 1 -- 172.21.15.98:0/3490663553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8934110080 msgr2=0x7f89341167b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.106+0000 7f893c790640 1 --2- 172.21.15.98:0/3490663553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8934110080 0x7f89341167b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f892c01fbc0 tx=0x7f892c01d380 comp rx=0 tx=0).stop 2024-05-09T08:55:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.110+0000 7f893c790640 1 -- 172.21.15.98:0/3490663553 shutdown_connections 2024-05-09T08:55:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.110+0000 7f893c790640 1 --2- 172.21.15.98:0/3490663553 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89200a2f00 0x7f89200a53c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.110+0000 7f893c790640 1 --2- 172.21.15.98:0/3490663553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f893411b010 0x7f8934116cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.110+0000 7f893c790640 1 --2- 172.21.15.98:0/3490663553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8934110080 0x7f89341167b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.110+0000 7f893c790640 1 --2- 172.21.15.98:0/3490663553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893410e0b0 0x7f893411a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.110+0000 7f893c790640 1 -- 172.21.15.98:0/3490663553 >> 172.21.15.98:0/3490663553 conn(0x7f893406c940 msgr2=0x7f893411c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.114+0000 7f893c790640 1 -- 172.21.15.98:0/3490663553 shutdown_connections 2024-05-09T08:55:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.116+0000 7f893c790640 1 -- 172.21.15.98:0/3490663553 wait complete. 2024-05-09T08:55:36.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:55:36.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.483+0000 7f5dcf59e640 1 -- 172.21.15.98:0/1345043096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dd0108830 msgr2=0x7f5dd0108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:36.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.483+0000 7f5dcf59e640 1 --2- 172.21.15.98:0/1345043096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dd0108830 0x7f5dd0108c90 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009860 tx=0x7f5dc402a3d0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.484+0000 7f5dcf59e640 1 -- 172.21.15.98:0/1345043096 shutdown_connections 2024-05-09T08:55:36.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.484+0000 7f5dcf59e640 1 --2- 172.21.15.98:0/1345043096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5dd01092b0 0x7f5dd011d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.484+0000 7f5dcf59e640 1 --2- 172.21.15.98:0/1345043096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dd0108830 0x7f5dd0108c90 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.484+0000 7f5dcf59e640 1 --2- 172.21.15.98:0/1345043096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dd010fed0 0x7f5dd01102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.484+0000 7f5dcf59e640 1 -- 172.21.15.98:0/1345043096 >> 172.21.15.98:0/1345043096 conn(0x7f5dd006c7b0 msgr2=0x7f5dd006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:36.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.489+0000 7f5dcf59e640 1 -- 172.21.15.98:0/1345043096 shutdown_connections 2024-05-09T08:55:36.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.489+0000 7f5dcf59e640 1 -- 172.21.15.98:0/1345043096 wait complete. 2024-05-09T08:55:36.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.490+0000 7f5dcf59e640 1 Processor -- start 2024-05-09T08:55:36.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.499+0000 7f5dcf59e640 1 -- start start 2024-05-09T08:55:36.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.499+0000 7f5dcf59e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5dd0108830 0x7f5dd011cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.499+0000 7f5dcf59e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dd01092b0 0x7f5dd0115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.499+0000 7f5dcf59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dd010fed0 0x7f5dd0116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.499+0000 7f5dcf59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd01208b0 con 0x7f5dd010fed0 2024-05-09T08:55:36.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.499+0000 7f5dcf59e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd0120730 con 0x7f5dd01092b0 2024-05-09T08:55:36.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.499+0000 7f5dcf59e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd0120a30 con 0x7f5dd0108830 2024-05-09T08:55:36.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.500+0000 7f5dce59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5dd0108830 0x7f5dd011cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.500+0000 7f5dce59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5dd0108830 0x7f5dd011cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:46796/0 (socket says 172.21.15.98:46796) 2024-05-09T08:55:36.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.500+0000 7f5dce59c640 1 -- 172.21.15.98:0/351499463 learned_addr learned my addr 172.21.15.98:0/351499463 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:36.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.501+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/351499463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dd01092b0 0x7f5dd0115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.502+0000 7f5dcdd9b640 1 -- 172.21.15.98:0/351499463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5dd0108830 msgr2=0x7f5dd011cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:36.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.502+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/351499463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5dd0108830 0x7f5dd011cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.502+0000 7f5dcdd9b640 1 -- 172.21.15.98:0/351499463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dd010fed0 msgr2=0x7f5dd0116190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:36.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.502+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/351499463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dd010fed0 0x7f5dd0116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.502+0000 7f5dcdd9b640 1 -- 172.21.15.98:0/351499463 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd0116750 con 0x7f5dd01092b0 2024-05-09T08:55:36.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.505+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/351499463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dd01092b0 0x7f5dd0115c50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009780 tx=0x7f5dc4030920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:36.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.508+0000 7f5dbf7fe640 1 -- 172.21.15.98:0/351499463 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4040070 con 0x7f5dd01092b0 2024-05-09T08:55:36.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.509+0000 7f5dcf59e640 1 -- 172.21.15.98:0/351499463 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd01169e0 con 0x7f5dd01092b0 2024-05-09T08:55:36.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.509+0000 7f5dcf59e640 1 -- 172.21.15.98:0/351499463 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd011a980 con 0x7f5dd01092b0 2024-05-09T08:55:36.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.513+0000 7f5dd7dc9740 1 Processor -- start 2024-05-09T08:55:36.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.513+0000 7f5dbf7fe640 1 -- 172.21.15.98:0/351499463 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dc4005590 con 0x7f5dd01092b0 2024-05-09T08:55:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.513+0000 7f5dbf7fe640 1 -- 172.21.15.98:0/351499463 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4004d30 con 0x7f5dd01092b0 2024-05-09T08:55:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.513+0000 7f5dd7dc9740 1 -- start start 2024-05-09T08:55:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.514+0000 7f5dd7dc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87520 0x562695a87900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.514+0000 7f5dd7dc9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a91fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.514+0000 7f5dd7dc9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 0x562695a94900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.515+0000 7f5dd7dc9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5626959ecc40 con 0x562695a87520 2024-05-09T08:55:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.515+0000 7f5dd7dc9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562695994580 con 0x562695a87e40 2024-05-09T08:55:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.515+0000 7f5dd7dc9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5626959e8c00 con 0x562695a92510 2024-05-09T08:55:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.515+0000 7f5dce59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a91fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.517+0000 7f5dced9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87520 0x562695a87900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.517+0000 7f5dced9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87520 0x562695a87900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37256/0 (socket says 172.21.15.98:37256) 2024-05-09T08:55:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.517+0000 7f5dced9d640 1 -- 172.21.15.98:0/2655711430 learned_addr learned my addr 172.21.15.98:0/2655711430 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.517+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/2655711430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 0x562695a94900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.517+0000 7f5dbf7fe640 1 -- 172.21.15.98:0/351499463 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5dc4039040 con 0x7f5dd01092b0 2024-05-09T08:55:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.519+0000 7f5dbf7fe640 1 --2- 172.21.15.98:0/351499463 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5db80a2f00 0x7f5db80a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.519+0000 7f5dbf7fe640 1 -- 172.21.15.98:0/351499463 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f5dc40f2ee0 con 0x7f5dd01092b0 2024-05-09T08:55:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.520+0000 7f5dced9d640 1 -- 172.21.15.98:0/2655711430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 msgr2=0x562695a94900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.520+0000 7f5dced9d640 1 --2- 172.21.15.98:0/2655711430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 0x562695a94900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.520+0000 7f5dced9d640 1 -- 172.21.15.98:0/2655711430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 msgr2=0x562695a91fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.520+0000 7f5dced9d640 1 --2- 172.21.15.98:0/2655711430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a91fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.520+0000 7f5dced9d640 1 -- 172.21.15.98:0/2655711430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626957003f0 con 0x562695a87520 2024-05-09T08:55:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.520+0000 7f5dce59c640 1 --2- 172.21.15.98:0/2655711430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a91fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:55:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.520+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/2655711430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 0x562695a94900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:55:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.521+0000 7f5dced9d640 1 --2- 172.21.15.98:0/2655711430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87520 0x562695a87900 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8009fd0 tx=0x7f5dc800e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c9d9f938e71754 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.524+0000 7f5dced9d640 1 --2- 172.21.15.98:0/351499463 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5db80a2f00 0x7f5db80a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.524+0000 7f5dbcff9640 1 -- 172.21.15.98:0/2655711430 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc8012650 con 0x562695a87520 2024-05-09T08:55:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.524+0000 7f5dbcff9640 1 -- 172.21.15.98:0/2655711430 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dc8012cd0 con 0x562695a87520 2024-05-09T08:55:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.524+0000 7f5dbcff9640 1 -- 172.21.15.98:0/2655711430 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc80193c0 con 0x562695a87520 2024-05-09T08:55:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.525+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/2655711430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87520 msgr2=0x562695a87900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.525+0000 7f5dced9d640 1 --2- 172.21.15.98:0/351499463 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5db80a2f00 0x7f5db80a53c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8011fd0 tx=0x7f5dc80196d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.525+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/2655711430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87520 0x562695a87900 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8009fd0 tx=0x7f5dc800e830 comp rx=0 tx=0).stop 2024-05-09T08:55:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.526+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/2655711430 shutdown_connections 2024-05-09T08:55:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.526+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/2655711430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 0x562695a94900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.526+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/2655711430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a91fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.526+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/2655711430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87520 0x562695a87900 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.526+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/2655711430 >> 172.21.15.98:0/2655711430 conn(0x562695a83840 msgr2=0x562695a83c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.531+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/2655711430 shutdown_connections 2024-05-09T08:55:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.533+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/2655711430 wait complete. 2024-05-09T08:55:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.533+0000 7f5dd7dc9740 1 -- start start 2024-05-09T08:55:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.533+0000 7f5dd7dc9740 1 Processor -- start 2024-05-09T08:55:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.535+0000 7f5dd7dc9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a87520 0x562695a8f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.535+0000 7f5dd7dc9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a8f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.535+0000 7f5dd7dc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a92510 0x562695a8fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.535+0000 7f5dd7dc9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562695a95300 con 0x562695a92510 2024-05-09T08:55:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.535+0000 7f5dd7dc9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5626959a7c40 con 0x562695a87e40 2024-05-09T08:55:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.535+0000 7f5dd7dc9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562695a95480 con 0x562695a87520 2024-05-09T08:55:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.535+0000 7f5dd7dc9740 10 client.0 _ll_register_callbacks cb 0x5626956ffbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.536+0000 7f5dced9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a92510 0x562695a8fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.536+0000 7f5dced9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a92510 0x562695a8fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37274/0 (socket says 172.21.15.98:37274) 2024-05-09T08:55:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.536+0000 7f5dced9d640 1 -- 172.21.15.98:0/1423531527 learned_addr learned my addr 172.21.15.98:0/1423531527 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.536+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/1423531527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a8f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.536+0000 7f5dce59c640 1 --2- 172.21.15.98:0/1423531527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a87520 0x562695a8f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.537+0000 7f5dcdd9b640 1 -- 172.21.15.98:0/1423531527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a87520 msgr2=0x562695a8f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.537+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/1423531527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a87520 0x562695a8f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.537+0000 7f5dcdd9b640 1 -- 172.21.15.98:0/1423531527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a92510 msgr2=0x562695a8fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.537+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/1423531527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a92510 0x562695a8fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.537+0000 7f5dcdd9b640 1 -- 172.21.15.98:0/1423531527 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626957ca7d0 con 0x562695a87e40 2024-05-09T08:55:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.542+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/1423531527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a8f990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40f0210 tx=0x7f5dc4030a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6721c33896aa09dd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.543+0000 7f5db5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:55:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.543+0000 7f5db5ffb640 1 -- 172.21.15.98:0/1423531527 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc40d7020 con 0x562695a87e40 2024-05-09T08:55:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.543+0000 7f5db5ffb640 1 -- 172.21.15.98:0/1423531527 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f5dc40bf440 con 0x562695a87e40 2024-05-09T08:55:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.543+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1423531527 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dc8012cd0 con 0x562695a87e40 2024-05-09T08:55:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.544+0000 7f5db5ffb640 1 -- 172.21.15.98:0/1423531527 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5dc40f06a0 con 0x562695a87e40 2024-05-09T08:55:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.544+0000 7f5dd7dc9740 10 client.26309 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.544+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1423531527 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562695aa1550 con 0x562695a87e40 2024-05-09T08:55:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.544+0000 7f5db5ffb640 1 -- 172.21.15.98:0/1423531527 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dc8012cd0 con 0x562695a87e40 2024-05-09T08:55:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.545+0000 7f5db5ffb640 1 -- 172.21.15.98:0/1423531527 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f5dc404d070 con 0x562695a87e40 2024-05-09T08:55:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.545+0000 7f5dd7dc9740 10 client.26309 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.545+0000 7f5dd7dc9740 10 client.26309 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:55:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.545+0000 7f5dd7dc9740 20 client.26309 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.545+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1423531527 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x562695aa29b0 0x562695aa4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.545+0000 7f5dd7dc9740 4 client.26309 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.545+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1423531527 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626959fa370 con 0x562695aa29b0 2024-05-09T08:55:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.545+0000 7f5dce59c640 1 --2- 172.21.15.98:0/1423531527 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x562695aa29b0 0x562695aa4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.546+0000 7f5db5ffb640 10 client.26309 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:55:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.546+0000 7f5dce59c640 1 --2- 172.21.15.98:0/1423531527 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x562695aa29b0 0x562695aa4dd0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6b82acd1f237213a server_cookie=1c5d06444d193c0c in_seq=0 out_seq=0 2024-05-09T08:55:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.547+0000 7f5db5ffb640 1 -- 172.21.15.98:0/1423531527 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5626959fa370 con 0x562695aa29b0 2024-05-09T08:55:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.547+0000 7f5db5ffb640 10 client.26309 handle_command_reply: tid=0 2024-05-09T08:55:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.547+0000 7f5dd7dc9740 1 client.26309 shutdown 2024-05-09T08:55:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.551+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1423531527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 msgr2=0x562695a8f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.551+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1423531527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a8f990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40f0210 tx=0x7f5dc4030a10 comp rx=0 tx=0).stop 2024-05-09T08:55:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.551+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1423531527 shutdown_connections 2024-05-09T08:55:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.551+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1423531527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a92510 0x562695a8fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.551+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1423531527 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x562695aa29b0 0x562695aa4dd0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.551+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1423531527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a8f990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.551+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1423531527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a87520 0x562695a8f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.551+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1423531527 >> 172.21.15.98:0/1423531527 conn(0x562695a83840 msgr2=0x562695a829c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.551+0000 7f5db5ffb640 0 client.26309 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:55:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.551+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1423531527 shutdown_connections 2024-05-09T08:55:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.551+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1423531527 wait complete. 2024-05-09T08:55:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.551+0000 7f5dd7dc9740 20 client.26309 trim_cache size 0 max 16384 2024-05-09T08:55:36.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:36 smithi098 ceph-mon[30092]: pgmap v153: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 750 MiB/s wr, 1.22k op/s 2024-05-09T08:55:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.594+0000 7f5dd7dc9740 1 Processor -- start 2024-05-09T08:55:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.594+0000 7f5dd7dc9740 1 -- start start 2024-05-09T08:55:36.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.594+0000 7f5dd7dc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87520 0x562695a9e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.594+0000 7f5dd7dc9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a82840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.595+0000 7f5dd7dc9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 0x562695aaa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.595+0000 7f5dd7dc9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5db0010030 con 0x562695a87520 2024-05-09T08:55:36.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.595+0000 7f5dd7dc9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5db0011b70 con 0x562695a87e40 2024-05-09T08:55:36.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.595+0000 7f5dd7dc9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5db0010700 con 0x562695a92510 2024-05-09T08:55:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.595+0000 7f5dcdd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a82840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.595+0000 7f5dcdd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a82840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45638/0 (socket says 172.21.15.98:45638) 2024-05-09T08:55:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.595+0000 7f5dcdd9b640 1 -- 172.21.15.98:0/1245520567 learned_addr learned my addr 172.21.15.98:0/1245520567 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.596+0000 7f5dced9d640 1 --2- 172.21.15.98:0/1245520567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 0x562695aaa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.596+0000 7f5dcdd9b640 1 -- 172.21.15.98:0/1245520567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 msgr2=0x562695aaa6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:36.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.596+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/1245520567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 0x562695aaa6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.596+0000 7f5dcdd9b640 1 -- 172.21.15.98:0/1245520567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87520 msgr2=0x562695a9e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:36.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.596+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/1245520567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87520 0x562695a9e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.596+0000 7f5dcdd9b640 1 -- 172.21.15.98:0/1245520567 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562695a8a2e0 con 0x562695a87e40 2024-05-09T08:55:36.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.596+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/1245520567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a82840 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40bb440 tx=0x7f5dc40c77c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3097aca7a932540f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:36.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.596+0000 7f5db7fff640 1 -- 172.21.15.98:0/1245520567 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc40d7030 con 0x562695a87e40 2024-05-09T08:55:36.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.597+0000 7f5db7fff640 1 -- 172.21.15.98:0/1245520567 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dc40df0b0 con 0x562695a87e40 2024-05-09T08:55:36.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.597+0000 7f5db7fff640 1 -- 172.21.15.98:0/1245520567 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4004ed0 con 0x562695a87e40 2024-05-09T08:55:36.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.598+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1245520567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 msgr2=0x562695a82840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:36.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.598+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1245520567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a82840 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40bb440 tx=0x7f5dc40c77c0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.598+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1245520567 shutdown_connections 2024-05-09T08:55:36.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.598+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1245520567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 0x562695aaa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.598+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1245520567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87e40 0x562695a82840 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.598+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1245520567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87520 0x562695a9e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.599+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1245520567 >> 172.21.15.98:0/1245520567 conn(0x562695a83840 msgr2=0x562695a82130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:36.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.599+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1245520567 shutdown_connections 2024-05-09T08:55:36.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.599+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1245520567 wait complete. 2024-05-09T08:55:36.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.599+0000 7f5dd7dc9740 1 -- start start 2024-05-09T08:55:36.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.600+0000 7f5dd7dc9740 1 Processor -- start 2024-05-09T08:55:36.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.600+0000 7f5dd7dc9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87520 0x562695a962b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.600+0000 7f5dd7dc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87e40 0x562695a967f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.601+0000 7f5dce59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87520 0x562695a962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.601+0000 7f5dce59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87520 0x562695a962b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45646/0 (socket says 172.21.15.98:45646) 2024-05-09T08:55:36.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.601+0000 7f5dce59c640 1 -- 172.21.15.98:0/1182294677 learned_addr learned my addr 172.21.15.98:0/1182294677 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:36.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.601+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1182294677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 0x562695a96d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.601+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1182294677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5db00114a0 con 0x562695a87e40 2024-05-09T08:55:36.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.601+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1182294677 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5db0010dd0 con 0x562695a87520 2024-05-09T08:55:36.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.601+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1182294677 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5db000e010 con 0x562695a92510 2024-05-09T08:55:36.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.601+0000 7f5dd7dc9740 10 client.0 _ll_register_callbacks cb 0x7f5dc40bb280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:36.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.603+0000 7f5dced9d640 1 --2- 172.21.15.98:0/1182294677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 0x562695a96d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.604+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/1182294677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87e40 0x562695a967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.605+0000 7f5dcdd9b640 1 -- 172.21.15.98:0/1182294677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 msgr2=0x562695a96d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:36.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.605+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/1182294677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 0x562695a96d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.605+0000 7f5dcdd9b640 1 -- 172.21.15.98:0/1182294677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87520 msgr2=0x562695a962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:36.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.605+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/1182294677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87520 0x562695a962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.605+0000 7f5dcdd9b640 1 -- 172.21.15.98:0/1182294677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562695b08060 con 0x562695a87e40 2024-05-09T08:55:36.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.605+0000 7f5dcdd9b640 1 --2- 172.21.15.98:0/1182294677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87e40 0x562695a967f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x562695a74eb0 tx=0x7f5dc40c7ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d92ed723bfa50bac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:36.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.608+0000 7f5db5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:55:36.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.608+0000 7f5db5ffb640 1 -- 172.21.15.98:0/1182294677 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc40d7030 con 0x562695a87e40 2024-05-09T08:55:36.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.608+0000 7f5db5ffb640 1 -- 172.21.15.98:0/1182294677 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f5dc40b9ba0 con 0x562695a87e40 2024-05-09T08:55:36.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.609+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1182294677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dc40df0b0 con 0x562695a87e40 2024-05-09T08:55:36.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.616+0000 7f5db5ffb640 1 -- 172.21.15.98:0/1182294677 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5dc40bf6e0 con 0x562695a87e40 2024-05-09T08:55:36.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.619+0000 7f5dd7dc9740 10 client.17022 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:36.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.619+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1182294677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562695b08060 con 0x562695a87e40 2024-05-09T08:55:36.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.623+0000 7f5db5ffb640 1 -- 172.21.15.98:0/1182294677 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dc40a2070 con 0x562695a87e40 2024-05-09T08:55:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.624+0000 7f5db5ffb640 1 -- 172.21.15.98:0/1182294677 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f5dc404d070 con 0x562695a87e40 2024-05-09T08:55:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.624+0000 7f5dd7dc9740 10 client.17022 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.624+0000 7f5dd7dc9740 10 client.17022 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:55:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.624+0000 7f5dd7dc9740 20 client.17022 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.625+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1182294677 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x562695b095c0 0x562695b0b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.625+0000 7f5dd7dc9740 4 client.17022 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:55:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.625+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1182294677 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f5dc40df0b0 con 0x562695b095c0 2024-05-09T08:55:36.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.627+0000 7f5dce59c640 1 --2- 172.21.15.98:0/1182294677 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x562695b095c0 0x562695b0b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:36.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.635+0000 7f5dce59c640 1 --2- 172.21.15.98:0/1182294677 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x562695b095c0 0x562695b0b9b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=27c952b40584b024 server_cookie=7bd22479536a7ee0 in_seq=0 out_seq=0 2024-05-09T08:55:36.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.635+0000 7f5db5ffb640 10 client.17022 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:55:36.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.635+0000 7f5db5ffb640 1 -- 172.21.15.98:0/1182294677 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f5dc40df0b0 con 0x562695b095c0 2024-05-09T08:55:36.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.635+0000 7f5db5ffb640 10 client.17022 handle_command_reply: tid=0 2024-05-09T08:55:36.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.635+0000 7f5dd7dc9740 1 client.17022 shutdown 2024-05-09T08:55:36.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.642+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1182294677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87e40 msgr2=0x562695a967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:36.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.642+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1182294677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87e40 0x562695a967f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x562695a74eb0 tx=0x7f5dc40c7ae0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.645+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1182294677 shutdown_connections 2024-05-09T08:55:36.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.645+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1182294677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562695a92510 0x562695a96d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.646+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1182294677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562695a87e40 0x562695a967f0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.646+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1182294677 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x562695b095c0 0x562695b0b9b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.646+0000 7f5dd7dc9740 1 --2- 172.21.15.98:0/1182294677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562695a87520 0x562695a962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.646+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1182294677 >> 172.21.15.98:0/1182294677 conn(0x562695a83840 msgr2=0x562695a829c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:36.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.647+0000 7f5db5ffb640 0 client.17022 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:55:36.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.647+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1182294677 shutdown_connections 2024-05-09T08:55:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.647+0000 7f5dd7dc9740 1 -- 172.21.15.98:0/1182294677 wait complete. 2024-05-09T08:55:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.647+0000 7f5dd7dc9740 20 client.17022 trim_cache size 0 max 16384 2024-05-09T08:55:36.647 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:55:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.652+0000 7f5dcf59e640 1 -- 172.21.15.98:0/351499463 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5db80a2f00 msgr2=0x7f5db80a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.652+0000 7f5dcf59e640 1 --2- 172.21.15.98:0/351499463 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5db80a2f00 0x7f5db80a53c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8011fd0 tx=0x7f5dc80196d0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.652+0000 7f5dcf59e640 1 -- 172.21.15.98:0/351499463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dd01092b0 msgr2=0x7f5dd0115c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.652+0000 7f5dcf59e640 1 --2- 172.21.15.98:0/351499463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dd01092b0 0x7f5dd0115c50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009780 tx=0x7f5dc4030920 comp rx=0 tx=0).stop 2024-05-09T08:55:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.653+0000 7f5dcf59e640 1 -- 172.21.15.98:0/351499463 shutdown_connections 2024-05-09T08:55:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.653+0000 7f5dcf59e640 1 --2- 172.21.15.98:0/351499463 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5db80a2f00 0x7f5db80a53c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.653+0000 7f5dcf59e640 1 --2- 172.21.15.98:0/351499463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dd010fed0 0x7f5dd0116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.653+0000 7f5dcf59e640 1 --2- 172.21.15.98:0/351499463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dd01092b0 0x7f5dd0115c50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.653+0000 7f5dcf59e640 1 --2- 172.21.15.98:0/351499463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5dd0108830 0x7f5dd011cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.653+0000 7f5dcf59e640 1 -- 172.21.15.98:0/351499463 >> 172.21.15.98:0/351499463 conn(0x7f5dd006c7b0 msgr2=0x7f5dd0070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.655+0000 7f5dcf59e640 1 -- 172.21.15.98:0/351499463 shutdown_connections 2024-05-09T08:55:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:36.656+0000 7f5dcf59e640 1 -- 172.21.15.98:0/351499463 wait complete. 2024-05-09T08:55:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:36 smithi142 ceph-mon[34625]: pgmap v153: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 750 MiB/s wr, 1.22k op/s 2024-05-09T08:55:36.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:55:36.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:36 smithi105 ceph-mon[34498]: pgmap v153: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 750 MiB/s wr, 1.22k op/s 2024-05-09T08:55:37.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.005+0000 7fc8513b0640 1 -- 172.21.15.98:0/3309725646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c10e2d0 msgr2=0x7fc84c10e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:37.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.005+0000 7fc8513b0640 1 --2- 172.21.15.98:0/3309725646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c10e2d0 0x7fc84c10e6b0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fc83c0085a0 tx=0x7fc83c00ac00 comp rx=0 tx=0).stop 2024-05-09T08:55:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.012+0000 7fc8513b0640 1 -- 172.21.15.98:0/3309725646 shutdown_connections 2024-05-09T08:55:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.012+0000 7fc8513b0640 1 --2- 172.21.15.98:0/3309725646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc84c1091f0 0x7fc84c115090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.012+0000 7fc8513b0640 1 --2- 172.21.15.98:0/3309725646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc84c108830 0x7fc84c108cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.012+0000 7fc8513b0640 1 --2- 172.21.15.98:0/3309725646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c10e2d0 0x7fc84c10e6b0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.012+0000 7fc8513b0640 1 -- 172.21.15.98:0/3309725646 >> 172.21.15.98:0/3309725646 conn(0x7fc84c06cc40 msgr2=0x7fc84c06d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:37.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.020+0000 7fc8513b0640 1 -- 172.21.15.98:0/3309725646 shutdown_connections 2024-05-09T08:55:37.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.020+0000 7fc8513b0640 1 -- 172.21.15.98:0/3309725646 wait complete. 2024-05-09T08:55:37.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.021+0000 7fc8513b0640 1 Processor -- start 2024-05-09T08:55:37.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.024+0000 7fc8513b0640 1 -- start start 2024-05-09T08:55:37.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.025+0000 7fc8513b0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c108830 0x7fc84c11e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.025+0000 7fc8513b0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc84c1091f0 0x7fc84c11e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.025+0000 7fc8513b0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc84c10e2d0 0x7fc84c118260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.025+0000 7fc8513b0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c116c90 con 0x7fc84c108830 2024-05-09T08:55:37.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.025+0000 7fc8513b0640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c116b10 con 0x7fc84c10e2d0 2024-05-09T08:55:37.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.025+0000 7fc8513b0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c116e10 con 0x7fc84c1091f0 2024-05-09T08:55:37.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.028+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc84c1091f0 0x7fc84c11e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.028+0000 7fc84affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c108830 0x7fc84c11e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.028+0000 7fc84affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c108830 0x7fc84c11e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37314/0 (socket says 172.21.15.98:37314) 2024-05-09T08:55:37.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.028+0000 7fc84affd640 1 -- 172.21.15.98:0/887609602 learned_addr learned my addr 172.21.15.98:0/887609602 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:37.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.028+0000 7fc84b7fe640 1 --2- 172.21.15.98:0/887609602 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc84c10e2d0 0x7fc84c118260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.030+0000 7fc84affd640 1 -- 172.21.15.98:0/887609602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc84c1091f0 msgr2=0x7fc84c11e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:37.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.030+0000 7fc84affd640 1 --2- 172.21.15.98:0/887609602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc84c1091f0 0x7fc84c11e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.030+0000 7fc84affd640 1 -- 172.21.15.98:0/887609602 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc84c10e2d0 msgr2=0x7fc84c118260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:37.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.031+0000 7fc84affd640 1 --2- 172.21.15.98:0/887609602 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc84c10e2d0 0x7fc84c118260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.031+0000 7fc84affd640 1 -- 172.21.15.98:0/887609602 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84c118b20 con 0x7fc84c108830 2024-05-09T08:55:37.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.031+0000 7fc84affd640 1 --2- 172.21.15.98:0/887609602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c108830 0x7fc84c11e1a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc83c000c00 tx=0x7fc83c02aaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:37.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.033+0000 7fc833fff640 1 -- 172.21.15.98:0/887609602 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c040070 con 0x7fc84c108830 2024-05-09T08:55:37.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.034+0000 7fc8513b0640 1 -- 172.21.15.98:0/887609602 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84c118db0 con 0x7fc84c108830 2024-05-09T08:55:37.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.034+0000 7fc8513b0640 1 -- 172.21.15.98:0/887609602 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc84c1bc310 con 0x7fc84c108830 2024-05-09T08:55:37.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.034+0000 7fc833fff640 1 -- 172.21.15.98:0/887609602 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc83c004260 con 0x7fc84c108830 2024-05-09T08:55:37.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.034+0000 7fc833fff640 1 -- 172.21.15.98:0/887609602 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c00a4f0 con 0x7fc84c108830 2024-05-09T08:55:37.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.042+0000 7fc85419d740 1 Processor -- start 2024-05-09T08:55:37.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.043+0000 7fc85419d740 1 -- start start 2024-05-09T08:55:37.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.043+0000 7fc85419d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063520 0x557436063900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.043+0000 7fc85419d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557436063e40 0x55743606dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.043+0000 7fc85419d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55743606e510 0x557436070900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.043+0000 7fc85419d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557435fc8c40 con 0x557436063e40 2024-05-09T08:55:37.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.043+0000 7fc85419d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557435f70580 con 0x55743606e510 2024-05-09T08:55:37.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.043+0000 7fc85419d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557435fc4c00 con 0x557436063520 2024-05-09T08:55:37.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.045+0000 7fc84b7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063520 0x557436063900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.045+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55743606e510 0x557436070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.046+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55743606e510 0x557436070900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45672/0 (socket says 172.21.15.98:45672) 2024-05-09T08:55:37.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.046+0000 7fc84a7fc640 1 -- 172.21.15.98:0/3430733926 learned_addr learned my addr 172.21.15.98:0/3430733926 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:37.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.046+0000 7fc833fff640 1 -- 172.21.15.98:0/887609602 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc83c00a690 con 0x7fc84c108830 2024-05-09T08:55:37.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.047+0000 7fc833fff640 1 --2- 172.21.15.98:0/887609602 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc8340a2e90 0x7fc8340a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.048+0000 7fc84a7fc640 1 -- 172.21.15.98:0/3430733926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063520 msgr2=0x557436063900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:37.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.049+0000 7fc84a7fc640 1 --2- 172.21.15.98:0/3430733926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063520 0x557436063900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.049+0000 7fc84a7fc640 1 -- 172.21.15.98:0/3430733926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557436063e40 msgr2=0x55743606dfd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:55:37.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.049+0000 7fc84a7fc640 1 --2- 172.21.15.98:0/3430733926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557436063e40 0x55743606dfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.049+0000 7fc84a7fc640 1 -- 172.21.15.98:0/3430733926 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557435cdc3f0 con 0x55743606e510 2024-05-09T08:55:37.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.049+0000 7fc833fff640 1 -- 172.21.15.98:0/887609602 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fc83c004410 con 0x7fc84c108830 2024-05-09T08:55:37.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.051+0000 7fc84b7fe640 1 --2- 172.21.15.98:0/887609602 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc8340a2e90 0x7fc8340a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.051+0000 7fc84a7fc640 1 --2- 172.21.15.98:0/3430733926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55743606e510 0x557436070900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc84000bb90 tx=0x7fc840005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3341013d14954bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:37.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.052+0000 7fc84b7fe640 1 --2- 172.21.15.98:0/887609602 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc8340a2e90 0x7fc8340a5350 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc844005fd0 tx=0x7fc844004a40 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:37.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.052+0000 7fc8317fa640 1 -- 172.21.15.98:0/3430733926 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840004230 con 0x55743606e510 2024-05-09T08:55:37.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.053+0000 7fc8317fa640 1 -- 172.21.15.98:0/3430733926 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8400043d0 con 0x55743606e510 2024-05-09T08:55:37.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.054+0000 7fc85419d740 1 -- 172.21.15.98:0/3430733926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55743606e510 msgr2=0x557436070900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:37.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.054+0000 7fc85419d740 1 --2- 172.21.15.98:0/3430733926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55743606e510 0x557436070900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc84000bb90 tx=0x7fc840005f90 comp rx=0 tx=0).stop 2024-05-09T08:55:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.056+0000 7fc85419d740 1 -- 172.21.15.98:0/3430733926 shutdown_connections 2024-05-09T08:55:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.056+0000 7fc85419d740 1 --2- 172.21.15.98:0/3430733926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55743606e510 0x557436070900 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.056+0000 7fc85419d740 1 --2- 172.21.15.98:0/3430733926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557436063e40 0x55743606dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.056+0000 7fc85419d740 1 --2- 172.21.15.98:0/3430733926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063520 0x557436063900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.056+0000 7fc85419d740 1 -- 172.21.15.98:0/3430733926 >> 172.21.15.98:0/3430733926 conn(0x55743605f840 msgr2=0x55743605fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.058+0000 7fc85419d740 1 -- 172.21.15.98:0/3430733926 shutdown_connections 2024-05-09T08:55:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.060+0000 7fc85419d740 1 -- 172.21.15.98:0/3430733926 wait complete. 2024-05-09T08:55:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.060+0000 7fc85419d740 1 -- start start 2024-05-09T08:55:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.061+0000 7fc85419d740 1 Processor -- start 2024-05-09T08:55:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.061+0000 7fc85419d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557436063520 0x55743606b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.061+0000 7fc85419d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063e40 0x55743606b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.061+0000 7fc85419d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55743606e510 0x55743606be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.061+0000 7fc85419d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557436071350 con 0x55743606e510 2024-05-09T08:55:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.061+0000 7fc85419d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557435f83c40 con 0x557436063520 2024-05-09T08:55:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.061+0000 7fc85419d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5574360714d0 con 0x557436063e40 2024-05-09T08:55:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.061+0000 7fc85419d740 10 client.0 _ll_register_callbacks cb 0x557435cdbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.062+0000 7fc84b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55743606e510 0x55743606be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.062+0000 7fc84b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55743606e510 0x55743606be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37328/0 (socket says 172.21.15.98:37328) 2024-05-09T08:55:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.062+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063e40 0x55743606b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.062+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063e40 0x55743606b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:46864/0 (socket says 172.21.15.98:46864) 2024-05-09T08:55:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.063+0000 7fc84a7fc640 1 -- 172.21.15.98:0/400680623 learned_addr learned my addr 172.21.15.98:0/400680623 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.063+0000 7fc84b7fe640 1 -- 172.21.15.98:0/400680623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063e40 msgr2=0x55743606b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.063+0000 7fc84b7fe640 1 --2- 172.21.15.98:0/400680623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063e40 0x55743606b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.063+0000 7fc84b7fe640 1 -- 172.21.15.98:0/400680623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557436063520 msgr2=0x55743606b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.064+0000 7fc84b7fe640 1 --2- 172.21.15.98:0/400680623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557436063520 0x55743606b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.064+0000 7fc84b7fe640 1 -- 172.21.15.98:0/400680623 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557435da67d0 con 0x55743606e510 2024-05-09T08:55:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.064+0000 7fc84a7fc640 1 --2- 172.21.15.98:0/400680623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063e40 0x55743606b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:55:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.068+0000 7fc84b7fe640 1 --2- 172.21.15.98:0/400680623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55743606e510 0x55743606be30 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc84400d680 tx=0x7fc84400f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99af741f352618cb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.069+0000 7fc82eff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:55:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.069+0000 7fc82eff5640 1 -- 172.21.15.98:0/400680623 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84401c650 con 0x55743606e510 2024-05-09T08:55:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.070+0000 7fc85419d740 1 -- 172.21.15.98:0/400680623 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8400043d0 con 0x55743606e510 2024-05-09T08:55:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.070+0000 7fc82eff5640 1 -- 172.21.15.98:0/400680623 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fc84401c7f0 con 0x55743606e510 2024-05-09T08:55:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.071+0000 7fc82eff5640 1 -- 172.21.15.98:0/400680623 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc844030050 con 0x55743606e510 2024-05-09T08:55:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.072+0000 7fc85419d740 10 client.17031 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.072+0000 7fc85419d740 1 -- 172.21.15.98:0/400680623 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55743607d5a0 con 0x55743606e510 2024-05-09T08:55:37.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.082+0000 7fc82eff5640 1 -- 172.21.15.98:0/400680623 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8400043d0 con 0x55743606e510 2024-05-09T08:55:37.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.082+0000 7fc82eff5640 1 -- 172.21.15.98:0/400680623 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fc84401ec40 con 0x55743606e510 2024-05-09T08:55:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.082+0000 7fc85419d740 10 client.17031 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.082+0000 7fc85419d740 10 client.17031 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.083+0000 7fc85419d740 20 client.17031 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.083+0000 7fc85419d740 1 --2- 172.21.15.98:0/400680623 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55743607ef30 0x5574360813d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.083+0000 7fc85419d740 4 client.17031 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.083+0000 7fc85419d740 1 -- 172.21.15.98:0/400680623 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557435fd6370 con 0x55743607ef30 2024-05-09T08:55:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.084+0000 7fc84affd640 1 --2- 172.21.15.98:0/400680623 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55743607ef30 0x5574360813d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.088+0000 7fc84affd640 1 --2- 172.21.15.98:0/400680623 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55743607ef30 0x5574360813d0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5a4cf6c57cb4260 server_cookie=fcc479899b00954e in_seq=0 out_seq=0 2024-05-09T08:55:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.089+0000 7fc82eff5640 10 client.17031 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.094+0000 7fc82eff5640 1 -- 172.21.15.98:0/400680623 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557435fd6370 con 0x55743607ef30 2024-05-09T08:55:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.095+0000 7fc82eff5640 10 client.17031 handle_command_reply: tid=0 2024-05-09T08:55:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.095+0000 7fc85419d740 1 client.17031 shutdown 2024-05-09T08:55:37.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.104+0000 7fc85419d740 1 -- 172.21.15.98:0/400680623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55743606e510 msgr2=0x55743606be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:37.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.104+0000 7fc85419d740 1 --2- 172.21.15.98:0/400680623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55743606e510 0x55743606be30 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc84400d680 tx=0x7fc84400f040 comp rx=0 tx=0).stop 2024-05-09T08:55:37.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.106+0000 7fc85419d740 1 -- 172.21.15.98:0/400680623 shutdown_connections 2024-05-09T08:55:37.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.106+0000 7fc85419d740 1 --2- 172.21.15.98:0/400680623 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55743607ef30 0x5574360813d0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.106+0000 7fc85419d740 1 --2- 172.21.15.98:0/400680623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55743606e510 0x55743606be30 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.106+0000 7fc85419d740 1 --2- 172.21.15.98:0/400680623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063e40 0x55743606b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.106+0000 7fc85419d740 1 --2- 172.21.15.98:0/400680623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557436063520 0x55743606b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.106+0000 7fc82eff5640 0 client.17031 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:37.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.106+0000 7fc85419d740 1 -- 172.21.15.98:0/400680623 >> 172.21.15.98:0/400680623 conn(0x55743605f840 msgr2=0x55743605e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:37.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.118+0000 7fc85419d740 1 -- 172.21.15.98:0/400680623 shutdown_connections 2024-05-09T08:55:37.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.121+0000 7fc85419d740 1 -- 172.21.15.98:0/400680623 wait complete. 2024-05-09T08:55:37.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.125+0000 7fc85419d740 20 client.17031 trim_cache size 0 max 16384 2024-05-09T08:55:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.160+0000 7fc85419d740 1 Processor -- start 2024-05-09T08:55:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.161+0000 7fc85419d740 1 -- start start 2024-05-09T08:55:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.161+0000 7fc85419d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557436063520 0x55743607a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.161+0000 7fc85419d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063e40 0x557436081320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.161+0000 7fc85419d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55743606e510 0x55743606a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.161+0000 7fc85419d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc8380100c0 con 0x55743606e510 2024-05-09T08:55:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.161+0000 7fc85419d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc838011c00 con 0x557436063520 2024-05-09T08:55:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.161+0000 7fc85419d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc838010790 con 0x557436063e40 2024-05-09T08:55:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.161+0000 7fc84affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557436063520 0x55743607a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.161+0000 7fc84affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557436063520 0x55743607a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45684/0 (socket says 172.21.15.98:45684) 2024-05-09T08:55:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.161+0000 7fc84affd640 1 -- 172.21.15.98:0/303988232 learned_addr learned my addr 172.21.15.98:0/303988232 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.161+0000 7fc84b7fe640 1 --2- 172.21.15.98:0/303988232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55743606e510 0x55743606a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.162+0000 7fc84a7fc640 1 --2- 172.21.15.98:0/303988232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063e40 0x557436081320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.162+0000 7fc84affd640 1 -- 172.21.15.98:0/303988232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063e40 msgr2=0x557436081320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.162+0000 7fc84affd640 1 --2- 172.21.15.98:0/303988232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063e40 0x557436081320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.162+0000 7fc84affd640 1 -- 172.21.15.98:0/303988232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55743606e510 msgr2=0x55743606a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.162+0000 7fc84affd640 1 --2- 172.21.15.98:0/303988232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55743606e510 0x55743606a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.162+0000 7fc84affd640 1 -- 172.21.15.98:0/303988232 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557436063320 con 0x557436063520 2024-05-09T08:55:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.163+0000 7fc84affd640 1 --2- 172.21.15.98:0/303988232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557436063520 0x55743607a2c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc83c02a5b0 tx=0x7fc83c0bb840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13d0a4767e7883bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.163+0000 7fc830ff9640 1 -- 172.21.15.98:0/303988232 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0b7be0 con 0x557436063520 2024-05-09T08:55:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.163+0000 7fc830ff9640 1 -- 172.21.15.98:0/303988232 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc83c042e50 con 0x557436063520 2024-05-09T08:55:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.163+0000 7fc830ff9640 1 -- 172.21.15.98:0/303988232 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0b7d80 con 0x557436063520 2024-05-09T08:55:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.166+0000 7fc85419d740 1 -- 172.21.15.98:0/303988232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557436063520 msgr2=0x55743607a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.166+0000 7fc85419d740 1 --2- 172.21.15.98:0/303988232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557436063520 0x55743607a2c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc83c02a5b0 tx=0x7fc83c0bb840 comp rx=0 tx=0).stop 2024-05-09T08:55:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.169+0000 7fc85419d740 1 -- 172.21.15.98:0/303988232 shutdown_connections 2024-05-09T08:55:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.169+0000 7fc85419d740 1 --2- 172.21.15.98:0/303988232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55743606e510 0x55743606a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.169+0000 7fc85419d740 1 --2- 172.21.15.98:0/303988232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063e40 0x557436081320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.169+0000 7fc85419d740 1 --2- 172.21.15.98:0/303988232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557436063520 0x55743607a2c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.170+0000 7fc85419d740 1 -- 172.21.15.98:0/303988232 >> 172.21.15.98:0/303988232 conn(0x55743605f840 msgr2=0x55743605fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.170+0000 7fc85419d740 1 -- 172.21.15.98:0/303988232 shutdown_connections 2024-05-09T08:55:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.170+0000 7fc85419d740 1 -- 172.21.15.98:0/303988232 wait complete. 2024-05-09T08:55:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.170+0000 7fc85419d740 1 -- start start 2024-05-09T08:55:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.171+0000 7fc85419d740 1 Processor -- start 2024-05-09T08:55:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.171+0000 7fc85419d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063520 0x557436076d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.171+0000 7fc85419d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557436063e40 0x557436067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.171+0000 7fc85419d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55743606e510 0x5574360685b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.171+0000 7fc85419d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc838011530 con 0x557436063e40 2024-05-09T08:55:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.171+0000 7fc85419d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc838010e60 con 0x55743606e510 2024-05-09T08:55:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.171+0000 7fc85419d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc83800e0a0 con 0x557436063520 2024-05-09T08:55:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.171+0000 7fc85419d740 10 client.0 _ll_register_callbacks cb 0x7fc84400d480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.171+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557436063e40 0x557436067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.172+0000 7fc84affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063520 0x557436076d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.172+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557436063e40 0x557436067ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37344/0 (socket says 172.21.15.98:37344) 2024-05-09T08:55:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.172+0000 7fc84a7fc640 1 -- 172.21.15.98:0/2541354936 learned_addr learned my addr 172.21.15.98:0/2541354936 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.172+0000 7fc84b7fe640 1 --2- 172.21.15.98:0/2541354936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55743606e510 0x5574360685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.172+0000 7fc84b7fe640 1 -- 172.21.15.98:0/2541354936 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063520 msgr2=0x557436076d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.172+0000 7fc84b7fe640 1 --2- 172.21.15.98:0/2541354936 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063520 0x557436076d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.172+0000 7fc84b7fe640 1 -- 172.21.15.98:0/2541354936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557436063e40 msgr2=0x557436067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.172+0000 7fc84b7fe640 1 --2- 172.21.15.98:0/2541354936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557436063e40 0x557436067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.172+0000 7fc84b7fe640 1 -- 172.21.15.98:0/2541354936 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557436074800 con 0x55743606e510 2024-05-09T08:55:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.172+0000 7fc84b7fe640 1 --2- 172.21.15.98:0/2541354936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55743606e510 0x5574360685b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc84400bcc0 tx=0x7fc844020d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=687125a516fa0412 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.173+0000 7fc82eff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:55:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.173+0000 7fc82eff5640 1 -- 172.21.15.98:0/2541354936 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844036680 con 0x55743606e510 2024-05-09T08:55:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.174+0000 7fc82eff5640 1 -- 172.21.15.98:0/2541354936 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fc844036820 con 0x55743606e510 2024-05-09T08:55:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.174+0000 7fc85419d740 1 -- 172.21.15.98:0/2541354936 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc83c042e50 con 0x55743606e510 2024-05-09T08:55:37.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.175+0000 7fc82eff5640 1 -- 172.21.15.98:0/2541354936 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc844046050 con 0x55743606e510 2024-05-09T08:55:37.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.175+0000 7fc85419d740 10 client.26330 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:37.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.175+0000 7fc85419d740 1 -- 172.21.15.98:0/2541354936 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5574360e56a0 con 0x55743606e510 2024-05-09T08:55:37.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.176+0000 7fc82eff5640 1 -- 172.21.15.98:0/2541354936 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc83c042e50 con 0x55743606e510 2024-05-09T08:55:37.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.176+0000 7fc82eff5640 1 -- 172.21.15.98:0/2541354936 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fc844033040 con 0x55743606e510 2024-05-09T08:55:37.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.176+0000 7fc85419d740 10 client.26330 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:37.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.176+0000 7fc85419d740 10 client.26330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:37.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.176+0000 7fc85419d740 20 client.26330 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:37.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.176+0000 7fc85419d740 1 --2- 172.21.15.98:0/2541354936 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5574360e5df0 0x5574360e8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:37.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.176+0000 7fc85419d740 4 client.26330 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:55:37.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.176+0000 7fc85419d740 1 -- 172.21.15.98:0/2541354936 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55743605f690 con 0x5574360e5df0 2024-05-09T08:55:37.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.176+0000 7fc84affd640 1 --2- 172.21.15.98:0/2541354936 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5574360e5df0 0x5574360e8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:37.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.178+0000 7fc84affd640 1 --2- 172.21.15.98:0/2541354936 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5574360e5df0 0x5574360e8210 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a2e8695c052ba67 server_cookie=f8ca06034b7d58d8 in_seq=0 out_seq=0 2024-05-09T08:55:37.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.179+0000 7fc82eff5640 10 client.26330 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:37.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.180+0000 7fc82eff5640 1 -- 172.21.15.98:0/2541354936 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55743605f690 con 0x5574360e5df0 2024-05-09T08:55:37.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.180+0000 7fc82eff5640 10 client.26330 handle_command_reply: tid=0 2024-05-09T08:55:37.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.180+0000 7fc85419d740 1 client.26330 shutdown 2024-05-09T08:55:37.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.183+0000 7fc85419d740 1 -- 172.21.15.98:0/2541354936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55743606e510 msgr2=0x5574360685b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:37.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.183+0000 7fc85419d740 1 --2- 172.21.15.98:0/2541354936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55743606e510 0x5574360685b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc84400bcc0 tx=0x7fc844020d80 comp rx=0 tx=0).stop 2024-05-09T08:55:37.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.183+0000 7fc85419d740 1 -- 172.21.15.98:0/2541354936 shutdown_connections 2024-05-09T08:55:37.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.183+0000 7fc85419d740 1 --2- 172.21.15.98:0/2541354936 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5574360e5df0 0x5574360e8210 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.183+0000 7fc85419d740 1 --2- 172.21.15.98:0/2541354936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55743606e510 0x5574360685b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.183+0000 7fc85419d740 1 --2- 172.21.15.98:0/2541354936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557436063e40 0x557436067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.183+0000 7fc85419d740 1 --2- 172.21.15.98:0/2541354936 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557436063520 0x557436076d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.183+0000 7fc85419d740 1 -- 172.21.15.98:0/2541354936 >> 172.21.15.98:0/2541354936 conn(0x55743605f840 msgr2=0x55743605e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:37.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.183+0000 7fc82eff5640 0 client.26330 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:37.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.184+0000 7fc85419d740 1 -- 172.21.15.98:0/2541354936 shutdown_connections 2024-05-09T08:55:37.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.184+0000 7fc85419d740 1 -- 172.21.15.98:0/2541354936 wait complete. 2024-05-09T08:55:37.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.185+0000 7fc85419d740 20 client.26330 trim_cache size 0 max 16384 2024-05-09T08:55:37.187 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:55:37.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.188+0000 7fc8513b0640 1 -- 172.21.15.98:0/887609602 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc8340a2e90 msgr2=0x7fc8340a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:37.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.188+0000 7fc8513b0640 1 --2- 172.21.15.98:0/887609602 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc8340a2e90 0x7fc8340a5350 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc844005fd0 tx=0x7fc844004a40 comp rx=0 tx=0).stop 2024-05-09T08:55:37.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.188+0000 7fc8513b0640 1 -- 172.21.15.98:0/887609602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c108830 msgr2=0x7fc84c11e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:37.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.188+0000 7fc8513b0640 1 --2- 172.21.15.98:0/887609602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c108830 0x7fc84c11e1a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc83c000c00 tx=0x7fc83c02aaf0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.188+0000 7fc8513b0640 1 -- 172.21.15.98:0/887609602 shutdown_connections 2024-05-09T08:55:37.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.188+0000 7fc8513b0640 1 --2- 172.21.15.98:0/887609602 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc8340a2e90 0x7fc8340a5350 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.188+0000 7fc8513b0640 1 --2- 172.21.15.98:0/887609602 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc84c10e2d0 0x7fc84c118260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.188+0000 7fc8513b0640 1 --2- 172.21.15.98:0/887609602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc84c1091f0 0x7fc84c11e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.188+0000 7fc8513b0640 1 --2- 172.21.15.98:0/887609602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c108830 0x7fc84c11e1a0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:37.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.188+0000 7fc8513b0640 1 -- 172.21.15.98:0/887609602 >> 172.21.15.98:0/887609602 conn(0x7fc84c06cc40 msgr2=0x7fc84c10ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:37.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.191+0000 7fc8513b0640 1 -- 172.21.15.98:0/887609602 shutdown_connections 2024-05-09T08:55:37.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:37.191+0000 7fc8513b0640 1 -- 172.21.15.98:0/887609602 wait complete. 2024-05-09T08:55:38.235 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:55:38.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:55:38.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:55:38.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.550+0000 7f12600cd640 1 -- 172.21.15.98:0/2079728596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1258109360 msgr2=0x7f1258114440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:38.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.550+0000 7f12600cd640 1 --2- 172.21.15.98:0/2079728596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1258109360 0x7f1258114440 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f1254009be0 tx=0x7f125402a6f0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.552+0000 7f12600cd640 1 -- 172.21.15.98:0/2079728596 shutdown_connections 2024-05-09T08:55:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.552+0000 7f12600cd640 1 --2- 172.21.15.98:0/2079728596 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1258114980 0x7f1258116d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.553+0000 7f12600cd640 1 --2- 172.21.15.98:0/2079728596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1258109360 0x7f1258114440 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.553+0000 7f12600cd640 1 --2- 172.21.15.98:0/2079728596 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1258108a40 0x7f1258108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.553+0000 7f12600cd640 1 -- 172.21.15.98:0/2079728596 >> 172.21.15.98:0/2079728596 conn(0x7f125806c940 msgr2=0x7f125806cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.553+0000 7f12600cd640 1 -- 172.21.15.98:0/2079728596 shutdown_connections 2024-05-09T08:55:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.553+0000 7f12600cd640 1 -- 172.21.15.98:0/2079728596 wait complete. 2024-05-09T08:55:38.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.554+0000 7f12600cd640 1 Processor -- start 2024-05-09T08:55:38.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.556+0000 7f12600cd640 1 -- start start 2024-05-09T08:55:38.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f12600cd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1258108a40 0x7f125810a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f12600cd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1258109360 0x7f125810a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f12600cd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1258114980 0x7f125810ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f12600cd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f125811a490 con 0x7f1258114980 2024-05-09T08:55:38.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f12600cd640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f125811a310 con 0x7f1258109360 2024-05-09T08:55:38.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f12600cd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f125811a610 con 0x7f1258108a40 2024-05-09T08:55:38.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f125d668640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1258109360 0x7f125810a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f125e66a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1258114980 0x7f125810ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f125d668640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1258109360 0x7f125810a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45704/0 (socket says 172.21.15.98:45704) 2024-05-09T08:55:38.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f125d668640 1 -- 172.21.15.98:0/2488216090 learned_addr learned my addr 172.21.15.98:0/2488216090 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:38.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f125e66a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1258114980 0x7f125810ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37362/0 (socket says 172.21.15.98:37362) 2024-05-09T08:55:38.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f125d668640 1 -- 172.21.15.98:0/2488216090 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1258108a40 msgr2=0x7f125810a2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:55:38.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f125d668640 1 --2- 172.21.15.98:0/2488216090 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1258108a40 0x7f125810a2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f125d668640 1 -- 172.21.15.98:0/2488216090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1258114980 msgr2=0x7f125810ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:38.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f125d668640 1 --2- 172.21.15.98:0/2488216090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1258114980 0x7f125810ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.557+0000 7f125d668640 1 -- 172.21.15.98:0/2488216090 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12581aef90 con 0x7f1258109360 2024-05-09T08:55:38.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.561+0000 7f125d668640 1 --2- 172.21.15.98:0/2488216090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1258109360 0x7f125810a7e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1254034040 tx=0x7f1254030710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:38.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.563+0000 7f124effd640 1 -- 172.21.15.98:0/2488216090 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254041020 con 0x7f1258109360 2024-05-09T08:55:38.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.563+0000 7f124effd640 1 -- 172.21.15.98:0/2488216090 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1254031920 con 0x7f1258109360 2024-05-09T08:55:38.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.563+0000 7f124effd640 1 -- 172.21.15.98:0/2488216090 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254036040 con 0x7f1258109360 2024-05-09T08:55:38.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.563+0000 7f12600cd640 1 -- 172.21.15.98:0/2488216090 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12581af220 con 0x7f1258109360 2024-05-09T08:55:38.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.563+0000 7f12600cd640 1 -- 172.21.15.98:0/2488216090 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12581af6c0 con 0x7f1258109360 2024-05-09T08:55:38.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.569+0000 7f1262ddc740 1 Processor -- start 2024-05-09T08:55:38.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.569+0000 7f1262ddc740 1 -- start start 2024-05-09T08:55:38.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.570+0000 7f1262ddc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d120b0 0x55ec45d103f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.570+0000 7f1262ddc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d109c0 0x55ec45d10e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.570+0000 7f1262ddc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d1db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.570+0000 7f1262ddc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ec45c75990 con 0x55ec45d109c0 2024-05-09T08:55:38.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.570+0000 7f1262ddc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ec45c1d5b0 con 0x55ec45d1b740 2024-05-09T08:55:38.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.570+0000 7f1262ddc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ec45c71950 con 0x55ec45d120b0 2024-05-09T08:55:38.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.572+0000 7f124effd640 1 -- 172.21.15.98:0/2488216090 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1254044020 con 0x7f1258109360 2024-05-09T08:55:38.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.574+0000 7f124effd640 1 --2- 172.21.15.98:0/2488216090 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f12440a2f00 0x7f12440a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.576+0000 7f125e66a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d120b0 0x55ec45d103f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.576+0000 7f124effd640 1 -- 172.21.15.98:0/2488216090 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f12540f00f0 con 0x7f1258109360 2024-05-09T08:55:38.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.576+0000 7f125d668640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d1db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.578+0000 7f125d668640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d1db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45714/0 (socket says 172.21.15.98:45714) 2024-05-09T08:55:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.578+0000 7f125d668640 1 -- 172.21.15.98:0/3696512064 learned_addr learned my addr 172.21.15.98:0/3696512064 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:38.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.578+0000 7f125de69640 1 --2- 172.21.15.98:0/3696512064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d109c0 0x55ec45d10e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.579+0000 7f125e66a640 1 --2- 172.21.15.98:0/2488216090 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f12440a2f00 0x7f12440a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.580+0000 7f125d668640 1 -- 172.21.15.98:0/3696512064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d120b0 msgr2=0x55ec45d103f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:38.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.580+0000 7f125d668640 1 --2- 172.21.15.98:0/3696512064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d120b0 0x55ec45d103f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.580+0000 7f125d668640 1 -- 172.21.15.98:0/3696512064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d109c0 msgr2=0x55ec45d10e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:38.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.580+0000 7f125d668640 1 --2- 172.21.15.98:0/3696512064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d109c0 0x55ec45d10e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.580+0000 7f125d668640 1 -- 172.21.15.98:0/3696512064 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec45a517e0 con 0x55ec45d1b740 2024-05-09T08:55:38.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.582+0000 7f125e66a640 1 --2- 172.21.15.98:0/2488216090 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f12440a2f00 0x7f12440a53c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f124800dfd0 tx=0x7f1248004700 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:38.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.582+0000 7f125d668640 1 --2- 172.21.15.98:0/3696512064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d1db30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f125402acb0 tx=0x7f12540bbdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a879f5f79c217a0c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:38.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.585+0000 7f1243fff640 1 -- 172.21.15.98:0/3696512064 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12540bc1f0 con 0x55ec45d1b740 2024-05-09T08:55:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.585+0000 7f1243fff640 1 -- 172.21.15.98:0/3696512064 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12540c05a0 con 0x55ec45d1b740 2024-05-09T08:55:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.585+0000 7f1243fff640 1 -- 172.21.15.98:0/3696512064 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12540ed6a0 con 0x55ec45d1b740 2024-05-09T08:55:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.587+0000 7f1262ddc740 1 -- 172.21.15.98:0/3696512064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 msgr2=0x55ec45d1db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.587+0000 7f1262ddc740 1 --2- 172.21.15.98:0/3696512064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d1db30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f125402acb0 tx=0x7f12540bbdf0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.587+0000 7f1262ddc740 1 -- 172.21.15.98:0/3696512064 shutdown_connections 2024-05-09T08:55:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.587+0000 7f1262ddc740 1 --2- 172.21.15.98:0/3696512064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d1db30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.587+0000 7f1262ddc740 1 --2- 172.21.15.98:0/3696512064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d109c0 0x55ec45d10e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.587+0000 7f1262ddc740 1 --2- 172.21.15.98:0/3696512064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d120b0 0x55ec45d103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.587+0000 7f1262ddc740 1 -- 172.21.15.98:0/3696512064 >> 172.21.15.98:0/3696512064 conn(0x55ec45cd90a0 msgr2=0x55ec45d0cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.589+0000 7f1262ddc740 1 -- 172.21.15.98:0/3696512064 shutdown_connections 2024-05-09T08:55:38.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:38 smithi098 ceph-mon[30092]: pgmap v154: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 827 MiB/s wr, 1.37k op/s 2024-05-09T08:55:38.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.589+0000 7f1262ddc740 1 -- 172.21.15.98:0/3696512064 wait complete. 2024-05-09T08:55:38.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.590+0000 7f1262ddc740 1 -- start start 2024-05-09T08:55:38.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.590+0000 7f1262ddc740 1 Processor -- start 2024-05-09T08:55:38.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.590+0000 7f1262ddc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d109c0 0x55ec45d185c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.590+0000 7f1262ddc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 0x55ec45d18b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.590+0000 7f1262ddc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d19040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.590+0000 7f1262ddc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ec45c1d5b0 con 0x55ec45d120b0 2024-05-09T08:55:38.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.590+0000 7f1262ddc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ec45c30b40 con 0x55ec45d1b740 2024-05-09T08:55:38.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.590+0000 7f1262ddc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ec45d26ba0 con 0x55ec45d109c0 2024-05-09T08:55:38.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.590+0000 7f1262ddc740 10 client.0 _ll_register_callbacks cb 0x55ec45c250f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:38.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.592+0000 7f125d668640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 0x55ec45d18b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.592+0000 7f125d668640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 0x55ec45d18b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37380/0 (socket says 172.21.15.98:37380) 2024-05-09T08:55:38.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.592+0000 7f125d668640 1 -- 172.21.15.98:0/1824784118 learned_addr learned my addr 172.21.15.98:0/1824784118 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:38.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.593+0000 7f125de69640 1 --2- 172.21.15.98:0/1824784118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d109c0 0x55ec45d185c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.593+0000 7f125e66a640 1 --2- 172.21.15.98:0/1824784118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d19040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.594+0000 7f125e66a640 1 -- 172.21.15.98:0/1824784118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d109c0 msgr2=0x55ec45d185c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:38.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.594+0000 7f125e66a640 1 --2- 172.21.15.98:0/1824784118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d109c0 0x55ec45d185c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.594+0000 7f125e66a640 1 -- 172.21.15.98:0/1824784118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 msgr2=0x55ec45d18b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:38.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.594+0000 7f125e66a640 1 --2- 172.21.15.98:0/1824784118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 0x55ec45d18b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.594+0000 7f125e66a640 1 -- 172.21.15.98:0/1824784118 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec45ab1f70 con 0x55ec45d1b740 2024-05-09T08:55:38.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.595+0000 7f125e66a640 1 --2- 172.21.15.98:0/1824784118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d19040 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1248019da0 tx=0x7f1248004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf524124651f537d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:38.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.596+0000 7f12417fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:55:38.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.596+0000 7f12417fa640 1 -- 172.21.15.98:0/1824784118 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248027060 con 0x55ec45d1b740 2024-05-09T08:55:38.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.596+0000 7f1262ddc740 1 -- 172.21.15.98:0/1824784118 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12540c05a0 con 0x55ec45d1b740 2024-05-09T08:55:38.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.596+0000 7f12417fa640 1 -- 172.21.15.98:0/1824784118 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f1248017c10 con 0x55ec45d1b740 2024-05-09T08:55:38.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.597+0000 7f12417fa640 1 -- 172.21.15.98:0/1824784118 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f124802e050 con 0x55ec45d1b740 2024-05-09T08:55:38.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.597+0000 7f1262ddc740 10 client.26342 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:38.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.597+0000 7f1262ddc740 1 -- 172.21.15.98:0/1824784118 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ec45949170 con 0x55ec45d1b740 2024-05-09T08:55:38.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.598+0000 7f12417fa640 1 -- 172.21.15.98:0/1824784118 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12540c05a0 con 0x55ec45d1b740 2024-05-09T08:55:38.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.599+0000 7f12417fa640 1 -- 172.21.15.98:0/1824784118 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f1248021070 con 0x55ec45d1b740 2024-05-09T08:55:38.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.599+0000 7f1262ddc740 10 client.26342 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:38.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.599+0000 7f1262ddc740 10 client.26342 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:38.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.599+0000 7f1262ddc740 20 client.26342 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:38.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.599+0000 7f1262ddc740 1 --2- 172.21.15.98:0/1824784118 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec45d34090 0x55ec45d364d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.599+0000 7f1262ddc740 4 client.26342 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:38.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.599+0000 7f1262ddc740 1 -- 172.21.15.98:0/1824784118 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec45c830c0 con 0x55ec45d34090 2024-05-09T08:55:38.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.600+0000 7f125de69640 1 --2- 172.21.15.98:0/1824784118 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec45d34090 0x55ec45d364d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.620+0000 7f125de69640 1 --2- 172.21.15.98:0/1824784118 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec45d34090 0x55ec45d364d0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0398e36246c23af server_cookie=112c74cb369df6f2 in_seq=0 out_seq=0 2024-05-09T08:55:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.620+0000 7f12417fa640 10 client.26342 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.629+0000 7f12417fa640 1 -- 172.21.15.98:0/1824784118 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ec45c830c0 con 0x55ec45d34090 2024-05-09T08:55:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.629+0000 7f12417fa640 10 client.26342 handle_command_reply: tid=0 2024-05-09T08:55:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.629+0000 7f1262ddc740 1 client.26342 shutdown 2024-05-09T08:55:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.633+0000 7f1262ddc740 1 -- 172.21.15.98:0/1824784118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 msgr2=0x55ec45d19040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.633+0000 7f1262ddc740 1 --2- 172.21.15.98:0/1824784118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d19040 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1248019da0 tx=0x7f1248004770 comp rx=0 tx=0).stop 2024-05-09T08:55:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.633+0000 7f1262ddc740 1 -- 172.21.15.98:0/1824784118 shutdown_connections 2024-05-09T08:55:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.633+0000 7f1262ddc740 1 --2- 172.21.15.98:0/1824784118 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec45d34090 0x55ec45d364d0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.633+0000 7f1262ddc740 1 --2- 172.21.15.98:0/1824784118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d19040 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.633+0000 7f1262ddc740 1 --2- 172.21.15.98:0/1824784118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 0x55ec45d18b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.633+0000 7f1262ddc740 1 --2- 172.21.15.98:0/1824784118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d109c0 0x55ec45d185c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.633+0000 7f1262ddc740 1 -- 172.21.15.98:0/1824784118 >> 172.21.15.98:0/1824784118 conn(0x55ec45cd90a0 msgr2=0x55ec45d12d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.634+0000 7f12417fa640 0 client.26342 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.634+0000 7f1262ddc740 1 -- 172.21.15.98:0/1824784118 shutdown_connections 2024-05-09T08:55:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.635+0000 7f1262ddc740 1 -- 172.21.15.98:0/1824784118 wait complete. 2024-05-09T08:55:38.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.638+0000 7f1262ddc740 20 client.26342 trim_cache size 0 max 16384 2024-05-09T08:55:38.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:38 smithi142 ceph-mon[34625]: pgmap v154: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 827 MiB/s wr, 1.37k op/s 2024-05-09T08:55:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.724+0000 7f1262ddc740 1 Processor -- start 2024-05-09T08:55:38.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.728+0000 7f1262ddc740 1 -- start start 2024-05-09T08:55:38.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.729+0000 7f1262ddc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d109c0 0x55ec45d2f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.729+0000 7f1262ddc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 0x55ec45d18e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.730+0000 7f1262ddc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d1b740 0x55ec45d0bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.730+0000 7f1262ddc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f123c00ff60 con 0x55ec45d120b0 2024-05-09T08:55:38.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.730+0000 7f1262ddc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f123c011aa0 con 0x55ec45d109c0 2024-05-09T08:55:38.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.730+0000 7f1262ddc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f123c010630 con 0x55ec45d1b740 2024-05-09T08:55:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.733+0000 7f125d668640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 0x55ec45d18e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.733+0000 7f125d668640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 0x55ec45d18e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37392/0 (socket says 172.21.15.98:37392) 2024-05-09T08:55:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.733+0000 7f125d668640 1 -- 172.21.15.98:0/2423504622 learned_addr learned my addr 172.21.15.98:0/2423504622 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.734+0000 7f125e66a640 1 --2- 172.21.15.98:0/2423504622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d1b740 0x55ec45d0bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.734+0000 7f125de69640 1 --2- 172.21.15.98:0/2423504622 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d109c0 0x55ec45d2f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:38 smithi105 ceph-mon[34498]: pgmap v154: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 827 MiB/s wr, 1.37k op/s 2024-05-09T08:55:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.737+0000 7f125de69640 1 -- 172.21.15.98:0/2423504622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d1b740 msgr2=0x55ec45d0bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.737+0000 7f125de69640 1 --2- 172.21.15.98:0/2423504622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d1b740 0x55ec45d0bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.737+0000 7f125de69640 1 -- 172.21.15.98:0/2423504622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 msgr2=0x55ec45d18e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.737+0000 7f125de69640 1 --2- 172.21.15.98:0/2423504622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 0x55ec45d18e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.737+0000 7f125de69640 1 -- 172.21.15.98:0/2423504622 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec45d327f0 con 0x55ec45d109c0 2024-05-09T08:55:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.737+0000 7f125de69640 1 --2- 172.21.15.98:0/2423504622 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d109c0 0x55ec45d2f8e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x55ec45d36d20 tx=0x7f125000d3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=793d7450fe4ad705 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:38.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.745+0000 7f12437fe640 1 -- 172.21.15.98:0/2423504622 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125000de50 con 0x55ec45d109c0 2024-05-09T08:55:38.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.745+0000 7f12437fe640 1 -- 172.21.15.98:0/2423504622 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f125001a330 con 0x55ec45d109c0 2024-05-09T08:55:38.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.745+0000 7f12437fe640 1 -- 172.21.15.98:0/2423504622 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12500052f0 con 0x55ec45d109c0 2024-05-09T08:55:38.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.747+0000 7f1262ddc740 1 -- 172.21.15.98:0/2423504622 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d109c0 msgr2=0x55ec45d2f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:38.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.747+0000 7f1262ddc740 1 --2- 172.21.15.98:0/2423504622 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d109c0 0x55ec45d2f8e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x55ec45d36d20 tx=0x7f125000d3c0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.747+0000 7f1262ddc740 1 -- 172.21.15.98:0/2423504622 shutdown_connections 2024-05-09T08:55:38.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.747+0000 7f1262ddc740 1 --2- 172.21.15.98:0/2423504622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d1b740 0x55ec45d0bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.747+0000 7f1262ddc740 1 --2- 172.21.15.98:0/2423504622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 0x55ec45d18e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.747+0000 7f1262ddc740 1 --2- 172.21.15.98:0/2423504622 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d109c0 0x55ec45d2f8e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.748+0000 7f1262ddc740 1 -- 172.21.15.98:0/2423504622 >> 172.21.15.98:0/2423504622 conn(0x55ec45cd90a0 msgr2=0x55ec45d0b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:38.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.748+0000 7f1262ddc740 1 -- 172.21.15.98:0/2423504622 shutdown_connections 2024-05-09T08:55:38.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.748+0000 7f1262ddc740 1 -- 172.21.15.98:0/2423504622 wait complete. 2024-05-09T08:55:38.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.749+0000 7f1262ddc740 1 -- start start 2024-05-09T08:55:38.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.750+0000 7f1262ddc740 1 Processor -- start 2024-05-09T08:55:38.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.751+0000 7f1262ddc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d109c0 0x55ec45d27990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.751+0000 7f1262ddc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 0x55ec45d27ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.751+0000 7f1262ddc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d28410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.751+0000 7f1262ddc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f123c0113d0 con 0x55ec45d120b0 2024-05-09T08:55:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.751+0000 7f125de69640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d109c0 0x55ec45d27990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.751+0000 7f125de69640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d109c0 0x55ec45d27990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:46908/0 (socket says 172.21.15.98:46908) 2024-05-09T08:55:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.751+0000 7f125de69640 1 -- 172.21.15.98:0/2510381987 learned_addr learned my addr 172.21.15.98:0/2510381987 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.753+0000 7f125d668640 1 --2- 172.21.15.98:0/2510381987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 0x55ec45d27ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.754+0000 7f1262ddc740 1 -- 172.21.15.98:0/2510381987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f123c010d00 con 0x55ec45d1b740 2024-05-09T08:55:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.754+0000 7f125e66a640 1 --2- 172.21.15.98:0/2510381987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d28410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.754+0000 7f1262ddc740 1 -- 172.21.15.98:0/2510381987 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f123c00df40 con 0x55ec45d109c0 2024-05-09T08:55:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.754+0000 7f1262ddc740 10 client.0 _ll_register_callbacks cb 0x7f1248019a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:38.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.758+0000 7f125de69640 1 -- 172.21.15.98:0/2510381987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 msgr2=0x55ec45d28410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:38.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.758+0000 7f125de69640 1 --2- 172.21.15.98:0/2510381987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d28410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.758+0000 7f125de69640 1 -- 172.21.15.98:0/2510381987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 msgr2=0x55ec45d27ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:38.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.758+0000 7f125de69640 1 --2- 172.21.15.98:0/2510381987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 0x55ec45d27ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:38.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.758+0000 7f125de69640 1 -- 172.21.15.98:0/2510381987 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec45d99800 con 0x55ec45d109c0 2024-05-09T08:55:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.761+0000 7f125de69640 1 --2- 172.21.15.98:0/2510381987 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d109c0 0x55ec45d27990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f125000dfb0 tx=0x7f1250003ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c392ebc3b7208a1c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.761+0000 7f12417fa640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:55:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.762+0000 7f12417fa640 1 -- 172.21.15.98:0/2510381987 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1250003e60 con 0x55ec45d109c0 2024-05-09T08:55:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.762+0000 7f12417fa640 1 -- 172.21.15.98:0/2510381987 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f1250005090 con 0x55ec45d109c0 2024-05-09T08:55:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.762+0000 7f1262ddc740 1 -- 172.21.15.98:0/2510381987 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f125001a330 con 0x55ec45d109c0 2024-05-09T08:55:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.764+0000 7f12417fa640 1 -- 172.21.15.98:0/2510381987 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1250034050 con 0x55ec45d109c0 2024-05-09T08:55:38.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.767+0000 7f1262ddc740 10 client.26302 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:38.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.767+0000 7f1262ddc740 1 -- 172.21.15.98:0/2510381987 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ec45d9aac0 con 0x55ec45d109c0 2024-05-09T08:55:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.769+0000 7f12417fa640 1 -- 172.21.15.98:0/2510381987 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f125001a330 con 0x55ec45d109c0 2024-05-09T08:55:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.769+0000 7f12417fa640 1 -- 172.21.15.98:0/2510381987 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f1250004800 con 0x55ec45d109c0 2024-05-09T08:55:38.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.770+0000 7f1262ddc740 10 client.26302 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.770+0000 7f1262ddc740 10 client.26302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.770+0000 7f1262ddc740 20 client.26302 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.770+0000 7f1262ddc740 1 --2- 172.21.15.98:0/2510381987 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec45d9b090 0x55ec45d9d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.770+0000 7f1262ddc740 4 client.26302 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:55:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.770+0000 7f1262ddc740 1 -- 172.21.15.98:0/2510381987 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ec45c7d410 con 0x55ec45d9b090 2024-05-09T08:55:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.771+0000 7f125d668640 1 --2- 172.21.15.98:0/2510381987 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec45d9b090 0x55ec45d9d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:38.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.780+0000 7f125d668640 1 --2- 172.21.15.98:0/2510381987 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec45d9b090 0x55ec45d9d4b0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa30cf464c30d0a6 server_cookie=879a36d39ca46a46 in_seq=0 out_seq=0 2024-05-09T08:55:38.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:38.780+0000 7f12417fa640 10 client.26302 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:39.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.261+0000 7f12417fa640 1 -- 172.21.15.98:0/2510381987 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ec45c7d410 con 0x55ec45d9b090 2024-05-09T08:55:39.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.261+0000 7f12417fa640 10 client.26302 handle_command_reply: tid=0 2024-05-09T08:55:39.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.261+0000 7f1262ddc740 1 client.26302 shutdown 2024-05-09T08:55:39.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.262+0000 7f1262ddc740 1 -- 172.21.15.98:0/2510381987 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d109c0 msgr2=0x55ec45d27990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:39.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.262+0000 7f1262ddc740 1 --2- 172.21.15.98:0/2510381987 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d109c0 0x55ec45d27990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f125000dfb0 tx=0x7f1250003ac0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.262+0000 7f1262ddc740 1 -- 172.21.15.98:0/2510381987 shutdown_connections 2024-05-09T08:55:39.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.262+0000 7f1262ddc740 1 --2- 172.21.15.98:0/2510381987 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec45d9b090 0x55ec45d9d4b0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.262+0000 7f1262ddc740 1 --2- 172.21.15.98:0/2510381987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec45d1b740 0x55ec45d28410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.262+0000 7f1262ddc740 1 --2- 172.21.15.98:0/2510381987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec45d120b0 0x55ec45d27ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.262+0000 7f1262ddc740 1 --2- 172.21.15.98:0/2510381987 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec45d109c0 0x55ec45d27990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.262+0000 7f1262ddc740 1 -- 172.21.15.98:0/2510381987 >> 172.21.15.98:0/2510381987 conn(0x55ec45cd90a0 msgr2=0x55ec45d0c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:39.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.262+0000 7f12417fa640 0 client.26302 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:39.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.263+0000 7f1262ddc740 1 -- 172.21.15.98:0/2510381987 shutdown_connections 2024-05-09T08:55:39.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.263+0000 7f1262ddc740 1 -- 172.21.15.98:0/2510381987 wait complete. 2024-05-09T08:55:39.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.264+0000 7f1262ddc740 20 client.26302 trim_cache size 0 max 16384 2024-05-09T08:55:39.264 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:55:39.264 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:55:39.264 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "375447eb-be0b-4843-b1db-b3d92f4ba4f4", 2024-05-09T08:55:39.264 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:55:39.265 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:55:39.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.272+0000 7f12600cd640 1 -- 172.21.15.98:0/2488216090 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f12440a2f00 msgr2=0x7f12440a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.272+0000 7f12600cd640 1 --2- 172.21.15.98:0/2488216090 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f12440a2f00 0x7f12440a53c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f124800dfd0 tx=0x7f1248004700 comp rx=0 tx=0).stop 2024-05-09T08:55:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.272+0000 7f12600cd640 1 -- 172.21.15.98:0/2488216090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1258109360 msgr2=0x7f125810a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.272+0000 7f12600cd640 1 --2- 172.21.15.98:0/2488216090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1258109360 0x7f125810a7e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1254034040 tx=0x7f1254030710 comp rx=0 tx=0).stop 2024-05-09T08:55:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.276+0000 7f12600cd640 1 -- 172.21.15.98:0/2488216090 shutdown_connections 2024-05-09T08:55:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.276+0000 7f12600cd640 1 --2- 172.21.15.98:0/2488216090 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f12440a2f00 0x7f12440a53c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.276+0000 7f12600cd640 1 --2- 172.21.15.98:0/2488216090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1258114980 0x7f125810ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.276+0000 7f12600cd640 1 --2- 172.21.15.98:0/2488216090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1258109360 0x7f125810a7e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.276+0000 7f12600cd640 1 --2- 172.21.15.98:0/2488216090 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1258108a40 0x7f125810a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.276+0000 7f12600cd640 1 -- 172.21.15.98:0/2488216090 >> 172.21.15.98:0/2488216090 conn(0x7f125806c940 msgr2=0x7f1258071a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.281+0000 7f12600cd640 1 -- 172.21.15.98:0/2488216090 shutdown_connections 2024-05-09T08:55:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.281+0000 7f12600cd640 1 -- 172.21.15.98:0/2488216090 wait complete. 2024-05-09T08:55:39.335 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:55:39.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:55:39.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.643+0000 7f305de7f640 1 -- 172.21.15.98:0/4238802345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f305810fed0 msgr2=0x7f30581102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:39.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.643+0000 7f305de7f640 1 --2- 172.21.15.98:0/4238802345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f305810fed0 0x7f30581102b0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3048009900 tx=0x7f304802a4b0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.647+0000 7f305de7f640 1 -- 172.21.15.98:0/4238802345 shutdown_connections 2024-05-09T08:55:39.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.647+0000 7f305de7f640 1 --2- 172.21.15.98:0/4238802345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30581092b0 0x7f305811d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.647+0000 7f305de7f640 1 --2- 172.21.15.98:0/4238802345 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3058108830 0x7f3058108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.647+0000 7f305de7f640 1 --2- 172.21.15.98:0/4238802345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f305810fed0 0x7f30581102b0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.647+0000 7f305de7f640 1 -- 172.21.15.98:0/4238802345 >> 172.21.15.98:0/4238802345 conn(0x7f305806c7b0 msgr2=0x7f305806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:39.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.650+0000 7f305de7f640 1 -- 172.21.15.98:0/4238802345 shutdown_connections 2024-05-09T08:55:39.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.653+0000 7f305de7f640 1 -- 172.21.15.98:0/4238802345 wait complete. 2024-05-09T08:55:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.654+0000 7f305de7f640 1 Processor -- start 2024-05-09T08:55:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.658+0000 7f305de7f640 1 -- start start 2024-05-09T08:55:39.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.665+0000 7f305de7f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3058108830 0x7f305811cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.665+0000 7f305de7f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30581092b0 0x7f3058115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.665+0000 7f305de7f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f305810fed0 0x7f3058116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.665+0000 7f305de7f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3058120950 con 0x7f3058108830 2024-05-09T08:55:39.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.665+0000 7f305de7f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f30581207d0 con 0x7f30581092b0 2024-05-09T08:55:39.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.665+0000 7f305de7f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3058120ad0 con 0x7f305810fed0 2024-05-09T08:55:39.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.666+0000 7f305ce7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3058108830 0x7f305811cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:39.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.666+0000 7f305ce7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3058108830 0x7f305811cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37420/0 (socket says 172.21.15.98:37420) 2024-05-09T08:55:39.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.666+0000 7f305ce7d640 1 -- 172.21.15.98:0/126130676 learned_addr learned my addr 172.21.15.98:0/126130676 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:39.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.669+0000 7f305d67e640 1 --2- 172.21.15.98:0/126130676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f305810fed0 0x7f3058116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:39.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.672+0000 7f305ce7d640 1 -- 172.21.15.98:0/126130676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f305810fed0 msgr2=0x7f3058116180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:39.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.672+0000 7f305ce7d640 1 --2- 172.21.15.98:0/126130676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f305810fed0 0x7f3058116180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.672+0000 7f305ce7d640 1 -- 172.21.15.98:0/126130676 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30581092b0 msgr2=0x7f3058115c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:39.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.672+0000 7f305ce7d640 1 --2- 172.21.15.98:0/126130676 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30581092b0 0x7f3058115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.672+0000 7f305ce7d640 1 -- 172.21.15.98:0/126130676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30581167c0 con 0x7f3058108830 2024-05-09T08:55:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.674+0000 7f305ce7d640 1 --2- 172.21.15.98:0/126130676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3058108830 0x7f305811cbc0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3048031980 tx=0x7f3048005650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.676+0000 7f3055ffb640 1 -- 172.21.15.98:0/126130676 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304803f030 con 0x7f3058108830 2024-05-09T08:55:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.676+0000 7f305de7f640 1 -- 172.21.15.98:0/126130676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3058116a50 con 0x7f3058108830 2024-05-09T08:55:39.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.676+0000 7f3055ffb640 1 -- 172.21.15.98:0/126130676 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3048004db0 con 0x7f3058108830 2024-05-09T08:55:39.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.676+0000 7f3055ffb640 1 -- 172.21.15.98:0/126130676 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3048035070 con 0x7f3058108830 2024-05-09T08:55:39.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.678+0000 7f305de7f640 1 -- 172.21.15.98:0/126130676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f305811aa20 con 0x7f3058108830 2024-05-09T08:55:39.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.688+0000 7f3061df5740 1 Processor -- start 2024-05-09T08:55:39.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.691+0000 7f3055ffb640 1 -- 172.21.15.98:0/126130676 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3048042030 con 0x7f3058108830 2024-05-09T08:55:39.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.691+0000 7f3061df5740 1 -- start start 2024-05-09T08:55:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.691+0000 7f3061df5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779470 0x55ec61779850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.691+0000 7f3061df5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61779d90 0x55ec61783f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.691+0000 7f3061df5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61786850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.691+0000 7f3061df5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ec616ded00 con 0x55ec61784460 2024-05-09T08:55:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.691+0000 7f3061df5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ec616864c0 con 0x55ec61779d90 2024-05-09T08:55:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.691+0000 7f3061df5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ec616dacc0 con 0x55ec61779470 2024-05-09T08:55:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.693+0000 7f305d67e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779470 0x55ec61779850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.693+0000 7f305d67e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779470 0x55ec61779850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:46930/0 (socket says 172.21.15.98:46930) 2024-05-09T08:55:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.693+0000 7f305d67e640 1 -- 172.21.15.98:0/2135678451 learned_addr learned my addr 172.21.15.98:0/2135678451 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:39.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.694+0000 7f3055ffb640 1 --2- 172.21.15.98:0/126130676 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f30440a2e40 0x7f30440a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.694+0000 7f3055ffb640 1 -- 172.21.15.98:0/126130676 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f30480ef950 con 0x7f3058108830 2024-05-09T08:55:39.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.695+0000 7f305ce7d640 1 --2- 172.21.15.98:0/2135678451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61779d90 0x55ec61783f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:39.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.696+0000 7f3057fff640 1 --2- 172.21.15.98:0/2135678451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61786850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:39.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.696+0000 7f305d67e640 1 --2- 172.21.15.98:0/126130676 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f30440a2e40 0x7f30440a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:39.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.696+0000 7f305d67e640 1 -- 172.21.15.98:0/2135678451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61779d90 msgr2=0x55ec61783f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:39.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.696+0000 7f305d67e640 1 --2- 172.21.15.98:0/2135678451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61779d90 0x55ec61783f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.696+0000 7f305d67e640 1 -- 172.21.15.98:0/2135678451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 msgr2=0x55ec61786850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:39.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.696+0000 7f305d67e640 1 --2- 172.21.15.98:0/2135678451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61786850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.696+0000 7f305d67e640 1 -- 172.21.15.98:0/2135678451 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec614bc7d0 con 0x55ec61779470 2024-05-09T08:55:39.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.698+0000 7f305d67e640 1 --2- 172.21.15.98:0/2135678451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779470 0x55ec61779850 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f304000bf00 tx=0x7f3040014660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cad8eb7da342aba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:39.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.700+0000 7f305d67e640 1 --2- 172.21.15.98:0/126130676 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f30440a2e40 0x7f30440a5300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3040014fd0 tx=0x7f3040019000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:39.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.701+0000 7f304effd640 1 -- 172.21.15.98:0/2135678451 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040017d30 con 0x55ec61779470 2024-05-09T08:55:39.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.701+0000 7f304effd640 1 -- 172.21.15.98:0/2135678451 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f304001b040 con 0x55ec61779470 2024-05-09T08:55:39.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.701+0000 7f304effd640 1 -- 172.21.15.98:0/2135678451 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304001e5e0 con 0x55ec61779470 2024-05-09T08:55:39.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.705+0000 7f3061df5740 1 -- 172.21.15.98:0/2135678451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779470 msgr2=0x55ec61779850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:39.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.705+0000 7f3061df5740 1 --2- 172.21.15.98:0/2135678451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779470 0x55ec61779850 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f304000bf00 tx=0x7f3040014660 comp rx=0 tx=0).stop 2024-05-09T08:55:39.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.707+0000 7f3061df5740 1 -- 172.21.15.98:0/2135678451 shutdown_connections 2024-05-09T08:55:39.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.707+0000 7f3061df5740 1 --2- 172.21.15.98:0/2135678451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61786850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.707+0000 7f3061df5740 1 --2- 172.21.15.98:0/2135678451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61779d90 0x55ec61783f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.707+0000 7f3061df5740 1 --2- 172.21.15.98:0/2135678451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779470 0x55ec61779850 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.707+0000 7f3061df5740 1 -- 172.21.15.98:0/2135678451 >> 172.21.15.98:0/2135678451 conn(0x55ec61775790 msgr2=0x55ec61775b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:39.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.714+0000 7f3061df5740 1 -- 172.21.15.98:0/2135678451 shutdown_connections 2024-05-09T08:55:39.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.716+0000 7f3061df5740 1 -- 172.21.15.98:0/2135678451 wait complete. 2024-05-09T08:55:39.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.716+0000 7f3061df5740 1 -- start start 2024-05-09T08:55:39.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.716+0000 7f3061df5740 1 Processor -- start 2024-05-09T08:55:39.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.717+0000 7f3061df5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61779470 0x55ec61781500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.717+0000 7f3061df5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779d90 0x55ec61781a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.717+0000 7f3061df5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61781f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.717+0000 7f3061df5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ec617871f0 con 0x55ec61784460 2024-05-09T08:55:39.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.717+0000 7f3061df5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ec61699b90 con 0x55ec61779470 2024-05-09T08:55:39.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.717+0000 7f3061df5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ec61787370 con 0x55ec61779d90 2024-05-09T08:55:39.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.717+0000 7f3061df5740 10 client.0 _ll_register_callbacks cb 0x55ec616d5da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:39.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.720+0000 7f305d67e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61781f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:39.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.720+0000 7f305d67e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61781f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37436/0 (socket says 172.21.15.98:37436) 2024-05-09T08:55:39.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.720+0000 7f305d67e640 1 -- 172.21.15.98:0/3209135084 learned_addr learned my addr 172.21.15.98:0/3209135084 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:39.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.720+0000 7f3057fff640 1 --2- 172.21.15.98:0/3209135084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779d90 0x55ec61781a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:39.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.721+0000 7f305d67e640 1 -- 172.21.15.98:0/3209135084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779d90 msgr2=0x55ec61781a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:39.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.721+0000 7f305d67e640 1 --2- 172.21.15.98:0/3209135084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779d90 0x55ec61781a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.721+0000 7f305d67e640 1 -- 172.21.15.98:0/3209135084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61779470 msgr2=0x55ec61781500 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:55:39.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.721+0000 7f305d67e640 1 --2- 172.21.15.98:0/3209135084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61779470 0x55ec61781500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.721+0000 7f305d67e640 1 -- 172.21.15.98:0/3209135084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec613f23f0 con 0x55ec61784460 2024-05-09T08:55:39.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.722+0000 7f305d67e640 1 --2- 172.21.15.98:0/3209135084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61781f80 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f304000b5f0 tx=0x7f3040017f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e64babeb4d3bb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:39.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.722+0000 7f3037fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:55:39.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.723+0000 7f3037fff640 1 -- 172.21.15.98:0/3209135084 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304001ebb0 con 0x55ec61784460 2024-05-09T08:55:39.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.723+0000 7f3061df5740 1 -- 172.21.15.98:0/3209135084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f304001b040 con 0x55ec61784460 2024-05-09T08:55:39.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.724+0000 7f3037fff640 1 -- 172.21.15.98:0/3209135084 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f304001ed50 con 0x55ec61784460 2024-05-09T08:55:39.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.725+0000 7f3037fff640 1 -- 172.21.15.98:0/3209135084 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3040039050 con 0x55ec61784460 2024-05-09T08:55:39.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.725+0000 7f3061df5740 10 client.17067 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:39.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.725+0000 7f3061df5740 1 -- 172.21.15.98:0/3209135084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ec613b20f0 con 0x55ec61784460 2024-05-09T08:55:39.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.726+0000 7f3037fff640 1 -- 172.21.15.98:0/3209135084 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f304001b040 con 0x55ec61784460 2024-05-09T08:55:39.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.727+0000 7f3037fff640 1 -- 172.21.15.98:0/3209135084 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3040030930 con 0x55ec61784460 2024-05-09T08:55:39.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.728+0000 7f3061df5740 10 client.17067 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:39.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.728+0000 7f3061df5740 10 client.17067 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:39.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.728+0000 7f3061df5740 20 client.17067 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:39.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.728+0000 7f3061df5740 1 --2- 172.21.15.98:0/3209135084 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec61794ba0 0x55ec61797060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.728+0000 7f3061df5740 4 client.17067 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:39.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.728+0000 7f3061df5740 1 -- 172.21.15.98:0/3209135084 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec616ec430 con 0x55ec61794ba0 2024-05-09T08:55:39.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.729+0000 7f305ce7d640 1 --2- 172.21.15.98:0/3209135084 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec61794ba0 0x55ec61797060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:39.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.735+0000 7f305ce7d640 1 --2- 172.21.15.98:0/3209135084 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec61794ba0 0x55ec61797060 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53ca8a5a5d59aa35 server_cookie=c8b82ef24546c654 in_seq=0 out_seq=0 2024-05-09T08:55:39.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.736+0000 7f3037fff640 10 client.17067 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:39.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.740+0000 7f3037fff640 1 -- 172.21.15.98:0/3209135084 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ec616ec430 con 0x55ec61794ba0 2024-05-09T08:55:39.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.740+0000 7f3037fff640 10 client.17067 handle_command_reply: tid=0 2024-05-09T08:55:39.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.740+0000 7f3061df5740 1 client.17067 shutdown 2024-05-09T08:55:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.751+0000 7f3061df5740 1 -- 172.21.15.98:0/3209135084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 msgr2=0x55ec61781f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.751+0000 7f3061df5740 1 --2- 172.21.15.98:0/3209135084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61781f80 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f304000b5f0 tx=0x7f3040017f60 comp rx=0 tx=0).stop 2024-05-09T08:55:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.751+0000 7f3061df5740 1 -- 172.21.15.98:0/3209135084 shutdown_connections 2024-05-09T08:55:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.751+0000 7f3061df5740 1 --2- 172.21.15.98:0/3209135084 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec61794ba0 0x55ec61797060 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.751+0000 7f3061df5740 1 --2- 172.21.15.98:0/3209135084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61781f80 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.751+0000 7f3061df5740 1 --2- 172.21.15.98:0/3209135084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779d90 0x55ec61781a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.751+0000 7f3061df5740 1 --2- 172.21.15.98:0/3209135084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61779470 0x55ec61781500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.751+0000 7f3061df5740 1 -- 172.21.15.98:0/3209135084 >> 172.21.15.98:0/3209135084 conn(0x55ec61775790 msgr2=0x55ec6177b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.752+0000 7f3037fff640 0 client.17067 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:39.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.753+0000 7f3061df5740 1 -- 172.21.15.98:0/3209135084 shutdown_connections 2024-05-09T08:55:39.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.753+0000 7f3061df5740 1 -- 172.21.15.98:0/3209135084 wait complete. 2024-05-09T08:55:39.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.754+0000 7f3061df5740 20 client.17067 trim_cache size 0 max 16384 2024-05-09T08:55:39.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.820+0000 7f3061df5740 1 Processor -- start 2024-05-09T08:55:39.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.821+0000 7f3061df5740 1 -- start start 2024-05-09T08:55:39.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.821+0000 7f3061df5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61779470 0x55ec617900b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.821+0000 7f3061df5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61779d90 0x55ec61781bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.821+0000 7f3061df5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61784460 0x55ec6179c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.822+0000 7f3061df5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30581c6200 con 0x55ec61779d90 2024-05-09T08:55:39.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.822+0000 7f3057fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61779d90 0x55ec61781bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:39.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.822+0000 7f3057fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61779d90 0x55ec61781bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37440/0 (socket says 172.21.15.98:37440) 2024-05-09T08:55:39.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.822+0000 7f3057fff640 1 -- 172.21.15.98:0/3126261502 learned_addr learned my addr 172.21.15.98:0/3126261502 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:39.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.822+0000 7f3061df5740 1 -- 172.21.15.98:0/3126261502 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f30581c7d40 con 0x55ec61779470 2024-05-09T08:55:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.822+0000 7f3061df5740 1 -- 172.21.15.98:0/3126261502 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f30581c68d0 con 0x55ec61784460 2024-05-09T08:55:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.823+0000 7f3057fff640 1 -- 172.21.15.98:0/3126261502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61784460 msgr2=0x55ec6179c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.823+0000 7f3057fff640 1 --2- 172.21.15.98:0/3126261502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61784460 0x55ec6179c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.823+0000 7f3057fff640 1 -- 172.21.15.98:0/3126261502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61779470 msgr2=0x55ec617900b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.823+0000 7f3057fff640 1 --2- 172.21.15.98:0/3126261502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61779470 0x55ec617900b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.823+0000 7f3057fff640 1 -- 172.21.15.98:0/3126261502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec617804b0 con 0x55ec61779d90 2024-05-09T08:55:39.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.823+0000 7f3057fff640 1 --2- 172.21.15.98:0/3126261502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61779d90 0x55ec61781bd0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3058117400 tx=0x7f30500127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1a61f3cbc33a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:39.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.824+0000 7f304e7fc640 1 -- 172.21.15.98:0/3126261502 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3050003b70 con 0x55ec61779d90 2024-05-09T08:55:39.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.825+0000 7f304e7fc640 1 -- 172.21.15.98:0/3126261502 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3050009e30 con 0x55ec61779d90 2024-05-09T08:55:39.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.825+0000 7f304e7fc640 1 -- 172.21.15.98:0/3126261502 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305001a870 con 0x55ec61779d90 2024-05-09T08:55:39.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.828+0000 7f3061df5740 1 -- 172.21.15.98:0/3126261502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61779d90 msgr2=0x55ec61781bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:39.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.828+0000 7f3061df5740 1 --2- 172.21.15.98:0/3126261502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61779d90 0x55ec61781bd0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3058117400 tx=0x7f30500127c0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.828+0000 7f3061df5740 1 -- 172.21.15.98:0/3126261502 shutdown_connections 2024-05-09T08:55:39.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.828+0000 7f3061df5740 1 --2- 172.21.15.98:0/3126261502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61784460 0x55ec6179c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.829+0000 7f3061df5740 1 --2- 172.21.15.98:0/3126261502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61779d90 0x55ec61781bd0 secure :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3058117400 tx=0x7f30500127c0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.829+0000 7f3061df5740 1 --2- 172.21.15.98:0/3126261502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61779470 0x55ec617900b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.829+0000 7f3061df5740 1 -- 172.21.15.98:0/3126261502 >> 172.21.15.98:0/3126261502 conn(0x55ec61775790 msgr2=0x55ec617814c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.834+0000 7f3061df5740 1 -- 172.21.15.98:0/3126261502 shutdown_connections 2024-05-09T08:55:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.834+0000 7f3061df5740 1 -- 172.21.15.98:0/3126261502 wait complete. 2024-05-09T08:55:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.835+0000 7f3061df5740 1 -- start start 2024-05-09T08:55:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.835+0000 7f3061df5740 1 Processor -- start 2024-05-09T08:55:39.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.836+0000 7f3061df5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779470 0x55ec6178cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.837+0000 7f3061df5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61787de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.837+0000 7f3057fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61787de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:39.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.837+0000 7f3057fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61787de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37442/0 (socket says 172.21.15.98:37442) 2024-05-09T08:55:39.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.837+0000 7f305ce7d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779470 0x55ec6178cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:39.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.837+0000 7f305ce7d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779470 0x55ec6178cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:46970/0 (socket says 172.21.15.98:46970) 2024-05-09T08:55:39.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.837+0000 7f3061df5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61788480 0x55ec617fa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.837+0000 7f3057fff640 1 -- 172.21.15.98:0/944454583 learned_addr learned my addr 172.21.15.98:0/944454583 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:39.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.837+0000 7f3061df5740 1 -- 172.21.15.98:0/944454583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30581c7670 con 0x55ec61784460 2024-05-09T08:55:39.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.838+0000 7f3061df5740 1 -- 172.21.15.98:0/944454583 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f30581c6fa0 con 0x55ec61788480 2024-05-09T08:55:39.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.838+0000 7f3061df5740 1 -- 172.21.15.98:0/944454583 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f30581c4290 con 0x55ec61779470 2024-05-09T08:55:39.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.838+0000 7f3061df5740 10 client.0 _ll_register_callbacks cb 0x7f304000bcd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:39.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.838+0000 7f3057fff640 1 -- 172.21.15.98:0/944454583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779470 msgr2=0x55ec6178cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:39.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.838+0000 7f3057fff640 1 --2- 172.21.15.98:0/944454583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779470 0x55ec6178cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.838+0000 7f3057fff640 1 -- 172.21.15.98:0/944454583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61788480 msgr2=0x55ec617fa1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:39.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.839+0000 7f3057fff640 1 --2- 172.21.15.98:0/944454583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61788480 0x55ec617fa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.839+0000 7f3057fff640 1 -- 172.21.15.98:0/944454583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec61779bc0 con 0x55ec61784460 2024-05-09T08:55:39.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.839+0000 7f3057fff640 1 --2- 172.21.15.98:0/944454583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61787de0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f3050009650 tx=0x7f3050003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e9ce36ce1f9155 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:39.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.839+0000 7f3037fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:55:39.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.840+0000 7f3037fff640 1 -- 172.21.15.98:0/944454583 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3050028020 con 0x55ec61784460 2024-05-09T08:55:39.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.840+0000 7f3037fff640 1 -- 172.21.15.98:0/944454583 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f3050004050 con 0x55ec61784460 2024-05-09T08:55:39.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.840+0000 7f3061df5740 1 -- 172.21.15.98:0/944454583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3050009e30 con 0x55ec61784460 2024-05-09T08:55:39.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.840+0000 7f3037fff640 1 -- 172.21.15.98:0/944454583 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3050030050 con 0x55ec61784460 2024-05-09T08:55:39.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.841+0000 7f3061df5740 10 client.17073 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:39.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.841+0000 7f3061df5740 1 -- 172.21.15.98:0/944454583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ec617fb580 con 0x55ec61784460 2024-05-09T08:55:39.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.846+0000 7f3037fff640 1 -- 172.21.15.98:0/944454583 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3050009e30 con 0x55ec61784460 2024-05-09T08:55:39.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.846+0000 7f3037fff640 1 -- 172.21.15.98:0/944454583 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3050023070 con 0x55ec61784460 2024-05-09T08:55:39.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.849+0000 7f3061df5740 10 client.17073 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:39.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.849+0000 7f3061df5740 10 client.17073 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:39.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.849+0000 7f3061df5740 20 client.17073 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:39.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.849+0000 7f3061df5740 1 --2- 172.21.15.98:0/944454583 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec617fbb50 0x55ec617fdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:39.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.849+0000 7f3061df5740 4 client.17073 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:55:39.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.849+0000 7f3061df5740 1 -- 172.21.15.98:0/944454583 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ec616e6780 con 0x55ec617fbb50 2024-05-09T08:55:39.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.854+0000 7f305ce7d640 1 --2- 172.21.15.98:0/944454583 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec617fbb50 0x55ec617fdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:39.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.880+0000 7f305ce7d640 1 --2- 172.21.15.98:0/944454583 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec617fbb50 0x55ec617fdf70 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e92a83be27cbcf3 server_cookie=5b5139a475fb3b51 in_seq=0 out_seq=0 2024-05-09T08:55:39.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.880+0000 7f3037fff640 10 client.17073 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.885+0000 7f3037fff640 1 -- 172.21.15.98:0/944454583 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55ec616e6780 con 0x55ec617fbb50 2024-05-09T08:55:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.885+0000 7f3037fff640 10 client.17073 handle_command_reply: tid=0 2024-05-09T08:55:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.885+0000 7f3061df5740 1 client.17073 shutdown 2024-05-09T08:55:39.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.899+0000 7f3061df5740 1 -- 172.21.15.98:0/944454583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 msgr2=0x55ec61787de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:39.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.899+0000 7f3061df5740 1 --2- 172.21.15.98:0/944454583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61787de0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f3050009650 tx=0x7f3050003c10 comp rx=0 tx=0).stop 2024-05-09T08:55:39.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.904+0000 7f3061df5740 1 -- 172.21.15.98:0/944454583 shutdown_connections 2024-05-09T08:55:39.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.904+0000 7f3061df5740 1 --2- 172.21.15.98:0/944454583 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ec617fbb50 0x55ec617fdf70 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.904+0000 7f3061df5740 1 --2- 172.21.15.98:0/944454583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec61788480 0x55ec617fa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.904+0000 7f3061df5740 1 --2- 172.21.15.98:0/944454583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec61784460 0x55ec61787de0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.904+0000 7f3061df5740 1 --2- 172.21.15.98:0/944454583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec61779470 0x55ec6178cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.904+0000 7f3061df5740 1 -- 172.21.15.98:0/944454583 >> 172.21.15.98:0/944454583 conn(0x55ec61775790 msgr2=0x55ec61781d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:39.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.904+0000 7f3037fff640 0 client.17073 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:39.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.908+0000 7f3061df5740 1 -- 172.21.15.98:0/944454583 shutdown_connections 2024-05-09T08:55:39.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.909+0000 7f3061df5740 1 -- 172.21.15.98:0/944454583 wait complete. 2024-05-09T08:55:39.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.911+0000 7f3061df5740 20 client.17073 trim_cache size 0 max 16384 2024-05-09T08:55:39.912 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:55:39.912 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "scrub active (7 inodes in the stack)", 2024-05-09T08:55:39.912 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:55:39.913 INFO:teuthology.orchestra.run.smithi098.stdout: "375447eb-be0b-4843-b1db-b3d92f4ba4f4": { 2024-05-09T08:55:39.913 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:55:39.913 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "375447eb-be0b-4843-b1db-b3d92f4ba4f4", 2024-05-09T08:55:39.913 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:55:39.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:55:39.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:55:39.913 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:55:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.917+0000 7f305de7f640 1 -- 172.21.15.98:0/126130676 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f30440a2e40 msgr2=0x7f30440a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.917+0000 7f305de7f640 1 --2- 172.21.15.98:0/126130676 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f30440a2e40 0x7f30440a5300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3040014fd0 tx=0x7f3040019000 comp rx=0 tx=0).stop 2024-05-09T08:55:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.917+0000 7f305de7f640 1 -- 172.21.15.98:0/126130676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3058108830 msgr2=0x7f305811cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.917+0000 7f305de7f640 1 --2- 172.21.15.98:0/126130676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3058108830 0x7f305811cbc0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3048031980 tx=0x7f3048005650 comp rx=0 tx=0).stop 2024-05-09T08:55:39.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.920+0000 7f305de7f640 1 -- 172.21.15.98:0/126130676 shutdown_connections 2024-05-09T08:55:39.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.920+0000 7f305de7f640 1 --2- 172.21.15.98:0/126130676 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f30440a2e40 0x7f30440a5300 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.920+0000 7f305de7f640 1 --2- 172.21.15.98:0/126130676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f305810fed0 0x7f3058116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.920+0000 7f305de7f640 1 --2- 172.21.15.98:0/126130676 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30581092b0 0x7f3058115c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.920+0000 7f305de7f640 1 --2- 172.21.15.98:0/126130676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3058108830 0x7f305811cbc0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:39.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.920+0000 7f305de7f640 1 -- 172.21.15.98:0/126130676 >> 172.21.15.98:0/126130676 conn(0x7f305806c7b0 msgr2=0x7f3058071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:39.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.928+0000 7f305de7f640 1 -- 172.21.15.98:0/126130676 shutdown_connections 2024-05-09T08:55:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:55:39.929+0000 7f305de7f640 1 -- 172.21.15.98:0/126130676 wait complete. 2024-05-09T08:55:39.953 INFO:tasks.cephfs.filesystem:scrub status for tag:375447eb-be0b-4843-b1db-b3d92f4ba4f4 - {'path': '/', 'tag': '375447eb-be0b-4843-b1db-b3d92f4ba4f4', 'options': 'recursive,force'} 2024-05-09T08:55:40.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:40 smithi098 ceph-mon[30092]: pgmap v155: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 723 MiB/s wr, 1.09k op/s 2024-05-09T08:55:40.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:40 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:55:40.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:40 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:55:40.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:40 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:55:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:40 smithi142 ceph-mon[34625]: pgmap v155: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 723 MiB/s wr, 1.09k op/s 2024-05-09T08:55:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:40 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:55:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:40 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:55:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:40 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:55:40.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:40 smithi105 ceph-mon[34498]: pgmap v155: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 723 MiB/s wr, 1.09k op/s 2024-05-09T08:55:40.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:40 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:55:40.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:40 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:55:40.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:40 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:55:42.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.507+0000 7fcc9721b640 1 -- 172.21.15.98:0/1661184390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9010fef0 msgr2=0x7fcc9010d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:42.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.507+0000 7fcc9721b640 1 --2- 172.21.15.98:0/1661184390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9010fef0 0x7fcc9010d9a0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fcc88009ca0 tx=0x7fcc8802a980 comp rx=0 tx=0).stop 2024-05-09T08:55:42.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.508+0000 7fcc9721b640 1 -- 172.21.15.98:0/1661184390 shutdown_connections 2024-05-09T08:55:42.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.508+0000 7fcc9721b640 1 --2- 172.21.15.98:0/1661184390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc9010e500 0x7fcc9011d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.508+0000 7fcc9721b640 1 --2- 172.21.15.98:0/1661184390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc901102d0 0x7fcc9010dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.508+0000 7fcc9721b640 1 --2- 172.21.15.98:0/1661184390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9010fef0 0x7fcc9010d9a0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.508+0000 7fcc9721b640 1 -- 172.21.15.98:0/1661184390 >> 172.21.15.98:0/1661184390 conn(0x7fcc9006c770 msgr2=0x7fcc9006cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.517+0000 7fcc9721b640 1 -- 172.21.15.98:0/1661184390 shutdown_connections 2024-05-09T08:55:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.518+0000 7fcc9721b640 1 -- 172.21.15.98:0/1661184390 wait complete. 2024-05-09T08:55:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.520+0000 7fcc9721b640 1 Processor -- start 2024-05-09T08:55:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.520+0000 7fcc9721b640 1 -- start start 2024-05-09T08:55:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.520+0000 7fcc9721b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc9010e500 0x7fcc901a6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.520+0000 7fcc9721b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9010fef0 0x7fcc901a7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.520+0000 7fcc9721b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc901102d0 0x7fcc901ab810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.520+0000 7fcc9721b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc90120a10 con 0x7fcc9010fef0 2024-05-09T08:55:42.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.520+0000 7fcc9721b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcc90120890 con 0x7fcc9010e500 2024-05-09T08:55:42.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.520+0000 7fcc9721b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcc90120b90 con 0x7fcc901102d0 2024-05-09T08:55:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.521+0000 7fcc96a1a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc901102d0 0x7fcc901ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.521+0000 7fcc96a1a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc901102d0 0x7fcc901ab810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:46980/0 (socket says 172.21.15.98:46980) 2024-05-09T08:55:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.521+0000 7fcc96a1a640 1 -- 172.21.15.98:0/3938008660 learned_addr learned my addr 172.21.15.98:0/3938008660 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.521+0000 7fcc96219640 1 --2- 172.21.15.98:0/3938008660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc9010e500 0x7fcc901a6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.522+0000 7fcc96a1a640 1 -- 172.21.15.98:0/3938008660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc9010e500 msgr2=0x7fcc901a6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.522+0000 7fcc96a1a640 1 --2- 172.21.15.98:0/3938008660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc9010e500 0x7fcc901a6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.522+0000 7fcc96a1a640 1 -- 172.21.15.98:0/3938008660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9010fef0 msgr2=0x7fcc901a7490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:55:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.522+0000 7fcc96a1a640 1 --2- 172.21.15.98:0/3938008660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9010fef0 0x7fcc901a7490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.522+0000 7fcc96a1a640 1 -- 172.21.15.98:0/3938008660 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc901abf10 con 0x7fcc901102d0 2024-05-09T08:55:42.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.522+0000 7fcc96a1a640 1 --2- 172.21.15.98:0/3938008660 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc901102d0 0x7fcc901ab810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcc8000b420 tx=0x7fcc8000b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:42.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.523+0000 7fcc877fe640 1 -- 172.21.15.98:0/3938008660 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc80011830 con 0x7fcc901102d0 2024-05-09T08:55:42.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.523+0000 7fcc9721b640 1 -- 172.21.15.98:0/3938008660 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc901ac200 con 0x7fcc901102d0 2024-05-09T08:55:42.523 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.523+0000 7fcc9721b640 1 -- 172.21.15.98:0/3938008660 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc901107f0 con 0x7fcc901102d0 2024-05-09T08:55:42.523 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.523+0000 7fcc877fe640 1 -- 172.21.15.98:0/3938008660 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc80016070 con 0x7fcc901102d0 2024-05-09T08:55:42.523 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.523+0000 7fcc877fe640 1 -- 172.21.15.98:0/3938008660 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8000f2e0 con 0x7fcc901102d0 2024-05-09T08:55:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.532+0000 7fcc877fe640 1 -- 172.21.15.98:0/3938008660 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcc8000f480 con 0x7fcc901102d0 2024-05-09T08:55:42.535 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.535+0000 7fcc877fe640 1 --2- 172.21.15.98:0/3938008660 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcc7c0a2e40 0x7fcc7c0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.535 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.535+0000 7fcc877fe640 1 -- 172.21.15.98:0/3938008660 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fcc800ced30 con 0x7fcc901102d0 2024-05-09T08:55:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.539+0000 7fcc9b18e740 1 Processor -- start 2024-05-09T08:55:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.540+0000 7fcc9b18e740 1 -- start start 2024-05-09T08:55:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.541+0000 7fcc9b18e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c045d0 0x55bed8c049b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.541+0000 7fcc9b18e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c04ef0 0x55bed8c0f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.541+0000 7fcc9b18e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c0f7b0 0x55bed8c11ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.541+0000 7fcc96219640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c045d0 0x55bed8c049b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.542+0000 7fcc96219640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c045d0 0x55bed8c049b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:46994/0 (socket says 172.21.15.98:46994) 2024-05-09T08:55:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.542+0000 7fcc96219640 1 -- 172.21.15.98:0/4228433208 learned_addr learned my addr 172.21.15.98:0/4228433208 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.542+0000 7fcc9b18e740 1 -- 172.21.15.98:0/4228433208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bed8b69340 con 0x55bed8c04ef0 2024-05-09T08:55:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.542+0000 7fcc96a1a640 1 --2- 172.21.15.98:0/4228433208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c0f7b0 0x55bed8c11ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.542+0000 7fcc9b18e740 1 -- 172.21.15.98:0/4228433208 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bed8b11290 con 0x55bed8c0f7b0 2024-05-09T08:55:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.542+0000 7fcc9b18e740 1 -- 172.21.15.98:0/4228433208 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55bed8b65300 con 0x55bed8c045d0 2024-05-09T08:55:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.544+0000 7fcc96219640 1 --2- 172.21.15.98:0/3938008660 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcc7c0a2e40 0x7fcc7c0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.544+0000 7fcc96219640 1 -- 172.21.15.98:0/4228433208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c0f7b0 msgr2=0x55bed8c11ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.544+0000 7fcc96219640 1 --2- 172.21.15.98:0/4228433208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c0f7b0 0x55bed8c11ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.544+0000 7fcc96219640 1 -- 172.21.15.98:0/4228433208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c04ef0 msgr2=0x55bed8c0f270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:55:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.544+0000 7fcc96219640 1 --2- 172.21.15.98:0/4228433208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c04ef0 0x55bed8c0f270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.544+0000 7fcc96219640 1 -- 172.21.15.98:0/4228433208 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bed8b358e0 con 0x55bed8c045d0 2024-05-09T08:55:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.545+0000 7fcc96219640 1 --2- 172.21.15.98:0/3938008660 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcc7c0a2e40 0x7fcc7c0a5300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcc88009ca0 tx=0x7fcc8803b040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.550+0000 7fcc96219640 1 --2- 172.21.15.98:0/4228433208 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c045d0 0x55bed8c049b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcc88005fa0 tx=0x7fcc88003a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ac364a358fb70ed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:42.553 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.554+0000 7fcc84bfe640 1 -- 172.21.15.98:0/4228433208 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8804b070 con 0x55bed8c045d0 2024-05-09T08:55:42.553 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.554+0000 7fcc84bfe640 1 -- 172.21.15.98:0/4228433208 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc88004040 con 0x55bed8c045d0 2024-05-09T08:55:42.553 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.554+0000 7fcc84bfe640 1 -- 172.21.15.98:0/4228433208 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc88034050 con 0x55bed8c045d0 2024-05-09T08:55:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.557+0000 7fcc9b18e740 1 -- 172.21.15.98:0/4228433208 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c045d0 msgr2=0x55bed8c049b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.557+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/4228433208 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c045d0 0x55bed8c049b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcc88005fa0 tx=0x7fcc88003a70 comp rx=0 tx=0).stop 2024-05-09T08:55:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.557+0000 7fcc9b18e740 1 -- 172.21.15.98:0/4228433208 shutdown_connections 2024-05-09T08:55:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.557+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/4228433208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c0f7b0 0x55bed8c11ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.557+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/4228433208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c04ef0 0x55bed8c0f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.557+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/4228433208 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c045d0 0x55bed8c049b0 secure :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcc88005fa0 tx=0x7fcc88003a70 comp rx=0 tx=0).stop 2024-05-09T08:55:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.557+0000 7fcc9b18e740 1 -- 172.21.15.98:0/4228433208 >> 172.21.15.98:0/4228433208 conn(0x55bed8c00ae0 msgr2=0x55bed8c00ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:42.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.564+0000 7fcc9b18e740 1 -- 172.21.15.98:0/4228433208 shutdown_connections 2024-05-09T08:55:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.567+0000 7fcc9b18e740 1 -- 172.21.15.98:0/4228433208 wait complete. 2024-05-09T08:55:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.567+0000 7fcc9b18e740 1 -- start start 2024-05-09T08:55:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.567+0000 7fcc9b18e740 1 Processor -- start 2024-05-09T08:55:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.572+0000 7fcc9b18e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c04ef0 0x55bed8c0c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.572+0000 7fcc9b18e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0f7b0 0x55bed8c0ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.572+0000 7fcc9b18e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c0d360 0x55bed8c1b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.572+0000 7fcc9b18e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bed8c12320 con 0x55bed8c0d360 2024-05-09T08:55:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.572+0000 7fcc9b18e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bed8b24790 con 0x55bed8c04ef0 2024-05-09T08:55:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.572+0000 7fcc9b18e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55bed8c124a0 con 0x55bed8c0f7b0 2024-05-09T08:55:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.572+0000 7fcc9b18e740 10 client.0 _ll_register_callbacks cb 0x55bed8b5fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.575+0000 7fcc95a18640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0f7b0 0x55bed8c0ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.575+0000 7fcc95a18640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0f7b0 0x55bed8c0ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:46996/0 (socket says 172.21.15.98:46996) 2024-05-09T08:55:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.575+0000 7fcc95a18640 1 -- 172.21.15.98:0/2671903429 learned_addr learned my addr 172.21.15.98:0/2671903429 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.577+0000 7fcc96a1a640 1 --2- 172.21.15.98:0/2671903429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c0d360 0x55bed8c1b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.578+0000 7fcc96219640 1 --2- 172.21.15.98:0/2671903429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c04ef0 0x55bed8c0c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.578+0000 7fcc95a18640 1 -- 172.21.15.98:0/2671903429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c04ef0 msgr2=0x55bed8c0c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.578+0000 7fcc95a18640 1 --2- 172.21.15.98:0/2671903429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c04ef0 0x55bed8c0c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.578+0000 7fcc95a18640 1 -- 172.21.15.98:0/2671903429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c0d360 msgr2=0x55bed8c1b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.578+0000 7fcc95a18640 1 --2- 172.21.15.98:0/2671903429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c0d360 0x55bed8c1b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.578+0000 7fcc95a18640 1 -- 172.21.15.98:0/2671903429 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bed887d210 con 0x55bed8c0f7b0 2024-05-09T08:55:42.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.579+0000 7fcc95a18640 1 --2- 172.21.15.98:0/2671903429 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0f7b0 0x55bed8c0ce20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c009dd0 tx=0x7fcc8c00b4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=525ce002557b634e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:42.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.579+0000 7fcc79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:55:42.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.580+0000 7fcc79ffb640 1 -- 172.21.15.98:0/2671903429 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c015020 con 0x55bed8c0f7b0 2024-05-09T08:55:42.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.580+0000 7fcc9b18e740 1 -- 172.21.15.98:0/2671903429 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc88004040 con 0x55bed8c0f7b0 2024-05-09T08:55:42.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.580+0000 7fcc79ffb640 1 -- 172.21.15.98:0/2671903429 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fcc8c011420 con 0x55bed8c0f7b0 2024-05-09T08:55:42.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.580+0000 7fcc79ffb640 1 -- 172.21.15.98:0/2671903429 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcc8c020050 con 0x55bed8c0f7b0 2024-05-09T08:55:42.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.586+0000 7fcc9b18e740 10 client.26323 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:42.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.586+0000 7fcc9b18e740 1 -- 172.21.15.98:0/2671903429 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bed8949620 con 0x55bed8c0f7b0 2024-05-09T08:55:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.588+0000 7fcc79ffb640 1 -- 172.21.15.98:0/2671903429 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc88004040 con 0x55bed8c0f7b0 2024-05-09T08:55:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.588+0000 7fcc79ffb640 1 -- 172.21.15.98:0/2671903429 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fcc8c0198b0 con 0x55bed8c0f7b0 2024-05-09T08:55:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.588+0000 7fcc9b18e740 10 client.26323 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.588+0000 7fcc9b18e740 10 client.26323 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.588+0000 7fcc9b18e740 20 client.26323 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.588+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/2671903429 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bed8c1ffb0 0x55bed8c22450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.588+0000 7fcc9b18e740 4 client.26323 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:42.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.588+0000 7fcc9b18e740 1 -- 172.21.15.98:0/2671903429 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bed8b76a70 con 0x55bed8c1ffb0 2024-05-09T08:55:42.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:42 smithi098 ceph-mon[30092]: pgmap v156: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 889 MiB/s wr, 1.20k op/s 2024-05-09T08:55:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.592+0000 7fcc96219640 1 --2- 172.21.15.98:0/2671903429 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bed8c1ffb0 0x55bed8c22450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:42.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.597+0000 7fcc96219640 1 --2- 172.21.15.98:0/2671903429 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bed8c1ffb0 0x55bed8c22450 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f06f9c949d670930 server_cookie=259b30cf48d6b2b5 in_seq=0 out_seq=0 2024-05-09T08:55:42.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.598+0000 7fcc79ffb640 10 client.26323 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.600+0000 7fcc79ffb640 1 -- 172.21.15.98:0/2671903429 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bed8b76a70 con 0x55bed8c1ffb0 2024-05-09T08:55:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.600+0000 7fcc79ffb640 10 client.26323 handle_command_reply: tid=0 2024-05-09T08:55:42.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.600+0000 7fcc9b18e740 1 client.26323 shutdown 2024-05-09T08:55:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.605+0000 7fcc9b18e740 1 -- 172.21.15.98:0/2671903429 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0f7b0 msgr2=0x55bed8c0ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.605+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/2671903429 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0f7b0 0x55bed8c0ce20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c009dd0 tx=0x7fcc8c00b4b0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.607+0000 7fcc9b18e740 1 -- 172.21.15.98:0/2671903429 shutdown_connections 2024-05-09T08:55:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.607+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/2671903429 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bed8c1ffb0 0x55bed8c22450 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.607+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/2671903429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c0d360 0x55bed8c1b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.607+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/2671903429 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0f7b0 0x55bed8c0ce20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.607+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/2671903429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c04ef0 0x55bed8c0c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.607+0000 7fcc9b18e740 1 -- 172.21.15.98:0/2671903429 >> 172.21.15.98:0/2671903429 conn(0x55bed8c00ae0 msgr2=0x55bed8c06860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.608+0000 7fcc79ffb640 0 client.26323 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:42.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.608+0000 7fcc9b18e740 1 -- 172.21.15.98:0/2671903429 shutdown_connections 2024-05-09T08:55:42.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.609+0000 7fcc9b18e740 1 -- 172.21.15.98:0/2671903429 wait complete. 2024-05-09T08:55:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.615+0000 7fcc9b18e740 20 client.26323 trim_cache size 0 max 16384 2024-05-09T08:55:42.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:42 smithi142 ceph-mon[34625]: pgmap v156: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 889 MiB/s wr, 1.20k op/s 2024-05-09T08:55:42.671 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.672+0000 7fcc9b18e740 1 Processor -- start 2024-05-09T08:55:42.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.675+0000 7fcc9b18e740 1 -- start start 2024-05-09T08:55:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.675+0000 7fcc9b18e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c04ef0 0x55bed8c27d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.675+0000 7fcc9b18e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c0d360 0x55bed8c282a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.675+0000 7fcc9b18e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0f7b0 0x55bed8c1b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.675+0000 7fcc9b18e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc74010090 con 0x55bed8c0d360 2024-05-09T08:55:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.675+0000 7fcc9b18e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcc74011bd0 con 0x55bed8c04ef0 2024-05-09T08:55:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.675+0000 7fcc9b18e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcc74010760 con 0x55bed8c0f7b0 2024-05-09T08:55:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.676+0000 7fcc96219640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c04ef0 0x55bed8c27d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.676+0000 7fcc96219640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c04ef0 0x55bed8c27d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:45846/0 (socket says 172.21.15.98:45846) 2024-05-09T08:55:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.676+0000 7fcc96219640 1 -- 172.21.15.98:0/3240388477 learned_addr learned my addr 172.21.15.98:0/3240388477 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.676+0000 7fcc96219640 1 -- 172.21.15.98:0/3240388477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0f7b0 msgr2=0x55bed8c1b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.676+0000 7fcc96219640 1 --2- 172.21.15.98:0/3240388477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0f7b0 0x55bed8c1b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.676+0000 7fcc96219640 1 -- 172.21.15.98:0/3240388477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c0d360 msgr2=0x55bed8c282a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.676+0000 7fcc96219640 1 --2- 172.21.15.98:0/3240388477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c0d360 0x55bed8c282a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.676+0000 7fcc96219640 1 -- 172.21.15.98:0/3240388477 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bed8c0ae50 con 0x55bed8c04ef0 2024-05-09T08:55:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.677+0000 7fcc96219640 1 --2- 172.21.15.98:0/3240388477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c04ef0 0x55bed8c27d60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55bed8c057e0 tx=0x7fcc8802a980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1acec6c28f7b58df server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.677+0000 7fcc7bfff640 1 -- 172.21.15.98:0/3240388477 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc88062070 con 0x55bed8c04ef0 2024-05-09T08:55:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.677+0000 7fcc7bfff640 1 -- 172.21.15.98:0/3240388477 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc88030c50 con 0x55bed8c04ef0 2024-05-09T08:55:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.677+0000 7fcc7bfff640 1 -- 172.21.15.98:0/3240388477 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8804b040 con 0x55bed8c04ef0 2024-05-09T08:55:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.678+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3240388477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c04ef0 msgr2=0x55bed8c27d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.678+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/3240388477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c04ef0 0x55bed8c27d60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55bed8c057e0 tx=0x7fcc8802a980 comp rx=0 tx=0).stop 2024-05-09T08:55:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.679+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3240388477 shutdown_connections 2024-05-09T08:55:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.679+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/3240388477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0f7b0 0x55bed8c1b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.679+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/3240388477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c0d360 0x55bed8c282a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.679+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/3240388477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c04ef0 0x55bed8c27d60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.679+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3240388477 >> 172.21.15.98:0/3240388477 conn(0x55bed8c00ae0 msgr2=0x55bed8c07bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:42.682 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.682+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3240388477 shutdown_connections 2024-05-09T08:55:42.682 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.682+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3240388477 wait complete. 2024-05-09T08:55:42.682 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.683+0000 7fcc9b18e740 1 -- start start 2024-05-09T08:55:42.682 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.683+0000 7fcc9b18e740 1 Processor -- start 2024-05-09T08:55:42.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.684+0000 7fcc9b18e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c04ef0 0x55bed8c09520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.684+0000 7fcc9b18e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0d360 0x55bed8c09a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.684+0000 7fcc96219640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c04ef0 0x55bed8c09520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:42.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.684+0000 7fcc96219640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c04ef0 0x55bed8c09520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37484/0 (socket says 172.21.15.98:37484) 2024-05-09T08:55:42.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.684+0000 7fcc96219640 1 -- 172.21.15.98:0/3531761905 learned_addr learned my addr 172.21.15.98:0/3531761905 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:42.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.684+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/3531761905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c0f7b0 0x55bed8c09fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.684+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3531761905 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc74011500 con 0x55bed8c04ef0 2024-05-09T08:55:42.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.684+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3531761905 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcc74010e30 con 0x55bed8c0f7b0 2024-05-09T08:55:42.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.684+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3531761905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7400e070 con 0x55bed8c0d360 2024-05-09T08:55:42.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.685+0000 7fcc9b18e740 10 client.0 _ll_register_callbacks cb 0x7fcc8c006e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:42.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.685+0000 7fcc96a1a640 1 --2- 172.21.15.98:0/3531761905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c0f7b0 0x55bed8c09fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:42.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.685+0000 7fcc95a18640 1 --2- 172.21.15.98:0/3531761905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0d360 0x55bed8c09a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:42.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.686+0000 7fcc95a18640 1 -- 172.21.15.98:0/3531761905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c0f7b0 msgr2=0x55bed8c09fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:42.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.686+0000 7fcc95a18640 1 --2- 172.21.15.98:0/3531761905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c0f7b0 0x55bed8c09fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.686+0000 7fcc95a18640 1 -- 172.21.15.98:0/3531761905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c04ef0 msgr2=0x55bed8c09520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:42.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.686+0000 7fcc95a18640 1 --2- 172.21.15.98:0/3531761905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c04ef0 0x55bed8c09520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:42.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.686+0000 7fcc95a18640 1 -- 172.21.15.98:0/3531761905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bed8c855d0 con 0x55bed8c0d360 2024-05-09T08:55:42.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.687+0000 7fcc95a18640 1 --2- 172.21.15.98:0/3531761905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0d360 0x55bed8c09a60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x55bed8b248e0 tx=0x7fcc8c006b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1af6b4108974db94 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:42.687 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.687+0000 7fcc79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:55:42.687 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.688+0000 7fcc79ffb640 1 -- 172.21.15.98:0/3531761905 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c015040 con 0x55bed8c0d360 2024-05-09T08:55:42.687 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.688+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3531761905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc88030c50 con 0x55bed8c0d360 2024-05-09T08:55:42.687 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.688+0000 7fcc79ffb640 1 -- 172.21.15.98:0/3531761905 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fcc8c0195a0 con 0x55bed8c0d360 2024-05-09T08:55:42.688 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.689+0000 7fcc79ffb640 1 -- 172.21.15.98:0/3531761905 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcc8c0197e0 con 0x55bed8c0d360 2024-05-09T08:55:42.688 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.689+0000 7fcc9b18e740 10 client.26326 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:42.688 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.689+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3531761905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bed8c86870 con 0x55bed8c0d360 2024-05-09T08:55:42.689 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.689+0000 7fcc79ffb640 1 -- 172.21.15.98:0/3531761905 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc8c019980 con 0x55bed8c0d360 2024-05-09T08:55:42.689 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.689+0000 7fcc79ffb640 1 -- 172.21.15.98:0/3531761905 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fcc8c025540 con 0x55bed8c0d360 2024-05-09T08:55:42.689 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.690+0000 7fcc9b18e740 10 client.26326 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:42.689 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.690+0000 7fcc9b18e740 10 client.26326 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:42.689 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.690+0000 7fcc9b18e740 20 client.26326 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:42.689 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.690+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/3531761905 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bed8c86f00 0x55bed8c89320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:42.690 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.690+0000 7fcc9b18e740 4 client.26326 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:55:42.690 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.690+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3531761905 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fcc88030c50 con 0x55bed8c86f00 2024-05-09T08:55:42.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.694+0000 7fcc96219640 1 --2- 172.21.15.98:0/3531761905 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bed8c86f00 0x55bed8c89320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.711+0000 7fcc79ffb640 10 client.26326 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:42.711+0000 7fcc96219640 1 --2- 172.21.15.98:0/3531761905 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bed8c86f00 0x55bed8c89320 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=493e0153669b6e5d server_cookie=a6727b8f32549c42 in_seq=0 out_seq=0 2024-05-09T08:55:42.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:42 smithi105 ceph-mon[34498]: pgmap v156: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 889 MiB/s wr, 1.20k op/s 2024-05-09T08:55:44.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:44 smithi098 ceph-mon[30092]: pgmap v157: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 668 MiB/s wr, 938 op/s 2024-05-09T08:55:44.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:44 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:55:44.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:44 smithi142 ceph-mon[34625]: pgmap v157: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 668 MiB/s wr, 938 op/s 2024-05-09T08:55:44.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:44 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:55:44.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:44 smithi105 ceph-mon[34498]: pgmap v157: 129 pgs: 129 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 668 MiB/s wr, 938 op/s 2024-05-09T08:55:44.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:44 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:55:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.238+0000 7fcc79ffb640 1 -- 172.21.15.98:0/3531761905 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fcc88030c50 con 0x55bed8c86f00 2024-05-09T08:55:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.238+0000 7fcc79ffb640 10 client.26326 handle_command_reply: tid=0 2024-05-09T08:55:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.238+0000 7fcc9b18e740 1 client.26326 shutdown 2024-05-09T08:55:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.239+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3531761905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0d360 msgr2=0x55bed8c09a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.239+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/3531761905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0d360 0x55bed8c09a60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x55bed8b248e0 tx=0x7fcc8c006b80 comp rx=0 tx=0).stop 2024-05-09T08:55:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.239+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3531761905 shutdown_connections 2024-05-09T08:55:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.239+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/3531761905 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bed8c86f00 0x55bed8c89320 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.239+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/3531761905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bed8c0f7b0 0x55bed8c09fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.239+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/3531761905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bed8c0d360 0x55bed8c09a60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.239+0000 7fcc9b18e740 1 --2- 172.21.15.98:0/3531761905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bed8c04ef0 0x55bed8c09520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.239+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3531761905 >> 172.21.15.98:0/3531761905 conn(0x55bed8c00ae0 msgr2=0x55bed8bff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.239+0000 7fcc79ffb640 0 client.26326 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.239+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3531761905 shutdown_connections 2024-05-09T08:55:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.239+0000 7fcc9b18e740 1 -- 172.21.15.98:0/3531761905 wait complete. 2024-05-09T08:55:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.240+0000 7fcc9b18e740 20 client.26326 trim_cache size 0 max 16384 2024-05-09T08:55:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:55:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:55:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:55:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:55:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.243+0000 7fcc9721b640 1 -- 172.21.15.98:0/3938008660 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcc7c0a2e40 msgr2=0x7fcc7c0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.243+0000 7fcc9721b640 1 --2- 172.21.15.98:0/3938008660 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcc7c0a2e40 0x7fcc7c0a5300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcc88009ca0 tx=0x7fcc8803b040 comp rx=0 tx=0).stop 2024-05-09T08:55:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.243+0000 7fcc9721b640 1 -- 172.21.15.98:0/3938008660 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc901102d0 msgr2=0x7fcc901ab810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.243+0000 7fcc9721b640 1 --2- 172.21.15.98:0/3938008660 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc901102d0 0x7fcc901ab810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcc8000b420 tx=0x7fcc8000b8e0 comp rx=0 tx=0).stop 2024-05-09T08:55:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.243+0000 7fcc9721b640 1 -- 172.21.15.98:0/3938008660 shutdown_connections 2024-05-09T08:55:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.243+0000 7fcc9721b640 1 --2- 172.21.15.98:0/3938008660 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcc7c0a2e40 0x7fcc7c0a5300 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.243+0000 7fcc9721b640 1 --2- 172.21.15.98:0/3938008660 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc901102d0 0x7fcc901ab810 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.243+0000 7fcc9721b640 1 --2- 172.21.15.98:0/3938008660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9010fef0 0x7fcc901a7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.243+0000 7fcc9721b640 1 --2- 172.21.15.98:0/3938008660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc9010e500 0x7fcc901a6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.243+0000 7fcc9721b640 1 -- 172.21.15.98:0/3938008660 >> 172.21.15.98:0/3938008660 conn(0x7fcc9006c770 msgr2=0x7fcc90070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.243+0000 7fcc9721b640 1 -- 172.21.15.98:0/3938008660 shutdown_connections 2024-05-09T08:55:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:45.243+0000 7fcc9721b640 1 -- 172.21.15.98:0/3938008660 wait complete. 2024-05-09T08:55:46.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:46 smithi098 ceph-mon[30092]: pgmap v158: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 783 MiB/s wr, 1.31k op/s 2024-05-09T08:55:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:46 smithi142 ceph-mon[34625]: pgmap v158: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 783 MiB/s wr, 1.31k op/s 2024-05-09T08:55:46.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:46 smithi105 ceph-mon[34498]: pgmap v158: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 783 MiB/s wr, 1.31k op/s 2024-05-09T08:55:47.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:47 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:55:47.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:47 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:55:47.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:47 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:55:48.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:48 smithi142 ceph-mon[34625]: pgmap v159: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 818 MiB/s wr, 1.58k op/s 2024-05-09T08:55:48.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:48 smithi105 ceph-mon[34498]: pgmap v159: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 818 MiB/s wr, 1.58k op/s 2024-05-09T08:55:48.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:48 smithi098 ceph-mon[30092]: pgmap v159: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 818 MiB/s wr, 1.58k op/s 2024-05-09T08:55:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:50 smithi142 ceph-mon[34625]: pgmap v160: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 741 MiB/s wr, 1.41k op/s 2024-05-09T08:55:50.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:50 smithi105 ceph-mon[34498]: pgmap v160: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 741 MiB/s wr, 1.41k op/s 2024-05-09T08:55:50.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:50 smithi098 ceph-mon[30092]: pgmap v160: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 741 MiB/s wr, 1.41k op/s 2024-05-09T08:55:52.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:52 smithi098 ceph-mon[30092]: pgmap v161: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 948 MiB/s wr, 1.73k op/s 2024-05-09T08:55:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:52 smithi142 ceph-mon[34625]: pgmap v161: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 948 MiB/s wr, 1.73k op/s 2024-05-09T08:55:52.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:52 smithi105 ceph-mon[34498]: pgmap v161: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 948 MiB/s wr, 1.73k op/s 2024-05-09T08:55:54.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:54 smithi142 ceph-mon[34625]: pgmap v162: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 686 MiB/s wr, 1.38k op/s 2024-05-09T08:55:54.662 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:54 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:55:54.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:54 smithi105 ceph-mon[34498]: pgmap v162: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 686 MiB/s wr, 1.38k op/s 2024-05-09T08:55:54.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:54 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:55:54.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:54 smithi098 ceph-mon[30092]: pgmap v162: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 686 MiB/s wr, 1.38k op/s 2024-05-09T08:55:54.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:54 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:55:56.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:56 smithi142 ceph-mon[34625]: pgmap v163: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 833 MiB/s wr, 1.75k op/s 2024-05-09T08:55:56.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:56 smithi105 ceph-mon[34498]: pgmap v163: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 833 MiB/s wr, 1.75k op/s 2024-05-09T08:55:56.841 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:56 smithi098 ceph-mon[30092]: pgmap v163: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 833 MiB/s wr, 1.75k op/s 2024-05-09T08:55:57.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:57 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:55:57.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:57 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:55:57.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:57 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:55:57.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:57 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:55:57.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:57 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:55:57.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:57 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:55:57.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:57 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:55:57.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:57 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:55:57.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:57 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:55:58.478 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.479+0000 7fdea7eca640 1 -- 172.21.15.98:0/3165300652 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdea011b010 msgr2=0x7fdea011d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:58.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.479+0000 7fdea7eca640 1 --2- 172.21.15.98:0/3165300652 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdea011b010 0x7fdea011d400 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fde9c009930 tx=0x7fde9c02a4f0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.484+0000 7fdea7eca640 1 -- 172.21.15.98:0/3165300652 shutdown_connections 2024-05-09T08:55:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.484+0000 7fdea7eca640 1 --2- 172.21.15.98:0/3165300652 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdea011b010 0x7fdea011d400 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.484+0000 7fdea7eca640 1 --2- 172.21.15.98:0/3165300652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdea0110400 0x7fdea0108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.484+0000 7fdea7eca640 1 --2- 172.21.15.98:0/3165300652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdea0110020 0x7fdea0108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.484+0000 7fdea7eca640 1 -- 172.21.15.98:0/3165300652 >> 172.21.15.98:0/3165300652 conn(0x7fdea006c900 msgr2=0x7fdea006cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.487+0000 7fdea7eca640 1 -- 172.21.15.98:0/3165300652 shutdown_connections 2024-05-09T08:55:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.490+0000 7fdea7eca640 1 -- 172.21.15.98:0/3165300652 wait complete. 2024-05-09T08:55:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.491+0000 7fdea7eca640 1 Processor -- start 2024-05-09T08:55:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.492+0000 7fdea7eca640 1 -- start start 2024-05-09T08:55:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.492+0000 7fdea7eca640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdea0110020 0x7fdea01a6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.492+0000 7fdea7eca640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdea011b010 0x7fdea01a7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.492+0000 7fdea7eca640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdea01ab8a0 0x7fdea01abd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.492+0000 7fdea7eca640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdea0120b60 con 0x7fdea01ab8a0 2024-05-09T08:55:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.492+0000 7fdea7eca640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdea01209e0 con 0x7fdea0110020 2024-05-09T08:55:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.492+0000 7fdea7eca640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdea0120ce0 con 0x7fdea011b010 2024-05-09T08:55:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.493+0000 7fdea6467640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdea01ab8a0 0x7fdea01abd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.493+0000 7fdea6467640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdea01ab8a0 0x7fdea01abd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35844/0 (socket says 172.21.15.98:35844) 2024-05-09T08:55:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.493+0000 7fdea6467640 1 -- 172.21.15.98:0/4243830257 learned_addr learned my addr 172.21.15.98:0/4243830257 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.493+0000 7fdea5465640 1 --2- 172.21.15.98:0/4243830257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdea011b010 0x7fdea01a7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.494+0000 7fdea6467640 1 -- 172.21.15.98:0/4243830257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdea011b010 msgr2=0x7fdea01a7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.494+0000 7fdea6467640 1 --2- 172.21.15.98:0/4243830257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdea011b010 0x7fdea01a7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.494+0000 7fdea6467640 1 -- 172.21.15.98:0/4243830257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdea0110020 msgr2=0x7fdea01a6fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:55:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.494+0000 7fdea6467640 1 --2- 172.21.15.98:0/4243830257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdea0110020 0x7fdea01a6fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.494+0000 7fdea6467640 1 -- 172.21.15.98:0/4243830257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea01ac390 con 0x7fdea01ab8a0 2024-05-09T08:55:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.495+0000 7fdea6467640 1 --2- 172.21.15.98:0/4243830257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdea01ab8a0 0x7fdea01abd50 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fde9c02aae0 tx=0x7fde9c039520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:58.496 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.497+0000 7fde96ffd640 1 -- 172.21.15.98:0/4243830257 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c040030 con 0x7fdea01ab8a0 2024-05-09T08:55:58.496 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.497+0000 7fdea7eca640 1 -- 172.21.15.98:0/4243830257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdea01ac620 con 0x7fdea01ab8a0 2024-05-09T08:55:58.496 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.497+0000 7fdea7eca640 1 -- 172.21.15.98:0/4243830257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdea01b3de0 con 0x7fdea01ab8a0 2024-05-09T08:55:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.497+0000 7fde96ffd640 1 -- 172.21.15.98:0/4243830257 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde9c030e40 con 0x7fdea01ab8a0 2024-05-09T08:55:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.497+0000 7fde96ffd640 1 -- 172.21.15.98:0/4243830257 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c035070 con 0x7fdea01ab8a0 2024-05-09T08:55:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.501+0000 7fdeaabd6740 1 Processor -- start 2024-05-09T08:55:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.503+0000 7fdeaabd6740 1 -- start start 2024-05-09T08:55:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.504+0000 7fdeaabd6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 0x558714d24830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.504+0000 7fdeaabd6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24d70 0x558714d2f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.504+0000 7fdeaabd6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d31a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.504+0000 7fdeaabd6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558714c89340 con 0x558714d2f630 2024-05-09T08:55:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.504+0000 7fdeaabd6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558714c31290 con 0x558714d24d70 2024-05-09T08:55:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.504+0000 7fdeaabd6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558714c85300 con 0x558714d24450 2024-05-09T08:55:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.504+0000 7fde96ffd640 1 -- 172.21.15.98:0/4243830257 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fde9c043030 con 0x7fdea01ab8a0 2024-05-09T08:55:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.505+0000 7fdea5465640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 0x558714d24830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.505+0000 7fdea5465640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 0x558714d24830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:49742/0 (socket says 172.21.15.98:49742) 2024-05-09T08:55:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.505+0000 7fdea5465640 1 -- 172.21.15.98:0/2689312734 learned_addr learned my addr 172.21.15.98:0/2689312734 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.506+0000 7fde96ffd640 1 --2- 172.21.15.98:0/4243830257 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fde8c0a2fc0 0x7fde8c0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.506+0000 7fdea5465640 1 -- 172.21.15.98:0/2689312734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24d70 msgr2=0x558714d2f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:55:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.506+0000 7fdea5465640 1 --2- 172.21.15.98:0/2689312734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24d70 0x558714d2f0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.506+0000 7fdea5465640 1 -- 172.21.15.98:0/2689312734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 msgr2=0x558714d31a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.506+0000 7fdea5465640 1 --2- 172.21.15.98:0/2689312734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d31a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.506+0000 7fdea5465640 1 -- 172.21.15.98:0/2689312734 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558714c558e0 con 0x558714d24450 2024-05-09T08:55:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.507+0000 7fde96ffd640 1 -- 172.21.15.98:0/4243830257 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fde9c0f0120 con 0x7fdea01ab8a0 2024-05-09T08:55:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.507+0000 7fdea5465640 1 --2- 172.21.15.98:0/2689312734 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 0x558714d24830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde98009f90 tx=0x7fde9800b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bca1a1fabc9f123c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.509+0000 7fde8bfff640 1 -- 172.21.15.98:0/2689312734 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9801b070 con 0x558714d24450 2024-05-09T08:55:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.510+0000 7fde8bfff640 1 -- 172.21.15.98:0/2689312734 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde98008db0 con 0x558714d24450 2024-05-09T08:55:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.511+0000 7fdeaabd6740 1 -- 172.21.15.98:0/2689312734 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 msgr2=0x558714d24830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.511+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/2689312734 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 0x558714d24830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde98009f90 tx=0x7fde9800b9a0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.513+0000 7fdea6467640 1 --2- 172.21.15.98:0/4243830257 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fde8c0a2fc0 0x7fde8c0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.513+0000 7fdeaabd6740 1 -- 172.21.15.98:0/2689312734 shutdown_connections 2024-05-09T08:55:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.513+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/2689312734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d31a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.513+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/2689312734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24d70 0x558714d2f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.513+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/2689312734 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 0x558714d24830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.513+0000 7fdeaabd6740 1 -- 172.21.15.98:0/2689312734 >> 172.21.15.98:0/2689312734 conn(0x558714d20960 msgr2=0x558714d20d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.517+0000 7fdeaabd6740 1 -- 172.21.15.98:0/2689312734 shutdown_connections 2024-05-09T08:55:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.520+0000 7fdeaabd6740 1 -- 172.21.15.98:0/2689312734 wait complete. 2024-05-09T08:55:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.520+0000 7fdeaabd6740 1 -- start start 2024-05-09T08:55:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.520+0000 7fdeaabd6740 1 Processor -- start 2024-05-09T08:55:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.529+0000 7fdea6467640 1 --2- 172.21.15.98:0/4243830257 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fde8c0a2fc0 0x7fde8c0a5480 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fde9c039590 tx=0x7fde9c009260 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.531+0000 7fdeaabd6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 0x558714d2c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.531+0000 7fdeaabd6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24d70 0x558714d2ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.531+0000 7fdeaabd6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d2cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.531+0000 7fdeaabd6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558714d32260 con 0x558714d2f630 2024-05-09T08:55:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.531+0000 7fdeaabd6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558714c44790 con 0x558714d24d70 2024-05-09T08:55:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.531+0000 7fdeaabd6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558714d323e0 con 0x558714d24450 2024-05-09T08:55:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.531+0000 7fdeaabd6740 10 client.0 _ll_register_callbacks cb 0x558714c7fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.531+0000 7fdea5465640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d2cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.531+0000 7fdea5465640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d2cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35866/0 (socket says 172.21.15.98:35866) 2024-05-09T08:55:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.531+0000 7fdea5465640 1 -- 172.21.15.98:0/1642853812 learned_addr learned my addr 172.21.15.98:0/1642853812 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.532+0000 7fdea5465640 1 --2- 172.21.15.98:0/1642853812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 0x558714d2c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:58.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.532+0000 7fdea5465640 1 -- 172.21.15.98:0/1642853812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 msgr2=0x558714d2c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:58.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.532+0000 7fdea5465640 1 --2- 172.21.15.98:0/1642853812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 0x558714d2c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.532+0000 7fdea5465640 1 -- 172.21.15.98:0/1642853812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24d70 msgr2=0x558714d2ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:58.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.532+0000 7fdea5465640 1 --2- 172.21.15.98:0/1642853812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24d70 0x558714d2ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.532+0000 7fdea5465640 1 -- 172.21.15.98:0/1642853812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55871499d210 con 0x558714d2f630 2024-05-09T08:55:58.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.533+0000 7fdea5465640 1 --2- 172.21.15.98:0/1642853812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d2cf70 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fde98016de0 tx=0x7fde98016970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665fa1950755a553 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:58.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.533+0000 7fde897fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:55:58.536 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.537+0000 7fde897fa640 1 -- 172.21.15.98:0/1642853812 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9801b070 con 0x558714d2f630 2024-05-09T08:55:58.536 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.537+0000 7fde897fa640 1 -- 172.21.15.98:0/1642853812 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fde98003c30 con 0x558714d2f630 2024-05-09T08:55:58.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.540+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1642853812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde98008db0 con 0x558714d2f630 2024-05-09T08:55:58.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.544+0000 7fde897fa640 1 -- 172.21.15.98:0/1642853812 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fde98003e30 con 0x558714d2f630 2024-05-09T08:55:58.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.544+0000 7fdeaabd6740 10 client.17088 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:58.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.544+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1642853812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558714a69620 con 0x558714d2f630 2024-05-09T08:55:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.549+0000 7fde897fa640 1 -- 172.21.15.98:0/1642853812 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde98008db0 con 0x558714d2f630 2024-05-09T08:55:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.549+0000 7fde897fa640 1 -- 172.21.15.98:0/1642853812 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fde98022070 con 0x558714d2f630 2024-05-09T08:55:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.549+0000 7fdeaabd6740 10 client.17088 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.549+0000 7fdeaabd6740 10 client.17088 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.549+0000 7fdeaabd6740 20 client.17088 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.549+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/1642853812 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558714d3f460 0x558714d41880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.549+0000 7fdeaabd6740 4 client.17088 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:55:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.549+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1642853812 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558714c96a70 con 0x558714d3f460 2024-05-09T08:55:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.550+0000 7fdea5c66640 1 --2- 172.21.15.98:0/1642853812 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558714d3f460 0x558714d41880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.560+0000 7fde897fa640 10 client.17088 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.560+0000 7fdea5c66640 1 --2- 172.21.15.98:0/1642853812 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558714d3f460 0x558714d41880 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b587d368fa749ebf server_cookie=a8494dbba7fa2482 in_seq=0 out_seq=0 2024-05-09T08:55:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.562+0000 7fde897fa640 1 -- 172.21.15.98:0/1642853812 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558714c96a70 con 0x558714d3f460 2024-05-09T08:55:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.562+0000 7fde897fa640 10 client.17088 handle_command_reply: tid=0 2024-05-09T08:55:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.563+0000 7fdeaabd6740 1 client.17088 shutdown 2024-05-09T08:55:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.564+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1642853812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 msgr2=0x558714d2cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.564+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/1642853812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d2cf70 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fde98016de0 tx=0x7fde98016970 comp rx=0 tx=0).stop 2024-05-09T08:55:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.568+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1642853812 shutdown_connections 2024-05-09T08:55:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.568+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/1642853812 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558714d3f460 0x558714d41880 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.568+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/1642853812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d2cf70 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.568+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/1642853812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24d70 0x558714d2ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.568+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/1642853812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 0x558714d2c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.568+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1642853812 >> 172.21.15.98:0/1642853812 conn(0x558714d20960 msgr2=0x558714d266e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.568+0000 7fde897fa640 0 client.17088 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.568+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1642853812 shutdown_connections 2024-05-09T08:55:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.569+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1642853812 wait complete. 2024-05-09T08:55:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.569+0000 7fdeaabd6740 20 client.17088 trim_cache size 0 max 16384 2024-05-09T08:55:58.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.600+0000 7fdeaabd6740 1 Processor -- start 2024-05-09T08:55:58.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.600+0000 7fdeaabd6740 1 -- start start 2024-05-09T08:55:58.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.600+0000 7fdeaabd6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24450 0x558714d3b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.600+0000 7fdeaabd6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24d70 0x558714d20d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.600+0000 7fdeaabd6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d2cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.600+0000 7fdeaabd6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde840100d0 con 0x558714d2f630 2024-05-09T08:55:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.600+0000 7fdeaabd6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fde84011c10 con 0x558714d24450 2024-05-09T08:55:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.600+0000 7fdeaabd6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fde840107a0 con 0x558714d24d70 2024-05-09T08:55:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.601+0000 7fdea5465640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d2cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.601+0000 7fdea5465640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d2cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35868/0 (socket says 172.21.15.98:35868) 2024-05-09T08:55:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.601+0000 7fdea5465640 1 -- 172.21.15.98:0/2962551024 learned_addr learned my addr 172.21.15.98:0/2962551024 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.601+0000 7fdea5465640 1 --2- 172.21.15.98:0/2962551024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24450 0x558714d3b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:58.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.603+0000 7fdea5465640 1 -- 172.21.15.98:0/2962551024 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24d70 msgr2=0x558714d20d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:55:58.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.603+0000 7fdea5465640 1 --2- 172.21.15.98:0/2962551024 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24d70 0x558714d20d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.603+0000 7fdea5465640 1 -- 172.21.15.98:0/2962551024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 msgr2=0x558714d2cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:58.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.603+0000 7fdea5465640 1 --2- 172.21.15.98:0/2962551024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d2cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.603+0000 7fdea5465640 1 -- 172.21.15.98:0/2962551024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558714d2b4a0 con 0x558714d24450 2024-05-09T08:55:58.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.605+0000 7fdea5465640 1 --2- 172.21.15.98:0/2962551024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24450 0x558714d3b200 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x558714d2db00 tx=0x7fde9800b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a819446785759e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.607+0000 7fde8b7fe640 1 -- 172.21.15.98:0/2962551024 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9801b070 con 0x558714d24450 2024-05-09T08:55:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.608+0000 7fde8b7fe640 1 -- 172.21.15.98:0/2962551024 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde98008db0 con 0x558714d24450 2024-05-09T08:55:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.609+0000 7fdeaabd6740 1 -- 172.21.15.98:0/2962551024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24450 msgr2=0x558714d3b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:55:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.609+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/2962551024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24450 0x558714d3b200 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x558714d2db00 tx=0x7fde9800b9a0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.610+0000 7fdeaabd6740 1 -- 172.21.15.98:0/2962551024 shutdown_connections 2024-05-09T08:55:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.610+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/2962551024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d2cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.610+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/2962551024 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24d70 0x558714d20d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.610+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/2962551024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24450 0x558714d3b200 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.610+0000 7fdeaabd6740 1 -- 172.21.15.98:0/2962551024 >> 172.21.15.98:0/2962551024 conn(0x558714d20960 msgr2=0x558714d1fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:55:58.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.614+0000 7fdeaabd6740 1 -- 172.21.15.98:0/2962551024 shutdown_connections 2024-05-09T08:55:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.615+0000 7fdeaabd6740 1 -- 172.21.15.98:0/2962551024 wait complete. 2024-05-09T08:55:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.616+0000 7fdeaabd6740 1 -- start start 2024-05-09T08:55:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.616+0000 7fdeaabd6740 1 Processor -- start 2024-05-09T08:55:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.617+0000 7fdeaabd6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 0x558714d37830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.617+0000 7fdeaabd6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24d70 0x558714d32ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.617+0000 7fdeaabd6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d330e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.617+0000 7fdeaabd6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde84011540 con 0x558714d2f630 2024-05-09T08:55:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.617+0000 7fdeaabd6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fde84010e70 con 0x558714d24d70 2024-05-09T08:55:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.617+0000 7fdeaabd6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fde8400e0b0 con 0x558714d24450 2024-05-09T08:55:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.618+0000 7fdeaabd6740 10 client.0 _ll_register_callbacks cb 0x7fde98019070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:55:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.618+0000 7fdea5c66640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24d70 0x558714d32ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.619+0000 7fdea5465640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.619+0000 7fdea5465640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d330e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35876/0 (socket says 172.21.15.98:35876) 2024-05-09T08:55:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.619+0000 7fdea5465640 1 -- 172.21.15.98:0/1593281797 learned_addr learned my addr 172.21.15.98:0/1593281797 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:55:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.620+0000 7fdea5465640 1 --2- 172.21.15.98:0/1593281797 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 0x558714d37830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.622+0000 7fdea5c66640 1 -- 172.21.15.98:0/1593281797 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 msgr2=0x558714d37830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.622+0000 7fdea5c66640 1 --2- 172.21.15.98:0/1593281797 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 0x558714d37830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.622+0000 7fdea5c66640 1 -- 172.21.15.98:0/1593281797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 msgr2=0x558714d330e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:55:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.622+0000 7fdea5c66640 1 --2- 172.21.15.98:0/1593281797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d330e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:55:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.622+0000 7fdea5c66640 1 -- 172.21.15.98:0/1593281797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558714d41bf0 con 0x558714d24d70 2024-05-09T08:55:58.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.628+0000 7fdea5c66640 1 --2- 172.21.15.98:0/1593281797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24d70 0x558714d32ba0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x558714d420d0 tx=0x7fde90009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44756e0ae1427b5a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:55:58.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.628+0000 7fde897fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:55:58.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.632+0000 7fde897fa640 1 -- 172.21.15.98:0/1593281797 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde90012b80 con 0x558714d24d70 2024-05-09T08:55:58.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.632+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1593281797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558714c90dc0 con 0x558714d24d70 2024-05-09T08:55:58.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.632+0000 7fde897fa640 1 -- 172.21.15.98:0/1593281797 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fde90012d20 con 0x558714d24d70 2024-05-09T08:55:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.633+0000 7fde897fa640 1 -- 172.21.15.98:0/1593281797 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fde90016050 con 0x558714d24d70 2024-05-09T08:55:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.633+0000 7fdeaabd6740 10 client.26372 fetch_fsmap learned FSMap version 35 2024-05-09T08:55:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.633+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1593281797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558714da5680 con 0x558714d24d70 2024-05-09T08:55:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.634+0000 7fde897fa640 1 -- 172.21.15.98:0/1593281797 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558714c90dc0 con 0x558714d24d70 2024-05-09T08:55:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.636+0000 7fde897fa640 1 -- 172.21.15.98:0/1593281797 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fde9001f7c0 con 0x558714d24d70 2024-05-09T08:55:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.636+0000 7fdeaabd6740 10 client.26372 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:55:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.636+0000 7fdeaabd6740 10 client.26372 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:55:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.636+0000 7fdeaabd6740 20 client.26372 populate_metadata read hostname 'smithi098' 2024-05-09T08:55:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.636+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/1593281797 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558714da5cb0 0x558714da80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:55:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.636+0000 7fdeaabd6740 4 client.26372 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:55:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.636+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1593281797 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558714c0e8e0 con 0x558714da5cb0 2024-05-09T08:55:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.636+0000 7fdea5465640 1 --2- 172.21.15.98:0/1593281797 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558714da5cb0 0x558714da80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:55:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.638+0000 7fdea5465640 1 --2- 172.21.15.98:0/1593281797 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558714da5cb0 0x558714da80d0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85dd42eceb36f45d server_cookie=496c87aec2ced5da in_seq=0 out_seq=0 2024-05-09T08:55:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:55:58.638+0000 7fde897fa640 10 client.26372 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:55:58.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:55:58 smithi105 ceph-mon[34498]: pgmap v164: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 838 MiB/s wr, 1.67k op/s 2024-05-09T08:55:58.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:55:58 smithi098 ceph-mon[30092]: pgmap v164: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 838 MiB/s wr, 1.67k op/s 2024-05-09T08:55:58.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:55:58 smithi142 ceph-mon[34625]: pgmap v164: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 838 MiB/s wr, 1.67k op/s 2024-05-09T08:56:00.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:00 smithi142 ceph-mon[34625]: pgmap v165: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 708 MiB/s wr, 1.30k op/s 2024-05-09T08:56:00.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:00 smithi105 ceph-mon[34498]: pgmap v165: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 708 MiB/s wr, 1.30k op/s 2024-05-09T08:56:00.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:00 smithi098 ceph-mon[30092]: pgmap v165: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 708 MiB/s wr, 1.30k op/s 2024-05-09T08:56:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.316+0000 7fde897fa640 1 -- 172.21.15.98:0/1593281797 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558714c0e8e0 con 0x558714da5cb0 2024-05-09T08:56:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.316+0000 7fde897fa640 10 client.26372 handle_command_reply: tid=0 2024-05-09T08:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.317+0000 7fdeaabd6740 1 client.26372 shutdown 2024-05-09T08:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.317+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1593281797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24d70 msgr2=0x558714d32ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.317+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/1593281797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24d70 0x558714d32ba0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x558714d420d0 tx=0x7fde90009f30 comp rx=0 tx=0).stop 2024-05-09T08:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.317+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1593281797 shutdown_connections 2024-05-09T08:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.317+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/1593281797 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558714da5cb0 0x558714da80d0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.317+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/1593281797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558714d2f630 0x558714d330e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.317+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/1593281797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558714d24d70 0x558714d32ba0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.317+0000 7fdeaabd6740 1 --2- 172.21.15.98:0/1593281797 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558714d24450 0x558714d37830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.317+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1593281797 >> 172.21.15.98:0/1593281797 conn(0x558714d20960 msgr2=0x558714d1fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.317+0000 7fde897fa640 0 client.26372 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.317+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1593281797 shutdown_connections 2024-05-09T08:56:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.317+0000 7fdeaabd6740 1 -- 172.21.15.98:0/1593281797 wait complete. 2024-05-09T08:56:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.318+0000 7fdeaabd6740 20 client.26372 trim_cache size 0 max 16384 2024-05-09T08:56:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:56:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:56:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:56:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:56:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.319+0000 7fdea7eca640 1 -- 172.21.15.98:0/4243830257 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fde8c0a2fc0 msgr2=0x7fde8c0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.319+0000 7fdea7eca640 1 --2- 172.21.15.98:0/4243830257 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fde8c0a2fc0 0x7fde8c0a5480 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fde9c039590 tx=0x7fde9c009260 comp rx=0 tx=0).stop 2024-05-09T08:56:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.319+0000 7fdea7eca640 1 -- 172.21.15.98:0/4243830257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdea01ab8a0 msgr2=0x7fdea01abd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.319+0000 7fdea7eca640 1 --2- 172.21.15.98:0/4243830257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdea01ab8a0 0x7fdea01abd50 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fde9c02aae0 tx=0x7fde9c039520 comp rx=0 tx=0).stop 2024-05-09T08:56:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.319+0000 7fdea7eca640 1 -- 172.21.15.98:0/4243830257 shutdown_connections 2024-05-09T08:56:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.319+0000 7fdea7eca640 1 --2- 172.21.15.98:0/4243830257 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fde8c0a2fc0 0x7fde8c0a5480 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.319+0000 7fdea7eca640 1 --2- 172.21.15.98:0/4243830257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdea01ab8a0 0x7fdea01abd50 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:02.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.319+0000 7fdea7eca640 1 --2- 172.21.15.98:0/4243830257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdea011b010 0x7fdea01a7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:02.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.319+0000 7fdea7eca640 1 --2- 172.21.15.98:0/4243830257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdea0110020 0x7fdea01a6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:02.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.319+0000 7fdea7eca640 1 -- 172.21.15.98:0/4243830257 >> 172.21.15.98:0/4243830257 conn(0x7fdea006c900 msgr2=0x7fdea010dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:02.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.319+0000 7fdea7eca640 1 -- 172.21.15.98:0/4243830257 shutdown_connections 2024-05-09T08:56:02.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:02.319+0000 7fdea7eca640 1 -- 172.21.15.98:0/4243830257 wait complete. 2024-05-09T08:56:02.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:02 smithi105 ceph-mon[34498]: pgmap v166: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 904 MiB/s wr, 1.66k op/s 2024-05-09T08:56:02.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:02 smithi098 ceph-mon[30092]: pgmap v166: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 904 MiB/s wr, 1.66k op/s 2024-05-09T08:56:02.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:02 smithi142 ceph-mon[34625]: pgmap v166: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 904 MiB/s wr, 1.66k op/s 2024-05-09T08:56:04.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:04 smithi105 ceph-mon[34498]: pgmap v167: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 629 MiB/s wr, 1.22k op/s 2024-05-09T08:56:04.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:04 smithi098 ceph-mon[30092]: pgmap v167: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 629 MiB/s wr, 1.22k op/s 2024-05-09T08:56:04.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:04 smithi142 ceph-mon[34625]: pgmap v167: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 629 MiB/s wr, 1.22k op/s 2024-05-09T08:56:06.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:06 smithi105 ceph-mon[34498]: pgmap v168: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 777 MiB/s wr, 1.45k op/s 2024-05-09T08:56:06.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:06 smithi098 ceph-mon[30092]: pgmap v168: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 777 MiB/s wr, 1.45k op/s 2024-05-09T08:56:06.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:06 smithi142 ceph-mon[34625]: pgmap v168: 129 pgs: 129 active+clean; 40 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 777 MiB/s wr, 1.45k op/s 2024-05-09T08:56:08.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:08 smithi105 ceph-mon[34498]: pgmap v169: 129 pgs: 129 active+clean; 40 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 836 MiB/s wr, 1.41k op/s 2024-05-09T08:56:08.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:08 smithi098 ceph-mon[30092]: pgmap v169: 129 pgs: 129 active+clean; 40 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 836 MiB/s wr, 1.41k op/s 2024-05-09T08:56:08.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:08 smithi142 ceph-mon[34625]: pgmap v169: 129 pgs: 129 active+clean; 40 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 836 MiB/s wr, 1.41k op/s 2024-05-09T08:56:09.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:56:09.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:56:10.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.235+0000 7efcaa5a9640 1 -- 172.21.15.98:0/2626760742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca410d170 msgr2=0x7efca4079b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:10.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.235+0000 7efcaa5a9640 1 --2- 172.21.15.98:0/2626760742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca410d170 0x7efca4079b40 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009ca0 tx=0x7efc9c02a980 comp rx=0 tx=0).stop 2024-05-09T08:56:10.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.236+0000 7efcaa5a9640 1 -- 172.21.15.98:0/2626760742 shutdown_connections 2024-05-09T08:56:10.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.236+0000 7efcaa5a9640 1 --2- 172.21.15.98:0/2626760742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca410d170 0x7efca4079b40 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.236+0000 7efcaa5a9640 1 --2- 172.21.15.98:0/2626760742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efca41093c0 0x7efca410ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.236+0000 7efcaa5a9640 1 --2- 172.21.15.98:0/2626760742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efca4108aa0 0x7efca4108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.236+0000 7efcaa5a9640 1 -- 172.21.15.98:0/2626760742 >> 172.21.15.98:0/2626760742 conn(0x7efca406c940 msgr2=0x7efca406cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:10.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.243+0000 7efcaa5a9640 1 -- 172.21.15.98:0/2626760742 shutdown_connections 2024-05-09T08:56:10.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.246+0000 7efcaa5a9640 1 -- 172.21.15.98:0/2626760742 wait complete. 2024-05-09T08:56:10.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.247+0000 7efcaa5a9640 1 Processor -- start 2024-05-09T08:56:10.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.251+0000 7efcaa5a9640 1 -- start start 2024-05-09T08:56:10.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.252+0000 7efcaa5a9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca4108aa0 0x7efca407e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.252+0000 7efcaa5a9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efca41093c0 0x7efca407ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.252+0000 7efcaa5a9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efca410d170 0x7efca407f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.252+0000 7efcaa5a9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efca407c570 con 0x7efca4108aa0 2024-05-09T08:56:10.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.252+0000 7efcaa5a9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efca407c3f0 con 0x7efca41093c0 2024-05-09T08:56:10.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.252+0000 7efcaa5a9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efca407c6f0 con 0x7efca410d170 2024-05-09T08:56:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.255+0000 7efca8b46640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efca410d170 0x7efca407f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.255+0000 7efca3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca4108aa0 0x7efca407e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.255+0000 7efca3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca4108aa0 0x7efca407e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39352/0 (socket says 172.21.15.98:39352) 2024-05-09T08:56:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.255+0000 7efca3fff640 1 -- 172.21.15.98:0/1023310340 learned_addr learned my addr 172.21.15.98:0/1023310340 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.256+0000 7efca37fe640 1 --2- 172.21.15.98:0/1023310340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efca41093c0 0x7efca407ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.256+0000 7efca3fff640 1 -- 172.21.15.98:0/1023310340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efca410d170 msgr2=0x7efca407f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.256+0000 7efca3fff640 1 --2- 172.21.15.98:0/1023310340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efca410d170 0x7efca407f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.256+0000 7efca3fff640 1 -- 172.21.15.98:0/1023310340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efca41093c0 msgr2=0x7efca407ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.256+0000 7efca3fff640 1 --2- 172.21.15.98:0/1023310340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efca41093c0 0x7efca407ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.256+0000 7efca3fff640 1 -- 172.21.15.98:0/1023310340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca41c42a0 con 0x7efca4108aa0 2024-05-09T08:56:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.259+0000 7efca3fff640 1 --2- 172.21.15.98:0/1023310340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca4108aa0 0x7efca407e6c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7efc9800e9f0 tx=0x7efc9800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.260+0000 7efca17fa640 1 -- 172.21.15.98:0/1023310340 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9800cd80 con 0x7efca4108aa0 2024-05-09T08:56:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.260+0000 7efca17fa640 1 -- 172.21.15.98:0/1023310340 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc98010910 con 0x7efca4108aa0 2024-05-09T08:56:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.260+0000 7efca17fa640 1 -- 172.21.15.98:0/1023310340 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc98017a20 con 0x7efca4108aa0 2024-05-09T08:56:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.265+0000 7efcaa5a9640 1 -- 172.21.15.98:0/1023310340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca41c4590 con 0x7efca4108aa0 2024-05-09T08:56:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.265+0000 7efcaa5a9640 1 -- 172.21.15.98:0/1023310340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca41c4a90 con 0x7efca4108aa0 2024-05-09T08:56:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.269+0000 7efcad397740 1 Processor -- start 2024-05-09T08:56:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.269+0000 7efcad397740 1 -- start start 2024-05-09T08:56:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.270+0000 7efcad397740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890280 0x55c686890660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.270+0000 7efcad397740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c686890ba0 0x55c68689ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.270+0000 7efcad397740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c68689b270 0x55c68689d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.270+0000 7efcad397740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c6867f5d00 con 0x55c68689b270 2024-05-09T08:56:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.270+0000 7efcad397740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c68679d4c0 con 0x55c686890ba0 2024-05-09T08:56:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.270+0000 7efcad397740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c6867f1cc0 con 0x55c686890280 2024-05-09T08:56:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.270+0000 7efca8b46640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890280 0x55c686890660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.270+0000 7efca8b46640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890280 0x55c686890660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39314/0 (socket says 172.21.15.98:39314) 2024-05-09T08:56:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.270+0000 7efca8b46640 1 -- 172.21.15.98:0/4073319013 learned_addr learned my addr 172.21.15.98:0/4073319013 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:10.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.273+0000 7efca3fff640 1 --2- 172.21.15.98:0/4073319013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c686890ba0 0x55c68689ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.275+0000 7efca8b46640 1 -- 172.21.15.98:0/4073319013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c686890ba0 msgr2=0x55c68689ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:10.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.275+0000 7efca8b46640 1 --2- 172.21.15.98:0/4073319013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c686890ba0 0x55c68689ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.275+0000 7efca8b46640 1 -- 172.21.15.98:0/4073319013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c68689b270 msgr2=0x55c68689d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:10.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.275+0000 7efca8b46640 1 --2- 172.21.15.98:0/4073319013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c68689b270 0x55c68689d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.275+0000 7efca8b46640 1 -- 172.21.15.98:0/4073319013 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6865d37d0 con 0x55c686890280 2024-05-09T08:56:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.284+0000 7efca17fa640 1 -- 172.21.15.98:0/1023310340 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7efc98010ac0 con 0x7efca4108aa0 2024-05-09T08:56:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.286+0000 7efca17fa640 1 --2- 172.21.15.98:0/1023310340 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efc8c0a2f00 0x7efc8c0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.286+0000 7efca17fa640 1 -- 172.21.15.98:0/1023310340 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7efc980d5890 con 0x7efca4108aa0 2024-05-09T08:56:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.286+0000 7efca8b46640 1 --2- 172.21.15.98:0/4073319013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890280 0x55c686890660 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009ca0 tx=0x7efc9c02fc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=873ff359d2057aa5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:10.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.289+0000 7efca3fff640 1 --2- 172.21.15.98:0/1023310340 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efc8c0a2f00 0x7efc8c0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.290+0000 7efca3fff640 1 --2- 172.21.15.98:0/1023310340 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efc8c0a2f00 0x7efc8c0a53c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x55c686891840 tx=0x7efc9800cf20 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:10.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.292+0000 7efc8a7fc640 1 -- 172.21.15.98:0/4073319013 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c048070 con 0x55c686890280 2024-05-09T08:56:10.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.292+0000 7efc8a7fc640 1 -- 172.21.15.98:0/4073319013 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc9c02fe80 con 0x55c686890280 2024-05-09T08:56:10.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.292+0000 7efc8a7fc640 1 -- 172.21.15.98:0/4073319013 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c034070 con 0x55c686890280 2024-05-09T08:56:10.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.296+0000 7efcad397740 1 -- 172.21.15.98:0/4073319013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890280 msgr2=0x55c686890660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:10.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.296+0000 7efcad397740 1 --2- 172.21.15.98:0/4073319013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890280 0x55c686890660 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009ca0 tx=0x7efc9c02fc50 comp rx=0 tx=0).stop 2024-05-09T08:56:10.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.298+0000 7efcad397740 1 -- 172.21.15.98:0/4073319013 shutdown_connections 2024-05-09T08:56:10.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.298+0000 7efcad397740 1 --2- 172.21.15.98:0/4073319013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c68689b270 0x55c68689d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.298+0000 7efcad397740 1 --2- 172.21.15.98:0/4073319013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c686890ba0 0x55c68689ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.298+0000 7efcad397740 1 --2- 172.21.15.98:0/4073319013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890280 0x55c686890660 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.298+0000 7efcad397740 1 -- 172.21.15.98:0/4073319013 >> 172.21.15.98:0/4073319013 conn(0x55c68688c5a0 msgr2=0x55c68688c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:10.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.305+0000 7efcad397740 1 -- 172.21.15.98:0/4073319013 shutdown_connections 2024-05-09T08:56:10.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.305+0000 7efcad397740 1 -- 172.21.15.98:0/4073319013 wait complete. 2024-05-09T08:56:10.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.306+0000 7efcad397740 1 -- start start 2024-05-09T08:56:10.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.306+0000 7efcad397740 1 Processor -- start 2024-05-09T08:56:10.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.307+0000 7efcad397740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c686890280 0x55c686898260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.307+0000 7efcad397740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 0x55c6868987a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.307+0000 7efcad397740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c68689b270 0x55c686898ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.307+0000 7efcad397740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c68689e000 con 0x55c68689b270 2024-05-09T08:56:10.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.307+0000 7efcad397740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c6867b0b90 con 0x55c686890280 2024-05-09T08:56:10.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.307+0000 7efcad397740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c68689e180 con 0x55c686890ba0 2024-05-09T08:56:10.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.307+0000 7efcad397740 10 client.0 _ll_register_callbacks cb 0x55c6867ecda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:10.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.308+0000 7efca8b46640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c686890280 0x55c686898260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.309+0000 7efca8b46640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c686890280 0x55c686898260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52240/0 (socket says 172.21.15.98:52240) 2024-05-09T08:56:10.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.309+0000 7efca8b46640 1 -- 172.21.15.98:0/2429522519 learned_addr learned my addr 172.21.15.98:0/2429522519 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:10.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.309+0000 7efca8b46640 1 --2- 172.21.15.98:0/2429522519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c68689b270 0x55c686898ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.309+0000 7efca37fe640 1 --2- 172.21.15.98:0/2429522519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 0x55c6868987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.309+0000 7efca8b46640 1 -- 172.21.15.98:0/2429522519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 msgr2=0x55c6868987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:10.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.309+0000 7efca8b46640 1 --2- 172.21.15.98:0/2429522519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 0x55c6868987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.309+0000 7efca8b46640 1 -- 172.21.15.98:0/2429522519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c68689b270 msgr2=0x55c686898ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:10.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.309+0000 7efca8b46640 1 --2- 172.21.15.98:0/2429522519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c68689b270 0x55c686898ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.309+0000 7efca8b46640 1 -- 172.21.15.98:0/2429522519 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6865093f0 con 0x55c686890280 2024-05-09T08:56:10.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.311+0000 7efca8b46640 1 --2- 172.21.15.98:0/2429522519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c686890280 0x55c686898260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009620 tx=0x7efc9c030680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d03b5c009148b62 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:10.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.313+0000 7efc837fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:56:10.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.313+0000 7efc837fe640 1 -- 172.21.15.98:0/2429522519 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c048070 con 0x55c686890280 2024-05-09T08:56:10.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.313+0000 7efcad397740 1 -- 172.21.15.98:0/2429522519 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc9c02fe80 con 0x55c686890280 2024-05-09T08:56:10.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.313+0000 7efc837fe640 1 -- 172.21.15.98:0/2429522519 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7efc9c02f5d0 con 0x55c686890280 2024-05-09T08:56:10.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.317+0000 7efc837fe640 1 -- 172.21.15.98:0/2429522519 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efc9c034070 con 0x55c686890280 2024-05-09T08:56:10.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.318+0000 7efcad397740 10 client.26381 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:10.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.318+0000 7efcad397740 1 -- 172.21.15.98:0/2429522519 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c6864c90f0 con 0x55c686890280 2024-05-09T08:56:10.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.320+0000 7efc837fe640 1 -- 172.21.15.98:0/2429522519 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc9c02fe80 con 0x55c686890280 2024-05-09T08:56:10.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.320+0000 7efc837fe640 1 -- 172.21.15.98:0/2429522519 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7efc9c043420 con 0x55c686890280 2024-05-09T08:56:10.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.320+0000 7efcad397740 10 client.26381 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:10.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.320+0000 7efcad397740 10 client.26381 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:10.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.320+0000 7efcad397740 20 client.26381 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:10.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.320+0000 7efcad397740 1 --2- 172.21.15.98:0/2429522519 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c6868ab6c0 0x55c6868adb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.320+0000 7efcad397740 4 client.26381 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:10.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.320+0000 7efcad397740 1 -- 172.21.15.98:0/2429522519 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c686803430 con 0x55c6868ab6c0 2024-05-09T08:56:10.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.321+0000 7efca37fe640 1 --2- 172.21.15.98:0/2429522519 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c6868ab6c0 0x55c6868adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.323+0000 7efca37fe640 1 --2- 172.21.15.98:0/2429522519 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c6868ab6c0 0x55c6868adb00 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60a1435cf4177999 server_cookie=e002bfffa90ef645 in_seq=0 out_seq=0 2024-05-09T08:56:10.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.323+0000 7efc837fe640 10 client.26381 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:10.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.325+0000 7efc837fe640 1 -- 172.21.15.98:0/2429522519 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c686803430 con 0x55c6868ab6c0 2024-05-09T08:56:10.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.325+0000 7efc837fe640 10 client.26381 handle_command_reply: tid=0 2024-05-09T08:56:10.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.325+0000 7efcad397740 1 client.26381 shutdown 2024-05-09T08:56:10.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.325+0000 7efcad397740 1 -- 172.21.15.98:0/2429522519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c686890280 msgr2=0x55c686898260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:10.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.325+0000 7efcad397740 1 --2- 172.21.15.98:0/2429522519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c686890280 0x55c686898260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009620 tx=0x7efc9c030680 comp rx=0 tx=0).stop 2024-05-09T08:56:10.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.325+0000 7efcad397740 1 -- 172.21.15.98:0/2429522519 shutdown_connections 2024-05-09T08:56:10.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.325+0000 7efcad397740 1 --2- 172.21.15.98:0/2429522519 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c6868ab6c0 0x55c6868adb00 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.325+0000 7efcad397740 1 --2- 172.21.15.98:0/2429522519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c68689b270 0x55c686898ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.325+0000 7efcad397740 1 --2- 172.21.15.98:0/2429522519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 0x55c6868987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.325+0000 7efcad397740 1 --2- 172.21.15.98:0/2429522519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c686890280 0x55c686898260 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.325+0000 7efcad397740 1 -- 172.21.15.98:0/2429522519 >> 172.21.15.98:0/2429522519 conn(0x55c68688c5a0 msgr2=0x55c6868923a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:10.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.325+0000 7efc837fe640 0 client.26381 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.326+0000 7efcad397740 1 -- 172.21.15.98:0/2429522519 shutdown_connections 2024-05-09T08:56:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.326+0000 7efcad397740 1 -- 172.21.15.98:0/2429522519 wait complete. 2024-05-09T08:56:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.326+0000 7efcad397740 20 client.26381 trim_cache size 0 max 16384 2024-05-09T08:56:10.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.361+0000 7efcad397740 1 Processor -- start 2024-05-09T08:56:10.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.361+0000 7efcad397740 1 -- start start 2024-05-09T08:56:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.362+0000 7efcad397740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c686890280 0x55c686893550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.362+0000 7efcad397740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 0x55c686892a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.362+0000 7efca8b46640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c686890280 0x55c686893550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.362+0000 7efca8b46640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c686890280 0x55c686893550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39366/0 (socket says 172.21.15.98:39366) 2024-05-09T08:56:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.363+0000 7efca8b46640 1 -- 172.21.15.98:0/3463354111 learned_addr learned my addr 172.21.15.98:0/3463354111 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.363+0000 7efcad397740 1 --2- 172.21.15.98:0/3463354111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c68689b270 0x55c6868b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.363+0000 7efcad397740 1 -- 172.21.15.98:0/3463354111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efca41ceeb0 con 0x55c686890280 2024-05-09T08:56:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.363+0000 7efcad397740 1 -- 172.21.15.98:0/3463354111 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efca41d09f0 con 0x55c68689b270 2024-05-09T08:56:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.363+0000 7efcad397740 1 -- 172.21.15.98:0/3463354111 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efca41cf580 con 0x55c686890ba0 2024-05-09T08:56:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.363+0000 7efca37fe640 1 --2- 172.21.15.98:0/3463354111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 0x55c686892a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.363+0000 7efca8b46640 1 --2- 172.21.15.98:0/3463354111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c68689b270 0x55c6868b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.363+0000 7efca37fe640 1 -- 172.21.15.98:0/3463354111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c68689b270 msgr2=0x55c6868b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.363+0000 7efca37fe640 1 --2- 172.21.15.98:0/3463354111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c68689b270 0x55c6868b3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.363+0000 7efca37fe640 1 -- 172.21.15.98:0/3463354111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c686890280 msgr2=0x55c686893550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.363+0000 7efca37fe640 1 --2- 172.21.15.98:0/3463354111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c686890280 0x55c686893550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.363+0000 7efca37fe640 1 -- 172.21.15.98:0/3463354111 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6868967d0 con 0x55c686890ba0 2024-05-09T08:56:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.363+0000 7efca8b46640 1 --2- 172.21.15.98:0/3463354111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c68689b270 0x55c6868b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:56:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.364+0000 7efca37fe640 1 --2- 172.21.15.98:0/3463354111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 0x55c686892a20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efc94006fd0 tx=0x7efc9400b6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=debdfba69ab4a83e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.364+0000 7efc89ffb640 1 -- 172.21.15.98:0/3463354111 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc94014590 con 0x55c686890ba0 2024-05-09T08:56:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.365+0000 7efc89ffb640 1 -- 172.21.15.98:0/3463354111 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc94014c10 con 0x55c686890ba0 2024-05-09T08:56:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.366+0000 7efcad397740 1 -- 172.21.15.98:0/3463354111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 msgr2=0x55c686892a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.366+0000 7efcad397740 1 --2- 172.21.15.98:0/3463354111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 0x55c686892a20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efc94006fd0 tx=0x7efc9400b6c0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.367+0000 7efcad397740 1 -- 172.21.15.98:0/3463354111 shutdown_connections 2024-05-09T08:56:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.367+0000 7efcad397740 1 --2- 172.21.15.98:0/3463354111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c68689b270 0x55c6868b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.367+0000 7efcad397740 1 --2- 172.21.15.98:0/3463354111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 0x55c686892a20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.367+0000 7efcad397740 1 --2- 172.21.15.98:0/3463354111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c686890280 0x55c686893550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.367+0000 7efcad397740 1 -- 172.21.15.98:0/3463354111 >> 172.21.15.98:0/3463354111 conn(0x55c68688c5a0 msgr2=0x55c68688af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:10.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.367+0000 7efcad397740 1 -- 172.21.15.98:0/3463354111 shutdown_connections 2024-05-09T08:56:10.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.367+0000 7efcad397740 1 -- 172.21.15.98:0/3463354111 wait complete. 2024-05-09T08:56:10.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.368+0000 7efcad397740 1 -- start start 2024-05-09T08:56:10.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.368+0000 7efcad397740 1 Processor -- start 2024-05-09T08:56:10.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.368+0000 7efcad397740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c686890280 0x55c6868a3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.369+0000 7efcad397740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 0x55c6868a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.369+0000 7efcad397740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c68689b270 0x55c686894a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.369+0000 7efcad397740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efca41d0320 con 0x55c686890280 2024-05-09T08:56:10.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.369+0000 7efca8b46640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c68689b270 0x55c686894a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.369+0000 7efca8b46640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c68689b270 0x55c686894a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52248/0 (socket says 172.21.15.98:52248) 2024-05-09T08:56:10.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.369+0000 7efca8b46640 1 -- 172.21.15.98:0/3496985183 learned_addr learned my addr 172.21.15.98:0/3496985183 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:10.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.369+0000 7efcad397740 1 -- 172.21.15.98:0/3496985183 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efca41cfc50 con 0x55c68689b270 2024-05-09T08:56:10.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.369+0000 7efca37fe640 1 --2- 172.21.15.98:0/3496985183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 0x55c6868a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.369+0000 7efcad397740 1 -- 172.21.15.98:0/3496985183 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efca41cce90 con 0x55c686890ba0 2024-05-09T08:56:10.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.369+0000 7efcad397740 10 client.0 _ll_register_callbacks cb 0x7efc9c046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:10.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.370+0000 7efca8b46640 1 --2- 172.21.15.98:0/3496985183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c686890280 0x55c6868a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.370+0000 7efca37fe640 1 -- 172.21.15.98:0/3496985183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c68689b270 msgr2=0x55c686894a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:10.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.370+0000 7efca37fe640 1 --2- 172.21.15.98:0/3496985183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c68689b270 0x55c686894a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.370+0000 7efca37fe640 1 -- 172.21.15.98:0/3496985183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c686890280 msgr2=0x55c6868a3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:10.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.370+0000 7efca37fe640 1 --2- 172.21.15.98:0/3496985183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c686890280 0x55c6868a3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.370+0000 7efca37fe640 1 -- 172.21.15.98:0/3496985183 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c686895250 con 0x55c686890ba0 2024-05-09T08:56:10.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.370+0000 7efca8b46640 1 --2- 172.21.15.98:0/3496985183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c686890280 0x55c6868a3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:56:10.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.370+0000 7efca37fe640 1 --2- 172.21.15.98:0/3496985183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 0x55c6868a3940 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efc94008010 tx=0x7efc94014ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e6af7052754aff3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:10.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.370+0000 7efc837fe640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:56:10.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.370+0000 7efc837fe640 1 -- 172.21.15.98:0/3496985183 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc94028070 con 0x55c686890ba0 2024-05-09T08:56:10.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.370+0000 7efc837fe640 1 -- 172.21.15.98:0/3496985183 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7efc94014590 con 0x55c686890ba0 2024-05-09T08:56:10.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.372+0000 7efcad397740 1 -- 172.21.15.98:0/3496985183 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc94014c10 con 0x55c686890ba0 2024-05-09T08:56:10.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.372+0000 7efc837fe640 1 -- 172.21.15.98:0/3496985183 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efc94032050 con 0x55c686890ba0 2024-05-09T08:56:10.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.383+0000 7efcad397740 10 client.26356 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:10.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.383+0000 7efcad397740 1 -- 172.21.15.98:0/3496985183 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c686911880 con 0x55c686890ba0 2024-05-09T08:56:10.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.383+0000 7efc837fe640 1 -- 172.21.15.98:0/3496985183 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc94014c10 con 0x55c686890ba0 2024-05-09T08:56:10.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.384+0000 7efc837fe640 1 -- 172.21.15.98:0/3496985183 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7efc94003820 con 0x55c686890ba0 2024-05-09T08:56:10.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.384+0000 7efcad397740 10 client.26356 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:10.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.384+0000 7efcad397740 10 client.26356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:10.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.384+0000 7efcad397740 20 client.26356 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:10.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.384+0000 7efcad397740 1 --2- 172.21.15.98:0/3496985183 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c686911ea0 0x55c6869142c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.384+0000 7efcad397740 4 client.26356 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:56:10.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.384+0000 7efcad397740 1 -- 172.21.15.98:0/3496985183 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c6867fd780 con 0x55c686911ea0 2024-05-09T08:56:10.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.389+0000 7efca8b46640 1 --2- 172.21.15.98:0/3496985183 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c686911ea0 0x55c6869142c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.390+0000 7efc837fe640 10 client.26356 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:10.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.390+0000 7efca8b46640 1 --2- 172.21.15.98:0/3496985183 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c686911ea0 0x55c6869142c0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f8e6cfcc0f2c338 server_cookie=c4f68dbc19cf86de in_seq=0 out_seq=0 2024-05-09T08:56:10.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.393+0000 7efc837fe640 1 -- 172.21.15.98:0/3496985183 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c6867fd780 con 0x55c686911ea0 2024-05-09T08:56:10.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.393+0000 7efc837fe640 10 client.26356 handle_command_reply: tid=0 2024-05-09T08:56:10.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.393+0000 7efcad397740 1 client.26356 shutdown 2024-05-09T08:56:10.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.398+0000 7efcad397740 1 -- 172.21.15.98:0/3496985183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 msgr2=0x55c6868a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:10.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.398+0000 7efcad397740 1 --2- 172.21.15.98:0/3496985183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 0x55c6868a3940 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efc94008010 tx=0x7efc94014ed0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.398+0000 7efcad397740 1 -- 172.21.15.98:0/3496985183 shutdown_connections 2024-05-09T08:56:10.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.398+0000 7efcad397740 1 --2- 172.21.15.98:0/3496985183 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c686911ea0 0x55c6869142c0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.398+0000 7efcad397740 1 --2- 172.21.15.98:0/3496985183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c68689b270 0x55c686894a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.398+0000 7efcad397740 1 --2- 172.21.15.98:0/3496985183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c686890ba0 0x55c6868a3940 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.398+0000 7efcad397740 1 --2- 172.21.15.98:0/3496985183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c686890280 0x55c6868a3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.398+0000 7efcad397740 1 -- 172.21.15.98:0/3496985183 >> 172.21.15.98:0/3496985183 conn(0x55c68688c5a0 msgr2=0x55c68688b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:10.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.398+0000 7efc837fe640 0 client.26356 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.398+0000 7efcad397740 1 -- 172.21.15.98:0/3496985183 shutdown_connections 2024-05-09T08:56:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.398+0000 7efcad397740 1 -- 172.21.15.98:0/3496985183 wait complete. 2024-05-09T08:56:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.399+0000 7efcad397740 20 client.26356 trim_cache size 0 max 16384 2024-05-09T08:56:10.399 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:56:10.399 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:56:10.399 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:56:10.399 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:56:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.401+0000 7efcaa5a9640 1 -- 172.21.15.98:0/1023310340 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efc8c0a2f00 msgr2=0x7efc8c0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.401+0000 7efcaa5a9640 1 --2- 172.21.15.98:0/1023310340 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efc8c0a2f00 0x7efc8c0a53c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x55c686891840 tx=0x7efc9800cf20 comp rx=0 tx=0).stop 2024-05-09T08:56:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.401+0000 7efcaa5a9640 1 -- 172.21.15.98:0/1023310340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca4108aa0 msgr2=0x7efca407e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.401+0000 7efcaa5a9640 1 --2- 172.21.15.98:0/1023310340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca4108aa0 0x7efca407e6c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7efc9800e9f0 tx=0x7efc9800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.401+0000 7efcaa5a9640 1 -- 172.21.15.98:0/1023310340 shutdown_connections 2024-05-09T08:56:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.401+0000 7efcaa5a9640 1 --2- 172.21.15.98:0/1023310340 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efc8c0a2f00 0x7efc8c0a53c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.401+0000 7efcaa5a9640 1 --2- 172.21.15.98:0/1023310340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efca410d170 0x7efca407f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.401+0000 7efcaa5a9640 1 --2- 172.21.15.98:0/1023310340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efca41093c0 0x7efca407ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.401+0000 7efcaa5a9640 1 --2- 172.21.15.98:0/1023310340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efca4108aa0 0x7efca407e6c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.401+0000 7efcaa5a9640 1 -- 172.21.15.98:0/1023310340 >> 172.21.15.98:0/1023310340 conn(0x7efca406c940 msgr2=0x7efca40717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.402+0000 7efcaa5a9640 1 -- 172.21.15.98:0/1023310340 shutdown_connections 2024-05-09T08:56:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.402+0000 7efcaa5a9640 1 -- 172.21.15.98:0/1023310340 wait complete. 2024-05-09T08:56:10.428 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:56:10.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:56:10.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:10 smithi105 ceph-mon[34498]: pgmap v170: 129 pgs: 129 active+clean; 40 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 706 MiB/s wr, 1.20k op/s 2024-05-09T08:56:10.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.755+0000 7eff6959a640 1 -- 172.21.15.98:0/2340309319 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff64031940 con 0x7eff6c1086f0 2024-05-09T08:56:10.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.755+0000 7eff6b59e640 1 -- 172.21.15.98:0/2340309319 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c1086f0 msgr2=0x7eff6c114b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:10.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.755+0000 7eff6b59e640 1 --2- 172.21.15.98:0/2340309319 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c1086f0 0x7eff6c114b30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7eff64009930 tx=0x7eff6402a480 comp rx=0 tx=0).stop 2024-05-09T08:56:10.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.757+0000 7eff6b59e640 1 -- 172.21.15.98:0/2340309319 shutdown_connections 2024-05-09T08:56:10.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.757+0000 7eff6b59e640 1 --2- 172.21.15.98:0/2340309319 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c1086f0 0x7eff6c114b30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.757+0000 7eff6b59e640 1 --2- 172.21.15.98:0/2340309319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff6c107d70 0x7eff6c1081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.757+0000 7eff6b59e640 1 --2- 172.21.15.98:0/2340309319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c10dd00 0x7eff6c10e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.757+0000 7eff6b59e640 1 -- 172.21.15.98:0/2340309319 >> 172.21.15.98:0/2340309319 conn(0x7eff6c077650 msgr2=0x7eff6c106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.765+0000 7eff6b59e640 1 -- 172.21.15.98:0/2340309319 shutdown_connections 2024-05-09T08:56:10.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.768+0000 7eff6b59e640 1 -- 172.21.15.98:0/2340309319 wait complete. 2024-05-09T08:56:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.770+0000 7eff6b59e640 1 Processor -- start 2024-05-09T08:56:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.772+0000 7eff6b59e640 1 -- start start 2024-05-09T08:56:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.772+0000 7eff6b59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c107d70 0x7eff6c19e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.772+0000 7eff6b59e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff6c1086f0 0x7eff6c19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.772+0000 7eff6b59e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c10dd00 0x7eff6c1a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.772+0000 7eff6b59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c1180e0 con 0x7eff6c107d70 2024-05-09T08:56:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.772+0000 7eff6b59e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c117f60 con 0x7eff6c1086f0 2024-05-09T08:56:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.772+0000 7eff6b59e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c118260 con 0x7eff6c10dd00 2024-05-09T08:56:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.773+0000 7eff6a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c107d70 0x7eff6c19e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.774+0000 7eff6a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c107d70 0x7eff6c19e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39410/0 (socket says 172.21.15.98:39410) 2024-05-09T08:56:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.774+0000 7eff6a59c640 1 -- 172.21.15.98:0/946093865 learned_addr learned my addr 172.21.15.98:0/946093865 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.774+0000 7eff69d9b640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff6c1086f0 0x7eff6c19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.775+0000 7eff6ad9d640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c10dd00 0x7eff6c1a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.775+0000 7eff6a59c640 1 -- 172.21.15.98:0/946093865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c10dd00 msgr2=0x7eff6c1a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.775+0000 7eff6a59c640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c10dd00 0x7eff6c1a2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.775+0000 7eff6a59c640 1 -- 172.21.15.98:0/946093865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff6c1086f0 msgr2=0x7eff6c19eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.775+0000 7eff6a59c640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff6c1086f0 0x7eff6c19eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.775+0000 7eff6a59c640 1 -- 172.21.15.98:0/946093865 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6c1a3530 con 0x7eff6c107d70 2024-05-09T08:56:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.775+0000 7eff69d9b640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff6c1086f0 0x7eff6c19eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:56:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.778+0000 7eff6a59c640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c107d70 0x7eff6c19e5c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00db80 tx=0x7eff5c00b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.784+0000 7eff637fe640 1 -- 172.21.15.98:0/946093865 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c017070 con 0x7eff6c107d70 2024-05-09T08:56:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.785+0000 7eff637fe640 1 -- 172.21.15.98:0/946093865 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff5c003b70 con 0x7eff6c107d70 2024-05-09T08:56:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.785+0000 7eff637fe640 1 -- 172.21.15.98:0/946093865 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0106c0 con 0x7eff6c107d70 2024-05-09T08:56:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.786+0000 7eff6b59e640 1 -- 172.21.15.98:0/946093865 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff6c1a3820 con 0x7eff6c107d70 2024-05-09T08:56:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.786+0000 7eff6b59e640 1 -- 172.21.15.98:0/946093865 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff6c07a080 con 0x7eff6c107d70 2024-05-09T08:56:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.789+0000 7eff6b59e640 1 -- 172.21.15.98:0/946093865 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff6c1093a0 con 0x7eff6c107d70 2024-05-09T08:56:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.792+0000 7eff637fe640 1 -- 172.21.15.98:0/946093865 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7eff5c004a40 con 0x7eff6c107d70 2024-05-09T08:56:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.794+0000 7eff637fe640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7eff580a2e80 0x7eff580a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.795+0000 7eff637fe640 1 -- 172.21.15.98:0/946093865 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7eff5c0cf6f0 con 0x7eff6c107d70 2024-05-09T08:56:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.795+0000 7eff69d9b640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7eff580a2e80 0x7eff580a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.798+0000 7eff69d9b640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7eff580a2e80 0x7eff580a5340 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7eff6c10d460 tx=0x7eff54008040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:10.804+0000 7eff637fe640 1 -- 172.21.15.98:0/946093865 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7eff5c020070 con 0x7eff6c107d70 2024-05-09T08:56:10.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:10 smithi098 ceph-mon[30092]: pgmap v170: 129 pgs: 129 active+clean; 40 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 706 MiB/s wr, 1.20k op/s 2024-05-09T08:56:10.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:10 smithi142 ceph-mon[34625]: pgmap v170: 129 pgs: 129 active+clean; 40 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 706 MiB/s wr, 1.20k op/s 2024-05-09T08:56:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.355+0000 7eff6b59e640 1 -- 172.21.15.98:0/946093865 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff6c19f8c0 con 0x7eff6c107d70 2024-05-09T08:56:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.357+0000 7eff637fe640 1 -- 172.21.15.98:0/946093865 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11944 (secure 0 0 0) 0x7eff5c09c050 con 0x7eff6c107d70 2024-05-09T08:56:11.360 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:56:11.360 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":16641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:54:25.881641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16749":{"gid":16749,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/3204012039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3204012039},{"type":"v1","addr":"172.21.15.105:6835","nonce":3204012039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24439,14667,24388,24461,24457]},"id":1}]} 2024-05-09T08:56:11.360 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 35 2024-05-09T08:56:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.364+0000 7eff6b59e640 1 -- 172.21.15.98:0/946093865 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7eff580a2e80 msgr2=0x7eff580a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.364+0000 7eff6b59e640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7eff580a2e80 0x7eff580a5340 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7eff6c10d460 tx=0x7eff54008040 comp rx=0 tx=0).stop 2024-05-09T08:56:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.364+0000 7eff6b59e640 1 -- 172.21.15.98:0/946093865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c107d70 msgr2=0x7eff6c19e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.364+0000 7eff6b59e640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c107d70 0x7eff6c19e5c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00db80 tx=0x7eff5c00b630 comp rx=0 tx=0).stop 2024-05-09T08:56:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.364+0000 7eff6b59e640 1 -- 172.21.15.98:0/946093865 shutdown_connections 2024-05-09T08:56:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.364+0000 7eff6b59e640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7eff580a2e80 0x7eff580a5340 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.364+0000 7eff6b59e640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff6c10dd00 0x7eff6c1a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.364+0000 7eff6b59e640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff6c1086f0 0x7eff6c19eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.364+0000 7eff6b59e640 1 --2- 172.21.15.98:0/946093865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c107d70 0x7eff6c19e5c0 secure :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00db80 tx=0x7eff5c00b630 comp rx=0 tx=0).stop 2024-05-09T08:56:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.364+0000 7eff6b59e640 1 -- 172.21.15.98:0/946093865 >> 172.21.15.98:0/946093865 conn(0x7eff6c077650 msgr2=0x7eff6c078760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.364+0000 7eff6b59e640 1 -- 172.21.15.98:0/946093865 shutdown_connections 2024-05-09T08:56:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.364+0000 7eff6b59e640 1 -- 172.21.15.98:0/946093865 wait complete. 2024-05-09T08:56:11.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:56:11.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.696+0000 7f4821ea5640 1 -- 172.21.15.98:0/3821110253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f481c105bc0 msgr2=0x7f481c106050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:11.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.696+0000 7f4821ea5640 1 --2- 172.21.15.98:0/3821110253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f481c105bc0 0x7f481c106050 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4810009900 tx=0x7f481002a540 comp rx=0 tx=0).stop 2024-05-09T08:56:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.698+0000 7f4821ea5640 1 -- 172.21.15.98:0/3821110253 shutdown_connections 2024-05-09T08:56:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.698+0000 7f4821ea5640 1 --2- 172.21.15.98:0/3821110253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f481c105bc0 0x7f481c106050 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.698+0000 7f4821ea5640 1 --2- 172.21.15.98:0/3821110253 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f481c1011c0 0x7f481c105680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.698+0000 7f4821ea5640 1 --2- 172.21.15.98:0/3821110253 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f481c1008a0 0x7f481c100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.698+0000 7f4821ea5640 1 -- 172.21.15.98:0/3821110253 >> 172.21.15.98:0/3821110253 conn(0x7f481c077830 msgr2=0x7f481c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.701+0000 7f4821ea5640 1 -- 172.21.15.98:0/3821110253 shutdown_connections 2024-05-09T08:56:11.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.703+0000 7f4821ea5640 1 -- 172.21.15.98:0/3821110253 wait complete. 2024-05-09T08:56:11.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.704+0000 7f4821ea5640 1 Processor -- start 2024-05-09T08:56:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.705+0000 7f4821ea5640 1 -- start start 2024-05-09T08:56:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.705+0000 7f4821ea5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f481c1008a0 0x7f481c073260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.705+0000 7f4821ea5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f481c1011c0 0x7f481c06d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.706+0000 7f4821ea5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f481c105bc0 0x7f481c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.706+0000 7f4821ea5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f481c06bca0 con 0x7f481c1011c0 2024-05-09T08:56:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.706+0000 7f4821ea5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f481c06bb20 con 0x7f481c105bc0 2024-05-09T08:56:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.706+0000 7f4821ea5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f481c06be20 con 0x7f481c1008a0 2024-05-09T08:56:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.706+0000 7f481b7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f481c1008a0 0x7f481c073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.706+0000 7f481b7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f481c1008a0 0x7f481c073260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39382/0 (socket says 172.21.15.98:39382) 2024-05-09T08:56:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.706+0000 7f481b7fe640 1 -- 172.21.15.98:0/2850093048 learned_addr learned my addr 172.21.15.98:0/2850093048 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.707+0000 7f481bfff640 1 --2- 172.21.15.98:0/2850093048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f481c105bc0 0x7f481c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.708+0000 7f481b7fe640 1 -- 172.21.15.98:0/2850093048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f481c105bc0 msgr2=0x7f481c06d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.708+0000 7f481b7fe640 1 --2- 172.21.15.98:0/2850093048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f481c105bc0 0x7f481c06d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.708+0000 7f481b7fe640 1 -- 172.21.15.98:0/2850093048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f481c1011c0 msgr2=0x7f481c06d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.708+0000 7f481b7fe640 1 --2- 172.21.15.98:0/2850093048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f481c1011c0 0x7f481c06d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.708+0000 7f481b7fe640 1 -- 172.21.15.98:0/2850093048 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f481c06e040 con 0x7f481c1008a0 2024-05-09T08:56:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.709+0000 7f481b7fe640 1 --2- 172.21.15.98:0/2850093048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f481c1008a0 0x7f481c073260 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f481400cce0 tx=0x7f481400a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.710+0000 7f4818ff9640 1 -- 172.21.15.98:0/2850093048 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4814004b20 con 0x7f481c1008a0 2024-05-09T08:56:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.710+0000 7f4821ea5640 1 -- 172.21.15.98:0/2850093048 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f481c1b34c0 con 0x7f481c1008a0 2024-05-09T08:56:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.710+0000 7f4821ea5640 1 -- 172.21.15.98:0/2850093048 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f481c1b3a60 con 0x7f481c1008a0 2024-05-09T08:56:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.711+0000 7f4818ff9640 1 -- 172.21.15.98:0/2850093048 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4814004cc0 con 0x7f481c1008a0 2024-05-09T08:56:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.711+0000 7f4818ff9640 1 -- 172.21.15.98:0/2850093048 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481400f7d0 con 0x7f481c1008a0 2024-05-09T08:56:11.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.714+0000 7f4818ff9640 1 -- 172.21.15.98:0/2850093048 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f481400f970 con 0x7f481c1008a0 2024-05-09T08:56:11.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.716+0000 7f4824bb9740 1 Processor -- start 2024-05-09T08:56:11.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.717+0000 7f4818ff9640 1 --2- 172.21.15.98:0/2850093048 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f48040a2e40 0x7f48040a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.718+0000 7f4824bb9740 1 -- start start 2024-05-09T08:56:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.718+0000 7f4818ff9640 1 -- 172.21.15.98:0/2850093048 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f48140cfc60 con 0x7f481c1008a0 2024-05-09T08:56:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.718+0000 7f4824bb9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4b330 0x55e078b4b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.718+0000 7f4824bb9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4bc50 0x55e078b55de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.718+0000 7f4824bb9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b56320 0x55e078b58710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.718+0000 7f4824bb9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e078ab0c40 con 0x55e078b56320 2024-05-09T08:56:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.718+0000 7f4824bb9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e078a58580 con 0x55e078b4b330 2024-05-09T08:56:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.718+0000 7f4824bb9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e078aacc00 con 0x55e078b4bc50 2024-05-09T08:56:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.719+0000 7f481affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4bc50 0x55e078b55de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.719+0000 7f481affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4bc50 0x55e078b55de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39384/0 (socket says 172.21.15.98:39384) 2024-05-09T08:56:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.719+0000 7f481affd640 1 -- 172.21.15.98:0/3099857869 learned_addr learned my addr 172.21.15.98:0/3099857869 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.719+0000 7f481b7fe640 1 --2- 172.21.15.98:0/3099857869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4b330 0x55e078b4b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.719+0000 7f481bfff640 1 --2- 172.21.15.98:0/2850093048 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f48040a2e40 0x7f48040a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.719+0000 7f481bfff640 1 --2- 172.21.15.98:0/3099857869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b56320 0x55e078b58710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.720+0000 7f481b7fe640 1 -- 172.21.15.98:0/3099857869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4bc50 msgr2=0x55e078b55de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.720+0000 7f481b7fe640 1 --2- 172.21.15.98:0/3099857869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4bc50 0x55e078b55de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.720+0000 7f481b7fe640 1 -- 172.21.15.98:0/3099857869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b56320 msgr2=0x55e078b58710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.720+0000 7f481b7fe640 1 --2- 172.21.15.98:0/3099857869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b56320 0x55e078b58710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.720+0000 7f481b7fe640 1 -- 172.21.15.98:0/3099857869 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0787c43f0 con 0x55e078b4b330 2024-05-09T08:56:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.721+0000 7f481b7fe640 1 --2- 172.21.15.98:0/3099857869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4b330 0x55e078b4b710 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f48140ccf90 tx=0x7f481409c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53bfdcf363891db server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:11.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.724+0000 7f481bfff640 1 --2- 172.21.15.98:0/2850093048 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f48040a2e40 0x7f48040a5300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4810005dc0 tx=0x7f4810005d50 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:11.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.725+0000 7f4801ffb640 1 -- 172.21.15.98:0/3099857869 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481401d070 con 0x55e078b4b330 2024-05-09T08:56:11.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.725+0000 7f4801ffb640 1 -- 172.21.15.98:0/3099857869 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48140a49d0 con 0x55e078b4b330 2024-05-09T08:56:11.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.725+0000 7f4801ffb640 1 -- 172.21.15.98:0/3099857869 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48140a4540 con 0x55e078b4b330 2024-05-09T08:56:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.731+0000 7f4824bb9740 1 -- 172.21.15.98:0/3099857869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4b330 msgr2=0x55e078b4b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.731+0000 7f4824bb9740 1 --2- 172.21.15.98:0/3099857869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4b330 0x55e078b4b710 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f48140ccf90 tx=0x7f481409c520 comp rx=0 tx=0).stop 2024-05-09T08:56:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.733+0000 7f4824bb9740 1 -- 172.21.15.98:0/3099857869 shutdown_connections 2024-05-09T08:56:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.733+0000 7f4824bb9740 1 --2- 172.21.15.98:0/3099857869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b56320 0x55e078b58710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.733+0000 7f4824bb9740 1 --2- 172.21.15.98:0/3099857869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4bc50 0x55e078b55de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.733+0000 7f4824bb9740 1 --2- 172.21.15.98:0/3099857869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4b330 0x55e078b4b710 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.733+0000 7f4824bb9740 1 -- 172.21.15.98:0/3099857869 >> 172.21.15.98:0/3099857869 conn(0x55e078b47650 msgr2=0x55e078b47a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.734+0000 7f4824bb9740 1 -- 172.21.15.98:0/3099857869 shutdown_connections 2024-05-09T08:56:11.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.735+0000 7f4824bb9740 1 -- 172.21.15.98:0/3099857869 wait complete. 2024-05-09T08:56:11.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.735+0000 7f4824bb9740 1 -- start start 2024-05-09T08:56:11.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.736+0000 7f4824bb9740 1 Processor -- start 2024-05-09T08:56:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.736+0000 7f4824bb9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4b330 0x55e078b532a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.736+0000 7f4824bb9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4bc50 0x55e078b537e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.736+0000 7f4824bb9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b56320 0x55e078b53d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.737+0000 7f481b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4b330 0x55e078b532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.737+0000 7f481b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4b330 0x55e078b532a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52298/0 (socket says 172.21.15.98:52298) 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.737+0000 7f481b7fe640 1 -- 172.21.15.98:0/3811502046 learned_addr learned my addr 172.21.15.98:0/3811502046 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.737+0000 7f4824bb9740 1 -- 172.21.15.98:0/3811502046 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e078b59110 con 0x55e078b56320 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.737+0000 7f4824bb9740 1 -- 172.21.15.98:0/3811502046 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e078a6bc40 con 0x55e078b4b330 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.737+0000 7f4824bb9740 1 -- 172.21.15.98:0/3811502046 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e078b59290 con 0x55e078b4bc50 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.737+0000 7f481affd640 1 --2- 172.21.15.98:0/3811502046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4bc50 0x55e078b537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.737+0000 7f4824bb9740 10 client.0 _ll_register_callbacks cb 0x55e0787c3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.737+0000 7f481b7fe640 1 -- 172.21.15.98:0/3811502046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4bc50 msgr2=0x55e078b537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.737+0000 7f481b7fe640 1 --2- 172.21.15.98:0/3811502046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4bc50 0x55e078b537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.737+0000 7f481b7fe640 1 -- 172.21.15.98:0/3811502046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b56320 msgr2=0x55e078b53d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.737+0000 7f481b7fe640 1 --2- 172.21.15.98:0/3811502046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b56320 0x55e078b53d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.737+0000 7f481b7fe640 1 -- 172.21.15.98:0/3811502046 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e07888e7d0 con 0x55e078b4b330 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.737+0000 7f481affd640 1 --2- 172.21.15.98:0/3811502046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4bc50 0x55e078b537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:56:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.738+0000 7f481b7fe640 1 --2- 172.21.15.98:0/3811502046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4b330 0x55e078b532a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4814098730 tx=0x7f481400fe90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13036462bb5a02c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.738+0000 7f47faffd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:56:11.739 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:11 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/946093865' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:56:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.739+0000 7f47faffd640 1 -- 172.21.15.98:0/3811502046 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48140b8030 con 0x55e078b4b330 2024-05-09T08:56:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.745+0000 7f4824bb9740 1 -- 172.21.15.98:0/3811502046 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48140a49d0 con 0x55e078b4b330 2024-05-09T08:56:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.745+0000 7f47faffd640 1 -- 172.21.15.98:0/3811502046 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f481409a060 con 0x55e078b4b330 2024-05-09T08:56:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.746+0000 7f47faffd640 1 -- 172.21.15.98:0/3811502046 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f481401d040 con 0x55e078b4b330 2024-05-09T08:56:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.746+0000 7f4824bb9740 10 client.26396 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.746+0000 7f4824bb9740 1 -- 172.21.15.98:0/3811502046 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e078b650a0 con 0x55e078b4b330 2024-05-09T08:56:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.747+0000 7f47faffd640 1 -- 172.21.15.98:0/3811502046 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48140a4540 con 0x55e078b4b330 2024-05-09T08:56:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.747+0000 7f47faffd640 1 -- 172.21.15.98:0/3811502046 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f481409ac20 con 0x55e078b4b330 2024-05-09T08:56:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.747+0000 7f4824bb9740 10 client.26396 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.747+0000 7f4824bb9740 10 client.26396 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:56:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.747+0000 7f4824bb9740 20 client.26396 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.747+0000 7f4824bb9740 1 --2- 172.21.15.98:0/3811502046 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55e078b665f0 0x55e078b68a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.747+0000 7f4824bb9740 4 client.26396 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.747+0000 7f4824bb9740 1 -- 172.21.15.98:0/3811502046 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f48140a49d0 con 0x55e078b665f0 2024-05-09T08:56:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.748+0000 7f481affd640 1 --2- 172.21.15.98:0/3811502046 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55e078b665f0 0x55e078b68a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.748+0000 7f47faffd640 10 client.26396 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:56:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.748+0000 7f481affd640 1 --2- 172.21.15.98:0/3811502046 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55e078b665f0 0x55e078b68a70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dfe4680b9947ad9e server_cookie=f21a96ab96cd1a8c in_seq=0 out_seq=0 2024-05-09T08:56:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.749+0000 7f47faffd640 1 -- 172.21.15.98:0/3811502046 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f48140a49d0 con 0x55e078b665f0 2024-05-09T08:56:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.749+0000 7f47faffd640 10 client.26396 handle_command_reply: tid=0 2024-05-09T08:56:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.750+0000 7f4824bb9740 1 client.26396 shutdown 2024-05-09T08:56:11.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.753+0000 7f4824bb9740 1 -- 172.21.15.98:0/3811502046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4b330 msgr2=0x55e078b532a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:11.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.753+0000 7f4824bb9740 1 --2- 172.21.15.98:0/3811502046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4b330 0x55e078b532a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4814098730 tx=0x7f481400fe90 comp rx=0 tx=0).stop 2024-05-09T08:56:11.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.755+0000 7f4824bb9740 1 -- 172.21.15.98:0/3811502046 shutdown_connections 2024-05-09T08:56:11.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.755+0000 7f4824bb9740 1 --2- 172.21.15.98:0/3811502046 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55e078b665f0 0x55e078b68a70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.755+0000 7f4824bb9740 1 --2- 172.21.15.98:0/3811502046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b56320 0x55e078b53d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.755+0000 7f4824bb9740 1 --2- 172.21.15.98:0/3811502046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4bc50 0x55e078b537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.755+0000 7f4824bb9740 1 --2- 172.21.15.98:0/3811502046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4b330 0x55e078b532a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.755+0000 7f47faffd640 0 client.26396 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:56:11.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.755+0000 7f4824bb9740 1 -- 172.21.15.98:0/3811502046 >> 172.21.15.98:0/3811502046 conn(0x55e078b47650 msgr2=0x55e078b4d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:11.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.759+0000 7f4824bb9740 1 -- 172.21.15.98:0/3811502046 shutdown_connections 2024-05-09T08:56:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.763+0000 7f4824bb9740 1 -- 172.21.15.98:0/3811502046 wait complete. 2024-05-09T08:56:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.765+0000 7f4824bb9740 20 client.26396 trim_cache size 0 max 16384 2024-05-09T08:56:11.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.792+0000 7f4824bb9740 1 Processor -- start 2024-05-09T08:56:11.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.793+0000 7f4824bb9740 1 -- start start 2024-05-09T08:56:11.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.793+0000 7f4824bb9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4b330 0x55e078b61fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.793+0000 7f4824bb9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4bc50 0x55e078b550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.793+0000 7f4824bb9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b56320 0x55e078b6e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.793+0000 7f4824bb9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4808010020 con 0x55e078b56320 2024-05-09T08:56:11.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.793+0000 7f4824bb9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4808011b60 con 0x55e078b4bc50 2024-05-09T08:56:11.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.793+0000 7f4824bb9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f48080106f0 con 0x55e078b4b330 2024-05-09T08:56:11.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.796+0000 7f481b7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4b330 0x55e078b61fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.796+0000 7f481b7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4b330 0x55e078b61fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39390/0 (socket says 172.21.15.98:39390) 2024-05-09T08:56:11.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.796+0000 7f481b7fe640 1 -- 172.21.15.98:0/1522308354 learned_addr learned my addr 172.21.15.98:0/1522308354 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:11.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.796+0000 7f481bfff640 1 --2- 172.21.15.98:0/1522308354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b56320 0x55e078b6e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.800+0000 7f481affd640 1 --2- 172.21.15.98:0/1522308354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4bc50 0x55e078b550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.800+0000 7f481b7fe640 1 -- 172.21.15.98:0/1522308354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4bc50 msgr2=0x55e078b550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:11.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.800+0000 7f481b7fe640 1 --2- 172.21.15.98:0/1522308354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4bc50 0x55e078b550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.800+0000 7f481b7fe640 1 -- 172.21.15.98:0/1522308354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b56320 msgr2=0x55e078b6e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:11.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.800+0000 7f481b7fe640 1 --2- 172.21.15.98:0/1522308354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b56320 0x55e078b6e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.800+0000 7f481b7fe640 1 -- 172.21.15.98:0/1522308354 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e078b690f0 con 0x55e078b4b330 2024-05-09T08:56:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.803+0000 7f481affd640 1 --2- 172.21.15.98:0/1522308354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4bc50 0x55e078b550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:56:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.804+0000 7f481b7fe640 1 --2- 172.21.15.98:0/1522308354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4b330 0x55e078b61fd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f481400ab20 tx=0x7f48140a12a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ea1eb58763d14a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.806+0000 7f48017fa640 1 -- 172.21.15.98:0/1522308354 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48140b8020 con 0x55e078b4b330 2024-05-09T08:56:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.806+0000 7f48017fa640 1 -- 172.21.15.98:0/1522308354 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48140851d0 con 0x55e078b4b330 2024-05-09T08:56:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.806+0000 7f48017fa640 1 -- 172.21.15.98:0/1522308354 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48140a47a0 con 0x55e078b4b330 2024-05-09T08:56:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.809+0000 7f4824bb9740 1 -- 172.21.15.98:0/1522308354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4b330 msgr2=0x55e078b61fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.810+0000 7f4824bb9740 1 --2- 172.21.15.98:0/1522308354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4b330 0x55e078b61fd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f481400ab20 tx=0x7f48140a12a0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.812+0000 7f4824bb9740 1 -- 172.21.15.98:0/1522308354 shutdown_connections 2024-05-09T08:56:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.812+0000 7f4824bb9740 1 --2- 172.21.15.98:0/1522308354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b56320 0x55e078b6e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.812+0000 7f4824bb9740 1 --2- 172.21.15.98:0/1522308354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4bc50 0x55e078b550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.812+0000 7f4824bb9740 1 --2- 172.21.15.98:0/1522308354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b4b330 0x55e078b61fd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.812+0000 7f4824bb9740 1 -- 172.21.15.98:0/1522308354 >> 172.21.15.98:0/1522308354 conn(0x55e078b47650 msgr2=0x55e078b549c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.815+0000 7f4824bb9740 1 -- 172.21.15.98:0/1522308354 shutdown_connections 2024-05-09T08:56:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.817+0000 7f4824bb9740 1 -- 172.21.15.98:0/1522308354 wait complete. 2024-05-09T08:56:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.817+0000 7f4824bb9740 1 -- start start 2024-05-09T08:56:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.818+0000 7f4824bb9740 1 Processor -- start 2024-05-09T08:56:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.818+0000 7f4824bb9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b4b330 0x55e078b59f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.818+0000 7f4824bb9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4bc50 0x55e078b5a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.818+0000 7f4824bb9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b56320 0x55e078b5a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.818+0000 7f4824bb9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4808011490 con 0x55e078b4b330 2024-05-09T08:56:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.818+0000 7f4824bb9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4808010dc0 con 0x55e078b4bc50 2024-05-09T08:56:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.818+0000 7f4824bb9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f480800e000 con 0x55e078b56320 2024-05-09T08:56:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.818+0000 7f4824bb9740 10 client.0 _ll_register_callbacks cb 0x7f481409b6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.819+0000 7f481bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b56320 0x55e078b5a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.819+0000 7f481bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b56320 0x55e078b5a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39404/0 (socket says 172.21.15.98:39404) 2024-05-09T08:56:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.819+0000 7f481bfff640 1 -- 172.21.15.98:0/1178443200 learned_addr learned my addr 172.21.15.98:0/1178443200 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.822+0000 7f481affd640 1 --2- 172.21.15.98:0/1178443200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4bc50 0x55e078b5a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.822+0000 7f481bfff640 1 -- 172.21.15.98:0/1178443200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4bc50 msgr2=0x55e078b5a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.822+0000 7f481bfff640 1 --2- 172.21.15.98:0/1178443200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4bc50 0x55e078b5a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.822+0000 7f481bfff640 1 -- 172.21.15.98:0/1178443200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b4b330 msgr2=0x55e078b59f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:11.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.822+0000 7f481bfff640 1 --2- 172.21.15.98:0/1178443200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b4b330 0x55e078b59f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.822+0000 7f481bfff640 1 -- 172.21.15.98:0/1178443200 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e078bdcc30 con 0x55e078b56320 2024-05-09T08:56:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.825+0000 7f481bfff640 1 --2- 172.21.15.98:0/1178443200 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b56320 0x55e078b5a9b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4810009230 tx=0x7f4810004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=107daecd14c7c0d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.825+0000 7f47faffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:56:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.825+0000 7f47faffd640 1 -- 172.21.15.98:0/1178443200 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4810035040 con 0x55e078b56320 2024-05-09T08:56:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.825+0000 7f4824bb9740 1 -- 172.21.15.98:0/1178443200 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48140851d0 con 0x55e078b56320 2024-05-09T08:56:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.826+0000 7f47faffd640 1 -- 172.21.15.98:0/1178443200 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f4810031950 con 0x55e078b56320 2024-05-09T08:56:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.826+0000 7f47faffd640 1 -- 172.21.15.98:0/1178443200 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4810031b50 con 0x55e078b56320 2024-05-09T08:56:11.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.828+0000 7f4824bb9740 10 client.26377 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:11.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.828+0000 7f4824bb9740 1 -- 172.21.15.98:0/1178443200 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e078bdde80 con 0x55e078b56320 2024-05-09T08:56:11.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.830+0000 7f47faffd640 1 -- 172.21.15.98:0/1178443200 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48140851d0 con 0x55e078b56320 2024-05-09T08:56:11.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.830+0000 7f47faffd640 1 -- 172.21.15.98:0/1178443200 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f4810051070 con 0x55e078b56320 2024-05-09T08:56:11.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.830+0000 7f4824bb9740 10 client.26377 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:11.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.830+0000 7f4824bb9740 10 client.26377 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:56:11.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.830+0000 7f4824bb9740 20 client.26377 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:11.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.830+0000 7f4824bb9740 1 --2- 172.21.15.98:0/1178443200 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55e078bde630 0x55e078be0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:11.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.830+0000 7f4824bb9740 4 client.26377 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:56:11.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.830+0000 7f4824bb9740 1 -- 172.21.15.98:0/1178443200 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e078abe370 con 0x55e078bde630 2024-05-09T08:56:11.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.834+0000 7f481b7fe640 1 --2- 172.21.15.98:0/1178443200 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55e078bde630 0x55e078be0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:11.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:11 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/946093865' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:56:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.840+0000 7f481b7fe640 1 --2- 172.21.15.98:0/1178443200 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55e078bde630 0x55e078be0a70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1dae7172587ac2b2 server_cookie=653f01fd823cfbe0 in_seq=0 out_seq=0 2024-05-09T08:56:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.841+0000 7f47faffd640 10 client.26377 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:56:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.842+0000 7f47faffd640 1 -- 172.21.15.98:0/1178443200 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e078abe370 con 0x55e078bde630 2024-05-09T08:56:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.842+0000 7f47faffd640 10 client.26377 handle_command_reply: tid=0 2024-05-09T08:56:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.843+0000 7f4824bb9740 1 client.26377 shutdown 2024-05-09T08:56:11.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.845+0000 7f4824bb9740 1 -- 172.21.15.98:0/1178443200 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b56320 msgr2=0x55e078b5a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:11.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.845+0000 7f4824bb9740 1 --2- 172.21.15.98:0/1178443200 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b56320 0x55e078b5a9b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4810009230 tx=0x7f4810004040 comp rx=0 tx=0).stop 2024-05-09T08:56:11.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.845+0000 7f4824bb9740 1 -- 172.21.15.98:0/1178443200 shutdown_connections 2024-05-09T08:56:11.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.845+0000 7f4824bb9740 1 --2- 172.21.15.98:0/1178443200 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55e078bde630 0x55e078be0a70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.845+0000 7f4824bb9740 1 --2- 172.21.15.98:0/1178443200 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e078b56320 0x55e078b5a9b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.845+0000 7f4824bb9740 1 --2- 172.21.15.98:0/1178443200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e078b4bc50 0x55e078b5a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.845+0000 7f4824bb9740 1 --2- 172.21.15.98:0/1178443200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e078b4b330 0x55e078b59f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.845+0000 7f4824bb9740 1 -- 172.21.15.98:0/1178443200 >> 172.21.15.98:0/1178443200 conn(0x55e078b47650 msgr2=0x55e078b467b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.845+0000 7f47faffd640 0 client.26377 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:56:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.845+0000 7f4824bb9740 1 -- 172.21.15.98:0/1178443200 shutdown_connections 2024-05-09T08:56:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.845+0000 7f4824bb9740 1 -- 172.21.15.98:0/1178443200 wait complete. 2024-05-09T08:56:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.846+0000 7f4824bb9740 20 client.26377 trim_cache size 0 max 16384 2024-05-09T08:56:11.846 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:56:11.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.855+0000 7f4821ea5640 1 -- 172.21.15.98:0/2850093048 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f48040a2e40 msgr2=0x7f48040a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:11.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.855+0000 7f4821ea5640 1 --2- 172.21.15.98:0/2850093048 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f48040a2e40 0x7f48040a5300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4810005dc0 tx=0x7f4810005d50 comp rx=0 tx=0).stop 2024-05-09T08:56:11.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.855+0000 7f4821ea5640 1 -- 172.21.15.98:0/2850093048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f481c1008a0 msgr2=0x7f481c073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:11.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.855+0000 7f4821ea5640 1 --2- 172.21.15.98:0/2850093048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f481c1008a0 0x7f481c073260 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f481400cce0 tx=0x7f481400a620 comp rx=0 tx=0).stop 2024-05-09T08:56:11.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.860+0000 7f4821ea5640 1 -- 172.21.15.98:0/2850093048 shutdown_connections 2024-05-09T08:56:11.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.860+0000 7f4821ea5640 1 --2- 172.21.15.98:0/2850093048 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f48040a2e40 0x7f48040a5300 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.860+0000 7f4821ea5640 1 --2- 172.21.15.98:0/2850093048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f481c105bc0 0x7f481c06d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.860+0000 7f4821ea5640 1 --2- 172.21.15.98:0/2850093048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f481c1011c0 0x7f481c06d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.860+0000 7f4821ea5640 1 --2- 172.21.15.98:0/2850093048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f481c1008a0 0x7f481c073260 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:11.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.860+0000 7f4821ea5640 1 -- 172.21.15.98:0/2850093048 >> 172.21.15.98:0/2850093048 conn(0x7f481c077830 msgr2=0x7f481c103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:11.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.862+0000 7f4821ea5640 1 -- 172.21.15.98:0/2850093048 shutdown_connections 2024-05-09T08:56:11.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:11.862+0000 7f4821ea5640 1 -- 172.21.15.98:0/2850093048 wait complete. 2024-05-09T08:56:11.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:56:11.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:11 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/946093865' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:56:12.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.168+0000 7faf5d0d9640 1 -- 172.21.15.98:0/4245236804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58106660 msgr2=0x7faf58106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.168+0000 7faf5d0d9640 1 --2- 172.21.15.98:0/4245236804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58106660 0x7faf58106a40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faf48009900 tx=0x7faf4802a3f0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.169+0000 7faf5d0d9640 1 -- 172.21.15.98:0/4245236804 shutdown_connections 2024-05-09T08:56:12.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.169+0000 7faf5d0d9640 1 --2- 172.21.15.98:0/4245236804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf58100f80 0x7faf5810d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.169+0000 7faf5d0d9640 1 --2- 172.21.15.98:0/4245236804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf581005c0 0x7faf58100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.169+0000 7faf5d0d9640 1 --2- 172.21.15.98:0/4245236804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58106660 0x7faf58106a40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.169+0000 7faf5d0d9640 1 -- 172.21.15.98:0/4245236804 >> 172.21.15.98:0/4245236804 conn(0x7faf580fc0b0 msgr2=0x7faf580fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:12.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.169+0000 7faf5d0d9640 1 -- 172.21.15.98:0/4245236804 shutdown_connections 2024-05-09T08:56:12.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.169+0000 7faf5d0d9640 1 -- 172.21.15.98:0/4245236804 wait complete. 2024-05-09T08:56:12.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.170+0000 7faf5d0d9640 1 Processor -- start 2024-05-09T08:56:12.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.173+0000 7faf5d0d9640 1 -- start start 2024-05-09T08:56:12.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.173+0000 7faf5d0d9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf581005c0 0x7faf58109840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.173+0000 7faf5d0d9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58100f80 0x7faf58109d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.173+0000 7faf5d0d9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf58106660 0x7faf5810a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.173+0000 7faf5d0d9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf5810f940 con 0x7faf581005c0 2024-05-09T08:56:12.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.173+0000 7faf5d0d9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faf5810f7c0 con 0x7faf58100f80 2024-05-09T08:56:12.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.173+0000 7faf5d0d9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7faf5810fac0 con 0x7faf58106660 2024-05-09T08:56:12.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.174+0000 7faf4ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58100f80 0x7faf58109d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.174+0000 7faf4ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58100f80 0x7faf58109d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52344/0 (socket says 172.21.15.98:52344) 2024-05-09T08:56:12.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.174+0000 7faf4ffff640 1 -- 172.21.15.98:0/28710271 learned_addr learned my addr 172.21.15.98:0/28710271 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:12.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.174+0000 7faf57fff640 1 --2- 172.21.15.98:0/28710271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf581005c0 0x7faf58109840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.174+0000 7faf4ffff640 1 -- 172.21.15.98:0/28710271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf58106660 msgr2=0x7faf5810a2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:56:12.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.174+0000 7faf4ffff640 1 --2- 172.21.15.98:0/28710271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf58106660 0x7faf5810a2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.174+0000 7faf4ffff640 1 -- 172.21.15.98:0/28710271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf581005c0 msgr2=0x7faf58109840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.174+0000 7faf4ffff640 1 --2- 172.21.15.98:0/28710271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf581005c0 0x7faf58109840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.174+0000 7faf4ffff640 1 -- 172.21.15.98:0/28710271 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf58073580 con 0x7faf58100f80 2024-05-09T08:56:12.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.174+0000 7faf57fff640 1 --2- 172.21.15.98:0/28710271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf581005c0 0x7faf58109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:56:12.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.174+0000 7faf4ffff640 1 --2- 172.21.15.98:0/28710271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58100f80 0x7faf58109d80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faf4000e9f0 tx=0x7faf4000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:12.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.175+0000 7faf55ffb640 1 -- 172.21.15.98:0/28710271 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4000cd80 con 0x7faf58100f80 2024-05-09T08:56:12.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.175+0000 7faf55ffb640 1 -- 172.21.15.98:0/28710271 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf40005500 con 0x7faf58100f80 2024-05-09T08:56:12.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.175+0000 7faf55ffb640 1 -- 172.21.15.98:0/28710271 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf40010690 con 0x7faf58100f80 2024-05-09T08:56:12.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.176+0000 7faf5d0d9640 1 -- 172.21.15.98:0/28710271 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf58073810 con 0x7faf58100f80 2024-05-09T08:56:12.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.176+0000 7faf5d0d9640 1 -- 172.21.15.98:0/28710271 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf58115370 con 0x7faf58100f80 2024-05-09T08:56:12.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.180+0000 7faf55ffb640 1 -- 172.21.15.98:0/28710271 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7faf400046c0 con 0x7faf58100f80 2024-05-09T08:56:12.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.181+0000 7faf55ffb640 1 --2- 172.21.15.98:0/28710271 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faf440b3f10 0x7faf440b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.182+0000 7faf55ffb640 1 -- 172.21.15.98:0/28710271 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7faf400cfc20 con 0x7faf58100f80 2024-05-09T08:56:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.183+0000 7faf57fff640 1 --2- 172.21.15.98:0/28710271 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faf440b3f10 0x7faf440b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.183+0000 7faf57fff640 1 --2- 172.21.15.98:0/28710271 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faf440b3f10 0x7faf440b63d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faf4802a960 tx=0x7faf4802a8f0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:12.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.194+0000 7faf61047740 1 Processor -- start 2024-05-09T08:56:12.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.196+0000 7faf61047740 1 -- start start 2024-05-09T08:56:12.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.196+0000 7faf61047740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb9751900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.196+0000 7faf61047740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb9751e40 0x55aeb975bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.196+0000 7faf61047740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb975c510 0x55aeb975e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.196+0000 7faf61047740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55aeb96b6c40 con 0x55aeb9751520 2024-05-09T08:56:12.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.196+0000 7faf61047740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55aeb965e580 con 0x55aeb9751e40 2024-05-09T08:56:12.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.196+0000 7faf61047740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55aeb96b2c00 con 0x55aeb975c510 2024-05-09T08:56:12.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.198+0000 7faf57fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb9751900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.198+0000 7faf57fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb9751900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39508/0 (socket says 172.21.15.98:39508) 2024-05-09T08:56:12.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.198+0000 7faf57fff640 1 -- 172.21.15.98:0/2368599163 learned_addr learned my addr 172.21.15.98:0/2368599163 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:12.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.199+0000 7faf4ffff640 1 --2- 172.21.15.98:0/2368599163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb9751e40 0x55aeb975bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.201+0000 7faf57fff640 1 -- 172.21.15.98:0/2368599163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb975c510 msgr2=0x55aeb975e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.201+0000 7faf57fff640 1 --2- 172.21.15.98:0/2368599163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb975c510 0x55aeb975e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.201+0000 7faf57fff640 1 -- 172.21.15.98:0/2368599163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb9751e40 msgr2=0x55aeb975bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.201+0000 7faf57fff640 1 --2- 172.21.15.98:0/2368599163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb9751e40 0x55aeb975bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.201+0000 7faf57fff640 1 -- 172.21.15.98:0/2368599163 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aeb93ca3f0 con 0x55aeb9751520 2024-05-09T08:56:12.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.203+0000 7faf4ffff640 1 --2- 172.21.15.98:0/2368599163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb9751e40 0x55aeb975bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:56:12.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.214+0000 7faf57fff640 1 --2- 172.21.15.98:0/2368599163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb9751900 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7faf58101700 tx=0x7faf4803c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b709cf9fee17ad31 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:12.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.218+0000 7faf4e7fc640 1 -- 172.21.15.98:0/2368599163 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf48035050 con 0x55aeb9751520 2024-05-09T08:56:12.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.219+0000 7faf4e7fc640 1 -- 172.21.15.98:0/2368599163 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf4804d070 con 0x55aeb9751520 2024-05-09T08:56:12.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.219+0000 7faf4e7fc640 1 -- 172.21.15.98:0/2368599163 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf48032550 con 0x55aeb9751520 2024-05-09T08:56:12.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.220+0000 7faf61047740 1 -- 172.21.15.98:0/2368599163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 msgr2=0x55aeb9751900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.220+0000 7faf61047740 1 --2- 172.21.15.98:0/2368599163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb9751900 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7faf58101700 tx=0x7faf4803c000 comp rx=0 tx=0).stop 2024-05-09T08:56:12.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.220+0000 7faf61047740 1 -- 172.21.15.98:0/2368599163 shutdown_connections 2024-05-09T08:56:12.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.220+0000 7faf61047740 1 --2- 172.21.15.98:0/2368599163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb975c510 0x55aeb975e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.220+0000 7faf61047740 1 --2- 172.21.15.98:0/2368599163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb9751e40 0x55aeb975bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.220+0000 7faf61047740 1 --2- 172.21.15.98:0/2368599163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb9751900 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.220+0000 7faf61047740 1 -- 172.21.15.98:0/2368599163 >> 172.21.15.98:0/2368599163 conn(0x55aeb974d840 msgr2=0x55aeb974dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:12.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.220+0000 7faf61047740 1 -- 172.21.15.98:0/2368599163 shutdown_connections 2024-05-09T08:56:12.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.220+0000 7faf61047740 1 -- 172.21.15.98:0/2368599163 wait complete. 2024-05-09T08:56:12.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.221+0000 7faf61047740 1 -- start start 2024-05-09T08:56:12.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.221+0000 7faf61047740 1 Processor -- start 2024-05-09T08:56:12.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.222+0000 7faf61047740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb97594f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.222+0000 7faf61047740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751e40 0x55aeb9759a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.222+0000 7faf61047740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb975c510 0x55aeb9759f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.222+0000 7faf61047740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55aeb975f2a0 con 0x55aeb9751520 2024-05-09T08:56:12.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.222+0000 7faf61047740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55aeb9671c40 con 0x55aeb975c510 2024-05-09T08:56:12.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.222+0000 7faf61047740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55aeb975f420 con 0x55aeb9751e40 2024-05-09T08:56:12.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.222+0000 7faf61047740 10 client.0 _ll_register_callbacks cb 0x55aeb93c9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:12.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.225+0000 7faf5c8d8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb975c510 0x55aeb9759f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.225+0000 7faf4ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751e40 0x55aeb9759a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.225+0000 7faf5c8d8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb975c510 0x55aeb9759f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52364/0 (socket says 172.21.15.98:52364) 2024-05-09T08:56:12.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.225+0000 7faf5c8d8640 1 -- 172.21.15.98:0/281377305 learned_addr learned my addr 172.21.15.98:0/281377305 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:12.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.226+0000 7faf5c8d8640 1 -- 172.21.15.98:0/281377305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751e40 msgr2=0x55aeb9759a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.226+0000 7faf5c8d8640 1 --2- 172.21.15.98:0/281377305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751e40 0x55aeb9759a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.226+0000 7faf5c8d8640 1 -- 172.21.15.98:0/281377305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 msgr2=0x55aeb97594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.226+0000 7faf5c8d8640 1 --2- 172.21.15.98:0/281377305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb97594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.226+0000 7faf5c8d8640 1 -- 172.21.15.98:0/281377305 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aeb94947d0 con 0x55aeb975c510 2024-05-09T08:56:12.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.227+0000 7faf5c8d8640 1 --2- 172.21.15.98:0/281377305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb975c510 0x55aeb9759f70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faf5000b390 tx=0x7faf5000b850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e57b604c780e5e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:12.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.229+0000 7faf377fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:56:12.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.229+0000 7faf377fe640 1 -- 172.21.15.98:0/281377305 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf500115a0 con 0x55aeb975c510 2024-05-09T08:56:12.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.229+0000 7faf61047740 1 -- 172.21.15.98:0/281377305 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf4804d070 con 0x55aeb975c510 2024-05-09T08:56:12.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.229+0000 7faf377fe640 1 -- 172.21.15.98:0/281377305 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7faf50011740 con 0x55aeb975c510 2024-05-09T08:56:12.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.231+0000 7faf377fe640 1 -- 172.21.15.98:0/281377305 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7faf50020050 con 0x55aeb975c510 2024-05-09T08:56:12.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.238+0000 7faf61047740 10 client.26405 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:12.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.238+0000 7faf61047740 1 -- 172.21.15.98:0/281377305 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55aeb976b510 con 0x55aeb975c510 2024-05-09T08:56:12.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.240+0000 7faf377fe640 1 -- 172.21.15.98:0/281377305 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf4804d070 con 0x55aeb975c510 2024-05-09T08:56:12.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.240+0000 7faf377fe640 1 -- 172.21.15.98:0/281377305 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7faf50011070 con 0x55aeb975c510 2024-05-09T08:56:12.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.240+0000 7faf61047740 10 client.26405 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:12.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.240+0000 7faf61047740 10 client.26405 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:56:12.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.240+0000 7faf61047740 20 client.26405 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:12.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.240+0000 7faf61047740 1 --2- 172.21.15.98:0/281377305 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55aeb976cdf0 0x55aeb976f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.240+0000 7faf61047740 4 client.26405 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:12.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.240+0000 7faf61047740 1 -- 172.21.15.98:0/281377305 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aeb96c4370 con 0x55aeb976cdf0 2024-05-09T08:56:12.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.241+0000 7faf57fff640 1 --2- 172.21.15.98:0/281377305 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55aeb976cdf0 0x55aeb976f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.242+0000 7faf57fff640 1 --2- 172.21.15.98:0/281377305 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55aeb976cdf0 0x55aeb976f2b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2651fbc20787b6af server_cookie=9a28e7c7fe70c17c in_seq=0 out_seq=0 2024-05-09T08:56:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.243+0000 7faf377fe640 10 client.26405 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:56:12.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.246+0000 7faf377fe640 1 -- 172.21.15.98:0/281377305 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aeb96c4370 con 0x55aeb976cdf0 2024-05-09T08:56:12.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.246+0000 7faf377fe640 10 client.26405 handle_command_reply: tid=0 2024-05-09T08:56:12.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.246+0000 7faf61047740 1 client.26405 shutdown 2024-05-09T08:56:12.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.247+0000 7faf61047740 1 -- 172.21.15.98:0/281377305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb975c510 msgr2=0x55aeb9759f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.247+0000 7faf61047740 1 --2- 172.21.15.98:0/281377305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb975c510 0x55aeb9759f70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faf5000b390 tx=0x7faf5000b850 comp rx=0 tx=0).stop 2024-05-09T08:56:12.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.247+0000 7faf61047740 1 -- 172.21.15.98:0/281377305 shutdown_connections 2024-05-09T08:56:12.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.247+0000 7faf61047740 1 --2- 172.21.15.98:0/281377305 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55aeb976cdf0 0x55aeb976f2b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.247+0000 7faf61047740 1 --2- 172.21.15.98:0/281377305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb975c510 0x55aeb9759f70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.247+0000 7faf61047740 1 --2- 172.21.15.98:0/281377305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751e40 0x55aeb9759a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.247+0000 7faf377fe640 0 client.26405 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:56:12.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.247+0000 7faf61047740 1 --2- 172.21.15.98:0/281377305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb97594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.247+0000 7faf61047740 1 -- 172.21.15.98:0/281377305 >> 172.21.15.98:0/281377305 conn(0x55aeb974d840 msgr2=0x55aeb9753540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:12.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.247+0000 7faf61047740 1 -- 172.21.15.98:0/281377305 shutdown_connections 2024-05-09T08:56:12.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.247+0000 7faf61047740 1 -- 172.21.15.98:0/281377305 wait complete. 2024-05-09T08:56:12.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.247+0000 7faf61047740 20 client.26405 trim_cache size 0 max 16384 2024-05-09T08:56:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.277+0000 7faf61047740 1 Processor -- start 2024-05-09T08:56:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.278+0000 7faf61047740 1 -- start start 2024-05-09T08:56:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.278+0000 7faf61047740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751520 0x55aeb9768160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.278+0000 7faf61047740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb9751e40 0x55aeb974d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.279+0000 7faf57fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751520 0x55aeb9768160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.279+0000 7faf57fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751520 0x55aeb9768160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39444/0 (socket says 172.21.15.98:39444) 2024-05-09T08:56:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.279+0000 7faf57fff640 1 -- 172.21.15.98:0/1418254505 learned_addr learned my addr 172.21.15.98:0/1418254505 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.279+0000 7faf61047740 1 --2- 172.21.15.98:0/1418254505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb975c510 0x55aeb9759bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.279+0000 7faf61047740 1 -- 172.21.15.98:0/1418254505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf581b5230 con 0x55aeb975c510 2024-05-09T08:56:12.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.279+0000 7faf5c8d8640 1 --2- 172.21.15.98:0/1418254505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb975c510 0x55aeb9759bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.279+0000 7faf61047740 1 -- 172.21.15.98:0/1418254505 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faf581b6d70 con 0x55aeb9751e40 2024-05-09T08:56:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.279+0000 7faf4ffff640 1 --2- 172.21.15.98:0/1418254505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb9751e40 0x55aeb974d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.280+0000 7faf61047740 1 -- 172.21.15.98:0/1418254505 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7faf581b5900 con 0x55aeb9751520 2024-05-09T08:56:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.280+0000 7faf5c8d8640 1 -- 172.21.15.98:0/1418254505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751520 msgr2=0x55aeb9768160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.280+0000 7faf5c8d8640 1 --2- 172.21.15.98:0/1418254505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751520 0x55aeb9768160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.280+0000 7faf5c8d8640 1 -- 172.21.15.98:0/1418254505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb9751e40 msgr2=0x55aeb974d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.280+0000 7faf5c8d8640 1 --2- 172.21.15.98:0/1418254505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb9751e40 0x55aeb974d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.280+0000 7faf5c8d8640 1 -- 172.21.15.98:0/1418254505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aeb976f930 con 0x55aeb975c510 2024-05-09T08:56:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.280+0000 7faf57fff640 1 --2- 172.21.15.98:0/1418254505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751520 0x55aeb9768160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:56:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.280+0000 7faf4ffff640 1 --2- 172.21.15.98:0/1418254505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb9751e40 0x55aeb974d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:56:12.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.284+0000 7faf5c8d8640 1 --2- 172.21.15.98:0/1418254505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb975c510 0x55aeb9759bc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7faf50002230 tx=0x7faf5001b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13deb1633325874c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:12.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.288+0000 7faf4dffb640 1 -- 172.21.15.98:0/1418254505 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5001b7a0 con 0x55aeb975c510 2024-05-09T08:56:12.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.288+0000 7faf4dffb640 1 -- 172.21.15.98:0/1418254505 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf50017070 con 0x55aeb975c510 2024-05-09T08:56:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.288+0000 7faf4dffb640 1 -- 172.21.15.98:0/1418254505 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf50012b80 con 0x55aeb975c510 2024-05-09T08:56:12.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.289+0000 7faf61047740 1 -- 172.21.15.98:0/1418254505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb975c510 msgr2=0x55aeb9759bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.289+0000 7faf61047740 1 --2- 172.21.15.98:0/1418254505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb975c510 0x55aeb9759bc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7faf50002230 tx=0x7faf5001b6f0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.296+0000 7faf61047740 1 -- 172.21.15.98:0/1418254505 shutdown_connections 2024-05-09T08:56:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.297+0000 7faf61047740 1 --2- 172.21.15.98:0/1418254505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb975c510 0x55aeb9759bc0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.297+0000 7faf61047740 1 --2- 172.21.15.98:0/1418254505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb9751e40 0x55aeb974d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.297+0000 7faf61047740 1 --2- 172.21.15.98:0/1418254505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751520 0x55aeb9768160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.297+0000 7faf61047740 1 -- 172.21.15.98:0/1418254505 >> 172.21.15.98:0/1418254505 conn(0x55aeb974d840 msgr2=0x55aeb97594b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.298+0000 7faf61047740 1 -- 172.21.15.98:0/1418254505 shutdown_connections 2024-05-09T08:56:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.298+0000 7faf61047740 1 -- 172.21.15.98:0/1418254505 wait complete. 2024-05-09T08:56:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.299+0000 7faf61047740 1 -- start start 2024-05-09T08:56:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.299+0000 7faf61047740 1 Processor -- start 2024-05-09T08:56:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.299+0000 7faf61047740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb97649f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.299+0000 7faf61047740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751e40 0x55aeb9755e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.299+0000 7faf61047740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb975c510 0x55aeb97563a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.299+0000 7faf61047740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf581b66a0 con 0x55aeb9751520 2024-05-09T08:56:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.299+0000 7faf61047740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faf581b5fd0 con 0x55aeb975c510 2024-05-09T08:56:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.299+0000 7faf61047740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7faf58068570 con 0x55aeb9751e40 2024-05-09T08:56:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.299+0000 7faf61047740 10 client.0 _ll_register_callbacks cb 0x7faf500094d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.300+0000 7faf57fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb97649f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.300+0000 7faf57fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb97649f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39540/0 (socket says 172.21.15.98:39540) 2024-05-09T08:56:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.300+0000 7faf57fff640 1 -- 172.21.15.98:0/820796556 learned_addr learned my addr 172.21.15.98:0/820796556 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.301+0000 7faf4ffff640 1 --2- 172.21.15.98:0/820796556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751e40 0x55aeb9755e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.301+0000 7faf5c8d8640 1 --2- 172.21.15.98:0/820796556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb975c510 0x55aeb97563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.301+0000 7faf57fff640 1 -- 172.21.15.98:0/820796556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751e40 msgr2=0x55aeb9755e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.301+0000 7faf57fff640 1 --2- 172.21.15.98:0/820796556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751e40 0x55aeb9755e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.302+0000 7faf57fff640 1 -- 172.21.15.98:0/820796556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb975c510 msgr2=0x55aeb97563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.302+0000 7faf57fff640 1 --2- 172.21.15.98:0/820796556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb975c510 0x55aeb97563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.302+0000 7faf57fff640 1 -- 172.21.15.98:0/820796556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aeb9756ba0 con 0x55aeb9751520 2024-05-09T08:56:12.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.302+0000 7faf57fff640 1 --2- 172.21.15.98:0/820796556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb97649f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7faf48005c40 tx=0x7faf480039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29d69e9aeb5de66 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.314+0000 7faf377fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:56:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.314+0000 7faf377fe640 1 -- 172.21.15.98:0/820796556 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf48035040 con 0x55aeb9751520 2024-05-09T08:56:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.314+0000 7faf377fe640 1 -- 172.21.15.98:0/820796556 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7faf480035f0 con 0x55aeb9751520 2024-05-09T08:56:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.314+0000 7faf61047740 1 -- 172.21.15.98:0/820796556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf50017070 con 0x55aeb9751520 2024-05-09T08:56:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.317+0000 7faf377fe640 1 -- 172.21.15.98:0/820796556 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7faf480037f0 con 0x55aeb9751520 2024-05-09T08:56:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.317+0000 7faf61047740 10 client.17133 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.317+0000 7faf61047740 1 -- 172.21.15.98:0/820796556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55aeb97d3460 con 0x55aeb9751520 2024-05-09T08:56:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.321+0000 7faf377fe640 1 -- 172.21.15.98:0/820796556 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf50017070 con 0x55aeb9751520 2024-05-09T08:56:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.323+0000 7faf377fe640 1 -- 172.21.15.98:0/820796556 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7faf48004c00 con 0x55aeb9751520 2024-05-09T08:56:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.323+0000 7faf61047740 10 client.17133 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.323+0000 7faf61047740 10 client.17133 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:56:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.323+0000 7faf61047740 20 client.17133 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.323+0000 7faf61047740 1 --2- 172.21.15.98:0/820796556 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55aeb97d3c30 0x55aeb97d6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.323+0000 7faf61047740 4 client.17133 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:56:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.323+0000 7faf61047740 1 -- 172.21.15.98:0/820796556 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aeb96be6c0 con 0x55aeb97d3c30 2024-05-09T08:56:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.325+0000 7faf4ffff640 1 --2- 172.21.15.98:0/820796556 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55aeb97d3c30 0x55aeb97d6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.329+0000 7faf4ffff640 1 --2- 172.21.15.98:0/820796556 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55aeb97d3c30 0x55aeb97d6050 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e757f433e166b0d2 server_cookie=565091312d97abf2 in_seq=0 out_seq=0 2024-05-09T08:56:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.329+0000 7faf377fe640 10 client.17133 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:56:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.329+0000 7faf377fe640 1 -- 172.21.15.98:0/820796556 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aeb96be6c0 con 0x55aeb97d3c30 2024-05-09T08:56:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.329+0000 7faf377fe640 10 client.17133 handle_command_reply: tid=0 2024-05-09T08:56:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.329+0000 7faf61047740 1 client.17133 shutdown 2024-05-09T08:56:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.330+0000 7faf61047740 1 -- 172.21.15.98:0/820796556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 msgr2=0x55aeb97649f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.330+0000 7faf61047740 1 --2- 172.21.15.98:0/820796556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb97649f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7faf48005c40 tx=0x7faf480039c0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.330+0000 7faf61047740 1 -- 172.21.15.98:0/820796556 shutdown_connections 2024-05-09T08:56:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.330+0000 7faf61047740 1 --2- 172.21.15.98:0/820796556 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55aeb97d3c30 0x55aeb97d6050 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.330+0000 7faf61047740 1 --2- 172.21.15.98:0/820796556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aeb975c510 0x55aeb97563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.330+0000 7faf61047740 1 --2- 172.21.15.98:0/820796556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55aeb9751e40 0x55aeb9755e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.330+0000 7faf61047740 1 --2- 172.21.15.98:0/820796556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55aeb9751520 0x55aeb97649f0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.330+0000 7faf377fe640 0 client.17133 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:56:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.330+0000 7faf61047740 1 -- 172.21.15.98:0/820796556 >> 172.21.15.98:0/820796556 conn(0x55aeb974d840 msgr2=0x55aeb9753500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.335+0000 7faf61047740 1 -- 172.21.15.98:0/820796556 shutdown_connections 2024-05-09T08:56:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.335+0000 7faf61047740 1 -- 172.21.15.98:0/820796556 wait complete. 2024-05-09T08:56:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.338+0000 7faf61047740 20 client.17133 trim_cache size 0 max 16384 2024-05-09T08:56:12.338 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:56:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.342+0000 7faf5d0d9640 1 -- 172.21.15.98:0/28710271 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faf440b3f10 msgr2=0x7faf440b63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.342+0000 7faf5d0d9640 1 --2- 172.21.15.98:0/28710271 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faf440b3f10 0x7faf440b63d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faf4802a960 tx=0x7faf4802a8f0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.342+0000 7faf5d0d9640 1 -- 172.21.15.98:0/28710271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58100f80 msgr2=0x7faf58109d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.342+0000 7faf5d0d9640 1 --2- 172.21.15.98:0/28710271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58100f80 0x7faf58109d80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faf4000e9f0 tx=0x7faf4000edf0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.348+0000 7faf5d0d9640 1 -- 172.21.15.98:0/28710271 shutdown_connections 2024-05-09T08:56:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.348+0000 7faf5d0d9640 1 --2- 172.21.15.98:0/28710271 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faf440b3f10 0x7faf440b63d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.348+0000 7faf5d0d9640 1 --2- 172.21.15.98:0/28710271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf58106660 0x7faf5810a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.348+0000 7faf5d0d9640 1 --2- 172.21.15.98:0/28710271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58100f80 0x7faf58109d80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.348+0000 7faf5d0d9640 1 --2- 172.21.15.98:0/28710271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf581005c0 0x7faf58109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.348+0000 7faf5d0d9640 1 -- 172.21.15.98:0/28710271 >> 172.21.15.98:0/28710271 conn(0x7faf580fc0b0 msgr2=0x7faf580fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.349+0000 7faf5d0d9640 1 -- 172.21.15.98:0/28710271 shutdown_connections 2024-05-09T08:56:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.349+0000 7faf5d0d9640 1 -- 172.21.15.98:0/28710271 wait complete. 2024-05-09T08:56:12.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:56:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.679+0000 7f138ed4d640 1 -- 172.21.15.98:0/3383382722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f138810fed0 msgr2=0x7f13881102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.679+0000 7f138ed4d640 1 --2- 172.21.15.98:0/3383382722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f138810fed0 0x7f13881102b0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f137801a910 tx=0x7f137803b4e0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.682+0000 7f138ed4d640 1 -- 172.21.15.98:0/3383382722 shutdown_connections 2024-05-09T08:56:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.682+0000 7f138ed4d640 1 --2- 172.21.15.98:0/3383382722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13881092b0 0x7f138811d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.682+0000 7f138ed4d640 1 --2- 172.21.15.98:0/3383382722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1388108830 0x7f1388108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.682+0000 7f138ed4d640 1 --2- 172.21.15.98:0/3383382722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f138810fed0 0x7f13881102b0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.682+0000 7f138ed4d640 1 -- 172.21.15.98:0/3383382722 >> 172.21.15.98:0/3383382722 conn(0x7f138806c7b0 msgr2=0x7f138806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.684+0000 7f138ed4d640 1 -- 172.21.15.98:0/3383382722 shutdown_connections 2024-05-09T08:56:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.685+0000 7f138ed4d640 1 -- 172.21.15.98:0/3383382722 wait complete. 2024-05-09T08:56:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.686+0000 7f138ed4d640 1 Processor -- start 2024-05-09T08:56:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.687+0000 7f138ed4d640 1 -- start start 2024-05-09T08:56:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.687+0000 7f138ed4d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1388108830 0x7f138811cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.687+0000 7f138ed4d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13881092b0 0x7f1388118e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.687+0000 7f138ed4d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f138810fed0 0x7f13881193d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.687+0000 7f138ed4d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13881131c0 con 0x7f13881092b0 2024-05-09T08:56:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.687+0000 7f138ed4d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1388113040 con 0x7f1388108830 2024-05-09T08:56:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.687+0000 7f138ed4d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1388113340 con 0x7f138810fed0 2024-05-09T08:56:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.687+0000 7f138dd4b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1388108830 0x7f138811cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.687+0000 7f138dd4b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1388108830 0x7f138811cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52406/0 (socket says 172.21.15.98:52406) 2024-05-09T08:56:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.687+0000 7f138dd4b640 1 -- 172.21.15.98:0/1548498841 learned_addr learned my addr 172.21.15.98:0/1548498841 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.688+0000 7f138dd4b640 1 -- 172.21.15.98:0/1548498841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f138810fed0 msgr2=0x7f13881193d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:56:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.688+0000 7f138dd4b640 1 --2- 172.21.15.98:0/1548498841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f138810fed0 0x7f13881193d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.688+0000 7f138dd4b640 1 -- 172.21.15.98:0/1548498841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13881092b0 msgr2=0x7f1388118e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:56:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.688+0000 7f138dd4b640 1 --2- 172.21.15.98:0/1548498841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13881092b0 0x7f1388118e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.688+0000 7f138dd4b640 1 -- 172.21.15.98:0/1548498841 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1388119b00 con 0x7f1388108830 2024-05-09T08:56:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.688+0000 7f138dd4b640 1 --2- 172.21.15.98:0/1548498841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1388108830 0x7f138811cd60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1378006010 tx=0x7f13780045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.689+0000 7f137effd640 1 -- 172.21.15.98:0/1548498841 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1378012020 con 0x7f1388108830 2024-05-09T08:56:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.689+0000 7f138ed4d640 1 -- 172.21.15.98:0/1548498841 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13881ccca0 con 0x7f1388108830 2024-05-09T08:56:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.689+0000 7f138ed4d640 1 -- 172.21.15.98:0/1548498841 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13881cd0a0 con 0x7f1388108830 2024-05-09T08:56:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.689+0000 7f137effd640 1 -- 172.21.15.98:0/1548498841 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1378005500 con 0x7f1388108830 2024-05-09T08:56:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.689+0000 7f137effd640 1 -- 172.21.15.98:0/1548498841 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1378046020 con 0x7f1388108830 2024-05-09T08:56:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.693+0000 7f1392d95740 1 Processor -- start 2024-05-09T08:56:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.696+0000 7f137effd640 1 -- 172.21.15.98:0/1548498841 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1378015020 con 0x7f1388108830 2024-05-09T08:56:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.697+0000 7f1392d95740 1 -- start start 2024-05-09T08:56:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.698+0000 7f1392d95740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55ddfd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.698+0000 7f1392d95740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55ddfde40 0x55c55de07fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.698+0000 7f1392d95740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de08510 0x55c55de0a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.698+0000 7f1392d95740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c55dd62c40 con 0x55c55ddfde40 2024-05-09T08:56:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.698+0000 7f1392d95740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c55dd0a580 con 0x55c55ddfd520 2024-05-09T08:56:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.698+0000 7f1392d95740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c55dd5ec00 con 0x55c55de08510 2024-05-09T08:56:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.698+0000 7f138d54a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de08510 0x55c55de0a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.698+0000 7f138d54a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de08510 0x55c55de0a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39476/0 (socket says 172.21.15.98:39476) 2024-05-09T08:56:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.698+0000 7f138d54a640 1 -- 172.21.15.98:0/738832671 learned_addr learned my addr 172.21.15.98:0/738832671 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.698+0000 7f138dd4b640 1 --2- 172.21.15.98:0/738832671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55ddfde40 0x55c55de07fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.698+0000 7f138e54c640 1 --2- 172.21.15.98:0/738832671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55ddfd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.699+0000 7f138d54a640 1 -- 172.21.15.98:0/738832671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 msgr2=0x55c55ddfd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.699+0000 7f138d54a640 1 --2- 172.21.15.98:0/738832671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55ddfd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.699+0000 7f138d54a640 1 -- 172.21.15.98:0/738832671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55ddfde40 msgr2=0x55c55de07fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.699+0000 7f138d54a640 1 --2- 172.21.15.98:0/738832671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55ddfde40 0x55c55de07fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.699+0000 7f138d54a640 1 -- 172.21.15.98:0/738832671 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c55da763f0 con 0x55c55de08510 2024-05-09T08:56:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.700+0000 7f138d54a640 1 --2- 172.21.15.98:0/738832671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de08510 0x55c55de0a900 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1384005fd0 tx=0x7f138400bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7937ff8ab1f5c3c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.700+0000 7f137effd640 1 --2- 172.21.15.98:0/1548498841 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f13740a2e00 0x7f13740a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.701+0000 7f137effd640 1 -- 172.21.15.98:0/1548498841 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f13780f13c0 con 0x7f1388108830 2024-05-09T08:56:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.701+0000 7f1373fff640 1 -- 172.21.15.98:0/738832671 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384016070 con 0x55c55de08510 2024-05-09T08:56:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.701+0000 7f1373fff640 1 -- 172.21.15.98:0/738832671 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f138400dc10 con 0x55c55de08510 2024-05-09T08:56:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.704+0000 7f1373fff640 1 -- 172.21.15.98:0/738832671 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384010460 con 0x55c55de08510 2024-05-09T08:56:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.705+0000 7f138e54c640 1 --2- 172.21.15.98:0/1548498841 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f13740a2e00 0x7f13740a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.705+0000 7f1392d95740 1 -- 172.21.15.98:0/738832671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de08510 msgr2=0x55c55de0a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.705+0000 7f1392d95740 1 --2- 172.21.15.98:0/738832671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de08510 0x55c55de0a900 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1384005fd0 tx=0x7f138400bf20 comp rx=0 tx=0).stop 2024-05-09T08:56:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.705+0000 7f1392d95740 1 -- 172.21.15.98:0/738832671 shutdown_connections 2024-05-09T08:56:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.705+0000 7f1392d95740 1 --2- 172.21.15.98:0/738832671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de08510 0x55c55de0a900 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.705+0000 7f1392d95740 1 --2- 172.21.15.98:0/738832671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55ddfde40 0x55c55de07fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.705+0000 7f1392d95740 1 --2- 172.21.15.98:0/738832671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55ddfd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.705+0000 7f1392d95740 1 -- 172.21.15.98:0/738832671 >> 172.21.15.98:0/738832671 conn(0x55c55ddf9840 msgr2=0x55c55ddf9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.707+0000 7f138e54c640 1 --2- 172.21.15.98:0/1548498841 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f13740a2e00 0x7f13740a52c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1380009e60 tx=0x7f1380009450 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.708+0000 7f1392d95740 1 -- 172.21.15.98:0/738832671 shutdown_connections 2024-05-09T08:56:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.709+0000 7f1392d95740 1 -- 172.21.15.98:0/738832671 wait complete. 2024-05-09T08:56:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.709+0000 7f1392d95740 1 -- start start 2024-05-09T08:56:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.709+0000 7f1392d95740 1 Processor -- start 2024-05-09T08:56:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.710+0000 7f1392d95740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55de05460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.710+0000 7f1392d95740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55ddfde40 0x55c55de059a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.710+0000 7f1392d95740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55de08510 0x55c55de05ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.710+0000 7f1392d95740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c55de0b300 con 0x55c55de08510 2024-05-09T08:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.710+0000 7f1392d95740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c55dd1dc40 con 0x55c55ddfd520 2024-05-09T08:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.710+0000 7f1392d95740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c55de0b480 con 0x55c55ddfde40 2024-05-09T08:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.710+0000 7f1392d95740 10 client.0 _ll_register_callbacks cb 0x55c55da75be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.710+0000 7f138d54a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55ddfde40 0x55c55de059a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.710+0000 7f138d54a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55ddfde40 0x55c55de059a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39480/0 (socket says 172.21.15.98:39480) 2024-05-09T08:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.710+0000 7f138d54a640 1 -- 172.21.15.98:0/3572482201 learned_addr learned my addr 172.21.15.98:0/3572482201 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.711+0000 7f138dd4b640 1 --2- 172.21.15.98:0/3572482201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55de05460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.711+0000 7f138d54a640 1 -- 172.21.15.98:0/3572482201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 msgr2=0x55c55de05460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.711+0000 7f138d54a640 1 --2- 172.21.15.98:0/3572482201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55de05460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.711+0000 7f138d54a640 1 -- 172.21.15.98:0/3572482201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55de08510 msgr2=0x55c55de05ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.711+0000 7f138d54a640 1 --2- 172.21.15.98:0/3572482201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55de08510 0x55c55de05ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.711+0000 7f138d54a640 1 -- 172.21.15.98:0/3572482201 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c55db407d0 con 0x55c55ddfde40 2024-05-09T08:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.711+0000 7f138d54a640 1 --2- 172.21.15.98:0/3572482201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55ddfde40 0x55c55de059a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f138400bfd0 tx=0x7f138400b690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7539e340820772a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.712+0000 7f13717fa640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:56:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.712+0000 7f13717fa640 1 -- 172.21.15.98:0/3572482201 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384016070 con 0x55c55ddfde40 2024-05-09T08:56:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.712+0000 7f1392d95740 1 -- 172.21.15.98:0/3572482201 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f138400dc10 con 0x55c55ddfde40 2024-05-09T08:56:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.713+0000 7f13717fa640 1 -- 172.21.15.98:0/3572482201 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f1384003d60 con 0x55c55ddfde40 2024-05-09T08:56:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.713+0000 7f13717fa640 1 -- 172.21.15.98:0/3572482201 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f138402c050 con 0x55c55ddfde40 2024-05-09T08:56:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.717+0000 7f1392d95740 10 client.26395 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.717+0000 7f1392d95740 1 -- 172.21.15.98:0/3572482201 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c55de17580 con 0x55c55ddfde40 2024-05-09T08:56:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.722+0000 7f13717fa640 1 -- 172.21.15.98:0/3572482201 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f138400dc10 con 0x55c55ddfde40 2024-05-09T08:56:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.722+0000 7f13717fa640 1 -- 172.21.15.98:0/3572482201 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f13840046c0 con 0x55c55ddfde40 2024-05-09T08:56:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.722+0000 7f1392d95740 10 client.26395 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.722+0000 7f1392d95740 10 client.26395 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:56:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.722+0000 7f1392d95740 20 client.26395 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.722+0000 7f1392d95740 1 --2- 172.21.15.98:0/3572482201 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c55de18e30 0x55c55de1b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.722+0000 7f1392d95740 4 client.26395 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.722+0000 7f1392d95740 1 -- 172.21.15.98:0/3572482201 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c55dd70370 con 0x55c55de18e30 2024-05-09T08:56:12.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.727+0000 7f138dd4b640 1 --2- 172.21.15.98:0/3572482201 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c55de18e30 0x55c55de1b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.734+0000 7f138dd4b640 1 --2- 172.21.15.98:0/3572482201 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c55de18e30 0x55c55de1b270 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6ed2bb215eadf61b server_cookie=cb78d1d7a4514ffe in_seq=0 out_seq=0 2024-05-09T08:56:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.734+0000 7f13717fa640 10 client.26395 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:56:12.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.735+0000 7f13717fa640 1 -- 172.21.15.98:0/3572482201 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c55dd70370 con 0x55c55de18e30 2024-05-09T08:56:12.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.735+0000 7f13717fa640 10 client.26395 handle_command_reply: tid=0 2024-05-09T08:56:12.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.735+0000 7f1392d95740 1 client.26395 shutdown 2024-05-09T08:56:12.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:12 smithi105 ceph-mon[34498]: pgmap v171: 129 pgs: 129 active+clean; 43 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 919 MiB/s wr, 2.34k op/s 2024-05-09T08:56:12.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.739+0000 7f1392d95740 1 -- 172.21.15.98:0/3572482201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55ddfde40 msgr2=0x55c55de059a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.739+0000 7f1392d95740 1 --2- 172.21.15.98:0/3572482201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55ddfde40 0x55c55de059a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f138400bfd0 tx=0x7f138400b690 comp rx=0 tx=0).stop 2024-05-09T08:56:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.739+0000 7f1392d95740 1 -- 172.21.15.98:0/3572482201 shutdown_connections 2024-05-09T08:56:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.739+0000 7f1392d95740 1 --2- 172.21.15.98:0/3572482201 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c55de18e30 0x55c55de1b270 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.739+0000 7f1392d95740 1 --2- 172.21.15.98:0/3572482201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55de08510 0x55c55de05ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.739+0000 7f1392d95740 1 --2- 172.21.15.98:0/3572482201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55ddfde40 0x55c55de059a0 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f138400bfd0 tx=0x7f138400b690 comp rx=0 tx=0).stop 2024-05-09T08:56:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.739+0000 7f1392d95740 1 --2- 172.21.15.98:0/3572482201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55de05460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.739+0000 7f1392d95740 1 -- 172.21.15.98:0/3572482201 >> 172.21.15.98:0/3572482201 conn(0x55c55ddf9840 msgr2=0x55c55ddff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.739+0000 7f13717fa640 0 client.26395 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:56:12.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.742+0000 7f1392d95740 1 -- 172.21.15.98:0/3572482201 shutdown_connections 2024-05-09T08:56:12.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.742+0000 7f1392d95740 1 -- 172.21.15.98:0/3572482201 wait complete. 2024-05-09T08:56:12.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.742+0000 7f1392d95740 20 client.26395 trim_cache size 0 max 16384 2024-05-09T08:56:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.786+0000 7f1392d95740 1 Processor -- start 2024-05-09T08:56:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.786+0000 7f1392d95740 1 -- start start 2024-05-09T08:56:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.786+0000 7f1392d95740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55ddf9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.786+0000 7f1392d95740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55de08510 0x55c55de05d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.786+0000 7f1392d95740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de04440 0x55c55de20b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.786+0000 7f1392d95740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f136c010080 con 0x55c55de08510 2024-05-09T08:56:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.786+0000 7f1392d95740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f136c011bc0 con 0x55c55ddfd520 2024-05-09T08:56:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.786+0000 7f1392d95740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f136c010750 con 0x55c55de04440 2024-05-09T08:56:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.787+0000 7f138dd4b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55ddf9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.787+0000 7f138dd4b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55ddf9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52440/0 (socket says 172.21.15.98:52440) 2024-05-09T08:56:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.787+0000 7f138e54c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de04440 0x55c55de20b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.787+0000 7f138dd4b640 1 -- 172.21.15.98:0/2696832501 learned_addr learned my addr 172.21.15.98:0/2696832501 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.787+0000 7f138dd4b640 1 -- 172.21.15.98:0/2696832501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de04440 msgr2=0x55c55de20b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.787+0000 7f138dd4b640 1 --2- 172.21.15.98:0/2696832501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de04440 0x55c55de20b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.787+0000 7f138dd4b640 1 -- 172.21.15.98:0/2696832501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55de08510 msgr2=0x55c55de05d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.788+0000 7f138dd4b640 1 --2- 172.21.15.98:0/2696832501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55de08510 0x55c55de05d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.788+0000 7f138dd4b640 1 -- 172.21.15.98:0/2696832501 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c55ddfdc30 con 0x55c55ddfd520 2024-05-09T08:56:12.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.788+0000 7f138dd4b640 1 --2- 172.21.15.98:0/2696832501 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55ddf9140 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1378000c00 tx=0x7f137800d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=331ae45b88a18cab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:12.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.796+0000 7f13737fe640 1 -- 172.21.15.98:0/2696832501 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13780c1870 con 0x55c55ddfd520 2024-05-09T08:56:12.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.796+0000 7f13737fe640 1 -- 172.21.15.98:0/2696832501 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13780c6130 con 0x55c55ddfd520 2024-05-09T08:56:12.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.796+0000 7f13737fe640 1 -- 172.21.15.98:0/2696832501 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13780ee970 con 0x55c55ddfd520 2024-05-09T08:56:12.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.799+0000 7f1392d95740 1 -- 172.21.15.98:0/2696832501 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 msgr2=0x55c55ddf9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.799+0000 7f1392d95740 1 --2- 172.21.15.98:0/2696832501 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55ddf9140 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1378000c00 tx=0x7f137800d000 comp rx=0 tx=0).stop 2024-05-09T08:56:12.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.799+0000 7f1392d95740 1 -- 172.21.15.98:0/2696832501 shutdown_connections 2024-05-09T08:56:12.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.800+0000 7f1392d95740 1 --2- 172.21.15.98:0/2696832501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de04440 0x55c55de20b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.800+0000 7f1392d95740 1 --2- 172.21.15.98:0/2696832501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55de08510 0x55c55de05d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.800+0000 7f1392d95740 1 --2- 172.21.15.98:0/2696832501 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55ddf9140 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.800+0000 7f1392d95740 1 -- 172.21.15.98:0/2696832501 >> 172.21.15.98:0/2696832501 conn(0x55c55ddf9840 msgr2=0x55c55de03d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:12.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.802+0000 7f1392d95740 1 -- 172.21.15.98:0/2696832501 shutdown_connections 2024-05-09T08:56:12.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.804+0000 7f1392d95740 1 -- 172.21.15.98:0/2696832501 wait complete. 2024-05-09T08:56:12.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.804+0000 7f1392d95740 1 -- start start 2024-05-09T08:56:12.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.805+0000 7f1392d95740 1 Processor -- start 2024-05-09T08:56:12.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.808+0000 7f1392d95740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55de0b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.808+0000 7f1392d95740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de04440 0x55c55de0bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.808+0000 7f1392d95740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55de08510 0x55c55de7d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.808+0000 7f1392d95740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f136c0114f0 con 0x55c55de08510 2024-05-09T08:56:12.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.808+0000 7f1392d95740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f136c010e20 con 0x55c55ddfd520 2024-05-09T08:56:12.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.808+0000 7f1392d95740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f136c00e060 con 0x55c55de04440 2024-05-09T08:56:12.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.808+0000 7f1392d95740 10 client.0 _ll_register_callbacks cb 0x7f138400b8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:12.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.808+0000 7f138dd4b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55de0b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.808+0000 7f138dd4b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55de0b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52442/0 (socket says 172.21.15.98:52442) 2024-05-09T08:56:12.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.808+0000 7f138dd4b640 1 -- 172.21.15.98:0/2675051141 learned_addr learned my addr 172.21.15.98:0/2675051141 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:12.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.808+0000 7f138d54a640 1 --2- 172.21.15.98:0/2675051141 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de04440 0x55c55de0bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.808+0000 7f138dd4b640 1 -- 172.21.15.98:0/2675051141 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de04440 msgr2=0x55c55de0bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.808+0000 7f138dd4b640 1 --2- 172.21.15.98:0/2675051141 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de04440 0x55c55de0bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.809+0000 7f138dd4b640 1 -- 172.21.15.98:0/2675051141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55de08510 msgr2=0x55c55de7d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:12.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.809+0000 7f138dd4b640 1 --2- 172.21.15.98:0/2675051141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55de08510 0x55c55de7d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.809+0000 7f138dd4b640 1 -- 172.21.15.98:0/2675051141 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c55de7d920 con 0x55c55ddfd520 2024-05-09T08:56:12.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.809+0000 7f138dd4b640 1 --2- 172.21.15.98:0/2675051141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55de0b720 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1378004eb0 tx=0x7f13780c18e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56d4cbd1fdd0c05e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:12.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.814+0000 7f13717fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:56:12.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.815+0000 7f13717fa640 1 -- 172.21.15.98:0/2675051141 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13780ee970 con 0x55c55ddfd520 2024-05-09T08:56:12.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.815+0000 7f13717fa640 1 -- 172.21.15.98:0/2675051141 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f13780b9b30 con 0x55c55ddfd520 2024-05-09T08:56:12.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.817+0000 7f1392d95740 1 -- 172.21.15.98:0/2675051141 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13780c6130 con 0x55c55ddfd520 2024-05-09T08:56:12.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.822+0000 7f13717fa640 1 -- 172.21.15.98:0/2675051141 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f13780bd720 con 0x55c55ddfd520 2024-05-09T08:56:12.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.824+0000 7f1392d95740 10 client.26429 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:12.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.824+0000 7f1392d95740 1 -- 172.21.15.98:0/2675051141 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c55de7ea30 con 0x55c55ddfd520 2024-05-09T08:56:12.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.824+0000 7f13717fa640 1 -- 172.21.15.98:0/2675051141 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13780c6130 con 0x55c55ddfd520 2024-05-09T08:56:12.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.824+0000 7f13717fa640 1 -- 172.21.15.98:0/2675051141 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f13780c5ca0 con 0x55c55ddfd520 2024-05-09T08:56:12.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.824+0000 7f1392d95740 10 client.26429 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:12.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.824+0000 7f1392d95740 10 client.26429 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:56:12.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.824+0000 7f1392d95740 20 client.26429 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:12.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.824+0000 7f1392d95740 1 --2- 172.21.15.98:0/2675051141 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c55de7eda0 0x55c55de811c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:12.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.824+0000 7f1392d95740 4 client.26429 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:56:12.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.824+0000 7f1392d95740 1 -- 172.21.15.98:0/2675051141 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c55dd6a6c0 con 0x55c55de7eda0 2024-05-09T08:56:12.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.825+0000 7f138d54a640 1 --2- 172.21.15.98:0/2675051141 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c55de7eda0 0x55c55de811c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:12.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.825+0000 7f13717fa640 10 client.26429 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:56:12.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.825+0000 7f138d54a640 1 --2- 172.21.15.98:0/2675051141 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c55de7eda0 0x55c55de811c0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f03b4945c7a5c8ca server_cookie=403a60a9f057f5f9 in_seq=0 out_seq=0 2024-05-09T08:56:12.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.825+0000 7f13717fa640 1 -- 172.21.15.98:0/2675051141 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c55dd6a6c0 con 0x55c55de7eda0 2024-05-09T08:56:12.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.825+0000 7f13717fa640 10 client.26429 handle_command_reply: tid=0 2024-05-09T08:56:12.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.826+0000 7f1392d95740 1 client.26429 shutdown 2024-05-09T08:56:12.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.826+0000 7f1392d95740 1 -- 172.21.15.98:0/2675051141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 msgr2=0x55c55de0b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.826+0000 7f1392d95740 1 --2- 172.21.15.98:0/2675051141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55de0b720 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1378004eb0 tx=0x7f13780c18e0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.830+0000 7f1392d95740 1 -- 172.21.15.98:0/2675051141 shutdown_connections 2024-05-09T08:56:12.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.830+0000 7f1392d95740 1 --2- 172.21.15.98:0/2675051141 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55c55de7eda0 0x55c55de811c0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.830+0000 7f1392d95740 1 --2- 172.21.15.98:0/2675051141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c55de08510 0x55c55de7d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.830+0000 7f1392d95740 1 --2- 172.21.15.98:0/2675051141 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c55de04440 0x55c55de0bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.830+0000 7f1392d95740 1 --2- 172.21.15.98:0/2675051141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c55ddfd520 0x55c55de0b720 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.830+0000 7f13717fa640 0 client.26429 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:56:12.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.830+0000 7f1392d95740 1 -- 172.21.15.98:0/2675051141 >> 172.21.15.98:0/2675051141 conn(0x55c55ddf9840 msgr2=0x55c55ddfecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:12.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.832+0000 7f1392d95740 1 -- 172.21.15.98:0/2675051141 shutdown_connections 2024-05-09T08:56:12.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.832+0000 7f1392d95740 1 -- 172.21.15.98:0/2675051141 wait complete. 2024-05-09T08:56:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.835+0000 7f1392d95740 20 client.26429 trim_cache size 0 max 16384 2024-05-09T08:56:12.836 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:56:12.840 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:12 smithi098 ceph-mon[30092]: pgmap v171: 129 pgs: 129 active+clean; 43 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 919 MiB/s wr, 2.34k op/s 2024-05-09T08:56:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.841+0000 7f138ed4d640 1 -- 172.21.15.98:0/1548498841 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f13740a2e00 msgr2=0x7f13740a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.841+0000 7f138ed4d640 1 --2- 172.21.15.98:0/1548498841 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f13740a2e00 0x7f13740a52c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1380009e60 tx=0x7f1380009450 comp rx=0 tx=0).stop 2024-05-09T08:56:12.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.841+0000 7f138ed4d640 1 -- 172.21.15.98:0/1548498841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1388108830 msgr2=0x7f138811cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:12.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.841+0000 7f138ed4d640 1 --2- 172.21.15.98:0/1548498841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1388108830 0x7f138811cd60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1378006010 tx=0x7f13780045e0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.844+0000 7f138ed4d640 1 -- 172.21.15.98:0/1548498841 shutdown_connections 2024-05-09T08:56:12.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.844+0000 7f138ed4d640 1 --2- 172.21.15.98:0/1548498841 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f13740a2e00 0x7f13740a52c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.844+0000 7f138ed4d640 1 --2- 172.21.15.98:0/1548498841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f138810fed0 0x7f13881193d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.844+0000 7f138ed4d640 1 --2- 172.21.15.98:0/1548498841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13881092b0 0x7f1388118e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.844+0000 7f138ed4d640 1 --2- 172.21.15.98:0/1548498841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1388108830 0x7f138811cd60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:12.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.844+0000 7f138ed4d640 1 -- 172.21.15.98:0/1548498841 >> 172.21.15.98:0/1548498841 conn(0x7f138806c7b0 msgr2=0x7f1388071890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:12.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.846+0000 7f138ed4d640 1 -- 172.21.15.98:0/1548498841 shutdown_connections 2024-05-09T08:56:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:12.849+0000 7f138ed4d640 1 -- 172.21.15.98:0/1548498841 wait complete. 2024-05-09T08:56:12.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:56:12.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:12 smithi142 ceph-mon[34625]: pgmap v171: 129 pgs: 129 active+clean; 43 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 919 MiB/s wr, 2.34k op/s 2024-05-09T08:56:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.165+0000 7fa03bfff640 1 -- 172.21.15.98:0/3415043995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa03c11ad50 msgr2=0x7fa03c11d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.165+0000 7fa03bfff640 1 --2- 172.21.15.98:0/3415043995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa03c11ad50 0x7fa03c11d140 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa034009ca0 tx=0x7fa03402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.165+0000 7fa03bfff640 1 -- 172.21.15.98:0/3415043995 shutdown_connections 2024-05-09T08:56:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.165+0000 7fa03bfff640 1 --2- 172.21.15.98:0/3415043995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa03c11ad50 0x7fa03c11d140 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.165+0000 7fa03bfff640 1 --2- 172.21.15.98:0/3415043995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa03c06dae0 0x7fa03c11a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.165+0000 7fa03bfff640 1 --2- 172.21.15.98:0/3415043995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa03c06d1c0 0x7fa03c06d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.165+0000 7fa03bfff640 1 -- 172.21.15.98:0/3415043995 >> 172.21.15.98:0/3415043995 conn(0x7fa03c06c6c0 msgr2=0x7fa03c10a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.165+0000 7fa03bfff640 1 -- 172.21.15.98:0/3415043995 shutdown_connections 2024-05-09T08:56:13.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.165+0000 7fa03bfff640 1 -- 172.21.15.98:0/3415043995 wait complete. 2024-05-09T08:56:13.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.166+0000 7fa03bfff640 1 Processor -- start 2024-05-09T08:56:13.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.167+0000 7fa03bfff640 1 -- start start 2024-05-09T08:56:13.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.167+0000 7fa03bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa03c06d1c0 0x7fa03c11a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.167+0000 7fa03bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa03c06dae0 0x7fa03c113370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.167+0000 7fa03bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa03c11ad50 0x7fa03c1138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.167+0000 7fa03bfff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c120890 con 0x7fa03c06dae0 2024-05-09T08:56:13.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.167+0000 7fa03bfff640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c120710 con 0x7fa03c11ad50 2024-05-09T08:56:13.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.167+0000 7fa03bfff640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c120a10 con 0x7fa03c06d1c0 2024-05-09T08:56:13.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.169+0000 7fa03affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa03c06d1c0 0x7fa03c11a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.169+0000 7fa03affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa03c06d1c0 0x7fa03c11a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39504/0 (socket says 172.21.15.98:39504) 2024-05-09T08:56:13.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.169+0000 7fa03affd640 1 -- 172.21.15.98:0/1328339992 learned_addr learned my addr 172.21.15.98:0/1328339992 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:13.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.169+0000 7fa03a7fc640 1 --2- 172.21.15.98:0/1328339992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa03c06dae0 0x7fa03c113370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.169+0000 7fa03b7fe640 1 --2- 172.21.15.98:0/1328339992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa03c11ad50 0x7fa03c1138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.170+0000 7fa03affd640 1 -- 172.21.15.98:0/1328339992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa03c11ad50 msgr2=0x7fa03c1138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.170+0000 7fa03affd640 1 --2- 172.21.15.98:0/1328339992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa03c11ad50 0x7fa03c1138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.170+0000 7fa03affd640 1 -- 172.21.15.98:0/1328339992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa03c06dae0 msgr2=0x7fa03c113370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.170+0000 7fa03affd640 1 --2- 172.21.15.98:0/1328339992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa03c06dae0 0x7fa03c113370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.170+0000 7fa03affd640 1 -- 172.21.15.98:0/1328339992 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa03c113e70 con 0x7fa03c06d1c0 2024-05-09T08:56:13.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.170+0000 7fa03affd640 1 --2- 172.21.15.98:0/1328339992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa03c06d1c0 0x7fa03c11a2f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa03000d770 tx=0x7fa03000dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:13.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.171+0000 7fa023fff640 1 -- 172.21.15.98:0/1328339992 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030017070 con 0x7fa03c06d1c0 2024-05-09T08:56:13.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.171+0000 7fa03bfff640 1 -- 172.21.15.98:0/1328339992 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa03c114160 con 0x7fa03c06d1c0 2024-05-09T08:56:13.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.171+0000 7fa023fff640 1 -- 172.21.15.98:0/1328339992 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa030005500 con 0x7fa03c06d1c0 2024-05-09T08:56:13.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.171+0000 7fa03bfff640 1 -- 172.21.15.98:0/1328339992 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa03c118140 con 0x7fa03c06d1c0 2024-05-09T08:56:13.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.171+0000 7fa023fff640 1 -- 172.21.15.98:0/1328339992 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030020060 con 0x7fa03c06d1c0 2024-05-09T08:56:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.177+0000 7fa043fc9740 1 Processor -- start 2024-05-09T08:56:13.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.178+0000 7fa043fc9740 1 -- start start 2024-05-09T08:56:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.179+0000 7fa043fc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509520 0x563481509900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.179+0000 7fa043fc9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481509e40 0x563481513fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.179+0000 7fa043fc9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481514510 0x563481516900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.179+0000 7fa043fc9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56348146ec40 con 0x563481509520 2024-05-09T08:56:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.179+0000 7fa043fc9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563481416580 con 0x563481509e40 2024-05-09T08:56:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.179+0000 7fa043fc9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56348146ac00 con 0x563481514510 2024-05-09T08:56:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.180+0000 7fa03b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509520 0x563481509900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.180+0000 7fa03b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509520 0x563481509900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39592/0 (socket says 172.21.15.98:39592) 2024-05-09T08:56:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.180+0000 7fa03b7fe640 1 -- 172.21.15.98:0/1066940665 learned_addr learned my addr 172.21.15.98:0/1066940665 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.180+0000 7fa023fff640 1 -- 172.21.15.98:0/1328339992 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa030009c30 con 0x7fa03c06d1c0 2024-05-09T08:56:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.181+0000 7fa03affd640 1 --2- 172.21.15.98:0/1066940665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481509e40 0x563481513fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.181+0000 7fa03a7fc640 1 --2- 172.21.15.98:0/1066940665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481514510 0x563481516900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.181+0000 7fa023fff640 1 --2- 172.21.15.98:0/1328339992 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa0240a2dc0 0x7fa0240a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.183+0000 7fa03b7fe640 1 -- 172.21.15.98:0/1066940665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481514510 msgr2=0x563481516900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.183+0000 7fa03b7fe640 1 --2- 172.21.15.98:0/1066940665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481514510 0x563481516900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.183+0000 7fa03b7fe640 1 -- 172.21.15.98:0/1066940665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481509e40 msgr2=0x563481513fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.184+0000 7fa03b7fe640 1 --2- 172.21.15.98:0/1066940665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481509e40 0x563481513fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.184+0000 7fa03b7fe640 1 -- 172.21.15.98:0/1066940665 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634811823f0 con 0x563481509520 2024-05-09T08:56:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.184+0000 7fa03b7fe640 1 --2- 172.21.15.98:0/1328339992 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa0240a2dc0 0x7fa0240a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.184+0000 7fa023fff640 1 -- 172.21.15.98:0/1328339992 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fa0300cf360 con 0x7fa03c06d1c0 2024-05-09T08:56:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.185+0000 7fa03b7fe640 1 --2- 172.21.15.98:0/1066940665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509520 0x563481509900 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa034005e30 tx=0x7fa0340314c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f191989ad53137bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:13.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.185+0000 7fa03b7fe640 1 --2- 172.21.15.98:0/1328339992 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa0240a2dc0 0x7fa0240a5280 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa03404b000 tx=0x7fa034048000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:13.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.185+0000 7fa0217fa640 1 -- 172.21.15.98:0/1066940665 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0340315b0 con 0x563481509520 2024-05-09T08:56:13.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.186+0000 7fa0217fa640 1 -- 172.21.15.98:0/1066940665 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa034036040 con 0x563481509520 2024-05-09T08:56:13.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.187+0000 7fa043fc9740 1 -- 172.21.15.98:0/1066940665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509520 msgr2=0x563481509900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.187+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1066940665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509520 0x563481509900 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa034005e30 tx=0x7fa0340314c0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.187+0000 7fa043fc9740 1 -- 172.21.15.98:0/1066940665 shutdown_connections 2024-05-09T08:56:13.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.187+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1066940665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481514510 0x563481516900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.187+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1066940665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481509e40 0x563481513fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.187+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1066940665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509520 0x563481509900 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.187+0000 7fa043fc9740 1 -- 172.21.15.98:0/1066940665 >> 172.21.15.98:0/1066940665 conn(0x563481505840 msgr2=0x563481505c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:13.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.189+0000 7fa043fc9740 1 -- 172.21.15.98:0/1066940665 shutdown_connections 2024-05-09T08:56:13.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.190+0000 7fa043fc9740 1 -- 172.21.15.98:0/1066940665 wait complete. 2024-05-09T08:56:13.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.190+0000 7fa043fc9740 1 -- start start 2024-05-09T08:56:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.190+0000 7fa043fc9740 1 Processor -- start 2024-05-09T08:56:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.191+0000 7fa043fc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509520 0x563481513f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.191+0000 7fa043fc9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509e40 0x5634815112d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.191+0000 7fa043fc9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481514510 0x563481511a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.191+0000 7fa043fc9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5634815173a0 con 0x563481509520 2024-05-09T08:56:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.191+0000 7fa043fc9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563481429c40 con 0x563481514510 2024-05-09T08:56:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.191+0000 7fa043fc9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563481517520 con 0x563481509e40 2024-05-09T08:56:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.191+0000 7fa043fc9740 10 client.0 _ll_register_callbacks cb 0x563481181be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.192+0000 7fa03affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509520 0x563481513f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.192+0000 7fa03affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509520 0x563481513f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39594/0 (socket says 172.21.15.98:39594) 2024-05-09T08:56:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.192+0000 7fa03affd640 1 -- 172.21.15.98:0/1350014456 learned_addr learned my addr 172.21.15.98:0/1350014456 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.193+0000 7fa03b7fe640 1 --2- 172.21.15.98:0/1350014456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481514510 0x563481511a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.194+0000 7fa03a7fc640 1 --2- 172.21.15.98:0/1350014456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509e40 0x5634815112d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.194+0000 7fa03affd640 1 -- 172.21.15.98:0/1350014456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509e40 msgr2=0x5634815112d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.194+0000 7fa03affd640 1 --2- 172.21.15.98:0/1350014456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509e40 0x5634815112d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.194+0000 7fa03affd640 1 -- 172.21.15.98:0/1350014456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481514510 msgr2=0x563481511a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.194+0000 7fa03affd640 1 --2- 172.21.15.98:0/1350014456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481514510 0x563481511a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.194+0000 7fa03affd640 1 -- 172.21.15.98:0/1350014456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56348124c7d0 con 0x563481509520 2024-05-09T08:56:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.194+0000 7fa03a7fc640 1 --2- 172.21.15.98:0/1350014456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509e40 0x5634815112d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:56:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.194+0000 7fa03affd640 1 --2- 172.21.15.98:0/1350014456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509520 0x563481513f40 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fa0300978c0 tx=0x7fa03009adc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27be813d79348058 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.194+0000 7fa01eff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:56:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.194+0000 7fa01eff5640 1 -- 172.21.15.98:0/1350014456 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030014050 con 0x563481509520 2024-05-09T08:56:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.194+0000 7fa043fc9740 1 -- 172.21.15.98:0/1350014456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa034036040 con 0x563481509520 2024-05-09T08:56:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.194+0000 7fa01eff5640 1 -- 172.21.15.98:0/1350014456 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fa03009a8b0 con 0x563481509520 2024-05-09T08:56:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.195+0000 7fa01eff5640 1 -- 172.21.15.98:0/1350014456 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa0300ccb10 con 0x563481509520 2024-05-09T08:56:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.195+0000 7fa043fc9740 10 client.17151 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:13.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.195+0000 7fa043fc9740 1 -- 172.21.15.98:0/1350014456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5634815234f0 con 0x563481509520 2024-05-09T08:56:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.195+0000 7fa01eff5640 1 -- 172.21.15.98:0/1350014456 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa034036040 con 0x563481509520 2024-05-09T08:56:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.196+0000 7fa01eff5640 1 -- 172.21.15.98:0/1350014456 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fa0300971b0 con 0x563481509520 2024-05-09T08:56:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.196+0000 7fa043fc9740 10 client.17151 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.196+0000 7fa043fc9740 10 client.17151 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:56:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.196+0000 7fa043fc9740 20 client.17151 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.196+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1350014456 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x563481524890 0x563481526ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.196+0000 7fa043fc9740 4 client.17151 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.196+0000 7fa043fc9740 1 -- 172.21.15.98:0/1350014456 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56348147c370 con 0x563481524890 2024-05-09T08:56:13.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.197+0000 7fa03a7fc640 1 --2- 172.21.15.98:0/1350014456 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x563481524890 0x563481526ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.198+0000 7fa01eff5640 10 client.17151 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:56:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.198+0000 7fa03a7fc640 1 --2- 172.21.15.98:0/1350014456 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x563481524890 0x563481526ce0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b87dcec4b52f5e61 server_cookie=67ee6505089cb70 in_seq=0 out_seq=0 2024-05-09T08:56:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.201+0000 7fa01eff5640 1 -- 172.21.15.98:0/1350014456 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56348147c370 con 0x563481524890 2024-05-09T08:56:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.201+0000 7fa01eff5640 10 client.17151 handle_command_reply: tid=0 2024-05-09T08:56:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.201+0000 7fa043fc9740 1 client.17151 shutdown 2024-05-09T08:56:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.202+0000 7fa043fc9740 1 -- 172.21.15.98:0/1350014456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509520 msgr2=0x563481513f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.202+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1350014456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509520 0x563481513f40 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fa0300978c0 tx=0x7fa03009adc0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.202+0000 7fa043fc9740 1 -- 172.21.15.98:0/1350014456 shutdown_connections 2024-05-09T08:56:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.202+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1350014456 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x563481524890 0x563481526ce0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.202+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1350014456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481514510 0x563481511a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.202+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1350014456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509e40 0x5634815112d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.202+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1350014456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509520 0x563481513f40 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.202+0000 7fa043fc9740 1 -- 172.21.15.98:0/1350014456 >> 172.21.15.98:0/1350014456 conn(0x563481505840 msgr2=0x56348150b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.202+0000 7fa01eff5640 0 client.17151 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:56:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.202+0000 7fa043fc9740 1 -- 172.21.15.98:0/1350014456 shutdown_connections 2024-05-09T08:56:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.202+0000 7fa043fc9740 1 -- 172.21.15.98:0/1350014456 wait complete. 2024-05-09T08:56:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.202+0000 7fa043fc9740 20 client.17151 trim_cache size 0 max 16384 2024-05-09T08:56:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.226+0000 7fa043fc9740 1 Processor -- start 2024-05-09T08:56:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.226+0000 7fa043fc9740 1 -- start start 2024-05-09T08:56:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.226+0000 7fa043fc9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509520 0x563481520340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.226+0000 7fa043fc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509e40 0x563481504840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.227+0000 7fa03affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509520 0x563481520340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.227+0000 7fa03affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509520 0x563481520340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39536/0 (socket says 172.21.15.98:39536) 2024-05-09T08:56:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.227+0000 7fa03a7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509e40 0x563481504840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.227+0000 7fa03a7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509e40 0x563481504840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39600/0 (socket says 172.21.15.98:39600) 2024-05-09T08:56:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.227+0000 7fa03affd640 1 -- 172.21.15.98:0/988273214 learned_addr learned my addr 172.21.15.98:0/988273214 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.227+0000 7fa043fc9740 1 --2- 172.21.15.98:0/988273214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481514510 0x56348152c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.227+0000 7fa043fc9740 1 -- 172.21.15.98:0/988273214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa02800ff60 con 0x563481509e40 2024-05-09T08:56:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.227+0000 7fa043fc9740 1 -- 172.21.15.98:0/988273214 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa028011aa0 con 0x563481514510 2024-05-09T08:56:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.227+0000 7fa043fc9740 1 -- 172.21.15.98:0/988273214 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa028010630 con 0x563481509520 2024-05-09T08:56:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.227+0000 7fa03a7fc640 1 -- 172.21.15.98:0/988273214 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509520 msgr2=0x563481520340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.227+0000 7fa03a7fc640 1 --2- 172.21.15.98:0/988273214 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509520 0x563481520340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.227+0000 7fa03a7fc640 1 -- 172.21.15.98:0/988273214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481514510 msgr2=0x56348152c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.227+0000 7fa03a7fc640 1 --2- 172.21.15.98:0/988273214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481514510 0x56348152c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.227+0000 7fa03a7fc640 1 -- 172.21.15.98:0/988273214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563481514340 con 0x563481509e40 2024-05-09T08:56:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.228+0000 7fa03a7fc640 1 --2- 172.21.15.98:0/988273214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509e40 0x563481504840 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x563481527590 tx=0x7fa02c012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c947b4125fe2e3cd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.228+0000 7fa020ff9640 1 -- 172.21.15.98:0/988273214 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c025070 con 0x563481509e40 2024-05-09T08:56:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.228+0000 7fa020ff9640 1 -- 172.21.15.98:0/988273214 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02c005510 con 0x563481509e40 2024-05-09T08:56:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.228+0000 7fa020ff9640 1 -- 172.21.15.98:0/988273214 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c004dd0 con 0x563481509e40 2024-05-09T08:56:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.229+0000 7fa043fc9740 1 -- 172.21.15.98:0/988273214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509e40 msgr2=0x563481504840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.229+0000 7fa043fc9740 1 --2- 172.21.15.98:0/988273214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509e40 0x563481504840 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x563481527590 tx=0x7fa02c012820 comp rx=0 tx=0).stop 2024-05-09T08:56:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.230+0000 7fa043fc9740 1 -- 172.21.15.98:0/988273214 shutdown_connections 2024-05-09T08:56:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.230+0000 7fa043fc9740 1 --2- 172.21.15.98:0/988273214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481514510 0x56348152c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.231+0000 7fa043fc9740 1 --2- 172.21.15.98:0/988273214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481509e40 0x563481504840 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.231+0000 7fa043fc9740 1 --2- 172.21.15.98:0/988273214 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509520 0x563481520340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.231+0000 7fa043fc9740 1 -- 172.21.15.98:0/988273214 >> 172.21.15.98:0/988273214 conn(0x563481505840 msgr2=0x563481504130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.231+0000 7fa043fc9740 1 -- 172.21.15.98:0/988273214 shutdown_connections 2024-05-09T08:56:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.231+0000 7fa043fc9740 1 -- 172.21.15.98:0/988273214 wait complete. 2024-05-09T08:56:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.231+0000 7fa043fc9740 1 -- start start 2024-05-09T08:56:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.232+0000 7fa043fc9740 1 Processor -- start 2024-05-09T08:56:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.232+0000 7fa043fc9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481509520 0x563481515610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.232+0000 7fa043fc9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509e40 0x563481515b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.232+0000 7fa03affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481509520 0x563481515610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.232+0000 7fa03affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481509520 0x563481515610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52496/0 (socket says 172.21.15.98:52496) 2024-05-09T08:56:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.233+0000 7fa03affd640 1 -- 172.21.15.98:0/1080537356 learned_addr learned my addr 172.21.15.98:0/1080537356 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.233+0000 7fa03a7fc640 1 --2- 172.21.15.98:0/1080537356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509e40 0x563481515b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.233+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1080537356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481514510 0x5634815160d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.233+0000 7fa043fc9740 1 -- 172.21.15.98:0/1080537356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa0280113d0 con 0x563481514510 2024-05-09T08:56:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.233+0000 7fa043fc9740 1 -- 172.21.15.98:0/1080537356 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa028010d00 con 0x563481509520 2024-05-09T08:56:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.233+0000 7fa043fc9740 1 -- 172.21.15.98:0/1080537356 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa02800df40 con 0x563481509e40 2024-05-09T08:56:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.233+0000 7fa043fc9740 10 client.0 _ll_register_callbacks cb 0x7fa03009b300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.234+0000 7fa03affd640 1 -- 172.21.15.98:0/1080537356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509e40 msgr2=0x563481515b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.234+0000 7fa03affd640 1 --2- 172.21.15.98:0/1080537356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509e40 0x563481515b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.234+0000 7fa03affd640 1 -- 172.21.15.98:0/1080537356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481514510 msgr2=0x5634815160d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.234+0000 7fa03affd640 1 --2- 172.21.15.98:0/1080537356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481514510 0x5634815160d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.234+0000 7fa03affd640 1 -- 172.21.15.98:0/1080537356 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56348151afe0 con 0x563481509520 2024-05-09T08:56:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.234+0000 7fa043fc9740 1 -- 172.21.15.98:0/1080537356 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa02c005510 con 0x563481509520 2024-05-09T08:56:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.234+0000 7fa03affd640 1 --2- 172.21.15.98:0/1080537356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481509520 0x563481515610 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa030097e30 tx=0x7fa03009aae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68e94e04b65e8ea8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.234+0000 7fa01eff5640 10 client.26444 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:56:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.235+0000 7fa01eff5640 1 -- 172.21.15.98:0/1080537356 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0300ab050 con 0x563481509520 2024-05-09T08:56:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.235+0000 7fa01eff5640 1 -- 172.21.15.98:0/1080537356 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fa0300b4320 con 0x563481509520 2024-05-09T08:56:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.235+0000 7fa01eff5640 1 -- 172.21.15.98:0/1080537356 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa0300ccb20 con 0x563481509520 2024-05-09T08:56:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.235+0000 7fa043fc9740 10 client.26444 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.235+0000 7fa043fc9740 1 -- 172.21.15.98:0/1080537356 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({fsmap=35}) -- 0x56348158b2d0 con 0x563481509520 2024-05-09T08:56:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.236+0000 7fa01eff5640 1 -- 172.21.15.98:0/1080537356 <== mon.2 v2:172.21.15.105:3300/0 4 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fa0300af390 con 0x563481509520 2024-05-09T08:56:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.236+0000 7fa043fc9740 10 client.26444 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.236+0000 7fa043fc9740 10 client.26444 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:56:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.236+0000 7fa043fc9740 20 client.26444 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.236+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1080537356 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56348158b8b0 0x56348158dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.236+0000 7fa043fc9740 4 client.26444 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:56:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.236+0000 7fa043fc9740 1 -- 172.21.15.98:0/1080537356 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5634814766c0 con 0x56348158b8b0 2024-05-09T08:56:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.236+0000 7fa03a7fc640 1 --2- 172.21.15.98:0/1080537356 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56348158b8b0 0x56348158dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.237+0000 7fa03a7fc640 1 --2- 172.21.15.98:0/1080537356 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56348158b8b0 0x56348158dd40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=86002c2252a47cd2 server_cookie=dcc34f24c1f842a2 in_seq=0 out_seq=0 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa01eff5640 10 client.26444 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa01eff5640 1 -- 172.21.15.98:0/1080537356 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5634814766c0 con 0x56348158b8b0 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa01eff5640 10 client.26444 handle_command_reply: tid=0 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa043fc9740 1 client.26444 shutdown 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa043fc9740 1 -- 172.21.15.98:0/1080537356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481509520 msgr2=0x563481515610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1080537356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481509520 0x563481515610 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa030097e30 tx=0x7fa03009aae0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa043fc9740 1 -- 172.21.15.98:0/1080537356 shutdown_connections 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1080537356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563481514510 0x5634815160d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1080537356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563481509e40 0x563481515b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1080537356 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56348158b8b0 0x56348158dd40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa01eff5640 0 client.26444 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa043fc9740 1 --2- 172.21.15.98:0/1080537356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563481509520 0x563481515610 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa043fc9740 1 -- 172.21.15.98:0/1080537356 >> 172.21.15.98:0/1080537356 conn(0x563481505840 msgr2=0x563481513210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa043fc9740 1 -- 172.21.15.98:0/1080537356 shutdown_connections 2024-05-09T08:56:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.238+0000 7fa043fc9740 1 -- 172.21.15.98:0/1080537356 wait complete. 2024-05-09T08:56:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.239+0000 7fa043fc9740 20 client.26444 trim_cache size 0 max 16384 2024-05-09T08:56:13.241 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:56:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.242+0000 7fa03bfff640 1 -- 172.21.15.98:0/1328339992 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa0240a2dc0 msgr2=0x7fa0240a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.242+0000 7fa03bfff640 1 --2- 172.21.15.98:0/1328339992 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa0240a2dc0 0x7fa0240a5280 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa03404b000 tx=0x7fa034048000 comp rx=0 tx=0).stop 2024-05-09T08:56:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.242+0000 7fa03bfff640 1 -- 172.21.15.98:0/1328339992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa03c06d1c0 msgr2=0x7fa03c11a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.242+0000 7fa03bfff640 1 --2- 172.21.15.98:0/1328339992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa03c06d1c0 0x7fa03c11a2f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa03000d770 tx=0x7fa03000dc30 comp rx=0 tx=0).stop 2024-05-09T08:56:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.243+0000 7fa03bfff640 1 -- 172.21.15.98:0/1328339992 shutdown_connections 2024-05-09T08:56:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.243+0000 7fa03bfff640 1 --2- 172.21.15.98:0/1328339992 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa0240a2dc0 0x7fa0240a5280 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.244+0000 7fa03bfff640 1 --2- 172.21.15.98:0/1328339992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa03c11ad50 0x7fa03c1138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.244+0000 7fa03bfff640 1 --2- 172.21.15.98:0/1328339992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa03c06dae0 0x7fa03c113370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.244+0000 7fa03bfff640 1 --2- 172.21.15.98:0/1328339992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa03c06d1c0 0x7fa03c11a2f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.244+0000 7fa03bfff640 1 -- 172.21.15.98:0/1328339992 >> 172.21.15.98:0/1328339992 conn(0x7fa03c06c6c0 msgr2=0x7fa03c11bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.246+0000 7fa03bfff640 1 -- 172.21.15.98:0/1328339992 shutdown_connections 2024-05-09T08:56:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.246+0000 7fa03bfff640 1 -- 172.21.15.98:0/1328339992 wait complete. 2024-05-09T08:56:13.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:56:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.494+0000 7f522caba640 1 -- 172.21.15.98:0/2562527668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f522810d130 msgr2=0x7f5228114d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.494+0000 7f522caba640 1 --2- 172.21.15.98:0/2562527668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f522810d130 0x7f5228114d00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5220009ca0 tx=0x7f522002a8d0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.495+0000 7f522caba640 1 -- 172.21.15.98:0/2562527668 shutdown_connections 2024-05-09T08:56:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.495+0000 7f522caba640 1 --2- 172.21.15.98:0/2562527668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f522810d130 0x7f5228114d00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.495+0000 7f522caba640 1 --2- 172.21.15.98:0/2562527668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5228108a60 0x7f522810ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.495+0000 7f522caba640 1 --2- 172.21.15.98:0/2562527668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52281080b0 0x7f5228108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.495+0000 7f522caba640 1 -- 172.21.15.98:0/2562527668 >> 172.21.15.98:0/2562527668 conn(0x7f5228075d70 msgr2=0x7f5228078190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.495+0000 7f522caba640 1 -- 172.21.15.98:0/2562527668 shutdown_connections 2024-05-09T08:56:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.495+0000 7f522caba640 1 -- 172.21.15.98:0/2562527668 wait complete. 2024-05-09T08:56:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.496+0000 7f522caba640 1 Processor -- start 2024-05-09T08:56:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.496+0000 7f522caba640 1 -- start start 2024-05-09T08:56:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.497+0000 7f522caba640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52281080b0 0x7f52281a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.497+0000 7f522caba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522810d130 0x7f52281a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.497+0000 7f522caba640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f522819e660 0x7f522819eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.498+0000 7f522659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52281080b0 0x7f52281a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.498+0000 7f522659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52281080b0 0x7f52281a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39562/0 (socket says 172.21.15.98:39562) 2024-05-09T08:56:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.498+0000 7f522659c640 1 -- 172.21.15.98:0/3606044244 learned_addr learned my addr 172.21.15.98:0/3606044244 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.498+0000 7f521dd9b640 1 --2- 172.21.15.98:0/3606044244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522810d130 0x7f52281a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.498+0000 7f5226d9d640 1 --2- 172.21.15.98:0/3606044244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f522819e660 0x7f522819eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.498+0000 7f522caba640 1 -- 172.21.15.98:0/3606044244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52281182c0 con 0x7f522810d130 2024-05-09T08:56:13.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.498+0000 7f522caba640 1 -- 172.21.15.98:0/3606044244 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5228118140 con 0x7f522819e660 2024-05-09T08:56:13.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.498+0000 7f522caba640 1 -- 172.21.15.98:0/3606044244 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5228118440 con 0x7f52281080b0 2024-05-09T08:56:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.499+0000 7f521dd9b640 1 -- 172.21.15.98:0/3606044244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52281080b0 msgr2=0x7f52281a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.499+0000 7f521dd9b640 1 --2- 172.21.15.98:0/3606044244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52281080b0 0x7f52281a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.499+0000 7f521dd9b640 1 -- 172.21.15.98:0/3606044244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f522819e660 msgr2=0x7f522819eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.499+0000 7f521dd9b640 1 --2- 172.21.15.98:0/3606044244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f522819e660 0x7f522819eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.499+0000 7f521dd9b640 1 -- 172.21.15.98:0/3606044244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522819f260 con 0x7f522810d130 2024-05-09T08:56:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.499+0000 7f521dd9b640 1 --2- 172.21.15.98:0/3606044244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522810d130 0x7f52281a4ae0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5210009fd0 tx=0x7f521000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.500+0000 7f521ffff640 1 -- 172.21.15.98:0/3606044244 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5210015020 con 0x7f522810d130 2024-05-09T08:56:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.500+0000 7f521ffff640 1 -- 172.21.15.98:0/3606044244 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5210011d10 con 0x7f522810d130 2024-05-09T08:56:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.500+0000 7f522caba640 1 -- 172.21.15.98:0/3606044244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52281ab340 con 0x7f522810d130 2024-05-09T08:56:13.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.501+0000 7f521ffff640 1 -- 172.21.15.98:0/3606044244 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521000f2e0 con 0x7f522810d130 2024-05-09T08:56:13.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.501+0000 7f522caba640 1 -- 172.21.15.98:0/3606044244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52281ab740 con 0x7f522810d130 2024-05-09T08:56:13.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.506+0000 7f522f7c8740 1 Processor -- start 2024-05-09T08:56:13.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.510+0000 7f521ffff640 1 -- 172.21.15.98:0/3606044244 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5210018020 con 0x7f522810d130 2024-05-09T08:56:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.510+0000 7f522f7c8740 1 -- start start 2024-05-09T08:56:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f522f7c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c076ec710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f522f7c8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076ecc50 0x556c076f6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f522f7c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076f7320 0x556c076f9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f522f7c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556c07651c40 con 0x556c076f7320 2024-05-09T08:56:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f522f7c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556c075f9580 con 0x556c076ec330 2024-05-09T08:56:13.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f522f7c8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556c0764dc00 con 0x556c076ecc50 2024-05-09T08:56:13.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f5226d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c076ec710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f522659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076ecc50 0x556c076f6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f5226d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c076ec710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52504/0 (socket says 172.21.15.98:52504) 2024-05-09T08:56:13.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f5226d9d640 1 -- 172.21.15.98:0/1834881404 learned_addr learned my addr 172.21.15.98:0/1834881404 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f521dd9b640 1 --2- 172.21.15.98:0/1834881404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076f7320 0x556c076f9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f5226d9d640 1 -- 172.21.15.98:0/1834881404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076ecc50 msgr2=0x556c076f6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f5226d9d640 1 --2- 172.21.15.98:0/1834881404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076ecc50 0x556c076f6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f5226d9d640 1 -- 172.21.15.98:0/1834881404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076f7320 msgr2=0x556c076f9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f5226d9d640 1 --2- 172.21.15.98:0/1834881404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076f7320 0x556c076f9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.511+0000 7f5226d9d640 1 -- 172.21.15.98:0/1834881404 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c073653f0 con 0x556c076ec330 2024-05-09T08:56:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.512+0000 7f521dd9b640 1 --2- 172.21.15.98:0/1834881404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076f7320 0x556c076f9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:56:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.512+0000 7f521ffff640 1 --2- 172.21.15.98:0/3606044244 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f52140a2c60 0x7f52140a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.512+0000 7f5226d9d640 1 --2- 172.21.15.98:0/1834881404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c076ec710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f522002f3d0 tx=0x7f522002f400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b8e1778ad17187d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.512+0000 7f521ffff640 1 -- 172.21.15.98:0/3606044244 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f52100d2060 con 0x7f522810d130 2024-05-09T08:56:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.513+0000 7f521cd99640 1 -- 172.21.15.98:0/1834881404 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5220031d80 con 0x556c076ec330 2024-05-09T08:56:13.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.513+0000 7f521cd99640 1 -- 172.21.15.98:0/1834881404 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5220036030 con 0x556c076ec330 2024-05-09T08:56:13.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.513+0000 7f521cd99640 1 -- 172.21.15.98:0/1834881404 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5220004370 con 0x556c076ec330 2024-05-09T08:56:13.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.513+0000 7f522659c640 1 --2- 172.21.15.98:0/3606044244 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f52140a2c60 0x7f52140a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.514+0000 7f522f7c8740 1 -- 172.21.15.98:0/1834881404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 msgr2=0x556c076ec710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.514+0000 7f522f7c8740 1 --2- 172.21.15.98:0/1834881404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c076ec710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f522002f3d0 tx=0x7f522002f400 comp rx=0 tx=0).stop 2024-05-09T08:56:13.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.515+0000 7f522f7c8740 1 -- 172.21.15.98:0/1834881404 shutdown_connections 2024-05-09T08:56:13.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.515+0000 7f522f7c8740 1 --2- 172.21.15.98:0/1834881404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076f7320 0x556c076f9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.515+0000 7f522f7c8740 1 --2- 172.21.15.98:0/1834881404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076ecc50 0x556c076f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.515+0000 7f522f7c8740 1 --2- 172.21.15.98:0/1834881404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c076ec710 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.515+0000 7f522f7c8740 1 -- 172.21.15.98:0/1834881404 >> 172.21.15.98:0/1834881404 conn(0x556c076e8650 msgr2=0x556c076e8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:13.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.515+0000 7f522f7c8740 1 -- 172.21.15.98:0/1834881404 shutdown_connections 2024-05-09T08:56:13.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.517+0000 7f522659c640 1 --2- 172.21.15.98:0/3606044244 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f52140a2c60 0x7f52140a5120 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x556c076ed8f0 tx=0x7f5218008040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:13.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.518+0000 7f522f7c8740 1 -- 172.21.15.98:0/1834881404 wait complete. 2024-05-09T08:56:13.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.518+0000 7f522f7c8740 1 -- start start 2024-05-09T08:56:13.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.518+0000 7f522f7c8740 1 Processor -- start 2024-05-09T08:56:13.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.519+0000 7f522f7c8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076ec330 0x556c076f6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.519+0000 7f522f7c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 0x556c076f4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.519+0000 7f522f7c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076f7320 0x556c076f48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.519+0000 7f522f7c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556c076fa100 con 0x556c076ecc50 2024-05-09T08:56:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.519+0000 7f522f7c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556c0760cc40 con 0x556c076f7320 2024-05-09T08:56:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.519+0000 7f522f7c8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556c076fa280 con 0x556c076ec330 2024-05-09T08:56:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.519+0000 7f522f7c8740 10 client.0 _ll_register_callbacks cb 0x556c07364be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.519+0000 7f522659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076ec330 0x556c076f6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.519+0000 7f522659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076ec330 0x556c076f6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39590/0 (socket says 172.21.15.98:39590) 2024-05-09T08:56:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.519+0000 7f522659c640 1 -- 172.21.15.98:0/2512250231 learned_addr learned my addr 172.21.15.98:0/2512250231 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.519+0000 7f5226d9d640 1 --2- 172.21.15.98:0/2512250231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076f7320 0x556c076f48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f521dd9b640 1 --2- 172.21.15.98:0/2512250231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 0x556c076f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f5226d9d640 1 -- 172.21.15.98:0/2512250231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076ec330 msgr2=0x556c076f6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f5226d9d640 1 --2- 172.21.15.98:0/2512250231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076ec330 0x556c076f6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f5226d9d640 1 -- 172.21.15.98:0/2512250231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 msgr2=0x556c076f4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f5226d9d640 1 --2- 172.21.15.98:0/2512250231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 0x556c076f4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f5226d9d640 1 -- 172.21.15.98:0/2512250231 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c0742f7d0 con 0x556c076f7320 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f522659c640 1 --2- 172.21.15.98:0/2512250231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076ec330 0x556c076f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f5226d9d640 1 --2- 172.21.15.98:0/2512250231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076f7320 0x556c076f48b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5220009ca0 tx=0x7f5220004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6fcea25b0f09e69 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f520dffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f520dffb640 1 -- 172.21.15.98:0/2512250231 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5220003c40 con 0x556c076f7320 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f522f7c8740 1 -- 172.21.15.98:0/2512250231 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5220036030 con 0x556c076f7320 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f520dffb640 1 -- 172.21.15.98:0/2512250231 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f522004a040 con 0x556c076f7320 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f520dffb640 1 -- 172.21.15.98:0/2512250231 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f522005c050 con 0x556c076f7320 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f522f7c8740 10 client.26453 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.520+0000 7f522f7c8740 1 -- 172.21.15.98:0/2512250231 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556c076fa100 con 0x556c076f7320 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.521+0000 7f520dffb640 1 -- 172.21.15.98:0/2512250231 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5220036030 con 0x556c076f7320 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.521+0000 7f520dffb640 1 -- 172.21.15.98:0/2512250231 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f522004e070 con 0x556c076f7320 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.521+0000 7f522f7c8740 10 client.26453 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.521+0000 7f522f7c8740 10 client.26453 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.521+0000 7f522f7c8740 20 client.26453 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.521+0000 7f522f7c8740 1 --2- 172.21.15.98:0/2512250231 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556c077184b0 0x556c0771a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.521+0000 7f522f7c8740 4 client.26453 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.521+0000 7f522f7c8740 1 -- 172.21.15.98:0/2512250231 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c0765f370 con 0x556c077184b0 2024-05-09T08:56:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.521+0000 7f522659c640 1 --2- 172.21.15.98:0/2512250231 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556c077184b0 0x556c0771a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.522+0000 7f522659c640 1 --2- 172.21.15.98:0/2512250231 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556c077184b0 0x556c0771a900 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed566b6153f1b445 server_cookie=e606615bc1adb508 in_seq=0 out_seq=0 2024-05-09T08:56:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.522+0000 7f520dffb640 10 client.26453 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.523+0000 7f520dffb640 1 -- 172.21.15.98:0/2512250231 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556c0765f370 con 0x556c077184b0 2024-05-09T08:56:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.523+0000 7f520dffb640 10 client.26453 handle_command_reply: tid=0 2024-05-09T08:56:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.523+0000 7f522f7c8740 1 client.26453 shutdown 2024-05-09T08:56:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.523+0000 7f522f7c8740 1 -- 172.21.15.98:0/2512250231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076f7320 msgr2=0x556c076f48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.523+0000 7f522f7c8740 1 --2- 172.21.15.98:0/2512250231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076f7320 0x556c076f48b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5220009ca0 tx=0x7f5220004a40 comp rx=0 tx=0).stop 2024-05-09T08:56:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.523+0000 7f522f7c8740 1 -- 172.21.15.98:0/2512250231 shutdown_connections 2024-05-09T08:56:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.523+0000 7f522f7c8740 1 --2- 172.21.15.98:0/2512250231 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556c077184b0 0x556c0771a900 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.523+0000 7f522f7c8740 1 --2- 172.21.15.98:0/2512250231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076f7320 0x556c076f48b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.523+0000 7f522f7c8740 1 --2- 172.21.15.98:0/2512250231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 0x556c076f4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.523+0000 7f522f7c8740 1 --2- 172.21.15.98:0/2512250231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076ec330 0x556c076f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.523+0000 7f522f7c8740 1 -- 172.21.15.98:0/2512250231 >> 172.21.15.98:0/2512250231 conn(0x556c076e8650 msgr2=0x556c076ee450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.524+0000 7f520dffb640 0 client.26453 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:13.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.526+0000 7f522f7c8740 1 -- 172.21.15.98:0/2512250231 shutdown_connections 2024-05-09T08:56:13.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.526+0000 7f522f7c8740 1 -- 172.21.15.98:0/2512250231 wait complete. 2024-05-09T08:56:13.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.526+0000 7f522f7c8740 20 client.26453 trim_cache size 0 max 16384 2024-05-09T08:56:13.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.558+0000 7f522f7c8740 1 Processor -- start 2024-05-09T08:56:13.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.558+0000 7f522f7c8740 1 -- start start 2024-05-09T08:56:13.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.558+0000 7f522f7c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c07702fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.558+0000 7f522f7c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 0x556c076e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.558+0000 7f522f7c8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076f7320 0x556c077201a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.558+0000 7f522f7c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52281ac8d0 con 0x556c076ecc50 2024-05-09T08:56:13.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.558+0000 7f522f7c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f52281ae410 con 0x556c076ec330 2024-05-09T08:56:13.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.558+0000 7f522f7c8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f52281acfa0 con 0x556c076f7320 2024-05-09T08:56:13.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.559+0000 7f5226d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076f7320 0x556c077201a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.559+0000 7f521dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 0x556c076e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.559+0000 7f521dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 0x556c076e8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39646/0 (socket says 172.21.15.98:39646) 2024-05-09T08:56:13.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.559+0000 7f521dd9b640 1 -- 172.21.15.98:0/3641422996 learned_addr learned my addr 172.21.15.98:0/3641422996 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:13.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.559+0000 7f522659c640 1 --2- 172.21.15.98:0/3641422996 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c07702fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.559+0000 7f521dd9b640 1 -- 172.21.15.98:0/3641422996 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076f7320 msgr2=0x556c077201a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.559+0000 7f521dd9b640 1 --2- 172.21.15.98:0/3641422996 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076f7320 0x556c077201a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.559+0000 7f521dd9b640 1 -- 172.21.15.98:0/3641422996 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 msgr2=0x556c07702fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.559+0000 7f521dd9b640 1 --2- 172.21.15.98:0/3641422996 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c07702fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.559+0000 7f521dd9b640 1 -- 172.21.15.98:0/3641422996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c076e3f30 con 0x556c076ecc50 2024-05-09T08:56:13.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.561+0000 7f521dd9b640 1 --2- 172.21.15.98:0/3641422996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 0x556c076e8a30 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f521009a3d0 tx=0x7f5210099f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85260aa9f4766743 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:13.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.562+0000 7f520ffff640 1 -- 172.21.15.98:0/3641422996 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521009a440 con 0x556c076ecc50 2024-05-09T08:56:13.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.562+0000 7f520ffff640 1 -- 172.21.15.98:0/3641422996 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52100d1020 con 0x556c076ecc50 2024-05-09T08:56:13.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.563+0000 7f522f7c8740 1 -- 172.21.15.98:0/3641422996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 msgr2=0x556c076e8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.563+0000 7f522f7c8740 1 --2- 172.21.15.98:0/3641422996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 0x556c076e8a30 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f521009a3d0 tx=0x7f5210099f80 comp rx=0 tx=0).stop 2024-05-09T08:56:13.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.564+0000 7f522f7c8740 1 -- 172.21.15.98:0/3641422996 shutdown_connections 2024-05-09T08:56:13.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.564+0000 7f522f7c8740 1 --2- 172.21.15.98:0/3641422996 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076f7320 0x556c077201a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.564+0000 7f522f7c8740 1 --2- 172.21.15.98:0/3641422996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 0x556c076e8a30 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.564+0000 7f522f7c8740 1 --2- 172.21.15.98:0/3641422996 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c07702fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.564+0000 7f522f7c8740 1 -- 172.21.15.98:0/3641422996 >> 172.21.15.98:0/3641422996 conn(0x556c076e8650 msgr2=0x556c076e3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:13.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.564+0000 7f522f7c8740 1 -- 172.21.15.98:0/3641422996 shutdown_connections 2024-05-09T08:56:13.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.564+0000 7f522f7c8740 1 -- 172.21.15.98:0/3641422996 wait complete. 2024-05-09T08:56:13.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.565+0000 7f522f7c8740 1 -- start start 2024-05-09T08:56:13.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.565+0000 7f522f7c8740 1 Processor -- start 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.565+0000 7f522f7c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c0777ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.565+0000 7f522f7c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 0x556c0777bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.565+0000 7f522f7c8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076f7320 0x556c0777c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.565+0000 7f522f7c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52281add40 con 0x556c076ecc50 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.565+0000 7f522f7c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f52281ad670 con 0x556c076ec330 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.565+0000 7f522f7c8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5228081770 con 0x556c076f7320 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.565+0000 7f522f7c8740 10 client.0 _ll_register_callbacks cb 0x7f5220031ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.566+0000 7f522659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c0777ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.566+0000 7f522659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c0777ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52542/0 (socket says 172.21.15.98:52542) 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.566+0000 7f522659c640 1 -- 172.21.15.98:0/254609751 learned_addr learned my addr 172.21.15.98:0/254609751 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.566+0000 7f522659c640 1 -- 172.21.15.98:0/254609751 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076f7320 msgr2=0x556c0777c490 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.566+0000 7f522659c640 1 --2- 172.21.15.98:0/254609751 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076f7320 0x556c0777c490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.566+0000 7f522659c640 1 -- 172.21.15.98:0/254609751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 msgr2=0x556c0777bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.566+0000 7f522659c640 1 --2- 172.21.15.98:0/254609751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 0x556c0777bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.566+0000 7f522659c640 1 -- 172.21.15.98:0/254609751 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c076ff690 con 0x556c076ec330 2024-05-09T08:56:13.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.567+0000 7f522659c640 1 --2- 172.21.15.98:0/254609751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c0777ba10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5218016780 tx=0x7f52180047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e731e7015dc7a09 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:13.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.570+0000 7f520dffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:56:13.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.570+0000 7f520dffb640 1 -- 172.21.15.98:0/254609751 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521801c040 con 0x556c076ec330 2024-05-09T08:56:13.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.570+0000 7f520dffb640 1 -- 172.21.15.98:0/254609751 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f5218023930 con 0x556c076ec330 2024-05-09T08:56:13.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.570+0000 7f522f7c8740 1 -- 172.21.15.98:0/254609751 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52100d1020 con 0x556c076ec330 2024-05-09T08:56:13.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.571+0000 7f520dffb640 1 -- 172.21.15.98:0/254609751 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f521803b050 con 0x556c076ec330 2024-05-09T08:56:13.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.571+0000 7f522f7c8740 10 client.26459 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:13.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.571+0000 7f522f7c8740 1 -- 172.21.15.98:0/254609751 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556c076f0ee0 con 0x556c076ec330 2024-05-09T08:56:13.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.571+0000 7f520dffb640 1 -- 172.21.15.98:0/254609751 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52100d1020 con 0x556c076ec330 2024-05-09T08:56:13.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.574+0000 7f520dffb640 1 -- 172.21.15.98:0/254609751 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f5218014c70 con 0x556c076ec330 2024-05-09T08:56:13.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.574+0000 7f522f7c8740 10 client.26459 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:13.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.574+0000 7f522f7c8740 10 client.26459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:13.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.574+0000 7f522f7c8740 20 client.26459 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:13.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.574+0000 7f522f7c8740 1 --2- 172.21.15.98:0/254609751 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556c0778fae0 0x556c07791f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:13.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.574+0000 7f522f7c8740 4 client.26459 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:56:13.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.574+0000 7f522f7c8740 1 -- 172.21.15.98:0/254609751 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c076596c0 con 0x556c0778fae0 2024-05-09T08:56:13.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.574+0000 7f521dd9b640 1 --2- 172.21.15.98:0/254609751 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556c0778fae0 0x556c07791f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:13.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.575+0000 7f521dd9b640 1 --2- 172.21.15.98:0/254609751 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556c0778fae0 0x556c07791f30 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=430db74266bf0a74 server_cookie=6efb93fd517539dd in_seq=0 out_seq=0 2024-05-09T08:56:13.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.575+0000 7f520dffb640 10 client.26459 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:13.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.575+0000 7f520dffb640 1 -- 172.21.15.98:0/254609751 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c076596c0 con 0x556c0778fae0 2024-05-09T08:56:13.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.575+0000 7f520dffb640 10 client.26459 handle_command_reply: tid=0 2024-05-09T08:56:13.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.576+0000 7f522f7c8740 1 client.26459 shutdown 2024-05-09T08:56:13.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.576+0000 7f522f7c8740 1 -- 172.21.15.98:0/254609751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 msgr2=0x556c0777ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.576+0000 7f522f7c8740 1 --2- 172.21.15.98:0/254609751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c0777ba10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5218016780 tx=0x7f52180047d0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.577+0000 7f522f7c8740 1 -- 172.21.15.98:0/254609751 shutdown_connections 2024-05-09T08:56:13.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.577+0000 7f522f7c8740 1 --2- 172.21.15.98:0/254609751 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556c0778fae0 0x556c07791f30 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.577+0000 7f522f7c8740 1 --2- 172.21.15.98:0/254609751 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556c076f7320 0x556c0777c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.577+0000 7f522f7c8740 1 --2- 172.21.15.98:0/254609751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556c076ecc50 0x556c0777bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.577+0000 7f522f7c8740 1 --2- 172.21.15.98:0/254609751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c076ec330 0x556c0777ba10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.577+0000 7f522f7c8740 1 -- 172.21.15.98:0/254609751 >> 172.21.15.98:0/254609751 conn(0x556c076e8650 msgr2=0x556c0771a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:13.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.578+0000 7f520dffb640 0 client.26459 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:13.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.579+0000 7f522f7c8740 1 -- 172.21.15.98:0/254609751 shutdown_connections 2024-05-09T08:56:13.582 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:56:13.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.579+0000 7f522f7c8740 1 -- 172.21.15.98:0/254609751 wait complete. 2024-05-09T08:56:13.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.580+0000 7f522f7c8740 20 client.26459 trim_cache size 0 max 16384 2024-05-09T08:56:13.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.583+0000 7f522caba640 1 -- 172.21.15.98:0/3606044244 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f52140a2c60 msgr2=0x7f52140a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.583+0000 7f522caba640 1 --2- 172.21.15.98:0/3606044244 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f52140a2c60 0x7f52140a5120 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x556c076ed8f0 tx=0x7f5218008040 comp rx=0 tx=0).stop 2024-05-09T08:56:13.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.583+0000 7f522caba640 1 -- 172.21.15.98:0/3606044244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522810d130 msgr2=0x7f52281a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:13.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.583+0000 7f522caba640 1 --2- 172.21.15.98:0/3606044244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522810d130 0x7f52281a4ae0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5210009fd0 tx=0x7f521000b870 comp rx=0 tx=0).stop 2024-05-09T08:56:13.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.583+0000 7f522caba640 1 -- 172.21.15.98:0/3606044244 shutdown_connections 2024-05-09T08:56:13.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.583+0000 7f522caba640 1 --2- 172.21.15.98:0/3606044244 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f52140a2c60 0x7f52140a5120 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.583+0000 7f522caba640 1 --2- 172.21.15.98:0/3606044244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f522819e660 0x7f522819eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.583+0000 7f522caba640 1 --2- 172.21.15.98:0/3606044244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f522810d130 0x7f52281a4ae0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.583+0000 7f522caba640 1 --2- 172.21.15.98:0/3606044244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52281080b0 0x7f52281a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:13.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.583+0000 7f522caba640 1 -- 172.21.15.98:0/3606044244 >> 172.21.15.98:0/3606044244 conn(0x7f5228075d70 msgr2=0x7f52280771c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:13.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.587+0000 7f522caba640 1 -- 172.21.15.98:0/3606044244 shutdown_connections 2024-05-09T08:56:13.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:13.587+0000 7f522caba640 1 -- 172.21.15.98:0/3606044244 wait complete. 2024-05-09T08:56:14.616 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:56:14.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:56:14.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:56:14.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:14 smithi098 ceph-mon[30092]: pgmap v172: 129 pgs: 129 active+clean; 43 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 634 MiB/s wr, 1.88k op/s 2024-05-09T08:56:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.900+0000 7fcabac2c640 1 -- 172.21.15.98:0/3088113624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab4100700 msgr2=0x7fcab4100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.900+0000 7fcabac2c640 1 --2- 172.21.15.98:0/3088113624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab4100700 0x7fcab4100b80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0009900 tx=0x7fcaa002a410 comp rx=0 tx=0).stop 2024-05-09T08:56:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.901+0000 7fcabac2c640 1 -- 172.21.15.98:0/3088113624 shutdown_connections 2024-05-09T08:56:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.901+0000 7fcabac2c640 1 --2- 172.21.15.98:0/3088113624 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcab41010c0 0x7fcab410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.901+0000 7fcabac2c640 1 --2- 172.21.15.98:0/3088113624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab4100700 0x7fcab4100b80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.901+0000 7fcabac2c640 1 --2- 172.21.15.98:0/3088113624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab4106710 0x7fcab4106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.901+0000 7fcabac2c640 1 -- 172.21.15.98:0/3088113624 >> 172.21.15.98:0/3088113624 conn(0x7fcab4077830 msgr2=0x7fcab4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:14.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.902+0000 7fcabac2c640 1 -- 172.21.15.98:0/3088113624 shutdown_connections 2024-05-09T08:56:14.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.902+0000 7fcabac2c640 1 -- 172.21.15.98:0/3088113624 wait complete. 2024-05-09T08:56:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.903+0000 7fcabac2c640 1 Processor -- start 2024-05-09T08:56:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.904+0000 7fcabac2c640 1 -- start start 2024-05-09T08:56:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.904+0000 7fcabac2c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab4100700 0x7fcab4072da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.904+0000 7fcabac2c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab41010c0 0x7fcab40732e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.904+0000 7fcabac2c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcab4106710 0x7fcab406ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.904+0000 7fcabac2c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcab406b820 con 0x7fcab4100700 2024-05-09T08:56:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.904+0000 7fcabac2c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcab406b6a0 con 0x7fcab41010c0 2024-05-09T08:56:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.904+0000 7fcabac2c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcab406b9a0 con 0x7fcab4106710 2024-05-09T08:56:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.904+0000 7fcab91c9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcab4106710 0x7fcab406ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.904+0000 7fcab91c9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcab4106710 0x7fcab406ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39630/0 (socket says 172.21.15.98:39630) 2024-05-09T08:56:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.904+0000 7fcab91c9640 1 -- 172.21.15.98:0/2171587814 learned_addr learned my addr 172.21.15.98:0/2171587814 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.904+0000 7fcab3fff640 1 --2- 172.21.15.98:0/2171587814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab41010c0 0x7fcab40732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.905+0000 7fcab91c9640 1 -- 172.21.15.98:0/2171587814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab41010c0 msgr2=0x7fcab40732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.905+0000 7fcab91c9640 1 --2- 172.21.15.98:0/2171587814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab41010c0 0x7fcab40732e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.905+0000 7fcab91c9640 1 -- 172.21.15.98:0/2171587814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab4100700 msgr2=0x7fcab4072da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.905+0000 7fcab91c9640 1 --2- 172.21.15.98:0/2171587814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab4100700 0x7fcab4072da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.905+0000 7fcab91c9640 1 -- 172.21.15.98:0/2171587814 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab406d720 con 0x7fcab4106710 2024-05-09T08:56:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.905+0000 7fcab91c9640 1 --2- 172.21.15.98:0/2171587814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcab4106710 0x7fcab406ce60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00efd0 tx=0x7fcaac009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.906+0000 7fcab1ffb640 1 -- 172.21.15.98:0/2171587814 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac01c070 con 0x7fcab4106710 2024-05-09T08:56:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.906+0000 7fcab1ffb640 1 -- 172.21.15.98:0/2171587814 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaac003b70 con 0x7fcab4106710 2024-05-09T08:56:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.906+0000 7fcab1ffb640 1 -- 172.21.15.98:0/2171587814 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac018860 con 0x7fcab4106710 2024-05-09T08:56:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.906+0000 7fcabac2c640 1 -- 172.21.15.98:0/2171587814 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcab406da10 con 0x7fcab4106710 2024-05-09T08:56:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.907+0000 7fcabac2c640 1 -- 172.21.15.98:0/2171587814 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcab41ab3e0 con 0x7fcab4106710 2024-05-09T08:56:14.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:14 smithi142 ceph-mon[34625]: pgmap v172: 129 pgs: 129 active+clean; 43 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 634 MiB/s wr, 1.88k op/s 2024-05-09T08:56:14.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.911+0000 7fcabda0f740 1 Processor -- start 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.911+0000 7fcab1ffb640 1 -- 172.21.15.98:0/2171587814 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcaac017070 con 0x7fcab4106710 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.911+0000 7fcabda0f740 1 -- start start 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.911+0000 7fcabda0f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0c80b0 0x56158e0c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.911+0000 7fcabda0f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 0x56158e0c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcabda0f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0d1740 0x56158e0d3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcabda0f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56158e02b990 con 0x56158e0d1740 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcabda0f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56158dfd35b0 con 0x56158e0c80b0 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcabda0f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56158e027950 con 0x56158e0c69c0 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcab3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0c80b0 0x56158e0c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcab3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0c80b0 0x56158e0c63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52564/0 (socket says 172.21.15.98:52564) 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcab3fff640 1 -- 172.21.15.98:0/1304039945 learned_addr learned my addr 172.21.15.98:0/1304039945 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcab89c8640 1 --2- 172.21.15.98:0/1304039945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 0x56158e0c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcab3fff640 1 --2- 172.21.15.98:0/1304039945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0d1740 0x56158e0d3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcab89c8640 1 -- 172.21.15.98:0/1304039945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0c80b0 msgr2=0x56158e0c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcab89c8640 1 --2- 172.21.15.98:0/1304039945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0c80b0 0x56158e0c63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcab89c8640 1 -- 172.21.15.98:0/1304039945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0d1740 msgr2=0x56158e0d3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:14.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcab89c8640 1 --2- 172.21.15.98:0/1304039945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0d1740 0x56158e0d3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcab89c8640 1 -- 172.21.15.98:0/1304039945 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56158de077e0 con 0x56158e0c69c0 2024-05-09T08:56:14.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.912+0000 7fcab89c8640 1 --2- 172.21.15.98:0/1304039945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 0x56158e0c6e40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009ac0 tx=0x7fcaa8009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79587c455b2d414e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:14.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.913+0000 7fcab1ffb640 1 --2- 172.21.15.98:0/2171587814 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcaa40a2e50 0x7fcaa40a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.913+0000 7fca9effd640 1 -- 172.21.15.98:0/1304039945 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa800ed10 con 0x56158e0c69c0 2024-05-09T08:56:14.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.913+0000 7fcab1ffb640 1 -- 172.21.15.98:0/2171587814 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fcaac0d3620 con 0x7fcab4106710 2024-05-09T08:56:14.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.913+0000 7fca9effd640 1 -- 172.21.15.98:0/1304039945 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaa8005020 con 0x56158e0c69c0 2024-05-09T08:56:14.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.913+0000 7fca9effd640 1 -- 172.21.15.98:0/1304039945 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa8004c40 con 0x56158e0c69c0 2024-05-09T08:56:14.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.914+0000 7fcab3fff640 1 --2- 172.21.15.98:0/2171587814 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcaa40a2e50 0x7fcaa40a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.914+0000 7fcabda0f740 1 -- 172.21.15.98:0/1304039945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 msgr2=0x56158e0c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:14.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.914+0000 7fcabda0f740 1 --2- 172.21.15.98:0/1304039945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 0x56158e0c6e40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009ac0 tx=0x7fcaa8009f80 comp rx=0 tx=0).stop 2024-05-09T08:56:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.915+0000 7fcabda0f740 1 -- 172.21.15.98:0/1304039945 shutdown_connections 2024-05-09T08:56:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.915+0000 7fcabda0f740 1 --2- 172.21.15.98:0/1304039945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0d1740 0x56158e0d3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.915+0000 7fcabda0f740 1 --2- 172.21.15.98:0/1304039945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 0x56158e0c6e40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.915+0000 7fcabda0f740 1 --2- 172.21.15.98:0/1304039945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0c80b0 0x56158e0c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.915+0000 7fcabda0f740 1 -- 172.21.15.98:0/1304039945 >> 172.21.15.98:0/1304039945 conn(0x56158e08f0a0 msgr2=0x56158e0c2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.915+0000 7fcab3fff640 1 --2- 172.21.15.98:0/2171587814 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcaa40a2e50 0x7fcaa40a5310 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcaa00057c0 tx=0x7fcaa0005a20 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.915+0000 7fcabda0f740 1 -- 172.21.15.98:0/1304039945 shutdown_connections 2024-05-09T08:56:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.915+0000 7fcabda0f740 1 -- 172.21.15.98:0/1304039945 wait complete. 2024-05-09T08:56:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.916+0000 7fcabda0f740 1 -- start start 2024-05-09T08:56:14.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.916+0000 7fcabda0f740 1 Processor -- start 2024-05-09T08:56:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.918+0000 7fcabda0f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c69c0 0x56158e0ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.918+0000 7fcabda0f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c80b0 0x56158e0cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.918+0000 7fcabda0f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.919+0000 7fcabda0f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56158e0d45d0 con 0x56158e0c69c0 2024-05-09T08:56:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.919+0000 7fcab3fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c80b0 0x56158e0cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.919+0000 7fcabda0f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56158dfe6b40 con 0x56158e0d1740 2024-05-09T08:56:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.919+0000 7fcabda0f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56158e0d4750 con 0x56158e0c80b0 2024-05-09T08:56:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.919+0000 7fcab3fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c80b0 0x56158e0cec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39644/0 (socket says 172.21.15.98:39644) 2024-05-09T08:56:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.919+0000 7fcab3fff640 1 -- 172.21.15.98:0/2664737703 learned_addr learned my addr 172.21.15.98:0/2664737703 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.919+0000 7fcab91c9640 1 --2- 172.21.15.98:0/2664737703 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.919+0000 7fcab89c8640 1 --2- 172.21.15.98:0/2664737703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c69c0 0x56158e0ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.920+0000 7fcabda0f740 10 client.0 _ll_register_callbacks cb 0x56158dfdb0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.920+0000 7fcab3fff640 1 -- 172.21.15.98:0/2664737703 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 msgr2=0x56158e0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.920+0000 7fcab3fff640 1 --2- 172.21.15.98:0/2664737703 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.920+0000 7fcab3fff640 1 -- 172.21.15.98:0/2664737703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c69c0 msgr2=0x56158e0ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:14.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.920+0000 7fcab3fff640 1 --2- 172.21.15.98:0/2664737703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c69c0 0x56158e0ce6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.920+0000 7fcab3fff640 1 -- 172.21.15.98:0/2664737703 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56158de67f70 con 0x56158e0c80b0 2024-05-09T08:56:14.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.920+0000 7fcab3fff640 1 --2- 172.21.15.98:0/2664737703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c80b0 0x56158e0cec10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcaa002a930 tx=0x7fcaa0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8322e9e6c5f5f72 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:14.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.920+0000 7fca93fff640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:56:14.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.921+0000 7fca93fff640 1 -- 172.21.15.98:0/2664737703 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa0035040 con 0x56158e0c80b0 2024-05-09T08:56:14.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.921+0000 7fca93fff640 1 -- 172.21.15.98:0/2664737703 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fcaa00317c0 con 0x56158e0c80b0 2024-05-09T08:56:14.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.921+0000 7fcabda0f740 1 -- 172.21.15.98:0/2664737703 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcaa8005020 con 0x56158e0c80b0 2024-05-09T08:56:14.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.922+0000 7fca93fff640 1 -- 172.21.15.98:0/2664737703 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcaa00044d0 con 0x56158e0c80b0 2024-05-09T08:56:14.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.922+0000 7fcabda0f740 10 client.26443 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:14.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.922+0000 7fcabda0f740 1 -- 172.21.15.98:0/2664737703 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56158dcff170 con 0x56158e0c80b0 2024-05-09T08:56:14.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.922+0000 7fca93fff640 1 -- 172.21.15.98:0/2664737703 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaa003d450 con 0x56158e0c80b0 2024-05-09T08:56:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.922+0000 7fca93fff640 1 -- 172.21.15.98:0/2664737703 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fcaa003a040 con 0x56158e0c80b0 2024-05-09T08:56:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.922+0000 7fcabda0f740 10 client.26443 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.922+0000 7fcabda0f740 10 client.26443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.922+0000 7fcabda0f740 20 client.26443 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.923+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2664737703 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56158e0e18c0 0x56158e0e3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.923+0000 7fcabda0f740 4 client.26443 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.923+0000 7fcabda0f740 1 -- 172.21.15.98:0/2664737703 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcaa8005020 con 0x56158e0e18c0 2024-05-09T08:56:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.923+0000 7fcab89c8640 1 --2- 172.21.15.98:0/2664737703 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56158e0e18c0 0x56158e0e3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.924+0000 7fca93fff640 10 client.26443 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.924+0000 7fcab89c8640 1 --2- 172.21.15.98:0/2664737703 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56158e0e18c0 0x56158e0e3cb0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5014687d97976b4 server_cookie=3a8c648ac408f089 in_seq=0 out_seq=0 2024-05-09T08:56:14.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.925+0000 7fca93fff640 1 -- 172.21.15.98:0/2664737703 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fcaa8005020 con 0x56158e0e18c0 2024-05-09T08:56:14.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.925+0000 7fca93fff640 10 client.26443 handle_command_reply: tid=0 2024-05-09T08:56:14.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.925+0000 7fcabda0f740 1 client.26443 shutdown 2024-05-09T08:56:14.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.926+0000 7fcabda0f740 1 -- 172.21.15.98:0/2664737703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c80b0 msgr2=0x56158e0cec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:14.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.926+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2664737703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c80b0 0x56158e0cec10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcaa002a930 tx=0x7fcaa0009260 comp rx=0 tx=0).stop 2024-05-09T08:56:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.926+0000 7fcabda0f740 1 -- 172.21.15.98:0/2664737703 shutdown_connections 2024-05-09T08:56:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.926+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2664737703 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56158e0e18c0 0x56158e0e3cb0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.926+0000 7fca93fff640 0 client.26443 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.926+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2664737703 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.926+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2664737703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c80b0 0x56158e0cec10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.926+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2664737703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c69c0 0x56158e0ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.926+0000 7fcabda0f740 1 -- 172.21.15.98:0/2664737703 >> 172.21.15.98:0/2664737703 conn(0x56158e08f0a0 msgr2=0x56158e0c8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.926+0000 7fcabda0f740 1 -- 172.21.15.98:0/2664737703 shutdown_connections 2024-05-09T08:56:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.926+0000 7fcabda0f740 1 -- 172.21.15.98:0/2664737703 wait complete. 2024-05-09T08:56:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.926+0000 7fcabda0f740 20 client.26443 trim_cache size 0 max 16384 2024-05-09T08:56:14.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.959+0000 7fcabda0f740 1 Processor -- start 2024-05-09T08:56:14.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.960+0000 7fcabda0f740 1 -- start start 2024-05-09T08:56:14.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.960+0000 7fcabda0f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 0x56158e0dd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.960+0000 7fcabda0f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c80b0 0x56158e0d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcabda0f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0e9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcabda0f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcab41b5400 con 0x56158e0c80b0 2024-05-09T08:56:14.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcab3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c80b0 0x56158e0d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcabda0f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcab41b6f40 con 0x56158e0d1740 2024-05-09T08:56:14.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcabda0f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcab41b5ad0 con 0x56158e0c69c0 2024-05-09T08:56:14.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcab89c8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 0x56158e0dd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcab3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c80b0 0x56158e0d0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39716/0 (socket says 172.21.15.98:39716) 2024-05-09T08:56:14.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcab3fff640 1 -- 172.21.15.98:0/2722233203 learned_addr learned my addr 172.21.15.98:0/2722233203 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:14.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcab91c9640 1 --2- 172.21.15.98:0/2722233203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0e9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcab91c9640 1 -- 172.21.15.98:0/2722233203 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 msgr2=0x56158e0dd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:14.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcab91c9640 1 --2- 172.21.15.98:0/2722233203 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 0x56158e0dd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcab91c9640 1 -- 172.21.15.98:0/2722233203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c80b0 msgr2=0x56158e0d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:14.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcab91c9640 1 --2- 172.21.15.98:0/2722233203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c80b0 0x56158e0d0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcab91c9640 1 -- 172.21.15.98:0/2722233203 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56158e0cd680 con 0x56158e0d1740 2024-05-09T08:56:14.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcab3fff640 1 --2- 172.21.15.98:0/2722233203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c80b0 0x56158e0d0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:56:14.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.961+0000 7fcab91c9640 1 --2- 172.21.15.98:0/2722233203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0e9560 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcaac0d09b0 tx=0x7fcaac09fe80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32b3c4a7a5994639 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:14.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.962+0000 7fca9e7fc640 1 -- 172.21.15.98:0/2722233203 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac0a3b30 con 0x56158e0d1740 2024-05-09T08:56:14.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.962+0000 7fca9e7fc640 1 -- 172.21.15.98:0/2722233203 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaac0a83f0 con 0x56158e0d1740 2024-05-09T08:56:14.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.962+0000 7fca9e7fc640 1 -- 172.21.15.98:0/2722233203 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac0d33e0 con 0x56158e0d1740 2024-05-09T08:56:14.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.963+0000 7fcabda0f740 1 -- 172.21.15.98:0/2722233203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 msgr2=0x56158e0e9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:14.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.963+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2722233203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0e9560 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcaac0d09b0 tx=0x7fcaac09fe80 comp rx=0 tx=0).stop 2024-05-09T08:56:14.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.963+0000 7fcabda0f740 1 -- 172.21.15.98:0/2722233203 shutdown_connections 2024-05-09T08:56:14.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.963+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2722233203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0e9560 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.963+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2722233203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c80b0 0x56158e0d0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.963+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2722233203 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 0x56158e0dd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.963+0000 7fcabda0f740 1 -- 172.21.15.98:0/2722233203 >> 172.21.15.98:0/2722233203 conn(0x56158e08f0a0 msgr2=0x56158e0cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:14.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.963+0000 7fcabda0f740 1 -- 172.21.15.98:0/2722233203 shutdown_connections 2024-05-09T08:56:14.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.963+0000 7fcabda0f740 1 -- 172.21.15.98:0/2722233203 wait complete. 2024-05-09T08:56:14.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.964+0000 7fcabda0f740 1 -- start start 2024-05-09T08:56:14.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.964+0000 7fcabda0f740 1 Processor -- start 2024-05-09T08:56:14.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.964+0000 7fcabda0f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 0x56158e0d99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.965+0000 7fcabda0f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c80b0 0x56158e0d9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.965+0000 7fcabda0f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0d69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.965+0000 7fcabda0f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcab41b6870 con 0x56158e0c80b0 2024-05-09T08:56:14.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.965+0000 7fcabda0f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcab41b61a0 con 0x56158e0d1740 2024-05-09T08:56:14.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.965+0000 7fcab91c9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0d69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.965+0000 7fcab91c9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0d69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52586/0 (socket says 172.21.15.98:52586) 2024-05-09T08:56:14.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.965+0000 7fcab91c9640 1 -- 172.21.15.98:0/2000642262 learned_addr learned my addr 172.21.15.98:0/2000642262 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:14.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.965+0000 7fcabda0f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcab41b33e0 con 0x56158e0c69c0 2024-05-09T08:56:14.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.965+0000 7fcabda0f740 10 client.0 _ll_register_callbacks cb 0x7fcaac0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:14.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.965+0000 7fcab3fff640 1 --2- 172.21.15.98:0/2000642262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c80b0 0x56158e0d9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.966+0000 7fcab91c9640 1 -- 172.21.15.98:0/2000642262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 msgr2=0x56158e0d99e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.966+0000 7fcab91c9640 1 --2- 172.21.15.98:0/2000642262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 0x56158e0d99e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.966+0000 7fcab91c9640 1 -- 172.21.15.98:0/2000642262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c80b0 msgr2=0x56158e0d9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.966+0000 7fcab91c9640 1 --2- 172.21.15.98:0/2000642262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c80b0 0x56158e0d9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.966+0000 7fcab91c9640 1 -- 172.21.15.98:0/2000642262 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56158e0d6ff0 con 0x56158e0d1740 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.966+0000 7fcabda0f740 1 -- 172.21.15.98:0/2000642262 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcaac0a83f0 con 0x56158e0d1740 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.966+0000 7fcab91c9640 1 --2- 172.21.15.98:0/2000642262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0d69b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcaac0d0fb0 tx=0x7fcaac09dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f62abefb086101 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.966+0000 7fca93fff640 10 client.26477 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.966+0000 7fca93fff640 1 -- 172.21.15.98:0/2000642262 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac0bc070 con 0x56158e0d1740 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.966+0000 7fca93fff640 1 -- 172.21.15.98:0/2000642262 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fcaac0d3380 con 0x56158e0d1740 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.966+0000 7fca93fff640 1 -- 172.21.15.98:0/2000642262 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcaac0a3b30 con 0x56158e0d1740 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.966+0000 7fcabda0f740 10 client.26477 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.966+0000 7fcabda0f740 1 -- 172.21.15.98:0/2000642262 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({fsmap=35}) -- 0x56158e150de0 con 0x56158e0d1740 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.967+0000 7fca93fff640 1 -- 172.21.15.98:0/2000642262 <== mon.2 v2:172.21.15.105:3300/0 4 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fcaac00c070 con 0x56158e0d1740 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.967+0000 7fcabda0f740 10 client.26477 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:14.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.967+0000 7fcabda0f740 10 client.26477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:14.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.967+0000 7fcabda0f740 20 client.26477 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:14.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.967+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2000642262 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56158e0e18c0 0x56158e1535c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:14.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.967+0000 7fcabda0f740 4 client.26477 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:56:14.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.967+0000 7fcabda0f740 1 -- 172.21.15.98:0/2000642262 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56158e0390c0 con 0x56158e0e18c0 2024-05-09T08:56:14.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.968+0000 7fcab89c8640 1 --2- 172.21.15.98:0/2000642262 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56158e0e18c0 0x56158e1535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:14.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.968+0000 7fca93fff640 10 client.26477 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:14.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.968+0000 7fcab89c8640 1 --2- 172.21.15.98:0/2000642262 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56158e0e18c0 0x56158e1535c0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8485bb0e3d628f86 server_cookie=e099dcafa891ca in_seq=0 out_seq=0 2024-05-09T08:56:14.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.970+0000 7fca93fff640 1 -- 172.21.15.98:0/2000642262 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56158e0390c0 con 0x56158e0e18c0 2024-05-09T08:56:14.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.970+0000 7fca93fff640 10 client.26477 handle_command_reply: tid=0 2024-05-09T08:56:14.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.970+0000 7fcabda0f740 1 client.26477 shutdown 2024-05-09T08:56:14.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.970+0000 7fcabda0f740 1 -- 172.21.15.98:0/2000642262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 msgr2=0x56158e0d69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:14.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.971+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2000642262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0d69b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcaac0d0fb0 tx=0x7fcaac09dc90 comp rx=0 tx=0).stop 2024-05-09T08:56:14.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.971+0000 7fcabda0f740 1 -- 172.21.15.98:0/2000642262 shutdown_connections 2024-05-09T08:56:14.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.971+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2000642262 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56158e0e18c0 0x56158e1535c0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.971+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2000642262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56158e0d1740 0x56158e0d69b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.971+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2000642262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56158e0c80b0 0x56158e0d9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.971+0000 7fca93fff640 0 client.26477 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:14.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.971+0000 7fcabda0f740 1 --2- 172.21.15.98:0/2000642262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56158e0c69c0 0x56158e0d99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.971+0000 7fcabda0f740 1 -- 172.21.15.98:0/2000642262 >> 172.21.15.98:0/2000642262 conn(0x56158e08f0a0 msgr2=0x56158e0c2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:14.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.971+0000 7fcabda0f740 1 -- 172.21.15.98:0/2000642262 shutdown_connections 2024-05-09T08:56:14.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.971+0000 7fcabda0f740 1 -- 172.21.15.98:0/2000642262 wait complete. 2024-05-09T08:56:14.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.971+0000 7fcabda0f740 20 client.26477 trim_cache size 0 max 16384 2024-05-09T08:56:14.972 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:56:14.972 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:56:14.972 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "57981285-d170-4b60-a0ae-32fcb72b9e5a", 2024-05-09T08:56:14.972 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:56:14.972 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:56:14.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.973+0000 7fcabac2c640 1 -- 172.21.15.98:0/2171587814 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcaa40a2e50 msgr2=0x7fcaa40a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:14.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.973+0000 7fcabac2c640 1 --2- 172.21.15.98:0/2171587814 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcaa40a2e50 0x7fcaa40a5310 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcaa00057c0 tx=0x7fcaa0005a20 comp rx=0 tx=0).stop 2024-05-09T08:56:14.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.973+0000 7fcabac2c640 1 -- 172.21.15.98:0/2171587814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcab4106710 msgr2=0x7fcab406ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:14.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.973+0000 7fcabac2c640 1 --2- 172.21.15.98:0/2171587814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcab4106710 0x7fcab406ce60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00efd0 tx=0x7fcaac009f90 comp rx=0 tx=0).stop 2024-05-09T08:56:14.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.973+0000 7fcabac2c640 1 -- 172.21.15.98:0/2171587814 shutdown_connections 2024-05-09T08:56:14.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.973+0000 7fcabac2c640 1 --2- 172.21.15.98:0/2171587814 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcaa40a2e50 0x7fcaa40a5310 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.973+0000 7fcabac2c640 1 --2- 172.21.15.98:0/2171587814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcab4106710 0x7fcab406ce60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.973+0000 7fcabac2c640 1 --2- 172.21.15.98:0/2171587814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab41010c0 0x7fcab40732e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.973+0000 7fcabac2c640 1 --2- 172.21.15.98:0/2171587814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcab4100700 0x7fcab4072da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:14.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.973+0000 7fcabac2c640 1 -- 172.21.15.98:0/2171587814 >> 172.21.15.98:0/2171587814 conn(0x7fcab4077830 msgr2=0x7fcab4102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:14.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.973+0000 7fcabac2c640 1 -- 172.21.15.98:0/2171587814 shutdown_connections 2024-05-09T08:56:14.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:14.973+0000 7fcabac2c640 1 -- 172.21.15.98:0/2171587814 wait complete. 2024-05-09T08:56:14.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:14 smithi105 ceph-mon[34498]: pgmap v172: 129 pgs: 129 active+clean; 43 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 634 MiB/s wr, 1.88k op/s 2024-05-09T08:56:14.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:56:14.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:56:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.195+0000 7fec88ed2640 1 -- 172.21.15.98:0/2812862571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec84103f10 msgr2=0x7fec84104390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.195+0000 7fec88ed2640 1 --2- 172.21.15.98:0/2812862571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec84103f10 0x7fec84104390 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fec78009900 tx=0x7fec7802a400 comp rx=0 tx=0).stop 2024-05-09T08:56:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.195+0000 7fec88ed2640 1 -- 172.21.15.98:0/2812862571 shutdown_connections 2024-05-09T08:56:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.195+0000 7fec88ed2640 1 --2- 172.21.15.98:0/2812862571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec841048d0 0x7fec84110830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.195+0000 7fec88ed2640 1 --2- 172.21.15.98:0/2812862571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec84103f10 0x7fec84104390 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.195+0000 7fec88ed2640 1 --2- 172.21.15.98:0/2812862571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec84108a50 0x7fec84108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.196+0000 7fec88ed2640 1 -- 172.21.15.98:0/2812862571 >> 172.21.15.98:0/2812862571 conn(0x7fec840775c0 msgr2=0x7fec840799e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.196+0000 7fec88ed2640 1 -- 172.21.15.98:0/2812862571 shutdown_connections 2024-05-09T08:56:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.196+0000 7fec88ed2640 1 -- 172.21.15.98:0/2812862571 wait complete. 2024-05-09T08:56:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.197+0000 7fec88ed2640 1 Processor -- start 2024-05-09T08:56:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.197+0000 7fec88ed2640 1 -- start start 2024-05-09T08:56:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.197+0000 7fec88ed2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec84103f10 0x7fec841106f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.197+0000 7fec88ed2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec841048d0 0x7fec8410d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.197+0000 7fec88ed2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec84108a50 0x7fec8410dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.197+0000 7fec88ed2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fec84113ed0 con 0x7fec84103f10 2024-05-09T08:56:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.197+0000 7fec88ed2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fec84113d50 con 0x7fec84108a50 2024-05-09T08:56:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.197+0000 7fec88ed2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fec84114050 con 0x7fec841048d0 2024-05-09T08:56:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.198+0000 7fec82d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec84108a50 0x7fec8410dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.198+0000 7fec82d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec84108a50 0x7fec8410dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52602/0 (socket says 172.21.15.98:52602) 2024-05-09T08:56:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.198+0000 7fec82d9d640 1 -- 172.21.15.98:0/4116134939 learned_addr learned my addr 172.21.15.98:0/4116134939 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.198+0000 7fec81d9b640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec841048d0 0x7fec8410d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.198+0000 7fec82d9d640 1 -- 172.21.15.98:0/4116134939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec841048d0 msgr2=0x7fec8410d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.198+0000 7fec82d9d640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec841048d0 0x7fec8410d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.198+0000 7fec82d9d640 1 -- 172.21.15.98:0/4116134939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec84103f10 msgr2=0x7fec841106f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.198+0000 7fec8259c640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec84103f10 0x7fec841106f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.198+0000 7fec82d9d640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec84103f10 0x7fec841106f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.198+0000 7fec82d9d640 1 -- 172.21.15.98:0/4116134939 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8410e650 con 0x7fec84108a50 2024-05-09T08:56:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.198+0000 7fec81d9b640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec841048d0 0x7fec8410d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:56:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.198+0000 7fec8259c640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec84103f10 0x7fec841106f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:56:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.198+0000 7fec82d9d640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec84108a50 0x7fec8410dd90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00eb10 tx=0x7fec7c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.199+0000 7fec737fe640 1 -- 172.21.15.98:0/4116134939 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c004b20 con 0x7fec84108a50 2024-05-09T08:56:15.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.199+0000 7fec88ed2640 1 -- 172.21.15.98:0/4116134939 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec8406b370 con 0x7fec84108a50 2024-05-09T08:56:15.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.199+0000 7fec88ed2640 1 -- 172.21.15.98:0/4116134939 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec8406b890 con 0x7fec84108a50 2024-05-09T08:56:15.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.199+0000 7fec737fe640 1 -- 172.21.15.98:0/4116134939 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec7c004cc0 con 0x7fec84108a50 2024-05-09T08:56:15.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.201+0000 7fec737fe640 1 -- 172.21.15.98:0/4116134939 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c010760 con 0x7fec84108a50 2024-05-09T08:56:15.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.204+0000 7fec8bbde740 1 Processor -- start 2024-05-09T08:56:15.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.204+0000 7fec8bbde740 1 -- start start 2024-05-09T08:56:15.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.204+0000 7fec8bbde740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23a4280 0x562ab23a4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.204+0000 7fec8bbde740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 0x562ab23aed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.205+0000 7fec8bbde740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23af270 0x562ab23b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.205+0000 7fec8bbde740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562ab2309d00 con 0x562ab23a4ba0 2024-05-09T08:56:15.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.205+0000 7fec8bbde740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562ab22b14c0 con 0x562ab23af270 2024-05-09T08:56:15.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.205+0000 7fec8bbde740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562ab2305cc0 con 0x562ab23a4280 2024-05-09T08:56:15.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.205+0000 7fec81d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23a4280 0x562ab23a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.205+0000 7fec81d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23a4280 0x562ab23a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39698/0 (socket says 172.21.15.98:39698) 2024-05-09T08:56:15.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.205+0000 7fec81d9b640 1 -- 172.21.15.98:0/3216086674 learned_addr learned my addr 172.21.15.98:0/3216086674 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:15.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.205+0000 7fec81d9b640 1 --2- 172.21.15.98:0/3216086674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23af270 0x562ab23b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.205+0000 7fec81d9b640 1 -- 172.21.15.98:0/3216086674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23af270 msgr2=0x562ab23b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:15.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.205+0000 7fec81d9b640 1 --2- 172.21.15.98:0/3216086674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23af270 0x562ab23b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.205+0000 7fec81d9b640 1 -- 172.21.15.98:0/3216086674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 msgr2=0x562ab23aed30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:56:15.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.205+0000 7fec81d9b640 1 --2- 172.21.15.98:0/3216086674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 0x562ab23aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.205+0000 7fec81d9b640 1 -- 172.21.15.98:0/3216086674 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ab20e77d0 con 0x562ab23a4280 2024-05-09T08:56:15.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.206+0000 7fec737fe640 1 -- 172.21.15.98:0/4116134939 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fec7c0109c0 con 0x7fec84108a50 2024-05-09T08:56:15.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.206+0000 7fec81d9b640 1 --2- 172.21.15.98:0/3216086674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23a4280 0x562ab23a4660 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fec78009230 tx=0x7fec78009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37217fc8141303a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:15.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.207+0000 7fec737fe640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fec6c0a2e00 0x7fec6c0a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.207+0000 7fec8259c640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fec6c0a2e00 0x7fec6c0a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.207+0000 7fec737fe640 1 -- 172.21.15.98:0/4116134939 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fec7c0d0ec0 con 0x7fec84108a50 2024-05-09T08:56:15.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.207+0000 7fec70ff9640 1 -- 172.21.15.98:0/3216086674 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78035040 con 0x562ab23a4280 2024-05-09T08:56:15.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.208+0000 7fec70ff9640 1 -- 172.21.15.98:0/3216086674 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec78005d10 con 0x562ab23a4280 2024-05-09T08:56:15.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.208+0000 7fec70ff9640 1 -- 172.21.15.98:0/3216086674 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78003d70 con 0x562ab23a4280 2024-05-09T08:56:15.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.209+0000 7fec8bbde740 1 -- 172.21.15.98:0/3216086674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23a4280 msgr2=0x562ab23a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.209+0000 7fec8bbde740 1 --2- 172.21.15.98:0/3216086674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23a4280 0x562ab23a4660 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fec78009230 tx=0x7fec78009560 comp rx=0 tx=0).stop 2024-05-09T08:56:15.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.209+0000 7fec8259c640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fec6c0a2e00 0x7fec6c0a52c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fec74006fd0 tx=0x7fec74008040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:15.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.209+0000 7fec8bbde740 1 -- 172.21.15.98:0/3216086674 shutdown_connections 2024-05-09T08:56:15.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.209+0000 7fec8bbde740 1 --2- 172.21.15.98:0/3216086674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23af270 0x562ab23b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.209+0000 7fec8bbde740 1 --2- 172.21.15.98:0/3216086674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 0x562ab23aed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.209+0000 7fec8bbde740 1 --2- 172.21.15.98:0/3216086674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23a4280 0x562ab23a4660 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.209+0000 7fec8bbde740 1 -- 172.21.15.98:0/3216086674 >> 172.21.15.98:0/3216086674 conn(0x562ab23a05a0 msgr2=0x562ab23a0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:15.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.209+0000 7fec8bbde740 1 -- 172.21.15.98:0/3216086674 shutdown_connections 2024-05-09T08:56:15.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.209+0000 7fec8bbde740 1 -- 172.21.15.98:0/3216086674 wait complete. 2024-05-09T08:56:15.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.210+0000 7fec8bbde740 1 -- start start 2024-05-09T08:56:15.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.210+0000 7fec8bbde740 1 Processor -- start 2024-05-09T08:56:15.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.210+0000 7fec8bbde740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23a4280 0x562ab23ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.210+0000 7fec8bbde740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 0x562ab23ac730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.210+0000 7fec8bbde740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23af270 0x562ab23acc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.210+0000 7fec8bbde740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562ab23b2000 con 0x562ab23a4ba0 2024-05-09T08:56:15.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.210+0000 7fec8bbde740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562ab22c4b90 con 0x562ab23af270 2024-05-09T08:56:15.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.210+0000 7fec8bbde740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562ab23b2180 con 0x562ab23a4280 2024-05-09T08:56:15.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.210+0000 7fec8bbde740 10 client.0 _ll_register_callbacks cb 0x562ab2300da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:15.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.211+0000 7fec8259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23a4280 0x562ab23ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.211+0000 7fec8259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23a4280 0x562ab23ac1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39704/0 (socket says 172.21.15.98:39704) 2024-05-09T08:56:15.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.211+0000 7fec8259c640 1 -- 172.21.15.98:0/532805354 learned_addr learned my addr 172.21.15.98:0/532805354 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:15.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.211+0000 7fec82d9d640 1 --2- 172.21.15.98:0/532805354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23af270 0x562ab23acc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.211+0000 7fec82d9d640 1 -- 172.21.15.98:0/532805354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23a4280 msgr2=0x562ab23ac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:15.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.211+0000 7fec82d9d640 1 --2- 172.21.15.98:0/532805354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23a4280 0x562ab23ac1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.211+0000 7fec82d9d640 1 -- 172.21.15.98:0/532805354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 msgr2=0x562ab23ac730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:15.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.211+0000 7fec82d9d640 1 --2- 172.21.15.98:0/532805354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 0x562ab23ac730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.211+0000 7fec82d9d640 1 -- 172.21.15.98:0/532805354 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ab201d3f0 con 0x562ab23af270 2024-05-09T08:56:15.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.211+0000 7fec82d9d640 1 --2- 172.21.15.98:0/532805354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23af270 0x562ab23acc70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0ce1f0 tx=0x7fec7c09c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf90d8d4fed67e42 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:15.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.211+0000 7fec69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:56:15.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.211+0000 7fec69ffb640 1 -- 172.21.15.98:0/532805354 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c0b5020 con 0x562ab23af270 2024-05-09T08:56:15.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.212+0000 7fec69ffb640 1 -- 172.21.15.98:0/532805354 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fec7c0a57a0 con 0x562ab23af270 2024-05-09T08:56:15.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.212+0000 7fec8bbde740 1 -- 172.21.15.98:0/532805354 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec78005d10 con 0x562ab23af270 2024-05-09T08:56:15.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.212+0000 7fec69ffb640 1 -- 172.21.15.98:0/532805354 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fec7c0ce670 con 0x562ab23af270 2024-05-09T08:56:15.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.212+0000 7fec8bbde740 10 client.26486 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:15.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.212+0000 7fec8bbde740 1 -- 172.21.15.98:0/532805354 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562ab1fdd0f0 con 0x562ab23af270 2024-05-09T08:56:15.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.212+0000 7fec69ffb640 1 -- 172.21.15.98:0/532805354 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec78005d10 con 0x562ab23af270 2024-05-09T08:56:15.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.212+0000 7fec69ffb640 1 -- 172.21.15.98:0/532805354 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fec7c09f030 con 0x562ab23af270 2024-05-09T08:56:15.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.212+0000 7fec8bbde740 10 client.26486 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:15.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.212+0000 7fec8bbde740 10 client.26486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:15.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.212+0000 7fec8bbde740 20 client.26486 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:15.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.212+0000 7fec8bbde740 1 --2- 172.21.15.98:0/532805354 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562ab23bf6b0 0x562ab23c1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.212+0000 7fec8bbde740 4 client.26486 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:15.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.212+0000 7fec8bbde740 1 -- 172.21.15.98:0/532805354 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ab2317430 con 0x562ab23bf6b0 2024-05-09T08:56:15.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.213+0000 7fec8259c640 1 --2- 172.21.15.98:0/532805354 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562ab23bf6b0 0x562ab23c1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.213+0000 7fec8259c640 1 --2- 172.21.15.98:0/532805354 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562ab23bf6b0 0x562ab23c1b00 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2381c809ace156f7 server_cookie=83679e0181d0c344 in_seq=0 out_seq=0 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.213+0000 7fec69ffb640 10 client.26486 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.214+0000 7fec69ffb640 1 -- 172.21.15.98:0/532805354 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562ab2317430 con 0x562ab23bf6b0 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.214+0000 7fec69ffb640 10 client.26486 handle_command_reply: tid=0 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.214+0000 7fec8bbde740 1 client.26486 shutdown 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.215+0000 7fec8bbde740 1 -- 172.21.15.98:0/532805354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23af270 msgr2=0x562ab23acc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.215+0000 7fec8bbde740 1 --2- 172.21.15.98:0/532805354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23af270 0x562ab23acc70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0ce1f0 tx=0x7fec7c09c9e0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.215+0000 7fec8bbde740 1 -- 172.21.15.98:0/532805354 shutdown_connections 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.215+0000 7fec8bbde740 1 --2- 172.21.15.98:0/532805354 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562ab23bf6b0 0x562ab23c1b00 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.215+0000 7fec8bbde740 1 --2- 172.21.15.98:0/532805354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23af270 0x562ab23acc70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.215+0000 7fec8bbde740 1 --2- 172.21.15.98:0/532805354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 0x562ab23ac730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.215+0000 7fec8bbde740 1 --2- 172.21.15.98:0/532805354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23a4280 0x562ab23ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.215+0000 7fec8bbde740 1 -- 172.21.15.98:0/532805354 >> 172.21.15.98:0/532805354 conn(0x562ab23a05a0 msgr2=0x562ab239f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.215+0000 7fec69ffb640 0 client.26486 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.215+0000 7fec8bbde740 1 -- 172.21.15.98:0/532805354 shutdown_connections 2024-05-09T08:56:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.215+0000 7fec8bbde740 1 -- 172.21.15.98:0/532805354 wait complete. 2024-05-09T08:56:15.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.215+0000 7fec8bbde740 20 client.26486 trim_cache size 0 max 16384 2024-05-09T08:56:15.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.246+0000 7fec8bbde740 1 Processor -- start 2024-05-09T08:56:15.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.246+0000 7fec8bbde740 1 -- start start 2024-05-09T08:56:15.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.247+0000 7fec8bbde740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4280 0x562ab23baf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.247+0000 7fec8bbde740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23a4ba0 0x562ab23a7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.247+0000 7fec8bbde740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23af270 0x562ab23a69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.247+0000 7fec8bbde740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fec64010040 con 0x562ab23a4280 2024-05-09T08:56:15.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.247+0000 7fec8bbde740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fec64011b80 con 0x562ab23a4ba0 2024-05-09T08:56:15.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.247+0000 7fec8bbde740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fec64010710 con 0x562ab23af270 2024-05-09T08:56:15.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.247+0000 7fec81d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23a4ba0 0x562ab23a7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.247+0000 7fec81d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23a4ba0 0x562ab23a7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:52628/0 (socket says 172.21.15.98:52628) 2024-05-09T08:56:15.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.247+0000 7fec81d9b640 1 -- 172.21.15.98:0/2391621888 learned_addr learned my addr 172.21.15.98:0/2391621888 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:15.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.247+0000 7fec82d9d640 1 --2- 172.21.15.98:0/2391621888 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23af270 0x562ab23a69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.248+0000 7fec81d9b640 1 -- 172.21.15.98:0/2391621888 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23af270 msgr2=0x562ab23a69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:15.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.248+0000 7fec81d9b640 1 --2- 172.21.15.98:0/2391621888 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23af270 0x562ab23a69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.248+0000 7fec81d9b640 1 -- 172.21.15.98:0/2391621888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4280 msgr2=0x562ab23baf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:15.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.248+0000 7fec81d9b640 1 --2- 172.21.15.98:0/2391621888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4280 0x562ab23baf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.248+0000 7fec81d9b640 1 -- 172.21.15.98:0/2391621888 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ab23a5510 con 0x562ab23a4ba0 2024-05-09T08:56:15.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.248+0000 7fec81d9b640 1 --2- 172.21.15.98:0/2391621888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23a4ba0 0x562ab23a7550 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fec78000c00 tx=0x7fec78009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfc546c2223f71bf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.251+0000 7fec6bfff640 1 -- 172.21.15.98:0/2391621888 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78071070 con 0x562ab23a4ba0 2024-05-09T08:56:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.251+0000 7fec6bfff640 1 -- 172.21.15.98:0/2391621888 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec78004c40 con 0x562ab23a4ba0 2024-05-09T08:56:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.251+0000 7fec6bfff640 1 -- 172.21.15.98:0/2391621888 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78035070 con 0x562ab23a4ba0 2024-05-09T08:56:15.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.252+0000 7fec8bbde740 1 -- 172.21.15.98:0/2391621888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23a4ba0 msgr2=0x562ab23a7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.252+0000 7fec8bbde740 1 --2- 172.21.15.98:0/2391621888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23a4ba0 0x562ab23a7550 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fec78000c00 tx=0x7fec78009260 comp rx=0 tx=0).stop 2024-05-09T08:56:15.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.252+0000 7fec8bbde740 1 -- 172.21.15.98:0/2391621888 shutdown_connections 2024-05-09T08:56:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.252+0000 7fec8bbde740 1 --2- 172.21.15.98:0/2391621888 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23af270 0x562ab23a69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.252+0000 7fec8bbde740 1 --2- 172.21.15.98:0/2391621888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23a4ba0 0x562ab23a7550 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.252+0000 7fec8bbde740 1 --2- 172.21.15.98:0/2391621888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4280 0x562ab23baf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.252+0000 7fec8bbde740 1 -- 172.21.15.98:0/2391621888 >> 172.21.15.98:0/2391621888 conn(0x562ab23a05a0 msgr2=0x562ab23ada20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.253+0000 7fec8bbde740 1 -- 172.21.15.98:0/2391621888 shutdown_connections 2024-05-09T08:56:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.253+0000 7fec8bbde740 1 -- 172.21.15.98:0/2391621888 wait complete. 2024-05-09T08:56:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.253+0000 7fec8bbde740 1 -- start start 2024-05-09T08:56:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.253+0000 7fec8bbde740 1 Processor -- start 2024-05-09T08:56:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.254+0000 7fec8bbde740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23a4280 0x562ab23a9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.254+0000 7fec8bbde740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 0x562ab23a9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.254+0000 7fec8bbde740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23af270 0x562ab23a9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.254+0000 7fec8bbde740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fec640114b0 con 0x562ab23a4ba0 2024-05-09T08:56:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.254+0000 7fec8bbde740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fec64010de0 con 0x562ab23a4280 2024-05-09T08:56:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.254+0000 7fec8bbde740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fec6400e020 con 0x562ab23af270 2024-05-09T08:56:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.254+0000 7fec8bbde740 10 client.0 _ll_register_callbacks cb 0x7fec7c09cb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:15.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.255+0000 7fec81d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 0x562ab23a9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.255+0000 7fec81d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 0x562ab23a9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39772/0 (socket says 172.21.15.98:39772) 2024-05-09T08:56:15.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.255+0000 7fec81d9b640 1 -- 172.21.15.98:0/1492244407 learned_addr learned my addr 172.21.15.98:0/1492244407 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:15.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.255+0000 7fec82d9d640 1 --2- 172.21.15.98:0/1492244407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23af270 0x562ab23a9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.255+0000 7fec8259c640 1 --2- 172.21.15.98:0/1492244407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23a4280 0x562ab23a9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.255+0000 7fec81d9b640 1 -- 172.21.15.98:0/1492244407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23af270 msgr2=0x562ab23a9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:15.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.255+0000 7fec81d9b640 1 --2- 172.21.15.98:0/1492244407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23af270 0x562ab23a9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.255+0000 7fec81d9b640 1 -- 172.21.15.98:0/1492244407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23a4280 msgr2=0x562ab23a9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:15.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.255+0000 7fec81d9b640 1 --2- 172.21.15.98:0/1492244407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23a4280 0x562ab23a9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.255+0000 7fec81d9b640 1 -- 172.21.15.98:0/1492244407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ab2424e30 con 0x562ab23a4ba0 2024-05-09T08:56:15.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.256+0000 7fec81d9b640 1 --2- 172.21.15.98:0/1492244407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 0x562ab23a9560 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fec7802aad0 tx=0x7fec78005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1152093a0067f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:15.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.256+0000 7fec69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:56:15.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.256+0000 7fec69ffb640 1 -- 172.21.15.98:0/1492244407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78035070 con 0x562ab23a4ba0 2024-05-09T08:56:15.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.256+0000 7fec69ffb640 1 -- 172.21.15.98:0/1492244407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fec78004760 con 0x562ab23a4ba0 2024-05-09T08:56:15.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.256+0000 7fec8bbde740 1 -- 172.21.15.98:0/1492244407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec78004c40 con 0x562ab23a4ba0 2024-05-09T08:56:15.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.256+0000 7fec69ffb640 1 -- 172.21.15.98:0/1492244407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fec78071040 con 0x562ab23a4ba0 2024-05-09T08:56:15.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.256+0000 7fec8bbde740 10 client.17184 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:15.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.256+0000 7fec8bbde740 1 -- 172.21.15.98:0/1492244407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562ab2426100 con 0x562ab23a4ba0 2024-05-09T08:56:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.257+0000 7fec69ffb640 1 -- 172.21.15.98:0/1492244407 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec78004c40 con 0x562ab23a4ba0 2024-05-09T08:56:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.257+0000 7fec69ffb640 1 -- 172.21.15.98:0/1492244407 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fec78074070 con 0x562ab23a4ba0 2024-05-09T08:56:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.258+0000 7fec8bbde740 10 client.17184 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.258+0000 7fec8bbde740 10 client.17184 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.258+0000 7fec8bbde740 20 client.17184 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.258+0000 7fec8bbde740 1 --2- 172.21.15.98:0/1492244407 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562ab2426330 0x562ab2428720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.258+0000 7fec8bbde740 4 client.17184 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:56:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.258+0000 7fec8bbde740 1 -- 172.21.15.98:0/1492244407 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562ab2311780 con 0x562ab2426330 2024-05-09T08:56:15.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.259+0000 7fec8259c640 1 --2- 172.21.15.98:0/1492244407 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562ab2426330 0x562ab2428720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.261+0000 7fec8259c640 1 --2- 172.21.15.98:0/1492244407 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562ab2426330 0x562ab2428720 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b4c1442d25e8f35 server_cookie=f9dd944f9cf5a190 in_seq=0 out_seq=0 2024-05-09T08:56:15.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.261+0000 7fec69ffb640 10 client.17184 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:15.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.261+0000 7fec69ffb640 1 -- 172.21.15.98:0/1492244407 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x562ab2311780 con 0x562ab2426330 2024-05-09T08:56:15.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.261+0000 7fec69ffb640 10 client.17184 handle_command_reply: tid=0 2024-05-09T08:56:15.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.261+0000 7fec8bbde740 1 client.17184 shutdown 2024-05-09T08:56:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.262+0000 7fec8bbde740 1 -- 172.21.15.98:0/1492244407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 msgr2=0x562ab23a9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.262+0000 7fec8bbde740 1 --2- 172.21.15.98:0/1492244407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 0x562ab23a9560 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fec7802aad0 tx=0x7fec78005770 comp rx=0 tx=0).stop 2024-05-09T08:56:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.262+0000 7fec8bbde740 1 -- 172.21.15.98:0/1492244407 shutdown_connections 2024-05-09T08:56:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.262+0000 7fec8bbde740 1 --2- 172.21.15.98:0/1492244407 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562ab2426330 0x562ab2428720 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.262+0000 7fec8bbde740 1 --2- 172.21.15.98:0/1492244407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562ab23af270 0x562ab23a9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.262+0000 7fec8bbde740 1 --2- 172.21.15.98:0/1492244407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ab23a4ba0 0x562ab23a9560 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.262+0000 7fec8bbde740 1 --2- 172.21.15.98:0/1492244407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ab23a4280 0x562ab23a9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.262+0000 7fec8bbde740 1 -- 172.21.15.98:0/1492244407 >> 172.21.15.98:0/1492244407 conn(0x562ab23a05a0 msgr2=0x562ab23ae2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:15.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.262+0000 7fec69ffb640 0 client.17184 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:15.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.262+0000 7fec8bbde740 1 -- 172.21.15.98:0/1492244407 shutdown_connections 2024-05-09T08:56:15.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.263+0000 7fec8bbde740 1 -- 172.21.15.98:0/1492244407 wait complete. 2024-05-09T08:56:15.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.264+0000 7fec8bbde740 20 client.17184 trim_cache size 0 max 16384 2024-05-09T08:56:15.263 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:56:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "scrub active (0 inodes in the stack)", 2024-05-09T08:56:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:56:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "57981285-d170-4b60-a0ae-32fcb72b9e5a": { 2024-05-09T08:56:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:56:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "57981285-d170-4b60-a0ae-32fcb72b9e5a", 2024-05-09T08:56:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:56:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:56:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:56:15.264 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:56:15.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.264+0000 7fec88ed2640 1 -- 172.21.15.98:0/4116134939 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fec6c0a2e00 msgr2=0x7fec6c0a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.264+0000 7fec88ed2640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fec6c0a2e00 0x7fec6c0a52c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fec74006fd0 tx=0x7fec74008040 comp rx=0 tx=0).stop 2024-05-09T08:56:15.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.264+0000 7fec88ed2640 1 -- 172.21.15.98:0/4116134939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec84108a50 msgr2=0x7fec8410dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.264+0000 7fec88ed2640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec84108a50 0x7fec8410dd90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00eb10 tx=0x7fec7c00c2d0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.265+0000 7fec88ed2640 1 -- 172.21.15.98:0/4116134939 shutdown_connections 2024-05-09T08:56:15.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.265+0000 7fec88ed2640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fec6c0a2e00 0x7fec6c0a52c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.265+0000 7fec88ed2640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec84108a50 0x7fec8410dd90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.265+0000 7fec88ed2640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec841048d0 0x7fec8410d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.265+0000 7fec88ed2640 1 --2- 172.21.15.98:0/4116134939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec84103f10 0x7fec841106f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.265+0000 7fec88ed2640 1 -- 172.21.15.98:0/4116134939 >> 172.21.15.98:0/4116134939 conn(0x7fec840775c0 msgr2=0x7fec84077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:15.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.265+0000 7fec88ed2640 1 -- 172.21.15.98:0/4116134939 shutdown_connections 2024-05-09T08:56:15.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:15.265+0000 7fec88ed2640 1 -- 172.21.15.98:0/4116134939 wait complete. 2024-05-09T08:56:15.288 INFO:tasks.cephfs.filesystem:scrub status for tag:57981285-d170-4b60-a0ae-32fcb72b9e5a - {'path': '/', 'tag': '57981285-d170-4b60-a0ae-32fcb72b9e5a', 'options': 'recursive,force'} 2024-05-09T08:56:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.540+0000 7fe4d7df4640 1 -- 172.21.15.98:0/2476704074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d01008b0 msgr2=0x7fe4d0100c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.540+0000 7fe4d7df4640 1 --2- 172.21.15.98:0/2476704074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d01008b0 0x7fe4d0100c90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc01a910 tx=0x7fe4cc03b590 comp rx=0 tx=0).stop 2024-05-09T08:56:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.540+0000 7fe4d7df4640 1 -- 172.21.15.98:0/2476704074 shutdown_connections 2024-05-09T08:56:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.540+0000 7fe4d7df4640 1 --2- 172.21.15.98:0/2476704074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4d0105bd0 0x7fe4d0106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.540+0000 7fe4d7df4640 1 --2- 172.21.15.98:0/2476704074 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4d01011d0 0x7fe4d0105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.540+0000 7fe4d7df4640 1 --2- 172.21.15.98:0/2476704074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d01008b0 0x7fe4d0100c90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.540+0000 7fe4d7df4640 1 -- 172.21.15.98:0/2476704074 >> 172.21.15.98:0/2476704074 conn(0x7fe4d00778b0 msgr2=0x7fe4d0077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.541+0000 7fe4d7df4640 1 -- 172.21.15.98:0/2476704074 shutdown_connections 2024-05-09T08:56:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.541+0000 7fe4d7df4640 1 -- 172.21.15.98:0/2476704074 wait complete. 2024-05-09T08:56:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.542+0000 7fe4d7df4640 1 Processor -- start 2024-05-09T08:56:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.542+0000 7fe4d7df4640 1 -- start start 2024-05-09T08:56:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.543+0000 7fe4d7df4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4d01008b0 0x7fe4d0072390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.543+0000 7fe4d7df4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4d01011d0 0x7fe4d00728d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.543+0000 7fe4d7df4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d0105bd0 0x7fe4d010d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.543+0000 7fe4d7df4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d01096e0 con 0x7fe4d01011d0 2024-05-09T08:56:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.543+0000 7fe4d7df4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d0109560 con 0x7fe4d01008b0 2024-05-09T08:56:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.543+0000 7fe4d7df4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d0109860 con 0x7fe4d0105bd0 2024-05-09T08:56:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.543+0000 7fe4d538f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4d01011d0 0x7fe4d00728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.544+0000 7fe4d538f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4d01011d0 0x7fe4d00728d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39798/0 (socket says 172.21.15.98:39798) 2024-05-09T08:56:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.544+0000 7fe4d538f640 1 -- 172.21.15.98:0/2803133543 learned_addr learned my addr 172.21.15.98:0/2803133543 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.544+0000 7fe4d6391640 1 --2- 172.21.15.98:0/2803133543 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d0105bd0 0x7fe4d010d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.544+0000 7fe4d538f640 1 -- 172.21.15.98:0/2803133543 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d0105bd0 msgr2=0x7fe4d010d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.544+0000 7fe4d538f640 1 --2- 172.21.15.98:0/2803133543 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d0105bd0 0x7fe4d010d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.544+0000 7fe4d538f640 1 -- 172.21.15.98:0/2803133543 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4d01008b0 msgr2=0x7fe4d0072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.546 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.544+0000 7fe4d538f640 1 --2- 172.21.15.98:0/2803133543 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4d01008b0 0x7fe4d0072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.546 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.544+0000 7fe4d538f640 1 -- 172.21.15.98:0/2803133543 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d010d540 con 0x7fe4d01011d0 2024-05-09T08:56:15.546 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.544+0000 7fe4d538f640 1 --2- 172.21.15.98:0/2803133543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4d01011d0 0x7fe4d00728d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000b480 tx=0x7fe4c000b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.546+0000 7fe4c6ffd640 1 -- 172.21.15.98:0/2803133543 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0014070 con 0x7fe4d01011d0 2024-05-09T08:56:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.546+0000 7fe4c6ffd640 1 -- 172.21.15.98:0/2803133543 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4c0010d50 con 0x7fe4d01011d0 2024-05-09T08:56:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.547+0000 7fe4c6ffd640 1 -- 172.21.15.98:0/2803133543 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0017360 con 0x7fe4d01011d0 2024-05-09T08:56:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.547+0000 7fe4d7df4640 1 -- 172.21.15.98:0/2803133543 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4d010d7d0 con 0x7fe4d01011d0 2024-05-09T08:56:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.547+0000 7fe4d7df4640 1 -- 172.21.15.98:0/2803133543 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4d010dca0 con 0x7fe4d01011d0 2024-05-09T08:56:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.554+0000 7fe4dabd6740 1 Processor -- start 2024-05-09T08:56:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.554+0000 7fe4dabd6740 1 -- start start 2024-05-09T08:56:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.555+0000 7fe4dabd6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f23450 0x558d12f23830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.555+0000 7fe4dabd6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23d70 0x558d12f2e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.555+0000 7fe4dabd6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f2e630 0x558d12f30a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.555+0000 7fe4dabd6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558d12e88340 con 0x558d12f23450 2024-05-09T08:56:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.555+0000 7fe4dabd6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558d12e30290 con 0x558d12f2e630 2024-05-09T08:56:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.555+0000 7fe4dabd6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558d12e84300 con 0x558d12f23d70 2024-05-09T08:56:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.555+0000 7fe4d5b90640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23d70 0x558d12f2e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.555+0000 7fe4d5b90640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23d70 0x558d12f2e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39744/0 (socket says 172.21.15.98:39744) 2024-05-09T08:56:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.555+0000 7fe4d5b90640 1 -- 172.21.15.98:0/2796570230 learned_addr learned my addr 172.21.15.98:0/2796570230 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.556+0000 7fe4d5b90640 1 -- 172.21.15.98:0/2796570230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f2e630 msgr2=0x558d12f30a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:56:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.556+0000 7fe4d5b90640 1 --2- 172.21.15.98:0/2796570230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f2e630 0x558d12f30a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.556+0000 7fe4d5b90640 1 -- 172.21.15.98:0/2796570230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f23450 msgr2=0x558d12f23830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.556+0000 7fe4d5b90640 1 --2- 172.21.15.98:0/2796570230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f23450 0x558d12f23830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.556+0000 7fe4d5b90640 1 -- 172.21.15.98:0/2796570230 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d12e548e0 con 0x558d12f23d70 2024-05-09T08:56:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.557+0000 7fe4c6ffd640 1 -- 172.21.15.98:0/2803133543 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe4c0017070 con 0x7fe4d01011d0 2024-05-09T08:56:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.558+0000 7fe4c6ffd640 1 --2- 172.21.15.98:0/2803133543 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe4bc0a2e20 0x7fe4bc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.560+0000 7fe4d6391640 1 --2- 172.21.15.98:0/2803133543 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe4bc0a2e20 0x7fe4bc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.561+0000 7fe4d5b90640 1 --2- 172.21.15.98:0/2796570230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23d70 0x558d12f2e0f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc004800 tx=0x7fe4cc004830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ec29172f4ae52a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.561+0000 7fe4bbfff640 1 -- 172.21.15.98:0/2796570230 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc04f020 con 0x558d12f23d70 2024-05-09T08:56:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.561+0000 7fe4bbfff640 1 -- 172.21.15.98:0/2796570230 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4cc004030 con 0x558d12f23d70 2024-05-09T08:56:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.561+0000 7fe4d6391640 1 --2- 172.21.15.98:0/2803133543 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe4bc0a2e20 0x7fe4bc0a52e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8009ea0 tx=0x7fe4c8009490 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.561+0000 7fe4c6ffd640 1 -- 172.21.15.98:0/2803133543 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fe4c00cec20 con 0x7fe4d01011d0 2024-05-09T08:56:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.562+0000 7fe4dabd6740 1 -- 172.21.15.98:0/2796570230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23d70 msgr2=0x558d12f2e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.562+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/2796570230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23d70 0x558d12f2e0f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc004800 tx=0x7fe4cc004830 comp rx=0 tx=0).stop 2024-05-09T08:56:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.562+0000 7fe4dabd6740 1 -- 172.21.15.98:0/2796570230 shutdown_connections 2024-05-09T08:56:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.563+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/2796570230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f2e630 0x558d12f30a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.563+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/2796570230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23d70 0x558d12f2e0f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.563+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/2796570230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f23450 0x558d12f23830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.563+0000 7fe4dabd6740 1 -- 172.21.15.98:0/2796570230 >> 172.21.15.98:0/2796570230 conn(0x558d12f1f960 msgr2=0x558d12f1fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.564+0000 7fe4dabd6740 1 -- 172.21.15.98:0/2796570230 shutdown_connections 2024-05-09T08:56:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.564+0000 7fe4dabd6740 1 -- 172.21.15.98:0/2796570230 wait complete. 2024-05-09T08:56:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.565+0000 7fe4dabd6740 1 -- start start 2024-05-09T08:56:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.565+0000 7fe4dabd6740 1 Processor -- start 2024-05-09T08:56:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.566+0000 7fe4dabd6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f23450 0x558d12f2b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.566+0000 7fe4dabd6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23d70 0x558d12f2ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.566+0000 7fe4dabd6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f2e630 0x558d12f2bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.566+0000 7fe4dabd6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558d12f31200 con 0x558d12f23450 2024-05-09T08:56:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.566+0000 7fe4dabd6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558d12e43790 con 0x558d12f23d70 2024-05-09T08:56:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.566+0000 7fe4dabd6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558d12f31380 con 0x558d12f2e630 2024-05-09T08:56:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.566+0000 7fe4dabd6740 10 client.0 _ll_register_callbacks cb 0x558d12e7ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.568+0000 7fe4d538f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23d70 0x558d12f2ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.568 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.568+0000 7fe4d538f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23d70 0x558d12f2ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:41732/0 (socket says 172.21.15.98:41732) 2024-05-09T08:56:15.568 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.568+0000 7fe4d538f640 1 -- 172.21.15.98:0/1412543642 learned_addr learned my addr 172.21.15.98:0/1412543642 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.568+0000 7fe4d6391640 1 --2- 172.21.15.98:0/1412543642 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f2e630 0x558d12f2bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.571+0000 7fe4d5b90640 1 --2- 172.21.15.98:0/1412543642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f23450 0x558d12f2b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.572+0000 7fe4d6391640 1 -- 172.21.15.98:0/1412543642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23d70 msgr2=0x558d12f2ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.572+0000 7fe4d6391640 1 --2- 172.21.15.98:0/1412543642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23d70 0x558d12f2ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.572+0000 7fe4d6391640 1 -- 172.21.15.98:0/1412543642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f23450 msgr2=0x558d12f2b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.572+0000 7fe4d6391640 1 --2- 172.21.15.98:0/1412543642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f23450 0x558d12f2b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.572+0000 7fe4d6391640 1 -- 172.21.15.98:0/1412543642 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d12b9c210 con 0x558d12f2e630 2024-05-09T08:56:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.573+0000 7fe4d5b90640 1 --2- 172.21.15.98:0/1412543642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f23450 0x558d12f2b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:56:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.574+0000 7fe4d6391640 1 --2- 172.21.15.98:0/1412543642 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f2e630 0x558d12f2bfc0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8004b90 tx=0x7fe4c8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a835fe1cc0a6c378 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.575+0000 7fe4b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:56:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.576+0000 7fe4b97fa640 1 -- 172.21.15.98:0/1412543642 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8027060 con 0x558d12f2e630 2024-05-09T08:56:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.576+0000 7fe4dabd6740 1 -- 172.21.15.98:0/1412543642 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4cc004030 con 0x558d12f2e630 2024-05-09T08:56:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.576+0000 7fe4b97fa640 1 -- 172.21.15.98:0/1412543642 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fe4c8012030 con 0x558d12f2e630 2024-05-09T08:56:15.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.578+0000 7fe4b97fa640 1 -- 172.21.15.98:0/1412543642 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe4c8017cb0 con 0x558d12f2e630 2024-05-09T08:56:15.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.578+0000 7fe4dabd6740 10 client.26476 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:15.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.579+0000 7fe4dabd6740 1 -- 172.21.15.98:0/1412543642 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558d12c68620 con 0x558d12f2e630 2024-05-09T08:56:15.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.579+0000 7fe4b97fa640 1 -- 172.21.15.98:0/1412543642 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4cc004030 con 0x558d12f2e630 2024-05-09T08:56:15.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.579+0000 7fe4b97fa640 1 -- 172.21.15.98:0/1412543642 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe4c802b990 con 0x558d12f2e630 2024-05-09T08:56:15.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.580+0000 7fe4dabd6740 10 client.26476 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:15.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.580+0000 7fe4dabd6740 10 client.26476 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:15.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.580+0000 7fe4dabd6740 20 client.26476 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:15.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.580+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/1412543642 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558d12f3e7b0 0x558d12f40bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.580+0000 7fe4dabd6740 4 client.26476 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:15.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.580+0000 7fe4dabd6740 1 -- 172.21.15.98:0/1412543642 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d12e95a70 con 0x558d12f3e7b0 2024-05-09T08:56:15.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.580+0000 7fe4d5b90640 1 --2- 172.21.15.98:0/1412543642 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558d12f3e7b0 0x558d12f40bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.584+0000 7fe4d5b90640 1 --2- 172.21.15.98:0/1412543642 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558d12f3e7b0 0x558d12f40bd0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b96a9ddaa79b7db0 server_cookie=826769ea1d19acf2 in_seq=0 out_seq=0 2024-05-09T08:56:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.584+0000 7fe4b97fa640 10 client.26476 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.590+0000 7fe4b97fa640 1 -- 172.21.15.98:0/1412543642 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d12e95a70 con 0x558d12f3e7b0 2024-05-09T08:56:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.590+0000 7fe4b97fa640 10 client.26476 handle_command_reply: tid=0 2024-05-09T08:56:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.590+0000 7fe4dabd6740 1 client.26476 shutdown 2024-05-09T08:56:15.591 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.591+0000 7fe4dabd6740 1 -- 172.21.15.98:0/1412543642 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f2e630 msgr2=0x558d12f2bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.591 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.591+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/1412543642 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f2e630 0x558d12f2bfc0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8004b90 tx=0x7fe4c8004a40 comp rx=0 tx=0).stop 2024-05-09T08:56:15.592 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.591+0000 7fe4dabd6740 1 -- 172.21.15.98:0/1412543642 shutdown_connections 2024-05-09T08:56:15.592 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.591+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/1412543642 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558d12f3e7b0 0x558d12f40bd0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.592 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.591+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/1412543642 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f2e630 0x558d12f2bfc0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.591+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/1412543642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23d70 0x558d12f2ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.591+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/1412543642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f23450 0x558d12f2b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.591+0000 7fe4dabd6740 1 -- 172.21.15.98:0/1412543642 >> 172.21.15.98:0/1412543642 conn(0x558d12f1f960 msgr2=0x558d12f256e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:15.593 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.593+0000 7fe4b97fa640 0 client.26476 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:15.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.595+0000 7fe4dabd6740 1 -- 172.21.15.98:0/1412543642 shutdown_connections 2024-05-09T08:56:15.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.597+0000 7fe4dabd6740 1 -- 172.21.15.98:0/1412543642 wait complete. 2024-05-09T08:56:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.598+0000 7fe4dabd6740 20 client.26476 trim_cache size 0 max 16384 2024-05-09T08:56:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.634+0000 7fe4dabd6740 1 Processor -- start 2024-05-09T08:56:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.634+0000 7fe4dabd6740 1 -- start start 2024-05-09T08:56:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.634+0000 7fe4dabd6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23450 0x558d12f3a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.634+0000 7fe4dabd6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23d70 0x558d12f2a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.634+0000 7fe4dabd6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f2e630 0x558d12f464c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.634+0000 7fe4dabd6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d01a4180 con 0x558d12f2e630 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.634+0000 7fe4dabd6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d01a5cc0 con 0x558d12f23d70 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.634+0000 7fe4dabd6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d01a4850 con 0x558d12f23450 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.634+0000 7fe4d6391640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f2e630 0x558d12f464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.635+0000 7fe4d6391640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f2e630 0x558d12f464c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39624/0 (socket says 172.21.15.98:39624) 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.635+0000 7fe4d6391640 1 -- 172.21.15.98:0/417307055 learned_addr learned my addr 172.21.15.98:0/417307055 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.635+0000 7fe4d5b90640 1 --2- 172.21.15.98:0/417307055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23450 0x558d12f3a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.635+0000 7fe4d6391640 1 -- 172.21.15.98:0/417307055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23450 msgr2=0x558d12f3a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.635+0000 7fe4d6391640 1 --2- 172.21.15.98:0/417307055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23450 0x558d12f3a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.635+0000 7fe4d6391640 1 -- 172.21.15.98:0/417307055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23d70 msgr2=0x558d12f2a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.635+0000 7fe4d6391640 1 --2- 172.21.15.98:0/417307055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23d70 0x558d12f2a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.635+0000 7fe4d6391640 1 -- 172.21.15.98:0/417307055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d12f23ba0 con 0x558d12f2e630 2024-05-09T08:56:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.635+0000 7fe4d6391640 1 --2- 172.21.15.98:0/417307055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f2e630 0x558d12f464c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800a010 tx=0x7fe4c8017f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2e080193697044 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.636+0000 7fe4bb7fe640 1 -- 172.21.15.98:0/417307055 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8031080 con 0x558d12f2e630 2024-05-09T08:56:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.636+0000 7fe4bb7fe640 1 -- 172.21.15.98:0/417307055 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4c802a350 con 0x558d12f2e630 2024-05-09T08:56:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.636+0000 7fe4bb7fe640 1 -- 172.21.15.98:0/417307055 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c802ab70 con 0x558d12f2e630 2024-05-09T08:56:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.637+0000 7fe4dabd6740 1 -- 172.21.15.98:0/417307055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f2e630 msgr2=0x558d12f464c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.637+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/417307055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f2e630 0x558d12f464c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800a010 tx=0x7fe4c8017f70 comp rx=0 tx=0).stop 2024-05-09T08:56:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.637+0000 7fe4dabd6740 1 -- 172.21.15.98:0/417307055 shutdown_connections 2024-05-09T08:56:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.637+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/417307055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f2e630 0x558d12f464c0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.638+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/417307055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23d70 0x558d12f2a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.638+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/417307055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23450 0x558d12f3a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.638+0000 7fe4dabd6740 1 -- 172.21.15.98:0/417307055 >> 172.21.15.98:0/417307055 conn(0x558d12f1f960 msgr2=0x558d12f29e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.638+0000 7fe4dabd6740 1 -- 172.21.15.98:0/417307055 shutdown_connections 2024-05-09T08:56:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.638+0000 7fe4dabd6740 1 -- 172.21.15.98:0/417307055 wait complete. 2024-05-09T08:56:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.639+0000 7fe4dabd6740 1 -- start start 2024-05-09T08:56:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.639+0000 7fe4dabd6740 1 Processor -- start 2024-05-09T08:56:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.639+0000 7fe4dabd6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23450 0x558d12f36af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.640+0000 7fe4dabd6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23d70 0x558d12f28c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.640+0000 7fe4dabd6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f2e630 0x558d12f29210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.640+0000 7fe4d5b90640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23450 0x558d12f36af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.640+0000 7fe4d5b90640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23450 0x558d12f36af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:41754/0 (socket says 172.21.15.98:41754) 2024-05-09T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.640+0000 7fe4d538f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23d70 0x558d12f28c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.640+0000 7fe4d538f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23d70 0x558d12f28c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:58620/0 (socket says 172.21.15.98:58620) 2024-05-09T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.640+0000 7fe4d5b90640 1 -- 172.21.15.98:0/411204958 learned_addr learned my addr 172.21.15.98:0/411204958 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.640+0000 7fe4dabd6740 1 -- 172.21.15.98:0/411204958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d01a55f0 con 0x558d12f2e630 2024-05-09T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.640+0000 7fe4dabd6740 1 -- 172.21.15.98:0/411204958 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d01a4f20 con 0x558d12f23450 2024-05-09T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.640+0000 7fe4dabd6740 1 -- 172.21.15.98:0/411204958 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d004f5f0 con 0x558d12f23d70 2024-05-09T08:56:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.641+0000 7fe4d538f640 1 -- 172.21.15.98:0/411204958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23450 msgr2=0x558d12f36af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.641+0000 7fe4d538f640 1 --2- 172.21.15.98:0/411204958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23450 0x558d12f36af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.641+0000 7fe4d538f640 1 -- 172.21.15.98:0/411204958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f2e630 msgr2=0x558d12f29210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.641+0000 7fe4d538f640 1 --2- 172.21.15.98:0/411204958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f2e630 0x558d12f29210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.641+0000 7fe4d538f640 1 -- 172.21.15.98:0/411204958 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d12f27a70 con 0x558d12f23d70 2024-05-09T08:56:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.641+0000 7fe4dabd6740 10 client.0 _ll_register_callbacks cb 0x7fe4c800ceb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.641+0000 7fe4d538f640 1 --2- 172.21.15.98:0/411204958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23d70 0x558d12f28c00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000be40 tx=0x7fe4c009b760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=718eab2754d46ca5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.641+0000 7fe4b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:56:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.642+0000 7fe4b97fa640 1 -- 172.21.15.98:0/411204958 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c009a570 con 0x558d12f23d70 2024-05-09T08:56:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.642+0000 7fe4b97fa640 1 -- 172.21.15.98:0/411204958 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fe4c0010870 con 0x558d12f23d70 2024-05-09T08:56:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.642+0000 7fe4dabd6740 1 -- 172.21.15.98:0/411204958 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4c802a350 con 0x558d12f23d70 2024-05-09T08:56:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.643+0000 7fe4b97fa640 1 -- 172.21.15.98:0/411204958 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe4c00a8050 con 0x558d12f23d70 2024-05-09T08:56:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.643+0000 7fe4dabd6740 10 client.26482 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.643+0000 7fe4dabd6740 1 -- 172.21.15.98:0/411204958 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({fsmap=35}) -- 0x558d12fa41f0 con 0x558d12f23d70 2024-05-09T08:56:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.643+0000 7fe4b97fa640 1 -- 172.21.15.98:0/411204958 <== mon.1 v2:172.21.15.142:3300/0 4 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe4c000f3c0 con 0x558d12f23d70 2024-05-09T08:56:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.643+0000 7fe4dabd6740 10 client.26482 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.644+0000 7fe4dabd6740 10 client.26482 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.644+0000 7fe4dabd6740 20 client.26482 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.644+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/411204958 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558d12fa4870 0x558d12fa6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.644+0000 7fe4dabd6740 4 client.26482 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:56:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.644+0000 7fe4d5b90640 1 --2- 172.21.15.98:0/411204958 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558d12fa4870 0x558d12fa6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.644+0000 7fe4dabd6740 1 -- 172.21.15.98:0/411204958 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fe4c802a350 con 0x558d12fa4870 2024-05-09T08:56:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.645+0000 7fe4d5b90640 1 --2- 172.21.15.98:0/411204958 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558d12fa4870 0x558d12fa6c90 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7afe385da48c477 server_cookie=f0238bae9ac47466 in_seq=0 out_seq=0 2024-05-09T08:56:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:15.645+0000 7fe4b97fa640 10 client.26482 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:15.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:15 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:56:15.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:15 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:56:15.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:15 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:56:15.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:15 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:56:15.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:15 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:56:15.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:15 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:56:15.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:15 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:56:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:15 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:56:15.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:15 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:56:16.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:16 smithi098 ceph-mon[30092]: pgmap v173: 129 pgs: 129 active+clean; 43 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 685 MiB/s wr, 2.27k op/s 2024-05-09T08:56:16.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:16 smithi142 ceph-mon[34625]: pgmap v173: 129 pgs: 129 active+clean; 43 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 685 MiB/s wr, 2.27k op/s 2024-05-09T08:56:16.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:16 smithi105 ceph-mon[34498]: pgmap v173: 129 pgs: 129 active+clean; 43 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 685 MiB/s wr, 2.27k op/s 2024-05-09T08:56:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.791+0000 7fe4b97fa640 1 -- 172.21.15.98:0/411204958 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fe4c802a350 con 0x558d12fa4870 2024-05-09T08:56:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.791+0000 7fe4b97fa640 10 client.26482 handle_command_reply: tid=0 2024-05-09T08:56:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.791+0000 7fe4dabd6740 1 client.26482 shutdown 2024-05-09T08:56:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.794+0000 7fe4dabd6740 1 -- 172.21.15.98:0/411204958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23d70 msgr2=0x558d12f28c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.794+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/411204958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23d70 0x558d12f28c00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000be40 tx=0x7fe4c009b760 comp rx=0 tx=0).stop 2024-05-09T08:56:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.794+0000 7fe4dabd6740 1 -- 172.21.15.98:0/411204958 shutdown_connections 2024-05-09T08:56:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.794+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/411204958 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x558d12fa4870 0x558d12fa6c90 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.794+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/411204958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d12f2e630 0x558d12f29210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.794+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/411204958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d12f23d70 0x558d12f28c00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.794+0000 7fe4dabd6740 1 --2- 172.21.15.98:0/411204958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d12f23450 0x558d12f36af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.794+0000 7fe4dabd6740 1 -- 172.21.15.98:0/411204958 >> 172.21.15.98:0/411204958 conn(0x558d12f1f960 msgr2=0x558d12f2a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.794+0000 7fe4b97fa640 0 client.26482 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.795+0000 7fe4dabd6740 1 -- 172.21.15.98:0/411204958 shutdown_connections 2024-05-09T08:56:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.795+0000 7fe4dabd6740 1 -- 172.21.15.98:0/411204958 wait complete. 2024-05-09T08:56:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.795+0000 7fe4dabd6740 20 client.26482 trim_cache size 0 max 16384 2024-05-09T08:56:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:56:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:56:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:56:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:56:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.800+0000 7fe4d7df4640 1 -- 172.21.15.98:0/2803133543 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe4bc0a2e20 msgr2=0x7fe4bc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.800+0000 7fe4d7df4640 1 --2- 172.21.15.98:0/2803133543 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe4bc0a2e20 0x7fe4bc0a52e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8009ea0 tx=0x7fe4c8009490 comp rx=0 tx=0).stop 2024-05-09T08:56:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.800+0000 7fe4d7df4640 1 -- 172.21.15.98:0/2803133543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4d01011d0 msgr2=0x7fe4d00728d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.800+0000 7fe4d7df4640 1 --2- 172.21.15.98:0/2803133543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4d01011d0 0x7fe4d00728d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000b480 tx=0x7fe4c000b940 comp rx=0 tx=0).stop 2024-05-09T08:56:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.801+0000 7fe4d7df4640 1 -- 172.21.15.98:0/2803133543 shutdown_connections 2024-05-09T08:56:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.801+0000 7fe4d7df4640 1 --2- 172.21.15.98:0/2803133543 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe4bc0a2e20 0x7fe4bc0a52e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.801+0000 7fe4d7df4640 1 --2- 172.21.15.98:0/2803133543 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d0105bd0 0x7fe4d010d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.801+0000 7fe4d7df4640 1 --2- 172.21.15.98:0/2803133543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4d01011d0 0x7fe4d00728d0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.801+0000 7fe4d7df4640 1 --2- 172.21.15.98:0/2803133543 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4d01008b0 0x7fe4d0072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.801+0000 7fe4d7df4640 1 -- 172.21.15.98:0/2803133543 >> 172.21.15.98:0/2803133543 conn(0x7fe4d00778b0 msgr2=0x7fe4d0103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.801+0000 7fe4d7df4640 1 -- 172.21.15.98:0/2803133543 shutdown_connections 2024-05-09T08:56:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:17.801+0000 7fe4d7df4640 1 -- 172.21.15.98:0/2803133543 wait complete. 2024-05-09T08:56:18.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:18 smithi098 ceph-mon[30092]: pgmap v174: 129 pgs: 129 active+clean; 43 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 678 MiB/s wr, 3.25k op/s 2024-05-09T08:56:18.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:18 smithi142 ceph-mon[34625]: pgmap v174: 129 pgs: 129 active+clean; 43 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 678 MiB/s wr, 3.25k op/s 2024-05-09T08:56:18.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:18 smithi105 ceph-mon[34498]: pgmap v174: 129 pgs: 129 active+clean; 43 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 678 MiB/s wr, 3.25k op/s 2024-05-09T08:56:20.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:20 smithi098 ceph-mon[30092]: pgmap v175: 129 pgs: 129 active+clean; 43 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 496 MiB/s wr, 3.02k op/s 2024-05-09T08:56:20.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:20 smithi142 ceph-mon[34625]: pgmap v175: 129 pgs: 129 active+clean; 43 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 496 MiB/s wr, 3.02k op/s 2024-05-09T08:56:20.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:20 smithi105 ceph-mon[34498]: pgmap v175: 129 pgs: 129 active+clean; 43 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 496 MiB/s wr, 3.02k op/s 2024-05-09T08:56:22.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:22 smithi098 ceph-mon[30092]: pgmap v176: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 691 MiB/s wr, 3.61k op/s 2024-05-09T08:56:22.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:22 smithi142 ceph-mon[34625]: pgmap v176: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 691 MiB/s wr, 3.61k op/s 2024-05-09T08:56:22.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:22 smithi105 ceph-mon[34498]: pgmap v176: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 691 MiB/s wr, 3.61k op/s 2024-05-09T08:56:24.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:24 smithi098 ceph-mon[30092]: pgmap v177: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 443 MiB/s wr, 2.40k op/s 2024-05-09T08:56:24.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:24 smithi142 ceph-mon[34625]: pgmap v177: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 443 MiB/s wr, 2.40k op/s 2024-05-09T08:56:24.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:24 smithi105 ceph-mon[34498]: pgmap v177: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 443 MiB/s wr, 2.40k op/s 2024-05-09T08:56:26.841 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:26 smithi098 ceph-mon[30092]: pgmap v178: 129 pgs: 129 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 509 MiB/s wr, 2.53k op/s 2024-05-09T08:56:26.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:26 smithi142 ceph-mon[34625]: pgmap v178: 129 pgs: 129 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 509 MiB/s wr, 2.53k op/s 2024-05-09T08:56:26.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:26 smithi105 ceph-mon[34498]: pgmap v178: 129 pgs: 129 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 509 MiB/s wr, 2.53k op/s 2024-05-09T08:56:28.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:28 smithi098 ceph-mon[30092]: pgmap v179: 129 pgs: 129 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 705 MiB/s wr, 2.29k op/s 2024-05-09T08:56:28.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:28 smithi142 ceph-mon[34625]: pgmap v179: 129 pgs: 129 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 705 MiB/s wr, 2.29k op/s 2024-05-09T08:56:28.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:28 smithi105 ceph-mon[34498]: pgmap v179: 129 pgs: 129 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 705 MiB/s wr, 2.29k op/s 2024-05-09T08:56:30.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:30 smithi098 ceph-mon[30092]: pgmap v180: 129 pgs: 129 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 564 MiB/s wr, 1.08k op/s 2024-05-09T08:56:30.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:30 smithi142 ceph-mon[34625]: pgmap v180: 129 pgs: 129 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 564 MiB/s wr, 1.08k op/s 2024-05-09T08:56:30.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:30 smithi105 ceph-mon[34498]: pgmap v180: 129 pgs: 129 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 564 MiB/s wr, 1.08k op/s 2024-05-09T08:56:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.085+0000 7f1984838640 1 -- 172.21.15.98:0/2431026866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f198011b010 msgr2=0x7f198011d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.085+0000 7f1984838640 1 --2- 172.21.15.98:0/2431026866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f198011b010 0x7f198011d400 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1970009900 tx=0x7f197002a4d0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.086+0000 7f1984838640 1 -- 172.21.15.98:0/2431026866 shutdown_connections 2024-05-09T08:56:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.086+0000 7f1984838640 1 --2- 172.21.15.98:0/2431026866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f198011b010 0x7f198011d400 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.086+0000 7f1984838640 1 --2- 172.21.15.98:0/2431026866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1980110400 0x7f1980108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.086+0000 7f1984838640 1 --2- 172.21.15.98:0/2431026866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1980110020 0x7f1980108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.086+0000 7f1984838640 1 -- 172.21.15.98:0/2431026866 >> 172.21.15.98:0/2431026866 conn(0x7f198006c900 msgr2=0x7f198006cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.088+0000 7f1984838640 1 -- 172.21.15.98:0/2431026866 shutdown_connections 2024-05-09T08:56:31.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.090+0000 7f1984838640 1 -- 172.21.15.98:0/2431026866 wait complete. 2024-05-09T08:56:31.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.091+0000 7f1984838640 1 Processor -- start 2024-05-09T08:56:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.091+0000 7f1984838640 1 -- start start 2024-05-09T08:56:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.092+0000 7f1984838640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1980110020 0x7f1980118490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.092+0000 7f1984838640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f198011b010 0x7f19801189d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.092+0000 7f1984838640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1980111590 0x7f1980111a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.092+0000 7f1984838640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1980120b80 con 0x7f198011b010 2024-05-09T08:56:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.092+0000 7f1984838640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1980120a00 con 0x7f1980111590 2024-05-09T08:56:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.092+0000 7f1984838640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1980120d00 con 0x7f1980110020 2024-05-09T08:56:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.093+0000 7f197ed9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1980111590 0x7f1980111a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.093+0000 7f197ed9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1980111590 0x7f1980111a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:42630/0 (socket says 172.21.15.98:42630) 2024-05-09T08:56:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.093+0000 7f197ed9d640 1 -- 172.21.15.98:0/313342168 learned_addr learned my addr 172.21.15.98:0/313342168 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.094+0000 7f197dd9b640 1 --2- 172.21.15.98:0/313342168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f198011b010 0x7f19801189d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.095+0000 7f197ed9d640 1 -- 172.21.15.98:0/313342168 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1980110020 msgr2=0x7f1980118490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.095+0000 7f197ed9d640 1 --2- 172.21.15.98:0/313342168 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1980110020 0x7f1980118490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.095 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.095+0000 7f197ed9d640 1 -- 172.21.15.98:0/313342168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f198011b010 msgr2=0x7f19801189d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:31.095 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.095+0000 7f197ed9d640 1 --2- 172.21.15.98:0/313342168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f198011b010 0x7f19801189d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.095 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.095+0000 7f197ed9d640 1 -- 172.21.15.98:0/313342168 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19801122d0 con 0x7f1980111590 2024-05-09T08:56:31.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.100+0000 7f197ed9d640 1 --2- 172.21.15.98:0/313342168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1980111590 0x7f1980111a10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1970004650 tx=0x7f1970003670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.101+0000 7f19777fe640 1 -- 172.21.15.98:0/313342168 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197003f030 con 0x7f1980111590 2024-05-09T08:56:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.102+0000 7f1984838640 1 -- 172.21.15.98:0/313342168 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1980116010 con 0x7f1980111590 2024-05-09T08:56:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.102+0000 7f1984838640 1 -- 172.21.15.98:0/313342168 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1980116500 con 0x7f1980111590 2024-05-09T08:56:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.102+0000 7f19777fe640 1 -- 172.21.15.98:0/313342168 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f197003b460 con 0x7f1980111590 2024-05-09T08:56:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.103+0000 7f19777fe640 1 -- 172.21.15.98:0/313342168 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970035070 con 0x7f1980111590 2024-05-09T08:56:31.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.110+0000 7f19777fe640 1 -- 172.21.15.98:0/313342168 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1970042030 con 0x7f1980111590 2024-05-09T08:56:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.115+0000 7f1987623740 1 Processor -- start 2024-05-09T08:56:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.116+0000 7f19777fe640 1 --2- 172.21.15.98:0/313342168 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f196c0a2e70 0x7f196c0a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.116+0000 7f19777fe640 1 -- 172.21.15.98:0/313342168 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f19700efdd0 con 0x7f1980111590 2024-05-09T08:56:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.118+0000 7f197dd9b640 1 --2- 172.21.15.98:0/313342168 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f196c0a2e70 0x7f196c0a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.121+0000 7f1987623740 1 -- start start 2024-05-09T08:56:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.121+0000 7f1987623740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55450 0x55b7e7e55830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.121+0000 7f1987623740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55d70 0x55b7e7e600f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.121+0000 7f1987623740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e62a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.121+0000 7f1987623740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b7e7dba340 con 0x55b7e7e55d70 2024-05-09T08:56:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.121+0000 7f1987623740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b7e7d62290 con 0x55b7e7e60630 2024-05-09T08:56:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.121+0000 7f1987623740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b7e7db6300 con 0x55b7e7e55450 2024-05-09T08:56:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.124+0000 7f197e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55450 0x55b7e7e55830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.124+0000 7f197e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55450 0x55b7e7e55830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:47416/0 (socket says 172.21.15.98:47416) 2024-05-09T08:56:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.124+0000 7f197e59c640 1 -- 172.21.15.98:0/126957189 learned_addr learned my addr 172.21.15.98:0/126957189 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.124+0000 7f197dd9b640 1 --2- 172.21.15.98:0/126957189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55d70 0x55b7e7e600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.125+0000 7f197dd9b640 1 --2- 172.21.15.98:0/313342168 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f196c0a2e70 0x7f196c0a5330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1968006fd0 tx=0x7f1968008040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.125+0000 7f197ed9d640 1 --2- 172.21.15.98:0/126957189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e62a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.125+0000 7f197e59c640 1 -- 172.21.15.98:0/126957189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 msgr2=0x55b7e7e62a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.125+0000 7f197e59c640 1 --2- 172.21.15.98:0/126957189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e62a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.125+0000 7f197e59c640 1 -- 172.21.15.98:0/126957189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55d70 msgr2=0x55b7e7e600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.125+0000 7f197e59c640 1 --2- 172.21.15.98:0/126957189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55d70 0x55b7e7e600f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.125+0000 7f197e59c640 1 -- 172.21.15.98:0/126957189 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7e7d868e0 con 0x55b7e7e55450 2024-05-09T08:56:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.126+0000 7f197e59c640 1 --2- 172.21.15.98:0/126957189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55450 0x55b7e7e55830 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1978009da0 tx=0x7f197800b420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=576ae40b77992e17 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.132+0000 7f1974ff9640 1 -- 172.21.15.98:0/126957189 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197800bd40 con 0x55b7e7e55450 2024-05-09T08:56:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.132+0000 7f1974ff9640 1 -- 172.21.15.98:0/126957189 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1978017070 con 0x55b7e7e55450 2024-05-09T08:56:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.133+0000 7f1987623740 1 -- 172.21.15.98:0/126957189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55450 msgr2=0x55b7e7e55830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.133+0000 7f1987623740 1 --2- 172.21.15.98:0/126957189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55450 0x55b7e7e55830 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1978009da0 tx=0x7f197800b420 comp rx=0 tx=0).stop 2024-05-09T08:56:31.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.141+0000 7f1987623740 1 -- 172.21.15.98:0/126957189 shutdown_connections 2024-05-09T08:56:31.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.142+0000 7f1987623740 1 --2- 172.21.15.98:0/126957189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e62a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.142+0000 7f1987623740 1 --2- 172.21.15.98:0/126957189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55d70 0x55b7e7e600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.142+0000 7f1987623740 1 --2- 172.21.15.98:0/126957189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55450 0x55b7e7e55830 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.142+0000 7f1987623740 1 -- 172.21.15.98:0/126957189 >> 172.21.15.98:0/126957189 conn(0x55b7e7e51960 msgr2=0x55b7e7e51d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:31.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.146+0000 7f1987623740 1 -- 172.21.15.98:0/126957189 shutdown_connections 2024-05-09T08:56:31.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.147+0000 7f1987623740 1 -- 172.21.15.98:0/126957189 wait complete. 2024-05-09T08:56:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.147+0000 7f1987623740 1 -- start start 2024-05-09T08:56:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.147+0000 7f1987623740 1 Processor -- start 2024-05-09T08:56:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.148+0000 7f1987623740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55450 0x55b7e7e5d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.148+0000 7f1987623740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55d70 0x55b7e7e5da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.148+0000 7f1987623740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e5df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.148+0000 7f1987623740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b7e7e63260 con 0x55b7e7e55d70 2024-05-09T08:56:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.148+0000 7f1987623740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b7e7d75790 con 0x55b7e7e60630 2024-05-09T08:56:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.148+0000 7f1987623740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b7e7e633e0 con 0x55b7e7e55450 2024-05-09T08:56:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.148+0000 7f1987623740 10 client.0 _ll_register_callbacks cb 0x55b7e7db0a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.149+0000 7f197e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55450 0x55b7e7e5d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.149+0000 7f197e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55450 0x55b7e7e5d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:47418/0 (socket says 172.21.15.98:47418) 2024-05-09T08:56:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.149+0000 7f197e59c640 1 -- 172.21.15.98:0/1128443096 learned_addr learned my addr 172.21.15.98:0/1128443096 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.150+0000 7f197dd9b640 1 --2- 172.21.15.98:0/1128443096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55d70 0x55b7e7e5da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.149+0000 7f197e59c640 1 -- 172.21.15.98:0/1128443096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 msgr2=0x55b7e7e5df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.150+0000 7f197e59c640 1 --2- 172.21.15.98:0/1128443096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e5df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.150+0000 7f197e59c640 1 -- 172.21.15.98:0/1128443096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55d70 msgr2=0x55b7e7e5da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.150+0000 7f197e59c640 1 --2- 172.21.15.98:0/1128443096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55d70 0x55b7e7e5da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.150+0000 7f197e59c640 1 -- 172.21.15.98:0/1128443096 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7e7ace210 con 0x55b7e7e55450 2024-05-09T08:56:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.150+0000 7f197dd9b640 1 --2- 172.21.15.98:0/1128443096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55d70 0x55b7e7e5da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:56:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.152+0000 7f197e59c640 1 --2- 172.21.15.98:0/1128443096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55450 0x55b7e7e5d4c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f197800bf70 tx=0x7f197800bfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=427e7c2b721b303b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.152+0000 7f1965ffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:56:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.152+0000 7f1965ffb640 1 -- 172.21.15.98:0/1128443096 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1978012840 con 0x55b7e7e55450 2024-05-09T08:56:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.153+0000 7f1965ffb640 1 -- 172.21.15.98:0/1128443096 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f19780129e0 con 0x55b7e7e55450 2024-05-09T08:56:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.153+0000 7f1987623740 1 -- 172.21.15.98:0/1128443096 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1978017070 con 0x55b7e7e55450 2024-05-09T08:56:31.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.153+0000 7f1965ffb640 1 -- 172.21.15.98:0/1128443096 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f19780070e0 con 0x55b7e7e55450 2024-05-09T08:56:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.153+0000 7f1987623740 10 client.26491 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.153+0000 7f1987623740 1 -- 172.21.15.98:0/1128443096 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b7e7b9a620 con 0x55b7e7e55450 2024-05-09T08:56:31.157 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.157+0000 7f1965ffb640 1 -- 172.21.15.98:0/1128443096 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1978017070 con 0x55b7e7e55450 2024-05-09T08:56:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.159+0000 7f1965ffb640 1 -- 172.21.15.98:0/1128443096 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f1978011070 con 0x55b7e7e55450 2024-05-09T08:56:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.159+0000 7f1987623740 10 client.26491 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.159+0000 7f1987623740 10 client.26491 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.159+0000 7f1987623740 20 client.26491 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.159+0000 7f1987623740 1 --2- 172.21.15.98:0/1128443096 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b7e7e70350 0x55b7e7e72740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.159+0000 7f1987623740 4 client.26491 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:31.159 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.159+0000 7f1987623740 1 -- 172.21.15.98:0/1128443096 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7e7dc7a70 con 0x55b7e7e70350 2024-05-09T08:56:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.165+0000 7f197dd9b640 1 --2- 172.21.15.98:0/1128443096 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b7e7e70350 0x55b7e7e72740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.172+0000 7f197dd9b640 1 --2- 172.21.15.98:0/1128443096 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b7e7e70350 0x55b7e7e72740 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4bdd284819c0219 server_cookie=31d9b748d1bd206 in_seq=0 out_seq=0 2024-05-09T08:56:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.172+0000 7f1965ffb640 10 client.26491 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:31.173 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.174+0000 7f1965ffb640 1 -- 172.21.15.98:0/1128443096 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b7e7dc7a70 con 0x55b7e7e70350 2024-05-09T08:56:31.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.174+0000 7f1965ffb640 10 client.26491 handle_command_reply: tid=0 2024-05-09T08:56:31.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.174+0000 7f1987623740 1 client.26491 shutdown 2024-05-09T08:56:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.178+0000 7f1987623740 1 -- 172.21.15.98:0/1128443096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55450 msgr2=0x55b7e7e5d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.178+0000 7f1987623740 1 --2- 172.21.15.98:0/1128443096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55450 0x55b7e7e5d4c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f197800bf70 tx=0x7f197800bfa0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.178+0000 7f1987623740 1 -- 172.21.15.98:0/1128443096 shutdown_connections 2024-05-09T08:56:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.178+0000 7f1987623740 1 --2- 172.21.15.98:0/1128443096 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b7e7e70350 0x55b7e7e72740 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.178+0000 7f1987623740 1 --2- 172.21.15.98:0/1128443096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e5df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.178+0000 7f1987623740 1 --2- 172.21.15.98:0/1128443096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55d70 0x55b7e7e5da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.178+0000 7f1987623740 1 --2- 172.21.15.98:0/1128443096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55450 0x55b7e7e5d4c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.178+0000 7f1987623740 1 -- 172.21.15.98:0/1128443096 >> 172.21.15.98:0/1128443096 conn(0x55b7e7e51960 msgr2=0x55b7e7e576e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.180+0000 7f1965ffb640 0 client.26491 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.182+0000 7f1987623740 1 -- 172.21.15.98:0/1128443096 shutdown_connections 2024-05-09T08:56:31.183 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.183+0000 7f1987623740 1 -- 172.21.15.98:0/1128443096 wait complete. 2024-05-09T08:56:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.192+0000 7f1987623740 20 client.26491 trim_cache size 0 max 16384 2024-05-09T08:56:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.229+0000 7f1987623740 1 Processor -- start 2024-05-09T08:56:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.229+0000 7f1987623740 1 -- start start 2024-05-09T08:56:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.230+0000 7f1987623740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55450 0x55b7e7e5fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.230+0000 7f1987623740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55d70 0x55b7e7e6c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.230+0000 7f1987623740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e5c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.230+0000 7f1987623740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f196000fe30 con 0x55b7e7e55450 2024-05-09T08:56:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.230+0000 7f1987623740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1960011970 con 0x55b7e7e60630 2024-05-09T08:56:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.230+0000 7f197dd9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55d70 0x55b7e7e6c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.231+0000 7f197ed9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e5c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.231+0000 7f197ed9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e5c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:42654/0 (socket says 172.21.15.98:42654) 2024-05-09T08:56:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.231+0000 7f197ed9d640 1 -- 172.21.15.98:0/807959572 learned_addr learned my addr 172.21.15.98:0/807959572 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.231+0000 7f1987623740 1 -- 172.21.15.98:0/807959572 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1960010500 con 0x55b7e7e55d70 2024-05-09T08:56:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.231+0000 7f197e59c640 1 --2- 172.21.15.98:0/807959572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55450 0x55b7e7e5fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.231+0000 7f197dd9b640 1 -- 172.21.15.98:0/807959572 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 msgr2=0x55b7e7e5c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.231+0000 7f197dd9b640 1 --2- 172.21.15.98:0/807959572 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e5c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.231+0000 7f197dd9b640 1 -- 172.21.15.98:0/807959572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55450 msgr2=0x55b7e7e5fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.231+0000 7f197dd9b640 1 --2- 172.21.15.98:0/807959572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55450 0x55b7e7e5fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.231+0000 7f197dd9b640 1 -- 172.21.15.98:0/807959572 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7e7e55ba0 con 0x55b7e7e55d70 2024-05-09T08:56:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.231+0000 7f197e59c640 1 --2- 172.21.15.98:0/807959572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55450 0x55b7e7e5fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:56:31.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.231+0000 7f197dd9b640 1 --2- 172.21.15.98:0/807959572 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55d70 0x55b7e7e6c1d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1968009690 tx=0x7f1968012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7de2a1a35c3b2d22 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:31.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.232+0000 7f1967fff640 1 -- 172.21.15.98:0/807959572 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1968027210 con 0x55b7e7e55d70 2024-05-09T08:56:31.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.232+0000 7f1967fff640 1 -- 172.21.15.98:0/807959572 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f196802d070 con 0x55b7e7e55d70 2024-05-09T08:56:31.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.232+0000 7f1967fff640 1 -- 172.21.15.98:0/807959572 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1968014a50 con 0x55b7e7e55d70 2024-05-09T08:56:31.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.234+0000 7f1987623740 1 -- 172.21.15.98:0/807959572 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55d70 msgr2=0x55b7e7e6c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:31.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.234+0000 7f1987623740 1 --2- 172.21.15.98:0/807959572 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55d70 0x55b7e7e6c1d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1968009690 tx=0x7f1968012000 comp rx=0 tx=0).stop 2024-05-09T08:56:31.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.237+0000 7f1987623740 1 -- 172.21.15.98:0/807959572 shutdown_connections 2024-05-09T08:56:31.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.237+0000 7f1987623740 1 --2- 172.21.15.98:0/807959572 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e5c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.237+0000 7f1987623740 1 --2- 172.21.15.98:0/807959572 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55d70 0x55b7e7e6c1d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.237+0000 7f1987623740 1 --2- 172.21.15.98:0/807959572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55450 0x55b7e7e5fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.237+0000 7f1987623740 1 -- 172.21.15.98:0/807959572 >> 172.21.15.98:0/807959572 conn(0x55b7e7e51960 msgr2=0x55b7e7e56e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:31.237 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.238+0000 7f1987623740 1 -- 172.21.15.98:0/807959572 shutdown_connections 2024-05-09T08:56:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.238+0000 7f1987623740 1 -- 172.21.15.98:0/807959572 wait complete. 2024-05-09T08:56:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.239+0000 7f1987623740 1 -- start start 2024-05-09T08:56:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.239+0000 7f1987623740 1 Processor -- start 2024-05-09T08:56:31.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.239+0000 7f1987623740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55450 0x55b7e7e5a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.239+0000 7f1987623740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55d70 0x55b7e7e5a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.239+0000 7f1987623740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e5ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.239+0000 7f1987623740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19600112a0 con 0x55b7e7e55450 2024-05-09T08:56:31.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.239+0000 7f1987623740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1960010bd0 con 0x55b7e7e60630 2024-05-09T08:56:31.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.239+0000 7f1987623740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f196000dec0 con 0x55b7e7e55d70 2024-05-09T08:56:31.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.239+0000 7f1987623740 10 client.0 _ll_register_callbacks cb 0x7f1968009580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:31.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.240+0000 7f197ed9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e5ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.240+0000 7f197ed9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e5ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:42664/0 (socket says 172.21.15.98:42664) 2024-05-09T08:56:31.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.240+0000 7f197ed9d640 1 -- 172.21.15.98:0/4077538791 learned_addr learned my addr 172.21.15.98:0/4077538791 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:31.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.240+0000 7f197dd9b640 1 --2- 172.21.15.98:0/4077538791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55d70 0x55b7e7e5a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.241+0000 7f197dd9b640 1 -- 172.21.15.98:0/4077538791 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 msgr2=0x55b7e7e5ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:31.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.241+0000 7f197dd9b640 1 --2- 172.21.15.98:0/4077538791 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e5ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.241+0000 7f197dd9b640 1 -- 172.21.15.98:0/4077538791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55450 msgr2=0x55b7e7e5a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:31.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.241+0000 7f197dd9b640 1 --2- 172.21.15.98:0/4077538791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55450 0x55b7e7e5a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:31.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.241+0000 7f197dd9b640 1 -- 172.21.15.98:0/4077538791 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7e7e66930 con 0x55b7e7e55d70 2024-05-09T08:56:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.243+0000 7f197dd9b640 1 --2- 172.21.15.98:0/4077538791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55d70 0x55b7e7e5a750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x55b7e7e60490 tx=0x7f1968009510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55b337ce7b5e2bb2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.243+0000 7f1965ffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:56:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.243+0000 7f1965ffb640 1 -- 172.21.15.98:0/4077538791 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1968037060 con 0x55b7e7e55d70 2024-05-09T08:56:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.244+0000 7f1965ffb640 1 -- 172.21.15.98:0/4077538791 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f1968014be0 con 0x55b7e7e55d70 2024-05-09T08:56:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.244+0000 7f1987623740 1 -- 172.21.15.98:0/4077538791 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f196802d070 con 0x55b7e7e55d70 2024-05-09T08:56:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.244+0000 7f1965ffb640 1 -- 172.21.15.98:0/4077538791 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1968014e00 con 0x55b7e7e55d70 2024-05-09T08:56:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.245+0000 7f1987623740 10 client.26497 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.245+0000 7f1987623740 1 -- 172.21.15.98:0/4077538791 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b7e7ed6990 con 0x55b7e7e55d70 2024-05-09T08:56:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.246+0000 7f1965ffb640 1 -- 172.21.15.98:0/4077538791 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f196802d070 con 0x55b7e7e55d70 2024-05-09T08:56:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.246+0000 7f1965ffb640 1 -- 172.21.15.98:0/4077538791 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f1968013070 con 0x55b7e7e55d70 2024-05-09T08:56:31.246 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.247+0000 7f1987623740 10 client.26497 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:31.246 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.247+0000 7f1987623740 10 client.26497 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:31.246 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.247+0000 7f1987623740 20 client.26497 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:31.246 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.247+0000 7f1987623740 1 --2- 172.21.15.98:0/4077538791 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b7e7ed6f30 0x55b7e7ed9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:31.246 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.247+0000 7f1987623740 4 client.26497 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:56:31.246 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.247+0000 7f1987623740 1 -- 172.21.15.98:0/4077538791 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b7e7dc1dc0 con 0x55b7e7ed6f30 2024-05-09T08:56:31.246 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.247+0000 7f197e59c640 1 --2- 172.21.15.98:0/4077538791 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b7e7ed6f30 0x55b7e7ed9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.248+0000 7f1965ffb640 10 client.26497 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:31.249+0000 7f197e59c640 1 --2- 172.21.15.98:0/4077538791 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b7e7ed6f30 0x55b7e7ed9350 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef0ec5e0298a9a19 server_cookie=cb890f477873626f in_seq=0 out_seq=0 2024-05-09T08:56:31.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:31 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:56:31.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:31 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:56:32.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:31 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:56:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.743+0000 7f1965ffb640 1 -- 172.21.15.98:0/4077538791 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b7e7dc1dc0 con 0x55b7e7ed6f30 2024-05-09T08:56:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.743+0000 7f1965ffb640 10 client.26497 handle_command_reply: tid=0 2024-05-09T08:56:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.743+0000 7f1987623740 1 client.26497 shutdown 2024-05-09T08:56:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.744+0000 7f1987623740 1 -- 172.21.15.98:0/4077538791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55d70 msgr2=0x55b7e7e5a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.744+0000 7f1987623740 1 --2- 172.21.15.98:0/4077538791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55d70 0x55b7e7e5a750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x55b7e7e60490 tx=0x7f1968009510 comp rx=0 tx=0).stop 2024-05-09T08:56:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.744+0000 7f1987623740 1 -- 172.21.15.98:0/4077538791 shutdown_connections 2024-05-09T08:56:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.744+0000 7f1987623740 1 --2- 172.21.15.98:0/4077538791 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b7e7ed6f30 0x55b7e7ed9350 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.744+0000 7f1987623740 1 --2- 172.21.15.98:0/4077538791 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b7e7e60630 0x55b7e7e5ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.744+0000 7f1987623740 1 --2- 172.21.15.98:0/4077538791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b7e7e55d70 0x55b7e7e5a750 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.744+0000 7f1987623740 1 --2- 172.21.15.98:0/4077538791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7e7e55450 0x55b7e7e5a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.744+0000 7f1987623740 1 -- 172.21.15.98:0/4077538791 >> 172.21.15.98:0/4077538791 conn(0x55b7e7e51960 msgr2=0x55b7e7e5c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.744+0000 7f1965ffb640 0 client.26497 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.744+0000 7f1987623740 1 -- 172.21.15.98:0/4077538791 shutdown_connections 2024-05-09T08:56:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.744+0000 7f1987623740 1 -- 172.21.15.98:0/4077538791 wait complete. 2024-05-09T08:56:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.744+0000 7f1987623740 20 client.26497 trim_cache size 0 max 16384 2024-05-09T08:56:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:56:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:56:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:56:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:56:32.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.747+0000 7f1984838640 1 -- 172.21.15.98:0/313342168 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f196c0a2e70 msgr2=0x7f196c0a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:32.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.747+0000 7f1984838640 1 --2- 172.21.15.98:0/313342168 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f196c0a2e70 0x7f196c0a5330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1968006fd0 tx=0x7f1968008040 comp rx=0 tx=0).stop 2024-05-09T08:56:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.747+0000 7f1984838640 1 -- 172.21.15.98:0/313342168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1980111590 msgr2=0x7f1980111a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.747+0000 7f1984838640 1 --2- 172.21.15.98:0/313342168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1980111590 0x7f1980111a10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1970004650 tx=0x7f1970003670 comp rx=0 tx=0).stop 2024-05-09T08:56:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.747+0000 7f1984838640 1 -- 172.21.15.98:0/313342168 shutdown_connections 2024-05-09T08:56:32.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.747+0000 7f1984838640 1 --2- 172.21.15.98:0/313342168 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f196c0a2e70 0x7f196c0a5330 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:32.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.747+0000 7f1984838640 1 --2- 172.21.15.98:0/313342168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1980111590 0x7f1980111a10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:32.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.748+0000 7f1984838640 1 --2- 172.21.15.98:0/313342168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f198011b010 0x7f19801189d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:32.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.748+0000 7f1984838640 1 --2- 172.21.15.98:0/313342168 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1980110020 0x7f1980118490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:32.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.748+0000 7f1984838640 1 -- 172.21.15.98:0/313342168 >> 172.21.15.98:0/313342168 conn(0x7f198006c900 msgr2=0x7f198010dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:32.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.748+0000 7f1984838640 1 -- 172.21.15.98:0/313342168 shutdown_connections 2024-05-09T08:56:32.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:32.748+0000 7f1984838640 1 -- 172.21.15.98:0/313342168 wait complete. 2024-05-09T08:56:32.886 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:32 smithi105 ceph-mon[34498]: pgmap v181: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 726 MiB/s wr, 1.48k op/s 2024-05-09T08:56:32.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:32 smithi142 ceph-mon[34625]: pgmap v181: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 726 MiB/s wr, 1.48k op/s 2024-05-09T08:56:33.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:32 smithi098 ceph-mon[30092]: pgmap v181: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 726 MiB/s wr, 1.48k op/s 2024-05-09T08:56:33.153 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7fae2232b700 time 2024-05-09T08:56:33.103736+0000 2024-05-09T08:56:33.153 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-05-09T08:56:33.153 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:56:33.153 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fae345463cd] 2024-05-09T08:56:33.153 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7fae34546593] 2024-05-09T08:56:33.153 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 3: (MDLog::_replay_thread()+0xd8) [0x5601c49540d8] 2024-05-09T08:56:33.153 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 4: (MDLog::ReplayThread::entry()+0x11) [0x5601c495dd01] 2024-05-09T08:56:33.153 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 5: /lib64/libpthread.so.0(+0x81ca) [0x7fae332c51ca] 2024-05-09T08:56:33.153 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 6: clone() 2024-05-09T08:56:33.154 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2024-05-09T08:56:33.103+0000 7fae2232b700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7fae2232b700 time 2024-05-09T08:56:33.103736+0000 2024-05-09T08:56:33.154 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-05-09T08:56:33.154 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2024-05-09T08:56:33.154 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:56:33.154 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fae345463cd] 2024-05-09T08:56:33.154 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7fae34546593] 2024-05-09T08:56:33.154 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 3: (MDLog::_replay_thread()+0xd8) [0x5601c49540d8] 2024-05-09T08:56:33.154 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 4: (MDLog::ReplayThread::entry()+0x11) [0x5601c495dd01] 2024-05-09T08:56:33.154 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 5: /lib64/libpthread.so.0(+0x81ca) [0x7fae332c51ca] 2024-05-09T08:56:33.154 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 6: clone() 2024-05-09T08:56:33.154 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2024-05-09T08:56:33.154 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: *** Caught signal (Aborted) ** 2024-05-09T08:56:33.154 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: in thread 7fae2232b700 thread_name:md_log_replay 2024-05-09T08:56:33.155 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:56:33.155 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fae332cfd20] 2024-05-09T08:56:33.155 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2: gsignal() 2024-05-09T08:56:33.155 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 3: abort() 2024-05-09T08:56:33.155 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fae34546427] 2024-05-09T08:56:33.156 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7fae34546593] 2024-05-09T08:56:33.156 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 6: (MDLog::_replay_thread()+0xd8) [0x5601c49540d8] 2024-05-09T08:56:33.156 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 7: (MDLog::ReplayThread::entry()+0x11) [0x5601c495dd01] 2024-05-09T08:56:33.156 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 8: /lib64/libpthread.so.0(+0x81ca) [0x7fae332c51ca] 2024-05-09T08:56:33.156 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 9: clone() 2024-05-09T08:56:33.156 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2024-05-09T08:56:33.104+0000 7fae2232b700 -1 *** Caught signal (Aborted) ** 2024-05-09T08:56:33.156 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: in thread 7fae2232b700 thread_name:md_log_replay 2024-05-09T08:56:33.156 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2024-05-09T08:56:33.156 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:56:33.156 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fae332cfd20] 2024-05-09T08:56:33.156 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2: gsignal() 2024-05-09T08:56:33.156 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 3: abort() 2024-05-09T08:56:33.156 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fae34546427] 2024-05-09T08:56:33.156 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7fae34546593] 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 6: (MDLog::_replay_thread()+0xd8) [0x5601c49540d8] 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 7: (MDLog::ReplayThread::entry()+0x11) [0x5601c495dd01] 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 8: /lib64/libpthread.so.0(+0x81ca) [0x7fae332c51ca] 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 9: clone() 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: -1> 2024-05-09T08:56:33.103+0000 7fae2232b700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7fae2232b700 time 2024-05-09T08:56:33.103736+0000 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fae345463cd] 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7fae34546593] 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 3: (MDLog::_replay_thread()+0xd8) [0x5601c49540d8] 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 4: (MDLog::ReplayThread::entry()+0x11) [0x5601c495dd01] 2024-05-09T08:56:33.157 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 5: /lib64/libpthread.so.0(+0x81ca) [0x7fae332c51ca] 2024-05-09T08:56:33.158 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 6: clone() 2024-05-09T08:56:33.158 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2024-05-09T08:56:33.158 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 0> 2024-05-09T08:56:33.104+0000 7fae2232b700 -1 *** Caught signal (Aborted) ** 2024-05-09T08:56:33.158 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: in thread 7fae2232b700 thread_name:md_log_replay 2024-05-09T08:56:33.158 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2024-05-09T08:56:33.158 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:56:33.158 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fae332cfd20] 2024-05-09T08:56:33.158 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2: gsignal() 2024-05-09T08:56:33.158 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 3: abort() 2024-05-09T08:56:33.158 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fae34546427] 2024-05-09T08:56:33.158 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7fae34546593] 2024-05-09T08:56:33.158 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 6: (MDLog::_replay_thread()+0xd8) [0x5601c49540d8] 2024-05-09T08:56:33.158 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 7: (MDLog::ReplayThread::entry()+0x11) [0x5601c495dd01] 2024-05-09T08:56:33.158 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 8: /lib64/libpthread.so.0(+0x81ca) [0x7fae332c51ca] 2024-05-09T08:56:33.159 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 9: clone() 2024-05-09T08:56:33.159 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T08:56:33.159 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2024-05-09T08:56:33.159 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: -1> 2024-05-09T08:56:33.103+0000 7fae2232b700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7fae2232b700 time 2024-05-09T08:56:33.103736+0000 2024-05-09T08:56:33.159 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2515-g96805dcf/rpm/el8/BUILD/ceph-19.0.0-2515-g96805dcf/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-05-09T08:56:33.159 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2024-05-09T08:56:33.159 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:56:33.159 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fae345463cd] 2024-05-09T08:56:33.159 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7fae34546593] 2024-05-09T08:56:33.159 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 3: (MDLog::_replay_thread()+0xd8) [0x5601c49540d8] 2024-05-09T08:56:33.159 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 4: (MDLog::ReplayThread::entry()+0x11) [0x5601c495dd01] 2024-05-09T08:56:33.159 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 5: /lib64/libpthread.so.0(+0x81ca) [0x7fae332c51ca] 2024-05-09T08:56:33.159 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 6: clone() 2024-05-09T08:56:33.159 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 0> 2024-05-09T08:56:33.104+0000 7fae2232b700 -1 *** Caught signal (Aborted) ** 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: in thread 7fae2232b700 thread_name:md_log_replay 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fae332cfd20] 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2: gsignal() 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 3: abort() 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fae34546427] 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d0593) [0x7fae34546593] 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 6: (MDLog::_replay_thread()+0xd8) [0x5601c49540d8] 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 7: (MDLog::ReplayThread::entry()+0x11) [0x5601c495dd01] 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 8: /lib64/libpthread.so.0(+0x81ca) [0x7fae332c51ca] 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 9: clone() 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T08:56:33.160 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68223]: 2024-05-09T08:56:33.487 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:33 smithi105 podman[68787]: 2024-05-09 08:56:33.274209755 +0000 UTC m=+0.101545827 container died aade97aca6951d5ec2fe8cf6455e2b33b26f8b775875e9b68219d81f99d42ce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:56:34.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:34 smithi142 ceph-mon[34625]: pgmap v182: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 602 MiB/s wr, 1.07k op/s 2024-05-09T08:56:34.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:34 smithi105 ceph-mon[34498]: pgmap v182: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 602 MiB/s wr, 1.07k op/s 2024-05-09T08:56:35.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:34 smithi098 ceph-mon[30092]: pgmap v182: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 602 MiB/s wr, 1.07k op/s 2024-05-09T08:56:36.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:36 smithi142 ceph-mon[34625]: pgmap v183: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 682 MiB/s wr, 1.15k op/s 2024-05-09T08:56:36.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:36 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:56:36.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:36 smithi105 ceph-mon[34498]: pgmap v183: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 682 MiB/s wr, 1.15k op/s 2024-05-09T08:56:36.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:36 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:56:37.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:36 smithi098 ceph-mon[30092]: pgmap v183: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 682 MiB/s wr, 1.15k op/s 2024-05-09T08:56:37.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:36 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:56:38.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:38 smithi142 ceph-mon[34625]: pgmap v184: 129 pgs: 129 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 858 MiB/s wr, 1.38k op/s 2024-05-09T08:56:38.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:38 smithi105 ceph-mon[34498]: pgmap v184: 129 pgs: 129 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 858 MiB/s wr, 1.38k op/s 2024-05-09T08:56:39.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:38 smithi098 ceph-mon[30092]: pgmap v184: 129 pgs: 129 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 858 MiB/s wr, 1.38k op/s 2024-05-09T08:56:40.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:40 smithi142 ceph-mon[34625]: pgmap v185: 129 pgs: 129 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 611 MiB/s wr, 1.22k op/s 2024-05-09T08:56:40.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:40 smithi105 ceph-mon[34498]: pgmap v185: 129 pgs: 129 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 611 MiB/s wr, 1.22k op/s 2024-05-09T08:56:41.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:40 smithi098 ceph-mon[30092]: pgmap v185: 129 pgs: 129 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 611 MiB/s wr, 1.22k op/s 2024-05-09T08:56:42.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:42 smithi142 ceph-mon[34625]: pgmap v186: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 783 MiB/s wr, 1.39k op/s 2024-05-09T08:56:42.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:42 smithi105 ceph-mon[34498]: pgmap v186: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 783 MiB/s wr, 1.39k op/s 2024-05-09T08:56:43.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:42 smithi098 ceph-mon[30092]: pgmap v186: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 783 MiB/s wr, 1.39k op/s 2024-05-09T08:56:43.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:43 smithi105 ceph-mon[34498]: pgmap v187: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 726 MiB/s wr, 993 op/s 2024-05-09T08:56:44.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:43 smithi098 ceph-mon[30092]: pgmap v187: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 726 MiB/s wr, 993 op/s 2024-05-09T08:56:44.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:43 smithi142 ceph-mon[34625]: pgmap v187: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 726 MiB/s wr, 993 op/s 2024-05-09T08:56:45.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:56:45.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:56:45.486 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:45 smithi105 podman[68787]: 2024-05-09 08:56:45.023729174 +0000 UTC m=+11.851065240 container remove aade97aca6951d5ec2fe8cf6455e2b33b26f8b775875e9b68219d81f99d42ce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2) 2024-05-09T08:56:45.487 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:45 smithi105 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-05-09T08:56:45.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.553+0000 7f1f9b410640 1 -- 172.21.15.98:0/3754488242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f94108830 msgr2=0x7f1f94108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:45.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.553+0000 7f1f9b410640 1 --2- 172.21.15.98:0/3754488242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f94108830 0x7f1f94108c90 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f1f90009900 tx=0x7f1f9002a4b0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.554+0000 7f1f9b410640 1 -- 172.21.15.98:0/3754488242 shutdown_connections 2024-05-09T08:56:45.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.554+0000 7f1f9b410640 1 --2- 172.21.15.98:0/3754488242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f941092b0 0x7f1f9411d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.554+0000 7f1f9b410640 1 --2- 172.21.15.98:0/3754488242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f94108830 0x7f1f94108c90 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.554+0000 7f1f9b410640 1 --2- 172.21.15.98:0/3754488242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f9410fed0 0x7f1f941102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.554+0000 7f1f9b410640 1 -- 172.21.15.98:0/3754488242 >> 172.21.15.98:0/3754488242 conn(0x7f1f9406c7b0 msgr2=0x7f1f9406cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:45.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.559+0000 7f1f9b410640 1 -- 172.21.15.98:0/3754488242 shutdown_connections 2024-05-09T08:56:45.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.560+0000 7f1f9b410640 1 -- 172.21.15.98:0/3754488242 wait complete. 2024-05-09T08:56:45.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.561+0000 7f1f9b410640 1 Processor -- start 2024-05-09T08:56:45.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.575+0000 7f1f9b410640 1 -- start start 2024-05-09T08:56:45.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.576+0000 7f1f9b410640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f94108830 0x7f1f9411c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.576+0000 7f1f9b410640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f941092b0 0x7f1f9411ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.576+0000 7f1f9b410640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f9410fed0 0x7f1f94116a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.576+0000 7f1f9b410640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1f94110e00 con 0x7f1f9410fed0 2024-05-09T08:56:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.576+0000 7f1f9b410640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1f94110c80 con 0x7f1f941092b0 2024-05-09T08:56:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.576+0000 7f1f9b410640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1f94110f80 con 0x7f1f94108830 2024-05-09T08:56:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.577+0000 7f1f99c0d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f941092b0 0x7f1f9411ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.577+0000 7f1f99c0d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f941092b0 0x7f1f9411ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54290/0 (socket says 172.21.15.98:54290) 2024-05-09T08:56:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.577+0000 7f1f99c0d640 1 -- 172.21.15.98:0/3091934191 learned_addr learned my addr 172.21.15.98:0/3091934191 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.577+0000 7f1f99c0d640 1 -- 172.21.15.98:0/3091934191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f94108830 msgr2=0x7f1f9411c970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:56:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.577+0000 7f1f99c0d640 1 --2- 172.21.15.98:0/3091934191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f94108830 0x7f1f9411c970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.577+0000 7f1f99c0d640 1 -- 172.21.15.98:0/3091934191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f9410fed0 msgr2=0x7f1f94116a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.577+0000 7f1f99c0d640 1 --2- 172.21.15.98:0/3091934191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f9410fed0 0x7f1f94116a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.577+0000 7f1f99c0d640 1 -- 172.21.15.98:0/3091934191 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f941172f0 con 0x7f1f941092b0 2024-05-09T08:56:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.578+0000 7f1f99c0d640 1 --2- 172.21.15.98:0/3091934191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f941092b0 0x7f1f9411ceb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1f9002aaa0 tx=0x7f1f90039520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.581+0000 7f1f8b7fe640 1 -- 172.21.15.98:0/3091934191 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f90040030 con 0x7f1f941092b0 2024-05-09T08:56:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.582+0000 7f1f8b7fe640 1 -- 172.21.15.98:0/3091934191 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f90030e60 con 0x7f1f941092b0 2024-05-09T08:56:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.583+0000 7f1f8b7fe640 1 -- 172.21.15.98:0/3091934191 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f90035070 con 0x7f1f941092b0 2024-05-09T08:56:45.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.583+0000 7f1f9b410640 1 -- 172.21.15.98:0/3091934191 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f94117580 con 0x7f1f941092b0 2024-05-09T08:56:45.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.583+0000 7f1f9b410640 1 -- 172.21.15.98:0/3091934191 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f941bc090 con 0x7f1f941092b0 2024-05-09T08:56:45.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.593+0000 7f1f8b7fe640 1 -- 172.21.15.98:0/3091934191 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1f90043030 con 0x7f1f941092b0 2024-05-09T08:56:45.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.596+0000 7f1f9f37d740 1 Processor -- start 2024-05-09T08:56:45.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.596+0000 7f1f8b7fe640 1 --2- 172.21.15.98:0/3091934191 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1f800a2ce0 0x7f1f800a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.597+0000 7f1f8b7fe640 1 -- 172.21.15.98:0/3091934191 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f1f900efe20 con 0x7f1f941092b0 2024-05-09T08:56:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.599+0000 7f1f9f37d740 1 -- start start 2024-05-09T08:56:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.599+0000 7f1f9f37d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b74430 0x55e9a5b74810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.599+0000 7f1f9f37d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b74d50 0x55e9a5b7eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.599+0000 7f1f9f37d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b7f420 0x55e9a5b81810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.599+0000 7f1f9f37d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e9a5ad9d00 con 0x55e9a5b74d50 2024-05-09T08:56:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.599+0000 7f1f9f37d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e9a5a814c0 con 0x55e9a5b74430 2024-05-09T08:56:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.599+0000 7f1f9f37d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e9a5ad5cc0 con 0x55e9a5b7f420 2024-05-09T08:56:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.600+0000 7f1f99c0d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b74d50 0x55e9a5b7eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.601+0000 7f1f99c0d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b74d50 0x55e9a5b7eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57300/0 (socket says 172.21.15.98:57300) 2024-05-09T08:56:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.601+0000 7f1f99c0d640 1 -- 172.21.15.98:0/1177203827 learned_addr learned my addr 172.21.15.98:0/1177203827 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.601+0000 7f1f9ac0f640 1 --2- 172.21.15.98:0/3091934191 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1f800a2ce0 0x7f1f800a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.601+0000 7f1f99c0d640 1 -- 172.21.15.98:0/1177203827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b7f420 msgr2=0x55e9a5b81810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:56:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.601+0000 7f1f99c0d640 1 --2- 172.21.15.98:0/1177203827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b7f420 0x55e9a5b81810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.601+0000 7f1f99c0d640 1 -- 172.21.15.98:0/1177203827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b74430 msgr2=0x55e9a5b74810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:56:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.601+0000 7f1f99c0d640 1 --2- 172.21.15.98:0/1177203827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b74430 0x55e9a5b74810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.601+0000 7f1f99c0d640 1 -- 172.21.15.98:0/1177203827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9a58b77d0 con 0x55e9a5b74d50 2024-05-09T08:56:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.602+0000 7f1f99c0d640 1 --2- 172.21.15.98:0/1177203827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b74d50 0x55e9a5b7eee0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f1f900c4660 tx=0x7f1f9003c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d46ceeb92662d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:45.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.607+0000 7f1f88ff9640 1 -- 172.21.15.98:0/1177203827 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f900bbf20 con 0x55e9a5b74d50 2024-05-09T08:56:45.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.607+0000 7f1f88ff9640 1 -- 172.21.15.98:0/1177203827 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f900c02d0 con 0x55e9a5b74d50 2024-05-09T08:56:45.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.608+0000 7f1f9f37d740 1 -- 172.21.15.98:0/1177203827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b74d50 msgr2=0x55e9a5b7eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:45.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.608+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/1177203827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b74d50 0x55e9a5b7eee0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f1f900c4660 tx=0x7f1f9003c470 comp rx=0 tx=0).stop 2024-05-09T08:56:45.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.608+0000 7f1f9ac0f640 1 --2- 172.21.15.98:0/3091934191 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1f800a2ce0 0x7f1f800a51a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x55e9a5b718e0 tx=0x7f1f84006c40 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:45.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.615+0000 7f1f9f37d740 1 -- 172.21.15.98:0/1177203827 shutdown_connections 2024-05-09T08:56:45.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.615+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/1177203827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b7f420 0x55e9a5b81810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.615+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/1177203827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b74d50 0x55e9a5b7eee0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.615+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/1177203827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b74430 0x55e9a5b74810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.615+0000 7f1f9f37d740 1 -- 172.21.15.98:0/1177203827 >> 172.21.15.98:0/1177203827 conn(0x55e9a5b70750 msgr2=0x55e9a5b70b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:45.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.619+0000 7f1f9f37d740 1 -- 172.21.15.98:0/1177203827 shutdown_connections 2024-05-09T08:56:45.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.620+0000 7f1f9f37d740 1 -- 172.21.15.98:0/1177203827 wait complete. 2024-05-09T08:56:45.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.620+0000 7f1f9f37d740 1 -- start start 2024-05-09T08:56:45.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.620+0000 7f1f9f37d740 1 Processor -- start 2024-05-09T08:56:45.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.621+0000 7f1f9f37d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b74430 0x55e9a5b7c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.621+0000 7f1f9f37d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b74d50 0x55e9a5b7c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.621+0000 7f1f9f37d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b7f420 0x55e9a5b7cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.621+0000 7f1f9f37d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e9a5b81fd0 con 0x55e9a5b7f420 2024-05-09T08:56:45.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.621+0000 7f1f9f37d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e9a5a94b90 con 0x55e9a5b74d50 2024-05-09T08:56:45.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.621+0000 7f1f9f37d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e9a5b82150 con 0x55e9a5b74430 2024-05-09T08:56:45.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.621+0000 7f1f9f37d740 10 client.0 _ll_register_callbacks cb 0x55e9a5ad0da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:45.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.621+0000 7f1f9ac0f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b7f420 0x55e9a5b7cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:45.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.621+0000 7f1f9ac0f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b7f420 0x55e9a5b7cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57304/0 (socket says 172.21.15.98:57304) 2024-05-09T08:56:45.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.621+0000 7f1f9ac0f640 1 -- 172.21.15.98:0/3917091300 learned_addr learned my addr 172.21.15.98:0/3917091300 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:45.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.622+0000 7f1f9a40e640 1 --2- 172.21.15.98:0/3917091300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b74430 0x55e9a5b7c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:45.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.625+0000 7f1f9a40e640 1 -- 172.21.15.98:0/3917091300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b74d50 msgr2=0x55e9a5b7c830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:56:45.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.625+0000 7f1f9a40e640 1 --2- 172.21.15.98:0/3917091300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b74d50 0x55e9a5b7c830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.625+0000 7f1f9a40e640 1 -- 172.21.15.98:0/3917091300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b7f420 msgr2=0x55e9a5b7cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:45.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.625+0000 7f1f9a40e640 1 --2- 172.21.15.98:0/3917091300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b7f420 0x55e9a5b7cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.625+0000 7f1f9a40e640 1 -- 172.21.15.98:0/3917091300 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9a57ed3f0 con 0x55e9a5b74430 2024-05-09T08:56:45.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.625+0000 7f1f9a40e640 1 --2- 172.21.15.98:0/3917091300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b74430 0x55e9a5b7c2f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c00e5f0 tx=0x7f1f8c00eab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc46bedea0231e5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:45.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.630+0000 7f1f7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:56:45.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.630+0000 7f1f7dffb640 1 -- 172.21.15.98:0/3917091300 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c004d20 con 0x55e9a5b74430 2024-05-09T08:56:45.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.631+0000 7f1f9f37d740 1 -- 172.21.15.98:0/3917091300 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f900c02d0 con 0x55e9a5b74430 2024-05-09T08:56:45.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.631+0000 7f1f7dffb640 1 -- 172.21.15.98:0/3917091300 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f1f8c0106a0 con 0x55e9a5b74430 2024-05-09T08:56:45.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.631+0000 7f1f7dffb640 1 -- 172.21.15.98:0/3917091300 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1f8c022050 con 0x55e9a5b74430 2024-05-09T08:56:45.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.632+0000 7f1f9f37d740 10 client.26500 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:45.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.632+0000 7f1f9f37d740 1 -- 172.21.15.98:0/3917091300 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e9a57ad0f0 con 0x55e9a5b74430 2024-05-09T08:56:45.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.633+0000 7f1f7dffb640 1 -- 172.21.15.98:0/3917091300 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f900c02d0 con 0x55e9a5b74430 2024-05-09T08:56:45.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.634+0000 7f1f7dffb640 1 -- 172.21.15.98:0/3917091300 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f1f8c00f040 con 0x55e9a5b74430 2024-05-09T08:56:45.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.634+0000 7f1f9f37d740 10 client.26500 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:45.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.634+0000 7f1f9f37d740 10 client.26500 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:45.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.634+0000 7f1f9f37d740 20 client.26500 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:45.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.634+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/3917091300 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e9a5b8f970 0x55e9a5b91e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.634+0000 7f1f9f37d740 4 client.26500 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:45.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.634+0000 7f1f9f37d740 1 -- 172.21.15.98:0/3917091300 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9a5ae7430 con 0x55e9a5b8f970 2024-05-09T08:56:45.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.634+0000 7f1f99c0d640 1 --2- 172.21.15.98:0/3917091300 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e9a5b8f970 0x55e9a5b91e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:45.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.636+0000 7f1f7dffb640 10 client.26500 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:45.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.637+0000 7f1f99c0d640 1 --2- 172.21.15.98:0/3917091300 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e9a5b8f970 0x55e9a5b91e10 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8508eccb05c2a7af server_cookie=56efba1d4d8a44cd in_seq=0 out_seq=0 2024-05-09T08:56:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.648+0000 7f1f7dffb640 1 -- 172.21.15.98:0/3917091300 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e9a5ae7430 con 0x55e9a5b8f970 2024-05-09T08:56:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.648+0000 7f1f7dffb640 10 client.26500 handle_command_reply: tid=0 2024-05-09T08:56:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.648+0000 7f1f9f37d740 1 client.26500 shutdown 2024-05-09T08:56:45.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.655+0000 7f1f9f37d740 1 -- 172.21.15.98:0/3917091300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b74430 msgr2=0x55e9a5b7c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:45.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.655+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/3917091300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b74430 0x55e9a5b7c2f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c00e5f0 tx=0x7f1f8c00eab0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.657+0000 7f1f9f37d740 1 -- 172.21.15.98:0/3917091300 shutdown_connections 2024-05-09T08:56:45.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.657+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/3917091300 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e9a5b8f970 0x55e9a5b91e10 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.657+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/3917091300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b7f420 0x55e9a5b7cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.657+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/3917091300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b74d50 0x55e9a5b7c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.657+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/3917091300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b74430 0x55e9a5b7c2f0 secure :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c00e5f0 tx=0x7f1f8c00eab0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.657+0000 7f1f9f37d740 1 -- 172.21.15.98:0/3917091300 >> 172.21.15.98:0/3917091300 conn(0x55e9a5b70750 msgr2=0x55e9a5b6fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:45.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.658+0000 7f1f7dffb640 0 client.26500 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:45.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.659+0000 7f1f9f37d740 1 -- 172.21.15.98:0/3917091300 shutdown_connections 2024-05-09T08:56:45.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.662+0000 7f1f9f37d740 1 -- 172.21.15.98:0/3917091300 wait complete. 2024-05-09T08:56:45.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.665+0000 7f1f9f37d740 20 client.26500 trim_cache size 0 max 16384 2024-05-09T08:56:45.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.726+0000 7f1f9f37d740 1 Processor -- start 2024-05-09T08:56:45.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.726+0000 7f1f9f37d740 1 -- start start 2024-05-09T08:56:45.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.726+0000 7f1f9f37d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b74d50 0x55e9a5b8ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.726+0000 7f1f9f37d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b7f420 0x55e9a5b7c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.726+0000 7f1f9f37d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b7eb30 0x55e9a5b976e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.726+0000 7f1f9f37d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1f941c5fd0 con 0x55e9a5b74d50 2024-05-09T08:56:45.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.726+0000 7f1f9f37d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1f941c7b10 con 0x55e9a5b7f420 2024-05-09T08:56:45.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.726+0000 7f1f9f37d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1f941c66a0 con 0x55e9a5b7eb30 2024-05-09T08:56:45.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.727+0000 7f1f99c0d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b7f420 0x55e9a5b7c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:45.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.727+0000 7f1f9ac0f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b7eb30 0x55e9a5b976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:45.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.727+0000 7f1f99c0d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b7f420 0x55e9a5b7c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54324/0 (socket says 172.21.15.98:54324) 2024-05-09T08:56:45.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.727+0000 7f1f99c0d640 1 -- 172.21.15.98:0/1629980862 learned_addr learned my addr 172.21.15.98:0/1629980862 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:45.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.729+0000 7f1f9a40e640 1 --2- 172.21.15.98:0/1629980862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b74d50 0x55e9a5b8ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:45.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.729+0000 7f1f99c0d640 1 -- 172.21.15.98:0/1629980862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b7eb30 msgr2=0x55e9a5b976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:45.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.729+0000 7f1f99c0d640 1 --2- 172.21.15.98:0/1629980862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b7eb30 0x55e9a5b976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.729+0000 7f1f99c0d640 1 -- 172.21.15.98:0/1629980862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b74d50 msgr2=0x55e9a5b8ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:45.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.729+0000 7f1f99c0d640 1 --2- 172.21.15.98:0/1629980862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b74d50 0x55e9a5b8ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.729+0000 7f1f99c0d640 1 -- 172.21.15.98:0/1629980862 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9a5b8e0a0 con 0x55e9a5b7f420 2024-05-09T08:56:45.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.730+0000 7f1f99c0d640 1 --2- 172.21.15.98:0/1629980862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b7f420 0x55e9a5b7c9c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1f900bbeb0 tx=0x7f1f9002a4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8898edeb3df65e96 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:45.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.731+0000 7f1f7ffff640 1 -- 172.21.15.98:0/1629980862 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f900bbf20 con 0x55e9a5b7f420 2024-05-09T08:56:45.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.732+0000 7f1f7ffff640 1 -- 172.21.15.98:0/1629980862 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f900c4b90 con 0x55e9a5b7f420 2024-05-09T08:56:45.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.733+0000 7f1f9f37d740 1 -- 172.21.15.98:0/1629980862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b7f420 msgr2=0x55e9a5b7c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:45.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.733+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/1629980862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b7f420 0x55e9a5b7c9c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1f900bbeb0 tx=0x7f1f9002a4b0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.736+0000 7f1f9f37d740 1 -- 172.21.15.98:0/1629980862 shutdown_connections 2024-05-09T08:56:45.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.736+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/1629980862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b7eb30 0x55e9a5b976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.736+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/1629980862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b7f420 0x55e9a5b7c9c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.736+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/1629980862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b74d50 0x55e9a5b8ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.736+0000 7f1f9f37d740 1 -- 172.21.15.98:0/1629980862 >> 172.21.15.98:0/1629980862 conn(0x55e9a5b70750 msgr2=0x55e9a5b7c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:45.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.739+0000 7f1f9f37d740 1 -- 172.21.15.98:0/1629980862 shutdown_connections 2024-05-09T08:56:45.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.740+0000 7f1f9f37d740 1 -- 172.21.15.98:0/1629980862 wait complete. 2024-05-09T08:56:45.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.741+0000 7f1f9f37d740 1 -- start start 2024-05-09T08:56:45.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.741+0000 7f1f9f37d740 1 Processor -- start 2024-05-09T08:56:45.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.744+0000 7f1f9f37d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b74d50 0x55e9a5b78e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.744+0000 7f1f9f37d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b7eb30 0x55e9a5b79390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.744+0000 7f1f9f37d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b7f420 0x55e9a5b798d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.744+0000 7f1f9f37d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1f941c7440 con 0x55e9a5b7eb30 2024-05-09T08:56:45.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.744+0000 7f1f9f37d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1f941c6d70 con 0x55e9a5b7f420 2024-05-09T08:56:45.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.744+0000 7f1f9f37d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1f941c3fb0 con 0x55e9a5b74d50 2024-05-09T08:56:45.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.744+0000 7f1f9f37d740 10 client.0 _ll_register_callbacks cb 0x7f1f8c0094a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:45.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.744+0000 7f1f99c0d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b7eb30 0x55e9a5b79390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:45.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.744+0000 7f1f99c0d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b7eb30 0x55e9a5b79390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57324/0 (socket says 172.21.15.98:57324) 2024-05-09T08:56:45.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.744+0000 7f1f99c0d640 1 -- 172.21.15.98:0/2296021139 learned_addr learned my addr 172.21.15.98:0/2296021139 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:45.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.745+0000 7f1f9ac0f640 1 --2- 172.21.15.98:0/2296021139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b7f420 0x55e9a5b798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:45.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.745+0000 7f1f9a40e640 1 --2- 172.21.15.98:0/2296021139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b74d50 0x55e9a5b78e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:45.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.749+0000 7f1f99c0d640 1 -- 172.21.15.98:0/2296021139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b74d50 msgr2=0x55e9a5b78e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:45.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.749+0000 7f1f99c0d640 1 --2- 172.21.15.98:0/2296021139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b74d50 0x55e9a5b78e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.749+0000 7f1f99c0d640 1 -- 172.21.15.98:0/2296021139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b7f420 msgr2=0x55e9a5b798d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:45.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.749+0000 7f1f99c0d640 1 --2- 172.21.15.98:0/2296021139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b7f420 0x55e9a5b798d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.749+0000 7f1f99c0d640 1 -- 172.21.15.98:0/2296021139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9a5b77ef0 con 0x55e9a5b7eb30 2024-05-09T08:56:45.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.750+0000 7f1f99c0d640 1 --2- 172.21.15.98:0/2296021139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b7eb30 0x55e9a5b79390 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1f900b88f0 tx=0x7f1f90005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b996bfffc9d64b8f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:45.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.751+0000 7f1f7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:56:45.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.752+0000 7f1f7dffb640 1 -- 172.21.15.98:0/2296021139 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f900cb020 con 0x55e9a5b7eb30 2024-05-09T08:56:45.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.752+0000 7f1f9f37d740 1 -- 172.21.15.98:0/2296021139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f900c4b90 con 0x55e9a5b7eb30 2024-05-09T08:56:45.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.752+0000 7f1f7dffb640 1 -- 172.21.15.98:0/2296021139 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f1f900c48a0 con 0x55e9a5b7eb30 2024-05-09T08:56:45.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.754+0000 7f1f7dffb640 1 -- 172.21.15.98:0/2296021139 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1f900ed5d0 con 0x55e9a5b7eb30 2024-05-09T08:56:45.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.755+0000 7f1f9f37d740 10 client.17214 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:45.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.755+0000 7f1f9f37d740 1 -- 172.21.15.98:0/2296021139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e9a5bf5f30 con 0x55e9a5b7eb30 2024-05-09T08:56:45.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.758+0000 7f1f7dffb640 1 -- 172.21.15.98:0/2296021139 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f900c4b90 con 0x55e9a5b7eb30 2024-05-09T08:56:45.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.758+0000 7f1f7dffb640 1 -- 172.21.15.98:0/2296021139 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f1f900bf070 con 0x55e9a5b7eb30 2024-05-09T08:56:45.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.759+0000 7f1f9f37d740 10 client.17214 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:45.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.759+0000 7f1f9f37d740 10 client.17214 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:45.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.759+0000 7f1f9f37d740 20 client.17214 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:45.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.759+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/2296021139 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e9a5bf6180 0x55e9a5bf8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:45.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.759+0000 7f1f9f37d740 4 client.17214 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:56:45.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.759+0000 7f1f9f37d740 1 -- 172.21.15.98:0/2296021139 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e9a5ae1780 con 0x55e9a5bf6180 2024-05-09T08:56:45.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.763+0000 7f1f9a40e640 1 --2- 172.21.15.98:0/2296021139 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e9a5bf6180 0x55e9a5bf8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.775+0000 7f1f9a40e640 1 --2- 172.21.15.98:0/2296021139 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e9a5bf6180 0x55e9a5bf8570 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c46bb3c3e4225afa server_cookie=2076fa6c448a4bd1 in_seq=0 out_seq=0 2024-05-09T08:56:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.775+0000 7f1f7dffb640 10 client.17214 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.781+0000 7f1f7dffb640 1 -- 172.21.15.98:0/2296021139 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e9a5ae1780 con 0x55e9a5bf6180 2024-05-09T08:56:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.781+0000 7f1f7dffb640 10 client.17214 handle_command_reply: tid=0 2024-05-09T08:56:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.781+0000 7f1f9f37d740 1 client.17214 shutdown 2024-05-09T08:56:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.789+0000 7f1f9f37d740 1 -- 172.21.15.98:0/2296021139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b7eb30 msgr2=0x55e9a5b79390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.789+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/2296021139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b7eb30 0x55e9a5b79390 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1f900b88f0 tx=0x7f1f90005d50 comp rx=0 tx=0).stop 2024-05-09T08:56:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.791+0000 7f1f9f37d740 1 -- 172.21.15.98:0/2296021139 shutdown_connections 2024-05-09T08:56:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.791+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/2296021139 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e9a5bf6180 0x55e9a5bf8570 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.791+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/2296021139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a5b7f420 0x55e9a5b798d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.791+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/2296021139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e9a5b7eb30 0x55e9a5b79390 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.791+0000 7f1f9f37d740 1 --2- 172.21.15.98:0/2296021139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e9a5b74d50 0x55e9a5b78e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.791+0000 7f1f9f37d740 1 -- 172.21.15.98:0/2296021139 >> 172.21.15.98:0/2296021139 conn(0x55e9a5b70750 msgr2=0x55e9a5b7e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.791+0000 7f1f7dffb640 0 client.17214 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:45.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.796+0000 7f1f9f37d740 1 -- 172.21.15.98:0/2296021139 shutdown_connections 2024-05-09T08:56:45.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.798+0000 7f1f9f37d740 1 -- 172.21.15.98:0/2296021139 wait complete. 2024-05-09T08:56:45.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.803+0000 7f1f9f37d740 20 client.17214 trim_cache size 0 max 16384 2024-05-09T08:56:45.807 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:56:45.807 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:56:45.807 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:56:45.807 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:56:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.812+0000 7f1f9b410640 1 -- 172.21.15.98:0/3091934191 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1f800a2ce0 msgr2=0x7f1f800a51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.812+0000 7f1f9b410640 1 --2- 172.21.15.98:0/3091934191 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1f800a2ce0 0x7f1f800a51a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x55e9a5b718e0 tx=0x7f1f84006c40 comp rx=0 tx=0).stop 2024-05-09T08:56:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.812+0000 7f1f9b410640 1 -- 172.21.15.98:0/3091934191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f941092b0 msgr2=0x7f1f9411ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.812+0000 7f1f9b410640 1 --2- 172.21.15.98:0/3091934191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f941092b0 0x7f1f9411ceb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1f9002aaa0 tx=0x7f1f90039520 comp rx=0 tx=0).stop 2024-05-09T08:56:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.813+0000 7f1f9b410640 1 -- 172.21.15.98:0/3091934191 shutdown_connections 2024-05-09T08:56:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.813+0000 7f1f9b410640 1 --2- 172.21.15.98:0/3091934191 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1f800a2ce0 0x7f1f800a51a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.813+0000 7f1f9b410640 1 --2- 172.21.15.98:0/3091934191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f9410fed0 0x7f1f94116a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.813+0000 7f1f9b410640 1 --2- 172.21.15.98:0/3091934191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f941092b0 0x7f1f9411ceb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.813+0000 7f1f9b410640 1 --2- 172.21.15.98:0/3091934191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f94108830 0x7f1f9411c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.813+0000 7f1f9b410640 1 -- 172.21.15.98:0/3091934191 >> 172.21.15.98:0/3091934191 conn(0x7f1f9406c7b0 msgr2=0x7f1f94070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:45.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.819+0000 7f1f9b410640 1 -- 172.21.15.98:0/3091934191 shutdown_connections 2024-05-09T08:56:45.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:45.819+0000 7f1f9b410640 1 -- 172.21.15.98:0/3091934191 wait complete. 2024-05-09T08:56:45.853 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:56:45.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:56:45.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:45 smithi105 ceph-mon[34498]: pgmap v188: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 703 MiB/s wr, 831 op/s 2024-05-09T08:56:45.987 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:45 smithi105 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-05-09T08:56:45.987 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:45 smithi105 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.e.service: Consumed 20.152s CPU time. 2024-05-09T08:56:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:45.993+0000 7ff8878e7640 1 -- 172.21.15.98:0/931132410 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff880110400 msgr2=0x7ff880108d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:45.993+0000 7ff8878e7640 1 --2- 172.21.15.98:0/931132410 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff880110400 0x7ff880108d70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff870009860 tx=0x7ff87002a3f0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:45.997+0000 7ff8878e7640 1 -- 172.21.15.98:0/931132410 shutdown_connections 2024-05-09T08:56:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:45.998+0000 7ff8878e7640 1 --2- 172.21.15.98:0/931132410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff88011b010 0x7ff88011d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:45.998+0000 7ff8878e7640 1 --2- 172.21.15.98:0/931132410 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff880110400 0x7ff880108d70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:45.998+0000 7ff8878e7640 1 --2- 172.21.15.98:0/931132410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff880110020 0x7ff880108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:45.998+0000 7ff8878e7640 1 -- 172.21.15.98:0/931132410 >> 172.21.15.98:0/931132410 conn(0x7ff88006c900 msgr2=0x7ff88006cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:46.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.004+0000 7ff8878e7640 1 -- 172.21.15.98:0/931132410 shutdown_connections 2024-05-09T08:56:46.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.005+0000 7ff8878e7640 1 -- 172.21.15.98:0/931132410 wait complete. 2024-05-09T08:56:46.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.006+0000 7ff8878e7640 1 Processor -- start 2024-05-09T08:56:46.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.006+0000 7ff8878e7640 1 -- start start 2024-05-09T08:56:46.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.007+0000 7ff8878e7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff880110020 0x7ff88011a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.007+0000 7ff8878e7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff88011b010 0x7ff8801167c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.007+0000 7ff8878e7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff880116d00 0x7ff8801171b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.007+0000 7ff8878e7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801131d0 con 0x7ff88011b010 2024-05-09T08:56:46.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.007+0000 7ff8878e7640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff880113050 con 0x7ff880116d00 2024-05-09T08:56:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.007+0000 7ff8878e7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff880113350 con 0x7ff880110020 2024-05-09T08:56:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.007+0000 7ff884e82640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff88011b010 0x7ff8801167c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.007+0000 7ff884e82640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff88011b010 0x7ff8801167c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57344/0 (socket says 172.21.15.98:57344) 2024-05-09T08:56:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.007+0000 7ff884e82640 1 -- 172.21.15.98:0/4002565673 learned_addr learned my addr 172.21.15.98:0/4002565673 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.007+0000 7ff885e84640 1 --2- 172.21.15.98:0/4002565673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff880116d00 0x7ff8801171b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.008+0000 7ff884e82640 1 -- 172.21.15.98:0/4002565673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff880110020 msgr2=0x7ff88011a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.008+0000 7ff884e82640 1 --2- 172.21.15.98:0/4002565673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff880110020 0x7ff88011a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.008+0000 7ff884e82640 1 -- 172.21.15.98:0/4002565673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff880116d00 msgr2=0x7ff8801171b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.008+0000 7ff884e82640 1 --2- 172.21.15.98:0/4002565673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff880116d00 0x7ff8801171b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.008+0000 7ff884e82640 1 -- 172.21.15.98:0/4002565673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8801cce10 con 0x7ff88011b010 2024-05-09T08:56:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.008+0000 7ff884e82640 1 --2- 172.21.15.98:0/4002565673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff88011b010 0x7ff8801167c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7ff870000c00 tx=0x7ff8700309c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.009+0000 7ff8767fc640 1 -- 172.21.15.98:0/4002565673 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870040070 con 0x7ff88011b010 2024-05-09T08:56:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.009+0000 7ff8767fc640 1 -- 172.21.15.98:0/4002565673 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff870005550 con 0x7ff88011b010 2024-05-09T08:56:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.009+0000 7ff8767fc640 1 -- 172.21.15.98:0/4002565673 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870004920 con 0x7ff88011b010 2024-05-09T08:56:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.009+0000 7ff8878e7640 1 -- 172.21.15.98:0/4002565673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8801cd0a0 con 0x7ff88011b010 2024-05-09T08:56:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.009+0000 7ff8878e7640 1 -- 172.21.15.98:0/4002565673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8801cd510 con 0x7ff88011b010 2024-05-09T08:56:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.013+0000 7ff88a5f1740 1 Processor -- start 2024-05-09T08:56:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.020+0000 7ff8767fc640 1 -- 172.21.15.98:0/4002565673 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff870043070 con 0x7ff88011b010 2024-05-09T08:56:46.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.027+0000 7ff88a5f1740 1 -- start start 2024-05-09T08:56:46.028 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.029+0000 7ff88a5f1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3f450 0x55a761a3f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.028 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.029+0000 7ff88a5f1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3fd70 0x55a761a4a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.029+0000 7ff88a5f1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a4a630 0x55a761a4ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.029+0000 7ff88a5f1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a7619a4340 con 0x55a761a3f450 2024-05-09T08:56:46.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.029+0000 7ff88a5f1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a76194c290 con 0x55a761a4a630 2024-05-09T08:56:46.032 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.029+0000 7ff88a5f1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a7619a0300 con 0x55a761a3fd70 2024-05-09T08:56:46.032 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.029+0000 7ff885683640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3fd70 0x55a761a4a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.029+0000 7ff885683640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3fd70 0x55a761a4a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:36494/0 (socket says 172.21.15.98:36494) 2024-05-09T08:56:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.029+0000 7ff885683640 1 -- 172.21.15.98:0/10018183 learned_addr learned my addr 172.21.15.98:0/10018183 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.029+0000 7ff885e84640 1 --2- 172.21.15.98:0/10018183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3f450 0x55a761a3f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.030+0000 7ff884e82640 1 --2- 172.21.15.98:0/10018183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a4a630 0x55a761a4ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.030+0000 7ff885e84640 1 -- 172.21.15.98:0/10018183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3fd70 msgr2=0x55a761a4a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.030+0000 7ff885e84640 1 --2- 172.21.15.98:0/10018183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3fd70 0x55a761a4a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.030+0000 7ff885e84640 1 -- 172.21.15.98:0/10018183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a4a630 msgr2=0x55a761a4ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.032+0000 7ff885e84640 1 --2- 172.21.15.98:0/10018183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a4a630 0x55a761a4ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.032+0000 7ff885e84640 1 -- 172.21.15.98:0/10018183 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7619708e0 con 0x55a761a3f450 2024-05-09T08:56:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.032+0000 7ff884e82640 1 --2- 172.21.15.98:0/10018183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a4a630 0x55a761a4ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:56:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.033+0000 7ff885e84640 1 --2- 172.21.15.98:0/10018183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3f450 0x55a761a3f830 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ff878005fd0 tx=0x7ff87801c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89cbf22132176fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:46.036 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.036+0000 7ff8767fc640 1 --2- 172.21.15.98:0/4002565673 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff86c0a2f80 0x7ff86c0a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.037 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.037+0000 7ff8767fc640 1 -- 172.21.15.98:0/4002565673 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7ff8700f0540 con 0x7ff88011b010 2024-05-09T08:56:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.040+0000 7ff86b7fe640 1 -- 172.21.15.98:0/10018183 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff878004d20 con 0x55a761a3f450 2024-05-09T08:56:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.040+0000 7ff86b7fe640 1 -- 172.21.15.98:0/10018183 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff878010a90 con 0x55a761a3f450 2024-05-09T08:56:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.040+0000 7ff86b7fe640 1 -- 172.21.15.98:0/10018183 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff878005670 con 0x55a761a3f450 2024-05-09T08:56:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.041+0000 7ff88a5f1740 1 -- 172.21.15.98:0/10018183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3f450 msgr2=0x55a761a3f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.041+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/10018183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3f450 0x55a761a3f830 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ff878005fd0 tx=0x7ff87801c800 comp rx=0 tx=0).stop 2024-05-09T08:56:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.041+0000 7ff88a5f1740 1 -- 172.21.15.98:0/10018183 shutdown_connections 2024-05-09T08:56:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.041+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/10018183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a4a630 0x55a761a4ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.041+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/10018183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3fd70 0x55a761a4a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.041+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/10018183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3f450 0x55a761a3f830 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.041+0000 7ff88a5f1740 1 -- 172.21.15.98:0/10018183 >> 172.21.15.98:0/10018183 conn(0x55a761a3b960 msgr2=0x55a761a3bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.042+0000 7ff885683640 1 --2- 172.21.15.98:0/4002565673 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff86c0a2f80 0x7ff86c0a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.044+0000 7ff88a5f1740 1 -- 172.21.15.98:0/10018183 shutdown_connections 2024-05-09T08:56:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.045+0000 7ff88a5f1740 1 -- 172.21.15.98:0/10018183 wait complete. 2024-05-09T08:56:46.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.046+0000 7ff88a5f1740 1 -- start start 2024-05-09T08:56:46.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.046+0000 7ff88a5f1740 1 Processor -- start 2024-05-09T08:56:46.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.051+0000 7ff88a5f1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a3f450 0x55a761a47590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.051+0000 7ff88a5f1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3fd70 0x55a761a47ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.051+0000 7ff88a5f1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a4a630 0x55a761a48010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.051+0000 7ff88a5f1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a761a4d260 con 0x55a761a4a630 2024-05-09T08:56:46.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.051+0000 7ff88a5f1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a76195f790 con 0x55a761a3f450 2024-05-09T08:56:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.051+0000 7ff88a5f1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a761a4d3e0 con 0x55a761a3fd70 2024-05-09T08:56:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.051+0000 7ff88a5f1740 10 client.0 _ll_register_callbacks cb 0x55a76199aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.051+0000 7ff885683640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a3f450 0x55a761a47590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.051+0000 7ff885683640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a3f450 0x55a761a47590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54386/0 (socket says 172.21.15.98:54386) 2024-05-09T08:56:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.051+0000 7ff885683640 1 -- 172.21.15.98:0/3469253808 learned_addr learned my addr 172.21.15.98:0/3469253808 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.052+0000 7ff884e82640 1 --2- 172.21.15.98:0/3469253808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3fd70 0x55a761a47ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.052+0000 7ff885e84640 1 --2- 172.21.15.98:0/3469253808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a4a630 0x55a761a48010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.052+0000 7ff885683640 1 -- 172.21.15.98:0/3469253808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3fd70 msgr2=0x55a761a47ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.052+0000 7ff885683640 1 --2- 172.21.15.98:0/3469253808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3fd70 0x55a761a47ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.052+0000 7ff885683640 1 -- 172.21.15.98:0/3469253808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a4a630 msgr2=0x55a761a48010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.052+0000 7ff885683640 1 --2- 172.21.15.98:0/3469253808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a4a630 0x55a761a48010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.052+0000 7ff885683640 1 -- 172.21.15.98:0/3469253808 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7616b8210 con 0x55a761a3f450 2024-05-09T08:56:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.052+0000 7ff885683640 1 --2- 172.21.15.98:0/3469253808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a3f450 0x55a761a47590 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff87c012680 tx=0x7ff87c012b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12c8f5f6c3f4d352 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.055+0000 7ff868ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:56:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.055+0000 7ff868ff9640 1 -- 172.21.15.98:0/3469253808 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c022070 con 0x55a761a3f450 2024-05-09T08:56:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.056+0000 7ff88a5f1740 1 -- 172.21.15.98:0/3469253808 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff878010a90 con 0x55a761a3f450 2024-05-09T08:56:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.056+0000 7ff868ff9640 1 -- 172.21.15.98:0/3469253808 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7ff87c00fad0 con 0x55a761a3f450 2024-05-09T08:56:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.057+0000 7ff868ff9640 1 -- 172.21.15.98:0/3469253808 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff87c00fcd0 con 0x55a761a3f450 2024-05-09T08:56:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.057+0000 7ff88a5f1740 10 client.26528 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.057+0000 7ff88a5f1740 1 -- 172.21.15.98:0/3469253808 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a761784620 con 0x55a761a3f450 2024-05-09T08:56:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.057+0000 7ff885683640 1 --2- 172.21.15.98:0/4002565673 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff86c0a2f80 0x7ff86c0a5440 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x55a761a409f0 tx=0x7ff87c0095d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.057+0000 7ff868ff9640 1 -- 172.21.15.98:0/3469253808 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff878010a90 con 0x55a761a3f450 2024-05-09T08:56:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.057+0000 7ff868ff9640 1 -- 172.21.15.98:0/3469253808 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7ff87c025030 con 0x55a761a3f450 2024-05-09T08:56:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.057+0000 7ff88a5f1740 10 client.26528 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.057+0000 7ff88a5f1740 10 client.26528 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.057+0000 7ff88a5f1740 20 client.26528 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.057+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/3469253808 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a761a5a750 0x55a761a5cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.057+0000 7ff88a5f1740 4 client.26528 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.057+0000 7ff88a5f1740 1 -- 172.21.15.98:0/3469253808 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7619b1a70 con 0x55a761a5a750 2024-05-09T08:56:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.058+0000 7ff884e82640 1 --2- 172.21.15.98:0/3469253808 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a761a5a750 0x55a761a5cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.059+0000 7ff884e82640 1 --2- 172.21.15.98:0/3469253808 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a761a5a750 0x55a761a5cc10 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a44165be71fa9a3e server_cookie=f18e5420a834bcce in_seq=0 out_seq=0 2024-05-09T08:56:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.059+0000 7ff868ff9640 10 client.26528 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.061+0000 7ff868ff9640 1 -- 172.21.15.98:0/3469253808 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a7619b1a70 con 0x55a761a5a750 2024-05-09T08:56:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.061+0000 7ff868ff9640 10 client.26528 handle_command_reply: tid=0 2024-05-09T08:56:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.061+0000 7ff88a5f1740 1 client.26528 shutdown 2024-05-09T08:56:46.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.062+0000 7ff88a5f1740 1 -- 172.21.15.98:0/3469253808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a3f450 msgr2=0x55a761a47590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:46.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.062+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/3469253808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a3f450 0x55a761a47590 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff87c012680 tx=0x7ff87c012b40 comp rx=0 tx=0).stop 2024-05-09T08:56:46.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.062+0000 7ff88a5f1740 1 -- 172.21.15.98:0/3469253808 shutdown_connections 2024-05-09T08:56:46.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.062+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/3469253808 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a761a5a750 0x55a761a5cc10 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.062+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/3469253808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a4a630 0x55a761a48010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.062+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/3469253808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3fd70 0x55a761a47ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.062+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/3469253808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a3f450 0x55a761a47590 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.062+0000 7ff88a5f1740 1 -- 172.21.15.98:0/3469253808 >> 172.21.15.98:0/3469253808 conn(0x55a761a3b960 msgr2=0x55a761a416e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:46.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.067+0000 7ff868ff9640 0 client.26528 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.067+0000 7ff88a5f1740 1 -- 172.21.15.98:0/3469253808 shutdown_connections 2024-05-09T08:56:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.068+0000 7ff88a5f1740 1 -- 172.21.15.98:0/3469253808 wait complete. 2024-05-09T08:56:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.068+0000 7ff88a5f1740 20 client.26528 trim_cache size 0 max 16384 2024-05-09T08:56:46.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:45 smithi098 ceph-mon[30092]: pgmap v188: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 703 MiB/s wr, 831 op/s 2024-05-09T08:56:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.109+0000 7ff88a5f1740 1 Processor -- start 2024-05-09T08:56:46.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.109+0000 7ff88a5f1740 1 -- start start 2024-05-09T08:56:46.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.109+0000 7ff88a5f1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3f450 0x55a761a49cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.109+0000 7ff88a5f1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3fd70 0x55a761a56120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.109+0000 7ff88a5f1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a4a630 0x55a761a62520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.110+0000 7ff88a5f1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801d7440 con 0x55a761a3fd70 2024-05-09T08:56:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.110+0000 7ff88a5f1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801d8f80 con 0x55a761a4a630 2024-05-09T08:56:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.110+0000 7ff88a5f1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801d7b10 con 0x55a761a3f450 2024-05-09T08:56:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.110+0000 7ff885683640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3f450 0x55a761a49cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.110+0000 7ff885683640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3f450 0x55a761a49cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:36520/0 (socket says 172.21.15.98:36520) 2024-05-09T08:56:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.110+0000 7ff885683640 1 -- 172.21.15.98:0/2369468986 learned_addr learned my addr 172.21.15.98:0/2369468986 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.110+0000 7ff885e84640 1 --2- 172.21.15.98:0/2369468986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a4a630 0x55a761a62520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.111+0000 7ff885e84640 1 -- 172.21.15.98:0/2369468986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3f450 msgr2=0x55a761a49cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.111+0000 7ff885e84640 1 --2- 172.21.15.98:0/2369468986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3f450 0x55a761a49cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.111+0000 7ff885e84640 1 -- 172.21.15.98:0/2369468986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3fd70 msgr2=0x55a761a56120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.111+0000 7ff885e84640 1 --2- 172.21.15.98:0/2369468986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3fd70 0x55a761a56120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.111+0000 7ff885e84640 1 -- 172.21.15.98:0/2369468986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a761a39f00 con 0x55a761a4a630 2024-05-09T08:56:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.111+0000 7ff885e84640 1 --2- 172.21.15.98:0/2369468986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a4a630 0x55a761a62520 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff87801cd00 tx=0x7ff878005c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b89993eff31b5deb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.113+0000 7ff86affd640 1 -- 172.21.15.98:0/2369468986 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff878003b70 con 0x55a761a4a630 2024-05-09T08:56:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.113+0000 7ff86affd640 1 -- 172.21.15.98:0/2369468986 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff878003d10 con 0x55a761a4a630 2024-05-09T08:56:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.113+0000 7ff86affd640 1 -- 172.21.15.98:0/2369468986 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8780054e0 con 0x55a761a4a630 2024-05-09T08:56:46.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.114+0000 7ff88a5f1740 1 -- 172.21.15.98:0/2369468986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a4a630 msgr2=0x55a761a62520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:46.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.114+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/2369468986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a4a630 0x55a761a62520 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff87801cd00 tx=0x7ff878005c80 comp rx=0 tx=0).stop 2024-05-09T08:56:46.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.119+0000 7ff88a5f1740 1 -- 172.21.15.98:0/2369468986 shutdown_connections 2024-05-09T08:56:46.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.119+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/2369468986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a4a630 0x55a761a62520 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.119+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/2369468986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3fd70 0x55a761a56120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.119+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/2369468986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a3f450 0x55a761a49cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.119+0000 7ff88a5f1740 1 -- 172.21.15.98:0/2369468986 >> 172.21.15.98:0/2369468986 conn(0x55a761a3b960 msgr2=0x55a761a42890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:46.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.120+0000 7ff88a5f1740 1 -- 172.21.15.98:0/2369468986 shutdown_connections 2024-05-09T08:56:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.121+0000 7ff88a5f1740 1 -- 172.21.15.98:0/2369468986 wait complete. 2024-05-09T08:56:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.122+0000 7ff88a5f1740 1 -- start start 2024-05-09T08:56:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.122+0000 7ff88a5f1740 1 Processor -- start 2024-05-09T08:56:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.122+0000 7ff88a5f1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a3f450 0x55a761a529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.122+0000 7ff88a5f1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3fd70 0x55a761a43f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.122+0000 7ff88a5f1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a4a630 0x55a761a44490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.122+0000 7ff88a5f1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801d88b0 con 0x55a761a3fd70 2024-05-09T08:56:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.122+0000 7ff88a5f1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801d81e0 con 0x55a761a3f450 2024-05-09T08:56:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.122+0000 7ff88a5f1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801d5420 con 0x55a761a4a630 2024-05-09T08:56:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.122+0000 7ff88a5f1740 10 client.0 _ll_register_callbacks cb 0x7ff87c0096c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.124+0000 7ff885e84640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a4a630 0x55a761a44490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.124+0000 7ff885e84640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a4a630 0x55a761a44490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:36526/0 (socket says 172.21.15.98:36526) 2024-05-09T08:56:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.124+0000 7ff885e84640 1 -- 172.21.15.98:0/662351444 learned_addr learned my addr 172.21.15.98:0/662351444 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.124+0000 7ff884e82640 1 --2- 172.21.15.98:0/662351444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3fd70 0x55a761a43f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.125+0000 7ff885683640 1 --2- 172.21.15.98:0/662351444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a3f450 0x55a761a529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.125+0000 7ff884e82640 1 -- 172.21.15.98:0/662351444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a4a630 msgr2=0x55a761a44490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.125+0000 7ff884e82640 1 --2- 172.21.15.98:0/662351444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a4a630 0x55a761a44490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.125+0000 7ff884e82640 1 -- 172.21.15.98:0/662351444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a3f450 msgr2=0x55a761a529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.125+0000 7ff884e82640 1 --2- 172.21.15.98:0/662351444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a3f450 0x55a761a529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.125+0000 7ff884e82640 1 -- 172.21.15.98:0/662351444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a761a44cc0 con 0x55a761a3fd70 2024-05-09T08:56:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.126+0000 7ff884e82640 1 --2- 172.21.15.98:0/662351444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3fd70 0x55a761a43f50 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7ff8700ed870 tx=0x7ff870005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90453b032137f765 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.127+0000 7ff868ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:56:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.127+0000 7ff868ff9640 1 -- 172.21.15.98:0/662351444 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87004a040 con 0x55a761a3fd70 2024-05-09T08:56:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.127+0000 7ff868ff9640 1 -- 172.21.15.98:0/662351444 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7ff8700ba940 con 0x55a761a3fd70 2024-05-09T08:56:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.127+0000 7ff88a5f1740 1 -- 172.21.15.98:0/662351444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff878003d10 con 0x55a761a3fd70 2024-05-09T08:56:46.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.131+0000 7ff868ff9640 1 -- 172.21.15.98:0/662351444 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff8700bc640 con 0x55a761a3fd70 2024-05-09T08:56:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.141+0000 7ff88a5f1740 10 client.17226 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.141+0000 7ff88a5f1740 1 -- 172.21.15.98:0/662351444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a761ac0dd0 con 0x55a761a3fd70 2024-05-09T08:56:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.144+0000 7ff868ff9640 1 -- 172.21.15.98:0/662351444 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff878003d10 con 0x55a761a3fd70 2024-05-09T08:56:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.144+0000 7ff868ff9640 1 -- 172.21.15.98:0/662351444 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7ff870031520 con 0x55a761a3fd70 2024-05-09T08:56:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.147+0000 7ff88a5f1740 10 client.17226 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.147+0000 7ff88a5f1740 10 client.17226 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.147+0000 7ff88a5f1740 20 client.17226 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.147+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/662351444 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a761ac1340 0x55a761ac3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.147+0000 7ff88a5f1740 4 client.17226 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:56:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.147+0000 7ff88a5f1740 1 -- 172.21.15.98:0/662351444 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a7619abdc0 con 0x55a761ac1340 2024-05-09T08:56:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.158+0000 7ff885683640 1 --2- 172.21.15.98:0/662351444 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a761ac1340 0x55a761ac3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:45 smithi142 ceph-mon[34625]: pgmap v188: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 703 MiB/s wr, 831 op/s 2024-05-09T08:56:46.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.182+0000 7ff885683640 1 --2- 172.21.15.98:0/662351444 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a761ac1340 0x55a761ac3760 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=685ee3ae39f18d99 server_cookie=5c6554c90faf041f in_seq=0 out_seq=0 2024-05-09T08:56:46.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:56:46.182+0000 7ff868ff9640 10 client.17226 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.200+0000 7f60ef509640 1 -- 172.21.15.98:0/1836291139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60e8108830 msgr2=0x7f60e8108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.200+0000 7f60ef509640 1 --2- 172.21.15.98:0/1836291139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60e8108830 0x7f60e8108cb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f60dc009860 tx=0x7f60dc02a3d0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.201+0000 7f60ef509640 1 -- 172.21.15.98:0/1836291139 shutdown_connections 2024-05-09T08:56:46.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.201+0000 7f60ef509640 1 --2- 172.21.15.98:0/1836291139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60e81092d0 0x7f60e811d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.201+0000 7f60ef509640 1 --2- 172.21.15.98:0/1836291139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60e8108830 0x7f60e8108cb0 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f60dc009860 tx=0x7f60dc02a3d0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.201+0000 7f60ef509640 1 --2- 172.21.15.98:0/1836291139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60e810fe30 0x7f60e8110210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.201+0000 7f60ef509640 1 -- 172.21.15.98:0/1836291139 >> 172.21.15.98:0/1836291139 conn(0x7f60e806c770 msgr2=0x7f60e806cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:46.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.202+0000 7f60ef509640 1 -- 172.21.15.98:0/1836291139 shutdown_connections 2024-05-09T08:56:46.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.202+0000 7f60ef509640 1 -- 172.21.15.98:0/1836291139 wait complete. 2024-05-09T08:56:46.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.203+0000 7f60ef509640 1 Processor -- start 2024-05-09T08:56:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.203+0000 7f60ef509640 1 -- start start 2024-05-09T08:56:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.203+0000 7f60ef509640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60e81092d0 0x7f60e811cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.203+0000 7f60ef509640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60e810fe30 0x7f60e8115e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.203+0000 7f60ef509640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60e8116380 0x7f60e8116830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.203+0000 7f60ef509640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f60e8120850 con 0x7f60e810fe30 2024-05-09T08:56:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.203+0000 7f60ef509640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f60e81206d0 con 0x7f60e81092d0 2024-05-09T08:56:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.203+0000 7f60ef509640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f60e81209d0 con 0x7f60e8116380 2024-05-09T08:56:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.204+0000 7f60edaa6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60e8116380 0x7f60e8116830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.204+0000 7f60edaa6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60e8116380 0x7f60e8116830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:36548/0 (socket says 172.21.15.98:36548) 2024-05-09T08:56:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.204+0000 7f60edaa6640 1 -- 172.21.15.98:0/2468543265 learned_addr learned my addr 172.21.15.98:0/2468543265 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.204+0000 7f60edaa6640 1 -- 172.21.15.98:0/2468543265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60e81092d0 msgr2=0x7f60e811cdc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:56:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.204+0000 7f60edaa6640 1 --2- 172.21.15.98:0/2468543265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60e81092d0 0x7f60e811cdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.204+0000 7f60edaa6640 1 -- 172.21.15.98:0/2468543265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60e810fe30 msgr2=0x7f60e8115e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.204+0000 7f60edaa6640 1 --2- 172.21.15.98:0/2468543265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60e810fe30 0x7f60e8115e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.204+0000 7f60edaa6640 1 -- 172.21.15.98:0/2468543265 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e811a9b0 con 0x7f60e8116380 2024-05-09T08:56:46.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.204+0000 7f60edaa6640 1 --2- 172.21.15.98:0/2468543265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60e8116380 0x7f60e8116830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009fd0 tx=0x7f60d800b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.206+0000 7f60e67fc640 1 -- 172.21.15.98:0/2468543265 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d8017070 con 0x7f60e8116380 2024-05-09T08:56:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.206+0000 7f60e67fc640 1 -- 172.21.15.98:0/2468543265 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60d8003e50 con 0x7f60e8116380 2024-05-09T08:56:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.206+0000 7f60e67fc640 1 -- 172.21.15.98:0/2468543265 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d80106c0 con 0x7f60e8116380 2024-05-09T08:56:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.208+0000 7f60ef509640 1 -- 172.21.15.98:0/2468543265 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60e811aca0 con 0x7f60e8116380 2024-05-09T08:56:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.208+0000 7f60ef509640 1 -- 172.21.15.98:0/2468543265 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60e811b170 con 0x7f60e8116380 2024-05-09T08:56:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.212+0000 7f60ef509640 1 -- 172.21.15.98:0/2468543265 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60e806da40 con 0x7f60e8116380 2024-05-09T08:56:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.219+0000 7f60e67fc640 1 -- 172.21.15.98:0/2468543265 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f60d8003970 con 0x7f60e8116380 2024-05-09T08:56:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.220+0000 7f60e67fc640 1 --2- 172.21.15.98:0/2468543265 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f60d00a2f00 0x7f60d00a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:46.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.220+0000 7f60e67fc640 1 -- 172.21.15.98:0/2468543265 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f60d80cf9c0 con 0x7f60e8116380 2024-05-09T08:56:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.221+0000 7f60e67fc640 1 -- 172.21.15.98:0/2468543265 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f60d80fd740 con 0x7f60e8116380 2024-05-09T08:56:46.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.222+0000 7f60ed2a5640 1 --2- 172.21.15.98:0/2468543265 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f60d00a2f00 0x7f60d00a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:46.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.223+0000 7f60ed2a5640 1 --2- 172.21.15.98:0/2468543265 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f60d00a2f00 0x7f60d00a53c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f60e0005fd0 tx=0x7f60e0009260 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:46.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.684+0000 7f60ef509640 1 -- 172.21.15.98:0/2468543265 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60e8117370 con 0x7f60e8116380 2024-05-09T08:56:46.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.686+0000 7f60e67fc640 1 -- 172.21.15.98:0/2468543265 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11944 (secure 0 0 0) 0x7f60d809c2c0 con 0x7f60e8116380 2024-05-09T08:56:46.687 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:56:46.688 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":16641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:54:25.881641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16749":{"gid":16749,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/3204012039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3204012039},{"type":"v1","addr":"172.21.15.105:6835","nonce":3204012039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24439,14667,24388,24461,24457]},"id":1}]} 2024-05-09T08:56:46.688 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 35 2024-05-09T08:56:46.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.702+0000 7f60ef509640 1 -- 172.21.15.98:0/2468543265 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f60d00a2f00 msgr2=0x7f60d00a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:46.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.702+0000 7f60ef509640 1 --2- 172.21.15.98:0/2468543265 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f60d00a2f00 0x7f60d00a53c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f60e0005fd0 tx=0x7f60e0009260 comp rx=0 tx=0).stop 2024-05-09T08:56:46.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.702+0000 7f60ef509640 1 -- 172.21.15.98:0/2468543265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60e8116380 msgr2=0x7f60e8116830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:46.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.702+0000 7f60ef509640 1 --2- 172.21.15.98:0/2468543265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60e8116380 0x7f60e8116830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009fd0 tx=0x7f60d800b260 comp rx=0 tx=0).stop 2024-05-09T08:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.703+0000 7f60ef509640 1 -- 172.21.15.98:0/2468543265 shutdown_connections 2024-05-09T08:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.703+0000 7f60ef509640 1 --2- 172.21.15.98:0/2468543265 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f60d00a2f00 0x7f60d00a53c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.703+0000 7f60ef509640 1 --2- 172.21.15.98:0/2468543265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60e8116380 0x7f60e8116830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.703+0000 7f60ef509640 1 --2- 172.21.15.98:0/2468543265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60e810fe30 0x7f60e8115e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.703+0000 7f60ef509640 1 --2- 172.21.15.98:0/2468543265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60e81092d0 0x7f60e811cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.703+0000 7f60ef509640 1 -- 172.21.15.98:0/2468543265 >> 172.21.15.98:0/2468543265 conn(0x7f60e806c770 msgr2=0x7f60e8070410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.704+0000 7f60ef509640 1 -- 172.21.15.98:0/2468543265 shutdown_connections 2024-05-09T08:56:46.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:46.704+0000 7f60ef509640 1 -- 172.21.15.98:0/2468543265 wait complete. 2024-05-09T08:56:46.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:56:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.007+0000 7f642a209640 1 -- 172.21.15.98:0/3223855020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f642411b010 msgr2=0x7f642411d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.007+0000 7f642a209640 1 --2- 172.21.15.98:0/3223855020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f642411b010 0x7f642411d400 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6414009860 tx=0x7f641402a360 comp rx=0 tx=0).stop 2024-05-09T08:56:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.012+0000 7f642a209640 1 -- 172.21.15.98:0/3223855020 shutdown_connections 2024-05-09T08:56:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.012+0000 7f642a209640 1 --2- 172.21.15.98:0/3223855020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f642411b010 0x7f642411d400 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.012+0000 7f642a209640 1 --2- 172.21.15.98:0/3223855020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64241088c0 0x7f6424108d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.012+0000 7f642a209640 1 --2- 172.21.15.98:0/3223855020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f642410fea0 0x7f6424110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.012+0000 7f642a209640 1 -- 172.21.15.98:0/3223855020 >> 172.21.15.98:0/3223855020 conn(0x7f642406c810 msgr2=0x7f642406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.013+0000 7f642a209640 1 -- 172.21.15.98:0/3223855020 shutdown_connections 2024-05-09T08:56:47.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.015+0000 7f642a209640 1 -- 172.21.15.98:0/3223855020 wait complete. 2024-05-09T08:56:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.017+0000 7f642a209640 1 Processor -- start 2024-05-09T08:56:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.018+0000 7f642a209640 1 -- start start 2024-05-09T08:56:47.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.020+0000 7f642a209640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64241088c0 0x7f642411a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.021+0000 7f642a209640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f642410fea0 0x7f6424114850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.021+0000 7f642a209640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f642411b010 0x7f6424114d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.021+0000 7f642a209640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6424111240 con 0x7f642410fea0 2024-05-09T08:56:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.021+0000 7f642a209640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f64241110c0 con 0x7f642411b010 2024-05-09T08:56:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.021+0000 7f642a209640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f64241113c0 con 0x7f64241088c0 2024-05-09T08:56:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.021+0000 7f6423fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f642411b010 0x7f6424114d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.021+0000 7f6423fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f642411b010 0x7f6424114d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54432/0 (socket says 172.21.15.98:54432) 2024-05-09T08:56:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.021+0000 7f6423fff640 1 -- 172.21.15.98:0/2066553455 learned_addr learned my addr 172.21.15.98:0/2066553455 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.022+0000 7f6423fff640 1 -- 172.21.15.98:0/2066553455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64241088c0 msgr2=0x7f642411a7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:56:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.022+0000 7f6423fff640 1 --2- 172.21.15.98:0/2066553455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64241088c0 0x7f642411a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.022+0000 7f6423fff640 1 -- 172.21.15.98:0/2066553455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f642410fea0 msgr2=0x7f6424114850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:56:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.022+0000 7f6423fff640 1 --2- 172.21.15.98:0/2066553455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f642410fea0 0x7f6424114850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.022+0000 7f6423fff640 1 -- 172.21.15.98:0/2066553455 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6424115560 con 0x7f642411b010 2024-05-09T08:56:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.022+0000 7f6423fff640 1 --2- 172.21.15.98:0/2066553455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f642411b010 0x7f6424114d90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f641402a950 tx=0x7f6414030ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.023+0000 7f6420ff9640 1 -- 172.21.15.98:0/2066553455 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6414037040 con 0x7f642411b010 2024-05-09T08:56:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.023+0000 7f6420ff9640 1 -- 172.21.15.98:0/2066553455 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6414040070 con 0x7f642411b010 2024-05-09T08:56:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.023+0000 7f6420ff9640 1 -- 172.21.15.98:0/2066553455 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6414003850 con 0x7f642411b010 2024-05-09T08:56:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.023+0000 7f642a209640 1 -- 172.21.15.98:0/2066553455 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64241bbb70 con 0x7f642411b010 2024-05-09T08:56:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.023+0000 7f642a209640 1 -- 172.21.15.98:0/2066553455 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64241bbf80 con 0x7f642411b010 2024-05-09T08:56:47.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.027+0000 7f6420ff9640 1 -- 172.21.15.98:0/2066553455 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f64140039f0 con 0x7f642411b010 2024-05-09T08:56:47.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.029+0000 7f6420ff9640 1 --2- 172.21.15.98:0/2066553455 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f640c0a2f00 0x7f640c0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.029+0000 7f6420ff9640 1 -- 172.21.15.98:0/2066553455 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f64140ef260 con 0x7f642411b010 2024-05-09T08:56:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.029+0000 7f642cff1740 1 Processor -- start 2024-05-09T08:56:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.029+0000 7f642cff1740 1 -- start start 2024-05-09T08:56:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.030+0000 7f642cff1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14177520 0x563c14177900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.030+0000 7f642cff1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177e40 0x563c14181fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.030+0000 7f642cff1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c14184900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.030+0000 7f642cff1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563c140dcc40 con 0x563c14177e40 2024-05-09T08:56:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.030+0000 7f642cff1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563c14084580 con 0x563c14177520 2024-05-09T08:56:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.030+0000 7f642cff1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563c140d8c00 con 0x563c14182510 2024-05-09T08:56:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.030+0000 7f6423fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c14184900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.030+0000 7f6423fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c14184900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:36566/0 (socket says 172.21.15.98:36566) 2024-05-09T08:56:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.030+0000 7f6423fff640 1 -- 172.21.15.98:0/307209205 learned_addr learned my addr 172.21.15.98:0/307209205 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.031+0000 7f6423fff640 1 -- 172.21.15.98:0/307209205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14177520 msgr2=0x563c14177900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:56:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.031+0000 7f6423fff640 1 --2- 172.21.15.98:0/307209205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14177520 0x563c14177900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.031+0000 7f6423fff640 1 -- 172.21.15.98:0/307209205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177e40 msgr2=0x563c14181fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.031+0000 7f6423fff640 1 --2- 172.21.15.98:0/307209205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177e40 0x563c14181fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.031+0000 7f6423fff640 1 -- 172.21.15.98:0/307209205 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c13df03f0 con 0x563c14182510 2024-05-09T08:56:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.031+0000 7f6423fff640 1 --2- 172.21.15.98:0/307209205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c14184900 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f64140b78d0 tx=0x7f64140bb300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f08a94cc723a855 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.031+0000 7f64237fe640 1 --2- 172.21.15.98:0/2066553455 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f640c0a2f00 0x7f640c0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.032+0000 7f6409ffb640 1 -- 172.21.15.98:0/307209205 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64140d4070 con 0x563c14182510 2024-05-09T08:56:47.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.032+0000 7f64237fe640 1 --2- 172.21.15.98:0/2066553455 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f640c0a2f00 0x7f640c0a53c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f641c005fd0 tx=0x7f641c0095c0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.032+0000 7f6409ffb640 1 -- 172.21.15.98:0/307209205 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6414032920 con 0x563c14182510 2024-05-09T08:56:47.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.032+0000 7f6409ffb640 1 -- 172.21.15.98:0/307209205 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64140c40e0 con 0x563c14182510 2024-05-09T08:56:47.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.033+0000 7f642cff1740 1 -- 172.21.15.98:0/307209205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 msgr2=0x563c14184900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.033+0000 7f642cff1740 1 --2- 172.21.15.98:0/307209205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c14184900 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f64140b78d0 tx=0x7f64140bb300 comp rx=0 tx=0).stop 2024-05-09T08:56:47.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.034+0000 7f642cff1740 1 -- 172.21.15.98:0/307209205 shutdown_connections 2024-05-09T08:56:47.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.034+0000 7f642cff1740 1 --2- 172.21.15.98:0/307209205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c14184900 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.034+0000 7f642cff1740 1 --2- 172.21.15.98:0/307209205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177e40 0x563c14181fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.034+0000 7f642cff1740 1 --2- 172.21.15.98:0/307209205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14177520 0x563c14177900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.034+0000 7f642cff1740 1 -- 172.21.15.98:0/307209205 >> 172.21.15.98:0/307209205 conn(0x563c14173840 msgr2=0x563c14173c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.036+0000 7f642cff1740 1 -- 172.21.15.98:0/307209205 shutdown_connections 2024-05-09T08:56:47.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.036+0000 7f642cff1740 1 -- 172.21.15.98:0/307209205 wait complete. 2024-05-09T08:56:47.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.036+0000 7f642cff1740 1 -- start start 2024-05-09T08:56:47.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.036+0000 7f642cff1740 1 Processor -- start 2024-05-09T08:56:47.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.037+0000 7f642cff1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177520 0x563c1417f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.037+0000 7f642cff1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14177e40 0x563c1417f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.037+0000 7f642cff1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14182510 0x563c1417fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.037+0000 7f642cff1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563c141850c0 con 0x563c14177520 2024-05-09T08:56:47.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.037+0000 7f642cff1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563c14097c40 con 0x563c14182510 2024-05-09T08:56:47.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.037+0000 7f642cff1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563c14185240 con 0x563c14177e40 2024-05-09T08:56:47.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.037+0000 7f642cff1740 10 client.0 _ll_register_callbacks cb 0x563c13defbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:47.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.038+0000 7f6422ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14177e40 0x563c1417f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.038+0000 7f6423fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14182510 0x563c1417fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.038+0000 7f6422ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14177e40 0x563c1417f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:36572/0 (socket says 172.21.15.98:36572) 2024-05-09T08:56:47.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.038+0000 7f6422ffd640 1 -- 172.21.15.98:0/1699953594 learned_addr learned my addr 172.21.15.98:0/1699953594 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:47.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.038+0000 7f64237fe640 1 --2- 172.21.15.98:0/1699953594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177520 0x563c1417f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.039+0000 7f6422ffd640 1 -- 172.21.15.98:0/1699953594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14182510 msgr2=0x563c1417fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.039+0000 7f6422ffd640 1 --2- 172.21.15.98:0/1699953594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14182510 0x563c1417fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.039+0000 7f6422ffd640 1 -- 172.21.15.98:0/1699953594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177520 msgr2=0x563c1417f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.039+0000 7f6422ffd640 1 --2- 172.21.15.98:0/1699953594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177520 0x563c1417f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.039+0000 7f6422ffd640 1 -- 172.21.15.98:0/1699953594 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c13eba7d0 con 0x563c14177e40 2024-05-09T08:56:47.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.039+0000 7f6422ffd640 1 --2- 172.21.15.98:0/1699953594 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14177e40 0x563c1417f9b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f64180058c0 tx=0x7f641800d6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8002dddbf85dc2cc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.039+0000 7f6402ffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:56:47.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.040+0000 7f6402ffd640 1 -- 172.21.15.98:0/1699953594 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6418016070 con 0x563c14177e40 2024-05-09T08:56:47.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.040+0000 7f642cff1740 1 -- 172.21.15.98:0/1699953594 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6414032920 con 0x563c14177e40 2024-05-09T08:56:47.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.040+0000 7f6402ffd640 1 -- 172.21.15.98:0/1699953594 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f6418010660 con 0x563c14177e40 2024-05-09T08:56:47.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.042+0000 7f6402ffd640 1 -- 172.21.15.98:0/1699953594 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6418022050 con 0x563c14177e40 2024-05-09T08:56:47.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.043+0000 7f642cff1740 10 client.26536 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:47.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.043+0000 7f642cff1740 1 -- 172.21.15.98:0/1699953594 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563c141912e0 con 0x563c14177e40 2024-05-09T08:56:47.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.043+0000 7f6402ffd640 1 -- 172.21.15.98:0/1699953594 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6414032920 con 0x563c14177e40 2024-05-09T08:56:47.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.043+0000 7f6402ffd640 1 -- 172.21.15.98:0/1699953594 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f641800f840 con 0x563c14177e40 2024-05-09T08:56:47.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.043+0000 7f642cff1740 10 client.26536 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:47.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.043+0000 7f642cff1740 10 client.26536 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:56:47.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.043+0000 7f642cff1740 20 client.26536 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:47.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.043+0000 7f642cff1740 1 --2- 172.21.15.98:0/1699953594 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563c14192c00 0x563c14195060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.043+0000 7f642cff1740 4 client.26536 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:47.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.043+0000 7f642cff1740 1 -- 172.21.15.98:0/1699953594 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c140ea370 con 0x563c14192c00 2024-05-09T08:56:47.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.044+0000 7f64237fe640 1 --2- 172.21.15.98:0/1699953594 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563c14192c00 0x563c14195060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.050+0000 7f64237fe640 1 --2- 172.21.15.98:0/1699953594 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563c14192c00 0x563c14195060 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2aaa08b2d79acbce server_cookie=62413c4ac8b73a10 in_seq=0 out_seq=0 2024-05-09T08:56:47.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.050+0000 7f6402ffd640 10 client.26536 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:56:47.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.051+0000 7f6402ffd640 1 -- 172.21.15.98:0/1699953594 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c140ea370 con 0x563c14192c00 2024-05-09T08:56:47.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.051+0000 7f6402ffd640 10 client.26536 handle_command_reply: tid=0 2024-05-09T08:56:47.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.051+0000 7f642cff1740 1 client.26536 shutdown 2024-05-09T08:56:47.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.054+0000 7f642cff1740 1 -- 172.21.15.98:0/1699953594 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14177e40 msgr2=0x563c1417f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.054+0000 7f642cff1740 1 --2- 172.21.15.98:0/1699953594 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14177e40 0x563c1417f9b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f64180058c0 tx=0x7f641800d6b0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.054+0000 7f642cff1740 1 -- 172.21.15.98:0/1699953594 shutdown_connections 2024-05-09T08:56:47.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.054+0000 7f642cff1740 1 --2- 172.21.15.98:0/1699953594 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563c14192c00 0x563c14195060 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.054+0000 7f642cff1740 1 --2- 172.21.15.98:0/1699953594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14182510 0x563c1417fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.054+0000 7f642cff1740 1 --2- 172.21.15.98:0/1699953594 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14177e40 0x563c1417f9b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.054+0000 7f642cff1740 1 --2- 172.21.15.98:0/1699953594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177520 0x563c1417f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.054+0000 7f642cff1740 1 -- 172.21.15.98:0/1699953594 >> 172.21.15.98:0/1699953594 conn(0x563c14173840 msgr2=0x563c141729c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:47.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.055+0000 7f6402ffd640 0 client.26536 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:56:47.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.056+0000 7f642cff1740 1 -- 172.21.15.98:0/1699953594 shutdown_connections 2024-05-09T08:56:47.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.056+0000 7f642cff1740 1 -- 172.21.15.98:0/1699953594 wait complete. 2024-05-09T08:56:47.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.059+0000 7f642cff1740 20 client.26536 trim_cache size 0 max 16384 2024-05-09T08:56:47.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.085+0000 7f642cff1740 1 Processor -- start 2024-05-09T08:56:47.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.086+0000 7f642cff1740 1 -- start start 2024-05-09T08:56:47.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.086+0000 7f642cff1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177520 0x563c1418df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.086+0000 7f642cff1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14177e40 0x563c1417fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.086+0000 7f642cff1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c14172840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.086+0000 7f642cff1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6410010030 con 0x563c14177520 2024-05-09T08:56:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.086+0000 7f642cff1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6410011b70 con 0x563c14177e40 2024-05-09T08:56:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.086+0000 7f642cff1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6410010700 con 0x563c14182510 2024-05-09T08:56:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.086+0000 7f6423fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c14172840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.086+0000 7f6423fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c14172840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:36576/0 (socket says 172.21.15.98:36576) 2024-05-09T08:56:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.086+0000 7f6423fff640 1 -- 172.21.15.98:0/1239834159 learned_addr learned my addr 172.21.15.98:0/1239834159 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.086+0000 7f6422ffd640 1 --2- 172.21.15.98:0/1239834159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14177e40 0x563c1417fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.087+0000 7f6423fff640 1 -- 172.21.15.98:0/1239834159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14177e40 msgr2=0x563c1417fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.087+0000 7f6423fff640 1 --2- 172.21.15.98:0/1239834159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14177e40 0x563c1417fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.087+0000 7f6423fff640 1 -- 172.21.15.98:0/1239834159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177520 msgr2=0x563c1418df80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:56:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.087+0000 7f6423fff640 1 --2- 172.21.15.98:0/1239834159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177520 0x563c1418df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.087+0000 7f6423fff640 1 -- 172.21.15.98:0/1239834159 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c1417d9e0 con 0x563c14182510 2024-05-09T08:56:47.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:46 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2468543265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:56:47.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.091+0000 7f6423fff640 1 --2- 172.21.15.98:0/1239834159 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c14172840 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6414030f50 tx=0x7f64140bb370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be13eccaa3f123d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.092+0000 7f64097fa640 1 -- 172.21.15.98:0/1239834159 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64140d4040 con 0x563c14182510 2024-05-09T08:56:47.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.092+0000 7f64097fa640 1 -- 172.21.15.98:0/1239834159 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6414032920 con 0x563c14182510 2024-05-09T08:56:47.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.092+0000 7f64097fa640 1 -- 172.21.15.98:0/1239834159 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64140bf820 con 0x563c14182510 2024-05-09T08:56:47.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.093+0000 7f642cff1740 1 -- 172.21.15.98:0/1239834159 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 msgr2=0x563c14172840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.093+0000 7f642cff1740 1 --2- 172.21.15.98:0/1239834159 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c14172840 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6414030f50 tx=0x7f64140bb370 comp rx=0 tx=0).stop 2024-05-09T08:56:47.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.093+0000 7f642cff1740 1 -- 172.21.15.98:0/1239834159 shutdown_connections 2024-05-09T08:56:47.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.093+0000 7f642cff1740 1 --2- 172.21.15.98:0/1239834159 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c14172840 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.093+0000 7f642cff1740 1 --2- 172.21.15.98:0/1239834159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14177e40 0x563c1417fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.093+0000 7f642cff1740 1 --2- 172.21.15.98:0/1239834159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177520 0x563c1418df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.093+0000 7f642cff1740 1 -- 172.21.15.98:0/1239834159 >> 172.21.15.98:0/1239834159 conn(0x563c14173840 msgr2=0x563c14172130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:47.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.097+0000 7f642cff1740 1 -- 172.21.15.98:0/1239834159 shutdown_connections 2024-05-09T08:56:47.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.097+0000 7f642cff1740 1 -- 172.21.15.98:0/1239834159 wait complete. 2024-05-09T08:56:47.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.098+0000 7f642cff1740 1 -- start start 2024-05-09T08:56:47.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.098+0000 7f642cff1740 1 Processor -- start 2024-05-09T08:56:47.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.099+0000 7f642cff1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14177520 0x563c1417c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.099+0000 7f642cff1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177e40 0x563c1417ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.099+0000 7f642cff1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c141f81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.099+0000 7f642cff1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f64100114a0 con 0x563c14177e40 2024-05-09T08:56:47.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.099+0000 7f6423fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c141f81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.099+0000 7f6423fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c141f81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:36584/0 (socket says 172.21.15.98:36584) 2024-05-09T08:56:47.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.099+0000 7f6423fff640 1 -- 172.21.15.98:0/3058011849 learned_addr learned my addr 172.21.15.98:0/3058011849 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:47.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.099+0000 7f642cff1740 1 -- 172.21.15.98:0/3058011849 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6410010dd0 con 0x563c14177520 2024-05-09T08:56:47.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.099+0000 7f642cff1740 1 -- 172.21.15.98:0/3058011849 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f641000e010 con 0x563c14182510 2024-05-09T08:56:47.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.101+0000 7f6422ffd640 1 --2- 172.21.15.98:0/3058011849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177e40 0x563c1417ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.101+0000 7f6423fff640 1 -- 172.21.15.98:0/3058011849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14177520 msgr2=0x563c1417c4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:56:47.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.101+0000 7f6423fff640 1 --2- 172.21.15.98:0/3058011849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14177520 0x563c1417c4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.101+0000 7f6423fff640 1 -- 172.21.15.98:0/3058011849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177e40 msgr2=0x563c1417ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.101+0000 7f6423fff640 1 --2- 172.21.15.98:0/3058011849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177e40 0x563c1417ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.101+0000 7f6423fff640 1 -- 172.21.15.98:0/3058011849 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c1417d2e0 con 0x563c14182510 2024-05-09T08:56:47.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.101+0000 7f6423fff640 1 --2- 172.21.15.98:0/3058011849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c141f81e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f64140b7e40 tx=0x7f6414005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb4cbdf3496604bc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.102+0000 7f6402ffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:56:47.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.102+0000 7f642cff1740 10 client.0 _ll_register_callbacks cb 0x7f6418005f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:47.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.102+0000 7f642cff1740 1 -- 172.21.15.98:0/3058011849 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6414032920 con 0x563c14182510 2024-05-09T08:56:47.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.102+0000 7f6402ffd640 1 -- 172.21.15.98:0/3058011849 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64140a1060 con 0x563c14182510 2024-05-09T08:56:47.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.102+0000 7f6402ffd640 1 -- 172.21.15.98:0/3058011849 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f64140b7ae0 con 0x563c14182510 2024-05-09T08:56:47.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.104+0000 7f6402ffd640 1 -- 172.21.15.98:0/3058011849 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f64140d4040 con 0x563c14182510 2024-05-09T08:56:47.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.104+0000 7f642cff1740 10 client.26542 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:47.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.104+0000 7f642cff1740 1 -- 172.21.15.98:0/3058011849 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({fsmap=35}) -- 0x563c141f94f0 con 0x563c14182510 2024-05-09T08:56:47.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.105+0000 7f6402ffd640 1 -- 172.21.15.98:0/3058011849 <== mon.1 v2:172.21.15.142:3300/0 4 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f641403a070 con 0x563c14182510 2024-05-09T08:56:47.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.105+0000 7f642cff1740 10 client.26542 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:47.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.105+0000 7f642cff1740 10 client.26542 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:56:47.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.105+0000 7f642cff1740 20 client.26542 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:47.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.105+0000 7f642cff1740 1 --2- 172.21.15.98:0/3058011849 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563c141f9a60 0x563c141fbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.105+0000 7f642cff1740 4 client.26542 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:56:47.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.105+0000 7f642cff1740 1 -- 172.21.15.98:0/3058011849 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f6414032920 con 0x563c141f9a60 2024-05-09T08:56:47.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.105+0000 7f64237fe640 1 --2- 172.21.15.98:0/3058011849 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563c141f9a60 0x563c141fbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.109+0000 7f64237fe640 1 --2- 172.21.15.98:0/3058011849 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563c141f9a60 0x563c141fbf20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=75d1028602b70dd4 server_cookie=2318c470d1af0151 in_seq=0 out_seq=0 2024-05-09T08:56:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.110+0000 7f6402ffd640 10 client.26542 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:56:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.110+0000 7f6402ffd640 1 -- 172.21.15.98:0/3058011849 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f6414032920 con 0x563c141f9a60 2024-05-09T08:56:47.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.110+0000 7f6402ffd640 10 client.26542 handle_command_reply: tid=0 2024-05-09T08:56:47.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.110+0000 7f642cff1740 1 client.26542 shutdown 2024-05-09T08:56:47.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.112+0000 7f642cff1740 1 -- 172.21.15.98:0/3058011849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 msgr2=0x563c141f81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.112+0000 7f642cff1740 1 --2- 172.21.15.98:0/3058011849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c141f81e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f64140b7e40 tx=0x7f6414005f90 comp rx=0 tx=0).stop 2024-05-09T08:56:47.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.116+0000 7f642cff1740 1 -- 172.21.15.98:0/3058011849 shutdown_connections 2024-05-09T08:56:47.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.116+0000 7f642cff1740 1 --2- 172.21.15.98:0/3058011849 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563c141f9a60 0x563c141fbf20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.116+0000 7f642cff1740 1 --2- 172.21.15.98:0/3058011849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c14182510 0x563c141f81e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.116+0000 7f642cff1740 1 --2- 172.21.15.98:0/3058011849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c14177e40 0x563c1417ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.117+0000 7f642cff1740 1 --2- 172.21.15.98:0/3058011849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c14177520 0x563c1417c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.117+0000 7f642cff1740 1 -- 172.21.15.98:0/3058011849 >> 172.21.15.98:0/3058011849 conn(0x563c14173840 msgr2=0x563c141729c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:47.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.118+0000 7f6402ffd640 0 client.26542 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:56:47.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.122+0000 7f642cff1740 1 -- 172.21.15.98:0/3058011849 shutdown_connections 2024-05-09T08:56:47.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.122+0000 7f642cff1740 1 -- 172.21.15.98:0/3058011849 wait complete. 2024-05-09T08:56:47.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.123+0000 7f642cff1740 20 client.26542 trim_cache size 0 max 16384 2024-05-09T08:56:47.124 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:56:47.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.126+0000 7f642a209640 1 -- 172.21.15.98:0/2066553455 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f640c0a2f00 msgr2=0x7f640c0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.126+0000 7f642a209640 1 --2- 172.21.15.98:0/2066553455 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f640c0a2f00 0x7f640c0a53c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f641c005fd0 tx=0x7f641c0095c0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.126+0000 7f642a209640 1 -- 172.21.15.98:0/2066553455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f642411b010 msgr2=0x7f6424114d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.126+0000 7f642a209640 1 --2- 172.21.15.98:0/2066553455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f642411b010 0x7f6424114d90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f641402a950 tx=0x7f6414030ee0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.126+0000 7f642a209640 1 -- 172.21.15.98:0/2066553455 shutdown_connections 2024-05-09T08:56:47.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.126+0000 7f642a209640 1 --2- 172.21.15.98:0/2066553455 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f640c0a2f00 0x7f640c0a53c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.126+0000 7f642a209640 1 --2- 172.21.15.98:0/2066553455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f642411b010 0x7f6424114d90 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.126+0000 7f642a209640 1 --2- 172.21.15.98:0/2066553455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f642410fea0 0x7f6424114850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.126+0000 7f642a209640 1 --2- 172.21.15.98:0/2066553455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64241088c0 0x7f642411a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.126+0000 7f642a209640 1 -- 172.21.15.98:0/2066553455 >> 172.21.15.98:0/2066553455 conn(0x7f642406c810 msgr2=0x7f642411b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:47.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.127+0000 7f642a209640 1 -- 172.21.15.98:0/2066553455 shutdown_connections 2024-05-09T08:56:47.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.128+0000 7f642a209640 1 -- 172.21.15.98:0/2066553455 wait complete. 2024-05-09T08:56:47.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:56:47.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:46 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2468543265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:56:47.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:46 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2468543265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:56:47.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.441+0000 7f53b6e38640 1 -- 172.21.15.98:0/4050220528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b01088c0 msgr2=0x7f53b0108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.441+0000 7f53b6e38640 1 --2- 172.21.15.98:0/4050220528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b01088c0 0x7f53b0108d40 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009930 tx=0x7f53a402a440 comp rx=0 tx=0).stop 2024-05-09T08:56:47.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.441+0000 7f53b6e38640 1 -- 172.21.15.98:0/4050220528 shutdown_connections 2024-05-09T08:56:47.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.441+0000 7f53b6e38640 1 --2- 172.21.15.98:0/4050220528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53b011b010 0x7f53b011d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.441+0000 7f53b6e38640 1 --2- 172.21.15.98:0/4050220528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b01088c0 0x7f53b0108d40 secure :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009930 tx=0x7f53a402a440 comp rx=0 tx=0).stop 2024-05-09T08:56:47.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.441+0000 7f53b6e38640 1 --2- 172.21.15.98:0/4050220528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53b010fea0 0x7f53b0110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.441+0000 7f53b6e38640 1 -- 172.21.15.98:0/4050220528 >> 172.21.15.98:0/4050220528 conn(0x7f53b006c810 msgr2=0x7f53b006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:47.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.442+0000 7f53b6e38640 1 -- 172.21.15.98:0/4050220528 shutdown_connections 2024-05-09T08:56:47.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.443+0000 7f53b6e38640 1 -- 172.21.15.98:0/4050220528 wait complete. 2024-05-09T08:56:47.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.444+0000 7f53b6e38640 1 Processor -- start 2024-05-09T08:56:47.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.445+0000 7f53b6e38640 1 -- start start 2024-05-09T08:56:47.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.445+0000 7f53b6e38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b010fea0 0x7f53b011a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.445+0000 7f53b6e38640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53b011b010 0x7f53b01138a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.445+0000 7f53b6e38640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53b0113de0 0x7f53b0114290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.445+0000 7f53b6e38640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53b0120890 con 0x7f53b010fea0 2024-05-09T08:56:47.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.445+0000 7f53b6e38640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f53b0120710 con 0x7f53b011b010 2024-05-09T08:56:47.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.445+0000 7f53b6e38640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f53b0120a10 con 0x7f53b0113de0 2024-05-09T08:56:47.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.446+0000 7f53b4bd4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b010fea0 0x7f53b011a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.446+0000 7f53b4bd4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b010fea0 0x7f53b011a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57458/0 (socket says 172.21.15.98:57458) 2024-05-09T08:56:47.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.446+0000 7f53b4bd4640 1 -- 172.21.15.98:0/1570111932 learned_addr learned my addr 172.21.15.98:0/1570111932 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:47.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.447+0000 7f53b4bd4640 1 -- 172.21.15.98:0/1570111932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53b0113de0 msgr2=0x7f53b0114290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.447+0000 7f53b4bd4640 1 --2- 172.21.15.98:0/1570111932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53b0113de0 0x7f53b0114290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.447+0000 7f53b4bd4640 1 -- 172.21.15.98:0/1570111932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53b011b010 msgr2=0x7f53b01138a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:56:47.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.447+0000 7f53b4bd4640 1 --2- 172.21.15.98:0/1570111932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53b011b010 0x7f53b01138a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.447+0000 7f53b4bd4640 1 -- 172.21.15.98:0/1570111932 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b0118470 con 0x7f53b010fea0 2024-05-09T08:56:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.453+0000 7f53b4bd4640 1 --2- 172.21.15.98:0/1570111932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b010fea0 0x7f53b011a820 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f53a000d950 tx=0x7f53a000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.456+0000 7f53adffb640 1 -- 172.21.15.98:0/1570111932 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0017070 con 0x7f53b010fea0 2024-05-09T08:56:47.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.457+0000 7f53adffb640 1 -- 172.21.15.98:0/1570111932 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a0009c30 con 0x7f53b010fea0 2024-05-09T08:56:47.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.457+0000 7f53adffb640 1 -- 172.21.15.98:0/1570111932 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a000f6b0 con 0x7f53b010fea0 2024-05-09T08:56:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.457+0000 7f53b6e38640 1 -- 172.21.15.98:0/1570111932 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b0118760 con 0x7f53b010fea0 2024-05-09T08:56:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.458+0000 7f53b6e38640 1 -- 172.21.15.98:0/1570111932 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b0118c60 con 0x7f53b010fea0 2024-05-09T08:56:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.463+0000 7f53b9c27740 1 Processor -- start 2024-05-09T08:56:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.465+0000 7f53b9c27740 1 -- start start 2024-05-09T08:56:47.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.465+0000 7f53b9c27740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffe520 0x558698ffe900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.465+0000 7f53b9c27740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558698ffee40 0x558699008fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.465+0000 7f53b9c27740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558699009510 0x55869900b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.465+0000 7f53b9c27740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558698f63c40 con 0x558699009510 2024-05-09T08:56:47.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.465+0000 7f53b9c27740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558698f0b580 con 0x558698ffe520 2024-05-09T08:56:47.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.465+0000 7f53b9c27740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558698f5fc00 con 0x558698ffee40 2024-05-09T08:56:47.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.470+0000 7f53b53d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffe520 0x558698ffe900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.474+0000 7f53adffb640 1 -- 172.21.15.98:0/1570111932 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f53a000f850 con 0x7f53b010fea0 2024-05-09T08:56:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.474+0000 7f53b53d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffe520 0x558698ffe900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54480/0 (socket says 172.21.15.98:54480) 2024-05-09T08:56:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.474+0000 7f53b53d5640 1 -- 172.21.15.98:0/2029366199 learned_addr learned my addr 172.21.15.98:0/2029366199 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.474+0000 7f53b4bd4640 1 --2- 172.21.15.98:0/2029366199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558698ffee40 0x558699008fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.475+0000 7f53affff640 1 --2- 172.21.15.98:0/2029366199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558699009510 0x55869900b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.475+0000 7f53b53d5640 1 -- 172.21.15.98:0/2029366199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558698ffee40 msgr2=0x558699008fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.479+0000 7f53b53d5640 1 --2- 172.21.15.98:0/2029366199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558698ffee40 0x558699008fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.479+0000 7f53b53d5640 1 -- 172.21.15.98:0/2029366199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558699009510 msgr2=0x55869900b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.479+0000 7f53b53d5640 1 --2- 172.21.15.98:0/2029366199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558699009510 0x55869900b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.479+0000 7f53b53d5640 1 -- 172.21.15.98:0/2029366199 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558698c773f0 con 0x558698ffe520 2024-05-09T08:56:47.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.479+0000 7f53b4bd4640 1 --2- 172.21.15.98:0/2029366199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558698ffee40 0x558699008fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:56:47.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.480+0000 7f53b53d5640 1 --2- 172.21.15.98:0/2029366199 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffe520 0x558698ffe900 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f53a8007c60 tx=0x7f53a800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5c57e167aa1c11 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.480+0000 7f53adffb640 1 --2- 172.21.15.98:0/1570111932 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f53980a2f00 0x7f53980a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.481+0000 7f5396ffd640 1 -- 172.21.15.98:0/2029366199 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8003b70 con 0x558698ffe520 2024-05-09T08:56:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.481+0000 7f5396ffd640 1 -- 172.21.15.98:0/2029366199 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a8003d10 con 0x558698ffe520 2024-05-09T08:56:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.481+0000 7f5396ffd640 1 -- 172.21.15.98:0/2029366199 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a800f8c0 con 0x558698ffe520 2024-05-09T08:56:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.481+0000 7f53b4bd4640 1 --2- 172.21.15.98:0/1570111932 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f53980a2f00 0x7f53980a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.482+0000 7f53adffb640 1 -- 172.21.15.98:0/1570111932 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f53a00cf370 con 0x7f53b010fea0 2024-05-09T08:56:47.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.482+0000 7f53b9c27740 1 -- 172.21.15.98:0/2029366199 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffe520 msgr2=0x558698ffe900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.482+0000 7f53b9c27740 1 --2- 172.21.15.98:0/2029366199 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffe520 0x558698ffe900 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f53a8007c60 tx=0x7f53a800c6a0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.482+0000 7f53b4bd4640 1 --2- 172.21.15.98:0/1570111932 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f53980a2f00 0x7f53980a53c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x558698fffae0 tx=0x7f53a00d4040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.485+0000 7f53b9c27740 1 -- 172.21.15.98:0/2029366199 shutdown_connections 2024-05-09T08:56:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.485+0000 7f53b9c27740 1 --2- 172.21.15.98:0/2029366199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558699009510 0x55869900b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.485+0000 7f53b9c27740 1 --2- 172.21.15.98:0/2029366199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558698ffee40 0x558699008fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.485+0000 7f53b9c27740 1 --2- 172.21.15.98:0/2029366199 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffe520 0x558698ffe900 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.485+0000 7f53b9c27740 1 -- 172.21.15.98:0/2029366199 >> 172.21.15.98:0/2029366199 conn(0x558698ffa840 msgr2=0x558698ffac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.485+0000 7f53b9c27740 1 -- 172.21.15.98:0/2029366199 shutdown_connections 2024-05-09T08:56:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.485+0000 7f53b9c27740 1 -- 172.21.15.98:0/2029366199 wait complete. 2024-05-09T08:56:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.486+0000 7f53b9c27740 1 -- start start 2024-05-09T08:56:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.486+0000 7f53b9c27740 1 Processor -- start 2024-05-09T08:56:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.487+0000 7f53b9c27740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffe520 0x558699006430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.487+0000 7f53b9c27740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffee40 0x558699006970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.487+0000 7f53b9c27740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558699009510 0x558699006eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.487+0000 7f53b9c27740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55869900c350 con 0x558698ffe520 2024-05-09T08:56:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.487+0000 7f53b9c27740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558698f1ec40 con 0x558698ffee40 2024-05-09T08:56:47.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.487+0000 7f53b9c27740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55869900c4d0 con 0x558699009510 2024-05-09T08:56:47.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.487+0000 7f53b9c27740 10 client.0 _ll_register_callbacks cb 0x558698c76be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:47.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.487+0000 7f53b53d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffe520 0x558699006430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.487+0000 7f53b53d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffe520 0x558699006430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57482/0 (socket says 172.21.15.98:57482) 2024-05-09T08:56:47.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.487+0000 7f53b53d5640 1 -- 172.21.15.98:0/147382681 learned_addr learned my addr 172.21.15.98:0/147382681 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:47.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.488+0000 7f53b53d5640 1 -- 172.21.15.98:0/147382681 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558699009510 msgr2=0x558699006eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.488+0000 7f53b53d5640 1 --2- 172.21.15.98:0/147382681 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558699009510 0x558699006eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.488+0000 7f53b53d5640 1 -- 172.21.15.98:0/147382681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffee40 msgr2=0x558699006970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.488+0000 7f53b53d5640 1 --2- 172.21.15.98:0/147382681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffee40 0x558699006970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.488+0000 7f53b53d5640 1 -- 172.21.15.98:0/147382681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558698d417d0 con 0x558698ffe520 2024-05-09T08:56:47.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.488+0000 7f53b53d5640 1 --2- 172.21.15.98:0/147382681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffe520 0x558699006430 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f53a8003f70 tx=0x7f53a800f4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56c14a21fd09a4b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.489+0000 7f538ffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:56:47.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.490+0000 7f538ffff640 1 -- 172.21.15.98:0/147382681 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8004d20 con 0x558698ffe520 2024-05-09T08:56:47.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.491+0000 7f538ffff640 1 -- 172.21.15.98:0/147382681 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f53a8004210 con 0x558698ffe520 2024-05-09T08:56:47.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.491+0000 7f53b9c27740 1 -- 172.21.15.98:0/147382681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53a8003d10 con 0x558698ffe520 2024-05-09T08:56:47.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.493+0000 7f538ffff640 1 -- 172.21.15.98:0/147382681 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f53a8004780 con 0x558698ffe520 2024-05-09T08:56:47.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.493+0000 7f53b9c27740 10 client.17250 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:47.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.493+0000 7f53b9c27740 1 -- 172.21.15.98:0/147382681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5586990184b0 con 0x558698ffe520 2024-05-09T08:56:47.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.495+0000 7f538ffff640 1 -- 172.21.15.98:0/147382681 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a8003d10 con 0x558698ffe520 2024-05-09T08:56:47.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.495+0000 7f538ffff640 1 -- 172.21.15.98:0/147382681 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f53a802f590 con 0x558698ffe520 2024-05-09T08:56:47.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.496+0000 7f53b9c27740 10 client.17250 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:47.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.496+0000 7f53b9c27740 10 client.17250 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:56:47.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.496+0000 7f53b9c27740 20 client.17250 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:47.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.496+0000 7f53b9c27740 1 --2- 172.21.15.98:0/147382681 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558699019d00 0x55869901c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.496+0000 7f53b9c27740 4 client.17250 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:47.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.496+0000 7f53b9c27740 1 -- 172.21.15.98:0/147382681 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558698f71370 con 0x558699019d00 2024-05-09T08:56:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.499+0000 7f53affff640 1 --2- 172.21.15.98:0/147382681 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558699019d00 0x55869901c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.503+0000 7f538ffff640 10 client.17250 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:56:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.503+0000 7f53affff640 1 --2- 172.21.15.98:0/147382681 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558699019d00 0x55869901c180 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3ea45636339f92cf server_cookie=e8e27745b482cb9b in_seq=0 out_seq=0 2024-05-09T08:56:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.506+0000 7f538ffff640 1 -- 172.21.15.98:0/147382681 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558698f71370 con 0x558699019d00 2024-05-09T08:56:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.506+0000 7f538ffff640 10 client.17250 handle_command_reply: tid=0 2024-05-09T08:56:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.506+0000 7f53b9c27740 1 client.17250 shutdown 2024-05-09T08:56:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.518+0000 7f53b9c27740 1 -- 172.21.15.98:0/147382681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffe520 msgr2=0x558699006430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.518+0000 7f53b9c27740 1 --2- 172.21.15.98:0/147382681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffe520 0x558699006430 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f53a8003f70 tx=0x7f53a800f4e0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.518+0000 7f53b9c27740 1 -- 172.21.15.98:0/147382681 shutdown_connections 2024-05-09T08:56:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.518+0000 7f53b9c27740 1 --2- 172.21.15.98:0/147382681 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558699019d00 0x55869901c180 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.518+0000 7f53b9c27740 1 --2- 172.21.15.98:0/147382681 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558699009510 0x558699006eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.518+0000 7f53b9c27740 1 --2- 172.21.15.98:0/147382681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffee40 0x558699006970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.518+0000 7f53b9c27740 1 --2- 172.21.15.98:0/147382681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffe520 0x558699006430 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.518+0000 7f53b9c27740 1 -- 172.21.15.98:0/147382681 >> 172.21.15.98:0/147382681 conn(0x558698ffa840 msgr2=0x558699000500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.518+0000 7f538ffff640 0 client.17250 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:56:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.523+0000 7f53b9c27740 1 -- 172.21.15.98:0/147382681 shutdown_connections 2024-05-09T08:56:47.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.524+0000 7f53b9c27740 1 -- 172.21.15.98:0/147382681 wait complete. 2024-05-09T08:56:47.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.524+0000 7f53b9c27740 20 client.17250 trim_cache size 0 max 16384 2024-05-09T08:56:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.580+0000 7f53b9c27740 1 Processor -- start 2024-05-09T08:56:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.582+0000 7f53b9c27740 1 -- start start 2024-05-09T08:56:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.584+0000 7f53b9c27740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffe520 0x558699015270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.585+0000 7f53b9c27740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffee40 0x5586990055b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.585+0000 7f53b9c27740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558699009510 0x558699000c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.585+0000 7f53b53d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffe520 0x558699015270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.585+0000 7f53b53d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffe520 0x558699015270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54486/0 (socket says 172.21.15.98:54486) 2024-05-09T08:56:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.585+0000 7f53b53d5640 1 -- 172.21.15.98:0/1351499233 learned_addr learned my addr 172.21.15.98:0/1351499233 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.585+0000 7f53b9c27740 1 -- 172.21.15.98:0/1351499233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f539c010090 con 0x558698ffee40 2024-05-09T08:56:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.585+0000 7f53b9c27740 1 -- 172.21.15.98:0/1351499233 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f539c011bd0 con 0x558698ffe520 2024-05-09T08:56:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.585+0000 7f53b9c27740 1 -- 172.21.15.98:0/1351499233 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f539c010760 con 0x558699009510 2024-05-09T08:56:47.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.586+0000 7f53b53d5640 1 --2- 172.21.15.98:0/1351499233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558699009510 0x558699000c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.586+0000 7f53affff640 1 --2- 172.21.15.98:0/1351499233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffee40 0x5586990055b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.587+0000 7f53affff640 1 -- 172.21.15.98:0/1351499233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558699009510 msgr2=0x558699000c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.588+0000 7f53affff640 1 --2- 172.21.15.98:0/1351499233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558699009510 0x558699000c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.588+0000 7f53affff640 1 -- 172.21.15.98:0/1351499233 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffe520 msgr2=0x558699015270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.588+0000 7f53affff640 1 --2- 172.21.15.98:0/1351499233 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffe520 0x558699015270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.588+0000 7f53affff640 1 -- 172.21.15.98:0/1351499233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586990011c0 con 0x558698ffee40 2024-05-09T08:56:47.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.588+0000 7f53b53d5640 1 --2- 172.21.15.98:0/1351499233 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffe520 0x558699015270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:56:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.589+0000 7f53affff640 1 --2- 172.21.15.98:0/1351499233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffee40 0x5586990055b0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f53a4008000 tx=0x7f53a4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d2ebf767bb387a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.590+0000 7f53967fc640 1 -- 172.21.15.98:0/1351499233 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4035020 con 0x558698ffee40 2024-05-09T08:56:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.590+0000 7f53967fc640 1 -- 172.21.15.98:0/1351499233 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a4004040 con 0x558698ffee40 2024-05-09T08:56:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.590+0000 7f53967fc640 1 -- 172.21.15.98:0/1351499233 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4031b60 con 0x558698ffee40 2024-05-09T08:56:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.591+0000 7f53b9c27740 1 -- 172.21.15.98:0/1351499233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffee40 msgr2=0x5586990055b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.591+0000 7f53b9c27740 1 --2- 172.21.15.98:0/1351499233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffee40 0x5586990055b0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f53a4008000 tx=0x7f53a4005fd0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.593+0000 7f53b9c27740 1 -- 172.21.15.98:0/1351499233 shutdown_connections 2024-05-09T08:56:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.593+0000 7f53b9c27740 1 --2- 172.21.15.98:0/1351499233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558699009510 0x558699000c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.593+0000 7f53b9c27740 1 --2- 172.21.15.98:0/1351499233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffee40 0x5586990055b0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.593+0000 7f53b9c27740 1 --2- 172.21.15.98:0/1351499233 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffe520 0x558699015270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.593+0000 7f53b9c27740 1 -- 172.21.15.98:0/1351499233 >> 172.21.15.98:0/1351499233 conn(0x558698ffa840 msgr2=0x558698ff9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.594+0000 7f53b9c27740 1 -- 172.21.15.98:0/1351499233 shutdown_connections 2024-05-09T08:56:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.594+0000 7f53b9c27740 1 -- 172.21.15.98:0/1351499233 wait complete. 2024-05-09T08:56:47.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.594+0000 7f53b9c27740 1 -- start start 2024-05-09T08:56:47.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.595+0000 7f53b9c27740 1 Processor -- start 2024-05-09T08:56:47.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.595+0000 7f53b9c27740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffe520 0x558699011b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.596+0000 7f53b9c27740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffee40 0x558699002dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.596+0000 7f53b9c27740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558699009510 0x558699003300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.596+0000 7f53b9c27740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f539c011500 con 0x558698ffe520 2024-05-09T08:56:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.596+0000 7f53b9c27740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f539c010e30 con 0x558698ffee40 2024-05-09T08:56:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.596+0000 7f53b9c27740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f539c00e070 con 0x558699009510 2024-05-09T08:56:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.596+0000 7f53affff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffee40 0x558699002dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.596+0000 7f53affff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffee40 0x558699002dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54490/0 (socket says 172.21.15.98:54490) 2024-05-09T08:56:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.596+0000 7f53affff640 1 -- 172.21.15.98:0/2008263301 learned_addr learned my addr 172.21.15.98:0/2008263301 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.597+0000 7f53b53d5640 1 --2- 172.21.15.98:0/2008263301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffe520 0x558699011b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.597+0000 7f53b9c27740 10 client.0 _ll_register_callbacks cb 0x7f53a4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.597+0000 7f53b53d5640 1 --2- 172.21.15.98:0/2008263301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558699009510 0x558699003300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.598+0000 7f53affff640 1 -- 172.21.15.98:0/2008263301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558699009510 msgr2=0x558699003300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.598+0000 7f53affff640 1 --2- 172.21.15.98:0/2008263301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558699009510 0x558699003300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.598+0000 7f53affff640 1 -- 172.21.15.98:0/2008263301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffe520 msgr2=0x558699011b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.598+0000 7f53affff640 1 --2- 172.21.15.98:0/2008263301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffe520 0x558699011b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.598+0000 7f53affff640 1 -- 172.21.15.98:0/2008263301 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558699003b00 con 0x558698ffee40 2024-05-09T08:56:47.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.598+0000 7f53affff640 1 --2- 172.21.15.98:0/2008263301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffee40 0x558699002dc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f53a4005940 tx=0x7f53a4004420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83312b981587b40d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.599+0000 7f538ffff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:56:47.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.599+0000 7f538ffff640 1 -- 172.21.15.98:0/2008263301 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4051060 con 0x558698ffee40 2024-05-09T08:56:47.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.599+0000 7f538ffff640 1 -- 172.21.15.98:0/2008263301 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f53a4031a90 con 0x558698ffee40 2024-05-09T08:56:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.599+0000 7f53b9c27740 1 -- 172.21.15.98:0/2008263301 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53a4004040 con 0x558698ffee40 2024-05-09T08:56:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.600+0000 7f538ffff640 1 -- 172.21.15.98:0/2008263301 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f53a4058050 con 0x558698ffee40 2024-05-09T08:56:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.600+0000 7f53b9c27740 10 client.26561 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.600+0000 7f53b9c27740 1 -- 172.21.15.98:0/2008263301 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558699080290 con 0x558698ffee40 2024-05-09T08:56:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.602+0000 7f538ffff640 1 -- 172.21.15.98:0/2008263301 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a4031cd0 con 0x558698ffee40 2024-05-09T08:56:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.602+0000 7f538ffff640 1 -- 172.21.15.98:0/2008263301 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f53a403a040 con 0x558698ffee40 2024-05-09T08:56:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.602+0000 7f53b9c27740 10 client.26561 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.602+0000 7f53b9c27740 10 client.26561 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:56:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.602+0000 7f53b9c27740 20 client.26561 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.602+0000 7f53b9c27740 1 --2- 172.21.15.98:0/2008263301 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558699080a00 0x558699082e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.603+0000 7f53b9c27740 4 client.26561 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:56:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.603+0000 7f53b9c27740 1 -- 172.21.15.98:0/2008263301 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f53a4004040 con 0x558699080a00 2024-05-09T08:56:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.603+0000 7f53b53d5640 1 --2- 172.21.15.98:0/2008263301 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558699080a00 0x558699082e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.610+0000 7f53b53d5640 1 --2- 172.21.15.98:0/2008263301 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558699080a00 0x558699082e20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1eeb261c4479fb1b server_cookie=66b28a4b0c620c44 in_seq=0 out_seq=0 2024-05-09T08:56:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.612+0000 7f538ffff640 10 client.26561 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:56:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.612+0000 7f538ffff640 1 -- 172.21.15.98:0/2008263301 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f53a4004040 con 0x558699080a00 2024-05-09T08:56:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.612+0000 7f538ffff640 10 client.26561 handle_command_reply: tid=0 2024-05-09T08:56:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.612+0000 7f53b9c27740 1 client.26561 shutdown 2024-05-09T08:56:47.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.614+0000 7f53b9c27740 1 -- 172.21.15.98:0/2008263301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffee40 msgr2=0x558699002dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.614+0000 7f53b9c27740 1 --2- 172.21.15.98:0/2008263301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffee40 0x558699002dc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f53a4005940 tx=0x7f53a4004420 comp rx=0 tx=0).stop 2024-05-09T08:56:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.614+0000 7f53b9c27740 1 -- 172.21.15.98:0/2008263301 shutdown_connections 2024-05-09T08:56:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.614+0000 7f53b9c27740 1 --2- 172.21.15.98:0/2008263301 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558699080a00 0x558699082e20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.615+0000 7f53b9c27740 1 --2- 172.21.15.98:0/2008263301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558699009510 0x558699003300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.615+0000 7f53b9c27740 1 --2- 172.21.15.98:0/2008263301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558698ffee40 0x558699002dc0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.615+0000 7f53b9c27740 1 --2- 172.21.15.98:0/2008263301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558698ffe520 0x558699011b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.615+0000 7f53b9c27740 1 -- 172.21.15.98:0/2008263301 >> 172.21.15.98:0/2008263301 conn(0x558698ffa840 msgr2=0x558698ff9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.616+0000 7f538ffff640 0 client.26561 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:56:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.616+0000 7f53b9c27740 1 -- 172.21.15.98:0/2008263301 shutdown_connections 2024-05-09T08:56:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.616+0000 7f53b9c27740 1 -- 172.21.15.98:0/2008263301 wait complete. 2024-05-09T08:56:47.616 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:56:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.616+0000 7f53b9c27740 20 client.26561 trim_cache size 0 max 16384 2024-05-09T08:56:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.617+0000 7f53b6e38640 1 -- 172.21.15.98:0/1570111932 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f53980a2f00 msgr2=0x7f53980a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.617+0000 7f53b6e38640 1 --2- 172.21.15.98:0/1570111932 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f53980a2f00 0x7f53980a53c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x558698fffae0 tx=0x7f53a00d4040 comp rx=0 tx=0).stop 2024-05-09T08:56:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.617+0000 7f53b6e38640 1 -- 172.21.15.98:0/1570111932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b010fea0 msgr2=0x7f53b011a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.617+0000 7f53b6e38640 1 --2- 172.21.15.98:0/1570111932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b010fea0 0x7f53b011a820 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f53a000d950 tx=0x7f53a000de10 comp rx=0 tx=0).stop 2024-05-09T08:56:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.617+0000 7f53b6e38640 1 -- 172.21.15.98:0/1570111932 shutdown_connections 2024-05-09T08:56:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.617+0000 7f53b6e38640 1 --2- 172.21.15.98:0/1570111932 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f53980a2f00 0x7f53980a53c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.617+0000 7f53b6e38640 1 --2- 172.21.15.98:0/1570111932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53b0113de0 0x7f53b0114290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.617+0000 7f53b6e38640 1 --2- 172.21.15.98:0/1570111932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53b011b010 0x7f53b01138a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.617+0000 7f53b6e38640 1 --2- 172.21.15.98:0/1570111932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b010fea0 0x7f53b011a820 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.617+0000 7f53b6e38640 1 -- 172.21.15.98:0/1570111932 >> 172.21.15.98:0/1570111932 conn(0x7f53b006c810 msgr2=0x7f53b011b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.618+0000 7f53b6e38640 1 -- 172.21.15.98:0/1570111932 shutdown_connections 2024-05-09T08:56:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.618+0000 7f53b6e38640 1 -- 172.21.15.98:0/1570111932 wait complete. 2024-05-09T08:56:47.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:56:47.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.931+0000 7f834d99e640 1 -- 172.21.15.98:0/1665409349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8348108830 msgr2=0x7f8348108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.931+0000 7f834d99e640 1 --2- 172.21.15.98:0/1665409349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8348108830 0x7f8348108c90 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f833c009900 tx=0x7f833c02a490 comp rx=0 tx=0).stop 2024-05-09T08:56:47.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.936+0000 7f834d99e640 1 -- 172.21.15.98:0/1665409349 shutdown_connections 2024-05-09T08:56:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.936+0000 7f834d99e640 1 --2- 172.21.15.98:0/1665409349 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83481092b0 0x7f834811d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.936+0000 7f834d99e640 1 --2- 172.21.15.98:0/1665409349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8348108830 0x7f8348108c90 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.936+0000 7f834d99e640 1 --2- 172.21.15.98:0/1665409349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f834810fed0 0x7f83481102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.936+0000 7f834d99e640 1 -- 172.21.15.98:0/1665409349 >> 172.21.15.98:0/1665409349 conn(0x7f834806c7b0 msgr2=0x7f834806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:47.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.939+0000 7f834d99e640 1 -- 172.21.15.98:0/1665409349 shutdown_connections 2024-05-09T08:56:47.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.939+0000 7f834d99e640 1 -- 172.21.15.98:0/1665409349 wait complete. 2024-05-09T08:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.941+0000 7f834d99e640 1 Processor -- start 2024-05-09T08:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.941+0000 7f834d99e640 1 -- start start 2024-05-09T08:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.941+0000 7f834d99e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8348108830 0x7f83481a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.941+0000 7f834d99e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f83481092b0 0x7f83481a72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.941+0000 7f834d99e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834810fed0 0x7f83481ab600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.941+0000 7f834d99e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83481208b0 con 0x7f8348108830 2024-05-09T08:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.941+0000 7f834d99e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8348120730 con 0x7f83481092b0 2024-05-09T08:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.941+0000 7f834d99e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8348120a30 con 0x7f834810fed0 2024-05-09T08:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.944+0000 7f834c99c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8348108830 0x7f83481a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.944+0000 7f834c99c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8348108830 0x7f83481a6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57506/0 (socket says 172.21.15.98:57506) 2024-05-09T08:56:47.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.944+0000 7f834c99c640 1 -- 172.21.15.98:0/2012874314 learned_addr learned my addr 172.21.15.98:0/2012874314 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:47.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.945+0000 7f834d19d640 1 --2- 172.21.15.98:0/2012874314 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834810fed0 0x7f83481ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.946+0000 7f8347fff640 1 --2- 172.21.15.98:0/2012874314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f83481092b0 0x7f83481a72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.948+0000 7f834c99c640 1 -- 172.21.15.98:0/2012874314 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834810fed0 msgr2=0x7f83481ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.948+0000 7f834c99c640 1 --2- 172.21.15.98:0/2012874314 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834810fed0 0x7f83481ab600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.948+0000 7f834c99c640 1 -- 172.21.15.98:0/2012874314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f83481092b0 msgr2=0x7f83481a72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.948+0000 7f834c99c640 1 --2- 172.21.15.98:0/2012874314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f83481092b0 0x7f83481a72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.948+0000 7f834c99c640 1 -- 172.21.15.98:0/2012874314 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83481abd30 con 0x7f8348108830 2024-05-09T08:56:47.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.955+0000 7f834c99c640 1 --2- 172.21.15.98:0/2012874314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8348108830 0x7f83481a6d90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f833800e9f0 tx=0x7f833800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.959+0000 7f8345ffb640 1 -- 172.21.15.98:0/2012874314 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833800cd80 con 0x7f8348108830 2024-05-09T08:56:47.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.959+0000 7f834d99e640 1 -- 172.21.15.98:0/2012874314 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83481ac020 con 0x7f8348108830 2024-05-09T08:56:47.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.959+0000 7f834d99e640 1 -- 172.21.15.98:0/2012874314 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8348079a90 con 0x7f8348108830 2024-05-09T08:56:47.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.960+0000 7f8345ffb640 1 -- 172.21.15.98:0/2012874314 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8338010910 con 0x7f8348108830 2024-05-09T08:56:47.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.960+0000 7f8345ffb640 1 -- 172.21.15.98:0/2012874314 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338018990 con 0x7f8348108830 2024-05-09T08:56:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.969+0000 7f8345ffb640 1 -- 172.21.15.98:0/2012874314 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8338018b30 con 0x7f8348108830 2024-05-09T08:56:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.970+0000 7f83519eb740 1 Processor -- start 2024-05-09T08:56:47.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.971+0000 7f83519eb740 1 -- start start 2024-05-09T08:56:47.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.971+0000 7f83519eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad44f0 0x55efadad48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.971+0000 7f83519eb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad4e10 0x55efadadefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.971+0000 7f83519eb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadadf4e0 0x55efadae18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.971+0000 7f83519eb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55efada39c40 con 0x55efadad4e10 2024-05-09T08:56:47.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.971+0000 7f83519eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55efad9e1580 con 0x55efadad44f0 2024-05-09T08:56:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.971+0000 7f83519eb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55efada35c00 con 0x55efadadf4e0 2024-05-09T08:56:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.972+0000 7f8345ffb640 1 --2- 172.21.15.98:0/2012874314 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f83300a2f10 0x7f83300a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.972+0000 7f834c99c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad44f0 0x55efadad48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.972+0000 7f8347fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad4e10 0x55efadadefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.972+0000 7f8347fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad4e10 0x55efadadefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57512/0 (socket says 172.21.15.98:57512) 2024-05-09T08:56:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.972+0000 7f8347fff640 1 -- 172.21.15.98:0/1530830866 learned_addr learned my addr 172.21.15.98:0/1530830866 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.972+0000 7f8345ffb640 1 -- 172.21.15.98:0/2012874314 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f83380d0b30 con 0x7f8348108830 2024-05-09T08:56:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.973+0000 7f8347fff640 1 -- 172.21.15.98:0/1530830866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadadf4e0 msgr2=0x55efadae18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.973+0000 7f8347fff640 1 --2- 172.21.15.98:0/1530830866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadadf4e0 0x55efadae18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.973+0000 7f8347fff640 1 -- 172.21.15.98:0/1530830866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad44f0 msgr2=0x55efadad48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.973+0000 7f8347fff640 1 --2- 172.21.15.98:0/1530830866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad44f0 0x55efadad48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.973+0000 7f8347fff640 1 -- 172.21.15.98:0/1530830866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efad74d3f0 con 0x55efadad4e10 2024-05-09T08:56:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.974+0000 7f8347fff640 1 --2- 172.21.15.98:0/1530830866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad4e10 0x55efadadefa0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f833c009230 tx=0x7f833c0315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af593dbb5a817f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.974+0000 7f832effd640 1 -- 172.21.15.98:0/1530830866 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c040070 con 0x55efadad4e10 2024-05-09T08:56:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.975+0000 7f832effd640 1 -- 172.21.15.98:0/1530830866 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c031920 con 0x55efadad4e10 2024-05-09T08:56:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.975+0000 7f834d19d640 1 --2- 172.21.15.98:0/2012874314 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f83300a2f10 0x7f83300a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.976+0000 7f83519eb740 1 -- 172.21.15.98:0/1530830866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad4e10 msgr2=0x55efadadefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.976+0000 7f83519eb740 1 --2- 172.21.15.98:0/1530830866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad4e10 0x55efadadefa0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f833c009230 tx=0x7f833c0315b0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.977+0000 7f83519eb740 1 -- 172.21.15.98:0/1530830866 shutdown_connections 2024-05-09T08:56:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.977+0000 7f83519eb740 1 --2- 172.21.15.98:0/1530830866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadadf4e0 0x55efadae18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.977+0000 7f83519eb740 1 --2- 172.21.15.98:0/1530830866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad4e10 0x55efadadefa0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.977+0000 7f83519eb740 1 --2- 172.21.15.98:0/1530830866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad44f0 0x55efadad48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.977+0000 7f83519eb740 1 -- 172.21.15.98:0/1530830866 >> 172.21.15.98:0/1530830866 conn(0x55efadad0810 msgr2=0x55efadad0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.977+0000 7f83519eb740 1 -- 172.21.15.98:0/1530830866 shutdown_connections 2024-05-09T08:56:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.977+0000 7f83519eb740 1 -- 172.21.15.98:0/1530830866 wait complete. 2024-05-09T08:56:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.978+0000 7f83519eb740 1 -- start start 2024-05-09T08:56:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.978+0000 7f83519eb740 1 Processor -- start 2024-05-09T08:56:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.978+0000 7f83519eb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad44f0 0x55efadadef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.978+0000 7f83519eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad4e10 0x55efadadc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.978+0000 7f83519eb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadadf4e0 0x55efadadca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.978+0000 7f83519eb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55efadae22c0 con 0x55efadad44f0 2024-05-09T08:56:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.978+0000 7f83519eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55efad9f4c40 con 0x55efadad4e10 2024-05-09T08:56:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.979+0000 7f83519eb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55efadae2440 con 0x55efadadf4e0 2024-05-09T08:56:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.979+0000 7f83519eb740 10 client.0 _ll_register_callbacks cb 0x55efad74cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.979+0000 7f834c99c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad44f0 0x55efadadef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.979+0000 7f834d19d640 1 --2- 172.21.15.98:0/2012874314 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f83300a2f10 0x7f83300a53d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x55efadad19a0 tx=0x7f83400118d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.980+0000 7f8347fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad4e10 0x55efadadc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.980+0000 7f8347fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad4e10 0x55efadadc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54522/0 (socket says 172.21.15.98:54522) 2024-05-09T08:56:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.980+0000 7f8347fff640 1 -- 172.21.15.98:0/4143749746 learned_addr learned my addr 172.21.15.98:0/4143749746 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.980+0000 7f834d19d640 1 --2- 172.21.15.98:0/4143749746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadadf4e0 0x55efadadca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:47.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.981+0000 7f834c99c640 1 -- 172.21.15.98:0/4143749746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadadf4e0 msgr2=0x55efadadca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.981+0000 7f834c99c640 1 --2- 172.21.15.98:0/4143749746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadadf4e0 0x55efadadca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.981+0000 7f834c99c640 1 -- 172.21.15.98:0/4143749746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad4e10 msgr2=0x55efadadc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:47.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.981+0000 7f834c99c640 1 --2- 172.21.15.98:0/4143749746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad4e10 0x55efadadc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:47.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.981+0000 7f834c99c640 1 -- 172.21.15.98:0/4143749746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efad8177d0 con 0x55efadad44f0 2024-05-09T08:56:47.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.982+0000 7f834c99c640 1 --2- 172.21.15.98:0/4143749746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad44f0 0x55efadadef90 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f83380cde60 tx=0x7f833809c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5a1d6a591dd64c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:47.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.982+0000 7f8327fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:56:47.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.983+0000 7f8327fff640 1 -- 172.21.15.98:0/4143749746 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338026040 con 0x55efadad44f0 2024-05-09T08:56:47.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.983+0000 7f83519eb740 1 -- 172.21.15.98:0/4143749746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f833c031920 con 0x55efadad44f0 2024-05-09T08:56:47.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.983+0000 7f8327fff640 1 -- 172.21.15.98:0/4143749746 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f83380106d0 con 0x55efadad44f0 2024-05-09T08:56:47.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.984+0000 7f8327fff640 1 -- 172.21.15.98:0/4143749746 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f83380a58a0 con 0x55efadad44f0 2024-05-09T08:56:47.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.984+0000 7f83519eb740 10 client.17268 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:47.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.984+0000 7f83519eb740 1 -- 172.21.15.98:0/4143749746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55efadaff4c0 con 0x55efadad44f0 2024-05-09T08:56:47.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.985+0000 7f8327fff640 1 -- 172.21.15.98:0/4143749746 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c031920 con 0x55efadad44f0 2024-05-09T08:56:47.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.985+0000 7f8327fff640 1 -- 172.21.15.98:0/4143749746 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f8338010b50 con 0x55efadad44f0 2024-05-09T08:56:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.985+0000 7f83519eb740 10 client.17268 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.986+0000 7f83519eb740 10 client.17268 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:56:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.986+0000 7f83519eb740 20 client.17268 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.986+0000 7f83519eb740 1 --2- 172.21.15.98:0/4143749746 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55efadb00e80 0x55efadb03320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.986+0000 7f83519eb740 4 client.17268 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.986+0000 7f83519eb740 1 -- 172.21.15.98:0/4143749746 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efada47370 con 0x55efadb00e80 2024-05-09T08:56:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.987+0000 7f8347fff640 1 --2- 172.21.15.98:0/4143749746 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55efadb00e80 0x55efadb03320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.998+0000 7f8347fff640 1 --2- 172.21.15.98:0/4143749746 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55efadb00e80 0x55efadb03320 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7a5d0d8d7cd71929 server_cookie=58f9e119966ecea in_seq=0 out_seq=0 2024-05-09T08:56:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:47.998+0000 7f8327fff640 10 client.17268 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:56:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.000+0000 7f8327fff640 1 -- 172.21.15.98:0/4143749746 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55efada47370 con 0x55efadb00e80 2024-05-09T08:56:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.000+0000 7f8327fff640 10 client.17268 handle_command_reply: tid=0 2024-05-09T08:56:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.001+0000 7f83519eb740 1 client.17268 shutdown 2024-05-09T08:56:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.007+0000 7f83519eb740 1 -- 172.21.15.98:0/4143749746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad44f0 msgr2=0x55efadadef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.007+0000 7f83519eb740 1 --2- 172.21.15.98:0/4143749746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad44f0 0x55efadadef90 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f83380cde60 tx=0x7f833809c680 comp rx=0 tx=0).stop 2024-05-09T08:56:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.014+0000 7f83519eb740 1 -- 172.21.15.98:0/4143749746 shutdown_connections 2024-05-09T08:56:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.014+0000 7f83519eb740 1 --2- 172.21.15.98:0/4143749746 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55efadb00e80 0x55efadb03320 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.014+0000 7f83519eb740 1 --2- 172.21.15.98:0/4143749746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadadf4e0 0x55efadadca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.014+0000 7f83519eb740 1 --2- 172.21.15.98:0/4143749746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad4e10 0x55efadadc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.014+0000 7f83519eb740 1 --2- 172.21.15.98:0/4143749746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad44f0 0x55efadadef90 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.014+0000 7f83519eb740 1 -- 172.21.15.98:0/4143749746 >> 172.21.15.98:0/4143749746 conn(0x55efadad0810 msgr2=0x55efadad64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.014+0000 7f8327fff640 0 client.17268 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:56:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.015+0000 7f83519eb740 1 -- 172.21.15.98:0/4143749746 shutdown_connections 2024-05-09T08:56:48.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.015+0000 7f83519eb740 1 -- 172.21.15.98:0/4143749746 wait complete. 2024-05-09T08:56:48.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.015+0000 7f83519eb740 20 client.17268 trim_cache size 0 max 16384 2024-05-09T08:56:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.061+0000 7f83519eb740 1 Processor -- start 2024-05-09T08:56:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.066+0000 7f83519eb740 1 -- start start 2024-05-09T08:56:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.066+0000 7f83519eb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadad44f0 0x55efadaeb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.066+0000 7f83519eb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad4e10 0x55efadb08c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.066+0000 7f83519eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadadf4e0 0x55efadb09170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.066+0000 7f83519eb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83340100d0 con 0x55efadad4e10 2024-05-09T08:56:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.067+0000 7f83519eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8334011c10 con 0x55efadadf4e0 2024-05-09T08:56:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.067+0000 7f83519eb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f83340107a0 con 0x55efadad44f0 2024-05-09T08:56:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.068+0000 7f8347fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad4e10 0x55efadb08c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.068+0000 7f8347fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad4e10 0x55efadb08c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57528/0 (socket says 172.21.15.98:57528) 2024-05-09T08:56:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.068+0000 7f8347fff640 1 -- 172.21.15.98:0/1835353852 learned_addr learned my addr 172.21.15.98:0/1835353852 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.068+0000 7f8347fff640 1 -- 172.21.15.98:0/1835353852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadad44f0 msgr2=0x55efadaeb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.068+0000 7f8347fff640 1 --2- 172.21.15.98:0/1835353852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadad44f0 0x55efadaeb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.068+0000 7f8347fff640 1 -- 172.21.15.98:0/1835353852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadadf4e0 msgr2=0x55efadb09170 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:56:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.069+0000 7f8347fff640 1 --2- 172.21.15.98:0/1835353852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadadf4e0 0x55efadb09170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.069+0000 7f8347fff640 1 -- 172.21.15.98:0/1835353852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efadad0230 con 0x55efadad4e10 2024-05-09T08:56:48.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.069+0000 7f8347fff640 1 --2- 172.21.15.98:0/1835353852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad4e10 0x55efadb08c30 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f833c031550 tx=0x7f833c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a20f9818d78c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:48.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.070+0000 7f832e7fc640 1 -- 172.21.15.98:0/1835353852 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c040050 con 0x55efadad4e10 2024-05-09T08:56:48.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.070+0000 7f832e7fc640 1 -- 172.21.15.98:0/1835353852 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c004300 con 0x55efadad4e10 2024-05-09T08:56:48.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.070+0000 7f832e7fc640 1 -- 172.21.15.98:0/1835353852 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c003a20 con 0x55efadad4e10 2024-05-09T08:56:48.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.074+0000 7f83519eb740 1 -- 172.21.15.98:0/1835353852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad4e10 msgr2=0x55efadb08c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.074+0000 7f83519eb740 1 --2- 172.21.15.98:0/1835353852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad4e10 0x55efadb08c30 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f833c031550 tx=0x7f833c005850 comp rx=0 tx=0).stop 2024-05-09T08:56:48.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.074+0000 7f83519eb740 1 -- 172.21.15.98:0/1835353852 shutdown_connections 2024-05-09T08:56:48.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.074+0000 7f83519eb740 1 --2- 172.21.15.98:0/1835353852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadadf4e0 0x55efadb09170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.074+0000 7f83519eb740 1 --2- 172.21.15.98:0/1835353852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadad4e10 0x55efadb08c30 secure :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f833c031550 tx=0x7f833c005850 comp rx=0 tx=0).stop 2024-05-09T08:56:48.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.074+0000 7f83519eb740 1 --2- 172.21.15.98:0/1835353852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadad44f0 0x55efadaeb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.074+0000 7f83519eb740 1 -- 172.21.15.98:0/1835353852 >> 172.21.15.98:0/1835353852 conn(0x55efadad0810 msgr2=0x55efadacb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:48.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.076+0000 7f83519eb740 1 -- 172.21.15.98:0/1835353852 shutdown_connections 2024-05-09T08:56:48.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.076+0000 7f83519eb740 1 -- 172.21.15.98:0/1835353852 wait complete. 2024-05-09T08:56:48.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.077+0000 7f83519eb740 1 -- start start 2024-05-09T08:56:48.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.077+0000 7f83519eb740 1 Processor -- start 2024-05-09T08:56:48.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.078+0000 7f83519eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad44f0 0x55efadad8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.078+0000 7f83519eb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadadf4e0 0x55efadad9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.078+0000 7f83519eb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadae6770 0x55efadae6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.079+0000 7f83519eb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8334011540 con 0x55efadae6770 2024-05-09T08:56:48.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.079+0000 7f83519eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8334010e70 con 0x55efadad44f0 2024-05-09T08:56:48.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.079+0000 7f83519eb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f833400e0b0 con 0x55efadadf4e0 2024-05-09T08:56:48.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.079+0000 7f834d19d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadae6770 0x55efadae6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.079+0000 7f834d19d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadae6770 0x55efadae6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57540/0 (socket says 172.21.15.98:57540) 2024-05-09T08:56:48.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.079+0000 7f834d19d640 1 -- 172.21.15.98:0/2370566020 learned_addr learned my addr 172.21.15.98:0/2370566020 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:48.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.079+0000 7f834c99c640 1 --2- 172.21.15.98:0/2370566020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad44f0 0x55efadad8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.079+0000 7f83519eb740 10 client.0 _ll_register_callbacks cb 0x7f8338009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:48.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.079+0000 7f8347fff640 1 --2- 172.21.15.98:0/2370566020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadadf4e0 0x55efadad9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.080+0000 7f834d19d640 1 -- 172.21.15.98:0/2370566020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadadf4e0 msgr2=0x55efadad9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:48.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.080+0000 7f834d19d640 1 --2- 172.21.15.98:0/2370566020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadadf4e0 0x55efadad9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.080+0000 7f834d19d640 1 -- 172.21.15.98:0/2370566020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad44f0 msgr2=0x55efadad8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:48.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.080+0000 7f834d19d640 1 --2- 172.21.15.98:0/2370566020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad44f0 0x55efadad8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.080+0000 7f834d19d640 1 -- 172.21.15.98:0/2370566020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efadad0640 con 0x55efadae6770 2024-05-09T08:56:48.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.080+0000 7f834d19d640 1 --2- 172.21.15.98:0/2370566020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadae6770 0x55efadae6bf0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f8340003bf0 tx=0x7f8340009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1fda4393e5da8e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:48.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.080+0000 7f8327fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:56:48.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.081+0000 7f8327fff640 1 -- 172.21.15.98:0/2370566020 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340005040 con 0x55efadae6770 2024-05-09T08:56:48.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.081+0000 7f83519eb740 1 -- 172.21.15.98:0/2370566020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f833c004300 con 0x55efadae6770 2024-05-09T08:56:48.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.082+0000 7f8327fff640 1 -- 172.21.15.98:0/2370566020 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f83400051e0 con 0x55efadae6770 2024-05-09T08:56:48.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.082+0000 7f8327fff640 1 -- 172.21.15.98:0/2370566020 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8340030050 con 0x55efadae6770 2024-05-09T08:56:48.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.083+0000 7f83519eb740 10 client.17274 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:48.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.083+0000 7f83519eb740 1 -- 172.21.15.98:0/2370566020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55efadb674a0 con 0x55efadae6770 2024-05-09T08:56:48.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.084+0000 7f8327fff640 1 -- 172.21.15.98:0/2370566020 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c004300 con 0x55efadae6770 2024-05-09T08:56:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.088+0000 7f8327fff640 1 -- 172.21.15.98:0/2370566020 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f834001fb90 con 0x55efadae6770 2024-05-09T08:56:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.088+0000 7f83519eb740 10 client.17274 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:48.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:47 smithi098 ceph-mon[30092]: pgmap v189: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 829 MiB/s wr, 1.06k op/s 2024-05-09T08:56:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.088+0000 7f83519eb740 10 client.17274 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:56:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.088+0000 7f83519eb740 20 client.17274 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.089+0000 7f83519eb740 1 --2- 172.21.15.98:0/2370566020 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55efadb67ae0 0x55efadb69f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.089+0000 7f83519eb740 4 client.17274 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:56:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.089+0000 7f83519eb740 1 -- 172.21.15.98:0/2370566020 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55efada416c0 con 0x55efadb67ae0 2024-05-09T08:56:48.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.096+0000 7f834c99c640 1 --2- 172.21.15.98:0/2370566020 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55efadb67ae0 0x55efadb69f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.116+0000 7f834c99c640 1 --2- 172.21.15.98:0/2370566020 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55efadb67ae0 0x55efadb69f00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=751f06053b0e4e79 server_cookie=3802c6a49c7455cd in_seq=0 out_seq=0 2024-05-09T08:56:48.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.116+0000 7f8327fff640 10 client.17274 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:56:48.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.117+0000 7f8327fff640 1 -- 172.21.15.98:0/2370566020 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55efada416c0 con 0x55efadb67ae0 2024-05-09T08:56:48.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.117+0000 7f8327fff640 10 client.17274 handle_command_reply: tid=0 2024-05-09T08:56:48.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.117+0000 7f83519eb740 1 client.17274 shutdown 2024-05-09T08:56:48.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.120+0000 7f83519eb740 1 -- 172.21.15.98:0/2370566020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadae6770 msgr2=0x55efadae6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.120+0000 7f83519eb740 1 --2- 172.21.15.98:0/2370566020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadae6770 0x55efadae6bf0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f8340003bf0 tx=0x7f8340009500 comp rx=0 tx=0).stop 2024-05-09T08:56:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.121+0000 7f83519eb740 1 -- 172.21.15.98:0/2370566020 shutdown_connections 2024-05-09T08:56:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.121+0000 7f83519eb740 1 --2- 172.21.15.98:0/2370566020 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55efadb67ae0 0x55efadb69f00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.121+0000 7f83519eb740 1 --2- 172.21.15.98:0/2370566020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efadae6770 0x55efadae6bf0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.121+0000 7f83519eb740 1 --2- 172.21.15.98:0/2370566020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55efadadf4e0 0x55efadad9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.121+0000 7f83519eb740 1 --2- 172.21.15.98:0/2370566020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efadad44f0 0x55efadad8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.121+0000 7f8327fff640 0 client.17274 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:56:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.121+0000 7f83519eb740 1 -- 172.21.15.98:0/2370566020 >> 172.21.15.98:0/2370566020 conn(0x55efadad0810 msgr2=0x55efadad75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.121+0000 7f83519eb740 1 -- 172.21.15.98:0/2370566020 shutdown_connections 2024-05-09T08:56:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.121+0000 7f83519eb740 1 -- 172.21.15.98:0/2370566020 wait complete. 2024-05-09T08:56:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.121+0000 7f83519eb740 20 client.17274 trim_cache size 0 max 16384 2024-05-09T08:56:48.122 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:56:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.126+0000 7f834d99e640 1 -- 172.21.15.98:0/2012874314 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f83300a2f10 msgr2=0x7f83300a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.126+0000 7f834d99e640 1 --2- 172.21.15.98:0/2012874314 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f83300a2f10 0x7f83300a53d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x55efadad19a0 tx=0x7f83400118d0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.126+0000 7f834d99e640 1 -- 172.21.15.98:0/2012874314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8348108830 msgr2=0x7f83481a6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.126+0000 7f834d99e640 1 --2- 172.21.15.98:0/2012874314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8348108830 0x7f83481a6d90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f833800e9f0 tx=0x7f833800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.127+0000 7f834d99e640 1 -- 172.21.15.98:0/2012874314 shutdown_connections 2024-05-09T08:56:48.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.127+0000 7f834d99e640 1 --2- 172.21.15.98:0/2012874314 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f83300a2f10 0x7f83300a53d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.127+0000 7f834d99e640 1 --2- 172.21.15.98:0/2012874314 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834810fed0 0x7f83481ab600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.127+0000 7f834d99e640 1 --2- 172.21.15.98:0/2012874314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f83481092b0 0x7f83481a72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.127+0000 7f834d99e640 1 --2- 172.21.15.98:0/2012874314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8348108830 0x7f83481a6d90 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.127+0000 7f834d99e640 1 -- 172.21.15.98:0/2012874314 >> 172.21.15.98:0/2012874314 conn(0x7f834806c7b0 msgr2=0x7f8348070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:48.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.128+0000 7f834d99e640 1 -- 172.21.15.98:0/2012874314 shutdown_connections 2024-05-09T08:56:48.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.128+0000 7f834d99e640 1 -- 172.21.15.98:0/2012874314 wait complete. 2024-05-09T08:56:48.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:47 smithi142 ceph-mon[34625]: pgmap v189: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 829 MiB/s wr, 1.06k op/s 2024-05-09T08:56:48.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:56:48.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:47 smithi105 ceph-mon[34498]: pgmap v189: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 829 MiB/s wr, 1.06k op/s 2024-05-09T08:56:48.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.427+0000 7fa1f339e640 1 -- 172.21.15.98:0/1097881670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec10e0b0 msgr2=0x7fa1ec10e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.427+0000 7fa1f339e640 1 --2- 172.21.15.98:0/1097881670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec10e0b0 0x7fa1ec10e530 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8009930 tx=0x7fa1e802a460 comp rx=0 tx=0).stop 2024-05-09T08:56:48.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.427+0000 7fa1f339e640 1 -- 172.21.15.98:0/1097881670 shutdown_connections 2024-05-09T08:56:48.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.427+0000 7fa1f339e640 1 --2- 172.21.15.98:0/1097881670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1ec11b010 0x7fa1ec11d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.427+0000 7fa1f339e640 1 --2- 172.21.15.98:0/1097881670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec10e0b0 0x7fa1ec10e530 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.427+0000 7fa1f339e640 1 --2- 172.21.15.98:0/1097881670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1ec110080 0x7fa1ec10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.427+0000 7fa1f339e640 1 -- 172.21.15.98:0/1097881670 >> 172.21.15.98:0/1097881670 conn(0x7fa1ec06c940 msgr2=0x7fa1ec06cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:48.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.430+0000 7fa1f339e640 1 -- 172.21.15.98:0/1097881670 shutdown_connections 2024-05-09T08:56:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.432+0000 7fa1f339e640 1 -- 172.21.15.98:0/1097881670 wait complete. 2024-05-09T08:56:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.433+0000 7fa1f339e640 1 Processor -- start 2024-05-09T08:56:48.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.434+0000 7fa1f339e640 1 -- start start 2024-05-09T08:56:48.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.435+0000 7fa1f339e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec10e0b0 0x7fa1ec118660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.435+0000 7fa1f339e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1ec110080 0x7fa1ec1116e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.435+0000 7fa1f339e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1ec11b010 0x7fa1ec111c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.435+0000 7fa1f339e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec120a40 con 0x7fa1ec10e0b0 2024-05-09T08:56:48.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.435+0000 7fa1f339e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec1208c0 con 0x7fa1ec11b010 2024-05-09T08:56:48.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.435+0000 7fa1f339e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec120bc0 con 0x7fa1ec110080 2024-05-09T08:56:48.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.437+0000 7fa1f193b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1ec11b010 0x7fa1ec111c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.437+0000 7fa1f0939640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1ec110080 0x7fa1ec1116e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.437+0000 7fa1f0939640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1ec110080 0x7fa1ec1116e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:36692/0 (socket says 172.21.15.98:36692) 2024-05-09T08:56:48.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.437+0000 7fa1f0939640 1 -- 172.21.15.98:0/985134962 learned_addr learned my addr 172.21.15.98:0/985134962 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:48.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.439+0000 7fa1f193b640 1 -- 172.21.15.98:0/985134962 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1ec110080 msgr2=0x7fa1ec1116e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.439+0000 7fa1f193b640 1 --2- 172.21.15.98:0/985134962 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1ec110080 0x7fa1ec1116e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.439+0000 7fa1f193b640 1 -- 172.21.15.98:0/985134962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec10e0b0 msgr2=0x7fa1ec118660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:56:48.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.439+0000 7fa1f193b640 1 --2- 172.21.15.98:0/985134962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec10e0b0 0x7fa1ec118660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.439+0000 7fa1f193b640 1 -- 172.21.15.98:0/985134962 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ec112260 con 0x7fa1ec11b010 2024-05-09T08:56:48.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.443+0000 7fa1f193b640 1 --2- 172.21.15.98:0/985134962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1ec11b010 0x7fa1ec111c20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400ca30 tx=0x7fa1e400cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:48.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.450+0000 7fa1e27fc640 1 -- 172.21.15.98:0/985134962 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4007d90 con 0x7fa1ec11b010 2024-05-09T08:56:48.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.450+0000 7fa1f339e640 1 -- 172.21.15.98:0/985134962 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1ec116180 con 0x7fa1ec11b010 2024-05-09T08:56:48.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.450+0000 7fa1f339e640 1 -- 172.21.15.98:0/985134962 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1ec1166a0 con 0x7fa1ec11b010 2024-05-09T08:56:48.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.450+0000 7fa1e27fc640 1 -- 172.21.15.98:0/985134962 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1e4005020 con 0x7fa1ec11b010 2024-05-09T08:56:48.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.451+0000 7fa1e27fc640 1 -- 172.21.15.98:0/985134962 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e400f780 con 0x7fa1ec11b010 2024-05-09T08:56:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.454+0000 7fa1f6188740 1 Processor -- start 2024-05-09T08:56:48.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.456+0000 7fa1e27fc640 1 -- 172.21.15.98:0/985134962 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa1e4004a00 con 0x7fa1ec11b010 2024-05-09T08:56:48.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.458+0000 7fa1f6188740 1 -- start start 2024-05-09T08:56:48.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.458+0000 7fa1e27fc640 1 --2- 172.21.15.98:0/985134962 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa1d80a2de0 0x7fa1d80a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.458+0000 7fa1e27fc640 1 -- 172.21.15.98:0/985134962 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fa1e40cf9a0 con 0x7fa1ec11b010 2024-05-09T08:56:48.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.460+0000 7fa1f6188740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5b330 0x55801ac5b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.461+0000 7fa1f0939640 1 --2- 172.21.15.98:0/985134962 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa1d80a2de0 0x7fa1d80a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.461+0000 7fa1f6188740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac5bc50 0x55801ac65de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.461+0000 7fa1f6188740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac66320 0x55801ac68710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.461+0000 7fa1f6188740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55801abc0c40 con 0x55801ac66320 2024-05-09T08:56:48.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.461+0000 7fa1f6188740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55801ab68580 con 0x55801ac5bc50 2024-05-09T08:56:48.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.461+0000 7fa1f6188740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55801abbcc00 con 0x55801ac5b330 2024-05-09T08:56:48.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.465+0000 7fa1f193b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac66320 0x55801ac68710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.465+0000 7fa1f193b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac66320 0x55801ac68710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57554/0 (socket says 172.21.15.98:57554) 2024-05-09T08:56:48.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.465+0000 7fa1f193b640 1 -- 172.21.15.98:0/3837320406 learned_addr learned my addr 172.21.15.98:0/3837320406 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:48.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.465+0000 7fa1f0939640 1 --2- 172.21.15.98:0/3837320406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac5bc50 0x55801ac65de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.470+0000 7fa1f113a640 1 --2- 172.21.15.98:0/3837320406 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5b330 0x55801ac5b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.471+0000 7fa1f193b640 1 -- 172.21.15.98:0/3837320406 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5b330 msgr2=0x55801ac5b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:48.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.471+0000 7fa1f193b640 1 --2- 172.21.15.98:0/3837320406 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5b330 0x55801ac5b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.471+0000 7fa1f193b640 1 -- 172.21.15.98:0/3837320406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac5bc50 msgr2=0x55801ac65de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:48.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.471+0000 7fa1f193b640 1 --2- 172.21.15.98:0/3837320406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac5bc50 0x55801ac65de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.471+0000 7fa1f193b640 1 -- 172.21.15.98:0/3837320406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55801a8d43f0 con 0x55801ac66320 2024-05-09T08:56:48.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.472+0000 7fa1f0939640 1 --2- 172.21.15.98:0/985134962 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa1d80a2de0 0x7fa1d80a52a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec112f30 tx=0x7fa1e802a960 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:48.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.479+0000 7fa1f193b640 1 --2- 172.21.15.98:0/3837320406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac66320 0x55801ac68710 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400ca00 tx=0x7fa1e409b530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3755a7fe53ba8720 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:48.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.481+0000 7fa1d77fe640 1 -- 172.21.15.98:0/3837320406 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e401d040 con 0x55801ac66320 2024-05-09T08:56:48.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.482+0000 7fa1d77fe640 1 -- 172.21.15.98:0/3837320406 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1e4098110 con 0x55801ac66320 2024-05-09T08:56:48.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.482+0000 7fa1d77fe640 1 -- 172.21.15.98:0/3837320406 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e409aef0 con 0x55801ac66320 2024-05-09T08:56:48.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.485+0000 7fa1f6188740 1 -- 172.21.15.98:0/3837320406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac66320 msgr2=0x55801ac68710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.485+0000 7fa1f6188740 1 --2- 172.21.15.98:0/3837320406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac66320 0x55801ac68710 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400ca00 tx=0x7fa1e409b530 comp rx=0 tx=0).stop 2024-05-09T08:56:48.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.490+0000 7fa1f6188740 1 -- 172.21.15.98:0/3837320406 shutdown_connections 2024-05-09T08:56:48.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.490+0000 7fa1f6188740 1 --2- 172.21.15.98:0/3837320406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac66320 0x55801ac68710 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.490+0000 7fa1f6188740 1 --2- 172.21.15.98:0/3837320406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac5bc50 0x55801ac65de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.490+0000 7fa1f6188740 1 --2- 172.21.15.98:0/3837320406 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5b330 0x55801ac5b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.490+0000 7fa1f6188740 1 -- 172.21.15.98:0/3837320406 >> 172.21.15.98:0/3837320406 conn(0x55801ac57650 msgr2=0x55801ac57a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:48.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.494+0000 7fa1f6188740 1 -- 172.21.15.98:0/3837320406 shutdown_connections 2024-05-09T08:56:48.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.494+0000 7fa1f6188740 1 -- 172.21.15.98:0/3837320406 wait complete. 2024-05-09T08:56:48.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.494+0000 7fa1f6188740 1 -- start start 2024-05-09T08:56:48.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.495+0000 7fa1f6188740 1 Processor -- start 2024-05-09T08:56:48.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.495+0000 7fa1f6188740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5b330 0x55801ac632e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.495+0000 7fa1f6188740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5bc50 0x55801ac63820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.495+0000 7fa1f6188740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac66320 0x55801ac63d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.495+0000 7fa1f6188740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55801ac69050 con 0x55801ac5b330 2024-05-09T08:56:48.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.495+0000 7fa1f6188740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55801ab7bc40 con 0x55801ac66320 2024-05-09T08:56:48.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.495+0000 7fa1f6188740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55801ac691d0 con 0x55801ac5bc50 2024-05-09T08:56:48.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.496+0000 7fa1f6188740 10 client.0 _ll_register_callbacks cb 0x55801a8d3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:48.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.498+0000 7fa1f0939640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5bc50 0x55801ac63820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.498+0000 7fa1f0939640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5bc50 0x55801ac63820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:36704/0 (socket says 172.21.15.98:36704) 2024-05-09T08:56:48.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.498+0000 7fa1f0939640 1 -- 172.21.15.98:0/1399129762 learned_addr learned my addr 172.21.15.98:0/1399129762 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:48.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.498+0000 7fa1f193b640 1 --2- 172.21.15.98:0/1399129762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac66320 0x55801ac63d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.499+0000 7fa1f113a640 1 --2- 172.21.15.98:0/1399129762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5b330 0x55801ac632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.499+0000 7fa1f193b640 1 -- 172.21.15.98:0/1399129762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5bc50 msgr2=0x55801ac63820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:48.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.499+0000 7fa1f193b640 1 --2- 172.21.15.98:0/1399129762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5bc50 0x55801ac63820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.499+0000 7fa1f193b640 1 -- 172.21.15.98:0/1399129762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5b330 msgr2=0x55801ac632e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:48.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.499+0000 7fa1f193b640 1 --2- 172.21.15.98:0/1399129762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5b330 0x55801ac632e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.499+0000 7fa1f193b640 1 -- 172.21.15.98:0/1399129762 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55801a99e7d0 con 0x55801ac66320 2024-05-09T08:56:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.505+0000 7fa1f193b640 1 --2- 172.21.15.98:0/1399129762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac66320 0x55801ac63d60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4004ff0 tx=0x7fa1e4097d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d34a0f9df969c6e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:48.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.512+0000 7fa1d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:56:48.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.512+0000 7fa1d4ff9640 1 -- 172.21.15.98:0/1399129762 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40b3070 con 0x55801ac66320 2024-05-09T08:56:48.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.512+0000 7fa1d4ff9640 1 -- 172.21.15.98:0/1399129762 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fa1e409b0f0 con 0x55801ac66320 2024-05-09T08:56:48.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.513+0000 7fa1f6188740 1 -- 172.21.15.98:0/1399129762 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1e4098110 con 0x55801ac66320 2024-05-09T08:56:48.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.515+0000 7fa1d4ff9640 1 -- 172.21.15.98:0/1399129762 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa1e401d040 con 0x55801ac66320 2024-05-09T08:56:48.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.515+0000 7fa1f6188740 10 client.26585 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:48.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.515+0000 7fa1f6188740 1 -- 172.21.15.98:0/1399129762 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55801ac752c0 con 0x55801ac66320 2024-05-09T08:56:48.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.516+0000 7fa1d4ff9640 1 -- 172.21.15.98:0/1399129762 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1e4098110 con 0x55801ac66320 2024-05-09T08:56:48.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.516+0000 7fa1d4ff9640 1 -- 172.21.15.98:0/1399129762 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fa1e409aae0 con 0x55801ac66320 2024-05-09T08:56:48.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.516+0000 7fa1f6188740 10 client.26585 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:48.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.516+0000 7fa1f6188740 10 client.26585 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:56:48.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.516+0000 7fa1f6188740 20 client.26585 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:48.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.516+0000 7fa1f6188740 1 --2- 172.21.15.98:0/1399129762 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55801ac76910 0x55801ac78d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.516+0000 7fa1f6188740 4 client.26585 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:48.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.516+0000 7fa1f6188740 1 -- 172.21.15.98:0/1399129762 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55801abce370 con 0x55801ac76910 2024-05-09T08:56:48.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.518+0000 7fa1f113a640 1 --2- 172.21.15.98:0/1399129762 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55801ac76910 0x55801ac78d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.538+0000 7fa1f113a640 1 --2- 172.21.15.98:0/1399129762 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55801ac76910 0x55801ac78d30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=83a613bc0a1d7ca server_cookie=3d6cbaa564f166a5 in_seq=0 out_seq=0 2024-05-09T08:56:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.539+0000 7fa1d4ff9640 10 client.26585 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:56:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.541+0000 7fa1d4ff9640 1 -- 172.21.15.98:0/1399129762 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55801abce370 con 0x55801ac76910 2024-05-09T08:56:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.541+0000 7fa1d4ff9640 10 client.26585 handle_command_reply: tid=0 2024-05-09T08:56:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.542+0000 7fa1f6188740 1 client.26585 shutdown 2024-05-09T08:56:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.546+0000 7fa1f6188740 1 -- 172.21.15.98:0/1399129762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac66320 msgr2=0x55801ac63d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.546+0000 7fa1f6188740 1 --2- 172.21.15.98:0/1399129762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac66320 0x55801ac63d60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4004ff0 tx=0x7fa1e4097d40 comp rx=0 tx=0).stop 2024-05-09T08:56:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.547+0000 7fa1f6188740 1 -- 172.21.15.98:0/1399129762 shutdown_connections 2024-05-09T08:56:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.547+0000 7fa1f6188740 1 --2- 172.21.15.98:0/1399129762 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55801ac76910 0x55801ac78d30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.547+0000 7fa1f6188740 1 --2- 172.21.15.98:0/1399129762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac66320 0x55801ac63d60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.547+0000 7fa1f6188740 1 --2- 172.21.15.98:0/1399129762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5bc50 0x55801ac63820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.547+0000 7fa1f6188740 1 --2- 172.21.15.98:0/1399129762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5b330 0x55801ac632e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.547+0000 7fa1f6188740 1 -- 172.21.15.98:0/1399129762 >> 172.21.15.98:0/1399129762 conn(0x55801ac57650 msgr2=0x55801ac5d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.547+0000 7fa1d4ff9640 0 client.26585 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:56:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.551+0000 7fa1f6188740 1 -- 172.21.15.98:0/1399129762 shutdown_connections 2024-05-09T08:56:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.553+0000 7fa1f6188740 1 -- 172.21.15.98:0/1399129762 wait complete. 2024-05-09T08:56:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.557+0000 7fa1f6188740 20 client.26585 trim_cache size 0 max 16384 2024-05-09T08:56:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.596+0000 7fa1f6188740 1 Processor -- start 2024-05-09T08:56:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.597+0000 7fa1f6188740 1 -- start start 2024-05-09T08:56:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.597+0000 7fa1f6188740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac5b330 0x55801ac56f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.597+0000 7fa1f6188740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5bc50 0x55801ac63b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.597+0000 7fa1f6188740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac66320 0x55801ac65110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.597+0000 7fa1f6188740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d0010030 con 0x55801ac5bc50 2024-05-09T08:56:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.597+0000 7fa1f6188740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d0011b70 con 0x55801ac5b330 2024-05-09T08:56:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.597+0000 7fa1f6188740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d0010700 con 0x55801ac66320 2024-05-09T08:56:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.599+0000 7fa1f0939640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5bc50 0x55801ac63b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.599+0000 7fa1f0939640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5bc50 0x55801ac63b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57582/0 (socket says 172.21.15.98:57582) 2024-05-09T08:56:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.599+0000 7fa1f0939640 1 -- 172.21.15.98:0/589678237 learned_addr learned my addr 172.21.15.98:0/589678237 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.599+0000 7fa1f193b640 1 --2- 172.21.15.98:0/589678237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac66320 0x55801ac65110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.600+0000 7fa1f113a640 1 --2- 172.21.15.98:0/589678237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac5b330 0x55801ac56f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.601+0000 7fa1f0939640 1 -- 172.21.15.98:0/589678237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac66320 msgr2=0x55801ac65110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.601+0000 7fa1f0939640 1 --2- 172.21.15.98:0/589678237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac66320 0x55801ac65110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.601+0000 7fa1f0939640 1 -- 172.21.15.98:0/589678237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac5b330 msgr2=0x55801ac56f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.601+0000 7fa1f0939640 1 --2- 172.21.15.98:0/589678237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac5b330 0x55801ac56f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.601+0000 7fa1f0939640 1 -- 172.21.15.98:0/589678237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55801ac48d20 con 0x55801ac5bc50 2024-05-09T08:56:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.606+0000 7fa1f0939640 1 --2- 172.21.15.98:0/589678237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5bc50 0x55801ac63b80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8009900 tx=0x7fa1e8045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b3c215aa481ad9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.607+0000 7fa1d6ffd640 1 -- 172.21.15.98:0/589678237 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8035040 con 0x55801ac5bc50 2024-05-09T08:56:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.607+0000 7fa1d6ffd640 1 -- 172.21.15.98:0/589678237 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1e8003ad0 con 0x55801ac5bc50 2024-05-09T08:56:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.607+0000 7fa1d6ffd640 1 -- 172.21.15.98:0/589678237 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e803c9f0 con 0x55801ac5bc50 2024-05-09T08:56:48.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.608+0000 7fa1f6188740 1 -- 172.21.15.98:0/589678237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5bc50 msgr2=0x55801ac63b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.608+0000 7fa1f6188740 1 --2- 172.21.15.98:0/589678237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5bc50 0x55801ac63b80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8009900 tx=0x7fa1e8045000 comp rx=0 tx=0).stop 2024-05-09T08:56:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.610+0000 7fa1f6188740 1 -- 172.21.15.98:0/589678237 shutdown_connections 2024-05-09T08:56:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.610+0000 7fa1f6188740 1 --2- 172.21.15.98:0/589678237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac66320 0x55801ac65110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.610+0000 7fa1f6188740 1 --2- 172.21.15.98:0/589678237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5bc50 0x55801ac63b80 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.610+0000 7fa1f6188740 1 --2- 172.21.15.98:0/589678237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac5b330 0x55801ac56f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.611+0000 7fa1f6188740 1 -- 172.21.15.98:0/589678237 >> 172.21.15.98:0/589678237 conn(0x55801ac57650 msgr2=0x55801ac64a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:48.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.613+0000 7fa1f6188740 1 -- 172.21.15.98:0/589678237 shutdown_connections 2024-05-09T08:56:48.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.614+0000 7fa1f6188740 1 -- 172.21.15.98:0/589678237 wait complete. 2024-05-09T08:56:48.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.614+0000 7fa1f6188740 1 -- start start 2024-05-09T08:56:48.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.614+0000 7fa1f6188740 1 Processor -- start 2024-05-09T08:56:48.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.615+0000 7fa1f6188740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5b330 0x55801ac6e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.615+0000 7fa1f6188740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5bc50 0x55801ac5fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.615+0000 7fa1f6188740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac66320 0x55801ac60510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.615+0000 7fa1f6188740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d00114a0 con 0x55801ac5bc50 2024-05-09T08:56:48.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.615+0000 7fa1f6188740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d0010dd0 con 0x55801ac66320 2024-05-09T08:56:48.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.615+0000 7fa1f6188740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d000e010 con 0x55801ac5b330 2024-05-09T08:56:48.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.615+0000 7fa1f6188740 10 client.0 _ll_register_callbacks cb 0x7fa1e409bb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:48.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.616+0000 7fa1f193b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac66320 0x55801ac60510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.616+0000 7fa1f193b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac66320 0x55801ac60510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54610/0 (socket says 172.21.15.98:54610) 2024-05-09T08:56:48.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.616+0000 7fa1f193b640 1 -- 172.21.15.98:0/4231717269 learned_addr learned my addr 172.21.15.98:0/4231717269 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:48.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.616+0000 7fa1f0939640 1 --2- 172.21.15.98:0/4231717269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5bc50 0x55801ac5fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.616+0000 7fa1f193b640 1 -- 172.21.15.98:0/4231717269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5b330 msgr2=0x55801ac6e9a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:56:48.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.616+0000 7fa1f193b640 1 --2- 172.21.15.98:0/4231717269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5b330 0x55801ac6e9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.616+0000 7fa1f193b640 1 -- 172.21.15.98:0/4231717269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5bc50 msgr2=0x55801ac5fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:48.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.616+0000 7fa1f193b640 1 --2- 172.21.15.98:0/4231717269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5bc50 0x55801ac5fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.616+0000 7fa1f193b640 1 -- 172.21.15.98:0/4231717269 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55801acdbe70 con 0x55801ac66320 2024-05-09T08:56:48.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.617+0000 7fa1f193b640 1 --2- 172.21.15.98:0/4231717269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac66320 0x55801ac60510 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4097f00 tx=0x7fa1e40ba4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c47143a6be616a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:48.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.618+0000 7fa1d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:56:48.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.619+0000 7fa1d4ff9640 1 -- 172.21.15.98:0/4231717269 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40a9060 con 0x55801ac66320 2024-05-09T08:56:48.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.619+0000 7fa1f6188740 1 -- 172.21.15.98:0/4231717269 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1e8003ad0 con 0x55801ac66320 2024-05-09T08:56:48.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.619+0000 7fa1d4ff9640 1 -- 172.21.15.98:0/4231717269 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fa1e40977f0 con 0x55801ac66320 2024-05-09T08:56:48.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.619+0000 7fa1d4ff9640 1 -- 172.21.15.98:0/4231717269 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa1e40b3040 con 0x55801ac66320 2024-05-09T08:56:48.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.619+0000 7fa1f6188740 10 client.26591 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:48.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.620+0000 7fa1f6188740 1 -- 172.21.15.98:0/4231717269 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55801acdbe70 con 0x55801ac66320 2024-05-09T08:56:48.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.622+0000 7fa1d4ff9640 1 -- 172.21.15.98:0/4231717269 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1e8003ad0 con 0x55801ac66320 2024-05-09T08:56:48.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.623+0000 7fa1d4ff9640 1 -- 172.21.15.98:0/4231717269 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fa1e40a4960 con 0x55801ac66320 2024-05-09T08:56:48.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.623+0000 7fa1f6188740 10 client.26591 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:48.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.623+0000 7fa1f6188740 10 client.26591 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:56:48.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.624+0000 7fa1f6188740 20 client.26591 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:48.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.624+0000 7fa1f6188740 1 --2- 172.21.15.98:0/4231717269 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55801acdd360 0x55801acdf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.624+0000 7fa1f6188740 4 client.26591 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:56:48.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.624+0000 7fa1f6188740 1 -- 172.21.15.98:0/4231717269 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55801abc86c0 con 0x55801acdd360 2024-05-09T08:56:48.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.624+0000 7fa1f113a640 1 --2- 172.21.15.98:0/4231717269 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55801acdd360 0x55801acdf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1d4ff9640 10 client.26591 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:56:48.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1f113a640 1 --2- 172.21.15.98:0/4231717269 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55801acdd360 0x55801acdf780 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9f9987edb2f2840d server_cookie=ea01457dcaf110f1 in_seq=0 out_seq=0 2024-05-09T08:56:48.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1d4ff9640 1 -- 172.21.15.98:0/4231717269 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55801abc86c0 con 0x55801acdd360 2024-05-09T08:56:48.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1d4ff9640 10 client.26591 handle_command_reply: tid=0 2024-05-09T08:56:48.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1f6188740 1 client.26591 shutdown 2024-05-09T08:56:48.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1f6188740 1 -- 172.21.15.98:0/4231717269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac66320 msgr2=0x55801ac60510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1f6188740 1 --2- 172.21.15.98:0/4231717269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac66320 0x55801ac60510 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4097f00 tx=0x7fa1e40ba4d0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1f6188740 1 -- 172.21.15.98:0/4231717269 shutdown_connections 2024-05-09T08:56:48.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1f6188740 1 --2- 172.21.15.98:0/4231717269 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55801acdd360 0x55801acdf780 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1f6188740 1 --2- 172.21.15.98:0/4231717269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55801ac66320 0x55801ac60510 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1f6188740 1 --2- 172.21.15.98:0/4231717269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55801ac5bc50 0x55801ac5fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1f6188740 1 --2- 172.21.15.98:0/4231717269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55801ac5b330 0x55801ac6e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1f6188740 1 -- 172.21.15.98:0/4231717269 >> 172.21.15.98:0/4231717269 conn(0x55801ac57650 msgr2=0x55801ac65220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:48.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1d4ff9640 0 client.26591 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:56:48.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.625+0000 7fa1f6188740 1 -- 172.21.15.98:0/4231717269 shutdown_connections 2024-05-09T08:56:48.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.626+0000 7fa1f6188740 1 -- 172.21.15.98:0/4231717269 wait complete. 2024-05-09T08:56:48.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.626+0000 7fa1f6188740 20 client.26591 trim_cache size 0 max 16384 2024-05-09T08:56:48.628 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:56:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.627+0000 7fa1f339e640 1 -- 172.21.15.98:0/985134962 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa1d80a2de0 msgr2=0x7fa1d80a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.627+0000 7fa1f339e640 1 --2- 172.21.15.98:0/985134962 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa1d80a2de0 0x7fa1d80a52a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec112f30 tx=0x7fa1e802a960 comp rx=0 tx=0).stop 2024-05-09T08:56:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.627+0000 7fa1f339e640 1 -- 172.21.15.98:0/985134962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1ec11b010 msgr2=0x7fa1ec111c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.627+0000 7fa1f339e640 1 --2- 172.21.15.98:0/985134962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1ec11b010 0x7fa1ec111c20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400ca30 tx=0x7fa1e400cef0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.627+0000 7fa1f339e640 1 -- 172.21.15.98:0/985134962 shutdown_connections 2024-05-09T08:56:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.627+0000 7fa1f339e640 1 --2- 172.21.15.98:0/985134962 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa1d80a2de0 0x7fa1d80a52a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.627+0000 7fa1f339e640 1 --2- 172.21.15.98:0/985134962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1ec11b010 0x7fa1ec111c20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.627+0000 7fa1f339e640 1 --2- 172.21.15.98:0/985134962 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1ec110080 0x7fa1ec1116e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.627+0000 7fa1f339e640 1 --2- 172.21.15.98:0/985134962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec10e0b0 0x7fa1ec118660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.627+0000 7fa1f339e640 1 -- 172.21.15.98:0/985134962 >> 172.21.15.98:0/985134962 conn(0x7fa1ec06c940 msgr2=0x7fa1ec072330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.627+0000 7fa1f339e640 1 -- 172.21.15.98:0/985134962 shutdown_connections 2024-05-09T08:56:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.627+0000 7fa1f339e640 1 -- 172.21.15.98:0/985134962 wait complete. 2024-05-09T08:56:48.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:56:48.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.940+0000 7fe5cfea5640 1 -- 172.21.15.98:0/3304277517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c81080b0 msgr2=0x7fe5c8108490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.940+0000 7fe5cfea5640 1 --2- 172.21.15.98:0/3304277517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c81080b0 0x7fe5c8108490 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009900 tx=0x7fe5b802a410 comp rx=0 tx=0).stop 2024-05-09T08:56:48.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.942+0000 7fe5cfea5640 1 -- 172.21.15.98:0/3304277517 shutdown_connections 2024-05-09T08:56:48.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.942+0000 7fe5cfea5640 1 --2- 172.21.15.98:0/3304277517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe5c810d130 0x7fe5c8114d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.942+0000 7fe5cfea5640 1 --2- 172.21.15.98:0/3304277517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5c8108a60 0x7fe5c810ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.942+0000 7fe5cfea5640 1 --2- 172.21.15.98:0/3304277517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c81080b0 0x7fe5c8108490 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.942+0000 7fe5cfea5640 1 -- 172.21.15.98:0/3304277517 >> 172.21.15.98:0/3304277517 conn(0x7fe5c8075d70 msgr2=0x7fe5c8078190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:48.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.950+0000 7fe5cfea5640 1 -- 172.21.15.98:0/3304277517 shutdown_connections 2024-05-09T08:56:48.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.953+0000 7fe5cfea5640 1 -- 172.21.15.98:0/3304277517 wait complete. 2024-05-09T08:56:48.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.954+0000 7fe5cfea5640 1 Processor -- start 2024-05-09T08:56:48.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.955+0000 7fe5cfea5640 1 -- start start 2024-05-09T08:56:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.955+0000 7fe5cfea5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5c81080b0 0x7fe5c8072d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.955+0000 7fe5cfea5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe5c810d130 0x7fe5c8073240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.955+0000 7fe5cfea5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c806cdc0 0x7fe5c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.955+0000 7fe5cfea5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c806b860 con 0x7fe5c806cdc0 2024-05-09T08:56:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.955+0000 7fe5cfea5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c806b6e0 con 0x7fe5c810d130 2024-05-09T08:56:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.955+0000 7fe5cfea5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c806b9e0 con 0x7fe5c81080b0 2024-05-09T08:56:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.956+0000 7fe5cd440640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe5c810d130 0x7fe5c8073240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.956+0000 7fe5cd440640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe5c810d130 0x7fe5c8073240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54636/0 (socket says 172.21.15.98:54636) 2024-05-09T08:56:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.956+0000 7fe5cd440640 1 -- 172.21.15.98:0/1787067373 learned_addr learned my addr 172.21.15.98:0/1787067373 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.956+0000 7fe5cdc41640 1 --2- 172.21.15.98:0/1787067373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5c81080b0 0x7fe5c8072d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.956+0000 7fe5ce442640 1 --2- 172.21.15.98:0/1787067373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c806cdc0 0x7fe5c806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.957+0000 7fe5cd440640 1 -- 172.21.15.98:0/1787067373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5c81080b0 msgr2=0x7fe5c8072d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.957+0000 7fe5cd440640 1 --2- 172.21.15.98:0/1787067373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5c81080b0 0x7fe5c8072d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.957+0000 7fe5cd440640 1 -- 172.21.15.98:0/1787067373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c806cdc0 msgr2=0x7fe5c806d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.957+0000 7fe5cd440640 1 --2- 172.21.15.98:0/1787067373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c806cdc0 0x7fe5c806d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.957+0000 7fe5cd440640 1 -- 172.21.15.98:0/1787067373 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c806d9f0 con 0x7fe5c810d130 2024-05-09T08:56:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.957+0000 7fe5cd440640 1 --2- 172.21.15.98:0/1787067373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe5c810d130 0x7fe5c8073240 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000b9e0 tx=0x7fe5c000bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.958+0000 7fe5beffd640 1 -- 172.21.15.98:0/1787067373 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0019070 con 0x7fe5c810d130 2024-05-09T08:56:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.958+0000 7fe5cfea5640 1 -- 172.21.15.98:0/1787067373 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c81b39d0 con 0x7fe5c810d130 2024-05-09T08:56:48.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.958+0000 7fe5cfea5640 1 -- 172.21.15.98:0/1787067373 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5c81b3e90 con 0x7fe5c810d130 2024-05-09T08:56:48.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.959+0000 7fe5beffd640 1 -- 172.21.15.98:0/1787067373 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5c000cba0 con 0x7fe5c810d130 2024-05-09T08:56:48.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.959+0000 7fe5beffd640 1 -- 172.21.15.98:0/1787067373 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c001c5a0 con 0x7fe5c810d130 2024-05-09T08:56:48.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.968+0000 7fe5beffd640 1 -- 172.21.15.98:0/1787067373 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe5c001c070 con 0x7fe5c810d130 2024-05-09T08:56:48.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.969+0000 7fe5d2bb4740 1 Processor -- start 2024-05-09T08:56:48.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.971+0000 7fe5beffd640 1 --2- 172.21.15.98:0/1787067373 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe5b40a2e40 0x7fe5b40a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.971+0000 7fe5beffd640 1 -- 172.21.15.98:0/1787067373 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fe5c00cec70 con 0x7fe5c810d130 2024-05-09T08:56:48.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.971+0000 7fe5d2bb4740 1 -- start start 2024-05-09T08:56:48.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.972+0000 7fe5ce442640 1 --2- 172.21.15.98:0/1787067373 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe5b40a2e40 0x7fe5b40a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.972+0000 7fe5d2bb4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3ed330 0x5582fd3ed710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.972+0000 7fe5d2bb4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3edc50 0x5582fd3f7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.972+0000 7fe5d2bb4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3f8320 0x5582fd3fa710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.972+0000 7fe5d2bb4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5582fd352c40 con 0x5582fd3f8320 2024-05-09T08:56:48.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.972+0000 7fe5d2bb4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5582fd2fa580 con 0x5582fd3ed330 2024-05-09T08:56:48.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.972+0000 7fe5d2bb4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5582fd34ec00 con 0x5582fd3edc50 2024-05-09T08:56:48.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.973+0000 7fe5cd440640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3edc50 0x5582fd3f7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.973+0000 7fe5cd440640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3edc50 0x5582fd3f7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:36742/0 (socket says 172.21.15.98:36742) 2024-05-09T08:56:48.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.973+0000 7fe5cd440640 1 -- 172.21.15.98:0/1636038787 learned_addr learned my addr 172.21.15.98:0/1636038787 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:48.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.973+0000 7fe5cdc41640 1 --2- 172.21.15.98:0/1636038787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3ed330 0x5582fd3ed710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.973+0000 7fe5ce442640 1 --2- 172.21.15.98:0/1636038787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3f8320 0x5582fd3fa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.973+0000 7fe5cd440640 1 -- 172.21.15.98:0/1636038787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3ed330 msgr2=0x5582fd3ed710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:48.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.973+0000 7fe5cd440640 1 --2- 172.21.15.98:0/1636038787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3ed330 0x5582fd3ed710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.973+0000 7fe5cd440640 1 -- 172.21.15.98:0/1636038787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3f8320 msgr2=0x5582fd3fa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:48.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.973+0000 7fe5cd440640 1 --2- 172.21.15.98:0/1636038787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3f8320 0x5582fd3fa710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.973+0000 7fe5cd440640 1 -- 172.21.15.98:0/1636038787 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582fd0663f0 con 0x5582fd3edc50 2024-05-09T08:56:48.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.974+0000 7fe5cd440640 1 --2- 172.21.15.98:0/1636038787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3edc50 0x5582fd3f7de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe5c001e030 tx=0x7fe5c009b640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e345b7605f00dd02 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:48.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.974+0000 7fe5ce442640 1 --2- 172.21.15.98:0/1787067373 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe5b40a2e40 0x7fe5b40a5300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4005fd0 tx=0x7fe5c40127c0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:48.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.974+0000 7fe5b3fff640 1 -- 172.21.15.98:0/1636038787 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c00a3af0 con 0x5582fd3edc50 2024-05-09T08:56:48.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.974+0000 7fe5b3fff640 1 -- 172.21.15.98:0/1636038787 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5c0099180 con 0x5582fd3edc50 2024-05-09T08:56:48.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.974+0000 7fe5b3fff640 1 -- 172.21.15.98:0/1636038787 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c00a38c0 con 0x5582fd3edc50 2024-05-09T08:56:48.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.977+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/1636038787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3edc50 msgr2=0x5582fd3f7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:56:48.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.977+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/1636038787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3edc50 0x5582fd3f7de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe5c001e030 tx=0x7fe5c009b640 comp rx=0 tx=0).stop 2024-05-09T08:56:48.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.980+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/1636038787 shutdown_connections 2024-05-09T08:56:48.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.980+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/1636038787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3f8320 0x5582fd3fa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.980+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/1636038787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3edc50 0x5582fd3f7de0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.980+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/1636038787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3ed330 0x5582fd3ed710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.980+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/1636038787 >> 172.21.15.98:0/1636038787 conn(0x5582fd3e9650 msgr2=0x5582fd3e9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:56:48.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.983+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/1636038787 shutdown_connections 2024-05-09T08:56:48.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.983+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/1636038787 wait complete. 2024-05-09T08:56:48.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.984+0000 7fe5d2bb4740 1 -- start start 2024-05-09T08:56:48.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.984+0000 7fe5d2bb4740 1 Processor -- start 2024-05-09T08:56:48.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.988+0000 7fe5d2bb4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3ed330 0x5582fd3f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.988+0000 7fe5d2bb4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3edc50 0x5582fd3f57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.988+0000 7fe5d2bb4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3f8320 0x5582fd3f5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:48.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.988+0000 7fe5d2bb4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5582fd3fb110 con 0x5582fd3edc50 2024-05-09T08:56:48.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.988+0000 7fe5d2bb4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5582fd30dc40 con 0x5582fd3ed330 2024-05-09T08:56:48.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.988+0000 7fe5d2bb4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5582fd3fb290 con 0x5582fd3f8320 2024-05-09T08:56:48.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.988+0000 7fe5d2bb4740 10 client.0 _ll_register_callbacks cb 0x5582fd065be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:56:48.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.989+0000 7fe5cd440640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3edc50 0x5582fd3f57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:48.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.989+0000 7fe5cd440640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3edc50 0x5582fd3f57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57618/0 (socket says 172.21.15.98:57618) 2024-05-09T08:56:48.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.989+0000 7fe5cd440640 1 -- 172.21.15.98:0/3141871228 learned_addr learned my addr 172.21.15.98:0/3141871228 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:56:48.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.989+0000 7fe5cd440640 1 -- 172.21.15.98:0/3141871228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3f8320 msgr2=0x5582fd3f5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:48.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.989+0000 7fe5cd440640 1 --2- 172.21.15.98:0/3141871228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3f8320 0x5582fd3f5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.989+0000 7fe5cd440640 1 -- 172.21.15.98:0/3141871228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3ed330 msgr2=0x5582fd3f52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:56:48.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.989+0000 7fe5cd440640 1 --2- 172.21.15.98:0/3141871228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3ed330 0x5582fd3f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:56:48.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.989+0000 7fe5cd440640 1 -- 172.21.15.98:0/3141871228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582fd1307d0 con 0x5582fd3edc50 2024-05-09T08:56:48.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.990+0000 7fe5cd440640 1 --2- 172.21.15.98:0/3141871228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3edc50 0x5582fd3f57e0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe5c00a35c0 tx=0x7fe5c00cc020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c82224cb52d29b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:56:48.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.992+0000 7fe5b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:56:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.992+0000 7fe5b17fa640 1 -- 172.21.15.98:0/3141871228 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c00a3af0 con 0x5582fd3edc50 2024-05-09T08:56:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.992+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3141871228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5c0099180 con 0x5582fd3edc50 2024-05-09T08:56:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.992+0000 7fe5b17fa640 1 -- 172.21.15.98:0/3141871228 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fe5c00a3870 con 0x5582fd3edc50 2024-05-09T08:56:48.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:48.998+0000 7fe5b17fa640 1 -- 172.21.15.98:0/3141871228 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe5c0082050 con 0x5582fd3edc50 2024-05-09T08:56:49.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:49.002+0000 7fe5d2bb4740 10 client.17301 fetch_fsmap learned FSMap version 35 2024-05-09T08:56:49.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:49.002+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3141871228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5582fd407350 con 0x5582fd3edc50 2024-05-09T08:56:49.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:49.006+0000 7fe5b17fa640 1 -- 172.21.15.98:0/3141871228 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5c0099180 con 0x5582fd3edc50 2024-05-09T08:56:49.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:49.006+0000 7fe5b17fa640 1 -- 172.21.15.98:0/3141871228 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe5c00a03c0 con 0x5582fd3edc50 2024-05-09T08:56:49.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:49.006+0000 7fe5d2bb4740 10 client.17301 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T08:56:49.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:49.006+0000 7fe5d2bb4740 10 client.17301 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:56:49.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:49.006+0000 7fe5d2bb4740 20 client.17301 populate_metadata read hostname 'smithi098' 2024-05-09T08:56:49.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:49.006+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3141871228 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5582fd4089d0 0x5582fd40ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:56:49.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:49.006+0000 7fe5d2bb4740 4 client.17301 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:56:49.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:49.006+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3141871228 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582fd360370 con 0x5582fd4089d0 2024-05-09T08:56:49.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:49.012+0000 7fe5cdc41640 1 --2- 172.21.15.98:0/3141871228 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5582fd4089d0 0x5582fd40ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:56:49.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:49.019+0000 7fe5cdc41640 1 --2- 172.21.15.98:0/3141871228 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5582fd4089d0 0x5582fd40ae30 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf809b0b4a871929 server_cookie=ad393463d85b0fc8 in_seq=0 out_seq=0 2024-05-09T08:56:49.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:56:49.019+0000 7fe5b17fa640 10 client.17301 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:56:50.162 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:49 smithi142 ceph-mon[34625]: pgmap v190: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 622 MiB/s wr, 790 op/s 2024-05-09T08:56:50.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:49 smithi105 ceph-mon[34498]: pgmap v190: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 622 MiB/s wr, 790 op/s 2024-05-09T08:56:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:49 smithi098 ceph-mon[30092]: pgmap v190: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 622 MiB/s wr, 790 op/s 2024-05-09T08:56:52.138 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:51 smithi098 ceph-mon[30092]: pgmap v191: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 783 MiB/s wr, 1.28k op/s 2024-05-09T08:56:52.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:51 smithi142 ceph-mon[34625]: pgmap v191: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 783 MiB/s wr, 1.28k op/s 2024-05-09T08:56:52.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:51 smithi105 ceph-mon[34498]: pgmap v191: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 783 MiB/s wr, 1.28k op/s 2024-05-09T08:56:54.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:53 smithi142 ceph-mon[34625]: pgmap v192: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 729 MiB/s wr, 1.39k op/s 2024-05-09T08:56:54.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:53 smithi142 ceph-mon[34625]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-09T08:56:54.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:53 smithi142 ceph-mon[34625]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-09T08:56:54.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:53 smithi105 ceph-mon[34498]: pgmap v192: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 729 MiB/s wr, 1.39k op/s 2024-05-09T08:56:54.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:53 smithi105 ceph-mon[34498]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-09T08:56:54.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:53 smithi105 ceph-mon[34498]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-09T08:56:54.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:53 smithi098 ceph-mon[30092]: pgmap v192: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 729 MiB/s wr, 1.39k op/s 2024-05-09T08:56:54.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:53 smithi098 ceph-mon[30092]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-09T08:56:54.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:53 smithi098 ceph-mon[30092]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-09T08:56:56.046 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:55 smithi105 ceph-mon[34498]: pgmap v193: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 609 MiB/s wr, 1.28k op/s 2024-05-09T08:56:56.046 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:55 smithi105 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 2. 2024-05-09T08:56:56.046 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:55 smithi105 systemd[1]: Stopped Ceph mds.e for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:56:56.046 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:55 smithi105 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.e.service: Consumed 20.152s CPU time. 2024-05-09T08:56:56.047 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:55 smithi105 systemd[1]: Starting Ceph mds.e for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T08:56:56.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:55 smithi142 ceph-mon[34625]: pgmap v193: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 609 MiB/s wr, 1.28k op/s 2024-05-09T08:56:56.305 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:56 smithi105 podman[68869]: 2024-05-09 08:56:55.94961969 +0000 UTC m=+0.019825284 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:56:56.305 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:56 smithi105 podman[68869]: 2024-05-09 08:56:56.116941009 +0000 UTC m=+0.187146601 container create 50591709f9973a4f7ec52cdb079accfa23e9c5457e76f8d6d978e73ed788bdc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:56:56.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:55 smithi098 ceph-mon[30092]: pgmap v193: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 609 MiB/s wr, 1.28k op/s 2024-05-09T08:56:56.560 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:56 smithi105 podman[68869]: 2024-05-09 08:56:56.412138882 +0000 UTC m=+0.482344479 container init 50591709f9973a4f7ec52cdb079accfa23e9c5457e76f8d6d978e73ed788bdc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:56:56.560 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:56 smithi105 podman[68869]: 2024-05-09 08:56:56.417462544 +0000 UTC m=+0.487668129 container start 50591709f9973a4f7ec52cdb079accfa23e9c5457e76f8d6d978e73ed788bdc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0) 2024-05-09T08:56:56.986 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:56 smithi105 bash[68869]: 50591709f9973a4f7ec52cdb079accfa23e9c5457e76f8d6d978e73ed788bdc1 2024-05-09T08:56:56.987 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:56 smithi105 systemd[1]: Started Ceph mds.e for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T08:56:56.987 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:56 smithi105 ceph-mds[68887]: -- 172.21.15.105:0/4040318702 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5610e7ccba40 con 0x5610e7c9e800 2024-05-09T08:56:56.987 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 08:56:56 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68883]: starting mds.e at 2024-05-09T08:56:58.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:57 smithi142 ceph-mon[34625]: pgmap v194: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 798 MiB/s wr, 1.39k op/s 2024-05-09T08:56:58.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:57 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.105:6834/561027115,v1:172.21.15.105:6835/561027115] up:boot 2024-05-09T08:56:58.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:57 smithi142 ceph-mon[34625]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:56:58.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:57 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:56:58.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:57 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:56:58.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:57 smithi105 ceph-mon[34498]: pgmap v194: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 798 MiB/s wr, 1.39k op/s 2024-05-09T08:56:58.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:57 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.105:6834/561027115,v1:172.21.15.105:6835/561027115] up:boot 2024-05-09T08:56:58.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:57 smithi105 ceph-mon[34498]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:56:58.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:57 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:56:58.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:57 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:56:58.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:57 smithi098 ceph-mon[30092]: pgmap v194: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 798 MiB/s wr, 1.39k op/s 2024-05-09T08:56:58.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:57 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.105:6834/561027115,v1:172.21.15.105:6835/561027115] up:boot 2024-05-09T08:56:58.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:57 smithi098 ceph-mon[30092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:56:58.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:57 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:56:58.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:57 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:57:00.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:59 smithi142 ceph-mon[34625]: pgmap v195: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 640 MiB/s wr, 1.13k op/s 2024-05-09T08:57:00.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:59 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:57:00.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:59 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:57:00.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:56:59 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:57:00.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:59 smithi105 ceph-mon[34498]: pgmap v195: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 640 MiB/s wr, 1.13k op/s 2024-05-09T08:57:00.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:59 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:57:00.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:59 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:57:00.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:56:59 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:57:00.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:59 smithi098 ceph-mon[30092]: pgmap v195: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 640 MiB/s wr, 1.13k op/s 2024-05-09T08:57:00.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:59 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:57:00.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:59 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:57:00.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:56:59 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:57:02.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:01 smithi142 ceph-mon[34625]: pgmap v196: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 793 MiB/s wr, 1.38k op/s 2024-05-09T08:57:02.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:01 smithi105 ceph-mon[34498]: pgmap v196: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 793 MiB/s wr, 1.38k op/s 2024-05-09T08:57:02.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:01 smithi098 ceph-mon[30092]: pgmap v196: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 793 MiB/s wr, 1.38k op/s 2024-05-09T08:57:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.708+0000 7ff868ff9640 1 -- 172.21.15.98:0/662351444 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a7619abdc0 con 0x55a761ac1340 2024-05-09T08:57:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.708+0000 7ff868ff9640 10 client.17226 handle_command_reply: tid=0 2024-05-09T08:57:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.708+0000 7ff88a5f1740 1 client.17226 shutdown 2024-05-09T08:57:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.708+0000 7fe5b17fa640 1 -- 172.21.15.98:0/3141871228 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582fd360370 con 0x5582fd4089d0 2024-05-09T08:57:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.708+0000 7fe5b17fa640 10 client.17301 handle_command_reply: tid=0 2024-05-09T08:57:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.708+0000 7fe5d2bb4740 1 client.17301 shutdown 2024-05-09T08:57:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.708+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3141871228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3edc50 msgr2=0x5582fd3f57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.708+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3141871228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3edc50 0x5582fd3f57e0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe5c00a35c0 tx=0x7fe5c00cc020 comp rx=0 tx=0).stop 2024-05-09T08:57:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.708+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3141871228 shutdown_connections 2024-05-09T08:57:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.708+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3141871228 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5582fd4089d0 0x5582fd40ae30 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.708+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3141871228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3f8320 0x5582fd3f5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.708+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3141871228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3edc50 0x5582fd3f57e0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.708+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3141871228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3ed330 0x5582fd3f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.708+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3141871228 >> 172.21.15.98:0/3141871228 conn(0x5582fd3e9650 msgr2=0x5582fd3ef350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.708+0000 7fe5b17fa640 0 client.17301 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.709+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3141871228 shutdown_connections 2024-05-09T08:57:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.709+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3141871228 wait complete. 2024-05-09T08:57:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.709+0000 7fe5d2bb4740 20 client.17301 trim_cache size 0 max 16384 2024-05-09T08:57:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.710+0000 7ff88a5f1740 1 -- 172.21.15.98:0/662351444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3fd70 msgr2=0x55a761a43f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.710+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/662351444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3fd70 0x55a761a43f50 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7ff8700ed870 tx=0x7ff870005700 comp rx=0 tx=0).stop 2024-05-09T08:57:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.710+0000 7ff88a5f1740 1 -- 172.21.15.98:0/662351444 shutdown_connections 2024-05-09T08:57:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.710+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/662351444 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a761ac1340 0x55a761ac3760 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.710+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/662351444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a761a4a630 0x55a761a44490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.710+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/662351444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a761a3fd70 0x55a761a43f50 secure :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7ff8700ed870 tx=0x7ff870005700 comp rx=0 tx=0).stop 2024-05-09T08:57:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.710+0000 7ff88a5f1740 1 --2- 172.21.15.98:0/662351444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a761a3f450 0x55a761a529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.710+0000 7ff88a5f1740 1 -- 172.21.15.98:0/662351444 >> 172.21.15.98:0/662351444 conn(0x55a761a3b960 msgr2=0x55a761a47530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.716+0000 7ff868ff9640 0 client.17226 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.716+0000 7ff88a5f1740 1 -- 172.21.15.98:0/662351444 shutdown_connections 2024-05-09T08:57:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.716+0000 7ff88a5f1740 1 -- 172.21.15.98:0/662351444 wait complete. 2024-05-09T08:57:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.716+0000 7ff88a5f1740 20 client.17226 trim_cache size 0 max 16384 2024-05-09T08:57:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:57:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:57:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:57:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:57:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.718+0000 7ff8878e7640 1 -- 172.21.15.98:0/4002565673 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff86c0a2f80 msgr2=0x7ff86c0a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.718+0000 7ff8878e7640 1 --2- 172.21.15.98:0/4002565673 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff86c0a2f80 0x7ff86c0a5440 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x55a761a409f0 tx=0x7ff87c0095d0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.718+0000 7ff8878e7640 1 -- 172.21.15.98:0/4002565673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff88011b010 msgr2=0x7ff8801167c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.718+0000 7ff8878e7640 1 --2- 172.21.15.98:0/4002565673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff88011b010 0x7ff8801167c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7ff870000c00 tx=0x7ff8700309c0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.719+0000 7ff8878e7640 1 -- 172.21.15.98:0/4002565673 shutdown_connections 2024-05-09T08:57:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.719+0000 7ff8878e7640 1 --2- 172.21.15.98:0/4002565673 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff86c0a2f80 0x7ff86c0a5440 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.719+0000 7ff8878e7640 1 --2- 172.21.15.98:0/4002565673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff880116d00 0x7ff8801171b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.719+0000 7ff8878e7640 1 --2- 172.21.15.98:0/4002565673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff88011b010 0x7ff8801167c0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.719+0000 7ff8878e7640 1 --2- 172.21.15.98:0/4002565673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff880110020 0x7ff88011a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.719+0000 7ff8878e7640 1 -- 172.21.15.98:0/4002565673 >> 172.21.15.98:0/4002565673 conn(0x7ff88006c900 msgr2=0x7ff88011b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.720+0000 7ff8878e7640 1 -- 172.21.15.98:0/4002565673 shutdown_connections 2024-05-09T08:57:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:03.720+0000 7ff8878e7640 1 -- 172.21.15.98:0/4002565673 wait complete. 2024-05-09T08:57:03.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.749+0000 7fe5d2bb4740 1 Processor -- start 2024-05-09T08:57:03.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.749+0000 7fe5d2bb4740 1 -- start start 2024-05-09T08:57:03.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.749+0000 7fe5d2bb4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3ed330 0x5582fd404030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:03.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.749+0000 7fe5d2bb4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3edc50 0x5582fd3f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:03.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.749+0000 7fe5d2bb4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3f8320 0x5582fd3ef190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:03.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.749+0000 7fe5d2bb4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac021040 con 0x5582fd3ed330 2024-05-09T08:57:03.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.749+0000 7fe5d2bb4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac022b80 con 0x5582fd3edc50 2024-05-09T08:57:03.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.749+0000 7fe5d2bb4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac021710 con 0x5582fd3f8320 2024-05-09T08:57:03.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.751+0000 7fe5ce442640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3f8320 0x5582fd3ef190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:03.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.751+0000 7fe5cd440640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3edc50 0x5582fd3f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:03.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.751+0000 7fe5cdc41640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3ed330 0x5582fd404030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:03.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.751+0000 7fe5cdc41640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3ed330 0x5582fd404030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57402/0 (socket says 172.21.15.98:57402) 2024-05-09T08:57:03.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.751+0000 7fe5cdc41640 1 -- 172.21.15.98:0/3029942659 learned_addr learned my addr 172.21.15.98:0/3029942659 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:03.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.752+0000 7fe5cdc41640 1 -- 172.21.15.98:0/3029942659 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3f8320 msgr2=0x5582fd3ef190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:03.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.752+0000 7fe5cdc41640 1 --2- 172.21.15.98:0/3029942659 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3f8320 0x5582fd3ef190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.752+0000 7fe5cdc41640 1 -- 172.21.15.98:0/3029942659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3edc50 msgr2=0x5582fd3f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:03.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.752+0000 7fe5cdc41640 1 --2- 172.21.15.98:0/3029942659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3edc50 0x5582fd3f5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.752+0000 7fe5cdc41640 1 -- 172.21.15.98:0/3029942659 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582fd3eda80 con 0x5582fd3ed330 2024-05-09T08:57:03.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.752+0000 7fe5cdc41640 1 --2- 172.21.15.98:0/3029942659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3ed330 0x5582fd404030 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009900 tx=0x7fe5b8031960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ada35214c6f239 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.755+0000 7fe5b37fe640 1 -- 172.21.15.98:0/3029942659 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b806a030 con 0x5582fd3ed330 2024-05-09T08:57:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.755+0000 7fe5b37fe640 1 -- 172.21.15.98:0/3029942659 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5b8066bd0 con 0x5582fd3ed330 2024-05-09T08:57:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.755+0000 7fe5b37fe640 1 -- 172.21.15.98:0/3029942659 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8035040 con 0x5582fd3ed330 2024-05-09T08:57:03.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.757+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3029942659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3ed330 msgr2=0x5582fd404030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:03.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.757+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3029942659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3ed330 0x5582fd404030 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009900 tx=0x7fe5b8031960 comp rx=0 tx=0).stop 2024-05-09T08:57:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.760+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3029942659 shutdown_connections 2024-05-09T08:57:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.760+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3029942659 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3f8320 0x5582fd3ef190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.760+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3029942659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3edc50 0x5582fd3f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.760+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3029942659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3ed330 0x5582fd404030 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.760+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3029942659 >> 172.21.15.98:0/3029942659 conn(0x5582fd3e9650 msgr2=0x5582fd3eea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.767+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3029942659 shutdown_connections 2024-05-09T08:57:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.770+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3029942659 wait complete. 2024-05-09T08:57:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.771+0000 7fe5d2bb4740 1 -- start start 2024-05-09T08:57:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.771+0000 7fe5d2bb4740 1 Processor -- start 2024-05-09T08:57:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.772+0000 7fe5d2bb4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3ed330 0x5582fd400540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.772+0000 7fe5d2bb4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3edc50 0x5582fd400a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.772+0000 7fe5d2bb4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3f8320 0x5582fd3f1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.772+0000 7fe5d2bb4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac0224b0 con 0x5582fd3f8320 2024-05-09T08:57:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.772+0000 7fe5d2bb4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac021de0 con 0x5582fd3ed330 2024-05-09T08:57:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.772+0000 7fe5d2bb4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac01f020 con 0x5582fd3edc50 2024-05-09T08:57:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.772+0000 7fe5d2bb4740 10 client.0 _ll_register_callbacks cb 0x7fe5c009a7e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:03.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.776+0000 7fe5ce442640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3f8320 0x5582fd3f1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:03.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.776+0000 7fe5ce442640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3f8320 0x5582fd3f1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57412/0 (socket says 172.21.15.98:57412) 2024-05-09T08:57:03.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.776+0000 7fe5ce442640 1 -- 172.21.15.98:0/3762586476 learned_addr learned my addr 172.21.15.98:0/3762586476 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:03.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.777+0000 7fe5cd440640 1 --2- 172.21.15.98:0/3762586476 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3edc50 0x5582fd400a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:03.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.777+0000 7fe5cdc41640 1 --2- 172.21.15.98:0/3762586476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3ed330 0x5582fd400540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:03.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.779+0000 7fe5cd440640 1 -- 172.21.15.98:0/3762586476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3ed330 msgr2=0x5582fd400540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:03.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.779+0000 7fe5cd440640 1 --2- 172.21.15.98:0/3762586476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3ed330 0x5582fd400540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.779+0000 7fe5cd440640 1 -- 172.21.15.98:0/3762586476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3f8320 msgr2=0x5582fd3f1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:03.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.779+0000 7fe5cd440640 1 --2- 172.21.15.98:0/3762586476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3f8320 0x5582fd3f1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.779+0000 7fe5cd440640 1 -- 172.21.15.98:0/3762586476 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582fd3f21a0 con 0x5582fd3edc50 2024-05-09T08:57:03.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.784+0000 7fe5cd440640 1 --2- 172.21.15.98:0/3762586476 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3edc50 0x5582fd400a80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000bfd0 tx=0x7fe5c00b3000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f11db31be540e38e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:03.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.787+0000 7fe5b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:57:03.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.788+0000 7fe5b17fa640 1 -- 172.21.15.98:0/3762586476 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c00a3af0 con 0x5582fd3edc50 2024-05-09T08:57:03.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.788+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3762586476 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5b8066bd0 con 0x5582fd3edc50 2024-05-09T08:57:03.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.788+0000 7fe5b17fa640 1 -- 172.21.15.98:0/3762586476 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7fe5c00cc2e0 con 0x5582fd3edc50 2024-05-09T08:57:03.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.789+0000 7fe5b17fa640 1 -- 172.21.15.98:0/3762586476 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe5c0099180 con 0x5582fd3edc50 2024-05-09T08:57:03.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.789+0000 7fe5d2bb4740 10 client.26587 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:03.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.789+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3762586476 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5582fd46ea90 con 0x5582fd3edc50 2024-05-09T08:57:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.806+0000 7fe5b17fa640 1 -- 172.21.15.98:0/3762586476 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5b8066bd0 con 0x5582fd3edc50 2024-05-09T08:57:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.807+0000 7fe5b17fa640 1 -- 172.21.15.98:0/3762586476 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe5c009a2d0 con 0x5582fd3edc50 2024-05-09T08:57:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.807+0000 7fe5d2bb4740 10 client.26587 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.807+0000 7fe5d2bb4740 10 client.26587 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.807+0000 7fe5d2bb4740 20 client.26587 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.807+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3762586476 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5582fd46eec0 0x5582fd4712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.807+0000 7fe5d2bb4740 4 client.26587 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:57:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.807+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3762586476 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5582fd35a6c0 con 0x5582fd46eec0 2024-05-09T08:57:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.816+0000 7fe5cdc41640 1 --2- 172.21.15.98:0/3762586476 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5582fd46eec0 0x5582fd4712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.825+0000 7fe5cdc41640 1 --2- 172.21.15.98:0/3762586476 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5582fd46eec0 0x5582fd4712b0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbecba9bb5ef372c server_cookie=5a26afb9db1c5cd6 in_seq=0 out_seq=0 2024-05-09T08:57:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.826+0000 7fe5b17fa640 10 client.26587 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.833+0000 7fe5b17fa640 1 -- 172.21.15.98:0/3762586476 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5582fd35a6c0 con 0x5582fd46eec0 2024-05-09T08:57:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.833+0000 7fe5b17fa640 10 client.26587 handle_command_reply: tid=0 2024-05-09T08:57:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.833+0000 7fe5d2bb4740 1 client.26587 shutdown 2024-05-09T08:57:03.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.837+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3762586476 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3edc50 msgr2=0x5582fd400a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:03.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.837+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3762586476 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3edc50 0x5582fd400a80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000bfd0 tx=0x7fe5c00b3000 comp rx=0 tx=0).stop 2024-05-09T08:57:03.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.844+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3762586476 shutdown_connections 2024-05-09T08:57:03.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.844+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3762586476 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5582fd46eec0 0x5582fd4712b0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.844+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3762586476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5582fd3f8320 0x5582fd3f1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.844+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3762586476 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5582fd3edc50 0x5582fd400a80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.844+0000 7fe5d2bb4740 1 --2- 172.21.15.98:0/3762586476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582fd3ed330 0x5582fd400540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.844+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3762586476 >> 172.21.15.98:0/3762586476 conn(0x5582fd3e9650 msgr2=0x5582fd3ef310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.844+0000 7fe5b17fa640 0 client.26587 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:03.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.849+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3762586476 shutdown_connections 2024-05-09T08:57:03.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.849+0000 7fe5d2bb4740 1 -- 172.21.15.98:0/3762586476 wait complete. 2024-05-09T08:57:03.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.849+0000 7fe5d2bb4740 20 client.26587 trim_cache size 0 max 16384 2024-05-09T08:57:03.849 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:57:03.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.858+0000 7fe5cfea5640 1 -- 172.21.15.98:0/1787067373 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe5b40a2e40 msgr2=0x7fe5b40a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:03.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.858+0000 7fe5cfea5640 1 --2- 172.21.15.98:0/1787067373 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe5b40a2e40 0x7fe5b40a5300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4005fd0 tx=0x7fe5c40127c0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.858+0000 7fe5cfea5640 1 -- 172.21.15.98:0/1787067373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe5c810d130 msgr2=0x7fe5c8073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:03.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.858+0000 7fe5cfea5640 1 --2- 172.21.15.98:0/1787067373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe5c810d130 0x7fe5c8073240 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000b9e0 tx=0x7fe5c000bea0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.858+0000 7fe5cfea5640 1 -- 172.21.15.98:0/1787067373 shutdown_connections 2024-05-09T08:57:03.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.858+0000 7fe5cfea5640 1 --2- 172.21.15.98:0/1787067373 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe5b40a2e40 0x7fe5b40a5300 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.858+0000 7fe5cfea5640 1 --2- 172.21.15.98:0/1787067373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5c806cdc0 0x7fe5c806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.858+0000 7fe5cfea5640 1 --2- 172.21.15.98:0/1787067373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe5c810d130 0x7fe5c8073240 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.858+0000 7fe5cfea5640 1 --2- 172.21.15.98:0/1787067373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5c81080b0 0x7fe5c8072d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:03.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.858+0000 7fe5cfea5640 1 -- 172.21.15.98:0/1787067373 >> 172.21.15.98:0/1787067373 conn(0x7fe5c8075d70 msgr2=0x7fe5c80768e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:03.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.859+0000 7fe5cfea5640 1 -- 172.21.15.98:0/1787067373 shutdown_connections 2024-05-09T08:57:03.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:03.859+0000 7fe5cfea5640 1 -- 172.21.15.98:0/1787067373 wait complete. 2024-05-09T08:57:04.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:03 smithi142 ceph-mon[34625]: pgmap v197: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 744 MiB/s wr, 1.07k op/s 2024-05-09T08:57:04.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:03 smithi105 ceph-mon[34498]: pgmap v197: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 744 MiB/s wr, 1.07k op/s 2024-05-09T08:57:04.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:03 smithi098 ceph-mon[30092]: pgmap v197: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 744 MiB/s wr, 1.07k op/s 2024-05-09T08:57:04.890 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:57:04.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:57:04.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:57:05.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.260+0000 7f391cdf6640 1 -- 172.21.15.98:0/3032363159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3918116830 msgr2=0x7f3918118c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:05.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.260+0000 7f391cdf6640 1 --2- 172.21.15.98:0/3032363159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3918116830 0x7f3918118c20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3900009900 tx=0x7f390002a440 comp rx=0 tx=0).stop 2024-05-09T08:57:05.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.264+0000 7f391cdf6640 1 -- 172.21.15.98:0/3032363159 shutdown_connections 2024-05-09T08:57:05.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.264+0000 7f391cdf6640 1 --2- 172.21.15.98:0/3032363159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3918116830 0x7f3918118c20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.264+0000 7f391cdf6640 1 --2- 172.21.15.98:0/3032363159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3918108d70 0x7f39181091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.264+0000 7f391cdf6640 1 --2- 172.21.15.98:0/3032363159 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f391810be00 0x7f3918108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.264+0000 7f391cdf6640 1 -- 172.21.15.98:0/3032363159 >> 172.21.15.98:0/3032363159 conn(0x7f391806c710 msgr2=0x7f391806cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:05.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.266+0000 7f391cdf6640 1 -- 172.21.15.98:0/3032363159 shutdown_connections 2024-05-09T08:57:05.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.269+0000 7f391cdf6640 1 -- 172.21.15.98:0/3032363159 wait complete. 2024-05-09T08:57:05.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.270+0000 7f391cdf6640 1 Processor -- start 2024-05-09T08:57:05.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.275+0000 7f391cdf6640 1 -- start start 2024-05-09T08:57:05.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.276+0000 7f391cdf6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3918108d70 0x7f39181160e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.276+0000 7f391cdf6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f391810be00 0x7f3918113180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.276+0000 7f391cdf6640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3918116830 0x7f39181136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.276+0000 7f391cdf6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f391811c4e0 con 0x7f3918108d70 2024-05-09T08:57:05.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.276+0000 7f391cdf6640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f391811c360 con 0x7f3918116830 2024-05-09T08:57:05.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.276+0000 7f391cdf6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f391811c660 con 0x7f391810be00 2024-05-09T08:57:05.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.277+0000 7f3917fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3918116830 0x7f39181136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:05.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.277+0000 7f3917fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3918116830 0x7f39181136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:42198/0 (socket says 172.21.15.98:42198) 2024-05-09T08:57:05.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.277+0000 7f3917fff640 1 -- 172.21.15.98:0/981153567 learned_addr learned my addr 172.21.15.98:0/981153567 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:05.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.278+0000 7f3917fff640 1 -- 172.21.15.98:0/981153567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f391810be00 msgr2=0x7f3918113180 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:57:05.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.278+0000 7f3917fff640 1 --2- 172.21.15.98:0/981153567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f391810be00 0x7f3918113180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.278+0000 7f3917fff640 1 -- 172.21.15.98:0/981153567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3918108d70 msgr2=0x7f39181160e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:05.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.278+0000 7f3917fff640 1 --2- 172.21.15.98:0/981153567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3918108d70 0x7f39181160e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.278+0000 7f3917fff640 1 -- 172.21.15.98:0/981153567 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3918113f50 con 0x7f3918116830 2024-05-09T08:57:05.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.279+0000 7f3917fff640 1 --2- 172.21.15.98:0/981153567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3918116830 0x7f39181136c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3900006010 tx=0x7f3900005650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:05.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.280+0000 7f3914ff9640 1 -- 172.21.15.98:0/981153567 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390003f070 con 0x7f3918116830 2024-05-09T08:57:05.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.280+0000 7f3914ff9640 1 -- 172.21.15.98:0/981153567 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3900004ba0 con 0x7f3918116830 2024-05-09T08:57:05.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.280+0000 7f3914ff9640 1 -- 172.21.15.98:0/981153567 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3900035030 con 0x7f3918116830 2024-05-09T08:57:05.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.281+0000 7f391cdf6640 1 -- 172.21.15.98:0/981153567 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39181b3280 con 0x7f3918116830 2024-05-09T08:57:05.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.282+0000 7f391cdf6640 1 -- 172.21.15.98:0/981153567 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39181b3800 con 0x7f3918116830 2024-05-09T08:57:05.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.288+0000 7f3920d65740 1 Processor -- start 2024-05-09T08:57:05.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.288+0000 7f3914ff9640 1 -- 172.21.15.98:0/981153567 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3900042070 con 0x7f3918116830 2024-05-09T08:57:05.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.288+0000 7f3920d65740 1 -- start start 2024-05-09T08:57:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.290+0000 7f3914ff9640 1 --2- 172.21.15.98:0/981153567 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f39040a2e90 0x7f39040a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.290+0000 7f3914ff9640 1 -- 172.21.15.98:0/981153567 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f39000efea0 con 0x7f3918116830 2024-05-09T08:57:05.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.291+0000 7f3920d65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8ede2c0 0x5638c8edc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.291+0000 7f3920d65740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8edcbd0 0x5638c8edd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.291+0000 7f3920d65740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ee7950 0x5638c8ee9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.291+0000 7f3916ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8edcbd0 0x5638c8edd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:05.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.291+0000 7f3916ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8edcbd0 0x5638c8edd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:33890/0 (socket says 172.21.15.98:33890) 2024-05-09T08:57:05.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.291+0000 7f3916ffd640 1 -- 172.21.15.98:0/3758817272 learned_addr learned my addr 172.21.15.98:0/3758817272 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:05.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.291+0000 7f3920d65740 1 -- 172.21.15.98:0/3758817272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5638c8e41990 con 0x5638c8ede2c0 2024-05-09T08:57:05.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.291+0000 7f3920d65740 1 -- 172.21.15.98:0/3758817272 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5638c8de95b0 con 0x5638c8ee7950 2024-05-09T08:57:05.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.291+0000 7f3920d65740 1 -- 172.21.15.98:0/3758817272 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5638c8e3d950 con 0x5638c8edcbd0 2024-05-09T08:57:05.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.292+0000 7f3916ffd640 1 -- 172.21.15.98:0/3758817272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ee7950 msgr2=0x5638c8ee9d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:57:05.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.292+0000 7f3916ffd640 1 --2- 172.21.15.98:0/3758817272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ee7950 0x5638c8ee9d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.292+0000 7f3916ffd640 1 -- 172.21.15.98:0/3758817272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8ede2c0 msgr2=0x5638c8edc600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:57:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.292+0000 7f3916ffd640 1 --2- 172.21.15.98:0/3758817272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8ede2c0 0x5638c8edc600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.292+0000 7f3916ffd640 1 -- 172.21.15.98:0/3758817272 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638c8c1d7e0 con 0x5638c8edcbd0 2024-05-09T08:57:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.292+0000 7f3916ffd640 1 --2- 172.21.15.98:0/3758817272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8edcbd0 0x5638c8edd050 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f391000bd20 tx=0x7f391000d420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f965710b00456f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.293+0000 7f390dffb640 1 -- 172.21.15.98:0/3758817272 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3910019070 con 0x5638c8edcbd0 2024-05-09T08:57:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.293+0000 7f3916ffd640 1 --2- 172.21.15.98:0/981153567 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f39040a2e90 0x7f39040a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.293+0000 7f390dffb640 1 -- 172.21.15.98:0/3758817272 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3910004db0 con 0x5638c8edcbd0 2024-05-09T08:57:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.293+0000 7f390dffb640 1 -- 172.21.15.98:0/3758817272 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3910014410 con 0x5638c8edcbd0 2024-05-09T08:57:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.294+0000 7f3920d65740 1 -- 172.21.15.98:0/3758817272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8edcbd0 msgr2=0x5638c8edd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.295+0000 7f3920d65740 1 --2- 172.21.15.98:0/3758817272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8edcbd0 0x5638c8edd050 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f391000bd20 tx=0x7f391000d420 comp rx=0 tx=0).stop 2024-05-09T08:57:05.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.296+0000 7f3920d65740 1 -- 172.21.15.98:0/3758817272 shutdown_connections 2024-05-09T08:57:05.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.296+0000 7f3920d65740 1 --2- 172.21.15.98:0/3758817272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ee7950 0x5638c8ee9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.296+0000 7f3920d65740 1 --2- 172.21.15.98:0/3758817272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8edcbd0 0x5638c8edd050 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.296+0000 7f3916ffd640 1 --2- 172.21.15.98:0/981153567 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f39040a2e90 0x7f39040a5350 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f391000bd20 tx=0x7f391000d420 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:05.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.296+0000 7f3920d65740 1 --2- 172.21.15.98:0/3758817272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8ede2c0 0x5638c8edc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.296+0000 7f3920d65740 1 -- 172.21.15.98:0/3758817272 >> 172.21.15.98:0/3758817272 conn(0x5638c8ea50a0 msgr2=0x5638c8ed9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:05.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.296+0000 7f3920d65740 1 -- 172.21.15.98:0/3758817272 shutdown_connections 2024-05-09T08:57:05.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.297+0000 7f3920d65740 1 -- 172.21.15.98:0/3758817272 wait complete. 2024-05-09T08:57:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.297+0000 7f3920d65740 1 -- start start 2024-05-09T08:57:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.298+0000 7f3920d65740 1 Processor -- start 2024-05-09T08:57:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.298+0000 7f3920d65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8edcbd0 0x5638c8ee4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.299+0000 7f3920d65740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ede2c0 0x5638c8ee4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.299+0000 7f3920d65740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8ee7950 0x5638c8ee5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.299+0000 7f3920d65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5638c8eea830 con 0x5638c8edcbd0 2024-05-09T08:57:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.299+0000 7f3920d65740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5638c8dfcb40 con 0x5638c8ede2c0 2024-05-09T08:57:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.299+0000 7f3920d65740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5638c8eea9b0 con 0x5638c8ee7950 2024-05-09T08:57:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.299+0000 7f3920d65740 10 client.0 _ll_register_callbacks cb 0x5638c8df10f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.299+0000 7f3916ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ede2c0 0x5638c8ee4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.299+0000 7f3916ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ede2c0 0x5638c8ee4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:42200/0 (socket says 172.21.15.98:42200) 2024-05-09T08:57:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.299+0000 7f3916ffd640 1 -- 172.21.15.98:0/1224403494 learned_addr learned my addr 172.21.15.98:0/1224403494 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:05.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.300+0000 7f3916ffd640 1 -- 172.21.15.98:0/1224403494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8ee7950 msgr2=0x5638c8ee5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:05.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.300+0000 7f3916ffd640 1 --2- 172.21.15.98:0/1224403494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8ee7950 0x5638c8ee5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.300+0000 7f3916ffd640 1 -- 172.21.15.98:0/1224403494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8edcbd0 msgr2=0x5638c8ee4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:05.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.300+0000 7f3916ffd640 1 --2- 172.21.15.98:0/1224403494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8edcbd0 0x5638c8ee4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.300+0000 7f3916ffd640 1 -- 172.21.15.98:0/1224403494 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638c8c7df70 con 0x5638c8ede2c0 2024-05-09T08:57:05.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.300+0000 7f3916ffd640 1 --2- 172.21.15.98:0/1224403494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ede2c0 0x5638c8ee4d40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f391000b470 tx=0x7f3910004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a7b11a7c19d8662 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.301+0000 7f38feffd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:57:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.305+0000 7f38feffd640 1 -- 172.21.15.98:0/1224403494 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3910019040 con 0x5638c8ede2c0 2024-05-09T08:57:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.305+0000 7f3920d65740 1 -- 172.21.15.98:0/1224403494 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3910004db0 con 0x5638c8ede2c0 2024-05-09T08:57:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.305+0000 7f38feffd640 1 -- 172.21.15.98:0/1224403494 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f39100064c0 con 0x5638c8ede2c0 2024-05-09T08:57:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.305+0000 7f38feffd640 1 -- 172.21.15.98:0/1224403494 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3910014410 con 0x5638c8ede2c0 2024-05-09T08:57:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.305+0000 7f3920d65740 10 client.26618 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.305+0000 7f3920d65740 1 -- 172.21.15.98:0/1224403494 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5638c8b15170 con 0x5638c8ede2c0 2024-05-09T08:57:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.306+0000 7f38feffd640 1 -- 172.21.15.98:0/1224403494 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3910004db0 con 0x5638c8ede2c0 2024-05-09T08:57:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.306+0000 7f38feffd640 1 -- 172.21.15.98:0/1224403494 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f391002c930 con 0x5638c8ede2c0 2024-05-09T08:57:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.306+0000 7f3920d65740 10 client.26618 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.306+0000 7f3920d65740 10 client.26618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.307+0000 7f3920d65740 20 client.26618 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.307+0000 7f3920d65740 1 --2- 172.21.15.98:0/1224403494 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638c8ef7da0 0x5638c8efa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.307+0000 7f3920d65740 4 client.26618 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:57:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.307+0000 7f3920d65740 1 -- 172.21.15.98:0/1224403494 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638c8e4f0c0 con 0x5638c8ef7da0 2024-05-09T08:57:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.308+0000 7f39177fe640 1 --2- 172.21.15.98:0/1224403494 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638c8ef7da0 0x5638c8efa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.313+0000 7f39177fe640 1 --2- 172.21.15.98:0/1224403494 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638c8ef7da0 0x5638c8efa240 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3467d490c7ca5df8 server_cookie=4c5fa7330636c2ae in_seq=0 out_seq=0 2024-05-09T08:57:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.313+0000 7f38feffd640 10 client.26618 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.320+0000 7f38feffd640 1 -- 172.21.15.98:0/1224403494 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5638c8e4f0c0 con 0x5638c8ef7da0 2024-05-09T08:57:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.320+0000 7f38feffd640 10 client.26618 handle_command_reply: tid=0 2024-05-09T08:57:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.320+0000 7f3920d65740 1 client.26618 shutdown 2024-05-09T08:57:05.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.320+0000 7f3920d65740 1 -- 172.21.15.98:0/1224403494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ede2c0 msgr2=0x5638c8ee4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:05.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.320+0000 7f3920d65740 1 --2- 172.21.15.98:0/1224403494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ede2c0 0x5638c8ee4d40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f391000b470 tx=0x7f3910004a40 comp rx=0 tx=0).stop 2024-05-09T08:57:05.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.325+0000 7f3920d65740 1 -- 172.21.15.98:0/1224403494 shutdown_connections 2024-05-09T08:57:05.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.325+0000 7f3920d65740 1 --2- 172.21.15.98:0/1224403494 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638c8ef7da0 0x5638c8efa240 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.325+0000 7f3920d65740 1 --2- 172.21.15.98:0/1224403494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8ee7950 0x5638c8ee5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.325+0000 7f3920d65740 1 --2- 172.21.15.98:0/1224403494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ede2c0 0x5638c8ee4d40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.325+0000 7f3920d65740 1 --2- 172.21.15.98:0/1224403494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8edcbd0 0x5638c8ee4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.325+0000 7f3920d65740 1 -- 172.21.15.98:0/1224403494 >> 172.21.15.98:0/1224403494 conn(0x5638c8ea50a0 msgr2=0x5638c8edefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:05.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.326+0000 7f38feffd640 0 client.26618 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:05.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.326+0000 7f3920d65740 1 -- 172.21.15.98:0/1224403494 shutdown_connections 2024-05-09T08:57:05.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.326+0000 7f3920d65740 1 -- 172.21.15.98:0/1224403494 wait complete. 2024-05-09T08:57:05.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.326+0000 7f3920d65740 20 client.26618 trim_cache size 0 max 16384 2024-05-09T08:57:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.369+0000 7f3920d65740 1 Processor -- start 2024-05-09T08:57:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.369+0000 7f3920d65740 1 -- start start 2024-05-09T08:57:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.369+0000 7f3920d65740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8edcbd0 0x5638c8ef3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.369+0000 7f3920d65740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8ede2c0 0x5638c8ed8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.369+0000 7f3920d65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8ee7950 0x5638c8ee50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.369+0000 7f3920d65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8010070 con 0x5638c8ee7950 2024-05-09T08:57:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.369+0000 7f3920d65740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8011bb0 con 0x5638c8edcbd0 2024-05-09T08:57:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.369+0000 7f3920d65740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8010740 con 0x5638c8ede2c0 2024-05-09T08:57:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.369+0000 7f3916ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8ede2c0 0x5638c8ed8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.369+0000 7f3916ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8ede2c0 0x5638c8ed8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:33908/0 (socket says 172.21.15.98:33908) 2024-05-09T08:57:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.369+0000 7f3916ffd640 1 -- 172.21.15.98:0/4287364573 learned_addr learned my addr 172.21.15.98:0/4287364573 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.369+0000 7f39177fe640 1 --2- 172.21.15.98:0/4287364573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8edcbd0 0x5638c8ef3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.370+0000 7f3917fff640 1 --2- 172.21.15.98:0/4287364573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8ee7950 0x5638c8ee50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.370+0000 7f3916ffd640 1 -- 172.21.15.98:0/4287364573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8edcbd0 msgr2=0x5638c8ef3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.370+0000 7f3916ffd640 1 --2- 172.21.15.98:0/4287364573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8edcbd0 0x5638c8ef3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.370+0000 7f3916ffd640 1 -- 172.21.15.98:0/4287364573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8ee7950 msgr2=0x5638c8ee50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.370+0000 7f3916ffd640 1 --2- 172.21.15.98:0/4287364573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8ee7950 0x5638c8ee50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.370+0000 7f3916ffd640 1 -- 172.21.15.98:0/4287364573 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638c8ee2d50 con 0x5638c8ede2c0 2024-05-09T08:57:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.370+0000 7f3916ffd640 1 --2- 172.21.15.98:0/4287364573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8ede2c0 0x5638c8ed8110 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3910014b60 tx=0x7f391000bbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=567d306f8ffa6e0b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.372+0000 7f390d7fa640 1 -- 172.21.15.98:0/4287364573 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3910019050 con 0x5638c8ede2c0 2024-05-09T08:57:05.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.372+0000 7f390d7fa640 1 -- 172.21.15.98:0/4287364573 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f391001e430 con 0x5638c8ede2c0 2024-05-09T08:57:05.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.373+0000 7f3920d65740 1 -- 172.21.15.98:0/4287364573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8ede2c0 msgr2=0x5638c8ed8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:05.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.373+0000 7f3920d65740 1 --2- 172.21.15.98:0/4287364573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8ede2c0 0x5638c8ed8110 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3910014b60 tx=0x7f391000bbf0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.384+0000 7f3920d65740 1 -- 172.21.15.98:0/4287364573 shutdown_connections 2024-05-09T08:57:05.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.384+0000 7f3920d65740 1 --2- 172.21.15.98:0/4287364573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8ee7950 0x5638c8ee50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.384+0000 7f3920d65740 1 --2- 172.21.15.98:0/4287364573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8ede2c0 0x5638c8ed8110 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.384+0000 7f3920d65740 1 --2- 172.21.15.98:0/4287364573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8edcbd0 0x5638c8ef3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.384+0000 7f3920d65740 1 -- 172.21.15.98:0/4287364573 >> 172.21.15.98:0/4287364573 conn(0x5638c8ea50a0 msgr2=0x5638c8ed7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:05.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.389+0000 7f3920d65740 1 -- 172.21.15.98:0/4287364573 shutdown_connections 2024-05-09T08:57:05.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.389+0000 7f3920d65740 1 -- 172.21.15.98:0/4287364573 wait complete. 2024-05-09T08:57:05.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.389+0000 7f3920d65740 1 -- start start 2024-05-09T08:57:05.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.390+0000 7f3920d65740 1 Processor -- start 2024-05-09T08:57:05.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.390+0000 7f3920d65740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8edcbd0 0x5638c8ee7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.390+0000 7f3920d65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8ede2c0 0x5638c8ee8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.390+0000 7f3920d65740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ee7950 0x5638c8ee87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.390+0000 7f3920d65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38f80114e0 con 0x5638c8ede2c0 2024-05-09T08:57:05.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.390+0000 7f3920d65740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8010e10 con 0x5638c8ee7950 2024-05-09T08:57:05.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.390+0000 7f3920d65740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f38f800e050 con 0x5638c8edcbd0 2024-05-09T08:57:05.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.390+0000 7f3920d65740 10 client.0 _ll_register_callbacks cb 0x7f3910017070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:05.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.391+0000 7f39177fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8edcbd0 0x5638c8ee7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:05.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.391+0000 7f3916ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8ede2c0 0x5638c8ee8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:05.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.391+0000 7f39177fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8edcbd0 0x5638c8ee7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:33914/0 (socket says 172.21.15.98:33914) 2024-05-09T08:57:05.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.391+0000 7f39177fe640 1 -- 172.21.15.98:0/558343183 learned_addr learned my addr 172.21.15.98:0/558343183 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:05.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.391+0000 7f3917fff640 1 --2- 172.21.15.98:0/558343183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ee7950 0x5638c8ee87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:05.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.397+0000 7f39177fe640 1 -- 172.21.15.98:0/558343183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ee7950 msgr2=0x5638c8ee87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:05.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.397+0000 7f39177fe640 1 --2- 172.21.15.98:0/558343183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ee7950 0x5638c8ee87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.397+0000 7f39177fe640 1 -- 172.21.15.98:0/558343183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8ede2c0 msgr2=0x5638c8ee8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:05.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.397+0000 7f39177fe640 1 --2- 172.21.15.98:0/558343183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8ede2c0 0x5638c8ee8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:05.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.397+0000 7f39177fe640 1 -- 172.21.15.98:0/558343183 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638c8eef050 con 0x5638c8edcbd0 2024-05-09T08:57:05.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.405+0000 7f39177fe640 1 --2- 172.21.15.98:0/558343183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8edcbd0 0x5638c8ee7d30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3908016850 tx=0x7f3908009b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e81e65daa70cd3f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:05.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.405+0000 7f38feffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-05-09T08:57:05.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.408+0000 7f38feffd640 1 -- 172.21.15.98:0/558343183 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3908023060 con 0x5638c8edcbd0 2024-05-09T08:57:05.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.408+0000 7f38feffd640 1 -- 172.21.15.98:0/558343183 <== mon.1 v2:172.21.15.142:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f3908010040 con 0x5638c8edcbd0 2024-05-09T08:57:05.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.410+0000 7f3920d65740 1 -- 172.21.15.98:0/558343183 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f391001e430 con 0x5638c8edcbd0 2024-05-09T08:57:05.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.418+0000 7f38feffd640 1 -- 172.21.15.98:0/558343183 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f39080156b0 con 0x5638c8edcbd0 2024-05-09T08:57:05.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.419+0000 7f3920d65740 10 client.26596 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:05.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.419+0000 7f3920d65740 1 -- 172.21.15.98:0/558343183 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5638c8ef03f0 con 0x5638c8edcbd0 2024-05-09T08:57:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.423+0000 7f38feffd640 1 -- 172.21.15.98:0/558343183 <== mon.1 v2:172.21.15.142:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f391001e430 con 0x5638c8edcbd0 2024-05-09T08:57:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.424+0000 7f38feffd640 1 -- 172.21.15.98:0/558343183 <== mon.1 v2:172.21.15.142:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f390801daf0 con 0x5638c8edcbd0 2024-05-09T08:57:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.425+0000 7f3920d65740 10 client.26596 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.425+0000 7f3920d65740 10 client.26596 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.425+0000 7f3920d65740 20 client.26596 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.425+0000 7f3920d65740 1 --2- 172.21.15.98:0/558343183 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638c8f6f670 0x5638c8f71a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.425+0000 7f3920d65740 4 client.26596 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:57:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.425+0000 7f3920d65740 1 -- 172.21.15.98:0/558343183 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5638c8e49410 con 0x5638c8f6f670 2024-05-09T08:57:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.429+0000 7f3916ffd640 1 --2- 172.21.15.98:0/558343183 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638c8f6f670 0x5638c8f71a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.444+0000 7f3916ffd640 1 --2- 172.21.15.98:0/558343183 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638c8f6f670 0x5638c8f71a60 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca783f6e0f3d908 server_cookie=e204ad3139fddd23 in_seq=0 out_seq=0 2024-05-09T08:57:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:05.444+0000 7f38feffd640 10 client.26596 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:06.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:05 smithi105 ceph-mon[34498]: pgmap v198: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 663 MiB/s wr, 825 op/s 2024-05-09T08:57:06.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:05 smithi098 ceph-mon[30092]: pgmap v198: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 663 MiB/s wr, 825 op/s 2024-05-09T08:57:06.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:05 smithi142 ceph-mon[34625]: pgmap v198: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 663 MiB/s wr, 825 op/s 2024-05-09T08:57:08.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:07 smithi105 ceph-mon[34498]: pgmap v199: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 855 MiB/s wr, 998 op/s 2024-05-09T08:57:08.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:07 smithi098 ceph-mon[30092]: pgmap v199: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 855 MiB/s wr, 998 op/s 2024-05-09T08:57:08.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:07 smithi142 ceph-mon[34625]: pgmap v199: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 855 MiB/s wr, 998 op/s 2024-05-09T08:57:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.515+0000 7f38feffd640 1 -- 172.21.15.98:0/558343183 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5638c8e49410 con 0x5638c8f6f670 2024-05-09T08:57:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.515+0000 7f38feffd640 10 client.26596 handle_command_reply: tid=0 2024-05-09T08:57:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.515+0000 7f3920d65740 1 client.26596 shutdown 2024-05-09T08:57:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.516+0000 7f3920d65740 1 -- 172.21.15.98:0/558343183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8edcbd0 msgr2=0x5638c8ee7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.516+0000 7f3920d65740 1 --2- 172.21.15.98:0/558343183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8edcbd0 0x5638c8ee7d30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3908016850 tx=0x7f3908009b00 comp rx=0 tx=0).stop 2024-05-09T08:57:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.516+0000 7f3920d65740 1 -- 172.21.15.98:0/558343183 shutdown_connections 2024-05-09T08:57:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.516+0000 7f3920d65740 1 --2- 172.21.15.98:0/558343183 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5638c8f6f670 0x5638c8f71a60 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.516+0000 7f3920d65740 1 --2- 172.21.15.98:0/558343183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5638c8ee7950 0x5638c8ee87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.516+0000 7f3920d65740 1 --2- 172.21.15.98:0/558343183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638c8ede2c0 0x5638c8ee8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.516+0000 7f3920d65740 1 --2- 172.21.15.98:0/558343183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638c8edcbd0 0x5638c8ee7d30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.516+0000 7f3920d65740 1 -- 172.21.15.98:0/558343183 >> 172.21.15.98:0/558343183 conn(0x5638c8ea50a0 msgr2=0x5638c8ed8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.516+0000 7f38feffd640 0 client.26596 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.516+0000 7f3920d65740 1 -- 172.21.15.98:0/558343183 shutdown_connections 2024-05-09T08:57:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.516+0000 7f3920d65740 1 -- 172.21.15.98:0/558343183 wait complete. 2024-05-09T08:57:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.517+0000 7f3920d65740 20 client.26596 trim_cache size 0 max 16384 2024-05-09T08:57:08.517 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:57:08.517 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:57:08.517 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "3a0291ac-d202-4443-b50d-9381e7e4f538", 2024-05-09T08:57:08.517 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:57:08.517 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:57:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.519+0000 7f391cdf6640 1 -- 172.21.15.98:0/981153567 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f39040a2e90 msgr2=0x7f39040a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.519+0000 7f391cdf6640 1 --2- 172.21.15.98:0/981153567 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f39040a2e90 0x7f39040a5350 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f391000bd20 tx=0x7f391000d420 comp rx=0 tx=0).stop 2024-05-09T08:57:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.519+0000 7f391cdf6640 1 -- 172.21.15.98:0/981153567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3918116830 msgr2=0x7f39181136c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.519+0000 7f391cdf6640 1 --2- 172.21.15.98:0/981153567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3918116830 0x7f39181136c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3900006010 tx=0x7f3900005650 comp rx=0 tx=0).stop 2024-05-09T08:57:08.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.524+0000 7f391cdf6640 1 -- 172.21.15.98:0/981153567 shutdown_connections 2024-05-09T08:57:08.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.524+0000 7f391cdf6640 1 --2- 172.21.15.98:0/981153567 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f39040a2e90 0x7f39040a5350 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.524+0000 7f391cdf6640 1 --2- 172.21.15.98:0/981153567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3918116830 0x7f39181136c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.524+0000 7f391cdf6640 1 --2- 172.21.15.98:0/981153567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f391810be00 0x7f3918113180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.524+0000 7f391cdf6640 1 --2- 172.21.15.98:0/981153567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3918108d70 0x7f39181160e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.524+0000 7f391cdf6640 1 -- 172.21.15.98:0/981153567 >> 172.21.15.98:0/981153567 conn(0x7f391806c710 msgr2=0x7f3918116c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:08.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.527+0000 7f391cdf6640 1 -- 172.21.15.98:0/981153567 shutdown_connections 2024-05-09T08:57:08.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.527+0000 7f391cdf6640 1 -- 172.21.15.98:0/981153567 wait complete. 2024-05-09T08:57:08.562 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:57:08.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:57:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.861+0000 7f2f2075a640 1 -- 172.21.15.98:0/3337427334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f1810d170 msgr2=0x7f2f18079b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.861+0000 7f2f2075a640 1 --2- 172.21.15.98:0/3337427334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f1810d170 0x7f2f18079b40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009ca0 tx=0x7f2f1002a980 comp rx=0 tx=0).stop 2024-05-09T08:57:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.862+0000 7f2f2075a640 1 -- 172.21.15.98:0/3337427334 shutdown_connections 2024-05-09T08:57:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.862+0000 7f2f2075a640 1 --2- 172.21.15.98:0/3337427334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f1810d170 0x7f2f18079b40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.862+0000 7f2f2075a640 1 --2- 172.21.15.98:0/3337427334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f181093c0 0x7f2f1810ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.862+0000 7f2f2075a640 1 --2- 172.21.15.98:0/3337427334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f18108aa0 0x7f2f18108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.862+0000 7f2f2075a640 1 -- 172.21.15.98:0/3337427334 >> 172.21.15.98:0/3337427334 conn(0x7f2f1806c940 msgr2=0x7f2f1806cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.864+0000 7f2f2075a640 1 -- 172.21.15.98:0/3337427334 shutdown_connections 2024-05-09T08:57:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.865+0000 7f2f2075a640 1 -- 172.21.15.98:0/3337427334 wait complete. 2024-05-09T08:57:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.866+0000 7f2f2075a640 1 Processor -- start 2024-05-09T08:57:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.866+0000 7f2f2075a640 1 -- start start 2024-05-09T08:57:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.866+0000 7f2f2075a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f18108aa0 0x7f2f18073d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.866+0000 7f2f2075a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f181093c0 0x7f2f180742a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.866+0000 7f2f2075a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f1810d170 0x7f2f180747e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.866+0000 7f2f2075a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1807d0a0 con 0x7f2f181093c0 2024-05-09T08:57:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.866+0000 7f2f2075a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1807cf20 con 0x7f2f1810d170 2024-05-09T08:57:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.866+0000 7f2f2075a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1807d220 con 0x7f2f18108aa0 2024-05-09T08:57:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.867+0000 7f2f1dcf5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f181093c0 0x7f2f180742a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.867+0000 7f2f1dcf5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f181093c0 0x7f2f180742a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44758/0 (socket says 172.21.15.98:44758) 2024-05-09T08:57:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.867+0000 7f2f1ecf7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f1810d170 0x7f2f180747e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.867+0000 7f2f1ecf7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f1810d170 0x7f2f180747e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54570/0 (socket says 172.21.15.98:54570) 2024-05-09T08:57:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.867+0000 7f2f1ecf7640 1 -- 172.21.15.98:0/3742558927 learned_addr learned my addr 172.21.15.98:0/3742558927 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.867+0000 7f2f1ecf7640 1 -- 172.21.15.98:0/3742558927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f18108aa0 msgr2=0x7f2f18073d60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:57:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.867+0000 7f2f1ecf7640 1 --2- 172.21.15.98:0/3742558927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f18108aa0 0x7f2f18073d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.867+0000 7f2f1ecf7640 1 -- 172.21.15.98:0/3742558927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f181093c0 msgr2=0x7f2f180742a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.867+0000 7f2f1ecf7640 1 --2- 172.21.15.98:0/3742558927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f181093c0 0x7f2f180742a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.867+0000 7f2f1ecf7640 1 -- 172.21.15.98:0/3742558927 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f181bbba0 con 0x7f2f1810d170 2024-05-09T08:57:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.867+0000 7f2f1ecf7640 1 --2- 172.21.15.98:0/3742558927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f1810d170 0x7f2f180747e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2f10030920 tx=0x7f2f10030950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.870+0000 7f2f0f7fe640 1 -- 172.21.15.98:0/3742558927 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f10047070 con 0x7f2f1810d170 2024-05-09T08:57:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.871+0000 7f2f0f7fe640 1 -- 172.21.15.98:0/3742558927 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f10004040 con 0x7f2f1810d170 2024-05-09T08:57:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.871+0000 7f2f2075a640 1 -- 172.21.15.98:0/3742558927 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f181bbe30 con 0x7f2f1810d170 2024-05-09T08:57:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.871+0000 7f2f2075a640 1 -- 172.21.15.98:0/3742558927 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f181bc2f0 con 0x7f2f1810d170 2024-05-09T08:57:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.871+0000 7f2f0f7fe640 1 -- 172.21.15.98:0/3742558927 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f10034070 con 0x7f2f1810d170 2024-05-09T08:57:08.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.877+0000 7f2f23468740 1 Processor -- start 2024-05-09T08:57:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.882+0000 7f2f23468740 1 -- start start 2024-05-09T08:57:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.882+0000 7f2f23468740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421affa280 0x56421affa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.882+0000 7f2f23468740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affaba0 0x56421b004d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.882+0000 7f2f23468740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421b005270 0x56421b007660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.882+0000 7f2f23468740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56421af5fd00 con 0x56421affaba0 2024-05-09T08:57:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.882+0000 7f2f23468740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56421af074c0 con 0x56421b005270 2024-05-09T08:57:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.882+0000 7f2f23468740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56421af5bcc0 con 0x56421affa280 2024-05-09T08:57:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.882+0000 7f2f0f7fe640 1 -- 172.21.15.98:0/3742558927 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2f1004a070 con 0x7f2f1810d170 2024-05-09T08:57:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.883+0000 7f2f0f7fe640 1 --2- 172.21.15.98:0/3742558927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2f040a2f00 0x7f2f040a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.884+0000 7f2f0f7fe640 1 -- 172.21.15.98:0/3742558927 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f2f100f0320 con 0x7f2f1810d170 2024-05-09T08:57:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.884+0000 7f2f1ecf7640 1 --2- 172.21.15.98:0/3742558927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2f040a2f00 0x7f2f040a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.885+0000 7f2f1dcf5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421b005270 0x56421b007660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.885+0000 7f2f1dcf5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421b005270 0x56421b007660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54572/0 (socket says 172.21.15.98:54572) 2024-05-09T08:57:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.885+0000 7f2f1dcf5640 1 -- 172.21.15.98:0/86679830 learned_addr learned my addr 172.21.15.98:0/86679830 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.885+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/86679830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421affa280 0x56421affa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.885+0000 7f2f1dcf5640 1 -- 172.21.15.98:0/86679830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421affa280 msgr2=0x56421affa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.885+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/86679830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421affa280 0x56421affa660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.885+0000 7f2f1dcf5640 1 -- 172.21.15.98:0/86679830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affaba0 msgr2=0x56421b004d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.885+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/86679830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affaba0 0x56421b004d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.885+0000 7f2f1dcf5640 1 -- 172.21.15.98:0/86679830 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56421ad3d7d0 con 0x56421b005270 2024-05-09T08:57:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.886+0000 7f2f1ecf7640 1 --2- 172.21.15.98:0/3742558927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2f040a2f00 0x7f2f040a53c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2f1002fb60 tx=0x7f2f100b8040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.886+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/86679830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421b005270 0x56421b007660 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2f1402cf30 tx=0x7f2f1402c240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c43218a6591b6bb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.891+0000 7f2f0cff9640 1 -- 172.21.15.98:0/86679830 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1403e070 con 0x56421b005270 2024-05-09T08:57:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.891+0000 7f2f0cff9640 1 -- 172.21.15.98:0/86679830 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f14039950 con 0x56421b005270 2024-05-09T08:57:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.892+0000 7f2f23468740 1 -- 172.21.15.98:0/86679830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421b005270 msgr2=0x56421b007660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.892+0000 7f2f23468740 1 --2- 172.21.15.98:0/86679830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421b005270 0x56421b007660 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2f1402cf30 tx=0x7f2f1402c240 comp rx=0 tx=0).stop 2024-05-09T08:57:08.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.899+0000 7f2f23468740 1 -- 172.21.15.98:0/86679830 shutdown_connections 2024-05-09T08:57:08.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.899+0000 7f2f23468740 1 --2- 172.21.15.98:0/86679830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421b005270 0x56421b007660 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.899+0000 7f2f23468740 1 --2- 172.21.15.98:0/86679830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affaba0 0x56421b004d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.899+0000 7f2f23468740 1 --2- 172.21.15.98:0/86679830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421affa280 0x56421affa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.899+0000 7f2f23468740 1 -- 172.21.15.98:0/86679830 >> 172.21.15.98:0/86679830 conn(0x56421aff65a0 msgr2=0x56421aff6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:08.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.906+0000 7f2f23468740 1 -- 172.21.15.98:0/86679830 shutdown_connections 2024-05-09T08:57:08.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.906+0000 7f2f23468740 1 -- 172.21.15.98:0/86679830 wait complete. 2024-05-09T08:57:08.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.907+0000 7f2f23468740 1 -- start start 2024-05-09T08:57:08.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.907+0000 7f2f23468740 1 Processor -- start 2024-05-09T08:57:08.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.907+0000 7f2f23468740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affa280 0x56421b002130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.907+0000 7f2f23468740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421affaba0 0x56421b002670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.907+0000 7f2f23468740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421b005270 0x56421b002bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.907+0000 7f2f23468740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56421b0080b0 con 0x56421affa280 2024-05-09T08:57:08.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.908+0000 7f2f23468740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56421af1ab90 con 0x56421b005270 2024-05-09T08:57:08.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.908+0000 7f2f23468740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56421b008230 con 0x56421affaba0 2024-05-09T08:57:08.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.908+0000 7f2f23468740 10 client.0 _ll_register_callbacks cb 0x56421af56da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:08.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.908+0000 7f2f1dcf5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421b005270 0x56421b002bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:08.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.908+0000 7f2f1dcf5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421b005270 0x56421b002bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54584/0 (socket says 172.21.15.98:54584) 2024-05-09T08:57:08.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.908+0000 7f2f1dcf5640 1 -- 172.21.15.98:0/781147312 learned_addr learned my addr 172.21.15.98:0/781147312 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:08.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.908+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/781147312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affa280 0x56421b002130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:08.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.909+0000 7f2f1dcf5640 1 -- 172.21.15.98:0/781147312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421affaba0 msgr2=0x56421b002670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:57:08.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.909+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/781147312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421affaba0 0x56421b002670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.909+0000 7f2f1dcf5640 1 -- 172.21.15.98:0/781147312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affa280 msgr2=0x56421b002130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:08.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.909+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/781147312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affa280 0x56421b002130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.909+0000 7f2f1dcf5640 1 -- 172.21.15.98:0/781147312 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56421ac733f0 con 0x56421b005270 2024-05-09T08:57:08.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.909+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/781147312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421b005270 0x56421b002bb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2f14036610 tx=0x7f2f14036510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4ba3629878dab11 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:08.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.909+0000 7f2f01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:57:08.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.909+0000 7f2f01ffb640 1 -- 172.21.15.98:0/781147312 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1404d060 con 0x56421b005270 2024-05-09T08:57:08.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.909+0000 7f2f23468740 1 -- 172.21.15.98:0/781147312 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f14039950 con 0x56421b005270 2024-05-09T08:57:08.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.910+0000 7f2f01ffb640 1 -- 172.21.15.98:0/781147312 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f2f14039670 con 0x56421b005270 2024-05-09T08:57:08.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.910+0000 7f2f01ffb640 1 -- 172.21.15.98:0/781147312 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2f14054050 con 0x56421b005270 2024-05-09T08:57:08.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.910+0000 7f2f23468740 10 client.26636 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:08.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.910+0000 7f2f23468740 1 -- 172.21.15.98:0/781147312 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56421ac330f0 con 0x56421b005270 2024-05-09T08:57:08.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.912+0000 7f2f01ffb640 1 -- 172.21.15.98:0/781147312 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f14039950 con 0x56421b005270 2024-05-09T08:57:08.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.912+0000 7f2f01ffb640 1 -- 172.21.15.98:0/781147312 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f2f14046a30 con 0x56421b005270 2024-05-09T08:57:08.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.912+0000 7f2f23468740 10 client.26636 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:08.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.912+0000 7f2f23468740 10 client.26636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:08.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.913+0000 7f2f23468740 20 client.26636 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:08.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.913+0000 7f2f23468740 1 --2- 172.21.15.98:0/781147312 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56421b015930 0x56421b017db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.913+0000 7f2f23468740 4 client.26636 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:57:08.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.913+0000 7f2f23468740 1 -- 172.21.15.98:0/781147312 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56421af6d430 con 0x56421b015930 2024-05-09T08:57:08.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.914+0000 7f2f1e4f6640 1 --2- 172.21.15.98:0/781147312 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56421b015930 0x56421b017db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:08.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.916+0000 7f2f1e4f6640 1 --2- 172.21.15.98:0/781147312 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56421b015930 0x56421b017db0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=372b0ca0f9e8c56d server_cookie=e714c6fa6dd77314 in_seq=0 out_seq=0 2024-05-09T08:57:08.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.917+0000 7f2f01ffb640 10 client.26636 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:08.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.919+0000 7f2f01ffb640 1 -- 172.21.15.98:0/781147312 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56421af6d430 con 0x56421b015930 2024-05-09T08:57:08.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.919+0000 7f2f01ffb640 10 client.26636 handle_command_reply: tid=0 2024-05-09T08:57:08.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.919+0000 7f2f23468740 1 client.26636 shutdown 2024-05-09T08:57:08.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.923+0000 7f2f23468740 1 -- 172.21.15.98:0/781147312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421b005270 msgr2=0x56421b002bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:08.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.923+0000 7f2f23468740 1 --2- 172.21.15.98:0/781147312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421b005270 0x56421b002bb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2f14036610 tx=0x7f2f14036510 comp rx=0 tx=0).stop 2024-05-09T08:57:08.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.924+0000 7f2f23468740 1 -- 172.21.15.98:0/781147312 shutdown_connections 2024-05-09T08:57:08.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.924+0000 7f2f23468740 1 --2- 172.21.15.98:0/781147312 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56421b015930 0x56421b017db0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.924+0000 7f2f23468740 1 --2- 172.21.15.98:0/781147312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421b005270 0x56421b002bb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.924+0000 7f2f23468740 1 --2- 172.21.15.98:0/781147312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421affaba0 0x56421b002670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.924+0000 7f2f23468740 1 --2- 172.21.15.98:0/781147312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affa280 0x56421b002130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.924+0000 7f2f23468740 1 -- 172.21.15.98:0/781147312 >> 172.21.15.98:0/781147312 conn(0x56421aff65a0 msgr2=0x56421aff57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:08.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.924+0000 7f2f01ffb640 0 client.26636 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:08.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.924+0000 7f2f23468740 1 -- 172.21.15.98:0/781147312 shutdown_connections 2024-05-09T08:57:08.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.924+0000 7f2f23468740 1 -- 172.21.15.98:0/781147312 wait complete. 2024-05-09T08:57:08.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.924+0000 7f2f23468740 20 client.26636 trim_cache size 0 max 16384 2024-05-09T08:57:08.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.964+0000 7f2f23468740 1 Processor -- start 2024-05-09T08:57:08.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.964+0000 7f2f23468740 1 -- start start 2024-05-09T08:57:08.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.964+0000 7f2f23468740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421aff65a0 0x56421b011050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.965+0000 7f2f23468740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affa280 0x56421b002800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.965+0000 7f2f1dcf5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421aff65a0 0x56421b011050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:08.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.965+0000 7f2f1dcf5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421aff65a0 0x56421b011050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:60900/0 (socket says 172.21.15.98:60900) 2024-05-09T08:57:08.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.965+0000 7f2f1dcf5640 1 -- 172.21.15.98:0/3457317303 learned_addr learned my addr 172.21.15.98:0/3457317303 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:08.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.965+0000 7f2f23468740 1 --2- 172.21.15.98:0/3457317303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421affaba0 0x56421b01d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.965+0000 7f2f23468740 1 -- 172.21.15.98:0/3457317303 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2efc0100d0 con 0x56421affa280 2024-05-09T08:57:08.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.965+0000 7f2f23468740 1 -- 172.21.15.98:0/3457317303 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2efc011c10 con 0x56421affaba0 2024-05-09T08:57:08.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.965+0000 7f2f23468740 1 -- 172.21.15.98:0/3457317303 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2efc0107a0 con 0x56421aff65a0 2024-05-09T08:57:08.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.966+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/3457317303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421affaba0 0x56421b01d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:08.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.966+0000 7f2f1dcf5640 1 -- 172.21.15.98:0/3457317303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421affaba0 msgr2=0x56421b01d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:08.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.966+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/3457317303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421affaba0 0x56421b01d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.966+0000 7f2f1dcf5640 1 -- 172.21.15.98:0/3457317303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affa280 msgr2=0x56421b002800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:08.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.966+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/3457317303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affa280 0x56421b002800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.966+0000 7f2f1dcf5640 1 -- 172.21.15.98:0/3457317303 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56421aff1d40 con 0x56421aff65a0 2024-05-09T08:57:08.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.966+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/3457317303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421aff65a0 0x56421b011050 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2f140047b0 tx=0x7f2f14046f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d70d5a467f1198f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:08.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.968+0000 7f2f03fff640 1 -- 172.21.15.98:0/3457317303 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1404d050 con 0x56421aff65a0 2024-05-09T08:57:08.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.968+0000 7f2f03fff640 1 -- 172.21.15.98:0/3457317303 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f14039470 con 0x56421aff65a0 2024-05-09T08:57:08.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.970+0000 7f2f23468740 1 -- 172.21.15.98:0/3457317303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421aff65a0 msgr2=0x56421b011050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:08.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.970+0000 7f2f23468740 1 --2- 172.21.15.98:0/3457317303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421aff65a0 0x56421b011050 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2f140047b0 tx=0x7f2f14046f70 comp rx=0 tx=0).stop 2024-05-09T08:57:08.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.970+0000 7f2f23468740 1 -- 172.21.15.98:0/3457317303 shutdown_connections 2024-05-09T08:57:08.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.970+0000 7f2f23468740 1 --2- 172.21.15.98:0/3457317303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421affaba0 0x56421b01d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.970+0000 7f2f23468740 1 --2- 172.21.15.98:0/3457317303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affa280 0x56421b002800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.970+0000 7f2f23468740 1 --2- 172.21.15.98:0/3457317303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421aff65a0 0x56421b011050 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.970+0000 7f2f23468740 1 -- 172.21.15.98:0/3457317303 >> 172.21.15.98:0/3457317303 conn(0x56421b000d70 msgr2=0x56421b0020f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:08.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.970+0000 7f2f23468740 1 -- 172.21.15.98:0/3457317303 shutdown_connections 2024-05-09T08:57:08.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.970+0000 7f2f23468740 1 -- 172.21.15.98:0/3457317303 wait complete. 2024-05-09T08:57:08.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.970+0000 7f2f23468740 1 -- start start 2024-05-09T08:57:08.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.971+0000 7f2f23468740 1 Processor -- start 2024-05-09T08:57:08.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.973+0000 7f2f23468740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affa280 0x56421b008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.973+0000 7f2f23468740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421affaba0 0x56421b008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.973+0000 7f2f23468740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421b005270 0x56421b073eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.973+0000 7f2f23468740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2efc011540 con 0x56421affa280 2024-05-09T08:57:08.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.973+0000 7f2f23468740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2efc010e70 con 0x56421affaba0 2024-05-09T08:57:08.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.973+0000 7f2f23468740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2efc00e0b0 con 0x56421b005270 2024-05-09T08:57:08.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.973+0000 7f2f23468740 10 client.0 _ll_register_callbacks cb 0x7f2f1403c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.974+0000 7f2f1e4f6640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421affaba0 0x56421b008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.974+0000 7f2f1e4f6640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421affaba0 0x56421b008c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:54602/0 (socket says 172.21.15.98:54602) 2024-05-09T08:57:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.974+0000 7f2f1e4f6640 1 -- 172.21.15.98:0/3476638033 learned_addr learned my addr 172.21.15.98:0/3476638033 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.974+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/3476638033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421b005270 0x56421b073eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.976+0000 7f2f1e4f6640 1 -- 172.21.15.98:0/3476638033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421b005270 msgr2=0x56421b073eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.976+0000 7f2f1e4f6640 1 --2- 172.21.15.98:0/3476638033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421b005270 0x56421b073eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.976+0000 7f2f1e4f6640 1 -- 172.21.15.98:0/3476638033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affa280 msgr2=0x56421b008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.976+0000 7f2f1e4f6640 1 --2- 172.21.15.98:0/3476638033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affa280 0x56421b008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.976+0000 7f2f1e4f6640 1 -- 172.21.15.98:0/3476638033 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56421b074470 con 0x56421affaba0 2024-05-09T08:57:08.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.977+0000 7f2f1e4f6640 1 --2- 172.21.15.98:0/3476638033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421affaba0 0x56421b008c60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2f08006d90 tx=0x7f2f08009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9244319f2d1d1689 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:08.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.981+0000 7f2f01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:57:08.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.981+0000 7f2f01ffb640 1 -- 172.21.15.98:0/3476638033 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f08012aa0 con 0x56421affaba0 2024-05-09T08:57:08.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.981+0000 7f2f01ffb640 1 -- 172.21.15.98:0/3476638033 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f2f08012c40 con 0x56421affaba0 2024-05-09T08:57:08.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.982+0000 7f2f23468740 1 -- 172.21.15.98:0/3476638033 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56421af67780 con 0x56421affaba0 2024-05-09T08:57:08.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.983+0000 7f2f01ffb640 1 -- 172.21.15.98:0/3476638033 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2f08012e60 con 0x56421affaba0 2024-05-09T08:57:08.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.986+0000 7f2f23468740 10 client.26642 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:08.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.986+0000 7f2f23468740 1 -- 172.21.15.98:0/3476638033 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56421b07b950 con 0x56421affaba0 2024-05-09T08:57:08.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.988+0000 7f2f01ffb640 1 -- 172.21.15.98:0/3476638033 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56421af67780 con 0x56421affaba0 2024-05-09T08:57:08.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.988+0000 7f2f01ffb640 1 -- 172.21.15.98:0/3476638033 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f2f0801f8a0 con 0x56421affaba0 2024-05-09T08:57:08.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.989+0000 7f2f23468740 10 client.26642 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:08.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.989+0000 7f2f23468740 10 client.26642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:08.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.989+0000 7f2f23468740 20 client.26642 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:08.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.989+0000 7f2f23468740 1 --2- 172.21.15.98:0/3476638033 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56421b07c010 0x56421b07e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:08.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.989+0000 7f2f23468740 4 client.26642 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:57:08.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.989+0000 7f2f23468740 1 -- 172.21.15.98:0/3476638033 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56421aee46f0 con 0x56421b07c010 2024-05-09T08:57:08.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.990+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/3476638033 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56421b07c010 0x56421b07e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:08.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.992+0000 7f2f01ffb640 10 client.26642 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:08.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.994+0000 7f2f1dcf5640 1 --2- 172.21.15.98:0/3476638033 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56421b07c010 0x56421b07e430 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=302e45ee9b29f729 server_cookie=6dc8ea018bd421ca in_seq=0 out_seq=0 2024-05-09T08:57:08.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.998+0000 7f2f01ffb640 1 -- 172.21.15.98:0/3476638033 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x56421aee46f0 con 0x56421b07c010 2024-05-09T08:57:08.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.998+0000 7f2f01ffb640 10 client.26642 handle_command_reply: tid=0 2024-05-09T08:57:08.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:08.998+0000 7f2f23468740 1 client.26642 shutdown 2024-05-09T08:57:09.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.001+0000 7f2f23468740 1 -- 172.21.15.98:0/3476638033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421affaba0 msgr2=0x56421b008c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:09.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.001+0000 7f2f23468740 1 --2- 172.21.15.98:0/3476638033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421affaba0 0x56421b008c60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2f08006d90 tx=0x7f2f08009ee0 comp rx=0 tx=0).stop 2024-05-09T08:57:09.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.002+0000 7f2f23468740 1 -- 172.21.15.98:0/3476638033 shutdown_connections 2024-05-09T08:57:09.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.002+0000 7f2f23468740 1 --2- 172.21.15.98:0/3476638033 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56421b07c010 0x56421b07e430 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:09.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.002+0000 7f2f23468740 1 --2- 172.21.15.98:0/3476638033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56421b005270 0x56421b073eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:09.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.002+0000 7f2f23468740 1 --2- 172.21.15.98:0/3476638033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56421affaba0 0x56421b008c60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:09.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.002+0000 7f2f23468740 1 --2- 172.21.15.98:0/3476638033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56421affa280 0x56421b008720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:09.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.002+0000 7f2f23468740 1 -- 172.21.15.98:0/3476638033 >> 172.21.15.98:0/3476638033 conn(0x56421b000d70 msgr2=0x56421affc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:09.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.002+0000 7f2f01ffb640 0 client.26642 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:09.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.003+0000 7f2f23468740 1 -- 172.21.15.98:0/3476638033 shutdown_connections 2024-05-09T08:57:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.006+0000 7f2f23468740 1 -- 172.21.15.98:0/3476638033 wait complete. 2024-05-09T08:57:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.006+0000 7f2f23468740 20 client.26642 trim_cache size 0 max 16384 2024-05-09T08:57:09.008 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:57:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "scrub active (2 inodes in the stack)", 2024-05-09T08:57:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:57:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: "3a0291ac-d202-4443-b50d-9381e7e4f538": { 2024-05-09T08:57:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:57:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "3a0291ac-d202-4443-b50d-9381e7e4f538", 2024-05-09T08:57:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:57:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:57:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:57:09.010 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:57:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.011+0000 7f2f2075a640 1 -- 172.21.15.98:0/3742558927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2f040a2f00 msgr2=0x7f2f040a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.011+0000 7f2f2075a640 1 --2- 172.21.15.98:0/3742558927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2f040a2f00 0x7f2f040a53c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2f1002fb60 tx=0x7f2f100b8040 comp rx=0 tx=0).stop 2024-05-09T08:57:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.011+0000 7f2f2075a640 1 -- 172.21.15.98:0/3742558927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f1810d170 msgr2=0x7f2f180747e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.011+0000 7f2f2075a640 1 --2- 172.21.15.98:0/3742558927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f1810d170 0x7f2f180747e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2f10030920 tx=0x7f2f10030950 comp rx=0 tx=0).stop 2024-05-09T08:57:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.011+0000 7f2f2075a640 1 -- 172.21.15.98:0/3742558927 shutdown_connections 2024-05-09T08:57:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.011+0000 7f2f2075a640 1 --2- 172.21.15.98:0/3742558927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2f040a2f00 0x7f2f040a53c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.011+0000 7f2f2075a640 1 --2- 172.21.15.98:0/3742558927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f1810d170 0x7f2f180747e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.012+0000 7f2f2075a640 1 --2- 172.21.15.98:0/3742558927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f181093c0 0x7f2f180742a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.012+0000 7f2f2075a640 1 --2- 172.21.15.98:0/3742558927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f18108aa0 0x7f2f18073d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.012+0000 7f2f2075a640 1 -- 172.21.15.98:0/3742558927 >> 172.21.15.98:0/3742558927 conn(0x7f2f1806c940 msgr2=0x7f2f18072880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.012+0000 7f2f2075a640 1 -- 172.21.15.98:0/3742558927 shutdown_connections 2024-05-09T08:57:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:09.012+0000 7f2f2075a640 1 -- 172.21.15.98:0/3742558927 wait complete. 2024-05-09T08:57:09.037 INFO:tasks.cephfs.filesystem:scrub status for tag:3a0291ac-d202-4443-b50d-9381e7e4f538 - {'path': '/', 'tag': '3a0291ac-d202-4443-b50d-9381e7e4f538', 'options': 'recursive,force'} 2024-05-09T08:57:09.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:08 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:57:09.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:08 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:57:09.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:08 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:57:09.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:08 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:57:09.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:08 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:57:09.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:08 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:57:09.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:08 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:57:09.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:08 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:57:09.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:08 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:57:10.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:09 smithi105 ceph-mon[34498]: pgmap v200: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 629 MiB/s wr, 820 op/s 2024-05-09T08:57:10.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:09 smithi098 ceph-mon[30092]: pgmap v200: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 629 MiB/s wr, 820 op/s 2024-05-09T08:57:10.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:09 smithi142 ceph-mon[34625]: pgmap v200: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 629 MiB/s wr, 820 op/s 2024-05-09T08:57:12.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:11 smithi105 ceph-mon[34498]: pgmap v201: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 772 MiB/s wr, 977 op/s 2024-05-09T08:57:12.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:11 smithi098 ceph-mon[30092]: pgmap v201: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 772 MiB/s wr, 977 op/s 2024-05-09T08:57:12.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:11 smithi142 ceph-mon[34625]: pgmap v201: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 772 MiB/s wr, 977 op/s 2024-05-09T08:57:14.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:13 smithi105 ceph-mon[34498]: pgmap v202: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 678 MiB/s wr, 761 op/s 2024-05-09T08:57:14.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:13 smithi098 ceph-mon[30092]: pgmap v202: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 678 MiB/s wr, 761 op/s 2024-05-09T08:57:14.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:13 smithi142 ceph-mon[34625]: pgmap v202: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 678 MiB/s wr, 761 op/s 2024-05-09T08:57:15.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:14 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:57:15.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:14 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:57:15.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:14 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:57:16.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:15 smithi098 ceph-mon[30092]: pgmap v203: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 646 MiB/s wr, 717 op/s 2024-05-09T08:57:16.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:15 smithi142 ceph-mon[34625]: pgmap v203: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 646 MiB/s wr, 717 op/s 2024-05-09T08:57:16.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:15 smithi105 ceph-mon[34498]: pgmap v203: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 646 MiB/s wr, 717 op/s 2024-05-09T08:57:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.835+0000 7f8941db7640 1 -- 172.21.15.98:0/2743649377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893c100a80 msgr2=0x7f893c100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.835+0000 7f8941db7640 1 --2- 172.21.15.98:0/2743649377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893c100a80 0x7f893c100f00 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f8924009900 tx=0x7f892402a410 comp rx=0 tx=0).stop 2024-05-09T08:57:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.835+0000 7f8941db7640 1 -- 172.21.15.98:0/2743649377 shutdown_connections 2024-05-09T08:57:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.835+0000 7f8941db7640 1 --2- 172.21.15.98:0/2743649377 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f893c101440 0x7f893c10d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.835+0000 7f8941db7640 1 --2- 172.21.15.98:0/2743649377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893c100a80 0x7f893c100f00 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.835+0000 7f8941db7640 1 --2- 172.21.15.98:0/2743649377 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f893c1056d0 0x7f893c105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.835+0000 7f8941db7640 1 -- 172.21.15.98:0/2743649377 >> 172.21.15.98:0/2743649377 conn(0x7f893c076530 msgr2=0x7f893c076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.835+0000 7f8941db7640 1 -- 172.21.15.98:0/2743649377 shutdown_connections 2024-05-09T08:57:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.835+0000 7f8941db7640 1 -- 172.21.15.98:0/2743649377 wait complete. 2024-05-09T08:57:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f8941db7640 1 Processor -- start 2024-05-09T08:57:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f8941db7640 1 -- start start 2024-05-09T08:57:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f8941db7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893c100a80 0x7f893c198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f8941db7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f893c101440 0x7f893c1992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f8941db7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f893c1056d0 0x7f893c1a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f8941db7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f893c110b00 con 0x7f893c100a80 2024-05-09T08:57:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f8941db7640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f893c110980 con 0x7f893c1056d0 2024-05-09T08:57:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f8941db7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f893c110c80 con 0x7f893c101440 2024-05-09T08:57:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f893b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893c100a80 0x7f893c198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f893b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893c100a80 0x7f893c198d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38570/0 (socket says 172.21.15.98:38570) 2024-05-09T08:57:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f893b7fe640 1 -- 172.21.15.98:0/3194668363 learned_addr learned my addr 172.21.15.98:0/3194668363 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f8933fff640 1 --2- 172.21.15.98:0/3194668363 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f893c101440 0x7f893c1992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f893bfff640 1 --2- 172.21.15.98:0/3194668363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f893c1056d0 0x7f893c1a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f893b7fe640 1 -- 172.21.15.98:0/3194668363 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f893c101440 msgr2=0x7f893c1992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f893b7fe640 1 --2- 172.21.15.98:0/3194668363 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f893c101440 0x7f893c1992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f893b7fe640 1 -- 172.21.15.98:0/3194668363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f893c1056d0 msgr2=0x7f893c1a3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f893b7fe640 1 --2- 172.21.15.98:0/3194668363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f893c1056d0 0x7f893c1a3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f893b7fe640 1 -- 172.21.15.98:0/3194668363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893c1a4340 con 0x7f893c100a80 2024-05-09T08:57:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.836+0000 7f893b7fe640 1 --2- 172.21.15.98:0/3194668363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893c100a80 0x7f893c198d80 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f892c00c890 tx=0x7f892c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.837+0000 7f89397fa640 1 -- 172.21.15.98:0/3194668363 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c004220 con 0x7f893c100a80 2024-05-09T08:57:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.837+0000 7f89397fa640 1 -- 172.21.15.98:0/3194668363 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f892c0043c0 con 0x7f893c100a80 2024-05-09T08:57:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.837+0000 7f8941db7640 1 -- 172.21.15.98:0/3194668363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f893c1a4510 con 0x7f893c100a80 2024-05-09T08:57:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.837+0000 7f89397fa640 1 -- 172.21.15.98:0/3194668363 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c004a00 con 0x7f893c100a80 2024-05-09T08:57:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.837+0000 7f8941db7640 1 -- 172.21.15.98:0/3194668363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f893c1a4a30 con 0x7f893c100a80 2024-05-09T08:57:16.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.839+0000 7f89397fa640 1 -- 172.21.15.98:0/3194668363 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f892c0079b0 con 0x7f893c100a80 2024-05-09T08:57:16.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.839+0000 7f8944ba1740 1 Processor -- start 2024-05-09T08:57:16.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.839+0000 7f8944ba1740 1 -- start start 2024-05-09T08:57:16.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.839+0000 7f8944ba1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6450 0x55e1730f6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.839+0000 7f8944ba1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6d70 0x55e1731010f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.839+0000 7f8944ba1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e173103a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.839+0000 7f8944ba1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e17305b340 con 0x55e173101630 2024-05-09T08:57:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.839+0000 7f8944ba1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e173003290 con 0x55e1730f6d70 2024-05-09T08:57:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.839+0000 7f8944ba1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e173057300 con 0x55e1730f6450 2024-05-09T08:57:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f8933fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e173103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f89397fa640 1 --2- 172.21.15.98:0/3194668363 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89280a2da0 0x7f89280a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f8933fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e173103a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38584/0 (socket says 172.21.15.98:38584) 2024-05-09T08:57:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f8933fff640 1 -- 172.21.15.98:0/3968042502 learned_addr learned my addr 172.21.15.98:0/3968042502 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f893b7fe640 1 --2- 172.21.15.98:0/3968042502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6d70 0x55e1731010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f893bfff640 1 --2- 172.21.15.98:0/3968042502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6450 0x55e1730f6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f893bfff640 1 --2- 172.21.15.98:0/3194668363 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89280a2da0 0x7f89280a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f8933fff640 1 -- 172.21.15.98:0/3968042502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6450 msgr2=0x55e1730f6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f8933fff640 1 --2- 172.21.15.98:0/3968042502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6450 0x55e1730f6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f8933fff640 1 -- 172.21.15.98:0/3968042502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6d70 msgr2=0x55e1731010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f8933fff640 1 --2- 172.21.15.98:0/3968042502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6d70 0x55e1731010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f8933fff640 1 -- 172.21.15.98:0/3968042502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1730278e0 con 0x55e173101630 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f893b7fe640 1 --2- 172.21.15.98:0/3968042502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6d70 0x55e1731010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f89397fa640 1 -- 172.21.15.98:0/3194668363 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f892c0cf3f0 con 0x7f893c100a80 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f8933fff640 1 --2- 172.21.15.98:0/3968042502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e173103a20 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f8924009900 tx=0x7f89240316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d4c4350fea2b24 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f893bfff640 1 --2- 172.21.15.98:0/3194668363 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89280a2da0 0x7f89280a5260 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8934005c40 tx=0x7f8934007520 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f8931ffb640 1 -- 172.21.15.98:0/3968042502 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924040070 con 0x55e173101630 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f8931ffb640 1 -- 172.21.15.98:0/3968042502 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8924005020 con 0x55e173101630 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.840+0000 7f8931ffb640 1 -- 172.21.15.98:0/3968042502 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924035040 con 0x55e173101630 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.841+0000 7f8944ba1740 1 -- 172.21.15.98:0/3968042502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 msgr2=0x55e173103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.841+0000 7f8944ba1740 1 --2- 172.21.15.98:0/3968042502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e173103a20 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f8924009900 tx=0x7f89240316e0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.841+0000 7f8944ba1740 1 -- 172.21.15.98:0/3968042502 shutdown_connections 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.841+0000 7f8944ba1740 1 --2- 172.21.15.98:0/3968042502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e173103a20 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.841+0000 7f8944ba1740 1 --2- 172.21.15.98:0/3968042502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6d70 0x55e1731010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.841+0000 7f8944ba1740 1 --2- 172.21.15.98:0/3968042502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6450 0x55e1730f6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.841+0000 7f8944ba1740 1 -- 172.21.15.98:0/3968042502 >> 172.21.15.98:0/3968042502 conn(0x55e1730f2960 msgr2=0x55e1730f2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.841+0000 7f8944ba1740 1 -- 172.21.15.98:0/3968042502 shutdown_connections 2024-05-09T08:57:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.841+0000 7f8944ba1740 1 -- 172.21.15.98:0/3968042502 wait complete. 2024-05-09T08:57:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.841+0000 7f8944ba1740 1 -- start start 2024-05-09T08:57:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8944ba1740 1 Processor -- start 2024-05-09T08:57:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8944ba1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1731010f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8944ba1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1730f6d70 0x55e1730fe480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8944ba1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e173101630 0x55e1730febc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8944ba1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e173104200 con 0x55e1730f6d70 2024-05-09T08:57:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8944ba1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e173016790 con 0x55e1730f6450 2024-05-09T08:57:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8944ba1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e173104380 con 0x55e173101630 2024-05-09T08:57:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8944ba1740 10 client.0 _ll_register_callbacks cb 0x55e173051a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8933fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1730f6d70 0x55e1730fe480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8933fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1730f6d70 0x55e1730fe480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38588/0 (socket says 172.21.15.98:38588) 2024-05-09T08:57:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8933fff640 1 -- 172.21.15.98:0/3594165742 learned_addr learned my addr 172.21.15.98:0/3594165742 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f893b7fe640 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1731010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f893bfff640 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e173101630 0x55e1730febc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8933fff640 1 -- 172.21.15.98:0/3594165742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e173101630 msgr2=0x55e1730febc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8933fff640 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e173101630 0x55e1730febc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8933fff640 1 -- 172.21.15.98:0/3594165742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 msgr2=0x55e1731010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8933fff640 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1731010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8933fff640 1 -- 172.21.15.98:0/3594165742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e172d6f210 con 0x55e1730f6d70 2024-05-09T08:57:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f893b7fe640 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1731010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f893bfff640 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e173101630 0x55e1730febc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f8933fff640 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1730f6d70 0x55e1730fe480 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8924031ba0 tx=0x7f8924009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1f21ec235bde3fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.842+0000 7f891affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f891affd640 1 -- 172.21.15.98:0/3594165742 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924040030 con 0x55e1730f6d70 2024-05-09T08:57:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f8944ba1740 1 -- 172.21.15.98:0/3594165742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8924005020 con 0x55e1730f6d70 2024-05-09T08:57:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f891affd640 1 -- 172.21.15.98:0/3594165742 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f8924004940 con 0x55e1730f6d70 2024-05-09T08:57:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f891affd640 1 -- 172.21.15.98:0/3594165742 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8924035070 con 0x55e1730f6d70 2024-05-09T08:57:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f8944ba1740 10 client.17334 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f8944ba1740 1 -- 172.21.15.98:0/3594165742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e172e3b620 con 0x55e1730f6d70 2024-05-09T08:57:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f891affd640 1 -- 172.21.15.98:0/3594165742 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8924005020 con 0x55e1730f6d70 2024-05-09T08:57:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f891affd640 1 -- 172.21.15.98:0/3594165742 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f892404e420 con 0x55e1730f6d70 2024-05-09T08:57:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f8944ba1740 10 client.17334 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f8944ba1740 10 client.17334 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f8944ba1740 20 client.17334 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f8944ba1740 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e173111060 0x55e173113450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f8944ba1740 4 client.17334 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:57:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f8944ba1740 1 -- 172.21.15.98:0/3594165742 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e173068a70 con 0x55e173111060 2024-05-09T08:57:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.843+0000 7f893b7fe640 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e173111060 0x55e173113450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.844+0000 7f891affd640 10 client.17334 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.844+0000 7f893b7fe640 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e173111060 0x55e173113450 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a6b127ccd6b671 server_cookie=6859466358d4b4e5 in_seq=0 out_seq=0 2024-05-09T08:57:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.844+0000 7f891affd640 1 -- 172.21.15.98:0/3594165742 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e173068a70 con 0x55e173111060 2024-05-09T08:57:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.844+0000 7f891affd640 10 client.17334 handle_command_reply: tid=0 2024-05-09T08:57:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.844+0000 7f8944ba1740 1 client.17334 shutdown 2024-05-09T08:57:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.845+0000 7f8944ba1740 1 -- 172.21.15.98:0/3594165742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1730f6d70 msgr2=0x55e1730fe480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.845+0000 7f8944ba1740 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1730f6d70 0x55e1730fe480 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8924031ba0 tx=0x7f8924009520 comp rx=0 tx=0).stop 2024-05-09T08:57:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.845+0000 7f8944ba1740 1 -- 172.21.15.98:0/3594165742 shutdown_connections 2024-05-09T08:57:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.845+0000 7f8944ba1740 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e173111060 0x55e173113450 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.845+0000 7f8944ba1740 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e173101630 0x55e1730febc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.845+0000 7f8944ba1740 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e1730f6d70 0x55e1730fe480 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.845+0000 7f891affd640 0 client.17334 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.845+0000 7f8944ba1740 1 --2- 172.21.15.98:0/3594165742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1731010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.845+0000 7f8944ba1740 1 -- 172.21.15.98:0/3594165742 >> 172.21.15.98:0/3594165742 conn(0x55e1730f2960 msgr2=0x55e1730f8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.845+0000 7f8944ba1740 1 -- 172.21.15.98:0/3594165742 shutdown_connections 2024-05-09T08:57:16.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.845+0000 7f8944ba1740 1 -- 172.21.15.98:0/3594165742 wait complete. 2024-05-09T08:57:16.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.845+0000 7f8944ba1740 20 client.17334 trim_cache size 0 max 16384 2024-05-09T08:57:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f8944ba1740 1 Processor -- start 2024-05-09T08:57:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f8944ba1740 1 -- start start 2024-05-09T08:57:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f8944ba1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1730f20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f8944ba1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6d70 0x55e173118a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f8944ba1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e173118f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f8944ba1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f893c1aebe0 con 0x55e173101630 2024-05-09T08:57:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f8944ba1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f893c1b0720 con 0x55e1730f6450 2024-05-09T08:57:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f8944ba1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f893c1af2b0 con 0x55e1730f6d70 2024-05-09T08:57:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f893bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e173118f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f893bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e173118f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38594/0 (socket says 172.21.15.98:38594) 2024-05-09T08:57:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f893bfff640 1 -- 172.21.15.98:0/1220839375 learned_addr learned my addr 172.21.15.98:0/1220839375 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f893b7fe640 1 --2- 172.21.15.98:0/1220839375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1730f20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f893bfff640 1 -- 172.21.15.98:0/1220839375 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6d70 msgr2=0x55e173118a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f893bfff640 1 --2- 172.21.15.98:0/1220839375 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6d70 0x55e173118a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f893bfff640 1 -- 172.21.15.98:0/1220839375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 msgr2=0x55e1730f20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f893bfff640 1 --2- 172.21.15.98:0/1220839375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1730f20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f893bfff640 1 -- 172.21.15.98:0/1220839375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1730f6ba0 con 0x55e173101630 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f893b7fe640 1 --2- 172.21.15.98:0/1220839375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1730f20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.860+0000 7f893bfff640 1 --2- 172.21.15.98:0/1220839375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e173118f60 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8934003c90 tx=0x7f8934007940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd4f09cf2ae0d3db server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.861+0000 7f89317fa640 1 -- 172.21.15.98:0/1220839375 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89340117a0 con 0x55e173101630 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.861+0000 7f89317fa640 1 -- 172.21.15.98:0/1220839375 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8934026070 con 0x55e173101630 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.861+0000 7f89317fa640 1 -- 172.21.15.98:0/1220839375 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8934021bc0 con 0x55e173101630 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.861+0000 7f8944ba1740 1 -- 172.21.15.98:0/1220839375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 msgr2=0x55e173118f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.861+0000 7f8944ba1740 1 --2- 172.21.15.98:0/1220839375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e173118f60 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8934003c90 tx=0x7f8934007940 comp rx=0 tx=0).stop 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.861+0000 7f8944ba1740 1 -- 172.21.15.98:0/1220839375 shutdown_connections 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.861+0000 7f8944ba1740 1 --2- 172.21.15.98:0/1220839375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e173118f60 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.861+0000 7f8944ba1740 1 --2- 172.21.15.98:0/1220839375 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6d70 0x55e173118a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.861+0000 7f8944ba1740 1 --2- 172.21.15.98:0/1220839375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1730f20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.861+0000 7f8944ba1740 1 -- 172.21.15.98:0/1220839375 >> 172.21.15.98:0/1220839375 conn(0x55e1730f2960 msgr2=0x55e1730f2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.861+0000 7f8944ba1740 1 -- 172.21.15.98:0/1220839375 shutdown_connections 2024-05-09T08:57:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.861+0000 7f8944ba1740 1 -- 172.21.15.98:0/1220839375 wait complete. 2024-05-09T08:57:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.861+0000 7f8944ba1740 1 -- start start 2024-05-09T08:57:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f8944ba1740 1 Processor -- start 2024-05-09T08:57:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f8944ba1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1730fb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f8944ba1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6d70 0x55e1730fb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f8944ba1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e1730fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f8944ba1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f893c1b0050 con 0x55e173101630 2024-05-09T08:57:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f8944ba1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f893c1af980 con 0x55e1730f6450 2024-05-09T08:57:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f8944ba1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f893c1acbc0 con 0x55e1730f6d70 2024-05-09T08:57:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f8944ba1740 10 client.0 _ll_register_callbacks cb 0x7f89240315b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f893bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e1730fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f893bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e1730fbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38604/0 (socket says 172.21.15.98:38604) 2024-05-09T08:57:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f893b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1730fb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f893bfff640 1 -- 172.21.15.98:0/1683728866 learned_addr learned my addr 172.21.15.98:0/1683728866 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f8933fff640 1 --2- 172.21.15.98:0/1683728866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6d70 0x55e1730fb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f893b7fe640 1 -- 172.21.15.98:0/1683728866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6d70 msgr2=0x55e1730fb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f893b7fe640 1 --2- 172.21.15.98:0/1683728866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6d70 0x55e1730fb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f893b7fe640 1 -- 172.21.15.98:0/1683728866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 msgr2=0x55e1730fbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f893b7fe640 1 --2- 172.21.15.98:0/1683728866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e1730fbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f893b7fe640 1 -- 172.21.15.98:0/1683728866 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1731763c0 con 0x55e1730f6450 2024-05-09T08:57:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f893bfff640 1 --2- 172.21.15.98:0/1683728866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e1730fbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:57:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.862+0000 7f8933fff640 1 --2- 172.21.15.98:0/1683728866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6d70 0x55e1730fb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:57:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f893b7fe640 1 --2- 172.21.15.98:0/1683728866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1730fb280 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f892c0cc780 tx=0x7f892c06d5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdeb6494da862212 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f891affd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T08:57:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f891affd640 1 -- 172.21.15.98:0/1683728866 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c01d040 con 0x55e1730f6450 2024-05-09T08:57:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f8944ba1740 1 -- 172.21.15.98:0/1683728866 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8934026070 con 0x55e1730f6450 2024-05-09T08:57:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f891affd640 1 -- 172.21.15.98:0/1683728866 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9474+0+0 (secure 0 0 0) 0x7f892c0cca00 con 0x55e1730f6450 2024-05-09T08:57:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f891affd640 1 -- 172.21.15.98:0/1683728866 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f892c097040 con 0x55e1730f6450 2024-05-09T08:57:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f8944ba1740 10 client.26654 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f8944ba1740 1 -- 172.21.15.98:0/1683728866 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e173177710 con 0x55e1730f6450 2024-05-09T08:57:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f891affd640 1 -- 172.21.15.98:0/1683728866 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8934026070 con 0x55e1730f6450 2024-05-09T08:57:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f891affd640 1 -- 172.21.15.98:0/1683728866 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f892c097c70 con 0x55e1730f6450 2024-05-09T08:57:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f8944ba1740 10 client.26654 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f8944ba1740 10 client.26654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f8944ba1740 20 client.26654 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f8944ba1740 1 --2- 172.21.15.98:0/1683728866 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e173177de0 0x55e17317a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f8944ba1740 4 client.26654 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:57:16.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.863+0000 7f8944ba1740 1 -- 172.21.15.98:0/1683728866 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e173062dc0 con 0x55e173177de0 2024-05-09T08:57:16.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.864+0000 7f8933fff640 1 --2- 172.21.15.98:0/1683728866 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e173177de0 0x55e17317a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:16.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.864+0000 7f891affd640 10 client.26654 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:16.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.864+0000 7f8933fff640 1 --2- 172.21.15.98:0/1683728866 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e173177de0 0x55e17317a200 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef83ff1b47b69845 server_cookie=35adc3366f200b9d in_seq=0 out_seq=0 2024-05-09T08:57:16.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.881+0000 7f891affd640 1 -- 172.21.15.98:0/1683728866 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e173062dc0 con 0x55e173177de0 2024-05-09T08:57:16.881 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.881+0000 7f891affd640 10 client.26654 handle_command_reply: tid=0 2024-05-09T08:57:16.881 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.881+0000 7f8944ba1740 1 client.26654 shutdown 2024-05-09T08:57:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.884+0000 7f8944ba1740 1 -- 172.21.15.98:0/1683728866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 msgr2=0x55e1730fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.884+0000 7f8944ba1740 1 --2- 172.21.15.98:0/1683728866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1730fb280 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f892c0cc780 tx=0x7f892c06d5c0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.884+0000 7f8944ba1740 1 -- 172.21.15.98:0/1683728866 shutdown_connections 2024-05-09T08:57:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.884+0000 7f8944ba1740 1 --2- 172.21.15.98:0/1683728866 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e173177de0 0x55e17317a200 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.884+0000 7f891affd640 0 client.26654 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.884+0000 7f8944ba1740 1 --2- 172.21.15.98:0/1683728866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e173101630 0x55e1730fbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.884+0000 7f8944ba1740 1 --2- 172.21.15.98:0/1683728866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1730f6d70 0x55e1730fb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.884+0000 7f8944ba1740 1 --2- 172.21.15.98:0/1683728866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e1730f6450 0x55e1730fb280 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.884+0000 7f8944ba1740 1 -- 172.21.15.98:0/1683728866 >> 172.21.15.98:0/1683728866 conn(0x55e1730f2960 msgr2=0x55e1730f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.884+0000 7f8944ba1740 1 -- 172.21.15.98:0/1683728866 shutdown_connections 2024-05-09T08:57:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.884+0000 7f8944ba1740 1 -- 172.21.15.98:0/1683728866 wait complete. 2024-05-09T08:57:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.884+0000 7f8944ba1740 20 client.26654 trim_cache size 0 max 16384 2024-05-09T08:57:16.885 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:57:16.885 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:57:16.885 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:57:16.885 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:57:16.886 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.886+0000 7f8941db7640 1 -- 172.21.15.98:0/3194668363 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89280a2da0 msgr2=0x7f89280a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:16.886 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.886+0000 7f8941db7640 1 --2- 172.21.15.98:0/3194668363 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89280a2da0 0x7f89280a5260 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8934005c40 tx=0x7f8934007520 comp rx=0 tx=0).stop 2024-05-09T08:57:16.886 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.886+0000 7f8941db7640 1 -- 172.21.15.98:0/3194668363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893c100a80 msgr2=0x7f893c198d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:16.886 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.886+0000 7f8941db7640 1 --2- 172.21.15.98:0/3194668363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893c100a80 0x7f893c198d80 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f892c00c890 tx=0x7f892c00cd50 comp rx=0 tx=0).stop 2024-05-09T08:57:16.886 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.887+0000 7f8941db7640 1 -- 172.21.15.98:0/3194668363 shutdown_connections 2024-05-09T08:57:16.886 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.887+0000 7f8941db7640 1 --2- 172.21.15.98:0/3194668363 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89280a2da0 0x7f89280a5260 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.886 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.887+0000 7f8941db7640 1 --2- 172.21.15.98:0/3194668363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f893c1056d0 0x7f893c1a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.886 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.887+0000 7f8941db7640 1 --2- 172.21.15.98:0/3194668363 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f893c101440 0x7f893c1992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.886 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.887+0000 7f8941db7640 1 --2- 172.21.15.98:0/3194668363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893c100a80 0x7f893c198d80 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:16.886 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.887+0000 7f8941db7640 1 -- 172.21.15.98:0/3194668363 >> 172.21.15.98:0/3194668363 conn(0x7f893c076530 msgr2=0x7f893c0fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:16.887 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.887+0000 7f8941db7640 1 -- 172.21.15.98:0/3194668363 shutdown_connections 2024-05-09T08:57:16.887 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:16.888+0000 7f8941db7640 1 -- 172.21.15.98:0/3194668363 wait complete. 2024-05-09T08:57:17.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:16 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:57:17.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:16 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:57:17.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:16 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:57:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:17 smithi098 ceph-mon[30092]: pgmap v204: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 811 MiB/s wr, 1.23k op/s 2024-05-09T08:57:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:17 smithi142 ceph-mon[34625]: pgmap v204: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 811 MiB/s wr, 1.23k op/s 2024-05-09T08:57:18.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:17 smithi105 ceph-mon[34498]: pgmap v204: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 811 MiB/s wr, 1.23k op/s 2024-05-09T08:57:20.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:19 smithi098 ceph-mon[30092]: pgmap v205: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 629 MiB/s wr, 1.16k op/s 2024-05-09T08:57:20.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:19 smithi142 ceph-mon[34625]: pgmap v205: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 629 MiB/s wr, 1.16k op/s 2024-05-09T08:57:20.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:19 smithi105 ceph-mon[34498]: pgmap v205: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 629 MiB/s wr, 1.16k op/s 2024-05-09T08:57:22.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:22 smithi098 ceph-mon[30092]: pgmap v206: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 616 MiB/s wr, 1.41k op/s 2024-05-09T08:57:22.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:22 smithi098 ceph-mon[30092]: osdmap e98: 12 total, 12 up, 12 in 2024-05-09T08:57:22.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:22 smithi142 ceph-mon[34625]: pgmap v206: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 616 MiB/s wr, 1.41k op/s 2024-05-09T08:57:22.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:22 smithi142 ceph-mon[34625]: osdmap e98: 12 total, 12 up, 12 in 2024-05-09T08:57:22.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:22 smithi105 ceph-mon[34498]: pgmap v206: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 616 MiB/s wr, 1.41k op/s 2024-05-09T08:57:22.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:22 smithi105 ceph-mon[34498]: osdmap e98: 12 total, 12 up, 12 in 2024-05-09T08:57:24.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:24 smithi098 ceph-mon[30092]: pgmap v208: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 403 MiB/s wr, 1.36k op/s 2024-05-09T08:57:24.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:24 smithi142 ceph-mon[34625]: pgmap v208: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 403 MiB/s wr, 1.36k op/s 2024-05-09T08:57:24.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:24 smithi105 ceph-mon[34498]: pgmap v208: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 403 MiB/s wr, 1.36k op/s 2024-05-09T08:57:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:26 smithi098 ceph-mon[30092]: pgmap v209: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 316 MiB/s wr, 1.23k op/s 2024-05-09T08:57:26.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:26 smithi142 ceph-mon[34625]: pgmap v209: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 316 MiB/s wr, 1.23k op/s 2024-05-09T08:57:26.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:26 smithi105 ceph-mon[34498]: pgmap v209: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 316 MiB/s wr, 1.23k op/s 2024-05-09T08:57:28.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:28 smithi098 ceph-mon[30092]: pgmap v210: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 103 MiB/s wr, 749 op/s 2024-05-09T08:57:28.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:28 smithi098 ceph-mon[30092]: osdmap e99: 12 total, 12 up, 12 in 2024-05-09T08:57:28.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:28 smithi142 ceph-mon[34625]: pgmap v210: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 103 MiB/s wr, 749 op/s 2024-05-09T08:57:28.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:28 smithi142 ceph-mon[34625]: osdmap e99: 12 total, 12 up, 12 in 2024-05-09T08:57:28.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:28 smithi105 ceph-mon[34498]: pgmap v210: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 103 MiB/s wr, 749 op/s 2024-05-09T08:57:28.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:28 smithi105 ceph-mon[34498]: osdmap e99: 12 total, 12 up, 12 in 2024-05-09T08:57:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.985+0000 7f63e52f5640 1 -- 172.21.15.98:0/4181680712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e01033e0 msgr2=0x7f63e010b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.985+0000 7f63e52f5640 1 --2- 172.21.15.98:0/4181680712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e01033e0 0x7f63e010b8c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009900 tx=0x7f63c802a480 comp rx=0 tx=0).stop 2024-05-09T08:57:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.985+0000 7f63e52f5640 1 -- 172.21.15.98:0/4181680712 shutdown_connections 2024-05-09T08:57:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.985+0000 7f63e52f5640 1 --2- 172.21.15.98:0/4181680712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63e010be90 0x7f63e010e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.985+0000 7f63e52f5640 1 --2- 172.21.15.98:0/4181680712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e01033e0 0x7f63e010b8c0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.985+0000 7f63e52f5640 1 --2- 172.21.15.98:0/4181680712 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63e0102ac0 0x7f63e0102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.985+0000 7f63e52f5640 1 -- 172.21.15.98:0/4181680712 >> 172.21.15.98:0/4181680712 conn(0x7f63e0077840 msgr2=0x7f63e0077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.985+0000 7f63e52f5640 1 -- 172.21.15.98:0/4181680712 shutdown_connections 2024-05-09T08:57:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.985+0000 7f63e52f5640 1 -- 172.21.15.98:0/4181680712 wait complete. 2024-05-09T08:57:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63e52f5640 1 Processor -- start 2024-05-09T08:57:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63e52f5640 1 -- start start 2024-05-09T08:57:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63e52f5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63e0102ac0 0x7f63e01980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63e52f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e01033e0 0x7f63e0198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63e52f5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63e010be90 0x7f63e019c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63e52f5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63e0111c40 con 0x7f63e01033e0 2024-05-09T08:57:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63e52f5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f63e0111ac0 con 0x7f63e010be90 2024-05-09T08:57:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63e52f5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f63e0111dc0 con 0x7f63e0102ac0 2024-05-09T08:57:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63d7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e01033e0 0x7f63e0198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63d7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e01033e0 0x7f63e0198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50302/0 (socket says 172.21.15.98:50302) 2024-05-09T08:57:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63d7fff640 1 -- 172.21.15.98:0/1326781528 learned_addr learned my addr 172.21.15.98:0/1326781528 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63deffd640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63e0102ac0 0x7f63e01980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63df7fe640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63e010be90 0x7f63e019c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63d7fff640 1 -- 172.21.15.98:0/1326781528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63e0102ac0 msgr2=0x7f63e01980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63d7fff640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63e0102ac0 0x7f63e01980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63d7fff640 1 -- 172.21.15.98:0/1326781528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63e010be90 msgr2=0x7f63e019c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.986+0000 7f63d7fff640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63e010be90 0x7f63e019c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.987+0000 7f63d7fff640 1 -- 172.21.15.98:0/1326781528 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e019d060 con 0x7f63e01033e0 2024-05-09T08:57:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.987+0000 7f63df7fe640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63e010be90 0x7f63e019c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.987+0000 7f63d7fff640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e01033e0 0x7f63e0198600 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f63c8006010 tx=0x7f63c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.987+0000 7f63dcff9640 1 -- 172.21.15.98:0/1326781528 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c803f030 con 0x7f63e01033e0 2024-05-09T08:57:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.987+0000 7f63dcff9640 1 -- 172.21.15.98:0/1326781528 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63c8003ce0 con 0x7f63e01033e0 2024-05-09T08:57:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.987+0000 7f63dcff9640 1 -- 172.21.15.98:0/1326781528 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c8035070 con 0x7f63e01033e0 2024-05-09T08:57:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.987+0000 7f63e52f5640 1 -- 172.21.15.98:0/1326781528 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63e019d2f0 con 0x7f63e01033e0 2024-05-09T08:57:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.987+0000 7f63e52f5640 1 -- 172.21.15.98:0/1326781528 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63e01a4be0 con 0x7f63e01033e0 2024-05-09T08:57:29.997 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.989+0000 7f63dcff9640 1 -- 172.21.15.98:0/1326781528 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f63c8042030 con 0x7f63e01033e0 2024-05-09T08:57:29.997 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.989+0000 7f63e8001740 1 Processor -- start 2024-05-09T08:57:29.997 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.990+0000 7f63e8001740 1 -- start start 2024-05-09T08:57:29.997 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.990+0000 7f63e8001740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793330 0x55eefc793710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.990+0000 7f63e8001740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc793c50 0x55eefc7a2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.990+0000 7f63e8001740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc7a2a40 0x55eefc7a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.990+0000 7f63e8001740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eefc6f8340 con 0x55eefc793330 2024-05-09T08:57:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.990+0000 7f63e8001740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55eefc6a0290 con 0x55eefc7a2a40 2024-05-09T08:57:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.990+0000 7f63e8001740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55eefc6f4300 con 0x55eefc793c50 2024-05-09T08:57:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.990+0000 7f63df7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793330 0x55eefc793710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.990+0000 7f63dcff9640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63cc0a2e20 0x7f63cc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.990+0000 7f63d7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc7a2a40 0x55eefc7a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.990+0000 7f63df7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793330 0x55eefc793710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50310/0 (socket says 172.21.15.98:50310) 2024-05-09T08:57:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.990+0000 7f63df7fe640 1 -- 172.21.15.98:0/1807505957 learned_addr learned my addr 172.21.15.98:0/1807505957 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.990+0000 7f63df7fe640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63cc0a2e20 0x7f63cc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.990+0000 7f63deffd640 1 --2- 172.21.15.98:0/1807505957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc793c50 0x55eefc7a2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.991+0000 7f63dcff9640 1 -- 172.21.15.98:0/1326781528 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f63c80efd20 con 0x7f63e01033e0 2024-05-09T08:57:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.991+0000 7f63df7fe640 1 -- 172.21.15.98:0/1807505957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc793c50 msgr2=0x55eefc7a2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.991+0000 7f63df7fe640 1 --2- 172.21.15.98:0/1807505957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc793c50 0x55eefc7a2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.991+0000 7f63df7fe640 1 -- 172.21.15.98:0/1807505957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc7a2a40 msgr2=0x55eefc7a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.991+0000 7f63df7fe640 1 --2- 172.21.15.98:0/1807505957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc7a2a40 0x55eefc7a4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.991+0000 7f63df7fe640 1 -- 172.21.15.98:0/1807505957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eefc6c48e0 con 0x55eefc793330 2024-05-09T08:57:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.991+0000 7f63df7fe640 1 --2- 172.21.15.98:0/1807505957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793330 0x55eefc793710 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f63d8015f00 tx=0x7f63d8012760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73edbf3dab977f6f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.991+0000 7f63df7fe640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63cc0a2e20 0x7f63cc0a52e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009fd0 tx=0x7f63d801d000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.991+0000 7f63d57fa640 1 -- 172.21.15.98:0/1807505957 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8025070 con 0x55eefc793330 2024-05-09T08:57:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.991+0000 7f63d57fa640 1 -- 172.21.15.98:0/1807505957 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63d8009c80 con 0x55eefc793330 2024-05-09T08:57:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.991+0000 7f63d57fa640 1 -- 172.21.15.98:0/1807505957 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8010400 con 0x55eefc793330 2024-05-09T08:57:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.992+0000 7f63e8001740 1 -- 172.21.15.98:0/1807505957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793330 msgr2=0x55eefc793710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.992+0000 7f63e8001740 1 --2- 172.21.15.98:0/1807505957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793330 0x55eefc793710 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f63d8015f00 tx=0x7f63d8012760 comp rx=0 tx=0).stop 2024-05-09T08:57:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.992+0000 7f63e8001740 1 -- 172.21.15.98:0/1807505957 shutdown_connections 2024-05-09T08:57:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.992+0000 7f63e8001740 1 --2- 172.21.15.98:0/1807505957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc7a2a40 0x55eefc7a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.992+0000 7f63e8001740 1 --2- 172.21.15.98:0/1807505957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc793c50 0x55eefc7a2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.992+0000 7f63e8001740 1 --2- 172.21.15.98:0/1807505957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793330 0x55eefc793710 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.992+0000 7f63e8001740 1 -- 172.21.15.98:0/1807505957 >> 172.21.15.98:0/1807505957 conn(0x55eefc78f960 msgr2=0x55eefc78fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.992+0000 7f63e8001740 1 -- 172.21.15.98:0/1807505957 shutdown_connections 2024-05-09T08:57:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.992+0000 7f63e8001740 1 -- 172.21.15.98:0/1807505957 wait complete. 2024-05-09T08:57:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.992+0000 7f63e8001740 1 -- start start 2024-05-09T08:57:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.992+0000 7f63e8001740 1 Processor -- start 2024-05-09T08:57:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63e8001740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc793330 0x55eefc79f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63e8001740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc793c50 0x55eefc79fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63e8001740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc7a2a40 0x55eefc7a0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63e8001740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eefc7a5610 con 0x55eefc7a2a40 2024-05-09T08:57:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63e8001740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55eefc6b3790 con 0x55eefc793c50 2024-05-09T08:57:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63e8001740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55eefc7a5790 con 0x55eefc793330 2024-05-09T08:57:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63e8001740 10 client.0 _ll_register_callbacks cb 0x55eefc6eea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63df7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc7a2a40 0x55eefc7a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63df7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc7a2a40 0x55eefc7a0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50318/0 (socket says 172.21.15.98:50318) 2024-05-09T08:57:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63df7fe640 1 -- 172.21.15.98:0/3215247380 learned_addr learned my addr 172.21.15.98:0/3215247380 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63deffd640 1 --2- 172.21.15.98:0/3215247380 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc793330 0x55eefc79f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63df7fe640 1 -- 172.21.15.98:0/3215247380 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc793330 msgr2=0x55eefc79f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63df7fe640 1 --2- 172.21.15.98:0/3215247380 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc793330 0x55eefc79f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63d7fff640 1 --2- 172.21.15.98:0/3215247380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc793c50 0x55eefc79fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63df7fe640 1 -- 172.21.15.98:0/3215247380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc793c50 msgr2=0x55eefc79fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63df7fe640 1 --2- 172.21.15.98:0/3215247380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc793c50 0x55eefc79fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63df7fe640 1 -- 172.21.15.98:0/3215247380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eefc40c210 con 0x55eefc7a2a40 2024-05-09T08:57:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63deffd640 1 --2- 172.21.15.98:0/3215247380 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc793330 0x55eefc79f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63df7fe640 1 --2- 172.21.15.98:0/3215247380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc7a2a40 0x55eefc7a0400 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f63d8021b50 tx=0x7f63d8009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7c211a76d592c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:30.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63c67fc640 1 -- 172.21.15.98:0/3215247380 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d802e030 con 0x55eefc7a2a40 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63e8001740 1 -- 172.21.15.98:0/3215247380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63d8009c80 con 0x55eefc7a2a40 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63c67fc640 1 -- 172.21.15.98:0/3215247380 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f63d8004c10 con 0x55eefc7a2a40 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63c67fc640 1 -- 172.21.15.98:0/3215247380 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f63d803b050 con 0x55eefc7a2a40 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63e8001740 10 client.17352 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.993+0000 7f63e8001740 1 -- 172.21.15.98:0/3215247380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55eefc4d8620 con 0x55eefc7a2a40 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.994+0000 7f63c67fc640 1 -- 172.21.15.98:0/3215247380 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63d8009c80 con 0x55eefc7a2a40 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.994+0000 7f63c67fc640 1 -- 172.21.15.98:0/3215247380 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f63d801e030 con 0x55eefc7a2a40 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.994+0000 7f63e8001740 10 client.17352 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.994+0000 7f63e8001740 10 client.17352 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.994+0000 7f63e8001740 20 client.17352 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.994+0000 7f63e8001740 1 --2- 172.21.15.98:0/3215247380 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eefc7b2710 0x55eefc7b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.994+0000 7f63e8001740 4 client.17352 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.994+0000 7f63e8001740 1 -- 172.21.15.98:0/3215247380 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eefc705a70 con 0x55eefc7b2710 2024-05-09T08:57:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.994+0000 7f63deffd640 1 --2- 172.21.15.98:0/3215247380 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eefc7b2710 0x55eefc7b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.995+0000 7f63c67fc640 10 client.17352 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.995+0000 7f63deffd640 1 --2- 172.21.15.98:0/3215247380 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eefc7b2710 0x55eefc7b4b00 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5017ae8b457239 server_cookie=44af9d5cb84b1f4a in_seq=0 out_seq=0 2024-05-09T08:57:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.995+0000 7f63c67fc640 1 -- 172.21.15.98:0/3215247380 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eefc705a70 con 0x55eefc7b2710 2024-05-09T08:57:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.995+0000 7f63c67fc640 10 client.17352 handle_command_reply: tid=0 2024-05-09T08:57:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.995+0000 7f63e8001740 1 client.17352 shutdown 2024-05-09T08:57:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.995+0000 7f63e8001740 1 -- 172.21.15.98:0/3215247380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc7a2a40 msgr2=0x55eefc7a0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.995+0000 7f63e8001740 1 --2- 172.21.15.98:0/3215247380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc7a2a40 0x55eefc7a0400 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f63d8021b50 tx=0x7f63d8009730 comp rx=0 tx=0).stop 2024-05-09T08:57:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.995+0000 7f63e8001740 1 -- 172.21.15.98:0/3215247380 shutdown_connections 2024-05-09T08:57:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.995+0000 7f63e8001740 1 --2- 172.21.15.98:0/3215247380 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eefc7b2710 0x55eefc7b4b00 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.995+0000 7f63e8001740 1 --2- 172.21.15.98:0/3215247380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc7a2a40 0x55eefc7a0400 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.995+0000 7f63e8001740 1 --2- 172.21.15.98:0/3215247380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc793c50 0x55eefc79fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.995+0000 7f63e8001740 1 --2- 172.21.15.98:0/3215247380 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc793330 0x55eefc79f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.995+0000 7f63c67fc640 0 client.17352 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.995+0000 7f63e8001740 1 -- 172.21.15.98:0/3215247380 >> 172.21.15.98:0/3215247380 conn(0x55eefc78f960 msgr2=0x55eefc7957d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.996+0000 7f63e8001740 1 -- 172.21.15.98:0/3215247380 shutdown_connections 2024-05-09T08:57:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.996+0000 7f63e8001740 1 -- 172.21.15.98:0/3215247380 wait complete. 2024-05-09T08:57:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:29.996+0000 7f63e8001740 20 client.17352 trim_cache size 0 max 16384 2024-05-09T08:57:30.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.010+0000 7f63e8001740 1 Processor -- start 2024-05-09T08:57:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.010+0000 7f63e8001740 1 -- start start 2024-05-09T08:57:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63e8001740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc793330 0x55eefc7a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63e8001740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793c50 0x55eefc7ae510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63e8001740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc7a2a40 0x55eefc79e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63e8001740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63e01acc00 con 0x55eefc793c50 2024-05-09T08:57:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63e8001740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f63e01ae740 con 0x55eefc793330 2024-05-09T08:57:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63e8001740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f63e01ad2d0 con 0x55eefc7a2a40 2024-05-09T08:57:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63d7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793c50 0x55eefc7ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63d7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793c50 0x55eefc7ae510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50334/0 (socket says 172.21.15.98:50334) 2024-05-09T08:57:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63d7fff640 1 -- 172.21.15.98:0/3241070919 learned_addr learned my addr 172.21.15.98:0/3241070919 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63d7fff640 1 -- 172.21.15.98:0/3241070919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc7a2a40 msgr2=0x55eefc79e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63d7fff640 1 --2- 172.21.15.98:0/3241070919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc7a2a40 0x55eefc79e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63d7fff640 1 -- 172.21.15.98:0/3241070919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc793330 msgr2=0x55eefc7a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63d7fff640 1 --2- 172.21.15.98:0/3241070919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc793330 0x55eefc7a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63d7fff640 1 -- 172.21.15.98:0/3241070919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eefc7948d0 con 0x55eefc793c50 2024-05-09T08:57:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63d7fff640 1 --2- 172.21.15.98:0/3241070919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793c50 0x55eefc7ae510 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f63c8000c00 tx=0x7f63c80e69c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f114e68b62074d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63d4ff9640 1 -- 172.21.15.98:0/3241070919 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c8049070 con 0x55eefc793c50 2024-05-09T08:57:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63d4ff9640 1 -- 172.21.15.98:0/3241070919 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63c803b580 con 0x55eefc793c50 2024-05-09T08:57:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.011+0000 7f63d4ff9640 1 -- 172.21.15.98:0/3241070919 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c8086070 con 0x55eefc793c50 2024-05-09T08:57:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.012+0000 7f63e8001740 1 -- 172.21.15.98:0/3241070919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793c50 msgr2=0x55eefc7ae510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.012+0000 7f63e8001740 1 --2- 172.21.15.98:0/3241070919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793c50 0x55eefc7ae510 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f63c8000c00 tx=0x7f63c80e69c0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.012+0000 7f63e8001740 1 -- 172.21.15.98:0/3241070919 shutdown_connections 2024-05-09T08:57:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.012+0000 7f63e8001740 1 --2- 172.21.15.98:0/3241070919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc7a2a40 0x55eefc79e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.012+0000 7f63e8001740 1 --2- 172.21.15.98:0/3241070919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793c50 0x55eefc7ae510 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.012+0000 7f63e8001740 1 --2- 172.21.15.98:0/3241070919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc793330 0x55eefc7a20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.012+0000 7f63e8001740 1 -- 172.21.15.98:0/3241070919 >> 172.21.15.98:0/3241070919 conn(0x55eefc78f960 msgr2=0x55eefc79e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.012+0000 7f63e8001740 1 -- 172.21.15.98:0/3241070919 shutdown_connections 2024-05-09T08:57:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.012+0000 7f63e8001740 1 -- 172.21.15.98:0/3241070919 wait complete. 2024-05-09T08:57:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.012+0000 7f63e8001740 1 -- start start 2024-05-09T08:57:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63e8001740 1 Processor -- start 2024-05-09T08:57:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63e8001740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc793330 0x55eefc7aafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63e8001740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793c50 0x55eefc79c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63e8001740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc7a2a40 0x55eefc79cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63e8001740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63e01ae070 con 0x55eefc793c50 2024-05-09T08:57:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63e8001740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f63e01ad9a0 con 0x55eefc793330 2024-05-09T08:57:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63e8001740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f63e01aac00 con 0x55eefc7a2a40 2024-05-09T08:57:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63e8001740 10 client.0 _ll_register_callbacks cb 0x7f63c80bbcc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63d7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793c50 0x55eefc79c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63d7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793c50 0x55eefc79c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50340/0 (socket says 172.21.15.98:50340) 2024-05-09T08:57:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63d7fff640 1 -- 172.21.15.98:0/3665788963 learned_addr learned my addr 172.21.15.98:0/3665788963 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63d7fff640 1 -- 172.21.15.98:0/3665788963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc7a2a40 msgr2=0x55eefc79cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63d7fff640 1 --2- 172.21.15.98:0/3665788963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc7a2a40 0x55eefc79cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63d7fff640 1 -- 172.21.15.98:0/3665788963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc793330 msgr2=0x55eefc7aafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63d7fff640 1 --2- 172.21.15.98:0/3665788963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc793330 0x55eefc7aafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63d7fff640 1 -- 172.21.15.98:0/3665788963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eefc8179a0 con 0x55eefc793c50 2024-05-09T08:57:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63d7fff640 1 --2- 172.21.15.98:0/3665788963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793c50 0x55eefc79c4c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f63c80bbe80 tx=0x7f63c803bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431513fdad53cb2f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63c67fc640 1 -- 172.21.15.98:0/3665788963 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c80d8030 con 0x55eefc793c50 2024-05-09T08:57:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63c67fc640 1 -- 172.21.15.98:0/3665788963 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f63c80f1070 con 0x55eefc793c50 2024-05-09T08:57:30.022 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.013+0000 7f63e8001740 1 -- 172.21.15.98:0/3665788963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55eefc6ffdc0 con 0x55eefc793c50 2024-05-09T08:57:30.022 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.014+0000 7f63c67fc640 1 -- 172.21.15.98:0/3665788963 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f63c8049070 con 0x55eefc793c50 2024-05-09T08:57:30.022 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.014+0000 7f63e8001740 10 client.17358 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:30.022 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.014+0000 7f63e8001740 1 -- 172.21.15.98:0/3665788963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55eefc818af0 con 0x55eefc793c50 2024-05-09T08:57:30.022 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.014+0000 7f63c67fc640 1 -- 172.21.15.98:0/3665788963 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55eefc6ffdc0 con 0x55eefc793c50 2024-05-09T08:57:30.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.014+0000 7f63c67fc640 1 -- 172.21.15.98:0/3665788963 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f63c80b8560 con 0x55eefc793c50 2024-05-09T08:57:30.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.014+0000 7f63e8001740 10 client.17358 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:30.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.014+0000 7f63e8001740 10 client.17358 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:30.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.014+0000 7f63e8001740 20 client.17358 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:30.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.014+0000 7f63e8001740 1 --2- 172.21.15.98:0/3665788963 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eefc819190 0x55eefc81b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:30.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.014+0000 7f63e8001740 4 client.17358 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:57:30.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.014+0000 7f63e8001740 1 -- 172.21.15.98:0/3665788963 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eefc67d8e0 con 0x55eefc819190 2024-05-09T08:57:30.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.014+0000 7f63deffd640 1 --2- 172.21.15.98:0/3665788963 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eefc819190 0x55eefc81b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:30.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.015+0000 7f63c67fc640 10 client.17358 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:30.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.015+0000 7f63deffd640 1 --2- 172.21.15.98:0/3665788963 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eefc819190 0x55eefc81b5b0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cca874bc5c719668 server_cookie=d90ba8ee34d44f91 in_seq=0 out_seq=0 2024-05-09T08:57:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.036+0000 7f63c67fc640 1 -- 172.21.15.98:0/3665788963 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eefc67d8e0 con 0x55eefc819190 2024-05-09T08:57:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.036+0000 7f63c67fc640 10 client.17358 handle_command_reply: tid=0 2024-05-09T08:57:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.036+0000 7f63e8001740 1 client.17358 shutdown 2024-05-09T08:57:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.036+0000 7f63e8001740 1 -- 172.21.15.98:0/3665788963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793c50 msgr2=0x55eefc79c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.036+0000 7f63e8001740 1 --2- 172.21.15.98:0/3665788963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793c50 0x55eefc79c4c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f63c80bbe80 tx=0x7f63c803bcb0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.036+0000 7f63e8001740 1 -- 172.21.15.98:0/3665788963 shutdown_connections 2024-05-09T08:57:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.036+0000 7f63e8001740 1 --2- 172.21.15.98:0/3665788963 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55eefc819190 0x55eefc81b5b0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.036+0000 7f63e8001740 1 --2- 172.21.15.98:0/3665788963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eefc7a2a40 0x55eefc79cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.036+0000 7f63c67fc640 0 client.17358 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.036+0000 7f63e8001740 1 --2- 172.21.15.98:0/3665788963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eefc793c50 0x55eefc79c4c0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.036+0000 7f63e8001740 1 --2- 172.21.15.98:0/3665788963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eefc793330 0x55eefc7aafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.036+0000 7f63e8001740 1 -- 172.21.15.98:0/3665788963 >> 172.21.15.98:0/3665788963 conn(0x55eefc78f960 msgr2=0x55eefc79e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e8001740 1 -- 172.21.15.98:0/3665788963 shutdown_connections 2024-05-09T08:57:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e8001740 1 -- 172.21.15.98:0/3665788963 wait complete. 2024-05-09T08:57:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e8001740 20 client.17358 trim_cache size 0 max 16384 2024-05-09T08:57:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:57:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:57:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:57:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:57:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e52f5640 1 -- 172.21.15.98:0/1326781528 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63cc0a2e20 msgr2=0x7f63cc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e52f5640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63cc0a2e20 0x7f63cc0a52e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009fd0 tx=0x7f63d801d000 comp rx=0 tx=0).stop 2024-05-09T08:57:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e52f5640 1 -- 172.21.15.98:0/1326781528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e01033e0 msgr2=0x7f63e0198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e52f5640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e01033e0 0x7f63e0198600 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f63c8006010 tx=0x7f63c80042f0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e52f5640 1 -- 172.21.15.98:0/1326781528 shutdown_connections 2024-05-09T08:57:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e52f5640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63cc0a2e20 0x7f63cc0a52e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e52f5640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63e010be90 0x7f63e019c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e52f5640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e01033e0 0x7f63e0198600 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e52f5640 1 --2- 172.21.15.98:0/1326781528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63e0102ac0 0x7f63e01980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e52f5640 1 -- 172.21.15.98:0/1326781528 >> 172.21.15.98:0/1326781528 conn(0x7f63e0077840 msgr2=0x7f63e00fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e52f5640 1 -- 172.21.15.98:0/1326781528 shutdown_connections 2024-05-09T08:57:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:30.037+0000 7f63e52f5640 1 -- 172.21.15.98:0/1326781528 wait complete. 2024-05-09T08:57:30.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:30 smithi098 ceph-mon[30092]: pgmap v212: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 83 MiB/s wr, 565 op/s 2024-05-09T08:57:30.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:30 smithi142 ceph-mon[34625]: pgmap v212: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 83 MiB/s wr, 565 op/s 2024-05-09T08:57:30.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:30 smithi105 ceph-mon[34498]: pgmap v212: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 83 MiB/s wr, 565 op/s 2024-05-09T08:57:32.289 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:32 smithi105 ceph-mon[34498]: pgmap v213: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 178 MiB/s wr, 1.38k op/s 2024-05-09T08:57:32.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:32 smithi098 ceph-mon[30092]: pgmap v213: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 178 MiB/s wr, 1.38k op/s 2024-05-09T08:57:32.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:32 smithi142 ceph-mon[34625]: pgmap v213: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 178 MiB/s wr, 1.38k op/s 2024-05-09T08:57:34.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:34 smithi098 ceph-mon[30092]: pgmap v214: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 208 MiB/s wr, 1.63k op/s 2024-05-09T08:57:34.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:34 smithi142 ceph-mon[34625]: pgmap v214: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 208 MiB/s wr, 1.63k op/s 2024-05-09T08:57:34.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:34 smithi105 ceph-mon[34498]: pgmap v214: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 208 MiB/s wr, 1.63k op/s 2024-05-09T08:57:36.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:36 smithi098 ceph-mon[30092]: pgmap v215: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 210 MiB/s wr, 1.68k op/s 2024-05-09T08:57:36.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:36 smithi142 ceph-mon[34625]: pgmap v215: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 210 MiB/s wr, 1.68k op/s 2024-05-09T08:57:36.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:36 smithi105 ceph-mon[34498]: pgmap v215: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 210 MiB/s wr, 1.68k op/s 2024-05-09T08:57:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:38 smithi098 ceph-mon[30092]: pgmap v216: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 180 MiB/s wr, 1.51k op/s 2024-05-09T08:57:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:38 smithi142 ceph-mon[34625]: pgmap v216: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 180 MiB/s wr, 1.51k op/s 2024-05-09T08:57:38.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:38 smithi105 ceph-mon[34498]: pgmap v216: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 180 MiB/s wr, 1.51k op/s 2024-05-09T08:57:39.038 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:57:39.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:57:39.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.156+0000 7fe44ba91640 1 -- 172.21.15.98:0/2416746524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe444107820 msgr2=0x7fe444107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.156+0000 7fe44ba91640 1 --2- 172.21.15.98:0/2416746524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe444107820 0x7fe444107c00 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe440009930 tx=0x7fe44002a420 comp rx=0 tx=0).stop 2024-05-09T08:57:39.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.156+0000 7fe44ba91640 1 -- 172.21.15.98:0/2416746524 shutdown_connections 2024-05-09T08:57:39.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.156+0000 7fe44ba91640 1 --2- 172.21.15.98:0/2416746524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe44406a870 0x7fe44410e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.156+0000 7fe44ba91640 1 --2- 172.21.15.98:0/2416746524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe444069eb0 0x7fe44406a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.156+0000 7fe44ba91640 1 --2- 172.21.15.98:0/2416746524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe444107820 0x7fe444107c00 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.156+0000 7fe44ba91640 1 -- 172.21.15.98:0/2416746524 >> 172.21.15.98:0/2416746524 conn(0x7fe444077790 msgr2=0x7fe444077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.156+0000 7fe44ba91640 1 -- 172.21.15.98:0/2416746524 shutdown_connections 2024-05-09T08:57:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.156+0000 7fe44ba91640 1 -- 172.21.15.98:0/2416746524 wait complete. 2024-05-09T08:57:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44ba91640 1 Processor -- start 2024-05-09T08:57:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44ba91640 1 -- start start 2024-05-09T08:57:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44ba91640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe444069eb0 0x7fe44419a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44ba91640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe44406a870 0x7fe44419a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44ba91640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe444107820 0x7fe44419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44ba91640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe444111c30 con 0x7fe444107820 2024-05-09T08:57:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44ba91640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe444111ab0 con 0x7fe444069eb0 2024-05-09T08:57:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44ba91640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe444111db0 con 0x7fe44406a870 2024-05-09T08:57:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44a02e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe444107820 0x7fe44419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44a02e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe444107820 0x7fe44419ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55208/0 (socket says 172.21.15.98:55208) 2024-05-09T08:57:39.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44a02e640 1 -- 172.21.15.98:0/756498557 learned_addr learned my addr 172.21.15.98:0/756498557 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44982d640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe444069eb0 0x7fe44419a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44902c640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe44406a870 0x7fe44419a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44a02e640 1 -- 172.21.15.98:0/756498557 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe44406a870 msgr2=0x7fe44419a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44a02e640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe44406a870 0x7fe44419a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44a02e640 1 -- 172.21.15.98:0/756498557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe444069eb0 msgr2=0x7fe44419a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.157+0000 7fe44a02e640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe444069eb0 0x7fe44419a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.158+0000 7fe44a02e640 1 -- 172.21.15.98:0/756498557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe44419f380 con 0x7fe444107820 2024-05-09T08:57:39.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.158+0000 7fe44902c640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe44406a870 0x7fe44419a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:39.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.158+0000 7fe44982d640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe444069eb0 0x7fe44419a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:39.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.158+0000 7fe44a02e640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe444107820 0x7fe44419ec50 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00cad0 tx=0x7fe43c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.158+0000 7fe43affd640 1 -- 172.21.15.98:0/756498557 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c00ae20 con 0x7fe444107820 2024-05-09T08:57:39.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.158+0000 7fe43affd640 1 -- 172.21.15.98:0/756498557 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe43c005020 con 0x7fe444107820 2024-05-09T08:57:39.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.158+0000 7fe43affd640 1 -- 172.21.15.98:0/756498557 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c00f6f0 con 0x7fe444107820 2024-05-09T08:57:39.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.158+0000 7fe44ba91640 1 -- 172.21.15.98:0/756498557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe44419f670 con 0x7fe444107820 2024-05-09T08:57:39.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.158+0000 7fe44ba91640 1 -- 172.21.15.98:0/756498557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4441a6f00 con 0x7fe444107820 2024-05-09T08:57:39.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.161+0000 7fe44e795740 1 Processor -- start 2024-05-09T08:57:39.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.161+0000 7fe43affd640 1 -- 172.21.15.98:0/756498557 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe43c0051d0 con 0x7fe444107820 2024-05-09T08:57:39.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.161+0000 7fe44e795740 1 -- start start 2024-05-09T08:57:39.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44e795740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7ea660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44e795740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7eaba0 0x5621cd7f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44e795740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7f5270 0x5621cd7f7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44e795740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5621cd74fd00 con 0x5621cd7ea280 2024-05-09T08:57:39.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44e795740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5621cd6f74c0 con 0x5621cd7f5270 2024-05-09T08:57:39.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44e795740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5621cd74bcc0 con 0x5621cd7eaba0 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44902c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7ea660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44902c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7ea660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55212/0 (socket says 172.21.15.98:55212) 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44902c640 1 -- 172.21.15.98:0/3665885033 learned_addr learned my addr 172.21.15.98:0/3665885033 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe43affd640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe4300a2d80 0x7fe4300a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44982d640 1 --2- 172.21.15.98:0/3665885033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7eaba0 0x5621cd7f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44902c640 1 -- 172.21.15.98:0/3665885033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7eaba0 msgr2=0x5621cd7f4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44902c640 1 --2- 172.21.15.98:0/3665885033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7eaba0 0x5621cd7f4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44902c640 1 -- 172.21.15.98:0/3665885033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7f5270 msgr2=0x5621cd7f7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44902c640 1 --2- 172.21.15.98:0/3665885033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7f5270 0x5621cd7f7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44902c640 1 -- 172.21.15.98:0/3665885033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621cd52d7d0 con 0x5621cd7ea280 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44a02e640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe4300a2d80 0x7fe4300a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe44902c640 1 --2- 172.21.15.98:0/3665885033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7ea660 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fe43400bdb0 tx=0x7fe434011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e568aea0e1b8ea7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe43affd640 1 -- 172.21.15.98:0/756498557 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fe43c0cf8b0 con 0x7fe444107820 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.162+0000 7fe42ffff640 1 -- 172.21.15.98:0/3665885033 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe434011e60 con 0x5621cd7ea280 2024-05-09T08:57:39.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.163+0000 7fe42ffff640 1 -- 172.21.15.98:0/3665885033 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe43401aae0 con 0x5621cd7ea280 2024-05-09T08:57:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.163+0000 7fe42ffff640 1 -- 172.21.15.98:0/3665885033 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe434012b60 con 0x5621cd7ea280 2024-05-09T08:57:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.163+0000 7fe44e795740 1 -- 172.21.15.98:0/3665885033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 msgr2=0x5621cd7ea660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.163+0000 7fe44a02e640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe4300a2d80 0x7fe4300a5240 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe43c0cd1f0 tx=0x7fe43c06d980 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.163+0000 7fe44e795740 1 --2- 172.21.15.98:0/3665885033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7ea660 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fe43400bdb0 tx=0x7fe434011750 comp rx=0 tx=0).stop 2024-05-09T08:57:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.163+0000 7fe44e795740 1 -- 172.21.15.98:0/3665885033 shutdown_connections 2024-05-09T08:57:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.163+0000 7fe44e795740 1 --2- 172.21.15.98:0/3665885033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7f5270 0x5621cd7f7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.163+0000 7fe44e795740 1 --2- 172.21.15.98:0/3665885033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7eaba0 0x5621cd7f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.163+0000 7fe44e795740 1 --2- 172.21.15.98:0/3665885033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7ea660 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.163+0000 7fe44e795740 1 -- 172.21.15.98:0/3665885033 >> 172.21.15.98:0/3665885033 conn(0x5621cd7e65a0 msgr2=0x5621cd7e6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.163+0000 7fe44e795740 1 -- 172.21.15.98:0/3665885033 shutdown_connections 2024-05-09T08:57:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.163+0000 7fe44e795740 1 -- 172.21.15.98:0/3665885033 wait complete. 2024-05-09T08:57:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.164+0000 7fe44e795740 1 -- start start 2024-05-09T08:57:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.164+0000 7fe44e795740 1 Processor -- start 2024-05-09T08:57:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.164+0000 7fe44e795740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7f2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.164+0000 7fe44e795740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7eaba0 0x5621cd7f2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.164+0000 7fe44e795740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7f5270 0x5621cd7f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.164+0000 7fe44e795740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5621cd7f8060 con 0x5621cd7ea280 2024-05-09T08:57:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.164+0000 7fe44e795740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5621cd70ab90 con 0x5621cd7eaba0 2024-05-09T08:57:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.164+0000 7fe44e795740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5621cd7f81e0 con 0x5621cd7f5270 2024-05-09T08:57:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.164+0000 7fe44e795740 10 client.0 _ll_register_callbacks cb 0x5621cd746da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.164+0000 7fe44902c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.164+0000 7fe44902c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7f2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55218/0 (socket says 172.21.15.98:55218) 2024-05-09T08:57:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.164+0000 7fe44902c640 1 -- 172.21.15.98:0/955785781 learned_addr learned my addr 172.21.15.98:0/955785781 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe44982d640 1 --2- 172.21.15.98:0/955785781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7eaba0 0x5621cd7f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe44902c640 1 -- 172.21.15.98:0/955785781 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7f5270 msgr2=0x5621cd7f2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe44902c640 1 --2- 172.21.15.98:0/955785781 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7f5270 0x5621cd7f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe44902c640 1 -- 172.21.15.98:0/955785781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7eaba0 msgr2=0x5621cd7f2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe44902c640 1 --2- 172.21.15.98:0/955785781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7eaba0 0x5621cd7f2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe44902c640 1 -- 172.21.15.98:0/955785781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621cd4633f0 con 0x5621cd7ea280 2024-05-09T08:57:39.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe44982d640 1 --2- 172.21.15.98:0/955785781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7eaba0 0x5621cd7f2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:39.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe44902c640 1 --2- 172.21.15.98:0/955785781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7f2210 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fe434012760 tx=0x7fe43400bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1ce97057d275a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe42d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:39.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe42d7fa640 1 -- 172.21.15.98:0/955785781 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43402b060 con 0x5621cd7ea280 2024-05-09T08:57:39.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe42d7fa640 1 -- 172.21.15.98:0/955785781 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fe43401a600 con 0x5621cd7ea280 2024-05-09T08:57:39.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe44e795740 1 -- 172.21.15.98:0/955785781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe43401aae0 con 0x5621cd7ea280 2024-05-09T08:57:39.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe42d7fa640 1 -- 172.21.15.98:0/955785781 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe434020070 con 0x5621cd7ea280 2024-05-09T08:57:39.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe44e795740 10 client.17370 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:39.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe44e795740 1 -- 172.21.15.98:0/955785781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5621cd4230f0 con 0x5621cd7ea280 2024-05-09T08:57:39.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.165+0000 7fe42d7fa640 1 -- 172.21.15.98:0/955785781 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe43401aae0 con 0x5621cd7ea280 2024-05-09T08:57:39.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.166+0000 7fe42d7fa640 1 -- 172.21.15.98:0/955785781 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe43400f6c0 con 0x5621cd7ea280 2024-05-09T08:57:39.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.166+0000 7fe44e795740 10 client.17370 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:39.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.166+0000 7fe44e795740 10 client.17370 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:39.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.166+0000 7fe44e795740 20 client.17370 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:39.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.166+0000 7fe44e795740 1 --2- 172.21.15.98:0/955785781 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5621cd805500 0x5621cd8078f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.166+0000 7fe44e795740 4 client.17370 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:57:39.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.166+0000 7fe44e795740 1 -- 172.21.15.98:0/955785781 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621cd75d430 con 0x5621cd805500 2024-05-09T08:57:39.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.166+0000 7fe44982d640 1 --2- 172.21.15.98:0/955785781 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5621cd805500 0x5621cd8078f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.167+0000 7fe42d7fa640 10 client.17370 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:39.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.167+0000 7fe44982d640 1 --2- 172.21.15.98:0/955785781 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5621cd805500 0x5621cd8078f0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae5b17ff839f6e64 server_cookie=a7682e22b3bc7b6 in_seq=0 out_seq=0 2024-05-09T08:57:39.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.167+0000 7fe42d7fa640 1 -- 172.21.15.98:0/955785781 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621cd75d430 con 0x5621cd805500 2024-05-09T08:57:39.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.167+0000 7fe42d7fa640 10 client.17370 handle_command_reply: tid=0 2024-05-09T08:57:39.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.167+0000 7fe44e795740 1 client.17370 shutdown 2024-05-09T08:57:39.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.167+0000 7fe44e795740 1 -- 172.21.15.98:0/955785781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 msgr2=0x5621cd7f2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.167+0000 7fe44e795740 1 --2- 172.21.15.98:0/955785781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7f2210 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fe434012760 tx=0x7fe43400bb70 comp rx=0 tx=0).stop 2024-05-09T08:57:39.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.167+0000 7fe44e795740 1 -- 172.21.15.98:0/955785781 shutdown_connections 2024-05-09T08:57:39.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.167+0000 7fe44e795740 1 --2- 172.21.15.98:0/955785781 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5621cd805500 0x5621cd8078f0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.167+0000 7fe44e795740 1 --2- 172.21.15.98:0/955785781 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7f5270 0x5621cd7f2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.167+0000 7fe42d7fa640 0 client.17370 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:39.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.167+0000 7fe44e795740 1 --2- 172.21.15.98:0/955785781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7eaba0 0x5621cd7f2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.167+0000 7fe44e795740 1 --2- 172.21.15.98:0/955785781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7f2210 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.167+0000 7fe44e795740 1 -- 172.21.15.98:0/955785781 >> 172.21.15.98:0/955785781 conn(0x5621cd7e65a0 msgr2=0x5621cd7ec3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.168+0000 7fe44e795740 1 -- 172.21.15.98:0/955785781 shutdown_connections 2024-05-09T08:57:39.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.168+0000 7fe44e795740 1 -- 172.21.15.98:0/955785781 wait complete. 2024-05-09T08:57:39.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.168+0000 7fe44e795740 20 client.17370 trim_cache size 0 max 16384 2024-05-09T08:57:39.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44e795740 1 Processor -- start 2024-05-09T08:57:39.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44e795740 1 -- start start 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44e795740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7ea280 0x5621cd800f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44e795740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7eaba0 0x5621cd7e1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44e795740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7f5270 0x5621cd80cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44e795740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe4441b1140 con 0x5621cd7eaba0 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44e795740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe4441b2c80 con 0x5621cd7f5270 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44e795740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe4441b1810 con 0x5621cd7ea280 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44982d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7eaba0 0x5621cd7e1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44982d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7eaba0 0x5621cd7e1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55230/0 (socket says 172.21.15.98:55230) 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44982d640 1 -- 172.21.15.98:0/1220705606 learned_addr learned my addr 172.21.15.98:0/1220705606 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44902c640 1 --2- 172.21.15.98:0/1220705606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7ea280 0x5621cd800f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44982d640 1 -- 172.21.15.98:0/1220705606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7ea280 msgr2=0x5621cd800f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44982d640 1 --2- 172.21.15.98:0/1220705606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7ea280 0x5621cd800f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44982d640 1 -- 172.21.15.98:0/1220705606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7f5270 msgr2=0x5621cd80cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44982d640 1 --2- 172.21.15.98:0/1220705606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7f5270 0x5621cd80cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44982d640 1 -- 172.21.15.98:0/1220705606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621cd7f11c0 con 0x5621cd7eaba0 2024-05-09T08:57:39.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44902c640 1 --2- 172.21.15.98:0/1220705606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7ea280 0x5621cd800f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:39.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.185+0000 7fe44982d640 1 --2- 172.21.15.98:0/1220705606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7eaba0 0x5621cd7e1670 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe440009930 tx=0x7fe440005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e00144eb671158 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.186+0000 7fe42f7fe640 1 -- 172.21.15.98:0/1220705606 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe440047070 con 0x5621cd7eaba0 2024-05-09T08:57:39.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.186+0000 7fe42f7fe640 1 -- 172.21.15.98:0/1220705606 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe440005520 con 0x5621cd7eaba0 2024-05-09T08:57:39.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.186+0000 7fe42f7fe640 1 -- 172.21.15.98:0/1220705606 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe440035040 con 0x5621cd7eaba0 2024-05-09T08:57:39.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.186+0000 7fe44e795740 1 -- 172.21.15.98:0/1220705606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7eaba0 msgr2=0x5621cd7e1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.186+0000 7fe44e795740 1 --2- 172.21.15.98:0/1220705606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7eaba0 0x5621cd7e1670 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe440009930 tx=0x7fe440005a20 comp rx=0 tx=0).stop 2024-05-09T08:57:39.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.186+0000 7fe44e795740 1 -- 172.21.15.98:0/1220705606 shutdown_connections 2024-05-09T08:57:39.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.186+0000 7fe44e795740 1 --2- 172.21.15.98:0/1220705606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7f5270 0x5621cd80cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.186+0000 7fe44e795740 1 --2- 172.21.15.98:0/1220705606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7eaba0 0x5621cd7e1670 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.186+0000 7fe44e795740 1 --2- 172.21.15.98:0/1220705606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7ea280 0x5621cd800f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.186+0000 7fe44e795740 1 -- 172.21.15.98:0/1220705606 >> 172.21.15.98:0/1220705606 conn(0x5621cd7e65a0 msgr2=0x5621cd7e57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.186+0000 7fe44e795740 1 -- 172.21.15.98:0/1220705606 shutdown_connections 2024-05-09T08:57:39.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.186+0000 7fe44e795740 1 -- 172.21.15.98:0/1220705606 wait complete. 2024-05-09T08:57:39.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.186+0000 7fe44e795740 1 -- start start 2024-05-09T08:57:39.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44e795740 1 Processor -- start 2024-05-09T08:57:39.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44e795740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7f95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44e795740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7eaba0 0x5621cd7f9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44e795740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7f5270 0x5621cd86a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44e795740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe4441b25b0 con 0x5621cd7ea280 2024-05-09T08:57:39.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44e795740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe4441b1ee0 con 0x5621cd7eaba0 2024-05-09T08:57:39.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44e795740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe4441af140 con 0x5621cd7f5270 2024-05-09T08:57:39.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44e795740 10 client.0 _ll_register_callbacks cb 0x7fe43400bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:39.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44902c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7f95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44902c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7f95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55246/0 (socket says 172.21.15.98:55246) 2024-05-09T08:57:39.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44902c640 1 -- 172.21.15.98:0/4137052916 learned_addr learned my addr 172.21.15.98:0/4137052916 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44982d640 1 --2- 172.21.15.98:0/4137052916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7eaba0 0x5621cd7f9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44902c640 1 --2- 172.21.15.98:0/4137052916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7f5270 0x5621cd86a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44902c640 1 -- 172.21.15.98:0/4137052916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7f5270 msgr2=0x5621cd86a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44902c640 1 --2- 172.21.15.98:0/4137052916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7f5270 0x5621cd86a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44902c640 1 -- 172.21.15.98:0/4137052916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7eaba0 msgr2=0x5621cd7f9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44902c640 1 --2- 172.21.15.98:0/4137052916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7eaba0 0x5621cd7f9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44902c640 1 -- 172.21.15.98:0/4137052916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621cd86acd0 con 0x5621cd7ea280 2024-05-09T08:57:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44982d640 1 --2- 172.21.15.98:0/4137052916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7eaba0 0x5621cd7f9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe44902c640 1 --2- 172.21.15.98:0/4137052916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7f95b0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fe4340059f0 tx=0x7fe43400b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a247e5af3353a53c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.187+0000 7fe42d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe42d7fa640 1 -- 172.21.15.98:0/4137052916 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43402b070 con 0x5621cd7ea280 2024-05-09T08:57:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe42d7fa640 1 -- 172.21.15.98:0/4137052916 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fe434005030 con 0x5621cd7ea280 2024-05-09T08:57:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe44e795740 1 -- 172.21.15.98:0/4137052916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe440005520 con 0x5621cd7ea280 2024-05-09T08:57:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe42d7fa640 1 -- 172.21.15.98:0/4137052916 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe434011e50 con 0x5621cd7ea280 2024-05-09T08:57:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe44e795740 10 client.17376 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe44e795740 1 -- 172.21.15.98:0/4137052916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5621cd86bf50 con 0x5621cd7ea280 2024-05-09T08:57:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe42d7fa640 1 -- 172.21.15.98:0/4137052916 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe440005520 con 0x5621cd7ea280 2024-05-09T08:57:39.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe42d7fa640 1 -- 172.21.15.98:0/4137052916 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe43401a400 con 0x5621cd7ea280 2024-05-09T08:57:39.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe44e795740 10 client.17376 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:39.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe44e795740 10 client.17376 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:39.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe44e795740 20 client.17376 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:39.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe44e795740 1 --2- 172.21.15.98:0/4137052916 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5621cd86c600 0x5621cd86ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe44e795740 4 client.17376 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:57:39.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe44e795740 1 -- 172.21.15.98:0/4137052916 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5621cd757780 con 0x5621cd86c600 2024-05-09T08:57:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.188+0000 7fe44982d640 1 --2- 172.21.15.98:0/4137052916 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5621cd86c600 0x5621cd86ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe42d7fa640 10 client.17376 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe44982d640 1 --2- 172.21.15.98:0/4137052916 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5621cd86c600 0x5621cd86ea20 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a02e2db5eed3877 server_cookie=63685b832301736 in_seq=0 out_seq=0 2024-05-09T08:57:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe42d7fa640 1 -- 172.21.15.98:0/4137052916 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5621cd757780 con 0x5621cd86c600 2024-05-09T08:57:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe42d7fa640 10 client.17376 handle_command_reply: tid=0 2024-05-09T08:57:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe44e795740 1 client.17376 shutdown 2024-05-09T08:57:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe44e795740 1 -- 172.21.15.98:0/4137052916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 msgr2=0x5621cd7f95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe44e795740 1 --2- 172.21.15.98:0/4137052916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7f95b0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fe4340059f0 tx=0x7fe43400b730 comp rx=0 tx=0).stop 2024-05-09T08:57:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe44e795740 1 -- 172.21.15.98:0/4137052916 shutdown_connections 2024-05-09T08:57:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe44e795740 1 --2- 172.21.15.98:0/4137052916 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5621cd86c600 0x5621cd86ea20 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe44e795740 1 --2- 172.21.15.98:0/4137052916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5621cd7f5270 0x5621cd86a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe44e795740 1 --2- 172.21.15.98:0/4137052916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621cd7eaba0 0x5621cd7f9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe44e795740 1 --2- 172.21.15.98:0/4137052916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621cd7ea280 0x5621cd7f95b0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe42d7fa640 0 client.17376 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:39.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe44e795740 1 -- 172.21.15.98:0/4137052916 >> 172.21.15.98:0/4137052916 conn(0x5621cd7e65a0 msgr2=0x5621cd7f21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe44e795740 1 -- 172.21.15.98:0/4137052916 shutdown_connections 2024-05-09T08:57:39.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe44e795740 1 -- 172.21.15.98:0/4137052916 wait complete. 2024-05-09T08:57:39.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.189+0000 7fe44e795740 20 client.17376 trim_cache size 0 max 16384 2024-05-09T08:57:39.198 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:57:39.199 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:57:39.199 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:57:39.199 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:57:39.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.190+0000 7fe44ba91640 1 -- 172.21.15.98:0/756498557 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe4300a2d80 msgr2=0x7fe4300a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.190+0000 7fe44ba91640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe4300a2d80 0x7fe4300a5240 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe43c0cd1f0 tx=0x7fe43c06d980 comp rx=0 tx=0).stop 2024-05-09T08:57:39.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.190+0000 7fe44ba91640 1 -- 172.21.15.98:0/756498557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe444107820 msgr2=0x7fe44419ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.190+0000 7fe44ba91640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe444107820 0x7fe44419ec50 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00cad0 tx=0x7fe43c00cf90 comp rx=0 tx=0).stop 2024-05-09T08:57:39.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.190+0000 7fe44ba91640 1 -- 172.21.15.98:0/756498557 shutdown_connections 2024-05-09T08:57:39.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.190+0000 7fe44ba91640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe4300a2d80 0x7fe4300a5240 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.190+0000 7fe44ba91640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe444107820 0x7fe44419ec50 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.190+0000 7fe44ba91640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe44406a870 0x7fe44419a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.190+0000 7fe44ba91640 1 --2- 172.21.15.98:0/756498557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe444069eb0 0x7fe44419a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.190+0000 7fe44ba91640 1 -- 172.21.15.98:0/756498557 >> 172.21.15.98:0/756498557 conn(0x7fe444077790 msgr2=0x7fe44410cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.190+0000 7fe44ba91640 1 -- 172.21.15.98:0/756498557 shutdown_connections 2024-05-09T08:57:39.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.190+0000 7fe44ba91640 1 -- 172.21.15.98:0/756498557 wait complete. 2024-05-09T08:57:39.202 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:57:39.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:57:39.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.304+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/1270372825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c58101800 msgr2=0x7f0c58101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.304+0000 7f0c5dd1b640 1 --2- 172.21.15.98:0/1270372825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c58101800 0x7f0c58101c80 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009930 tx=0x7f0c4c02a420 comp rx=0 tx=0).stop 2024-05-09T08:57:39.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.304+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/1270372825 shutdown_connections 2024-05-09T08:57:39.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.304+0000 7f0c5dd1b640 1 --2- 172.21.15.98:0/1270372825 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c581021c0 0x7f0c5810e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.304+0000 7f0c5dd1b640 1 --2- 172.21.15.98:0/1270372825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c58101800 0x7f0c58101c80 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.304+0000 7f0c5dd1b640 1 --2- 172.21.15.98:0/1270372825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c58107810 0x7f0c58107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.304+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/1270372825 >> 172.21.15.98:0/1270372825 conn(0x7f0c58078bf0 msgr2=0x7f0c58079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.304+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/1270372825 shutdown_connections 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.304+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/1270372825 wait complete. 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c5dd1b640 1 Processor -- start 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c5dd1b640 1 -- start start 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c5dd1b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c58101800 0x7f0c581980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c5dd1b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c581021c0 0x7f0c58198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c5dd1b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c58107810 0x7f0c5819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c5dd1b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c58111c20 con 0x7f0c58107810 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c5dd1b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0c58111aa0 con 0x7f0c581021c0 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c5dd1b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0c58111da0 con 0x7f0c58101800 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c57fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c58107810 0x7f0c5819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c57fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c58107810 0x7f0c5819c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55258/0 (socket says 172.21.15.98:55258) 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c57fff640 1 -- 172.21.15.98:0/399187889 learned_addr learned my addr 172.21.15.98:0/399187889 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c577fe640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c58101800 0x7f0c581980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c56ffd640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c581021c0 0x7f0c58198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c57fff640 1 -- 172.21.15.98:0/399187889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c58101800 msgr2=0x7f0c581980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c57fff640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c58101800 0x7f0c581980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c57fff640 1 -- 172.21.15.98:0/399187889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c581021c0 msgr2=0x7f0c58198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c57fff640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c581021c0 0x7f0c58198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c57fff640 1 -- 172.21.15.98:0/399187889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5819d0b0 con 0x7f0c58107810 2024-05-09T08:57:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.305+0000 7f0c56ffd640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c581021c0 0x7f0c58198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.306+0000 7f0c57fff640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c58107810 0x7f0c5819c980 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000cce0 tx=0x7f0c5000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.306+0000 7f0c54ff9640 1 -- 172.21.15.98:0/399187889 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5000f450 con 0x7f0c58107810 2024-05-09T08:57:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.306+0000 7f0c54ff9640 1 -- 172.21.15.98:0/399187889 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c5000fad0 con 0x7f0c58107810 2024-05-09T08:57:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.306+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/399187889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c5819d3a0 con 0x7f0c58107810 2024-05-09T08:57:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.306+0000 7f0c54ff9640 1 -- 172.21.15.98:0/399187889 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c50018c10 con 0x7f0c58107810 2024-05-09T08:57:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.306+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/399187889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c581a4c30 con 0x7f0c58107810 2024-05-09T08:57:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.307+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/399187889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c58068aa0 con 0x7f0c58107810 2024-05-09T08:57:39.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.308+0000 7f0c54ff9640 1 -- 172.21.15.98:0/399187889 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0c5000f5f0 con 0x7f0c58107810 2024-05-09T08:57:39.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.309+0000 7f0c54ff9640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0c400a2d00 0x7f0c400a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.309+0000 7f0c577fe640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0c400a2d00 0x7f0c400a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.309+0000 7f0c54ff9640 1 -- 172.21.15.98:0/399187889 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f0c500cff50 con 0x7f0c58107810 2024-05-09T08:57:39.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.309+0000 7f0c577fe640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0c400a2d00 0x7f0c400a51c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0c480058c0 tx=0x7f0c4800a2d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.313+0000 7f0c54ff9640 1 -- 172.21.15.98:0/399187889 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0c50014030 con 0x7f0c58107810 2024-05-09T08:57:39.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.526+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/399187889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c58109da0 con 0x7f0c58107810 2024-05-09T08:57:39.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.526+0000 7f0c54ff9640 1 -- 172.21.15.98:0/399187889 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11941 (secure 0 0 0) 0x7f0c5009c760 con 0x7f0c58107810 2024-05-09T08:57:39.526 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:57:39.526 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":26581,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/561027115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":561027115},{"type":"v1","addr":"172.21.15.105:6835","nonce":561027115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:56:52.891273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16641":{"gid":16641,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T08:57:39.526 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 37 2024-05-09T08:57:39.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.528+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/399187889 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0c400a2d00 msgr2=0x7f0c400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.528+0000 7f0c5dd1b640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0c400a2d00 0x7f0c400a51c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0c480058c0 tx=0x7f0c4800a2d0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.528+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/399187889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c58107810 msgr2=0x7f0c5819c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.528+0000 7f0c5dd1b640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c58107810 0x7f0c5819c980 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000cce0 tx=0x7f0c5000a620 comp rx=0 tx=0).stop 2024-05-09T08:57:39.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.528+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/399187889 shutdown_connections 2024-05-09T08:57:39.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.528+0000 7f0c5dd1b640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0c400a2d00 0x7f0c400a51c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.528+0000 7f0c5dd1b640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c58107810 0x7f0c5819c980 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.528+0000 7f0c5dd1b640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c581021c0 0x7f0c58198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.528+0000 7f0c5dd1b640 1 --2- 172.21.15.98:0/399187889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c58101800 0x7f0c581980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.528+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/399187889 >> 172.21.15.98:0/399187889 conn(0x7f0c58078bf0 msgr2=0x7f0c581041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.529+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/399187889 shutdown_connections 2024-05-09T08:57:39.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.529+0000 7f0c5dd1b640 1 -- 172.21.15.98:0/399187889 wait complete. 2024-05-09T08:57:39.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:57:39.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.644+0000 7f5bd8d5a640 1 -- 172.21.15.98:0/3792303787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd4101860 msgr2=0x7f5bd410d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.644+0000 7f5bd8d5a640 1 --2- 172.21.15.98:0/3792303787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd4101860 0x7f5bd410d570 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009ca0 tx=0x7f5bcc02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.644+0000 7f5bd8d5a640 1 -- 172.21.15.98:0/3792303787 shutdown_connections 2024-05-09T08:57:39.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.644+0000 7f5bd8d5a640 1 --2- 172.21.15.98:0/3792303787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd4101860 0x7f5bd410d570 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.644+0000 7f5bd8d5a640 1 --2- 172.21.15.98:0/3792303787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bd4100ea0 0x7f5bd4101320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.644+0000 7f5bd8d5a640 1 --2- 172.21.15.98:0/3792303787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bd41056e0 0x7f5bd4105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.644+0000 7f5bd8d5a640 1 -- 172.21.15.98:0/3792303787 >> 172.21.15.98:0/3792303787 conn(0x7f5bd4076cf0 msgr2=0x7f5bd40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.644+0000 7f5bd8d5a640 1 -- 172.21.15.98:0/3792303787 shutdown_connections 2024-05-09T08:57:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.644+0000 7f5bd8d5a640 1 -- 172.21.15.98:0/3792303787 wait complete. 2024-05-09T08:57:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd8d5a640 1 Processor -- start 2024-05-09T08:57:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd8d5a640 1 -- start start 2024-05-09T08:57:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd8d5a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bd4100ea0 0x7f5bd410e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd8d5a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd4101860 0x7f5bd410e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd8d5a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bd41056e0 0x7f5bd410ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd8d5a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd4111c20 con 0x7f5bd4101860 2024-05-09T08:57:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd8d5a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd4111aa0 con 0x7f5bd4100ea0 2024-05-09T08:57:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd8d5a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd4111da0 con 0x7f5bd41056e0 2024-05-09T08:57:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd4101860 0x7f5bd410e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd4101860 0x7f5bd410e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55274/0 (socket says 172.21.15.98:55274) 2024-05-09T08:57:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd1d9b640 1 -- 172.21.15.98:0/1750539286 learned_addr learned my addr 172.21.15.98:0/1750539286 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/1750539286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bd41056e0 0x7f5bd410ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd259c640 1 --2- 172.21.15.98:0/1750539286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bd4100ea0 0x7f5bd410e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd1d9b640 1 -- 172.21.15.98:0/1750539286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bd41056e0 msgr2=0x7f5bd410ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd1d9b640 1 --2- 172.21.15.98:0/1750539286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bd41056e0 0x7f5bd410ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd1d9b640 1 -- 172.21.15.98:0/1750539286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bd4100ea0 msgr2=0x7f5bd410e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd1d9b640 1 --2- 172.21.15.98:0/1750539286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bd4100ea0 0x7f5bd410e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.645+0000 7f5bd1d9b640 1 -- 172.21.15.98:0/1750539286 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd41a6620 con 0x7f5bd4101860 2024-05-09T08:57:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.646+0000 7f5bd1d9b640 1 --2- 172.21.15.98:0/1750539286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd4101860 0x7f5bd410e540 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5bc400b4a0 tx=0x7f5bc400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.646+0000 7f5bcb7fe640 1 -- 172.21.15.98:0/1750539286 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4014070 con 0x7f5bd4101860 2024-05-09T08:57:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.646+0000 7f5bcb7fe640 1 -- 172.21.15.98:0/1750539286 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc4010af0 con 0x7f5bd4101860 2024-05-09T08:57:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.646+0000 7f5bd8d5a640 1 -- 172.21.15.98:0/1750539286 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd41a6910 con 0x7f5bd4101860 2024-05-09T08:57:39.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.646+0000 7f5bcb7fe640 1 -- 172.21.15.98:0/1750539286 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4017d10 con 0x7f5bd4101860 2024-05-09T08:57:39.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.646+0000 7f5bd8d5a640 1 -- 172.21.15.98:0/1750539286 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd41a6eb0 con 0x7f5bd4101860 2024-05-09T08:57:39.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.648+0000 7f5bcb7fe640 1 -- 172.21.15.98:0/1750539286 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5bc4017070 con 0x7f5bd4101860 2024-05-09T08:57:39.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.648+0000 7f5bdba67740 1 Processor -- start 2024-05-09T08:57:39.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bdba67740 1 -- start start 2024-05-09T08:57:39.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bdba67740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 0x563e4c979710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bdba67740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c979c50 0x563e4c983de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bdba67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c984320 0x563e4c986710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bcb7fe640 1 --2- 172.21.15.98:0/1750539286 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5bc00a2d80 0x7f5bc00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bdba67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563e4c8dec40 con 0x563e4c984320 2024-05-09T08:57:39.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bdba67740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563e4c886580 con 0x563e4c979c50 2024-05-09T08:57:39.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bdba67740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563e4c8dac00 con 0x563e4c979330 2024-05-09T08:57:39.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bd1d9b640 1 --2- 172.21.15.98:0/1750539286 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5bc00a2d80 0x7f5bc00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bd2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c984320 0x563e4c986710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bd2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c984320 0x563e4c986710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55276/0 (socket says 172.21.15.98:55276) 2024-05-09T08:57:39.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bd2d9d640 1 -- 172.21.15.98:0/1654177650 learned_addr learned my addr 172.21.15.98:0/1654177650 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/1654177650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 0x563e4c979710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bd259c640 1 --2- 172.21.15.98:0/1654177650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c979c50 0x563e4c983de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bcb7fe640 1 -- 172.21.15.98:0/1750539286 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f5bc40ce9f0 con 0x7f5bd4101860 2024-05-09T08:57:39.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bd2d9d640 1 -- 172.21.15.98:0/1654177650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 msgr2=0x563e4c979710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/1654177650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 0x563e4c979710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bd2d9d640 1 -- 172.21.15.98:0/1654177650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c979c50 msgr2=0x563e4c983de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/1654177650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c979c50 0x563e4c983de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bd2d9d640 1 -- 172.21.15.98:0/1654177650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e4c5f23f0 con 0x563e4c984320 2024-05-09T08:57:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bd259c640 1 --2- 172.21.15.98:0/1654177650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c979c50 0x563e4c983de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:57:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.649+0000 7f5bd1d9b640 1 --2- 172.21.15.98:0/1750539286 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5bc00a2d80 0x7f5bc00a5240 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4065d00 tx=0x7f5bc406f250 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.650+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/1654177650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c984320 0x563e4c986710 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009ca0 tx=0x7f5bcc0093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd2ce74de98f06f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.650+0000 7f5bc8ff9640 1 -- 172.21.15.98:0/1654177650 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc003970 con 0x563e4c984320 2024-05-09T08:57:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.650+0000 7f5bc8ff9640 1 -- 172.21.15.98:0/1654177650 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bcc036070 con 0x563e4c984320 2024-05-09T08:57:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.650+0000 7f5bc8ff9640 1 -- 172.21.15.98:0/1654177650 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc004350 con 0x563e4c984320 2024-05-09T08:57:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.650+0000 7f5bdba67740 1 -- 172.21.15.98:0/1654177650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c984320 msgr2=0x563e4c986710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.650+0000 7f5bdba67740 1 --2- 172.21.15.98:0/1654177650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c984320 0x563e4c986710 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009ca0 tx=0x7f5bcc0093d0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.650+0000 7f5bdba67740 1 -- 172.21.15.98:0/1654177650 shutdown_connections 2024-05-09T08:57:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.650+0000 7f5bdba67740 1 --2- 172.21.15.98:0/1654177650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c984320 0x563e4c986710 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.650+0000 7f5bdba67740 1 --2- 172.21.15.98:0/1654177650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c979c50 0x563e4c983de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.650+0000 7f5bdba67740 1 --2- 172.21.15.98:0/1654177650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 0x563e4c979710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.650+0000 7f5bdba67740 1 -- 172.21.15.98:0/1654177650 >> 172.21.15.98:0/1654177650 conn(0x563e4c975650 msgr2=0x563e4c975a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bdba67740 1 -- 172.21.15.98:0/1654177650 shutdown_connections 2024-05-09T08:57:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bdba67740 1 -- 172.21.15.98:0/1654177650 wait complete. 2024-05-09T08:57:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bdba67740 1 -- start start 2024-05-09T08:57:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bdba67740 1 Processor -- start 2024-05-09T08:57:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bdba67740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 0x563e4c983dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bdba67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c9811d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bdba67740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c984320 0x563e4c981910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bdba67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563e4c986f30 con 0x563e4c979c50 2024-05-09T08:57:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bdba67740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563e4c899c40 con 0x563e4c984320 2024-05-09T08:57:39.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bdba67740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563e4c9870b0 con 0x563e4c979330 2024-05-09T08:57:39.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bdba67740 10 client.0 _ll_register_callbacks cb 0x563e4c5f1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:39.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bd259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c9811d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bd259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c9811d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55282/0 (socket says 172.21.15.98:55282) 2024-05-09T08:57:39.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bd259c640 1 -- 172.21.15.98:0/1470730517 learned_addr learned my addr 172.21.15.98:0/1470730517 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.651+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/1470730517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c984320 0x563e4c981910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/1470730517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 0x563e4c983dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bd259c640 1 -- 172.21.15.98:0/1470730517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 msgr2=0x563e4c983dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bd259c640 1 --2- 172.21.15.98:0/1470730517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 0x563e4c983dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bd259c640 1 -- 172.21.15.98:0/1470730517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c984320 msgr2=0x563e4c981910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bd259c640 1 --2- 172.21.15.98:0/1470730517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c984320 0x563e4c981910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bd259c640 1 -- 172.21.15.98:0/1470730517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e4c6bc7d0 con 0x563e4c979c50 2024-05-09T08:57:39.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/1470730517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 0x563e4c983dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bd259c640 1 --2- 172.21.15.98:0/1470730517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c9811d0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc00baa0 tx=0x7f5bbc00bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe046a372fa4f13d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bb9ffb640 1 -- 172.21.15.98:0/1470730517 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc003df0 con 0x563e4c979c50 2024-05-09T08:57:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bb9ffb640 1 -- 172.21.15.98:0/1470730517 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f5bbc011440 con 0x563e4c979c50 2024-05-09T08:57:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bdba67740 1 -- 172.21.15.98:0/1470730517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bcc036070 con 0x563e4c979c50 2024-05-09T08:57:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bb9ffb640 1 -- 172.21.15.98:0/1470730517 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5bbc011640 con 0x563e4c979c50 2024-05-09T08:57:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bdba67740 10 client.17394 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bdba67740 1 -- 172.21.15.98:0/1470730517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563e4c9930e0 con 0x563e4c979c50 2024-05-09T08:57:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bb9ffb640 1 -- 172.21.15.98:0/1470730517 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bcc036070 con 0x563e4c979c50 2024-05-09T08:57:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bb9ffb640 1 -- 172.21.15.98:0/1470730517 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f5bbc01b930 con 0x563e4c979c50 2024-05-09T08:57:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bdba67740 10 client.17394 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bdba67740 10 client.17394 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:57:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bdba67740 20 client.17394 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bdba67740 1 --2- 172.21.15.98:0/1470730517 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563e4c994640 0x563e4c996a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bdba67740 4 client.17394 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:57:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.652+0000 7f5bdba67740 1 -- 172.21.15.98:0/1470730517 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e4c8ec370 con 0x563e4c994640 2024-05-09T08:57:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.653+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/1470730517 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563e4c994640 0x563e4c996a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.653+0000 7f5bb9ffb640 10 client.17394 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:57:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.653+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/1470730517 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563e4c994640 0x563e4c996a60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8f2671958d3eb418 server_cookie=596d2e48b8401e6e in_seq=0 out_seq=0 2024-05-09T08:57:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bb9ffb640 1 -- 172.21.15.98:0/1470730517 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563e4c8ec370 con 0x563e4c994640 2024-05-09T08:57:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bb9ffb640 10 client.17394 handle_command_reply: tid=0 2024-05-09T08:57:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bdba67740 1 client.17394 shutdown 2024-05-09T08:57:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bdba67740 1 -- 172.21.15.98:0/1470730517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 msgr2=0x563e4c9811d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bdba67740 1 --2- 172.21.15.98:0/1470730517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c9811d0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc00baa0 tx=0x7f5bbc00bf60 comp rx=0 tx=0).stop 2024-05-09T08:57:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bdba67740 1 -- 172.21.15.98:0/1470730517 shutdown_connections 2024-05-09T08:57:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bdba67740 1 --2- 172.21.15.98:0/1470730517 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563e4c994640 0x563e4c996a60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bdba67740 1 --2- 172.21.15.98:0/1470730517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c984320 0x563e4c981910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bdba67740 1 --2- 172.21.15.98:0/1470730517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c9811d0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bdba67740 1 --2- 172.21.15.98:0/1470730517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 0x563e4c983dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bdba67740 1 -- 172.21.15.98:0/1470730517 >> 172.21.15.98:0/1470730517 conn(0x563e4c975650 msgr2=0x563e4c97b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bb9ffb640 0 client.17394 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:57:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bdba67740 1 -- 172.21.15.98:0/1470730517 shutdown_connections 2024-05-09T08:57:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bdba67740 1 -- 172.21.15.98:0/1470730517 wait complete. 2024-05-09T08:57:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.654+0000 7f5bdba67740 20 client.17394 trim_cache size 0 max 16384 2024-05-09T08:57:39.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.667+0000 7f5bdba67740 1 Processor -- start 2024-05-09T08:57:39.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.667+0000 7f5bdba67740 1 -- start start 2024-05-09T08:57:39.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.667+0000 7f5bdba67740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 0x563e4c98fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.667+0000 7f5bdba67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c99c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.667+0000 7f5bdba67740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c984320 0x563e4c99c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.667+0000 7f5bdba67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41b11a0 con 0x563e4c979c50 2024-05-09T08:57:39.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.667+0000 7f5bdba67740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41b2ce0 con 0x563e4c984320 2024-05-09T08:57:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.667+0000 7f5bdba67740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41b1870 con 0x563e4c979330 2024-05-09T08:57:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.668+0000 7f5bd259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c99c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.668+0000 7f5bd259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c99c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55294/0 (socket says 172.21.15.98:55294) 2024-05-09T08:57:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.668+0000 7f5bd259c640 1 -- 172.21.15.98:0/1821483282 learned_addr learned my addr 172.21.15.98:0/1821483282 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.668+0000 7f5bd259c640 1 -- 172.21.15.98:0/1821483282 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 msgr2=0x563e4c98fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.668+0000 7f5bd259c640 1 --2- 172.21.15.98:0/1821483282 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 0x563e4c98fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.668+0000 7f5bd259c640 1 -- 172.21.15.98:0/1821483282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c984320 msgr2=0x563e4c99c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.668+0000 7f5bd259c640 1 --2- 172.21.15.98:0/1821483282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c984320 0x563e4c99c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.668+0000 7f5bd259c640 1 -- 172.21.15.98:0/1821483282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e4c981c80 con 0x563e4c979c50 2024-05-09T08:57:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.668+0000 7f5bd259c640 1 --2- 172.21.15.98:0/1821483282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c99c310 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc01be20 tx=0x7f5bbc0115d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63200c9f8dbaf287 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.668+0000 7f5bbbfff640 1 -- 172.21.15.98:0/1821483282 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc005850 con 0x563e4c979c50 2024-05-09T08:57:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.668+0000 7f5bbbfff640 1 -- 172.21.15.98:0/1821483282 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bbc00b400 con 0x563e4c979c50 2024-05-09T08:57:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.668+0000 7f5bbbfff640 1 -- 172.21.15.98:0/1821483282 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc010da0 con 0x563e4c979c50 2024-05-09T08:57:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 -- 172.21.15.98:0/1821483282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 msgr2=0x563e4c99c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 --2- 172.21.15.98:0/1821483282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c99c310 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc01be20 tx=0x7f5bbc0115d0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 -- 172.21.15.98:0/1821483282 shutdown_connections 2024-05-09T08:57:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 --2- 172.21.15.98:0/1821483282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c984320 0x563e4c99c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 --2- 172.21.15.98:0/1821483282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c99c310 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 --2- 172.21.15.98:0/1821483282 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c979330 0x563e4c98fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 -- 172.21.15.98:0/1821483282 >> 172.21.15.98:0/1821483282 conn(0x563e4c975650 msgr2=0x563e4c975a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 -- 172.21.15.98:0/1821483282 shutdown_connections 2024-05-09T08:57:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 -- 172.21.15.98:0/1821483282 wait complete. 2024-05-09T08:57:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 -- start start 2024-05-09T08:57:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 Processor -- start 2024-05-09T08:57:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c979330 0x563e4c98c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c97da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c984320 0x563e4c97df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41b2610 con 0x563e4c979c50 2024-05-09T08:57:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41b1f40 con 0x563e4c979330 2024-05-09T08:57:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41af1a0 con 0x563e4c984320 2024-05-09T08:57:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bdba67740 10 client.0 _ll_register_callbacks cb 0x7f5bbc004420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bd259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c97da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bd259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c97da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55300/0 (socket says 172.21.15.98:55300) 2024-05-09T08:57:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bd259c640 1 -- 172.21.15.98:0/4083785045 learned_addr learned my addr 172.21.15.98:0/4083785045 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/4083785045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c979330 0x563e4c98c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.669+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/4083785045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c984320 0x563e4c97df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bd259c640 1 -- 172.21.15.98:0/4083785045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c984320 msgr2=0x563e4c97df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bd259c640 1 --2- 172.21.15.98:0/4083785045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c984320 0x563e4c97df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bd259c640 1 -- 172.21.15.98:0/4083785045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c979330 msgr2=0x563e4c98c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bd259c640 1 --2- 172.21.15.98:0/4083785045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c979330 0x563e4c98c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bd259c640 1 -- 172.21.15.98:0/4083785045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e4c97e4c0 con 0x563e4c979c50 2024-05-09T08:57:39.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/4083785045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c984320 0x563e4c97df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:39.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bd259c640 1 --2- 172.21.15.98:0/4083785045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c97da40 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc010c90 tx=0x7f5bbc010cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54c243b1e901555 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:39.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bb9ffb640 1 -- 172.21.15.98:0/4083785045 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc037070 con 0x563e4c979c50 2024-05-09T08:57:39.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bb9ffb640 1 -- 172.21.15.98:0/4083785045 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f5bbc005250 con 0x563e4c979c50 2024-05-09T08:57:39.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bdba67740 1 -- 172.21.15.98:0/4083785045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bbc00b400 con 0x563e4c979c50 2024-05-09T08:57:39.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bb9ffb640 1 -- 172.21.15.98:0/4083785045 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5bbc043050 con 0x563e4c979c50 2024-05-09T08:57:39.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bdba67740 10 client.17400 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bdba67740 1 -- 172.21.15.98:0/4083785045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563e4c9fa8d0 con 0x563e4c979c50 2024-05-09T08:57:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bb9ffb640 1 -- 172.21.15.98:0/4083785045 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bbc00b400 con 0x563e4c979c50 2024-05-09T08:57:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bb9ffb640 1 -- 172.21.15.98:0/4083785045 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f5bbc01b420 con 0x563e4c979c50 2024-05-09T08:57:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bdba67740 10 client.17400 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bdba67740 10 client.17400 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:57:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bdba67740 20 client.17400 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bdba67740 1 --2- 172.21.15.98:0/4083785045 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563e4c9faeb0 0x563e4c9fd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bdba67740 4 client.17400 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:57:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.670+0000 7f5bdba67740 1 -- 172.21.15.98:0/4083785045 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563e4c8e66c0 con 0x563e4c9faeb0 2024-05-09T08:57:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.671+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/4083785045 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563e4c9faeb0 0x563e4c9fd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.671+0000 7f5bb9ffb640 10 client.17400 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:57:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.671+0000 7f5bd2d9d640 1 --2- 172.21.15.98:0/4083785045 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563e4c9faeb0 0x563e4c9fd2d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=97b31cd7818b03f1 server_cookie=a3d5d72148e8c7c8 in_seq=0 out_seq=0 2024-05-09T08:57:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.671+0000 7f5bb9ffb640 1 -- 172.21.15.98:0/4083785045 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563e4c8e66c0 con 0x563e4c9faeb0 2024-05-09T08:57:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.671+0000 7f5bb9ffb640 10 client.17400 handle_command_reply: tid=0 2024-05-09T08:57:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.671+0000 7f5bdba67740 1 client.17400 shutdown 2024-05-09T08:57:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bdba67740 1 -- 172.21.15.98:0/4083785045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 msgr2=0x563e4c97da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bdba67740 1 --2- 172.21.15.98:0/4083785045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c97da40 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc010c90 tx=0x7f5bbc010cc0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bdba67740 1 -- 172.21.15.98:0/4083785045 shutdown_connections 2024-05-09T08:57:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bdba67740 1 --2- 172.21.15.98:0/4083785045 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563e4c9faeb0 0x563e4c9fd2d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bdba67740 1 --2- 172.21.15.98:0/4083785045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563e4c984320 0x563e4c97df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bdba67740 1 --2- 172.21.15.98:0/4083785045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e4c979c50 0x563e4c97da40 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bb9ffb640 0 client.17400 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:57:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bdba67740 1 --2- 172.21.15.98:0/4083785045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563e4c979330 0x563e4c98c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bdba67740 1 -- 172.21.15.98:0/4083785045 >> 172.21.15.98:0/4083785045 conn(0x563e4c975650 msgr2=0x563e4c982880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bdba67740 1 -- 172.21.15.98:0/4083785045 shutdown_connections 2024-05-09T08:57:39.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bdba67740 1 -- 172.21.15.98:0/4083785045 wait complete. 2024-05-09T08:57:39.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bdba67740 20 client.17400 trim_cache size 0 max 16384 2024-05-09T08:57:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:57:39.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bd8d5a640 1 -- 172.21.15.98:0/1750539286 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5bc00a2d80 msgr2=0x7f5bc00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bd8d5a640 1 --2- 172.21.15.98:0/1750539286 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5bc00a2d80 0x7f5bc00a5240 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4065d00 tx=0x7f5bc406f250 comp rx=0 tx=0).stop 2024-05-09T08:57:39.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bd8d5a640 1 -- 172.21.15.98:0/1750539286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd4101860 msgr2=0x7f5bd410e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bd8d5a640 1 --2- 172.21.15.98:0/1750539286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd4101860 0x7f5bd410e540 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5bc400b4a0 tx=0x7f5bc400b960 comp rx=0 tx=0).stop 2024-05-09T08:57:39.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bd8d5a640 1 -- 172.21.15.98:0/1750539286 shutdown_connections 2024-05-09T08:57:39.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bd8d5a640 1 --2- 172.21.15.98:0/1750539286 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5bc00a2d80 0x7f5bc00a5240 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bd8d5a640 1 --2- 172.21.15.98:0/1750539286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bd41056e0 0x7f5bd410ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bd8d5a640 1 --2- 172.21.15.98:0/1750539286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd4101860 0x7f5bd410e540 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bd8d5a640 1 --2- 172.21.15.98:0/1750539286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bd4100ea0 0x7f5bd410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bd8d5a640 1 -- 172.21.15.98:0/1750539286 >> 172.21.15.98:0/1750539286 conn(0x7f5bd4076cf0 msgr2=0x7f5bd41033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bd8d5a640 1 -- 172.21.15.98:0/1750539286 shutdown_connections 2024-05-09T08:57:39.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.672+0000 7f5bd8d5a640 1 -- 172.21.15.98:0/1750539286 wait complete. 2024-05-09T08:57:39.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:57:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.786+0000 7fa9d3f29640 1 -- 172.21.15.98:0/2127260303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9cc101800 msgr2=0x7fa9cc101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.786+0000 7fa9d3f29640 1 --2- 172.21.15.98:0/2127260303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9cc101800 0x7fa9cc101c80 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc009930 tx=0x7fa9bc02a420 comp rx=0 tx=0).stop 2024-05-09T08:57:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.786+0000 7fa9d3f29640 1 -- 172.21.15.98:0/2127260303 shutdown_connections 2024-05-09T08:57:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.786+0000 7fa9d3f29640 1 --2- 172.21.15.98:0/2127260303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9cc1021c0 0x7fa9cc10e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.786+0000 7fa9d3f29640 1 --2- 172.21.15.98:0/2127260303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9cc101800 0x7fa9cc101c80 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.786+0000 7fa9d3f29640 1 --2- 172.21.15.98:0/2127260303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9cc107810 0x7fa9cc107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.786+0000 7fa9d3f29640 1 -- 172.21.15.98:0/2127260303 >> 172.21.15.98:0/2127260303 conn(0x7fa9cc078c80 msgr2=0x7fa9cc079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.786+0000 7fa9d3f29640 1 -- 172.21.15.98:0/2127260303 shutdown_connections 2024-05-09T08:57:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.786+0000 7fa9d3f29640 1 -- 172.21.15.98:0/2127260303 wait complete. 2024-05-09T08:57:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d3f29640 1 Processor -- start 2024-05-09T08:57:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d3f29640 1 -- start start 2024-05-09T08:57:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d3f29640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9cc101800 0x7fa9cc1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d3f29640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9cc1021c0 0x7fa9cc1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d3f29640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9cc107810 0x7fa9cc19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d3f29640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc111c20 con 0x7fa9cc1021c0 2024-05-09T08:57:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d3f29640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc111aa0 con 0x7fa9cc101800 2024-05-09T08:57:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d3f29640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc111da0 con 0x7fa9cc107810 2024-05-09T08:57:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d14c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9cc1021c0 0x7fa9cc1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d14c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9cc1021c0 0x7fa9cc1985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55312/0 (socket says 172.21.15.98:55312) 2024-05-09T08:57:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d14c4640 1 -- 172.21.15.98:0/134313298 learned_addr learned my addr 172.21.15.98:0/134313298 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d24c6640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9cc107810 0x7fa9cc19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9cc101800 0x7fa9cc1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d14c4640 1 -- 172.21.15.98:0/134313298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9cc107810 msgr2=0x7fa9cc19c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d14c4640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9cc107810 0x7fa9cc19c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d14c4640 1 -- 172.21.15.98:0/134313298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9cc101800 msgr2=0x7fa9cc1980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d14c4640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9cc101800 0x7fa9cc1980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d14c4640 1 -- 172.21.15.98:0/134313298 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9cc19d040 con 0x7fa9cc1021c0 2024-05-09T08:57:39.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9cc101800 0x7fa9cc1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:39.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.787+0000 7fa9d14c4640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9cc1021c0 0x7fa9cc1985e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc009a50 tx=0x7fa9bc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.788+0000 7fa9c2ffd640 1 -- 172.21.15.98:0/134313298 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc03f030 con 0x7fa9cc1021c0 2024-05-09T08:57:39.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.788+0000 7fa9c2ffd640 1 -- 172.21.15.98:0/134313298 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9bc003ce0 con 0x7fa9cc1021c0 2024-05-09T08:57:39.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.788+0000 7fa9d3f29640 1 -- 172.21.15.98:0/134313298 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9cc19d2d0 con 0x7fa9cc1021c0 2024-05-09T08:57:39.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.788+0000 7fa9c2ffd640 1 -- 172.21.15.98:0/134313298 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc035070 con 0x7fa9cc1021c0 2024-05-09T08:57:39.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.788+0000 7fa9d3f29640 1 -- 172.21.15.98:0/134313298 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9cc1a4bc0 con 0x7fa9cc1021c0 2024-05-09T08:57:39.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.790+0000 7fa9c2ffd640 1 -- 172.21.15.98:0/134313298 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa9bc042030 con 0x7fa9cc1021c0 2024-05-09T08:57:39.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.790+0000 7fa9d6c3a740 1 Processor -- start 2024-05-09T08:57:39.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.790+0000 7fa9d6c3a740 1 -- start start 2024-05-09T08:57:39.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d6c3a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399210 0x56388f3995f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d6c3a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399b30 0x56388f3a81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d6c3a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f3a8730 0x56388f3aab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d6c3a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56388f2fec40 con 0x56388f399b30 2024-05-09T08:57:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d6c3a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56388f2a6580 con 0x56388f3a8730 2024-05-09T08:57:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d6c3a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56388f2fac00 con 0x56388f399210 2024-05-09T08:57:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d1cc5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399b30 0x56388f3a81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d1cc5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399b30 0x56388f3a81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55314/0 (socket says 172.21.15.98:55314) 2024-05-09T08:57:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d1cc5640 1 -- 172.21.15.98:0/3097454545 learned_addr learned my addr 172.21.15.98:0/3097454545 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d24c6640 1 --2- 172.21.15.98:0/3097454545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399210 0x56388f3995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9c2ffd640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa9b80a2d80 0x7fa9b80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d1cc5640 1 -- 172.21.15.98:0/3097454545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399210 msgr2=0x56388f3995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/3097454545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399210 0x56388f3995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d14c4640 1 --2- 172.21.15.98:0/3097454545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f3a8730 0x56388f3aab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d1cc5640 1 -- 172.21.15.98:0/3097454545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f3a8730 msgr2=0x56388f3aab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/3097454545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f3a8730 0x56388f3aab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d1cc5640 1 -- 172.21.15.98:0/3097454545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56388f0123f0 con 0x56388f399b30 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d24c6640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa9b80a2d80 0x7fa9b80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d14c4640 1 --2- 172.21.15.98:0/3097454545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f3a8730 0x56388f3aab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9c2ffd640 1 -- 172.21.15.98:0/134313298 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fa9bc0efd70 con 0x7fa9cc1021c0 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/3097454545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399b30 0x56388f3a81f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8008be0 tx=0x7fa9c800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab406d8378cdc41 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9d24c6640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa9b80a2d80 0x7fa9b80a5240 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4007310 tx=0x7fa9c4004a40 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.791+0000 7fa9b7fff640 1 -- 172.21.15.98:0/3097454545 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8014070 con 0x56388f399b30 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.792+0000 7fa9b7fff640 1 -- 172.21.15.98:0/3097454545 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9c80047a0 con 0x56388f399b30 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.792+0000 7fa9b7fff640 1 -- 172.21.15.98:0/3097454545 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8010650 con 0x56388f399b30 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.792+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/3097454545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399b30 msgr2=0x56388f3a81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.792+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/3097454545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399b30 0x56388f3a81f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8008be0 tx=0x7fa9c800e720 comp rx=0 tx=0).stop 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.792+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/3097454545 shutdown_connections 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.792+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/3097454545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f3a8730 0x56388f3aab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.792+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/3097454545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399b30 0x56388f3a81f0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.792+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/3097454545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399210 0x56388f3995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.792+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/3097454545 >> 172.21.15.98:0/3097454545 conn(0x56388f395650 msgr2=0x56388f395a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.792+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/3097454545 shutdown_connections 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.792+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/3097454545 wait complete. 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d6c3a740 1 -- start start 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d6c3a740 1 Processor -- start 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d6c3a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399210 0x56388f3a5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d6c3a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f399b30 0x56388f3a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d6c3a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f3a8730 0x56388f3a6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d6c3a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56388f3ab570 con 0x56388f3a8730 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d6c3a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56388f2b9c40 con 0x56388f399b30 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d6c3a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56388f3ab6f0 con 0x56388f399210 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d6c3a740 10 client.0 _ll_register_callbacks cb 0x56388f011be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d24c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f3a8730 0x56388f3a6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d24c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f3a8730 0x56388f3a6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55330/0 (socket says 172.21.15.98:55330) 2024-05-09T08:57:39.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d24c6640 1 -- 172.21.15.98:0/4086965948 learned_addr learned my addr 172.21.15.98:0/4086965948 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d14c4640 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f399b30 0x56388f3a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399210 0x56388f3a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d24c6640 1 -- 172.21.15.98:0/4086965948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399210 msgr2=0x56388f3a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.793+0000 7fa9d24c6640 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399210 0x56388f3a5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d24c6640 1 -- 172.21.15.98:0/4086965948 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f399b30 msgr2=0x56388f3a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d24c6640 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f399b30 0x56388f3a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d24c6640 1 -- 172.21.15.98:0/4086965948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56388f0dc7d0 con 0x56388f3a8730 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399210 0x56388f3a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d14c4640 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f399b30 0x56388f3a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d24c6640 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f3a8730 0x56388f3a6090 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4012e50 tx=0x7fa9c4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f70bb36266fa1e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9b57fa640 1 -- 172.21.15.98:0/4086965948 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4027670 con 0x56388f3a8730 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/4086965948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9c80047a0 con 0x56388f3a8730 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9b57fa640 1 -- 172.21.15.98:0/4086965948 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fa9c401e450 con 0x56388f3a8730 2024-05-09T08:57:39.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9b57fa640 1 -- 172.21.15.98:0/4086965948 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa9c4027810 con 0x56388f3a8730 2024-05-09T08:57:39.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d6c3a740 10 client.17412 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:39.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/4086965948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56388f3b7730 con 0x56388f3a8730 2024-05-09T08:57:39.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9b57fa640 1 -- 172.21.15.98:0/4086965948 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9c80047a0 con 0x56388f3a8730 2024-05-09T08:57:39.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9b57fa640 1 -- 172.21.15.98:0/4086965948 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa9c401b440 con 0x56388f3a8730 2024-05-09T08:57:39.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d6c3a740 10 client.17412 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:39.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d6c3a740 10 client.17412 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:57:39.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d6c3a740 20 client.17412 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:39.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x56388f3b8bf0 0x56388f3bb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d6c3a740 4 client.17412 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:57:39.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.794+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/4086965948 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56388f30c370 con 0x56388f3b8bf0 2024-05-09T08:57:39.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.795+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x56388f3b8bf0 0x56388f3bb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.795+0000 7fa9b57fa640 10 client.17412 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:57:39.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.795+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x56388f3b8bf0 0x56388f3bb040 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a6ebd7de6fa5fa3a server_cookie=bd431356884ba9d4 in_seq=0 out_seq=0 2024-05-09T08:57:39.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9b57fa640 1 -- 172.21.15.98:0/4086965948 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56388f30c370 con 0x56388f3b8bf0 2024-05-09T08:57:39.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9b57fa640 10 client.17412 handle_command_reply: tid=0 2024-05-09T08:57:39.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9d6c3a740 1 client.17412 shutdown 2024-05-09T08:57:39.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/4086965948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f3a8730 msgr2=0x56388f3a6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f3a8730 0x56388f3a6090 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4012e50 tx=0x7fa9c4004ab0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/4086965948 shutdown_connections 2024-05-09T08:57:39.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x56388f3b8bf0 0x56388f3bb040 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f3a8730 0x56388f3a6090 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f399b30 0x56388f3a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/4086965948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399210 0x56388f3a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9b57fa640 0 client.17412 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:57:39.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/4086965948 >> 172.21.15.98:0/4086965948 conn(0x56388f395650 msgr2=0x56388f39b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/4086965948 shutdown_connections 2024-05-09T08:57:39.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/4086965948 wait complete. 2024-05-09T08:57:39.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.796+0000 7fa9d6c3a740 20 client.17412 trim_cache size 0 max 16384 2024-05-09T08:57:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.809+0000 7fa9d6c3a740 1 Processor -- start 2024-05-09T08:57:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.809+0000 7fa9d6c3a740 1 -- start start 2024-05-09T08:57:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.809+0000 7fa9d6c3a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399210 0x56388f3b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.809+0000 7fa9d6c3a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f399b30 0x56388f3a4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.809+0000 7fa9d6c3a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f3a8730 0x56388f3c08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.809+0000 7fa9d6c3a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc1acbc0 con 0x56388f399210 2024-05-09T08:57:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.809+0000 7fa9d6c3a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc1ae700 con 0x56388f399b30 2024-05-09T08:57:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.809+0000 7fa9d6c3a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc1ad290 con 0x56388f3a8730 2024-05-09T08:57:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.809+0000 7fa9d1cc5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399210 0x56388f3b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.810+0000 7fa9d1cc5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399210 0x56388f3b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55342/0 (socket says 172.21.15.98:55342) 2024-05-09T08:57:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.810+0000 7fa9d1cc5640 1 -- 172.21.15.98:0/4113001427 learned_addr learned my addr 172.21.15.98:0/4113001427 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.810+0000 7fa9d1cc5640 1 -- 172.21.15.98:0/4113001427 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f3a8730 msgr2=0x56388f3c08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.810+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/4113001427 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f3a8730 0x56388f3c08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.810+0000 7fa9d1cc5640 1 -- 172.21.15.98:0/4113001427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f399b30 msgr2=0x56388f3a4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.810+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/4113001427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f399b30 0x56388f3a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.810+0000 7fa9d1cc5640 1 -- 172.21.15.98:0/4113001427 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56388f3911a0 con 0x56388f399210 2024-05-09T08:57:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.810+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/4113001427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399210 0x56388f3b44d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8008f40 tx=0x7fa9c800ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f311830e3ce956 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.810+0000 7fa9b77fe640 1 -- 172.21.15.98:0/4113001427 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8014040 con 0x56388f399210 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.810+0000 7fa9b77fe640 1 -- 172.21.15.98:0/4113001427 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9c8005510 con 0x56388f399210 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.810+0000 7fa9b77fe640 1 -- 172.21.15.98:0/4113001427 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8004c60 con 0x56388f399210 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/4113001427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399210 msgr2=0x56388f3b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/4113001427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399210 0x56388f3b44d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8008f40 tx=0x7fa9c800ef50 comp rx=0 tx=0).stop 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/4113001427 shutdown_connections 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/4113001427 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f3a8730 0x56388f3c08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/4113001427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f399b30 0x56388f3a4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/4113001427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399210 0x56388f3b44d0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/4113001427 >> 172.21.15.98:0/4113001427 conn(0x56388f395650 msgr2=0x56388f395a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/4113001427 shutdown_connections 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/4113001427 wait complete. 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 -- start start 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 Processor -- start 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399210 0x56388f3b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399b30 0x56388f3ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f3a8730 0x56388f3ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc1ae030 con 0x56388f399210 2024-05-09T08:57:39.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc1ad960 con 0x56388f3a8730 2024-05-09T08:57:39.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc1aabc0 con 0x56388f399b30 2024-05-09T08:57:39.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d6c3a740 10 client.0 _ll_register_callbacks cb 0x7fa9c4007c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:39.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d1cc5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399210 0x56388f3b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d1cc5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399210 0x56388f3b0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55350/0 (socket says 172.21.15.98:55350) 2024-05-09T08:57:39.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.811+0000 7fa9d1cc5640 1 -- 172.21.15.98:0/220947306 learned_addr learned my addr 172.21.15.98:0/220947306 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9d24c6640 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f3a8730 0x56388f3ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9d14c4640 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399b30 0x56388f3ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9d1cc5640 1 -- 172.21.15.98:0/220947306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399b30 msgr2=0x56388f3ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399b30 0x56388f3ac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9d1cc5640 1 -- 172.21.15.98:0/220947306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f3a8730 msgr2=0x56388f3ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f3a8730 0x56388f3ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9d1cc5640 1 -- 172.21.15.98:0/220947306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56388f3ad080 con 0x56388f399210 2024-05-09T08:57:39.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9d14c4640 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399b30 0x56388f3ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:39.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9d24c6640 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f3a8730 0x56388f3ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:39.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9d1cc5640 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399210 0x56388f3b0ed0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800ede0 tx=0x7fa9c8004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5162ac3d04cbb2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:39.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9b57fa640 1 -- 172.21.15.98:0/220947306 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8014040 con 0x56388f399210 2024-05-09T08:57:39.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9b57fa640 1 -- 172.21.15.98:0/220947306 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fa9c802e450 con 0x56388f399210 2024-05-09T08:57:39.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/220947306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9c8005510 con 0x56388f399210 2024-05-09T08:57:39.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9b57fa640 1 -- 172.21.15.98:0/220947306 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa9c8037050 con 0x56388f399210 2024-05-09T08:57:39.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9d6c3a740 10 client.17418 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:39.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/220947306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56388f41f260 con 0x56388f399210 2024-05-09T08:57:39.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9b57fa640 1 -- 172.21.15.98:0/220947306 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9c8005510 con 0x56388f399210 2024-05-09T08:57:39.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.812+0000 7fa9b57fa640 1 -- 172.21.15.98:0/220947306 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa9c8010810 con 0x56388f399210 2024-05-09T08:57:39.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.813+0000 7fa9d6c3a740 10 client.17418 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:39.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.813+0000 7fa9d6c3a740 10 client.17418 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:57:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.813+0000 7fa9d6c3a740 20 client.17418 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.813+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x56388f41f7e0 0x56388f421bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.813+0000 7fa9d6c3a740 4 client.17418 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:57:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.813+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/220947306 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56388f3066c0 con 0x56388f41f7e0 2024-05-09T08:57:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.813+0000 7fa9d14c4640 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x56388f41f7e0 0x56388f421bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.813+0000 7fa9b57fa640 10 client.17418 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:57:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.813+0000 7fa9d14c4640 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x56388f41f7e0 0x56388f421bd0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=edd7dfcb37ef57c8 server_cookie=538f99b1a3299e1e in_seq=0 out_seq=0 2024-05-09T08:57:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.813+0000 7fa9b57fa640 1 -- 172.21.15.98:0/220947306 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56388f3066c0 con 0x56388f41f7e0 2024-05-09T08:57:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.813+0000 7fa9b57fa640 10 client.17418 handle_command_reply: tid=0 2024-05-09T08:57:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.813+0000 7fa9d6c3a740 1 client.17418 shutdown 2024-05-09T08:57:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.813+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/220947306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399210 msgr2=0x56388f3b0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.813+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399210 0x56388f3b0ed0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800ede0 tx=0x7fa9c8004700 comp rx=0 tx=0).stop 2024-05-09T08:57:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/220947306 shutdown_connections 2024-05-09T08:57:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x56388f41f7e0 0x56388f421bd0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56388f3a8730 0x56388f3ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56388f399b30 0x56388f3ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d6c3a740 1 --2- 172.21.15.98:0/220947306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56388f399210 0x56388f3b0ed0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/220947306 >> 172.21.15.98:0/220947306 conn(0x56388f395650 msgr2=0x56388f393f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9b57fa640 0 client.17418 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:57:39.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/220947306 shutdown_connections 2024-05-09T08:57:39.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d6c3a740 1 -- 172.21.15.98:0/220947306 wait complete. 2024-05-09T08:57:39.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d6c3a740 20 client.17418 trim_cache size 0 max 16384 2024-05-09T08:57:39.820 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:57:39.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d3f29640 1 -- 172.21.15.98:0/134313298 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa9b80a2d80 msgr2=0x7fa9b80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d3f29640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa9b80a2d80 0x7fa9b80a5240 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4007310 tx=0x7fa9c4004a40 comp rx=0 tx=0).stop 2024-05-09T08:57:39.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d3f29640 1 -- 172.21.15.98:0/134313298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9cc1021c0 msgr2=0x7fa9cc1985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d3f29640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9cc1021c0 0x7fa9cc1985e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc009a50 tx=0x7fa9bc004270 comp rx=0 tx=0).stop 2024-05-09T08:57:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d3f29640 1 -- 172.21.15.98:0/134313298 shutdown_connections 2024-05-09T08:57:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d3f29640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa9b80a2d80 0x7fa9b80a5240 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d3f29640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9cc107810 0x7fa9cc19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d3f29640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9cc1021c0 0x7fa9cc1985e0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d3f29640 1 --2- 172.21.15.98:0/134313298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9cc101800 0x7fa9cc1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d3f29640 1 -- 172.21.15.98:0/134313298 >> 172.21.15.98:0/134313298 conn(0x7fa9cc078c80 msgr2=0x7fa9cc0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d3f29640 1 -- 172.21.15.98:0/134313298 shutdown_connections 2024-05-09T08:57:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.814+0000 7fa9d3f29640 1 -- 172.21.15.98:0/134313298 wait complete. 2024-05-09T08:57:39.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:57:39.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.929+0000 7f2e04f48640 1 -- 172.21.15.98:0/2016252306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e00100700 msgr2=0x7f2e00100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.929+0000 7f2e04f48640 1 --2- 172.21.15.98:0/2016252306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e00100700 0x7f2e00100b80 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f2df4009930 tx=0x7f2df402a420 comp rx=0 tx=0).stop 2024-05-09T08:57:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.929+0000 7f2e04f48640 1 -- 172.21.15.98:0/2016252306 shutdown_connections 2024-05-09T08:57:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.929+0000 7f2e04f48640 1 --2- 172.21.15.98:0/2016252306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e001010c0 0x7f2e0010d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.929+0000 7f2e04f48640 1 --2- 172.21.15.98:0/2016252306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e00100700 0x7f2e00100b80 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.929+0000 7f2e04f48640 1 --2- 172.21.15.98:0/2016252306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e00106710 0x7f2e00106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.929+0000 7f2e04f48640 1 -- 172.21.15.98:0/2016252306 >> 172.21.15.98:0/2016252306 conn(0x7f2e00077830 msgr2=0x7f2e00077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.929+0000 7f2e04f48640 1 -- 172.21.15.98:0/2016252306 shutdown_connections 2024-05-09T08:57:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.929+0000 7f2e04f48640 1 -- 172.21.15.98:0/2016252306 wait complete. 2024-05-09T08:57:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.929+0000 7f2e04f48640 1 Processor -- start 2024-05-09T08:57:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2e04f48640 1 -- start start 2024-05-09T08:57:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2e04f48640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e00100700 0x7f2e00198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2e04f48640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e001010c0 0x7f2e00198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2e04f48640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e00106710 0x7f2e0019ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2e04f48640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e00111c20 con 0x7f2e00106710 2024-05-09T08:57:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2e04f48640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2e00111aa0 con 0x7f2e00100700 2024-05-09T08:57:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2e04f48640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2e00111da0 con 0x7f2e001010c0 2024-05-09T08:57:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2dfed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e00106710 0x7f2e0019ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2dfed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e00106710 0x7f2e0019ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55380/0 (socket says 172.21.15.98:55380) 2024-05-09T08:57:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2dfed9d640 1 -- 172.21.15.98:0/2623523649 learned_addr learned my addr 172.21.15.98:0/2623523649 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/2623523649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e001010c0 0x7f2e00198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2dfe59c640 1 --2- 172.21.15.98:0/2623523649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e00100700 0x7f2e00198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2dfed9d640 1 -- 172.21.15.98:0/2623523649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e001010c0 msgr2=0x7f2e00198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2dfed9d640 1 --2- 172.21.15.98:0/2623523649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e001010c0 0x7f2e00198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2dfed9d640 1 -- 172.21.15.98:0/2623523649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e00100700 msgr2=0x7f2e00198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2dfed9d640 1 --2- 172.21.15.98:0/2623523649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e00100700 0x7f2e00198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2dfed9d640 1 -- 172.21.15.98:0/2623523649 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e0019d130 con 0x7f2e00106710 2024-05-09T08:57:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2dfed9d640 1 --2- 172.21.15.98:0/2623523649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e00106710 0x7f2e0019ca00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f2df800cad0 tx=0x7f2df800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.930+0000 7f2def7fe640 1 -- 172.21.15.98:0/2623523649 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df800ae20 con 0x7f2e00106710 2024-05-09T08:57:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.931+0000 7f2e04f48640 1 -- 172.21.15.98:0/2623523649 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e0019d420 con 0x7f2e00106710 2024-05-09T08:57:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.931+0000 7f2def7fe640 1 -- 172.21.15.98:0/2623523649 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2df8005020 con 0x7f2e00106710 2024-05-09T08:57:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.931+0000 7f2def7fe640 1 -- 172.21.15.98:0/2623523649 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df800f6b0 con 0x7f2e00106710 2024-05-09T08:57:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.931+0000 7f2e04f48640 1 -- 172.21.15.98:0/2623523649 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e001a4cb0 con 0x7f2e00106710 2024-05-09T08:57:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.933+0000 7f2def7fe640 1 -- 172.21.15.98:0/2623523649 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2df80051d0 con 0x7f2e00106710 2024-05-09T08:57:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.933+0000 7f2e07c4b740 1 Processor -- start 2024-05-09T08:57:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2e07c4b740 1 -- start start 2024-05-09T08:57:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2e07c4b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9d080 0x55f56af9d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2e07c4b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56af9da30 0x55f56afac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2e07c4b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56afac730 0x55f56afaeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2e07c4b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f56af02c40 con 0x55f56afac730 2024-05-09T08:57:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2e07c4b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f56aeaa580 con 0x55f56af9d080 2024-05-09T08:57:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2e07c4b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f56aefec00 con 0x55f56af9da30 2024-05-09T08:57:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2dfdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56afac730 0x55f56afaeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2dfdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56afac730 0x55f56afaeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55386/0 (socket says 172.21.15.98:55386) 2024-05-09T08:57:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2dfdd9b640 1 -- 172.21.15.98:0/1317025865 learned_addr learned my addr 172.21.15.98:0/1317025865 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/1317025865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9d080 0x55f56af9d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2dfe59c640 1 --2- 172.21.15.98:0/1317025865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56af9da30 0x55f56afac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2def7fe640 1 --2- 172.21.15.98:0/2623523649 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2de80a2d00 0x7f2de80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2dfdd9b640 1 -- 172.21.15.98:0/1317025865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56af9da30 msgr2=0x55f56afac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/1317025865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56af9da30 0x55f56afac1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2dfdd9b640 1 -- 172.21.15.98:0/1317025865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9d080 msgr2=0x55f56af9d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/1317025865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9d080 0x55f56af9d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.934+0000 7f2dfdd9b640 1 -- 172.21.15.98:0/1317025865 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f56ac163f0 con 0x55f56afac730 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.935+0000 7f2dfe59c640 1 --2- 172.21.15.98:0/1317025865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56af9da30 0x55f56afac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.935+0000 7f2dfed9d640 1 --2- 172.21.15.98:0/2623523649 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2de80a2d00 0x7f2de80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.935+0000 7f2def7fe640 1 -- 172.21.15.98:0/2623523649 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f2df80cfc50 con 0x7f2e00106710 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.935+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/1317025865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56afac730 0x55f56afaeb20 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f2df402a940 tx=0x7f2df4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c0dd2c30f05a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.935+0000 7f2decff9640 1 -- 172.21.15.98:0/1317025865 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df4035070 con 0x55f56afac730 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.935+0000 7f2decff9640 1 -- 172.21.15.98:0/1317025865 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2df4031a10 con 0x55f56afac730 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.935+0000 7f2decff9640 1 -- 172.21.15.98:0/1317025865 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df4004a90 con 0x55f56afac730 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 -- 172.21.15.98:0/1317025865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56afac730 msgr2=0x55f56afaeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/1317025865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56afac730 0x55f56afaeb20 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f2df402a940 tx=0x7f2df4005fd0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2dfed9d640 1 --2- 172.21.15.98:0/2623523649 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2de80a2d00 0x7f2de80a51c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2df80a4440 tx=0x7f2df8097900 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 -- 172.21.15.98:0/1317025865 shutdown_connections 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/1317025865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56afac730 0x55f56afaeb20 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/1317025865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56af9da30 0x55f56afac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/1317025865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9d080 0x55f56af9d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 -- 172.21.15.98:0/1317025865 >> 172.21.15.98:0/1317025865 conn(0x55f56af99650 msgr2=0x55f56af99a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 -- 172.21.15.98:0/1317025865 shutdown_connections 2024-05-09T08:57:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 -- 172.21.15.98:0/1317025865 wait complete. 2024-05-09T08:57:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 -- start start 2024-05-09T08:57:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 Processor -- start 2024-05-09T08:57:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56af9d080 0x55f56afa96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9da30 0x55f56afa9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56afac730 0x55f56afaa120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f56afaf490 con 0x55f56af9da30 2024-05-09T08:57:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f56aebdc40 con 0x55f56afac730 2024-05-09T08:57:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f56afaf610 con 0x55f56af9d080 2024-05-09T08:57:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2e07c4b740 10 client.0 _ll_register_callbacks cb 0x55f56ac15be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2dfe59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9da30 0x55f56afa9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2dfe59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9da30 0x55f56afa9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55394/0 (socket says 172.21.15.98:55394) 2024-05-09T08:57:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.936+0000 7f2dfe59c640 1 -- 172.21.15.98:0/1183922108 learned_addr learned my addr 172.21.15.98:0/1183922108 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2dfe59c640 1 -- 172.21.15.98:0/1183922108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56af9d080 msgr2=0x55f56afa96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/1183922108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56af9d080 0x55f56afa96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2dfe59c640 1 --2- 172.21.15.98:0/1183922108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56af9d080 0x55f56afa96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2dfe59c640 1 -- 172.21.15.98:0/1183922108 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56afac730 msgr2=0x55f56afaa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2dfe59c640 1 --2- 172.21.15.98:0/1183922108 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56afac730 0x55f56afaa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2dfe59c640 1 -- 172.21.15.98:0/1183922108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f56ace07d0 con 0x55f56af9da30 2024-05-09T08:57:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2dfe59c640 1 --2- 172.21.15.98:0/1183922108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9da30 0x55f56afa9be0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f2df000f6c0 tx=0x7f2df000fb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e3a12b2748ff36 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2de5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2de5ffb640 1 -- 172.21.15.98:0/1183922108 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0016070 con 0x55f56af9da30 2024-05-09T08:57:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2de5ffb640 1 -- 172.21.15.98:0/1183922108 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f2df000bb00 con 0x55f56af9da30 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2e07c4b740 1 -- 172.21.15.98:0/1183922108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f56af10370 con 0x55f56af9da30 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2de5ffb640 1 -- 172.21.15.98:0/1183922108 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2df0025050 con 0x55f56af9da30 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2e07c4b740 10 client.17430 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2e07c4b740 1 -- 172.21.15.98:0/1183922108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f56afbb730 con 0x55f56af9da30 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2de5ffb640 1 -- 172.21.15.98:0/1183922108 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f56af10370 con 0x55f56af9da30 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2de5ffb640 1 -- 172.21.15.98:0/1183922108 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f2df001a400 con 0x55f56af9da30 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2e07c4b740 10 client.17430 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2e07c4b740 10 client.17430 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2e07c4b740 20 client.17430 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.937+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/1183922108 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f56afbcc90 0x55f56afbf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.938+0000 7f2e07c4b740 4 client.17430 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.938+0000 7f2e07c4b740 1 -- 172.21.15.98:0/1183922108 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f56af0a6c0 con 0x55f56afbcc90 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.938+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/1183922108 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f56afbcc90 0x55f56afbf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.938+0000 7f2de5ffb640 10 client.17430 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:57:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.938+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/1183922108 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f56afbcc90 0x55f56afbf0b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=56f96ba8c631f9aa server_cookie=127288dff1268553 in_seq=0 out_seq=0 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2de5ffb640 1 -- 172.21.15.98:0/1183922108 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f56af0a6c0 con 0x55f56afbcc90 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2de5ffb640 10 client.17430 handle_command_reply: tid=0 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2e07c4b740 1 client.17430 shutdown 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2e07c4b740 1 -- 172.21.15.98:0/1183922108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9da30 msgr2=0x55f56afa9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/1183922108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9da30 0x55f56afa9be0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f2df000f6c0 tx=0x7f2df000fb80 comp rx=0 tx=0).stop 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2e07c4b740 1 -- 172.21.15.98:0/1183922108 shutdown_connections 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/1183922108 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f56afbcc90 0x55f56afbf0b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/1183922108 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56afac730 0x55f56afaa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2de5ffb640 0 client.17430 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/1183922108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9da30 0x55f56afa9be0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/1183922108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56af9d080 0x55f56afa96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2e07c4b740 1 -- 172.21.15.98:0/1183922108 >> 172.21.15.98:0/1183922108 conn(0x55f56af99650 msgr2=0x55f56af9f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2e07c4b740 1 -- 172.21.15.98:0/1183922108 shutdown_connections 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2e07c4b740 1 -- 172.21.15.98:0/1183922108 wait complete. 2024-05-09T08:57:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.939+0000 7f2e07c4b740 20 client.17430 trim_cache size 0 max 16384 2024-05-09T08:57:39.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2e07c4b740 1 Processor -- start 2024-05-09T08:57:39.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2e07c4b740 1 -- start start 2024-05-09T08:57:39.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2e07c4b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9d080 0x55f56afb83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2e07c4b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9da30 0x55f56afa9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2e07c4b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56afac730 0x55f56af98630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2e07c4b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e001acd70 con 0x55f56af9d080 2024-05-09T08:57:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2e07c4b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2e001ae8b0 con 0x55f56af9da30 2024-05-09T08:57:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2e07c4b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2e001ad440 con 0x55f56afac730 2024-05-09T08:57:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2dfdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9d080 0x55f56afb83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2dfdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9d080 0x55f56afb83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55408/0 (socket says 172.21.15.98:55408) 2024-05-09T08:57:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2dfdd9b640 1 -- 172.21.15.98:0/2868030973 learned_addr learned my addr 172.21.15.98:0/2868030973 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2dfe59c640 1 --2- 172.21.15.98:0/2868030973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9da30 0x55f56afa9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/2868030973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56afac730 0x55f56af98630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2dfdd9b640 1 -- 172.21.15.98:0/2868030973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56afac730 msgr2=0x55f56af98630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/2868030973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56afac730 0x55f56af98630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2dfdd9b640 1 -- 172.21.15.98:0/2868030973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9da30 msgr2=0x55f56afa9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/2868030973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9da30 0x55f56afa9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2dfdd9b640 1 -- 172.21.15.98:0/2868030973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f56af9a860 con 0x55f56af9d080 2024-05-09T08:57:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.953+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/2868030973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9d080 0x55f56afb83c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f2df4004590 tx=0x7f2df402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e598f37822d08655 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.954+0000 7f2de7fff640 1 -- 172.21.15.98:0/2868030973 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df4035040 con 0x55f56af9d080 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.954+0000 7f2de7fff640 1 -- 172.21.15.98:0/2868030973 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2df4005600 con 0x55f56af9d080 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.954+0000 7f2de7fff640 1 -- 172.21.15.98:0/2868030973 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df4004a90 con 0x55f56af9d080 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.954+0000 7f2e07c4b740 1 -- 172.21.15.98:0/2868030973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9d080 msgr2=0x55f56afb83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.954+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/2868030973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9d080 0x55f56afb83c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f2df4004590 tx=0x7f2df402a420 comp rx=0 tx=0).stop 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.954+0000 7f2e07c4b740 1 -- 172.21.15.98:0/2868030973 shutdown_connections 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.954+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/2868030973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56afac730 0x55f56af98630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.954+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/2868030973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9da30 0x55f56afa9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.954+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/2868030973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9d080 0x55f56afb83c0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.954+0000 7f2e07c4b740 1 -- 172.21.15.98:0/2868030973 >> 172.21.15.98:0/2868030973 conn(0x55f56af99650 msgr2=0x55f56af97f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.954+0000 7f2e07c4b740 1 -- 172.21.15.98:0/2868030973 shutdown_connections 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.954+0000 7f2e07c4b740 1 -- 172.21.15.98:0/2868030973 wait complete. 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.954+0000 7f2e07c4b740 1 -- start start 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2e07c4b740 1 Processor -- start 2024-05-09T08:57:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2e07c4b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9d080 0x55f56afb4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2e07c4b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9da30 0x55f56afa6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2e07c4b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56afac730 0x55f56afa65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2e07c4b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e001ae1e0 con 0x55f56af9da30 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2e07c4b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2e001adb10 con 0x55f56af9d080 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2e07c4b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2e001aad70 con 0x55f56afac730 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2e07c4b740 10 client.0 _ll_register_callbacks cb 0x7f2df0005ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2dfe59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9da30 0x55f56afa6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2dfe59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9da30 0x55f56afa6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55422/0 (socket says 172.21.15.98:55422) 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2dfe59c640 1 -- 172.21.15.98:0/278662614 learned_addr learned my addr 172.21.15.98:0/278662614 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/278662614 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56afac730 0x55f56afa65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/278662614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9d080 0x55f56afb4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2dfe59c640 1 -- 172.21.15.98:0/278662614 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56afac730 msgr2=0x55f56afa65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2dfe59c640 1 --2- 172.21.15.98:0/278662614 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56afac730 0x55f56afa65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2dfe59c640 1 -- 172.21.15.98:0/278662614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9d080 msgr2=0x55f56afb4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2dfe59c640 1 --2- 172.21.15.98:0/278662614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9d080 0x55f56afb4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2dfe59c640 1 -- 172.21.15.98:0/278662614 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f56afa6dd0 con 0x55f56af9da30 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/278662614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9d080 0x55f56afb4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2dfe59c640 1 --2- 172.21.15.98:0/278662614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9da30 0x55f56afa6060 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x55f56afa8f00 tx=0x7f2df000d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84eb37b6c9ace08 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.955+0000 7f2de5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2de5ffb640 1 -- 172.21.15.98:0/278662614 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0016040 con 0x55f56af9da30 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2de5ffb640 1 -- 172.21.15.98:0/278662614 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f2df000b900 con 0x55f56af9da30 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2e07c4b740 1 -- 172.21.15.98:0/278662614 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2df4005600 con 0x55f56af9da30 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2de5ffb640 1 -- 172.21.15.98:0/278662614 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2df003b050 con 0x55f56af9da30 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2e07c4b740 10 client.17436 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2e07c4b740 1 -- 172.21.15.98:0/278662614 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f56b023250 con 0x55f56af9da30 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2de5ffb640 1 -- 172.21.15.98:0/278662614 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2df4005600 con 0x55f56af9da30 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2de5ffb640 1 -- 172.21.15.98:0/278662614 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f2df0004660 con 0x55f56af9da30 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2e07c4b740 10 client.17436 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2e07c4b740 10 client.17436 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:57:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2e07c4b740 20 client.17436 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/278662614 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f56b023a00 0x55f56b025e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2e07c4b740 4 client.17436 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2e07c4b740 1 -- 172.21.15.98:0/278662614 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f56ae87820 con 0x55f56b023a00 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.956+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/278662614 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f56b023a00 0x55f56b025e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2de5ffb640 10 client.17436 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2dfdd9b640 1 --2- 172.21.15.98:0/278662614 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f56b023a00 0x55f56b025e20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=400dd9e81a83fd5d server_cookie=2d78bb9c6ce802ae in_seq=0 out_seq=0 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2de5ffb640 1 -- 172.21.15.98:0/278662614 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f56ae87820 con 0x55f56b023a00 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2de5ffb640 10 client.17436 handle_command_reply: tid=0 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2e07c4b740 1 client.17436 shutdown 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2e07c4b740 1 -- 172.21.15.98:0/278662614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9da30 msgr2=0x55f56afa6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/278662614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9da30 0x55f56afa6060 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x55f56afa8f00 tx=0x7f2df000d660 comp rx=0 tx=0).stop 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2e07c4b740 1 -- 172.21.15.98:0/278662614 shutdown_connections 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/278662614 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f56b023a00 0x55f56b025e20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/278662614 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f56afac730 0x55f56afa65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2de5ffb640 0 client.17436 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:57:39.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/278662614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f56af9da30 0x55f56afa6060 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2e07c4b740 1 --2- 172.21.15.98:0/278662614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f56af9d080 0x55f56afb4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2e07c4b740 1 -- 172.21.15.98:0/278662614 >> 172.21.15.98:0/278662614 conn(0x55f56af99650 msgr2=0x55f56af987b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2e07c4b740 1 -- 172.21.15.98:0/278662614 shutdown_connections 2024-05-09T08:57:39.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2e07c4b740 1 -- 172.21.15.98:0/278662614 wait complete. 2024-05-09T08:57:39.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.957+0000 7f2e07c4b740 20 client.17436 trim_cache size 0 max 16384 2024-05-09T08:57:39.963 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:57:39.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.958+0000 7f2e04f48640 1 -- 172.21.15.98:0/2623523649 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2de80a2d00 msgr2=0x7f2de80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.958+0000 7f2e04f48640 1 --2- 172.21.15.98:0/2623523649 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2de80a2d00 0x7f2de80a51c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2df80a4440 tx=0x7f2df8097900 comp rx=0 tx=0).stop 2024-05-09T08:57:39.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.958+0000 7f2e04f48640 1 -- 172.21.15.98:0/2623523649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e00106710 msgr2=0x7f2e0019ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:39.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.958+0000 7f2e04f48640 1 --2- 172.21.15.98:0/2623523649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e00106710 0x7f2e0019ca00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f2df800cad0 tx=0x7f2df800cf90 comp rx=0 tx=0).stop 2024-05-09T08:57:39.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.958+0000 7f2e04f48640 1 -- 172.21.15.98:0/2623523649 shutdown_connections 2024-05-09T08:57:39.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.958+0000 7f2e04f48640 1 --2- 172.21.15.98:0/2623523649 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2de80a2d00 0x7f2de80a51c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.958+0000 7f2e04f48640 1 --2- 172.21.15.98:0/2623523649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e00106710 0x7f2e0019ca00 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.958+0000 7f2e04f48640 1 --2- 172.21.15.98:0/2623523649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e001010c0 0x7f2e00198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.958+0000 7f2e04f48640 1 --2- 172.21.15.98:0/2623523649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e00100700 0x7f2e00198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:39.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.958+0000 7f2e04f48640 1 -- 172.21.15.98:0/2623523649 >> 172.21.15.98:0/2623523649 conn(0x7f2e00077830 msgr2=0x7f2e00102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:39.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.958+0000 7f2e04f48640 1 -- 172.21.15.98:0/2623523649 shutdown_connections 2024-05-09T08:57:39.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:39.958+0000 7f2e04f48640 1 -- 172.21.15.98:0/2623523649 wait complete. 2024-05-09T08:57:39.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:57:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.072+0000 7fe31ee15640 1 -- 172.21.15.98:0/4167307393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3181010c0 msgr2=0x7fe31810d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 --2- 172.21.15.98:0/4167307393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3181010c0 0x7fe31810d5d0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe310009930 tx=0x7fe31002a420 comp rx=0 tx=0).stop 2024-05-09T08:57:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 -- 172.21.15.98:0/4167307393 shutdown_connections 2024-05-09T08:57:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 --2- 172.21.15.98:0/4167307393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3181010c0 0x7fe31810d5d0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 --2- 172.21.15.98:0/4167307393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe318100700 0x7fe318100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 --2- 172.21.15.98:0/4167307393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe318106710 0x7fe318106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 -- 172.21.15.98:0/4167307393 >> 172.21.15.98:0/4167307393 conn(0x7fe318077830 msgr2=0x7fe318077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 -- 172.21.15.98:0/4167307393 shutdown_connections 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 -- 172.21.15.98:0/4167307393 wait complete. 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 Processor -- start 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 -- start start 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe318100700 0x7fe318198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3181010c0 0x7fe318199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe318106710 0x7fe3181a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe318110b20 con 0x7fe318100700 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe3181109a0 con 0x7fe3181010c0 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31ee15640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe318110ca0 con 0x7fe318106710 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31cbb1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe318100700 0x7fe318198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31cbb1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe318100700 0x7fe318198f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55446/0 (socket says 172.21.15.98:55446) 2024-05-09T08:57:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.073+0000 7fe31cbb1640 1 -- 172.21.15.98:0/2805139983 learned_addr learned my addr 172.21.15.98:0/2805139983 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.074+0000 7fe31cbb1640 1 -- 172.21.15.98:0/2805139983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe318106710 msgr2=0x7fe3181a3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.074+0000 7fe317fff640 1 --2- 172.21.15.98:0/2805139983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3181010c0 0x7fe318199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.074+0000 7fe31cbb1640 1 --2- 172.21.15.98:0/2805139983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe318106710 0x7fe3181a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.074+0000 7fe31cbb1640 1 -- 172.21.15.98:0/2805139983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3181010c0 msgr2=0x7fe318199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.074+0000 7fe31cbb1640 1 --2- 172.21.15.98:0/2805139983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3181010c0 0x7fe318199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.074+0000 7fe31cbb1640 1 -- 172.21.15.98:0/2805139983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3181a4500 con 0x7fe318100700 2024-05-09T08:57:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.074+0000 7fe31cbb1640 1 --2- 172.21.15.98:0/2805139983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe318100700 0x7fe318198f40 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fe30800d770 tx=0x7fe30800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.074+0000 7fe315ffb640 1 -- 172.21.15.98:0/2805139983 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308017070 con 0x7fe318100700 2024-05-09T08:57:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.074+0000 7fe315ffb640 1 -- 172.21.15.98:0/2805139983 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe308005500 con 0x7fe318100700 2024-05-09T08:57:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.074+0000 7fe31ee15640 1 -- 172.21.15.98:0/2805139983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3181a46d0 con 0x7fe318100700 2024-05-09T08:57:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.074+0000 7fe315ffb640 1 -- 172.21.15.98:0/2805139983 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308004e00 con 0x7fe318100700 2024-05-09T08:57:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.074+0000 7fe31ee15640 1 -- 172.21.15.98:0/2805139983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3181a4bc0 con 0x7fe318100700 2024-05-09T08:57:40.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.076+0000 7fe315ffb640 1 -- 172.21.15.98:0/2805139983 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe308004fa0 con 0x7fe318100700 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe321bff740 1 Processor -- start 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe321bff740 1 -- start start 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe315ffb640 1 --2- 172.21.15.98:0/2805139983 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe3040a2d80 0x7fe3040a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe321bff740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef685000 0x55d7ef6853e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe321bff740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef6941f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe321bff740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef694730 0x55d7ef696b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe321bff740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d7ef5eac40 con 0x55d7ef685920 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe321bff740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d7ef592580 con 0x55d7ef685000 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe321bff740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d7ef5e6c00 con 0x55d7ef694730 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe31d3b2640 1 --2- 172.21.15.98:0/2805139983 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe3040a2d80 0x7fe3040a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe317fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef6941f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe317fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef6941f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55460/0 (socket says 172.21.15.98:55460) 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe31cbb1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef685000 0x55d7ef6853e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe317fff640 1 -- 172.21.15.98:0/3520810726 learned_addr learned my addr 172.21.15.98:0/3520810726 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe315ffb640 1 -- 172.21.15.98:0/2805139983 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fe3080cf1f0 con 0x7fe318100700 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe317fff640 1 -- 172.21.15.98:0/3520810726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef694730 msgr2=0x55d7ef696b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe317fff640 1 --2- 172.21.15.98:0/3520810726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef694730 0x55d7ef696b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe317fff640 1 -- 172.21.15.98:0/3520810726 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef685000 msgr2=0x55d7ef6853e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe317fff640 1 --2- 172.21.15.98:0/3520810726 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef685000 0x55d7ef6853e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.077+0000 7fe317fff640 1 -- 172.21.15.98:0/3520810726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7ef2fe3f0 con 0x55d7ef685920 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.078+0000 7fe31d3b2640 1 --2- 172.21.15.98:0/2805139983 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe3040a2d80 0x7fe3040a5240 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe3100057c0 tx=0x7fe310031490 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.078+0000 7fe317fff640 1 --2- 172.21.15.98:0/3520810726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef6941f0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fe300008df0 tx=0x7fe30000d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a0af2a3e154ce3e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.078+0000 7fe30effd640 1 -- 172.21.15.98:0/3520810726 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30000f830 con 0x55d7ef685920 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.078+0000 7fe30effd640 1 -- 172.21.15.98:0/3520810726 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe30000de20 con 0x55d7ef685920 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.078+0000 7fe30effd640 1 -- 172.21.15.98:0/3520810726 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30000e3b0 con 0x55d7ef685920 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.078+0000 7fe321bff740 1 -- 172.21.15.98:0/3520810726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 msgr2=0x55d7ef6941f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.078+0000 7fe321bff740 1 --2- 172.21.15.98:0/3520810726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef6941f0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fe300008df0 tx=0x7fe30000d710 comp rx=0 tx=0).stop 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 -- 172.21.15.98:0/3520810726 shutdown_connections 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 --2- 172.21.15.98:0/3520810726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef694730 0x55d7ef696b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 --2- 172.21.15.98:0/3520810726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef6941f0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 --2- 172.21.15.98:0/3520810726 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef685000 0x55d7ef6853e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 -- 172.21.15.98:0/3520810726 >> 172.21.15.98:0/3520810726 conn(0x55d7ef681650 msgr2=0x55d7ef681a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 -- 172.21.15.98:0/3520810726 shutdown_connections 2024-05-09T08:57:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 -- 172.21.15.98:0/3520810726 wait complete. 2024-05-09T08:57:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 -- start start 2024-05-09T08:57:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 Processor -- start 2024-05-09T08:57:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef685000 0x55d7ef6916b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef691bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef694730 0x55d7ef692130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d7ef6974f0 con 0x55d7ef685920 2024-05-09T08:57:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d7ef5a5c40 con 0x55d7ef685000 2024-05-09T08:57:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d7ef697670 con 0x55d7ef694730 2024-05-09T08:57:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe321bff740 10 client.0 _ll_register_callbacks cb 0x55d7ef2fdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe317fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef691bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.079+0000 7fe317fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef691bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55474/0 (socket says 172.21.15.98:55474) 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe317fff640 1 -- 172.21.15.98:0/3067646775 learned_addr learned my addr 172.21.15.98:0/3067646775 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe31cbb1640 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef685000 0x55d7ef6916b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe31d3b2640 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef694730 0x55d7ef692130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe317fff640 1 -- 172.21.15.98:0/3067646775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef694730 msgr2=0x55d7ef692130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe317fff640 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef694730 0x55d7ef692130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe317fff640 1 -- 172.21.15.98:0/3067646775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef685000 msgr2=0x55d7ef6916b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe317fff640 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef685000 0x55d7ef6916b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe317fff640 1 -- 172.21.15.98:0/3067646775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7ef3c87d0 con 0x55d7ef685920 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe31d3b2640 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef694730 0x55d7ef692130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe31cbb1640 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef685000 0x55d7ef6916b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe317fff640 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef691bf0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fe3000189f0 tx=0x7fe3000082b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c193ad848a44d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe2f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe2f7fff640 1 -- 172.21.15.98:0/3067646775 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe300028060 con 0x55d7ef685920 2024-05-09T08:57:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe2f7fff640 1 -- 172.21.15.98:0/3067646775 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fe30000f630 con 0x55d7ef685920 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe321bff740 1 -- 172.21.15.98:0/3067646775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe30000de20 con 0x55d7ef685920 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe2f7fff640 1 -- 172.21.15.98:0/3067646775 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe30002f050 con 0x55d7ef685920 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe321bff740 10 client.17448 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe321bff740 1 -- 172.21.15.98:0/3067646775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d7ef6a3710 con 0x55d7ef685920 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.080+0000 7fe2f7fff640 1 -- 172.21.15.98:0/3067646775 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe30000de20 con 0x55d7ef685920 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.081+0000 7fe2f7fff640 1 -- 172.21.15.98:0/3067646775 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe300005380 con 0x55d7ef685920 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.081+0000 7fe321bff740 10 client.17448 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.081+0000 7fe321bff740 10 client.17448 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.081+0000 7fe321bff740 20 client.17448 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.081+0000 7fe321bff740 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d7ef6a4b70 0x55d7ef6a6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.081+0000 7fe321bff740 4 client.17448 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.081+0000 7fe321bff740 1 -- 172.21.15.98:0/3067646775 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7ef5f8370 con 0x55d7ef6a4b70 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.081+0000 7fe31cbb1640 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d7ef6a4b70 0x55d7ef6a6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.081+0000 7fe2f7fff640 10 client.17448 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:57:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.081+0000 7fe31cbb1640 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d7ef6a4b70 0x55d7ef6a6f60 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=adfb3bc65a389a88 server_cookie=d4148265ccab4d84 in_seq=0 out_seq=0 2024-05-09T08:57:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe2f7fff640 1 -- 172.21.15.98:0/3067646775 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7ef5f8370 con 0x55d7ef6a4b70 2024-05-09T08:57:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe2f7fff640 10 client.17448 handle_command_reply: tid=0 2024-05-09T08:57:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe321bff740 1 client.17448 shutdown 2024-05-09T08:57:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe321bff740 1 -- 172.21.15.98:0/3067646775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 msgr2=0x55d7ef691bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe321bff740 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef691bf0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fe3000189f0 tx=0x7fe3000082b0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe321bff740 1 -- 172.21.15.98:0/3067646775 shutdown_connections 2024-05-09T08:57:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe321bff740 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef694730 0x55d7ef692130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe321bff740 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef691bf0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe321bff740 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d7ef6a4b70 0x55d7ef6a6f60 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe321bff740 1 --2- 172.21.15.98:0/3067646775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef685000 0x55d7ef6916b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe2f7fff640 0 client.17448 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:57:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe321bff740 1 -- 172.21.15.98:0/3067646775 >> 172.21.15.98:0/3067646775 conn(0x55d7ef681650 msgr2=0x55d7ef6874c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe321bff740 1 -- 172.21.15.98:0/3067646775 shutdown_connections 2024-05-09T08:57:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe321bff740 1 -- 172.21.15.98:0/3067646775 wait complete. 2024-05-09T08:57:40.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.082+0000 7fe321bff740 20 client.17448 trim_cache size 0 max 16384 2024-05-09T08:57:40.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:40 smithi098 ceph-mon[30092]: pgmap v217: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 159 MiB/s wr, 1.33k op/s 2024-05-09T08:57:40.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:40 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/399187889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:57:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe321bff740 1 Processor -- start 2024-05-09T08:57:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe321bff740 1 -- start start 2024-05-09T08:57:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe321bff740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef685000 0x55d7ef693de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe321bff740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef6a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe321bff740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef694730 0x55d7ef690830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe321bff740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3181acbb0 con 0x55d7ef685920 2024-05-09T08:57:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe321bff740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe3181ae6f0 con 0x55d7ef694730 2024-05-09T08:57:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe321bff740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe3181ad280 con 0x55d7ef685000 2024-05-09T08:57:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe317fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef6a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe317fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef6a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55482/0 (socket says 172.21.15.98:55482) 2024-05-09T08:57:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe317fff640 1 -- 172.21.15.98:0/319598089 learned_addr learned my addr 172.21.15.98:0/319598089 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe31cbb1640 1 --2- 172.21.15.98:0/319598089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef685000 0x55d7ef693de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe317fff640 1 -- 172.21.15.98:0/319598089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef685000 msgr2=0x55d7ef693de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe317fff640 1 --2- 172.21.15.98:0/319598089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef685000 0x55d7ef693de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe317fff640 1 -- 172.21.15.98:0/319598089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef694730 msgr2=0x55d7ef690830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe317fff640 1 --2- 172.21.15.98:0/319598089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef694730 0x55d7ef690830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe317fff640 1 -- 172.21.15.98:0/319598089 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7ef697650 con 0x55d7ef685920 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe317fff640 1 --2- 172.21.15.98:0/319598089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef6a03f0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe300002230 tx=0x7fe300008690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cedb112a651f1a75 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.096+0000 7fe30e7fc640 1 -- 172.21.15.98:0/319598089 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe300028040 con 0x55d7ef685920 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.097+0000 7fe30e7fc640 1 -- 172.21.15.98:0/319598089 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3000084b0 con 0x55d7ef685920 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.097+0000 7fe30e7fc640 1 -- 172.21.15.98:0/319598089 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe300023e60 con 0x55d7ef685920 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.097+0000 7fe321bff740 1 -- 172.21.15.98:0/319598089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 msgr2=0x55d7ef6a03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.097+0000 7fe321bff740 1 --2- 172.21.15.98:0/319598089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef6a03f0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe300002230 tx=0x7fe300008690 comp rx=0 tx=0).stop 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.097+0000 7fe321bff740 1 -- 172.21.15.98:0/319598089 shutdown_connections 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.097+0000 7fe321bff740 1 --2- 172.21.15.98:0/319598089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef694730 0x55d7ef690830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.097+0000 7fe321bff740 1 --2- 172.21.15.98:0/319598089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef6a03f0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.097+0000 7fe321bff740 1 --2- 172.21.15.98:0/319598089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef685000 0x55d7ef693de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.097+0000 7fe321bff740 1 -- 172.21.15.98:0/319598089 >> 172.21.15.98:0/319598089 conn(0x55d7ef681650 msgr2=0x55d7ef67ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.097+0000 7fe321bff740 1 -- 172.21.15.98:0/319598089 shutdown_connections 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.097+0000 7fe321bff740 1 -- 172.21.15.98:0/319598089 wait complete. 2024-05-09T08:57:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.097+0000 7fe321bff740 1 -- start start 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe321bff740 1 Processor -- start 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe321bff740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef685000 0x55d7ef69cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe321bff740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef68df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe321bff740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef694730 0x55d7ef68e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe321bff740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3181ae020 con 0x55d7ef685920 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe321bff740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe3181ad950 con 0x55d7ef694730 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe321bff740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe3181aabb0 con 0x55d7ef685000 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe321bff740 10 client.0 _ll_register_callbacks cb 0x7fe30800de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe317fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef68df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe317fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef68df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55484/0 (socket says 172.21.15.98:55484) 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe317fff640 1 -- 172.21.15.98:0/3222409297 learned_addr learned my addr 172.21.15.98:0/3222409297 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe31cbb1640 1 --2- 172.21.15.98:0/3222409297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef685000 0x55d7ef69cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe317fff640 1 -- 172.21.15.98:0/3222409297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef685000 msgr2=0x55d7ef69cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe317fff640 1 --2- 172.21.15.98:0/3222409297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef685000 0x55d7ef69cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe31d3b2640 1 --2- 172.21.15.98:0/3222409297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef694730 0x55d7ef68e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe317fff640 1 -- 172.21.15.98:0/3222409297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef694730 msgr2=0x55d7ef68e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe317fff640 1 --2- 172.21.15.98:0/3222409297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef694730 0x55d7ef68e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe317fff640 1 -- 172.21.15.98:0/3222409297 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7ef68ec00 con 0x55d7ef685920 2024-05-09T08:57:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe317fff640 1 --2- 172.21.15.98:0/3222409297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef68df80 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fe300008b80 tx=0x7fe300004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e08c4012ad9e7d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.098+0000 7fe2f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe2f7fff640 1 -- 172.21.15.98:0/3222409297 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe300028070 con 0x55d7ef685920 2024-05-09T08:57:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe2f7fff640 1 -- 172.21.15.98:0/3222409297 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fe300018c60 con 0x55d7ef685920 2024-05-09T08:57:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe321bff740 1 -- 172.21.15.98:0/3222409297 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3000084b0 con 0x55d7ef685920 2024-05-09T08:57:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe2f7fff640 1 -- 172.21.15.98:0/3222409297 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe300018e60 con 0x55d7ef685920 2024-05-09T08:57:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe321bff740 10 client.17454 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe321bff740 1 -- 172.21.15.98:0/3222409297 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d7ef70ad20 con 0x55d7ef685920 2024-05-09T08:57:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe2f7fff640 1 -- 172.21.15.98:0/3222409297 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3000084b0 con 0x55d7ef685920 2024-05-09T08:57:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe2f7fff640 1 -- 172.21.15.98:0/3222409297 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe30000f630 con 0x55d7ef685920 2024-05-09T08:57:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe321bff740 10 client.17454 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe321bff740 10 client.17454 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe321bff740 20 client.17454 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe321bff740 1 --2- 172.21.15.98:0/3222409297 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d7ef70b400 0x55d7ef70d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe321bff740 4 client.17454 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe321bff740 1 -- 172.21.15.98:0/3222409297 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d7ef5f26c0 con 0x55d7ef70b400 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.099+0000 7fe31cbb1640 1 --2- 172.21.15.98:0/3222409297 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d7ef70b400 0x55d7ef70d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe2f7fff640 10 client.17454 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe31cbb1640 1 --2- 172.21.15.98:0/3222409297 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d7ef70b400 0x55d7ef70d820 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f7595197b4962dcb server_cookie=87bf26358dd0c2c0 in_seq=0 out_seq=0 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe2f7fff640 1 -- 172.21.15.98:0/3222409297 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d7ef5f26c0 con 0x55d7ef70b400 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe2f7fff640 10 client.17454 handle_command_reply: tid=0 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe321bff740 1 client.17454 shutdown 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe321bff740 1 -- 172.21.15.98:0/3222409297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 msgr2=0x55d7ef68df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe321bff740 1 --2- 172.21.15.98:0/3222409297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef68df80 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fe300008b80 tx=0x7fe300004d50 comp rx=0 tx=0).stop 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe321bff740 1 -- 172.21.15.98:0/3222409297 shutdown_connections 2024-05-09T08:57:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe321bff740 1 --2- 172.21.15.98:0/3222409297 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55d7ef70b400 0x55d7ef70d820 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe321bff740 1 --2- 172.21.15.98:0/3222409297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7ef694730 0x55d7ef68e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe321bff740 1 --2- 172.21.15.98:0/3222409297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7ef685920 0x55d7ef68df80 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe2f7fff640 0 client.17454 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:57:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe321bff740 1 --2- 172.21.15.98:0/3222409297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7ef685000 0x55d7ef69cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe321bff740 1 -- 172.21.15.98:0/3222409297 >> 172.21.15.98:0/3222409297 conn(0x55d7ef681650 msgr2=0x55d7ef6807b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe321bff740 1 -- 172.21.15.98:0/3222409297 shutdown_connections 2024-05-09T08:57:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe321bff740 1 -- 172.21.15.98:0/3222409297 wait complete. 2024-05-09T08:57:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.100+0000 7fe321bff740 20 client.17454 trim_cache size 0 max 16384 2024-05-09T08:57:40.106 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:57:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.101+0000 7fe31ee15640 1 -- 172.21.15.98:0/2805139983 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe3040a2d80 msgr2=0x7fe3040a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.101+0000 7fe31ee15640 1 --2- 172.21.15.98:0/2805139983 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe3040a2d80 0x7fe3040a5240 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe3100057c0 tx=0x7fe310031490 comp rx=0 tx=0).stop 2024-05-09T08:57:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.101+0000 7fe31ee15640 1 -- 172.21.15.98:0/2805139983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe318100700 msgr2=0x7fe318198f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.101+0000 7fe31ee15640 1 --2- 172.21.15.98:0/2805139983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe318100700 0x7fe318198f40 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fe30800d770 tx=0x7fe30800dc30 comp rx=0 tx=0).stop 2024-05-09T08:57:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.101+0000 7fe31ee15640 1 -- 172.21.15.98:0/2805139983 shutdown_connections 2024-05-09T08:57:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.101+0000 7fe31ee15640 1 --2- 172.21.15.98:0/2805139983 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe3040a2d80 0x7fe3040a5240 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.101+0000 7fe31ee15640 1 --2- 172.21.15.98:0/2805139983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe318106710 0x7fe3181a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.101+0000 7fe31ee15640 1 --2- 172.21.15.98:0/2805139983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3181010c0 0x7fe318199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.101+0000 7fe31ee15640 1 --2- 172.21.15.98:0/2805139983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe318100700 0x7fe318198f40 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.101+0000 7fe31ee15640 1 -- 172.21.15.98:0/2805139983 >> 172.21.15.98:0/2805139983 conn(0x7fe318077830 msgr2=0x7fe318102ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.101+0000 7fe31ee15640 1 -- 172.21.15.98:0/2805139983 shutdown_connections 2024-05-09T08:57:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.101+0000 7fe31ee15640 1 -- 172.21.15.98:0/2805139983 wait complete. 2024-05-09T08:57:40.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:57:40.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.215+0000 7f7f4a35a640 1 -- 172.21.15.98:0/4048110788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f44101800 msgr2=0x7f7f44101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.215+0000 7f7f4a35a640 1 --2- 172.21.15.98:0/4048110788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f44101800 0x7f7f44101c80 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009930 tx=0x7f7f3802a420 comp rx=0 tx=0).stop 2024-05-09T08:57:40.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.215+0000 7f7f4a35a640 1 -- 172.21.15.98:0/4048110788 shutdown_connections 2024-05-09T08:57:40.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.215+0000 7f7f4a35a640 1 --2- 172.21.15.98:0/4048110788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f441021c0 0x7f7f4410e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.215+0000 7f7f4a35a640 1 --2- 172.21.15.98:0/4048110788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f44101800 0x7f7f44101c80 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.215+0000 7f7f4a35a640 1 --2- 172.21.15.98:0/4048110788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f44107810 0x7f7f44107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.215+0000 7f7f4a35a640 1 -- 172.21.15.98:0/4048110788 >> 172.21.15.98:0/4048110788 conn(0x7f7f44078c80 msgr2=0x7f7f44079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:40.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.215+0000 7f7f4a35a640 1 -- 172.21.15.98:0/4048110788 shutdown_connections 2024-05-09T08:57:40.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.215+0000 7f7f4a35a640 1 -- 172.21.15.98:0/4048110788 wait complete. 2024-05-09T08:57:40.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.216+0000 7f7f4a35a640 1 Processor -- start 2024-05-09T08:57:40.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.216+0000 7f7f4a35a640 1 -- start start 2024-05-09T08:57:40.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.216+0000 7f7f4a35a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f44101800 0x7f7f44198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.216+0000 7f7f4a35a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441021c0 0x7f7f441986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.216+0000 7f7f4a35a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f44107810 0x7f7f4419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.216+0000 7f7f4a35a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f44111c20 con 0x7f7f441021c0 2024-05-09T08:57:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.216+0000 7f7f4a35a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7f44111aa0 con 0x7f7f44101800 2024-05-09T08:57:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.216+0000 7f7f4a35a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7f44111da0 con 0x7f7f44107810 2024-05-09T08:57:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.216+0000 7f7f437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441021c0 0x7f7f441986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.216+0000 7f7f437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441021c0 0x7f7f441986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55512/0 (socket says 172.21.15.98:55512) 2024-05-09T08:57:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.216+0000 7f7f437fe640 1 -- 172.21.15.98:0/4292866038 learned_addr learned my addr 172.21.15.98:0/4292866038 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.217+0000 7f7f43fff640 1 --2- 172.21.15.98:0/4292866038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f44101800 0x7f7f44198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.217+0000 7f7f488f7640 1 --2- 172.21.15.98:0/4292866038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f44107810 0x7f7f4419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.217+0000 7f7f437fe640 1 -- 172.21.15.98:0/4292866038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f44107810 msgr2=0x7f7f4419ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.217+0000 7f7f437fe640 1 --2- 172.21.15.98:0/4292866038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f44107810 0x7f7f4419ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.217+0000 7f7f437fe640 1 -- 172.21.15.98:0/4292866038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f44101800 msgr2=0x7f7f44198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.217+0000 7f7f437fe640 1 --2- 172.21.15.98:0/4292866038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f44101800 0x7f7f44198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.217+0000 7f7f437fe640 1 -- 172.21.15.98:0/4292866038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f4419d170 con 0x7f7f441021c0 2024-05-09T08:57:40.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.217+0000 7f7f437fe640 1 --2- 172.21.15.98:0/4292866038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441021c0 0x7f7f441986c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009230 tx=0x7f7f38004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:40.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.217+0000 7f7f417fa640 1 -- 172.21.15.98:0/4292866038 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3803f030 con 0x7f7f441021c0 2024-05-09T08:57:40.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.217+0000 7f7f4a35a640 1 -- 172.21.15.98:0/4292866038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f4419d400 con 0x7f7f441021c0 2024-05-09T08:57:40.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.217+0000 7f7f417fa640 1 -- 172.21.15.98:0/4292866038 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f38003ad0 con 0x7f7f441021c0 2024-05-09T08:57:40.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.217+0000 7f7f417fa640 1 -- 172.21.15.98:0/4292866038 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f38035070 con 0x7f7f441021c0 2024-05-09T08:57:40.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.217+0000 7f7f4a35a640 1 -- 172.21.15.98:0/4292866038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f441a4cf0 con 0x7f7f441021c0 2024-05-09T08:57:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.219+0000 7f7f417fa640 1 -- 172.21.15.98:0/4292866038 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7f38042030 con 0x7f7f441021c0 2024-05-09T08:57:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f4d067740 1 Processor -- start 2024-05-09T08:57:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f4d067740 1 -- start start 2024-05-09T08:57:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f4d067740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28330 0x561a5ab28710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f4d067740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab28c50 0x561a5ab32de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f4d067740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab33320 0x561a5ab35710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f4d067740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561a5aa8dc40 con 0x561a5ab28330 2024-05-09T08:57:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f4d067740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x561a5aa35580 con 0x561a5ab33320 2024-05-09T08:57:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f4d067740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561a5aa89c00 con 0x561a5ab28c50 2024-05-09T08:57:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f488f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28330 0x561a5ab28710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f488f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28330 0x561a5ab28710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55524/0 (socket says 172.21.15.98:55524) 2024-05-09T08:57:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f417fa640 1 --2- 172.21.15.98:0/4292866038 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f7f2c0a2d00 0x7f7f2c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f488f7640 1 -- 172.21.15.98:0/519279068 learned_addr learned my addr 172.21.15.98:0/519279068 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f43fff640 1 --2- 172.21.15.98:0/519279068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab28c50 0x561a5ab32de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f488f7640 1 --2- 172.21.15.98:0/4292866038 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f7f2c0a2d00 0x7f7f2c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f417fa640 1 -- 172.21.15.98:0/4292866038 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f7f380efe10 con 0x7f7f441021c0 2024-05-09T08:57:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f437fe640 1 --2- 172.21.15.98:0/519279068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab33320 0x561a5ab35710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f488f7640 1 -- 172.21.15.98:0/519279068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab28c50 msgr2=0x561a5ab32de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f488f7640 1 --2- 172.21.15.98:0/519279068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab28c50 0x561a5ab32de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f488f7640 1 -- 172.21.15.98:0/519279068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab33320 msgr2=0x561a5ab35710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f488f7640 1 --2- 172.21.15.98:0/519279068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab33320 0x561a5ab35710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.220+0000 7f7f488f7640 1 -- 172.21.15.98:0/519279068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a5a7a13f0 con 0x561a5ab28330 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.221+0000 7f7f488f7640 1 --2- 172.21.15.98:0/519279068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28330 0x561a5ab28710 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c0149c0 tx=0x7f7f3c014e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b887d85f5b003494 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.221+0000 7f7f2a7fc640 1 -- 172.21.15.98:0/519279068 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c01eb40 con 0x561a5ab28330 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.221+0000 7f7f2a7fc640 1 -- 172.21.15.98:0/519279068 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f3c023070 con 0x561a5ab28330 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.221+0000 7f7f2a7fc640 1 -- 172.21.15.98:0/519279068 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c013ae0 con 0x561a5ab28330 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.221+0000 7f7f4d067740 1 -- 172.21.15.98:0/519279068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28330 msgr2=0x561a5ab28710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.221+0000 7f7f488f7640 1 --2- 172.21.15.98:0/4292866038 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f7f2c0a2d00 0x7f7f2c0a51c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c022000 tx=0x7f7f3c020000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.221+0000 7f7f4d067740 1 --2- 172.21.15.98:0/519279068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28330 0x561a5ab28710 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c0149c0 tx=0x7f7f3c014e80 comp rx=0 tx=0).stop 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.221+0000 7f7f4d067740 1 -- 172.21.15.98:0/519279068 shutdown_connections 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.221+0000 7f7f4d067740 1 --2- 172.21.15.98:0/519279068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab33320 0x561a5ab35710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.221+0000 7f7f4d067740 1 --2- 172.21.15.98:0/519279068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab28c50 0x561a5ab32de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.221+0000 7f7f4d067740 1 --2- 172.21.15.98:0/519279068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28330 0x561a5ab28710 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.221+0000 7f7f4d067740 1 -- 172.21.15.98:0/519279068 >> 172.21.15.98:0/519279068 conn(0x561a5ab24650 msgr2=0x561a5ab24a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.221+0000 7f7f4d067740 1 -- 172.21.15.98:0/519279068 shutdown_connections 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.221+0000 7f7f4d067740 1 -- 172.21.15.98:0/519279068 wait complete. 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.222+0000 7f7f4d067740 1 -- start start 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.222+0000 7f7f4d067740 1 Processor -- start 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.222+0000 7f7f4d067740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28330 0x561a5ab302c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.222+0000 7f7f4d067740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28c50 0x561a5ab30800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.222+0000 7f7f4d067740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab33320 0x561a5ab30d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.222+0000 7f7f4d067740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561a5ab36110 con 0x561a5ab28330 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.222+0000 7f7f4d067740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x561a5aa48c40 con 0x561a5ab28c50 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.222+0000 7f7f4d067740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561a5ab36290 con 0x561a5ab33320 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.222+0000 7f7f4d067740 10 client.0 _ll_register_callbacks cb 0x561a5a7a0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.222+0000 7f7f43fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28330 0x561a5ab302c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.222+0000 7f7f43fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28330 0x561a5ab302c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55534/0 (socket says 172.21.15.98:55534) 2024-05-09T08:57:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.222+0000 7f7f43fff640 1 -- 172.21.15.98:0/1947342592 learned_addr learned my addr 172.21.15.98:0/1947342592 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.222+0000 7f7f437fe640 1 --2- 172.21.15.98:0/1947342592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28c50 0x561a5ab30800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f488f7640 1 --2- 172.21.15.98:0/1947342592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab33320 0x561a5ab30d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f43fff640 1 -- 172.21.15.98:0/1947342592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab33320 msgr2=0x561a5ab30d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f43fff640 1 --2- 172.21.15.98:0/1947342592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab33320 0x561a5ab30d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f43fff640 1 -- 172.21.15.98:0/1947342592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28c50 msgr2=0x561a5ab30800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f43fff640 1 --2- 172.21.15.98:0/1947342592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28c50 0x561a5ab30800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f43fff640 1 -- 172.21.15.98:0/1947342592 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a5a86b7d0 con 0x561a5ab28330 2024-05-09T08:57:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f43fff640 1 --2- 172.21.15.98:0/1947342592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28330 0x561a5ab302c0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400be70 tx=0x7f7f34012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a769c4f7883116 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f237fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f237fe640 1 -- 172.21.15.98:0/1947342592 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3401f060 con 0x561a5ab28330 2024-05-09T08:57:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f237fe640 1 -- 172.21.15.98:0/1947342592 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f7f340108a0 con 0x561a5ab28330 2024-05-09T08:57:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f4d067740 1 -- 172.21.15.98:0/1947342592 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f3c023070 con 0x561a5ab28330 2024-05-09T08:57:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f237fe640 1 -- 172.21.15.98:0/1947342592 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7f34010aa0 con 0x561a5ab28330 2024-05-09T08:57:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f4d067740 10 client.17466 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f4d067740 1 -- 172.21.15.98:0/1947342592 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561a5ab42370 con 0x561a5ab28330 2024-05-09T08:57:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f237fe640 1 -- 172.21.15.98:0/1947342592 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f3c023070 con 0x561a5ab28330 2024-05-09T08:57:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f237fe640 1 -- 172.21.15.98:0/1947342592 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f7f34019030 con 0x561a5ab28330 2024-05-09T08:57:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f4d067740 10 client.17466 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f4d067740 10 client.17466 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.223+0000 7f7f4d067740 20 client.17466 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.224+0000 7f7f4d067740 1 --2- 172.21.15.98:0/1947342592 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561a5ab43830 0x561a5ab45c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.224+0000 7f7f4d067740 4 client.17466 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:57:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.224+0000 7f7f4d067740 1 -- 172.21.15.98:0/1947342592 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a5aa9b370 con 0x561a5ab43830 2024-05-09T08:57:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.224+0000 7f7f437fe640 1 --2- 172.21.15.98:0/1947342592 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561a5ab43830 0x561a5ab45c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.224+0000 7f7f437fe640 1 --2- 172.21.15.98:0/1947342592 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561a5ab43830 0x561a5ab45c80 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60e130733ce485dc server_cookie=31a49e049672b105 in_seq=0 out_seq=0 2024-05-09T08:57:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.224+0000 7f7f237fe640 10 client.17466 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f237fe640 1 -- 172.21.15.98:0/1947342592 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561a5aa9b370 con 0x561a5ab43830 2024-05-09T08:57:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f237fe640 10 client.17466 handle_command_reply: tid=0 2024-05-09T08:57:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f4d067740 1 client.17466 shutdown 2024-05-09T08:57:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f4d067740 1 -- 172.21.15.98:0/1947342592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28330 msgr2=0x561a5ab302c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f4d067740 1 --2- 172.21.15.98:0/1947342592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28330 0x561a5ab302c0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400be70 tx=0x7f7f34012820 comp rx=0 tx=0).stop 2024-05-09T08:57:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f4d067740 1 -- 172.21.15.98:0/1947342592 shutdown_connections 2024-05-09T08:57:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f4d067740 1 --2- 172.21.15.98:0/1947342592 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561a5ab43830 0x561a5ab45c80 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f4d067740 1 --2- 172.21.15.98:0/1947342592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab33320 0x561a5ab30d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f4d067740 1 --2- 172.21.15.98:0/1947342592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28c50 0x561a5ab30800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f237fe640 0 client.17466 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f4d067740 1 --2- 172.21.15.98:0/1947342592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28330 0x561a5ab302c0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f4d067740 1 -- 172.21.15.98:0/1947342592 >> 172.21.15.98:0/1947342592 conn(0x561a5ab24650 msgr2=0x561a5ab2a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f4d067740 1 -- 172.21.15.98:0/1947342592 shutdown_connections 2024-05-09T08:57:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f4d067740 1 -- 172.21.15.98:0/1947342592 wait complete. 2024-05-09T08:57:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.225+0000 7f7f4d067740 20 client.17466 trim_cache size 0 max 16384 2024-05-09T08:57:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.238+0000 7f7f4d067740 1 Processor -- start 2024-05-09T08:57:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.238+0000 7f7f4d067740 1 -- start start 2024-05-09T08:57:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.238+0000 7f7f4d067740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28330 0x561a5ab3efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.238+0000 7f7f4d067740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab28c50 0x561a5ab2aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.238+0000 7f7f4d067740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab33320 0x561a5ab3b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.238+0000 7f7f4d067740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f441aee70 con 0x561a5ab33320 2024-05-09T08:57:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.238+0000 7f7f4d067740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7f441b09b0 con 0x561a5ab28330 2024-05-09T08:57:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.238+0000 7f7f4d067740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7f441af540 con 0x561a5ab28c50 2024-05-09T08:57:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.239+0000 7f7f488f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab33320 0x561a5ab3b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.239+0000 7f7f488f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab33320 0x561a5ab3b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55542/0 (socket says 172.21.15.98:55542) 2024-05-09T08:57:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.239+0000 7f7f488f7640 1 -- 172.21.15.98:0/302925109 learned_addr learned my addr 172.21.15.98:0/302925109 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.239+0000 7f7f437fe640 1 --2- 172.21.15.98:0/302925109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab28c50 0x561a5ab2aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.239+0000 7f7f488f7640 1 -- 172.21.15.98:0/302925109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab28c50 msgr2=0x561a5ab2aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.239+0000 7f7f488f7640 1 --2- 172.21.15.98:0/302925109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab28c50 0x561a5ab2aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.239+0000 7f7f43fff640 1 --2- 172.21.15.98:0/302925109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28330 0x561a5ab3efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.239+0000 7f7f488f7640 1 -- 172.21.15.98:0/302925109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28330 msgr2=0x561a5ab3efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.239+0000 7f7f488f7640 1 --2- 172.21.15.98:0/302925109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28330 0x561a5ab3efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.239+0000 7f7f488f7640 1 -- 172.21.15.98:0/302925109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a5ab24480 con 0x561a5ab33320 2024-05-09T08:57:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.239+0000 7f7f488f7640 1 --2- 172.21.15.98:0/302925109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab33320 0x561a5ab3b8c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c013f50 tx=0x7f7f3c012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=352261d67401c54e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.239+0000 7f7f29ffb640 1 -- 172.21.15.98:0/302925109 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c01e940 con 0x561a5ab33320 2024-05-09T08:57:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.239+0000 7f7f29ffb640 1 -- 172.21.15.98:0/302925109 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f3c023070 con 0x561a5ab33320 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.239+0000 7f7f29ffb640 1 -- 172.21.15.98:0/302925109 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c01edd0 con 0x561a5ab33320 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 -- 172.21.15.98:0/302925109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab33320 msgr2=0x561a5ab3b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 --2- 172.21.15.98:0/302925109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab33320 0x561a5ab3b8c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c013f50 tx=0x7f7f3c012040 comp rx=0 tx=0).stop 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 -- 172.21.15.98:0/302925109 shutdown_connections 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 --2- 172.21.15.98:0/302925109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab33320 0x561a5ab3b8c0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 --2- 172.21.15.98:0/302925109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab28c50 0x561a5ab2aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 --2- 172.21.15.98:0/302925109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28330 0x561a5ab3efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 -- 172.21.15.98:0/302925109 >> 172.21.15.98:0/302925109 conn(0x561a5ab24650 msgr2=0x561a5ab22f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 -- 172.21.15.98:0/302925109 shutdown_connections 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 -- 172.21.15.98:0/302925109 wait complete. 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 -- start start 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 Processor -- start 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28330 0x561a5aba7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28c50 0x561a5aba7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab33320 0x561a5aba1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f441b02e0 con 0x561a5ab28c50 2024-05-09T08:57:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7f441afc10 con 0x561a5ab28330 2024-05-09T08:57:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7f441ace70 con 0x561a5ab33320 2024-05-09T08:57:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.240+0000 7f7f4d067740 10 client.0 _ll_register_callbacks cb 0x7f7f3400b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28c50 0x561a5aba7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28c50 0x561a5aba7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55558/0 (socket says 172.21.15.98:55558) 2024-05-09T08:57:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f437fe640 1 -- 172.21.15.98:0/18116290 learned_addr learned my addr 172.21.15.98:0/18116290 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f43fff640 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28330 0x561a5aba7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f488f7640 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab33320 0x561a5aba1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f437fe640 1 -- 172.21.15.98:0/18116290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab33320 msgr2=0x561a5aba1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f437fe640 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab33320 0x561a5aba1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f437fe640 1 -- 172.21.15.98:0/18116290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28330 msgr2=0x561a5aba7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f437fe640 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28330 0x561a5aba7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f437fe640 1 -- 172.21.15.98:0/18116290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a5aba2580 con 0x561a5ab28c50 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f43fff640 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28330 0x561a5aba7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f488f7640 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab33320 0x561a5aba1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f437fe640 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28c50 0x561a5aba7a90 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f7f380ed200 tx=0x7f7f38009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85295d86a8d33083 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f237fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f237fe640 1 -- 172.21.15.98:0/18116290 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f38049040 con 0x561a5ab28c50 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f237fe640 1 -- 172.21.15.98:0/18116290 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f7f380ed3d0 con 0x561a5ab28c50 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f4d067740 1 -- 172.21.15.98:0/18116290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f3c023070 con 0x561a5ab28c50 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f237fe640 1 -- 172.21.15.98:0/18116290 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7f380c02d0 con 0x561a5ab28c50 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f4d067740 10 client.17472 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f4d067740 1 -- 172.21.15.98:0/18116290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561a5abaecb0 con 0x561a5ab28c50 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.241+0000 7f7f237fe640 1 -- 172.21.15.98:0/18116290 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f380d7070 con 0x561a5ab28c50 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.242+0000 7f7f237fe640 1 -- 172.21.15.98:0/18116290 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f7f380bf070 con 0x561a5ab28c50 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.242+0000 7f7f4d067740 10 client.17472 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.242+0000 7f7f4d067740 10 client.17472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:40.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.242+0000 7f7f4d067740 20 client.17472 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:40.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.242+0000 7f7f4d067740 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561a5abaf3e0 0x561a5abb1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:40.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.242+0000 7f7f4d067740 4 client.17472 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:57:40.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.242+0000 7f7f4d067740 1 -- 172.21.15.98:0/18116290 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561a5aa956c0 con 0x561a5abaf3e0 2024-05-09T08:57:40.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.242+0000 7f7f43fff640 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561a5abaf3e0 0x561a5abb1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:40.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.242+0000 7f7f237fe640 10 client.17472 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:40.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.242+0000 7f7f43fff640 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561a5abaf3e0 0x561a5abb1800 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=591dba6978fd4101 server_cookie=f20e6966d81df733 in_seq=0 out_seq=0 2024-05-09T08:57:40.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.242+0000 7f7f237fe640 1 -- 172.21.15.98:0/18116290 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561a5aa956c0 con 0x561a5abaf3e0 2024-05-09T08:57:40.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.242+0000 7f7f237fe640 10 client.17472 handle_command_reply: tid=0 2024-05-09T08:57:40.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.242+0000 7f7f4d067740 1 client.17472 shutdown 2024-05-09T08:57:40.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4d067740 1 -- 172.21.15.98:0/18116290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28c50 msgr2=0x561a5aba7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4d067740 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28c50 0x561a5aba7a90 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f7f380ed200 tx=0x7f7f38009260 comp rx=0 tx=0).stop 2024-05-09T08:57:40.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4d067740 1 -- 172.21.15.98:0/18116290 shutdown_connections 2024-05-09T08:57:40.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4d067740 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561a5abaf3e0 0x561a5abb1800 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4d067740 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561a5ab33320 0x561a5aba1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4d067740 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561a5ab28c50 0x561a5aba7a90 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f237fe640 0 client.17472 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:40.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4d067740 1 --2- 172.21.15.98:0/18116290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561a5ab28330 0x561a5aba7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4d067740 1 -- 172.21.15.98:0/18116290 >> 172.21.15.98:0/18116290 conn(0x561a5ab24650 msgr2=0x561a5ab237b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:40.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4d067740 1 -- 172.21.15.98:0/18116290 shutdown_connections 2024-05-09T08:57:40.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4d067740 1 -- 172.21.15.98:0/18116290 wait complete. 2024-05-09T08:57:40.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4d067740 20 client.17472 trim_cache size 0 max 16384 2024-05-09T08:57:40.251 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:57:40.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4a35a640 1 -- 172.21.15.98:0/4292866038 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f7f2c0a2d00 msgr2=0x7f7f2c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4a35a640 1 --2- 172.21.15.98:0/4292866038 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f7f2c0a2d00 0x7f7f2c0a51c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c022000 tx=0x7f7f3c020000 comp rx=0 tx=0).stop 2024-05-09T08:57:40.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4a35a640 1 -- 172.21.15.98:0/4292866038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441021c0 msgr2=0x7f7f441986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:40.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4a35a640 1 --2- 172.21.15.98:0/4292866038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441021c0 0x7f7f441986c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009230 tx=0x7f7f38004270 comp rx=0 tx=0).stop 2024-05-09T08:57:40.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4a35a640 1 -- 172.21.15.98:0/4292866038 shutdown_connections 2024-05-09T08:57:40.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4a35a640 1 --2- 172.21.15.98:0/4292866038 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f7f2c0a2d00 0x7f7f2c0a51c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4a35a640 1 --2- 172.21.15.98:0/4292866038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f44107810 0x7f7f4419ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4a35a640 1 --2- 172.21.15.98:0/4292866038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441021c0 0x7f7f441986c0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4a35a640 1 --2- 172.21.15.98:0/4292866038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f44101800 0x7f7f44198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:40.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.243+0000 7f7f4a35a640 1 -- 172.21.15.98:0/4292866038 >> 172.21.15.98:0/4292866038 conn(0x7f7f44078c80 msgr2=0x7f7f440ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:40.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.244+0000 7f7f4a35a640 1 -- 172.21.15.98:0/4292866038 shutdown_connections 2024-05-09T08:57:40.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:40.244+0000 7f7f4a35a640 1 -- 172.21.15.98:0/4292866038 wait complete. 2024-05-09T08:57:40.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:40 smithi142 ceph-mon[34625]: pgmap v217: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 159 MiB/s wr, 1.33k op/s 2024-05-09T08:57:40.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:40 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/399187889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:57:40.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:40 smithi105 ceph-mon[34498]: pgmap v217: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 159 MiB/s wr, 1.33k op/s 2024-05-09T08:57:40.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:40 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/399187889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:57:41.254 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:57:41.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:57:41.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:57:41.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.381+0000 7fa4c1b22640 1 -- 172.21.15.98:0/2647079925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4bc069ef0 msgr2=0x7fa4bc06a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.381+0000 7fa4c1b22640 1 --2- 172.21.15.98:0/2647079925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4bc069ef0 0x7fa4bc06a370 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009930 tx=0x7fa4a402a420 comp rx=0 tx=0).stop 2024-05-09T08:57:41.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.381+0000 7fa4c1b22640 1 -- 172.21.15.98:0/2647079925 shutdown_connections 2024-05-09T08:57:41.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.381+0000 7fa4c1b22640 1 --2- 172.21.15.98:0/2647079925 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4bc06a8b0 0x7fa4bc10e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.381+0000 7fa4c1b22640 1 --2- 172.21.15.98:0/2647079925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4bc069ef0 0x7fa4bc06a370 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.381+0000 7fa4c1b22640 1 --2- 172.21.15.98:0/2647079925 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa4bc107820 0x7fa4bc107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.381+0000 7fa4c1b22640 1 -- 172.21.15.98:0/2647079925 >> 172.21.15.98:0/2647079925 conn(0x7fa4bc077790 msgr2=0x7fa4bc077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.381+0000 7fa4c1b22640 1 -- 172.21.15.98:0/2647079925 shutdown_connections 2024-05-09T08:57:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.381+0000 7fa4c1b22640 1 -- 172.21.15.98:0/2647079925 wait complete. 2024-05-09T08:57:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.381+0000 7fa4c1b22640 1 Processor -- start 2024-05-09T08:57:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.381+0000 7fa4c1b22640 1 -- start start 2024-05-09T08:57:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4c1b22640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4bc069ef0 0x7fa4bc1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4c1b22640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa4bc06a8b0 0x7fa4bc198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4c1b22640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4bc107820 0x7fa4bc19cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4c1b22640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc111cb0 con 0x7fa4bc107820 2024-05-09T08:57:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4c1b22640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc111b30 con 0x7fa4bc069ef0 2024-05-09T08:57:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4c1b22640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc111e30 con 0x7fa4bc06a8b0 2024-05-09T08:57:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4bbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4bc107820 0x7fa4bc19cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4bbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4bc107820 0x7fa4bc19cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55574/0 (socket says 172.21.15.98:55574) 2024-05-09T08:57:41.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4bbfff640 1 -- 172.21.15.98:0/583148243 learned_addr learned my addr 172.21.15.98:0/583148243 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:41.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4bb7fe640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4bc069ef0 0x7fa4bc1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa4bc06a8b0 0x7fa4bc198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4bbfff640 1 -- 172.21.15.98:0/583148243 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa4bc06a8b0 msgr2=0x7fa4bc198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4bbfff640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa4bc06a8b0 0x7fa4bc198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4bbfff640 1 -- 172.21.15.98:0/583148243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4bc069ef0 msgr2=0x7fa4bc1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4bbfff640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4bc069ef0 0x7fa4bc1981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4bbfff640 1 -- 172.21.15.98:0/583148243 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4bc19d1e0 con 0x7fa4bc107820 2024-05-09T08:57:41.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4bb7fe640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4bc069ef0 0x7fa4bc1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:41.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa4bc06a8b0 0x7fa4bc198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:41.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4bbfff640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4bc107820 0x7fa4bc19cab0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400b970 tx=0x7fa4b400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4b97fa640 1 -- 172.21.15.98:0/583148243 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4019070 con 0x7fa4bc107820 2024-05-09T08:57:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4b97fa640 1 -- 172.21.15.98:0/583148243 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4b400cb40 con 0x7fa4bc107820 2024-05-09T08:57:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4b97fa640 1 -- 172.21.15.98:0/583148243 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4007650 con 0x7fa4bc107820 2024-05-09T08:57:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.382+0000 7fa4c1b22640 1 -- 172.21.15.98:0/583148243 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4bc19d4d0 con 0x7fa4bc107820 2024-05-09T08:57:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.383+0000 7fa4c1b22640 1 -- 172.21.15.98:0/583148243 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4bc1a4d60 con 0x7fa4bc107820 2024-05-09T08:57:41.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.385+0000 7fa4b97fa640 1 -- 172.21.15.98:0/583148243 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa4b401c070 con 0x7fa4bc107820 2024-05-09T08:57:41.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.385+0000 7fa4c4833740 1 Processor -- start 2024-05-09T08:57:41.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.385+0000 7fa4c4833740 1 -- start start 2024-05-09T08:57:41.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.385+0000 7fa4c4833740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207d0b0 0x5637c207b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.385+0000 7fa4c4833740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c207b9c0 0x5637c207be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.385+0000 7fa4c4833740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c2086740 0x5637c2088b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.385+0000 7fa4c4833740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5637c1fe0990 con 0x5637c2086740 2024-05-09T08:57:41.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.385+0000 7fa4c4833740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5637c1f885b0 con 0x5637c207d0b0 2024-05-09T08:57:41.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.385+0000 7fa4c4833740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5637c1fdc950 con 0x5637c207b9c0 2024-05-09T08:57:41.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.385+0000 7fa4b3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c2086740 0x5637c2088b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4b3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c2086740 0x5637c2088b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55580/0 (socket says 172.21.15.98:55580) 2024-05-09T08:57:41.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4b3fff640 1 -- 172.21.15.98:0/350375347 learned_addr learned my addr 172.21.15.98:0/350375347 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:41.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4b97fa640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa4a80a2d00 0x7fa4a80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4bb7fe640 1 --2- 172.21.15.98:0/350375347 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c207b9c0 0x5637c207be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/350375347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207d0b0 0x5637c207b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4bbfff640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa4a80a2d00 0x7fa4a80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4b3fff640 1 -- 172.21.15.98:0/350375347 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c207b9c0 msgr2=0x5637c207be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/350375347 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c207b9c0 0x5637c207be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4b3fff640 1 -- 172.21.15.98:0/350375347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207d0b0 msgr2=0x5637c207b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/350375347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207d0b0 0x5637c207b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4b3fff640 1 -- 172.21.15.98:0/350375347 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637c1dbc7e0 con 0x5637c2086740 2024-05-09T08:57:41.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4bb7fe640 1 --2- 172.21.15.98:0/350375347 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c207b9c0 0x5637c207be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4b97fa640 1 -- 172.21.15.98:0/583148243 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fa4b40cede0 con 0x7fa4bc107820 2024-05-09T08:57:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/350375347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c2086740 0x5637c2088b30 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4005e90 tx=0x7fa4a4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5cc1fd4a153aef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4b1ffb640 1 -- 172.21.15.98:0/350375347 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4045070 con 0x5637c2086740 2024-05-09T08:57:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4bbfff640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa4a80a2d00 0x7fa4a80a51c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa4b409ad00 tx=0x7fa4b406cfc0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4b1ffb640 1 -- 172.21.15.98:0/350375347 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4a4031980 con 0x5637c2086740 2024-05-09T08:57:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.386+0000 7fa4b1ffb640 1 -- 172.21.15.98:0/350375347 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4035040 con 0x5637c2086740 2024-05-09T08:57:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.387+0000 7fa4c4833740 1 -- 172.21.15.98:0/350375347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c2086740 msgr2=0x5637c2088b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.387+0000 7fa4c4833740 1 --2- 172.21.15.98:0/350375347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c2086740 0x5637c2088b30 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4005e90 tx=0x7fa4a4000bc0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.387+0000 7fa4c4833740 1 -- 172.21.15.98:0/350375347 shutdown_connections 2024-05-09T08:57:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.387+0000 7fa4c4833740 1 --2- 172.21.15.98:0/350375347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c2086740 0x5637c2088b30 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.387+0000 7fa4c4833740 1 --2- 172.21.15.98:0/350375347 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c207b9c0 0x5637c207be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.387+0000 7fa4c4833740 1 --2- 172.21.15.98:0/350375347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207d0b0 0x5637c207b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.387+0000 7fa4c4833740 1 -- 172.21.15.98:0/350375347 >> 172.21.15.98:0/350375347 conn(0x5637c20440a0 msgr2=0x5637c2077f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.387+0000 7fa4c4833740 1 -- 172.21.15.98:0/350375347 shutdown_connections 2024-05-09T08:57:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.387+0000 7fa4c4833740 1 -- 172.21.15.98:0/350375347 wait complete. 2024-05-09T08:57:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.387+0000 7fa4c4833740 1 -- start start 2024-05-09T08:57:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4c4833740 1 Processor -- start 2024-05-09T08:57:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4c4833740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c207b9c0 0x5637c2083710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4c4833740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207d0b0 0x5637c2083c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4c4833740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c2086740 0x5637c2084190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4c4833740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5637c2089570 con 0x5637c207d0b0 2024-05-09T08:57:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4c4833740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5637c1f9bb40 con 0x5637c2086740 2024-05-09T08:57:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4c4833740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5637c20896f0 con 0x5637c207b9c0 2024-05-09T08:57:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4c4833740 10 client.0 _ll_register_callbacks cb 0x5637c1f900f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4bb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207d0b0 0x5637c2083c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4bb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207d0b0 0x5637c2083c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55584/0 (socket says 172.21.15.98:55584) 2024-05-09T08:57:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4bb7fe640 1 -- 172.21.15.98:0/1316376764 learned_addr learned my addr 172.21.15.98:0/1316376764 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/1316376764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c2086740 0x5637c2084190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/1316376764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c207b9c0 0x5637c2083710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4bb7fe640 1 -- 172.21.15.98:0/1316376764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c207b9c0 msgr2=0x5637c2083710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4bb7fe640 1 --2- 172.21.15.98:0/1316376764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c207b9c0 0x5637c2083710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4bb7fe640 1 -- 172.21.15.98:0/1316376764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c2086740 msgr2=0x5637c2084190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4bb7fe640 1 --2- 172.21.15.98:0/1316376764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c2086740 0x5637c2084190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4bb7fe640 1 -- 172.21.15.98:0/1316376764 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637c1e1cf70 con 0x5637c207d0b0 2024-05-09T08:57:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.388+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/1316376764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c2086740 0x5637c2084190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa4bb7fe640 1 --2- 172.21.15.98:0/1316376764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207d0b0 0x5637c2083c50 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00e2d0 tx=0x7fa4ac00e790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3cadd0207bf5250 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa49affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa49affd640 1 -- 172.21.15.98:0/1316376764 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac018020 con 0x5637c207d0b0 2024-05-09T08:57:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa49affd640 1 -- 172.21.15.98:0/1316376764 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fa4ac013660 con 0x5637c207d0b0 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa4c4833740 1 -- 172.21.15.98:0/1316376764 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4a4031980 con 0x5637c207d0b0 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa49affd640 1 -- 172.21.15.98:0/1316376764 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa4ac023050 con 0x5637c207d0b0 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa4c4833740 10 client.17484 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa4c4833740 1 -- 172.21.15.98:0/1316376764 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5637c1cb4170 con 0x5637c207d0b0 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa49affd640 1 -- 172.21.15.98:0/1316376764 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4a4031980 con 0x5637c207d0b0 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa49affd640 1 -- 172.21.15.98:0/1316376764 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa4ac014b60 con 0x5637c207d0b0 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa4c4833740 10 client.17484 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa4c4833740 10 client.17484 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa4c4833740 20 client.17484 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa4c4833740 1 --2- 172.21.15.98:0/1316376764 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5637c2096940 0x5637c2098da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa4c4833740 4 client.17484 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.389+0000 7fa4c4833740 1 -- 172.21.15.98:0/1316376764 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637c1fee0c0 con 0x5637c2096940 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.390+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/1316376764 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5637c2096940 0x5637c2098da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.390+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/1316376764 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5637c2096940 0x5637c2098da0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b0e524e6643bca9 server_cookie=e081e97ce00bebe9 in_seq=0 out_seq=0 2024-05-09T08:57:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.390+0000 7fa49affd640 10 client.17484 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa49affd640 1 -- 172.21.15.98:0/1316376764 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5637c1fee0c0 con 0x5637c2096940 2024-05-09T08:57:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa49affd640 10 client.17484 handle_command_reply: tid=0 2024-05-09T08:57:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa4c4833740 1 client.17484 shutdown 2024-05-09T08:57:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa4c4833740 1 -- 172.21.15.98:0/1316376764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207d0b0 msgr2=0x5637c2083c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa4c4833740 1 --2- 172.21.15.98:0/1316376764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207d0b0 0x5637c2083c50 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00e2d0 tx=0x7fa4ac00e790 comp rx=0 tx=0).stop 2024-05-09T08:57:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa4c4833740 1 -- 172.21.15.98:0/1316376764 shutdown_connections 2024-05-09T08:57:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa4c4833740 1 --2- 172.21.15.98:0/1316376764 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5637c2096940 0x5637c2098da0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa4c4833740 1 --2- 172.21.15.98:0/1316376764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c2086740 0x5637c2084190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa4c4833740 1 --2- 172.21.15.98:0/1316376764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207d0b0 0x5637c2083c50 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa49affd640 0 client.17484 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa4c4833740 1 --2- 172.21.15.98:0/1316376764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c207b9c0 0x5637c2083710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa4c4833740 1 -- 172.21.15.98:0/1316376764 >> 172.21.15.98:0/1316376764 conn(0x5637c20440a0 msgr2=0x5637c207dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa4c4833740 1 -- 172.21.15.98:0/1316376764 shutdown_connections 2024-05-09T08:57:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa4c4833740 1 -- 172.21.15.98:0/1316376764 wait complete. 2024-05-09T08:57:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.391+0000 7fa4c4833740 20 client.17484 trim_cache size 0 max 16384 2024-05-09T08:57:41.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.409+0000 7fa4c4833740 1 Processor -- start 2024-05-09T08:57:41.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.409+0000 7fa4c4833740 1 -- start start 2024-05-09T08:57:41.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.409+0000 7fa4c4833740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207b9c0 0x5637c2092470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.409+0000 7fa4c4833740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207d0b0 0x5637c2076f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.409+0000 7fa4c4833740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c2086740 0x5637c209e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.409+0000 7fa4c4833740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc1aeef0 con 0x5637c207b9c0 2024-05-09T08:57:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.409+0000 7fa4c4833740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc1b0a30 con 0x5637c207d0b0 2024-05-09T08:57:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.409+0000 7fa4c4833740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc1af5c0 con 0x5637c2086740 2024-05-09T08:57:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.409+0000 7fa4b3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207b9c0 0x5637c2092470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.409+0000 7fa4b3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207b9c0 0x5637c2092470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55594/0 (socket says 172.21.15.98:55594) 2024-05-09T08:57:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.409+0000 7fa4b3fff640 1 -- 172.21.15.98:0/2965813587 learned_addr learned my addr 172.21.15.98:0/2965813587 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.410+0000 7fa4bb7fe640 1 --2- 172.21.15.98:0/2965813587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207d0b0 0x5637c2076f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.410+0000 7fa4b3fff640 1 -- 172.21.15.98:0/2965813587 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c2086740 msgr2=0x5637c209e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.410+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/2965813587 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c2086740 0x5637c209e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.410+0000 7fa4b3fff640 1 -- 172.21.15.98:0/2965813587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207d0b0 msgr2=0x5637c2076f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.410+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/2965813587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207d0b0 0x5637c2076f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.410+0000 7fa4b3fff640 1 -- 172.21.15.98:0/2965813587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637c2076470 con 0x5637c207b9c0 2024-05-09T08:57:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.410+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/2965813587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207b9c0 0x5637c2092470 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4043040 tx=0x7fa4a4031b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a210248b1b6c307 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.410+0000 7fa4b17fa640 1 -- 172.21.15.98:0/2965813587 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4055070 con 0x5637c207b9c0 2024-05-09T08:57:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.410+0000 7fa4b17fa640 1 -- 172.21.15.98:0/2965813587 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4a404fe50 con 0x5637c207b9c0 2024-05-09T08:57:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.410+0000 7fa4b17fa640 1 -- 172.21.15.98:0/2965813587 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4035040 con 0x5637c207b9c0 2024-05-09T08:57:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 -- 172.21.15.98:0/2965813587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207b9c0 msgr2=0x5637c2092470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 --2- 172.21.15.98:0/2965813587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207b9c0 0x5637c2092470 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4043040 tx=0x7fa4a4031b30 comp rx=0 tx=0).stop 2024-05-09T08:57:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 -- 172.21.15.98:0/2965813587 shutdown_connections 2024-05-09T08:57:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 --2- 172.21.15.98:0/2965813587 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c2086740 0x5637c209e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 --2- 172.21.15.98:0/2965813587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207d0b0 0x5637c2076f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 --2- 172.21.15.98:0/2965813587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207b9c0 0x5637c2092470 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 -- 172.21.15.98:0/2965813587 >> 172.21.15.98:0/2965813587 conn(0x5637c20440a0 msgr2=0x5637c20767f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 -- 172.21.15.98:0/2965813587 shutdown_connections 2024-05-09T08:57:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 -- 172.21.15.98:0/2965813587 wait complete. 2024-05-09T08:57:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 -- start start 2024-05-09T08:57:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 Processor -- start 2024-05-09T08:57:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207b9c0 0x5637c208e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207d0b0 0x5637c208eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c2086740 0x5637c208b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc1b0360 con 0x5637c207d0b0 2024-05-09T08:57:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc1afc90 con 0x5637c207b9c0 2024-05-09T08:57:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc1acef0 con 0x5637c2086740 2024-05-09T08:57:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.411+0000 7fa4c4833740 10 client.0 _ll_register_callbacks cb 0x7fa4ac00c170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4bb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207d0b0 0x5637c208eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4bb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207d0b0 0x5637c208eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55608/0 (socket says 172.21.15.98:55608) 2024-05-09T08:57:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4bb7fe640 1 -- 172.21.15.98:0/1521978027 learned_addr learned my addr 172.21.15.98:0/1521978027 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/1521978027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c2086740 0x5637c208b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/1521978027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207b9c0 0x5637c208e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4bb7fe640 1 -- 172.21.15.98:0/1521978027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c2086740 msgr2=0x5637c208b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4bb7fe640 1 --2- 172.21.15.98:0/1521978027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c2086740 0x5637c208b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4bb7fe640 1 -- 172.21.15.98:0/1521978027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207b9c0 msgr2=0x5637c208e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4bb7fe640 1 --2- 172.21.15.98:0/1521978027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207b9c0 0x5637c208e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4bb7fe640 1 -- 172.21.15.98:0/1521978027 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637c208c040 con 0x5637c207d0b0 2024-05-09T08:57:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/1521978027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207b9c0 0x5637c208e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4bb7fe640 1 --2- 172.21.15.98:0/1521978027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207d0b0 0x5637c208eec0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac014f50 tx=0x7fa4ac006da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73e5b4e20c2bbd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa49affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa49affd640 1 -- 172.21.15.98:0/1521978027 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac02d070 con 0x5637c207d0b0 2024-05-09T08:57:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa49affd640 1 -- 172.21.15.98:0/1521978027 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fa4ac012b70 con 0x5637c207d0b0 2024-05-09T08:57:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4c4833740 1 -- 172.21.15.98:0/1521978027 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4a404fe50 con 0x5637c207d0b0 2024-05-09T08:57:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa49affd640 1 -- 172.21.15.98:0/1521978027 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa4ac018040 con 0x5637c207d0b0 2024-05-09T08:57:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4c4833740 10 client.17490 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa4c4833740 1 -- 172.21.15.98:0/1521978027 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5637c20fd910 con 0x5637c207d0b0 2024-05-09T08:57:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.412+0000 7fa49affd640 1 -- 172.21.15.98:0/1521978027 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4a404fe50 con 0x5637c207d0b0 2024-05-09T08:57:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.413+0000 7fa49affd640 1 -- 172.21.15.98:0/1521978027 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa4ac030020 con 0x5637c207d0b0 2024-05-09T08:57:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.413+0000 7fa4c4833740 10 client.17490 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.413+0000 7fa4c4833740 10 client.17490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.413+0000 7fa4c4833740 20 client.17490 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.413+0000 7fa4c4833740 1 --2- 172.21.15.98:0/1521978027 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5637c20fe0b0 0x5637c21004d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.413+0000 7fa4c4833740 4 client.17490 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:57:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.413+0000 7fa4c4833740 1 -- 172.21.15.98:0/1521978027 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5637c1fe8410 con 0x5637c20fe0b0 2024-05-09T08:57:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.413+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/1521978027 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5637c20fe0b0 0x5637c21004d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.413+0000 7fa49affd640 10 client.17490 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.413+0000 7fa4b3fff640 1 --2- 172.21.15.98:0/1521978027 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5637c20fe0b0 0x5637c21004d0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=256d4219ba6ba3c server_cookie=9162b9a68b8c3e4d in_seq=0 out_seq=0 2024-05-09T08:57:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa49affd640 1 -- 172.21.15.98:0/1521978027 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5637c1fe8410 con 0x5637c20fe0b0 2024-05-09T08:57:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa49affd640 10 client.17490 handle_command_reply: tid=0 2024-05-09T08:57:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa4c4833740 1 client.17490 shutdown 2024-05-09T08:57:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa4c4833740 1 -- 172.21.15.98:0/1521978027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207d0b0 msgr2=0x5637c208eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa4c4833740 1 --2- 172.21.15.98:0/1521978027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207d0b0 0x5637c208eec0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac014f50 tx=0x7fa4ac006da0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa4c4833740 1 -- 172.21.15.98:0/1521978027 shutdown_connections 2024-05-09T08:57:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa4c4833740 1 --2- 172.21.15.98:0/1521978027 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5637c20fe0b0 0x5637c21004d0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa4c4833740 1 --2- 172.21.15.98:0/1521978027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5637c2086740 0x5637c208b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa4c4833740 1 --2- 172.21.15.98:0/1521978027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637c207d0b0 0x5637c208eec0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa4c4833740 1 --2- 172.21.15.98:0/1521978027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5637c207b9c0 0x5637c208e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa4c4833740 1 -- 172.21.15.98:0/1521978027 >> 172.21.15.98:0/1521978027 conn(0x5637c20440a0 msgr2=0x5637c2077080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:41.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa49affd640 0 client.17490 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:41.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa4c4833740 1 -- 172.21.15.98:0/1521978027 shutdown_connections 2024-05-09T08:57:41.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa4c4833740 1 -- 172.21.15.98:0/1521978027 wait complete. 2024-05-09T08:57:41.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.415+0000 7fa4c4833740 20 client.17490 trim_cache size 0 max 16384 2024-05-09T08:57:41.421 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:57:41.421 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:57:41.421 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "cb87b23f-eb73-4650-b3ac-a2babd37a208", 2024-05-09T08:57:41.421 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:57:41.421 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:57:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.416+0000 7fa4c1b22640 1 -- 172.21.15.98:0/583148243 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa4a80a2d00 msgr2=0x7fa4a80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.416+0000 7fa4c1b22640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa4a80a2d00 0x7fa4a80a51c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa4b409ad00 tx=0x7fa4b406cfc0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.416+0000 7fa4c1b22640 1 -- 172.21.15.98:0/583148243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4bc107820 msgr2=0x7fa4bc19cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.416+0000 7fa4c1b22640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4bc107820 0x7fa4bc19cab0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400b970 tx=0x7fa4b400be30 comp rx=0 tx=0).stop 2024-05-09T08:57:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.416+0000 7fa4c1b22640 1 -- 172.21.15.98:0/583148243 shutdown_connections 2024-05-09T08:57:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.416+0000 7fa4c1b22640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa4a80a2d00 0x7fa4a80a51c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.416+0000 7fa4c1b22640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4bc107820 0x7fa4bc19cab0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.416+0000 7fa4c1b22640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa4bc06a8b0 0x7fa4bc198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.416+0000 7fa4c1b22640 1 --2- 172.21.15.98:0/583148243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4bc069ef0 0x7fa4bc1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.416+0000 7fa4c1b22640 1 -- 172.21.15.98:0/583148243 >> 172.21.15.98:0/583148243 conn(0x7fa4bc077790 msgr2=0x7fa4bc0fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.416+0000 7fa4c1b22640 1 -- 172.21.15.98:0/583148243 shutdown_connections 2024-05-09T08:57:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.416+0000 7fa4c1b22640 1 -- 172.21.15.98:0/583148243 wait complete. 2024-05-09T08:57:41.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:57:41.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:57:41.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.532+0000 7f1bbe1ed640 1 -- 172.21.15.98:0/820563567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb8106710 msgr2=0x7f1bb8106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.532+0000 7f1bbe1ed640 1 --2- 172.21.15.98:0/820563567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb8106710 0x7f1bb8106af0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009930 tx=0x7f1ba802a420 comp rx=0 tx=0).stop 2024-05-09T08:57:41.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.532+0000 7f1bbe1ed640 1 -- 172.21.15.98:0/820563567 shutdown_connections 2024-05-09T08:57:41.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.532+0000 7f1bbe1ed640 1 --2- 172.21.15.98:0/820563567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bb81010c0 0x7f1bb810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.532+0000 7f1bbe1ed640 1 --2- 172.21.15.98:0/820563567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bb8100700 0x7f1bb8100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.532+0000 7f1bbe1ed640 1 --2- 172.21.15.98:0/820563567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb8106710 0x7f1bb8106af0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.532+0000 7f1bbe1ed640 1 -- 172.21.15.98:0/820563567 >> 172.21.15.98:0/820563567 conn(0x7f1bb8077830 msgr2=0x7f1bb8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:41.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.532+0000 7f1bbe1ed640 1 -- 172.21.15.98:0/820563567 shutdown_connections 2024-05-09T08:57:41.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.532+0000 7f1bbe1ed640 1 -- 172.21.15.98:0/820563567 wait complete. 2024-05-09T08:57:41.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bbe1ed640 1 Processor -- start 2024-05-09T08:57:41.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bbe1ed640 1 -- start start 2024-05-09T08:57:41.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bbe1ed640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bb8100700 0x7f1bb8198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bbe1ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bb81010c0 0x7f1bb8198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bbe1ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb8106710 0x7f1bb819c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bbe1ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb8110b20 con 0x7f1bb8106710 2024-05-09T08:57:41.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bbe1ed640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81109a0 con 0x7f1bb81010c0 2024-05-09T08:57:41.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bbe1ed640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb8110ca0 con 0x7f1bb8100700 2024-05-09T08:57:41.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bb7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb8106710 0x7f1bb819c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bb7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb8106710 0x7f1bb819c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55628/0 (socket says 172.21.15.98:55628) 2024-05-09T08:57:41.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bb7fff640 1 -- 172.21.15.98:0/4108845092 learned_addr learned my addr 172.21.15.98:0/4108845092 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:41.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/4108845092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bb81010c0 0x7f1bb8198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bb77fe640 1 --2- 172.21.15.98:0/4108845092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bb8100700 0x7f1bb8198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bb7fff640 1 -- 172.21.15.98:0/4108845092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bb8100700 msgr2=0x7f1bb8198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bb7fff640 1 --2- 172.21.15.98:0/4108845092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bb8100700 0x7f1bb8198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bb7fff640 1 -- 172.21.15.98:0/4108845092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bb81010c0 msgr2=0x7f1bb8198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bb7fff640 1 --2- 172.21.15.98:0/4108845092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bb81010c0 0x7f1bb8198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.533+0000 7f1bb7fff640 1 -- 172.21.15.98:0/4108845092 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb819d0f0 con 0x7f1bb8106710 2024-05-09T08:57:41.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.534+0000 7f1bb7fff640 1 --2- 172.21.15.98:0/4108845092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb8106710 0x7f1bb819c9c0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000cad0 tx=0x7f1bb000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:41.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.534+0000 7f1bb4ff9640 1 -- 172.21.15.98:0/4108845092 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb000ae20 con 0x7f1bb8106710 2024-05-09T08:57:41.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.534+0000 7f1bb4ff9640 1 -- 172.21.15.98:0/4108845092 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bb0005020 con 0x7f1bb8106710 2024-05-09T08:57:41.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.534+0000 7f1bbe1ed640 1 -- 172.21.15.98:0/4108845092 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb819d3e0 con 0x7f1bb8106710 2024-05-09T08:57:41.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.534+0000 7f1bb4ff9640 1 -- 172.21.15.98:0/4108845092 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb000f6f0 con 0x7f1bb8106710 2024-05-09T08:57:41.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.534+0000 7f1bbe1ed640 1 -- 172.21.15.98:0/4108845092 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb81a4c70 con 0x7f1bb8106710 2024-05-09T08:57:41.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.536+0000 7f1bb4ff9640 1 -- 172.21.15.98:0/4108845092 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1bb00051d0 con 0x7f1bb8106710 2024-05-09T08:57:41.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.536+0000 7f1bc0fce740 1 Processor -- start 2024-05-09T08:57:41.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bc0fce740 1 -- start start 2024-05-09T08:57:41.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bc0fce740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236180 0x563c2e236560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bc0fce740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e236aa0 0x563c2e245140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bc0fce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e247a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bc0fce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563c2e19bd00 con 0x563c2e245680 2024-05-09T08:57:41.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bc0fce740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563c2e1434c0 con 0x563c2e236180 2024-05-09T08:57:41.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bc0fce740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563c2e197cc0 con 0x563c2e236aa0 2024-05-09T08:57:41.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bb6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e247a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bb6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e247a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55636/0 (socket says 172.21.15.98:55636) 2024-05-09T08:57:41.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/1516700455 learned_addr learned my addr 172.21.15.98:0/1516700455 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:41.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bb77fe640 1 --2- 172.21.15.98:0/1516700455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e236aa0 0x563c2e245140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bb4ff9640 1 --2- 172.21.15.98:0/4108845092 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1ba00a2d80 0x7f1ba00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/1516700455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236180 0x563c2e236560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/1516700455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e236aa0 msgr2=0x563c2e245140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/1516700455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e236aa0 0x563c2e245140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bb7fff640 1 --2- 172.21.15.98:0/4108845092 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1ba00a2d80 0x7f1ba00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/1516700455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236180 msgr2=0x563c2e236560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/1516700455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236180 0x563c2e236560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/1516700455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c2df797d0 con 0x563c2e245680 2024-05-09T08:57:41.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bb4ff9640 1 -- 172.21.15.98:0/4108845092 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f1bb00cf820 con 0x7f1bb8106710 2024-05-09T08:57:41.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.537+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/1516700455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e247a70 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f1bac013550 tx=0x7f1bac00a770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e96cfac64170f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:41.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.538+0000 7f1b9dffb640 1 -- 172.21.15.98:0/1516700455 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac019030 con 0x563c2e245680 2024-05-09T08:57:41.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.538+0000 7f1b9dffb640 1 -- 172.21.15.98:0/1516700455 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bac005500 con 0x563c2e245680 2024-05-09T08:57:41.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.538+0000 7f1b9dffb640 1 -- 172.21.15.98:0/1516700455 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac004d90 con 0x563c2e245680 2024-05-09T08:57:41.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.538+0000 7f1bc0fce740 1 -- 172.21.15.98:0/1516700455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 msgr2=0x563c2e247a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.538+0000 7f1bb7fff640 1 --2- 172.21.15.98:0/4108845092 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1ba00a2d80 0x7f1ba00a5240 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1bb009b8c0 tx=0x7f1bb006d8f0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:41.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.538+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/1516700455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e247a70 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f1bac013550 tx=0x7f1bac00a770 comp rx=0 tx=0).stop 2024-05-09T08:57:41.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.538+0000 7f1bc0fce740 1 -- 172.21.15.98:0/1516700455 shutdown_connections 2024-05-09T08:57:41.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.538+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/1516700455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e247a70 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.538+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/1516700455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e236aa0 0x563c2e245140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.538+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/1516700455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236180 0x563c2e236560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.538+0000 7f1bc0fce740 1 -- 172.21.15.98:0/1516700455 >> 172.21.15.98:0/1516700455 conn(0x563c2e2325a0 msgr2=0x563c2e232980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:41.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.538+0000 7f1bc0fce740 1 -- 172.21.15.98:0/1516700455 shutdown_connections 2024-05-09T08:57:41.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.538+0000 7f1bc0fce740 1 -- 172.21.15.98:0/1516700455 wait complete. 2024-05-09T08:57:41.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.539+0000 7f1bc0fce740 1 -- start start 2024-05-09T08:57:41.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.539+0000 7f1bc0fce740 1 Processor -- start 2024-05-09T08:57:41.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.539+0000 7f1bc0fce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e236180 0x563c2e242620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.539+0000 7f1bc0fce740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236aa0 0x563c2e242b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.539+0000 7f1bc0fce740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e245680 0x563c2e2430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.539+0000 7f1bc0fce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563c2e248440 con 0x563c2e236180 2024-05-09T08:57:41.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.539+0000 7f1bc0fce740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563c2e156b90 con 0x563c2e236aa0 2024-05-09T08:57:41.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.539+0000 7f1bc0fce740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563c2e2485c0 con 0x563c2e245680 2024-05-09T08:57:41.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.539+0000 7f1bc0fce740 10 client.0 _ll_register_callbacks cb 0x563c2e192da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:41.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.539+0000 7f1bb6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e236180 0x563c2e242620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.539+0000 7f1bb6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e236180 0x563c2e242620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55650/0 (socket says 172.21.15.98:55650) 2024-05-09T08:57:41.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.539+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/896736070 learned_addr learned my addr 172.21.15.98:0/896736070 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:41.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.539+0000 7f1bb77fe640 1 --2- 172.21.15.98:0/896736070 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236aa0 0x563c2e242b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/896736070 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e245680 msgr2=0x563c2e2430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/896736070 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e245680 0x563c2e2430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/896736070 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236aa0 msgr2=0x563c2e242b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/896736070 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236aa0 0x563c2e242b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/896736070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c2deaf3f0 con 0x563c2e236180 2024-05-09T08:57:41.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/896736070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e236180 0x563c2e242620 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f1bac011390 tx=0x7f1bac005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8cbf019c70beaf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:41.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1b96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:41.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1b96ffd640 1 -- 172.21.15.98:0/896736070 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac019040 con 0x563c2e236180 2024-05-09T08:57:41.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1b96ffd640 1 -- 172.21.15.98:0/896736070 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f1bac004b10 con 0x563c2e236180 2024-05-09T08:57:41.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1bc0fce740 1 -- 172.21.15.98:0/896736070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bac005500 con 0x563c2e236180 2024-05-09T08:57:41.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1b96ffd640 1 -- 172.21.15.98:0/896736070 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1bac030050 con 0x563c2e236180 2024-05-09T08:57:41.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1bc0fce740 10 client.17502 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:41.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1bc0fce740 1 -- 172.21.15.98:0/896736070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563c2de6f0f0 con 0x563c2e236180 2024-05-09T08:57:41.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1b96ffd640 1 -- 172.21.15.98:0/896736070 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bac005500 con 0x563c2e236180 2024-05-09T08:57:41.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1b96ffd640 1 -- 172.21.15.98:0/896736070 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f1bac025020 con 0x563c2e236180 2024-05-09T08:57:41.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1bc0fce740 10 client.17502 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:41.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1bc0fce740 10 client.17502 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:41.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.540+0000 7f1bc0fce740 20 client.17502 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:41.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.541+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/896736070 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563c2e2559d0 0x563c2e257dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.541+0000 7f1bc0fce740 4 client.17502 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:57:41.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.541+0000 7f1bc0fce740 1 -- 172.21.15.98:0/896736070 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c2e1a9430 con 0x563c2e2559d0 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.541+0000 7f1bb77fe640 1 --2- 172.21.15.98:0/896736070 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563c2e2559d0 0x563c2e257dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.541+0000 7f1b96ffd640 10 client.17502 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.541+0000 7f1bb77fe640 1 --2- 172.21.15.98:0/896736070 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563c2e2559d0 0x563c2e257dc0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c9b830dd906783b server_cookie=b7ce43cd2e548ba2 in_seq=0 out_seq=0 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1b96ffd640 1 -- 172.21.15.98:0/896736070 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c2e1a9430 con 0x563c2e2559d0 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1b96ffd640 10 client.17502 handle_command_reply: tid=0 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1bc0fce740 1 client.17502 shutdown 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1bc0fce740 1 -- 172.21.15.98:0/896736070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e236180 msgr2=0x563c2e242620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/896736070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e236180 0x563c2e242620 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f1bac011390 tx=0x7f1bac005da0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1bc0fce740 1 -- 172.21.15.98:0/896736070 shutdown_connections 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/896736070 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563c2e2559d0 0x563c2e257dc0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/896736070 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e245680 0x563c2e2430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/896736070 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236aa0 0x563c2e242b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1b96ffd640 0 client.17502 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/896736070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e236180 0x563c2e242620 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1bc0fce740 1 -- 172.21.15.98:0/896736070 >> 172.21.15.98:0/896736070 conn(0x563c2e2325a0 msgr2=0x563c2e2383d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:41.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1bc0fce740 1 -- 172.21.15.98:0/896736070 shutdown_connections 2024-05-09T08:57:41.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1bc0fce740 1 -- 172.21.15.98:0/896736070 wait complete. 2024-05-09T08:57:41.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.542+0000 7f1bc0fce740 20 client.17502 trim_cache size 0 max 16384 2024-05-09T08:57:41.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bc0fce740 1 Processor -- start 2024-05-09T08:57:41.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bc0fce740 1 -- start start 2024-05-09T08:57:41.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bc0fce740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e236180 0x563c2e251340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bc0fce740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236aa0 0x563c2e231660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bc0fce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e25d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bc0fce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81acce0 con 0x563c2e245680 2024-05-09T08:57:41.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bc0fce740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81ae820 con 0x563c2e236aa0 2024-05-09T08:57:41.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bc0fce740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81ad3b0 con 0x563c2e236180 2024-05-09T08:57:41.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bb6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e25d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bb6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e25d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55666/0 (socket says 172.21.15.98:55666) 2024-05-09T08:57:41.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/731545784 learned_addr learned my addr 172.21.15.98:0/731545784 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:41.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/731545784 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e236180 msgr2=0x563c2e251340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/731545784 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e236180 0x563c2e251340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/731545784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236aa0 msgr2=0x563c2e231660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/731545784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236aa0 0x563c2e231660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.559+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/731545784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c2e2368d0 con 0x563c2e245680 2024-05-09T08:57:41.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.560+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/731545784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e25d500 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f1bac011360 tx=0x7f1bac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925e9a71b8635ca3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:41.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.560+0000 7f1b9d7fa640 1 -- 172.21.15.98:0/731545784 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac01eaf0 con 0x563c2e245680 2024-05-09T08:57:41.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.560+0000 7f1b9d7fa640 1 -- 172.21.15.98:0/731545784 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bac01ec90 con 0x563c2e245680 2024-05-09T08:57:41.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.560+0000 7f1b9d7fa640 1 -- 172.21.15.98:0/731545784 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac004e80 con 0x563c2e245680 2024-05-09T08:57:41.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.560+0000 7f1bc0fce740 1 -- 172.21.15.98:0/731545784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 msgr2=0x563c2e25d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.560+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/731545784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e25d500 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f1bac011360 tx=0x7f1bac0046c0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.560+0000 7f1bc0fce740 1 -- 172.21.15.98:0/731545784 shutdown_connections 2024-05-09T08:57:41.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.560+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/731545784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e25d500 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.560+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/731545784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236aa0 0x563c2e231660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.560+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/731545784 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e236180 0x563c2e251340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.560+0000 7f1bc0fce740 1 -- 172.21.15.98:0/731545784 >> 172.21.15.98:0/731545784 conn(0x563c2e2325a0 msgr2=0x563c2e230f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:41.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.560+0000 7f1bc0fce740 1 -- 172.21.15.98:0/731545784 shutdown_connections 2024-05-09T08:57:41.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.560+0000 7f1bc0fce740 1 -- 172.21.15.98:0/731545784 wait complete. 2024-05-09T08:57:41.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.561+0000 7f1bc0fce740 1 -- start start 2024-05-09T08:57:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.561+0000 7f1bc0fce740 1 Processor -- start 2024-05-09T08:57:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.561+0000 7f1bc0fce740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e236180 0x563c2e24d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.561+0000 7f1bc0fce740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236aa0 0x563c2e24dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.561+0000 7f1bc0fce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e248ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.561+0000 7f1bc0fce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81ae150 con 0x563c2e245680 2024-05-09T08:57:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.561+0000 7f1bc0fce740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81ada80 con 0x563c2e236aa0 2024-05-09T08:57:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.561+0000 7f1bc0fce740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81aace0 con 0x563c2e236180 2024-05-09T08:57:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.561+0000 7f1bc0fce740 10 client.0 _ll_register_callbacks cb 0x7f1ba8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.561+0000 7f1bb6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e248ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.561+0000 7f1bb6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e248ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55682/0 (socket says 172.21.15.98:55682) 2024-05-09T08:57:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.561+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/2188909217 learned_addr learned my addr 172.21.15.98:0/2188909217 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.561+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/2188909217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e236180 0x563c2e24d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.561+0000 7f1bb77fe640 1 --2- 172.21.15.98:0/2188909217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236aa0 0x563c2e24dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/2188909217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e236180 msgr2=0x563c2e24d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/2188909217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e236180 0x563c2e24d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/2188909217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236aa0 msgr2=0x563c2e24dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/2188909217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236aa0 0x563c2e24dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bb6ffd640 1 -- 172.21.15.98:0/2188909217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c2e2492e0 con 0x563c2e245680 2024-05-09T08:57:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bb6ffd640 1 --2- 172.21.15.98:0/2188909217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e248ae0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f1bac005450 tx=0x7f1bac011040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671424a99a970779 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1b96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1b96ffd640 1 -- 172.21.15.98:0/2188909217 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac019030 con 0x563c2e245680 2024-05-09T08:57:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1b96ffd640 1 -- 172.21.15.98:0/2188909217 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f1bac013020 con 0x563c2e245680 2024-05-09T08:57:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bc0fce740 1 -- 172.21.15.98:0/2188909217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bac01ec90 con 0x563c2e245680 2024-05-09T08:57:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1b96ffd640 1 -- 172.21.15.98:0/2188909217 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1bac013220 con 0x563c2e245680 2024-05-09T08:57:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bc0fce740 10 client.17508 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bc0fce740 1 -- 172.21.15.98:0/2188909217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563c2e2bb890 con 0x563c2e245680 2024-05-09T08:57:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1b96ffd640 1 -- 172.21.15.98:0/2188909217 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bac01ec90 con 0x563c2e245680 2024-05-09T08:57:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1b96ffd640 1 -- 172.21.15.98:0/2188909217 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f1bac025050 con 0x563c2e245680 2024-05-09T08:57:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bc0fce740 10 client.17508 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bc0fce740 10 client.17508 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bc0fce740 20 client.17508 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/2188909217 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563c2e2bbe40 0x563c2e2be230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bc0fce740 4 client.17508 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:57:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.562+0000 7f1bc0fce740 1 -- 172.21.15.98:0/2188909217 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563c2e1a3780 con 0x563c2e2bbe40 2024-05-09T08:57:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1bb77fe640 1 --2- 172.21.15.98:0/2188909217 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563c2e2bbe40 0x563c2e2be230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1b96ffd640 10 client.17508 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1bb77fe640 1 --2- 172.21.15.98:0/2188909217 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563c2e2bbe40 0x563c2e2be230 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25ceb44eb404f445 server_cookie=b90d3287beaefe52 in_seq=0 out_seq=0 2024-05-09T08:57:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1b96ffd640 1 -- 172.21.15.98:0/2188909217 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563c2e1a3780 con 0x563c2e2bbe40 2024-05-09T08:57:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1b96ffd640 10 client.17508 handle_command_reply: tid=0 2024-05-09T08:57:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1bc0fce740 1 client.17508 shutdown 2024-05-09T08:57:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1bc0fce740 1 -- 172.21.15.98:0/2188909217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 msgr2=0x563c2e248ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/2188909217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e248ae0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f1bac005450 tx=0x7f1bac011040 comp rx=0 tx=0).stop 2024-05-09T08:57:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1bc0fce740 1 -- 172.21.15.98:0/2188909217 shutdown_connections 2024-05-09T08:57:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/2188909217 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563c2e2bbe40 0x563c2e2be230 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/2188909217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c2e245680 0x563c2e248ae0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/2188909217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563c2e236aa0 0x563c2e24dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1bc0fce740 1 --2- 172.21.15.98:0/2188909217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563c2e236180 0x563c2e24d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1b96ffd640 0 client.17508 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1bc0fce740 1 -- 172.21.15.98:0/2188909217 >> 172.21.15.98:0/2188909217 conn(0x563c2e2325a0 msgr2=0x563c2e242e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1bc0fce740 1 -- 172.21.15.98:0/2188909217 shutdown_connections 2024-05-09T08:57:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1bc0fce740 1 -- 172.21.15.98:0/2188909217 wait complete. 2024-05-09T08:57:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.563+0000 7f1bc0fce740 20 client.17508 trim_cache size 0 max 16384 2024-05-09T08:57:41.573 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:57:41.573 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:57:41.573 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:57:41.573 INFO:teuthology.orchestra.run.smithi098.stdout: "cb87b23f-eb73-4650-b3ac-a2babd37a208": { 2024-05-09T08:57:41.573 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:57:41.573 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "cb87b23f-eb73-4650-b3ac-a2babd37a208", 2024-05-09T08:57:41.573 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:57:41.573 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:57:41.573 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:57:41.574 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:57:41.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.564+0000 7f1bbe1ed640 1 -- 172.21.15.98:0/4108845092 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1ba00a2d80 msgr2=0x7f1ba00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.564+0000 7f1bbe1ed640 1 --2- 172.21.15.98:0/4108845092 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1ba00a2d80 0x7f1ba00a5240 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1bb009b8c0 tx=0x7f1bb006d8f0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.564+0000 7f1bbe1ed640 1 -- 172.21.15.98:0/4108845092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb8106710 msgr2=0x7f1bb819c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.564+0000 7f1bbe1ed640 1 --2- 172.21.15.98:0/4108845092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb8106710 0x7f1bb819c9c0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000cad0 tx=0x7f1bb000cf90 comp rx=0 tx=0).stop 2024-05-09T08:57:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.564+0000 7f1bbe1ed640 1 -- 172.21.15.98:0/4108845092 shutdown_connections 2024-05-09T08:57:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.564+0000 7f1bbe1ed640 1 --2- 172.21.15.98:0/4108845092 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1ba00a2d80 0x7f1ba00a5240 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.564+0000 7f1bbe1ed640 1 --2- 172.21.15.98:0/4108845092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb8106710 0x7f1bb819c9c0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.564+0000 7f1bbe1ed640 1 --2- 172.21.15.98:0/4108845092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bb81010c0 0x7f1bb8198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.564+0000 7f1bbe1ed640 1 --2- 172.21.15.98:0/4108845092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bb8100700 0x7f1bb8198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.564+0000 7f1bbe1ed640 1 -- 172.21.15.98:0/4108845092 >> 172.21.15.98:0/4108845092 conn(0x7f1bb8077830 msgr2=0x7f1bb81023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.564+0000 7f1bbe1ed640 1 -- 172.21.15.98:0/4108845092 shutdown_connections 2024-05-09T08:57:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:57:41.564+0000 7f1bbe1ed640 1 -- 172.21.15.98:0/4108845092 wait complete. 2024-05-09T08:57:41.577 INFO:tasks.cephfs.filesystem:scrub status for tag:cb87b23f-eb73-4650-b3ac-a2babd37a208 - {'path': '/', 'tag': 'cb87b23f-eb73-4650-b3ac-a2babd37a208', 'options': 'recursive,force'} 2024-05-09T08:57:42.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:42 smithi098 ceph-mon[30092]: pgmap v218: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 131 MiB/s wr, 1.10k op/s 2024-05-09T08:57:42.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:42 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:57:42.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:42 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:57:42.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:42 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:57:42.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:42 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:57:42.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:42 smithi142 ceph-mon[34625]: pgmap v218: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 131 MiB/s wr, 1.10k op/s 2024-05-09T08:57:42.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:42 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:57:42.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:42 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:57:42.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:42 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:57:42.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:42 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:57:42.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:42 smithi105 ceph-mon[34498]: pgmap v218: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 131 MiB/s wr, 1.10k op/s 2024-05-09T08:57:42.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:42 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:57:42.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:42 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:57:42.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:42 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:57:42.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:42 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:57:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.140+0000 7f5bd505d640 1 -- 172.21.15.98:0/2289811889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0100a80 msgr2=0x7f5bd0100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.140+0000 7f5bd505d640 1 --2- 172.21.15.98:0/2289811889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0100a80 0x7f5bd0100f00 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009900 tx=0x7f5bb802a410 comp rx=0 tx=0).stop 2024-05-09T08:57:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.140+0000 7f5bd505d640 1 -- 172.21.15.98:0/2289811889 shutdown_connections 2024-05-09T08:57:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.140+0000 7f5bd505d640 1 --2- 172.21.15.98:0/2289811889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bd0101440 0x7f5bd010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.140+0000 7f5bd505d640 1 --2- 172.21.15.98:0/2289811889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0100a80 0x7f5bd0100f00 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.140+0000 7f5bd505d640 1 --2- 172.21.15.98:0/2289811889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bd01056e0 0x7f5bd0105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.140+0000 7f5bd505d640 1 -- 172.21.15.98:0/2289811889 >> 172.21.15.98:0/2289811889 conn(0x7f5bd0076530 msgr2=0x7f5bd0076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.140+0000 7f5bd505d640 1 -- 172.21.15.98:0/2289811889 shutdown_connections 2024-05-09T08:57:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.140+0000 7f5bd505d640 1 -- 172.21.15.98:0/2289811889 wait complete. 2024-05-09T08:57:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bd505d640 1 Processor -- start 2024-05-09T08:57:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bd505d640 1 -- start start 2024-05-09T08:57:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bd505d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bd0100a80 0x7f5bd010e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bd505d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0101440 0x7f5bd010e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bd505d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bd01056e0 0x7f5bd010ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bd505d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd0111c20 con 0x7f5bd0101440 2024-05-09T08:57:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bd505d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd0111aa0 con 0x7f5bd0100a80 2024-05-09T08:57:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bd505d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd0111da0 con 0x7f5bd01056e0 2024-05-09T08:57:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bc7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0101440 0x7f5bd010e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bc7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0101440 0x7f5bd010e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55710/0 (socket says 172.21.15.98:55710) 2024-05-09T08:57:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bc7fff640 1 -- 172.21.15.98:0/4004252576 learned_addr learned my addr 172.21.15.98:0/4004252576 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bcf59e640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bd01056e0 0x7f5bd010ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bced9d640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bd0100a80 0x7f5bd010e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bc7fff640 1 -- 172.21.15.98:0/4004252576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bd01056e0 msgr2=0x7f5bd010ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bc7fff640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bd01056e0 0x7f5bd010ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bc7fff640 1 -- 172.21.15.98:0/4004252576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bd0100a80 msgr2=0x7f5bd010e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.141+0000 7f5bc7fff640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bd0100a80 0x7f5bd010e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.142+0000 7f5bc7fff640 1 -- 172.21.15.98:0/4004252576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd01a63c0 con 0x7f5bd0101440 2024-05-09T08:57:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.142+0000 7f5bced9d640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bd0100a80 0x7f5bd010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.142+0000 7f5bc7fff640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0101440 0x7f5bd010e540 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8006010 tx=0x7f5bb8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.142+0000 7f5bccd99640 1 -- 172.21.15.98:0/4004252576 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb803f070 con 0x7f5bd0101440 2024-05-09T08:57:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.142+0000 7f5bd505d640 1 -- 172.21.15.98:0/4004252576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd01a6650 con 0x7f5bd0101440 2024-05-09T08:57:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.142+0000 7f5bccd99640 1 -- 172.21.15.98:0/4004252576 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bb8031c30 con 0x7f5bd0101440 2024-05-09T08:57:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.142+0000 7f5bd505d640 1 -- 172.21.15.98:0/4004252576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd01a6b90 con 0x7f5bd0101440 2024-05-09T08:57:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.142+0000 7f5bccd99640 1 -- 172.21.15.98:0/4004252576 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8049060 con 0x7f5bd0101440 2024-05-09T08:57:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.145+0000 7f5bccd99640 1 -- 172.21.15.98:0/4004252576 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5bb8042070 con 0x7f5bd0101440 2024-05-09T08:57:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.145+0000 7f5bd7d6e740 1 Processor -- start 2024-05-09T08:57:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.145+0000 7f5bd7d6e740 1 -- start start 2024-05-09T08:57:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bd7d6e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3e5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bd7d6e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5d70 0x556acf3f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bd7d6e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 0x556acf3f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bd7d6e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556acf34a340 con 0x556acf3e5450 2024-05-09T08:57:43.157 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bd7d6e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556acf2f2290 con 0x556acf3f0630 2024-05-09T08:57:43.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bd7d6e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556acf346300 con 0x556acf3e5d70 2024-05-09T08:57:43.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bcf59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3e5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bcf59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3e5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55726/0 (socket says 172.21.15.98:55726) 2024-05-09T08:57:43.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bcf59e640 1 -- 172.21.15.98:0/105768399 learned_addr learned my addr 172.21.15.98:0/105768399 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:43.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bced9d640 1 --2- 172.21.15.98:0/105768399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5d70 0x556acf3f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bccd99640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5bbc0a2dd0 0x7f5bbc0a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.158 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bc7fff640 1 --2- 172.21.15.98:0/105768399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 0x556acf3f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bcf59e640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5bbc0a2dd0 0x7f5bbc0a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bced9d640 1 -- 172.21.15.98:0/105768399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 msgr2=0x556acf3f2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bced9d640 1 --2- 172.21.15.98:0/105768399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 0x556acf3f2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bced9d640 1 -- 172.21.15.98:0/105768399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 msgr2=0x556acf3e5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bced9d640 1 --2- 172.21.15.98:0/105768399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3e5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bced9d640 1 -- 172.21.15.98:0/105768399 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556acf3168e0 con 0x556acf3e5d70 2024-05-09T08:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.146+0000 7f5bcf59e640 1 --2- 172.21.15.98:0/105768399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3e5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:57:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.147+0000 7f5bccd99640 1 -- 172.21.15.98:0/4004252576 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f5bb80efeb0 con 0x7f5bd0101440 2024-05-09T08:57:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.147+0000 7f5bced9d640 1 --2- 172.21.15.98:0/105768399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5d70 0x556acf3f00f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0008b30 tx=0x7f5bc000e350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31757e6a6ba7c051 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.147+0000 7f5bcf59e640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5bbc0a2dd0 0x7f5bbc0a5290 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8005c40 tx=0x7f5bc8004ab0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.147+0000 7f5bc57fa640 1 -- 172.21.15.98:0/105768399 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0014070 con 0x556acf3e5d70 2024-05-09T08:57:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.147+0000 7f5bc57fa640 1 -- 172.21.15.98:0/105768399 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc0005500 con 0x556acf3e5d70 2024-05-09T08:57:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.147+0000 7f5bc57fa640 1 -- 172.21.15.98:0/105768399 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00106e0 con 0x556acf3e5d70 2024-05-09T08:57:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.148+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/105768399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5d70 msgr2=0x556acf3f00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.148+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/105768399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5d70 0x556acf3f00f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0008b30 tx=0x7f5bc000e350 comp rx=0 tx=0).stop 2024-05-09T08:57:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.148+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/105768399 shutdown_connections 2024-05-09T08:57:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.148+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/105768399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 0x556acf3f2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.148+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/105768399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5d70 0x556acf3f00f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.148+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/105768399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3e5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.148+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/105768399 >> 172.21.15.98:0/105768399 conn(0x556acf3e1960 msgr2=0x556acf3e1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.148+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/105768399 shutdown_connections 2024-05-09T08:57:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.148+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/105768399 wait complete. 2024-05-09T08:57:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.148+0000 7f5bd7d6e740 1 -- start start 2024-05-09T08:57:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.148+0000 7f5bd7d6e740 1 Processor -- start 2024-05-09T08:57:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bd7d6e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3ed570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bd7d6e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3e5d70 0x556acf3edab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bd7d6e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3f0630 0x556acf3edff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bd7d6e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556acf3f3260 con 0x556acf3e5450 2024-05-09T08:57:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bd7d6e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556acf305790 con 0x556acf3e5d70 2024-05-09T08:57:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bd7d6e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556acf3f33e0 con 0x556acf3f0630 2024-05-09T08:57:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bd7d6e740 10 client.0 _ll_register_callbacks cb 0x556acf340a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bced9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3ed570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bced9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3ed570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55736/0 (socket says 172.21.15.98:55736) 2024-05-09T08:57:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bced9d640 1 -- 172.21.15.98:0/2031160698 learned_addr learned my addr 172.21.15.98:0/2031160698 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bcf59e640 1 --2- 172.21.15.98:0/2031160698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3f0630 0x556acf3edff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bced9d640 1 -- 172.21.15.98:0/2031160698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3f0630 msgr2=0x556acf3edff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bced9d640 1 --2- 172.21.15.98:0/2031160698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3f0630 0x556acf3edff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bced9d640 1 -- 172.21.15.98:0/2031160698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3e5d70 msgr2=0x556acf3edab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bc7fff640 1 --2- 172.21.15.98:0/2031160698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3e5d70 0x556acf3edab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bced9d640 1 --2- 172.21.15.98:0/2031160698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3e5d70 0x556acf3edab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bced9d640 1 -- 172.21.15.98:0/2031160698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556acf05e210 con 0x556acf3e5450 2024-05-09T08:57:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bced9d640 1 --2- 172.21.15.98:0/2031160698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3ed570 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0000c00 tx=0x7f5bc000efc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c227236229d92e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bb67fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bb67fc640 1 -- 172.21.15.98:0/2031160698 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0027060 con 0x556acf3e5450 2024-05-09T08:57:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bb67fc640 1 -- 172.21.15.98:0/2031160698 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f5bc00104a0 con 0x556acf3e5450 2024-05-09T08:57:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.149+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/2031160698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bc0005500 con 0x556acf3e5450 2024-05-09T08:57:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.150+0000 7f5bb67fc640 1 -- 172.21.15.98:0/2031160698 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5bc002e050 con 0x556acf3e5450 2024-05-09T08:57:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.150+0000 7f5bd7d6e740 10 client.17520 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.150+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/2031160698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556acf12a620 con 0x556acf3e5450 2024-05-09T08:57:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.150+0000 7f5bb67fc640 1 -- 172.21.15.98:0/2031160698 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc0005500 con 0x556acf3e5450 2024-05-09T08:57:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.150+0000 7f5bb67fc640 1 -- 172.21.15.98:0/2031160698 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f5bc0018670 con 0x556acf3e5450 2024-05-09T08:57:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.150+0000 7f5bd7d6e740 10 client.17520 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.150+0000 7f5bd7d6e740 10 client.17520 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.150+0000 7f5bd7d6e740 20 client.17520 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.150+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/2031160698 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556acf400220 0x556acf402610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.150+0000 7f5bd7d6e740 4 client.17520 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:57:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.150+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/2031160698 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556acf357a70 con 0x556acf400220 2024-05-09T08:57:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.150+0000 7f5bc7fff640 1 --2- 172.21.15.98:0/2031160698 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556acf400220 0x556acf402610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.151+0000 7f5bb67fc640 10 client.17520 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.151+0000 7f5bc7fff640 1 --2- 172.21.15.98:0/2031160698 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556acf400220 0x556acf402610 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a94c8107162f53f server_cookie=280d59566893c2e3 in_seq=0 out_seq=0 2024-05-09T08:57:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.151+0000 7f5bb67fc640 1 -- 172.21.15.98:0/2031160698 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556acf357a70 con 0x556acf400220 2024-05-09T08:57:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.152+0000 7f5bb67fc640 10 client.17520 handle_command_reply: tid=0 2024-05-09T08:57:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.152+0000 7f5bd7d6e740 1 client.17520 shutdown 2024-05-09T08:57:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.152+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/2031160698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 msgr2=0x556acf3ed570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.152+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/2031160698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3ed570 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0000c00 tx=0x7f5bc000efc0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.152+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/2031160698 shutdown_connections 2024-05-09T08:57:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.152+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/2031160698 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556acf400220 0x556acf402610 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.152+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/2031160698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3f0630 0x556acf3edff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.152+0000 7f5bb67fc640 0 client.17520 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.152+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/2031160698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3e5d70 0x556acf3edab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.152+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/2031160698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3ed570 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.152+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/2031160698 >> 172.21.15.98:0/2031160698 conn(0x556acf3e1960 msgr2=0x556acf3e76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.152+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/2031160698 shutdown_connections 2024-05-09T08:57:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.152+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/2031160698 wait complete. 2024-05-09T08:57:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.152+0000 7f5bd7d6e740 20 client.17520 trim_cache size 0 max 16384 2024-05-09T08:57:43.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.167+0000 7f5bd7d6e740 1 Processor -- start 2024-05-09T08:57:43.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.167+0000 7f5bd7d6e740 1 -- start start 2024-05-09T08:57:43.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bd7d6e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3fc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bd7d6e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5d70 0x556acf3ec6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bd7d6e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 0x556acf3ef3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bd7d6e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01b0f70 con 0x556acf3e5450 2024-05-09T08:57:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bd7d6e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01b2ab0 con 0x556acf3f0630 2024-05-09T08:57:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bd7d6e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01b1640 con 0x556acf3e5d70 2024-05-09T08:57:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bced9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3fc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bced9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3fc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55744/0 (socket says 172.21.15.98:55744) 2024-05-09T08:57:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bced9d640 1 -- 172.21.15.98:0/2428735161 learned_addr learned my addr 172.21.15.98:0/2428735161 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bced9d640 1 -- 172.21.15.98:0/2428735161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5d70 msgr2=0x556acf3ec6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bced9d640 1 --2- 172.21.15.98:0/2428735161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5d70 0x556acf3ec6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bced9d640 1 -- 172.21.15.98:0/2428735161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 msgr2=0x556acf3ef3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bced9d640 1 --2- 172.21.15.98:0/2428735161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 0x556acf3ef3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bced9d640 1 -- 172.21.15.98:0/2428735161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556acf3dff00 con 0x556acf3e5450 2024-05-09T08:57:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bced9d640 1 --2- 172.21.15.98:0/2428735161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3fc120 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0018ac0 tx=0x7f5bc000e850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ef4d6be3ce6f85 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bc4ff9640 1 -- 172.21.15.98:0/2428735161 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0027030 con 0x556acf3e5450 2024-05-09T08:57:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bc4ff9640 1 -- 172.21.15.98:0/2428735161 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc0005490 con 0x556acf3e5450 2024-05-09T08:57:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.168+0000 7f5bc4ff9640 1 -- 172.21.15.98:0/2428735161 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0014070 con 0x556acf3e5450 2024-05-09T08:57:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.169+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/2428735161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 msgr2=0x556acf3fc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.169+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/2428735161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3fc120 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0018ac0 tx=0x7f5bc000e850 comp rx=0 tx=0).stop 2024-05-09T08:57:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.169+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/2428735161 shutdown_connections 2024-05-09T08:57:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.169+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/2428735161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 0x556acf3ef3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.169+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/2428735161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5d70 0x556acf3ec6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.169+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/2428735161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5450 0x556acf3fc120 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.169+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/2428735161 >> 172.21.15.98:0/2428735161 conn(0x556acf3e1960 msgr2=0x556acf3eec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.169+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/2428735161 shutdown_connections 2024-05-09T08:57:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.169+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/2428735161 wait complete. 2024-05-09T08:57:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.169+0000 7f5bd7d6e740 1 -- start start 2024-05-09T08:57:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bd7d6e740 1 Processor -- start 2024-05-09T08:57:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bd7d6e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5450 0x556acf3f8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bd7d6e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5d70 0x556acf3ea150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bd7d6e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 0x556acf3ea890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bd7d6e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01b23e0 con 0x556acf3e5d70 2024-05-09T08:57:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bd7d6e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01b1d10 con 0x556acf3f0630 2024-05-09T08:57:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bd7d6e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01aef70 con 0x556acf3e5450 2024-05-09T08:57:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bd7d6e740 10 client.0 _ll_register_callbacks cb 0x7f5bc8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bc7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5d70 0x556acf3ea150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bc7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5d70 0x556acf3ea150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55754/0 (socket says 172.21.15.98:55754) 2024-05-09T08:57:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bc7fff640 1 -- 172.21.15.98:0/1726786585 learned_addr learned my addr 172.21.15.98:0/1726786585 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bced9d640 1 --2- 172.21.15.98:0/1726786585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5450 0x556acf3f8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bcf59e640 1 --2- 172.21.15.98:0/1726786585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 0x556acf3ea890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bc7fff640 1 -- 172.21.15.98:0/1726786585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5450 msgr2=0x556acf3f8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bc7fff640 1 --2- 172.21.15.98:0/1726786585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5450 0x556acf3f8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bc7fff640 1 -- 172.21.15.98:0/1726786585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 msgr2=0x556acf3ea890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bc7fff640 1 --2- 172.21.15.98:0/1726786585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 0x556acf3ea890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bc7fff640 1 -- 172.21.15.98:0/1726786585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556acf4654b0 con 0x556acf3e5d70 2024-05-09T08:57:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.170+0000 7f5bcf59e640 1 --2- 172.21.15.98:0/1726786585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 0x556acf3ea890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bc7fff640 1 --2- 172.21.15.98:0/1726786585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5d70 0x556acf3ea150 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f5bb803b740 tx=0x7f5bb8009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a57083026598c31 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bb67fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bb67fc640 1 -- 172.21.15.98:0/1726786585 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb80dc0b0 con 0x556acf3e5d70 2024-05-09T08:57:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bb67fc640 1 -- 172.21.15.98:0/1726786585 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7f5bb8031750 con 0x556acf3e5d70 2024-05-09T08:57:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/1726786585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bc0005490 con 0x556acf3e5d70 2024-05-09T08:57:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bb67fc640 1 -- 172.21.15.98:0/1726786585 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5bb80bc6c0 con 0x556acf3e5d70 2024-05-09T08:57:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bd7d6e740 10 client.17526 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/1726786585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556acf4666e0 con 0x556acf3e5d70 2024-05-09T08:57:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bb67fc640 1 -- 172.21.15.98:0/1726786585 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc0005490 con 0x556acf3e5d70 2024-05-09T08:57:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bb67fc640 1 -- 172.21.15.98:0/1726786585 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f5bb80be050 con 0x556acf3e5d70 2024-05-09T08:57:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bd7d6e740 10 client.17526 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bd7d6e740 10 client.17526 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bd7d6e740 20 client.17526 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/1726786585 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556acf466e60 0x556acf469280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bd7d6e740 4 client.17526 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:57:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.171+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/1726786585 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556acf351dc0 con 0x556acf466e60 2024-05-09T08:57:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.172+0000 7f5bced9d640 1 --2- 172.21.15.98:0/1726786585 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556acf466e60 0x556acf469280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.172+0000 7f5bb67fc640 10 client.17526 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.172+0000 7f5bced9d640 1 --2- 172.21.15.98:0/1726786585 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556acf466e60 0x556acf469280 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba897554c2c9361a server_cookie=f7278151289be5b7 in_seq=0 out_seq=0 2024-05-09T08:57:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bb67fc640 1 -- 172.21.15.98:0/1726786585 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556acf351dc0 con 0x556acf466e60 2024-05-09T08:57:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bb67fc640 10 client.17526 handle_command_reply: tid=0 2024-05-09T08:57:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd7d6e740 1 client.17526 shutdown 2024-05-09T08:57:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/1726786585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5d70 msgr2=0x556acf3ea150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/1726786585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5d70 0x556acf3ea150 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f5bb803b740 tx=0x7f5bb8009620 comp rx=0 tx=0).stop 2024-05-09T08:57:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/1726786585 shutdown_connections 2024-05-09T08:57:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/1726786585 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x556acf466e60 0x556acf469280 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/1726786585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556acf3f0630 0x556acf3ea890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/1726786585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556acf3e5d70 0x556acf3ea150 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd7d6e740 1 --2- 172.21.15.98:0/1726786585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556acf3e5450 0x556acf3f8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bb67fc640 0 client.17526 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/1726786585 >> 172.21.15.98:0/1726786585 conn(0x556acf3e1960 msgr2=0x556acf3e0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/1726786585 shutdown_connections 2024-05-09T08:57:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd7d6e740 1 -- 172.21.15.98:0/1726786585 wait complete. 2024-05-09T08:57:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd7d6e740 20 client.17526 trim_cache size 0 max 16384 2024-05-09T08:57:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:57:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:57:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:57:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:57:43.190 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd505d640 1 -- 172.21.15.98:0/4004252576 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5bbc0a2dd0 msgr2=0x7f5bbc0a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd505d640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5bbc0a2dd0 0x7f5bbc0a5290 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8005c40 tx=0x7f5bc8004ab0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd505d640 1 -- 172.21.15.98:0/4004252576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0101440 msgr2=0x7f5bd010e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.187+0000 7f5bd505d640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0101440 0x7f5bd010e540 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8006010 tx=0x7f5bb8003c10 comp rx=0 tx=0).stop 2024-05-09T08:57:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.188+0000 7f5bd505d640 1 -- 172.21.15.98:0/4004252576 shutdown_connections 2024-05-09T08:57:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.188+0000 7f5bd505d640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5bbc0a2dd0 0x7f5bbc0a5290 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.188+0000 7f5bd505d640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bd01056e0 0x7f5bd010ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.188+0000 7f5bd505d640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0101440 0x7f5bd010e540 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.188+0000 7f5bd505d640 1 --2- 172.21.15.98:0/4004252576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bd0100a80 0x7f5bd010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.188+0000 7f5bd505d640 1 -- 172.21.15.98:0/4004252576 >> 172.21.15.98:0/4004252576 conn(0x7f5bd0076530 msgr2=0x7f5bd00fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.188+0000 7f5bd505d640 1 -- 172.21.15.98:0/4004252576 shutdown_connections 2024-05-09T08:57:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:43.188+0000 7f5bd505d640 1 -- 172.21.15.98:0/4004252576 wait complete. 2024-05-09T08:57:44.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:44 smithi098 ceph-mon[30092]: pgmap v219: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 MiB/s wr, 360 op/s 2024-05-09T08:57:44.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:44 smithi142 ceph-mon[34625]: pgmap v219: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 MiB/s wr, 360 op/s 2024-05-09T08:57:44.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:44 smithi105 ceph-mon[34498]: pgmap v219: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 MiB/s wr, 360 op/s 2024-05-09T08:57:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:46 smithi098 ceph-mon[30092]: pgmap v220: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 87 op/s 2024-05-09T08:57:46.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:46 smithi142 ceph-mon[34625]: pgmap v220: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 87 op/s 2024-05-09T08:57:46.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:46 smithi105 ceph-mon[34498]: pgmap v220: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 87 op/s 2024-05-09T08:57:47.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:47 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:57:47.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:47 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:57:47.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:47 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:57:48.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:48 smithi098 ceph-mon[30092]: pgmap v221: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-05-09T08:57:48.783 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:48 smithi105 ceph-mon[34498]: pgmap v221: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-05-09T08:57:48.784 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:48 smithi142 ceph-mon[34625]: pgmap v221: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-05-09T08:57:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:50 smithi098 ceph-mon[30092]: pgmap v222: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 KiB/s wr, 15 op/s 2024-05-09T08:57:50.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:50 smithi142 ceph-mon[34625]: pgmap v222: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 KiB/s wr, 15 op/s 2024-05-09T08:57:50.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:50 smithi105 ceph-mon[34498]: pgmap v222: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 KiB/s wr, 15 op/s 2024-05-09T08:57:52.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:52 smithi098 ceph-mon[30092]: pgmap v223: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.9 KiB/s wr, 17 op/s 2024-05-09T08:57:52.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:52 smithi142 ceph-mon[34625]: pgmap v223: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.9 KiB/s wr, 17 op/s 2024-05-09T08:57:52.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:52 smithi105 ceph-mon[34498]: pgmap v223: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.9 KiB/s wr, 17 op/s 2024-05-09T08:57:54.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:54 smithi098 ceph-mon[30092]: pgmap v224: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-05-09T08:57:54.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:54 smithi142 ceph-mon[34625]: pgmap v224: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-05-09T08:57:54.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:54 smithi105 ceph-mon[34498]: pgmap v224: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-05-09T08:57:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.288+0000 7fe850d30640 1 -- 172.21.15.98:0/4054651763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe84c106720 msgr2=0x7fe84c106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.288+0000 7fe850d30640 1 --2- 172.21.15.98:0/4054651763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe84c106720 0x7fe84c106b00 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fe834009930 tx=0x7fe83402a420 comp rx=0 tx=0).stop 2024-05-09T08:57:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.288+0000 7fe850d30640 1 -- 172.21.15.98:0/4054651763 shutdown_connections 2024-05-09T08:57:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.288+0000 7fe850d30640 1 --2- 172.21.15.98:0/4054651763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe84c101120 0x7fe84c10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.288+0000 7fe850d30640 1 --2- 172.21.15.98:0/4054651763 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe84c100760 0x7fe84c100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.288+0000 7fe850d30640 1 --2- 172.21.15.98:0/4054651763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe84c106720 0x7fe84c106b00 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.288+0000 7fe850d30640 1 -- 172.21.15.98:0/4054651763 >> 172.21.15.98:0/4054651763 conn(0x7fe84c0778b0 msgr2=0x7fe84c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.288+0000 7fe850d30640 1 -- 172.21.15.98:0/4054651763 shutdown_connections 2024-05-09T08:57:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.288+0000 7fe850d30640 1 -- 172.21.15.98:0/4054651763 wait complete. 2024-05-09T08:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.289+0000 7fe850d30640 1 Processor -- start 2024-05-09T08:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.289+0000 7fe850d30640 1 -- start start 2024-05-09T08:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.289+0000 7fe850d30640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe84c100760 0x7fe84c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.289+0000 7fe850d30640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe84c101120 0x7fe84c1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.289+0000 7fe850d30640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe84c106720 0x7fe84c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.289+0000 7fe850d30640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c111c30 con 0x7fe84c100760 2024-05-09T08:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.289+0000 7fe850d30640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c111ab0 con 0x7fe84c101120 2024-05-09T08:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.289+0000 7fe850d30640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c111db0 con 0x7fe84c106720 2024-05-09T08:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe84a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe84c100760 0x7fe84c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe84a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe84c100760 0x7fe84c197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53904/0 (socket says 172.21.15.98:53904) 2024-05-09T08:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe84a59c640 1 -- 172.21.15.98:0/1081168983 learned_addr learned my addr 172.21.15.98:0/1081168983 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe849d9b640 1 --2- 172.21.15.98:0/1081168983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe84c101120 0x7fe84c1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe84ad9d640 1 --2- 172.21.15.98:0/1081168983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe84c106720 0x7fe84c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe84a59c640 1 -- 172.21.15.98:0/1081168983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe84c106720 msgr2=0x7fe84c19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe84a59c640 1 --2- 172.21.15.98:0/1081168983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe84c106720 0x7fe84c19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe84a59c640 1 -- 172.21.15.98:0/1081168983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe84c101120 msgr2=0x7fe84c1984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe84a59c640 1 --2- 172.21.15.98:0/1081168983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe84c101120 0x7fe84c1984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe84a59c640 1 -- 172.21.15.98:0/1081168983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84c19cfa0 con 0x7fe84c100760 2024-05-09T08:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe84a59c640 1 --2- 172.21.15.98:0/1081168983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe84c100760 0x7fe84c197fb0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fe834009230 tx=0x7fe834004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe8437fe640 1 -- 172.21.15.98:0/1081168983 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83403f030 con 0x7fe84c100760 2024-05-09T08:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe850d30640 1 -- 172.21.15.98:0/1081168983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84c19d230 con 0x7fe84c100760 2024-05-09T08:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe8437fe640 1 -- 172.21.15.98:0/1081168983 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe834003ad0 con 0x7fe84c100760 2024-05-09T08:57:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe8437fe640 1 -- 172.21.15.98:0/1081168983 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe834035070 con 0x7fe84c100760 2024-05-09T08:57:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.290+0000 7fe850d30640 1 -- 172.21.15.98:0/1081168983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe84c1a4b20 con 0x7fe84c100760 2024-05-09T08:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.292+0000 7fe8437fe640 1 -- 172.21.15.98:0/1081168983 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe834042030 con 0x7fe84c100760 2024-05-09T08:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.293+0000 7fe853a3e740 1 Processor -- start 2024-05-09T08:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.293+0000 7fe853a3e740 1 -- start start 2024-05-09T08:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.293+0000 7fe8437fe640 1 --2- 172.21.15.98:0/1081168983 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8380a2da0 0x7fe8380a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe853a3e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264d5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe853a3e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5d70 0x555a264e00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe853a3e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264e0630 0x555a264e2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe853a3e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555a2643a340 con 0x555a264d5450 2024-05-09T08:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe853a3e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555a263e2290 con 0x555a264d5d70 2024-05-09T08:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe853a3e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555a26436300 con 0x555a264e0630 2024-05-09T08:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe84ad9d640 1 --2- 172.21.15.98:0/1081168983 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8380a2da0 0x7fe8380a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe84a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264d5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe8437fe640 1 -- 172.21.15.98:0/1081168983 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fe8340efd80 con 0x7fe84c100760 2024-05-09T08:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe84a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264d5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53912/0 (socket says 172.21.15.98:53912) 2024-05-09T08:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe84a59c640 1 -- 172.21.15.98:0/3047639345 learned_addr learned my addr 172.21.15.98:0/3047639345 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe84a59c640 1 -- 172.21.15.98:0/3047639345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264e0630 msgr2=0x555a264e2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe84ad9d640 1 --2- 172.21.15.98:0/3047639345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264e0630 0x555a264e2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe849d9b640 1 --2- 172.21.15.98:0/3047639345 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5d70 0x555a264e00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe84a59c640 1 --2- 172.21.15.98:0/3047639345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264e0630 0x555a264e2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe84a59c640 1 -- 172.21.15.98:0/3047639345 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5d70 msgr2=0x555a264e00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe84a59c640 1 --2- 172.21.15.98:0/3047639345 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5d70 0x555a264e00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe84a59c640 1 -- 172.21.15.98:0/3047639345 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a264068e0 con 0x555a264d5450 2024-05-09T08:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe84a59c640 1 --2- 172.21.15.98:0/3047639345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264d5830 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fe8340bbb30 tx=0x7fe83402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d90cf1de5da01b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe84ad9d640 1 --2- 172.21.15.98:0/1081168983 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8380a2da0 0x7fe8380a5260 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe8440092f0 tx=0x7fe844009f90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe840ff9640 1 -- 172.21.15.98:0/3047639345 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe834049040 con 0x555a264d5450 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe840ff9640 1 -- 172.21.15.98:0/3047639345 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8340bc530 con 0x555a264d5450 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.294+0000 7fe840ff9640 1 -- 172.21.15.98:0/3047639345 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83409fcf0 con 0x555a264d5450 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.295+0000 7fe853a3e740 1 -- 172.21.15.98:0/3047639345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 msgr2=0x555a264d5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.295+0000 7fe853a3e740 1 --2- 172.21.15.98:0/3047639345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264d5830 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fe8340bbb30 tx=0x7fe83402a420 comp rx=0 tx=0).stop 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.295+0000 7fe853a3e740 1 -- 172.21.15.98:0/3047639345 shutdown_connections 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.295+0000 7fe853a3e740 1 --2- 172.21.15.98:0/3047639345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264e0630 0x555a264e2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.295+0000 7fe853a3e740 1 --2- 172.21.15.98:0/3047639345 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5d70 0x555a264e00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.295+0000 7fe853a3e740 1 --2- 172.21.15.98:0/3047639345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264d5830 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.295+0000 7fe853a3e740 1 -- 172.21.15.98:0/3047639345 >> 172.21.15.98:0/3047639345 conn(0x555a264d1960 msgr2=0x555a264d1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.295+0000 7fe853a3e740 1 -- 172.21.15.98:0/3047639345 shutdown_connections 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.295+0000 7fe853a3e740 1 -- 172.21.15.98:0/3047639345 wait complete. 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe853a3e740 1 -- start start 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe853a3e740 1 Processor -- start 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe853a3e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264dd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe853a3e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5d70 0x555a264ddaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe853a3e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264e0630 0x555a264de030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe853a3e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555a264e3020 con 0x555a264d5450 2024-05-09T08:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe853a3e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555a263f5790 con 0x555a264d5d70 2024-05-09T08:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe853a3e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555a264e31a0 con 0x555a264e0630 2024-05-09T08:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe853a3e740 10 client.0 _ll_register_callbacks cb 0x555a26430a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe84a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264dd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe84a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264dd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53920/0 (socket says 172.21.15.98:53920) 2024-05-09T08:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe84a59c640 1 -- 172.21.15.98:0/4034918192 learned_addr learned my addr 172.21.15.98:0/4034918192 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe849d9b640 1 --2- 172.21.15.98:0/4034918192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5d70 0x555a264ddaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe84ad9d640 1 --2- 172.21.15.98:0/4034918192 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264e0630 0x555a264de030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe84a59c640 1 -- 172.21.15.98:0/4034918192 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264e0630 msgr2=0x555a264de030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe84a59c640 1 --2- 172.21.15.98:0/4034918192 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264e0630 0x555a264de030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe84a59c640 1 -- 172.21.15.98:0/4034918192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5d70 msgr2=0x555a264ddaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe84a59c640 1 --2- 172.21.15.98:0/4034918192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5d70 0x555a264ddaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe84a59c640 1 -- 172.21.15.98:0/4034918192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a2614e210 con 0x555a264d5450 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe849d9b640 1 --2- 172.21.15.98:0/4034918192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5d70 0x555a264ddaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe84a59c640 1 --2- 172.21.15.98:0/4034918192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264dd5b0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fe83403d040 tx=0x7fe8340c01e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc4e5187c277b6c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.296+0000 7fe831ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe831ffb640 1 -- 172.21.15.98:0/4034918192 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe834049070 con 0x555a264d5450 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe831ffb640 1 -- 172.21.15.98:0/4034918192 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fe8340ed2e0 con 0x555a264d5450 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe853a3e740 1 -- 172.21.15.98:0/4034918192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8340bc530 con 0x555a264d5450 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe831ffb640 1 -- 172.21.15.98:0/4034918192 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe8340ed4e0 con 0x555a264d5450 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe853a3e740 10 client.17538 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe853a3e740 1 -- 172.21.15.98:0/4034918192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x555a2621a620 con 0x555a264d5450 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe831ffb640 1 -- 172.21.15.98:0/4034918192 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8340bc530 con 0x555a264d5450 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe831ffb640 1 -- 172.21.15.98:0/4034918192 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe8340bb280 con 0x555a264d5450 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe853a3e740 10 client.17538 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe853a3e740 10 client.17538 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe853a3e740 20 client.17538 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe853a3e740 1 --2- 172.21.15.98:0/4034918192 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a264eff80 0x555a264f2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe853a3e740 4 client.17538 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe853a3e740 1 -- 172.21.15.98:0/4034918192 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a26447a70 con 0x555a264eff80 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.297+0000 7fe849d9b640 1 --2- 172.21.15.98:0/4034918192 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a264eff80 0x555a264f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.298+0000 7fe831ffb640 10 client.17538 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.298+0000 7fe849d9b640 1 --2- 172.21.15.98:0/4034918192 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a264eff80 0x555a264f2370 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=858dc09ea48d1dda server_cookie=5b545ff99382108c in_seq=0 out_seq=0 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.298+0000 7fe831ffb640 1 -- 172.21.15.98:0/4034918192 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a26447a70 con 0x555a264eff80 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.298+0000 7fe831ffb640 10 client.17538 handle_command_reply: tid=0 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.298+0000 7fe853a3e740 1 client.17538 shutdown 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.299+0000 7fe853a3e740 1 -- 172.21.15.98:0/4034918192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 msgr2=0x555a264dd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.299+0000 7fe853a3e740 1 --2- 172.21.15.98:0/4034918192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264dd5b0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fe83403d040 tx=0x7fe8340c01e0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.299+0000 7fe853a3e740 1 -- 172.21.15.98:0/4034918192 shutdown_connections 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.299+0000 7fe853a3e740 1 --2- 172.21.15.98:0/4034918192 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a264eff80 0x555a264f2370 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.299+0000 7fe853a3e740 1 --2- 172.21.15.98:0/4034918192 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264e0630 0x555a264de030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.299+0000 7fe831ffb640 0 client.17538 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.299+0000 7fe853a3e740 1 --2- 172.21.15.98:0/4034918192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5d70 0x555a264ddaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.299+0000 7fe853a3e740 1 --2- 172.21.15.98:0/4034918192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264dd5b0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.299+0000 7fe853a3e740 1 -- 172.21.15.98:0/4034918192 >> 172.21.15.98:0/4034918192 conn(0x555a264d1960 msgr2=0x555a264d76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.299+0000 7fe853a3e740 1 -- 172.21.15.98:0/4034918192 shutdown_connections 2024-05-09T08:57:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.299+0000 7fe853a3e740 1 -- 172.21.15.98:0/4034918192 wait complete. 2024-05-09T08:57:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.299+0000 7fe853a3e740 20 client.17538 trim_cache size 0 max 16384 2024-05-09T08:57:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe853a3e740 1 Processor -- start 2024-05-09T08:57:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe853a3e740 1 -- start start 2024-05-09T08:57:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe853a3e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5450 0x555a264ebe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe853a3e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264d5d70 0x555a264f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe853a3e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264e0630 0x555a264f80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe853a3e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c1aed40 con 0x555a264e0630 2024-05-09T08:57:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe853a3e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c1b0880 con 0x555a264d5450 2024-05-09T08:57:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe853a3e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c1af410 con 0x555a264d5d70 2024-05-09T08:57:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe84ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264e0630 0x555a264f80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe84ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264e0630 0x555a264f80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53924/0 (socket says 172.21.15.98:53924) 2024-05-09T08:57:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe84ad9d640 1 -- 172.21.15.98:0/4119488749 learned_addr learned my addr 172.21.15.98:0/4119488749 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe84a59c640 1 --2- 172.21.15.98:0/4119488749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5450 0x555a264ebe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe849d9b640 1 --2- 172.21.15.98:0/4119488749 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264d5d70 0x555a264f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe84ad9d640 1 -- 172.21.15.98:0/4119488749 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264d5d70 msgr2=0x555a264f7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe84ad9d640 1 --2- 172.21.15.98:0/4119488749 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264d5d70 0x555a264f7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe84ad9d640 1 -- 172.21.15.98:0/4119488749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5450 msgr2=0x555a264ebe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe84ad9d640 1 --2- 172.21.15.98:0/4119488749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5450 0x555a264ebe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe84ad9d640 1 -- 172.21.15.98:0/4119488749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a264cff00 con 0x555a264e0630 2024-05-09T08:57:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe84a59c640 1 --2- 172.21.15.98:0/4119488749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5450 0x555a264ebe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:57:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.314+0000 7fe84ad9d640 1 --2- 172.21.15.98:0/4119488749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264e0630 0x555a264f80b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fe844012480 tx=0x7fe844017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b44a20570bfca445 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.315+0000 7fe833fff640 1 -- 172.21.15.98:0/4119488749 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe844004d20 con 0x555a264e0630 2024-05-09T08:57:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.315+0000 7fe833fff640 1 -- 172.21.15.98:0/4119488749 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe844005500 con 0x555a264e0630 2024-05-09T08:57:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.315+0000 7fe833fff640 1 -- 172.21.15.98:0/4119488749 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe844020a60 con 0x555a264e0630 2024-05-09T08:57:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.315+0000 7fe853a3e740 1 -- 172.21.15.98:0/4119488749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264e0630 msgr2=0x555a264f80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.315+0000 7fe853a3e740 1 --2- 172.21.15.98:0/4119488749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264e0630 0x555a264f80b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fe844012480 tx=0x7fe844017000 comp rx=0 tx=0).stop 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.315+0000 7fe853a3e740 1 -- 172.21.15.98:0/4119488749 shutdown_connections 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.315+0000 7fe853a3e740 1 --2- 172.21.15.98:0/4119488749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264e0630 0x555a264f80b0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.315+0000 7fe853a3e740 1 --2- 172.21.15.98:0/4119488749 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264d5d70 0x555a264f7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.315+0000 7fe853a3e740 1 --2- 172.21.15.98:0/4119488749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264d5450 0x555a264ebe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.315+0000 7fe853a3e740 1 -- 172.21.15.98:0/4119488749 >> 172.21.15.98:0/4119488749 conn(0x555a264d1960 msgr2=0x555a264d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.315+0000 7fe853a3e740 1 -- 172.21.15.98:0/4119488749 shutdown_connections 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.315+0000 7fe853a3e740 1 -- 172.21.15.98:0/4119488749 wait complete. 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe853a3e740 1 -- start start 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe853a3e740 1 Processor -- start 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe853a3e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264e8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe853a3e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264d5d70 0x555a264e52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe853a3e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264e0630 0x555a264e5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe853a3e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c1b01b0 con 0x555a264d5450 2024-05-09T08:57:56.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe853a3e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c1afae0 con 0x555a264e0630 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe853a3e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c1acd40 con 0x555a264d5d70 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe853a3e740 10 client.0 _ll_register_callbacks cb 0x7fe8340bb790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe84a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264e8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe84a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264e8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53936/0 (socket says 172.21.15.98:53936) 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe84a59c640 1 -- 172.21.15.98:0/333505980 learned_addr learned my addr 172.21.15.98:0/333505980 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe849d9b640 1 --2- 172.21.15.98:0/333505980 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264d5d70 0x555a264e52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe84a59c640 1 -- 172.21.15.98:0/333505980 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264d5d70 msgr2=0x555a264e52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe84a59c640 1 --2- 172.21.15.98:0/333505980 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264d5d70 0x555a264e52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe84ad9d640 1 --2- 172.21.15.98:0/333505980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264e0630 0x555a264e5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe84a59c640 1 -- 172.21.15.98:0/333505980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264e0630 msgr2=0x555a264e5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe84a59c640 1 --2- 172.21.15.98:0/333505980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264e0630 0x555a264e5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe84a59c640 1 -- 172.21.15.98:0/333505980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a264e5e70 con 0x555a264d5450 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe84a59c640 1 --2- 172.21.15.98:0/333505980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264e8510 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fe8340bbdc0 tx=0x7fe8340045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802a769046f444a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.316+0000 7fe831ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:57:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe831ffb640 1 -- 172.21.15.98:0/333505980 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8340ed4f0 con 0x555a264d5450 2024-05-09T08:57:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe831ffb640 1 -- 172.21.15.98:0/333505980 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9562+0+0 (secure 0 0 0) 0x7fe834031bd0 con 0x555a264d5450 2024-05-09T08:57:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe853a3e740 1 -- 172.21.15.98:0/333505980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe844005500 con 0x555a264d5450 2024-05-09T08:57:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe831ffb640 1 -- 172.21.15.98:0/333505980 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe8340b7b80 con 0x555a264d5450 2024-05-09T08:57:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe853a3e740 10 client.17544 fetch_fsmap learned FSMap version 37 2024-05-09T08:57:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe853a3e740 1 -- 172.21.15.98:0/333505980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x555a264e5e70 con 0x555a264d5450 2024-05-09T08:57:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe831ffb640 1 -- 172.21.15.98:0/333505980 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe844005500 con 0x555a264d5450 2024-05-09T08:57:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe831ffb640 1 -- 172.21.15.98:0/333505980 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe8340c4610 con 0x555a264d5450 2024-05-09T08:57:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe853a3e740 10 client.17544 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:57:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe853a3e740 10 client.17544 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:57:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe853a3e740 20 client.17544 populate_metadata read hostname 'smithi098' 2024-05-09T08:57:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe853a3e740 1 --2- 172.21.15.98:0/333505980 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a264eff80 0x555a26559500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:57:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe853a3e740 4 client.17544 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:57:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe853a3e740 1 -- 172.21.15.98:0/333505980 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555a26441dc0 con 0x555a264eff80 2024-05-09T08:57:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.317+0000 7fe849d9b640 1 --2- 172.21.15.98:0/333505980 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a264eff80 0x555a26559500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:57:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.318+0000 7fe831ffb640 10 client.17544 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.318+0000 7fe849d9b640 1 --2- 172.21.15.98:0/333505980 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a264eff80 0x555a26559500 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b4d16cc36f44ece server_cookie=bd2902892a29e778 in_seq=0 out_seq=0 2024-05-09T08:57:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe831ffb640 1 -- 172.21.15.98:0/333505980 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555a26441dc0 con 0x555a264eff80 2024-05-09T08:57:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe831ffb640 10 client.17544 handle_command_reply: tid=0 2024-05-09T08:57:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe853a3e740 1 client.17544 shutdown 2024-05-09T08:57:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe853a3e740 1 -- 172.21.15.98:0/333505980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 msgr2=0x555a264e8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe853a3e740 1 --2- 172.21.15.98:0/333505980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264e8510 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fe8340bbdc0 tx=0x7fe8340045f0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe853a3e740 1 -- 172.21.15.98:0/333505980 shutdown_connections 2024-05-09T08:57:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe853a3e740 1 --2- 172.21.15.98:0/333505980 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555a264eff80 0x555a26559500 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe853a3e740 1 --2- 172.21.15.98:0/333505980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555a264e0630 0x555a264e5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe831ffb640 0 client.17544 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:57:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe853a3e740 1 --2- 172.21.15.98:0/333505980 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a264d5d70 0x555a264e52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe853a3e740 1 --2- 172.21.15.98:0/333505980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555a264d5450 0x555a264e8510 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe853a3e740 1 -- 172.21.15.98:0/333505980 >> 172.21.15.98:0/333505980 conn(0x555a264d1960 msgr2=0x555a264dd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe853a3e740 1 -- 172.21.15.98:0/333505980 shutdown_connections 2024-05-09T08:57:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe853a3e740 1 -- 172.21.15.98:0/333505980 wait complete. 2024-05-09T08:57:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.328+0000 7fe853a3e740 20 client.17544 trim_cache size 0 max 16384 2024-05-09T08:57:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:57:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:57:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:57:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:57:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.329+0000 7fe850d30640 1 -- 172.21.15.98:0/1081168983 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8380a2da0 msgr2=0x7fe8380a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.329+0000 7fe850d30640 1 --2- 172.21.15.98:0/1081168983 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8380a2da0 0x7fe8380a5260 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe8440092f0 tx=0x7fe844009f90 comp rx=0 tx=0).stop 2024-05-09T08:57:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.329+0000 7fe850d30640 1 -- 172.21.15.98:0/1081168983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe84c100760 msgr2=0x7fe84c197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:57:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.329+0000 7fe850d30640 1 --2- 172.21.15.98:0/1081168983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe84c100760 0x7fe84c197fb0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fe834009230 tx=0x7fe834004270 comp rx=0 tx=0).stop 2024-05-09T08:57:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.329+0000 7fe850d30640 1 -- 172.21.15.98:0/1081168983 shutdown_connections 2024-05-09T08:57:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.329+0000 7fe850d30640 1 --2- 172.21.15.98:0/1081168983 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe8380a2da0 0x7fe8380a5260 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.329+0000 7fe850d30640 1 --2- 172.21.15.98:0/1081168983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe84c106720 0x7fe84c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.329+0000 7fe850d30640 1 --2- 172.21.15.98:0/1081168983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe84c101120 0x7fe84c1984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.329+0000 7fe850d30640 1 --2- 172.21.15.98:0/1081168983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe84c100760 0x7fe84c197fb0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:57:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.329+0000 7fe850d30640 1 -- 172.21.15.98:0/1081168983 >> 172.21.15.98:0/1081168983 conn(0x7fe84c0778b0 msgr2=0x7fe84c1031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.329+0000 7fe850d30640 1 -- 172.21.15.98:0/1081168983 shutdown_connections 2024-05-09T08:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:57:56.329+0000 7fe850d30640 1 -- 172.21.15.98:0/1081168983 wait complete. 2024-05-09T08:57:56.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:56 smithi098 ceph-mon[30092]: pgmap v225: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-05-09T08:57:56.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:56 smithi142 ceph-mon[34625]: pgmap v225: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-05-09T08:57:56.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:56 smithi105 ceph-mon[34498]: pgmap v225: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-05-09T08:57:58.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:57:58 smithi098 ceph-mon[30092]: pgmap v226: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-05-09T08:57:58.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:57:58 smithi142 ceph-mon[34625]: pgmap v226: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-05-09T08:57:58.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:57:58 smithi105 ceph-mon[34498]: pgmap v226: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-05-09T08:58:00.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:00 smithi098 ceph-mon[30092]: pgmap v227: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.0 KiB/s wr, 11 op/s 2024-05-09T08:58:00.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:00 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:58:00.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:00 smithi142 ceph-mon[34625]: pgmap v227: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.0 KiB/s wr, 11 op/s 2024-05-09T08:58:00.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:00 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:58:00.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:00 smithi105 ceph-mon[34498]: pgmap v227: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.0 KiB/s wr, 11 op/s 2024-05-09T08:58:00.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:00 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:58:01.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:01 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:58:01.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:01 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:58:01.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:01 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:58:01.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:01 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:58:01.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:01 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:58:01.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:01 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:58:01.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:01 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:58:01.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:01 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:58:01.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:01 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:58:02.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:02 smithi105 ceph-mon[34498]: pgmap v228: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.6 KiB/s wr, 14 op/s 2024-05-09T08:58:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:02 smithi098 ceph-mon[30092]: pgmap v228: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.6 KiB/s wr, 14 op/s 2024-05-09T08:58:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:02 smithi142 ceph-mon[34625]: pgmap v228: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.6 KiB/s wr, 14 op/s 2024-05-09T08:58:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:04 smithi105 ceph-mon[34498]: pgmap v229: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.6 KiB/s wr, 12 op/s 2024-05-09T08:58:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:04 smithi098 ceph-mon[30092]: pgmap v229: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.6 KiB/s wr, 12 op/s 2024-05-09T08:58:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:04 smithi142 ceph-mon[34625]: pgmap v229: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.6 KiB/s wr, 12 op/s 2024-05-09T08:58:06.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:06 smithi105 ceph-mon[34498]: pgmap v230: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 27 KiB/s wr, 26 op/s 2024-05-09T08:58:06.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:06 smithi098 ceph-mon[30092]: pgmap v230: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 27 KiB/s wr, 26 op/s 2024-05-09T08:58:06.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:06 smithi142 ceph-mon[34625]: pgmap v230: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 27 KiB/s wr, 26 op/s 2024-05-09T08:58:07.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:07 smithi098 ceph-mon[30092]: osdmap e100: 12 total, 12 up, 12 in 2024-05-09T08:58:07.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:07 smithi142 ceph-mon[34625]: osdmap e100: 12 total, 12 up, 12 in 2024-05-09T08:58:07.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:07 smithi105 ceph-mon[34498]: osdmap e100: 12 total, 12 up, 12 in 2024-05-09T08:58:08.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:08 smithi098 ceph-mon[30092]: pgmap v232: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 57 KiB/s wr, 33 op/s 2024-05-09T08:58:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:08 smithi142 ceph-mon[34625]: pgmap v232: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 57 KiB/s wr, 33 op/s 2024-05-09T08:58:08.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:08 smithi105 ceph-mon[34498]: pgmap v232: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 57 KiB/s wr, 33 op/s 2024-05-09T08:58:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.426+0000 7fb6c574f640 1 -- 172.21.15.98:0/2471201194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c0100760 msgr2=0x7fb6c0100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.426+0000 7fb6c574f640 1 --2- 172.21.15.98:0/2471201194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c0100760 0x7fb6c0100be0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009930 tx=0x7fb6b402a420 comp rx=0 tx=0).stop 2024-05-09T08:58:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.427+0000 7fb6c574f640 1 -- 172.21.15.98:0/2471201194 shutdown_connections 2024-05-09T08:58:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.427+0000 7fb6c574f640 1 --2- 172.21.15.98:0/2471201194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6c0101120 0x7fb6c010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.427+0000 7fb6c574f640 1 --2- 172.21.15.98:0/2471201194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c0100760 0x7fb6c0100be0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.427+0000 7fb6c574f640 1 --2- 172.21.15.98:0/2471201194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c0106720 0x7fb6c0106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.427+0000 7fb6c574f640 1 -- 172.21.15.98:0/2471201194 >> 172.21.15.98:0/2471201194 conn(0x7fb6c00778b0 msgr2=0x7fb6c0077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.427+0000 7fb6c574f640 1 -- 172.21.15.98:0/2471201194 shutdown_connections 2024-05-09T08:58:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.427+0000 7fb6c574f640 1 -- 172.21.15.98:0/2471201194 wait complete. 2024-05-09T08:58:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.427+0000 7fb6c574f640 1 Processor -- start 2024-05-09T08:58:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.427+0000 7fb6c574f640 1 -- start start 2024-05-09T08:58:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6c574f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c0100760 0x7fb6c0198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6c574f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6c0101120 0x7fb6c01993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6c574f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c0106720 0x7fb6c01a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6c574f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c0110b30 con 0x7fb6c0106720 2024-05-09T08:58:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6c574f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01109b0 con 0x7fb6c0100760 2024-05-09T08:58:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6c574f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c0110cb0 con 0x7fb6c0101120 2024-05-09T08:58:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6bf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c0106720 0x7fb6c01a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6bf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c0106720 0x7fb6c01a3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56988/0 (socket says 172.21.15.98:56988) 2024-05-09T08:58:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6bf7fe640 1 -- 172.21.15.98:0/1909889662 learned_addr learned my addr 172.21.15.98:0/1909889662 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6beffd640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c0100760 0x7fb6c0198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6c0101120 0x7fb6c01993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6bf7fe640 1 -- 172.21.15.98:0/1909889662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6c0101120 msgr2=0x7fb6c01993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6bf7fe640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6c0101120 0x7fb6c01993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6bf7fe640 1 -- 172.21.15.98:0/1909889662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c0100760 msgr2=0x7fb6c0198e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6bf7fe640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c0100760 0x7fb6c0198e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6bf7fe640 1 -- 172.21.15.98:0/1909889662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c01a4430 con 0x7fb6c0106720 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6beffd640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c0100760 0x7fb6c0198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6c0101120 0x7fb6c01993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6bf7fe640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c0106720 0x7fb6c01a3ef0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800eaa0 tx=0x7fb6b800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6a7fff640 1 -- 172.21.15.98:0/1909889662 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b800cdb0 con 0x7fb6c0106720 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6c574f640 1 -- 172.21.15.98:0/1909889662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c01a4660 con 0x7fb6c0106720 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6a7fff640 1 -- 172.21.15.98:0/1909889662 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6b8005020 con 0x7fb6c0106720 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6c574f640 1 -- 172.21.15.98:0/1909889662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c01a4bd0 con 0x7fb6c0106720 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.428+0000 7fb6a7fff640 1 -- 172.21.15.98:0/1909889662 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b80116d0 con 0x7fb6c0106720 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.431+0000 7fb6a7fff640 1 -- 172.21.15.98:0/1909889662 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb6b80054d0 con 0x7fb6c0106720 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.431+0000 7fb6c8456740 1 Processor -- start 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.431+0000 7fb6c8456740 1 -- start start 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.431+0000 7fb6c8456740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 0x55e753aa5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.431+0000 7fb6c8456740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 0x55e753ab4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.431+0000 7fb6c8456740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ab6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.431+0000 7fb6c8456740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e753a0a340 con 0x55e753ab4a40 2024-05-09T08:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.431+0000 7fb6c8456740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e7539b2290 con 0x55e753aa5390 2024-05-09T08:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.431+0000 7fb6c8456740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e753a06300 con 0x55e753aa5cb0 2024-05-09T08:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.431+0000 7fb6a7fff640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb6a80a2da0 0x7fb6a80a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.431+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ab6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ab6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57004/0 (socket says 172.21.15.98:57004) 2024-05-09T08:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6be7fc640 1 -- 172.21.15.98:0/810721103 learned_addr learned my addr 172.21.15.98:0/810721103 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6bf7fe640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb6a80a2da0 0x7fb6a80a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/810721103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 0x55e753aa5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6a7fff640 1 -- 172.21.15.98:0/1909889662 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fb6b80cfb90 con 0x7fb6c0106720 2024-05-09T08:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6beffd640 1 --2- 172.21.15.98:0/810721103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 0x55e753ab4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6be7fc640 1 -- 172.21.15.98:0/810721103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 msgr2=0x55e753ab4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/810721103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 0x55e753ab4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6be7fc640 1 -- 172.21.15.98:0/810721103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 msgr2=0x55e753aa5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/810721103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 0x55e753aa5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6be7fc640 1 -- 172.21.15.98:0/810721103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7539d68e0 con 0x55e753ab4a40 2024-05-09T08:58:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/810721103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ab6e30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4031550 tx=0x7fb6b4005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1032605ff5e14e04 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6bf7fe640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb6a80a2da0 0x7fb6a80a5260 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80ccf20 tx=0x7fb6b806dc00 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6a57fa640 1 -- 172.21.15.98:0/810721103 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4035050 con 0x55e753ab4a40 2024-05-09T08:58:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6a57fa640 1 -- 172.21.15.98:0/810721103 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6b4005040 con 0x55e753ab4a40 2024-05-09T08:58:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.432+0000 7fb6a57fa640 1 -- 172.21.15.98:0/810721103 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4004920 con 0x55e753ab4a40 2024-05-09T08:58:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.433+0000 7fb6c8456740 1 -- 172.21.15.98:0/810721103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 msgr2=0x55e753ab6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.433+0000 7fb6c8456740 1 --2- 172.21.15.98:0/810721103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ab6e30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4031550 tx=0x7fb6b4005bb0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.433+0000 7fb6c8456740 1 -- 172.21.15.98:0/810721103 shutdown_connections 2024-05-09T08:58:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.433+0000 7fb6c8456740 1 --2- 172.21.15.98:0/810721103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ab6e30 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.433+0000 7fb6c8456740 1 --2- 172.21.15.98:0/810721103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 0x55e753ab4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.433+0000 7fb6c8456740 1 --2- 172.21.15.98:0/810721103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 0x55e753aa5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.433+0000 7fb6c8456740 1 -- 172.21.15.98:0/810721103 >> 172.21.15.98:0/810721103 conn(0x55e753aa1960 msgr2=0x55e753aa1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.433+0000 7fb6c8456740 1 -- 172.21.15.98:0/810721103 shutdown_connections 2024-05-09T08:58:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.433+0000 7fb6c8456740 1 -- 172.21.15.98:0/810721103 wait complete. 2024-05-09T08:58:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.433+0000 7fb6c8456740 1 -- start start 2024-05-09T08:58:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.433+0000 7fb6c8456740 1 Processor -- start 2024-05-09T08:58:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6c8456740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5390 0x55e753ab1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6c8456740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5cb0 0x55e753ab1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6c8456740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ab24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6c8456740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e753ab75b0 con 0x55e753ab4a40 2024-05-09T08:58:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6c8456740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e7539c5790 con 0x55e753aa5cb0 2024-05-09T08:58:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6c8456740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e753ab7730 con 0x55e753aa5390 2024-05-09T08:58:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6c8456740 10 client.0 _ll_register_callbacks cb 0x55e753a00a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ab24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ab24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57008/0 (socket says 172.21.15.98:57008) 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6be7fc640 1 -- 172.21.15.98:0/3636163365 learned_addr learned my addr 172.21.15.98:0/3636163365 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/3636163365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5390 0x55e753ab1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6beffd640 1 --2- 172.21.15.98:0/3636163365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5cb0 0x55e753ab1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6be7fc640 1 -- 172.21.15.98:0/3636163365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5390 msgr2=0x55e753ab1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/3636163365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5390 0x55e753ab1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6be7fc640 1 -- 172.21.15.98:0/3636163365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5cb0 msgr2=0x55e753ab1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/3636163365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5cb0 0x55e753ab1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6be7fc640 1 -- 172.21.15.98:0/3636163365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e75371e210 con 0x55e753ab4a40 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6beffd640 1 --2- 172.21.15.98:0/3636163365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5cb0 0x55e753ab1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/3636163365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ab24a0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fb6b402a940 tx=0x7fb6b4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613c5b817e4763eb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.434+0000 7fb6a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.435+0000 7fb6a2ff5640 1 -- 172.21.15.98:0/3636163365 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4035040 con 0x55e753ab4a40 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.435+0000 7fb6a2ff5640 1 -- 172.21.15.98:0/3636163365 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fb6b4004ad0 con 0x55e753ab4a40 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.435+0000 7fb6c8456740 1 -- 172.21.15.98:0/3636163365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6b4005040 con 0x55e753ab4a40 2024-05-09T08:58:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.435+0000 7fb6a2ff5640 1 -- 172.21.15.98:0/3636163365 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb6b4004cd0 con 0x55e753ab4a40 2024-05-09T08:58:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.435+0000 7fb6c8456740 10 client.17556 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.435+0000 7fb6c8456740 1 -- 172.21.15.98:0/3636163365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e7537ea620 con 0x55e753ab4a40 2024-05-09T08:58:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.435+0000 7fb6a2ff5640 1 -- 172.21.15.98:0/3636163365 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6b4005040 con 0x55e753ab4a40 2024-05-09T08:58:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.435+0000 7fb6a2ff5640 1 -- 172.21.15.98:0/3636163365 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb6b404a5e0 con 0x55e753ab4a40 2024-05-09T08:58:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.435+0000 7fb6c8456740 10 client.17556 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.435+0000 7fb6c8456740 10 client.17556 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.435+0000 7fb6c8456740 20 client.17556 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.435+0000 7fb6c8456740 1 --2- 172.21.15.98:0/3636163365 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e753ac4630 0x55e753ac6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.435+0000 7fb6c8456740 4 client.17556 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.435+0000 7fb6c8456740 1 -- 172.21.15.98:0/3636163365 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e753a17a70 con 0x55e753ac4630 2024-05-09T08:58:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.436+0000 7fb6beffd640 1 --2- 172.21.15.98:0/3636163365 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e753ac4630 0x55e753ac6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.436+0000 7fb6a2ff5640 10 client.17556 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.436+0000 7fb6beffd640 1 --2- 172.21.15.98:0/3636163365 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e753ac4630 0x55e753ac6a20 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2620d031d040637 server_cookie=d78eede77311a904 in_seq=0 out_seq=0 2024-05-09T08:58:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6a2ff5640 1 -- 172.21.15.98:0/3636163365 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e753a17a70 con 0x55e753ac4630 2024-05-09T08:58:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6a2ff5640 10 client.17556 handle_command_reply: tid=0 2024-05-09T08:58:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6c8456740 1 client.17556 shutdown 2024-05-09T08:58:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6c8456740 1 -- 172.21.15.98:0/3636163365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 msgr2=0x55e753ab24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6c8456740 1 --2- 172.21.15.98:0/3636163365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ab24a0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fb6b402a940 tx=0x7fb6b4009260 comp rx=0 tx=0).stop 2024-05-09T08:58:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6c8456740 1 -- 172.21.15.98:0/3636163365 shutdown_connections 2024-05-09T08:58:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6c8456740 1 --2- 172.21.15.98:0/3636163365 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e753ac4630 0x55e753ac6a20 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6c8456740 1 --2- 172.21.15.98:0/3636163365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ab24a0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6c8456740 1 --2- 172.21.15.98:0/3636163365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5cb0 0x55e753ab1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6c8456740 1 --2- 172.21.15.98:0/3636163365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5390 0x55e753ab1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6a2ff5640 0 client.17556 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6c8456740 1 -- 172.21.15.98:0/3636163365 >> 172.21.15.98:0/3636163365 conn(0x55e753aa1960 msgr2=0x55e753aa76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6c8456740 1 -- 172.21.15.98:0/3636163365 shutdown_connections 2024-05-09T08:58:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6c8456740 1 -- 172.21.15.98:0/3636163365 wait complete. 2024-05-09T08:58:09.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.437+0000 7fb6c8456740 20 client.17556 trim_cache size 0 max 16384 2024-05-09T08:58:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.452+0000 7fb6c8456740 1 Processor -- start 2024-05-09T08:58:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.452+0000 7fb6c8456740 1 -- start start 2024-05-09T08:58:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.452+0000 7fb6c8456740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 0x55e753ab4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.452+0000 7fb6c8456740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 0x55e753ac9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.452+0000 7fb6c8456740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ac9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.452+0000 7fb6c8456740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01acde0 con 0x55e753ab4a40 2024-05-09T08:58:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.452+0000 7fb6c8456740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01ae920 con 0x55e753aa5390 2024-05-09T08:58:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.452+0000 7fb6c8456740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01ad4b0 con 0x55e753aa5cb0 2024-05-09T08:58:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.452+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ac9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.452+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ac9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57024/0 (socket says 172.21.15.98:57024) 2024-05-09T08:58:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.452+0000 7fb6be7fc640 1 -- 172.21.15.98:0/1911147734 learned_addr learned my addr 172.21.15.98:0/1911147734 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.452+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/1911147734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 0x55e753ab4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.452+0000 7fb6beffd640 1 --2- 172.21.15.98:0/1911147734 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 0x55e753ac9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6be7fc640 1 -- 172.21.15.98:0/1911147734 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 msgr2=0x55e753ac9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/1911147734 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 0x55e753ac9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6be7fc640 1 -- 172.21.15.98:0/1911147734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 msgr2=0x55e753ab4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/1911147734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 0x55e753ab4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6be7fc640 1 -- 172.21.15.98:0/1911147734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e753acc860 con 0x55e753ab4a40 2024-05-09T08:58:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/1911147734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ac9f80 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4005eb0 tx=0x7fb6b4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47785029b9cdb867 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6a4ff9640 1 -- 172.21.15.98:0/1911147734 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4035070 con 0x55e753ab4a40 2024-05-09T08:58:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6a4ff9640 1 -- 172.21.15.98:0/1911147734 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6b4050070 con 0x55e753ab4a40 2024-05-09T08:58:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6a4ff9640 1 -- 172.21.15.98:0/1911147734 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b404a420 con 0x55e753ab4a40 2024-05-09T08:58:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6c8456740 1 -- 172.21.15.98:0/1911147734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 msgr2=0x55e753ac9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6c8456740 1 --2- 172.21.15.98:0/1911147734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ac9f80 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4005eb0 tx=0x7fb6b4000bc0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6c8456740 1 -- 172.21.15.98:0/1911147734 shutdown_connections 2024-05-09T08:58:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6c8456740 1 --2- 172.21.15.98:0/1911147734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ac9f80 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6c8456740 1 --2- 172.21.15.98:0/1911147734 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 0x55e753ac9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.453+0000 7fb6c8456740 1 --2- 172.21.15.98:0/1911147734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 0x55e753ab4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6c8456740 1 -- 172.21.15.98:0/1911147734 >> 172.21.15.98:0/1911147734 conn(0x55e753aa1960 msgr2=0x55e753aa0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6c8456740 1 -- 172.21.15.98:0/1911147734 shutdown_connections 2024-05-09T08:58:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6c8456740 1 -- 172.21.15.98:0/1911147734 wait complete. 2024-05-09T08:58:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6c8456740 1 -- start start 2024-05-09T08:58:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6c8456740 1 Processor -- start 2024-05-09T08:58:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6c8456740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 0x55e753ac9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6c8456740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 0x55e753ac81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6c8456740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ac8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6c8456740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01ae250 con 0x55e753ab4a40 2024-05-09T08:58:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6c8456740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01adb80 con 0x55e753aa5390 2024-05-09T08:58:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6c8456740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01aade0 con 0x55e753aa5cb0 2024-05-09T08:58:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6c8456740 10 client.0 _ll_register_callbacks cb 0x7fb6b4031660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ac8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ac8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57032/0 (socket says 172.21.15.98:57032) 2024-05-09T08:58:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6be7fc640 1 -- 172.21.15.98:0/1718345583 learned_addr learned my addr 172.21.15.98:0/1718345583 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/1718345583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 0x55e753ac9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.454+0000 7fb6beffd640 1 --2- 172.21.15.98:0/1718345583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 0x55e753ac81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6be7fc640 1 -- 172.21.15.98:0/1718345583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 msgr2=0x55e753ac81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/1718345583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 0x55e753ac81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6be7fc640 1 -- 172.21.15.98:0/1718345583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 msgr2=0x55e753ac9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/1718345583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 0x55e753ac9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6be7fc640 1 -- 172.21.15.98:0/1718345583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e753ac8fe0 con 0x55e753ab4a40 2024-05-09T08:58:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6beffd640 1 --2- 172.21.15.98:0/1718345583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 0x55e753ac81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6be7fc640 1 --2- 172.21.15.98:0/1718345583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ac8810 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4005c10 tx=0x7fb6b40048d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ceed24e52a8be96 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6a2ff5640 1 -- 172.21.15.98:0/1718345583 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4035030 con 0x55e753ab4a40 2024-05-09T08:58:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6a2ff5640 1 -- 172.21.15.98:0/1718345583 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fb6b40317b0 con 0x55e753ab4a40 2024-05-09T08:58:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6c8456740 1 -- 172.21.15.98:0/1718345583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6b4050070 con 0x55e753ab4a40 2024-05-09T08:58:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6a2ff5640 1 -- 172.21.15.98:0/1718345583 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb6b4004e70 con 0x55e753ab4a40 2024-05-09T08:58:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6c8456740 10 client.17562 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6c8456740 1 -- 172.21.15.98:0/1718345583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e753b2b930 con 0x55e753ab4a40 2024-05-09T08:58:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6a2ff5640 1 -- 172.21.15.98:0/1718345583 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6b4050070 con 0x55e753ab4a40 2024-05-09T08:58:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6a2ff5640 1 -- 172.21.15.98:0/1718345583 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb6b4068020 con 0x55e753ab4a40 2024-05-09T08:58:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6c8456740 10 client.17562 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6c8456740 10 client.17562 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6c8456740 20 client.17562 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.455+0000 7fb6c8456740 1 --2- 172.21.15.98:0/1718345583 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e753b2bf60 0x55e753b2e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.456+0000 7fb6c8456740 4 client.17562 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:58:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.456+0000 7fb6c8456740 1 -- 172.21.15.98:0/1718345583 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e753a11dc0 con 0x55e753b2bf60 2024-05-09T08:58:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.456+0000 7fb6beffd640 1 --2- 172.21.15.98:0/1718345583 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e753b2bf60 0x55e753b2e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.456+0000 7fb6beffd640 1 --2- 172.21.15.98:0/1718345583 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e753b2bf60 0x55e753b2e380 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=977fe0723ad4e63 server_cookie=66f297556592310f in_seq=0 out_seq=0 2024-05-09T08:58:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.456+0000 7fb6a2ff5640 10 client.17562 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:09.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.464+0000 7fb6a2ff5640 1 -- 172.21.15.98:0/1718345583 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e753a11dc0 con 0x55e753b2bf60 2024-05-09T08:58:09.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.464+0000 7fb6a2ff5640 10 client.17562 handle_command_reply: tid=0 2024-05-09T08:58:09.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.464+0000 7fb6c8456740 1 client.17562 shutdown 2024-05-09T08:58:09.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.464+0000 7fb6c8456740 1 -- 172.21.15.98:0/1718345583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 msgr2=0x55e753ac8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.464+0000 7fb6c8456740 1 --2- 172.21.15.98:0/1718345583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ac8810 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4005c10 tx=0x7fb6b40048d0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.464+0000 7fb6c8456740 1 -- 172.21.15.98:0/1718345583 shutdown_connections 2024-05-09T08:58:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.464+0000 7fb6c8456740 1 --2- 172.21.15.98:0/1718345583 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e753b2bf60 0x55e753b2e380 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.464+0000 7fb6c8456740 1 --2- 172.21.15.98:0/1718345583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e753ab4a40 0x55e753ac8810 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.464+0000 7fb6c8456740 1 --2- 172.21.15.98:0/1718345583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e753aa5cb0 0x55e753ac81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.464+0000 7fb6c8456740 1 --2- 172.21.15.98:0/1718345583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e753aa5390 0x55e753ac9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.464+0000 7fb6a2ff5640 0 client.17562 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.464+0000 7fb6c8456740 1 -- 172.21.15.98:0/1718345583 >> 172.21.15.98:0/1718345583 conn(0x55e753aa1960 msgr2=0x55e753aa6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.464+0000 7fb6c8456740 1 -- 172.21.15.98:0/1718345583 shutdown_connections 2024-05-09T08:58:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.464+0000 7fb6c8456740 1 -- 172.21.15.98:0/1718345583 wait complete. 2024-05-09T08:58:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.465+0000 7fb6c8456740 20 client.17562 trim_cache size 0 max 16384 2024-05-09T08:58:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:58:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:58:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:58:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:58:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.465+0000 7fb6c574f640 1 -- 172.21.15.98:0/1909889662 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb6a80a2da0 msgr2=0x7fb6a80a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.465+0000 7fb6c574f640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb6a80a2da0 0x7fb6a80a5260 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80ccf20 tx=0x7fb6b806dc00 comp rx=0 tx=0).stop 2024-05-09T08:58:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.465+0000 7fb6c574f640 1 -- 172.21.15.98:0/1909889662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c0106720 msgr2=0x7fb6c01a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.465+0000 7fb6c574f640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c0106720 0x7fb6c01a3ef0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800eaa0 tx=0x7fb6b800ef60 comp rx=0 tx=0).stop 2024-05-09T08:58:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.465+0000 7fb6c574f640 1 -- 172.21.15.98:0/1909889662 shutdown_connections 2024-05-09T08:58:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.465+0000 7fb6c574f640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb6a80a2da0 0x7fb6a80a5260 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.465+0000 7fb6c574f640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6c0106720 0x7fb6c01a3ef0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.465+0000 7fb6c574f640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6c0101120 0x7fb6c01993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.465+0000 7fb6c574f640 1 --2- 172.21.15.98:0/1909889662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c0100760 0x7fb6c0198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.465+0000 7fb6c574f640 1 -- 172.21.15.98:0/1909889662 >> 172.21.15.98:0/1909889662 conn(0x7fb6c00778b0 msgr2=0x7fb6c0103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.465+0000 7fb6c574f640 1 -- 172.21.15.98:0/1909889662 shutdown_connections 2024-05-09T08:58:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:09.465+0000 7fb6c574f640 1 -- 172.21.15.98:0/1909889662 wait complete. 2024-05-09T08:58:10.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:10 smithi098 ceph-mon[30092]: pgmap v233: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 56 KiB/s wr, 32 op/s 2024-05-09T08:58:10.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:10 smithi142 ceph-mon[34625]: pgmap v233: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 56 KiB/s wr, 32 op/s 2024-05-09T08:58:10.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:10 smithi105 ceph-mon[34498]: pgmap v233: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 56 KiB/s wr, 32 op/s 2024-05-09T08:58:11.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:58:11.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:58:11.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.694+0000 7fe784638640 1 -- 172.21.15.98:0/2863716162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe77c101850 msgr2=0x7fe77c101cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:11.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.694+0000 7fe784638640 1 --2- 172.21.15.98:0/2863716162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe77c101850 0x7fe77c101cd0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009930 tx=0x7fe76c02a420 comp rx=0 tx=0).stop 2024-05-09T08:58:11.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.694+0000 7fe784638640 1 -- 172.21.15.98:0/2863716162 shutdown_connections 2024-05-09T08:58:11.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.694+0000 7fe784638640 1 --2- 172.21.15.98:0/2863716162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe77c102210 0x7fe77c10e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.694+0000 7fe784638640 1 --2- 172.21.15.98:0/2863716162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe77c101850 0x7fe77c101cd0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.694+0000 7fe784638640 1 --2- 172.21.15.98:0/2863716162 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe77c107860 0x7fe77c107c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.694+0000 7fe784638640 1 -- 172.21.15.98:0/2863716162 >> 172.21.15.98:0/2863716162 conn(0x7fe77c0fd340 msgr2=0x7fe77c0ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.694+0000 7fe784638640 1 -- 172.21.15.98:0/2863716162 shutdown_connections 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.694+0000 7fe784638640 1 -- 172.21.15.98:0/2863716162 wait complete. 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.695+0000 7fe784638640 1 Processor -- start 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.695+0000 7fe784638640 1 -- start start 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.695+0000 7fe784638640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe77c101850 0x7fe77c19a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.695+0000 7fe784638640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe77c102210 0x7fe77c19a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.695+0000 7fe784638640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe77c107860 0x7fe77c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.695+0000 7fe784638640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c111c70 con 0x7fe77c102210 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.695+0000 7fe784638640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c111af0 con 0x7fe77c101850 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.695+0000 7fe784638640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c111df0 con 0x7fe77c107860 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.695+0000 7fe781bd3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe77c102210 0x7fe77c19a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.695+0000 7fe781bd3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe77c102210 0x7fe77c19a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57046/0 (socket says 172.21.15.98:57046) 2024-05-09T08:58:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.695+0000 7fe781bd3640 1 -- 172.21.15.98:0/2043277922 learned_addr learned my addr 172.21.15.98:0/2043277922 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.695+0000 7fe7823d4640 1 --2- 172.21.15.98:0/2043277922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe77c101850 0x7fe77c19a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.695+0000 7fe782bd5640 1 --2- 172.21.15.98:0/2043277922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe77c107860 0x7fe77c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.696+0000 7fe781bd3640 1 -- 172.21.15.98:0/2043277922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe77c107860 msgr2=0x7fe77c19ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.696+0000 7fe781bd3640 1 --2- 172.21.15.98:0/2043277922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe77c107860 0x7fe77c19ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.696+0000 7fe781bd3640 1 -- 172.21.15.98:0/2043277922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe77c101850 msgr2=0x7fe77c19a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.696+0000 7fe781bd3640 1 --2- 172.21.15.98:0/2043277922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe77c101850 0x7fe77c19a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.696+0000 7fe781bd3640 1 -- 172.21.15.98:0/2043277922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe77c19f3a0 con 0x7fe77c102210 2024-05-09T08:58:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.696+0000 7fe781bd3640 1 --2- 172.21.15.98:0/2043277922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe77c102210 0x7fe77c19a8f0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009230 tx=0x7fe76c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.696+0000 7fe7737fe640 1 -- 172.21.15.98:0/2043277922 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c03f030 con 0x7fe77c102210 2024-05-09T08:58:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.696+0000 7fe784638640 1 -- 172.21.15.98:0/2043277922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe77c19f630 con 0x7fe77c102210 2024-05-09T08:58:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.696+0000 7fe7737fe640 1 -- 172.21.15.98:0/2043277922 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe76c003ad0 con 0x7fe77c102210 2024-05-09T08:58:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.696+0000 7fe7737fe640 1 -- 172.21.15.98:0/2043277922 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c035070 con 0x7fe77c102210 2024-05-09T08:58:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.696+0000 7fe784638640 1 -- 172.21.15.98:0/2043277922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe77c1a6f20 con 0x7fe77c102210 2024-05-09T08:58:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.699+0000 7fe7737fe640 1 -- 172.21.15.98:0/2043277922 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe76c042030 con 0x7fe77c102210 2024-05-09T08:58:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.699+0000 7fe787421740 1 Processor -- start 2024-05-09T08:58:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe787421740 1 -- start start 2024-05-09T08:58:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe787421740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f52280 0x559789f52660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe787421740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f52ba0 0x559789f5cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe787421740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f5d270 0x559789f5f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe787421740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559789eb7d00 con 0x559789f52ba0 2024-05-09T08:58:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe787421740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559789e5f4c0 con 0x559789f5d270 2024-05-09T08:58:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe787421740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559789eb3cc0 con 0x559789f52280 2024-05-09T08:58:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe7823d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f52ba0 0x559789f5cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe7823d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f52ba0 0x559789f5cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57050/0 (socket says 172.21.15.98:57050) 2024-05-09T08:58:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe7823d4640 1 -- 172.21.15.98:0/58740136 learned_addr learned my addr 172.21.15.98:0/58740136 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe781bd3640 1 --2- 172.21.15.98:0/58740136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f5d270 0x559789f5f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe7737fe640 1 --2- 172.21.15.98:0/2043277922 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe7680a2d00 0x7fe7680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe782bd5640 1 --2- 172.21.15.98:0/58740136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f52280 0x559789f52660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe7823d4640 1 -- 172.21.15.98:0/58740136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f52280 msgr2=0x559789f52660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe7823d4640 1 --2- 172.21.15.98:0/58740136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f52280 0x559789f52660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe7823d4640 1 -- 172.21.15.98:0/58740136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f5d270 msgr2=0x559789f5f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe7823d4640 1 --2- 172.21.15.98:0/58740136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f5d270 0x559789f5f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe7823d4640 1 -- 172.21.15.98:0/58740136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559789c957d0 con 0x559789f52ba0 2024-05-09T08:58:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.700+0000 7fe782bd5640 1 --2- 172.21.15.98:0/58740136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f52280 0x559789f52660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.701+0000 7fe782bd5640 1 --2- 172.21.15.98:0/2043277922 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe7680a2d00 0x7fe7680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.701+0000 7fe7737fe640 1 -- 172.21.15.98:0/2043277922 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fe76c0efd10 con 0x7fe77c102210 2024-05-09T08:58:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.701+0000 7fe7823d4640 1 --2- 172.21.15.98:0/58740136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f52ba0 0x559789f5cd30 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fe77800ab90 tx=0x7fe77800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=132434e90f8adba3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.701+0000 7fe770bfe640 1 -- 172.21.15.98:0/58740136 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77801d070 con 0x559789f52ba0 2024-05-09T08:58:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.701+0000 7fe782bd5640 1 --2- 172.21.15.98:0/2043277922 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe7680a2d00 0x7fe7680a51c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe774005fd0 tx=0x7fe774004a40 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.701+0000 7fe770bfe640 1 -- 172.21.15.98:0/58740136 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe778018b20 con 0x559789f52ba0 2024-05-09T08:58:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.701+0000 7fe770bfe640 1 -- 172.21.15.98:0/58740136 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe778010a80 con 0x559789f52ba0 2024-05-09T08:58:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 -- 172.21.15.98:0/58740136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f52ba0 msgr2=0x559789f5cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 --2- 172.21.15.98:0/58740136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f52ba0 0x559789f5cd30 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fe77800ab90 tx=0x7fe77800f790 comp rx=0 tx=0).stop 2024-05-09T08:58:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 -- 172.21.15.98:0/58740136 shutdown_connections 2024-05-09T08:58:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 --2- 172.21.15.98:0/58740136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f5d270 0x559789f5f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 --2- 172.21.15.98:0/58740136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f52ba0 0x559789f5cd30 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 --2- 172.21.15.98:0/58740136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f52280 0x559789f52660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 -- 172.21.15.98:0/58740136 >> 172.21.15.98:0/58740136 conn(0x559789f4e5a0 msgr2=0x559789f4e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 -- 172.21.15.98:0/58740136 shutdown_connections 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 -- 172.21.15.98:0/58740136 wait complete. 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 -- start start 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 Processor -- start 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52280 0x559789f5a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f52ba0 0x559789f5a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f5d270 0x559789f5ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559789f60060 con 0x559789f5d270 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559789e72b90 con 0x559789f52280 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559789f601e0 con 0x559789f52ba0 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.702+0000 7fe787421740 10 client.0 _ll_register_callbacks cb 0x559789eaeda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe782bd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f5d270 0x559789f5ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe782bd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f5d270 0x559789f5ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57062/0 (socket says 172.21.15.98:57062) 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe782bd5640 1 -- 172.21.15.98:0/230773120 learned_addr learned my addr 172.21.15.98:0/230773120 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe7823d4640 1 --2- 172.21.15.98:0/230773120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52280 0x559789f5a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe782bd5640 1 -- 172.21.15.98:0/230773120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f52ba0 msgr2=0x559789f5a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe782bd5640 1 --2- 172.21.15.98:0/230773120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f52ba0 0x559789f5a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe782bd5640 1 -- 172.21.15.98:0/230773120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52280 msgr2=0x559789f5a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe782bd5640 1 --2- 172.21.15.98:0/230773120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52280 0x559789f5a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe782bd5640 1 -- 172.21.15.98:0/230773120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559789bcb3f0 con 0x559789f5d270 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe7823d4640 1 --2- 172.21.15.98:0/230773120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52280 0x559789f5a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe782bd5640 1 --2- 172.21.15.98:0/230773120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f5d270 0x559789f5ac40 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fe774012940 tx=0x7fe774009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10848f339b9911fb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe765ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe765ffb640 1 -- 172.21.15.98:0/230773120 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe774023070 con 0x559789f5d270 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe787421740 1 -- 172.21.15.98:0/230773120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe778018b20 con 0x559789f5d270 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe765ffb640 1 -- 172.21.15.98:0/230773120 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fe77400ece0 con 0x559789f5d270 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe765ffb640 1 -- 172.21.15.98:0/230773120 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe77402f050 con 0x559789f5d270 2024-05-09T08:58:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe787421740 10 client.17574 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe787421740 1 -- 172.21.15.98:0/230773120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559789b8b0f0 con 0x559789f5d270 2024-05-09T08:58:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.703+0000 7fe765ffb640 1 -- 172.21.15.98:0/230773120 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe778018b20 con 0x559789f5d270 2024-05-09T08:58:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.704+0000 7fe765ffb640 1 -- 172.21.15.98:0/230773120 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe77400f040 con 0x559789f5d270 2024-05-09T08:58:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.704+0000 7fe787421740 10 client.17574 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.704+0000 7fe787421740 10 client.17574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.704+0000 7fe787421740 20 client.17574 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.704+0000 7fe787421740 1 --2- 172.21.15.98:0/230773120 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559789f6d590 0x559789f6fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.704+0000 7fe787421740 4 client.17574 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.704+0000 7fe787421740 1 -- 172.21.15.98:0/230773120 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559789ec5430 con 0x559789f6d590 2024-05-09T08:58:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.704+0000 7fe7823d4640 1 --2- 172.21.15.98:0/230773120 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559789f6d590 0x559789f6fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.704+0000 7fe7823d4640 1 --2- 172.21.15.98:0/230773120 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559789f6d590 0x559789f6fa00 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e9edb35e419c9bc server_cookie=ff0314ebca9ca7ef in_seq=0 out_seq=0 2024-05-09T08:58:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.704+0000 7fe765ffb640 10 client.17574 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe765ffb640 1 -- 172.21.15.98:0/230773120 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559789ec5430 con 0x559789f6d590 2024-05-09T08:58:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe765ffb640 10 client.17574 handle_command_reply: tid=0 2024-05-09T08:58:11.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe787421740 1 client.17574 shutdown 2024-05-09T08:58:11.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe787421740 1 -- 172.21.15.98:0/230773120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f5d270 msgr2=0x559789f5ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:11.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe787421740 1 --2- 172.21.15.98:0/230773120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f5d270 0x559789f5ac40 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fe774012940 tx=0x7fe774009600 comp rx=0 tx=0).stop 2024-05-09T08:58:11.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe787421740 1 -- 172.21.15.98:0/230773120 shutdown_connections 2024-05-09T08:58:11.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe787421740 1 --2- 172.21.15.98:0/230773120 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559789f6d590 0x559789f6fa00 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe787421740 1 --2- 172.21.15.98:0/230773120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f5d270 0x559789f5ac40 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe787421740 1 --2- 172.21.15.98:0/230773120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f52ba0 0x559789f5a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe765ffb640 0 client.17574 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:11.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe787421740 1 --2- 172.21.15.98:0/230773120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52280 0x559789f5a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe787421740 1 -- 172.21.15.98:0/230773120 >> 172.21.15.98:0/230773120 conn(0x559789f4e5a0 msgr2=0x559789f4d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:11.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe787421740 1 -- 172.21.15.98:0/230773120 shutdown_connections 2024-05-09T08:58:11.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe787421740 1 -- 172.21.15.98:0/230773120 wait complete. 2024-05-09T08:58:11.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.705+0000 7fe787421740 20 client.17574 trim_cache size 0 max 16384 2024-05-09T08:58:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe787421740 1 Processor -- start 2024-05-09T08:58:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe787421740 1 -- start start 2024-05-09T08:58:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe787421740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f52280 0x559789f68f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe787421740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52ba0 0x559789f752d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe787421740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f5d270 0x559789f75810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe787421740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c1b1110 con 0x559789f5d270 2024-05-09T08:58:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe787421740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c1b2c50 con 0x559789f52ba0 2024-05-09T08:58:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe787421740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c1b17e0 con 0x559789f52280 2024-05-09T08:58:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe782bd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f5d270 0x559789f75810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe782bd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f5d270 0x559789f75810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57066/0 (socket says 172.21.15.98:57066) 2024-05-09T08:58:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe782bd5640 1 -- 172.21.15.98:0/1753381489 learned_addr learned my addr 172.21.15.98:0/1753381489 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe781bd3640 1 --2- 172.21.15.98:0/1753381489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52ba0 0x559789f752d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe782bd5640 1 -- 172.21.15.98:0/1753381489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f52280 msgr2=0x559789f68f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe782bd5640 1 --2- 172.21.15.98:0/1753381489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f52280 0x559789f68f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe782bd5640 1 -- 172.21.15.98:0/1753381489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52ba0 msgr2=0x559789f752d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe782bd5640 1 --2- 172.21.15.98:0/1753381489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52ba0 0x559789f752d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe782bd5640 1 -- 172.21.15.98:0/1753381489 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559789f6c190 con 0x559789f5d270 2024-05-09T08:58:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.723+0000 7fe782bd5640 1 --2- 172.21.15.98:0/1753381489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f5d270 0x559789f75810 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fe774000f80 tx=0x7fe77400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8580a5f16c50add server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.724+0000 7fe767fff640 1 -- 172.21.15.98:0/1753381489 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe774023070 con 0x559789f5d270 2024-05-09T08:58:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.724+0000 7fe767fff640 1 -- 172.21.15.98:0/1753381489 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe774038070 con 0x559789f5d270 2024-05-09T08:58:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.724+0000 7fe767fff640 1 -- 172.21.15.98:0/1753381489 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77401f930 con 0x559789f5d270 2024-05-09T08:58:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.724+0000 7fe787421740 1 -- 172.21.15.98:0/1753381489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f5d270 msgr2=0x559789f75810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.724+0000 7fe787421740 1 --2- 172.21.15.98:0/1753381489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f5d270 0x559789f75810 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fe774000f80 tx=0x7fe77400ef60 comp rx=0 tx=0).stop 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.724+0000 7fe787421740 1 -- 172.21.15.98:0/1753381489 shutdown_connections 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.724+0000 7fe787421740 1 --2- 172.21.15.98:0/1753381489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f5d270 0x559789f75810 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.724+0000 7fe787421740 1 --2- 172.21.15.98:0/1753381489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52ba0 0x559789f752d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.724+0000 7fe787421740 1 --2- 172.21.15.98:0/1753381489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f52280 0x559789f68f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.724+0000 7fe787421740 1 -- 172.21.15.98:0/1753381489 >> 172.21.15.98:0/1753381489 conn(0x559789f4e5a0 msgr2=0x559789f5aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.724+0000 7fe787421740 1 -- 172.21.15.98:0/1753381489 shutdown_connections 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.724+0000 7fe787421740 1 -- 172.21.15.98:0/1753381489 wait complete. 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.724+0000 7fe787421740 1 -- start start 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe787421740 1 Processor -- start 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe787421740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52280 0x559789f57010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe787421740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f52ba0 0x559789f57550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe787421740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f5d270 0x559789f57a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe787421740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c1b2580 con 0x559789f52ba0 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe787421740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c1b1eb0 con 0x559789f52280 2024-05-09T08:58:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe787421740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c1af110 con 0x559789f5d270 2024-05-09T08:58:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe787421740 10 client.0 _ll_register_callbacks cb 0x7fe774009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe781bd3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f52ba0 0x559789f57550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe781bd3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f52ba0 0x559789f57550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57078/0 (socket says 172.21.15.98:57078) 2024-05-09T08:58:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe781bd3640 1 -- 172.21.15.98:0/3564470982 learned_addr learned my addr 172.21.15.98:0/3564470982 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe782bd5640 1 --2- 172.21.15.98:0/3564470982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f5d270 0x559789f57a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe7823d4640 1 --2- 172.21.15.98:0/3564470982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52280 0x559789f57010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe781bd3640 1 -- 172.21.15.98:0/3564470982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f5d270 msgr2=0x559789f57a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe781bd3640 1 --2- 172.21.15.98:0/3564470982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f5d270 0x559789f57a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe781bd3640 1 -- 172.21.15.98:0/3564470982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52280 msgr2=0x559789f57010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe781bd3640 1 --2- 172.21.15.98:0/3564470982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52280 0x559789f57010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe781bd3640 1 -- 172.21.15.98:0/3564470982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559789fd2d10 con 0x559789f52ba0 2024-05-09T08:58:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe781bd3640 1 --2- 172.21.15.98:0/3564470982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f52ba0 0x559789f57550 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009a50 tx=0x7fe76c0058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b177dc23a25f3b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe765ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe765ffb640 1 -- 172.21.15.98:0/3564470982 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c049040 con 0x559789f52ba0 2024-05-09T08:58:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe765ffb640 1 -- 172.21.15.98:0/3564470982 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fe76c003c80 con 0x559789f52ba0 2024-05-09T08:58:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.725+0000 7fe787421740 1 -- 172.21.15.98:0/3564470982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe774038070 con 0x559789f52ba0 2024-05-09T08:58:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.726+0000 7fe765ffb640 1 -- 172.21.15.98:0/3564470982 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe76c0bbe70 con 0x559789f52ba0 2024-05-09T08:58:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.726+0000 7fe787421740 10 client.17580 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.726+0000 7fe787421740 1 -- 172.21.15.98:0/3564470982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559789fd4040 con 0x559789f52ba0 2024-05-09T08:58:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.726+0000 7fe765ffb640 1 -- 172.21.15.98:0/3564470982 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe774038070 con 0x559789f52ba0 2024-05-09T08:58:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.726+0000 7fe765ffb640 1 -- 172.21.15.98:0/3564470982 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe76c031bd0 con 0x559789f52ba0 2024-05-09T08:58:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.726+0000 7fe787421740 10 client.17580 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.726+0000 7fe787421740 10 client.17580 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.726+0000 7fe787421740 20 client.17580 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.726+0000 7fe787421740 1 --2- 172.21.15.98:0/3564470982 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559789fd47e0 0x559789fd6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.726+0000 7fe787421740 4 client.17580 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.726+0000 7fe787421740 1 -- 172.21.15.98:0/3564470982 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559789ebf780 con 0x559789fd47e0 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.726+0000 7fe7823d4640 1 --2- 172.21.15.98:0/3564470982 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559789fd47e0 0x559789fd6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.726+0000 7fe765ffb640 10 client.17580 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.726+0000 7fe7823d4640 1 --2- 172.21.15.98:0/3564470982 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559789fd47e0 0x559789fd6c00 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fd5e8f42a0bdd14 server_cookie=501c130585daea9 in_seq=0 out_seq=0 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe765ffb640 1 -- 172.21.15.98:0/3564470982 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559789ebf780 con 0x559789fd47e0 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe765ffb640 10 client.17580 handle_command_reply: tid=0 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe787421740 1 client.17580 shutdown 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe787421740 1 -- 172.21.15.98:0/3564470982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f52ba0 msgr2=0x559789f57550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe787421740 1 --2- 172.21.15.98:0/3564470982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f52ba0 0x559789f57550 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009a50 tx=0x7fe76c0058f0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe787421740 1 -- 172.21.15.98:0/3564470982 shutdown_connections 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe787421740 1 --2- 172.21.15.98:0/3564470982 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559789fd47e0 0x559789fd6c00 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe787421740 1 --2- 172.21.15.98:0/3564470982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559789f5d270 0x559789f57a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe787421740 1 --2- 172.21.15.98:0/3564470982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559789f52ba0 0x559789f57550 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe787421740 1 --2- 172.21.15.98:0/3564470982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559789f52280 0x559789f57010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe765ffb640 0 client.17580 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe787421740 1 -- 172.21.15.98:0/3564470982 >> 172.21.15.98:0/3564470982 conn(0x559789f4e5a0 msgr2=0x559789f539d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe787421740 1 -- 172.21.15.98:0/3564470982 shutdown_connections 2024-05-09T08:58:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe787421740 1 -- 172.21.15.98:0/3564470982 wait complete. 2024-05-09T08:58:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe787421740 20 client.17580 trim_cache size 0 max 16384 2024-05-09T08:58:11.734 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:58:11.734 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:58:11.734 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:58:11.734 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:58:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe784638640 1 -- 172.21.15.98:0/2043277922 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe7680a2d00 msgr2=0x7fe7680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe784638640 1 --2- 172.21.15.98:0/2043277922 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe7680a2d00 0x7fe7680a51c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe774005fd0 tx=0x7fe774004a40 comp rx=0 tx=0).stop 2024-05-09T08:58:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe784638640 1 -- 172.21.15.98:0/2043277922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe77c102210 msgr2=0x7fe77c19a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe784638640 1 --2- 172.21.15.98:0/2043277922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe77c102210 0x7fe77c19a8f0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009230 tx=0x7fe76c004270 comp rx=0 tx=0).stop 2024-05-09T08:58:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe784638640 1 -- 172.21.15.98:0/2043277922 shutdown_connections 2024-05-09T08:58:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe784638640 1 --2- 172.21.15.98:0/2043277922 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe7680a2d00 0x7fe7680a51c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe784638640 1 --2- 172.21.15.98:0/2043277922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe77c107860 0x7fe77c19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe784638640 1 --2- 172.21.15.98:0/2043277922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe77c102210 0x7fe77c19a8f0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe784638640 1 --2- 172.21.15.98:0/2043277922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe77c101850 0x7fe77c19a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.727+0000 7fe784638640 1 -- 172.21.15.98:0/2043277922 >> 172.21.15.98:0/2043277922 conn(0x7fe77c0fd340 msgr2=0x7fe77c10d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.728+0000 7fe784638640 1 -- 172.21.15.98:0/2043277922 shutdown_connections 2024-05-09T08:58:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.728+0000 7fe784638640 1 -- 172.21.15.98:0/2043277922 wait complete. 2024-05-09T08:58:11.738 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:58:11.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:58:11.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.840+0000 7efddef42640 1 -- 172.21.15.98:0/2567514156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100780 msgr2=0x7efdd8100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:11.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.840+0000 7efddef42640 1 --2- 172.21.15.98:0/2567514156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100780 0x7efdd8100c00 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7efdc4009930 tx=0x7efdc402a420 comp rx=0 tx=0).stop 2024-05-09T08:58:11.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.840+0000 7efddef42640 1 -- 172.21.15.98:0/2567514156 shutdown_connections 2024-05-09T08:58:11.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.840+0000 7efddef42640 1 --2- 172.21.15.98:0/2567514156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdd8101140 0x7efdd810d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.840+0000 7efddef42640 1 --2- 172.21.15.98:0/2567514156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100780 0x7efdd8100c00 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.840+0000 7efddef42640 1 --2- 172.21.15.98:0/2567514156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd8106740 0x7efdd8106b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.840+0000 7efddef42640 1 -- 172.21.15.98:0/2567514156 >> 172.21.15.98:0/2567514156 conn(0x7efdd8077890 msgr2=0x7efdd8077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:11.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.840+0000 7efddef42640 1 -- 172.21.15.98:0/2567514156 shutdown_connections 2024-05-09T08:58:11.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.840+0000 7efddef42640 1 -- 172.21.15.98:0/2567514156 wait complete. 2024-05-09T08:58:11.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddef42640 1 Processor -- start 2024-05-09T08:58:11.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddef42640 1 -- start start 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddef42640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100780 0x7efdd81970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddef42640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdd8101140 0x7efdd8197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddef42640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd8106740 0x7efdd819b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddef42640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdd8110b50 con 0x7efdd8100780 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddef42640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efdd81109d0 con 0x7efdd8101140 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddef42640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efdd8110cd0 con 0x7efdd8106740 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddccde640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100780 0x7efdd81970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddccde640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100780 0x7efdd81970d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57104/0 (socket says 172.21.15.98:57104) 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddccde640 1 -- 172.21.15.98:0/2251245057 learned_addr learned my addr 172.21.15.98:0/2251245057 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efdd7fff640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdd8101140 0x7efdd8197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddd4df640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd8106740 0x7efdd819b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddccde640 1 -- 172.21.15.98:0/2251245057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd8106740 msgr2=0x7efdd819b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddccde640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd8106740 0x7efdd819b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddccde640 1 -- 172.21.15.98:0/2251245057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdd8101140 msgr2=0x7efdd8197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddccde640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdd8101140 0x7efdd8197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:11.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddccde640 1 -- 172.21.15.98:0/2251245057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd819c0c0 con 0x7efdd8100780 2024-05-09T08:58:11.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efdd7fff640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdd8101140 0x7efdd8197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:11.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.841+0000 7efddccde640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100780 0x7efdd81970d0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7efdcc00c970 tx=0x7efdcc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:11.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.842+0000 7efdd5ffb640 1 -- 172.21.15.98:0/2251245057 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdcc007bb0 con 0x7efdd8100780 2024-05-09T08:58:11.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.842+0000 7efdd5ffb640 1 -- 172.21.15.98:0/2251245057 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdcc007d50 con 0x7efdd8100780 2024-05-09T08:58:11.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.842+0000 7efddef42640 1 -- 172.21.15.98:0/2251245057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd819c3b0 con 0x7efdd8100780 2024-05-09T08:58:11.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.842+0000 7efdd5ffb640 1 -- 172.21.15.98:0/2251245057 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdcc00f450 con 0x7efdd8100780 2024-05-09T08:58:11.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.842+0000 7efddef42640 1 -- 172.21.15.98:0/2251245057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd81a3c40 con 0x7efdd8100780 2024-05-09T08:58:11.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.843+0000 7efddef42640 1 -- 172.21.15.98:0/2251245057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdd8068aa0 con 0x7efdd8100780 2024-05-09T08:58:11.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.844+0000 7efdd5ffb640 1 -- 172.21.15.98:0/2251245057 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7efdcc005020 con 0x7efdd8100780 2024-05-09T08:58:11.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.844+0000 7efdd5ffb640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efdc80a2d00 0x7efdc80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:11.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.844+0000 7efdd7fff640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efdc80a2d00 0x7efdc80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:11.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.845+0000 7efdd5ffb640 1 -- 172.21.15.98:0/2251245057 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7efdcc0cf700 con 0x7efdd8100780 2024-05-09T08:58:11.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.845+0000 7efdd7fff640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efdc80a2d00 0x7efdc80a51c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efdc4000c00 tx=0x7efdc4005a20 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:11.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:11.849+0000 7efdd5ffb640 1 -- 172.21.15.98:0/2251245057 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efdcc014030 con 0x7efdd8100780 2024-05-09T08:58:12.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.062+0000 7efddef42640 1 -- 172.21.15.98:0/2251245057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efdd8198420 con 0x7efdd8100780 2024-05-09T08:58:12.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.063+0000 7efdd5ffb640 1 -- 172.21.15.98:0/2251245057 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11941 (secure 0 0 0) 0x7efdcc09beb0 con 0x7efdd8100780 2024-05-09T08:58:12.063 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:58:12.063 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":26581,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/561027115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":561027115},{"type":"v1","addr":"172.21.15.105:6835","nonce":561027115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:56:52.891273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16641":{"gid":16641,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T08:58:12.063 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 37 2024-05-09T08:58:12.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.065+0000 7efddef42640 1 -- 172.21.15.98:0/2251245057 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efdc80a2d00 msgr2=0x7efdc80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.065+0000 7efddef42640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efdc80a2d00 0x7efdc80a51c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efdc4000c00 tx=0x7efdc4005a20 comp rx=0 tx=0).stop 2024-05-09T08:58:12.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.065+0000 7efddef42640 1 -- 172.21.15.98:0/2251245057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100780 msgr2=0x7efdd81970d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.065+0000 7efddef42640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100780 0x7efdd81970d0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7efdcc00c970 tx=0x7efdcc00ce30 comp rx=0 tx=0).stop 2024-05-09T08:58:12.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.065+0000 7efddef42640 1 -- 172.21.15.98:0/2251245057 shutdown_connections 2024-05-09T08:58:12.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.065+0000 7efddef42640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efdc80a2d00 0x7efdc80a51c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.065+0000 7efddef42640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd8106740 0x7efdd819b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.065+0000 7efddef42640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdd8101140 0x7efdd8197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.065+0000 7efddef42640 1 --2- 172.21.15.98:0/2251245057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd8100780 0x7efdd81970d0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.065+0000 7efddef42640 1 -- 172.21.15.98:0/2251245057 >> 172.21.15.98:0/2251245057 conn(0x7efdd8077890 msgr2=0x7efdd8103400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.065+0000 7efddef42640 1 -- 172.21.15.98:0/2251245057 shutdown_connections 2024-05-09T08:58:12.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.065+0000 7efddef42640 1 -- 172.21.15.98:0/2251245057 wait complete. 2024-05-09T08:58:12.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:58:12.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.180+0000 7fae3c658640 1 -- 172.21.15.98:0/1664845273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae34107810 msgr2=0x7fae34107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.180+0000 7fae3c658640 1 --2- 172.21.15.98:0/1664845273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae34107810 0x7fae34107bf0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fae24009cd0 tx=0x7fae2402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.180+0000 7fae3c658640 1 -- 172.21.15.98:0/1664845273 shutdown_connections 2024-05-09T08:58:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.180+0000 7fae3c658640 1 --2- 172.21.15.98:0/1664845273 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae341021c0 0x7fae3410e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.180+0000 7fae3c658640 1 --2- 172.21.15.98:0/1664845273 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae34101800 0x7fae34101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.180+0000 7fae3c658640 1 --2- 172.21.15.98:0/1664845273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae34107810 0x7fae34107bf0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.180+0000 7fae3c658640 1 -- 172.21.15.98:0/1664845273 >> 172.21.15.98:0/1664845273 conn(0x7fae34078c80 msgr2=0x7fae34079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.180+0000 7fae3c658640 1 -- 172.21.15.98:0/1664845273 shutdown_connections 2024-05-09T08:58:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.180+0000 7fae3c658640 1 -- 172.21.15.98:0/1664845273 wait complete. 2024-05-09T08:58:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.181+0000 7fae3c658640 1 Processor -- start 2024-05-09T08:58:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.181+0000 7fae3c658640 1 -- start start 2024-05-09T08:58:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.181+0000 7fae3c658640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae34101800 0x7fae34198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.181+0000 7fae3c658640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae341021c0 0x7fae34198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.181+0000 7fae3c658640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae34107810 0x7fae3419ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.181+0000 7fae3c658640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae34111be0 con 0x7fae341021c0 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.181+0000 7fae3c658640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fae34111a60 con 0x7fae34107810 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.181+0000 7fae3c658640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fae34111d60 con 0x7fae34101800 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.181+0000 7fae39bf3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae341021c0 0x7fae34198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.181+0000 7fae39bf3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae341021c0 0x7fae34198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57122/0 (socket says 172.21.15.98:57122) 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.181+0000 7fae39bf3640 1 -- 172.21.15.98:0/808336737 learned_addr learned my addr 172.21.15.98:0/808336737 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.181+0000 7fae3abf5640 1 --2- 172.21.15.98:0/808336737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae34107810 0x7fae3419ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.181+0000 7fae3a3f4640 1 --2- 172.21.15.98:0/808336737 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae34101800 0x7fae34198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.182+0000 7fae39bf3640 1 -- 172.21.15.98:0/808336737 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae34101800 msgr2=0x7fae34198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.182+0000 7fae39bf3640 1 --2- 172.21.15.98:0/808336737 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae34101800 0x7fae34198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.182+0000 7fae39bf3640 1 -- 172.21.15.98:0/808336737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae34107810 msgr2=0x7fae3419ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.182+0000 7fae39bf3640 1 --2- 172.21.15.98:0/808336737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae34107810 0x7fae3419ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.182+0000 7fae39bf3640 1 -- 172.21.15.98:0/808336737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae3419d130 con 0x7fae341021c0 2024-05-09T08:58:12.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.182+0000 7fae39bf3640 1 --2- 172.21.15.98:0/808336737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae341021c0 0x7fae34198680 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fae2800e9f0 tx=0x7fae2800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.182+0000 7fae337fe640 1 -- 172.21.15.98:0/808336737 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2800cd80 con 0x7fae341021c0 2024-05-09T08:58:12.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.182+0000 7fae337fe640 1 -- 172.21.15.98:0/808336737 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae28010910 con 0x7fae341021c0 2024-05-09T08:58:12.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.182+0000 7fae3c658640 1 -- 172.21.15.98:0/808336737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae3419d420 con 0x7fae341021c0 2024-05-09T08:58:12.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.182+0000 7fae337fe640 1 -- 172.21.15.98:0/808336737 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae28018990 con 0x7fae341021c0 2024-05-09T08:58:12.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.182+0000 7fae3c658640 1 -- 172.21.15.98:0/808336737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae341a4cb0 con 0x7fae341021c0 2024-05-09T08:58:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.184+0000 7fae337fe640 1 -- 172.21.15.98:0/808336737 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fae28010ac0 con 0x7fae341021c0 2024-05-09T08:58:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.185+0000 7fae3f363740 1 Processor -- start 2024-05-09T08:58:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.185+0000 7fae3f363740 1 -- start start 2024-05-09T08:58:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.185+0000 7fae3f363740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 0x565445185710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.185+0000 7fae3f363740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x56544518fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.185+0000 7fae3f363740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 0x565445192710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.185+0000 7fae3f363740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5654450eac40 con 0x565445185c50 2024-05-09T08:58:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.185+0000 7fae3f363740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x565445092580 con 0x565445185330 2024-05-09T08:58:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.185+0000 7fae3f363740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5654450e6c00 con 0x565445190320 2024-05-09T08:58:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.185+0000 7fae3a3f4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x56544518fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae3a3f4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x56544518fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57138/0 (socket says 172.21.15.98:57138) 2024-05-09T08:58:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae3a3f4640 1 -- 172.21.15.98:0/3249175287 learned_addr learned my addr 172.21.15.98:0/3249175287 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae337fe640 1 --2- 172.21.15.98:0/808336737 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fae1c0a2d80 0x7fae1c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae3abf5640 1 --2- 172.21.15.98:0/3249175287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 0x565445185710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae39bf3640 1 --2- 172.21.15.98:0/3249175287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 0x565445192710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae3abf5640 1 --2- 172.21.15.98:0/808336737 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fae1c0a2d80 0x7fae1c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae337fe640 1 -- 172.21.15.98:0/808336737 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fae280cfb90 con 0x7fae341021c0 2024-05-09T08:58:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae3a3f4640 1 -- 172.21.15.98:0/3249175287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 msgr2=0x565445192710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae3a3f4640 1 --2- 172.21.15.98:0/3249175287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 0x565445192710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae3a3f4640 1 -- 172.21.15.98:0/3249175287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 msgr2=0x565445185710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae3a3f4640 1 --2- 172.21.15.98:0/3249175287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 0x565445185710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae3a3f4640 1 -- 172.21.15.98:0/3249175287 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565444dfe3f0 con 0x565445185c50 2024-05-09T08:58:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae39bf3640 1 --2- 172.21.15.98:0/3249175287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 0x565445192710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae3a3f4640 1 --2- 172.21.15.98:0/3249175287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x56544518fde0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fae24009cd0 tx=0x7fae24009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade5e35867ccc0a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae3abf5640 1 --2- 172.21.15.98:0/808336737 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fae1c0a2d80 0x7fae1c0a5240 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fae2c00bba0 tx=0x7fae2c004730 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae30bfe640 1 -- 172.21.15.98:0/3249175287 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae24048070 con 0x565445185c50 2024-05-09T08:58:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae30bfe640 1 -- 172.21.15.98:0/3249175287 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae24036070 con 0x565445185c50 2024-05-09T08:58:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.186+0000 7fae30bfe640 1 -- 172.21.15.98:0/3249175287 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae24004410 con 0x565445185c50 2024-05-09T08:58:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.187+0000 7fae3f363740 1 -- 172.21.15.98:0/3249175287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 msgr2=0x56544518fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.187+0000 7fae3f363740 1 --2- 172.21.15.98:0/3249175287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x56544518fde0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fae24009cd0 tx=0x7fae24009a20 comp rx=0 tx=0).stop 2024-05-09T08:58:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.187+0000 7fae3f363740 1 -- 172.21.15.98:0/3249175287 shutdown_connections 2024-05-09T08:58:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.187+0000 7fae3f363740 1 --2- 172.21.15.98:0/3249175287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 0x565445192710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.187+0000 7fae3f363740 1 --2- 172.21.15.98:0/3249175287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x56544518fde0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.187+0000 7fae3f363740 1 --2- 172.21.15.98:0/3249175287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 0x565445185710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.187+0000 7fae3f363740 1 -- 172.21.15.98:0/3249175287 >> 172.21.15.98:0/3249175287 conn(0x565445181650 msgr2=0x565445181a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.187+0000 7fae3f363740 1 -- 172.21.15.98:0/3249175287 shutdown_connections 2024-05-09T08:58:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.187+0000 7fae3f363740 1 -- 172.21.15.98:0/3249175287 wait complete. 2024-05-09T08:58:12.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.187+0000 7fae3f363740 1 -- start start 2024-05-09T08:58:12.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.187+0000 7fae3f363740 1 Processor -- start 2024-05-09T08:58:12.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae3f363740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 0x56544518d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae3f363740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x56544518d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae3f363740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 0x56544518dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae3f363740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5654451930b0 con 0x565445185c50 2024-05-09T08:58:12.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae3f363740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5654450a5c40 con 0x565445185330 2024-05-09T08:58:12.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae3f363740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x565445193230 con 0x565445190320 2024-05-09T08:58:12.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae3f363740 10 client.0 _ll_register_callbacks cb 0x565444dfdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:12.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae39bf3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x56544518d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae39bf3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x56544518d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57150/0 (socket says 172.21.15.98:57150) 2024-05-09T08:58:12.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae39bf3640 1 -- 172.21.15.98:0/3247187277 learned_addr learned my addr 172.21.15.98:0/3247187277 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae3abf5640 1 --2- 172.21.15.98:0/3247187277 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 0x56544518dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae39bf3640 1 -- 172.21.15.98:0/3247187277 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 msgr2=0x56544518dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae39bf3640 1 --2- 172.21.15.98:0/3247187277 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 0x56544518dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae39bf3640 1 -- 172.21.15.98:0/3247187277 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 msgr2=0x56544518d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae39bf3640 1 --2- 172.21.15.98:0/3247187277 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 0x56544518d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae39bf3640 1 -- 172.21.15.98:0/3247187277 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565444ec87d0 con 0x565445185c50 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae3abf5640 1 --2- 172.21.15.98:0/3247187277 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 0x56544518dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae39bf3640 1 --2- 172.21.15.98:0/3247187277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x56544518d7c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fae280980f0 tx=0x7fae2801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a1b524c871bea5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae19ffb640 1 -- 172.21.15.98:0/3247187277 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae280a4850 con 0x565445185c50 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae19ffb640 1 -- 172.21.15.98:0/3247187277 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fae28014d20 con 0x565445185c50 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae3f363740 1 -- 172.21.15.98:0/3247187277 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae24036070 con 0x565445185c50 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae19ffb640 1 -- 172.21.15.98:0/3247187277 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fae280cd090 con 0x565445185c50 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae3f363740 10 client.17598 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.188+0000 7fae3f363740 1 -- 172.21.15.98:0/3247187277 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56544519f2a0 con 0x565445185c50 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.189+0000 7fae19ffb640 1 -- 172.21.15.98:0/3247187277 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae24036070 con 0x565445185c50 2024-05-09T08:58:12.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.189+0000 7fae19ffb640 1 -- 172.21.15.98:0/3247187277 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fae280963d0 con 0x565445185c50 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.189+0000 7fae3f363740 10 client.17598 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.189+0000 7fae3f363740 10 client.17598 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.189+0000 7fae3f363740 20 client.17598 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.189+0000 7fae3f363740 1 --2- 172.21.15.98:0/3247187277 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5654451a0760 0x5654451a2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.189+0000 7fae3f363740 4 client.17598 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.189+0000 7fae3f363740 1 -- 172.21.15.98:0/3247187277 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654450f8370 con 0x5654451a0760 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.189+0000 7fae3a3f4640 1 --2- 172.21.15.98:0/3247187277 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5654451a0760 0x5654451a2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.190+0000 7fae19ffb640 10 client.17598 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.190+0000 7fae3a3f4640 1 --2- 172.21.15.98:0/3247187277 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5654451a0760 0x5654451a2be0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=579ec090906efbbe server_cookie=f40eb4a912b68054 in_seq=0 out_seq=0 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.190+0000 7fae19ffb640 1 -- 172.21.15.98:0/3247187277 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5654450f8370 con 0x5654451a0760 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.190+0000 7fae19ffb640 10 client.17598 handle_command_reply: tid=0 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.190+0000 7fae3f363740 1 client.17598 shutdown 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.190+0000 7fae3f363740 1 -- 172.21.15.98:0/3247187277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 msgr2=0x56544518d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.190+0000 7fae3f363740 1 --2- 172.21.15.98:0/3247187277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x56544518d7c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fae280980f0 tx=0x7fae2801f000 comp rx=0 tx=0).stop 2024-05-09T08:58:12.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.190+0000 7fae3f363740 1 -- 172.21.15.98:0/3247187277 shutdown_connections 2024-05-09T08:58:12.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.190+0000 7fae3f363740 1 --2- 172.21.15.98:0/3247187277 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5654451a0760 0x5654451a2be0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.190+0000 7fae3f363740 1 --2- 172.21.15.98:0/3247187277 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 0x56544518dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.190+0000 7fae19ffb640 0 client.17598 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:58:12.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.190+0000 7fae3f363740 1 --2- 172.21.15.98:0/3247187277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x56544518d7c0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.191+0000 7fae3f363740 1 --2- 172.21.15.98:0/3247187277 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 0x56544518d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.191+0000 7fae3f363740 1 -- 172.21.15.98:0/3247187277 >> 172.21.15.98:0/3247187277 conn(0x565445181650 msgr2=0x5654451807b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.191+0000 7fae3f363740 1 -- 172.21.15.98:0/3247187277 shutdown_connections 2024-05-09T08:58:12.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.191+0000 7fae3f363740 1 -- 172.21.15.98:0/3247187277 wait complete. 2024-05-09T08:58:12.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.191+0000 7fae3f363740 20 client.17598 trim_cache size 0 max 16384 2024-05-09T08:58:12.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae3f363740 1 Processor -- start 2024-05-09T08:58:12.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae3f363740 1 -- start start 2024-05-09T08:58:12.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae3f363740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 0x56544519bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae3f363740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x565445187a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae3f363740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 0x565445180630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae3f363740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae341aee90 con 0x565445185c50 2024-05-09T08:58:12.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae3f363740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fae341b09d0 con 0x565445185330 2024-05-09T08:58:12.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae3f363740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fae341af560 con 0x565445190320 2024-05-09T08:58:12.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae39bf3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x565445187a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae39bf3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x565445187a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57164/0 (socket says 172.21.15.98:57164) 2024-05-09T08:58:12.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae39bf3640 1 -- 172.21.15.98:0/865346798 learned_addr learned my addr 172.21.15.98:0/865346798 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae3a3f4640 1 --2- 172.21.15.98:0/865346798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 0x56544519bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae39bf3640 1 -- 172.21.15.98:0/865346798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 msgr2=0x565445180630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae39bf3640 1 --2- 172.21.15.98:0/865346798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 0x565445180630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae39bf3640 1 -- 172.21.15.98:0/865346798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 msgr2=0x56544519bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae39bf3640 1 --2- 172.21.15.98:0/865346798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 0x56544519bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae39bf3640 1 -- 172.21.15.98:0/865346798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56544517d1a0 con 0x565445185c50 2024-05-09T08:58:12.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae3a3f4640 1 --2- 172.21.15.98:0/865346798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 0x56544519bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:12.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae39bf3640 1 --2- 172.21.15.98:0/865346798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x565445187a90 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fae2809bc20 tx=0x7fae280cfe50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b68c4f4a912bc00 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae1bfff640 1 -- 172.21.15.98:0/865346798 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae280cd230 con 0x565445185c50 2024-05-09T08:58:12.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae1bfff640 1 -- 172.21.15.98:0/865346798 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae280a4850 con 0x565445185c50 2024-05-09T08:58:12.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.204+0000 7fae1bfff640 1 -- 172.21.15.98:0/865346798 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2809b130 con 0x565445185c50 2024-05-09T08:58:12.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.205+0000 7fae3f363740 1 -- 172.21.15.98:0/865346798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 msgr2=0x565445187a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.205+0000 7fae3f363740 1 --2- 172.21.15.98:0/865346798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x565445187a90 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fae2809bc20 tx=0x7fae280cfe50 comp rx=0 tx=0).stop 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.205+0000 7fae3f363740 1 -- 172.21.15.98:0/865346798 shutdown_connections 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.205+0000 7fae3f363740 1 --2- 172.21.15.98:0/865346798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445190320 0x565445180630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.205+0000 7fae3f363740 1 --2- 172.21.15.98:0/865346798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185c50 0x565445187a90 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.205+0000 7fae3f363740 1 --2- 172.21.15.98:0/865346798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445185330 0x56544519bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.205+0000 7fae3f363740 1 -- 172.21.15.98:0/865346798 >> 172.21.15.98:0/865346798 conn(0x565445181650 msgr2=0x56544517ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.205+0000 7fae3f363740 1 -- 172.21.15.98:0/865346798 shutdown_connections 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.205+0000 7fae3f363740 1 -- 172.21.15.98:0/865346798 wait complete. 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.205+0000 7fae3f363740 1 -- start start 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.205+0000 7fae3f363740 1 Processor -- start 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3f363740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185330 0x565445194300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3f363740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445185c50 0x565445194840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3f363740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445190320 0x565445205d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3f363740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae341b0300 con 0x565445185330 2024-05-09T08:58:12.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3f363740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fae341afc30 con 0x565445190320 2024-05-09T08:58:12.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3f363740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fae341ace90 con 0x565445185c50 2024-05-09T08:58:12.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3f363740 10 client.0 _ll_register_callbacks cb 0x7fae28009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:12.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3a3f4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185330 0x565445194300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3a3f4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185330 0x565445194300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57180/0 (socket says 172.21.15.98:57180) 2024-05-09T08:58:12.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3a3f4640 1 -- 172.21.15.98:0/3344012396 learned_addr learned my addr 172.21.15.98:0/3344012396 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3abf5640 1 --2- 172.21.15.98:0/3344012396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445190320 0x565445205d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3a3f4640 1 -- 172.21.15.98:0/3344012396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445185c50 msgr2=0x565445194840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3a3f4640 1 --2- 172.21.15.98:0/3344012396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445185c50 0x565445194840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3a3f4640 1 -- 172.21.15.98:0/3344012396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445190320 msgr2=0x565445205d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3a3f4640 1 --2- 172.21.15.98:0/3344012396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445190320 0x565445205d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3a3f4640 1 -- 172.21.15.98:0/3344012396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565445206260 con 0x565445185330 2024-05-09T08:58:12.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3a3f4640 1 --2- 172.21.15.98:0/3344012396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185330 0x565445194300 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fae24046040 tx=0x7fae24009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a19ec262f88028 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:12.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae19ffb640 1 -- 172.21.15.98:0/3344012396 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae24048040 con 0x565445185330 2024-05-09T08:58:12.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae19ffb640 1 -- 172.21.15.98:0/3344012396 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fae24031670 con 0x565445185330 2024-05-09T08:58:12.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3f363740 1 -- 172.21.15.98:0/3344012396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae280a4850 con 0x565445185330 2024-05-09T08:58:12.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae19ffb640 1 -- 172.21.15.98:0/3344012396 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fae2405f050 con 0x565445185330 2024-05-09T08:58:12.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3f363740 10 client.17604 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:12.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae3f363740 1 -- 172.21.15.98:0/3344012396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x565445207330 con 0x565445185330 2024-05-09T08:58:12.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.206+0000 7fae19ffb640 1 -- 172.21.15.98:0/3344012396 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae280a4850 con 0x565445185330 2024-05-09T08:58:12.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.207+0000 7fae19ffb640 1 -- 172.21.15.98:0/3344012396 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fae24042030 con 0x565445185330 2024-05-09T08:58:12.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.207+0000 7fae3f363740 10 client.17604 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:12.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.207+0000 7fae3f363740 10 client.17604 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:58:12.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.207+0000 7fae3f363740 20 client.17604 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:12.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.207+0000 7fae3f363740 1 --2- 172.21.15.98:0/3344012396 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x565445207740 0x565445209b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.207+0000 7fae3f363740 4 client.17604 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:58:12.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.207+0000 7fae3f363740 1 -- 172.21.15.98:0/3344012396 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5654450f26c0 con 0x565445207740 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.207+0000 7fae39bf3640 1 --2- 172.21.15.98:0/3344012396 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x565445207740 0x565445209b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.207+0000 7fae19ffb640 10 client.17604 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.207+0000 7fae39bf3640 1 --2- 172.21.15.98:0/3344012396 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x565445207740 0x565445209b60 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ac6f4434c81352bd server_cookie=d30324001a57d0ef in_seq=0 out_seq=0 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.207+0000 7fae19ffb640 1 -- 172.21.15.98:0/3344012396 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5654450f26c0 con 0x565445207740 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.207+0000 7fae19ffb640 10 client.17604 handle_command_reply: tid=0 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.207+0000 7fae3f363740 1 client.17604 shutdown 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3f363740 1 -- 172.21.15.98:0/3344012396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185330 msgr2=0x565445194300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3f363740 1 --2- 172.21.15.98:0/3344012396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185330 0x565445194300 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fae24046040 tx=0x7fae24009580 comp rx=0 tx=0).stop 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3f363740 1 -- 172.21.15.98:0/3344012396 shutdown_connections 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3f363740 1 --2- 172.21.15.98:0/3344012396 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x565445207740 0x565445209b60 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3f363740 1 --2- 172.21.15.98:0/3344012396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565445190320 0x565445205d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae19ffb640 0 client.17604 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3f363740 1 --2- 172.21.15.98:0/3344012396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x565445185c50 0x565445194840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3f363740 1 --2- 172.21.15.98:0/3344012396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565445185330 0x565445194300 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3f363740 1 -- 172.21.15.98:0/3344012396 >> 172.21.15.98:0/3344012396 conn(0x565445181650 msgr2=0x5654451807b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3f363740 1 -- 172.21.15.98:0/3344012396 shutdown_connections 2024-05-09T08:58:12.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3f363740 1 -- 172.21.15.98:0/3344012396 wait complete. 2024-05-09T08:58:12.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3f363740 20 client.17604 trim_cache size 0 max 16384 2024-05-09T08:58:12.216 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:58:12.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3c658640 1 -- 172.21.15.98:0/808336737 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fae1c0a2d80 msgr2=0x7fae1c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3c658640 1 --2- 172.21.15.98:0/808336737 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fae1c0a2d80 0x7fae1c0a5240 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fae2c00bba0 tx=0x7fae2c004730 comp rx=0 tx=0).stop 2024-05-09T08:58:12.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3c658640 1 -- 172.21.15.98:0/808336737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae341021c0 msgr2=0x7fae34198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3c658640 1 --2- 172.21.15.98:0/808336737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae341021c0 0x7fae34198680 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fae2800e9f0 tx=0x7fae2800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3c658640 1 -- 172.21.15.98:0/808336737 shutdown_connections 2024-05-09T08:58:12.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3c658640 1 --2- 172.21.15.98:0/808336737 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fae1c0a2d80 0x7fae1c0a5240 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3c658640 1 --2- 172.21.15.98:0/808336737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae34107810 0x7fae3419ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3c658640 1 --2- 172.21.15.98:0/808336737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae341021c0 0x7fae34198680 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3c658640 1 --2- 172.21.15.98:0/808336737 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae34101800 0x7fae34198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3c658640 1 -- 172.21.15.98:0/808336737 >> 172.21.15.98:0/808336737 conn(0x7fae34078c80 msgr2=0x7fae340ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3c658640 1 -- 172.21.15.98:0/808336737 shutdown_connections 2024-05-09T08:58:12.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.208+0000 7fae3c658640 1 -- 172.21.15.98:0/808336737 wait complete. 2024-05-09T08:58:12.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:58:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.324+0000 7fde328e1640 1 -- 172.21.15.98:0/3045222042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde2c106710 msgr2=0x7fde2c106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.324+0000 7fde328e1640 1 --2- 172.21.15.98:0/3045222042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde2c106710 0x7fde2c106af0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fde1c009930 tx=0x7fde1c02a420 comp rx=0 tx=0).stop 2024-05-09T08:58:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.324+0000 7fde328e1640 1 -- 172.21.15.98:0/3045222042 shutdown_connections 2024-05-09T08:58:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.324+0000 7fde328e1640 1 --2- 172.21.15.98:0/3045222042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde2c1010c0 0x7fde2c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.324+0000 7fde328e1640 1 --2- 172.21.15.98:0/3045222042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde2c100700 0x7fde2c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.324+0000 7fde328e1640 1 --2- 172.21.15.98:0/3045222042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde2c106710 0x7fde2c106af0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.324+0000 7fde328e1640 1 -- 172.21.15.98:0/3045222042 >> 172.21.15.98:0/3045222042 conn(0x7fde2c077830 msgr2=0x7fde2c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.324+0000 7fde328e1640 1 -- 172.21.15.98:0/3045222042 shutdown_connections 2024-05-09T08:58:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.324+0000 7fde328e1640 1 -- 172.21.15.98:0/3045222042 wait complete. 2024-05-09T08:58:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.324+0000 7fde328e1640 1 Processor -- start 2024-05-09T08:58:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.324+0000 7fde328e1640 1 -- start start 2024-05-09T08:58:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde328e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde2c100700 0x7fde2c198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde328e1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde2c1010c0 0x7fde2c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde328e1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde2c106710 0x7fde2c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde328e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c111c40 con 0x7fde2c100700 2024-05-09T08:58:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde328e1640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c111ac0 con 0x7fde2c1010c0 2024-05-09T08:58:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde328e1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c111dc0 con 0x7fde2c106710 2024-05-09T08:58:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde2bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde2c100700 0x7fde2c198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde2bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde2c100700 0x7fde2c198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57190/0 (socket says 172.21.15.98:57190) 2024-05-09T08:58:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde2bfff640 1 -- 172.21.15.98:0/238675321 learned_addr learned my addr 172.21.15.98:0/238675321 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde2bfff640 1 -- 172.21.15.98:0/238675321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde2c106710 msgr2=0x7fde2c19ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde30e7e640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde2c106710 0x7fde2c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde2b7fe640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde2c1010c0 0x7fde2c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde2bfff640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde2c106710 0x7fde2c19ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde2bfff640 1 -- 172.21.15.98:0/238675321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde2c1010c0 msgr2=0x7fde2c1986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde2bfff640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde2c1010c0 0x7fde2c1986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde2bfff640 1 -- 172.21.15.98:0/238675321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2c19d150 con 0x7fde2c100700 2024-05-09T08:58:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde30e7e640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde2c106710 0x7fde2c19ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde2b7fe640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde2c1010c0 0x7fde2c1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde2bfff640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde2c100700 0x7fde2c198160 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fde1c009230 tx=0x7fde1c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde297fa640 1 -- 172.21.15.98:0/238675321 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c03f030 con 0x7fde2c100700 2024-05-09T08:58:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde297fa640 1 -- 172.21.15.98:0/238675321 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde1c003ad0 con 0x7fde2c100700 2024-05-09T08:58:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde328e1640 1 -- 172.21.15.98:0/238675321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde2c19d3e0 con 0x7fde2c100700 2024-05-09T08:58:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde297fa640 1 -- 172.21.15.98:0/238675321 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c035070 con 0x7fde2c100700 2024-05-09T08:58:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.325+0000 7fde328e1640 1 -- 172.21.15.98:0/238675321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde2c1a4cd0 con 0x7fde2c100700 2024-05-09T08:58:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.328+0000 7fde297fa640 1 -- 172.21.15.98:0/238675321 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fde1c042030 con 0x7fde2c100700 2024-05-09T08:58:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.328+0000 7fde355f2740 1 Processor -- start 2024-05-09T08:58:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.328+0000 7fde355f2740 1 -- start start 2024-05-09T08:58:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.328+0000 7fde355f2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7330 0x5612312a7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.328+0000 7fde355f2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312a7c50 0x5612312b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.328+0000 7fde355f2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312b2320 0x5612312b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.328+0000 7fde355f2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56123120cc40 con 0x5612312b2320 2024-05-09T08:58:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.328+0000 7fde355f2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5612311b4580 con 0x5612312a7c50 2024-05-09T08:58:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.328+0000 7fde355f2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561231208c00 con 0x5612312a7330 2024-05-09T08:58:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.328+0000 7fde2b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312b2320 0x5612312b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.328+0000 7fde2b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312b2320 0x5612312b4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57200/0 (socket says 172.21.15.98:57200) 2024-05-09T08:58:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.328+0000 7fde2b7fe640 1 -- 172.21.15.98:0/2860314139 learned_addr learned my addr 172.21.15.98:0/2860314139 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.328+0000 7fde297fa640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fde180a2d80 0x7fde180a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde2bfff640 1 --2- 172.21.15.98:0/2860314139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312a7c50 0x5612312b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde30e7e640 1 --2- 172.21.15.98:0/2860314139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7330 0x5612312a7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde30e7e640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fde180a2d80 0x7fde180a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde2b7fe640 1 -- 172.21.15.98:0/2860314139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7330 msgr2=0x5612312a7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde2b7fe640 1 --2- 172.21.15.98:0/2860314139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7330 0x5612312a7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde2b7fe640 1 -- 172.21.15.98:0/2860314139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312a7c50 msgr2=0x5612312b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde2b7fe640 1 --2- 172.21.15.98:0/2860314139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312a7c50 0x5612312b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde2b7fe640 1 -- 172.21.15.98:0/2860314139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561230f203f0 con 0x5612312b2320 2024-05-09T08:58:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde2bfff640 1 --2- 172.21.15.98:0/2860314139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312a7c50 0x5612312b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde297fa640 1 -- 172.21.15.98:0/238675321 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fde1c0efdb0 con 0x7fde2c100700 2024-05-09T08:58:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde2b7fe640 1 --2- 172.21.15.98:0/2860314139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312b2320 0x5612312b4710 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fde14009c20 tx=0x7fde1400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc857d7b4e06279 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde227fc640 1 -- 172.21.15.98:0/2860314139 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14016070 con 0x5612312b2320 2024-05-09T08:58:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde30e7e640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fde180a2d80 0x7fde180a5240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fde2400c4f0 tx=0x7fde24008220 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde227fc640 1 -- 172.21.15.98:0/2860314139 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde14011950 con 0x5612312b2320 2024-05-09T08:58:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.329+0000 7fde227fc640 1 -- 172.21.15.98:0/2860314139 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14010870 con 0x5612312b2320 2024-05-09T08:58:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.330+0000 7fde355f2740 1 -- 172.21.15.98:0/2860314139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312b2320 msgr2=0x5612312b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.330+0000 7fde355f2740 1 --2- 172.21.15.98:0/2860314139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312b2320 0x5612312b4710 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fde14009c20 tx=0x7fde1400e6c0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.330+0000 7fde355f2740 1 -- 172.21.15.98:0/2860314139 shutdown_connections 2024-05-09T08:58:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.330+0000 7fde355f2740 1 --2- 172.21.15.98:0/2860314139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312b2320 0x5612312b4710 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.330+0000 7fde355f2740 1 --2- 172.21.15.98:0/2860314139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312a7c50 0x5612312b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.330+0000 7fde355f2740 1 --2- 172.21.15.98:0/2860314139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7330 0x5612312a7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.330+0000 7fde355f2740 1 -- 172.21.15.98:0/2860314139 >> 172.21.15.98:0/2860314139 conn(0x5612312a3650 msgr2=0x5612312a3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.330+0000 7fde355f2740 1 -- 172.21.15.98:0/2860314139 shutdown_connections 2024-05-09T08:58:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.330+0000 7fde355f2740 1 -- 172.21.15.98:0/2860314139 wait complete. 2024-05-09T08:58:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.330+0000 7fde355f2740 1 -- start start 2024-05-09T08:58:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.330+0000 7fde355f2740 1 Processor -- start 2024-05-09T08:58:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde355f2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312a7330 0x5612312af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde355f2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7c50 0x5612312af740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde355f2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312b2320 0x5612312afc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde355f2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5612312b5110 con 0x5612312a7330 2024-05-09T08:58:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde355f2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5612311c7c40 con 0x5612312b2320 2024-05-09T08:58:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde355f2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5612312b5290 con 0x5612312a7c50 2024-05-09T08:58:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde355f2740 10 client.0 _ll_register_callbacks cb 0x561230f1fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde2bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312a7330 0x5612312af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde2bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312a7330 0x5612312af200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57212/0 (socket says 172.21.15.98:57212) 2024-05-09T08:58:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde2bfff640 1 -- 172.21.15.98:0/2655635693 learned_addr learned my addr 172.21.15.98:0/2655635693 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde2b7fe640 1 --2- 172.21.15.98:0/2655635693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7c50 0x5612312af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde2bfff640 1 -- 172.21.15.98:0/2655635693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7c50 msgr2=0x5612312af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde2bfff640 1 --2- 172.21.15.98:0/2655635693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7c50 0x5612312af740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde2bfff640 1 -- 172.21.15.98:0/2655635693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312b2320 msgr2=0x5612312afc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde2bfff640 1 --2- 172.21.15.98:0/2655635693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312b2320 0x5612312afc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde2bfff640 1 -- 172.21.15.98:0/2655635693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561230fea7d0 con 0x5612312a7330 2024-05-09T08:58:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde2bfff640 1 --2- 172.21.15.98:0/2655635693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312a7330 0x5612312af200 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fde1c08ddf0 tx=0x7fde1c0bb880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61aa35a654a92068 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde0b7fe640 1 -- 172.21.15.98:0/2655635693 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c049040 con 0x5612312a7330 2024-05-09T08:58:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde0b7fe640 1 -- 172.21.15.98:0/2655635693 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fde1c0b84d0 con 0x5612312a7330 2024-05-09T08:58:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde355f2740 1 -- 172.21.15.98:0/2655635693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde14011950 con 0x5612312a7330 2024-05-09T08:58:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde0b7fe640 1 -- 172.21.15.98:0/2655635693 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fde1c0da050 con 0x5612312a7330 2024-05-09T08:58:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde355f2740 10 client.17616 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.331+0000 7fde355f2740 1 -- 172.21.15.98:0/2655635693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5612312c1200 con 0x5612312a7330 2024-05-09T08:58:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.332+0000 7fde0b7fe640 1 -- 172.21.15.98:0/2655635693 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde14011950 con 0x5612312a7330 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.332+0000 7fde0b7fe640 1 -- 172.21.15.98:0/2655635693 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fde1c0bc0c0 con 0x5612312a7330 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.332+0000 7fde355f2740 10 client.17616 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.332+0000 7fde355f2740 10 client.17616 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.332+0000 7fde355f2740 20 client.17616 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.332+0000 7fde355f2740 1 --2- 172.21.15.98:0/2655635693 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5612312c2610 0x5612312c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.332+0000 7fde355f2740 4 client.17616 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.332+0000 7fde355f2740 1 -- 172.21.15.98:0/2655635693 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56123121a370 con 0x5612312c2610 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.332+0000 7fde2b7fe640 1 --2- 172.21.15.98:0/2655635693 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5612312c2610 0x5612312c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.332+0000 7fde0b7fe640 10 client.17616 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.332+0000 7fde2b7fe640 1 --2- 172.21.15.98:0/2655635693 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5612312c2610 0x5612312c4a80 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e3ed56d3560ccdec server_cookie=fe71fe8b6238cdf5 in_seq=0 out_seq=0 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.333+0000 7fde0b7fe640 1 -- 172.21.15.98:0/2655635693 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56123121a370 con 0x5612312c2610 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.333+0000 7fde0b7fe640 10 client.17616 handle_command_reply: tid=0 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.333+0000 7fde355f2740 1 client.17616 shutdown 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.333+0000 7fde355f2740 1 -- 172.21.15.98:0/2655635693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312a7330 msgr2=0x5612312af200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.333+0000 7fde355f2740 1 --2- 172.21.15.98:0/2655635693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312a7330 0x5612312af200 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fde1c08ddf0 tx=0x7fde1c0bb880 comp rx=0 tx=0).stop 2024-05-09T08:58:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.333+0000 7fde355f2740 1 -- 172.21.15.98:0/2655635693 shutdown_connections 2024-05-09T08:58:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.333+0000 7fde355f2740 1 --2- 172.21.15.98:0/2655635693 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5612312c2610 0x5612312c4a80 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.333+0000 7fde355f2740 1 --2- 172.21.15.98:0/2655635693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312b2320 0x5612312afc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.333+0000 7fde0b7fe640 0 client.17616 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:58:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.333+0000 7fde355f2740 1 --2- 172.21.15.98:0/2655635693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7c50 0x5612312af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.333+0000 7fde355f2740 1 --2- 172.21.15.98:0/2655635693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312a7330 0x5612312af200 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.333+0000 7fde355f2740 1 -- 172.21.15.98:0/2655635693 >> 172.21.15.98:0/2655635693 conn(0x5612312a3650 msgr2=0x5612312a9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.333+0000 7fde355f2740 1 -- 172.21.15.98:0/2655635693 shutdown_connections 2024-05-09T08:58:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.333+0000 7fde355f2740 1 -- 172.21.15.98:0/2655635693 wait complete. 2024-05-09T08:58:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.334+0000 7fde355f2740 20 client.17616 trim_cache size 0 max 16384 2024-05-09T08:58:12.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:12 smithi098 ceph-mon[30092]: pgmap v234: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-09T08:58:12.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:12 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2251245057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:58:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.346+0000 7fde355f2740 1 Processor -- start 2024-05-09T08:58:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde355f2740 1 -- start start 2024-05-09T08:58:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde355f2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7330 0x5612312b1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde355f2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312a7c50 0x5612312be000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde355f2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312b2320 0x5612312ba8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde355f2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c1aef30 con 0x5612312a7c50 2024-05-09T08:58:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde355f2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c1b0a70 con 0x5612312b2320 2024-05-09T08:58:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde355f2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c1af600 con 0x5612312a7330 2024-05-09T08:58:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde2b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312a7c50 0x5612312be000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde2b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312a7c50 0x5612312be000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57222/0 (socket says 172.21.15.98:57222) 2024-05-09T08:58:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde2b7fe640 1 -- 172.21.15.98:0/2951998633 learned_addr learned my addr 172.21.15.98:0/2951998633 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde2bfff640 1 --2- 172.21.15.98:0/2951998633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7330 0x5612312b1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde2b7fe640 1 -- 172.21.15.98:0/2951998633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7330 msgr2=0x5612312b1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde2b7fe640 1 --2- 172.21.15.98:0/2951998633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7330 0x5612312b1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde2b7fe640 1 -- 172.21.15.98:0/2951998633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312b2320 msgr2=0x5612312ba8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde30e7e640 1 --2- 172.21.15.98:0/2951998633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312b2320 0x5612312ba8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde2b7fe640 1 --2- 172.21.15.98:0/2951998633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312b2320 0x5612312ba8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde2b7fe640 1 -- 172.21.15.98:0/2951998633 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612312a3480 con 0x5612312a7c50 2024-05-09T08:58:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde30e7e640 1 --2- 172.21.15.98:0/2951998633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312b2320 0x5612312ba8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde2b7fe640 1 --2- 172.21.15.98:0/2951998633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312a7c50 0x5612312be000 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x5612312a47e0 tx=0x7fde14009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=165b55d733546f61 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde21ffb640 1 -- 172.21.15.98:0/2951998633 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1402d020 con 0x5612312a7c50 2024-05-09T08:58:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde21ffb640 1 -- 172.21.15.98:0/2951998633 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde14004620 con 0x5612312a7c50 2024-05-09T08:58:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.347+0000 7fde21ffb640 1 -- 172.21.15.98:0/2951998633 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14016040 con 0x5612312a7c50 2024-05-09T08:58:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.348+0000 7fde355f2740 1 -- 172.21.15.98:0/2951998633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312a7c50 msgr2=0x5612312be000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.348+0000 7fde355f2740 1 --2- 172.21.15.98:0/2951998633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312a7c50 0x5612312be000 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x5612312a47e0 tx=0x7fde14009760 comp rx=0 tx=0).stop 2024-05-09T08:58:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.348+0000 7fde355f2740 1 -- 172.21.15.98:0/2951998633 shutdown_connections 2024-05-09T08:58:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.348+0000 7fde355f2740 1 --2- 172.21.15.98:0/2951998633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312b2320 0x5612312ba8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.348+0000 7fde355f2740 1 --2- 172.21.15.98:0/2951998633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312a7c50 0x5612312be000 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.348+0000 7fde355f2740 1 --2- 172.21.15.98:0/2951998633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7330 0x5612312b1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.348+0000 7fde355f2740 1 -- 172.21.15.98:0/2951998633 >> 172.21.15.98:0/2951998633 conn(0x5612312a3650 msgr2=0x5612312a27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.348+0000 7fde355f2740 1 -- 172.21.15.98:0/2951998633 shutdown_connections 2024-05-09T08:58:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.348+0000 7fde355f2740 1 -- 172.21.15.98:0/2951998633 wait complete. 2024-05-09T08:58:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.348+0000 7fde355f2740 1 -- start start 2024-05-09T08:58:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde355f2740 1 Processor -- start 2024-05-09T08:58:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde355f2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312a7330 0x5612312ac6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde355f2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7c50 0x5612312acc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde355f2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312b2320 0x5612312ab470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde355f2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c1b03a0 con 0x5612312b2320 2024-05-09T08:58:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde355f2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c1afcd0 con 0x5612312a7330 2024-05-09T08:58:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde355f2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c1acf30 con 0x5612312a7c50 2024-05-09T08:58:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde355f2740 10 client.0 _ll_register_callbacks cb 0x7fde1c0b8160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde30e7e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312b2320 0x5612312ab470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde30e7e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312b2320 0x5612312ab470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57224/0 (socket says 172.21.15.98:57224) 2024-05-09T08:58:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde30e7e640 1 -- 172.21.15.98:0/4070778813 learned_addr learned my addr 172.21.15.98:0/4070778813 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde2b7fe640 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7c50 0x5612312acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde2bfff640 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312a7330 0x5612312ac6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde30e7e640 1 -- 172.21.15.98:0/4070778813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7c50 msgr2=0x5612312acc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde30e7e640 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7c50 0x5612312acc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde30e7e640 1 -- 172.21.15.98:0/4070778813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312a7330 msgr2=0x5612312ac6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde30e7e640 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312a7330 0x5612312ac6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde30e7e640 1 -- 172.21.15.98:0/4070778813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612312a4810 con 0x5612312b2320 2024-05-09T08:58:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde2b7fe640 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7c50 0x5612312acc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde2bfff640 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312a7330 0x5612312ac6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde30e7e640 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312b2320 0x5612312ab470 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fde2400eb60 tx=0x7fde24004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2941a39a83e699c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.349+0000 7fde0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde0b7fe640 1 -- 172.21.15.98:0/4070778813 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2401ab20 con 0x5612312b2320 2024-05-09T08:58:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde0b7fe640 1 -- 172.21.15.98:0/4070778813 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fde2401acc0 con 0x5612312b2320 2024-05-09T08:58:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde355f2740 1 -- 172.21.15.98:0/4070778813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde14004620 con 0x5612312b2320 2024-05-09T08:58:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde0b7fe640 1 -- 172.21.15.98:0/4070778813 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fde2402d050 con 0x5612312b2320 2024-05-09T08:58:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde355f2740 10 client.17622 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde355f2740 1 -- 172.21.15.98:0/4070778813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56123132d370 con 0x5612312b2320 2024-05-09T08:58:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde0b7fe640 1 -- 172.21.15.98:0/4070778813 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde14004620 con 0x5612312b2320 2024-05-09T08:58:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde0b7fe640 1 -- 172.21.15.98:0/4070778813 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fde24021030 con 0x5612312b2320 2024-05-09T08:58:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde355f2740 10 client.17622 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde355f2740 10 client.17622 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:58:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde355f2740 20 client.17622 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde355f2740 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x56123132db70 0x56123132ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde355f2740 4 client.17622 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:58:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde355f2740 1 -- 172.21.15.98:0/4070778813 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5612312146c0 con 0x56123132db70 2024-05-09T08:58:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.350+0000 7fde2bfff640 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x56123132db70 0x56123132ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde0b7fe640 10 client.17622 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:58:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde2bfff640 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x56123132db70 0x56123132ff90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fa93ff46aa6cf2b1 server_cookie=9f93d264c566ef6d in_seq=0 out_seq=0 2024-05-09T08:58:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde0b7fe640 1 -- 172.21.15.98:0/4070778813 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5612312146c0 con 0x56123132db70 2024-05-09T08:58:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde0b7fe640 10 client.17622 handle_command_reply: tid=0 2024-05-09T08:58:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde355f2740 1 client.17622 shutdown 2024-05-09T08:58:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde355f2740 1 -- 172.21.15.98:0/4070778813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312b2320 msgr2=0x5612312ab470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde355f2740 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312b2320 0x5612312ab470 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fde2400eb60 tx=0x7fde24004b50 comp rx=0 tx=0).stop 2024-05-09T08:58:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde355f2740 1 -- 172.21.15.98:0/4070778813 shutdown_connections 2024-05-09T08:58:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde355f2740 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x56123132db70 0x56123132ff90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde355f2740 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612312b2320 0x5612312ab470 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde0b7fe640 0 client.17622 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:58:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde355f2740 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5612312a7c50 0x5612312acc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde355f2740 1 --2- 172.21.15.98:0/4070778813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5612312a7330 0x5612312ac6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde355f2740 1 -- 172.21.15.98:0/4070778813 >> 172.21.15.98:0/4070778813 conn(0x5612312a3650 msgr2=0x5612312af1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde355f2740 1 -- 172.21.15.98:0/4070778813 shutdown_connections 2024-05-09T08:58:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde355f2740 1 -- 172.21.15.98:0/4070778813 wait complete. 2024-05-09T08:58:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.351+0000 7fde355f2740 20 client.17622 trim_cache size 0 max 16384 2024-05-09T08:58:12.360 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:58:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.352+0000 7fde328e1640 1 -- 172.21.15.98:0/238675321 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fde180a2d80 msgr2=0x7fde180a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.352+0000 7fde328e1640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fde180a2d80 0x7fde180a5240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fde2400c4f0 tx=0x7fde24008220 comp rx=0 tx=0).stop 2024-05-09T08:58:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.352+0000 7fde328e1640 1 -- 172.21.15.98:0/238675321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde2c100700 msgr2=0x7fde2c198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.352+0000 7fde328e1640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde2c100700 0x7fde2c198160 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fde1c009230 tx=0x7fde1c004270 comp rx=0 tx=0).stop 2024-05-09T08:58:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.352+0000 7fde328e1640 1 -- 172.21.15.98:0/238675321 shutdown_connections 2024-05-09T08:58:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.352+0000 7fde328e1640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fde180a2d80 0x7fde180a5240 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.352+0000 7fde328e1640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde2c106710 0x7fde2c19ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.352+0000 7fde328e1640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde2c1010c0 0x7fde2c1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.352+0000 7fde328e1640 1 --2- 172.21.15.98:0/238675321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde2c100700 0x7fde2c198160 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.352+0000 7fde328e1640 1 -- 172.21.15.98:0/238675321 >> 172.21.15.98:0/238675321 conn(0x7fde2c077830 msgr2=0x7fde2c102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.352+0000 7fde328e1640 1 -- 172.21.15.98:0/238675321 shutdown_connections 2024-05-09T08:58:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.352+0000 7fde328e1640 1 -- 172.21.15.98:0/238675321 wait complete. 2024-05-09T08:58:12.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:58:12.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.466+0000 7fd523529640 1 -- 172.21.15.98:0/778319016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd51c100700 msgr2=0x7fd51c100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.466+0000 7fd523529640 1 --2- 172.21.15.98:0/778319016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd51c100700 0x7fd51c100b80 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fd50c009930 tx=0x7fd50c02a420 comp rx=0 tx=0).stop 2024-05-09T08:58:12.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.466+0000 7fd523529640 1 -- 172.21.15.98:0/778319016 shutdown_connections 2024-05-09T08:58:12.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.466+0000 7fd523529640 1 --2- 172.21.15.98:0/778319016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd51c1010c0 0x7fd51c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.466+0000 7fd523529640 1 --2- 172.21.15.98:0/778319016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd51c100700 0x7fd51c100b80 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.466+0000 7fd523529640 1 --2- 172.21.15.98:0/778319016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd51c106710 0x7fd51c106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.466+0000 7fd523529640 1 -- 172.21.15.98:0/778319016 >> 172.21.15.98:0/778319016 conn(0x7fd51c076ab0 msgr2=0x7fd51c076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.467+0000 7fd523529640 1 -- 172.21.15.98:0/778319016 shutdown_connections 2024-05-09T08:58:12.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.467+0000 7fd523529640 1 -- 172.21.15.98:0/778319016 wait complete. 2024-05-09T08:58:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.467+0000 7fd523529640 1 Processor -- start 2024-05-09T08:58:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.467+0000 7fd523529640 1 -- start start 2024-05-09T08:58:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.467+0000 7fd523529640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd51c100700 0x7fd51c198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.467+0000 7fd523529640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd51c1010c0 0x7fd51c1994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.467+0000 7fd523529640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd51c106710 0x7fd51c1a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.467+0000 7fd523529640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c110b20 con 0x7fd51c1010c0 2024-05-09T08:58:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.467+0000 7fd523529640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c1109a0 con 0x7fd51c106710 2024-05-09T08:58:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.467+0000 7fd523529640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c110ca0 con 0x7fd51c100700 2024-05-09T08:58:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.467+0000 7fd520ac4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd51c1010c0 0x7fd51c1994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.467+0000 7fd520ac4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd51c1010c0 0x7fd51c1994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57242/0 (socket says 172.21.15.98:57242) 2024-05-09T08:58:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.467+0000 7fd520ac4640 1 -- 172.21.15.98:0/1263547728 learned_addr learned my addr 172.21.15.98:0/1263547728 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.468+0000 7fd521ac6640 1 --2- 172.21.15.98:0/1263547728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd51c106710 0x7fd51c1a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.468+0000 7fd520ac4640 1 -- 172.21.15.98:0/1263547728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd51c100700 msgr2=0x7fd51c198f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.468+0000 7fd520ac4640 1 --2- 172.21.15.98:0/1263547728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd51c100700 0x7fd51c198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.468+0000 7fd520ac4640 1 -- 172.21.15.98:0/1263547728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd51c106710 msgr2=0x7fd51c1a3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.468+0000 7fd520ac4640 1 --2- 172.21.15.98:0/1263547728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd51c106710 0x7fd51c1a3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.468+0000 7fd520ac4640 1 -- 172.21.15.98:0/1263547728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51c1a4520 con 0x7fd51c1010c0 2024-05-09T08:58:12.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.468+0000 7fd521ac6640 1 --2- 172.21.15.98:0/1263547728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd51c106710 0x7fd51c1a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:12.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.468+0000 7fd520ac4640 1 --2- 172.21.15.98:0/1263547728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd51c1010c0 0x7fd51c1994a0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd50c009230 tx=0x7fd50c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.468+0000 7fd5127fc640 1 -- 172.21.15.98:0/1263547728 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c03f030 con 0x7fd51c1010c0 2024-05-09T08:58:12.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.468+0000 7fd523529640 1 -- 172.21.15.98:0/1263547728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd51c1a46f0 con 0x7fd51c1010c0 2024-05-09T08:58:12.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.468+0000 7fd5127fc640 1 -- 172.21.15.98:0/1263547728 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd50c003ad0 con 0x7fd51c1010c0 2024-05-09T08:58:12.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.468+0000 7fd5127fc640 1 -- 172.21.15.98:0/1263547728 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c035070 con 0x7fd51c1010c0 2024-05-09T08:58:12.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.468+0000 7fd523529640 1 -- 172.21.15.98:0/1263547728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd51c1a4bb0 con 0x7fd51c1010c0 2024-05-09T08:58:12.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.470+0000 7fd5127fc640 1 -- 172.21.15.98:0/1263547728 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd50c042030 con 0x7fd51c1010c0 2024-05-09T08:58:12.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd52623a740 1 Processor -- start 2024-05-09T08:58:12.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd52623a740 1 -- start start 2024-05-09T08:58:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd5127fc640 1 --2- 172.21.15.98:0/1263547728 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd5080a2d00 0x7fd5080a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd52623a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc0b330 0x55f29dc0b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd52623a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 0x55f29dc15de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd52623a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc16320 0x55f29dc18710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd52623a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f29db70c40 con 0x55f29dc16320 2024-05-09T08:58:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd52623a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f29db18580 con 0x55f29dc0b330 2024-05-09T08:58:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd52623a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f29db6cc00 con 0x55f29dc0bc50 2024-05-09T08:58:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd521ac6640 1 --2- 172.21.15.98:0/1263547728 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd5080a2d00 0x7fd5080a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd521ac6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc16320 0x55f29dc18710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd5127fc640 1 -- 172.21.15.98:0/1263547728 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fd50c0efe80 con 0x7fd51c1010c0 2024-05-09T08:58:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd521ac6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc16320 0x55f29dc18710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57252/0 (socket says 172.21.15.98:57252) 2024-05-09T08:58:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd521ac6640 1 -- 172.21.15.98:0/2950326540 learned_addr learned my addr 172.21.15.98:0/2950326540 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd520ac4640 1 --2- 172.21.15.98:0/2950326540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 0x55f29dc15de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd5212c5640 1 --2- 172.21.15.98:0/2950326540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc0b330 0x55f29dc0b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.471+0000 7fd520ac4640 1 -- 172.21.15.98:0/2950326540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc0b330 msgr2=0x55f29dc0b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.472+0000 7fd520ac4640 1 --2- 172.21.15.98:0/2950326540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc0b330 0x55f29dc0b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.472+0000 7fd520ac4640 1 -- 172.21.15.98:0/2950326540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc16320 msgr2=0x55f29dc18710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.472+0000 7fd520ac4640 1 --2- 172.21.15.98:0/2950326540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc16320 0x55f29dc18710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.472+0000 7fd520ac4640 1 -- 172.21.15.98:0/2950326540 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f29d8843f0 con 0x55f29dc0bc50 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.472+0000 7fd521ac6640 1 --2- 172.21.15.98:0/2950326540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc16320 0x55f29dc18710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.472+0000 7fd521ac6640 1 --2- 172.21.15.98:0/1263547728 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd5080a2d00 0x7fd5080a51c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd514012790 tx=0x7fd51400d9d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.472+0000 7fd520ac4640 1 --2- 172.21.15.98:0/2950326540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 0x55f29dc15de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0ed160 tx=0x7fd50c02a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfe6d15115d80818 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.472+0000 7fd5077fe640 1 -- 172.21.15.98:0/2950326540 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c0ce840 con 0x55f29dc0bc50 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.472+0000 7fd5077fe640 1 -- 172.21.15.98:0/2950326540 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd50c0bbf30 con 0x55f29dc0bc50 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.472+0000 7fd5077fe640 1 -- 172.21.15.98:0/2950326540 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c003c80 con 0x55f29dc0bc50 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 -- 172.21.15.98:0/2950326540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 msgr2=0x55f29dc15de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 --2- 172.21.15.98:0/2950326540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 0x55f29dc15de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0ed160 tx=0x7fd50c02a420 comp rx=0 tx=0).stop 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 -- 172.21.15.98:0/2950326540 shutdown_connections 2024-05-09T08:58:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 --2- 172.21.15.98:0/2950326540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc16320 0x55f29dc18710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 --2- 172.21.15.98:0/2950326540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 0x55f29dc15de0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 --2- 172.21.15.98:0/2950326540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc0b330 0x55f29dc0b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 -- 172.21.15.98:0/2950326540 >> 172.21.15.98:0/2950326540 conn(0x55f29dc07650 msgr2=0x55f29dc07a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 -- 172.21.15.98:0/2950326540 shutdown_connections 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 -- 172.21.15.98:0/2950326540 wait complete. 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 -- start start 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 Processor -- start 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0b330 0x55f29dc132b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 0x55f29dc137f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc16320 0x55f29dc13d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f29dc18ed0 con 0x55f29dc0b330 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f29db2bc40 con 0x55f29dc16320 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f29dc19050 con 0x55f29dc0bc50 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.473+0000 7fd52623a740 10 client.0 _ll_register_callbacks cb 0x55f29d883be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd5212c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0b330 0x55f29dc132b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd5212c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0b330 0x55f29dc132b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57264/0 (socket says 172.21.15.98:57264) 2024-05-09T08:58:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd5212c5640 1 -- 172.21.15.98:0/3324348349 learned_addr learned my addr 172.21.15.98:0/3324348349 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd521ac6640 1 --2- 172.21.15.98:0/3324348349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc16320 0x55f29dc13d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd520ac4640 1 --2- 172.21.15.98:0/3324348349 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 0x55f29dc137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd5212c5640 1 -- 172.21.15.98:0/3324348349 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 msgr2=0x55f29dc137f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd5212c5640 1 --2- 172.21.15.98:0/3324348349 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 0x55f29dc137f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd5212c5640 1 -- 172.21.15.98:0/3324348349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc16320 msgr2=0x55f29dc13d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd5212c5640 1 --2- 172.21.15.98:0/3324348349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc16320 0x55f29dc13d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd5212c5640 1 -- 172.21.15.98:0/3324348349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f29d94e7d0 con 0x55f29dc0b330 2024-05-09T08:58:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd5212c5640 1 --2- 172.21.15.98:0/3324348349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0b330 0x55f29dc132b0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fd518011690 tx=0x7fd518011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c37fabc0da4ed5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd504ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd504ff9640 1 -- 172.21.15.98:0/3324348349 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518004d20 con 0x55f29dc0b330 2024-05-09T08:58:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd504ff9640 1 -- 172.21.15.98:0/3324348349 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fd51800f940 con 0x55f29dc0b330 2024-05-09T08:58:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd52623a740 1 -- 172.21.15.98:0/3324348349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd50c0bbf30 con 0x55f29dc0b330 2024-05-09T08:58:12.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd504ff9640 1 -- 172.21.15.98:0/3324348349 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd518011e90 con 0x55f29dc0b330 2024-05-09T08:58:12.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd52623a740 10 client.17634 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:12.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd52623a740 1 -- 172.21.15.98:0/3324348349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f29dc25020 con 0x55f29dc0b330 2024-05-09T08:58:12.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.474+0000 7fd504ff9640 1 -- 172.21.15.98:0/3324348349 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd50c0bbf30 con 0x55f29dc0b330 2024-05-09T08:58:12.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.475+0000 7fd504ff9640 1 -- 172.21.15.98:0/3324348349 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd51800f040 con 0x55f29dc0b330 2024-05-09T08:58:12.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.475+0000 7fd52623a740 10 client.17634 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:12.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.475+0000 7fd52623a740 10 client.17634 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:58:12.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.475+0000 7fd52623a740 20 client.17634 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:12.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.475+0000 7fd52623a740 1 --2- 172.21.15.98:0/3324348349 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f29dc26530 0x55f29dc289a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.475+0000 7fd52623a740 4 client.17634 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:12.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.475+0000 7fd52623a740 1 -- 172.21.15.98:0/3324348349 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f29db7e370 con 0x55f29dc26530 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.475+0000 7fd520ac4640 1 --2- 172.21.15.98:0/3324348349 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f29dc26530 0x55f29dc289a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.475+0000 7fd504ff9640 10 client.17634 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.475+0000 7fd520ac4640 1 --2- 172.21.15.98:0/3324348349 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f29dc26530 0x55f29dc289a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=846bdf98fa9c409a server_cookie=b9572735ac44c2ea in_seq=0 out_seq=0 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd504ff9640 1 -- 172.21.15.98:0/3324348349 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f29db7e370 con 0x55f29dc26530 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd504ff9640 10 client.17634 handle_command_reply: tid=0 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd52623a740 1 client.17634 shutdown 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd52623a740 1 -- 172.21.15.98:0/3324348349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0b330 msgr2=0x55f29dc132b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd52623a740 1 --2- 172.21.15.98:0/3324348349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0b330 0x55f29dc132b0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fd518011690 tx=0x7fd518011b50 comp rx=0 tx=0).stop 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd52623a740 1 -- 172.21.15.98:0/3324348349 shutdown_connections 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd52623a740 1 --2- 172.21.15.98:0/3324348349 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f29dc26530 0x55f29dc289a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd52623a740 1 --2- 172.21.15.98:0/3324348349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc16320 0x55f29dc13d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd52623a740 1 --2- 172.21.15.98:0/3324348349 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 0x55f29dc137f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd52623a740 1 --2- 172.21.15.98:0/3324348349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0b330 0x55f29dc132b0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd504ff9640 0 client.17634 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:58:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd52623a740 1 -- 172.21.15.98:0/3324348349 >> 172.21.15.98:0/3324348349 conn(0x55f29dc07650 msgr2=0x55f29dc0d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd52623a740 1 -- 172.21.15.98:0/3324348349 shutdown_connections 2024-05-09T08:58:12.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd52623a740 1 -- 172.21.15.98:0/3324348349 wait complete. 2024-05-09T08:58:12.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.476+0000 7fd52623a740 20 client.17634 trim_cache size 0 max 16384 2024-05-09T08:58:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.489+0000 7fd52623a740 1 Processor -- start 2024-05-09T08:58:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.489+0000 7fd52623a740 1 -- start start 2024-05-09T08:58:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.489+0000 7fd52623a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0b330 0x55f29dc21b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.489+0000 7fd52623a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 0x55f29dc0da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd52623a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc16320 0x55f29dc2e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd52623a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c1af010 con 0x55f29dc0b330 2024-05-09T08:58:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd52623a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c1b0b50 con 0x55f29dc16320 2024-05-09T08:58:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd52623a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c1af6e0 con 0x55f29dc0bc50 2024-05-09T08:58:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd5212c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0b330 0x55f29dc21b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd5212c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0b330 0x55f29dc21b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57270/0 (socket says 172.21.15.98:57270) 2024-05-09T08:58:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd5212c5640 1 -- 172.21.15.98:0/145719219 learned_addr learned my addr 172.21.15.98:0/145719219 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd520ac4640 1 --2- 172.21.15.98:0/145719219 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 0x55f29dc0da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd5212c5640 1 -- 172.21.15.98:0/145719219 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 msgr2=0x55f29dc0da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd521ac6640 1 --2- 172.21.15.98:0/145719219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc16320 0x55f29dc2e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd5212c5640 1 --2- 172.21.15.98:0/145719219 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 0x55f29dc0da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd5212c5640 1 -- 172.21.15.98:0/145719219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc16320 msgr2=0x55f29dc2e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd5212c5640 1 --2- 172.21.15.98:0/145719219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc16320 0x55f29dc2e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd5212c5640 1 -- 172.21.15.98:0/145719219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f29dc14210 con 0x55f29dc0b330 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd520ac4640 1 --2- 172.21.15.98:0/145719219 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 0x55f29dc0da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd521ac6640 1 --2- 172.21.15.98:0/145719219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc16320 0x55f29dc2e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd5212c5640 1 --2- 172.21.15.98:0/145719219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0b330 0x55f29dc21b70 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fd518015020 tx=0x7fd518009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbeee1bc9094cc31 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd506ffd640 1 -- 172.21.15.98:0/145719219 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51802c020 con 0x55f29dc0b330 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd506ffd640 1 -- 172.21.15.98:0/145719219 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd51800f730 con 0x55f29dc0b330 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.490+0000 7fd506ffd640 1 -- 172.21.15.98:0/145719219 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51801a520 con 0x55f29dc0b330 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.491+0000 7fd52623a740 1 -- 172.21.15.98:0/145719219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0b330 msgr2=0x55f29dc21b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.491+0000 7fd52623a740 1 --2- 172.21.15.98:0/145719219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0b330 0x55f29dc21b70 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fd518015020 tx=0x7fd518009ca0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.491+0000 7fd52623a740 1 -- 172.21.15.98:0/145719219 shutdown_connections 2024-05-09T08:58:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.491+0000 7fd52623a740 1 --2- 172.21.15.98:0/145719219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc16320 0x55f29dc2e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.491+0000 7fd52623a740 1 --2- 172.21.15.98:0/145719219 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc0bc50 0x55f29dc0da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.491+0000 7fd52623a740 1 --2- 172.21.15.98:0/145719219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0b330 0x55f29dc21b70 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.491+0000 7fd52623a740 1 -- 172.21.15.98:0/145719219 >> 172.21.15.98:0/145719219 conn(0x55f29dc07650 msgr2=0x55f29dc13b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.491+0000 7fd52623a740 1 -- 172.21.15.98:0/145719219 shutdown_connections 2024-05-09T08:58:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.491+0000 7fd52623a740 1 -- 172.21.15.98:0/145719219 wait complete. 2024-05-09T08:58:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.491+0000 7fd52623a740 1 -- start start 2024-05-09T08:58:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.491+0000 7fd52623a740 1 Processor -- start 2024-05-09T08:58:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.491+0000 7fd52623a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc0b330 0x55f29dc0fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd52623a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0bc50 0x55f29dc10370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd52623a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc16320 0x55f29dc108b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd52623a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c1b0480 con 0x55f29dc0bc50 2024-05-09T08:58:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd52623a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c1afdb0 con 0x55f29dc0b330 2024-05-09T08:58:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd52623a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c1ad010 con 0x55f29dc16320 2024-05-09T08:58:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd52623a740 10 client.0 _ll_register_callbacks cb 0x7fd518009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd520ac4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0bc50 0x55f29dc10370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd520ac4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0bc50 0x55f29dc10370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57284/0 (socket says 172.21.15.98:57284) 2024-05-09T08:58:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd520ac4640 1 -- 172.21.15.98:0/4104470390 learned_addr learned my addr 172.21.15.98:0/4104470390 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd5212c5640 1 --2- 172.21.15.98:0/4104470390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc0b330 0x55f29dc0fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd521ac6640 1 --2- 172.21.15.98:0/4104470390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc16320 0x55f29dc108b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd520ac4640 1 -- 172.21.15.98:0/4104470390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc16320 msgr2=0x55f29dc108b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd520ac4640 1 --2- 172.21.15.98:0/4104470390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc16320 0x55f29dc108b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd520ac4640 1 -- 172.21.15.98:0/4104470390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc0b330 msgr2=0x55f29dc0fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd520ac4640 1 --2- 172.21.15.98:0/4104470390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc0b330 0x55f29dc0fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd520ac4640 1 -- 172.21.15.98:0/4104470390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f29dc8bae0 con 0x55f29dc0bc50 2024-05-09T08:58:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd520ac4640 1 --2- 172.21.15.98:0/4104470390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0bc50 0x55f29dc10370 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fd50c032c00 tx=0x7fd50c005a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ba7931b3ef4be server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd504ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd504ff9640 1 -- 172.21.15.98:0/4104470390 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c049040 con 0x55f29dc0bc50 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd504ff9640 1 -- 172.21.15.98:0/4104470390 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fd50c0ba230 con 0x55f29dc0bc50 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd52623a740 1 -- 172.21.15.98:0/4104470390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd51800f730 con 0x55f29dc0bc50 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd504ff9640 1 -- 172.21.15.98:0/4104470390 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd50c0c02e0 con 0x55f29dc0bc50 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd52623a740 10 client.17640 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd52623a740 1 -- 172.21.15.98:0/4104470390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f29dc8cbf0 con 0x55f29dc0bc50 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.492+0000 7fd504ff9640 1 -- 172.21.15.98:0/4104470390 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd50c0d7030 con 0x55f29dc0bc50 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.493+0000 7fd504ff9640 1 -- 172.21.15.98:0/4104470390 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd50c0ed3e0 con 0x55f29dc0bc50 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.493+0000 7fd52623a740 10 client.17640 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.493+0000 7fd52623a740 10 client.17640 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.493+0000 7fd52623a740 20 client.17640 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.493+0000 7fd52623a740 1 --2- 172.21.15.98:0/4104470390 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f29dc8d230 0x55f29dc8f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.493+0000 7fd52623a740 4 client.17640 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.493+0000 7fd52623a740 1 -- 172.21.15.98:0/4104470390 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f29db786c0 con 0x55f29dc8d230 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.493+0000 7fd5212c5640 1 --2- 172.21.15.98:0/4104470390 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f29dc8d230 0x55f29dc8f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.493+0000 7fd5212c5640 1 --2- 172.21.15.98:0/4104470390 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f29dc8d230 0x55f29dc8f650 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4698c023927b03c6 server_cookie=f824ab57fe4b601b in_seq=0 out_seq=0 2024-05-09T08:58:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.493+0000 7fd504ff9640 10 client.17640 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:58:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.493+0000 7fd504ff9640 1 -- 172.21.15.98:0/4104470390 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f29db786c0 con 0x55f29dc8d230 2024-05-09T08:58:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd504ff9640 10 client.17640 handle_command_reply: tid=0 2024-05-09T08:58:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd52623a740 1 client.17640 shutdown 2024-05-09T08:58:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd52623a740 1 -- 172.21.15.98:0/4104470390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0bc50 msgr2=0x55f29dc10370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd52623a740 1 --2- 172.21.15.98:0/4104470390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0bc50 0x55f29dc10370 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fd50c032c00 tx=0x7fd50c005a60 comp rx=0 tx=0).stop 2024-05-09T08:58:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd52623a740 1 -- 172.21.15.98:0/4104470390 shutdown_connections 2024-05-09T08:58:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd52623a740 1 --2- 172.21.15.98:0/4104470390 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55f29dc8d230 0x55f29dc8f650 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd52623a740 1 --2- 172.21.15.98:0/4104470390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f29dc16320 0x55f29dc108b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd504ff9640 0 client.17640 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:58:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd52623a740 1 --2- 172.21.15.98:0/4104470390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f29dc0bc50 0x55f29dc10370 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd52623a740 1 --2- 172.21.15.98:0/4104470390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29dc0b330 0x55f29dc0fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd52623a740 1 -- 172.21.15.98:0/4104470390 >> 172.21.15.98:0/4104470390 conn(0x55f29dc07650 msgr2=0x55f29dc05f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd52623a740 1 -- 172.21.15.98:0/4104470390 shutdown_connections 2024-05-09T08:58:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd52623a740 1 -- 172.21.15.98:0/4104470390 wait complete. 2024-05-09T08:58:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd52623a740 20 client.17640 trim_cache size 0 max 16384 2024-05-09T08:58:12.501 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:58:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd523529640 1 -- 172.21.15.98:0/1263547728 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd5080a2d00 msgr2=0x7fd5080a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd523529640 1 --2- 172.21.15.98:0/1263547728 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd5080a2d00 0x7fd5080a51c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd514012790 tx=0x7fd51400d9d0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd523529640 1 -- 172.21.15.98:0/1263547728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd51c1010c0 msgr2=0x7fd51c1994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd523529640 1 --2- 172.21.15.98:0/1263547728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd51c1010c0 0x7fd51c1994a0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd50c009230 tx=0x7fd50c004270 comp rx=0 tx=0).stop 2024-05-09T08:58:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd523529640 1 -- 172.21.15.98:0/1263547728 shutdown_connections 2024-05-09T08:58:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd523529640 1 --2- 172.21.15.98:0/1263547728 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd5080a2d00 0x7fd5080a51c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd523529640 1 --2- 172.21.15.98:0/1263547728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd51c106710 0x7fd51c1a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd523529640 1 --2- 172.21.15.98:0/1263547728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd51c1010c0 0x7fd51c1994a0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd523529640 1 --2- 172.21.15.98:0/1263547728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd51c100700 0x7fd51c198f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.494+0000 7fd523529640 1 -- 172.21.15.98:0/1263547728 >> 172.21.15.98:0/1263547728 conn(0x7fd51c076ab0 msgr2=0x7fd51c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.495+0000 7fd523529640 1 -- 172.21.15.98:0/1263547728 shutdown_connections 2024-05-09T08:58:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.495+0000 7fd523529640 1 -- 172.21.15.98:0/1263547728 wait complete. 2024-05-09T08:58:12.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:58:12.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.608+0000 7fd16339e640 1 -- 172.21.15.98:0/2607423765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c106710 msgr2=0x7fd15c106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.608+0000 7fd16339e640 1 --2- 172.21.15.98:0/2607423765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c106710 0x7fd15c106af0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fd158009930 tx=0x7fd15802a420 comp rx=0 tx=0).stop 2024-05-09T08:58:12.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.608+0000 7fd16339e640 1 -- 172.21.15.98:0/2607423765 shutdown_connections 2024-05-09T08:58:12.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.608+0000 7fd16339e640 1 --2- 172.21.15.98:0/2607423765 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd15c1010c0 0x7fd15c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.608+0000 7fd16339e640 1 --2- 172.21.15.98:0/2607423765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd15c100700 0x7fd15c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.608+0000 7fd16339e640 1 --2- 172.21.15.98:0/2607423765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c106710 0x7fd15c106af0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.608+0000 7fd16339e640 1 -- 172.21.15.98:0/2607423765 >> 172.21.15.98:0/2607423765 conn(0x7fd15c077830 msgr2=0x7fd15c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.608+0000 7fd16339e640 1 -- 172.21.15.98:0/2607423765 shutdown_connections 2024-05-09T08:58:12.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.608+0000 7fd16339e640 1 -- 172.21.15.98:0/2607423765 wait complete. 2024-05-09T08:58:12.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.609+0000 7fd16339e640 1 Processor -- start 2024-05-09T08:58:12.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.609+0000 7fd16339e640 1 -- start start 2024-05-09T08:58:12.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.609+0000 7fd16339e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c100700 0x7fd15c198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.609+0000 7fd16339e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd15c1010c0 0x7fd15c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.609+0000 7fd16339e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd15c106710 0x7fd15c19c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.609+0000 7fd16339e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd15c110b20 con 0x7fd15c100700 2024-05-09T08:58:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.609+0000 7fd16339e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd15c1109a0 con 0x7fd15c1010c0 2024-05-09T08:58:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.609+0000 7fd16339e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd15c110ca0 con 0x7fd15c106710 2024-05-09T08:58:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.609+0000 7fd16113a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c100700 0x7fd15c198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.609+0000 7fd16113a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c100700 0x7fd15c198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57302/0 (socket says 172.21.15.98:57302) 2024-05-09T08:58:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.609+0000 7fd16113a640 1 -- 172.21.15.98:0/1637615951 learned_addr learned my addr 172.21.15.98:0/1637615951 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.609+0000 7fd160939640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd15c1010c0 0x7fd15c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.610+0000 7fd16113a640 1 -- 172.21.15.98:0/1637615951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd15c106710 msgr2=0x7fd15c19c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.610+0000 7fd16193b640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd15c106710 0x7fd15c19c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.610+0000 7fd16113a640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd15c106710 0x7fd15c19c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.610+0000 7fd16113a640 1 -- 172.21.15.98:0/1637615951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd15c1010c0 msgr2=0x7fd15c198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.610+0000 7fd16113a640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd15c1010c0 0x7fd15c198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.610+0000 7fd16113a640 1 -- 172.21.15.98:0/1637615951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15c19d120 con 0x7fd15c100700 2024-05-09T08:58:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.610+0000 7fd160939640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd15c1010c0 0x7fd15c198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.610+0000 7fd16193b640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd15c106710 0x7fd15c19c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.610+0000 7fd16113a640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c100700 0x7fd15c198130 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fd158009a50 tx=0x7fd158004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.610+0000 7fd1527fc640 1 -- 172.21.15.98:0/1637615951 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15803f030 con 0x7fd15c100700 2024-05-09T08:58:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.610+0000 7fd16339e640 1 -- 172.21.15.98:0/1637615951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd15c19d3b0 con 0x7fd15c100700 2024-05-09T08:58:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.610+0000 7fd1527fc640 1 -- 172.21.15.98:0/1637615951 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd158003ce0 con 0x7fd15c100700 2024-05-09T08:58:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.610+0000 7fd1527fc640 1 -- 172.21.15.98:0/1637615951 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd158035070 con 0x7fd15c100700 2024-05-09T08:58:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.610+0000 7fd16339e640 1 -- 172.21.15.98:0/1637615951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd15c1a4ca0 con 0x7fd15c100700 2024-05-09T08:58:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.612+0000 7fd1527fc640 1 -- 172.21.15.98:0/1637615951 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd158042030 con 0x7fd15c100700 2024-05-09T08:58:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.613+0000 7fd166187740 1 Processor -- start 2024-05-09T08:58:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.613+0000 7fd166187740 1 -- start start 2024-05-09T08:58:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.613+0000 7fd166187740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6330 0x55c7be0f6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.613+0000 7fd166187740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6c50 0x55c7be100de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.613+0000 7fd166187740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be101320 0x55c7be103710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.613+0000 7fd166187740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c7be05bc40 con 0x55c7be0f6330 2024-05-09T08:58:12.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.613+0000 7fd166187740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c7be003580 con 0x55c7be101320 2024-05-09T08:58:12.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.613+0000 7fd166187740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c7be057c00 con 0x55c7be0f6c50 2024-05-09T08:58:12.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.613+0000 7fd16193b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6330 0x55c7be0f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.613+0000 7fd16193b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6330 0x55c7be0f6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57318/0 (socket says 172.21.15.98:57318) 2024-05-09T08:58:12.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.613+0000 7fd16193b640 1 -- 172.21.15.98:0/4060415492 learned_addr learned my addr 172.21.15.98:0/4060415492 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.613+0000 7fd16113a640 1 --2- 172.21.15.98:0/4060415492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6c50 0x55c7be100de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd160939640 1 --2- 172.21.15.98:0/4060415492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be101320 0x55c7be103710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd1527fc640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd1480a2d00 0x7fd1480a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd16193b640 1 -- 172.21.15.98:0/4060415492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6c50 msgr2=0x55c7be100de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd16193b640 1 --2- 172.21.15.98:0/4060415492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6c50 0x55c7be100de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd16193b640 1 -- 172.21.15.98:0/4060415492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be101320 msgr2=0x55c7be103710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd16193b640 1 --2- 172.21.15.98:0/4060415492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be101320 0x55c7be103710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd16193b640 1 -- 172.21.15.98:0/4060415492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7bdd6f3f0 con 0x55c7be0f6330 2024-05-09T08:58:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd160939640 1 --2- 172.21.15.98:0/4060415492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be101320 0x55c7be103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd16113a640 1 --2- 172.21.15.98:0/4060415492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6c50 0x55c7be100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd16193b640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd1480a2d00 0x7fd1480a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd1527fc640 1 -- 172.21.15.98:0/1637615951 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fd1580eff00 con 0x7fd15c100700 2024-05-09T08:58:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd16193b640 1 --2- 172.21.15.98:0/4060415492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6330 0x55c7be0f6710 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd15400c370 tx=0x7fd15400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c3c01f63e8bc2d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd1477fe640 1 -- 172.21.15.98:0/4060415492 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd154016070 con 0x55c7be0f6330 2024-05-09T08:58:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd1477fe640 1 -- 172.21.15.98:0/4060415492 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd154011d00 con 0x55c7be0f6330 2024-05-09T08:58:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.614+0000 7fd1477fe640 1 -- 172.21.15.98:0/4060415492 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15401bc20 con 0x55c7be0f6330 2024-05-09T08:58:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.615+0000 7fd166187740 1 -- 172.21.15.98:0/4060415492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6330 msgr2=0x55c7be0f6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.615+0000 7fd16193b640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd1480a2d00 0x7fd1480a51c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd15401d000 tx=0x7fd15401e000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.615+0000 7fd166187740 1 --2- 172.21.15.98:0/4060415492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6330 0x55c7be0f6710 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd15400c370 tx=0x7fd15400c830 comp rx=0 tx=0).stop 2024-05-09T08:58:12.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.615+0000 7fd166187740 1 -- 172.21.15.98:0/4060415492 shutdown_connections 2024-05-09T08:58:12.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.615+0000 7fd166187740 1 --2- 172.21.15.98:0/4060415492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be101320 0x55c7be103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.615+0000 7fd166187740 1 --2- 172.21.15.98:0/4060415492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6c50 0x55c7be100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.615+0000 7fd166187740 1 --2- 172.21.15.98:0/4060415492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6330 0x55c7be0f6710 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.615+0000 7fd166187740 1 -- 172.21.15.98:0/4060415492 >> 172.21.15.98:0/4060415492 conn(0x55c7be0f2650 msgr2=0x55c7be0f2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.615+0000 7fd166187740 1 -- 172.21.15.98:0/4060415492 shutdown_connections 2024-05-09T08:58:12.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.615+0000 7fd166187740 1 -- 172.21.15.98:0/4060415492 wait complete. 2024-05-09T08:58:12.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.615+0000 7fd166187740 1 -- start start 2024-05-09T08:58:12.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.615+0000 7fd166187740 1 Processor -- start 2024-05-09T08:58:12.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd166187740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6330 0x55c7be0fe1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd166187740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6c50 0x55c7be0fe700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd166187740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be101320 0x55c7be0fec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd166187740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c7be104110 con 0x55c7be0f6330 2024-05-09T08:58:12.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd166187740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c7be016c40 con 0x55c7be101320 2024-05-09T08:58:12.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd166187740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c7be104290 con 0x55c7be0f6c50 2024-05-09T08:58:12.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd166187740 10 client.0 _ll_register_callbacks cb 0x55c7bdd6ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:12.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd16113a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6330 0x55c7be0fe1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd16113a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6330 0x55c7be0fe1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57324/0 (socket says 172.21.15.98:57324) 2024-05-09T08:58:12.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd16113a640 1 -- 172.21.15.98:0/2219471132 learned_addr learned my addr 172.21.15.98:0/2219471132 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd160939640 1 --2- 172.21.15.98:0/2219471132 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6c50 0x55c7be0fe700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd16113a640 1 -- 172.21.15.98:0/2219471132 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6c50 msgr2=0x55c7be0fe700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd16113a640 1 --2- 172.21.15.98:0/2219471132 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6c50 0x55c7be0fe700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd16113a640 1 -- 172.21.15.98:0/2219471132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be101320 msgr2=0x55c7be0fec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd16113a640 1 --2- 172.21.15.98:0/2219471132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be101320 0x55c7be0fec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd16113a640 1 -- 172.21.15.98:0/2219471132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7bde397d0 con 0x55c7be0f6330 2024-05-09T08:58:12.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd16113a640 1 --2- 172.21.15.98:0/2219471132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6330 0x55c7be0fe1c0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd158031550 tx=0x7fd15802a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1138450ac5d3a30 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd144ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:12.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd144ff9640 1 -- 172.21.15.98:0/2219471132 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd158049050 con 0x55c7be0f6330 2024-05-09T08:58:12.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd144ff9640 1 -- 172.21.15.98:0/2219471132 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fd1580b8620 con 0x55c7be0f6330 2024-05-09T08:58:12.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd166187740 1 -- 172.21.15.98:0/2219471132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd154011d00 con 0x55c7be0f6330 2024-05-09T08:58:12.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd144ff9640 1 -- 172.21.15.98:0/2219471132 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd1580bb910 con 0x55c7be0f6330 2024-05-09T08:58:12.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd166187740 10 client.17652 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:12.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.616+0000 7fd166187740 1 -- 172.21.15.98:0/2219471132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c7be110250 con 0x55c7be0f6330 2024-05-09T08:58:12.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.617+0000 7fd144ff9640 1 -- 172.21.15.98:0/2219471132 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd154011d00 con 0x55c7be0f6330 2024-05-09T08:58:12.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.617+0000 7fd144ff9640 1 -- 172.21.15.98:0/2219471132 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd15803b460 con 0x55c7be0f6330 2024-05-09T08:58:12.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.617+0000 7fd166187740 10 client.17652 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:12.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.617+0000 7fd166187740 10 client.17652 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:58:12.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.617+0000 7fd166187740 20 client.17652 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:12.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.617+0000 7fd166187740 1 --2- 172.21.15.98:0/2219471132 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c7be111640 0x55c7be113a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.617+0000 7fd166187740 4 client.17652 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:12.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.617+0000 7fd166187740 1 -- 172.21.15.98:0/2219471132 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7be069370 con 0x55c7be111640 2024-05-09T08:58:12.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.617+0000 7fd160939640 1 --2- 172.21.15.98:0/2219471132 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c7be111640 0x55c7be113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.617+0000 7fd160939640 1 --2- 172.21.15.98:0/2219471132 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c7be111640 0x55c7be113a70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5739ae1fbac81552 server_cookie=d806d98d7351b97b in_seq=0 out_seq=0 2024-05-09T08:58:12.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.617+0000 7fd144ff9640 10 client.17652 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:58:12.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.618+0000 7fd144ff9640 1 -- 172.21.15.98:0/2219471132 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c7be069370 con 0x55c7be111640 2024-05-09T08:58:12.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.618+0000 7fd144ff9640 10 client.17652 handle_command_reply: tid=0 2024-05-09T08:58:12.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.618+0000 7fd166187740 1 client.17652 shutdown 2024-05-09T08:58:12.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.618+0000 7fd166187740 1 -- 172.21.15.98:0/2219471132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6330 msgr2=0x55c7be0fe1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.618+0000 7fd166187740 1 --2- 172.21.15.98:0/2219471132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6330 0x55c7be0fe1c0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd158031550 tx=0x7fd15802a420 comp rx=0 tx=0).stop 2024-05-09T08:58:12.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.618+0000 7fd166187740 1 -- 172.21.15.98:0/2219471132 shutdown_connections 2024-05-09T08:58:12.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.618+0000 7fd166187740 1 --2- 172.21.15.98:0/2219471132 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c7be111640 0x55c7be113a70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.618+0000 7fd166187740 1 --2- 172.21.15.98:0/2219471132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be101320 0x55c7be0fec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.618+0000 7fd166187740 1 --2- 172.21.15.98:0/2219471132 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6c50 0x55c7be0fe700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.618+0000 7fd144ff9640 0 client.17652 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:58:12.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.618+0000 7fd166187740 1 --2- 172.21.15.98:0/2219471132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6330 0x55c7be0fe1c0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.618+0000 7fd166187740 1 -- 172.21.15.98:0/2219471132 >> 172.21.15.98:0/2219471132 conn(0x55c7be0f2650 msgr2=0x55c7be0f8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.618+0000 7fd166187740 1 -- 172.21.15.98:0/2219471132 shutdown_connections 2024-05-09T08:58:12.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.618+0000 7fd166187740 1 -- 172.21.15.98:0/2219471132 wait complete. 2024-05-09T08:58:12.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.619+0000 7fd166187740 20 client.17652 trim_cache size 0 max 16384 2024-05-09T08:58:12.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd166187740 1 Processor -- start 2024-05-09T08:58:12.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd166187740 1 -- start start 2024-05-09T08:58:12.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd166187740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be0f6330 0x55c7be10d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd166187740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6c50 0x55c7be0fd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd166187740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be101320 0x55c7be0f8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd166187740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd15c1aef20 con 0x55c7be0f6c50 2024-05-09T08:58:12.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd166187740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd15c1b0a60 con 0x55c7be0f6330 2024-05-09T08:58:12.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd166187740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd15c1af5f0 con 0x55c7be101320 2024-05-09T08:58:12.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd160939640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6c50 0x55c7be0fd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd160939640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6c50 0x55c7be0fd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57326/0 (socket says 172.21.15.98:57326) 2024-05-09T08:58:12.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd160939640 1 -- 172.21.15.98:0/1139495967 learned_addr learned my addr 172.21.15.98:0/1139495967 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd16193b640 1 --2- 172.21.15.98:0/1139495967 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be101320 0x55c7be0f8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd160939640 1 -- 172.21.15.98:0/1139495967 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be101320 msgr2=0x55c7be0f8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd160939640 1 --2- 172.21.15.98:0/1139495967 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be101320 0x55c7be0f8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd160939640 1 -- 172.21.15.98:0/1139495967 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be0f6330 msgr2=0x55c7be10d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd160939640 1 --2- 172.21.15.98:0/1139495967 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be0f6330 0x55c7be10d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd160939640 1 -- 172.21.15.98:0/1139495967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7be0f71a0 con 0x55c7be0f6c50 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd16193b640 1 --2- 172.21.15.98:0/1139495967 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be101320 0x55c7be0f8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd160939640 1 --2- 172.21.15.98:0/1139495967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6c50 0x55c7be0fd360 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x55c7be0f37e0 tx=0x7fd14c005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=633e486f0f136b21 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.632+0000 7fd146ffd640 1 -- 172.21.15.98:0/1139495967 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0130d0 con 0x55c7be0f6c50 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.633+0000 7fd146ffd640 1 -- 172.21.15.98:0/1139495967 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd14c013270 con 0x55c7be0f6c50 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.633+0000 7fd146ffd640 1 -- 172.21.15.98:0/1139495967 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c011c30 con 0x55c7be0f6c50 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.633+0000 7fd166187740 1 -- 172.21.15.98:0/1139495967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6c50 msgr2=0x55c7be0fd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.633+0000 7fd166187740 1 --2- 172.21.15.98:0/1139495967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6c50 0x55c7be0fd360 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x55c7be0f37e0 tx=0x7fd14c005d20 comp rx=0 tx=0).stop 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.633+0000 7fd166187740 1 -- 172.21.15.98:0/1139495967 shutdown_connections 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.633+0000 7fd166187740 1 --2- 172.21.15.98:0/1139495967 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be101320 0x55c7be0f8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.633+0000 7fd166187740 1 --2- 172.21.15.98:0/1139495967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be0f6c50 0x55c7be0fd360 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.633+0000 7fd166187740 1 --2- 172.21.15.98:0/1139495967 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be0f6330 0x55c7be10d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.633+0000 7fd166187740 1 -- 172.21.15.98:0/1139495967 >> 172.21.15.98:0/1139495967 conn(0x55c7be0f2650 msgr2=0x55c7be0f7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.633+0000 7fd166187740 1 -- 172.21.15.98:0/1139495967 shutdown_connections 2024-05-09T08:58:12.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.633+0000 7fd166187740 1 -- 172.21.15.98:0/1139495967 wait complete. 2024-05-09T08:58:12.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.633+0000 7fd166187740 1 -- start start 2024-05-09T08:58:12.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.633+0000 7fd166187740 1 Processor -- start 2024-05-09T08:58:12.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd166187740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6330 0x55c7be0fb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd166187740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be0f6c50 0x55c7be0fb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd166187740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be101320 0x55c7be0fba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd166187740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd15c1b0390 con 0x55c7be101320 2024-05-09T08:58:12.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd166187740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd15c1afcc0 con 0x55c7be0f6c50 2024-05-09T08:58:12.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd166187740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd15c1acf20 con 0x55c7be0f6330 2024-05-09T08:58:12.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd166187740 10 client.0 _ll_register_callbacks cb 0x7fd158009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:12.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd16193b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be101320 0x55c7be0fba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd16193b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be101320 0x55c7be0fba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57332/0 (socket says 172.21.15.98:57332) 2024-05-09T08:58:12.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd16193b640 1 -- 172.21.15.98:0/3187999713 learned_addr learned my addr 172.21.15.98:0/3187999713 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd160939640 1 --2- 172.21.15.98:0/3187999713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be0f6c50 0x55c7be0fb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd16113a640 1 --2- 172.21.15.98:0/3187999713 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6330 0x55c7be0fb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd16193b640 1 -- 172.21.15.98:0/3187999713 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6330 msgr2=0x55c7be0fb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd16193b640 1 --2- 172.21.15.98:0/3187999713 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6330 0x55c7be0fb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd16193b640 1 -- 172.21.15.98:0/3187999713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be0f6c50 msgr2=0x55c7be0fb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd16193b640 1 --2- 172.21.15.98:0/3187999713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be0f6c50 0x55c7be0fb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd16193b640 1 -- 172.21.15.98:0/3187999713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7be176cc0 con 0x55c7be101320 2024-05-09T08:58:12.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd160939640 1 --2- 172.21.15.98:0/3187999713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be0f6c50 0x55c7be0fb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd16193b640 1 --2- 172.21.15.98:0/3187999713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be101320 0x55c7be0fba80 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fd154011d50 tx=0x7fd154011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6599e8cca09fa113 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.634+0000 7fd144ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd144ff9640 1 -- 172.21.15.98:0/3187999713 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd154016040 con 0x55c7be101320 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd144ff9640 1 -- 172.21.15.98:0/3187999713 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7fd15401b810 con 0x55c7be101320 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd166187740 1 -- 172.21.15.98:0/3187999713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd14c013270 con 0x55c7be101320 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd144ff9640 1 -- 172.21.15.98:0/3187999713 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd154038050 con 0x55c7be101320 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd166187740 10 client.17658 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd166187740 1 -- 172.21.15.98:0/3187999713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c7be177f30 con 0x55c7be101320 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd144ff9640 1 -- 172.21.15.98:0/3187999713 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd14c013270 con 0x55c7be101320 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd144ff9640 1 -- 172.21.15.98:0/3187999713 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd15401a070 con 0x55c7be101320 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd166187740 10 client.17658 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd166187740 10 client.17658 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd166187740 20 client.17658 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd166187740 1 --2- 172.21.15.98:0/3187999713 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c7be1783e0 0x55c7be17a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd166187740 4 client.17658 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd166187740 1 -- 172.21.15.98:0/3187999713 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c7be0636c0 con 0x55c7be1783e0 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.635+0000 7fd16113a640 1 --2- 172.21.15.98:0/3187999713 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c7be1783e0 0x55c7be17a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd144ff9640 10 client.17658 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd16113a640 1 --2- 172.21.15.98:0/3187999713 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c7be1783e0 0x55c7be17a800 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cc68af0ba7d8ddfc server_cookie=52f17bdd8c318234 in_seq=0 out_seq=0 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd144ff9640 1 -- 172.21.15.98:0/3187999713 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c7be0636c0 con 0x55c7be1783e0 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd144ff9640 10 client.17658 handle_command_reply: tid=0 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd166187740 1 client.17658 shutdown 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd166187740 1 -- 172.21.15.98:0/3187999713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be101320 msgr2=0x55c7be0fba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd166187740 1 --2- 172.21.15.98:0/3187999713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be101320 0x55c7be0fba80 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fd154011d50 tx=0x7fd154011d80 comp rx=0 tx=0).stop 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd166187740 1 -- 172.21.15.98:0/3187999713 shutdown_connections 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd166187740 1 --2- 172.21.15.98:0/3187999713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7be101320 0x55c7be0fba80 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd166187740 1 --2- 172.21.15.98:0/3187999713 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55c7be1783e0 0x55c7be17a800 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd166187740 1 --2- 172.21.15.98:0/3187999713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7be0f6c50 0x55c7be0fb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd166187740 1 --2- 172.21.15.98:0/3187999713 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7be0f6330 0x55c7be0fb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd144ff9640 0 client.17658 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:58:12.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd166187740 1 -- 172.21.15.98:0/3187999713 >> 172.21.15.98:0/3187999713 conn(0x55c7be0f2650 msgr2=0x55c7be0f8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd166187740 1 -- 172.21.15.98:0/3187999713 shutdown_connections 2024-05-09T08:58:12.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd166187740 1 -- 172.21.15.98:0/3187999713 wait complete. 2024-05-09T08:58:12.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.636+0000 7fd166187740 20 client.17658 trim_cache size 0 max 16384 2024-05-09T08:58:12.645 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:58:12.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.637+0000 7fd16339e640 1 -- 172.21.15.98:0/1637615951 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd1480a2d00 msgr2=0x7fd1480a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.637+0000 7fd16339e640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd1480a2d00 0x7fd1480a51c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd15401d000 tx=0x7fd15401e000 comp rx=0 tx=0).stop 2024-05-09T08:58:12.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.637+0000 7fd16339e640 1 -- 172.21.15.98:0/1637615951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c100700 msgr2=0x7fd15c198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.637+0000 7fd16339e640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c100700 0x7fd15c198130 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fd158009a50 tx=0x7fd158004270 comp rx=0 tx=0).stop 2024-05-09T08:58:12.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.637+0000 7fd16339e640 1 -- 172.21.15.98:0/1637615951 shutdown_connections 2024-05-09T08:58:12.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.637+0000 7fd16339e640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd1480a2d00 0x7fd1480a51c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.637+0000 7fd16339e640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd15c106710 0x7fd15c19c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.637+0000 7fd16339e640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd15c1010c0 0x7fd15c198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.637+0000 7fd16339e640 1 --2- 172.21.15.98:0/1637615951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c100700 0x7fd15c198130 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.637+0000 7fd16339e640 1 -- 172.21.15.98:0/1637615951 >> 172.21.15.98:0/1637615951 conn(0x7fd15c077830 msgr2=0x7fd15c0ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.637+0000 7fd16339e640 1 -- 172.21.15.98:0/1637615951 shutdown_connections 2024-05-09T08:58:12.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.637+0000 7fd16339e640 1 -- 172.21.15.98:0/1637615951 wait complete. 2024-05-09T08:58:12.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:58:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:12 smithi142 ceph-mon[34625]: pgmap v234: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-09T08:58:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:12 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2251245057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:58:12.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:12 smithi105 ceph-mon[34498]: pgmap v234: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-09T08:58:12.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:12 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2251245057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:58:12.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.751+0000 7f6407d63640 1 -- 172.21.15.98:0/2254387776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6400106760 msgr2=0x7f6400069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.751+0000 7f6407d63640 1 --2- 172.21.15.98:0/2254387776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6400106760 0x7f6400069eb0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f63fc009900 tx=0x7f63fc02a410 comp rx=0 tx=0).stop 2024-05-09T08:58:12.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.751+0000 7f6407d63640 1 -- 172.21.15.98:0/2254387776 shutdown_connections 2024-05-09T08:58:12.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.751+0000 7f6407d63640 1 --2- 172.21.15.98:0/2254387776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6400104260 0x7f64001046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.751+0000 7f6407d63640 1 --2- 172.21.15.98:0/2254387776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f640006a3f0 0x7f640006a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.751+0000 7f6407d63640 1 --2- 172.21.15.98:0/2254387776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6400106760 0x7f6400069eb0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.751+0000 7f6407d63640 1 -- 172.21.15.98:0/2254387776 >> 172.21.15.98:0/2254387776 conn(0x7f6400077790 msgr2=0x7f6400077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.752+0000 7f6407d63640 1 -- 172.21.15.98:0/2254387776 shutdown_connections 2024-05-09T08:58:12.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.752+0000 7f6407d63640 1 -- 172.21.15.98:0/2254387776 wait complete. 2024-05-09T08:58:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.752+0000 7f6407d63640 1 Processor -- start 2024-05-09T08:58:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.752+0000 7f6407d63640 1 -- start start 2024-05-09T08:58:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.752+0000 7f6407d63640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f640006a3f0 0x7f640019e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.752+0000 7f6407d63640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6400104260 0x7f640019ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.752+0000 7f6407d63640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6400106760 0x7f64001a3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.752+0000 7f6407d63640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6400118240 con 0x7f640006a3f0 2024-05-09T08:58:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.752+0000 7f6407d63640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f64001180c0 con 0x7f6400104260 2024-05-09T08:58:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.752+0000 7f6407d63640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f64001183c0 con 0x7f6400106760 2024-05-09T08:58:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.752+0000 7f6405aff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f640006a3f0 0x7f640019e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.752+0000 7f6405aff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f640006a3f0 0x7f640019e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57360/0 (socket says 172.21.15.98:57360) 2024-05-09T08:58:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.752+0000 7f6405aff640 1 -- 172.21.15.98:0/603381002 learned_addr learned my addr 172.21.15.98:0/603381002 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.753+0000 7f64052fe640 1 --2- 172.21.15.98:0/603381002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6400104260 0x7f640019ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.753+0000 7f6405aff640 1 -- 172.21.15.98:0/603381002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6400106760 msgr2=0x7f64001a3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.753+0000 7f6405aff640 1 --2- 172.21.15.98:0/603381002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6400106760 0x7f64001a3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.753+0000 7f6405aff640 1 -- 172.21.15.98:0/603381002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6400104260 msgr2=0x7f640019ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.753+0000 7f6405aff640 1 --2- 172.21.15.98:0/603381002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6400104260 0x7f640019ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.753+0000 7f6405aff640 1 -- 172.21.15.98:0/603381002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64001a3730 con 0x7f640006a3f0 2024-05-09T08:58:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.753+0000 7f6405aff640 1 --2- 172.21.15.98:0/603381002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f640006a3f0 0x7f640019e740 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f63fc006010 tx=0x7f63fc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.753+0000 7f63f6ffd640 1 -- 172.21.15.98:0/603381002 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03f030 con 0x7f640006a3f0 2024-05-09T08:58:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.753+0000 7f63f6ffd640 1 -- 172.21.15.98:0/603381002 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63fc003ce0 con 0x7f640006a3f0 2024-05-09T08:58:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.753+0000 7f6407d63640 1 -- 172.21.15.98:0/603381002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64001a39c0 con 0x7f640006a3f0 2024-05-09T08:58:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.753+0000 7f63f6ffd640 1 -- 172.21.15.98:0/603381002 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63fc035070 con 0x7f640006a3f0 2024-05-09T08:58:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.753+0000 7f6407d63640 1 -- 172.21.15.98:0/603381002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64001ab2b0 con 0x7f640006a3f0 2024-05-09T08:58:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.755+0000 7f63f6ffd640 1 -- 172.21.15.98:0/603381002 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f63fc042030 con 0x7f640006a3f0 2024-05-09T08:58:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.756+0000 7f640aa6a740 1 Processor -- start 2024-05-09T08:58:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.756+0000 7f640aa6a740 1 -- start start 2024-05-09T08:58:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.756+0000 7f63f6ffd640 1 --2- 172.21.15.98:0/603381002 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63ec0a2d00 0x7f63ec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.756+0000 7f640aa6a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568b0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.756+0000 7f640aa6a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568bade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.756+0000 7f640aa6a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568bd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.756+0000 7f640aa6a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fd56815c40 con 0x55fd568bb320 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.756+0000 7f640aa6a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fd567bd580 con 0x55fd568b0330 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.756+0000 7f640aa6a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fd56811c00 con 0x55fd568b0c50 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f63f6ffd640 1 -- 172.21.15.98:0/603381002 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7f63fc0efee0 con 0x7f640006a3f0 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f6406300640 1 --2- 172.21.15.98:0/603381002 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63ec0a2d00 0x7f63ec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f6405aff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f6405aff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568b0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:50150/0 (socket says 172.21.15.98:50150) 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f6405aff640 1 -- 172.21.15.98:0/217707062 learned_addr learned my addr 172.21.15.98:0/217707062 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f6406300640 1 --2- 172.21.15.98:0/217707062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568bd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f64052fe640 1 --2- 172.21.15.98:0/217707062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568bade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f6405aff640 1 -- 172.21.15.98:0/217707062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 msgr2=0x55fd568bade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f6405aff640 1 --2- 172.21.15.98:0/217707062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568bade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f6405aff640 1 -- 172.21.15.98:0/217707062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 msgr2=0x55fd568bd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f6405aff640 1 --2- 172.21.15.98:0/217707062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568bd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f6405aff640 1 -- 172.21.15.98:0/217707062 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd565293f0 con 0x55fd568b0330 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f6406300640 1 --2- 172.21.15.98:0/603381002 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63ec0a2d00 0x7f63ec0a51c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f63f8005c40 tx=0x7f63f8009c70 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f6405aff640 1 --2- 172.21.15.98:0/217707062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568b0710 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f63fc0c4730 tx=0x7f63fc0b8390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac26dbafa5a90d01 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f63ebfff640 1 -- 172.21.15.98:0/217707062 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63fc049040 con 0x55fd568b0330 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f63ebfff640 1 -- 172.21.15.98:0/217707062 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63fc03b580 con 0x55fd568b0330 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.757+0000 7f63ebfff640 1 -- 172.21.15.98:0/217707062 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0bc6f0 con 0x55fd568b0330 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.758+0000 7f640aa6a740 1 -- 172.21.15.98:0/217707062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 msgr2=0x55fd568b0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.758+0000 7f640aa6a740 1 --2- 172.21.15.98:0/217707062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568b0710 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f63fc0c4730 tx=0x7f63fc0b8390 comp rx=0 tx=0).stop 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.758+0000 7f640aa6a740 1 -- 172.21.15.98:0/217707062 shutdown_connections 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.758+0000 7f640aa6a740 1 --2- 172.21.15.98:0/217707062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568bd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.758+0000 7f640aa6a740 1 --2- 172.21.15.98:0/217707062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568bade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.758+0000 7f640aa6a740 1 --2- 172.21.15.98:0/217707062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568b0710 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.758+0000 7f640aa6a740 1 -- 172.21.15.98:0/217707062 >> 172.21.15.98:0/217707062 conn(0x55fd568ac650 msgr2=0x55fd568aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.758+0000 7f640aa6a740 1 -- 172.21.15.98:0/217707062 shutdown_connections 2024-05-09T08:58:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.758+0000 7f640aa6a740 1 -- 172.21.15.98:0/217707062 wait complete. 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.758+0000 7f640aa6a740 1 -- start start 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.758+0000 7f640aa6a740 1 Processor -- start 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f640aa6a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568b0330 0x55fd568b8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f640aa6a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568b87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f640aa6a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568bb320 0x55fd568b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f640aa6a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fd568bded0 con 0x55fd568b0330 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f640aa6a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fd567d0c40 con 0x55fd568bb320 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f640aa6a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fd568be050 con 0x55fd568b0c50 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f640aa6a740 10 client.0 _ll_register_callbacks cb 0x55fd56528be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f6405aff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568b0330 0x55fd568b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f6405aff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568b0330 0x55fd568b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57368/0 (socket says 172.21.15.98:57368) 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f6405aff640 1 -- 172.21.15.98:0/1925660702 learned_addr learned my addr 172.21.15.98:0/1925660702 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f64052fe640 1 --2- 172.21.15.98:0/1925660702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568b87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f6406300640 1 --2- 172.21.15.98:0/1925660702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568bb320 0x55fd568b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f6405aff640 1 -- 172.21.15.98:0/1925660702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 msgr2=0x55fd568b87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f6405aff640 1 --2- 172.21.15.98:0/1925660702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568b87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f6405aff640 1 -- 172.21.15.98:0/1925660702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568bb320 msgr2=0x55fd568b8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f6405aff640 1 --2- 172.21.15.98:0/1925660702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568bb320 0x55fd568b8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f6405aff640 1 -- 172.21.15.98:0/1925660702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd565f37d0 con 0x55fd568b0330 2024-05-09T08:58:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f6406300640 1 --2- 172.21.15.98:0/1925660702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568bb320 0x55fd568b8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f6405aff640 1 --2- 172.21.15.98:0/1925660702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568b0330 0x55fd568b8260 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f63fc03d040 tx=0x7f63fc0ba500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359375c9f1e9b53 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.759+0000 7f63e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f63e97fa640 1 -- 172.21.15.98:0/1925660702 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0c9060 con 0x55fd568b0330 2024-05-09T08:58:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f63e97fa640 1 -- 172.21.15.98:0/1925660702 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7f63fc0cecd0 con 0x55fd568b0330 2024-05-09T08:58:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f640aa6a740 1 -- 172.21.15.98:0/1925660702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fd56823370 con 0x55fd568b0330 2024-05-09T08:58:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f63e97fa640 1 -- 172.21.15.98:0/1925660702 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f63fc0cd070 con 0x55fd568b0330 2024-05-09T08:58:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f640aa6a740 10 client.17670 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f640aa6a740 1 -- 172.21.15.98:0/1925660702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55fd568bded0 con 0x55fd568b0330 2024-05-09T08:58:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f63e97fa640 1 -- 172.21.15.98:0/1925660702 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63fc0d7030 con 0x55fd568b0330 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f63e97fa640 1 -- 172.21.15.98:0/1925660702 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f63fc0ed3e0 con 0x55fd568b0330 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f640aa6a740 10 client.17670 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f640aa6a740 10 client.17670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f640aa6a740 20 client.17670 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f640aa6a740 1 --2- 172.21.15.98:0/1925660702 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fd568cb240 0x55fd568cd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f640aa6a740 4 client.17670 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f640aa6a740 1 -- 172.21.15.98:0/1925660702 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd5681d6c0 con 0x55fd568cb240 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.760+0000 7f64052fe640 1 --2- 172.21.15.98:0/1925660702 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fd568cb240 0x55fd568cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.761+0000 7f63e97fa640 10 client.17670 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.761+0000 7f64052fe640 1 --2- 172.21.15.98:0/1925660702 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fd568cb240 0x55fd568cd670 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83af6d9ad5ce10c7 server_cookie=b9bd584094e74fdb in_seq=0 out_seq=0 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.761+0000 7f63e97fa640 1 -- 172.21.15.98:0/1925660702 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd5681d6c0 con 0x55fd568cb240 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.761+0000 7f63e97fa640 10 client.17670 handle_command_reply: tid=0 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.761+0000 7f640aa6a740 1 client.17670 shutdown 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.762+0000 7f640aa6a740 1 -- 172.21.15.98:0/1925660702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568b0330 msgr2=0x55fd568b8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.762+0000 7f640aa6a740 1 --2- 172.21.15.98:0/1925660702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568b0330 0x55fd568b8260 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f63fc03d040 tx=0x7f63fc0ba500 comp rx=0 tx=0).stop 2024-05-09T08:58:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.762+0000 7f640aa6a740 1 -- 172.21.15.98:0/1925660702 shutdown_connections 2024-05-09T08:58:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.762+0000 7f640aa6a740 1 --2- 172.21.15.98:0/1925660702 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fd568cb240 0x55fd568cd670 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.762+0000 7f640aa6a740 1 --2- 172.21.15.98:0/1925660702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568bb320 0x55fd568b8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.762+0000 7f640aa6a740 1 --2- 172.21.15.98:0/1925660702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568b87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.762+0000 7f640aa6a740 1 --2- 172.21.15.98:0/1925660702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568b0330 0x55fd568b8260 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.762+0000 7f63e97fa640 0 client.17670 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.762+0000 7f640aa6a740 1 -- 172.21.15.98:0/1925660702 >> 172.21.15.98:0/1925660702 conn(0x55fd568ac650 msgr2=0x55fd568b2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.762+0000 7f640aa6a740 1 -- 172.21.15.98:0/1925660702 shutdown_connections 2024-05-09T08:58:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.762+0000 7f640aa6a740 1 -- 172.21.15.98:0/1925660702 wait complete. 2024-05-09T08:58:12.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.762+0000 7f640aa6a740 20 client.17670 trim_cache size 0 max 16384 2024-05-09T08:58:12.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.775+0000 7f640aa6a740 1 Processor -- start 2024-05-09T08:58:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.775+0000 7f640aa6a740 1 -- start start 2024-05-09T08:58:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.775+0000 7f640aa6a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568ba990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.775+0000 7f640aa6a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.775+0000 7f640aa6a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568d2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.775+0000 7f640aa6a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f64001b5640 con 0x55fd568bb320 2024-05-09T08:58:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.775+0000 7f640aa6a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f64001b7180 con 0x55fd568b0330 2024-05-09T08:58:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.775+0000 7f640aa6a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f64001b5d10 con 0x55fd568b0c50 2024-05-09T08:58:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f6406300640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568d2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f6406300640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568d2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57380/0 (socket says 172.21.15.98:57380) 2024-05-09T08:58:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f6406300640 1 -- 172.21.15.98:0/1373083723 learned_addr learned my addr 172.21.15.98:0/1373083723 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f64052fe640 1 --2- 172.21.15.98:0/1373083723 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f6405aff640 1 --2- 172.21.15.98:0/1373083723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568ba990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f6406300640 1 -- 172.21.15.98:0/1373083723 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 msgr2=0x55fd568c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f6406300640 1 --2- 172.21.15.98:0/1373083723 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f6406300640 1 -- 172.21.15.98:0/1373083723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 msgr2=0x55fd568ba990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f6406300640 1 --2- 172.21.15.98:0/1373083723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568ba990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f6406300640 1 -- 172.21.15.98:0/1373083723 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd568b77c0 con 0x55fd568bb320 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f6405aff640 1 --2- 172.21.15.98:0/1373083723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568ba990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f6406300640 1 --2- 172.21.15.98:0/1373083723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568d2f10 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f63f8012330 tx=0x7f63f801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59da1b8195209ab2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f63eb7fe640 1 -- 172.21.15.98:0/1373083723 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8012360 con 0x55fd568bb320 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f63eb7fe640 1 -- 172.21.15.98:0/1373083723 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f8004510 con 0x55fd568bb320 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.776+0000 7f63eb7fe640 1 -- 172.21.15.98:0/1373083723 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8011a10 con 0x55fd568bb320 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.777+0000 7f640aa6a740 1 -- 172.21.15.98:0/1373083723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 msgr2=0x55fd568d2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.777+0000 7f640aa6a740 1 --2- 172.21.15.98:0/1373083723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568d2f10 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f63f8012330 tx=0x7f63f801c000 comp rx=0 tx=0).stop 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.777+0000 7f640aa6a740 1 -- 172.21.15.98:0/1373083723 shutdown_connections 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.777+0000 7f640aa6a740 1 --2- 172.21.15.98:0/1373083723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568d2f10 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.777+0000 7f640aa6a740 1 --2- 172.21.15.98:0/1373083723 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.777+0000 7f640aa6a740 1 --2- 172.21.15.98:0/1373083723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568ba990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.777+0000 7f640aa6a740 1 -- 172.21.15.98:0/1373083723 >> 172.21.15.98:0/1373083723 conn(0x55fd568ac650 msgr2=0x55fd568ab7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.777+0000 7f640aa6a740 1 -- 172.21.15.98:0/1373083723 shutdown_connections 2024-05-09T08:58:12.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.777+0000 7f640aa6a740 1 -- 172.21.15.98:0/1373083723 wait complete. 2024-05-09T08:58:12.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.777+0000 7f640aa6a740 1 -- start start 2024-05-09T08:58:12.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.777+0000 7f640aa6a740 1 Processor -- start 2024-05-09T08:58:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f640aa6a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f640aa6a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568b56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f640aa6a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f640aa6a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f64001b6ab0 con 0x55fd568bb320 2024-05-09T08:58:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f640aa6a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f64001b63e0 con 0x55fd568b0330 2024-05-09T08:58:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f640aa6a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f64001b3640 con 0x55fd568b0c50 2024-05-09T08:58:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f640aa6a740 10 client.0 _ll_register_callbacks cb 0x7f63f8012530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f6406300640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f6406300640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568b5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57392/0 (socket says 172.21.15.98:57392) 2024-05-09T08:58:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f6406300640 1 -- 172.21.15.98:0/2448440909 learned_addr learned my addr 172.21.15.98:0/2448440909 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f64052fe640 1 --2- 172.21.15.98:0/2448440909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568b56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f6405aff640 1 --2- 172.21.15.98:0/2448440909 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f6406300640 1 -- 172.21.15.98:0/2448440909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 msgr2=0x55fd568b56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f6406300640 1 --2- 172.21.15.98:0/2448440909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568b56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f6406300640 1 -- 172.21.15.98:0/2448440909 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 msgr2=0x55fd568b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f6406300640 1 --2- 172.21.15.98:0/2448440909 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f6406300640 1 -- 172.21.15.98:0/2448440909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd56930980 con 0x55fd568bb320 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f6405aff640 1 --2- 172.21.15.98:0/2448440909 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f6406300640 1 --2- 172.21.15.98:0/2448440909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568b5c10 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009610 tx=0x7f63f80048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c894ff87b9d791e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.778+0000 7f63e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f63e97fa640 1 -- 172.21.15.98:0/2448440909 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8038060 con 0x55fd568bb320 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f640aa6a740 1 -- 172.21.15.98:0/2448440909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63f8004510 con 0x55fd568bb320 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f63e97fa640 1 -- 172.21.15.98:0/2448440909 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9650+0+0 (secure 0 0 0) 0x7f63f8004240 con 0x55fd568bb320 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f63e97fa640 1 -- 172.21.15.98:0/2448440909 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f63f803f050 con 0x55fd568bb320 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f640aa6a740 10 client.17676 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f640aa6a740 1 -- 172.21.15.98:0/2448440909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55fd56931bb0 con 0x55fd568bb320 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f63e97fa640 1 -- 172.21.15.98:0/2448440909 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f8004510 con 0x55fd568bb320 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f63e97fa640 1 -- 172.21.15.98:0/2448440909 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f63f801f440 con 0x55fd568bb320 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f640aa6a740 10 client.17676 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f640aa6a740 10 client.17676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f640aa6a740 20 client.17676 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:12.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f640aa6a740 1 --2- 172.21.15.98:0/2448440909 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fd56932270 0x55fd56934660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f640aa6a740 4 client.17676 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f640aa6a740 1 -- 172.21.15.98:0/2448440909 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd5679a820 con 0x55fd56932270 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.779+0000 7f6405aff640 1 --2- 172.21.15.98:0/2448440909 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fd56932270 0x55fd56934660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f63e97fa640 10 client.17676 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f6405aff640 1 --2- 172.21.15.98:0/2448440909 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fd56932270 0x55fd56934660 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52f5a046ac05c392 server_cookie=fe1de1c2bf2de22a in_seq=0 out_seq=0 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f63e97fa640 1 -- 172.21.15.98:0/2448440909 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd5679a820 con 0x55fd56932270 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f63e97fa640 10 client.17676 handle_command_reply: tid=0 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f640aa6a740 1 client.17676 shutdown 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f640aa6a740 1 -- 172.21.15.98:0/2448440909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 msgr2=0x55fd568b5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f640aa6a740 1 --2- 172.21.15.98:0/2448440909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568b5c10 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009610 tx=0x7f63f80048f0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f640aa6a740 1 -- 172.21.15.98:0/2448440909 shutdown_connections 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f640aa6a740 1 --2- 172.21.15.98:0/2448440909 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fd56932270 0x55fd56934660 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f640aa6a740 1 --2- 172.21.15.98:0/2448440909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd568bb320 0x55fd568b5c10 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f640aa6a740 1 --2- 172.21.15.98:0/2448440909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd568b0c50 0x55fd568b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f640aa6a740 1 --2- 172.21.15.98:0/2448440909 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd568b0330 0x55fd568b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f63e97fa640 0 client.17676 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f640aa6a740 1 -- 172.21.15.98:0/2448440909 >> 172.21.15.98:0/2448440909 conn(0x55fd568ac650 msgr2=0x55fd568b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f640aa6a740 1 -- 172.21.15.98:0/2448440909 shutdown_connections 2024-05-09T08:58:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f640aa6a740 1 -- 172.21.15.98:0/2448440909 wait complete. 2024-05-09T08:58:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.780+0000 7f640aa6a740 20 client.17676 trim_cache size 0 max 16384 2024-05-09T08:58:12.786 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:58:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.781+0000 7f6407d63640 1 -- 172.21.15.98:0/603381002 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63ec0a2d00 msgr2=0x7f63ec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.781+0000 7f6407d63640 1 --2- 172.21.15.98:0/603381002 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63ec0a2d00 0x7f63ec0a51c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f63f8005c40 tx=0x7f63f8009c70 comp rx=0 tx=0).stop 2024-05-09T08:58:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.781+0000 7f6407d63640 1 -- 172.21.15.98:0/603381002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f640006a3f0 msgr2=0x7f640019e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.781+0000 7f6407d63640 1 --2- 172.21.15.98:0/603381002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f640006a3f0 0x7f640019e740 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f63fc006010 tx=0x7f63fc0042f0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.781+0000 7f6407d63640 1 -- 172.21.15.98:0/603381002 shutdown_connections 2024-05-09T08:58:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.781+0000 7f6407d63640 1 --2- 172.21.15.98:0/603381002 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63ec0a2d00 0x7f63ec0a51c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.781+0000 7f6407d63640 1 --2- 172.21.15.98:0/603381002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6400106760 0x7f64001a3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.781+0000 7f6407d63640 1 --2- 172.21.15.98:0/603381002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6400104260 0x7f640019ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.781+0000 7f6407d63640 1 --2- 172.21.15.98:0/603381002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f640006a3f0 0x7f640019e740 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:12.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.781+0000 7f6407d63640 1 -- 172.21.15.98:0/603381002 >> 172.21.15.98:0/603381002 conn(0x7f6400077790 msgr2=0x7f6400103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:12.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.781+0000 7f6407d63640 1 -- 172.21.15.98:0/603381002 shutdown_connections 2024-05-09T08:58:12.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:12.781+0000 7f6407d63640 1 -- 172.21.15.98:0/603381002 wait complete. 2024-05-09T08:58:13.791 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:58:13.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:58:13.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:58:13.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.909+0000 7fc96e563640 1 -- 172.21.15.98:0/2043315493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9681021c0 msgr2=0x7fc96810e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:13.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.909+0000 7fc96e563640 1 --2- 172.21.15.98:0/2043315493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9681021c0 0x7fc96810e6d0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fc960009cd0 tx=0x7fc96002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.909+0000 7fc96e563640 1 -- 172.21.15.98:0/2043315493 shutdown_connections 2024-05-09T08:58:13.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.909+0000 7fc96e563640 1 --2- 172.21.15.98:0/2043315493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9681021c0 0x7fc96810e6d0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.909+0000 7fc96e563640 1 --2- 172.21.15.98:0/2043315493 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc968101800 0x7fc968101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.909+0000 7fc96e563640 1 --2- 172.21.15.98:0/2043315493 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc968107810 0x7fc968107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.909+0000 7fc96e563640 1 -- 172.21.15.98:0/2043315493 >> 172.21.15.98:0/2043315493 conn(0x7fc968078c80 msgr2=0x7fc968079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:13.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.909+0000 7fc96e563640 1 -- 172.21.15.98:0/2043315493 shutdown_connections 2024-05-09T08:58:13.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.909+0000 7fc96e563640 1 -- 172.21.15.98:0/2043315493 wait complete. 2024-05-09T08:58:13.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.910+0000 7fc96e563640 1 Processor -- start 2024-05-09T08:58:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.910+0000 7fc96e563640 1 -- start start 2024-05-09T08:58:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.910+0000 7fc96e563640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc968101800 0x7fc968198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.910+0000 7fc96e563640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9681021c0 0x7fc9681986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.910+0000 7fc96e563640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc968107810 0x7fc96819ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.910+0000 7fc96e563640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc968111c20 con 0x7fc968101800 2024-05-09T08:58:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.910+0000 7fc96e563640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc968111aa0 con 0x7fc9681021c0 2024-05-09T08:58:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.910+0000 7fc96e563640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc968111da0 con 0x7fc968107810 2024-05-09T08:58:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.910+0000 7fc967fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc968101800 0x7fc968198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.910+0000 7fc967fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc968101800 0x7fc968198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57408/0 (socket says 172.21.15.98:57408) 2024-05-09T08:58:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.910+0000 7fc967fff640 1 -- 172.21.15.98:0/4081799973 learned_addr learned my addr 172.21.15.98:0/4081799973 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.911+0000 7fc9677fe640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9681021c0 0x7fc9681986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.911+0000 7fc96cb00640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc968107810 0x7fc96819ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.911+0000 7fc967fff640 1 -- 172.21.15.98:0/4081799973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc968107810 msgr2=0x7fc96819ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.911+0000 7fc967fff640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc968107810 0x7fc96819ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.911+0000 7fc967fff640 1 -- 172.21.15.98:0/4081799973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9681021c0 msgr2=0x7fc9681986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.911+0000 7fc967fff640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9681021c0 0x7fc9681986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.911+0000 7fc967fff640 1 -- 172.21.15.98:0/4081799973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96819d150 con 0x7fc968101800 2024-05-09T08:58:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.911+0000 7fc96cb00640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc968107810 0x7fc96819ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.911+0000 7fc967fff640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc968101800 0x7fc968198160 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fc95800b4a0 tx=0x7fc95800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.911+0000 7fc9657fa640 1 -- 172.21.15.98:0/4081799973 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc958014070 con 0x7fc968101800 2024-05-09T08:58:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.911+0000 7fc9657fa640 1 -- 172.21.15.98:0/4081799973 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc958010af0 con 0x7fc968101800 2024-05-09T08:58:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.911+0000 7fc96e563640 1 -- 172.21.15.98:0/4081799973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc96819d440 con 0x7fc968101800 2024-05-09T08:58:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.911+0000 7fc9657fa640 1 -- 172.21.15.98:0/4081799973 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc958017d10 con 0x7fc968101800 2024-05-09T08:58:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.911+0000 7fc96e563640 1 -- 172.21.15.98:0/4081799973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9681a4cd0 con 0x7fc968101800 2024-05-09T08:58:13.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.913+0000 7fc9657fa640 1 -- 172.21.15.98:0/4081799973 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc958017070 con 0x7fc968101800 2024-05-09T08:58:13.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc97127a740 1 Processor -- start 2024-05-09T08:58:13.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc97127a740 1 -- start start 2024-05-09T08:58:13.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc9657fa640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc9500a2d00 0x7fc9500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc97127a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d573f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc97127a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 0x5588f1d57e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc97127a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d64b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc97127a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5588f1cbc990 con 0x5588f1d62740 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc97127a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5588f1c645b0 con 0x5588f1d590b0 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc97127a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5588f1cb8950 con 0x5588f1d579c0 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc96cb00640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc9500a2d00 0x7fc9500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc9657fa640 1 -- 172.21.15.98:0/4081799973 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fc9580cebd0 con 0x7fc968101800 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc967fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc96cb00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d64b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc967fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d573f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:50188/0 (socket says 172.21.15.98:50188) 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc96cb00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d64b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57420/0 (socket says 172.21.15.98:57420) 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc967fff640 1 -- 172.21.15.98:0/992938176 learned_addr learned my addr 172.21.15.98:0/992938176 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.914+0000 7fc9677fe640 1 --2- 172.21.15.98:0/992938176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 0x5588f1d57e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.915+0000 7fc967fff640 1 -- 172.21.15.98:0/992938176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 msgr2=0x5588f1d57e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.915+0000 7fc967fff640 1 --2- 172.21.15.98:0/992938176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 0x5588f1d57e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.915+0000 7fc967fff640 1 -- 172.21.15.98:0/992938176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 msgr2=0x5588f1d64b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.915+0000 7fc967fff640 1 --2- 172.21.15.98:0/992938176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d64b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.915+0000 7fc967fff640 1 -- 172.21.15.98:0/992938176 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588f1a987e0 con 0x5588f1d590b0 2024-05-09T08:58:13.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.915+0000 7fc9677fe640 1 --2- 172.21.15.98:0/992938176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 0x5588f1d57e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:13.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.915+0000 7fc96cb00640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc9500a2d00 0x7fc9500a51c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc960009a60 tx=0x7fc9600099f0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:13.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.915+0000 7fc967fff640 1 --2- 172.21.15.98:0/992938176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d573f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc958097760 tx=0x7fc95809aa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bebfbe4b41d0b6e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:13.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.915+0000 7fc94e7fc640 1 -- 172.21.15.98:0/992938176 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95801e020 con 0x5588f1d590b0 2024-05-09T08:58:13.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.915+0000 7fc94e7fc640 1 -- 172.21.15.98:0/992938176 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc95809b490 con 0x5588f1d590b0 2024-05-09T08:58:13.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.915+0000 7fc94e7fc640 1 -- 172.21.15.98:0/992938176 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9580a3a00 con 0x5588f1d590b0 2024-05-09T08:58:13.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.916+0000 7fc97127a740 1 -- 172.21.15.98:0/992938176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 msgr2=0x5588f1d573f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:13.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.916+0000 7fc97127a740 1 --2- 172.21.15.98:0/992938176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d573f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc958097760 tx=0x7fc95809aa00 comp rx=0 tx=0).stop 2024-05-09T08:58:13.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.916+0000 7fc97127a740 1 -- 172.21.15.98:0/992938176 shutdown_connections 2024-05-09T08:58:13.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.916+0000 7fc97127a740 1 --2- 172.21.15.98:0/992938176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d64b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.916+0000 7fc97127a740 1 --2- 172.21.15.98:0/992938176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 0x5588f1d57e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.916+0000 7fc97127a740 1 --2- 172.21.15.98:0/992938176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d573f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.916+0000 7fc97127a740 1 -- 172.21.15.98:0/992938176 >> 172.21.15.98:0/992938176 conn(0x5588f1d200a0 msgr2=0x5588f1d53f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:13.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.916+0000 7fc97127a740 1 -- 172.21.15.98:0/992938176 shutdown_connections 2024-05-09T08:58:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.916+0000 7fc97127a740 1 -- 172.21.15.98:0/992938176 wait complete. 2024-05-09T08:58:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.916+0000 7fc97127a740 1 -- start start 2024-05-09T08:58:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.916+0000 7fc97127a740 1 Processor -- start 2024-05-09T08:58:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.916+0000 7fc97127a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 0x5588f1d5f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.916+0000 7fc97127a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d5fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.916+0000 7fc97127a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc97127a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5588f1d655d0 con 0x5588f1d62740 2024-05-09T08:58:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc97127a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5588f1c77b40 con 0x5588f1d590b0 2024-05-09T08:58:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc97127a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5588f1d65750 con 0x5588f1d579c0 2024-05-09T08:58:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc97127a740 10 client.0 _ll_register_callbacks cb 0x5588f1c6c0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc96cb00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc96cb00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d600c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57426/0 (socket says 172.21.15.98:57426) 2024-05-09T08:58:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc96cb00640 1 -- 172.21.15.98:0/3999895916 learned_addr learned my addr 172.21.15.98:0/3999895916 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc9677fe640 1 --2- 172.21.15.98:0/3999895916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d5fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc967fff640 1 --2- 172.21.15.98:0/3999895916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 0x5588f1d5f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc96cb00640 1 -- 172.21.15.98:0/3999895916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 msgr2=0x5588f1d5f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:13.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc96cb00640 1 --2- 172.21.15.98:0/3999895916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 0x5588f1d5f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc96cb00640 1 -- 172.21.15.98:0/3999895916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 msgr2=0x5588f1d5fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:13.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc96cb00640 1 --2- 172.21.15.98:0/3999895916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d5fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc96cb00640 1 -- 172.21.15.98:0/3999895916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588f1af8f70 con 0x5588f1d62740 2024-05-09T08:58:13.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc96cb00640 1 --2- 172.21.15.98:0/3999895916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d600c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc960035000 tx=0x7fc960046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121269142517eff8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:13.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc9477fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:13.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc9477fe640 1 -- 172.21.15.98:0/3999895916 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96004f030 con 0x5588f1d62740 2024-05-09T08:58:13.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc9477fe640 1 -- 172.21.15.98:0/3999895916 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fc9600046c0 con 0x5588f1d62740 2024-05-09T08:58:13.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc97127a740 1 -- 172.21.15.98:0/3999895916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc95809b490 con 0x5588f1d62740 2024-05-09T08:58:13.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.917+0000 7fc9477fe640 1 -- 172.21.15.98:0/3999895916 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc9600305a0 con 0x5588f1d62740 2024-05-09T08:58:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.918+0000 7fc97127a740 10 client.17688 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.918+0000 7fc97127a740 1 -- 172.21.15.98:0/3999895916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5588f1990170 con 0x5588f1d62740 2024-05-09T08:58:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.918+0000 7fc9477fe640 1 -- 172.21.15.98:0/3999895916 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc95809b490 con 0x5588f1d62740 2024-05-09T08:58:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.918+0000 7fc9477fe640 1 -- 172.21.15.98:0/3999895916 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fc960030070 con 0x5588f1d62740 2024-05-09T08:58:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.918+0000 7fc97127a740 10 client.17688 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.918+0000 7fc97127a740 10 client.17688 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.918+0000 7fc97127a740 20 client.17688 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.918+0000 7fc97127a740 1 --2- 172.21.15.98:0/3999895916 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5588f1d72810 0x5588f1d74c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.918+0000 7fc97127a740 4 client.17688 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.918+0000 7fc97127a740 1 -- 172.21.15.98:0/3999895916 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588f1cca0c0 con 0x5588f1d72810 2024-05-09T08:58:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.918+0000 7fc967fff640 1 --2- 172.21.15.98:0/3999895916 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5588f1d72810 0x5588f1d74c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.918+0000 7fc9477fe640 10 client.17688 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.918+0000 7fc967fff640 1 --2- 172.21.15.98:0/3999895916 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5588f1d72810 0x5588f1d74c20 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61fb798101dfcf0a server_cookie=915da82772213292 in_seq=0 out_seq=0 2024-05-09T08:58:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.919+0000 7fc9477fe640 1 -- 172.21.15.98:0/3999895916 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588f1cca0c0 con 0x5588f1d72810 2024-05-09T08:58:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.919+0000 7fc9477fe640 10 client.17688 handle_command_reply: tid=0 2024-05-09T08:58:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.919+0000 7fc97127a740 1 client.17688 shutdown 2024-05-09T08:58:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.919+0000 7fc97127a740 1 -- 172.21.15.98:0/3999895916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 msgr2=0x5588f1d600c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.919+0000 7fc97127a740 1 --2- 172.21.15.98:0/3999895916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d600c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc960035000 tx=0x7fc960046000 comp rx=0 tx=0).stop 2024-05-09T08:58:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.919+0000 7fc97127a740 1 -- 172.21.15.98:0/3999895916 shutdown_connections 2024-05-09T08:58:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.919+0000 7fc97127a740 1 --2- 172.21.15.98:0/3999895916 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5588f1d72810 0x5588f1d74c20 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.919+0000 7fc97127a740 1 --2- 172.21.15.98:0/3999895916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d600c0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.919+0000 7fc97127a740 1 --2- 172.21.15.98:0/3999895916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d5fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.919+0000 7fc9477fe640 0 client.17688 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.919+0000 7fc97127a740 1 --2- 172.21.15.98:0/3999895916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 0x5588f1d5f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.920+0000 7fc97127a740 1 -- 172.21.15.98:0/3999895916 >> 172.21.15.98:0/3999895916 conn(0x5588f1d200a0 msgr2=0x5588f1d59d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.920+0000 7fc97127a740 1 -- 172.21.15.98:0/3999895916 shutdown_connections 2024-05-09T08:58:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.920+0000 7fc97127a740 1 -- 172.21.15.98:0/3999895916 wait complete. 2024-05-09T08:58:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.920+0000 7fc97127a740 20 client.17688 trim_cache size 0 max 16384 2024-05-09T08:58:13.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.937+0000 7fc97127a740 1 Processor -- start 2024-05-09T08:58:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.937+0000 7fc97127a740 1 -- start start 2024-05-09T08:58:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.937+0000 7fc97127a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 0x5588f1d6e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.937+0000 7fc97127a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d52f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.937+0000 7fc97127a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d7a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.937+0000 7fc97127a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9681aef60 con 0x5588f1d62740 2024-05-09T08:58:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.937+0000 7fc97127a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc9681b0aa0 con 0x5588f1d590b0 2024-05-09T08:58:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.937+0000 7fc97127a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc9681af630 con 0x5588f1d579c0 2024-05-09T08:58:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc96cb00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d7a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc96cb00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d7a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57442/0 (socket says 172.21.15.98:57442) 2024-05-09T08:58:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc96cb00640 1 -- 172.21.15.98:0/1669245599 learned_addr learned my addr 172.21.15.98:0/1669245599 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc9677fe640 1 --2- 172.21.15.98:0/1669245599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d52f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc967fff640 1 --2- 172.21.15.98:0/1669245599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 0x5588f1d6e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc96cb00640 1 -- 172.21.15.98:0/1669245599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 msgr2=0x5588f1d6e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc96cb00640 1 --2- 172.21.15.98:0/1669245599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 0x5588f1d6e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc96cb00640 1 -- 172.21.15.98:0/1669245599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 msgr2=0x5588f1d52f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc96cb00640 1 --2- 172.21.15.98:0/1669245599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d52f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc96cb00640 1 -- 172.21.15.98:0/1669245599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588f1d5b240 con 0x5588f1d62740 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc9677fe640 1 --2- 172.21.15.98:0/1669245599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d52f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc96cb00640 1 --2- 172.21.15.98:0/1669245599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d7a4f0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc96002aff0 tx=0x7fc960009c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c44d8bbaecee2d41 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc94dffb640 1 -- 172.21.15.98:0/1669245599 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc960065070 con 0x5588f1d62740 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc94dffb640 1 -- 172.21.15.98:0/1669245599 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc960036040 con 0x5588f1d62740 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.938+0000 7fc94dffb640 1 -- 172.21.15.98:0/1669245599 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96004f030 con 0x5588f1d62740 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.939+0000 7fc97127a740 1 -- 172.21.15.98:0/1669245599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 msgr2=0x5588f1d7a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.939+0000 7fc97127a740 1 --2- 172.21.15.98:0/1669245599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d7a4f0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc96002aff0 tx=0x7fc960009c60 comp rx=0 tx=0).stop 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.939+0000 7fc97127a740 1 -- 172.21.15.98:0/1669245599 shutdown_connections 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.939+0000 7fc97127a740 1 --2- 172.21.15.98:0/1669245599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d62740 0x5588f1d7a4f0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.939+0000 7fc97127a740 1 --2- 172.21.15.98:0/1669245599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d590b0 0x5588f1d52f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.939+0000 7fc97127a740 1 --2- 172.21.15.98:0/1669245599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d579c0 0x5588f1d6e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.939+0000 7fc97127a740 1 -- 172.21.15.98:0/1669245599 >> 172.21.15.98:0/1669245599 conn(0x5588f1d200a0 msgr2=0x5588f1d527f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.939+0000 7fc97127a740 1 -- 172.21.15.98:0/1669245599 shutdown_connections 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.939+0000 7fc97127a740 1 -- 172.21.15.98:0/1669245599 wait complete. 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.939+0000 7fc97127a740 1 -- start start 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.939+0000 7fc97127a740 1 Processor -- start 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc97127a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d579c0 0x5588f1d6a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc97127a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d590b0 0x5588f1d6af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc97127a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d62740 0x5588f1d679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc97127a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9681b03d0 con 0x5588f1d579c0 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc97127a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc9681afd00 con 0x5588f1d62740 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc97127a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc9681acf60 con 0x5588f1d590b0 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc97127a740 10 client.0 _ll_register_callbacks cb 0x7fc96002af20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc967fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d579c0 0x5588f1d6a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc967fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d579c0 0x5588f1d6a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57458/0 (socket says 172.21.15.98:57458) 2024-05-09T08:58:13.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc967fff640 1 -- 172.21.15.98:0/2360048050 learned_addr learned my addr 172.21.15.98:0/2360048050 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:13.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc96cb00640 1 --2- 172.21.15.98:0/2360048050 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d62740 0x5588f1d679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc967fff640 1 -- 172.21.15.98:0/2360048050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d590b0 msgr2=0x5588f1d6af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:13.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc967fff640 1 --2- 172.21.15.98:0/2360048050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d590b0 0x5588f1d6af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc967fff640 1 -- 172.21.15.98:0/2360048050 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d62740 msgr2=0x5588f1d679b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:13.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc967fff640 1 --2- 172.21.15.98:0/2360048050 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d62740 0x5588f1d679b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc967fff640 1 -- 172.21.15.98:0/2360048050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588f1d67fa0 con 0x5588f1d579c0 2024-05-09T08:58:13.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc96cb00640 1 --2- 172.21.15.98:0/2360048050 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d62740 0x5588f1d679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:13.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc967fff640 1 --2- 172.21.15.98:0/2360048050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d579c0 0x5588f1d6a9e0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fc95809aa90 tx=0x7fc9580047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c27a2b8dbb9d5760 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:13.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc9477fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:13.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc9477fe640 1 -- 172.21.15.98:0/2360048050 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95801e040 con 0x5588f1d579c0 2024-05-09T08:58:13.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc9477fe640 1 -- 172.21.15.98:0/2360048050 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fc958010ca0 con 0x5588f1d579c0 2024-05-09T08:58:13.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc97127a740 1 -- 172.21.15.98:0/2360048050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc960036040 con 0x5588f1d579c0 2024-05-09T08:58:13.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc9477fe640 1 -- 172.21.15.98:0/2360048050 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc958098eb0 con 0x5588f1d579c0 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc97127a740 10 client.17694 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.940+0000 7fc97127a740 1 -- 172.21.15.98:0/2360048050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5588f1dd9670 con 0x5588f1d579c0 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.941+0000 7fc9477fe640 1 -- 172.21.15.98:0/2360048050 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc960036040 con 0x5588f1d579c0 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.941+0000 7fc9477fe640 1 -- 172.21.15.98:0/2360048050 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fc9580cc0a0 con 0x5588f1d579c0 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.941+0000 7fc97127a740 10 client.17694 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.941+0000 7fc97127a740 10 client.17694 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.941+0000 7fc97127a740 20 client.17694 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.941+0000 7fc97127a740 1 --2- 172.21.15.98:0/2360048050 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5588f1dd9a00 0x5588f1ddbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.941+0000 7fc97127a740 4 client.17694 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.941+0000 7fc97127a740 1 -- 172.21.15.98:0/2360048050 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5588f1cc4410 con 0x5588f1dd9a00 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.941+0000 7fc9677fe640 1 --2- 172.21.15.98:0/2360048050 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5588f1dd9a00 0x5588f1ddbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.941+0000 7fc9477fe640 10 client.17694 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.941+0000 7fc9677fe640 1 --2- 172.21.15.98:0/2360048050 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5588f1dd9a00 0x5588f1ddbdf0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ead072ae8b181bc server_cookie=e9f446bfa0d433f8 in_seq=0 out_seq=0 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.942+0000 7fc9477fe640 1 -- 172.21.15.98:0/2360048050 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5588f1cc4410 con 0x5588f1dd9a00 2024-05-09T08:58:13.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.942+0000 7fc9477fe640 10 client.17694 handle_command_reply: tid=0 2024-05-09T08:58:13.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.942+0000 7fc97127a740 1 client.17694 shutdown 2024-05-09T08:58:13.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc97127a740 1 -- 172.21.15.98:0/2360048050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d579c0 msgr2=0x5588f1d6a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:13.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc97127a740 1 --2- 172.21.15.98:0/2360048050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d579c0 0x5588f1d6a9e0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fc95809aa90 tx=0x7fc9580047d0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc97127a740 1 -- 172.21.15.98:0/2360048050 shutdown_connections 2024-05-09T08:58:13.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc97127a740 1 --2- 172.21.15.98:0/2360048050 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5588f1dd9a00 0x5588f1ddbdf0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc9477fe640 0 client.17694 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:13.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc97127a740 1 --2- 172.21.15.98:0/2360048050 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588f1d62740 0x5588f1d679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc97127a740 1 --2- 172.21.15.98:0/2360048050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5588f1d590b0 0x5588f1d6af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc97127a740 1 --2- 172.21.15.98:0/2360048050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588f1d579c0 0x5588f1d6a9e0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc97127a740 1 -- 172.21.15.98:0/2360048050 >> 172.21.15.98:0/2360048050 conn(0x5588f1d200a0 msgr2=0x5588f1d53080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:13.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc97127a740 1 -- 172.21.15.98:0/2360048050 shutdown_connections 2024-05-09T08:58:13.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc97127a740 1 -- 172.21.15.98:0/2360048050 wait complete. 2024-05-09T08:58:13.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc97127a740 20 client.17694 trim_cache size 0 max 16384 2024-05-09T08:58:13.949 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:58:13.949 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:58:13.949 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "9459b057-7b1c-4a1a-8565-2756aa5b6652", 2024-05-09T08:58:13.949 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:58:13.949 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:58:13.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc96e563640 1 -- 172.21.15.98:0/4081799973 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc9500a2d00 msgr2=0x7fc9500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:13.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc96e563640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc9500a2d00 0x7fc9500a51c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc960009a60 tx=0x7fc9600099f0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc96e563640 1 -- 172.21.15.98:0/4081799973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc968101800 msgr2=0x7fc968198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:13.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc96e563640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc968101800 0x7fc968198160 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fc95800b4a0 tx=0x7fc95800b960 comp rx=0 tx=0).stop 2024-05-09T08:58:13.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc96e563640 1 -- 172.21.15.98:0/4081799973 shutdown_connections 2024-05-09T08:58:13.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc96e563640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc9500a2d00 0x7fc9500a51c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc96e563640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc968107810 0x7fc96819ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc96e563640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9681021c0 0x7fc9681986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc96e563640 1 --2- 172.21.15.98:0/4081799973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc968101800 0x7fc968198160 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:13.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.943+0000 7fc96e563640 1 -- 172.21.15.98:0/4081799973 >> 172.21.15.98:0/4081799973 conn(0x7fc968078c80 msgr2=0x7fc9680ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:13.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.944+0000 7fc96e563640 1 -- 172.21.15.98:0/4081799973 shutdown_connections 2024-05-09T08:58:13.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:13.944+0000 7fc96e563640 1 -- 172.21.15.98:0/4081799973 wait complete. 2024-05-09T08:58:13.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:58:13.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:58:14.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.058+0000 7fded65dc640 1 -- 172.21.15.98:0/2148800512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded0107810 msgr2=0x7fded0107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:14.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.058+0000 7fded65dc640 1 --2- 172.21.15.98:0/2148800512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded0107810 0x7fded0107bf0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009930 tx=0x7fdec002a420 comp rx=0 tx=0).stop 2024-05-09T08:58:14.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.058+0000 7fded65dc640 1 -- 172.21.15.98:0/2148800512 shutdown_connections 2024-05-09T08:58:14.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.058+0000 7fded65dc640 1 --2- 172.21.15.98:0/2148800512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fded01021c0 0x7fded010e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.058+0000 7fded65dc640 1 --2- 172.21.15.98:0/2148800512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fded0101800 0x7fded0101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.058+0000 7fded65dc640 1 --2- 172.21.15.98:0/2148800512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded0107810 0x7fded0107bf0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.058+0000 7fded65dc640 1 -- 172.21.15.98:0/2148800512 >> 172.21.15.98:0/2148800512 conn(0x7fded0078c80 msgr2=0x7fded0079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:14.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.058+0000 7fded65dc640 1 -- 172.21.15.98:0/2148800512 shutdown_connections 2024-05-09T08:58:14.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.058+0000 7fded65dc640 1 -- 172.21.15.98:0/2148800512 wait complete. 2024-05-09T08:58:14.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.058+0000 7fded65dc640 1 Processor -- start 2024-05-09T08:58:14.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded65dc640 1 -- start start 2024-05-09T08:58:14.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded65dc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fded0101800 0x7fded01980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded65dc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fded01021c0 0x7fded0198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded65dc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded0107810 0x7fded019c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded65dc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fded0111be0 con 0x7fded0107810 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded65dc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fded0111a60 con 0x7fded01021c0 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded65dc640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fded0111d60 con 0x7fded0101800 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded4b79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded0107810 0x7fded019c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded4b79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded0107810 0x7fded019c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57484/0 (socket says 172.21.15.98:57484) 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded4b79640 1 -- 172.21.15.98:0/3487809484 learned_addr learned my addr 172.21.15.98:0/3487809484 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fdecf7fe640 1 --2- 172.21.15.98:0/3487809484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fded01021c0 0x7fded0198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fdecffff640 1 --2- 172.21.15.98:0/3487809484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fded0101800 0x7fded01980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded4b79640 1 -- 172.21.15.98:0/3487809484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fded0101800 msgr2=0x7fded01980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded4b79640 1 --2- 172.21.15.98:0/3487809484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fded0101800 0x7fded01980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded4b79640 1 -- 172.21.15.98:0/3487809484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fded01021c0 msgr2=0x7fded0198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded4b79640 1 --2- 172.21.15.98:0/3487809484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fded01021c0 0x7fded0198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.059+0000 7fded4b79640 1 -- 172.21.15.98:0/3487809484 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded019d0d0 con 0x7fded0107810 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.060+0000 7fded4b79640 1 --2- 172.21.15.98:0/3487809484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded0107810 0x7fded019c9a0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fdec8009fd0 tx=0x7fdec800def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.060+0000 7fdecd7fa640 1 -- 172.21.15.98:0/3487809484 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec8017070 con 0x7fded0107810 2024-05-09T08:58:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.060+0000 7fdecd7fa640 1 -- 172.21.15.98:0/3487809484 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdec8003e50 con 0x7fded0107810 2024-05-09T08:58:14.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.060+0000 7fded65dc640 1 -- 172.21.15.98:0/3487809484 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded019d3c0 con 0x7fded0107810 2024-05-09T08:58:14.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.060+0000 7fdecd7fa640 1 -- 172.21.15.98:0/3487809484 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec8010460 con 0x7fded0107810 2024-05-09T08:58:14.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.060+0000 7fded65dc640 1 -- 172.21.15.98:0/3487809484 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded01a4c50 con 0x7fded0107810 2024-05-09T08:58:14.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.062+0000 7fdecd7fa640 1 -- 172.21.15.98:0/3487809484 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fdec8003970 con 0x7fded0107810 2024-05-09T08:58:14.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.062+0000 7fded93c6740 1 Processor -- start 2024-05-09T08:58:14.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.062+0000 7fded93c6740 1 -- start start 2024-05-09T08:58:14.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.062+0000 7fded93c6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6280 0x56500b8e6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.062+0000 7fded93c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6ba0 0x56500b8f0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.062+0000 7fded93c6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8f1270 0x56500b8f3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.062+0000 7fded93c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56500b84bd00 con 0x56500b8e6ba0 2024-05-09T08:58:14.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.062+0000 7fded93c6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56500b7f34c0 con 0x56500b8f1270 2024-05-09T08:58:14.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.062+0000 7fded93c6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56500b847cc0 con 0x56500b8e6280 2024-05-09T08:58:14.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdecffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6ba0 0x56500b8f0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdecffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6ba0 0x56500b8f0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57500/0 (socket says 172.21.15.98:57500) 2024-05-09T08:58:14.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdecffff640 1 -- 172.21.15.98:0/2947447532 learned_addr learned my addr 172.21.15.98:0/2947447532 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdecd7fa640 1 --2- 172.21.15.98:0/3487809484 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdeb80a2d00 0x7fdeb80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdecf7fe640 1 --2- 172.21.15.98:0/2947447532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8f1270 0x56500b8f3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fded4b79640 1 --2- 172.21.15.98:0/3487809484 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdeb80a2d00 0x7fdeb80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdecf7fe640 1 --2- 172.21.15.98:0/2947447532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6280 0x56500b8e6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdecffff640 1 -- 172.21.15.98:0/2947447532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6280 msgr2=0x56500b8e6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdecffff640 1 --2- 172.21.15.98:0/2947447532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6280 0x56500b8e6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdecffff640 1 -- 172.21.15.98:0/2947447532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8f1270 msgr2=0x56500b8f3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdecffff640 1 --2- 172.21.15.98:0/2947447532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8f1270 0x56500b8f3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdecffff640 1 -- 172.21.15.98:0/2947447532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56500b6297d0 con 0x56500b8e6ba0 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdecd7fa640 1 -- 172.21.15.98:0/3487809484 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fdec80cf500 con 0x7fded0107810 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdecf7fe640 1 --2- 172.21.15.98:0/2947447532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6280 0x56500b8e6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdecffff640 1 --2- 172.21.15.98:0/2947447532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6ba0 0x56500b8f0d30 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009930 tx=0x7fdec0032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a027089444aee83f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fded4b79640 1 --2- 172.21.15.98:0/3487809484 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdeb80a2d00 0x7fdeb80a51c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdec80cc8f0 tx=0x7fdec8086040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdeb67fc640 1 -- 172.21.15.98:0/2947447532 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec003f070 con 0x56500b8e6ba0 2024-05-09T08:58:14.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdeb67fc640 1 -- 172.21.15.98:0/2947447532 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdec00317b0 con 0x56500b8e6ba0 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.063+0000 7fdeb67fc640 1 -- 172.21.15.98:0/2947447532 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec0035070 con 0x56500b8e6ba0 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.064+0000 7fded93c6740 1 -- 172.21.15.98:0/2947447532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6ba0 msgr2=0x56500b8f0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.064+0000 7fded93c6740 1 --2- 172.21.15.98:0/2947447532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6ba0 0x56500b8f0d30 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009930 tx=0x7fdec0032ce0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.064+0000 7fded93c6740 1 -- 172.21.15.98:0/2947447532 shutdown_connections 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.064+0000 7fded93c6740 1 --2- 172.21.15.98:0/2947447532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8f1270 0x56500b8f3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.064+0000 7fded93c6740 1 --2- 172.21.15.98:0/2947447532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6ba0 0x56500b8f0d30 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.064+0000 7fded93c6740 1 --2- 172.21.15.98:0/2947447532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6280 0x56500b8e6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.064+0000 7fded93c6740 1 -- 172.21.15.98:0/2947447532 >> 172.21.15.98:0/2947447532 conn(0x56500b8e25a0 msgr2=0x56500b8e2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.064+0000 7fded93c6740 1 -- 172.21.15.98:0/2947447532 shutdown_connections 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.064+0000 7fded93c6740 1 -- 172.21.15.98:0/2947447532 wait complete. 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.064+0000 7fded93c6740 1 -- start start 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fded93c6740 1 Processor -- start 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fded93c6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8e6280 0x56500b8ee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fded93c6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6ba0 0x56500b8ee730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fded93c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8f1270 0x56500b8eec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fded93c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56500b8f4000 con 0x56500b8f1270 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fded93c6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56500b806b90 con 0x56500b8e6280 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fded93c6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56500b8f4180 con 0x56500b8e6ba0 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fded93c6740 10 client.0 _ll_register_callbacks cb 0x56500b842da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fdecf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8f1270 0x56500b8eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fdecf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8f1270 0x56500b8eec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57504/0 (socket says 172.21.15.98:57504) 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fdecf7fe640 1 -- 172.21.15.98:0/1579980337 learned_addr learned my addr 172.21.15.98:0/1579980337 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fdecf7fe640 1 --2- 172.21.15.98:0/1579980337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8e6280 0x56500b8ee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fdecffff640 1 --2- 172.21.15.98:0/1579980337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6ba0 0x56500b8ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fdecf7fe640 1 -- 172.21.15.98:0/1579980337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6ba0 msgr2=0x56500b8ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fdecf7fe640 1 --2- 172.21.15.98:0/1579980337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6ba0 0x56500b8ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fdecf7fe640 1 -- 172.21.15.98:0/1579980337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8e6280 msgr2=0x56500b8ee1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fdecf7fe640 1 --2- 172.21.15.98:0/1579980337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8e6280 0x56500b8ee1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fdecf7fe640 1 -- 172.21.15.98:0/1579980337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56500b55f3f0 con 0x56500b8f1270 2024-05-09T08:58:14.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fdecf7fe640 1 --2- 172.21.15.98:0/1579980337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8f1270 0x56500b8eec70 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fdec4011fc0 tx=0x7fdec40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f81aeb7b8216f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:14.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fdeaf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:14.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.065+0000 7fdeaf7fe640 1 -- 172.21.15.98:0/1579980337 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4005220 con 0x56500b8f1270 2024-05-09T08:58:14.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.066+0000 7fdeaf7fe640 1 -- 172.21.15.98:0/1579980337 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fdec40053c0 con 0x56500b8f1270 2024-05-09T08:58:14.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.066+0000 7fded93c6740 1 -- 172.21.15.98:0/1579980337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdec00317b0 con 0x56500b8f1270 2024-05-09T08:58:14.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.066+0000 7fdeaf7fe640 1 -- 172.21.15.98:0/1579980337 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fdec402a050 con 0x56500b8f1270 2024-05-09T08:58:14.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.066+0000 7fded93c6740 10 client.17706 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:14.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.066+0000 7fded93c6740 1 -- 172.21.15.98:0/1579980337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56500b51f0f0 con 0x56500b8f1270 2024-05-09T08:58:14.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.066+0000 7fdeaf7fe640 1 -- 172.21.15.98:0/1579980337 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdec00317b0 con 0x56500b8f1270 2024-05-09T08:58:14.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.066+0000 7fdeaf7fe640 1 -- 172.21.15.98:0/1579980337 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fdec400d6a0 con 0x56500b8f1270 2024-05-09T08:58:14.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.066+0000 7fded93c6740 10 client.17706 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:14.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.066+0000 7fded93c6740 10 client.17706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:14.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.066+0000 7fded93c6740 20 client.17706 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:14.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.066+0000 7fded93c6740 1 --2- 172.21.15.98:0/1579980337 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56500b901590 0x56500b9039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.066+0000 7fded93c6740 4 client.17706 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.066+0000 7fded93c6740 1 -- 172.21.15.98:0/1579980337 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56500b859430 con 0x56500b901590 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.066+0000 7fdecffff640 1 --2- 172.21.15.98:0/1579980337 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56500b901590 0x56500b9039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.067+0000 7fdeaf7fe640 10 client.17706 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.067+0000 7fdecffff640 1 --2- 172.21.15.98:0/1579980337 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56500b901590 0x56500b9039f0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98877f586d578115 server_cookie=d73645357b1dcf35 in_seq=0 out_seq=0 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.067+0000 7fdeaf7fe640 1 -- 172.21.15.98:0/1579980337 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56500b859430 con 0x56500b901590 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.067+0000 7fdeaf7fe640 10 client.17706 handle_command_reply: tid=0 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.067+0000 7fded93c6740 1 client.17706 shutdown 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.067+0000 7fded93c6740 1 -- 172.21.15.98:0/1579980337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8f1270 msgr2=0x56500b8eec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.067+0000 7fded93c6740 1 --2- 172.21.15.98:0/1579980337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8f1270 0x56500b8eec70 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fdec4011fc0 tx=0x7fdec40046c0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.068+0000 7fded93c6740 1 -- 172.21.15.98:0/1579980337 shutdown_connections 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.068+0000 7fded93c6740 1 --2- 172.21.15.98:0/1579980337 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56500b901590 0x56500b9039f0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.068+0000 7fded93c6740 1 --2- 172.21.15.98:0/1579980337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8f1270 0x56500b8eec70 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.068+0000 7fdeaf7fe640 0 client.17706 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.068+0000 7fded93c6740 1 --2- 172.21.15.98:0/1579980337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6ba0 0x56500b8ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.068+0000 7fded93c6740 1 --2- 172.21.15.98:0/1579980337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8e6280 0x56500b8ee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.068+0000 7fded93c6740 1 -- 172.21.15.98:0/1579980337 >> 172.21.15.98:0/1579980337 conn(0x56500b8e25a0 msgr2=0x56500b8e17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:14.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.068+0000 7fded93c6740 1 -- 172.21.15.98:0/1579980337 shutdown_connections 2024-05-09T08:58:14.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.068+0000 7fded93c6740 1 -- 172.21.15.98:0/1579980337 wait complete. 2024-05-09T08:58:14.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.068+0000 7fded93c6740 20 client.17706 trim_cache size 0 max 16384 2024-05-09T08:58:14.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fded93c6740 1 Processor -- start 2024-05-09T08:58:14.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fded93c6740 1 -- start start 2024-05-09T08:58:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fded93c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6280 0x56500b8fcf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fded93c6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6ba0 0x56500b8e8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fded93c6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8f1270 0x56500b9092c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fded93c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fded01aeda0 con 0x56500b8e6280 2024-05-09T08:58:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fded93c6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fded01b08e0 con 0x56500b8f1270 2024-05-09T08:58:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fded93c6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fded01af470 con 0x56500b8e6ba0 2024-05-09T08:58:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fdecf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6280 0x56500b8fcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fdecf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6280 0x56500b8fcf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57516/0 (socket says 172.21.15.98:57516) 2024-05-09T08:58:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fdecf7fe640 1 -- 172.21.15.98:0/144032255 learned_addr learned my addr 172.21.15.98:0/144032255 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fdecffff640 1 --2- 172.21.15.98:0/144032255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6ba0 0x56500b8e8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fdecf7fe640 1 -- 172.21.15.98:0/144032255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6ba0 msgr2=0x56500b8e8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fdecf7fe640 1 --2- 172.21.15.98:0/144032255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6ba0 0x56500b8e8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fdecf7fe640 1 -- 172.21.15.98:0/144032255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8f1270 msgr2=0x56500b9092c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:14.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fdecf7fe640 1 --2- 172.21.15.98:0/144032255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8f1270 0x56500b9092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fdecf7fe640 1 -- 172.21.15.98:0/144032255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56500b8e37d0 con 0x56500b8e6280 2024-05-09T08:58:14.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fdecffff640 1 --2- 172.21.15.98:0/144032255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6ba0 0x56500b8e8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:14.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.085+0000 7fdecf7fe640 1 --2- 172.21.15.98:0/144032255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6280 0x56500b8fcf20 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fdec400daf0 tx=0x7fdec400af20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d79cc79190d1f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:14.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.086+0000 7fdeb5ffb640 1 -- 172.21.15.98:0/144032255 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec400d610 con 0x56500b8e6280 2024-05-09T08:58:14.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.086+0000 7fdeb5ffb640 1 -- 172.21.15.98:0/144032255 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdec400d7b0 con 0x56500b8e6280 2024-05-09T08:58:14.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.086+0000 7fdeb5ffb640 1 -- 172.21.15.98:0/144032255 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4004990 con 0x56500b8e6280 2024-05-09T08:58:14.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.086+0000 7fded93c6740 1 -- 172.21.15.98:0/144032255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6280 msgr2=0x56500b8fcf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:14.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.086+0000 7fded93c6740 1 --2- 172.21.15.98:0/144032255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6280 0x56500b8fcf20 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fdec400daf0 tx=0x7fdec400af20 comp rx=0 tx=0).stop 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.086+0000 7fded93c6740 1 -- 172.21.15.98:0/144032255 shutdown_connections 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.086+0000 7fded93c6740 1 --2- 172.21.15.98:0/144032255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8f1270 0x56500b9092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.086+0000 7fded93c6740 1 --2- 172.21.15.98:0/144032255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8e6ba0 0x56500b8e8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.086+0000 7fded93c6740 1 --2- 172.21.15.98:0/144032255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6280 0x56500b8fcf20 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.086+0000 7fded93c6740 1 -- 172.21.15.98:0/144032255 >> 172.21.15.98:0/144032255 conn(0x56500b8e25a0 msgr2=0x56500b8e7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.086+0000 7fded93c6740 1 -- 172.21.15.98:0/144032255 shutdown_connections 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.086+0000 7fded93c6740 1 -- 172.21.15.98:0/144032255 wait complete. 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.086+0000 7fded93c6740 1 -- start start 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fded93c6740 1 Processor -- start 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fded93c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6280 0x56500b8f95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fded93c6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8e6ba0 0x56500b8f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fded93c6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8f1270 0x56500b8f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fded93c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fded01b0210 con 0x56500b8e6280 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fded93c6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fded01afb40 con 0x56500b8e6ba0 2024-05-09T08:58:14.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fded93c6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fded01acda0 con 0x56500b8f1270 2024-05-09T08:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fded93c6740 10 client.0 _ll_register_callbacks cb 0x7fdec400ad00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fdecf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6280 0x56500b8f95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fdecf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6280 0x56500b8f95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57522/0 (socket says 172.21.15.98:57522) 2024-05-09T08:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fdecf7fe640 1 -- 172.21.15.98:0/1482079105 learned_addr learned my addr 172.21.15.98:0/1482079105 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fdecffff640 1 --2- 172.21.15.98:0/1482079105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8e6ba0 0x56500b8f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fdecf7fe640 1 -- 172.21.15.98:0/1482079105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8f1270 msgr2=0x56500b8f4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fdecf7fe640 1 --2- 172.21.15.98:0/1482079105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8f1270 0x56500b8f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fdecf7fe640 1 -- 172.21.15.98:0/1482079105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8e6ba0 msgr2=0x56500b8f49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fdecf7fe640 1 --2- 172.21.15.98:0/1482079105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8e6ba0 0x56500b8f49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fdecf7fe640 1 -- 172.21.15.98:0/1482079105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56500b8f54e0 con 0x56500b8e6280 2024-05-09T08:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fdecffff640 1 --2- 172.21.15.98:0/1482079105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8e6ba0 0x56500b8f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fdecf7fe640 1 --2- 172.21.15.98:0/1482079105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6280 0x56500b8f95f0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fdec40058c0 tx=0x7fdec4011570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed142a47d99df80 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.087+0000 7fdeaf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fdeaf7fe640 1 -- 172.21.15.98:0/1482079105 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec403d070 con 0x56500b8e6280 2024-05-09T08:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fdeaf7fe640 1 -- 172.21.15.98:0/1482079105 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fdec4037660 con 0x56500b8e6280 2024-05-09T08:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fded93c6740 1 -- 172.21.15.98:0/1482079105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdec400d7b0 con 0x56500b8e6280 2024-05-09T08:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fdeaf7fe640 1 -- 172.21.15.98:0/1482079105 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fdec4037860 con 0x56500b8e6280 2024-05-09T08:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fded93c6740 10 client.17712 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fded93c6740 1 -- 172.21.15.98:0/1482079105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56500b9678b0 con 0x56500b8e6280 2024-05-09T08:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fdeaf7fe640 1 -- 172.21.15.98:0/1482079105 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdec400d7b0 con 0x56500b8e6280 2024-05-09T08:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fdeaf7fe640 1 -- 172.21.15.98:0/1482079105 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fdec401fa80 con 0x56500b8e6280 2024-05-09T08:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fded93c6740 10 client.17712 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fded93c6740 10 client.17712 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fded93c6740 20 client.17712 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fded93c6740 1 --2- 172.21.15.98:0/1482079105 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56500b967e60 0x56500b96a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fded93c6740 4 client.17712 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:58:14.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fded93c6740 1 -- 172.21.15.98:0/1482079105 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56500b853780 con 0x56500b967e60 2024-05-09T08:58:14.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:13 smithi098 ceph-mon[30092]: pgmap v235: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-05-09T08:58:14.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:13 smithi098 ceph-mon[30092]: osdmap e101: 12 total, 12 up, 12 in 2024-05-09T08:58:14.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.088+0000 7fdecffff640 1 --2- 172.21.15.98:0/1482079105 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56500b967e60 0x56500b96a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:14.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fdecffff640 1 --2- 172.21.15.98:0/1482079105 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56500b967e60 0x56500b96a280 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5271442fcaa3a10a server_cookie=32f302b714a32575 in_seq=0 out_seq=0 2024-05-09T08:58:14.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fdeaf7fe640 10 client.17712 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fdeaf7fe640 1 -- 172.21.15.98:0/1482079105 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56500b853780 con 0x56500b967e60 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fdeaf7fe640 10 client.17712 handle_command_reply: tid=0 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fded93c6740 1 client.17712 shutdown 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fded93c6740 1 -- 172.21.15.98:0/1482079105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6280 msgr2=0x56500b8f95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fded93c6740 1 --2- 172.21.15.98:0/1482079105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6280 0x56500b8f95f0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fdec40058c0 tx=0x7fdec4011570 comp rx=0 tx=0).stop 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fded93c6740 1 -- 172.21.15.98:0/1482079105 shutdown_connections 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fded93c6740 1 --2- 172.21.15.98:0/1482079105 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56500b967e60 0x56500b96a280 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fded93c6740 1 --2- 172.21.15.98:0/1482079105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500b8f1270 0x56500b8f4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fdeaf7fe640 0 client.17712 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fded93c6740 1 --2- 172.21.15.98:0/1482079105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56500b8e6ba0 0x56500b8f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fded93c6740 1 --2- 172.21.15.98:0/1482079105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56500b8e6280 0x56500b8f95f0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fded93c6740 1 -- 172.21.15.98:0/1482079105 >> 172.21.15.98:0/1482079105 conn(0x56500b8e25a0 msgr2=0x56500b8eea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fded93c6740 1 -- 172.21.15.98:0/1482079105 shutdown_connections 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fded93c6740 1 -- 172.21.15.98:0/1482079105 wait complete. 2024-05-09T08:58:14.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.089+0000 7fded93c6740 20 client.17712 trim_cache size 0 max 16384 2024-05-09T08:58:14.098 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:58:14.098 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:58:14.098 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:58:14.098 INFO:teuthology.orchestra.run.smithi098.stdout: "9459b057-7b1c-4a1a-8565-2756aa5b6652": { 2024-05-09T08:58:14.098 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:58:14.098 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "9459b057-7b1c-4a1a-8565-2756aa5b6652", 2024-05-09T08:58:14.098 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:58:14.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:58:14.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:58:14.098 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:58:14.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.090+0000 7fded65dc640 1 -- 172.21.15.98:0/3487809484 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdeb80a2d00 msgr2=0x7fdeb80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:14.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.090+0000 7fded65dc640 1 --2- 172.21.15.98:0/3487809484 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdeb80a2d00 0x7fdeb80a51c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdec80cc8f0 tx=0x7fdec8086040 comp rx=0 tx=0).stop 2024-05-09T08:58:14.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.090+0000 7fded65dc640 1 -- 172.21.15.98:0/3487809484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded0107810 msgr2=0x7fded019c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:14.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.090+0000 7fded65dc640 1 --2- 172.21.15.98:0/3487809484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded0107810 0x7fded019c9a0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fdec8009fd0 tx=0x7fdec800def0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.090+0000 7fded65dc640 1 -- 172.21.15.98:0/3487809484 shutdown_connections 2024-05-09T08:58:14.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.090+0000 7fded65dc640 1 --2- 172.21.15.98:0/3487809484 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdeb80a2d00 0x7fdeb80a51c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.090+0000 7fded65dc640 1 --2- 172.21.15.98:0/3487809484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded0107810 0x7fded019c9a0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.090+0000 7fded65dc640 1 --2- 172.21.15.98:0/3487809484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fded01021c0 0x7fded0198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.090+0000 7fded65dc640 1 --2- 172.21.15.98:0/3487809484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fded0101800 0x7fded01980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:14.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.090+0000 7fded65dc640 1 -- 172.21.15.98:0/3487809484 >> 172.21.15.98:0/3487809484 conn(0x7fded0078c80 msgr2=0x7fded00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:14.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.090+0000 7fded65dc640 1 -- 172.21.15.98:0/3487809484 shutdown_connections 2024-05-09T08:58:14.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:14.090+0000 7fded65dc640 1 -- 172.21.15.98:0/3487809484 wait complete. 2024-05-09T08:58:14.101 INFO:tasks.cephfs.filesystem:scrub status for tag:9459b057-7b1c-4a1a-8565-2756aa5b6652 - {'path': '/', 'tag': '9459b057-7b1c-4a1a-8565-2756aa5b6652', 'options': 'recursive,force'} 2024-05-09T08:58:14.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:13 smithi105 ceph-mon[34498]: pgmap v235: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-05-09T08:58:14.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:13 smithi105 ceph-mon[34498]: osdmap e101: 12 total, 12 up, 12 in 2024-05-09T08:58:14.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:13 smithi142 ceph-mon[34625]: pgmap v235: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-05-09T08:58:14.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:13 smithi142 ceph-mon[34625]: osdmap e101: 12 total, 12 up, 12 in 2024-05-09T08:58:15.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:14 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:58:15.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:14 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:58:15.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:14 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:58:15.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:14 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:58:15.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:14 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:58:15.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:14 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:58:15.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:14 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:58:15.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:14 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:58:15.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:14 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:58:15.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:14 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:58:15.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:14 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:58:15.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:14 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:58:16.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:15 smithi105 ceph-mon[34498]: pgmap v237: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-09T08:58:16.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:15 smithi098 ceph-mon[30092]: pgmap v237: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-09T08:58:16.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:15 smithi142 ceph-mon[34625]: pgmap v237: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-09T08:58:17.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:16 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:58:17.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:16 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:58:17.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:16 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:58:18.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:17 smithi105 ceph-mon[34498]: pgmap v238: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-05-09T08:58:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:17 smithi098 ceph-mon[30092]: pgmap v238: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-05-09T08:58:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:17 smithi142 ceph-mon[34625]: pgmap v238: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-05-09T08:58:20.242 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:19 smithi105 ceph-mon[34498]: pgmap v239: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-05-09T08:58:20.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:19 smithi098 ceph-mon[30092]: pgmap v239: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-05-09T08:58:20.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:19 smithi142 ceph-mon[34625]: pgmap v239: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-05-09T08:58:22.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:21 smithi105 ceph-mon[34498]: pgmap v240: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 511 B/s wr, 14 op/s 2024-05-09T08:58:22.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:21 smithi098 ceph-mon[30092]: pgmap v240: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 511 B/s wr, 14 op/s 2024-05-09T08:58:22.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:21 smithi142 ceph-mon[34625]: pgmap v240: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 511 B/s wr, 14 op/s 2024-05-09T08:58:22.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.563+0000 7f803d2c3640 1 -- 172.21.15.98:0/1440870380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8038101840 msgr2=0x7f8038101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:22.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.563+0000 7f803d2c3640 1 --2- 172.21.15.98:0/1440870380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8038101840 0x7f8038101cc0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f8020009930 tx=0x7f802002a420 comp rx=0 tx=0).stop 2024-05-09T08:58:22.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.563+0000 7f803d2c3640 1 -- 172.21.15.98:0/1440870380 shutdown_connections 2024-05-09T08:58:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.563+0000 7f803d2c3640 1 --2- 172.21.15.98:0/1440870380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8038102200 0x7f803810e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.563+0000 7f803d2c3640 1 --2- 172.21.15.98:0/1440870380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8038101840 0x7f8038101cc0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.563+0000 7f803d2c3640 1 --2- 172.21.15.98:0/1440870380 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8038107850 0x7f8038107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.563+0000 7f803d2c3640 1 -- 172.21.15.98:0/1440870380 >> 172.21.15.98:0/1440870380 conn(0x7f8038078cc0 msgr2=0x7f80380790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.563+0000 7f803d2c3640 1 -- 172.21.15.98:0/1440870380 shutdown_connections 2024-05-09T08:58:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.563+0000 7f803d2c3640 1 -- 172.21.15.98:0/1440870380 wait complete. 2024-05-09T08:58:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.564+0000 7f803d2c3640 1 Processor -- start 2024-05-09T08:58:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.564+0000 7f803d2c3640 1 -- start start 2024-05-09T08:58:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.564+0000 7f803d2c3640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8038101840 0x7f80381980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.564+0000 7f803d2c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8038102200 0x7f8038198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.564+0000 7f803d2c3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8038107850 0x7f803819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.564+0000 7f803d2c3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8038111c60 con 0x7f8038102200 2024-05-09T08:58:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.564+0000 7f803d2c3640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8038111ae0 con 0x7f8038107850 2024-05-09T08:58:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.564+0000 7f803d2c3640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8038111de0 con 0x7f8038101840 2024-05-09T08:58:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.564+0000 7f802ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8038102200 0x7f8038198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.564+0000 7f802ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8038102200 0x7f8038198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60816/0 (socket says 172.21.15.98:60816) 2024-05-09T08:58:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.564+0000 7f802ffff640 1 -- 172.21.15.98:0/4124740701 learned_addr learned my addr 172.21.15.98:0/4124740701 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.564+0000 7f80377fe640 1 --2- 172.21.15.98:0/4124740701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8038107850 0x7f803819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.564+0000 7f8036ffd640 1 --2- 172.21.15.98:0/4124740701 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8038101840 0x7f80381980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.565+0000 7f802ffff640 1 -- 172.21.15.98:0/4124740701 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8038101840 msgr2=0x7f80381980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.565+0000 7f802ffff640 1 --2- 172.21.15.98:0/4124740701 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8038101840 0x7f80381980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.565+0000 7f802ffff640 1 -- 172.21.15.98:0/4124740701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8038107850 msgr2=0x7f803819c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.565+0000 7f802ffff640 1 --2- 172.21.15.98:0/4124740701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8038107850 0x7f803819c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.565+0000 7f802ffff640 1 -- 172.21.15.98:0/4124740701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f803819d080 con 0x7f8038102200 2024-05-09T08:58:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.565+0000 7f802ffff640 1 --2- 172.21.15.98:0/4124740701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8038102200 0x7f8038198620 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f8020009230 tx=0x7f8020004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.565+0000 7f8034ff9640 1 -- 172.21.15.98:0/4124740701 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802003f030 con 0x7f8038102200 2024-05-09T08:58:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.565+0000 7f8034ff9640 1 -- 172.21.15.98:0/4124740701 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8020003ad0 con 0x7f8038102200 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.565+0000 7f803d2c3640 1 -- 172.21.15.98:0/4124740701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f803819d310 con 0x7f8038102200 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.565+0000 7f8034ff9640 1 -- 172.21.15.98:0/4124740701 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8020035070 con 0x7f8038102200 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.565+0000 7f803d2c3640 1 -- 172.21.15.98:0/4124740701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80381a4c00 con 0x7f8038102200 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.567+0000 7f8034ff9640 1 -- 172.21.15.98:0/4124740701 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8020042030 con 0x7f8038102200 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f803ffcb740 1 Processor -- start 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f803ffcb740 1 -- start start 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f803ffcb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c92350330 0x564c92350710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f803ffcb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 0x564c9235f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f803ffcb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c9235fa40 0x564c92361e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f8034ff9640 1 --2- 172.21.15.98:0/4124740701 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f80240a2da0 0x7f80240a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f803ffcb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564c922b5340 con 0x564c92350330 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f803ffcb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564c9225d290 con 0x564c92350c50 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f803ffcb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564c922b1300 con 0x564c9235fa40 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f80377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c92350330 0x564c92350710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f80377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c92350330 0x564c92350710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60828/0 (socket says 172.21.15.98:60828) 2024-05-09T08:58:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f80377fe640 1 -- 172.21.15.98:0/681940494 learned_addr learned my addr 172.21.15.98:0/681940494 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f8036ffd640 1 --2- 172.21.15.98:0/681940494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 0x564c9235f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f80377fe640 1 --2- 172.21.15.98:0/4124740701 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f80240a2da0 0x7f80240a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f802ffff640 1 --2- 172.21.15.98:0/681940494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c9235fa40 0x564c92361e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.568+0000 7f8034ff9640 1 -- 172.21.15.98:0/4124740701 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f80200efd30 con 0x7f8038102200 2024-05-09T08:58:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.569+0000 7f8036ffd640 1 -- 172.21.15.98:0/681940494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c9235fa40 msgr2=0x564c92361e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.569+0000 7f8036ffd640 1 --2- 172.21.15.98:0/681940494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c9235fa40 0x564c92361e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.569+0000 7f8036ffd640 1 -- 172.21.15.98:0/681940494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c92350330 msgr2=0x564c92350710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.569+0000 7f8036ffd640 1 --2- 172.21.15.98:0/681940494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c92350330 0x564c92350710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.569+0000 7f8036ffd640 1 -- 172.21.15.98:0/681940494 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c922818e0 con 0x564c92350c50 2024-05-09T08:58:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.569+0000 7f80377fe640 1 --2- 172.21.15.98:0/681940494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c92350330 0x564c92350710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:58:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.569+0000 7f80377fe640 1 --2- 172.21.15.98:0/4124740701 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f80240a2da0 0x7f80240a5260 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8030009fd0 tx=0x7f80300095f0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.569+0000 7f8036ffd640 1 --2- 172.21.15.98:0/681940494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 0x564c9235f500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f802800bb90 tx=0x7f802800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12eeeb053b54152a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.569+0000 7f802d7fa640 1 -- 172.21.15.98:0/681940494 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8028015070 con 0x564c92350c50 2024-05-09T08:58:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.569+0000 7f802d7fa640 1 -- 172.21.15.98:0/681940494 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8028005500 con 0x564c92350c50 2024-05-09T08:58:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.569+0000 7f802d7fa640 1 -- 172.21.15.98:0/681940494 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8028011440 con 0x564c92350c50 2024-05-09T08:58:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.570+0000 7f803ffcb740 1 -- 172.21.15.98:0/681940494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 msgr2=0x564c9235f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.570+0000 7f803ffcb740 1 --2- 172.21.15.98:0/681940494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 0x564c9235f500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f802800bb90 tx=0x7f802800f730 comp rx=0 tx=0).stop 2024-05-09T08:58:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.570+0000 7f803ffcb740 1 -- 172.21.15.98:0/681940494 shutdown_connections 2024-05-09T08:58:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.570+0000 7f803ffcb740 1 --2- 172.21.15.98:0/681940494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c9235fa40 0x564c92361e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.570+0000 7f803ffcb740 1 --2- 172.21.15.98:0/681940494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 0x564c9235f500 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.570+0000 7f803ffcb740 1 --2- 172.21.15.98:0/681940494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c92350330 0x564c92350710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.570+0000 7f803ffcb740 1 -- 172.21.15.98:0/681940494 >> 172.21.15.98:0/681940494 conn(0x564c9234c960 msgr2=0x564c9234cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.570+0000 7f803ffcb740 1 -- 172.21.15.98:0/681940494 shutdown_connections 2024-05-09T08:58:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.570+0000 7f803ffcb740 1 -- 172.21.15.98:0/681940494 wait complete. 2024-05-09T08:58:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.570+0000 7f803ffcb740 1 -- start start 2024-05-09T08:58:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.570+0000 7f803ffcb740 1 Processor -- start 2024-05-09T08:58:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f803ffcb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c92350330 0x564c9235c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f803ffcb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 0x564c9235cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f803ffcb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c9235fa40 0x564c9235d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f803ffcb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564c92362670 con 0x564c9235fa40 2024-05-09T08:58:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f803ffcb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564c92270790 con 0x564c92350c50 2024-05-09T08:58:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f803ffcb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564c923627f0 con 0x564c92350330 2024-05-09T08:58:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f803ffcb740 10 client.0 _ll_register_callbacks cb 0x564c922aba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f80377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c9235fa40 0x564c9235d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f80377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c9235fa40 0x564c9235d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60842/0 (socket says 172.21.15.98:60842) 2024-05-09T08:58:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f80377fe640 1 -- 172.21.15.98:0/232434210 learned_addr learned my addr 172.21.15.98:0/232434210 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f802ffff640 1 --2- 172.21.15.98:0/232434210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 0x564c9235cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f8036ffd640 1 --2- 172.21.15.98:0/232434210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c92350330 0x564c9235c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f80377fe640 1 -- 172.21.15.98:0/232434210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c92350330 msgr2=0x564c9235c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f80377fe640 1 --2- 172.21.15.98:0/232434210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c92350330 0x564c9235c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f80377fe640 1 -- 172.21.15.98:0/232434210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 msgr2=0x564c9235cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f80377fe640 1 --2- 172.21.15.98:0/232434210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 0x564c9235cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f80377fe640 1 -- 172.21.15.98:0/232434210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c91fc9210 con 0x564c9235fa40 2024-05-09T08:58:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f8036ffd640 1 --2- 172.21.15.98:0/232434210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c92350330 0x564c9235c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f80377fe640 1 --2- 172.21.15.98:0/232434210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c9235fa40 0x564c9235d460 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8030009f10 tx=0x7f8030004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce88c46db010347e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f801e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f801e7fc640 1 -- 172.21.15.98:0/232434210 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8030025030 con 0x564c9235fa40 2024-05-09T08:58:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f801e7fc640 1 -- 172.21.15.98:0/232434210 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f803001b070 con 0x564c9235fa40 2024-05-09T08:58:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.571+0000 7f803ffcb740 1 -- 172.21.15.98:0/232434210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8028005500 con 0x564c9235fa40 2024-05-09T08:58:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.572+0000 7f801e7fc640 1 -- 172.21.15.98:0/232434210 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8030032050 con 0x564c9235fa40 2024-05-09T08:58:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.572+0000 7f803ffcb740 10 client.17724 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.572+0000 7f803ffcb740 1 -- 172.21.15.98:0/232434210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564c92095620 con 0x564c9235fa40 2024-05-09T08:58:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.572+0000 7f801e7fc640 1 -- 172.21.15.98:0/232434210 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8028005500 con 0x564c9235fa40 2024-05-09T08:58:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.572+0000 7f801e7fc640 1 -- 172.21.15.98:0/232434210 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f803001e050 con 0x564c9235fa40 2024-05-09T08:58:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.572+0000 7f803ffcb740 10 client.17724 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.572+0000 7f803ffcb740 10 client.17724 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.572+0000 7f803ffcb740 20 client.17724 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.572+0000 7f803ffcb740 1 --2- 172.21.15.98:0/232434210 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564c9236f810 0x564c92371c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.572+0000 7f803ffcb740 4 client.17724 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.572+0000 7f803ffcb740 1 -- 172.21.15.98:0/232434210 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c922c2a70 con 0x564c9236f810 2024-05-09T08:58:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.572+0000 7f8036ffd640 1 --2- 172.21.15.98:0/232434210 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564c9236f810 0x564c92371c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.573+0000 7f801e7fc640 10 client.17724 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.573+0000 7f8036ffd640 1 --2- 172.21.15.98:0/232434210 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564c9236f810 0x564c92371c20 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62059b7cb4315447 server_cookie=4dd84c3aaf894edb in_seq=0 out_seq=0 2024-05-09T08:58:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.573+0000 7f801e7fc640 1 -- 172.21.15.98:0/232434210 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564c922c2a70 con 0x564c9236f810 2024-05-09T08:58:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.573+0000 7f801e7fc640 10 client.17724 handle_command_reply: tid=0 2024-05-09T08:58:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.573+0000 7f803ffcb740 1 client.17724 shutdown 2024-05-09T08:58:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.573+0000 7f803ffcb740 1 -- 172.21.15.98:0/232434210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c9235fa40 msgr2=0x564c9235d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.573+0000 7f803ffcb740 1 --2- 172.21.15.98:0/232434210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c9235fa40 0x564c9235d460 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8030009f10 tx=0x7f8030004a40 comp rx=0 tx=0).stop 2024-05-09T08:58:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.574+0000 7f803ffcb740 1 -- 172.21.15.98:0/232434210 shutdown_connections 2024-05-09T08:58:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.574+0000 7f803ffcb740 1 --2- 172.21.15.98:0/232434210 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564c9236f810 0x564c92371c20 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.574+0000 7f803ffcb740 1 --2- 172.21.15.98:0/232434210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c9235fa40 0x564c9235d460 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.574+0000 7f803ffcb740 1 --2- 172.21.15.98:0/232434210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 0x564c9235cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.574+0000 7f803ffcb740 1 --2- 172.21.15.98:0/232434210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c92350330 0x564c9235c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.574+0000 7f801e7fc640 0 client.17724 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.574+0000 7f803ffcb740 1 -- 172.21.15.98:0/232434210 >> 172.21.15.98:0/232434210 conn(0x564c9234c960 msgr2=0x564c92352770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.574+0000 7f803ffcb740 1 -- 172.21.15.98:0/232434210 shutdown_connections 2024-05-09T08:58:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.574+0000 7f803ffcb740 1 -- 172.21.15.98:0/232434210 wait complete. 2024-05-09T08:58:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.574+0000 7f803ffcb740 20 client.17724 trim_cache size 0 max 16384 2024-05-09T08:58:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.589+0000 7f803ffcb740 1 Processor -- start 2024-05-09T08:58:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.589+0000 7f803ffcb740 1 -- start start 2024-05-09T08:58:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.589+0000 7f803ffcb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350330 0x564c9235f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.589+0000 7f803ffcb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c92350c50 0x564c9236b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.589+0000 7f803ffcb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c9235fa40 0x564c9235d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.589+0000 7f803ffcb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80381acb80 con 0x564c9235fa40 2024-05-09T08:58:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.589+0000 7f803ffcb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f80381ae6c0 con 0x564c92350330 2024-05-09T08:58:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.589+0000 7f803ffcb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f80381ad250 con 0x564c92350c50 2024-05-09T08:58:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.589+0000 7f80377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c9235fa40 0x564c9235d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.589+0000 7f80377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c9235fa40 0x564c9235d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60848/0 (socket says 172.21.15.98:60848) 2024-05-09T08:58:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.589+0000 7f80377fe640 1 -- 172.21.15.98:0/1342150166 learned_addr learned my addr 172.21.15.98:0/1342150166 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.590+0000 7f8036ffd640 1 --2- 172.21.15.98:0/1342150166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350330 0x564c9235f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.590+0000 7f80377fe640 1 -- 172.21.15.98:0/1342150166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c92350c50 msgr2=0x564c9236b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.590+0000 7f80377fe640 1 --2- 172.21.15.98:0/1342150166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c92350c50 0x564c9236b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.590+0000 7f80377fe640 1 -- 172.21.15.98:0/1342150166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350330 msgr2=0x564c9235f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.590+0000 7f80377fe640 1 --2- 172.21.15.98:0/1342150166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350330 0x564c9235f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.590+0000 7f80377fe640 1 -- 172.21.15.98:0/1342150166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c9235b990 con 0x564c9235fa40 2024-05-09T08:58:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.590+0000 7f80377fe640 1 --2- 172.21.15.98:0/1342150166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c9235fa40 0x564c9235d230 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f8030009be0 tx=0x7f803000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600af4d75e7675e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.590+0000 7f802cff9640 1 -- 172.21.15.98:0/1342150166 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8030025070 con 0x564c9235fa40 2024-05-09T08:58:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.590+0000 7f802cff9640 1 -- 172.21.15.98:0/1342150166 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8030005bd0 con 0x564c9235fa40 2024-05-09T08:58:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.590+0000 7f802cff9640 1 -- 172.21.15.98:0/1342150166 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80300216e0 con 0x564c9235fa40 2024-05-09T08:58:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 -- 172.21.15.98:0/1342150166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c9235fa40 msgr2=0x564c9235d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 --2- 172.21.15.98:0/1342150166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c9235fa40 0x564c9235d230 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f8030009be0 tx=0x7f803000bf90 comp rx=0 tx=0).stop 2024-05-09T08:58:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 -- 172.21.15.98:0/1342150166 shutdown_connections 2024-05-09T08:58:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 --2- 172.21.15.98:0/1342150166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c9235fa40 0x564c9235d230 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 --2- 172.21.15.98:0/1342150166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c92350c50 0x564c9236b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 --2- 172.21.15.98:0/1342150166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350330 0x564c9235f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 -- 172.21.15.98:0/1342150166 >> 172.21.15.98:0/1342150166 conn(0x564c9234c960 msgr2=0x564c9234cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 -- 172.21.15.98:0/1342150166 shutdown_connections 2024-05-09T08:58:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 -- 172.21.15.98:0/1342150166 wait complete. 2024-05-09T08:58:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 -- start start 2024-05-09T08:58:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 Processor -- start 2024-05-09T08:58:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c92350330 0x564c92367e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 0x564c92363170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c9235fa40 0x564c923636b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80381adff0 con 0x564c92350330 2024-05-09T08:58:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f80381ad920 con 0x564c92350c50 2024-05-09T08:58:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.591+0000 7f803ffcb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f80381aab80 con 0x564c9235fa40 2024-05-09T08:58:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f803ffcb740 10 client.0 _ll_register_callbacks cb 0x7f8030009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f8036ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c92350330 0x564c92367e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f8036ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c92350330 0x564c92367e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60862/0 (socket says 172.21.15.98:60862) 2024-05-09T08:58:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f8036ffd640 1 -- 172.21.15.98:0/2273098799 learned_addr learned my addr 172.21.15.98:0/2273098799 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f802ffff640 1 --2- 172.21.15.98:0/2273098799 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 0x564c92363170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f8036ffd640 1 -- 172.21.15.98:0/2273098799 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c9235fa40 msgr2=0x564c923636b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f80377fe640 1 --2- 172.21.15.98:0/2273098799 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c9235fa40 0x564c923636b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f8036ffd640 1 --2- 172.21.15.98:0/2273098799 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c9235fa40 0x564c923636b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f8036ffd640 1 -- 172.21.15.98:0/2273098799 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 msgr2=0x564c92363170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f8036ffd640 1 --2- 172.21.15.98:0/2273098799 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 0x564c92363170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f8036ffd640 1 -- 172.21.15.98:0/2273098799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c92363eb0 con 0x564c92350330 2024-05-09T08:58:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f80377fe640 1 --2- 172.21.15.98:0/2273098799 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c9235fa40 0x564c923636b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f8036ffd640 1 --2- 172.21.15.98:0/2273098799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c92350330 0x564c92367e00 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f8028004bf0 tx=0x7f802800b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59292ebdd8d41efa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f801e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f801e7fc640 1 -- 172.21.15.98:0/2273098799 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8028015040 con 0x564c92350330 2024-05-09T08:58:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f801e7fc640 1 -- 172.21.15.98:0/2273098799 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f8028026420 con 0x564c92350330 2024-05-09T08:58:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f803ffcb740 1 -- 172.21.15.98:0/2273098799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8030005bd0 con 0x564c92350330 2024-05-09T08:58:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f801e7fc640 1 -- 172.21.15.98:0/2273098799 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8028035050 con 0x564c92350330 2024-05-09T08:58:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f803ffcb740 10 client.17730 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.592+0000 7f803ffcb740 1 -- 172.21.15.98:0/2273098799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564c923d5d80 con 0x564c92350330 2024-05-09T08:58:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.593+0000 7f801e7fc640 1 -- 172.21.15.98:0/2273098799 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8030005bd0 con 0x564c92350330 2024-05-09T08:58:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.593+0000 7f801e7fc640 1 -- 172.21.15.98:0/2273098799 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f8028005170 con 0x564c92350330 2024-05-09T08:58:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.593+0000 7f803ffcb740 10 client.17730 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.593+0000 7f803ffcb740 10 client.17730 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:22.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.593+0000 7f803ffcb740 20 client.17730 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:22.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.593+0000 7f803ffcb740 1 --2- 172.21.15.98:0/2273098799 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564c923d6220 0x564c923d8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:22.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.593+0000 7f803ffcb740 4 client.17730 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:58:22.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.593+0000 7f803ffcb740 1 -- 172.21.15.98:0/2273098799 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564c922bcdc0 con 0x564c923d6220 2024-05-09T08:58:22.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.593+0000 7f802ffff640 1 --2- 172.21.15.98:0/2273098799 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564c923d6220 0x564c923d8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:22.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.593+0000 7f801e7fc640 10 client.17730 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:22.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.593+0000 7f802ffff640 1 --2- 172.21.15.98:0/2273098799 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564c923d6220 0x564c923d8640 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a72af2e7ccb96614 server_cookie=79a2b2ce6a892034 in_seq=0 out_seq=0 2024-05-09T08:58:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f801e7fc640 1 -- 172.21.15.98:0/2273098799 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564c922bcdc0 con 0x564c923d6220 2024-05-09T08:58:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f801e7fc640 10 client.17730 handle_command_reply: tid=0 2024-05-09T08:58:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f803ffcb740 1 client.17730 shutdown 2024-05-09T08:58:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f803ffcb740 1 -- 172.21.15.98:0/2273098799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c92350330 msgr2=0x564c92367e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f803ffcb740 1 --2- 172.21.15.98:0/2273098799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c92350330 0x564c92367e00 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f8028004bf0 tx=0x7f802800b720 comp rx=0 tx=0).stop 2024-05-09T08:58:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f803ffcb740 1 -- 172.21.15.98:0/2273098799 shutdown_connections 2024-05-09T08:58:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f803ffcb740 1 --2- 172.21.15.98:0/2273098799 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x564c923d6220 0x564c923d8640 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f803ffcb740 1 --2- 172.21.15.98:0/2273098799 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564c9235fa40 0x564c923636b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f803ffcb740 1 --2- 172.21.15.98:0/2273098799 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c92350c50 0x564c92363170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f801e7fc640 0 client.17730 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f803ffcb740 1 --2- 172.21.15.98:0/2273098799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564c92350330 0x564c92367e00 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f803ffcb740 1 -- 172.21.15.98:0/2273098799 >> 172.21.15.98:0/2273098799 conn(0x564c9234c960 msgr2=0x564c9235d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f803ffcb740 1 -- 172.21.15.98:0/2273098799 shutdown_connections 2024-05-09T08:58:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f803ffcb740 1 -- 172.21.15.98:0/2273098799 wait complete. 2024-05-09T08:58:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.605+0000 7f803ffcb740 20 client.17730 trim_cache size 0 max 16384 2024-05-09T08:58:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:58:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:58:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:58:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:58:22.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.606+0000 7f803d2c3640 1 -- 172.21.15.98:0/4124740701 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f80240a2da0 msgr2=0x7f80240a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:22.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.606+0000 7f803d2c3640 1 --2- 172.21.15.98:0/4124740701 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f80240a2da0 0x7f80240a5260 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8030009fd0 tx=0x7f80300095f0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.606+0000 7f803d2c3640 1 -- 172.21.15.98:0/4124740701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8038102200 msgr2=0x7f8038198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.606+0000 7f803d2c3640 1 --2- 172.21.15.98:0/4124740701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8038102200 0x7f8038198620 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f8020009230 tx=0x7f8020004270 comp rx=0 tx=0).stop 2024-05-09T08:58:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.606+0000 7f803d2c3640 1 -- 172.21.15.98:0/4124740701 shutdown_connections 2024-05-09T08:58:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.606+0000 7f803d2c3640 1 --2- 172.21.15.98:0/4124740701 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f80240a2da0 0x7f80240a5260 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.606+0000 7f803d2c3640 1 --2- 172.21.15.98:0/4124740701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8038107850 0x7f803819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.606+0000 7f803d2c3640 1 --2- 172.21.15.98:0/4124740701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8038102200 0x7f8038198620 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.606+0000 7f803d2c3640 1 --2- 172.21.15.98:0/4124740701 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8038101840 0x7f80381980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.606+0000 7f803d2c3640 1 -- 172.21.15.98:0/4124740701 >> 172.21.15.98:0/4124740701 conn(0x7f8038078cc0 msgr2=0x7f80380ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.606+0000 7f803d2c3640 1 -- 172.21.15.98:0/4124740701 shutdown_connections 2024-05-09T08:58:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:22.606+0000 7f803d2c3640 1 -- 172.21.15.98:0/4124740701 wait complete. 2024-05-09T08:58:24.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:23 smithi105 ceph-mon[34498]: pgmap v241: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-05-09T08:58:24.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:23 smithi098 ceph-mon[30092]: pgmap v241: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-05-09T08:58:24.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:23 smithi142 ceph-mon[34625]: pgmap v241: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-05-09T08:58:26.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:25 smithi105 ceph-mon[34498]: pgmap v242: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 13 op/s 2024-05-09T08:58:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:25 smithi098 ceph-mon[30092]: pgmap v242: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 13 op/s 2024-05-09T08:58:26.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:25 smithi142 ceph-mon[34625]: pgmap v242: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 13 op/s 2024-05-09T08:58:28.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:27 smithi105 ceph-mon[34498]: pgmap v243: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.7 KiB/s wr, 17 op/s 2024-05-09T08:58:28.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:27 smithi098 ceph-mon[30092]: pgmap v243: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.7 KiB/s wr, 17 op/s 2024-05-09T08:58:28.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:27 smithi142 ceph-mon[34625]: pgmap v243: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.7 KiB/s wr, 17 op/s 2024-05-09T08:58:30.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:29 smithi105 ceph-mon[34498]: pgmap v244: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.1 KiB/s wr, 12 op/s 2024-05-09T08:58:30.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:29 smithi098 ceph-mon[30092]: pgmap v244: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.1 KiB/s wr, 12 op/s 2024-05-09T08:58:30.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:29 smithi142 ceph-mon[34625]: pgmap v244: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.1 KiB/s wr, 12 op/s 2024-05-09T08:58:32.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:31 smithi142 ceph-mon[34625]: pgmap v245: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 KiB/s wr, 14 op/s 2024-05-09T08:58:32.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:31 smithi105 ceph-mon[34498]: pgmap v245: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 KiB/s wr, 14 op/s 2024-05-09T08:58:32.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:31 smithi098 ceph-mon[30092]: pgmap v245: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 KiB/s wr, 14 op/s 2024-05-09T08:58:34.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:33 smithi105 ceph-mon[34498]: pgmap v246: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.1 KiB/s wr, 12 op/s 2024-05-09T08:58:34.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:33 smithi098 ceph-mon[30092]: pgmap v246: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.1 KiB/s wr, 12 op/s 2024-05-09T08:58:34.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:33 smithi142 ceph-mon[34625]: pgmap v246: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.1 KiB/s wr, 12 op/s 2024-05-09T08:58:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.703+0000 7f27363a5640 1 -- 172.21.15.98:0/2048054057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2730102ae0 msgr2=0x7f2730102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.703+0000 7f27363a5640 1 --2- 172.21.15.98:0/2048054057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2730102ae0 0x7f2730102ec0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f2720009900 tx=0x7f272002a480 comp rx=0 tx=0).stop 2024-05-09T08:58:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.703+0000 7f27363a5640 1 -- 172.21.15.98:0/2048054057 shutdown_connections 2024-05-09T08:58:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.703+0000 7f27363a5640 1 --2- 172.21.15.98:0/2048054057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f273010beb0 0x7f273010e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.703+0000 7f27363a5640 1 --2- 172.21.15.98:0/2048054057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2730103400 0x7f273010b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.703+0000 7f27363a5640 1 --2- 172.21.15.98:0/2048054057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2730102ae0 0x7f2730102ec0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.703+0000 7f27363a5640 1 -- 172.21.15.98:0/2048054057 >> 172.21.15.98:0/2048054057 conn(0x7f2730076ab0 msgr2=0x7f2730076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.703+0000 7f27363a5640 1 -- 172.21.15.98:0/2048054057 shutdown_connections 2024-05-09T08:58:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.703+0000 7f27363a5640 1 -- 172.21.15.98:0/2048054057 wait complete. 2024-05-09T08:58:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.704+0000 7f27363a5640 1 Processor -- start 2024-05-09T08:58:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.704+0000 7f27363a5640 1 -- start start 2024-05-09T08:58:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.704+0000 7f27363a5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2730102ae0 0x7f2730100730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.704+0000 7f27363a5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2730103400 0x7f2730100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.704+0000 7f27363a5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f273010beb0 0x7f27301011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.704+0000 7f27363a5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2730111c40 con 0x7f2730103400 2024-05-09T08:58:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.704+0000 7f27363a5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2730111ac0 con 0x7f273010beb0 2024-05-09T08:58:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.704+0000 7f27363a5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2730111dc0 con 0x7f2730102ae0 2024-05-09T08:58:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f272f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2730103400 0x7f2730100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f272f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2730103400 0x7f2730100c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53576/0 (socket says 172.21.15.98:53576) 2024-05-09T08:58:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f272f7fe640 1 -- 172.21.15.98:0/741985626 learned_addr learned my addr 172.21.15.98:0/741985626 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f2734942640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f273010beb0 0x7f27301011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f272ffff640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2730102ae0 0x7f2730100730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f272f7fe640 1 -- 172.21.15.98:0/741985626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2730102ae0 msgr2=0x7f2730100730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f272f7fe640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2730102ae0 0x7f2730100730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f272f7fe640 1 -- 172.21.15.98:0/741985626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f273010beb0 msgr2=0x7f27301011b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f272f7fe640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f273010beb0 0x7f27301011b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f272f7fe640 1 -- 172.21.15.98:0/741985626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27301a6560 con 0x7f2730103400 2024-05-09T08:58:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f2734942640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f273010beb0 0x7f27301011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f272f7fe640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2730103400 0x7f2730100c70 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f271800d950 tx=0x7f271800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f272d7fa640 1 -- 172.21.15.98:0/741985626 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2718017070 con 0x7f2730103400 2024-05-09T08:58:35.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f27363a5640 1 -- 172.21.15.98:0/741985626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27301a6850 con 0x7f2730103400 2024-05-09T08:58:35.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f272d7fa640 1 -- 172.21.15.98:0/741985626 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2718009c30 con 0x7f2730103400 2024-05-09T08:58:35.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f272d7fa640 1 -- 172.21.15.98:0/741985626 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271800f6b0 con 0x7f2730103400 2024-05-09T08:58:35.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.705+0000 7f27363a5640 1 -- 172.21.15.98:0/741985626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27301a6df0 con 0x7f2730103400 2024-05-09T08:58:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.707+0000 7f272d7fa640 1 -- 172.21.15.98:0/741985626 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f27180109d0 con 0x7f2730103400 2024-05-09T08:58:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f273918e740 1 Processor -- start 2024-05-09T08:58:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f273918e740 1 -- start start 2024-05-09T08:58:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f273918e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814450 0x5590d6814830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f273918e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d6814d70 0x5590d681f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f273918e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d681f630 0x5590d6821a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f273918e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5590d6779340 con 0x5590d6814d70 2024-05-09T08:58:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f272d7fa640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f271c0a2da0 0x7f271c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f273918e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5590d6721290 con 0x5590d6814450 2024-05-09T08:58:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f273918e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5590d6775300 con 0x5590d681f630 2024-05-09T08:58:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f272ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d6814d70 0x5590d681f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f272ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d6814d70 0x5590d681f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53592/0 (socket says 172.21.15.98:53592) 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f272ffff640 1 -- 172.21.15.98:0/554668672 learned_addr learned my addr 172.21.15.98:0/554668672 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f2734942640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f271c0a2da0 0x7f271c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f272d7fa640 1 -- 172.21.15.98:0/741985626 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f27180cf4e0 con 0x7f2730103400 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f2734942640 1 --2- 172.21.15.98:0/554668672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814450 0x5590d6814830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.708+0000 7f272f7fe640 1 --2- 172.21.15.98:0/554668672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d681f630 0x5590d6821a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.709+0000 7f272ffff640 1 -- 172.21.15.98:0/554668672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d681f630 msgr2=0x5590d6821a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.709+0000 7f272ffff640 1 --2- 172.21.15.98:0/554668672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d681f630 0x5590d6821a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.709+0000 7f272ffff640 1 -- 172.21.15.98:0/554668672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814450 msgr2=0x5590d6814830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.709+0000 7f272ffff640 1 --2- 172.21.15.98:0/554668672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814450 0x5590d6814830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.709+0000 7f272ffff640 1 -- 172.21.15.98:0/554668672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590d67458e0 con 0x5590d6814d70 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.709+0000 7f2734942640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f271c0a2da0 0x7f271c0a5260 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f273006b140 tx=0x7f2728009cf0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.709+0000 7f272ffff640 1 --2- 172.21.15.98:0/554668672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d6814d70 0x5590d681f0f0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f2720006010 tx=0x7f2720031510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1888231ac18c0b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.709+0000 7f27267fc640 1 -- 172.21.15.98:0/554668672 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2720035030 con 0x5590d6814d70 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.709+0000 7f27267fc640 1 -- 172.21.15.98:0/554668672 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2720032e30 con 0x5590d6814d70 2024-05-09T08:58:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.709+0000 7f27267fc640 1 -- 172.21.15.98:0/554668672 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27200046c0 con 0x5590d6814d70 2024-05-09T08:58:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.710+0000 7f273918e740 1 -- 172.21.15.98:0/554668672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d6814d70 msgr2=0x5590d681f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.710+0000 7f273918e740 1 --2- 172.21.15.98:0/554668672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d6814d70 0x5590d681f0f0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f2720006010 tx=0x7f2720031510 comp rx=0 tx=0).stop 2024-05-09T08:58:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.710+0000 7f273918e740 1 -- 172.21.15.98:0/554668672 shutdown_connections 2024-05-09T08:58:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.710+0000 7f273918e740 1 --2- 172.21.15.98:0/554668672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d681f630 0x5590d6821a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.710+0000 7f273918e740 1 --2- 172.21.15.98:0/554668672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d6814d70 0x5590d681f0f0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.710+0000 7f273918e740 1 --2- 172.21.15.98:0/554668672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814450 0x5590d6814830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.710+0000 7f273918e740 1 -- 172.21.15.98:0/554668672 >> 172.21.15.98:0/554668672 conn(0x5590d6810960 msgr2=0x5590d6810d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.710+0000 7f273918e740 1 -- 172.21.15.98:0/554668672 shutdown_connections 2024-05-09T08:58:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.710+0000 7f273918e740 1 -- 172.21.15.98:0/554668672 wait complete. 2024-05-09T08:58:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.710+0000 7f273918e740 1 -- start start 2024-05-09T08:58:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f273918e740 1 Processor -- start 2024-05-09T08:58:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f273918e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814450 0x5590d681c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f273918e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d6814d70 0x5590d681ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f273918e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d681f630 0x5590d681cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f273918e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5590d6822200 con 0x5590d681f630 2024-05-09T08:58:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f273918e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5590d6734790 con 0x5590d6814450 2024-05-09T08:58:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f273918e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5590d6822380 con 0x5590d6814d70 2024-05-09T08:58:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f273918e740 10 client.0 _ll_register_callbacks cb 0x5590d676fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f2734942640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d681f630 0x5590d681cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f2734942640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d681f630 0x5590d681cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53608/0 (socket says 172.21.15.98:53608) 2024-05-09T08:58:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f2734942640 1 -- 172.21.15.98:0/84990306 learned_addr learned my addr 172.21.15.98:0/84990306 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f272ffff640 1 --2- 172.21.15.98:0/84990306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814450 0x5590d681c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f272f7fe640 1 --2- 172.21.15.98:0/84990306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d6814d70 0x5590d681ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f2734942640 1 -- 172.21.15.98:0/84990306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d6814d70 msgr2=0x5590d681ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f2734942640 1 --2- 172.21.15.98:0/84990306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d6814d70 0x5590d681ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f2734942640 1 -- 172.21.15.98:0/84990306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814450 msgr2=0x5590d681c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f2734942640 1 --2- 172.21.15.98:0/84990306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814450 0x5590d681c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f2734942640 1 -- 172.21.15.98:0/84990306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590d648d210 con 0x5590d681f630 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f272ffff640 1 --2- 172.21.15.98:0/84990306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814450 0x5590d681c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f2734942640 1 --2- 172.21.15.98:0/84990306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d681f630 0x5590d681cf70 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f2728010c30 tx=0x7f2728009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726a1dbbdfe589f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.711+0000 7f270f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.712+0000 7f270f7fe640 1 -- 172.21.15.98:0/84990306 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2728004300 con 0x5590d681f630 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.712+0000 7f270f7fe640 1 -- 172.21.15.98:0/84990306 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f27280044a0 con 0x5590d681f630 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.712+0000 7f273918e740 1 -- 172.21.15.98:0/84990306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2720032e30 con 0x5590d681f630 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.712+0000 7f270f7fe640 1 -- 172.21.15.98:0/84990306 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f27280046a0 con 0x5590d681f630 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.712+0000 7f273918e740 10 client.17742 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.712+0000 7f273918e740 1 -- 172.21.15.98:0/84990306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5590d6559620 con 0x5590d681f630 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.712+0000 7f270f7fe640 1 -- 172.21.15.98:0/84990306 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2720032e30 con 0x5590d681f630 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.712+0000 7f270f7fe640 1 -- 172.21.15.98:0/84990306 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f272801c070 con 0x5590d681f630 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.712+0000 7f273918e740 10 client.17742 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.712+0000 7f273918e740 10 client.17742 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.712+0000 7f273918e740 20 client.17742 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.712+0000 7f273918e740 1 --2- 172.21.15.98:0/84990306 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5590d682f2f0 0x5590d6831720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.712+0000 7f273918e740 4 client.17742 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.712+0000 7f273918e740 1 -- 172.21.15.98:0/84990306 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590d6786a70 con 0x5590d682f2f0 2024-05-09T08:58:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.713+0000 7f272ffff640 1 --2- 172.21.15.98:0/84990306 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5590d682f2f0 0x5590d6831720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.713+0000 7f272ffff640 1 --2- 172.21.15.98:0/84990306 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5590d682f2f0 0x5590d6831720 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ee9c7375085c060 server_cookie=5140e2c9b0d004b0 in_seq=0 out_seq=0 2024-05-09T08:58:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.713+0000 7f270f7fe640 10 client.17742 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f270f7fe640 1 -- 172.21.15.98:0/84990306 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5590d6786a70 con 0x5590d682f2f0 2024-05-09T08:58:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f270f7fe640 10 client.17742 handle_command_reply: tid=0 2024-05-09T08:58:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f273918e740 1 client.17742 shutdown 2024-05-09T08:58:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f273918e740 1 -- 172.21.15.98:0/84990306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d681f630 msgr2=0x5590d681cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f273918e740 1 --2- 172.21.15.98:0/84990306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d681f630 0x5590d681cf70 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f2728010c30 tx=0x7f2728009240 comp rx=0 tx=0).stop 2024-05-09T08:58:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f273918e740 1 -- 172.21.15.98:0/84990306 shutdown_connections 2024-05-09T08:58:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f273918e740 1 --2- 172.21.15.98:0/84990306 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5590d682f2f0 0x5590d6831720 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f273918e740 1 --2- 172.21.15.98:0/84990306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d681f630 0x5590d681cf70 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f273918e740 1 --2- 172.21.15.98:0/84990306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d6814d70 0x5590d681ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f273918e740 1 --2- 172.21.15.98:0/84990306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814450 0x5590d681c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f270f7fe640 0 client.17742 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f273918e740 1 -- 172.21.15.98:0/84990306 >> 172.21.15.98:0/84990306 conn(0x5590d6810960 msgr2=0x5590d68166e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f273918e740 1 -- 172.21.15.98:0/84990306 shutdown_connections 2024-05-09T08:58:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f273918e740 1 -- 172.21.15.98:0/84990306 wait complete. 2024-05-09T08:58:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.714+0000 7f273918e740 20 client.17742 trim_cache size 0 max 16384 2024-05-09T08:58:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f273918e740 1 Processor -- start 2024-05-09T08:58:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f273918e740 1 -- start start 2024-05-09T08:58:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f273918e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d6814450 0x5590d682b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f273918e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814d70 0x5590d6816e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f273918e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d681f630 0x5590d6810d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f273918e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f27301b0fc0 con 0x5590d681f630 2024-05-09T08:58:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f273918e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f27301b2b00 con 0x5590d6814d70 2024-05-09T08:58:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f273918e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f27301b1690 con 0x5590d6814450 2024-05-09T08:58:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f2734942640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d681f630 0x5590d6810d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f2734942640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d681f630 0x5590d6810d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53610/0 (socket says 172.21.15.98:53610) 2024-05-09T08:58:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f2734942640 1 -- 172.21.15.98:0/80259679 learned_addr learned my addr 172.21.15.98:0/80259679 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f2734942640 1 -- 172.21.15.98:0/80259679 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d6814450 msgr2=0x5590d682b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f2734942640 1 --2- 172.21.15.98:0/80259679 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d6814450 0x5590d682b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f2734942640 1 -- 172.21.15.98:0/80259679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814d70 msgr2=0x5590d6816e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f2734942640 1 --2- 172.21.15.98:0/80259679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814d70 0x5590d6816e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f2734942640 1 -- 172.21.15.98:0/80259679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590d681b4a0 con 0x5590d681f630 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.729+0000 7f2734942640 1 --2- 172.21.15.98:0/80259679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d681f630 0x5590d6810d40 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f2728013970 tx=0x7f27280094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce84a4cb92d4dc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.730+0000 7f2725ffb640 1 -- 172.21.15.98:0/80259679 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272803a020 con 0x5590d681f630 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.730+0000 7f2725ffb640 1 -- 172.21.15.98:0/80259679 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2728003d40 con 0x5590d681f630 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.730+0000 7f2725ffb640 1 -- 172.21.15.98:0/80259679 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2728004360 con 0x5590d681f630 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.730+0000 7f273918e740 1 -- 172.21.15.98:0/80259679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d681f630 msgr2=0x5590d6810d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.730+0000 7f273918e740 1 --2- 172.21.15.98:0/80259679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d681f630 0x5590d6810d40 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f2728013970 tx=0x7f27280094c0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.730+0000 7f273918e740 1 -- 172.21.15.98:0/80259679 shutdown_connections 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.730+0000 7f273918e740 1 --2- 172.21.15.98:0/80259679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d681f630 0x5590d6810d40 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.730+0000 7f273918e740 1 --2- 172.21.15.98:0/80259679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d6814d70 0x5590d6816e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.730+0000 7f273918e740 1 --2- 172.21.15.98:0/80259679 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d6814450 0x5590d682b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.730+0000 7f273918e740 1 -- 172.21.15.98:0/80259679 >> 172.21.15.98:0/80259679 conn(0x5590d6810960 msgr2=0x5590d680fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.730+0000 7f273918e740 1 -- 172.21.15.98:0/80259679 shutdown_connections 2024-05-09T08:58:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.730+0000 7f273918e740 1 -- 172.21.15.98:0/80259679 wait complete. 2024-05-09T08:58:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.730+0000 7f273918e740 1 -- start start 2024-05-09T08:58:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f273918e740 1 Processor -- start 2024-05-09T08:58:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f273918e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d6814450 0x5590d68194c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f273918e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d6814d70 0x5590d6819a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f273918e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d681f630 0x5590d6819f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f273918e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f27301b2430 con 0x5590d6814450 2024-05-09T08:58:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f273918e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f27301b1d60 con 0x5590d681f630 2024-05-09T08:58:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f273918e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f27301aefc0 con 0x5590d6814d70 2024-05-09T08:58:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f273918e740 10 client.0 _ll_register_callbacks cb 0x7f2728009690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f272ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d6814450 0x5590d68194c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f272ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d6814450 0x5590d68194c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53618/0 (socket says 172.21.15.98:53618) 2024-05-09T08:58:35.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f272ffff640 1 -- 172.21.15.98:0/1420586975 learned_addr learned my addr 172.21.15.98:0/1420586975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f2734942640 1 --2- 172.21.15.98:0/1420586975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d681f630 0x5590d6819f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f272f7fe640 1 --2- 172.21.15.98:0/1420586975 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d6814d70 0x5590d6819a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f272ffff640 1 -- 172.21.15.98:0/1420586975 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d6814d70 msgr2=0x5590d6819a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f272ffff640 1 --2- 172.21.15.98:0/1420586975 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d6814d70 0x5590d6819a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f272ffff640 1 -- 172.21.15.98:0/1420586975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d681f630 msgr2=0x5590d6819f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f272ffff640 1 --2- 172.21.15.98:0/1420586975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d681f630 0x5590d6819f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.731+0000 7f272ffff640 1 -- 172.21.15.98:0/1420586975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590d6814ba0 con 0x5590d6814450 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f272ffff640 1 --2- 172.21.15.98:0/1420586975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d6814450 0x5590d68194c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f2720009900 tx=0x7f2720005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61634333721631e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f270f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f270f7fe640 1 -- 172.21.15.98:0/1420586975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2720052020 con 0x5590d6814450 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f270f7fe640 1 -- 172.21.15.98:0/1420586975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f27200317b0 con 0x5590d6814450 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f273918e740 1 -- 172.21.15.98:0/1420586975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2728003d40 con 0x5590d6814450 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f270f7fe640 1 -- 172.21.15.98:0/1420586975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f272005d050 con 0x5590d6814450 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f273918e740 10 client.17748 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f273918e740 1 -- 172.21.15.98:0/1420586975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5590d6895c80 con 0x5590d6814450 2024-05-09T08:58:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f270f7fe640 1 -- 172.21.15.98:0/1420586975 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2728003d40 con 0x5590d6814450 2024-05-09T08:58:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f270f7fe640 1 -- 172.21.15.98:0/1420586975 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f2720004750 con 0x5590d6814450 2024-05-09T08:58:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f273918e740 10 client.17748 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f273918e740 10 client.17748 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f273918e740 20 client.17748 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f273918e740 1 --2- 172.21.15.98:0/1420586975 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5590d6895fc0 0x5590d68983b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f273918e740 4 client.17748 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:58:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f273918e740 1 -- 172.21.15.98:0/1420586975 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5590d6780dc0 con 0x5590d6895fc0 2024-05-09T08:58:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.732+0000 7f272f7fe640 1 --2- 172.21.15.98:0/1420586975 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5590d6895fc0 0x5590d68983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.733+0000 7f272f7fe640 1 --2- 172.21.15.98:0/1420586975 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5590d6895fc0 0x5590d68983b0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=466e5496d12e844 server_cookie=253ebb14ddde3aed in_seq=0 out_seq=0 2024-05-09T08:58:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.733+0000 7f270f7fe640 10 client.17748 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:35.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f270f7fe640 1 -- 172.21.15.98:0/1420586975 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5590d6780dc0 con 0x5590d6895fc0 2024-05-09T08:58:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f270f7fe640 10 client.17748 handle_command_reply: tid=0 2024-05-09T08:58:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f273918e740 1 client.17748 shutdown 2024-05-09T08:58:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f273918e740 1 -- 172.21.15.98:0/1420586975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d6814450 msgr2=0x5590d68194c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f273918e740 1 --2- 172.21.15.98:0/1420586975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d6814450 0x5590d68194c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f2720009900 tx=0x7f2720005a20 comp rx=0 tx=0).stop 2024-05-09T08:58:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f273918e740 1 -- 172.21.15.98:0/1420586975 shutdown_connections 2024-05-09T08:58:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f273918e740 1 --2- 172.21.15.98:0/1420586975 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5590d6895fc0 0x5590d68983b0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f273918e740 1 --2- 172.21.15.98:0/1420586975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d681f630 0x5590d6819f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f273918e740 1 --2- 172.21.15.98:0/1420586975 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5590d6814d70 0x5590d6819a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f273918e740 1 --2- 172.21.15.98:0/1420586975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590d6814450 0x5590d68194c0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f270f7fe640 0 client.17748 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f273918e740 1 -- 172.21.15.98:0/1420586975 >> 172.21.15.98:0/1420586975 conn(0x5590d6810960 msgr2=0x5590d680fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f273918e740 1 -- 172.21.15.98:0/1420586975 shutdown_connections 2024-05-09T08:58:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f273918e740 1 -- 172.21.15.98:0/1420586975 wait complete. 2024-05-09T08:58:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.741+0000 7f273918e740 20 client.17748 trim_cache size 0 max 16384 2024-05-09T08:58:35.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:58:35.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:58:35.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:58:35.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:58:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.742+0000 7f27363a5640 1 -- 172.21.15.98:0/741985626 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f271c0a2da0 msgr2=0x7f271c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.742+0000 7f27363a5640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f271c0a2da0 0x7f271c0a5260 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f273006b140 tx=0x7f2728009cf0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.742+0000 7f27363a5640 1 -- 172.21.15.98:0/741985626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2730103400 msgr2=0x7f2730100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.742+0000 7f27363a5640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2730103400 0x7f2730100c70 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f271800d950 tx=0x7f271800de10 comp rx=0 tx=0).stop 2024-05-09T08:58:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.742+0000 7f27363a5640 1 -- 172.21.15.98:0/741985626 shutdown_connections 2024-05-09T08:58:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.742+0000 7f27363a5640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f271c0a2da0 0x7f271c0a5260 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.742+0000 7f27363a5640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f273010beb0 0x7f27301011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.742+0000 7f27363a5640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2730103400 0x7f2730100c70 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.742+0000 7f27363a5640 1 --2- 172.21.15.98:0/741985626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2730102ae0 0x7f2730100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.742+0000 7f27363a5640 1 -- 172.21.15.98:0/741985626 >> 172.21.15.98:0/741985626 conn(0x7f2730076ab0 msgr2=0x7f27300fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.742+0000 7f27363a5640 1 -- 172.21.15.98:0/741985626 shutdown_connections 2024-05-09T08:58:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:35.742+0000 7f27363a5640 1 -- 172.21.15.98:0/741985626 wait complete. 2024-05-09T08:58:36.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:35 smithi105 ceph-mon[34498]: pgmap v247: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.4 KiB/s wr, 11 op/s 2024-05-09T08:58:36.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:35 smithi098 ceph-mon[30092]: pgmap v247: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.4 KiB/s wr, 11 op/s 2024-05-09T08:58:36.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:35 smithi142 ceph-mon[34625]: pgmap v247: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.4 KiB/s wr, 11 op/s 2024-05-09T08:58:38.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:37 smithi105 ceph-mon[34498]: pgmap v248: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.2 KiB/s wr, 13 op/s 2024-05-09T08:58:38.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:37 smithi098 ceph-mon[30092]: pgmap v248: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.2 KiB/s wr, 13 op/s 2024-05-09T08:58:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:37 smithi142 ceph-mon[34625]: pgmap v248: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.2 KiB/s wr, 13 op/s 2024-05-09T08:58:40.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:39 smithi105 ceph-mon[34498]: pgmap v249: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-09T08:58:40.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:39 smithi098 ceph-mon[30092]: pgmap v249: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-09T08:58:40.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:39 smithi142 ceph-mon[34625]: pgmap v249: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-09T08:58:42.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:41 smithi105 ceph-mon[34498]: pgmap v250: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.6 KiB/s wr, 13 op/s 2024-05-09T08:58:42.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:41 smithi098 ceph-mon[30092]: pgmap v250: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.6 KiB/s wr, 13 op/s 2024-05-09T08:58:42.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:41 smithi142 ceph-mon[34625]: pgmap v250: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.6 KiB/s wr, 13 op/s 2024-05-09T08:58:44.102 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:58:44.103 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:58:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.220+0000 7fd799eee640 1 -- 172.21.15.98:0/1252712758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd794100700 msgr2=0x7fd794100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.220+0000 7fd799eee640 1 --2- 172.21.15.98:0/1252712758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd794100700 0x7fd794100b80 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fd77c009930 tx=0x7fd77c02a420 comp rx=0 tx=0).stop 2024-05-09T08:58:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.220+0000 7fd799eee640 1 -- 172.21.15.98:0/1252712758 shutdown_connections 2024-05-09T08:58:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.220+0000 7fd799eee640 1 --2- 172.21.15.98:0/1252712758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7941010c0 0x7fd79410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.220+0000 7fd799eee640 1 --2- 172.21.15.98:0/1252712758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd794100700 0x7fd794100b80 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.220+0000 7fd799eee640 1 --2- 172.21.15.98:0/1252712758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd794106710 0x7fd794106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.220+0000 7fd799eee640 1 -- 172.21.15.98:0/1252712758 >> 172.21.15.98:0/1252712758 conn(0x7fd794077830 msgr2=0x7fd794077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.220+0000 7fd799eee640 1 -- 172.21.15.98:0/1252712758 shutdown_connections 2024-05-09T08:58:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.220+0000 7fd799eee640 1 -- 172.21.15.98:0/1252712758 wait complete. 2024-05-09T08:58:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd799eee640 1 Processor -- start 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd799eee640 1 -- start start 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd799eee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd794100700 0x7fd794198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd799eee640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7941010c0 0x7fd794198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd799eee640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd794106710 0x7fd79419c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd799eee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd794111c20 con 0x7fd794100700 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd799eee640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd794111aa0 con 0x7fd7941010c0 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd799eee640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd794111da0 con 0x7fd794106710 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd7937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd794100700 0x7fd794198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd7937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd794100700 0x7fd794198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53638/0 (socket says 172.21.15.98:53638) 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd7937fe640 1 -- 172.21.15.98:0/3897167736 learned_addr learned my addr 172.21.15.98:0/3897167736 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd793fff640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd794106710 0x7fd79419c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd78bfff640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7941010c0 0x7fd794198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd7937fe640 1 -- 172.21.15.98:0/3897167736 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd794106710 msgr2=0x7fd79419c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd7937fe640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd794106710 0x7fd79419c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd7937fe640 1 -- 172.21.15.98:0/3897167736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7941010c0 msgr2=0x7fd794198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd7937fe640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7941010c0 0x7fd794198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd7937fe640 1 -- 172.21.15.98:0/3897167736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79419d110 con 0x7fd794100700 2024-05-09T08:58:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd78bfff640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7941010c0 0x7fd794198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.221+0000 7fd7937fe640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd794100700 0x7fd794198120 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fd78400c890 tx=0x7fd78400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.222+0000 7fd7917fa640 1 -- 172.21.15.98:0/3897167736 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784004220 con 0x7fd794100700 2024-05-09T08:58:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.222+0000 7fd799eee640 1 -- 172.21.15.98:0/3897167736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd79419d400 con 0x7fd794100700 2024-05-09T08:58:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.222+0000 7fd7917fa640 1 -- 172.21.15.98:0/3897167736 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7840043c0 con 0x7fd794100700 2024-05-09T08:58:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.222+0000 7fd7917fa640 1 -- 172.21.15.98:0/3897167736 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784004a00 con 0x7fd794100700 2024-05-09T08:58:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.222+0000 7fd799eee640 1 -- 172.21.15.98:0/3897167736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7941a4c90 con 0x7fd794100700 2024-05-09T08:58:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.224+0000 7fd7917fa640 1 -- 172.21.15.98:0/3897167736 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd7840079b0 con 0x7fd794100700 2024-05-09T08:58:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.224+0000 7fd79cbf7740 1 Processor -- start 2024-05-09T08:58:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.224+0000 7fd79cbf7740 1 -- start start 2024-05-09T08:58:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.224+0000 7fd79cbf7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c10862280 0x559c10862660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.224+0000 7fd79cbf7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862ba0 0x559c1086cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.224+0000 7fd79cbf7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c1086d270 0x559c1086f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.224+0000 7fd79cbf7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559c107c7d00 con 0x559c10862280 2024-05-09T08:58:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.224+0000 7fd79cbf7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559c1076f4c0 con 0x559c1086d270 2024-05-09T08:58:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.224+0000 7fd79cbf7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559c107c3cc0 con 0x559c10862ba0 2024-05-09T08:58:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd793fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c10862280 0x559c10862660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd7917fa640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd7800a2d80 0x7fd7800a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd793fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c10862280 0x559c10862660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53654/0 (socket says 172.21.15.98:53654) 2024-05-09T08:58:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd793fff640 1 -- 172.21.15.98:0/1026845647 learned_addr learned my addr 172.21.15.98:0/1026845647 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd793fff640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd7800a2d80 0x7fd7800a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd78bfff640 1 --2- 172.21.15.98:0/1026845647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c1086d270 0x559c1086f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd7917fa640 1 -- 172.21.15.98:0/3897167736 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fd7840cf600 con 0x7fd794100700 2024-05-09T08:58:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd793fff640 1 -- 172.21.15.98:0/1026845647 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862ba0 msgr2=0x559c1086cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd7937fe640 1 --2- 172.21.15.98:0/1026845647 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862ba0 0x559c1086cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd793fff640 1 --2- 172.21.15.98:0/1026845647 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862ba0 0x559c1086cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd793fff640 1 -- 172.21.15.98:0/1026845647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c1086d270 msgr2=0x559c1086f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd793fff640 1 --2- 172.21.15.98:0/1026845647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c1086d270 0x559c1086f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd793fff640 1 -- 172.21.15.98:0/1026845647 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c105a57d0 con 0x559c10862280 2024-05-09T08:58:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd78bfff640 1 --2- 172.21.15.98:0/1026845647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c1086d270 0x559c1086f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd7937fe640 1 --2- 172.21.15.98:0/1026845647 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862ba0 0x559c1086cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd793fff640 1 --2- 172.21.15.98:0/1026845647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c10862280 0x559c10862660 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd78c012890 tx=0x7fd78c012d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512ec2b3e527bde7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd789ffb640 1 -- 172.21.15.98:0/1026845647 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd78c003e00 con 0x559c10862280 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd789ffb640 1 -- 172.21.15.98:0/1026845647 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd78c004b60 con 0x559c10862280 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd789ffb640 1 -- 172.21.15.98:0/1026845647 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0052b0 con 0x559c10862280 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.225+0000 7fd793fff640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd7800a2d80 0x7fd7800a5240 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd78c01f000 tx=0x7fd78c01e000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.226+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1026845647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c10862280 msgr2=0x559c10862660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.226+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/1026845647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c10862280 0x559c10862660 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd78c012890 tx=0x7fd78c012d50 comp rx=0 tx=0).stop 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.226+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1026845647 shutdown_connections 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.226+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/1026845647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c1086d270 0x559c1086f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.226+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/1026845647 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862ba0 0x559c1086cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.226+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/1026845647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c10862280 0x559c10862660 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.226+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1026845647 >> 172.21.15.98:0/1026845647 conn(0x559c1085e5a0 msgr2=0x559c1085e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.226+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1026845647 shutdown_connections 2024-05-09T08:58:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.226+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1026845647 wait complete. 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.226+0000 7fd79cbf7740 1 -- start start 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd79cbf7740 1 Processor -- start 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd79cbf7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862280 0x559c1086a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd79cbf7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c10862ba0 0x559c1086a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd79cbf7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c1086d270 0x559c1086ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd79cbf7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559c10870060 con 0x559c10862ba0 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd79cbf7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559c10782b90 con 0x559c1086d270 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd79cbf7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559c108701e0 con 0x559c10862280 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd79cbf7740 10 client.0 _ll_register_callbacks cb 0x559c107beda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd78bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c10862ba0 0x559c1086a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd78bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c10862ba0 0x559c1086a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53664/0 (socket says 172.21.15.98:53664) 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd78bfff640 1 -- 172.21.15.98:0/2411910117 learned_addr learned my addr 172.21.15.98:0/2411910117 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd7937fe640 1 --2- 172.21.15.98:0/2411910117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862280 0x559c1086a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd793fff640 1 --2- 172.21.15.98:0/2411910117 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c1086d270 0x559c1086ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd78bfff640 1 -- 172.21.15.98:0/2411910117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862280 msgr2=0x559c1086a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd78bfff640 1 --2- 172.21.15.98:0/2411910117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862280 0x559c1086a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd78bfff640 1 -- 172.21.15.98:0/2411910117 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c1086d270 msgr2=0x559c1086ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd78bfff640 1 --2- 172.21.15.98:0/2411910117 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c1086d270 0x559c1086ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd78bfff640 1 -- 172.21.15.98:0/2411910117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c104db3f0 con 0x559c10862ba0 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd7937fe640 1 --2- 172.21.15.98:0/2411910117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862280 0x559c1086a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd78bfff640 1 --2- 172.21.15.98:0/2411910117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c10862ba0 0x559c1086a6f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fd77c005ab0 tx=0x7fd77c0315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f4176e668c5aa06 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.227+0000 7fd772ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.228+0000 7fd772ffd640 1 -- 172.21.15.98:0/2411910117 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c03f070 con 0x559c10862ba0 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.228+0000 7fd772ffd640 1 -- 172.21.15.98:0/2411910117 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fd77c004040 con 0x559c10862ba0 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.228+0000 7fd79cbf7740 1 -- 172.21.15.98:0/2411910117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd78c004b60 con 0x559c10862ba0 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.228+0000 7fd772ffd640 1 -- 172.21.15.98:0/2411910117 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd77c04b050 con 0x559c10862ba0 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.228+0000 7fd79cbf7740 10 client.17760 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.228+0000 7fd79cbf7740 1 -- 172.21.15.98:0/2411910117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559c1049b0f0 con 0x559c10862ba0 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.228+0000 7fd772ffd640 1 -- 172.21.15.98:0/2411910117 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd78c004b60 con 0x559c10862ba0 2024-05-09T08:58:44.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.228+0000 7fd772ffd640 1 -- 172.21.15.98:0/2411910117 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd77c0049c0 con 0x559c10862ba0 2024-05-09T08:58:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.228+0000 7fd79cbf7740 10 client.17760 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.228+0000 7fd79cbf7740 10 client.17760 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.228+0000 7fd79cbf7740 20 client.17760 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.228+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/2411910117 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c1087d400 0x559c1087f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.228+0000 7fd79cbf7740 4 client.17760 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.228+0000 7fd79cbf7740 1 -- 172.21.15.98:0/2411910117 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c107d5430 con 0x559c1087d400 2024-05-09T08:58:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.229+0000 7fd7937fe640 1 --2- 172.21.15.98:0/2411910117 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c1087d400 0x559c1087f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.229+0000 7fd7937fe640 1 --2- 172.21.15.98:0/2411910117 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c1087d400 0x559c1087f8a0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50dee0936d59beb0 server_cookie=2062058385288707 in_seq=0 out_seq=0 2024-05-09T08:58:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.229+0000 7fd772ffd640 10 client.17760 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd772ffd640 1 -- 172.21.15.98:0/2411910117 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559c107d5430 con 0x559c1087d400 2024-05-09T08:58:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd772ffd640 10 client.17760 handle_command_reply: tid=0 2024-05-09T08:58:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd79cbf7740 1 client.17760 shutdown 2024-05-09T08:58:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd79cbf7740 1 -- 172.21.15.98:0/2411910117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c10862ba0 msgr2=0x559c1086a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/2411910117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c10862ba0 0x559c1086a6f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fd77c005ab0 tx=0x7fd77c0315b0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd79cbf7740 1 -- 172.21.15.98:0/2411910117 shutdown_connections 2024-05-09T08:58:44.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/2411910117 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c1087d400 0x559c1087f8a0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/2411910117 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c1086d270 0x559c1086ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/2411910117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c10862ba0 0x559c1086a6f0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd772ffd640 0 client.17760 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:44.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/2411910117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862280 0x559c1086a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd79cbf7740 1 -- 172.21.15.98:0/2411910117 >> 172.21.15.98:0/2411910117 conn(0x559c1085e5a0 msgr2=0x559c108642a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd79cbf7740 1 -- 172.21.15.98:0/2411910117 shutdown_connections 2024-05-09T08:58:44.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd79cbf7740 1 -- 172.21.15.98:0/2411910117 wait complete. 2024-05-09T08:58:44.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.230+0000 7fd79cbf7740 20 client.17760 trim_cache size 0 max 16384 2024-05-09T08:58:44.237 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:43 smithi105 ceph-mon[34498]: pgmap v251: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-05-09T08:58:44.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.247+0000 7fd79cbf7740 1 Processor -- start 2024-05-09T08:58:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.247+0000 7fd79cbf7740 1 -- start start 2024-05-09T08:58:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd79cbf7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c10862280 0x559c1086c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd79cbf7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862ba0 0x559c10878f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd79cbf7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c1086d270 0x559c10875840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd79cbf7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd7941aefd0 con 0x559c1086d270 2024-05-09T08:58:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd79cbf7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd7941b0b10 con 0x559c10862280 2024-05-09T08:58:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd79cbf7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd7941af6a0 con 0x559c10862ba0 2024-05-09T08:58:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd793fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c1086d270 0x559c10875840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd793fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c1086d270 0x559c10875840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53676/0 (socket says 172.21.15.98:53676) 2024-05-09T08:58:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd793fff640 1 -- 172.21.15.98:0/1084891706 learned_addr learned my addr 172.21.15.98:0/1084891706 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd793fff640 1 -- 172.21.15.98:0/1084891706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862ba0 msgr2=0x559c10878f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd793fff640 1 --2- 172.21.15.98:0/1084891706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862ba0 0x559c10878f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd793fff640 1 -- 172.21.15.98:0/1084891706 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c10862280 msgr2=0x559c1086c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd793fff640 1 --2- 172.21.15.98:0/1084891706 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c10862280 0x559c1086c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd793fff640 1 -- 172.21.15.98:0/1084891706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c108629d0 con 0x559c1086d270 2024-05-09T08:58:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd793fff640 1 --2- 172.21.15.98:0/1084891706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c1086d270 0x559c10875840 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd78c010ae0 tx=0x7fd78c008610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39284563a126f8d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd7897fa640 1 -- 172.21.15.98:0/1084891706 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd78c037070 con 0x559c1086d270 2024-05-09T08:58:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd7897fa640 1 -- 172.21.15.98:0/1084891706 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd78c012480 con 0x559c1086d270 2024-05-09T08:58:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.248+0000 7fd7897fa640 1 -- 172.21.15.98:0/1084891706 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0055d0 con 0x559c1086d270 2024-05-09T08:58:44.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.249+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1084891706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c1086d270 msgr2=0x559c10875840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.249+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/1084891706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c1086d270 0x559c10875840 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd78c010ae0 tx=0x7fd78c008610 comp rx=0 tx=0).stop 2024-05-09T08:58:44.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.249+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1084891706 shutdown_connections 2024-05-09T08:58:44.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.249+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/1084891706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c1086d270 0x559c10875840 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.249+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/1084891706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862ba0 0x559c10878f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.249+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/1084891706 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c10862280 0x559c1086c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.249+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1084891706 >> 172.21.15.98:0/1084891706 conn(0x559c1085e5a0 msgr2=0x559c1086a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.249+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1084891706 shutdown_connections 2024-05-09T08:58:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.249+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1084891706 wait complete. 2024-05-09T08:58:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.249+0000 7fd79cbf7740 1 -- start start 2024-05-09T08:58:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.249+0000 7fd79cbf7740 1 Processor -- start 2024-05-09T08:58:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd79cbf7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c10862280 0x559c10866420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd79cbf7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862ba0 0x559c10866960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd79cbf7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c1086d270 0x559c10866ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd79cbf7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd7941b0440 con 0x559c1086d270 2024-05-09T08:58:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd79cbf7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd7941afd70 con 0x559c10862280 2024-05-09T08:58:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd79cbf7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd7941acfd0 con 0x559c10862ba0 2024-05-09T08:58:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd79cbf7740 10 client.0 _ll_register_callbacks cb 0x7fd77c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd793fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c1086d270 0x559c10866ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd793fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c1086d270 0x559c10866ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53680/0 (socket says 172.21.15.98:53680) 2024-05-09T08:58:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd793fff640 1 -- 172.21.15.98:0/1127895628 learned_addr learned my addr 172.21.15.98:0/1127895628 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd7937fe640 1 --2- 172.21.15.98:0/1127895628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c10862280 0x559c10866420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd793fff640 1 -- 172.21.15.98:0/1127895628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862ba0 msgr2=0x559c10866960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd793fff640 1 --2- 172.21.15.98:0/1127895628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862ba0 0x559c10866960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd793fff640 1 -- 172.21.15.98:0/1127895628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c10862280 msgr2=0x559c10866420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd793fff640 1 --2- 172.21.15.98:0/1127895628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c10862280 0x559c10866420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd793fff640 1 -- 172.21.15.98:0/1127895628 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c108676a0 con 0x559c1086d270 2024-05-09T08:58:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd7937fe640 1 --2- 172.21.15.98:0/1127895628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c10862280 0x559c10866420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd793fff640 1 --2- 172.21.15.98:0/1127895628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c1086d270 0x559c10866ea0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd78c010740 tx=0x7fd78c0050d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7017a9ea8384ff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd772ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd772ffd640 1 -- 172.21.15.98:0/1127895628 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd78c037070 con 0x559c1086d270 2024-05-09T08:58:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd772ffd640 1 -- 172.21.15.98:0/1127895628 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fd78c004c10 con 0x559c1086d270 2024-05-09T08:58:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1127895628 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd78c012480 con 0x559c1086d270 2024-05-09T08:58:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd772ffd640 1 -- 172.21.15.98:0/1127895628 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd78c004e10 con 0x559c1086d270 2024-05-09T08:58:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd79cbf7740 10 client.17766 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.250+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1127895628 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559c108e7eb0 con 0x559c1086d270 2024-05-09T08:58:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.251+0000 7fd772ffd640 1 -- 172.21.15.98:0/1127895628 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd78c012480 con 0x559c1086d270 2024-05-09T08:58:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.251+0000 7fd772ffd640 1 -- 172.21.15.98:0/1127895628 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd78c032440 con 0x559c1086d270 2024-05-09T08:58:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.251+0000 7fd79cbf7740 10 client.17766 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.251+0000 7fd79cbf7740 10 client.17766 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.251+0000 7fd79cbf7740 20 client.17766 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.251+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/1127895628 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c108e83b0 0x559c108ea7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.251+0000 7fd79cbf7740 4 client.17766 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:58:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.251+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1127895628 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559c107cf780 con 0x559c108e83b0 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.251+0000 7fd7937fe640 1 --2- 172.21.15.98:0/1127895628 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c108e83b0 0x559c108ea7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd772ffd640 10 client.17766 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd7937fe640 1 --2- 172.21.15.98:0/1127895628 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c108e83b0 0x559c108ea7d0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23ca85027a9b623c server_cookie=88fb2f4e21d2ee99 in_seq=0 out_seq=0 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd772ffd640 1 -- 172.21.15.98:0/1127895628 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559c107cf780 con 0x559c108e83b0 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd772ffd640 10 client.17766 handle_command_reply: tid=0 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd79cbf7740 1 client.17766 shutdown 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1127895628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c1086d270 msgr2=0x559c10866ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/1127895628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c1086d270 0x559c10866ea0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd78c010740 tx=0x7fd78c0050d0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1127895628 shutdown_connections 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/1127895628 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c108e83b0 0x559c108ea7d0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/1127895628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c1086d270 0x559c10866ea0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/1127895628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c10862ba0 0x559c10866960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd79cbf7740 1 --2- 172.21.15.98:0/1127895628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c10862280 0x559c10866420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1127895628 >> 172.21.15.98:0/1127895628 conn(0x559c1085e5a0 msgr2=0x559c10864260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd772ffd640 0 client.17766 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:44.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1127895628 shutdown_connections 2024-05-09T08:58:44.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd79cbf7740 1 -- 172.21.15.98:0/1127895628 wait complete. 2024-05-09T08:58:44.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.252+0000 7fd79cbf7740 20 client.17766 trim_cache size 0 max 16384 2024-05-09T08:58:44.260 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:58:44.261 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:58:44.261 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:58:44.261 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:58:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.253+0000 7fd799eee640 1 -- 172.21.15.98:0/3897167736 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd7800a2d80 msgr2=0x7fd7800a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.253+0000 7fd799eee640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd7800a2d80 0x7fd7800a5240 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd78c01f000 tx=0x7fd78c01e000 comp rx=0 tx=0).stop 2024-05-09T08:58:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.253+0000 7fd799eee640 1 -- 172.21.15.98:0/3897167736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd794100700 msgr2=0x7fd794198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.253+0000 7fd799eee640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd794100700 0x7fd794198120 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fd78400c890 tx=0x7fd78400cd50 comp rx=0 tx=0).stop 2024-05-09T08:58:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.253+0000 7fd799eee640 1 -- 172.21.15.98:0/3897167736 shutdown_connections 2024-05-09T08:58:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.253+0000 7fd799eee640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd7800a2d80 0x7fd7800a5240 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.253+0000 7fd799eee640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd794106710 0x7fd79419c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.253+0000 7fd799eee640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7941010c0 0x7fd794198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.253+0000 7fd799eee640 1 --2- 172.21.15.98:0/3897167736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd794100700 0x7fd794198120 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.253+0000 7fd799eee640 1 -- 172.21.15.98:0/3897167736 >> 172.21.15.98:0/3897167736 conn(0x7fd794077830 msgr2=0x7fd794102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.253+0000 7fd799eee640 1 -- 172.21.15.98:0/3897167736 shutdown_connections 2024-05-09T08:58:44.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.253+0000 7fd799eee640 1 -- 172.21.15.98:0/3897167736 wait complete. 2024-05-09T08:58:44.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:43 smithi098 ceph-mon[30092]: pgmap v251: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-05-09T08:58:44.264 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:58:44.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:58:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.366+0000 7f25a544c640 1 -- 172.21.15.98:0/1881957010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a01033e0 msgr2=0x7f25a010b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.366+0000 7f25a544c640 1 --2- 172.21.15.98:0/1881957010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a01033e0 0x7f25a010b8c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f2588009900 tx=0x7f258802a510 comp rx=0 tx=0).stop 2024-05-09T08:58:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.366+0000 7f25a544c640 1 -- 172.21.15.98:0/1881957010 shutdown_connections 2024-05-09T08:58:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.366+0000 7f25a544c640 1 --2- 172.21.15.98:0/1881957010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a010be00 0x7f25a010e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.366+0000 7f25a544c640 1 --2- 172.21.15.98:0/1881957010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a01033e0 0x7f25a010b8c0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.366+0000 7f25a544c640 1 --2- 172.21.15.98:0/1881957010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25a0102ac0 0x7f25a0102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.366+0000 7f25a544c640 1 -- 172.21.15.98:0/1881957010 >> 172.21.15.98:0/1881957010 conn(0x7f25a0077830 msgr2=0x7f25a0077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.366+0000 7f25a544c640 1 -- 172.21.15.98:0/1881957010 shutdown_connections 2024-05-09T08:58:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.366+0000 7f25a544c640 1 -- 172.21.15.98:0/1881957010 wait complete. 2024-05-09T08:58:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.367+0000 7f25a544c640 1 Processor -- start 2024-05-09T08:58:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.367+0000 7f25a544c640 1 -- start start 2024-05-09T08:58:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.367+0000 7f25a544c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a0102ac0 0x7f25a0198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.367+0000 7f25a544c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a01033e0 0x7f25a01986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.367+0000 7f25a544c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25a010be00 0x7f25a019c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.367+0000 7f25a544c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25a0111c00 con 0x7f25a01033e0 2024-05-09T08:58:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.367+0000 7f25a544c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f25a0111a80 con 0x7f25a0102ac0 2024-05-09T08:58:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.367+0000 7f25a544c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f25a0111d80 con 0x7f25a010be00 2024-05-09T08:58:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.367+0000 7f25967fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a01033e0 0x7f25a01986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.367+0000 7f25967fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a01033e0 0x7f25a01986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53690/0 (socket says 172.21.15.98:53690) 2024-05-09T08:58:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.367+0000 7f25967fc640 1 -- 172.21.15.98:0/3905185594 learned_addr learned my addr 172.21.15.98:0/3905185594 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.367+0000 7f259effd640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a0102ac0 0x7f25a0198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.368+0000 7f259f7fe640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25a010be00 0x7f25a019c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.368+0000 7f25967fc640 1 -- 172.21.15.98:0/3905185594 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25a010be00 msgr2=0x7f25a019c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.368+0000 7f25967fc640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25a010be00 0x7f25a019c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.368+0000 7f25967fc640 1 -- 172.21.15.98:0/3905185594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a0102ac0 msgr2=0x7f25a0198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.368+0000 7f25967fc640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a0102ac0 0x7f25a0198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.368+0000 7f25967fc640 1 -- 172.21.15.98:0/3905185594 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a019d120 con 0x7f25a01033e0 2024-05-09T08:58:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.368+0000 7f259f7fe640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25a010be00 0x7f25a019c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.368+0000 7f25967fc640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a01033e0 0x7f25a01986c0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f2588006010 tx=0x7f25880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.368+0000 7f259cff9640 1 -- 172.21.15.98:0/3905185594 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258803f030 con 0x7f25a01033e0 2024-05-09T08:58:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.368+0000 7f259cff9640 1 -- 172.21.15.98:0/3905185594 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2588003ce0 con 0x7f25a01033e0 2024-05-09T08:58:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.368+0000 7f25a544c640 1 -- 172.21.15.98:0/3905185594 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a019d3b0 con 0x7f25a01033e0 2024-05-09T08:58:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.368+0000 7f259cff9640 1 -- 172.21.15.98:0/3905185594 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2588035070 con 0x7f25a01033e0 2024-05-09T08:58:44.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.368+0000 7f25a544c640 1 -- 172.21.15.98:0/3905185594 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a01a4ca0 con 0x7f25a01033e0 2024-05-09T08:58:44.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.369+0000 7f25a544c640 1 -- 172.21.15.98:0/3905185594 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a0068650 con 0x7f25a01033e0 2024-05-09T08:58:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.370+0000 7f259cff9640 1 -- 172.21.15.98:0/3905185594 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2588042030 con 0x7f25a01033e0 2024-05-09T08:58:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.371+0000 7f259cff9640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f258c0a2d00 0x7f258c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.371+0000 7f259cff9640 1 -- 172.21.15.98:0/3905185594 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f25880efd70 con 0x7f25a01033e0 2024-05-09T08:58:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.371+0000 7f259effd640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f258c0a2d00 0x7f258c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.371+0000 7f259effd640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f258c0a2d00 0x7f258c0a51c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2590005fd0 tx=0x7f259000b390 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.375+0000 7f259cff9640 1 -- 172.21.15.98:0/3905185594 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f25880bc640 con 0x7f25a01033e0 2024-05-09T08:58:44.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:43 smithi142 ceph-mon[34625]: pgmap v251: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-05-09T08:58:44.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.589+0000 7f25a544c640 1 -- 172.21.15.98:0/3905185594 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25a0106f80 con 0x7f25a01033e0 2024-05-09T08:58:44.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.590+0000 7f259cff9640 1 -- 172.21.15.98:0/3905185594 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11941 (secure 0 0 0) 0x7f25880bbf40 con 0x7f25a01033e0 2024-05-09T08:58:44.589 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:58:44.590 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":26581,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/561027115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":561027115},{"type":"v1","addr":"172.21.15.105:6835","nonce":561027115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:56:52.891273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16641":{"gid":16641,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T08:58:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 37 2024-05-09T08:58:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.592+0000 7f25a544c640 1 -- 172.21.15.98:0/3905185594 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f258c0a2d00 msgr2=0x7f258c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.592+0000 7f25a544c640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f258c0a2d00 0x7f258c0a51c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2590005fd0 tx=0x7f259000b390 comp rx=0 tx=0).stop 2024-05-09T08:58:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.592+0000 7f25a544c640 1 -- 172.21.15.98:0/3905185594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a01033e0 msgr2=0x7f25a01986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.592+0000 7f25a544c640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a01033e0 0x7f25a01986c0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f2588006010 tx=0x7f25880042f0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.592+0000 7f25a544c640 1 -- 172.21.15.98:0/3905185594 shutdown_connections 2024-05-09T08:58:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.592+0000 7f25a544c640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f258c0a2d00 0x7f258c0a51c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.592+0000 7f25a544c640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25a010be00 0x7f25a019c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.592+0000 7f25a544c640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a01033e0 0x7f25a01986c0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.592+0000 7f25a544c640 1 --2- 172.21.15.98:0/3905185594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a0102ac0 0x7f25a0198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.592+0000 7f25a544c640 1 -- 172.21.15.98:0/3905185594 >> 172.21.15.98:0/3905185594 conn(0x7f25a0077830 msgr2=0x7f25a00fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.592+0000 7f25a544c640 1 -- 172.21.15.98:0/3905185594 shutdown_connections 2024-05-09T08:58:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.593+0000 7f25a544c640 1 -- 172.21.15.98:0/3905185594 wait complete. 2024-05-09T08:58:44.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:58:44.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.707+0000 7fb91f3ed640 1 -- 172.21.15.98:0/3685688140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb91806a7f0 msgr2=0x7fb91810e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.707+0000 7fb91f3ed640 1 --2- 172.21.15.98:0/3685688140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb91806a7f0 0x7fb91810e6d0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fb910009cd0 tx=0x7fb91002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.707+0000 7fb91f3ed640 1 -- 172.21.15.98:0/3685688140 shutdown_connections 2024-05-09T08:58:44.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.707+0000 7fb91f3ed640 1 --2- 172.21.15.98:0/3685688140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb91806a7f0 0x7fb91810e6d0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.707+0000 7fb91f3ed640 1 --2- 172.21.15.98:0/3685688140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb918069e30 0x7fb91806a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.707+0000 7fb91f3ed640 1 --2- 172.21.15.98:0/3685688140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb918107810 0x7fb918107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.707+0000 7fb91f3ed640 1 -- 172.21.15.98:0/3685688140 >> 172.21.15.98:0/3685688140 conn(0x7fb918076530 msgr2=0x7fb918076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.707+0000 7fb91f3ed640 1 -- 172.21.15.98:0/3685688140 shutdown_connections 2024-05-09T08:58:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.707+0000 7fb91f3ed640 1 -- 172.21.15.98:0/3685688140 wait complete. 2024-05-09T08:58:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91f3ed640 1 Processor -- start 2024-05-09T08:58:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91f3ed640 1 -- start start 2024-05-09T08:58:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91f3ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb918069e30 0x7fb918198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91f3ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb91806a7f0 0x7fb9181985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91f3ed640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb918107810 0x7fb91819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91f3ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb918111c20 con 0x7fb918069e30 2024-05-09T08:58:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91f3ed640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb918111aa0 con 0x7fb91806a7f0 2024-05-09T08:58:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91f3ed640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb918111da0 con 0x7fb918107810 2024-05-09T08:58:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91d189640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb918069e30 0x7fb918198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91d189640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb918069e30 0x7fb918198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53702/0 (socket says 172.21.15.98:53702) 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91d189640 1 -- 172.21.15.98:0/1052847151 learned_addr learned my addr 172.21.15.98:0/1052847151 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91c988640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb91806a7f0 0x7fb9181985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91d98a640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb918107810 0x7fb91819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91d189640 1 -- 172.21.15.98:0/1052847151 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb918107810 msgr2=0x7fb91819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91d189640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb918107810 0x7fb91819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91d189640 1 -- 172.21.15.98:0/1052847151 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb91806a7f0 msgr2=0x7fb9181985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91d189640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb91806a7f0 0x7fb9181985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91d189640 1 -- 172.21.15.98:0/1052847151 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91819d020 con 0x7fb918069e30 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91c988640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb91806a7f0 0x7fb9181985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.708+0000 7fb91d98a640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb918107810 0x7fb91819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.709+0000 7fb91d189640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb918069e30 0x7fb918198080 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fb91400d770 tx=0x7fb91400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.709+0000 7fb90e7fc640 1 -- 172.21.15.98:0/1052847151 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb914017070 con 0x7fb918069e30 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.709+0000 7fb90e7fc640 1 -- 172.21.15.98:0/1052847151 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb914005500 con 0x7fb918069e30 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.709+0000 7fb91f3ed640 1 -- 172.21.15.98:0/1052847151 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb91819d310 con 0x7fb918069e30 2024-05-09T08:58:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.709+0000 7fb90e7fc640 1 -- 172.21.15.98:0/1052847151 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb914004e00 con 0x7fb918069e30 2024-05-09T08:58:44.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.709+0000 7fb91f3ed640 1 -- 172.21.15.98:0/1052847151 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9181a4ba0 con 0x7fb918069e30 2024-05-09T08:58:44.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.711+0000 7fb90e7fc640 1 -- 172.21.15.98:0/1052847151 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb914004fa0 con 0x7fb918069e30 2024-05-09T08:58:44.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.711+0000 7fb9221ce740 1 Processor -- start 2024-05-09T08:58:44.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.711+0000 7fb9221ce740 1 -- start start 2024-05-09T08:58:44.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb9221ce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013fe210 0x5576013fe5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb9221ce740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 0x55760140d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb9221ce740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 0x55760140fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb9221ce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557601363c40 con 0x5576013fe210 2024-05-09T08:58:44.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb9221ce740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55760130b580 con 0x5576013feb30 2024-05-09T08:58:44.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb9221ce740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55760135fc00 con 0x55760140d730 2024-05-09T08:58:44.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb90e7fc640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb9040a2d00 0x7fb9040a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb91d98a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013fe210 0x5576013fe5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb91d98a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013fe210 0x5576013fe5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53704/0 (socket says 172.21.15.98:53704) 2024-05-09T08:58:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb91d98a640 1 -- 172.21.15.98:0/2121947359 learned_addr learned my addr 172.21.15.98:0/2121947359 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb91d98a640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb9040a2d00 0x7fb9040a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb91c988640 1 --2- 172.21.15.98:0/2121947359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 0x55760140fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb90e7fc640 1 -- 172.21.15.98:0/1052847151 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fb9140cf2d0 con 0x7fb918069e30 2024-05-09T08:58:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb91d189640 1 --2- 172.21.15.98:0/2121947359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 0x55760140d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb91d98a640 1 -- 172.21.15.98:0/2121947359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 msgr2=0x55760140fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb91d98a640 1 --2- 172.21.15.98:0/2121947359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 0x55760140fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb91d98a640 1 -- 172.21.15.98:0/2121947359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 msgr2=0x55760140d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb91d98a640 1 --2- 172.21.15.98:0/2121947359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 0x55760140d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb91d98a640 1 -- 172.21.15.98:0/2121947359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576010773f0 con 0x5576013fe210 2024-05-09T08:58:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb91d189640 1 --2- 172.21.15.98:0/2121947359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 0x55760140d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb91d98a640 1 --2- 172.21.15.98:0/2121947359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013fe210 0x5576013fe5f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fb910009430 tx=0x7fb910031c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15c868cbd9c20f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb9037fe640 1 -- 172.21.15.98:0/2121947359 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91004f070 con 0x5576013fe210 2024-05-09T08:58:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb9037fe640 1 -- 172.21.15.98:0/2121947359 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb910036040 con 0x5576013fe210 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.712+0000 7fb9037fe640 1 -- 172.21.15.98:0/2121947359 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb910030960 con 0x5576013fe210 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.713+0000 7fb91d98a640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb9040a2d00 0x7fb9040a51c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb91004c000 tx=0x7fb910048000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.713+0000 7fb9221ce740 1 -- 172.21.15.98:0/2121947359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013fe210 msgr2=0x5576013fe5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.713+0000 7fb9221ce740 1 --2- 172.21.15.98:0/2121947359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013fe210 0x5576013fe5f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fb910009430 tx=0x7fb910031c00 comp rx=0 tx=0).stop 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.713+0000 7fb9221ce740 1 -- 172.21.15.98:0/2121947359 shutdown_connections 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.713+0000 7fb9221ce740 1 --2- 172.21.15.98:0/2121947359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 0x55760140fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.713+0000 7fb9221ce740 1 --2- 172.21.15.98:0/2121947359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 0x55760140d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.713+0000 7fb9221ce740 1 --2- 172.21.15.98:0/2121947359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013fe210 0x5576013fe5f0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.713+0000 7fb9221ce740 1 -- 172.21.15.98:0/2121947359 >> 172.21.15.98:0/2121947359 conn(0x5576013fa650 msgr2=0x5576013faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.713+0000 7fb9221ce740 1 -- 172.21.15.98:0/2121947359 shutdown_connections 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.713+0000 7fb9221ce740 1 -- 172.21.15.98:0/2121947359 wait complete. 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb9221ce740 1 -- start start 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb9221ce740 1 Processor -- start 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb9221ce740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5576013fe210 0x55760140a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb9221ce740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 0x55760140abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb9221ce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55760140d730 0x55760140b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb9221ce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557601410570 con 0x55760140d730 2024-05-09T08:58:44.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb9221ce740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55760131ec40 con 0x5576013feb30 2024-05-09T08:58:44.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb9221ce740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5576014106f0 con 0x5576013fe210 2024-05-09T08:58:44.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb9221ce740 10 client.0 _ll_register_callbacks cb 0x557601076be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:44.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb91d98a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55760140d730 0x55760140b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb91d98a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55760140d730 0x55760140b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53718/0 (socket says 172.21.15.98:53718) 2024-05-09T08:58:44.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb91d98a640 1 -- 172.21.15.98:0/2839062769 learned_addr learned my addr 172.21.15.98:0/2839062769 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb91d189640 1 --2- 172.21.15.98:0/2839062769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5576013fe210 0x55760140a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb91c988640 1 --2- 172.21.15.98:0/2839062769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 0x55760140abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb91d98a640 1 -- 172.21.15.98:0/2839062769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5576013fe210 msgr2=0x55760140a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb91d98a640 1 --2- 172.21.15.98:0/2839062769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5576013fe210 0x55760140a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb91d98a640 1 -- 172.21.15.98:0/2839062769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 msgr2=0x55760140abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb91d98a640 1 --2- 172.21.15.98:0/2839062769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 0x55760140abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb91d98a640 1 -- 172.21.15.98:0/2839062769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576011417d0 con 0x55760140d730 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.714+0000 7fb91d189640 1 --2- 172.21.15.98:0/2839062769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5576013fe210 0x55760140a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb91d98a640 1 --2- 172.21.15.98:0/2839062769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55760140d730 0x55760140b130 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fb9100306a0 tx=0x7fb910030e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a0a0ed3cca7334 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb900ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb900ff9640 1 -- 172.21.15.98:0/2839062769 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91004f030 con 0x55760140d730 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb900ff9640 1 -- 172.21.15.98:0/2839062769 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fb910003e00 con 0x55760140d730 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb9221ce740 1 -- 172.21.15.98:0/2839062769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb910036040 con 0x55760140d730 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb900ff9640 1 -- 172.21.15.98:0/2839062769 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb910061050 con 0x55760140d730 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb9221ce740 10 client.17784 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb9221ce740 1 -- 172.21.15.98:0/2839062769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557601410570 con 0x55760140d730 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb900ff9640 1 -- 172.21.15.98:0/2839062769 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb910036040 con 0x55760140d730 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb900ff9640 1 -- 172.21.15.98:0/2839062769 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb910030070 con 0x55760140d730 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb9221ce740 10 client.17784 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb9221ce740 10 client.17784 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:58:44.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb9221ce740 20 client.17784 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb9221ce740 1 --2- 172.21.15.98:0/2839062769 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55760141d8e0 0x55760141fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb9221ce740 4 client.17784 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.715+0000 7fb9221ce740 1 -- 172.21.15.98:0/2839062769 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557601371370 con 0x55760141d8e0 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.716+0000 7fb91d189640 1 --2- 172.21.15.98:0/2839062769 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55760141d8e0 0x55760141fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.716+0000 7fb900ff9640 10 client.17784 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.716+0000 7fb91d189640 1 --2- 172.21.15.98:0/2839062769 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55760141d8e0 0x55760141fcd0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ea8b1690f8c47ba2 server_cookie=b8e10aac1985e48c in_seq=0 out_seq=0 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb900ff9640 1 -- 172.21.15.98:0/2839062769 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557601371370 con 0x55760141d8e0 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb900ff9640 10 client.17784 handle_command_reply: tid=0 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb9221ce740 1 client.17784 shutdown 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb9221ce740 1 -- 172.21.15.98:0/2839062769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55760140d730 msgr2=0x55760140b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb9221ce740 1 --2- 172.21.15.98:0/2839062769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55760140d730 0x55760140b130 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fb9100306a0 tx=0x7fb910030e10 comp rx=0 tx=0).stop 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb9221ce740 1 -- 172.21.15.98:0/2839062769 shutdown_connections 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb9221ce740 1 --2- 172.21.15.98:0/2839062769 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55760141d8e0 0x55760141fcd0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb9221ce740 1 --2- 172.21.15.98:0/2839062769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55760140d730 0x55760140b130 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb9221ce740 1 --2- 172.21.15.98:0/2839062769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 0x55760140abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb900ff9640 0 client.17784 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:58:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb9221ce740 1 --2- 172.21.15.98:0/2839062769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5576013fe210 0x55760140a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb9221ce740 1 -- 172.21.15.98:0/2839062769 >> 172.21.15.98:0/2839062769 conn(0x5576013fa650 msgr2=0x5576014003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb9221ce740 1 -- 172.21.15.98:0/2839062769 shutdown_connections 2024-05-09T08:58:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb9221ce740 1 -- 172.21.15.98:0/2839062769 wait complete. 2024-05-09T08:58:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.717+0000 7fb9221ce740 20 client.17784 trim_cache size 0 max 16384 2024-05-09T08:58:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.730+0000 7fb9221ce740 1 Processor -- start 2024-05-09T08:58:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.730+0000 7fb9221ce740 1 -- start start 2024-05-09T08:58:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.730+0000 7fb9221ce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013fe210 0x55760140cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.730+0000 7fb9221ce740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 0x5576014194d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.730+0000 7fb9221ce740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 0x557601425530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.730+0000 7fb9221ce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9181acbc0 con 0x5576013fe210 2024-05-09T08:58:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.730+0000 7fb9221ce740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb9181ae700 con 0x5576013feb30 2024-05-09T08:58:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.730+0000 7fb9221ce740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb9181ad290 con 0x55760140d730 2024-05-09T08:58:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.730+0000 7fb91d189640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013fe210 0x55760140cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.730+0000 7fb91d189640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013fe210 0x55760140cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53722/0 (socket says 172.21.15.98:53722) 2024-05-09T08:58:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.730+0000 7fb91d189640 1 -- 172.21.15.98:0/2078953934 learned_addr learned my addr 172.21.15.98:0/2078953934 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb91d98a640 1 --2- 172.21.15.98:0/2078953934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 0x557601425530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb91d189640 1 -- 172.21.15.98:0/2078953934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 msgr2=0x557601425530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb91d189640 1 --2- 172.21.15.98:0/2078953934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 0x557601425530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb91d189640 1 -- 172.21.15.98:0/2078953934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 msgr2=0x5576014194d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb91d189640 1 --2- 172.21.15.98:0/2078953934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 0x5576014194d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb91d189640 1 -- 172.21.15.98:0/2078953934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557601420300 con 0x5576013fe210 2024-05-09T08:58:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb91d189640 1 --2- 172.21.15.98:0/2078953934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013fe210 0x55760140cde0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fb91400d740 tx=0x7fb9140aba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153fe5c5d70dd9b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb902ffd640 1 -- 172.21.15.98:0/2078953934 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb914020040 con 0x5576013fe210 2024-05-09T08:58:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb902ffd640 1 -- 172.21.15.98:0/2078953934 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb91409bae0 con 0x5576013fe210 2024-05-09T08:58:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb902ffd640 1 -- 172.21.15.98:0/2078953934 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb914005320 con 0x5576013fe210 2024-05-09T08:58:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb9221ce740 1 -- 172.21.15.98:0/2078953934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013fe210 msgr2=0x55760140cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb9221ce740 1 --2- 172.21.15.98:0/2078953934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013fe210 0x55760140cde0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fb91400d740 tx=0x7fb9140aba40 comp rx=0 tx=0).stop 2024-05-09T08:58:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb9221ce740 1 -- 172.21.15.98:0/2078953934 shutdown_connections 2024-05-09T08:58:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb9221ce740 1 --2- 172.21.15.98:0/2078953934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 0x557601425530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb9221ce740 1 --2- 172.21.15.98:0/2078953934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013feb30 0x5576014194d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb9221ce740 1 --2- 172.21.15.98:0/2078953934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013fe210 0x55760140cde0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb9221ce740 1 -- 172.21.15.98:0/2078953934 >> 172.21.15.98:0/2078953934 conn(0x5576013fa650 msgr2=0x5576013f97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb9221ce740 1 -- 172.21.15.98:0/2078953934 shutdown_connections 2024-05-09T08:58:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.731+0000 7fb9221ce740 1 -- 172.21.15.98:0/2078953934 wait complete. 2024-05-09T08:58:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb9221ce740 1 -- start start 2024-05-09T08:58:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb9221ce740 1 Processor -- start 2024-05-09T08:58:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb9221ce740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013fe210 0x557601415b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb9221ce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013feb30 0x557601410fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb9221ce740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 0x557601411500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb9221ce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9181ae030 con 0x5576013feb30 2024-05-09T08:58:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb9221ce740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb9181ad960 con 0x5576013fe210 2024-05-09T08:58:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb9221ce740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb9181aabc0 con 0x55760140d730 2024-05-09T08:58:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb9221ce740 10 client.0 _ll_register_callbacks cb 0x7fb91004d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb91c988640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013feb30 0x557601410fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb91c988640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013feb30 0x557601410fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53736/0 (socket says 172.21.15.98:53736) 2024-05-09T08:58:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb91c988640 1 -- 172.21.15.98:0/1429364814 learned_addr learned my addr 172.21.15.98:0/1429364814 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb91d98a640 1 --2- 172.21.15.98:0/1429364814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 0x557601411500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb91d189640 1 --2- 172.21.15.98:0/1429364814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013fe210 0x557601415b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb91c988640 1 -- 172.21.15.98:0/1429364814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 msgr2=0x557601411500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb91c988640 1 --2- 172.21.15.98:0/1429364814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 0x557601411500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb91c988640 1 -- 172.21.15.98:0/1429364814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013fe210 msgr2=0x557601415b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb91c988640 1 --2- 172.21.15.98:0/1429364814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013fe210 0x557601415b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.732+0000 7fb91c988640 1 -- 172.21.15.98:0/1429364814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55760140fee0 con 0x5576013feb30 2024-05-09T08:58:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb91d189640 1 --2- 172.21.15.98:0/1429364814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013fe210 0x557601415b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb91c988640 1 --2- 172.21.15.98:0/1429364814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013feb30 0x557601410fc0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fb918100550 tx=0x7fb908012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f872964c7420ec8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb900ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb900ff9640 1 -- 172.21.15.98:0/1429364814 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb908010d60 con 0x5576013feb30 2024-05-09T08:58:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb900ff9640 1 -- 172.21.15.98:0/1429364814 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fb908005020 con 0x5576013feb30 2024-05-09T08:58:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb9221ce740 1 -- 172.21.15.98:0/1429364814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb91409bae0 con 0x5576013feb30 2024-05-09T08:58:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb900ff9640 1 -- 172.21.15.98:0/1429364814 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb908025050 con 0x5576013feb30 2024-05-09T08:58:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb9221ce740 10 client.17790 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb9221ce740 1 -- 172.21.15.98:0/1429364814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557601483a20 con 0x5576013feb30 2024-05-09T08:58:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb900ff9640 1 -- 172.21.15.98:0/1429364814 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb91409bae0 con 0x5576013feb30 2024-05-09T08:58:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb900ff9640 1 -- 172.21.15.98:0/1429364814 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb908019030 con 0x5576013feb30 2024-05-09T08:58:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb9221ce740 10 client.17790 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb9221ce740 10 client.17790 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:58:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb9221ce740 20 client.17790 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb9221ce740 1 --2- 172.21.15.98:0/1429364814 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x557601484250 0x557601486670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb9221ce740 4 client.17790 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:58:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.733+0000 7fb9221ce740 1 -- 172.21.15.98:0/1429364814 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55760136b6c0 con 0x557601484250 2024-05-09T08:58:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb91d189640 1 --2- 172.21.15.98:0/1429364814 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x557601484250 0x557601486670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb900ff9640 10 client.17790 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:58:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb91d189640 1 --2- 172.21.15.98:0/1429364814 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x557601484250 0x557601486670 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d439290818885e7d server_cookie=59726297638568a4 in_seq=0 out_seq=0 2024-05-09T08:58:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb900ff9640 1 -- 172.21.15.98:0/1429364814 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55760136b6c0 con 0x557601484250 2024-05-09T08:58:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb900ff9640 10 client.17790 handle_command_reply: tid=0 2024-05-09T08:58:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb9221ce740 1 client.17790 shutdown 2024-05-09T08:58:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb9221ce740 1 -- 172.21.15.98:0/1429364814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013feb30 msgr2=0x557601410fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb9221ce740 1 --2- 172.21.15.98:0/1429364814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013feb30 0x557601410fc0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fb918100550 tx=0x7fb908012b60 comp rx=0 tx=0).stop 2024-05-09T08:58:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb9221ce740 1 -- 172.21.15.98:0/1429364814 shutdown_connections 2024-05-09T08:58:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb9221ce740 1 --2- 172.21.15.98:0/1429364814 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x557601484250 0x557601486670 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb9221ce740 1 --2- 172.21.15.98:0/1429364814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55760140d730 0x557601411500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb900ff9640 0 client.17790 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:58:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb9221ce740 1 --2- 172.21.15.98:0/1429364814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576013feb30 0x557601410fc0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb9221ce740 1 --2- 172.21.15.98:0/1429364814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5576013fe210 0x557601415b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb9221ce740 1 -- 172.21.15.98:0/1429364814 >> 172.21.15.98:0/1429364814 conn(0x5576013fa650 msgr2=0x557601408e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.734+0000 7fb9221ce740 1 -- 172.21.15.98:0/1429364814 shutdown_connections 2024-05-09T08:58:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.735+0000 7fb9221ce740 1 -- 172.21.15.98:0/1429364814 wait complete. 2024-05-09T08:58:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.735+0000 7fb9221ce740 20 client.17790 trim_cache size 0 max 16384 2024-05-09T08:58:44.742 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:58:44.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.735+0000 7fb91f3ed640 1 -- 172.21.15.98:0/1052847151 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb9040a2d00 msgr2=0x7fb9040a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.735+0000 7fb91f3ed640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb9040a2d00 0x7fb9040a51c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb91004c000 tx=0x7fb910048000 comp rx=0 tx=0).stop 2024-05-09T08:58:44.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.735+0000 7fb91f3ed640 1 -- 172.21.15.98:0/1052847151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb918069e30 msgr2=0x7fb918198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.735+0000 7fb91f3ed640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb918069e30 0x7fb918198080 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fb91400d770 tx=0x7fb91400dc30 comp rx=0 tx=0).stop 2024-05-09T08:58:44.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.735+0000 7fb91f3ed640 1 -- 172.21.15.98:0/1052847151 shutdown_connections 2024-05-09T08:58:44.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.735+0000 7fb91f3ed640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb9040a2d00 0x7fb9040a51c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.735+0000 7fb91f3ed640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb918107810 0x7fb91819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.735+0000 7fb91f3ed640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb91806a7f0 0x7fb9181985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.735+0000 7fb91f3ed640 1 --2- 172.21.15.98:0/1052847151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb918069e30 0x7fb918198080 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.735+0000 7fb91f3ed640 1 -- 172.21.15.98:0/1052847151 >> 172.21.15.98:0/1052847151 conn(0x7fb918076530 msgr2=0x7fb9180fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.735+0000 7fb91f3ed640 1 -- 172.21.15.98:0/1052847151 shutdown_connections 2024-05-09T08:58:44.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.735+0000 7fb91f3ed640 1 -- 172.21.15.98:0/1052847151 wait complete. 2024-05-09T08:58:44.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:58:44.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.850+0000 7f9baa25d640 1 -- 172.21.15.98:0/1611863703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ba41021c0 msgr2=0x7f9ba410e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.850+0000 7f9baa25d640 1 --2- 172.21.15.98:0/1611863703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ba41021c0 0x7f9ba410e6d0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009cd0 tx=0x7f9b9c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.850+0000 7f9baa25d640 1 -- 172.21.15.98:0/1611863703 shutdown_connections 2024-05-09T08:58:44.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.850+0000 7f9baa25d640 1 --2- 172.21.15.98:0/1611863703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ba41021c0 0x7f9ba410e6d0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.850+0000 7f9baa25d640 1 --2- 172.21.15.98:0/1611863703 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ba4101800 0x7f9ba4101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.850+0000 7f9baa25d640 1 --2- 172.21.15.98:0/1611863703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ba4107810 0x7f9ba4107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.850+0000 7f9baa25d640 1 -- 172.21.15.98:0/1611863703 >> 172.21.15.98:0/1611863703 conn(0x7f9ba4078c80 msgr2=0x7f9ba4079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.850+0000 7f9baa25d640 1 -- 172.21.15.98:0/1611863703 shutdown_connections 2024-05-09T08:58:44.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.851+0000 7f9baa25d640 1 -- 172.21.15.98:0/1611863703 wait complete. 2024-05-09T08:58:44.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.851+0000 7f9baa25d640 1 Processor -- start 2024-05-09T08:58:44.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.851+0000 7f9baa25d640 1 -- start start 2024-05-09T08:58:44.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9baa25d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ba4101800 0x7f9ba4198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9baa25d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ba41021c0 0x7f9ba4198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9baa25d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ba4107810 0x7f9ba419ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9baa25d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba4111c20 con 0x7f9ba4101800 2024-05-09T08:58:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9baa25d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba4111aa0 con 0x7f9ba4107810 2024-05-09T08:58:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9baa25d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba4111da0 con 0x7f9ba41021c0 2024-05-09T08:58:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9ba37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ba4101800 0x7f9ba4198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9ba37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ba4101800 0x7f9ba4198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53758/0 (socket says 172.21.15.98:53758) 2024-05-09T08:58:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9ba37fe640 1 -- 172.21.15.98:0/656516922 learned_addr learned my addr 172.21.15.98:0/656516922 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9ba3fff640 1 --2- 172.21.15.98:0/656516922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ba4107810 0x7f9ba419ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9ba37fe640 1 -- 172.21.15.98:0/656516922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ba41021c0 msgr2=0x7f9ba4198680 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:58:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9ba37fe640 1 --2- 172.21.15.98:0/656516922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ba41021c0 0x7f9ba4198680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9ba37fe640 1 -- 172.21.15.98:0/656516922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ba4107810 msgr2=0x7f9ba419ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9ba37fe640 1 --2- 172.21.15.98:0/656516922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ba4107810 0x7f9ba419ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9ba37fe640 1 -- 172.21.15.98:0/656516922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba419d130 con 0x7f9ba4101800 2024-05-09T08:58:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9ba37fe640 1 --2- 172.21.15.98:0/656516922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ba4101800 0x7f9ba4198140 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400c970 tx=0x7f9b9400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9ba17fa640 1 -- 172.21.15.98:0/656516922 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94007bb0 con 0x7f9ba4101800 2024-05-09T08:58:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9baa25d640 1 -- 172.21.15.98:0/656516922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba419d420 con 0x7f9ba4101800 2024-05-09T08:58:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9ba17fa640 1 -- 172.21.15.98:0/656516922 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b94007d50 con 0x7f9ba4101800 2024-05-09T08:58:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9ba17fa640 1 -- 172.21.15.98:0/656516922 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94018800 con 0x7f9ba4101800 2024-05-09T08:58:44.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.852+0000 7f9baa25d640 1 -- 172.21.15.98:0/656516922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba41a4cb0 con 0x7f9ba4101800 2024-05-09T08:58:44.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.854+0000 7f9ba17fa640 1 -- 172.21.15.98:0/656516922 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9b94018a60 con 0x7f9ba4101800 2024-05-09T08:58:44.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9bacf6b740 1 Processor -- start 2024-05-09T08:58:44.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9bacf6b740 1 -- start start 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9ba17fa640 1 --2- 172.21.15.98:0/656516922 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9b900a2d00 0x7f9b900a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9bacf6b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2d330 0x55ba22a2d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9bacf6b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a2dc50 0x55ba22a37de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9bacf6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a38320 0x55ba22a3a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9bacf6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ba22992c40 con 0x55ba22a38320 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9bacf6b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ba2293a580 con 0x55ba22a2d330 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9bacf6b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ba2298ec00 con 0x55ba22a2dc50 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9ba3fff640 1 --2- 172.21.15.98:0/656516922 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9b900a2d00 0x7f9b900a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9ba3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a38320 0x55ba22a3a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9ba17fa640 1 -- 172.21.15.98:0/656516922 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f9b940cf900 con 0x7f9ba4101800 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9ba3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a38320 0x55ba22a3a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53772/0 (socket says 172.21.15.98:53772) 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9ba3fff640 1 -- 172.21.15.98:0/1015628120 learned_addr learned my addr 172.21.15.98:0/1015628120 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9ba37fe640 1 --2- 172.21.15.98:0/1015628120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2d330 0x55ba22a2d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.855+0000 7f9b9bfff640 1 --2- 172.21.15.98:0/1015628120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a2dc50 0x55ba22a37de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.856+0000 7f9ba3fff640 1 -- 172.21.15.98:0/1015628120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a2dc50 msgr2=0x55ba22a37de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.856+0000 7f9ba3fff640 1 --2- 172.21.15.98:0/1015628120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a2dc50 0x55ba22a37de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.856+0000 7f9ba3fff640 1 -- 172.21.15.98:0/1015628120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2d330 msgr2=0x55ba22a2d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.856+0000 7f9ba3fff640 1 --2- 172.21.15.98:0/1015628120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2d330 0x55ba22a2d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.856+0000 7f9ba3fff640 1 -- 172.21.15.98:0/1015628120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba226a63f0 con 0x55ba22a38320 2024-05-09T08:58:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.856+0000 7f9ba37fe640 1 --2- 172.21.15.98:0/1015628120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2d330 0x55ba22a2d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:58:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.856+0000 7f9ba3fff640 1 --2- 172.21.15.98:0/1015628120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a38320 0x55ba22a3a710 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c035000 tx=0x7f9b9c031b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc5bca82ab4dd87 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.856+0000 7f9ba3fff640 1 --2- 172.21.15.98:0/656516922 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9b900a2d00 0x7f9b900a51c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009a60 tx=0x7f9b9c042000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.856+0000 7f9b99ffb640 1 -- 172.21.15.98:0/1015628120 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c04d070 con 0x55ba22a38320 2024-05-09T08:58:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.856+0000 7f9b99ffb640 1 -- 172.21.15.98:0/1015628120 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b9c036040 con 0x55ba22a38320 2024-05-09T08:58:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.856+0000 7f9b99ffb640 1 -- 172.21.15.98:0/1015628120 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c0308b0 con 0x55ba22a38320 2024-05-09T08:58:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.857+0000 7f9bacf6b740 1 -- 172.21.15.98:0/1015628120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a38320 msgr2=0x55ba22a3a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.857+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/1015628120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a38320 0x55ba22a3a710 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c035000 tx=0x7f9b9c031b10 comp rx=0 tx=0).stop 2024-05-09T08:58:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.857+0000 7f9bacf6b740 1 -- 172.21.15.98:0/1015628120 shutdown_connections 2024-05-09T08:58:44.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.857+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/1015628120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a38320 0x55ba22a3a710 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.857+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/1015628120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a2dc50 0x55ba22a37de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.857+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/1015628120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2d330 0x55ba22a2d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.857+0000 7f9bacf6b740 1 -- 172.21.15.98:0/1015628120 >> 172.21.15.98:0/1015628120 conn(0x55ba22a29650 msgr2=0x55ba22a29a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.857+0000 7f9bacf6b740 1 -- 172.21.15.98:0/1015628120 shutdown_connections 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.857+0000 7f9bacf6b740 1 -- 172.21.15.98:0/1015628120 wait complete. 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.857+0000 7f9bacf6b740 1 -- start start 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.857+0000 7f9bacf6b740 1 Processor -- start 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9bacf6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a352c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9bacf6b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a2dc50 0x55ba22a35800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9bacf6b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a38320 0x55ba22a35d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9bacf6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ba22a3b0b0 con 0x55ba22a2d330 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9bacf6b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ba2294dc40 con 0x55ba22a38320 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9bacf6b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ba22a3b230 con 0x55ba22a2dc50 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9bacf6b740 10 client.0 _ll_register_callbacks cb 0x55ba226a5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:44.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9ba37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a352c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9ba37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a352c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53786/0 (socket says 172.21.15.98:53786) 2024-05-09T08:58:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9ba37fe640 1 -- 172.21.15.98:0/712374499 learned_addr learned my addr 172.21.15.98:0/712374499 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9b9bfff640 1 --2- 172.21.15.98:0/712374499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a2dc50 0x55ba22a35800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9ba3fff640 1 --2- 172.21.15.98:0/712374499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a38320 0x55ba22a35d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9ba37fe640 1 -- 172.21.15.98:0/712374499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a2dc50 msgr2=0x55ba22a35800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9ba37fe640 1 --2- 172.21.15.98:0/712374499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a2dc50 0x55ba22a35800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9ba37fe640 1 -- 172.21.15.98:0/712374499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a38320 msgr2=0x55ba22a35d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9ba37fe640 1 --2- 172.21.15.98:0/712374499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a38320 0x55ba22a35d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9ba37fe640 1 -- 172.21.15.98:0/712374499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba227707d0 con 0x55ba22a2d330 2024-05-09T08:58:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9b9bfff640 1 --2- 172.21.15.98:0/712374499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a2dc50 0x55ba22a35800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9ba37fe640 1 --2- 172.21.15.98:0/712374499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a352c0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f9b940cd240 tx=0x7f9b9409b4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de458d1b575350ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9b82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9b82ffd640 1 -- 172.21.15.98:0/712374499 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9409c3a0 con 0x55ba22a2d330 2024-05-09T08:58:44.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9b82ffd640 1 -- 172.21.15.98:0/712374499 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f9b94098130 con 0x55ba22a2d330 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9bacf6b740 1 -- 172.21.15.98:0/712374499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b9c036040 con 0x55ba22a2d330 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9b82ffd640 1 -- 172.21.15.98:0/712374499 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9b9409cd40 con 0x55ba22a2d330 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9bacf6b740 10 client.17802 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.858+0000 7f9bacf6b740 1 -- 172.21.15.98:0/712374499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ba22a472b0 con 0x55ba22a2d330 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.859+0000 7f9b82ffd640 1 -- 172.21.15.98:0/712374499 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b9c036040 con 0x55ba22a2d330 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.859+0000 7f9b82ffd640 1 -- 172.21.15.98:0/712374499 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f9b94020030 con 0x55ba22a2d330 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.859+0000 7f9bacf6b740 10 client.17802 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.859+0000 7f9bacf6b740 10 client.17802 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.859+0000 7f9bacf6b740 20 client.17802 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.859+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/712374499 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55ba22a48710 0x55ba22a4ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.859+0000 7f9bacf6b740 4 client.17802 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.859+0000 7f9bacf6b740 1 -- 172.21.15.98:0/712374499 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba229a0370 con 0x55ba22a48710 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.860+0000 7f9b9bfff640 1 --2- 172.21.15.98:0/712374499 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55ba22a48710 0x55ba22a4ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.860+0000 7f9b9bfff640 1 --2- 172.21.15.98:0/712374499 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55ba22a48710 0x55ba22a4ab40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=98e64742090b0fc server_cookie=517e8fb4522041de in_seq=0 out_seq=0 2024-05-09T08:58:44.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.860+0000 7f9b82ffd640 10 client.17802 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9b82ffd640 1 -- 172.21.15.98:0/712374499 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ba229a0370 con 0x55ba22a48710 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9b82ffd640 10 client.17802 handle_command_reply: tid=0 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9bacf6b740 1 client.17802 shutdown 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9bacf6b740 1 -- 172.21.15.98:0/712374499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 msgr2=0x55ba22a352c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/712374499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a352c0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f9b940cd240 tx=0x7f9b9409b4e0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9bacf6b740 1 -- 172.21.15.98:0/712374499 shutdown_connections 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/712374499 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55ba22a48710 0x55ba22a4ab40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/712374499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a38320 0x55ba22a35d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9b82ffd640 0 client.17802 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/712374499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a2dc50 0x55ba22a35800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/712374499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a352c0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9bacf6b740 1 -- 172.21.15.98:0/712374499 >> 172.21.15.98:0/712374499 conn(0x55ba22a29650 msgr2=0x55ba22a2f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9bacf6b740 1 -- 172.21.15.98:0/712374499 shutdown_connections 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9bacf6b740 1 -- 172.21.15.98:0/712374499 wait complete. 2024-05-09T08:58:44.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.861+0000 7f9bacf6b740 20 client.17802 trim_cache size 0 max 16384 2024-05-09T08:58:44.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.874+0000 7f9bacf6b740 1 Processor -- start 2024-05-09T08:58:44.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.874+0000 7f9bacf6b740 1 -- start start 2024-05-09T08:58:44.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.874+0000 7f9bacf6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a43f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.874+0000 7f9bacf6b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2dc50 0x55ba22a2fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.874+0000 7f9bacf6b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a38320 0x55ba22a50410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.874+0000 7f9bacf6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba41aee40 con 0x55ba22a2d330 2024-05-09T08:58:44.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.874+0000 7f9bacf6b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba41b0980 con 0x55ba22a2dc50 2024-05-09T08:58:44.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.874+0000 7f9bacf6b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba41af510 con 0x55ba22a38320 2024-05-09T08:58:44.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.874+0000 7f9ba37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a43f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.874+0000 7f9ba37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a43f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53792/0 (socket says 172.21.15.98:53792) 2024-05-09T08:58:44.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.874+0000 7f9ba37fe640 1 -- 172.21.15.98:0/1652646793 learned_addr learned my addr 172.21.15.98:0/1652646793 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9b9bfff640 1 --2- 172.21.15.98:0/1652646793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2dc50 0x55ba22a2fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9ba37fe640 1 -- 172.21.15.98:0/1652646793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a38320 msgr2=0x55ba22a50410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9ba37fe640 1 --2- 172.21.15.98:0/1652646793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a38320 0x55ba22a50410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9ba37fe640 1 -- 172.21.15.98:0/1652646793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2dc50 msgr2=0x55ba22a2fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9ba37fe640 1 --2- 172.21.15.98:0/1652646793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2dc50 0x55ba22a2fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9ba37fe640 1 -- 172.21.15.98:0/1652646793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba22a1ad20 con 0x55ba22a2d330 2024-05-09T08:58:44.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9ba37fe640 1 --2- 172.21.15.98:0/1652646793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a43f70 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f9b9409ba50 tx=0x7f9b940076c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf20c68a454d5ea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9b997fa640 1 -- 172.21.15.98:0/1652646793 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b940ba3e0 con 0x55ba22a2d330 2024-05-09T08:58:44.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9b997fa640 1 -- 172.21.15.98:0/1652646793 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b9407e070 con 0x55ba22a2d330 2024-05-09T08:58:44.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9b997fa640 1 -- 172.21.15.98:0/1652646793 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9409af10 con 0x55ba22a2d330 2024-05-09T08:58:44.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9bacf6b740 1 -- 172.21.15.98:0/1652646793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 msgr2=0x55ba22a43f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/1652646793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a43f70 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f9b9409ba50 tx=0x7f9b940076c0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9bacf6b740 1 -- 172.21.15.98:0/1652646793 shutdown_connections 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/1652646793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a38320 0x55ba22a50410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/1652646793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2dc50 0x55ba22a2fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.875+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/1652646793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a43f70 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9bacf6b740 1 -- 172.21.15.98:0/1652646793 >> 172.21.15.98:0/1652646793 conn(0x55ba22a29650 msgr2=0x55ba22a287b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9bacf6b740 1 -- 172.21.15.98:0/1652646793 shutdown_connections 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9bacf6b740 1 -- 172.21.15.98:0/1652646793 wait complete. 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9bacf6b740 1 -- start start 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9bacf6b740 1 Processor -- start 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9bacf6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a40480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9bacf6b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2dc50 0x55ba22a409c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9bacf6b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a38320 0x55ba22a3d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9bacf6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba41b02b0 con 0x55ba22a2d330 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9bacf6b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba41afbe0 con 0x55ba22a2dc50 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9bacf6b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba41ace40 con 0x55ba22a38320 2024-05-09T08:58:44.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9bacf6b740 10 client.0 _ll_register_callbacks cb 0x7f9b9400c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:44.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9ba37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a40480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9ba37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a40480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53806/0 (socket says 172.21.15.98:53806) 2024-05-09T08:58:44.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.876+0000 7f9ba37fe640 1 -- 172.21.15.98:0/451220960 learned_addr learned my addr 172.21.15.98:0/451220960 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9ba37fe640 1 -- 172.21.15.98:0/451220960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a38320 msgr2=0x55ba22a3d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9ba37fe640 1 --2- 172.21.15.98:0/451220960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a38320 0x55ba22a3d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9ba37fe640 1 -- 172.21.15.98:0/451220960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2dc50 msgr2=0x55ba22a409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:44.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9b9bfff640 1 --2- 172.21.15.98:0/451220960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2dc50 0x55ba22a409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9ba37fe640 1 --2- 172.21.15.98:0/451220960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2dc50 0x55ba22a409c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9ba37fe640 1 -- 172.21.15.98:0/451220960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba22a3dc50 con 0x55ba22a2d330 2024-05-09T08:58:44.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9ba37fe640 1 --2- 172.21.15.98:0/451220960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a40480 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f9b940ba360 tx=0x7f9b9409b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c869917ff41e4a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9b82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:44.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9b82ffd640 1 -- 172.21.15.98:0/451220960 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94097a10 con 0x55ba22a2d330 2024-05-09T08:58:44.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9b82ffd640 1 -- 172.21.15.98:0/451220960 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f9b9409a980 con 0x55ba22a2d330 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9bacf6b740 1 -- 172.21.15.98:0/451220960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b9407e070 con 0x55ba22a2d330 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9b82ffd640 1 -- 172.21.15.98:0/451220960 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9b9409fdc0 con 0x55ba22a2d330 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9bacf6b740 10 client.17808 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9bacf6b740 1 -- 172.21.15.98:0/451220960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ba22aaf6d0 con 0x55ba22a2d330 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9b82ffd640 1 -- 172.21.15.98:0/451220960 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b9407e070 con 0x55ba22a2d330 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9b82ffd640 1 -- 172.21.15.98:0/451220960 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f9b94020070 con 0x55ba22a2d330 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9bacf6b740 10 client.17808 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9bacf6b740 10 client.17808 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9bacf6b740 20 client.17808 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/451220960 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55ba22aafba0 0x55ba22ab1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9bacf6b740 4 client.17808 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.877+0000 7f9bacf6b740 1 -- 172.21.15.98:0/451220960 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ba2299a6c0 con 0x55ba22aafba0 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.878+0000 7f9b9bfff640 1 --2- 172.21.15.98:0/451220960 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55ba22aafba0 0x55ba22ab1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.878+0000 7f9b82ffd640 10 client.17808 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:58:44.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.878+0000 7f9b9bfff640 1 --2- 172.21.15.98:0/451220960 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55ba22aafba0 0x55ba22ab1f90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bb16e92acdf3b7e server_cookie=c259d3c8a55a294d in_seq=0 out_seq=0 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.878+0000 7f9b82ffd640 1 -- 172.21.15.98:0/451220960 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ba2299a6c0 con 0x55ba22aafba0 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.878+0000 7f9b82ffd640 10 client.17808 handle_command_reply: tid=0 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.878+0000 7f9bacf6b740 1 client.17808 shutdown 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9bacf6b740 1 -- 172.21.15.98:0/451220960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 msgr2=0x55ba22a40480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/451220960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a40480 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f9b940ba360 tx=0x7f9b9409b9c0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9bacf6b740 1 -- 172.21.15.98:0/451220960 shutdown_connections 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/451220960 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55ba22aafba0 0x55ba22ab1f90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/451220960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ba22a38320 0x55ba22a3d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9b82ffd640 0 client.17808 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/451220960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ba22a2dc50 0x55ba22a409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9bacf6b740 1 --2- 172.21.15.98:0/451220960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba22a2d330 0x55ba22a40480 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9bacf6b740 1 -- 172.21.15.98:0/451220960 >> 172.21.15.98:0/451220960 conn(0x55ba22a29650 msgr2=0x55ba22a36970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9bacf6b740 1 -- 172.21.15.98:0/451220960 shutdown_connections 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9bacf6b740 1 -- 172.21.15.98:0/451220960 wait complete. 2024-05-09T08:58:44.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9bacf6b740 20 client.17808 trim_cache size 0 max 16384 2024-05-09T08:58:44.885 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:58:44.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9baa25d640 1 -- 172.21.15.98:0/656516922 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9b900a2d00 msgr2=0x7f9b900a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9baa25d640 1 --2- 172.21.15.98:0/656516922 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9b900a2d00 0x7f9b900a51c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009a60 tx=0x7f9b9c042000 comp rx=0 tx=0).stop 2024-05-09T08:58:44.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9baa25d640 1 -- 172.21.15.98:0/656516922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ba4101800 msgr2=0x7f9ba4198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9baa25d640 1 --2- 172.21.15.98:0/656516922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ba4101800 0x7f9ba4198140 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400c970 tx=0x7f9b9400ce30 comp rx=0 tx=0).stop 2024-05-09T08:58:44.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9baa25d640 1 -- 172.21.15.98:0/656516922 shutdown_connections 2024-05-09T08:58:44.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9baa25d640 1 --2- 172.21.15.98:0/656516922 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9b900a2d00 0x7f9b900a51c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9baa25d640 1 --2- 172.21.15.98:0/656516922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ba4107810 0x7f9ba419ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9baa25d640 1 --2- 172.21.15.98:0/656516922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ba41021c0 0x7f9ba4198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9baa25d640 1 --2- 172.21.15.98:0/656516922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ba4101800 0x7f9ba4198140 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9baa25d640 1 -- 172.21.15.98:0/656516922 >> 172.21.15.98:0/656516922 conn(0x7f9ba4078c80 msgr2=0x7f9ba40ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9baa25d640 1 -- 172.21.15.98:0/656516922 shutdown_connections 2024-05-09T08:58:44.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.879+0000 7f9baa25d640 1 -- 172.21.15.98:0/656516922 wait complete. 2024-05-09T08:58:44.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:58:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.995+0000 7f3a83c31640 1 -- 172.21.15.98:0/892044326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a7c105be0 msgr2=0x7f3a7c106050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.995+0000 7f3a83c31640 1 --2- 172.21.15.98:0/892044326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a7c105be0 0x7f3a7c106050 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009ca0 tx=0x7f3a7402a950 comp rx=0 tx=0).stop 2024-05-09T08:58:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.995+0000 7f3a83c31640 1 -- 172.21.15.98:0/892044326 shutdown_connections 2024-05-09T08:58:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.995+0000 7f3a83c31640 1 --2- 172.21.15.98:0/892044326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a7c105be0 0x7f3a7c106050 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.995+0000 7f3a83c31640 1 --2- 172.21.15.98:0/892044326 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a7c1011e0 0x7f3a7c1056a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.995+0000 7f3a83c31640 1 --2- 172.21.15.98:0/892044326 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a7c1008c0 0x7f3a7c100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.995+0000 7f3a83c31640 1 -- 172.21.15.98:0/892044326 >> 172.21.15.98:0/892044326 conn(0x7f3a7c077890 msgr2=0x7f3a7c077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.995+0000 7f3a83c31640 1 -- 172.21.15.98:0/892044326 shutdown_connections 2024-05-09T08:58:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.995+0000 7f3a83c31640 1 -- 172.21.15.98:0/892044326 wait complete. 2024-05-09T08:58:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a83c31640 1 Processor -- start 2024-05-09T08:58:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a83c31640 1 -- start start 2024-05-09T08:58:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a83c31640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a7c1008c0 0x7f3a7c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a83c31640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a7c1011e0 0x7f3a7c19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a83c31640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a7c105be0 0x7f3a7c1a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a83c31640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1181d0 con 0x7f3a7c1011e0 2024-05-09T08:58:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a83c31640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c118050 con 0x7f3a7c1008c0 2024-05-09T08:58:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a83c31640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c118350 con 0x7f3a7c105be0 2024-05-09T08:58:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a811cc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a7c1011e0 0x7f3a7c19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a811cc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a7c1011e0 0x7f3a7c19ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53826/0 (socket says 172.21.15.98:53826) 2024-05-09T08:58:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a811cc640 1 -- 172.21.15.98:0/3824682875 learned_addr learned my addr 172.21.15.98:0/3824682875 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a821ce640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a7c105be0 0x7f3a7c1a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a819cd640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a7c1008c0 0x7f3a7c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a811cc640 1 -- 172.21.15.98:0/3824682875 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a7c105be0 msgr2=0x7f3a7c1a2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a811cc640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a7c105be0 0x7f3a7c1a2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a811cc640 1 -- 172.21.15.98:0/3824682875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a7c1008c0 msgr2=0x7f3a7c19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a811cc640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a7c1008c0 0x7f3a7c19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a811cc640 1 -- 172.21.15.98:0/3824682875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a7c1a3670 con 0x7f3a7c1011e0 2024-05-09T08:58:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a819cd640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a7c1008c0 0x7f3a7c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.996+0000 7f3a811cc640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a7c1011e0 0x7f3a7c19ec10 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00d950 tx=0x7f3a6c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.997+0000 7f3a72ffd640 1 -- 172.21.15.98:0/3824682875 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c017030 con 0x7f3a7c1011e0 2024-05-09T08:58:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.997+0000 7f3a83c31640 1 -- 172.21.15.98:0/3824682875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a7c1a3960 con 0x7f3a7c1011e0 2024-05-09T08:58:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.997+0000 7f3a72ffd640 1 -- 172.21.15.98:0/3824682875 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6c004020 con 0x7f3a7c1011e0 2024-05-09T08:58:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.997+0000 7f3a72ffd640 1 -- 172.21.15.98:0/3824682875 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c0047a0 con 0x7f3a7c1011e0 2024-05-09T08:58:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.997+0000 7f3a83c31640 1 -- 172.21.15.98:0/3824682875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a7c1ab1f0 con 0x7f3a7c1011e0 2024-05-09T08:58:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.999+0000 7f3a72ffd640 1 -- 172.21.15.98:0/3824682875 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3a6c0041d0 con 0x7f3a7c1011e0 2024-05-09T08:58:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:44.999+0000 7f3a86a1a740 1 Processor -- start 2024-05-09T08:58:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a86a1a740 1 -- start start 2024-05-09T08:58:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a86a1a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ec330 0x562f836ec710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a72ffd640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3a680a2d00 0x7f3a680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a86a1a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836ecc50 0x562f836f6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a86a1a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836f7320 0x562f836f9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a86a1a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562f83651c40 con 0x562f836ec330 2024-05-09T08:58:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a86a1a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562f835f9580 con 0x562f836ecc50 2024-05-09T08:58:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a86a1a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562f8364dc00 con 0x562f836f7320 2024-05-09T08:58:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a821ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ec330 0x562f836ec710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a819cd640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3a680a2d00 0x7f3a680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a821ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ec330 0x562f836ec710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53836/0 (socket says 172.21.15.98:53836) 2024-05-09T08:58:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a821ce640 1 -- 172.21.15.98:0/462877473 learned_addr learned my addr 172.21.15.98:0/462877473 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a72ffd640 1 -- 172.21.15.98:0/3824682875 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f3a6c0cf2b0 con 0x7f3a7c1011e0 2024-05-09T08:58:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a811cc640 1 --2- 172.21.15.98:0/462877473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836ecc50 0x562f836f6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a821ce640 1 --2- 172.21.15.98:0/462877473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836f7320 0x562f836f9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a821ce640 1 -- 172.21.15.98:0/462877473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836f7320 msgr2=0x562f836f9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a821ce640 1 --2- 172.21.15.98:0/462877473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836f7320 0x562f836f9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a821ce640 1 -- 172.21.15.98:0/462877473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836ecc50 msgr2=0x562f836f6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a821ce640 1 --2- 172.21.15.98:0/462877473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836ecc50 0x562f836f6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a821ce640 1 -- 172.21.15.98:0/462877473 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f833653f0 con 0x562f836ec330 2024-05-09T08:58:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a819cd640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3a680a2d00 0x7f3a680a51c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3a78009b40 tx=0x7f3a78005e50 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.000+0000 7f3a821ce640 1 --2- 172.21.15.98:0/462877473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ec330 0x562f836ec710 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f3a74005fa0 tx=0x7f3a74005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c069fc2ddac4bf3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.001+0000 7f3a67fff640 1 -- 172.21.15.98:0/462877473 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74034040 con 0x562f836ec330 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.001+0000 7f3a67fff640 1 -- 172.21.15.98:0/462877473 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a74003e50 con 0x562f836ec330 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.001+0000 7f3a67fff640 1 -- 172.21.15.98:0/462877473 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74004620 con 0x562f836ec330 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.001+0000 7f3a86a1a740 1 -- 172.21.15.98:0/462877473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ec330 msgr2=0x562f836ec710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.001+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/462877473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ec330 0x562f836ec710 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f3a74005fa0 tx=0x7f3a74005df0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.001+0000 7f3a86a1a740 1 -- 172.21.15.98:0/462877473 shutdown_connections 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.001+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/462877473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836f7320 0x562f836f9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.001+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/462877473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836ecc50 0x562f836f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.001+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/462877473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ec330 0x562f836ec710 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.001+0000 7f3a86a1a740 1 -- 172.21.15.98:0/462877473 >> 172.21.15.98:0/462877473 conn(0x562f836e8650 msgr2=0x562f836e8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.001+0000 7f3a86a1a740 1 -- 172.21.15.98:0/462877473 shutdown_connections 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.001+0000 7f3a86a1a740 1 -- 172.21.15.98:0/462877473 wait complete. 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.002+0000 7f3a86a1a740 1 -- start start 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.002+0000 7f3a86a1a740 1 Processor -- start 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.002+0000 7f3a86a1a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 0x562f836f4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.002+0000 7f3a86a1a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ecc50 0x562f836f47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.002+0000 7f3a86a1a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836f7320 0x562f836f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.002+0000 7f3a86a1a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562f836fa0b0 con 0x562f836ecc50 2024-05-09T08:58:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.002+0000 7f3a86a1a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562f8360cc40 con 0x562f836f7320 2024-05-09T08:58:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.002+0000 7f3a86a1a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562f836fa230 con 0x562f836ec330 2024-05-09T08:58:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.002+0000 7f3a86a1a740 10 client.0 _ll_register_callbacks cb 0x562f83364be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.002+0000 7f3a811cc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ecc50 0x562f836f47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.002+0000 7f3a811cc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ecc50 0x562f836f47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53838/0 (socket says 172.21.15.98:53838) 2024-05-09T08:58:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.002+0000 7f3a811cc640 1 -- 172.21.15.98:0/3413115729 learned_addr learned my addr 172.21.15.98:0/3413115729 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.002+0000 7f3a821ce640 1 --2- 172.21.15.98:0/3413115729 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836f7320 0x562f836f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a819cd640 1 --2- 172.21.15.98:0/3413115729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 0x562f836f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a811cc640 1 -- 172.21.15.98:0/3413115729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 msgr2=0x562f836f4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a811cc640 1 --2- 172.21.15.98:0/3413115729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 0x562f836f4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a811cc640 1 -- 172.21.15.98:0/3413115729 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836f7320 msgr2=0x562f836f4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a811cc640 1 --2- 172.21.15.98:0/3413115729 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836f7320 0x562f836f4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a811cc640 1 -- 172.21.15.98:0/3413115729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f8342f7d0 con 0x562f836ecc50 2024-05-09T08:58:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a819cd640 1 --2- 172.21.15.98:0/3413115729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 0x562f836f4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a811cc640 1 --2- 172.21.15.98:0/3413115729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ecc50 0x562f836f47d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0cc650 tx=0x7f3a6c09ade0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369c2dc4b491c543 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a657fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a657fa640 1 -- 172.21.15.98:0/3413115729 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c09bac0 con 0x562f836ecc50 2024-05-09T08:58:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a657fa640 1 -- 172.21.15.98:0/3413115729 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f3a6c0cc870 con 0x562f836ecc50 2024-05-09T08:58:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3413115729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a74003e50 con 0x562f836ecc50 2024-05-09T08:58:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a657fa640 1 -- 172.21.15.98:0/3413115729 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3a6c00e350 con 0x562f836ecc50 2024-05-09T08:58:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a86a1a740 10 client.17820 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3413115729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562f837062e0 con 0x562f836ecc50 2024-05-09T08:58:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a657fa640 1 -- 172.21.15.98:0/3413115729 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a74003e50 con 0x562f836ecc50 2024-05-09T08:58:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a657fa640 1 -- 172.21.15.98:0/3413115729 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3a6c097a30 con 0x562f836ecc50 2024-05-09T08:58:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a86a1a740 10 client.17820 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a86a1a740 10 client.17820 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:58:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a86a1a740 20 client.17820 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3413115729 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562f837076f0 0x562f83709b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a86a1a740 4 client.17820 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.003+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3413115729 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f8365f370 con 0x562f837076f0 2024-05-09T08:58:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.004+0000 7f3a819cd640 1 --2- 172.21.15.98:0/3413115729 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562f837076f0 0x562f83709b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.004+0000 7f3a819cd640 1 --2- 172.21.15.98:0/3413115729 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562f837076f0 0x562f83709b00 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e49124bad7c3c9ff server_cookie=858e2847d9b68fb4 in_seq=0 out_seq=0 2024-05-09T08:58:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.004+0000 7f3a657fa640 10 client.17820 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:58:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a657fa640 1 -- 172.21.15.98:0/3413115729 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562f8365f370 con 0x562f837076f0 2024-05-09T08:58:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a657fa640 10 client.17820 handle_command_reply: tid=0 2024-05-09T08:58:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a86a1a740 1 client.17820 shutdown 2024-05-09T08:58:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3413115729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ecc50 msgr2=0x562f836f47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3413115729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ecc50 0x562f836f47d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0cc650 tx=0x7f3a6c09ade0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3413115729 shutdown_connections 2024-05-09T08:58:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3413115729 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562f837076f0 0x562f83709b00 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3413115729 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836f7320 0x562f836f4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a657fa640 0 client.17820 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:58:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3413115729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ecc50 0x562f836f47d0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3413115729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 0x562f836f4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3413115729 >> 172.21.15.98:0/3413115729 conn(0x562f836e8650 msgr2=0x562f836ee350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3413115729 shutdown_connections 2024-05-09T08:58:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3413115729 wait complete. 2024-05-09T08:58:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.005+0000 7f3a86a1a740 20 client.17820 trim_cache size 0 max 16384 2024-05-09T08:58:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.018+0000 7f3a86a1a740 1 Processor -- start 2024-05-09T08:58:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.018+0000 7f3a86a1a740 1 -- start start 2024-05-09T08:58:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.018+0000 7f3a86a1a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 0x562f83702f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.018+0000 7f3a86a1a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836ecc50 0x562f836eea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.018+0000 7f3a86a1a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836f7320 0x562f8370f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.018+0000 7f3a86a1a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b53f0 con 0x562f836f7320 2024-05-09T08:58:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.018+0000 7f3a86a1a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b6f30 con 0x562f836ecc50 2024-05-09T08:58:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.018+0000 7f3a86a1a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b5ac0 con 0x562f836ec330 2024-05-09T08:58:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.019+0000 7f3a821ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836f7320 0x562f8370f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.019+0000 7f3a821ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836f7320 0x562f8370f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53848/0 (socket says 172.21.15.98:53848) 2024-05-09T08:58:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.019+0000 7f3a821ce640 1 -- 172.21.15.98:0/3315559043 learned_addr learned my addr 172.21.15.98:0/3315559043 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.019+0000 7f3a819cd640 1 --2- 172.21.15.98:0/3315559043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 0x562f83702f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.019+0000 7f3a821ce640 1 -- 172.21.15.98:0/3315559043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 msgr2=0x562f83702f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.019+0000 7f3a821ce640 1 --2- 172.21.15.98:0/3315559043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 0x562f83702f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.019+0000 7f3a821ce640 1 -- 172.21.15.98:0/3315559043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836ecc50 msgr2=0x562f836eea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.019+0000 7f3a821ce640 1 --2- 172.21.15.98:0/3315559043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836ecc50 0x562f836eea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.019+0000 7f3a821ce640 1 -- 172.21.15.98:0/3315559043 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f836e41a0 con 0x562f836f7320 2024-05-09T08:58:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.019+0000 7f3a819cd640 1 --2- 172.21.15.98:0/3315559043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 0x562f83702f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.019+0000 7f3a821ce640 1 --2- 172.21.15.98:0/3315559043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836f7320 0x562f8370f380 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009a60 tx=0x7f3a74004200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe1c9e04195b288 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.019+0000 7f3a677fe640 1 -- 172.21.15.98:0/3315559043 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74034040 con 0x562f836f7320 2024-05-09T08:58:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.019+0000 7f3a677fe640 1 -- 172.21.15.98:0/3315559043 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a7402fc20 con 0x562f836f7320 2024-05-09T08:58:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.019+0000 7f3a677fe640 1 -- 172.21.15.98:0/3315559043 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74003e70 con 0x562f836f7320 2024-05-09T08:58:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3315559043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836f7320 msgr2=0x562f8370f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3315559043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836f7320 0x562f8370f380 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009a60 tx=0x7f3a74004200 comp rx=0 tx=0).stop 2024-05-09T08:58:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3315559043 shutdown_connections 2024-05-09T08:58:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3315559043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836f7320 0x562f8370f380 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3315559043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836ecc50 0x562f836eea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3315559043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 0x562f83702f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3315559043 >> 172.21.15.98:0/3315559043 conn(0x562f836e8650 msgr2=0x562f836f4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3315559043 shutdown_connections 2024-05-09T08:58:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3315559043 wait complete. 2024-05-09T08:58:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 -- start start 2024-05-09T08:58:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 Processor -- start 2024-05-09T08:58:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 0x562f836ff480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ecc50 0x562f836ff9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836f7320 0x562f836fc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b6860 con 0x562f836ecc50 2024-05-09T08:58:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b6190 con 0x562f836f7320 2024-05-09T08:58:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b33f0 con 0x562f836ec330 2024-05-09T08:58:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a86a1a740 10 client.0 _ll_register_callbacks cb 0x7f3a6c005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.020+0000 7f3a811cc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ecc50 0x562f836ff9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a811cc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ecc50 0x562f836ff9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53852/0 (socket says 172.21.15.98:53852) 2024-05-09T08:58:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a811cc640 1 -- 172.21.15.98:0/3598871342 learned_addr learned my addr 172.21.15.98:0/3598871342 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a819cd640 1 --2- 172.21.15.98:0/3598871342 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 0x562f836ff480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a821ce640 1 --2- 172.21.15.98:0/3598871342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836f7320 0x562f836fc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a811cc640 1 -- 172.21.15.98:0/3598871342 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 msgr2=0x562f836ff480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a811cc640 1 --2- 172.21.15.98:0/3598871342 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 0x562f836ff480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a811cc640 1 -- 172.21.15.98:0/3598871342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836f7320 msgr2=0x562f836fc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a811cc640 1 --2- 172.21.15.98:0/3598871342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836f7320 0x562f836fc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a811cc640 1 -- 172.21.15.98:0/3598871342 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f836fca10 con 0x562f836ecc50 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a811cc640 1 --2- 172.21.15.98:0/3598871342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ecc50 0x562f836ff9c0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c09f930 tx=0x7f3a6c005ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c206e2079becf0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a657fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a657fa640 1 -- 172.21.15.98:0/3598871342 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c09af00 con 0x562f836ecc50 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a657fa640 1 -- 172.21.15.98:0/3598871342 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f3a6c097a30 con 0x562f836ecc50 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3598871342 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a7402fc20 con 0x562f836ecc50 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a657fa640 1 -- 172.21.15.98:0/3598871342 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3a6c09f770 con 0x562f836ecc50 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a86a1a740 10 client.17826 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3598871342 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562f8376e410 con 0x562f836ecc50 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a657fa640 1 -- 172.21.15.98:0/3598871342 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a7402fc20 con 0x562f836ecc50 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a657fa640 1 -- 172.21.15.98:0/3598871342 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3a6c09a280 con 0x562f836ecc50 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a86a1a740 10 client.17826 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a86a1a740 10 client.17826 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a86a1a740 20 client.17826 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3598871342 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562f8376eb50 0x562f83770f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a86a1a740 4 client.17826 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.021+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3598871342 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562f836596c0 con 0x562f8376eb50 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a819cd640 1 --2- 172.21.15.98:0/3598871342 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562f8376eb50 0x562f83770f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a657fa640 10 client.17826 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a819cd640 1 --2- 172.21.15.98:0/3598871342 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562f8376eb50 0x562f83770f70 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b5adfe750d84345e server_cookie=8b19cf5e6a95a70b in_seq=0 out_seq=0 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a657fa640 1 -- 172.21.15.98:0/3598871342 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562f836596c0 con 0x562f8376eb50 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a657fa640 10 client.17826 handle_command_reply: tid=0 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a86a1a740 1 client.17826 shutdown 2024-05-09T08:58:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3598871342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ecc50 msgr2=0x562f836ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3598871342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ecc50 0x562f836ff9c0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c09f930 tx=0x7f3a6c005ab0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3598871342 shutdown_connections 2024-05-09T08:58:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3598871342 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x562f8376eb50 0x562f83770f70 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3598871342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f836f7320 0x562f836fc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a657fa640 0 client.17826 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:58:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3598871342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f836ecc50 0x562f836ff9c0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a86a1a740 1 --2- 172.21.15.98:0/3598871342 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f836ec330 0x562f836ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.022+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3598871342 >> 172.21.15.98:0/3598871342 conn(0x562f836e8650 msgr2=0x562f836e6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3598871342 shutdown_connections 2024-05-09T08:58:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a86a1a740 1 -- 172.21.15.98:0/3598871342 wait complete. 2024-05-09T08:58:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a86a1a740 20 client.17826 trim_cache size 0 max 16384 2024-05-09T08:58:45.030 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:58:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a83c31640 1 -- 172.21.15.98:0/3824682875 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3a680a2d00 msgr2=0x7f3a680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a83c31640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3a680a2d00 0x7f3a680a51c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3a78009b40 tx=0x7f3a78005e50 comp rx=0 tx=0).stop 2024-05-09T08:58:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a83c31640 1 -- 172.21.15.98:0/3824682875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a7c1011e0 msgr2=0x7f3a7c19ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a83c31640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a7c1011e0 0x7f3a7c19ec10 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00d950 tx=0x7f3a6c00de10 comp rx=0 tx=0).stop 2024-05-09T08:58:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a83c31640 1 -- 172.21.15.98:0/3824682875 shutdown_connections 2024-05-09T08:58:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a83c31640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3a680a2d00 0x7f3a680a51c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a83c31640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a7c105be0 0x7f3a7c1a2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a83c31640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a7c1011e0 0x7f3a7c19ec10 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a83c31640 1 --2- 172.21.15.98:0/3824682875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a7c1008c0 0x7f3a7c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a83c31640 1 -- 172.21.15.98:0/3824682875 >> 172.21.15.98:0/3824682875 conn(0x7f3a7c077890 msgr2=0x7f3a7c103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a83c31640 1 -- 172.21.15.98:0/3824682875 shutdown_connections 2024-05-09T08:58:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.023+0000 7f3a83c31640 1 -- 172.21.15.98:0/3824682875 wait complete. 2024-05-09T08:58:45.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:58:45.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.137+0000 7fe7f867d640 1 -- 172.21.15.98:0/1984801934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7f0102ac0 msgr2=0x7fe7f0102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.137+0000 7fe7f867d640 1 --2- 172.21.15.98:0/1984801934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7f0102ac0 0x7fe7f0102ea0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0009900 tx=0x7fe7e002a510 comp rx=0 tx=0).stop 2024-05-09T08:58:45.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.137+0000 7fe7f867d640 1 -- 172.21.15.98:0/1984801934 shutdown_connections 2024-05-09T08:58:45.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.137+0000 7fe7f867d640 1 --2- 172.21.15.98:0/1984801934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f010be00 0x7fe7f010e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.137+0000 7fe7f867d640 1 --2- 172.21.15.98:0/1984801934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7f01033e0 0x7fe7f010b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.137+0000 7fe7f867d640 1 --2- 172.21.15.98:0/1984801934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7f0102ac0 0x7fe7f0102ea0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.137+0000 7fe7f867d640 1 -- 172.21.15.98:0/1984801934 >> 172.21.15.98:0/1984801934 conn(0x7fe7f0076ab0 msgr2=0x7fe7f0076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.137+0000 7fe7f867d640 1 -- 172.21.15.98:0/1984801934 shutdown_connections 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.137+0000 7fe7f867d640 1 -- 172.21.15.98:0/1984801934 wait complete. 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f867d640 1 Processor -- start 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f867d640 1 -- start start 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f867d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7f0102ac0 0x7fe7f0198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f867d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f01033e0 0x7fe7f0198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f867d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7f010be00 0x7fe7f019c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f867d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f0111c00 con 0x7fe7f010be00 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f867d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f0111a80 con 0x7fe7f01033e0 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f867d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f0111d80 con 0x7fe7f0102ac0 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f6c1a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7f010be00 0x7fe7f019c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f6c1a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7f010be00 0x7fe7f019c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53856/0 (socket says 172.21.15.98:53856) 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f6c1a640 1 -- 172.21.15.98:0/1970289431 learned_addr learned my addr 172.21.15.98:0/1970289431 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/1970289431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f01033e0 0x7fe7f0198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f6419640 1 --2- 172.21.15.98:0/1970289431 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7f0102ac0 0x7fe7f0198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f6c1a640 1 -- 172.21.15.98:0/1970289431 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7f0102ac0 msgr2=0x7fe7f0198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f6c1a640 1 --2- 172.21.15.98:0/1970289431 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7f0102ac0 0x7fe7f0198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f6c1a640 1 -- 172.21.15.98:0/1970289431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f01033e0 msgr2=0x7fe7f0198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f6c1a640 1 --2- 172.21.15.98:0/1970289431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f01033e0 0x7fe7f0198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.138+0000 7fe7f6c1a640 1 -- 172.21.15.98:0/1970289431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f019d0f0 con 0x7fe7f010be00 2024-05-09T08:58:45.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.139+0000 7fe7f6c1a640 1 --2- 172.21.15.98:0/1970289431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7f010be00 0x7fe7f019c9c0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800d750 tx=0x7fe7e800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.139+0000 7fe7e77fe640 1 -- 172.21.15.98:0/1970289431 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8017070 con 0x7fe7f010be00 2024-05-09T08:58:45.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.139+0000 7fe7f867d640 1 -- 172.21.15.98:0/1970289431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f019d3e0 con 0x7fe7f010be00 2024-05-09T08:58:45.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.139+0000 7fe7e77fe640 1 -- 172.21.15.98:0/1970289431 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e8005500 con 0x7fe7f010be00 2024-05-09T08:58:45.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.139+0000 7fe7e77fe640 1 -- 172.21.15.98:0/1970289431 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8004d90 con 0x7fe7f010be00 2024-05-09T08:58:45.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.139+0000 7fe7f867d640 1 -- 172.21.15.98:0/1970289431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f01a4c70 con 0x7fe7f010be00 2024-05-09T08:58:45.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.141+0000 7fe7e77fe640 1 -- 172.21.15.98:0/1970289431 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe7e8004f30 con 0x7fe7f010be00 2024-05-09T08:58:45.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.141+0000 7fe7fb38f740 1 Processor -- start 2024-05-09T08:58:45.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.141+0000 7fe7fb38f740 1 -- start start 2024-05-09T08:58:45.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7fb38f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2330 0x55caa96b2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7fb38f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2c50 0x55caa96bcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7fb38f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 0x55caa96bf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7fb38f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55caa9617c40 con 0x55caa96b2c50 2024-05-09T08:58:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7fb38f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55caa95bf580 con 0x55caa96b2330 2024-05-09T08:58:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7fb38f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55caa9613c00 con 0x55caa96bd320 2024-05-09T08:58:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7f6419640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2c50 0x55caa96bcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7f6419640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2c50 0x55caa96bcde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53872/0 (socket says 172.21.15.98:53872) 2024-05-09T08:58:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7f6419640 1 -- 172.21.15.98:0/1808863991 learned_addr learned my addr 172.21.15.98:0/1808863991 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7e77fe640 1 --2- 172.21.15.98:0/1970289431 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe7dc0a2d00 0x7fe7dc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/1808863991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2330 0x55caa96b2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7f6c1a640 1 --2- 172.21.15.98:0/1970289431 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe7dc0a2d00 0x7fe7dc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7f6419640 1 -- 172.21.15.98:0/1808863991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 msgr2=0x55caa96bf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/1808863991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 0x55caa96bf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7f6419640 1 --2- 172.21.15.98:0/1808863991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 0x55caa96bf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7f6419640 1 -- 172.21.15.98:0/1808863991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2330 msgr2=0x55caa96b2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7f6419640 1 --2- 172.21.15.98:0/1808863991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2330 0x55caa96b2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7f6419640 1 -- 172.21.15.98:0/1808863991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caa932b3f0 con 0x55caa96b2c50 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/1808863991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 0x55caa96bf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7e77fe640 1 -- 172.21.15.98:0/1970289431 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fe7e80cf230 con 0x7fe7f010be00 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.142+0000 7fe7f6419640 1 --2- 172.21.15.98:0/1808863991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2c50 0x55caa96bcde0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0033000 tx=0x7fe7e0032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cfee97ed32bdb1b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.143+0000 7fe7f6c1a640 1 --2- 172.21.15.98:0/1970289431 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe7dc0a2d00 0x7fe7dc0a51c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe7e809b270 tx=0x7fe7e806d2a0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.143+0000 7fe7e4ff9640 1 -- 172.21.15.98:0/1808863991 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e003f070 con 0x55caa96b2c50 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.143+0000 7fe7e4ff9640 1 -- 172.21.15.98:0/1808863991 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e00317b0 con 0x55caa96b2c50 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.143+0000 7fe7e4ff9640 1 -- 172.21.15.98:0/1808863991 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0035070 con 0x55caa96b2c50 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.143+0000 7fe7fb38f740 1 -- 172.21.15.98:0/1808863991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2c50 msgr2=0x55caa96bcde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.143+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/1808863991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2c50 0x55caa96bcde0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0033000 tx=0x7fe7e0032ce0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.143+0000 7fe7fb38f740 1 -- 172.21.15.98:0/1808863991 shutdown_connections 2024-05-09T08:58:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.143+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/1808863991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 0x55caa96bf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.143+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/1808863991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2c50 0x55caa96bcde0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.143+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/1808863991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2330 0x55caa96b2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.143+0000 7fe7fb38f740 1 -- 172.21.15.98:0/1808863991 >> 172.21.15.98:0/1808863991 conn(0x55caa96ae650 msgr2=0x55caa96aea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.143+0000 7fe7fb38f740 1 -- 172.21.15.98:0/1808863991 shutdown_connections 2024-05-09T08:58:45.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.143+0000 7fe7fb38f740 1 -- 172.21.15.98:0/1808863991 wait complete. 2024-05-09T08:58:45.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7fb38f740 1 -- start start 2024-05-09T08:58:45.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7fb38f740 1 Processor -- start 2024-05-09T08:58:45.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7fb38f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2330 0x55caa96ba260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7fb38f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2c50 0x55caa96ba7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7fb38f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 0x55caa96bace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7fb38f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55caa96c00b0 con 0x55caa96b2330 2024-05-09T08:58:45.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7fb38f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55caa95d2c40 con 0x55caa96b2c50 2024-05-09T08:58:45.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7fb38f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55caa96c0230 con 0x55caa96bd320 2024-05-09T08:58:45.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7fb38f740 10 client.0 _ll_register_callbacks cb 0x55caa932abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7f5c18640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2330 0x55caa96ba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7f5c18640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2330 0x55caa96ba260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53874/0 (socket says 172.21.15.98:53874) 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7f5c18640 1 -- 172.21.15.98:0/4205913274 learned_addr learned my addr 172.21.15.98:0/4205913274 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7f6419640 1 --2- 172.21.15.98:0/4205913274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2c50 0x55caa96ba7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7f5c18640 1 -- 172.21.15.98:0/4205913274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 msgr2=0x55caa96bace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/4205913274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 0x55caa96bace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7f5c18640 1 -- 172.21.15.98:0/4205913274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2c50 msgr2=0x55caa96ba7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/4205913274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2c50 0x55caa96ba7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7f5c18640 1 -- 172.21.15.98:0/4205913274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55caa93f57d0 con 0x55caa96b2330 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.144+0000 7fe7f6419640 1 --2- 172.21.15.98:0/4205913274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2c50 0x55caa96ba7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/4205913274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2330 0x55caa96ba260 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00afd0 tx=0x7fe7ec011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73416876495e9439 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7d9ffb640 1 -- 172.21.15.98:0/4205913274 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec0142d0 con 0x55caa96b2330 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7d9ffb640 1 -- 172.21.15.98:0/4205913274 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fe7ec003970 con 0x55caa96b2330 2024-05-09T08:58:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7fb38f740 1 -- 172.21.15.98:0/4205913274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7e00317b0 con 0x55caa96b2330 2024-05-09T08:58:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7d9ffb640 1 -- 172.21.15.98:0/4205913274 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe7ec024050 con 0x55caa96b2330 2024-05-09T08:58:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7fb38f740 10 client.17838 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7fb38f740 1 -- 172.21.15.98:0/4205913274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55caa96cc2c0 con 0x55caa96b2330 2024-05-09T08:58:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7d9ffb640 1 -- 172.21.15.98:0/4205913274 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e00317b0 con 0x55caa96b2330 2024-05-09T08:58:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7d9ffb640 1 -- 172.21.15.98:0/4205913274 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe7ec00f660 con 0x55caa96b2330 2024-05-09T08:58:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7fb38f740 10 client.17838 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7fb38f740 10 client.17838 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:58:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7fb38f740 20 client.17838 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/4205913274 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55caa96cd790 0x55caa96cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7fb38f740 4 client.17838 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.145+0000 7fe7fb38f740 1 -- 172.21.15.98:0/4205913274 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55caa9625370 con 0x55caa96cd790 2024-05-09T08:58:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.146+0000 7fe7f6419640 1 --2- 172.21.15.98:0/4205913274 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55caa96cd790 0x55caa96cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.146+0000 7fe7f6419640 1 --2- 172.21.15.98:0/4205913274 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55caa96cd790 0x55caa96cfc10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ea7471cf5be9d71c server_cookie=a95ccc967186e1eb in_seq=0 out_seq=0 2024-05-09T08:58:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.146+0000 7fe7d9ffb640 10 client.17838 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7d9ffb640 1 -- 172.21.15.98:0/4205913274 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55caa9625370 con 0x55caa96cd790 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7d9ffb640 10 client.17838 handle_command_reply: tid=0 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7fb38f740 1 client.17838 shutdown 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7fb38f740 1 -- 172.21.15.98:0/4205913274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2330 msgr2=0x55caa96ba260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/4205913274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2330 0x55caa96ba260 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00afd0 tx=0x7fe7ec011810 comp rx=0 tx=0).stop 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7fb38f740 1 -- 172.21.15.98:0/4205913274 shutdown_connections 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/4205913274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 0x55caa96bace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/4205913274 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55caa96cd790 0x55caa96cfc10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/4205913274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2c50 0x55caa96ba7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/4205913274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2330 0x55caa96ba260 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7d9ffb640 0 client.17838 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7fb38f740 1 -- 172.21.15.98:0/4205913274 >> 172.21.15.98:0/4205913274 conn(0x55caa96ae650 msgr2=0x55caa96b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7fb38f740 1 -- 172.21.15.98:0/4205913274 shutdown_connections 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7fb38f740 1 -- 172.21.15.98:0/4205913274 wait complete. 2024-05-09T08:58:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.147+0000 7fe7fb38f740 20 client.17838 trim_cache size 0 max 16384 2024-05-09T08:58:45.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.160+0000 7fe7fb38f740 1 Processor -- start 2024-05-09T08:58:45.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.160+0000 7fe7fb38f740 1 -- start start 2024-05-09T08:58:45.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.160+0000 7fe7fb38f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96b2330 0x55caa96c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.160+0000 7fe7fb38f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2c50 0x55caa96bc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.160+0000 7fe7fb38f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96bd320 0x55caa96d54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.160+0000 7fe7fb38f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f01aedf0 con 0x55caa96bd320 2024-05-09T08:58:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.160+0000 7fe7fb38f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f01b0930 con 0x55caa96b2c50 2024-05-09T08:58:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.160+0000 7fe7fb38f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f01af4c0 con 0x55caa96b2330 2024-05-09T08:58:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.160+0000 7fe7f5c18640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96bd320 0x55caa96d54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.160+0000 7fe7f5c18640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96bd320 0x55caa96d54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53882/0 (socket says 172.21.15.98:53882) 2024-05-09T08:58:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.160+0000 7fe7f5c18640 1 -- 172.21.15.98:0/1079573918 learned_addr learned my addr 172.21.15.98:0/1079573918 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.161+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/1079573918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96b2330 0x55caa96c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.161+0000 7fe7f6419640 1 --2- 172.21.15.98:0/1079573918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2c50 0x55caa96bc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.161+0000 7fe7f5c18640 1 -- 172.21.15.98:0/1079573918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96b2330 msgr2=0x55caa96c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.161+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/1079573918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96b2330 0x55caa96c8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.161+0000 7fe7f5c18640 1 -- 172.21.15.98:0/1079573918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2c50 msgr2=0x55caa96bc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.161+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/1079573918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2c50 0x55caa96bc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.161+0000 7fe7f5c18640 1 -- 172.21.15.98:0/1079573918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caa969fd20 con 0x55caa96bd320 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.161+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/1079573918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96bd320 0x55caa96d54e0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00a820 tx=0x7fe7ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c623236cc14c1e97 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.161+0000 7fe7dbfff640 1 -- 172.21.15.98:0/1079573918 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec02d020 con 0x55caa96bd320 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.161+0000 7fe7dbfff640 1 -- 172.21.15.98:0/1079573918 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7ec018030 con 0x55caa96bd320 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.161+0000 7fe7dbfff640 1 -- 172.21.15.98:0/1079573918 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec01da90 con 0x55caa96bd320 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 -- 172.21.15.98:0/1079573918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96bd320 msgr2=0x55caa96d54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/1079573918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96bd320 0x55caa96d54e0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00a820 tx=0x7fe7ec004a40 comp rx=0 tx=0).stop 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 -- 172.21.15.98:0/1079573918 shutdown_connections 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/1079573918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96bd320 0x55caa96d54e0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/1079573918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2c50 0x55caa96bc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/1079573918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96b2330 0x55caa96c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 -- 172.21.15.98:0/1079573918 >> 172.21.15.98:0/1079573918 conn(0x55caa96ae650 msgr2=0x55caa96aea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 -- 172.21.15.98:0/1079573918 shutdown_connections 2024-05-09T08:58:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 -- 172.21.15.98:0/1079573918 wait complete. 2024-05-09T08:58:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 -- start start 2024-05-09T08:58:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 Processor -- start 2024-05-09T08:58:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2330 0x55caa96c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2c50 0x55caa96b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 0x55caa96b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f01b0260 con 0x55caa96b2c50 2024-05-09T08:58:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f01afb90 con 0x55caa96b2330 2024-05-09T08:58:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f01acdf0 con 0x55caa96bd320 2024-05-09T08:58:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7fb38f740 10 client.0 _ll_register_callbacks cb 0x7fe7ec00a770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7f6419640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2c50 0x55caa96b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7f6419640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2c50 0x55caa96b6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53894/0 (socket says 172.21.15.98:53894) 2024-05-09T08:58:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7f6419640 1 -- 172.21.15.98:0/785372934 learned_addr learned my addr 172.21.15.98:0/785372934 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.162+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/785372934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 0x55caa96b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/785372934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2330 0x55caa96c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7f6419640 1 -- 172.21.15.98:0/785372934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 msgr2=0x55caa96b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7f6419640 1 --2- 172.21.15.98:0/785372934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 0x55caa96b6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7f6419640 1 -- 172.21.15.98:0/785372934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2330 msgr2=0x55caa96c5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7f6419640 1 --2- 172.21.15.98:0/785372934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2330 0x55caa96c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7f6419640 1 -- 172.21.15.98:0/785372934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55caa96b7660 con 0x55caa96b2c50 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7f6419640 1 --2- 172.21.15.98:0/785372934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2c50 0x55caa96b6ab0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0005400 tx=0x7fe7e00052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe3c57795e384c8a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7d9ffb640 1 -- 172.21.15.98:0/785372934 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0050070 con 0x55caa96b2c50 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7d9ffb640 1 -- 172.21.15.98:0/785372934 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fe7e004cdc0 con 0x55caa96b2c50 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7fb38f740 1 -- 172.21.15.98:0/785372934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7ec018030 con 0x55caa96b2c50 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7d9ffb640 1 -- 172.21.15.98:0/785372934 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe7e005c050 con 0x55caa96b2c50 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7fb38f740 10 client.17844 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7fb38f740 1 -- 172.21.15.98:0/785372934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55caa9733c70 con 0x55caa96b2c50 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7d9ffb640 1 -- 172.21.15.98:0/785372934 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7ec018030 con 0x55caa96b2c50 2024-05-09T08:58:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7d9ffb640 1 -- 172.21.15.98:0/785372934 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe7e004c330 con 0x55caa96b2c50 2024-05-09T08:58:45.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7fb38f740 10 client.17844 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:45.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7fb38f740 10 client.17844 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:58:45.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7fb38f740 20 client.17844 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:45.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/785372934 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55caa9733fd0 0x55caa97363f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7fb38f740 4 client.17844 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:58:45.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.163+0000 7fe7fb38f740 1 -- 172.21.15.98:0/785372934 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55caa961f6c0 con 0x55caa9733fd0 2024-05-09T08:58:45.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/785372934 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55caa9733fd0 0x55caa97363f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7d9ffb640 10 client.17844 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:58:45.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7f5c18640 1 --2- 172.21.15.98:0/785372934 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55caa9733fd0 0x55caa97363f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4b727994e34fb80f server_cookie=a52fecf7c61fdcac in_seq=0 out_seq=0 2024-05-09T08:58:45.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7d9ffb640 1 -- 172.21.15.98:0/785372934 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55caa961f6c0 con 0x55caa9733fd0 2024-05-09T08:58:45.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7d9ffb640 10 client.17844 handle_command_reply: tid=0 2024-05-09T08:58:45.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7fb38f740 1 client.17844 shutdown 2024-05-09T08:58:45.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7fb38f740 1 -- 172.21.15.98:0/785372934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2c50 msgr2=0x55caa96b6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/785372934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2c50 0x55caa96b6ab0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0005400 tx=0x7fe7e00052d0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7fb38f740 1 -- 172.21.15.98:0/785372934 shutdown_connections 2024-05-09T08:58:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/785372934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55caa96bd320 0x55caa96b6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/785372934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55caa96b2c50 0x55caa96b6ab0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/785372934 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55caa9733fd0 0x55caa97363f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7fb38f740 1 --2- 172.21.15.98:0/785372934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55caa96b2330 0x55caa96c5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7d9ffb640 0 client.17844 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:58:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7fb38f740 1 -- 172.21.15.98:0/785372934 >> 172.21.15.98:0/785372934 conn(0x55caa96ae650 msgr2=0x55caa96baab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7fb38f740 1 -- 172.21.15.98:0/785372934 shutdown_connections 2024-05-09T08:58:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7fb38f740 1 -- 172.21.15.98:0/785372934 wait complete. 2024-05-09T08:58:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.164+0000 7fe7fb38f740 20 client.17844 trim_cache size 0 max 16384 2024-05-09T08:58:45.171 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:58:45.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.165+0000 7fe7f867d640 1 -- 172.21.15.98:0/1970289431 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe7dc0a2d00 msgr2=0x7fe7dc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.165+0000 7fe7f867d640 1 --2- 172.21.15.98:0/1970289431 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe7dc0a2d00 0x7fe7dc0a51c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe7e809b270 tx=0x7fe7e806d2a0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.165+0000 7fe7f867d640 1 -- 172.21.15.98:0/1970289431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7f010be00 msgr2=0x7fe7f019c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.165+0000 7fe7f867d640 1 --2- 172.21.15.98:0/1970289431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7f010be00 0x7fe7f019c9c0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800d750 tx=0x7fe7e800dc10 comp rx=0 tx=0).stop 2024-05-09T08:58:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.165+0000 7fe7f867d640 1 -- 172.21.15.98:0/1970289431 shutdown_connections 2024-05-09T08:58:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.165+0000 7fe7f867d640 1 --2- 172.21.15.98:0/1970289431 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe7dc0a2d00 0x7fe7dc0a51c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.165+0000 7fe7f867d640 1 --2- 172.21.15.98:0/1970289431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7f010be00 0x7fe7f019c9c0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.165+0000 7fe7f867d640 1 --2- 172.21.15.98:0/1970289431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f01033e0 0x7fe7f0198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.165+0000 7fe7f867d640 1 --2- 172.21.15.98:0/1970289431 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7f0102ac0 0x7fe7f0198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.165+0000 7fe7f867d640 1 -- 172.21.15.98:0/1970289431 >> 172.21.15.98:0/1970289431 conn(0x7fe7f0076ab0 msgr2=0x7fe7f00fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.165+0000 7fe7f867d640 1 -- 172.21.15.98:0/1970289431 shutdown_connections 2024-05-09T08:58:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.165+0000 7fe7f867d640 1 -- 172.21.15.98:0/1970289431 wait complete. 2024-05-09T08:58:45.174 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:44 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3905185594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:58:45.176 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:58:45.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:44 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3905185594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:58:45.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.280+0000 7fadf3238640 1 -- 172.21.15.98:0/1542007184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadec100a80 msgr2=0x7fadec100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.280+0000 7fadf3238640 1 --2- 172.21.15.98:0/1542007184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadec100a80 0x7fadec100ee0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fadd8009900 tx=0x7fadd802a410 comp rx=0 tx=0).stop 2024-05-09T08:58:45.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.280+0000 7fadf3238640 1 -- 172.21.15.98:0/1542007184 shutdown_connections 2024-05-09T08:58:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.280+0000 7fadf3238640 1 --2- 172.21.15.98:0/1542007184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadec101420 0x7fadec10d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.280+0000 7fadf3238640 1 --2- 172.21.15.98:0/1542007184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadec100a80 0x7fadec100ee0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.280+0000 7fadf3238640 1 --2- 172.21.15.98:0/1542007184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fadec106780 0x7fadec106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.280+0000 7fadf3238640 1 -- 172.21.15.98:0/1542007184 >> 172.21.15.98:0/1542007184 conn(0x7fadec076530 msgr2=0x7fadec076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.280+0000 7fadf3238640 1 -- 172.21.15.98:0/1542007184 shutdown_connections 2024-05-09T08:58:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.280+0000 7fadf3238640 1 -- 172.21.15.98:0/1542007184 wait complete. 2024-05-09T08:58:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.280+0000 7fadf3238640 1 Processor -- start 2024-05-09T08:58:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.280+0000 7fadf3238640 1 -- start start 2024-05-09T08:58:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf3238640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fadec100a80 0x7fadec1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf3238640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadec101420 0x7fadec198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf3238640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadec106780 0x7fadec19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf3238640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fadec111bc0 con 0x7fadec106780 2024-05-09T08:58:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf3238640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fadec111a40 con 0x7fadec100a80 2024-05-09T08:58:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf3238640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fadec111d40 con 0x7fadec101420 2024-05-09T08:58:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf17d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadec106780 0x7fadec19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf17d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadec106780 0x7fadec19c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53910/0 (socket says 172.21.15.98:53910) 2024-05-09T08:58:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf17d5640 1 -- 172.21.15.98:0/3652470960 learned_addr learned my addr 172.21.15.98:0/3652470960 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf0fd4640 1 --2- 172.21.15.98:0/3652470960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fadec100a80 0x7fadec1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadebfff640 1 --2- 172.21.15.98:0/3652470960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadec101420 0x7fadec198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf17d5640 1 -- 172.21.15.98:0/3652470960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadec101420 msgr2=0x7fadec198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf17d5640 1 --2- 172.21.15.98:0/3652470960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadec101420 0x7fadec198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf17d5640 1 -- 172.21.15.98:0/3652470960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fadec100a80 msgr2=0x7fadec1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf17d5640 1 --2- 172.21.15.98:0/3652470960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fadec100a80 0x7fadec1980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf17d5640 1 -- 172.21.15.98:0/3652470960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadec19d060 con 0x7fadec106780 2024-05-09T08:58:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf17d5640 1 --2- 172.21.15.98:0/3652470960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadec106780 0x7fadec19c930 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fade400cad0 tx=0x7fade400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fade9ffb640 1 -- 172.21.15.98:0/3652470960 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade400ae20 con 0x7fadec106780 2024-05-09T08:58:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf3238640 1 -- 172.21.15.98:0/3652470960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadec19d350 con 0x7fadec106780 2024-05-09T08:58:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fade9ffb640 1 -- 172.21.15.98:0/3652470960 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fade4005020 con 0x7fadec106780 2024-05-09T08:58:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fade9ffb640 1 -- 172.21.15.98:0/3652470960 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade400f6f0 con 0x7fadec106780 2024-05-09T08:58:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.281+0000 7fadf3238640 1 -- 172.21.15.98:0/3652470960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadec1a4be0 con 0x7fadec106780 2024-05-09T08:58:45.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fade9ffb640 1 -- 172.21.15.98:0/3652470960 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fade40051d0 con 0x7fadec106780 2024-05-09T08:58:45.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fadf6029740 1 Processor -- start 2024-05-09T08:58:45.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fadf6029740 1 -- start start 2024-05-09T08:58:45.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fadf6029740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25e330 0x555f9e25e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fadf6029740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 0x555f9e268de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fadf6029740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e269320 0x555f9e26b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fadf6029740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555f9e1c3c40 con 0x555f9e25e330 2024-05-09T08:58:45.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fadf6029740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555f9e16b580 con 0x555f9e269320 2024-05-09T08:58:45.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fadf6029740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555f9e1bfc00 con 0x555f9e25ec50 2024-05-09T08:58:45.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fade9ffb640 1 --2- 172.21.15.98:0/3652470960 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faddc0a2d00 0x7faddc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fadebfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25e330 0x555f9e25e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fadebfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25e330 0x555f9e25e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53914/0 (socket says 172.21.15.98:53914) 2024-05-09T08:58:45.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fadebfff640 1 -- 172.21.15.98:0/3770668017 learned_addr learned my addr 172.21.15.98:0/3770668017 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:45.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fadf17d5640 1 --2- 172.21.15.98:0/3652470960 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faddc0a2d00 0x7faddc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fade9ffb640 1 -- 172.21.15.98:0/3652470960 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fade40cf920 con 0x7fadec106780 2024-05-09T08:58:45.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fadf0fd4640 1 --2- 172.21.15.98:0/3770668017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 0x555f9e268de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.284+0000 7fadebfff640 1 --2- 172.21.15.98:0/3770668017 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e269320 0x555f9e26b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.285+0000 7fadebfff640 1 -- 172.21.15.98:0/3770668017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 msgr2=0x555f9e268de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.285+0000 7fadebfff640 1 --2- 172.21.15.98:0/3770668017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 0x555f9e268de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.285+0000 7fadebfff640 1 -- 172.21.15.98:0/3770668017 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e269320 msgr2=0x555f9e26b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.285+0000 7fadebfff640 1 --2- 172.21.15.98:0/3770668017 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e269320 0x555f9e26b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.285+0000 7fadebfff640 1 -- 172.21.15.98:0/3770668017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f9ded73f0 con 0x555f9e25e330 2024-05-09T08:58:45.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.285+0000 7fadebfff640 1 --2- 172.21.15.98:0/3770668017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25e330 0x555f9e25e710 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fadd802a930 tx=0x7fadd8005d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ce8275e8751f89 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.285+0000 7fadf17d5640 1 --2- 172.21.15.98:0/3652470960 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faddc0a2d00 0x7faddc0a51c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fade40ccd10 tx=0x7fade4005960 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.285+0000 7fadd6ffd640 1 -- 172.21.15.98:0/3770668017 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8043020 con 0x555f9e25e330 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.285+0000 7fadd6ffd640 1 -- 172.21.15.98:0/3770668017 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadd80318b0 con 0x555f9e25e330 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.285+0000 7fadd6ffd640 1 -- 172.21.15.98:0/3770668017 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8035050 con 0x555f9e25e330 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 -- 172.21.15.98:0/3770668017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25e330 msgr2=0x555f9e25e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 --2- 172.21.15.98:0/3770668017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25e330 0x555f9e25e710 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fadd802a930 tx=0x7fadd8005d70 comp rx=0 tx=0).stop 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 -- 172.21.15.98:0/3770668017 shutdown_connections 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 --2- 172.21.15.98:0/3770668017 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e269320 0x555f9e26b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 --2- 172.21.15.98:0/3770668017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 0x555f9e268de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 --2- 172.21.15.98:0/3770668017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25e330 0x555f9e25e710 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 -- 172.21.15.98:0/3770668017 >> 172.21.15.98:0/3770668017 conn(0x555f9e25a650 msgr2=0x555f9e25aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 -- 172.21.15.98:0/3770668017 shutdown_connections 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 -- 172.21.15.98:0/3770668017 wait complete. 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 -- start start 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 Processor -- start 2024-05-09T08:58:45.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e25e330 0x555f9e266260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 0x555f9e2667a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e269320 0x555f9e266ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555f9e26c0b0 con 0x555f9e269320 2024-05-09T08:58:45.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555f9e17ec40 con 0x555f9e25e330 2024-05-09T08:58:45.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555f9e26c230 con 0x555f9e25ec50 2024-05-09T08:58:45.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.286+0000 7fadf6029740 10 client.0 _ll_register_callbacks cb 0x555f9ded6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:45.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadebfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e269320 0x555f9e266ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadebfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e269320 0x555f9e266ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53928/0 (socket says 172.21.15.98:53928) 2024-05-09T08:58:45.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadebfff640 1 -- 172.21.15.98:0/3455249787 learned_addr learned my addr 172.21.15.98:0/3455249787 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:45.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadf0fd4640 1 --2- 172.21.15.98:0/3455249787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 0x555f9e2667a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadebfff640 1 --2- 172.21.15.98:0/3455249787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e25e330 0x555f9e266260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadebfff640 1 -- 172.21.15.98:0/3455249787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 msgr2=0x555f9e2667a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadebfff640 1 --2- 172.21.15.98:0/3455249787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 0x555f9e2667a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadebfff640 1 -- 172.21.15.98:0/3455249787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e25e330 msgr2=0x555f9e266260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadebfff640 1 --2- 172.21.15.98:0/3455249787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e25e330 0x555f9e266260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadebfff640 1 -- 172.21.15.98:0/3455249787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f9dfa17d0 con 0x555f9e269320 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadebfff640 1 --2- 172.21.15.98:0/3455249787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e269320 0x555f9e266ce0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fadd8005c40 tx=0x7fadd8005490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6d555400476350 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadcbfff640 1 -- 172.21.15.98:0/3455249787 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8035070 con 0x555f9e269320 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadcbfff640 1 -- 172.21.15.98:0/3455249787 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fadd8004260 con 0x555f9e269320 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadf6029740 1 -- 172.21.15.98:0/3455249787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadd80318b0 con 0x555f9e269320 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadcbfff640 1 -- 172.21.15.98:0/3455249787 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fadd8058050 con 0x555f9e269320 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadf6029740 10 client.17856 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadf6029740 1 -- 172.21.15.98:0/3455249787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x555f9e2782a0 con 0x555f9e269320 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.287+0000 7fadcbfff640 1 -- 172.21.15.98:0/3455249787 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadd80318b0 con 0x555f9e269320 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.288+0000 7fadcbfff640 1 -- 172.21.15.98:0/3455249787 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fadd8003790 con 0x555f9e269320 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.288+0000 7fadf6029740 10 client.17856 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.288+0000 7fadf6029740 10 client.17856 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.288+0000 7fadf6029740 20 client.17856 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.288+0000 7fadf6029740 1 --2- 172.21.15.98:0/3455249787 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555f9e2795f0 0x555f9e27b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.288+0000 7fadf6029740 4 client.17856 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.288+0000 7fadf6029740 1 -- 172.21.15.98:0/3455249787 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f9e1d1370 con 0x555f9e2795f0 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.288+0000 7fadf0fd4640 1 --2- 172.21.15.98:0/3455249787 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555f9e2795f0 0x555f9e27b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.288+0000 7fadcbfff640 10 client.17856 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.288+0000 7fadf0fd4640 1 --2- 172.21.15.98:0/3455249787 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555f9e2795f0 0x555f9e27b9e0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e9d18335ccc0c80 server_cookie=6b7951a7a5dc0767 in_seq=0 out_seq=0 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.289+0000 7fadcbfff640 1 -- 172.21.15.98:0/3455249787 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f9e1d1370 con 0x555f9e2795f0 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.289+0000 7fadcbfff640 10 client.17856 handle_command_reply: tid=0 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.289+0000 7fadf6029740 1 client.17856 shutdown 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.289+0000 7fadf6029740 1 -- 172.21.15.98:0/3455249787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e269320 msgr2=0x555f9e266ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.289+0000 7fadf6029740 1 --2- 172.21.15.98:0/3455249787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e269320 0x555f9e266ce0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fadd8005c40 tx=0x7fadd8005490 comp rx=0 tx=0).stop 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.289+0000 7fadf6029740 1 -- 172.21.15.98:0/3455249787 shutdown_connections 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.289+0000 7fadf6029740 1 --2- 172.21.15.98:0/3455249787 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555f9e2795f0 0x555f9e27b9e0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.289+0000 7fadf6029740 1 --2- 172.21.15.98:0/3455249787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e269320 0x555f9e266ce0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.289+0000 7fadf6029740 1 --2- 172.21.15.98:0/3455249787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 0x555f9e2667a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.289+0000 7fadcbfff640 0 client.17856 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:45.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.289+0000 7fadf6029740 1 --2- 172.21.15.98:0/3455249787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e25e330 0x555f9e266260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.289+0000 7fadf6029740 1 -- 172.21.15.98:0/3455249787 >> 172.21.15.98:0/3455249787 conn(0x555f9e25a650 msgr2=0x555f9e260350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.289+0000 7fadf6029740 1 -- 172.21.15.98:0/3455249787 shutdown_connections 2024-05-09T08:58:45.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.289+0000 7fadf6029740 1 -- 172.21.15.98:0/3455249787 wait complete. 2024-05-09T08:58:45.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.290+0000 7fadf6029740 20 client.17856 trim_cache size 0 max 16384 2024-05-09T08:58:45.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.302+0000 7fadf6029740 1 Processor -- start 2024-05-09T08:58:45.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadf6029740 1 -- start start 2024-05-09T08:58:45.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadf6029740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25e330 0x555f9e268990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadf6029740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 0x555f9e274fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadf6029740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e269320 0x555f9e280fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadf6029740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fadec1aed70 con 0x555f9e25e330 2024-05-09T08:58:45.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadf6029740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fadec1b08b0 con 0x555f9e269320 2024-05-09T08:58:45.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadf6029740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fadec1af440 con 0x555f9e25ec50 2024-05-09T08:58:45.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadebfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25e330 0x555f9e268990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadebfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25e330 0x555f9e268990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53938/0 (socket says 172.21.15.98:53938) 2024-05-09T08:58:45.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadebfff640 1 -- 172.21.15.98:0/2703244662 learned_addr learned my addr 172.21.15.98:0/2703244662 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:45.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadf0fd4640 1 --2- 172.21.15.98:0/2703244662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 0x555f9e274fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadebfff640 1 -- 172.21.15.98:0/2703244662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 msgr2=0x555f9e274fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadebfff640 1 --2- 172.21.15.98:0/2703244662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 0x555f9e274fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadebfff640 1 -- 172.21.15.98:0/2703244662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e269320 msgr2=0x555f9e280fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadebfff640 1 --2- 172.21.15.98:0/2703244662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e269320 0x555f9e280fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadebfff640 1 -- 172.21.15.98:0/2703244662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f9e25ea80 con 0x555f9e25e330 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadebfff640 1 --2- 172.21.15.98:0/2703244662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25e330 0x555f9e268990 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fadd8006010 tx=0x7fadd8005500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0b58ce1fbf174 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadd67fc640 1 -- 172.21.15.98:0/2703244662 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8035070 con 0x555f9e25e330 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadd67fc640 1 -- 172.21.15.98:0/2703244662 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadd8004260 con 0x555f9e25e330 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.303+0000 7fadd67fc640 1 -- 172.21.15.98:0/2703244662 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8043030 con 0x555f9e25e330 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.304+0000 7fadf6029740 1 -- 172.21.15.98:0/2703244662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25e330 msgr2=0x555f9e268990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.304+0000 7fadf6029740 1 --2- 172.21.15.98:0/2703244662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25e330 0x555f9e268990 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fadd8006010 tx=0x7fadd8005500 comp rx=0 tx=0).stop 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.304+0000 7fadf6029740 1 -- 172.21.15.98:0/2703244662 shutdown_connections 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.304+0000 7fadf6029740 1 --2- 172.21.15.98:0/2703244662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e269320 0x555f9e280fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.304+0000 7fadf6029740 1 --2- 172.21.15.98:0/2703244662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25ec50 0x555f9e274fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.304+0000 7fadf6029740 1 --2- 172.21.15.98:0/2703244662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25e330 0x555f9e268990 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.304+0000 7fadf6029740 1 -- 172.21.15.98:0/2703244662 >> 172.21.15.98:0/2703244662 conn(0x555f9e25a650 msgr2=0x555f9e2597b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.304+0000 7fadf6029740 1 -- 172.21.15.98:0/2703244662 shutdown_connections 2024-05-09T08:58:45.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.304+0000 7fadf6029740 1 -- 172.21.15.98:0/2703244662 wait complete. 2024-05-09T08:58:45.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.304+0000 7fadf6029740 1 -- start start 2024-05-09T08:58:45.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.304+0000 7fadf6029740 1 Processor -- start 2024-05-09T08:58:45.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf6029740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25e330 0x555f9e2718a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf6029740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25ec50 0x555f9e26cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf6029740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e269320 0x555f9e26d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf6029740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fadec1b01e0 con 0x555f9e25ec50 2024-05-09T08:58:45.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf6029740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fadec1afb10 con 0x555f9e269320 2024-05-09T08:58:45.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf6029740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fadec1acd70 con 0x555f9e25e330 2024-05-09T08:58:45.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf6029740 10 client.0 _ll_register_callbacks cb 0x7fade00097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:45.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf0fd4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25ec50 0x555f9e26cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf0fd4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25ec50 0x555f9e26cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53952/0 (socket says 172.21.15.98:53952) 2024-05-09T08:58:45.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf0fd4640 1 -- 172.21.15.98:0/1918910814 learned_addr learned my addr 172.21.15.98:0/1918910814 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:45.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadebfff640 1 --2- 172.21.15.98:0/1918910814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25e330 0x555f9e2718a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadebfff640 1 --2- 172.21.15.98:0/1918910814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e269320 0x555f9e26d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf0fd4640 1 -- 172.21.15.98:0/1918910814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25e330 msgr2=0x555f9e2718a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf0fd4640 1 --2- 172.21.15.98:0/1918910814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25e330 0x555f9e2718a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf0fd4640 1 -- 172.21.15.98:0/1918910814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e269320 msgr2=0x555f9e26d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:45.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf0fd4640 1 --2- 172.21.15.98:0/1918910814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e269320 0x555f9e26d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf0fd4640 1 -- 172.21.15.98:0/1918910814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f9e26d950 con 0x555f9e25ec50 2024-05-09T08:58:45.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadebfff640 1 --2- 172.21.15.98:0/1918910814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25e330 0x555f9e2718a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:45.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadf0fd4640 1 --2- 172.21.15.98:0/1918910814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25ec50 0x555f9e26cc10 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fade0009e50 tx=0x7fade00096f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac77a54b144f3d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:45.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:45.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.305+0000 7fadcbfff640 1 -- 172.21.15.98:0/1918910814 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade0023070 con 0x555f9e25ec50 2024-05-09T08:58:45.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.306+0000 7fadf6029740 1 -- 172.21.15.98:0/1918910814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadd8004260 con 0x555f9e25ec50 2024-05-09T08:58:45.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.306+0000 7fadcbfff640 1 -- 172.21.15.98:0/1918910814 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fade001f3a0 con 0x555f9e25ec50 2024-05-09T08:58:45.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.306+0000 7fadcbfff640 1 -- 172.21.15.98:0/1918910814 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fade002f050 con 0x555f9e25ec50 2024-05-09T08:58:45.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.306+0000 7fadf6029740 10 client.17862 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:45.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.306+0000 7fadf6029740 1 -- 172.21.15.98:0/1918910814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x555f9e2df900 con 0x555f9e25ec50 2024-05-09T08:58:45.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.306+0000 7fadcbfff640 1 -- 172.21.15.98:0/1918910814 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadd8004260 con 0x555f9e25ec50 2024-05-09T08:58:45.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.306+0000 7fadcbfff640 1 -- 172.21.15.98:0/1918910814 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fade0028ce0 con 0x555f9e25ec50 2024-05-09T08:58:45.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.306+0000 7fadf6029740 10 client.17862 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:45.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.306+0000 7fadf6029740 10 client.17862 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:45.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.306+0000 7fadf6029740 20 client.17862 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:45.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.306+0000 7fadf6029740 1 --2- 172.21.15.98:0/1918910814 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555f9e2e0060 0x555f9e2e2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:45.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.306+0000 7fadf6029740 4 client.17862 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:58:45.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.306+0000 7fadf6029740 1 -- 172.21.15.98:0/1918910814 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555f9e1cb6c0 con 0x555f9e2e0060 2024-05-09T08:58:45.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.306+0000 7fadebfff640 1 --2- 172.21.15.98:0/1918910814 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555f9e2e0060 0x555f9e2e2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:45.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadcbfff640 10 client.17862 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:45.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadebfff640 1 --2- 172.21.15.98:0/1918910814 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555f9e2e0060 0x555f9e2e2480 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3743a43a96982465 server_cookie=8e6fa6e94eb219a5 in_seq=0 out_seq=0 2024-05-09T08:58:45.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadcbfff640 1 -- 172.21.15.98:0/1918910814 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555f9e1cb6c0 con 0x555f9e2e0060 2024-05-09T08:58:45.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadcbfff640 10 client.17862 handle_command_reply: tid=0 2024-05-09T08:58:45.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadf6029740 1 client.17862 shutdown 2024-05-09T08:58:45.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadf6029740 1 -- 172.21.15.98:0/1918910814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25ec50 msgr2=0x555f9e26cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadf6029740 1 --2- 172.21.15.98:0/1918910814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25ec50 0x555f9e26cc10 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fade0009e50 tx=0x7fade00096f0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadf6029740 1 -- 172.21.15.98:0/1918910814 shutdown_connections 2024-05-09T08:58:45.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadf6029740 1 --2- 172.21.15.98:0/1918910814 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x555f9e2e0060 0x555f9e2e2480 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadf6029740 1 --2- 172.21.15.98:0/1918910814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f9e269320 0x555f9e26d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadf6029740 1 --2- 172.21.15.98:0/1918910814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555f9e25ec50 0x555f9e26cc10 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadcbfff640 0 client.17862 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:45.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadf6029740 1 --2- 172.21.15.98:0/1918910814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555f9e25e330 0x555f9e2718a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadf6029740 1 -- 172.21.15.98:0/1918910814 >> 172.21.15.98:0/1918910814 conn(0x555f9e25a650 msgr2=0x555f9e25fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadf6029740 1 -- 172.21.15.98:0/1918910814 shutdown_connections 2024-05-09T08:58:45.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadf6029740 1 -- 172.21.15.98:0/1918910814 wait complete. 2024-05-09T08:58:45.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.307+0000 7fadf6029740 20 client.17862 trim_cache size 0 max 16384 2024-05-09T08:58:45.316 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:58:45.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.308+0000 7fadf3238640 1 -- 172.21.15.98:0/3652470960 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faddc0a2d00 msgr2=0x7faddc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.308+0000 7fadf3238640 1 --2- 172.21.15.98:0/3652470960 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faddc0a2d00 0x7faddc0a51c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fade40ccd10 tx=0x7fade4005960 comp rx=0 tx=0).stop 2024-05-09T08:58:45.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.308+0000 7fadf3238640 1 -- 172.21.15.98:0/3652470960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadec106780 msgr2=0x7fadec19c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:45.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.308+0000 7fadf3238640 1 --2- 172.21.15.98:0/3652470960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadec106780 0x7fadec19c930 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fade400cad0 tx=0x7fade400cf90 comp rx=0 tx=0).stop 2024-05-09T08:58:45.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.308+0000 7fadf3238640 1 -- 172.21.15.98:0/3652470960 shutdown_connections 2024-05-09T08:58:45.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.308+0000 7fadf3238640 1 --2- 172.21.15.98:0/3652470960 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7faddc0a2d00 0x7faddc0a51c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.308+0000 7fadf3238640 1 --2- 172.21.15.98:0/3652470960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadec106780 0x7fadec19c930 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.308+0000 7fadf3238640 1 --2- 172.21.15.98:0/3652470960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadec101420 0x7fadec198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.308+0000 7fadf3238640 1 --2- 172.21.15.98:0/3652470960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fadec100a80 0x7fadec1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:45.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.308+0000 7fadf3238640 1 -- 172.21.15.98:0/3652470960 >> 172.21.15.98:0/3652470960 conn(0x7fadec076530 msgr2=0x7fadec0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:45.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.308+0000 7fadf3238640 1 -- 172.21.15.98:0/3652470960 shutdown_connections 2024-05-09T08:58:45.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:45.308+0000 7fadf3238640 1 -- 172.21.15.98:0/3652470960 wait complete. 2024-05-09T08:58:45.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:44 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3905185594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:58:46.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:45 smithi105 ceph-mon[34498]: pgmap v252: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-05-09T08:58:46.319 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:58:46.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:58:46.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:58:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:45 smithi098 ceph-mon[30092]: pgmap v252: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-05-09T08:58:46.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:45 smithi142 ceph-mon[34625]: pgmap v252: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-05-09T08:58:46.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.438+0000 7fa80e431640 1 -- 172.21.15.98:0/2585716148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8081010d0 msgr2=0x7fa80810d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.438+0000 7fa80e431640 1 --2- 172.21.15.98:0/2585716148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8081010d0 0x7fa80810d5e0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fa800009cd0 tx=0x7fa80002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.438+0000 7fa80e431640 1 -- 172.21.15.98:0/2585716148 shutdown_connections 2024-05-09T08:58:46.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.438+0000 7fa80e431640 1 --2- 172.21.15.98:0/2585716148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8081010d0 0x7fa80810d5e0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.438+0000 7fa80e431640 1 --2- 172.21.15.98:0/2585716148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa808100710 0x7fa808100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.438+0000 7fa80e431640 1 --2- 172.21.15.98:0/2585716148 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa808106720 0x7fa808106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.438+0000 7fa80e431640 1 -- 172.21.15.98:0/2585716148 >> 172.21.15.98:0/2585716148 conn(0x7fa808077790 msgr2=0x7fa808077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:46.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.438+0000 7fa80e431640 1 -- 172.21.15.98:0/2585716148 shutdown_connections 2024-05-09T08:58:46.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.438+0000 7fa80e431640 1 -- 172.21.15.98:0/2585716148 wait complete. 2024-05-09T08:58:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.439+0000 7fa80e431640 1 Processor -- start 2024-05-09T08:58:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.439+0000 7fa80e431640 1 -- start start 2024-05-09T08:58:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.439+0000 7fa80e431640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa808100710 0x7fa808196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.439+0000 7fa80e431640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8081010d0 0x7fa808197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.439+0000 7fa80e431640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa808106720 0x7fa80819b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.439+0000 7fa80e431640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa808110b30 con 0x7fa8081010d0 2024-05-09T08:58:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.439+0000 7fa80e431640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa8081109b0 con 0x7fa808106720 2024-05-09T08:58:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.439+0000 7fa80e431640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa808110cb0 con 0x7fa808100710 2024-05-09T08:58:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.439+0000 7fa8077fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8081010d0 0x7fa808197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.439+0000 7fa8077fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8081010d0 0x7fa808197490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54956/0 (socket says 172.21.15.98:54956) 2024-05-09T08:58:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.439+0000 7fa8077fe640 1 -- 172.21.15.98:0/1675811174 learned_addr learned my addr 172.21.15.98:0/1675811174 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.440+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa808106720 0x7fa80819b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.440+0000 7fa807fff640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa808100710 0x7fa808196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.440+0000 7fa8077fe640 1 -- 172.21.15.98:0/1675811174 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa808100710 msgr2=0x7fa808196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.440+0000 7fa8077fe640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa808100710 0x7fa808196f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.440+0000 7fa8077fe640 1 -- 172.21.15.98:0/1675811174 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa808106720 msgr2=0x7fa80819b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.440+0000 7fa8077fe640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa808106720 0x7fa80819b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.440+0000 7fa8077fe640 1 -- 172.21.15.98:0/1675811174 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa80819bef0 con 0x7fa8081010d0 2024-05-09T08:58:46.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.440+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa808106720 0x7fa80819b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:46.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.440+0000 7fa8077fe640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8081010d0 0x7fa808197490 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00d770 tx=0x7fa7fc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:46.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.440+0000 7fa8057fa640 1 -- 172.21.15.98:0/1675811174 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc017070 con 0x7fa8081010d0 2024-05-09T08:58:46.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.440+0000 7fa80e431640 1 -- 172.21.15.98:0/1675811174 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa80819c1e0 con 0x7fa8081010d0 2024-05-09T08:58:46.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.440+0000 7fa8057fa640 1 -- 172.21.15.98:0/1675811174 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7fc005500 con 0x7fa8081010d0 2024-05-09T08:58:46.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.440+0000 7fa8057fa640 1 -- 172.21.15.98:0/1675811174 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc004e00 con 0x7fa8081010d0 2024-05-09T08:58:46.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.440+0000 7fa80e431640 1 -- 172.21.15.98:0/1675811174 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8081a3a70 con 0x7fa8081010d0 2024-05-09T08:58:46.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.442+0000 7fa8057fa640 1 -- 172.21.15.98:0/1675811174 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa7fc004fa0 con 0x7fa8081010d0 2024-05-09T08:58:46.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa811219740 1 Processor -- start 2024-05-09T08:58:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa811219740 1 -- start start 2024-05-09T08:58:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa811219740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c00b0 0x55cec78be3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa811219740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78bee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa8057fa640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa7f00a2d00 0x7fa7f00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa811219740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c9740 0x55cec78cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa811219740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cec7823990 con 0x55cec78be9c0 2024-05-09T08:58:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa811219740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cec77cb5b0 con 0x55cec78c00b0 2024-05-09T08:58:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa811219740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cec781f950 con 0x55cec78c9740 2024-05-09T08:58:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa807fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78bee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa807fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78bee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54962/0 (socket says 172.21.15.98:54962) 2024-05-09T08:58:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa807fff640 1 -- 172.21.15.98:0/3960856048 learned_addr learned my addr 172.21.15.98:0/3960856048 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa8077fe640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa7f00a2d00 0x7fa7f00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/3960856048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c9740 0x55cec78cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/3960856048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c00b0 0x55cec78be3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa807fff640 1 -- 172.21.15.98:0/3960856048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c9740 msgr2=0x55cec78cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa807fff640 1 --2- 172.21.15.98:0/3960856048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c9740 0x55cec78cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa807fff640 1 -- 172.21.15.98:0/3960856048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c00b0 msgr2=0x55cec78be3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa807fff640 1 --2- 172.21.15.98:0/3960856048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c00b0 0x55cec78be3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa8057fa640 1 -- 172.21.15.98:0/1675811174 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fa7fc0cf390 con 0x7fa8081010d0 2024-05-09T08:58:46.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa807fff640 1 -- 172.21.15.98:0/3960856048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cec75ff7e0 con 0x55cec78be9c0 2024-05-09T08:58:46.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.443+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/3960856048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c00b0 0x55cec78be3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:46.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.444+0000 7fa807fff640 1 --2- 172.21.15.98:0/3960856048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78bee40 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800bac0 tx=0x7fa7f800bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1445d8db9a5366c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:46.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.444+0000 7fa8077fe640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa7f00a2d00 0x7fa7f00a51c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc066490 tx=0x7fa7fc06f040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:46.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.444+0000 7fa7ee7fc640 1 -- 172.21.15.98:0/3960856048 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8017420 con 0x55cec78be9c0 2024-05-09T08:58:46.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.444+0000 7fa7ee7fc640 1 -- 172.21.15.98:0/3960856048 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7f8017aa0 con 0x55cec78be9c0 2024-05-09T08:58:46.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.444+0000 7fa7ee7fc640 1 -- 172.21.15.98:0/3960856048 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f800fa50 con 0x55cec78be9c0 2024-05-09T08:58:46.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.444+0000 7fa811219740 1 -- 172.21.15.98:0/3960856048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 msgr2=0x55cec78bee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.444+0000 7fa811219740 1 --2- 172.21.15.98:0/3960856048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78bee40 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800bac0 tx=0x7fa7f800bf80 comp rx=0 tx=0).stop 2024-05-09T08:58:46.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.444+0000 7fa811219740 1 -- 172.21.15.98:0/3960856048 shutdown_connections 2024-05-09T08:58:46.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.444+0000 7fa811219740 1 --2- 172.21.15.98:0/3960856048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c9740 0x55cec78cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.444+0000 7fa811219740 1 --2- 172.21.15.98:0/3960856048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78bee40 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.444+0000 7fa811219740 1 --2- 172.21.15.98:0/3960856048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c00b0 0x55cec78be3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.444+0000 7fa811219740 1 -- 172.21.15.98:0/3960856048 >> 172.21.15.98:0/3960856048 conn(0x55cec78870a0 msgr2=0x55cec78baf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:46.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.445+0000 7fa811219740 1 -- 172.21.15.98:0/3960856048 shutdown_connections 2024-05-09T08:58:46.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.445+0000 7fa811219740 1 -- 172.21.15.98:0/3960856048 wait complete. 2024-05-09T08:58:46.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.445+0000 7fa811219740 1 -- start start 2024-05-09T08:58:46.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.445+0000 7fa811219740 1 Processor -- start 2024-05-09T08:58:46.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.445+0000 7fa811219740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.445+0000 7fa811219740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c00b0 0x55cec78c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.445+0000 7fa811219740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c9740 0x55cec78c7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.445+0000 7fa811219740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cec78cc5d0 con 0x55cec78be9c0 2024-05-09T08:58:46.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.445+0000 7fa811219740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cec77deb40 con 0x55cec78c9740 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.445+0000 7fa811219740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cec78cc750 con 0x55cec78c00b0 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.445+0000 7fa811219740 10 client.0 _ll_register_callbacks cb 0x55cec77d30f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.445+0000 7fa80c9ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.445+0000 7fa80c9ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54970/0 (socket says 172.21.15.98:54970) 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.445+0000 7fa80c9ce640 1 -- 172.21.15.98:0/11870121 learned_addr learned my addr 172.21.15.98:0/11870121 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.446+0000 7fa807fff640 1 --2- 172.21.15.98:0/11870121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c00b0 0x55cec78c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.446+0000 7fa80c9ce640 1 -- 172.21.15.98:0/11870121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c00b0 msgr2=0x55cec78c6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.446+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/11870121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c00b0 0x55cec78c6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.446+0000 7fa80c9ce640 1 -- 172.21.15.98:0/11870121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c9740 msgr2=0x55cec78c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.446+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/11870121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c9740 0x55cec78c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.446+0000 7fa80c9ce640 1 -- 172.21.15.98:0/11870121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cec765ff70 con 0x55cec78be9c0 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.446+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/11870121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78c65d0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fa800009cd0 tx=0x7fa800031cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709aac2332cd66dd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.446+0000 7fa7e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.446+0000 7fa7e77fe640 1 -- 172.21.15.98:0/11870121 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa800003d80 con 0x55cec78be9c0 2024-05-09T08:58:46.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.446+0000 7fa7e77fe640 1 -- 172.21.15.98:0/11870121 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fa800004040 con 0x55cec78be9c0 2024-05-09T08:58:46.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.446+0000 7fa811219740 1 -- 172.21.15.98:0/11870121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7f8017aa0 con 0x55cec78be9c0 2024-05-09T08:58:46.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.446+0000 7fa7e77fe640 1 -- 172.21.15.98:0/11870121 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa800004240 con 0x55cec78be9c0 2024-05-09T08:58:46.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.446+0000 7fa811219740 10 client.17874 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:46.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.446+0000 7fa811219740 1 -- 172.21.15.98:0/11870121 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cec74f7170 con 0x55cec78be9c0 2024-05-09T08:58:46.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.447+0000 7fa7e77fe640 1 -- 172.21.15.98:0/11870121 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7f8017aa0 con 0x55cec78be9c0 2024-05-09T08:58:46.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.447+0000 7fa7e77fe640 1 -- 172.21.15.98:0/11870121 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa800030070 con 0x55cec78be9c0 2024-05-09T08:58:46.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.447+0000 7fa811219740 10 client.17874 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:46.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.447+0000 7fa811219740 10 client.17874 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:46.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.447+0000 7fa811219740 20 client.17874 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:46.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.447+0000 7fa811219740 1 --2- 172.21.15.98:0/11870121 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cec78d9750 0x55cec78dbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.447+0000 7fa811219740 4 client.17874 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:46.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.447+0000 7fa811219740 1 -- 172.21.15.98:0/11870121 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cec78310c0 con 0x55cec78d9750 2024-05-09T08:58:46.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.447+0000 7fa807fff640 1 --2- 172.21.15.98:0/11870121 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cec78d9750 0x55cec78dbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.447+0000 7fa807fff640 1 --2- 172.21.15.98:0/11870121 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cec78d9750 0x55cec78dbb80 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7750735228464ffb server_cookie=5fd6dc0ca6b7abaa in_seq=0 out_seq=0 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.447+0000 7fa7e77fe640 10 client.17874 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.448+0000 7fa7e77fe640 1 -- 172.21.15.98:0/11870121 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cec78310c0 con 0x55cec78d9750 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.448+0000 7fa7e77fe640 10 client.17874 handle_command_reply: tid=0 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.448+0000 7fa811219740 1 client.17874 shutdown 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.449+0000 7fa811219740 1 -- 172.21.15.98:0/11870121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 msgr2=0x55cec78c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.449+0000 7fa811219740 1 --2- 172.21.15.98:0/11870121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78c65d0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fa800009cd0 tx=0x7fa800031cc0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.449+0000 7fa811219740 1 -- 172.21.15.98:0/11870121 shutdown_connections 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.449+0000 7fa811219740 1 --2- 172.21.15.98:0/11870121 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cec78d9750 0x55cec78dbb80 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.449+0000 7fa811219740 1 --2- 172.21.15.98:0/11870121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c9740 0x55cec78c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.449+0000 7fa7e77fe640 0 client.17874 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.449+0000 7fa811219740 1 --2- 172.21.15.98:0/11870121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c00b0 0x55cec78c6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.449+0000 7fa811219740 1 --2- 172.21.15.98:0/11870121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78c65d0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.449+0000 7fa811219740 1 -- 172.21.15.98:0/11870121 >> 172.21.15.98:0/11870121 conn(0x55cec78870a0 msgr2=0x55cec78c0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.449+0000 7fa811219740 1 -- 172.21.15.98:0/11870121 shutdown_connections 2024-05-09T08:58:46.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.449+0000 7fa811219740 1 -- 172.21.15.98:0/11870121 wait complete. 2024-05-09T08:58:46.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.449+0000 7fa811219740 20 client.17874 trim_cache size 0 max 16384 2024-05-09T08:58:46.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.466+0000 7fa811219740 1 Processor -- start 2024-05-09T08:58:46.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.466+0000 7fa811219740 1 -- start start 2024-05-09T08:58:46.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.466+0000 7fa811219740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78d5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.466+0000 7fa811219740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c00b0 0x55cec78b9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.466+0000 7fa811219740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c9740 0x55cec78e1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.466+0000 7fa811219740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8081acb90 con 0x55cec78be9c0 2024-05-09T08:58:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.466+0000 7fa811219740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa8081ae6d0 con 0x55cec78c9740 2024-05-09T08:58:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.466+0000 7fa811219740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa8081ad260 con 0x55cec78c00b0 2024-05-09T08:58:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.467+0000 7fa80c9ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78d5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.467+0000 7fa80c9ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78d5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54980/0 (socket says 172.21.15.98:54980) 2024-05-09T08:58:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.467+0000 7fa80c9ce640 1 -- 172.21.15.98:0/2199505421 learned_addr learned my addr 172.21.15.98:0/2199505421 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.467+0000 7fa807fff640 1 --2- 172.21.15.98:0/2199505421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c00b0 0x55cec78b9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.467+0000 7fa80c9ce640 1 -- 172.21.15.98:0/2199505421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c00b0 msgr2=0x55cec78b9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.467+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/2199505421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c00b0 0x55cec78b9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.467+0000 7fa80c9ce640 1 -- 172.21.15.98:0/2199505421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c9740 msgr2=0x55cec78e1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.467+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/2199505421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c9740 0x55cec78e1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.467+0000 7fa80c9ce640 1 -- 172.21.15.98:0/2199505421 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cec78bb130 con 0x55cec78be9c0 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.467+0000 7fa807fff640 1 --2- 172.21.15.98:0/2199505421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c00b0 0x55cec78b9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.467+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/2199505421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78d5500 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fa80002ae00 tx=0x7fa8000093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf749a6762e3a30 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.467+0000 7fa7edffb640 1 -- 172.21.15.98:0/2199505421 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa800030eb0 con 0x55cec78be9c0 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.467+0000 7fa7edffb640 1 -- 172.21.15.98:0/2199505421 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa800043070 con 0x55cec78be9c0 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.467+0000 7fa7edffb640 1 -- 172.21.15.98:0/2199505421 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa800030920 con 0x55cec78be9c0 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.468+0000 7fa811219740 1 -- 172.21.15.98:0/2199505421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 msgr2=0x55cec78d5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.468+0000 7fa811219740 1 --2- 172.21.15.98:0/2199505421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78d5500 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fa80002ae00 tx=0x7fa8000093e0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.468+0000 7fa811219740 1 -- 172.21.15.98:0/2199505421 shutdown_connections 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.468+0000 7fa811219740 1 --2- 172.21.15.98:0/2199505421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c9740 0x55cec78e1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.468+0000 7fa811219740 1 --2- 172.21.15.98:0/2199505421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78c00b0 0x55cec78b9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.468+0000 7fa811219740 1 --2- 172.21.15.98:0/2199505421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78be9c0 0x55cec78d5500 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.468+0000 7fa811219740 1 -- 172.21.15.98:0/2199505421 >> 172.21.15.98:0/2199505421 conn(0x55cec78870a0 msgr2=0x55cec78b97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.468+0000 7fa811219740 1 -- 172.21.15.98:0/2199505421 shutdown_connections 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.468+0000 7fa811219740 1 -- 172.21.15.98:0/2199505421 wait complete. 2024-05-09T08:58:46.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.468+0000 7fa811219740 1 -- start start 2024-05-09T08:58:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.468+0000 7fa811219740 1 Processor -- start 2024-05-09T08:58:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa811219740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78be9c0 0x55cec78c3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa811219740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c00b0 0x55cec78c3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa811219740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78c9740 0x55cec793ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa811219740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8081ae000 con 0x55cec78c9740 2024-05-09T08:58:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa811219740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa8081ad930 con 0x55cec78c00b0 2024-05-09T08:58:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa811219740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa8081aab90 con 0x55cec78be9c0 2024-05-09T08:58:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa811219740 10 client.0 _ll_register_callbacks cb 0x7fa800031950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa80c9ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78c9740 0x55cec793ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa80c9ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78c9740 0x55cec793ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54992/0 (socket says 172.21.15.98:54992) 2024-05-09T08:58:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa80c9ce640 1 -- 172.21.15.98:0/2083805110 learned_addr learned my addr 172.21.15.98:0/2083805110 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa807fff640 1 --2- 172.21.15.98:0/2083805110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c00b0 0x55cec78c3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/2083805110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78be9c0 0x55cec78c3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa80c9ce640 1 -- 172.21.15.98:0/2083805110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78be9c0 msgr2=0x55cec78c3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/2083805110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78be9c0 0x55cec78c3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa80c9ce640 1 -- 172.21.15.98:0/2083805110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c00b0 msgr2=0x55cec78c3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/2083805110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c00b0 0x55cec78c3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa80c9ce640 1 -- 172.21.15.98:0/2083805110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cec793f200 con 0x55cec78c9740 2024-05-09T08:58:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa807fff640 1 --2- 172.21.15.98:0/2083805110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c00b0 0x55cec78c3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa80c9ce640 1 --2- 172.21.15.98:0/2083805110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78c9740 0x55cec793ecc0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fa800031a60 tx=0x7fa8000095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5afe1e18588509 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa7e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa7e77fe640 1 -- 172.21.15.98:0/2083805110 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa800058020 con 0x55cec78c9740 2024-05-09T08:58:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa7e77fe640 1 -- 172.21.15.98:0/2083805110 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7fa800004000 con 0x55cec78c9740 2024-05-09T08:58:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa811219740 1 -- 172.21.15.98:0/2083805110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa800043070 con 0x55cec78c9740 2024-05-09T08:58:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa7e77fe640 1 -- 172.21.15.98:0/2083805110 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa8000308b0 con 0x55cec78c9740 2024-05-09T08:58:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa811219740 10 client.17880 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:46.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.469+0000 7fa811219740 1 -- 172.21.15.98:0/2083805110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cec7940310 con 0x55cec78c9740 2024-05-09T08:58:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.470+0000 7fa7e77fe640 1 -- 172.21.15.98:0/2083805110 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa800043070 con 0x55cec78c9740 2024-05-09T08:58:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.470+0000 7fa7e77fe640 1 -- 172.21.15.98:0/2083805110 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa800054a80 con 0x55cec78c9740 2024-05-09T08:58:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.470+0000 7fa811219740 10 client.17880 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.470+0000 7fa811219740 10 client.17880 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.470+0000 7fa811219740 20 client.17880 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.470+0000 7fa811219740 1 --2- 172.21.15.98:0/2083805110 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cec79407e0 0x55cec7942bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.470+0000 7fa811219740 4 client.17880 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:58:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.470+0000 7fa811219740 1 -- 172.21.15.98:0/2083805110 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cec782b410 con 0x55cec79407e0 2024-05-09T08:58:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.470+0000 7fa807fff640 1 --2- 172.21.15.98:0/2083805110 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cec79407e0 0x55cec7942bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.470+0000 7fa7e77fe640 10 client.17880 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.470+0000 7fa807fff640 1 --2- 172.21.15.98:0/2083805110 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cec79407e0 0x55cec7942bd0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c996badc6748fe1 server_cookie=501e485a27c850cb in_seq=0 out_seq=0 2024-05-09T08:58:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.471+0000 7fa7e77fe640 1 -- 172.21.15.98:0/2083805110 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cec782b410 con 0x55cec79407e0 2024-05-09T08:58:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.471+0000 7fa7e77fe640 10 client.17880 handle_command_reply: tid=0 2024-05-09T08:58:46.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.471+0000 7fa811219740 1 client.17880 shutdown 2024-05-09T08:58:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa811219740 1 -- 172.21.15.98:0/2083805110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78c9740 msgr2=0x55cec793ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa811219740 1 --2- 172.21.15.98:0/2083805110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78c9740 0x55cec793ecc0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fa800031a60 tx=0x7fa8000095a0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa811219740 1 -- 172.21.15.98:0/2083805110 shutdown_connections 2024-05-09T08:58:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa811219740 1 --2- 172.21.15.98:0/2083805110 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cec79407e0 0x55cec7942bd0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa811219740 1 --2- 172.21.15.98:0/2083805110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cec78c9740 0x55cec793ecc0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa7e77fe640 0 client.17880 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa811219740 1 --2- 172.21.15.98:0/2083805110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cec78c00b0 0x55cec78c3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa811219740 1 --2- 172.21.15.98:0/2083805110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cec78be9c0 0x55cec78c3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa811219740 1 -- 172.21.15.98:0/2083805110 >> 172.21.15.98:0/2083805110 conn(0x55cec78870a0 msgr2=0x55cec78ba080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa811219740 1 -- 172.21.15.98:0/2083805110 shutdown_connections 2024-05-09T08:58:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa811219740 1 -- 172.21.15.98:0/2083805110 wait complete. 2024-05-09T08:58:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa811219740 20 client.17880 trim_cache size 0 max 16384 2024-05-09T08:58:46.478 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:58:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:58:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "c53a794f-b42b-4719-9e6e-fed2fa0c2102", 2024-05-09T08:58:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:58:46.478 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:58:46.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa80e431640 1 -- 172.21.15.98:0/1675811174 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa7f00a2d00 msgr2=0x7fa7f00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa80e431640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa7f00a2d00 0x7fa7f00a51c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc066490 tx=0x7fa7fc06f040 comp rx=0 tx=0).stop 2024-05-09T08:58:46.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa80e431640 1 -- 172.21.15.98:0/1675811174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8081010d0 msgr2=0x7fa808197490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.472+0000 7fa80e431640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8081010d0 0x7fa808197490 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00d770 tx=0x7fa7fc00dc30 comp rx=0 tx=0).stop 2024-05-09T08:58:46.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.473+0000 7fa80e431640 1 -- 172.21.15.98:0/1675811174 shutdown_connections 2024-05-09T08:58:46.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.473+0000 7fa80e431640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa7f00a2d00 0x7fa7f00a51c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.473+0000 7fa80e431640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa808106720 0x7fa80819b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.473+0000 7fa80e431640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8081010d0 0x7fa808197490 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.473+0000 7fa80e431640 1 --2- 172.21.15.98:0/1675811174 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa808100710 0x7fa808196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.473+0000 7fa80e431640 1 -- 172.21.15.98:0/1675811174 >> 172.21.15.98:0/1675811174 conn(0x7fa808077790 msgr2=0x7fa8081031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:46.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.473+0000 7fa80e431640 1 -- 172.21.15.98:0/1675811174 shutdown_connections 2024-05-09T08:58:46.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.473+0000 7fa80e431640 1 -- 172.21.15.98:0/1675811174 wait complete. 2024-05-09T08:58:46.483 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:58:46.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:58:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.627+0000 7f00f3a8c640 1 -- 172.21.15.98:0/3599592355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00ec100700 msgr2=0x7f00ec100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.627+0000 7f00f3a8c640 1 --2- 172.21.15.98:0/3599592355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00ec100700 0x7f00ec100b80 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009930 tx=0x7f00e802a420 comp rx=0 tx=0).stop 2024-05-09T08:58:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.627+0000 7f00f3a8c640 1 -- 172.21.15.98:0/3599592355 shutdown_connections 2024-05-09T08:58:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.627+0000 7f00f3a8c640 1 --2- 172.21.15.98:0/3599592355 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00ec1010c0 0x7f00ec10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.627+0000 7f00f3a8c640 1 --2- 172.21.15.98:0/3599592355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00ec100700 0x7f00ec100b80 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.627+0000 7f00f3a8c640 1 --2- 172.21.15.98:0/3599592355 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00ec106710 0x7f00ec106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.627+0000 7f00f3a8c640 1 -- 172.21.15.98:0/3599592355 >> 172.21.15.98:0/3599592355 conn(0x7f00ec077830 msgr2=0x7f00ec077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.627+0000 7f00f3a8c640 1 -- 172.21.15.98:0/3599592355 shutdown_connections 2024-05-09T08:58:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.627+0000 7f00f3a8c640 1 -- 172.21.15.98:0/3599592355 wait complete. 2024-05-09T08:58:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f3a8c640 1 Processor -- start 2024-05-09T08:58:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f3a8c640 1 -- start start 2024-05-09T08:58:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f3a8c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00ec100700 0x7f00ec198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f3a8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00ec1010c0 0x7f00ec1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f3a8c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00ec106710 0x7f00ec19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f3a8c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec111c20 con 0x7f00ec1010c0 2024-05-09T08:58:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f3a8c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec111aa0 con 0x7f00ec106710 2024-05-09T08:58:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f3a8c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec111da0 con 0x7f00ec100700 2024-05-09T08:58:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f1027640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00ec1010c0 0x7f00ec1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f1027640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00ec1010c0 0x7f00ec1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55012/0 (socket says 172.21.15.98:55012) 2024-05-09T08:58:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f1027640 1 -- 172.21.15.98:0/221812641 learned_addr learned my addr 172.21.15.98:0/221812641 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f1027640 1 -- 172.21.15.98:0/221812641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00ec100700 msgr2=0x7f00ec198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f1027640 1 --2- 172.21.15.98:0/221812641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00ec100700 0x7f00ec198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f1027640 1 -- 172.21.15.98:0/221812641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00ec106710 msgr2=0x7f00ec19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f1027640 1 --2- 172.21.15.98:0/221812641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00ec106710 0x7f00ec19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f1027640 1 -- 172.21.15.98:0/221812641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00ec19d170 con 0x7f00ec1010c0 2024-05-09T08:58:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f1027640 1 --2- 172.21.15.98:0/221812641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00ec1010c0 0x7f00ec1986c0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009230 tx=0x7f00e8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00deffd640 1 -- 172.21.15.98:0/221812641 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e803f030 con 0x7f00ec1010c0 2024-05-09T08:58:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00deffd640 1 -- 172.21.15.98:0/221812641 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00e8003ad0 con 0x7f00ec1010c0 2024-05-09T08:58:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00f3a8c640 1 -- 172.21.15.98:0/221812641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00ec19d400 con 0x7f00ec1010c0 2024-05-09T08:58:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.628+0000 7f00deffd640 1 -- 172.21.15.98:0/221812641 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e8035070 con 0x7f00ec1010c0 2024-05-09T08:58:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.629+0000 7f00f3a8c640 1 -- 172.21.15.98:0/221812641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00ec1a4cf0 con 0x7f00ec1010c0 2024-05-09T08:58:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.630+0000 7f00deffd640 1 -- 172.21.15.98:0/221812641 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f00e8042030 con 0x7f00ec1010c0 2024-05-09T08:58:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.631+0000 7f00f6799740 1 Processor -- start 2024-05-09T08:58:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.631+0000 7f00f6799740 1 -- start start 2024-05-09T08:58:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.631+0000 7f00f6799740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089280 0x562364089660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.631+0000 7f00f6799740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089ba0 0x562364093d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.631+0000 7f00deffd640 1 --2- 172.21.15.98:0/221812641 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00d80a2d00 0x7f00d80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.631+0000 7f00f6799740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 0x562364096660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.631+0000 7f00f6799740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562363feed00 con 0x562364089280 2024-05-09T08:58:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.631+0000 7f00f6799740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562363f964c0 con 0x562364089ba0 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.631+0000 7f00f6799740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562363feacc0 con 0x562364094270 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00f2029640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089280 0x562364089660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00f2029640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089280 0x562364089660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55024/0 (socket says 172.21.15.98:55024) 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00f2029640 1 -- 172.21.15.98:0/3124812625 learned_addr learned my addr 172.21.15.98:0/3124812625 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00deffd640 1 -- 172.21.15.98:0/221812641 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f00e80ef750 con 0x7f00ec1010c0 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00f1027640 1 --2- 172.21.15.98:0/221812641 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00d80a2d00 0x7f00d80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00f1828640 1 --2- 172.21.15.98:0/3124812625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089ba0 0x562364093d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00f2029640 1 -- 172.21.15.98:0/3124812625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 msgr2=0x562364096660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00f2029640 1 --2- 172.21.15.98:0/3124812625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 0x562364096660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00f2029640 1 -- 172.21.15.98:0/3124812625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089ba0 msgr2=0x562364093d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00f2029640 1 --2- 172.21.15.98:0/3124812625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089ba0 0x562364093d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00f2029640 1 -- 172.21.15.98:0/3124812625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562363dcc7d0 con 0x562364089280 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00f1828640 1 --2- 172.21.15.98:0/3124812625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089ba0 0x562364093d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00f2029640 1 --2- 172.21.15.98:0/3124812625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089280 0x562364089660 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009e20 tx=0x7f00e40106e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b587a800e7eb4c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00f1027640 1 --2- 172.21.15.98:0/221812641 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00d80a2d00 0x7f00d80a51c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f00e8000c00 tx=0x7f00e80a6040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00d7fff640 1 -- 172.21.15.98:0/3124812625 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e4016070 con 0x562364089280 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00d7fff640 1 -- 172.21.15.98:0/3124812625 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00e4004d20 con 0x562364089280 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.632+0000 7f00d7fff640 1 -- 172.21.15.98:0/3124812625 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e4012680 con 0x562364089280 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.633+0000 7f00f6799740 1 -- 172.21.15.98:0/3124812625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089280 msgr2=0x562364089660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.633+0000 7f00f6799740 1 --2- 172.21.15.98:0/3124812625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089280 0x562364089660 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009e20 tx=0x7f00e40106e0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.633+0000 7f00f6799740 1 -- 172.21.15.98:0/3124812625 shutdown_connections 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.633+0000 7f00f6799740 1 --2- 172.21.15.98:0/3124812625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 0x562364096660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.633+0000 7f00f6799740 1 --2- 172.21.15.98:0/3124812625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089ba0 0x562364093d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.633+0000 7f00f6799740 1 --2- 172.21.15.98:0/3124812625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089280 0x562364089660 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.633+0000 7f00f6799740 1 -- 172.21.15.98:0/3124812625 >> 172.21.15.98:0/3124812625 conn(0x5623640855a0 msgr2=0x562364085980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.633+0000 7f00f6799740 1 -- 172.21.15.98:0/3124812625 shutdown_connections 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.633+0000 7f00f6799740 1 -- 172.21.15.98:0/3124812625 wait complete. 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.633+0000 7f00f6799740 1 -- start start 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.633+0000 7f00f6799740 1 Processor -- start 2024-05-09T08:58:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f6799740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089280 0x562364091210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f6799740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089ba0 0x562364091750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f6799740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 0x562364091c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f6799740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562364097060 con 0x562364089ba0 2024-05-09T08:58:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f6799740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562363fa9b90 con 0x562364089280 2024-05-09T08:58:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f6799740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5623640971e0 con 0x562364094270 2024-05-09T08:58:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f6799740 10 client.0 _ll_register_callbacks cb 0x562363fe5da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f1828640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089ba0 0x562364091750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f1828640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089ba0 0x562364091750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55026/0 (socket says 172.21.15.98:55026) 2024-05-09T08:58:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f1828640 1 -- 172.21.15.98:0/1232661 learned_addr learned my addr 172.21.15.98:0/1232661 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f2029640 1 --2- 172.21.15.98:0/1232661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089280 0x562364091210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f2029640 1 --2- 172.21.15.98:0/1232661 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 0x562364091c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f1828640 1 -- 172.21.15.98:0/1232661 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 msgr2=0x562364091c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f1828640 1 --2- 172.21.15.98:0/1232661 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 0x562364091c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f1828640 1 -- 172.21.15.98:0/1232661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089280 msgr2=0x562364091210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f1828640 1 --2- 172.21.15.98:0/1232661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089280 0x562364091210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f1828640 1 -- 172.21.15.98:0/1232661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562363d023f0 con 0x562364089ba0 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f2029640 1 --2- 172.21.15.98:0/1232661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089280 0x562364091210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00f1828640 1 --2- 172.21.15.98:0/1232661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089ba0 0x562364091750 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f00e000d6a0 tx=0x7f00e000db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c1bb7380eca3e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.634+0000 7f00d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00d57fa640 1 -- 172.21.15.98:0/1232661 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e000dea0 con 0x562364089ba0 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00d57fa640 1 -- 172.21.15.98:0/1232661 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f00e0004220 con 0x562364089ba0 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00f6799740 1 -- 172.21.15.98:0/1232661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00e4004d20 con 0x562364089ba0 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00d57fa640 1 -- 172.21.15.98:0/1232661 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f00e0004420 con 0x562364089ba0 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00f6799740 10 client.17892 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00f6799740 1 -- 172.21.15.98:0/1232661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562363cc20f0 con 0x562364089ba0 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00d57fa640 1 -- 172.21.15.98:0/1232661 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00e4004d20 con 0x562364089ba0 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00d57fa640 1 -- 172.21.15.98:0/1232661 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f00e0003b50 con 0x562364089ba0 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00f6799740 10 client.17892 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00f6799740 10 client.17892 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00f6799740 20 client.17892 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00f6799740 1 --2- 172.21.15.98:0/1232661 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5623640a4610 0x5623640a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00f6799740 4 client.17892 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00f6799740 1 -- 172.21.15.98:0/1232661 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562363ffc430 con 0x5623640a4610 2024-05-09T08:58:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.635+0000 7f00f2029640 1 --2- 172.21.15.98:0/1232661 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5623640a4610 0x5623640a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.636+0000 7f00f2029640 1 --2- 172.21.15.98:0/1232661 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5623640a4610 0x5623640a6a80 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4edf6a321fed9ff4 server_cookie=7896d3ba09f325b4 in_seq=0 out_seq=0 2024-05-09T08:58:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.636+0000 7f00d57fa640 10 client.17892 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.636+0000 7f00d57fa640 1 -- 172.21.15.98:0/1232661 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562363ffc430 con 0x5623640a4610 2024-05-09T08:58:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.636+0000 7f00d57fa640 10 client.17892 handle_command_reply: tid=0 2024-05-09T08:58:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.636+0000 7f00f6799740 1 client.17892 shutdown 2024-05-09T08:58:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.637+0000 7f00f6799740 1 -- 172.21.15.98:0/1232661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089ba0 msgr2=0x562364091750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.637+0000 7f00f6799740 1 --2- 172.21.15.98:0/1232661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089ba0 0x562364091750 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f00e000d6a0 tx=0x7f00e000db60 comp rx=0 tx=0).stop 2024-05-09T08:58:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.637+0000 7f00f6799740 1 -- 172.21.15.98:0/1232661 shutdown_connections 2024-05-09T08:58:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.637+0000 7f00f6799740 1 --2- 172.21.15.98:0/1232661 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5623640a4610 0x5623640a6a80 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.637+0000 7f00f6799740 1 --2- 172.21.15.98:0/1232661 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 0x562364091c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.637+0000 7f00f6799740 1 --2- 172.21.15.98:0/1232661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089ba0 0x562364091750 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.637+0000 7f00d57fa640 0 client.17892 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.637+0000 7f00f6799740 1 --2- 172.21.15.98:0/1232661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089280 0x562364091210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.637+0000 7f00f6799740 1 -- 172.21.15.98:0/1232661 >> 172.21.15.98:0/1232661 conn(0x5623640855a0 msgr2=0x56236408b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.637+0000 7f00f6799740 1 -- 172.21.15.98:0/1232661 shutdown_connections 2024-05-09T08:58:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.637+0000 7f00f6799740 1 -- 172.21.15.98:0/1232661 wait complete. 2024-05-09T08:58:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.637+0000 7f00f6799740 20 client.17892 trim_cache size 0 max 16384 2024-05-09T08:58:46.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.654+0000 7f00f6799740 1 Processor -- start 2024-05-09T08:58:46.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.654+0000 7f00f6799740 1 -- start start 2024-05-09T08:58:46.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.654+0000 7f00f6799740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089280 0x56236408b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.654+0000 7f00f6799740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089ba0 0x5623640ac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.654+0000 7f00f6799740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 0x5623640ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.654+0000 7f00f6799740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec1aef10 con 0x562364089ba0 2024-05-09T08:58:46.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.654+0000 7f00f6799740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec1b0a50 con 0x562364089280 2024-05-09T08:58:46.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.654+0000 7f00f6799740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec1af5e0 con 0x562364094270 2024-05-09T08:58:46.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.654+0000 7f00f1828640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089ba0 0x5623640ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.654+0000 7f00f1828640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089ba0 0x5623640ac390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55028/0 (socket says 172.21.15.98:55028) 2024-05-09T08:58:46.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.654+0000 7f00f1828640 1 -- 172.21.15.98:0/2983269392 learned_addr learned my addr 172.21.15.98:0/2983269392 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:46.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.654+0000 7f00f2029640 1 --2- 172.21.15.98:0/2983269392 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089280 0x56236408b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.654+0000 7f00f1828640 1 -- 172.21.15.98:0/2983269392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 msgr2=0x5623640ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00f1828640 1 --2- 172.21.15.98:0/2983269392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 0x5623640ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00f1828640 1 -- 172.21.15.98:0/2983269392 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089280 msgr2=0x56236408b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00f1828640 1 --2- 172.21.15.98:0/2983269392 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089280 0x56236408b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00f1828640 1 -- 172.21.15.98:0/2983269392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623640969d0 con 0x562364089ba0 2024-05-09T08:58:46.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00f1828640 1 --2- 172.21.15.98:0/2983269392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089ba0 0x5623640ac390 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f00e0003f90 tx=0x7f00e0008680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b3bd791dec59cd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:46.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00d77fe640 1 -- 172.21.15.98:0/2983269392 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e002a070 con 0x562364089ba0 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00d77fe640 1 -- 172.21.15.98:0/2983269392 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00e0005bb0 con 0x562364089ba0 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00d77fe640 1 -- 172.21.15.98:0/2983269392 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e0003b40 con 0x562364089ba0 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00f6799740 1 -- 172.21.15.98:0/2983269392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089ba0 msgr2=0x5623640ac390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00f6799740 1 --2- 172.21.15.98:0/2983269392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089ba0 0x5623640ac390 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f00e0003f90 tx=0x7f00e0008680 comp rx=0 tx=0).stop 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00f6799740 1 -- 172.21.15.98:0/2983269392 shutdown_connections 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00f6799740 1 --2- 172.21.15.98:0/2983269392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 0x5623640ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00f6799740 1 --2- 172.21.15.98:0/2983269392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089ba0 0x5623640ac390 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00f6799740 1 --2- 172.21.15.98:0/2983269392 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089280 0x56236408b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00f6799740 1 -- 172.21.15.98:0/2983269392 >> 172.21.15.98:0/2983269392 conn(0x5623640855a0 msgr2=0x562364092a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00f6799740 1 -- 172.21.15.98:0/2983269392 shutdown_connections 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.655+0000 7f00f6799740 1 -- 172.21.15.98:0/2983269392 wait complete. 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.656+0000 7f00f6799740 1 -- start start 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.656+0000 7f00f6799740 1 Processor -- start 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.656+0000 7f00f6799740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089280 0x56236409c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.656+0000 7f00f6799740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089ba0 0x562364099430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.656+0000 7f00f6799740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 0x562364099970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.656+0000 7f00f6799740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec1b0380 con 0x562364089280 2024-05-09T08:58:46.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.656+0000 7f00f6799740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec1afcb0 con 0x562364089ba0 2024-05-09T08:58:46.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.656+0000 7f00f6799740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec1acf10 con 0x562364094270 2024-05-09T08:58:46.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.656+0000 7f00f6799740 10 client.0 _ll_register_callbacks cb 0x7f00e0005ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:46.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.656+0000 7f00f2029640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089280 0x56236409c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.656+0000 7f00f2029640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089280 0x56236409c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55040/0 (socket says 172.21.15.98:55040) 2024-05-09T08:58:46.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.656+0000 7f00f2029640 1 -- 172.21.15.98:0/3442650254 learned_addr learned my addr 172.21.15.98:0/3442650254 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:46.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.656+0000 7f00f1828640 1 --2- 172.21.15.98:0/3442650254 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089ba0 0x562364099430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f2029640 1 -- 172.21.15.98:0/3442650254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 msgr2=0x562364099970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f2029640 1 --2- 172.21.15.98:0/3442650254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 0x562364099970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f2029640 1 -- 172.21.15.98:0/3442650254 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089ba0 msgr2=0x562364099430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:46.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f2029640 1 --2- 172.21.15.98:0/3442650254 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089ba0 0x562364099430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f2029640 1 -- 172.21.15.98:0/3442650254 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562364099eb0 con 0x562364089280 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f2029640 1 --2- 172.21.15.98:0/3442650254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089280 0x56236409c650 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009c30 tx=0x7f00e4005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7859211a674ea9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00d57fa640 1 -- 172.21.15.98:0/3442650254 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e4023040 con 0x562364089280 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00d57fa640 1 -- 172.21.15.98:0/3442650254 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f00e4003970 con 0x562364089280 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f6799740 1 -- 172.21.15.98:0/3442650254 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00e0005bb0 con 0x562364089280 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00d57fa640 1 -- 172.21.15.98:0/3442650254 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f00e4016040 con 0x562364089280 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f6799740 10 client.17898 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f6799740 1 -- 172.21.15.98:0/3442650254 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56236410b9c0 con 0x562364089280 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00d57fa640 1 -- 172.21.15.98:0/3442650254 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00e0005bb0 con 0x562364089280 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00d57fa640 1 -- 172.21.15.98:0/3442650254 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f00e4005000 con 0x562364089280 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f6799740 10 client.17898 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f6799740 10 client.17898 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f6799740 20 client.17898 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f6799740 1 --2- 172.21.15.98:0/3442650254 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56236410bfa0 0x56236410e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:46.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f6799740 4 client.17898 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.657+0000 7f00f6799740 1 -- 172.21.15.98:0/3442650254 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562363ff6780 con 0x56236410bfa0 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.658+0000 7f00f1828640 1 --2- 172.21.15.98:0/3442650254 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56236410bfa0 0x56236410e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.658+0000 7f00d57fa640 10 client.17898 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.658+0000 7f00f1828640 1 --2- 172.21.15.98:0/3442650254 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56236410bfa0 0x56236410e390 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a25fa867e2ea172 server_cookie=34f9ca0f229acef9 in_seq=0 out_seq=0 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.658+0000 7f00d57fa640 1 -- 172.21.15.98:0/3442650254 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562363ff6780 con 0x56236410bfa0 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.658+0000 7f00d57fa640 10 client.17898 handle_command_reply: tid=0 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.658+0000 7f00f6799740 1 client.17898 shutdown 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.658+0000 7f00f6799740 1 -- 172.21.15.98:0/3442650254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089280 msgr2=0x56236409c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.658+0000 7f00f6799740 1 --2- 172.21.15.98:0/3442650254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089280 0x56236409c650 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009c30 tx=0x7f00e4005bd0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f6799740 1 -- 172.21.15.98:0/3442650254 shutdown_connections 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f6799740 1 --2- 172.21.15.98:0/3442650254 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56236410bfa0 0x56236410e390 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f6799740 1 --2- 172.21.15.98:0/3442650254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562364094270 0x562364099970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f6799740 1 --2- 172.21.15.98:0/3442650254 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562364089ba0 0x562364099430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f6799740 1 --2- 172.21.15.98:0/3442650254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562364089280 0x56236409c650 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00d57fa640 0 client.17898 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:46.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f6799740 1 -- 172.21.15.98:0/3442650254 >> 172.21.15.98:0/3442650254 conn(0x5623640855a0 msgr2=0x5623640932d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:46.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f6799740 1 -- 172.21.15.98:0/3442650254 shutdown_connections 2024-05-09T08:58:46.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f6799740 1 -- 172.21.15.98:0/3442650254 wait complete. 2024-05-09T08:58:46.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f6799740 20 client.17898 trim_cache size 0 max 16384 2024-05-09T08:58:46.665 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:58:46.665 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:58:46.665 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:58:46.665 INFO:teuthology.orchestra.run.smithi098.stdout: "c53a794f-b42b-4719-9e6e-fed2fa0c2102": { 2024-05-09T08:58:46.665 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:58:46.666 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "c53a794f-b42b-4719-9e6e-fed2fa0c2102", 2024-05-09T08:58:46.666 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:58:46.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:58:46.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:58:46.666 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:58:46.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f3a8c640 1 -- 172.21.15.98:0/221812641 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00d80a2d00 msgr2=0x7f00d80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f3a8c640 1 --2- 172.21.15.98:0/221812641 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00d80a2d00 0x7f00d80a51c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f00e8000c00 tx=0x7f00e80a6040 comp rx=0 tx=0).stop 2024-05-09T08:58:46.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f3a8c640 1 -- 172.21.15.98:0/221812641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00ec1010c0 msgr2=0x7f00ec1986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:46.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f3a8c640 1 --2- 172.21.15.98:0/221812641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00ec1010c0 0x7f00ec1986c0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009230 tx=0x7f00e8004270 comp rx=0 tx=0).stop 2024-05-09T08:58:46.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f3a8c640 1 -- 172.21.15.98:0/221812641 shutdown_connections 2024-05-09T08:58:46.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f3a8c640 1 --2- 172.21.15.98:0/221812641 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00d80a2d00 0x7f00d80a51c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f3a8c640 1 --2- 172.21.15.98:0/221812641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00ec106710 0x7f00ec19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f3a8c640 1 --2- 172.21.15.98:0/221812641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00ec1010c0 0x7f00ec1986c0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f3a8c640 1 --2- 172.21.15.98:0/221812641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00ec100700 0x7f00ec198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:46.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f3a8c640 1 -- 172.21.15.98:0/221812641 >> 172.21.15.98:0/221812641 conn(0x7f00ec077830 msgr2=0x7f00ec103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:46.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f3a8c640 1 -- 172.21.15.98:0/221812641 shutdown_connections 2024-05-09T08:58:46.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:58:46.659+0000 7f00f3a8c640 1 -- 172.21.15.98:0/221812641 wait complete. 2024-05-09T08:58:46.669 INFO:tasks.cephfs.filesystem:scrub status for tag:c53a794f-b42b-4719-9e6e-fed2fa0c2102 - {'path': '/', 'tag': 'c53a794f-b42b-4719-9e6e-fed2fa0c2102', 'options': 'recursive,force'} 2024-05-09T08:58:47.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:46 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:58:47.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:46 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:58:47.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:46 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:58:47.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:46 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:58:47.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:46 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:58:47.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:46 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:58:47.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:46 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:58:47.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:46 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:58:47.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:46 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:58:47.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:46 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:58:47.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:46 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:58:47.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:46 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:58:48.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:47 smithi098 ceph-mon[30092]: pgmap v253: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.6 KiB/s wr, 14 op/s 2024-05-09T08:58:48.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:47 smithi142 ceph-mon[34625]: pgmap v253: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.6 KiB/s wr, 14 op/s 2024-05-09T08:58:48.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:47 smithi105 ceph-mon[34498]: pgmap v253: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.6 KiB/s wr, 14 op/s 2024-05-09T08:58:48.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.840+0000 7f002ffd5640 1 -- 172.21.15.98:0/268427244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0028069850 msgr2=0x7f0028100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.840+0000 7f002ffd5640 1 --2- 172.21.15.98:0/268427244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0028069850 0x7f0028100110 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f0024009cd0 tx=0x7f002402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.840+0000 7f002ffd5640 1 -- 172.21.15.98:0/268427244 shutdown_connections 2024-05-09T08:58:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.840+0000 7f002ffd5640 1 --2- 172.21.15.98:0/268427244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0028104b00 0x7f0028104f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.840+0000 7f002ffd5640 1 --2- 172.21.15.98:0/268427244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0028069cc0 0x7f0028100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.840+0000 7f002ffd5640 1 --2- 172.21.15.98:0/268427244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0028069850 0x7f0028100110 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.840+0000 7f002ffd5640 1 -- 172.21.15.98:0/268427244 >> 172.21.15.98:0/268427244 conn(0x7f0028076c50 msgr2=0x7f00280fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.840+0000 7f002ffd5640 1 -- 172.21.15.98:0/268427244 shutdown_connections 2024-05-09T08:58:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.840+0000 7f002ffd5640 1 -- 172.21.15.98:0/268427244 wait complete. 2024-05-09T08:58:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.840+0000 7f002ffd5640 1 Processor -- start 2024-05-09T08:58:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.840+0000 7f002ffd5640 1 -- start start 2024-05-09T08:58:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002ffd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0028069850 0x7f0028198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002ffd5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0028069cc0 0x7f00281994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002ffd5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0028104b00 0x7f00281a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002ffd5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0028110bc0 con 0x7f0028069850 2024-05-09T08:58:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002ffd5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0028110a40 con 0x7f0028069cc0 2024-05-09T08:58:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002ffd5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0028110d40 con 0x7f0028104b00 2024-05-09T08:58:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002dd71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0028069850 0x7f0028198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002dd71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0028069850 0x7f0028198fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55060/0 (socket says 172.21.15.98:55060) 2024-05-09T08:58:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002dd71640 1 -- 172.21.15.98:0/1406414678 learned_addr learned my addr 172.21.15.98:0/1406414678 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002dd71640 1 -- 172.21.15.98:0/1406414678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0028104b00 msgr2=0x7f00281a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002d570640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0028069cc0 0x7f00281994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002e572640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0028104b00 0x7f00281a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002dd71640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0028104b00 0x7f00281a4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002dd71640 1 -- 172.21.15.98:0/1406414678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0028069cc0 msgr2=0x7f00281994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:48.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002dd71640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0028069cc0 0x7f00281994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002dd71640 1 -- 172.21.15.98:0/1406414678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00281a4560 con 0x7f0028069850 2024-05-09T08:58:48.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002d570640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0028069cc0 0x7f00281994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:48.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f002dd71640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0028069850 0x7f0028198fa0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f002402aed0 tx=0x7f0024003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:48.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.841+0000 7f001effd640 1 -- 172.21.15.98:0/1406414678 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0024004450 con 0x7f0028069850 2024-05-09T08:58:48.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.842+0000 7f001effd640 1 -- 172.21.15.98:0/1406414678 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0024003e10 con 0x7f0028069850 2024-05-09T08:58:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.842+0000 7f002ffd5640 1 -- 172.21.15.98:0/1406414678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00281a4730 con 0x7f0028069850 2024-05-09T08:58:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.842+0000 7f001effd640 1 -- 172.21.15.98:0/1406414678 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00240466e0 con 0x7f0028069850 2024-05-09T08:58:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.842+0000 7f002ffd5640 1 -- 172.21.15.98:0/1406414678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00281a4bf0 con 0x7f0028069850 2024-05-09T08:58:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.844+0000 7f001effd640 1 -- 172.21.15.98:0/1406414678 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f00240305a0 con 0x7f0028069850 2024-05-09T08:58:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.844+0000 7f0032dc3740 1 Processor -- start 2024-05-09T08:58:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f0032dc3740 1 -- start start 2024-05-09T08:58:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f0032dc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4dfa450 0x55dbd4dfa830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f0032dc3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4dfad70 0x55dbd4e050f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f0032dc3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4e05630 0x55dbd4e07a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f0032dc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dbd4d5f340 con 0x55dbd4dfa450 2024-05-09T08:58:48.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f0032dc3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55dbd4d07290 con 0x55dbd4e05630 2024-05-09T08:58:48.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f0032dc3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55dbd4d5b300 con 0x55dbd4dfad70 2024-05-09T08:58:48.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f002e572640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4dfa450 0x55dbd4dfa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f001effd640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00140a2da0 0x7f00140a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f002e572640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4dfa450 0x55dbd4dfa830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55068/0 (socket says 172.21.15.98:55068) 2024-05-09T08:58:48.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f002e572640 1 -- 172.21.15.98:0/2941576813 learned_addr learned my addr 172.21.15.98:0/2941576813 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:48.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f002d570640 1 --2- 172.21.15.98:0/2941576813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4e05630 0x55dbd4e07a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f002e572640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00140a2da0 0x7f00140a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f001effd640 1 -- 172.21.15.98:0/1406414678 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f002402e030 con 0x7f0028069850 2024-05-09T08:58:48.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f002dd71640 1 --2- 172.21.15.98:0/2941576813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4dfad70 0x55dbd4e050f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f002e572640 1 -- 172.21.15.98:0/2941576813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4dfad70 msgr2=0x55dbd4e050f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:48.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f002e572640 1 --2- 172.21.15.98:0/2941576813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4dfad70 0x55dbd4e050f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f002e572640 1 -- 172.21.15.98:0/2941576813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4e05630 msgr2=0x55dbd4e07a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:48.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f002e572640 1 --2- 172.21.15.98:0/2941576813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4e05630 0x55dbd4e07a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.845+0000 7f002e572640 1 -- 172.21.15.98:0/2941576813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbd4d2b8e0 con 0x55dbd4dfa450 2024-05-09T08:58:48.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.846+0000 7f002e572640 1 --2- 172.21.15.98:0/2941576813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4dfa450 0x55dbd4dfa830 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f00200145f0 tx=0x7f0020014ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a0f6d616d8883f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:48.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.846+0000 7f0013fff640 1 -- 172.21.15.98:0/2941576813 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020021070 con 0x55dbd4dfa450 2024-05-09T08:58:48.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.846+0000 7f0013fff640 1 -- 172.21.15.98:0/2941576813 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f002000bad0 con 0x55dbd4dfa450 2024-05-09T08:58:48.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.846+0000 7f0013fff640 1 -- 172.21.15.98:0/2941576813 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002000f6a0 con 0x55dbd4dfa450 2024-05-09T08:58:48.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.846+0000 7f002e572640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00140a2da0 0x7f00140a5260 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f002001e000 tx=0x7f002001b000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:48.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.847+0000 7f0032dc3740 1 -- 172.21.15.98:0/2941576813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4dfa450 msgr2=0x55dbd4dfa830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:48.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.847+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2941576813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4dfa450 0x55dbd4dfa830 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f00200145f0 tx=0x7f0020014ab0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.847+0000 7f0032dc3740 1 -- 172.21.15.98:0/2941576813 shutdown_connections 2024-05-09T08:58:48.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.847+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2941576813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4e05630 0x55dbd4e07a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.847+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2941576813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4dfad70 0x55dbd4e050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.847+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2941576813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4dfa450 0x55dbd4dfa830 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.847+0000 7f0032dc3740 1 -- 172.21.15.98:0/2941576813 >> 172.21.15.98:0/2941576813 conn(0x55dbd4df6960 msgr2=0x55dbd4df6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:48.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.847+0000 7f0032dc3740 1 -- 172.21.15.98:0/2941576813 shutdown_connections 2024-05-09T08:58:48.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.847+0000 7f0032dc3740 1 -- 172.21.15.98:0/2941576813 wait complete. 2024-05-09T08:58:48.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.847+0000 7f0032dc3740 1 -- start start 2024-05-09T08:58:48.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.847+0000 7f0032dc3740 1 Processor -- start 2024-05-09T08:58:48.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f0032dc3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4dfa450 0x55dbd4e025c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f0032dc3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 0x55dbd4e02b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f0032dc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4e05630 0x55dbd4e03040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f0032dc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dbd4e08260 con 0x55dbd4e05630 2024-05-09T08:58:48.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f0032dc3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55dbd4d1a790 con 0x55dbd4dfad70 2024-05-09T08:58:48.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f0032dc3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55dbd4e083e0 con 0x55dbd4dfa450 2024-05-09T08:58:48.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f0032dc3740 10 client.0 _ll_register_callbacks cb 0x55dbd4d55a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:48.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f002e572640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4e05630 0x55dbd4e03040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f002e572640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4e05630 0x55dbd4e03040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55074/0 (socket says 172.21.15.98:55074) 2024-05-09T08:58:48.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f002e572640 1 -- 172.21.15.98:0/2607526559 learned_addr learned my addr 172.21.15.98:0/2607526559 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:48.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f002d570640 1 --2- 172.21.15.98:0/2607526559 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 0x55dbd4e02b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f002e572640 1 -- 172.21.15.98:0/2607526559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4dfa450 msgr2=0x55dbd4e025c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:48.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f002e572640 1 --2- 172.21.15.98:0/2607526559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4dfa450 0x55dbd4e025c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f002e572640 1 -- 172.21.15.98:0/2607526559 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 msgr2=0x55dbd4e02b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:48.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f002e572640 1 --2- 172.21.15.98:0/2607526559 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 0x55dbd4e02b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f002e572640 1 -- 172.21.15.98:0/2607526559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbd4a73210 con 0x55dbd4e05630 2024-05-09T08:58:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f002e572640 1 --2- 172.21.15.98:0/2607526559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4e05630 0x55dbd4e03040 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f0020014fb0 tx=0x7f002000fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a468a688937ab2ca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f00117fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f00117fa640 1 -- 172.21.15.98:0/2607526559 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020021040 con 0x55dbd4e05630 2024-05-09T08:58:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f0032dc3740 1 -- 172.21.15.98:0/2607526559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f002000bad0 con 0x55dbd4e05630 2024-05-09T08:58:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f00117fa640 1 -- 172.21.15.98:0/2607526559 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f002001c9c0 con 0x55dbd4e05630 2024-05-09T08:58:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f00117fa640 1 -- 172.21.15.98:0/2607526559 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f002000b370 con 0x55dbd4e05630 2024-05-09T08:58:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f0032dc3740 10 client.17910 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.848+0000 7f0032dc3740 1 -- 172.21.15.98:0/2607526559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55dbd4b3f620 con 0x55dbd4e05630 2024-05-09T08:58:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.849+0000 7f00117fa640 1 -- 172.21.15.98:0/2607526559 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f002000bad0 con 0x55dbd4e05630 2024-05-09T08:58:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.849+0000 7f00117fa640 1 -- 172.21.15.98:0/2607526559 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f002001b070 con 0x55dbd4e05630 2024-05-09T08:58:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.849+0000 7f0032dc3740 10 client.17910 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.849+0000 7f0032dc3740 10 client.17910 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.849+0000 7f0032dc3740 20 client.17910 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.849+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2607526559 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55dbd4e152d0 0x55dbd4e176c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.849+0000 7f0032dc3740 4 client.17910 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:58:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.849+0000 7f0032dc3740 1 -- 172.21.15.98:0/2607526559 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbd4d6ca70 con 0x55dbd4e152d0 2024-05-09T08:58:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.849+0000 7f002dd71640 1 --2- 172.21.15.98:0/2607526559 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55dbd4e152d0 0x55dbd4e176c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.850+0000 7f00117fa640 10 client.17910 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.850+0000 7f002dd71640 1 --2- 172.21.15.98:0/2607526559 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55dbd4e152d0 0x55dbd4e176c0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad534733b784f01 server_cookie=d4f663be244679db in_seq=0 out_seq=0 2024-05-09T08:58:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.850+0000 7f00117fa640 1 -- 172.21.15.98:0/2607526559 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dbd4d6ca70 con 0x55dbd4e152d0 2024-05-09T08:58:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.850+0000 7f00117fa640 10 client.17910 handle_command_reply: tid=0 2024-05-09T08:58:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.850+0000 7f0032dc3740 1 client.17910 shutdown 2024-05-09T08:58:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.850+0000 7f0032dc3740 1 -- 172.21.15.98:0/2607526559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4e05630 msgr2=0x55dbd4e03040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.850+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2607526559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4e05630 0x55dbd4e03040 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f0020014fb0 tx=0x7f002000fba0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.850+0000 7f0032dc3740 1 -- 172.21.15.98:0/2607526559 shutdown_connections 2024-05-09T08:58:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.850+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2607526559 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55dbd4e152d0 0x55dbd4e176c0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.850+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2607526559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4e05630 0x55dbd4e03040 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.850+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2607526559 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 0x55dbd4e02b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.850+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2607526559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4dfa450 0x55dbd4e025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.850+0000 7f00117fa640 0 client.17910 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.850+0000 7f0032dc3740 1 -- 172.21.15.98:0/2607526559 >> 172.21.15.98:0/2607526559 conn(0x55dbd4df6960 msgr2=0x55dbd4dfc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.851+0000 7f0032dc3740 1 -- 172.21.15.98:0/2607526559 shutdown_connections 2024-05-09T08:58:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.851+0000 7f0032dc3740 1 -- 172.21.15.98:0/2607526559 wait complete. 2024-05-09T08:58:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.851+0000 7f0032dc3740 20 client.17910 trim_cache size 0 max 16384 2024-05-09T08:58:48.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.865+0000 7f0032dc3740 1 Processor -- start 2024-05-09T08:58:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.865+0000 7f0032dc3740 1 -- start start 2024-05-09T08:58:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f0032dc3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4dfa450 0x55dbd4e11120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f0032dc3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 0x55dbd4dfce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f0032dc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4e05630 0x55dbd4e1cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f0032dc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f00281aeed0 con 0x55dbd4e05630 2024-05-09T08:58:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f0032dc3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f00281b0a10 con 0x55dbd4dfad70 2024-05-09T08:58:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f0032dc3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f00281af5a0 con 0x55dbd4dfa450 2024-05-09T08:58:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f002e572640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4e05630 0x55dbd4e1cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f002e572640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4e05630 0x55dbd4e1cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55076/0 (socket says 172.21.15.98:55076) 2024-05-09T08:58:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f002e572640 1 -- 172.21.15.98:0/2602040200 learned_addr learned my addr 172.21.15.98:0/2602040200 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f002d570640 1 --2- 172.21.15.98:0/2602040200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 0x55dbd4dfce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f002e572640 1 -- 172.21.15.98:0/2602040200 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4dfa450 msgr2=0x55dbd4e11120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:58:48.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f002e572640 1 --2- 172.21.15.98:0/2602040200 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4dfa450 0x55dbd4e11120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f002e572640 1 -- 172.21.15.98:0/2602040200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 msgr2=0x55dbd4dfce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:48.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f002e572640 1 --2- 172.21.15.98:0/2602040200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 0x55dbd4dfce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f002e572640 1 -- 172.21.15.98:0/2602040200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbd4dfb3e0 con 0x55dbd4e05630 2024-05-09T08:58:48.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f002d570640 1 --2- 172.21.15.98:0/2602040200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 0x55dbd4dfce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:48.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f002e572640 1 --2- 172.21.15.98:0/2602040200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4e05630 0x55dbd4e1cf90 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f002000f020 tx=0x7f0020014ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e0751a4cd02d65 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:48.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f00137fe640 1 -- 172.21.15.98:0/2602040200 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020018070 con 0x55dbd4e05630 2024-05-09T08:58:48.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f00137fe640 1 -- 172.21.15.98:0/2602040200 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0020005a10 con 0x55dbd4e05630 2024-05-09T08:58:48.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.866+0000 7f00137fe640 1 -- 172.21.15.98:0/2602040200 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002001c7c0 con 0x55dbd4e05630 2024-05-09T08:58:48.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.867+0000 7f0032dc3740 1 -- 172.21.15.98:0/2602040200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4e05630 msgr2=0x55dbd4e1cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:48.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.867+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2602040200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4e05630 0x55dbd4e1cf90 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f002000f020 tx=0x7f0020014ab0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.867+0000 7f0032dc3740 1 -- 172.21.15.98:0/2602040200 shutdown_connections 2024-05-09T08:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.867+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2602040200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4e05630 0x55dbd4e1cf90 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.867+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2602040200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 0x55dbd4dfce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.867+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2602040200 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4dfa450 0x55dbd4e11120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.867+0000 7f0032dc3740 1 -- 172.21.15.98:0/2602040200 >> 172.21.15.98:0/2602040200 conn(0x55dbd4df6960 msgr2=0x55dbd4df5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.867+0000 7f0032dc3740 1 -- 172.21.15.98:0/2602040200 shutdown_connections 2024-05-09T08:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.867+0000 7f0032dc3740 1 -- 172.21.15.98:0/2602040200 wait complete. 2024-05-09T08:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.867+0000 7f0032dc3740 1 -- start start 2024-05-09T08:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.867+0000 7f0032dc3740 1 Processor -- start 2024-05-09T08:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f0032dc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4dfa450 0x55dbd4dff390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f0032dc3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 0x55dbd4dff8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f0032dc3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4e05630 0x55dbd4dffe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f0032dc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f00281b0340 con 0x55dbd4dfa450 2024-05-09T08:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f0032dc3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f00281afc70 con 0x55dbd4dfad70 2024-05-09T08:58:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f0032dc3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f00281aced0 con 0x55dbd4e05630 2024-05-09T08:58:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f0032dc3740 10 client.0 _ll_register_callbacks cb 0x7f002001f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:58:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f002dd71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4dfa450 0x55dbd4dff390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f002dd71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4dfa450 0x55dbd4dff390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55092/0 (socket says 172.21.15.98:55092) 2024-05-09T08:58:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f002dd71640 1 -- 172.21.15.98:0/2424441758 learned_addr learned my addr 172.21.15.98:0/2424441758 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:58:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f002e572640 1 --2- 172.21.15.98:0/2424441758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4e05630 0x55dbd4dffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f002dd71640 1 -- 172.21.15.98:0/2424441758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4e05630 msgr2=0x55dbd4dffe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f002dd71640 1 --2- 172.21.15.98:0/2424441758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4e05630 0x55dbd4dffe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f002dd71640 1 -- 172.21.15.98:0/2424441758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 msgr2=0x55dbd4dff8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:58:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f002dd71640 1 --2- 172.21.15.98:0/2424441758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 0x55dbd4dff8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f002dd71640 1 -- 172.21.15.98:0/2424441758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbd4e7a880 con 0x55dbd4dfa450 2024-05-09T08:58:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f002e572640 1 --2- 172.21.15.98:0/2424441758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4e05630 0x55dbd4dffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:58:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f002dd71640 1 --2- 172.21.15.98:0/2424441758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4dfa450 0x55dbd4dff390 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f00240ee5e0 tx=0x7f00240dbf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44f8d30b4425dd8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:58:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f00117fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f00117fa640 1 -- 172.21.15.98:0/2424441758 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00240a2ae0 con 0x55dbd4dfa450 2024-05-09T08:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f00117fa640 1 -- 172.21.15.98:0/2424441758 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f00240045f0 con 0x55dbd4dfa450 2024-05-09T08:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.868+0000 7f0032dc3740 1 -- 172.21.15.98:0/2424441758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0020005a10 con 0x55dbd4dfa450 2024-05-09T08:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.869+0000 7f00117fa640 1 -- 172.21.15.98:0/2424441758 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f00240c5fc0 con 0x55dbd4dfa450 2024-05-09T08:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.869+0000 7f0032dc3740 10 client.17916 fetch_fsmap learned FSMap version 37 2024-05-09T08:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.869+0000 7f0032dc3740 1 -- 172.21.15.98:0/2424441758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55dbd4e7bb40 con 0x55dbd4dfa450 2024-05-09T08:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.869+0000 7f00117fa640 1 -- 172.21.15.98:0/2424441758 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0020005a10 con 0x55dbd4dfa450 2024-05-09T08:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.869+0000 7f00117fa640 1 -- 172.21.15.98:0/2424441758 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f002404e070 con 0x55dbd4dfa450 2024-05-09T08:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.869+0000 7f0032dc3740 10 client.17916 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.869+0000 7f0032dc3740 10 client.17916 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.869+0000 7f0032dc3740 20 client.17916 populate_metadata read hostname 'smithi098' 2024-05-09T08:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.869+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2424441758 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55dbd4e7c220 0x55dbd4e7e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:58:48.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.869+0000 7f0032dc3740 4 client.17916 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.869+0000 7f0032dc3740 1 -- 172.21.15.98:0/2424441758 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dbd4d66dc0 con 0x55dbd4e7c220 2024-05-09T08:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.869+0000 7f002d570640 1 --2- 172.21.15.98:0/2424441758 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55dbd4e7c220 0x55dbd4e7e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.870+0000 7f00117fa640 10 client.17916 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.870+0000 7f002d570640 1 --2- 172.21.15.98:0/2424441758 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55dbd4e7c220 0x55dbd4e7e640 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=702d7e7034db97af server_cookie=5c37aaa6246becd2 in_seq=0 out_seq=0 2024-05-09T08:58:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f00117fa640 1 -- 172.21.15.98:0/2424441758 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dbd4d66dc0 con 0x55dbd4e7c220 2024-05-09T08:58:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f00117fa640 10 client.17916 handle_command_reply: tid=0 2024-05-09T08:58:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f0032dc3740 1 client.17916 shutdown 2024-05-09T08:58:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f0032dc3740 1 -- 172.21.15.98:0/2424441758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4dfa450 msgr2=0x55dbd4dff390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:48.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2424441758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4dfa450 0x55dbd4dff390 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f00240ee5e0 tx=0x7f00240dbf90 comp rx=0 tx=0).stop 2024-05-09T08:58:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f0032dc3740 1 -- 172.21.15.98:0/2424441758 shutdown_connections 2024-05-09T08:58:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2424441758 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55dbd4e7c220 0x55dbd4e7e640 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2424441758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dbd4e05630 0x55dbd4dffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2424441758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dbd4dfad70 0x55dbd4dff8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f0032dc3740 1 --2- 172.21.15.98:0/2424441758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dbd4dfa450 0x55dbd4dff390 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f00117fa640 0 client.17916 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:58:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f0032dc3740 1 -- 172.21.15.98:0/2424441758 >> 172.21.15.98:0/2424441758 conn(0x55dbd4df6960 msgr2=0x55dbd4e023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f0032dc3740 1 -- 172.21.15.98:0/2424441758 shutdown_connections 2024-05-09T08:58:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f0032dc3740 1 -- 172.21.15.98:0/2424441758 wait complete. 2024-05-09T08:58:48.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.874+0000 7f0032dc3740 20 client.17916 trim_cache size 0 max 16384 2024-05-09T08:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:58:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.875+0000 7f002ffd5640 1 -- 172.21.15.98:0/1406414678 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00140a2da0 msgr2=0x7f00140a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.875+0000 7f002ffd5640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00140a2da0 0x7f00140a5260 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f002001e000 tx=0x7f002001b000 comp rx=0 tx=0).stop 2024-05-09T08:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.875+0000 7f002ffd5640 1 -- 172.21.15.98:0/1406414678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0028069850 msgr2=0x7f0028198fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.875+0000 7f002ffd5640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0028069850 0x7f0028198fa0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f002402aed0 tx=0x7f0024003c20 comp rx=0 tx=0).stop 2024-05-09T08:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.875+0000 7f002ffd5640 1 -- 172.21.15.98:0/1406414678 shutdown_connections 2024-05-09T08:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.875+0000 7f002ffd5640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00140a2da0 0x7f00140a5260 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.875+0000 7f002ffd5640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0028104b00 0x7f00281a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.875+0000 7f002ffd5640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0028069cc0 0x7f00281994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.875+0000 7f002ffd5640 1 --2- 172.21.15.98:0/1406414678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0028069850 0x7f0028198fa0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.875+0000 7f002ffd5640 1 -- 172.21.15.98:0/1406414678 >> 172.21.15.98:0/1406414678 conn(0x7f0028076c50 msgr2=0x7f0028102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:58:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.875+0000 7f002ffd5640 1 -- 172.21.15.98:0/1406414678 shutdown_connections 2024-05-09T08:58:48.881 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:58:48.875+0000 7f002ffd5640 1 -- 172.21.15.98:0/1406414678 wait complete. 2024-05-09T08:58:50.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:49 smithi098 ceph-mon[30092]: pgmap v254: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-05-09T08:58:50.657 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:49 smithi105 ceph-mon[34498]: pgmap v254: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-05-09T08:58:50.657 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:49 smithi142 ceph-mon[34625]: pgmap v254: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-05-09T08:58:52.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:51 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:58:52.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:51 smithi098 ceph-mon[30092]: pgmap v255: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 1.3 KiB/s wr, 14 op/s 2024-05-09T08:58:52.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:51 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:58:52.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:51 smithi142 ceph-mon[34625]: pgmap v255: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 1.3 KiB/s wr, 14 op/s 2024-05-09T08:58:52.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:51 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:58:52.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:51 smithi105 ceph-mon[34498]: pgmap v255: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 1.3 KiB/s wr, 14 op/s 2024-05-09T08:58:54.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:54 smithi098 ceph-mon[30092]: pgmap v256: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:58:54.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:54 smithi142 ceph-mon[34625]: pgmap v256: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:58:54.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:54 smithi105 ceph-mon[34498]: pgmap v256: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:58:56.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:56 smithi098 ceph-mon[30092]: pgmap v257: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:58:56.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:56 smithi142 ceph-mon[34625]: pgmap v257: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:58:56.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:56 smithi105 ceph-mon[34498]: pgmap v257: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:58:58.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:58:58 smithi098 ceph-mon[30092]: pgmap v258: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T08:58:58.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:58:58 smithi142 ceph-mon[34625]: pgmap v258: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T08:58:58.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:58:58 smithi105 ceph-mon[34498]: pgmap v258: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T08:59:00.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:00 smithi098 ceph-mon[30092]: pgmap v259: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:59:00.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:00 smithi142 ceph-mon[34625]: pgmap v259: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:59:00.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:00 smithi105 ceph-mon[34498]: pgmap v259: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:59:01.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:01 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:59:01.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:01 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:59:01.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:01 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:59:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.970+0000 7f859d8a5640 1 -- 172.21.15.98:0/1855003413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85981056d0 msgr2=0x7f8598105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.970+0000 7f859d8a5640 1 --2- 172.21.15.98:0/1855003413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85981056d0 0x7f8598105ab0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f8588009900 tx=0x7f858802a410 comp rx=0 tx=0).stop 2024-05-09T08:59:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.970+0000 7f859d8a5640 1 -- 172.21.15.98:0/1855003413 shutdown_connections 2024-05-09T08:59:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.970+0000 7f859d8a5640 1 --2- 172.21.15.98:0/1855003413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8598101440 0x7f859810d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.970+0000 7f859d8a5640 1 --2- 172.21.15.98:0/1855003413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8598100a80 0x7f8598100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.970+0000 7f859d8a5640 1 --2- 172.21.15.98:0/1855003413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85981056d0 0x7f8598105ab0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.970+0000 7f859d8a5640 1 -- 172.21.15.98:0/1855003413 >> 172.21.15.98:0/1855003413 conn(0x7f8598076530 msgr2=0x7f8598076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.970+0000 7f859d8a5640 1 -- 172.21.15.98:0/1855003413 shutdown_connections 2024-05-09T08:59:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.970+0000 7f859d8a5640 1 -- 172.21.15.98:0/1855003413 wait complete. 2024-05-09T08:59:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f859d8a5640 1 Processor -- start 2024-05-09T08:59:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f859d8a5640 1 -- start start 2024-05-09T08:59:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f859d8a5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8598100a80 0x7f8598197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f859d8a5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8598101440 0x7f85981984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f859d8a5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85981056d0 0x7f859819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f859d8a5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8598110ac0 con 0x7f8598100a80 2024-05-09T08:59:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f859d8a5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8598110940 con 0x7f8598101440 2024-05-09T08:59:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f859d8a5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8598110c40 con 0x7f85981056d0 2024-05-09T08:59:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f8596ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8598100a80 0x7f8598197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f8596ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8598100a80 0x7f8598197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56678/0 (socket says 172.21.15.98:56678) 2024-05-09T08:59:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f8596ffd640 1 -- 172.21.15.98:0/2617077033 learned_addr learned my addr 172.21.15.98:0/2617077033 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f858e7fc640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8598101440 0x7f85981984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f85977fe640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85981056d0 0x7f859819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f8596ffd640 1 -- 172.21.15.98:0/2617077033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85981056d0 msgr2=0x7f859819c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f8596ffd640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85981056d0 0x7f859819c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f8596ffd640 1 -- 172.21.15.98:0/2617077033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8598101440 msgr2=0x7f85981984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f8596ffd640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8598101440 0x7f85981984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f8596ffd640 1 -- 172.21.15.98:0/2617077033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f859819cfa0 con 0x7f8598100a80 2024-05-09T08:59:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.971+0000 7f858e7fc640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8598101440 0x7f85981984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.972+0000 7f8596ffd640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8598100a80 0x7f8598197fb0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f8588005ab0 tx=0x7f85880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.972+0000 7f859c8a3640 1 -- 172.21.15.98:0/2617077033 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858803f030 con 0x7f8598100a80 2024-05-09T08:59:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.972+0000 7f859d8a5640 1 -- 172.21.15.98:0/2617077033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f859819d230 con 0x7f8598100a80 2024-05-09T08:59:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.972+0000 7f859c8a3640 1 -- 172.21.15.98:0/2617077033 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8588003ce0 con 0x7f8598100a80 2024-05-09T08:59:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.972+0000 7f859c8a3640 1 -- 172.21.15.98:0/2617077033 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8588035070 con 0x7f8598100a80 2024-05-09T08:59:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.972+0000 7f859d8a5640 1 -- 172.21.15.98:0/2617077033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85981a4b20 con 0x7f8598100a80 2024-05-09T08:59:01.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.974+0000 7f859c8a3640 1 -- 172.21.15.98:0/2617077033 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8588042030 con 0x7f8598100a80 2024-05-09T08:59:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.974+0000 7f85a05b8740 1 Processor -- start 2024-05-09T08:59:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.974+0000 7f85a05b8740 1 -- start start 2024-05-09T08:59:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f85a05b8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cc7450 0x559d47cc7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f85a05b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47cd20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f85a05b8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cd2630 0x559d47cd4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f85a05b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559d47c2c340 con 0x559d47cc7d70 2024-05-09T08:59:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f85a05b8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559d47bd4290 con 0x559d47cc7450 2024-05-09T08:59:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f85a05b8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559d47c28300 con 0x559d47cd2630 2024-05-09T08:59:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f8596ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47cd20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f8596ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47cd20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56682/0 (socket says 172.21.15.98:56682) 2024-05-09T08:59:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f8596ffd640 1 -- 172.21.15.98:0/801175796 learned_addr learned my addr 172.21.15.98:0/801175796 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f859c8a3640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f85840a2da0 0x7f85840a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f858e7fc640 1 --2- 172.21.15.98:0/801175796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cd2630 0x559d47cd4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f85977fe640 1 --2- 172.21.15.98:0/801175796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cc7450 0x559d47cc7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f85977fe640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f85840a2da0 0x7f85840a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f859c8a3640 1 -- 172.21.15.98:0/2617077033 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f85880f01e0 con 0x7f8598100a80 2024-05-09T08:59:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f8596ffd640 1 -- 172.21.15.98:0/801175796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cd2630 msgr2=0x559d47cd4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f8596ffd640 1 --2- 172.21.15.98:0/801175796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cd2630 0x559d47cd4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f8596ffd640 1 -- 172.21.15.98:0/801175796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cc7450 msgr2=0x559d47cc7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f8596ffd640 1 --2- 172.21.15.98:0/801175796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cc7450 0x559d47cc7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f8596ffd640 1 -- 172.21.15.98:0/801175796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d47bf88e0 con 0x559d47cc7d70 2024-05-09T08:59:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.975+0000 7f8596ffd640 1 --2- 172.21.15.98:0/801175796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47cd20f0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f85880bbfd0 tx=0x7f858802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=addac1d6326f4961 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.976+0000 7f85977fe640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f85840a2da0 0x7f85840a5260 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8590005fd0 tx=0x7f8590005db0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.976+0000 7f858dffb640 1 -- 172.21.15.98:0/801175796 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8588049070 con 0x559d47cc7d70 2024-05-09T08:59:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.976+0000 7f858dffb640 1 -- 172.21.15.98:0/801175796 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85880bc9d0 con 0x559d47cc7d70 2024-05-09T08:59:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.976+0000 7f858dffb640 1 -- 172.21.15.98:0/801175796 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85880b8940 con 0x559d47cc7d70 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.976+0000 7f85a05b8740 1 -- 172.21.15.98:0/801175796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 msgr2=0x559d47cd20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.976+0000 7f85a05b8740 1 --2- 172.21.15.98:0/801175796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47cd20f0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f85880bbfd0 tx=0x7f858802a410 comp rx=0 tx=0).stop 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.976+0000 7f85a05b8740 1 -- 172.21.15.98:0/801175796 shutdown_connections 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.976+0000 7f85a05b8740 1 --2- 172.21.15.98:0/801175796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cd2630 0x559d47cd4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.976+0000 7f85a05b8740 1 --2- 172.21.15.98:0/801175796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47cd20f0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.976+0000 7f85a05b8740 1 --2- 172.21.15.98:0/801175796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cc7450 0x559d47cc7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.976+0000 7f85a05b8740 1 -- 172.21.15.98:0/801175796 >> 172.21.15.98:0/801175796 conn(0x559d47cc3960 msgr2=0x559d47cc3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.976+0000 7f85a05b8740 1 -- 172.21.15.98:0/801175796 shutdown_connections 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.976+0000 7f85a05b8740 1 -- 172.21.15.98:0/801175796 wait complete. 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.977+0000 7f85a05b8740 1 -- start start 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.977+0000 7f85a05b8740 1 Processor -- start 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.977+0000 7f85a05b8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cc7450 0x559d47ccf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.977+0000 7f85a05b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47ccfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.977+0000 7f85a05b8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cd2630 0x559d47cd0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.977+0000 7f85a05b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559d47c2c340 con 0x559d47cc7d70 2024-05-09T08:59:01.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.977+0000 7f85a05b8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559d47be7790 con 0x559d47cd2630 2024-05-09T08:59:01.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.977+0000 7f85a05b8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559d47cd5020 con 0x559d47cc7450 2024-05-09T08:59:01.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.977+0000 7f85a05b8740 10 client.0 _ll_register_callbacks cb 0x559d47c22a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:01.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.977+0000 7f858e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47ccfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:01.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.977+0000 7f858e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47ccfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56684/0 (socket says 172.21.15.98:56684) 2024-05-09T08:59:01.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.977+0000 7f858e7fc640 1 -- 172.21.15.98:0/2442467122 learned_addr learned my addr 172.21.15.98:0/2442467122 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:01.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.977+0000 7f85977fe640 1 --2- 172.21.15.98:0/2442467122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cd2630 0x559d47cd0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:01.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f8596ffd640 1 --2- 172.21.15.98:0/2442467122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cc7450 0x559d47ccf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:01.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f858e7fc640 1 -- 172.21.15.98:0/2442467122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cc7450 msgr2=0x559d47ccf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:01.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f858e7fc640 1 --2- 172.21.15.98:0/2442467122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cc7450 0x559d47ccf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f858e7fc640 1 -- 172.21.15.98:0/2442467122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cd2630 msgr2=0x559d47cd0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:01.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f858e7fc640 1 --2- 172.21.15.98:0/2442467122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cd2630 0x559d47cd0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f858e7fc640 1 -- 172.21.15.98:0/2442467122 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d47940210 con 0x559d47cc7d70 2024-05-09T08:59:01.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f858e7fc640 1 --2- 172.21.15.98:0/2442467122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47ccfb10 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f8580011850 tx=0x7f8580011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c5a92b44b5ec2d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f8576ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f8576ffd640 1 -- 172.21.15.98:0/2442467122 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858000fe70 con 0x559d47cc7d70 2024-05-09T08:59:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f8576ffd640 1 -- 172.21.15.98:0/2442467122 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f8580004020 con 0x559d47cc7d70 2024-05-09T08:59:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f85a05b8740 1 -- 172.21.15.98:0/2442467122 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85880bc9d0 con 0x559d47cc7d70 2024-05-09T08:59:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f8576ffd640 1 -- 172.21.15.98:0/2442467122 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8580004220 con 0x559d47cc7d70 2024-05-09T08:59:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f85a05b8740 10 client.17928 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f85a05b8740 1 -- 172.21.15.98:0/2442467122 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559d47a0c620 con 0x559d47cc7d70 2024-05-09T08:59:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.978+0000 7f8576ffd640 1 -- 172.21.15.98:0/2442467122 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85880bc9d0 con 0x559d47cc7d70 2024-05-09T08:59:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.979+0000 7f8576ffd640 1 -- 172.21.15.98:0/2442467122 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f8580018030 con 0x559d47cc7d70 2024-05-09T08:59:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.979+0000 7f85a05b8740 10 client.17928 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.979+0000 7f85a05b8740 10 client.17928 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.979+0000 7f85a05b8740 20 client.17928 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.979+0000 7f85a05b8740 1 --2- 172.21.15.98:0/2442467122 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d47ce2150 0x559d47ce45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.979+0000 7f85a05b8740 4 client.17928 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.979+0000 7f85a05b8740 1 -- 172.21.15.98:0/2442467122 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d47c39a70 con 0x559d47ce2150 2024-05-09T08:59:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.979+0000 7f8596ffd640 1 --2- 172.21.15.98:0/2442467122 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d47ce2150 0x559d47ce45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.979+0000 7f8596ffd640 1 --2- 172.21.15.98:0/2442467122 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d47ce2150 0x559d47ce45a0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90304ada6446d8c server_cookie=c5fa72ea63365134 in_seq=0 out_seq=0 2024-05-09T08:59:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.979+0000 7f8576ffd640 10 client.17928 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.980+0000 7f8576ffd640 1 -- 172.21.15.98:0/2442467122 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559d47c39a70 con 0x559d47ce2150 2024-05-09T08:59:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.980+0000 7f8576ffd640 10 client.17928 handle_command_reply: tid=0 2024-05-09T08:59:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.980+0000 7f85a05b8740 1 client.17928 shutdown 2024-05-09T08:59:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.980+0000 7f85a05b8740 1 -- 172.21.15.98:0/2442467122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 msgr2=0x559d47ccfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.980+0000 7f85a05b8740 1 --2- 172.21.15.98:0/2442467122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47ccfb10 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f8580011850 tx=0x7f8580011c50 comp rx=0 tx=0).stop 2024-05-09T08:59:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.980+0000 7f85a05b8740 1 -- 172.21.15.98:0/2442467122 shutdown_connections 2024-05-09T08:59:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.980+0000 7f85a05b8740 1 --2- 172.21.15.98:0/2442467122 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d47ce2150 0x559d47ce45a0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.980+0000 7f85a05b8740 1 --2- 172.21.15.98:0/2442467122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cd2630 0x559d47cd0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.980+0000 7f85a05b8740 1 --2- 172.21.15.98:0/2442467122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47ccfb10 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.980+0000 7f85a05b8740 1 --2- 172.21.15.98:0/2442467122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cc7450 0x559d47ccf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:01.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.980+0000 7f8576ffd640 0 client.17928 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:01.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.980+0000 7f85a05b8740 1 -- 172.21.15.98:0/2442467122 >> 172.21.15.98:0/2442467122 conn(0x559d47cc3960 msgr2=0x559d47cc96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:01.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.981+0000 7f85a05b8740 1 -- 172.21.15.98:0/2442467122 shutdown_connections 2024-05-09T08:59:01.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.981+0000 7f85a05b8740 1 -- 172.21.15.98:0/2442467122 wait complete. 2024-05-09T08:59:01.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.981+0000 7f85a05b8740 20 client.17928 trim_cache size 0 max 16384 2024-05-09T08:59:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.995+0000 7f85a05b8740 1 Processor -- start 2024-05-09T08:59:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.995+0000 7f85a05b8740 1 -- start start 2024-05-09T08:59:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.995+0000 7f85a05b8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cc7450 0x559d47cddd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.995+0000 7f85a05b8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cc7d70 0x559d47cc9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.995+0000 7f85a05b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cd2630 0x559d47cc3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.995+0000 7f85a05b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85981aedc0 con 0x559d47cd2630 2024-05-09T08:59:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.995+0000 7f85a05b8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f85981b0900 con 0x559d47cc7d70 2024-05-09T08:59:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.995+0000 7f85a05b8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f85981af490 con 0x559d47cc7450 2024-05-09T08:59:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f85977fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cd2630 0x559d47cc3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f85977fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cd2630 0x559d47cc3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56696/0 (socket says 172.21.15.98:56696) 2024-05-09T08:59:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f85977fe640 1 -- 172.21.15.98:0/2391292426 learned_addr learned my addr 172.21.15.98:0/2391292426 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f8596ffd640 1 --2- 172.21.15.98:0/2391292426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cc7450 0x559d47cddd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f85977fe640 1 -- 172.21.15.98:0/2391292426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cc7450 msgr2=0x559d47cddd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f85977fe640 1 --2- 172.21.15.98:0/2391292426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cc7450 0x559d47cddd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f85977fe640 1 -- 172.21.15.98:0/2391292426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cc7d70 msgr2=0x559d47cc9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f858e7fc640 1 --2- 172.21.15.98:0/2391292426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cc7d70 0x559d47cc9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f85977fe640 1 --2- 172.21.15.98:0/2391292426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cc7d70 0x559d47cc9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f85977fe640 1 -- 172.21.15.98:0/2391292426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d47cce580 con 0x559d47cd2630 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f858e7fc640 1 --2- 172.21.15.98:0/2391292426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cc7d70 0x559d47cc9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f85977fe640 1 --2- 172.21.15.98:0/2391292426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cd2630 0x559d47cc3d40 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f8590007920 tx=0x7f8590005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a920ef7bbe1426df server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f858d7fa640 1 -- 172.21.15.98:0/2391292426 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859001e510 con 0x559d47cd2630 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f858d7fa640 1 -- 172.21.15.98:0/2391292426 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f859001eb90 con 0x559d47cd2630 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.996+0000 7f858d7fa640 1 -- 172.21.15.98:0/2391292426 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859000fd60 con 0x559d47cd2630 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.997+0000 7f85a05b8740 1 -- 172.21.15.98:0/2391292426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cd2630 msgr2=0x559d47cc3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.997+0000 7f85a05b8740 1 --2- 172.21.15.98:0/2391292426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cd2630 0x559d47cc3d40 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f8590007920 tx=0x7f8590005f00 comp rx=0 tx=0).stop 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.997+0000 7f85a05b8740 1 -- 172.21.15.98:0/2391292426 shutdown_connections 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.997+0000 7f85a05b8740 1 --2- 172.21.15.98:0/2391292426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cd2630 0x559d47cc3d40 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.997+0000 7f85a05b8740 1 --2- 172.21.15.98:0/2391292426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cc7d70 0x559d47cc9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.997+0000 7f85a05b8740 1 --2- 172.21.15.98:0/2391292426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cc7450 0x559d47cddd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.997+0000 7f85a05b8740 1 -- 172.21.15.98:0/2391292426 >> 172.21.15.98:0/2391292426 conn(0x559d47cc3960 msgr2=0x559d47cc2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.997+0000 7f85a05b8740 1 -- 172.21.15.98:0/2391292426 shutdown_connections 2024-05-09T08:59:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.997+0000 7f85a05b8740 1 -- 172.21.15.98:0/2391292426 wait complete. 2024-05-09T08:59:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.997+0000 7f85a05b8740 1 -- start start 2024-05-09T08:59:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.997+0000 7f85a05b8740 1 Processor -- start 2024-05-09T08:59:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f85a05b8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cc7450 0x559d47cda2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f85a05b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47cda7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f85a05b8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cd2630 0x559d47cd7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f85a05b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85981b0230 con 0x559d47cc7d70 2024-05-09T08:59:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f85a05b8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f85981afb60 con 0x559d47cd2630 2024-05-09T08:59:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f85a05b8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f85981acdc0 con 0x559d47cc7450 2024-05-09T08:59:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f85a05b8740 10 client.0 _ll_register_callbacks cb 0x7f858000c830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f858e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47cda7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f858e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47cda7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56712/0 (socket says 172.21.15.98:56712) 2024-05-09T08:59:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f858e7fc640 1 -- 172.21.15.98:0/1023578153 learned_addr learned my addr 172.21.15.98:0/1023578153 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f858e7fc640 1 -- 172.21.15.98:0/1023578153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cc7450 msgr2=0x559d47cda2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f85977fe640 1 --2- 172.21.15.98:0/1023578153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cd2630 0x559d47cd7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f858e7fc640 1 --2- 172.21.15.98:0/1023578153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cc7450 0x559d47cda2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f858e7fc640 1 -- 172.21.15.98:0/1023578153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cd2630 msgr2=0x559d47cd7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f858e7fc640 1 --2- 172.21.15.98:0/1023578153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cd2630 0x559d47cd7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f858e7fc640 1 -- 172.21.15.98:0/1023578153 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d47cd7800 con 0x559d47cc7d70 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f85977fe640 1 --2- 172.21.15.98:0/1023578153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cd2630 0x559d47cd7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f858e7fc640 1 --2- 172.21.15.98:0/1023578153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47cda7e0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f85800058c0 tx=0x7f858000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4859844eb8bc73f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f8576ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f8576ffd640 1 -- 172.21.15.98:0/1023578153 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8580011040 con 0x559d47cc7d70 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f8576ffd640 1 -- 172.21.15.98:0/1023578153 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9562+0+0 (secure 0 0 0) 0x7f85800111e0 con 0x559d47cc7d70 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.998+0000 7f85a05b8740 1 -- 172.21.15.98:0/1023578153 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f859001eb90 con 0x559d47cc7d70 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.999+0000 7f8576ffd640 1 -- 172.21.15.98:0/1023578153 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8580037050 con 0x559d47cc7d70 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.999+0000 7f85a05b8740 10 client.17934 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.999+0000 7f85a05b8740 1 -- 172.21.15.98:0/1023578153 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559d47d4d370 con 0x559d47cc7d70 2024-05-09T08:59:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.999+0000 7f8576ffd640 1 -- 172.21.15.98:0/1023578153 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f859001eb90 con 0x559d47cc7d70 2024-05-09T08:59:02.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.999+0000 7f8576ffd640 1 -- 172.21.15.98:0/1023578153 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f8580018070 con 0x559d47cc7d70 2024-05-09T08:59:02.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.999+0000 7f85a05b8740 10 client.17934 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:02.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.999+0000 7f85a05b8740 10 client.17934 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:02.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.999+0000 7f85a05b8740 20 client.17934 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:02.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.999+0000 7f85a05b8740 1 --2- 172.21.15.98:0/1023578153 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d47d4db70 0x559d47d4ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:02.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.999+0000 7f85a05b8740 4 client.17934 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:59:02.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.999+0000 7f85a05b8740 1 -- 172.21.15.98:0/1023578153 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559d47c33dc0 con 0x559d47d4db70 2024-05-09T08:59:02.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:01.999+0000 7f8596ffd640 1 --2- 172.21.15.98:0/1023578153 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d47d4db70 0x559d47d4ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:02.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.000+0000 7f8596ffd640 1 --2- 172.21.15.98:0/1023578153 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d47d4db70 0x559d47d4ff90 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dfbdce68cda7d5f server_cookie=d90ce1bcd54066c3 in_seq=0 out_seq=0 2024-05-09T08:59:02.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.000+0000 7f8576ffd640 10 client.17934 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.006+0000 7f8576ffd640 1 -- 172.21.15.98:0/1023578153 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559d47c33dc0 con 0x559d47d4db70 2024-05-09T08:59:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.006+0000 7f8576ffd640 10 client.17934 handle_command_reply: tid=0 2024-05-09T08:59:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f85a05b8740 1 client.17934 shutdown 2024-05-09T08:59:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f85a05b8740 1 -- 172.21.15.98:0/1023578153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 msgr2=0x559d47cda7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f85a05b8740 1 --2- 172.21.15.98:0/1023578153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47cda7e0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f85800058c0 tx=0x7f858000f790 comp rx=0 tx=0).stop 2024-05-09T08:59:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f85a05b8740 1 -- 172.21.15.98:0/1023578153 shutdown_connections 2024-05-09T08:59:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f85a05b8740 1 --2- 172.21.15.98:0/1023578153 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d47d4db70 0x559d47d4ff90 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f85a05b8740 1 --2- 172.21.15.98:0/1023578153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d47cd2630 0x559d47cd7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f85a05b8740 1 --2- 172.21.15.98:0/1023578153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d47cc7d70 0x559d47cda7e0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f8576ffd640 0 client.17934 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f85a05b8740 1 --2- 172.21.15.98:0/1023578153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d47cc7450 0x559d47cda2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f85a05b8740 1 -- 172.21.15.98:0/1023578153 >> 172.21.15.98:0/1023578153 conn(0x559d47cc3960 msgr2=0x559d47cc2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f85a05b8740 1 -- 172.21.15.98:0/1023578153 shutdown_connections 2024-05-09T08:59:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f85a05b8740 1 -- 172.21.15.98:0/1023578153 wait complete. 2024-05-09T08:59:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f85a05b8740 20 client.17934 trim_cache size 0 max 16384 2024-05-09T08:59:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:59:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:59:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:59:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:59:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f859d8a5640 1 -- 172.21.15.98:0/2617077033 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f85840a2da0 msgr2=0x7f85840a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f859d8a5640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f85840a2da0 0x7f85840a5260 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8590005fd0 tx=0x7f8590005db0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f859d8a5640 1 -- 172.21.15.98:0/2617077033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8598100a80 msgr2=0x7f8598197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.007+0000 7f859d8a5640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8598100a80 0x7f8598197fb0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f8588005ab0 tx=0x7f85880042f0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.008+0000 7f859d8a5640 1 -- 172.21.15.98:0/2617077033 shutdown_connections 2024-05-09T08:59:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.008+0000 7f859d8a5640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f85840a2da0 0x7f85840a5260 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.008+0000 7f859d8a5640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85981056d0 0x7f859819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.008+0000 7f859d8a5640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8598101440 0x7f85981984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.008+0000 7f859d8a5640 1 --2- 172.21.15.98:0/2617077033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8598100a80 0x7f8598197fb0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.008+0000 7f859d8a5640 1 -- 172.21.15.98:0/2617077033 >> 172.21.15.98:0/2617077033 conn(0x7f8598076530 msgr2=0x7f8598104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.008+0000 7f859d8a5640 1 -- 172.21.15.98:0/2617077033 shutdown_connections 2024-05-09T08:59:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:02.008+0000 7f859d8a5640 1 -- 172.21.15.98:0/2617077033 wait complete. 2024-05-09T08:59:02.310 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:02 smithi142 ceph-mon[34625]: pgmap v260: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:59:02.310 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:02 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:59:02.310 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:02 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:59:02.310 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:02 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:59:02.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:02 smithi098 ceph-mon[30092]: pgmap v260: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:59:02.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:02 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:59:02.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:02 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:59:02.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:02 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:59:02.388 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:02 smithi105 ceph-mon[34498]: pgmap v260: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:59:02.388 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:02 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:59:02.388 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:02 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:59:02.388 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:02 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T08:59:04.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:04 smithi098 ceph-mon[30092]: pgmap v261: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T08:59:04.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:04 smithi142 ceph-mon[34625]: pgmap v261: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T08:59:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:04 smithi105 ceph-mon[34498]: pgmap v261: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T08:59:06.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:06 smithi098 ceph-mon[30092]: pgmap v262: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-05-09T08:59:06.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:06 smithi142 ceph-mon[34625]: pgmap v262: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-05-09T08:59:06.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:06 smithi105 ceph-mon[34498]: pgmap v262: 129 pgs: 129 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-05-09T08:59:07.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:07 smithi098 ceph-mon[30092]: osdmap e102: 12 total, 12 up, 12 in 2024-05-09T08:59:07.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:07 smithi142 ceph-mon[34625]: osdmap e102: 12 total, 12 up, 12 in 2024-05-09T08:59:07.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:07 smithi105 ceph-mon[34498]: osdmap e102: 12 total, 12 up, 12 in 2024-05-09T08:59:08.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:08 smithi098 ceph-mon[30092]: pgmap v264: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-09T08:59:08.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:08 smithi142 ceph-mon[34625]: pgmap v264: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-09T08:59:08.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:08 smithi105 ceph-mon[34498]: pgmap v264: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-09T08:59:10.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:10 smithi098 ceph-mon[30092]: pgmap v265: 129 pgs: 6 active+clean+snaptrim, 106 active+clean, 17 active+clean+snaptrim_wait; 49 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-09T08:59:10.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:10 smithi142 ceph-mon[34625]: pgmap v265: 129 pgs: 6 active+clean+snaptrim, 106 active+clean, 17 active+clean+snaptrim_wait; 49 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-09T08:59:10.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:10 smithi105 ceph-mon[34498]: pgmap v265: 129 pgs: 6 active+clean+snaptrim, 106 active+clean, 17 active+clean+snaptrim_wait; 49 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-09T08:59:12.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:12 smithi098 ceph-mon[30092]: pgmap v266: 129 pgs: 6 active+clean+snaptrim, 106 active+clean, 17 active+clean+snaptrim_wait; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-09T08:59:12.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:12 smithi142 ceph-mon[34625]: pgmap v266: 129 pgs: 6 active+clean+snaptrim, 106 active+clean, 17 active+clean+snaptrim_wait; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-09T08:59:12.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:12 smithi105 ceph-mon[34498]: pgmap v266: 129 pgs: 6 active+clean+snaptrim, 106 active+clean, 17 active+clean+snaptrim_wait; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-09T08:59:14.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:14 smithi098 ceph-mon[30092]: pgmap v267: 129 pgs: 2 active+clean+snaptrim, 123 active+clean, 4 active+clean+snaptrim_wait; 41 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T08:59:14.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:14 smithi142 ceph-mon[34625]: pgmap v267: 129 pgs: 2 active+clean+snaptrim, 123 active+clean, 4 active+clean+snaptrim_wait; 41 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T08:59:14.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:14 smithi105 ceph-mon[34498]: pgmap v267: 129 pgs: 2 active+clean+snaptrim, 123 active+clean, 4 active+clean+snaptrim_wait; 41 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T08:59:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.106+0000 7f11e1c9a640 1 -- 172.21.15.98:0/3442595884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11dc106720 msgr2=0x7f11dc106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.106+0000 7f11e1c9a640 1 --2- 172.21.15.98:0/3442595884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11dc106720 0x7f11dc106b00 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f11cc009930 tx=0x7f11cc02a420 comp rx=0 tx=0).stop 2024-05-09T08:59:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.106+0000 7f11e1c9a640 1 -- 172.21.15.98:0/3442595884 shutdown_connections 2024-05-09T08:59:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.106+0000 7f11e1c9a640 1 --2- 172.21.15.98:0/3442595884 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11dc101120 0x7f11dc10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.106+0000 7f11e1c9a640 1 --2- 172.21.15.98:0/3442595884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11dc100760 0x7f11dc100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.106+0000 7f11e1c9a640 1 --2- 172.21.15.98:0/3442595884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11dc106720 0x7f11dc106b00 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.106+0000 7f11e1c9a640 1 -- 172.21.15.98:0/3442595884 >> 172.21.15.98:0/3442595884 conn(0x7f11dc0778b0 msgr2=0x7f11dc077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.106+0000 7f11e1c9a640 1 -- 172.21.15.98:0/3442595884 shutdown_connections 2024-05-09T08:59:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.107+0000 7f11e1c9a640 1 -- 172.21.15.98:0/3442595884 wait complete. 2024-05-09T08:59:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.107+0000 7f11e1c9a640 1 Processor -- start 2024-05-09T08:59:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.107+0000 7f11e1c9a640 1 -- start start 2024-05-09T08:59:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11e1c9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11dc100760 0x7f11dc198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11e1c9a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11dc101120 0x7f11dc199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11e1c9a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11dc106720 0x7f11dc1a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11e1c9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc110b30 con 0x7f11dc100760 2024-05-09T08:59:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11e1c9a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1109b0 con 0x7f11dc106720 2024-05-09T08:59:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11e1c9a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc110cb0 con 0x7f11dc101120 2024-05-09T08:59:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11db7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11dc100760 0x7f11dc198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11db7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11dc100760 0x7f11dc198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57490/0 (socket says 172.21.15.98:57490) 2024-05-09T08:59:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11db7fe640 1 -- 172.21.15.98:0/1729705071 learned_addr learned my addr 172.21.15.98:0/1729705071 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11daffd640 1 --2- 172.21.15.98:0/1729705071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11dc101120 0x7f11dc199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11dbfff640 1 --2- 172.21.15.98:0/1729705071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11dc106720 0x7f11dc1a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11db7fe640 1 -- 172.21.15.98:0/1729705071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11dc101120 msgr2=0x7f11dc199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11db7fe640 1 --2- 172.21.15.98:0/1729705071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11dc101120 0x7f11dc199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11db7fe640 1 -- 172.21.15.98:0/1729705071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11dc106720 msgr2=0x7f11dc1a3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:15.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11db7fe640 1 --2- 172.21.15.98:0/1729705071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11dc106720 0x7f11dc1a3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11db7fe640 1 -- 172.21.15.98:0/1729705071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11dc1a44b0 con 0x7f11dc100760 2024-05-09T08:59:15.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.108+0000 7f11db7fe640 1 --2- 172.21.15.98:0/1729705071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11dc100760 0x7f11dc198ef0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f11cc009230 tx=0x7f11cc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:15.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.109+0000 7f11d8ff9640 1 -- 172.21.15.98:0/1729705071 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11cc03f030 con 0x7f11dc100760 2024-05-09T08:59:15.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.109+0000 7f11e1c9a640 1 -- 172.21.15.98:0/1729705071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11dc1a4680 con 0x7f11dc100760 2024-05-09T08:59:15.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.109+0000 7f11d8ff9640 1 -- 172.21.15.98:0/1729705071 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11cc003ad0 con 0x7f11dc100760 2024-05-09T08:59:15.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.109+0000 7f11d8ff9640 1 -- 172.21.15.98:0/1729705071 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11cc035070 con 0x7f11dc100760 2024-05-09T08:59:15.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.109+0000 7f11e1c9a640 1 -- 172.21.15.98:0/1729705071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11dc1a4b40 con 0x7f11dc100760 2024-05-09T08:59:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.112+0000 7f11d8ff9640 1 -- 172.21.15.98:0/1729705071 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f11cc042030 con 0x7f11dc100760 2024-05-09T08:59:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.112+0000 7f11e49a7740 1 Processor -- start 2024-05-09T08:59:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.112+0000 7f11e49a7740 1 -- start start 2024-05-09T08:59:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.112+0000 7f11e49a7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2aad450 0x55feb2aad830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.112+0000 7f11e49a7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aadd70 0x55feb2ab80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.112+0000 7f11e49a7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2ab8630 0x55feb2abaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.112+0000 7f11e49a7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55feb2a12340 con 0x55feb2ab8630 2024-05-09T08:59:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.112+0000 7f11e49a7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55feb29ba290 con 0x55feb2aadd70 2024-05-09T08:59:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.112+0000 7f11e49a7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55feb2a0e300 con 0x55feb2aad450 2024-05-09T08:59:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.112+0000 7f11daffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2ab8630 0x55feb2abaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.112+0000 7f11d8ff9640 1 --2- 172.21.15.98:0/1729705071 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11c40a2da0 0x7f11c40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.112+0000 7f11daffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2ab8630 0x55feb2abaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57496/0 (socket says 172.21.15.98:57496) 2024-05-09T08:59:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.112+0000 7f11daffd640 1 -- 172.21.15.98:0/1661134015 learned_addr learned my addr 172.21.15.98:0/1661134015 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.113+0000 7f11db7fe640 1 --2- 172.21.15.98:0/1661134015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aadd70 0x55feb2ab80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.113+0000 7f11dbfff640 1 --2- 172.21.15.98:0/1661134015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2aad450 0x55feb2aad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.113+0000 7f11dbfff640 1 --2- 172.21.15.98:0/1729705071 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11c40a2da0 0x7f11c40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.113+0000 7f11d8ff9640 1 -- 172.21.15.98:0/1729705071 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7f11cc0efce0 con 0x7f11dc100760 2024-05-09T08:59:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.113+0000 7f11daffd640 1 -- 172.21.15.98:0/1661134015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2aad450 msgr2=0x55feb2aad830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.113+0000 7f11daffd640 1 --2- 172.21.15.98:0/1661134015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2aad450 0x55feb2aad830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.113+0000 7f11daffd640 1 -- 172.21.15.98:0/1661134015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aadd70 msgr2=0x55feb2ab80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.113+0000 7f11daffd640 1 --2- 172.21.15.98:0/1661134015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aadd70 0x55feb2ab80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.113+0000 7f11daffd640 1 -- 172.21.15.98:0/1661134015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55feb29de8e0 con 0x55feb2ab8630 2024-05-09T08:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.113+0000 7f11daffd640 1 --2- 172.21.15.98:0/1661134015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2ab8630 0x55feb2abaa20 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f11d000b740 tx=0x7f11d000bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224e23a6e6d5281d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.113+0000 7f11dbfff640 1 --2- 172.21.15.98:0/1729705071 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11c40a2da0 0x7f11c40a5260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f11d400a780 tx=0x7f11d400a4d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.113+0000 7f11c1ffb640 1 -- 172.21.15.98:0/1661134015 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0003dd0 con 0x55feb2ab8630 2024-05-09T08:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.113+0000 7f11c1ffb640 1 -- 172.21.15.98:0/1661134015 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11d0015070 con 0x55feb2ab8630 2024-05-09T08:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.113+0000 7f11c1ffb640 1 -- 172.21.15.98:0/1661134015 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0004e40 con 0x55feb2ab8630 2024-05-09T08:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.114+0000 7f11e49a7740 1 -- 172.21.15.98:0/1661134015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2ab8630 msgr2=0x55feb2abaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.114+0000 7f11e49a7740 1 --2- 172.21.15.98:0/1661134015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2ab8630 0x55feb2abaa20 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f11d000b740 tx=0x7f11d000bc00 comp rx=0 tx=0).stop 2024-05-09T08:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.114+0000 7f11e49a7740 1 -- 172.21.15.98:0/1661134015 shutdown_connections 2024-05-09T08:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.114+0000 7f11e49a7740 1 --2- 172.21.15.98:0/1661134015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2ab8630 0x55feb2abaa20 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.114+0000 7f11e49a7740 1 --2- 172.21.15.98:0/1661134015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aadd70 0x55feb2ab80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.114+0000 7f11e49a7740 1 --2- 172.21.15.98:0/1661134015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2aad450 0x55feb2aad830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.114+0000 7f11e49a7740 1 -- 172.21.15.98:0/1661134015 >> 172.21.15.98:0/1661134015 conn(0x55feb2aa9960 msgr2=0x55feb2aa9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.114+0000 7f11e49a7740 1 -- 172.21.15.98:0/1661134015 shutdown_connections 2024-05-09T08:59:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.114+0000 7f11e49a7740 1 -- 172.21.15.98:0/1661134015 wait complete. 2024-05-09T08:59:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.114+0000 7f11e49a7740 1 -- start start 2024-05-09T08:59:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11e49a7740 1 Processor -- start 2024-05-09T08:59:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11e49a7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2aad450 0x55feb2ab55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11e49a7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aadd70 0x55feb2ab5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11e49a7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2ab8630 0x55feb2ab6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11e49a7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55feb2abb260 con 0x55feb2aadd70 2024-05-09T08:59:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11e49a7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55feb29cd790 con 0x55feb2ab8630 2024-05-09T08:59:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11e49a7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55feb2abb3e0 con 0x55feb2aad450 2024-05-09T08:59:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11e49a7740 10 client.0 _ll_register_callbacks cb 0x55feb2a08a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11daffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aadd70 0x55feb2ab5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11daffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aadd70 0x55feb2ab5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57508/0 (socket says 172.21.15.98:57508) 2024-05-09T08:59:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11daffd640 1 -- 172.21.15.98:0/170880053 learned_addr learned my addr 172.21.15.98:0/170880053 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11db7fe640 1 --2- 172.21.15.98:0/170880053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2aad450 0x55feb2ab55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11dbfff640 1 --2- 172.21.15.98:0/170880053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2ab8630 0x55feb2ab6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11daffd640 1 -- 172.21.15.98:0/170880053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2aad450 msgr2=0x55feb2ab55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11daffd640 1 --2- 172.21.15.98:0/170880053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2aad450 0x55feb2ab55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11daffd640 1 -- 172.21.15.98:0/170880053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2ab8630 msgr2=0x55feb2ab6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11daffd640 1 --2- 172.21.15.98:0/170880053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2ab8630 0x55feb2ab6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11daffd640 1 -- 172.21.15.98:0/170880053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55feb2726210 con 0x55feb2aadd70 2024-05-09T08:59:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11dbfff640 1 --2- 172.21.15.98:0/170880053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2ab8630 0x55feb2ab6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11daffd640 1 --2- 172.21.15.98:0/170880053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aadd70 0x55feb2ab5af0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f11d0004ae0 tx=0x7f11d0004380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d86beeb9cd9ae5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.115+0000 7f11baffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.116+0000 7f11baffd640 1 -- 172.21.15.98:0/170880053 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0028060 con 0x55feb2aadd70 2024-05-09T08:59:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.116+0000 7f11baffd640 1 -- 172.21.15.98:0/170880053 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7f11d0011440 con 0x55feb2aadd70 2024-05-09T08:59:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.116+0000 7f11e49a7740 1 -- 172.21.15.98:0/170880053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11d0015070 con 0x55feb2aadd70 2024-05-09T08:59:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.116+0000 7f11baffd640 1 -- 172.21.15.98:0/170880053 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f11d002f050 con 0x55feb2aadd70 2024-05-09T08:59:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.116+0000 7f11e49a7740 10 client.17946 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.116+0000 7f11e49a7740 1 -- 172.21.15.98:0/170880053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55feb27f2620 con 0x55feb2aadd70 2024-05-09T08:59:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.116+0000 7f11baffd640 1 -- 172.21.15.98:0/170880053 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11d0015070 con 0x55feb2aadd70 2024-05-09T08:59:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.116+0000 7f11baffd640 1 -- 172.21.15.98:0/170880053 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f11d0022070 con 0x55feb2aadd70 2024-05-09T08:59:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.116+0000 7f11e49a7740 10 client.17946 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.116+0000 7f11e49a7740 10 client.17946 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.116+0000 7f11e49a7740 20 client.17946 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.116+0000 7f11e49a7740 1 --2- 172.21.15.98:0/170880053 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55feb2ac82a0 0x55feb2aca690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.116+0000 7f11e49a7740 4 client.17946 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.116+0000 7f11e49a7740 1 -- 172.21.15.98:0/170880053 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55feb2a1fa70 con 0x55feb2ac82a0 2024-05-09T08:59:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.117+0000 7f11db7fe640 1 --2- 172.21.15.98:0/170880053 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55feb2ac82a0 0x55feb2aca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.117+0000 7f11baffd640 10 client.17946 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.117+0000 7f11db7fe640 1 --2- 172.21.15.98:0/170880053 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55feb2ac82a0 0x55feb2aca690 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84dc21888cccdec8 server_cookie=dfb1788879fe26e6 in_seq=0 out_seq=0 2024-05-09T08:59:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11baffd640 1 -- 172.21.15.98:0/170880053 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55feb2a1fa70 con 0x55feb2ac82a0 2024-05-09T08:59:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11baffd640 10 client.17946 handle_command_reply: tid=0 2024-05-09T08:59:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11e49a7740 1 client.17946 shutdown 2024-05-09T08:59:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11e49a7740 1 -- 172.21.15.98:0/170880053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aadd70 msgr2=0x55feb2ab5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11e49a7740 1 --2- 172.21.15.98:0/170880053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aadd70 0x55feb2ab5af0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f11d0004ae0 tx=0x7f11d0004380 comp rx=0 tx=0).stop 2024-05-09T08:59:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11e49a7740 1 -- 172.21.15.98:0/170880053 shutdown_connections 2024-05-09T08:59:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11e49a7740 1 --2- 172.21.15.98:0/170880053 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55feb2ac82a0 0x55feb2aca690 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11e49a7740 1 --2- 172.21.15.98:0/170880053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2ab8630 0x55feb2ab6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11e49a7740 1 --2- 172.21.15.98:0/170880053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aadd70 0x55feb2ab5af0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11baffd640 0 client.17946 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11e49a7740 1 --2- 172.21.15.98:0/170880053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2aad450 0x55feb2ab55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11e49a7740 1 -- 172.21.15.98:0/170880053 >> 172.21.15.98:0/170880053 conn(0x55feb2aa9960 msgr2=0x55feb2aaf690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11e49a7740 1 -- 172.21.15.98:0/170880053 shutdown_connections 2024-05-09T08:59:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11e49a7740 1 -- 172.21.15.98:0/170880053 wait complete. 2024-05-09T08:59:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.118+0000 7f11e49a7740 20 client.17946 trim_cache size 0 max 16384 2024-05-09T08:59:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11e49a7740 1 Processor -- start 2024-05-09T08:59:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11e49a7740 1 -- start start 2024-05-09T08:59:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11e49a7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aad450 0x55feb2ac4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11e49a7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aadd70 0x55feb2ab5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11e49a7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2ab8630 0x55feb2acfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11e49a7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1aef40 con 0x55feb2aad450 2024-05-09T08:59:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11e49a7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1b0a80 con 0x55feb2aadd70 2024-05-09T08:59:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11e49a7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1af610 con 0x55feb2ab8630 2024-05-09T08:59:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11db7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aad450 0x55feb2ac4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11db7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aad450 0x55feb2ac4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57524/0 (socket says 172.21.15.98:57524) 2024-05-09T08:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11db7fe640 1 -- 172.21.15.98:0/2131503609 learned_addr learned my addr 172.21.15.98:0/2131503609 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11db7fe640 1 -- 172.21.15.98:0/2131503609 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2ab8630 msgr2=0x55feb2acfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11db7fe640 1 --2- 172.21.15.98:0/2131503609 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2ab8630 0x55feb2acfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11db7fe640 1 -- 172.21.15.98:0/2131503609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aadd70 msgr2=0x55feb2ab5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11db7fe640 1 --2- 172.21.15.98:0/2131503609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aadd70 0x55feb2ab5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11db7fe640 1 -- 172.21.15.98:0/2131503609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55feb2aae3e0 con 0x55feb2aad450 2024-05-09T08:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11db7fe640 1 --2- 172.21.15.98:0/2131503609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aad450 0x55feb2ac4120 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f11cc0bbdd0 tx=0x7f11cc0f21f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f404d79e83c03bc1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11c17fa640 1 -- 172.21.15.98:0/2131503609 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11cc049040 con 0x55feb2aad450 2024-05-09T08:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11c17fa640 1 -- 172.21.15.98:0/2131503609 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11cc03b460 con 0x55feb2aad450 2024-05-09T08:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.133+0000 7f11c17fa640 1 -- 172.21.15.98:0/2131503609 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11cc0ed3a0 con 0x55feb2aad450 2024-05-09T08:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.134+0000 7f11e49a7740 1 -- 172.21.15.98:0/2131503609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aad450 msgr2=0x55feb2ac4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.134+0000 7f11e49a7740 1 --2- 172.21.15.98:0/2131503609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aad450 0x55feb2ac4120 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f11cc0bbdd0 tx=0x7f11cc0f21f0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.134+0000 7f11e49a7740 1 -- 172.21.15.98:0/2131503609 shutdown_connections 2024-05-09T08:59:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.134+0000 7f11e49a7740 1 --2- 172.21.15.98:0/2131503609 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2ab8630 0x55feb2acfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.134+0000 7f11e49a7740 1 --2- 172.21.15.98:0/2131503609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aadd70 0x55feb2ab5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.134+0000 7f11e49a7740 1 --2- 172.21.15.98:0/2131503609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aad450 0x55feb2ac4120 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.134+0000 7f11e49a7740 1 -- 172.21.15.98:0/2131503609 >> 172.21.15.98:0/2131503609 conn(0x55feb2aa9960 msgr2=0x55feb2ab5570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.134+0000 7f11e49a7740 1 -- 172.21.15.98:0/2131503609 shutdown_connections 2024-05-09T08:59:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.134+0000 7f11e49a7740 1 -- 172.21.15.98:0/2131503609 wait complete. 2024-05-09T08:59:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.134+0000 7f11e49a7740 1 -- start start 2024-05-09T08:59:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11e49a7740 1 Processor -- start 2024-05-09T08:59:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11e49a7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aad450 0x55feb2ac0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11e49a7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aadd70 0x55feb2abd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11e49a7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2ab8630 0x55feb2abdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11e49a7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1b03b0 con 0x55feb2aadd70 2024-05-09T08:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11e49a7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1afce0 con 0x55feb2aad450 2024-05-09T08:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11e49a7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1acf40 con 0x55feb2ab8630 2024-05-09T08:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11e49a7740 10 client.0 _ll_register_callbacks cb 0x7f11cc0bb980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11daffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aadd70 0x55feb2abd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11daffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aadd70 0x55feb2abd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57528/0 (socket says 172.21.15.98:57528) 2024-05-09T08:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11daffd640 1 -- 172.21.15.98:0/1989992741 learned_addr learned my addr 172.21.15.98:0/1989992741 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11db7fe640 1 --2- 172.21.15.98:0/1989992741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aad450 0x55feb2ac0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11daffd640 1 -- 172.21.15.98:0/1989992741 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2ab8630 msgr2=0x55feb2abdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11daffd640 1 --2- 172.21.15.98:0/1989992741 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2ab8630 0x55feb2abdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11daffd640 1 -- 172.21.15.98:0/1989992741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aad450 msgr2=0x55feb2ac0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11daffd640 1 --2- 172.21.15.98:0/1989992741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aad450 0x55feb2ac0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11daffd640 1 -- 172.21.15.98:0/1989992741 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55feb2b2e540 con 0x55feb2aadd70 2024-05-09T08:59:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11db7fe640 1 --2- 172.21.15.98:0/1989992741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aad450 0x55feb2ac0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11daffd640 1 --2- 172.21.15.98:0/1989992741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aadd70 0x55feb2abd600 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f11d0019970 tx=0x7f11d00048d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa40aed6702e2056 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11baffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11baffd640 1 -- 172.21.15.98:0/1989992741 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0028050 con 0x55feb2aadd70 2024-05-09T08:59:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11baffd640 1 -- 172.21.15.98:0/1989992741 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7f11d0019420 con 0x55feb2aadd70 2024-05-09T08:59:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.135+0000 7f11e49a7740 1 -- 172.21.15.98:0/1989992741 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11cc03b460 con 0x55feb2aadd70 2024-05-09T08:59:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.136+0000 7f11baffd640 1 -- 172.21.15.98:0/1989992741 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f11d0019620 con 0x55feb2aadd70 2024-05-09T08:59:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.136+0000 7f11e49a7740 10 client.17952 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.136+0000 7f11e49a7740 1 -- 172.21.15.98:0/1989992741 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55feb2b2f620 con 0x55feb2aadd70 2024-05-09T08:59:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.136+0000 7f11baffd640 1 -- 172.21.15.98:0/1989992741 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11cc03b460 con 0x55feb2aadd70 2024-05-09T08:59:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.136+0000 7f11baffd640 1 -- 172.21.15.98:0/1989992741 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f11d0022040 con 0x55feb2aadd70 2024-05-09T08:59:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.136+0000 7f11e49a7740 10 client.17952 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.136+0000 7f11e49a7740 10 client.17952 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.136+0000 7f11e49a7740 20 client.17952 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.136+0000 7f11e49a7740 1 --2- 172.21.15.98:0/1989992741 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55feb2b2fcb0 0x55feb2b320d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.136+0000 7f11e49a7740 4 client.17952 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:59:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.136+0000 7f11e49a7740 1 -- 172.21.15.98:0/1989992741 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55feb2aa97b0 con 0x55feb2b2fcb0 2024-05-09T08:59:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.136+0000 7f11db7fe640 1 --2- 172.21.15.98:0/1989992741 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55feb2b2fcb0 0x55feb2b320d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.137+0000 7f11baffd640 10 client.17952 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.137+0000 7f11db7fe640 1 --2- 172.21.15.98:0/1989992741 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55feb2b2fcb0 0x55feb2b320d0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79011786095c2319 server_cookie=9ada28ea660b7a in_seq=0 out_seq=0 2024-05-09T08:59:15.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11baffd640 1 -- 172.21.15.98:0/1989992741 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55feb2aa97b0 con 0x55feb2b2fcb0 2024-05-09T08:59:15.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11baffd640 10 client.17952 handle_command_reply: tid=0 2024-05-09T08:59:15.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11e49a7740 1 client.17952 shutdown 2024-05-09T08:59:15.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11e49a7740 1 -- 172.21.15.98:0/1989992741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aadd70 msgr2=0x55feb2abd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:15.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11e49a7740 1 --2- 172.21.15.98:0/1989992741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aadd70 0x55feb2abd600 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f11d0019970 tx=0x7f11d00048d0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11e49a7740 1 -- 172.21.15.98:0/1989992741 shutdown_connections 2024-05-09T08:59:15.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11e49a7740 1 --2- 172.21.15.98:0/1989992741 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55feb2b2fcb0 0x55feb2b320d0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11e49a7740 1 --2- 172.21.15.98:0/1989992741 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55feb2ab8630 0x55feb2abdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11e49a7740 1 --2- 172.21.15.98:0/1989992741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55feb2aadd70 0x55feb2abd600 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11baffd640 0 client.17952 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11e49a7740 1 --2- 172.21.15.98:0/1989992741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55feb2aad450 0x55feb2ac0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11e49a7740 1 -- 172.21.15.98:0/1989992741 >> 172.21.15.98:0/1989992741 conn(0x55feb2aa9960 msgr2=0x55feb2ab5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11e49a7740 1 -- 172.21.15.98:0/1989992741 shutdown_connections 2024-05-09T08:59:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11e49a7740 1 -- 172.21.15.98:0/1989992741 wait complete. 2024-05-09T08:59:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.143+0000 7f11e49a7740 20 client.17952 trim_cache size 0 max 16384 2024-05-09T08:59:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:59:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:59:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:59:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.144+0000 7f11e1c9a640 1 -- 172.21.15.98:0/1729705071 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11c40a2da0 msgr2=0x7f11c40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.144+0000 7f11e1c9a640 1 --2- 172.21.15.98:0/1729705071 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11c40a2da0 0x7f11c40a5260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f11d400a780 tx=0x7f11d400a4d0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.144+0000 7f11e1c9a640 1 -- 172.21.15.98:0/1729705071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11dc100760 msgr2=0x7f11dc198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.144+0000 7f11e1c9a640 1 --2- 172.21.15.98:0/1729705071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11dc100760 0x7f11dc198ef0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f11cc009230 tx=0x7f11cc004270 comp rx=0 tx=0).stop 2024-05-09T08:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.144+0000 7f11e1c9a640 1 -- 172.21.15.98:0/1729705071 shutdown_connections 2024-05-09T08:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.144+0000 7f11e1c9a640 1 --2- 172.21.15.98:0/1729705071 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11c40a2da0 0x7f11c40a5260 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.144+0000 7f11e1c9a640 1 --2- 172.21.15.98:0/1729705071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11dc106720 0x7f11dc1a3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.144+0000 7f11e1c9a640 1 --2- 172.21.15.98:0/1729705071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11dc101120 0x7f11dc199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.144+0000 7f11e1c9a640 1 --2- 172.21.15.98:0/1729705071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11dc100760 0x7f11dc198ef0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.144+0000 7f11e1c9a640 1 -- 172.21.15.98:0/1729705071 >> 172.21.15.98:0/1729705071 conn(0x7f11dc0778b0 msgr2=0x7f11dc103280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.144+0000 7f11e1c9a640 1 -- 172.21.15.98:0/1729705071 shutdown_connections 2024-05-09T08:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:15.144+0000 7f11e1c9a640 1 -- 172.21.15.98:0/1729705071 wait complete. 2024-05-09T08:59:16.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:16 smithi098 ceph-mon[30092]: pgmap v268: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 28 KiB/s wr, 13 op/s 2024-05-09T08:59:16.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:16 smithi142 ceph-mon[34625]: pgmap v268: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 28 KiB/s wr, 13 op/s 2024-05-09T08:59:16.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:16 smithi105 ceph-mon[34498]: pgmap v268: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 28 KiB/s wr, 13 op/s 2024-05-09T08:59:16.671 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:59:16.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:59:16.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.787+0000 7fb2d9f77640 1 -- 172.21.15.98:0/3317683825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2d4107810 msgr2=0x7fb2d4107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:16.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.787+0000 7fb2d9f77640 1 --2- 172.21.15.98:0/3317683825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2d4107810 0x7fb2d4107bf0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4009930 tx=0x7fb2c402a420 comp rx=0 tx=0).stop 2024-05-09T08:59:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.787+0000 7fb2d9f77640 1 -- 172.21.15.98:0/3317683825 shutdown_connections 2024-05-09T08:59:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.787+0000 7fb2d9f77640 1 --2- 172.21.15.98:0/3317683825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2d41021c0 0x7fb2d410e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.787+0000 7fb2d9f77640 1 --2- 172.21.15.98:0/3317683825 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2d4101800 0x7fb2d4101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.787+0000 7fb2d9f77640 1 --2- 172.21.15.98:0/3317683825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2d4107810 0x7fb2d4107bf0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.787+0000 7fb2d9f77640 1 -- 172.21.15.98:0/3317683825 >> 172.21.15.98:0/3317683825 conn(0x7fb2d40fd2d0 msgr2=0x7fb2d40ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.787+0000 7fb2d9f77640 1 -- 172.21.15.98:0/3317683825 shutdown_connections 2024-05-09T08:59:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.787+0000 7fb2d9f77640 1 -- 172.21.15.98:0/3317683825 wait complete. 2024-05-09T08:59:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.788+0000 7fb2d9f77640 1 Processor -- start 2024-05-09T08:59:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.788+0000 7fb2d9f77640 1 -- start start 2024-05-09T08:59:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.788+0000 7fb2d9f77640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2d4101800 0x7fb2d41981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.788+0000 7fb2d9f77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2d41021c0 0x7fb2d4198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.788+0000 7fb2d9f77640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2d4107810 0x7fb2d419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.788+0000 7fb2d9f77640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d4111c20 con 0x7fb2d41021c0 2024-05-09T08:59:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.788+0000 7fb2d9f77640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d4111aa0 con 0x7fb2d4107810 2024-05-09T08:59:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.788+0000 7fb2d9f77640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d4111da0 con 0x7fb2d4101800 2024-05-09T08:59:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2d41021c0 0x7fb2d4198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2d41021c0 0x7fb2d4198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57172/0 (socket says 172.21.15.98:57172) 2024-05-09T08:59:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/4163586488 learned_addr learned my addr 172.21.15.98:0/4163586488 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d3fff640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2d4107810 0x7fb2d419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d37fe640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2d4101800 0x7fb2d41981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/4163586488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2d4101800 msgr2=0x7fb2d41981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d2ffd640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2d4101800 0x7fb2d41981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/4163586488 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2d4107810 msgr2=0x7fb2d419ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d2ffd640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2d4107810 0x7fb2d419ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/4163586488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d419d180 con 0x7fb2d41021c0 2024-05-09T08:59:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d3fff640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2d4107810 0x7fb2d419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d2ffd640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2d41021c0 0x7fb2d4198720 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800d950 tx=0x7fb2c800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:16.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d0ff9640 1 -- 172.21.15.98:0/4163586488 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8017070 con 0x7fb2d41021c0 2024-05-09T08:59:16.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d9f77640 1 -- 172.21.15.98:0/4163586488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2d419d470 con 0x7fb2d41021c0 2024-05-09T08:59:16.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d0ff9640 1 -- 172.21.15.98:0/4163586488 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2c8009c30 con 0x7fb2d41021c0 2024-05-09T08:59:16.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d0ff9640 1 -- 172.21.15.98:0/4163586488 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c800f6b0 con 0x7fb2d41021c0 2024-05-09T08:59:16.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.789+0000 7fb2d9f77640 1 -- 172.21.15.98:0/4163586488 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d41a4d00 con 0x7fb2d41021c0 2024-05-09T08:59:16.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.791+0000 7fb2d0ff9640 1 -- 172.21.15.98:0/4163586488 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb2c80109d0 con 0x7fb2d41021c0 2024-05-09T08:59:16.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2dcd65740 1 Processor -- start 2024-05-09T08:59:16.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2dcd65740 1 -- start start 2024-05-09T08:59:16.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2dcd65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30180 0x55e350c30560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2dcd65740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c30aa0 0x55e350c3f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2dcd65740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c3f680 0x55e350c41a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2dcd65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e350b95d00 con 0x55e350c30180 2024-05-09T08:59:16.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2dcd65740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e350b3d4c0 con 0x55e350c30aa0 2024-05-09T08:59:16.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2dcd65740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e350b91cc0 con 0x55e350c3f680 2024-05-09T08:59:16.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2d0ff9640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb2bc0a2d00 0x7fb2bc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2d3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30180 0x55e350c30560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2d2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c3f680 0x55e350c41a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2d3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30180 0x55e350c30560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57186/0 (socket says 172.21.15.98:57186) 2024-05-09T08:59:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2d3fff640 1 -- 172.21.15.98:0/1087507561 learned_addr learned my addr 172.21.15.98:0/1087507561 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2d3fff640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb2bc0a2d00 0x7fb2bc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2d37fe640 1 --2- 172.21.15.98:0/1087507561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c30aa0 0x55e350c3f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.792+0000 7fb2d0ff9640 1 -- 172.21.15.98:0/4163586488 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7fb2c80cf580 con 0x7fb2d41021c0 2024-05-09T08:59:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.793+0000 7fb2d3fff640 1 -- 172.21.15.98:0/1087507561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c3f680 msgr2=0x55e350c41a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.793+0000 7fb2d3fff640 1 --2- 172.21.15.98:0/1087507561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c3f680 0x55e350c41a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.793+0000 7fb2d3fff640 1 -- 172.21.15.98:0/1087507561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c30aa0 msgr2=0x55e350c3f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.793+0000 7fb2d3fff640 1 --2- 172.21.15.98:0/1087507561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c30aa0 0x55e350c3f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.793+0000 7fb2d3fff640 1 -- 172.21.15.98:0/1087507561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3509737d0 con 0x55e350c30180 2024-05-09T08:59:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.793+0000 7fb2d3fff640 1 --2- 172.21.15.98:0/1087507561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30180 0x55e350c30560 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00afa0 tx=0x7fb2cc016700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a918e7c69fd04d85 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.793+0000 7fb2d3fff640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb2bc0a2d00 0x7fb2bc0a51c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00afd0 tx=0x7fb2cc01a000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.793+0000 7fb2b9ffb640 1 -- 172.21.15.98:0/1087507561 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc026070 con 0x55e350c30180 2024-05-09T08:59:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.793+0000 7fb2b9ffb640 1 -- 172.21.15.98:0/1087507561 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2cc009d00 con 0x55e350c30180 2024-05-09T08:59:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.793+0000 7fb2b9ffb640 1 -- 172.21.15.98:0/1087507561 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc013040 con 0x55e350c30180 2024-05-09T08:59:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.794+0000 7fb2dcd65740 1 -- 172.21.15.98:0/1087507561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30180 msgr2=0x55e350c30560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.794+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/1087507561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30180 0x55e350c30560 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00afa0 tx=0x7fb2cc016700 comp rx=0 tx=0).stop 2024-05-09T08:59:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.794+0000 7fb2dcd65740 1 -- 172.21.15.98:0/1087507561 shutdown_connections 2024-05-09T08:59:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.794+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/1087507561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c3f680 0x55e350c41a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.794+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/1087507561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c30aa0 0x55e350c3f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.794+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/1087507561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30180 0x55e350c30560 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.794+0000 7fb2dcd65740 1 -- 172.21.15.98:0/1087507561 >> 172.21.15.98:0/1087507561 conn(0x55e350c2c5a0 msgr2=0x55e350c2c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.794+0000 7fb2dcd65740 1 -- 172.21.15.98:0/1087507561 shutdown_connections 2024-05-09T08:59:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.794+0000 7fb2dcd65740 1 -- 172.21.15.98:0/1087507561 wait complete. 2024-05-09T08:59:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.794+0000 7fb2dcd65740 1 -- start start 2024-05-09T08:59:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.794+0000 7fb2dcd65740 1 Processor -- start 2024-05-09T08:59:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2dcd65740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c30180 0x55e350c3c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2dcd65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c3cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2dcd65740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c3f680 0x55e350c3d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2dcd65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e350c423e0 con 0x55e350c30aa0 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2dcd65740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e350b50b90 con 0x55e350c3f680 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2dcd65740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e350c42560 con 0x55e350c30180 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2dcd65740 10 client.0 _ll_register_callbacks cb 0x55e350b8cda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c3cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c3cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57200/0 (socket says 172.21.15.98:57200) 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/3666358475 learned_addr learned my addr 172.21.15.98:0/3666358475 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2d3fff640 1 --2- 172.21.15.98:0/3666358475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c3f680 0x55e350c3d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2d37fe640 1 --2- 172.21.15.98:0/3666358475 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c30180 0x55e350c3c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/3666358475 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c30180 msgr2=0x55e350c3c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2d2ffd640 1 --2- 172.21.15.98:0/3666358475 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c30180 0x55e350c3c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/3666358475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c3f680 msgr2=0x55e350c3d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2d2ffd640 1 --2- 172.21.15.98:0/3666358475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c3f680 0x55e350c3d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/3666358475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3508a93f0 con 0x55e350c30aa0 2024-05-09T08:59:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2d3fff640 1 --2- 172.21.15.98:0/3666358475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c3f680 0x55e350c3d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2d2ffd640 1 --2- 172.21.15.98:0/3666358475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c3cb50 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8000c00 tx=0x7fb2c806b490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9feed7c3f3761a40 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2b2ffd640 1 -- 172.21.15.98:0/3666358475 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8020040 con 0x55e350c30aa0 2024-05-09T08:59:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2b2ffd640 1 -- 172.21.15.98:0/3666358475 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7fb2c80a3e10 con 0x55e350c30aa0 2024-05-09T08:59:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2dcd65740 1 -- 172.21.15.98:0/3666358475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2cc009d00 con 0x55e350c30aa0 2024-05-09T08:59:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2b2ffd640 1 -- 172.21.15.98:0/3666358475 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb2c80666c0 con 0x55e350c30aa0 2024-05-09T08:59:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2dcd65740 10 client.17964 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.795+0000 7fb2dcd65740 1 -- 172.21.15.98:0/3666358475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e3508690f0 con 0x55e350c30aa0 2024-05-09T08:59:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.796+0000 7fb2b2ffd640 1 -- 172.21.15.98:0/3666358475 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2cc009d00 con 0x55e350c30aa0 2024-05-09T08:59:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.796+0000 7fb2b2ffd640 1 -- 172.21.15.98:0/3666358475 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb2c806a1f0 con 0x55e350c30aa0 2024-05-09T08:59:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.796+0000 7fb2dcd65740 10 client.17964 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.796+0000 7fb2dcd65740 10 client.17964 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.796+0000 7fb2dcd65740 20 client.17964 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.796+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/3666358475 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e350c4f8c0 0x55e350c51cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.796+0000 7fb2dcd65740 4 client.17964 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.796+0000 7fb2dcd65740 1 -- 172.21.15.98:0/3666358475 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e350ba3430 con 0x55e350c4f8c0 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.796+0000 7fb2d37fe640 1 --2- 172.21.15.98:0/3666358475 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e350c4f8c0 0x55e350c51cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.796+0000 7fb2d37fe640 1 --2- 172.21.15.98:0/3666358475 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e350c4f8c0 0x55e350c51cd0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbef90a6ea4919d5 server_cookie=619a7a91c3618eeb in_seq=0 out_seq=0 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.796+0000 7fb2b2ffd640 10 client.17964 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.797+0000 7fb2b2ffd640 1 -- 172.21.15.98:0/3666358475 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e350ba3430 con 0x55e350c4f8c0 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.797+0000 7fb2b2ffd640 10 client.17964 handle_command_reply: tid=0 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.797+0000 7fb2dcd65740 1 client.17964 shutdown 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.797+0000 7fb2dcd65740 1 -- 172.21.15.98:0/3666358475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 msgr2=0x55e350c3cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.797+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/3666358475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c3cb50 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8000c00 tx=0x7fb2c806b490 comp rx=0 tx=0).stop 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.797+0000 7fb2dcd65740 1 -- 172.21.15.98:0/3666358475 shutdown_connections 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.797+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/3666358475 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e350c4f8c0 0x55e350c51cd0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.797+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/3666358475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c3f680 0x55e350c3d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.797+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/3666358475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c3cb50 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.797+0000 7fb2b2ffd640 0 client.17964 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:16.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.797+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/3666358475 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c30180 0x55e350c3c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.798+0000 7fb2dcd65740 1 -- 172.21.15.98:0/3666358475 >> 172.21.15.98:0/3666358475 conn(0x55e350c2c5a0 msgr2=0x55e350c32410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:16.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.798+0000 7fb2dcd65740 1 -- 172.21.15.98:0/3666358475 shutdown_connections 2024-05-09T08:59:16.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.798+0000 7fb2dcd65740 1 -- 172.21.15.98:0/3666358475 wait complete. 2024-05-09T08:59:16.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.798+0000 7fb2dcd65740 20 client.17964 trim_cache size 0 max 16384 2024-05-09T08:59:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.815+0000 7fb2dcd65740 1 Processor -- start 2024-05-09T08:59:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.815+0000 7fb2dcd65740 1 -- start start 2024-05-09T08:59:16.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.815+0000 7fb2dcd65740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c30180 0x55e350c4b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.815+0000 7fb2dcd65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c2b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.815+0000 7fb2dcd65740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c3f680 0x55e350c575a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.815+0000 7fb2dcd65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d41ace70 con 0x55e350c30aa0 2024-05-09T08:59:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.815+0000 7fb2dcd65740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d41ae9b0 con 0x55e350c30180 2024-05-09T08:59:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.815+0000 7fb2dcd65740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d41ad540 con 0x55e350c3f680 2024-05-09T08:59:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.815+0000 7fb2d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c2b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.815+0000 7fb2d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c2b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57202/0 (socket says 172.21.15.98:57202) 2024-05-09T08:59:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.815+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/587669705 learned_addr learned my addr 172.21.15.98:0/587669705 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.816+0000 7fb2d37fe640 1 --2- 172.21.15.98:0/587669705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c30180 0x55e350c4b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.816+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/587669705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c3f680 msgr2=0x55e350c575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.816+0000 7fb2d2ffd640 1 --2- 172.21.15.98:0/587669705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c3f680 0x55e350c575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.816+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/587669705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c30180 msgr2=0x55e350c4b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.816+0000 7fb2d2ffd640 1 --2- 172.21.15.98:0/587669705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c30180 0x55e350c4b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.816+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/587669705 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e350c41a70 con 0x55e350c30aa0 2024-05-09T08:59:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.816+0000 7fb2d2ffd640 1 --2- 172.21.15.98:0/587669705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c2b660 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80058c0 tx=0x7fb2c800bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0a469bc91ba00d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.816+0000 7fb2b97fa640 1 -- 172.21.15.98:0/587669705 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8020070 con 0x55e350c30aa0 2024-05-09T08:59:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.816+0000 7fb2b97fa640 1 -- 172.21.15.98:0/587669705 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2c809b630 con 0x55e350c30aa0 2024-05-09T08:59:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.816+0000 7fb2b97fa640 1 -- 172.21.15.98:0/587669705 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8097ca0 con 0x55e350c30aa0 2024-05-09T08:59:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 -- 172.21.15.98:0/587669705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 msgr2=0x55e350c2b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/587669705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c2b660 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80058c0 tx=0x7fb2c800bf60 comp rx=0 tx=0).stop 2024-05-09T08:59:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 -- 172.21.15.98:0/587669705 shutdown_connections 2024-05-09T08:59:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/587669705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c3f680 0x55e350c575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/587669705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c2b660 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/587669705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c30180 0x55e350c4b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 -- 172.21.15.98:0/587669705 >> 172.21.15.98:0/587669705 conn(0x55e350c2c5a0 msgr2=0x55e350c2af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 -- 172.21.15.98:0/587669705 shutdown_connections 2024-05-09T08:59:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 -- 172.21.15.98:0/587669705 wait complete. 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 -- start start 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 Processor -- start 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c30180 0x55e350c431e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c43720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c3f680 0x55e350c43c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d41ae2e0 con 0x55e350c30aa0 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d41adc10 con 0x55e350c3f680 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d41aae70 con 0x55e350c30180 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2dcd65740 10 client.0 _ll_register_callbacks cb 0x7fb2c80860b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.817+0000 7fb2d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c43720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c43720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57218/0 (socket says 172.21.15.98:57218) 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/2223031718 learned_addr learned my addr 172.21.15.98:0/2223031718 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/2223031718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c30180 msgr2=0x55e350c431e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2d2ffd640 1 --2- 172.21.15.98:0/2223031718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c30180 0x55e350c431e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/2223031718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c3f680 msgr2=0x55e350c43c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2d3fff640 1 --2- 172.21.15.98:0/2223031718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c3f680 0x55e350c43c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2d2ffd640 1 --2- 172.21.15.98:0/2223031718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c3f680 0x55e350c43c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2d2ffd640 1 -- 172.21.15.98:0/2223031718 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e350cb4e40 con 0x55e350c30aa0 2024-05-09T08:59:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2d2ffd640 1 --2- 172.21.15.98:0/2223031718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c43720 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fb2c809b7f0 tx=0x7fb2c8005af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f23a39edd085c9d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2b2ffd640 1 -- 172.21.15.98:0/2223031718 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8020070 con 0x55e350c30aa0 2024-05-09T08:59:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2b2ffd640 1 -- 172.21.15.98:0/2223031718 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7fb2c80cf320 con 0x55e350c30aa0 2024-05-09T08:59:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2dcd65740 1 -- 172.21.15.98:0/2223031718 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e350b9d780 con 0x55e350c30aa0 2024-05-09T08:59:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2b2ffd640 1 -- 172.21.15.98:0/2223031718 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb2c80b4050 con 0x55e350c30aa0 2024-05-09T08:59:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2dcd65740 10 client.17970 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2dcd65740 1 -- 172.21.15.98:0/2223031718 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e350cb4e40 con 0x55e350c30aa0 2024-05-09T08:59:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2b2ffd640 1 -- 172.21.15.98:0/2223031718 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e350b9d780 con 0x55e350c30aa0 2024-05-09T08:59:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.818+0000 7fb2b2ffd640 1 -- 172.21.15.98:0/2223031718 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb2c80afb50 con 0x55e350c30aa0 2024-05-09T08:59:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.819+0000 7fb2dcd65740 10 client.17970 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.819+0000 7fb2dcd65740 10 client.17970 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.819+0000 7fb2dcd65740 20 client.17970 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.819+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/2223031718 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e350cb63c0 0x55e350cb87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.819+0000 7fb2dcd65740 4 client.17970 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:59:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.819+0000 7fb2dcd65740 1 -- 172.21.15.98:0/2223031718 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e350b1a6f0 con 0x55e350cb63c0 2024-05-09T08:59:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.819+0000 7fb2d37fe640 1 --2- 172.21.15.98:0/2223031718 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e350cb63c0 0x55e350cb87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.819+0000 7fb2b2ffd640 10 client.17970 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.819+0000 7fb2d37fe640 1 --2- 172.21.15.98:0/2223031718 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e350cb63c0 0x55e350cb87b0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cf2397051698b5d server_cookie=efd9420481992af6 in_seq=0 out_seq=0 2024-05-09T08:59:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.819+0000 7fb2b2ffd640 1 -- 172.21.15.98:0/2223031718 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e350b1a6f0 con 0x55e350cb63c0 2024-05-09T08:59:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.819+0000 7fb2b2ffd640 10 client.17970 handle_command_reply: tid=0 2024-05-09T08:59:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.819+0000 7fb2dcd65740 1 client.17970 shutdown 2024-05-09T08:59:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2dcd65740 1 -- 172.21.15.98:0/2223031718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 msgr2=0x55e350c43720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/2223031718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c43720 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fb2c809b7f0 tx=0x7fb2c8005af0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2dcd65740 1 -- 172.21.15.98:0/2223031718 shutdown_connections 2024-05-09T08:59:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/2223031718 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55e350cb63c0 0x55e350cb87b0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/2223031718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e350c3f680 0x55e350c43c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/2223031718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e350c30aa0 0x55e350c43720 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2dcd65740 1 --2- 172.21.15.98:0/2223031718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e350c30180 0x55e350c431e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2b2ffd640 0 client.17970 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2dcd65740 1 -- 172.21.15.98:0/2223031718 >> 172.21.15.98:0/2223031718 conn(0x55e350c2c5a0 msgr2=0x55e350c2b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2dcd65740 1 -- 172.21.15.98:0/2223031718 shutdown_connections 2024-05-09T08:59:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2dcd65740 1 -- 172.21.15.98:0/2223031718 wait complete. 2024-05-09T08:59:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2dcd65740 20 client.17970 trim_cache size 0 max 16384 2024-05-09T08:59:16.827 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:59:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:59:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:59:16.827 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:59:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2d9f77640 1 -- 172.21.15.98:0/4163586488 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb2bc0a2d00 msgr2=0x7fb2bc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2d9f77640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb2bc0a2d00 0x7fb2bc0a51c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00afd0 tx=0x7fb2cc01a000 comp rx=0 tx=0).stop 2024-05-09T08:59:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2d9f77640 1 -- 172.21.15.98:0/4163586488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2d41021c0 msgr2=0x7fb2d4198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2d9f77640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2d41021c0 0x7fb2d4198720 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800d950 tx=0x7fb2c800de10 comp rx=0 tx=0).stop 2024-05-09T08:59:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2d9f77640 1 -- 172.21.15.98:0/4163586488 shutdown_connections 2024-05-09T08:59:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2d9f77640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb2bc0a2d00 0x7fb2bc0a51c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2d9f77640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2d4107810 0x7fb2d419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2d9f77640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2d41021c0 0x7fb2d4198720 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2d9f77640 1 --2- 172.21.15.98:0/4163586488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2d4101800 0x7fb2d41981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2d9f77640 1 -- 172.21.15.98:0/4163586488 >> 172.21.15.98:0/4163586488 conn(0x7fb2d40fd2d0 msgr2=0x7fb2d40fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2d9f77640 1 -- 172.21.15.98:0/4163586488 shutdown_connections 2024-05-09T08:59:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.820+0000 7fb2d9f77640 1 -- 172.21.15.98:0/4163586488 wait complete. 2024-05-09T08:59:16.831 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:59:16.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:59:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.932+0000 7f38fc59d640 1 -- 172.21.15.98:0/1588734970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38f4100da0 msgr2=0x7f38f41064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.932+0000 7f38fc59d640 1 --2- 172.21.15.98:0/1588734970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38f4100da0 0x7f38f41064d0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f38e4009900 tx=0x7f38e402a410 comp rx=0 tx=0).stop 2024-05-09T08:59:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.932+0000 7f38fc59d640 1 -- 172.21.15.98:0/1588734970 shutdown_connections 2024-05-09T08:59:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.932+0000 7f38fc59d640 1 --2- 172.21.15.98:0/1588734970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f38f4106bb0 0x7f38f410e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.932+0000 7f38fc59d640 1 --2- 172.21.15.98:0/1588734970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38f4100da0 0x7f38f41064d0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.932+0000 7f38fc59d640 1 --2- 172.21.15.98:0/1588734970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f41003f0 0x7f38f41007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.932+0000 7f38fc59d640 1 -- 172.21.15.98:0/1588734970 >> 172.21.15.98:0/1588734970 conn(0x7f38f4075a10 msgr2=0x7f38f4075e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.932+0000 7f38fc59d640 1 -- 172.21.15.98:0/1588734970 shutdown_connections 2024-05-09T08:59:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.932+0000 7f38fc59d640 1 -- 172.21.15.98:0/1588734970 wait complete. 2024-05-09T08:59:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fc59d640 1 Processor -- start 2024-05-09T08:59:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fc59d640 1 -- start start 2024-05-09T08:59:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fc59d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38f41003f0 0x7f38f41980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fc59d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f38f4106bb0 0x7f38f4198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fc59d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f419c9a0 0x7f38f419ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fc59d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38f4111c80 con 0x7f38f41003f0 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fc59d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f38f4111b00 con 0x7f38f4106bb0 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fc59d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f38f4111e00 con 0x7f38f419c9a0 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fa339640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38f41003f0 0x7f38f41980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fa339640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38f41003f0 0x7f38f41980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57230/0 (socket says 172.21.15.98:57230) 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fa339640 1 -- 172.21.15.98:0/3388101927 learned_addr learned my addr 172.21.15.98:0/3388101927 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fab3a640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f419c9a0 0x7f38f419ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38f9b38640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f38f4106bb0 0x7f38f4198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fa339640 1 -- 172.21.15.98:0/3388101927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f419c9a0 msgr2=0x7f38f419ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fa339640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f419c9a0 0x7f38f419ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fa339640 1 -- 172.21.15.98:0/3388101927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f38f4106bb0 msgr2=0x7f38f4198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fa339640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f38f4106bb0 0x7f38f4198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fa339640 1 -- 172.21.15.98:0/3388101927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f419d490 con 0x7f38f41003f0 2024-05-09T08:59:16.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38f9b38640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f38f4106bb0 0x7f38f4198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:16.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.933+0000 7f38fa339640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38f41003f0 0x7f38f41980e0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f38ec00c970 tx=0x7f38ec00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:16.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.934+0000 7f38eb7fe640 1 -- 172.21.15.98:0/3388101927 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ec007bb0 con 0x7f38f41003f0 2024-05-09T08:59:16.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.934+0000 7f38fc59d640 1 -- 172.21.15.98:0/3388101927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38f419d780 con 0x7f38f41003f0 2024-05-09T08:59:16.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.934+0000 7f38eb7fe640 1 -- 172.21.15.98:0/3388101927 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38ec007d50 con 0x7f38f41003f0 2024-05-09T08:59:16.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.934+0000 7f38eb7fe640 1 -- 172.21.15.98:0/3388101927 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ec00f450 con 0x7f38f41003f0 2024-05-09T08:59:16.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.934+0000 7f38fc59d640 1 -- 172.21.15.98:0/3388101927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38f41a4ef0 con 0x7f38f41003f0 2024-05-09T08:59:16.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.935+0000 7f38fc59d640 1 -- 172.21.15.98:0/3388101927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38f4068aa0 con 0x7f38f41003f0 2024-05-09T08:59:16.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.936+0000 7f38eb7fe640 1 -- 172.21.15.98:0/3388101927 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f38ec004fe0 con 0x7f38f41003f0 2024-05-09T08:59:16.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.937+0000 7f38eb7fe640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f38e00a2d00 0x7f38e00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:16.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.937+0000 7f38f9b38640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f38e00a2d00 0x7f38e00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:16.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.937+0000 7f38eb7fe640 1 -- 172.21.15.98:0/3388101927 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7f38ec0cfb40 con 0x7f38f41003f0 2024-05-09T08:59:16.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.938+0000 7f38f9b38640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f38e00a2d00 0x7f38e00a51c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f38e4000c00 tx=0x7f38e4005a20 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:16.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:16.941+0000 7f38eb7fe640 1 -- 172.21.15.98:0/3388101927 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f38ec014030 con 0x7f38f41003f0 2024-05-09T08:59:17.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.156+0000 7f38fc59d640 1 -- 172.21.15.98:0/3388101927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38f4109d90 con 0x7f38f41003f0 2024-05-09T08:59:17.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.157+0000 7f38eb7fe640 1 -- 172.21.15.98:0/3388101927 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11941 (secure 0 0 0) 0x7f38ec09c2f0 con 0x7f38f41003f0 2024-05-09T08:59:17.156 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:59:17.156 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":26581,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/561027115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":561027115},{"type":"v1","addr":"172.21.15.105:6835","nonce":561027115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:56:52.891273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16641":{"gid":16641,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T08:59:17.156 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 37 2024-05-09T08:59:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.159+0000 7f38fc59d640 1 -- 172.21.15.98:0/3388101927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f38e00a2d00 msgr2=0x7f38e00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.159+0000 7f38fc59d640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f38e00a2d00 0x7f38e00a51c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f38e4000c00 tx=0x7f38e4005a20 comp rx=0 tx=0).stop 2024-05-09T08:59:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.159+0000 7f38fc59d640 1 -- 172.21.15.98:0/3388101927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38f41003f0 msgr2=0x7f38f41980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.159+0000 7f38fc59d640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38f41003f0 0x7f38f41980e0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f38ec00c970 tx=0x7f38ec00ce30 comp rx=0 tx=0).stop 2024-05-09T08:59:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.159+0000 7f38fc59d640 1 -- 172.21.15.98:0/3388101927 shutdown_connections 2024-05-09T08:59:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.159+0000 7f38fc59d640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f38e00a2d00 0x7f38e00a51c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.159+0000 7f38fc59d640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f419c9a0 0x7f38f419ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.159+0000 7f38fc59d640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f38f4106bb0 0x7f38f4198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.159+0000 7f38fc59d640 1 --2- 172.21.15.98:0/3388101927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38f41003f0 0x7f38f41980e0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.159+0000 7f38fc59d640 1 -- 172.21.15.98:0/3388101927 >> 172.21.15.98:0/3388101927 conn(0x7f38f4075a10 msgr2=0x7f38f40fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.159+0000 7f38fc59d640 1 -- 172.21.15.98:0/3388101927 shutdown_connections 2024-05-09T08:59:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.159+0000 7f38fc59d640 1 -- 172.21.15.98:0/3388101927 wait complete. 2024-05-09T08:59:17.169 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:59:17.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.274+0000 7f6ffef3b640 1 -- 172.21.15.98:0/2541707836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff8102ab0 msgr2=0x7f6ff8102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.274+0000 7f6ffef3b640 1 --2- 172.21.15.98:0/2541707836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff8102ab0 0x7f6ff8102e90 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4009900 tx=0x7f6ff402a510 comp rx=0 tx=0).stop 2024-05-09T08:59:17.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.274+0000 7f6ffef3b640 1 -- 172.21.15.98:0/2541707836 shutdown_connections 2024-05-09T08:59:17.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.274+0000 7f6ffef3b640 1 --2- 172.21.15.98:0/2541707836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ff810bdf0 0x7f6ff810e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.274+0000 7f6ffef3b640 1 --2- 172.21.15.98:0/2541707836 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ff81033d0 0x7f6ff810b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.274+0000 7f6ffef3b640 1 --2- 172.21.15.98:0/2541707836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff8102ab0 0x7f6ff8102e90 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.274+0000 7f6ffef3b640 1 -- 172.21.15.98:0/2541707836 >> 172.21.15.98:0/2541707836 conn(0x7f6ff8077830 msgr2=0x7f6ff8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.274+0000 7f6ffef3b640 1 -- 172.21.15.98:0/2541707836 shutdown_connections 2024-05-09T08:59:17.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.274+0000 7f6ffef3b640 1 -- 172.21.15.98:0/2541707836 wait complete. 2024-05-09T08:59:17.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffef3b640 1 Processor -- start 2024-05-09T08:59:17.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffef3b640 1 -- start start 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffef3b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ff8102ab0 0x7f6ff8100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffef3b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ff81033d0 0x7f6ff8100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffef3b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff810bdf0 0x7f6ff8101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffef3b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff8111bf0 con 0x7f6ff810bdf0 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffef3b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff8111a70 con 0x7f6ff81033d0 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffef3b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff8111d70 con 0x7f6ff8102ab0 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffd4d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff810bdf0 0x7f6ff8101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffd4d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff810bdf0 0x7f6ff8101180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57244/0 (socket says 172.21.15.98:57244) 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffd4d8640 1 -- 172.21.15.98:0/3941083531 learned_addr learned my addr 172.21.15.98:0/3941083531 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffccd7640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ff8102ab0 0x7f6ff8100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6feffff640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ff81033d0 0x7f6ff8100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffd4d8640 1 -- 172.21.15.98:0/3941083531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ff8102ab0 msgr2=0x7f6ff8100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffd4d8640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ff8102ab0 0x7f6ff8100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffd4d8640 1 -- 172.21.15.98:0/3941083531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ff81033d0 msgr2=0x7f6ff8100c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffd4d8640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ff81033d0 0x7f6ff8100c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffd4d8640 1 -- 172.21.15.98:0/3941083531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff81a65f0 con 0x7f6ff810bdf0 2024-05-09T08:59:17.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6feffff640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ff81033d0 0x7f6ff8100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:17.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.275+0000 7f6ffd4d8640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff810bdf0 0x7f6ff8101180 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000eb10 tx=0x7f6ff000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.276+0000 7f6fedffb640 1 -- 172.21.15.98:0/3941083531 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0004b20 con 0x7f6ff810bdf0 2024-05-09T08:59:17.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.276+0000 7f6fedffb640 1 -- 172.21.15.98:0/3941083531 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ff0004cc0 con 0x7f6ff810bdf0 2024-05-09T08:59:17.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.276+0000 7f6ffef3b640 1 -- 172.21.15.98:0/3941083531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff81a68e0 con 0x7f6ff810bdf0 2024-05-09T08:59:17.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.276+0000 7f6fedffb640 1 -- 172.21.15.98:0/3941083531 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0010760 con 0x7f6ff810bdf0 2024-05-09T08:59:17.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.276+0000 7f6ffef3b640 1 -- 172.21.15.98:0/3941083531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ff81a6de0 con 0x7f6ff810bdf0 2024-05-09T08:59:17.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.278+0000 7f6fedffb640 1 -- 172.21.15.98:0/3941083531 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6ff0005020 con 0x7f6ff810bdf0 2024-05-09T08:59:17.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.278+0000 7f7001c47740 1 Processor -- start 2024-05-09T08:59:17.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.278+0000 7f7001c47740 1 -- start start 2024-05-09T08:59:17.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.278+0000 7f6fedffb640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6fe00a2d00 0x7f6fe00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f7001c47740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8330 0x55b8df2e8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f7001c47740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8c50 0x55b8df2f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f7001c47740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2f3320 0x55b8df2f5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f7001c47740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b8df24dc40 con 0x55b8df2f3320 2024-05-09T08:59:17.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f7001c47740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b8df1f5580 con 0x55b8df2e8330 2024-05-09T08:59:17.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f7001c47740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b8df249c00 con 0x55b8df2e8c50 2024-05-09T08:59:17.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6fedffb640 1 -- 172.21.15.98:0/3941083531 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7f6ff00cfe60 con 0x7f6ff810bdf0 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6feffff640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6fe00a2d00 0x7f6fe00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6ffd4d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2f3320 0x55b8df2f5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6ffd4d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2f3320 0x55b8df2f5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57254/0 (socket says 172.21.15.98:57254) 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6ffd4d8640 1 -- 172.21.15.98:0/578057203 learned_addr learned my addr 172.21.15.98:0/578057203 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6ffccd7640 1 --2- 172.21.15.98:0/578057203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8330 0x55b8df2e8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6ffd4d8640 1 -- 172.21.15.98:0/578057203 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8c50 msgr2=0x55b8df2f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6ffd4d8640 1 --2- 172.21.15.98:0/578057203 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8c50 0x55b8df2f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6ffd4d8640 1 -- 172.21.15.98:0/578057203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8330 msgr2=0x55b8df2e8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6ffd4d8640 1 --2- 172.21.15.98:0/578057203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8330 0x55b8df2e8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6ffd4d8640 1 -- 172.21.15.98:0/578057203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8def613f0 con 0x55b8df2f3320 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6ffccd7640 1 --2- 172.21.15.98:0/578057203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8330 0x55b8df2e8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6ffd4d8640 1 --2- 172.21.15.98:0/578057203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2f3320 0x55b8df2f5710 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0098370 tx=0x7f6ff009bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92ef918910f5c51 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6feffff640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6fe00a2d00 0x7f6fe00a51c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8005fd0 tx=0x7f6fe800a660 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6fdeffd640 1 -- 172.21.15.98:0/578057203 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff009c610 con 0x55b8df2f3320 2024-05-09T08:59:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6fdeffd640 1 -- 172.21.15.98:0/578057203 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ff009bf10 con 0x55b8df2f3320 2024-05-09T08:59:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.279+0000 7f6fdeffd640 1 -- 172.21.15.98:0/578057203 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff00cd3c0 con 0x55b8df2f3320 2024-05-09T08:59:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.280+0000 7f7001c47740 1 -- 172.21.15.98:0/578057203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2f3320 msgr2=0x55b8df2f5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.280+0000 7f7001c47740 1 --2- 172.21.15.98:0/578057203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2f3320 0x55b8df2f5710 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0098370 tx=0x7f6ff009bda0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.280+0000 7f7001c47740 1 -- 172.21.15.98:0/578057203 shutdown_connections 2024-05-09T08:59:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.280+0000 7f7001c47740 1 --2- 172.21.15.98:0/578057203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2f3320 0x55b8df2f5710 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.280+0000 7f7001c47740 1 --2- 172.21.15.98:0/578057203 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8c50 0x55b8df2f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.280+0000 7f7001c47740 1 --2- 172.21.15.98:0/578057203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8330 0x55b8df2e8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.280+0000 7f7001c47740 1 -- 172.21.15.98:0/578057203 >> 172.21.15.98:0/578057203 conn(0x55b8df2e4650 msgr2=0x55b8df2e4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.280+0000 7f7001c47740 1 -- 172.21.15.98:0/578057203 shutdown_connections 2024-05-09T08:59:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.280+0000 7f7001c47740 1 -- 172.21.15.98:0/578057203 wait complete. 2024-05-09T08:59:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.280+0000 7f7001c47740 1 -- start start 2024-05-09T08:59:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.280+0000 7f7001c47740 1 Processor -- start 2024-05-09T08:59:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f7001c47740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8330 0x55b8df2f0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f7001c47740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2e8c50 0x55b8df2f0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f7001c47740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2f3320 0x55b8df2f0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f7001c47740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b8df2f6110 con 0x55b8df2e8c50 2024-05-09T08:59:17.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f7001c47740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b8df208c40 con 0x55b8df2f3320 2024-05-09T08:59:17.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f7001c47740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b8df2f6290 con 0x55b8df2e8330 2024-05-09T08:59:17.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f7001c47740 10 client.0 _ll_register_callbacks cb 0x55b8def60be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:17.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f6feffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2e8c50 0x55b8df2f0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f6feffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2e8c50 0x55b8df2f0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57266/0 (socket says 172.21.15.98:57266) 2024-05-09T08:59:17.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f6feffff640 1 -- 172.21.15.98:0/3102425210 learned_addr learned my addr 172.21.15.98:0/3102425210 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f6ffd4d8640 1 --2- 172.21.15.98:0/3102425210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2f3320 0x55b8df2f0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f6ffccd7640 1 --2- 172.21.15.98:0/3102425210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8330 0x55b8df2f0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f6feffff640 1 -- 172.21.15.98:0/3102425210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8330 msgr2=0x55b8df2f0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f6feffff640 1 --2- 172.21.15.98:0/3102425210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8330 0x55b8df2f0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f6feffff640 1 -- 172.21.15.98:0/3102425210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2f3320 msgr2=0x55b8df2f0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f6feffff640 1 --2- 172.21.15.98:0/3102425210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2f3320 0x55b8df2f0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f6feffff640 1 -- 172.21.15.98:0/3102425210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8df02b7d0 con 0x55b8df2e8c50 2024-05-09T08:59:17.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f6ffd4d8640 1 --2- 172.21.15.98:0/3102425210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2f3320 0x55b8df2f0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:17.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f6feffff640 1 --2- 172.21.15.98:0/3102425210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2e8c50 0x55b8df2f0780 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8012960 tx=0x7f6fe80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f4daff82db7b63 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f6fd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:17.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.281+0000 7f6fd7fff640 1 -- 172.21.15.98:0/3102425210 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8023030 con 0x55b8df2e8c50 2024-05-09T08:59:17.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.282+0000 7f6fd7fff640 1 -- 172.21.15.98:0/3102425210 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7f6fe8011b70 con 0x55b8df2e8c50 2024-05-09T08:59:17.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.282+0000 7f7001c47740 1 -- 172.21.15.98:0/3102425210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ff009bf10 con 0x55b8df2e8c50 2024-05-09T08:59:17.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.282+0000 7f6fd7fff640 1 -- 172.21.15.98:0/3102425210 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6fe802d050 con 0x55b8df2e8c50 2024-05-09T08:59:17.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.282+0000 7f7001c47740 10 client.17988 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:17.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.282+0000 7f7001c47740 1 -- 172.21.15.98:0/3102425210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b8df302300 con 0x55b8df2e8c50 2024-05-09T08:59:17.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.282+0000 7f6fd7fff640 1 -- 172.21.15.98:0/3102425210 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ff009bf10 con 0x55b8df2e8c50 2024-05-09T08:59:17.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.282+0000 7f6fd7fff640 1 -- 172.21.15.98:0/3102425210 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f6fe8004b50 con 0x55b8df2e8c50 2024-05-09T08:59:17.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.282+0000 7f7001c47740 10 client.17988 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:17.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.282+0000 7f7001c47740 10 client.17988 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:59:17.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.282+0000 7f7001c47740 20 client.17988 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:17.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.282+0000 7f7001c47740 1 --2- 172.21.15.98:0/3102425210 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b8df303810 0x55b8df305c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.282+0000 7f7001c47740 4 client.17988 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:17.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.282+0000 7f7001c47740 1 -- 172.21.15.98:0/3102425210 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8df25b370 con 0x55b8df303810 2024-05-09T08:59:17.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.282+0000 7f6ffccd7640 1 --2- 172.21.15.98:0/3102425210 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b8df303810 0x55b8df305c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.283+0000 7f6ffccd7640 1 --2- 172.21.15.98:0/3102425210 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b8df303810 0x55b8df305c70 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7a4665b65e678fec server_cookie=fb855a5bf1cd06ba in_seq=0 out_seq=0 2024-05-09T08:59:17.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.283+0000 7f6fd7fff640 10 client.17988 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:59:17.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.283+0000 7f6fd7fff640 1 -- 172.21.15.98:0/3102425210 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b8df25b370 con 0x55b8df303810 2024-05-09T08:59:17.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.283+0000 7f6fd7fff640 10 client.17988 handle_command_reply: tid=0 2024-05-09T08:59:17.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.283+0000 7f7001c47740 1 client.17988 shutdown 2024-05-09T08:59:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.284+0000 7f7001c47740 1 -- 172.21.15.98:0/3102425210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2e8c50 msgr2=0x55b8df2f0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.284+0000 7f7001c47740 1 --2- 172.21.15.98:0/3102425210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2e8c50 0x55b8df2f0780 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8012960 tx=0x7f6fe80046c0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.284+0000 7f7001c47740 1 -- 172.21.15.98:0/3102425210 shutdown_connections 2024-05-09T08:59:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.284+0000 7f7001c47740 1 --2- 172.21.15.98:0/3102425210 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b8df303810 0x55b8df305c70 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.284+0000 7f7001c47740 1 --2- 172.21.15.98:0/3102425210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2f3320 0x55b8df2f0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.284+0000 7f7001c47740 1 --2- 172.21.15.98:0/3102425210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2e8c50 0x55b8df2f0780 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.284+0000 7f7001c47740 1 --2- 172.21.15.98:0/3102425210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8330 0x55b8df2f0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.284+0000 7f6fd7fff640 0 client.17988 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:59:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.284+0000 7f7001c47740 1 -- 172.21.15.98:0/3102425210 >> 172.21.15.98:0/3102425210 conn(0x55b8df2e4650 msgr2=0x55b8df2ea450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.284+0000 7f7001c47740 1 -- 172.21.15.98:0/3102425210 shutdown_connections 2024-05-09T08:59:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.284+0000 7f7001c47740 1 -- 172.21.15.98:0/3102425210 wait complete. 2024-05-09T08:59:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.284+0000 7f7001c47740 20 client.17988 trim_cache size 0 max 16384 2024-05-09T08:59:17.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f7001c47740 1 Processor -- start 2024-05-09T08:59:17.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f7001c47740 1 -- start start 2024-05-09T08:59:17.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f7001c47740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2e8330 0x55b8df2e2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f7001c47740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8c50 0x55b8df2eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f7001c47740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2f3320 0x55b8df2ef9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f7001c47740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81b1230 con 0x55b8df2e8330 2024-05-09T08:59:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f7001c47740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81b2d70 con 0x55b8df2e8c50 2024-05-09T08:59:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f7001c47740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81b1900 con 0x55b8df2f3320 2024-05-09T08:59:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f6ffccd7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2e8330 0x55b8df2e2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f6ffccd7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2e8330 0x55b8df2e2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57278/0 (socket says 172.21.15.98:57278) 2024-05-09T08:59:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f6ffccd7640 1 -- 172.21.15.98:0/115886353 learned_addr learned my addr 172.21.15.98:0/115886353 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f6ffd4d8640 1 --2- 172.21.15.98:0/115886353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2f3320 0x55b8df2ef9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f6ffccd7640 1 -- 172.21.15.98:0/115886353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2f3320 msgr2=0x55b8df2ef9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f6ffccd7640 1 --2- 172.21.15.98:0/115886353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2f3320 0x55b8df2ef9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f6feffff640 1 --2- 172.21.15.98:0/115886353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8c50 0x55b8df2eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f6ffccd7640 1 -- 172.21.15.98:0/115886353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8c50 msgr2=0x55b8df2eaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f6ffccd7640 1 --2- 172.21.15.98:0/115886353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8c50 0x55b8df2eaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f6ffccd7640 1 -- 172.21.15.98:0/115886353 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8df2e3dd0 con 0x55b8df2e8330 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f6ffd4d8640 1 --2- 172.21.15.98:0/115886353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2f3320 0x55b8df2ef9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.297+0000 7f6feffff640 1 --2- 172.21.15.98:0/115886353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8c50 0x55b8df2eaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.298+0000 7f6ffccd7640 1 --2- 172.21.15.98:0/115886353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2e8330 0x55b8df2e2f20 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4009230 tx=0x7f6ff40058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42eb86625dda4991 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.298+0000 7f6fde7fc640 1 -- 172.21.15.98:0/115886353 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4049030 con 0x55b8df2e8330 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.298+0000 7f6fde7fc640 1 -- 172.21.15.98:0/115886353 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ff4004310 con 0x55b8df2e8330 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.298+0000 7f6fde7fc640 1 -- 172.21.15.98:0/115886353 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4035040 con 0x55b8df2e8330 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.298+0000 7f7001c47740 1 -- 172.21.15.98:0/115886353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2e8330 msgr2=0x55b8df2e2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.298+0000 7f7001c47740 1 --2- 172.21.15.98:0/115886353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2e8330 0x55b8df2e2f20 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4009230 tx=0x7f6ff40058d0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.298+0000 7f7001c47740 1 -- 172.21.15.98:0/115886353 shutdown_connections 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.298+0000 7f7001c47740 1 --2- 172.21.15.98:0/115886353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2f3320 0x55b8df2ef9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.298+0000 7f7001c47740 1 --2- 172.21.15.98:0/115886353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8c50 0x55b8df2eaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.298+0000 7f7001c47740 1 --2- 172.21.15.98:0/115886353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2e8330 0x55b8df2e2f20 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.298+0000 7f7001c47740 1 -- 172.21.15.98:0/115886353 >> 172.21.15.98:0/115886353 conn(0x55b8df2e4650 msgr2=0x55b8df2f21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.298+0000 7f7001c47740 1 -- 172.21.15.98:0/115886353 shutdown_connections 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.298+0000 7f7001c47740 1 -- 172.21.15.98:0/115886353 wait complete. 2024-05-09T08:59:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f7001c47740 1 -- start start 2024-05-09T08:59:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f7001c47740 1 Processor -- start 2024-05-09T08:59:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f7001c47740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8330 0x55b8df2fb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f7001c47740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8c50 0x55b8df2f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f7001c47740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2f3320 0x55b8df2f6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f7001c47740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81b26a0 con 0x55b8df2f3320 2024-05-09T08:59:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f7001c47740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81b1fd0 con 0x55b8df2e8330 2024-05-09T08:59:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f7001c47740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81af230 con 0x55b8df2e8c50 2024-05-09T08:59:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f7001c47740 10 client.0 _ll_register_callbacks cb 0x7f6fe8012660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f6ffd4d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2f3320 0x55b8df2f6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f6ffd4d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2f3320 0x55b8df2f6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57292/0 (socket says 172.21.15.98:57292) 2024-05-09T08:59:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f6ffd4d8640 1 -- 172.21.15.98:0/732221143 learned_addr learned my addr 172.21.15.98:0/732221143 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f6ffccd7640 1 --2- 172.21.15.98:0/732221143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8330 0x55b8df2fb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f6feffff640 1 --2- 172.21.15.98:0/732221143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8c50 0x55b8df2f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f6ffd4d8640 1 -- 172.21.15.98:0/732221143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8c50 msgr2=0x55b8df2f6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f6ffd4d8640 1 --2- 172.21.15.98:0/732221143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8c50 0x55b8df2f6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f6ffd4d8640 1 -- 172.21.15.98:0/732221143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8330 msgr2=0x55b8df2fb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f6ffd4d8640 1 --2- 172.21.15.98:0/732221143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8330 0x55b8df2fb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f6ffd4d8640 1 -- 172.21.15.98:0/732221143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8df2e8a80 con 0x55b8df2f3320 2024-05-09T08:59:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.299+0000 7f6feffff640 1 --2- 172.21.15.98:0/732221143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8c50 0x55b8df2f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f6ffd4d8640 1 --2- 172.21.15.98:0/732221143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2f3320 0x55b8df2f6f80 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00049d0 tx=0x7f6ff009a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21265e2f3749ebe3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f6fd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f6fd7fff640 1 -- 172.21.15.98:0/732221143 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff009bf10 con 0x55b8df2f3320 2024-05-09T08:59:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f6fd7fff640 1 -- 172.21.15.98:0/732221143 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7f6ff009c170 con 0x55b8df2f3320 2024-05-09T08:59:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f7001c47740 1 -- 172.21.15.98:0/732221143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ff4004310 con 0x55b8df2f3320 2024-05-09T08:59:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f6fd7fff640 1 -- 172.21.15.98:0/732221143 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6ff0014430 con 0x55b8df2f3320 2024-05-09T08:59:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f7001c47740 10 client.17994 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f7001c47740 1 -- 172.21.15.98:0/732221143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b8df369980 con 0x55b8df2f3320 2024-05-09T08:59:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f6fd7fff640 1 -- 172.21.15.98:0/732221143 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ff4004310 con 0x55b8df2f3320 2024-05-09T08:59:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f6fd7fff640 1 -- 172.21.15.98:0/732221143 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f6ff00a4780 con 0x55b8df2f3320 2024-05-09T08:59:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f7001c47740 10 client.17994 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f7001c47740 10 client.17994 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:59:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f7001c47740 20 client.17994 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f7001c47740 1 --2- 172.21.15.98:0/732221143 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b8df369c50 0x55b8df36c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f7001c47740 4 client.17994 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:59:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f7001c47740 1 -- 172.21.15.98:0/732221143 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b8df2556c0 con 0x55b8df369c50 2024-05-09T08:59:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.300+0000 7f6ffccd7640 1 --2- 172.21.15.98:0/732221143 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b8df369c50 0x55b8df36c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f6fd7fff640 10 client.17994 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:59:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f6ffccd7640 1 --2- 172.21.15.98:0/732221143 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b8df369c50 0x55b8df36c040 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4576d845650720cb server_cookie=36828ebb90c7efe9 in_seq=0 out_seq=0 2024-05-09T08:59:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f6fd7fff640 1 -- 172.21.15.98:0/732221143 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b8df2556c0 con 0x55b8df369c50 2024-05-09T08:59:17.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f6fd7fff640 10 client.17994 handle_command_reply: tid=0 2024-05-09T08:59:17.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f7001c47740 1 client.17994 shutdown 2024-05-09T08:59:17.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f7001c47740 1 -- 172.21.15.98:0/732221143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2f3320 msgr2=0x55b8df2f6f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f7001c47740 1 --2- 172.21.15.98:0/732221143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2f3320 0x55b8df2f6f80 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00049d0 tx=0x7f6ff009a420 comp rx=0 tx=0).stop 2024-05-09T08:59:17.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f7001c47740 1 -- 172.21.15.98:0/732221143 shutdown_connections 2024-05-09T08:59:17.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f7001c47740 1 --2- 172.21.15.98:0/732221143 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55b8df369c50 0x55b8df36c040 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f7001c47740 1 --2- 172.21.15.98:0/732221143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8df2f3320 0x55b8df2f6f80 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f7001c47740 1 --2- 172.21.15.98:0/732221143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8df2e8c50 0x55b8df2f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f7001c47740 1 --2- 172.21.15.98:0/732221143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b8df2e8330 0x55b8df2fb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f6fd7fff640 0 client.17994 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:59:17.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f7001c47740 1 -- 172.21.15.98:0/732221143 >> 172.21.15.98:0/732221143 conn(0x55b8df2e4650 msgr2=0x55b8df2f1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f7001c47740 1 -- 172.21.15.98:0/732221143 shutdown_connections 2024-05-09T08:59:17.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f7001c47740 1 -- 172.21.15.98:0/732221143 wait complete. 2024-05-09T08:59:17.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.301+0000 7f7001c47740 20 client.17994 trim_cache size 0 max 16384 2024-05-09T08:59:17.309 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:59:17.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.302+0000 7f6ffef3b640 1 -- 172.21.15.98:0/3941083531 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6fe00a2d00 msgr2=0x7f6fe00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.302+0000 7f6ffef3b640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6fe00a2d00 0x7f6fe00a51c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8005fd0 tx=0x7f6fe800a660 comp rx=0 tx=0).stop 2024-05-09T08:59:17.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.302+0000 7f6ffef3b640 1 -- 172.21.15.98:0/3941083531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff810bdf0 msgr2=0x7f6ff8101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.302+0000 7f6ffef3b640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff810bdf0 0x7f6ff8101180 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000eb10 tx=0x7f6ff000c2d0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.302+0000 7f6ffef3b640 1 -- 172.21.15.98:0/3941083531 shutdown_connections 2024-05-09T08:59:17.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.302+0000 7f6ffef3b640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6fe00a2d00 0x7f6fe00a51c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.302+0000 7f6ffef3b640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff810bdf0 0x7f6ff8101180 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.302+0000 7f6ffef3b640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ff81033d0 0x7f6ff8100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.302+0000 7f6ffef3b640 1 --2- 172.21.15.98:0/3941083531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ff8102ab0 0x7f6ff8100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.302+0000 7f6ffef3b640 1 -- 172.21.15.98:0/3941083531 >> 172.21.15.98:0/3941083531 conn(0x7f6ff8077830 msgr2=0x7f6ff80fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.302+0000 7f6ffef3b640 1 -- 172.21.15.98:0/3941083531 shutdown_connections 2024-05-09T08:59:17.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.302+0000 7f6ffef3b640 1 -- 172.21.15.98:0/3941083531 wait complete. 2024-05-09T08:59:17.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:59:17.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.415+0000 7fb429238640 1 -- 172.21.15.98:0/1304986857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb424107810 msgr2=0x7fb424107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.415+0000 7fb429238640 1 --2- 172.21.15.98:0/1304986857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb424107810 0x7fb424107bf0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fb414009930 tx=0x7fb41402a420 comp rx=0 tx=0).stop 2024-05-09T08:59:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.415+0000 7fb429238640 1 -- 172.21.15.98:0/1304986857 shutdown_connections 2024-05-09T08:59:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.415+0000 7fb429238640 1 --2- 172.21.15.98:0/1304986857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb4241021c0 0x7fb42410e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.415+0000 7fb429238640 1 --2- 172.21.15.98:0/1304986857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb424101800 0x7fb424101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.415+0000 7fb429238640 1 --2- 172.21.15.98:0/1304986857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb424107810 0x7fb424107bf0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.415+0000 7fb429238640 1 -- 172.21.15.98:0/1304986857 >> 172.21.15.98:0/1304986857 conn(0x7fb4240fd270 msgr2=0x7fb4240ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.415+0000 7fb429238640 1 -- 172.21.15.98:0/1304986857 shutdown_connections 2024-05-09T08:59:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.415+0000 7fb429238640 1 -- 172.21.15.98:0/1304986857 wait complete. 2024-05-09T08:59:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.416+0000 7fb429238640 1 Processor -- start 2024-05-09T08:59:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.416+0000 7fb429238640 1 -- start start 2024-05-09T08:59:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.416+0000 7fb429238640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb424101800 0x7fb4241981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.416+0000 7fb429238640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4241021c0 0x7fb4241986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.416+0000 7fb429238640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb424107810 0x7fb42419ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.416+0000 7fb429238640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb424111c20 con 0x7fb424101800 2024-05-09T08:59:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.416+0000 7fb429238640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb424111aa0 con 0x7fb4241021c0 2024-05-09T08:59:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.416+0000 7fb429238640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb424111da0 con 0x7fb424107810 2024-05-09T08:59:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.416+0000 7fb422d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb424101800 0x7fb4241981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.416+0000 7fb422d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb424101800 0x7fb4241981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57306/0 (socket says 172.21.15.98:57306) 2024-05-09T08:59:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.416+0000 7fb422d9d640 1 -- 172.21.15.98:0/1001749804 learned_addr learned my addr 172.21.15.98:0/1001749804 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.416+0000 7fb42259c640 1 --2- 172.21.15.98:0/1001749804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4241021c0 0x7fb4241986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.417+0000 7fb42359e640 1 --2- 172.21.15.98:0/1001749804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb424107810 0x7fb42419ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.417+0000 7fb422d9d640 1 -- 172.21.15.98:0/1001749804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb424107810 msgr2=0x7fb42419ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.417+0000 7fb422d9d640 1 --2- 172.21.15.98:0/1001749804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb424107810 0x7fb42419ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.417+0000 7fb422d9d640 1 -- 172.21.15.98:0/1001749804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4241021c0 msgr2=0x7fb4241986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.417+0000 7fb422d9d640 1 --2- 172.21.15.98:0/1001749804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4241021c0 0x7fb4241986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.417+0000 7fb422d9d640 1 -- 172.21.15.98:0/1001749804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42419d140 con 0x7fb424101800 2024-05-09T08:59:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.417+0000 7fb422d9d640 1 --2- 172.21.15.98:0/1001749804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb424101800 0x7fb4241981a0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fb414009230 tx=0x7fb414004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.417+0000 7fb41bfff640 1 -- 172.21.15.98:0/1001749804 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41403f030 con 0x7fb424101800 2024-05-09T08:59:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.417+0000 7fb429238640 1 -- 172.21.15.98:0/1001749804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb42419d3d0 con 0x7fb424101800 2024-05-09T08:59:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.417+0000 7fb41bfff640 1 -- 172.21.15.98:0/1001749804 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb414003ad0 con 0x7fb424101800 2024-05-09T08:59:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.417+0000 7fb41bfff640 1 -- 172.21.15.98:0/1001749804 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414035070 con 0x7fb424101800 2024-05-09T08:59:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.417+0000 7fb429238640 1 -- 172.21.15.98:0/1001749804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4241a4cc0 con 0x7fb424101800 2024-05-09T08:59:17.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.419+0000 7fb41bfff640 1 -- 172.21.15.98:0/1001749804 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb414042030 con 0x7fb424101800 2024-05-09T08:59:17.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.419+0000 7fb42c021740 1 Processor -- start 2024-05-09T08:59:17.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.419+0000 7fb42c021740 1 -- start start 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42c021740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd668710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42c021740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd668c50 0x55a7fd672de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42c021740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd673320 0x55a7fd675710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42c021740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a7fd5cdc40 con 0x55a7fd668c50 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42c021740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a7fd575580 con 0x55a7fd673320 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42c021740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a7fd5c9c00 con 0x55a7fd668330 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb41bfff640 1 --2- 172.21.15.98:0/1001749804 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb4100a2d00 0x7fb4100a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42359e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd668710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42359e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd668710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:55654/0 (socket says 172.21.15.98:55654) 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb422d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd668c50 0x55a7fd672de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42359e640 1 -- 172.21.15.98:0/2646323919 learned_addr learned my addr 172.21.15.98:0/2646323919 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42259c640 1 --2- 172.21.15.98:0/2646323919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd673320 0x55a7fd675710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42359e640 1 --2- 172.21.15.98:0/1001749804 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb4100a2d00 0x7fb4100a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb41bfff640 1 -- 172.21.15.98:0/1001749804 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7fb4140efce0 con 0x7fb424101800 2024-05-09T08:59:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42359e640 1 -- 172.21.15.98:0/2646323919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd673320 msgr2=0x55a7fd675710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42359e640 1 --2- 172.21.15.98:0/2646323919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd673320 0x55a7fd675710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42359e640 1 -- 172.21.15.98:0/2646323919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd668c50 msgr2=0x55a7fd672de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42359e640 1 --2- 172.21.15.98:0/2646323919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd668c50 0x55a7fd672de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.420+0000 7fb42359e640 1 -- 172.21.15.98:0/2646323919 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7fd2e13f0 con 0x55a7fd668330 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.421+0000 7fb42359e640 1 --2- 172.21.15.98:0/1001749804 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb4100a2d00 0x7fb4100a51c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009fd0 tx=0x7fb41c01c000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.421+0000 7fb42359e640 1 --2- 172.21.15.98:0/2646323919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd668710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb41c014e00 tx=0x7fb41c0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3541e9c01faf229 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.421+0000 7fb4197fa640 1 -- 172.21.15.98:0/2646323919 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c00ccb0 con 0x55a7fd668330 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.421+0000 7fb4197fa640 1 -- 172.21.15.98:0/2646323919 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb41c00ce50 con 0x55a7fd668330 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.421+0000 7fb4197fa640 1 -- 172.21.15.98:0/2646323919 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c004710 con 0x55a7fd668330 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 -- 172.21.15.98:0/2646323919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 msgr2=0x55a7fd668710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 --2- 172.21.15.98:0/2646323919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd668710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb41c014e00 tx=0x7fb41c0117b0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 -- 172.21.15.98:0/2646323919 shutdown_connections 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 --2- 172.21.15.98:0/2646323919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd673320 0x55a7fd675710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 --2- 172.21.15.98:0/2646323919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd668c50 0x55a7fd672de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 --2- 172.21.15.98:0/2646323919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd668710 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 -- 172.21.15.98:0/2646323919 >> 172.21.15.98:0/2646323919 conn(0x55a7fd664650 msgr2=0x55a7fd664a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 -- 172.21.15.98:0/2646323919 shutdown_connections 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 -- 172.21.15.98:0/2646323919 wait complete. 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 -- start start 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 Processor -- start 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd668330 0x55a7fd670240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668c50 0x55a7fd670780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd673320 0x55a7fd670cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a7fd67a4c0 con 0x55a7fd673320 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a7fd588c40 con 0x55a7fd668330 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a7fd67a640 con 0x55a7fd668c50 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42c021740 10 client.0 _ll_register_callbacks cb 0x55a7fd2e0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd673320 0x55a7fd670cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd673320 0x55a7fd670cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57318/0 (socket says 172.21.15.98:57318) 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.422+0000 7fb42359e640 1 -- 172.21.15.98:0/4063822445 learned_addr learned my addr 172.21.15.98:0/4063822445 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb422d9d640 1 --2- 172.21.15.98:0/4063822445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd668330 0x55a7fd670240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb42259c640 1 --2- 172.21.15.98:0/4063822445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668c50 0x55a7fd670780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb42359e640 1 -- 172.21.15.98:0/4063822445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668c50 msgr2=0x55a7fd670780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb42359e640 1 --2- 172.21.15.98:0/4063822445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668c50 0x55a7fd670780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb42359e640 1 -- 172.21.15.98:0/4063822445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd668330 msgr2=0x55a7fd670240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb42359e640 1 --2- 172.21.15.98:0/4063822445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd668330 0x55a7fd670240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb42359e640 1 -- 172.21.15.98:0/4063822445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7fd3ab7d0 con 0x55a7fd673320 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb42259c640 1 --2- 172.21.15.98:0/4063822445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668c50 0x55a7fd670780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb42359e640 1 --2- 172.21.15.98:0/4063822445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd673320 0x55a7fd670cc0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fb41c014540 tx=0x7fb41c014aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=122ec397af9b6498 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb40a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb40a7fc640 1 -- 172.21.15.98:0/4063822445 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c009c20 con 0x55a7fd673320 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb40a7fc640 1 -- 172.21.15.98:0/4063822445 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7fb41c004410 con 0x55a7fd673320 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb42c021740 1 -- 172.21.15.98:0/4063822445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb41c00ce50 con 0x55a7fd673320 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb40a7fc640 1 -- 172.21.15.98:0/4063822445 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb41c004610 con 0x55a7fd673320 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb42c021740 10 client.18006 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:17.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb42c021740 1 -- 172.21.15.98:0/4063822445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a7fd6866d0 con 0x55a7fd673320 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.423+0000 7fb40a7fc640 1 -- 172.21.15.98:0/4063822445 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb41c00ce50 con 0x55a7fd673320 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.424+0000 7fb40a7fc640 1 -- 172.21.15.98:0/4063822445 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb41c02d070 con 0x55a7fd673320 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.424+0000 7fb42c021740 10 client.18006 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.424+0000 7fb42c021740 10 client.18006 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.424+0000 7fb42c021740 20 client.18006 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.424+0000 7fb42c021740 1 --2- 172.21.15.98:0/4063822445 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a7fd687ae0 0x55a7fd689ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.424+0000 7fb42c021740 4 client.18006 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.424+0000 7fb42c021740 1 -- 172.21.15.98:0/4063822445 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7fd5db370 con 0x55a7fd687ae0 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.424+0000 7fb422d9d640 1 --2- 172.21.15.98:0/4063822445 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a7fd687ae0 0x55a7fd689ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb422d9d640 1 --2- 172.21.15.98:0/4063822445 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a7fd687ae0 0x55a7fd689ed0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=96338b15cb14dbf9 server_cookie=63335ce8756582b5 in_seq=0 out_seq=0 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb40a7fc640 10 client.18006 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb40a7fc640 1 -- 172.21.15.98:0/4063822445 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a7fd5db370 con 0x55a7fd687ae0 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb40a7fc640 10 client.18006 handle_command_reply: tid=0 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb42c021740 1 client.18006 shutdown 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb42c021740 1 -- 172.21.15.98:0/4063822445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd673320 msgr2=0x55a7fd670cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb42c021740 1 --2- 172.21.15.98:0/4063822445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd673320 0x55a7fd670cc0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fb41c014540 tx=0x7fb41c014aa0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb42c021740 1 -- 172.21.15.98:0/4063822445 shutdown_connections 2024-05-09T08:59:17.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb42c021740 1 --2- 172.21.15.98:0/4063822445 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a7fd687ae0 0x55a7fd689ed0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb42c021740 1 --2- 172.21.15.98:0/4063822445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd673320 0x55a7fd670cc0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb42c021740 1 --2- 172.21.15.98:0/4063822445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668c50 0x55a7fd670780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb42c021740 1 --2- 172.21.15.98:0/4063822445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd668330 0x55a7fd670240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb40a7fc640 0 client.18006 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:59:17.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb42c021740 1 -- 172.21.15.98:0/4063822445 >> 172.21.15.98:0/4063822445 conn(0x55a7fd664650 msgr2=0x55a7fd66a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.425+0000 7fb42c021740 1 -- 172.21.15.98:0/4063822445 shutdown_connections 2024-05-09T08:59:17.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.426+0000 7fb42c021740 1 -- 172.21.15.98:0/4063822445 wait complete. 2024-05-09T08:59:17.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.426+0000 7fb42c021740 20 client.18006 trim_cache size 0 max 16384 2024-05-09T08:59:17.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42c021740 1 Processor -- start 2024-05-09T08:59:17.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42c021740 1 -- start start 2024-05-09T08:59:17.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42c021740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd6833b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42c021740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd668c50 0x55a7fd66aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42c021740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd673320 0x55a7fd66f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42c021740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4241acdc0 con 0x55a7fd668c50 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42c021740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb4241ae900 con 0x55a7fd673320 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42c021740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb4241ad490 con 0x55a7fd668330 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd668c50 0x55a7fd66aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd668c50 0x55a7fd66aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57320/0 (socket says 172.21.15.98:57320) 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42259c640 1 -- 172.21.15.98:0/4026985843 learned_addr learned my addr 172.21.15.98:0/4026985843 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb422d9d640 1 --2- 172.21.15.98:0/4026985843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd6833b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42259c640 1 -- 172.21.15.98:0/4026985843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 msgr2=0x55a7fd6833b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42259c640 1 --2- 172.21.15.98:0/4026985843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd6833b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42259c640 1 -- 172.21.15.98:0/4026985843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd673320 msgr2=0x55a7fd66f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42259c640 1 --2- 172.21.15.98:0/4026985843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd673320 0x55a7fd66f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb42259c640 1 -- 172.21.15.98:0/4026985843 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7fd668a80 con 0x55a7fd668c50 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.439+0000 7fb422d9d640 1 --2- 172.21.15.98:0/4026985843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd6833b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.440+0000 7fb42259c640 1 --2- 172.21.15.98:0/4026985843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd668c50 0x55a7fd66aa90 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fb40c0045b0 tx=0x7fb40c00bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49cbf920b5a705e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.440+0000 7fb418ff9640 1 -- 172.21.15.98:0/4026985843 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c00bdf0 con 0x55a7fd668c50 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.440+0000 7fb418ff9640 1 -- 172.21.15.98:0/4026985843 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb40c011440 con 0x55a7fd668c50 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.440+0000 7fb418ff9640 1 -- 172.21.15.98:0/4026985843 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c010690 con 0x55a7fd668c50 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.440+0000 7fb42c021740 1 -- 172.21.15.98:0/4026985843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd668c50 msgr2=0x55a7fd66aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.440+0000 7fb42c021740 1 --2- 172.21.15.98:0/4026985843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd668c50 0x55a7fd66aa90 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fb40c0045b0 tx=0x7fb40c00bc00 comp rx=0 tx=0).stop 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.440+0000 7fb42c021740 1 -- 172.21.15.98:0/4026985843 shutdown_connections 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.440+0000 7fb42c021740 1 --2- 172.21.15.98:0/4026985843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd673320 0x55a7fd66f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.440+0000 7fb42c021740 1 --2- 172.21.15.98:0/4026985843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd668c50 0x55a7fd66aa90 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.440+0000 7fb42c021740 1 --2- 172.21.15.98:0/4026985843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd6833b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42c021740 1 -- 172.21.15.98:0/4026985843 >> 172.21.15.98:0/4026985843 conn(0x55a7fd664650 msgr2=0x55a7fd662f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42c021740 1 -- 172.21.15.98:0/4026985843 shutdown_connections 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42c021740 1 -- 172.21.15.98:0/4026985843 wait complete. 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42c021740 1 -- start start 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42c021740 1 Processor -- start 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42c021740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd67f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42c021740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd668c50 0x55a7fd67ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42c021740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd673320 0x55a7fd67b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42c021740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4241ae230 con 0x55a7fd673320 2024-05-09T08:59:17.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42c021740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb4241adb60 con 0x55a7fd668c50 2024-05-09T08:59:17.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42c021740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb4241aadc0 con 0x55a7fd668330 2024-05-09T08:59:17.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42c021740 10 client.0 _ll_register_callbacks cb 0x7fb41c014f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:17.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd673320 0x55a7fd67b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd673320 0x55a7fd67b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57334/0 (socket says 172.21.15.98:57334) 2024-05-09T08:59:17.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42359e640 1 -- 172.21.15.98:0/2283191621 learned_addr learned my addr 172.21.15.98:0/2283191621 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb42259c640 1 --2- 172.21.15.98:0/2283191621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd668c50 0x55a7fd67ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.441+0000 7fb422d9d640 1 --2- 172.21.15.98:0/2283191621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd67f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42359e640 1 -- 172.21.15.98:0/2283191621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 msgr2=0x55a7fd67f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42359e640 1 --2- 172.21.15.98:0/2283191621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd67f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42359e640 1 -- 172.21.15.98:0/2283191621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd668c50 msgr2=0x55a7fd67ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42359e640 1 --2- 172.21.15.98:0/2283191621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd668c50 0x55a7fd67ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42359e640 1 -- 172.21.15.98:0/2283191621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7fd67b8a0 con 0x55a7fd673320 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42359e640 1 --2- 172.21.15.98:0/2283191621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd673320 0x55a7fd67b360 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fb41c002230 tx=0x7fb41c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2965d493136aa30 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb40a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb40a7fc640 1 -- 172.21.15.98:0/2283191621 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c011cb0 con 0x55a7fd673320 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42c021740 1 -- 172.21.15.98:0/2283191621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb40c011440 con 0x55a7fd673320 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb40a7fc640 1 -- 172.21.15.98:0/2283191621 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7fb41c014260 con 0x55a7fd673320 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb40a7fc640 1 -- 172.21.15.98:0/2283191621 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb41c0042c0 con 0x55a7fd673320 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42c021740 10 client.18012 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42c021740 1 -- 172.21.15.98:0/2283191621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a7fd6edc80 con 0x55a7fd673320 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb40a7fc640 1 -- 172.21.15.98:0/2283191621 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb40c011440 con 0x55a7fd673320 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb40a7fc640 1 -- 172.21.15.98:0/2283191621 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb41c02d070 con 0x55a7fd673320 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42c021740 10 client.18012 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42c021740 10 client.18012 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42c021740 20 client.18012 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42c021740 1 --2- 172.21.15.98:0/2283191621 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a7fd6ee2e0 0x55a7fd6f06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42c021740 4 client.18012 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:59:17.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.442+0000 7fb42c021740 1 -- 172.21.15.98:0/2283191621 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a7fd5d56c0 con 0x55a7fd6ee2e0 2024-05-09T08:59:17.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb422d9d640 1 --2- 172.21.15.98:0/2283191621 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a7fd6ee2e0 0x55a7fd6f06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb40a7fc640 10 client.18012 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:59:17.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb422d9d640 1 --2- 172.21.15.98:0/2283191621 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a7fd6ee2e0 0x55a7fd6f06d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6a34b47bab991362 server_cookie=efe986405937985f in_seq=0 out_seq=0 2024-05-09T08:59:17.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb40a7fc640 1 -- 172.21.15.98:0/2283191621 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a7fd5d56c0 con 0x55a7fd6ee2e0 2024-05-09T08:59:17.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb40a7fc640 10 client.18012 handle_command_reply: tid=0 2024-05-09T08:59:17.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb42c021740 1 client.18012 shutdown 2024-05-09T08:59:17.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb42c021740 1 -- 172.21.15.98:0/2283191621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd673320 msgr2=0x55a7fd67b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb42c021740 1 --2- 172.21.15.98:0/2283191621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd673320 0x55a7fd67b360 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fb41c002230 tx=0x7fb41c005fd0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb42c021740 1 -- 172.21.15.98:0/2283191621 shutdown_connections 2024-05-09T08:59:17.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb42c021740 1 --2- 172.21.15.98:0/2283191621 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a7fd6ee2e0 0x55a7fd6f06d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb42c021740 1 --2- 172.21.15.98:0/2283191621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fd673320 0x55a7fd67b360 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb42c021740 1 --2- 172.21.15.98:0/2283191621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a7fd668c50 0x55a7fd67ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb40a7fc640 0 client.18012 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:59:17.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb42c021740 1 --2- 172.21.15.98:0/2283191621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a7fd668330 0x55a7fd67f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb42c021740 1 -- 172.21.15.98:0/2283191621 >> 172.21.15.98:0/2283191621 conn(0x55a7fd664650 msgr2=0x55a7fd6701e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb42c021740 1 -- 172.21.15.98:0/2283191621 shutdown_connections 2024-05-09T08:59:17.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb42c021740 1 -- 172.21.15.98:0/2283191621 wait complete. 2024-05-09T08:59:17.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.443+0000 7fb42c021740 20 client.18012 trim_cache size 0 max 16384 2024-05-09T08:59:17.450 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:59:17.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.444+0000 7fb429238640 1 -- 172.21.15.98:0/1001749804 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb4100a2d00 msgr2=0x7fb4100a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.444+0000 7fb429238640 1 --2- 172.21.15.98:0/1001749804 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb4100a2d00 0x7fb4100a51c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009fd0 tx=0x7fb41c01c000 comp rx=0 tx=0).stop 2024-05-09T08:59:17.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.444+0000 7fb429238640 1 -- 172.21.15.98:0/1001749804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb424101800 msgr2=0x7fb4241981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.444+0000 7fb429238640 1 --2- 172.21.15.98:0/1001749804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb424101800 0x7fb4241981a0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fb414009230 tx=0x7fb414004270 comp rx=0 tx=0).stop 2024-05-09T08:59:17.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.444+0000 7fb429238640 1 -- 172.21.15.98:0/1001749804 shutdown_connections 2024-05-09T08:59:17.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.444+0000 7fb429238640 1 --2- 172.21.15.98:0/1001749804 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb4100a2d00 0x7fb4100a51c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.444+0000 7fb429238640 1 --2- 172.21.15.98:0/1001749804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb424107810 0x7fb42419ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.444+0000 7fb429238640 1 --2- 172.21.15.98:0/1001749804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4241021c0 0x7fb4241986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.444+0000 7fb429238640 1 --2- 172.21.15.98:0/1001749804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb424101800 0x7fb4241981a0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.444+0000 7fb429238640 1 -- 172.21.15.98:0/1001749804 >> 172.21.15.98:0/1001749804 conn(0x7fb4240fd270 msgr2=0x7fb4240fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.444+0000 7fb429238640 1 -- 172.21.15.98:0/1001749804 shutdown_connections 2024-05-09T08:59:17.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.444+0000 7fb429238640 1 -- 172.21.15.98:0/1001749804 wait complete. 2024-05-09T08:59:17.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:59:17.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.557+0000 7f3b4149a640 1 -- 172.21.15.98:0/276480414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c107860 msgr2=0x7f3b3c107c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.557+0000 7f3b4149a640 1 --2- 172.21.15.98:0/276480414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c107860 0x7f3b3c107c40 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009930 tx=0x7f3b2c02a420 comp rx=0 tx=0).stop 2024-05-09T08:59:17.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.557+0000 7f3b4149a640 1 -- 172.21.15.98:0/276480414 shutdown_connections 2024-05-09T08:59:17.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.557+0000 7f3b4149a640 1 --2- 172.21.15.98:0/276480414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b3c102210 0x7f3b3c10e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.557+0000 7f3b4149a640 1 --2- 172.21.15.98:0/276480414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b3c101850 0x7f3b3c101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.557+0000 7f3b4149a640 1 --2- 172.21.15.98:0/276480414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c107860 0x7f3b3c107c40 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.557+0000 7f3b4149a640 1 -- 172.21.15.98:0/276480414 >> 172.21.15.98:0/276480414 conn(0x7f3b3c0fd340 msgr2=0x7f3b3c0ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.557+0000 7f3b4149a640 1 -- 172.21.15.98:0/276480414 shutdown_connections 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.557+0000 7f3b4149a640 1 -- 172.21.15.98:0/276480414 wait complete. 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.558+0000 7f3b4149a640 1 Processor -- start 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.558+0000 7f3b4149a640 1 -- start start 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.558+0000 7f3b4149a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c101850 0x7f3b3c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.558+0000 7f3b4149a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b3c102210 0x7f3b3c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.558+0000 7f3b4149a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b3c107860 0x7f3b3c0745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.558+0000 7f3b4149a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c111c30 con 0x7f3b3c101850 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.558+0000 7f3b4149a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c111ab0 con 0x7f3b3c107860 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.558+0000 7f3b4149a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c111db0 con 0x7f3b3c102210 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.558+0000 7f3b3affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c101850 0x7f3b3c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.558+0000 7f3b3affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c101850 0x7f3b3c073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57348/0 (socket says 172.21.15.98:57348) 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.558+0000 7f3b3affd640 1 -- 172.21.15.98:0/3699280388 learned_addr learned my addr 172.21.15.98:0/3699280388 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.559+0000 7f3b3b7fe640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b3c107860 0x7f3b3c0745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.559+0000 7f3b3a7fc640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b3c102210 0x7f3b3c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.559+0000 7f3b3affd640 1 -- 172.21.15.98:0/3699280388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b3c102210 msgr2=0x7f3b3c074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.559+0000 7f3b3affd640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b3c102210 0x7f3b3c074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.559+0000 7f3b3affd640 1 -- 172.21.15.98:0/3699280388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b3c107860 msgr2=0x7f3b3c0745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.559+0000 7f3b3affd640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b3c107860 0x7f3b3c0745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.559+0000 7f3b3affd640 1 -- 172.21.15.98:0/3699280388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3c1a6670 con 0x7f3b3c101850 2024-05-09T08:59:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.559+0000 7f3b3a7fc640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b3c102210 0x7f3b3c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.559+0000 7f3b3affd640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c101850 0x7f3b3c073b30 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009230 tx=0x7f3b2c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.559+0000 7f3b33fff640 1 -- 172.21.15.98:0/3699280388 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c03f030 con 0x7f3b3c101850 2024-05-09T08:59:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.559+0000 7f3b4149a640 1 -- 172.21.15.98:0/3699280388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b3c1a6900 con 0x7f3b3c101850 2024-05-09T08:59:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.559+0000 7f3b33fff640 1 -- 172.21.15.98:0/3699280388 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b2c003ad0 con 0x7f3b3c101850 2024-05-09T08:59:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.559+0000 7f3b33fff640 1 -- 172.21.15.98:0/3699280388 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c035070 con 0x7f3b3c101850 2024-05-09T08:59:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.559+0000 7f3b4149a640 1 -- 172.21.15.98:0/3699280388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b3c1a6df0 con 0x7f3b3c101850 2024-05-09T08:59:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.561+0000 7f3b33fff640 1 -- 172.21.15.98:0/3699280388 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3b2c042030 con 0x7f3b3c101850 2024-05-09T08:59:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b441a2740 1 Processor -- start 2024-05-09T08:59:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b441a2740 1 -- start start 2024-05-09T08:59:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b441a2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482d9330 0x5631482d9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b441a2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 0x5631482e3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b441a2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482e4320 0x5631482e6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b441a2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56314823ec40 con 0x5631482d9330 2024-05-09T08:59:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b441a2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5631481e6580 con 0x5631482e4320 2024-05-09T08:59:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b441a2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56314823ac00 con 0x5631482d9c50 2024-05-09T08:59:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b3b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482d9330 0x5631482d9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b3b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482d9330 0x5631482d9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57354/0 (socket says 172.21.15.98:57354) 2024-05-09T08:59:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b3b7fe640 1 -- 172.21.15.98:0/677655217 learned_addr learned my addr 172.21.15.98:0/677655217 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b33fff640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3b280a2d00 0x7f3b280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b3a7fc640 1 --2- 172.21.15.98:0/677655217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482e4320 0x5631482e6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b3affd640 1 --2- 172.21.15.98:0/677655217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 0x5631482e3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.562+0000 7f3b3b7fe640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3b280a2d00 0x7f3b280a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.563+0000 7f3b3b7fe640 1 -- 172.21.15.98:0/677655217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 msgr2=0x5631482e3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.563+0000 7f3b3b7fe640 1 --2- 172.21.15.98:0/677655217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 0x5631482e3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.563+0000 7f3b3b7fe640 1 -- 172.21.15.98:0/677655217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482e4320 msgr2=0x5631482e6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.563+0000 7f3b33fff640 1 -- 172.21.15.98:0/3699280388 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7f3b2c0efdb0 con 0x7f3b3c101850 2024-05-09T08:59:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.563+0000 7f3b3b7fe640 1 --2- 172.21.15.98:0/677655217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482e4320 0x5631482e6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.563+0000 7f3b3b7fe640 1 -- 172.21.15.98:0/677655217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563147f523f0 con 0x5631482d9330 2024-05-09T08:59:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.563+0000 7f3b3a7fc640 1 --2- 172.21.15.98:0/677655217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482e4320 0x5631482e6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.563+0000 7f3b3b7fe640 1 --2- 172.21.15.98:0/677655217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482d9330 0x5631482d9710 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f3b34013940 tx=0x7f3b34013e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6edc048ba1ee6cc8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.563+0000 7f3b317fa640 1 -- 172.21.15.98:0/677655217 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b34021030 con 0x5631482d9330 2024-05-09T08:59:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.563+0000 7f3b317fa640 1 -- 172.21.15.98:0/677655217 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b340043a0 con 0x5631482d9330 2024-05-09T08:59:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.563+0000 7f3b317fa640 1 -- 172.21.15.98:0/677655217 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3401d860 con 0x5631482d9330 2024-05-09T08:59:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.563+0000 7f3b3b7fe640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3b280a2d00 0x7f3b280a51c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3b34021000 tx=0x7f3b34020000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 -- 172.21.15.98:0/677655217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482d9330 msgr2=0x5631482d9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 --2- 172.21.15.98:0/677655217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482d9330 0x5631482d9710 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f3b34013940 tx=0x7f3b34013e00 comp rx=0 tx=0).stop 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 -- 172.21.15.98:0/677655217 shutdown_connections 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 --2- 172.21.15.98:0/677655217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482e4320 0x5631482e6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 --2- 172.21.15.98:0/677655217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 0x5631482e3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 --2- 172.21.15.98:0/677655217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482d9330 0x5631482d9710 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 -- 172.21.15.98:0/677655217 >> 172.21.15.98:0/677655217 conn(0x5631482d5650 msgr2=0x5631482d5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 -- 172.21.15.98:0/677655217 shutdown_connections 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 -- 172.21.15.98:0/677655217 wait complete. 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 -- start start 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 Processor -- start 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9330 0x5631482e1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 0x5631482e17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482e4320 0x5631482e1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5631482e7110 con 0x5631482e4320 2024-05-09T08:59:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5631481f9c40 con 0x5631482d9330 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5631482e7290 con 0x5631482d9c50 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b441a2740 10 client.0 _ll_register_callbacks cb 0x563147f51be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b3b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482e4320 0x5631482e1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b3b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482e4320 0x5631482e1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57368/0 (socket says 172.21.15.98:57368) 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.564+0000 7f3b3b7fe640 1 -- 172.21.15.98:0/4053430139 learned_addr learned my addr 172.21.15.98:0/4053430139 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b3a7fc640 1 --2- 172.21.15.98:0/4053430139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 0x5631482e17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b3affd640 1 --2- 172.21.15.98:0/4053430139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9330 0x5631482e1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b3b7fe640 1 -- 172.21.15.98:0/4053430139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 msgr2=0x5631482e17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b3b7fe640 1 --2- 172.21.15.98:0/4053430139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 0x5631482e17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b3b7fe640 1 -- 172.21.15.98:0/4053430139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9330 msgr2=0x5631482e1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b3b7fe640 1 --2- 172.21.15.98:0/4053430139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9330 0x5631482e1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b3b7fe640 1 -- 172.21.15.98:0/4053430139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56314801c7d0 con 0x5631482e4320 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b3b7fe640 1 --2- 172.21.15.98:0/4053430139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482e4320 0x5631482e1ce0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400aea0 tx=0x7f3b3400a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbdb43b421098c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b227fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b227fc640 1 -- 172.21.15.98:0/4053430139 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b34003ec0 con 0x5631482e4320 2024-05-09T08:59:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b227fc640 1 -- 172.21.15.98:0/4053430139 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7f3b34004060 con 0x5631482e4320 2024-05-09T08:59:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b441a2740 1 -- 172.21.15.98:0/4053430139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b340043a0 con 0x5631482e4320 2024-05-09T08:59:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b227fc640 1 -- 172.21.15.98:0/4053430139 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3b3403b050 con 0x5631482e4320 2024-05-09T08:59:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b441a2740 10 client.18024 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b441a2740 1 -- 172.21.15.98:0/4053430139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5631482f3300 con 0x5631482e4320 2024-05-09T08:59:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.565+0000 7f3b227fc640 1 -- 172.21.15.98:0/4053430139 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b340043a0 con 0x5631482e4320 2024-05-09T08:59:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.566+0000 7f3b227fc640 1 -- 172.21.15.98:0/4053430139 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3b34005330 con 0x5631482e4320 2024-05-09T08:59:17.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.566+0000 7f3b441a2740 10 client.18024 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:17.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.566+0000 7f3b441a2740 10 client.18024 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:59:17.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.566+0000 7f3b441a2740 20 client.18024 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:17.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.566+0000 7f3b441a2740 1 --2- 172.21.15.98:0/4053430139 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5631482f4700 0x5631482f6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.566+0000 7f3b441a2740 4 client.18024 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:17.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.566+0000 7f3b441a2740 1 -- 172.21.15.98:0/4053430139 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56314824c370 con 0x5631482f4700 2024-05-09T08:59:17.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.566+0000 7f3b3affd640 1 --2- 172.21.15.98:0/4053430139 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5631482f4700 0x5631482f6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.566+0000 7f3b227fc640 10 client.18024 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:59:17.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.566+0000 7f3b3affd640 1 --2- 172.21.15.98:0/4053430139 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5631482f4700 0x5631482f6af0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c21a5e0819c09de7 server_cookie=a85f98541697be2e in_seq=0 out_seq=0 2024-05-09T08:59:17.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b227fc640 1 -- 172.21.15.98:0/4053430139 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56314824c370 con 0x5631482f4700 2024-05-09T08:59:17.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b227fc640 10 client.18024 handle_command_reply: tid=0 2024-05-09T08:59:17.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b441a2740 1 client.18024 shutdown 2024-05-09T08:59:17.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b441a2740 1 -- 172.21.15.98:0/4053430139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482e4320 msgr2=0x5631482e1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b441a2740 1 --2- 172.21.15.98:0/4053430139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482e4320 0x5631482e1ce0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400aea0 tx=0x7f3b3400a540 comp rx=0 tx=0).stop 2024-05-09T08:59:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b441a2740 1 -- 172.21.15.98:0/4053430139 shutdown_connections 2024-05-09T08:59:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b441a2740 1 --2- 172.21.15.98:0/4053430139 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5631482f4700 0x5631482f6af0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b441a2740 1 --2- 172.21.15.98:0/4053430139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482e4320 0x5631482e1ce0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b441a2740 1 --2- 172.21.15.98:0/4053430139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 0x5631482e17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b227fc640 0 client.18024 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:59:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b441a2740 1 --2- 172.21.15.98:0/4053430139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9330 0x5631482e1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b441a2740 1 -- 172.21.15.98:0/4053430139 >> 172.21.15.98:0/4053430139 conn(0x5631482d5650 msgr2=0x5631482db350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b441a2740 1 -- 172.21.15.98:0/4053430139 shutdown_connections 2024-05-09T08:59:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b441a2740 1 -- 172.21.15.98:0/4053430139 wait complete. 2024-05-09T08:59:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.567+0000 7f3b441a2740 20 client.18024 trim_cache size 0 max 16384 2024-05-09T08:59:17.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.580+0000 7f3b441a2740 1 Processor -- start 2024-05-09T08:59:17.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.580+0000 7f3b441a2740 1 -- start start 2024-05-09T08:59:17.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b441a2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482d9330 0x5631482f0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b441a2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9c50 0x5631482e3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b441a2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482e4320 0x5631482fc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b441a2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c1b1140 con 0x5631482d9330 2024-05-09T08:59:17.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b441a2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c1b2c80 con 0x5631482d9c50 2024-05-09T08:59:17.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b441a2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c1b1810 con 0x5631482e4320 2024-05-09T08:59:17.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b3affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482d9330 0x5631482f0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b3affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482d9330 0x5631482f0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57382/0 (socket says 172.21.15.98:57382) 2024-05-09T08:59:17.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b3affd640 1 -- 172.21.15.98:0/1804044085 learned_addr learned my addr 172.21.15.98:0/1804044085 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b3affd640 1 -- 172.21.15.98:0/1804044085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482e4320 msgr2=0x5631482fc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b3affd640 1 --2- 172.21.15.98:0/1804044085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482e4320 0x5631482fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b3affd640 1 -- 172.21.15.98:0/1804044085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9c50 msgr2=0x5631482e3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b3affd640 1 --2- 172.21.15.98:0/1804044085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9c50 0x5631482e3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b3affd640 1 -- 172.21.15.98:0/1804044085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631482e6ff0 con 0x5631482d9330 2024-05-09T08:59:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b3affd640 1 --2- 172.21.15.98:0/1804044085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482d9330 0x5631482f0000 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0bbb60 tx=0x7f3b2c0dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ddcf687d41bcff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b30ff9640 1 -- 172.21.15.98:0/1804044085 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c049040 con 0x5631482d9330 2024-05-09T08:59:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b30ff9640 1 -- 172.21.15.98:0/1804044085 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b2c0c4ad0 con 0x5631482d9330 2024-05-09T08:59:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.581+0000 7f3b30ff9640 1 -- 172.21.15.98:0/1804044085 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c003c80 con 0x5631482d9330 2024-05-09T08:59:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.582+0000 7f3b441a2740 1 -- 172.21.15.98:0/1804044085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482d9330 msgr2=0x5631482f0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.582+0000 7f3b441a2740 1 --2- 172.21.15.98:0/1804044085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482d9330 0x5631482f0000 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0bbb60 tx=0x7f3b2c0dc040 comp rx=0 tx=0).stop 2024-05-09T08:59:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.582+0000 7f3b441a2740 1 -- 172.21.15.98:0/1804044085 shutdown_connections 2024-05-09T08:59:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.582+0000 7f3b441a2740 1 --2- 172.21.15.98:0/1804044085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482e4320 0x5631482fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.582+0000 7f3b441a2740 1 --2- 172.21.15.98:0/1804044085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9c50 0x5631482e3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.582+0000 7f3b441a2740 1 --2- 172.21.15.98:0/1804044085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482d9330 0x5631482f0000 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.582+0000 7f3b441a2740 1 -- 172.21.15.98:0/1804044085 >> 172.21.15.98:0/1804044085 conn(0x5631482d5650 msgr2=0x5631482e2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.582+0000 7f3b441a2740 1 -- 172.21.15.98:0/1804044085 shutdown_connections 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.582+0000 7f3b441a2740 1 -- 172.21.15.98:0/1804044085 wait complete. 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.582+0000 7f3b441a2740 1 -- start start 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.582+0000 7f3b441a2740 1 Processor -- start 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b441a2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9330 0x5631482ec510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b441a2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 0x5631482eca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b441a2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482e4320 0x5631482e94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b441a2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c1b25b0 con 0x5631482e4320 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b441a2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c1b1ee0 con 0x5631482d9330 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b441a2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c1af140 con 0x5631482d9c50 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b441a2740 10 client.0 _ll_register_callbacks cb 0x7f3b34013710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b3b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482e4320 0x5631482e94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b3b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482e4320 0x5631482e94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57386/0 (socket says 172.21.15.98:57386) 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b3b7fe640 1 -- 172.21.15.98:0/204758291 learned_addr learned my addr 172.21.15.98:0/204758291 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b3a7fc640 1 --2- 172.21.15.98:0/204758291 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 0x5631482eca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b3affd640 1 --2- 172.21.15.98:0/204758291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9330 0x5631482ec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b3b7fe640 1 -- 172.21.15.98:0/204758291 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 msgr2=0x5631482eca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b3b7fe640 1 --2- 172.21.15.98:0/204758291 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 0x5631482eca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b3b7fe640 1 -- 172.21.15.98:0/204758291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9330 msgr2=0x5631482ec510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b3b7fe640 1 --2- 172.21.15.98:0/204758291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9330 0x5631482ec510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b3b7fe640 1 -- 172.21.15.98:0/204758291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631482e9af0 con 0x5631482e4320 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b3affd640 1 --2- 172.21.15.98:0/204758291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9330 0x5631482ec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b3b7fe640 1 --2- 172.21.15.98:0/204758291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482e4320 0x5631482e94e0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400a580 tx=0x7f3b3400ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fddea78cd21d6c4d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b227fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b227fc640 1 -- 172.21.15.98:0/204758291 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b34043070 con 0x5631482e4320 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b227fc640 1 -- 172.21.15.98:0/204758291 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7f3b34003ec0 con 0x5631482e4320 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b441a2740 1 -- 172.21.15.98:0/204758291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b2c0c4ad0 con 0x5631482e4320 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b227fc640 1 -- 172.21.15.98:0/204758291 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3b3404f050 con 0x5631482e4320 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b441a2740 10 client.18030 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.583+0000 7f3b441a2740 1 -- 172.21.15.98:0/204758291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56314835b390 con 0x5631482e4320 2024-05-09T08:59:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.584+0000 7f3b227fc640 1 -- 172.21.15.98:0/204758291 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b2c0c4ad0 con 0x5631482e4320 2024-05-09T08:59:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.584+0000 7f3b227fc640 1 -- 172.21.15.98:0/204758291 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3b3402d9a0 con 0x5631482e4320 2024-05-09T08:59:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.584+0000 7f3b441a2740 10 client.18030 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.584+0000 7f3b441a2740 10 client.18030 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:59:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.584+0000 7f3b441a2740 20 client.18030 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.584+0000 7f3b441a2740 1 --2- 172.21.15.98:0/204758291 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x56314835b970 0x56314835dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.584+0000 7f3b441a2740 4 client.18030 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:59:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.584+0000 7f3b441a2740 1 -- 172.21.15.98:0/204758291 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5631482466c0 con 0x56314835b970 2024-05-09T08:59:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.584+0000 7f3b3affd640 1 --2- 172.21.15.98:0/204758291 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x56314835b970 0x56314835dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.584+0000 7f3b227fc640 10 client.18030 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:59:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.584+0000 7f3b3affd640 1 --2- 172.21.15.98:0/204758291 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x56314835b970 0x56314835dd90 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=40deba74b9cd915 server_cookie=998490dfb676e9c4 in_seq=0 out_seq=0 2024-05-09T08:59:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b227fc640 1 -- 172.21.15.98:0/204758291 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5631482466c0 con 0x56314835b970 2024-05-09T08:59:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b227fc640 10 client.18030 handle_command_reply: tid=0 2024-05-09T08:59:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b441a2740 1 client.18030 shutdown 2024-05-09T08:59:17.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b441a2740 1 -- 172.21.15.98:0/204758291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482e4320 msgr2=0x5631482e94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b441a2740 1 --2- 172.21.15.98:0/204758291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482e4320 0x5631482e94e0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400a580 tx=0x7f3b3400ab60 comp rx=0 tx=0).stop 2024-05-09T08:59:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b441a2740 1 -- 172.21.15.98:0/204758291 shutdown_connections 2024-05-09T08:59:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b441a2740 1 --2- 172.21.15.98:0/204758291 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x56314835b970 0x56314835dd90 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b441a2740 1 --2- 172.21.15.98:0/204758291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631482e4320 0x5631482e94e0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b441a2740 1 --2- 172.21.15.98:0/204758291 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631482d9c50 0x5631482eca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b227fc640 0 client.18030 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:59:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b441a2740 1 --2- 172.21.15.98:0/204758291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631482d9330 0x5631482ec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b441a2740 1 -- 172.21.15.98:0/204758291 >> 172.21.15.98:0/204758291 conn(0x5631482d5650 msgr2=0x5631482e1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b441a2740 1 -- 172.21.15.98:0/204758291 shutdown_connections 2024-05-09T08:59:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b441a2740 1 -- 172.21.15.98:0/204758291 wait complete. 2024-05-09T08:59:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b441a2740 20 client.18030 trim_cache size 0 max 16384 2024-05-09T08:59:17.591 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:59:17.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b4149a640 1 -- 172.21.15.98:0/3699280388 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3b280a2d00 msgr2=0x7f3b280a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b4149a640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3b280a2d00 0x7f3b280a51c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3b34021000 tx=0x7f3b34020000 comp rx=0 tx=0).stop 2024-05-09T08:59:17.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b4149a640 1 -- 172.21.15.98:0/3699280388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c101850 msgr2=0x7f3b3c073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b4149a640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c101850 0x7f3b3c073b30 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009230 tx=0x7f3b2c004270 comp rx=0 tx=0).stop 2024-05-09T08:59:17.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b4149a640 1 -- 172.21.15.98:0/3699280388 shutdown_connections 2024-05-09T08:59:17.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b4149a640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3b280a2d00 0x7f3b280a51c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b4149a640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b3c107860 0x7f3b3c0745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b4149a640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b3c102210 0x7f3b3c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b4149a640 1 --2- 172.21.15.98:0/3699280388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b3c101850 0x7f3b3c073b30 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.585+0000 7f3b4149a640 1 -- 172.21.15.98:0/3699280388 >> 172.21.15.98:0/3699280388 conn(0x7f3b3c0fd340 msgr2=0x7f3b3c0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.586+0000 7f3b4149a640 1 -- 172.21.15.98:0/3699280388 shutdown_connections 2024-05-09T08:59:17.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.586+0000 7f3b4149a640 1 -- 172.21.15.98:0/3699280388 wait complete. 2024-05-09T08:59:17.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:59:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.700+0000 7f2c3422c640 1 -- 172.21.15.98:0/4006775045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c2c100700 msgr2=0x7f2c2c100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.700+0000 7f2c3422c640 1 --2- 172.21.15.98:0/4006775045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c2c100700 0x7f2c2c100b80 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009930 tx=0x7f2c1c02a420 comp rx=0 tx=0).stop 2024-05-09T08:59:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.700+0000 7f2c3422c640 1 -- 172.21.15.98:0/4006775045 shutdown_connections 2024-05-09T08:59:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.700+0000 7f2c3422c640 1 --2- 172.21.15.98:0/4006775045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c2c1010c0 0x7f2c2c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.700+0000 7f2c3422c640 1 --2- 172.21.15.98:0/4006775045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c2c100700 0x7f2c2c100b80 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.700+0000 7f2c3422c640 1 --2- 172.21.15.98:0/4006775045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c2c106710 0x7f2c2c106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.700+0000 7f2c3422c640 1 -- 172.21.15.98:0/4006775045 >> 172.21.15.98:0/4006775045 conn(0x7f2c2c077830 msgr2=0x7f2c2c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.700+0000 7f2c3422c640 1 -- 172.21.15.98:0/4006775045 shutdown_connections 2024-05-09T08:59:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.700+0000 7f2c3422c640 1 -- 172.21.15.98:0/4006775045 wait complete. 2024-05-09T08:59:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c3422c640 1 Processor -- start 2024-05-09T08:59:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c3422c640 1 -- start start 2024-05-09T08:59:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c3422c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c2c100700 0x7f2c2c198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c3422c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c2c1010c0 0x7f2c2c1994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c3422c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c2c106710 0x7f2c2c1a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c3422c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2c110b20 con 0x7f2c2c106710 2024-05-09T08:59:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c3422c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2c1109a0 con 0x7f2c2c100700 2024-05-09T08:59:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c3422c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2c110ca0 con 0x7f2c2c1010c0 2024-05-09T08:59:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c327c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c2c106710 0x7f2c2c1a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c327c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c2c106710 0x7f2c2c1a4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57394/0 (socket says 172.21.15.98:57394) 2024-05-09T08:59:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c327c9640 1 -- 172.21.15.98:0/1659961691 learned_addr learned my addr 172.21.15.98:0/1659961691 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c317c7640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c2c1010c0 0x7f2c2c1994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c31fc8640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c2c100700 0x7f2c2c198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c327c9640 1 -- 172.21.15.98:0/1659961691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c2c1010c0 msgr2=0x7f2c2c1994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c327c9640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c2c1010c0 0x7f2c2c1994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c327c9640 1 -- 172.21.15.98:0/1659961691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c2c100700 msgr2=0x7f2c2c198fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c327c9640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c2c100700 0x7f2c2c198fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c327c9640 1 -- 172.21.15.98:0/1659961691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c2c1a4560 con 0x7f2c2c106710 2024-05-09T08:59:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c317c7640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c2c1010c0 0x7f2c2c1994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.701+0000 7f2c327c9640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c2c106710 0x7f2c2c1a4020 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f2c2400eb10 tx=0x7f2c2400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.702+0000 7f2c22ffd640 1 -- 172.21.15.98:0/1659961691 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c24004b20 con 0x7f2c2c106710 2024-05-09T08:59:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.702+0000 7f2c3422c640 1 -- 172.21.15.98:0/1659961691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c2c1a4730 con 0x7f2c2c106710 2024-05-09T08:59:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.702+0000 7f2c22ffd640 1 -- 172.21.15.98:0/1659961691 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c24004cc0 con 0x7f2c2c106710 2024-05-09T08:59:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.702+0000 7f2c22ffd640 1 -- 172.21.15.98:0/1659961691 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c24010760 con 0x7f2c2c106710 2024-05-09T08:59:17.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.702+0000 7f2c3422c640 1 -- 172.21.15.98:0/1659961691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c2c1a4bf0 con 0x7f2c2c106710 2024-05-09T08:59:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.704+0000 7f2c22ffd640 1 -- 172.21.15.98:0/1659961691 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2c24005020 con 0x7f2c2c106710 2024-05-09T08:59:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.704+0000 7f2c37010740 1 Processor -- start 2024-05-09T08:59:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.704+0000 7f2c37010740 1 -- start start 2024-05-09T08:59:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.704+0000 7f2c22ffd640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2c180a2d00 0x7f2c180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c37010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8ba98440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c37010740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 0x555d8baa71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c37010740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8baa7730 0x555d8baa9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c22ffd640 1 -- 172.21.15.98:0/1659961691 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7f2c240d00b0 con 0x7f2c2c106710 2024-05-09T08:59:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c37010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555d8b9fdc40 con 0x555d8ba98060 2024-05-09T08:59:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c37010740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555d8b9a5580 con 0x555d8ba98a10 2024-05-09T08:59:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c37010740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555d8b9f9c00 con 0x555d8baa7730 2024-05-09T08:59:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c317c7640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2c180a2d00 0x7f2c180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c31fc8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8ba98440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c31fc8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8ba98440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57402/0 (socket says 172.21.15.98:57402) 2024-05-09T08:59:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c31fc8640 1 -- 172.21.15.98:0/1086799115 learned_addr learned my addr 172.21.15.98:0/1086799115 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c327c9640 1 --2- 172.21.15.98:0/1086799115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8baa7730 0x555d8baa9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c317c7640 1 --2- 172.21.15.98:0/1086799115 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 0x555d8baa71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c31fc8640 1 -- 172.21.15.98:0/1086799115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8baa7730 msgr2=0x555d8baa9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c31fc8640 1 --2- 172.21.15.98:0/1086799115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8baa7730 0x555d8baa9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c31fc8640 1 -- 172.21.15.98:0/1086799115 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 msgr2=0x555d8baa71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c31fc8640 1 --2- 172.21.15.98:0/1086799115 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 0x555d8baa71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c31fc8640 1 -- 172.21.15.98:0/1086799115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d8b7113f0 con 0x555d8ba98060 2024-05-09T08:59:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c317c7640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2c180a2d00 0x7f2c180a51c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c000c00 tx=0x7f2c1c032d90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c31fc8640 1 --2- 172.21.15.98:0/1086799115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8ba98440 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009c00 tx=0x7f2c280113e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8cb3da96b89069 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c17fff640 1 -- 172.21.15.98:0/1086799115 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c28011db0 con 0x555d8ba98060 2024-05-09T08:59:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c17fff640 1 -- 172.21.15.98:0/1086799115 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c28012930 con 0x555d8ba98060 2024-05-09T08:59:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.705+0000 7f2c17fff640 1 -- 172.21.15.98:0/1086799115 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2800f980 con 0x555d8ba98060 2024-05-09T08:59:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.706+0000 7f2c37010740 1 -- 172.21.15.98:0/1086799115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 msgr2=0x555d8ba98440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.706+0000 7f2c37010740 1 --2- 172.21.15.98:0/1086799115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8ba98440 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009c00 tx=0x7f2c280113e0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.706+0000 7f2c37010740 1 -- 172.21.15.98:0/1086799115 shutdown_connections 2024-05-09T08:59:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.706+0000 7f2c37010740 1 --2- 172.21.15.98:0/1086799115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8baa7730 0x555d8baa9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.706+0000 7f2c37010740 1 --2- 172.21.15.98:0/1086799115 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 0x555d8baa71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.706+0000 7f2c37010740 1 --2- 172.21.15.98:0/1086799115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8ba98440 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.706+0000 7f2c37010740 1 -- 172.21.15.98:0/1086799115 >> 172.21.15.98:0/1086799115 conn(0x555d8ba94650 msgr2=0x555d8ba94a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.706+0000 7f2c37010740 1 -- 172.21.15.98:0/1086799115 shutdown_connections 2024-05-09T08:59:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.706+0000 7f2c37010740 1 -- 172.21.15.98:0/1086799115 wait complete. 2024-05-09T08:59:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.706+0000 7f2c37010740 1 -- start start 2024-05-09T08:59:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c37010740 1 Processor -- start 2024-05-09T08:59:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c37010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8baa4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c37010740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8ba98a10 0x555d8baa4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c37010740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8baa7730 0x555d8baa50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c37010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555d8baaa4f0 con 0x555d8ba98060 2024-05-09T08:59:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c37010740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555d8b9b8c40 con 0x555d8baa7730 2024-05-09T08:59:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c37010740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555d8baaa670 con 0x555d8ba98a10 2024-05-09T08:59:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c37010740 10 client.0 _ll_register_callbacks cb 0x555d8b710be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c31fc8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8baa4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c31fc8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8baa4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57414/0 (socket says 172.21.15.98:57414) 2024-05-09T08:59:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c31fc8640 1 -- 172.21.15.98:0/2542697968 learned_addr learned my addr 172.21.15.98:0/2542697968 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c317c7640 1 --2- 172.21.15.98:0/2542697968 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8ba98a10 0x555d8baa4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c31fc8640 1 -- 172.21.15.98:0/2542697968 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8ba98a10 msgr2=0x555d8baa4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c31fc8640 1 --2- 172.21.15.98:0/2542697968 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8ba98a10 0x555d8baa4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c31fc8640 1 -- 172.21.15.98:0/2542697968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8baa7730 msgr2=0x555d8baa50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c31fc8640 1 --2- 172.21.15.98:0/2542697968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8baa7730 0x555d8baa50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c31fc8640 1 -- 172.21.15.98:0/2542697968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d8b7db7d0 con 0x555d8ba98060 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c31fc8640 1 --2- 172.21.15.98:0/2542697968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8baa4630 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f2c28012700 tx=0x7f2c28012730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ecab892a33016a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.707+0000 7f2c157fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c157fa640 1 -- 172.21.15.98:0/2542697968 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c28028020 con 0x555d8ba98060 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c157fa640 1 -- 172.21.15.98:0/2542697968 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7f2c28012cf0 con 0x555d8ba98060 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c37010740 1 -- 172.21.15.98:0/2542697968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c28012930 con 0x555d8ba98060 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c157fa640 1 -- 172.21.15.98:0/2542697968 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2c28033050 con 0x555d8ba98060 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c37010740 10 client.18042 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c37010740 1 -- 172.21.15.98:0/2542697968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x555d8bab66a0 con 0x555d8ba98060 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c157fa640 1 -- 172.21.15.98:0/2542697968 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c28012930 con 0x555d8ba98060 2024-05-09T08:59:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c157fa640 1 -- 172.21.15.98:0/2542697968 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f2c2800fba0 con 0x555d8ba98060 2024-05-09T08:59:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c37010740 10 client.18042 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c37010740 10 client.18042 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:59:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c37010740 20 client.18042 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c37010740 1 --2- 172.21.15.98:0/2542697968 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x555d8bab7b00 0x555d8bab9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c37010740 4 client.18042 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c37010740 1 -- 172.21.15.98:0/2542697968 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d8ba0b370 con 0x555d8bab7b00 2024-05-09T08:59:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.708+0000 7f2c317c7640 1 --2- 172.21.15.98:0/2542697968 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x555d8bab7b00 0x555d8bab9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.709+0000 7f2c317c7640 1 --2- 172.21.15.98:0/2542697968 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x555d8bab7b00 0x555d8bab9ef0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=53dd58c4826992ec server_cookie=2d7256812551b634 in_seq=0 out_seq=0 2024-05-09T08:59:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.709+0000 7f2c157fa640 10 client.18042 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:59:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.709+0000 7f2c157fa640 1 -- 172.21.15.98:0/2542697968 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d8ba0b370 con 0x555d8bab7b00 2024-05-09T08:59:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.709+0000 7f2c157fa640 10 client.18042 handle_command_reply: tid=0 2024-05-09T08:59:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.710+0000 7f2c37010740 1 client.18042 shutdown 2024-05-09T08:59:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.710+0000 7f2c37010740 1 -- 172.21.15.98:0/2542697968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 msgr2=0x555d8baa4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.710+0000 7f2c37010740 1 --2- 172.21.15.98:0/2542697968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8baa4630 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f2c28012700 tx=0x7f2c28012730 comp rx=0 tx=0).stop 2024-05-09T08:59:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.710+0000 7f2c37010740 1 -- 172.21.15.98:0/2542697968 shutdown_connections 2024-05-09T08:59:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.710+0000 7f2c37010740 1 --2- 172.21.15.98:0/2542697968 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x555d8bab7b00 0x555d8bab9ef0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.710+0000 7f2c37010740 1 --2- 172.21.15.98:0/2542697968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8baa7730 0x555d8baa50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.710+0000 7f2c37010740 1 --2- 172.21.15.98:0/2542697968 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8ba98a10 0x555d8baa4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.710+0000 7f2c157fa640 0 client.18042 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:59:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.710+0000 7f2c37010740 1 --2- 172.21.15.98:0/2542697968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8baa4630 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.710+0000 7f2c37010740 1 -- 172.21.15.98:0/2542697968 >> 172.21.15.98:0/2542697968 conn(0x555d8ba94650 msgr2=0x555d8ba9a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.710+0000 7f2c37010740 1 -- 172.21.15.98:0/2542697968 shutdown_connections 2024-05-09T08:59:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.710+0000 7f2c37010740 1 -- 172.21.15.98:0/2542697968 wait complete. 2024-05-09T08:59:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.710+0000 7f2c37010740 20 client.18042 trim_cache size 0 max 16384 2024-05-09T08:59:17.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c37010740 1 Processor -- start 2024-05-09T08:59:17.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c37010740 1 -- start start 2024-05-09T08:59:17.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c37010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8bab33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c37010740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 0x555d8baa3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c37010740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8baa7730 0x555d8babf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c37010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2c1ace90 con 0x555d8ba98060 2024-05-09T08:59:17.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c37010740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2c1ae9d0 con 0x555d8ba98a10 2024-05-09T08:59:17.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c37010740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2c1ad560 con 0x555d8baa7730 2024-05-09T08:59:17.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c31fc8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8bab33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c31fc8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8bab33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57418/0 (socket says 172.21.15.98:57418) 2024-05-09T08:59:17.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c31fc8640 1 -- 172.21.15.98:0/570395563 learned_addr learned my addr 172.21.15.98:0/570395563 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c317c7640 1 --2- 172.21.15.98:0/570395563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 0x555d8baa3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c31fc8640 1 -- 172.21.15.98:0/570395563 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8baa7730 msgr2=0x555d8babf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c31fc8640 1 --2- 172.21.15.98:0/570395563 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8baa7730 0x555d8babf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c31fc8640 1 -- 172.21.15.98:0/570395563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 msgr2=0x555d8baa3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c31fc8640 1 --2- 172.21.15.98:0/570395563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 0x555d8baa3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c31fc8640 1 -- 172.21.15.98:0/570395563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d8ba85d20 con 0x555d8ba98060 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.723+0000 7f2c317c7640 1 --2- 172.21.15.98:0/570395563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 0x555d8baa3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.724+0000 7f2c31fc8640 1 --2- 172.21.15.98:0/570395563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8bab33f0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f2c28012820 tx=0x7f2c28012850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab9657278823dd8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.724+0000 7f2c177fe640 1 -- 172.21.15.98:0/570395563 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2803b020 con 0x555d8ba98060 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.724+0000 7f2c177fe640 1 -- 172.21.15.98:0/570395563 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c28009630 con 0x555d8ba98060 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.724+0000 7f2c177fe640 1 -- 172.21.15.98:0/570395563 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c28028040 con 0x555d8ba98060 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.724+0000 7f2c37010740 1 -- 172.21.15.98:0/570395563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 msgr2=0x555d8bab33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.724+0000 7f2c37010740 1 --2- 172.21.15.98:0/570395563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8bab33f0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f2c28012820 tx=0x7f2c28012850 comp rx=0 tx=0).stop 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.724+0000 7f2c37010740 1 -- 172.21.15.98:0/570395563 shutdown_connections 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.724+0000 7f2c37010740 1 --2- 172.21.15.98:0/570395563 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8baa7730 0x555d8babf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.724+0000 7f2c37010740 1 --2- 172.21.15.98:0/570395563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 0x555d8baa3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.724+0000 7f2c37010740 1 --2- 172.21.15.98:0/570395563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8bab33f0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.724+0000 7f2c37010740 1 -- 172.21.15.98:0/570395563 >> 172.21.15.98:0/570395563 conn(0x555d8ba94650 msgr2=0x555d8ba937b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.724+0000 7f2c37010740 1 -- 172.21.15.98:0/570395563 shutdown_connections 2024-05-09T08:59:17.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.724+0000 7f2c37010740 1 -- 172.21.15.98:0/570395563 wait complete. 2024-05-09T08:59:17.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c37010740 1 -- start start 2024-05-09T08:59:17.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c37010740 1 Processor -- start 2024-05-09T08:59:17.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c37010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8baafca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c37010740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 0x555d8baab010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c37010740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8baa7730 0x555d8baab550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c37010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2c1ae300 con 0x555d8ba98060 2024-05-09T08:59:17.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c37010740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2c1adc30 con 0x555d8ba98a10 2024-05-09T08:59:17.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c37010740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2c1aae90 con 0x555d8baa7730 2024-05-09T08:59:17.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c37010740 10 client.0 _ll_register_callbacks cb 0x7f2c28009a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:17.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c31fc8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8baafca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c31fc8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8baafca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57422/0 (socket says 172.21.15.98:57422) 2024-05-09T08:59:17.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c31fc8640 1 -- 172.21.15.98:0/571519382 learned_addr learned my addr 172.21.15.98:0/571519382 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c317c7640 1 --2- 172.21.15.98:0/571519382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 0x555d8baab010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c31fc8640 1 -- 172.21.15.98:0/571519382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8baa7730 msgr2=0x555d8baab550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c31fc8640 1 --2- 172.21.15.98:0/571519382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8baa7730 0x555d8baab550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c31fc8640 1 -- 172.21.15.98:0/571519382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 msgr2=0x555d8baab010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.725+0000 7f2c31fc8640 1 --2- 172.21.15.98:0/571519382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 0x555d8baab010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c31fc8640 1 -- 172.21.15.98:0/571519382 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d8baabd80 con 0x555d8ba98060 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c31fc8640 1 --2- 172.21.15.98:0/571519382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8baafca0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f2c280036e0 tx=0x7f2c28009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57941ccf1b8fa144 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c157fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c157fa640 1 -- 172.21.15.98:0/571519382 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c28045020 con 0x555d8ba98060 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c157fa640 1 -- 172.21.15.98:0/571519382 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7f2c28003960 con 0x555d8ba98060 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c37010740 1 -- 172.21.15.98:0/571519382 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c28009630 con 0x555d8ba98060 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c157fa640 1 -- 172.21.15.98:0/571519382 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2c2804d050 con 0x555d8ba98060 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c37010740 10 client.18048 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c37010740 1 -- 172.21.15.98:0/571519382 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x555d8bb1dd50 con 0x555d8ba98060 2024-05-09T08:59:17.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c157fa640 1 -- 172.21.15.98:0/571519382 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c28009630 con 0x555d8ba98060 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c157fa640 1 -- 172.21.15.98:0/571519382 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f2c28024450 con 0x555d8ba98060 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c37010740 10 client.18048 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c37010740 10 client.18048 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c37010740 20 client.18048 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c37010740 1 --2- 172.21.15.98:0/571519382 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x555d8bb1e300 0x555d8bb20730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c37010740 4 client.18048 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.726+0000 7f2c37010740 1 -- 172.21.15.98:0/571519382 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555d8ba056c0 con 0x555d8bb1e300 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c317c7640 1 --2- 172.21.15.98:0/571519382 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x555d8bb1e300 0x555d8bb20730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c157fa640 10 client.18048 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c317c7640 1 --2- 172.21.15.98:0/571519382 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x555d8bb1e300 0x555d8bb20730 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cf3c579ec84b1606 server_cookie=d14fa3e341e63174 in_seq=0 out_seq=0 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c157fa640 1 -- 172.21.15.98:0/571519382 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555d8ba056c0 con 0x555d8bb1e300 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c157fa640 10 client.18048 handle_command_reply: tid=0 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c37010740 1 client.18048 shutdown 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c37010740 1 -- 172.21.15.98:0/571519382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 msgr2=0x555d8baafca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c37010740 1 --2- 172.21.15.98:0/571519382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8baafca0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f2c280036e0 tx=0x7f2c28009260 comp rx=0 tx=0).stop 2024-05-09T08:59:17.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c37010740 1 -- 172.21.15.98:0/571519382 shutdown_connections 2024-05-09T08:59:17.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c37010740 1 --2- 172.21.15.98:0/571519382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555d8baa7730 0x555d8baab550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c37010740 1 --2- 172.21.15.98:0/571519382 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x555d8bb1e300 0x555d8bb20730 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c37010740 1 --2- 172.21.15.98:0/571519382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555d8ba98a10 0x555d8baab010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c157fa640 0 client.18048 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:59:17.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c37010740 1 --2- 172.21.15.98:0/571519382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d8ba98060 0x555d8baafca0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c37010740 1 -- 172.21.15.98:0/571519382 >> 172.21.15.98:0/571519382 conn(0x555d8ba94650 msgr2=0x555d8baa5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c37010740 1 -- 172.21.15.98:0/571519382 shutdown_connections 2024-05-09T08:59:17.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.727+0000 7f2c37010740 1 -- 172.21.15.98:0/571519382 wait complete. 2024-05-09T08:59:17.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.728+0000 7f2c37010740 20 client.18048 trim_cache size 0 max 16384 2024-05-09T08:59:17.733 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:59:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.728+0000 7f2c3422c640 1 -- 172.21.15.98:0/1659961691 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2c180a2d00 msgr2=0x7f2c180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.728+0000 7f2c3422c640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2c180a2d00 0x7f2c180a51c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c000c00 tx=0x7f2c1c032d90 comp rx=0 tx=0).stop 2024-05-09T08:59:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.728+0000 7f2c3422c640 1 -- 172.21.15.98:0/1659961691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c2c106710 msgr2=0x7f2c2c1a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.728+0000 7f2c3422c640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c2c106710 0x7f2c2c1a4020 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f2c2400eb10 tx=0x7f2c2400c2d0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.728+0000 7f2c3422c640 1 -- 172.21.15.98:0/1659961691 shutdown_connections 2024-05-09T08:59:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.728+0000 7f2c3422c640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2c180a2d00 0x7f2c180a51c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.728+0000 7f2c3422c640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c2c106710 0x7f2c2c1a4020 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.728+0000 7f2c3422c640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c2c1010c0 0x7f2c2c1994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.728+0000 7f2c3422c640 1 --2- 172.21.15.98:0/1659961691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c2c100700 0x7f2c2c198fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.728+0000 7f2c3422c640 1 -- 172.21.15.98:0/1659961691 >> 172.21.15.98:0/1659961691 conn(0x7f2c2c077830 msgr2=0x7f2c2c102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.728+0000 7f2c3422c640 1 -- 172.21.15.98:0/1659961691 shutdown_connections 2024-05-09T08:59:17.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.728+0000 7f2c3422c640 1 -- 172.21.15.98:0/1659961691 wait complete. 2024-05-09T08:59:17.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:59:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.841+0000 7fb1ada3f640 1 -- 172.21.15.98:0/956184210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a8106900 msgr2=0x7fb1a8106ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.841+0000 7fb1ada3f640 1 --2- 172.21.15.98:0/956184210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a8106900 0x7fb1a8106ce0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb198009cd0 tx=0x7fb19802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.841+0000 7fb1ada3f640 1 -- 172.21.15.98:0/956184210 shutdown_connections 2024-05-09T08:59:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.841+0000 7fb1ada3f640 1 --2- 172.21.15.98:0/956184210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1a81013d0 0x7fb1a810d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.842+0000 7fb1ada3f640 1 --2- 172.21.15.98:0/956184210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a8100a10 0x7fb1a8100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.842+0000 7fb1ada3f640 1 --2- 172.21.15.98:0/956184210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a8106900 0x7fb1a8106ce0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.842+0000 7fb1ada3f640 1 -- 172.21.15.98:0/956184210 >> 172.21.15.98:0/956184210 conn(0x7fb1a8077710 msgr2=0x7fb1a8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.842+0000 7fb1ada3f640 1 -- 172.21.15.98:0/956184210 shutdown_connections 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.842+0000 7fb1ada3f640 1 -- 172.21.15.98:0/956184210 wait complete. 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.842+0000 7fb1ada3f640 1 Processor -- start 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.842+0000 7fb1ada3f640 1 -- start start 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.842+0000 7fb1ada3f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1a8100a10 0x7fb1a8198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.842+0000 7fb1ada3f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a81013d0 0x7fb1a8199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.842+0000 7fb1ada3f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a8106900 0x7fb1a81a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.842+0000 7fb1ada3f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a8110bd0 con 0x7fb1a81013d0 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.842+0000 7fb1ada3f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a8110a50 con 0x7fb1a8106900 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.842+0000 7fb1ada3f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a8110d50 con 0x7fb1a8100a10 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1a67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a81013d0 0x7fb1a8199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1a67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a81013d0 0x7fb1a8199530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57432/0 (socket says 172.21.15.98:57432) 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1a67fc640 1 -- 172.21.15.98:0/3309747711 learned_addr learned my addr 172.21.15.98:0/3309747711 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1a77fe640 1 --2- 172.21.15.98:0/3309747711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a8106900 0x7fb1a81a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1a6ffd640 1 --2- 172.21.15.98:0/3309747711 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1a8100a10 0x7fb1a8198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1a67fc640 1 -- 172.21.15.98:0/3309747711 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1a8100a10 msgr2=0x7fb1a8198ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1a67fc640 1 --2- 172.21.15.98:0/3309747711 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1a8100a10 0x7fb1a8198ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1a67fc640 1 -- 172.21.15.98:0/3309747711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a8106900 msgr2=0x7fb1a81a4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1a67fc640 1 --2- 172.21.15.98:0/3309747711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a8106900 0x7fb1a81a4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1a67fc640 1 -- 172.21.15.98:0/3309747711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a81a45b0 con 0x7fb1a81013d0 2024-05-09T08:59:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1a67fc640 1 --2- 172.21.15.98:0/3309747711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a81013d0 0x7fb1a8199530 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00d950 tx=0x7fb19c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1aca3d640 1 -- 172.21.15.98:0/3309747711 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c017030 con 0x7fb1a81013d0 2024-05-09T08:59:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1aca3d640 1 -- 172.21.15.98:0/3309747711 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb19c004020 con 0x7fb1a81013d0 2024-05-09T08:59:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1ada3f640 1 -- 172.21.15.98:0/3309747711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a81a4780 con 0x7fb1a81013d0 2024-05-09T08:59:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1aca3d640 1 -- 172.21.15.98:0/3309747711 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c0047a0 con 0x7fb1a81013d0 2024-05-09T08:59:17.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.843+0000 7fb1ada3f640 1 -- 172.21.15.98:0/3309747711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a81a4ca0 con 0x7fb1a81013d0 2024-05-09T08:59:17.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.845+0000 7fb1aca3d640 1 -- 172.21.15.98:0/3309747711 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb19c0041d0 con 0x7fb1a81013d0 2024-05-09T08:59:17.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1b082b740 1 Processor -- start 2024-05-09T08:59:17.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1b082b740 1 -- start start 2024-05-09T08:59:17.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1b082b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2dd330 0x5626ee2dd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1b082b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2ddc50 0x5626ee2e7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1b082b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2e8320 0x5626ee2ea710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1b082b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5626ee242c40 con 0x5626ee2dd330 2024-05-09T08:59:17.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1b082b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5626ee1ea580 con 0x5626ee2e8320 2024-05-09T08:59:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1b082b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5626ee23ec00 con 0x5626ee2ddc50 2024-05-09T08:59:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1aca3d640 1 --2- 172.21.15.98:0/3309747711 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb1900a2d00 0x7fb1900a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2dd330 0x5626ee2dd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2dd330 0x5626ee2dd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57438/0 (socket says 172.21.15.98:57438) 2024-05-09T08:59:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1a77fe640 1 -- 172.21.15.98:0/3197602419 learned_addr learned my addr 172.21.15.98:0/3197602419 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1a67fc640 1 --2- 172.21.15.98:0/3197602419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2e8320 0x5626ee2ea710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1a77fe640 1 --2- 172.21.15.98:0/3309747711 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb1900a2d00 0x7fb1900a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1a6ffd640 1 --2- 172.21.15.98:0/3197602419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2ddc50 0x5626ee2e7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1aca3d640 1 -- 172.21.15.98:0/3309747711 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7fb19c0cf280 con 0x7fb1a81013d0 2024-05-09T08:59:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1a77fe640 1 -- 172.21.15.98:0/3197602419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2ddc50 msgr2=0x5626ee2e7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1a77fe640 1 --2- 172.21.15.98:0/3197602419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2ddc50 0x5626ee2e7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1a77fe640 1 -- 172.21.15.98:0/3197602419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2e8320 msgr2=0x5626ee2ea710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1a77fe640 1 --2- 172.21.15.98:0/3197602419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2e8320 0x5626ee2ea710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.846+0000 7fb1a77fe640 1 -- 172.21.15.98:0/3197602419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626edf563f0 con 0x5626ee2dd330 2024-05-09T08:59:17.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.847+0000 7fb1a77fe640 1 --2- 172.21.15.98:0/3197602419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2dd330 0x5626ee2dd710 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00129f0 tx=0x7fb1a0012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3fb6d4e8f9f7996 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.847+0000 7fb1a77fe640 1 --2- 172.21.15.98:0/3309747711 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb1900a2d00 0x7fb1900a51c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0010740 tx=0x7fb1a0017000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.847+0000 7fb18dffb640 1 -- 172.21.15.98:0/3197602419 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0024070 con 0x5626ee2dd330 2024-05-09T08:59:17.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.847+0000 7fb18dffb640 1 -- 172.21.15.98:0/3197602419 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1a000b9c0 con 0x5626ee2dd330 2024-05-09T08:59:17.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.847+0000 7fb18dffb640 1 -- 172.21.15.98:0/3197602419 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0004d70 con 0x5626ee2dd330 2024-05-09T08:59:17.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.847+0000 7fb1b082b740 1 -- 172.21.15.98:0/3197602419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2dd330 msgr2=0x5626ee2dd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.847+0000 7fb1b082b740 1 --2- 172.21.15.98:0/3197602419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2dd330 0x5626ee2dd710 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00129f0 tx=0x7fb1a0012eb0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 -- 172.21.15.98:0/3197602419 shutdown_connections 2024-05-09T08:59:17.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 --2- 172.21.15.98:0/3197602419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2e8320 0x5626ee2ea710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 --2- 172.21.15.98:0/3197602419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2ddc50 0x5626ee2e7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 --2- 172.21.15.98:0/3197602419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2dd330 0x5626ee2dd710 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 -- 172.21.15.98:0/3197602419 >> 172.21.15.98:0/3197602419 conn(0x5626ee2d9650 msgr2=0x5626ee2d9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 -- 172.21.15.98:0/3197602419 shutdown_connections 2024-05-09T08:59:17.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 -- 172.21.15.98:0/3197602419 wait complete. 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 -- start start 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 Processor -- start 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2dd330 0x5626ee2e5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2ddc50 0x5626ee2e5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2e8320 0x5626ee2e5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5626ee2eb0b0 con 0x5626ee2dd330 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5626ee1fdc40 con 0x5626ee2ddc50 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5626ee2eb230 con 0x5626ee2e8320 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1b082b740 10 client.0 _ll_register_callbacks cb 0x5626edf55be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.848+0000 7fb1a6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2dd330 0x5626ee2e5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb1a6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2dd330 0x5626ee2e5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57440/0 (socket says 172.21.15.98:57440) 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/3802925927 learned_addr learned my addr 172.21.15.98:0/3802925927 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb1a67fc640 1 --2- 172.21.15.98:0/3802925927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2ddc50 0x5626ee2e5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/3802925927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2e8320 msgr2=0x5626ee2e5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb1a77fe640 1 --2- 172.21.15.98:0/3802925927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2e8320 0x5626ee2e5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb1a6ffd640 1 --2- 172.21.15.98:0/3802925927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2e8320 0x5626ee2e5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/3802925927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2ddc50 msgr2=0x5626ee2e5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb1a6ffd640 1 --2- 172.21.15.98:0/3802925927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2ddc50 0x5626ee2e5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb1a6ffd640 1 -- 172.21.15.98:0/3802925927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626ee0207d0 con 0x5626ee2dd330 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb1a67fc640 1 --2- 172.21.15.98:0/3802925927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2ddc50 0x5626ee2e5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb1a6ffd640 1 --2- 172.21.15.98:0/3802925927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2dd330 0x5626ee2e5350 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fb198009cd0 tx=0x7fb198031c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3519ea261c4fdac0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb186ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb186ffd640 1 -- 172.21.15.98:0/3802925927 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb198003970 con 0x5626ee2dd330 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb186ffd640 1 -- 172.21.15.98:0/3802925927 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7fb198003b10 con 0x5626ee2dd330 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb1b082b740 1 -- 172.21.15.98:0/3802925927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1a000b9c0 con 0x5626ee2dd330 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb186ffd640 1 -- 172.21.15.98:0/3802925927 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb198003d10 con 0x5626ee2dd330 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb1b082b740 10 client.18060 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.849+0000 7fb1b082b740 1 -- 172.21.15.98:0/3802925927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5626ee2f7350 con 0x5626ee2dd330 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.850+0000 7fb186ffd640 1 -- 172.21.15.98:0/3802925927 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1a000b9c0 con 0x5626ee2dd330 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.850+0000 7fb186ffd640 1 -- 172.21.15.98:0/3802925927 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb198030070 con 0x5626ee2dd330 2024-05-09T08:59:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.850+0000 7fb1b082b740 10 client.18060 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.850+0000 7fb1b082b740 10 client.18060 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.850+0000 7fb1b082b740 20 client.18060 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.850+0000 7fb1b082b740 1 --2- 172.21.15.98:0/3802925927 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5626ee2f8860 0x5626ee2facf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.850+0000 7fb1b082b740 4 client.18060 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.850+0000 7fb1b082b740 1 -- 172.21.15.98:0/3802925927 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626ee250370 con 0x5626ee2f8860 2024-05-09T08:59:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.850+0000 7fb1a67fc640 1 --2- 172.21.15.98:0/3802925927 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5626ee2f8860 0x5626ee2facf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.850+0000 7fb186ffd640 10 client.18060 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.850+0000 7fb1a67fc640 1 --2- 172.21.15.98:0/3802925927 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5626ee2f8860 0x5626ee2facf0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3beb80eed545b6f server_cookie=e63b432ae083e814 in_seq=0 out_seq=0 2024-05-09T08:59:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.851+0000 7fb186ffd640 1 -- 172.21.15.98:0/3802925927 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5626ee250370 con 0x5626ee2f8860 2024-05-09T08:59:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.851+0000 7fb186ffd640 10 client.18060 handle_command_reply: tid=0 2024-05-09T08:59:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.851+0000 7fb1b082b740 1 client.18060 shutdown 2024-05-09T08:59:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.851+0000 7fb1b082b740 1 -- 172.21.15.98:0/3802925927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2dd330 msgr2=0x5626ee2e5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.851+0000 7fb1b082b740 1 --2- 172.21.15.98:0/3802925927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2dd330 0x5626ee2e5350 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fb198009cd0 tx=0x7fb198031c30 comp rx=0 tx=0).stop 2024-05-09T08:59:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.851+0000 7fb1b082b740 1 -- 172.21.15.98:0/3802925927 shutdown_connections 2024-05-09T08:59:17.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.851+0000 7fb1b082b740 1 --2- 172.21.15.98:0/3802925927 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5626ee2f8860 0x5626ee2facf0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.851+0000 7fb1b082b740 1 --2- 172.21.15.98:0/3802925927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2e8320 0x5626ee2e5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.851+0000 7fb1b082b740 1 --2- 172.21.15.98:0/3802925927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2ddc50 0x5626ee2e5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.851+0000 7fb186ffd640 0 client.18060 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:17.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.851+0000 7fb1b082b740 1 --2- 172.21.15.98:0/3802925927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2dd330 0x5626ee2e5350 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.851+0000 7fb1b082b740 1 -- 172.21.15.98:0/3802925927 >> 172.21.15.98:0/3802925927 conn(0x5626ee2d9650 msgr2=0x5626ee2df450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.851+0000 7fb1b082b740 1 -- 172.21.15.98:0/3802925927 shutdown_connections 2024-05-09T08:59:17.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.851+0000 7fb1b082b740 1 -- 172.21.15.98:0/3802925927 wait complete. 2024-05-09T08:59:17.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.852+0000 7fb1b082b740 20 client.18060 trim_cache size 0 max 16384 2024-05-09T08:59:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.864+0000 7fb1b082b740 1 Processor -- start 2024-05-09T08:59:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.864+0000 7fb1b082b740 1 -- start start 2024-05-09T08:59:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1b082b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2dd330 0x5626ee2f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1b082b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2ddc50 0x5626ee3005c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1b082b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2e8320 0x5626ee300b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1b082b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a81af090 con 0x5626ee2e8320 2024-05-09T08:59:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1b082b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a81b0bd0 con 0x5626ee2dd330 2024-05-09T08:59:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1b082b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a81af760 con 0x5626ee2ddc50 2024-05-09T08:59:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2e8320 0x5626ee300b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2e8320 0x5626ee300b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57444/0 (socket says 172.21.15.98:57444) 2024-05-09T08:59:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1a77fe640 1 -- 172.21.15.98:0/2861004687 learned_addr learned my addr 172.21.15.98:0/2861004687 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1a77fe640 1 -- 172.21.15.98:0/2861004687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2ddc50 msgr2=0x5626ee3005c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1a77fe640 1 --2- 172.21.15.98:0/2861004687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2ddc50 0x5626ee3005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1a77fe640 1 -- 172.21.15.98:0/2861004687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2dd330 msgr2=0x5626ee2f3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1a77fe640 1 --2- 172.21.15.98:0/2861004687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2dd330 0x5626ee2f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1a77fe640 1 -- 172.21.15.98:0/2861004687 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626ee2e4320 con 0x5626ee2e8320 2024-05-09T08:59:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb1a77fe640 1 --2- 172.21.15.98:0/2861004687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2e8320 0x5626ee300b00 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fb1a000b680 tx=0x7fb1a000b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3d6f9ccde153b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb18d7fa640 1 -- 172.21.15.98:0/2861004687 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0024070 con 0x5626ee2e8320 2024-05-09T08:59:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb18d7fa640 1 -- 172.21.15.98:0/2861004687 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1a0012510 con 0x5626ee2e8320 2024-05-09T08:59:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.865+0000 7fb18d7fa640 1 -- 172.21.15.98:0/2861004687 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a00055a0 con 0x5626ee2e8320 2024-05-09T08:59:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.866+0000 7fb1b082b740 1 -- 172.21.15.98:0/2861004687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2e8320 msgr2=0x5626ee300b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.866+0000 7fb1b082b740 1 --2- 172.21.15.98:0/2861004687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2e8320 0x5626ee300b00 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fb1a000b680 tx=0x7fb1a000b6b0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.866+0000 7fb1b082b740 1 -- 172.21.15.98:0/2861004687 shutdown_connections 2024-05-09T08:59:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.866+0000 7fb1b082b740 1 --2- 172.21.15.98:0/2861004687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2e8320 0x5626ee300b00 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.866+0000 7fb1b082b740 1 --2- 172.21.15.98:0/2861004687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2ddc50 0x5626ee3005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.866+0000 7fb1b082b740 1 --2- 172.21.15.98:0/2861004687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2dd330 0x5626ee2f3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.866+0000 7fb1b082b740 1 -- 172.21.15.98:0/2861004687 >> 172.21.15.98:0/2861004687 conn(0x5626ee2d9650 msgr2=0x5626ee2d87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.866+0000 7fb1b082b740 1 -- 172.21.15.98:0/2861004687 shutdown_connections 2024-05-09T08:59:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.866+0000 7fb1b082b740 1 -- 172.21.15.98:0/2861004687 wait complete. 2024-05-09T08:59:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.866+0000 7fb1b082b740 1 -- start start 2024-05-09T08:59:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.866+0000 7fb1b082b740 1 Processor -- start 2024-05-09T08:59:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1b082b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2dd330 0x5626ee2ed450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1b082b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2ddc50 0x5626ee2ed990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1b082b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2e8320 0x5626ee2edef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1b082b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a81b0500 con 0x5626ee2e8320 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1b082b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a81afe30 con 0x5626ee2dd330 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1b082b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a81ad090 con 0x5626ee2ddc50 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1b082b740 10 client.0 _ll_register_callbacks cb 0x7fb198009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2e8320 0x5626ee2edef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2e8320 0x5626ee2edef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57450/0 (socket says 172.21.15.98:57450) 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1a77fe640 1 -- 172.21.15.98:0/4270454885 learned_addr learned my addr 172.21.15.98:0/4270454885 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1a6ffd640 1 --2- 172.21.15.98:0/4270454885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2dd330 0x5626ee2ed450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1a67fc640 1 --2- 172.21.15.98:0/4270454885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2ddc50 0x5626ee2ed990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1a77fe640 1 -- 172.21.15.98:0/4270454885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2ddc50 msgr2=0x5626ee2ed990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1a77fe640 1 --2- 172.21.15.98:0/4270454885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2ddc50 0x5626ee2ed990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1a77fe640 1 -- 172.21.15.98:0/4270454885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2dd330 msgr2=0x5626ee2ed450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1a77fe640 1 --2- 172.21.15.98:0/4270454885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2dd330 0x5626ee2ed450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1a77fe640 1 -- 172.21.15.98:0/4270454885 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626ee35f110 con 0x5626ee2e8320 2024-05-09T08:59:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1a77fe640 1 --2- 172.21.15.98:0/4270454885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2e8320 0x5626ee2edef0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fb1a000b3c0 tx=0x7fb1a000b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98ed5f502c66272e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb186ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb186ffd640 1 -- 172.21.15.98:0/4270454885 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0034070 con 0x5626ee2e8320 2024-05-09T08:59:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb186ffd640 1 -- 172.21.15.98:0/4270454885 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9650+0+0 (secure 0 0 0) 0x7fb1a0004040 con 0x5626ee2e8320 2024-05-09T08:59:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1b082b740 1 -- 172.21.15.98:0/4270454885 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1a0012510 con 0x5626ee2e8320 2024-05-09T08:59:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb186ffd640 1 -- 172.21.15.98:0/4270454885 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb1a0024070 con 0x5626ee2e8320 2024-05-09T08:59:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1b082b740 10 client.18066 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb1b082b740 1 -- 172.21.15.98:0/4270454885 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5626ee360410 con 0x5626ee2e8320 2024-05-09T08:59:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.867+0000 7fb186ffd640 1 -- 172.21.15.98:0/4270454885 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1a0012510 con 0x5626ee2e8320 2024-05-09T08:59:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.868+0000 7fb186ffd640 1 -- 172.21.15.98:0/4270454885 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb1a0003660 con 0x5626ee2e8320 2024-05-09T08:59:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.868+0000 7fb1b082b740 10 client.18066 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.868+0000 7fb1b082b740 10 client.18066 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.868+0000 7fb1b082b740 20 client.18066 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.868+0000 7fb1b082b740 1 --2- 172.21.15.98:0/4270454885 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5626ee3608e0 0x5626ee362cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.868+0000 7fb1b082b740 4 client.18066 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.868+0000 7fb1b082b740 1 -- 172.21.15.98:0/4270454885 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5626ee24a6c0 con 0x5626ee3608e0 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.868+0000 7fb1a6ffd640 1 --2- 172.21.15.98:0/4270454885 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5626ee3608e0 0x5626ee362cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.868+0000 7fb186ffd640 10 client.18066 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.868+0000 7fb1a6ffd640 1 --2- 172.21.15.98:0/4270454885 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5626ee3608e0 0x5626ee362cd0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a2ea7a9f148d1a6 server_cookie=5df856d1be37f5b1 in_seq=0 out_seq=0 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.868+0000 7fb186ffd640 1 -- 172.21.15.98:0/4270454885 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5626ee24a6c0 con 0x5626ee3608e0 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.868+0000 7fb186ffd640 10 client.18066 handle_command_reply: tid=0 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.868+0000 7fb1b082b740 1 client.18066 shutdown 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1b082b740 1 -- 172.21.15.98:0/4270454885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2e8320 msgr2=0x5626ee2edef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1b082b740 1 --2- 172.21.15.98:0/4270454885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2e8320 0x5626ee2edef0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fb1a000b3c0 tx=0x7fb1a000b9c0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1b082b740 1 -- 172.21.15.98:0/4270454885 shutdown_connections 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1b082b740 1 --2- 172.21.15.98:0/4270454885 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5626ee3608e0 0x5626ee362cd0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1b082b740 1 --2- 172.21.15.98:0/4270454885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5626ee2e8320 0x5626ee2edef0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1b082b740 1 --2- 172.21.15.98:0/4270454885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626ee2ddc50 0x5626ee2ed990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb186ffd640 0 client.18066 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:17.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1b082b740 1 --2- 172.21.15.98:0/4270454885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5626ee2dd330 0x5626ee2ed450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1b082b740 1 -- 172.21.15.98:0/4270454885 >> 172.21.15.98:0/4270454885 conn(0x5626ee2d9650 msgr2=0x5626ee2dea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1b082b740 1 -- 172.21.15.98:0/4270454885 shutdown_connections 2024-05-09T08:59:17.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1b082b740 1 -- 172.21.15.98:0/4270454885 wait complete. 2024-05-09T08:59:17.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1b082b740 20 client.18066 trim_cache size 0 max 16384 2024-05-09T08:59:17.875 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:59:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1ada3f640 1 -- 172.21.15.98:0/3309747711 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb1900a2d00 msgr2=0x7fb1900a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1ada3f640 1 --2- 172.21.15.98:0/3309747711 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb1900a2d00 0x7fb1900a51c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0010740 tx=0x7fb1a0017000 comp rx=0 tx=0).stop 2024-05-09T08:59:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1ada3f640 1 -- 172.21.15.98:0/3309747711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a81013d0 msgr2=0x7fb1a8199530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1ada3f640 1 --2- 172.21.15.98:0/3309747711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a81013d0 0x7fb1a8199530 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00d950 tx=0x7fb19c00de10 comp rx=0 tx=0).stop 2024-05-09T08:59:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1ada3f640 1 -- 172.21.15.98:0/3309747711 shutdown_connections 2024-05-09T08:59:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1ada3f640 1 --2- 172.21.15.98:0/3309747711 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb1900a2d00 0x7fb1900a51c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1ada3f640 1 --2- 172.21.15.98:0/3309747711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a8106900 0x7fb1a81a4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1ada3f640 1 --2- 172.21.15.98:0/3309747711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a81013d0 0x7fb1a8199530 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1ada3f640 1 --2- 172.21.15.98:0/3309747711 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1a8100a10 0x7fb1a8198ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1ada3f640 1 -- 172.21.15.98:0/3309747711 >> 172.21.15.98:0/3309747711 conn(0x7fb1a8077710 msgr2=0x7fb1a80ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.869+0000 7fb1ada3f640 1 -- 172.21.15.98:0/3309747711 shutdown_connections 2024-05-09T08:59:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:17.870+0000 7fb1ada3f640 1 -- 172.21.15.98:0/3309747711 wait complete. 2024-05-09T08:59:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:18 smithi098 ceph-mon[30092]: pgmap v269: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 284 B/s wr, 10 op/s 2024-05-09T08:59:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:18 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3388101927' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:59:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:18 smithi098 ceph-mon[30092]: osdmap e103: 12 total, 12 up, 12 in 2024-05-09T08:59:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:18 smithi142 ceph-mon[34625]: pgmap v269: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 284 B/s wr, 10 op/s 2024-05-09T08:59:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:18 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3388101927' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:59:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:18 smithi142 ceph-mon[34625]: osdmap e103: 12 total, 12 up, 12 in 2024-05-09T08:59:18.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:18 smithi105 ceph-mon[34498]: pgmap v269: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 284 B/s wr, 10 op/s 2024-05-09T08:59:18.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:18 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3388101927' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:59:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:18 smithi105 ceph-mon[34498]: osdmap e103: 12 total, 12 up, 12 in 2024-05-09T08:59:18.880 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:59:18.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:59:18.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:59:18.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.998+0000 7f080099e640 1 -- 172.21.15.98:0/2028135566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07fc1010c0 msgr2=0x7f07fc10d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:18.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.998+0000 7f080099e640 1 --2- 172.21.15.98:0/2028135566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07fc1010c0 0x7f07fc10d5d0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009cd0 tx=0x7f07f402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:59:18.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.998+0000 7f080099e640 1 -- 172.21.15.98:0/2028135566 shutdown_connections 2024-05-09T08:59:18.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.998+0000 7f080099e640 1 --2- 172.21.15.98:0/2028135566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07fc1010c0 0x7f07fc10d5d0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:18.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.998+0000 7f080099e640 1 --2- 172.21.15.98:0/2028135566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07fc100700 0x7f07fc100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:18.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.998+0000 7f080099e640 1 --2- 172.21.15.98:0/2028135566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07fc106710 0x7f07fc106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:18.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.998+0000 7f080099e640 1 -- 172.21.15.98:0/2028135566 >> 172.21.15.98:0/2028135566 conn(0x7f07fc077830 msgr2=0x7f07fc077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:18.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.998+0000 7f080099e640 1 -- 172.21.15.98:0/2028135566 shutdown_connections 2024-05-09T08:59:18.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.998+0000 7f080099e640 1 -- 172.21.15.98:0/2028135566 wait complete. 2024-05-09T08:59:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f080099e640 1 Processor -- start 2024-05-09T08:59:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f080099e640 1 -- start start 2024-05-09T08:59:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f080099e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07fc100700 0x7f07fc196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f080099e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07fc1010c0 0x7f07fc1973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f080099e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07fc106710 0x7f07fc19b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f080099e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc110b30 con 0x7f07fc100700 2024-05-09T08:59:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f080099e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc1109b0 con 0x7f07fc1010c0 2024-05-09T08:59:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f080099e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc110cb0 con 0x7f07fc106710 2024-05-09T08:59:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f07fa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07fc100700 0x7f07fc196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f07fa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07fc100700 0x7f07fc196e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57464/0 (socket says 172.21.15.98:57464) 2024-05-09T08:59:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f07fa59c640 1 -- 172.21.15.98:0/625831356 learned_addr learned my addr 172.21.15.98:0/625831356 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f07fad9d640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07fc106710 0x7f07fc19b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f07f3fff640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07fc1010c0 0x7f07fc1973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f07fa59c640 1 -- 172.21.15.98:0/625831356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07fc106710 msgr2=0x7f07fc19b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f07fa59c640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07fc106710 0x7f07fc19b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f07fa59c640 1 -- 172.21.15.98:0/625831356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07fc1010c0 msgr2=0x7f07fc1973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f07fa59c640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07fc1010c0 0x7f07fc1973b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f07fa59c640 1 -- 172.21.15.98:0/625831356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07fc19be60 con 0x7f07fc100700 2024-05-09T08:59:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f07f3fff640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07fc1010c0 0x7f07fc1973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:18.999+0000 7f07fa59c640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07fc100700 0x7f07fc196e70 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f07ec00c920 tx=0x7f07ec00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.000+0000 7f07f37fe640 1 -- 172.21.15.98:0/625831356 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec007c70 con 0x7f07fc100700 2024-05-09T08:59:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.000+0000 7f07f37fe640 1 -- 172.21.15.98:0/625831356 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07ec007e10 con 0x7f07fc100700 2024-05-09T08:59:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.000+0000 7f080099e640 1 -- 172.21.15.98:0/625831356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07fc19c150 con 0x7f07fc100700 2024-05-09T08:59:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.000+0000 7f07f37fe640 1 -- 172.21.15.98:0/625831356 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec018890 con 0x7f07fc100700 2024-05-09T08:59:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.000+0000 7f080099e640 1 -- 172.21.15.98:0/625831356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07fc1a39e0 con 0x7f07fc100700 2024-05-09T08:59:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.002+0000 7f07f37fe640 1 -- 172.21.15.98:0/625831356 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f07ec018a70 con 0x7f07fc100700 2024-05-09T08:59:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.002+0000 7f0803789740 1 Processor -- start 2024-05-09T08:59:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f0803789740 1 -- start start 2024-05-09T08:59:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f0803789740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55930221d2c0 0x55930221b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f0803789740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221bbd0 0x55930221c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f0803789740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559302226950 0x559302228d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f0803789740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55930217e990 con 0x55930221bbd0 2024-05-09T08:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f0803789740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5593021265b0 con 0x559302226950 2024-05-09T08:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f0803789740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55930217a950 con 0x55930221d2c0 2024-05-09T08:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f07fa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221bbd0 0x55930221c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f07f37fe640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f07e80a2d00 0x7f07e80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f07fa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221bbd0 0x55930221c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57480/0 (socket says 172.21.15.98:57480) 2024-05-09T08:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f07fa59c640 1 -- 172.21.15.98:0/1498299007 learned_addr learned my addr 172.21.15.98:0/1498299007 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f07fad9d640 1 --2- 172.21.15.98:0/1498299007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55930221d2c0 0x55930221b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f07f3fff640 1 --2- 172.21.15.98:0/1498299007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559302226950 0x559302228d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f07fad9d640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f07e80a2d00 0x7f07e80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f07f37fe640 1 -- 172.21.15.98:0/625831356 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f07ec0cfa40 con 0x7f07fc100700 2024-05-09T08:59:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f07fa59c640 1 -- 172.21.15.98:0/1498299007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55930221d2c0 msgr2=0x55930221b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f07fa59c640 1 --2- 172.21.15.98:0/1498299007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55930221d2c0 0x55930221b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f07fa59c640 1 -- 172.21.15.98:0/1498299007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559302226950 msgr2=0x559302228d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.003+0000 7f07fa59c640 1 --2- 172.21.15.98:0/1498299007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559302226950 0x559302228d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.004+0000 7f07fa59c640 1 -- 172.21.15.98:0/1498299007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559301f5a7e0 con 0x55930221bbd0 2024-05-09T08:59:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.004+0000 7f07fa59c640 1 --2- 172.21.15.98:0/1498299007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221bbd0 0x55930221c050 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f07ec0984c0 tx=0x7f07ec020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea42b4afd9ea502 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.004+0000 7f07fad9d640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f07e80a2d00 0x7f07e80a51c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f07f4006010 tx=0x7f07f40095c0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.004+0000 7f07f0ff9640 1 -- 172.21.15.98:0/1498299007 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec09baf0 con 0x55930221bbd0 2024-05-09T08:59:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.004+0000 7f07f0ff9640 1 -- 172.21.15.98:0/1498299007 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07ec0a4760 con 0x55930221bbd0 2024-05-09T08:59:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.004+0000 7f07f0ff9640 1 -- 172.21.15.98:0/1498299007 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec0ccfa0 con 0x55930221bbd0 2024-05-09T08:59:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 -- 172.21.15.98:0/1498299007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221bbd0 msgr2=0x55930221c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 --2- 172.21.15.98:0/1498299007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221bbd0 0x55930221c050 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f07ec0984c0 tx=0x7f07ec020000 comp rx=0 tx=0).stop 2024-05-09T08:59:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 -- 172.21.15.98:0/1498299007 shutdown_connections 2024-05-09T08:59:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 --2- 172.21.15.98:0/1498299007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559302226950 0x559302228d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 --2- 172.21.15.98:0/1498299007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221bbd0 0x55930221c050 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 --2- 172.21.15.98:0/1498299007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55930221d2c0 0x55930221b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 -- 172.21.15.98:0/1498299007 >> 172.21.15.98:0/1498299007 conn(0x5593021e20a0 msgr2=0x559302218130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 -- 172.21.15.98:0/1498299007 shutdown_connections 2024-05-09T08:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 -- 172.21.15.98:0/1498299007 wait complete. 2024-05-09T08:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 -- start start 2024-05-09T08:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 Processor -- start 2024-05-09T08:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55930221bbd0 0x5593022238c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221d2c0 0x559302223e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559302226950 0x559302224340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559302229720 con 0x55930221d2c0 2024-05-09T08:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559302139b40 con 0x559302226950 2024-05-09T08:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5593022298a0 con 0x55930221bbd0 2024-05-09T08:59:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f0803789740 10 client.0 _ll_register_callbacks cb 0x55930212e0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f07f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221d2c0 0x559302223e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f07f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221d2c0 0x559302223e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57494/0 (socket says 172.21.15.98:57494) 2024-05-09T08:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f07f3fff640 1 -- 172.21.15.98:0/752175133 learned_addr learned my addr 172.21.15.98:0/752175133 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.005+0000 7f07fad9d640 1 --2- 172.21.15.98:0/752175133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559302226950 0x559302224340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f07fa59c640 1 --2- 172.21.15.98:0/752175133 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55930221bbd0 0x5593022238c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f07f3fff640 1 -- 172.21.15.98:0/752175133 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55930221bbd0 msgr2=0x5593022238c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f07f3fff640 1 --2- 172.21.15.98:0/752175133 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55930221bbd0 0x5593022238c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f07f3fff640 1 -- 172.21.15.98:0/752175133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559302226950 msgr2=0x559302224340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f07f3fff640 1 --2- 172.21.15.98:0/752175133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559302226950 0x559302224340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f07f3fff640 1 -- 172.21.15.98:0/752175133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559301fbaf70 con 0x55930221d2c0 2024-05-09T08:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f07f3fff640 1 --2- 172.21.15.98:0/752175133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221d2c0 0x559302223e00 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f07e4010a50 tx=0x7f07e4010e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52aa51ef0d6031b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f07e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f07e1ffb640 1 -- 172.21.15.98:0/752175133 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e401d060 con 0x55930221d2c0 2024-05-09T08:59:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f07e1ffb640 1 -- 172.21.15.98:0/752175133 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f07e400eb20 con 0x55930221d2c0 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f0803789740 1 -- 172.21.15.98:0/752175133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07ec0a4760 con 0x55930221d2c0 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f07e1ffb640 1 -- 172.21.15.98:0/752175133 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f07e4024050 con 0x55930221d2c0 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f0803789740 10 client.18078 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f0803789740 1 -- 172.21.15.98:0/752175133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559301e52170 con 0x55930221d2c0 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f07e1ffb640 1 -- 172.21.15.98:0/752175133 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07ec0a4760 con 0x55930221d2c0 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.006+0000 7f07e1ffb640 1 -- 172.21.15.98:0/752175133 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f07e400d3a0 con 0x55930221d2c0 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.007+0000 7f0803789740 10 client.18078 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.007+0000 7f0803789740 10 client.18078 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.007+0000 7f0803789740 20 client.18078 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.007+0000 7f0803789740 1 --2- 172.21.15.98:0/752175133 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559302236aa0 0x559302238eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.007+0000 7f0803789740 4 client.18078 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.007+0000 7f0803789740 1 -- 172.21.15.98:0/752175133 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55930218c0c0 con 0x559302236aa0 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.007+0000 7f07fa59c640 1 --2- 172.21.15.98:0/752175133 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559302236aa0 0x559302238eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.007+0000 7f07e1ffb640 10 client.18078 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.007+0000 7f07fa59c640 1 --2- 172.21.15.98:0/752175133 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559302236aa0 0x559302238eb0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fb084fa9d145ab8 server_cookie=9a6522bfe206b56a in_seq=0 out_seq=0 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.008+0000 7f07e1ffb640 1 -- 172.21.15.98:0/752175133 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55930218c0c0 con 0x559302236aa0 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.008+0000 7f07e1ffb640 10 client.18078 handle_command_reply: tid=0 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.008+0000 7f0803789740 1 client.18078 shutdown 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.008+0000 7f0803789740 1 -- 172.21.15.98:0/752175133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221d2c0 msgr2=0x559302223e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.008+0000 7f0803789740 1 --2- 172.21.15.98:0/752175133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221d2c0 0x559302223e00 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f07e4010a50 tx=0x7f07e4010e50 comp rx=0 tx=0).stop 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.008+0000 7f0803789740 1 -- 172.21.15.98:0/752175133 shutdown_connections 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.008+0000 7f0803789740 1 --2- 172.21.15.98:0/752175133 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559302236aa0 0x559302238eb0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.008+0000 7f0803789740 1 --2- 172.21.15.98:0/752175133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559302226950 0x559302224340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.008+0000 7f07e1ffb640 0 client.18078 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.008+0000 7f0803789740 1 --2- 172.21.15.98:0/752175133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221d2c0 0x559302223e00 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.008+0000 7f0803789740 1 --2- 172.21.15.98:0/752175133 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55930221bbd0 0x5593022238c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.008+0000 7f0803789740 1 -- 172.21.15.98:0/752175133 >> 172.21.15.98:0/752175133 conn(0x5593021e20a0 msgr2=0x559302215080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.009+0000 7f0803789740 1 -- 172.21.15.98:0/752175133 shutdown_connections 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.009+0000 7f0803789740 1 -- 172.21.15.98:0/752175133 wait complete. 2024-05-09T08:59:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.009+0000 7f0803789740 20 client.18078 trim_cache size 0 max 16384 2024-05-09T08:59:19.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.026+0000 7f0803789740 1 Processor -- start 2024-05-09T08:59:19.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.026+0000 7f0803789740 1 -- start start 2024-05-09T08:59:19.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.026+0000 7f0803789740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55930221bbd0 0x559302232650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.026+0000 7f0803789740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221d2c0 0x55930222ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.026+0000 7f0803789740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559302226950 0x559302223f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.026+0000 7f0803789740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc1acc10 con 0x55930221d2c0 2024-05-09T08:59:19.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.026+0000 7f0803789740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc1ae750 con 0x55930221bbd0 2024-05-09T08:59:19.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.026+0000 7f0803789740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc1ad2e0 con 0x559302226950 2024-05-09T08:59:19.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.026+0000 7f07f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221d2c0 0x55930222ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.026+0000 7f07f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221d2c0 0x55930222ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57500/0 (socket says 172.21.15.98:57500) 2024-05-09T08:59:19.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.026+0000 7f07f3fff640 1 -- 172.21.15.98:0/823510890 learned_addr learned my addr 172.21.15.98:0/823510890 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:19.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.026+0000 7f07fad9d640 1 --2- 172.21.15.98:0/823510890 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559302226950 0x559302223f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f07f3fff640 1 -- 172.21.15.98:0/823510890 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559302226950 msgr2=0x559302223f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f07fa59c640 1 --2- 172.21.15.98:0/823510890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55930221bbd0 0x559302232650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f07f3fff640 1 --2- 172.21.15.98:0/823510890 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559302226950 0x559302223f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f07f3fff640 1 -- 172.21.15.98:0/823510890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55930221bbd0 msgr2=0x559302232650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f07f3fff640 1 --2- 172.21.15.98:0/823510890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55930221bbd0 0x559302232650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f07f3fff640 1 -- 172.21.15.98:0/823510890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559302222870 con 0x55930221d2c0 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f07fad9d640 1 --2- 172.21.15.98:0/823510890 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559302226950 0x559302223f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f07fa59c640 1 --2- 172.21.15.98:0/823510890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55930221bbd0 0x559302232650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f07f3fff640 1 --2- 172.21.15.98:0/823510890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221d2c0 0x55930222ef40 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f07e400b310 tx=0x7f07e4010640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a208339788c823b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f07e3fff640 1 -- 172.21.15.98:0/823510890 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e400d9e0 con 0x55930221d2c0 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f07e3fff640 1 -- 172.21.15.98:0/823510890 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07e402d070 con 0x55930221d2c0 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f07e3fff640 1 -- 172.21.15.98:0/823510890 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e4005580 con 0x55930221d2c0 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f0803789740 1 -- 172.21.15.98:0/823510890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221d2c0 msgr2=0x55930222ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f0803789740 1 --2- 172.21.15.98:0/823510890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221d2c0 0x55930222ef40 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f07e400b310 tx=0x7f07e4010640 comp rx=0 tx=0).stop 2024-05-09T08:59:19.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f0803789740 1 -- 172.21.15.98:0/823510890 shutdown_connections 2024-05-09T08:59:19.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f0803789740 1 --2- 172.21.15.98:0/823510890 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559302226950 0x559302223f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f0803789740 1 --2- 172.21.15.98:0/823510890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221d2c0 0x55930222ef40 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f0803789740 1 --2- 172.21.15.98:0/823510890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55930221bbd0 0x559302232650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f0803789740 1 -- 172.21.15.98:0/823510890 >> 172.21.15.98:0/823510890 conn(0x5593021e20a0 msgr2=0x559302223880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:19.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f0803789740 1 -- 172.21.15.98:0/823510890 shutdown_connections 2024-05-09T08:59:19.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.027+0000 7f0803789740 1 -- 172.21.15.98:0/823510890 wait complete. 2024-05-09T08:59:19.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f0803789740 1 -- start start 2024-05-09T08:59:19.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f0803789740 1 Processor -- start 2024-05-09T08:59:19.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f0803789740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221bbd0 0x559302217dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f0803789740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55930221d2c0 0x559302216170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f0803789740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559302226950 0x5593022166b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f0803789740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc1ae080 con 0x55930221bbd0 2024-05-09T08:59:19.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f0803789740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc1ad9b0 con 0x55930221d2c0 2024-05-09T08:59:19.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f0803789740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc1aac10 con 0x559302226950 2024-05-09T08:59:19.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f0803789740 10 client.0 _ll_register_callbacks cb 0x7f07e400b790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:19.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f07fa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221bbd0 0x559302217dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f07fa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221bbd0 0x559302217dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57512/0 (socket says 172.21.15.98:57512) 2024-05-09T08:59:19.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f07fa59c640 1 -- 172.21.15.98:0/1747010902 learned_addr learned my addr 172.21.15.98:0/1747010902 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:19.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f07f3fff640 1 --2- 172.21.15.98:0/1747010902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55930221d2c0 0x559302216170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f07fa59c640 1 -- 172.21.15.98:0/1747010902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559302226950 msgr2=0x5593022166b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f07fad9d640 1 --2- 172.21.15.98:0/1747010902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559302226950 0x5593022166b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f07fa59c640 1 --2- 172.21.15.98:0/1747010902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559302226950 0x5593022166b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f07fa59c640 1 -- 172.21.15.98:0/1747010902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55930221d2c0 msgr2=0x559302216170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f07fa59c640 1 --2- 172.21.15.98:0/1747010902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55930221d2c0 0x559302216170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.028+0000 7f07fa59c640 1 -- 172.21.15.98:0/1747010902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559302216bf0 con 0x55930221bbd0 2024-05-09T08:59:19.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f07fa59c640 1 --2- 172.21.15.98:0/1747010902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221bbd0 0x559302217dd0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f07ec00fee0 tx=0x7f07ec00c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3589efeb8fa596ca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:19.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f07e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:19.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f07e1ffb640 1 -- 172.21.15.98:0/1747010902 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec0a4760 con 0x55930221bbd0 2024-05-09T08:59:19.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f07e1ffb640 1 -- 172.21.15.98:0/1747010902 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f07ec00fb30 con 0x55930221bbd0 2024-05-09T08:59:19.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f0803789740 1 -- 172.21.15.98:0/1747010902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07e402d070 con 0x55930221bbd0 2024-05-09T08:59:19.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f07e1ffb640 1 -- 172.21.15.98:0/1747010902 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f07ec098160 con 0x55930221bbd0 2024-05-09T08:59:19.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f0803789740 10 client.18084 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:19.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f0803789740 1 -- 172.21.15.98:0/1747010902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55930229fca0 con 0x55930221bbd0 2024-05-09T08:59:19.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f07e1ffb640 1 -- 172.21.15.98:0/1747010902 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07e402d070 con 0x55930221bbd0 2024-05-09T08:59:19.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f07e1ffb640 1 -- 172.21.15.98:0/1747010902 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f07ec0a4360 con 0x55930221bbd0 2024-05-09T08:59:19.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f0803789740 10 client.18084 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:19.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f0803789740 10 client.18084 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:19.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f0803789740 20 client.18084 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:19.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f0803789740 1 --2- 172.21.15.98:0/1747010902 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55930229ff90 0x5593022a2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f0803789740 4 client.18084 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:59:19.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f0803789740 1 -- 172.21.15.98:0/1747010902 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559302186410 con 0x55930229ff90 2024-05-09T08:59:19.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.029+0000 7f07f3fff640 1 --2- 172.21.15.98:0/1747010902 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55930229ff90 0x5593022a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.030+0000 7f07e1ffb640 10 client.18084 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:19.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.030+0000 7f07f3fff640 1 --2- 172.21.15.98:0/1747010902 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55930229ff90 0x5593022a2380 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed7a3514ae809b27 server_cookie=b17fb8db2a864390 in_seq=0 out_seq=0 2024-05-09T08:59:19.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f07e1ffb640 1 -- 172.21.15.98:0/1747010902 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559302186410 con 0x55930229ff90 2024-05-09T08:59:19.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f07e1ffb640 10 client.18084 handle_command_reply: tid=0 2024-05-09T08:59:19.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f0803789740 1 client.18084 shutdown 2024-05-09T08:59:19.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f0803789740 1 -- 172.21.15.98:0/1747010902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221bbd0 msgr2=0x559302217dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f0803789740 1 --2- 172.21.15.98:0/1747010902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221bbd0 0x559302217dd0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f07ec00fee0 tx=0x7f07ec00c870 comp rx=0 tx=0).stop 2024-05-09T08:59:19.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f0803789740 1 -- 172.21.15.98:0/1747010902 shutdown_connections 2024-05-09T08:59:19.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f0803789740 1 --2- 172.21.15.98:0/1747010902 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55930229ff90 0x5593022a2380 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f07e1ffb640 0 client.18084 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:19.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f0803789740 1 --2- 172.21.15.98:0/1747010902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559302226950 0x5593022166b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f0803789740 1 --2- 172.21.15.98:0/1747010902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55930221d2c0 0x559302216170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f0803789740 1 --2- 172.21.15.98:0/1747010902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55930221bbd0 0x559302217dd0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f0803789740 1 -- 172.21.15.98:0/1747010902 >> 172.21.15.98:0/1747010902 conn(0x5593021e20a0 msgr2=0x559302223860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:19.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f0803789740 1 -- 172.21.15.98:0/1747010902 shutdown_connections 2024-05-09T08:59:19.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f0803789740 1 -- 172.21.15.98:0/1747010902 wait complete. 2024-05-09T08:59:19.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.031+0000 7f0803789740 20 client.18084 trim_cache size 0 max 16384 2024-05-09T08:59:19.037 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:59:19.037 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:59:19.037 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "aecb8913-ac8d-4934-b826-92dea511cc3c", 2024-05-09T08:59:19.037 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:59:19.037 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:59:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.032+0000 7f080099e640 1 -- 172.21.15.98:0/625831356 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f07e80a2d00 msgr2=0x7f07e80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.032+0000 7f080099e640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f07e80a2d00 0x7f07e80a51c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f07f4006010 tx=0x7f07f40095c0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.032+0000 7f080099e640 1 -- 172.21.15.98:0/625831356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07fc100700 msgr2=0x7f07fc196e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.032+0000 7f080099e640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07fc100700 0x7f07fc196e70 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f07ec00c920 tx=0x7f07ec00cde0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.032+0000 7f080099e640 1 -- 172.21.15.98:0/625831356 shutdown_connections 2024-05-09T08:59:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.032+0000 7f080099e640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f07e80a2d00 0x7f07e80a51c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.032+0000 7f080099e640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07fc106710 0x7f07fc19b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.032+0000 7f080099e640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07fc1010c0 0x7f07fc1973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.032+0000 7f080099e640 1 --2- 172.21.15.98:0/625831356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07fc100700 0x7f07fc196e70 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.032+0000 7f080099e640 1 -- 172.21.15.98:0/625831356 >> 172.21.15.98:0/625831356 conn(0x7f07fc077830 msgr2=0x7f07fc102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:19.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.032+0000 7f080099e640 1 -- 172.21.15.98:0/625831356 shutdown_connections 2024-05-09T08:59:19.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.032+0000 7f080099e640 1 -- 172.21.15.98:0/625831356 wait complete. 2024-05-09T08:59:19.043 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:59:19.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:59:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.147+0000 7fd5adb90640 1 -- 172.21.15.98:0/3187721114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a8106720 msgr2=0x7fd5a8106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.147+0000 7fd5adb90640 1 --2- 172.21.15.98:0/3187721114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a8106720 0x7fd5a8106b00 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fd598009930 tx=0x7fd59802a420 comp rx=0 tx=0).stop 2024-05-09T08:59:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.147+0000 7fd5adb90640 1 -- 172.21.15.98:0/3187721114 shutdown_connections 2024-05-09T08:59:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.147+0000 7fd5adb90640 1 --2- 172.21.15.98:0/3187721114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a81010d0 0x7fd5a810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.147+0000 7fd5adb90640 1 --2- 172.21.15.98:0/3187721114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5a8100710 0x7fd5a8100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.147+0000 7fd5adb90640 1 --2- 172.21.15.98:0/3187721114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a8106720 0x7fd5a8106b00 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.147+0000 7fd5adb90640 1 -- 172.21.15.98:0/3187721114 >> 172.21.15.98:0/3187721114 conn(0x7fd5a8077790 msgr2=0x7fd5a8077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.147+0000 7fd5adb90640 1 -- 172.21.15.98:0/3187721114 shutdown_connections 2024-05-09T08:59:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.147+0000 7fd5adb90640 1 -- 172.21.15.98:0/3187721114 wait complete. 2024-05-09T08:59:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5adb90640 1 Processor -- start 2024-05-09T08:59:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5adb90640 1 -- start start 2024-05-09T08:59:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5adb90640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5a8100710 0x7fd5a8198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5adb90640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a81010d0 0x7fd5a81986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5adb90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a8106720 0x7fd5a819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5adb90640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a8111c30 con 0x7fd5a8106720 2024-05-09T08:59:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5adb90640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a8111ab0 con 0x7fd5a8100710 2024-05-09T08:59:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5adb90640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a8111db0 con 0x7fd5a81010d0 2024-05-09T08:59:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5a7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a8106720 0x7fd5a819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5a7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a8106720 0x7fd5a819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57526/0 (socket says 172.21.15.98:57526) 2024-05-09T08:59:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5a7fff640 1 -- 172.21.15.98:0/2459006818 learned_addr learned my addr 172.21.15.98:0/2459006818 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5a77fe640 1 --2- 172.21.15.98:0/2459006818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5a8100710 0x7fd5a8198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/2459006818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a81010d0 0x7fd5a81986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5a7fff640 1 -- 172.21.15.98:0/2459006818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a81010d0 msgr2=0x7fd5a81986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5a7fff640 1 --2- 172.21.15.98:0/2459006818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a81010d0 0x7fd5a81986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5a7fff640 1 -- 172.21.15.98:0/2459006818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5a8100710 msgr2=0x7fd5a8198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5a7fff640 1 --2- 172.21.15.98:0/2459006818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5a8100710 0x7fd5a8198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.148+0000 7fd5a7fff640 1 -- 172.21.15.98:0/2459006818 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a819d180 con 0x7fd5a8106720 2024-05-09T08:59:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.149+0000 7fd5a7fff640 1 --2- 172.21.15.98:0/2459006818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a8106720 0x7fd5a819ca50 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000eaa0 tx=0x7fd5a000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.149+0000 7fd5a4ff9640 1 -- 172.21.15.98:0/2459006818 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a000cdb0 con 0x7fd5a8106720 2024-05-09T08:59:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.149+0000 7fd5adb90640 1 -- 172.21.15.98:0/2459006818 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a819d470 con 0x7fd5a8106720 2024-05-09T08:59:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.149+0000 7fd5a4ff9640 1 -- 172.21.15.98:0/2459006818 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5a0005020 con 0x7fd5a8106720 2024-05-09T08:59:19.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.149+0000 7fd5a4ff9640 1 -- 172.21.15.98:0/2459006818 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a00116b0 con 0x7fd5a8106720 2024-05-09T08:59:19.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.149+0000 7fd5adb90640 1 -- 172.21.15.98:0/2459006818 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a81a4d00 con 0x7fd5a8106720 2024-05-09T08:59:19.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.151+0000 7fd5a4ff9640 1 -- 172.21.15.98:0/2459006818 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd5a0005560 con 0x7fd5a8106720 2024-05-09T08:59:19.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.151+0000 7fd5b097c740 1 Processor -- start 2024-05-09T08:59:19.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.151+0000 7fd5b097c740 1 -- start start 2024-05-09T08:59:19.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5b097c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fc280 0x5594515fc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5b097c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fcba0 0x559451606d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5b097c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 0x559451609660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5b097c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559451561d00 con 0x5594515fcba0 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5b097c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5594515094c0 con 0x5594515fc280 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5b097c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55945155dcc0 con 0x559451607270 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fcba0 0x559451606d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fcba0 0x559451606d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57532/0 (socket says 172.21.15.98:57532) 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a77fe640 1 -- 172.21.15.98:0/1855859841 learned_addr learned my addr 172.21.15.98:0/1855859841 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/1855859841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 0x559451609660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/1855859841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fc280 0x5594515fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a4ff9640 1 --2- 172.21.15.98:0/2459006818 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd5900a2d00 0x7fd5900a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a77fe640 1 -- 172.21.15.98:0/1855859841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 msgr2=0x559451609660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a77fe640 1 --2- 172.21.15.98:0/1855859841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 0x559451609660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a77fe640 1 -- 172.21.15.98:0/1855859841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fc280 msgr2=0x5594515fc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a77fe640 1 --2- 172.21.15.98:0/1855859841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fc280 0x5594515fc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a77fe640 1 -- 172.21.15.98:0/1855859841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55945133f7d0 con 0x5594515fcba0 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/1855859841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fc280 0x5594515fc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a7fff640 1 --2- 172.21.15.98:0/2459006818 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd5900a2d00 0x7fd5900a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a4ff9640 1 -- 172.21.15.98:0/2459006818 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fd5a00cf9c0 con 0x7fd5a8106720 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd5a77fe640 1 --2- 172.21.15.98:0/1855859841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fcba0 0x559451606d30 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fd59802a940 tx=0x7fd598005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e38476af0d099719 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd58dffb640 1 -- 172.21.15.98:0/1855859841 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598040070 con 0x5594515fcba0 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd58dffb640 1 -- 172.21.15.98:0/1855859841 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5980319f0 con 0x5594515fcba0 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.152+0000 7fd58dffb640 1 -- 172.21.15.98:0/1855859841 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598035040 con 0x5594515fcba0 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.153+0000 7fd5b097c740 1 -- 172.21.15.98:0/1855859841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fcba0 msgr2=0x559451606d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.153+0000 7fd5a7fff640 1 --2- 172.21.15.98:0/2459006818 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd5900a2d00 0x7fd5900a51c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000efd0 tx=0x7fd5a006db40 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.153+0000 7fd5b097c740 1 --2- 172.21.15.98:0/1855859841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fcba0 0x559451606d30 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fd59802a940 tx=0x7fd598005c00 comp rx=0 tx=0).stop 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.153+0000 7fd5b097c740 1 -- 172.21.15.98:0/1855859841 shutdown_connections 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.153+0000 7fd5b097c740 1 --2- 172.21.15.98:0/1855859841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 0x559451609660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.153+0000 7fd5b097c740 1 --2- 172.21.15.98:0/1855859841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fcba0 0x559451606d30 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.153+0000 7fd5b097c740 1 --2- 172.21.15.98:0/1855859841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fc280 0x5594515fc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.153+0000 7fd5b097c740 1 -- 172.21.15.98:0/1855859841 >> 172.21.15.98:0/1855859841 conn(0x5594515f85a0 msgr2=0x5594515f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.153+0000 7fd5b097c740 1 -- 172.21.15.98:0/1855859841 shutdown_connections 2024-05-09T08:59:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.153+0000 7fd5b097c740 1 -- 172.21.15.98:0/1855859841 wait complete. 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.153+0000 7fd5b097c740 1 -- start start 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5b097c740 1 Processor -- start 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5b097c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fc280 0x559451604080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5b097c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fcba0 0x5594516045c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5b097c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 0x559451604b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5b097c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55945160a000 con 0x5594515fcba0 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5b097c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55945151cb90 con 0x5594515fc280 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5b097c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55945160a180 con 0x559451607270 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5b097c740 10 client.0 _ll_register_callbacks cb 0x559451558da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fcba0 0x5594516045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fcba0 0x5594516045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57542/0 (socket says 172.21.15.98:57542) 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5a77fe640 1 -- 172.21.15.98:0/3953114035 learned_addr learned my addr 172.21.15.98:0/3953114035 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/3953114035 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fc280 0x559451604080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/3953114035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 0x559451604b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5a77fe640 1 -- 172.21.15.98:0/3953114035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 msgr2=0x559451604b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5a77fe640 1 --2- 172.21.15.98:0/3953114035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 0x559451604b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5a77fe640 1 -- 172.21.15.98:0/3953114035 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fc280 msgr2=0x559451604080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5a77fe640 1 --2- 172.21.15.98:0/3953114035 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fc280 0x559451604080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5a77fe640 1 -- 172.21.15.98:0/3953114035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594512753f0 con 0x5594515fcba0 2024-05-09T08:59:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.154+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/3953114035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 0x559451604b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd5a77fe640 1 --2- 172.21.15.98:0/3953114035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fcba0 0x5594516045c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fd598006010 tx=0x7fd59802a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=849e9b0dff334fa0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd586ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd586ffd640 1 -- 172.21.15.98:0/3953114035 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598040030 con 0x5594515fcba0 2024-05-09T08:59:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd586ffd640 1 -- 172.21.15.98:0/3953114035 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fd598004260 con 0x5594515fcba0 2024-05-09T08:59:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd5b097c740 1 -- 172.21.15.98:0/3953114035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5980319f0 con 0x5594515fcba0 2024-05-09T08:59:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd586ffd640 1 -- 172.21.15.98:0/3953114035 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd598056050 con 0x5594515fcba0 2024-05-09T08:59:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd5b097c740 10 client.18096 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd5b097c740 1 -- 172.21.15.98:0/3953114035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5594512350f0 con 0x5594515fcba0 2024-05-09T08:59:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd586ffd640 1 -- 172.21.15.98:0/3953114035 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5980319f0 con 0x5594515fcba0 2024-05-09T08:59:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd586ffd640 1 -- 172.21.15.98:0/3953114035 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd598003850 con 0x5594515fcba0 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd5b097c740 10 client.18096 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd5b097c740 10 client.18096 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd5b097c740 20 client.18096 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd5b097c740 1 --2- 172.21.15.98:0/3953114035 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559451617250 0x559451619640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd5b097c740 4 client.18096 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.155+0000 7fd5b097c740 1 -- 172.21.15.98:0/3953114035 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55945156f430 con 0x559451617250 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.156+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/3953114035 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559451617250 0x559451619640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.156+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/3953114035 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559451617250 0x559451619640 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41c386ec90914feb server_cookie=930107f585614fcc in_seq=0 out_seq=0 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.156+0000 7fd586ffd640 10 client.18096 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.156+0000 7fd586ffd640 1 -- 172.21.15.98:0/3953114035 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55945156f430 con 0x559451617250 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.157+0000 7fd586ffd640 10 client.18096 handle_command_reply: tid=0 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.157+0000 7fd5b097c740 1 client.18096 shutdown 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.157+0000 7fd5b097c740 1 -- 172.21.15.98:0/3953114035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fcba0 msgr2=0x5594516045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.157+0000 7fd5b097c740 1 --2- 172.21.15.98:0/3953114035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fcba0 0x5594516045c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fd598006010 tx=0x7fd59802a420 comp rx=0 tx=0).stop 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.157+0000 7fd5b097c740 1 -- 172.21.15.98:0/3953114035 shutdown_connections 2024-05-09T08:59:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.157+0000 7fd5b097c740 1 --2- 172.21.15.98:0/3953114035 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559451617250 0x559451619640 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.157+0000 7fd5b097c740 1 --2- 172.21.15.98:0/3953114035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 0x559451604b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.157+0000 7fd5b097c740 1 --2- 172.21.15.98:0/3953114035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fcba0 0x5594516045c0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.157+0000 7fd586ffd640 0 client.18096 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:19.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.157+0000 7fd5b097c740 1 --2- 172.21.15.98:0/3953114035 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fc280 0x559451604080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.157+0000 7fd5b097c740 1 -- 172.21.15.98:0/3953114035 >> 172.21.15.98:0/3953114035 conn(0x5594515f85a0 msgr2=0x5594515f77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:19.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.157+0000 7fd5b097c740 1 -- 172.21.15.98:0/3953114035 shutdown_connections 2024-05-09T08:59:19.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.157+0000 7fd5b097c740 1 -- 172.21.15.98:0/3953114035 wait complete. 2024-05-09T08:59:19.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.157+0000 7fd5b097c740 20 client.18096 trim_cache size 0 max 16384 2024-05-09T08:59:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5b097c740 1 Processor -- start 2024-05-09T08:59:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5b097c740 1 -- start start 2024-05-09T08:59:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5b097c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fc280 0x559451612ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5b097c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fcba0 0x5594515fe520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5b097c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 0x5594516031e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5b097c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a81aee40 con 0x5594515fc280 2024-05-09T08:59:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5b097c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a81b0980 con 0x5594515fcba0 2024-05-09T08:59:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5b097c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a81af510 con 0x559451607270 2024-05-09T08:59:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5a6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fc280 0x559451612ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5a6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fc280 0x559451612ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57546/0 (socket says 172.21.15.98:57546) 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5a6ffd640 1 -- 172.21.15.98:0/144872627 learned_addr learned my addr 172.21.15.98:0/144872627 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5a6ffd640 1 -- 172.21.15.98:0/144872627 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 msgr2=0x5594516031e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/144872627 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 0x5594516031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5a6ffd640 1 -- 172.21.15.98:0/144872627 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fcba0 msgr2=0x5594515fe520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/144872627 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fcba0 0x5594515fe520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.174+0000 7fd5a6ffd640 1 -- 172.21.15.98:0/144872627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594515fc9d0 con 0x5594515fc280 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.175+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/144872627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fc280 0x559451612ef0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0042e0 tx=0x7fd59c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=debbceb67d149936 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.175+0000 7fd58d7fa640 1 -- 172.21.15.98:0/144872627 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c025020 con 0x5594515fc280 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.175+0000 7fd58d7fa640 1 -- 172.21.15.98:0/144872627 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd59c005500 con 0x5594515fc280 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.175+0000 7fd58d7fa640 1 -- 172.21.15.98:0/144872627 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c0212d0 con 0x5594515fc280 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.175+0000 7fd5b097c740 1 -- 172.21.15.98:0/144872627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fc280 msgr2=0x559451612ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.175+0000 7fd5b097c740 1 --2- 172.21.15.98:0/144872627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fc280 0x559451612ef0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0042e0 tx=0x7fd59c00b870 comp rx=0 tx=0).stop 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.175+0000 7fd5b097c740 1 -- 172.21.15.98:0/144872627 shutdown_connections 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.175+0000 7fd5b097c740 1 --2- 172.21.15.98:0/144872627 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559451607270 0x5594516031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.175+0000 7fd5b097c740 1 --2- 172.21.15.98:0/144872627 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5594515fcba0 0x5594515fe520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.175+0000 7fd5b097c740 1 --2- 172.21.15.98:0/144872627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fc280 0x559451612ef0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.175+0000 7fd5b097c740 1 -- 172.21.15.98:0/144872627 >> 172.21.15.98:0/144872627 conn(0x5594515f85a0 msgr2=0x559451604040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.175+0000 7fd5b097c740 1 -- 172.21.15.98:0/144872627 shutdown_connections 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.175+0000 7fd5b097c740 1 -- 172.21.15.98:0/144872627 wait complete. 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.176+0000 7fd5b097c740 1 -- start start 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.176+0000 7fd5b097c740 1 Processor -- start 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.176+0000 7fd5b097c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fc280 0x559451600470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.176+0000 7fd5b097c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5594515fcba0 0x5594516009b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.176+0000 7fd5b097c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559451607270 0x559451600ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.176+0000 7fd5b097c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a81b02b0 con 0x5594515fc280 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.176+0000 7fd5b097c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a81afbe0 con 0x559451607270 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.176+0000 7fd5b097c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a81ace40 con 0x5594515fcba0 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.176+0000 7fd5b097c740 10 client.0 _ll_register_callbacks cb 0x7fd59802aae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.176+0000 7fd5a6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fc280 0x559451600470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.176+0000 7fd5a6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fc280 0x559451600470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57548/0 (socket says 172.21.15.98:57548) 2024-05-09T08:59:19.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.176+0000 7fd5a6ffd640 1 -- 172.21.15.98:0/865423084 learned_addr learned my addr 172.21.15.98:0/865423084 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:19.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.176+0000 7fd5a77fe640 1 --2- 172.21.15.98:0/865423084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5594515fcba0 0x5594516009b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.176+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/865423084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559451607270 0x559451600ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5a6ffd640 1 -- 172.21.15.98:0/865423084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5594515fcba0 msgr2=0x5594516009b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/865423084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5594515fcba0 0x5594516009b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5a6ffd640 1 -- 172.21.15.98:0/865423084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559451607270 msgr2=0x559451600ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:19.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/865423084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559451607270 0x559451600ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5a6ffd640 1 -- 172.21.15.98:0/865423084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55945160e6a0 con 0x5594515fc280 2024-05-09T08:59:19.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5a77fe640 1 --2- 172.21.15.98:0/865423084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5594515fcba0 0x5594516009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:19.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5a6ffd640 1 --2- 172.21.15.98:0/865423084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fc280 0x559451600470 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fd59c00c010 tx=0x7fd59c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbef302f38d010ca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:19.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd586ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:19.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd586ffd640 1 -- 172.21.15.98:0/865423084 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c034060 con 0x5594515fc280 2024-05-09T08:59:19.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd586ffd640 1 -- 172.21.15.98:0/865423084 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fd59c021050 con 0x5594515fc280 2024-05-09T08:59:19.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5b097c740 1 -- 172.21.15.98:0/865423084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd59c005500 con 0x5594515fc280 2024-05-09T08:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd586ffd640 1 -- 172.21.15.98:0/865423084 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd59c03b050 con 0x5594515fc280 2024-05-09T08:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5b097c740 10 client.18102 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5b097c740 1 -- 172.21.15.98:0/865423084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55945160f970 con 0x5594515fc280 2024-05-09T08:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd586ffd640 1 -- 172.21.15.98:0/865423084 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd59c005500 con 0x5594515fc280 2024-05-09T08:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd586ffd640 1 -- 172.21.15.98:0/865423084 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd59c012650 con 0x5594515fc280 2024-05-09T08:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5b097c740 10 client.18102 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5b097c740 10 client.18102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5b097c740 20 client.18102 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5b097c740 1 --2- 172.21.15.98:0/865423084 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55945167d5c0 0x55945167f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5b097c740 4 client.18102 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.177+0000 7fd5b097c740 1 -- 172.21.15.98:0/865423084 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559451569780 con 0x55945167d5c0 2024-05-09T08:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.178+0000 7fd5a77fe640 1 --2- 172.21.15.98:0/865423084 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55945167d5c0 0x55945167f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.178+0000 7fd586ffd640 10 client.18102 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.178+0000 7fd5a77fe640 1 --2- 172.21.15.98:0/865423084 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55945167d5c0 0x55945167f9b0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4858f5f86f969424 server_cookie=cf4dc60238d1a7b5 in_seq=0 out_seq=0 2024-05-09T08:59:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.178+0000 7fd586ffd640 1 -- 172.21.15.98:0/865423084 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559451569780 con 0x55945167d5c0 2024-05-09T08:59:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.178+0000 7fd586ffd640 10 client.18102 handle_command_reply: tid=0 2024-05-09T08:59:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.178+0000 7fd5b097c740 1 client.18102 shutdown 2024-05-09T08:59:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.178+0000 7fd5b097c740 1 -- 172.21.15.98:0/865423084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fc280 msgr2=0x559451600470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.178+0000 7fd5b097c740 1 --2- 172.21.15.98:0/865423084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fc280 0x559451600470 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fd59c00c010 tx=0x7fd59c004d10 comp rx=0 tx=0).stop 2024-05-09T08:59:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5b097c740 1 -- 172.21.15.98:0/865423084 shutdown_connections 2024-05-09T08:59:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5b097c740 1 --2- 172.21.15.98:0/865423084 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55945167d5c0 0x55945167f9b0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5b097c740 1 --2- 172.21.15.98:0/865423084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559451607270 0x559451600ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5b097c740 1 --2- 172.21.15.98:0/865423084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5594515fcba0 0x5594516009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd586ffd640 0 client.18102 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5b097c740 1 --2- 172.21.15.98:0/865423084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594515fc280 0x559451600470 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5b097c740 1 -- 172.21.15.98:0/865423084 >> 172.21.15.98:0/865423084 conn(0x5594515f85a0 msgr2=0x559451603060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5b097c740 1 -- 172.21.15.98:0/865423084 shutdown_connections 2024-05-09T08:59:19.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5b097c740 1 -- 172.21.15.98:0/865423084 wait complete. 2024-05-09T08:59:19.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5b097c740 20 client.18102 trim_cache size 0 max 16384 2024-05-09T08:59:19.185 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:59:19.185 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:59:19.185 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:59:19.185 INFO:teuthology.orchestra.run.smithi098.stdout: "aecb8913-ac8d-4934-b826-92dea511cc3c": { 2024-05-09T08:59:19.185 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:59:19.185 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "aecb8913-ac8d-4934-b826-92dea511cc3c", 2024-05-09T08:59:19.185 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:59:19.185 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:59:19.185 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:59:19.186 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:59:19.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5adb90640 1 -- 172.21.15.98:0/2459006818 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd5900a2d00 msgr2=0x7fd5900a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5adb90640 1 --2- 172.21.15.98:0/2459006818 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd5900a2d00 0x7fd5900a51c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000efd0 tx=0x7fd5a006db40 comp rx=0 tx=0).stop 2024-05-09T08:59:19.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5adb90640 1 -- 172.21.15.98:0/2459006818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a8106720 msgr2=0x7fd5a819ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:19.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5adb90640 1 --2- 172.21.15.98:0/2459006818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a8106720 0x7fd5a819ca50 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000eaa0 tx=0x7fd5a000ef60 comp rx=0 tx=0).stop 2024-05-09T08:59:19.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5adb90640 1 -- 172.21.15.98:0/2459006818 shutdown_connections 2024-05-09T08:59:19.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5adb90640 1 --2- 172.21.15.98:0/2459006818 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd5900a2d00 0x7fd5900a51c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5adb90640 1 --2- 172.21.15.98:0/2459006818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a8106720 0x7fd5a819ca50 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5adb90640 1 --2- 172.21.15.98:0/2459006818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a81010d0 0x7fd5a81986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5adb90640 1 --2- 172.21.15.98:0/2459006818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5a8100710 0x7fd5a8198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:19.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5adb90640 1 -- 172.21.15.98:0/2459006818 >> 172.21.15.98:0/2459006818 conn(0x7fd5a8077790 msgr2=0x7fd5a81032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:19.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5adb90640 1 -- 172.21.15.98:0/2459006818 shutdown_connections 2024-05-09T08:59:19.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:19.179+0000 7fd5adb90640 1 -- 172.21.15.98:0/2459006818 wait complete. 2024-05-09T08:59:19.190 INFO:tasks.cephfs.filesystem:scrub status for tag:aecb8913-ac8d-4934-b826-92dea511cc3c - {'path': '/', 'tag': 'aecb8913-ac8d-4934-b826-92dea511cc3c', 'options': 'recursive,force'} 2024-05-09T08:59:20.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:20 smithi098 ceph-mon[30092]: pgmap v271: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-05-09T08:59:20.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:20 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:59:20.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:20 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:59:20.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:20 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:59:20.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:20 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:59:20.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:20 smithi142 ceph-mon[34625]: pgmap v271: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-05-09T08:59:20.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:20 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:59:20.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:20 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:59:20.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:20 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:59:20.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:20 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:59:20.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:20 smithi105 ceph-mon[34498]: pgmap v271: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-05-09T08:59:20.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:20 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:59:20.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:20 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:59:20.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:20 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:59:20.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:20 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:59:22.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:22 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:59:22.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:22 smithi098 ceph-mon[30092]: pgmap v272: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 511 B/s wr, 13 op/s 2024-05-09T08:59:22.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:22 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:59:22.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:22 smithi142 ceph-mon[34625]: pgmap v272: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 511 B/s wr, 13 op/s 2024-05-09T08:59:22.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:22 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:59:22.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:22 smithi105 ceph-mon[34498]: pgmap v272: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 511 B/s wr, 13 op/s 2024-05-09T08:59:24.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:24 smithi098 ceph-mon[30092]: pgmap v273: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 409 B/s wr, 13 op/s 2024-05-09T08:59:24.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:24 smithi142 ceph-mon[34625]: pgmap v273: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 409 B/s wr, 13 op/s 2024-05-09T08:59:24.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:24 smithi105 ceph-mon[34498]: pgmap v273: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 409 B/s wr, 13 op/s 2024-05-09T08:59:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:26 smithi098 ceph-mon[30092]: pgmap v274: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 409 B/s wr, 14 op/s 2024-05-09T08:59:26.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:26 smithi142 ceph-mon[34625]: pgmap v274: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 409 B/s wr, 14 op/s 2024-05-09T08:59:26.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:26 smithi105 ceph-mon[34498]: pgmap v274: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 409 B/s wr, 14 op/s 2024-05-09T08:59:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.241+0000 7fe395cb3640 1 -- 172.21.15.98:0/1800218691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe390107820 msgr2=0x7fe390107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.241+0000 7fe395cb3640 1 --2- 172.21.15.98:0/1800218691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe390107820 0x7fe390107c00 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fe380009930 tx=0x7fe38002a420 comp rx=0 tx=0).stop 2024-05-09T08:59:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.241+0000 7fe395cb3640 1 -- 172.21.15.98:0/1800218691 shutdown_connections 2024-05-09T08:59:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.241+0000 7fe395cb3640 1 --2- 172.21.15.98:0/1800218691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3901021d0 0x7fe39010e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.241+0000 7fe395cb3640 1 --2- 172.21.15.98:0/1800218691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe390101810 0x7fe390101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.241+0000 7fe395cb3640 1 --2- 172.21.15.98:0/1800218691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe390107820 0x7fe390107c00 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.241+0000 7fe395cb3640 1 -- 172.21.15.98:0/1800218691 >> 172.21.15.98:0/1800218691 conn(0x7fe3900fd2e0 msgr2=0x7fe3900ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.241+0000 7fe395cb3640 1 -- 172.21.15.98:0/1800218691 shutdown_connections 2024-05-09T08:59:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.241+0000 7fe395cb3640 1 -- 172.21.15.98:0/1800218691 wait complete. 2024-05-09T08:59:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.242+0000 7fe395cb3640 1 Processor -- start 2024-05-09T08:59:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.242+0000 7fe395cb3640 1 -- start start 2024-05-09T08:59:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.242+0000 7fe395cb3640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe390101810 0x7fe3901980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.242+0000 7fe395cb3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3901021d0 0x7fe390198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.242+0000 7fe395cb3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe390107820 0x7fe39019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.242+0000 7fe395cb3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe390111c30 con 0x7fe390107820 2024-05-09T08:59:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.242+0000 7fe395cb3640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe390111ab0 con 0x7fe3901021d0 2024-05-09T08:59:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.242+0000 7fe395cb3640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe390111db0 con 0x7fe390101810 2024-05-09T08:59:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.242+0000 7fe38ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe390107820 0x7fe39019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.242+0000 7fe38ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe390107820 0x7fe39019c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43078/0 (socket says 172.21.15.98:43078) 2024-05-09T08:59:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.242+0000 7fe38ffff640 1 -- 172.21.15.98:0/1648752052 learned_addr learned my addr 172.21.15.98:0/1648752052 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.243+0000 7fe387fff640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3901021d0 0x7fe390198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.243+0000 7fe38f7fe640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe390101810 0x7fe3901980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.243+0000 7fe38ffff640 1 -- 172.21.15.98:0/1648752052 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe390101810 msgr2=0x7fe3901980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.243+0000 7fe38ffff640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe390101810 0x7fe3901980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.243+0000 7fe38ffff640 1 -- 172.21.15.98:0/1648752052 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3901021d0 msgr2=0x7fe390198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.243+0000 7fe38ffff640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3901021d0 0x7fe390198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.243+0000 7fe38ffff640 1 -- 172.21.15.98:0/1648752052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe39019d070 con 0x7fe390107820 2024-05-09T08:59:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.243+0000 7fe387fff640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3901021d0 0x7fe390198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.243+0000 7fe38f7fe640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe390101810 0x7fe3901980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.243+0000 7fe38ffff640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe390107820 0x7fe39019c940 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fe38800eb10 tx=0x7fe38800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.243+0000 7fe38d7fa640 1 -- 172.21.15.98:0/1648752052 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe388004b20 con 0x7fe390107820 2024-05-09T08:59:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.243+0000 7fe38d7fa640 1 -- 172.21.15.98:0/1648752052 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe388004cc0 con 0x7fe390107820 2024-05-09T08:59:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.243+0000 7fe395cb3640 1 -- 172.21.15.98:0/1648752052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe39019d360 con 0x7fe390107820 2024-05-09T08:59:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.243+0000 7fe38d7fa640 1 -- 172.21.15.98:0/1648752052 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe388010760 con 0x7fe390107820 2024-05-09T08:59:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.244+0000 7fe395cb3640 1 -- 172.21.15.98:0/1648752052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3901a4bf0 con 0x7fe390107820 2024-05-09T08:59:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.246+0000 7fe38d7fa640 1 -- 172.21.15.98:0/1648752052 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe388005020 con 0x7fe390107820 2024-05-09T08:59:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.246+0000 7fe3989c3740 1 Processor -- start 2024-05-09T08:59:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.246+0000 7fe3989c3740 1 -- start start 2024-05-09T08:59:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.246+0000 7fe3989c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab972450 0x55c5ab972830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.246+0000 7fe38d7fa640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe37c0a2e20 0x7fe37c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.246+0000 7fe3989c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab972d70 0x55c5ab97d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.246+0000 7fe3989c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab97d630 0x55c5ab97fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.246+0000 7fe3989c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c5ab8d7340 con 0x55c5ab972d70 2024-05-09T08:59:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.246+0000 7fe3989c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c5ab87f290 con 0x55c5ab972450 2024-05-09T08:59:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.246+0000 7fe3989c3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c5ab8d3300 con 0x55c5ab97d630 2024-05-09T08:59:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe38f7fe640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe37c0a2e20 0x7fe37c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe387fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab972d70 0x55c5ab97d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe387fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab972d70 0x55c5ab97d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43090/0 (socket says 172.21.15.98:43090) 2024-05-09T08:59:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe387fff640 1 -- 172.21.15.98:0/1976126825 learned_addr learned my addr 172.21.15.98:0/1976126825 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe38d7fa640 1 -- 172.21.15.98:0/1648752052 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fe3880cff50 con 0x7fe390107820 2024-05-09T08:59:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe38ffff640 1 --2- 172.21.15.98:0/1976126825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab97d630 0x55c5ab97fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe387fff640 1 --2- 172.21.15.98:0/1976126825 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab972450 0x55c5ab972830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe387fff640 1 -- 172.21.15.98:0/1976126825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab97d630 msgr2=0x55c5ab97fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe387fff640 1 --2- 172.21.15.98:0/1976126825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab97d630 0x55c5ab97fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe387fff640 1 -- 172.21.15.98:0/1976126825 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab972450 msgr2=0x55c5ab972830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe387fff640 1 --2- 172.21.15.98:0/1976126825 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab972450 0x55c5ab972830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe387fff640 1 -- 172.21.15.98:0/1976126825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5ab8a38e0 con 0x55c5ab972d70 2024-05-09T08:59:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe38f7fe640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe37c0a2e20 0x7fe37c0a52e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe380000c00 tx=0x7fe380032c70 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe387fff640 1 --2- 172.21.15.98:0/1976126825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab972d70 0x55c5ab97d0f0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fe378012630 tx=0x7fe378009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb5f8c24ae6835c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe385ffb640 1 -- 172.21.15.98:0/1976126825 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe378020070 con 0x55c5ab972d70 2024-05-09T08:59:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe385ffb640 1 -- 172.21.15.98:0/1976126825 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe37801cc40 con 0x55c5ab972d70 2024-05-09T08:59:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.247+0000 7fe385ffb640 1 -- 172.21.15.98:0/1976126825 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe378010e70 con 0x55c5ab972d70 2024-05-09T08:59:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.248+0000 7fe3989c3740 1 -- 172.21.15.98:0/1976126825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab972d70 msgr2=0x55c5ab97d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.248+0000 7fe3989c3740 1 --2- 172.21.15.98:0/1976126825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab972d70 0x55c5ab97d0f0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fe378012630 tx=0x7fe378009810 comp rx=0 tx=0).stop 2024-05-09T08:59:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.248+0000 7fe3989c3740 1 -- 172.21.15.98:0/1976126825 shutdown_connections 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.248+0000 7fe3989c3740 1 --2- 172.21.15.98:0/1976126825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab97d630 0x55c5ab97fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.248+0000 7fe3989c3740 1 --2- 172.21.15.98:0/1976126825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab972d70 0x55c5ab97d0f0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.248+0000 7fe3989c3740 1 --2- 172.21.15.98:0/1976126825 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab972450 0x55c5ab972830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.248+0000 7fe3989c3740 1 -- 172.21.15.98:0/1976126825 >> 172.21.15.98:0/1976126825 conn(0x55c5ab96e960 msgr2=0x55c5ab96ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.248+0000 7fe3989c3740 1 -- 172.21.15.98:0/1976126825 shutdown_connections 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.248+0000 7fe3989c3740 1 -- 172.21.15.98:0/1976126825 wait complete. 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.248+0000 7fe3989c3740 1 -- start start 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe3989c3740 1 Processor -- start 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe3989c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab972450 0x55c5ab97a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe3989c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab972d70 0x55c5ab97aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe3989c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab97d630 0x55c5ab97b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe3989c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c5ab980260 con 0x55c5ab972450 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe3989c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c5ab892790 con 0x55c5ab97d630 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe3989c3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c5ab9803e0 con 0x55c5ab972d70 2024-05-09T08:59:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe3989c3740 10 client.0 _ll_register_callbacks cb 0x55c5ab8cda20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe38f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab972450 0x55c5ab97a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe38f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab972450 0x55c5ab97a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43102/0 (socket says 172.21.15.98:43102) 2024-05-09T08:59:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe38f7fe640 1 -- 172.21.15.98:0/2127543198 learned_addr learned my addr 172.21.15.98:0/2127543198 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe387fff640 1 --2- 172.21.15.98:0/2127543198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab972d70 0x55c5ab97aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe38ffff640 1 --2- 172.21.15.98:0/2127543198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab97d630 0x55c5ab97b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe38f7fe640 1 -- 172.21.15.98:0/2127543198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab972d70 msgr2=0x55c5ab97aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe38f7fe640 1 --2- 172.21.15.98:0/2127543198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab972d70 0x55c5ab97aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe38f7fe640 1 -- 172.21.15.98:0/2127543198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab97d630 msgr2=0x55c5ab97b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe38f7fe640 1 --2- 172.21.15.98:0/2127543198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab97d630 0x55c5ab97b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.249+0000 7fe38f7fe640 1 -- 172.21.15.98:0/2127543198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5ab5eb210 con 0x55c5ab972450 2024-05-09T08:59:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe38f7fe640 1 --2- 172.21.15.98:0/2127543198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab972450 0x55c5ab97a5b0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fe380031810 tx=0x7fe380009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d4a9ef7a700844 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe36effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe36effd640 1 -- 172.21.15.98:0/2127543198 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38004b020 con 0x55c5ab972450 2024-05-09T08:59:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe36effd640 1 -- 172.21.15.98:0/2127543198 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fe3800037f0 con 0x55c5ab972450 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe3989c3740 1 -- 172.21.15.98:0/2127543198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe37801cc40 con 0x55c5ab972450 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe36effd640 1 -- 172.21.15.98:0/2127543198 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe380056050 con 0x55c5ab972450 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe3989c3740 10 client.18114 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe3989c3740 1 -- 172.21.15.98:0/2127543198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c5ab6b7620 con 0x55c5ab972450 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe36effd640 1 -- 172.21.15.98:0/2127543198 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe37801cc40 con 0x55c5ab972450 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe36effd640 1 -- 172.21.15.98:0/2127543198 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe38003b9f0 con 0x55c5ab972450 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe3989c3740 10 client.18114 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe3989c3740 10 client.18114 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe3989c3740 20 client.18114 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe3989c3740 1 --2- 172.21.15.98:0/2127543198 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c5ab98d3b0 0x55c5ab98f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe3989c3740 4 client.18114 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.250+0000 7fe3989c3740 1 -- 172.21.15.98:0/2127543198 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5ab8e4a70 con 0x55c5ab98d3b0 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.251+0000 7fe387fff640 1 --2- 172.21.15.98:0/2127543198 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c5ab98d3b0 0x55c5ab98f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.251+0000 7fe36effd640 10 client.18114 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.251+0000 7fe387fff640 1 --2- 172.21.15.98:0/2127543198 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c5ab98d3b0 0x55c5ab98f7a0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9652b2adc5846ca server_cookie=859e5d9229ab16ce in_seq=0 out_seq=0 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe36effd640 1 -- 172.21.15.98:0/2127543198 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c5ab8e4a70 con 0x55c5ab98d3b0 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe36effd640 10 client.18114 handle_command_reply: tid=0 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe3989c3740 1 client.18114 shutdown 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe3989c3740 1 -- 172.21.15.98:0/2127543198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab972450 msgr2=0x55c5ab97a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe3989c3740 1 --2- 172.21.15.98:0/2127543198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab972450 0x55c5ab97a5b0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fe380031810 tx=0x7fe380009260 comp rx=0 tx=0).stop 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe3989c3740 1 -- 172.21.15.98:0/2127543198 shutdown_connections 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe3989c3740 1 --2- 172.21.15.98:0/2127543198 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c5ab98d3b0 0x55c5ab98f7a0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe3989c3740 1 --2- 172.21.15.98:0/2127543198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab97d630 0x55c5ab97b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe3989c3740 1 --2- 172.21.15.98:0/2127543198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab972d70 0x55c5ab97aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe3989c3740 1 --2- 172.21.15.98:0/2127543198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab972450 0x55c5ab97a5b0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe3989c3740 1 -- 172.21.15.98:0/2127543198 >> 172.21.15.98:0/2127543198 conn(0x55c5ab96e960 msgr2=0x55c5ab9746e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe36effd640 0 client.18114 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe3989c3740 1 -- 172.21.15.98:0/2127543198 shutdown_connections 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe3989c3740 1 -- 172.21.15.98:0/2127543198 wait complete. 2024-05-09T08:59:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.252+0000 7fe3989c3740 20 client.18114 trim_cache size 0 max 16384 2024-05-09T08:59:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.267+0000 7fe3989c3740 1 Processor -- start 2024-05-09T08:59:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.267+0000 7fe3989c3740 1 -- start start 2024-05-09T08:59:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.267+0000 7fe3989c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab972450 0x55c5ab97cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.267+0000 7fe3989c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab972d70 0x55c5ab989120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.267+0000 7fe3989c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab97d630 0x55c5ab974e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.267+0000 7fe3989c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3901acd30 con 0x55c5ab97d630 2024-05-09T08:59:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe3989c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe3901ae870 con 0x55c5ab972d70 2024-05-09T08:59:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe3989c3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe3901ad400 con 0x55c5ab972450 2024-05-09T08:59:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe38ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab97d630 0x55c5ab974e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe38ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab97d630 0x55c5ab974e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43112/0 (socket says 172.21.15.98:43112) 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe38ffff640 1 -- 172.21.15.98:0/589616422 learned_addr learned my addr 172.21.15.98:0/589616422 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe38ffff640 1 -- 172.21.15.98:0/589616422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab972450 msgr2=0x55c5ab97cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe38f7fe640 1 --2- 172.21.15.98:0/589616422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab972450 0x55c5ab97cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe38ffff640 1 --2- 172.21.15.98:0/589616422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab972450 0x55c5ab97cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe38ffff640 1 -- 172.21.15.98:0/589616422 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab972d70 msgr2=0x55c5ab989120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe38ffff640 1 --2- 172.21.15.98:0/589616422 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab972d70 0x55c5ab989120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe38ffff640 1 -- 172.21.15.98:0/589616422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5ab971c50 con 0x55c5ab97d630 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe38ffff640 1 --2- 172.21.15.98:0/589616422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab97d630 0x55c5ab974e60 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fe3880cd340 tx=0x7fe38809bba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53608f330331d9dd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe3857fa640 1 -- 172.21.15.98:0/589616422 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38809c760 con 0x55c5ab97d630 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe3857fa640 1 -- 172.21.15.98:0/589616422 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3880ca970 con 0x55c5ab97d630 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.268+0000 7fe3857fa640 1 -- 172.21.15.98:0/589616422 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe388016dd0 con 0x55c5ab97d630 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.269+0000 7fe3989c3740 1 -- 172.21.15.98:0/589616422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab97d630 msgr2=0x55c5ab974e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.269+0000 7fe3989c3740 1 --2- 172.21.15.98:0/589616422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab97d630 0x55c5ab974e60 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fe3880cd340 tx=0x7fe38809bba0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.269+0000 7fe3989c3740 1 -- 172.21.15.98:0/589616422 shutdown_connections 2024-05-09T08:59:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.269+0000 7fe3989c3740 1 --2- 172.21.15.98:0/589616422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab97d630 0x55c5ab974e60 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.269+0000 7fe3989c3740 1 --2- 172.21.15.98:0/589616422 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab972d70 0x55c5ab989120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.269+0000 7fe3989c3740 1 --2- 172.21.15.98:0/589616422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab972450 0x55c5ab97cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.269+0000 7fe3989c3740 1 -- 172.21.15.98:0/589616422 >> 172.21.15.98:0/589616422 conn(0x55c5ab96e960 msgr2=0x55c5ab975870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.269+0000 7fe3989c3740 1 -- 172.21.15.98:0/589616422 shutdown_connections 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.269+0000 7fe3989c3740 1 -- 172.21.15.98:0/589616422 wait complete. 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.269+0000 7fe3989c3740 1 -- start start 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.269+0000 7fe3989c3740 1 Processor -- start 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe3989c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab972450 0x55c5ab985970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe3989c3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab972d70 0x55c5ab980d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe3989c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab97d630 0x55c5ab981270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe3989c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3901ae1a0 con 0x55c5ab97d630 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe3989c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe3901adad0 con 0x55c5ab972450 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe3989c3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe3901aad30 con 0x55c5ab972d70 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe3989c3740 10 client.0 _ll_register_callbacks cb 0x7fe378005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe38ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab97d630 0x55c5ab981270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe38ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab97d630 0x55c5ab981270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43118/0 (socket says 172.21.15.98:43118) 2024-05-09T08:59:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe38ffff640 1 -- 172.21.15.98:0/2636155210 learned_addr learned my addr 172.21.15.98:0/2636155210 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe38f7fe640 1 --2- 172.21.15.98:0/2636155210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab972450 0x55c5ab985970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe38ffff640 1 -- 172.21.15.98:0/2636155210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab972d70 msgr2=0x55c5ab980d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe38ffff640 1 --2- 172.21.15.98:0/2636155210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab972d70 0x55c5ab980d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe38ffff640 1 -- 172.21.15.98:0/2636155210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab972450 msgr2=0x55c5ab985970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe38ffff640 1 --2- 172.21.15.98:0/2636155210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab972450 0x55c5ab985970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe38ffff640 1 -- 172.21.15.98:0/2636155210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5ab981a20 con 0x55c5ab97d630 2024-05-09T08:59:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe38ffff640 1 --2- 172.21.15.98:0/2636155210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab97d630 0x55c5ab981270 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fe3880049d0 tx=0x7fe38809c1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e299c49230596d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe36effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe36effd640 1 -- 172.21.15.98:0/2636155210 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38809c760 con 0x55c5ab97d630 2024-05-09T08:59:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe36effd640 1 -- 172.21.15.98:0/2636155210 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fe3880a0110 con 0x55c5ab97d630 2024-05-09T08:59:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe3989c3740 1 -- 172.21.15.98:0/2636155210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3880ca970 con 0x55c5ab97d630 2024-05-09T08:59:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe36effd640 1 -- 172.21.15.98:0/2636155210 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe3880a0410 con 0x55c5ab97d630 2024-05-09T08:59:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe3989c3740 10 client.18120 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.270+0000 7fe3989c3740 1 -- 172.21.15.98:0/2636155210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c5ab9f3870 con 0x55c5ab97d630 2024-05-09T08:59:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.271+0000 7fe36effd640 1 -- 172.21.15.98:0/2636155210 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3880ca970 con 0x55c5ab97d630 2024-05-09T08:59:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.271+0000 7fe36effd640 1 -- 172.21.15.98:0/2636155210 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe3880cabb0 con 0x55c5ab97d630 2024-05-09T08:59:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.271+0000 7fe3989c3740 10 client.18120 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.271+0000 7fe3989c3740 10 client.18120 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.271+0000 7fe3989c3740 20 client.18120 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.271+0000 7fe3989c3740 1 --2- 172.21.15.98:0/2636155210 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c5ab9f3ee0 0x55c5ab9f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.271+0000 7fe3989c3740 4 client.18120 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:59:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.271+0000 7fe3989c3740 1 -- 172.21.15.98:0/2636155210 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c5ab8dedc0 con 0x55c5ab9f3ee0 2024-05-09T08:59:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.271+0000 7fe38f7fe640 1 --2- 172.21.15.98:0/2636155210 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c5ab9f3ee0 0x55c5ab9f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.271+0000 7fe36effd640 10 client.18120 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.271+0000 7fe38f7fe640 1 --2- 172.21.15.98:0/2636155210 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c5ab9f3ee0 0x55c5ab9f62d0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe7052a706c7720c server_cookie=44f2c8c12aea181c in_seq=0 out_seq=0 2024-05-09T08:59:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe36effd640 1 -- 172.21.15.98:0/2636155210 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c5ab8dedc0 con 0x55c5ab9f3ee0 2024-05-09T08:59:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe36effd640 10 client.18120 handle_command_reply: tid=0 2024-05-09T08:59:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe3989c3740 1 client.18120 shutdown 2024-05-09T08:59:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe3989c3740 1 -- 172.21.15.98:0/2636155210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab97d630 msgr2=0x55c5ab981270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe3989c3740 1 --2- 172.21.15.98:0/2636155210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab97d630 0x55c5ab981270 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fe3880049d0 tx=0x7fe38809c1a0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe3989c3740 1 -- 172.21.15.98:0/2636155210 shutdown_connections 2024-05-09T08:59:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe3989c3740 1 --2- 172.21.15.98:0/2636155210 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55c5ab9f3ee0 0x55c5ab9f62d0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe3989c3740 1 --2- 172.21.15.98:0/2636155210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5ab97d630 0x55c5ab981270 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe3989c3740 1 --2- 172.21.15.98:0/2636155210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c5ab972d70 0x55c5ab980d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe36effd640 0 client.18120 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe3989c3740 1 --2- 172.21.15.98:0/2636155210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5ab972450 0x55c5ab985970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe3989c3740 1 -- 172.21.15.98:0/2636155210 >> 172.21.15.98:0/2636155210 conn(0x55c5ab96e960 msgr2=0x55c5ab96d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe3989c3740 1 -- 172.21.15.98:0/2636155210 shutdown_connections 2024-05-09T08:59:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe3989c3740 1 -- 172.21.15.98:0/2636155210 wait complete. 2024-05-09T08:59:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe3989c3740 20 client.18120 trim_cache size 0 max 16384 2024-05-09T08:59:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:59:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:59:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:59:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:59:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe395cb3640 1 -- 172.21.15.98:0/1648752052 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe37c0a2e20 msgr2=0x7fe37c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe395cb3640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe37c0a2e20 0x7fe37c0a52e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe380000c00 tx=0x7fe380032c70 comp rx=0 tx=0).stop 2024-05-09T08:59:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe395cb3640 1 -- 172.21.15.98:0/1648752052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe390107820 msgr2=0x7fe39019c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.276+0000 7fe395cb3640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe390107820 0x7fe39019c940 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fe38800eb10 tx=0x7fe38800c2d0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.277+0000 7fe395cb3640 1 -- 172.21.15.98:0/1648752052 shutdown_connections 2024-05-09T08:59:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.277+0000 7fe395cb3640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe37c0a2e20 0x7fe37c0a52e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.277+0000 7fe395cb3640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe390107820 0x7fe39019c940 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.277+0000 7fe395cb3640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3901021d0 0x7fe390198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.277+0000 7fe395cb3640 1 --2- 172.21.15.98:0/1648752052 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe390101810 0x7fe3901980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.277+0000 7fe395cb3640 1 -- 172.21.15.98:0/1648752052 >> 172.21.15.98:0/1648752052 conn(0x7fe3900fd2e0 msgr2=0x7fe3900fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.277+0000 7fe395cb3640 1 -- 172.21.15.98:0/1648752052 shutdown_connections 2024-05-09T08:59:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:28.277+0000 7fe395cb3640 1 -- 172.21.15.98:0/1648752052 wait complete. 2024-05-09T08:59:28.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:28 smithi098 ceph-mon[30092]: pgmap v275: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 307 B/s wr, 13 op/s 2024-05-09T08:59:28.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:28 smithi142 ceph-mon[34625]: pgmap v275: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 307 B/s wr, 13 op/s 2024-05-09T08:59:28.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:28 smithi105 ceph-mon[34498]: pgmap v275: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 307 B/s wr, 13 op/s 2024-05-09T08:59:30.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:30 smithi142 ceph-mon[34625]: pgmap v276: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 282 B/s wr, 13 op/s 2024-05-09T08:59:30.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:30 smithi105 ceph-mon[34498]: pgmap v276: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 282 B/s wr, 13 op/s 2024-05-09T08:59:30.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:30 smithi098 ceph-mon[30092]: pgmap v276: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 282 B/s wr, 13 op/s 2024-05-09T08:59:32.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:32 smithi142 ceph-mon[34625]: pgmap v277: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 426 B/s wr, 14 op/s 2024-05-09T08:59:32.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:32 smithi105 ceph-mon[34498]: pgmap v277: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 426 B/s wr, 14 op/s 2024-05-09T08:59:32.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:32 smithi098 ceph-mon[30092]: pgmap v277: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 426 B/s wr, 14 op/s 2024-05-09T08:59:34.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:34 smithi142 ceph-mon[34625]: pgmap v278: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-05-09T08:59:34.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:34 smithi105 ceph-mon[34498]: pgmap v278: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-05-09T08:59:34.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:34 smithi098 ceph-mon[30092]: pgmap v278: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-05-09T08:59:36.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:36 smithi142 ceph-mon[34625]: pgmap v279: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:59:36.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:36 smithi105 ceph-mon[34498]: pgmap v279: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:59:36.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:36 smithi098 ceph-mon[30092]: pgmap v279: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:59:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:38 smithi142 ceph-mon[34625]: pgmap v280: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:59:38.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:38 smithi105 ceph-mon[34498]: pgmap v280: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:59:38.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:38 smithi098 ceph-mon[30092]: pgmap v280: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:59:40.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:40 smithi142 ceph-mon[34625]: pgmap v281: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:59:40.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:40 smithi105 ceph-mon[34498]: pgmap v281: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:59:40.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:40 smithi098 ceph-mon[30092]: pgmap v281: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:59:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.373+0000 7f02d7f70640 1 -- 172.21.15.98:0/40968348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02d0106720 msgr2=0x7f02d0106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.373+0000 7f02d7f70640 1 --2- 172.21.15.98:0/40968348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02d0106720 0x7f02d0106b00 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009930 tx=0x7f02cc02a420 comp rx=0 tx=0).stop 2024-05-09T08:59:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.373+0000 7f02d7f70640 1 -- 172.21.15.98:0/40968348 shutdown_connections 2024-05-09T08:59:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.373+0000 7f02d7f70640 1 --2- 172.21.15.98:0/40968348 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02d0101120 0x7f02d010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.373+0000 7f02d7f70640 1 --2- 172.21.15.98:0/40968348 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d0100760 0x7f02d0100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.373+0000 7f02d7f70640 1 --2- 172.21.15.98:0/40968348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02d0106720 0x7f02d0106b00 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.373+0000 7f02d7f70640 1 -- 172.21.15.98:0/40968348 >> 172.21.15.98:0/40968348 conn(0x7f02d00778b0 msgr2=0x7f02d0077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.373+0000 7f02d7f70640 1 -- 172.21.15.98:0/40968348 shutdown_connections 2024-05-09T08:59:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.373+0000 7f02d7f70640 1 -- 172.21.15.98:0/40968348 wait complete. 2024-05-09T08:59:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d7f70640 1 Processor -- start 2024-05-09T08:59:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d7f70640 1 -- start start 2024-05-09T08:59:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d7f70640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02d0100760 0x7f02d0198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d7f70640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d0101120 0x7f02d01986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d7f70640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02d0106720 0x7f02d019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d7f70640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02d0110b30 con 0x7f02d0100760 2024-05-09T08:59:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d7f70640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f02d01109b0 con 0x7f02d0106720 2024-05-09T08:59:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d7f70640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f02d0110cb0 con 0x7f02d0101120 2024-05-09T08:59:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d5d0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02d0100760 0x7f02d0198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d5d0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02d0100760 0x7f02d0198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36294/0 (socket says 172.21.15.98:36294) 2024-05-09T08:59:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d5d0c640 1 -- 172.21.15.98:0/2650606914 learned_addr learned my addr 172.21.15.98:0/2650606914 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d650d640 1 --2- 172.21.15.98:0/2650606914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02d0106720 0x7f02d019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d550b640 1 --2- 172.21.15.98:0/2650606914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d0101120 0x7f02d01986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d5d0c640 1 -- 172.21.15.98:0/2650606914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d0101120 msgr2=0x7f02d01986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d5d0c640 1 --2- 172.21.15.98:0/2650606914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d0101120 0x7f02d01986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.374+0000 7f02d5d0c640 1 -- 172.21.15.98:0/2650606914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02d0106720 msgr2=0x7f02d019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.375+0000 7f02d5d0c640 1 --2- 172.21.15.98:0/2650606914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02d0106720 0x7f02d019ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.375+0000 7f02d5d0c640 1 -- 172.21.15.98:0/2650606914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d019d170 con 0x7f02d0100760 2024-05-09T08:59:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.375+0000 7f02d5d0c640 1 --2- 172.21.15.98:0/2650606914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02d0100760 0x7f02d0198180 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009230 tx=0x7f02cc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.375+0000 7f02c6ffd640 1 -- 172.21.15.98:0/2650606914 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02cc03f030 con 0x7f02d0100760 2024-05-09T08:59:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.375+0000 7f02d7f70640 1 -- 172.21.15.98:0/2650606914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02d019d400 con 0x7f02d0100760 2024-05-09T08:59:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.375+0000 7f02c6ffd640 1 -- 172.21.15.98:0/2650606914 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02cc003ad0 con 0x7f02d0100760 2024-05-09T08:59:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.375+0000 7f02c6ffd640 1 -- 172.21.15.98:0/2650606914 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02cc035070 con 0x7f02d0100760 2024-05-09T08:59:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.375+0000 7f02d7f70640 1 -- 172.21.15.98:0/2650606914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02d01a4cf0 con 0x7f02d0100760 2024-05-09T08:59:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.377+0000 7f02c6ffd640 1 -- 172.21.15.98:0/2650606914 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f02cc042030 con 0x7f02d0100760 2024-05-09T08:59:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02dac84740 1 Processor -- start 2024-05-09T08:59:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02dac84740 1 -- start start 2024-05-09T08:59:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02c6ffd640 1 --2- 172.21.15.98:0/2650606914 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f02bc0a2da0 0x7f02bc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02dac84740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9a2450 0x55cbac9a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02dac84740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2d70 0x55cbac9ad0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02dac84740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9ad630 0x55cbac9afa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02dac84740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cbac907340 con 0x55cbac9ad630 2024-05-09T08:59:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02dac84740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cbac8af290 con 0x55cbac9a2d70 2024-05-09T08:59:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02dac84740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cbac903300 con 0x55cbac9a2450 2024-05-09T08:59:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02d650d640 1 --2- 172.21.15.98:0/2650606914 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f02bc0a2da0 0x7f02bc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02d550b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2d70 0x55cbac9ad0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02d550b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2d70 0x55cbac9ad0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:48652/0 (socket says 172.21.15.98:48652) 2024-05-09T08:59:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02c6ffd640 1 -- 172.21.15.98:0/2650606914 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f02cc0efcb0 con 0x7f02d0100760 2024-05-09T08:59:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02d650d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9ad630 0x55cbac9afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.378+0000 7f02d550b640 1 -- 172.21.15.98:0/1264862201 learned_addr learned my addr 172.21.15.98:0/1264862201 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.379+0000 7f02d5d0c640 1 --2- 172.21.15.98:0/1264862201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9a2450 0x55cbac9a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.379+0000 7f02d550b640 1 -- 172.21.15.98:0/1264862201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9a2450 msgr2=0x55cbac9a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.379+0000 7f02d550b640 1 --2- 172.21.15.98:0/1264862201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9a2450 0x55cbac9a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.379+0000 7f02d550b640 1 -- 172.21.15.98:0/1264862201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9ad630 msgr2=0x55cbac9afa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.379+0000 7f02d550b640 1 --2- 172.21.15.98:0/1264862201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9ad630 0x55cbac9afa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.379+0000 7f02d550b640 1 -- 172.21.15.98:0/1264862201 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbac8d38e0 con 0x55cbac9a2d70 2024-05-09T08:59:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.379+0000 7f02d5d0c640 1 --2- 172.21.15.98:0/1264862201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9a2450 0x55cbac9a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.379+0000 7f02d650d640 1 --2- 172.21.15.98:0/2650606914 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f02bc0a2da0 0x7f02bc0a5260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f02c8009fd0 tx=0x7f02c8009da0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.379+0000 7f02d550b640 1 --2- 172.21.15.98:0/1264862201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2d70 0x55cbac9ad0f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f02c000b740 tx=0x7f02c000bc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f434f990b309238c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.379+0000 7f02bbfff640 1 -- 172.21.15.98:0/1264862201 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0003dd0 con 0x55cbac9a2d70 2024-05-09T08:59:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.379+0000 7f02bbfff640 1 -- 172.21.15.98:0/1264862201 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02c0015070 con 0x55cbac9a2d70 2024-05-09T08:59:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.379+0000 7f02bbfff640 1 -- 172.21.15.98:0/1264862201 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0004dc0 con 0x55cbac9a2d70 2024-05-09T08:59:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.380+0000 7f02dac84740 1 -- 172.21.15.98:0/1264862201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2d70 msgr2=0x55cbac9ad0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.380+0000 7f02dac84740 1 --2- 172.21.15.98:0/1264862201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2d70 0x55cbac9ad0f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f02c000b740 tx=0x7f02c000bc00 comp rx=0 tx=0).stop 2024-05-09T08:59:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.380+0000 7f02dac84740 1 -- 172.21.15.98:0/1264862201 shutdown_connections 2024-05-09T08:59:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.380+0000 7f02dac84740 1 --2- 172.21.15.98:0/1264862201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9ad630 0x55cbac9afa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.380+0000 7f02dac84740 1 --2- 172.21.15.98:0/1264862201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2d70 0x55cbac9ad0f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.380+0000 7f02dac84740 1 --2- 172.21.15.98:0/1264862201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9a2450 0x55cbac9a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.380+0000 7f02dac84740 1 -- 172.21.15.98:0/1264862201 >> 172.21.15.98:0/1264862201 conn(0x55cbac99e960 msgr2=0x55cbac99ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.380+0000 7f02dac84740 1 -- 172.21.15.98:0/1264862201 shutdown_connections 2024-05-09T08:59:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.380+0000 7f02dac84740 1 -- 172.21.15.98:0/1264862201 wait complete. 2024-05-09T08:59:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.380+0000 7f02dac84740 1 -- start start 2024-05-09T08:59:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02dac84740 1 Processor -- start 2024-05-09T08:59:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02dac84740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 0x55cbac9aa540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02dac84740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9aaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02dac84740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 0x55cbac9aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02dac84740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cbac9b0260 con 0x55cbac9a2d70 2024-05-09T08:59:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02dac84740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cbac8c2790 con 0x55cbac9a2450 2024-05-09T08:59:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02dac84740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cbac9b03e0 con 0x55cbac9ad630 2024-05-09T08:59:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02dac84740 10 client.0 _ll_register_callbacks cb 0x55cbac8fda20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02d550b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9aaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02d5d0c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 0x55cbac9aa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02d550b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9aaa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36316/0 (socket says 172.21.15.98:36316) 2024-05-09T08:59:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02d5d0c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 0x55cbac9aa540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:48668/0 (socket says 172.21.15.98:48668) 2024-05-09T08:59:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02d550b640 1 -- 172.21.15.98:0/254425229 learned_addr learned my addr 172.21.15.98:0/254425229 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02d650d640 1 --2- 172.21.15.98:0/254425229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 0x55cbac9aafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02d550b640 1 -- 172.21.15.98:0/254425229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 msgr2=0x55cbac9aafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02d550b640 1 --2- 172.21.15.98:0/254425229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 0x55cbac9aafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02d550b640 1 -- 172.21.15.98:0/254425229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 msgr2=0x55cbac9aa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02d550b640 1 --2- 172.21.15.98:0/254425229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 0x55cbac9aa540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.381+0000 7f02d550b640 1 -- 172.21.15.98:0/254425229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbac61b210 con 0x55cbac9a2d70 2024-05-09T08:59:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02d5d0c640 1 --2- 172.21.15.98:0/254425229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 0x55cbac9aa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02d550b640 1 --2- 172.21.15.98:0/254425229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9aaa80 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f02c00057a0 tx=0x7f02c000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaecd10b97700c57 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02b97fa640 1 -- 172.21.15.98:0/254425229 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0028060 con 0x55cbac9a2d70 2024-05-09T08:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02b97fa640 1 -- 172.21.15.98:0/254425229 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f02c0004b40 con 0x55cbac9a2d70 2024-05-09T08:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02dac84740 1 -- 172.21.15.98:0/254425229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02c0015070 con 0x55cbac9a2d70 2024-05-09T08:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02b97fa640 1 -- 172.21.15.98:0/254425229 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f02c002f050 con 0x55cbac9a2d70 2024-05-09T08:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02dac84740 10 client.18132 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02dac84740 1 -- 172.21.15.98:0/254425229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cbac6e7620 con 0x55cbac9a2d70 2024-05-09T08:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02b97fa640 1 -- 172.21.15.98:0/254425229 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02c0015070 con 0x55cbac9a2d70 2024-05-09T08:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02b97fa640 1 -- 172.21.15.98:0/254425229 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f02c0022030 con 0x55cbac9a2d70 2024-05-09T08:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02dac84740 10 client.18132 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02dac84740 10 client.18132 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02dac84740 20 client.18132 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.382+0000 7f02dac84740 1 --2- 172.21.15.98:0/254425229 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cbac9bd240 0x55cbac9bf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.383+0000 7f02dac84740 4 client.18132 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.383+0000 7f02dac84740 1 -- 172.21.15.98:0/254425229 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbac914a70 con 0x55cbac9bd240 2024-05-09T08:59:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.383+0000 7f02d5d0c640 1 --2- 172.21.15.98:0/254425229 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cbac9bd240 0x55cbac9bf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.383+0000 7f02b97fa640 10 client.18132 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.383+0000 7f02d5d0c640 1 --2- 172.21.15.98:0/254425229 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cbac9bd240 0x55cbac9bf630 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe1b963611db260b server_cookie=eedc1a70ad2afd2a in_seq=0 out_seq=0 2024-05-09T08:59:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02b97fa640 1 -- 172.21.15.98:0/254425229 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cbac914a70 con 0x55cbac9bd240 2024-05-09T08:59:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02b97fa640 10 client.18132 handle_command_reply: tid=0 2024-05-09T08:59:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02dac84740 1 client.18132 shutdown 2024-05-09T08:59:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02dac84740 1 -- 172.21.15.98:0/254425229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 msgr2=0x55cbac9aaa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02dac84740 1 --2- 172.21.15.98:0/254425229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9aaa80 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f02c00057a0 tx=0x7f02c000bf90 comp rx=0 tx=0).stop 2024-05-09T08:59:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02dac84740 1 -- 172.21.15.98:0/254425229 shutdown_connections 2024-05-09T08:59:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02dac84740 1 --2- 172.21.15.98:0/254425229 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cbac9bd240 0x55cbac9bf630 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02dac84740 1 --2- 172.21.15.98:0/254425229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 0x55cbac9aafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02dac84740 1 --2- 172.21.15.98:0/254425229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9aaa80 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02b97fa640 0 client.18132 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02dac84740 1 --2- 172.21.15.98:0/254425229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 0x55cbac9aa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02dac84740 1 -- 172.21.15.98:0/254425229 >> 172.21.15.98:0/254425229 conn(0x55cbac99e960 msgr2=0x55cbac9a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02dac84740 1 -- 172.21.15.98:0/254425229 shutdown_connections 2024-05-09T08:59:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02dac84740 1 -- 172.21.15.98:0/254425229 wait complete. 2024-05-09T08:59:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.384+0000 7f02dac84740 20 client.18132 trim_cache size 0 max 16384 2024-05-09T08:59:41.400 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.399+0000 7f02dac84740 1 Processor -- start 2024-05-09T08:59:41.400 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.399+0000 7f02dac84740 1 -- start start 2024-05-09T08:59:41.400 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.399+0000 7f02dac84740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 0x55cbac9b9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.399+0000 7f02dac84740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9ac370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.399+0000 7f02dac84740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 0x55cbac9c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.399+0000 7f02dac84740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02d01aedf0 con 0x55cbac9a2d70 2024-05-09T08:59:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.399+0000 7f02dac84740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f02d01b0930 con 0x55cbac9a2450 2024-05-09T08:59:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.399+0000 7f02dac84740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f02d01af4c0 con 0x55cbac9ad630 2024-05-09T08:59:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.399+0000 7f02d550b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9ac370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02d550b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9ac370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36320/0 (socket says 172.21.15.98:36320) 2024-05-09T08:59:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02d550b640 1 -- 172.21.15.98:0/810642425 learned_addr learned my addr 172.21.15.98:0/810642425 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02d5d0c640 1 --2- 172.21.15.98:0/810642425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 0x55cbac9b9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02d650d640 1 --2- 172.21.15.98:0/810642425 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 0x55cbac9c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02d550b640 1 -- 172.21.15.98:0/810642425 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 msgr2=0x55cbac9c4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02d550b640 1 --2- 172.21.15.98:0/810642425 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 0x55cbac9c4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02d550b640 1 -- 172.21.15.98:0/810642425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 msgr2=0x55cbac9b9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02d550b640 1 --2- 172.21.15.98:0/810642425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 0x55cbac9b9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02d550b640 1 -- 172.21.15.98:0/810642425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbac99cf00 con 0x55cbac9a2d70 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02d650d640 1 --2- 172.21.15.98:0/810642425 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 0x55cbac9c4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02d5d0c640 1 --2- 172.21.15.98:0/810642425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 0x55cbac9b9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02d550b640 1 --2- 172.21.15.98:0/810642425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9ac370 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f02c0005770 tx=0x7f02c0003af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d47c1458ec1768 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02bb7fe640 1 -- 172.21.15.98:0/810642425 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0028070 con 0x55cbac9a2d70 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02bb7fe640 1 -- 172.21.15.98:0/810642425 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02c0015070 con 0x55cbac9a2d70 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.400+0000 7f02bb7fe640 1 -- 172.21.15.98:0/810642425 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0019900 con 0x55cbac9a2d70 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.401+0000 7f02dac84740 1 -- 172.21.15.98:0/810642425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 msgr2=0x55cbac9ac370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.401+0000 7f02dac84740 1 --2- 172.21.15.98:0/810642425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9ac370 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f02c0005770 tx=0x7f02c0003af0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.401+0000 7f02dac84740 1 -- 172.21.15.98:0/810642425 shutdown_connections 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.401+0000 7f02dac84740 1 --2- 172.21.15.98:0/810642425 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 0x55cbac9c4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.401+0000 7f02dac84740 1 --2- 172.21.15.98:0/810642425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9ac370 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.401+0000 7f02dac84740 1 --2- 172.21.15.98:0/810642425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 0x55cbac9b9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.401+0000 7f02dac84740 1 -- 172.21.15.98:0/810642425 >> 172.21.15.98:0/810642425 conn(0x55cbac99e960 msgr2=0x55cbac9abc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.401+0000 7f02dac84740 1 -- 172.21.15.98:0/810642425 shutdown_connections 2024-05-09T08:59:41.403 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.401+0000 7f02dac84740 1 -- 172.21.15.98:0/810642425 wait complete. 2024-05-09T08:59:41.403 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.401+0000 7f02dac84740 1 -- start start 2024-05-09T08:59:41.403 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.401+0000 7f02dac84740 1 Processor -- start 2024-05-09T08:59:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02dac84740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 0x55cbac9a70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02dac84740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9a7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02dac84740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 0x55cbac9a7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02dac84740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02d01b0260 con 0x55cbac9a2d70 2024-05-09T08:59:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02dac84740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f02d01afb90 con 0x55cbac9a2450 2024-05-09T08:59:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02dac84740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f02d01acdf0 con 0x55cbac9ad630 2024-05-09T08:59:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02dac84740 10 client.0 _ll_register_callbacks cb 0x7f02c80126f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02d550b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9a7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02d550b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9a7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36322/0 (socket says 172.21.15.98:36322) 2024-05-09T08:59:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02d550b640 1 -- 172.21.15.98:0/1941795242 learned_addr learned my addr 172.21.15.98:0/1941795242 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02d550b640 1 -- 172.21.15.98:0/1941795242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 msgr2=0x55cbac9a7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02d550b640 1 --2- 172.21.15.98:0/1941795242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 0x55cbac9a7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02d550b640 1 -- 172.21.15.98:0/1941795242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 msgr2=0x55cbac9a70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02d550b640 1 --2- 172.21.15.98:0/1941795242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 0x55cbac9a70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02d550b640 1 -- 172.21.15.98:0/1941795242 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbaca22470 con 0x55cbac9a2d70 2024-05-09T08:59:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02d550b640 1 --2- 172.21.15.98:0/1941795242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9a7630 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f02c0003ba0 tx=0x7f02c0005ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ceb0dec24126b2e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02b97fa640 1 -- 172.21.15.98:0/1941795242 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0028070 con 0x55cbac9a2d70 2024-05-09T08:59:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02b97fa640 1 -- 172.21.15.98:0/1941795242 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f02c0011440 con 0x55cbac9a2d70 2024-05-09T08:59:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02dac84740 1 -- 172.21.15.98:0/1941795242 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02c0015070 con 0x55cbac9a2d70 2024-05-09T08:59:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02b97fa640 1 -- 172.21.15.98:0/1941795242 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f02c0011e60 con 0x55cbac9a2d70 2024-05-09T08:59:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02dac84740 10 client.18138 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.402+0000 7f02dac84740 1 -- 172.21.15.98:0/1941795242 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cbaca23630 con 0x55cbac9a2d70 2024-05-09T08:59:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.403+0000 7f02b97fa640 1 -- 172.21.15.98:0/1941795242 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02c0015070 con 0x55cbac9a2d70 2024-05-09T08:59:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.403+0000 7f02b97fa640 1 -- 172.21.15.98:0/1941795242 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f02c0022040 con 0x55cbac9a2d70 2024-05-09T08:59:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.403+0000 7f02dac84740 10 client.18138 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.403+0000 7f02dac84740 10 client.18138 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:41.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.403+0000 7f02dac84740 20 client.18138 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:41.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.403+0000 7f02dac84740 1 --2- 172.21.15.98:0/1941795242 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cbaca23d60 0x55cbaca261c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:41.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.403+0000 7f02dac84740 4 client.18138 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:59:41.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.403+0000 7f02dac84740 1 -- 172.21.15.98:0/1941795242 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cbac90edc0 con 0x55cbaca23d60 2024-05-09T08:59:41.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.403+0000 7f02d5d0c640 1 --2- 172.21.15.98:0/1941795242 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cbaca23d60 0x55cbaca261c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:41.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.403+0000 7f02d5d0c640 1 --2- 172.21.15.98:0/1941795242 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cbaca23d60 0x55cbaca261c0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d337fc183432d1bb server_cookie=792f8b16dbecbe33 in_seq=0 out_seq=0 2024-05-09T08:59:41.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.403+0000 7f02b97fa640 10 client.18138 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.407+0000 7f02b97fa640 1 -- 172.21.15.98:0/1941795242 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cbac90edc0 con 0x55cbaca23d60 2024-05-09T08:59:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.407+0000 7f02b97fa640 10 client.18138 handle_command_reply: tid=0 2024-05-09T08:59:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.407+0000 7f02dac84740 1 client.18138 shutdown 2024-05-09T08:59:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02dac84740 1 -- 172.21.15.98:0/1941795242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 msgr2=0x55cbac9a7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02dac84740 1 --2- 172.21.15.98:0/1941795242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9a7630 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f02c0003ba0 tx=0x7f02c0005ec0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02dac84740 1 -- 172.21.15.98:0/1941795242 shutdown_connections 2024-05-09T08:59:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02dac84740 1 --2- 172.21.15.98:0/1941795242 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cbaca23d60 0x55cbaca261c0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02dac84740 1 --2- 172.21.15.98:0/1941795242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbac9ad630 0x55cbac9a7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02dac84740 1 --2- 172.21.15.98:0/1941795242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cbac9a2d70 0x55cbac9a7630 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02b97fa640 0 client.18138 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02dac84740 1 --2- 172.21.15.98:0/1941795242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbac9a2450 0x55cbac9a70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02dac84740 1 -- 172.21.15.98:0/1941795242 >> 172.21.15.98:0/1941795242 conn(0x55cbac99e960 msgr2=0x55cbac99da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02dac84740 1 -- 172.21.15.98:0/1941795242 shutdown_connections 2024-05-09T08:59:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02dac84740 1 -- 172.21.15.98:0/1941795242 wait complete. 2024-05-09T08:59:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02dac84740 20 client.18138 trim_cache size 0 max 16384 2024-05-09T08:59:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:59:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:59:41.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:59:41.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:59:41.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02d7f70640 1 -- 172.21.15.98:0/2650606914 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f02bc0a2da0 msgr2=0x7f02bc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:41.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02d7f70640 1 --2- 172.21.15.98:0/2650606914 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f02bc0a2da0 0x7f02bc0a5260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f02c8009fd0 tx=0x7f02c8009da0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02d7f70640 1 -- 172.21.15.98:0/2650606914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02d0100760 msgr2=0x7f02d0198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02d7f70640 1 --2- 172.21.15.98:0/2650606914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02d0100760 0x7f02d0198180 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009230 tx=0x7f02cc004270 comp rx=0 tx=0).stop 2024-05-09T08:59:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02d7f70640 1 -- 172.21.15.98:0/2650606914 shutdown_connections 2024-05-09T08:59:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02d7f70640 1 --2- 172.21.15.98:0/2650606914 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f02bc0a2da0 0x7f02bc0a5260 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02d7f70640 1 --2- 172.21.15.98:0/2650606914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02d0106720 0x7f02d019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02d7f70640 1 --2- 172.21.15.98:0/2650606914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d0101120 0x7f02d01986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02d7f70640 1 --2- 172.21.15.98:0/2650606914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02d0100760 0x7f02d0198180 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.408+0000 7f02d7f70640 1 -- 172.21.15.98:0/2650606914 >> 172.21.15.98:0/2650606914 conn(0x7f02d00778b0 msgr2=0x7f02d00692a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.409+0000 7f02d7f70640 1 -- 172.21.15.98:0/2650606914 shutdown_connections 2024-05-09T08:59:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:41.409+0000 7f02d7f70640 1 -- 172.21.15.98:0/2650606914 wait complete. 2024-05-09T08:59:42.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:42 smithi142 ceph-mon[34625]: pgmap v282: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:59:42.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:42 smithi105 ceph-mon[34498]: pgmap v282: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:59:42.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:42 smithi098 ceph-mon[30092]: pgmap v282: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:59:44.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:44 smithi142 ceph-mon[34625]: pgmap v283: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-09T08:59:44.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:44 smithi105 ceph-mon[34498]: pgmap v283: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-09T08:59:44.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:44 smithi098 ceph-mon[30092]: pgmap v283: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-09T08:59:46.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:46 smithi142 ceph-mon[34625]: pgmap v284: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-09T08:59:46.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:46 smithi105 ceph-mon[34498]: pgmap v284: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-09T08:59:46.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:46 smithi098 ceph-mon[30092]: pgmap v284: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-09T08:59:48.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:48 smithi142 ceph-mon[34625]: pgmap v285: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:59:48.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:48 smithi105 ceph-mon[34498]: pgmap v285: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:59:48.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:48 smithi098 ceph-mon[30092]: pgmap v285: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:59:49.191 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:59:49.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:59:49.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.307+0000 7fdfd29e7640 1 -- 172.21.15.98:0/1495440032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfcc1010d0 msgr2=0x7fdfcc10d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.307+0000 7fdfd29e7640 1 --2- 172.21.15.98:0/1495440032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfcc1010d0 0x7fdfcc10d5e0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009cd0 tx=0x7fdfc402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.307+0000 7fdfd29e7640 1 -- 172.21.15.98:0/1495440032 shutdown_connections 2024-05-09T08:59:49.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.307+0000 7fdfd29e7640 1 --2- 172.21.15.98:0/1495440032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfcc1010d0 0x7fdfcc10d5e0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.307+0000 7fdfd29e7640 1 --2- 172.21.15.98:0/1495440032 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfcc100710 0x7fdfcc100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.307+0000 7fdfd29e7640 1 --2- 172.21.15.98:0/1495440032 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfcc106720 0x7fdfcc106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.307+0000 7fdfd29e7640 1 -- 172.21.15.98:0/1495440032 >> 172.21.15.98:0/1495440032 conn(0x7fdfcc077790 msgr2=0x7fdfcc077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.307+0000 7fdfd29e7640 1 -- 172.21.15.98:0/1495440032 shutdown_connections 2024-05-09T08:59:49.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.307+0000 7fdfd29e7640 1 -- 172.21.15.98:0/1495440032 wait complete. 2024-05-09T08:59:49.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.308+0000 7fdfd29e7640 1 Processor -- start 2024-05-09T08:59:49.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.308+0000 7fdfd29e7640 1 -- start start 2024-05-09T08:59:49.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.308+0000 7fdfd29e7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfcc100710 0x7fdfcc1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.308+0000 7fdfd29e7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfcc1010d0 0x7fdfcc198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.308+0000 7fdfd29e7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfcc106720 0x7fdfcc19c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.308+0000 7fdfd29e7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc111c30 con 0x7fdfcc100710 2024-05-09T08:59:49.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.308+0000 7fdfd29e7640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc111ab0 con 0x7fdfcc1010d0 2024-05-09T08:59:49.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.308+0000 7fdfd29e7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc111db0 con 0x7fdfcc106720 2024-05-09T08:59:49.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.308+0000 7fdfcbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfcc100710 0x7fdfcc1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.308+0000 7fdfcbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfcc100710 0x7fdfcc1980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36724/0 (socket says 172.21.15.98:36724) 2024-05-09T08:59:49.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.308+0000 7fdfcbfff640 1 -- 172.21.15.98:0/3589625451 learned_addr learned my addr 172.21.15.98:0/3589625451 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.308+0000 7fdfcb7fe640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfcc1010d0 0x7fdfcc198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.308+0000 7fdfd0f84640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfcc106720 0x7fdfcc19c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.309+0000 7fdfcbfff640 1 -- 172.21.15.98:0/3589625451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfcc106720 msgr2=0x7fdfcc19c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.309+0000 7fdfcbfff640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfcc106720 0x7fdfcc19c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.309+0000 7fdfcbfff640 1 -- 172.21.15.98:0/3589625451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfcc1010d0 msgr2=0x7fdfcc198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.309+0000 7fdfcbfff640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfcc1010d0 0x7fdfcc198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.309+0000 7fdfcbfff640 1 -- 172.21.15.98:0/3589625451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfcc19d0c0 con 0x7fdfcc100710 2024-05-09T08:59:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.309+0000 7fdfcb7fe640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfcc1010d0 0x7fdfcc198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:49.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.309+0000 7fdfcbfff640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfcc100710 0x7fdfcc1980d0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00d770 tx=0x7fdfbc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.309+0000 7fdfc97fa640 1 -- 172.21.15.98:0/3589625451 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc017070 con 0x7fdfcc100710 2024-05-09T08:59:49.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.309+0000 7fdfc97fa640 1 -- 172.21.15.98:0/3589625451 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfbc005500 con 0x7fdfcc100710 2024-05-09T08:59:49.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.309+0000 7fdfd29e7640 1 -- 172.21.15.98:0/3589625451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfcc19d3b0 con 0x7fdfcc100710 2024-05-09T08:59:49.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.309+0000 7fdfc97fa640 1 -- 172.21.15.98:0/3589625451 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc004e00 con 0x7fdfcc100710 2024-05-09T08:59:49.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.309+0000 7fdfd29e7640 1 -- 172.21.15.98:0/3589625451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfcc1a4c40 con 0x7fdfcc100710 2024-05-09T08:59:49.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.311+0000 7fdfc97fa640 1 -- 172.21.15.98:0/3589625451 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fdfbc004fa0 con 0x7fdfcc100710 2024-05-09T08:59:49.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfd57d4740 1 Processor -- start 2024-05-09T08:59:49.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfd57d4740 1 -- start start 2024-05-09T08:59:49.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfd57d4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c25280 0x559d63c25660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfd57d4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c25ba0 0x559d63c2fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfd57d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c30270 0x559d63c32660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfd57d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559d63b8ad00 con 0x559d63c30270 2024-05-09T08:59:49.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfd57d4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559d63b324c0 con 0x559d63c25280 2024-05-09T08:59:49.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfd57d4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559d63b86cc0 con 0x559d63c25ba0 2024-05-09T08:59:49.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfc97fa640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdfb40a2d00 0x7fdfb40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfcb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c30270 0x559d63c32660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfcb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c30270 0x559d63c32660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36740/0 (socket says 172.21.15.98:36740) 2024-05-09T08:59:49.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfcb7fe640 1 -- 172.21.15.98:0/815252870 learned_addr learned my addr 172.21.15.98:0/815252870 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfcbfff640 1 --2- 172.21.15.98:0/815252870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c25ba0 0x559d63c2fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfd0f84640 1 --2- 172.21.15.98:0/815252870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c25280 0x559d63c25660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfd0f84640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdfb40a2d00 0x7fdfb40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfc97fa640 1 -- 172.21.15.98:0/3589625451 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fdfbc0cf2d0 con 0x7fdfcc100710 2024-05-09T08:59:49.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfcb7fe640 1 -- 172.21.15.98:0/815252870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c25ba0 msgr2=0x559d63c2fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfcb7fe640 1 --2- 172.21.15.98:0/815252870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c25ba0 0x559d63c2fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfcb7fe640 1 -- 172.21.15.98:0/815252870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c25280 msgr2=0x559d63c25660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfcb7fe640 1 --2- 172.21.15.98:0/815252870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c25280 0x559d63c25660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.312+0000 7fdfcb7fe640 1 -- 172.21.15.98:0/815252870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d639687d0 con 0x559d63c30270 2024-05-09T08:59:49.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.313+0000 7fdfcb7fe640 1 --2- 172.21.15.98:0/815252870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c30270 0x559d63c32660 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000bd30 tx=0x7fdfc000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7942523afd41baae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.313+0000 7fdfd0f84640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdfb40a2d00 0x7fdfb40a51c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4004af0 tx=0x7fdfc40095c0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.313+0000 7fdfb27fc640 1 -- 172.21.15.98:0/815252870 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0015070 con 0x559d63c30270 2024-05-09T08:59:49.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.313+0000 7fdfb27fc640 1 -- 172.21.15.98:0/815252870 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc000be20 con 0x559d63c30270 2024-05-09T08:59:49.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.313+0000 7fdfb27fc640 1 -- 172.21.15.98:0/815252870 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0011a10 con 0x559d63c30270 2024-05-09T08:59:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.313+0000 7fdfd57d4740 1 -- 172.21.15.98:0/815252870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c30270 msgr2=0x559d63c32660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.313+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/815252870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c30270 0x559d63c32660 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000bd30 tx=0x7fdfc000f730 comp rx=0 tx=0).stop 2024-05-09T08:59:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 -- 172.21.15.98:0/815252870 shutdown_connections 2024-05-09T08:59:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/815252870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c30270 0x559d63c32660 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/815252870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c25ba0 0x559d63c2fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/815252870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c25280 0x559d63c25660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 -- 172.21.15.98:0/815252870 >> 172.21.15.98:0/815252870 conn(0x559d63c215a0 msgr2=0x559d63c21980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 -- 172.21.15.98:0/815252870 shutdown_connections 2024-05-09T08:59:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 -- 172.21.15.98:0/815252870 wait complete. 2024-05-09T08:59:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 -- start start 2024-05-09T08:59:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 Processor -- start 2024-05-09T08:59:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c25280 0x559d63c2d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c25ba0 0x559d63c2d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c30270 0x559d63c2dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559d63c33060 con 0x559d63c30270 2024-05-09T08:59:49.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559d63b45b90 con 0x559d63c25280 2024-05-09T08:59:49.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559d63c331e0 con 0x559d63c25ba0 2024-05-09T08:59:49.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.314+0000 7fdfd57d4740 10 client.0 _ll_register_callbacks cb 0x559d63b81da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:49.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfd0f84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c30270 0x559d63c2dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfd0f84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c30270 0x559d63c2dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36752/0 (socket says 172.21.15.98:36752) 2024-05-09T08:59:49.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfd0f84640 1 -- 172.21.15.98:0/419010565 learned_addr learned my addr 172.21.15.98:0/419010565 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfcbfff640 1 --2- 172.21.15.98:0/419010565 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c25280 0x559d63c2d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfcb7fe640 1 --2- 172.21.15.98:0/419010565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c25ba0 0x559d63c2d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfd0f84640 1 -- 172.21.15.98:0/419010565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c25ba0 msgr2=0x559d63c2d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfd0f84640 1 --2- 172.21.15.98:0/419010565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c25ba0 0x559d63c2d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfd0f84640 1 -- 172.21.15.98:0/419010565 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c25280 msgr2=0x559d63c2d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfd0f84640 1 --2- 172.21.15.98:0/419010565 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c25280 0x559d63c2d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfd0f84640 1 -- 172.21.15.98:0/419010565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d6389e3f0 con 0x559d63c30270 2024-05-09T08:59:49.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfd0f84640 1 --2- 172.21.15.98:0/419010565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c30270 0x559d63c2dc40 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009cd0 tx=0x7fdfc4009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcdaa042f7376b46 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:49.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfab7fe640 1 -- 172.21.15.98:0/419010565 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4051070 con 0x559d63c30270 2024-05-09T08:59:49.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfab7fe640 1 -- 172.21.15.98:0/419010565 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fdfc4004490 con 0x559d63c30270 2024-05-09T08:59:49.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfd57d4740 1 -- 172.21.15.98:0/419010565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfc000be20 con 0x559d63c30270 2024-05-09T08:59:49.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfab7fe640 1 -- 172.21.15.98:0/419010565 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fdfc4004690 con 0x559d63c30270 2024-05-09T08:59:49.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfd57d4740 10 client.18150 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:49.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.315+0000 7fdfd57d4740 1 -- 172.21.15.98:0/419010565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559d6385e0f0 con 0x559d63c30270 2024-05-09T08:59:49.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.316+0000 7fdfab7fe640 1 -- 172.21.15.98:0/419010565 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc000be20 con 0x559d63c30270 2024-05-09T08:59:49.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.316+0000 7fdfab7fe640 1 -- 172.21.15.98:0/419010565 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fdfc4049040 con 0x559d63c30270 2024-05-09T08:59:49.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.316+0000 7fdfd57d4740 10 client.18150 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:49.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.316+0000 7fdfd57d4740 10 client.18150 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:49.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.316+0000 7fdfd57d4740 20 client.18150 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:49.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.316+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/419010565 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d63c40540 0x559d63c42990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.316+0000 7fdfd57d4740 4 client.18150 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:49.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.316+0000 7fdfd57d4740 1 -- 172.21.15.98:0/419010565 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d63b98430 con 0x559d63c40540 2024-05-09T08:59:49.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.316+0000 7fdfcbfff640 1 --2- 172.21.15.98:0/419010565 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d63c40540 0x559d63c42990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.317+0000 7fdfab7fe640 10 client.18150 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:49.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.317+0000 7fdfcbfff640 1 --2- 172.21.15.98:0/419010565 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d63c40540 0x559d63c42990 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27ab2e0a900b5a86 server_cookie=58e188f51a48daf in_seq=0 out_seq=0 2024-05-09T08:59:49.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.317+0000 7fdfab7fe640 1 -- 172.21.15.98:0/419010565 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559d63b98430 con 0x559d63c40540 2024-05-09T08:59:49.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.317+0000 7fdfab7fe640 10 client.18150 handle_command_reply: tid=0 2024-05-09T08:59:49.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.317+0000 7fdfd57d4740 1 client.18150 shutdown 2024-05-09T08:59:49.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.318+0000 7fdfd57d4740 1 -- 172.21.15.98:0/419010565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c30270 msgr2=0x559d63c2dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.318+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/419010565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c30270 0x559d63c2dc40 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009cd0 tx=0x7fdfc4009900 comp rx=0 tx=0).stop 2024-05-09T08:59:49.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.318+0000 7fdfd57d4740 1 -- 172.21.15.98:0/419010565 shutdown_connections 2024-05-09T08:59:49.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.318+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/419010565 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d63c40540 0x559d63c42990 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.318+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/419010565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c30270 0x559d63c2dc40 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.318+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/419010565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c25ba0 0x559d63c2d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.318+0000 7fdfab7fe640 0 client.18150 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:49.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.318+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/419010565 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c25280 0x559d63c2d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.318+0000 7fdfd57d4740 1 -- 172.21.15.98:0/419010565 >> 172.21.15.98:0/419010565 conn(0x559d63c215a0 msgr2=0x559d63c273a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.318+0000 7fdfd57d4740 1 -- 172.21.15.98:0/419010565 shutdown_connections 2024-05-09T08:59:49.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.318+0000 7fdfd57d4740 1 -- 172.21.15.98:0/419010565 wait complete. 2024-05-09T08:59:49.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.318+0000 7fdfd57d4740 20 client.18150 trim_cache size 0 max 16384 2024-05-09T08:59:49.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.335+0000 7fdfd57d4740 1 Processor -- start 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.335+0000 7fdfd57d4740 1 -- start start 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.335+0000 7fdfd57d4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c25280 0x559d63c3bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.335+0000 7fdfd57d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c25ba0 0x559d63c279e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.335+0000 7fdfd57d4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c30270 0x559d63c48260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.335+0000 7fdfd57d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc1aeeb0 con 0x559d63c25ba0 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.335+0000 7fdfd57d4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc1b09f0 con 0x559d63c30270 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.335+0000 7fdfd57d4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc1af580 con 0x559d63c25280 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.336+0000 7fdfcb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c25ba0 0x559d63c279e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.336+0000 7fdfcb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c25ba0 0x559d63c279e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36758/0 (socket says 172.21.15.98:36758) 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.336+0000 7fdfcb7fe640 1 -- 172.21.15.98:0/1413665660 learned_addr learned my addr 172.21.15.98:0/1413665660 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.336+0000 7fdfcb7fe640 1 -- 172.21.15.98:0/1413665660 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c25280 msgr2=0x559d63c3bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.336+0000 7fdfcb7fe640 1 --2- 172.21.15.98:0/1413665660 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c25280 0x559d63c3bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.336+0000 7fdfcb7fe640 1 -- 172.21.15.98:0/1413665660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c30270 msgr2=0x559d63c48260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.336+0000 7fdfcb7fe640 1 --2- 172.21.15.98:0/1413665660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c30270 0x559d63c48260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.336+0000 7fdfcb7fe640 1 -- 172.21.15.98:0/1413665660 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d63c260f0 con 0x559d63c25ba0 2024-05-09T08:59:49.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.336+0000 7fdfcb7fe640 1 --2- 172.21.15.98:0/1413665660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c25ba0 0x559d63c279e0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00115c0 tx=0x7fdfc00115f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53440811e2075f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.336+0000 7fdfb1ffb640 1 -- 172.21.15.98:0/1413665660 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0015040 con 0x559d63c25ba0 2024-05-09T08:59:49.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.336+0000 7fdfb1ffb640 1 -- 172.21.15.98:0/1413665660 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc0005520 con 0x559d63c25ba0 2024-05-09T08:59:49.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.336+0000 7fdfb1ffb640 1 -- 172.21.15.98:0/1413665660 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0011660 con 0x559d63c25ba0 2024-05-09T08:59:49.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.337+0000 7fdfd57d4740 1 -- 172.21.15.98:0/1413665660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c25ba0 msgr2=0x559d63c279e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.337+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/1413665660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c25ba0 0x559d63c279e0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00115c0 tx=0x7fdfc00115f0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.337+0000 7fdfd57d4740 1 -- 172.21.15.98:0/1413665660 shutdown_connections 2024-05-09T08:59:49.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.337+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/1413665660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c30270 0x559d63c48260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.337+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/1413665660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c25ba0 0x559d63c279e0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.337+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/1413665660 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c25280 0x559d63c3bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.337+0000 7fdfd57d4740 1 -- 172.21.15.98:0/1413665660 >> 172.21.15.98:0/1413665660 conn(0x559d63c215a0 msgr2=0x559d63c207e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.337+0000 7fdfd57d4740 1 -- 172.21.15.98:0/1413665660 shutdown_connections 2024-05-09T08:59:49.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.337+0000 7fdfd57d4740 1 -- 172.21.15.98:0/1413665660 wait complete. 2024-05-09T08:59:49.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.337+0000 7fdfd57d4740 1 -- start start 2024-05-09T08:59:49.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.337+0000 7fdfd57d4740 1 Processor -- start 2024-05-09T08:59:49.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.337+0000 7fdfd57d4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c25280 0x559d63c387e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfd57d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c25ba0 0x559d63c29b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfd57d4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c30270 0x559d63c2a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfd57d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc1b0320 con 0x559d63c25ba0 2024-05-09T08:59:49.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfd57d4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc1afc50 con 0x559d63c25280 2024-05-09T08:59:49.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfd57d4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc1aceb0 con 0x559d63c30270 2024-05-09T08:59:49.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfd57d4740 10 client.0 _ll_register_callbacks cb 0x7fdfc4031da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:49.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfcb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c25ba0 0x559d63c29b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfcb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c25ba0 0x559d63c29b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36764/0 (socket says 172.21.15.98:36764) 2024-05-09T08:59:49.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfcb7fe640 1 -- 172.21.15.98:0/2076020468 learned_addr learned my addr 172.21.15.98:0/2076020468 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfd0f84640 1 --2- 172.21.15.98:0/2076020468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c30270 0x559d63c2a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfcb7fe640 1 -- 172.21.15.98:0/2076020468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c30270 msgr2=0x559d63c2a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfcb7fe640 1 --2- 172.21.15.98:0/2076020468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c30270 0x559d63c2a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfcb7fe640 1 -- 172.21.15.98:0/2076020468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c25280 msgr2=0x559d63c387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfcb7fe640 1 --2- 172.21.15.98:0/2076020468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c25280 0x559d63c387e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfcb7fe640 1 -- 172.21.15.98:0/2076020468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d63c2a8b0 con 0x559d63c25ba0 2024-05-09T08:59:49.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfcb7fe640 1 --2- 172.21.15.98:0/2076020468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c25ba0 0x559d63c29b40 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x559d63c20f40 tx=0x7fdfc000ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de31deafbf79fe4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:49.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfab7fe640 1 -- 172.21.15.98:0/2076020468 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0030020 con 0x559d63c25ba0 2024-05-09T08:59:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfab7fe640 1 -- 172.21.15.98:0/2076020468 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fdfc0004a90 con 0x559d63c25ba0 2024-05-09T08:59:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfd57d4740 1 -- 172.21.15.98:0/2076020468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfc0005520 con 0x559d63c25ba0 2024-05-09T08:59:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfab7fe640 1 -- 172.21.15.98:0/2076020468 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fdfc003b050 con 0x559d63c25ba0 2024-05-09T08:59:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfd57d4740 10 client.18156 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.338+0000 7fdfd57d4740 1 -- 172.21.15.98:0/2076020468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559d63ca6b00 con 0x559d63c25ba0 2024-05-09T08:59:49.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.339+0000 7fdfab7fe640 1 -- 172.21.15.98:0/2076020468 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc0005520 con 0x559d63c25ba0 2024-05-09T08:59:49.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.339+0000 7fdfab7fe640 1 -- 172.21.15.98:0/2076020468 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fdfc002c480 con 0x559d63c25ba0 2024-05-09T08:59:49.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.339+0000 7fdfd57d4740 10 client.18156 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:49.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.339+0000 7fdfd57d4740 10 client.18156 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:49.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.339+0000 7fdfd57d4740 20 client.18156 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:49.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.339+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/2076020468 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d63ca7000 0x559d63ca9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.339+0000 7fdfd57d4740 4 client.18156 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:59:49.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.339+0000 7fdfd57d4740 1 -- 172.21.15.98:0/2076020468 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559d63b92780 con 0x559d63ca7000 2024-05-09T08:59:49.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.339+0000 7fdfcbfff640 1 --2- 172.21.15.98:0/2076020468 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d63ca7000 0x559d63ca9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.339+0000 7fdfcbfff640 1 --2- 172.21.15.98:0/2076020468 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d63ca7000 0x559d63ca9420 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb3b908f9a3446df server_cookie=d0ba46b02807e577 in_seq=0 out_seq=0 2024-05-09T08:59:49.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.339+0000 7fdfab7fe640 10 client.18156 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:49.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfab7fe640 1 -- 172.21.15.98:0/2076020468 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559d63b92780 con 0x559d63ca7000 2024-05-09T08:59:49.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfab7fe640 10 client.18156 handle_command_reply: tid=0 2024-05-09T08:59:49.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd57d4740 1 client.18156 shutdown 2024-05-09T08:59:49.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd57d4740 1 -- 172.21.15.98:0/2076020468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c25ba0 msgr2=0x559d63c29b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/2076020468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c25ba0 0x559d63c29b40 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x559d63c20f40 tx=0x7fdfc000ba70 comp rx=0 tx=0).stop 2024-05-09T08:59:49.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd57d4740 1 -- 172.21.15.98:0/2076020468 shutdown_connections 2024-05-09T08:59:49.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/2076020468 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559d63ca7000 0x559d63ca9420 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/2076020468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559d63c30270 0x559d63c2a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfab7fe640 0 client.18156 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:49.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/2076020468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559d63c25ba0 0x559d63c29b40 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd57d4740 1 --2- 172.21.15.98:0/2076020468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559d63c25280 0x559d63c387e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd57d4740 1 -- 172.21.15.98:0/2076020468 >> 172.21.15.98:0/2076020468 conn(0x559d63c215a0 msgr2=0x559d63c2d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd57d4740 1 -- 172.21.15.98:0/2076020468 shutdown_connections 2024-05-09T08:59:49.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd57d4740 1 -- 172.21.15.98:0/2076020468 wait complete. 2024-05-09T08:59:49.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd57d4740 20 client.18156 trim_cache size 0 max 16384 2024-05-09T08:59:49.351 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:59:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T08:59:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T08:59:49.352 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:59:49.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd29e7640 1 -- 172.21.15.98:0/3589625451 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdfb40a2d00 msgr2=0x7fdfb40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd29e7640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdfb40a2d00 0x7fdfb40a51c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4004af0 tx=0x7fdfc40095c0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd29e7640 1 -- 172.21.15.98:0/3589625451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfcc100710 msgr2=0x7fdfcc1980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.340+0000 7fdfd29e7640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfcc100710 0x7fdfcc1980d0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00d770 tx=0x7fdfbc00dc30 comp rx=0 tx=0).stop 2024-05-09T08:59:49.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.341+0000 7fdfd29e7640 1 -- 172.21.15.98:0/3589625451 shutdown_connections 2024-05-09T08:59:49.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.341+0000 7fdfd29e7640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fdfb40a2d00 0x7fdfb40a51c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.341+0000 7fdfd29e7640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfcc106720 0x7fdfcc19c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.341+0000 7fdfd29e7640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdfcc1010d0 0x7fdfcc198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.341+0000 7fdfd29e7640 1 --2- 172.21.15.98:0/3589625451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfcc100710 0x7fdfcc1980d0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.341+0000 7fdfd29e7640 1 -- 172.21.15.98:0/3589625451 >> 172.21.15.98:0/3589625451 conn(0x7fdfcc077790 msgr2=0x7fdfcc1031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.341+0000 7fdfd29e7640 1 -- 172.21.15.98:0/3589625451 shutdown_connections 2024-05-09T08:59:49.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.341+0000 7fdfd29e7640 1 -- 172.21.15.98:0/3589625451 wait complete. 2024-05-09T08:59:49.356 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:59:49.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:59:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.459+0000 7f36046d2640 1 -- 172.21.15.98:0/2809025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc1003f0 msgr2=0x7f35fc1007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.459+0000 7f36046d2640 1 --2- 172.21.15.98:0/2809025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc1003f0 0x7f35fc1007d0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009900 tx=0x7f35f802a410 comp rx=0 tx=0).stop 2024-05-09T08:59:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.459+0000 7f36046d2640 1 -- 172.21.15.98:0/2809025 shutdown_connections 2024-05-09T08:59:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.459+0000 7f36046d2640 1 --2- 172.21.15.98:0/2809025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35fc106bb0 0x7f35fc10e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.459+0000 7f36046d2640 1 --2- 172.21.15.98:0/2809025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35fc100da0 0x7f35fc1064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.459+0000 7f36046d2640 1 --2- 172.21.15.98:0/2809025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc1003f0 0x7f35fc1007d0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.459+0000 7f36046d2640 1 -- 172.21.15.98:0/2809025 >> 172.21.15.98:0/2809025 conn(0x7f35fc075a10 msgr2=0x7f35fc075e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.459+0000 7f36046d2640 1 -- 172.21.15.98:0/2809025 shutdown_connections 2024-05-09T08:59:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.459+0000 7f36046d2640 1 -- 172.21.15.98:0/2809025 wait complete. 2024-05-09T08:59:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.459+0000 7f36046d2640 1 Processor -- start 2024-05-09T08:59:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.459+0000 7f36046d2640 1 -- start start 2024-05-09T08:59:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.459+0000 7f36046d2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35fc1003f0 0x7f35fc198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.459+0000 7f36046d2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc106bb0 0x7f35fc198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f36046d2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35fc19ca00 0x7f35fc19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f36046d2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc111c80 con 0x7f35fc106bb0 2024-05-09T08:59:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f36046d2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc111b00 con 0x7f35fc19ca00 2024-05-09T08:59:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f36046d2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc111e00 con 0x7f35fc1003f0 2024-05-09T08:59:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f3601c6d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc106bb0 0x7f35fc198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f3601c6d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc106bb0 0x7f35fc198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36780/0 (socket says 172.21.15.98:36780) 2024-05-09T08:59:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f3601c6d640 1 -- 172.21.15.98:0/3188097495 learned_addr learned my addr 172.21.15.98:0/3188097495 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f360246e640 1 --2- 172.21.15.98:0/3188097495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35fc1003f0 0x7f35fc198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f3601c6d640 1 -- 172.21.15.98:0/3188097495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35fc1003f0 msgr2=0x7f35fc198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f3601c6d640 1 --2- 172.21.15.98:0/3188097495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35fc1003f0 0x7f35fc198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f3601c6d640 1 -- 172.21.15.98:0/3188097495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35fc19ca00 msgr2=0x7f35fc19ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f3601c6d640 1 --2- 172.21.15.98:0/3188097495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35fc19ca00 0x7f35fc19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f3601c6d640 1 -- 172.21.15.98:0/3188097495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35fc19d4f0 con 0x7f35fc106bb0 2024-05-09T08:59:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f3601c6d640 1 --2- 172.21.15.98:0/3188097495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc106bb0 0x7f35fc198680 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f35ec00c970 tx=0x7f35ec00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f35f37fe640 1 -- 172.21.15.98:0/3188097495 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec014070 con 0x7f35fc106bb0 2024-05-09T08:59:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f35f37fe640 1 -- 172.21.15.98:0/3188097495 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35ec004440 con 0x7f35fc106bb0 2024-05-09T08:59:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f35f37fe640 1 -- 172.21.15.98:0/3188097495 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec00f450 con 0x7f35fc106bb0 2024-05-09T08:59:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f36046d2640 1 -- 172.21.15.98:0/3188097495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35fc19d7e0 con 0x7f35fc106bb0 2024-05-09T08:59:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.460+0000 7f36046d2640 1 -- 172.21.15.98:0/3188097495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35fc1a4fd0 con 0x7f35fc106bb0 2024-05-09T08:59:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.461+0000 7f36046d2640 1 -- 172.21.15.98:0/3188097495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35fc068aa0 con 0x7f35fc106bb0 2024-05-09T08:59:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.463+0000 7f35f37fe640 1 -- 172.21.15.98:0/3188097495 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f35ec017070 con 0x7f35fc106bb0 2024-05-09T08:59:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.464+0000 7f35f37fe640 1 --2- 172.21.15.98:0/3188097495 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f35e80a2d00 0x7f35e80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.464+0000 7f360246e640 1 --2- 172.21.15.98:0/3188097495 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f35e80a2d00 0x7f35e80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.464+0000 7f35f37fe640 1 -- 172.21.15.98:0/3188097495 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f35ec0cfba0 con 0x7f35fc106bb0 2024-05-09T08:59:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.464+0000 7f360246e640 1 --2- 172.21.15.98:0/3188097495 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f35e80a2d00 0x7f35e80a51c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f35f8005ab0 tx=0x7f35f8005a20 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.467+0000 7f35f37fe640 1 -- 172.21.15.98:0/3188097495 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f35ec01e020 con 0x7f35fc106bb0 2024-05-09T08:59:49.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.680+0000 7f36046d2640 1 -- 172.21.15.98:0/3188097495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35fc199490 con 0x7f35fc106bb0 2024-05-09T08:59:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.681+0000 7f35f37fe640 1 -- 172.21.15.98:0/3188097495 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11941 (secure 0 0 0) 0x7f35fc199490 con 0x7f35fc106bb0 2024-05-09T08:59:49.681 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T08:59:49.681 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":26581,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/561027115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":561027115},{"type":"v1","addr":"172.21.15.105:6835","nonce":561027115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:56:52.891273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16641":{"gid":16641,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T08:59:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 37 2024-05-09T08:59:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.683+0000 7f36046d2640 1 -- 172.21.15.98:0/3188097495 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f35e80a2d00 msgr2=0x7f35e80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.683+0000 7f36046d2640 1 --2- 172.21.15.98:0/3188097495 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f35e80a2d00 0x7f35e80a51c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f35f8005ab0 tx=0x7f35f8005a20 comp rx=0 tx=0).stop 2024-05-09T08:59:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.683+0000 7f36046d2640 1 -- 172.21.15.98:0/3188097495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc106bb0 msgr2=0x7f35fc198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.683+0000 7f36046d2640 1 --2- 172.21.15.98:0/3188097495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc106bb0 0x7f35fc198680 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f35ec00c970 tx=0x7f35ec00ce30 comp rx=0 tx=0).stop 2024-05-09T08:59:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.683+0000 7f36046d2640 1 -- 172.21.15.98:0/3188097495 shutdown_connections 2024-05-09T08:59:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.683+0000 7f36046d2640 1 --2- 172.21.15.98:0/3188097495 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f35e80a2d00 0x7f35e80a51c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.683+0000 7f36046d2640 1 --2- 172.21.15.98:0/3188097495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35fc19ca00 0x7f35fc19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.683+0000 7f36046d2640 1 --2- 172.21.15.98:0/3188097495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35fc106bb0 0x7f35fc198680 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.683+0000 7f36046d2640 1 --2- 172.21.15.98:0/3188097495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35fc1003f0 0x7f35fc198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.683+0000 7f36046d2640 1 -- 172.21.15.98:0/3188097495 >> 172.21.15.98:0/3188097495 conn(0x7f35fc075a10 msgr2=0x7f35fc0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.683+0000 7f36046d2640 1 -- 172.21.15.98:0/3188097495 shutdown_connections 2024-05-09T08:59:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.683+0000 7f36046d2640 1 -- 172.21.15.98:0/3188097495 wait complete. 2024-05-09T08:59:49.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:59:49.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.798+0000 7f2e2ecee640 1 -- 172.21.15.98:0/653515082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2810bdf0 msgr2=0x7f2e2810e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.798+0000 7f2e2ecee640 1 --2- 172.21.15.98:0/653515082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2810bdf0 0x7f2e2810e290 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f2e20009ca0 tx=0x7f2e2002a9e0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.798+0000 7f2e2ecee640 1 -- 172.21.15.98:0/653515082 shutdown_connections 2024-05-09T08:59:49.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.798+0000 7f2e2ecee640 1 --2- 172.21.15.98:0/653515082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2810bdf0 0x7f2e2810e290 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.798+0000 7f2e2ecee640 1 --2- 172.21.15.98:0/653515082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e281033d0 0x7f2e2810b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.798+0000 7f2e2ecee640 1 --2- 172.21.15.98:0/653515082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e28102ab0 0x7f2e28102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.798+0000 7f2e2ecee640 1 -- 172.21.15.98:0/653515082 >> 172.21.15.98:0/653515082 conn(0x7f2e28077830 msgr2=0x7f2e28077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.798+0000 7f2e2ecee640 1 -- 172.21.15.98:0/653515082 shutdown_connections 2024-05-09T08:59:49.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.798+0000 7f2e2ecee640 1 -- 172.21.15.98:0/653515082 wait complete. 2024-05-09T08:59:49.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2ecee640 1 Processor -- start 2024-05-09T08:59:49.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2ecee640 1 -- start start 2024-05-09T08:59:49.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2ecee640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e28102ab0 0x7f2e28198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2ecee640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e281033d0 0x7f2e28198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2ecee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2810bdf0 0x7f2e2819c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2ecee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e28111c30 con 0x7f2e2810bdf0 2024-05-09T08:59:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2ecee640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2e28111ab0 con 0x7f2e28102ab0 2024-05-09T08:59:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2ecee640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2e28111db0 con 0x7f2e281033d0 2024-05-09T08:59:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2d28b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2810bdf0 0x7f2e2819c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2d28b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2810bdf0 0x7f2e2819c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36796/0 (socket says 172.21.15.98:36796) 2024-05-09T08:59:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2d28b640 1 -- 172.21.15.98:0/726532896 learned_addr learned my addr 172.21.15.98:0/726532896 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e27fff640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e281033d0 0x7f2e28198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2ca8a640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e28102ab0 0x7f2e28198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2d28b640 1 -- 172.21.15.98:0/726532896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e281033d0 msgr2=0x7f2e28198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2d28b640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e281033d0 0x7f2e28198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2d28b640 1 -- 172.21.15.98:0/726532896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e28102ab0 msgr2=0x7f2e28198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2d28b640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e28102ab0 0x7f2e28198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2d28b640 1 -- 172.21.15.98:0/726532896 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2819d0d0 con 0x7f2e2810bdf0 2024-05-09T08:59:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2ca8a640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e28102ab0 0x7f2e28198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.799+0000 7f2e2d28b640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2810bdf0 0x7f2e2819c9a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f2e20009dc0 tx=0x7f2e20003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.800+0000 7f2e25ffb640 1 -- 172.21.15.98:0/726532896 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e20046070 con 0x7f2e2810bdf0 2024-05-09T08:59:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.800+0000 7f2e25ffb640 1 -- 172.21.15.98:0/726532896 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e2002e870 con 0x7f2e2810bdf0 2024-05-09T08:59:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.800+0000 7f2e2ecee640 1 -- 172.21.15.98:0/726532896 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e2819d360 con 0x7f2e2810bdf0 2024-05-09T08:59:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.800+0000 7f2e25ffb640 1 -- 172.21.15.98:0/726532896 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e20033070 con 0x7f2e2810bdf0 2024-05-09T08:59:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.800+0000 7f2e2ecee640 1 -- 172.21.15.98:0/726532896 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e281a4c50 con 0x7f2e2810bdf0 2024-05-09T08:59:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.802+0000 7f2e25ffb640 1 -- 172.21.15.98:0/726532896 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2e20049070 con 0x7f2e2810bdf0 2024-05-09T08:59:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.802+0000 7f2e319fa740 1 Processor -- start 2024-05-09T08:59:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.802+0000 7f2e319fa740 1 -- start start 2024-05-09T08:59:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e319fa740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bed330 0x55c443bed710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e319fa740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bedc50 0x55c443bf7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e319fa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443bfa710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e319fa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c443b52c40 con 0x55c443bf8320 2024-05-09T08:59:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e319fa740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c443afa580 con 0x55c443bed330 2024-05-09T08:59:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e319fa740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c443b4ec00 con 0x55c443bedc50 2024-05-09T08:59:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e25ffb640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2e180a2d00 0x7f2e180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e27fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443bfa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e27fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443bfa710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36808/0 (socket says 172.21.15.98:36808) 2024-05-09T08:59:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e27fff640 1 -- 172.21.15.98:0/601128712 learned_addr learned my addr 172.21.15.98:0/601128712 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e2d28b640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2e180a2d00 0x7f2e180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e25ffb640 1 -- 172.21.15.98:0/726532896 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f2e200f0540 con 0x7f2e2810bdf0 2024-05-09T08:59:49.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e27fff640 1 --2- 172.21.15.98:0/601128712 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bed330 0x55c443bed710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e2ca8a640 1 --2- 172.21.15.98:0/601128712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bedc50 0x55c443bf7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e27fff640 1 -- 172.21.15.98:0/601128712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bedc50 msgr2=0x55c443bf7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e27fff640 1 --2- 172.21.15.98:0/601128712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bedc50 0x55c443bf7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e27fff640 1 -- 172.21.15.98:0/601128712 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bed330 msgr2=0x55c443bed710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e27fff640 1 --2- 172.21.15.98:0/601128712 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bed330 0x55c443bed710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e27fff640 1 -- 172.21.15.98:0/601128712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4438663f0 con 0x55c443bf8320 2024-05-09T08:59:49.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e27fff640 1 --2- 172.21.15.98:0/601128712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443bfa710 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2e140124f0 tx=0x7f2e1400c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c116788050e6df server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e2d28b640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2e180a2d00 0x7f2e180a51c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2e200ed930 tx=0x7f2e200a7040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e12ffd640 1 -- 172.21.15.98:0/601128712 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e14018030 con 0x55c443bf8320 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e12ffd640 1 -- 172.21.15.98:0/601128712 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e14005500 con 0x55c443bf8320 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.803+0000 7f2e12ffd640 1 -- 172.21.15.98:0/601128712 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e140108a0 con 0x55c443bf8320 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.804+0000 7f2e319fa740 1 -- 172.21.15.98:0/601128712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 msgr2=0x55c443bfa710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.804+0000 7f2e319fa740 1 --2- 172.21.15.98:0/601128712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443bfa710 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2e140124f0 tx=0x7f2e1400c5c0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.804+0000 7f2e319fa740 1 -- 172.21.15.98:0/601128712 shutdown_connections 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.804+0000 7f2e319fa740 1 --2- 172.21.15.98:0/601128712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443bfa710 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.804+0000 7f2e319fa740 1 --2- 172.21.15.98:0/601128712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bedc50 0x55c443bf7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.804+0000 7f2e319fa740 1 --2- 172.21.15.98:0/601128712 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bed330 0x55c443bed710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.804+0000 7f2e319fa740 1 -- 172.21.15.98:0/601128712 >> 172.21.15.98:0/601128712 conn(0x55c443be9650 msgr2=0x55c443be9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.804+0000 7f2e319fa740 1 -- 172.21.15.98:0/601128712 shutdown_connections 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.804+0000 7f2e319fa740 1 -- 172.21.15.98:0/601128712 wait complete. 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.804+0000 7f2e319fa740 1 -- start start 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.804+0000 7f2e319fa740 1 Processor -- start 2024-05-09T08:59:49.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e319fa740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bed330 0x55c443bf52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e319fa740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bedc50 0x55c443bf5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e319fa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443bf5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e319fa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c443bfb110 con 0x55c443bf8320 2024-05-09T08:59:49.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e319fa740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c443b0dc40 con 0x55c443bedc50 2024-05-09T08:59:49.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e319fa740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c443bfb290 con 0x55c443bed330 2024-05-09T08:59:49.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e319fa740 10 client.0 _ll_register_callbacks cb 0x55c443865be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:49.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e27fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443bf5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e27fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443bf5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36816/0 (socket says 172.21.15.98:36816) 2024-05-09T08:59:49.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e27fff640 1 -- 172.21.15.98:0/2922587573 learned_addr learned my addr 172.21.15.98:0/2922587573 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e27fff640 1 --2- 172.21.15.98:0/2922587573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bed330 0x55c443bf52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e2ca8a640 1 --2- 172.21.15.98:0/2922587573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bedc50 0x55c443bf5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e27fff640 1 -- 172.21.15.98:0/2922587573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bed330 msgr2=0x55c443bf52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e27fff640 1 --2- 172.21.15.98:0/2922587573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bed330 0x55c443bf52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e27fff640 1 -- 172.21.15.98:0/2922587573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bedc50 msgr2=0x55c443bf5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e27fff640 1 --2- 172.21.15.98:0/2922587573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bedc50 0x55c443bf5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e27fff640 1 -- 172.21.15.98:0/2922587573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4439307d0 con 0x55c443bf8320 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e27fff640 1 --2- 172.21.15.98:0/2922587573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443bf5d70 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f2e14004c40 tx=0x7f2e14004d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3925437a99c472 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.805+0000 7f2e07fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e07fff640 1 -- 172.21.15.98:0/2922587573 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e14018040 con 0x55c443bf8320 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e07fff640 1 -- 172.21.15.98:0/2922587573 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f2e14004fc0 con 0x55c443bf8320 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e319fa740 1 -- 172.21.15.98:0/2922587573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e14005500 con 0x55c443bf8320 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e07fff640 1 -- 172.21.15.98:0/2922587573 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2e14032050 con 0x55c443bf8320 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e319fa740 10 client.18174 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e319fa740 1 -- 172.21.15.98:0/2922587573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c443c07380 con 0x55c443bf8320 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e07fff640 1 -- 172.21.15.98:0/2922587573 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e14005500 con 0x55c443bf8320 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e07fff640 1 -- 172.21.15.98:0/2922587573 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f2e1402a020 con 0x55c443bf8320 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e319fa740 10 client.18174 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e319fa740 10 client.18174 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e319fa740 20 client.18174 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:49.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e319fa740 1 --2- 172.21.15.98:0/2922587573 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55c443c08730 0x55c443c0ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e319fa740 4 client.18174 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e319fa740 1 -- 172.21.15.98:0/2922587573 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c443b60370 con 0x55c443c08730 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.806+0000 7f2e2ca8a640 1 --2- 172.21.15.98:0/2922587573 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55c443c08730 0x55c443c0ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.807+0000 7f2e07fff640 10 client.18174 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.807+0000 7f2e2ca8a640 1 --2- 172.21.15.98:0/2922587573 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55c443c08730 0x55c443c0ab20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e823a3f3d736bf23 server_cookie=e81471958c0ea67 in_seq=0 out_seq=0 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.807+0000 7f2e07fff640 1 -- 172.21.15.98:0/2922587573 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c443b60370 con 0x55c443c08730 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.807+0000 7f2e07fff640 10 client.18174 handle_command_reply: tid=0 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.808+0000 7f2e319fa740 1 client.18174 shutdown 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.808+0000 7f2e319fa740 1 -- 172.21.15.98:0/2922587573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 msgr2=0x55c443bf5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.808+0000 7f2e319fa740 1 --2- 172.21.15.98:0/2922587573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443bf5d70 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f2e14004c40 tx=0x7f2e14004d40 comp rx=0 tx=0).stop 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.808+0000 7f2e319fa740 1 -- 172.21.15.98:0/2922587573 shutdown_connections 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.808+0000 7f2e319fa740 1 --2- 172.21.15.98:0/2922587573 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55c443c08730 0x55c443c0ab20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.808+0000 7f2e319fa740 1 --2- 172.21.15.98:0/2922587573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443bf5d70 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.808+0000 7f2e319fa740 1 --2- 172.21.15.98:0/2922587573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bedc50 0x55c443bf5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.808+0000 7f2e07fff640 0 client.18174 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:59:49.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.808+0000 7f2e319fa740 1 --2- 172.21.15.98:0/2922587573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bed330 0x55c443bf52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.808+0000 7f2e319fa740 1 -- 172.21.15.98:0/2922587573 >> 172.21.15.98:0/2922587573 conn(0x55c443be9650 msgr2=0x55c443bef450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.808+0000 7f2e319fa740 1 -- 172.21.15.98:0/2922587573 shutdown_connections 2024-05-09T08:59:49.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.808+0000 7f2e319fa740 1 -- 172.21.15.98:0/2922587573 wait complete. 2024-05-09T08:59:49.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.808+0000 7f2e319fa740 20 client.18174 trim_cache size 0 max 16384 2024-05-09T08:59:49.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.821+0000 7f2e319fa740 1 Processor -- start 2024-05-09T08:59:49.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.821+0000 7f2e319fa740 1 -- start start 2024-05-09T08:59:49.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.821+0000 7f2e319fa740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bed330 0x55c443bf7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.821+0000 7f2e319fa740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bedc50 0x55c443c04000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.821+0000 7f2e319fa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443befad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.821+0000 7f2e319fa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e281aee10 con 0x55c443bf8320 2024-05-09T08:59:49.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.821+0000 7f2e319fa740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2e281b0950 con 0x55c443bedc50 2024-05-09T08:59:49.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.821+0000 7f2e319fa740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2e281af4e0 con 0x55c443bed330 2024-05-09T08:59:49.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.821+0000 7f2e27fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443befad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.821+0000 7f2e27fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443befad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36826/0 (socket says 172.21.15.98:36826) 2024-05-09T08:59:49.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.821+0000 7f2e27fff640 1 -- 172.21.15.98:0/3018901084 learned_addr learned my addr 172.21.15.98:0/3018901084 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e27fff640 1 -- 172.21.15.98:0/3018901084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bed330 msgr2=0x55c443bf7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e27fff640 1 --2- 172.21.15.98:0/3018901084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bed330 0x55c443bf7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e27fff640 1 -- 172.21.15.98:0/3018901084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bedc50 msgr2=0x55c443c04000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e27fff640 1 --2- 172.21.15.98:0/3018901084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bedc50 0x55c443c04000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e27fff640 1 -- 172.21.15.98:0/3018901084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c443bdad20 con 0x55c443bf8320 2024-05-09T08:59:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e27fff640 1 --2- 172.21.15.98:0/3018901084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443befad0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f2e1401d7e0 tx=0x7f2e1401d810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab9130c4e4205e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e127fc640 1 -- 172.21.15.98:0/3018901084 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e14018040 con 0x55c443bf8320 2024-05-09T08:59:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e127fc640 1 -- 172.21.15.98:0/3018901084 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e14012e40 con 0x55c443bf8320 2024-05-09T08:59:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e127fc640 1 -- 172.21.15.98:0/3018901084 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e14004f40 con 0x55c443bf8320 2024-05-09T08:59:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e319fa740 1 -- 172.21.15.98:0/3018901084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 msgr2=0x55c443befad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e319fa740 1 --2- 172.21.15.98:0/3018901084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443befad0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f2e1401d7e0 tx=0x7f2e1401d810 comp rx=0 tx=0).stop 2024-05-09T08:59:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e319fa740 1 -- 172.21.15.98:0/3018901084 shutdown_connections 2024-05-09T08:59:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e319fa740 1 --2- 172.21.15.98:0/3018901084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bf8320 0x55c443befad0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e319fa740 1 --2- 172.21.15.98:0/3018901084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bedc50 0x55c443c04000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e319fa740 1 --2- 172.21.15.98:0/3018901084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bed330 0x55c443bf7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e319fa740 1 -- 172.21.15.98:0/3018901084 >> 172.21.15.98:0/3018901084 conn(0x55c443be9650 msgr2=0x55c443be87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e319fa740 1 -- 172.21.15.98:0/3018901084 shutdown_connections 2024-05-09T08:59:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.822+0000 7f2e319fa740 1 -- 172.21.15.98:0/3018901084 wait complete. 2024-05-09T08:59:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.823+0000 7f2e319fa740 1 -- start start 2024-05-09T08:59:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.823+0000 7f2e319fa740 1 Processor -- start 2024-05-09T08:59:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.823+0000 7f2e319fa740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bed330 0x55c443c00620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.823+0000 7f2e319fa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bedc50 0x55c443bfbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.823+0000 7f2e319fa740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bf8320 0x55c443bfc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.823+0000 7f2e319fa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e281b0280 con 0x55c443bedc50 2024-05-09T08:59:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.823+0000 7f2e319fa740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2e281afbb0 con 0x55c443bf8320 2024-05-09T08:59:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.823+0000 7f2e319fa740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2e281ace10 con 0x55c443bed330 2024-05-09T08:59:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.823+0000 7f2e319fa740 10 client.0 _ll_register_callbacks cb 0x7f2e1400cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.823+0000 7f2e2ca8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bedc50 0x55c443bfbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.823+0000 7f2e2ca8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bedc50 0x55c443bfbad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36834/0 (socket says 172.21.15.98:36834) 2024-05-09T08:59:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.823+0000 7f2e2ca8a640 1 -- 172.21.15.98:0/181551912 learned_addr learned my addr 172.21.15.98:0/181551912 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e27fff640 1 --2- 172.21.15.98:0/181551912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bf8320 0x55c443bfc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e2ca8a640 1 -- 172.21.15.98:0/181551912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bed330 msgr2=0x55c443c00620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e2ca8a640 1 --2- 172.21.15.98:0/181551912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bed330 0x55c443c00620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e2ca8a640 1 -- 172.21.15.98:0/181551912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bf8320 msgr2=0x55c443bfc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e2ca8a640 1 --2- 172.21.15.98:0/181551912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bf8320 0x55c443bfc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e2ca8a640 1 -- 172.21.15.98:0/181551912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c443bfc550 con 0x55c443bedc50 2024-05-09T08:59:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e27fff640 1 --2- 172.21.15.98:0/181551912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bf8320 0x55c443bfc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e2ca8a640 1 --2- 172.21.15.98:0/181551912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bedc50 0x55c443bfbad0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x55c443bf4b50 tx=0x7f2e1c0144d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc245ed47f0cf51a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e07fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e07fff640 1 -- 172.21.15.98:0/181551912 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c01f070 con 0x55c443bedc50 2024-05-09T08:59:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e07fff640 1 -- 172.21.15.98:0/181551912 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f2e1c014d80 con 0x55c443bedc50 2024-05-09T08:59:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e319fa740 1 -- 172.21.15.98:0/181551912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c443b5a6c0 con 0x55c443bedc50 2024-05-09T08:59:49.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e07fff640 1 -- 172.21.15.98:0/181551912 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2e1c02a050 con 0x55c443bedc50 2024-05-09T08:59:49.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e319fa740 10 client.18180 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:49.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e319fa740 1 -- 172.21.15.98:0/181551912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c443c72d40 con 0x55c443bedc50 2024-05-09T08:59:49.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e07fff640 1 -- 172.21.15.98:0/181551912 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c443b5a6c0 con 0x55c443bedc50 2024-05-09T08:59:49.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e07fff640 1 -- 172.21.15.98:0/181551912 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f2e1c022460 con 0x55c443bedc50 2024-05-09T08:59:49.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e319fa740 10 client.18180 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:49.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e319fa740 10 client.18180 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T08:59:49.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e319fa740 20 client.18180 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:49.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e319fa740 1 --2- 172.21.15.98:0/181551912 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55c443c734a0 0x55c443c758c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e319fa740 4 client.18180 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:59:49.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.824+0000 7f2e319fa740 1 -- 172.21.15.98:0/181551912 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c443ad7820 con 0x55c443c734a0 2024-05-09T08:59:49.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.825+0000 7f2e27fff640 1 --2- 172.21.15.98:0/181551912 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55c443c734a0 0x55c443c758c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.825+0000 7f2e27fff640 1 --2- 172.21.15.98:0/181551912 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55c443c734a0 0x55c443c758c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8a054a0c1a6de496 server_cookie=5e682c7298dea3a in_seq=0 out_seq=0 2024-05-09T08:59:49.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.825+0000 7f2e07fff640 10 client.18180 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e07fff640 1 -- 172.21.15.98:0/181551912 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c443ad7820 con 0x55c443c734a0 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e07fff640 10 client.18180 handle_command_reply: tid=0 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e319fa740 1 client.18180 shutdown 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e319fa740 1 -- 172.21.15.98:0/181551912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bedc50 msgr2=0x55c443bfbad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e319fa740 1 --2- 172.21.15.98:0/181551912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bedc50 0x55c443bfbad0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x55c443bf4b50 tx=0x7f2e1c0144d0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e319fa740 1 -- 172.21.15.98:0/181551912 shutdown_connections 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e319fa740 1 --2- 172.21.15.98:0/181551912 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55c443c734a0 0x55c443c758c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e319fa740 1 --2- 172.21.15.98:0/181551912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c443bf8320 0x55c443bfc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e319fa740 1 --2- 172.21.15.98:0/181551912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c443bedc50 0x55c443bfbad0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e07fff640 0 client.18180 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e319fa740 1 --2- 172.21.15.98:0/181551912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c443bed330 0x55c443c00620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e319fa740 1 -- 172.21.15.98:0/181551912 >> 172.21.15.98:0/181551912 conn(0x55c443be9650 msgr2=0x55c443bf52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e319fa740 1 -- 172.21.15.98:0/181551912 shutdown_connections 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e319fa740 1 -- 172.21.15.98:0/181551912 wait complete. 2024-05-09T08:59:49.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e319fa740 20 client.18180 trim_cache size 0 max 16384 2024-05-09T08:59:49.834 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:59:49.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e2ecee640 1 -- 172.21.15.98:0/726532896 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2e180a2d00 msgr2=0x7f2e180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e2ecee640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2e180a2d00 0x7f2e180a51c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2e200ed930 tx=0x7f2e200a7040 comp rx=0 tx=0).stop 2024-05-09T08:59:49.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e2ecee640 1 -- 172.21.15.98:0/726532896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2810bdf0 msgr2=0x7f2e2819c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e2ecee640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2810bdf0 0x7f2e2819c9a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f2e20009dc0 tx=0x7f2e20003f10 comp rx=0 tx=0).stop 2024-05-09T08:59:49.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e2ecee640 1 -- 172.21.15.98:0/726532896 shutdown_connections 2024-05-09T08:59:49.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e2ecee640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f2e180a2d00 0x7f2e180a51c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e2ecee640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e2810bdf0 0x7f2e2819c9a0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e2ecee640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e281033d0 0x7f2e28198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e2ecee640 1 --2- 172.21.15.98:0/726532896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e28102ab0 0x7f2e28198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.826+0000 7f2e2ecee640 1 -- 172.21.15.98:0/726532896 >> 172.21.15.98:0/726532896 conn(0x7f2e28077830 msgr2=0x7f2e280fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.827+0000 7f2e2ecee640 1 -- 172.21.15.98:0/726532896 shutdown_connections 2024-05-09T08:59:49.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.827+0000 7f2e2ecee640 1 -- 172.21.15.98:0/726532896 wait complete. 2024-05-09T08:59:49.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:59:49.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.940+0000 7f084bbc4640 1 -- 172.21.15.98:0/2388750348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0844074420 msgr2=0x7f084410dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.940+0000 7f084bbc4640 1 --2- 172.21.15.98:0/2388750348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0844074420 0x7f084410dbd0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f0834009900 tx=0x7f083402a480 comp rx=0 tx=0).stop 2024-05-09T08:59:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.940+0000 7f084bbc4640 1 -- 172.21.15.98:0/2388750348 shutdown_connections 2024-05-09T08:59:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.940+0000 7f084bbc4640 1 --2- 172.21.15.98:0/2388750348 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f084410e110 0x7f0844110500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.940+0000 7f084bbc4640 1 --2- 172.21.15.98:0/2388750348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0844074420 0x7f084410dbd0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.940+0000 7f084bbc4640 1 --2- 172.21.15.98:0/2388750348 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0844073b00 0x7f0844073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.940+0000 7f084bbc4640 1 -- 172.21.15.98:0/2388750348 >> 172.21.15.98:0/2388750348 conn(0x7f0844078c80 msgr2=0x7f0844079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.940+0000 7f084bbc4640 1 -- 172.21.15.98:0/2388750348 shutdown_connections 2024-05-09T08:59:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.940+0000 7f084bbc4640 1 -- 172.21.15.98:0/2388750348 wait complete. 2024-05-09T08:59:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.941+0000 7f084bbc4640 1 Processor -- start 2024-05-09T08:59:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.941+0000 7f084bbc4640 1 -- start start 2024-05-09T08:59:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.941+0000 7f084bbc4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0844073b00 0x7f084419a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.941+0000 7f084bbc4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0844074420 0x7f084419a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.941+0000 7f084bbc4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f084410e110 0x7f084419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.941+0000 7f084bbc4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0844113e10 con 0x7f0844074420 2024-05-09T08:59:49.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.941+0000 7f084bbc4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0844113c90 con 0x7f0844073b00 2024-05-09T08:59:49.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.941+0000 7f084bbc4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0844113f90 con 0x7f084410e110 2024-05-09T08:59:49.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.941+0000 7f084915f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0844074420 0x7f084419a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.941+0000 7f084915f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0844074420 0x7f084419a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36852/0 (socket says 172.21.15.98:36852) 2024-05-09T08:59:49.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.941+0000 7f084915f640 1 -- 172.21.15.98:0/2265166810 learned_addr learned my addr 172.21.15.98:0/2265166810 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.941+0000 7f0849960640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0844073b00 0x7f084419a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.941+0000 7f084a161640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f084410e110 0x7f084419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.942+0000 7f084915f640 1 -- 172.21.15.98:0/2265166810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f084410e110 msgr2=0x7f084419ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.942+0000 7f084915f640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f084410e110 0x7f084419ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.942+0000 7f084915f640 1 -- 172.21.15.98:0/2265166810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0844073b00 msgr2=0x7f084419a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.942+0000 7f084915f640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0844073b00 0x7f084419a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.942+0000 7f084915f640 1 -- 172.21.15.98:0/2265166810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084419f340 con 0x7f0844074420 2024-05-09T08:59:49.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.942+0000 7f084a161640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f084410e110 0x7f084419ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:49.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.942+0000 7f084915f640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0844074420 0x7f084419a890 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f0834006010 tx=0x7f08340042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.942+0000 7f083affd640 1 -- 172.21.15.98:0/2265166810 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083403f030 con 0x7f0844074420 2024-05-09T08:59:49.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.942+0000 7f083affd640 1 -- 172.21.15.98:0/2265166810 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0834003ce0 con 0x7f0844074420 2024-05-09T08:59:49.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.942+0000 7f084bbc4640 1 -- 172.21.15.98:0/2265166810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f084419f5d0 con 0x7f0844074420 2024-05-09T08:59:49.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.942+0000 7f083affd640 1 -- 172.21.15.98:0/2265166810 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834035070 con 0x7f0844074420 2024-05-09T08:59:49.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.942+0000 7f084bbc4640 1 -- 172.21.15.98:0/2265166810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08441a6ec0 con 0x7f0844074420 2024-05-09T08:59:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.945+0000 7f083affd640 1 -- 172.21.15.98:0/2265166810 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0834042030 con 0x7f0844074420 2024-05-09T08:59:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.945+0000 7f084e9a8740 1 Processor -- start 2024-05-09T08:59:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.945+0000 7f084e9a8740 1 -- start start 2024-05-09T08:59:49.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.945+0000 7f084e9a8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc8d330 0x564dacc8d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.945+0000 7f084e9a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc97de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.945+0000 7f084e9a8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 0x564dacc9a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.945+0000 7f084e9a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564dacbf2c40 con 0x564dacc8dc50 2024-05-09T08:59:49.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f084e9a8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564dacb9a580 con 0x564dacc98320 2024-05-09T08:59:49.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f084e9a8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564dacbeec00 con 0x564dacc8d330 2024-05-09T08:59:49.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f0849960640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc97de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f0849960640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc97de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36856/0 (socket says 172.21.15.98:36856) 2024-05-09T08:59:49.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f0849960640 1 -- 172.21.15.98:0/395958263 learned_addr learned my addr 172.21.15.98:0/395958263 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f084a161640 1 --2- 172.21.15.98:0/395958263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc8d330 0x564dacc8d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f083affd640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f08300a2d00 0x7f08300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f0849960640 1 -- 172.21.15.98:0/395958263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc8d330 msgr2=0x564dacc8d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f084915f640 1 --2- 172.21.15.98:0/395958263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 0x564dacc9a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f0849960640 1 --2- 172.21.15.98:0/395958263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc8d330 0x564dacc8d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f0849960640 1 -- 172.21.15.98:0/395958263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 msgr2=0x564dacc9a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f0849960640 1 --2- 172.21.15.98:0/395958263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 0x564dacc9a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f0849960640 1 -- 172.21.15.98:0/395958263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564dac9063f0 con 0x564dacc8dc50 2024-05-09T08:59:49.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f084915f640 1 --2- 172.21.15.98:0/395958263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 0x564dacc9a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:49.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f084a161640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f08300a2d00 0x7f08300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f083affd640 1 -- 172.21.15.98:0/2265166810 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f08340efdd0 con 0x7f0844074420 2024-05-09T08:59:49.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f0849960640 1 --2- 172.21.15.98:0/395958263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc97de0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f084000bd10 tx=0x7f08400107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab322ebe2a73942 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f082ffff640 1 -- 172.21.15.98:0/395958263 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0840004030 con 0x564dacc8dc50 2024-05-09T08:59:49.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f084a161640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f08300a2d00 0x7f08300a51c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f084406b100 tx=0x7f083c004a40 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f082ffff640 1 -- 172.21.15.98:0/395958263 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08400041d0 con 0x564dacc8dc50 2024-05-09T08:59:49.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.946+0000 7f082ffff640 1 -- 172.21.15.98:0/395958263 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0840011820 con 0x564dacc8dc50 2024-05-09T08:59:49.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.947+0000 7f084e9a8740 1 -- 172.21.15.98:0/395958263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 msgr2=0x564dacc97de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.947+0000 7f084e9a8740 1 --2- 172.21.15.98:0/395958263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc97de0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f084000bd10 tx=0x7f08400107a0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.947+0000 7f084e9a8740 1 -- 172.21.15.98:0/395958263 shutdown_connections 2024-05-09T08:59:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.947+0000 7f084e9a8740 1 --2- 172.21.15.98:0/395958263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 0x564dacc9a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.947+0000 7f084e9a8740 1 --2- 172.21.15.98:0/395958263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc97de0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.947+0000 7f084e9a8740 1 --2- 172.21.15.98:0/395958263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc8d330 0x564dacc8d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.947+0000 7f084e9a8740 1 -- 172.21.15.98:0/395958263 >> 172.21.15.98:0/395958263 conn(0x564dacc89650 msgr2=0x564dacc89a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.947+0000 7f084e9a8740 1 -- 172.21.15.98:0/395958263 shutdown_connections 2024-05-09T08:59:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.947+0000 7f084e9a8740 1 -- 172.21.15.98:0/395958263 wait complete. 2024-05-09T08:59:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084e9a8740 1 -- start start 2024-05-09T08:59:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084e9a8740 1 Processor -- start 2024-05-09T08:59:49.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084e9a8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc8d330 0x564dacc95210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084e9a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc95750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084e9a8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 0x564dacc95c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084e9a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564dacc9b110 con 0x564dacc8dc50 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084e9a8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564dacbadc40 con 0x564dacc98320 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084e9a8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564dacc9b290 con 0x564dacc8d330 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084e9a8740 10 client.0 _ll_register_callbacks cb 0x564dac905be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084915f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc95750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084915f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc95750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36868/0 (socket says 172.21.15.98:36868) 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084915f640 1 -- 172.21.15.98:0/2816266307 learned_addr learned my addr 172.21.15.98:0/2816266307 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084a161640 1 --2- 172.21.15.98:0/2816266307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 0x564dacc95c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f0849960640 1 --2- 172.21.15.98:0/2816266307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc8d330 0x564dacc95210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084915f640 1 -- 172.21.15.98:0/2816266307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc8d330 msgr2=0x564dacc95210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084915f640 1 --2- 172.21.15.98:0/2816266307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc8d330 0x564dacc95210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084915f640 1 -- 172.21.15.98:0/2816266307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 msgr2=0x564dacc95c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084915f640 1 --2- 172.21.15.98:0/2816266307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 0x564dacc95c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.948+0000 7f084915f640 1 -- 172.21.15.98:0/2816266307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564dac9d07d0 con 0x564dacc8dc50 2024-05-09T08:59:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f084915f640 1 --2- 172.21.15.98:0/2816266307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc95750 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f0834000c00 tx=0x7f0834009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58680069c647bdec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f082d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f082d7fa640 1 -- 172.21.15.98:0/2816266307 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834049030 con 0x564dacc8dc50 2024-05-09T08:59:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f082d7fa640 1 -- 172.21.15.98:0/2816266307 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f08340c4720 con 0x564dacc8dc50 2024-05-09T08:59:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f084e9a8740 1 -- 172.21.15.98:0/2816266307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08400041d0 con 0x564dacc8dc50 2024-05-09T08:59:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f082d7fa640 1 -- 172.21.15.98:0/2816266307 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f083403bcb0 con 0x564dacc8dc50 2024-05-09T08:59:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f084e9a8740 10 client.18192 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f084e9a8740 1 -- 172.21.15.98:0/2816266307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564dacca7230 con 0x564dacc8dc50 2024-05-09T08:59:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f082d7fa640 1 -- 172.21.15.98:0/2816266307 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08400041d0 con 0x564dacc8dc50 2024-05-09T08:59:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f082d7fa640 1 -- 172.21.15.98:0/2816266307 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f08340bb390 con 0x564dacc8dc50 2024-05-09T08:59:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f084e9a8740 10 client.18192 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f084e9a8740 10 client.18192 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:59:49.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f084e9a8740 20 client.18192 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f084e9a8740 1 --2- 172.21.15.98:0/2816266307 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x564dacca8580 0x564daccaa990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f084e9a8740 4 client.18192 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.949+0000 7f084e9a8740 1 -- 172.21.15.98:0/2816266307 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564dacc00370 con 0x564dacca8580 2024-05-09T08:59:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.950+0000 7f0849960640 1 --2- 172.21.15.98:0/2816266307 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x564dacca8580 0x564daccaa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.950+0000 7f082d7fa640 10 client.18192 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:59:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.950+0000 7f0849960640 1 --2- 172.21.15.98:0/2816266307 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x564dacca8580 0x564daccaa990 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6c6cd4fe087542e2 server_cookie=347b41e0be57e72c in_seq=0 out_seq=0 2024-05-09T08:59:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f082d7fa640 1 -- 172.21.15.98:0/2816266307 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564dacc00370 con 0x564dacca8580 2024-05-09T08:59:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f082d7fa640 10 client.18192 handle_command_reply: tid=0 2024-05-09T08:59:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f084e9a8740 1 client.18192 shutdown 2024-05-09T08:59:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f084e9a8740 1 -- 172.21.15.98:0/2816266307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 msgr2=0x564dacc95750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f084e9a8740 1 --2- 172.21.15.98:0/2816266307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc95750 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f0834000c00 tx=0x7f0834009620 comp rx=0 tx=0).stop 2024-05-09T08:59:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f084e9a8740 1 -- 172.21.15.98:0/2816266307 shutdown_connections 2024-05-09T08:59:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f084e9a8740 1 --2- 172.21.15.98:0/2816266307 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x564dacca8580 0x564daccaa990 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f084e9a8740 1 --2- 172.21.15.98:0/2816266307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 0x564dacc95c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f084e9a8740 1 --2- 172.21.15.98:0/2816266307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc95750 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f082d7fa640 0 client.18192 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:59:49.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f084e9a8740 1 --2- 172.21.15.98:0/2816266307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc8d330 0x564dacc95210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f084e9a8740 1 -- 172.21.15.98:0/2816266307 >> 172.21.15.98:0/2816266307 conn(0x564dacc89650 msgr2=0x564dacc887b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f084e9a8740 1 -- 172.21.15.98:0/2816266307 shutdown_connections 2024-05-09T08:59:49.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f084e9a8740 1 -- 172.21.15.98:0/2816266307 wait complete. 2024-05-09T08:59:49.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.951+0000 7f084e9a8740 20 client.18192 trim_cache size 0 max 16384 2024-05-09T08:59:49.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.964+0000 7f084e9a8740 1 Processor -- start 2024-05-09T08:59:49.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.964+0000 7f084e9a8740 1 -- start start 2024-05-09T08:59:49.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.964+0000 7f084e9a8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc8d330 0x564dacca4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.964+0000 7f084e9a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564daccb0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.964+0000 7f084e9a8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 0x564daccb0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.964+0000 7f084e9a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08441b1050 con 0x564dacc8dc50 2024-05-09T08:59:49.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.964+0000 7f084e9a8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f08441b2b90 con 0x564dacc98320 2024-05-09T08:59:49.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.964+0000 7f084e9a8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f08441b1720 con 0x564dacc8d330 2024-05-09T08:59:49.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.964+0000 7f084915f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564daccb0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.964+0000 7f084915f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564daccb0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36870/0 (socket says 172.21.15.98:36870) 2024-05-09T08:59:49.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.964+0000 7f084915f640 1 -- 172.21.15.98:0/44958418 learned_addr learned my addr 172.21.15.98:0/44958418 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f084915f640 1 -- 172.21.15.98:0/44958418 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc8d330 msgr2=0x564dacca4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f084915f640 1 --2- 172.21.15.98:0/44958418 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc8d330 0x564dacca4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f084915f640 1 -- 172.21.15.98:0/44958418 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 msgr2=0x564daccb0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f084915f640 1 --2- 172.21.15.98:0/44958418 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 0x564daccb0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f084915f640 1 -- 172.21.15.98:0/44958418 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564dacc84f30 con 0x564dacc8dc50 2024-05-09T08:59:49.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f084915f640 1 --2- 172.21.15.98:0/44958418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564daccb0230 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f08340bb730 tx=0x7f0834005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77fc86c8e82d1fd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f082f7fe640 1 -- 172.21.15.98:0/44958418 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834049030 con 0x564dacc8dc50 2024-05-09T08:59:49.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f082f7fe640 1 -- 172.21.15.98:0/44958418 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f083403b580 con 0x564dacc8dc50 2024-05-09T08:59:49.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f082f7fe640 1 -- 172.21.15.98:0/44958418 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08340044e0 con 0x564dacc8dc50 2024-05-09T08:59:49.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f084e9a8740 1 -- 172.21.15.98:0/44958418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 msgr2=0x564daccb0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f084e9a8740 1 --2- 172.21.15.98:0/44958418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564daccb0230 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f08340bb730 tx=0x7f0834005890 comp rx=0 tx=0).stop 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f084e9a8740 1 -- 172.21.15.98:0/44958418 shutdown_connections 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f084e9a8740 1 --2- 172.21.15.98:0/44958418 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc98320 0x564daccb0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f084e9a8740 1 --2- 172.21.15.98:0/44958418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564daccb0230 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f084e9a8740 1 --2- 172.21.15.98:0/44958418 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc8d330 0x564dacca4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.965+0000 7f084e9a8740 1 -- 172.21.15.98:0/44958418 >> 172.21.15.98:0/44958418 conn(0x564dacc89650 msgr2=0x564dacc90500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.966+0000 7f084e9a8740 1 -- 172.21.15.98:0/44958418 shutdown_connections 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.966+0000 7f084e9a8740 1 -- 172.21.15.98:0/44958418 wait complete. 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.966+0000 7f084e9a8740 1 -- start start 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.966+0000 7f084e9a8740 1 Processor -- start 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.966+0000 7f084e9a8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc8d330 0x564dacc9c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.966+0000 7f084e9a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc9c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.966+0000 7f084e9a8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc98320 0x564dacc9cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.966+0000 7f084e9a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08441b24c0 con 0x564dacc8dc50 2024-05-09T08:59:49.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.966+0000 7f084e9a8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f08441b1df0 con 0x564dacc8d330 2024-05-09T08:59:49.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.966+0000 7f084e9a8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f08441af050 con 0x564dacc98320 2024-05-09T08:59:49.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.966+0000 7f084e9a8740 10 client.0 _ll_register_callbacks cb 0x7f08340b82b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:49.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.966+0000 7f084915f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc9c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.966+0000 7f084915f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc9c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36880/0 (socket says 172.21.15.98:36880) 2024-05-09T08:59:49.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.966+0000 7f084915f640 1 -- 172.21.15.98:0/3021634773 learned_addr learned my addr 172.21.15.98:0/3021634773 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:49.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f0849960640 1 --2- 172.21.15.98:0/3021634773 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc8d330 0x564dacc9c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084915f640 1 -- 172.21.15.98:0/3021634773 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc98320 msgr2=0x564dacc9cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084915f640 1 --2- 172.21.15.98:0/3021634773 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc98320 0x564dacc9cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084915f640 1 -- 172.21.15.98:0/3021634773 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc8d330 msgr2=0x564dacc9c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:49.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084915f640 1 --2- 172.21.15.98:0/3021634773 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc8d330 0x564dacc9c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084915f640 1 -- 172.21.15.98:0/3021634773 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564dacd0dca0 con 0x564dacc8dc50 2024-05-09T08:59:49.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f0849960640 1 --2- 172.21.15.98:0/3021634773 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc8d330 0x564dacc9c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:49.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084915f640 1 --2- 172.21.15.98:0/3021634773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc9c680 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f08340c4680 tx=0x7f0834005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74dce27c4fc0ed9b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:49.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f082d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:49.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f082d7fa640 1 -- 172.21.15.98:0/3021634773 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834049070 con 0x564dacc8dc50 2024-05-09T08:59:49.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f082d7fa640 1 -- 172.21.15.98:0/3021634773 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f08340c4920 con 0x564dacc8dc50 2024-05-09T08:59:49.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084e9a8740 1 -- 172.21.15.98:0/3021634773 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564dacbfa6c0 con 0x564dacc8dc50 2024-05-09T08:59:49.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f082d7fa640 1 -- 172.21.15.98:0/3021634773 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f08340c4c00 con 0x564dacc8dc50 2024-05-09T08:59:49.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084e9a8740 10 client.18198 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:49.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084e9a8740 1 -- 172.21.15.98:0/3021634773 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564dacd0dca0 con 0x564dacc8dc50 2024-05-09T08:59:49.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f082d7fa640 1 -- 172.21.15.98:0/3021634773 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564dacbfa6c0 con 0x564dacc8dc50 2024-05-09T08:59:49.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f082d7fa640 1 -- 172.21.15.98:0/3021634773 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f0834031c60 con 0x564dacc8dc50 2024-05-09T08:59:49.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084e9a8740 10 client.18198 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:49.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084e9a8740 10 client.18198 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T08:59:49.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084e9a8740 20 client.18198 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:49.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084e9a8740 1 --2- 172.21.15.98:0/3021634773 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x564dacd0f2c0 0x564dacd116e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:49.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084e9a8740 4 client.18198 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.967+0000 7f084e9a8740 1 -- 172.21.15.98:0/3021634773 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564dacb77820 con 0x564dacd0f2c0 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f0849960640 1 --2- 172.21.15.98:0/3021634773 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x564dacd0f2c0 0x564dacd116e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f082d7fa640 10 client.18198 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f0849960640 1 --2- 172.21.15.98:0/3021634773 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x564dacd0f2c0 0x564dacd116e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8ca5992b6541e4e3 server_cookie=4cf4710fd4853429 in_seq=0 out_seq=0 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f082d7fa640 1 -- 172.21.15.98:0/3021634773 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564dacb77820 con 0x564dacd0f2c0 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f082d7fa640 10 client.18198 handle_command_reply: tid=0 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f084e9a8740 1 client.18198 shutdown 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f084e9a8740 1 -- 172.21.15.98:0/3021634773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 msgr2=0x564dacc9c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f084e9a8740 1 --2- 172.21.15.98:0/3021634773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc9c680 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f08340c4680 tx=0x7f0834005900 comp rx=0 tx=0).stop 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f084e9a8740 1 -- 172.21.15.98:0/3021634773 shutdown_connections 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f084e9a8740 1 --2- 172.21.15.98:0/3021634773 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x564dacd0f2c0 0x564dacd116e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f084e9a8740 1 --2- 172.21.15.98:0/3021634773 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564dacc98320 0x564dacc9cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f084e9a8740 1 --2- 172.21.15.98:0/3021634773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564dacc8dc50 0x564dacc9c680 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f082d7fa640 0 client.18198 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T08:59:49.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f084e9a8740 1 --2- 172.21.15.98:0/3021634773 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564dacc8d330 0x564dacc9c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f084e9a8740 1 -- 172.21.15.98:0/3021634773 >> 172.21.15.98:0/3021634773 conn(0x564dacc89650 msgr2=0x564dacc87f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f084e9a8740 1 -- 172.21.15.98:0/3021634773 shutdown_connections 2024-05-09T08:59:49.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f084e9a8740 1 -- 172.21.15.98:0/3021634773 wait complete. 2024-05-09T08:59:49.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.968+0000 7f084e9a8740 20 client.18198 trim_cache size 0 max 16384 2024-05-09T08:59:49.977 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:59:49.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.969+0000 7f084bbc4640 1 -- 172.21.15.98:0/2265166810 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f08300a2d00 msgr2=0x7f08300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.969+0000 7f084bbc4640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f08300a2d00 0x7f08300a51c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f084406b100 tx=0x7f083c004a40 comp rx=0 tx=0).stop 2024-05-09T08:59:49.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.969+0000 7f084bbc4640 1 -- 172.21.15.98:0/2265166810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0844074420 msgr2=0x7f084419a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:49.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.969+0000 7f084bbc4640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0844074420 0x7f084419a890 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f0834006010 tx=0x7f08340042f0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.969+0000 7f084bbc4640 1 -- 172.21.15.98:0/2265166810 shutdown_connections 2024-05-09T08:59:49.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.969+0000 7f084bbc4640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f08300a2d00 0x7f08300a51c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.969+0000 7f084bbc4640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f084410e110 0x7f084419ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.969+0000 7f084bbc4640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0844074420 0x7f084419a890 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.969+0000 7f084bbc4640 1 --2- 172.21.15.98:0/2265166810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0844073b00 0x7f084419a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:49.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.969+0000 7f084bbc4640 1 -- 172.21.15.98:0/2265166810 >> 172.21.15.98:0/2265166810 conn(0x7f0844078c80 msgr2=0x7f08440fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:49.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.969+0000 7f084bbc4640 1 -- 172.21.15.98:0/2265166810 shutdown_connections 2024-05-09T08:59:49.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:49.969+0000 7f084bbc4640 1 -- 172.21.15.98:0/2265166810 wait complete. 2024-05-09T08:59:49.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:59:50.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.084+0000 7fc683f61640 1 -- 172.21.15.98:0/3628928180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc67c0ffd30 msgr2=0x7fc67c100190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.084+0000 7fc683f61640 1 --2- 172.21.15.98:0/3628928180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc67c0ffd30 0x7fc67c100190 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fc66c009900 tx=0x7fc66c02a410 comp rx=0 tx=0).stop 2024-05-09T08:59:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.084+0000 7fc683f61640 1 -- 172.21.15.98:0/3628928180 shutdown_connections 2024-05-09T08:59:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.084+0000 7fc683f61640 1 --2- 172.21.15.98:0/3628928180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67c1007b0 0x7fc67c104470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.084+0000 7fc683f61640 1 --2- 172.21.15.98:0/3628928180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc67c0ffd30 0x7fc67c100190 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.084+0000 7fc683f61640 1 --2- 172.21.15.98:0/3628928180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc67c106970 0x7fc67c106d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.084+0000 7fc683f61640 1 -- 172.21.15.98:0/3628928180 >> 172.21.15.98:0/3628928180 conn(0x7fc67c076cc0 msgr2=0x7fc67c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.084+0000 7fc683f61640 1 -- 172.21.15.98:0/3628928180 shutdown_connections 2024-05-09T08:59:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.084+0000 7fc683f61640 1 -- 172.21.15.98:0/3628928180 wait complete. 2024-05-09T08:59:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc683f61640 1 Processor -- start 2024-05-09T08:59:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc683f61640 1 -- start start 2024-05-09T08:59:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc683f61640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc67c0ffd30 0x7fc67c1a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc683f61640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc67c1007b0 0x7fc67c1a4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc683f61640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67c106970 0x7fc67c19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc683f61640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1183c0 con 0x7fc67c0ffd30 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc683f61640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c118240 con 0x7fc67c1007b0 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc683f61640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c118540 con 0x7fc67c106970 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc681cfd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc67c0ffd30 0x7fc67c1a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc681cfd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc67c0ffd30 0x7fc67c1a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36904/0 (socket says 172.21.15.98:36904) 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc681cfd640 1 -- 172.21.15.98:0/2491778732 learned_addr learned my addr 172.21.15.98:0/2491778732 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc6814fc640 1 --2- 172.21.15.98:0/2491778732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc67c1007b0 0x7fc67c1a4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc681cfd640 1 -- 172.21.15.98:0/2491778732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67c106970 msgr2=0x7fc67c19e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc681cfd640 1 --2- 172.21.15.98:0/2491778732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67c106970 0x7fc67c19e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc681cfd640 1 -- 172.21.15.98:0/2491778732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc67c1007b0 msgr2=0x7fc67c1a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc681cfd640 1 --2- 172.21.15.98:0/2491778732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc67c1007b0 0x7fc67c1a4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.085+0000 7fc681cfd640 1 -- 172.21.15.98:0/2491778732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc67c19efe0 con 0x7fc67c0ffd30 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.086+0000 7fc681cfd640 1 --2- 172.21.15.98:0/2491778732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc67c0ffd30 0x7fc67c1a4660 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc67800d950 tx=0x7fc67800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.086+0000 7fc672ffd640 1 -- 172.21.15.98:0/2491778732 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc678017070 con 0x7fc67c0ffd30 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.086+0000 7fc683f61640 1 -- 172.21.15.98:0/2491778732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc67c19f2d0 con 0x7fc67c0ffd30 2024-05-09T08:59:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.086+0000 7fc672ffd640 1 -- 172.21.15.98:0/2491778732 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc678009c30 con 0x7fc67c0ffd30 2024-05-09T08:59:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.086+0000 7fc683f61640 1 -- 172.21.15.98:0/2491778732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc67c1ab5e0 con 0x7fc67c0ffd30 2024-05-09T08:59:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.086+0000 7fc672ffd640 1 -- 172.21.15.98:0/2491778732 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc678020060 con 0x7fc67c0ffd30 2024-05-09T08:59:50.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc672ffd640 1 -- 172.21.15.98:0/2491778732 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc6780109d0 con 0x7fc67c0ffd30 2024-05-09T08:59:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc686c6b740 1 Processor -- start 2024-05-09T08:59:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc686c6b740 1 -- start start 2024-05-09T08:59:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc686c6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132b2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc686c6b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 0x55e0132bcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc686c6b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132bf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc686c6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e013217c40 con 0x55e0132b2330 2024-05-09T08:59:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc686c6b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e0131bf580 con 0x55e0132bd320 2024-05-09T08:59:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc686c6b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e013213c00 con 0x55e0132b2c50 2024-05-09T08:59:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc6824fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132b2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc6824fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132b2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36910/0 (socket says 172.21.15.98:36910) 2024-05-09T08:59:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc6824fe640 1 -- 172.21.15.98:0/965210 learned_addr learned my addr 172.21.15.98:0/965210 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc672ffd640 1 --2- 172.21.15.98:0/2491778732 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc6680a2d00 0x7fc6680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc6814fc640 1 --2- 172.21.15.98:0/965210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132bf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.089+0000 7fc681cfd640 1 --2- 172.21.15.98:0/965210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 0x55e0132bcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.090+0000 7fc6824fe640 1 -- 172.21.15.98:0/965210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 msgr2=0x55e0132bcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.090+0000 7fc6824fe640 1 --2- 172.21.15.98:0/965210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 0x55e0132bcde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.090+0000 7fc6824fe640 1 -- 172.21.15.98:0/965210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 msgr2=0x55e0132bf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.090+0000 7fc6824fe640 1 --2- 172.21.15.98:0/965210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132bf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.090+0000 7fc6824fe640 1 -- 172.21.15.98:0/965210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e012f2b3f0 con 0x55e0132b2330 2024-05-09T08:59:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.090+0000 7fc6814fc640 1 --2- 172.21.15.98:0/965210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132bf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.090+0000 7fc672ffd640 1 -- 172.21.15.98:0/2491778732 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fc6780cf4e0 con 0x7fc67c0ffd30 2024-05-09T08:59:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.090+0000 7fc6824fe640 1 --2- 172.21.15.98:0/2491778732 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc6680a2d00 0x7fc6680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.090+0000 7fc6824fe640 1 --2- 172.21.15.98:0/965210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132b2710 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc674009f00 tx=0x7fc67400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957943fec2db8994 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.090+0000 7fc667fff640 1 -- 172.21.15.98:0/965210 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67401b550 con 0x55e0132b2330 2024-05-09T08:59:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.090+0000 7fc6824fe640 1 --2- 172.21.15.98:0/2491778732 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc6680a2d00 0x7fc6680a51c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc67401f000 tx=0x7fc674020000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.090+0000 7fc667fff640 1 -- 172.21.15.98:0/965210 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc67401bbd0 con 0x55e0132b2330 2024-05-09T08:59:50.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.090+0000 7fc667fff640 1 -- 172.21.15.98:0/965210 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc674011bb0 con 0x55e0132b2330 2024-05-09T08:59:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.091+0000 7fc686c6b740 1 -- 172.21.15.98:0/965210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 msgr2=0x55e0132b2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.091+0000 7fc686c6b740 1 --2- 172.21.15.98:0/965210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132b2710 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc674009f00 tx=0x7fc67400e6c0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.091+0000 7fc686c6b740 1 -- 172.21.15.98:0/965210 shutdown_connections 2024-05-09T08:59:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.091+0000 7fc686c6b740 1 --2- 172.21.15.98:0/965210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132bf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.091+0000 7fc686c6b740 1 --2- 172.21.15.98:0/965210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 0x55e0132bcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.091+0000 7fc686c6b740 1 --2- 172.21.15.98:0/965210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132b2710 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.091+0000 7fc686c6b740 1 -- 172.21.15.98:0/965210 >> 172.21.15.98:0/965210 conn(0x55e0132ae650 msgr2=0x55e0132aea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.091+0000 7fc686c6b740 1 -- 172.21.15.98:0/965210 shutdown_connections 2024-05-09T08:59:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.091+0000 7fc686c6b740 1 -- 172.21.15.98:0/965210 wait complete. 2024-05-09T08:59:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.091+0000 7fc686c6b740 1 -- start start 2024-05-09T08:59:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.091+0000 7fc686c6b740 1 Processor -- start 2024-05-09T08:59:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc686c6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132ba300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc686c6b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 0x55e0132ba840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc686c6b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132bad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc686c6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e0132c0110 con 0x55e0132b2330 2024-05-09T08:59:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc686c6b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e0131d2c40 con 0x55e0132bd320 2024-05-09T08:59:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc686c6b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e0132c0290 con 0x55e0132b2c50 2024-05-09T08:59:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc686c6b740 10 client.0 _ll_register_callbacks cb 0x55e012f2abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc681cfd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132ba300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc681cfd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132ba300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36918/0 (socket says 172.21.15.98:36918) 2024-05-09T08:59:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc681cfd640 1 -- 172.21.15.98:0/27032470 learned_addr learned my addr 172.21.15.98:0/27032470 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc6824fe640 1 --2- 172.21.15.98:0/27032470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132bad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc681cfd640 1 -- 172.21.15.98:0/27032470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 msgr2=0x55e0132ba840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc681cfd640 1 --2- 172.21.15.98:0/27032470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 0x55e0132ba840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc681cfd640 1 -- 172.21.15.98:0/27032470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 msgr2=0x55e0132bad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc681cfd640 1 --2- 172.21.15.98:0/27032470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132bad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc681cfd640 1 -- 172.21.15.98:0/27032470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e012ff57d0 con 0x55e0132b2330 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc6824fe640 1 --2- 172.21.15.98:0/27032470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132bad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc681cfd640 1 --2- 172.21.15.98:0/27032470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132ba300 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fc678010fd0 tx=0x7fc67809b0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632d4760d69eb15e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc6657fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc6657fa640 1 -- 172.21.15.98:0/27032470 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67809fa40 con 0x55e0132b2330 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc6657fa640 1 -- 172.21.15.98:0/27032470 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fc6780cf170 con 0x55e0132b2330 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc686c6b740 1 -- 172.21.15.98:0/27032470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc67401bbd0 con 0x55e0132b2330 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc6657fa640 1 -- 172.21.15.98:0/27032470 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc6780a4300 con 0x55e0132b2330 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc686c6b740 10 client.18210 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.092+0000 7fc686c6b740 1 -- 172.21.15.98:0/27032470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e0132cc360 con 0x55e0132b2330 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.093+0000 7fc6657fa640 1 -- 172.21.15.98:0/27032470 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc67401bbd0 con 0x55e0132b2330 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.093+0000 7fc6657fa640 1 -- 172.21.15.98:0/27032470 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fc6780afbb0 con 0x55e0132b2330 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.093+0000 7fc686c6b740 10 client.18210 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.093+0000 7fc686c6b740 10 client.18210 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.093+0000 7fc686c6b740 20 client.18210 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.093+0000 7fc686c6b740 1 --2- 172.21.15.98:0/27032470 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e0132cd770 0x55e0132cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.093+0000 7fc686c6b740 4 client.18210 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.093+0000 7fc686c6b740 1 -- 172.21.15.98:0/27032470 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e013225370 con 0x55e0132cd770 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.093+0000 7fc6814fc640 1 --2- 172.21.15.98:0/27032470 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e0132cd770 0x55e0132cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.093+0000 7fc6814fc640 1 --2- 172.21.15.98:0/27032470 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e0132cd770 0x55e0132cfba0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ef0c853781cdafe2 server_cookie=f439c1d026e342c6 in_seq=0 out_seq=0 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.093+0000 7fc6657fa640 10 client.18210 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.094+0000 7fc6657fa640 1 -- 172.21.15.98:0/27032470 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e013225370 con 0x55e0132cd770 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.094+0000 7fc6657fa640 10 client.18210 handle_command_reply: tid=0 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.094+0000 7fc686c6b740 1 client.18210 shutdown 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.094+0000 7fc686c6b740 1 -- 172.21.15.98:0/27032470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 msgr2=0x55e0132ba300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.094+0000 7fc686c6b740 1 --2- 172.21.15.98:0/27032470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132ba300 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fc678010fd0 tx=0x7fc67809b0b0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.094+0000 7fc686c6b740 1 -- 172.21.15.98:0/27032470 shutdown_connections 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.094+0000 7fc686c6b740 1 --2- 172.21.15.98:0/27032470 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e0132cd770 0x55e0132cfba0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.094+0000 7fc686c6b740 1 --2- 172.21.15.98:0/27032470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132bad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.094+0000 7fc686c6b740 1 --2- 172.21.15.98:0/27032470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 0x55e0132ba840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.094+0000 7fc686c6b740 1 --2- 172.21.15.98:0/27032470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132ba300 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.094+0000 7fc6657fa640 0 client.18210 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.094+0000 7fc686c6b740 1 -- 172.21.15.98:0/27032470 >> 172.21.15.98:0/27032470 conn(0x55e0132ae650 msgr2=0x55e0132b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.094+0000 7fc686c6b740 1 -- 172.21.15.98:0/27032470 shutdown_connections 2024-05-09T08:59:50.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.094+0000 7fc686c6b740 1 -- 172.21.15.98:0/27032470 wait complete. 2024-05-09T08:59:50.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.095+0000 7fc686c6b740 20 client.18210 trim_cache size 0 max 16384 2024-05-09T08:59:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc686c6b740 1 Processor -- start 2024-05-09T08:59:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc686c6b740 1 -- start start 2024-05-09T08:59:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc686c6b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2330 0x55e0132bca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc686c6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2c50 0x55e0132c8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc686c6b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132d5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc686c6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1b5420 con 0x55e0132b2c50 2024-05-09T08:59:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc686c6b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1b6f60 con 0x55e0132bd320 2024-05-09T08:59:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc686c6b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1b5af0 con 0x55e0132b2330 2024-05-09T08:59:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc6814fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2c50 0x55e0132c8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc6814fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2c50 0x55e0132c8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36926/0 (socket says 172.21.15.98:36926) 2024-05-09T08:59:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc6814fc640 1 -- 172.21.15.98:0/3490955989 learned_addr learned my addr 172.21.15.98:0/3490955989 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc6814fc640 1 -- 172.21.15.98:0/3490955989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2330 msgr2=0x55e0132bca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc6814fc640 1 --2- 172.21.15.98:0/3490955989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2330 0x55e0132bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc6814fc640 1 -- 172.21.15.98:0/3490955989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 msgr2=0x55e0132d5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc6814fc640 1 --2- 172.21.15.98:0/3490955989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132d5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc6814fc640 1 -- 172.21.15.98:0/3490955989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0132ae1a0 con 0x55e0132b2c50 2024-05-09T08:59:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc6814fc640 1 --2- 172.21.15.98:0/3490955989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2c50 0x55e0132c8fd0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fc66c009900 tx=0x7fc66c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b0393f9c3c2220a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.108+0000 7fc6677fe640 1 -- 172.21.15.98:0/3490955989 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c035040 con 0x55e0132b2c50 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.109+0000 7fc6677fe640 1 -- 172.21.15.98:0/3490955989 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc66c042b50 con 0x55e0132b2c50 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.109+0000 7fc6677fe640 1 -- 172.21.15.98:0/3490955989 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c042250 con 0x55e0132b2c50 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.109+0000 7fc686c6b740 1 -- 172.21.15.98:0/3490955989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2c50 msgr2=0x55e0132c8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.109+0000 7fc686c6b740 1 --2- 172.21.15.98:0/3490955989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2c50 0x55e0132c8fd0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fc66c009900 tx=0x7fc66c005a20 comp rx=0 tx=0).stop 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.109+0000 7fc686c6b740 1 -- 172.21.15.98:0/3490955989 shutdown_connections 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.109+0000 7fc686c6b740 1 --2- 172.21.15.98:0/3490955989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132d5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.109+0000 7fc686c6b740 1 --2- 172.21.15.98:0/3490955989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2c50 0x55e0132c8fd0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.109+0000 7fc686c6b740 1 --2- 172.21.15.98:0/3490955989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2330 0x55e0132bca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.109+0000 7fc686c6b740 1 -- 172.21.15.98:0/3490955989 >> 172.21.15.98:0/3490955989 conn(0x55e0132ae650 msgr2=0x55e0132aea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.109+0000 7fc686c6b740 1 -- 172.21.15.98:0/3490955989 shutdown_connections 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.109+0000 7fc686c6b740 1 -- 172.21.15.98:0/3490955989 wait complete. 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.109+0000 7fc686c6b740 1 -- start start 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.109+0000 7fc686c6b740 1 Processor -- start 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc686c6b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132b6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc686c6b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 0x55e0132b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc686c6b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc686c6b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1b6890 con 0x55e0132b2330 2024-05-09T08:59:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc686c6b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1b61c0 con 0x55e0132bd320 2024-05-09T08:59:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc686c6b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1b3420 con 0x55e0132b2c50 2024-05-09T08:59:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc686c6b740 10 client.0 _ll_register_callbacks cb 0x7fc67800df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc681cfd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132b6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc681cfd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132b6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36928/0 (socket says 172.21.15.98:36928) 2024-05-09T08:59:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc681cfd640 1 -- 172.21.15.98:0/1224516445 learned_addr learned my addr 172.21.15.98:0/1224516445 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc6824fe640 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc6814fc640 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 0x55e0132b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc681cfd640 1 -- 172.21.15.98:0/1224516445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 msgr2=0x55e0132b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc681cfd640 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 0x55e0132b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc681cfd640 1 -- 172.21.15.98:0/1224516445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 msgr2=0x55e0132b7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc681cfd640 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132b7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc681cfd640 1 -- 172.21.15.98:0/1224516445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e013332cb0 con 0x55e0132b2330 2024-05-09T08:59:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc6824fe640 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132b7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc6814fc640 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 0x55e0132b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc681cfd640 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132b6fb0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fc6780058c0 tx=0x7fc678097930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0334a940dfffc4c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc6657fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc6657fa640 1 -- 172.21.15.98:0/1224516445 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6780b9060 con 0x55e0132b2330 2024-05-09T08:59:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc6657fa640 1 -- 172.21.15.98:0/1224516445 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fc6780cf170 con 0x55e0132b2330 2024-05-09T08:59:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.110+0000 7fc686c6b740 1 -- 172.21.15.98:0/1224516445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc66c042b50 con 0x55e0132b2330 2024-05-09T08:59:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.111+0000 7fc6657fa640 1 -- 172.21.15.98:0/1224516445 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc67809fa40 con 0x55e0132b2330 2024-05-09T08:59:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.111+0000 7fc686c6b740 10 client.18216 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.111+0000 7fc686c6b740 1 -- 172.21.15.98:0/1224516445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e013333f90 con 0x55e0132b2330 2024-05-09T08:59:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.111+0000 7fc6657fa640 1 -- 172.21.15.98:0/1224516445 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc66c042b50 con 0x55e0132b2330 2024-05-09T08:59:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.111+0000 7fc6657fa640 1 -- 172.21.15.98:0/1224516445 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fc67809a550 con 0x55e0132b2330 2024-05-09T08:59:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.111+0000 7fc686c6b740 10 client.18216 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.111+0000 7fc686c6b740 10 client.18216 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.111+0000 7fc686c6b740 20 client.18216 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.111+0000 7fc686c6b740 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e013334690 0x55e013336ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.111+0000 7fc686c6b740 4 client.18216 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.111+0000 7fc686c6b740 1 -- 172.21.15.98:0/1224516445 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e01321f6c0 con 0x55e013334690 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.111+0000 7fc6814fc640 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e013334690 0x55e013336ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc6657fa640 10 client.18216 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc6814fc640 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e013334690 0x55e013336ab0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f606a57e849f6090 server_cookie=2b5905b895a9c098 in_seq=0 out_seq=0 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc6657fa640 1 -- 172.21.15.98:0/1224516445 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e01321f6c0 con 0x55e013334690 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc6657fa640 10 client.18216 handle_command_reply: tid=0 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc686c6b740 1 client.18216 shutdown 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc686c6b740 1 -- 172.21.15.98:0/1224516445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 msgr2=0x55e0132b6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc686c6b740 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132b6fb0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fc6780058c0 tx=0x7fc678097930 comp rx=0 tx=0).stop 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc686c6b740 1 -- 172.21.15.98:0/1224516445 shutdown_connections 2024-05-09T08:59:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc686c6b740 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e013334690 0x55e013336ab0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc686c6b740 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e0132bd320 0x55e0132b7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc6657fa640 0 client.18216 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T08:59:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc686c6b740 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0132b2c50 0x55e0132b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc686c6b740 1 --2- 172.21.15.98:0/1224516445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0132b2330 0x55e0132b6fb0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc686c6b740 1 -- 172.21.15.98:0/1224516445 >> 172.21.15.98:0/1224516445 conn(0x55e0132ae650 msgr2=0x55e0132acf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc686c6b740 1 -- 172.21.15.98:0/1224516445 shutdown_connections 2024-05-09T08:59:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc686c6b740 1 -- 172.21.15.98:0/1224516445 wait complete. 2024-05-09T08:59:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.112+0000 7fc686c6b740 20 client.18216 trim_cache size 0 max 16384 2024-05-09T08:59:50.118 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:59:50.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.113+0000 7fc683f61640 1 -- 172.21.15.98:0/2491778732 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc6680a2d00 msgr2=0x7fc6680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.113+0000 7fc683f61640 1 --2- 172.21.15.98:0/2491778732 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc6680a2d00 0x7fc6680a51c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc67401f000 tx=0x7fc674020000 comp rx=0 tx=0).stop 2024-05-09T08:59:50.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.113+0000 7fc683f61640 1 -- 172.21.15.98:0/2491778732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc67c0ffd30 msgr2=0x7fc67c1a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.113+0000 7fc683f61640 1 --2- 172.21.15.98:0/2491778732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc67c0ffd30 0x7fc67c1a4660 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc67800d950 tx=0x7fc67800de10 comp rx=0 tx=0).stop 2024-05-09T08:59:50.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.113+0000 7fc683f61640 1 -- 172.21.15.98:0/2491778732 shutdown_connections 2024-05-09T08:59:50.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.113+0000 7fc683f61640 1 --2- 172.21.15.98:0/2491778732 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc6680a2d00 0x7fc6680a51c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.113+0000 7fc683f61640 1 --2- 172.21.15.98:0/2491778732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67c106970 0x7fc67c19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.113+0000 7fc683f61640 1 --2- 172.21.15.98:0/2491778732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc67c1007b0 0x7fc67c1a4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.113+0000 7fc683f61640 1 --2- 172.21.15.98:0/2491778732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc67c0ffd30 0x7fc67c1a4660 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.113+0000 7fc683f61640 1 -- 172.21.15.98:0/2491778732 >> 172.21.15.98:0/2491778732 conn(0x7fc67c076cc0 msgr2=0x7fc67c102000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.113+0000 7fc683f61640 1 -- 172.21.15.98:0/2491778732 shutdown_connections 2024-05-09T08:59:50.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.113+0000 7fc683f61640 1 -- 172.21.15.98:0/2491778732 wait complete. 2024-05-09T08:59:50.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:59:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.226+0000 7f3e247a2640 1 -- 172.21.15.98:0/2652774051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e1c100d80 msgr2=0x7f3e1c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.226+0000 7f3e247a2640 1 --2- 172.21.15.98:0/2652774051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e1c100d80 0x7f3e1c1064b0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f3e10009900 tx=0x7f3e1002a410 comp rx=0 tx=0).stop 2024-05-09T08:59:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.226+0000 7f3e247a2640 1 -- 172.21.15.98:0/2652774051 shutdown_connections 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.226+0000 7f3e247a2640 1 --2- 172.21.15.98:0/2652774051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e1c106b90 0x7f3e1c10e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.226+0000 7f3e247a2640 1 --2- 172.21.15.98:0/2652774051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e1c100d80 0x7f3e1c1064b0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.226+0000 7f3e247a2640 1 --2- 172.21.15.98:0/2652774051 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e1c1003d0 0x7f3e1c1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.226+0000 7f3e247a2640 1 -- 172.21.15.98:0/2652774051 >> 172.21.15.98:0/2652774051 conn(0x7f3e1c0759c0 msgr2=0x7f3e1c075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.226+0000 7f3e247a2640 1 -- 172.21.15.98:0/2652774051 shutdown_connections 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.226+0000 7f3e247a2640 1 -- 172.21.15.98:0/2652774051 wait complete. 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.227+0000 7f3e247a2640 1 Processor -- start 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.227+0000 7f3e247a2640 1 -- start start 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.227+0000 7f3e247a2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e1c1003d0 0x7f3e1c198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.227+0000 7f3e247a2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e1c106b90 0x7f3e1c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.227+0000 7f3e247a2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e1c19ca40 0x7f3e1c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.227+0000 7f3e247a2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c111c80 con 0x7f3e1c19ca40 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.227+0000 7f3e247a2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c111b00 con 0x7f3e1c1003d0 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.227+0000 7f3e247a2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c111e00 con 0x7f3e1c106b90 2024-05-09T08:59:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.227+0000 7f3e22d3f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e1c19ca40 0x7f3e1c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.227+0000 7f3e22d3f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e1c19ca40 0x7f3e1c19cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36938/0 (socket says 172.21.15.98:36938) 2024-05-09T08:59:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.227+0000 7f3e22d3f640 1 -- 172.21.15.98:0/439847532 learned_addr learned my addr 172.21.15.98:0/439847532 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e2253e640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e1c1003d0 0x7f3e1c198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e21d3d640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e1c106b90 0x7f3e1c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e22d3f640 1 -- 172.21.15.98:0/439847532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e1c106b90 msgr2=0x7f3e1c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e22d3f640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e1c106b90 0x7f3e1c1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e22d3f640 1 -- 172.21.15.98:0/439847532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e1c1003d0 msgr2=0x7f3e1c198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e22d3f640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e1c1003d0 0x7f3e1c198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e22d3f640 1 -- 172.21.15.98:0/439847532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1c19d530 con 0x7f3e1c19ca40 2024-05-09T08:59:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e2253e640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e1c1003d0 0x7f3e1c198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e21d3d640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e1c106b90 0x7f3e1c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e22d3f640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e1c19ca40 0x7f3e1c19cef0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400d750 tx=0x7f3e1400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e1b7fe640 1 -- 172.21.15.98:0/439847532 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e14017070 con 0x7f3e1c19ca40 2024-05-09T08:59:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e1b7fe640 1 -- 172.21.15.98:0/439847532 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e14005500 con 0x7f3e1c19ca40 2024-05-09T08:59:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e247a2640 1 -- 172.21.15.98:0/439847532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1c19d820 con 0x7f3e1c19ca40 2024-05-09T08:59:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e1b7fe640 1 -- 172.21.15.98:0/439847532 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e14004d90 con 0x7f3e1c19ca40 2024-05-09T08:59:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.228+0000 7f3e247a2640 1 -- 172.21.15.98:0/439847532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e1c1a5010 con 0x7f3e1c19ca40 2024-05-09T08:59:50.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.231+0000 7f3e1b7fe640 1 -- 172.21.15.98:0/439847532 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3e14004f30 con 0x7f3e1c19ca40 2024-05-09T08:59:50.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.231+0000 7f3e27592740 1 Processor -- start 2024-05-09T08:59:50.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.231+0000 7f3e27592740 1 -- start start 2024-05-09T08:59:50.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.231+0000 7f3e27592740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7230 0x56168e0d7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.231+0000 7f3e27592740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7b50 0x56168e0e61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.231+0000 7f3e27592740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0e6730 0x56168e0e8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.231+0000 7f3e27592740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56168e03cc40 con 0x56168e0d7b50 2024-05-09T08:59:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.231+0000 7f3e27592740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56168dfe4580 con 0x56168e0d7230 2024-05-09T08:59:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.231+0000 7f3e27592740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56168e038c00 con 0x56168e0e6730 2024-05-09T08:59:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e2253e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7b50 0x56168e0e61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e2253e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7b50 0x56168e0e61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36942/0 (socket says 172.21.15.98:36942) 2024-05-09T08:59:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e2253e640 1 -- 172.21.15.98:0/2014735880 learned_addr learned my addr 172.21.15.98:0/2014735880 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e21d3d640 1 --2- 172.21.15.98:0/2014735880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7230 0x56168e0d7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e2253e640 1 -- 172.21.15.98:0/2014735880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0e6730 msgr2=0x56168e0e8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e2253e640 1 --2- 172.21.15.98:0/2014735880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0e6730 0x56168e0e8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e2253e640 1 -- 172.21.15.98:0/2014735880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7230 msgr2=0x56168e0d7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e1b7fe640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e040a2d00 0x7f3e040a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e2253e640 1 --2- 172.21.15.98:0/2014735880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7230 0x56168e0d7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e2253e640 1 -- 172.21.15.98:0/2014735880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56168dd503f0 con 0x56168e0d7b50 2024-05-09T08:59:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e21d3d640 1 --2- 172.21.15.98:0/2014735880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7230 0x56168e0d7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e22d3f640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e040a2d00 0x7f3e040a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e2253e640 1 --2- 172.21.15.98:0/2014735880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7b50 0x56168e0e61f0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0103a0 tx=0x7f3e0c010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1891b295a8b8e54 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e1b7fe640 1 -- 172.21.15.98:0/439847532 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f3e140cf210 con 0x7f3e1c19ca40 2024-05-09T08:59:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e18ff9640 1 -- 172.21.15.98:0/2014735880 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c016070 con 0x56168e0d7b50 2024-05-09T08:59:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e18ff9640 1 -- 172.21.15.98:0/2014735880 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e0c012bf0 con 0x56168e0d7b50 2024-05-09T08:59:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.232+0000 7f3e18ff9640 1 -- 172.21.15.98:0/2014735880 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c01bd30 con 0x56168e0d7b50 2024-05-09T08:59:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.233+0000 7f3e27592740 1 -- 172.21.15.98:0/2014735880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7b50 msgr2=0x56168e0e61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.233+0000 7f3e22d3f640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e040a2d00 0x7f3e040a51c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3e1409b2b0 tx=0x7f3e14077040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.233+0000 7f3e27592740 1 --2- 172.21.15.98:0/2014735880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7b50 0x56168e0e61f0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0103a0 tx=0x7f3e0c010860 comp rx=0 tx=0).stop 2024-05-09T08:59:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.233+0000 7f3e27592740 1 -- 172.21.15.98:0/2014735880 shutdown_connections 2024-05-09T08:59:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.233+0000 7f3e27592740 1 --2- 172.21.15.98:0/2014735880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0e6730 0x56168e0e8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.233+0000 7f3e27592740 1 --2- 172.21.15.98:0/2014735880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7b50 0x56168e0e61f0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.233+0000 7f3e27592740 1 --2- 172.21.15.98:0/2014735880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7230 0x56168e0d7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.233+0000 7f3e27592740 1 -- 172.21.15.98:0/2014735880 >> 172.21.15.98:0/2014735880 conn(0x56168e0d3650 msgr2=0x56168e0d3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.233+0000 7f3e27592740 1 -- 172.21.15.98:0/2014735880 shutdown_connections 2024-05-09T08:59:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.233+0000 7f3e27592740 1 -- 172.21.15.98:0/2014735880 wait complete. 2024-05-09T08:59:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.233+0000 7f3e27592740 1 -- start start 2024-05-09T08:59:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.233+0000 7f3e27592740 1 Processor -- start 2024-05-09T08:59:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e27592740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7230 0x56168e0e36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e27592740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7b50 0x56168e0e3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e27592740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0e6730 0x56168e0e4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e27592740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56168e0e94f0 con 0x56168e0d7b50 2024-05-09T08:59:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e27592740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56168dff7c40 con 0x56168e0d7230 2024-05-09T08:59:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e27592740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56168e0e9670 con 0x56168e0e6730 2024-05-09T08:59:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e27592740 10 client.0 _ll_register_callbacks cb 0x56168dd4fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e2253e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7b50 0x56168e0e3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e2253e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7b50 0x56168e0e3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36958/0 (socket says 172.21.15.98:36958) 2024-05-09T08:59:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e2253e640 1 -- 172.21.15.98:0/3126857626 learned_addr learned my addr 172.21.15.98:0/3126857626 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e2253e640 1 -- 172.21.15.98:0/3126857626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0e6730 msgr2=0x56168e0e4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e2253e640 1 --2- 172.21.15.98:0/3126857626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0e6730 0x56168e0e4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e2253e640 1 -- 172.21.15.98:0/3126857626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7230 msgr2=0x56168e0e36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e2253e640 1 --2- 172.21.15.98:0/3126857626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7230 0x56168e0e36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e2253e640 1 -- 172.21.15.98:0/3126857626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56168de1a7d0 con 0x56168e0d7b50 2024-05-09T08:59:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e2253e640 1 --2- 172.21.15.98:0/3126857626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7b50 0x56168e0e3c00 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c01ba10 tx=0x7f3e0c01b510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ebbe72e899f485c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e01ffb640 1 -- 172.21.15.98:0/3126857626 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c029060 con 0x56168e0d7b50 2024-05-09T08:59:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e01ffb640 1 -- 172.21.15.98:0/3126857626 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f3e0c01bcc0 con 0x56168e0d7b50 2024-05-09T08:59:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.234+0000 7f3e27592740 1 -- 172.21.15.98:0/3126857626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e0c012bf0 con 0x56168e0d7b50 2024-05-09T08:59:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.235+0000 7f3e01ffb640 1 -- 172.21.15.98:0/3126857626 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3e0c030050 con 0x56168e0d7b50 2024-05-09T08:59:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.235+0000 7f3e27592740 10 client.18228 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.235+0000 7f3e27592740 1 -- 172.21.15.98:0/3126857626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56168e0f57b0 con 0x56168e0d7b50 2024-05-09T08:59:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.235+0000 7f3e01ffb640 1 -- 172.21.15.98:0/3126857626 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e0c012bf0 con 0x56168e0d7b50 2024-05-09T08:59:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.235+0000 7f3e01ffb640 1 -- 172.21.15.98:0/3126857626 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3e0c00f040 con 0x56168e0d7b50 2024-05-09T08:59:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.235+0000 7f3e27592740 10 client.18228 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.235+0000 7f3e27592740 10 client.18228 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:59:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.235+0000 7f3e27592740 20 client.18228 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.235+0000 7f3e27592740 1 --2- 172.21.15.98:0/3126857626 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56168e0f6c10 0x56168e0f9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.235+0000 7f3e27592740 4 client.18228 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.235+0000 7f3e27592740 1 -- 172.21.15.98:0/3126857626 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56168e04a370 con 0x56168e0f6c10 2024-05-09T08:59:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.235+0000 7f3e21d3d640 1 --2- 172.21.15.98:0/3126857626 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56168e0f6c10 0x56168e0f9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e01ffb640 10 client.18228 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:59:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e21d3d640 1 --2- 172.21.15.98:0/3126857626 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56168e0f6c10 0x56168e0f9000 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5db7ed8c35baf6d8 server_cookie=9a08b35efa52b23 in_seq=0 out_seq=0 2024-05-09T08:59:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e01ffb640 1 -- 172.21.15.98:0/3126857626 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56168e04a370 con 0x56168e0f6c10 2024-05-09T08:59:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e01ffb640 10 client.18228 handle_command_reply: tid=0 2024-05-09T08:59:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e27592740 1 client.18228 shutdown 2024-05-09T08:59:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e27592740 1 -- 172.21.15.98:0/3126857626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7b50 msgr2=0x56168e0e3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e27592740 1 --2- 172.21.15.98:0/3126857626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7b50 0x56168e0e3c00 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c01ba10 tx=0x7f3e0c01b510 comp rx=0 tx=0).stop 2024-05-09T08:59:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e27592740 1 -- 172.21.15.98:0/3126857626 shutdown_connections 2024-05-09T08:59:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e27592740 1 --2- 172.21.15.98:0/3126857626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0e6730 0x56168e0e4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e27592740 1 --2- 172.21.15.98:0/3126857626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7b50 0x56168e0e3c00 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e27592740 1 --2- 172.21.15.98:0/3126857626 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56168e0f6c10 0x56168e0f9000 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e27592740 1 --2- 172.21.15.98:0/3126857626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7230 0x56168e0e36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e01ffb640 0 client.18228 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:59:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e27592740 1 -- 172.21.15.98:0/3126857626 >> 172.21.15.98:0/3126857626 conn(0x56168e0d3650 msgr2=0x56168e0d93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e27592740 1 -- 172.21.15.98:0/3126857626 shutdown_connections 2024-05-09T08:59:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.236+0000 7f3e27592740 1 -- 172.21.15.98:0/3126857626 wait complete. 2024-05-09T08:59:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.237+0000 7f3e27592740 20 client.18228 trim_cache size 0 max 16384 2024-05-09T08:59:50.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.249+0000 7f3e27592740 1 Processor -- start 2024-05-09T08:59:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.249+0000 7f3e27592740 1 -- start start 2024-05-09T08:59:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e27592740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7230 0x56168e0e5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e27592740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0d7b50 0x56168e0e0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e27592740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0e6730 0x56168e0e0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e27592740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1accc0 con 0x56168e0d7230 2024-05-09T08:59:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e27592740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1ae800 con 0x56168e0e6730 2024-05-09T08:59:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e27592740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1ad390 con 0x56168e0d7b50 2024-05-09T08:59:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e21d3d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7230 0x56168e0e5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e21d3d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7230 0x56168e0e5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36962/0 (socket says 172.21.15.98:36962) 2024-05-09T08:59:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e21d3d640 1 -- 172.21.15.98:0/897851701 learned_addr learned my addr 172.21.15.98:0/897851701 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e21d3d640 1 -- 172.21.15.98:0/897851701 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0d7b50 msgr2=0x56168e0e0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e21d3d640 1 --2- 172.21.15.98:0/897851701 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0d7b50 0x56168e0e0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e21d3d640 1 -- 172.21.15.98:0/897851701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0e6730 msgr2=0x56168e0e0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e21d3d640 1 --2- 172.21.15.98:0/897851701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0e6730 0x56168e0e0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e21d3d640 1 -- 172.21.15.98:0/897851701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56168e0d7980 con 0x56168e0d7230 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e21d3d640 1 --2- 172.21.15.98:0/897851701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7230 0x56168e0e5df0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f3e10005c40 tx=0x7f3e10032e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d9311760e893e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e03fff640 1 -- 172.21.15.98:0/897851701 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10035040 con 0x56168e0d7230 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e03fff640 1 -- 172.21.15.98:0/897851701 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e10005520 con 0x56168e0d7230 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.250+0000 7f3e03fff640 1 -- 172.21.15.98:0/897851701 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10004d90 con 0x56168e0d7230 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.251+0000 7f3e27592740 1 -- 172.21.15.98:0/897851701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7230 msgr2=0x56168e0e5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.251+0000 7f3e27592740 1 --2- 172.21.15.98:0/897851701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7230 0x56168e0e5df0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f3e10005c40 tx=0x7f3e10032e10 comp rx=0 tx=0).stop 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.251+0000 7f3e27592740 1 -- 172.21.15.98:0/897851701 shutdown_connections 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.251+0000 7f3e27592740 1 --2- 172.21.15.98:0/897851701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0e6730 0x56168e0e0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.251+0000 7f3e27592740 1 --2- 172.21.15.98:0/897851701 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0d7b50 0x56168e0e0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.251+0000 7f3e27592740 1 --2- 172.21.15.98:0/897851701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7230 0x56168e0e5df0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.251+0000 7f3e27592740 1 -- 172.21.15.98:0/897851701 >> 172.21.15.98:0/897851701 conn(0x56168e0d3650 msgr2=0x56168e0de9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.251+0000 7f3e27592740 1 -- 172.21.15.98:0/897851701 shutdown_connections 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.251+0000 7f3e27592740 1 -- 172.21.15.98:0/897851701 wait complete. 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.251+0000 7f3e27592740 1 -- start start 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.251+0000 7f3e27592740 1 Processor -- start 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e27592740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7230 0x56168e0f9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e27592740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7b50 0x56168e0fa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e27592740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0e6730 0x56168e0fa720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e27592740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1ae130 con 0x56168e0d7230 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e27592740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1ada60 con 0x56168e0d7b50 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e27592740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1aacc0 con 0x56168e0e6730 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e27592740 10 client.0 _ll_register_callbacks cb 0x7f3e0c009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e21d3d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7230 0x56168e0f9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e21d3d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7230 0x56168e0f9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36970/0 (socket says 172.21.15.98:36970) 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e21d3d640 1 -- 172.21.15.98:0/2660708899 learned_addr learned my addr 172.21.15.98:0/2660708899 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e21d3d640 1 -- 172.21.15.98:0/2660708899 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0e6730 msgr2=0x56168e0fa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e21d3d640 1 --2- 172.21.15.98:0/2660708899 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0e6730 0x56168e0fa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e21d3d640 1 -- 172.21.15.98:0/2660708899 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7b50 msgr2=0x56168e0fa1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e2253e640 1 --2- 172.21.15.98:0/2660708899 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7b50 0x56168e0fa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e21d3d640 1 --2- 172.21.15.98:0/2660708899 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7b50 0x56168e0fa1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e21d3d640 1 -- 172.21.15.98:0/2660708899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56168e15c040 con 0x56168e0d7230 2024-05-09T08:59:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e21d3d640 1 --2- 172.21.15.98:0/2660708899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7230 0x56168e0f9ca0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f3e10005710 tx=0x7f3e1002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f873132134117d2b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e01ffb640 1 -- 172.21.15.98:0/2660708899 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10035040 con 0x56168e0d7230 2024-05-09T08:59:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e01ffb640 1 -- 172.21.15.98:0/2660708899 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f3e10004250 con 0x56168e0d7230 2024-05-09T08:59:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e27592740 1 -- 172.21.15.98:0/2660708899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e10005520 con 0x56168e0d7230 2024-05-09T08:59:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e01ffb640 1 -- 172.21.15.98:0/2660708899 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3e10004450 con 0x56168e0d7230 2024-05-09T08:59:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e27592740 10 client.18234 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.252+0000 7f3e27592740 1 -- 172.21.15.98:0/2660708899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56168e15d320 con 0x56168e0d7230 2024-05-09T08:59:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.253+0000 7f3e01ffb640 1 -- 172.21.15.98:0/2660708899 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e10005520 con 0x56168e0d7230 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.253+0000 7f3e01ffb640 1 -- 172.21.15.98:0/2660708899 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3e10045020 con 0x56168e0d7230 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.253+0000 7f3e27592740 10 client.18234 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.253+0000 7f3e27592740 10 client.18234 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.253+0000 7f3e27592740 20 client.18234 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.253+0000 7f3e27592740 1 --2- 172.21.15.98:0/2660708899 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56168e15d980 0x56168e15fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.253+0000 7f3e27592740 4 client.18234 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.253+0000 7f3e27592740 1 -- 172.21.15.98:0/2660708899 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56168e0446c0 con 0x56168e15d980 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.253+0000 7f3e2253e640 1 --2- 172.21.15.98:0/2660708899 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56168e15d980 0x56168e15fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.253+0000 7f3e01ffb640 10 client.18234 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.253+0000 7f3e2253e640 1 --2- 172.21.15.98:0/2660708899 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56168e15d980 0x56168e15fda0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b837c27a48a5b110 server_cookie=8b9a2e60404ee27d in_seq=0 out_seq=0 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e01ffb640 1 -- 172.21.15.98:0/2660708899 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56168e0446c0 con 0x56168e15d980 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e01ffb640 10 client.18234 handle_command_reply: tid=0 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e27592740 1 client.18234 shutdown 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e27592740 1 -- 172.21.15.98:0/2660708899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7230 msgr2=0x56168e0f9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e27592740 1 --2- 172.21.15.98:0/2660708899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7230 0x56168e0f9ca0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f3e10005710 tx=0x7f3e1002a410 comp rx=0 tx=0).stop 2024-05-09T08:59:50.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e27592740 1 -- 172.21.15.98:0/2660708899 shutdown_connections 2024-05-09T08:59:50.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e27592740 1 --2- 172.21.15.98:0/2660708899 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56168e0e6730 0x56168e0fa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e27592740 1 --2- 172.21.15.98:0/2660708899 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x56168e15d980 0x56168e15fda0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e27592740 1 --2- 172.21.15.98:0/2660708899 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56168e0d7b50 0x56168e0fa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e27592740 1 --2- 172.21.15.98:0/2660708899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56168e0d7230 0x56168e0f9ca0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e01ffb640 0 client.18234 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T08:59:50.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e27592740 1 -- 172.21.15.98:0/2660708899 >> 172.21.15.98:0/2660708899 conn(0x56168e0d3650 msgr2=0x56168e0d1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e27592740 1 -- 172.21.15.98:0/2660708899 shutdown_connections 2024-05-09T08:59:50.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e27592740 1 -- 172.21.15.98:0/2660708899 wait complete. 2024-05-09T08:59:50.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e27592740 20 client.18234 trim_cache size 0 max 16384 2024-05-09T08:59:50.260 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:59:50.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e247a2640 1 -- 172.21.15.98:0/439847532 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e040a2d00 msgr2=0x7f3e040a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e247a2640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e040a2d00 0x7f3e040a51c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3e1409b2b0 tx=0x7f3e14077040 comp rx=0 tx=0).stop 2024-05-09T08:59:50.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e247a2640 1 -- 172.21.15.98:0/439847532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e1c19ca40 msgr2=0x7f3e1c19cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e247a2640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e1c19ca40 0x7f3e1c19cef0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400d750 tx=0x7f3e1400dc10 comp rx=0 tx=0).stop 2024-05-09T08:59:50.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e247a2640 1 -- 172.21.15.98:0/439847532 shutdown_connections 2024-05-09T08:59:50.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e247a2640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e040a2d00 0x7f3e040a51c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e247a2640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e1c19ca40 0x7f3e1c19cef0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e247a2640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e1c106b90 0x7f3e1c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e247a2640 1 --2- 172.21.15.98:0/439847532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e1c1003d0 0x7f3e1c198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e247a2640 1 -- 172.21.15.98:0/439847532 >> 172.21.15.98:0/439847532 conn(0x7f3e1c0759c0 msgr2=0x7f3e1c0ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e247a2640 1 -- 172.21.15.98:0/439847532 shutdown_connections 2024-05-09T08:59:50.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.254+0000 7f3e247a2640 1 -- 172.21.15.98:0/439847532 wait complete. 2024-05-09T08:59:50.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:50 smithi098 ceph-mon[30092]: pgmap v286: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:59:50.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:50 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3188097495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:59:50.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:59:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.368+0000 7fcee33dc640 1 -- 172.21.15.98:0/247648538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcedc101800 msgr2=0x7fcedc101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.368+0000 7fcee33dc640 1 --2- 172.21.15.98:0/247648538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcedc101800 0x7fcedc101c80 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fced0009930 tx=0x7fced002a420 comp rx=0 tx=0).stop 2024-05-09T08:59:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.368+0000 7fcee33dc640 1 -- 172.21.15.98:0/247648538 shutdown_connections 2024-05-09T08:59:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.368+0000 7fcee33dc640 1 --2- 172.21.15.98:0/247648538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcedc1021c0 0x7fcedc10e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.368+0000 7fcee33dc640 1 --2- 172.21.15.98:0/247648538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcedc101800 0x7fcedc101c80 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.368+0000 7fcee33dc640 1 --2- 172.21.15.98:0/247648538 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcedc107810 0x7fcedc107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.368+0000 7fcee33dc640 1 -- 172.21.15.98:0/247648538 >> 172.21.15.98:0/247648538 conn(0x7fcedc0fd270 msgr2=0x7fcedc0ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.368+0000 7fcee33dc640 1 -- 172.21.15.98:0/247648538 shutdown_connections 2024-05-09T08:59:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.368+0000 7fcee33dc640 1 -- 172.21.15.98:0/247648538 wait complete. 2024-05-09T08:59:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.368+0000 7fcee33dc640 1 Processor -- start 2024-05-09T08:59:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.368+0000 7fcee33dc640 1 -- start start 2024-05-09T08:59:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee33dc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcedc101800 0x7fcedc198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee33dc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcedc1021c0 0x7fcedc198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee33dc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcedc107810 0x7fcedc19c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee33dc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc111c20 con 0x7fcedc107810 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee33dc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc111aa0 con 0x7fcedc1021c0 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee33dc640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc111da0 con 0x7fcedc101800 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee1979640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcedc107810 0x7fcedc19c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee1979640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcedc107810 0x7fcedc19c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36992/0 (socket says 172.21.15.98:36992) 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee1979640 1 -- 172.21.15.98:0/3845210592 learned_addr learned my addr 172.21.15.98:0/3845210592 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee0977640 1 --2- 172.21.15.98:0/3845210592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcedc1021c0 0x7fcedc198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee1979640 1 -- 172.21.15.98:0/3845210592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcedc101800 msgr2=0x7fcedc198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee1979640 1 --2- 172.21.15.98:0/3845210592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcedc101800 0x7fcedc198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee1979640 1 -- 172.21.15.98:0/3845210592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcedc1021c0 msgr2=0x7fcedc198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee1979640 1 --2- 172.21.15.98:0/3845210592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcedc1021c0 0x7fcedc198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee1979640 1 -- 172.21.15.98:0/3845210592 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcedc19d0f0 con 0x7fcedc107810 2024-05-09T08:59:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee0977640 1 --2- 172.21.15.98:0/3845210592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcedc1021c0 0x7fcedc198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee1979640 1 --2- 172.21.15.98:0/3845210592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcedc107810 0x7fcedc19c9c0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fced400cce0 tx=0x7fced400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcece7fc640 1 -- 172.21.15.98:0/3845210592 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced400f450 con 0x7fcedc107810 2024-05-09T08:59:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcece7fc640 1 -- 172.21.15.98:0/3845210592 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced400fad0 con 0x7fcedc107810 2024-05-09T08:59:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcece7fc640 1 -- 172.21.15.98:0/3845210592 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced4018c10 con 0x7fcedc107810 2024-05-09T08:59:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee33dc640 1 -- 172.21.15.98:0/3845210592 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcedc19d3e0 con 0x7fcedc107810 2024-05-09T08:59:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.369+0000 7fcee33dc640 1 -- 172.21.15.98:0/3845210592 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcedc1a4c70 con 0x7fcedc107810 2024-05-09T08:59:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.371+0000 7fcece7fc640 1 -- 172.21.15.98:0/3845210592 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fced400f5f0 con 0x7fcedc107810 2024-05-09T08:59:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcee61c9740 1 Processor -- start 2024-05-09T08:59:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcee61c9740 1 -- start start 2024-05-09T08:59:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcee61c9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 0x55bb494d8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcece7fc640 1 --2- 172.21.15.98:0/3845210592 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcec80a2d80 0x7fcec80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcee61c9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494d8c50 0x55bb494e2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcee61c9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494e3320 0x55bb494e5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcee61c9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bb4943dc40 con 0x55bb494e3320 2024-05-09T08:59:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcee61c9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bb493e5580 con 0x55bb494d8c50 2024-05-09T08:59:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcee61c9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55bb49439c00 con 0x55bb494d8330 2024-05-09T08:59:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcee1979640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494e3320 0x55bb494e5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcee1178640 1 --2- 172.21.15.98:0/3845210592 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcec80a2d80 0x7fcec80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcee1979640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494e3320 0x55bb494e5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36998/0 (socket says 172.21.15.98:36998) 2024-05-09T08:59:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcee1979640 1 -- 172.21.15.98:0/1603076111 learned_addr learned my addr 172.21.15.98:0/1603076111 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcee0977640 1 --2- 172.21.15.98:0/1603076111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 0x55bb494d8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.372+0000 7fcece7fc640 1 -- 172.21.15.98:0/3845210592 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fced40cff50 con 0x7fcedc107810 2024-05-09T08:59:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.373+0000 7fcee0977640 1 --2- 172.21.15.98:0/1603076111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494d8c50 0x55bb494e2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.373+0000 7fcee1979640 1 -- 172.21.15.98:0/1603076111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 msgr2=0x55bb494d8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.373+0000 7fcee1979640 1 --2- 172.21.15.98:0/1603076111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 0x55bb494d8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.373+0000 7fcee1979640 1 -- 172.21.15.98:0/1603076111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494d8c50 msgr2=0x55bb494e2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.373+0000 7fcee1979640 1 --2- 172.21.15.98:0/1603076111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494d8c50 0x55bb494e2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.373+0000 7fcee1979640 1 -- 172.21.15.98:0/1603076111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb491513f0 con 0x55bb494e3320 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.373+0000 7fcee1178640 1 --2- 172.21.15.98:0/3845210592 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcec80a2d80 0x7fcec80a5240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fced800a860 tx=0x7fced8005b60 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.373+0000 7fcee1979640 1 --2- 172.21.15.98:0/1603076111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494e3320 0x55bb494e5710 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fced4012040 tx=0x7fced40186b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2abf7f225b640b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.373+0000 7fcec77fe640 1 -- 172.21.15.98:0/1603076111 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced409bba0 con 0x55bb494e3320 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.373+0000 7fcec77fe640 1 -- 172.21.15.98:0/1603076111 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced409c760 con 0x55bb494e3320 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.373+0000 7fcec77fe640 1 -- 172.21.15.98:0/1603076111 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced4009290 con 0x55bb494e3320 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.374+0000 7fcee61c9740 1 -- 172.21.15.98:0/1603076111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494e3320 msgr2=0x55bb494e5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.374+0000 7fcee61c9740 1 --2- 172.21.15.98:0/1603076111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494e3320 0x55bb494e5710 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fced4012040 tx=0x7fced40186b0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.374+0000 7fcee61c9740 1 -- 172.21.15.98:0/1603076111 shutdown_connections 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.374+0000 7fcee61c9740 1 --2- 172.21.15.98:0/1603076111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494e3320 0x55bb494e5710 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.374+0000 7fcee61c9740 1 --2- 172.21.15.98:0/1603076111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494d8c50 0x55bb494e2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.374+0000 7fcee61c9740 1 --2- 172.21.15.98:0/1603076111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 0x55bb494d8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.374+0000 7fcee61c9740 1 -- 172.21.15.98:0/1603076111 >> 172.21.15.98:0/1603076111 conn(0x55bb494d4650 msgr2=0x55bb494d4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.374+0000 7fcee61c9740 1 -- 172.21.15.98:0/1603076111 shutdown_connections 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.374+0000 7fcee61c9740 1 -- 172.21.15.98:0/1603076111 wait complete. 2024-05-09T08:59:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.374+0000 7fcee61c9740 1 -- start start 2024-05-09T08:59:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.374+0000 7fcee61c9740 1 Processor -- start 2024-05-09T08:59:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.374+0000 7fcee61c9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8330 0x55bb494e01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee61c9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8c50 0x55bb494e0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee61c9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494e3320 0x55bb494e0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee61c9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bb494e6110 con 0x55bb494d8330 2024-05-09T08:59:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee61c9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bb493f8c40 con 0x55bb494e3320 2024-05-09T08:59:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee61c9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55bb494e6290 con 0x55bb494d8c50 2024-05-09T08:59:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee1178640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8330 0x55bb494e01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee61c9740 10 client.0 _ll_register_callbacks cb 0x55bb49150be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee1178640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8330 0x55bb494e01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37008/0 (socket says 172.21.15.98:37008) 2024-05-09T08:59:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee1178640 1 -- 172.21.15.98:0/190052061 learned_addr learned my addr 172.21.15.98:0/190052061 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee0977640 1 --2- 172.21.15.98:0/190052061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8c50 0x55bb494e0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee1178640 1 -- 172.21.15.98:0/190052061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8c50 msgr2=0x55bb494e0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee1178640 1 --2- 172.21.15.98:0/190052061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8c50 0x55bb494e0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee1178640 1 -- 172.21.15.98:0/190052061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494e3320 msgr2=0x55bb494e0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee1178640 1 --2- 172.21.15.98:0/190052061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494e3320 0x55bb494e0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee1178640 1 -- 172.21.15.98:0/190052061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb4921b7d0 con 0x55bb494d8330 2024-05-09T08:59:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee0977640 1 --2- 172.21.15.98:0/190052061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8c50 0x55bb494e0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee1178640 1 --2- 172.21.15.98:0/190052061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8330 0x55bb494e01f0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fced800f7a0 tx=0x7fced80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4dc3e0c5b6cc9d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcec4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcec4ff9640 1 -- 172.21.15.98:0/190052061 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8020070 con 0x55bb494d8330 2024-05-09T08:59:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcec4ff9640 1 -- 172.21.15.98:0/190052061 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fced800ea30 con 0x55bb494d8330 2024-05-09T08:59:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee61c9740 1 -- 172.21.15.98:0/190052061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fced409c760 con 0x55bb494d8330 2024-05-09T08:59:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcec4ff9640 1 -- 172.21.15.98:0/190052061 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fced802c050 con 0x55bb494d8330 2024-05-09T08:59:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee61c9740 10 client.18246 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.375+0000 7fcee61c9740 1 -- 172.21.15.98:0/190052061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bb494f22a0 con 0x55bb494d8330 2024-05-09T08:59:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.376+0000 7fcec4ff9640 1 -- 172.21.15.98:0/190052061 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced409c760 con 0x55bb494d8330 2024-05-09T08:59:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.376+0000 7fcec4ff9640 1 -- 172.21.15.98:0/190052061 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fced801b070 con 0x55bb494d8330 2024-05-09T08:59:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.376+0000 7fcee61c9740 10 client.18246 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.376+0000 7fcee61c9740 10 client.18246 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.376+0000 7fcee61c9740 20 client.18246 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.376+0000 7fcee61c9740 1 --2- 172.21.15.98:0/190052061 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bb494f3700 0x55bb494f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.376+0000 7fcee61c9740 4 client.18246 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.376+0000 7fcee61c9740 1 -- 172.21.15.98:0/190052061 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb4944b370 con 0x55bb494f3700 2024-05-09T08:59:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.376+0000 7fcee0977640 1 --2- 172.21.15.98:0/190052061 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bb494f3700 0x55bb494f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.376+0000 7fcec4ff9640 10 client.18246 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.376+0000 7fcee0977640 1 --2- 172.21.15.98:0/190052061 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bb494f3700 0x55bb494f5b70 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59c1348b4fbdf57d server_cookie=72fc5304db6d4ca3 in_seq=0 out_seq=0 2024-05-09T08:59:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.377+0000 7fcec4ff9640 1 -- 172.21.15.98:0/190052061 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb4944b370 con 0x55bb494f3700 2024-05-09T08:59:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.377+0000 7fcec4ff9640 10 client.18246 handle_command_reply: tid=0 2024-05-09T08:59:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.377+0000 7fcee61c9740 1 client.18246 shutdown 2024-05-09T08:59:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.377+0000 7fcee61c9740 1 -- 172.21.15.98:0/190052061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8330 msgr2=0x55bb494e01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.377+0000 7fcee61c9740 1 --2- 172.21.15.98:0/190052061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8330 0x55bb494e01f0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fced800f7a0 tx=0x7fced80046c0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.377+0000 7fcee61c9740 1 -- 172.21.15.98:0/190052061 shutdown_connections 2024-05-09T08:59:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.377+0000 7fcee61c9740 1 --2- 172.21.15.98:0/190052061 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bb494f3700 0x55bb494f5b70 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.377+0000 7fcee61c9740 1 --2- 172.21.15.98:0/190052061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494e3320 0x55bb494e0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.377+0000 7fcec4ff9640 0 client.18246 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.377+0000 7fcee61c9740 1 --2- 172.21.15.98:0/190052061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8c50 0x55bb494e0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.377+0000 7fcee61c9740 1 --2- 172.21.15.98:0/190052061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8330 0x55bb494e01f0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.377+0000 7fcee61c9740 1 -- 172.21.15.98:0/190052061 >> 172.21.15.98:0/190052061 conn(0x55bb494d4650 msgr2=0x55bb494da450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.377+0000 7fcee61c9740 1 -- 172.21.15.98:0/190052061 shutdown_connections 2024-05-09T08:59:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.377+0000 7fcee61c9740 1 -- 172.21.15.98:0/190052061 wait complete. 2024-05-09T08:59:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.378+0000 7fcee61c9740 20 client.18246 trim_cache size 0 max 16384 2024-05-09T08:59:50.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee61c9740 1 Processor -- start 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee61c9740 1 -- start start 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee61c9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 0x55bb494eefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee61c9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8c50 0x55bb494daba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee61c9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494e3320 0x55bb494d3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee61c9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc1aee80 con 0x55bb494d8c50 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee61c9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc1b09c0 con 0x55bb494e3320 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee61c9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc1af550 con 0x55bb494d8330 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee0977640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8c50 0x55bb494daba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee0977640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8c50 0x55bb494daba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37020/0 (socket says 172.21.15.98:37020) 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee0977640 1 -- 172.21.15.98:0/493238110 learned_addr learned my addr 172.21.15.98:0/493238110 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee0977640 1 -- 172.21.15.98:0/493238110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 msgr2=0x55bb494eefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee0977640 1 --2- 172.21.15.98:0/493238110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 0x55bb494eefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee0977640 1 -- 172.21.15.98:0/493238110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494e3320 msgr2=0x55bb494d3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee0977640 1 --2- 172.21.15.98:0/493238110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494e3320 0x55bb494d3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee0977640 1 -- 172.21.15.98:0/493238110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb494c5d20 con 0x55bb494d8c50 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcee0977640 1 --2- 172.21.15.98:0/493238110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8c50 0x55bb494daba0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fced0000c00 tx=0x7fced0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ea99f8149a81319 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcec6ffd640 1 -- 172.21.15.98:0/493238110 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced0035040 con 0x55bb494d8c50 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcec6ffd640 1 -- 172.21.15.98:0/493238110 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced0049070 con 0x55bb494d8c50 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.391+0000 7fcec6ffd640 1 -- 172.21.15.98:0/493238110 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced00053c0 con 0x55bb494d8c50 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.392+0000 7fcee61c9740 1 -- 172.21.15.98:0/493238110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8c50 msgr2=0x55bb494daba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.392+0000 7fcee61c9740 1 --2- 172.21.15.98:0/493238110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8c50 0x55bb494daba0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fced0000c00 tx=0x7fced0009260 comp rx=0 tx=0).stop 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.392+0000 7fcee61c9740 1 -- 172.21.15.98:0/493238110 shutdown_connections 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.392+0000 7fcee61c9740 1 --2- 172.21.15.98:0/493238110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494e3320 0x55bb494d3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.392+0000 7fcee61c9740 1 --2- 172.21.15.98:0/493238110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8c50 0x55bb494daba0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.392+0000 7fcee61c9740 1 --2- 172.21.15.98:0/493238110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 0x55bb494eefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.392+0000 7fcee61c9740 1 -- 172.21.15.98:0/493238110 >> 172.21.15.98:0/493238110 conn(0x55bb494d4650 msgr2=0x55bb494d2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.392+0000 7fcee61c9740 1 -- 172.21.15.98:0/493238110 shutdown_connections 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.392+0000 7fcee61c9740 1 -- 172.21.15.98:0/493238110 wait complete. 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.392+0000 7fcee61c9740 1 -- start start 2024-05-09T08:59:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.392+0000 7fcee61c9740 1 Processor -- start 2024-05-09T08:59:50.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee61c9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 0x55bb494dcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee61c9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8c50 0x55bb494dd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee61c9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494e3320 0x55bb494dd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee61c9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc1b02f0 con 0x55bb494d8c50 2024-05-09T08:59:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee61c9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc1afc20 con 0x55bb494e3320 2024-05-09T08:59:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee61c9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc1ace80 con 0x55bb494d8330 2024-05-09T08:59:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee61c9740 10 client.0 _ll_register_callbacks cb 0x7fced800f450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee0977640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8c50 0x55bb494dd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee0977640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8c50 0x55bb494dd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37022/0 (socket says 172.21.15.98:37022) 2024-05-09T08:59:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee0977640 1 -- 172.21.15.98:0/959806507 learned_addr learned my addr 172.21.15.98:0/959806507 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee0977640 1 -- 172.21.15.98:0/959806507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 msgr2=0x55bb494dcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee1178640 1 --2- 172.21.15.98:0/959806507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 0x55bb494dcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee0977640 1 --2- 172.21.15.98:0/959806507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 0x55bb494dcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee0977640 1 -- 172.21.15.98:0/959806507 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494e3320 msgr2=0x55bb494dd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee0977640 1 --2- 172.21.15.98:0/959806507 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494e3320 0x55bb494dd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee0977640 1 -- 172.21.15.98:0/959806507 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb49558cb0 con 0x55bb494d8c50 2024-05-09T08:59:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee1178640 1 --2- 172.21.15.98:0/959806507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 0x55bb494dcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee0977640 1 --2- 172.21.15.98:0/959806507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8c50 0x55bb494dd2f0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fced0009230 tx=0x7fced0005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a951d03dcf386525 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcec4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcec4ff9640 1 -- 172.21.15.98:0/959806507 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced0051020 con 0x55bb494d8c50 2024-05-09T08:59:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcec4ff9640 1 -- 172.21.15.98:0/959806507 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fced0005350 con 0x55bb494d8c50 2024-05-09T08:59:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.393+0000 7fcee61c9740 1 -- 172.21.15.98:0/959806507 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fced0049070 con 0x55bb494d8c50 2024-05-09T08:59:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.394+0000 7fcec4ff9640 1 -- 172.21.15.98:0/959806507 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fced0035040 con 0x55bb494d8c50 2024-05-09T08:59:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.394+0000 7fcee61c9740 10 client.18252 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.394+0000 7fcee61c9740 1 -- 172.21.15.98:0/959806507 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bb49559eb0 con 0x55bb494d8c50 2024-05-09T08:59:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.394+0000 7fcec4ff9640 1 -- 172.21.15.98:0/959806507 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced0049070 con 0x55bb494d8c50 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.394+0000 7fcec4ff9640 1 -- 172.21.15.98:0/959806507 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fced003a040 con 0x55bb494d8c50 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.394+0000 7fcee61c9740 10 client.18252 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.394+0000 7fcee61c9740 10 client.18252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.394+0000 7fcee61c9740 20 client.18252 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.394+0000 7fcee61c9740 1 --2- 172.21.15.98:0/959806507 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bb4955a600 0x55bb4955ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.394+0000 7fcee61c9740 4 client.18252 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.394+0000 7fcee61c9740 1 -- 172.21.15.98:0/959806507 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb494456c0 con 0x55bb4955a600 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.394+0000 7fcee1178640 1 --2- 172.21.15.98:0/959806507 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bb4955a600 0x55bb4955ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcec4ff9640 10 client.18252 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee1178640 1 --2- 172.21.15.98:0/959806507 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bb4955a600 0x55bb4955ca20 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a92651b480e5b65f server_cookie=80ea29a217003e9b in_seq=0 out_seq=0 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcec4ff9640 1 -- 172.21.15.98:0/959806507 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb494456c0 con 0x55bb4955a600 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcec4ff9640 10 client.18252 handle_command_reply: tid=0 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee61c9740 1 client.18252 shutdown 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee61c9740 1 -- 172.21.15.98:0/959806507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8c50 msgr2=0x55bb494dd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee61c9740 1 --2- 172.21.15.98:0/959806507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8c50 0x55bb494dd2f0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fced0009230 tx=0x7fced0005c90 comp rx=0 tx=0).stop 2024-05-09T08:59:50.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee61c9740 1 -- 172.21.15.98:0/959806507 shutdown_connections 2024-05-09T08:59:50.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee61c9740 1 --2- 172.21.15.98:0/959806507 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55bb4955a600 0x55bb4955ca20 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee61c9740 1 --2- 172.21.15.98:0/959806507 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb494e3320 0x55bb494dd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee61c9740 1 --2- 172.21.15.98:0/959806507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bb494d8c50 0x55bb494dd2f0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcec4ff9640 0 client.18252 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:50.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee61c9740 1 --2- 172.21.15.98:0/959806507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bb494d8330 0x55bb494dcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee61c9740 1 -- 172.21.15.98:0/959806507 >> 172.21.15.98:0/959806507 conn(0x55bb494d4650 msgr2=0x55bb494e0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee61c9740 1 -- 172.21.15.98:0/959806507 shutdown_connections 2024-05-09T08:59:50.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee61c9740 1 -- 172.21.15.98:0/959806507 wait complete. 2024-05-09T08:59:50.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee61c9740 20 client.18252 trim_cache size 0 max 16384 2024-05-09T08:59:50.401 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T08:59:50.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee33dc640 1 -- 172.21.15.98:0/3845210592 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcec80a2d80 msgr2=0x7fcec80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee33dc640 1 --2- 172.21.15.98:0/3845210592 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcec80a2d80 0x7fcec80a5240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fced800a860 tx=0x7fced8005b60 comp rx=0 tx=0).stop 2024-05-09T08:59:50.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee33dc640 1 -- 172.21.15.98:0/3845210592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcedc107810 msgr2=0x7fcedc19c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:50.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.395+0000 7fcee33dc640 1 --2- 172.21.15.98:0/3845210592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcedc107810 0x7fcedc19c9c0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fced400cce0 tx=0x7fced400a620 comp rx=0 tx=0).stop 2024-05-09T08:59:50.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.396+0000 7fcee33dc640 1 -- 172.21.15.98:0/3845210592 shutdown_connections 2024-05-09T08:59:50.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.396+0000 7fcee33dc640 1 --2- 172.21.15.98:0/3845210592 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcec80a2d80 0x7fcec80a5240 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.396+0000 7fcee33dc640 1 --2- 172.21.15.98:0/3845210592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcedc107810 0x7fcedc19c9c0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.396+0000 7fcee33dc640 1 --2- 172.21.15.98:0/3845210592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcedc1021c0 0x7fcedc198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.396+0000 7fcee33dc640 1 --2- 172.21.15.98:0/3845210592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcedc101800 0x7fcedc198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:50.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.396+0000 7fcee33dc640 1 -- 172.21.15.98:0/3845210592 >> 172.21.15.98:0/3845210592 conn(0x7fcedc0fd270 msgr2=0x7fcedc0fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:50.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.396+0000 7fcee33dc640 1 -- 172.21.15.98:0/3845210592 shutdown_connections 2024-05-09T08:59:50.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:50.396+0000 7fcee33dc640 1 -- 172.21.15.98:0/3845210592 wait complete. 2024-05-09T08:59:50.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:50 smithi142 ceph-mon[34625]: pgmap v286: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:59:50.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:50 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3188097495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:59:50.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:50 smithi105 ceph-mon[34498]: pgmap v286: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:59:50.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:50 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3188097495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:59:51.407 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:59:51.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:59:51.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:59:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.526+0000 7f6e4cf9d640 1 -- 172.21.15.98:0/412178869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e48106780 msgr2=0x7f6e48106b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.526+0000 7f6e4cf9d640 1 --2- 172.21.15.98:0/412178869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e48106780 0x7f6e48106b60 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f6e38009900 tx=0x7f6e3802a410 comp rx=0 tx=0).stop 2024-05-09T08:59:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.526+0000 7f6e4cf9d640 1 -- 172.21.15.98:0/412178869 shutdown_connections 2024-05-09T08:59:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.526+0000 7f6e4cf9d640 1 --2- 172.21.15.98:0/412178869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e48101420 0x7f6e4810d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.526+0000 7f6e4cf9d640 1 --2- 172.21.15.98:0/412178869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e48100a80 0x7f6e48100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.526+0000 7f6e4cf9d640 1 --2- 172.21.15.98:0/412178869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e48106780 0x7f6e48106b60 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.526+0000 7f6e4cf9d640 1 -- 172.21.15.98:0/412178869 >> 172.21.15.98:0/412178869 conn(0x7f6e48076530 msgr2=0x7f6e48076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.526+0000 7f6e4cf9d640 1 -- 172.21.15.98:0/412178869 shutdown_connections 2024-05-09T08:59:51.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.526+0000 7f6e4cf9d640 1 -- 172.21.15.98:0/412178869 wait complete. 2024-05-09T08:59:51.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.527+0000 7f6e4cf9d640 1 Processor -- start 2024-05-09T08:59:51.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.527+0000 7f6e4cf9d640 1 -- start start 2024-05-09T08:59:51.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.527+0000 7f6e4cf9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e48100a80 0x7f6e48196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.527+0000 7f6e4cf9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e48101420 0x7f6e48197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.527+0000 7f6e4cf9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e48106780 0x7f6e4819b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.527+0000 7f6e4cf9d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6e48110ad0 con 0x7f6e48101420 2024-05-09T08:59:51.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.527+0000 7f6e4cf9d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6e48110950 con 0x7f6e48100a80 2024-05-09T08:59:51.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.527+0000 7f6e4cf9d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6e48110c50 con 0x7f6e48106780 2024-05-09T08:59:51.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.527+0000 7f6e3dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e48101420 0x7f6e48197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.527+0000 7f6e3dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e48101420 0x7f6e48197430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37042/0 (socket says 172.21.15.98:37042) 2024-05-09T08:59:51.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.527+0000 7f6e3dd9b640 1 -- 172.21.15.98:0/3294787272 learned_addr learned my addr 172.21.15.98:0/3294787272 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:51.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.527+0000 7f6e46d9d640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e48106780 0x7f6e4819b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.528+0000 7f6e4659c640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e48100a80 0x7f6e48196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.528+0000 7f6e3dd9b640 1 -- 172.21.15.98:0/3294787272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e48106780 msgr2=0x7f6e4819b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.528+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e48106780 0x7f6e4819b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.528+0000 7f6e3dd9b640 1 -- 172.21.15.98:0/3294787272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e48100a80 msgr2=0x7f6e48196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.528+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e48100a80 0x7f6e48196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.528+0000 7f6e3dd9b640 1 -- 172.21.15.98:0/3294787272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4819bee0 con 0x7f6e48101420 2024-05-09T08:59:51.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.528+0000 7f6e4659c640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e48100a80 0x7f6e48196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:51.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.528+0000 7f6e46d9d640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e48106780 0x7f6e4819b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:51.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.528+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e48101420 0x7f6e48197430 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f6e3000d770 tx=0x7f6e3000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:51.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.528+0000 7f6e3ffff640 1 -- 172.21.15.98:0/3294787272 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e30017070 con 0x7f6e48101420 2024-05-09T08:59:51.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.528+0000 7f6e4cf9d640 1 -- 172.21.15.98:0/3294787272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e4819c1d0 con 0x7f6e48101420 2024-05-09T08:59:51.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.528+0000 7f6e3ffff640 1 -- 172.21.15.98:0/3294787272 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e30005500 con 0x7f6e48101420 2024-05-09T08:59:51.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.528+0000 7f6e3ffff640 1 -- 172.21.15.98:0/3294787272 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e30004e00 con 0x7f6e48101420 2024-05-09T08:59:51.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.528+0000 7f6e4cf9d640 1 -- 172.21.15.98:0/3294787272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e481a3a60 con 0x7f6e48101420 2024-05-09T08:59:51.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.530+0000 7f6e3ffff640 1 -- 172.21.15.98:0/3294787272 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6e30004fa0 con 0x7f6e48101420 2024-05-09T08:59:51.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.531+0000 7f6e4fd81740 1 Processor -- start 2024-05-09T08:59:51.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.531+0000 7f6e4fd81740 1 -- start start 2024-05-09T08:59:51.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.531+0000 7f6e4fd81740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b6590b0 0x557a5b6573f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.531+0000 7f6e4fd81740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6579c0 0x557a5b657e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.531+0000 7f6e4fd81740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b662740 0x557a5b664b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.531+0000 7f6e4fd81740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557a5b5bc990 con 0x557a5b662740 2024-05-09T08:59:51.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.531+0000 7f6e4fd81740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557a5b5645b0 con 0x557a5b6590b0 2024-05-09T08:59:51.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.531+0000 7f6e4fd81740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557a5b5b8950 con 0x557a5b6579c0 2024-05-09T08:59:51.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.531+0000 7f6e3dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b662740 0x557a5b664b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.531+0000 7f6e3dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b662740 0x557a5b664b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37052/0 (socket says 172.21.15.98:37052) 2024-05-09T08:59:51.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.531+0000 7f6e3dd9b640 1 -- 172.21.15.98:0/2266979860 learned_addr learned my addr 172.21.15.98:0/2266979860 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:51.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.531+0000 7f6e4659c640 1 --2- 172.21.15.98:0/2266979860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6579c0 0x557a5b657e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.531+0000 7f6e46d9d640 1 --2- 172.21.15.98:0/2266979860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b6590b0 0x557a5b6573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.531+0000 7f6e3ffff640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6e340a2d00 0x7f6e340a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.532+0000 7f6e3dd9b640 1 -- 172.21.15.98:0/2266979860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6579c0 msgr2=0x557a5b657e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.532+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/2266979860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6579c0 0x557a5b657e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.532+0000 7f6e3dd9b640 1 -- 172.21.15.98:0/2266979860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b6590b0 msgr2=0x557a5b6573f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.532+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/2266979860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b6590b0 0x557a5b6573f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.532+0000 7f6e46d9d640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6e340a2d00 0x7f6e340a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.532+0000 7f6e3dd9b640 1 -- 172.21.15.98:0/2266979860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a5b3987e0 con 0x557a5b662740 2024-05-09T08:59:51.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.532+0000 7f6e4659c640 1 --2- 172.21.15.98:0/2266979860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6579c0 0x557a5b657e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:59:51.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.532+0000 7f6e3ffff640 1 -- 172.21.15.98:0/3294787272 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f6e300cf340 con 0x7f6e48101420 2024-05-09T08:59:51.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.532+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/2266979860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b662740 0x557a5b664b30 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f6e300cc5f0 tx=0x7f6e30022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4aebd1da54b898d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:51.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.532+0000 7f6e46d9d640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6e340a2d00 0x7f6e340a51c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6e40012930 tx=0x7f6e40004ab0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.532+0000 7f6e3cd99640 1 -- 172.21.15.98:0/2266979860 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3009af90 con 0x557a5b662740 2024-05-09T08:59:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.532+0000 7f6e3cd99640 1 -- 172.21.15.98:0/2266979860 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e300046c0 con 0x557a5b662740 2024-05-09T08:59:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.532+0000 7f6e3cd99640 1 -- 172.21.15.98:0/2266979860 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e300056b0 con 0x557a5b662740 2024-05-09T08:59:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.533+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2266979860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b662740 msgr2=0x557a5b664b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.533+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/2266979860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b662740 0x557a5b664b30 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f6e300cc5f0 tx=0x7f6e30022000 comp rx=0 tx=0).stop 2024-05-09T08:59:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.533+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2266979860 shutdown_connections 2024-05-09T08:59:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.533+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/2266979860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b662740 0x557a5b664b30 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.533+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/2266979860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6579c0 0x557a5b657e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.533+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/2266979860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b6590b0 0x557a5b6573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.533+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2266979860 >> 172.21.15.98:0/2266979860 conn(0x557a5b6200a0 msgr2=0x557a5b653f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.533+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2266979860 shutdown_connections 2024-05-09T08:59:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.533+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2266979860 wait complete. 2024-05-09T08:59:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.533+0000 7f6e4fd81740 1 -- start start 2024-05-09T08:59:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.533+0000 7f6e4fd81740 1 Processor -- start 2024-05-09T08:59:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4fd81740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6579c0 0x557a5b665990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4fd81740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6590b0 0x557a5b665ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4fd81740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b662740 0x557a5b666410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4fd81740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557a5b665510 con 0x557a5b6579c0 2024-05-09T08:59:51.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4fd81740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557a5b577b40 con 0x557a5b662740 2024-05-09T08:59:51.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4fd81740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557a5b665690 con 0x557a5b6590b0 2024-05-09T08:59:51.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4fd81740 10 client.0 _ll_register_callbacks cb 0x557a5b56c0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:51.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6579c0 0x557a5b665990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6579c0 0x557a5b665990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37060/0 (socket says 172.21.15.98:37060) 2024-05-09T08:59:51.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4659c640 1 -- 172.21.15.98:0/2653675753 learned_addr learned my addr 172.21.15.98:0/2653675753 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:51.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/2653675753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6590b0 0x557a5b665ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4659c640 1 -- 172.21.15.98:0/2653675753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6590b0 msgr2=0x557a5b665ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4659c640 1 --2- 172.21.15.98:0/2653675753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6590b0 0x557a5b665ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4659c640 1 -- 172.21.15.98:0/2653675753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b662740 msgr2=0x557a5b666410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4659c640 1 --2- 172.21.15.98:0/2653675753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b662740 0x557a5b666410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4659c640 1 -- 172.21.15.98:0/2653675753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a5b3f8f70 con 0x557a5b6579c0 2024-05-09T08:59:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/2653675753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6590b0 0x557a5b665ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:59:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4659c640 1 --2- 172.21.15.98:0/2653675753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6579c0 0x557a5b665990 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f6e38009230 tx=0x7f6e380317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd469e5f99785b37 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e2dffb640 1 -- 172.21.15.98:0/2653675753 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e38040070 con 0x557a5b6579c0 2024-05-09T08:59:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e2dffb640 1 -- 172.21.15.98:0/2653675753 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f6e38004450 con 0x557a5b6579c0 2024-05-09T08:59:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2653675753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557a5b5ca0c0 con 0x557a5b6579c0 2024-05-09T08:59:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e2dffb640 1 -- 172.21.15.98:0/2653675753 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6e3804c050 con 0x557a5b6579c0 2024-05-09T08:59:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4fd81740 10 client.18264 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.534+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2653675753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557a5b290170 con 0x557a5b6579c0 2024-05-09T08:59:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.535+0000 7f6e2dffb640 1 -- 172.21.15.98:0/2653675753 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557a5b5ca0c0 con 0x557a5b6579c0 2024-05-09T08:59:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.535+0000 7f6e2dffb640 1 -- 172.21.15.98:0/2653675753 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f6e38004c90 con 0x557a5b6579c0 2024-05-09T08:59:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.535+0000 7f6e4fd81740 10 client.18264 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.535+0000 7f6e4fd81740 10 client.18264 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.535+0000 7f6e4fd81740 20 client.18264 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.535+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/2653675753 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557a5b674990 0x557a5b676e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.535+0000 7f6e4fd81740 4 client.18264 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.535+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2653675753 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a5b5c4410 con 0x557a5b674990 2024-05-09T08:59:51.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.535+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/2653675753 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557a5b674990 0x557a5b676e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e2dffb640 10 client.18264 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/2653675753 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557a5b674990 0x557a5b676e50 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60fdd3c6ecbf769c server_cookie=912f9b5e5446a549 in_seq=0 out_seq=0 2024-05-09T08:59:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e2dffb640 1 -- 172.21.15.98:0/2653675753 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557a5b5c4410 con 0x557a5b674990 2024-05-09T08:59:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e2dffb640 10 client.18264 handle_command_reply: tid=0 2024-05-09T08:59:51.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e4fd81740 1 client.18264 shutdown 2024-05-09T08:59:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2653675753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6579c0 msgr2=0x557a5b665990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/2653675753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6579c0 0x557a5b665990 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f6e38009230 tx=0x7f6e380317e0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2653675753 shutdown_connections 2024-05-09T08:59:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/2653675753 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557a5b674990 0x557a5b676e50 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/2653675753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b662740 0x557a5b666410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e2dffb640 0 client.18264 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/2653675753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6590b0 0x557a5b665ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/2653675753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6579c0 0x557a5b665990 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2653675753 >> 172.21.15.98:0/2653675753 conn(0x557a5b6200a0 msgr2=0x557a5b659d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.536+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2653675753 shutdown_connections 2024-05-09T08:59:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.537+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2653675753 wait complete. 2024-05-09T08:59:51.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.537+0000 7f6e4fd81740 20 client.18264 trim_cache size 0 max 16384 2024-05-09T08:59:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.554+0000 7f6e4fd81740 1 Processor -- start 2024-05-09T08:59:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.554+0000 7f6e4fd81740 1 -- start start 2024-05-09T08:59:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.554+0000 7f6e4fd81740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b6579c0 0x557a5b67c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.554+0000 7f6e4fd81740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6590b0 0x557a5b67ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.554+0000 7f6e4fd81740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b662740 0x557a5b663b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.554+0000 7f6e4fd81740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6e481acb60 con 0x557a5b6590b0 2024-05-09T08:59:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.554+0000 7f6e4fd81740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6e481ae6a0 con 0x557a5b6579c0 2024-05-09T08:59:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e4fd81740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6e481ad230 con 0x557a5b662740 2024-05-09T08:59:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e3dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6590b0 0x557a5b67ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e3dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6590b0 0x557a5b67ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37070/0 (socket says 172.21.15.98:37070) 2024-05-09T08:59:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e3dd9b640 1 -- 172.21.15.98:0/2222761261 learned_addr learned my addr 172.21.15.98:0/2222761261 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:51.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e4659c640 1 --2- 172.21.15.98:0/2222761261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b6579c0 0x557a5b67c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e3dd9b640 1 -- 172.21.15.98:0/2222761261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b662740 msgr2=0x557a5b663b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/2222761261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b662740 0x557a5b663b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e3dd9b640 1 -- 172.21.15.98:0/2222761261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b6579c0 msgr2=0x557a5b67c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/2222761261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b6579c0 0x557a5b67c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e3dd9b640 1 -- 172.21.15.98:0/2222761261 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a5b661da0 con 0x557a5b6590b0 2024-05-09T08:59:51.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e4659c640 1 --2- 172.21.15.98:0/2222761261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b6579c0 0x557a5b67c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:51.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/2222761261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6590b0 0x557a5b67ccb0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f6e3000dfa0 tx=0x7f6e30097040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a141ad3538a580a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:51.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e2ffff640 1 -- 172.21.15.98:0/2222761261 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e30090950 con 0x557a5b6590b0 2024-05-09T08:59:51.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e2ffff640 1 -- 172.21.15.98:0/2222761261 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e300d2b20 con 0x557a5b6590b0 2024-05-09T08:59:51.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.555+0000 7f6e2ffff640 1 -- 172.21.15.98:0/2222761261 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3009adb0 con 0x557a5b6590b0 2024-05-09T08:59:51.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2222761261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6590b0 msgr2=0x557a5b67ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/2222761261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6590b0 0x557a5b67ccb0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f6e3000dfa0 tx=0x7f6e30097040 comp rx=0 tx=0).stop 2024-05-09T08:59:51.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2222761261 shutdown_connections 2024-05-09T08:59:51.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/2222761261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b662740 0x557a5b663b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/2222761261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6590b0 0x557a5b67ccb0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/2222761261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b6579c0 0x557a5b67c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2222761261 >> 172.21.15.98:0/2222761261 conn(0x557a5b6200a0 msgr2=0x557a5b65dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:51.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2222761261 shutdown_connections 2024-05-09T08:59:51.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 -- 172.21.15.98:0/2222761261 wait complete. 2024-05-09T08:59:51.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 -- start start 2024-05-09T08:59:51.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 Processor -- start 2024-05-09T08:59:51.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6579c0 0x557a5b66b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6590b0 0x557a5b66bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b662740 0x557a5b668980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6e481adfd0 con 0x557a5b6579c0 2024-05-09T08:59:51.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6e481ad900 con 0x557a5b662740 2024-05-09T08:59:51.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6e481aab60 con 0x557a5b6590b0 2024-05-09T08:59:51.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4fd81740 10 client.0 _ll_register_callbacks cb 0x7f6e38009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:51.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6579c0 0x557a5b66b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6579c0 0x557a5b66b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37074/0 (socket says 172.21.15.98:37074) 2024-05-09T08:59:51.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.556+0000 7f6e4659c640 1 -- 172.21.15.98:0/530238223 learned_addr learned my addr 172.21.15.98:0/530238223 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4659c640 1 -- 172.21.15.98:0/530238223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6590b0 msgr2=0x557a5b66bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4659c640 1 --2- 172.21.15.98:0/530238223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6590b0 0x557a5b66bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4659c640 1 -- 172.21.15.98:0/530238223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b662740 msgr2=0x557a5b668980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4659c640 1 --2- 172.21.15.98:0/530238223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b662740 0x557a5b668980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4659c640 1 -- 172.21.15.98:0/530238223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a5b6691b0 con 0x557a5b6579c0 2024-05-09T08:59:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4659c640 1 --2- 172.21.15.98:0/530238223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6579c0 0x557a5b66b9b0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f6e38005c40 tx=0x7f6e38031b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc33193b0cb14a4a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e2dffb640 1 -- 172.21.15.98:0/530238223 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e38040050 con 0x557a5b6579c0 2024-05-09T08:59:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e2dffb640 1 -- 172.21.15.98:0/530238223 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7f6e38003e30 con 0x557a5b6579c0 2024-05-09T08:59:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4fd81740 1 -- 172.21.15.98:0/530238223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557a5b541da0 con 0x557a5b6579c0 2024-05-09T08:59:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e2dffb640 1 -- 172.21.15.98:0/530238223 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6e3805e050 con 0x557a5b6579c0 2024-05-09T08:59:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4fd81740 10 client.18270 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4fd81740 1 -- 172.21.15.98:0/530238223 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557a5b6dba60 con 0x557a5b6579c0 2024-05-09T08:59:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e2dffb640 1 -- 172.21.15.98:0/530238223 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557a5b541da0 con 0x557a5b6579c0 2024-05-09T08:59:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e2dffb640 1 -- 172.21.15.98:0/530238223 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f6e380047e0 con 0x557a5b6579c0 2024-05-09T08:59:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4fd81740 10 client.18270 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4fd81740 10 client.18270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4fd81740 20 client.18270 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/530238223 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557a5b6dc230 0x557a5b6de6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4fd81740 4 client.18270 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:59:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.557+0000 7f6e4fd81740 1 -- 172.21.15.98:0/530238223 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557a5b58dcd0 con 0x557a5b6dc230 2024-05-09T08:59:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.558+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/530238223 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557a5b6dc230 0x557a5b6de6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.558+0000 7f6e2dffb640 10 client.18270 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.558+0000 7f6e3dd9b640 1 --2- 172.21.15.98:0/530238223 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557a5b6dc230 0x557a5b6de6b0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e51adc78f89f491 server_cookie=4e8298b805e7a451 in_seq=0 out_seq=0 2024-05-09T08:59:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e2dffb640 1 -- 172.21.15.98:0/530238223 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557a5b58dcd0 con 0x557a5b6dc230 2024-05-09T08:59:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e2dffb640 10 client.18270 handle_command_reply: tid=0 2024-05-09T08:59:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e4fd81740 1 client.18270 shutdown 2024-05-09T08:59:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e4fd81740 1 -- 172.21.15.98:0/530238223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6579c0 msgr2=0x557a5b66b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/530238223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6579c0 0x557a5b66b9b0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f6e38005c40 tx=0x7f6e38031b00 comp rx=0 tx=0).stop 2024-05-09T08:59:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e4fd81740 1 -- 172.21.15.98:0/530238223 shutdown_connections 2024-05-09T08:59:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/530238223 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x557a5b6dc230 0x557a5b6de6b0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/530238223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a5b662740 0x557a5b668980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/530238223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a5b6590b0 0x557a5b66bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e2dffb640 0 client.18270 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e4fd81740 1 --2- 172.21.15.98:0/530238223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a5b6579c0 0x557a5b66b9b0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e4fd81740 1 -- 172.21.15.98:0/530238223 >> 172.21.15.98:0/530238223 conn(0x557a5b6200a0 msgr2=0x557a5b6594e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e4fd81740 1 -- 172.21.15.98:0/530238223 shutdown_connections 2024-05-09T08:59:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e4fd81740 1 -- 172.21.15.98:0/530238223 wait complete. 2024-05-09T08:59:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.559+0000 7f6e4fd81740 20 client.18270 trim_cache size 0 max 16384 2024-05-09T08:59:51.577 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:59:51.577 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T08:59:51.577 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "adb68c40-d966-40c2-8764-d4663589f841", 2024-05-09T08:59:51.577 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T08:59:51.577 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:59:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.560+0000 7f6e4cf9d640 1 -- 172.21.15.98:0/3294787272 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6e340a2d00 msgr2=0x7f6e340a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.560+0000 7f6e4cf9d640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6e340a2d00 0x7f6e340a51c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6e40012930 tx=0x7f6e40004ab0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.560+0000 7f6e4cf9d640 1 -- 172.21.15.98:0/3294787272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e48101420 msgr2=0x7f6e48197430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.560+0000 7f6e4cf9d640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e48101420 0x7f6e48197430 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f6e3000d770 tx=0x7f6e3000dc30 comp rx=0 tx=0).stop 2024-05-09T08:59:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.560+0000 7f6e4cf9d640 1 -- 172.21.15.98:0/3294787272 shutdown_connections 2024-05-09T08:59:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.560+0000 7f6e4cf9d640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6e340a2d00 0x7f6e340a51c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.560+0000 7f6e4cf9d640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e48106780 0x7f6e4819b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.560+0000 7f6e4cf9d640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e48101420 0x7f6e48197430 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.560+0000 7f6e4cf9d640 1 --2- 172.21.15.98:0/3294787272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e48100a80 0x7f6e48196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.560+0000 7f6e4cf9d640 1 -- 172.21.15.98:0/3294787272 >> 172.21.15.98:0/3294787272 conn(0x7f6e48076530 msgr2=0x7f6e480ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.560+0000 7f6e4cf9d640 1 -- 172.21.15.98:0/3294787272 shutdown_connections 2024-05-09T08:59:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.560+0000 7f6e4cf9d640 1 -- 172.21.15.98:0/3294787272 wait complete. 2024-05-09T08:59:51.581 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:59:51.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:59:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.685+0000 7fa76724d640 1 -- 172.21.15.98:0/880939730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7601010c0 msgr2=0x7fa76010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.685+0000 7fa76724d640 1 --2- 172.21.15.98:0/880939730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7601010c0 0x7fa76010d5d0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fa758009930 tx=0x7fa75802a420 comp rx=0 tx=0).stop 2024-05-09T08:59:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.685+0000 7fa76724d640 1 -- 172.21.15.98:0/880939730 shutdown_connections 2024-05-09T08:59:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.685+0000 7fa76724d640 1 --2- 172.21.15.98:0/880939730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7601010c0 0x7fa76010d5d0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.685+0000 7fa76724d640 1 --2- 172.21.15.98:0/880939730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa760100700 0x7fa760100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.685+0000 7fa76724d640 1 --2- 172.21.15.98:0/880939730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa760106710 0x7fa760106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.685+0000 7fa76724d640 1 -- 172.21.15.98:0/880939730 >> 172.21.15.98:0/880939730 conn(0x7fa760077830 msgr2=0x7fa760077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.685+0000 7fa76724d640 1 -- 172.21.15.98:0/880939730 shutdown_connections 2024-05-09T08:59:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.685+0000 7fa76724d640 1 -- 172.21.15.98:0/880939730 wait complete. 2024-05-09T08:59:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.685+0000 7fa76724d640 1 Processor -- start 2024-05-09T08:59:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.685+0000 7fa76724d640 1 -- start start 2024-05-09T08:59:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa76724d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa760100700 0x7fa760198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa76724d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7601010c0 0x7fa760198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa76724d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa760106710 0x7fa76019c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa76724d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa760110b20 con 0x7fa760106710 2024-05-09T08:59:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa76724d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601109a0 con 0x7fa7601010c0 2024-05-09T08:59:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa76724d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa760110ca0 con 0x7fa760100700 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa7657ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa760106710 0x7fa76019c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa7657ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa760106710 0x7fa76019c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37080/0 (socket says 172.21.15.98:37080) 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa7657ea640 1 -- 172.21.15.98:0/2168625492 learned_addr learned my addr 172.21.15.98:0/2168625492 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa75ffff640 1 --2- 172.21.15.98:0/2168625492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7601010c0 0x7fa760198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa7657ea640 1 -- 172.21.15.98:0/2168625492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa760100700 msgr2=0x7fa760198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa7657ea640 1 --2- 172.21.15.98:0/2168625492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa760100700 0x7fa760198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa7657ea640 1 -- 172.21.15.98:0/2168625492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7601010c0 msgr2=0x7fa760198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa7657ea640 1 --2- 172.21.15.98:0/2168625492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7601010c0 0x7fa760198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa7657ea640 1 -- 172.21.15.98:0/2168625492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76019d0f0 con 0x7fa760106710 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa7657ea640 1 --2- 172.21.15.98:0/2168625492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa760106710 0x7fa76019c9c0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fa758005ab0 tx=0x7fa7580040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa75dffb640 1 -- 172.21.15.98:0/2168625492 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75803f030 con 0x7fa760106710 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa76724d640 1 -- 172.21.15.98:0/2168625492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa76019d380 con 0x7fa760106710 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa75dffb640 1 -- 172.21.15.98:0/2168625492 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7580044c0 con 0x7fa760106710 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.686+0000 7fa75dffb640 1 -- 172.21.15.98:0/2168625492 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758035070 con 0x7fa760106710 2024-05-09T08:59:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.687+0000 7fa76724d640 1 -- 172.21.15.98:0/2168625492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7601a4c70 con 0x7fa760106710 2024-05-09T08:59:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa75dffb640 1 -- 172.21.15.98:0/2168625492 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa758042030 con 0x7fa760106710 2024-05-09T08:59:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa769bfd740 1 Processor -- start 2024-05-09T08:59:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa769bfd740 1 -- start start 2024-05-09T08:59:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa75dffb640 1 --2- 172.21.15.98:0/2168625492 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa74c0a2d00 0x7fa74c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa769bfd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fcfb0 0x562bbf1fd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa769bfd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1fd960 0x562bbf20c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa769bfd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf20c680 0x562bbf20ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa769bfd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562bbf162d00 con 0x562bbf1fcfb0 2024-05-09T08:59:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa769bfd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562bbf10a4c0 con 0x562bbf20c680 2024-05-09T08:59:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa769bfd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562bbf15ecc0 con 0x562bbf1fd960 2024-05-09T08:59:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa764fe9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fcfb0 0x562bbf1fd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa75ffff640 1 --2- 172.21.15.98:0/2168625492 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa74c0a2d00 0x7fa74c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa764fe9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fcfb0 0x562bbf1fd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37096/0 (socket says 172.21.15.98:37096) 2024-05-09T08:59:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa764fe9640 1 -- 172.21.15.98:0/2074523335 learned_addr learned my addr 172.21.15.98:0/2074523335 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.689+0000 7fa75dffb640 1 -- 172.21.15.98:0/2168625492 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fa7580efc80 con 0x7fa760106710 2024-05-09T08:59:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.690+0000 7fa7657ea640 1 --2- 172.21.15.98:0/2074523335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf20c680 0x562bbf20ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.690+0000 7fa764fe9640 1 -- 172.21.15.98:0/2074523335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1fd960 msgr2=0x562bbf20c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.690+0000 7fa764fe9640 1 --2- 172.21.15.98:0/2074523335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1fd960 0x562bbf20c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.690+0000 7fa764fe9640 1 -- 172.21.15.98:0/2074523335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf20c680 msgr2=0x562bbf20ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.690+0000 7fa764fe9640 1 --2- 172.21.15.98:0/2074523335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf20c680 0x562bbf20ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.690+0000 7fa764fe9640 1 -- 172.21.15.98:0/2074523335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bbef407d0 con 0x562bbf1fcfb0 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.690+0000 7fa75ffff640 1 --2- 172.21.15.98:0/2168625492 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa74c0a2d00 0x7fa74c0a51c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa7601002f0 tx=0x7fa7480041e0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.690+0000 7fa764fe9640 1 --2- 172.21.15.98:0/2074523335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fcfb0 0x562bbf1fd390 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fa750009e20 tx=0x7fa75000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9186e46d97e470f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.690+0000 7fa756ffd640 1 -- 172.21.15.98:0/2074523335 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750004b20 con 0x562bbf1fcfb0 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.690+0000 7fa756ffd640 1 -- 172.21.15.98:0/2074523335 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa750004cc0 con 0x562bbf1fcfb0 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.690+0000 7fa756ffd640 1 -- 172.21.15.98:0/2074523335 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750018970 con 0x562bbf1fcfb0 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.691+0000 7fa769bfd740 1 -- 172.21.15.98:0/2074523335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fcfb0 msgr2=0x562bbf1fd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.691+0000 7fa769bfd740 1 --2- 172.21.15.98:0/2074523335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fcfb0 0x562bbf1fd390 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fa750009e20 tx=0x7fa75000e720 comp rx=0 tx=0).stop 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.691+0000 7fa769bfd740 1 -- 172.21.15.98:0/2074523335 shutdown_connections 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.691+0000 7fa769bfd740 1 --2- 172.21.15.98:0/2074523335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf20c680 0x562bbf20ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.691+0000 7fa769bfd740 1 --2- 172.21.15.98:0/2074523335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1fd960 0x562bbf20c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.691+0000 7fa769bfd740 1 --2- 172.21.15.98:0/2074523335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fcfb0 0x562bbf1fd390 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.691+0000 7fa769bfd740 1 -- 172.21.15.98:0/2074523335 >> 172.21.15.98:0/2074523335 conn(0x562bbf1f95a0 msgr2=0x562bbf1f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.691+0000 7fa769bfd740 1 -- 172.21.15.98:0/2074523335 shutdown_connections 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.691+0000 7fa769bfd740 1 -- 172.21.15.98:0/2074523335 wait complete. 2024-05-09T08:59:51.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.691+0000 7fa769bfd740 1 -- start start 2024-05-09T08:59:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.691+0000 7fa769bfd740 1 Processor -- start 2024-05-09T08:59:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa769bfd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf1fcfb0 0x562bbf2095b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa769bfd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fd960 0x562bbf209af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa769bfd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf20c680 0x562bbf20a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa769bfd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562bbf20f440 con 0x562bbf1fd960 2024-05-09T08:59:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa769bfd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562bbf11db90 con 0x562bbf1fcfb0 2024-05-09T08:59:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa769bfd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562bbf20f5c0 con 0x562bbf20c680 2024-05-09T08:59:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa769bfd740 10 client.0 _ll_register_callbacks cb 0x562bbf159da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa75ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fd960 0x562bbf209af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa75ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fd960 0x562bbf209af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37098/0 (socket says 172.21.15.98:37098) 2024-05-09T08:59:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa75ffff640 1 -- 172.21.15.98:0/2015255867 learned_addr learned my addr 172.21.15.98:0/2015255867 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa764fe9640 1 --2- 172.21.15.98:0/2015255867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf1fcfb0 0x562bbf2095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa7657ea640 1 --2- 172.21.15.98:0/2015255867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf20c680 0x562bbf20a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa75ffff640 1 -- 172.21.15.98:0/2015255867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf20c680 msgr2=0x562bbf20a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa75ffff640 1 --2- 172.21.15.98:0/2015255867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf20c680 0x562bbf20a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa75ffff640 1 -- 172.21.15.98:0/2015255867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf1fcfb0 msgr2=0x562bbf2095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa75ffff640 1 --2- 172.21.15.98:0/2015255867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf1fcfb0 0x562bbf2095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa75ffff640 1 -- 172.21.15.98:0/2015255867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bbee763f0 con 0x562bbf1fd960 2024-05-09T08:59:51.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa7657ea640 1 --2- 172.21.15.98:0/2015255867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf20c680 0x562bbf20a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:51.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa75ffff640 1 --2- 172.21.15.98:0/2015255867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fd960 0x562bbf209af0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fa748003ed0 tx=0x7fa748004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2521b41ffaeb1aa3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:51.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa73ffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:51.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa73ffff640 1 -- 172.21.15.98:0/2015255867 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa748023070 con 0x562bbf1fd960 2024-05-09T08:59:51.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa73ffff640 1 -- 172.21.15.98:0/2015255867 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fa74801f830 con 0x562bbf1fd960 2024-05-09T08:59:51.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa769bfd740 1 -- 172.21.15.98:0/2015255867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa750004cc0 con 0x562bbf1fd960 2024-05-09T08:59:51.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa73ffff640 1 -- 172.21.15.98:0/2015255867 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa74802f050 con 0x562bbf1fd960 2024-05-09T08:59:51.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa769bfd740 10 client.18282 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:51.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.692+0000 7fa769bfd740 1 -- 172.21.15.98:0/2015255867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562bbee360f0 con 0x562bbf1fd960 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.693+0000 7fa73ffff640 1 -- 172.21.15.98:0/2015255867 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa750004cc0 con 0x562bbf1fd960 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.693+0000 7fa73ffff640 1 -- 172.21.15.98:0/2015255867 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa74800fab0 con 0x562bbf1fd960 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.693+0000 7fa769bfd740 10 client.18282 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.693+0000 7fa769bfd740 10 client.18282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.693+0000 7fa769bfd740 20 client.18282 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.693+0000 7fa769bfd740 1 --2- 172.21.15.98:0/2015255867 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562bbf21c9b0 0x562bbf21edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.693+0000 7fa769bfd740 4 client.18282 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.693+0000 7fa769bfd740 1 -- 172.21.15.98:0/2015255867 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bbf170430 con 0x562bbf21c9b0 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.693+0000 7fa764fe9640 1 --2- 172.21.15.98:0/2015255867 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562bbf21c9b0 0x562bbf21edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.694+0000 7fa73ffff640 10 client.18282 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.694+0000 7fa764fe9640 1 --2- 172.21.15.98:0/2015255867 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562bbf21c9b0 0x562bbf21edf0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a66dff4be572263a server_cookie=739908ed39cfdbf9 in_seq=0 out_seq=0 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.694+0000 7fa73ffff640 1 -- 172.21.15.98:0/2015255867 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562bbf170430 con 0x562bbf21c9b0 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.694+0000 7fa73ffff640 10 client.18282 handle_command_reply: tid=0 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.694+0000 7fa769bfd740 1 client.18282 shutdown 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.694+0000 7fa769bfd740 1 -- 172.21.15.98:0/2015255867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fd960 msgr2=0x562bbf209af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.694+0000 7fa769bfd740 1 --2- 172.21.15.98:0/2015255867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fd960 0x562bbf209af0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fa748003ed0 tx=0x7fa748004730 comp rx=0 tx=0).stop 2024-05-09T08:59:51.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.694+0000 7fa769bfd740 1 -- 172.21.15.98:0/2015255867 shutdown_connections 2024-05-09T08:59:51.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.694+0000 7fa769bfd740 1 --2- 172.21.15.98:0/2015255867 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562bbf21c9b0 0x562bbf21edf0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.694+0000 7fa769bfd740 1 --2- 172.21.15.98:0/2015255867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf20c680 0x562bbf20a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.694+0000 7fa769bfd740 1 --2- 172.21.15.98:0/2015255867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fd960 0x562bbf209af0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.694+0000 7fa769bfd740 1 --2- 172.21.15.98:0/2015255867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf1fcfb0 0x562bbf2095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.694+0000 7fa73ffff640 0 client.18282 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:51.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.694+0000 7fa769bfd740 1 -- 172.21.15.98:0/2015255867 >> 172.21.15.98:0/2015255867 conn(0x562bbf1f95a0 msgr2=0x562bbf1ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:51.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.695+0000 7fa769bfd740 1 -- 172.21.15.98:0/2015255867 shutdown_connections 2024-05-09T08:59:51.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.695+0000 7fa769bfd740 1 -- 172.21.15.98:0/2015255867 wait complete. 2024-05-09T08:59:51.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.695+0000 7fa769bfd740 20 client.18282 trim_cache size 0 max 16384 2024-05-09T08:59:51.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa769bfd740 1 Processor -- start 2024-05-09T08:59:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa769bfd740 1 -- start start 2024-05-09T08:59:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa769bfd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1f95a0 0x562bbf218340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa769bfd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf1fcfb0 0x562bbf207e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa769bfd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1fd960 0x562bbf2246c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa769bfd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601acd00 con 0x562bbf1f95a0 2024-05-09T08:59:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa769bfd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601ae840 con 0x562bbf1fcfb0 2024-05-09T08:59:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa769bfd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601ad3d0 con 0x562bbf1fd960 2024-05-09T08:59:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa764fe9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1f95a0 0x562bbf218340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa764fe9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1f95a0 0x562bbf218340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37110/0 (socket says 172.21.15.98:37110) 2024-05-09T08:59:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa764fe9640 1 -- 172.21.15.98:0/812409148 learned_addr learned my addr 172.21.15.98:0/812409148 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa7657ea640 1 --2- 172.21.15.98:0/812409148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1fd960 0x562bbf2246c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa764fe9640 1 -- 172.21.15.98:0/812409148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1fd960 msgr2=0x562bbf2246c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa764fe9640 1 --2- 172.21.15.98:0/812409148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1fd960 0x562bbf2246c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa764fe9640 1 -- 172.21.15.98:0/812409148 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf1fcfb0 msgr2=0x562bbf207e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa764fe9640 1 --2- 172.21.15.98:0/812409148 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf1fcfb0 0x562bbf207e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa764fe9640 1 -- 172.21.15.98:0/812409148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bbf1fd700 con 0x562bbf1f95a0 2024-05-09T08:59:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa7657ea640 1 --2- 172.21.15.98:0/812409148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1fd960 0x562bbf2246c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa764fe9640 1 --2- 172.21.15.98:0/812409148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1f95a0 0x562bbf218340 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fa750005b40 tx=0x7fa75001c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473651b1e182c173 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.712+0000 7fa7567fc640 1 -- 172.21.15.98:0/812409148 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75002d070 con 0x562bbf1f95a0 2024-05-09T08:59:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.713+0000 7fa7567fc640 1 -- 172.21.15.98:0/812409148 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa750028650 con 0x562bbf1f95a0 2024-05-09T08:59:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.713+0000 7fa7567fc640 1 -- 172.21.15.98:0/812409148 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750018c40 con 0x562bbf1f95a0 2024-05-09T08:59:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.713+0000 7fa769bfd740 1 -- 172.21.15.98:0/812409148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1f95a0 msgr2=0x562bbf218340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.713+0000 7fa769bfd740 1 --2- 172.21.15.98:0/812409148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1f95a0 0x562bbf218340 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fa750005b40 tx=0x7fa75001c040 comp rx=0 tx=0).stop 2024-05-09T08:59:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.713+0000 7fa769bfd740 1 -- 172.21.15.98:0/812409148 shutdown_connections 2024-05-09T08:59:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.713+0000 7fa769bfd740 1 --2- 172.21.15.98:0/812409148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1fd960 0x562bbf2246c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.713+0000 7fa769bfd740 1 --2- 172.21.15.98:0/812409148 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf1fcfb0 0x562bbf207e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.713+0000 7fa769bfd740 1 --2- 172.21.15.98:0/812409148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1f95a0 0x562bbf218340 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.713+0000 7fa769bfd740 1 -- 172.21.15.98:0/812409148 >> 172.21.15.98:0/812409148 conn(0x562bbf1f49e0 msgr2=0x562bbf20b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:51.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.713+0000 7fa769bfd740 1 -- 172.21.15.98:0/812409148 shutdown_connections 2024-05-09T08:59:51.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.713+0000 7fa769bfd740 1 -- 172.21.15.98:0/812409148 wait complete. 2024-05-09T08:59:51.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.713+0000 7fa769bfd740 1 -- start start 2024-05-09T08:59:51.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa769bfd740 1 Processor -- start 2024-05-09T08:59:51.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa769bfd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1f95a0 0x562bbf210a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa769bfd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf1fcfb0 0x562bbf210fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa769bfd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fd960 0x562bbf27bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa769bfd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601ae170 con 0x562bbf1fd960 2024-05-09T08:59:51.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa769bfd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601adaa0 con 0x562bbf1fcfb0 2024-05-09T08:59:51.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa769bfd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601aad00 con 0x562bbf1f95a0 2024-05-09T08:59:51.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa769bfd740 10 client.0 _ll_register_callbacks cb 0x7fa748004430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:51.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa7657ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fd960 0x562bbf27bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa7657ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fd960 0x562bbf27bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37112/0 (socket says 172.21.15.98:37112) 2024-05-09T08:59:51.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa7657ea640 1 -- 172.21.15.98:0/216169458 learned_addr learned my addr 172.21.15.98:0/216169458 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:51.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa764fe9640 1 --2- 172.21.15.98:0/216169458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1f95a0 0x562bbf210a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa75ffff640 1 --2- 172.21.15.98:0/216169458 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf1fcfb0 0x562bbf210fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa7657ea640 1 -- 172.21.15.98:0/216169458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1f95a0 msgr2=0x562bbf210a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa7657ea640 1 --2- 172.21.15.98:0/216169458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1f95a0 0x562bbf210a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa7657ea640 1 -- 172.21.15.98:0/216169458 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf1fcfb0 msgr2=0x562bbf210fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:51.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa7657ea640 1 --2- 172.21.15.98:0/216169458 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf1fcfb0 0x562bbf210fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa7657ea640 1 -- 172.21.15.98:0/216169458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bbf27c4c0 con 0x562bbf1fd960 2024-05-09T08:59:51.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa7657ea640 1 --2- 172.21.15.98:0/216169458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fd960 0x562bbf27bf00 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fa7580ed070 tx=0x7fa7580bc490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26884f5fe4057aed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:51.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.714+0000 7fa73ffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:51.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa73ffff640 1 -- 172.21.15.98:0/216169458 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758049070 con 0x562bbf1fd960 2024-05-09T08:59:51.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa73ffff640 1 -- 172.21.15.98:0/216169458 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fa758003cb0 con 0x562bbf1fd960 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa769bfd740 1 -- 172.21.15.98:0/216169458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa750028650 con 0x562bbf1fd960 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa73ffff640 1 -- 172.21.15.98:0/216169458 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa7580c4a00 con 0x562bbf1fd960 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa769bfd740 10 client.18288 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa769bfd740 1 -- 172.21.15.98:0/216169458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562bbf283830 con 0x562bbf1fd960 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa73ffff640 1 -- 172.21.15.98:0/216169458 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa750028650 con 0x562bbf1fd960 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa73ffff640 1 -- 172.21.15.98:0/216169458 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa7580ab410 con 0x562bbf1fd960 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa769bfd740 10 client.18288 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa769bfd740 10 client.18288 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa769bfd740 20 client.18288 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa769bfd740 1 --2- 172.21.15.98:0/216169458 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562bbf20c680 0x562bbf286010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa769bfd740 4 client.18288 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa769bfd740 1 -- 172.21.15.98:0/216169458 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562bbf16a780 con 0x562bbf20c680 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.715+0000 7fa764fe9640 1 --2- 172.21.15.98:0/216169458 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562bbf20c680 0x562bbf286010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa73ffff640 10 client.18288 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa764fe9640 1 --2- 172.21.15.98:0/216169458 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562bbf20c680 0x562bbf286010 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23adcb3a3f3c14a9 server_cookie=9a27b086ded357e in_seq=0 out_seq=0 2024-05-09T08:59:51.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa73ffff640 1 -- 172.21.15.98:0/216169458 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562bbf16a780 con 0x562bbf20c680 2024-05-09T08:59:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa73ffff640 10 client.18288 handle_command_reply: tid=0 2024-05-09T08:59:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa769bfd740 1 client.18288 shutdown 2024-05-09T08:59:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa769bfd740 1 -- 172.21.15.98:0/216169458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fd960 msgr2=0x562bbf27bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa769bfd740 1 --2- 172.21.15.98:0/216169458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fd960 0x562bbf27bf00 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fa7580ed070 tx=0x7fa7580bc490 comp rx=0 tx=0).stop 2024-05-09T08:59:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa769bfd740 1 -- 172.21.15.98:0/216169458 shutdown_connections 2024-05-09T08:59:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa769bfd740 1 --2- 172.21.15.98:0/216169458 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562bbf20c680 0x562bbf286010 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa769bfd740 1 --2- 172.21.15.98:0/216169458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562bbf1fd960 0x562bbf27bf00 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa769bfd740 1 --2- 172.21.15.98:0/216169458 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562bbf1fcfb0 0x562bbf210fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa73ffff640 0 client.18288 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa769bfd740 1 --2- 172.21.15.98:0/216169458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562bbf1f95a0 0x562bbf210a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa769bfd740 1 -- 172.21.15.98:0/216169458 >> 172.21.15.98:0/216169458 conn(0x562bbf1f49e0 msgr2=0x562bbf1ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa769bfd740 1 -- 172.21.15.98:0/216169458 shutdown_connections 2024-05-09T08:59:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa769bfd740 1 -- 172.21.15.98:0/216169458 wait complete. 2024-05-09T08:59:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.716+0000 7fa769bfd740 20 client.18288 trim_cache size 0 max 16384 2024-05-09T08:59:51.727 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T08:59:51.727 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:59:51.727 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T08:59:51.727 INFO:teuthology.orchestra.run.smithi098.stdout: "adb68c40-d966-40c2-8764-d4663589f841": { 2024-05-09T08:59:51.727 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T08:59:51.727 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "adb68c40-d966-40c2-8764-d4663589f841", 2024-05-09T08:59:51.727 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T08:59:51.727 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:59:51.727 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T08:59:51.727 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T08:59:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.717+0000 7fa76724d640 1 -- 172.21.15.98:0/2168625492 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa74c0a2d00 msgr2=0x7fa74c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.717+0000 7fa76724d640 1 --2- 172.21.15.98:0/2168625492 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa74c0a2d00 0x7fa74c0a51c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa7601002f0 tx=0x7fa7480041e0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.717+0000 7fa76724d640 1 -- 172.21.15.98:0/2168625492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa760106710 msgr2=0x7fa76019c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.717+0000 7fa76724d640 1 --2- 172.21.15.98:0/2168625492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa760106710 0x7fa76019c9c0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fa758005ab0 tx=0x7fa7580040c0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.717+0000 7fa76724d640 1 -- 172.21.15.98:0/2168625492 shutdown_connections 2024-05-09T08:59:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.717+0000 7fa76724d640 1 --2- 172.21.15.98:0/2168625492 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa74c0a2d00 0x7fa74c0a51c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.717+0000 7fa76724d640 1 --2- 172.21.15.98:0/2168625492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa760106710 0x7fa76019c9c0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.717+0000 7fa76724d640 1 --2- 172.21.15.98:0/2168625492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7601010c0 0x7fa760198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.717+0000 7fa76724d640 1 --2- 172.21.15.98:0/2168625492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa760100700 0x7fa760198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.717+0000 7fa76724d640 1 -- 172.21.15.98:0/2168625492 >> 172.21.15.98:0/2168625492 conn(0x7fa760077830 msgr2=0x7fa760068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:51.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.717+0000 7fa76724d640 1 -- 172.21.15.98:0/2168625492 shutdown_connections 2024-05-09T08:59:51.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T08:59:51.717+0000 7fa76724d640 1 -- 172.21.15.98:0/2168625492 wait complete. 2024-05-09T08:59:51.730 INFO:tasks.cephfs.filesystem:scrub status for tag:adb68c40-d966-40c2-8764-d4663589f841 - {'path': '/', 'tag': 'adb68c40-d966-40c2-8764-d4663589f841', 'options': 'recursive,force'} 2024-05-09T08:59:52.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:52 smithi142 ceph-mon[34625]: pgmap v287: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:59:52.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:52 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T08:59:52.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:52 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:59:52.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:52 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T08:59:52.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:52 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T08:59:52.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:52 smithi105 ceph-mon[34498]: pgmap v287: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:59:52.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:52 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T08:59:52.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:52 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:59:52.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:52 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T08:59:52.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:52 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T08:59:52.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:52 smithi098 ceph-mon[30092]: pgmap v287: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:59:52.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:52 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T08:59:52.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:52 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:59:52.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:52 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T08:59:52.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:52 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T08:59:54.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:54 smithi142 ceph-mon[34625]: pgmap v288: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:59:54.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:54 smithi105 ceph-mon[34498]: pgmap v288: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:59:54.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.506+0000 7fcf36864640 1 -- 172.21.15.98:0/1432655497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf301033e0 msgr2=0x7fcf3010b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.506+0000 7fcf36864640 1 --2- 172.21.15.98:0/1432655497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf301033e0 0x7fcf3010b8c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009900 tx=0x7fcf2402a510 comp rx=0 tx=0).stop 2024-05-09T08:59:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.506+0000 7fcf36864640 1 -- 172.21.15.98:0/1432655497 shutdown_connections 2024-05-09T08:59:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.506+0000 7fcf36864640 1 --2- 172.21.15.98:0/1432655497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf3010be00 0x7fcf3010e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.506+0000 7fcf36864640 1 --2- 172.21.15.98:0/1432655497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf301033e0 0x7fcf3010b8c0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.506+0000 7fcf36864640 1 --2- 172.21.15.98:0/1432655497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf30102ac0 0x7fcf30102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.506+0000 7fcf36864640 1 -- 172.21.15.98:0/1432655497 >> 172.21.15.98:0/1432655497 conn(0x7fcf300778b0 msgr2=0x7fcf30077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.506+0000 7fcf36864640 1 -- 172.21.15.98:0/1432655497 shutdown_connections 2024-05-09T08:59:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.506+0000 7fcf36864640 1 -- 172.21.15.98:0/1432655497 wait complete. 2024-05-09T08:59:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.506+0000 7fcf36864640 1 Processor -- start 2024-05-09T08:59:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.506+0000 7fcf36864640 1 -- start start 2024-05-09T08:59:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf36864640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf30102ac0 0x7fcf301980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf36864640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf301033e0 0x7fcf30198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf36864640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3010be00 0x7fcf3019c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf36864640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf30111c00 con 0x7fcf3010be00 2024-05-09T08:59:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf36864640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcf30111a80 con 0x7fcf301033e0 2024-05-09T08:59:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf36864640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcf30111d80 con 0x7fcf30102ac0 2024-05-09T08:59:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf34e01640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3010be00 0x7fcf3019c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf34e01640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3010be00 0x7fcf3019c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37128/0 (socket says 172.21.15.98:37128) 2024-05-09T08:59:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf34e01640 1 -- 172.21.15.98:0/3922799688 learned_addr learned my addr 172.21.15.98:0/3922799688 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf301033e0 0x7fcf30198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf2ffff640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf30102ac0 0x7fcf301980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf34e01640 1 -- 172.21.15.98:0/3922799688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf30102ac0 msgr2=0x7fcf301980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf34e01640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf30102ac0 0x7fcf301980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf34e01640 1 -- 172.21.15.98:0/3922799688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf301033e0 msgr2=0x7fcf30198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf34e01640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf301033e0 0x7fcf30198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf34e01640 1 -- 172.21.15.98:0/3922799688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf3019d060 con 0x7fcf3010be00 2024-05-09T08:59:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf301033e0 0x7fcf30198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:59:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.507+0000 7fcf34e01640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3010be00 0x7fcf3019c930 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800eaa0 tx=0x7fcf2800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.508+0000 7fcf2d7fa640 1 -- 172.21.15.98:0/3922799688 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2800cdb0 con 0x7fcf3010be00 2024-05-09T08:59:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.508+0000 7fcf2d7fa640 1 -- 172.21.15.98:0/3922799688 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf28005020 con 0x7fcf3010be00 2024-05-09T08:59:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.508+0000 7fcf36864640 1 -- 172.21.15.98:0/3922799688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf3019d350 con 0x7fcf3010be00 2024-05-09T08:59:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.508+0000 7fcf2d7fa640 1 -- 172.21.15.98:0/3922799688 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf280116d0 con 0x7fcf3010be00 2024-05-09T08:59:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.508+0000 7fcf36864640 1 -- 172.21.15.98:0/3922799688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf301a4be0 con 0x7fcf3010be00 2024-05-09T08:59:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.510+0000 7fcf2d7fa640 1 -- 172.21.15.98:0/3922799688 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcf280054d0 con 0x7fcf3010be00 2024-05-09T08:59:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.510+0000 7fcf39573740 1 Processor -- start 2024-05-09T08:59:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.510+0000 7fcf39573740 1 -- start start 2024-05-09T08:59:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf39573740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56a450 0x559c8e56a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf39573740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e56ad70 0x559c8e5750f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf39573740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e575630 0x559c8e577a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf39573740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559c8e4cf340 con 0x559c8e56ad70 2024-05-09T08:59:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf39573740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559c8e477290 con 0x559c8e56a450 2024-05-09T08:59:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf39573740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559c8e4cb300 con 0x559c8e575630 2024-05-09T08:59:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf2ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e56ad70 0x559c8e5750f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf2ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e56ad70 0x559c8e5750f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37140/0 (socket says 172.21.15.98:37140) 2024-05-09T08:59:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf2ffff640 1 -- 172.21.15.98:0/4161713947 learned_addr learned my addr 172.21.15.98:0/4161713947 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf2d7fa640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcf180a2da0 0x7fcf180a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/4161713947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56a450 0x559c8e56a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/4161713947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e575630 0x559c8e577a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf34e01640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcf180a2da0 0x7fcf180a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf2ffff640 1 -- 172.21.15.98:0/4161713947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e575630 msgr2=0x559c8e577a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf2ffff640 1 --2- 172.21.15.98:0/4161713947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e575630 0x559c8e577a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf2ffff640 1 -- 172.21.15.98:0/4161713947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56a450 msgr2=0x559c8e56a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf2ffff640 1 --2- 172.21.15.98:0/4161713947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56a450 0x559c8e56a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf2ffff640 1 -- 172.21.15.98:0/4161713947 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c8e49b8e0 con 0x559c8e56ad70 2024-05-09T08:59:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/4161713947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e575630 0x559c8e577a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:59:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf2d7fa640 1 -- 172.21.15.98:0/3922799688 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fcf280cf990 con 0x7fcf3010be00 2024-05-09T08:59:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf2ffff640 1 --2- 172.21.15.98:0/4161713947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e56ad70 0x559c8e5750f0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fcf20009cf0 tx=0x7fcf20011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa68d64767f74328 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.511+0000 7fcf167fc640 1 -- 172.21.15.98:0/4161713947 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf20004230 con 0x559c8e56ad70 2024-05-09T08:59:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.512+0000 7fcf34e01640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcf180a2da0 0x7fcf180a5260 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcf280ccd80 tx=0x7fcf2806db10 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.512+0000 7fcf167fc640 1 -- 172.21.15.98:0/4161713947 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf200043d0 con 0x559c8e56ad70 2024-05-09T08:59:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.512+0000 7fcf167fc640 1 -- 172.21.15.98:0/4161713947 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf200198c0 con 0x559c8e56ad70 2024-05-09T08:59:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.512+0000 7fcf39573740 1 -- 172.21.15.98:0/4161713947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e56ad70 msgr2=0x559c8e5750f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.512+0000 7fcf39573740 1 --2- 172.21.15.98:0/4161713947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e56ad70 0x559c8e5750f0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fcf20009cf0 tx=0x7fcf20011810 comp rx=0 tx=0).stop 2024-05-09T08:59:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.512+0000 7fcf39573740 1 -- 172.21.15.98:0/4161713947 shutdown_connections 2024-05-09T08:59:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.512+0000 7fcf39573740 1 --2- 172.21.15.98:0/4161713947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e575630 0x559c8e577a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.512+0000 7fcf39573740 1 --2- 172.21.15.98:0/4161713947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e56ad70 0x559c8e5750f0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.512+0000 7fcf39573740 1 --2- 172.21.15.98:0/4161713947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56a450 0x559c8e56a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.512+0000 7fcf39573740 1 -- 172.21.15.98:0/4161713947 >> 172.21.15.98:0/4161713947 conn(0x559c8e566960 msgr2=0x559c8e566d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.512+0000 7fcf39573740 1 -- 172.21.15.98:0/4161713947 shutdown_connections 2024-05-09T08:59:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.512+0000 7fcf39573740 1 -- 172.21.15.98:0/4161713947 wait complete. 2024-05-09T08:59:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.513+0000 7fcf39573740 1 -- start start 2024-05-09T08:59:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.513+0000 7fcf39573740 1 Processor -- start 2024-05-09T08:59:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.513+0000 7fcf39573740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56a450 0x559c8e57f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.513+0000 7fcf39573740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56ad70 0x559c8e57fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.513+0000 7fcf39573740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e575630 0x559c8e5798e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.513+0000 7fcf39573740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559c8e578260 con 0x559c8e575630 2024-05-09T08:59:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.513+0000 7fcf39573740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559c8e48a790 con 0x559c8e56a450 2024-05-09T08:59:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.513+0000 7fcf39573740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559c8e5783e0 con 0x559c8e56ad70 2024-05-09T08:59:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.513+0000 7fcf39573740 10 client.0 _ll_register_callbacks cb 0x559c8e4c5a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.513+0000 7fcf2f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e575630 0x559c8e5798e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.513+0000 7fcf2f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e575630 0x559c8e5798e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37156/0 (socket says 172.21.15.98:37156) 2024-05-09T08:59:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.513+0000 7fcf2f7fe640 1 -- 172.21.15.98:0/3686828456 learned_addr learned my addr 172.21.15.98:0/3686828456 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.513+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/3686828456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56a450 0x559c8e57f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.513+0000 7fcf2ffff640 1 --2- 172.21.15.98:0/3686828456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56ad70 0x559c8e57fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.514+0000 7fcf2f7fe640 1 -- 172.21.15.98:0/3686828456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56ad70 msgr2=0x559c8e57fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.514+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/3686828456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56ad70 0x559c8e57fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.514+0000 7fcf2f7fe640 1 -- 172.21.15.98:0/3686828456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56a450 msgr2=0x559c8e57f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.514+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/3686828456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56a450 0x559c8e57f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.514+0000 7fcf2f7fe640 1 -- 172.21.15.98:0/3686828456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c8e1e3210 con 0x559c8e575630 2024-05-09T08:59:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.514+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/3686828456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e575630 0x559c8e5798e0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009900 tx=0x7fcf2402a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b143f3af05f816 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.514+0000 7fcf0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.514+0000 7fcf0f7fe640 1 -- 172.21.15.98:0/3686828456 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf24035070 con 0x559c8e575630 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.514+0000 7fcf39573740 1 -- 172.21.15.98:0/3686828456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf200043d0 con 0x559c8e575630 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.514+0000 7fcf0f7fe640 1 -- 172.21.15.98:0/3686828456 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fcf24004480 con 0x559c8e575630 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.514+0000 7fcf0f7fe640 1 -- 172.21.15.98:0/3686828456 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fcf24053050 con 0x559c8e575630 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.514+0000 7fcf39573740 10 client.18300 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.514+0000 7fcf39573740 1 -- 172.21.15.98:0/3686828456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559c8e2af620 con 0x559c8e575630 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.514+0000 7fcf0f7fe640 1 -- 172.21.15.98:0/3686828456 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf200043d0 con 0x559c8e575630 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.515+0000 7fcf0f7fe640 1 -- 172.21.15.98:0/3686828456 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fcf2403c600 con 0x559c8e575630 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.515+0000 7fcf39573740 10 client.18300 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.515+0000 7fcf39573740 10 client.18300 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.515+0000 7fcf39573740 20 client.18300 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.515+0000 7fcf39573740 1 --2- 172.21.15.98:0/3686828456 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c8e589630 0x559c8e58ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.515+0000 7fcf39573740 4 client.18300 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:59:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.515+0000 7fcf39573740 1 -- 172.21.15.98:0/3686828456 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c8e4dca70 con 0x559c8e589630 2024-05-09T08:59:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.515+0000 7fcf2ffff640 1 --2- 172.21.15.98:0/3686828456 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c8e589630 0x559c8e58ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.515+0000 7fcf2ffff640 1 --2- 172.21.15.98:0/3686828456 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c8e589630 0x559c8e58ba20 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d578c50501bdede7 server_cookie=301b06353bfd9ef6 in_seq=0 out_seq=0 2024-05-09T08:59:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.515+0000 7fcf0f7fe640 10 client.18300 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.516+0000 7fcf0f7fe640 1 -- 172.21.15.98:0/3686828456 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559c8e4dca70 con 0x559c8e589630 2024-05-09T08:59:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.516+0000 7fcf0f7fe640 10 client.18300 handle_command_reply: tid=0 2024-05-09T08:59:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.516+0000 7fcf39573740 1 client.18300 shutdown 2024-05-09T08:59:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.517+0000 7fcf39573740 1 -- 172.21.15.98:0/3686828456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e575630 msgr2=0x559c8e5798e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.517+0000 7fcf39573740 1 --2- 172.21.15.98:0/3686828456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e575630 0x559c8e5798e0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009900 tx=0x7fcf2402a510 comp rx=0 tx=0).stop 2024-05-09T08:59:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.517+0000 7fcf39573740 1 -- 172.21.15.98:0/3686828456 shutdown_connections 2024-05-09T08:59:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.517+0000 7fcf39573740 1 --2- 172.21.15.98:0/3686828456 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c8e589630 0x559c8e58ba20 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.517+0000 7fcf39573740 1 --2- 172.21.15.98:0/3686828456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e575630 0x559c8e5798e0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.517+0000 7fcf39573740 1 --2- 172.21.15.98:0/3686828456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56ad70 0x559c8e57fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.517+0000 7fcf39573740 1 --2- 172.21.15.98:0/3686828456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56a450 0x559c8e57f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.517+0000 7fcf0f7fe640 0 client.18300 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.517+0000 7fcf39573740 1 -- 172.21.15.98:0/3686828456 >> 172.21.15.98:0/3686828456 conn(0x559c8e566960 msgr2=0x559c8e56c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.517+0000 7fcf39573740 1 -- 172.21.15.98:0/3686828456 shutdown_connections 2024-05-09T08:59:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.517+0000 7fcf39573740 1 -- 172.21.15.98:0/3686828456 wait complete. 2024-05-09T08:59:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.517+0000 7fcf39573740 20 client.18300 trim_cache size 0 max 16384 2024-05-09T08:59:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.531+0000 7fcf39573740 1 Processor -- start 2024-05-09T08:59:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf39573740 1 -- start start 2024-05-09T08:59:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf39573740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e56a450 0x559c8e5912e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf39573740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56ad70 0x559c8e591820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf39573740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e575630 0x559c8e57f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf39573740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf301acce0 con 0x559c8e56a450 2024-05-09T08:59:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf39573740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcf301ae820 con 0x559c8e575630 2024-05-09T08:59:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf39573740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcf301ad3b0 con 0x559c8e56ad70 2024-05-09T08:59:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf2f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e56a450 0x559c8e5912e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf2f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e56a450 0x559c8e5912e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37172/0 (socket says 172.21.15.98:37172) 2024-05-09T08:59:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf2f7fe640 1 -- 172.21.15.98:0/133226333 learned_addr learned my addr 172.21.15.98:0/133226333 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf2ffff640 1 --2- 172.21.15.98:0/133226333 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56ad70 0x559c8e591820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf2f7fe640 1 -- 172.21.15.98:0/133226333 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56ad70 msgr2=0x559c8e591820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/133226333 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56ad70 0x559c8e591820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf2f7fe640 1 -- 172.21.15.98:0/133226333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e575630 msgr2=0x559c8e57f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/133226333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e575630 0x559c8e57f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf2f7fe640 1 -- 172.21.15.98:0/133226333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c8e564f00 con 0x559c8e56a450 2024-05-09T08:59:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/133226333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e56a450 0x559c8e5912e0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fcf2402acd0 tx=0x7fcf24009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292cfd62ecab9527 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf15ffb640 1 -- 172.21.15.98:0/133226333 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf24035040 con 0x559c8e56a450 2024-05-09T08:59:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf15ffb640 1 -- 172.21.15.98:0/133226333 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf24041070 con 0x559c8e56a450 2024-05-09T08:59:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.532+0000 7fcf15ffb640 1 -- 172.21.15.98:0/133226333 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf24003870 con 0x559c8e56a450 2024-05-09T08:59:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.533+0000 7fcf39573740 1 -- 172.21.15.98:0/133226333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e56a450 msgr2=0x559c8e5912e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.533+0000 7fcf39573740 1 --2- 172.21.15.98:0/133226333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e56a450 0x559c8e5912e0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fcf2402acd0 tx=0x7fcf24009830 comp rx=0 tx=0).stop 2024-05-09T08:59:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.533+0000 7fcf39573740 1 -- 172.21.15.98:0/133226333 shutdown_connections 2024-05-09T08:59:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.533+0000 7fcf39573740 1 --2- 172.21.15.98:0/133226333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e575630 0x559c8e57f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.533+0000 7fcf39573740 1 --2- 172.21.15.98:0/133226333 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56ad70 0x559c8e591820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.533+0000 7fcf39573740 1 --2- 172.21.15.98:0/133226333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e56a450 0x559c8e5912e0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.533+0000 7fcf39573740 1 -- 172.21.15.98:0/133226333 >> 172.21.15.98:0/133226333 conn(0x559c8e566960 msgr2=0x559c8e579170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.533+0000 7fcf39573740 1 -- 172.21.15.98:0/133226333 shutdown_connections 2024-05-09T08:59:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.533+0000 7fcf39573740 1 -- 172.21.15.98:0/133226333 wait complete. 2024-05-09T08:59:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.533+0000 7fcf39573740 1 -- start start 2024-05-09T08:59:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf39573740 1 Processor -- start 2024-05-09T08:59:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf39573740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56a450 0x559c8e5856c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf39573740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56ad70 0x559c8e585c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf39573740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e575630 0x559c8e582b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf39573740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf301ae150 con 0x559c8e575630 2024-05-09T08:59:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf39573740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcf301ada80 con 0x559c8e56ad70 2024-05-09T08:59:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf39573740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcf301aace0 con 0x559c8e56a450 2024-05-09T08:59:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf39573740 10 client.0 _ll_register_callbacks cb 0x7fcf24009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:59:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf2f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e575630 0x559c8e582b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf2f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e575630 0x559c8e582b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37186/0 (socket says 172.21.15.98:37186) 2024-05-09T08:59:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf2f7fe640 1 -- 172.21.15.98:0/963057522 learned_addr learned my addr 172.21.15.98:0/963057522 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T08:59:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/963057522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56a450 0x559c8e5856c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf2ffff640 1 --2- 172.21.15.98:0/963057522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56ad70 0x559c8e585c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf2f7fe640 1 -- 172.21.15.98:0/963057522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56a450 msgr2=0x559c8e5856c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/963057522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56a450 0x559c8e5856c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf2f7fe640 1 -- 172.21.15.98:0/963057522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56ad70 msgr2=0x559c8e585c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:59:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/963057522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56ad70 0x559c8e585c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf2f7fe640 1 -- 172.21.15.98:0/963057522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c8e583370 con 0x559c8e575630 2024-05-09T08:59:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf2f7fe640 1 --2- 172.21.15.98:0/963057522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e575630 0x559c8e582b70 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009900 tx=0x7fcf24046040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45f6cd2c04562115 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:59:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.534+0000 7fcf0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T08:59:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf0f7fe640 1 -- 172.21.15.98:0/963057522 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf24035070 con 0x559c8e575630 2024-05-09T08:59:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf0f7fe640 1 -- 172.21.15.98:0/963057522 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9562+0+0 (secure 0 0 0) 0x7fcf240035f0 con 0x559c8e575630 2024-05-09T08:59:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf39573740 1 -- 172.21.15.98:0/963057522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf24041070 con 0x559c8e575630 2024-05-09T08:59:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf0f7fe640 1 -- 172.21.15.98:0/963057522 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fcf24031a00 con 0x559c8e575630 2024-05-09T08:59:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf39573740 10 client.18306 fetch_fsmap learned FSMap version 37 2024-05-09T08:59:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf39573740 1 -- 172.21.15.98:0/963057522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559c8e5ef590 con 0x559c8e575630 2024-05-09T08:59:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf0f7fe640 1 -- 172.21.15.98:0/963057522 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf24041070 con 0x559c8e575630 2024-05-09T08:59:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf0f7fe640 1 -- 172.21.15.98:0/963057522 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fcf24005020 con 0x559c8e575630 2024-05-09T08:59:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf39573740 10 client.18306 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T08:59:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf39573740 10 client.18306 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T08:59:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf39573740 20 client.18306 populate_metadata read hostname 'smithi098' 2024-05-09T08:59:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf39573740 1 --2- 172.21.15.98:0/963057522 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c8e5efbf0 0x559c8e5f2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:59:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf39573740 4 client.18306 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:59:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf39573740 1 -- 172.21.15.98:0/963057522 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559c8e4d6dc0 con 0x559c8e5efbf0 2024-05-09T08:59:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.535+0000 7fcf2ffff640 1 --2- 172.21.15.98:0/963057522 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c8e5efbf0 0x559c8e5f2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:59:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.536+0000 7fcf0f7fe640 10 client.18306 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.536+0000 7fcf2ffff640 1 --2- 172.21.15.98:0/963057522 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c8e5efbf0 0x559c8e5f2010 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1334a42ffd3c5b3a server_cookie=407d8072dd04ccf in_seq=0 out_seq=0 2024-05-09T08:59:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf0f7fe640 1 -- 172.21.15.98:0/963057522 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559c8e4d6dc0 con 0x559c8e5efbf0 2024-05-09T08:59:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf0f7fe640 10 client.18306 handle_command_reply: tid=0 2024-05-09T08:59:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf39573740 1 client.18306 shutdown 2024-05-09T08:59:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf39573740 1 -- 172.21.15.98:0/963057522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e575630 msgr2=0x559c8e582b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf39573740 1 --2- 172.21.15.98:0/963057522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e575630 0x559c8e582b70 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009900 tx=0x7fcf24046040 comp rx=0 tx=0).stop 2024-05-09T08:59:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf39573740 1 -- 172.21.15.98:0/963057522 shutdown_connections 2024-05-09T08:59:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf39573740 1 --2- 172.21.15.98:0/963057522 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559c8e5efbf0 0x559c8e5f2010 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf39573740 1 --2- 172.21.15.98:0/963057522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559c8e575630 0x559c8e582b70 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf0f7fe640 0 client.18306 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T08:59:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf39573740 1 --2- 172.21.15.98:0/963057522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c8e56ad70 0x559c8e585c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf39573740 1 --2- 172.21.15.98:0/963057522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c8e56a450 0x559c8e5856c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf39573740 1 -- 172.21.15.98:0/963057522 >> 172.21.15.98:0/963057522 conn(0x559c8e566960 msgr2=0x559c8e57f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf39573740 1 -- 172.21.15.98:0/963057522 shutdown_connections 2024-05-09T08:59:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf39573740 1 -- 172.21.15.98:0/963057522 wait complete. 2024-05-09T08:59:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.540+0000 7fcf39573740 20 client.18306 trim_cache size 0 max 16384 2024-05-09T08:59:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T08:59:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T08:59:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T08:59:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T08:59:54.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.541+0000 7fcf36864640 1 -- 172.21.15.98:0/3922799688 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcf180a2da0 msgr2=0x7fcf180a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:54.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.541+0000 7fcf36864640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcf180a2da0 0x7fcf180a5260 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcf280ccd80 tx=0x7fcf2806db10 comp rx=0 tx=0).stop 2024-05-09T08:59:54.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.541+0000 7fcf36864640 1 -- 172.21.15.98:0/3922799688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3010be00 msgr2=0x7fcf3019c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:59:54.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.541+0000 7fcf36864640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3010be00 0x7fcf3019c930 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800eaa0 tx=0x7fcf2800ef60 comp rx=0 tx=0).stop 2024-05-09T08:59:54.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.541+0000 7fcf36864640 1 -- 172.21.15.98:0/3922799688 shutdown_connections 2024-05-09T08:59:54.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.541+0000 7fcf36864640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcf180a2da0 0x7fcf180a5260 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.541+0000 7fcf36864640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3010be00 0x7fcf3019c930 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.541+0000 7fcf36864640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf301033e0 0x7fcf30198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.541+0000 7fcf36864640 1 --2- 172.21.15.98:0/3922799688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf30102ac0 0x7fcf301980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:59:54.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.541+0000 7fcf36864640 1 -- 172.21.15.98:0/3922799688 >> 172.21.15.98:0/3922799688 conn(0x7fcf300778b0 msgr2=0x7fcf300fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:59:54.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.541+0000 7fcf36864640 1 -- 172.21.15.98:0/3922799688 shutdown_connections 2024-05-09T08:59:54.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T08:59:54.541+0000 7fcf36864640 1 -- 172.21.15.98:0/3922799688 wait complete. 2024-05-09T08:59:54.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:54 smithi098 ceph-mon[30092]: pgmap v288: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:59:56.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:56 smithi142 ceph-mon[34625]: pgmap v289: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-05-09T08:59:56.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:56 smithi105 ceph-mon[34498]: pgmap v289: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-05-09T08:59:56.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:56 smithi098 ceph-mon[30092]: pgmap v289: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-05-09T08:59:57.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:57 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T08:59:57.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:57 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T08:59:57.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:57 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T08:59:58.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 08:59:58 smithi142 ceph-mon[34625]: pgmap v290: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 511 B/s wr, 15 op/s 2024-05-09T08:59:58.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 08:59:58 smithi105 ceph-mon[34498]: pgmap v290: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 511 B/s wr, 15 op/s 2024-05-09T08:59:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 08:59:58 smithi098 ceph-mon[30092]: pgmap v290: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 511 B/s wr, 15 op/s 2024-05-09T09:00:00.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:00 smithi142 ceph-mon[34625]: pgmap v291: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:00:00.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:00 smithi142 ceph-mon[34625]: overall HEALTH_OK 2024-05-09T09:00:00.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:00 smithi105 ceph-mon[34498]: pgmap v291: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:00:00.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:00 smithi105 ceph-mon[34498]: overall HEALTH_OK 2024-05-09T09:00:00.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:00 smithi098 ceph-mon[30092]: pgmap v291: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:00:00.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:00 smithi098 ceph-mon[30092]: overall HEALTH_OK 2024-05-09T09:00:02.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:02 smithi142 ceph-mon[34625]: pgmap v292: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:00:02.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:02 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:00:02.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:02 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:00:02.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:02 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:00:02.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:02 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:00:02.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:02 smithi105 ceph-mon[34498]: pgmap v292: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:00:02.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:02 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:00:02.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:02 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:00:02.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:02 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:00:02.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:02 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:00:02.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:02 smithi098 ceph-mon[30092]: pgmap v292: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:00:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:02 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:00:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:02 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:00:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:02 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:00:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:02 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:00:04.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:04 smithi142 ceph-mon[34625]: pgmap v293: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-05-09T09:00:04.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:04 smithi105 ceph-mon[34498]: pgmap v293: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-05-09T09:00:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:04 smithi098 ceph-mon[30092]: pgmap v293: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-05-09T09:00:06.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:06 smithi142 ceph-mon[34625]: pgmap v294: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-05-09T09:00:06.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:06 smithi105 ceph-mon[34498]: pgmap v294: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-05-09T09:00:06.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:06 smithi098 ceph-mon[30092]: pgmap v294: 129 pgs: 129 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-05-09T09:00:07.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:07 smithi105 ceph-mon[34498]: osdmap e104: 12 total, 12 up, 12 in 2024-05-09T09:00:07.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:07 smithi098 ceph-mon[30092]: osdmap e104: 12 total, 12 up, 12 in 2024-05-09T09:00:07.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:07 smithi142 ceph-mon[34625]: osdmap e104: 12 total, 12 up, 12 in 2024-05-09T09:00:07.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.682+0000 7fb86ea12640 1 -- 172.21.15.98:0/1627948995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8681007b0 msgr2=0x7fb868104470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:07.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.682+0000 7fb86ea12640 1 --2- 172.21.15.98:0/1627948995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8681007b0 0x7fb868104470 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fb858009860 tx=0x7fb85802a380 comp rx=0 tx=0).stop 2024-05-09T09:00:07.689 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.689+0000 7fb86ea12640 1 -- 172.21.15.98:0/1627948995 shutdown_connections 2024-05-09T09:00:07.690 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.689+0000 7fb86ea12640 1 --2- 172.21.15.98:0/1627948995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8681007b0 0x7fb868104470 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.690 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.689+0000 7fb86ea12640 1 --2- 172.21.15.98:0/1627948995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8680ffd30 0x7fb868100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.690 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.689+0000 7fb86ea12640 1 --2- 172.21.15.98:0/1627948995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb868106970 0x7fb868106d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.690 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.689+0000 7fb86ea12640 1 -- 172.21.15.98:0/1627948995 >> 172.21.15.98:0/1627948995 conn(0x7fb868076cc0 msgr2=0x7fb8680fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.696+0000 7fb86ea12640 1 -- 172.21.15.98:0/1627948995 shutdown_connections 2024-05-09T09:00:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.698+0000 7fb86ea12640 1 -- 172.21.15.98:0/1627948995 wait complete. 2024-05-09T09:00:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.699+0000 7fb86ea12640 1 Processor -- start 2024-05-09T09:00:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.700+0000 7fb86ea12640 1 -- start start 2024-05-09T09:00:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.700+0000 7fb86ea12640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8680ffd30 0x7fb8681a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.700+0000 7fb86ea12640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8681007b0 0x7fb8681a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.700+0000 7fb86ea12640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb868106970 0x7fb86819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.700+0000 7fb86ea12640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb868118460 con 0x7fb8680ffd30 2024-05-09T09:00:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.700+0000 7fb86ea12640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb8681182e0 con 0x7fb868106970 2024-05-09T09:00:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.700+0000 7fb86ea12640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb8681185e0 con 0x7fb8681007b0 2024-05-09T09:00:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.701+0000 7fb86cfaf640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb868106970 0x7fb86819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.701+0000 7fb86cfaf640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb868106970 0x7fb86819e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:44640/0 (socket says 172.21.15.98:44640) 2024-05-09T09:00:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.701+0000 7fb86cfaf640 1 -- 172.21.15.98:0/1181063831 learned_addr learned my addr 172.21.15.98:0/1181063831 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.701+0000 7fb86cfaf640 1 -- 172.21.15.98:0/1181063831 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8681007b0 msgr2=0x7fb8681a4ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T09:00:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.701+0000 7fb867fff640 1 --2- 172.21.15.98:0/1181063831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8680ffd30 0x7fb8681a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.701+0000 7fb86cfaf640 1 --2- 172.21.15.98:0/1181063831 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8681007b0 0x7fb8681a4ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.701+0000 7fb86cfaf640 1 -- 172.21.15.98:0/1181063831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8680ffd30 msgr2=0x7fb8681a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.701+0000 7fb86cfaf640 1 --2- 172.21.15.98:0/1181063831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8680ffd30 0x7fb8681a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.701+0000 7fb86cfaf640 1 -- 172.21.15.98:0/1181063831 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb86819ee70 con 0x7fb868106970 2024-05-09T09:00:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.701+0000 7fb867fff640 1 --2- 172.21.15.98:0/1181063831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8680ffd30 0x7fb8681a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.702+0000 7fb86cfaf640 1 --2- 172.21.15.98:0/1181063831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb868106970 0x7fb86819e660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb858009980 tx=0x7fb858030ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.702+0000 7fb8657fa640 1 -- 172.21.15.98:0/1181063831 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb858040020 con 0x7fb868106970 2024-05-09T09:00:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.703+0000 7fb8657fa640 1 -- 172.21.15.98:0/1181063831 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb858032740 con 0x7fb868106970 2024-05-09T09:00:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.703+0000 7fb8657fa640 1 -- 172.21.15.98:0/1181063831 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb858037040 con 0x7fb868106970 2024-05-09T09:00:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.704+0000 7fb86ea12640 1 -- 172.21.15.98:0/1181063831 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb86819f100 con 0x7fb868106970 2024-05-09T09:00:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.705+0000 7fb86ea12640 1 -- 172.21.15.98:0/1181063831 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb868107460 con 0x7fb868106970 2024-05-09T09:00:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.708+0000 7fb8717fa740 1 Processor -- start 2024-05-09T09:00:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.709+0000 7fb8717fa740 1 -- start start 2024-05-09T09:00:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb8717fa740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403db450 0x5613403db830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb8717fa740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403dbd70 0x5613403e60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb8717fa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403e8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb8717fa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561340340340 con 0x5613403e6630 2024-05-09T09:00:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb8717fa740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5613402e8290 con 0x5613403dbd70 2024-05-09T09:00:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb8717fa740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56134033c300 con 0x5613403db450 2024-05-09T09:00:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb8657fa640 1 -- 172.21.15.98:0/1181063831 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb858043020 con 0x7fb868106970 2024-05-09T09:00:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb867fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403dbd70 0x5613403e60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb867fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403dbd70 0x5613403e60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:44646/0 (socket says 172.21.15.98:44646) 2024-05-09T09:00:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb867fff640 1 -- 172.21.15.98:0/2561517796 learned_addr learned my addr 172.21.15.98:0/2561517796 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb867fff640 1 -- 172.21.15.98:0/2561517796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403db450 msgr2=0x5613403db830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T09:00:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb867fff640 1 --2- 172.21.15.98:0/2561517796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403db450 0x5613403db830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb867fff640 1 -- 172.21.15.98:0/2561517796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 msgr2=0x5613403e8a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T09:00:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb867fff640 1 --2- 172.21.15.98:0/2561517796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403e8a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.710+0000 7fb867fff640 1 -- 172.21.15.98:0/2561517796 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56134030c8e0 con 0x5613403dbd70 2024-05-09T09:00:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.711+0000 7fb867fff640 1 --2- 172.21.15.98:0/2561517796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403dbd70 0x5613403e60f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb860009f00 tx=0x7fb86000f7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65b8e6940572135e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.711+0000 7fb8657fa640 1 --2- 172.21.15.98:0/1181063831 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8500a2f80 0x7fb8500a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.712+0000 7fb8657fa640 1 -- 172.21.15.98:0/1181063831 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9650+0+0 (secure 0 0 0) 0x7fb8580ef2f0 con 0x7fb868106970 2024-05-09T09:00:07.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.714+0000 7fb84e7fc640 1 -- 172.21.15.98:0/2561517796 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86001c070 con 0x5613403dbd70 2024-05-09T09:00:07.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.714+0000 7fb84e7fc640 1 -- 172.21.15.98:0/2561517796 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb86000db50 con 0x5613403dbd70 2024-05-09T09:00:07.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.714+0000 7fb84e7fc640 1 -- 172.21.15.98:0/2561517796 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb860017d40 con 0x5613403dbd70 2024-05-09T09:00:07.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.717+0000 7fb8677fe640 1 --2- 172.21.15.98:0/1181063831 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8500a2f80 0x7fb8500a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.717+0000 7fb8717fa740 1 -- 172.21.15.98:0/2561517796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403dbd70 msgr2=0x5613403e60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.717+0000 7fb8717fa740 1 --2- 172.21.15.98:0/2561517796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403dbd70 0x5613403e60f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb860009f00 tx=0x7fb86000f7d0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.718+0000 7fb8717fa740 1 -- 172.21.15.98:0/2561517796 shutdown_connections 2024-05-09T09:00:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.718+0000 7fb8717fa740 1 --2- 172.21.15.98:0/2561517796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403e8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.718+0000 7fb8717fa740 1 --2- 172.21.15.98:0/2561517796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403dbd70 0x5613403e60f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.718+0000 7fb8717fa740 1 --2- 172.21.15.98:0/2561517796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403db450 0x5613403db830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.718+0000 7fb8717fa740 1 -- 172.21.15.98:0/2561517796 >> 172.21.15.98:0/2561517796 conn(0x5613403d7960 msgr2=0x5613403d7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.718+0000 7fb8677fe640 1 --2- 172.21.15.98:0/1181063831 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8500a2f80 0x7fb8500a5440 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x5613403d8d80 tx=0x7fb85c006c00 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.720+0000 7fb8717fa740 1 -- 172.21.15.98:0/2561517796 shutdown_connections 2024-05-09T09:00:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.724+0000 7fb8717fa740 1 -- 172.21.15.98:0/2561517796 wait complete. 2024-05-09T09:00:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.725+0000 7fb8717fa740 1 -- start start 2024-05-09T09:00:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.725+0000 7fb8717fa740 1 Processor -- start 2024-05-09T09:00:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.725+0000 7fb8717fa740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403db450 0x5613403e3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.725+0000 7fb8717fa740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403dbd70 0x5613403e3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.725+0000 7fb8717fa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403e3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.725+0000 7fb8717fa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5613403e9260 con 0x5613403e6630 2024-05-09T09:00:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.725+0000 7fb8717fa740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5613402fb790 con 0x5613403db450 2024-05-09T09:00:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.725+0000 7fb8717fa740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5613403e93e0 con 0x5613403dbd70 2024-05-09T09:00:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.725+0000 7fb8717fa740 10 client.0 _ll_register_callbacks cb 0x561340336a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.728+0000 7fb86cfaf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403e3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.728+0000 7fb86cfaf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403e3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45974/0 (socket says 172.21.15.98:45974) 2024-05-09T09:00:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.728+0000 7fb86cfaf640 1 -- 172.21.15.98:0/4292472303 learned_addr learned my addr 172.21.15.98:0/4292472303 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.729+0000 7fb867fff640 1 --2- 172.21.15.98:0/4292472303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403db450 0x5613403e3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.729+0000 7fb8677fe640 1 --2- 172.21.15.98:0/4292472303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403dbd70 0x5613403e3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.730+0000 7fb867fff640 1 -- 172.21.15.98:0/4292472303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403dbd70 msgr2=0x5613403e3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.730+0000 7fb867fff640 1 --2- 172.21.15.98:0/4292472303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403dbd70 0x5613403e3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.730+0000 7fb867fff640 1 -- 172.21.15.98:0/4292472303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 msgr2=0x5613403e3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.730+0000 7fb867fff640 1 --2- 172.21.15.98:0/4292472303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403e3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.730+0000 7fb867fff640 1 -- 172.21.15.98:0/4292472303 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561340054210 con 0x5613403db450 2024-05-09T09:00:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.732+0000 7fb867fff640 1 --2- 172.21.15.98:0/4292472303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403db450 0x5613403e3550 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb860006010 tx=0x7fb860009c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e09d3386ded270e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.737+0000 7fb8477fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T09:00:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.737+0000 7fb8477fe640 1 -- 172.21.15.98:0/4292472303 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86001c040 con 0x5613403db450 2024-05-09T09:00:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.737+0000 7fb8477fe640 1 -- 172.21.15.98:0/4292472303 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9650+0+0 (secure 0 0 0) 0x7fb86000d7f0 con 0x5613403db450 2024-05-09T09:00:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.737+0000 7fb8717fa740 1 -- 172.21.15.98:0/4292472303 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb86000db50 con 0x5613403db450 2024-05-09T09:00:07.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.741+0000 7fb8477fe640 1 -- 172.21.15.98:0/4292472303 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb860017a80 con 0x5613403db450 2024-05-09T09:00:07.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.741+0000 7fb8717fa740 10 client.27230 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:07.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.741+0000 7fb8717fa740 1 -- 172.21.15.98:0/4292472303 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561340120620 con 0x5613403db450 2024-05-09T09:00:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.748+0000 7fb8477fe640 1 -- 172.21.15.98:0/4292472303 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb86000db50 con 0x5613403db450 2024-05-09T09:00:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.748+0000 7fb8477fe640 1 -- 172.21.15.98:0/4292472303 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb86001f070 con 0x5613403db450 2024-05-09T09:00:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.748+0000 7fb8717fa740 10 client.27230 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.748+0000 7fb8717fa740 10 client.27230 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.748+0000 7fb8717fa740 20 client.27230 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.749+0000 7fb8717fa740 1 --2- 172.21.15.98:0/4292472303 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561340407310 0x561340409700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.749+0000 7fb8717fa740 4 client.27230 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.749+0000 7fb8717fa740 1 -- 172.21.15.98:0/4292472303 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56134034da70 con 0x561340407310 2024-05-09T09:00:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.749+0000 7fb8677fe640 1 --2- 172.21.15.98:0/4292472303 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561340407310 0x561340409700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.751+0000 7fb8677fe640 1 --2- 172.21.15.98:0/4292472303 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561340407310 0x561340409700 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6656a04ce9136e7a server_cookie=b03353043253cd15 in_seq=0 out_seq=0 2024-05-09T09:00:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.754+0000 7fb8477fe640 10 client.27230 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.754+0000 7fb8477fe640 1 -- 172.21.15.98:0/4292472303 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56134034da70 con 0x561340407310 2024-05-09T09:00:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.754+0000 7fb8477fe640 10 client.27230 handle_command_reply: tid=0 2024-05-09T09:00:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.754+0000 7fb8717fa740 1 client.27230 shutdown 2024-05-09T09:00:07.761 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.760+0000 7fb8717fa740 1 -- 172.21.15.98:0/4292472303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403db450 msgr2=0x5613403e3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:07.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.760+0000 7fb8717fa740 1 --2- 172.21.15.98:0/4292472303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403db450 0x5613403e3550 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb860006010 tx=0x7fb860009c70 comp rx=0 tx=0).stop 2024-05-09T09:00:07.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.764+0000 7fb8717fa740 1 -- 172.21.15.98:0/4292472303 shutdown_connections 2024-05-09T09:00:07.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.764+0000 7fb8717fa740 1 --2- 172.21.15.98:0/4292472303 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561340407310 0x561340409700 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.764+0000 7fb8717fa740 1 --2- 172.21.15.98:0/4292472303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403e3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.764+0000 7fb8717fa740 1 --2- 172.21.15.98:0/4292472303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403dbd70 0x5613403e3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.764+0000 7fb8717fa740 1 --2- 172.21.15.98:0/4292472303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403db450 0x5613403e3550 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.764+0000 7fb8717fa740 1 -- 172.21.15.98:0/4292472303 >> 172.21.15.98:0/4292472303 conn(0x5613403d7960 msgr2=0x5613403dd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:07.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.766+0000 7fb8477fe640 0 client.27230 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:07.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.767+0000 7fb8717fa740 1 -- 172.21.15.98:0/4292472303 shutdown_connections 2024-05-09T09:00:07.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.768+0000 7fb8717fa740 1 -- 172.21.15.98:0/4292472303 wait complete. 2024-05-09T09:00:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.768+0000 7fb8717fa740 20 client.27230 trim_cache size 0 max 16384 2024-05-09T09:00:07.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.800+0000 7fb8717fa740 1 Processor -- start 2024-05-09T09:00:07.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.801+0000 7fb8717fa740 1 -- start start 2024-05-09T09:00:07.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.801+0000 7fb8717fa740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403db450 0x5613403f2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.801+0000 7fb8717fa740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403dbd70 0x5613403e3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.801+0000 7fb8717fa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403e4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.801+0000 7fb8717fa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb86810a170 con 0x5613403e6630 2024-05-09T09:00:07.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.802+0000 7fb8717fa740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb86810bcb0 con 0x5613403dbd70 2024-05-09T09:00:07.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.802+0000 7fb8717fa740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb86810a840 con 0x5613403db450 2024-05-09T09:00:07.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.802+0000 7fb86cfaf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403e4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:07.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.802+0000 7fb86cfaf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403e4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45990/0 (socket says 172.21.15.98:45990) 2024-05-09T09:00:07.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.802+0000 7fb86cfaf640 1 -- 172.21.15.98:0/402634847 learned_addr learned my addr 172.21.15.98:0/402634847 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:07.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.802+0000 7fb867fff640 1 --2- 172.21.15.98:0/402634847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403db450 0x5613403f2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:07.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.802+0000 7fb8677fe640 1 --2- 172.21.15.98:0/402634847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403dbd70 0x5613403e3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:07.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.803+0000 7fb867fff640 1 -- 172.21.15.98:0/402634847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403dbd70 msgr2=0x5613403e3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:07.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.803+0000 7fb867fff640 1 --2- 172.21.15.98:0/402634847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403dbd70 0x5613403e3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.803+0000 7fb867fff640 1 -- 172.21.15.98:0/402634847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 msgr2=0x5613403e4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:07.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.803+0000 7fb867fff640 1 --2- 172.21.15.98:0/402634847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403e4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.803+0000 7fb867fff640 1 -- 172.21.15.98:0/402634847 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613403d5f00 con 0x5613403db450 2024-05-09T09:00:07.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.803+0000 7fb867fff640 1 --2- 172.21.15.98:0/402634847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403db450 0x5613403f2180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb8600042a0 tx=0x7fb860009d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f809a516a24b206 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:07.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.804+0000 7fb84dffb640 1 -- 172.21.15.98:0/402634847 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86001c070 con 0x5613403db450 2024-05-09T09:00:07.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.804+0000 7fb84dffb640 1 -- 172.21.15.98:0/402634847 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb86000d7f0 con 0x5613403db450 2024-05-09T09:00:07.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.804+0000 7fb8717fa740 1 -- 172.21.15.98:0/402634847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403db450 msgr2=0x5613403f2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:07.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.805+0000 7fb8717fa740 1 --2- 172.21.15.98:0/402634847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403db450 0x5613403f2180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb8600042a0 tx=0x7fb860009d80 comp rx=0 tx=0).stop 2024-05-09T09:00:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.806+0000 7fb8717fa740 1 -- 172.21.15.98:0/402634847 shutdown_connections 2024-05-09T09:00:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.806+0000 7fb8717fa740 1 --2- 172.21.15.98:0/402634847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403e4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.806+0000 7fb8717fa740 1 --2- 172.21.15.98:0/402634847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403dbd70 0x5613403e3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.806+0000 7fb8717fa740 1 --2- 172.21.15.98:0/402634847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403db450 0x5613403f2180 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.806+0000 7fb8717fa740 1 -- 172.21.15.98:0/402634847 >> 172.21.15.98:0/402634847 conn(0x5613403d7960 msgr2=0x5613403e3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.809+0000 7fb8717fa740 1 -- 172.21.15.98:0/402634847 shutdown_connections 2024-05-09T09:00:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.811+0000 7fb8717fa740 1 -- 172.21.15.98:0/402634847 wait complete. 2024-05-09T09:00:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.812+0000 7fb8717fa740 1 -- start start 2024-05-09T09:00:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.812+0000 7fb8717fa740 1 Processor -- start 2024-05-09T09:00:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.812+0000 7fb8717fa740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403db450 0x5613403ee690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.812+0000 7fb8717fa740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403dbd70 0x5613403eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.812+0000 7fb8717fa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403eb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.812+0000 7fb8717fa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb86810b5e0 con 0x5613403e6630 2024-05-09T09:00:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.812+0000 7fb8717fa740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb86810af10 con 0x5613403db450 2024-05-09T09:00:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.812+0000 7fb8717fa740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb86804f5f0 con 0x5613403dbd70 2024-05-09T09:00:07.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.812+0000 7fb8717fa740 10 client.0 _ll_register_callbacks cb 0x7fb86001a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:07.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.813+0000 7fb867fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403db450 0x5613403ee690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:07.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.813+0000 7fb867fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403db450 0x5613403ee690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:44670/0 (socket says 172.21.15.98:44670) 2024-05-09T09:00:07.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.813+0000 7fb867fff640 1 -- 172.21.15.98:0/1108866718 learned_addr learned my addr 172.21.15.98:0/1108866718 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:07.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.813+0000 7fb86cfaf640 1 --2- 172.21.15.98:0/1108866718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403eb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.813+0000 7fb867fff640 1 -- 172.21.15.98:0/1108866718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403dbd70 msgr2=0x5613403eebd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:00:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.813+0000 7fb867fff640 1 --2- 172.21.15.98:0/1108866718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403dbd70 0x5613403eebd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.813+0000 7fb867fff640 1 -- 172.21.15.98:0/1108866718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 msgr2=0x5613403eb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.813+0000 7fb867fff640 1 --2- 172.21.15.98:0/1108866718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403eb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.813+0000 7fb867fff640 1 -- 172.21.15.98:0/1108866718 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613403ebe90 con 0x5613403db450 2024-05-09T09:00:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.814+0000 7fb867fff640 1 --2- 172.21.15.98:0/1108866718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403db450 0x5613403ee690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb86000f810 tx=0x7fb860009280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cb435adc3915675 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.814+0000 7fb8477fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T09:00:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.814+0000 7fb8477fe640 1 -- 172.21.15.98:0/1108866718 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86001c070 con 0x5613403db450 2024-05-09T09:00:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.814+0000 7fb8717fa740 1 -- 172.21.15.98:0/1108866718 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb86000d7f0 con 0x5613403db450 2024-05-09T09:00:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.814+0000 7fb8477fe640 1 -- 172.21.15.98:0/1108866718 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9650+0+0 (secure 0 0 0) 0x7fb8600049d0 con 0x5613403db450 2024-05-09T09:00:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.815+0000 7fb8477fe640 1 -- 172.21.15.98:0/1108866718 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb860039050 con 0x5613403db450 2024-05-09T09:00:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.817+0000 7fb8717fa740 10 client.27236 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.817+0000 7fb8717fa740 1 -- 172.21.15.98:0/1108866718 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56134046e130 con 0x5613403db450 2024-05-09T09:00:07.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.818+0000 7fb8477fe640 1 -- 172.21.15.98:0/1108866718 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb86000d7f0 con 0x5613403db450 2024-05-09T09:00:07.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.818+0000 7fb8477fe640 1 -- 172.21.15.98:0/1108866718 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb860034030 con 0x5613403db450 2024-05-09T09:00:07.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.819+0000 7fb8717fa740 10 client.27236 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:07.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.819+0000 7fb8717fa740 10 client.27236 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:07.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.819+0000 7fb8717fa740 20 client.27236 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:07.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.819+0000 7fb8717fa740 1 --2- 172.21.15.98:0/1108866718 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56134046e6e0 0x561340470ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.819+0000 7fb8717fa740 4 client.27236 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:00:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.819+0000 7fb8717fa740 1 -- 172.21.15.98:0/1108866718 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561340347dc0 con 0x56134046e6e0 2024-05-09T09:00:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.821+0000 7fb8677fe640 1 --2- 172.21.15.98:0/1108866718 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56134046e6e0 0x561340470ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.823+0000 7fb8677fe640 1 --2- 172.21.15.98:0/1108866718 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56134046e6e0 0x561340470ad0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a85381e5639eb6b1 server_cookie=3726175deef8970a in_seq=0 out_seq=0 2024-05-09T09:00:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.823+0000 7fb8477fe640 10 client.27236 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.846+0000 7fb8477fe640 1 -- 172.21.15.98:0/1108866718 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561340347dc0 con 0x56134046e6e0 2024-05-09T09:00:07.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.846+0000 7fb8477fe640 10 client.27236 handle_command_reply: tid=0 2024-05-09T09:00:07.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.846+0000 7fb8717fa740 1 client.27236 shutdown 2024-05-09T09:00:07.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.858+0000 7fb8717fa740 1 -- 172.21.15.98:0/1108866718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403db450 msgr2=0x5613403ee690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:07.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.858+0000 7fb8717fa740 1 --2- 172.21.15.98:0/1108866718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403db450 0x5613403ee690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb86000f810 tx=0x7fb860009280 comp rx=0 tx=0).stop 2024-05-09T09:00:07.858 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.859+0000 7fb8717fa740 1 -- 172.21.15.98:0/1108866718 shutdown_connections 2024-05-09T09:00:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.859+0000 7fb8717fa740 1 --2- 172.21.15.98:0/1108866718 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56134046e6e0 0x561340470ad0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.859+0000 7fb8717fa740 1 --2- 172.21.15.98:0/1108866718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613403e6630 0x5613403eb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.859+0000 7fb8717fa740 1 --2- 172.21.15.98:0/1108866718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5613403dbd70 0x5613403eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.859+0000 7fb8717fa740 1 --2- 172.21.15.98:0/1108866718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5613403db450 0x5613403ee690 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.859+0000 7fb8717fa740 1 -- 172.21.15.98:0/1108866718 >> 172.21.15.98:0/1108866718 conn(0x5613403d7960 msgr2=0x5613403e3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.860+0000 7fb8477fe640 0 client.27236 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.861+0000 7fb8717fa740 1 -- 172.21.15.98:0/1108866718 shutdown_connections 2024-05-09T09:00:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.861+0000 7fb8717fa740 1 -- 172.21.15.98:0/1108866718 wait complete. 2024-05-09T09:00:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.864+0000 7fb8717fa740 20 client.27236 trim_cache size 0 max 16384 2024-05-09T09:00:07.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T09:00:07.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T09:00:07.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T09:00:07.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T09:00:07.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.866+0000 7fb86ea12640 1 -- 172.21.15.98:0/1181063831 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8500a2f80 msgr2=0x7fb8500a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.866+0000 7fb86ea12640 1 --2- 172.21.15.98:0/1181063831 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8500a2f80 0x7fb8500a5440 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x5613403d8d80 tx=0x7fb85c006c00 comp rx=0 tx=0).stop 2024-05-09T09:00:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.866+0000 7fb86ea12640 1 -- 172.21.15.98:0/1181063831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb868106970 msgr2=0x7fb86819e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.866+0000 7fb86ea12640 1 --2- 172.21.15.98:0/1181063831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb868106970 0x7fb86819e660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb858009980 tx=0x7fb858030ea0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.868+0000 7fb86ea12640 1 -- 172.21.15.98:0/1181063831 shutdown_connections 2024-05-09T09:00:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.868+0000 7fb86ea12640 1 --2- 172.21.15.98:0/1181063831 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb8500a2f80 0x7fb8500a5440 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.868+0000 7fb86ea12640 1 --2- 172.21.15.98:0/1181063831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb868106970 0x7fb86819e660 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.868+0000 7fb86ea12640 1 --2- 172.21.15.98:0/1181063831 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8681007b0 0x7fb8681a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.868+0000 7fb86ea12640 1 --2- 172.21.15.98:0/1181063831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8680ffd30 0x7fb8681a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.868+0000 7fb86ea12640 1 -- 172.21.15.98:0/1181063831 >> 172.21.15.98:0/1181063831 conn(0x7fb868076cc0 msgr2=0x7fb8681022d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.868+0000 7fb86ea12640 1 -- 172.21.15.98:0/1181063831 shutdown_connections 2024-05-09T09:00:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:07.868+0000 7fb86ea12640 1 -- 172.21.15.98:0/1181063831 wait complete. 2024-05-09T09:00:08.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:08 smithi105 ceph-mon[34498]: pgmap v296: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-05-09T09:00:08.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:08 smithi098 ceph-mon[30092]: pgmap v296: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-05-09T09:00:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:08 smithi142 ceph-mon[34625]: pgmap v296: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-05-09T09:00:10.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:10 smithi105 ceph-mon[34498]: pgmap v297: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 39 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-05-09T09:00:10.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:10 smithi098 ceph-mon[30092]: pgmap v297: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 39 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-05-09T09:00:10.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:10 smithi142 ceph-mon[34625]: pgmap v297: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 39 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-05-09T09:00:12.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:12 smithi105 ceph-mon[34498]: pgmap v298: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 33 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-05-09T09:00:12.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:12 smithi098 ceph-mon[30092]: pgmap v298: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 33 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-05-09T09:00:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:12 smithi142 ceph-mon[34625]: pgmap v298: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 33 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-05-09T09:00:14.475 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:14 smithi105 ceph-mon[34498]: pgmap v299: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 31 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-05-09T09:00:14.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:14 smithi098 ceph-mon[30092]: pgmap v299: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 31 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-05-09T09:00:14.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:14 smithi142 ceph-mon[34625]: pgmap v299: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 31 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-05-09T09:00:16.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:16 smithi105 ceph-mon[34498]: pgmap v300: 129 pgs: 129 active+clean; 30 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 28 KiB/s wr, 13 op/s 2024-05-09T09:00:16.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:16 smithi098 ceph-mon[30092]: pgmap v300: 129 pgs: 129 active+clean; 30 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 28 KiB/s wr, 13 op/s 2024-05-09T09:00:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:16 smithi142 ceph-mon[34625]: pgmap v300: 129 pgs: 129 active+clean; 30 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 28 KiB/s wr, 13 op/s 2024-05-09T09:00:18.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:18 smithi105 ceph-mon[34498]: pgmap v301: 129 pgs: 129 active+clean; 30 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 959 B/s wr, 11 op/s 2024-05-09T09:00:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:18 smithi105 ceph-mon[34498]: osdmap e105: 12 total, 12 up, 12 in 2024-05-09T09:00:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:18 smithi098 ceph-mon[30092]: pgmap v301: 129 pgs: 129 active+clean; 30 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 959 B/s wr, 11 op/s 2024-05-09T09:00:18.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:18 smithi098 ceph-mon[30092]: osdmap e105: 12 total, 12 up, 12 in 2024-05-09T09:00:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:18 smithi142 ceph-mon[34625]: pgmap v301: 129 pgs: 129 active+clean; 30 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 959 B/s wr, 11 op/s 2024-05-09T09:00:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:18 smithi142 ceph-mon[34625]: osdmap e105: 12 total, 12 up, 12 in 2024-05-09T09:00:20.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:20 smithi105 ceph-mon[34498]: pgmap v303: 129 pgs: 129 active+clean; 30 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 921 B/s wr, 10 op/s 2024-05-09T09:00:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:20 smithi098 ceph-mon[30092]: pgmap v303: 129 pgs: 129 active+clean; 30 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 921 B/s wr, 10 op/s 2024-05-09T09:00:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:20 smithi142 ceph-mon[34625]: pgmap v303: 129 pgs: 129 active+clean; 30 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 921 B/s wr, 10 op/s 2024-05-09T09:00:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.978+0000 7f3055c10640 1 -- 172.21.15.98:0/3578091135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3050100760 msgr2=0x7f3050100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.978+0000 7f3055c10640 1 --2- 172.21.15.98:0/3578091135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3050100760 0x7f3050100be0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f3044009930 tx=0x7f304402a420 comp rx=0 tx=0).stop 2024-05-09T09:00:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.978+0000 7f3055c10640 1 -- 172.21.15.98:0/3578091135 shutdown_connections 2024-05-09T09:00:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.978+0000 7f3055c10640 1 --2- 172.21.15.98:0/3578091135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3050101120 0x7f305010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.978+0000 7f3055c10640 1 --2- 172.21.15.98:0/3578091135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3050100760 0x7f3050100be0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.978+0000 7f3055c10640 1 --2- 172.21.15.98:0/3578091135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3050106720 0x7f3050106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.978+0000 7f3055c10640 1 -- 172.21.15.98:0/3578091135 >> 172.21.15.98:0/3578091135 conn(0x7f30500778b0 msgr2=0x7f3050077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.978+0000 7f3055c10640 1 -- 172.21.15.98:0/3578091135 shutdown_connections 2024-05-09T09:00:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.978+0000 7f3055c10640 1 -- 172.21.15.98:0/3578091135 wait complete. 2024-05-09T09:00:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.979+0000 7f3055c10640 1 Processor -- start 2024-05-09T09:00:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.979+0000 7f3055c10640 1 -- start start 2024-05-09T09:00:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.979+0000 7f3055c10640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3050100760 0x7f3050198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.979+0000 7f3055c10640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3050101120 0x7f3050198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.979+0000 7f3055c10640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3050106720 0x7f305019c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.979+0000 7f3055c10640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3050110b30 con 0x7f3050101120 2024-05-09T09:00:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.979+0000 7f3055c10640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f30501109b0 con 0x7f3050106720 2024-05-09T09:00:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.979+0000 7f3055c10640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3050110cb0 con 0x7f3050100760 2024-05-09T09:00:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3050101120 0x7f3050198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3050101120 0x7f3050198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34962/0 (socket says 172.21.15.98:34962) 2024-05-09T09:00:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304effd640 1 -- 172.21.15.98:0/3994372328 learned_addr learned my addr 172.21.15.98:0/3994372328 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304ffff640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3050106720 0x7f305019c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304f7fe640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3050100760 0x7f3050198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304effd640 1 -- 172.21.15.98:0/3994372328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3050100760 msgr2=0x7f3050198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304effd640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3050100760 0x7f3050198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304effd640 1 -- 172.21.15.98:0/3994372328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3050106720 msgr2=0x7f305019c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304effd640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3050106720 0x7f305019c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304effd640 1 -- 172.21.15.98:0/3994372328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f305019d110 con 0x7f3050101120 2024-05-09T09:00:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304ffff640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3050106720 0x7f305019c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304effd640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3050101120 0x7f3050198660 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f3044009230 tx=0x7f3044004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304cff9640 1 -- 172.21.15.98:0/3994372328 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304403f030 con 0x7f3050101120 2024-05-09T09:00:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304cff9640 1 -- 172.21.15.98:0/3994372328 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3044003ad0 con 0x7f3050101120 2024-05-09T09:00:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f3055c10640 1 -- 172.21.15.98:0/3994372328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f305019d3a0 con 0x7f3050101120 2024-05-09T09:00:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f304cff9640 1 -- 172.21.15.98:0/3994372328 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3044035070 con 0x7f3050101120 2024-05-09T09:00:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.980+0000 7f3055c10640 1 -- 172.21.15.98:0/3994372328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30501a4c90 con 0x7f3050101120 2024-05-09T09:00:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.983+0000 7f304cff9640 1 -- 172.21.15.98:0/3994372328 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3044042030 con 0x7f3050101120 2024-05-09T09:00:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.983+0000 7f30589f5740 1 Processor -- start 2024-05-09T09:00:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f30589f5740 1 -- start start 2024-05-09T09:00:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f30589f5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f27860 0x55d4f8f27c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f30589f5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f28180 0x55d4f8f32500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f30589f5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f32a40 0x55d4f8f34e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f30589f5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d4f8e88340 con 0x55d4f8f28180 2024-05-09T09:00:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f30589f5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d4f8e30290 con 0x55d4f8f32a40 2024-05-09T09:00:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f30589f5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d4f8e84300 con 0x55d4f8f27860 2024-05-09T09:00:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f28180 0x55d4f8f32500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f28180 0x55d4f8f32500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34966/0 (socket says 172.21.15.98:34966) 2024-05-09T09:00:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304f7fe640 1 -- 172.21.15.98:0/3431899953 learned_addr learned my addr 172.21.15.98:0/3431899953 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304ffff640 1 --2- 172.21.15.98:0/3431899953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f27860 0x55d4f8f27c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304cff9640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f30380a2da0 0x7f30380a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304effd640 1 --2- 172.21.15.98:0/3431899953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f32a40 0x55d4f8f34e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304f7fe640 1 -- 172.21.15.98:0/3431899953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f27860 msgr2=0x55d4f8f27c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304f7fe640 1 --2- 172.21.15.98:0/3431899953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f27860 0x55d4f8f27c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304f7fe640 1 -- 172.21.15.98:0/3431899953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f32a40 msgr2=0x55d4f8f34e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304f7fe640 1 --2- 172.21.15.98:0/3431899953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f32a40 0x55d4f8f34e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304ffff640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f30380a2da0 0x7f30380a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304f7fe640 1 -- 172.21.15.98:0/3431899953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4f8e548e0 con 0x55d4f8f28180 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304effd640 1 --2- 172.21.15.98:0/3431899953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f32a40 0x55d4f8f34e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304cff9640 1 -- 172.21.15.98:0/3994372328 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f30440efe10 con 0x7f3050101120 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.984+0000 7f304f7fe640 1 --2- 172.21.15.98:0/3431899953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f28180 0x55d4f8f32500 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f3040009af0 tx=0x7f304000e2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccac9649faceb1d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.985+0000 7f3035ffb640 1 -- 172.21.15.98:0/3431899953 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040016070 con 0x55d4f8f28180 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.985+0000 7f3035ffb640 1 -- 172.21.15.98:0/3431899953 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3040005500 con 0x55d4f8f28180 2024-05-09T09:00:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.985+0000 7f3035ffb640 1 -- 172.21.15.98:0/3431899953 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040011470 con 0x55d4f8f28180 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.985+0000 7f30589f5740 1 -- 172.21.15.98:0/3431899953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f28180 msgr2=0x55d4f8f32500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.985+0000 7f304ffff640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f30380a2da0 0x7f30380a5260 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3048012590 tx=0x7f3048004ab0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.985+0000 7f30589f5740 1 --2- 172.21.15.98:0/3431899953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f28180 0x55d4f8f32500 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f3040009af0 tx=0x7f304000e2f0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.985+0000 7f30589f5740 1 -- 172.21.15.98:0/3431899953 shutdown_connections 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.985+0000 7f30589f5740 1 --2- 172.21.15.98:0/3431899953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f32a40 0x55d4f8f34e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.985+0000 7f30589f5740 1 --2- 172.21.15.98:0/3431899953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f28180 0x55d4f8f32500 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.985+0000 7f30589f5740 1 --2- 172.21.15.98:0/3431899953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f27860 0x55d4f8f27c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.985+0000 7f30589f5740 1 -- 172.21.15.98:0/3431899953 >> 172.21.15.98:0/3431899953 conn(0x55d4f8f23d70 msgr2=0x55d4f8f24150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.985+0000 7f30589f5740 1 -- 172.21.15.98:0/3431899953 shutdown_connections 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.986+0000 7f30589f5740 1 -- 172.21.15.98:0/3431899953 wait complete. 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.986+0000 7f30589f5740 1 -- start start 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.986+0000 7f30589f5740 1 Processor -- start 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.986+0000 7f30589f5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f27860 0x55d4f8f2f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.986+0000 7f30589f5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f28180 0x55d4f8f2fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.986+0000 7f30589f5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f32a40 0x55d4f8f30330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.986+0000 7f30589f5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d4f8f35670 con 0x55d4f8f32a40 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.986+0000 7f30589f5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d4f8e43790 con 0x55d4f8f28180 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.986+0000 7f30589f5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d4f8f357f0 con 0x55d4f8f27860 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.986+0000 7f30589f5740 10 client.0 _ll_register_callbacks cb 0x55d4f8e7ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f304ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f32a40 0x55d4f8f30330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f304ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f32a40 0x55d4f8f30330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34974/0 (socket says 172.21.15.98:34974) 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f304ffff640 1 -- 172.21.15.98:0/3137191055 learned_addr learned my addr 172.21.15.98:0/3137191055 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f304effd640 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f28180 0x55d4f8f2fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f304f7fe640 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f27860 0x55d4f8f2f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f304ffff640 1 -- 172.21.15.98:0/3137191055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f27860 msgr2=0x55d4f8f2f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f304ffff640 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f27860 0x55d4f8f2f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f304ffff640 1 -- 172.21.15.98:0/3137191055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f28180 msgr2=0x55d4f8f2fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f304ffff640 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f28180 0x55d4f8f2fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f304ffff640 1 -- 172.21.15.98:0/3137191055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4f8b9c210 con 0x55d4f8f32a40 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f304effd640 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f28180 0x55d4f8f2fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f304f7fe640 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f27860 0x55d4f8f2f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f304ffff640 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f32a40 0x55d4f8f30330 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f3048010f50 tx=0x7f304800a5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dd31c44467eb08d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f302effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f302effd640 1 -- 172.21.15.98:0/3137191055 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304801cca0 con 0x55d4f8f32a40 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f302effd640 1 -- 172.21.15.98:0/3137191055 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f304801a400 con 0x55d4f8f32a40 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f30589f5740 1 -- 172.21.15.98:0/3137191055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3040005500 con 0x55d4f8f32a40 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f302effd640 1 -- 172.21.15.98:0/3137191055 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3048031050 con 0x55d4f8f32a40 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f30589f5740 10 client.18327 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.987+0000 7f30589f5740 1 -- 172.21.15.98:0/3137191055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d4f8c68620 con 0x55d4f8f32a40 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.988+0000 7f302effd640 1 -- 172.21.15.98:0/3137191055 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3040005500 con 0x55d4f8f32a40 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.988+0000 7f302effd640 1 -- 172.21.15.98:0/3137191055 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f304800f740 con 0x55d4f8f32a40 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.988+0000 7f30589f5740 10 client.18327 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.988+0000 7f30589f5740 10 client.18327 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.988+0000 7f30589f5740 20 client.18327 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.988+0000 7f30589f5740 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d4f8f426e0 0x55d4f8f44af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.988+0000 7f30589f5740 4 client.18327 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.988+0000 7f30589f5740 1 -- 172.21.15.98:0/3137191055 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4f8e95a70 con 0x55d4f8f426e0 2024-05-09T09:00:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.988+0000 7f304f7fe640 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d4f8f426e0 0x55d4f8f44af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.988+0000 7f302effd640 10 client.18327 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.988+0000 7f304f7fe640 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d4f8f426e0 0x55d4f8f44af0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2c9a7dc7f33b388 server_cookie=559257fa1f53baa3 in_seq=0 out_seq=0 2024-05-09T09:00:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.989+0000 7f302effd640 1 -- 172.21.15.98:0/3137191055 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d4f8e95a70 con 0x55d4f8f426e0 2024-05-09T09:00:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.989+0000 7f302effd640 10 client.18327 handle_command_reply: tid=0 2024-05-09T09:00:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.989+0000 7f30589f5740 1 client.18327 shutdown 2024-05-09T09:00:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.989+0000 7f30589f5740 1 -- 172.21.15.98:0/3137191055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f32a40 msgr2=0x55d4f8f30330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.989+0000 7f30589f5740 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f32a40 0x55d4f8f30330 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f3048010f50 tx=0x7f304800a5a0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.989+0000 7f30589f5740 1 -- 172.21.15.98:0/3137191055 shutdown_connections 2024-05-09T09:00:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.989+0000 7f30589f5740 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d4f8f426e0 0x55d4f8f44af0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.989+0000 7f30589f5740 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f32a40 0x55d4f8f30330 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.989+0000 7f30589f5740 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f28180 0x55d4f8f2fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.989+0000 7f30589f5740 1 --2- 172.21.15.98:0/3137191055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f27860 0x55d4f8f2f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:20.996 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.989+0000 7f302effd640 0 client.18327 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:20.996 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.989+0000 7f30589f5740 1 -- 172.21.15.98:0/3137191055 >> 172.21.15.98:0/3137191055 conn(0x55d4f8f23d70 msgr2=0x55d4f8f29aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:20.996 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.989+0000 7f30589f5740 1 -- 172.21.15.98:0/3137191055 shutdown_connections 2024-05-09T09:00:20.996 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.990+0000 7f30589f5740 1 -- 172.21.15.98:0/3137191055 wait complete. 2024-05-09T09:00:20.996 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:20.990+0000 7f30589f5740 20 client.18327 trim_cache size 0 max 16384 2024-05-09T09:00:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.004+0000 7f30589f5740 1 Processor -- start 2024-05-09T09:00:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.004+0000 7f30589f5740 1 -- start start 2024-05-09T09:00:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.004+0000 7f30589f5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f27860 0x55d4f8f31fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.004+0000 7f30589f5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f28180 0x55d4f8f2a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.004+0000 7f30589f5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f32a40 0x55d4f8f4a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.004+0000 7f30589f5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30501accc0 con 0x55d4f8f32a40 2024-05-09T09:00:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.004+0000 7f30589f5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f30501ae800 con 0x55d4f8f27860 2024-05-09T09:00:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.004+0000 7f30589f5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f30501ad390 con 0x55d4f8f28180 2024-05-09T09:00:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f304ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f32a40 0x55d4f8f4a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f304ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f32a40 0x55d4f8f4a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34978/0 (socket says 172.21.15.98:34978) 2024-05-09T09:00:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f304ffff640 1 -- 172.21.15.98:0/3460485110 learned_addr learned my addr 172.21.15.98:0/3460485110 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f304effd640 1 --2- 172.21.15.98:0/3460485110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f28180 0x55d4f8f2a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f304f7fe640 1 --2- 172.21.15.98:0/3460485110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f27860 0x55d4f8f31fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f304ffff640 1 -- 172.21.15.98:0/3460485110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f28180 msgr2=0x55d4f8f2a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f304ffff640 1 --2- 172.21.15.98:0/3460485110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f28180 0x55d4f8f2a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f304ffff640 1 -- 172.21.15.98:0/3460485110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f27860 msgr2=0x55d4f8f31fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f304ffff640 1 --2- 172.21.15.98:0/3460485110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f27860 0x55d4f8f31fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f304ffff640 1 -- 172.21.15.98:0/3460485110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4f8f1ac30 con 0x55d4f8f32a40 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f304effd640 1 --2- 172.21.15.98:0/3460485110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f28180 0x55d4f8f2a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f304f7fe640 1 --2- 172.21.15.98:0/3460485110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f27860 0x55d4f8f31fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f304ffff640 1 --2- 172.21.15.98:0/3460485110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f32a40 0x55d4f8f4a400 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f3048026490 tx=0x7f304800aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab52fdfcfdf81bd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f30357fa640 1 -- 172.21.15.98:0/3460485110 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304803a070 con 0x55d4f8f32a40 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f30357fa640 1 -- 172.21.15.98:0/3460485110 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f304800f920 con 0x55d4f8f32a40 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.005+0000 7f30357fa640 1 -- 172.21.15.98:0/3460485110 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30480267b0 con 0x55d4f8f32a40 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.006+0000 7f30589f5740 1 -- 172.21.15.98:0/3460485110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f32a40 msgr2=0x55d4f8f4a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.006+0000 7f30589f5740 1 --2- 172.21.15.98:0/3460485110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f32a40 0x55d4f8f4a400 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f3048026490 tx=0x7f304800aa90 comp rx=0 tx=0).stop 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.006+0000 7f30589f5740 1 -- 172.21.15.98:0/3460485110 shutdown_connections 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.006+0000 7f30589f5740 1 --2- 172.21.15.98:0/3460485110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f32a40 0x55d4f8f4a400 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.006+0000 7f30589f5740 1 --2- 172.21.15.98:0/3460485110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f28180 0x55d4f8f2a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.006+0000 7f30589f5740 1 --2- 172.21.15.98:0/3460485110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f27860 0x55d4f8f31fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.006+0000 7f30589f5740 1 -- 172.21.15.98:0/3460485110 >> 172.21.15.98:0/3460485110 conn(0x55d4f8f23d70 msgr2=0x55d4f8f1e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.006+0000 7f30589f5740 1 -- 172.21.15.98:0/3460485110 shutdown_connections 2024-05-09T09:00:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.006+0000 7f30589f5740 1 -- 172.21.15.98:0/3460485110 wait complete. 2024-05-09T09:00:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.006+0000 7f30589f5740 1 -- start start 2024-05-09T09:00:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.006+0000 7f30589f5740 1 Processor -- start 2024-05-09T09:00:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.006+0000 7f30589f5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f27860 0x55d4f8f2c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.006+0000 7f30589f5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f28180 0x55d4f8f2ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.006+0000 7f30589f5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f32a40 0x55d4f8fa7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f30589f5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30501ae130 con 0x55d4f8f27860 2024-05-09T09:00:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f30589f5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f30501ada60 con 0x55d4f8f32a40 2024-05-09T09:00:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f30589f5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f30501aacc0 con 0x55d4f8f28180 2024-05-09T09:00:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f30589f5740 10 client.0 _ll_register_callbacks cb 0x55d4f8f32990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f304f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f27860 0x55d4f8f2c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f304f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f27860 0x55d4f8f2c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34980/0 (socket says 172.21.15.98:34980) 2024-05-09T09:00:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f304f7fe640 1 -- 172.21.15.98:0/485007106 learned_addr learned my addr 172.21.15.98:0/485007106 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f304effd640 1 --2- 172.21.15.98:0/485007106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f28180 0x55d4f8f2ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f304ffff640 1 --2- 172.21.15.98:0/485007106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f32a40 0x55d4f8fa7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f304f7fe640 1 -- 172.21.15.98:0/485007106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f28180 msgr2=0x55d4f8f2ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f304f7fe640 1 --2- 172.21.15.98:0/485007106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f28180 0x55d4f8f2ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f304f7fe640 1 -- 172.21.15.98:0/485007106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f32a40 msgr2=0x55d4f8fa7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f304f7fe640 1 --2- 172.21.15.98:0/485007106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f32a40 0x55d4f8fa7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f304f7fe640 1 -- 172.21.15.98:0/485007106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4f8fa81b0 con 0x55d4f8f27860 2024-05-09T09:00:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f304effd640 1 --2- 172.21.15.98:0/485007106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f28180 0x55d4f8f2ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f304f7fe640 1 --2- 172.21.15.98:0/485007106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f27860 0x55d4f8f2c920 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f3040004b60 tx=0x7f3040009400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a70ba47f3e95fa5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f302effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f302effd640 1 -- 172.21.15.98:0/485007106 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040016040 con 0x55d4f8f27860 2024-05-09T09:00:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f302effd640 1 -- 172.21.15.98:0/485007106 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f304000ed20 con 0x55d4f8f27860 2024-05-09T09:00:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f30589f5740 1 -- 172.21.15.98:0/485007106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d4f8e8fdc0 con 0x55d4f8f27860 2024-05-09T09:00:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f302effd640 1 -- 172.21.15.98:0/485007106 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3040027020 con 0x55d4f8f27860 2024-05-09T09:00:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f30589f5740 10 client.18333 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f30589f5740 1 -- 172.21.15.98:0/485007106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d4f8fa9290 con 0x55d4f8f27860 2024-05-09T09:00:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.007+0000 7f302effd640 1 -- 172.21.15.98:0/485007106 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d4f8e8fdc0 con 0x55d4f8f27860 2024-05-09T09:00:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.008+0000 7f302effd640 1 -- 172.21.15.98:0/485007106 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f304000f040 con 0x55d4f8f27860 2024-05-09T09:00:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.008+0000 7f30589f5740 10 client.18333 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.008+0000 7f30589f5740 10 client.18333 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.008+0000 7f30589f5740 20 client.18333 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.008+0000 7f30589f5740 1 --2- 172.21.15.98:0/485007106 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d4f8fa9870 0x55d4f8fabc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.008+0000 7f30589f5740 4 client.18333 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:00:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.008+0000 7f30589f5740 1 -- 172.21.15.98:0/485007106 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d4f8e0d8e0 con 0x55d4f8fa9870 2024-05-09T09:00:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.008+0000 7f304effd640 1 --2- 172.21.15.98:0/485007106 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d4f8fa9870 0x55d4f8fabc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.008+0000 7f302effd640 10 client.18333 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.008+0000 7f304effd640 1 --2- 172.21.15.98:0/485007106 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d4f8fa9870 0x55d4f8fabc60 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1742bfde617aab54 server_cookie=41f6715ead416b13 in_seq=0 out_seq=0 2024-05-09T09:00:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f302effd640 1 -- 172.21.15.98:0/485007106 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d4f8e0d8e0 con 0x55d4f8fa9870 2024-05-09T09:00:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f302effd640 10 client.18333 handle_command_reply: tid=0 2024-05-09T09:00:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f30589f5740 1 client.18333 shutdown 2024-05-09T09:00:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f30589f5740 1 -- 172.21.15.98:0/485007106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f27860 msgr2=0x55d4f8f2c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f30589f5740 1 --2- 172.21.15.98:0/485007106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f27860 0x55d4f8f2c920 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f3040004b60 tx=0x7f3040009400 comp rx=0 tx=0).stop 2024-05-09T09:00:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f30589f5740 1 -- 172.21.15.98:0/485007106 shutdown_connections 2024-05-09T09:00:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f30589f5740 1 --2- 172.21.15.98:0/485007106 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d4f8fa9870 0x55d4f8fabc60 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f30589f5740 1 --2- 172.21.15.98:0/485007106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4f8f32a40 0x55d4f8fa7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f30589f5740 1 --2- 172.21.15.98:0/485007106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d4f8f28180 0x55d4f8f2ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f302effd640 0 client.18333 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f30589f5740 1 --2- 172.21.15.98:0/485007106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d4f8f27860 0x55d4f8f2c920 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f30589f5740 1 -- 172.21.15.98:0/485007106 >> 172.21.15.98:0/485007106 conn(0x55d4f8f23d70 msgr2=0x55d4f8f29210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f30589f5740 1 -- 172.21.15.98:0/485007106 shutdown_connections 2024-05-09T09:00:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f30589f5740 1 -- 172.21.15.98:0/485007106 wait complete. 2024-05-09T09:00:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f30589f5740 20 client.18333 trim_cache size 0 max 16384 2024-05-09T09:00:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T09:00:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T09:00:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T09:00:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T09:00:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f3055c10640 1 -- 172.21.15.98:0/3994372328 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f30380a2da0 msgr2=0x7f30380a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f3055c10640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f30380a2da0 0x7f30380a5260 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3048012590 tx=0x7f3048004ab0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f3055c10640 1 -- 172.21.15.98:0/3994372328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3050101120 msgr2=0x7f3050198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.013+0000 7f3055c10640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3050101120 0x7f3050198660 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f3044009230 tx=0x7f3044004270 comp rx=0 tx=0).stop 2024-05-09T09:00:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.014+0000 7f3055c10640 1 -- 172.21.15.98:0/3994372328 shutdown_connections 2024-05-09T09:00:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.014+0000 7f3055c10640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f30380a2da0 0x7f30380a5260 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.014+0000 7f3055c10640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3050106720 0x7f305019c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.014+0000 7f3055c10640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3050101120 0x7f3050198660 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.014+0000 7f3055c10640 1 --2- 172.21.15.98:0/3994372328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3050100760 0x7f3050198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.014+0000 7f3055c10640 1 -- 172.21.15.98:0/3994372328 >> 172.21.15.98:0/3994372328 conn(0x7f30500778b0 msgr2=0x7f3050069280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.014+0000 7f3055c10640 1 -- 172.21.15.98:0/3994372328 shutdown_connections 2024-05-09T09:00:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:21.014+0000 7f3055c10640 1 -- 172.21.15.98:0/3994372328 wait complete. 2024-05-09T09:00:21.731 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:00:21.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:00:21.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.847+0000 7f3e8d15a640 1 -- 172.21.15.98:0/2507215870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e88101800 msgr2=0x7f3e88101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.847+0000 7f3e8d15a640 1 --2- 172.21.15.98:0/2507215870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e88101800 0x7f3e88101c80 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c009930 tx=0x7f3e7c02a420 comp rx=0 tx=0).stop 2024-05-09T09:00:21.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.847+0000 7f3e8d15a640 1 -- 172.21.15.98:0/2507215870 shutdown_connections 2024-05-09T09:00:21.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.847+0000 7f3e8d15a640 1 --2- 172.21.15.98:0/2507215870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e881021c0 0x7f3e8810e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.847+0000 7f3e8d15a640 1 --2- 172.21.15.98:0/2507215870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e88101800 0x7f3e88101c80 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.847+0000 7f3e8d15a640 1 --2- 172.21.15.98:0/2507215870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e88107810 0x7f3e88107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.847+0000 7f3e8d15a640 1 -- 172.21.15.98:0/2507215870 >> 172.21.15.98:0/2507215870 conn(0x7f3e88078c80 msgr2=0x7f3e88079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:21.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.847+0000 7f3e8d15a640 1 -- 172.21.15.98:0/2507215870 shutdown_connections 2024-05-09T09:00:21.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.847+0000 7f3e8d15a640 1 -- 172.21.15.98:0/2507215870 wait complete. 2024-05-09T09:00:21.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.848+0000 7f3e8d15a640 1 Processor -- start 2024-05-09T09:00:21.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.848+0000 7f3e8d15a640 1 -- start start 2024-05-09T09:00:21.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8d15a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e88101800 0x7f3e88198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8d15a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e881021c0 0x7f3e88198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8d15a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e88107810 0x7f3e8819ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8d15a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e88111c40 con 0x7f3e88107810 2024-05-09T09:00:21.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8d15a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3e88111ac0 con 0x7f3e88101800 2024-05-09T09:00:21.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8d15a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3e88111dc0 con 0x7f3e881021c0 2024-05-09T09:00:21.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e88107810 0x7f3e8819ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e88107810 0x7f3e8819ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34986/0 (socket says 172.21.15.98:34986) 2024-05-09T09:00:21.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8759e640 1 -- 172.21.15.98:0/1996667401 learned_addr learned my addr 172.21.15.98:0/1996667401 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:21.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e86d9d640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e88101800 0x7f3e88198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8659c640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e881021c0 0x7f3e88198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8759e640 1 -- 172.21.15.98:0/1996667401 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e881021c0 msgr2=0x7f3e88198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8759e640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e881021c0 0x7f3e88198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8759e640 1 -- 172.21.15.98:0/1996667401 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e88101800 msgr2=0x7f3e88198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8759e640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e88101800 0x7f3e88198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8759e640 1 -- 172.21.15.98:0/1996667401 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8819d1c0 con 0x7f3e88107810 2024-05-09T09:00:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e86d9d640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e88101800 0x7f3e88198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8759e640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e88107810 0x7f3e8819ca90 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009fd0 tx=0x7f3e8000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e6ffff640 1 -- 172.21.15.98:0/1996667401 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e80011830 con 0x7f3e88107810 2024-05-09T09:00:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8d15a640 1 -- 172.21.15.98:0/1996667401 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e8819d4b0 con 0x7f3e88107810 2024-05-09T09:00:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e6ffff640 1 -- 172.21.15.98:0/1996667401 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e80016070 con 0x7f3e88107810 2024-05-09T09:00:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e6ffff640 1 -- 172.21.15.98:0/1996667401 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8000f2e0 con 0x7f3e88107810 2024-05-09T09:00:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.849+0000 7f3e8d15a640 1 -- 172.21.15.98:0/1996667401 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e881a4d40 con 0x7f3e88107810 2024-05-09T09:00:21.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.852+0000 7f3e6ffff640 1 -- 172.21.15.98:0/1996667401 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3e800119d0 con 0x7f3e88107810 2024-05-09T09:00:21.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.852+0000 7f3e8fe66740 1 Processor -- start 2024-05-09T09:00:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.852+0000 7f3e8fe66740 1 -- start start 2024-05-09T09:00:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.852+0000 7f3e8fe66740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bb2280 0x55ef99bb2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.852+0000 7f3e8fe66740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bb2ba0 0x55ef99bbcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.852+0000 7f3e8fe66740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bbd270 0x55ef99bbf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.852+0000 7f3e8fe66740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ef99b17d00 con 0x55ef99bbd270 2024-05-09T09:00:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.852+0000 7f3e8fe66740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ef99abf4c0 con 0x55ef99bb2280 2024-05-09T09:00:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.852+0000 7f3e8fe66740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ef99b13cc0 con 0x55ef99bb2ba0 2024-05-09T09:00:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e8659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bbd270 0x55ef99bbf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e8659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bbd270 0x55ef99bbf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34992/0 (socket says 172.21.15.98:34992) 2024-05-09T09:00:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e8659c640 1 -- 172.21.15.98:0/2550115406 learned_addr learned my addr 172.21.15.98:0/2550115406 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e86d9d640 1 --2- 172.21.15.98:0/2550115406 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bb2ba0 0x55ef99bbcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e8659c640 1 --2- 172.21.15.98:0/2550115406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bb2280 0x55ef99bb2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e6ffff640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e700a2d00 0x7f3e700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e8659c640 1 -- 172.21.15.98:0/2550115406 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bb2ba0 msgr2=0x55ef99bbcd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e8659c640 1 --2- 172.21.15.98:0/2550115406 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bb2ba0 0x55ef99bbcd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e8659c640 1 -- 172.21.15.98:0/2550115406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bb2280 msgr2=0x55ef99bb2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e8659c640 1 --2- 172.21.15.98:0/2550115406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bb2280 0x55ef99bb2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e8659c640 1 -- 172.21.15.98:0/2550115406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef998f57d0 con 0x55ef99bbd270 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e8759e640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e700a2d00 0x7f3e700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e6ffff640 1 -- 172.21.15.98:0/1996667401 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f3e800cef10 con 0x7f3e88107810 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e8659c640 1 --2- 172.21.15.98:0/2550115406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bbd270 0x55ef99bbf660 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c005ab0 tx=0x7f3e7c0058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df4c9ce0c0771fc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e6d7fa640 1 -- 172.21.15.98:0/2550115406 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c043030 con 0x55ef99bbd270 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e6d7fa640 1 -- 172.21.15.98:0/2550115406 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e7c005020 con 0x55ef99bbd270 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.853+0000 7f3e6d7fa640 1 -- 172.21.15.98:0/2550115406 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c035050 con 0x55ef99bbd270 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.854+0000 7f3e8fe66740 1 -- 172.21.15.98:0/2550115406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bbd270 msgr2=0x55ef99bbf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.854+0000 7f3e8759e640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e700a2d00 0x7f3e700a51c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3e80097aa0 tx=0x7f3e800660f0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.854+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/2550115406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bbd270 0x55ef99bbf660 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c005ab0 tx=0x7f3e7c0058b0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.854+0000 7f3e8fe66740 1 -- 172.21.15.98:0/2550115406 shutdown_connections 2024-05-09T09:00:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.854+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/2550115406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bbd270 0x55ef99bbf660 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.854+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/2550115406 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bb2ba0 0x55ef99bbcd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.854+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/2550115406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bb2280 0x55ef99bb2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.854+0000 7f3e8fe66740 1 -- 172.21.15.98:0/2550115406 >> 172.21.15.98:0/2550115406 conn(0x55ef99bae5a0 msgr2=0x55ef99bae980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.854+0000 7f3e8fe66740 1 -- 172.21.15.98:0/2550115406 shutdown_connections 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.854+0000 7f3e8fe66740 1 -- 172.21.15.98:0/2550115406 wait complete. 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.854+0000 7f3e8fe66740 1 -- start start 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.854+0000 7f3e8fe66740 1 Processor -- start 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e8fe66740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bb2280 0x55ef99bba250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e8fe66740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2ba0 0x55ef99bba790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e8fe66740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bbd270 0x55ef99bbacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e8fe66740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ef99bc0000 con 0x55ef99bb2ba0 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e8fe66740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ef99ad2b90 con 0x55ef99bb2280 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e8fe66740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ef99bc0180 con 0x55ef99bbd270 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e8fe66740 10 client.0 _ll_register_callbacks cb 0x55ef99b0eda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e86d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2ba0 0x55ef99bba790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e86d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2ba0 0x55ef99bba790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34996/0 (socket says 172.21.15.98:34996) 2024-05-09T09:00:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e86d9d640 1 -- 172.21.15.98:0/3690089880 learned_addr learned my addr 172.21.15.98:0/3690089880 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e8659c640 1 --2- 172.21.15.98:0/3690089880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bb2280 0x55ef99bba250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e8659c640 1 --2- 172.21.15.98:0/3690089880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bbd270 0x55ef99bbacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e86d9d640 1 -- 172.21.15.98:0/3690089880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bbd270 msgr2=0x55ef99bbacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e86d9d640 1 --2- 172.21.15.98:0/3690089880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bbd270 0x55ef99bbacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e86d9d640 1 -- 172.21.15.98:0/3690089880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bb2280 msgr2=0x55ef99bba250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e86d9d640 1 --2- 172.21.15.98:0/3690089880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bb2280 0x55ef99bba250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e86d9d640 1 -- 172.21.15.98:0/3690089880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef9982b3f0 con 0x55ef99bb2ba0 2024-05-09T09:00:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e8659c640 1 --2- 172.21.15.98:0/3690089880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bbd270 0x55ef99bbacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e86d9d640 1 --2- 172.21.15.98:0/3690089880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2ba0 0x55ef99bba790 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f3e78012860 tx=0x7f3e78012d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286fc9cee495ef9d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e6aff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e6aff5640 1 -- 172.21.15.98:0/3690089880 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e78010dc0 con 0x55ef99bb2ba0 2024-05-09T09:00:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e6aff5640 1 -- 172.21.15.98:0/3690089880 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f3e78003970 con 0x55ef99bb2ba0 2024-05-09T09:00:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.855+0000 7f3e8fe66740 1 -- 172.21.15.98:0/3690089880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e7c005020 con 0x55ef99bb2ba0 2024-05-09T09:00:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.856+0000 7f3e6aff5640 1 -- 172.21.15.98:0/3690089880 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3e78003b70 con 0x55ef99bb2ba0 2024-05-09T09:00:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.856+0000 7f3e8fe66740 10 client.18345 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.856+0000 7f3e8fe66740 1 -- 172.21.15.98:0/3690089880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ef997eb0f0 con 0x55ef99bb2ba0 2024-05-09T09:00:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.856+0000 7f3e6aff5640 1 -- 172.21.15.98:0/3690089880 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e7c005020 con 0x55ef99bb2ba0 2024-05-09T09:00:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.856+0000 7f3e6aff5640 1 -- 172.21.15.98:0/3690089880 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3e78019070 con 0x55ef99bb2ba0 2024-05-09T09:00:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.856+0000 7f3e8fe66740 10 client.18345 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.856+0000 7f3e8fe66740 10 client.18345 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.856+0000 7f3e8fe66740 20 client.18345 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.856+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/3690089880 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ef99bcd640 0x55ef99bcfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.856+0000 7f3e8fe66740 4 client.18345 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.856+0000 7f3e8fe66740 1 -- 172.21.15.98:0/3690089880 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef99b25430 con 0x55ef99bcd640 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.856+0000 7f3e8659c640 1 --2- 172.21.15.98:0/3690089880 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ef99bcd640 0x55ef99bcfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.857+0000 7f3e6aff5640 10 client.18345 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.857+0000 7f3e8659c640 1 --2- 172.21.15.98:0/3690089880 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ef99bcd640 0x55ef99bcfae0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=401f1e327575a61 server_cookie=2be5ed3cd8dd7af6 in_seq=0 out_seq=0 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.857+0000 7f3e6aff5640 1 -- 172.21.15.98:0/3690089880 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef99b25430 con 0x55ef99bcd640 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.857+0000 7f3e6aff5640 10 client.18345 handle_command_reply: tid=0 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.857+0000 7f3e8fe66740 1 client.18345 shutdown 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.858+0000 7f3e8fe66740 1 -- 172.21.15.98:0/3690089880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2ba0 msgr2=0x55ef99bba790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.858+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/3690089880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2ba0 0x55ef99bba790 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f3e78012860 tx=0x7f3e78012d20 comp rx=0 tx=0).stop 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.858+0000 7f3e8fe66740 1 -- 172.21.15.98:0/3690089880 shutdown_connections 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.858+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/3690089880 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ef99bcd640 0x55ef99bcfae0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.858+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/3690089880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bbd270 0x55ef99bbacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.858+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/3690089880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2ba0 0x55ef99bba790 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.858+0000 7f3e6aff5640 0 client.18345 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.858+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/3690089880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bb2280 0x55ef99bba250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.858+0000 7f3e8fe66740 1 -- 172.21.15.98:0/3690089880 >> 172.21.15.98:0/3690089880 conn(0x55ef99bae5a0 msgr2=0x55ef99bb43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.858+0000 7f3e8fe66740 1 -- 172.21.15.98:0/3690089880 shutdown_connections 2024-05-09T09:00:21.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.858+0000 7f3e8fe66740 1 -- 172.21.15.98:0/3690089880 wait complete. 2024-05-09T09:00:21.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.858+0000 7f3e8fe66740 20 client.18345 trim_cache size 0 max 16384 2024-05-09T09:00:21.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.875+0000 7f3e8fe66740 1 Processor -- start 2024-05-09T09:00:21.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.875+0000 7f3e8fe66740 1 -- start start 2024-05-09T09:00:21.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.875+0000 7f3e8fe66740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2280 0x55ef99bc8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.875+0000 7f3e8fe66740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bb2ba0 0x55ef99bb4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.875+0000 7f3e8fe66740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bbd270 0x55ef99bd53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.875+0000 7f3e8fe66740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e881aeeb0 con 0x55ef99bb2280 2024-05-09T09:00:21.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.875+0000 7f3e8fe66740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3e881b09f0 con 0x55ef99bb2ba0 2024-05-09T09:00:21.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.875+0000 7f3e8fe66740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3e881af580 con 0x55ef99bbd270 2024-05-09T09:00:21.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.875+0000 7f3e8659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2280 0x55ef99bc8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.875+0000 7f3e8659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2280 0x55ef99bc8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35008/0 (socket says 172.21.15.98:35008) 2024-05-09T09:00:21.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.875+0000 7f3e8659c640 1 -- 172.21.15.98:0/1468041760 learned_addr learned my addr 172.21.15.98:0/1468041760 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:21.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.876+0000 7f3e8659c640 1 -- 172.21.15.98:0/1468041760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bbd270 msgr2=0x55ef99bd53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.876+0000 7f3e8659c640 1 --2- 172.21.15.98:0/1468041760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bbd270 0x55ef99bd53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.876+0000 7f3e8659c640 1 -- 172.21.15.98:0/1468041760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bb2ba0 msgr2=0x55ef99bb4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.876+0000 7f3e8659c640 1 --2- 172.21.15.98:0/1468041760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bb2ba0 0x55ef99bb4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.876+0000 7f3e8659c640 1 -- 172.21.15.98:0/1468041760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef99baf7d0 con 0x55ef99bb2280 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.876+0000 7f3e8659c640 1 --2- 172.21.15.98:0/1468041760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2280 0x55ef99bc8ec0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x55ef99baf730 tx=0x7f3e7c0318a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b6588faa0ab762 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.876+0000 7f3e6cff9640 1 -- 172.21.15.98:0/1468041760 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c056070 con 0x55ef99bb2280 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.876+0000 7f3e6cff9640 1 -- 172.21.15.98:0/1468041760 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e7c009520 con 0x55ef99bb2280 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.876+0000 7f3e6cff9640 1 -- 172.21.15.98:0/1468041760 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c043040 con 0x55ef99bb2280 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.876+0000 7f3e8fe66740 1 -- 172.21.15.98:0/1468041760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2280 msgr2=0x55ef99bc8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.876+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/1468041760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2280 0x55ef99bc8ec0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x55ef99baf730 tx=0x7f3e7c0318a0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 -- 172.21.15.98:0/1468041760 shutdown_connections 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/1468041760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bbd270 0x55ef99bd53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/1468041760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bb2ba0 0x55ef99bb4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/1468041760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2280 0x55ef99bc8ec0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 -- 172.21.15.98:0/1468041760 >> 172.21.15.98:0/1468041760 conn(0x55ef99bae5a0 msgr2=0x55ef99bbaaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 -- 172.21.15.98:0/1468041760 shutdown_connections 2024-05-09T09:00:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 -- 172.21.15.98:0/1468041760 wait complete. 2024-05-09T09:00:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 -- start start 2024-05-09T09:00:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 Processor -- start 2024-05-09T09:00:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2280 0x55ef99bb7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bb2ba0 0x55ef99bb7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bbd270 0x55ef99c3b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e881b0320 con 0x55ef99bb2280 2024-05-09T09:00:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3e881afc50 con 0x55ef99bbd270 2024-05-09T09:00:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3e881aceb0 con 0x55ef99bb2ba0 2024-05-09T09:00:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8fe66740 10 client.0 _ll_register_callbacks cb 0x7f3e7800aad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2280 0x55ef99bb7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2280 0x55ef99bb7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35018/0 (socket says 172.21.15.98:35018) 2024-05-09T09:00:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8659c640 1 -- 172.21.15.98:0/1560112110 learned_addr learned my addr 172.21.15.98:0/1560112110 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:21.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e86d9d640 1 --2- 172.21.15.98:0/1560112110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bb2ba0 0x55ef99bb7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8659c640 1 -- 172.21.15.98:0/1560112110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bb2ba0 msgr2=0x55ef99bb7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:21.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8659c640 1 --2- 172.21.15.98:0/1560112110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bb2ba0 0x55ef99bb7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8659c640 1 -- 172.21.15.98:0/1560112110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bbd270 msgr2=0x55ef99c3b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:21.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8659c640 1 --2- 172.21.15.98:0/1560112110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bbd270 0x55ef99c3b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.877+0000 7f3e8659c640 1 -- 172.21.15.98:0/1560112110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef99c3b9b0 con 0x55ef99bb2280 2024-05-09T09:00:21.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e8659c640 1 --2- 172.21.15.98:0/1560112110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2280 0x55ef99bb7550 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0092c0 tx=0x7f3e7c0044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c9d8703e1a27f59 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:21.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e6aff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:21.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e6aff5640 1 -- 172.21.15.98:0/1560112110 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c043070 con 0x55ef99bb2280 2024-05-09T09:00:21.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e6aff5640 1 -- 172.21.15.98:0/1560112110 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f3e7c051580 con 0x55ef99bb2280 2024-05-09T09:00:21.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e8fe66740 1 -- 172.21.15.98:0/1560112110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e7c009520 con 0x55ef99bb2280 2024-05-09T09:00:21.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e6aff5640 1 -- 172.21.15.98:0/1560112110 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3e7c064050 con 0x55ef99bb2280 2024-05-09T09:00:21.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e8fe66740 10 client.18351 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:21.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e8fe66740 1 -- 172.21.15.98:0/1560112110 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ef99c3cbc0 con 0x55ef99bb2280 2024-05-09T09:00:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e6aff5640 1 -- 172.21.15.98:0/1560112110 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e7c009520 con 0x55ef99bb2280 2024-05-09T09:00:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e6aff5640 1 -- 172.21.15.98:0/1560112110 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3e7c03a480 con 0x55ef99bb2280 2024-05-09T09:00:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e8fe66740 10 client.18351 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e8fe66740 10 client.18351 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e8fe66740 20 client.18351 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/1560112110 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ef99c3cdf0 0x55ef99c3f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e8fe66740 4 client.18351 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:00:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.878+0000 7f3e8fe66740 1 -- 172.21.15.98:0/1560112110 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ef99b1f780 con 0x55ef99c3cdf0 2024-05-09T09:00:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e86d9d640 1 --2- 172.21.15.98:0/1560112110 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ef99c3cdf0 0x55ef99c3f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e6aff5640 10 client.18351 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e86d9d640 1 --2- 172.21.15.98:0/1560112110 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ef99c3cdf0 0x55ef99c3f1e0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8497af14f272ae9f server_cookie=a9c93178087b7009 in_seq=0 out_seq=0 2024-05-09T09:00:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e6aff5640 1 -- 172.21.15.98:0/1560112110 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ef99b1f780 con 0x55ef99c3cdf0 2024-05-09T09:00:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e6aff5640 10 client.18351 handle_command_reply: tid=0 2024-05-09T09:00:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e8fe66740 1 client.18351 shutdown 2024-05-09T09:00:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e8fe66740 1 -- 172.21.15.98:0/1560112110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2280 msgr2=0x55ef99bb7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/1560112110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2280 0x55ef99bb7550 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0092c0 tx=0x7f3e7c0044f0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e8fe66740 1 -- 172.21.15.98:0/1560112110 shutdown_connections 2024-05-09T09:00:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/1560112110 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55ef99c3cdf0 0x55ef99c3f1e0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/1560112110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef99bbd270 0x55ef99c3b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/1560112110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef99bb2ba0 0x55ef99bb7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e8fe66740 1 --2- 172.21.15.98:0/1560112110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ef99bb2280 0x55ef99bb7550 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e6aff5640 0 client.18351 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e8fe66740 1 -- 172.21.15.98:0/1560112110 >> 172.21.15.98:0/1560112110 conn(0x55ef99bae5a0 msgr2=0x55ef99bacf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e8fe66740 1 -- 172.21.15.98:0/1560112110 shutdown_connections 2024-05-09T09:00:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.879+0000 7f3e8fe66740 1 -- 172.21.15.98:0/1560112110 wait complete. 2024-05-09T09:00:21.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.880+0000 7f3e8fe66740 20 client.18351 trim_cache size 0 max 16384 2024-05-09T09:00:21.886 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:00:21.886 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T09:00:21.886 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T09:00:21.886 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:00:21.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.880+0000 7f3e8d15a640 1 -- 172.21.15.98:0/1996667401 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e700a2d00 msgr2=0x7f3e700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.880+0000 7f3e8d15a640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e700a2d00 0x7f3e700a51c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3e80097aa0 tx=0x7f3e800660f0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.880+0000 7f3e8d15a640 1 -- 172.21.15.98:0/1996667401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e88107810 msgr2=0x7f3e8819ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.880+0000 7f3e8d15a640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e88107810 0x7f3e8819ca90 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009fd0 tx=0x7f3e8000b8e0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.880+0000 7f3e8d15a640 1 -- 172.21.15.98:0/1996667401 shutdown_connections 2024-05-09T09:00:21.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.880+0000 7f3e8d15a640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3e700a2d00 0x7f3e700a51c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.880+0000 7f3e8d15a640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e88107810 0x7f3e8819ca90 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.880+0000 7f3e8d15a640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e881021c0 0x7f3e88198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.880+0000 7f3e8d15a640 1 --2- 172.21.15.98:0/1996667401 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e88101800 0x7f3e88198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.880+0000 7f3e8d15a640 1 -- 172.21.15.98:0/1996667401 >> 172.21.15.98:0/1996667401 conn(0x7f3e88078c80 msgr2=0x7f3e880ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:21.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.880+0000 7f3e8d15a640 1 -- 172.21.15.98:0/1996667401 shutdown_connections 2024-05-09T09:00:21.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.880+0000 7f3e8d15a640 1 -- 172.21.15.98:0/1996667401 wait complete. 2024-05-09T09:00:21.890 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T09:00:21.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:00:21.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.993+0000 7f9541af5640 1 -- 172.21.15.98:0/1930857734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c107810 msgr2=0x7f953c107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.993+0000 7f9541af5640 1 --2- 172.21.15.98:0/1930857734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c107810 0x7f953c107bf0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f952c009930 tx=0x7f952c02a420 comp rx=0 tx=0).stop 2024-05-09T09:00:21.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.993+0000 7f9541af5640 1 -- 172.21.15.98:0/1930857734 shutdown_connections 2024-05-09T09:00:21.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.993+0000 7f9541af5640 1 --2- 172.21.15.98:0/1930857734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f953c1021c0 0x7f953c10e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.993+0000 7f9541af5640 1 --2- 172.21.15.98:0/1930857734 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f953c101800 0x7f953c101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.993+0000 7f9541af5640 1 --2- 172.21.15.98:0/1930857734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c107810 0x7f953c107bf0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.993+0000 7f9541af5640 1 -- 172.21.15.98:0/1930857734 >> 172.21.15.98:0/1930857734 conn(0x7f953c078bf0 msgr2=0x7f953c079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.993+0000 7f9541af5640 1 -- 172.21.15.98:0/1930857734 shutdown_connections 2024-05-09T09:00:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.993+0000 7f9541af5640 1 -- 172.21.15.98:0/1930857734 wait complete. 2024-05-09T09:00:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.993+0000 7f9541af5640 1 Processor -- start 2024-05-09T09:00:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f9541af5640 1 -- start start 2024-05-09T09:00:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f9541af5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f953c101800 0x7f953c19a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f9541af5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f953c1021c0 0x7f953c19a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f9541af5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c107810 0x7f953c19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f9541af5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f953c111be0 con 0x7f953c107810 2024-05-09T09:00:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f9541af5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f953c111a60 con 0x7f953c101800 2024-05-09T09:00:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f9541af5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f953c111d60 con 0x7f953c1021c0 2024-05-09T09:00:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f953bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c107810 0x7f953c19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f953bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c107810 0x7f953c19ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35034/0 (socket says 172.21.15.98:35034) 2024-05-09T09:00:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f953bfff640 1 -- 172.21.15.98:0/2029427446 learned_addr learned my addr 172.21.15.98:0/2029427446 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f953affd640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f953c1021c0 0x7f953c19a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f953b7fe640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f953c101800 0x7f953c19a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f953bfff640 1 -- 172.21.15.98:0/2029427446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f953c1021c0 msgr2=0x7f953c19a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f953bfff640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f953c1021c0 0x7f953c19a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f953bfff640 1 -- 172.21.15.98:0/2029427446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f953c101800 msgr2=0x7f953c19a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f953bfff640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f953c101800 0x7f953c19a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f953bfff640 1 -- 172.21.15.98:0/2029427446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f953c19f320 con 0x7f953c107810 2024-05-09T09:00:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f953affd640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f953c1021c0 0x7f953c19a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f953b7fe640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f953c101800 0x7f953c19a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.994+0000 7f953bfff640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c107810 0x7f953c19ebf0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f953400cad0 tx=0x7f953400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.995+0000 7f9538ff9640 1 -- 172.21.15.98:0/2029427446 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953400ae20 con 0x7f953c107810 2024-05-09T09:00:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.995+0000 7f9538ff9640 1 -- 172.21.15.98:0/2029427446 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9534005020 con 0x7f953c107810 2024-05-09T09:00:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.995+0000 7f9541af5640 1 -- 172.21.15.98:0/2029427446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f953c19f610 con 0x7f953c107810 2024-05-09T09:00:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.995+0000 7f9538ff9640 1 -- 172.21.15.98:0/2029427446 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953400f6f0 con 0x7f953c107810 2024-05-09T09:00:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.995+0000 7f9541af5640 1 -- 172.21.15.98:0/2029427446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f953c1a6ea0 con 0x7f953c107810 2024-05-09T09:00:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.996+0000 7f9541af5640 1 -- 172.21.15.98:0/2029427446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f953c068aa0 con 0x7f953c107810 2024-05-09T09:00:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.997+0000 7f9538ff9640 1 -- 172.21.15.98:0/2029427446 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f95340051d0 con 0x7f953c107810 2024-05-09T09:00:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.997+0000 7f9538ff9640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f95240a2d00 0x7f95240a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.998+0000 7f953b7fe640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f95240a2d00 0x7f95240a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.998+0000 7f9538ff9640 1 -- 172.21.15.98:0/2029427446 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f95340cfcc0 con 0x7f953c107810 2024-05-09T09:00:22.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:21.998+0000 7f953b7fe640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f95240a2d00 0x7f95240a51c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f952c000c00 tx=0x7f952c032c70 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.001+0000 7f9538ff9640 1 -- 172.21.15.98:0/2029427446 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9534014030 con 0x7f953c107810 2024-05-09T09:00:22.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.219+0000 7f9541af5640 1 -- 172.21.15.98:0/2029427446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f953c1a7140 con 0x7f953c107810 2024-05-09T09:00:22.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.220+0000 7f9538ff9640 1 -- 172.21.15.98:0/2029427446 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11941 (secure 0 0 0) 0x7f953409c4d0 con 0x7f953c107810 2024-05-09T09:00:22.220 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T09:00:22.220 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":26581,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/561027115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":561027115},{"type":"v1","addr":"172.21.15.105:6835","nonce":561027115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:56:52.891273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16641":{"gid":16641,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T09:00:22.220 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 37 2024-05-09T09:00:22.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.222+0000 7f9541af5640 1 -- 172.21.15.98:0/2029427446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f95240a2d00 msgr2=0x7f95240a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.222+0000 7f9541af5640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f95240a2d00 0x7f95240a51c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f952c000c00 tx=0x7f952c032c70 comp rx=0 tx=0).stop 2024-05-09T09:00:22.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.222+0000 7f9541af5640 1 -- 172.21.15.98:0/2029427446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c107810 msgr2=0x7f953c19ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.222+0000 7f9541af5640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c107810 0x7f953c19ebf0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f953400cad0 tx=0x7f953400cf90 comp rx=0 tx=0).stop 2024-05-09T09:00:22.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.222+0000 7f9541af5640 1 -- 172.21.15.98:0/2029427446 shutdown_connections 2024-05-09T09:00:22.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.222+0000 7f9541af5640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f95240a2d00 0x7f95240a51c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.222+0000 7f9541af5640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c107810 0x7f953c19ebf0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.222+0000 7f9541af5640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f953c1021c0 0x7f953c19a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.222+0000 7f9541af5640 1 --2- 172.21.15.98:0/2029427446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f953c101800 0x7f953c19a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.222+0000 7f9541af5640 1 -- 172.21.15.98:0/2029427446 >> 172.21.15.98:0/2029427446 conn(0x7f953c078bf0 msgr2=0x7f953c10cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.223+0000 7f9541af5640 1 -- 172.21.15.98:0/2029427446 shutdown_connections 2024-05-09T09:00:22.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.223+0000 7f9541af5640 1 -- 172.21.15.98:0/2029427446 wait complete. 2024-05-09T09:00:22.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T09:00:22.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.339+0000 7fa89809a640 1 -- 172.21.15.98:0/1775280349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8901033d0 msgr2=0x7fa89010b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.339+0000 7fa89809a640 1 --2- 172.21.15.98:0/1775280349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8901033d0 0x7fa89010b8b0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fa880009900 tx=0x7fa88002a510 comp rx=0 tx=0).stop 2024-05-09T09:00:22.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.339+0000 7fa89809a640 1 -- 172.21.15.98:0/1775280349 shutdown_connections 2024-05-09T09:00:22.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.339+0000 7fa89809a640 1 --2- 172.21.15.98:0/1775280349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa89010bdf0 0x7fa89010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.339+0000 7fa89809a640 1 --2- 172.21.15.98:0/1775280349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8901033d0 0x7fa89010b8b0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.339+0000 7fa89809a640 1 --2- 172.21.15.98:0/1775280349 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa890102ab0 0x7fa890102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.339+0000 7fa89809a640 1 -- 172.21.15.98:0/1775280349 >> 172.21.15.98:0/1775280349 conn(0x7fa890077830 msgr2=0x7fa890077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.339+0000 7fa89809a640 1 -- 172.21.15.98:0/1775280349 shutdown_connections 2024-05-09T09:00:22.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.339+0000 7fa89809a640 1 -- 172.21.15.98:0/1775280349 wait complete. 2024-05-09T09:00:22.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa89809a640 1 Processor -- start 2024-05-09T09:00:22.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa89809a640 1 -- start start 2024-05-09T09:00:22.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa89809a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa890102ab0 0x7fa890198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa89809a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8901033d0 0x7fa8901986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa89809a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa89010bdf0 0x7fa89019c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa89809a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa890111c30 con 0x7fa8901033d0 2024-05-09T09:00:22.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa89809a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa890111ab0 con 0x7fa890102ab0 2024-05-09T09:00:22.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa89809a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa890111db0 con 0x7fa89010bdf0 2024-05-09T09:00:22.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa895635640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8901033d0 0x7fa8901986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa895635640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8901033d0 0x7fa8901986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35062/0 (socket says 172.21.15.98:35062) 2024-05-09T09:00:22.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa895635640 1 -- 172.21.15.98:0/1773478045 learned_addr learned my addr 172.21.15.98:0/1773478045 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa895e36640 1 --2- 172.21.15.98:0/1773478045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa890102ab0 0x7fa890198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa896637640 1 --2- 172.21.15.98:0/1773478045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa89010bdf0 0x7fa89019c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa895635640 1 -- 172.21.15.98:0/1773478045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa89010bdf0 msgr2=0x7fa89019c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa895635640 1 --2- 172.21.15.98:0/1773478045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa89010bdf0 0x7fa89019c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa895635640 1 -- 172.21.15.98:0/1773478045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa890102ab0 msgr2=0x7fa890198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa895635640 1 --2- 172.21.15.98:0/1773478045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa890102ab0 0x7fa890198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.340+0000 7fa895635640 1 -- 172.21.15.98:0/1773478045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa89019d110 con 0x7fa8901033d0 2024-05-09T09:00:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.341+0000 7fa895635640 1 --2- 172.21.15.98:0/1773478045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8901033d0 0x7fa8901986b0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fa880005ab0 tx=0x7fa8800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.341+0000 7fa886ffd640 1 -- 172.21.15.98:0/1773478045 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88003f030 con 0x7fa8901033d0 2024-05-09T09:00:22.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.341+0000 7fa886ffd640 1 -- 172.21.15.98:0/1773478045 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa880003ce0 con 0x7fa8901033d0 2024-05-09T09:00:22.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.341+0000 7fa89809a640 1 -- 172.21.15.98:0/1773478045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa89019d3a0 con 0x7fa8901033d0 2024-05-09T09:00:22.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.341+0000 7fa886ffd640 1 -- 172.21.15.98:0/1773478045 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880035070 con 0x7fa8901033d0 2024-05-09T09:00:22.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.341+0000 7fa89809a640 1 -- 172.21.15.98:0/1773478045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8901a4c90 con 0x7fa8901033d0 2024-05-09T09:00:22.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.343+0000 7fa886ffd640 1 -- 172.21.15.98:0/1773478045 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa880042030 con 0x7fa8901033d0 2024-05-09T09:00:22.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.343+0000 7fa89ada4740 1 Processor -- start 2024-05-09T09:00:22.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa89ada4740 1 -- start start 2024-05-09T09:00:22.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa89ada4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767f210 0x559e6767f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa89ada4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6767fb30 0x559e6768e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa89ada4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6768e730 0x559e67690b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa89ada4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559e675e4c40 con 0x559e6767f210 2024-05-09T09:00:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa89ada4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559e6758c580 con 0x559e6768e730 2024-05-09T09:00:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa89ada4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559e675e0c00 con 0x559e6767fb30 2024-05-09T09:00:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa886ffd640 1 --2- 172.21.15.98:0/1773478045 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa87c0a2d00 0x7fa87c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa896637640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767f210 0x559e6767f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa895e36640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6767fb30 0x559e6768e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa896637640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767f210 0x559e6767f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35066/0 (socket says 172.21.15.98:35066) 2024-05-09T09:00:22.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa895e36640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6767fb30 0x559e6768e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:49972/0 (socket says 172.21.15.98:49972) 2024-05-09T09:00:22.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa896637640 1 -- 172.21.15.98:0/1851247746 learned_addr learned my addr 172.21.15.98:0/1851247746 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa896637640 1 --2- 172.21.15.98:0/1773478045 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa87c0a2d00 0x7fa87c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa886ffd640 1 -- 172.21.15.98:0/1773478045 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7fa8800efe10 con 0x7fa8901033d0 2024-05-09T09:00:22.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa896637640 1 -- 172.21.15.98:0/1851247746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6767fb30 msgr2=0x559e6768e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa896637640 1 --2- 172.21.15.98:0/1851247746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6767fb30 0x559e6768e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa895635640 1 --2- 172.21.15.98:0/1851247746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6768e730 0x559e67690b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa896637640 1 -- 172.21.15.98:0/1851247746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6768e730 msgr2=0x559e67690b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa896637640 1 --2- 172.21.15.98:0/1851247746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6768e730 0x559e67690b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa896637640 1 -- 172.21.15.98:0/1851247746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e672f83f0 con 0x559e6767f210 2024-05-09T09:00:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa895e36640 1 --2- 172.21.15.98:0/1851247746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6767fb30 0x559e6768e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:00:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.344+0000 7fa895635640 1 --2- 172.21.15.98:0/1851247746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6768e730 0x559e67690b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.345+0000 7fa896637640 1 --2- 172.21.15.98:0/1851247746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767f210 0x559e6767f5f0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fa888013e60 tx=0x7fa8880107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=470d71226ac8aaf4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.345+0000 7fa896637640 1 --2- 172.21.15.98:0/1773478045 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa87c0a2d00 0x7fa87c0a51c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa888010ca0 tx=0x7fa888015000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.345+0000 7fa87bfff640 1 -- 172.21.15.98:0/1851247746 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88800ae10 con 0x559e6767f210 2024-05-09T09:00:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.345+0000 7fa87bfff640 1 -- 172.21.15.98:0/1851247746 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa888007be0 con 0x559e6767f210 2024-05-09T09:00:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.345+0000 7fa87bfff640 1 -- 172.21.15.98:0/1851247746 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa888005900 con 0x559e6767f210 2024-05-09T09:00:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.346+0000 7fa89ada4740 1 -- 172.21.15.98:0/1851247746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767f210 msgr2=0x559e6767f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.346+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1851247746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767f210 0x559e6767f5f0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fa888013e60 tx=0x7fa8880107a0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.346+0000 7fa89ada4740 1 -- 172.21.15.98:0/1851247746 shutdown_connections 2024-05-09T09:00:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.346+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1851247746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6768e730 0x559e67690b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.346+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1851247746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6767fb30 0x559e6768e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.346+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1851247746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767f210 0x559e6767f5f0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.346+0000 7fa89ada4740 1 -- 172.21.15.98:0/1851247746 >> 172.21.15.98:0/1851247746 conn(0x559e6767b650 msgr2=0x559e6767ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.346+0000 7fa89ada4740 1 -- 172.21.15.98:0/1851247746 shutdown_connections 2024-05-09T09:00:22.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.346+0000 7fa89ada4740 1 -- 172.21.15.98:0/1851247746 wait complete. 2024-05-09T09:00:22.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.346+0000 7fa89ada4740 1 -- start start 2024-05-09T09:00:22.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.346+0000 7fa89ada4740 1 Processor -- start 2024-05-09T09:00:22.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa89ada4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767f210 0x559e6768b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa89ada4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6767fb30 0x559e6768bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa89ada4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6768e730 0x559e6768c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa89ada4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559e67691510 con 0x559e6767f210 2024-05-09T09:00:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa89ada4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559e6759fc40 con 0x559e6767fb30 2024-05-09T09:00:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa89ada4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559e67691690 con 0x559e6768e730 2024-05-09T09:00:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa89ada4740 10 client.0 _ll_register_callbacks cb 0x559e672f7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa895e36640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767f210 0x559e6768b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa895e36640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767f210 0x559e6768b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35080/0 (socket says 172.21.15.98:35080) 2024-05-09T09:00:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa895e36640 1 -- 172.21.15.98:0/1470009 learned_addr learned my addr 172.21.15.98:0/1470009 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa895635640 1 --2- 172.21.15.98:0/1470009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6767fb30 0x559e6768bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa896637640 1 --2- 172.21.15.98:0/1470009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6768e730 0x559e6768c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa895e36640 1 -- 172.21.15.98:0/1470009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6768e730 msgr2=0x559e6768c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa895e36640 1 --2- 172.21.15.98:0/1470009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6768e730 0x559e6768c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa895e36640 1 -- 172.21.15.98:0/1470009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6767fb30 msgr2=0x559e6768bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa895e36640 1 --2- 172.21.15.98:0/1470009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6767fb30 0x559e6768bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa895e36640 1 -- 172.21.15.98:0/1470009 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e673c27d0 con 0x559e6767f210 2024-05-09T09:00:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa895635640 1 --2- 172.21.15.98:0/1470009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6767fb30 0x559e6768bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa895e36640 1 --2- 172.21.15.98:0/1470009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767f210 0x559e6768b700 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fa88c011610 tx=0x7fa88c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da09bead836756cb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa8797fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa8797fa640 1 -- 172.21.15.98:0/1470009 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88c00f750 con 0x559e6767f210 2024-05-09T09:00:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa8797fa640 1 -- 172.21.15.98:0/1470009 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7fa88c00f8f0 con 0x559e6767f210 2024-05-09T09:00:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.347+0000 7fa89ada4740 1 -- 172.21.15.98:0/1470009 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa888007be0 con 0x559e6767f210 2024-05-09T09:00:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.348+0000 7fa8797fa640 1 -- 172.21.15.98:0/1470009 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa88c026050 con 0x559e6767f210 2024-05-09T09:00:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.348+0000 7fa89ada4740 10 client.18369 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.348+0000 7fa89ada4740 1 -- 172.21.15.98:0/1470009 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559e6769d790 con 0x559e6767f210 2024-05-09T09:00:22.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.348+0000 7fa8797fa640 1 -- 172.21.15.98:0/1470009 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa888007be0 con 0x559e6767f210 2024-05-09T09:00:22.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.348+0000 7fa8797fa640 1 -- 172.21.15.98:0/1470009 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa88c01d400 con 0x559e6767f210 2024-05-09T09:00:22.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.348+0000 7fa89ada4740 10 client.18369 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:22.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.348+0000 7fa89ada4740 10 client.18369 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T09:00:22.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.348+0000 7fa89ada4740 20 client.18369 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:22.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.348+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1470009 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559e6769ec50 0x559e676a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.348+0000 7fa89ada4740 4 client.18369 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:22.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.348+0000 7fa89ada4740 1 -- 172.21.15.98:0/1470009 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e675f2370 con 0x559e6769ec50 2024-05-09T09:00:22.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.348+0000 7fa895635640 1 --2- 172.21.15.98:0/1470009 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559e6769ec50 0x559e676a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.349+0000 7fa8797fa640 10 client.18369 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T09:00:22.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.349+0000 7fa895635640 1 --2- 172.21.15.98:0/1470009 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559e6769ec50 0x559e676a10c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ccc5d5806d73db2e server_cookie=7154beb4a0173a2b in_seq=0 out_seq=0 2024-05-09T09:00:22.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.349+0000 7fa8797fa640 1 -- 172.21.15.98:0/1470009 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559e675f2370 con 0x559e6769ec50 2024-05-09T09:00:22.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.349+0000 7fa8797fa640 10 client.18369 handle_command_reply: tid=0 2024-05-09T09:00:22.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.349+0000 7fa89ada4740 1 client.18369 shutdown 2024-05-09T09:00:22.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.350+0000 7fa89ada4740 1 -- 172.21.15.98:0/1470009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767f210 msgr2=0x559e6768b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.350+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1470009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767f210 0x559e6768b700 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fa88c011610 tx=0x7fa88c011ad0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.350+0000 7fa89ada4740 1 -- 172.21.15.98:0/1470009 shutdown_connections 2024-05-09T09:00:22.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.350+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1470009 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559e6769ec50 0x559e676a10c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.350+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1470009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6768e730 0x559e6768c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.350+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1470009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6767fb30 0x559e6768bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.350+0000 7fa8797fa640 0 client.18369 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T09:00:22.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.350+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1470009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767f210 0x559e6768b700 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.350+0000 7fa89ada4740 1 -- 172.21.15.98:0/1470009 >> 172.21.15.98:0/1470009 conn(0x559e6767b650 msgr2=0x559e67681400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.350+0000 7fa89ada4740 1 -- 172.21.15.98:0/1470009 shutdown_connections 2024-05-09T09:00:22.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.350+0000 7fa89ada4740 1 -- 172.21.15.98:0/1470009 wait complete. 2024-05-09T09:00:22.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.350+0000 7fa89ada4740 20 client.18369 trim_cache size 0 max 16384 2024-05-09T09:00:22.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.363+0000 7fa89ada4740 1 Processor -- start 2024-05-09T09:00:22.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.363+0000 7fa89ada4740 1 -- start start 2024-05-09T09:00:22.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.363+0000 7fa89ada4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6767f210 0x559e6768de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.363+0000 7fa89ada4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767fb30 0x559e6769a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.363+0000 7fa89ada4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6768e730 0x559e6767a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.363+0000 7fa89ada4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8901acbc0 con 0x559e6767fb30 2024-05-09T09:00:22.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.363+0000 7fa89ada4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa8901ae700 con 0x559e6767f210 2024-05-09T09:00:22.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.363+0000 7fa89ada4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa8901ad290 con 0x559e6768e730 2024-05-09T09:00:22.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.364+0000 7fa895635640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767fb30 0x559e6769a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.364+0000 7fa895635640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767fb30 0x559e6769a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35094/0 (socket says 172.21.15.98:35094) 2024-05-09T09:00:22.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.364+0000 7fa895635640 1 -- 172.21.15.98:0/194224154 learned_addr learned my addr 172.21.15.98:0/194224154 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.364+0000 7fa895635640 1 -- 172.21.15.98:0/194224154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6768e730 msgr2=0x559e6767a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.364+0000 7fa895635640 1 --2- 172.21.15.98:0/194224154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6768e730 0x559e6767a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.364+0000 7fa895635640 1 -- 172.21.15.98:0/194224154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6767f210 msgr2=0x559e6768de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.364+0000 7fa895e36640 1 --2- 172.21.15.98:0/194224154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6767f210 0x559e6768de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.364+0000 7fa895635640 1 --2- 172.21.15.98:0/194224154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6767f210 0x559e6768de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.364+0000 7fa895635640 1 -- 172.21.15.98:0/194224154 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e67690b20 con 0x559e6767fb30 2024-05-09T09:00:22.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.364+0000 7fa895e36640 1 --2- 172.21.15.98:0/194224154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6767f210 0x559e6768de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:22.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.364+0000 7fa895635640 1 --2- 172.21.15.98:0/194224154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767fb30 0x559e6769a440 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fa880000c00 tx=0x7fa8800dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa59bd40c9cccab9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.364+0000 7fa87b7fe640 1 -- 172.21.15.98:0/194224154 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880049040 con 0x559e6767fb30 2024-05-09T09:00:22.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.364+0000 7fa87b7fe640 1 -- 172.21.15.98:0/194224154 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8800bbf20 con 0x559e6767fb30 2024-05-09T09:00:22.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.364+0000 7fa87b7fe640 1 -- 172.21.15.98:0/194224154 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8800ed3d0 con 0x559e6767fb30 2024-05-09T09:00:22.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.365+0000 7fa89ada4740 1 -- 172.21.15.98:0/194224154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767fb30 msgr2=0x559e6769a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.365+0000 7fa89ada4740 1 --2- 172.21.15.98:0/194224154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767fb30 0x559e6769a440 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fa880000c00 tx=0x7fa8800dc040 comp rx=0 tx=0).stop 2024-05-09T09:00:22.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.365+0000 7fa89ada4740 1 -- 172.21.15.98:0/194224154 shutdown_connections 2024-05-09T09:00:22.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.365+0000 7fa89ada4740 1 --2- 172.21.15.98:0/194224154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6768e730 0x559e6767a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.365+0000 7fa89ada4740 1 --2- 172.21.15.98:0/194224154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767fb30 0x559e6769a440 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.365+0000 7fa89ada4740 1 --2- 172.21.15.98:0/194224154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6767f210 0x559e6768de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.365+0000 7fa89ada4740 1 -- 172.21.15.98:0/194224154 >> 172.21.15.98:0/194224154 conn(0x559e6767b650 msgr2=0x559e67679f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.365+0000 7fa89ada4740 1 -- 172.21.15.98:0/194224154 shutdown_connections 2024-05-09T09:00:22.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.365+0000 7fa89ada4740 1 -- 172.21.15.98:0/194224154 wait complete. 2024-05-09T09:00:22.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.365+0000 7fa89ada4740 1 -- start start 2024-05-09T09:00:22.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.365+0000 7fa89ada4740 1 Processor -- start 2024-05-09T09:00:22.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa89ada4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6767f210 0x559e67692b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa89ada4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767fb30 0x559e67693050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa89ada4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6768e730 0x559e67704200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa89ada4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8901ae030 con 0x559e6767fb30 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa89ada4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa8901ad960 con 0x559e6768e730 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa89ada4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa8901aabc0 con 0x559e6767f210 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa89ada4740 10 client.0 _ll_register_callbacks cb 0x7fa88c0095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa895635640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767fb30 0x559e67693050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa895635640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767fb30 0x559e67693050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35096/0 (socket says 172.21.15.98:35096) 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa895635640 1 -- 172.21.15.98:0/1419013936 learned_addr learned my addr 172.21.15.98:0/1419013936 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa896637640 1 --2- 172.21.15.98:0/1419013936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6768e730 0x559e67704200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa895635640 1 -- 172.21.15.98:0/1419013936 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6767f210 msgr2=0x559e67692b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa895635640 1 --2- 172.21.15.98:0/1419013936 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6767f210 0x559e67692b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa895635640 1 -- 172.21.15.98:0/1419013936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6768e730 msgr2=0x559e67704200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa895635640 1 --2- 172.21.15.98:0/1419013936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6768e730 0x559e67704200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa895635640 1 -- 172.21.15.98:0/1419013936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e677047c0 con 0x559e6767fb30 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa896637640 1 --2- 172.21.15.98:0/1419013936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6768e730 0x559e67704200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa895635640 1 --2- 172.21.15.98:0/1419013936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767fb30 0x559e67693050 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fa8800ed750 tx=0x7fa88002a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc1f197d8b0fe1d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa8797fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa8797fa640 1 -- 172.21.15.98:0/1419013936 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880049070 con 0x559e6767fb30 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa89ada4740 1 -- 172.21.15.98:0/1419013936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8800bbf20 con 0x559e6767fb30 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.366+0000 7fa8797fa640 1 -- 172.21.15.98:0/1419013936 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7fa8800ed3d0 con 0x559e6767fb30 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.367+0000 7fa8797fa640 1 -- 172.21.15.98:0/1419013936 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa8800c4b90 con 0x559e6767fb30 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.367+0000 7fa89ada4740 10 client.18375 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.367+0000 7fa89ada4740 1 -- 172.21.15.98:0/1419013936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559e67705a10 con 0x559e6767fb30 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.367+0000 7fa8797fa640 1 -- 172.21.15.98:0/1419013936 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8800bbf20 con 0x559e6767fb30 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.367+0000 7fa8797fa640 1 -- 172.21.15.98:0/1419013936 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa8800bf070 con 0x559e6767fb30 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.367+0000 7fa89ada4740 10 client.18375 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.367+0000 7fa89ada4740 10 client.18375 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.367+0000 7fa89ada4740 20 client.18375 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.367+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1419013936 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559e677060f0 0x559e67708510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.367+0000 7fa89ada4740 4 client.18375 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.367+0000 7fa89ada4740 1 -- 172.21.15.98:0/1419013936 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559e675ec6c0 con 0x559e677060f0 2024-05-09T09:00:22.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.367+0000 7fa895e36640 1 --2- 172.21.15.98:0/1419013936 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559e677060f0 0x559e67708510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa8797fa640 10 client.18375 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T09:00:22.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa895e36640 1 --2- 172.21.15.98:0/1419013936 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559e677060f0 0x559e67708510 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=84bc42aeb48ce5a3 server_cookie=4b6bcc58f6bbf0b0 in_seq=0 out_seq=0 2024-05-09T09:00:22.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa8797fa640 1 -- 172.21.15.98:0/1419013936 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559e675ec6c0 con 0x559e677060f0 2024-05-09T09:00:22.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa8797fa640 10 client.18375 handle_command_reply: tid=0 2024-05-09T09:00:22.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa89ada4740 1 client.18375 shutdown 2024-05-09T09:00:22.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa89ada4740 1 -- 172.21.15.98:0/1419013936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767fb30 msgr2=0x559e67693050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1419013936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767fb30 0x559e67693050 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fa8800ed750 tx=0x7fa88002a510 comp rx=0 tx=0).stop 2024-05-09T09:00:22.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa89ada4740 1 -- 172.21.15.98:0/1419013936 shutdown_connections 2024-05-09T09:00:22.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1419013936 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559e677060f0 0x559e67708510 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1419013936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559e6768e730 0x559e67704200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa8797fa640 0 client.18375 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T09:00:22.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1419013936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e6767fb30 0x559e67693050 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa89ada4740 1 --2- 172.21.15.98:0/1419013936 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559e6767f210 0x559e67692b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa89ada4740 1 -- 172.21.15.98:0/1419013936 >> 172.21.15.98:0/1419013936 conn(0x559e6767b650 msgr2=0x559e6767a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa89ada4740 1 -- 172.21.15.98:0/1419013936 shutdown_connections 2024-05-09T09:00:22.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa89ada4740 1 -- 172.21.15.98:0/1419013936 wait complete. 2024-05-09T09:00:22.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.368+0000 7fa89ada4740 20 client.18375 trim_cache size 0 max 16384 2024-05-09T09:00:22.391 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:00:22.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.369+0000 7fa89809a640 1 -- 172.21.15.98:0/1773478045 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa87c0a2d00 msgr2=0x7fa87c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.369+0000 7fa89809a640 1 --2- 172.21.15.98:0/1773478045 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa87c0a2d00 0x7fa87c0a51c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa888010ca0 tx=0x7fa888015000 comp rx=0 tx=0).stop 2024-05-09T09:00:22.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.369+0000 7fa89809a640 1 -- 172.21.15.98:0/1773478045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8901033d0 msgr2=0x7fa8901986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.369+0000 7fa89809a640 1 --2- 172.21.15.98:0/1773478045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8901033d0 0x7fa8901986b0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fa880005ab0 tx=0x7fa8800042f0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.369+0000 7fa89809a640 1 -- 172.21.15.98:0/1773478045 shutdown_connections 2024-05-09T09:00:22.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.369+0000 7fa89809a640 1 --2- 172.21.15.98:0/1773478045 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fa87c0a2d00 0x7fa87c0a51c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.369+0000 7fa89809a640 1 --2- 172.21.15.98:0/1773478045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa89010bdf0 0x7fa89019c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.369+0000 7fa89809a640 1 --2- 172.21.15.98:0/1773478045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8901033d0 0x7fa8901986b0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.369+0000 7fa89809a640 1 --2- 172.21.15.98:0/1773478045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa890102ab0 0x7fa890198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.369+0000 7fa89809a640 1 -- 172.21.15.98:0/1773478045 >> 172.21.15.98:0/1773478045 conn(0x7fa890077830 msgr2=0x7fa8900fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.369+0000 7fa89809a640 1 -- 172.21.15.98:0/1773478045 shutdown_connections 2024-05-09T09:00:22.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.369+0000 7fa89809a640 1 -- 172.21.15.98:0/1773478045 wait complete. 2024-05-09T09:00:22.393 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:22 smithi098 ceph-mon[30092]: pgmap v304: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-05-09T09:00:22.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T09:00:22.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:22 smithi105 ceph-mon[34498]: pgmap v304: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-05-09T09:00:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.502+0000 7ff0e9109640 1 -- 172.21.15.98:0/1030562680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0e41033d0 msgr2=0x7ff0e410b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.502+0000 7ff0e9109640 1 --2- 172.21.15.98:0/1030562680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0e41033d0 0x7ff0e410b8b0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009900 tx=0x7ff0cc02a510 comp rx=0 tx=0).stop 2024-05-09T09:00:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.502+0000 7ff0e9109640 1 -- 172.21.15.98:0/1030562680 shutdown_connections 2024-05-09T09:00:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.502+0000 7ff0e9109640 1 --2- 172.21.15.98:0/1030562680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0e410bdf0 0x7ff0e410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.502+0000 7ff0e9109640 1 --2- 172.21.15.98:0/1030562680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0e41033d0 0x7ff0e410b8b0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.502+0000 7ff0e9109640 1 --2- 172.21.15.98:0/1030562680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0e4102ab0 0x7ff0e4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.502+0000 7ff0e9109640 1 -- 172.21.15.98:0/1030562680 >> 172.21.15.98:0/1030562680 conn(0x7ff0e4077830 msgr2=0x7ff0e4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.502+0000 7ff0e9109640 1 -- 172.21.15.98:0/1030562680 shutdown_connections 2024-05-09T09:00:22.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.502+0000 7ff0e9109640 1 -- 172.21.15.98:0/1030562680 wait complete. 2024-05-09T09:00:22.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.503+0000 7ff0e9109640 1 Processor -- start 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.503+0000 7ff0e9109640 1 -- start start 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.503+0000 7ff0e9109640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0e4102ab0 0x7ff0e41980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.503+0000 7ff0e9109640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0e41033d0 0x7ff0e41985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.503+0000 7ff0e9109640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0e410bdf0 0x7ff0e419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.503+0000 7ff0e9109640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e4111bf0 con 0x7ff0e41033d0 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.503+0000 7ff0e9109640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e4111a70 con 0x7ff0e4102ab0 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.503+0000 7ff0e9109640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e4111d70 con 0x7ff0e410bdf0 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.503+0000 7ff0dbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0e41033d0 0x7ff0e41985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.504+0000 7ff0dbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0e41033d0 0x7ff0e41985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35118/0 (socket says 172.21.15.98:35118) 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.504+0000 7ff0dbfff640 1 -- 172.21.15.98:0/708551477 learned_addr learned my addr 172.21.15.98:0/708551477 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.504+0000 7ff0dbfff640 1 -- 172.21.15.98:0/708551477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0e410bdf0 msgr2=0x7ff0e419c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.504+0000 7ff0dbfff640 1 --2- 172.21.15.98:0/708551477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0e410bdf0 0x7ff0e419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.504+0000 7ff0dbfff640 1 -- 172.21.15.98:0/708551477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0e4102ab0 msgr2=0x7ff0e41980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.504+0000 7ff0dbfff640 1 --2- 172.21.15.98:0/708551477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0e4102ab0 0x7ff0e41980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.504+0000 7ff0dbfff640 1 -- 172.21.15.98:0/708551477 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e419d0a0 con 0x7ff0e41033d0 2024-05-09T09:00:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.504+0000 7ff0dbfff640 1 --2- 172.21.15.98:0/708551477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0e41033d0 0x7ff0e41985f0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc006010 tx=0x7ff0cc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.504+0000 7ff0e0d99640 1 -- 172.21.15.98:0/708551477 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc03f030 con 0x7ff0e41033d0 2024-05-09T09:00:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.504+0000 7ff0e9109640 1 -- 172.21.15.98:0/708551477 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0e419d330 con 0x7ff0e41033d0 2024-05-09T09:00:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.504+0000 7ff0e0d99640 1 -- 172.21.15.98:0/708551477 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0cc003ce0 con 0x7ff0e41033d0 2024-05-09T09:00:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.504+0000 7ff0e0d99640 1 -- 172.21.15.98:0/708551477 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc035070 con 0x7ff0e41033d0 2024-05-09T09:00:22.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.504+0000 7ff0e9109640 1 -- 172.21.15.98:0/708551477 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0e41a4c20 con 0x7ff0e41033d0 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.506+0000 7ff0e0d99640 1 -- 172.21.15.98:0/708551477 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff0cc042030 con 0x7ff0e41033d0 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.507+0000 7ff0ebe15740 1 Processor -- start 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.507+0000 7ff0ebe15740 1 -- start start 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.507+0000 7ff0ebe15740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873ea0330 0x558873ea0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.507+0000 7ff0ebe15740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 0x558873eaade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.507+0000 7ff0ebe15740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873eab320 0x558873ead710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.507+0000 7ff0ebe15740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558873e05c40 con 0x558873eab320 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.507+0000 7ff0ebe15740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558873dad580 con 0x558873ea0c50 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.507+0000 7ff0ebe15740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558873e01c00 con 0x558873ea0330 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.507+0000 7ff0dbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873eab320 0x558873ead710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.507+0000 7ff0dbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873eab320 0x558873ead710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35134/0 (socket says 172.21.15.98:35134) 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.507+0000 7ff0dbfff640 1 -- 172.21.15.98:0/3622615946 learned_addr learned my addr 172.21.15.98:0/3622615946 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.507+0000 7ff0e359e640 1 --2- 172.21.15.98:0/3622615946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873ea0330 0x558873ea0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.507+0000 7ff0e2d9d640 1 --2- 172.21.15.98:0/3622615946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 0x558873eaade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.507+0000 7ff0e0d99640 1 --2- 172.21.15.98:0/708551477 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff0d00a2d00 0x7ff0d00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.508+0000 7ff0e359e640 1 --2- 172.21.15.98:0/708551477 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff0d00a2d00 0x7ff0d00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.508+0000 7ff0e0d99640 1 -- 172.21.15.98:0/708551477 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7ff0cc0efef0 con 0x7ff0e41033d0 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.508+0000 7ff0dbfff640 1 -- 172.21.15.98:0/3622615946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873ea0330 msgr2=0x558873ea0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.508+0000 7ff0dbfff640 1 --2- 172.21.15.98:0/3622615946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873ea0330 0x558873ea0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.508+0000 7ff0dbfff640 1 -- 172.21.15.98:0/3622615946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 msgr2=0x558873eaade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.508+0000 7ff0dbfff640 1 --2- 172.21.15.98:0/3622615946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 0x558873eaade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.508+0000 7ff0dbfff640 1 -- 172.21.15.98:0/3622615946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558873b193f0 con 0x558873eab320 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.508+0000 7ff0e2d9d640 1 --2- 172.21.15.98:0/3622615946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 0x558873eaade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.508+0000 7ff0e359e640 1 --2- 172.21.15.98:0/3622615946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873ea0330 0x558873ea0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.508+0000 7ff0dbfff640 1 --2- 172.21.15.98:0/3622615946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873eab320 0x558873ead710 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0bc050 tx=0x7ff0cc02a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df82d2178d5678b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.508+0000 7ff0d97fa640 1 -- 172.21.15.98:0/3622615946 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc049070 con 0x558873eab320 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.508+0000 7ff0e359e640 1 --2- 172.21.15.98:0/708551477 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff0d00a2d00 0x7ff0d00a51c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc009270 tx=0x7ff0dc009500 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.508+0000 7ff0d97fa640 1 -- 172.21.15.98:0/3622615946 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0cc0dc0e0 con 0x558873eab320 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.508+0000 7ff0d97fa640 1 -- 172.21.15.98:0/3622615946 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc0bc0c0 con 0x558873eab320 2024-05-09T09:00:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.509+0000 7ff0ebe15740 1 -- 172.21.15.98:0/3622615946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873eab320 msgr2=0x558873ead710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.509+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/3622615946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873eab320 0x558873ead710 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0bc050 tx=0x7ff0cc02a510 comp rx=0 tx=0).stop 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.509+0000 7ff0ebe15740 1 -- 172.21.15.98:0/3622615946 shutdown_connections 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.509+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/3622615946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873eab320 0x558873ead710 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.509+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/3622615946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 0x558873eaade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.509+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/3622615946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873ea0330 0x558873ea0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.509+0000 7ff0ebe15740 1 -- 172.21.15.98:0/3622615946 >> 172.21.15.98:0/3622615946 conn(0x558873e9c650 msgr2=0x558873e9ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.509+0000 7ff0ebe15740 1 -- 172.21.15.98:0/3622615946 shutdown_connections 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.509+0000 7ff0ebe15740 1 -- 172.21.15.98:0/3622615946 wait complete. 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.509+0000 7ff0ebe15740 1 -- start start 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.509+0000 7ff0ebe15740 1 Processor -- start 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0ebe15740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0330 0x558873ea82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0ebe15740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873ea0c50 0x558873ea87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0ebe15740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873eab320 0x558873ea8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0ebe15740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558873e05c40 con 0x558873ea0c50 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0ebe15740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558873dc0c40 con 0x558873ea0330 2024-05-09T09:00:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0ebe15740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558873eaded0 con 0x558873eab320 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0ebe15740 10 client.0 _ll_register_callbacks cb 0x558873b18be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0dbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873ea0c50 0x558873ea87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0dbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873ea0c50 0x558873ea87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35142/0 (socket says 172.21.15.98:35142) 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0dbfff640 1 -- 172.21.15.98:0/1716653607 learned_addr learned my addr 172.21.15.98:0/1716653607 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0e2d9d640 1 --2- 172.21.15.98:0/1716653607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0330 0x558873ea82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0e359e640 1 --2- 172.21.15.98:0/1716653607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873eab320 0x558873ea8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0dbfff640 1 -- 172.21.15.98:0/1716653607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873eab320 msgr2=0x558873ea8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0dbfff640 1 --2- 172.21.15.98:0/1716653607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873eab320 0x558873ea8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0dbfff640 1 -- 172.21.15.98:0/1716653607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0330 msgr2=0x558873ea82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0dbfff640 1 --2- 172.21.15.98:0/1716653607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0330 0x558873ea82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0dbfff640 1 -- 172.21.15.98:0/1716653607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558873be37d0 con 0x558873ea0c50 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0e359e640 1 --2- 172.21.15.98:0/1716653607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873eab320 0x558873ea8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0dbfff640 1 --2- 172.21.15.98:0/1716653607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873ea0c50 0x558873ea87f0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0bbdc0 tx=0x7ff0cc0f3670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cebf6ec23906aea3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0ca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/1716653607 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc049040 con 0x558873ea0c50 2024-05-09T09:00:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/1716653607 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7ff0cc0c0120 con 0x558873ea0c50 2024-05-09T09:00:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.510+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1716653607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558873e13370 con 0x558873ea0c50 2024-05-09T09:00:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.511+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/1716653607 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff0cc0bbc00 con 0x558873ea0c50 2024-05-09T09:00:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.511+0000 7ff0ebe15740 10 client.18387 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.511+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1716653607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558873ebe390 con 0x558873ea0c50 2024-05-09T09:00:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.511+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/1716653607 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558873e13370 con 0x558873ea0c50 2024-05-09T09:00:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.511+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/1716653607 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7ff0cc0d8050 con 0x558873ea0c50 2024-05-09T09:00:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.511+0000 7ff0ebe15740 10 client.18387 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.511+0000 7ff0ebe15740 10 client.18387 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T09:00:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.511+0000 7ff0ebe15740 20 client.18387 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.511+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/1716653607 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558873ebf6e0 0x558873ec1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.511+0000 7ff0ebe15740 4 client.18387 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.511+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1716653607 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558873e0d6c0 con 0x558873ebf6e0 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.511+0000 7ff0e2d9d640 1 --2- 172.21.15.98:0/1716653607 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558873ebf6e0 0x558873ec1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.512+0000 7ff0e2d9d640 1 --2- 172.21.15.98:0/1716653607 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558873ebf6e0 0x558873ec1ad0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c6b7d80504b25cc5 server_cookie=52fb274681b3880 in_seq=0 out_seq=0 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.512+0000 7ff0ca7fc640 10 client.18387 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.512+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/1716653607 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558873e0d6c0 con 0x558873ebf6e0 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.512+0000 7ff0ca7fc640 10 client.18387 handle_command_reply: tid=0 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.512+0000 7ff0ebe15740 1 client.18387 shutdown 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.513+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1716653607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873ea0c50 msgr2=0x558873ea87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.513+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/1716653607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873ea0c50 0x558873ea87f0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0bbdc0 tx=0x7ff0cc0f3670 comp rx=0 tx=0).stop 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.513+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1716653607 shutdown_connections 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.513+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/1716653607 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558873ebf6e0 0x558873ec1ad0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.513+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/1716653607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873eab320 0x558873ea8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.513+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/1716653607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873ea0c50 0x558873ea87f0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.513+0000 7ff0ca7fc640 0 client.18387 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.513+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/1716653607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0330 0x558873ea82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.513+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1716653607 >> 172.21.15.98:0/1716653607 conn(0x558873e9c650 msgr2=0x558873ea2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.513+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1716653607 shutdown_connections 2024-05-09T09:00:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.513+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1716653607 wait complete. 2024-05-09T09:00:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.513+0000 7ff0ebe15740 20 client.18387 trim_cache size 0 max 16384 2024-05-09T09:00:22.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.526+0000 7ff0ebe15740 1 Processor -- start 2024-05-09T09:00:22.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.526+0000 7ff0ebe15740 1 -- start start 2024-05-09T09:00:22.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.526+0000 7ff0ebe15740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873ea0330 0x558873ea2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.526+0000 7ff0ebe15740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 0x558873ea8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.526+0000 7ff0ebe15740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873eab320 0x558873eaa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.526+0000 7ff0ebe15740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e41acc20 con 0x558873ea0330 2024-05-09T09:00:22.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.526+0000 7ff0ebe15740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e41ae760 con 0x558873ea0c50 2024-05-09T09:00:22.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.526+0000 7ff0ebe15740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e41ad2f0 con 0x558873eab320 2024-05-09T09:00:22.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.527+0000 7ff0e2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873ea0330 0x558873ea2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.527+0000 7ff0e2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873ea0330 0x558873ea2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35146/0 (socket says 172.21.15.98:35146) 2024-05-09T09:00:22.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.527+0000 7ff0e2d9d640 1 -- 172.21.15.98:0/555012265 learned_addr learned my addr 172.21.15.98:0/555012265 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.527+0000 7ff0dbfff640 1 --2- 172.21.15.98:0/555012265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 0x558873ea8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.527+0000 7ff0e359e640 1 --2- 172.21.15.98:0/555012265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873eab320 0x558873eaa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.527+0000 7ff0e2d9d640 1 -- 172.21.15.98:0/555012265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873eab320 msgr2=0x558873eaa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.527+0000 7ff0e2d9d640 1 --2- 172.21.15.98:0/555012265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873eab320 0x558873eaa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.527+0000 7ff0e2d9d640 1 -- 172.21.15.98:0/555012265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 msgr2=0x558873ea8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.527+0000 7ff0e2d9d640 1 --2- 172.21.15.98:0/555012265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 0x558873ea8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.527+0000 7ff0e2d9d640 1 -- 172.21.15.98:0/555012265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558873ea6830 con 0x558873ea0330 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.527+0000 7ff0dbfff640 1 --2- 172.21.15.98:0/555012265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 0x558873ea8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.527+0000 7ff0e2d9d640 1 --2- 172.21.15.98:0/555012265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873ea0330 0x558873ea2a90 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x558873ec21f0 tx=0x7ff0d4012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=215827573f9d89a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.527+0000 7ff0d8ff9640 1 -- 172.21.15.98:0/555012265 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d40120d0 con 0x558873ea0330 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.528+0000 7ff0d8ff9640 1 -- 172.21.15.98:0/555012265 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0d4012270 con 0x558873ea0330 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.528+0000 7ff0d8ff9640 1 -- 172.21.15.98:0/555012265 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4010a70 con 0x558873ea0330 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.528+0000 7ff0ebe15740 1 -- 172.21.15.98:0/555012265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873ea0330 msgr2=0x558873ea2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.528+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/555012265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873ea0330 0x558873ea2a90 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x558873ec21f0 tx=0x7ff0d4012820 comp rx=0 tx=0).stop 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.528+0000 7ff0ebe15740 1 -- 172.21.15.98:0/555012265 shutdown_connections 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.528+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/555012265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873eab320 0x558873eaa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.528+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/555012265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 0x558873ea8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.528+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/555012265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873ea0330 0x558873ea2a90 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.528+0000 7ff0ebe15740 1 -- 172.21.15.98:0/555012265 >> 172.21.15.98:0/555012265 conn(0x558873e9c650 msgr2=0x558873ea99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.528+0000 7ff0ebe15740 1 -- 172.21.15.98:0/555012265 shutdown_connections 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.528+0000 7ff0ebe15740 1 -- 172.21.15.98:0/555012265 wait complete. 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.529+0000 7ff0ebe15740 1 -- start start 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.529+0000 7ff0ebe15740 1 Processor -- start 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.529+0000 7ff0ebe15740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873ea0330 0x558873eb33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.529+0000 7ff0ebe15740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 0x558873eb3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.529+0000 7ff0ebe15740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873eab320 0x558873f24af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.529+0000 7ff0ebe15740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e41ae090 con 0x558873eab320 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.529+0000 7ff0ebe15740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e41ad9c0 con 0x558873ea0c50 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.529+0000 7ff0ebe15740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e41aac20 con 0x558873ea0330 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.529+0000 7ff0ebe15740 10 client.0 _ll_register_callbacks cb 0x7ff0d4005f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.529+0000 7ff0e359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873eab320 0x558873f24af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.529+0000 7ff0e359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873eab320 0x558873f24af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35150/0 (socket says 172.21.15.98:35150) 2024-05-09T09:00:22.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.529+0000 7ff0e359e640 1 -- 172.21.15.98:0/1731536508 learned_addr learned my addr 172.21.15.98:0/1731536508 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.529+0000 7ff0dbfff640 1 --2- 172.21.15.98:0/1731536508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 0x558873eb3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.529+0000 7ff0e2d9d640 1 --2- 172.21.15.98:0/1731536508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873ea0330 0x558873eb33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.530+0000 7ff0e359e640 1 -- 172.21.15.98:0/1731536508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873ea0330 msgr2=0x558873eb33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.530+0000 7ff0e359e640 1 --2- 172.21.15.98:0/1731536508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873ea0330 0x558873eb33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.530+0000 7ff0e359e640 1 -- 172.21.15.98:0/1731536508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 msgr2=0x558873eb3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.530+0000 7ff0e359e640 1 --2- 172.21.15.98:0/1731536508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 0x558873eb3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.530+0000 7ff0e359e640 1 -- 172.21.15.98:0/1731536508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558873f25030 con 0x558873eab320 2024-05-09T09:00:22.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.530+0000 7ff0e359e640 1 --2- 172.21.15.98:0/1731536508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873eab320 0x558873f24af0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc010b70 tx=0x7ff0dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe4502a6d085185 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.530+0000 7ff0ca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:22.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.530+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/1731536508 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc018070 con 0x558873eab320 2024-05-09T09:00:22.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.530+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/1731536508 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7ff0dc00fc10 con 0x558873eab320 2024-05-09T09:00:22.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.530+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1731536508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0d4012270 con 0x558873eab320 2024-05-09T09:00:22.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.530+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/1731536508 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff0dc034050 con 0x558873eab320 2024-05-09T09:00:22.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.530+0000 7ff0ebe15740 10 client.18393 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:22.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.530+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1731536508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558873f26210 con 0x558873eab320 2024-05-09T09:00:22.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.530+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/1731536508 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0d4012270 con 0x558873eab320 2024-05-09T09:00:22.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.531+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/1731536508 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7ff0dc004580 con 0x558873eab320 2024-05-09T09:00:22.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.531+0000 7ff0ebe15740 10 client.18393 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:22.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.531+0000 7ff0ebe15740 10 client.18393 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T09:00:22.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.531+0000 7ff0ebe15740 20 client.18393 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:22.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.531+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/1731536508 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558873f269e0 0x558873f28e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.531+0000 7ff0ebe15740 4 client.18393 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:00:22.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.531+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1731536508 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558873d8a820 con 0x558873f269e0 2024-05-09T09:00:22.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.531+0000 7ff0e2d9d640 1 --2- 172.21.15.98:0/1731536508 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558873f269e0 0x558873f28e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.531+0000 7ff0e2d9d640 1 --2- 172.21.15.98:0/1731536508 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558873f269e0 0x558873f28e00 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f1bcbe73f60833a1 server_cookie=f68741c279c358bc in_seq=0 out_seq=0 2024-05-09T09:00:22.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.531+0000 7ff0ca7fc640 10 client.18393 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T09:00:22.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/1731536508 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558873d8a820 con 0x558873f269e0 2024-05-09T09:00:22.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ca7fc640 10 client.18393 handle_command_reply: tid=0 2024-05-09T09:00:22.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ebe15740 1 client.18393 shutdown 2024-05-09T09:00:22.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1731536508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873eab320 msgr2=0x558873f24af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/1731536508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873eab320 0x558873f24af0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc010b70 tx=0x7ff0dc004a40 comp rx=0 tx=0).stop 2024-05-09T09:00:22.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1731536508 shutdown_connections 2024-05-09T09:00:22.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/1731536508 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x558873f269e0 0x558873f28e00 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/1731536508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558873eab320 0x558873f24af0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ca7fc640 0 client.18393 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T09:00:22.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/1731536508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558873ea0c50 0x558873eb3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ebe15740 1 --2- 172.21.15.98:0/1731536508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558873ea0330 0x558873eb33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1731536508 >> 172.21.15.98:0/1731536508 conn(0x558873e9c650 msgr2=0x558873e9b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1731536508 shutdown_connections 2024-05-09T09:00:22.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ebe15740 1 -- 172.21.15.98:0/1731536508 wait complete. 2024-05-09T09:00:22.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0ebe15740 20 client.18393 trim_cache size 0 max 16384 2024-05-09T09:00:22.539 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:00:22.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0e9109640 1 -- 172.21.15.98:0/708551477 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff0d00a2d00 msgr2=0x7ff0d00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0e9109640 1 --2- 172.21.15.98:0/708551477 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff0d00a2d00 0x7ff0d00a51c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc009270 tx=0x7ff0dc009500 comp rx=0 tx=0).stop 2024-05-09T09:00:22.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0e9109640 1 -- 172.21.15.98:0/708551477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0e41033d0 msgr2=0x7ff0e41985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0e9109640 1 --2- 172.21.15.98:0/708551477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0e41033d0 0x7ff0e41985f0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc006010 tx=0x7ff0cc0042f0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0e9109640 1 -- 172.21.15.98:0/708551477 shutdown_connections 2024-05-09T09:00:22.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0e9109640 1 --2- 172.21.15.98:0/708551477 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff0d00a2d00 0x7ff0d00a51c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0e9109640 1 --2- 172.21.15.98:0/708551477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0e410bdf0 0x7ff0e419c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0e9109640 1 --2- 172.21.15.98:0/708551477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0e41033d0 0x7ff0e41985f0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0e9109640 1 --2- 172.21.15.98:0/708551477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0e4102ab0 0x7ff0e41980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.532+0000 7ff0e9109640 1 -- 172.21.15.98:0/708551477 >> 172.21.15.98:0/708551477 conn(0x7ff0e4077830 msgr2=0x7ff0e40fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.533+0000 7ff0e9109640 1 -- 172.21.15.98:0/708551477 shutdown_connections 2024-05-09T09:00:22.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.533+0000 7ff0e9109640 1 -- 172.21.15.98:0/708551477 wait complete. 2024-05-09T09:00:22.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T09:00:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.647+0000 7f4806612640 1 -- 172.21.15.98:0/1899159603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800107810 msgr2=0x7f4800107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.647+0000 7f4806612640 1 --2- 172.21.15.98:0/1899159603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800107810 0x7f4800107bf0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009930 tx=0x7f47f002a420 comp rx=0 tx=0).stop 2024-05-09T09:00:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.647+0000 7f4806612640 1 -- 172.21.15.98:0/1899159603 shutdown_connections 2024-05-09T09:00:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.647+0000 7f4806612640 1 --2- 172.21.15.98:0/1899159603 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48001021c0 0x7f480010e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.647+0000 7f4806612640 1 --2- 172.21.15.98:0/1899159603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4800101800 0x7f4800101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.647+0000 7f4806612640 1 --2- 172.21.15.98:0/1899159603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800107810 0x7f4800107bf0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.647+0000 7f4806612640 1 -- 172.21.15.98:0/1899159603 >> 172.21.15.98:0/1899159603 conn(0x7f48000fd270 msgr2=0x7f48000ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.647+0000 7f4806612640 1 -- 172.21.15.98:0/1899159603 shutdown_connections 2024-05-09T09:00:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.647+0000 7f4806612640 1 -- 172.21.15.98:0/1899159603 wait complete. 2024-05-09T09:00:22.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.648+0000 7f4806612640 1 Processor -- start 2024-05-09T09:00:22.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.648+0000 7f4806612640 1 -- start start 2024-05-09T09:00:22.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.648+0000 7f4806612640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4800101800 0x7f4800198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.648+0000 7f4806612640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48001021c0 0x7f48001986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.648+0000 7f4806612640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800107810 0x7f480019ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.648+0000 7f4806612640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4800111c20 con 0x7f4800107810 2024-05-09T09:00:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.648+0000 7f4806612640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4800111aa0 con 0x7f4800101800 2024-05-09T09:00:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.648+0000 7f4806612640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4800111da0 con 0x7f48001021c0 2024-05-09T09:00:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.648+0000 7f4804baf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800107810 0x7f480019ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.648+0000 7f4804baf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800107810 0x7f480019ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35176/0 (socket says 172.21.15.98:35176) 2024-05-09T09:00:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.648+0000 7f4804baf640 1 -- 172.21.15.98:0/983097467 learned_addr learned my addr 172.21.15.98:0/983097467 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.648+0000 7f47fffff640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4800101800 0x7f4800198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.648+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48001021c0 0x7f48001986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.649+0000 7f4804baf640 1 -- 172.21.15.98:0/983097467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48001021c0 msgr2=0x7f48001986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.649+0000 7f4804baf640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48001021c0 0x7f48001986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.649+0000 7f4804baf640 1 -- 172.21.15.98:0/983097467 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4800101800 msgr2=0x7f4800198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.649+0000 7f4804baf640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4800101800 0x7f4800198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.649+0000 7f4804baf640 1 -- 172.21.15.98:0/983097467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480019d150 con 0x7f4800107810 2024-05-09T09:00:22.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.649+0000 7f47fffff640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4800101800 0x7f4800198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:00:22.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.649+0000 7f4804baf640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800107810 0x7f480019ca20 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f47f800eaa0 tx=0x7f47f800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.649+0000 7f47fd7fa640 1 -- 172.21.15.98:0/983097467 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f800cdb0 con 0x7f4800107810 2024-05-09T09:00:22.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.649+0000 7f4806612640 1 -- 172.21.15.98:0/983097467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f480019d440 con 0x7f4800107810 2024-05-09T09:00:22.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.649+0000 7f47fd7fa640 1 -- 172.21.15.98:0/983097467 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f8005020 con 0x7f4800107810 2024-05-09T09:00:22.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.649+0000 7f47fd7fa640 1 -- 172.21.15.98:0/983097467 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f80116d0 con 0x7f4800107810 2024-05-09T09:00:22.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.649+0000 7f4806612640 1 -- 172.21.15.98:0/983097467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48001a4cd0 con 0x7f4800107810 2024-05-09T09:00:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:22 smithi142 ceph-mon[34625]: pgmap v304: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-05-09T09:00:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.651+0000 7f47fd7fa640 1 -- 172.21.15.98:0/983097467 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f47f80054d0 con 0x7f4800107810 2024-05-09T09:00:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.652+0000 7f48093f6740 1 Processor -- start 2024-05-09T09:00:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.652+0000 7f48093f6740 1 -- start start 2024-05-09T09:00:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.652+0000 7f48093f6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ea2330 0x5585c1ea2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.652+0000 7f48093f6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2c50 0x5585c1eacde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.652+0000 7f48093f6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ead320 0x5585c1eaf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.652+0000 7f48093f6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5585c1e07c40 con 0x5585c1ead320 2024-05-09T09:00:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.652+0000 7f48093f6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5585c1daf580 con 0x5585c1ea2330 2024-05-09T09:00:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.652+0000 7f48093f6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5585c1e03c00 con 0x5585c1ea2c50 2024-05-09T09:00:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.652+0000 7f47fd7fa640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f47e80a2d00 0x7f47e80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.652+0000 7f47ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ead320 0x5585c1eaf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.652+0000 7f47ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ead320 0x5585c1eaf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35192/0 (socket says 172.21.15.98:35192) 2024-05-09T09:00:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.652+0000 7f47ff7fe640 1 -- 172.21.15.98:0/4155102057 learned_addr learned my addr 172.21.15.98:0/4155102057 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.652+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/4155102057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ea2330 0x5585c1ea2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.652+0000 7f4804baf640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f47e80a2d00 0x7f47e80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.653+0000 7f47fffff640 1 --2- 172.21.15.98:0/4155102057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2c50 0x5585c1eacde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.653+0000 7f47fd7fa640 1 -- 172.21.15.98:0/983097467 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f47f80cf9f0 con 0x7f4800107810 2024-05-09T09:00:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.653+0000 7f47ff7fe640 1 -- 172.21.15.98:0/4155102057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2c50 msgr2=0x5585c1eacde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.653+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/4155102057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2c50 0x5585c1eacde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.653+0000 7f47ff7fe640 1 -- 172.21.15.98:0/4155102057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ea2330 msgr2=0x5585c1ea2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.653+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/4155102057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ea2330 0x5585c1ea2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.653+0000 7f47ff7fe640 1 -- 172.21.15.98:0/4155102057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585c1b1b3f0 con 0x5585c1ead320 2024-05-09T09:00:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.653+0000 7f4804baf640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f47e80a2d00 0x7f47e80a51c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f47f8098520 tx=0x7f47f8000f40 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.653+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/4155102057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ead320 0x5585c1eaf710 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f47f40093e0 tx=0x7f47f4009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48621fc8a2a21a4d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.653+0000 7f47e67fc640 1 -- 172.21.15.98:0/4155102057 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f400f7c0 con 0x5585c1ead320 2024-05-09T09:00:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.653+0000 7f47e67fc640 1 -- 172.21.15.98:0/4155102057 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f400fe40 con 0x5585c1ead320 2024-05-09T09:00:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.653+0000 7f47e67fc640 1 -- 172.21.15.98:0/4155102057 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f400ee40 con 0x5585c1ead320 2024-05-09T09:00:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.654+0000 7f48093f6740 1 -- 172.21.15.98:0/4155102057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ead320 msgr2=0x5585c1eaf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.654+0000 7f48093f6740 1 --2- 172.21.15.98:0/4155102057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ead320 0x5585c1eaf710 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f47f40093e0 tx=0x7f47f4009370 comp rx=0 tx=0).stop 2024-05-09T09:00:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.654+0000 7f48093f6740 1 -- 172.21.15.98:0/4155102057 shutdown_connections 2024-05-09T09:00:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.654+0000 7f48093f6740 1 --2- 172.21.15.98:0/4155102057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ead320 0x5585c1eaf710 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.654+0000 7f48093f6740 1 --2- 172.21.15.98:0/4155102057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2c50 0x5585c1eacde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.654+0000 7f48093f6740 1 --2- 172.21.15.98:0/4155102057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ea2330 0x5585c1ea2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.654+0000 7f48093f6740 1 -- 172.21.15.98:0/4155102057 >> 172.21.15.98:0/4155102057 conn(0x5585c1e9e650 msgr2=0x5585c1e9ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.654+0000 7f48093f6740 1 -- 172.21.15.98:0/4155102057 shutdown_connections 2024-05-09T09:00:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.654+0000 7f48093f6740 1 -- 172.21.15.98:0/4155102057 wait complete. 2024-05-09T09:00:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.654+0000 7f48093f6740 1 -- start start 2024-05-09T09:00:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f48093f6740 1 Processor -- start 2024-05-09T09:00:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f48093f6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2330 0x5585c1eaa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f48093f6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eaa830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f48093f6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ead320 0x5585c1eaad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f48093f6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5585c1eb0110 con 0x5585c1ea2c50 2024-05-09T09:00:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f48093f6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5585c1dc2c40 con 0x5585c1ead320 2024-05-09T09:00:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f48093f6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5585c1eb0290 con 0x5585c1ea2330 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f48093f6740 10 client.0 _ll_register_callbacks cb 0x5585c1b1abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f47fffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eaa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f47fffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eaa830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35198/0 (socket says 172.21.15.98:35198) 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f47fffff640 1 -- 172.21.15.98:0/3147065255 learned_addr learned my addr 172.21.15.98:0/3147065255 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/3147065255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2330 0x5585c1eaa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/3147065255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ead320 0x5585c1eaad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f47fffff640 1 -- 172.21.15.98:0/3147065255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2330 msgr2=0x5585c1eaa2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f47fffff640 1 --2- 172.21.15.98:0/3147065255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2330 0x5585c1eaa2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f47fffff640 1 -- 172.21.15.98:0/3147065255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ead320 msgr2=0x5585c1eaad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f47fffff640 1 --2- 172.21.15.98:0/3147065255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ead320 0x5585c1eaad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f47fffff640 1 -- 172.21.15.98:0/3147065255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585c1be57d0 con 0x5585c1ea2c50 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.655+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/3147065255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ead320 0x5585c1eaad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f47fffff640 1 --2- 172.21.15.98:0/3147065255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eaa830 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009930 tx=0x7f47f00317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f4ee9ae6e75269 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f47df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f47df7fe640 1 -- 172.21.15.98:0/3147065255 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0035040 con 0x5585c1ea2c50 2024-05-09T09:00:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f47df7fe640 1 -- 172.21.15.98:0/3147065255 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f47f0004250 con 0x5585c1ea2c50 2024-05-09T09:00:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f48093f6740 1 -- 172.21.15.98:0/3147065255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47f400fe40 con 0x5585c1ea2c50 2024-05-09T09:00:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f47df7fe640 1 -- 172.21.15.98:0/3147065255 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f47f0031a10 con 0x5585c1ea2c50 2024-05-09T09:00:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f48093f6740 10 client.18405 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f48093f6740 1 -- 172.21.15.98:0/3147065255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5585c1ebc350 con 0x5585c1ea2c50 2024-05-09T09:00:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f47df7fe640 1 -- 172.21.15.98:0/3147065255 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f400fe40 con 0x5585c1ea2c50 2024-05-09T09:00:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f47df7fe640 1 -- 172.21.15.98:0/3147065255 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f47f0004bb0 con 0x5585c1ea2c50 2024-05-09T09:00:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f48093f6740 10 client.18405 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f48093f6740 10 client.18405 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T09:00:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f48093f6740 20 client.18405 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f48093f6740 1 --2- 172.21.15.98:0/3147065255 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5585c1ebd810 0x5585c1ebfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f48093f6740 4 client.18405 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.656+0000 7f48093f6740 1 -- 172.21.15.98:0/3147065255 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585c1e15370 con 0x5585c1ebd810 2024-05-09T09:00:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.657+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/3147065255 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5585c1ebd810 0x5585c1ebfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.657+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/3147065255 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5585c1ebd810 0x5585c1ebfc50 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d0539e7382444e11 server_cookie=96a9f4538529c0b9 in_seq=0 out_seq=0 2024-05-09T09:00:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.657+0000 7f47df7fe640 10 client.18405 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T09:00:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f47df7fe640 1 -- 172.21.15.98:0/3147065255 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5585c1e15370 con 0x5585c1ebd810 2024-05-09T09:00:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f47df7fe640 10 client.18405 handle_command_reply: tid=0 2024-05-09T09:00:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f48093f6740 1 client.18405 shutdown 2024-05-09T09:00:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f48093f6740 1 -- 172.21.15.98:0/3147065255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 msgr2=0x5585c1eaa830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f48093f6740 1 --2- 172.21.15.98:0/3147065255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eaa830 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009930 tx=0x7f47f00317e0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f48093f6740 1 -- 172.21.15.98:0/3147065255 shutdown_connections 2024-05-09T09:00:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f48093f6740 1 --2- 172.21.15.98:0/3147065255 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5585c1ebd810 0x5585c1ebfc50 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f48093f6740 1 --2- 172.21.15.98:0/3147065255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ead320 0x5585c1eaad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f48093f6740 1 --2- 172.21.15.98:0/3147065255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eaa830 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f47df7fe640 0 client.18405 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T09:00:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f48093f6740 1 --2- 172.21.15.98:0/3147065255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2330 0x5585c1eaa2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f48093f6740 1 -- 172.21.15.98:0/3147065255 >> 172.21.15.98:0/3147065255 conn(0x5585c1e9e650 msgr2=0x5585c1ea4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f48093f6740 1 -- 172.21.15.98:0/3147065255 shutdown_connections 2024-05-09T09:00:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f48093f6740 1 -- 172.21.15.98:0/3147065255 wait complete. 2024-05-09T09:00:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.658+0000 7f48093f6740 20 client.18405 trim_cache size 0 max 16384 2024-05-09T09:00:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.671+0000 7f48093f6740 1 Processor -- start 2024-05-09T09:00:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.671+0000 7f48093f6740 1 -- start start 2024-05-09T09:00:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.671+0000 7f48093f6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ea2330 0x5585c1eb8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.671+0000 7f48093f6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eaabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.671+0000 7f48093f6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ead320 0x5585c1ec5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.671+0000 7f48093f6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48001af000 con 0x5585c1ea2c50 2024-05-09T09:00:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.671+0000 7f48093f6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f48001b0b40 con 0x5585c1ea2330 2024-05-09T09:00:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.671+0000 7f48093f6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f48001af6d0 con 0x5585c1ead320 2024-05-09T09:00:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.671+0000 7f47fffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eaabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.671+0000 7f47fffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eaabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35202/0 (socket says 172.21.15.98:35202) 2024-05-09T09:00:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.671+0000 7f47fffff640 1 -- 172.21.15.98:0/725630580 learned_addr learned my addr 172.21.15.98:0/725630580 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.672+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/725630580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ea2330 0x5585c1eb8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.672+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/725630580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ead320 0x5585c1ec5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.672+0000 7f47fffff640 1 -- 172.21.15.98:0/725630580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ead320 msgr2=0x5585c1ec5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.672+0000 7f47fffff640 1 --2- 172.21.15.98:0/725630580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ead320 0x5585c1ec5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.672+0000 7f47fffff640 1 -- 172.21.15.98:0/725630580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ea2330 msgr2=0x5585c1eb8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.672+0000 7f47fffff640 1 --2- 172.21.15.98:0/725630580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ea2330 0x5585c1eb8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.672+0000 7f47fffff640 1 -- 172.21.15.98:0/725630580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585c1e8fd20 con 0x5585c1ea2c50 2024-05-09T09:00:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.672+0000 7f47fffff640 1 --2- 172.21.15.98:0/725630580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eaabd0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f47f0004f80 tx=0x7f47f002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2d556bf5d3c682 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.672+0000 7f47e5ffb640 1 -- 172.21.15.98:0/725630580 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0057070 con 0x5585c1ea2c50 2024-05-09T09:00:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.672+0000 7f47e5ffb640 1 -- 172.21.15.98:0/725630580 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f0004c50 con 0x5585c1ea2c50 2024-05-09T09:00:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.672+0000 7f47e5ffb640 1 -- 172.21.15.98:0/725630580 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0035030 con 0x5585c1ea2c50 2024-05-09T09:00:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.673+0000 7f48093f6740 1 -- 172.21.15.98:0/725630580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 msgr2=0x5585c1eaabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.673+0000 7f48093f6740 1 --2- 172.21.15.98:0/725630580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eaabd0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f47f0004f80 tx=0x7f47f002a420 comp rx=0 tx=0).stop 2024-05-09T09:00:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.673+0000 7f48093f6740 1 -- 172.21.15.98:0/725630580 shutdown_connections 2024-05-09T09:00:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.673+0000 7f48093f6740 1 --2- 172.21.15.98:0/725630580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ead320 0x5585c1ec5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.673+0000 7f48093f6740 1 --2- 172.21.15.98:0/725630580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eaabd0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.673+0000 7f48093f6740 1 --2- 172.21.15.98:0/725630580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ea2330 0x5585c1eb8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.673+0000 7f48093f6740 1 -- 172.21.15.98:0/725630580 >> 172.21.15.98:0/725630580 conn(0x5585c1e9e650 msgr2=0x5585c1e9d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.673+0000 7f48093f6740 1 -- 172.21.15.98:0/725630580 shutdown_connections 2024-05-09T09:00:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.673+0000 7f48093f6740 1 -- 172.21.15.98:0/725630580 wait complete. 2024-05-09T09:00:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.673+0000 7f48093f6740 1 -- start start 2024-05-09T09:00:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.673+0000 7f48093f6740 1 Processor -- start 2024-05-09T09:00:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f48093f6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2330 0x5585c1eb57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f48093f6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eb2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f48093f6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ead320 0x5585c1eb30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f48093f6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48001b0470 con 0x5585c1ea2c50 2024-05-09T09:00:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f48093f6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f48001afda0 con 0x5585c1ead320 2024-05-09T09:00:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f48093f6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f48001ad000 con 0x5585c1ea2330 2024-05-09T09:00:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f48093f6740 10 client.0 _ll_register_callbacks cb 0x7f47f0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:22.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47fffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eb2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47fffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eb2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35204/0 (socket says 172.21.15.98:35204) 2024-05-09T09:00:22.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47fffff640 1 -- 172.21.15.98:0/944599713 learned_addr learned my addr 172.21.15.98:0/944599713 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/944599713 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2330 0x5585c1eb57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/944599713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ead320 0x5585c1eb30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47fffff640 1 -- 172.21.15.98:0/944599713 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2330 msgr2=0x5585c1eb57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47fffff640 1 --2- 172.21.15.98:0/944599713 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2330 0x5585c1eb57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47fffff640 1 -- 172.21.15.98:0/944599713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ead320 msgr2=0x5585c1eb30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47fffff640 1 --2- 172.21.15.98:0/944599713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ead320 0x5585c1eb30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47fffff640 1 -- 172.21.15.98:0/944599713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585c1eb37e0 con 0x5585c1ea2c50 2024-05-09T09:00:22.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47fffff640 1 --2- 172.21.15.98:0/944599713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eb2b60 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f47f00096c0 tx=0x7f47f0005600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f4408ed5a42e82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47df7fe640 1 -- 172.21.15.98:0/944599713 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0060060 con 0x5585c1ea2c50 2024-05-09T09:00:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47df7fe640 1 -- 172.21.15.98:0/944599713 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f47f0004970 con 0x5585c1ea2c50 2024-05-09T09:00:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f48093f6740 1 -- 172.21.15.98:0/944599713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47f0004c50 con 0x5585c1ea2c50 2024-05-09T09:00:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f47df7fe640 1 -- 172.21.15.98:0/944599713 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f47f0035070 con 0x5585c1ea2c50 2024-05-09T09:00:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f48093f6740 10 client.18411 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.674+0000 7f48093f6740 1 -- 172.21.15.98:0/944599713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5585c1f23cd0 con 0x5585c1ea2c50 2024-05-09T09:00:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.675+0000 7f47df7fe640 1 -- 172.21.15.98:0/944599713 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f0004c50 con 0x5585c1ea2c50 2024-05-09T09:00:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.675+0000 7f47df7fe640 1 -- 172.21.15.98:0/944599713 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f47f005a070 con 0x5585c1ea2c50 2024-05-09T09:00:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.675+0000 7f48093f6740 10 client.18411 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.675+0000 7f48093f6740 10 client.18411 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T09:00:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.675+0000 7f48093f6740 20 client.18411 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.675+0000 7f48093f6740 1 --2- 172.21.15.98:0/944599713 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5585c1f24330 0x5585c1f26750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.675+0000 7f48093f6740 4 client.18411 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:00:22.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.675+0000 7f48093f6740 1 -- 172.21.15.98:0/944599713 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5585c1e0f6c0 con 0x5585c1f24330 2024-05-09T09:00:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.675+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/944599713 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5585c1f24330 0x5585c1f26750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.675+0000 7f47df7fe640 10 client.18411 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T09:00:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.675+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/944599713 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5585c1f24330 0x5585c1f26750 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dae771d7362b9fd3 server_cookie=a2433b6b4ed9a18f in_seq=0 out_seq=0 2024-05-09T09:00:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f47df7fe640 1 -- 172.21.15.98:0/944599713 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5585c1e0f6c0 con 0x5585c1f24330 2024-05-09T09:00:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f47df7fe640 10 client.18411 handle_command_reply: tid=0 2024-05-09T09:00:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f48093f6740 1 client.18411 shutdown 2024-05-09T09:00:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f48093f6740 1 -- 172.21.15.98:0/944599713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 msgr2=0x5585c1eb2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f48093f6740 1 --2- 172.21.15.98:0/944599713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eb2b60 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f47f00096c0 tx=0x7f47f0005600 comp rx=0 tx=0).stop 2024-05-09T09:00:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f48093f6740 1 -- 172.21.15.98:0/944599713 shutdown_connections 2024-05-09T09:00:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f48093f6740 1 --2- 172.21.15.98:0/944599713 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x5585c1f24330 0x5585c1f26750 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f47df7fe640 0 client.18411 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T09:00:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f48093f6740 1 --2- 172.21.15.98:0/944599713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585c1ead320 0x5585c1eb30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f48093f6740 1 --2- 172.21.15.98:0/944599713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5585c1ea2c50 0x5585c1eb2b60 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f48093f6740 1 --2- 172.21.15.98:0/944599713 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5585c1ea2330 0x5585c1eb57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f48093f6740 1 -- 172.21.15.98:0/944599713 >> 172.21.15.98:0/944599713 conn(0x5585c1e9e650 msgr2=0x5585c1e9d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f48093f6740 1 -- 172.21.15.98:0/944599713 shutdown_connections 2024-05-09T09:00:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f48093f6740 1 -- 172.21.15.98:0/944599713 wait complete. 2024-05-09T09:00:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.676+0000 7f48093f6740 20 client.18411 trim_cache size 0 max 16384 2024-05-09T09:00:22.683 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:00:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.677+0000 7f4806612640 1 -- 172.21.15.98:0/983097467 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f47e80a2d00 msgr2=0x7f47e80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.677+0000 7f4806612640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f47e80a2d00 0x7f47e80a51c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f47f8098520 tx=0x7f47f8000f40 comp rx=0 tx=0).stop 2024-05-09T09:00:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.677+0000 7f4806612640 1 -- 172.21.15.98:0/983097467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800107810 msgr2=0x7f480019ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.677+0000 7f4806612640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800107810 0x7f480019ca20 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f47f800eaa0 tx=0x7f47f800ef60 comp rx=0 tx=0).stop 2024-05-09T09:00:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.677+0000 7f4806612640 1 -- 172.21.15.98:0/983097467 shutdown_connections 2024-05-09T09:00:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.677+0000 7f4806612640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f47e80a2d00 0x7f47e80a51c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.677+0000 7f4806612640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800107810 0x7f480019ca20 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.677+0000 7f4806612640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48001021c0 0x7f48001986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.677+0000 7f4806612640 1 --2- 172.21.15.98:0/983097467 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4800101800 0x7f4800198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.677+0000 7f4806612640 1 -- 172.21.15.98:0/983097467 >> 172.21.15.98:0/983097467 conn(0x7f48000fd270 msgr2=0x7f48000fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.677+0000 7f4806612640 1 -- 172.21.15.98:0/983097467 shutdown_connections 2024-05-09T09:00:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.677+0000 7f4806612640 1 -- 172.21.15.98:0/983097467 wait complete. 2024-05-09T09:00:22.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T09:00:22.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.809+0000 7efee2e4d640 1 -- 172.21.15.98:0/3105446219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efedc106710 msgr2=0x7efedc106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.809+0000 7efee2e4d640 1 --2- 172.21.15.98:0/3105446219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efedc106710 0x7efedc106af0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7efed0009930 tx=0x7efed002a420 comp rx=0 tx=0).stop 2024-05-09T09:00:22.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.809+0000 7efee2e4d640 1 -- 172.21.15.98:0/3105446219 shutdown_connections 2024-05-09T09:00:22.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.809+0000 7efee2e4d640 1 --2- 172.21.15.98:0/3105446219 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc1010c0 0x7efedc10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.809+0000 7efee2e4d640 1 --2- 172.21.15.98:0/3105446219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efedc100700 0x7efedc100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.809+0000 7efee2e4d640 1 --2- 172.21.15.98:0/3105446219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efedc106710 0x7efedc106af0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.809+0000 7efee2e4d640 1 -- 172.21.15.98:0/3105446219 >> 172.21.15.98:0/3105446219 conn(0x7efedc077830 msgr2=0x7efedc077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.809+0000 7efee2e4d640 1 -- 172.21.15.98:0/3105446219 shutdown_connections 2024-05-09T09:00:22.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.809+0000 7efee2e4d640 1 -- 172.21.15.98:0/3105446219 wait complete. 2024-05-09T09:00:22.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.810+0000 7efee2e4d640 1 Processor -- start 2024-05-09T09:00:22.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.810+0000 7efee2e4d640 1 -- start start 2024-05-09T09:00:22.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.810+0000 7efee2e4d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efedc100700 0x7efedc198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee2e4d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc1010c0 0x7efedc1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee2e4d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efedc106710 0x7efedc19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee2e4d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efedc110b20 con 0x7efedc106710 2024-05-09T09:00:22.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee2e4d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efedc1109a0 con 0x7efedc100700 2024-05-09T09:00:22.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee2e4d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efedc110ca0 con 0x7efedc1010c0 2024-05-09T09:00:22.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee13ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efedc106710 0x7efedc19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee13ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efedc106710 0x7efedc19ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35228/0 (socket says 172.21.15.98:35228) 2024-05-09T09:00:22.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee13ea640 1 -- 172.21.15.98:0/3960864869 learned_addr learned my addr 172.21.15.98:0/3960864869 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee0be9640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efedc100700 0x7efedc198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efedbfff640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc1010c0 0x7efedc1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee13ea640 1 -- 172.21.15.98:0/3960864869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc1010c0 msgr2=0x7efedc1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee13ea640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc1010c0 0x7efedc1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee13ea640 1 -- 172.21.15.98:0/3960864869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efedc100700 msgr2=0x7efedc198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee13ea640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efedc100700 0x7efedc198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee13ea640 1 -- 172.21.15.98:0/3960864869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efedc19d170 con 0x7efedc106710 2024-05-09T09:00:22.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efedbfff640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc1010c0 0x7efedc1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:22.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee13ea640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efedc106710 0x7efedc19ca40 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7efed400eaa0 tx=0x7efed400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efed9ffb640 1 -- 172.21.15.98:0/3960864869 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed400cdb0 con 0x7efedc106710 2024-05-09T09:00:22.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efee2e4d640 1 -- 172.21.15.98:0/3960864869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efedc19d460 con 0x7efedc106710 2024-05-09T09:00:22.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efed9ffb640 1 -- 172.21.15.98:0/3960864869 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efed4005020 con 0x7efedc106710 2024-05-09T09:00:22.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.811+0000 7efed9ffb640 1 -- 172.21.15.98:0/3960864869 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed40116d0 con 0x7efedc106710 2024-05-09T09:00:22.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.812+0000 7efee2e4d640 1 -- 172.21.15.98:0/3960864869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efedc1a4cf0 con 0x7efedc106710 2024-05-09T09:00:22.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efed9ffb640 1 -- 172.21.15.98:0/3960864869 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7efed40054d0 con 0x7efedc106710 2024-05-09T09:00:22.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efee5c39740 1 Processor -- start 2024-05-09T09:00:22.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efee5c39740 1 -- start start 2024-05-09T09:00:22.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efee5c39740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13330 0x559cd9a13710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efee5c39740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13c50 0x559cd9a1dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efee5c39740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a1e320 0x559cd9a20710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efee5c39740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559cd9978c40 con 0x559cd9a13330 2024-05-09T09:00:22.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efee5c39740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559cd9920580 con 0x559cd9a13c50 2024-05-09T09:00:22.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efee5c39740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559cd9974c00 con 0x559cd9a1e320 2024-05-09T09:00:22.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efed9ffb640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efecc0a2d00 0x7efecc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efedbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13330 0x559cd9a13710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efedbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13330 0x559cd9a13710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35234/0 (socket says 172.21.15.98:35234) 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efedbfff640 1 -- 172.21.15.98:0/2287735317 learned_addr learned my addr 172.21.15.98:0/2287735317 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efee0be9640 1 --2- 172.21.15.98:0/2287735317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13c50 0x559cd9a1dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.814+0000 7efee13ea640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efecc0a2d00 0x7efecc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.815+0000 7efedbfff640 1 --2- 172.21.15.98:0/2287735317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a1e320 0x559cd9a20710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.815+0000 7efedbfff640 1 -- 172.21.15.98:0/2287735317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a1e320 msgr2=0x559cd9a20710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.815+0000 7efedbfff640 1 --2- 172.21.15.98:0/2287735317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a1e320 0x559cd9a20710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.815+0000 7efedbfff640 1 -- 172.21.15.98:0/2287735317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13c50 msgr2=0x559cd9a1dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.815+0000 7efedbfff640 1 --2- 172.21.15.98:0/2287735317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13c50 0x559cd9a1dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.815+0000 7efedbfff640 1 -- 172.21.15.98:0/2287735317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cd968c3f0 con 0x559cd9a13330 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.815+0000 7efed9ffb640 1 -- 172.21.15.98:0/3960864869 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7efed40cfaf0 con 0x7efedc106710 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.815+0000 7efedbfff640 1 --2- 172.21.15.98:0/2287735317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13330 0x559cd9a13710 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7efec8015560 tx=0x7efec8009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a5010c053f41a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.815+0000 7efee13ea640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efecc0a2d00 0x7efecc0a51c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efed4098060 tx=0x7efed4097800 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.815+0000 7efec6ffd640 1 -- 172.21.15.98:0/2287735317 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec80134c0 con 0x559cd9a13330 2024-05-09T09:00:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.815+0000 7efec6ffd640 1 -- 172.21.15.98:0/2287735317 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efec801b070 con 0x559cd9a13330 2024-05-09T09:00:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.815+0000 7efec6ffd640 1 -- 172.21.15.98:0/2287735317 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec8012740 con 0x559cd9a13330 2024-05-09T09:00:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.816+0000 7efee5c39740 1 -- 172.21.15.98:0/2287735317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13330 msgr2=0x559cd9a13710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.816+0000 7efee5c39740 1 --2- 172.21.15.98:0/2287735317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13330 0x559cd9a13710 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7efec8015560 tx=0x7efec8009630 comp rx=0 tx=0).stop 2024-05-09T09:00:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.816+0000 7efee5c39740 1 -- 172.21.15.98:0/2287735317 shutdown_connections 2024-05-09T09:00:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.816+0000 7efee5c39740 1 --2- 172.21.15.98:0/2287735317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a1e320 0x559cd9a20710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.816+0000 7efee5c39740 1 --2- 172.21.15.98:0/2287735317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13c50 0x559cd9a1dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.816+0000 7efee5c39740 1 --2- 172.21.15.98:0/2287735317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13330 0x559cd9a13710 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.816+0000 7efee5c39740 1 -- 172.21.15.98:0/2287735317 >> 172.21.15.98:0/2287735317 conn(0x559cd9a0f650 msgr2=0x559cd9a0fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.816+0000 7efee5c39740 1 -- 172.21.15.98:0/2287735317 shutdown_connections 2024-05-09T09:00:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.816+0000 7efee5c39740 1 -- 172.21.15.98:0/2287735317 wait complete. 2024-05-09T09:00:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.816+0000 7efee5c39740 1 -- start start 2024-05-09T09:00:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.816+0000 7efee5c39740 1 Processor -- start 2024-05-09T09:00:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee5c39740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13330 0x559cd9a1b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee5c39740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13c50 0x559cd9a1b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee5c39740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a1e320 0x559cd9a1bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee5c39740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559cd9a21110 con 0x559cd9a13c50 2024-05-09T09:00:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee5c39740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559cd9933c40 con 0x559cd9a13330 2024-05-09T09:00:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee5c39740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559cd9a21290 con 0x559cd9a1e320 2024-05-09T09:00:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee5c39740 10 client.0 _ll_register_callbacks cb 0x559cd968bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee0be9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13c50 0x559cd9a1b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee0be9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13c50 0x559cd9a1b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35240/0 (socket says 172.21.15.98:35240) 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee0be9640 1 -- 172.21.15.98:0/3375285505 learned_addr learned my addr 172.21.15.98:0/3375285505 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efedbfff640 1 --2- 172.21.15.98:0/3375285505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13330 0x559cd9a1b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efedbfff640 1 --2- 172.21.15.98:0/3375285505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a1e320 0x559cd9a1bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee0be9640 1 -- 172.21.15.98:0/3375285505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a1e320 msgr2=0x559cd9a1bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee0be9640 1 --2- 172.21.15.98:0/3375285505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a1e320 0x559cd9a1bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee0be9640 1 -- 172.21.15.98:0/3375285505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13330 msgr2=0x559cd9a1b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee0be9640 1 --2- 172.21.15.98:0/3375285505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13330 0x559cd9a1b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee0be9640 1 -- 172.21.15.98:0/3375285505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cd97567d0 con 0x559cd9a13c50 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efedbfff640 1 --2- 172.21.15.98:0/3375285505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13330 0x559cd9a1b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee0be9640 1 --2- 172.21.15.98:0/3375285505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13c50 0x559cd9a1b7c0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7efed002a940 tx=0x7efed0031850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6d57d5818f191 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efebbfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efebbfff640 1 -- 172.21.15.98:0/3375285505 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed0035040 con 0x559cd9a13c50 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efebbfff640 1 -- 172.21.15.98:0/3375285505 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7efed0004460 con 0x559cd9a13c50 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.817+0000 7efee5c39740 1 -- 172.21.15.98:0/3375285505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efec801b070 con 0x559cd9a13c50 2024-05-09T09:00:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.818+0000 7efebbfff640 1 -- 172.21.15.98:0/3375285505 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7efed0005030 con 0x559cd9a13c50 2024-05-09T09:00:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.818+0000 7efee5c39740 10 client.18423 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.818+0000 7efee5c39740 1 -- 172.21.15.98:0/3375285505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559cd9a2d2e0 con 0x559cd9a13c50 2024-05-09T09:00:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.818+0000 7efebbfff640 1 -- 172.21.15.98:0/3375285505 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efec801b070 con 0x559cd9a13c50 2024-05-09T09:00:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.818+0000 7efebbfff640 1 -- 172.21.15.98:0/3375285505 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7efed00035f0 con 0x559cd9a13c50 2024-05-09T09:00:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.818+0000 7efee5c39740 10 client.18423 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.818+0000 7efee5c39740 10 client.18423 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:00:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.818+0000 7efee5c39740 20 client.18423 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.818+0000 7efee5c39740 1 --2- 172.21.15.98:0/3375285505 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559cd9a2e7a0 0x559cd9a30b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.818+0000 7efee5c39740 4 client.18423 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.818+0000 7efee5c39740 1 -- 172.21.15.98:0/3375285505 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559cd9986370 con 0x559cd9a2e7a0 2024-05-09T09:00:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.818+0000 7efedbfff640 1 --2- 172.21.15.98:0/3375285505 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559cd9a2e7a0 0x559cd9a30b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.819+0000 7efedbfff640 1 --2- 172.21.15.98:0/3375285505 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559cd9a2e7a0 0x559cd9a30b90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=963746f03549f8a6 server_cookie=cf78a20eb3cfa8fe in_seq=0 out_seq=0 2024-05-09T09:00:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.819+0000 7efebbfff640 10 client.18423 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T09:00:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.819+0000 7efebbfff640 1 -- 172.21.15.98:0/3375285505 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559cd9986370 con 0x559cd9a2e7a0 2024-05-09T09:00:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.819+0000 7efebbfff640 10 client.18423 handle_command_reply: tid=0 2024-05-09T09:00:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.819+0000 7efee5c39740 1 client.18423 shutdown 2024-05-09T09:00:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.819+0000 7efee5c39740 1 -- 172.21.15.98:0/3375285505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13c50 msgr2=0x559cd9a1b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.819+0000 7efee5c39740 1 --2- 172.21.15.98:0/3375285505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13c50 0x559cd9a1b7c0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7efed002a940 tx=0x7efed0031850 comp rx=0 tx=0).stop 2024-05-09T09:00:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.819+0000 7efee5c39740 1 -- 172.21.15.98:0/3375285505 shutdown_connections 2024-05-09T09:00:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.819+0000 7efee5c39740 1 --2- 172.21.15.98:0/3375285505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a1e320 0x559cd9a1bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.819+0000 7efee5c39740 1 --2- 172.21.15.98:0/3375285505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13c50 0x559cd9a1b7c0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.819+0000 7efee5c39740 1 --2- 172.21.15.98:0/3375285505 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559cd9a2e7a0 0x559cd9a30b90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.820+0000 7efee5c39740 1 --2- 172.21.15.98:0/3375285505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13330 0x559cd9a1b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.820+0000 7efee5c39740 1 -- 172.21.15.98:0/3375285505 >> 172.21.15.98:0/3375285505 conn(0x559cd9a0f650 msgr2=0x559cd9a15350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.820+0000 7efebbfff640 0 client.18423 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T09:00:22.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.820+0000 7efee5c39740 1 -- 172.21.15.98:0/3375285505 shutdown_connections 2024-05-09T09:00:22.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.820+0000 7efee5c39740 1 -- 172.21.15.98:0/3375285505 wait complete. 2024-05-09T09:00:22.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.820+0000 7efee5c39740 20 client.18423 trim_cache size 0 max 16384 2024-05-09T09:00:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.833+0000 7efee5c39740 1 Processor -- start 2024-05-09T09:00:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.833+0000 7efee5c39740 1 -- start start 2024-05-09T09:00:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.833+0000 7efee5c39740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13330 0x559cd9a0ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.833+0000 7efee5c39740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a13c50 0x559cd9a29fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.833+0000 7efee5c39740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a1e320 0x559cd9a36460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.833+0000 7efee5c39740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efedc1aef60 con 0x559cd9a1e320 2024-05-09T09:00:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.833+0000 7efee5c39740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efedc1b0aa0 con 0x559cd9a13330 2024-05-09T09:00:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.833+0000 7efee5c39740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efedc1af630 con 0x559cd9a13c50 2024-05-09T09:00:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.834+0000 7efedbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a1e320 0x559cd9a36460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.834+0000 7efedbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a1e320 0x559cd9a36460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35256/0 (socket says 172.21.15.98:35256) 2024-05-09T09:00:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.834+0000 7efedbfff640 1 -- 172.21.15.98:0/368659823 learned_addr learned my addr 172.21.15.98:0/368659823 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.834+0000 7efee0be9640 1 --2- 172.21.15.98:0/368659823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a13c50 0x559cd9a29fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.834+0000 7efedbfff640 1 --2- 172.21.15.98:0/368659823 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13330 0x559cd9a0ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.834+0000 7efedbfff640 1 -- 172.21.15.98:0/368659823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a13c50 msgr2=0x559cd9a29fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.834+0000 7efedbfff640 1 --2- 172.21.15.98:0/368659823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a13c50 0x559cd9a29fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.834+0000 7efedbfff640 1 -- 172.21.15.98:0/368659823 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13330 msgr2=0x559cd9a0ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.834+0000 7efedbfff640 1 --2- 172.21.15.98:0/368659823 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13330 0x559cd9a0ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.834+0000 7efedbfff640 1 -- 172.21.15.98:0/368659823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cd9a00d20 con 0x559cd9a1e320 2024-05-09T09:00:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.834+0000 7efedbfff640 1 --2- 172.21.15.98:0/368659823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a1e320 0x559cd9a36460 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7efec8015530 tx=0x7efec8023040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2837597c72cb400a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.834+0000 7efec67fc640 1 -- 172.21.15.98:0/368659823 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec8032020 con 0x559cd9a1e320 2024-05-09T09:00:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.834+0000 7efec67fc640 1 -- 172.21.15.98:0/368659823 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efec801b070 con 0x559cd9a1e320 2024-05-09T09:00:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.834+0000 7efec67fc640 1 -- 172.21.15.98:0/368659823 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec8005290 con 0x559cd9a1e320 2024-05-09T09:00:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.835+0000 7efee5c39740 1 -- 172.21.15.98:0/368659823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a1e320 msgr2=0x559cd9a36460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.835+0000 7efee5c39740 1 --2- 172.21.15.98:0/368659823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a1e320 0x559cd9a36460 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7efec8015530 tx=0x7efec8023040 comp rx=0 tx=0).stop 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.835+0000 7efee5c39740 1 -- 172.21.15.98:0/368659823 shutdown_connections 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.835+0000 7efee5c39740 1 --2- 172.21.15.98:0/368659823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a1e320 0x559cd9a36460 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.835+0000 7efee5c39740 1 --2- 172.21.15.98:0/368659823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a13c50 0x559cd9a29fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.835+0000 7efee5c39740 1 --2- 172.21.15.98:0/368659823 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a13330 0x559cd9a0ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.835+0000 7efee5c39740 1 -- 172.21.15.98:0/368659823 >> 172.21.15.98:0/368659823 conn(0x559cd9a0f650 msgr2=0x559cd9a1d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.835+0000 7efee5c39740 1 -- 172.21.15.98:0/368659823 shutdown_connections 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.835+0000 7efee5c39740 1 -- 172.21.15.98:0/368659823 wait complete. 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.835+0000 7efee5c39740 1 -- start start 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.835+0000 7efee5c39740 1 Processor -- start 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee5c39740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a13330 0x559cd9a22ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee5c39740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13c50 0x559cd9a22ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee5c39740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a1e320 0x559cd9a8cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee5c39740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efedc1b03d0 con 0x559cd9a13c50 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee5c39740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efedc1afd00 con 0x559cd9a1e320 2024-05-09T09:00:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee5c39740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efedc1acf60 con 0x559cd9a13330 2024-05-09T09:00:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee5c39740 10 client.0 _ll_register_callbacks cb 0x7efed00314d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee0be9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13c50 0x559cd9a22ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee0be9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13c50 0x559cd9a22ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35258/0 (socket says 172.21.15.98:35258) 2024-05-09T09:00:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee0be9640 1 -- 172.21.15.98:0/3254317631 learned_addr learned my addr 172.21.15.98:0/3254317631 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efedbfff640 1 --2- 172.21.15.98:0/3254317631 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a1e320 0x559cd9a8cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efedbfff640 1 --2- 172.21.15.98:0/3254317631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a13330 0x559cd9a22ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee0be9640 1 -- 172.21.15.98:0/3254317631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a13330 msgr2=0x559cd9a22ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee0be9640 1 --2- 172.21.15.98:0/3254317631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a13330 0x559cd9a22ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee0be9640 1 -- 172.21.15.98:0/3254317631 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a1e320 msgr2=0x559cd9a8cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee0be9640 1 --2- 172.21.15.98:0/3254317631 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a1e320 0x559cd9a8cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee0be9640 1 -- 172.21.15.98:0/3254317631 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cd9a8d470 con 0x559cd9a13c50 2024-05-09T09:00:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efee0be9640 1 --2- 172.21.15.98:0/3254317631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13c50 0x559cd9a22ff0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7efed0009850 tx=0x7efed0003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=180469e1b3759b9e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.836+0000 7efebbfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efebbfff640 1 -- 172.21.15.98:0/3254317631 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed0035040 con 0x559cd9a13c50 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efebbfff640 1 -- 172.21.15.98:0/3254317631 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7efed0031b00 con 0x559cd9a13c50 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efee5c39740 1 -- 172.21.15.98:0/3254317631 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efec801b070 con 0x559cd9a13c50 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efebbfff640 1 -- 172.21.15.98:0/3254317631 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7efed0060050 con 0x559cd9a13c50 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efee5c39740 10 client.18429 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efee5c39740 1 -- 172.21.15.98:0/3254317631 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559cd9a95800 con 0x559cd9a13c50 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efebbfff640 1 -- 172.21.15.98:0/3254317631 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efec801b070 con 0x559cd9a13c50 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efebbfff640 1 -- 172.21.15.98:0/3254317631 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7efed00049f0 con 0x559cd9a13c50 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efee5c39740 10 client.18429 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efee5c39740 10 client.18429 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efee5c39740 20 client.18429 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efee5c39740 1 --2- 172.21.15.98:0/3254317631 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559cd9a96000 0x559cd9a98420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efee5c39740 4 client.18429 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efee5c39740 1 -- 172.21.15.98:0/3254317631 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559cd99806c0 con 0x559cd9a96000 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.837+0000 7efedbfff640 1 --2- 172.21.15.98:0/3254317631 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559cd9a96000 0x559cd9a98420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efebbfff640 10 client.18429 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T09:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efedbfff640 1 --2- 172.21.15.98:0/3254317631 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559cd9a96000 0x559cd9a98420 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=29320bf38495bc71 server_cookie=79626a1df2c1cb6d in_seq=0 out_seq=0 2024-05-09T09:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efebbfff640 1 -- 172.21.15.98:0/3254317631 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559cd99806c0 con 0x559cd9a96000 2024-05-09T09:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efebbfff640 10 client.18429 handle_command_reply: tid=0 2024-05-09T09:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efee5c39740 1 client.18429 shutdown 2024-05-09T09:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efee5c39740 1 -- 172.21.15.98:0/3254317631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13c50 msgr2=0x559cd9a22ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efee5c39740 1 --2- 172.21.15.98:0/3254317631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13c50 0x559cd9a22ff0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7efed0009850 tx=0x7efed0003e30 comp rx=0 tx=0).stop 2024-05-09T09:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efee5c39740 1 -- 172.21.15.98:0/3254317631 shutdown_connections 2024-05-09T09:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efee5c39740 1 --2- 172.21.15.98:0/3254317631 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559cd9a96000 0x559cd9a98420 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efee5c39740 1 --2- 172.21.15.98:0/3254317631 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cd9a1e320 0x559cd9a8cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efee5c39740 1 --2- 172.21.15.98:0/3254317631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559cd9a13c50 0x559cd9a22ff0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efee5c39740 1 --2- 172.21.15.98:0/3254317631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559cd9a13330 0x559cd9a22ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efebbfff640 0 client.18429 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T09:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efee5c39740 1 -- 172.21.15.98:0/3254317631 >> 172.21.15.98:0/3254317631 conn(0x559cd9a0f650 msgr2=0x559cd9a1d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efee5c39740 1 -- 172.21.15.98:0/3254317631 shutdown_connections 2024-05-09T09:00:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efee5c39740 1 -- 172.21.15.98:0/3254317631 wait complete. 2024-05-09T09:00:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.838+0000 7efee5c39740 20 client.18429 trim_cache size 0 max 16384 2024-05-09T09:00:22.847 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:00:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.839+0000 7efee2e4d640 1 -- 172.21.15.98:0/3960864869 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efecc0a2d00 msgr2=0x7efecc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.839+0000 7efee2e4d640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efecc0a2d00 0x7efecc0a51c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efed4098060 tx=0x7efed4097800 comp rx=0 tx=0).stop 2024-05-09T09:00:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.839+0000 7efee2e4d640 1 -- 172.21.15.98:0/3960864869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efedc106710 msgr2=0x7efedc19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.839+0000 7efee2e4d640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efedc106710 0x7efedc19ca40 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7efed400eaa0 tx=0x7efed400ef60 comp rx=0 tx=0).stop 2024-05-09T09:00:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.839+0000 7efee2e4d640 1 -- 172.21.15.98:0/3960864869 shutdown_connections 2024-05-09T09:00:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.839+0000 7efee2e4d640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efecc0a2d00 0x7efecc0a51c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.839+0000 7efee2e4d640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efedc106710 0x7efedc19ca40 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.839+0000 7efee2e4d640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc1010c0 0x7efedc1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.839+0000 7efee2e4d640 1 --2- 172.21.15.98:0/3960864869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efedc100700 0x7efedc198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.839+0000 7efee2e4d640 1 -- 172.21.15.98:0/3960864869 >> 172.21.15.98:0/3960864869 conn(0x7efedc077830 msgr2=0x7efedc068da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.839+0000 7efee2e4d640 1 -- 172.21.15.98:0/3960864869 shutdown_connections 2024-05-09T09:00:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.839+0000 7efee2e4d640 1 -- 172.21.15.98:0/3960864869 wait complete. 2024-05-09T09:00:22.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T09:00:22.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.957+0000 7f11d842a640 1 -- 172.21.15.98:0/2064254583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d01010c0 msgr2=0x7f11d010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.957+0000 7f11d842a640 1 --2- 172.21.15.98:0/2064254583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d01010c0 0x7f11d010d5d0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f11cc009930 tx=0x7f11cc02a420 comp rx=0 tx=0).stop 2024-05-09T09:00:22.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.957+0000 7f11d842a640 1 -- 172.21.15.98:0/2064254583 shutdown_connections 2024-05-09T09:00:22.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.957+0000 7f11d842a640 1 --2- 172.21.15.98:0/2064254583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d01010c0 0x7f11d010d5d0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.957+0000 7f11d842a640 1 --2- 172.21.15.98:0/2064254583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11d0100700 0x7f11d0100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.957+0000 7f11d842a640 1 --2- 172.21.15.98:0/2064254583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11d0106710 0x7f11d0106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.957+0000 7f11d842a640 1 -- 172.21.15.98:0/2064254583 >> 172.21.15.98:0/2064254583 conn(0x7f11d0077830 msgr2=0x7f11d0077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.957+0000 7f11d842a640 1 -- 172.21.15.98:0/2064254583 shutdown_connections 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.957+0000 7f11d842a640 1 -- 172.21.15.98:0/2064254583 wait complete. 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.957+0000 7f11d842a640 1 Processor -- start 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d842a640 1 -- start start 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d842a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d0100700 0x7f11d0198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d842a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11d01010c0 0x7f11d0199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d842a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11d0106710 0x7f11d01a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d842a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11d0110b20 con 0x7f11d0100700 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d842a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f11d01109a0 con 0x7f11d0106710 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d842a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f11d0110ca0 con 0x7f11d01010c0 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d61c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d0100700 0x7f11d0198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d61c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d0100700 0x7f11d0198ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35272/0 (socket says 172.21.15.98:35272) 2024-05-09T09:00:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d61c6640 1 -- 172.21.15.98:0/28933041 learned_addr learned my addr 172.21.15.98:0/28933041 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d69c7640 1 --2- 172.21.15.98:0/28933041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11d0106710 0x7f11d01a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d61c6640 1 -- 172.21.15.98:0/28933041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11d01010c0 msgr2=0x7f11d0199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d61c6640 1 --2- 172.21.15.98:0/28933041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11d01010c0 0x7f11d0199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d61c6640 1 -- 172.21.15.98:0/28933041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11d0106710 msgr2=0x7f11d01a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d61c6640 1 --2- 172.21.15.98:0/28933041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11d0106710 0x7f11d01a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d61c6640 1 -- 172.21.15.98:0/28933041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d01a44a0 con 0x7f11d0100700 2024-05-09T09:00:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d69c7640 1 --2- 172.21.15.98:0/28933041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11d0106710 0x7f11d01a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d61c6640 1 --2- 172.21.15.98:0/28933041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d0100700 0x7f11d0198ee0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f11c800db80 tx=0x7f11c800b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11c77fe640 1 -- 172.21.15.98:0/28933041 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c8017070 con 0x7f11d0100700 2024-05-09T09:00:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11d842a640 1 -- 172.21.15.98:0/28933041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d01a4670 con 0x7f11d0100700 2024-05-09T09:00:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11c77fe640 1 -- 172.21.15.98:0/28933041 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c8003b70 con 0x7f11d0100700 2024-05-09T09:00:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.958+0000 7f11c77fe640 1 -- 172.21.15.98:0/28933041 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c80106c0 con 0x7f11d0100700 2024-05-09T09:00:22.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.959+0000 7f11d842a640 1 -- 172.21.15.98:0/28933041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d01a4b90 con 0x7f11d0100700 2024-05-09T09:00:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.961+0000 7f11c77fe640 1 -- 172.21.15.98:0/28933041 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f11c8004020 con 0x7f11d0100700 2024-05-09T09:00:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.961+0000 7f11db214740 1 Processor -- start 2024-05-09T09:00:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11db214740 1 -- start start 2024-05-09T09:00:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11db214740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24e7330 0x55edd24e7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11db214740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11db214740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24f2320 0x55edd24f4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11db214740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55edd244cc40 con 0x55edd24e7c50 2024-05-09T09:00:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11db214740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55edd23f4580 con 0x55edd24e7330 2024-05-09T09:00:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11db214740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55edd2448c00 con 0x55edd24f2320 2024-05-09T09:00:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11c77fe640 1 --2- 172.21.15.98:0/28933041 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11bc0a2d00 0x7f11bc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11d61c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11d69c7640 1 --2- 172.21.15.98:0/28933041 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11bc0a2d00 0x7f11bc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11d61c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24f1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35288/0 (socket says 172.21.15.98:35288) 2024-05-09T09:00:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11d61c6640 1 -- 172.21.15.98:0/1045886191 learned_addr learned my addr 172.21.15.98:0/1045886191 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11d59c5640 1 --2- 172.21.15.98:0/1045886191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24f2320 0x55edd24f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11d69c7640 1 --2- 172.21.15.98:0/1045886191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24e7330 0x55edd24e7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11c77fe640 1 -- 172.21.15.98:0/28933041 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f11c80cf6d0 con 0x7f11d0100700 2024-05-09T09:00:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11d61c6640 1 -- 172.21.15.98:0/1045886191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24f2320 msgr2=0x55edd24f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11d61c6640 1 --2- 172.21.15.98:0/1045886191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24f2320 0x55edd24f4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11d61c6640 1 -- 172.21.15.98:0/1045886191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24e7330 msgr2=0x55edd24e7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11d61c6640 1 --2- 172.21.15.98:0/1045886191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24e7330 0x55edd24e7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11d61c6640 1 -- 172.21.15.98:0/1045886191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edd21603f0 con 0x55edd24e7c50 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.962+0000 7f11d59c5640 1 --2- 172.21.15.98:0/1045886191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24f2320 0x55edd24f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.963+0000 7f11d69c7640 1 --2- 172.21.15.98:0/28933041 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11bc0a2d00 0x7f11bc0a51c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f11cc031550 tx=0x7f11cc032ea0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.963+0000 7f11d61c6640 1 --2- 172.21.15.98:0/1045886191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24f1de0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f11c800bb30 tx=0x7f11c806e7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65cbc1f8b163e09 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.963+0000 7f11c4bfe640 1 -- 172.21.15.98:0/1045886191 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c809be80 con 0x55edd24e7c50 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.963+0000 7f11c4bfe640 1 -- 172.21.15.98:0/1045886191 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c809fb30 con 0x55edd24e7c50 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.963+0000 7f11c4bfe640 1 -- 172.21.15.98:0/1045886191 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c8099b10 con 0x55edd24e7c50 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.963+0000 7f11db214740 1 -- 172.21.15.98:0/1045886191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 msgr2=0x55edd24f1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.963+0000 7f11db214740 1 --2- 172.21.15.98:0/1045886191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24f1de0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f11c800bb30 tx=0x7f11c806e7f0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 -- 172.21.15.98:0/1045886191 shutdown_connections 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 --2- 172.21.15.98:0/1045886191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24f2320 0x55edd24f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 --2- 172.21.15.98:0/1045886191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24f1de0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 --2- 172.21.15.98:0/1045886191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24e7330 0x55edd24e7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 -- 172.21.15.98:0/1045886191 >> 172.21.15.98:0/1045886191 conn(0x55edd24e3650 msgr2=0x55edd24e3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 -- 172.21.15.98:0/1045886191 shutdown_connections 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 -- 172.21.15.98:0/1045886191 wait complete. 2024-05-09T09:00:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 -- start start 2024-05-09T09:00:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 Processor -- start 2024-05-09T09:00:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24e7330 0x55edd24ef280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24ef7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24f2320 0x55edd24efd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55edd24f5110 con 0x55edd24e7c50 2024-05-09T09:00:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55edd2407c40 con 0x55edd24f2320 2024-05-09T09:00:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55edd24f5290 con 0x55edd24e7330 2024-05-09T09:00:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11db214740 10 client.0 _ll_register_callbacks cb 0x55edd215fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11d59c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24ef7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11d59c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24ef7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35304/0 (socket says 172.21.15.98:35304) 2024-05-09T09:00:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.964+0000 7f11d59c5640 1 -- 172.21.15.98:0/1709953458 learned_addr learned my addr 172.21.15.98:0/1709953458 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11d61c6640 1 --2- 172.21.15.98:0/1709953458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24e7330 0x55edd24ef280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11d69c7640 1 --2- 172.21.15.98:0/1709953458 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24f2320 0x55edd24efd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11d59c5640 1 -- 172.21.15.98:0/1709953458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24e7330 msgr2=0x55edd24ef280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11d59c5640 1 --2- 172.21.15.98:0/1709953458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24e7330 0x55edd24ef280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11d59c5640 1 -- 172.21.15.98:0/1709953458 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24f2320 msgr2=0x55edd24efd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11d59c5640 1 --2- 172.21.15.98:0/1709953458 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24f2320 0x55edd24efd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11d59c5640 1 -- 172.21.15.98:0/1709953458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edd222a7d0 con 0x55edd24e7c50 2024-05-09T09:00:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11d59c5640 1 --2- 172.21.15.98:0/1709953458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24ef7c0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f11c000f670 tx=0x7f11c000fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d477290c64919aa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11b9ffb640 1 -- 172.21.15.98:0/1709953458 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c001d060 con 0x55edd24e7c50 2024-05-09T09:00:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11b9ffb640 1 -- 172.21.15.98:0/1709953458 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f11c000db20 con 0x55edd24e7c50 2024-05-09T09:00:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11db214740 1 -- 172.21.15.98:0/1709953458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11c809fb30 con 0x55edd24e7c50 2024-05-09T09:00:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11b9ffb640 1 -- 172.21.15.98:0/1709953458 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f11c0024050 con 0x55edd24e7c50 2024-05-09T09:00:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11db214740 10 client.18441 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11db214740 1 -- 172.21.15.98:0/1709953458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55edd2501340 con 0x55edd24e7c50 2024-05-09T09:00:22.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.965+0000 7f11b9ffb640 1 -- 172.21.15.98:0/1709953458 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c809fb30 con 0x55edd24e7c50 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.966+0000 7f11b9ffb640 1 -- 172.21.15.98:0/1709953458 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f11c0016340 con 0x55edd24e7c50 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.966+0000 7f11db214740 10 client.18441 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.966+0000 7f11db214740 10 client.18441 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.966+0000 7f11db214740 20 client.18441 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.966+0000 7f11db214740 1 --2- 172.21.15.98:0/1709953458 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55edd2502850 0x55edd2504c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.966+0000 7f11db214740 4 client.18441 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.966+0000 7f11db214740 1 -- 172.21.15.98:0/1709953458 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edd245a370 con 0x55edd2502850 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.966+0000 7f11d61c6640 1 --2- 172.21.15.98:0/1709953458 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55edd2502850 0x55edd2504c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.966+0000 7f11b9ffb640 10 client.18441 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.966+0000 7f11d61c6640 1 --2- 172.21.15.98:0/1709953458 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55edd2502850 0x55edd2504c60 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57f91e246870c4e3 server_cookie=b8b8ca1f74db5646 in_seq=0 out_seq=0 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.967+0000 7f11b9ffb640 1 -- 172.21.15.98:0/1709953458 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55edd245a370 con 0x55edd2502850 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.967+0000 7f11b9ffb640 10 client.18441 handle_command_reply: tid=0 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.967+0000 7f11db214740 1 client.18441 shutdown 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.967+0000 7f11db214740 1 -- 172.21.15.98:0/1709953458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 msgr2=0x55edd24ef7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.967+0000 7f11db214740 1 --2- 172.21.15.98:0/1709953458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24ef7c0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f11c000f670 tx=0x7f11c000fb30 comp rx=0 tx=0).stop 2024-05-09T09:00:22.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.967+0000 7f11db214740 1 -- 172.21.15.98:0/1709953458 shutdown_connections 2024-05-09T09:00:22.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.967+0000 7f11db214740 1 --2- 172.21.15.98:0/1709953458 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55edd2502850 0x55edd2504c60 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.967+0000 7f11db214740 1 --2- 172.21.15.98:0/1709953458 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24f2320 0x55edd24efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.967+0000 7f11b9ffb640 0 client.18441 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:22.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.967+0000 7f11db214740 1 --2- 172.21.15.98:0/1709953458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24ef7c0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.967+0000 7f11db214740 1 --2- 172.21.15.98:0/1709953458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24e7330 0x55edd24ef280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.967+0000 7f11db214740 1 -- 172.21.15.98:0/1709953458 >> 172.21.15.98:0/1709953458 conn(0x55edd24e3650 msgr2=0x55edd24e9350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.967+0000 7f11db214740 1 -- 172.21.15.98:0/1709953458 shutdown_connections 2024-05-09T09:00:22.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.968+0000 7f11db214740 1 -- 172.21.15.98:0/1709953458 wait complete. 2024-05-09T09:00:22.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.968+0000 7f11db214740 20 client.18441 trim_cache size 0 max 16384 2024-05-09T09:00:22.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11db214740 1 Processor -- start 2024-05-09T09:00:22.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11db214740 1 -- start start 2024-05-09T09:00:22.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11db214740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24e7330 0x55edd24f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11db214740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24fdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11db214740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24f2320 0x55edd24e9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11db214740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11d01acc90 con 0x55edd24e7c50 2024-05-09T09:00:22.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11db214740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f11d01ae7d0 con 0x55edd24e7330 2024-05-09T09:00:22.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11db214740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f11d01ad360 con 0x55edd24f2320 2024-05-09T09:00:22.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11d59c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24fdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11d59c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24fdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35320/0 (socket says 172.21.15.98:35320) 2024-05-09T09:00:22.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11d59c5640 1 -- 172.21.15.98:0/2275023934 learned_addr learned my addr 172.21.15.98:0/2275023934 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11d69c7640 1 --2- 172.21.15.98:0/2275023934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24f2320 0x55edd24e9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11d61c6640 1 --2- 172.21.15.98:0/2275023934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24e7330 0x55edd24f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11d59c5640 1 -- 172.21.15.98:0/2275023934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24f2320 msgr2=0x55edd24e9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11d59c5640 1 --2- 172.21.15.98:0/2275023934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24f2320 0x55edd24e9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11d59c5640 1 -- 172.21.15.98:0/2275023934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24e7330 msgr2=0x55edd24f19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11d59c5640 1 --2- 172.21.15.98:0/2275023934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24e7330 0x55edd24f19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11d59c5640 1 -- 172.21.15.98:0/2275023934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edd24ed7d0 con 0x55edd24e7c50 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.981+0000 7f11d59c5640 1 --2- 172.21.15.98:0/2275023934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24fdfd0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f11c0001eb0 tx=0x7f11c0005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644674c40a1fbbc2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.982+0000 7f11bbfff640 1 -- 172.21.15.98:0/2275023934 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c002c020 con 0x55edd24e7c50 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.982+0000 7f11bbfff640 1 -- 172.21.15.98:0/2275023934 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c000ae70 con 0x55edd24e7c50 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.982+0000 7f11bbfff640 1 -- 172.21.15.98:0/2275023934 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0005040 con 0x55edd24e7c50 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.982+0000 7f11db214740 1 -- 172.21.15.98:0/2275023934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 msgr2=0x55edd24fdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.982+0000 7f11db214740 1 --2- 172.21.15.98:0/2275023934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24fdfd0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f11c0001eb0 tx=0x7f11c0005b90 comp rx=0 tx=0).stop 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.982+0000 7f11db214740 1 -- 172.21.15.98:0/2275023934 shutdown_connections 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.982+0000 7f11db214740 1 --2- 172.21.15.98:0/2275023934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24f2320 0x55edd24e9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.982+0000 7f11db214740 1 --2- 172.21.15.98:0/2275023934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7c50 0x55edd24fdfd0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.982+0000 7f11db214740 1 --2- 172.21.15.98:0/2275023934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24e7330 0x55edd24f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.982+0000 7f11db214740 1 -- 172.21.15.98:0/2275023934 >> 172.21.15.98:0/2275023934 conn(0x55edd24e3650 msgr2=0x55edd24e3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.982+0000 7f11db214740 1 -- 172.21.15.98:0/2275023934 shutdown_connections 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.982+0000 7f11db214740 1 -- 172.21.15.98:0/2275023934 wait complete. 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11db214740 1 -- start start 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11db214740 1 Processor -- start 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11db214740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7330 0x55edd24f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11db214740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24e7c50 0x55edd24f6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11db214740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24f2320 0x55edd24f69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11db214740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11d01ae100 con 0x55edd24e7330 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11db214740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f11d01ada30 con 0x55edd24f2320 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11db214740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f11d01aac90 con 0x55edd24e7c50 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11db214740 10 client.0 _ll_register_callbacks cb 0x7f11c000a570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11d61c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7330 0x55edd24f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11d61c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7330 0x55edd24f5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35322/0 (socket says 172.21.15.98:35322) 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11d61c6640 1 -- 172.21.15.98:0/826064034 learned_addr learned my addr 172.21.15.98:0/826064034 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11d59c5640 1 --2- 172.21.15.98:0/826064034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24e7c50 0x55edd24f6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11d69c7640 1 --2- 172.21.15.98:0/826064034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24f2320 0x55edd24f69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11d61c6640 1 -- 172.21.15.98:0/826064034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24e7c50 msgr2=0x55edd24f6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11d61c6640 1 --2- 172.21.15.98:0/826064034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24e7c50 0x55edd24f6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11d61c6640 1 -- 172.21.15.98:0/826064034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24f2320 msgr2=0x55edd24f69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11d61c6640 1 --2- 172.21.15.98:0/826064034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24f2320 0x55edd24f69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11d61c6640 1 -- 172.21.15.98:0/826064034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edd2567e30 con 0x55edd24e7330 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.983+0000 7f11d69c7640 1 --2- 172.21.15.98:0/826064034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24f2320 0x55edd24f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11d61c6640 1 --2- 172.21.15.98:0/826064034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7330 0x55edd24f5f50 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f11c8098150 tx=0x7f11c800df10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc768bb0e3c366cb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11b9ffb640 1 -- 172.21.15.98:0/826064034 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c809be80 con 0x55edd24e7330 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11b9ffb640 1 -- 172.21.15.98:0/826064034 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f11c8004310 con 0x55edd24e7330 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11db214740 1 -- 172.21.15.98:0/826064034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11c000ae70 con 0x55edd24e7330 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11b9ffb640 1 -- 172.21.15.98:0/826064034 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f11c809b780 con 0x55edd24e7330 2024-05-09T09:00:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11db214740 10 client.18447 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11db214740 1 -- 172.21.15.98:0/826064034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55edd2569130 con 0x55edd24e7330 2024-05-09T09:00:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11b9ffb640 1 -- 172.21.15.98:0/826064034 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c000ae70 con 0x55edd24e7330 2024-05-09T09:00:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11b9ffb640 1 -- 172.21.15.98:0/826064034 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f11c8097040 con 0x55edd24e7330 2024-05-09T09:00:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11db214740 10 client.18447 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11db214740 10 client.18447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11db214740 20 client.18447 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11db214740 1 --2- 172.21.15.98:0/826064034 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55edd2569490 0x55edd256b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11db214740 4 client.18447 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:00:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.984+0000 7f11db214740 1 -- 172.21.15.98:0/826064034 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55edd24546c0 con 0x55edd2569490 2024-05-09T09:00:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11d59c5640 1 --2- 172.21.15.98:0/826064034 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55edd2569490 0x55edd256b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11b9ffb640 10 client.18447 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11d59c5640 1 --2- 172.21.15.98:0/826064034 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55edd2569490 0x55edd256b880 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbd895f5415b1967 server_cookie=57675b52a79d3f3f in_seq=0 out_seq=0 2024-05-09T09:00:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11b9ffb640 1 -- 172.21.15.98:0/826064034 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55edd24546c0 con 0x55edd2569490 2024-05-09T09:00:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11b9ffb640 10 client.18447 handle_command_reply: tid=0 2024-05-09T09:00:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11db214740 1 client.18447 shutdown 2024-05-09T09:00:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11db214740 1 -- 172.21.15.98:0/826064034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7330 msgr2=0x55edd24f5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11db214740 1 --2- 172.21.15.98:0/826064034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7330 0x55edd24f5f50 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f11c8098150 tx=0x7f11c800df10 comp rx=0 tx=0).stop 2024-05-09T09:00:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11db214740 1 -- 172.21.15.98:0/826064034 shutdown_connections 2024-05-09T09:00:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11db214740 1 --2- 172.21.15.98:0/826064034 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55edd2569490 0x55edd256b880 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11db214740 1 --2- 172.21.15.98:0/826064034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55edd24f2320 0x55edd24f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11b9ffb640 0 client.18447 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11db214740 1 --2- 172.21.15.98:0/826064034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55edd24e7c50 0x55edd24f6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11db214740 1 --2- 172.21.15.98:0/826064034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edd24e7330 0x55edd24f5f50 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11db214740 1 -- 172.21.15.98:0/826064034 >> 172.21.15.98:0/826064034 conn(0x55edd24e3650 msgr2=0x55edd24e1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.985+0000 7f11db214740 1 -- 172.21.15.98:0/826064034 shutdown_connections 2024-05-09T09:00:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.986+0000 7f11db214740 1 -- 172.21.15.98:0/826064034 wait complete. 2024-05-09T09:00:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.986+0000 7f11db214740 20 client.18447 trim_cache size 0 max 16384 2024-05-09T09:00:22.992 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:00:22.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.986+0000 7f11d842a640 1 -- 172.21.15.98:0/28933041 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11bc0a2d00 msgr2=0x7f11bc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.986+0000 7f11d842a640 1 --2- 172.21.15.98:0/28933041 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11bc0a2d00 0x7f11bc0a51c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f11cc031550 tx=0x7f11cc032ea0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.986+0000 7f11d842a640 1 -- 172.21.15.98:0/28933041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d0100700 msgr2=0x7f11d0198ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:22.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.986+0000 7f11d842a640 1 --2- 172.21.15.98:0/28933041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d0100700 0x7f11d0198ee0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f11c800db80 tx=0x7f11c800b630 comp rx=0 tx=0).stop 2024-05-09T09:00:22.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.986+0000 7f11d842a640 1 -- 172.21.15.98:0/28933041 shutdown_connections 2024-05-09T09:00:22.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.986+0000 7f11d842a640 1 --2- 172.21.15.98:0/28933041 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f11bc0a2d00 0x7f11bc0a51c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.986+0000 7f11d842a640 1 --2- 172.21.15.98:0/28933041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11d0106710 0x7f11d01a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.986+0000 7f11d842a640 1 --2- 172.21.15.98:0/28933041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11d01010c0 0x7f11d0199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.986+0000 7f11d842a640 1 --2- 172.21.15.98:0/28933041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11d0100700 0x7f11d0198ee0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:22.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.986+0000 7f11d842a640 1 -- 172.21.15.98:0/28933041 >> 172.21.15.98:0/28933041 conn(0x7f11d0077830 msgr2=0x7f11d0102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:22.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.986+0000 7f11d842a640 1 -- 172.21.15.98:0/28933041 shutdown_connections 2024-05-09T09:00:22.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:22.986+0000 7f11d842a640 1 -- 172.21.15.98:0/28933041 wait complete. 2024-05-09T09:00:23.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:23 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2029427446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:00:23.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:23 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2029427446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:00:23.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:23 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2029427446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:00:23.997 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T09:00:23.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:00:23.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T09:00:24.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.110+0000 7f0ed8f70640 1 -- 172.21.15.98:0/42734015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ed4069eb0 msgr2=0x7f0ed406a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.110+0000 7f0ed8f70640 1 --2- 172.21.15.98:0/42734015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ed4069eb0 0x7f0ed406a330 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009930 tx=0x7f0ebc02a420 comp rx=0 tx=0).stop 2024-05-09T09:00:24.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.110+0000 7f0ed8f70640 1 -- 172.21.15.98:0/42734015 shutdown_connections 2024-05-09T09:00:24.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.110+0000 7f0ed8f70640 1 --2- 172.21.15.98:0/42734015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ed406a870 0x7f0ed410e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.110+0000 7f0ed8f70640 1 --2- 172.21.15.98:0/42734015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ed4069eb0 0x7f0ed406a330 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.110+0000 7f0ed8f70640 1 --2- 172.21.15.98:0/42734015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ed4107820 0x7f0ed4107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.110+0000 7f0ed8f70640 1 -- 172.21.15.98:0/42734015 >> 172.21.15.98:0/42734015 conn(0x7f0ed4077790 msgr2=0x7f0ed4077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:24.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.110+0000 7f0ed8f70640 1 -- 172.21.15.98:0/42734015 shutdown_connections 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.110+0000 7f0ed8f70640 1 -- 172.21.15.98:0/42734015 wait complete. 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.111+0000 7f0ed8f70640 1 Processor -- start 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.111+0000 7f0ed8f70640 1 -- start start 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.111+0000 7f0ed8f70640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ed4069eb0 0x7f0ed4198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.111+0000 7f0ed8f70640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ed406a870 0x7f0ed41986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.111+0000 7f0ed8f70640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ed4107820 0x7f0ed419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.111+0000 7f0ed8f70640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed4111c50 con 0x7f0ed4069eb0 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.111+0000 7f0ed8f70640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed4111ad0 con 0x7f0ed406a870 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.111+0000 7f0ed8f70640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed4111dd0 con 0x7f0ed4107820 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.111+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ed4069eb0 0x7f0ed4198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.111+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ed4069eb0 0x7f0ed4198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35344/0 (socket says 172.21.15.98:35344) 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.111+0000 7f0ed259c640 1 -- 172.21.15.98:0/3520005851 learned_addr learned my addr 172.21.15.98:0/3520005851 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.111+0000 7f0ecbfff640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ed406a870 0x7f0ed41986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.111+0000 7f0ed2d9d640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ed4107820 0x7f0ed419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.112+0000 7f0ed259c640 1 -- 172.21.15.98:0/3520005851 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ed4107820 msgr2=0x7f0ed419ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.112+0000 7f0ed259c640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ed4107820 0x7f0ed419ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.112+0000 7f0ed259c640 1 -- 172.21.15.98:0/3520005851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ed406a870 msgr2=0x7f0ed41986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.112+0000 7f0ed259c640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ed406a870 0x7f0ed41986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.112+0000 7f0ed259c640 1 -- 172.21.15.98:0/3520005851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed419d180 con 0x7f0ed4069eb0 2024-05-09T09:00:24.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.112+0000 7f0ecbfff640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ed406a870 0x7f0ed41986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:24.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.112+0000 7f0ed2d9d640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ed4107820 0x7f0ed419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:24.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.112+0000 7f0ed259c640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ed4069eb0 0x7f0ed4198190 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f0ec400b4a0 tx=0x7f0ec400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:24.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.112+0000 7f0ecb7fe640 1 -- 172.21.15.98:0/3520005851 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4014070 con 0x7f0ed4069eb0 2024-05-09T09:00:24.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.112+0000 7f0ecb7fe640 1 -- 172.21.15.98:0/3520005851 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ec4010af0 con 0x7f0ed4069eb0 2024-05-09T09:00:24.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.112+0000 7f0ecb7fe640 1 -- 172.21.15.98:0/3520005851 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4017d10 con 0x7f0ed4069eb0 2024-05-09T09:00:24.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.112+0000 7f0ed8f70640 1 -- 172.21.15.98:0/3520005851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed419d470 con 0x7f0ed4069eb0 2024-05-09T09:00:24.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.112+0000 7f0ed8f70640 1 -- 172.21.15.98:0/3520005851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed41a4d00 con 0x7f0ed4069eb0 2024-05-09T09:00:24.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.114+0000 7f0ecb7fe640 1 -- 172.21.15.98:0/3520005851 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0ec4017070 con 0x7f0ed4069eb0 2024-05-09T09:00:24.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.114+0000 7f0edbc80740 1 Processor -- start 2024-05-09T09:00:24.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0edbc80740 1 -- start start 2024-05-09T09:00:24.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0edbc80740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a210913f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0edbc80740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210919c0 0x559a21091e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0edbc80740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a2109eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0ecb7fe640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0ec00a2d00 0x7f0ec00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0edbc80740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559a20ff6990 con 0x559a210930b0 2024-05-09T09:00:24.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0edbc80740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559a20f9e5b0 con 0x559a2109c740 2024-05-09T09:00:24.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0edbc80740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559a20ff2950 con 0x559a210919c0 2024-05-09T09:00:24.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0ed2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a210913f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0ed2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a210913f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35346/0 (socket says 172.21.15.98:35346) 2024-05-09T09:00:24.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0ed2d9d640 1 -- 172.21.15.98:0/4257502240 learned_addr learned my addr 172.21.15.98:0/4257502240 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:24.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0ed2d9d640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0ec00a2d00 0x7f0ec00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0ecb7fe640 1 -- 172.21.15.98:0/3520005851 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f0ec40ceb70 con 0x7f0ed4069eb0 2024-05-09T09:00:24.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0ed259c640 1 --2- 172.21.15.98:0/4257502240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210919c0 0x559a21091e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0ecbfff640 1 --2- 172.21.15.98:0/4257502240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a2109eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0ed2d9d640 1 -- 172.21.15.98:0/4257502240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210919c0 msgr2=0x559a21091e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0ed2d9d640 1 --2- 172.21.15.98:0/4257502240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210919c0 0x559a21091e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0ed2d9d640 1 -- 172.21.15.98:0/4257502240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 msgr2=0x559a2109eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0ed2d9d640 1 --2- 172.21.15.98:0/4257502240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a2109eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.115+0000 7f0ed2d9d640 1 -- 172.21.15.98:0/4257502240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a20dd27e0 con 0x559a210930b0 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.116+0000 7f0ed2d9d640 1 --2- 172.21.15.98:0/4257502240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a210913f0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc012da0 tx=0x7f0ecc00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db45df2423659f76 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.116+0000 7f0ed2d9d640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0ec00a2d00 0x7f0ec00a51c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00afd0 tx=0x7f0ecc01a000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.116+0000 7f0ec8ff9640 1 -- 172.21.15.98:0/4257502240 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc019cb0 con 0x559a210930b0 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.116+0000 7f0ec8ff9640 1 -- 172.21.15.98:0/4257502240 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ecc019e50 con 0x559a210930b0 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.116+0000 7f0ec8ff9640 1 -- 172.21.15.98:0/4257502240 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc01c670 con 0x559a210930b0 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 -- 172.21.15.98:0/4257502240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 msgr2=0x559a210913f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 --2- 172.21.15.98:0/4257502240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a210913f0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc012da0 tx=0x7f0ecc00f730 comp rx=0 tx=0).stop 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 -- 172.21.15.98:0/4257502240 shutdown_connections 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 --2- 172.21.15.98:0/4257502240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a2109eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 --2- 172.21.15.98:0/4257502240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210919c0 0x559a21091e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 --2- 172.21.15.98:0/4257502240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a210913f0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 -- 172.21.15.98:0/4257502240 >> 172.21.15.98:0/4257502240 conn(0x559a2105a0a0 msgr2=0x559a2108df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 -- 172.21.15.98:0/4257502240 shutdown_connections 2024-05-09T09:00:24.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 -- 172.21.15.98:0/4257502240 wait complete. 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 -- start start 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 Processor -- start 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210919c0 0x559a210996b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a21099bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a2109a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559a2109f570 con 0x559a210930b0 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559a20fb1b40 con 0x559a2109c740 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559a2109f6f0 con 0x559a210919c0 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0edbc80740 10 client.0 _ll_register_callbacks cb 0x559a20fa60f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0ecbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a21099bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0ecbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a21099bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35354/0 (socket says 172.21.15.98:35354) 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.117+0000 7f0ecbfff640 1 -- 172.21.15.98:0/336395716 learned_addr learned my addr 172.21.15.98:0/336395716 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0ed2d9d640 1 --2- 172.21.15.98:0/336395716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a2109a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0ecbfff640 1 -- 172.21.15.98:0/336395716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210919c0 msgr2=0x559a210996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0ecbfff640 1 --2- 172.21.15.98:0/336395716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210919c0 0x559a210996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0ecbfff640 1 -- 172.21.15.98:0/336395716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 msgr2=0x559a2109a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0ecbfff640 1 --2- 172.21.15.98:0/336395716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a2109a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0ecbfff640 1 -- 172.21.15.98:0/336395716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a20e32f70 con 0x559a210930b0 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0ed2d9d640 1 --2- 172.21.15.98:0/336395716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a2109a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0ecbfff640 1 --2- 172.21.15.98:0/336395716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a21099bf0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009900 tx=0x7f0ebc0317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb944a1abacc5c78 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0eb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0eb9ffb640 1 -- 172.21.15.98:0/336395716 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc040070 con 0x559a210930b0 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0eb9ffb640 1 -- 172.21.15.98:0/336395716 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f0ebc004250 con 0x559a210930b0 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0edbc80740 1 -- 172.21.15.98:0/336395716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ecc019e50 con 0x559a210930b0 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0eb9ffb640 1 -- 172.21.15.98:0/336395716 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0ebc035040 con 0x559a210930b0 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0edbc80740 10 client.18459 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0edbc80740 1 -- 172.21.15.98:0/336395716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559a20cca170 con 0x559a210930b0 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.118+0000 7f0eb9ffb640 1 -- 172.21.15.98:0/336395716 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ecc019e50 con 0x559a210930b0 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.119+0000 7f0eb9ffb640 1 -- 172.21.15.98:0/336395716 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f0ebc004a30 con 0x559a210930b0 2024-05-09T09:00:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.119+0000 7f0edbc80740 10 client.18459 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.119+0000 7f0edbc80740 10 client.18459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.119+0000 7f0edbc80740 20 client.18459 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.119+0000 7f0edbc80740 1 --2- 172.21.15.98:0/336395716 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559a210ac810 0x559a210aec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.119+0000 7f0edbc80740 4 client.18459 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.119+0000 7f0edbc80740 1 -- 172.21.15.98:0/336395716 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a210040c0 con 0x559a210ac810 2024-05-09T09:00:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.119+0000 7f0ed259c640 1 --2- 172.21.15.98:0/336395716 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559a210ac810 0x559a210aec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.119+0000 7f0ed259c640 1 --2- 172.21.15.98:0/336395716 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559a210ac810 0x559a210aec20 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45d878a5f651203a server_cookie=f3c7775189bafe40 in_seq=0 out_seq=0 2024-05-09T09:00:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.119+0000 7f0eb9ffb640 10 client.18459 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0eb9ffb640 1 -- 172.21.15.98:0/336395716 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559a210040c0 con 0x559a210ac810 2024-05-09T09:00:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0eb9ffb640 10 client.18459 handle_command_reply: tid=0 2024-05-09T09:00:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0edbc80740 1 client.18459 shutdown 2024-05-09T09:00:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0edbc80740 1 -- 172.21.15.98:0/336395716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 msgr2=0x559a21099bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0edbc80740 1 --2- 172.21.15.98:0/336395716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a21099bf0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009900 tx=0x7f0ebc0317e0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0edbc80740 1 -- 172.21.15.98:0/336395716 shutdown_connections 2024-05-09T09:00:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0edbc80740 1 --2- 172.21.15.98:0/336395716 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559a210ac810 0x559a210aec20 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0edbc80740 1 --2- 172.21.15.98:0/336395716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a2109a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0eb9ffb640 0 client.18459 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0edbc80740 1 --2- 172.21.15.98:0/336395716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a21099bf0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0edbc80740 1 --2- 172.21.15.98:0/336395716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210919c0 0x559a210996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0edbc80740 1 -- 172.21.15.98:0/336395716 >> 172.21.15.98:0/336395716 conn(0x559a2105a0a0 msgr2=0x559a21093d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0edbc80740 1 -- 172.21.15.98:0/336395716 shutdown_connections 2024-05-09T09:00:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0edbc80740 1 -- 172.21.15.98:0/336395716 wait complete. 2024-05-09T09:00:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.120+0000 7f0edbc80740 20 client.18459 trim_cache size 0 max 16384 2024-05-09T09:00:24.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0edbc80740 1 Processor -- start 2024-05-09T09:00:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0edbc80740 1 -- start start 2024-05-09T09:00:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0edbc80740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210919c0 0x559a210a8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0edbc80740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a2108cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0edbc80740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a210b4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0edbc80740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41af020 con 0x559a210930b0 2024-05-09T09:00:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0edbc80740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41b0b60 con 0x559a2109c740 2024-05-09T09:00:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0edbc80740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41af6f0 con 0x559a210919c0 2024-05-09T09:00:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0ecbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a2108cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0ecbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a2108cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35366/0 (socket says 172.21.15.98:35366) 2024-05-09T09:00:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0ecbfff640 1 -- 172.21.15.98:0/3998610807 learned_addr learned my addr 172.21.15.98:0/3998610807 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0ecbfff640 1 -- 172.21.15.98:0/3998610807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210919c0 msgr2=0x559a210a8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0ecbfff640 1 --2- 172.21.15.98:0/3998610807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210919c0 0x559a210a8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0ecbfff640 1 -- 172.21.15.98:0/3998610807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 msgr2=0x559a210b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0ecbfff640 1 --2- 172.21.15.98:0/3998610807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a210b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0ecbfff640 1 -- 172.21.15.98:0/3998610807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a2108e130 con 0x559a210930b0 2024-05-09T09:00:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0ecbfff640 1 --2- 172.21.15.98:0/3998610807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a2108cf00 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009930 tx=0x7f0ebc02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae4912b845b65126 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0ebbfff640 1 -- 172.21.15.98:0/3998610807 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc040070 con 0x559a210930b0 2024-05-09T09:00:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0ebbfff640 1 -- 172.21.15.98:0/3998610807 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ebc0035f0 con 0x559a210930b0 2024-05-09T09:00:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.138+0000 7f0ebbfff640 1 -- 172.21.15.98:0/3998610807 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc004a20 con 0x559a210930b0 2024-05-09T09:00:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.139+0000 7f0edbc80740 1 -- 172.21.15.98:0/3998610807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 msgr2=0x559a2108cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.139+0000 7f0edbc80740 1 --2- 172.21.15.98:0/3998610807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a2108cf00 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009930 tx=0x7f0ebc02a420 comp rx=0 tx=0).stop 2024-05-09T09:00:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.139+0000 7f0edbc80740 1 -- 172.21.15.98:0/3998610807 shutdown_connections 2024-05-09T09:00:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.139+0000 7f0edbc80740 1 --2- 172.21.15.98:0/3998610807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a210b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.139+0000 7f0edbc80740 1 --2- 172.21.15.98:0/3998610807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210930b0 0x559a2108cf00 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.139+0000 7f0edbc80740 1 --2- 172.21.15.98:0/3998610807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210919c0 0x559a210a8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.139+0000 7f0edbc80740 1 -- 172.21.15.98:0/3998610807 >> 172.21.15.98:0/3998610807 conn(0x559a2105a0a0 msgr2=0x559a2108c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.139+0000 7f0edbc80740 1 -- 172.21.15.98:0/3998610807 shutdown_connections 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.139+0000 7f0edbc80740 1 -- 172.21.15.98:0/3998610807 wait complete. 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.139+0000 7f0edbc80740 1 -- start start 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.139+0000 7f0edbc80740 1 Processor -- start 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0edbc80740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210919c0 0x559a210a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0edbc80740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210930b0 0x559a210a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0edbc80740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a210a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0edbc80740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41b0490 con 0x559a210919c0 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0edbc80740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41afdc0 con 0x559a2109c740 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0edbc80740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41ad020 con 0x559a210930b0 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0edbc80740 10 client.0 _ll_register_callbacks cb 0x7f0ebc032d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210919c0 0x559a210a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210919c0 0x559a210a4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35380/0 (socket says 172.21.15.98:35380) 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0ed259c640 1 -- 172.21.15.98:0/1764580861 learned_addr learned my addr 172.21.15.98:0/1764580861 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0ecbfff640 1 --2- 172.21.15.98:0/1764580861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210930b0 0x559a210a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0ed259c640 1 -- 172.21.15.98:0/1764580861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210930b0 msgr2=0x559a210a4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0ed259c640 1 --2- 172.21.15.98:0/1764580861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210930b0 0x559a210a4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0ed2d9d640 1 --2- 172.21.15.98:0/1764580861 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a210a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0ed259c640 1 -- 172.21.15.98:0/1764580861 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 msgr2=0x559a210a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0ed259c640 1 --2- 172.21.15.98:0/1764580861 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a210a1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0ed259c640 1 -- 172.21.15.98:0/1764580861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a210a2270 con 0x559a210919c0 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0ed2d9d640 1 --2- 172.21.15.98:0/1764580861 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a210a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0ed259c640 1 --2- 172.21.15.98:0/1764580861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210919c0 0x559a210a4980 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f0ec409ad20 tx=0x7f0ec4017730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b186b28f3e6aaf3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0eb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0eb9ffb640 1 -- 172.21.15.98:0/1764580861 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec409f140 con 0x559a210919c0 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0edbc80740 1 -- 172.21.15.98:0/1764580861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ebc0035f0 con 0x559a210919c0 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0eb9ffb640 1 -- 172.21.15.98:0/1764580861 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f0ec4010ca0 con 0x559a210919c0 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0eb9ffb640 1 -- 172.21.15.98:0/1764580861 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0ec40a3a00 con 0x559a210919c0 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0edbc80740 10 client.18465 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0edbc80740 1 -- 172.21.15.98:0/1764580861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559a21113a30 con 0x559a210919c0 2024-05-09T09:00:24.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.140+0000 7f0eb9ffb640 1 -- 172.21.15.98:0/1764580861 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ebc0035f0 con 0x559a210919c0 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.141+0000 7f0eb9ffb640 1 -- 172.21.15.98:0/1764580861 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f0ec40aac00 con 0x559a210919c0 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.141+0000 7f0edbc80740 10 client.18465 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.141+0000 7f0edbc80740 10 client.18465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.141+0000 7f0edbc80740 20 client.18465 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.141+0000 7f0edbc80740 1 --2- 172.21.15.98:0/1764580861 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559a21114100 0x559a211164f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.141+0000 7f0edbc80740 4 client.18465 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.141+0000 7f0edbc80740 1 -- 172.21.15.98:0/1764580861 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559a20ffe410 con 0x559a21114100 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.141+0000 7f0ecbfff640 1 --2- 172.21.15.98:0/1764580861 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559a21114100 0x559a211164f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.141+0000 7f0eb9ffb640 10 client.18465 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.141+0000 7f0ecbfff640 1 --2- 172.21.15.98:0/1764580861 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559a21114100 0x559a211164f0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc9c1736e502a98e server_cookie=fa7069efeab98ab in_seq=0 out_seq=0 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.142+0000 7f0eb9ffb640 1 -- 172.21.15.98:0/1764580861 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559a20ffe410 con 0x559a21114100 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.142+0000 7f0eb9ffb640 10 client.18465 handle_command_reply: tid=0 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.142+0000 7f0edbc80740 1 client.18465 shutdown 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.142+0000 7f0edbc80740 1 -- 172.21.15.98:0/1764580861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210919c0 msgr2=0x559a210a4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.142+0000 7f0edbc80740 1 --2- 172.21.15.98:0/1764580861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210919c0 0x559a210a4980 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f0ec409ad20 tx=0x7f0ec4017730 comp rx=0 tx=0).stop 2024-05-09T09:00:24.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.142+0000 7f0edbc80740 1 -- 172.21.15.98:0/1764580861 shutdown_connections 2024-05-09T09:00:24.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.142+0000 7f0edbc80740 1 --2- 172.21.15.98:0/1764580861 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x559a21114100 0x559a211164f0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.142+0000 7f0edbc80740 1 --2- 172.21.15.98:0/1764580861 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559a2109c740 0x559a210a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.142+0000 7f0edbc80740 1 --2- 172.21.15.98:0/1764580861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559a210930b0 0x559a210a4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.142+0000 7f0edbc80740 1 --2- 172.21.15.98:0/1764580861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559a210919c0 0x559a210a4980 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.142+0000 7f0edbc80740 1 -- 172.21.15.98:0/1764580861 >> 172.21.15.98:0/1764580861 conn(0x559a2105a0a0 msgr2=0x559a2108d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:24.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.142+0000 7f0eb9ffb640 0 client.18465 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:24.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0edbc80740 1 -- 172.21.15.98:0/1764580861 shutdown_connections 2024-05-09T09:00:24.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0edbc80740 1 -- 172.21.15.98:0/1764580861 wait complete. 2024-05-09T09:00:24.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0edbc80740 20 client.18465 trim_cache size 0 max 16384 2024-05-09T09:00:24.148 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:00:24.149 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T09:00:24.149 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "6f55763a-ce0c-453a-9960-bb9fc0968151", 2024-05-09T09:00:24.149 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T09:00:24.149 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:00:24.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0ed8f70640 1 -- 172.21.15.98:0/3520005851 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0ec00a2d00 msgr2=0x7f0ec00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0ed8f70640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0ec00a2d00 0x7f0ec00a51c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00afd0 tx=0x7f0ecc01a000 comp rx=0 tx=0).stop 2024-05-09T09:00:24.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0ed8f70640 1 -- 172.21.15.98:0/3520005851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ed4069eb0 msgr2=0x7f0ed4198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0ed8f70640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ed4069eb0 0x7f0ed4198190 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f0ec400b4a0 tx=0x7f0ec400b960 comp rx=0 tx=0).stop 2024-05-09T09:00:24.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0ed8f70640 1 -- 172.21.15.98:0/3520005851 shutdown_connections 2024-05-09T09:00:24.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0ed8f70640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0ec00a2d00 0x7f0ec00a51c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0ed8f70640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ed4107820 0x7f0ed419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0ed8f70640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ed406a870 0x7f0ed41986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0ed8f70640 1 --2- 172.21.15.98:0/3520005851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ed4069eb0 0x7f0ed4198190 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0ed8f70640 1 -- 172.21.15.98:0/3520005851 >> 172.21.15.98:0/3520005851 conn(0x7f0ed4077790 msgr2=0x7f0ed40fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:24.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0ed8f70640 1 -- 172.21.15.98:0/3520005851 shutdown_connections 2024-05-09T09:00:24.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.143+0000 7f0ed8f70640 1 -- 172.21.15.98:0/3520005851 wait complete. 2024-05-09T09:00:24.154 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:00:24.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:00:24.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.257+0000 7f6319c38640 1 -- 172.21.15.98:0/1334245085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6314102ab0 msgr2=0x7f6314102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.257+0000 7f6319c38640 1 --2- 172.21.15.98:0/1334245085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6314102ab0 0x7f6314102e90 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f6304009900 tx=0x7f630402a510 comp rx=0 tx=0).stop 2024-05-09T09:00:24.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.257+0000 7f6319c38640 1 -- 172.21.15.98:0/1334245085 shutdown_connections 2024-05-09T09:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.257+0000 7f6319c38640 1 --2- 172.21.15.98:0/1334245085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f631410bdf0 0x7f631410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.257+0000 7f6319c38640 1 --2- 172.21.15.98:0/1334245085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63141033d0 0x7f631410b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.257+0000 7f6319c38640 1 --2- 172.21.15.98:0/1334245085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6314102ab0 0x7f6314102e90 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.257+0000 7f6319c38640 1 -- 172.21.15.98:0/1334245085 >> 172.21.15.98:0/1334245085 conn(0x7f6314077830 msgr2=0x7f6314077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.257+0000 7f6319c38640 1 -- 172.21.15.98:0/1334245085 shutdown_connections 2024-05-09T09:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.258+0000 7f6319c38640 1 -- 172.21.15.98:0/1334245085 wait complete. 2024-05-09T09:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.258+0000 7f6319c38640 1 Processor -- start 2024-05-09T09:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.258+0000 7f6319c38640 1 -- start start 2024-05-09T09:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.258+0000 7f6319c38640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6314102ab0 0x7f6314100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.258+0000 7f6319c38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63141033d0 0x7f6314100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.258+0000 7f6319c38640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f631410bdf0 0x7f6314101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.258+0000 7f6319c38640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6314111c30 con 0x7f63141033d0 2024-05-09T09:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.258+0000 7f6319c38640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6314111ab0 con 0x7f6314102ab0 2024-05-09T09:00:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.258+0000 7f6319c38640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6314111db0 con 0x7f631410bdf0 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f630affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63141033d0 0x7f6314100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f630affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63141033d0 0x7f6314100c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35396/0 (socket says 172.21.15.98:35396) 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f630affd640 1 -- 172.21.15.98:0/4240649590 learned_addr learned my addr 172.21.15.98:0/4240649590 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f63137fe640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6314102ab0 0x7f6314100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f6313fff640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f631410bdf0 0x7f6314101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f630affd640 1 -- 172.21.15.98:0/4240649590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f631410bdf0 msgr2=0x7f6314101180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f630affd640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f631410bdf0 0x7f6314101180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f630affd640 1 -- 172.21.15.98:0/4240649590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6314102ab0 msgr2=0x7f6314100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f630affd640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6314102ab0 0x7f6314100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f630affd640 1 -- 172.21.15.98:0/4240649590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63141a6630 con 0x7f63141033d0 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f6313fff640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f631410bdf0 0x7f6314101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f630affd640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63141033d0 0x7f6314100c40 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00c890 tx=0x7f62fc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f63117fa640 1 -- 172.21.15.98:0/4240649590 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc004220 con 0x7f63141033d0 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f6319c38640 1 -- 172.21.15.98:0/4240649590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63141a6920 con 0x7f63141033d0 2024-05-09T09:00:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f63117fa640 1 -- 172.21.15.98:0/4240649590 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62fc0043c0 con 0x7f63141033d0 2024-05-09T09:00:24.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f63117fa640 1 -- 172.21.15.98:0/4240649590 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc004a00 con 0x7f63141033d0 2024-05-09T09:00:24.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.259+0000 7f6319c38640 1 -- 172.21.15.98:0/4240649590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63141a6ec0 con 0x7f63141033d0 2024-05-09T09:00:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.262+0000 7f63117fa640 1 -- 172.21.15.98:0/4240649590 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f62fc0079b0 con 0x7f63141033d0 2024-05-09T09:00:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.262+0000 7f631ca1e740 1 Processor -- start 2024-05-09T09:00:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.262+0000 7f631ca1e740 1 -- start start 2024-05-09T09:00:24.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.262+0000 7f631ca1e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811da280 0x55b1811da660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.262+0000 7f631ca1e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811daba0 0x55b1811e4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.262+0000 7f631ca1e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811e5270 0x55b1811e7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.262+0000 7f631ca1e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b18113fd00 con 0x55b1811e5270 2024-05-09T09:00:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.262+0000 7f631ca1e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b1810e74c0 con 0x55b1811daba0 2024-05-09T09:00:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.262+0000 7f631ca1e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b18113bcc0 con 0x55b1811da280 2024-05-09T09:00:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f630affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811e5270 0x55b1811e7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f630affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811e5270 0x55b1811e7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35410/0 (socket says 172.21.15.98:35410) 2024-05-09T09:00:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f630affd640 1 -- 172.21.15.98:0/2112958086 learned_addr learned my addr 172.21.15.98:0/2112958086 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f6313fff640 1 --2- 172.21.15.98:0/2112958086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811da280 0x55b1811da660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f63137fe640 1 --2- 172.21.15.98:0/2112958086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811daba0 0x55b1811e4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f63117fa640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63000a2d00 0x7f63000a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f6313fff640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63000a2d00 0x7f63000a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f630affd640 1 -- 172.21.15.98:0/2112958086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811da280 msgr2=0x55b1811da660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f630affd640 1 --2- 172.21.15.98:0/2112958086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811da280 0x55b1811da660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f630affd640 1 -- 172.21.15.98:0/2112958086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811daba0 msgr2=0x55b1811e4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f630affd640 1 --2- 172.21.15.98:0/2112958086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811daba0 0x55b1811e4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f63117fa640 1 -- 172.21.15.98:0/4240649590 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f62fc0cf610 con 0x7f63141033d0 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f630affd640 1 -- 172.21.15.98:0/2112958086 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b180f1d7d0 con 0x55b1811e5270 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f630affd640 1 --2- 172.21.15.98:0/2112958086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811e5270 0x55b1811e7660 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f62fc0cc8c0 tx=0x7f62fc0054a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9edcbde8e13296f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f6313fff640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63000a2d00 0x7f63000a51c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f630c013870 tx=0x7f630c004a40 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f6309ffb640 1 -- 172.21.15.98:0/2112958086 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc01d020 con 0x55b1811e5270 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f6309ffb640 1 -- 172.21.15.98:0/2112958086 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62fc086110 con 0x55b1811e5270 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.263+0000 7f6309ffb640 1 -- 172.21.15.98:0/2112958086 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc09fad0 con 0x55b1811e5270 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.264+0000 7f631ca1e740 1 -- 172.21.15.98:0/2112958086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811e5270 msgr2=0x55b1811e7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.264+0000 7f631ca1e740 1 --2- 172.21.15.98:0/2112958086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811e5270 0x55b1811e7660 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f62fc0cc8c0 tx=0x7f62fc0054a0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.264+0000 7f631ca1e740 1 -- 172.21.15.98:0/2112958086 shutdown_connections 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.264+0000 7f631ca1e740 1 --2- 172.21.15.98:0/2112958086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811e5270 0x55b1811e7660 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.264+0000 7f631ca1e740 1 --2- 172.21.15.98:0/2112958086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811daba0 0x55b1811e4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.264+0000 7f631ca1e740 1 --2- 172.21.15.98:0/2112958086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811da280 0x55b1811da660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.264+0000 7f631ca1e740 1 -- 172.21.15.98:0/2112958086 >> 172.21.15.98:0/2112958086 conn(0x55b1811d65a0 msgr2=0x55b1811d6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:24.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.264+0000 7f631ca1e740 1 -- 172.21.15.98:0/2112958086 shutdown_connections 2024-05-09T09:00:24.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.264+0000 7f631ca1e740 1 -- 172.21.15.98:0/2112958086 wait complete. 2024-05-09T09:00:24.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.264+0000 7f631ca1e740 1 -- start start 2024-05-09T09:00:24.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f631ca1e740 1 Processor -- start 2024-05-09T09:00:24.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f631ca1e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811da280 0x55b1811e2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f631ca1e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811daba0 0x55b1811e2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f631ca1e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811e5270 0x55b1811e2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f631ca1e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b1811e7fa0 con 0x55b1811e5270 2024-05-09T09:00:24.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f631ca1e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b1810fab90 con 0x55b1811daba0 2024-05-09T09:00:24.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f631ca1e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b1811e8120 con 0x55b1811da280 2024-05-09T09:00:24.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f631ca1e740 10 client.0 _ll_register_callbacks cb 0x55b181136da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:24.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f6313fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811e5270 0x55b1811e2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f6313fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811e5270 0x55b1811e2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35412/0 (socket says 172.21.15.98:35412) 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f6313fff640 1 -- 172.21.15.98:0/1390759716 learned_addr learned my addr 172.21.15.98:0/1390759716 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f63137fe640 1 --2- 172.21.15.98:0/1390759716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811da280 0x55b1811e2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f630affd640 1 --2- 172.21.15.98:0/1390759716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811daba0 0x55b1811e2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f6313fff640 1 -- 172.21.15.98:0/1390759716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811da280 msgr2=0x55b1811e2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f6313fff640 1 --2- 172.21.15.98:0/1390759716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811da280 0x55b1811e2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f6313fff640 1 -- 172.21.15.98:0/1390759716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811daba0 msgr2=0x55b1811e2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f6313fff640 1 --2- 172.21.15.98:0/1390759716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811daba0 0x55b1811e2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f6313fff640 1 -- 172.21.15.98:0/1390759716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b180e533f0 con 0x55b1811e5270 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f6313fff640 1 --2- 172.21.15.98:0/1390759716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811e5270 0x55b1811e2bd0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f630c010f70 tx=0x7f630c004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d23c8e11c5e3cb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.265+0000 7f62f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f62f2ffd640 1 -- 172.21.15.98:0/1390759716 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630c022070 con 0x55b1811e5270 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f62f2ffd640 1 -- 172.21.15.98:0/1390759716 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f630c01e800 con 0x55b1811e5270 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f631ca1e740 1 -- 172.21.15.98:0/1390759716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62fc086110 con 0x55b1811e5270 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f62f2ffd640 1 -- 172.21.15.98:0/1390759716 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f630c02e050 con 0x55b1811e5270 2024-05-09T09:00:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f631ca1e740 10 client.18477 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:24.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f631ca1e740 1 -- 172.21.15.98:0/1390759716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b180e130f0 con 0x55b1811e5270 2024-05-09T09:00:24.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f62f2ffd640 1 -- 172.21.15.98:0/1390759716 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62fc086110 con 0x55b1811e5270 2024-05-09T09:00:24.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f62f2ffd640 1 -- 172.21.15.98:0/1390759716 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f630c01d070 con 0x55b1811e5270 2024-05-09T09:00:24.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f631ca1e740 10 client.18477 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:24.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f631ca1e740 10 client.18477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:24.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f631ca1e740 20 client.18477 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:24.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1390759716 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b1811f5470 0x55b1811f78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f631ca1e740 4 client.18477 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:24.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f631ca1e740 1 -- 172.21.15.98:0/1390759716 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b18114d430 con 0x55b1811f5470 2024-05-09T09:00:24.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.266+0000 7f63137fe640 1 --2- 172.21.15.98:0/1390759716 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b1811f5470 0x55b1811f78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.267+0000 7f62f2ffd640 10 client.18477 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:24.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.267+0000 7f63137fe640 1 --2- 172.21.15.98:0/1390759716 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b1811f5470 0x55b1811f78a0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=690b62da30e1d502 server_cookie=54e0021729d3bd4e in_seq=0 out_seq=0 2024-05-09T09:00:24.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.267+0000 7f62f2ffd640 1 -- 172.21.15.98:0/1390759716 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b18114d430 con 0x55b1811f5470 2024-05-09T09:00:24.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.267+0000 7f62f2ffd640 10 client.18477 handle_command_reply: tid=0 2024-05-09T09:00:24.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.268+0000 7f631ca1e740 1 client.18477 shutdown 2024-05-09T09:00:24.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.268+0000 7f631ca1e740 1 -- 172.21.15.98:0/1390759716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811e5270 msgr2=0x55b1811e2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.268+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1390759716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811e5270 0x55b1811e2bd0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f630c010f70 tx=0x7f630c004b50 comp rx=0 tx=0).stop 2024-05-09T09:00:24.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.268+0000 7f631ca1e740 1 -- 172.21.15.98:0/1390759716 shutdown_connections 2024-05-09T09:00:24.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.268+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1390759716 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b1811f5470 0x55b1811f78a0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.268+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1390759716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811e5270 0x55b1811e2bd0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.268+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1390759716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811daba0 0x55b1811e2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.268+0000 7f62f2ffd640 0 client.18477 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:24.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.268+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1390759716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811da280 0x55b1811e2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.268+0000 7f631ca1e740 1 -- 172.21.15.98:0/1390759716 >> 172.21.15.98:0/1390759716 conn(0x55b1811d65a0 msgr2=0x55b1811dc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:24.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.268+0000 7f631ca1e740 1 -- 172.21.15.98:0/1390759716 shutdown_connections 2024-05-09T09:00:24.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.268+0000 7f631ca1e740 1 -- 172.21.15.98:0/1390759716 wait complete. 2024-05-09T09:00:24.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.268+0000 7f631ca1e740 20 client.18477 trim_cache size 0 max 16384 2024-05-09T09:00:24.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.285+0000 7f631ca1e740 1 Processor -- start 2024-05-09T09:00:24.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.285+0000 7f631ca1e740 1 -- start start 2024-05-09T09:00:24.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.285+0000 7f631ca1e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811da280 0x55b1811f0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.285+0000 7f631ca1e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811daba0 0x55b1811dca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.285+0000 7f631ca1e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811e5270 0x55b1811fd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.285+0000 7f631ca1e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63141b1170 con 0x55b1811da280 2024-05-09T09:00:24.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.285+0000 7f631ca1e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f63141b2cb0 con 0x55b1811e5270 2024-05-09T09:00:24.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.285+0000 7f631ca1e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f63141b1840 con 0x55b1811daba0 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.285+0000 7f63137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811da280 0x55b1811f0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.285+0000 7f63137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811da280 0x55b1811f0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35414/0 (socket says 172.21.15.98:35414) 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.285+0000 7f63137fe640 1 -- 172.21.15.98:0/1654466446 learned_addr learned my addr 172.21.15.98:0/1654466446 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f63137fe640 1 -- 172.21.15.98:0/1654466446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811daba0 msgr2=0x55b1811dca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f63137fe640 1 --2- 172.21.15.98:0/1654466446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811daba0 0x55b1811dca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f6313fff640 1 --2- 172.21.15.98:0/1654466446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811e5270 0x55b1811fd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f63137fe640 1 -- 172.21.15.98:0/1654466446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811e5270 msgr2=0x55b1811fd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f63137fe640 1 --2- 172.21.15.98:0/1654466446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811e5270 0x55b1811fd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f63137fe640 1 -- 172.21.15.98:0/1654466446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1811db510 con 0x55b1811da280 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f6313fff640 1 --2- 172.21.15.98:0/1654466446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811e5270 0x55b1811fd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f63137fe640 1 --2- 172.21.15.98:0/1654466446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811da280 0x55b1811f0e60 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f630402aa30 tx=0x7f6304031ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9401ef1fc6226e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f63097fa640 1 -- 172.21.15.98:0/1654466446 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630404b070 con 0x55b1811da280 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f63097fa640 1 -- 172.21.15.98:0/1654466446 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6304047900 con 0x55b1811da280 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f63097fa640 1 -- 172.21.15.98:0/1654466446 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6304035040 con 0x55b1811da280 2024-05-09T09:00:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f631ca1e740 1 -- 172.21.15.98:0/1654466446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811da280 msgr2=0x55b1811f0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1654466446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811da280 0x55b1811f0e60 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f630402aa30 tx=0x7f6304031ac0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f631ca1e740 1 -- 172.21.15.98:0/1654466446 shutdown_connections 2024-05-09T09:00:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1654466446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811e5270 0x55b1811fd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1654466446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811daba0 0x55b1811dca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.286+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1654466446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811da280 0x55b1811f0e60 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f631ca1e740 1 -- 172.21.15.98:0/1654466446 >> 172.21.15.98:0/1654466446 conn(0x55b1811d65a0 msgr2=0x55b1811e29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f631ca1e740 1 -- 172.21.15.98:0/1654466446 shutdown_connections 2024-05-09T09:00:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f631ca1e740 1 -- 172.21.15.98:0/1654466446 wait complete. 2024-05-09T09:00:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f631ca1e740 1 -- start start 2024-05-09T09:00:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f631ca1e740 1 Processor -- start 2024-05-09T09:00:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f631ca1e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811da280 0x55b1811e8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f631ca1e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811daba0 0x55b1811e93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f631ca1e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811e5270 0x55b1811e9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f631ca1e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63141b25e0 con 0x55b1811daba0 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f631ca1e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f63141b1f10 con 0x55b1811da280 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f631ca1e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f63141af170 con 0x55b1811e5270 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f631ca1e740 10 client.0 _ll_register_callbacks cb 0x7f630c013450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f630affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811daba0 0x55b1811e93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f630affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811daba0 0x55b1811e93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35426/0 (socket says 172.21.15.98:35426) 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.287+0000 7f630affd640 1 -- 172.21.15.98:0/1029698867 learned_addr learned my addr 172.21.15.98:0/1029698867 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f6313fff640 1 --2- 172.21.15.98:0/1029698867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811e5270 0x55b1811e9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f63137fe640 1 --2- 172.21.15.98:0/1029698867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811da280 0x55b1811e8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f630affd640 1 -- 172.21.15.98:0/1029698867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811e5270 msgr2=0x55b1811e9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f630affd640 1 --2- 172.21.15.98:0/1029698867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811e5270 0x55b1811e9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f630affd640 1 -- 172.21.15.98:0/1029698867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811da280 msgr2=0x55b1811e8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f630affd640 1 --2- 172.21.15.98:0/1029698867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811da280 0x55b1811e8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f630affd640 1 -- 172.21.15.98:0/1029698867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1811db720 con 0x55b1811daba0 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f6313fff640 1 --2- 172.21.15.98:0/1029698867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811e5270 0x55b1811e9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:24.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f630affd640 1 --2- 172.21.15.98:0/1029698867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811daba0 0x55b1811e93e0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f62fc0ccf50 tx=0x7f62fc00a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74bdd0b760adfb89 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:24.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f62f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:24.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f62f2ffd640 1 -- 172.21.15.98:0/1029698867 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc0b4060 con 0x55b1811daba0 2024-05-09T09:00:24.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f62f2ffd640 1 -- 172.21.15.98:0/1029698867 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f62fc007ca0 con 0x55b1811daba0 2024-05-09T09:00:24.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f631ca1e740 1 -- 172.21.15.98:0/1029698867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6304047900 con 0x55b1811daba0 2024-05-09T09:00:24.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f62f2ffd640 1 -- 172.21.15.98:0/1029698867 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f62fc01d040 con 0x55b1811daba0 2024-05-09T09:00:24.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f631ca1e740 10 client.18483 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:24.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f631ca1e740 1 -- 172.21.15.98:0/1029698867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b18125bb90 con 0x55b1811daba0 2024-05-09T09:00:24.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f62f2ffd640 1 -- 172.21.15.98:0/1029698867 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6304047900 con 0x55b1811daba0 2024-05-09T09:00:24.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f62f2ffd640 1 -- 172.21.15.98:0/1029698867 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f62fc097470 con 0x55b1811daba0 2024-05-09T09:00:24.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f631ca1e740 10 client.18483 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:24.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f631ca1e740 10 client.18483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:24.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f631ca1e740 20 client.18483 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:24.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1029698867 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b18125bf90 0x55b18125e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:24.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f631ca1e740 4 client.18483 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.288+0000 7f631ca1e740 1 -- 172.21.15.98:0/1029698867 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b181147780 con 0x55b18125bf90 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f63137fe640 1 --2- 172.21.15.98:0/1029698867 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b18125bf90 0x55b18125e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f62f2ffd640 10 client.18483 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f63137fe640 1 --2- 172.21.15.98:0/1029698867 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b18125bf90 0x55b18125e380 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6ca1402deaa6c9e server_cookie=134b5fa3a5baa534 in_seq=0 out_seq=0 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f62f2ffd640 1 -- 172.21.15.98:0/1029698867 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b181147780 con 0x55b18125bf90 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f62f2ffd640 10 client.18483 handle_command_reply: tid=0 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f631ca1e740 1 client.18483 shutdown 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f631ca1e740 1 -- 172.21.15.98:0/1029698867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811daba0 msgr2=0x55b1811e93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1029698867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811daba0 0x55b1811e93e0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f62fc0ccf50 tx=0x7f62fc00a410 comp rx=0 tx=0).stop 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f631ca1e740 1 -- 172.21.15.98:0/1029698867 shutdown_connections 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1029698867 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b18125bf90 0x55b18125e380 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1029698867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1811e5270 0x55b1811e9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1029698867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1811daba0 0x55b1811e93e0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f631ca1e740 1 --2- 172.21.15.98:0/1029698867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1811da280 0x55b1811e8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f62f2ffd640 0 client.18483 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:24.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f631ca1e740 1 -- 172.21.15.98:0/1029698867 >> 172.21.15.98:0/1029698867 conn(0x55b1811d65a0 msgr2=0x55b1811d4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:24.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f631ca1e740 1 -- 172.21.15.98:0/1029698867 shutdown_connections 2024-05-09T09:00:24.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.289+0000 7f631ca1e740 1 -- 172.21.15.98:0/1029698867 wait complete. 2024-05-09T09:00:24.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.290+0000 7f631ca1e740 20 client.18483 trim_cache size 0 max 16384 2024-05-09T09:00:24.296 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:00:24.296 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T09:00:24.296 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T09:00:24.296 INFO:teuthology.orchestra.run.smithi098.stdout: "6f55763a-ce0c-453a-9960-bb9fc0968151": { 2024-05-09T09:00:24.296 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T09:00:24.296 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "6f55763a-ce0c-453a-9960-bb9fc0968151", 2024-05-09T09:00:24.297 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T09:00:24.297 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:00:24.297 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:00:24.297 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:00:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.290+0000 7f6319c38640 1 -- 172.21.15.98:0/4240649590 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63000a2d00 msgr2=0x7f63000a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.290+0000 7f6319c38640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63000a2d00 0x7f63000a51c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f630c013870 tx=0x7f630c004a40 comp rx=0 tx=0).stop 2024-05-09T09:00:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.290+0000 7f6319c38640 1 -- 172.21.15.98:0/4240649590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63141033d0 msgr2=0x7f6314100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.290+0000 7f6319c38640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63141033d0 0x7f6314100c40 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00c890 tx=0x7f62fc00cd50 comp rx=0 tx=0).stop 2024-05-09T09:00:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.290+0000 7f6319c38640 1 -- 172.21.15.98:0/4240649590 shutdown_connections 2024-05-09T09:00:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.290+0000 7f6319c38640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f63000a2d00 0x7f63000a51c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.290+0000 7f6319c38640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f631410bdf0 0x7f6314101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.290+0000 7f6319c38640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63141033d0 0x7f6314100c40 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.290+0000 7f6319c38640 1 --2- 172.21.15.98:0/4240649590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6314102ab0 0x7f6314100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.290+0000 7f6319c38640 1 -- 172.21.15.98:0/4240649590 >> 172.21.15.98:0/4240649590 conn(0x7f6314077830 msgr2=0x7f63140fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.290+0000 7f6319c38640 1 -- 172.21.15.98:0/4240649590 shutdown_connections 2024-05-09T09:00:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:24.290+0000 7f6319c38640 1 -- 172.21.15.98:0/4240649590 wait complete. 2024-05-09T09:00:24.299 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:24 smithi098 ceph-mon[30092]: pgmap v305: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-05-09T09:00:24.300 INFO:tasks.cephfs.filesystem:scrub status for tag:6f55763a-ce0c-453a-9960-bb9fc0968151 - {'path': '/', 'tag': '6f55763a-ce0c-453a-9960-bb9fc0968151', 'options': 'recursive,force'} 2024-05-09T09:00:24.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:24 smithi105 ceph-mon[34498]: pgmap v305: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-05-09T09:00:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:24 smithi142 ceph-mon[34625]: pgmap v305: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-05-09T09:00:25.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:25 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T09:00:25.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:25 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T09:00:25.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:25 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T09:00:25.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:25 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T09:00:25.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:25 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T09:00:25.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:25 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T09:00:25.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:25 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T09:00:25.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:25 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T09:00:25.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:25 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T09:00:25.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:25 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T09:00:25.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:25 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T09:00:25.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:25 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T09:00:26.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:26 smithi105 ceph-mon[34498]: pgmap v306: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 102 B/s wr, 10 op/s 2024-05-09T09:00:26.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:26 smithi098 ceph-mon[30092]: pgmap v306: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 102 B/s wr, 10 op/s 2024-05-09T09:00:26.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:26 smithi142 ceph-mon[34625]: pgmap v306: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 102 B/s wr, 10 op/s 2024-05-09T09:00:27.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:27 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T09:00:27.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:27 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T09:00:27.696 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:27 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T09:00:28.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:28 smithi105 ceph-mon[34498]: pgmap v307: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 307 B/s wr, 13 op/s 2024-05-09T09:00:28.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:28 smithi098 ceph-mon[30092]: pgmap v307: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 307 B/s wr, 13 op/s 2024-05-09T09:00:28.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:28 smithi142 ceph-mon[34625]: pgmap v307: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 307 B/s wr, 13 op/s 2024-05-09T09:00:30.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:30 smithi105 ceph-mon[34498]: pgmap v308: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 282 B/s wr, 13 op/s 2024-05-09T09:00:30.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:30 smithi098 ceph-mon[30092]: pgmap v308: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 282 B/s wr, 13 op/s 2024-05-09T09:00:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:30 smithi142 ceph-mon[34625]: pgmap v308: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 282 B/s wr, 13 op/s 2024-05-09T09:00:32.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:32 smithi105 ceph-mon[34498]: pgmap v309: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:00:32.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:32 smithi098 ceph-mon[30092]: pgmap v309: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:00:32.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:32 smithi142 ceph-mon[34625]: pgmap v309: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:00:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.108+0000 7f157adb7640 1 -- 172.21.15.98:0/1353582129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157410e1f0 msgr2=0x7f15741105e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.108+0000 7f157adb7640 1 --2- 172.21.15.98:0/1353582129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157410e1f0 0x7f15741105e0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f155c009900 tx=0x7f155c02a410 comp rx=0 tx=0).stop 2024-05-09T09:00:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.108+0000 7f157adb7640 1 -- 172.21.15.98:0/1353582129 shutdown_connections 2024-05-09T09:00:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.108+0000 7f157adb7640 1 --2- 172.21.15.98:0/1353582129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157410e1f0 0x7f15741105e0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.108+0000 7f157adb7640 1 --2- 172.21.15.98:0/1353582129 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1574100f60 0x7f157410dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.108+0000 7f157adb7640 1 --2- 172.21.15.98:0/1353582129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1574100640 0x7f1574100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.108+0000 7f157adb7640 1 -- 172.21.15.98:0/1353582129 >> 172.21.15.98:0/1353582129 conn(0x7f15740778b0 msgr2=0x7f1574077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.108+0000 7f157adb7640 1 -- 172.21.15.98:0/1353582129 shutdown_connections 2024-05-09T09:00:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.108+0000 7f157adb7640 1 -- 172.21.15.98:0/1353582129 wait complete. 2024-05-09T09:00:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.108+0000 7f157adb7640 1 Processor -- start 2024-05-09T09:00:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f157adb7640 1 -- start start 2024-05-09T09:00:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f157adb7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1574100640 0x7f157419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f157adb7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1574100f60 0x7f157419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f157adb7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f157410e1f0 0x7f157419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f157adb7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1574113e20 con 0x7f1574100640 2024-05-09T09:00:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f157adb7640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1574113ca0 con 0x7f157410e1f0 2024-05-09T09:00:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f157adb7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1574113fa0 con 0x7f1574100f60 2024-05-09T09:00:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f1578b53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1574100640 0x7f157419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f1578b53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1574100640 0x7f157419a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44136/0 (socket says 172.21.15.98:44136) 2024-05-09T09:00:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f1578b53640 1 -- 172.21.15.98:0/406461511 learned_addr learned my addr 172.21.15.98:0/406461511 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f1573fff640 1 --2- 172.21.15.98:0/406461511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1574100f60 0x7f157419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f1579354640 1 --2- 172.21.15.98:0/406461511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f157410e1f0 0x7f157419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f1578b53640 1 -- 172.21.15.98:0/406461511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1574100f60 msgr2=0x7f157419a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f1578b53640 1 --2- 172.21.15.98:0/406461511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1574100f60 0x7f157419a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f1578b53640 1 -- 172.21.15.98:0/406461511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f157410e1f0 msgr2=0x7f157419eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f1578b53640 1 --2- 172.21.15.98:0/406461511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f157410e1f0 0x7f157419eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f1578b53640 1 -- 172.21.15.98:0/406461511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157419f270 con 0x7f1574100640 2024-05-09T09:00:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f1578b53640 1 --2- 172.21.15.98:0/406461511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1574100640 0x7f157419a280 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f156c00e9f0 tx=0x7f156c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.109+0000 7f1571ffb640 1 -- 172.21.15.98:0/406461511 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c00cd80 con 0x7f1574100640 2024-05-09T09:00:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.110+0000 7f157adb7640 1 -- 172.21.15.98:0/406461511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f157419f560 con 0x7f1574100640 2024-05-09T09:00:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.110+0000 7f1571ffb640 1 -- 172.21.15.98:0/406461511 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f156c005500 con 0x7f1574100640 2024-05-09T09:00:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.110+0000 7f1571ffb640 1 -- 172.21.15.98:0/406461511 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c010690 con 0x7f1574100640 2024-05-09T09:00:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.110+0000 7f157adb7640 1 -- 172.21.15.98:0/406461511 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15741a6df0 con 0x7f1574100640 2024-05-09T09:00:34.119 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.112+0000 7f1571ffb640 1 -- 172.21.15.98:0/406461511 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f156c0046c0 con 0x7f1574100640 2024-05-09T09:00:34.119 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f157dba3740 1 Processor -- start 2024-05-09T09:00:34.119 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f157dba3740 1 -- start start 2024-05-09T09:00:34.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1571ffb640 1 --2- 172.21.15.98:0/406461511 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f15600a2da0 0x7f15600a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f157dba3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628450 0x562dc5628830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f157dba3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5628d70 0x562dc56330f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f157dba3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5633630 0x562dc5635a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f157dba3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562dc558d340 con 0x562dc5628450 2024-05-09T09:00:34.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f157dba3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562dc5535290 con 0x562dc5628d70 2024-05-09T09:00:34.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f157dba3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562dc5589300 con 0x562dc5633630 2024-05-09T09:00:34.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1579354640 1 --2- 172.21.15.98:0/406461511 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f15600a2da0 0x7f15600a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1571ffb640 1 -- 172.21.15.98:0/406461511 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f156c0cfc80 con 0x7f1574100640 2024-05-09T09:00:34.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1578b53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628450 0x562dc5628830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1578b53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628450 0x562dc5628830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44146/0 (socket says 172.21.15.98:44146) 2024-05-09T09:00:34.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1578b53640 1 -- 172.21.15.98:0/1181326397 learned_addr learned my addr 172.21.15.98:0/1181326397 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:34.120 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1573fff640 1 --2- 172.21.15.98:0/1181326397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5628d70 0x562dc56330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1579354640 1 --2- 172.21.15.98:0/1181326397 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5633630 0x562dc5635a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1578b53640 1 -- 172.21.15.98:0/1181326397 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5633630 msgr2=0x562dc5635a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1578b53640 1 --2- 172.21.15.98:0/1181326397 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5633630 0x562dc5635a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1578b53640 1 -- 172.21.15.98:0/1181326397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5628d70 msgr2=0x562dc56330f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1578b53640 1 --2- 172.21.15.98:0/1181326397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5628d70 0x562dc56330f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1578b53640 1 -- 172.21.15.98:0/1181326397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dc55598e0 con 0x562dc5628450 2024-05-09T09:00:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1578b53640 1 --2- 172.21.15.98:0/1181326397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628450 0x562dc5628830 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f156c098760 tx=0x7f156c09b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5eccb418278b15b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f1579354640 1 --2- 172.21.15.98:0/406461511 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f15600a2da0 0x7f15600a5260 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f155c006010 tx=0x7f155c031db0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.113+0000 7f156affd640 1 -- 172.21.15.98:0/1181326397 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c09c490 con 0x562dc5628450 2024-05-09T09:00:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.114+0000 7f156affd640 1 -- 172.21.15.98:0/1181326397 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f156c09bd90 con 0x562dc5628450 2024-05-09T09:00:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.114+0000 7f156affd640 1 -- 172.21.15.98:0/1181326397 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c04aaf0 con 0x562dc5628450 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.114+0000 7f157dba3740 1 -- 172.21.15.98:0/1181326397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628450 msgr2=0x562dc5628830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.114+0000 7f157dba3740 1 --2- 172.21.15.98:0/1181326397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628450 0x562dc5628830 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f156c098760 tx=0x7f156c09b990 comp rx=0 tx=0).stop 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.114+0000 7f157dba3740 1 -- 172.21.15.98:0/1181326397 shutdown_connections 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.114+0000 7f157dba3740 1 --2- 172.21.15.98:0/1181326397 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5633630 0x562dc5635a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.114+0000 7f157dba3740 1 --2- 172.21.15.98:0/1181326397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5628d70 0x562dc56330f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.114+0000 7f157dba3740 1 --2- 172.21.15.98:0/1181326397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628450 0x562dc5628830 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.114+0000 7f157dba3740 1 -- 172.21.15.98:0/1181326397 >> 172.21.15.98:0/1181326397 conn(0x562dc5624960 msgr2=0x562dc5624d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.114+0000 7f157dba3740 1 -- 172.21.15.98:0/1181326397 shutdown_connections 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.114+0000 7f157dba3740 1 -- 172.21.15.98:0/1181326397 wait complete. 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f157dba3740 1 -- start start 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f157dba3740 1 Processor -- start 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f157dba3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628450 0x562dc5630610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f157dba3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5628d70 0x562dc5630b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f157dba3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5633630 0x562dc5631090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f157dba3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562dc5636260 con 0x562dc5628450 2024-05-09T09:00:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f157dba3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562dc5548790 con 0x562dc5633630 2024-05-09T09:00:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f157dba3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562dc56363e0 con 0x562dc5628d70 2024-05-09T09:00:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f157dba3740 10 client.0 _ll_register_callbacks cb 0x562dc5583a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f1578b53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628450 0x562dc5630610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f1578b53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628450 0x562dc5630610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44152/0 (socket says 172.21.15.98:44152) 2024-05-09T09:00:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f1578b53640 1 -- 172.21.15.98:0/668881224 learned_addr learned my addr 172.21.15.98:0/668881224 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f1579354640 1 --2- 172.21.15.98:0/668881224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5633630 0x562dc5631090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f1578b53640 1 -- 172.21.15.98:0/668881224 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5628d70 msgr2=0x562dc5630b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f1578b53640 1 --2- 172.21.15.98:0/668881224 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5628d70 0x562dc5630b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f1578b53640 1 -- 172.21.15.98:0/668881224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5633630 msgr2=0x562dc5631090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f1578b53640 1 --2- 172.21.15.98:0/668881224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5633630 0x562dc5631090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.115+0000 7f1578b53640 1 -- 172.21.15.98:0/668881224 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dc52a1210 con 0x562dc5628450 2024-05-09T09:00:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f1578b53640 1 --2- 172.21.15.98:0/668881224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628450 0x562dc5630610 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f156c09a040 tx=0x7f156c09ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a618d5d2b8a22a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f1553fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f1553fff640 1 -- 172.21.15.98:0/668881224 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c0a4a00 con 0x562dc5628450 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f1553fff640 1 -- 172.21.15.98:0/668881224 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f156c04f180 con 0x562dc5628450 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f157dba3740 1 -- 172.21.15.98:0/668881224 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f156c09bd90 con 0x562dc5628450 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f1553fff640 1 -- 172.21.15.98:0/668881224 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f156c09c490 con 0x562dc5628450 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f157dba3740 10 client.18495 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f157dba3740 1 -- 172.21.15.98:0/668881224 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562dc536d620 con 0x562dc5628450 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f1553fff640 1 -- 172.21.15.98:0/668881224 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f156c09bd90 con 0x562dc5628450 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f1553fff640 1 -- 172.21.15.98:0/668881224 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f156c09b300 con 0x562dc5628450 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f157dba3740 10 client.18495 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f157dba3740 10 client.18495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f157dba3740 20 client.18495 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f157dba3740 1 --2- 172.21.15.98:0/668881224 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562dc5643260 0x562dc5645650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f157dba3740 4 client.18495 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.116+0000 7f157dba3740 1 -- 172.21.15.98:0/668881224 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562dc559aa70 con 0x562dc5643260 2024-05-09T09:00:34.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.117+0000 7f1573fff640 1 --2- 172.21.15.98:0/668881224 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562dc5643260 0x562dc5645650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.117+0000 7f1553fff640 10 client.18495 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.117+0000 7f1573fff640 1 --2- 172.21.15.98:0/668881224 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562dc5643260 0x562dc5645650 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1b7e46fd90fc9cd server_cookie=1650569a8f8a70b3 in_seq=0 out_seq=0 2024-05-09T09:00:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f1553fff640 1 -- 172.21.15.98:0/668881224 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562dc559aa70 con 0x562dc5643260 2024-05-09T09:00:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f1553fff640 10 client.18495 handle_command_reply: tid=0 2024-05-09T09:00:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f157dba3740 1 client.18495 shutdown 2024-05-09T09:00:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f157dba3740 1 -- 172.21.15.98:0/668881224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628450 msgr2=0x562dc5630610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f157dba3740 1 --2- 172.21.15.98:0/668881224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628450 0x562dc5630610 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f156c09a040 tx=0x7f156c09ba00 comp rx=0 tx=0).stop 2024-05-09T09:00:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f157dba3740 1 -- 172.21.15.98:0/668881224 shutdown_connections 2024-05-09T09:00:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f157dba3740 1 --2- 172.21.15.98:0/668881224 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562dc5643260 0x562dc5645650 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f157dba3740 1 --2- 172.21.15.98:0/668881224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5633630 0x562dc5631090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f157dba3740 1 --2- 172.21.15.98:0/668881224 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5628d70 0x562dc5630b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f1553fff640 0 client.18495 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f157dba3740 1 --2- 172.21.15.98:0/668881224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628450 0x562dc5630610 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f157dba3740 1 -- 172.21.15.98:0/668881224 >> 172.21.15.98:0/668881224 conn(0x562dc5624960 msgr2=0x562dc562a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f157dba3740 1 -- 172.21.15.98:0/668881224 shutdown_connections 2024-05-09T09:00:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f157dba3740 1 -- 172.21.15.98:0/668881224 wait complete. 2024-05-09T09:00:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.118+0000 7f157dba3740 20 client.18495 trim_cache size 0 max 16384 2024-05-09T09:00:34.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f157dba3740 1 Processor -- start 2024-05-09T09:00:34.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f157dba3740 1 -- start start 2024-05-09T09:00:34.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f157dba3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5628450 0x562dc5632d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f157dba3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628d70 0x562dc563ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f157dba3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5633630 0x562dc562d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f157dba3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f15741b10c0 con 0x562dc5628d70 2024-05-09T09:00:34.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f157dba3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f15741b2c00 con 0x562dc5628450 2024-05-09T09:00:34.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f157dba3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f15741b1790 con 0x562dc5633630 2024-05-09T09:00:34.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f1573fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628d70 0x562dc563ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f1573fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628d70 0x562dc563ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44164/0 (socket says 172.21.15.98:44164) 2024-05-09T09:00:34.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f1573fff640 1 -- 172.21.15.98:0/509160351 learned_addr learned my addr 172.21.15.98:0/509160351 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:34.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f1573fff640 1 -- 172.21.15.98:0/509160351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5633630 msgr2=0x562dc562d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:34.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f1573fff640 1 --2- 172.21.15.98:0/509160351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5633630 0x562dc562d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f1578b53640 1 --2- 172.21.15.98:0/509160351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5628450 0x562dc5632d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f1573fff640 1 -- 172.21.15.98:0/509160351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5628450 msgr2=0x562dc5632d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:34.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f1573fff640 1 --2- 172.21.15.98:0/509160351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5628450 0x562dc5632d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f1573fff640 1 -- 172.21.15.98:0/509160351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dc5622f00 con 0x562dc5628d70 2024-05-09T09:00:34.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f1578b53640 1 --2- 172.21.15.98:0/509160351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5628450 0x562dc5632d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:34.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.133+0000 7f1573fff640 1 --2- 172.21.15.98:0/509160351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628d70 0x562dc563ed40 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x562dc56299f0 tx=0x7f1564009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5052b83b8c310cb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:34.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.134+0000 7f156a7fc640 1 -- 172.21.15.98:0/509160351 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156401adc0 con 0x562dc5628d70 2024-05-09T09:00:34.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.134+0000 7f156a7fc640 1 -- 172.21.15.98:0/509160351 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f156401f070 con 0x562dc5628d70 2024-05-09T09:00:34.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.134+0000 7f156a7fc640 1 -- 172.21.15.98:0/509160351 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15640043f0 con 0x562dc5628d70 2024-05-09T09:00:34.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.134+0000 7f157dba3740 1 -- 172.21.15.98:0/509160351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628d70 msgr2=0x562dc563ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:34.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.134+0000 7f157dba3740 1 --2- 172.21.15.98:0/509160351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628d70 0x562dc563ed40 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x562dc56299f0 tx=0x7f1564009760 comp rx=0 tx=0).stop 2024-05-09T09:00:34.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.134+0000 7f157dba3740 1 -- 172.21.15.98:0/509160351 shutdown_connections 2024-05-09T09:00:34.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.134+0000 7f157dba3740 1 --2- 172.21.15.98:0/509160351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5633630 0x562dc562d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.134+0000 7f157dba3740 1 --2- 172.21.15.98:0/509160351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628d70 0x562dc563ed40 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.134+0000 7f157dba3740 1 --2- 172.21.15.98:0/509160351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5628450 0x562dc5632d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.134+0000 7f157dba3740 1 -- 172.21.15.98:0/509160351 >> 172.21.15.98:0/509160351 conn(0x562dc5624960 msgr2=0x562dc563e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.134+0000 7f157dba3740 1 -- 172.21.15.98:0/509160351 shutdown_connections 2024-05-09T09:00:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.134+0000 7f157dba3740 1 -- 172.21.15.98:0/509160351 wait complete. 2024-05-09T09:00:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.134+0000 7f157dba3740 1 -- start start 2024-05-09T09:00:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f157dba3740 1 Processor -- start 2024-05-09T09:00:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f157dba3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5628450 0x562dc5648840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f157dba3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628d70 0x562dc5646be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f157dba3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5633630 0x562dc5647120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f157dba3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f15741b2530 con 0x562dc5628d70 2024-05-09T09:00:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f157dba3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f15741b1e60 con 0x562dc5633630 2024-05-09T09:00:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f157dba3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f15741af0c0 con 0x562dc5628450 2024-05-09T09:00:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f157dba3740 10 client.0 _ll_register_callbacks cb 0x7f156c04e030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:34.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f1573fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628d70 0x562dc5646be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f1573fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628d70 0x562dc5646be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44178/0 (socket says 172.21.15.98:44178) 2024-05-09T09:00:34.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f1573fff640 1 -- 172.21.15.98:0/522744812 learned_addr learned my addr 172.21.15.98:0/522744812 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:34.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f1579354640 1 --2- 172.21.15.98:0/522744812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5633630 0x562dc5647120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f1578b53640 1 --2- 172.21.15.98:0/522744812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5628450 0x562dc5648840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f1573fff640 1 -- 172.21.15.98:0/522744812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5628450 msgr2=0x562dc5648840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:34.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f1573fff640 1 --2- 172.21.15.98:0/522744812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5628450 0x562dc5648840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f1573fff640 1 -- 172.21.15.98:0/522744812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5633630 msgr2=0x562dc5647120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:34.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f1573fff640 1 --2- 172.21.15.98:0/522744812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5633630 0x562dc5647120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f1573fff640 1 -- 172.21.15.98:0/522744812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dc5647760 con 0x562dc5628d70 2024-05-09T09:00:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.135+0000 7f1579354640 1 --2- 172.21.15.98:0/522744812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5633630 0x562dc5647120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f1573fff640 1 --2- 172.21.15.98:0/522744812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628d70 0x562dc5646be0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f1564000f80 tx=0x7f156401ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8bb47e31c20ac6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f1553fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f1553fff640 1 -- 172.21.15.98:0/522744812 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1564004e50 con 0x562dc5628d70 2024-05-09T09:00:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f157dba3740 1 -- 172.21.15.98:0/522744812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f156401f070 con 0x562dc5628d70 2024-05-09T09:00:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f1553fff640 1 -- 172.21.15.98:0/522744812 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f1564003a30 con 0x562dc5628d70 2024-05-09T09:00:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f1553fff640 1 -- 172.21.15.98:0/522744812 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1564003c30 con 0x562dc5628d70 2024-05-09T09:00:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f157dba3740 10 client.18501 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f157dba3740 1 -- 172.21.15.98:0/522744812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562dc56aa240 con 0x562dc5628d70 2024-05-09T09:00:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f1553fff640 1 -- 172.21.15.98:0/522744812 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f156401f070 con 0x562dc5628d70 2024-05-09T09:00:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f1553fff640 1 -- 172.21.15.98:0/522744812 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f1564004610 con 0x562dc5628d70 2024-05-09T09:00:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f157dba3740 10 client.18501 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f157dba3740 10 client.18501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f157dba3740 20 client.18501 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f157dba3740 1 --2- 172.21.15.98:0/522744812 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562dc56aa970 0x562dc56acd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f157dba3740 4 client.18501 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:00:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f157dba3740 1 -- 172.21.15.98:0/522744812 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562dc5594dc0 con 0x562dc56aa970 2024-05-09T09:00:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.136+0000 7f1578b53640 1 --2- 172.21.15.98:0/522744812 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562dc56aa970 0x562dc56acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.137+0000 7f1553fff640 10 client.18501 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.137+0000 7f1578b53640 1 --2- 172.21.15.98:0/522744812 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562dc56aa970 0x562dc56acd90 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65e3f7649b7fa14f server_cookie=7df2d3512a55937f in_seq=0 out_seq=0 2024-05-09T09:00:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f1553fff640 1 -- 172.21.15.98:0/522744812 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562dc5594dc0 con 0x562dc56aa970 2024-05-09T09:00:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f1553fff640 10 client.18501 handle_command_reply: tid=0 2024-05-09T09:00:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f157dba3740 1 client.18501 shutdown 2024-05-09T09:00:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f157dba3740 1 -- 172.21.15.98:0/522744812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628d70 msgr2=0x562dc5646be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f157dba3740 1 --2- 172.21.15.98:0/522744812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628d70 0x562dc5646be0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f1564000f80 tx=0x7f156401ad70 comp rx=0 tx=0).stop 2024-05-09T09:00:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f157dba3740 1 -- 172.21.15.98:0/522744812 shutdown_connections 2024-05-09T09:00:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f157dba3740 1 --2- 172.21.15.98:0/522744812 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562dc56aa970 0x562dc56acd90 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f157dba3740 1 --2- 172.21.15.98:0/522744812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562dc5633630 0x562dc5647120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f157dba3740 1 --2- 172.21.15.98:0/522744812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562dc5628d70 0x562dc5646be0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f1553fff640 0 client.18501 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:34.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f157dba3740 1 --2- 172.21.15.98:0/522744812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562dc5628450 0x562dc5648840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f157dba3740 1 -- 172.21.15.98:0/522744812 >> 172.21.15.98:0/522744812 conn(0x562dc5624960 msgr2=0x562dc562a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:34.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f157dba3740 1 -- 172.21.15.98:0/522744812 shutdown_connections 2024-05-09T09:00:34.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f157dba3740 1 -- 172.21.15.98:0/522744812 wait complete. 2024-05-09T09:00:34.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.141+0000 7f157dba3740 20 client.18501 trim_cache size 0 max 16384 2024-05-09T09:00:34.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T09:00:34.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T09:00:34.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T09:00:34.145 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T09:00:34.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.142+0000 7f157adb7640 1 -- 172.21.15.98:0/406461511 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f15600a2da0 msgr2=0x7f15600a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:34.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.142+0000 7f157adb7640 1 --2- 172.21.15.98:0/406461511 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f15600a2da0 0x7f15600a5260 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f155c006010 tx=0x7f155c031db0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.142+0000 7f157adb7640 1 -- 172.21.15.98:0/406461511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1574100640 msgr2=0x7f157419a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:34.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.142+0000 7f157adb7640 1 --2- 172.21.15.98:0/406461511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1574100640 0x7f157419a280 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f156c00e9f0 tx=0x7f156c00edf0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.142+0000 7f157adb7640 1 -- 172.21.15.98:0/406461511 shutdown_connections 2024-05-09T09:00:34.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.142+0000 7f157adb7640 1 --2- 172.21.15.98:0/406461511 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f15600a2da0 0x7f15600a5260 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.142+0000 7f157adb7640 1 --2- 172.21.15.98:0/406461511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f157410e1f0 0x7f157419eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.142+0000 7f157adb7640 1 --2- 172.21.15.98:0/406461511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1574100f60 0x7f157419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.142+0000 7f157adb7640 1 --2- 172.21.15.98:0/406461511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1574100640 0x7f157419a280 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:34.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.142+0000 7f157adb7640 1 -- 172.21.15.98:0/406461511 >> 172.21.15.98:0/406461511 conn(0x7f15740778b0 msgr2=0x7f157410f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:34.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.142+0000 7f157adb7640 1 -- 172.21.15.98:0/406461511 shutdown_connections 2024-05-09T09:00:34.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:34.142+0000 7f157adb7640 1 -- 172.21.15.98:0/406461511 wait complete. 2024-05-09T09:00:34.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:34 smithi105 ceph-mon[34498]: pgmap v310: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:00:34.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:34 smithi098 ceph-mon[30092]: pgmap v310: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:00:34.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:34 smithi142 ceph-mon[34625]: pgmap v310: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:00:36.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:36 smithi105 ceph-mon[34498]: pgmap v311: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-05-09T09:00:36.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:36 smithi098 ceph-mon[30092]: pgmap v311: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-05-09T09:00:36.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:36 smithi142 ceph-mon[34625]: pgmap v311: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-05-09T09:00:38.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:38 smithi098 ceph-mon[30092]: pgmap v312: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 426 B/s wr, 14 op/s 2024-05-09T09:00:38.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:38 smithi142 ceph-mon[34625]: pgmap v312: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 426 B/s wr, 14 op/s 2024-05-09T09:00:38.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:38 smithi105 ceph-mon[34498]: pgmap v312: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 426 B/s wr, 14 op/s 2024-05-09T09:00:40.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:40 smithi098 ceph-mon[30092]: pgmap v313: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:40.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:40 smithi142 ceph-mon[34625]: pgmap v313: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:40.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:40 smithi105 ceph-mon[34498]: pgmap v313: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:42.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:42 smithi098 ceph-mon[30092]: pgmap v314: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:00:42.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:42 smithi142 ceph-mon[34625]: pgmap v314: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:00:42.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:42 smithi105 ceph-mon[34498]: pgmap v314: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:00:44.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:44 smithi098 ceph-mon[30092]: pgmap v315: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:44.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:44 smithi142 ceph-mon[34625]: pgmap v315: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:44.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:44 smithi105 ceph-mon[34498]: pgmap v315: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:46.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:46 smithi098 ceph-mon[30092]: pgmap v316: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:46.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:46 smithi142 ceph-mon[34625]: pgmap v316: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:46.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:46 smithi105 ceph-mon[34498]: pgmap v316: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.240+0000 7f5604a64640 1 -- 172.21.15.98:0/341318969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5600100640 msgr2=0x7f5600100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.240+0000 7f5604a64640 1 --2- 172.21.15.98:0/341318969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5600100640 0x7f5600100a20 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009900 tx=0x7f55f002a410 comp rx=0 tx=0).stop 2024-05-09T09:00:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.240+0000 7f5604a64640 1 -- 172.21.15.98:0/341318969 shutdown_connections 2024-05-09T09:00:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.240+0000 7f5604a64640 1 --2- 172.21.15.98:0/341318969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f560010e1f0 0x7f56001105e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.240+0000 7f5604a64640 1 --2- 172.21.15.98:0/341318969 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5600100f60 0x7f560010dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.240+0000 7f5604a64640 1 --2- 172.21.15.98:0/341318969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5600100640 0x7f5600100a20 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.240+0000 7f5604a64640 1 -- 172.21.15.98:0/341318969 >> 172.21.15.98:0/341318969 conn(0x7f56000778b0 msgr2=0x7f5600077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.241+0000 7f5604a64640 1 -- 172.21.15.98:0/341318969 shutdown_connections 2024-05-09T09:00:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.241+0000 7f5604a64640 1 -- 172.21.15.98:0/341318969 wait complete. 2024-05-09T09:00:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.241+0000 7f5604a64640 1 Processor -- start 2024-05-09T09:00:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f5604a64640 1 -- start start 2024-05-09T09:00:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f5604a64640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5600100640 0x7f560019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f5604a64640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5600100f60 0x7f560019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f5604a64640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f560010e1f0 0x7f560019eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f5604a64640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5600113de0 con 0x7f5600100f60 2024-05-09T09:00:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f5604a64640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5600113c60 con 0x7f5600100640 2024-05-09T09:00:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f5604a64640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5600113f60 con 0x7f560010e1f0 2024-05-09T09:00:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f55f5d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5600100f60 0x7f560019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f55f5d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5600100f60 0x7f560019a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51972/0 (socket says 172.21.15.98:51972) 2024-05-09T09:00:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f55f5d9b640 1 -- 172.21.15.98:0/1689889754 learned_addr learned my addr 172.21.15.98:0/1689889754 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f55fed9d640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f560010e1f0 0x7f560019eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f55fe59c640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5600100640 0x7f560019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f55f5d9b640 1 -- 172.21.15.98:0/1689889754 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f560010e1f0 msgr2=0x7f560019eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f55f5d9b640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f560010e1f0 0x7f560019eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f55f5d9b640 1 -- 172.21.15.98:0/1689889754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5600100640 msgr2=0x7f560019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f55f5d9b640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5600100640 0x7f560019a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f55f5d9b640 1 -- 172.21.15.98:0/1689889754 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f560019f250 con 0x7f5600100f60 2024-05-09T09:00:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f55fe59c640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5600100640 0x7f560019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f55f5d9b640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5600100f60 0x7f560019a7a0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f55e800c920 tx=0x7f55e800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.242+0000 7f55f7fff640 1 -- 172.21.15.98:0/1689889754 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e8007c70 con 0x7f5600100f60 2024-05-09T09:00:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.243+0000 7f5604a64640 1 -- 172.21.15.98:0/1689889754 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f560019f540 con 0x7f5600100f60 2024-05-09T09:00:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.243+0000 7f55f7fff640 1 -- 172.21.15.98:0/1689889754 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55e8007e10 con 0x7f5600100f60 2024-05-09T09:00:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.243+0000 7f55f7fff640 1 -- 172.21.15.98:0/1689889754 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e8018890 con 0x7f5600100f60 2024-05-09T09:00:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.243+0000 7f5604a64640 1 -- 172.21.15.98:0/1689889754 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56001a6dd0 con 0x7f5600100f60 2024-05-09T09:00:47.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.246+0000 7f5607771740 1 Processor -- start 2024-05-09T09:00:47.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.246+0000 7f55f7fff640 1 -- 172.21.15.98:0/1689889754 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f55e8018a70 con 0x7f5600100f60 2024-05-09T09:00:47.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.246+0000 7f5607771740 1 -- start start 2024-05-09T09:00:47.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.246+0000 7f5607771740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab27f450 0x5593ab27f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.246+0000 7f5607771740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27fd70 0x5593ab28a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.246+0000 7f5607771740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab28a630 0x5593ab28ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.246+0000 7f5607771740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5593ab1e4340 con 0x5593ab27fd70 2024-05-09T09:00:47.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.246+0000 7f5607771740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5593ab18c290 con 0x5593ab27f450 2024-05-09T09:00:47.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.246+0000 7f5607771740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5593ab1e0300 con 0x5593ab28a630 2024-05-09T09:00:47.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.246+0000 7f55fe59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27fd70 0x5593ab28a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.246+0000 7f55fe59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27fd70 0x5593ab28a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51988/0 (socket says 172.21.15.98:51988) 2024-05-09T09:00:47.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.246+0000 7f55fe59c640 1 -- 172.21.15.98:0/1542068917 learned_addr learned my addr 172.21.15.98:0/1542068917 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:47.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.247+0000 7f55f5d9b640 1 --2- 172.21.15.98:0/1542068917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab28a630 0x5593ab28ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.247+0000 7f55fed9d640 1 --2- 172.21.15.98:0/1542068917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab27f450 0x5593ab27f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.247+0000 7f55fe59c640 1 -- 172.21.15.98:0/1542068917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab28a630 msgr2=0x5593ab28ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:47.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.247+0000 7f55fe59c640 1 --2- 172.21.15.98:0/1542068917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab28a630 0x5593ab28ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.247+0000 7f55fe59c640 1 -- 172.21.15.98:0/1542068917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab27f450 msgr2=0x5593ab27f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.247+0000 7f55fe59c640 1 --2- 172.21.15.98:0/1542068917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab27f450 0x5593ab27f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.247+0000 7f55fe59c640 1 -- 172.21.15.98:0/1542068917 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593ab1b08e0 con 0x5593ab27fd70 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.247+0000 7f55f7fff640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f55ec0a2da0 0x7f55ec0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.247+0000 7f55fe59c640 1 --2- 172.21.15.98:0/1542068917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27fd70 0x5593ab28a0f0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f55f0000c00 tx=0x7f55f0032d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbfe8cba399b3d80 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.247+0000 7f55fed9d640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f55ec0a2da0 0x7f55ec0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.247+0000 7f55f7fff640 1 -- 172.21.15.98:0/1689889754 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f55e80cfa60 con 0x7f5600100f60 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.247+0000 7f55f4d99640 1 -- 172.21.15.98:0/1542068917 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f003f070 con 0x5593ab27fd70 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.247+0000 7f55f4d99640 1 -- 172.21.15.98:0/1542068917 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55f00317b0 con 0x5593ab27fd70 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.247+0000 7f55f4d99640 1 -- 172.21.15.98:0/1542068917 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0035070 con 0x5593ab27fd70 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.248+0000 7f5607771740 1 -- 172.21.15.98:0/1542068917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27fd70 msgr2=0x5593ab28a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.248+0000 7f55fed9d640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f55ec0a2da0 0x7f55ec0a5260 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f55f8005c40 tx=0x7f55f800a350 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.248+0000 7f5607771740 1 --2- 172.21.15.98:0/1542068917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27fd70 0x5593ab28a0f0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f55f0000c00 tx=0x7f55f0032d30 comp rx=0 tx=0).stop 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.248+0000 7f5607771740 1 -- 172.21.15.98:0/1542068917 shutdown_connections 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.248+0000 7f5607771740 1 --2- 172.21.15.98:0/1542068917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab28a630 0x5593ab28ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.248+0000 7f5607771740 1 --2- 172.21.15.98:0/1542068917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27fd70 0x5593ab28a0f0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.248+0000 7f5607771740 1 --2- 172.21.15.98:0/1542068917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab27f450 0x5593ab27f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.248+0000 7f5607771740 1 -- 172.21.15.98:0/1542068917 >> 172.21.15.98:0/1542068917 conn(0x5593ab27b960 msgr2=0x5593ab27bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.248+0000 7f5607771740 1 -- 172.21.15.98:0/1542068917 shutdown_connections 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.248+0000 7f5607771740 1 -- 172.21.15.98:0/1542068917 wait complete. 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.248+0000 7f5607771740 1 -- start start 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f5607771740 1 Processor -- start 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f5607771740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27f450 0x5593ab287510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f5607771740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27fd70 0x5593ab287a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f5607771740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab28a630 0x5593ab287f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f5607771740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5593ab28d200 con 0x5593ab27f450 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f5607771740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5593ab19f790 con 0x5593ab28a630 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f5607771740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5593ab28d380 con 0x5593ab27fd70 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f5607771740 10 client.0 _ll_register_callbacks cb 0x5593ab1daa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f55fe59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27f450 0x5593ab287510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f55fe59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27f450 0x5593ab287510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52002/0 (socket says 172.21.15.98:52002) 2024-05-09T09:00:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f55fe59c640 1 -- 172.21.15.98:0/1165334405 learned_addr learned my addr 172.21.15.98:0/1165334405 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f55f5d9b640 1 --2- 172.21.15.98:0/1165334405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27fd70 0x5593ab287a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f55fed9d640 1 --2- 172.21.15.98:0/1165334405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab28a630 0x5593ab287f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f55fe59c640 1 -- 172.21.15.98:0/1165334405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27fd70 msgr2=0x5593ab287a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f55fe59c640 1 --2- 172.21.15.98:0/1165334405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27fd70 0x5593ab287a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f55fe59c640 1 -- 172.21.15.98:0/1165334405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab28a630 msgr2=0x5593ab287f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f55fe59c640 1 --2- 172.21.15.98:0/1165334405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab28a630 0x5593ab287f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f55fe59c640 1 -- 172.21.15.98:0/1165334405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593aaef8210 con 0x5593ab27f450 2024-05-09T09:00:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f55fed9d640 1 --2- 172.21.15.98:0/1165334405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab28a630 0x5593ab287f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f55fe59c640 1 --2- 172.21.15.98:0/1165334405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27f450 0x5593ab287510 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f55f0032da0 tx=0x7f55f00315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffdd369528e4204e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.249+0000 7f55e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.250+0000 7f55e5ffb640 1 -- 172.21.15.98:0/1165334405 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f003f030 con 0x5593ab27f450 2024-05-09T09:00:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.250+0000 7f55e5ffb640 1 -- 172.21.15.98:0/1165334405 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f55f0004460 con 0x5593ab27f450 2024-05-09T09:00:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.250+0000 7f5607771740 1 -- 172.21.15.98:0/1165334405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55f00317b0 con 0x5593ab27f450 2024-05-09T09:00:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.252+0000 7f55e5ffb640 1 -- 172.21.15.98:0/1165334405 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f55f0054050 con 0x5593ab27f450 2024-05-09T09:00:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.252+0000 7f5607771740 10 client.18513 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.252+0000 7f5607771740 1 -- 172.21.15.98:0/1165334405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5593aafc4620 con 0x5593ab27f450 2024-05-09T09:00:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.253+0000 7f55e5ffb640 1 -- 172.21.15.98:0/1165334405 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55f00317b0 con 0x5593ab27f450 2024-05-09T09:00:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.253+0000 7f55e5ffb640 1 -- 172.21.15.98:0/1165334405 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f55f0004aa0 con 0x5593ab27f450 2024-05-09T09:00:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.253+0000 7f5607771740 10 client.18513 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.253+0000 7f5607771740 10 client.18513 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.253+0000 7f5607771740 20 client.18513 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.253+0000 7f5607771740 1 --2- 172.21.15.98:0/1165334405 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5593ab29a100 0x5593ab29c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.253+0000 7f5607771740 4 client.18513 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.253+0000 7f5607771740 1 -- 172.21.15.98:0/1165334405 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593ab1f1a70 con 0x5593ab29a100 2024-05-09T09:00:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.253+0000 7f55f5d9b640 1 --2- 172.21.15.98:0/1165334405 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5593ab29a100 0x5593ab29c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.253+0000 7f55f5d9b640 1 --2- 172.21.15.98:0/1165334405 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5593ab29a100 0x5593ab29c4f0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5baae7bef3ff1ed6 server_cookie=59086bef0babe202 in_seq=0 out_seq=0 2024-05-09T09:00:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.253+0000 7f55e5ffb640 10 client.18513 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.254+0000 7f55e5ffb640 1 -- 172.21.15.98:0/1165334405 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5593ab1f1a70 con 0x5593ab29a100 2024-05-09T09:00:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.254+0000 7f55e5ffb640 10 client.18513 handle_command_reply: tid=0 2024-05-09T09:00:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.254+0000 7f5607771740 1 client.18513 shutdown 2024-05-09T09:00:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.254+0000 7f5607771740 1 -- 172.21.15.98:0/1165334405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27f450 msgr2=0x5593ab287510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.254+0000 7f5607771740 1 --2- 172.21.15.98:0/1165334405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27f450 0x5593ab287510 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f55f0032da0 tx=0x7f55f00315b0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.254+0000 7f5607771740 1 -- 172.21.15.98:0/1165334405 shutdown_connections 2024-05-09T09:00:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.254+0000 7f5607771740 1 --2- 172.21.15.98:0/1165334405 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5593ab29a100 0x5593ab29c4f0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.254+0000 7f5607771740 1 --2- 172.21.15.98:0/1165334405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab28a630 0x5593ab287f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.254+0000 7f55e5ffb640 0 client.18513 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.254+0000 7f5607771740 1 --2- 172.21.15.98:0/1165334405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27fd70 0x5593ab287a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.254+0000 7f5607771740 1 --2- 172.21.15.98:0/1165334405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27f450 0x5593ab287510 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.254+0000 7f5607771740 1 -- 172.21.15.98:0/1165334405 >> 172.21.15.98:0/1165334405 conn(0x5593ab27b960 msgr2=0x5593ab2816e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.255+0000 7f5607771740 1 -- 172.21.15.98:0/1165334405 shutdown_connections 2024-05-09T09:00:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.255+0000 7f5607771740 1 -- 172.21.15.98:0/1165334405 wait complete. 2024-05-09T09:00:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.255+0000 7f5607771740 20 client.18513 trim_cache size 0 max 16384 2024-05-09T09:00:47.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.269+0000 7f5607771740 1 Processor -- start 2024-05-09T09:00:47.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.269+0000 7f5607771740 1 -- start start 2024-05-09T09:00:47.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.269+0000 7f5607771740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27f450 0x5593ab281d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f5607771740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab27fd70 0x5593ab27a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f5607771740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab28a630 0x5593ab286a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f5607771740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56001b1010 con 0x5593ab28a630 2024-05-09T09:00:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f5607771740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f56001b2b50 con 0x5593ab27fd70 2024-05-09T09:00:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f5607771740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f56001b16e0 con 0x5593ab27f450 2024-05-09T09:00:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55fed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab28a630 0x5593ab286a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55fed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab28a630 0x5593ab286a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52014/0 (socket says 172.21.15.98:52014) 2024-05-09T09:00:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55fed9d640 1 -- 172.21.15.98:0/93999002 learned_addr learned my addr 172.21.15.98:0/93999002 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55fe59c640 1 --2- 172.21.15.98:0/93999002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27f450 0x5593ab281d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55fed9d640 1 -- 172.21.15.98:0/93999002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27f450 msgr2=0x5593ab281d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55fed9d640 1 --2- 172.21.15.98:0/93999002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27f450 0x5593ab281d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55fed9d640 1 -- 172.21.15.98:0/93999002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab27fd70 msgr2=0x5593ab27a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55f5d9b640 1 --2- 172.21.15.98:0/93999002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab27fd70 0x5593ab27a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55fed9d640 1 --2- 172.21.15.98:0/93999002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab27fd70 0x5593ab27a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55fed9d640 1 -- 172.21.15.98:0/93999002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593ab29c860 con 0x5593ab28a630 2024-05-09T09:00:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55fe59c640 1 --2- 172.21.15.98:0/93999002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27f450 0x5593ab281d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55f5d9b640 1 --2- 172.21.15.98:0/93999002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab27fd70 0x5593ab27a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55fed9d640 1 --2- 172.21.15.98:0/93999002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab28a630 0x5593ab286a70 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f55f80127a0 tx=0x7f55f8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c6d9886ccd0921 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55e7fff640 1 -- 172.21.15.98:0/93999002 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f8004d20 con 0x5593ab28a630 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55e7fff640 1 -- 172.21.15.98:0/93999002 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55f800fe30 con 0x5593ab28a630 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.270+0000 7f55e7fff640 1 -- 172.21.15.98:0/93999002 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f8005960 con 0x5593ab28a630 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.271+0000 7f5607771740 1 -- 172.21.15.98:0/93999002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab28a630 msgr2=0x5593ab286a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.271+0000 7f5607771740 1 --2- 172.21.15.98:0/93999002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab28a630 0x5593ab286a70 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f55f80127a0 tx=0x7f55f8014000 comp rx=0 tx=0).stop 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.271+0000 7f5607771740 1 -- 172.21.15.98:0/93999002 shutdown_connections 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.271+0000 7f5607771740 1 --2- 172.21.15.98:0/93999002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab28a630 0x5593ab286a70 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.271+0000 7f5607771740 1 --2- 172.21.15.98:0/93999002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab27fd70 0x5593ab27a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.271+0000 7f5607771740 1 --2- 172.21.15.98:0/93999002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27f450 0x5593ab281d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.271+0000 7f5607771740 1 -- 172.21.15.98:0/93999002 >> 172.21.15.98:0/93999002 conn(0x5593ab27b960 msgr2=0x5593ab27bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.271+0000 7f5607771740 1 -- 172.21.15.98:0/93999002 shutdown_connections 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.271+0000 7f5607771740 1 -- 172.21.15.98:0/93999002 wait complete. 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.271+0000 7f5607771740 1 -- start start 2024-05-09T09:00:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.271+0000 7f5607771740 1 Processor -- start 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f5607771740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27f450 0x5593ab28d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f5607771740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27fd70 0x5593ab28dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f5607771740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab28a630 0x5593ab2fe2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f5607771740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56001b2480 con 0x5593ab27f450 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f5607771740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f56001b1db0 con 0x5593ab28a630 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f5607771740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f56001af010 con 0x5593ab27fd70 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f5607771740 10 client.0 _ll_register_callbacks cb 0x7f55f0032c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f55fe59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27f450 0x5593ab28d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f55fe59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27f450 0x5593ab28d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52030/0 (socket says 172.21.15.98:52030) 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f55fe59c640 1 -- 172.21.15.98:0/3609943490 learned_addr learned my addr 172.21.15.98:0/3609943490 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f55fe59c640 1 -- 172.21.15.98:0/3609943490 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27fd70 msgr2=0x5593ab28dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f55fe59c640 1 --2- 172.21.15.98:0/3609943490 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27fd70 0x5593ab28dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f55fe59c640 1 -- 172.21.15.98:0/3609943490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab28a630 msgr2=0x5593ab2fe2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f55fe59c640 1 --2- 172.21.15.98:0/3609943490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab28a630 0x5593ab2fe2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f55fe59c640 1 -- 172.21.15.98:0/3609943490 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593ab2fe830 con 0x5593ab27f450 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f55fe59c640 1 --2- 172.21.15.98:0/3609943490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27f450 0x5593ab28d590 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f55f002a450 tx=0x7f55f00316b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc11734deb29c88 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f55e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f55e5ffb640 1 -- 172.21.15.98:0/3609943490 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f003f030 con 0x5593ab27f450 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f55e5ffb640 1 -- 172.21.15.98:0/3609943490 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f55f0031be0 con 0x5593ab27f450 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f5607771740 1 -- 172.21.15.98:0/3609943490 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55f800fe30 con 0x5593ab27f450 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f55e5ffb640 1 -- 172.21.15.98:0/3609943490 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f55f0035040 con 0x5593ab27f450 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f5607771740 10 client.18519 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f5607771740 1 -- 172.21.15.98:0/3609943490 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5593ab2ff8c0 con 0x5593ab27f450 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.272+0000 7f55e5ffb640 1 -- 172.21.15.98:0/3609943490 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55f800fe30 con 0x5593ab27f450 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.273+0000 7f55e5ffb640 1 -- 172.21.15.98:0/3609943490 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f55f00035f0 con 0x5593ab27f450 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.273+0000 7f5607771740 10 client.18519 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.273+0000 7f5607771740 10 client.18519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.273+0000 7f5607771740 20 client.18519 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.273+0000 7f5607771740 1 --2- 172.21.15.98:0/3609943490 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5593ab29a100 0x5593ab3020f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.273+0000 7f5607771740 4 client.18519 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:00:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.273+0000 7f5607771740 1 -- 172.21.15.98:0/3609943490 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5593ab1ebdc0 con 0x5593ab29a100 2024-05-09T09:00:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.273+0000 7f55f5d9b640 1 --2- 172.21.15.98:0/3609943490 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5593ab29a100 0x5593ab3020f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.273+0000 7f55f5d9b640 1 --2- 172.21.15.98:0/3609943490 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5593ab29a100 0x5593ab3020f0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97ced685b992fbff server_cookie=6d367e7f747d87a0 in_seq=0 out_seq=0 2024-05-09T09:00:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.273+0000 7f55e5ffb640 10 client.18519 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:47.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f55e5ffb640 1 -- 172.21.15.98:0/3609943490 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5593ab1ebdc0 con 0x5593ab29a100 2024-05-09T09:00:47.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f55e5ffb640 10 client.18519 handle_command_reply: tid=0 2024-05-09T09:00:47.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f5607771740 1 client.18519 shutdown 2024-05-09T09:00:47.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f5607771740 1 -- 172.21.15.98:0/3609943490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27f450 msgr2=0x5593ab28d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:47.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f5607771740 1 --2- 172.21.15.98:0/3609943490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27f450 0x5593ab28d590 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f55f002a450 tx=0x7f55f00316b0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f5607771740 1 -- 172.21.15.98:0/3609943490 shutdown_connections 2024-05-09T09:00:47.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f5607771740 1 --2- 172.21.15.98:0/3609943490 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5593ab29a100 0x5593ab3020f0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f5607771740 1 --2- 172.21.15.98:0/3609943490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593ab28a630 0x5593ab2fe2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f55e5ffb640 0 client.18519 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:47.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f5607771740 1 --2- 172.21.15.98:0/3609943490 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593ab27fd70 0x5593ab28dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f5607771740 1 --2- 172.21.15.98:0/3609943490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5593ab27f450 0x5593ab28d590 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f5607771740 1 -- 172.21.15.98:0/3609943490 >> 172.21.15.98:0/3609943490 conn(0x5593ab27b960 msgr2=0x5593ab2950b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:47.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f5607771740 1 -- 172.21.15.98:0/3609943490 shutdown_connections 2024-05-09T09:00:47.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f5607771740 1 -- 172.21.15.98:0/3609943490 wait complete. 2024-05-09T09:00:47.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.278+0000 7f5607771740 20 client.18519 trim_cache size 0 max 16384 2024-05-09T09:00:47.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T09:00:47.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T09:00:47.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T09:00:47.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T09:00:47.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.279+0000 7f5604a64640 1 -- 172.21.15.98:0/1689889754 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f55ec0a2da0 msgr2=0x7f55ec0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:47.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.279+0000 7f5604a64640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f55ec0a2da0 0x7f55ec0a5260 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f55f8005c40 tx=0x7f55f800a350 comp rx=0 tx=0).stop 2024-05-09T09:00:47.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.279+0000 7f5604a64640 1 -- 172.21.15.98:0/1689889754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5600100f60 msgr2=0x7f560019a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:47.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.279+0000 7f5604a64640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5600100f60 0x7f560019a7a0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f55e800c920 tx=0x7f55e800cde0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.279+0000 7f5604a64640 1 -- 172.21.15.98:0/1689889754 shutdown_connections 2024-05-09T09:00:47.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.279+0000 7f5604a64640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f55ec0a2da0 0x7f55ec0a5260 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.279+0000 7f5604a64640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f560010e1f0 0x7f560019eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.279+0000 7f5604a64640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5600100f60 0x7f560019a7a0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.279+0000 7f5604a64640 1 --2- 172.21.15.98:0/1689889754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5600100640 0x7f560019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:47.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.279+0000 7f5604a64640 1 -- 172.21.15.98:0/1689889754 >> 172.21.15.98:0/1689889754 conn(0x7f56000778b0 msgr2=0x7f560010f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:47.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.279+0000 7f5604a64640 1 -- 172.21.15.98:0/1689889754 shutdown_connections 2024-05-09T09:00:47.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:00:47.279+0000 7f5604a64640 1 -- 172.21.15.98:0/1689889754 wait complete. 2024-05-09T09:00:48.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:48 smithi098 ceph-mon[30092]: pgmap v317: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:00:48.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:48 smithi142 ceph-mon[34625]: pgmap v317: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:00:48.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:48 smithi105 ceph-mon[34498]: pgmap v317: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:00:50.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:50 smithi098 ceph-mon[30092]: pgmap v318: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-05-09T09:00:50.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:50 smithi142 ceph-mon[34625]: pgmap v318: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-05-09T09:00:50.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:50 smithi105 ceph-mon[34498]: pgmap v318: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-05-09T09:00:52.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:52 smithi098 ceph-mon[30092]: pgmap v319: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 11 op/s 2024-05-09T09:00:52.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:52 smithi142 ceph-mon[34625]: pgmap v319: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 11 op/s 2024-05-09T09:00:52.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:52 smithi105 ceph-mon[34498]: pgmap v319: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 11 op/s 2024-05-09T09:00:54.301 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:00:54.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:00:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.418+0000 7f40e521c640 1 -- 172.21.15.98:0/2194967457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40e01021c0 msgr2=0x7f40e010e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.418+0000 7f40e521c640 1 --2- 172.21.15.98:0/2194967457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40e01021c0 0x7f40e010e6d0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f40d8009cd0 tx=0x7f40d802a8e0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.418+0000 7f40e521c640 1 -- 172.21.15.98:0/2194967457 shutdown_connections 2024-05-09T09:00:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.418+0000 7f40e521c640 1 --2- 172.21.15.98:0/2194967457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40e01021c0 0x7f40e010e6d0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.418+0000 7f40e521c640 1 --2- 172.21.15.98:0/2194967457 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40e0101800 0x7f40e0101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.418+0000 7f40e521c640 1 --2- 172.21.15.98:0/2194967457 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40e0107810 0x7f40e0107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.418+0000 7f40e521c640 1 -- 172.21.15.98:0/2194967457 >> 172.21.15.98:0/2194967457 conn(0x7f40e00fd270 msgr2=0x7f40e00ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:54.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.418+0000 7f40e521c640 1 -- 172.21.15.98:0/2194967457 shutdown_connections 2024-05-09T09:00:54.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.418+0000 7f40e521c640 1 -- 172.21.15.98:0/2194967457 wait complete. 2024-05-09T09:00:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.419+0000 7f40e521c640 1 Processor -- start 2024-05-09T09:00:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.419+0000 7f40e521c640 1 -- start start 2024-05-09T09:00:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.419+0000 7f40e521c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40e0101800 0x7f40e0073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40e521c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40e01021c0 0x7f40e0074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40e521c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40e0107810 0x7f40e00745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40e521c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f40e0111be0 con 0x7f40e01021c0 2024-05-09T09:00:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40e521c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f40e0111a60 con 0x7f40e0107810 2024-05-09T09:00:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40e521c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f40e0111d60 con 0x7f40e0101800 2024-05-09T09:00:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40de59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40e01021c0 0x7f40e0074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40de59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40e01021c0 0x7f40e0074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52050/0 (socket says 172.21.15.98:52050) 2024-05-09T09:00:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40de59c640 1 -- 172.21.15.98:0/2668117871 learned_addr learned my addr 172.21.15.98:0/2668117871 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40df59e640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40e0107810 0x7f40e00745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40ded9d640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40e0101800 0x7f40e0073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40de59c640 1 -- 172.21.15.98:0/2668117871 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40e0101800 msgr2=0x7f40e0073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40de59c640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40e0101800 0x7f40e0073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40de59c640 1 -- 172.21.15.98:0/2668117871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40e0107810 msgr2=0x7f40e00745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40de59c640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40e0107810 0x7f40e00745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40de59c640 1 -- 172.21.15.98:0/2668117871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e01a6640 con 0x7f40e01021c0 2024-05-09T09:00:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40df59e640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40e0107810 0x7f40e00745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40de59c640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40e01021c0 0x7f40e0074070 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f40d400e9f0 tx=0x7f40d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40c7fff640 1 -- 172.21.15.98:0/2668117871 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d400cd80 con 0x7f40e01021c0 2024-05-09T09:00:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40e521c640 1 -- 172.21.15.98:0/2668117871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40e01a6930 con 0x7f40e01021c0 2024-05-09T09:00:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40c7fff640 1 -- 172.21.15.98:0/2668117871 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40d4010910 con 0x7f40e01021c0 2024-05-09T09:00:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40c7fff640 1 -- 172.21.15.98:0/2668117871 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d40188f0 con 0x7f40e01021c0 2024-05-09T09:00:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.420+0000 7f40e521c640 1 -- 172.21.15.98:0/2668117871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40e01a6ed0 con 0x7f40e01021c0 2024-05-09T09:00:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.423+0000 7f40c7fff640 1 -- 172.21.15.98:0/2668117871 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f40d4010ac0 con 0x7f40e01021c0 2024-05-09T09:00:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.423+0000 7f40e8007740 1 Processor -- start 2024-05-09T09:00:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.423+0000 7f40e8007740 1 -- start start 2024-05-09T09:00:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.423+0000 7f40e8007740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69a280 0x55667b69a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.423+0000 7f40c7fff640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f40c80a2d00 0x7f40c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.423+0000 7f40e8007740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69aba0 0x55667b6a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.423+0000 7f40e8007740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b6a5270 0x55667b6a7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.423+0000 7f40e8007740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55667b5ffd00 con 0x55667b69aba0 2024-05-09T09:00:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.423+0000 7f40e8007740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55667b5a74c0 con 0x55667b69a280 2024-05-09T09:00:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.423+0000 7f40e8007740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55667b5fbcc0 con 0x55667b6a5270 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40ded9d640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f40c80a2d00 0x7f40c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40de59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69aba0 0x55667b6a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40c7fff640 1 -- 172.21.15.98:0/2668117871 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f40d40cfd50 con 0x7f40e01021c0 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40de59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69aba0 0x55667b6a4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52052/0 (socket says 172.21.15.98:52052) 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40de59c640 1 -- 172.21.15.98:0/3953132037 learned_addr learned my addr 172.21.15.98:0/3953132037 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40df59e640 1 --2- 172.21.15.98:0/3953132037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69a280 0x55667b69a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40df59e640 1 --2- 172.21.15.98:0/3953132037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b6a5270 0x55667b6a7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40de59c640 1 -- 172.21.15.98:0/3953132037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b6a5270 msgr2=0x55667b6a7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40de59c640 1 --2- 172.21.15.98:0/3953132037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b6a5270 0x55667b6a7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40de59c640 1 -- 172.21.15.98:0/3953132037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69a280 msgr2=0x55667b69a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40de59c640 1 --2- 172.21.15.98:0/3953132037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69a280 0x55667b69a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40de59c640 1 -- 172.21.15.98:0/3953132037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55667b3dd7d0 con 0x55667b69aba0 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40ded9d640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f40c80a2d00 0x7f40c80a51c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f40d0005fd0 tx=0x7f40d0005850 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40de59c640 1 --2- 172.21.15.98:0/3953132037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69aba0 0x55667b6a4d30 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f40d4097b40 tx=0x7f40d4025360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a2df405956583e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40c57fa640 1 -- 172.21.15.98:0/3953132037 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d40b0110 con 0x55667b69aba0 2024-05-09T09:00:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40c57fa640 1 -- 172.21.15.98:0/3953132037 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40d40a5990 con 0x55667b69aba0 2024-05-09T09:00:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.424+0000 7f40c57fa640 1 -- 172.21.15.98:0/3953132037 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d40984d0 con 0x55667b69aba0 2024-05-09T09:00:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.425+0000 7f40e8007740 1 -- 172.21.15.98:0/3953132037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69aba0 msgr2=0x55667b6a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.425+0000 7f40e8007740 1 --2- 172.21.15.98:0/3953132037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69aba0 0x55667b6a4d30 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f40d4097b40 tx=0x7f40d4025360 comp rx=0 tx=0).stop 2024-05-09T09:00:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.425+0000 7f40e8007740 1 -- 172.21.15.98:0/3953132037 shutdown_connections 2024-05-09T09:00:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.425+0000 7f40e8007740 1 --2- 172.21.15.98:0/3953132037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b6a5270 0x55667b6a7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.425+0000 7f40e8007740 1 --2- 172.21.15.98:0/3953132037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69aba0 0x55667b6a4d30 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.425+0000 7f40e8007740 1 --2- 172.21.15.98:0/3953132037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69a280 0x55667b69a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.425+0000 7f40e8007740 1 -- 172.21.15.98:0/3953132037 >> 172.21.15.98:0/3953132037 conn(0x55667b6965a0 msgr2=0x55667b696980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.425+0000 7f40e8007740 1 -- 172.21.15.98:0/3953132037 shutdown_connections 2024-05-09T09:00:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.425+0000 7f40e8007740 1 -- 172.21.15.98:0/3953132037 wait complete. 2024-05-09T09:00:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.425+0000 7f40e8007740 1 -- start start 2024-05-09T09:00:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.425+0000 7f40e8007740 1 Processor -- start 2024-05-09T09:00:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40e8007740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69a280 0x55667b6a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40e8007740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b69aba0 0x55667b6a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40e8007740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b6a5270 0x55667b6a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40e8007740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55667b6a8000 con 0x55667b69a280 2024-05-09T09:00:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40e8007740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55667b5bab90 con 0x55667b6a5270 2024-05-09T09:00:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40e8007740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55667b6a8180 con 0x55667b69aba0 2024-05-09T09:00:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40e8007740 10 client.0 _ll_register_callbacks cb 0x55667b5f6da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40ded9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69a280 0x55667b6a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40ded9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69a280 0x55667b6a21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52066/0 (socket says 172.21.15.98:52066) 2024-05-09T09:00:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40ded9d640 1 -- 172.21.15.98:0/1326267674 learned_addr learned my addr 172.21.15.98:0/1326267674 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40de59c640 1 --2- 172.21.15.98:0/1326267674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b69aba0 0x55667b6a26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40ded9d640 1 -- 172.21.15.98:0/1326267674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b69aba0 msgr2=0x55667b6a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40ded9d640 1 --2- 172.21.15.98:0/1326267674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b69aba0 0x55667b6a26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40ded9d640 1 -- 172.21.15.98:0/1326267674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b6a5270 msgr2=0x55667b6a2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40ded9d640 1 --2- 172.21.15.98:0/1326267674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b6a5270 0x55667b6a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40ded9d640 1 -- 172.21.15.98:0/1326267674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55667b3133f0 con 0x55667b69a280 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40ded9d640 1 --2- 172.21.15.98:0/1326267674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69a280 0x55667b6a21a0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f40d000b940 tx=0x7f40d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15883293eb40b285 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40c2ff5640 1 -- 172.21.15.98:0/1326267674 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0022070 con 0x55667b69a280 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40c2ff5640 1 -- 172.21.15.98:0/1326267674 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f40d001e860 con 0x55667b69a280 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40e8007740 1 -- 172.21.15.98:0/1326267674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40d40a5990 con 0x55667b69a280 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40c2ff5640 1 -- 172.21.15.98:0/1326267674 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f40d002e050 con 0x55667b69a280 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40e8007740 10 client.18531 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.426+0000 7f40e8007740 1 -- 172.21.15.98:0/1326267674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55667b2d30f0 con 0x55667b69a280 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.427+0000 7f40c2ff5640 1 -- 172.21.15.98:0/1326267674 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40d40a5990 con 0x55667b69a280 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.427+0000 7f40c2ff5640 1 -- 172.21.15.98:0/1326267674 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f40d0010b50 con 0x55667b69a280 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.427+0000 7f40e8007740 10 client.18531 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.427+0000 7f40e8007740 10 client.18531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.427+0000 7f40e8007740 20 client.18531 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.427+0000 7f40e8007740 1 --2- 172.21.15.98:0/1326267674 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55667b6b5560 0x55667b6b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.427+0000 7f40e8007740 4 client.18531 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.427+0000 7f40e8007740 1 -- 172.21.15.98:0/1326267674 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55667b60d430 con 0x55667b6b5560 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.427+0000 7f40de59c640 1 --2- 172.21.15.98:0/1326267674 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55667b6b5560 0x55667b6b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.427+0000 7f40c2ff5640 10 client.18531 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.427+0000 7f40de59c640 1 --2- 172.21.15.98:0/1326267674 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55667b6b5560 0x55667b6b7a00 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39b6ec58c9d29b3a server_cookie=71a42a73db2f5e58 in_seq=0 out_seq=0 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40c2ff5640 1 -- 172.21.15.98:0/1326267674 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55667b60d430 con 0x55667b6b5560 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40c2ff5640 10 client.18531 handle_command_reply: tid=0 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40e8007740 1 client.18531 shutdown 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40e8007740 1 -- 172.21.15.98:0/1326267674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69a280 msgr2=0x55667b6a21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40e8007740 1 --2- 172.21.15.98:0/1326267674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69a280 0x55667b6a21a0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f40d000b940 tx=0x7f40d00046c0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40e8007740 1 -- 172.21.15.98:0/1326267674 shutdown_connections 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40e8007740 1 --2- 172.21.15.98:0/1326267674 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55667b6b5560 0x55667b6b7a00 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40e8007740 1 --2- 172.21.15.98:0/1326267674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b6a5270 0x55667b6a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40e8007740 1 --2- 172.21.15.98:0/1326267674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b69aba0 0x55667b6a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40c2ff5640 0 client.18531 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40e8007740 1 --2- 172.21.15.98:0/1326267674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69a280 0x55667b6a21a0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40e8007740 1 -- 172.21.15.98:0/1326267674 >> 172.21.15.98:0/1326267674 conn(0x55667b6965a0 msgr2=0x55667b6957e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40e8007740 1 -- 172.21.15.98:0/1326267674 shutdown_connections 2024-05-09T09:00:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40e8007740 1 -- 172.21.15.98:0/1326267674 wait complete. 2024-05-09T09:00:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.428+0000 7f40e8007740 20 client.18531 trim_cache size 0 max 16384 2024-05-09T09:00:54.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40e8007740 1 Processor -- start 2024-05-09T09:00:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40e8007740 1 -- start start 2024-05-09T09:00:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40e8007740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69a280 0x55667b6b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40e8007740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69aba0 0x55667b69c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40e8007740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b6a5270 0x55667b6bd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40e8007740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f40e01b1110 con 0x55667b69a280 2024-05-09T09:00:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40e8007740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f40e01b2c50 con 0x55667b69aba0 2024-05-09T09:00:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40e8007740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f40e01b17e0 con 0x55667b6a5270 2024-05-09T09:00:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40ded9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69a280 0x55667b6b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40ded9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69a280 0x55667b6b0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52070/0 (socket says 172.21.15.98:52070) 2024-05-09T09:00:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40ded9d640 1 -- 172.21.15.98:0/3379224591 learned_addr learned my addr 172.21.15.98:0/3379224591 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40de59c640 1 --2- 172.21.15.98:0/3379224591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69aba0 0x55667b69c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40ded9d640 1 -- 172.21.15.98:0/3379224591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b6a5270 msgr2=0x55667b6bd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40ded9d640 1 --2- 172.21.15.98:0/3379224591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b6a5270 0x55667b6bd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40ded9d640 1 -- 172.21.15.98:0/3379224591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69aba0 msgr2=0x55667b69c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40ded9d640 1 --2- 172.21.15.98:0/3379224591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69aba0 0x55667b69c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40ded9d640 1 -- 172.21.15.98:0/3379224591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55667b69b0a0 con 0x55667b69a280 2024-05-09T09:00:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40de59c640 1 --2- 172.21.15.98:0/3379224591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69aba0 0x55667b69c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.446+0000 7f40ded9d640 1 --2- 172.21.15.98:0/3379224591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69a280 0x55667b6b0ef0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f40d0000c00 tx=0x7f40d0005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d75df124d3911119 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.447+0000 7f40c4ff9640 1 -- 172.21.15.98:0/3379224591 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0036070 con 0x55667b69a280 2024-05-09T09:00:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.447+0000 7f40c4ff9640 1 -- 172.21.15.98:0/3379224591 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40d0026e50 con 0x55667b69a280 2024-05-09T09:00:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.447+0000 7f40c4ff9640 1 -- 172.21.15.98:0/3379224591 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0022050 con 0x55667b69a280 2024-05-09T09:00:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.447+0000 7f40e8007740 1 -- 172.21.15.98:0/3379224591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69a280 msgr2=0x55667b6b0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.447+0000 7f40e8007740 1 --2- 172.21.15.98:0/3379224591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69a280 0x55667b6b0ef0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f40d0000c00 tx=0x7f40d0005c70 comp rx=0 tx=0).stop 2024-05-09T09:00:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.447+0000 7f40e8007740 1 -- 172.21.15.98:0/3379224591 shutdown_connections 2024-05-09T09:00:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.447+0000 7f40e8007740 1 --2- 172.21.15.98:0/3379224591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b6a5270 0x55667b6bd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.447+0000 7f40e8007740 1 --2- 172.21.15.98:0/3379224591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69aba0 0x55667b69c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.447+0000 7f40e8007740 1 --2- 172.21.15.98:0/3379224591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b69a280 0x55667b6b0ef0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.447+0000 7f40e8007740 1 -- 172.21.15.98:0/3379224591 >> 172.21.15.98:0/3379224591 conn(0x55667b6965a0 msgr2=0x55667b69b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.447+0000 7f40e8007740 1 -- 172.21.15.98:0/3379224591 shutdown_connections 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.447+0000 7f40e8007740 1 -- 172.21.15.98:0/3379224591 wait complete. 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.447+0000 7f40e8007740 1 -- start start 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40e8007740 1 Processor -- start 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40e8007740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b69a280 0x55667b6ad6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40e8007740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69aba0 0x55667b6a8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40e8007740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b6a5270 0x55667b6a8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40e8007740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f40e01b2580 con 0x55667b6a5270 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40e8007740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f40e01b1eb0 con 0x55667b69aba0 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40e8007740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f40e01af110 con 0x55667b69a280 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40e8007740 10 client.0 _ll_register_callbacks cb 0x7f40d0015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40df59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b6a5270 0x55667b6a8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40df59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b6a5270 0x55667b6a8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52084/0 (socket says 172.21.15.98:52084) 2024-05-09T09:00:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40df59e640 1 -- 172.21.15.98:0/1208382879 learned_addr learned my addr 172.21.15.98:0/1208382879 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40ded9d640 1 --2- 172.21.15.98:0/1208382879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b69a280 0x55667b6ad6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40de59c640 1 --2- 172.21.15.98:0/1208382879 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69aba0 0x55667b6a8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40df59e640 1 -- 172.21.15.98:0/1208382879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b69a280 msgr2=0x55667b6ad6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40df59e640 1 --2- 172.21.15.98:0/1208382879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b69a280 0x55667b6ad6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40df59e640 1 -- 172.21.15.98:0/1208382879 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69aba0 msgr2=0x55667b6a8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40df59e640 1 --2- 172.21.15.98:0/1208382879 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69aba0 0x55667b6a8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40df59e640 1 -- 172.21.15.98:0/1208382879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55667b6a96e0 con 0x55667b6a5270 2024-05-09T09:00:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40df59e640 1 --2- 172.21.15.98:0/1208382879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b6a5270 0x55667b6a8fa0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f40d8009cd0 tx=0x7f40d8031af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2dd27127eef1b38 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.448+0000 7f40c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40c2ff5640 1 -- 172.21.15.98:0/1208382879 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d8003b70 con 0x55667b6a5270 2024-05-09T09:00:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40c2ff5640 1 -- 172.21.15.98:0/1208382879 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f40d8003d10 con 0x55667b6a5270 2024-05-09T09:00:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40e8007740 1 -- 172.21.15.98:0/1208382879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40d0026e50 con 0x55667b6a5270 2024-05-09T09:00:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40c2ff5640 1 -- 172.21.15.98:0/1208382879 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f40d8055050 con 0x55667b6a5270 2024-05-09T09:00:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40e8007740 10 client.18537 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40e8007740 1 -- 172.21.15.98:0/1208382879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55667b71b9f0 con 0x55667b6a5270 2024-05-09T09:00:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40c2ff5640 1 -- 172.21.15.98:0/1208382879 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40d0026e50 con 0x55667b6a5270 2024-05-09T09:00:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40c2ff5640 1 -- 172.21.15.98:0/1208382879 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f40d8030070 con 0x55667b6a5270 2024-05-09T09:00:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40e8007740 10 client.18537 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40e8007740 10 client.18537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40e8007740 20 client.18537 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40e8007740 1 --2- 172.21.15.98:0/1208382879 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55667b71c160 0x55667b71e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40e8007740 4 client.18537 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:00:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40e8007740 1 -- 172.21.15.98:0/1208382879 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55667b607780 con 0x55667b71c160 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.449+0000 7f40ded9d640 1 --2- 172.21.15.98:0/1208382879 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55667b71c160 0x55667b71e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40c2ff5640 10 client.18537 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40ded9d640 1 --2- 172.21.15.98:0/1208382879 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55667b71c160 0x55667b71e580 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ee64213e2600930 server_cookie=81434e71ce9b1171 in_seq=0 out_seq=0 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40c2ff5640 1 -- 172.21.15.98:0/1208382879 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55667b607780 con 0x55667b71c160 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40c2ff5640 10 client.18537 handle_command_reply: tid=0 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40e8007740 1 client.18537 shutdown 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40e8007740 1 -- 172.21.15.98:0/1208382879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b6a5270 msgr2=0x55667b6a8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40e8007740 1 --2- 172.21.15.98:0/1208382879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b6a5270 0x55667b6a8fa0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f40d8009cd0 tx=0x7f40d8031af0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40e8007740 1 -- 172.21.15.98:0/1208382879 shutdown_connections 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40e8007740 1 --2- 172.21.15.98:0/1208382879 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55667b71c160 0x55667b71e580 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40e8007740 1 --2- 172.21.15.98:0/1208382879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55667b6a5270 0x55667b6a8fa0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40e8007740 1 --2- 172.21.15.98:0/1208382879 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55667b69aba0 0x55667b6a8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40c2ff5640 0 client.18537 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40e8007740 1 --2- 172.21.15.98:0/1208382879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55667b69a280 0x55667b6ad6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40e8007740 1 -- 172.21.15.98:0/1208382879 >> 172.21.15.98:0/1208382879 conn(0x55667b6965a0 msgr2=0x55667b6a0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40e8007740 1 -- 172.21.15.98:0/1208382879 shutdown_connections 2024-05-09T09:00:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40e8007740 1 -- 172.21.15.98:0/1208382879 wait complete. 2024-05-09T09:00:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.450+0000 7f40e8007740 20 client.18537 trim_cache size 0 max 16384 2024-05-09T09:00:54.458 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:00:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T09:00:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T09:00:54.458 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:00:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.451+0000 7f40e521c640 1 -- 172.21.15.98:0/2668117871 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f40c80a2d00 msgr2=0x7f40c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.451+0000 7f40e521c640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f40c80a2d00 0x7f40c80a51c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f40d0005fd0 tx=0x7f40d0005850 comp rx=0 tx=0).stop 2024-05-09T09:00:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.451+0000 7f40e521c640 1 -- 172.21.15.98:0/2668117871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40e01021c0 msgr2=0x7f40e0074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.451+0000 7f40e521c640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40e01021c0 0x7f40e0074070 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f40d400e9f0 tx=0x7f40d400eeb0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.451+0000 7f40e521c640 1 -- 172.21.15.98:0/2668117871 shutdown_connections 2024-05-09T09:00:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.451+0000 7f40e521c640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f40c80a2d00 0x7f40c80a51c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.451+0000 7f40e521c640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40e0107810 0x7f40e00745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.451+0000 7f40e521c640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40e01021c0 0x7f40e0074070 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.451+0000 7f40e521c640 1 --2- 172.21.15.98:0/2668117871 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40e0101800 0x7f40e0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.451+0000 7f40e521c640 1 -- 172.21.15.98:0/2668117871 >> 172.21.15.98:0/2668117871 conn(0x7f40e00fd270 msgr2=0x7f40e00fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.451+0000 7f40e521c640 1 -- 172.21.15.98:0/2668117871 shutdown_connections 2024-05-09T09:00:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.451+0000 7f40e521c640 1 -- 172.21.15.98:0/2668117871 wait complete. 2024-05-09T09:00:54.461 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T09:00:54.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:00:54.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.565+0000 7f892c514640 1 -- 172.21.15.98:0/3484768994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8924101800 msgr2=0x7f8924101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.565+0000 7f892c514640 1 --2- 172.21.15.98:0/3484768994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8924101800 0x7f8924101c80 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f8914009930 tx=0x7f891402a420 comp rx=0 tx=0).stop 2024-05-09T09:00:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.565+0000 7f892c514640 1 -- 172.21.15.98:0/3484768994 shutdown_connections 2024-05-09T09:00:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.565+0000 7f892c514640 1 --2- 172.21.15.98:0/3484768994 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f89241021c0 0x7f892410e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.565+0000 7f892c514640 1 --2- 172.21.15.98:0/3484768994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8924101800 0x7f8924101c80 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.565+0000 7f892c514640 1 --2- 172.21.15.98:0/3484768994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8924107810 0x7f8924107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.565+0000 7f892c514640 1 -- 172.21.15.98:0/3484768994 >> 172.21.15.98:0/3484768994 conn(0x7f8924078bf0 msgr2=0x7f8924079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.565+0000 7f892c514640 1 -- 172.21.15.98:0/3484768994 shutdown_connections 2024-05-09T09:00:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.565+0000 7f892c514640 1 -- 172.21.15.98:0/3484768994 wait complete. 2024-05-09T09:00:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f892c514640 1 Processor -- start 2024-05-09T09:00:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f892c514640 1 -- start start 2024-05-09T09:00:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f892c514640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8924101800 0x7f8924198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f892c514640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89241021c0 0x7f8924198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f892c514640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8924107810 0x7f892419c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f892c514640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8924111c20 con 0x7f89241021c0 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f892c514640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8924111aa0 con 0x7f8924107810 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f892c514640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8924111da0 con 0x7f8924101800 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f8929aaf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89241021c0 0x7f8924198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f8929aaf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89241021c0 0x7f8924198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52096/0 (socket says 172.21.15.98:52096) 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f8929aaf640 1 -- 172.21.15.98:0/2346178472 learned_addr learned my addr 172.21.15.98:0/2346178472 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f892a2b0640 1 --2- 172.21.15.98:0/2346178472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8924101800 0x7f8924198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f892aab1640 1 --2- 172.21.15.98:0/2346178472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8924107810 0x7f892419c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f8929aaf640 1 -- 172.21.15.98:0/2346178472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8924101800 msgr2=0x7f8924198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f8929aaf640 1 --2- 172.21.15.98:0/2346178472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8924101800 0x7f8924198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f8929aaf640 1 -- 172.21.15.98:0/2346178472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8924107810 msgr2=0x7f892419c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f8929aaf640 1 --2- 172.21.15.98:0/2346178472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8924107810 0x7f892419c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.566+0000 7f8929aaf640 1 -- 172.21.15.98:0/2346178472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f892419d0f0 con 0x7f89241021c0 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.567+0000 7f8929aaf640 1 --2- 172.21.15.98:0/2346178472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89241021c0 0x7f8924198640 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f8914009230 tx=0x7f8914004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.567+0000 7f891b7fe640 1 -- 172.21.15.98:0/2346178472 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891403f030 con 0x7f89241021c0 2024-05-09T09:00:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.567+0000 7f892c514640 1 -- 172.21.15.98:0/2346178472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f892419d380 con 0x7f89241021c0 2024-05-09T09:00:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.567+0000 7f891b7fe640 1 -- 172.21.15.98:0/2346178472 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8914003ad0 con 0x7f89241021c0 2024-05-09T09:00:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.567+0000 7f891b7fe640 1 -- 172.21.15.98:0/2346178472 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8914035070 con 0x7f89241021c0 2024-05-09T09:00:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.567+0000 7f892c514640 1 -- 172.21.15.98:0/2346178472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89241a4c70 con 0x7f89241021c0 2024-05-09T09:00:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.568+0000 7f892c514640 1 -- 172.21.15.98:0/2346178472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8924068aa0 con 0x7f89241021c0 2024-05-09T09:00:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.569+0000 7f891b7fe640 1 -- 172.21.15.98:0/2346178472 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8914042030 con 0x7f89241021c0 2024-05-09T09:00:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.569+0000 7f891b7fe640 1 --2- 172.21.15.98:0/2346178472 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89100a2d00 0x7f89100a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.570+0000 7f892a2b0640 1 --2- 172.21.15.98:0/2346178472 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89100a2d00 0x7f89100a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.570+0000 7f891b7fe640 1 -- 172.21.15.98:0/2346178472 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f89140f0100 con 0x7f89241021c0 2024-05-09T09:00:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.570+0000 7f892a2b0640 1 --2- 172.21.15.98:0/2346178472 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89100a2d00 0x7f89100a51c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f89200058c0 tx=0x7f892000a2d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.573+0000 7f891b7fe640 1 -- 172.21.15.98:0/2346178472 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f89140bc910 con 0x7f89241021c0 2024-05-09T09:00:54.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:54 smithi098 ceph-mon[30092]: pgmap v320: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:54.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:54 smithi142 ceph-mon[34625]: pgmap v320: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:54.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:54 smithi105 ceph-mon[34498]: pgmap v320: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.790+0000 7f892c514640 1 -- 172.21.15.98:0/2346178472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8924109d70 con 0x7f89241021c0 2024-05-09T09:00:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.791+0000 7f891b7fe640 1 -- 172.21.15.98:0/2346178472 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11941 (secure 0 0 0) 0x7f89140bc210 con 0x7f89241021c0 2024-05-09T09:00:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T09:00:54.791 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":26581,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/561027115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":561027115},{"type":"v1","addr":"172.21.15.105:6835","nonce":561027115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:56:52.891273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16641":{"gid":16641,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T09:00:54.791 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 37 2024-05-09T09:00:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.793+0000 7f892c514640 1 -- 172.21.15.98:0/2346178472 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89100a2d00 msgr2=0x7f89100a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.793+0000 7f892c514640 1 --2- 172.21.15.98:0/2346178472 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89100a2d00 0x7f89100a51c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f89200058c0 tx=0x7f892000a2d0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.793+0000 7f892c514640 1 -- 172.21.15.98:0/2346178472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89241021c0 msgr2=0x7f8924198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.793+0000 7f892c514640 1 --2- 172.21.15.98:0/2346178472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89241021c0 0x7f8924198640 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f8914009230 tx=0x7f8914004270 comp rx=0 tx=0).stop 2024-05-09T09:00:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.793+0000 7f892c514640 1 -- 172.21.15.98:0/2346178472 shutdown_connections 2024-05-09T09:00:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.794+0000 7f892c514640 1 --2- 172.21.15.98:0/2346178472 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f89100a2d00 0x7f89100a51c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.794+0000 7f892c514640 1 --2- 172.21.15.98:0/2346178472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8924107810 0x7f892419c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.794+0000 7f892c514640 1 --2- 172.21.15.98:0/2346178472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89241021c0 0x7f8924198640 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.794+0000 7f892c514640 1 --2- 172.21.15.98:0/2346178472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8924101800 0x7f8924198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.794+0000 7f892c514640 1 -- 172.21.15.98:0/2346178472 >> 172.21.15.98:0/2346178472 conn(0x7f8924078bf0 msgr2=0x7f8924104200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:54.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.794+0000 7f892c514640 1 -- 172.21.15.98:0/2346178472 shutdown_connections 2024-05-09T09:00:54.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.794+0000 7f892c514640 1 -- 172.21.15.98:0/2346178472 wait complete. 2024-05-09T09:00:54.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T09:00:54.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.909+0000 7f5431aba640 1 -- 172.21.15.98:0/1335201118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c1010c0 msgr2=0x7f542c10d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.909+0000 7f5431aba640 1 --2- 172.21.15.98:0/1335201118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c1010c0 0x7f542c10d5d0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f5424009cd0 tx=0x7f542402a8e0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.909+0000 7f5431aba640 1 -- 172.21.15.98:0/1335201118 shutdown_connections 2024-05-09T09:00:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.909+0000 7f5431aba640 1 --2- 172.21.15.98:0/1335201118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c1010c0 0x7f542c10d5d0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.909+0000 7f5431aba640 1 --2- 172.21.15.98:0/1335201118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f542c100700 0x7f542c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.909+0000 7f5431aba640 1 --2- 172.21.15.98:0/1335201118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f542c106710 0x7f542c106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.909+0000 7f5431aba640 1 -- 172.21.15.98:0/1335201118 >> 172.21.15.98:0/1335201118 conn(0x7f542c077830 msgr2=0x7f542c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.909+0000 7f5431aba640 1 -- 172.21.15.98:0/1335201118 shutdown_connections 2024-05-09T09:00:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.909+0000 7f5431aba640 1 -- 172.21.15.98:0/1335201118 wait complete. 2024-05-09T09:00:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.909+0000 7f5431aba640 1 Processor -- start 2024-05-09T09:00:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.909+0000 7f5431aba640 1 -- start start 2024-05-09T09:00:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f5431aba640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f542c100700 0x7f542c198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f5431aba640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f542c1010c0 0x7f542c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f5431aba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c106710 0x7f542c19cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f5431aba640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f542c110ba0 con 0x7f542c106710 2024-05-09T09:00:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f5431aba640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f542c110a20 con 0x7f542c100700 2024-05-09T09:00:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f5431aba640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f542c110d20 con 0x7f542c1010c0 2024-05-09T09:00:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f542bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c106710 0x7f542c19cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f542bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c106710 0x7f542c19cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52120/0 (socket says 172.21.15.98:52120) 2024-05-09T09:00:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f542bfff640 1 -- 172.21.15.98:0/2970150421 learned_addr learned my addr 172.21.15.98:0/2970150421 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f542b7fe640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f542c100700 0x7f542c198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f542bfff640 1 -- 172.21.15.98:0/2970150421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f542c1010c0 msgr2=0x7f542c198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f542affd640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f542c1010c0 0x7f542c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f542bfff640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f542c1010c0 0x7f542c198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f542bfff640 1 -- 172.21.15.98:0/2970150421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f542c100700 msgr2=0x7f542c198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f542bfff640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f542c100700 0x7f542c198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f542bfff640 1 -- 172.21.15.98:0/2970150421 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542c19d200 con 0x7f542c106710 2024-05-09T09:00:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f542b7fe640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f542c100700 0x7f542c198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f542affd640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f542c1010c0 0x7f542c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f542bfff640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c106710 0x7f542c19cad0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f5424006010 tx=0x7f5424003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f5428ff9640 1 -- 172.21.15.98:0/2970150421 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424004450 con 0x7f542c106710 2024-05-09T09:00:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f5428ff9640 1 -- 172.21.15.98:0/2970150421 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5424003e10 con 0x7f542c106710 2024-05-09T09:00:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f5431aba640 1 -- 172.21.15.98:0/2970150421 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f542c19d490 con 0x7f542c106710 2024-05-09T09:00:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f5428ff9640 1 -- 172.21.15.98:0/2970150421 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54240466e0 con 0x7f542c106710 2024-05-09T09:00:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.910+0000 7f5431aba640 1 -- 172.21.15.98:0/2970150421 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f542c1a4d80 con 0x7f542c106710 2024-05-09T09:00:54.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f5428ff9640 1 -- 172.21.15.98:0/2970150421 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f54240305a0 con 0x7f542c106710 2024-05-09T09:00:54.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f54347c6740 1 Processor -- start 2024-05-09T09:00:54.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f54347c6740 1 -- start start 2024-05-09T09:00:54.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f54347c6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8849330 0x563cc8849710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f54347c6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 0x563cc8853de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f54347c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8854320 0x563cc8856710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f54347c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563cc87aec40 con 0x563cc8854320 2024-05-09T09:00:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f54347c6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563cc8756580 con 0x563cc8849c50 2024-05-09T09:00:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f54347c6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563cc87aac00 con 0x563cc8849330 2024-05-09T09:00:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f542affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8854320 0x563cc8856710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f5428ff9640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54180a2d00 0x7f54180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f542affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8854320 0x563cc8856710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52130/0 (socket says 172.21.15.98:52130) 2024-05-09T09:00:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f542affd640 1 -- 172.21.15.98:0/2213562860 learned_addr learned my addr 172.21.15.98:0/2213562860 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f542b7fe640 1 --2- 172.21.15.98:0/2213562860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 0x563cc8853de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f542bfff640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54180a2d00 0x7f54180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.913+0000 7f542affd640 1 --2- 172.21.15.98:0/2213562860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8849330 0x563cc8849710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.914+0000 7f542affd640 1 -- 172.21.15.98:0/2213562860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8849330 msgr2=0x563cc8849710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.914+0000 7f542affd640 1 --2- 172.21.15.98:0/2213562860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8849330 0x563cc8849710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.914+0000 7f542affd640 1 -- 172.21.15.98:0/2213562860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 msgr2=0x563cc8853de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.914+0000 7f542affd640 1 --2- 172.21.15.98:0/2213562860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 0x563cc8853de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.914+0000 7f542affd640 1 -- 172.21.15.98:0/2213562860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cc84c23f0 con 0x563cc8854320 2024-05-09T09:00:54.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.914+0000 7f5428ff9640 1 -- 172.21.15.98:0/2970150421 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f542402e030 con 0x7f542c106710 2024-05-09T09:00:54.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.914+0000 7f542affd640 1 --2- 172.21.15.98:0/2213562860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8854320 0x563cc8856710 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f541c00cf10 tx=0x7f541c00c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=733a9592497af24a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:54.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.914+0000 7f542bfff640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54180a2d00 0x7f54180a51c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f54240b9820 tx=0x7f5424090430 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:54.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.914+0000 7f5421ffb640 1 -- 172.21.15.98:0/2213562860 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c01b070 con 0x563cc8854320 2024-05-09T09:00:54.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.914+0000 7f5421ffb640 1 -- 172.21.15.98:0/2213562860 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f541c00ee40 con 0x563cc8854320 2024-05-09T09:00:54.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.914+0000 7f5421ffb640 1 -- 172.21.15.98:0/2213562860 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c00f040 con 0x563cc8854320 2024-05-09T09:00:54.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 -- 172.21.15.98:0/2213562860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8854320 msgr2=0x563cc8856710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 --2- 172.21.15.98:0/2213562860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8854320 0x563cc8856710 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f541c00cf10 tx=0x7f541c00c410 comp rx=0 tx=0).stop 2024-05-09T09:00:54.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 -- 172.21.15.98:0/2213562860 shutdown_connections 2024-05-09T09:00:54.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 --2- 172.21.15.98:0/2213562860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8854320 0x563cc8856710 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 --2- 172.21.15.98:0/2213562860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 0x563cc8853de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 --2- 172.21.15.98:0/2213562860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8849330 0x563cc8849710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 -- 172.21.15.98:0/2213562860 >> 172.21.15.98:0/2213562860 conn(0x563cc8845650 msgr2=0x563cc8845a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 -- 172.21.15.98:0/2213562860 shutdown_connections 2024-05-09T09:00:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 -- 172.21.15.98:0/2213562860 wait complete. 2024-05-09T09:00:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 -- start start 2024-05-09T09:00:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 Processor -- start 2024-05-09T09:00:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849330 0x563cc88512e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 0x563cc8851820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 0x563cc8851d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563cc8857110 con 0x563cc8849330 2024-05-09T09:00:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563cc8769c40 con 0x563cc8849c50 2024-05-09T09:00:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563cc8857290 con 0x563cc8854320 2024-05-09T09:00:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f54347c6740 10 client.0 _ll_register_callbacks cb 0x563cc84c1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.915+0000 7f542affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849330 0x563cc88512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f542affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849330 0x563cc88512e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52142/0 (socket says 172.21.15.98:52142) 2024-05-09T09:00:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f542affd640 1 -- 172.21.15.98:0/3211835161 learned_addr learned my addr 172.21.15.98:0/3211835161 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f542b7fe640 1 --2- 172.21.15.98:0/3211835161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 0x563cc8851820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f542affd640 1 -- 172.21.15.98:0/3211835161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 msgr2=0x563cc8851d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f542affd640 1 --2- 172.21.15.98:0/3211835161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 0x563cc8851d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f542affd640 1 -- 172.21.15.98:0/3211835161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 msgr2=0x563cc8851820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f542affd640 1 --2- 172.21.15.98:0/3211835161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 0x563cc8851820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f542affd640 1 -- 172.21.15.98:0/3211835161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cc858c7d0 con 0x563cc8849330 2024-05-09T09:00:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f542affd640 1 --2- 172.21.15.98:0/3211835161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849330 0x563cc88512e0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f541c010010 tx=0x7f541c016d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fdebad10ffc4bf0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f540affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f540affd640 1 -- 172.21.15.98:0/3211835161 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c01b030 con 0x563cc8849330 2024-05-09T09:00:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f540affd640 1 -- 172.21.15.98:0/3211835161 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f541c00ec40 con 0x563cc8849330 2024-05-09T09:00:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f54347c6740 1 -- 172.21.15.98:0/3211835161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563cc87bc370 con 0x563cc8849330 2024-05-09T09:00:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f540affd640 1 -- 172.21.15.98:0/3211835161 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f541c02f050 con 0x563cc8849330 2024-05-09T09:00:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f54347c6740 10 client.18555 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.916+0000 7f54347c6740 1 -- 172.21.15.98:0/3211835161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563cc8863380 con 0x563cc8849330 2024-05-09T09:00:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.917+0000 7f540affd640 1 -- 172.21.15.98:0/3211835161 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563cc87bc370 con 0x563cc8849330 2024-05-09T09:00:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.917+0000 7f540affd640 1 -- 172.21.15.98:0/3211835161 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f541c015070 con 0x563cc8849330 2024-05-09T09:00:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.917+0000 7f54347c6740 10 client.18555 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.917+0000 7f54347c6740 10 client.18555 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T09:00:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.917+0000 7f54347c6740 20 client.18555 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.917+0000 7f54347c6740 1 --2- 172.21.15.98:0/3211835161 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563cc8864720 0x563cc8866b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.917+0000 7f54347c6740 4 client.18555 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.917+0000 7f54347c6740 1 -- 172.21.15.98:0/3211835161 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cc87b66c0 con 0x563cc8864720 2024-05-09T09:00:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.917+0000 7f542b7fe640 1 --2- 172.21.15.98:0/3211835161 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563cc8864720 0x563cc8866b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.917+0000 7f542b7fe640 1 --2- 172.21.15.98:0/3211835161 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563cc8864720 0x563cc8866b10 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6ce46765daf0cd6d server_cookie=cd6c6324d813fae6 in_seq=0 out_seq=0 2024-05-09T09:00:54.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.917+0000 7f540affd640 10 client.18555 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T09:00:54.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.918+0000 7f540affd640 1 -- 172.21.15.98:0/3211835161 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563cc87b66c0 con 0x563cc8864720 2024-05-09T09:00:54.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.918+0000 7f540affd640 10 client.18555 handle_command_reply: tid=0 2024-05-09T09:00:54.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.918+0000 7f54347c6740 1 client.18555 shutdown 2024-05-09T09:00:54.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.918+0000 7f54347c6740 1 -- 172.21.15.98:0/3211835161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849330 msgr2=0x563cc88512e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.918+0000 7f54347c6740 1 --2- 172.21.15.98:0/3211835161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849330 0x563cc88512e0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f541c010010 tx=0x7f541c016d30 comp rx=0 tx=0).stop 2024-05-09T09:00:54.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.918+0000 7f54347c6740 1 -- 172.21.15.98:0/3211835161 shutdown_connections 2024-05-09T09:00:54.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.918+0000 7f54347c6740 1 --2- 172.21.15.98:0/3211835161 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563cc8864720 0x563cc8866b10 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.918+0000 7f54347c6740 1 --2- 172.21.15.98:0/3211835161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 0x563cc8851d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.918+0000 7f54347c6740 1 --2- 172.21.15.98:0/3211835161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 0x563cc8851820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.918+0000 7f540affd640 0 client.18555 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T09:00:54.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.918+0000 7f54347c6740 1 --2- 172.21.15.98:0/3211835161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849330 0x563cc88512e0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.918+0000 7f54347c6740 1 -- 172.21.15.98:0/3211835161 >> 172.21.15.98:0/3211835161 conn(0x563cc8845650 msgr2=0x563cc884b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:54.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.918+0000 7f54347c6740 1 -- 172.21.15.98:0/3211835161 shutdown_connections 2024-05-09T09:00:54.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.918+0000 7f54347c6740 1 -- 172.21.15.98:0/3211835161 wait complete. 2024-05-09T09:00:54.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.919+0000 7f54347c6740 20 client.18555 trim_cache size 0 max 16384 2024-05-09T09:00:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.931+0000 7f54347c6740 1 Processor -- start 2024-05-09T09:00:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f54347c6740 1 -- start start 2024-05-09T09:00:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f54347c6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849330 0x563cc8860000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f54347c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849c50 0x563cc884bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f54347c6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 0x563cc886c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f54347c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1aee60 con 0x563cc8849c50 2024-05-09T09:00:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f54347c6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1b09a0 con 0x563cc8849330 2024-05-09T09:00:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f54347c6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1af530 con 0x563cc8854320 2024-05-09T09:00:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f542b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849c50 0x563cc884bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f542b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849c50 0x563cc884bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52156/0 (socket says 172.21.15.98:52156) 2024-05-09T09:00:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f542b7fe640 1 -- 172.21.15.98:0/324136983 learned_addr learned my addr 172.21.15.98:0/324136983 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f542b7fe640 1 -- 172.21.15.98:0/324136983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 msgr2=0x563cc886c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f542affd640 1 --2- 172.21.15.98:0/324136983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 0x563cc886c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f542b7fe640 1 --2- 172.21.15.98:0/324136983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 0x563cc886c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f542b7fe640 1 -- 172.21.15.98:0/324136983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849330 msgr2=0x563cc8860000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f542b7fe640 1 --2- 172.21.15.98:0/324136983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849330 0x563cc8860000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f542b7fe640 1 -- 172.21.15.98:0/324136983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cc8856ca0 con 0x563cc8849c50 2024-05-09T09:00:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f542affd640 1 --2- 172.21.15.98:0/324136983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 0x563cc886c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f542b7fe640 1 --2- 172.21.15.98:0/324136983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849c50 0x563cc884bba0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x563cc88670d0 tx=0x7f5414007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ca92148b500496 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f54217fa640 1 -- 172.21.15.98:0/324136983 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414016a40 con 0x563cc8849c50 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f54217fa640 1 -- 172.21.15.98:0/324136983 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f541401f030 con 0x563cc8849c50 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.932+0000 7f54217fa640 1 -- 172.21.15.98:0/324136983 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54140143b0 con 0x563cc8849c50 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.933+0000 7f54347c6740 1 -- 172.21.15.98:0/324136983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849c50 msgr2=0x563cc884bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.933+0000 7f54347c6740 1 --2- 172.21.15.98:0/324136983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849c50 0x563cc884bba0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x563cc88670d0 tx=0x7f5414007f10 comp rx=0 tx=0).stop 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.933+0000 7f54347c6740 1 -- 172.21.15.98:0/324136983 shutdown_connections 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.933+0000 7f54347c6740 1 --2- 172.21.15.98:0/324136983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 0x563cc886c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.933+0000 7f54347c6740 1 --2- 172.21.15.98:0/324136983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849c50 0x563cc884bba0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.933+0000 7f54347c6740 1 --2- 172.21.15.98:0/324136983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849330 0x563cc8860000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.933+0000 7f54347c6740 1 -- 172.21.15.98:0/324136983 >> 172.21.15.98:0/324136983 conn(0x563cc8845650 msgr2=0x563cc88447b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.933+0000 7f54347c6740 1 -- 172.21.15.98:0/324136983 shutdown_connections 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.933+0000 7f54347c6740 1 -- 172.21.15.98:0/324136983 wait complete. 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.933+0000 7f54347c6740 1 -- start start 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f54347c6740 1 Processor -- start 2024-05-09T09:00:54.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f54347c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849330 0x563cc885c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f54347c6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 0x563cc8857d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f54347c6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 0x563cc8858320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f54347c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1b02d0 con 0x563cc8849330 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f54347c6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1afc00 con 0x563cc8849c50 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f54347c6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1ace60 con 0x563cc8854320 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f54347c6740 10 client.0 _ll_register_callbacks cb 0x7f541c019070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f542affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849330 0x563cc885c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f542affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849330 0x563cc885c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52162/0 (socket says 172.21.15.98:52162) 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f542affd640 1 -- 172.21.15.98:0/2498344818 learned_addr learned my addr 172.21.15.98:0/2498344818 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f542affd640 1 --2- 172.21.15.98:0/2498344818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 0x563cc8858320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f542affd640 1 -- 172.21.15.98:0/2498344818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 msgr2=0x563cc8858320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f542affd640 1 --2- 172.21.15.98:0/2498344818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 0x563cc8858320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f542b7fe640 1 --2- 172.21.15.98:0/2498344818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 0x563cc8857d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f542affd640 1 -- 172.21.15.98:0/2498344818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 msgr2=0x563cc8857d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f542affd640 1 --2- 172.21.15.98:0/2498344818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 0x563cc8857d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f542affd640 1 -- 172.21.15.98:0/2498344818 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cc8858b20 con 0x563cc8849330 2024-05-09T09:00:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f542affd640 1 --2- 172.21.15.98:0/2498344818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849330 0x563cc885c9a0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f541c000c00 tx=0x7f541c021850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8658964a2aa9ce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f540affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f540affd640 1 -- 172.21.15.98:0/2498344818 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c038060 con 0x563cc8849330 2024-05-09T09:00:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f540affd640 1 -- 172.21.15.98:0/2498344818 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f541c015540 con 0x563cc8849330 2024-05-09T09:00:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.934+0000 7f54347c6740 1 -- 172.21.15.98:0/2498344818 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f541401f030 con 0x563cc8849330 2024-05-09T09:00:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.935+0000 7f540affd640 1 -- 172.21.15.98:0/2498344818 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f541c03f050 con 0x563cc8849330 2024-05-09T09:00:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.935+0000 7f54347c6740 10 client.18561 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.935+0000 7f54347c6740 1 -- 172.21.15.98:0/2498344818 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563cc88ca9e0 con 0x563cc8849330 2024-05-09T09:00:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.935+0000 7f540affd640 1 -- 172.21.15.98:0/2498344818 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f541401f030 con 0x563cc8849330 2024-05-09T09:00:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.935+0000 7f540affd640 1 -- 172.21.15.98:0/2498344818 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f541c015040 con 0x563cc8849330 2024-05-09T09:00:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.935+0000 7f54347c6740 10 client.18561 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.935+0000 7f54347c6740 10 client.18561 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T09:00:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.935+0000 7f54347c6740 20 client.18561 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.935+0000 7f54347c6740 1 --2- 172.21.15.98:0/2498344818 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563cc88cb040 0x563cc88cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.935+0000 7f54347c6740 4 client.18561 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.935+0000 7f54347c6740 1 -- 172.21.15.98:0/2498344818 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563cc8733820 con 0x563cc88cb040 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.935+0000 7f542b7fe640 1 --2- 172.21.15.98:0/2498344818 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563cc88cb040 0x563cc88cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.935+0000 7f542b7fe640 1 --2- 172.21.15.98:0/2498344818 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563cc88cb040 0x563cc88cd460 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=12411ab4177c2868 server_cookie=71be5752c7b673e9 in_seq=0 out_seq=0 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.935+0000 7f540affd640 10 client.18561 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f540affd640 1 -- 172.21.15.98:0/2498344818 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563cc8733820 con 0x563cc88cb040 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f540affd640 10 client.18561 handle_command_reply: tid=0 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f54347c6740 1 client.18561 shutdown 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f54347c6740 1 -- 172.21.15.98:0/2498344818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849330 msgr2=0x563cc885c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f54347c6740 1 --2- 172.21.15.98:0/2498344818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849330 0x563cc885c9a0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f541c000c00 tx=0x7f541c021850 comp rx=0 tx=0).stop 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f54347c6740 1 -- 172.21.15.98:0/2498344818 shutdown_connections 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f54347c6740 1 --2- 172.21.15.98:0/2498344818 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x563cc88cb040 0x563cc88cd460 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f54347c6740 1 --2- 172.21.15.98:0/2498344818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563cc8854320 0x563cc8858320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f540affd640 0 client.18561 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T09:00:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f54347c6740 1 --2- 172.21.15.98:0/2498344818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563cc8849c50 0x563cc8857d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f54347c6740 1 --2- 172.21.15.98:0/2498344818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563cc8849330 0x563cc885c9a0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f54347c6740 1 -- 172.21.15.98:0/2498344818 >> 172.21.15.98:0/2498344818 conn(0x563cc8845650 msgr2=0x563cc88512a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f54347c6740 1 -- 172.21.15.98:0/2498344818 shutdown_connections 2024-05-09T09:00:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f54347c6740 1 -- 172.21.15.98:0/2498344818 wait complete. 2024-05-09T09:00:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f54347c6740 20 client.18561 trim_cache size 0 max 16384 2024-05-09T09:00:54.943 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:00:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f5431aba640 1 -- 172.21.15.98:0/2970150421 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54180a2d00 msgr2=0x7f54180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f5431aba640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54180a2d00 0x7f54180a51c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f54240b9820 tx=0x7f5424090430 comp rx=0 tx=0).stop 2024-05-09T09:00:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f5431aba640 1 -- 172.21.15.98:0/2970150421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c106710 msgr2=0x7f542c19cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.936+0000 7f5431aba640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c106710 0x7f542c19cad0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f5424006010 tx=0x7f5424003c20 comp rx=0 tx=0).stop 2024-05-09T09:00:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.937+0000 7f5431aba640 1 -- 172.21.15.98:0/2970150421 shutdown_connections 2024-05-09T09:00:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.937+0000 7f5431aba640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54180a2d00 0x7f54180a51c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.937+0000 7f5431aba640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c106710 0x7f542c19cad0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.937+0000 7f5431aba640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f542c1010c0 0x7f542c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.937+0000 7f5431aba640 1 --2- 172.21.15.98:0/2970150421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f542c100700 0x7f542c198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:54.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.937+0000 7f5431aba640 1 -- 172.21.15.98:0/2970150421 >> 172.21.15.98:0/2970150421 conn(0x7f542c077830 msgr2=0x7f542c068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:54.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.937+0000 7f5431aba640 1 -- 172.21.15.98:0/2970150421 shutdown_connections 2024-05-09T09:00:54.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:54.937+0000 7f5431aba640 1 -- 172.21.15.98:0/2970150421 wait complete. 2024-05-09T09:00:54.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T09:00:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.051+0000 7ff4d4c2a640 1 -- 172.21.15.98:0/418514883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d01010c0 msgr2=0x7ff4d010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.051+0000 7ff4d4c2a640 1 --2- 172.21.15.98:0/418514883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d01010c0 0x7ff4d010d5d0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009930 tx=0x7ff4c802a420 comp rx=0 tx=0).stop 2024-05-09T09:00:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.052+0000 7ff4d4c2a640 1 -- 172.21.15.98:0/418514883 shutdown_connections 2024-05-09T09:00:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.052+0000 7ff4d4c2a640 1 --2- 172.21.15.98:0/418514883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d01010c0 0x7ff4d010d5d0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.052+0000 7ff4d4c2a640 1 --2- 172.21.15.98:0/418514883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4d0100700 0x7ff4d0100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.052+0000 7ff4d4c2a640 1 --2- 172.21.15.98:0/418514883 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4d0106710 0x7ff4d0106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.052+0000 7ff4d4c2a640 1 -- 172.21.15.98:0/418514883 >> 172.21.15.98:0/418514883 conn(0x7ff4d0077830 msgr2=0x7ff4d0077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.052+0000 7ff4d4c2a640 1 -- 172.21.15.98:0/418514883 shutdown_connections 2024-05-09T09:00:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.052+0000 7ff4d4c2a640 1 -- 172.21.15.98:0/418514883 wait complete. 2024-05-09T09:00:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.052+0000 7ff4d4c2a640 1 Processor -- start 2024-05-09T09:00:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.052+0000 7ff4d4c2a640 1 -- start start 2024-05-09T09:00:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4d4c2a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4d0100700 0x7ff4d019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4d4c2a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d01010c0 0x7ff4d019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4d4c2a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4d0106710 0x7ff4d019ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4d4c2a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d0110ae0 con 0x7ff4d01010c0 2024-05-09T09:00:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4d4c2a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d0110960 con 0x7ff4d0106710 2024-05-09T09:00:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4d4c2a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d0110c60 con 0x7ff4d0100700 2024-05-09T09:00:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4cdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d01010c0 0x7ff4d019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4cdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d01010c0 0x7ff4d019a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52184/0 (socket says 172.21.15.98:52184) 2024-05-09T09:00:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4cdd9b640 1 -- 172.21.15.98:0/932345337 learned_addr learned my addr 172.21.15.98:0/932345337 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4ced9d640 1 --2- 172.21.15.98:0/932345337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4d0106710 0x7ff4d019ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4ce59c640 1 --2- 172.21.15.98:0/932345337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4d0100700 0x7ff4d019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4cdd9b640 1 -- 172.21.15.98:0/932345337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4d0100700 msgr2=0x7ff4d019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4cdd9b640 1 --2- 172.21.15.98:0/932345337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4d0100700 0x7ff4d019a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4cdd9b640 1 -- 172.21.15.98:0/932345337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4d0106710 msgr2=0x7ff4d019ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4cdd9b640 1 --2- 172.21.15.98:0/932345337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4d0106710 0x7ff4d019ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4cdd9b640 1 -- 172.21.15.98:0/932345337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d019f170 con 0x7ff4d01010c0 2024-05-09T09:00:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4cdd9b640 1 --2- 172.21.15.98:0/932345337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d01010c0 0x7ff4d019a6c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ff4c400c920 tx=0x7ff4c400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4bf7fe640 1 -- 172.21.15.98:0/932345337 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4007c70 con 0x7ff4d01010c0 2024-05-09T09:00:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4bf7fe640 1 -- 172.21.15.98:0/932345337 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c4007e10 con 0x7ff4d01010c0 2024-05-09T09:00:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4d4c2a640 1 -- 172.21.15.98:0/932345337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d019f460 con 0x7ff4d01010c0 2024-05-09T09:00:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4bf7fe640 1 -- 172.21.15.98:0/932345337 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4018890 con 0x7ff4d01010c0 2024-05-09T09:00:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.053+0000 7ff4d4c2a640 1 -- 172.21.15.98:0/932345337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d01a6cf0 con 0x7ff4d01010c0 2024-05-09T09:00:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.056+0000 7ff4bf7fe640 1 -- 172.21.15.98:0/932345337 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff4c4018a70 con 0x7ff4d01010c0 2024-05-09T09:00:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.056+0000 7ff4d7a12740 1 Processor -- start 2024-05-09T09:00:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.056+0000 7ff4d7a12740 1 -- start start 2024-05-09T09:00:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.056+0000 7ff4d7a12740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718330 0x560a40718710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.056+0000 7ff4d7a12740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40718c50 0x560a40722de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.056+0000 7ff4d7a12740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40723320 0x560a40725710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.056+0000 7ff4d7a12740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560a4067dc40 con 0x560a40723320 2024-05-09T09:00:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.056+0000 7ff4d7a12740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560a40625580 con 0x560a40718330 2024-05-09T09:00:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.056+0000 7ff4d7a12740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x560a40679c00 con 0x560a40718c50 2024-05-09T09:00:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4bf7fe640 1 --2- 172.21.15.98:0/932345337 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff4b80a2d00 0x7ff4b80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4cdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40723320 0x560a40725710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4cdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40723320 0x560a40725710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52190/0 (socket says 172.21.15.98:52190) 2024-05-09T09:00:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4cdd9b640 1 -- 172.21.15.98:0/1688501023 learned_addr learned my addr 172.21.15.98:0/1688501023 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4ced9d640 1 --2- 172.21.15.98:0/1688501023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718330 0x560a40718710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4ce59c640 1 --2- 172.21.15.98:0/1688501023 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40718c50 0x560a40722de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4ced9d640 1 --2- 172.21.15.98:0/932345337 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff4b80a2d00 0x7ff4b80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4bf7fe640 1 -- 172.21.15.98:0/932345337 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7ff4c40cf9b0 con 0x7ff4d01010c0 2024-05-09T09:00:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4cdd9b640 1 -- 172.21.15.98:0/1688501023 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40718c50 msgr2=0x560a40722de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4cdd9b640 1 --2- 172.21.15.98:0/1688501023 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40718c50 0x560a40722de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4cdd9b640 1 -- 172.21.15.98:0/1688501023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718330 msgr2=0x560a40718710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4cdd9b640 1 --2- 172.21.15.98:0/1688501023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718330 0x560a40718710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4cdd9b640 1 -- 172.21.15.98:0/1688501023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a403913f0 con 0x560a40723320 2024-05-09T09:00:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4ce59c640 1 --2- 172.21.15.98:0/1688501023 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40718c50 0x560a40722de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4cdd9b640 1 --2- 172.21.15.98:0/1688501023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40723320 0x560a40725710 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40ccda0 tx=0x7ff4c4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921cb598ed3c2aa5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4ced9d640 1 --2- 172.21.15.98:0/932345337 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff4b80a2d00 0x7ff4b80a51c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8031550 tx=0x7ff4c8009260 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4bcff9640 1 -- 172.21.15.98:0/1688501023 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c40cfe60 con 0x560a40723320 2024-05-09T09:00:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4bcff9640 1 -- 172.21.15.98:0/1688501023 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c40a4730 con 0x560a40723320 2024-05-09T09:00:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.057+0000 7ff4bcff9640 1 -- 172.21.15.98:0/1688501023 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4098130 con 0x560a40723320 2024-05-09T09:00:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.058+0000 7ff4d7a12740 1 -- 172.21.15.98:0/1688501023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40723320 msgr2=0x560a40725710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.058+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/1688501023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40723320 0x560a40725710 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40ccda0 tx=0x7ff4c4000bc0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.058+0000 7ff4d7a12740 1 -- 172.21.15.98:0/1688501023 shutdown_connections 2024-05-09T09:00:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.058+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/1688501023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40723320 0x560a40725710 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.058+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/1688501023 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40718c50 0x560a40722de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.058+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/1688501023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718330 0x560a40718710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.058+0000 7ff4d7a12740 1 -- 172.21.15.98:0/1688501023 >> 172.21.15.98:0/1688501023 conn(0x560a40714650 msgr2=0x560a40714a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.058+0000 7ff4d7a12740 1 -- 172.21.15.98:0/1688501023 shutdown_connections 2024-05-09T09:00:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.058+0000 7ff4d7a12740 1 -- 172.21.15.98:0/1688501023 wait complete. 2024-05-09T09:00:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.058+0000 7ff4d7a12740 1 -- start start 2024-05-09T09:00:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4d7a12740 1 Processor -- start 2024-05-09T09:00:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4d7a12740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718330 0x560a407202b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4d7a12740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40718c50 0x560a407207f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4d7a12740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40723320 0x560a40720d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4d7a12740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560a407260b0 con 0x560a40718330 2024-05-09T09:00:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4d7a12740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560a40638c40 con 0x560a40723320 2024-05-09T09:00:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4d7a12740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x560a40726230 con 0x560a40718c50 2024-05-09T09:00:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4d7a12740 10 client.0 _ll_register_callbacks cb 0x560a40390be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4ce59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718330 0x560a407202b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4ce59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718330 0x560a407202b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52202/0 (socket says 172.21.15.98:52202) 2024-05-09T09:00:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4ce59c640 1 -- 172.21.15.98:0/4245416858 learned_addr learned my addr 172.21.15.98:0/4245416858 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4cdd9b640 1 --2- 172.21.15.98:0/4245416858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40718c50 0x560a407207f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4ce59c640 1 -- 172.21.15.98:0/4245416858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40718c50 msgr2=0x560a407207f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4ce59c640 1 --2- 172.21.15.98:0/4245416858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40718c50 0x560a407207f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4ce59c640 1 -- 172.21.15.98:0/4245416858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40723320 msgr2=0x560a40720d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4ce59c640 1 --2- 172.21.15.98:0/4245416858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40723320 0x560a40720d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4ce59c640 1 -- 172.21.15.98:0/4245416858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a4045b7d0 con 0x560a40718330 2024-05-09T09:00:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4ce59c640 1 --2- 172.21.15.98:0/4245416858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718330 0x560a407202b0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000fa90 tx=0x7ff4c000ff50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c2b084129aa45fb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4b5ffb640 1 -- 172.21.15.98:0/4245416858 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c000de50 con 0x560a40718330 2024-05-09T09:00:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4b5ffb640 1 -- 172.21.15.98:0/4245416858 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7ff4c0004b20 con 0x560a40718330 2024-05-09T09:00:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.059+0000 7ff4d7a12740 1 -- 172.21.15.98:0/4245416858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4c40a4730 con 0x560a40718330 2024-05-09T09:00:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.060+0000 7ff4b5ffb640 1 -- 172.21.15.98:0/4245416858 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff4c0004d20 con 0x560a40718330 2024-05-09T09:00:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.060+0000 7ff4d7a12740 10 client.18573 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.060+0000 7ff4d7a12740 1 -- 172.21.15.98:0/4245416858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560a40732340 con 0x560a40718330 2024-05-09T09:00:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.060+0000 7ff4b5ffb640 1 -- 172.21.15.98:0/4245416858 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c40a4730 con 0x560a40718330 2024-05-09T09:00:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.060+0000 7ff4b5ffb640 1 -- 172.21.15.98:0/4245416858 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7ff4c001a690 con 0x560a40718330 2024-05-09T09:00:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.060+0000 7ff4d7a12740 10 client.18573 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.060+0000 7ff4d7a12740 10 client.18573 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T09:00:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.060+0000 7ff4d7a12740 20 client.18573 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.060+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/4245416858 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560a40733850 0x560a40735ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.060+0000 7ff4d7a12740 4 client.18573 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.060+0000 7ff4d7a12740 1 -- 172.21.15.98:0/4245416858 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a4068b370 con 0x560a40733850 2024-05-09T09:00:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.060+0000 7ff4cdd9b640 1 --2- 172.21.15.98:0/4245416858 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560a40733850 0x560a40735ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.061+0000 7ff4b5ffb640 10 client.18573 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T09:00:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.061+0000 7ff4cdd9b640 1 --2- 172.21.15.98:0/4245416858 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560a40733850 0x560a40735ca0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=db3399e4c4ae1c88 server_cookie=59185624351ef60a in_seq=0 out_seq=0 2024-05-09T09:00:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.061+0000 7ff4b5ffb640 1 -- 172.21.15.98:0/4245416858 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560a4068b370 con 0x560a40733850 2024-05-09T09:00:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.061+0000 7ff4b5ffb640 10 client.18573 handle_command_reply: tid=0 2024-05-09T09:00:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.061+0000 7ff4d7a12740 1 client.18573 shutdown 2024-05-09T09:00:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.061+0000 7ff4d7a12740 1 -- 172.21.15.98:0/4245416858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718330 msgr2=0x560a407202b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.061+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/4245416858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718330 0x560a407202b0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000fa90 tx=0x7ff4c000ff50 comp rx=0 tx=0).stop 2024-05-09T09:00:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.061+0000 7ff4d7a12740 1 -- 172.21.15.98:0/4245416858 shutdown_connections 2024-05-09T09:00:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.061+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/4245416858 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560a40733850 0x560a40735ca0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.061+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/4245416858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40723320 0x560a40720d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.061+0000 7ff4b5ffb640 0 client.18573 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T09:00:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.061+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/4245416858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40718c50 0x560a407207f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.061+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/4245416858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718330 0x560a407202b0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.061+0000 7ff4d7a12740 1 -- 172.21.15.98:0/4245416858 >> 172.21.15.98:0/4245416858 conn(0x560a40714650 msgr2=0x560a4071a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.062+0000 7ff4d7a12740 1 -- 172.21.15.98:0/4245416858 shutdown_connections 2024-05-09T09:00:55.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.062+0000 7ff4d7a12740 1 -- 172.21.15.98:0/4245416858 wait complete. 2024-05-09T09:00:55.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.062+0000 7ff4d7a12740 20 client.18573 trim_cache size 0 max 16384 2024-05-09T09:00:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4d7a12740 1 Processor -- start 2024-05-09T09:00:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4d7a12740 1 -- start start 2024-05-09T09:00:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4d7a12740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718330 0x560a4072efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4d7a12740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718c50 0x560a4071aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4d7a12740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40723320 0x560a4073b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4d7a12740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01b1000 con 0x560a40718330 2024-05-09T09:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4d7a12740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01b2b40 con 0x560a40718c50 2024-05-09T09:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4d7a12740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01b16d0 con 0x560a40723320 2024-05-09T09:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4ce59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718330 0x560a4072efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4ce59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718330 0x560a4072efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52206/0 (socket says 172.21.15.98:52206) 2024-05-09T09:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4ce59c640 1 -- 172.21.15.98:0/3857896503 learned_addr learned my addr 172.21.15.98:0/3857896503 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4ced9d640 1 --2- 172.21.15.98:0/3857896503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40723320 0x560a4073b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4ce59c640 1 -- 172.21.15.98:0/3857896503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40723320 msgr2=0x560a4073b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4ce59c640 1 --2- 172.21.15.98:0/3857896503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40723320 0x560a4073b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4cdd9b640 1 --2- 172.21.15.98:0/3857896503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718c50 0x560a4071aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4ce59c640 1 -- 172.21.15.98:0/3857896503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718c50 msgr2=0x560a4071aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4ce59c640 1 --2- 172.21.15.98:0/3857896503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718c50 0x560a4071aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4ce59c640 1 -- 172.21.15.98:0/3857896503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a40705d20 con 0x560a40718330 2024-05-09T09:00:55.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4cdd9b640 1 --2- 172.21.15.98:0/3857896503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718c50 0x560a4071aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4ce59c640 1 --2- 172.21.15.98:0/3857896503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718330 0x560a4072efa0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7ff4c001ab30 tx=0x7ff4c000d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5825d435568818fb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4b7fff640 1 -- 172.21.15.98:0/3857896503 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c002c070 con 0x560a40718330 2024-05-09T09:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.075+0000 7ff4b7fff640 1 -- 172.21.15.98:0/3857896503 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c000dc50 con 0x560a40718330 2024-05-09T09:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.076+0000 7ff4b7fff640 1 -- 172.21.15.98:0/3857896503 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0028a30 con 0x560a40718330 2024-05-09T09:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.076+0000 7ff4d7a12740 1 -- 172.21.15.98:0/3857896503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718330 msgr2=0x560a4072efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.076+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/3857896503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718330 0x560a4072efa0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7ff4c001ab30 tx=0x7ff4c000d710 comp rx=0 tx=0).stop 2024-05-09T09:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.076+0000 7ff4d7a12740 1 -- 172.21.15.98:0/3857896503 shutdown_connections 2024-05-09T09:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.076+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/3857896503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40723320 0x560a4073b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.076+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/3857896503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718c50 0x560a4071aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.076+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/3857896503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718330 0x560a4072efa0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.076+0000 7ff4d7a12740 1 -- 172.21.15.98:0/3857896503 >> 172.21.15.98:0/3857896503 conn(0x560a40714650 msgr2=0x560a407137b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.076+0000 7ff4d7a12740 1 -- 172.21.15.98:0/3857896503 shutdown_connections 2024-05-09T09:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.076+0000 7ff4d7a12740 1 -- 172.21.15.98:0/3857896503 wait complete. 2024-05-09T09:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.076+0000 7ff4d7a12740 1 -- start start 2024-05-09T09:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.076+0000 7ff4d7a12740 1 Processor -- start 2024-05-09T09:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4d7a12740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718330 0x560a40727170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4d7a12740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718c50 0x560a407276b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4d7a12740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40723320 0x560a40727bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4d7a12740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01b2470 con 0x560a40718c50 2024-05-09T09:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4d7a12740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01b1da0 con 0x560a40718330 2024-05-09T09:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4d7a12740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01af000 con 0x560a40723320 2024-05-09T09:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4d7a12740 10 client.0 _ll_register_callbacks cb 0x7ff4c0008580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4cdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718c50 0x560a407276b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4cdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718c50 0x560a407276b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52218/0 (socket says 172.21.15.98:52218) 2024-05-09T09:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4cdd9b640 1 -- 172.21.15.98:0/213812912 learned_addr learned my addr 172.21.15.98:0/213812912 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4ce59c640 1 --2- 172.21.15.98:0/213812912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718330 0x560a40727170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4ced9d640 1 --2- 172.21.15.98:0/213812912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40723320 0x560a40727bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4cdd9b640 1 -- 172.21.15.98:0/213812912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40723320 msgr2=0x560a40727bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4cdd9b640 1 --2- 172.21.15.98:0/213812912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40723320 0x560a40727bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4cdd9b640 1 -- 172.21.15.98:0/213812912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718330 msgr2=0x560a40727170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4cdd9b640 1 --2- 172.21.15.98:0/213812912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718330 0x560a40727170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4cdd9b640 1 -- 172.21.15.98:0/213812912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a407990c0 con 0x560a40718c50 2024-05-09T09:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4ce59c640 1 --2- 172.21.15.98:0/213812912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718330 0x560a40727170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4cdd9b640 1 --2- 172.21.15.98:0/213812912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718c50 0x560a407276b0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7ff4c400c8f0 tx=0x7ff4c4097dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14dc899062250c52 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4b5ffb640 1 -- 172.21.15.98:0/213812912 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c40cfe60 con 0x560a40718c50 2024-05-09T09:00:55.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4b5ffb640 1 -- 172.21.15.98:0/213812912 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7ff4c4018d60 con 0x560a40718c50 2024-05-09T09:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4d7a12740 1 -- 172.21.15.98:0/213812912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4c000dc50 con 0x560a40718c50 2024-05-09T09:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4b5ffb640 1 -- 172.21.15.98:0/213812912 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff4c40a4730 con 0x560a40718c50 2024-05-09T09:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4d7a12740 10 client.18579 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.077+0000 7ff4d7a12740 1 -- 172.21.15.98:0/213812912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560a4079a330 con 0x560a40718c50 2024-05-09T09:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.078+0000 7ff4b5ffb640 1 -- 172.21.15.98:0/213812912 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c000dc50 con 0x560a40718c50 2024-05-09T09:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.078+0000 7ff4b5ffb640 1 -- 172.21.15.98:0/213812912 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7ff4c40a42a0 con 0x560a40718c50 2024-05-09T09:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.078+0000 7ff4d7a12740 10 client.18579 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.078+0000 7ff4d7a12740 10 client.18579 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T09:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.078+0000 7ff4d7a12740 20 client.18579 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.078+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/213812912 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560a40733850 0x560a4079c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.078+0000 7ff4d7a12740 4 client.18579 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.078+0000 7ff4d7a12740 1 -- 172.21.15.98:0/213812912 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560a406856c0 con 0x560a40733850 2024-05-09T09:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.078+0000 7ff4ce59c640 1 --2- 172.21.15.98:0/213812912 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560a40733850 0x560a4079c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.078+0000 7ff4b5ffb640 10 client.18579 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T09:00:55.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.078+0000 7ff4ce59c640 1 --2- 172.21.15.98:0/213812912 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560a40733850 0x560a4079c620 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=197f4646ffc10154 server_cookie=a85a7e5578d0ff18 in_seq=0 out_seq=0 2024-05-09T09:00:55.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4b5ffb640 1 -- 172.21.15.98:0/213812912 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560a406856c0 con 0x560a40733850 2024-05-09T09:00:55.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4b5ffb640 10 client.18579 handle_command_reply: tid=0 2024-05-09T09:00:55.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d7a12740 1 client.18579 shutdown 2024-05-09T09:00:55.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d7a12740 1 -- 172.21.15.98:0/213812912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718c50 msgr2=0x560a407276b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/213812912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718c50 0x560a407276b0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7ff4c400c8f0 tx=0x7ff4c4097dc0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d7a12740 1 -- 172.21.15.98:0/213812912 shutdown_connections 2024-05-09T09:00:55.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/213812912 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x560a40733850 0x560a4079c620 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/213812912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560a40723320 0x560a40727bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4b5ffb640 0 client.18579 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T09:00:55.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/213812912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560a40718c50 0x560a407276b0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d7a12740 1 --2- 172.21.15.98:0/213812912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560a40718330 0x560a40727170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d7a12740 1 -- 172.21.15.98:0/213812912 >> 172.21.15.98:0/213812912 conn(0x560a40714650 msgr2=0x560a407137b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d7a12740 1 -- 172.21.15.98:0/213812912 shutdown_connections 2024-05-09T09:00:55.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d7a12740 1 -- 172.21.15.98:0/213812912 wait complete. 2024-05-09T09:00:55.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d7a12740 20 client.18579 trim_cache size 0 max 16384 2024-05-09T09:00:55.087 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:00:55.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d4c2a640 1 -- 172.21.15.98:0/932345337 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff4b80a2d00 msgr2=0x7ff4b80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d4c2a640 1 --2- 172.21.15.98:0/932345337 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff4b80a2d00 0x7ff4b80a51c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8031550 tx=0x7ff4c8009260 comp rx=0 tx=0).stop 2024-05-09T09:00:55.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d4c2a640 1 -- 172.21.15.98:0/932345337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d01010c0 msgr2=0x7ff4d019a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d4c2a640 1 --2- 172.21.15.98:0/932345337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d01010c0 0x7ff4d019a6c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ff4c400c920 tx=0x7ff4c400cde0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d4c2a640 1 -- 172.21.15.98:0/932345337 shutdown_connections 2024-05-09T09:00:55.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d4c2a640 1 --2- 172.21.15.98:0/932345337 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff4b80a2d00 0x7ff4b80a51c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d4c2a640 1 --2- 172.21.15.98:0/932345337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4d0106710 0x7ff4d019ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d4c2a640 1 --2- 172.21.15.98:0/932345337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d01010c0 0x7ff4d019a6c0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d4c2a640 1 --2- 172.21.15.98:0/932345337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4d0100700 0x7ff4d019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d4c2a640 1 -- 172.21.15.98:0/932345337 >> 172.21.15.98:0/932345337 conn(0x7ff4d0077830 msgr2=0x7ff4d0104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d4c2a640 1 -- 172.21.15.98:0/932345337 shutdown_connections 2024-05-09T09:00:55.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.079+0000 7ff4d4c2a640 1 -- 172.21.15.98:0/932345337 wait complete. 2024-05-09T09:00:55.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T09:00:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.193+0000 7f0112163640 1 -- 172.21.15.98:0/2619282736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f010c102ab0 msgr2=0x7f010c102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.193+0000 7f0112163640 1 --2- 172.21.15.98:0/2619282736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f010c102ab0 0x7f010c102e90 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f00fc009900 tx=0x7f00fc02a510 comp rx=0 tx=0).stop 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.193+0000 7f0112163640 1 -- 172.21.15.98:0/2619282736 shutdown_connections 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.193+0000 7f0112163640 1 --2- 172.21.15.98:0/2619282736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f010c10bdf0 0x7f010c10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.193+0000 7f0112163640 1 --2- 172.21.15.98:0/2619282736 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f010c1033d0 0x7f010c10b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.193+0000 7f0112163640 1 --2- 172.21.15.98:0/2619282736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f010c102ab0 0x7f010c102e90 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.193+0000 7f0112163640 1 -- 172.21.15.98:0/2619282736 >> 172.21.15.98:0/2619282736 conn(0x7f010c077830 msgr2=0x7f010c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.193+0000 7f0112163640 1 -- 172.21.15.98:0/2619282736 shutdown_connections 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.193+0000 7f0112163640 1 -- 172.21.15.98:0/2619282736 wait complete. 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.193+0000 7f0112163640 1 Processor -- start 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.193+0000 7f0112163640 1 -- start start 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.193+0000 7f0112163640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f010c102ab0 0x7f010c198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f0112163640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f010c1033d0 0x7f010c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f0112163640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f010c10bdf0 0x7f010c19c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f0112163640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f010c111bf0 con 0x7f010c10bdf0 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f0112163640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f010c111a70 con 0x7f010c102ab0 2024-05-09T09:00:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f0112163640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f010c111d70 con 0x7f010c1033d0 2024-05-09T09:00:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f010bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f010c10bdf0 0x7f010c19c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f010bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f010c10bdf0 0x7f010c19c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52236/0 (socket says 172.21.15.98:52236) 2024-05-09T09:00:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f010bfff640 1 -- 172.21.15.98:0/2933314290 learned_addr learned my addr 172.21.15.98:0/2933314290 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f010b7fe640 1 --2- 172.21.15.98:0/2933314290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f010c102ab0 0x7f010c198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f010bfff640 1 -- 172.21.15.98:0/2933314290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f010c1033d0 msgr2=0x7f010c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f010bfff640 1 --2- 172.21.15.98:0/2933314290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f010c1033d0 0x7f010c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f010bfff640 1 -- 172.21.15.98:0/2933314290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f010c102ab0 msgr2=0x7f010c198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f010bfff640 1 --2- 172.21.15.98:0/2933314290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f010c102ab0 0x7f010c198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f010bfff640 1 -- 172.21.15.98:0/2933314290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010c19d100 con 0x7f010c10bdf0 2024-05-09T09:00:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f010bfff640 1 --2- 172.21.15.98:0/2933314290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f010c10bdf0 0x7f010c19c9d0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f010400eaa0 tx=0x7f010400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f01097fa640 1 -- 172.21.15.98:0/2933314290 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010400cdb0 con 0x7f010c10bdf0 2024-05-09T09:00:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f0112163640 1 -- 172.21.15.98:0/2933314290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f010c19d3f0 con 0x7f010c10bdf0 2024-05-09T09:00:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f01097fa640 1 -- 172.21.15.98:0/2933314290 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0104005020 con 0x7f010c10bdf0 2024-05-09T09:00:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f01097fa640 1 -- 172.21.15.98:0/2933314290 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01040116b0 con 0x7f010c10bdf0 2024-05-09T09:00:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.194+0000 7f0112163640 1 -- 172.21.15.98:0/2933314290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f010c1a4c80 con 0x7f010c10bdf0 2024-05-09T09:00:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.196+0000 7f01097fa640 1 -- 172.21.15.98:0/2933314290 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0104005560 con 0x7f010c10bdf0 2024-05-09T09:00:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f0114e75740 1 Processor -- start 2024-05-09T09:00:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f0114e75740 1 -- start start 2024-05-09T09:00:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f0114e75740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da94330 0x55fd5da94710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f01097fa640 1 --2- 172.21.15.98:0/2933314290 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00f80a2d00 0x7f00f80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f0114e75740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94c50 0x55fd5da9ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f0114e75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da9f320 0x55fd5daa1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f0114e75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fd5d9f9c40 con 0x55fd5da9f320 2024-05-09T09:00:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f0114e75740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fd5d9a1580 con 0x55fd5da94c50 2024-05-09T09:00:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f0114e75740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fd5d9f5c00 con 0x55fd5da94330 2024-05-09T09:00:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f010bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da9f320 0x55fd5daa1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f0103fff640 1 --2- 172.21.15.98:0/2933314290 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00f80a2d00 0x7f00f80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f010bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da9f320 0x55fd5daa1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52240/0 (socket says 172.21.15.98:52240) 2024-05-09T09:00:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f010bfff640 1 -- 172.21.15.98:0/1384267931 learned_addr learned my addr 172.21.15.98:0/1384267931 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f01097fa640 1 -- 172.21.15.98:0/2933314290 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f01040cfa70 con 0x7f010c10bdf0 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f010b7fe640 1 --2- 172.21.15.98:0/1384267931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da94330 0x55fd5da94710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f0103fff640 1 --2- 172.21.15.98:0/1384267931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94c50 0x55fd5da9ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f010bfff640 1 -- 172.21.15.98:0/1384267931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da94330 msgr2=0x55fd5da94710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.197+0000 7f010bfff640 1 --2- 172.21.15.98:0/1384267931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da94330 0x55fd5da94710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f010bfff640 1 -- 172.21.15.98:0/1384267931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94c50 msgr2=0x55fd5da9ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f010bfff640 1 --2- 172.21.15.98:0/1384267931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94c50 0x55fd5da9ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f010bfff640 1 -- 172.21.15.98:0/1384267931 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd5d70d3f0 con 0x55fd5da9f320 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f0103fff640 1 --2- 172.21.15.98:0/2933314290 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00f80a2d00 0x7f00f80a51c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f010c10db20 tx=0x7f00f40137d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f010bfff640 1 --2- 172.21.15.98:0/1384267931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da9f320 0x55fd5daa1710 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f01040092f0 tx=0x7f01040058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e2e46216758d20 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f0101ffb640 1 -- 172.21.15.98:0/1384267931 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010401f070 con 0x55fd5da9f320 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f0101ffb640 1 -- 172.21.15.98:0/1384267931 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f010409c280 con 0x55fd5da9f320 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f0101ffb640 1 -- 172.21.15.98:0/1384267931 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01040a47f0 con 0x55fd5da9f320 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f0114e75740 1 -- 172.21.15.98:0/1384267931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da9f320 msgr2=0x55fd5daa1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f0114e75740 1 --2- 172.21.15.98:0/1384267931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da9f320 0x55fd5daa1710 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f01040092f0 tx=0x7f01040058d0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f0114e75740 1 -- 172.21.15.98:0/1384267931 shutdown_connections 2024-05-09T09:00:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f0114e75740 1 --2- 172.21.15.98:0/1384267931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da9f320 0x55fd5daa1710 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f0114e75740 1 --2- 172.21.15.98:0/1384267931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94c50 0x55fd5da9ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f0114e75740 1 --2- 172.21.15.98:0/1384267931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da94330 0x55fd5da94710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.198+0000 7f0114e75740 1 -- 172.21.15.98:0/1384267931 >> 172.21.15.98:0/1384267931 conn(0x55fd5da90650 msgr2=0x55fd5da90a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f0114e75740 1 -- 172.21.15.98:0/1384267931 shutdown_connections 2024-05-09T09:00:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f0114e75740 1 -- 172.21.15.98:0/1384267931 wait complete. 2024-05-09T09:00:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f0114e75740 1 -- start start 2024-05-09T09:00:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f0114e75740 1 Processor -- start 2024-05-09T09:00:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f0114e75740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94330 0x55fd5da9c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f0114e75740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da94c50 0x55fd5da9c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f0114e75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da9f320 0x55fd5da9cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f0114e75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fd5daa20b0 con 0x55fd5da9f320 2024-05-09T09:00:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f0114e75740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fd5d9b4c40 con 0x55fd5da94330 2024-05-09T09:00:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f0114e75740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fd5daa2230 con 0x55fd5da94c50 2024-05-09T09:00:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f0114e75740 10 client.0 _ll_register_callbacks cb 0x55fd5d70cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f010bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da9f320 0x55fd5da9cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f010bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da9f320 0x55fd5da9cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52252/0 (socket says 172.21.15.98:52252) 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f010bfff640 1 -- 172.21.15.98:0/3165122670 learned_addr learned my addr 172.21.15.98:0/3165122670 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.199+0000 7f010b7fe640 1 --2- 172.21.15.98:0/3165122670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94330 0x55fd5da9c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f010bfff640 1 -- 172.21.15.98:0/3165122670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da94c50 msgr2=0x55fd5da9c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f010bfff640 1 --2- 172.21.15.98:0/3165122670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da94c50 0x55fd5da9c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f010bfff640 1 -- 172.21.15.98:0/3165122670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94330 msgr2=0x55fd5da9c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f010bfff640 1 --2- 172.21.15.98:0/3165122670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94330 0x55fd5da9c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f010bfff640 1 -- 172.21.15.98:0/3165122670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd5d7d77d0 con 0x55fd5da9f320 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f010bfff640 1 --2- 172.21.15.98:0/3165122670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da9f320 0x55fd5da9cd90 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f01040cce60 tx=0x7f0104005400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f679bee1914ab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f00eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f00eaffd640 1 -- 172.21.15.98:0/3165122670 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010401f040 con 0x55fd5da9f320 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f00eaffd640 1 -- 172.21.15.98:0/3165122670 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f01040ae450 con 0x55fd5da9f320 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f0114e75740 1 -- 172.21.15.98:0/3165122670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f010409c280 con 0x55fd5da9f320 2024-05-09T09:00:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f00eaffd640 1 -- 172.21.15.98:0/3165122670 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f01040a47f0 con 0x55fd5da9f320 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f0114e75740 10 client.18591 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f0114e75740 1 -- 172.21.15.98:0/3165122670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55fd5daae330 con 0x55fd5da9f320 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f00eaffd640 1 -- 172.21.15.98:0/3165122670 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f010409c280 con 0x55fd5da9f320 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f00eaffd640 1 -- 172.21.15.98:0/3165122670 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f010400f040 con 0x55fd5da9f320 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f0114e75740 10 client.18591 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f0114e75740 10 client.18591 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f0114e75740 20 client.18591 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f0114e75740 1 --2- 172.21.15.98:0/3165122670 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55fd5daaf4b0 0x55fd5dab18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f0114e75740 4 client.18591 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.200+0000 7f0114e75740 1 -- 172.21.15.98:0/3165122670 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd5da07370 con 0x55fd5daaf4b0 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.201+0000 7f010b7fe640 1 --2- 172.21.15.98:0/3165122670 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55fd5daaf4b0 0x55fd5dab18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.201+0000 7f010b7fe640 1 --2- 172.21.15.98:0/3165122670 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55fd5daaf4b0 0x55fd5dab18a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f895c8f48125054c server_cookie=cde038e13feef7b4 in_seq=0 out_seq=0 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.201+0000 7f00eaffd640 10 client.18591 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f00eaffd640 1 -- 172.21.15.98:0/3165122670 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd5da07370 con 0x55fd5daaf4b0 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f00eaffd640 10 client.18591 handle_command_reply: tid=0 2024-05-09T09:00:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f0114e75740 1 client.18591 shutdown 2024-05-09T09:00:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f0114e75740 1 -- 172.21.15.98:0/3165122670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da9f320 msgr2=0x55fd5da9cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f0114e75740 1 --2- 172.21.15.98:0/3165122670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da9f320 0x55fd5da9cd90 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f01040cce60 tx=0x7f0104005400 comp rx=0 tx=0).stop 2024-05-09T09:00:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f0114e75740 1 -- 172.21.15.98:0/3165122670 shutdown_connections 2024-05-09T09:00:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f0114e75740 1 --2- 172.21.15.98:0/3165122670 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55fd5daaf4b0 0x55fd5dab18a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f0114e75740 1 --2- 172.21.15.98:0/3165122670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da9f320 0x55fd5da9cd90 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f00eaffd640 0 client.18591 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T09:00:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f0114e75740 1 --2- 172.21.15.98:0/3165122670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da94c50 0x55fd5da9c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f0114e75740 1 --2- 172.21.15.98:0/3165122670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94330 0x55fd5da9c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f0114e75740 1 -- 172.21.15.98:0/3165122670 >> 172.21.15.98:0/3165122670 conn(0x55fd5da90650 msgr2=0x55fd5da96450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f0114e75740 1 -- 172.21.15.98:0/3165122670 shutdown_connections 2024-05-09T09:00:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f0114e75740 1 -- 172.21.15.98:0/3165122670 wait complete. 2024-05-09T09:00:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.202+0000 7f0114e75740 20 client.18591 trim_cache size 0 max 16384 2024-05-09T09:00:55.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f0114e75740 1 Processor -- start 2024-05-09T09:00:55.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f0114e75740 1 -- start start 2024-05-09T09:00:55.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f0114e75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da94330 0x55fd5da96b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f0114e75740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da94c50 0x55fd5da9cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f0114e75740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da9f320 0x55fd5dab6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f0114e75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f010c1aef10 con 0x55fd5da94330 2024-05-09T09:00:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f0114e75740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f010c1b0a50 con 0x55fd5da9f320 2024-05-09T09:00:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f0114e75740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f010c1af5e0 con 0x55fd5da94c50 2024-05-09T09:00:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f010b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da94330 0x55fd5da96b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f010b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da94330 0x55fd5da96b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52254/0 (socket says 172.21.15.98:52254) 2024-05-09T09:00:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f010b7fe640 1 -- 172.21.15.98:0/1112555443 learned_addr learned my addr 172.21.15.98:0/1112555443 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f0103fff640 1 --2- 172.21.15.98:0/1112555443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da94c50 0x55fd5da9cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f010bfff640 1 --2- 172.21.15.98:0/1112555443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da9f320 0x55fd5dab6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f010b7fe640 1 -- 172.21.15.98:0/1112555443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da94c50 msgr2=0x55fd5da9cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f010b7fe640 1 --2- 172.21.15.98:0/1112555443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da94c50 0x55fd5da9cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f010b7fe640 1 -- 172.21.15.98:0/1112555443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da9f320 msgr2=0x55fd5dab6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f010b7fe640 1 --2- 172.21.15.98:0/1112555443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da9f320 0x55fd5dab6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f010b7fe640 1 -- 172.21.15.98:0/1112555443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd5da94a80 con 0x55fd5da94330 2024-05-09T09:00:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f010bfff640 1 --2- 172.21.15.98:0/1112555443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da9f320 0x55fd5dab6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.216+0000 7f010b7fe640 1 --2- 172.21.15.98:0/1112555443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da94330 0x55fd5da96b70 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f00fc009230 tx=0x7f00fc0058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330cd4e852a93c5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.217+0000 7f01017fa640 1 -- 172.21.15.98:0/1112555443 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc04a070 con 0x55fd5da94330 2024-05-09T09:00:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.217+0000 7f01017fa640 1 -- 172.21.15.98:0/1112555443 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00fc046dc0 con 0x55fd5da94330 2024-05-09T09:00:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.217+0000 7f01017fa640 1 -- 172.21.15.98:0/1112555443 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc035040 con 0x55fd5da94330 2024-05-09T09:00:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.217+0000 7f0114e75740 1 -- 172.21.15.98:0/1112555443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da94330 msgr2=0x55fd5da96b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.217+0000 7f0114e75740 1 --2- 172.21.15.98:0/1112555443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da94330 0x55fd5da96b70 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f00fc009230 tx=0x7f00fc0058d0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.217+0000 7f0114e75740 1 -- 172.21.15.98:0/1112555443 shutdown_connections 2024-05-09T09:00:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.217+0000 7f0114e75740 1 --2- 172.21.15.98:0/1112555443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da9f320 0x55fd5dab6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.217+0000 7f0114e75740 1 --2- 172.21.15.98:0/1112555443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da94c50 0x55fd5da9cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.217+0000 7f0114e75740 1 --2- 172.21.15.98:0/1112555443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da94330 0x55fd5da96b70 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.217+0000 7f0114e75740 1 -- 172.21.15.98:0/1112555443 >> 172.21.15.98:0/1112555443 conn(0x55fd5da90650 msgr2=0x55fd5da8f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.217+0000 7f0114e75740 1 -- 172.21.15.98:0/1112555443 shutdown_connections 2024-05-09T09:00:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.217+0000 7f0114e75740 1 -- 172.21.15.98:0/1112555443 wait complete. 2024-05-09T09:00:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0114e75740 1 -- start start 2024-05-09T09:00:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0114e75740 1 Processor -- start 2024-05-09T09:00:55.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0114e75740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94330 0x55fd5daa76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0114e75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da94c50 0x55fd5daa2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0114e75740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da9f320 0x55fd5daa2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0114e75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f010c1b0380 con 0x55fd5da94c50 2024-05-09T09:00:55.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0114e75740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f010c1afcb0 con 0x55fd5da94330 2024-05-09T09:00:55.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0114e75740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f010c1acf10 con 0x55fd5da9f320 2024-05-09T09:00:55.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0114e75740 10 client.0 _ll_register_callbacks cb 0x7f0104092040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:55.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0103fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da94c50 0x55fd5daa2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0103fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da94c50 0x55fd5daa2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52258/0 (socket says 172.21.15.98:52258) 2024-05-09T09:00:55.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0103fff640 1 -- 172.21.15.98:0/3027675033 learned_addr learned my addr 172.21.15.98:0/3027675033 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f010bfff640 1 --2- 172.21.15.98:0/3027675033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da9f320 0x55fd5daa2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f010b7fe640 1 --2- 172.21.15.98:0/3027675033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94330 0x55fd5daa76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0103fff640 1 -- 172.21.15.98:0/3027675033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da9f320 msgr2=0x55fd5daa2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0103fff640 1 --2- 172.21.15.98:0/3027675033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da9f320 0x55fd5daa2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0103fff640 1 -- 172.21.15.98:0/3027675033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94330 msgr2=0x55fd5daa76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0103fff640 1 --2- 172.21.15.98:0/3027675033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94330 0x55fd5daa76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0103fff640 1 -- 172.21.15.98:0/3027675033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd5daa35f0 con 0x55fd5da94c50 2024-05-09T09:00:55.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f010bfff640 1 --2- 172.21.15.98:0/3027675033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da9f320 0x55fd5daa2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:55.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f0103fff640 1 --2- 172.21.15.98:0/3027675033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da94c50 0x55fd5daa2a70 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f00f400a420 tx=0x7f00f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f30243586732fca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.218+0000 7f00eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:55.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.219+0000 7f00eaffd640 1 -- 172.21.15.98:0/3027675033 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f4011c80 con 0x55fd5da94c50 2024-05-09T09:00:55.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.219+0000 7f00eaffd640 1 -- 172.21.15.98:0/3027675033 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f00f4011e20 con 0x55fd5da94c50 2024-05-09T09:00:55.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.219+0000 7f0114e75740 1 -- 172.21.15.98:0/3027675033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00fc046dc0 con 0x55fd5da94c50 2024-05-09T09:00:55.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.219+0000 7f00eaffd640 1 -- 172.21.15.98:0/3027675033 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f00f4030050 con 0x55fd5da94c50 2024-05-09T09:00:55.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.219+0000 7f0114e75740 10 client.18597 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:55.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.219+0000 7f0114e75740 1 -- 172.21.15.98:0/3027675033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55fd5db154b0 con 0x55fd5da94c50 2024-05-09T09:00:55.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.219+0000 7f00eaffd640 1 -- 172.21.15.98:0/3027675033 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00fc046dc0 con 0x55fd5da94c50 2024-05-09T09:00:55.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.219+0000 7f00eaffd640 1 -- 172.21.15.98:0/3027675033 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f00f400f040 con 0x55fd5da94c50 2024-05-09T09:00:55.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.219+0000 7f0114e75740 10 client.18597 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:55.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.219+0000 7f0114e75740 10 client.18597 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T09:00:55.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.219+0000 7f0114e75740 20 client.18597 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:55.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.219+0000 7f0114e75740 1 --2- 172.21.15.98:0/3027675033 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55fd5db15cb0 0x55fd5db180d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.219+0000 7f0114e75740 4 client.18597 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:00:55.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.219+0000 7f0114e75740 1 -- 172.21.15.98:0/3027675033 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd5da016c0 con 0x55fd5db15cb0 2024-05-09T09:00:55.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f010b7fe640 1 --2- 172.21.15.98:0/3027675033 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55fd5db15cb0 0x55fd5db180d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f010b7fe640 1 --2- 172.21.15.98:0/3027675033 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55fd5db15cb0 0x55fd5db180d0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4e6abce24983d802 server_cookie=f1445a3037ec6b7d in_seq=0 out_seq=0 2024-05-09T09:00:55.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f00eaffd640 10 client.18597 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T09:00:55.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f00eaffd640 1 -- 172.21.15.98:0/3027675033 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd5da016c0 con 0x55fd5db15cb0 2024-05-09T09:00:55.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f00eaffd640 10 client.18597 handle_command_reply: tid=0 2024-05-09T09:00:55.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f0114e75740 1 client.18597 shutdown 2024-05-09T09:00:55.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f0114e75740 1 -- 172.21.15.98:0/3027675033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da94c50 msgr2=0x55fd5daa2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f0114e75740 1 --2- 172.21.15.98:0/3027675033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da94c50 0x55fd5daa2a70 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f00f400a420 tx=0x7f00f40046c0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f0114e75740 1 -- 172.21.15.98:0/3027675033 shutdown_connections 2024-05-09T09:00:55.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f0114e75740 1 --2- 172.21.15.98:0/3027675033 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55fd5db15cb0 0x55fd5db180d0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f0114e75740 1 --2- 172.21.15.98:0/3027675033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fd5da9f320 0x55fd5daa2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f00eaffd640 0 client.18597 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T09:00:55.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f0114e75740 1 --2- 172.21.15.98:0/3027675033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fd5da94c50 0x55fd5daa2a70 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f0114e75740 1 --2- 172.21.15.98:0/3027675033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fd5da94330 0x55fd5daa76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f0114e75740 1 -- 172.21.15.98:0/3027675033 >> 172.21.15.98:0/3027675033 conn(0x55fd5da90650 msgr2=0x55fd5da9d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f0114e75740 1 -- 172.21.15.98:0/3027675033 shutdown_connections 2024-05-09T09:00:55.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f0114e75740 1 -- 172.21.15.98:0/3027675033 wait complete. 2024-05-09T09:00:55.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.220+0000 7f0114e75740 20 client.18597 trim_cache size 0 max 16384 2024-05-09T09:00:55.227 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:00:55.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.221+0000 7f0112163640 1 -- 172.21.15.98:0/2933314290 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00f80a2d00 msgr2=0x7f00f80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.221+0000 7f0112163640 1 --2- 172.21.15.98:0/2933314290 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00f80a2d00 0x7f00f80a51c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f010c10db20 tx=0x7f00f40137d0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.221+0000 7f0112163640 1 -- 172.21.15.98:0/2933314290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f010c10bdf0 msgr2=0x7f010c19c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.221+0000 7f0112163640 1 --2- 172.21.15.98:0/2933314290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f010c10bdf0 0x7f010c19c9d0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f010400eaa0 tx=0x7f010400ef60 comp rx=0 tx=0).stop 2024-05-09T09:00:55.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.221+0000 7f0112163640 1 -- 172.21.15.98:0/2933314290 shutdown_connections 2024-05-09T09:00:55.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.221+0000 7f0112163640 1 --2- 172.21.15.98:0/2933314290 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f00f80a2d00 0x7f00f80a51c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.221+0000 7f0112163640 1 --2- 172.21.15.98:0/2933314290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f010c10bdf0 0x7f010c19c9d0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.221+0000 7f0112163640 1 --2- 172.21.15.98:0/2933314290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f010c1033d0 0x7f010c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.221+0000 7f0112163640 1 --2- 172.21.15.98:0/2933314290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f010c102ab0 0x7f010c198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.221+0000 7f0112163640 1 -- 172.21.15.98:0/2933314290 >> 172.21.15.98:0/2933314290 conn(0x7f010c077830 msgr2=0x7f010c0fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.221+0000 7f0112163640 1 -- 172.21.15.98:0/2933314290 shutdown_connections 2024-05-09T09:00:55.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.221+0000 7f0112163640 1 -- 172.21.15.98:0/2933314290 wait complete. 2024-05-09T09:00:55.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T09:00:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.335+0000 7f4d3b79b640 1 -- 172.21.15.98:0/4128238051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d34107860 msgr2=0x7f4d34107c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.335+0000 7f4d3b79b640 1 --2- 172.21.15.98:0/4128238051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d34107860 0x7f4d34107c40 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009cd0 tx=0x7f4d2402a8e0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.335+0000 7f4d3b79b640 1 -- 172.21.15.98:0/4128238051 shutdown_connections 2024-05-09T09:00:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.335+0000 7f4d3b79b640 1 --2- 172.21.15.98:0/4128238051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d34102210 0x7f4d3410e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.335+0000 7f4d3b79b640 1 --2- 172.21.15.98:0/4128238051 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d34101850 0x7f4d34101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.335+0000 7f4d3b79b640 1 --2- 172.21.15.98:0/4128238051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d34107860 0x7f4d34107c40 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.335+0000 7f4d3b79b640 1 -- 172.21.15.98:0/4128238051 >> 172.21.15.98:0/4128238051 conn(0x7f4d340fd340 msgr2=0x7f4d340ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.335+0000 7f4d3b79b640 1 -- 172.21.15.98:0/4128238051 shutdown_connections 2024-05-09T09:00:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.335+0000 7f4d3b79b640 1 -- 172.21.15.98:0/4128238051 wait complete. 2024-05-09T09:00:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.336+0000 7f4d3b79b640 1 Processor -- start 2024-05-09T09:00:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.336+0000 7f4d3b79b640 1 -- start start 2024-05-09T09:00:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.336+0000 7f4d3b79b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d34101850 0x7f4d34073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.336+0000 7f4d3b79b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d34102210 0x7f4d34074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.336+0000 7f4d3b79b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d34107860 0x7f4d340745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.336+0000 7f4d3b79b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d34111c70 con 0x7f4d34102210 2024-05-09T09:00:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.336+0000 7f4d3b79b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4d34111af0 con 0x7f4d34101850 2024-05-09T09:00:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.336+0000 7f4d3b79b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4d34111df0 con 0x7f4d34107860 2024-05-09T09:00:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.336+0000 7f4d38d36640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d34102210 0x7f4d34074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.336+0000 7f4d38d36640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d34102210 0x7f4d34074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52278/0 (socket says 172.21.15.98:52278) 2024-05-09T09:00:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.336+0000 7f4d38d36640 1 -- 172.21.15.98:0/3558904340 learned_addr learned my addr 172.21.15.98:0/3558904340 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.336+0000 7f4d39d38640 1 --2- 172.21.15.98:0/3558904340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d34107860 0x7f4d340745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.336+0000 7f4d39537640 1 --2- 172.21.15.98:0/3558904340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d34101850 0x7f4d34073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.337+0000 7f4d38d36640 1 -- 172.21.15.98:0/3558904340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d34107860 msgr2=0x7f4d340745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.337+0000 7f4d38d36640 1 --2- 172.21.15.98:0/3558904340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d34107860 0x7f4d340745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.337+0000 7f4d38d36640 1 -- 172.21.15.98:0/3558904340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d34101850 msgr2=0x7f4d34073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.337+0000 7f4d38d36640 1 --2- 172.21.15.98:0/3558904340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d34101850 0x7f4d34073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.337+0000 7f4d38d36640 1 -- 172.21.15.98:0/3558904340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d341a6630 con 0x7f4d34102210 2024-05-09T09:00:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.337+0000 7f4d38d36640 1 --2- 172.21.15.98:0/3558904340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d34102210 0x7f4d34074070 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f4d2800d950 tx=0x7f4d2800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.337+0000 7f4d327fc640 1 -- 172.21.15.98:0/3558904340 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28017070 con 0x7f4d34102210 2024-05-09T09:00:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.337+0000 7f4d3b79b640 1 -- 172.21.15.98:0/3558904340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d341a6920 con 0x7f4d34102210 2024-05-09T09:00:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.337+0000 7f4d327fc640 1 -- 172.21.15.98:0/3558904340 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d28009c30 con 0x7f4d34102210 2024-05-09T09:00:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.337+0000 7f4d327fc640 1 -- 172.21.15.98:0/3558904340 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2800f6b0 con 0x7f4d34102210 2024-05-09T09:00:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.337+0000 7f4d3b79b640 1 -- 172.21.15.98:0/3558904340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d341a6e70 con 0x7f4d34102210 2024-05-09T09:00:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.339+0000 7f4d327fc640 1 -- 172.21.15.98:0/3558904340 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4d280109d0 con 0x7f4d34102210 2024-05-09T09:00:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d3e583740 1 Processor -- start 2024-05-09T09:00:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d3e583740 1 -- start start 2024-05-09T09:00:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d327fc640 1 --2- 172.21.15.98:0/3558904340 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4d1c0a2d00 0x7f4d1c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d3e583740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58136330 0x55bc58136710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d3e583740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136c50 0x55bc58140de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d3e583740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58141320 0x55bc58143710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d39d38640 1 --2- 172.21.15.98:0/3558904340 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4d1c0a2d00 0x7f4d1c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d3e583740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bc5809bc40 con 0x55bc58136c50 2024-05-09T09:00:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d3e583740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bc58043580 con 0x55bc58141320 2024-05-09T09:00:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d3e583740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55bc58097c00 con 0x55bc58136330 2024-05-09T09:00:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d38d36640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136c50 0x55bc58140de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d327fc640 1 -- 172.21.15.98:0/3558904340 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f4d280cf440 con 0x7f4d34102210 2024-05-09T09:00:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d38d36640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136c50 0x55bc58140de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52280/0 (socket says 172.21.15.98:52280) 2024-05-09T09:00:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d38d36640 1 -- 172.21.15.98:0/382568732 learned_addr learned my addr 172.21.15.98:0/382568732 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d39537640 1 --2- 172.21.15.98:0/382568732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58136330 0x55bc58136710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.340+0000 7f4d39d38640 1 --2- 172.21.15.98:0/382568732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58141320 0x55bc58143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.341+0000 7f4d38d36640 1 -- 172.21.15.98:0/382568732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58136330 msgr2=0x55bc58136710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.341+0000 7f4d38d36640 1 --2- 172.21.15.98:0/382568732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58136330 0x55bc58136710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.341+0000 7f4d38d36640 1 -- 172.21.15.98:0/382568732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58141320 msgr2=0x55bc58143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.341+0000 7f4d38d36640 1 --2- 172.21.15.98:0/382568732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58141320 0x55bc58143710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.341+0000 7f4d38d36640 1 -- 172.21.15.98:0/382568732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc57daf3f0 con 0x55bc58136c50 2024-05-09T09:00:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.341+0000 7f4d38d36640 1 --2- 172.21.15.98:0/382568732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136c50 0x55bc58140de0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f4d28097a60 tx=0x7f4d28005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7830bb05ae02012 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.341+0000 7f4d39d38640 1 --2- 172.21.15.98:0/3558904340 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4d1c0a2d00 0x7f4d1c0a51c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c00afd0 tx=0x7f4d2c00af40 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.341+0000 7f4d1b7fe640 1 -- 172.21.15.98:0/382568732 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28020040 con 0x55bc58136c50 2024-05-09T09:00:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.341+0000 7f4d1b7fe640 1 -- 172.21.15.98:0/382568732 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d2806d160 con 0x55bc58136c50 2024-05-09T09:00:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.341+0000 7f4d1b7fe640 1 -- 172.21.15.98:0/382568732 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d280cca00 con 0x55bc58136c50 2024-05-09T09:00:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.341+0000 7f4d3e583740 1 -- 172.21.15.98:0/382568732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136c50 msgr2=0x55bc58140de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.341+0000 7f4d3e583740 1 --2- 172.21.15.98:0/382568732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136c50 0x55bc58140de0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f4d28097a60 tx=0x7f4d28005f90 comp rx=0 tx=0).stop 2024-05-09T09:00:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 -- 172.21.15.98:0/382568732 shutdown_connections 2024-05-09T09:00:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 --2- 172.21.15.98:0/382568732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58141320 0x55bc58143710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 --2- 172.21.15.98:0/382568732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136c50 0x55bc58140de0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 --2- 172.21.15.98:0/382568732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58136330 0x55bc58136710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 -- 172.21.15.98:0/382568732 >> 172.21.15.98:0/382568732 conn(0x55bc58132650 msgr2=0x55bc58132a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 -- 172.21.15.98:0/382568732 shutdown_connections 2024-05-09T09:00:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 -- 172.21.15.98:0/382568732 wait complete. 2024-05-09T09:00:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 -- start start 2024-05-09T09:00:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 Processor -- start 2024-05-09T09:00:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58136330 0x55bc5813e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58136c50 0x55bc5813e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58141320 0x55bc5813ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bc58144110 con 0x55bc58141320 2024-05-09T09:00:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bc58056c40 con 0x55bc58136c50 2024-05-09T09:00:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55bc58144290 con 0x55bc58136330 2024-05-09T09:00:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d3e583740 10 client.0 _ll_register_callbacks cb 0x55bc57daebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.342+0000 7f4d39d38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58141320 0x55bc5813ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d39d38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58141320 0x55bc5813ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52296/0 (socket says 172.21.15.98:52296) 2024-05-09T09:00:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d39d38640 1 -- 172.21.15.98:0/864108907 learned_addr learned my addr 172.21.15.98:0/864108907 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d38d36640 1 --2- 172.21.15.98:0/864108907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58136c50 0x55bc5813e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d39d38640 1 -- 172.21.15.98:0/864108907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58136330 msgr2=0x55bc5813e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d39537640 1 --2- 172.21.15.98:0/864108907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58136330 0x55bc5813e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d39d38640 1 --2- 172.21.15.98:0/864108907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58136330 0x55bc5813e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d39d38640 1 -- 172.21.15.98:0/864108907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58136c50 msgr2=0x55bc5813e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d39d38640 1 --2- 172.21.15.98:0/864108907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58136c50 0x55bc5813e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d39d38640 1 -- 172.21.15.98:0/864108907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc57e797d0 con 0x55bc58141320 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d39d38640 1 --2- 172.21.15.98:0/864108907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58141320 0x55bc5813ed20 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c011c90 tx=0x7f4d2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d163508b58f01218 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d18ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d18ff9640 1 -- 172.21.15.98:0/864108907 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c024070 con 0x55bc58141320 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d18ff9640 1 -- 172.21.15.98:0/864108907 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f4d2c005220 con 0x55bc58141320 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d3e583740 1 -- 172.21.15.98:0/864108907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d2806d160 con 0x55bc58141320 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d18ff9640 1 -- 172.21.15.98:0/864108907 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f4d2c031050 con 0x55bc58141320 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d3e583740 10 client.18609 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d3e583740 1 -- 172.21.15.98:0/864108907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bc58150320 con 0x55bc58141320 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.343+0000 7f4d18ff9640 1 -- 172.21.15.98:0/864108907 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d2806d160 con 0x55bc58141320 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.344+0000 7f4d18ff9640 1 -- 172.21.15.98:0/864108907 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f4d2c01f070 con 0x55bc58141320 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.344+0000 7f4d3e583740 10 client.18609 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.344+0000 7f4d3e583740 10 client.18609 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.344+0000 7f4d3e583740 20 client.18609 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.344+0000 7f4d3e583740 1 --2- 172.21.15.98:0/864108907 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55bc581517e0 0x55bc58153c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.344+0000 7f4d3e583740 4 client.18609 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.344+0000 7f4d3e583740 1 -- 172.21.15.98:0/864108907 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc580a9370 con 0x55bc581517e0 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.344+0000 7f4d39537640 1 --2- 172.21.15.98:0/864108907 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55bc581517e0 0x55bc58153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.344+0000 7f4d39537640 1 --2- 172.21.15.98:0/864108907 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55bc581517e0 0x55bc58153c10 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d3b4821b90c51818 server_cookie=acad49be16d6493c in_seq=0 out_seq=0 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.344+0000 7f4d18ff9640 10 client.18609 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d18ff9640 1 -- 172.21.15.98:0/864108907 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bc580a9370 con 0x55bc581517e0 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d18ff9640 10 client.18609 handle_command_reply: tid=0 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d3e583740 1 client.18609 shutdown 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d3e583740 1 -- 172.21.15.98:0/864108907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58141320 msgr2=0x55bc5813ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d3e583740 1 --2- 172.21.15.98:0/864108907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58141320 0x55bc5813ed20 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c011c90 tx=0x7f4d2c004730 comp rx=0 tx=0).stop 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d3e583740 1 -- 172.21.15.98:0/864108907 shutdown_connections 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d3e583740 1 --2- 172.21.15.98:0/864108907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58141320 0x55bc5813ed20 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d3e583740 1 --2- 172.21.15.98:0/864108907 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55bc581517e0 0x55bc58153c10 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d3e583740 1 --2- 172.21.15.98:0/864108907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58136c50 0x55bc5813e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d3e583740 1 --2- 172.21.15.98:0/864108907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58136330 0x55bc5813e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d18ff9640 0 client.18609 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T09:00:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d3e583740 1 -- 172.21.15.98:0/864108907 >> 172.21.15.98:0/864108907 conn(0x55bc58132650 msgr2=0x55bc581317b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d3e583740 1 -- 172.21.15.98:0/864108907 shutdown_connections 2024-05-09T09:00:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d3e583740 1 -- 172.21.15.98:0/864108907 wait complete. 2024-05-09T09:00:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.345+0000 7f4d3e583740 20 client.18609 trim_cache size 0 max 16384 2024-05-09T09:00:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.359+0000 7f4d3e583740 1 Processor -- start 2024-05-09T09:00:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.359+0000 7f4d3e583740 1 -- start start 2024-05-09T09:00:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.359+0000 7f4d3e583740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58136330 0x55bc581409d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.359+0000 7f4d3e583740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136c50 0x55bc5814d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.359+0000 7f4d3e583740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58141320 0x55bc58131630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.359+0000 7f4d3e583740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d341b1040 con 0x55bc58136c50 2024-05-09T09:00:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.359+0000 7f4d3e583740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4d341b2b80 con 0x55bc58141320 2024-05-09T09:00:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.359+0000 7f4d3e583740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4d341b1710 con 0x55bc58136330 2024-05-09T09:00:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.359+0000 7f4d38d36640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136c50 0x55bc5814d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.359+0000 7f4d38d36640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136c50 0x55bc5814d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52308/0 (socket says 172.21.15.98:52308) 2024-05-09T09:00:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.359+0000 7f4d38d36640 1 -- 172.21.15.98:0/1232918390 learned_addr learned my addr 172.21.15.98:0/1232918390 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.359+0000 7f4d39d38640 1 --2- 172.21.15.98:0/1232918390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58141320 0x55bc58131630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.359+0000 7f4d38d36640 1 -- 172.21.15.98:0/1232918390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58136330 msgr2=0x55bc581409d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:00:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.359+0000 7f4d38d36640 1 --2- 172.21.15.98:0/1232918390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58136330 0x55bc581409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d38d36640 1 -- 172.21.15.98:0/1232918390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58141320 msgr2=0x55bc58131630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d38d36640 1 --2- 172.21.15.98:0/1232918390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58141320 0x55bc58131630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d38d36640 1 -- 172.21.15.98:0/1232918390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc58123d20 con 0x55bc58136c50 2024-05-09T09:00:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d39d38640 1 --2- 172.21.15.98:0/1232918390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58141320 0x55bc58131630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d38d36640 1 --2- 172.21.15.98:0/1232918390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136c50 0x55bc5814d000 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f4d280ccd80 tx=0x7f4d28005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cae9dc02537530 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d1affd640 1 -- 172.21.15.98:0/1232918390 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d280b4070 con 0x55bc58136c50 2024-05-09T09:00:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d1affd640 1 -- 172.21.15.98:0/1232918390 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d2809b600 con 0x55bc58136c50 2024-05-09T09:00:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d1affd640 1 -- 172.21.15.98:0/1232918390 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28020070 con 0x55bc58136c50 2024-05-09T09:00:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d3e583740 1 -- 172.21.15.98:0/1232918390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136c50 msgr2=0x55bc5814d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d3e583740 1 --2- 172.21.15.98:0/1232918390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136c50 0x55bc5814d000 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f4d280ccd80 tx=0x7f4d28005880 comp rx=0 tx=0).stop 2024-05-09T09:00:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d3e583740 1 -- 172.21.15.98:0/1232918390 shutdown_connections 2024-05-09T09:00:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d3e583740 1 --2- 172.21.15.98:0/1232918390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58141320 0x55bc58131630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d3e583740 1 --2- 172.21.15.98:0/1232918390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136c50 0x55bc5814d000 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d3e583740 1 --2- 172.21.15.98:0/1232918390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58136330 0x55bc581409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.360+0000 7f4d3e583740 1 -- 172.21.15.98:0/1232918390 >> 172.21.15.98:0/1232918390 conn(0x55bc58132650 msgr2=0x55bc58130f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.361+0000 7f4d3e583740 1 -- 172.21.15.98:0/1232918390 shutdown_connections 2024-05-09T09:00:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.361+0000 7f4d3e583740 1 -- 172.21.15.98:0/1232918390 wait complete. 2024-05-09T09:00:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.361+0000 7f4d3e583740 1 -- start start 2024-05-09T09:00:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.361+0000 7f4d3e583740 1 Processor -- start 2024-05-09T09:00:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.361+0000 7f4d3e583740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136330 0x55bc5813b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.361+0000 7f4d3e583740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58136c50 0x55bc5813b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.361+0000 7f4d3e583740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58141320 0x55bc5813bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.361+0000 7f4d3e583740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d341b24b0 con 0x55bc58136330 2024-05-09T09:00:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.361+0000 7f4d3e583740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4d341b1de0 con 0x55bc58136c50 2024-05-09T09:00:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.361+0000 7f4d3e583740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4d341af040 con 0x55bc58141320 2024-05-09T09:00:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.361+0000 7f4d3e583740 10 client.0 _ll_register_callbacks cb 0x7f4d2c005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.361+0000 7f4d39537640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136330 0x55bc5813b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.361+0000 7f4d39537640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136330 0x55bc5813b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52316/0 (socket says 172.21.15.98:52316) 2024-05-09T09:00:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.361+0000 7f4d39537640 1 -- 172.21.15.98:0/1584106608 learned_addr learned my addr 172.21.15.98:0/1584106608 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d39537640 1 -- 172.21.15.98:0/1584106608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58141320 msgr2=0x55bc5813bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d39537640 1 --2- 172.21.15.98:0/1584106608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58141320 0x55bc5813bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d38d36640 1 --2- 172.21.15.98:0/1584106608 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58136c50 0x55bc5813b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d39537640 1 -- 172.21.15.98:0/1584106608 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58136c50 msgr2=0x55bc5813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d39537640 1 --2- 172.21.15.98:0/1584106608 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58136c50 0x55bc5813b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d39537640 1 -- 172.21.15.98:0/1584106608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc581b6e50 con 0x55bc58136330 2024-05-09T09:00:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d38d36640 1 --2- 172.21.15.98:0/1584106608 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58136c50 0x55bc5813b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d39537640 1 --2- 172.21.15.98:0/1584106608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136330 0x55bc5813b0b0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f4d2402ae00 tx=0x7f4d240316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f3292a9c5b4efc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d18ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d18ff9640 1 -- 172.21.15.98:0/1584106608 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24031ad0 con 0x55bc58136330 2024-05-09T09:00:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d18ff9640 1 -- 172.21.15.98:0/1584106608 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f4d24031c70 con 0x55bc58136330 2024-05-09T09:00:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d3e583740 1 -- 172.21.15.98:0/1584106608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d2809b600 con 0x55bc58136330 2024-05-09T09:00:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d18ff9640 1 -- 172.21.15.98:0/1584106608 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f4d24003970 con 0x55bc58136330 2024-05-09T09:00:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d3e583740 10 client.18615 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d3e583740 1 -- 172.21.15.98:0/1584106608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bc581b80d0 con 0x55bc58136330 2024-05-09T09:00:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d18ff9640 1 -- 172.21.15.98:0/1584106608 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d2809b600 con 0x55bc58136330 2024-05-09T09:00:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d18ff9640 1 -- 172.21.15.98:0/1584106608 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f4d24030070 con 0x55bc58136330 2024-05-09T09:00:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d3e583740 10 client.18615 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d3e583740 10 client.18615 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:00:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d3e583740 20 client.18615 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d3e583740 1 --2- 172.21.15.98:0/1584106608 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55bc581b88b0 0x55bc581bacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d3e583740 4 client.18615 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.362+0000 7f4d3e583740 1 -- 172.21.15.98:0/1584106608 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bc580a36c0 con 0x55bc581b88b0 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d38d36640 1 --2- 172.21.15.98:0/1584106608 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55bc581b88b0 0x55bc581bacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d18ff9640 10 client.18615 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d38d36640 1 --2- 172.21.15.98:0/1584106608 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55bc581b88b0 0x55bc581bacd0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1681edb0fe30a0e7 server_cookie=c8999dab181972af in_seq=0 out_seq=0 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d18ff9640 1 -- 172.21.15.98:0/1584106608 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bc580a36c0 con 0x55bc581b88b0 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d18ff9640 10 client.18615 handle_command_reply: tid=0 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d3e583740 1 client.18615 shutdown 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d3e583740 1 -- 172.21.15.98:0/1584106608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136330 msgr2=0x55bc5813b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d3e583740 1 --2- 172.21.15.98:0/1584106608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136330 0x55bc5813b0b0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f4d2402ae00 tx=0x7f4d240316e0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d3e583740 1 -- 172.21.15.98:0/1584106608 shutdown_connections 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d3e583740 1 --2- 172.21.15.98:0/1584106608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bc58141320 0x55bc5813bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d3e583740 1 --2- 172.21.15.98:0/1584106608 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55bc581b88b0 0x55bc581bacd0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d3e583740 1 --2- 172.21.15.98:0/1584106608 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc58136c50 0x55bc5813b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d3e583740 1 --2- 172.21.15.98:0/1584106608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc58136330 0x55bc5813b0b0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d18ff9640 0 client.18615 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T09:00:55.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d3e583740 1 -- 172.21.15.98:0/1584106608 >> 172.21.15.98:0/1584106608 conn(0x55bc58132650 msgr2=0x55bc581317b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d3e583740 1 -- 172.21.15.98:0/1584106608 shutdown_connections 2024-05-09T09:00:55.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.363+0000 7f4d3e583740 1 -- 172.21.15.98:0/1584106608 wait complete. 2024-05-09T09:00:55.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.364+0000 7f4d3e583740 20 client.18615 trim_cache size 0 max 16384 2024-05-09T09:00:55.370 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:00:55.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.364+0000 7f4d3b79b640 1 -- 172.21.15.98:0/3558904340 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4d1c0a2d00 msgr2=0x7f4d1c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.364+0000 7f4d3b79b640 1 --2- 172.21.15.98:0/3558904340 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4d1c0a2d00 0x7f4d1c0a51c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c00afd0 tx=0x7f4d2c00af40 comp rx=0 tx=0).stop 2024-05-09T09:00:55.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.364+0000 7f4d3b79b640 1 -- 172.21.15.98:0/3558904340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d34102210 msgr2=0x7f4d34074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.364+0000 7f4d3b79b640 1 --2- 172.21.15.98:0/3558904340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d34102210 0x7f4d34074070 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f4d2800d950 tx=0x7f4d2800de10 comp rx=0 tx=0).stop 2024-05-09T09:00:55.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.364+0000 7f4d3b79b640 1 -- 172.21.15.98:0/3558904340 shutdown_connections 2024-05-09T09:00:55.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.364+0000 7f4d3b79b640 1 --2- 172.21.15.98:0/3558904340 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4d1c0a2d00 0x7f4d1c0a51c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.364+0000 7f4d3b79b640 1 --2- 172.21.15.98:0/3558904340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d34107860 0x7f4d340745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.364+0000 7f4d3b79b640 1 --2- 172.21.15.98:0/3558904340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d34102210 0x7f4d34074070 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.364+0000 7f4d3b79b640 1 --2- 172.21.15.98:0/3558904340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d34101850 0x7f4d34073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.364+0000 7f4d3b79b640 1 -- 172.21.15.98:0/3558904340 >> 172.21.15.98:0/3558904340 conn(0x7f4d340fd340 msgr2=0x7f4d340fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.364+0000 7f4d3b79b640 1 -- 172.21.15.98:0/3558904340 shutdown_connections 2024-05-09T09:00:55.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.364+0000 7f4d3b79b640 1 -- 172.21.15.98:0/3558904340 wait complete. 2024-05-09T09:00:55.373 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:55 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2346178472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:00:55.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T09:00:55.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.478+0000 7fc421f89640 1 -- 172.21.15.98:0/180030744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc41c107810 msgr2=0x7fc41c107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.478+0000 7fc421f89640 1 --2- 172.21.15.98:0/180030744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc41c107810 0x7fc41c107bf0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009930 tx=0x7fc40c02a420 comp rx=0 tx=0).stop 2024-05-09T09:00:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.478+0000 7fc421f89640 1 -- 172.21.15.98:0/180030744 shutdown_connections 2024-05-09T09:00:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.478+0000 7fc421f89640 1 --2- 172.21.15.98:0/180030744 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc41c1021c0 0x7fc41c10e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.478+0000 7fc421f89640 1 --2- 172.21.15.98:0/180030744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc41c101800 0x7fc41c101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.478+0000 7fc421f89640 1 --2- 172.21.15.98:0/180030744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc41c107810 0x7fc41c107bf0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.478+0000 7fc421f89640 1 -- 172.21.15.98:0/180030744 >> 172.21.15.98:0/180030744 conn(0x7fc41c078c80 msgr2=0x7fc41c079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.478+0000 7fc421f89640 1 -- 172.21.15.98:0/180030744 shutdown_connections 2024-05-09T09:00:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.479+0000 7fc421f89640 1 -- 172.21.15.98:0/180030744 wait complete. 2024-05-09T09:00:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.479+0000 7fc421f89640 1 Processor -- start 2024-05-09T09:00:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.479+0000 7fc421f89640 1 -- start start 2024-05-09T09:00:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.479+0000 7fc421f89640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc41c101800 0x7fc41c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.479+0000 7fc421f89640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc41c1021c0 0x7fc41c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.479+0000 7fc421f89640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc41c107810 0x7fc41c0745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.479+0000 7fc421f89640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c111be0 con 0x7fc41c101800 2024-05-09T09:00:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.479+0000 7fc421f89640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c111a60 con 0x7fc41c107810 2024-05-09T09:00:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.479+0000 7fc421f89640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c111d60 con 0x7fc41c1021c0 2024-05-09T09:00:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc41b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc41c101800 0x7fc41c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc41b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc41c101800 0x7fc41c073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52346/0 (socket says 172.21.15.98:52346) 2024-05-09T09:00:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc41b7fe640 1 -- 172.21.15.98:0/3274233201 learned_addr learned my addr 172.21.15.98:0/3274233201 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc412ffd640 1 --2- 172.21.15.98:0/3274233201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc41c1021c0 0x7fc41c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc41bfff640 1 --2- 172.21.15.98:0/3274233201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc41c107810 0x7fc41c0745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc41b7fe640 1 -- 172.21.15.98:0/3274233201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc41c1021c0 msgr2=0x7fc41c074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc41b7fe640 1 --2- 172.21.15.98:0/3274233201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc41c1021c0 0x7fc41c074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc41b7fe640 1 -- 172.21.15.98:0/3274233201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc41c107810 msgr2=0x7fc41c0745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc41b7fe640 1 --2- 172.21.15.98:0/3274233201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc41c107810 0x7fc41c0745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc41b7fe640 1 -- 172.21.15.98:0/3274233201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc41c1a6600 con 0x7fc41c101800 2024-05-09T09:00:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc41b7fe640 1 --2- 172.21.15.98:0/3274233201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc41c101800 0x7fc41c073b30 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009a50 tx=0x7fc40c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc4197fa640 1 -- 172.21.15.98:0/3274233201 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c03f030 con 0x7fc41c101800 2024-05-09T09:00:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc421f89640 1 -- 172.21.15.98:0/3274233201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc41c1a6890 con 0x7fc41c101800 2024-05-09T09:00:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc4197fa640 1 -- 172.21.15.98:0/3274233201 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc40c003ce0 con 0x7fc41c101800 2024-05-09T09:00:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc4197fa640 1 -- 172.21.15.98:0/3274233201 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c035070 con 0x7fc41c101800 2024-05-09T09:00:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.480+0000 7fc421f89640 1 -- 172.21.15.98:0/3274233201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc41c1a6d80 con 0x7fc41c101800 2024-05-09T09:00:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.482+0000 7fc4197fa640 1 -- 172.21.15.98:0/3274233201 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc40c042030 con 0x7fc41c101800 2024-05-09T09:00:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.483+0000 7fc424d72740 1 Processor -- start 2024-05-09T09:00:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.483+0000 7fc424d72740 1 -- start start 2024-05-09T09:00:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.483+0000 7fc424d72740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bdda330 0x562f6bdda710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.483+0000 7fc424d72740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bddac50 0x562f6bde4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.483+0000 7fc424d72740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bde5320 0x562f6bde7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.483+0000 7fc424d72740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562f6bd3fc40 con 0x562f6bde5320 2024-05-09T09:00:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.483+0000 7fc424d72740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562f6bce7580 con 0x562f6bddac50 2024-05-09T09:00:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.483+0000 7fc424d72740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562f6bd3bc00 con 0x562f6bdda330 2024-05-09T09:00:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.483+0000 7fc412ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bde5320 0x562f6bde7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.483+0000 7fc412ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bde5320 0x562f6bde7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52350/0 (socket says 172.21.15.98:52350) 2024-05-09T09:00:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.483+0000 7fc4197fa640 1 --2- 172.21.15.98:0/3274233201 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc4080a2d00 0x7fc4080a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.483+0000 7fc412ffd640 1 -- 172.21.15.98:0/158591261 learned_addr learned my addr 172.21.15.98:0/158591261 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.484+0000 7fc41bfff640 1 --2- 172.21.15.98:0/158591261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bdda330 0x562f6bdda710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.483+0000 7fc41b7fe640 1 --2- 172.21.15.98:0/158591261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bddac50 0x562f6bde4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.484+0000 7fc41bfff640 1 --2- 172.21.15.98:0/3274233201 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc4080a2d00 0x7fc4080a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.484+0000 7fc412ffd640 1 -- 172.21.15.98:0/158591261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bdda330 msgr2=0x562f6bdda710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.484+0000 7fc412ffd640 1 --2- 172.21.15.98:0/158591261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bdda330 0x562f6bdda710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.484+0000 7fc412ffd640 1 -- 172.21.15.98:0/158591261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bddac50 msgr2=0x562f6bde4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.484+0000 7fc412ffd640 1 --2- 172.21.15.98:0/158591261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bddac50 0x562f6bde4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.484+0000 7fc4197fa640 1 -- 172.21.15.98:0/3274233201 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7fc40c0eff10 con 0x7fc41c101800 2024-05-09T09:00:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.484+0000 7fc412ffd640 1 -- 172.21.15.98:0/158591261 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f6ba533f0 con 0x562f6bde5320 2024-05-09T09:00:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.484+0000 7fc41bfff640 1 --2- 172.21.15.98:0/158591261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bdda330 0x562f6bdda710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.484+0000 7fc412ffd640 1 --2- 172.21.15.98:0/158591261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bde5320 0x562f6bde7710 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fc40400cbc0 tx=0x7fc40400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230aafdd9915d5c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.484+0000 7fc41bfff640 1 --2- 172.21.15.98:0/3274233201 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc4080a2d00 0x7fc4080a51c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc41400bfd0 tx=0x7fc414004a40 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.484+0000 7fc411ffb640 1 -- 172.21.15.98:0/158591261 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc404010630 con 0x562f6bde5320 2024-05-09T09:00:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.484+0000 7fc411ffb640 1 -- 172.21.15.98:0/158591261 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc404010cb0 con 0x562f6bde5320 2024-05-09T09:00:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.484+0000 7fc411ffb640 1 -- 172.21.15.98:0/158591261 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40401ac00 con 0x562f6bde5320 2024-05-09T09:00:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.485+0000 7fc424d72740 1 -- 172.21.15.98:0/158591261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bde5320 msgr2=0x562f6bde7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.485+0000 7fc424d72740 1 --2- 172.21.15.98:0/158591261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bde5320 0x562f6bde7710 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fc40400cbc0 tx=0x7fc40400e720 comp rx=0 tx=0).stop 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.485+0000 7fc424d72740 1 -- 172.21.15.98:0/158591261 shutdown_connections 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.485+0000 7fc424d72740 1 --2- 172.21.15.98:0/158591261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bde5320 0x562f6bde7710 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.485+0000 7fc424d72740 1 --2- 172.21.15.98:0/158591261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bddac50 0x562f6bde4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.485+0000 7fc424d72740 1 --2- 172.21.15.98:0/158591261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bdda330 0x562f6bdda710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.485+0000 7fc424d72740 1 -- 172.21.15.98:0/158591261 >> 172.21.15.98:0/158591261 conn(0x562f6bdd6650 msgr2=0x562f6bdd6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.485+0000 7fc424d72740 1 -- 172.21.15.98:0/158591261 shutdown_connections 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.485+0000 7fc424d72740 1 -- 172.21.15.98:0/158591261 wait complete. 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.485+0000 7fc424d72740 1 -- start start 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.485+0000 7fc424d72740 1 Processor -- start 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc424d72740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bdda330 0x562f6bde2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc424d72740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bddac50 0x562f6bde2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc424d72740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bde5320 0x562f6bde2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc424d72740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562f6bde8110 con 0x562f6bddac50 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc424d72740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562f6bcfac40 con 0x562f6bde5320 2024-05-09T09:00:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc424d72740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562f6bde8290 con 0x562f6bdda330 2024-05-09T09:00:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc424d72740 10 client.0 _ll_register_callbacks cb 0x562f6ba52be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc412ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bddac50 0x562f6bde2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc412ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bddac50 0x562f6bde2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52352/0 (socket says 172.21.15.98:52352) 2024-05-09T09:00:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc412ffd640 1 -- 172.21.15.98:0/712083416 learned_addr learned my addr 172.21.15.98:0/712083416 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc41b7fe640 1 --2- 172.21.15.98:0/712083416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bdda330 0x562f6bde2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc41bfff640 1 --2- 172.21.15.98:0/712083416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bde5320 0x562f6bde2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc412ffd640 1 -- 172.21.15.98:0/712083416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bdda330 msgr2=0x562f6bde2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc412ffd640 1 --2- 172.21.15.98:0/712083416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bdda330 0x562f6bde2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc412ffd640 1 -- 172.21.15.98:0/712083416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bde5320 msgr2=0x562f6bde2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc412ffd640 1 --2- 172.21.15.98:0/712083416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bde5320 0x562f6bde2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc412ffd640 1 -- 172.21.15.98:0/712083416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f6bb1d7d0 con 0x562f6bddac50 2024-05-09T09:00:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc412ffd640 1 --2- 172.21.15.98:0/712083416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bddac50 0x562f6bde2790 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fc404000c00 tx=0x7fc40400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=370015b07120523c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc3faffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc3faffd640 1 -- 172.21.15.98:0/712083416 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc404010e60 con 0x562f6bddac50 2024-05-09T09:00:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc3faffd640 1 -- 172.21.15.98:0/712083416 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7fc404005260 con 0x562f6bddac50 2024-05-09T09:00:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc424d72740 1 -- 172.21.15.98:0/712083416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc404010cb0 con 0x562f6bddac50 2024-05-09T09:00:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc3faffd640 1 -- 172.21.15.98:0/712083416 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc40401ae60 con 0x562f6bddac50 2024-05-09T09:00:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc424d72740 10 client.18627 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.486+0000 7fc424d72740 1 -- 172.21.15.98:0/712083416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562f6bdf42b0 con 0x562f6bddac50 2024-05-09T09:00:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.487+0000 7fc3faffd640 1 -- 172.21.15.98:0/712083416 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc404010cb0 con 0x562f6bddac50 2024-05-09T09:00:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.487+0000 7fc3faffd640 1 -- 172.21.15.98:0/712083416 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fc4040046c0 con 0x562f6bddac50 2024-05-09T09:00:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.487+0000 7fc424d72740 10 client.18627 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.487+0000 7fc424d72740 10 client.18627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.487+0000 7fc424d72740 20 client.18627 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.487+0000 7fc424d72740 1 --2- 172.21.15.98:0/712083416 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562f6bdf5710 0x562f6bdf7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.487+0000 7fc424d72740 4 client.18627 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.487+0000 7fc424d72740 1 -- 172.21.15.98:0/712083416 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f6bd4d370 con 0x562f6bdf5710 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.487+0000 7fc41b7fe640 1 --2- 172.21.15.98:0/712083416 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562f6bdf5710 0x562f6bdf7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.487+0000 7fc3faffd640 10 client.18627 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.487+0000 7fc41b7fe640 1 --2- 172.21.15.98:0/712083416 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562f6bdf5710 0x562f6bdf7b00 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca47e7ad445cf2d server_cookie=e21314c61d290d44 in_seq=0 out_seq=0 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc3faffd640 1 -- 172.21.15.98:0/712083416 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562f6bd4d370 con 0x562f6bdf5710 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc3faffd640 10 client.18627 handle_command_reply: tid=0 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc424d72740 1 client.18627 shutdown 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc424d72740 1 -- 172.21.15.98:0/712083416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bddac50 msgr2=0x562f6bde2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc424d72740 1 --2- 172.21.15.98:0/712083416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bddac50 0x562f6bde2790 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fc404000c00 tx=0x7fc40400c900 comp rx=0 tx=0).stop 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc424d72740 1 -- 172.21.15.98:0/712083416 shutdown_connections 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc424d72740 1 --2- 172.21.15.98:0/712083416 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562f6bdf5710 0x562f6bdf7b00 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc424d72740 1 --2- 172.21.15.98:0/712083416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bde5320 0x562f6bde2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc424d72740 1 --2- 172.21.15.98:0/712083416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bddac50 0x562f6bde2790 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc3faffd640 0 client.18627 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc424d72740 1 --2- 172.21.15.98:0/712083416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bdda330 0x562f6bde2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc424d72740 1 -- 172.21.15.98:0/712083416 >> 172.21.15.98:0/712083416 conn(0x562f6bdd6650 msgr2=0x562f6bddc450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc424d72740 1 -- 172.21.15.98:0/712083416 shutdown_connections 2024-05-09T09:00:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc424d72740 1 -- 172.21.15.98:0/712083416 wait complete. 2024-05-09T09:00:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.488+0000 7fc424d72740 20 client.18627 trim_cache size 0 max 16384 2024-05-09T09:00:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc424d72740 1 Processor -- start 2024-05-09T09:00:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc424d72740 1 -- start start 2024-05-09T09:00:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc424d72740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bdda330 0x562f6bdf0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc424d72740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bddac50 0x562f6bde13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc424d72740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bde5320 0x562f6bddcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc424d72740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c1b1050 con 0x562f6bdda330 2024-05-09T09:00:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc424d72740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c1b2b90 con 0x562f6bde5320 2024-05-09T09:00:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc424d72740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c1b1720 con 0x562f6bddac50 2024-05-09T09:00:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc41b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bdda330 0x562f6bdf0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc41b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bdda330 0x562f6bdf0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52368/0 (socket says 172.21.15.98:52368) 2024-05-09T09:00:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc41b7fe640 1 -- 172.21.15.98:0/2477231257 learned_addr learned my addr 172.21.15.98:0/2477231257 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc41b7fe640 1 -- 172.21.15.98:0/2477231257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bddac50 msgr2=0x562f6bde13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc41b7fe640 1 --2- 172.21.15.98:0/2477231257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bddac50 0x562f6bde13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc41b7fe640 1 -- 172.21.15.98:0/2477231257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bde5320 msgr2=0x562f6bddcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc41b7fe640 1 --2- 172.21.15.98:0/2477231257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bde5320 0x562f6bddcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc41b7fe640 1 -- 172.21.15.98:0/2477231257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f6bdf80c0 con 0x562f6bdda330 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc41b7fe640 1 --2- 172.21.15.98:0/2477231257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bdda330 0x562f6bdf0fd0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fc40c0b83d0 tx=0x7fc40c08f030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe1a4774e636575 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc4117fa640 1 -- 172.21.15.98:0/2477231257 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c049040 con 0x562f6bdda330 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc4117fa640 1 -- 172.21.15.98:0/2477231257 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc40c0bbfc0 con 0x562f6bdda330 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.502+0000 7fc4117fa640 1 -- 172.21.15.98:0/2477231257 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c0b8630 con 0x562f6bdda330 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.503+0000 7fc424d72740 1 -- 172.21.15.98:0/2477231257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bdda330 msgr2=0x562f6bdf0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.503+0000 7fc424d72740 1 --2- 172.21.15.98:0/2477231257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bdda330 0x562f6bdf0fd0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fc40c0b83d0 tx=0x7fc40c08f030 comp rx=0 tx=0).stop 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.503+0000 7fc424d72740 1 -- 172.21.15.98:0/2477231257 shutdown_connections 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.503+0000 7fc424d72740 1 --2- 172.21.15.98:0/2477231257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bde5320 0x562f6bddcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.503+0000 7fc424d72740 1 --2- 172.21.15.98:0/2477231257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bddac50 0x562f6bde13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.503+0000 7fc424d72740 1 --2- 172.21.15.98:0/2477231257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bdda330 0x562f6bdf0fd0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.503+0000 7fc424d72740 1 -- 172.21.15.98:0/2477231257 >> 172.21.15.98:0/2477231257 conn(0x562f6bdd6650 msgr2=0x562f6bde3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.503+0000 7fc424d72740 1 -- 172.21.15.98:0/2477231257 shutdown_connections 2024-05-09T09:00:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.503+0000 7fc424d72740 1 -- 172.21.15.98:0/2477231257 wait complete. 2024-05-09T09:00:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.503+0000 7fc424d72740 1 -- start start 2024-05-09T09:00:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc424d72740 1 Processor -- start 2024-05-09T09:00:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc424d72740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bdda330 0x562f6bde8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc424d72740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bddac50 0x562f6bde93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc424d72740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bde5320 0x562f6bde9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc424d72740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c1b24c0 con 0x562f6bddac50 2024-05-09T09:00:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc424d72740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c1b1df0 con 0x562f6bdda330 2024-05-09T09:00:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc424d72740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c1af050 con 0x562f6bde5320 2024-05-09T09:00:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc424d72740 10 client.0 _ll_register_callbacks cb 0x7fc404014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc412ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bddac50 0x562f6bde93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc412ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bddac50 0x562f6bde93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52382/0 (socket says 172.21.15.98:52382) 2024-05-09T09:00:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc412ffd640 1 -- 172.21.15.98:0/1113722929 learned_addr learned my addr 172.21.15.98:0/1113722929 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc41bfff640 1 --2- 172.21.15.98:0/1113722929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bde5320 0x562f6bde9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc412ffd640 1 -- 172.21.15.98:0/1113722929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bde5320 msgr2=0x562f6bde9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc412ffd640 1 --2- 172.21.15.98:0/1113722929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bde5320 0x562f6bde9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc412ffd640 1 -- 172.21.15.98:0/1113722929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bdda330 msgr2=0x562f6bde8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc412ffd640 1 --2- 172.21.15.98:0/1113722929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bdda330 0x562f6bde8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc412ffd640 1 -- 172.21.15.98:0/1113722929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f6be63370 con 0x562f6bddac50 2024-05-09T09:00:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc412ffd640 1 --2- 172.21.15.98:0/1113722929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bddac50 0x562f6bde93e0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fc40401a590 tx=0x7fc404004890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2be10678d590caa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc3faffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc3faffd640 1 -- 172.21.15.98:0/1113722929 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40401ac90 con 0x562f6bddac50 2024-05-09T09:00:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc3faffd640 1 -- 172.21.15.98:0/1113722929 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7fc40401a680 con 0x562f6bddac50 2024-05-09T09:00:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.504+0000 7fc424d72740 1 -- 172.21.15.98:0/1113722929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc40c0bbfc0 con 0x562f6bddac50 2024-05-09T09:00:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.505+0000 7fc3faffd640 1 -- 172.21.15.98:0/1113722929 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc40400ca10 con 0x562f6bddac50 2024-05-09T09:00:55.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.505+0000 7fc424d72740 10 client.18633 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.505+0000 7fc424d72740 1 -- 172.21.15.98:0/1113722929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562f6be64610 con 0x562f6bddac50 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.505+0000 7fc3faffd640 1 -- 172.21.15.98:0/1113722929 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc40c0bbfc0 con 0x562f6bddac50 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.505+0000 7fc3faffd640 1 -- 172.21.15.98:0/1113722929 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fc404030030 con 0x562f6bddac50 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.505+0000 7fc424d72740 10 client.18633 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.505+0000 7fc424d72740 10 client.18633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.505+0000 7fc424d72740 20 client.18633 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.505+0000 7fc424d72740 1 --2- 172.21.15.98:0/1113722929 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562f6be64d80 0x562f6be671a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.505+0000 7fc424d72740 4 client.18633 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.505+0000 7fc424d72740 1 -- 172.21.15.98:0/1113722929 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562f6bd476c0 con 0x562f6be64d80 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.505+0000 7fc41b7fe640 1 --2- 172.21.15.98:0/1113722929 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562f6be64d80 0x562f6be671a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc3faffd640 10 client.18633 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.505+0000 7fc41b7fe640 1 --2- 172.21.15.98:0/1113722929 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562f6be64d80 0x562f6be671a0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3baa111f4618ac0 server_cookie=8c7092a5b3976ff0 in_seq=0 out_seq=0 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc3faffd640 1 -- 172.21.15.98:0/1113722929 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562f6bd476c0 con 0x562f6be64d80 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc3faffd640 10 client.18633 handle_command_reply: tid=0 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc424d72740 1 client.18633 shutdown 2024-05-09T09:00:55.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc424d72740 1 -- 172.21.15.98:0/1113722929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bddac50 msgr2=0x562f6bde93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc424d72740 1 --2- 172.21.15.98:0/1113722929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bddac50 0x562f6bde93e0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fc40401a590 tx=0x7fc404004890 comp rx=0 tx=0).stop 2024-05-09T09:00:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc424d72740 1 -- 172.21.15.98:0/1113722929 shutdown_connections 2024-05-09T09:00:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc424d72740 1 --2- 172.21.15.98:0/1113722929 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x562f6be64d80 0x562f6be671a0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc424d72740 1 --2- 172.21.15.98:0/1113722929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562f6bde5320 0x562f6bde9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc424d72740 1 --2- 172.21.15.98:0/1113722929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f6bddac50 0x562f6bde93e0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc3faffd640 0 client.18633 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc424d72740 1 --2- 172.21.15.98:0/1113722929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f6bdda330 0x562f6bde8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc424d72740 1 -- 172.21.15.98:0/1113722929 >> 172.21.15.98:0/1113722929 conn(0x562f6bdd6650 msgr2=0x562f6bde2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc424d72740 1 -- 172.21.15.98:0/1113722929 shutdown_connections 2024-05-09T09:00:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc424d72740 1 -- 172.21.15.98:0/1113722929 wait complete. 2024-05-09T09:00:55.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc424d72740 20 client.18633 trim_cache size 0 max 16384 2024-05-09T09:00:55.512 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:00:55.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc421f89640 1 -- 172.21.15.98:0/3274233201 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc4080a2d00 msgr2=0x7fc4080a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc421f89640 1 --2- 172.21.15.98:0/3274233201 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc4080a2d00 0x7fc4080a51c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc41400bfd0 tx=0x7fc414004a40 comp rx=0 tx=0).stop 2024-05-09T09:00:55.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc421f89640 1 -- 172.21.15.98:0/3274233201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc41c101800 msgr2=0x7fc41c073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:55.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.506+0000 7fc421f89640 1 --2- 172.21.15.98:0/3274233201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc41c101800 0x7fc41c073b30 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009a50 tx=0x7fc40c004270 comp rx=0 tx=0).stop 2024-05-09T09:00:55.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.507+0000 7fc421f89640 1 -- 172.21.15.98:0/3274233201 shutdown_connections 2024-05-09T09:00:55.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.507+0000 7fc421f89640 1 --2- 172.21.15.98:0/3274233201 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc4080a2d00 0x7fc4080a51c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.507+0000 7fc421f89640 1 --2- 172.21.15.98:0/3274233201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc41c107810 0x7fc41c0745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.507+0000 7fc421f89640 1 --2- 172.21.15.98:0/3274233201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc41c1021c0 0x7fc41c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.507+0000 7fc421f89640 1 --2- 172.21.15.98:0/3274233201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc41c101800 0x7fc41c073b30 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:55.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.507+0000 7fc421f89640 1 -- 172.21.15.98:0/3274233201 >> 172.21.15.98:0/3274233201 conn(0x7fc41c078c80 msgr2=0x7fc41c0ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:55.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.507+0000 7fc421f89640 1 -- 172.21.15.98:0/3274233201 shutdown_connections 2024-05-09T09:00:55.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:55.507+0000 7fc421f89640 1 -- 172.21.15.98:0/3274233201 wait complete. 2024-05-09T09:00:55.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:55 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2346178472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:00:55.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:55 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2346178472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:00:56.518 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T09:00:56.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:00:56.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T09:00:56.545 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:56 smithi098 ceph-mon[30092]: pgmap v321: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:56.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.637+0000 7f092e6f5640 1 -- 172.21.15.98:0/337694980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0928106b90 msgr2=0x7f092810e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.637+0000 7f092e6f5640 1 --2- 172.21.15.98:0/337694980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0928106b90 0x7f092810e7a0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f0920009ca0 tx=0x7f092002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.637+0000 7f092e6f5640 1 -- 172.21.15.98:0/337694980 shutdown_connections 2024-05-09T09:00:56.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.637+0000 7f092e6f5640 1 --2- 172.21.15.98:0/337694980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0928106b90 0x7f092810e7a0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.637+0000 7f092e6f5640 1 --2- 172.21.15.98:0/337694980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0928100d80 0x7f09281064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.637+0000 7f092e6f5640 1 --2- 172.21.15.98:0/337694980 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09281003d0 0x7f09281007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.637+0000 7f092e6f5640 1 -- 172.21.15.98:0/337694980 >> 172.21.15.98:0/337694980 conn(0x7f09280759c0 msgr2=0x7f0928075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:56.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.637+0000 7f092e6f5640 1 -- 172.21.15.98:0/337694980 shutdown_connections 2024-05-09T09:00:56.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.637+0000 7f092e6f5640 1 -- 172.21.15.98:0/337694980 wait complete. 2024-05-09T09:00:56.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.638+0000 7f092e6f5640 1 Processor -- start 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.638+0000 7f092e6f5640 1 -- start start 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.638+0000 7f092e6f5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09281003d0 0x7f0928101340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.638+0000 7f092e6f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0928106b90 0x7f0928101880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.638+0000 7f092e6f5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0928101dc0 0x7f09281a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.638+0000 7f092e6f5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0928111c80 con 0x7f0928106b90 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.638+0000 7f092e6f5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0928111b00 con 0x7f0928101dc0 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.638+0000 7f092e6f5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0928111e00 con 0x7f09281003d0 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.638+0000 7f09277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0928106b90 0x7f0928101880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.638+0000 7f09277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0928106b90 0x7f0928101880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59658/0 (socket says 172.21.15.98:59658) 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.638+0000 7f09277fe640 1 -- 172.21.15.98:0/1787705984 learned_addr learned my addr 172.21.15.98:0/1787705984 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.638+0000 7f092cc92640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0928101dc0 0x7f09281a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.639+0000 7f09277fe640 1 -- 172.21.15.98:0/1787705984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09281003d0 msgr2=0x7f0928101340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.639+0000 7f0927fff640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09281003d0 0x7f0928101340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.639+0000 7f09277fe640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09281003d0 0x7f0928101340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.639+0000 7f09277fe640 1 -- 172.21.15.98:0/1787705984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0928101dc0 msgr2=0x7f09281a6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.639+0000 7f09277fe640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0928101dc0 0x7f09281a6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.639+0000 7f09277fe640 1 -- 172.21.15.98:0/1787705984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09281a6a10 con 0x7f0928106b90 2024-05-09T09:00:56.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.639+0000 7f092cc92640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0928101dc0 0x7f09281a6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:56.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.639+0000 7f0927fff640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09281003d0 0x7f0928101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:56.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.639+0000 7f09277fe640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0928106b90 0x7f0928101880 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f091c00d770 tx=0x7f091c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:56.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.639+0000 7f09257fa640 1 -- 172.21.15.98:0/1787705984 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c017070 con 0x7f0928106b90 2024-05-09T09:00:56.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.639+0000 7f092e6f5640 1 -- 172.21.15.98:0/1787705984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09281a6d00 con 0x7f0928106b90 2024-05-09T09:00:56.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.639+0000 7f09257fa640 1 -- 172.21.15.98:0/1787705984 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f091c009c30 con 0x7f0928106b90 2024-05-09T09:00:56.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.639+0000 7f092e6f5640 1 -- 172.21.15.98:0/1787705984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09281a7240 con 0x7f0928106b90 2024-05-09T09:00:56.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.639+0000 7f09257fa640 1 -- 172.21.15.98:0/1787705984 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c021060 con 0x7f0928106b90 2024-05-09T09:00:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.642+0000 7f09257fa640 1 -- 172.21.15.98:0/1787705984 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f091c01a070 con 0x7f0928106b90 2024-05-09T09:00:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.642+0000 7f0931403740 1 Processor -- start 2024-05-09T09:00:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.642+0000 7f0931403740 1 -- start start 2024-05-09T09:00:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.642+0000 7f0931403740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19e33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.642+0000 7f0931403740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 0x5610b19e3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.642+0000 7f0931403740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 0x5610b19f0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.642+0000 7f0931403740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5610b1948990 con 0x5610b19e50b0 2024-05-09T09:00:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.642+0000 7f0931403740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5610b18f05b0 con 0x5610b19ee740 2024-05-09T09:00:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.642+0000 7f0931403740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5610b1944950 con 0x5610b19e39c0 2024-05-09T09:00:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.642+0000 7f092cc92640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19e33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.642+0000 7f092cc92640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19e33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59660/0 (socket says 172.21.15.98:59660) 2024-05-09T09:00:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.642+0000 7f092cc92640 1 -- 172.21.15.98:0/896874175 learned_addr learned my addr 172.21.15.98:0/896874175 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f09277fe640 1 --2- 172.21.15.98:0/896874175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 0x5610b19f0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f0927fff640 1 --2- 172.21.15.98:0/896874175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 0x5610b19e3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f092cc92640 1 -- 172.21.15.98:0/896874175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 msgr2=0x5610b19e3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f092cc92640 1 --2- 172.21.15.98:0/896874175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 0x5610b19e3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f092cc92640 1 -- 172.21.15.98:0/896874175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 msgr2=0x5610b19f0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f09257fa640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09100a2d30 0x7f09100a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f092cc92640 1 --2- 172.21.15.98:0/896874175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 0x5610b19f0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f092cc92640 1 -- 172.21.15.98:0/896874175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610b17247e0 con 0x5610b19e50b0 2024-05-09T09:00:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f0927fff640 1 --2- 172.21.15.98:0/896874175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 0x5610b19e3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f09257fa640 1 -- 172.21.15.98:0/1787705984 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f091c0cf700 con 0x7f0928106b90 2024-05-09T09:00:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f092cc92640 1 --2- 172.21.15.98:0/896874175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19e33f0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f0920005f60 tx=0x7f0920005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd42def9ed982c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f092cc92640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09100a2d30 0x7f09100a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f090e7fc640 1 -- 172.21.15.98:0/896874175 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092004b070 con 0x5610b19e50b0 2024-05-09T09:00:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f090e7fc640 1 -- 172.21.15.98:0/896874175 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0920036070 con 0x5610b19e50b0 2024-05-09T09:00:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.643+0000 7f090e7fc640 1 -- 172.21.15.98:0/896874175 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920004840 con 0x5610b19e50b0 2024-05-09T09:00:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.644+0000 7f0931403740 1 -- 172.21.15.98:0/896874175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 msgr2=0x5610b19e33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.644+0000 7f092cc92640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09100a2d30 0x7f09100a51f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0920048000 tx=0x7f0920049000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.644+0000 7f0931403740 1 --2- 172.21.15.98:0/896874175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19e33f0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f0920005f60 tx=0x7f0920005f90 comp rx=0 tx=0).stop 2024-05-09T09:00:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.644+0000 7f0931403740 1 -- 172.21.15.98:0/896874175 shutdown_connections 2024-05-09T09:00:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.644+0000 7f0931403740 1 --2- 172.21.15.98:0/896874175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 0x5610b19f0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.644+0000 7f0931403740 1 --2- 172.21.15.98:0/896874175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 0x5610b19e3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.644+0000 7f0931403740 1 --2- 172.21.15.98:0/896874175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19e33f0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.644+0000 7f0931403740 1 -- 172.21.15.98:0/896874175 >> 172.21.15.98:0/896874175 conn(0x5610b19ac0a0 msgr2=0x5610b19dff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.644+0000 7f0931403740 1 -- 172.21.15.98:0/896874175 shutdown_connections 2024-05-09T09:00:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.644+0000 7f0931403740 1 -- 172.21.15.98:0/896874175 wait complete. 2024-05-09T09:00:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.644+0000 7f0931403740 1 -- start start 2024-05-09T09:00:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f0931403740 1 Processor -- start 2024-05-09T09:00:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f0931403740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 0x5610b19eb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f0931403740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19ebb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f0931403740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 0x5610b19ec070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f0931403740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5610b19f1450 con 0x5610b19e50b0 2024-05-09T09:00:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f0931403740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5610b1903b40 con 0x5610b19ee740 2024-05-09T09:00:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f0931403740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5610b19f15d0 con 0x5610b19e39c0 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f0931403740 10 client.0 _ll_register_callbacks cb 0x5610b18f80f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f09277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19ebb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f09277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19ebb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59666/0 (socket says 172.21.15.98:59666) 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f09277fe640 1 -- 172.21.15.98:0/458862060 learned_addr learned my addr 172.21.15.98:0/458862060 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f0927fff640 1 --2- 172.21.15.98:0/458862060 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 0x5610b19eb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f09277fe640 1 -- 172.21.15.98:0/458862060 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 msgr2=0x5610b19eb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f09277fe640 1 --2- 172.21.15.98:0/458862060 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 0x5610b19eb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f09277fe640 1 -- 172.21.15.98:0/458862060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 msgr2=0x5610b19ec070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f09277fe640 1 --2- 172.21.15.98:0/458862060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 0x5610b19ec070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f09277fe640 1 -- 172.21.15.98:0/458862060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610b1784f70 con 0x5610b19e50b0 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f0927fff640 1 --2- 172.21.15.98:0/458862060 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 0x5610b19eb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f09277fe640 1 --2- 172.21.15.98:0/458862060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19ebb30 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f091c009610 tx=0x7f091c09b170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998c058c8952e928 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f09077fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f09077fe640 1 -- 172.21.15.98:0/458862060 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c09b810 con 0x5610b19e50b0 2024-05-09T09:00:56.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f09077fe640 1 -- 172.21.15.98:0/458862060 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f091c097e80 con 0x5610b19e50b0 2024-05-09T09:00:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.645+0000 7f0931403740 1 -- 172.21.15.98:0/458862060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0920036070 con 0x5610b19e50b0 2024-05-09T09:00:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.646+0000 7f09077fe640 1 -- 172.21.15.98:0/458862060 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f091c09fbc0 con 0x5610b19e50b0 2024-05-09T09:00:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.646+0000 7f0931403740 10 client.18645 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.646+0000 7f0931403740 1 -- 172.21.15.98:0/458862060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5610b161c170 con 0x5610b19e50b0 2024-05-09T09:00:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.646+0000 7f09077fe640 1 -- 172.21.15.98:0/458862060 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0920036070 con 0x5610b19e50b0 2024-05-09T09:00:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.646+0000 7f09077fe640 1 -- 172.21.15.98:0/458862060 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f091c0cfbb0 con 0x5610b19e50b0 2024-05-09T09:00:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.646+0000 7f0931403740 10 client.18645 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.646+0000 7f0931403740 10 client.18645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.646+0000 7f0931403740 20 client.18645 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.646+0000 7f0931403740 1 --2- 172.21.15.98:0/458862060 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5610b19fe500 0x5610b1a00950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.646+0000 7f0931403740 4 client.18645 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.646+0000 7f0931403740 1 -- 172.21.15.98:0/458862060 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610b19560c0 con 0x5610b19fe500 2024-05-09T09:00:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.646+0000 7f0927fff640 1 --2- 172.21.15.98:0/458862060 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5610b19fe500 0x5610b1a00950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.647+0000 7f0927fff640 1 --2- 172.21.15.98:0/458862060 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5610b19fe500 0x5610b1a00950 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34be18ee85b5f9d7 server_cookie=cee2d1d0def1fe1e in_seq=0 out_seq=0 2024-05-09T09:00:56.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.647+0000 7f09077fe640 10 client.18645 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:56.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.647+0000 7f09077fe640 1 -- 172.21.15.98:0/458862060 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5610b19560c0 con 0x5610b19fe500 2024-05-09T09:00:56.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.647+0000 7f09077fe640 10 client.18645 handle_command_reply: tid=0 2024-05-09T09:00:56.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.647+0000 7f0931403740 1 client.18645 shutdown 2024-05-09T09:00:56.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.647+0000 7f0931403740 1 -- 172.21.15.98:0/458862060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 msgr2=0x5610b19ebb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.647+0000 7f0931403740 1 --2- 172.21.15.98:0/458862060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19ebb30 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f091c009610 tx=0x7f091c09b170 comp rx=0 tx=0).stop 2024-05-09T09:00:56.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.648+0000 7f0931403740 1 -- 172.21.15.98:0/458862060 shutdown_connections 2024-05-09T09:00:56.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.648+0000 7f0931403740 1 --2- 172.21.15.98:0/458862060 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5610b19fe500 0x5610b1a00950 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.648+0000 7f0931403740 1 --2- 172.21.15.98:0/458862060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 0x5610b19ec070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.648+0000 7f0931403740 1 --2- 172.21.15.98:0/458862060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19ebb30 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.648+0000 7f09077fe640 0 client.18645 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:56.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.648+0000 7f0931403740 1 --2- 172.21.15.98:0/458862060 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 0x5610b19eb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.648+0000 7f0931403740 1 -- 172.21.15.98:0/458862060 >> 172.21.15.98:0/458862060 conn(0x5610b19ac0a0 msgr2=0x5610b19e5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:56.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.648+0000 7f0931403740 1 -- 172.21.15.98:0/458862060 shutdown_connections 2024-05-09T09:00:56.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.648+0000 7f0931403740 1 -- 172.21.15.98:0/458862060 wait complete. 2024-05-09T09:00:56.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.648+0000 7f0931403740 20 client.18645 trim_cache size 0 max 16384 2024-05-09T09:00:56.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:56 smithi142 ceph-mon[34625]: pgmap v321: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:56.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.665+0000 7f0931403740 1 Processor -- start 2024-05-09T09:00:56.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.665+0000 7f0931403740 1 -- start start 2024-05-09T09:00:56.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.665+0000 7f0931403740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 0x5610b19fa350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.665+0000 7f0931403740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19def00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.665+0000 7f0931403740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 0x5610b1a06220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.665+0000 7f0931403740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09281b0fc0 con 0x5610b19e50b0 2024-05-09T09:00:56.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.665+0000 7f0931403740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f09281b2b00 con 0x5610b19ee740 2024-05-09T09:00:56.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.665+0000 7f0931403740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f09281b1690 con 0x5610b19e39c0 2024-05-09T09:00:56.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.666+0000 7f09277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19def00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.666+0000 7f09277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19def00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59682/0 (socket says 172.21.15.98:59682) 2024-05-09T09:00:56.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.666+0000 7f09277fe640 1 -- 172.21.15.98:0/811143501 learned_addr learned my addr 172.21.15.98:0/811143501 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:56.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.666+0000 7f09277fe640 1 -- 172.21.15.98:0/811143501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 msgr2=0x5610b19fa350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.666+0000 7f09277fe640 1 --2- 172.21.15.98:0/811143501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 0x5610b19fa350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.666+0000 7f09277fe640 1 -- 172.21.15.98:0/811143501 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 msgr2=0x5610b1a06220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.666+0000 7f092cc92640 1 --2- 172.21.15.98:0/811143501 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 0x5610b1a06220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.666+0000 7f09277fe640 1 --2- 172.21.15.98:0/811143501 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 0x5610b1a06220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.666+0000 7f09277fe640 1 -- 172.21.15.98:0/811143501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610b19ea5a0 con 0x5610b19e50b0 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.666+0000 7f092cc92640 1 --2- 172.21.15.98:0/811143501 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 0x5610b1a06220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.666+0000 7f09277fe640 1 --2- 172.21.15.98:0/811143501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19def00 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f091c09b7a0 tx=0x7f091c09b2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4cd3b0b28a3be7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.666+0000 7f090dffb640 1 -- 172.21.15.98:0/811143501 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c0b4070 con 0x5610b19e50b0 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.666+0000 7f090dffb640 1 -- 172.21.15.98:0/811143501 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f091c0a4480 con 0x5610b19e50b0 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.666+0000 7f090dffb640 1 -- 172.21.15.98:0/811143501 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c015030 con 0x5610b19e50b0 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 -- 172.21.15.98:0/811143501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 msgr2=0x5610b19def00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 --2- 172.21.15.98:0/811143501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19def00 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f091c09b7a0 tx=0x7f091c09b2a0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 -- 172.21.15.98:0/811143501 shutdown_connections 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 --2- 172.21.15.98:0/811143501 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19ee740 0x5610b1a06220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 --2- 172.21.15.98:0/811143501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e50b0 0x5610b19def00 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 --2- 172.21.15.98:0/811143501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19e39c0 0x5610b19fa350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 -- 172.21.15.98:0/811143501 >> 172.21.15.98:0/811143501 conn(0x5610b19ac0a0 msgr2=0x5610b19de7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 -- 172.21.15.98:0/811143501 shutdown_connections 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 -- 172.21.15.98:0/811143501 wait complete. 2024-05-09T09:00:56.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 -- start start 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 Processor -- start 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e39c0 0x5610b19f6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19e50b0 0x5610b19f6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19ee740 0x5610b19f39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09281b2430 con 0x5610b19e39c0 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f09281b1d60 con 0x5610b19e50b0 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f09281aefc0 con 0x5610b19ee740 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.667+0000 7f0931403740 10 client.0 _ll_register_callbacks cb 0x7f091c09b5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f0927fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e39c0 0x5610b19f6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f0927fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e39c0 0x5610b19f6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59684/0 (socket says 172.21.15.98:59684) 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f0927fff640 1 -- 172.21.15.98:0/4049897163 learned_addr learned my addr 172.21.15.98:0/4049897163 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f09277fe640 1 --2- 172.21.15.98:0/4049897163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19e50b0 0x5610b19f6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f0927fff640 1 -- 172.21.15.98:0/4049897163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19ee740 msgr2=0x5610b19f39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f0927fff640 1 --2- 172.21.15.98:0/4049897163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19ee740 0x5610b19f39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f0927fff640 1 -- 172.21.15.98:0/4049897163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19e50b0 msgr2=0x5610b19f6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f0927fff640 1 --2- 172.21.15.98:0/4049897163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19e50b0 0x5610b19f6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f0927fff640 1 -- 172.21.15.98:0/4049897163 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610b19e0130 con 0x5610b19e39c0 2024-05-09T09:00:56.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f0927fff640 1 --2- 172.21.15.98:0/4049897163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e39c0 0x5610b19f6860 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x5610b1a01190 tx=0x7f0918012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e8d6987921d2a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:56.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f09077fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:56.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f09077fe640 1 -- 172.21.15.98:0/4049897163 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0918004030 con 0x5610b19e39c0 2024-05-09T09:00:56.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f09077fe640 1 -- 172.21.15.98:0/4049897163 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f09180041d0 con 0x5610b19e39c0 2024-05-09T09:00:56.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f0931403740 1 -- 172.21.15.98:0/4049897163 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f091c0a4480 con 0x5610b19e39c0 2024-05-09T09:00:56.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f09077fe640 1 -- 172.21.15.98:0/4049897163 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f091802a050 con 0x5610b19e39c0 2024-05-09T09:00:56.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f0931403740 10 client.18651 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:56.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f0931403740 1 -- 172.21.15.98:0/4049897163 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5610b1a65510 con 0x5610b19e39c0 2024-05-09T09:00:56.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.668+0000 7f09077fe640 1 -- 172.21.15.98:0/4049897163 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f091c0a4480 con 0x5610b19e39c0 2024-05-09T09:00:56.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.669+0000 7f09077fe640 1 -- 172.21.15.98:0/4049897163 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f0918003860 con 0x5610b19e39c0 2024-05-09T09:00:56.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.669+0000 7f0931403740 10 client.18651 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.669+0000 7f0931403740 10 client.18651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.669+0000 7f0931403740 20 client.18651 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.669+0000 7f0931403740 1 --2- 172.21.15.98:0/4049897163 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5610b1a65ce0 0x5610b1a68100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.669+0000 7f0931403740 4 client.18651 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T09:00:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.669+0000 7f0931403740 1 -- 172.21.15.98:0/4049897163 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5610b1950410 con 0x5610b1a65ce0 2024-05-09T09:00:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.669+0000 7f09277fe640 1 --2- 172.21.15.98:0/4049897163 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5610b1a65ce0 0x5610b1a68100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.669+0000 7f09077fe640 10 client.18651 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.669+0000 7f09277fe640 1 --2- 172.21.15.98:0/4049897163 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5610b1a65ce0 0x5610b1a68100 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fab52e472f197bf server_cookie=1f8ca36e16492b75 in_seq=0 out_seq=0 2024-05-09T09:00:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.670+0000 7f09077fe640 1 -- 172.21.15.98:0/4049897163 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5610b1950410 con 0x5610b1a65ce0 2024-05-09T09:00:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.670+0000 7f09077fe640 10 client.18651 handle_command_reply: tid=0 2024-05-09T09:00:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.670+0000 7f0931403740 1 client.18651 shutdown 2024-05-09T09:00:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f0931403740 1 -- 172.21.15.98:0/4049897163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e39c0 msgr2=0x5610b19f6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f0931403740 1 --2- 172.21.15.98:0/4049897163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e39c0 0x5610b19f6860 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x5610b1a01190 tx=0x7f0918012820 comp rx=0 tx=0).stop 2024-05-09T09:00:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f0931403740 1 -- 172.21.15.98:0/4049897163 shutdown_connections 2024-05-09T09:00:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f0931403740 1 --2- 172.21.15.98:0/4049897163 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5610b1a65ce0 0x5610b1a68100 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f0931403740 1 --2- 172.21.15.98:0/4049897163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5610b19ee740 0x5610b19f39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f0931403740 1 --2- 172.21.15.98:0/4049897163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5610b19e50b0 0x5610b19f6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f0931403740 1 --2- 172.21.15.98:0/4049897163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610b19e39c0 0x5610b19f6860 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f0931403740 1 -- 172.21.15.98:0/4049897163 >> 172.21.15.98:0/4049897163 conn(0x5610b19ac0a0 msgr2=0x5610b19df080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f09077fe640 0 client.18651 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f0931403740 1 -- 172.21.15.98:0/4049897163 shutdown_connections 2024-05-09T09:00:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f0931403740 1 -- 172.21.15.98:0/4049897163 wait complete. 2024-05-09T09:00:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f0931403740 20 client.18651 trim_cache size 0 max 16384 2024-05-09T09:00:56.676 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:00:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T09:00:56.676 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "a449e44c-e424-4bf1-8b9c-315b48196fde", 2024-05-09T09:00:56.677 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T09:00:56.677 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:00:56.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f092e6f5640 1 -- 172.21.15.98:0/1787705984 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09100a2d30 msgr2=0x7f09100a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f092e6f5640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09100a2d30 0x7f09100a51f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0920048000 tx=0x7f0920049000 comp rx=0 tx=0).stop 2024-05-09T09:00:56.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f092e6f5640 1 -- 172.21.15.98:0/1787705984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0928106b90 msgr2=0x7f0928101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.671+0000 7f092e6f5640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0928106b90 0x7f0928101880 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f091c00d770 tx=0x7f091c00dc30 comp rx=0 tx=0).stop 2024-05-09T09:00:56.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.672+0000 7f092e6f5640 1 -- 172.21.15.98:0/1787705984 shutdown_connections 2024-05-09T09:00:56.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.672+0000 7f092e6f5640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09100a2d30 0x7f09100a51f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.672+0000 7f092e6f5640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0928101dc0 0x7f09281a6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.672+0000 7f092e6f5640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0928106b90 0x7f0928101880 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.672+0000 7f092e6f5640 1 --2- 172.21.15.98:0/1787705984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09281003d0 0x7f0928101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.672+0000 7f092e6f5640 1 -- 172.21.15.98:0/1787705984 >> 172.21.15.98:0/1787705984 conn(0x7f09280759c0 msgr2=0x7f09280fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:56.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.672+0000 7f092e6f5640 1 -- 172.21.15.98:0/1787705984 shutdown_connections 2024-05-09T09:00:56.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.672+0000 7f092e6f5640 1 -- 172.21.15.98:0/1787705984 wait complete. 2024-05-09T09:00:56.682 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:00:56.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:00:56.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:56 smithi105 ceph-mon[34498]: pgmap v321: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:00:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.826+0000 7f359fdc8640 1 -- 172.21.15.98:0/634996451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598102ac0 msgr2=0x7f3598102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.826+0000 7f359fdc8640 1 --2- 172.21.15.98:0/634996451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598102ac0 0x7f3598102ea0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f3594009900 tx=0x7f359402a510 comp rx=0 tx=0).stop 2024-05-09T09:00:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.826+0000 7f359fdc8640 1 -- 172.21.15.98:0/634996451 shutdown_connections 2024-05-09T09:00:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.826+0000 7f359fdc8640 1 --2- 172.21.15.98:0/634996451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359810be00 0x7f359810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.826+0000 7f359fdc8640 1 --2- 172.21.15.98:0/634996451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35981033e0 0x7f359810b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.826+0000 7f359fdc8640 1 --2- 172.21.15.98:0/634996451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598102ac0 0x7f3598102ea0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.826+0000 7f359fdc8640 1 -- 172.21.15.98:0/634996451 >> 172.21.15.98:0/634996451 conn(0x7f3598076ab0 msgr2=0x7f3598076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.826+0000 7f359fdc8640 1 -- 172.21.15.98:0/634996451 shutdown_connections 2024-05-09T09:00:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.826+0000 7f359fdc8640 1 -- 172.21.15.98:0/634996451 wait complete. 2024-05-09T09:00:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359fdc8640 1 Processor -- start 2024-05-09T09:00:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359fdc8640 1 -- start start 2024-05-09T09:00:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359fdc8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598102ac0 0x7f3598198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359fdc8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35981033e0 0x7f3598198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359fdc8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359810be00 0x7f359819c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359fdc8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3598111c00 con 0x7f3598102ac0 2024-05-09T09:00:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359fdc8640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3598111a80 con 0x7f35981033e0 2024-05-09T09:00:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359fdc8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3598111d80 con 0x7f359810be00 2024-05-09T09:00:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359db64640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598102ac0 0x7f3598198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359db64640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598102ac0 0x7f3598198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59706/0 (socket says 172.21.15.98:59706) 2024-05-09T09:00:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359db64640 1 -- 172.21.15.98:0/2421692499 learned_addr learned my addr 172.21.15.98:0/2421692499 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359d363640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35981033e0 0x7f3598198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359e365640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359810be00 0x7f359819c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359db64640 1 -- 172.21.15.98:0/2421692499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359810be00 msgr2=0x7f359819c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359db64640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359810be00 0x7f359819c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359db64640 1 -- 172.21.15.98:0/2421692499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35981033e0 msgr2=0x7f3598198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359db64640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35981033e0 0x7f3598198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359db64640 1 -- 172.21.15.98:0/2421692499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359819d0f0 con 0x7f3598102ac0 2024-05-09T09:00:56.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.827+0000 7f359e365640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359810be00 0x7f359819c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:56.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.828+0000 7f359db64640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598102ac0 0x7f3598198100 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f3594006010 tx=0x7f35940042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:56.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.828+0000 7f358effd640 1 -- 172.21.15.98:0/2421692499 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359403f030 con 0x7f3598102ac0 2024-05-09T09:00:56.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.828+0000 7f358effd640 1 -- 172.21.15.98:0/2421692499 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3594003ce0 con 0x7f3598102ac0 2024-05-09T09:00:56.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.828+0000 7f359fdc8640 1 -- 172.21.15.98:0/2421692499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f359819d380 con 0x7f3598102ac0 2024-05-09T09:00:56.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.828+0000 7f358effd640 1 -- 172.21.15.98:0/2421692499 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594035070 con 0x7f3598102ac0 2024-05-09T09:00:56.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.828+0000 7f359fdc8640 1 -- 172.21.15.98:0/2421692499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35981a4c70 con 0x7f3598102ac0 2024-05-09T09:00:56.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.830+0000 7f358effd640 1 -- 172.21.15.98:0/2421692499 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3594042030 con 0x7f3598102ac0 2024-05-09T09:00:56.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.830+0000 7f35a2bb1740 1 Processor -- start 2024-05-09T09:00:56.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f35a2bb1740 1 -- start start 2024-05-09T09:00:56.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f35a2bb1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937d160 0x55fec937d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f35a2bb1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937da80 0x55fec938c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f35a2bb1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec938c680 0x55fec938ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f35a2bb1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fec92e2d00 con 0x55fec937da80 2024-05-09T09:00:56.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f35a2bb1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fec928a4c0 con 0x55fec937d160 2024-05-09T09:00:56.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f35a2bb1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fec92decc0 con 0x55fec938c680 2024-05-09T09:00:56.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f358effd640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f35840a2d80 0x7f35840a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f359db64640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937da80 0x55fec938c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f359db64640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937da80 0x55fec938c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59708/0 (socket says 172.21.15.98:59708) 2024-05-09T09:00:56.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f359db64640 1 -- 172.21.15.98:0/835904912 learned_addr learned my addr 172.21.15.98:0/835904912 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:56.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f359e365640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f35840a2d80 0x7f35840a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f359d363640 1 --2- 172.21.15.98:0/835904912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec938c680 0x55fec938ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f359e365640 1 --2- 172.21.15.98:0/835904912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937d160 0x55fec937d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f358effd640 1 -- 172.21.15.98:0/2421692499 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f35940efd20 con 0x7f3598102ac0 2024-05-09T09:00:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f359db64640 1 -- 172.21.15.98:0/835904912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec938c680 msgr2=0x55fec938ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.831+0000 7f359db64640 1 --2- 172.21.15.98:0/835904912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec938c680 0x55fec938ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.832+0000 7f359db64640 1 -- 172.21.15.98:0/835904912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937d160 msgr2=0x55fec937d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.832+0000 7f359db64640 1 --2- 172.21.15.98:0/835904912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937d160 0x55fec937d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.832+0000 7f359db64640 1 -- 172.21.15.98:0/835904912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fec90c07d0 con 0x55fec937da80 2024-05-09T09:00:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.832+0000 7f359d363640 1 --2- 172.21.15.98:0/835904912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec938c680 0x55fec938ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.832+0000 7f359e365640 1 --2- 172.21.15.98:0/835904912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937d160 0x55fec937d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:00:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.832+0000 7f359db64640 1 --2- 172.21.15.98:0/835904912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937da80 0x55fec938c140 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f35940c4d20 tx=0x7f35940bbb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a159fe28b114e3e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.832+0000 7f359e365640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f35840a2d80 0x7f35840a5240 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3590005fd0 tx=0x7f359000a2f0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.832+0000 7f3583fff640 1 -- 172.21.15.98:0/835904912 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594049070 con 0x55fec937da80 2024-05-09T09:00:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.832+0000 7f3583fff640 1 -- 172.21.15.98:0/835904912 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3594036e80 con 0x55fec937da80 2024-05-09T09:00:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.832+0000 7f3583fff640 1 -- 172.21.15.98:0/835904912 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35940bbef0 con 0x55fec937da80 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.833+0000 7f35a2bb1740 1 -- 172.21.15.98:0/835904912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937da80 msgr2=0x55fec938c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.833+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/835904912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937da80 0x55fec938c140 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f35940c4d20 tx=0x7f35940bbb60 comp rx=0 tx=0).stop 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.833+0000 7f35a2bb1740 1 -- 172.21.15.98:0/835904912 shutdown_connections 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.833+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/835904912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec938c680 0x55fec938ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.833+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/835904912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937da80 0x55fec938c140 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.833+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/835904912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937d160 0x55fec937d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.833+0000 7f35a2bb1740 1 -- 172.21.15.98:0/835904912 >> 172.21.15.98:0/835904912 conn(0x55fec93795a0 msgr2=0x55fec9379980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.833+0000 7f35a2bb1740 1 -- 172.21.15.98:0/835904912 shutdown_connections 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.833+0000 7f35a2bb1740 1 -- 172.21.15.98:0/835904912 wait complete. 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.833+0000 7f35a2bb1740 1 -- start start 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.833+0000 7f35a2bb1740 1 Processor -- start 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35a2bb1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937d160 0x55fec9389550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35a2bb1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec937da80 0x55fec9389a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35a2bb1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec938c680 0x55fec9389fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35a2bb1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fec938f280 con 0x55fec937d160 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35a2bb1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fec929db90 con 0x55fec938c680 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35a2bb1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fec938f400 con 0x55fec937da80 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f359db64640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937d160 0x55fec9389550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35a2bb1740 10 client.0 _ll_register_callbacks cb 0x55fec92d9da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f359db64640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937d160 0x55fec9389550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59712/0 (socket says 172.21.15.98:59712) 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f359db64640 1 -- 172.21.15.98:0/4102976430 learned_addr learned my addr 172.21.15.98:0/4102976430 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f359d363640 1 --2- 172.21.15.98:0/4102976430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec937da80 0x55fec9389a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f359db64640 1 -- 172.21.15.98:0/4102976430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec937da80 msgr2=0x55fec9389a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f359db64640 1 --2- 172.21.15.98:0/4102976430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec937da80 0x55fec9389a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f359db64640 1 -- 172.21.15.98:0/4102976430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec938c680 msgr2=0x55fec9389fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f359db64640 1 --2- 172.21.15.98:0/4102976430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec938c680 0x55fec9389fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f359db64640 1 -- 172.21.15.98:0/4102976430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fec8ff63f0 con 0x55fec937d160 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f359db64640 1 --2- 172.21.15.98:0/4102976430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937d160 0x55fec9389550 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f35940ed110 tx=0x7f35940bc670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671fcb5c14428e9e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35817fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35817fa640 1 -- 172.21.15.98:0/4102976430 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594049040 con 0x55fec937d160 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35817fa640 1 -- 172.21.15.98:0/4102976430 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f35940b8560 con 0x55fec937d160 2024-05-09T09:00:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4102976430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fec92f0430 con 0x55fec937d160 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35817fa640 1 -- 172.21.15.98:0/4102976430 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f359403bcb0 con 0x55fec937d160 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35a2bb1740 10 client.18663 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4102976430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55fec8fb60f0 con 0x55fec937d160 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.834+0000 7f35817fa640 1 -- 172.21.15.98:0/4102976430 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fec92f0430 con 0x55fec937d160 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.835+0000 7f35817fa640 1 -- 172.21.15.98:0/4102976430 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f35940c46d0 con 0x55fec937d160 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.835+0000 7f35a2bb1740 10 client.18663 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.835+0000 7f35a2bb1740 10 client.18663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.835+0000 7f35a2bb1740 20 client.18663 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.835+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/4102976430 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fec939c540 0x55fec939e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.835+0000 7f35a2bb1740 4 client.18663 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.835+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4102976430 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fec92ea780 con 0x55fec939c540 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.835+0000 7f359d363640 1 --2- 172.21.15.98:0/4102976430 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fec939c540 0x55fec939e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.835+0000 7f35817fa640 10 client.18663 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.835+0000 7f359d363640 1 --2- 172.21.15.98:0/4102976430 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fec939c540 0x55fec939e930 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b121562642bb5d83 server_cookie=8834dd51ec8b46f6 in_seq=0 out_seq=0 2024-05-09T09:00:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35817fa640 1 -- 172.21.15.98:0/4102976430 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fec92ea780 con 0x55fec939c540 2024-05-09T09:00:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35817fa640 10 client.18663 handle_command_reply: tid=0 2024-05-09T09:00:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35a2bb1740 1 client.18663 shutdown 2024-05-09T09:00:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4102976430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937d160 msgr2=0x55fec9389550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/4102976430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937d160 0x55fec9389550 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f35940ed110 tx=0x7f35940bc670 comp rx=0 tx=0).stop 2024-05-09T09:00:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4102976430 shutdown_connections 2024-05-09T09:00:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/4102976430 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fec939c540 0x55fec939e930 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/4102976430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec938c680 0x55fec9389fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/4102976430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec937da80 0x55fec9389a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/4102976430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937d160 0x55fec9389550 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35817fa640 0 client.18663 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4102976430 >> 172.21.15.98:0/4102976430 conn(0x55fec93795a0 msgr2=0x55fec937f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4102976430 shutdown_connections 2024-05-09T09:00:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4102976430 wait complete. 2024-05-09T09:00:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.836+0000 7f35a2bb1740 20 client.18663 trim_cache size 0 max 16384 2024-05-09T09:00:56.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f35a2bb1740 1 Processor -- start 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f35a2bb1740 1 -- start start 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f35a2bb1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937d160 0x55fec9398090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f35a2bb1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec937da80 0x55fec93787e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f35a2bb1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec938c680 0x55fec93a40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f35a2bb1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35981acc70 con 0x55fec938c680 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f35a2bb1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f35981ae7b0 con 0x55fec937d160 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f35a2bb1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f35981ad340 con 0x55fec937da80 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f359e365640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec938c680 0x55fec93a40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f359e365640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec938c680 0x55fec93a40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59728/0 (socket says 172.21.15.98:59728) 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f359e365640 1 -- 172.21.15.98:0/2800178269 learned_addr learned my addr 172.21.15.98:0/2800178269 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f359e365640 1 -- 172.21.15.98:0/2800178269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec937da80 msgr2=0x55fec93787e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f359e365640 1 --2- 172.21.15.98:0/2800178269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec937da80 0x55fec93787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f359e365640 1 -- 172.21.15.98:0/2800178269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937d160 msgr2=0x55fec9398090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f359e365640 1 --2- 172.21.15.98:0/2800178269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937d160 0x55fec9398090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f359e365640 1 -- 172.21.15.98:0/2800178269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fec93790d0 con 0x55fec938c680 2024-05-09T09:00:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f359e365640 1 --2- 172.21.15.98:0/2800178269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec938c680 0x55fec93a40a0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f359000a5c0 tx=0x7f3590004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e57ac78029ba03d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f35837fe640 1 -- 172.21.15.98:0/2800178269 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359000d840 con 0x55fec938c680 2024-05-09T09:00:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.854+0000 7f35837fe640 1 -- 172.21.15.98:0/2800178269 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3590011470 con 0x55fec938c680 2024-05-09T09:00:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.855+0000 7f35837fe640 1 -- 172.21.15.98:0/2800178269 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359001d670 con 0x55fec938c680 2024-05-09T09:00:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.855+0000 7f35a2bb1740 1 -- 172.21.15.98:0/2800178269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec938c680 msgr2=0x55fec93a40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.855+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/2800178269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec938c680 0x55fec93a40a0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f359000a5c0 tx=0x7f3590004ab0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.855+0000 7f35a2bb1740 1 -- 172.21.15.98:0/2800178269 shutdown_connections 2024-05-09T09:00:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.855+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/2800178269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec938c680 0x55fec93a40a0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.855+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/2800178269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec937da80 0x55fec93787e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.855+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/2800178269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937d160 0x55fec9398090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.855+0000 7f35a2bb1740 1 -- 172.21.15.98:0/2800178269 >> 172.21.15.98:0/2800178269 conn(0x55fec93795a0 msgr2=0x55fec9388490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.855+0000 7f35a2bb1740 1 -- 172.21.15.98:0/2800178269 shutdown_connections 2024-05-09T09:00:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.855+0000 7f35a2bb1740 1 -- 172.21.15.98:0/2800178269 wait complete. 2024-05-09T09:00:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.855+0000 7f35a2bb1740 1 -- start start 2024-05-09T09:00:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f35a2bb1740 1 Processor -- start 2024-05-09T09:00:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f35a2bb1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937d160 0x55fec9390250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f35a2bb1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937da80 0x55fec9390790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f35a2bb1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec938c680 0x55fec9390cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f35a2bb1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35981ae0e0 con 0x55fec937d160 2024-05-09T09:00:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f35a2bb1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f35981ada10 con 0x55fec937da80 2024-05-09T09:00:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f35a2bb1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f35981aac70 con 0x55fec938c680 2024-05-09T09:00:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f35a2bb1740 10 client.0 _ll_register_callbacks cb 0x7f3594005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:00:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f359db64640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937d160 0x55fec9390250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f359db64640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937d160 0x55fec9390250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59732/0 (socket says 172.21.15.98:59732) 2024-05-09T09:00:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f359db64640 1 -- 172.21.15.98:0/4046845881 learned_addr learned my addr 172.21.15.98:0/4046845881 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:00:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f359db64640 1 -- 172.21.15.98:0/4046845881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec938c680 msgr2=0x55fec9390cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f359e365640 1 --2- 172.21.15.98:0/4046845881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec938c680 0x55fec9390cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f359d363640 1 --2- 172.21.15.98:0/4046845881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937da80 0x55fec9390790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f359db64640 1 --2- 172.21.15.98:0/4046845881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec938c680 0x55fec9390cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f359db64640 1 -- 172.21.15.98:0/4046845881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937da80 msgr2=0x55fec9390790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:00:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f359db64640 1 --2- 172.21.15.98:0/4046845881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937da80 0x55fec9390790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f359db64640 1 -- 172.21.15.98:0/4046845881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fec9401bb0 con 0x55fec937d160 2024-05-09T09:00:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f359e365640 1 --2- 172.21.15.98:0/4046845881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec938c680 0x55fec9390cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:00:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f359db64640 1 --2- 172.21.15.98:0/4046845881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937d160 0x55fec9390250 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f3594005940 tx=0x7f35940bc510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03c7dd0c5e0c152 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:00:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f35817fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:00:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.856+0000 7f35817fa640 1 -- 172.21.15.98:0/4046845881 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594049070 con 0x55fec937d160 2024-05-09T09:00:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.857+0000 7f35817fa640 1 -- 172.21.15.98:0/4046845881 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7f3594031bd0 con 0x55fec937d160 2024-05-09T09:00:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.857+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4046845881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3590011470 con 0x55fec937d160 2024-05-09T09:00:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.857+0000 7f35817fa640 1 -- 172.21.15.98:0/4046845881 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f35940bbcc0 con 0x55fec937d160 2024-05-09T09:00:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.857+0000 7f35a2bb1740 10 client.18669 fetch_fsmap learned FSMap version 37 2024-05-09T09:00:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.857+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4046845881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55fec9401bb0 con 0x55fec937d160 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.857+0000 7f35817fa640 1 -- 172.21.15.98:0/4046845881 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3590011470 con 0x55fec937d160 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.857+0000 7f35817fa640 1 -- 172.21.15.98:0/4046845881 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f35940b1080 con 0x55fec937d160 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.857+0000 7f35a2bb1740 10 client.18669 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.857+0000 7f35a2bb1740 10 client.18669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.857+0000 7f35a2bb1740 20 client.18669 populate_metadata read hostname 'smithi098' 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.857+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/4046845881 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fec9403220 0x55fec9405610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.857+0000 7f35a2bb1740 4 client.18669 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.857+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4046845881 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fec92676f0 con 0x55fec9403220 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.857+0000 7f359d363640 1 --2- 172.21.15.98:0/4046845881 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fec9403220 0x55fec9405610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f359d363640 1 --2- 172.21.15.98:0/4046845881 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fec9403220 0x55fec9405610 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=595f457f161b75fb server_cookie=a88ea8f7b9b63821 in_seq=0 out_seq=0 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35817fa640 10 client.18669 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35817fa640 1 -- 172.21.15.98:0/4046845881 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fec92676f0 con 0x55fec9403220 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35817fa640 10 client.18669 handle_command_reply: tid=0 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35a2bb1740 1 client.18669 shutdown 2024-05-09T09:00:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4046845881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937d160 msgr2=0x55fec9390250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/4046845881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937d160 0x55fec9390250 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f3594005940 tx=0x7f35940bc510 comp rx=0 tx=0).stop 2024-05-09T09:00:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4046845881 shutdown_connections 2024-05-09T09:00:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/4046845881 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55fec9403220 0x55fec9405610 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/4046845881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fec938c680 0x55fec9390cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/4046845881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fec937da80 0x55fec9390790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35817fa640 0 client.18669 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:00:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35a2bb1740 1 --2- 172.21.15.98:0/4046845881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fec937d160 0x55fec9390250 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4046845881 >> 172.21.15.98:0/4046845881 conn(0x55fec93795a0 msgr2=0x55fec9389d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4046845881 shutdown_connections 2024-05-09T09:00:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35a2bb1740 1 -- 172.21.15.98:0/4046845881 wait complete. 2024-05-09T09:00:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.858+0000 7f35a2bb1740 20 client.18669 trim_cache size 0 max 16384 2024-05-09T09:00:56.865 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:00:56.865 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T09:00:56.865 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T09:00:56.865 INFO:teuthology.orchestra.run.smithi098.stdout: "a449e44c-e424-4bf1-8b9c-315b48196fde": { 2024-05-09T09:00:56.865 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T09:00:56.865 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "a449e44c-e424-4bf1-8b9c-315b48196fde", 2024-05-09T09:00:56.865 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T09:00:56.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:00:56.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:00:56.865 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:00:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.859+0000 7f359fdc8640 1 -- 172.21.15.98:0/2421692499 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f35840a2d80 msgr2=0x7f35840a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.859+0000 7f359fdc8640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f35840a2d80 0x7f35840a5240 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3590005fd0 tx=0x7f359000a2f0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.859+0000 7f359fdc8640 1 -- 172.21.15.98:0/2421692499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598102ac0 msgr2=0x7f3598198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:00:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.859+0000 7f359fdc8640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598102ac0 0x7f3598198100 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f3594006010 tx=0x7f35940042f0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.859+0000 7f359fdc8640 1 -- 172.21.15.98:0/2421692499 shutdown_connections 2024-05-09T09:00:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.859+0000 7f359fdc8640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f35840a2d80 0x7f35840a5240 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.859+0000 7f359fdc8640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359810be00 0x7f359819c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.859+0000 7f359fdc8640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35981033e0 0x7f3598198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.859+0000 7f359fdc8640 1 --2- 172.21.15.98:0/2421692499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598102ac0 0x7f3598198100 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:00:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.859+0000 7f359fdc8640 1 -- 172.21.15.98:0/2421692499 >> 172.21.15.98:0/2421692499 conn(0x7f3598076ab0 msgr2=0x7f35980fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:00:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.859+0000 7f359fdc8640 1 -- 172.21.15.98:0/2421692499 shutdown_connections 2024-05-09T09:00:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:00:56.859+0000 7f359fdc8640 1 -- 172.21.15.98:0/2421692499 wait complete. 2024-05-09T09:00:56.869 INFO:tasks.cephfs.filesystem:scrub status for tag:a449e44c-e424-4bf1-8b9c-315b48196fde - {'path': '/', 'tag': 'a449e44c-e424-4bf1-8b9c-315b48196fde', 'options': 'recursive,force'} 2024-05-09T09:00:57.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:57 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T09:00:57.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:57 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T09:00:57.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:57 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T09:00:57.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:57 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T09:00:57.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:57 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T09:00:57.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:57 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T09:00:57.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:57 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T09:00:57.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:57 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T09:00:57.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:57 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T09:00:57.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:57 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T09:00:57.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:57 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T09:00:57.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:57 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T09:00:58.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:00:58 smithi098 ceph-mon[30092]: pgmap v322: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:00:58.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:00:58 smithi142 ceph-mon[34625]: pgmap v322: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:00:58.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:00:58 smithi105 ceph-mon[34498]: pgmap v322: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:01:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.375+0000 7ff3c9683640 1 -- 172.21.15.98:0/763487608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3c4102200 msgr2=0x7ff3c410e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.375+0000 7ff3c9683640 1 --2- 172.21.15.98:0/763487608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3c4102200 0x7ff3c410e710 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009cd0 tx=0x7ff3bc02a8e0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.375+0000 7ff3c9683640 1 -- 172.21.15.98:0/763487608 shutdown_connections 2024-05-09T09:01:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.375+0000 7ff3c9683640 1 --2- 172.21.15.98:0/763487608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3c4102200 0x7ff3c410e710 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.375+0000 7ff3c9683640 1 --2- 172.21.15.98:0/763487608 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3c4101840 0x7ff3c4101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.375+0000 7ff3c9683640 1 --2- 172.21.15.98:0/763487608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4107850 0x7ff3c4107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.375+0000 7ff3c9683640 1 -- 172.21.15.98:0/763487608 >> 172.21.15.98:0/763487608 conn(0x7ff3c4078cc0 msgr2=0x7ff3c40790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:00.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.375+0000 7ff3c9683640 1 -- 172.21.15.98:0/763487608 shutdown_connections 2024-05-09T09:01:00.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.375+0000 7ff3c9683640 1 -- 172.21.15.98:0/763487608 wait complete. 2024-05-09T09:01:00.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.376+0000 7ff3c9683640 1 Processor -- start 2024-05-09T09:01:00.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.376+0000 7ff3c9683640 1 -- start start 2024-05-09T09:01:00.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.376+0000 7ff3c9683640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4101840 0x7ff3c4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.376+0000 7ff3c9683640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3c4102200 0x7ff3c41985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.376+0000 7ff3c9683640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3c4107850 0x7ff3c419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.376+0000 7ff3c9683640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c4111c60 con 0x7ff3c4102200 2024-05-09T09:01:00.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.376+0000 7ff3c9683640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c4111ae0 con 0x7ff3c4107850 2024-05-09T09:01:00.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.376+0000 7ff3c9683640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c4111de0 con 0x7ff3c4101840 2024-05-09T09:01:00.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3ba7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3c4102200 0x7ff3c41985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3ba7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3c4102200 0x7ff3c41985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59750/0 (socket says 172.21.15.98:59750) 2024-05-09T09:01:00.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3ba7fc640 1 -- 172.21.15.98:0/599469864 learned_addr learned my addr 172.21.15.98:0/599469864 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:00.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3c2ffd640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4101840 0x7ff3c4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3ba7fc640 1 -- 172.21.15.98:0/599469864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4101840 msgr2=0x7ff3c4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:00.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3ba7fc640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4101840 0x7ff3c4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3ba7fc640 1 -- 172.21.15.98:0/599469864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3c4107850 msgr2=0x7ff3c419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:00.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3c37fe640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3c4107850 0x7ff3c419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3ba7fc640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3c4107850 0x7ff3c419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3ba7fc640 1 -- 172.21.15.98:0/599469864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c419d000 con 0x7ff3c4102200 2024-05-09T09:01:00.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3c2ffd640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4101840 0x7ff3c4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:00.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3c37fe640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3c4107850 0x7ff3c419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:00.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3ba7fc640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3c4102200 0x7ff3c41985a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac00d950 tx=0x7ff3ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:00.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3c0ff9640 1 -- 172.21.15.98:0/599469864 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac017070 con 0x7ff3c4102200 2024-05-09T09:01:00.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3c0ff9640 1 -- 172.21.15.98:0/599469864 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3ac009c30 con 0x7ff3c4102200 2024-05-09T09:01:00.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3c9683640 1 -- 172.21.15.98:0/599469864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c419d2f0 con 0x7ff3c4102200 2024-05-09T09:01:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3c0ff9640 1 -- 172.21.15.98:0/599469864 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac00f6b0 con 0x7ff3c4102200 2024-05-09T09:01:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.377+0000 7ff3c9683640 1 -- 172.21.15.98:0/599469864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c41a4b80 con 0x7ff3c4102200 2024-05-09T09:01:00.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.380+0000 7ff3c0ff9640 1 -- 172.21.15.98:0/599469864 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff3ac0109d0 con 0x7ff3c4102200 2024-05-09T09:01:00.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.380+0000 7ff3cc38b740 1 Processor -- start 2024-05-09T09:01:00.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.380+0000 7ff3cc38b740 1 -- start start 2024-05-09T09:01:00.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3cc38b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 0x561f17fd0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3cc38b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fd0d70 0x561f17fdb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3cc38b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fdb630 0x561f17fdda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3cc38b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561f17f35340 con 0x561f17fd0d70 2024-05-09T09:01:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3cc38b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x561f17edd290 con 0x561f17fdb630 2024-05-09T09:01:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3cc38b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561f17f31300 con 0x561f17fd0450 2024-05-09T09:01:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3c2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fd0d70 0x561f17fdb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3c2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fd0d70 0x561f17fdb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59754/0 (socket says 172.21.15.98:59754) 2024-05-09T09:01:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3c2ffd640 1 -- 172.21.15.98:0/3191760811 learned_addr learned my addr 172.21.15.98:0/3191760811 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3c0ff9640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff3b00a2da0 0x7ff3b00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3ba7fc640 1 --2- 172.21.15.98:0/3191760811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fdb630 0x561f17fdda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3c37fe640 1 --2- 172.21.15.98:0/3191760811 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 0x561f17fd0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3c37fe640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff3b00a2da0 0x7ff3b00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3c2ffd640 1 -- 172.21.15.98:0/3191760811 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 msgr2=0x561f17fd0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3c2ffd640 1 --2- 172.21.15.98:0/3191760811 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 0x561f17fd0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3c2ffd640 1 -- 172.21.15.98:0/3191760811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fdb630 msgr2=0x561f17fdda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3c2ffd640 1 --2- 172.21.15.98:0/3191760811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fdb630 0x561f17fdda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3c2ffd640 1 -- 172.21.15.98:0/3191760811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f17f018e0 con 0x561f17fd0d70 2024-05-09T09:01:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3c0ff9640 1 -- 172.21.15.98:0/599469864 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7ff3ac0cf560 con 0x7ff3c4102200 2024-05-09T09:01:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.381+0000 7ff3c2ffd640 1 --2- 172.21.15.98:0/3191760811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fd0d70 0x561f17fdb0f0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400baf0 tx=0x7ff3b400d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f18cc20ada14caf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.382+0000 7ff3c37fe640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff3b00a2da0 0x7ff3b00a5260 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009a60 tx=0x7ff3bc0095c0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.382+0000 7ff3b97fa640 1 -- 172.21.15.98:0/3191760811 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4017420 con 0x561f17fd0d70 2024-05-09T09:01:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.382+0000 7ff3b97fa640 1 -- 172.21.15.98:0/3191760811 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3b4017aa0 con 0x561f17fd0d70 2024-05-09T09:01:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.382+0000 7ff3b97fa640 1 -- 172.21.15.98:0/3191760811 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b400fb40 con 0x561f17fd0d70 2024-05-09T09:01:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.383+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3191760811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fd0d70 msgr2=0x561f17fdb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.383+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/3191760811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fd0d70 0x561f17fdb0f0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400baf0 tx=0x7ff3b400d340 comp rx=0 tx=0).stop 2024-05-09T09:01:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.383+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3191760811 shutdown_connections 2024-05-09T09:01:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.383+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/3191760811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fdb630 0x561f17fdda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.383+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/3191760811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fd0d70 0x561f17fdb0f0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.383+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/3191760811 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 0x561f17fd0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.383+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3191760811 >> 172.21.15.98:0/3191760811 conn(0x561f17fcc960 msgr2=0x561f17fccd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.383+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3191760811 shutdown_connections 2024-05-09T09:01:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.383+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3191760811 wait complete. 2024-05-09T09:01:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.383+0000 7ff3cc38b740 1 -- start start 2024-05-09T09:01:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.383+0000 7ff3cc38b740 1 Processor -- start 2024-05-09T09:01:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3cc38b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 0x561f17fd8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3cc38b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fd0d70 0x561f17fd8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3cc38b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fdb630 0x561f17fd8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3cc38b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561f17fde260 con 0x561f17fdb630 2024-05-09T09:01:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3cc38b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x561f17ef0790 con 0x561f17fd0d70 2024-05-09T09:01:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3cc38b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561f17fde3e0 con 0x561f17fd0450 2024-05-09T09:01:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3cc38b740 10 client.0 _ll_register_callbacks cb 0x561f17f2ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3c2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 0x561f17fd8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3c2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 0x561f17fd8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.98:39400/0 (socket says 172.21.15.98:39400) 2024-05-09T09:01:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3c37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fdb630 0x561f17fd8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3c2ffd640 1 -- 172.21.15.98:0/3025981274 learned_addr learned my addr 172.21.15.98:0/3025981274 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3ba7fc640 1 --2- 172.21.15.98:0/3025981274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fd0d70 0x561f17fd8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3c37fe640 1 -- 172.21.15.98:0/3025981274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 msgr2=0x561f17fd8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3c37fe640 1 --2- 172.21.15.98:0/3025981274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 0x561f17fd8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3c37fe640 1 -- 172.21.15.98:0/3025981274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fd0d70 msgr2=0x561f17fd8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3c37fe640 1 --2- 172.21.15.98:0/3025981274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fd0d70 0x561f17fd8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3c37fe640 1 -- 172.21.15.98:0/3025981274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f17c49210 con 0x561f17fdb630 2024-05-09T09:01:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3ba7fc640 1 --2- 172.21.15.98:0/3025981274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fd0d70 0x561f17fd8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3c2ffd640 1 --2- 172.21.15.98:0/3025981274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 0x561f17fd8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:01:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3c37fe640 1 --2- 172.21.15.98:0/3025981274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fdb630 0x561f17fd8fd0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc006010 tx=0x7ff3bc02a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59873a9e94176fbf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.384+0000 7ff3aa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.385+0000 7ff3aa7fc640 1 -- 172.21.15.98:0/3025981274 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc04d070 con 0x561f17fdb630 2024-05-09T09:01:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.385+0000 7ff3aa7fc640 1 -- 172.21.15.98:0/3025981274 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7ff3bc004790 con 0x561f17fdb630 2024-05-09T09:01:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.385+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3025981274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3b4017aa0 con 0x561f17fdb630 2024-05-09T09:01:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.385+0000 7ff3aa7fc640 1 -- 172.21.15.98:0/3025981274 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff3bc056620 con 0x561f17fdb630 2024-05-09T09:01:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.385+0000 7ff3cc38b740 10 client.18681 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.385+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3025981274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561f17d15620 con 0x561f17fdb630 2024-05-09T09:01:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.385+0000 7ff3aa7fc640 1 -- 172.21.15.98:0/3025981274 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3b4017aa0 con 0x561f17fdb630 2024-05-09T09:01:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.385+0000 7ff3aa7fc640 1 -- 172.21.15.98:0/3025981274 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7ff3bc030070 con 0x561f17fdb630 2024-05-09T09:01:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.385+0000 7ff3cc38b740 10 client.18681 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.385+0000 7ff3cc38b740 10 client.18681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.385+0000 7ff3cc38b740 20 client.18681 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.385+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/3025981274 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561f17feb2c0 0x561f17fed6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.385+0000 7ff3cc38b740 4 client.18681 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:01:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.385+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3025981274 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f17f42a70 con 0x561f17feb2c0 2024-05-09T09:01:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.386+0000 7ff3c2ffd640 1 --2- 172.21.15.98:0/3025981274 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561f17feb2c0 0x561f17fed6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.386+0000 7ff3c2ffd640 1 --2- 172.21.15.98:0/3025981274 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561f17feb2c0 0x561f17fed6b0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ad9cdb13738ad99 server_cookie=559b957d5c79cd30 in_seq=0 out_seq=0 2024-05-09T09:01:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.386+0000 7ff3aa7fc640 10 client.18681 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3aa7fc640 1 -- 172.21.15.98:0/3025981274 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561f17f42a70 con 0x561f17feb2c0 2024-05-09T09:01:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3aa7fc640 10 client.18681 handle_command_reply: tid=0 2024-05-09T09:01:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3cc38b740 1 client.18681 shutdown 2024-05-09T09:01:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3025981274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fdb630 msgr2=0x561f17fd8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/3025981274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fdb630 0x561f17fd8fd0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc006010 tx=0x7ff3bc02a8e0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3025981274 shutdown_connections 2024-05-09T09:01:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/3025981274 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561f17feb2c0 0x561f17fed6b0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3aa7fc640 0 client.18681 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/3025981274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fdb630 0x561f17fd8fd0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/3025981274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fd0d70 0x561f17fd8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/3025981274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 0x561f17fd8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3025981274 >> 172.21.15.98:0/3025981274 conn(0x561f17fcc960 msgr2=0x561f17fd2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3025981274 shutdown_connections 2024-05-09T09:01:00.397 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3025981274 wait complete. 2024-05-09T09:01:00.397 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.387+0000 7ff3cc38b740 20 client.18681 trim_cache size 0 max 16384 2024-05-09T09:01:00.403 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.404+0000 7ff3cc38b740 1 Processor -- start 2024-05-09T09:01:00.404 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.404+0000 7ff3cc38b740 1 -- start start 2024-05-09T09:01:00.405 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3cc38b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 0x561f17fdac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.405 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3cc38b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fd0d70 0x561f17fe7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3cc38b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fdb630 0x561f17fd2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3cc38b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c41aedc0 con 0x561f17fd0d70 2024-05-09T09:01:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3cc38b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c41b0900 con 0x561f17fdb630 2024-05-09T09:01:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3cc38b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c41af490 con 0x561f17fd0450 2024-05-09T09:01:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3ba7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fd0d70 0x561f17fe7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3c2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 0x561f17fdac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3c37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fdb630 0x561f17fd2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3ba7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fd0d70 0x561f17fe7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59766/0 (socket says 172.21.15.98:59766) 2024-05-09T09:01:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3ba7fc640 1 -- 172.21.15.98:0/3183137516 learned_addr learned my addr 172.21.15.98:0/3183137516 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3ba7fc640 1 -- 172.21.15.98:0/3183137516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 msgr2=0x561f17fdac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:00.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3ba7fc640 1 --2- 172.21.15.98:0/3183137516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 0x561f17fdac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3ba7fc640 1 -- 172.21.15.98:0/3183137516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fdb630 msgr2=0x561f17fd2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:00.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3ba7fc640 1 --2- 172.21.15.98:0/3183137516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fdb630 0x561f17fd2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3ba7fc640 1 -- 172.21.15.98:0/3183137516 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f17fcaf00 con 0x561f17fd0d70 2024-05-09T09:01:00.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.405+0000 7ff3c37fe640 1 --2- 172.21.15.98:0/3183137516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fdb630 0x561f17fd2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:01:00.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.406+0000 7ff3ba7fc640 1 --2- 172.21.15.98:0/3183137516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fd0d70 0x561f17fe7120 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac098040 tx=0x7ff3ac06e6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937b3571adf8ea21 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:00.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.406+0000 7ff3b8ff9640 1 -- 172.21.15.98:0/3183137516 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac020040 con 0x561f17fd0d70 2024-05-09T09:01:00.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.406+0000 7ff3b8ff9640 1 -- 172.21.15.98:0/3183137516 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3ac009de0 con 0x561f17fd0d70 2024-05-09T09:01:00.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.406+0000 7ff3b8ff9640 1 -- 172.21.15.98:0/3183137516 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac0ccb20 con 0x561f17fd0d70 2024-05-09T09:01:00.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.407+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3183137516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fd0d70 msgr2=0x561f17fe7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:00.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.407+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/3183137516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fd0d70 0x561f17fe7120 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac098040 tx=0x7ff3ac06e6e0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.407+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3183137516 shutdown_connections 2024-05-09T09:01:00.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.407+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/3183137516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fdb630 0x561f17fd2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.407+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/3183137516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fd0d70 0x561f17fe7120 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.407+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/3183137516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0450 0x561f17fdac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.407+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3183137516 >> 172.21.15.98:0/3183137516 conn(0x561f17fcc960 msgr2=0x561f17fd1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:00.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.407+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3183137516 shutdown_connections 2024-05-09T09:01:00.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.407+0000 7ff3cc38b740 1 -- 172.21.15.98:0/3183137516 wait complete. 2024-05-09T09:01:00.413 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.407+0000 7ff3cc38b740 1 -- start start 2024-05-09T09:01:00.413 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.407+0000 7ff3cc38b740 1 Processor -- start 2024-05-09T09:01:00.413 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3cc38b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fd0450 0x561f17fd5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.413 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3cc38b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0d70 0x561f17fd59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.413 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3cc38b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fdb630 0x561f17fd5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.413 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3cc38b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c41b0230 con 0x561f17fdb630 2024-05-09T09:01:00.413 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3cc38b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c41afb60 con 0x561f17fd0450 2024-05-09T09:01:00.413 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3cc38b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c41acdc0 con 0x561f17fd0d70 2024-05-09T09:01:00.413 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3cc38b740 10 client.0 _ll_register_callbacks cb 0x7ff3c406b0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:00.413 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3c37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fdb630 0x561f17fd5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.413 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3ba7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0d70 0x561f17fd59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.413 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3c37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fdb630 0x561f17fd5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59774/0 (socket says 172.21.15.98:59774) 2024-05-09T09:01:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3c37fe640 1 -- 172.21.15.98:0/73439448 learned_addr learned my addr 172.21.15.98:0/73439448 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3c2ffd640 1 --2- 172.21.15.98:0/73439448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fd0450 0x561f17fd5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3c37fe640 1 -- 172.21.15.98:0/73439448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0d70 msgr2=0x561f17fd59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3c37fe640 1 --2- 172.21.15.98:0/73439448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0d70 0x561f17fd59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3c37fe640 1 -- 172.21.15.98:0/73439448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fd0450 msgr2=0x561f17fd5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3c37fe640 1 --2- 172.21.15.98:0/73439448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fd0450 0x561f17fd5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3c37fe640 1 -- 172.21.15.98:0/73439448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f18050a30 con 0x561f17fdb630 2024-05-09T09:01:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3ba7fc640 1 --2- 172.21.15.98:0/73439448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0d70 0x561f17fd59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:01:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3c37fe640 1 --2- 172.21.15.98:0/73439448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fdb630 0x561f17fd5ef0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc006010 tx=0x7ff3bc030f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d05242a6cfe61f73 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.408+0000 7ff3aa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3aa7fc640 1 -- 172.21.15.98:0/73439448 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc063030 con 0x561f17fdb630 2024-05-09T09:01:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3aa7fc640 1 -- 172.21.15.98:0/73439448 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9562+0+0 (secure 0 0 0) 0x7ff3bc004b40 con 0x561f17fdb630 2024-05-09T09:01:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3cc38b740 1 -- 172.21.15.98:0/73439448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3ac009de0 con 0x561f17fdb630 2024-05-09T09:01:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3aa7fc640 1 -- 172.21.15.98:0/73439448 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff3bc06d050 con 0x561f17fdb630 2024-05-09T09:01:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3cc38b740 10 client.18687 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3cc38b740 1 -- 172.21.15.98:0/73439448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561f18051d00 con 0x561f17fdb630 2024-05-09T09:01:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3aa7fc640 1 -- 172.21.15.98:0/73439448 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3ac009de0 con 0x561f17fdb630 2024-05-09T09:01:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3aa7fc640 1 -- 172.21.15.98:0/73439448 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7ff3bc0309b0 con 0x561f17fdb630 2024-05-09T09:01:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3cc38b740 10 client.18687 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3cc38b740 10 client.18687 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3cc38b740 20 client.18687 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/73439448 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561f17feb2c0 0x561f18054540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3cc38b740 4 client.18687 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:01:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3cc38b740 1 -- 172.21.15.98:0/73439448 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f17f3cdc0 con 0x561f17feb2c0 2024-05-09T09:01:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.409+0000 7ff3c2ffd640 1 --2- 172.21.15.98:0/73439448 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561f17feb2c0 0x561f18054540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:00.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.410+0000 7ff3aa7fc640 10 client.18687 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:00.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.410+0000 7ff3c2ffd640 1 --2- 172.21.15.98:0/73439448 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561f17feb2c0 0x561f18054540 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fde3bb4cecc39d8 server_cookie=48276b3b074580aa in_seq=0 out_seq=0 2024-05-09T09:01:00.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.469+0000 7ff3aa7fc640 1 -- 172.21.15.98:0/73439448 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f17f3cdc0 con 0x561f17feb2c0 2024-05-09T09:01:00.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.469+0000 7ff3aa7fc640 10 client.18687 handle_command_reply: tid=0 2024-05-09T09:01:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.469+0000 7ff3cc38b740 1 client.18687 shutdown 2024-05-09T09:01:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.469+0000 7ff3cc38b740 1 -- 172.21.15.98:0/73439448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fdb630 msgr2=0x561f17fd5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.469+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/73439448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fdb630 0x561f17fd5ef0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc006010 tx=0x7ff3bc030f60 comp rx=0 tx=0).stop 2024-05-09T09:01:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.469+0000 7ff3cc38b740 1 -- 172.21.15.98:0/73439448 shutdown_connections 2024-05-09T09:01:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.469+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/73439448 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x561f17feb2c0 0x561f18054540 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.469+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/73439448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561f17fdb630 0x561f17fd5ef0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.469+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/73439448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f17fd0d70 0x561f17fd59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.469+0000 7ff3aa7fc640 0 client.18687 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.469+0000 7ff3cc38b740 1 --2- 172.21.15.98:0/73439448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561f17fd0450 0x561f17fd5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.469+0000 7ff3cc38b740 1 -- 172.21.15.98:0/73439448 >> 172.21.15.98:0/73439448 conn(0x561f17fcc960 msgr2=0x561f17fd2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.469+0000 7ff3cc38b740 1 -- 172.21.15.98:0/73439448 shutdown_connections 2024-05-09T09:01:00.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.469+0000 7ff3cc38b740 1 -- 172.21.15.98:0/73439448 wait complete. 2024-05-09T09:01:00.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.470+0000 7ff3cc38b740 20 client.18687 trim_cache size 0 max 16384 2024-05-09T09:01:00.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T09:01:00.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T09:01:00.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T09:01:00.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T09:01:00.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.470+0000 7ff3c9683640 1 -- 172.21.15.98:0/599469864 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff3b00a2da0 msgr2=0x7ff3b00a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:00.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.470+0000 7ff3c9683640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff3b00a2da0 0x7ff3b00a5260 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009a60 tx=0x7ff3bc0095c0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.470+0000 7ff3c9683640 1 -- 172.21.15.98:0/599469864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3c4102200 msgr2=0x7ff3c41985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:00.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.470+0000 7ff3c9683640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3c4102200 0x7ff3c41985a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac00d950 tx=0x7ff3ac00de10 comp rx=0 tx=0).stop 2024-05-09T09:01:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.470+0000 7ff3c9683640 1 -- 172.21.15.98:0/599469864 shutdown_connections 2024-05-09T09:01:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.470+0000 7ff3c9683640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff3b00a2da0 0x7ff3b00a5260 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.470+0000 7ff3c9683640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3c4107850 0x7ff3c419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.470+0000 7ff3c9683640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3c4102200 0x7ff3c41985a0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.470+0000 7ff3c9683640 1 --2- 172.21.15.98:0/599469864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4101840 0x7ff3c4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.470+0000 7ff3c9683640 1 -- 172.21.15.98:0/599469864 >> 172.21.15.98:0/599469864 conn(0x7ff3c4078cc0 msgr2=0x7ff3c40feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.470+0000 7ff3c9683640 1 -- 172.21.15.98:0/599469864 shutdown_connections 2024-05-09T09:01:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:00.470+0000 7ff3c9683640 1 -- 172.21.15.98:0/599469864 wait complete. 2024-05-09T09:01:00.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:00 smithi098 ceph-mon[30092]: pgmap v323: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:01:00.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:00 smithi142 ceph-mon[34625]: pgmap v323: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:01:00.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:00 smithi105 ceph-mon[34498]: pgmap v323: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:01:02.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:02 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T09:01:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:02 smithi098 ceph-mon[30092]: pgmap v324: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:01:02.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:02 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:01:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:02 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T09:01:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:02 smithi142 ceph-mon[34625]: pgmap v324: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:01:02.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:02 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:01:02.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:02 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T09:01:02.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:02 smithi105 ceph-mon[34498]: pgmap v324: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:01:02.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:02 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:01:04.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:04 smithi098 ceph-mon[30092]: pgmap v325: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-05-09T09:01:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:04 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:04 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:04 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:04.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:04 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:04 smithi142 ceph-mon[34625]: pgmap v325: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-05-09T09:01:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:04 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:04 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:04 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:04.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:04 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:04.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:04 smithi105 ceph-mon[34498]: pgmap v325: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-05-09T09:01:04.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:04 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:04.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:04 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:04 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:04.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:04 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-05-09T09:01:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: Unable to set osd_memory_target on smithi142 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-05-09T09:01:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-09T09:01:05.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-09T09:01:05.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:01:05.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:01:05.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: Unable to set osd_memory_target on smithi142 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:01:05.659 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:01:05.660 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:05.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-05-09T09:01:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: Unable to set osd_memory_target on smithi142 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-05-09T09:01:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-09T09:01:05.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-09T09:01:05.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:01:05.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:01:05.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:01:05.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:01:06.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:06 smithi142 ceph-mon[34625]: pgmap v326: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-05-09T09:01:06.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:06 smithi142 ceph-mon[34625]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-05-09T09:01:06.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:06 smithi142 ceph-mon[34625]: Unable to set osd_memory_target on smithi098 to 184766668: error parsing value: Value '184766668' is below minimum 939524096 2024-05-09T09:01:06.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:06 smithi105 ceph-mon[34498]: pgmap v326: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-05-09T09:01:06.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:06 smithi105 ceph-mon[34498]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-05-09T09:01:06.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:06 smithi105 ceph-mon[34498]: Unable to set osd_memory_target on smithi098 to 184766668: error parsing value: Value '184766668' is below minimum 939524096 2024-05-09T09:01:06.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:06 smithi098 ceph-mon[30092]: pgmap v326: 129 pgs: 129 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-05-09T09:01:06.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:06 smithi098 ceph-mon[30092]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-05-09T09:01:06.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:06 smithi098 ceph-mon[30092]: Unable to set osd_memory_target on smithi098 to 184766668: error parsing value: Value '184766668' is below minimum 939524096 2024-05-09T09:01:07.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:07 smithi142 ceph-mon[34625]: osdmap e106: 12 total, 12 up, 12 in 2024-05-09T09:01:07.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:07 smithi105 ceph-mon[34498]: osdmap e106: 12 total, 12 up, 12 in 2024-05-09T09:01:07.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:07 smithi098 ceph-mon[30092]: osdmap e106: 12 total, 12 up, 12 in 2024-05-09T09:01:08.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:08 smithi142 ceph-mon[34625]: pgmap v328: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 54 KiB/s wr, 34 op/s 2024-05-09T09:01:08.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:08 smithi105 ceph-mon[34498]: pgmap v328: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 54 KiB/s wr, 34 op/s 2024-05-09T09:01:08.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:08 smithi098 ceph-mon[30092]: pgmap v328: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 54 KiB/s wr, 34 op/s 2024-05-09T09:01:10.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:10 smithi142 ceph-mon[34625]: pgmap v329: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 54 KiB/s wr, 34 op/s 2024-05-09T09:01:10.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:10 smithi105 ceph-mon[34498]: pgmap v329: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 54 KiB/s wr, 34 op/s 2024-05-09T09:01:10.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:10 smithi098 ceph-mon[30092]: pgmap v329: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 54 KiB/s wr, 34 op/s 2024-05-09T09:01:12.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:12 smithi142 ceph-mon[34625]: pgmap v330: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 24 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-05-09T09:01:12.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:12 smithi105 ceph-mon[34498]: pgmap v330: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 24 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-05-09T09:01:12.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:12 smithi098 ceph-mon[30092]: pgmap v330: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 24 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-05-09T09:01:13.570 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.569+0000 7ffa78688640 1 -- 172.21.15.98:0/157245749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa70102200 msgr2=0x7ffa7010e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.569+0000 7ffa78688640 1 --2- 172.21.15.98:0/157245749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa70102200 0x7ffa7010e710 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7ffa68009cd0 tx=0x7ffa6802a8e0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.569+0000 7ffa78688640 1 -- 172.21.15.98:0/157245749 shutdown_connections 2024-05-09T09:01:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.569+0000 7ffa78688640 1 --2- 172.21.15.98:0/157245749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa70102200 0x7ffa7010e710 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.569+0000 7ffa78688640 1 --2- 172.21.15.98:0/157245749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa70101840 0x7ffa70101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.569+0000 7ffa78688640 1 --2- 172.21.15.98:0/157245749 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa70107850 0x7ffa70107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.569+0000 7ffa78688640 1 -- 172.21.15.98:0/157245749 >> 172.21.15.98:0/157245749 conn(0x7ffa70078cc0 msgr2=0x7ffa700790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.569+0000 7ffa78688640 1 -- 172.21.15.98:0/157245749 shutdown_connections 2024-05-09T09:01:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.569+0000 7ffa78688640 1 -- 172.21.15.98:0/157245749 wait complete. 2024-05-09T09:01:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa78688640 1 Processor -- start 2024-05-09T09:01:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa78688640 1 -- start start 2024-05-09T09:01:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa78688640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa70101840 0x7ffa70198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa78688640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa70102200 0x7ffa70198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa78688640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa70107850 0x7ffa7019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa78688640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa70111cb0 con 0x7ffa70107850 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa78688640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffa70111b30 con 0x7ffa70101840 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa78688640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffa70111e30 con 0x7ffa70102200 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa76c25640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa70107850 0x7ffa7019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa76c25640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa70107850 0x7ffa7019c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57754/0 (socket says 172.21.15.98:57754) 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa76c25640 1 -- 172.21.15.98:0/2274008479 learned_addr learned my addr 172.21.15.98:0/2274008479 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa75c23640 1 --2- 172.21.15.98:0/2274008479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa70102200 0x7ffa70198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa76424640 1 --2- 172.21.15.98:0/2274008479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa70101840 0x7ffa70198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa76c25640 1 -- 172.21.15.98:0/2274008479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa70102200 msgr2=0x7ffa70198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa76c25640 1 --2- 172.21.15.98:0/2274008479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa70102200 0x7ffa70198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa76c25640 1 -- 172.21.15.98:0/2274008479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa70101840 msgr2=0x7ffa70198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa76c25640 1 --2- 172.21.15.98:0/2274008479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa70101840 0x7ffa70198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.570+0000 7ffa76c25640 1 -- 172.21.15.98:0/2274008479 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7019d000 con 0x7ffa70107850 2024-05-09T09:01:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.571+0000 7ffa76c25640 1 --2- 172.21.15.98:0/2274008479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa70107850 0x7ffa7019c8d0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7ffa68033040 tx=0x7ffa68031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.571+0000 7ffa677fe640 1 -- 172.21.15.98:0/2274008479 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa68004040 con 0x7ffa70107850 2024-05-09T09:01:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.571+0000 7ffa677fe640 1 -- 172.21.15.98:0/2274008479 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa680046c0 con 0x7ffa70107850 2024-05-09T09:01:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.571+0000 7ffa78688640 1 -- 172.21.15.98:0/2274008479 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa7019d290 con 0x7ffa70107850 2024-05-09T09:01:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.571+0000 7ffa677fe640 1 -- 172.21.15.98:0/2274008479 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa68046480 con 0x7ffa70107850 2024-05-09T09:01:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.571+0000 7ffa78688640 1 -- 172.21.15.98:0/2274008479 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa701a4b80 con 0x7ffa70107850 2024-05-09T09:01:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.573+0000 7ffa677fe640 1 -- 172.21.15.98:0/2274008479 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ffa680305a0 con 0x7ffa70107850 2024-05-09T09:01:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.573+0000 7ffa7b394740 1 Processor -- start 2024-05-09T09:01:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa7b394740 1 -- start start 2024-05-09T09:01:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa7b394740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151450 0x560029151830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa7b394740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560029151d70 0x56002915c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa7b394740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56002915c630 0x56002915ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa7b394740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5600290b6340 con 0x56002915c630 2024-05-09T09:01:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa7b394740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56002905e290 con 0x560029151d70 2024-05-09T09:01:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa7b394740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5600290b2300 con 0x560029151450 2024-05-09T09:01:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa75c23640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56002915c630 0x56002915ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa75c23640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56002915c630 0x56002915ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57756/0 (socket says 172.21.15.98:57756) 2024-05-09T09:01:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa75c23640 1 -- 172.21.15.98:0/1368368854 learned_addr learned my addr 172.21.15.98:0/1368368854 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa76424640 1 --2- 172.21.15.98:0/1368368854 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560029151d70 0x56002915c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa75c23640 1 --2- 172.21.15.98:0/1368368854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151450 0x560029151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa677fe640 1 --2- 172.21.15.98:0/2274008479 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffa5c0a2e20 0x7ffa5c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa76c25640 1 --2- 172.21.15.98:0/2274008479 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffa5c0a2e20 0x7ffa5c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa75c23640 1 -- 172.21.15.98:0/1368368854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151450 msgr2=0x560029151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa75c23640 1 --2- 172.21.15.98:0/1368368854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151450 0x560029151830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa75c23640 1 -- 172.21.15.98:0/1368368854 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560029151d70 msgr2=0x56002915c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.574+0000 7ffa75c23640 1 --2- 172.21.15.98:0/1368368854 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560029151d70 0x56002915c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.575+0000 7ffa75c23640 1 -- 172.21.15.98:0/1368368854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600290828e0 con 0x56002915c630 2024-05-09T09:01:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.575+0000 7ffa677fe640 1 -- 172.21.15.98:0/2274008479 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9650+0+0 (secure 0 0 0) 0x7ffa6802e030 con 0x7ffa70107850 2024-05-09T09:01:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.575+0000 7ffa75c23640 1 --2- 172.21.15.98:0/1368368854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56002915c630 0x56002915ea20 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7ffa600149b0 tx=0x7ffa6000c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e478839fb63c189 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.575+0000 7ffa64ff9640 1 -- 172.21.15.98:0/1368368854 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa600142d0 con 0x56002915c630 2024-05-09T09:01:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.575+0000 7ffa76c25640 1 --2- 172.21.15.98:0/2274008479 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffa5c0a2e20 0x7ffa5c0a52e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ffa680bcda0 tx=0x7ffa680095c0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.575+0000 7ffa64ff9640 1 -- 172.21.15.98:0/1368368854 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa60005500 con 0x56002915c630 2024-05-09T09:01:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.575+0000 7ffa64ff9640 1 -- 172.21.15.98:0/1368368854 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6000f9c0 con 0x56002915c630 2024-05-09T09:01:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.576+0000 7ffa7b394740 1 -- 172.21.15.98:0/1368368854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56002915c630 msgr2=0x56002915ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.576+0000 7ffa7b394740 1 --2- 172.21.15.98:0/1368368854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56002915c630 0x56002915ea20 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7ffa600149b0 tx=0x7ffa6000c4b0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.576+0000 7ffa7b394740 1 -- 172.21.15.98:0/1368368854 shutdown_connections 2024-05-09T09:01:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.576+0000 7ffa7b394740 1 --2- 172.21.15.98:0/1368368854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56002915c630 0x56002915ea20 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.576+0000 7ffa7b394740 1 --2- 172.21.15.98:0/1368368854 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560029151d70 0x56002915c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.576+0000 7ffa7b394740 1 --2- 172.21.15.98:0/1368368854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151450 0x560029151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.576+0000 7ffa7b394740 1 -- 172.21.15.98:0/1368368854 >> 172.21.15.98:0/1368368854 conn(0x56002914d960 msgr2=0x56002914dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.576+0000 7ffa7b394740 1 -- 172.21.15.98:0/1368368854 shutdown_connections 2024-05-09T09:01:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.576+0000 7ffa7b394740 1 -- 172.21.15.98:0/1368368854 wait complete. 2024-05-09T09:01:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.576+0000 7ffa7b394740 1 -- start start 2024-05-09T09:01:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.576+0000 7ffa7b394740 1 Processor -- start 2024-05-09T09:01:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa7b394740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560029151450 0x5600291595e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa7b394740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151d70 0x560029159b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa7b394740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56002915c630 0x56002915a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa7b394740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56002915f260 con 0x560029151450 2024-05-09T09:01:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa7b394740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560029071790 con 0x56002915c630 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa7b394740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56002915f3e0 con 0x560029151d70 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa7b394740 10 client.0 _ll_register_callbacks cb 0x5600290aca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa75c23640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560029151450 0x5600291595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa75c23640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560029151450 0x5600291595e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57766/0 (socket says 172.21.15.98:57766) 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa75c23640 1 -- 172.21.15.98:0/3165903191 learned_addr learned my addr 172.21.15.98:0/3165903191 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa76424640 1 --2- 172.21.15.98:0/3165903191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151d70 0x560029159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa75c23640 1 -- 172.21.15.98:0/3165903191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151d70 msgr2=0x560029159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa75c23640 1 --2- 172.21.15.98:0/3165903191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151d70 0x560029159b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa75c23640 1 -- 172.21.15.98:0/3165903191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56002915c630 msgr2=0x56002915a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa75c23640 1 --2- 172.21.15.98:0/3165903191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56002915c630 0x56002915a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa75c23640 1 -- 172.21.15.98:0/3165903191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560028dca210 con 0x560029151450 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa76424640 1 --2- 172.21.15.98:0/3165903191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151d70 0x560029159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa75c23640 1 --2- 172.21.15.98:0/3165903191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560029151450 0x5600291595e0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7ffa60014980 tx=0x7ffa600056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf51ca68c847f1a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa59ffb640 1 -- 172.21.15.98:0/3165903191 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60028030 con 0x560029151450 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa59ffb640 1 -- 172.21.15.98:0/3165903191 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9650+0+0 (secure 0 0 0) 0x7ffa60005230 con 0x560029151450 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa7b394740 1 -- 172.21.15.98:0/3165903191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa60005500 con 0x560029151450 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.577+0000 7ffa59ffb640 1 -- 172.21.15.98:0/3165903191 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ffa60032050 con 0x560029151450 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.578+0000 7ffa7b394740 10 client.18699 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.578+0000 7ffa7b394740 1 -- 172.21.15.98:0/3165903191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560028e96620 con 0x560029151450 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.578+0000 7ffa59ffb640 1 -- 172.21.15.98:0/3165903191 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa60005500 con 0x560029151450 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.578+0000 7ffa59ffb640 1 -- 172.21.15.98:0/3165903191 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7ffa6000f8a0 con 0x560029151450 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.578+0000 7ffa7b394740 10 client.18699 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.578+0000 7ffa7b394740 10 client.18699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.578+0000 7ffa7b394740 20 client.18699 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.578+0000 7ffa7b394740 1 --2- 172.21.15.98:0/3165903191 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56002916c380 0x56002916e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.578+0000 7ffa7b394740 4 client.18699 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.578+0000 7ffa7b394740 1 -- 172.21.15.98:0/3165903191 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600290c3a70 con 0x56002916c380 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.578+0000 7ffa76424640 1 --2- 172.21.15.98:0/3165903191 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56002916c380 0x56002916e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.579+0000 7ffa76424640 1 --2- 172.21.15.98:0/3165903191 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56002916c380 0x56002916e770 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e103770fd114757 server_cookie=a496fce8d712cab9 in_seq=0 out_seq=0 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.579+0000 7ffa59ffb640 10 client.18699 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.579+0000 7ffa59ffb640 1 -- 172.21.15.98:0/3165903191 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5600290c3a70 con 0x56002916c380 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.579+0000 7ffa59ffb640 10 client.18699 handle_command_reply: tid=0 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.579+0000 7ffa7b394740 1 client.18699 shutdown 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.579+0000 7ffa7b394740 1 -- 172.21.15.98:0/3165903191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560029151450 msgr2=0x5600291595e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.579+0000 7ffa7b394740 1 --2- 172.21.15.98:0/3165903191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560029151450 0x5600291595e0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7ffa60014980 tx=0x7ffa600056b0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.579+0000 7ffa7b394740 1 -- 172.21.15.98:0/3165903191 shutdown_connections 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.579+0000 7ffa7b394740 1 --2- 172.21.15.98:0/3165903191 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x56002916c380 0x56002916e770 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.579+0000 7ffa7b394740 1 --2- 172.21.15.98:0/3165903191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56002915c630 0x56002915a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.579+0000 7ffa59ffb640 0 client.18699 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.579+0000 7ffa7b394740 1 --2- 172.21.15.98:0/3165903191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151d70 0x560029159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.579+0000 7ffa7b394740 1 --2- 172.21.15.98:0/3165903191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560029151450 0x5600291595e0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.579+0000 7ffa7b394740 1 -- 172.21.15.98:0/3165903191 >> 172.21.15.98:0/3165903191 conn(0x56002914d960 msgr2=0x560029153690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.580+0000 7ffa7b394740 1 -- 172.21.15.98:0/3165903191 shutdown_connections 2024-05-09T09:01:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.580+0000 7ffa7b394740 1 -- 172.21.15.98:0/3165903191 wait complete. 2024-05-09T09:01:13.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.580+0000 7ffa7b394740 20 client.18699 trim_cache size 0 max 16384 2024-05-09T09:01:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.594+0000 7ffa7b394740 1 Processor -- start 2024-05-09T09:01:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.594+0000 7ffa7b394740 1 -- start start 2024-05-09T09:01:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.594+0000 7ffa7b394740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151450 0x560029168120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.594+0000 7ffa7b394740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560029151d70 0x560029153e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.594+0000 7ffa7b394740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56002915c630 0x560029159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.594+0000 7ffa7b394740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa701aee40 con 0x56002915c630 2024-05-09T09:01:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.594+0000 7ffa7b394740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffa701b0980 con 0x560029151d70 2024-05-09T09:01:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.594+0000 7ffa7b394740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffa701af510 con 0x560029151450 2024-05-09T09:01:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.595+0000 7ffa75c23640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56002915c630 0x560029159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.595+0000 7ffa75c23640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56002915c630 0x560029159cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57776/0 (socket says 172.21.15.98:57776) 2024-05-09T09:01:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.595+0000 7ffa75c23640 1 -- 172.21.15.98:0/1254460441 learned_addr learned my addr 172.21.15.98:0/1254460441 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.595+0000 7ffa75c23640 1 -- 172.21.15.98:0/1254460441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151450 msgr2=0x560029168120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.595+0000 7ffa75c23640 1 --2- 172.21.15.98:0/1254460441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151450 0x560029168120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.595+0000 7ffa75c23640 1 -- 172.21.15.98:0/1254460441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560029151d70 msgr2=0x560029153e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.595+0000 7ffa75c23640 1 --2- 172.21.15.98:0/1254460441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560029151d70 0x560029153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.595+0000 7ffa75c23640 1 -- 172.21.15.98:0/1254460441 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560029157b90 con 0x56002915c630 2024-05-09T09:01:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.595+0000 7ffa75c23640 1 --2- 172.21.15.98:0/1254460441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56002915c630 0x560029159cb0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7ffa60001eb0 tx=0x7ffa60004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f32e552b80fdaca7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.595+0000 7ffa5bfff640 1 -- 172.21.15.98:0/1254460441 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60028030 con 0x56002915c630 2024-05-09T09:01:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.595+0000 7ffa5bfff640 1 -- 172.21.15.98:0/1254460441 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa60029040 con 0x56002915c630 2024-05-09T09:01:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.595+0000 7ffa5bfff640 1 -- 172.21.15.98:0/1254460441 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60004680 con 0x56002915c630 2024-05-09T09:01:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 -- 172.21.15.98:0/1254460441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56002915c630 msgr2=0x560029159cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 --2- 172.21.15.98:0/1254460441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56002915c630 0x560029159cb0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7ffa60001eb0 tx=0x7ffa60004ee0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 -- 172.21.15.98:0/1254460441 shutdown_connections 2024-05-09T09:01:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 --2- 172.21.15.98:0/1254460441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56002915c630 0x560029159cb0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 --2- 172.21.15.98:0/1254460441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560029151d70 0x560029153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 --2- 172.21.15.98:0/1254460441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x560029151450 0x560029168120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 -- 172.21.15.98:0/1254460441 >> 172.21.15.98:0/1254460441 conn(0x56002914d960 msgr2=0x5600291595a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 -- 172.21.15.98:0/1254460441 shutdown_connections 2024-05-09T09:01:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 -- 172.21.15.98:0/1254460441 wait complete. 2024-05-09T09:01:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 -- start start 2024-05-09T09:01:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 Processor -- start 2024-05-09T09:01:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560029151450 0x560029164920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560029151d70 0x560029155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56002915c630 0x5600291564f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.596+0000 7ffa7b394740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa701b02b0 con 0x560029151d70 2024-05-09T09:01:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa7b394740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffa701afbe0 con 0x560029151450 2024-05-09T09:01:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa7b394740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffa701ace40 con 0x56002915c630 2024-05-09T09:01:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa7b394740 10 client.0 _ll_register_callbacks cb 0x7ffa6c005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa76424640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560029151d70 0x560029155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa76424640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560029151d70 0x560029155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57784/0 (socket says 172.21.15.98:57784) 2024-05-09T09:01:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa76424640 1 -- 172.21.15.98:0/1135776249 learned_addr learned my addr 172.21.15.98:0/1135776249 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa76424640 1 -- 172.21.15.98:0/1135776249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56002915c630 msgr2=0x5600291564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa76424640 1 --2- 172.21.15.98:0/1135776249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56002915c630 0x5600291564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa76424640 1 -- 172.21.15.98:0/1135776249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560029151450 msgr2=0x560029164920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa76424640 1 --2- 172.21.15.98:0/1135776249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560029151450 0x560029164920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa76424640 1 -- 172.21.15.98:0/1135776249 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560029156c30 con 0x560029151d70 2024-05-09T09:01:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa76424640 1 --2- 172.21.15.98:0/1135776249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560029151d70 0x560029155fb0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x56002916ed40 tx=0x7ffa6c01af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e1dafd8e5f47f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa59ffb640 1 -- 172.21.15.98:0/1135776249 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c013e60 con 0x560029151d70 2024-05-09T09:01:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa59ffb640 1 -- 172.21.15.98:0/1135776249 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9650+0+0 (secure 0 0 0) 0x7ffa6c01a330 con 0x560029151d70 2024-05-09T09:01:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa7b394740 1 -- 172.21.15.98:0/1135776249 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa60029040 con 0x560029151d70 2024-05-09T09:01:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa59ffb640 1 -- 172.21.15.98:0/1135776249 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ffa6c02b050 con 0x560029151d70 2024-05-09T09:01:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa7b394740 10 client.18705 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa7b394740 1 -- 172.21.15.98:0/1135776249 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5600291d2600 con 0x560029151d70 2024-05-09T09:01:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.597+0000 7ffa59ffb640 1 -- 172.21.15.98:0/1135776249 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa60029040 con 0x560029151d70 2024-05-09T09:01:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.598+0000 7ffa59ffb640 1 -- 172.21.15.98:0/1135776249 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7ffa6c00f040 con 0x560029151d70 2024-05-09T09:01:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.598+0000 7ffa7b394740 10 client.18705 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.598+0000 7ffa7b394740 10 client.18705 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.598+0000 7ffa7b394740 20 client.18705 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.598+0000 7ffa7b394740 1 --2- 172.21.15.98:0/1135776249 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5600291d2db0 0x5600291d51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.598+0000 7ffa7b394740 4 client.18705 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:01:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.598+0000 7ffa7b394740 1 -- 172.21.15.98:0/1135776249 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5600290bddc0 con 0x5600291d2db0 2024-05-09T09:01:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.598+0000 7ffa75c23640 1 --2- 172.21.15.98:0/1135776249 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5600291d2db0 0x5600291d51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.598+0000 7ffa75c23640 1 --2- 172.21.15.98:0/1135776249 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5600291d2db0 0x5600291d51d0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4149d70d28651a53 server_cookie=ef583642d347c003 in_seq=0 out_seq=0 2024-05-09T09:01:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.598+0000 7ffa59ffb640 10 client.18705 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa59ffb640 1 -- 172.21.15.98:0/1135776249 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5600290bddc0 con 0x5600291d2db0 2024-05-09T09:01:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa59ffb640 10 client.18705 handle_command_reply: tid=0 2024-05-09T09:01:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa7b394740 1 client.18705 shutdown 2024-05-09T09:01:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa7b394740 1 -- 172.21.15.98:0/1135776249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560029151d70 msgr2=0x560029155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa7b394740 1 --2- 172.21.15.98:0/1135776249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560029151d70 0x560029155fb0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x56002916ed40 tx=0x7ffa6c01af90 comp rx=0 tx=0).stop 2024-05-09T09:01:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa7b394740 1 -- 172.21.15.98:0/1135776249 shutdown_connections 2024-05-09T09:01:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa7b394740 1 --2- 172.21.15.98:0/1135776249 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5600291d2db0 0x5600291d51d0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa7b394740 1 --2- 172.21.15.98:0/1135776249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56002915c630 0x5600291564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa59ffb640 0 client.18705 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:13.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa7b394740 1 --2- 172.21.15.98:0/1135776249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560029151d70 0x560029155fb0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa7b394740 1 --2- 172.21.15.98:0/1135776249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560029151450 0x560029164920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa7b394740 1 -- 172.21.15.98:0/1135776249 >> 172.21.15.98:0/1135776249 conn(0x56002914d960 msgr2=0x56002914c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:13.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa7b394740 1 -- 172.21.15.98:0/1135776249 shutdown_connections 2024-05-09T09:01:13.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa7b394740 1 -- 172.21.15.98:0/1135776249 wait complete. 2024-05-09T09:01:13.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.606+0000 7ffa7b394740 20 client.18705 trim_cache size 0 max 16384 2024-05-09T09:01:13.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T09:01:13.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T09:01:13.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T09:01:13.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T09:01:13.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.607+0000 7ffa78688640 1 -- 172.21.15.98:0/2274008479 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffa5c0a2e20 msgr2=0x7ffa5c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:13.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.607+0000 7ffa78688640 1 --2- 172.21.15.98:0/2274008479 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffa5c0a2e20 0x7ffa5c0a52e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ffa680bcda0 tx=0x7ffa680095c0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.607+0000 7ffa78688640 1 -- 172.21.15.98:0/2274008479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa70107850 msgr2=0x7ffa7019c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:13.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.607+0000 7ffa78688640 1 --2- 172.21.15.98:0/2274008479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa70107850 0x7ffa7019c8d0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7ffa68033040 tx=0x7ffa68031e60 comp rx=0 tx=0).stop 2024-05-09T09:01:13.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.607+0000 7ffa78688640 1 -- 172.21.15.98:0/2274008479 shutdown_connections 2024-05-09T09:01:13.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.607+0000 7ffa78688640 1 --2- 172.21.15.98:0/2274008479 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ffa5c0a2e20 0x7ffa5c0a52e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.607+0000 7ffa78688640 1 --2- 172.21.15.98:0/2274008479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa70107850 0x7ffa7019c8d0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.607+0000 7ffa78688640 1 --2- 172.21.15.98:0/2274008479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa70102200 0x7ffa70198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.607+0000 7ffa78688640 1 --2- 172.21.15.98:0/2274008479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa70101840 0x7ffa70198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:13.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.607+0000 7ffa78688640 1 -- 172.21.15.98:0/2274008479 >> 172.21.15.98:0/2274008479 conn(0x7ffa70078cc0 msgr2=0x7ffa700fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:13.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.607+0000 7ffa78688640 1 -- 172.21.15.98:0/2274008479 shutdown_connections 2024-05-09T09:01:13.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:13.607+0000 7ffa78688640 1 -- 172.21.15.98:0/2274008479 wait complete. 2024-05-09T09:01:14.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:14 smithi142 ceph-mon[34625]: pgmap v331: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 21 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:01:14.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:14 smithi105 ceph-mon[34498]: pgmap v331: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 21 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:01:14.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:14 smithi098 ceph-mon[30092]: pgmap v331: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 21 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:01:16.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:16 smithi142 ceph-mon[34625]: pgmap v332: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 28 KiB/s wr, 13 op/s 2024-05-09T09:01:16.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:16 smithi105 ceph-mon[34498]: pgmap v332: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 28 KiB/s wr, 13 op/s 2024-05-09T09:01:16.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:16 smithi098 ceph-mon[30092]: pgmap v332: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 28 KiB/s wr, 13 op/s 2024-05-09T09:01:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:18 smithi142 ceph-mon[34625]: pgmap v333: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 976 B/s wr, 11 op/s 2024-05-09T09:01:18.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:18 smithi142 ceph-mon[34625]: osdmap e107: 12 total, 12 up, 12 in 2024-05-09T09:01:18.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:18 smithi105 ceph-mon[34498]: pgmap v333: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 976 B/s wr, 11 op/s 2024-05-09T09:01:18.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:18 smithi105 ceph-mon[34498]: osdmap e107: 12 total, 12 up, 12 in 2024-05-09T09:01:18.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:18 smithi098 ceph-mon[30092]: pgmap v333: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 976 B/s wr, 11 op/s 2024-05-09T09:01:18.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:18 smithi098 ceph-mon[30092]: osdmap e107: 12 total, 12 up, 12 in 2024-05-09T09:01:20.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:20 smithi142 ceph-mon[34625]: pgmap v335: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 921 B/s wr, 11 op/s 2024-05-09T09:01:20.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:20 smithi105 ceph-mon[34498]: pgmap v335: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 921 B/s wr, 11 op/s 2024-05-09T09:01:20.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:20 smithi098 ceph-mon[30092]: pgmap v335: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 921 B/s wr, 11 op/s 2024-05-09T09:01:22.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:22 smithi098 ceph-mon[30092]: pgmap v336: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 921 B/s wr, 11 op/s 2024-05-09T09:01:22.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:22 smithi142 ceph-mon[34625]: pgmap v336: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 921 B/s wr, 11 op/s 2024-05-09T09:01:22.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:22 smithi105 ceph-mon[34498]: pgmap v336: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 921 B/s wr, 11 op/s 2024-05-09T09:01:24.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:24 smithi142 ceph-mon[34625]: pgmap v337: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 818 B/s wr, 10 op/s 2024-05-09T09:01:24.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:24 smithi105 ceph-mon[34498]: pgmap v337: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 818 B/s wr, 10 op/s 2024-05-09T09:01:24.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:24 smithi098 ceph-mon[30092]: pgmap v337: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 818 B/s wr, 10 op/s 2024-05-09T09:01:26.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:26 smithi142 ceph-mon[34625]: pgmap v338: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 716 B/s wr, 10 op/s 2024-05-09T09:01:26.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.707+0000 7fcfe9c31640 1 -- 172.21.15.98:0/4207872512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe41021d0 msgr2=0x7fcfe410e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:26.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.707+0000 7fcfe9c31640 1 --2- 172.21.15.98:0/4207872512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe41021d0 0x7fcfe410e6e0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009cd0 tx=0x7fcfdc02a8e0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.707+0000 7fcfe9c31640 1 -- 172.21.15.98:0/4207872512 shutdown_connections 2024-05-09T09:01:26.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.707+0000 7fcfe9c31640 1 --2- 172.21.15.98:0/4207872512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe41021d0 0x7fcfe410e6e0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.707+0000 7fcfe9c31640 1 --2- 172.21.15.98:0/4207872512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe4101810 0x7fcfe4101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.707+0000 7fcfe9c31640 1 --2- 172.21.15.98:0/4207872512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcfe4107820 0x7fcfe4107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.707+0000 7fcfe9c31640 1 -- 172.21.15.98:0/4207872512 >> 172.21.15.98:0/4207872512 conn(0x7fcfe4078cc0 msgr2=0x7fcfe40790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.707+0000 7fcfe9c31640 1 -- 172.21.15.98:0/4207872512 shutdown_connections 2024-05-09T09:01:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.707+0000 7fcfe9c31640 1 -- 172.21.15.98:0/4207872512 wait complete. 2024-05-09T09:01:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.707+0000 7fcfe9c31640 1 Processor -- start 2024-05-09T09:01:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.707+0000 7fcfe9c31640 1 -- start start 2024-05-09T09:01:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfe9c31640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcfe4101810 0x7fcfe4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfe9c31640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe41021d0 0x7fcfe4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfe9c31640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe4107820 0x7fcfe419c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfe9c31640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe4111c30 con 0x7fcfe41021d0 2024-05-09T09:01:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfe9c31640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe4111ab0 con 0x7fcfe4107820 2024-05-09T09:01:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfe9c31640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe4111db0 con 0x7fcfe4101810 2024-05-09T09:01:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfdbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe41021d0 0x7fcfe4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfdbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe41021d0 0x7fcfe4198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46926/0 (socket says 172.21.15.98:46926) 2024-05-09T09:01:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfdbfff640 1 -- 172.21.15.98:0/2497443028 learned_addr learned my addr 172.21.15.98:0/2497443028 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfe3fff640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe4107820 0x7fcfe419c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfe37fe640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcfe4101810 0x7fcfe4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfdbfff640 1 -- 172.21.15.98:0/2497443028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcfe4101810 msgr2=0x7fcfe4198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfdbfff640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcfe4101810 0x7fcfe4198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfdbfff640 1 -- 172.21.15.98:0/2497443028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe4107820 msgr2=0x7fcfe419c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfdbfff640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe4107820 0x7fcfe419c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfdbfff640 1 -- 172.21.15.98:0/2497443028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe419cff0 con 0x7fcfe41021d0 2024-05-09T09:01:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfe37fe640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcfe4101810 0x7fcfe4198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfdbfff640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe41021d0 0x7fcfe4198590 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00c890 tx=0x7fcfcc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfe17fa640 1 -- 172.21.15.98:0/2497443028 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc004220 con 0x7fcfe41021d0 2024-05-09T09:01:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfe17fa640 1 -- 172.21.15.98:0/2497443028 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfcc0043c0 con 0x7fcfe41021d0 2024-05-09T09:01:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfe9c31640 1 -- 172.21.15.98:0/2497443028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe419d2e0 con 0x7fcfe41021d0 2024-05-09T09:01:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfe17fa640 1 -- 172.21.15.98:0/2497443028 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc004a00 con 0x7fcfe41021d0 2024-05-09T09:01:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.708+0000 7fcfe9c31640 1 -- 172.21.15.98:0/2497443028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe41a4b70 con 0x7fcfe41021d0 2024-05-09T09:01:26.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.711+0000 7fcfe17fa640 1 -- 172.21.15.98:0/2497443028 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcfcc0079b0 con 0x7fcfe41021d0 2024-05-09T09:01:26.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.711+0000 7fcfeca1a740 1 Processor -- start 2024-05-09T09:01:26.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfeca1a740 1 -- start start 2024-05-09T09:01:26.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfeca1a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 0x55cc79b27830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.717 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfeca1a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b320f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfeca1a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b32630 0x55cc79b34a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfeca1a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cc79a8c340 con 0x55cc79b27d70 2024-05-09T09:01:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfeca1a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cc79a34290 con 0x55cc79b32630 2024-05-09T09:01:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfeca1a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cc79a88300 con 0x55cc79b27450 2024-05-09T09:01:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfe17fa640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcfd00a2da0 0x7fcfd00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfe37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b320f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfe37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b320f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46930/0 (socket says 172.21.15.98:46930) 2024-05-09T09:01:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfe37fe640 1 -- 172.21.15.98:0/1250571408 learned_addr learned my addr 172.21.15.98:0/1250571408 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfe3fff640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcfd00a2da0 0x7fcfd00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfe17fa640 1 -- 172.21.15.98:0/2497443028 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fcfcc0cf560 con 0x7fcfe41021d0 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfe3fff640 1 --2- 172.21.15.98:0/1250571408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 0x55cc79b27830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfdbfff640 1 --2- 172.21.15.98:0/1250571408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b32630 0x55cc79b34a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfe37fe640 1 -- 172.21.15.98:0/1250571408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 msgr2=0x55cc79b27830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfe37fe640 1 --2- 172.21.15.98:0/1250571408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 0x55cc79b27830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfe37fe640 1 -- 172.21.15.98:0/1250571408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b32630 msgr2=0x55cc79b34a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfe37fe640 1 --2- 172.21.15.98:0/1250571408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b32630 0x55cc79b34a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfe37fe640 1 -- 172.21.15.98:0/1250571408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc79a588e0 con 0x55cc79b27d70 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfdbfff640 1 --2- 172.21.15.98:0/1250571408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b32630 0x55cc79b34a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.712+0000 7fcfe37fe640 1 --2- 172.21.15.98:0/1250571408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b320f0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009bd0 tx=0x7fcfd400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4083183cbb4d3fc2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.713+0000 7fcfe3fff640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcfd00a2da0 0x7fcfd00a5260 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009430 tx=0x7fcfdc009360 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.713+0000 7fcfd9ffb640 1 -- 172.21.15.98:0/1250571408 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4016070 con 0x55cc79b27d70 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.713+0000 7fcfd9ffb640 1 -- 172.21.15.98:0/1250571408 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfd4011950 con 0x55cc79b27d70 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.713+0000 7fcfd9ffb640 1 -- 172.21.15.98:0/1250571408 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd40108a0 con 0x55cc79b27d70 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.713+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1250571408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 msgr2=0x55cc79b320f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.713+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/1250571408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b320f0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009bd0 tx=0x7fcfd400e6c0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.713+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1250571408 shutdown_connections 2024-05-09T09:01:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.713+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/1250571408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b32630 0x55cc79b34a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.713+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/1250571408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b320f0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.713+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/1250571408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 0x55cc79b27830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.713+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1250571408 >> 172.21.15.98:0/1250571408 conn(0x55cc79b23960 msgr2=0x55cc79b23d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.713+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1250571408 shutdown_connections 2024-05-09T09:01:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.713+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1250571408 wait complete. 2024-05-09T09:01:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.714+0000 7fcfeca1a740 1 -- start start 2024-05-09T09:01:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.714+0000 7fcfeca1a740 1 Processor -- start 2024-05-09T09:01:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.714+0000 7fcfeca1a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b27450 0x55cc79b2f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.714+0000 7fcfeca1a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b2fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.714+0000 7fcfeca1a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b32630 0x55cc79b30000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.714+0000 7fcfeca1a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cc79b35260 con 0x55cc79b27d70 2024-05-09T09:01:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.714+0000 7fcfeca1a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cc79a47790 con 0x55cc79b27450 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.714+0000 7fcfeca1a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cc79b353e0 con 0x55cc79b32630 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.714+0000 7fcfeca1a740 10 client.0 _ll_register_callbacks cb 0x55cc79a82a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.714+0000 7fcfdbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b2fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.714+0000 7fcfdbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b2fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46942/0 (socket says 172.21.15.98:46942) 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.714+0000 7fcfdbfff640 1 -- 172.21.15.98:0/292282026 learned_addr learned my addr 172.21.15.98:0/292282026 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.714+0000 7fcfe37fe640 1 --2- 172.21.15.98:0/292282026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b27450 0x55cc79b2f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfdbfff640 1 -- 172.21.15.98:0/292282026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b32630 msgr2=0x55cc79b30000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfdbfff640 1 --2- 172.21.15.98:0/292282026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b32630 0x55cc79b30000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfdbfff640 1 -- 172.21.15.98:0/292282026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b27450 msgr2=0x55cc79b2f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfdbfff640 1 --2- 172.21.15.98:0/292282026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b27450 0x55cc79b2f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfdbfff640 1 -- 172.21.15.98:0/292282026 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc797a0210 con 0x55cc79b27d70 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfdbfff640 1 --2- 172.21.15.98:0/292282026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b2fac0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc007590 tx=0x7fcfcc01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b65e5ee3023d69 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfc2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfc2ffd640 1 -- 172.21.15.98:0/292282026 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc0b4060 con 0x55cc79b27d70 2024-05-09T09:01:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfc2ffd640 1 -- 172.21.15.98:0/292282026 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fcfcc007ca0 con 0x55cc79b27d70 2024-05-09T09:01:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfeca1a740 1 -- 172.21.15.98:0/292282026 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfd4011950 con 0x55cc79b27d70 2024-05-09T09:01:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfc2ffd640 1 -- 172.21.15.98:0/292282026 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fcfcc086070 con 0x55cc79b27d70 2024-05-09T09:01:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfeca1a740 10 client.18717 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfeca1a740 1 -- 172.21.15.98:0/292282026 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cc7986c620 con 0x55cc79b27d70 2024-05-09T09:01:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfc2ffd640 1 -- 172.21.15.98:0/292282026 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfd4011950 con 0x55cc79b27d70 2024-05-09T09:01:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfc2ffd640 1 -- 172.21.15.98:0/292282026 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fcfcc0ccb20 con 0x55cc79b27d70 2024-05-09T09:01:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfeca1a740 10 client.18717 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfeca1a740 10 client.18717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfeca1a740 20 client.18717 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/292282026 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cc79b42260 0x55cc79b44650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfeca1a740 4 client.18717 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:01:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.715+0000 7fcfeca1a740 1 -- 172.21.15.98:0/292282026 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc79a99a70 con 0x55cc79b42260 2024-05-09T09:01:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.716+0000 7fcfe37fe640 1 --2- 172.21.15.98:0/292282026 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cc79b42260 0x55cc79b44650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.716+0000 7fcfc2ffd640 10 client.18717 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.716+0000 7fcfe37fe640 1 --2- 172.21.15.98:0/292282026 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cc79b42260 0x55cc79b44650 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ea74b873803f5d4 server_cookie=58ebd57b626b4a02 in_seq=0 out_seq=0 2024-05-09T09:01:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfc2ffd640 1 -- 172.21.15.98:0/292282026 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cc79a99a70 con 0x55cc79b42260 2024-05-09T09:01:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfc2ffd640 10 client.18717 handle_command_reply: tid=0 2024-05-09T09:01:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfeca1a740 1 client.18717 shutdown 2024-05-09T09:01:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfeca1a740 1 -- 172.21.15.98:0/292282026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 msgr2=0x55cc79b2fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/292282026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b2fac0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc007590 tx=0x7fcfcc01b000 comp rx=0 tx=0).stop 2024-05-09T09:01:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfeca1a740 1 -- 172.21.15.98:0/292282026 shutdown_connections 2024-05-09T09:01:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/292282026 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cc79b42260 0x55cc79b44650 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/292282026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b32630 0x55cc79b30000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfc2ffd640 0 client.18717 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/292282026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b2fac0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/292282026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b27450 0x55cc79b2f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfeca1a740 1 -- 172.21.15.98:0/292282026 >> 172.21.15.98:0/292282026 conn(0x55cc79b23960 msgr2=0x55cc79b29690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfeca1a740 1 -- 172.21.15.98:0/292282026 shutdown_connections 2024-05-09T09:01:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfeca1a740 1 -- 172.21.15.98:0/292282026 wait complete. 2024-05-09T09:01:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.717+0000 7fcfeca1a740 20 client.18717 trim_cache size 0 max 16384 2024-05-09T09:01:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfeca1a740 1 Processor -- start 2024-05-09T09:01:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfeca1a740 1 -- start start 2024-05-09T09:01:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfeca1a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 0x55cc79b3e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfeca1a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b2e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfeca1a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b32630 0x55cc79b23180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfeca1a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe41aee80 con 0x55cc79b27d70 2024-05-09T09:01:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfeca1a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe41b09c0 con 0x55cc79b32630 2024-05-09T09:01:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfeca1a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe41af550 con 0x55cc79b27450 2024-05-09T09:01:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfdbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b2e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfdbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b2e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46948/0 (socket says 172.21.15.98:46948) 2024-05-09T09:01:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfdbfff640 1 -- 172.21.15.98:0/1594177362 learned_addr learned my addr 172.21.15.98:0/1594177362 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfdbfff640 1 -- 172.21.15.98:0/1594177362 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 msgr2=0x55cc79b3e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfdbfff640 1 --2- 172.21.15.98:0/1594177362 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 0x55cc79b3e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfdbfff640 1 -- 172.21.15.98:0/1594177362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b32630 msgr2=0x55cc79b23180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfdbfff640 1 --2- 172.21.15.98:0/1594177362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b32630 0x55cc79b23180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.732+0000 7fcfdbfff640 1 -- 172.21.15.98:0/1594177362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc79b27ba0 con 0x55cc79b27d70 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.733+0000 7fcfdbfff640 1 --2- 172.21.15.98:0/1594177362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b2e700 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc09b600 tx=0x7fcfcc0ba170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80d2999f01a9e7f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.733+0000 7fcfd97fa640 1 -- 172.21.15.98:0/1594177362 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc01d040 con 0x55cc79b27d70 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.733+0000 7fcfd97fa640 1 -- 172.21.15.98:0/1594177362 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfcc09b670 con 0x55cc79b27d70 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.733+0000 7fcfd97fa640 1 -- 172.21.15.98:0/1594177362 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc0b4040 con 0x55cc79b27d70 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.733+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1594177362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 msgr2=0x55cc79b2e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.733+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/1594177362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b2e700 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc09b600 tx=0x7fcfcc0ba170 comp rx=0 tx=0).stop 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.733+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1594177362 shutdown_connections 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.733+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/1594177362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b32630 0x55cc79b23180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.733+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/1594177362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b27d70 0x55cc79b2e700 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.733+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/1594177362 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 0x55cc79b3e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.733+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1594177362 >> 172.21.15.98:0/1594177362 conn(0x55cc79b23960 msgr2=0x55cc79b28e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:26.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.733+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1594177362 shutdown_connections 2024-05-09T09:01:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.733+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1594177362 wait complete. 2024-05-09T09:01:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfeca1a740 1 -- start start 2024-05-09T09:01:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfeca1a740 1 Processor -- start 2024-05-09T09:01:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfeca1a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 0x55cc79b2b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfeca1a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b27d70 0x55cc79b2bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfeca1a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b32630 0x55cc79b2c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfeca1a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe41b02f0 con 0x55cc79b32630 2024-05-09T09:01:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfeca1a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe41afc20 con 0x55cc79b27d70 2024-05-09T09:01:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfeca1a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe41ace80 con 0x55cc79b27450 2024-05-09T09:01:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfeca1a740 10 client.0 _ll_register_callbacks cb 0x7fcfcc004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfe3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b32630 0x55cc79b2c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfe3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b32630 0x55cc79b2c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46954/0 (socket says 172.21.15.98:46954) 2024-05-09T09:01:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfe3fff640 1 -- 172.21.15.98:0/1935218239 learned_addr learned my addr 172.21.15.98:0/1935218239 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:26.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfe37fe640 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 0x55cc79b2b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfe3fff640 1 -- 172.21.15.98:0/1935218239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 msgr2=0x55cc79b2b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:26.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfe3fff640 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 0x55cc79b2b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfdbfff640 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b27d70 0x55cc79b2bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfe3fff640 1 -- 172.21.15.98:0/1935218239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b27d70 msgr2=0x55cc79b2bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:26.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfe3fff640 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b27d70 0x55cc79b2bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfe3fff640 1 -- 172.21.15.98:0/1935218239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc79b2c900 con 0x55cc79b32630 2024-05-09T09:01:26.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfe37fe640 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 0x55cc79b2b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:26.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfdbfff640 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b27d70 0x55cc79b2bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:26.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfe3fff640 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b32630 0x55cc79b2c290 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009cd0 tx=0x7fcfdc009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f38dcee5f7eac97 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:26.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.734+0000 7fcfc2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:26.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfc2ffd640 1 -- 172.21.15.98:0/1935218239 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc04f070 con 0x55cc79b32630 2024-05-09T09:01:26.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfc2ffd640 1 -- 172.21.15.98:0/1935218239 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fcfdc004720 con 0x55cc79b32630 2024-05-09T09:01:26.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1935218239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cc79a93dc0 con 0x55cc79b32630 2024-05-09T09:01:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfc2ffd640 1 -- 172.21.15.98:0/1935218239 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fcfdc004b40 con 0x55cc79b32630 2024-05-09T09:01:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfeca1a740 10 client.18723 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1935218239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cc79bacc10 con 0x55cc79b32630 2024-05-09T09:01:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfc2ffd640 1 -- 172.21.15.98:0/1935218239 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55cc79a93dc0 con 0x55cc79b32630 2024-05-09T09:01:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfc2ffd640 1 -- 172.21.15.98:0/1935218239 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fcfdc048040 con 0x55cc79b32630 2024-05-09T09:01:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfeca1a740 10 client.18723 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfeca1a740 10 client.18723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfeca1a740 20 client.18723 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cc79bad390 0x55cc79baf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfeca1a740 4 client.18723 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:01:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1935218239 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cc79a118e0 con 0x55cc79bad390 2024-05-09T09:01:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.735+0000 7fcfe37fe640 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cc79bad390 0x55cc79baf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.736+0000 7fcfc2ffd640 10 client.18723 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.736+0000 7fcfe37fe640 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cc79bad390 0x55cc79baf7b0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d813d4209b8201 server_cookie=dfc4736235d837fe in_seq=0 out_seq=0 2024-05-09T09:01:26.741 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:26 smithi105 ceph-mon[34498]: pgmap v338: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 716 B/s wr, 10 op/s 2024-05-09T09:01:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfc2ffd640 1 -- 172.21.15.98:0/1935218239 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cc79a118e0 con 0x55cc79bad390 2024-05-09T09:01:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfc2ffd640 10 client.18723 handle_command_reply: tid=0 2024-05-09T09:01:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfeca1a740 1 client.18723 shutdown 2024-05-09T09:01:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1935218239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b32630 msgr2=0x55cc79b2c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b32630 0x55cc79b2c290 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009cd0 tx=0x7fcfdc009800 comp rx=0 tx=0).stop 2024-05-09T09:01:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1935218239 shutdown_connections 2024-05-09T09:01:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cc79bad390 0x55cc79baf7b0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc79b32630 0x55cc79b2c290 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cc79b27d70 0x55cc79b2bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfc2ffd640 0 client.18723 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfeca1a740 1 --2- 172.21.15.98:0/1935218239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc79b27450 0x55cc79b2b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1935218239 >> 172.21.15.98:0/1935218239 conn(0x55cc79b23960 msgr2=0x55cc79b29690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1935218239 shutdown_connections 2024-05-09T09:01:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfeca1a740 1 -- 172.21.15.98:0/1935218239 wait complete. 2024-05-09T09:01:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.740+0000 7fcfeca1a740 20 client.18723 trim_cache size 0 max 16384 2024-05-09T09:01:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T09:01:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T09:01:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T09:01:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T09:01:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.741+0000 7fcfe9c31640 1 -- 172.21.15.98:0/2497443028 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcfd00a2da0 msgr2=0x7fcfd00a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.741+0000 7fcfe9c31640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcfd00a2da0 0x7fcfd00a5260 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009430 tx=0x7fcfdc009360 comp rx=0 tx=0).stop 2024-05-09T09:01:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.741+0000 7fcfe9c31640 1 -- 172.21.15.98:0/2497443028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe41021d0 msgr2=0x7fcfe4198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.741+0000 7fcfe9c31640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe41021d0 0x7fcfe4198590 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00c890 tx=0x7fcfcc00cd50 comp rx=0 tx=0).stop 2024-05-09T09:01:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.741+0000 7fcfe9c31640 1 -- 172.21.15.98:0/2497443028 shutdown_connections 2024-05-09T09:01:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.741+0000 7fcfe9c31640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fcfd00a2da0 0x7fcfd00a5260 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.741+0000 7fcfe9c31640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe4107820 0x7fcfe419c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.741+0000 7fcfe9c31640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe41021d0 0x7fcfe4198590 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.741+0000 7fcfe9c31640 1 --2- 172.21.15.98:0/2497443028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcfe4101810 0x7fcfe4198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.741+0000 7fcfe9c31640 1 -- 172.21.15.98:0/2497443028 >> 172.21.15.98:0/2497443028 conn(0x7fcfe4078cc0 msgr2=0x7fcfe40ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.741+0000 7fcfe9c31640 1 -- 172.21.15.98:0/2497443028 shutdown_connections 2024-05-09T09:01:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:26.741+0000 7fcfe9c31640 1 -- 172.21.15.98:0/2497443028 wait complete. 2024-05-09T09:01:26.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:26 smithi098 ceph-mon[30092]: pgmap v338: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 716 B/s wr, 10 op/s 2024-05-09T09:01:26.870 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:01:26.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:01:26.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.977+0000 7f67dcf09640 1 -- 172.21.15.98:0/3652240952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8106720 msgr2=0x7f67d8106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:26.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.977+0000 7f67dcf09640 1 --2- 172.21.15.98:0/3652240952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8106720 0x7f67d8106b00 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f67c8009930 tx=0x7f67c802a420 comp rx=0 tx=0).stop 2024-05-09T09:01:26.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.977+0000 7f67dcf09640 1 -- 172.21.15.98:0/3652240952 shutdown_connections 2024-05-09T09:01:26.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.977+0000 7f67dcf09640 1 --2- 172.21.15.98:0/3652240952 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67d81010d0 0x7f67d810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.977+0000 7f67dcf09640 1 --2- 172.21.15.98:0/3652240952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d8100710 0x7f67d8100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.977+0000 7f67dcf09640 1 --2- 172.21.15.98:0/3652240952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8106720 0x7f67d8106b00 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.977+0000 7f67dcf09640 1 -- 172.21.15.98:0/3652240952 >> 172.21.15.98:0/3652240952 conn(0x7f67d8077790 msgr2=0x7f67d8077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:26.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.977+0000 7f67dcf09640 1 -- 172.21.15.98:0/3652240952 shutdown_connections 2024-05-09T09:01:26.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.977+0000 7f67dcf09640 1 -- 172.21.15.98:0/3652240952 wait complete. 2024-05-09T09:01:26.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.978+0000 7f67dcf09640 1 Processor -- start 2024-05-09T09:01:26.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.978+0000 7f67dcf09640 1 -- start start 2024-05-09T09:01:26.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.978+0000 7f67dcf09640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d8100710 0x7f67d8198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67dcf09640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67d81010d0 0x7f67d8199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67dcf09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8106720 0x7f67d81a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67dcf09640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f67d8110af0 con 0x7f67d8106720 2024-05-09T09:01:26.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67dcf09640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f67d8110970 con 0x7f67d8100710 2024-05-09T09:01:26.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67dcf09640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f67d8110c70 con 0x7f67d81010d0 2024-05-09T09:01:26.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67d6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8106720 0x7f67d81a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67d6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8106720 0x7f67d81a3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46964/0 (socket says 172.21.15.98:46964) 2024-05-09T09:01:26.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67d6d9d640 1 -- 172.21.15.98:0/2678843257 learned_addr learned my addr 172.21.15.98:0/2678843257 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:26.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67cdd9b640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67d81010d0 0x7f67d8199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67d659c640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d8100710 0x7f67d8198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67d6d9d640 1 -- 172.21.15.98:0/2678843257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67d81010d0 msgr2=0x7f67d8199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:26.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67d6d9d640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67d81010d0 0x7f67d8199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67d6d9d640 1 -- 172.21.15.98:0/2678843257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d8100710 msgr2=0x7f67d8198ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:26.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67d6d9d640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d8100710 0x7f67d8198ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67d6d9d640 1 -- 172.21.15.98:0/2678843257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d81a44b0 con 0x7f67d8106720 2024-05-09T09:01:26.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67d659c640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d8100710 0x7f67d8198ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:26.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67d6d9d640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8106720 0x7f67d81a3f70 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f67d0009fd0 tx=0x7f67d000def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:26.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67cffff640 1 -- 172.21.15.98:0/2678843257 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0017070 con 0x7f67d8106720 2024-05-09T09:01:26.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67dcf09640 1 -- 172.21.15.98:0/2678843257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d81a4680 con 0x7f67d8106720 2024-05-09T09:01:26.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67cffff640 1 -- 172.21.15.98:0/2678843257 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67d0003e50 con 0x7f67d8106720 2024-05-09T09:01:26.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67cffff640 1 -- 172.21.15.98:0/2678843257 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0020060 con 0x7f67d8106720 2024-05-09T09:01:26.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.979+0000 7f67dcf09640 1 -- 172.21.15.98:0/2678843257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d81a4ba0 con 0x7f67d8106720 2024-05-09T09:01:26.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.982+0000 7f67dfc10740 1 Processor -- start 2024-05-09T09:01:26.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.982+0000 7f67cffff640 1 -- 172.21.15.98:0/2678843257 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f67d0004000 con 0x7f67d8106720 2024-05-09T09:01:26.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67dfc10740 1 -- start start 2024-05-09T09:01:26.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67dfc10740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4280 0x55b2c70f4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67dfc10740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 0x55b2c70fed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67dfc10740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70ff270 0x55b2c7101660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67dfc10740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b2c7059d00 con 0x55b2c70f4280 2024-05-09T09:01:26.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67dfc10740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b2c70014c0 con 0x55b2c70f4ba0 2024-05-09T09:01:26.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67dfc10740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b2c7055cc0 con 0x55b2c70ff270 2024-05-09T09:01:26.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67cdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4280 0x55b2c70f4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67cdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4280 0x55b2c70f4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46980/0 (socket says 172.21.15.98:46980) 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67cdd9b640 1 -- 172.21.15.98:0/4026291239 learned_addr learned my addr 172.21.15.98:0/4026291239 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67cffff640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f67c40a2d00 0x7f67c40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67cdd9b640 1 -- 172.21.15.98:0/4026291239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70ff270 msgr2=0x55b2c7101660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67cdd9b640 1 --2- 172.21.15.98:0/4026291239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70ff270 0x55b2c7101660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67d659c640 1 --2- 172.21.15.98:0/4026291239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 0x55b2c70fed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67cdd9b640 1 -- 172.21.15.98:0/4026291239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 msgr2=0x55b2c70fed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67cdd9b640 1 --2- 172.21.15.98:0/4026291239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 0x55b2c70fed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67cdd9b640 1 -- 172.21.15.98:0/4026291239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2c6e377d0 con 0x55b2c70f4280 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67d659c640 1 --2- 172.21.15.98:0/4026291239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 0x55b2c70fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67d6d9d640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f67c40a2d00 0x7f67c40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67cffff640 1 -- 172.21.15.98:0/2678843257 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f67d00cf500 con 0x7f67d8106720 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.983+0000 7f67cdd9b640 1 --2- 172.21.15.98:0/4026291239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4280 0x55b2c70f4660 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f67c000ad40 tx=0x7f67c00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa124634a3393cf8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.984+0000 7f67ccd99640 1 -- 172.21.15.98:0/4026291239 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0012450 con 0x55b2c70f4280 2024-05-09T09:01:26.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.984+0000 7f67ccd99640 1 -- 172.21.15.98:0/4026291239 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c0012ad0 con 0x55b2c70f4280 2024-05-09T09:01:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.984+0000 7f67ccd99640 1 -- 172.21.15.98:0/4026291239 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c000faa0 con 0x55b2c70f4280 2024-05-09T09:01:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.984+0000 7f67dfc10740 1 -- 172.21.15.98:0/4026291239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4280 msgr2=0x55b2c70f4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.984+0000 7f67d6d9d640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f67c40a2d00 0x7f67c40a51c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f67d00cc990 tx=0x7f67d00861e0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.984+0000 7f67dfc10740 1 --2- 172.21.15.98:0/4026291239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4280 0x55b2c70f4660 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f67c000ad40 tx=0x7f67c00117b0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.984+0000 7f67dfc10740 1 -- 172.21.15.98:0/4026291239 shutdown_connections 2024-05-09T09:01:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.984+0000 7f67dfc10740 1 --2- 172.21.15.98:0/4026291239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70ff270 0x55b2c7101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.984+0000 7f67dfc10740 1 --2- 172.21.15.98:0/4026291239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 0x55b2c70fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.984+0000 7f67dfc10740 1 --2- 172.21.15.98:0/4026291239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4280 0x55b2c70f4660 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.984+0000 7f67dfc10740 1 -- 172.21.15.98:0/4026291239 >> 172.21.15.98:0/4026291239 conn(0x55b2c70f05a0 msgr2=0x55b2c70f0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67dfc10740 1 -- 172.21.15.98:0/4026291239 shutdown_connections 2024-05-09T09:01:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67dfc10740 1 -- 172.21.15.98:0/4026291239 wait complete. 2024-05-09T09:01:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67dfc10740 1 -- start start 2024-05-09T09:01:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67dfc10740 1 Processor -- start 2024-05-09T09:01:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67dfc10740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4280 0x55b2c70fc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67dfc10740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 0x55b2c70fc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67dfc10740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70ff270 0x55b2c70fccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67dfc10740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b2c7102060 con 0x55b2c70f4280 2024-05-09T09:01:26.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67dfc10740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b2c7014b90 con 0x55b2c70f4ba0 2024-05-09T09:01:26.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67dfc10740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b2c71021e0 con 0x55b2c70ff270 2024-05-09T09:01:26.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67dfc10740 10 client.0 _ll_register_callbacks cb 0x55b2c7050da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:26.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67cdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4280 0x55b2c70fc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67cdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4280 0x55b2c70fc240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46996/0 (socket says 172.21.15.98:46996) 2024-05-09T09:01:26.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67cdd9b640 1 -- 172.21.15.98:0/3878246761 learned_addr learned my addr 172.21.15.98:0/3878246761 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:26.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67d659c640 1 --2- 172.21.15.98:0/3878246761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 0x55b2c70fc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67cdd9b640 1 -- 172.21.15.98:0/3878246761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70ff270 msgr2=0x55b2c70fccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:26.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67cdd9b640 1 --2- 172.21.15.98:0/3878246761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70ff270 0x55b2c70fccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67cdd9b640 1 -- 172.21.15.98:0/3878246761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 msgr2=0x55b2c70fc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:26.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67cdd9b640 1 --2- 172.21.15.98:0/3878246761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 0x55b2c70fc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.985+0000 7f67cdd9b640 1 -- 172.21.15.98:0/3878246761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2c6d6d3f0 con 0x55b2c70f4280 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67cdd9b640 1 --2- 172.21.15.98:0/3878246761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4280 0x55b2c70fc240 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f67c00057c0 tx=0x7f67c0012990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7035125717bb8d7d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67bdffb640 1 -- 172.21.15.98:0/3878246761 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0005310 con 0x55b2c70f4280 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67bdffb640 1 -- 172.21.15.98:0/3878246761 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f67c00054b0 con 0x55b2c70f4280 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67dfc10740 1 -- 172.21.15.98:0/3878246761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67c0012ad0 con 0x55b2c70f4280 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67bdffb640 1 -- 172.21.15.98:0/3878246761 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f67c002f050 con 0x55b2c70f4280 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67dfc10740 10 client.18735 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67dfc10740 1 -- 172.21.15.98:0/3878246761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b2c6d2d0f0 con 0x55b2c70f4280 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67bdffb640 1 -- 172.21.15.98:0/3878246761 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c0012ad0 con 0x55b2c70f4280 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67bdffb640 1 -- 172.21.15.98:0/3878246761 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f67c0004c30 con 0x55b2c70f4280 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67dfc10740 10 client.18735 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67dfc10740 10 client.18735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67dfc10740 20 client.18735 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67dfc10740 1 --2- 172.21.15.98:0/3878246761 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b2c710f480 0x55b2c7111870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:26.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67dfc10740 4 client.18735 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.986+0000 7f67dfc10740 1 -- 172.21.15.98:0/3878246761 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2c7067430 con 0x55b2c710f480 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.987+0000 7f67d659c640 1 --2- 172.21.15.98:0/3878246761 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b2c710f480 0x55b2c7111870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.987+0000 7f67bdffb640 10 client.18735 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.987+0000 7f67d659c640 1 --2- 172.21.15.98:0/3878246761 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b2c710f480 0x55b2c7111870 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8694c2c586adb32e server_cookie=9c485e5c35370817 in_seq=0 out_seq=0 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67bdffb640 1 -- 172.21.15.98:0/3878246761 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b2c7067430 con 0x55b2c710f480 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67bdffb640 10 client.18735 handle_command_reply: tid=0 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67dfc10740 1 client.18735 shutdown 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67dfc10740 1 -- 172.21.15.98:0/3878246761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4280 msgr2=0x55b2c70fc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67dfc10740 1 --2- 172.21.15.98:0/3878246761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4280 0x55b2c70fc240 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f67c00057c0 tx=0x7f67c0012990 comp rx=0 tx=0).stop 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67dfc10740 1 -- 172.21.15.98:0/3878246761 shutdown_connections 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67dfc10740 1 --2- 172.21.15.98:0/3878246761 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b2c710f480 0x55b2c7111870 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67dfc10740 1 --2- 172.21.15.98:0/3878246761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70ff270 0x55b2c70fccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67bdffb640 0 client.18735 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67dfc10740 1 --2- 172.21.15.98:0/3878246761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 0x55b2c70fc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67dfc10740 1 --2- 172.21.15.98:0/3878246761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4280 0x55b2c70fc240 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:26.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67dfc10740 1 -- 172.21.15.98:0/3878246761 >> 172.21.15.98:0/3878246761 conn(0x55b2c70f05a0 msgr2=0x55b2c70ef7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:26.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67dfc10740 1 -- 172.21.15.98:0/3878246761 shutdown_connections 2024-05-09T09:01:26.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67dfc10740 1 -- 172.21.15.98:0/3878246761 wait complete. 2024-05-09T09:01:26.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:26.988+0000 7f67dfc10740 20 client.18735 trim_cache size 0 max 16384 2024-05-09T09:01:27.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.005+0000 7f67dfc10740 1 Processor -- start 2024-05-09T09:01:27.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.005+0000 7f67dfc10740 1 -- start start 2024-05-09T09:01:27.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.005+0000 7f67dfc10740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70f4280 0x55b2c70fe970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.005+0000 7f67dfc10740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4ba0 0x55b2c710af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.005+0000 7f67dfc10740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70ff270 0x55b2c70fc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.005+0000 7f67dfc10740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f67d81aeea0 con 0x55b2c70f4ba0 2024-05-09T09:01:27.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.005+0000 7f67dfc10740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f67d81b09e0 con 0x55b2c70ff270 2024-05-09T09:01:27.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.005+0000 7f67dfc10740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f67d81af570 con 0x55b2c70f4280 2024-05-09T09:01:27.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.005+0000 7f67d659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4ba0 0x55b2c710af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.005+0000 7f67d659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4ba0 0x55b2c710af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46998/0 (socket says 172.21.15.98:46998) 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.005+0000 7f67d659c640 1 -- 172.21.15.98:0/2379437155 learned_addr learned my addr 172.21.15.98:0/2379437155 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67d659c640 1 -- 172.21.15.98:0/2379437155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70f4280 msgr2=0x55b2c70fe970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67d659c640 1 --2- 172.21.15.98:0/2379437155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70f4280 0x55b2c70fe970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67d659c640 1 -- 172.21.15.98:0/2379437155 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70ff270 msgr2=0x55b2c70fc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67d659c640 1 --2- 172.21.15.98:0/2379437155 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70ff270 0x55b2c70fc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67d659c640 1 -- 172.21.15.98:0/2379437155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2c70ebce0 con 0x55b2c70f4ba0 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67d659c640 1 --2- 172.21.15.98:0/2379437155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4ba0 0x55b2c710af20 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f67c8000c00 tx=0x7f67c8031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc45169f468ef6ea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67bffff640 1 -- 172.21.15.98:0/2379437155 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c8035040 con 0x55b2c70f4ba0 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67bffff640 1 -- 172.21.15.98:0/2379437155 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c8045070 con 0x55b2c70f4ba0 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67bffff640 1 -- 172.21.15.98:0/2379437155 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c8040750 con 0x55b2c70f4ba0 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67dfc10740 1 -- 172.21.15.98:0/2379437155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4ba0 msgr2=0x55b2c710af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67dfc10740 1 --2- 172.21.15.98:0/2379437155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4ba0 0x55b2c710af20 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f67c8000c00 tx=0x7f67c8031400 comp rx=0 tx=0).stop 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67dfc10740 1 -- 172.21.15.98:0/2379437155 shutdown_connections 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67dfc10740 1 --2- 172.21.15.98:0/2379437155 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70ff270 0x55b2c70fc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67dfc10740 1 --2- 172.21.15.98:0/2379437155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70f4ba0 0x55b2c710af20 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67dfc10740 1 --2- 172.21.15.98:0/2379437155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70f4280 0x55b2c70fe970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67dfc10740 1 -- 172.21.15.98:0/2379437155 >> 172.21.15.98:0/2379437155 conn(0x55b2c70f05a0 msgr2=0x55b2c70fc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67dfc10740 1 -- 172.21.15.98:0/2379437155 shutdown_connections 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.006+0000 7f67dfc10740 1 -- 172.21.15.98:0/2379437155 wait complete. 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67dfc10740 1 -- start start 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67dfc10740 1 Processor -- start 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67dfc10740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70f4280 0x55b2c7103110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67dfc10740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 0x55b2c7103650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67dfc10740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70ff270 0x55b2c7103bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67dfc10740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f67d81b0310 con 0x55b2c70ff270 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67dfc10740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f67d81afc40 con 0x55b2c70f4ba0 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67dfc10740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f67d81acea0 con 0x55b2c70f4280 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67dfc10740 10 client.0 _ll_register_callbacks cb 0x7f67c000ac10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67cdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70ff270 0x55b2c7103bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67cdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70ff270 0x55b2c7103bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47008/0 (socket says 172.21.15.98:47008) 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67cdd9b640 1 -- 172.21.15.98:0/1364505911 learned_addr learned my addr 172.21.15.98:0/1364505911 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67d659c640 1 --2- 172.21.15.98:0/1364505911 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 0x55b2c7103650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67cdd9b640 1 -- 172.21.15.98:0/1364505911 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70f4280 msgr2=0x55b2c7103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67cdd9b640 1 --2- 172.21.15.98:0/1364505911 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70f4280 0x55b2c7103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67cdd9b640 1 -- 172.21.15.98:0/1364505911 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 msgr2=0x55b2c7103650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67cdd9b640 1 --2- 172.21.15.98:0/1364505911 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 0x55b2c7103650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67cdd9b640 1 -- 172.21.15.98:0/1364505911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2c71749d0 con 0x55b2c70ff270 2024-05-09T09:01:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.007+0000 7f67d659c640 1 --2- 172.21.15.98:0/1364505911 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 0x55b2c7103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67cdd9b640 1 --2- 172.21.15.98:0/1364505911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70ff270 0x55b2c7103bb0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f67c0001eb0 tx=0x7f67c00266f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc12b2efef34c8c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67bdffb640 1 -- 172.21.15.98:0/1364505911 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0026ac0 con 0x55b2c70ff270 2024-05-09T09:01:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67bdffb640 1 -- 172.21.15.98:0/1364505911 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f67c0026c60 con 0x55b2c70ff270 2024-05-09T09:01:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67dfc10740 1 -- 172.21.15.98:0/1364505911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67c8045070 con 0x55b2c70ff270 2024-05-09T09:01:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67bdffb640 1 -- 172.21.15.98:0/1364505911 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f67c003e050 con 0x55b2c70ff270 2024-05-09T09:01:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67dfc10740 10 client.18741 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:27.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67dfc10740 1 -- 172.21.15.98:0/1364505911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b2c7175d40 con 0x55b2c70ff270 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67bdffb640 1 -- 172.21.15.98:0/1364505911 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c8045070 con 0x55b2c70ff270 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67bdffb640 1 -- 172.21.15.98:0/1364505911 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f67c00049d0 con 0x55b2c70ff270 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67dfc10740 10 client.18741 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67dfc10740 10 client.18741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67dfc10740 20 client.18741 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67dfc10740 1 --2- 172.21.15.98:0/1364505911 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b2c71764c0 0x55b2c71788e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67dfc10740 4 client.18741 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67dfc10740 1 -- 172.21.15.98:0/1364505911 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b2c7061780 con 0x55b2c71764c0 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.008+0000 7f67d659c640 1 --2- 172.21.15.98:0/1364505911 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b2c71764c0 0x55b2c71788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67bdffb640 10 client.18741 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67d659c640 1 --2- 172.21.15.98:0/1364505911 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b2c71764c0 0x55b2c71788e0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ee88490cb0ee30c server_cookie=3bd9b2a4335891be in_seq=0 out_seq=0 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67bdffb640 1 -- 172.21.15.98:0/1364505911 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b2c7061780 con 0x55b2c71764c0 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67bdffb640 10 client.18741 handle_command_reply: tid=0 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67dfc10740 1 client.18741 shutdown 2024-05-09T09:01:27.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67dfc10740 1 -- 172.21.15.98:0/1364505911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70ff270 msgr2=0x55b2c7103bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67dfc10740 1 --2- 172.21.15.98:0/1364505911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70ff270 0x55b2c7103bb0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f67c0001eb0 tx=0x7f67c00266f0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67dfc10740 1 -- 172.21.15.98:0/1364505911 shutdown_connections 2024-05-09T09:01:27.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67dfc10740 1 --2- 172.21.15.98:0/1364505911 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55b2c71764c0 0x55b2c71788e0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67dfc10740 1 --2- 172.21.15.98:0/1364505911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2c70ff270 0x55b2c7103bb0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67bdffb640 0 client.18741 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:27.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67dfc10740 1 --2- 172.21.15.98:0/1364505911 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b2c70f4ba0 0x55b2c7103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67dfc10740 1 --2- 172.21.15.98:0/1364505911 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2c70f4280 0x55b2c7103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67dfc10740 1 -- 172.21.15.98:0/1364505911 >> 172.21.15.98:0/1364505911 conn(0x55b2c70f05a0 msgr2=0x55b2c70fca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67dfc10740 1 -- 172.21.15.98:0/1364505911 shutdown_connections 2024-05-09T09:01:27.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67dfc10740 1 -- 172.21.15.98:0/1364505911 wait complete. 2024-05-09T09:01:27.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.009+0000 7f67dfc10740 20 client.18741 trim_cache size 0 max 16384 2024-05-09T09:01:27.016 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:01:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T09:01:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T09:01:27.016 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:01:27.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.010+0000 7f67dcf09640 1 -- 172.21.15.98:0/2678843257 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f67c40a2d00 msgr2=0x7f67c40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.010+0000 7f67dcf09640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f67c40a2d00 0x7f67c40a51c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f67d00cc990 tx=0x7f67d00861e0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.010+0000 7f67dcf09640 1 -- 172.21.15.98:0/2678843257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8106720 msgr2=0x7f67d81a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.010+0000 7f67dcf09640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8106720 0x7f67d81a3f70 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f67d0009fd0 tx=0x7f67d000def0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.010+0000 7f67dcf09640 1 -- 172.21.15.98:0/2678843257 shutdown_connections 2024-05-09T09:01:27.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.010+0000 7f67dcf09640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f67c40a2d00 0x7f67c40a51c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.010+0000 7f67dcf09640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8106720 0x7f67d81a3f70 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.010+0000 7f67dcf09640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67d81010d0 0x7f67d8199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.010+0000 7f67dcf09640 1 --2- 172.21.15.98:0/2678843257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d8100710 0x7f67d8198ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.010+0000 7f67dcf09640 1 -- 172.21.15.98:0/2678843257 >> 172.21.15.98:0/2678843257 conn(0x7f67d8077790 msgr2=0x7f67d81032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.010+0000 7f67dcf09640 1 -- 172.21.15.98:0/2678843257 shutdown_connections 2024-05-09T09:01:27.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.010+0000 7f67dcf09640 1 -- 172.21.15.98:0/2678843257 wait complete. 2024-05-09T09:01:27.020 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T09:01:27.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:01:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.123+0000 7efdf57f9640 1 -- 172.21.15.98:0/210454448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0100700 msgr2=0x7efdf0100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.123+0000 7efdf57f9640 1 --2- 172.21.15.98:0/210454448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0100700 0x7efdf0100b80 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7efde4009930 tx=0x7efde402a420 comp rx=0 tx=0).stop 2024-05-09T09:01:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.123+0000 7efdf57f9640 1 -- 172.21.15.98:0/210454448 shutdown_connections 2024-05-09T09:01:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.123+0000 7efdf57f9640 1 --2- 172.21.15.98:0/210454448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf01010c0 0x7efdf010d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.123+0000 7efdf57f9640 1 --2- 172.21.15.98:0/210454448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0100700 0x7efdf0100b80 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.123+0000 7efdf57f9640 1 --2- 172.21.15.98:0/210454448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdf0106710 0x7efdf0106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.123+0000 7efdf57f9640 1 -- 172.21.15.98:0/210454448 >> 172.21.15.98:0/210454448 conn(0x7efdf0076ab0 msgr2=0x7efdf0076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.123+0000 7efdf57f9640 1 -- 172.21.15.98:0/210454448 shutdown_connections 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.123+0000 7efdf57f9640 1 -- 172.21.15.98:0/210454448 wait complete. 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.123+0000 7efdf57f9640 1 Processor -- start 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.123+0000 7efdf57f9640 1 -- start start 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdf57f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0100700 0x7efdf0198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdf57f9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf01010c0 0x7efdf01994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdf57f9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdf0106710 0x7efdf01a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdf57f9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdf0110b20 con 0x7efdf0100700 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdf57f9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efdf01109a0 con 0x7efdf01010c0 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdf57f9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efdf0110ca0 con 0x7efdf0106710 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdeeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0100700 0x7efdf0198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdeeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0100700 0x7efdf0198f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47020/0 (socket says 172.21.15.98:47020) 2024-05-09T09:01:27.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdeeffd640 1 -- 172.21.15.98:0/2416373337 learned_addr learned my addr 172.21.15.98:0/2416373337 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdee7fc640 1 --2- 172.21.15.98:0/2416373337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf01010c0 0x7efdf01994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdef7fe640 1 --2- 172.21.15.98:0/2416373337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdf0106710 0x7efdf01a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdeeffd640 1 -- 172.21.15.98:0/2416373337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdf0106710 msgr2=0x7efdf01a4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdeeffd640 1 --2- 172.21.15.98:0/2416373337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdf0106710 0x7efdf01a4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdeeffd640 1 -- 172.21.15.98:0/2416373337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf01010c0 msgr2=0x7efdf01994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdeeffd640 1 --2- 172.21.15.98:0/2416373337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf01010c0 0x7efdf01994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdeeffd640 1 -- 172.21.15.98:0/2416373337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf01a4540 con 0x7efdf0100700 2024-05-09T09:01:27.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdeeffd640 1 --2- 172.21.15.98:0/2416373337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0100700 0x7efdf0198f80 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7efde000ea80 tx=0x7efde000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdd7fff640 1 -- 172.21.15.98:0/2416373337 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde000ce40 con 0x7efdf0100700 2024-05-09T09:01:27.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdf57f9640 1 -- 172.21.15.98:0/2416373337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdf01a4710 con 0x7efdf0100700 2024-05-09T09:01:27.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdd7fff640 1 -- 172.21.15.98:0/2416373337 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efde0005020 con 0x7efdf0100700 2024-05-09T09:01:27.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.124+0000 7efdd7fff640 1 -- 172.21.15.98:0/2416373337 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde0011760 con 0x7efdf0100700 2024-05-09T09:01:27.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.125+0000 7efdf57f9640 1 -- 172.21.15.98:0/2416373337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdf01a4c30 con 0x7efdf0100700 2024-05-09T09:01:27.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.125+0000 7efdf57f9640 1 -- 172.21.15.98:0/2416373337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdf0101e40 con 0x7efdf0100700 2024-05-09T09:01:27.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.127+0000 7efdd7fff640 1 -- 172.21.15.98:0/2416373337 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7efde00051d0 con 0x7efdf0100700 2024-05-09T09:01:27.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.128+0000 7efdd7fff640 1 --2- 172.21.15.98:0/2416373337 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efdd80a2d00 0x7efdd80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.128+0000 7efdee7fc640 1 --2- 172.21.15.98:0/2416373337 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efdd80a2d00 0x7efdd80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.128+0000 7efdd7fff640 1 -- 172.21.15.98:0/2416373337 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7efde00cfed0 con 0x7efdf0100700 2024-05-09T09:01:27.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.128+0000 7efdee7fc640 1 --2- 172.21.15.98:0/2416373337 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efdd80a2d00 0x7efdd80a51c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efde4009900 tx=0x7efde40058d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.131+0000 7efdd7fff640 1 -- 172.21.15.98:0/2416373337 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efde0016030 con 0x7efdf0100700 2024-05-09T09:01:27.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.346+0000 7efdf57f9640 1 -- 172.21.15.98:0/2416373337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efdf0108c70 con 0x7efdf0100700 2024-05-09T09:01:27.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.346+0000 7efdd7fff640 1 -- 172.21.15.98:0/2416373337 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11941 (secure 0 0 0) 0x7efde009c680 con 0x7efdf0100700 2024-05-09T09:01:27.346 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T09:01:27.346 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":26581,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/561027115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":561027115},{"type":"v1","addr":"172.21.15.105:6835","nonce":561027115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:56:52.891273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16641":{"gid":16641,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T09:01:27.346 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 37 2024-05-09T09:01:27.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.348+0000 7efdf57f9640 1 -- 172.21.15.98:0/2416373337 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efdd80a2d00 msgr2=0x7efdd80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.348+0000 7efdf57f9640 1 --2- 172.21.15.98:0/2416373337 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efdd80a2d00 0x7efdd80a51c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efde4009900 tx=0x7efde40058d0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.349+0000 7efdf57f9640 1 -- 172.21.15.98:0/2416373337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0100700 msgr2=0x7efdf0198f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.349+0000 7efdf57f9640 1 --2- 172.21.15.98:0/2416373337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0100700 0x7efdf0198f80 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7efde000ea80 tx=0x7efde000ef40 comp rx=0 tx=0).stop 2024-05-09T09:01:27.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.349+0000 7efdf57f9640 1 -- 172.21.15.98:0/2416373337 shutdown_connections 2024-05-09T09:01:27.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.349+0000 7efdf57f9640 1 --2- 172.21.15.98:0/2416373337 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7efdd80a2d00 0x7efdd80a51c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.349+0000 7efdf57f9640 1 --2- 172.21.15.98:0/2416373337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdf0106710 0x7efdf01a4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.349+0000 7efdf57f9640 1 --2- 172.21.15.98:0/2416373337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf01010c0 0x7efdf01994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.349+0000 7efdf57f9640 1 --2- 172.21.15.98:0/2416373337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0100700 0x7efdf0198f80 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.349+0000 7efdf57f9640 1 -- 172.21.15.98:0/2416373337 >> 172.21.15.98:0/2416373337 conn(0x7efdf0076ab0 msgr2=0x7efdf00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.349+0000 7efdf57f9640 1 -- 172.21.15.98:0/2416373337 shutdown_connections 2024-05-09T09:01:27.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.349+0000 7efdf57f9640 1 -- 172.21.15.98:0/2416373337 wait complete. 2024-05-09T09:01:27.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T09:01:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.464+0000 7fe550c9e640 1 -- 172.21.15.98:0/714382903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c1021d0 msgr2=0x7fe54c10e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.464+0000 7fe550c9e640 1 --2- 172.21.15.98:0/714382903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c1021d0 0x7fe54c10e6e0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fe544009cd0 tx=0x7fe54402a8e0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.465+0000 7fe550c9e640 1 -- 172.21.15.98:0/714382903 shutdown_connections 2024-05-09T09:01:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.465+0000 7fe550c9e640 1 --2- 172.21.15.98:0/714382903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c1021d0 0x7fe54c10e6e0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.465+0000 7fe550c9e640 1 --2- 172.21.15.98:0/714382903 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe54c101810 0x7fe54c101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.465+0000 7fe550c9e640 1 --2- 172.21.15.98:0/714382903 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe54c107820 0x7fe54c107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.465+0000 7fe550c9e640 1 -- 172.21.15.98:0/714382903 >> 172.21.15.98:0/714382903 conn(0x7fe54c078c30 msgr2=0x7fe54c079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.465+0000 7fe550c9e640 1 -- 172.21.15.98:0/714382903 shutdown_connections 2024-05-09T09:01:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.465+0000 7fe550c9e640 1 -- 172.21.15.98:0/714382903 wait complete. 2024-05-09T09:01:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.465+0000 7fe550c9e640 1 Processor -- start 2024-05-09T09:01:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.465+0000 7fe550c9e640 1 -- start start 2024-05-09T09:01:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe550c9e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe54c101810 0x7fe54c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe550c9e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe54c1021d0 0x7fe54c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe550c9e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c107820 0x7fe54c0745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe550c9e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c111bf0 con 0x7fe54c107820 2024-05-09T09:01:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe550c9e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c111a70 con 0x7fe54c1021d0 2024-05-09T09:01:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe550c9e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c111d70 con 0x7fe54c101810 2024-05-09T09:01:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe54ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c107820 0x7fe54c0745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe54ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c107820 0x7fe54c0745b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47034/0 (socket says 172.21.15.98:47034) 2024-05-09T09:01:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe54ad9d640 1 -- 172.21.15.98:0/3895345416 learned_addr learned my addr 172.21.15.98:0/3895345416 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe54a59c640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe54c101810 0x7fe54c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe541d9b640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe54c1021d0 0x7fe54c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe54ad9d640 1 -- 172.21.15.98:0/3895345416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe54c101810 msgr2=0x7fe54c073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe54ad9d640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe54c101810 0x7fe54c073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe54ad9d640 1 -- 172.21.15.98:0/3895345416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe54c1021d0 msgr2=0x7fe54c074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe54ad9d640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe54c1021d0 0x7fe54c074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe54ad9d640 1 -- 172.21.15.98:0/3895345416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54c1a65b0 con 0x7fe54c107820 2024-05-09T09:01:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe541d9b640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe54c1021d0 0x7fe54c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe54ad9d640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c107820 0x7fe54c0745b0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fe544005cc0 tx=0x7fe544003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.466+0000 7fe543fff640 1 -- 172.21.15.98:0/3895345416 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe544004040 con 0x7fe54c107820 2024-05-09T09:01:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.467+0000 7fe550c9e640 1 -- 172.21.15.98:0/3895345416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe54c1a6840 con 0x7fe54c107820 2024-05-09T09:01:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.467+0000 7fe543fff640 1 -- 172.21.15.98:0/3895345416 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5440046c0 con 0x7fe54c107820 2024-05-09T09:01:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.467+0000 7fe543fff640 1 -- 172.21.15.98:0/3895345416 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe544046480 con 0x7fe54c107820 2024-05-09T09:01:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.467+0000 7fe550c9e640 1 -- 172.21.15.98:0/3895345416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe54c1a6d90 con 0x7fe54c107820 2024-05-09T09:01:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe543fff640 1 -- 172.21.15.98:0/3895345416 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe5440305a0 con 0x7fe54c107820 2024-05-09T09:01:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe5539ac740 1 Processor -- start 2024-05-09T09:01:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe5539ac740 1 -- start start 2024-05-09T09:01:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe5539ac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3b330 0x5631fce3b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe5539ac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3bc50 0x5631fce45de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe5539ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce48710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe5539ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5631fcda0c40 con 0x5631fce46320 2024-05-09T09:01:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe5539ac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5631fcd48580 con 0x5631fce3b330 2024-05-09T09:01:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe5539ac740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5631fcd9cc00 con 0x5631fce3bc50 2024-05-09T09:01:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe541d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce48710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe541d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce48710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47044/0 (socket says 172.21.15.98:47044) 2024-05-09T09:01:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe541d9b640 1 -- 172.21.15.98:0/2501183378 learned_addr learned my addr 172.21.15.98:0/2501183378 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe543fff640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe5380a2d00 0x7fe5380a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe54a59c640 1 --2- 172.21.15.98:0/2501183378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3bc50 0x5631fce45de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.469+0000 7fe541d9b640 1 --2- 172.21.15.98:0/2501183378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3b330 0x5631fce3b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.470+0000 7fe54ad9d640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe5380a2d00 0x7fe5380a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.470+0000 7fe541d9b640 1 -- 172.21.15.98:0/2501183378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3bc50 msgr2=0x5631fce45de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.470+0000 7fe541d9b640 1 --2- 172.21.15.98:0/2501183378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3bc50 0x5631fce45de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.470+0000 7fe541d9b640 1 -- 172.21.15.98:0/2501183378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3b330 msgr2=0x5631fce3b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.470+0000 7fe541d9b640 1 --2- 172.21.15.98:0/2501183378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3b330 0x5631fce3b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.470+0000 7fe541d9b640 1 -- 172.21.15.98:0/2501183378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631fcab43f0 con 0x5631fce46320 2024-05-09T09:01:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.470+0000 7fe543fff640 1 -- 172.21.15.98:0/3895345416 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fe54402e030 con 0x7fe54c107820 2024-05-09T09:01:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.470+0000 7fe541d9b640 1 --2- 172.21.15.98:0/2501183378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce48710 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fe534013690 tx=0x7fe53400c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d7742adcbba690 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.470+0000 7fe540d99640 1 -- 172.21.15.98:0/2501183378 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534019030 con 0x5631fce46320 2024-05-09T09:01:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.470+0000 7fe540d99640 1 -- 172.21.15.98:0/2501183378 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe534011ab0 con 0x5631fce46320 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.470+0000 7fe540d99640 1 -- 172.21.15.98:0/2501183378 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53400fc50 con 0x5631fce46320 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.470+0000 7fe54ad9d640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe5380a2d00 0x7fe5380a51c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe5440bcea0 tx=0x7fe54408f160 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 -- 172.21.15.98:0/2501183378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 msgr2=0x5631fce48710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 --2- 172.21.15.98:0/2501183378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce48710 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fe534013690 tx=0x7fe53400c750 comp rx=0 tx=0).stop 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 -- 172.21.15.98:0/2501183378 shutdown_connections 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 --2- 172.21.15.98:0/2501183378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce48710 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 --2- 172.21.15.98:0/2501183378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3bc50 0x5631fce45de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 --2- 172.21.15.98:0/2501183378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3b330 0x5631fce3b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 -- 172.21.15.98:0/2501183378 >> 172.21.15.98:0/2501183378 conn(0x5631fce37650 msgr2=0x5631fce37a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 -- 172.21.15.98:0/2501183378 shutdown_connections 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 -- 172.21.15.98:0/2501183378 wait complete. 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 -- start start 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 Processor -- start 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3b330 0x5631fce432f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3bc50 0x5631fce43830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce43d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5631fce49110 con 0x5631fce46320 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5631fcd5bc40 con 0x5631fce3bc50 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5631fce49290 con 0x5631fce3b330 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe5539ac740 10 client.0 _ll_register_callbacks cb 0x5631fcab3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe541d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce43d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe541d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce43d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47060/0 (socket says 172.21.15.98:47060) 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.471+0000 7fe541d9b640 1 -- 172.21.15.98:0/338677517 learned_addr learned my addr 172.21.15.98:0/338677517 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe54a59c640 1 --2- 172.21.15.98:0/338677517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3bc50 0x5631fce43830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe541d9b640 1 --2- 172.21.15.98:0/338677517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3b330 0x5631fce432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe541d9b640 1 -- 172.21.15.98:0/338677517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3b330 msgr2=0x5631fce432f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe541d9b640 1 --2- 172.21.15.98:0/338677517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3b330 0x5631fce432f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe541d9b640 1 -- 172.21.15.98:0/338677517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3bc50 msgr2=0x5631fce43830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe541d9b640 1 --2- 172.21.15.98:0/338677517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3bc50 0x5631fce43830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe541d9b640 1 -- 172.21.15.98:0/338677517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631fcb7e7d0 con 0x5631fce46320 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe541d9b640 1 --2- 172.21.15.98:0/338677517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce43d70 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fe534005790 tx=0x7fe534015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d818f6f53dbf1d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe531ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe531ffb640 1 -- 172.21.15.98:0/338677517 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534019070 con 0x5631fce46320 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe531ffb640 1 -- 172.21.15.98:0/338677517 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fe5340115d0 con 0x5631fce46320 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe5539ac740 1 -- 172.21.15.98:0/338677517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe534011ab0 con 0x5631fce46320 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe531ffb640 1 -- 172.21.15.98:0/338677517 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe534031050 con 0x5631fce46320 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe5539ac740 10 client.18759 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe5539ac740 1 -- 172.21.15.98:0/338677517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5631fce55360 con 0x5631fce46320 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.472+0000 7fe531ffb640 1 -- 172.21.15.98:0/338677517 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe534011ab0 con 0x5631fce46320 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.473+0000 7fe531ffb640 1 -- 172.21.15.98:0/338677517 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe5340035f0 con 0x5631fce46320 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.473+0000 7fe5539ac740 10 client.18759 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.473+0000 7fe5539ac740 10 client.18759 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.473+0000 7fe5539ac740 20 client.18759 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.473+0000 7fe5539ac740 1 --2- 172.21.15.98:0/338677517 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5631fce56760 0x5631fce58b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.473+0000 7fe5539ac740 4 client.18759 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:01:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.473+0000 7fe5539ac740 1 -- 172.21.15.98:0/338677517 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631fcdae370 con 0x5631fce56760 2024-05-09T09:01:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.473+0000 7fe54a59c640 1 --2- 172.21.15.98:0/338677517 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5631fce56760 0x5631fce58b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.473+0000 7fe531ffb640 10 client.18759 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T09:01:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.473+0000 7fe54a59c640 1 --2- 172.21.15.98:0/338677517 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5631fce56760 0x5631fce58b50 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3d22218ea30aa258 server_cookie=a115f2aecb8a245d in_seq=0 out_seq=0 2024-05-09T09:01:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe531ffb640 1 -- 172.21.15.98:0/338677517 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5631fcdae370 con 0x5631fce56760 2024-05-09T09:01:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe531ffb640 10 client.18759 handle_command_reply: tid=0 2024-05-09T09:01:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe5539ac740 1 client.18759 shutdown 2024-05-09T09:01:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe5539ac740 1 -- 172.21.15.98:0/338677517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 msgr2=0x5631fce43d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe5539ac740 1 --2- 172.21.15.98:0/338677517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce43d70 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fe534005790 tx=0x7fe534015040 comp rx=0 tx=0).stop 2024-05-09T09:01:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe5539ac740 1 -- 172.21.15.98:0/338677517 shutdown_connections 2024-05-09T09:01:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe5539ac740 1 --2- 172.21.15.98:0/338677517 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5631fce56760 0x5631fce58b50 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe5539ac740 1 --2- 172.21.15.98:0/338677517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce43d70 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe531ffb640 0 client.18759 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T09:01:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe5539ac740 1 --2- 172.21.15.98:0/338677517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3bc50 0x5631fce43830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe5539ac740 1 --2- 172.21.15.98:0/338677517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3b330 0x5631fce432f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe5539ac740 1 -- 172.21.15.98:0/338677517 >> 172.21.15.98:0/338677517 conn(0x5631fce37650 msgr2=0x5631fce3d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe5539ac740 1 -- 172.21.15.98:0/338677517 shutdown_connections 2024-05-09T09:01:27.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe5539ac740 1 -- 172.21.15.98:0/338677517 wait complete. 2024-05-09T09:01:27.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.474+0000 7fe5539ac740 20 client.18759 trim_cache size 0 max 16384 2024-05-09T09:01:27.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.487+0000 7fe5539ac740 1 Processor -- start 2024-05-09T09:01:27.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.487+0000 7fe5539ac740 1 -- start start 2024-05-09T09:01:27.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe5539ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce3b330 0x5631fce369f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe5539ac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3bc50 0x5631fce52000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe5539ac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce46320 0x5631fce3dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe5539ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c1b0f80 con 0x5631fce3b330 2024-05-09T09:01:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe5539ac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c1b2ac0 con 0x5631fce46320 2024-05-09T09:01:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe5539ac740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c1b1650 con 0x5631fce3bc50 2024-05-09T09:01:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe541d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce3b330 0x5631fce369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe541d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce3b330 0x5631fce369f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47068/0 (socket says 172.21.15.98:47068) 2024-05-09T09:01:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe541d9b640 1 -- 172.21.15.98:0/3334077777 learned_addr learned my addr 172.21.15.98:0/3334077777 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe541d9b640 1 -- 172.21.15.98:0/3334077777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3bc50 msgr2=0x5631fce52000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe541d9b640 1 --2- 172.21.15.98:0/3334077777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3bc50 0x5631fce52000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe541d9b640 1 -- 172.21.15.98:0/3334077777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce46320 msgr2=0x5631fce3dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe541d9b640 1 --2- 172.21.15.98:0/3334077777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce46320 0x5631fce3dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe541d9b640 1 -- 172.21.15.98:0/3334077777 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631fce28d20 con 0x5631fce3b330 2024-05-09T09:01:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe541d9b640 1 --2- 172.21.15.98:0/3334077777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce3b330 0x5631fce369f0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fe5340039c0 tx=0x7fe53400ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48ec67b40da5f31 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe533fff640 1 -- 172.21.15.98:0/3334077777 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53403a070 con 0x5631fce3b330 2024-05-09T09:01:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe533fff640 1 -- 172.21.15.98:0/3334077777 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe53400fdf0 con 0x5631fce3b330 2024-05-09T09:01:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.488+0000 7fe533fff640 1 -- 172.21.15.98:0/3334077777 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534019040 con 0x5631fce3b330 2024-05-09T09:01:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.489+0000 7fe5539ac740 1 -- 172.21.15.98:0/3334077777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce3b330 msgr2=0x5631fce369f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.489+0000 7fe5539ac740 1 --2- 172.21.15.98:0/3334077777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce3b330 0x5631fce369f0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fe5340039c0 tx=0x7fe53400ce40 comp rx=0 tx=0).stop 2024-05-09T09:01:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.489+0000 7fe5539ac740 1 -- 172.21.15.98:0/3334077777 shutdown_connections 2024-05-09T09:01:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.489+0000 7fe5539ac740 1 --2- 172.21.15.98:0/3334077777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce46320 0x5631fce3dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.489+0000 7fe5539ac740 1 --2- 172.21.15.98:0/3334077777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3bc50 0x5631fce52000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.489+0000 7fe5539ac740 1 --2- 172.21.15.98:0/3334077777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce3b330 0x5631fce369f0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.489+0000 7fe5539ac740 1 -- 172.21.15.98:0/3334077777 >> 172.21.15.98:0/3334077777 conn(0x5631fce37650 msgr2=0x5631fce43b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.489+0000 7fe5539ac740 1 -- 172.21.15.98:0/3334077777 shutdown_connections 2024-05-09T09:01:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.489+0000 7fe5539ac740 1 -- 172.21.15.98:0/3334077777 wait complete. 2024-05-09T09:01:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.489+0000 7fe5539ac740 1 -- start start 2024-05-09T09:01:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.489+0000 7fe5539ac740 1 Processor -- start 2024-05-09T09:01:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.489+0000 7fe5539ac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3b330 0x5631fce4a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.489+0000 7fe5539ac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3bc50 0x5631fce4a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.489+0000 7fe5539ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce4ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe5539ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c1b23f0 con 0x5631fce46320 2024-05-09T09:01:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe5539ac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c1b1d20 con 0x5631fce3bc50 2024-05-09T09:01:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe5539ac740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c1aef80 con 0x5631fce3b330 2024-05-09T09:01:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe5539ac740 10 client.0 _ll_register_callbacks cb 0x7fe53c00b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe541d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce4ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe541d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce4ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47074/0 (socket says 172.21.15.98:47074) 2024-05-09T09:01:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe541d9b640 1 -- 172.21.15.98:0/3391532341 learned_addr learned my addr 172.21.15.98:0/3391532341 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe541d9b640 1 --2- 172.21.15.98:0/3391532341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3b330 0x5631fce4a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe54a59c640 1 --2- 172.21.15.98:0/3391532341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3bc50 0x5631fce4a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe541d9b640 1 -- 172.21.15.98:0/3391532341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3b330 msgr2=0x5631fce4a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe541d9b640 1 --2- 172.21.15.98:0/3391532341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3b330 0x5631fce4a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe541d9b640 1 -- 172.21.15.98:0/3391532341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3bc50 msgr2=0x5631fce4a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe541d9b640 1 --2- 172.21.15.98:0/3391532341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3bc50 0x5631fce4a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe541d9b640 1 -- 172.21.15.98:0/3391532341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631fcebbb80 con 0x5631fce46320 2024-05-09T09:01:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe54a59c640 1 --2- 172.21.15.98:0/3391532341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3bc50 0x5631fce4a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe541d9b640 1 --2- 172.21.15.98:0/3391532341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce4ab00 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fe534016040 tx=0x7fe534003690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9045d47befed3f60 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe531ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe531ffb640 1 -- 172.21.15.98:0/3391532341 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53403a040 con 0x5631fce46320 2024-05-09T09:01:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe531ffb640 1 -- 172.21.15.98:0/3391532341 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fe5340115d0 con 0x5631fce46320 2024-05-09T09:01:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe5539ac740 1 -- 172.21.15.98:0/3391532341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe53400fdf0 con 0x5631fce46320 2024-05-09T09:01:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe531ffb640 1 -- 172.21.15.98:0/3391532341 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe5340117d0 con 0x5631fce46320 2024-05-09T09:01:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe5539ac740 10 client.18765 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.490+0000 7fe5539ac740 1 -- 172.21.15.98:0/3391532341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5631fcebce60 con 0x5631fce46320 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.491+0000 7fe531ffb640 1 -- 172.21.15.98:0/3391532341 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe53400fdf0 con 0x5631fce46320 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.491+0000 7fe531ffb640 1 -- 172.21.15.98:0/3391532341 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe534040060 con 0x5631fce46320 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.491+0000 7fe5539ac740 10 client.18765 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.491+0000 7fe5539ac740 10 client.18765 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.491+0000 7fe5539ac740 20 client.18765 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.491+0000 7fe5539ac740 1 --2- 172.21.15.98:0/3391532341 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5631fcebd460 0x5631fcebf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.491+0000 7fe5539ac740 4 client.18765 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.491+0000 7fe5539ac740 1 -- 172.21.15.98:0/3391532341 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5631fcda86c0 con 0x5631fcebd460 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.491+0000 7fe54a59c640 1 --2- 172.21.15.98:0/3391532341 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5631fcebd460 0x5631fcebf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.491+0000 7fe531ffb640 10 client.18765 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.491+0000 7fe54a59c640 1 --2- 172.21.15.98:0/3391532341 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5631fcebd460 0x5631fcebf850 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ac9c85907d9a4b84 server_cookie=319f926b000dca21 in_seq=0 out_seq=0 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe531ffb640 1 -- 172.21.15.98:0/3391532341 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5631fcda86c0 con 0x5631fcebd460 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe531ffb640 10 client.18765 handle_command_reply: tid=0 2024-05-09T09:01:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe5539ac740 1 client.18765 shutdown 2024-05-09T09:01:27.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe5539ac740 1 -- 172.21.15.98:0/3391532341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 msgr2=0x5631fce4ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe5539ac740 1 --2- 172.21.15.98:0/3391532341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce4ab00 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fe534016040 tx=0x7fe534003690 comp rx=0 tx=0).stop 2024-05-09T09:01:27.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe5539ac740 1 -- 172.21.15.98:0/3391532341 shutdown_connections 2024-05-09T09:01:27.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe5539ac740 1 --2- 172.21.15.98:0/3391532341 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x5631fcebd460 0x5631fcebf850 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe5539ac740 1 --2- 172.21.15.98:0/3391532341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631fce46320 0x5631fce4ab00 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe5539ac740 1 --2- 172.21.15.98:0/3391532341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631fce3bc50 0x5631fce4a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe5539ac740 1 --2- 172.21.15.98:0/3391532341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5631fce3b330 0x5631fce4a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe531ffb640 0 client.18765 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T09:01:27.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe5539ac740 1 -- 172.21.15.98:0/3391532341 >> 172.21.15.98:0/3391532341 conn(0x5631fce37650 msgr2=0x5631fce41840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe5539ac740 1 -- 172.21.15.98:0/3391532341 shutdown_connections 2024-05-09T09:01:27.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe5539ac740 1 -- 172.21.15.98:0/3391532341 wait complete. 2024-05-09T09:01:27.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe5539ac740 20 client.18765 trim_cache size 0 max 16384 2024-05-09T09:01:27.499 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe550c9e640 1 -- 172.21.15.98:0/3895345416 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe5380a2d00 msgr2=0x7fe5380a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe550c9e640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe5380a2d00 0x7fe5380a51c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe5440bcea0 tx=0x7fe54408f160 comp rx=0 tx=0).stop 2024-05-09T09:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe550c9e640 1 -- 172.21.15.98:0/3895345416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c107820 msgr2=0x7fe54c0745b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe550c9e640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c107820 0x7fe54c0745b0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fe544005cc0 tx=0x7fe544003970 comp rx=0 tx=0).stop 2024-05-09T09:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe550c9e640 1 -- 172.21.15.98:0/3895345416 shutdown_connections 2024-05-09T09:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe550c9e640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe5380a2d00 0x7fe5380a51c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe550c9e640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c107820 0x7fe54c0745b0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe550c9e640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe54c1021d0 0x7fe54c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe550c9e640 1 --2- 172.21.15.98:0/3895345416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe54c101810 0x7fe54c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.492+0000 7fe550c9e640 1 -- 172.21.15.98:0/3895345416 >> 172.21.15.98:0/3895345416 conn(0x7fe54c078c30 msgr2=0x7fe54c104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.493+0000 7fe550c9e640 1 -- 172.21.15.98:0/3895345416 shutdown_connections 2024-05-09T09:01:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.493+0000 7fe550c9e640 1 -- 172.21.15.98:0/3895345416 wait complete. 2024-05-09T09:01:27.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T09:01:27.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 -- 172.21.15.98:0/1468990344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c0006a580 msgr2=0x7f0c0010e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 --2- 172.21.15.98:0/1468990344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c0006a580 0x7f0c0010e690 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8009900 tx=0x7f0bf802a480 comp rx=0 tx=0).stop 2024-05-09T09:01:27.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 -- 172.21.15.98:0/1468990344 shutdown_connections 2024-05-09T09:01:27.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 --2- 172.21.15.98:0/1468990344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c0006a580 0x7f0c0010e690 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 --2- 172.21.15.98:0/1468990344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c00108140 0x7f0c00069e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 --2- 172.21.15.98:0/1468990344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c00107820 0x7f0c00107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 -- 172.21.15.98:0/1468990344 >> 172.21.15.98:0/1468990344 conn(0x7f0c00076530 msgr2=0x7f0c00076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 -- 172.21.15.98:0/1468990344 shutdown_connections 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 -- 172.21.15.98:0/1468990344 wait complete. 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 Processor -- start 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 -- start start 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c0006a580 0x7f0c001980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c00107820 0x7f0c00198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00108140 0x7f0c0019c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c00111c50 con 0x7f0c00108140 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0c00111ad0 con 0x7f0c0006a580 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.607+0000 7f0c05563640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0c00111dd0 con 0x7f0c00107820 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0bff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00108140 0x7f0c0019c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0bff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00108140 0x7f0c0019c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47094/0 (socket says 172.21.15.98:47094) 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0bff7fe640 1 -- 172.21.15.98:0/1435167728 learned_addr learned my addr 172.21.15.98:0/1435167728 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0bfe7fc640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c00107820 0x7f0c00198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0bff7fe640 1 -- 172.21.15.98:0/1435167728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c00107820 msgr2=0x7f0c00198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0bff7fe640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c00107820 0x7f0c00198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0bfeffd640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c0006a580 0x7f0c001980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0bff7fe640 1 -- 172.21.15.98:0/1435167728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c0006a580 msgr2=0x7f0c001980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0bff7fe640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c0006a580 0x7f0c001980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0bff7fe640 1 -- 172.21.15.98:0/1435167728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0019d0e0 con 0x7f0c00108140 2024-05-09T09:01:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0bfe7fc640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c00107820 0x7f0c00198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0bfeffd640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c0006a580 0x7f0c001980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0bff7fe640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00108140 0x7f0c0019c9b0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8004770 tx=0x7f0bf8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0be7fff640 1 -- 172.21.15.98:0/1435167728 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf803f070 con 0x7f0c00108140 2024-05-09T09:01:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0be7fff640 1 -- 172.21.15.98:0/1435167728 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bf8031c30 con 0x7f0c00108140 2024-05-09T09:01:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0c05563640 1 -- 172.21.15.98:0/1435167728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c0019d370 con 0x7f0c00108140 2024-05-09T09:01:27.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0be7fff640 1 -- 172.21.15.98:0/1435167728 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8035030 con 0x7f0c00108140 2024-05-09T09:01:27.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.608+0000 7f0c05563640 1 -- 172.21.15.98:0/1435167728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c001a4c60 con 0x7f0c00108140 2024-05-09T09:01:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0be7fff640 1 -- 172.21.15.98:0/1435167728 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0bf8042070 con 0x7f0c00108140 2024-05-09T09:01:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0c08275740 1 Processor -- start 2024-05-09T09:01:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0c08275740 1 -- start start 2024-05-09T09:01:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0c08275740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823330 0x55b1cd823710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0c08275740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd823c50 0x55b1cd82dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0c08275740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd82e320 0x55b1cd830710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0c08275740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b1cd788c40 con 0x55b1cd82e320 2024-05-09T09:01:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0c08275740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b1cd730580 con 0x55b1cd823c50 2024-05-09T09:01:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0c08275740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b1cd784c00 con 0x55b1cd823330 2024-05-09T09:01:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0be7fff640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0be80a2d00 0x7f0be80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0bfe7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd82e320 0x55b1cd830710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0bfe7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd82e320 0x55b1cd830710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47096/0 (socket says 172.21.15.98:47096) 2024-05-09T09:01:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0bfe7fc640 1 -- 172.21.15.98:0/237625717 learned_addr learned my addr 172.21.15.98:0/237625717 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0bff7fe640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0be80a2d00 0x7f0be80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.611+0000 7f0bfeffd640 1 --2- 172.21.15.98:0/237625717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd823c50 0x55b1cd82dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.612+0000 7f0bfe7fc640 1 --2- 172.21.15.98:0/237625717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823330 0x55b1cd823710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.612+0000 7f0be7fff640 1 -- 172.21.15.98:0/1435167728 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f0bf80f0150 con 0x7f0c00108140 2024-05-09T09:01:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.612+0000 7f0bfe7fc640 1 -- 172.21.15.98:0/237625717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823330 msgr2=0x55b1cd823710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.612+0000 7f0bfe7fc640 1 --2- 172.21.15.98:0/237625717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823330 0x55b1cd823710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.612+0000 7f0bfe7fc640 1 -- 172.21.15.98:0/237625717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd823c50 msgr2=0x55b1cd82dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.612+0000 7f0bfe7fc640 1 --2- 172.21.15.98:0/237625717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd823c50 0x55b1cd82dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.612+0000 7f0bfe7fc640 1 -- 172.21.15.98:0/237625717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1cd49c3f0 con 0x55b1cd82e320 2024-05-09T09:01:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.612+0000 7f0bff7fe640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0be80a2d00 0x7f0be80a51c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0bf80b8c30 tx=0x7f0bf80a6d90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.612+0000 7f0bfe7fc640 1 --2- 172.21.15.98:0/237625717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd82e320 0x55b1cd830710 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4007e80 tx=0x7f0bf40073e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5fa52545285c82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.612+0000 7f0be57fa640 1 -- 172.21.15.98:0/237625717 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf401c070 con 0x55b1cd82e320 2024-05-09T09:01:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.612+0000 7f0be57fa640 1 -- 172.21.15.98:0/237625717 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bf4010040 con 0x55b1cd82e320 2024-05-09T09:01:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.612+0000 7f0be57fa640 1 -- 172.21.15.98:0/237625717 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4014730 con 0x55b1cd82e320 2024-05-09T09:01:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 -- 172.21.15.98:0/237625717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd82e320 msgr2=0x55b1cd830710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 --2- 172.21.15.98:0/237625717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd82e320 0x55b1cd830710 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4007e80 tx=0x7f0bf40073e0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 -- 172.21.15.98:0/237625717 shutdown_connections 2024-05-09T09:01:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 --2- 172.21.15.98:0/237625717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd82e320 0x55b1cd830710 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 --2- 172.21.15.98:0/237625717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd823c50 0x55b1cd82dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 --2- 172.21.15.98:0/237625717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823330 0x55b1cd823710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 -- 172.21.15.98:0/237625717 >> 172.21.15.98:0/237625717 conn(0x55b1cd81f650 msgr2=0x55b1cd81fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 -- 172.21.15.98:0/237625717 shutdown_connections 2024-05-09T09:01:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 -- 172.21.15.98:0/237625717 wait complete. 2024-05-09T09:01:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 -- start start 2024-05-09T09:01:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 Processor -- start 2024-05-09T09:01:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd823330 0x55b1cd82b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823c50 0x55b1cd82b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd82e320 0x55b1cd82bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b1cd831110 con 0x55b1cd82e320 2024-05-09T09:01:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b1cd743c40 con 0x55b1cd823330 2024-05-09T09:01:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b1cd831290 con 0x55b1cd823c50 2024-05-09T09:01:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.613+0000 7f0c08275740 10 client.0 _ll_register_callbacks cb 0x55b1cd49bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0bfe7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd82e320 0x55b1cd82bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0bfe7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd82e320 0x55b1cd82bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47104/0 (socket says 172.21.15.98:47104) 2024-05-09T09:01:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0bfe7fc640 1 -- 172.21.15.98:0/1406951590 learned_addr learned my addr 172.21.15.98:0/1406951590 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0bfe7fc640 1 --2- 172.21.15.98:0/1406951590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd823330 0x55b1cd82b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0bfeffd640 1 --2- 172.21.15.98:0/1406951590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823c50 0x55b1cd82b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0bfe7fc640 1 -- 172.21.15.98:0/1406951590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823c50 msgr2=0x55b1cd82b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0bfe7fc640 1 --2- 172.21.15.98:0/1406951590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823c50 0x55b1cd82b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0bfe7fc640 1 -- 172.21.15.98:0/1406951590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd823330 msgr2=0x55b1cd82b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0bfe7fc640 1 --2- 172.21.15.98:0/1406951590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd823330 0x55b1cd82b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0bfe7fc640 1 -- 172.21.15.98:0/1406951590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1cd5667d0 con 0x55b1cd82e320 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0bfe7fc640 1 --2- 172.21.15.98:0/1406951590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd82e320 0x55b1cd82bd70 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4007fd0 tx=0x7f0bf4017f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d987576922e3969 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0be2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0be2ff5640 1 -- 172.21.15.98:0/1406951590 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf401c040 con 0x55b1cd82e320 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0be2ff5640 1 -- 172.21.15.98:0/1406951590 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f0bf4017c40 con 0x55b1cd82e320 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0c08275740 1 -- 172.21.15.98:0/1406951590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0bf4010040 con 0x55b1cd82e320 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0be2ff5640 1 -- 172.21.15.98:0/1406951590 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0bf4033050 con 0x55b1cd82e320 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0c08275740 10 client.18777 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.614+0000 7f0c08275740 1 -- 172.21.15.98:0/1406951590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b1cd83d380 con 0x55b1cd82e320 2024-05-09T09:01:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.615+0000 7f0be2ff5640 1 -- 172.21.15.98:0/1406951590 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bf4010040 con 0x55b1cd82e320 2024-05-09T09:01:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.615+0000 7f0be2ff5640 1 -- 172.21.15.98:0/1406951590 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f0bf4022070 con 0x55b1cd82e320 2024-05-09T09:01:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.615+0000 7f0c08275740 10 client.18777 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.615+0000 7f0c08275740 10 client.18777 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T09:01:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.615+0000 7f0c08275740 20 client.18777 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.615+0000 7f0c08275740 1 --2- 172.21.15.98:0/1406951590 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55b1cd83e730 0x55b1cd840b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.615+0000 7f0c08275740 4 client.18777 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:01:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.615+0000 7f0c08275740 1 -- 172.21.15.98:0/1406951590 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1cd796370 con 0x55b1cd83e730 2024-05-09T09:01:27.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.616+0000 7f0bfeffd640 1 --2- 172.21.15.98:0/1406951590 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55b1cd83e730 0x55b1cd840b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.618+0000 7f0bfeffd640 1 --2- 172.21.15.98:0/1406951590 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55b1cd83e730 0x55b1cd840b20 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b91a7c468f42caf0 server_cookie=22c92f5801625e6b in_seq=0 out_seq=0 2024-05-09T09:01:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.618+0000 7f0be2ff5640 10 client.18777 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T09:01:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.620+0000 7f0be2ff5640 1 -- 172.21.15.98:0/1406951590 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b1cd796370 con 0x55b1cd83e730 2024-05-09T09:01:27.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.620+0000 7f0be2ff5640 10 client.18777 handle_command_reply: tid=0 2024-05-09T09:01:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.620+0000 7f0c08275740 1 client.18777 shutdown 2024-05-09T09:01:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.621+0000 7f0c08275740 1 -- 172.21.15.98:0/1406951590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd82e320 msgr2=0x55b1cd82bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.621+0000 7f0c08275740 1 --2- 172.21.15.98:0/1406951590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd82e320 0x55b1cd82bd70 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4007fd0 tx=0x7f0bf4017f30 comp rx=0 tx=0).stop 2024-05-09T09:01:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.621+0000 7f0c08275740 1 -- 172.21.15.98:0/1406951590 shutdown_connections 2024-05-09T09:01:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.621+0000 7f0c08275740 1 --2- 172.21.15.98:0/1406951590 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55b1cd83e730 0x55b1cd840b20 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.621+0000 7f0c08275740 1 --2- 172.21.15.98:0/1406951590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd82e320 0x55b1cd82bd70 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.621+0000 7f0be2ff5640 0 client.18777 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T09:01:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.621+0000 7f0c08275740 1 --2- 172.21.15.98:0/1406951590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823c50 0x55b1cd82b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.621+0000 7f0c08275740 1 --2- 172.21.15.98:0/1406951590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd823330 0x55b1cd82b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.621+0000 7f0c08275740 1 -- 172.21.15.98:0/1406951590 >> 172.21.15.98:0/1406951590 conn(0x55b1cd81f650 msgr2=0x55b1cd825450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.621+0000 7f0c08275740 1 -- 172.21.15.98:0/1406951590 shutdown_connections 2024-05-09T09:01:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.621+0000 7f0c08275740 1 -- 172.21.15.98:0/1406951590 wait complete. 2024-05-09T09:01:27.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.621+0000 7f0c08275740 20 client.18777 trim_cache size 0 max 16384 2024-05-09T09:01:27.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0c08275740 1 Processor -- start 2024-05-09T09:01:27.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0c08275740 1 -- start start 2024-05-09T09:01:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0c08275740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823330 0x55b1cd81e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0c08275740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd823c50 0x55b1cd83a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0c08275740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd82e320 0x55b1cd825ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0c08275740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c001aeec0 con 0x55b1cd823c50 2024-05-09T09:01:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0c08275740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0c001b0a00 con 0x55b1cd82e320 2024-05-09T09:01:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0c08275740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0c001af590 con 0x55b1cd823330 2024-05-09T09:01:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0bfeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd823c50 0x55b1cd83a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0bfeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd823c50 0x55b1cd83a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47116/0 (socket says 172.21.15.98:47116) 2024-05-09T09:01:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0bfeffd640 1 -- 172.21.15.98:0/3595567081 learned_addr learned my addr 172.21.15.98:0/3595567081 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0bfeffd640 1 -- 172.21.15.98:0/3595567081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823330 msgr2=0x55b1cd81e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0bfeffd640 1 --2- 172.21.15.98:0/3595567081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823330 0x55b1cd81e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0bfeffd640 1 -- 172.21.15.98:0/3595567081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd82e320 msgr2=0x55b1cd825ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0bfeffd640 1 --2- 172.21.15.98:0/3595567081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd82e320 0x55b1cd825ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0bfeffd640 1 -- 172.21.15.98:0/3595567081 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1cd810d20 con 0x55b1cd823c50 2024-05-09T09:01:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0bfeffd640 1 --2- 172.21.15.98:0/3595567081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd823c50 0x55b1cd83a000 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0000f80 tx=0x7f0bf0004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2cf3fff9daaf28c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0be4ff9640 1 -- 172.21.15.98:0/3595567081 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf001d670 con 0x55b1cd823c50 2024-05-09T09:01:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0be4ff9640 1 -- 172.21.15.98:0/3595567081 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bf0023070 con 0x55b1cd823c50 2024-05-09T09:01:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.634+0000 7f0be4ff9640 1 -- 172.21.15.98:0/3595567081 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf001c630 con 0x55b1cd823c50 2024-05-09T09:01:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.635+0000 7f0c08275740 1 -- 172.21.15.98:0/3595567081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd823c50 msgr2=0x55b1cd83a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.635+0000 7f0c08275740 1 --2- 172.21.15.98:0/3595567081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd823c50 0x55b1cd83a000 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0000f80 tx=0x7f0bf0004410 comp rx=0 tx=0).stop 2024-05-09T09:01:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.635+0000 7f0c08275740 1 -- 172.21.15.98:0/3595567081 shutdown_connections 2024-05-09T09:01:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.635+0000 7f0c08275740 1 --2- 172.21.15.98:0/3595567081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd82e320 0x55b1cd825ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.635+0000 7f0c08275740 1 --2- 172.21.15.98:0/3595567081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd823c50 0x55b1cd83a000 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.635+0000 7f0c08275740 1 --2- 172.21.15.98:0/3595567081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823330 0x55b1cd81e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.635+0000 7f0c08275740 1 -- 172.21.15.98:0/3595567081 >> 172.21.15.98:0/3595567081 conn(0x55b1cd81f650 msgr2=0x55b1cd82a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.635+0000 7f0c08275740 1 -- 172.21.15.98:0/3595567081 shutdown_connections 2024-05-09T09:01:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.635+0000 7f0c08275740 1 -- 172.21.15.98:0/3595567081 wait complete. 2024-05-09T09:01:27.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.635+0000 7f0c08275740 1 -- start start 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.635+0000 7f0c08275740 1 Processor -- start 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0c08275740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823330 0x55b1cd832030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0c08275740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd823c50 0x55b1cd832570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0c08275740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd82e320 0x55b1cd832ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0c08275740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c001b0330 con 0x55b1cd823c50 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0c08275740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0c001afc60 con 0x55b1cd82e320 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0c08275740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0c001acec0 con 0x55b1cd823330 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0c08275740 10 client.0 _ll_register_callbacks cb 0x7f0bf401a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0bfeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd823c50 0x55b1cd832570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0bfeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd823c50 0x55b1cd832570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47132/0 (socket says 172.21.15.98:47132) 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0bfeffd640 1 -- 172.21.15.98:0/3614212226 learned_addr learned my addr 172.21.15.98:0/3614212226 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0bfeffd640 1 -- 172.21.15.98:0/3614212226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823330 msgr2=0x55b1cd832030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0bfeffd640 1 --2- 172.21.15.98:0/3614212226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823330 0x55b1cd832030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0bfeffd640 1 -- 172.21.15.98:0/3614212226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd82e320 msgr2=0x55b1cd832ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0bfeffd640 1 --2- 172.21.15.98:0/3614212226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd82e320 0x55b1cd832ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0bfeffd640 1 -- 172.21.15.98:0/3614212226 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1cd8a3bd0 con 0x55b1cd823c50 2024-05-09T09:01:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0bfeffd640 1 --2- 172.21.15.98:0/3614212226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd823c50 0x55b1cd832570 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00049b0 tx=0x7f0bf0003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74280bb811944a94 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0be2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0be2ff5640 1 -- 172.21.15.98:0/3614212226 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf001d880 con 0x55b1cd823c50 2024-05-09T09:01:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0c08275740 1 -- 172.21.15.98:0/3614212226 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0bf0023070 con 0x55b1cd823c50 2024-05-09T09:01:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0be2ff5640 1 -- 172.21.15.98:0/3614212226 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f0bf00125f0 con 0x55b1cd823c50 2024-05-09T09:01:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0be2ff5640 1 -- 172.21.15.98:0/3614212226 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0bf00127f0 con 0x55b1cd823c50 2024-05-09T09:01:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0c08275740 10 client.18783 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.636+0000 7f0c08275740 1 -- 172.21.15.98:0/3614212226 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b1cd8a4ed0 con 0x55b1cd823c50 2024-05-09T09:01:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.637+0000 7f0be2ff5640 1 -- 172.21.15.98:0/3614212226 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bf0023070 con 0x55b1cd823c50 2024-05-09T09:01:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.637+0000 7f0be2ff5640 1 -- 172.21.15.98:0/3614212226 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f0bf002e3a0 con 0x55b1cd823c50 2024-05-09T09:01:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.637+0000 7f0c08275740 10 client.18783 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.637+0000 7f0c08275740 10 client.18783 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T09:01:27.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.637+0000 7f0c08275740 20 client.18783 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.637+0000 7f0c08275740 1 --2- 172.21.15.98:0/3614212226 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55b1cd8a5610 0x55b1cd8a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.637+0000 7f0c08275740 4 client.18783 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:01:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.637+0000 7f0c08275740 1 -- 172.21.15.98:0/3614212226 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b1cd7906c0 con 0x55b1cd8a5610 2024-05-09T09:01:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.637+0000 7f0bfe7fc640 1 --2- 172.21.15.98:0/3614212226 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55b1cd8a5610 0x55b1cd8a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.637+0000 7f0be2ff5640 10 client.18783 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T09:01:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.637+0000 7f0bfe7fc640 1 --2- 172.21.15.98:0/3614212226 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55b1cd8a5610 0x55b1cd8a7a30 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=63f12c514d65f12c server_cookie=41940e3a381514a1 in_seq=0 out_seq=0 2024-05-09T09:01:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0be2ff5640 1 -- 172.21.15.98:0/3614212226 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b1cd7906c0 con 0x55b1cd8a5610 2024-05-09T09:01:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0be2ff5640 10 client.18783 handle_command_reply: tid=0 2024-05-09T09:01:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c08275740 1 client.18783 shutdown 2024-05-09T09:01:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c08275740 1 -- 172.21.15.98:0/3614212226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd823c50 msgr2=0x55b1cd832570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c08275740 1 --2- 172.21.15.98:0/3614212226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd823c50 0x55b1cd832570 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00049b0 tx=0x7f0bf0003eb0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c08275740 1 -- 172.21.15.98:0/3614212226 shutdown_connections 2024-05-09T09:01:27.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c08275740 1 --2- 172.21.15.98:0/3614212226 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55b1cd8a5610 0x55b1cd8a7a30 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c08275740 1 --2- 172.21.15.98:0/3614212226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b1cd82e320 0x55b1cd832ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c08275740 1 --2- 172.21.15.98:0/3614212226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1cd823c50 0x55b1cd832570 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0be2ff5640 0 client.18783 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T09:01:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c08275740 1 --2- 172.21.15.98:0/3614212226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b1cd823330 0x55b1cd832030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c08275740 1 -- 172.21.15.98:0/3614212226 >> 172.21.15.98:0/3614212226 conn(0x55b1cd81f650 msgr2=0x55b1cd82b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c08275740 1 -- 172.21.15.98:0/3614212226 shutdown_connections 2024-05-09T09:01:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c08275740 1 -- 172.21.15.98:0/3614212226 wait complete. 2024-05-09T09:01:27.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c08275740 20 client.18783 trim_cache size 0 max 16384 2024-05-09T09:01:27.644 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:01:27.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c05563640 1 -- 172.21.15.98:0/1435167728 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0be80a2d00 msgr2=0x7f0be80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c05563640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0be80a2d00 0x7f0be80a51c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0bf80b8c30 tx=0x7f0bf80a6d90 comp rx=0 tx=0).stop 2024-05-09T09:01:27.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c05563640 1 -- 172.21.15.98:0/1435167728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00108140 msgr2=0x7f0c0019c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c05563640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00108140 0x7f0c0019c9b0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8004770 tx=0x7f0bf8003c10 comp rx=0 tx=0).stop 2024-05-09T09:01:27.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c05563640 1 -- 172.21.15.98:0/1435167728 shutdown_connections 2024-05-09T09:01:27.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c05563640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0be80a2d00 0x7f0be80a51c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c05563640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c00108140 0x7f0c0019c9b0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c05563640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c00107820 0x7f0c00198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c05563640 1 --2- 172.21.15.98:0/1435167728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c0006a580 0x7f0c001980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c05563640 1 -- 172.21.15.98:0/1435167728 >> 172.21.15.98:0/1435167728 conn(0x7f0c00076530 msgr2=0x7f0c000ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c05563640 1 -- 172.21.15.98:0/1435167728 shutdown_connections 2024-05-09T09:01:27.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.638+0000 7f0c05563640 1 -- 172.21.15.98:0/1435167728 wait complete. 2024-05-09T09:01:27.647 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:27 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2416373337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:01:27.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T09:01:27.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:27 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2416373337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:01:27.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:27 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2416373337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:01:27.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.753+0000 7f79b41c1640 1 -- 172.21.15.98:0/2978185058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac069eb0 msgr2=0x7f79ac06a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.753+0000 7f79b41c1640 1 --2- 172.21.15.98:0/2978185058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac069eb0 0x7f79ac06a330 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f799c009930 tx=0x7f799c02a420 comp rx=0 tx=0).stop 2024-05-09T09:01:27.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.753+0000 7f79b41c1640 1 -- 172.21.15.98:0/2978185058 shutdown_connections 2024-05-09T09:01:27.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.753+0000 7f79b41c1640 1 --2- 172.21.15.98:0/2978185058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79ac06a870 0x7f79ac10e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.753+0000 7f79b41c1640 1 --2- 172.21.15.98:0/2978185058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac069eb0 0x7f79ac06a330 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.753+0000 7f79b41c1640 1 --2- 172.21.15.98:0/2978185058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79ac107820 0x7f79ac107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.753+0000 7f79b41c1640 1 -- 172.21.15.98:0/2978185058 >> 172.21.15.98:0/2978185058 conn(0x7f79ac077790 msgr2=0x7f79ac077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.753+0000 7f79b41c1640 1 -- 172.21.15.98:0/2978185058 shutdown_connections 2024-05-09T09:01:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.753+0000 7f79b41c1640 1 -- 172.21.15.98:0/2978185058 wait complete. 2024-05-09T09:01:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b41c1640 1 Processor -- start 2024-05-09T09:01:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b41c1640 1 -- start start 2024-05-09T09:01:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b41c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac069eb0 0x7f79ac100760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b41c1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79ac06a870 0x7f79ac100ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b41c1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79ac107820 0x7f79ac1011e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b41c1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac111c80 con 0x7f79ac069eb0 2024-05-09T09:01:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b41c1640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac111b00 con 0x7f79ac06a870 2024-05-09T09:01:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b41c1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac111e00 con 0x7f79ac107820 2024-05-09T09:01:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b1f5d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac069eb0 0x7f79ac100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b1f5d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac069eb0 0x7f79ac100760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47138/0 (socket says 172.21.15.98:47138) 2024-05-09T09:01:27.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b1f5d640 1 -- 172.21.15.98:0/85271762 learned_addr learned my addr 172.21.15.98:0/85271762 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b175c640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79ac06a870 0x7f79ac100ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b275e640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79ac107820 0x7f79ac1011e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b1f5d640 1 -- 172.21.15.98:0/85271762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79ac107820 msgr2=0x7f79ac1011e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b1f5d640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79ac107820 0x7f79ac1011e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b1f5d640 1 -- 172.21.15.98:0/85271762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79ac06a870 msgr2=0x7f79ac100ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b1f5d640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79ac06a870 0x7f79ac100ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b1f5d640 1 -- 172.21.15.98:0/85271762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ac1a66c0 con 0x7f79ac069eb0 2024-05-09T09:01:27.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b175c640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79ac06a870 0x7f79ac100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:27.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.754+0000 7f79b1f5d640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac069eb0 0x7f79ac100760 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f79a800c970 tx=0x7f79a800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.755+0000 7f79a2ffd640 1 -- 172.21.15.98:0/85271762 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a8007bb0 con 0x7f79ac069eb0 2024-05-09T09:01:27.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.755+0000 7f79a2ffd640 1 -- 172.21.15.98:0/85271762 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79a8007d50 con 0x7f79ac069eb0 2024-05-09T09:01:27.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.755+0000 7f79a2ffd640 1 -- 172.21.15.98:0/85271762 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a800f450 con 0x7f79ac069eb0 2024-05-09T09:01:27.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.755+0000 7f79b41c1640 1 -- 172.21.15.98:0/85271762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79ac1a69b0 con 0x7f79ac069eb0 2024-05-09T09:01:27.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.755+0000 7f79b41c1640 1 -- 172.21.15.98:0/85271762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79ac1a6e80 con 0x7f79ac069eb0 2024-05-09T09:01:27.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.757+0000 7f79a2ffd640 1 -- 172.21.15.98:0/85271762 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f79a8005020 con 0x7f79ac069eb0 2024-05-09T09:01:27.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.757+0000 7f79b6fae740 1 Processor -- start 2024-05-09T09:01:27.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.757+0000 7f79b6fae740 1 -- start start 2024-05-09T09:01:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b6fae740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d576431330 0x55d576431710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b6fae740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431c50 0x55d57643bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b6fae740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d57643c320 0x55d57643e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b6fae740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d576396c40 con 0x55d57643c320 2024-05-09T09:01:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b6fae740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d57633e580 con 0x55d576431c50 2024-05-09T09:01:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b6fae740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d576392c00 con 0x55d576431330 2024-05-09T09:01:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79a2ffd640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f79980a2d00 0x7f79980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b175c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d57643c320 0x55d57643e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b175c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d57643c320 0x55d57643e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47144/0 (socket says 172.21.15.98:47144) 2024-05-09T09:01:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b175c640 1 -- 172.21.15.98:0/995945261 learned_addr learned my addr 172.21.15.98:0/995945261 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b1f5d640 1 --2- 172.21.15.98:0/995945261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431c50 0x55d57643bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b275e640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f79980a2d00 0x7f79980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b275e640 1 --2- 172.21.15.98:0/995945261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d576431330 0x55d576431710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79a2ffd640 1 -- 172.21.15.98:0/85271762 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f79a80cf680 con 0x7f79ac069eb0 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b175c640 1 -- 172.21.15.98:0/995945261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d576431330 msgr2=0x55d576431710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b175c640 1 --2- 172.21.15.98:0/995945261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d576431330 0x55d576431710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b175c640 1 -- 172.21.15.98:0/995945261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431c50 msgr2=0x55d57643bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b175c640 1 --2- 172.21.15.98:0/995945261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431c50 0x55d57643bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b175c640 1 -- 172.21.15.98:0/995945261 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5760aa3f0 con 0x55d57643c320 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.758+0000 7f79b175c640 1 --2- 172.21.15.98:0/995945261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d57643c320 0x55d57643e710 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f799c02aa10 tx=0x7f799c004200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa16fc080da93b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.759+0000 7f79b275e640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f79980a2d00 0x7f79980a51c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009fd0 tx=0x7f79a4009440 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.759+0000 7f7997fff640 1 -- 172.21.15.98:0/995945261 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c035030 con 0x55d57643c320 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.759+0000 7f7997fff640 1 -- 172.21.15.98:0/995945261 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f799c005520 con 0x55d57643c320 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.759+0000 7f7997fff640 1 -- 172.21.15.98:0/995945261 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c003850 con 0x55d57643c320 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.759+0000 7f79b6fae740 1 -- 172.21.15.98:0/995945261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d57643c320 msgr2=0x55d57643e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.759+0000 7f79b6fae740 1 --2- 172.21.15.98:0/995945261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d57643c320 0x55d57643e710 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f799c02aa10 tx=0x7f799c004200 comp rx=0 tx=0).stop 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.759+0000 7f79b6fae740 1 -- 172.21.15.98:0/995945261 shutdown_connections 2024-05-09T09:01:27.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.759+0000 7f79b6fae740 1 --2- 172.21.15.98:0/995945261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d57643c320 0x55d57643e710 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.759+0000 7f79b6fae740 1 --2- 172.21.15.98:0/995945261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431c50 0x55d57643bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.759+0000 7f79b6fae740 1 --2- 172.21.15.98:0/995945261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d576431330 0x55d576431710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.759+0000 7f79b6fae740 1 -- 172.21.15.98:0/995945261 >> 172.21.15.98:0/995945261 conn(0x55d57642d650 msgr2=0x55d57642da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.759+0000 7f79b6fae740 1 -- 172.21.15.98:0/995945261 shutdown_connections 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.759+0000 7f79b6fae740 1 -- 172.21.15.98:0/995945261 wait complete. 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.760+0000 7f79b6fae740 1 -- start start 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.760+0000 7f79b6fae740 1 Processor -- start 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.760+0000 7f79b6fae740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431330 0x55d576439220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.760+0000 7f79b6fae740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431c50 0x55d576439760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.760+0000 7f79b6fae740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d57643c320 0x55d576439ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.760+0000 7f79b6fae740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d57643f0b0 con 0x55d576431c50 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.760+0000 7f79b6fae740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d576351c40 con 0x55d576431330 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.760+0000 7f79b6fae740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d57643f230 con 0x55d57643c320 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.760+0000 7f79b6fae740 10 client.0 _ll_register_callbacks cb 0x55d5760a9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:27.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.760+0000 7f79b175c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431c50 0x55d576439760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.760+0000 7f79b175c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431c50 0x55d576439760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47152/0 (socket says 172.21.15.98:47152) 2024-05-09T09:01:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.760+0000 7f79b175c640 1 -- 172.21.15.98:0/2203476195 learned_addr learned my addr 172.21.15.98:0/2203476195 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.760+0000 7f79b1f5d640 1 --2- 172.21.15.98:0/2203476195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431330 0x55d576439220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79b275e640 1 --2- 172.21.15.98:0/2203476195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d57643c320 0x55d576439ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79b175c640 1 -- 172.21.15.98:0/2203476195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d57643c320 msgr2=0x55d576439ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79b175c640 1 --2- 172.21.15.98:0/2203476195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d57643c320 0x55d576439ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79b175c640 1 -- 172.21.15.98:0/2203476195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431330 msgr2=0x55d576439220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79b175c640 1 --2- 172.21.15.98:0/2203476195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431330 0x55d576439220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79b175c640 1 -- 172.21.15.98:0/2203476195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5761747d0 con 0x55d576431c50 2024-05-09T09:01:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79b1f5d640 1 --2- 172.21.15.98:0/2203476195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431330 0x55d576439220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79b175c640 1 --2- 172.21.15.98:0/2203476195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431c50 0x55d576439760 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f799c02a940 tx=0x7f799c004b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c1ceedd6ebade5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79957fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79957fa640 1 -- 172.21.15.98:0/2203476195 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c035070 con 0x55d576431c50 2024-05-09T09:01:27.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79957fa640 1 -- 172.21.15.98:0/2203476195 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f799c005010 con 0x55d576431c50 2024-05-09T09:01:27.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79b6fae740 1 -- 172.21.15.98:0/2203476195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f799c005520 con 0x55d576431c50 2024-05-09T09:01:27.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79957fa640 1 -- 172.21.15.98:0/2203476195 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f799c005210 con 0x55d576431c50 2024-05-09T09:01:27.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79b6fae740 10 client.18795 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:27.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79b6fae740 1 -- 172.21.15.98:0/2203476195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d57644b220 con 0x55d576431c50 2024-05-09T09:01:27.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79957fa640 1 -- 172.21.15.98:0/2203476195 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f799c005520 con 0x55d576431c50 2024-05-09T09:01:27.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79957fa640 1 -- 172.21.15.98:0/2203476195 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f799c031740 con 0x55d576431c50 2024-05-09T09:01:27.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79b6fae740 10 client.18795 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:27.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79b6fae740 10 client.18795 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T09:01:27.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.761+0000 7f79b6fae740 20 client.18795 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:27.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.762+0000 7f79b6fae740 1 --2- 172.21.15.98:0/2203476195 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d57644c570 0x55d57644e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.762+0000 7f79b6fae740 4 client.18795 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:01:27.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.762+0000 7f79b6fae740 1 -- 172.21.15.98:0/2203476195 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5763a4370 con 0x55d57644c570 2024-05-09T09:01:27.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.762+0000 7f79b1f5d640 1 --2- 172.21.15.98:0/2203476195 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d57644c570 0x55d57644e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.762+0000 7f79957fa640 10 client.18795 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T09:01:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.762+0000 7f79b1f5d640 1 --2- 172.21.15.98:0/2203476195 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d57644c570 0x55d57644e960 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=587cb1d4e30cc7ef server_cookie=9bf993a2ccb381a7 in_seq=0 out_seq=0 2024-05-09T09:01:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79957fa640 1 -- 172.21.15.98:0/2203476195 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d5763a4370 con 0x55d57644c570 2024-05-09T09:01:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79957fa640 10 client.18795 handle_command_reply: tid=0 2024-05-09T09:01:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79b6fae740 1 client.18795 shutdown 2024-05-09T09:01:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79b6fae740 1 -- 172.21.15.98:0/2203476195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431c50 msgr2=0x55d576439760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79b6fae740 1 --2- 172.21.15.98:0/2203476195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431c50 0x55d576439760 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f799c02a940 tx=0x7f799c004b10 comp rx=0 tx=0).stop 2024-05-09T09:01:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79b6fae740 1 -- 172.21.15.98:0/2203476195 shutdown_connections 2024-05-09T09:01:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79b6fae740 1 --2- 172.21.15.98:0/2203476195 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d57644c570 0x55d57644e960 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79b6fae740 1 --2- 172.21.15.98:0/2203476195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d57643c320 0x55d576439ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79b6fae740 1 --2- 172.21.15.98:0/2203476195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431c50 0x55d576439760 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79b6fae740 1 --2- 172.21.15.98:0/2203476195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431330 0x55d576439220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79957fa640 0 client.18795 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T09:01:27.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79b6fae740 1 -- 172.21.15.98:0/2203476195 >> 172.21.15.98:0/2203476195 conn(0x55d57642d650 msgr2=0x55d576433450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79b6fae740 1 -- 172.21.15.98:0/2203476195 shutdown_connections 2024-05-09T09:01:27.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79b6fae740 1 -- 172.21.15.98:0/2203476195 wait complete. 2024-05-09T09:01:27.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.763+0000 7f79b6fae740 20 client.18795 trim_cache size 0 max 16384 2024-05-09T09:01:27.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.776+0000 7f79b6fae740 1 Processor -- start 2024-05-09T09:01:27.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.776+0000 7f79b6fae740 1 -- start start 2024-05-09T09:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.776+0000 7f79b6fae740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431330 0x55d5764383c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.776+0000 7f79b6fae740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431c50 0x55d5764364e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b6fae740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d57643c320 0x55d576436a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b6fae740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac1b1010 con 0x55d576431c50 2024-05-09T09:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b6fae740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac1b2b50 con 0x55d576431330 2024-05-09T09:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b6fae740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac1b16e0 con 0x55d57643c320 2024-05-09T09:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b175c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431c50 0x55d5764364e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b175c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431c50 0x55d5764364e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47154/0 (socket says 172.21.15.98:47154) 2024-05-09T09:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b175c640 1 -- 172.21.15.98:0/11263332 learned_addr learned my addr 172.21.15.98:0/11263332 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b275e640 1 --2- 172.21.15.98:0/11263332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d57643c320 0x55d576436a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b175c640 1 -- 172.21.15.98:0/11263332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d57643c320 msgr2=0x55d576436a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b175c640 1 --2- 172.21.15.98:0/11263332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d57643c320 0x55d576436a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b175c640 1 -- 172.21.15.98:0/11263332 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431330 msgr2=0x55d5764383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b175c640 1 --2- 172.21.15.98:0/11263332 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431330 0x55d5764383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b175c640 1 -- 172.21.15.98:0/11263332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d576431a80 con 0x55d576431c50 2024-05-09T09:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b275e640 1 --2- 172.21.15.98:0/11263332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d57643c320 0x55d576436a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79b175c640 1 --2- 172.21.15.98:0/11263332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431c50 0x55d5764364e0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f799c02a460 tx=0x7f799c0051a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f45a5258dc138f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79977fe640 1 -- 172.21.15.98:0/11263332 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c035040 con 0x55d576431c50 2024-05-09T09:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79977fe640 1 -- 172.21.15.98:0/11263332 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f799c004b90 con 0x55d576431c50 2024-05-09T09:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.777+0000 7f79977fe640 1 -- 172.21.15.98:0/11263332 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c003ac0 con 0x55d576431c50 2024-05-09T09:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.778+0000 7f79b6fae740 1 -- 172.21.15.98:0/11263332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431c50 msgr2=0x55d5764364e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.778+0000 7f79b6fae740 1 --2- 172.21.15.98:0/11263332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431c50 0x55d5764364e0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f799c02a460 tx=0x7f799c0051a0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.778+0000 7f79b6fae740 1 -- 172.21.15.98:0/11263332 shutdown_connections 2024-05-09T09:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.778+0000 7f79b6fae740 1 --2- 172.21.15.98:0/11263332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d57643c320 0x55d576436a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.778+0000 7f79b6fae740 1 --2- 172.21.15.98:0/11263332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431c50 0x55d5764364e0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.778+0000 7f79b6fae740 1 --2- 172.21.15.98:0/11263332 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d576431330 0x55d5764383c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.778+0000 7f79b6fae740 1 -- 172.21.15.98:0/11263332 >> 172.21.15.98:0/11263332 conn(0x55d57642d650 msgr2=0x55d57642da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.778+0000 7f79b6fae740 1 -- 172.21.15.98:0/11263332 shutdown_connections 2024-05-09T09:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.778+0000 7f79b6fae740 1 -- 172.21.15.98:0/11263332 wait complete. 2024-05-09T09:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.778+0000 7f79b6fae740 1 -- start start 2024-05-09T09:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.778+0000 7f79b6fae740 1 Processor -- start 2024-05-09T09:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b6fae740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431330 0x55d57644f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b6fae740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d576431c50 0x55d57644fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b6fae740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d57643c320 0x55d5764500a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b6fae740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac1b2480 con 0x55d576431330 2024-05-09T09:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b6fae740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac1b1db0 con 0x55d57643c320 2024-05-09T09:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b6fae740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac1af010 con 0x55d576431c50 2024-05-09T09:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b6fae740 10 client.0 _ll_register_callbacks cb 0x7f79a809b560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b1f5d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431330 0x55d57644f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b1f5d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431330 0x55d57644f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47170/0 (socket says 172.21.15.98:47170) 2024-05-09T09:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b1f5d640 1 -- 172.21.15.98:0/201088200 learned_addr learned my addr 172.21.15.98:0/201088200 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b275e640 1 --2- 172.21.15.98:0/201088200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d57643c320 0x55d5764500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b1f5d640 1 -- 172.21.15.98:0/201088200 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d576431c50 msgr2=0x55d57644fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b1f5d640 1 --2- 172.21.15.98:0/201088200 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d576431c50 0x55d57644fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b1f5d640 1 -- 172.21.15.98:0/201088200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d57643c320 msgr2=0x55d5764500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b1f5d640 1 --2- 172.21.15.98:0/201088200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d57643c320 0x55d5764500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b1f5d640 1 -- 172.21.15.98:0/201088200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5764b19a0 con 0x55d576431330 2024-05-09T09:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b1f5d640 1 --2- 172.21.15.98:0/201088200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431330 0x55d57644f620 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f79a8007fd0 tx=0x7f79a806a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34609ef6957d6ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79957fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79957fa640 1 -- 172.21.15.98:0/201088200 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a801d040 con 0x55d576431330 2024-05-09T09:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79957fa640 1 -- 172.21.15.98:0/201088200 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f79a80860b0 con 0x55d576431330 2024-05-09T09:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b6fae740 1 -- 172.21.15.98:0/201088200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f799c004b90 con 0x55d576431330 2024-05-09T09:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79957fa640 1 -- 172.21.15.98:0/201088200 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f79a809b790 con 0x55d576431330 2024-05-09T09:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b6fae740 10 client.18801 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79b6fae740 1 -- 172.21.15.98:0/201088200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d5764b2c70 con 0x55d576431330 2024-05-09T09:01:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.779+0000 7f79957fa640 1 -- 172.21.15.98:0/201088200 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f799c004b90 con 0x55d576431330 2024-05-09T09:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.780+0000 7f79957fa640 1 -- 172.21.15.98:0/201088200 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f79a8005310 con 0x55d576431330 2024-05-09T09:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.780+0000 7f79b6fae740 10 client.18801 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.780+0000 7f79b6fae740 10 client.18801 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T09:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.780+0000 7f79b6fae740 20 client.18801 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.780+0000 7f79b6fae740 1 --2- 172.21.15.98:0/201088200 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d5764b33c0 0x55d5764b5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.780+0000 7f79b6fae740 4 client.18801 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.780+0000 7f79b6fae740 1 -- 172.21.15.98:0/201088200 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d57639e6c0 con 0x55d5764b33c0 2024-05-09T09:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.780+0000 7f79b175c640 1 --2- 172.21.15.98:0/201088200 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d5764b33c0 0x55d5764b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.780+0000 7f79957fa640 10 client.18801 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T09:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.780+0000 7f79b175c640 1 --2- 172.21.15.98:0/201088200 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d5764b33c0 0x55d5764b5800 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5af3c5128850d2bd server_cookie=17a286507fb65eaa in_seq=0 out_seq=0 2024-05-09T09:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.780+0000 7f79957fa640 1 -- 172.21.15.98:0/201088200 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d57639e6c0 con 0x55d5764b33c0 2024-05-09T09:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.780+0000 7f79957fa640 10 client.18801 handle_command_reply: tid=0 2024-05-09T09:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.780+0000 7f79b6fae740 1 client.18801 shutdown 2024-05-09T09:01:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b6fae740 1 -- 172.21.15.98:0/201088200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431330 msgr2=0x55d57644f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b6fae740 1 --2- 172.21.15.98:0/201088200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431330 0x55d57644f620 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f79a8007fd0 tx=0x7f79a806a6c0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b6fae740 1 -- 172.21.15.98:0/201088200 shutdown_connections 2024-05-09T09:01:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b6fae740 1 --2- 172.21.15.98:0/201088200 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55d5764b33c0 0x55d5764b5800 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b6fae740 1 --2- 172.21.15.98:0/201088200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d57643c320 0x55d5764500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b6fae740 1 --2- 172.21.15.98:0/201088200 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d576431c50 0x55d57644fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b6fae740 1 --2- 172.21.15.98:0/201088200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d576431330 0x55d57644f620 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79957fa640 0 client.18801 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T09:01:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b6fae740 1 -- 172.21.15.98:0/201088200 >> 172.21.15.98:0/201088200 conn(0x55d57642d650 msgr2=0x55d5764391c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b6fae740 1 -- 172.21.15.98:0/201088200 shutdown_connections 2024-05-09T09:01:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b6fae740 1 -- 172.21.15.98:0/201088200 wait complete. 2024-05-09T09:01:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b6fae740 20 client.18801 trim_cache size 0 max 16384 2024-05-09T09:01:27.788 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:01:27.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b41c1640 1 -- 172.21.15.98:0/85271762 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f79980a2d00 msgr2=0x7f79980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b41c1640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f79980a2d00 0x7f79980a51c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009fd0 tx=0x7f79a4009440 comp rx=0 tx=0).stop 2024-05-09T09:01:27.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b41c1640 1 -- 172.21.15.98:0/85271762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac069eb0 msgr2=0x7f79ac100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b41c1640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac069eb0 0x7f79ac100760 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f79a800c970 tx=0x7f79a800ce30 comp rx=0 tx=0).stop 2024-05-09T09:01:27.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b41c1640 1 -- 172.21.15.98:0/85271762 shutdown_connections 2024-05-09T09:01:27.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b41c1640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f79980a2d00 0x7f79980a51c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b41c1640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79ac107820 0x7f79ac1011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b41c1640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79ac06a870 0x7f79ac100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b41c1640 1 --2- 172.21.15.98:0/85271762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac069eb0 0x7f79ac100760 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b41c1640 1 -- 172.21.15.98:0/85271762 >> 172.21.15.98:0/85271762 conn(0x7f79ac077790 msgr2=0x7f79ac0fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b41c1640 1 -- 172.21.15.98:0/85271762 shutdown_connections 2024-05-09T09:01:27.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.781+0000 7f79b41c1640 1 -- 172.21.15.98:0/85271762 wait complete. 2024-05-09T09:01:27.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T09:01:27.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.896+0000 7fad9499e640 1 -- 172.21.15.98:0/637318895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad9006a750 msgr2=0x7fad9010e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.896+0000 7fad9499e640 1 --2- 172.21.15.98:0/637318895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad9006a750 0x7fad9010e6c0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fad88009ca0 tx=0x7fad8802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.896+0000 7fad9499e640 1 -- 172.21.15.98:0/637318895 shutdown_connections 2024-05-09T09:01:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.896+0000 7fad9499e640 1 --2- 172.21.15.98:0/637318895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad9006a750 0x7fad9010e6c0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.896+0000 7fad9499e640 1 --2- 172.21.15.98:0/637318895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad90069db0 0x7fad9006a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.896+0000 7fad9499e640 1 --2- 172.21.15.98:0/637318895 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad90107870 0x7fad90107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.896+0000 7fad9499e640 1 -- 172.21.15.98:0/637318895 >> 172.21.15.98:0/637318895 conn(0x7fad90077500 msgr2=0x7fad90077910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.896+0000 7fad9499e640 1 -- 172.21.15.98:0/637318895 shutdown_connections 2024-05-09T09:01:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.896+0000 7fad9499e640 1 -- 172.21.15.98:0/637318895 wait complete. 2024-05-09T09:01:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad9499e640 1 Processor -- start 2024-05-09T09:01:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad9499e640 1 -- start start 2024-05-09T09:01:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad9499e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad90069db0 0x7fad901980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad9499e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad9006a750 0x7fad90198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad9499e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad90107870 0x7fad9019c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad9499e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad90111c50 con 0x7fad90107870 2024-05-09T09:01:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad9499e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fad90111ad0 con 0x7fad9006a750 2024-05-09T09:01:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad9499e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fad90111dd0 con 0x7fad90069db0 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad8ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad90107870 0x7fad9019c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad8ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad90107870 0x7fad9019c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47186/0 (socket says 172.21.15.98:47186) 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad8ffff640 1 -- 172.21.15.98:0/471224369 learned_addr learned my addr 172.21.15.98:0/471224369 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad86ffd640 1 --2- 172.21.15.98:0/471224369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad9006a750 0x7fad90198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad8f7fe640 1 --2- 172.21.15.98:0/471224369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad90069db0 0x7fad901980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad8ffff640 1 -- 172.21.15.98:0/471224369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad90069db0 msgr2=0x7fad901980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad8ffff640 1 --2- 172.21.15.98:0/471224369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad90069db0 0x7fad901980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad8ffff640 1 -- 172.21.15.98:0/471224369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad9006a750 msgr2=0x7fad90198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad8ffff640 1 --2- 172.21.15.98:0/471224369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad9006a750 0x7fad90198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.897+0000 7fad8ffff640 1 -- 172.21.15.98:0/471224369 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad9019d0c0 con 0x7fad90107870 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.898+0000 7fad8ffff640 1 --2- 172.21.15.98:0/471224369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad90107870 0x7fad9019c990 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fad88006010 tx=0x7fad88003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.898+0000 7fad8d7fa640 1 -- 172.21.15.98:0/471224369 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad88031650 con 0x7fad90107870 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.898+0000 7fad8d7fa640 1 -- 172.21.15.98:0/471224369 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad880317f0 con 0x7fad90107870 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.898+0000 7fad8d7fa640 1 -- 172.21.15.98:0/471224369 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad88030940 con 0x7fad90107870 2024-05-09T09:01:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.898+0000 7fad9499e640 1 -- 172.21.15.98:0/471224369 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad9019d350 con 0x7fad90107870 2024-05-09T09:01:27.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.898+0000 7fad9499e640 1 -- 172.21.15.98:0/471224369 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad901a4c40 con 0x7fad90107870 2024-05-09T09:01:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.900+0000 7fad8d7fa640 1 -- 172.21.15.98:0/471224369 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fad88030ae0 con 0x7fad90107870 2024-05-09T09:01:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad989ec740 1 Processor -- start 2024-05-09T09:01:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad989ec740 1 -- start start 2024-05-09T09:01:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad989ec740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e3693d0 0x55a62e3697b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad989ec740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e369cf0 0x55a62e3783b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad989ec740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e37ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad989ec740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a62e2cec40 con 0x55a62e3788f0 2024-05-09T09:01:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad989ec740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a62e276580 con 0x55a62e369cf0 2024-05-09T09:01:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad989ec740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a62e2cac00 con 0x55a62e3693d0 2024-05-09T09:01:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad8d7fa640 1 --2- 172.21.15.98:0/471224369 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fad7c0a2d00 0x7fad7c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad86ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e37ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad86ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e37ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47190/0 (socket says 172.21.15.98:47190) 2024-05-09T09:01:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad86ffd640 1 -- 172.21.15.98:0/2634480054 learned_addr learned my addr 172.21.15.98:0/2634480054 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad86ffd640 1 --2- 172.21.15.98:0/2634480054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e3693d0 0x55a62e3697b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad8ffff640 1 --2- 172.21.15.98:0/471224369 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fad7c0a2d00 0x7fad7c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad8f7fe640 1 --2- 172.21.15.98:0/2634480054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e369cf0 0x55a62e3783b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad8d7fa640 1 -- 172.21.15.98:0/471224369 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fad880f0ac0 con 0x7fad90107870 2024-05-09T09:01:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad86ffd640 1 -- 172.21.15.98:0/2634480054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e3693d0 msgr2=0x55a62e3697b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad86ffd640 1 --2- 172.21.15.98:0/2634480054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e3693d0 0x55a62e3697b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad86ffd640 1 -- 172.21.15.98:0/2634480054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e369cf0 msgr2=0x55a62e3783b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad86ffd640 1 --2- 172.21.15.98:0/2634480054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e369cf0 0x55a62e3783b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.901+0000 7fad86ffd640 1 -- 172.21.15.98:0/2634480054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a62dfe23f0 con 0x55a62e3788f0 2024-05-09T09:01:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.902+0000 7fad8f7fe640 1 --2- 172.21.15.98:0/2634480054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e369cf0 0x55a62e3783b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.902+0000 7fad86ffd640 1 --2- 172.21.15.98:0/2634480054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e37ace0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fad780127c0 tx=0x7fad7800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0fedf5215f64672 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.902+0000 7fad8ffff640 1 --2- 172.21.15.98:0/471224369 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fad7c0a2d00 0x7fad7c0a51c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fad880edfc0 tx=0x7fad8808eee0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.902+0000 7fad85ffb640 1 -- 172.21.15.98:0/2634480054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78012e00 con 0x55a62e3788f0 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.902+0000 7fad85ffb640 1 -- 172.21.15.98:0/2634480054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad78010c50 con 0x55a62e3788f0 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.902+0000 7fad85ffb640 1 -- 172.21.15.98:0/2634480054 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7801eb50 con 0x55a62e3788f0 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 -- 172.21.15.98:0/2634480054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 msgr2=0x55a62e37ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 --2- 172.21.15.98:0/2634480054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e37ace0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fad780127c0 tx=0x7fad7800b7c0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 -- 172.21.15.98:0/2634480054 shutdown_connections 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 --2- 172.21.15.98:0/2634480054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e37ace0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 --2- 172.21.15.98:0/2634480054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e369cf0 0x55a62e3783b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 --2- 172.21.15.98:0/2634480054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e3693d0 0x55a62e3697b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 -- 172.21.15.98:0/2634480054 >> 172.21.15.98:0/2634480054 conn(0x55a62e365810 msgr2=0x55a62e365bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 -- 172.21.15.98:0/2634480054 shutdown_connections 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 -- 172.21.15.98:0/2634480054 wait complete. 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 -- start start 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 Processor -- start 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 0x55a62e375820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e369cf0 0x55a62e375d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e3762a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a62e37b730 con 0x55a62e3788f0 2024-05-09T09:01:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a62e289c40 con 0x55a62e3693d0 2024-05-09T09:01:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a62e37b8b0 con 0x55a62e369cf0 2024-05-09T09:01:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad989ec740 10 client.0 _ll_register_callbacks cb 0x55a62dfe1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad86ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e3762a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad86ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e3762a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47198/0 (socket says 172.21.15.98:47198) 2024-05-09T09:01:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad86ffd640 1 -- 172.21.15.98:0/1995814145 learned_addr learned my addr 172.21.15.98:0/1995814145 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.903+0000 7fad86ffd640 1 --2- 172.21.15.98:0/1995814145 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 0x55a62e375820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad86ffd640 1 -- 172.21.15.98:0/1995814145 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e369cf0 msgr2=0x55a62e375d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad86ffd640 1 --2- 172.21.15.98:0/1995814145 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e369cf0 0x55a62e375d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad86ffd640 1 -- 172.21.15.98:0/1995814145 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 msgr2=0x55a62e375820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad86ffd640 1 --2- 172.21.15.98:0/1995814145 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 0x55a62e375820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad86ffd640 1 -- 172.21.15.98:0/1995814145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a62e0ac7d0 con 0x55a62e3788f0 2024-05-09T09:01:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad86ffd640 1 --2- 172.21.15.98:0/1995814145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e3762a0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fad7801e450 tx=0x7fad7801e6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c12f2834f93c2f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad6effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad6effd640 1 -- 172.21.15.98:0/1995814145 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7802f060 con 0x55a62e3788f0 2024-05-09T09:01:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad6effd640 1 -- 172.21.15.98:0/1995814145 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fad78012e00 con 0x55a62e3788f0 2024-05-09T09:01:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad989ec740 1 -- 172.21.15.98:0/1995814145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad78010c50 con 0x55a62e3788f0 2024-05-09T09:01:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad6effd640 1 -- 172.21.15.98:0/1995814145 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fad78010e00 con 0x55a62e3788f0 2024-05-09T09:01:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad989ec740 10 client.18813 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad989ec740 1 -- 172.21.15.98:0/1995814145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a62e387970 con 0x55a62e3788f0 2024-05-09T09:01:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad6effd640 1 -- 172.21.15.98:0/1995814145 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad78010c50 con 0x55a62e3788f0 2024-05-09T09:01:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad6effd640 1 -- 172.21.15.98:0/1995814145 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fad7801f6c0 con 0x55a62e3788f0 2024-05-09T09:01:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad989ec740 10 client.18813 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad989ec740 10 client.18813 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:01:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad989ec740 20 client.18813 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:27.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad989ec740 1 --2- 172.21.15.98:0/1995814145 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55a62e388dd0 0x55a62e38b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad989ec740 4 client.18813 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:01:27.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.904+0000 7fad989ec740 1 -- 172.21.15.98:0/1995814145 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a62e2dc370 con 0x55a62e388dd0 2024-05-09T09:01:27.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.905+0000 7fad8f7fe640 1 --2- 172.21.15.98:0/1995814145 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55a62e388dd0 0x55a62e38b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.905+0000 7fad6effd640 10 client.18813 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T09:01:27.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.905+0000 7fad8f7fe640 1 --2- 172.21.15.98:0/1995814145 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55a62e388dd0 0x55a62e38b1c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8788c93eac877e22 server_cookie=508bfaf469e30165 in_seq=0 out_seq=0 2024-05-09T09:01:27.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad6effd640 1 -- 172.21.15.98:0/1995814145 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a62e2dc370 con 0x55a62e388dd0 2024-05-09T09:01:27.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad6effd640 10 client.18813 handle_command_reply: tid=0 2024-05-09T09:01:27.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad989ec740 1 client.18813 shutdown 2024-05-09T09:01:27.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad989ec740 1 -- 172.21.15.98:0/1995814145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 msgr2=0x55a62e3762a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad989ec740 1 --2- 172.21.15.98:0/1995814145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e3762a0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fad7801e450 tx=0x7fad7801e6f0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad989ec740 1 -- 172.21.15.98:0/1995814145 shutdown_connections 2024-05-09T09:01:27.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad989ec740 1 --2- 172.21.15.98:0/1995814145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e3762a0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad989ec740 1 --2- 172.21.15.98:0/1995814145 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e369cf0 0x55a62e375d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad989ec740 1 --2- 172.21.15.98:0/1995814145 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55a62e388dd0 0x55a62e38b1c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad989ec740 1 --2- 172.21.15.98:0/1995814145 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 0x55a62e375820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad989ec740 1 -- 172.21.15.98:0/1995814145 >> 172.21.15.98:0/1995814145 conn(0x55a62e365810 msgr2=0x55a62e36b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad6effd640 0 client.18813 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T09:01:27.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad989ec740 1 -- 172.21.15.98:0/1995814145 shutdown_connections 2024-05-09T09:01:27.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad989ec740 1 -- 172.21.15.98:0/1995814145 wait complete. 2024-05-09T09:01:27.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.906+0000 7fad989ec740 20 client.18813 trim_cache size 0 max 16384 2024-05-09T09:01:27.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad989ec740 1 Processor -- start 2024-05-09T09:01:27.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad989ec740 1 -- start start 2024-05-09T09:01:27.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad989ec740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 0x55a62e377f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad989ec740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e369cf0 0x55a62e384690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad989ec740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e3788f0 0x55a62e375ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad989ec740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad901acab0 con 0x55a62e369cf0 2024-05-09T09:01:27.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad989ec740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fad901ae5f0 con 0x55a62e3693d0 2024-05-09T09:01:27.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad989ec740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fad901ad180 con 0x55a62e3788f0 2024-05-09T09:01:27.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad8f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e369cf0 0x55a62e384690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad8f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e369cf0 0x55a62e384690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47208/0 (socket says 172.21.15.98:47208) 2024-05-09T09:01:27.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad8f7fe640 1 -- 172.21.15.98:0/1112560904 learned_addr learned my addr 172.21.15.98:0/1112560904 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad86ffd640 1 --2- 172.21.15.98:0/1112560904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 0x55a62e377f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad8f7fe640 1 -- 172.21.15.98:0/1112560904 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e3788f0 msgr2=0x55a62e375ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad8f7fe640 1 --2- 172.21.15.98:0/1112560904 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e3788f0 0x55a62e375ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad8f7fe640 1 -- 172.21.15.98:0/1112560904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 msgr2=0x55a62e377f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad8f7fe640 1 --2- 172.21.15.98:0/1112560904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 0x55a62e377f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.919+0000 7fad8f7fe640 1 -- 172.21.15.98:0/1112560904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a62e37b0a0 con 0x55a62e369cf0 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.920+0000 7fad8f7fe640 1 --2- 172.21.15.98:0/1112560904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e369cf0 0x55a62e384690 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fad80009330 tx=0x7fad80016800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb789da1c98af8da server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.920+0000 7fad857fa640 1 -- 172.21.15.98:0/1112560904 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad80016e40 con 0x55a62e369cf0 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.920+0000 7fad857fa640 1 -- 172.21.15.98:0/1112560904 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad80009d70 con 0x55a62e369cf0 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.920+0000 7fad857fa640 1 -- 172.21.15.98:0/1112560904 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad800146c0 con 0x55a62e369cf0 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.920+0000 7fad989ec740 1 -- 172.21.15.98:0/1112560904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e369cf0 msgr2=0x55a62e384690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.920+0000 7fad989ec740 1 --2- 172.21.15.98:0/1112560904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e369cf0 0x55a62e384690 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fad80009330 tx=0x7fad80016800 comp rx=0 tx=0).stop 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.920+0000 7fad989ec740 1 -- 172.21.15.98:0/1112560904 shutdown_connections 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.920+0000 7fad989ec740 1 --2- 172.21.15.98:0/1112560904 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e3788f0 0x55a62e375ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.920+0000 7fad989ec740 1 --2- 172.21.15.98:0/1112560904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e369cf0 0x55a62e384690 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.920+0000 7fad989ec740 1 --2- 172.21.15.98:0/1112560904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 0x55a62e377f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.920+0000 7fad989ec740 1 -- 172.21.15.98:0/1112560904 >> 172.21.15.98:0/1112560904 conn(0x55a62e365810 msgr2=0x55a62e3757e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.920+0000 7fad989ec740 1 -- 172.21.15.98:0/1112560904 shutdown_connections 2024-05-09T09:01:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.920+0000 7fad989ec740 1 -- 172.21.15.98:0/1112560904 wait complete. 2024-05-09T09:01:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad989ec740 1 -- start start 2024-05-09T09:01:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad989ec740 1 Processor -- start 2024-05-09T09:01:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad989ec740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 0x55a62e380bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad989ec740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e369cf0 0x55a62e3810f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad989ec740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e37c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad989ec740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad901adf20 con 0x55a62e3788f0 2024-05-09T09:01:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad989ec740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fad901ad850 con 0x55a62e3693d0 2024-05-09T09:01:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad989ec740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fad901aaab0 con 0x55a62e369cf0 2024-05-09T09:01:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad989ec740 10 client.0 _ll_register_callbacks cb 0x7fad7800be90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad86ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e37c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad86ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e37c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47210/0 (socket says 172.21.15.98:47210) 2024-05-09T09:01:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad86ffd640 1 -- 172.21.15.98:0/132502888 learned_addr learned my addr 172.21.15.98:0/132502888 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad86ffd640 1 --2- 172.21.15.98:0/132502888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 0x55a62e380bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad86ffd640 1 -- 172.21.15.98:0/132502888 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e369cf0 msgr2=0x55a62e3810f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad86ffd640 1 --2- 172.21.15.98:0/132502888 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e369cf0 0x55a62e3810f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad86ffd640 1 -- 172.21.15.98:0/132502888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 msgr2=0x55a62e380bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad86ffd640 1 --2- 172.21.15.98:0/132502888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 0x55a62e380bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad86ffd640 1 -- 172.21.15.98:0/132502888 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a62e37cb20 con 0x55a62e3788f0 2024-05-09T09:01:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad86ffd640 1 --2- 172.21.15.98:0/132502888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e37c320 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fad780127c0 tx=0x7fad78010fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207639fee1daa3a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.921+0000 7fad6effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad6effd640 1 -- 172.21.15.98:0/132502888 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7802f040 con 0x55a62e3788f0 2024-05-09T09:01:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad6effd640 1 -- 172.21.15.98:0/132502888 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fad7800b390 con 0x55a62e3788f0 2024-05-09T09:01:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad989ec740 1 -- 172.21.15.98:0/132502888 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad80009d70 con 0x55a62e3788f0 2024-05-09T09:01:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad6effd640 1 -- 172.21.15.98:0/132502888 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fad7801f420 con 0x55a62e3788f0 2024-05-09T09:01:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad989ec740 10 client.18819 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad989ec740 1 -- 172.21.15.98:0/132502888 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a62e3eee00 con 0x55a62e3788f0 2024-05-09T09:01:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad6effd640 1 -- 172.21.15.98:0/132502888 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad80009d70 con 0x55a62e3788f0 2024-05-09T09:01:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad6effd640 1 -- 172.21.15.98:0/132502888 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fad7801e920 con 0x55a62e3788f0 2024-05-09T09:01:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad989ec740 10 client.18819 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad989ec740 10 client.18819 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:01:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad989ec740 20 client.18819 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad989ec740 1 --2- 172.21.15.98:0/132502888 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55a62e3ef360 0x55a62e3f1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad989ec740 4 client.18819 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:01:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad989ec740 1 -- 172.21.15.98:0/132502888 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a62e2d66c0 con 0x55a62e3ef360 2024-05-09T09:01:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.922+0000 7fad8f7fe640 1 --2- 172.21.15.98:0/132502888 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55a62e3ef360 0x55a62e3f1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad8f7fe640 1 --2- 172.21.15.98:0/132502888 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55a62e3ef360 0x55a62e3f1780 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=67773134e6a40ab2 server_cookie=26576a10508b8340 in_seq=0 out_seq=0 2024-05-09T09:01:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad6effd640 10 client.18819 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T09:01:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad6effd640 1 -- 172.21.15.98:0/132502888 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a62e2d66c0 con 0x55a62e3ef360 2024-05-09T09:01:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad6effd640 10 client.18819 handle_command_reply: tid=0 2024-05-09T09:01:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad989ec740 1 client.18819 shutdown 2024-05-09T09:01:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad989ec740 1 -- 172.21.15.98:0/132502888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 msgr2=0x55a62e37c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad989ec740 1 --2- 172.21.15.98:0/132502888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e37c320 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fad780127c0 tx=0x7fad78010fa0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad989ec740 1 -- 172.21.15.98:0/132502888 shutdown_connections 2024-05-09T09:01:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad989ec740 1 --2- 172.21.15.98:0/132502888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a62e3788f0 0x55a62e37c320 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad989ec740 1 --2- 172.21.15.98:0/132502888 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a62e369cf0 0x55a62e3810f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad989ec740 1 --2- 172.21.15.98:0/132502888 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55a62e3ef360 0x55a62e3f1780 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad989ec740 1 --2- 172.21.15.98:0/132502888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a62e3693d0 0x55a62e380bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad6effd640 0 client.18819 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T09:01:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad989ec740 1 -- 172.21.15.98:0/132502888 >> 172.21.15.98:0/132502888 conn(0x55a62e365810 msgr2=0x55a62e376070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad989ec740 1 -- 172.21.15.98:0/132502888 shutdown_connections 2024-05-09T09:01:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad989ec740 1 -- 172.21.15.98:0/132502888 wait complete. 2024-05-09T09:01:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.923+0000 7fad989ec740 20 client.18819 trim_cache size 0 max 16384 2024-05-09T09:01:27.930 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:01:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.924+0000 7fad9499e640 1 -- 172.21.15.98:0/471224369 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fad7c0a2d00 msgr2=0x7fad7c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.924+0000 7fad9499e640 1 --2- 172.21.15.98:0/471224369 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fad7c0a2d00 0x7fad7c0a51c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fad880edfc0 tx=0x7fad8808eee0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.924+0000 7fad9499e640 1 -- 172.21.15.98:0/471224369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad90107870 msgr2=0x7fad9019c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.924+0000 7fad9499e640 1 --2- 172.21.15.98:0/471224369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad90107870 0x7fad9019c990 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fad88006010 tx=0x7fad88003ce0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.924+0000 7fad9499e640 1 -- 172.21.15.98:0/471224369 shutdown_connections 2024-05-09T09:01:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.924+0000 7fad9499e640 1 --2- 172.21.15.98:0/471224369 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fad7c0a2d00 0x7fad7c0a51c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.924+0000 7fad9499e640 1 --2- 172.21.15.98:0/471224369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad90107870 0x7fad9019c990 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.924+0000 7fad9499e640 1 --2- 172.21.15.98:0/471224369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad9006a750 0x7fad90198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.924+0000 7fad9499e640 1 --2- 172.21.15.98:0/471224369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad90069db0 0x7fad901980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.924+0000 7fad9499e640 1 -- 172.21.15.98:0/471224369 >> 172.21.15.98:0/471224369 conn(0x7fad90077500 msgr2=0x7fad900fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.924+0000 7fad9499e640 1 -- 172.21.15.98:0/471224369 shutdown_connections 2024-05-09T09:01:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:27.924+0000 7fad9499e640 1 -- 172.21.15.98:0/471224369 wait complete. 2024-05-09T09:01:27.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T09:01:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.038+0000 7f864879e640 1 -- 172.21.15.98:0/1076689219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f864006a870 msgr2=0x7f864010e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.038+0000 7f864879e640 1 --2- 172.21.15.98:0/1076689219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f864006a870 0x7f864010e6e0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f8638009cd0 tx=0x7f863802a8e0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.038+0000 7f864879e640 1 -- 172.21.15.98:0/1076689219 shutdown_connections 2024-05-09T09:01:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.038+0000 7f864879e640 1 --2- 172.21.15.98:0/1076689219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f864006a870 0x7f864010e6e0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.038+0000 7f864879e640 1 --2- 172.21.15.98:0/1076689219 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8640069eb0 0x7f864006a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.038+0000 7f864879e640 1 --2- 172.21.15.98:0/1076689219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8640107820 0x7f8640107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.038+0000 7f864879e640 1 -- 172.21.15.98:0/1076689219 >> 172.21.15.98:0/1076689219 conn(0x7f8640077790 msgr2=0x7f8640077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.038+0000 7f864879e640 1 -- 172.21.15.98:0/1076689219 shutdown_connections 2024-05-09T09:01:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.038+0000 7f864879e640 1 -- 172.21.15.98:0/1076689219 wait complete. 2024-05-09T09:01:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.039+0000 7f864879e640 1 Processor -- start 2024-05-09T09:01:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.039+0000 7f864879e640 1 -- start start 2024-05-09T09:01:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.039+0000 7f864879e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8640069eb0 0x7f86401981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.039+0000 7f864879e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f864006a870 0x7f86401986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.039+0000 7f864879e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8640107820 0x7f864019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.039+0000 7f864879e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8640111c30 con 0x7f8640069eb0 2024-05-09T09:01:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.039+0000 7f864879e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8640111ab0 con 0x7f864006a870 2024-05-09T09:01:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.039+0000 7f864879e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8640111db0 con 0x7f8640107820 2024-05-09T09:01:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.039+0000 7f864653a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8640069eb0 0x7f86401981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.039+0000 7f864653a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8640069eb0 0x7f86401981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47236/0 (socket says 172.21.15.98:47236) 2024-05-09T09:01:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.039+0000 7f864653a640 1 -- 172.21.15.98:0/3683370497 learned_addr learned my addr 172.21.15.98:0/3683370497 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.039+0000 7f8645d39640 1 --2- 172.21.15.98:0/3683370497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f864006a870 0x7f86401986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.040+0000 7f864653a640 1 -- 172.21.15.98:0/3683370497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8640107820 msgr2=0x7f864019ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.040+0000 7f864653a640 1 --2- 172.21.15.98:0/3683370497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8640107820 0x7f864019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.040+0000 7f864653a640 1 -- 172.21.15.98:0/3683370497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f864006a870 msgr2=0x7f86401986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.040+0000 7f864653a640 1 --2- 172.21.15.98:0/3683370497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f864006a870 0x7f86401986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.040+0000 7f864653a640 1 -- 172.21.15.98:0/3683370497 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864019d1a0 con 0x7f8640069eb0 2024-05-09T09:01:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.040+0000 7f864653a640 1 --2- 172.21.15.98:0/3683370497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8640069eb0 0x7f86401981b0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f863c00d950 tx=0x7f863c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.040+0000 7f86377fe640 1 -- 172.21.15.98:0/3683370497 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c017070 con 0x7f8640069eb0 2024-05-09T09:01:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.040+0000 7f864879e640 1 -- 172.21.15.98:0/3683370497 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f864019d490 con 0x7f8640069eb0 2024-05-09T09:01:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.040+0000 7f86377fe640 1 -- 172.21.15.98:0/3683370497 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f863c009c30 con 0x7f8640069eb0 2024-05-09T09:01:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.040+0000 7f86377fe640 1 -- 172.21.15.98:0/3683370497 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c00f6b0 con 0x7f8640069eb0 2024-05-09T09:01:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.040+0000 7f864879e640 1 -- 172.21.15.98:0/3683370497 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86401a4d20 con 0x7f8640069eb0 2024-05-09T09:01:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.042+0000 7f86377fe640 1 -- 172.21.15.98:0/3683370497 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f863c0109d0 con 0x7f8640069eb0 2024-05-09T09:01:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f864b584740 1 Processor -- start 2024-05-09T09:01:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f864b584740 1 -- start start 2024-05-09T09:01:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f864b584740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 0x5630f1d7e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f864b584740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1d8d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f864b584740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1d8fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f864b584740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5630f1ce3c40 con 0x5630f1d8d730 2024-05-09T09:01:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f864b584740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5630f1c8b580 con 0x5630f1d7eb30 2024-05-09T09:01:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f864b584740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5630f1cdfc00 con 0x5630f1d7e210 2024-05-09T09:01:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f86377fe640 1 --2- 172.21.15.98:0/3683370497 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f862c0a2d00 0x7f862c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f8645d39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1d8fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f8645d39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1d8fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47240/0 (socket says 172.21.15.98:47240) 2024-05-09T09:01:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f8645d39640 1 -- 172.21.15.98:0/1851358226 learned_addr learned my addr 172.21.15.98:0/1851358226 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f8646d3b640 1 --2- 172.21.15.98:0/3683370497 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f862c0a2d00 0x7f862c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f8646d3b640 1 --2- 172.21.15.98:0/1851358226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 0x5630f1d7e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f86377fe640 1 -- 172.21.15.98:0/3683370497 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f863c0cf3f0 con 0x7f8640069eb0 2024-05-09T09:01:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f864653a640 1 --2- 172.21.15.98:0/1851358226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1d8d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f8645d39640 1 -- 172.21.15.98:0/1851358226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 msgr2=0x5630f1d7e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f8645d39640 1 --2- 172.21.15.98:0/1851358226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 0x5630f1d7e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f8645d39640 1 -- 172.21.15.98:0/1851358226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 msgr2=0x5630f1d8d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f8645d39640 1 --2- 172.21.15.98:0/1851358226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1d8d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.043+0000 7f8645d39640 1 -- 172.21.15.98:0/1851358226 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630f19f73f0 con 0x5630f1d8d730 2024-05-09T09:01:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.044+0000 7f8645d39640 1 --2- 172.21.15.98:0/1851358226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1d8fb20 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f863000b740 tx=0x7f863000bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b8d6fc91720688 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.044+0000 7f8646d3b640 1 --2- 172.21.15.98:0/3683370497 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f862c0a2d00 0x7f862c0a51c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8638009a60 tx=0x7f86380093a0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.044+0000 7f8634ff9640 1 -- 172.21.15.98:0/1851358226 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8630003dd0 con 0x5630f1d8d730 2024-05-09T09:01:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.044+0000 7f8634ff9640 1 -- 172.21.15.98:0/1851358226 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8630015070 con 0x5630f1d8d730 2024-05-09T09:01:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.044+0000 7f8634ff9640 1 -- 172.21.15.98:0/1851358226 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8630004d80 con 0x5630f1d8d730 2024-05-09T09:01:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.044+0000 7f864b584740 1 -- 172.21.15.98:0/1851358226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 msgr2=0x5630f1d8fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.044+0000 7f864b584740 1 --2- 172.21.15.98:0/1851358226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1d8fb20 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f863000b740 tx=0x7f863000bc00 comp rx=0 tx=0).stop 2024-05-09T09:01:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 -- 172.21.15.98:0/1851358226 shutdown_connections 2024-05-09T09:01:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 --2- 172.21.15.98:0/1851358226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1d8fb20 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 --2- 172.21.15.98:0/1851358226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1d8d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 --2- 172.21.15.98:0/1851358226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 0x5630f1d7e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 -- 172.21.15.98:0/1851358226 >> 172.21.15.98:0/1851358226 conn(0x5630f1d7a650 msgr2=0x5630f1d7aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 -- 172.21.15.98:0/1851358226 shutdown_connections 2024-05-09T09:01:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 -- 172.21.15.98:0/1851358226 wait complete. 2024-05-09T09:01:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 -- start start 2024-05-09T09:01:28.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 Processor -- start 2024-05-09T09:01:28.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 0x5630f1d8a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1d8abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1d8b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5630f1d90570 con 0x5630f1d8d730 2024-05-09T09:01:28.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5630f1c9ec40 con 0x5630f1d7eb30 2024-05-09T09:01:28.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5630f1d906f0 con 0x5630f1d7e210 2024-05-09T09:01:28.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864b584740 10 client.0 _ll_register_callbacks cb 0x5630f19f6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:28.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f8646d3b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1d8b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f8646d3b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1d8b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47246/0 (socket says 172.21.15.98:47246) 2024-05-09T09:01:28.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f8646d3b640 1 -- 172.21.15.98:0/1215299408 learned_addr learned my addr 172.21.15.98:0/1215299408 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:28.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.045+0000 7f864653a640 1 --2- 172.21.15.98:0/1215299408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 0x5630f1d8a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f8645d39640 1 --2- 172.21.15.98:0/1215299408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1d8abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f8646d3b640 1 -- 172.21.15.98:0/1215299408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 msgr2=0x5630f1d8a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f8646d3b640 1 --2- 172.21.15.98:0/1215299408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 0x5630f1d8a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f8646d3b640 1 -- 172.21.15.98:0/1215299408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 msgr2=0x5630f1d8abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f8646d3b640 1 --2- 172.21.15.98:0/1215299408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1d8abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f8646d3b640 1 -- 172.21.15.98:0/1215299408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630f1ac17d0 con 0x5630f1d8d730 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f864653a640 1 --2- 172.21.15.98:0/1215299408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 0x5630f1d8a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f8646d3b640 1 --2- 172.21.15.98:0/1215299408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1d8b0f0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f8638031dc0 tx=0x7f8638009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca315a7eb39643f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f8629ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f8629ffb640 1 -- 172.21.15.98:0/1215299408 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863804e070 con 0x5630f1d8d730 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f8629ffb640 1 -- 172.21.15.98:0/1215299408 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f8638049650 con 0x5630f1d8d730 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f864b584740 1 -- 172.21.15.98:0/1215299408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8630015070 con 0x5630f1d8d730 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f8629ffb640 1 -- 172.21.15.98:0/1215299408 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f863805a050 con 0x5630f1d8d730 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f864b584740 10 client.18831 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f864b584740 1 -- 172.21.15.98:0/1215299408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5630f1d9c7d0 con 0x5630f1d8d730 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f8629ffb640 1 -- 172.21.15.98:0/1215299408 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8630015070 con 0x5630f1d8d730 2024-05-09T09:01:28.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f8629ffb640 1 -- 172.21.15.98:0/1215299408 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f8638030070 con 0x5630f1d8d730 2024-05-09T09:01:28.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f864b584740 10 client.18831 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:28.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f864b584740 10 client.18831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:28.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f864b584740 20 client.18831 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:28.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.046+0000 7f864b584740 1 --2- 172.21.15.98:0/1215299408 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5630f1d9dc50 0x5630f1da00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.047+0000 7f864b584740 4 client.18831 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:01:28.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.047+0000 7f864b584740 1 -- 172.21.15.98:0/1215299408 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630f1cf1370 con 0x5630f1d9dc50 2024-05-09T09:01:28.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.047+0000 7f864653a640 1 --2- 172.21.15.98:0/1215299408 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5630f1d9dc50 0x5630f1da00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.047+0000 7f8629ffb640 10 client.18831 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:28.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.047+0000 7f864653a640 1 --2- 172.21.15.98:0/1215299408 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5630f1d9dc50 0x5630f1da00b0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e225501f76ccc010 server_cookie=b11748501abe99f7 in_seq=0 out_seq=0 2024-05-09T09:01:28.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f8629ffb640 1 -- 172.21.15.98:0/1215299408 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5630f1cf1370 con 0x5630f1d9dc50 2024-05-09T09:01:28.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f8629ffb640 10 client.18831 handle_command_reply: tid=0 2024-05-09T09:01:28.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f864b584740 1 client.18831 shutdown 2024-05-09T09:01:28.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f864b584740 1 -- 172.21.15.98:0/1215299408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 msgr2=0x5630f1d8b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:28.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f864b584740 1 --2- 172.21.15.98:0/1215299408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1d8b0f0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f8638031dc0 tx=0x7f8638009890 comp rx=0 tx=0).stop 2024-05-09T09:01:28.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f864b584740 1 -- 172.21.15.98:0/1215299408 shutdown_connections 2024-05-09T09:01:28.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f864b584740 1 --2- 172.21.15.98:0/1215299408 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5630f1d9dc50 0x5630f1da00b0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f864b584740 1 --2- 172.21.15.98:0/1215299408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1d8b0f0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f8629ffb640 0 client.18831 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:28.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f864b584740 1 --2- 172.21.15.98:0/1215299408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1d8abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f864b584740 1 --2- 172.21.15.98:0/1215299408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 0x5630f1d8a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f864b584740 1 -- 172.21.15.98:0/1215299408 >> 172.21.15.98:0/1215299408 conn(0x5630f1d7a650 msgr2=0x5630f1d80400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:28.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f864b584740 1 -- 172.21.15.98:0/1215299408 shutdown_connections 2024-05-09T09:01:28.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f864b584740 1 -- 172.21.15.98:0/1215299408 wait complete. 2024-05-09T09:01:28.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.048+0000 7f864b584740 20 client.18831 trim_cache size 0 max 16384 2024-05-09T09:01:28.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.061+0000 7f864b584740 1 Processor -- start 2024-05-09T09:01:28.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.061+0000 7f864b584740 1 -- start start 2024-05-09T09:01:28.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.061+0000 7f864b584740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 0x5630f1d994d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.061+0000 7f864b584740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1da5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.061+0000 7f864b584740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1da5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.061+0000 7f864b584740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86401acc60 con 0x5630f1d8d730 2024-05-09T09:01:28.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.061+0000 7f864b584740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f86401ae7a0 con 0x5630f1d7eb30 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.061+0000 7f864b584740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f86401ad330 con 0x5630f1d7e210 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f8646d3b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1da5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f8646d3b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1da5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47262/0 (socket says 172.21.15.98:47262) 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f8646d3b640 1 -- 172.21.15.98:0/502645398 learned_addr learned my addr 172.21.15.98:0/502645398 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f864653a640 1 --2- 172.21.15.98:0/502645398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 0x5630f1d994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f8645d39640 1 --2- 172.21.15.98:0/502645398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1da5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f8646d3b640 1 -- 172.21.15.98:0/502645398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 msgr2=0x5630f1d994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f8646d3b640 1 --2- 172.21.15.98:0/502645398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 0x5630f1d994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f8646d3b640 1 -- 172.21.15.98:0/502645398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 msgr2=0x5630f1da5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f8646d3b640 1 --2- 172.21.15.98:0/502645398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1da5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f8646d3b640 1 -- 172.21.15.98:0/502645398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630f1d7e960 con 0x5630f1d8d730 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f8645d39640 1 --2- 172.21.15.98:0/502645398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1da5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f8646d3b640 1 --2- 172.21.15.98:0/502645398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1da5ec0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f863802ae00 tx=0x7f8638030e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aafd516c0de689cd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f862bfff640 1 -- 172.21.15.98:0/502645398 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863804e070 con 0x5630f1d8d730 2024-05-09T09:01:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f862bfff640 1 -- 172.21.15.98:0/502645398 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8638057020 con 0x5630f1d8d730 2024-05-09T09:01:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.062+0000 7f862bfff640 1 -- 172.21.15.98:0/502645398 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8638043040 con 0x5630f1d8d730 2024-05-09T09:01:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 -- 172.21.15.98:0/502645398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 msgr2=0x5630f1da5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 --2- 172.21.15.98:0/502645398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1da5ec0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f863802ae00 tx=0x7f8638030e00 comp rx=0 tx=0).stop 2024-05-09T09:01:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 -- 172.21.15.98:0/502645398 shutdown_connections 2024-05-09T09:01:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 --2- 172.21.15.98:0/502645398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d8d730 0x5630f1da5ec0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 --2- 172.21.15.98:0/502645398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1da5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 --2- 172.21.15.98:0/502645398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d7e210 0x5630f1d994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 -- 172.21.15.98:0/502645398 >> 172.21.15.98:0/502645398 conn(0x5630f1d7a650 msgr2=0x5630f1d897d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 -- 172.21.15.98:0/502645398 shutdown_connections 2024-05-09T09:01:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 -- 172.21.15.98:0/502645398 wait complete. 2024-05-09T09:01:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 -- start start 2024-05-09T09:01:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 Processor -- start 2024-05-09T09:01:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d7e210 0x5630f1d91500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1d91a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d8d730 0x5630f1d91f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86401ae0d0 con 0x5630f1d7e210 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f86401ada00 con 0x5630f1d7eb30 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f86401aac60 con 0x5630f1d8d730 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864b584740 10 client.0 _ll_register_callbacks cb 0x7f8638009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864653a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d7e210 0x5630f1d91500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864653a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d7e210 0x5630f1d91500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47266/0 (socket says 172.21.15.98:47266) 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f864653a640 1 -- 172.21.15.98:0/353102585 learned_addr learned my addr 172.21.15.98:0/353102585 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.063+0000 7f8645d39640 1 --2- 172.21.15.98:0/353102585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1d91a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864653a640 1 -- 172.21.15.98:0/353102585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d8d730 msgr2=0x5630f1d91f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864653a640 1 --2- 172.21.15.98:0/353102585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d8d730 0x5630f1d91f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864653a640 1 -- 172.21.15.98:0/353102585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 msgr2=0x5630f1d91a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864653a640 1 --2- 172.21.15.98:0/353102585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1d91a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864653a640 1 -- 172.21.15.98:0/353102585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630f1e032f0 con 0x5630f1d7e210 2024-05-09T09:01:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f8645d39640 1 --2- 172.21.15.98:0/353102585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1d91a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864653a640 1 --2- 172.21.15.98:0/353102585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d7e210 0x5630f1d91500 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f863c09b490 tx=0x7f863c005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8550e62aef2cddb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f8629ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f8629ffb640 1 -- 172.21.15.98:0/353102585 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c020040 con 0x5630f1d7e210 2024-05-09T09:01:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f8629ffb640 1 -- 172.21.15.98:0/353102585 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f863c097b70 con 0x5630f1d7e210 2024-05-09T09:01:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864b584740 1 -- 172.21.15.98:0/353102585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8638057020 con 0x5630f1d7e210 2024-05-09T09:01:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f8629ffb640 1 -- 172.21.15.98:0/353102585 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f863c09f8b0 con 0x5630f1d7e210 2024-05-09T09:01:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864b584740 10 client.18837 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864b584740 1 -- 172.21.15.98:0/353102585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5630f1e045c0 con 0x5630f1d7e210 2024-05-09T09:01:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f8629ffb640 1 -- 172.21.15.98:0/353102585 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8638057020 con 0x5630f1d7e210 2024-05-09T09:01:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f8629ffb640 1 -- 172.21.15.98:0/353102585 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f863c09a9e0 con 0x5630f1d7e210 2024-05-09T09:01:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864b584740 10 client.18837 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864b584740 10 client.18837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864b584740 20 client.18837 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864b584740 1 --2- 172.21.15.98:0/353102585 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5630f1d9dc50 0x5630f1e06d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864b584740 4 client.18837 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.064+0000 7f864b584740 1 -- 172.21.15.98:0/353102585 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5630f1ceb6c0 con 0x5630f1d9dc50 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f8645d39640 1 --2- 172.21.15.98:0/353102585 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5630f1d9dc50 0x5630f1e06d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f8629ffb640 10 client.18837 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f8645d39640 1 --2- 172.21.15.98:0/353102585 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5630f1d9dc50 0x5630f1e06d00 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b95ab0017c4a7040 server_cookie=3b3053df56e45d1e in_seq=0 out_seq=0 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f8629ffb640 1 -- 172.21.15.98:0/353102585 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5630f1ceb6c0 con 0x5630f1d9dc50 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f8629ffb640 10 client.18837 handle_command_reply: tid=0 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f864b584740 1 client.18837 shutdown 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f864b584740 1 -- 172.21.15.98:0/353102585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d7e210 msgr2=0x5630f1d91500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f864b584740 1 --2- 172.21.15.98:0/353102585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d7e210 0x5630f1d91500 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f863c09b490 tx=0x7f863c005d30 comp rx=0 tx=0).stop 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f864b584740 1 -- 172.21.15.98:0/353102585 shutdown_connections 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f864b584740 1 --2- 172.21.15.98:0/353102585 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5630f1d9dc50 0x5630f1e06d00 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f864b584740 1 --2- 172.21.15.98:0/353102585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5630f1d8d730 0x5630f1d91f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f8629ffb640 0 client.18837 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:28.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f864b584740 1 --2- 172.21.15.98:0/353102585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5630f1d7eb30 0x5630f1d91a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f864b584740 1 --2- 172.21.15.98:0/353102585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630f1d7e210 0x5630f1d91500 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f864b584740 1 -- 172.21.15.98:0/353102585 >> 172.21.15.98:0/353102585 conn(0x5630f1d7a650 msgr2=0x5630f1d78f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f864b584740 1 -- 172.21.15.98:0/353102585 shutdown_connections 2024-05-09T09:01:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.065+0000 7f864b584740 1 -- 172.21.15.98:0/353102585 wait complete. 2024-05-09T09:01:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.066+0000 7f864b584740 20 client.18837 trim_cache size 0 max 16384 2024-05-09T09:01:28.072 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:01:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.066+0000 7f864879e640 1 -- 172.21.15.98:0/3683370497 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f862c0a2d00 msgr2=0x7f862c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.066+0000 7f864879e640 1 --2- 172.21.15.98:0/3683370497 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f862c0a2d00 0x7f862c0a51c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8638009a60 tx=0x7f86380093a0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.066+0000 7f864879e640 1 -- 172.21.15.98:0/3683370497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8640069eb0 msgr2=0x7f86401981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.066+0000 7f864879e640 1 --2- 172.21.15.98:0/3683370497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8640069eb0 0x7f86401981b0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f863c00d950 tx=0x7f863c00de10 comp rx=0 tx=0).stop 2024-05-09T09:01:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.066+0000 7f864879e640 1 -- 172.21.15.98:0/3683370497 shutdown_connections 2024-05-09T09:01:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.066+0000 7f864879e640 1 --2- 172.21.15.98:0/3683370497 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f862c0a2d00 0x7f862c0a51c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.066+0000 7f864879e640 1 --2- 172.21.15.98:0/3683370497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8640107820 0x7f864019ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.066+0000 7f864879e640 1 --2- 172.21.15.98:0/3683370497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f864006a870 0x7f86401986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.066+0000 7f864879e640 1 --2- 172.21.15.98:0/3683370497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8640069eb0 0x7f86401981b0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:28.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.066+0000 7f864879e640 1 -- 172.21.15.98:0/3683370497 >> 172.21.15.98:0/3683370497 conn(0x7f8640077790 msgr2=0x7f86400fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:28.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.066+0000 7f864879e640 1 -- 172.21.15.98:0/3683370497 shutdown_connections 2024-05-09T09:01:28.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:28.066+0000 7f864879e640 1 -- 172.21.15.98:0/3683370497 wait complete. 2024-05-09T09:01:28.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:28 smithi142 ceph-mon[34625]: pgmap v339: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-09T09:01:28.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:28 smithi105 ceph-mon[34498]: pgmap v339: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-09T09:01:28.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:28 smithi098 ceph-mon[30092]: pgmap v339: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-09T09:01:29.076 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T09:01:29.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:01:29.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T09:01:29.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.194+0000 7f0ccf563640 1 -- 172.21.15.98:0/2576931238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc80ffd30 msgr2=0x7f0cc81001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.194+0000 7f0ccf563640 1 --2- 172.21.15.98:0/2576931238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc80ffd30 0x7f0cc81001b0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009930 tx=0x7f0cb802a420 comp rx=0 tx=0).stop 2024-05-09T09:01:29.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 -- 172.21.15.98:0/2576931238 shutdown_connections 2024-05-09T09:01:29.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 --2- 172.21.15.98:0/2576931238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0cc81006f0 0x7f0cc810e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 --2- 172.21.15.98:0/2576931238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc80ffd30 0x7f0cc81001b0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 --2- 172.21.15.98:0/2576931238 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cc8106810 0x7f0cc8106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 -- 172.21.15.98:0/2576931238 >> 172.21.15.98:0/2576931238 conn(0x7f0cc8076cf0 msgr2=0x7f0cc80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:29.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 -- 172.21.15.98:0/2576931238 shutdown_connections 2024-05-09T09:01:29.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 -- 172.21.15.98:0/2576931238 wait complete. 2024-05-09T09:01:29.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 Processor -- start 2024-05-09T09:01:29.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 -- start start 2024-05-09T09:01:29.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0cc80ffd30 0x7f0cc81981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cc81006f0 0x7f0cc8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc8106810 0x7f0cc819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc8111cf0 con 0x7f0cc8106810 2024-05-09T09:01:29.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc8111b70 con 0x7f0cc81006f0 2024-05-09T09:01:29.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.195+0000 7f0ccf563640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc8111e70 con 0x7f0cc80ffd30 2024-05-09T09:01:29.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0ccdb00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc8106810 0x7f0cc819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0ccdb00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc8106810 0x7f0cc819ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47294/0 (socket says 172.21.15.98:47294) 2024-05-09T09:01:29.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0ccdb00640 1 -- 172.21.15.98:0/3833175306 learned_addr learned my addr 172.21.15.98:0/3833175306 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:29.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0ccd2ff640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0cc80ffd30 0x7f0cc81981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0cccafe640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cc81006f0 0x7f0cc8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0ccdb00640 1 -- 172.21.15.98:0/3833175306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0cc80ffd30 msgr2=0x7f0cc81981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0ccdb00640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0cc80ffd30 0x7f0cc81981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0ccdb00640 1 -- 172.21.15.98:0/3833175306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cc81006f0 msgr2=0x7f0cc8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0ccdb00640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cc81006f0 0x7f0cc8198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0ccdb00640 1 -- 172.21.15.98:0/3833175306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cc819d170 con 0x7f0cc8106810 2024-05-09T09:01:29.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0ccd2ff640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0cc80ffd30 0x7f0cc81981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:29.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0ccdb00640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc8106810 0x7f0cc819ca40 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f0cc000eaa0 tx=0x7f0cc000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:29.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0cbe7fc640 1 -- 172.21.15.98:0/3833175306 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000cdb0 con 0x7f0cc8106810 2024-05-09T09:01:29.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0cbe7fc640 1 -- 172.21.15.98:0/3833175306 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cc0005020 con 0x7f0cc8106810 2024-05-09T09:01:29.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0ccf563640 1 -- 172.21.15.98:0/3833175306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cc819d460 con 0x7f0cc8106810 2024-05-09T09:01:29.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0cbe7fc640 1 -- 172.21.15.98:0/3833175306 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc00116b0 con 0x7f0cc8106810 2024-05-09T09:01:29.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.196+0000 7f0ccf563640 1 -- 172.21.15.98:0/3833175306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cc81a4cf0 con 0x7f0cc8106810 2024-05-09T09:01:29.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.199+0000 7f0cbe7fc640 1 -- 172.21.15.98:0/3833175306 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0cc0005560 con 0x7f0cc8106810 2024-05-09T09:01:29.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.199+0000 7f0cd2279740 1 Processor -- start 2024-05-09T09:01:29.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.199+0000 7f0cd2279740 1 -- start start 2024-05-09T09:01:29.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cd2279740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99e0b0 0x55da7c99c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cd2279740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c99c9c0 0x55da7c99ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cd2279740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c9a7740 0x55da7c9a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cd2279740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55da7c901990 con 0x55da7c99e0b0 2024-05-09T09:01:29.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cd2279740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55da7c8a95b0 con 0x55da7c9a7740 2024-05-09T09:01:29.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cd2279740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55da7c8fd950 con 0x55da7c99c9c0 2024-05-09T09:01:29.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cccafe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99e0b0 0x55da7c99c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cccafe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99e0b0 0x55da7c99c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47300/0 (socket says 172.21.15.98:47300) 2024-05-09T09:01:29.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cccafe640 1 -- 172.21.15.98:0/3462251128 learned_addr learned my addr 172.21.15.98:0/3462251128 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:29.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0ccd2ff640 1 --2- 172.21.15.98:0/3462251128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c99c9c0 0x55da7c99ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cbe7fc640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0cb40a2d00 0x7f0cb40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cccafe640 1 -- 172.21.15.98:0/3462251128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c99c9c0 msgr2=0x55da7c99ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cccafe640 1 --2- 172.21.15.98:0/3462251128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c99c9c0 0x55da7c99ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cccafe640 1 -- 172.21.15.98:0/3462251128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c9a7740 msgr2=0x55da7c9a9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cccafe640 1 --2- 172.21.15.98:0/3462251128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c9a7740 0x55da7c9a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cccafe640 1 -- 172.21.15.98:0/3462251128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da7c6dd7e0 con 0x55da7c99e0b0 2024-05-09T09:01:29.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0ccdb00640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0cb40a2d00 0x7f0cb40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cbe7fc640 1 -- 172.21.15.98:0/3833175306 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f0cc00cfb60 con 0x7f0cc8106810 2024-05-09T09:01:29.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.200+0000 7f0cccafe640 1 --2- 172.21.15.98:0/3462251128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99e0b0 0x55da7c99c3f0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f0cb802a940 tx=0x7f0cb8032c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d7ba4136ac64f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:29.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.201+0000 7f0cb37fe640 1 -- 172.21.15.98:0/3462251128 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8035050 con 0x55da7c99e0b0 2024-05-09T09:01:29.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.201+0000 7f0cb37fe640 1 -- 172.21.15.98:0/3462251128 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cb8032e20 con 0x55da7c99e0b0 2024-05-09T09:01:29.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.201+0000 7f0cb37fe640 1 -- 172.21.15.98:0/3462251128 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb80054a0 con 0x55da7c99e0b0 2024-05-09T09:01:29.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.201+0000 7f0ccdb00640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0cb40a2d00 0x7f0cb40a51c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0cc00ccf50 tx=0x7f0cc0051040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:29.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.201+0000 7f0cd2279740 1 -- 172.21.15.98:0/3462251128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99e0b0 msgr2=0x55da7c99c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.201+0000 7f0cd2279740 1 --2- 172.21.15.98:0/3462251128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99e0b0 0x55da7c99c3f0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f0cb802a940 tx=0x7f0cb8032c30 comp rx=0 tx=0).stop 2024-05-09T09:01:29.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.201+0000 7f0cd2279740 1 -- 172.21.15.98:0/3462251128 shutdown_connections 2024-05-09T09:01:29.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.201+0000 7f0cd2279740 1 --2- 172.21.15.98:0/3462251128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c9a7740 0x55da7c9a9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.201+0000 7f0cd2279740 1 --2- 172.21.15.98:0/3462251128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c99c9c0 0x55da7c99ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.201+0000 7f0cd2279740 1 --2- 172.21.15.98:0/3462251128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99e0b0 0x55da7c99c3f0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.201+0000 7f0cd2279740 1 -- 172.21.15.98:0/3462251128 >> 172.21.15.98:0/3462251128 conn(0x55da7c9650a0 msgr2=0x55da7c998f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:29.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.201+0000 7f0cd2279740 1 -- 172.21.15.98:0/3462251128 shutdown_connections 2024-05-09T09:01:29.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.201+0000 7f0cd2279740 1 -- 172.21.15.98:0/3462251128 wait complete. 2024-05-09T09:01:29.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.202+0000 7f0cd2279740 1 -- start start 2024-05-09T09:01:29.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.202+0000 7f0cd2279740 1 Processor -- start 2024-05-09T09:01:29.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.202+0000 7f0cd2279740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c99c9c0 0x55da7c9a4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.202+0000 7f0cd2279740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c99e0b0 0x55da7c9a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.202+0000 7f0cd2279740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c9a7740 0x55da7c9a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.202+0000 7f0cd2279740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55da7c9aa570 con 0x55da7c9a7740 2024-05-09T09:01:29.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.202+0000 7f0cd2279740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55da7c8bcb40 con 0x55da7c99c9c0 2024-05-09T09:01:29.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.202+0000 7f0cd2279740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55da7c9aa6f0 con 0x55da7c99e0b0 2024-05-09T09:01:29.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.202+0000 7f0cd2279740 10 client.0 _ll_register_callbacks cb 0x55da7c8b10f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:29.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.202+0000 7f0cccafe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c9a7740 0x55da7c9a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.202+0000 7f0cccafe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c9a7740 0x55da7c9a50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47312/0 (socket says 172.21.15.98:47312) 2024-05-09T09:01:29.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.202+0000 7f0cccafe640 1 -- 172.21.15.98:0/1152133583 learned_addr learned my addr 172.21.15.98:0/1152133583 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:29.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.202+0000 7f0ccd2ff640 1 --2- 172.21.15.98:0/1152133583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c99e0b0 0x55da7c9a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cccafe640 1 --2- 172.21.15.98:0/1152133583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c99c9c0 0x55da7c9a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cccafe640 1 -- 172.21.15.98:0/1152133583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c99e0b0 msgr2=0x55da7c9a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cccafe640 1 --2- 172.21.15.98:0/1152133583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c99e0b0 0x55da7c9a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cccafe640 1 -- 172.21.15.98:0/1152133583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c99c9c0 msgr2=0x55da7c9a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cccafe640 1 --2- 172.21.15.98:0/1152133583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c99c9c0 0x55da7c9a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cccafe640 1 -- 172.21.15.98:0/1152133583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da7c73df70 con 0x55da7c9a7740 2024-05-09T09:01:29.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cccafe640 1 --2- 172.21.15.98:0/1152133583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c9a7740 0x55da7c9a50a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8031550 tx=0x7f0cb80098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e6b5c8d9b720aa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:29.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:29.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cb0ff9640 1 -- 172.21.15.98:0/1152133583 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8035070 con 0x55da7c9a7740 2024-05-09T09:01:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cb0ff9640 1 -- 172.21.15.98:0/1152133583 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f0cb8004300 con 0x55da7c9a7740 2024-05-09T09:01:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cd2279740 1 -- 172.21.15.98:0/1152133583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cb8032e20 con 0x55da7c9a7740 2024-05-09T09:01:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cb0ff9640 1 -- 172.21.15.98:0/1152133583 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0cb8059050 con 0x55da7c9a7740 2024-05-09T09:01:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cd2279740 10 client.18849 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cd2279740 1 -- 172.21.15.98:0/1152133583 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55da7c5d5170 con 0x55da7c9a7740 2024-05-09T09:01:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.203+0000 7f0cb0ff9640 1 -- 172.21.15.98:0/1152133583 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cb8032e20 con 0x55da7c9a7740 2024-05-09T09:01:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.204+0000 7f0cb0ff9640 1 -- 172.21.15.98:0/1152133583 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f0cb8003780 con 0x55da7c9a7740 2024-05-09T09:01:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.204+0000 7f0cd2279740 10 client.18849 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.204+0000 7f0cd2279740 10 client.18849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.204+0000 7f0cd2279740 20 client.18849 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.204+0000 7f0cd2279740 1 --2- 172.21.15.98:0/1152133583 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55da7c9b7680 0x55da7c9b9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.204+0000 7f0cd2279740 4 client.18849 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:01:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.204+0000 7f0cd2279740 1 -- 172.21.15.98:0/1152133583 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da7c90f0c0 con 0x55da7c9b7680 2024-05-09T09:01:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.204+0000 7f0ccd2ff640 1 --2- 172.21.15.98:0/1152133583 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55da7c9b7680 0x55da7c9b9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.204+0000 7f0ccd2ff640 1 --2- 172.21.15.98:0/1152133583 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55da7c9b7680 0x55da7c9b9a70 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ef37bff6ad7f488 server_cookie=76869d37255e58b8 in_seq=0 out_seq=0 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.204+0000 7f0cb0ff9640 10 client.18849 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cb0ff9640 1 -- 172.21.15.98:0/1152133583 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55da7c90f0c0 con 0x55da7c9b7680 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cb0ff9640 10 client.18849 handle_command_reply: tid=0 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cd2279740 1 client.18849 shutdown 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cd2279740 1 -- 172.21.15.98:0/1152133583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c9a7740 msgr2=0x55da7c9a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cd2279740 1 --2- 172.21.15.98:0/1152133583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c9a7740 0x55da7c9a50a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8031550 tx=0x7f0cb80098a0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cd2279740 1 -- 172.21.15.98:0/1152133583 shutdown_connections 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cd2279740 1 --2- 172.21.15.98:0/1152133583 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55da7c9b7680 0x55da7c9b9a70 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cd2279740 1 --2- 172.21.15.98:0/1152133583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c9a7740 0x55da7c9a50a0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cd2279740 1 --2- 172.21.15.98:0/1152133583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c99e0b0 0x55da7c9a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cb0ff9640 0 client.18849 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cd2279740 1 --2- 172.21.15.98:0/1152133583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c99c9c0 0x55da7c9a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cd2279740 1 -- 172.21.15.98:0/1152133583 >> 172.21.15.98:0/1152133583 conn(0x55da7c9650a0 msgr2=0x55da7c99ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:29.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cd2279740 1 -- 172.21.15.98:0/1152133583 shutdown_connections 2024-05-09T09:01:29.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cd2279740 1 -- 172.21.15.98:0/1152133583 wait complete. 2024-05-09T09:01:29.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.205+0000 7f0cd2279740 20 client.18849 trim_cache size 0 max 16384 2024-05-09T09:01:29.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cd2279740 1 Processor -- start 2024-05-09T09:01:29.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cd2279740 1 -- start start 2024-05-09T09:01:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cd2279740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99c9c0 0x55da7c9a3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cd2279740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c99e0b0 0x55da7c9bca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cd2279740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c9a7740 0x55da7c9bcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cd2279740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc81aedf0 con 0x55da7c99c9c0 2024-05-09T09:01:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cd2279740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc81b0930 con 0x55da7c99e0b0 2024-05-09T09:01:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cd2279740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc81af4c0 con 0x55da7c9a7740 2024-05-09T09:01:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cccafe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99c9c0 0x55da7c9a3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cccafe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99c9c0 0x55da7c9a3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47328/0 (socket says 172.21.15.98:47328) 2024-05-09T09:01:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cccafe640 1 -- 172.21.15.98:0/1419059610 learned_addr learned my addr 172.21.15.98:0/1419059610 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cccafe640 1 -- 172.21.15.98:0/1419059610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c9a7740 msgr2=0x55da7c9bcfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cccafe640 1 --2- 172.21.15.98:0/1419059610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c9a7740 0x55da7c9bcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cccafe640 1 -- 172.21.15.98:0/1419059610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c99e0b0 msgr2=0x55da7c9bca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cccafe640 1 --2- 172.21.15.98:0/1419059610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c99e0b0 0x55da7c9bca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cccafe640 1 -- 172.21.15.98:0/1419059610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da7c9a0240 con 0x55da7c99c9c0 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cccafe640 1 --2- 172.21.15.98:0/1419059610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99c9c0 0x55da7c9a3510 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009930 tx=0x7f0cb8031620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715d56cd0867bac9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cb2ffd640 1 -- 172.21.15.98:0/1419059610 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8035040 con 0x55da7c99c9c0 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cb2ffd640 1 -- 172.21.15.98:0/1419059610 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cb8003dc0 con 0x55da7c99c9c0 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.223+0000 7f0cb2ffd640 1 -- 172.21.15.98:0/1419059610 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb803d620 con 0x55da7c99c9c0 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.224+0000 7f0cd2279740 1 -- 172.21.15.98:0/1419059610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99c9c0 msgr2=0x55da7c9a3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.224+0000 7f0cd2279740 1 --2- 172.21.15.98:0/1419059610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99c9c0 0x55da7c9a3510 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009930 tx=0x7f0cb8031620 comp rx=0 tx=0).stop 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.224+0000 7f0cd2279740 1 -- 172.21.15.98:0/1419059610 shutdown_connections 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.224+0000 7f0cd2279740 1 --2- 172.21.15.98:0/1419059610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c9a7740 0x55da7c9bcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.224+0000 7f0cd2279740 1 --2- 172.21.15.98:0/1419059610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c99e0b0 0x55da7c9bca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.224+0000 7f0cd2279740 1 --2- 172.21.15.98:0/1419059610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99c9c0 0x55da7c9a3510 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.224+0000 7f0cd2279740 1 -- 172.21.15.98:0/1419059610 >> 172.21.15.98:0/1419059610 conn(0x55da7c9650a0 msgr2=0x55da7c9a2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.224+0000 7f0cd2279740 1 -- 172.21.15.98:0/1419059610 shutdown_connections 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.224+0000 7f0cd2279740 1 -- 172.21.15.98:0/1419059610 wait complete. 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.224+0000 7f0cd2279740 1 -- start start 2024-05-09T09:01:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.224+0000 7f0cd2279740 1 Processor -- start 2024-05-09T09:01:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0cd2279740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c99c9c0 0x55da7c9babe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0cd2279740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99e0b0 0x55da7c9bb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0cd2279740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c9a7740 0x55da7ca1c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0cd2279740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc81b0260 con 0x55da7c99e0b0 2024-05-09T09:01:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0cd2279740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc81afb90 con 0x55da7c9a7740 2024-05-09T09:01:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0cd2279740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc81acdf0 con 0x55da7c99c9c0 2024-05-09T09:01:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0cd2279740 10 client.0 _ll_register_callbacks cb 0x7f0cb8005bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0ccd2ff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99e0b0 0x55da7c9bb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0ccd2ff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99e0b0 0x55da7c9bb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47344/0 (socket says 172.21.15.98:47344) 2024-05-09T09:01:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0ccd2ff640 1 -- 172.21.15.98:0/3352634784 learned_addr learned my addr 172.21.15.98:0/3352634784 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0ccd2ff640 1 -- 172.21.15.98:0/3352634784 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c99c9c0 msgr2=0x55da7c9babe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0ccd2ff640 1 --2- 172.21.15.98:0/3352634784 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c99c9c0 0x55da7c9babe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0ccd2ff640 1 -- 172.21.15.98:0/3352634784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c9a7740 msgr2=0x55da7ca1c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0ccd2ff640 1 --2- 172.21.15.98:0/3352634784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c9a7740 0x55da7ca1c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0ccd2ff640 1 -- 172.21.15.98:0/3352634784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da7ca1cda0 con 0x55da7c99e0b0 2024-05-09T09:01:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0ccd2ff640 1 --2- 172.21.15.98:0/3352634784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99e0b0 0x55da7c9bb120 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f0cc40058c0 tx=0x7f0cc400b320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ab880a1436400c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0cb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0cb0ff9640 1 -- 172.21.15.98:0/3352634784 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4022020 con 0x55da7c99e0b0 2024-05-09T09:01:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0cb0ff9640 1 -- 172.21.15.98:0/3352634784 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f0cc401e620 con 0x55da7c99e0b0 2024-05-09T09:01:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.225+0000 7f0cd2279740 1 -- 172.21.15.98:0/3352634784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cb8003dc0 con 0x55da7c99e0b0 2024-05-09T09:01:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.226+0000 7f0cb0ff9640 1 -- 172.21.15.98:0/3352634784 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0cc402d050 con 0x55da7c99e0b0 2024-05-09T09:01:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.226+0000 7f0cd2279740 10 client.18855 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.226+0000 7f0cd2279740 1 -- 172.21.15.98:0/3352634784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55da7ca1e140 con 0x55da7c99e0b0 2024-05-09T09:01:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.226+0000 7f0cb0ff9640 1 -- 172.21.15.98:0/3352634784 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cb8003dc0 con 0x55da7c99e0b0 2024-05-09T09:01:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.226+0000 7f0cb0ff9640 1 -- 172.21.15.98:0/3352634784 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f0cc4011440 con 0x55da7c99e0b0 2024-05-09T09:01:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.226+0000 7f0cd2279740 10 client.18855 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.226+0000 7f0cd2279740 10 client.18855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.226+0000 7f0cd2279740 20 client.18855 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.226+0000 7f0cd2279740 1 --2- 172.21.15.98:0/3352634784 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55da7ca1e940 0x55da7ca20d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.226+0000 7f0cd2279740 4 client.18855 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.226+0000 7f0cd2279740 1 -- 172.21.15.98:0/3352634784 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55da7c909410 con 0x55da7ca1e940 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.226+0000 7f0cccafe640 1 --2- 172.21.15.98:0/3352634784 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55da7ca1e940 0x55da7ca20d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.226+0000 7f0cccafe640 1 --2- 172.21.15.98:0/3352634784 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55da7ca1e940 0x55da7ca20d60 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4291883d36c9e383 server_cookie=f767722473537bf6 in_seq=0 out_seq=0 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.226+0000 7f0cb0ff9640 10 client.18855 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cb0ff9640 1 -- 172.21.15.98:0/3352634784 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55da7c909410 con 0x55da7ca1e940 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cb0ff9640 10 client.18855 handle_command_reply: tid=0 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cd2279740 1 client.18855 shutdown 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cd2279740 1 -- 172.21.15.98:0/3352634784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99e0b0 msgr2=0x55da7c9bb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cd2279740 1 --2- 172.21.15.98:0/3352634784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99e0b0 0x55da7c9bb120 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f0cc40058c0 tx=0x7f0cc400b320 comp rx=0 tx=0).stop 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cd2279740 1 -- 172.21.15.98:0/3352634784 shutdown_connections 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cd2279740 1 --2- 172.21.15.98:0/3352634784 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55da7ca1e940 0x55da7ca20d60 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cb0ff9640 0 client.18855 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:29.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cd2279740 1 --2- 172.21.15.98:0/3352634784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da7c9a7740 0x55da7ca1c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cd2279740 1 --2- 172.21.15.98:0/3352634784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7c99e0b0 0x55da7c9bb120 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cd2279740 1 --2- 172.21.15.98:0/3352634784 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55da7c99c9c0 0x55da7c9babe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cd2279740 1 -- 172.21.15.98:0/3352634784 >> 172.21.15.98:0/3352634784 conn(0x55da7c9650a0 msgr2=0x55da7c9b3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:29.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cd2279740 1 -- 172.21.15.98:0/3352634784 shutdown_connections 2024-05-09T09:01:29.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cd2279740 1 -- 172.21.15.98:0/3352634784 wait complete. 2024-05-09T09:01:29.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.228+0000 7f0cd2279740 20 client.18855 trim_cache size 0 max 16384 2024-05-09T09:01:29.233 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:01:29.233 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T09:01:29.233 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "8c645698-5e0d-48eb-bcdc-5fba2d5b407e", 2024-05-09T09:01:29.233 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T09:01:29.234 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:01:29.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.229+0000 7f0ccf563640 1 -- 172.21.15.98:0/3833175306 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0cb40a2d00 msgr2=0x7f0cb40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.229+0000 7f0ccf563640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0cb40a2d00 0x7f0cb40a51c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0cc00ccf50 tx=0x7f0cc0051040 comp rx=0 tx=0).stop 2024-05-09T09:01:29.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.229+0000 7f0ccf563640 1 -- 172.21.15.98:0/3833175306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc8106810 msgr2=0x7f0cc819ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.229+0000 7f0ccf563640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc8106810 0x7f0cc819ca40 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f0cc000eaa0 tx=0x7f0cc000ef60 comp rx=0 tx=0).stop 2024-05-09T09:01:29.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.229+0000 7f0ccf563640 1 -- 172.21.15.98:0/3833175306 shutdown_connections 2024-05-09T09:01:29.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.229+0000 7f0ccf563640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f0cb40a2d00 0x7f0cb40a51c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.229+0000 7f0ccf563640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc8106810 0x7f0cc819ca40 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.229+0000 7f0ccf563640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cc81006f0 0x7f0cc8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.229+0000 7f0ccf563640 1 --2- 172.21.15.98:0/3833175306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0cc80ffd30 0x7f0cc81981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.229+0000 7f0ccf563640 1 -- 172.21.15.98:0/3833175306 >> 172.21.15.98:0/3833175306 conn(0x7f0cc8076cf0 msgr2=0x7f0cc8104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:29.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.229+0000 7f0ccf563640 1 -- 172.21.15.98:0/3833175306 shutdown_connections 2024-05-09T09:01:29.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.229+0000 7f0ccf563640 1 -- 172.21.15.98:0/3833175306 wait complete. 2024-05-09T09:01:29.239 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:01:29.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:01:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.343+0000 7f5e76dd5640 1 -- 172.21.15.98:0/1264472892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e70100aa0 msgr2=0x7f5e70104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.343+0000 7f5e76dd5640 1 --2- 172.21.15.98:0/1264472892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e70100aa0 0x7f5e70104ae0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c009900 tx=0x7f5e5c02a410 comp rx=0 tx=0).stop 2024-05-09T09:01:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.343+0000 7f5e76dd5640 1 -- 172.21.15.98:0/1264472892 shutdown_connections 2024-05-09T09:01:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.343+0000 7f5e76dd5640 1 --2- 172.21.15.98:0/1264472892 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e701051c0 0x7f5e7010cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.343+0000 7f5e76dd5640 1 --2- 172.21.15.98:0/1264472892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e70100aa0 0x7f5e70104ae0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.343+0000 7f5e76dd5640 1 --2- 172.21.15.98:0/1264472892 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e701000f0 0x7f5e701004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.343+0000 7f5e76dd5640 1 -- 172.21.15.98:0/1264472892 >> 172.21.15.98:0/1264472892 conn(0x7f5e70076cf0 msgr2=0x7f5e700fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.343+0000 7f5e76dd5640 1 -- 172.21.15.98:0/1264472892 shutdown_connections 2024-05-09T09:01:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.343+0000 7f5e76dd5640 1 -- 172.21.15.98:0/1264472892 wait complete. 2024-05-09T09:01:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.343+0000 7f5e76dd5640 1 Processor -- start 2024-05-09T09:01:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e76dd5640 1 -- start start 2024-05-09T09:01:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e76dd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e701000f0 0x7f5e70199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e76dd5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e701051c0 0x7f5e701996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e76dd5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e70199c40 0x7f5e701a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e76dd5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e70110bc0 con 0x7f5e701000f0 2024-05-09T09:01:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e76dd5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5e70110a40 con 0x7f5e70199c40 2024-05-09T09:01:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e76dd5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5e70110d40 con 0x7f5e701051c0 2024-05-09T09:01:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e74b71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e701000f0 0x7f5e70199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e74b71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e701000f0 0x7f5e70199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47362/0 (socket says 172.21.15.98:47362) 2024-05-09T09:01:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e74b71640 1 -- 172.21.15.98:0/1454508285 learned_addr learned my addr 172.21.15.98:0/1454508285 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e75372640 1 --2- 172.21.15.98:0/1454508285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e70199c40 0x7f5e701a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e74b71640 1 -- 172.21.15.98:0/1454508285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e701051c0 msgr2=0x7f5e701996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e74b71640 1 --2- 172.21.15.98:0/1454508285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e701051c0 0x7f5e701996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e74b71640 1 -- 172.21.15.98:0/1454508285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e70199c40 msgr2=0x7f5e701a41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e74b71640 1 --2- 172.21.15.98:0/1454508285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e70199c40 0x7f5e701a41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e74b71640 1 -- 172.21.15.98:0/1454508285 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e701a49f0 con 0x7f5e701000f0 2024-05-09T09:01:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e75372640 1 --2- 172.21.15.98:0/1454508285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e70199c40 0x7f5e701a41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e74b71640 1 --2- 172.21.15.98:0/1454508285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e701000f0 0x7f5e70199170 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f5e6400b4a0 tx=0x7f5e6400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e6dffb640 1 -- 172.21.15.98:0/1454508285 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e64014070 con 0x7f5e701000f0 2024-05-09T09:01:29.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e6dffb640 1 -- 172.21.15.98:0/1454508285 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e64010af0 con 0x7f5e701000f0 2024-05-09T09:01:29.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e6dffb640 1 -- 172.21.15.98:0/1454508285 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e64017d10 con 0x7f5e701000f0 2024-05-09T09:01:29.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.344+0000 7f5e76dd5640 1 -- 172.21.15.98:0/1454508285 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e701a4ce0 con 0x7f5e701000f0 2024-05-09T09:01:29.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.345+0000 7f5e76dd5640 1 -- 172.21.15.98:0/1454508285 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e701a5180 con 0x7f5e701000f0 2024-05-09T09:01:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.347+0000 7f5e6dffb640 1 -- 172.21.15.98:0/1454508285 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5e64017070 con 0x7f5e701000f0 2024-05-09T09:01:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.347+0000 7f5e79bb6740 1 Processor -- start 2024-05-09T09:01:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.347+0000 7f5e79bb6740 1 -- start start 2024-05-09T09:01:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.347+0000 7f5e6dffb640 1 --2- 172.21.15.98:0/1454508285 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5e600a2d00 0x7f5e600a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e79bb6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7e160 0x55934ec7e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e79bb6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec7ea80 0x55934ec8d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e79bb6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec8d680 0x55934ec8fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e79bb6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55934ebe3d00 con 0x55934ec7ea80 2024-05-09T09:01:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e79bb6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55934eb8b4c0 con 0x55934ec8d680 2024-05-09T09:01:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e79bb6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55934ebdfcc0 con 0x55934ec7e160 2024-05-09T09:01:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e75372640 1 --2- 172.21.15.98:0/1454508285 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5e600a2d00 0x7f5e600a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e6dffb640 1 -- 172.21.15.98:0/1454508285 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f5e640cebd0 con 0x7f5e701000f0 2024-05-09T09:01:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec7ea80 0x55934ec8d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec7ea80 0x55934ec8d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47372/0 (socket says 172.21.15.98:47372) 2024-05-09T09:01:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e6ffff640 1 -- 172.21.15.98:0/325080491 learned_addr learned my addr 172.21.15.98:0/325080491 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e74b71640 1 --2- 172.21.15.98:0/325080491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7e160 0x55934ec7e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e75372640 1 --2- 172.21.15.98:0/325080491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec8d680 0x55934ec8fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e6ffff640 1 -- 172.21.15.98:0/325080491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7e160 msgr2=0x55934ec7e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e6ffff640 1 --2- 172.21.15.98:0/325080491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7e160 0x55934ec7e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e6ffff640 1 -- 172.21.15.98:0/325080491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec8d680 msgr2=0x55934ec8fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e6ffff640 1 --2- 172.21.15.98:0/325080491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec8d680 0x55934ec8fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e6ffff640 1 -- 172.21.15.98:0/325080491 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55934e9c17d0 con 0x55934ec7ea80 2024-05-09T09:01:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e75372640 1 --2- 172.21.15.98:0/325080491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec8d680 0x55934ec8fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e75372640 1 --2- 172.21.15.98:0/1454508285 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5e600a2d00 0x7f5e600a51c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5e68007fa0 tx=0x7f5e68007900 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e6ffff640 1 --2- 172.21.15.98:0/325080491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec7ea80 0x55934ec8d140 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c02a930 tx=0x7f5e5c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c5a4cab876e349 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e5affd640 1 -- 172.21.15.98:0/325080491 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c040070 con 0x55934ec7ea80 2024-05-09T09:01:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e5affd640 1 -- 172.21.15.98:0/325080491 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e5c004040 con 0x55934ec7ea80 2024-05-09T09:01:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.348+0000 7f5e5affd640 1 -- 172.21.15.98:0/325080491 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c035040 con 0x55934ec7ea80 2024-05-09T09:01:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.349+0000 7f5e79bb6740 1 -- 172.21.15.98:0/325080491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec7ea80 msgr2=0x55934ec8d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.349+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/325080491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec7ea80 0x55934ec8d140 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c02a930 tx=0x7f5e5c005dd0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.349+0000 7f5e79bb6740 1 -- 172.21.15.98:0/325080491 shutdown_connections 2024-05-09T09:01:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.349+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/325080491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec8d680 0x55934ec8fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.349+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/325080491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec7ea80 0x55934ec8d140 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.349+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/325080491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7e160 0x55934ec7e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.349+0000 7f5e79bb6740 1 -- 172.21.15.98:0/325080491 >> 172.21.15.98:0/325080491 conn(0x55934ec7a5a0 msgr2=0x55934ec7a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.349+0000 7f5e79bb6740 1 -- 172.21.15.98:0/325080491 shutdown_connections 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.349+0000 7f5e79bb6740 1 -- 172.21.15.98:0/325080491 wait complete. 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.349+0000 7f5e79bb6740 1 -- start start 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.349+0000 7f5e79bb6740 1 Processor -- start 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e79bb6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7e160 0x55934ec8a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e79bb6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7ea80 0x55934ec8aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e79bb6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec8d680 0x55934ec8afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e79bb6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55934ec90460 con 0x55934ec8d680 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e79bb6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55934eb9eb90 con 0x55934ec7e160 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e79bb6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55934ec905e0 con 0x55934ec7ea80 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e79bb6740 10 client.0 _ll_register_callbacks cb 0x55934ebdada0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e75372640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec8d680 0x55934ec8afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e75372640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec8d680 0x55934ec8afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47388/0 (socket says 172.21.15.98:47388) 2024-05-09T09:01:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e75372640 1 -- 172.21.15.98:0/1781801670 learned_addr learned my addr 172.21.15.98:0/1781801670 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e6ffff640 1 --2- 172.21.15.98:0/1781801670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7ea80 0x55934ec8aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e74b71640 1 --2- 172.21.15.98:0/1781801670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7e160 0x55934ec8a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e75372640 1 -- 172.21.15.98:0/1781801670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7ea80 msgr2=0x55934ec8aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e75372640 1 --2- 172.21.15.98:0/1781801670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7ea80 0x55934ec8aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e75372640 1 -- 172.21.15.98:0/1781801670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7e160 msgr2=0x55934ec8a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e75372640 1 --2- 172.21.15.98:0/1781801670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7e160 0x55934ec8a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e75372640 1 -- 172.21.15.98:0/1781801670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55934e8f73f0 con 0x55934ec8d680 2024-05-09T09:01:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e74b71640 1 --2- 172.21.15.98:0/1781801670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7e160 0x55934ec8a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e75372640 1 --2- 172.21.15.98:0/1781801670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec8d680 0x55934ec8afd0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f5e68011b20 tx=0x7f5e68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40c49a55540c6cf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.350+0000 7f5e4ffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e4ffff640 1 -- 172.21.15.98:0/1781801670 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e68022020 con 0x55934ec8d680 2024-05-09T09:01:29.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e4ffff640 1 -- 172.21.15.98:0/1781801670 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f5e68010c00 con 0x55934ec8d680 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1781801670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e5c004040 con 0x55934ec8d680 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e4ffff640 1 -- 172.21.15.98:0/1781801670 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5e6802d050 con 0x55934ec8d680 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e79bb6740 10 client.18867 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1781801670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55934e8b70f0 con 0x55934ec8d680 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e4ffff640 1 -- 172.21.15.98:0/1781801670 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e5c004040 con 0x55934ec8d680 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e4ffff640 1 -- 172.21.15.98:0/1781801670 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f5e6801d430 con 0x55934ec8d680 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e79bb6740 10 client.18867 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e79bb6740 10 client.18867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e79bb6740 20 client.18867 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/1781801670 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55934ec9da10 0x55934ec9feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e79bb6740 4 client.18867 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1781801670 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55934ebf1430 con 0x55934ec9da10 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.351+0000 7f5e74b71640 1 --2- 172.21.15.98:0/1781801670 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55934ec9da10 0x55934ec9feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.352+0000 7f5e4ffff640 10 client.18867 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:29.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.352+0000 7f5e74b71640 1 --2- 172.21.15.98:0/1781801670 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55934ec9da10 0x55934ec9feb0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc81bf75e27d88a9 server_cookie=e5a97360190a7fc in_seq=0 out_seq=0 2024-05-09T09:01:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.352+0000 7f5e4ffff640 1 -- 172.21.15.98:0/1781801670 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55934ebf1430 con 0x55934ec9da10 2024-05-09T09:01:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.352+0000 7f5e4ffff640 10 client.18867 handle_command_reply: tid=0 2024-05-09T09:01:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.353+0000 7f5e79bb6740 1 client.18867 shutdown 2024-05-09T09:01:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.353+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1781801670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec8d680 msgr2=0x55934ec8afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.353+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/1781801670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec8d680 0x55934ec8afd0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f5e68011b20 tx=0x7f5e68004a40 comp rx=0 tx=0).stop 2024-05-09T09:01:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.353+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1781801670 shutdown_connections 2024-05-09T09:01:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.353+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/1781801670 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55934ec9da10 0x55934ec9feb0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.353+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/1781801670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec8d680 0x55934ec8afd0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.353+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/1781801670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7ea80 0x55934ec8aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.353+0000 7f5e4ffff640 0 client.18867 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.353+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/1781801670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7e160 0x55934ec8a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.353+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1781801670 >> 172.21.15.98:0/1781801670 conn(0x55934ec7a5a0 msgr2=0x55934ec80350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.353+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1781801670 shutdown_connections 2024-05-09T09:01:29.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.353+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1781801670 wait complete. 2024-05-09T09:01:29.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.353+0000 7f5e79bb6740 20 client.18867 trim_cache size 0 max 16384 2024-05-09T09:01:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.370+0000 7f5e79bb6740 1 Processor -- start 2024-05-09T09:01:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.370+0000 7f5e79bb6740 1 -- start start 2024-05-09T09:01:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.370+0000 7f5e79bb6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7e160 0x55934ec993f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.370+0000 7f5e79bb6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7ea80 0x55934ec79660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.370+0000 7f5e79bb6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec8d680 0x55934eca5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.370+0000 7f5e79bb6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701acce0 con 0x55934ec8d680 2024-05-09T09:01:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.370+0000 7f5e79bb6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701ae820 con 0x55934ec7ea80 2024-05-09T09:01:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.370+0000 7f5e79bb6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701ad3b0 con 0x55934ec7e160 2024-05-09T09:01:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.370+0000 7f5e75372640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec8d680 0x55934eca5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.370+0000 7f5e75372640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec8d680 0x55934eca5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47400/0 (socket says 172.21.15.98:47400) 2024-05-09T09:01:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.370+0000 7f5e75372640 1 -- 172.21.15.98:0/31390246 learned_addr learned my addr 172.21.15.98:0/31390246 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e75372640 1 -- 172.21.15.98:0/31390246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7e160 msgr2=0x55934ec993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e74b71640 1 --2- 172.21.15.98:0/31390246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7e160 0x55934ec993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e6ffff640 1 --2- 172.21.15.98:0/31390246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7ea80 0x55934ec79660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e75372640 1 --2- 172.21.15.98:0/31390246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7e160 0x55934ec993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e75372640 1 -- 172.21.15.98:0/31390246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7ea80 msgr2=0x55934ec79660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e75372640 1 --2- 172.21.15.98:0/31390246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7ea80 0x55934ec79660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e75372640 1 -- 172.21.15.98:0/31390246 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55934ec8fa70 con 0x55934ec8d680 2024-05-09T09:01:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e74b71640 1 --2- 172.21.15.98:0/31390246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7e160 0x55934ec993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e6ffff640 1 --2- 172.21.15.98:0/31390246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7ea80 0x55934ec79660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e75372640 1 --2- 172.21.15.98:0/31390246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec8d680 0x55934eca5780 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800ec10 tx=0x7f5e68007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab80f8da12a23a20 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e5a7fc640 1 -- 172.21.15.98:0/31390246 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e68022070 con 0x55934ec8d680 2024-05-09T09:01:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e5a7fc640 1 -- 172.21.15.98:0/31390246 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e68035030 con 0x55934ec8d680 2024-05-09T09:01:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e5a7fc640 1 -- 172.21.15.98:0/31390246 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6801e730 con 0x55934ec8d680 2024-05-09T09:01:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e79bb6740 1 -- 172.21.15.98:0/31390246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec8d680 msgr2=0x55934eca5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/31390246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec8d680 0x55934eca5780 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800ec10 tx=0x7f5e68007000 comp rx=0 tx=0).stop 2024-05-09T09:01:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e79bb6740 1 -- 172.21.15.98:0/31390246 shutdown_connections 2024-05-09T09:01:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/31390246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec8d680 0x55934eca5780 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/31390246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7ea80 0x55934ec79660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.371+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/31390246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec7e160 0x55934ec993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e79bb6740 1 -- 172.21.15.98:0/31390246 >> 172.21.15.98:0/31390246 conn(0x55934ec7a5a0 msgr2=0x55934ec78f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e79bb6740 1 -- 172.21.15.98:0/31390246 shutdown_connections 2024-05-09T09:01:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e79bb6740 1 -- 172.21.15.98:0/31390246 wait complete. 2024-05-09T09:01:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e79bb6740 1 -- start start 2024-05-09T09:01:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e79bb6740 1 Processor -- start 2024-05-09T09:01:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e79bb6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7e160 0x55934ec95d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e79bb6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec7ea80 0x55934ec910b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e79bb6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec8d680 0x55934ec91610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e79bb6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701ae150 con 0x55934ec7ea80 2024-05-09T09:01:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e79bb6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701ada80 con 0x55934ec7e160 2024-05-09T09:01:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e79bb6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701aace0 con 0x55934ec8d680 2024-05-09T09:01:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e79bb6740 10 client.0 _ll_register_callbacks cb 0x7f5e68007c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec7ea80 0x55934ec910b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec7ea80 0x55934ec910b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47410/0 (socket says 172.21.15.98:47410) 2024-05-09T09:01:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e6ffff640 1 -- 172.21.15.98:0/1240330559 learned_addr learned my addr 172.21.15.98:0/1240330559 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.372+0000 7f5e74b71640 1 --2- 172.21.15.98:0/1240330559 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7e160 0x55934ec95d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e6ffff640 1 -- 172.21.15.98:0/1240330559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec8d680 msgr2=0x55934ec91610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e6ffff640 1 --2- 172.21.15.98:0/1240330559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec8d680 0x55934ec91610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e6ffff640 1 -- 172.21.15.98:0/1240330559 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7e160 msgr2=0x55934ec95d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e6ffff640 1 --2- 172.21.15.98:0/1240330559 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7e160 0x55934ec95d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e6ffff640 1 -- 172.21.15.98:0/1240330559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55934ec91e40 con 0x55934ec7ea80 2024-05-09T09:01:29.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e6ffff640 1 --2- 172.21.15.98:0/1240330559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec7ea80 0x55934ec910b0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c005e40 tx=0x7f5e5c032f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46846a9fa4d7bf89 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:29.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e4ffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:29.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e4ffff640 1 -- 172.21.15.98:0/1240330559 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c04c070 con 0x55934ec7ea80 2024-05-09T09:01:29.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e4ffff640 1 -- 172.21.15.98:0/1240330559 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f5e5c0319c0 con 0x55934ec7ea80 2024-05-09T09:01:29.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1240330559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e68035030 con 0x55934ec7ea80 2024-05-09T09:01:29.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e4ffff640 1 -- 172.21.15.98:0/1240330559 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5e5c058050 con 0x55934ec7ea80 2024-05-09T09:01:29.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e79bb6740 10 client.18873 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:29.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1240330559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55934ed04030 con 0x55934ec7ea80 2024-05-09T09:01:29.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e4ffff640 1 -- 172.21.15.98:0/1240330559 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e68035030 con 0x55934ec7ea80 2024-05-09T09:01:29.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e4ffff640 1 -- 172.21.15.98:0/1240330559 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f5e5c03a650 con 0x55934ec7ea80 2024-05-09T09:01:29.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e79bb6740 10 client.18873 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:29.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e79bb6740 10 client.18873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:29.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e79bb6740 20 client.18873 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:29.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/1240330559 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55934ed04360 0x55934ed06750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:29.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e79bb6740 4 client.18873 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:01:29.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.373+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1240330559 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55934ebeb780 con 0x55934ed04360 2024-05-09T09:01:29.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e74b71640 1 --2- 172.21.15.98:0/1240330559 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55934ed04360 0x55934ed06750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:29.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e4ffff640 10 client.18873 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:29.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e74b71640 1 --2- 172.21.15.98:0/1240330559 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55934ed04360 0x55934ed06750 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c08df5754da728d2 server_cookie=9b30d57cd95f17dd in_seq=0 out_seq=0 2024-05-09T09:01:29.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e4ffff640 1 -- 172.21.15.98:0/1240330559 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55934ebeb780 con 0x55934ed04360 2024-05-09T09:01:29.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e4ffff640 10 client.18873 handle_command_reply: tid=0 2024-05-09T09:01:29.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e79bb6740 1 client.18873 shutdown 2024-05-09T09:01:29.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1240330559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec7ea80 msgr2=0x55934ec910b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/1240330559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec7ea80 0x55934ec910b0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c005e40 tx=0x7f5e5c032f60 comp rx=0 tx=0).stop 2024-05-09T09:01:29.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1240330559 shutdown_connections 2024-05-09T09:01:29.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/1240330559 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55934ed04360 0x55934ed06750 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/1240330559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55934ec8d680 0x55934ec91610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/1240330559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55934ec7ea80 0x55934ec910b0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e79bb6740 1 --2- 172.21.15.98:0/1240330559 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55934ec7e160 0x55934ec95d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1240330559 >> 172.21.15.98:0/1240330559 conn(0x55934ec7a5a0 msgr2=0x55934ec8ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:29.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e4ffff640 0 client.18873 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:29.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.374+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1240330559 shutdown_connections 2024-05-09T09:01:29.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.375+0000 7f5e79bb6740 1 -- 172.21.15.98:0/1240330559 wait complete. 2024-05-09T09:01:29.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.375+0000 7f5e79bb6740 20 client.18873 trim_cache size 0 max 16384 2024-05-09T09:01:29.381 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:01:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T09:01:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T09:01:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "8c645698-5e0d-48eb-bcdc-5fba2d5b407e": { 2024-05-09T09:01:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T09:01:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "8c645698-5e0d-48eb-bcdc-5fba2d5b407e", 2024-05-09T09:01:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T09:01:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:01:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:01:29.382 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:01:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.375+0000 7f5e76dd5640 1 -- 172.21.15.98:0/1454508285 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5e600a2d00 msgr2=0x7f5e600a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.375+0000 7f5e76dd5640 1 --2- 172.21.15.98:0/1454508285 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5e600a2d00 0x7f5e600a51c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5e68007fa0 tx=0x7f5e68007900 comp rx=0 tx=0).stop 2024-05-09T09:01:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.375+0000 7f5e76dd5640 1 -- 172.21.15.98:0/1454508285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e701000f0 msgr2=0x7f5e70199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.375+0000 7f5e76dd5640 1 --2- 172.21.15.98:0/1454508285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e701000f0 0x7f5e70199170 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f5e6400b4a0 tx=0x7f5e6400b960 comp rx=0 tx=0).stop 2024-05-09T09:01:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.375+0000 7f5e76dd5640 1 -- 172.21.15.98:0/1454508285 shutdown_connections 2024-05-09T09:01:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.375+0000 7f5e76dd5640 1 --2- 172.21.15.98:0/1454508285 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5e600a2d00 0x7f5e600a51c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.375+0000 7f5e76dd5640 1 --2- 172.21.15.98:0/1454508285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e70199c40 0x7f5e701a41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.375+0000 7f5e76dd5640 1 --2- 172.21.15.98:0/1454508285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e701051c0 0x7f5e701996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.375+0000 7f5e76dd5640 1 --2- 172.21.15.98:0/1454508285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e701000f0 0x7f5e70199170 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.375+0000 7f5e76dd5640 1 -- 172.21.15.98:0/1454508285 >> 172.21.15.98:0/1454508285 conn(0x7f5e70076cf0 msgr2=0x7f5e700fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:29.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.375+0000 7f5e76dd5640 1 -- 172.21.15.98:0/1454508285 shutdown_connections 2024-05-09T09:01:29.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:29.375+0000 7f5e76dd5640 1 -- 172.21.15.98:0/1454508285 wait complete. 2024-05-09T09:01:29.386 INFO:tasks.cephfs.filesystem:scrub status for tag:8c645698-5e0d-48eb-bcdc-5fba2d5b407e - {'path': '/', 'tag': '8c645698-5e0d-48eb-bcdc-5fba2d5b407e', 'options': 'recursive,force'} 2024-05-09T09:01:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:30 smithi142 ceph-mon[34625]: pgmap v340: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-09T09:01:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:30 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T09:01:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:30 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T09:01:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:30 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T09:01:30.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:30 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T09:01:30.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:30 smithi105 ceph-mon[34498]: pgmap v340: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-09T09:01:30.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:30 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T09:01:30.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:30 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T09:01:30.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:30 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T09:01:30.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:30 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T09:01:30.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:30 smithi098 ceph-mon[30092]: pgmap v340: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-09T09:01:30.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:30 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T09:01:30.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:30 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T09:01:30.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:30 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T09:01:30.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:30 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T09:01:32.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:32 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T09:01:32.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:32 smithi105 ceph-mon[34498]: pgmap v341: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 170 B/s wr, 13 op/s 2024-05-09T09:01:32.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:32 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T09:01:32.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:32 smithi098 ceph-mon[30092]: pgmap v341: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 170 B/s wr, 13 op/s 2024-05-09T09:01:32.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:32 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T09:01:32.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:32 smithi142 ceph-mon[34625]: pgmap v341: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 170 B/s wr, 13 op/s 2024-05-09T09:01:34.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:34 smithi105 ceph-mon[34498]: pgmap v342: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:01:34.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:34 smithi098 ceph-mon[30092]: pgmap v342: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:01:34.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:34 smithi142 ceph-mon[34625]: pgmap v342: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:01:36.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:36 smithi105 ceph-mon[34498]: pgmap v343: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:01:36.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:36 smithi098 ceph-mon[30092]: pgmap v343: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:01:36.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:36 smithi142 ceph-mon[34625]: pgmap v343: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T09:01:38.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:38 smithi105 ceph-mon[34498]: pgmap v344: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:01:38.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:38 smithi098 ceph-mon[30092]: pgmap v344: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:01:38.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:38 smithi142 ceph-mon[34625]: pgmap v344: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:01:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.838+0000 7f573cbdb640 1 -- 172.21.15.98:0/263054789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5738100110 msgr2=0x7f57381004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.838+0000 7f573cbdb640 1 --2- 172.21.15.98:0/263054789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5738100110 0x7f57381004f0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f5728009900 tx=0x7f572802a410 comp rx=0 tx=0).stop 2024-05-09T09:01:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.838+0000 7f573cbdb640 1 -- 172.21.15.98:0/263054789 shutdown_connections 2024-05-09T09:01:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.838+0000 7f573cbdb640 1 --2- 172.21.15.98:0/263054789 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57381051e0 0x7f573810cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.838+0000 7f573cbdb640 1 --2- 172.21.15.98:0/263054789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5738100ac0 0x7f5738104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.838+0000 7f573cbdb640 1 --2- 172.21.15.98:0/263054789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5738100110 0x7f57381004f0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.838+0000 7f573cbdb640 1 -- 172.21.15.98:0/263054789 >> 172.21.15.98:0/263054789 conn(0x7f5738076c50 msgr2=0x7f57380fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:39.839 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.838+0000 7f573cbdb640 1 -- 172.21.15.98:0/263054789 shutdown_connections 2024-05-09T09:01:39.839 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.838+0000 7f573cbdb640 1 -- 172.21.15.98:0/263054789 wait complete. 2024-05-09T09:01:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.839+0000 7f573cbdb640 1 Processor -- start 2024-05-09T09:01:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.839+0000 7f573cbdb640 1 -- start start 2024-05-09T09:01:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.839+0000 7f573cbdb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5738100110 0x7f5738198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f573cbdb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57381051e0 0x7f5738199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f573cbdb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57381999d0 0x7f57381a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f573cbdb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5738110b60 con 0x7f57381051e0 2024-05-09T09:01:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f573cbdb640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f57381109e0 con 0x7f57381999d0 2024-05-09T09:01:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f573cbdb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5738110ce0 con 0x7f5738100110 2024-05-09T09:01:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f572ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57381051e0 0x7f5738199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f572ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57381051e0 0x7f5738199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58914/0 (socket says 172.21.15.98:58914) 2024-05-09T09:01:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f572ffff640 1 -- 172.21.15.98:0/1923927828 learned_addr learned my addr 172.21.15.98:0/1923927828 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f573659c640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5738100110 0x7f5738198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f5736d9d640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57381999d0 0x7f57381a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f572ffff640 1 -- 172.21.15.98:0/1923927828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5738100110 msgr2=0x7f5738198f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f572ffff640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5738100110 0x7f5738198f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f572ffff640 1 -- 172.21.15.98:0/1923927828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57381999d0 msgr2=0x7f57381a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f572ffff640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57381999d0 0x7f57381a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f572ffff640 1 -- 172.21.15.98:0/1923927828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57381a4750 con 0x7f57381051e0 2024-05-09T09:01:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f573659c640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5738100110 0x7f5738198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f5736d9d640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57381999d0 0x7f57381a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f572ffff640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57381051e0 0x7f5738199440 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f572000d740 tx=0x7f572000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f572f7fe640 1 -- 172.21.15.98:0/1923927828 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5720017070 con 0x7f57381051e0 2024-05-09T09:01:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f572f7fe640 1 -- 172.21.15.98:0/1923927828 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f572000de50 con 0x7f57381051e0 2024-05-09T09:01:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f573cbdb640 1 -- 172.21.15.98:0/1923927828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57381a4a40 con 0x7f57381051e0 2024-05-09T09:01:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f572f7fe640 1 -- 172.21.15.98:0/1923927828 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5720004e00 con 0x7f57381051e0 2024-05-09T09:01:39.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.840+0000 7f573cbdb640 1 -- 172.21.15.98:0/1923927828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57381a4f40 con 0x7f57381051e0 2024-05-09T09:01:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.843+0000 7f572f7fe640 1 -- 172.21.15.98:0/1923927828 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5720004fa0 con 0x7f57381051e0 2024-05-09T09:01:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.844+0000 7f573f9bd740 1 Processor -- start 2024-05-09T09:01:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.844+0000 7f573f9bd740 1 -- start start 2024-05-09T09:01:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.844+0000 7f573f9bd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585450 0x5618fc585830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.844+0000 7f573f9bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585d70 0x5618fc5900f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.844+0000 7f573f9bd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 0x5618fc592a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.844+0000 7f573f9bd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5618fc4ea340 con 0x5618fc585450 2024-05-09T09:01:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.844+0000 7f573f9bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5618fc492290 con 0x5618fc585d70 2024-05-09T09:01:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.844+0000 7f573f9bd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5618fc4e6300 con 0x5618fc590630 2024-05-09T09:01:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.844+0000 7f5736d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585450 0x5618fc585830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.844+0000 7f5736d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585450 0x5618fc585830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58920/0 (socket says 172.21.15.98:58920) 2024-05-09T09:01:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.844+0000 7f5736d9d640 1 -- 172.21.15.98:0/3075062733 learned_addr learned my addr 172.21.15.98:0/3075062733 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.844+0000 7f573659c640 1 --2- 172.21.15.98:0/3075062733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585d70 0x5618fc5900f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.844+0000 7f572f7fe640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f57240a2da0 0x7f57240a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.844+0000 7f572ffff640 1 --2- 172.21.15.98:0/3075062733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 0x5618fc592a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.845+0000 7f5736d9d640 1 -- 172.21.15.98:0/3075062733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 msgr2=0x5618fc592a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.845+0000 7f5736d9d640 1 --2- 172.21.15.98:0/3075062733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 0x5618fc592a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.845+0000 7f5736d9d640 1 -- 172.21.15.98:0/3075062733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585d70 msgr2=0x5618fc5900f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.845+0000 7f5736d9d640 1 --2- 172.21.15.98:0/3075062733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585d70 0x5618fc5900f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.845+0000 7f5736d9d640 1 -- 172.21.15.98:0/3075062733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618fc4b68e0 con 0x5618fc585450 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.845+0000 7f572ffff640 1 --2- 172.21.15.98:0/3075062733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 0x5618fc592a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.845+0000 7f572f7fe640 1 -- 172.21.15.98:0/1923927828 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f57200cf310 con 0x7f57381051e0 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.845+0000 7f5736d9d640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f57240a2da0 0x7f57240a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.845+0000 7f5736d9d640 1 --2- 172.21.15.98:0/3075062733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585450 0x5618fc585830 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f5730009fa0 tx=0x7f573000f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369cc0a8ffe6e72b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.845+0000 7f572cff9640 1 -- 172.21.15.98:0/3075062733 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730012890 con 0x5618fc585450 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.845+0000 7f572cff9640 1 -- 172.21.15.98:0/3075062733 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f573001aa20 con 0x5618fc585450 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.845+0000 7f572cff9640 1 -- 172.21.15.98:0/3075062733 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730023970 con 0x5618fc585450 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.846+0000 7f573f9bd740 1 -- 172.21.15.98:0/3075062733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585450 msgr2=0x5618fc585830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.846+0000 7f5736d9d640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f57240a2da0 0x7f57240a5260 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f573001e000 tx=0x7f573001b000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.846+0000 7f573f9bd740 1 --2- 172.21.15.98:0/3075062733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585450 0x5618fc585830 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f5730009fa0 tx=0x7f573000f7f0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.846+0000 7f573f9bd740 1 -- 172.21.15.98:0/3075062733 shutdown_connections 2024-05-09T09:01:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.846+0000 7f573f9bd740 1 --2- 172.21.15.98:0/3075062733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 0x5618fc592a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.846+0000 7f573f9bd740 1 --2- 172.21.15.98:0/3075062733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585d70 0x5618fc5900f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.846+0000 7f573f9bd740 1 --2- 172.21.15.98:0/3075062733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585450 0x5618fc585830 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.846+0000 7f573f9bd740 1 -- 172.21.15.98:0/3075062733 >> 172.21.15.98:0/3075062733 conn(0x5618fc581960 msgr2=0x5618fc581d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.846+0000 7f573f9bd740 1 -- 172.21.15.98:0/3075062733 shutdown_connections 2024-05-09T09:01:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.846+0000 7f573f9bd740 1 -- 172.21.15.98:0/3075062733 wait complete. 2024-05-09T09:01:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.846+0000 7f573f9bd740 1 -- start start 2024-05-09T09:01:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f573f9bd740 1 Processor -- start 2024-05-09T09:01:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f573f9bd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc585450 0x5618fc58d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f573f9bd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585d70 0x5618fc58da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f573f9bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc590630 0x5618fc58df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f573f9bd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5618fc593260 con 0x5618fc585d70 2024-05-09T09:01:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f573f9bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5618fc4a5790 con 0x5618fc590630 2024-05-09T09:01:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f573f9bd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5618fc5933e0 con 0x5618fc585450 2024-05-09T09:01:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f573f9bd740 10 client.0 _ll_register_callbacks cb 0x5618fc4e0a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f572ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585d70 0x5618fc58da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f572ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585d70 0x5618fc58da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58924/0 (socket says 172.21.15.98:58924) 2024-05-09T09:01:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f572ffff640 1 -- 172.21.15.98:0/3288276829 learned_addr learned my addr 172.21.15.98:0/3288276829 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f573659c640 1 --2- 172.21.15.98:0/3288276829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc585450 0x5618fc58d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f5736d9d640 1 --2- 172.21.15.98:0/3288276829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc590630 0x5618fc58df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f572ffff640 1 -- 172.21.15.98:0/3288276829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc585450 msgr2=0x5618fc58d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f572ffff640 1 --2- 172.21.15.98:0/3288276829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc585450 0x5618fc58d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f572ffff640 1 -- 172.21.15.98:0/3288276829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc590630 msgr2=0x5618fc58df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f572ffff640 1 --2- 172.21.15.98:0/3288276829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc590630 0x5618fc58df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f572ffff640 1 -- 172.21.15.98:0/3288276829 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618fc1fe210 con 0x5618fc585d70 2024-05-09T09:01:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f573659c640 1 --2- 172.21.15.98:0/3288276829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc585450 0x5618fc58d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f572ffff640 1 --2- 172.21.15.98:0/3288276829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585d70 0x5618fc58da30 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f57200ccc50 tx=0x7f572006b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52bdb84d80284a5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f571dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.847+0000 7f571dffb640 1 -- 172.21.15.98:0/3288276829 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5720020040 con 0x5618fc585d70 2024-05-09T09:01:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.848+0000 7f571dffb640 1 -- 172.21.15.98:0/3288276829 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f57200cc8d0 con 0x5618fc585d70 2024-05-09T09:01:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.848+0000 7f573f9bd740 1 -- 172.21.15.98:0/3288276829 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f573001aa20 con 0x5618fc585d70 2024-05-09T09:01:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.848+0000 7f571dffb640 1 -- 172.21.15.98:0/3288276829 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f57200cf170 con 0x5618fc585d70 2024-05-09T09:01:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.848+0000 7f573f9bd740 10 client.18885 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.848+0000 7f573f9bd740 1 -- 172.21.15.98:0/3288276829 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5618fc2ca620 con 0x5618fc585d70 2024-05-09T09:01:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.848+0000 7f571dffb640 1 -- 172.21.15.98:0/3288276829 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f573001aa20 con 0x5618fc585d70 2024-05-09T09:01:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.848+0000 7f571dffb640 1 -- 172.21.15.98:0/3288276829 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f572006c070 con 0x5618fc585d70 2024-05-09T09:01:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.848+0000 7f573f9bd740 10 client.18885 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.848+0000 7f573f9bd740 10 client.18885 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.848+0000 7f573f9bd740 20 client.18885 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.848+0000 7f573f9bd740 1 --2- 172.21.15.98:0/3288276829 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5618fc5a0220 0x5618fc5a2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.848+0000 7f573f9bd740 4 client.18885 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:01:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.848+0000 7f573f9bd740 1 -- 172.21.15.98:0/3288276829 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618fc4f7a70 con 0x5618fc5a0220 2024-05-09T09:01:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.848+0000 7f573659c640 1 --2- 172.21.15.98:0/3288276829 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5618fc5a0220 0x5618fc5a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.849+0000 7f571dffb640 10 client.18885 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.849+0000 7f573659c640 1 --2- 172.21.15.98:0/3288276829 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5618fc5a0220 0x5618fc5a2610 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5f2da8fde0eac0b server_cookie=49b1ff5a632071da in_seq=0 out_seq=0 2024-05-09T09:01:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.849+0000 7f571dffb640 1 -- 172.21.15.98:0/3288276829 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5618fc4f7a70 con 0x5618fc5a0220 2024-05-09T09:01:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.849+0000 7f571dffb640 10 client.18885 handle_command_reply: tid=0 2024-05-09T09:01:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.849+0000 7f573f9bd740 1 client.18885 shutdown 2024-05-09T09:01:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.849+0000 7f573f9bd740 1 -- 172.21.15.98:0/3288276829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585d70 msgr2=0x5618fc58da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.849+0000 7f573f9bd740 1 --2- 172.21.15.98:0/3288276829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585d70 0x5618fc58da30 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f57200ccc50 tx=0x7f572006b280 comp rx=0 tx=0).stop 2024-05-09T09:01:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.849+0000 7f573f9bd740 1 -- 172.21.15.98:0/3288276829 shutdown_connections 2024-05-09T09:01:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.849+0000 7f573f9bd740 1 --2- 172.21.15.98:0/3288276829 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5618fc5a0220 0x5618fc5a2610 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.850+0000 7f573f9bd740 1 --2- 172.21.15.98:0/3288276829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc590630 0x5618fc58df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.850+0000 7f571dffb640 0 client.18885 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.850+0000 7f573f9bd740 1 --2- 172.21.15.98:0/3288276829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585d70 0x5618fc58da30 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.850+0000 7f573f9bd740 1 --2- 172.21.15.98:0/3288276829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc585450 0x5618fc58d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.850+0000 7f573f9bd740 1 -- 172.21.15.98:0/3288276829 >> 172.21.15.98:0/3288276829 conn(0x5618fc581960 msgr2=0x5618fc5876e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.850+0000 7f573f9bd740 1 -- 172.21.15.98:0/3288276829 shutdown_connections 2024-05-09T09:01:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.850+0000 7f573f9bd740 1 -- 172.21.15.98:0/3288276829 wait complete. 2024-05-09T09:01:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.850+0000 7f573f9bd740 20 client.18885 trim_cache size 0 max 16384 2024-05-09T09:01:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.864+0000 7f573f9bd740 1 Processor -- start 2024-05-09T09:01:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.864+0000 7f573f9bd740 1 -- start start 2024-05-09T09:01:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f573f9bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585450 0x5618fc58fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f573f9bd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585d70 0x5618fc59c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f573f9bd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 0x5618fc5a7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f573f9bd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57381aedc0 con 0x5618fc585d70 2024-05-09T09:01:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f573f9bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f57381b0900 con 0x5618fc585450 2024-05-09T09:01:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f573f9bd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f57381af490 con 0x5618fc590630 2024-05-09T09:01:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f572ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585d70 0x5618fc59c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f572ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585d70 0x5618fc59c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58940/0 (socket says 172.21.15.98:58940) 2024-05-09T09:01:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f572ffff640 1 -- 172.21.15.98:0/3336695512 learned_addr learned my addr 172.21.15.98:0/3336695512 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f573659c640 1 --2- 172.21.15.98:0/3336695512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585450 0x5618fc58fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f5736d9d640 1 --2- 172.21.15.98:0/3336695512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 0x5618fc5a7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f572ffff640 1 -- 172.21.15.98:0/3336695512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 msgr2=0x5618fc5a7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f572ffff640 1 --2- 172.21.15.98:0/3336695512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 0x5618fc5a7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f572ffff640 1 -- 172.21.15.98:0/3336695512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585450 msgr2=0x5618fc58fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f572ffff640 1 --2- 172.21.15.98:0/3336695512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585450 0x5618fc58fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f572ffff640 1 -- 172.21.15.98:0/3336695512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618fc5a2980 con 0x5618fc585d70 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f5736d9d640 1 --2- 172.21.15.98:0/3336695512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 0x5618fc5a7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f572ffff640 1 --2- 172.21.15.98:0/3336695512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585d70 0x5618fc59c120 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f572009b120 tx=0x7f57200a40e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1000b0177a4f3f5c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f571ffff640 1 -- 172.21.15.98:0/3336695512 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5720014070 con 0x5618fc585d70 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f571ffff640 1 -- 172.21.15.98:0/3336695512 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f572009bb20 con 0x5618fc585d70 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.865+0000 7f571ffff640 1 -- 172.21.15.98:0/3336695512 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5720020050 con 0x5618fc585d70 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.866+0000 7f573f9bd740 1 -- 172.21.15.98:0/3336695512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585d70 msgr2=0x5618fc59c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.866+0000 7f573f9bd740 1 --2- 172.21.15.98:0/3336695512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585d70 0x5618fc59c120 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f572009b120 tx=0x7f57200a40e0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.866+0000 7f573f9bd740 1 -- 172.21.15.98:0/3336695512 shutdown_connections 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.866+0000 7f573f9bd740 1 --2- 172.21.15.98:0/3336695512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 0x5618fc5a7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.866+0000 7f573f9bd740 1 --2- 172.21.15.98:0/3336695512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585d70 0x5618fc59c120 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.866+0000 7f573f9bd740 1 --2- 172.21.15.98:0/3336695512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585450 0x5618fc58fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.866+0000 7f573f9bd740 1 -- 172.21.15.98:0/3336695512 >> 172.21.15.98:0/3336695512 conn(0x5618fc581960 msgr2=0x5618fc580a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.866+0000 7f573f9bd740 1 -- 172.21.15.98:0/3336695512 shutdown_connections 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.866+0000 7f573f9bd740 1 -- 172.21.15.98:0/3336695512 wait complete. 2024-05-09T09:01:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.866+0000 7f573f9bd740 1 -- start start 2024-05-09T09:01:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.866+0000 7f573f9bd740 1 Processor -- start 2024-05-09T09:01:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573f9bd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585450 0x5618fc5fea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573f9bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585d70 0x5618fc5fef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573f9bd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 0x5618fc594940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573f9bd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57381b0230 con 0x5618fc585450 2024-05-09T09:01:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573f9bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f57381afb60 con 0x5618fc585d70 2024-05-09T09:01:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573f9bd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f57381acdc0 con 0x5618fc590630 2024-05-09T09:01:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573f9bd740 10 client.0 _ll_register_callbacks cb 0x7f5720097720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585450 0x5618fc5fea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585450 0x5618fc5fea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58954/0 (socket says 172.21.15.98:58954) 2024-05-09T09:01:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573659c640 1 -- 172.21.15.98:0/2934834440 learned_addr learned my addr 172.21.15.98:0/2934834440 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f572ffff640 1 --2- 172.21.15.98:0/2934834440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585d70 0x5618fc5fef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f5736d9d640 1 --2- 172.21.15.98:0/2934834440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 0x5618fc594940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573659c640 1 -- 172.21.15.98:0/2934834440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 msgr2=0x5618fc594940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573659c640 1 --2- 172.21.15.98:0/2934834440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 0x5618fc594940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573659c640 1 -- 172.21.15.98:0/2934834440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585d70 msgr2=0x5618fc5fef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573659c640 1 --2- 172.21.15.98:0/2934834440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585d70 0x5618fc5fef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573659c640 1 -- 172.21.15.98:0/2934834440 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618fc595030 con 0x5618fc585450 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f572ffff640 1 --2- 172.21.15.98:0/2934834440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585d70 0x5618fc5fef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573659c640 1 --2- 172.21.15.98:0/2934834440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585450 0x5618fc5fea20 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f5728031550 tx=0x7f5728004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea5ef5c05775d95 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f571dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f571dffb640 1 -- 172.21.15.98:0/2934834440 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5728035040 con 0x5618fc585450 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f571dffb640 1 -- 172.21.15.98:0/2934834440 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f5728005240 con 0x5618fc585450 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573f9bd740 1 -- 172.21.15.98:0/2934834440 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f572009bb20 con 0x5618fc585450 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f571dffb640 1 -- 172.21.15.98:0/2934834440 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5728054050 con 0x5618fc585450 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573f9bd740 10 client.18891 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.867+0000 7f573f9bd740 1 -- 172.21.15.98:0/2934834440 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5618fc6072d0 con 0x5618fc585450 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.868+0000 7f571dffb640 1 -- 172.21.15.98:0/2934834440 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f572009bb20 con 0x5618fc585450 2024-05-09T09:01:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.868+0000 7f571dffb640 1 -- 172.21.15.98:0/2934834440 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f57280047e0 con 0x5618fc585450 2024-05-09T09:01:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.868+0000 7f573f9bd740 10 client.18891 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.868+0000 7f573f9bd740 10 client.18891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.868+0000 7f573f9bd740 20 client.18891 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.868+0000 7f573f9bd740 1 --2- 172.21.15.98:0/2934834440 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5618fc607a50 0x5618fc609e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.868+0000 7f573f9bd740 4 client.18891 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:01:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.868+0000 7f573f9bd740 1 -- 172.21.15.98:0/2934834440 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5618fc4f1dc0 con 0x5618fc607a50 2024-05-09T09:01:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.868+0000 7f572ffff640 1 --2- 172.21.15.98:0/2934834440 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5618fc607a50 0x5618fc609e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.868+0000 7f571dffb640 10 client.18891 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.868+0000 7f572ffff640 1 --2- 172.21.15.98:0/2934834440 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5618fc607a50 0x5618fc609e70 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f424defa70c6707a server_cookie=43e67beb120a5f2d in_seq=0 out_seq=0 2024-05-09T09:01:39.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.872+0000 7f571dffb640 1 -- 172.21.15.98:0/2934834440 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5618fc4f1dc0 con 0x5618fc607a50 2024-05-09T09:01:39.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.872+0000 7f571dffb640 10 client.18891 handle_command_reply: tid=0 2024-05-09T09:01:39.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.872+0000 7f573f9bd740 1 client.18891 shutdown 2024-05-09T09:01:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.872+0000 7f573f9bd740 1 -- 172.21.15.98:0/2934834440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585450 msgr2=0x5618fc5fea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.872+0000 7f573f9bd740 1 --2- 172.21.15.98:0/2934834440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585450 0x5618fc5fea20 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f5728031550 tx=0x7f5728004610 comp rx=0 tx=0).stop 2024-05-09T09:01:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.872+0000 7f573f9bd740 1 -- 172.21.15.98:0/2934834440 shutdown_connections 2024-05-09T09:01:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.872+0000 7f573f9bd740 1 --2- 172.21.15.98:0/2934834440 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5618fc607a50 0x5618fc609e70 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.872+0000 7f573f9bd740 1 --2- 172.21.15.98:0/2934834440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5618fc590630 0x5618fc594940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.872+0000 7f571dffb640 0 client.18891 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.872+0000 7f573f9bd740 1 --2- 172.21.15.98:0/2934834440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5618fc585d70 0x5618fc5fef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.872+0000 7f573f9bd740 1 --2- 172.21.15.98:0/2934834440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5618fc585450 0x5618fc5fea20 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.872+0000 7f573f9bd740 1 -- 172.21.15.98:0/2934834440 >> 172.21.15.98:0/2934834440 conn(0x5618fc581960 msgr2=0x5618fc58d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573f9bd740 1 -- 172.21.15.98:0/2934834440 shutdown_connections 2024-05-09T09:01:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573f9bd740 1 -- 172.21.15.98:0/2934834440 wait complete. 2024-05-09T09:01:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573f9bd740 20 client.18891 trim_cache size 0 max 16384 2024-05-09T09:01:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T09:01:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T09:01:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T09:01:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T09:01:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573cbdb640 1 -- 172.21.15.98:0/1923927828 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f57240a2da0 msgr2=0x7f57240a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573cbdb640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f57240a2da0 0x7f57240a5260 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f573001e000 tx=0x7f573001b000 comp rx=0 tx=0).stop 2024-05-09T09:01:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573cbdb640 1 -- 172.21.15.98:0/1923927828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57381051e0 msgr2=0x7f5738199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573cbdb640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57381051e0 0x7f5738199440 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f572000d740 tx=0x7f572000dc00 comp rx=0 tx=0).stop 2024-05-09T09:01:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573cbdb640 1 -- 172.21.15.98:0/1923927828 shutdown_connections 2024-05-09T09:01:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573cbdb640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f57240a2da0 0x7f57240a5260 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573cbdb640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57381999d0 0x7f57381a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573cbdb640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57381051e0 0x7f5738199440 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573cbdb640 1 --2- 172.21.15.98:0/1923927828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5738100110 0x7f5738198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573cbdb640 1 -- 172.21.15.98:0/1923927828 >> 172.21.15.98:0/1923927828 conn(0x7f5738076c50 msgr2=0x7f57381039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573cbdb640 1 -- 172.21.15.98:0/1923927828 shutdown_connections 2024-05-09T09:01:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:39.873+0000 7f573cbdb640 1 -- 172.21.15.98:0/1923927828 wait complete. 2024-05-09T09:01:40.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:40 smithi105 ceph-mon[34498]: pgmap v345: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:01:40.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:40 smithi098 ceph-mon[30092]: pgmap v345: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:01:40.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:40 smithi142 ceph-mon[34625]: pgmap v345: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:01:42.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:42 smithi105 ceph-mon[34498]: pgmap v346: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:01:42.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:42 smithi098 ceph-mon[30092]: pgmap v346: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:01:42.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:42 smithi142 ceph-mon[34625]: pgmap v346: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-09T09:01:44.658 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:44 smithi142 ceph-mon[34625]: pgmap v347: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 10 op/s 2024-05-09T09:01:44.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:44 smithi105 ceph-mon[34498]: pgmap v347: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 10 op/s 2024-05-09T09:01:44.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:44 smithi098 ceph-mon[30092]: pgmap v347: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 10 op/s 2024-05-09T09:01:46.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:46 smithi105 ceph-mon[34498]: pgmap v348: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-09T09:01:46.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:46 smithi098 ceph-mon[30092]: pgmap v348: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-09T09:01:46.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:46 smithi142 ceph-mon[34625]: pgmap v348: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-09T09:01:48.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:48 smithi105 ceph-mon[34498]: pgmap v349: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:01:48.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:48 smithi098 ceph-mon[30092]: pgmap v349: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:01:48.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:48 smithi142 ceph-mon[34625]: pgmap v349: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:01:50.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:50 smithi105 ceph-mon[34498]: pgmap v350: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:01:50.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:50 smithi098 ceph-mon[30092]: pgmap v350: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:01:50.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:50 smithi142 ceph-mon[34625]: pgmap v350: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:01:52.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:52 smithi105 ceph-mon[34498]: pgmap v351: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:01:52.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:52 smithi098 ceph-mon[30092]: pgmap v351: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:01:52.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:52 smithi142 ceph-mon[34625]: pgmap v351: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:01:52.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.972+0000 7f980bbd5640 1 -- 172.21.15.98:0/3448861177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9804101840 msgr2=0x7f9804101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:52.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.972+0000 7f980bbd5640 1 --2- 172.21.15.98:0/3448861177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9804101840 0x7f9804101cc0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009930 tx=0x7f97f402a420 comp rx=0 tx=0).stop 2024-05-09T09:01:52.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.972+0000 7f980bbd5640 1 -- 172.21.15.98:0/3448861177 shutdown_connections 2024-05-09T09:01:52.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.972+0000 7f980bbd5640 1 --2- 172.21.15.98:0/3448861177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9804102200 0x7f980410e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.972+0000 7f980bbd5640 1 --2- 172.21.15.98:0/3448861177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9804101840 0x7f9804101cc0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.972+0000 7f980bbd5640 1 --2- 172.21.15.98:0/3448861177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9804107850 0x7f9804107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.972+0000 7f980bbd5640 1 -- 172.21.15.98:0/3448861177 >> 172.21.15.98:0/3448861177 conn(0x7f9804078cc0 msgr2=0x7f98040790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:52.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.972+0000 7f980bbd5640 1 -- 172.21.15.98:0/3448861177 shutdown_connections 2024-05-09T09:01:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.972+0000 7f980bbd5640 1 -- 172.21.15.98:0/3448861177 wait complete. 2024-05-09T09:01:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980bbd5640 1 Processor -- start 2024-05-09T09:01:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980bbd5640 1 -- start start 2024-05-09T09:01:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980bbd5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9804101840 0x7f9804198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980bbd5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9804102200 0x7f98041985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980bbd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9804107850 0x7f980419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980bbd5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9804111c60 con 0x7f9804107850 2024-05-09T09:01:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980bbd5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9804111ae0 con 0x7f9804101840 2024-05-09T09:01:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980bbd5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9804111de0 con 0x7f9804102200 2024-05-09T09:01:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980a172640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9804107850 0x7f980419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980a172640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9804107850 0x7f980419c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33296/0 (socket says 172.21.15.98:33296) 2024-05-09T09:01:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980a172640 1 -- 172.21.15.98:0/1210728231 learned_addr learned my addr 172.21.15.98:0/1210728231 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f9809971640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9804101840 0x7f9804198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f9809170640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9804102200 0x7f98041985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980a172640 1 -- 172.21.15.98:0/1210728231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9804102200 msgr2=0x7f98041985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980a172640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9804102200 0x7f98041985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980a172640 1 -- 172.21.15.98:0/1210728231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9804101840 msgr2=0x7f9804198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980a172640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9804101840 0x7f9804198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980a172640 1 -- 172.21.15.98:0/1210728231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f980419d050 con 0x7f9804107850 2024-05-09T09:01:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f9809170640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9804102200 0x7f98041985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.973+0000 7f980a172640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9804107850 0x7f980419c920 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00e960 tx=0x7f97fc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.974+0000 7f97faffd640 1 -- 172.21.15.98:0/1210728231 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97fc004d20 con 0x7f9804107850 2024-05-09T09:01:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.974+0000 7f97faffd640 1 -- 172.21.15.98:0/1210728231 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97fc005020 con 0x7f9804107850 2024-05-09T09:01:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.974+0000 7f980bbd5640 1 -- 172.21.15.98:0/1210728231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f980419d340 con 0x7f9804107850 2024-05-09T09:01:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.974+0000 7f97faffd640 1 -- 172.21.15.98:0/1210728231 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97fc0116d0 con 0x7f9804107850 2024-05-09T09:01:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.974+0000 7f980bbd5640 1 -- 172.21.15.98:0/1210728231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98041a4bd0 con 0x7f9804107850 2024-05-09T09:01:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.976+0000 7f97faffd640 1 -- 172.21.15.98:0/1210728231 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f97fc005560 con 0x7f9804107850 2024-05-09T09:01:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.976+0000 7f980e9be740 1 Processor -- start 2024-05-09T09:01:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f980e9be740 1 -- start start 2024-05-09T09:01:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f980e9be740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc48830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f980e9be740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc530f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f980e9be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc55a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f980e9be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55720cbad340 con 0x55720cc53630 2024-05-09T09:01:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f980e9be740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55720cb55290 con 0x55720cc48450 2024-05-09T09:01:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f980e9be740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55720cba9300 con 0x55720cc48d70 2024-05-09T09:01:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f9809170640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc55a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f97faffd640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f97f00a2e20 0x7f97f00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f9809170640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc55a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33304/0 (socket says 172.21.15.98:33304) 2024-05-09T09:01:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f9809170640 1 -- 172.21.15.98:0/3982667772 learned_addr learned my addr 172.21.15.98:0/3982667772 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f9809170640 1 --2- 172.21.15.98:0/3982667772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc48830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f9809971640 1 --2- 172.21.15.98:0/3982667772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f980a172640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f97f00a2e20 0x7f97f00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f9809170640 1 -- 172.21.15.98:0/3982667772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 msgr2=0x55720cc530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f9809170640 1 --2- 172.21.15.98:0/3982667772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc530f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f9809170640 1 -- 172.21.15.98:0/3982667772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 msgr2=0x55720cc48830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f9809170640 1 --2- 172.21.15.98:0/3982667772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc48830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.977+0000 7f97faffd640 1 -- 172.21.15.98:0/1210728231 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f97fc0cf960 con 0x7f9804107850 2024-05-09T09:01:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.978+0000 7f9809170640 1 -- 172.21.15.98:0/3982667772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55720cb798e0 con 0x55720cc53630 2024-05-09T09:01:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.978+0000 7f9809971640 1 --2- 172.21.15.98:0/3982667772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:01:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.978+0000 7f9809170640 1 --2- 172.21.15.98:0/3982667772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc55a20 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f97f40057c0 tx=0x7f97f4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fd53d2df20f230 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.978+0000 7f980a172640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f97f00a2e20 0x7f97f00a52e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0ccd50 tx=0x7f97fc06eae0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.978+0000 7f97effff640 1 -- 172.21.15.98:0/3982667772 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f4045070 con 0x55720cc53630 2024-05-09T09:01:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.978+0000 7f97effff640 1 -- 172.21.15.98:0/3982667772 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97f40035f0 con 0x55720cc53630 2024-05-09T09:01:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.978+0000 7f97effff640 1 -- 172.21.15.98:0/3982667772 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f4035040 con 0x55720cc53630 2024-05-09T09:01:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.979+0000 7f980e9be740 1 -- 172.21.15.98:0/3982667772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 msgr2=0x55720cc55a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.979+0000 7f980e9be740 1 --2- 172.21.15.98:0/3982667772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc55a20 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f97f40057c0 tx=0x7f97f4000bc0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.979+0000 7f980e9be740 1 -- 172.21.15.98:0/3982667772 shutdown_connections 2024-05-09T09:01:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.979+0000 7f980e9be740 1 --2- 172.21.15.98:0/3982667772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc55a20 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.979+0000 7f980e9be740 1 --2- 172.21.15.98:0/3982667772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.979+0000 7f980e9be740 1 --2- 172.21.15.98:0/3982667772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc48830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.979+0000 7f980e9be740 1 -- 172.21.15.98:0/3982667772 >> 172.21.15.98:0/3982667772 conn(0x55720cc44960 msgr2=0x55720cc44d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.979+0000 7f980e9be740 1 -- 172.21.15.98:0/3982667772 shutdown_connections 2024-05-09T09:01:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.979+0000 7f980e9be740 1 -- 172.21.15.98:0/3982667772 wait complete. 2024-05-09T09:01:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.979+0000 7f980e9be740 1 -- start start 2024-05-09T09:01:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.979+0000 7f980e9be740 1 Processor -- start 2024-05-09T09:01:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f980e9be740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc505b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f980e9be740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc50af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f980e9be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc51030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f980e9be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55720cc56200 con 0x55720cc53630 2024-05-09T09:01:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f980e9be740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55720cb68790 con 0x55720cc48450 2024-05-09T09:01:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f980e9be740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55720cc56380 con 0x55720cc48d70 2024-05-09T09:01:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f980e9be740 10 client.0 _ll_register_callbacks cb 0x55720cba3a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f9809170640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc51030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f9809170640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc51030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33314/0 (socket says 172.21.15.98:33314) 2024-05-09T09:01:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f9809170640 1 -- 172.21.15.98:0/55216510 learned_addr learned my addr 172.21.15.98:0/55216510 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f9809971640 1 --2- 172.21.15.98:0/55216510 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc50af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f9809170640 1 --2- 172.21.15.98:0/55216510 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f9809170640 1 -- 172.21.15.98:0/55216510 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 msgr2=0x55720cc50af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f9809170640 1 --2- 172.21.15.98:0/55216510 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc50af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f9809170640 1 -- 172.21.15.98:0/55216510 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 msgr2=0x55720cc505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f9809170640 1 --2- 172.21.15.98:0/55216510 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f9809170640 1 -- 172.21.15.98:0/55216510 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55720c8c1210 con 0x55720cc53630 2024-05-09T09:01:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f9809170640 1 --2- 172.21.15.98:0/55216510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc51030 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f97f4005c40 tx=0x7f97f402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe71ab31b7b18f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f97ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.980+0000 7f97ed7fa640 1 -- 172.21.15.98:0/55216510 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f4035040 con 0x55720cc53630 2024-05-09T09:01:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.981+0000 7f97ed7fa640 1 -- 172.21.15.98:0/55216510 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f97f40039a0 con 0x55720cc53630 2024-05-09T09:01:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.981+0000 7f980e9be740 1 -- 172.21.15.98:0/55216510 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97f40035f0 con 0x55720cc53630 2024-05-09T09:01:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.981+0000 7f97ed7fa640 1 -- 172.21.15.98:0/55216510 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f97f4045040 con 0x55720cc53630 2024-05-09T09:01:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.981+0000 7f980e9be740 10 client.18903 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.981+0000 7f980e9be740 1 -- 172.21.15.98:0/55216510 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55720c98d620 con 0x55720cc53630 2024-05-09T09:01:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.981+0000 7f97ed7fa640 1 -- 172.21.15.98:0/55216510 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97f40035f0 con 0x55720cc53630 2024-05-09T09:01:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.981+0000 7f97ed7fa640 1 -- 172.21.15.98:0/55216510 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f97f403a040 con 0x55720cc53630 2024-05-09T09:01:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.981+0000 7f980e9be740 10 client.18903 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.981+0000 7f980e9be740 10 client.18903 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.981+0000 7f980e9be740 20 client.18903 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.981+0000 7f980e9be740 1 --2- 172.21.15.98:0/55216510 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55720cc631d0 0x55720cc655c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.981+0000 7f980e9be740 4 client.18903 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:01:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.981+0000 7f980e9be740 1 -- 172.21.15.98:0/55216510 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55720cbbaa70 con 0x55720cc631d0 2024-05-09T09:01:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.981+0000 7f9809971640 1 --2- 172.21.15.98:0/55216510 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55720cc631d0 0x55720cc655c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.982+0000 7f97ed7fa640 10 client.18903 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.982+0000 7f9809971640 1 --2- 172.21.15.98:0/55216510 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55720cc631d0 0x55720cc655c0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be92f9bc4f7ee server_cookie=535b940a753b86a3 in_seq=0 out_seq=0 2024-05-09T09:01:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.982+0000 7f97ed7fa640 1 -- 172.21.15.98:0/55216510 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55720cbbaa70 con 0x55720cc631d0 2024-05-09T09:01:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.982+0000 7f97ed7fa640 10 client.18903 handle_command_reply: tid=0 2024-05-09T09:01:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.982+0000 7f980e9be740 1 client.18903 shutdown 2024-05-09T09:01:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.983+0000 7f980e9be740 1 -- 172.21.15.98:0/55216510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 msgr2=0x55720cc51030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.983+0000 7f980e9be740 1 --2- 172.21.15.98:0/55216510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc51030 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f97f4005c40 tx=0x7f97f402a420 comp rx=0 tx=0).stop 2024-05-09T09:01:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.983+0000 7f980e9be740 1 -- 172.21.15.98:0/55216510 shutdown_connections 2024-05-09T09:01:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.983+0000 7f980e9be740 1 --2- 172.21.15.98:0/55216510 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55720cc631d0 0x55720cc655c0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.983+0000 7f980e9be740 1 --2- 172.21.15.98:0/55216510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc51030 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.983+0000 7f980e9be740 1 --2- 172.21.15.98:0/55216510 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc50af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.983+0000 7f980e9be740 1 --2- 172.21.15.98:0/55216510 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.983+0000 7f97ed7fa640 0 client.18903 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.983+0000 7f980e9be740 1 -- 172.21.15.98:0/55216510 >> 172.21.15.98:0/55216510 conn(0x55720cc44960 msgr2=0x55720cc4a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.983+0000 7f980e9be740 1 -- 172.21.15.98:0/55216510 shutdown_connections 2024-05-09T09:01:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.983+0000 7f980e9be740 1 -- 172.21.15.98:0/55216510 wait complete. 2024-05-09T09:01:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.983+0000 7f980e9be740 20 client.18903 trim_cache size 0 max 16384 2024-05-09T09:01:53.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f980e9be740 1 Processor -- start 2024-05-09T09:01:53.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f980e9be740 1 -- start start 2024-05-09T09:01:53.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f980e9be740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc52ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:53.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f980e9be740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc4ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:53.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f980e9be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc4d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:53.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f980e9be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f98041aee80 con 0x55720cc53630 2024-05-09T09:01:53.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f980e9be740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f98041b09c0 con 0x55720cc48450 2024-05-09T09:01:53.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f980e9be740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f98041af550 con 0x55720cc48d70 2024-05-09T09:01:53.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f9809170640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc4d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:53.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f9809170640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc4d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33322/0 (socket says 172.21.15.98:33322) 2024-05-09T09:01:53.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f9809170640 1 -- 172.21.15.98:0/2733708155 learned_addr learned my addr 172.21.15.98:0/2733708155 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:53.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f9809170640 1 -- 172.21.15.98:0/2733708155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 msgr2=0x55720cc4ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:53.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f9809971640 1 --2- 172.21.15.98:0/2733708155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc4ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:53.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f9809170640 1 --2- 172.21.15.98:0/2733708155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc4ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f9809170640 1 -- 172.21.15.98:0/2733708155 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 msgr2=0x55720cc52ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:53.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f9809170640 1 --2- 172.21.15.98:0/2733708155 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc52ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f9809170640 1 -- 172.21.15.98:0/2733708155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55720cc44d40 con 0x55720cc53630 2024-05-09T09:01:53.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f9809971640 1 --2- 172.21.15.98:0/2733708155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc4ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:01:53.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f9809170640 1 --2- 172.21.15.98:0/2733708155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc4d830 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f97f403ec50 tx=0x7f97f403ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2863d70ec7b5ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.998+0000 7f97ef7fe640 1 -- 172.21.15.98:0/2733708155 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f405d070 con 0x55720cc53630 2024-05-09T09:01:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.999+0000 7f97ef7fe640 1 -- 172.21.15.98:0/2733708155 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97f403ee50 con 0x55720cc53630 2024-05-09T09:01:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.999+0000 7f97ef7fe640 1 -- 172.21.15.98:0/2733708155 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f4035070 con 0x55720cc53630 2024-05-09T09:01:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.999+0000 7f980e9be740 1 -- 172.21.15.98:0/2733708155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 msgr2=0x55720cc4d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.999+0000 7f980e9be740 1 --2- 172.21.15.98:0/2733708155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc4d830 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f97f403ec50 tx=0x7f97f403ed50 comp rx=0 tx=0).stop 2024-05-09T09:01:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.999+0000 7f980e9be740 1 -- 172.21.15.98:0/2733708155 shutdown_connections 2024-05-09T09:01:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.999+0000 7f980e9be740 1 --2- 172.21.15.98:0/2733708155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc4d830 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.999+0000 7f980e9be740 1 --2- 172.21.15.98:0/2733708155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc4ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.999+0000 7f980e9be740 1 --2- 172.21.15.98:0/2733708155 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc52ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.999+0000 7f980e9be740 1 -- 172.21.15.98:0/2733708155 >> 172.21.15.98:0/2733708155 conn(0x55720cc44960 msgr2=0x55720cc43a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.999+0000 7f980e9be740 1 -- 172.21.15.98:0/2733708155 shutdown_connections 2024-05-09T09:01:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.999+0000 7f980e9be740 1 -- 172.21.15.98:0/2733708155 wait complete. 2024-05-09T09:01:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:52.999+0000 7f980e9be740 1 -- start start 2024-05-09T09:01:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f980e9be740 1 Processor -- start 2024-05-09T09:01:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f980e9be740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc68b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f980e9be740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc65ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f980e9be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc66630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f980e9be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f98041b02f0 con 0x55720cc53630 2024-05-09T09:01:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f980e9be740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f98041afc20 con 0x55720cc48450 2024-05-09T09:01:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f980e9be740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f98041ace80 con 0x55720cc48d70 2024-05-09T09:01:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f980e9be740 10 client.0 _ll_register_callbacks cb 0x7f97f4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f9809170640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc66630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f9809170640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc66630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33334/0 (socket says 172.21.15.98:33334) 2024-05-09T09:01:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f9809170640 1 -- 172.21.15.98:0/1958258142 learned_addr learned my addr 172.21.15.98:0/1958258142 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f9809971640 1 --2- 172.21.15.98:0/1958258142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc65ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f9809170640 1 --2- 172.21.15.98:0/1958258142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc68b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f9809170640 1 -- 172.21.15.98:0/1958258142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 msgr2=0x55720cc65ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f9809170640 1 --2- 172.21.15.98:0/1958258142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc65ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f9809170640 1 -- 172.21.15.98:0/1958258142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 msgr2=0x55720cc68b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f9809170640 1 --2- 172.21.15.98:0/1958258142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc68b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.000+0000 7f9809170640 1 -- 172.21.15.98:0/1958258142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55720ccc83b0 con 0x55720cc53630 2024-05-09T09:01:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f9809170640 1 --2- 172.21.15.98:0/1958258142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc66630 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f97f4004650 tx=0x7f97f403e6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f91ffb63635173 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f97ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f97ed7fa640 1 -- 172.21.15.98:0/1958258142 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f405d070 con 0x55720cc53630 2024-05-09T09:01:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f97ed7fa640 1 -- 172.21.15.98:0/1958258142 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f97f4004290 con 0x55720cc53630 2024-05-09T09:01:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f980e9be740 1 -- 172.21.15.98:0/1958258142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97f403ee50 con 0x55720cc53630 2024-05-09T09:01:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f97ed7fa640 1 -- 172.21.15.98:0/1958258142 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f97f406e050 con 0x55720cc53630 2024-05-09T09:01:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f980e9be740 10 client.18909 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f980e9be740 1 -- 172.21.15.98:0/1958258142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55720ccc9490 con 0x55720cc53630 2024-05-09T09:01:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f97ed7fa640 1 -- 172.21.15.98:0/1958258142 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97f403ee50 con 0x55720cc53630 2024-05-09T09:01:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f97ed7fa640 1 -- 172.21.15.98:0/1958258142 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f97f403a070 con 0x55720cc53630 2024-05-09T09:01:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f980e9be740 10 client.18909 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f980e9be740 10 client.18909 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f980e9be740 20 client.18909 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f980e9be740 1 --2- 172.21.15.98:0/1958258142 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55720ccc9b10 0x55720cccbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f980e9be740 4 client.18909 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:01:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.001+0000 7f980e9be740 1 -- 172.21.15.98:0/1958258142 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55720cbb4dc0 con 0x55720ccc9b10 2024-05-09T09:01:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.002+0000 7f9809971640 1 --2- 172.21.15.98:0/1958258142 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55720ccc9b10 0x55720cccbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.002+0000 7f9809971640 1 --2- 172.21.15.98:0/1958258142 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55720ccc9b10 0x55720cccbf00 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e503a91e929d3b5d server_cookie=7404b11ceffec8a2 in_seq=0 out_seq=0 2024-05-09T09:01:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.002+0000 7f97ed7fa640 10 client.18909 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:53.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f97ed7fa640 1 -- 172.21.15.98:0/1958258142 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55720cbb4dc0 con 0x55720ccc9b10 2024-05-09T09:01:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f97ed7fa640 10 client.18909 handle_command_reply: tid=0 2024-05-09T09:01:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f980e9be740 1 client.18909 shutdown 2024-05-09T09:01:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f980e9be740 1 -- 172.21.15.98:0/1958258142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 msgr2=0x55720cc66630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f980e9be740 1 --2- 172.21.15.98:0/1958258142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc66630 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f97f4004650 tx=0x7f97f403e6b0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f980e9be740 1 -- 172.21.15.98:0/1958258142 shutdown_connections 2024-05-09T09:01:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f980e9be740 1 --2- 172.21.15.98:0/1958258142 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55720ccc9b10 0x55720cccbf00 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f980e9be740 1 --2- 172.21.15.98:0/1958258142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55720cc53630 0x55720cc66630 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f980e9be740 1 --2- 172.21.15.98:0/1958258142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55720cc48d70 0x55720cc65ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f97ed7fa640 0 client.18909 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f980e9be740 1 --2- 172.21.15.98:0/1958258142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55720cc48450 0x55720cc68b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f980e9be740 1 -- 172.21.15.98:0/1958258142 >> 172.21.15.98:0/1958258142 conn(0x55720cc44960 msgr2=0x55720cc4ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f980e9be740 1 -- 172.21.15.98:0/1958258142 shutdown_connections 2024-05-09T09:01:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f980e9be740 1 -- 172.21.15.98:0/1958258142 wait complete. 2024-05-09T09:01:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.006+0000 7f980e9be740 20 client.18909 trim_cache size 0 max 16384 2024-05-09T09:01:53.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T09:01:53.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T09:01:53.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T09:01:53.010 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T09:01:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.007+0000 7f980bbd5640 1 -- 172.21.15.98:0/1210728231 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f97f00a2e20 msgr2=0x7f97f00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.007+0000 7f980bbd5640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f97f00a2e20 0x7f97f00a52e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0ccd50 tx=0x7f97fc06eae0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.007+0000 7f980bbd5640 1 -- 172.21.15.98:0/1210728231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9804107850 msgr2=0x7f980419c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.007+0000 7f980bbd5640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9804107850 0x7f980419c920 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00e960 tx=0x7f97fc00ee20 comp rx=0 tx=0).stop 2024-05-09T09:01:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.007+0000 7f980bbd5640 1 -- 172.21.15.98:0/1210728231 shutdown_connections 2024-05-09T09:01:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.007+0000 7f980bbd5640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f97f00a2e20 0x7f97f00a52e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.007+0000 7f980bbd5640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9804107850 0x7f980419c920 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.007+0000 7f980bbd5640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9804102200 0x7f98041985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.007+0000 7f980bbd5640 1 --2- 172.21.15.98:0/1210728231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9804101840 0x7f9804198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.007+0000 7f980bbd5640 1 -- 172.21.15.98:0/1210728231 >> 172.21.15.98:0/1210728231 conn(0x7f9804078cc0 msgr2=0x7f98040ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.007+0000 7f980bbd5640 1 -- 172.21.15.98:0/1210728231 shutdown_connections 2024-05-09T09:01:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:01:53.007+0000 7f980bbd5640 1 -- 172.21.15.98:0/1210728231 wait complete. 2024-05-09T09:01:54.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:54 smithi105 ceph-mon[34498]: pgmap v352: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:01:54.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:54 smithi098 ceph-mon[30092]: pgmap v352: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:01:54.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:54 smithi142 ceph-mon[34625]: pgmap v352: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:01:56.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:56 smithi105 ceph-mon[34498]: pgmap v353: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 9 op/s 2024-05-09T09:01:56.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:56 smithi098 ceph-mon[30092]: pgmap v353: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 9 op/s 2024-05-09T09:01:56.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:56 smithi142 ceph-mon[34625]: pgmap v353: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 9 op/s 2024-05-09T09:01:58.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:01:58 smithi105 ceph-mon[34498]: pgmap v354: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-05-09T09:01:58.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:01:58 smithi098 ceph-mon[30092]: pgmap v354: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-05-09T09:01:58.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:01:58 smithi142 ceph-mon[34625]: pgmap v354: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-05-09T09:01:59.386 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:01:59.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:01:59.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.504+0000 7fb609b90640 1 -- 172.21.15.98:0/3857935112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6041021c0 msgr2=0x7fb60410e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.504+0000 7fb609b90640 1 --2- 172.21.15.98:0/3857935112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6041021c0 0x7fb60410e6d0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc009cd0 tx=0x7fb5fc02a8e0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.504+0000 7fb609b90640 1 -- 172.21.15.98:0/3857935112 shutdown_connections 2024-05-09T09:01:59.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.504+0000 7fb609b90640 1 --2- 172.21.15.98:0/3857935112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6041021c0 0x7fb60410e6d0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.504+0000 7fb609b90640 1 --2- 172.21.15.98:0/3857935112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb604101800 0x7fb604101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.504+0000 7fb609b90640 1 --2- 172.21.15.98:0/3857935112 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb604107810 0x7fb604107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.504+0000 7fb609b90640 1 -- 172.21.15.98:0/3857935112 >> 172.21.15.98:0/3857935112 conn(0x7fb6040fd270 msgr2=0x7fb6040ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.504+0000 7fb609b90640 1 -- 172.21.15.98:0/3857935112 shutdown_connections 2024-05-09T09:01:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.504+0000 7fb609b90640 1 -- 172.21.15.98:0/3857935112 wait complete. 2024-05-09T09:01:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.505+0000 7fb609b90640 1 Processor -- start 2024-05-09T09:01:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.505+0000 7fb609b90640 1 -- start start 2024-05-09T09:01:59.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.505+0000 7fb609b90640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb604101800 0x7fb6041981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.505+0000 7fb609b90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6041021c0 0x7fb604198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.505+0000 7fb609b90640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb604107810 0x7fb60419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.505+0000 7fb609b90640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb604111c20 con 0x7fb6041021c0 2024-05-09T09:01:59.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.505+0000 7fb609b90640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb604111aa0 con 0x7fb604107810 2024-05-09T09:01:59.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.505+0000 7fb609b90640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb604111da0 con 0x7fb604101800 2024-05-09T09:01:59.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb602ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6041021c0 0x7fb604198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb602ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6041021c0 0x7fb604198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37652/0 (socket says 172.21.15.98:37652) 2024-05-09T09:01:59.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb602ffd640 1 -- 172.21.15.98:0/1598026536 learned_addr learned my addr 172.21.15.98:0/1598026536 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:59.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb603fff640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb604107810 0x7fb60419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb6037fe640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb604101800 0x7fb6041981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb602ffd640 1 -- 172.21.15.98:0/1598026536 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb604101800 msgr2=0x7fb6041981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb602ffd640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb604101800 0x7fb6041981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb602ffd640 1 -- 172.21.15.98:0/1598026536 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb604107810 msgr2=0x7fb60419ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb602ffd640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb604107810 0x7fb60419ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb602ffd640 1 -- 172.21.15.98:0/1598026536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60419d1b0 con 0x7fb6041021c0 2024-05-09T09:01:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb6037fe640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb604101800 0x7fb6041981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb602ffd640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6041021c0 0x7fb604198700 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fb5f800d950 tx=0x7fb5f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb600ff9640 1 -- 172.21.15.98:0/1598026536 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f8017030 con 0x7fb6041021c0 2024-05-09T09:01:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb600ff9640 1 -- 172.21.15.98:0/1598026536 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5f8004020 con 0x7fb6041021c0 2024-05-09T09:01:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb609b90640 1 -- 172.21.15.98:0/1598026536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb60419d4a0 con 0x7fb6041021c0 2024-05-09T09:01:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb600ff9640 1 -- 172.21.15.98:0/1598026536 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f80047a0 con 0x7fb6041021c0 2024-05-09T09:01:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.506+0000 7fb609b90640 1 -- 172.21.15.98:0/1598026536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6041a4d30 con 0x7fb6041021c0 2024-05-09T09:01:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.509+0000 7fb600ff9640 1 -- 172.21.15.98:0/1598026536 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb5f80041d0 con 0x7fb6041021c0 2024-05-09T09:01:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.509+0000 7fb60c979740 1 Processor -- start 2024-05-09T09:01:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.509+0000 7fb60c979740 1 -- start start 2024-05-09T09:01:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb60c979740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a989280 0x55cd4a989660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb60c979740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989ba0 0x55cd4a993d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb60c979740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a994270 0x55cd4a996660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb60c979740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cd4a8eed00 con 0x55cd4a989ba0 2024-05-09T09:01:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb60c979740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cd4a8964c0 con 0x55cd4a994270 2024-05-09T09:01:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb60c979740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cd4a8eacc0 con 0x55cd4a989280 2024-05-09T09:01:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb6037fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989ba0 0x55cd4a993d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb6037fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989ba0 0x55cd4a993d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37662/0 (socket says 172.21.15.98:37662) 2024-05-09T09:01:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb6037fe640 1 -- 172.21.15.98:0/458284800 learned_addr learned my addr 172.21.15.98:0/458284800 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb603fff640 1 --2- 172.21.15.98:0/458284800 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a989280 0x55cd4a989660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb600ff9640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb5ec0a2d80 0x7fb5ec0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb602ffd640 1 --2- 172.21.15.98:0/458284800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a994270 0x55cd4a996660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb603fff640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb5ec0a2d80 0x7fb5ec0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb6037fe640 1 -- 172.21.15.98:0/458284800 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a989280 msgr2=0x55cd4a989660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb6037fe640 1 --2- 172.21.15.98:0/458284800 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a989280 0x55cd4a989660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb6037fe640 1 -- 172.21.15.98:0/458284800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a994270 msgr2=0x55cd4a996660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb6037fe640 1 --2- 172.21.15.98:0/458284800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a994270 0x55cd4a996660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb6037fe640 1 -- 172.21.15.98:0/458284800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd4a6cc7d0 con 0x55cd4a989ba0 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb600ff9640 1 -- 172.21.15.98:0/1598026536 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fb5f80cf1c0 con 0x7fb6041021c0 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb6037fe640 1 --2- 172.21.15.98:0/458284800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989ba0 0x55cd4a993d30 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4007850 tx=0x7fb5f4007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4193369e4981ed88 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.510+0000 7fb5e9ffb640 1 -- 172.21.15.98:0/458284800 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f400d950 con 0x55cd4a989ba0 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.511+0000 7fb5e9ffb640 1 -- 172.21.15.98:0/458284800 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5f4018070 con 0x55cd4a989ba0 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.511+0000 7fb5e9ffb640 1 -- 172.21.15.98:0/458284800 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f40123e0 con 0x55cd4a989ba0 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.511+0000 7fb60c979740 1 -- 172.21.15.98:0/458284800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989ba0 msgr2=0x55cd4a993d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.511+0000 7fb60c979740 1 --2- 172.21.15.98:0/458284800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989ba0 0x55cd4a993d30 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4007850 tx=0x7fb5f4007d10 comp rx=0 tx=0).stop 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.511+0000 7fb603fff640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb5ec0a2d80 0x7fb5ec0a5240 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc004ab0 tx=0x7fb5fc02a8e0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.511+0000 7fb60c979740 1 -- 172.21.15.98:0/458284800 shutdown_connections 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.511+0000 7fb60c979740 1 --2- 172.21.15.98:0/458284800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a994270 0x55cd4a996660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.511+0000 7fb60c979740 1 --2- 172.21.15.98:0/458284800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989ba0 0x55cd4a993d30 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.511+0000 7fb60c979740 1 --2- 172.21.15.98:0/458284800 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a989280 0x55cd4a989660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.511+0000 7fb60c979740 1 -- 172.21.15.98:0/458284800 >> 172.21.15.98:0/458284800 conn(0x55cd4a9855a0 msgr2=0x55cd4a985980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.511+0000 7fb60c979740 1 -- 172.21.15.98:0/458284800 shutdown_connections 2024-05-09T09:01:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.511+0000 7fb60c979740 1 -- 172.21.15.98:0/458284800 wait complete. 2024-05-09T09:01:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.512+0000 7fb60c979740 1 -- start start 2024-05-09T09:01:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.512+0000 7fb60c979740 1 Processor -- start 2024-05-09T09:01:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.512+0000 7fb60c979740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a989280 0x55cd4a991190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.512+0000 7fb60c979740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a989ba0 0x55cd4a9916d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.512+0000 7fb60c979740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a994270 0x55cd4a991c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.512+0000 7fb60c979740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cd4a997060 con 0x55cd4a994270 2024-05-09T09:01:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.512+0000 7fb60c979740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cd4a8a9b90 con 0x55cd4a989280 2024-05-09T09:01:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.512+0000 7fb60c979740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cd4a9971e0 con 0x55cd4a989ba0 2024-05-09T09:01:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.512+0000 7fb60c979740 10 client.0 _ll_register_callbacks cb 0x55cd4a8e5da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.512+0000 7fb603fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a994270 0x55cd4a991c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.512+0000 7fb603fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a994270 0x55cd4a991c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37676/0 (socket says 172.21.15.98:37676) 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.512+0000 7fb603fff640 1 -- 172.21.15.98:0/3065435061 learned_addr learned my addr 172.21.15.98:0/3065435061 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.512+0000 7fb602ffd640 1 --2- 172.21.15.98:0/3065435061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a989ba0 0x55cd4a9916d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.512+0000 7fb6037fe640 1 --2- 172.21.15.98:0/3065435061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a989280 0x55cd4a991190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb603fff640 1 -- 172.21.15.98:0/3065435061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a989ba0 msgr2=0x55cd4a9916d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb603fff640 1 --2- 172.21.15.98:0/3065435061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a989ba0 0x55cd4a9916d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb603fff640 1 -- 172.21.15.98:0/3065435061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a989280 msgr2=0x55cd4a991190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb603fff640 1 --2- 172.21.15.98:0/3065435061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a989280 0x55cd4a991190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb603fff640 1 -- 172.21.15.98:0/3065435061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd4a6023f0 con 0x55cd4a994270 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb603fff640 1 --2- 172.21.15.98:0/3065435061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a994270 0x55cd4a991c10 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc031dc0 tx=0x7fb5fc0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d42a21c7e37a85ee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb5e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb5e2ffd640 1 -- 172.21.15.98:0/3065435061 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc04e070 con 0x55cd4a994270 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb60c979740 1 -- 172.21.15.98:0/3065435061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5f4018070 con 0x55cd4a994270 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb5e2ffd640 1 -- 172.21.15.98:0/3065435061 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fb5fc004750 con 0x55cd4a994270 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb5e2ffd640 1 -- 172.21.15.98:0/3065435061 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb5fc004b40 con 0x55cd4a994270 2024-05-09T09:01:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb60c979740 10 client.18921 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb60c979740 1 -- 172.21.15.98:0/3065435061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cd4a5c20f0 con 0x55cd4a994270 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb5e2ffd640 1 -- 172.21.15.98:0/3065435061 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5f4018070 con 0x55cd4a994270 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb5e2ffd640 1 -- 172.21.15.98:0/3065435061 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb5fc030070 con 0x55cd4a994270 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb60c979740 10 client.18921 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.513+0000 7fb60c979740 10 client.18921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.514+0000 7fb60c979740 20 client.18921 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.514+0000 7fb60c979740 1 --2- 172.21.15.98:0/3065435061 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cd4a9a4560 0x55cd4a9a6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.514+0000 7fb60c979740 4 client.18921 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.514+0000 7fb60c979740 1 -- 172.21.15.98:0/3065435061 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd4a8fc430 con 0x55cd4a9a4560 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.514+0000 7fb6037fe640 1 --2- 172.21.15.98:0/3065435061 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cd4a9a4560 0x55cd4a9a6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.514+0000 7fb6037fe640 1 --2- 172.21.15.98:0/3065435061 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cd4a9a4560 0x55cd4a9a6a00 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c694f9bda2e985c server_cookie=8429cde798754e87 in_seq=0 out_seq=0 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.514+0000 7fb5e2ffd640 10 client.18921 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb5e2ffd640 1 -- 172.21.15.98:0/3065435061 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cd4a8fc430 con 0x55cd4a9a4560 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb5e2ffd640 10 client.18921 handle_command_reply: tid=0 2024-05-09T09:01:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb60c979740 1 client.18921 shutdown 2024-05-09T09:01:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb60c979740 1 -- 172.21.15.98:0/3065435061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a994270 msgr2=0x55cd4a991c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb60c979740 1 --2- 172.21.15.98:0/3065435061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a994270 0x55cd4a991c10 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc031dc0 tx=0x7fb5fc0095c0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb60c979740 1 -- 172.21.15.98:0/3065435061 shutdown_connections 2024-05-09T09:01:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb60c979740 1 --2- 172.21.15.98:0/3065435061 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cd4a9a4560 0x55cd4a9a6a00 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb60c979740 1 --2- 172.21.15.98:0/3065435061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a994270 0x55cd4a991c10 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb60c979740 1 --2- 172.21.15.98:0/3065435061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a989ba0 0x55cd4a9916d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb5e2ffd640 0 client.18921 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb60c979740 1 --2- 172.21.15.98:0/3065435061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a989280 0x55cd4a991190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb60c979740 1 -- 172.21.15.98:0/3065435061 >> 172.21.15.98:0/3065435061 conn(0x55cd4a9855a0 msgr2=0x55cd4a9847e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb60c979740 1 -- 172.21.15.98:0/3065435061 shutdown_connections 2024-05-09T09:01:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb60c979740 1 -- 172.21.15.98:0/3065435061 wait complete. 2024-05-09T09:01:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.515+0000 7fb60c979740 20 client.18921 trim_cache size 0 max 16384 2024-05-09T09:01:59.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.532+0000 7fb60c979740 1 Processor -- start 2024-05-09T09:01:59.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.532+0000 7fb60c979740 1 -- start start 2024-05-09T09:01:59.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb60c979740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a989280 0x55cd4a99ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb60c979740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989ba0 0x55cd4a98b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb60c979740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a994270 0x55cd4a9ac320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb60c979740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6041aef00 con 0x55cd4a989ba0 2024-05-09T09:01:59.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb60c979740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb6041b0a40 con 0x55cd4a994270 2024-05-09T09:01:59.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb60c979740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb6041af5d0 con 0x55cd4a989280 2024-05-09T09:01:59.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb602ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989ba0 0x55cd4a98b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb602ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989ba0 0x55cd4a98b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37690/0 (socket says 172.21.15.98:37690) 2024-05-09T09:01:59.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb602ffd640 1 -- 172.21.15.98:0/1290157716 learned_addr learned my addr 172.21.15.98:0/1290157716 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:59.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb602ffd640 1 -- 172.21.15.98:0/1290157716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a989280 msgr2=0x55cd4a99ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:59.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb602ffd640 1 --2- 172.21.15.98:0/1290157716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a989280 0x55cd4a99ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb602ffd640 1 -- 172.21.15.98:0/1290157716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a994270 msgr2=0x55cd4a9ac320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:59.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb602ffd640 1 --2- 172.21.15.98:0/1290157716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a994270 0x55cd4a9ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb602ffd640 1 -- 172.21.15.98:0/1290157716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd4a9865a0 con 0x55cd4a989ba0 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb602ffd640 1 --2- 172.21.15.98:0/1290157716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989ba0 0x55cd4a98b9e0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fb5f809b260 tx=0x7fb5f80af040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ed4c03b280134d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb5e97fa640 1 -- 172.21.15.98:0/1290157716 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f809b9d0 con 0x55cd4a989ba0 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb5e97fa640 1 -- 172.21.15.98:0/1290157716 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5f809b2d0 con 0x55cd4a989ba0 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.533+0000 7fb5e97fa640 1 -- 172.21.15.98:0/1290157716 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f80bb0b0 con 0x55cd4a989ba0 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.534+0000 7fb60c979740 1 -- 172.21.15.98:0/1290157716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989ba0 msgr2=0x55cd4a98b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.534+0000 7fb60c979740 1 --2- 172.21.15.98:0/1290157716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989ba0 0x55cd4a98b9e0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fb5f809b260 tx=0x7fb5f80af040 comp rx=0 tx=0).stop 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.534+0000 7fb60c979740 1 -- 172.21.15.98:0/1290157716 shutdown_connections 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.534+0000 7fb60c979740 1 --2- 172.21.15.98:0/1290157716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a994270 0x55cd4a9ac320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.534+0000 7fb60c979740 1 --2- 172.21.15.98:0/1290157716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989ba0 0x55cd4a98b9e0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.534+0000 7fb60c979740 1 --2- 172.21.15.98:0/1290157716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a989280 0x55cd4a99ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.534+0000 7fb60c979740 1 -- 172.21.15.98:0/1290157716 >> 172.21.15.98:0/1290157716 conn(0x55cd4a9855a0 msgr2=0x55cd4a9929c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.534+0000 7fb60c979740 1 -- 172.21.15.98:0/1290157716 shutdown_connections 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.534+0000 7fb60c979740 1 -- 172.21.15.98:0/1290157716 wait complete. 2024-05-09T09:01:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.534+0000 7fb60c979740 1 -- start start 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.534+0000 7fb60c979740 1 Processor -- start 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb60c979740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989280 0x55cd4a99c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb60c979740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a989ba0 0x55cd4a999430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb60c979740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a994270 0x55cd4a999970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb60c979740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6041b0370 con 0x55cd4a989280 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb60c979740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb6041afca0 con 0x55cd4a989ba0 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb60c979740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb6041acf50 con 0x55cd4a994270 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb60c979740 10 client.0 _ll_register_callbacks cb 0x7fb5fc031500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb6037fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989280 0x55cd4a99c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb6037fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989280 0x55cd4a99c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37706/0 (socket says 172.21.15.98:37706) 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb6037fe640 1 -- 172.21.15.98:0/3036052708 learned_addr learned my addr 172.21.15.98:0/3036052708 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb603fff640 1 --2- 172.21.15.98:0/3036052708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a994270 0x55cd4a999970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb6037fe640 1 -- 172.21.15.98:0/3036052708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a994270 msgr2=0x55cd4a999970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb6037fe640 1 --2- 172.21.15.98:0/3036052708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a994270 0x55cd4a999970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb6037fe640 1 -- 172.21.15.98:0/3036052708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a989ba0 msgr2=0x55cd4a999430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb6037fe640 1 --2- 172.21.15.98:0/3036052708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a989ba0 0x55cd4a999430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb6037fe640 1 -- 172.21.15.98:0/3036052708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd4a99a1a0 con 0x55cd4a989280 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb6037fe640 1 --2- 172.21.15.98:0/3036052708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989280 0x55cd4a99c810 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4007540 tx=0x7fb5f4007fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36fa2a4267f0fa6a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb5e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb5e2ffd640 1 -- 172.21.15.98:0/3036052708 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4027bd0 con 0x55cd4a989280 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb5e2ffd640 1 -- 172.21.15.98:0/3036052708 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fb5f4027d70 con 0x55cd4a989280 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.535+0000 7fb60c979740 1 -- 172.21.15.98:0/3036052708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5f809b2d0 con 0x55cd4a989280 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.536+0000 7fb5e2ffd640 1 -- 172.21.15.98:0/3036052708 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb5f4036050 con 0x55cd4a989280 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.536+0000 7fb60c979740 10 client.18927 fetch_fsmap learned FSMap version 37 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.536+0000 7fb60c979740 1 -- 172.21.15.98:0/3036052708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cd4aa0bbf0 con 0x55cd4a989280 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.536+0000 7fb5e2ffd640 1 -- 172.21.15.98:0/3036052708 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5f809b2d0 con 0x55cd4a989280 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.536+0000 7fb5e2ffd640 1 -- 172.21.15.98:0/3036052708 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb5f4027370 con 0x55cd4a989280 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.536+0000 7fb60c979740 10 client.18927 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.536+0000 7fb60c979740 10 client.18927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:01:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.536+0000 7fb60c979740 20 client.18927 populate_metadata read hostname 'smithi098' 2024-05-09T09:01:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.536+0000 7fb60c979740 1 --2- 172.21.15.98:0/3036052708 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cd4aa0c090 0x55cd4aa0e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.536+0000 7fb60c979740 4 client.18927 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:01:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.536+0000 7fb60c979740 1 -- 172.21.15.98:0/3036052708 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cd4a8f6780 con 0x55cd4aa0c090 2024-05-09T09:01:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.536+0000 7fb602ffd640 1 --2- 172.21.15.98:0/3036052708 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cd4aa0c090 0x55cd4aa0e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb5e2ffd640 10 client.18927 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb602ffd640 1 --2- 172.21.15.98:0/3036052708 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cd4aa0c090 0x55cd4aa0e4b0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2a6204a668edd99 server_cookie=14a370e01a019792 in_seq=0 out_seq=0 2024-05-09T09:01:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb5e2ffd640 1 -- 172.21.15.98:0/3036052708 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cd4a8f6780 con 0x55cd4aa0c090 2024-05-09T09:01:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb5e2ffd640 10 client.18927 handle_command_reply: tid=0 2024-05-09T09:01:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb60c979740 1 client.18927 shutdown 2024-05-09T09:01:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb60c979740 1 -- 172.21.15.98:0/3036052708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989280 msgr2=0x55cd4a99c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb60c979740 1 --2- 172.21.15.98:0/3036052708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989280 0x55cd4a99c810 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4007540 tx=0x7fb5f4007fa0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb60c979740 1 -- 172.21.15.98:0/3036052708 shutdown_connections 2024-05-09T09:01:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb60c979740 1 --2- 172.21.15.98:0/3036052708 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55cd4aa0c090 0x55cd4aa0e4b0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb60c979740 1 --2- 172.21.15.98:0/3036052708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cd4a994270 0x55cd4a999970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb60c979740 1 --2- 172.21.15.98:0/3036052708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd4a989ba0 0x55cd4a999430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb5e2ffd640 0 client.18927 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:01:59.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb60c979740 1 --2- 172.21.15.98:0/3036052708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd4a989280 0x55cd4a99c810 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb60c979740 1 -- 172.21.15.98:0/3036052708 >> 172.21.15.98:0/3036052708 conn(0x55cd4a9855a0 msgr2=0x55cd4a993250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:59.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb60c979740 1 -- 172.21.15.98:0/3036052708 shutdown_connections 2024-05-09T09:01:59.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb60c979740 1 -- 172.21.15.98:0/3036052708 wait complete. 2024-05-09T09:01:59.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.537+0000 7fb60c979740 20 client.18927 trim_cache size 0 max 16384 2024-05-09T09:01:59.543 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:01:59.543 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T09:01:59.543 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T09:01:59.544 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:01:59.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.538+0000 7fb609b90640 1 -- 172.21.15.98:0/1598026536 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb5ec0a2d80 msgr2=0x7fb5ec0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.538+0000 7fb609b90640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb5ec0a2d80 0x7fb5ec0a5240 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc004ab0 tx=0x7fb5fc02a8e0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.538+0000 7fb609b90640 1 -- 172.21.15.98:0/1598026536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6041021c0 msgr2=0x7fb604198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.538+0000 7fb609b90640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6041021c0 0x7fb604198700 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fb5f800d950 tx=0x7fb5f800de10 comp rx=0 tx=0).stop 2024-05-09T09:01:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.538+0000 7fb609b90640 1 -- 172.21.15.98:0/1598026536 shutdown_connections 2024-05-09T09:01:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.538+0000 7fb609b90640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb5ec0a2d80 0x7fb5ec0a5240 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.538+0000 7fb609b90640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb604107810 0x7fb60419ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.538+0000 7fb609b90640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6041021c0 0x7fb604198700 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.538+0000 7fb609b90640 1 --2- 172.21.15.98:0/1598026536 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb604101800 0x7fb6041981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.538+0000 7fb609b90640 1 -- 172.21.15.98:0/1598026536 >> 172.21.15.98:0/1598026536 conn(0x7fb6040fd270 msgr2=0x7fb6040fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.538+0000 7fb609b90640 1 -- 172.21.15.98:0/1598026536 shutdown_connections 2024-05-09T09:01:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.538+0000 7fb609b90640 1 -- 172.21.15.98:0/1598026536 wait complete. 2024-05-09T09:01:59.548 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T09:01:59.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:01:59.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.651+0000 7f09e8c38640 1 -- 172.21.15.98:0/2366704934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09e410be60 msgr2=0x7f09e410e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.651+0000 7f09e8c38640 1 --2- 172.21.15.98:0/2366704934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09e410be60 0x7f09e410e290 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009900 tx=0x7f09d402a480 comp rx=0 tx=0).stop 2024-05-09T09:01:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.651+0000 7f09e8c38640 1 -- 172.21.15.98:0/2366704934 shutdown_connections 2024-05-09T09:01:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.651+0000 7f09e8c38640 1 --2- 172.21.15.98:0/2366704934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09e410be60 0x7f09e410e290 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.651+0000 7f09e8c38640 1 --2- 172.21.15.98:0/2366704934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09e41033b0 0x7f09e410b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.651+0000 7f09e8c38640 1 --2- 172.21.15.98:0/2366704934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f09e4102a90 0x7f09e4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.651+0000 7f09e8c38640 1 -- 172.21.15.98:0/2366704934 >> 172.21.15.98:0/2366704934 conn(0x7f09e4076ab0 msgr2=0x7f09e4076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.651+0000 7f09e8c38640 1 -- 172.21.15.98:0/2366704934 shutdown_connections 2024-05-09T09:01:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.651+0000 7f09e8c38640 1 -- 172.21.15.98:0/2366704934 wait complete. 2024-05-09T09:01:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.652+0000 7f09e8c38640 1 Processor -- start 2024-05-09T09:01:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.652+0000 7f09e8c38640 1 -- start start 2024-05-09T09:01:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.652+0000 7f09e8c38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09e4102a90 0x7f09e41006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.652+0000 7f09e8c38640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f09e41033b0 0x7f09e4100c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.652+0000 7f09e8c38640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09e410be60 0x7f09e4101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.652+0000 7f09e8c38640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09e4111bf0 con 0x7f09e4102a90 2024-05-09T09:01:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.652+0000 7f09e8c38640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f09e4111a70 con 0x7f09e41033b0 2024-05-09T09:01:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.652+0000 7f09e8c38640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f09e4111d70 con 0x7f09e410be60 2024-05-09T09:01:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.652+0000 7f09e259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09e4102a90 0x7f09e41006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.652+0000 7f09e259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09e4102a90 0x7f09e41006e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37712/0 (socket says 172.21.15.98:37712) 2024-05-09T09:01:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.652+0000 7f09e259c640 1 -- 172.21.15.98:0/3520996217 learned_addr learned my addr 172.21.15.98:0/3520996217 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.653+0000 7f09e1d9b640 1 --2- 172.21.15.98:0/3520996217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f09e41033b0 0x7f09e4100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.653+0000 7f09e259c640 1 -- 172.21.15.98:0/3520996217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09e410be60 msgr2=0x7f09e4101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.653+0000 7f09e259c640 1 --2- 172.21.15.98:0/3520996217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09e410be60 0x7f09e4101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.653+0000 7f09e259c640 1 -- 172.21.15.98:0/3520996217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f09e41033b0 msgr2=0x7f09e4100c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.653+0000 7f09e259c640 1 --2- 172.21.15.98:0/3520996217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f09e41033b0 0x7f09e4100c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.653+0000 7f09e259c640 1 -- 172.21.15.98:0/3520996217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e41a65d0 con 0x7f09e4102a90 2024-05-09T09:01:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.653+0000 7f09e259c640 1 --2- 172.21.15.98:0/3520996217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09e4102a90 0x7f09e41006e0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00ebc0 tx=0x7f09dc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.653+0000 7f09db7fe640 1 -- 172.21.15.98:0/3520996217 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc010630 con 0x7f09e4102a90 2024-05-09T09:01:59.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.653+0000 7f09db7fe640 1 -- 172.21.15.98:0/3520996217 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09dc010cb0 con 0x7f09e4102a90 2024-05-09T09:01:59.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.653+0000 7f09e8c38640 1 -- 172.21.15.98:0/3520996217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e41a68c0 con 0x7f09e4102a90 2024-05-09T09:01:59.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.653+0000 7f09db7fe640 1 -- 172.21.15.98:0/3520996217 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc018bd0 con 0x7f09e4102a90 2024-05-09T09:01:59.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.653+0000 7f09e8c38640 1 -- 172.21.15.98:0/3520996217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e41a6e10 con 0x7f09e4102a90 2024-05-09T09:01:59.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.654+0000 7f09e8c38640 1 -- 172.21.15.98:0/3520996217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09e4068aa0 con 0x7f09e4102a90 2024-05-09T09:01:59.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.655+0000 7f09db7fe640 1 -- 172.21.15.98:0/3520996217 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f09dc0107d0 con 0x7f09e4102a90 2024-05-09T09:01:59.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.656+0000 7f09db7fe640 1 --2- 172.21.15.98:0/3520996217 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09d00a2d00 0x7f09d00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.656+0000 7f09e1d9b640 1 --2- 172.21.15.98:0/3520996217 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09d00a2d00 0x7f09d00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.656+0000 7f09db7fe640 1 -- 172.21.15.98:0/3520996217 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f09dc0cffe0 con 0x7f09e4102a90 2024-05-09T09:01:59.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.657+0000 7f09e1d9b640 1 --2- 172.21.15.98:0/3520996217 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09d00a2d00 0x7f09d00a51c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f09e4102170 tx=0x7f09cc005f90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:59.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.661+0000 7f09db7fe640 1 -- 172.21.15.98:0/3520996217 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f09dc09c7f0 con 0x7f09e4102a90 2024-05-09T09:01:59.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.877+0000 7f09e8c38640 1 -- 172.21.15.98:0/3520996217 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09e4106f50 con 0x7f09e4102a90 2024-05-09T09:01:59.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.877+0000 7f09db7fe640 1 -- 172.21.15.98:0/3520996217 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11941 (secure 0 0 0) 0x7f09dc09c0f0 con 0x7f09e4102a90 2024-05-09T09:01:59.877 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T09:01:59.877 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":26581,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/561027115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":561027115},{"type":"v1","addr":"172.21.15.105:6835","nonce":561027115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:56:52.891273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16641":{"gid":16641,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T09:01:59.877 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 37 2024-05-09T09:01:59.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.879+0000 7f09e8c38640 1 -- 172.21.15.98:0/3520996217 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09d00a2d00 msgr2=0x7f09d00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.879+0000 7f09e8c38640 1 --2- 172.21.15.98:0/3520996217 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09d00a2d00 0x7f09d00a51c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f09e4102170 tx=0x7f09cc005f90 comp rx=0 tx=0).stop 2024-05-09T09:01:59.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.879+0000 7f09e8c38640 1 -- 172.21.15.98:0/3520996217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09e4102a90 msgr2=0x7f09e41006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.879+0000 7f09e8c38640 1 --2- 172.21.15.98:0/3520996217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09e4102a90 0x7f09e41006e0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00ebc0 tx=0x7f09dc00c6a0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.879+0000 7f09e8c38640 1 -- 172.21.15.98:0/3520996217 shutdown_connections 2024-05-09T09:01:59.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.879+0000 7f09e8c38640 1 --2- 172.21.15.98:0/3520996217 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f09d00a2d00 0x7f09d00a51c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.879+0000 7f09e8c38640 1 --2- 172.21.15.98:0/3520996217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09e410be60 0x7f09e4101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.879+0000 7f09e8c38640 1 --2- 172.21.15.98:0/3520996217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f09e41033b0 0x7f09e4100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.879+0000 7f09e8c38640 1 --2- 172.21.15.98:0/3520996217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09e4102a90 0x7f09e41006e0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.879+0000 7f09e8c38640 1 -- 172.21.15.98:0/3520996217 >> 172.21.15.98:0/3520996217 conn(0x7f09e4076ab0 msgr2=0x7f09e40fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:59.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.879+0000 7f09e8c38640 1 -- 172.21.15.98:0/3520996217 shutdown_connections 2024-05-09T09:01:59.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.879+0000 7f09e8c38640 1 -- 172.21.15.98:0/3520996217 wait complete. 2024-05-09T09:01:59.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T09:01:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.994+0000 7fd1029c1640 1 -- 172.21.15.98:0/801992576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0fc107810 msgr2=0x7fd0fc107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.994+0000 7fd1029c1640 1 --2- 172.21.15.98:0/801992576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0fc107810 0x7fd0fc107bf0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009930 tx=0x7fd0ec02a420 comp rx=0 tx=0).stop 2024-05-09T09:01:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.994+0000 7fd1029c1640 1 -- 172.21.15.98:0/801992576 shutdown_connections 2024-05-09T09:01:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.994+0000 7fd1029c1640 1 --2- 172.21.15.98:0/801992576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0fc1021c0 0x7fd0fc10e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.994+0000 7fd1029c1640 1 --2- 172.21.15.98:0/801992576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0fc101800 0x7fd0fc101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.994+0000 7fd1029c1640 1 --2- 172.21.15.98:0/801992576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0fc107810 0x7fd0fc107bf0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.994+0000 7fd1029c1640 1 -- 172.21.15.98:0/801992576 >> 172.21.15.98:0/801992576 conn(0x7fd0fc0fd270 msgr2=0x7fd0fc0ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:01:59.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.994+0000 7fd1029c1640 1 -- 172.21.15.98:0/801992576 shutdown_connections 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.994+0000 7fd1029c1640 1 -- 172.21.15.98:0/801992576 wait complete. 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.995+0000 7fd1029c1640 1 Processor -- start 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.995+0000 7fd1029c1640 1 -- start start 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.995+0000 7fd1029c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0fc101800 0x7fd0fc198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.995+0000 7fd1029c1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0fc1021c0 0x7fd0fc1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.995+0000 7fd1029c1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0fc107810 0x7fd0fc19c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.995+0000 7fd1029c1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc111be0 con 0x7fd0fc101800 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.995+0000 7fd1029c1640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc111a60 con 0x7fd0fc107810 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.995+0000 7fd1029c1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc111d60 con 0x7fd0fc1021c0 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.995+0000 7fd0fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0fc101800 0x7fd0fc198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.995+0000 7fd0fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0fc101800 0x7fd0fc198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37730/0 (socket says 172.21.15.98:37730) 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.995+0000 7fd0fbfff640 1 -- 172.21.15.98:0/2762012875 learned_addr learned my addr 172.21.15.98:0/2762012875 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.995+0000 7fd100f5e640 1 --2- 172.21.15.98:0/2762012875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0fc107810 0x7fd0fc19c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.996+0000 7fd0fbfff640 1 -- 172.21.15.98:0/2762012875 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0fc1021c0 msgr2=0x7fd0fc1986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.996+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/2762012875 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0fc1021c0 0x7fd0fc1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.996+0000 7fd0fbfff640 1 -- 172.21.15.98:0/2762012875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0fc107810 msgr2=0x7fd0fc19c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:01:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.996+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/2762012875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0fc107810 0x7fd0fc19c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:01:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.996+0000 7fd0fbfff640 1 -- 172.21.15.98:0/2762012875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0fc19d100 con 0x7fd0fc101800 2024-05-09T09:01:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.996+0000 7fd100f5e640 1 --2- 172.21.15.98:0/2762012875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0fc107810 0x7fd0fc19c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:01:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.996+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/2762012875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0fc101800 0x7fd0fc198160 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009230 tx=0x7fd0ec004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:01:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.996+0000 7fd0f97fa640 1 -- 172.21.15.98:0/2762012875 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec03f030 con 0x7fd0fc101800 2024-05-09T09:01:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.996+0000 7fd0f97fa640 1 -- 172.21.15.98:0/2762012875 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0ec003ad0 con 0x7fd0fc101800 2024-05-09T09:01:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.996+0000 7fd1029c1640 1 -- 172.21.15.98:0/2762012875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0fc19d390 con 0x7fd0fc101800 2024-05-09T09:01:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.996+0000 7fd0f97fa640 1 -- 172.21.15.98:0/2762012875 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec035070 con 0x7fd0fc101800 2024-05-09T09:01:59.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.996+0000 7fd1029c1640 1 -- 172.21.15.98:0/2762012875 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0fc1a4c80 con 0x7fd0fc101800 2024-05-09T09:02:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.998+0000 7fd0f97fa640 1 -- 172.21.15.98:0/2762012875 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd0ec042030 con 0x7fd0fc101800 2024-05-09T09:02:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd1057ac740 1 Processor -- start 2024-05-09T09:02:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd1057ac740 1 -- start start 2024-05-09T09:02:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd1057ac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d26f330 0x559f4d26f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd1057ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d279de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd1057ac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d27a320 0x559f4d27c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd1057ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559f4d1d4c40 con 0x559f4d26fc50 2024-05-09T09:02:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd1057ac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559f4d17c580 con 0x559f4d26f330 2024-05-09T09:02:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd1057ac740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559f4d1d0c00 con 0x559f4d27a320 2024-05-09T09:02:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd0fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d279de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd0fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d279de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37738/0 (socket says 172.21.15.98:37738) 2024-05-09T09:02:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd0fbfff640 1 -- 172.21.15.98:0/3458670577 learned_addr learned my addr 172.21.15.98:0/3458670577 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd0fb7fe640 1 --2- 172.21.15.98:0/3458670577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d27a320 0x559f4d27c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd0f97fa640 1 --2- 172.21.15.98:0/2762012875 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd0e40a2d00 0x7fd0e40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd100f5e640 1 --2- 172.21.15.98:0/3458670577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d26f330 0x559f4d26f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd0fbfff640 1 -- 172.21.15.98:0/3458670577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d27a320 msgr2=0x559f4d27c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/3458670577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d27a320 0x559f4d27c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd0fbfff640 1 -- 172.21.15.98:0/3458670577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d26f330 msgr2=0x559f4d26f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/3458670577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d26f330 0x559f4d26f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd0fbfff640 1 -- 172.21.15.98:0/3458670577 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f4cee83f0 con 0x559f4d26fc50 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd100f5e640 1 --2- 172.21.15.98:0/3458670577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d26f330 0x559f4d26f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd100f5e640 1 --2- 172.21.15.98:0/2762012875 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd0e40a2d00 0x7fd0e40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:01:59.999+0000 7fd0f97fa640 1 -- 172.21.15.98:0/2762012875 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fd0ec0efe10 con 0x7fd0fc101800 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.000+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/3458670577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d279de0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec0ed0c0 tx=0x7fd0ec04b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ac6e80a065632e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.000+0000 7fd0e27fc640 1 -- 172.21.15.98:0/3458670577 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec049070 con 0x559f4d26fc50 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.000+0000 7fd100f5e640 1 --2- 172.21.15.98:0/2762012875 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd0e40a2d00 0x7fd0e40a51c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4005fd0 tx=0x7fd0f4004a40 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.000+0000 7fd0e27fc640 1 -- 172.21.15.98:0/3458670577 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0ec0bc620 con 0x559f4d26fc50 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.000+0000 7fd0e27fc640 1 -- 172.21.15.98:0/3458670577 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec02fdf0 con 0x559f4d26fc50 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.000+0000 7fd1057ac740 1 -- 172.21.15.98:0/3458670577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 msgr2=0x559f4d279de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.000+0000 7fd1057ac740 1 --2- 172.21.15.98:0/3458670577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d279de0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec0ed0c0 tx=0x7fd0ec04b000 comp rx=0 tx=0).stop 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.000+0000 7fd1057ac740 1 -- 172.21.15.98:0/3458670577 shutdown_connections 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.000+0000 7fd1057ac740 1 --2- 172.21.15.98:0/3458670577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d27a320 0x559f4d27c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.000+0000 7fd1057ac740 1 --2- 172.21.15.98:0/3458670577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d279de0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.000+0000 7fd1057ac740 1 --2- 172.21.15.98:0/3458670577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d26f330 0x559f4d26f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.000+0000 7fd1057ac740 1 -- 172.21.15.98:0/3458670577 >> 172.21.15.98:0/3458670577 conn(0x559f4d26b650 msgr2=0x559f4d26ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd1057ac740 1 -- 172.21.15.98:0/3458670577 shutdown_connections 2024-05-09T09:02:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd1057ac740 1 -- 172.21.15.98:0/3458670577 wait complete. 2024-05-09T09:02:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd1057ac740 1 -- start start 2024-05-09T09:02:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd1057ac740 1 Processor -- start 2024-05-09T09:02:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd1057ac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d26f330 0x559f4d277260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd1057ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d2777a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd1057ac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 0x559f4d277ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd1057ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559f4d27ced0 con 0x559f4d26fc50 2024-05-09T09:02:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd1057ac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559f4d18fc40 con 0x559f4d27a320 2024-05-09T09:02:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd1057ac740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559f4d27d050 con 0x559f4d26f330 2024-05-09T09:02:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd1057ac740 10 client.0 _ll_register_callbacks cb 0x559f4cee7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd0fb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d2777a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd0fb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d2777a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37750/0 (socket says 172.21.15.98:37750) 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd0fb7fe640 1 -- 172.21.15.98:0/354131667 learned_addr learned my addr 172.21.15.98:0/354131667 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.001+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/354131667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d26f330 0x559f4d277260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd100f5e640 1 --2- 172.21.15.98:0/354131667 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 0x559f4d277ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd0fb7fe640 1 -- 172.21.15.98:0/354131667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d26f330 msgr2=0x559f4d277260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd0fb7fe640 1 --2- 172.21.15.98:0/354131667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d26f330 0x559f4d277260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd0fb7fe640 1 -- 172.21.15.98:0/354131667 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 msgr2=0x559f4d277ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd0fb7fe640 1 --2- 172.21.15.98:0/354131667 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 0x559f4d277ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd0fb7fe640 1 -- 172.21.15.98:0/354131667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f4cfb27d0 con 0x559f4d26fc50 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd100f5e640 1 --2- 172.21.15.98:0/354131667 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 0x559f4d277ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd0fb7fe640 1 --2- 172.21.15.98:0/354131667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d2777a0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00038a0 tx=0x7fd0f0003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8003ca6371ab7878 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd0db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd0db7fe640 1 -- 172.21.15.98:0/354131667 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f001a470 con 0x559f4d26fc50 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd0db7fe640 1 -- 172.21.15.98:0/354131667 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fd0f001a610 con 0x559f4d26fc50 2024-05-09T09:02:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd1057ac740 1 -- 172.21.15.98:0/354131667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0ec0bc620 con 0x559f4d26fc50 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd0db7fe640 1 -- 172.21.15.98:0/354131667 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd0f0023050 con 0x559f4d26fc50 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd1057ac740 10 client.18945 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd1057ac740 1 -- 172.21.15.98:0/354131667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559f4d289060 con 0x559f4d26fc50 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd0db7fe640 1 -- 172.21.15.98:0/354131667 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0ec0bc620 con 0x559f4d26fc50 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd0db7fe640 1 -- 172.21.15.98:0/354131667 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd0f00118f0 con 0x559f4d26fc50 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd1057ac740 10 client.18945 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd1057ac740 10 client.18945 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd1057ac740 20 client.18945 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd1057ac740 1 --2- 172.21.15.98:0/354131667 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559f4d28a5c0 0x559f4d28ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd1057ac740 4 client.18945 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.002+0000 7fd1057ac740 1 -- 172.21.15.98:0/354131667 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f4d1e2370 con 0x559f4d28a5c0 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.003+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/354131667 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559f4d28a5c0 0x559f4d28ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.003+0000 7fd0db7fe640 10 client.18945 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.003+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/354131667 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559f4d28a5c0 0x559f4d28ca00 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=46270e6043a6964e server_cookie=175690828af8e6d0 in_seq=0 out_seq=0 2024-05-09T09:02:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd0db7fe640 1 -- 172.21.15.98:0/354131667 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f4d1e2370 con 0x559f4d28a5c0 2024-05-09T09:02:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd0db7fe640 10 client.18945 handle_command_reply: tid=0 2024-05-09T09:02:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd1057ac740 1 client.18945 shutdown 2024-05-09T09:02:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd1057ac740 1 -- 172.21.15.98:0/354131667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 msgr2=0x559f4d2777a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd1057ac740 1 --2- 172.21.15.98:0/354131667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d2777a0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00038a0 tx=0x7fd0f0003d60 comp rx=0 tx=0).stop 2024-05-09T09:02:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd1057ac740 1 -- 172.21.15.98:0/354131667 shutdown_connections 2024-05-09T09:02:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd1057ac740 1 --2- 172.21.15.98:0/354131667 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559f4d28a5c0 0x559f4d28ca00 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd1057ac740 1 --2- 172.21.15.98:0/354131667 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 0x559f4d277ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd1057ac740 1 --2- 172.21.15.98:0/354131667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d2777a0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd0db7fe640 0 client.18945 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T09:02:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd1057ac740 1 --2- 172.21.15.98:0/354131667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d26f330 0x559f4d277260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd1057ac740 1 -- 172.21.15.98:0/354131667 >> 172.21.15.98:0/354131667 conn(0x559f4d26b650 msgr2=0x559f4d271330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd1057ac740 1 -- 172.21.15.98:0/354131667 shutdown_connections 2024-05-09T09:02:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd1057ac740 1 -- 172.21.15.98:0/354131667 wait complete. 2024-05-09T09:02:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.004+0000 7fd1057ac740 20 client.18945 trim_cache size 0 max 16384 2024-05-09T09:02:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.017+0000 7fd1057ac740 1 Processor -- start 2024-05-09T09:02:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.017+0000 7fd1057ac740 1 -- start start 2024-05-09T09:02:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd1057ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26f330 0x559f4d26af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd1057ac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d26fc50 0x559f4d285bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd1057ac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 0x559f4d279090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd1057ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc1aeef0 con 0x559f4d26f330 2024-05-09T09:02:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd1057ac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc1b0a30 con 0x559f4d27a320 2024-05-09T09:02:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd1057ac740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc1af5c0 con 0x559f4d26fc50 2024-05-09T09:02:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd0fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26f330 0x559f4d26af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd0fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26f330 0x559f4d26af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37756/0 (socket says 172.21.15.98:37756) 2024-05-09T09:02:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd0fbfff640 1 -- 172.21.15.98:0/1213269443 learned_addr learned my addr 172.21.15.98:0/1213269443 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd0fbfff640 1 -- 172.21.15.98:0/1213269443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d26fc50 msgr2=0x559f4d285bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/1213269443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d26fc50 0x559f4d285bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd0fbfff640 1 -- 172.21.15.98:0/1213269443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 msgr2=0x559f4d279090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/1213269443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 0x559f4d279090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd0fbfff640 1 -- 172.21.15.98:0/1213269443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f4d270150 con 0x559f4d26f330 2024-05-09T09:02:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/1213269443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26f330 0x559f4d26af30 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec0bbeb0 tx=0x7fd0ec005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38b7c3846e69dc9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd0e1ffb640 1 -- 172.21.15.98:0/1213269443 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec049040 con 0x559f4d26f330 2024-05-09T09:02:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd0e1ffb640 1 -- 172.21.15.98:0/1213269443 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0ec0bc620 con 0x559f4d26f330 2024-05-09T09:02:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.018+0000 7fd0e1ffb640 1 -- 172.21.15.98:0/1213269443 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec03b460 con 0x559f4d26f330 2024-05-09T09:02:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.019+0000 7fd1057ac740 1 -- 172.21.15.98:0/1213269443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26f330 msgr2=0x559f4d26af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.019+0000 7fd1057ac740 1 --2- 172.21.15.98:0/1213269443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26f330 0x559f4d26af30 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec0bbeb0 tx=0x7fd0ec005e50 comp rx=0 tx=0).stop 2024-05-09T09:02:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.019+0000 7fd1057ac740 1 -- 172.21.15.98:0/1213269443 shutdown_connections 2024-05-09T09:02:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.019+0000 7fd1057ac740 1 --2- 172.21.15.98:0/1213269443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 0x559f4d279090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.019+0000 7fd1057ac740 1 --2- 172.21.15.98:0/1213269443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d26fc50 0x559f4d285bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.019+0000 7fd1057ac740 1 --2- 172.21.15.98:0/1213269443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26f330 0x559f4d26af30 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.019+0000 7fd1057ac740 1 -- 172.21.15.98:0/1213269443 >> 172.21.15.98:0/1213269443 conn(0x559f4d26b650 msgr2=0x559f4d278980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.019+0000 7fd1057ac740 1 -- 172.21.15.98:0/1213269443 shutdown_connections 2024-05-09T09:02:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.019+0000 7fd1057ac740 1 -- 172.21.15.98:0/1213269443 wait complete. 2024-05-09T09:02:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.019+0000 7fd1057ac740 1 -- start start 2024-05-09T09:02:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd1057ac740 1 Processor -- start 2024-05-09T09:02:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd1057ac740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d26f330 0x559f4d274370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd1057ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d2748b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd1057ac740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 0x559f4d2efb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd1057ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc1b0360 con 0x559f4d26fc50 2024-05-09T09:02:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd1057ac740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc1afc90 con 0x559f4d27a320 2024-05-09T09:02:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd1057ac740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc1acef0 con 0x559f4d26f330 2024-05-09T09:02:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd1057ac740 10 client.0 _ll_register_callbacks cb 0x7fd0f0004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd0fb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d2748b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd0fb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d2748b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37764/0 (socket says 172.21.15.98:37764) 2024-05-09T09:02:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd0fb7fe640 1 -- 172.21.15.98:0/3648828859 learned_addr learned my addr 172.21.15.98:0/3648828859 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/3648828859 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d26f330 0x559f4d274370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd100f5e640 1 --2- 172.21.15.98:0/3648828859 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 0x559f4d2efb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd0fb7fe640 1 -- 172.21.15.98:0/3648828859 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d26f330 msgr2=0x559f4d274370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd0fb7fe640 1 --2- 172.21.15.98:0/3648828859 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d26f330 0x559f4d274370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd0fb7fe640 1 -- 172.21.15.98:0/3648828859 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 msgr2=0x559f4d2efb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd0fb7fe640 1 --2- 172.21.15.98:0/3648828859 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 0x559f4d2efb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd0fb7fe640 1 -- 172.21.15.98:0/3648828859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f4d2f0080 con 0x559f4d26fc50 2024-05-09T09:02:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd100f5e640 1 --2- 172.21.15.98:0/3648828859 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 0x559f4d2efb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd0fb7fe640 1 --2- 172.21.15.98:0/3648828859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d2748b0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0001eb0 tx=0x7fd0f000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571f0a26912c0297 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.020+0000 7fd0db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd0db7fe640 1 -- 172.21.15.98:0/3648828859 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0005850 con 0x559f4d26fc50 2024-05-09T09:02:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd0db7fe640 1 -- 172.21.15.98:0/3648828859 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fd0f0011440 con 0x559f4d26fc50 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd1057ac740 1 -- 172.21.15.98:0/3648828859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0ec0bc620 con 0x559f4d26fc50 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd0db7fe640 1 -- 172.21.15.98:0/3648828859 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd0f0011660 con 0x559f4d26fc50 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd1057ac740 10 client.18951 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd1057ac740 1 -- 172.21.15.98:0/3648828859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559f4d2f11e0 con 0x559f4d26fc50 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd0db7fe640 1 -- 172.21.15.98:0/3648828859 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0ec0bc620 con 0x559f4d26fc50 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd0db7fe640 1 -- 172.21.15.98:0/3648828859 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd0f001acc0 con 0x559f4d26fc50 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd1057ac740 10 client.18951 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd1057ac740 10 client.18951 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd1057ac740 20 client.18951 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd1057ac740 1 --2- 172.21.15.98:0/3648828859 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559f4d2f1990 0x559f4d2f3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd1057ac740 4 client.18951 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd1057ac740 1 -- 172.21.15.98:0/3648828859 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559f4d1dc6c0 con 0x559f4d2f1990 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.021+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/3648828859 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559f4d2f1990 0x559f4d2f3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd0db7fe640 10 client.18951 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T09:02:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd0fbfff640 1 --2- 172.21.15.98:0/3648828859 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559f4d2f1990 0x559f4d2f3db0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=288ef58ce0facea9 server_cookie=70bfecb04d9a786b in_seq=0 out_seq=0 2024-05-09T09:02:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd0db7fe640 1 -- 172.21.15.98:0/3648828859 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559f4d1dc6c0 con 0x559f4d2f1990 2024-05-09T09:02:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd0db7fe640 10 client.18951 handle_command_reply: tid=0 2024-05-09T09:02:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd1057ac740 1 client.18951 shutdown 2024-05-09T09:02:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd1057ac740 1 -- 172.21.15.98:0/3648828859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 msgr2=0x559f4d2748b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd1057ac740 1 --2- 172.21.15.98:0/3648828859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d2748b0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0001eb0 tx=0x7fd0f000b960 comp rx=0 tx=0).stop 2024-05-09T09:02:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd1057ac740 1 -- 172.21.15.98:0/3648828859 shutdown_connections 2024-05-09T09:02:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd1057ac740 1 --2- 172.21.15.98:0/3648828859 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x559f4d2f1990 0x559f4d2f3db0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd1057ac740 1 --2- 172.21.15.98:0/3648828859 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559f4d27a320 0x559f4d2efb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd1057ac740 1 --2- 172.21.15.98:0/3648828859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f4d26fc50 0x559f4d2748b0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd1057ac740 1 --2- 172.21.15.98:0/3648828859 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559f4d26f330 0x559f4d274370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd0db7fe640 0 client.18951 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T09:02:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd1057ac740 1 -- 172.21.15.98:0/3648828859 >> 172.21.15.98:0/3648828859 conn(0x559f4d26b650 msgr2=0x559f4d277a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd1057ac740 1 -- 172.21.15.98:0/3648828859 shutdown_connections 2024-05-09T09:02:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd1057ac740 1 -- 172.21.15.98:0/3648828859 wait complete. 2024-05-09T09:02:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.022+0000 7fd1057ac740 20 client.18951 trim_cache size 0 max 16384 2024-05-09T09:02:00.029 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:02:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.023+0000 7fd1029c1640 1 -- 172.21.15.98:0/2762012875 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd0e40a2d00 msgr2=0x7fd0e40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.023+0000 7fd1029c1640 1 --2- 172.21.15.98:0/2762012875 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd0e40a2d00 0x7fd0e40a51c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4005fd0 tx=0x7fd0f4004a40 comp rx=0 tx=0).stop 2024-05-09T09:02:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.023+0000 7fd1029c1640 1 -- 172.21.15.98:0/2762012875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0fc101800 msgr2=0x7fd0fc198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.023+0000 7fd1029c1640 1 --2- 172.21.15.98:0/2762012875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0fc101800 0x7fd0fc198160 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009230 tx=0x7fd0ec004270 comp rx=0 tx=0).stop 2024-05-09T09:02:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.023+0000 7fd1029c1640 1 -- 172.21.15.98:0/2762012875 shutdown_connections 2024-05-09T09:02:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.023+0000 7fd1029c1640 1 --2- 172.21.15.98:0/2762012875 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd0e40a2d00 0x7fd0e40a51c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.023+0000 7fd1029c1640 1 --2- 172.21.15.98:0/2762012875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0fc107810 0x7fd0fc19c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.023+0000 7fd1029c1640 1 --2- 172.21.15.98:0/2762012875 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0fc1021c0 0x7fd0fc1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.023+0000 7fd1029c1640 1 --2- 172.21.15.98:0/2762012875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0fc101800 0x7fd0fc198160 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.023+0000 7fd1029c1640 1 -- 172.21.15.98:0/2762012875 >> 172.21.15.98:0/2762012875 conn(0x7fd0fc0fd270 msgr2=0x7fd0fc0fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.023+0000 7fd1029c1640 1 -- 172.21.15.98:0/2762012875 shutdown_connections 2024-05-09T09:02:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.023+0000 7fd1029c1640 1 -- 172.21.15.98:0/2762012875 wait complete. 2024-05-09T09:02:00.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T09:02:00.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.137+0000 7f649a590640 1 -- 172.21.15.98:0/4081118885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64941021c0 msgr2=0x7f649410e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.137+0000 7f649a590640 1 --2- 172.21.15.98:0/4081118885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64941021c0 0x7f649410e6d0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f648c009cd0 tx=0x7f648c02a8e0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.137+0000 7f649a590640 1 -- 172.21.15.98:0/4081118885 shutdown_connections 2024-05-09T09:02:00.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.137+0000 7f649a590640 1 --2- 172.21.15.98:0/4081118885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64941021c0 0x7f649410e6d0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.137+0000 7f649a590640 1 --2- 172.21.15.98:0/4081118885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6494101800 0x7f6494101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.137+0000 7f649a590640 1 --2- 172.21.15.98:0/4081118885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6494107810 0x7f6494107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.137+0000 7f649a590640 1 -- 172.21.15.98:0/4081118885 >> 172.21.15.98:0/4081118885 conn(0x7f6494078c80 msgr2=0x7f6494079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.137+0000 7f649a590640 1 -- 172.21.15.98:0/4081118885 shutdown_connections 2024-05-09T09:02:00.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.137+0000 7f649a590640 1 -- 172.21.15.98:0/4081118885 wait complete. 2024-05-09T09:02:00.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f649a590640 1 Processor -- start 2024-05-09T09:02:00.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f649a590640 1 -- start start 2024-05-09T09:02:00.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f649a590640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6494101800 0x7f6494198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f649a590640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64941021c0 0x7f64941986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f649a590640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6494107810 0x7f649419ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f649a590640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6494111c20 con 0x7f64941021c0 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f649a590640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6494111aa0 con 0x7f6494101800 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f649a590640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6494111da0 con 0x7f6494107810 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f64937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64941021c0 0x7f64941986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f64937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64941021c0 0x7f64941986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37784/0 (socket says 172.21.15.98:37784) 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f64937fe640 1 -- 172.21.15.98:0/2258385566 learned_addr learned my addr 172.21.15.98:0/2258385566 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f6493fff640 1 --2- 172.21.15.98:0/2258385566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6494101800 0x7f6494198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f64937fe640 1 -- 172.21.15.98:0/2258385566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6494107810 msgr2=0x7f649419ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f6498b2d640 1 --2- 172.21.15.98:0/2258385566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6494107810 0x7f649419ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f64937fe640 1 --2- 172.21.15.98:0/2258385566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6494107810 0x7f649419ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f64937fe640 1 -- 172.21.15.98:0/2258385566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6494101800 msgr2=0x7f6494198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f64937fe640 1 --2- 172.21.15.98:0/2258385566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6494101800 0x7f6494198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.138+0000 7f64937fe640 1 -- 172.21.15.98:0/2258385566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f649419d150 con 0x7f64941021c0 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.139+0000 7f64937fe640 1 --2- 172.21.15.98:0/2258385566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64941021c0 0x7f64941986a0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f648800d950 tx=0x7f648800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.139+0000 7f64917fa640 1 -- 172.21.15.98:0/2258385566 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6488017070 con 0x7f64941021c0 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.139+0000 7f64917fa640 1 -- 172.21.15.98:0/2258385566 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f648800de80 con 0x7f64941021c0 2024-05-09T09:02:00.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.139+0000 7f649a590640 1 -- 172.21.15.98:0/2258385566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f649419d440 con 0x7f64941021c0 2024-05-09T09:02:00.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.139+0000 7f64917fa640 1 -- 172.21.15.98:0/2258385566 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f648800f620 con 0x7f64941021c0 2024-05-09T09:02:00.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.139+0000 7f649a590640 1 -- 172.21.15.98:0/2258385566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64941a4cd0 con 0x7f64941021c0 2024-05-09T09:02:00.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f649d378740 1 Processor -- start 2024-05-09T09:02:00.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f649d378740 1 -- start start 2024-05-09T09:02:00.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f64917fa640 1 -- 172.21.15.98:0/2258385566 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6488009c30 con 0x7f64941021c0 2024-05-09T09:02:00.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f649d378740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a275ffb330 0x55a275ffb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f649d378740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a276005de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f649d378740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a276006320 0x55a276008710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f649d378740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a275f60c40 con 0x55a275ffbc50 2024-05-09T09:02:00.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f649d378740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a275f08580 con 0x55a276006320 2024-05-09T09:02:00.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f649d378740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a275f5cc00 con 0x55a275ffb330 2024-05-09T09:02:00.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f6493fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a276005de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f6493fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a276005de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37790/0 (socket says 172.21.15.98:37790) 2024-05-09T09:02:00.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f6493fff640 1 -- 172.21.15.98:0/251388736 learned_addr learned my addr 172.21.15.98:0/251388736 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f64937fe640 1 --2- 172.21.15.98:0/251388736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a276006320 0x55a276008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f6498b2d640 1 --2- 172.21.15.98:0/251388736 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a275ffb330 0x55a275ffb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f6493fff640 1 -- 172.21.15.98:0/251388736 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a275ffb330 msgr2=0x55a275ffb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f6493fff640 1 --2- 172.21.15.98:0/251388736 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a275ffb330 0x55a275ffb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f6493fff640 1 -- 172.21.15.98:0/251388736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a276006320 msgr2=0x55a276008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.142+0000 7f6493fff640 1 --2- 172.21.15.98:0/251388736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a276006320 0x55a276008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.143+0000 7f6493fff640 1 -- 172.21.15.98:0/251388736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a275c743f0 con 0x55a275ffbc50 2024-05-09T09:02:00.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.143+0000 7f64917fa640 1 --2- 172.21.15.98:0/2258385566 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f647c0a2d00 0x7f647c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.143+0000 7f6493fff640 1 --2- 172.21.15.98:0/251388736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a276005de0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f6484009a70 tx=0x7f6484009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17afb9b6c4d4add9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.143+0000 7f6498b2d640 1 --2- 172.21.15.98:0/2258385566 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f647c0a2d00 0x7f647c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.143+0000 7f647a7fc640 1 -- 172.21.15.98:0/251388736 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6484017070 con 0x55a275ffbc50 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.143+0000 7f647a7fc640 1 -- 172.21.15.98:0/251388736 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6484013470 con 0x55a275ffbc50 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.143+0000 7f64917fa640 1 -- 172.21.15.98:0/2258385566 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f64880cf4d0 con 0x7f64941021c0 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.143+0000 7f647a7fc640 1 -- 172.21.15.98:0/251388736 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6484012630 con 0x55a275ffbc50 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 -- 172.21.15.98:0/251388736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 msgr2=0x55a276005de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f6498b2d640 1 --2- 172.21.15.98:0/2258385566 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f647c0a2d00 0x7f647c0a51c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f648c009a60 tx=0x7f648c005ef0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 --2- 172.21.15.98:0/251388736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a276005de0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f6484009a70 tx=0x7f6484009f30 comp rx=0 tx=0).stop 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 -- 172.21.15.98:0/251388736 shutdown_connections 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 --2- 172.21.15.98:0/251388736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a276006320 0x55a276008710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 --2- 172.21.15.98:0/251388736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a276005de0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 --2- 172.21.15.98:0/251388736 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a275ffb330 0x55a275ffb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 -- 172.21.15.98:0/251388736 >> 172.21.15.98:0/251388736 conn(0x55a275ff7650 msgr2=0x55a275ff7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 -- 172.21.15.98:0/251388736 shutdown_connections 2024-05-09T09:02:00.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 -- 172.21.15.98:0/251388736 wait complete. 2024-05-09T09:02:00.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 -- start start 2024-05-09T09:02:00.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 Processor -- start 2024-05-09T09:02:00.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a275ffb330 0x55a276003230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a276003770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a276006320 0x55a276003cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a27600d520 con 0x55a275ffbc50 2024-05-09T09:02:00.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a275f1bc40 con 0x55a275ffb330 2024-05-09T09:02:00.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a27600d6a0 con 0x55a276006320 2024-05-09T09:02:00.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.144+0000 7f649d378740 10 client.0 _ll_register_callbacks cb 0x55a275c73be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:00.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a276003770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a276003770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37794/0 (socket says 172.21.15.98:37794) 2024-05-09T09:02:00.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64937fe640 1 -- 172.21.15.98:0/4282479172 learned_addr learned my addr 172.21.15.98:0/4282479172 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f6493fff640 1 --2- 172.21.15.98:0/4282479172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a275ffb330 0x55a276003230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64937fe640 1 -- 172.21.15.98:0/4282479172 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a276006320 msgr2=0x55a276003cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64937fe640 1 --2- 172.21.15.98:0/4282479172 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a276006320 0x55a276003cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64937fe640 1 -- 172.21.15.98:0/4282479172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a275ffb330 msgr2=0x55a276003230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64937fe640 1 --2- 172.21.15.98:0/4282479172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a275ffb330 0x55a276003230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64937fe640 1 -- 172.21.15.98:0/4282479172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a275d3e7d0 con 0x55a275ffbc50 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64937fe640 1 --2- 172.21.15.98:0/4282479172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a276003770 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f6488005fd0 tx=0x7f648809b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d980d61accd12466 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64737fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64737fe640 1 -- 172.21.15.98:0/4282479172 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6488020050 con 0x55a275ffbc50 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64737fe640 1 -- 172.21.15.98:0/4282479172 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f64880cf120 con 0x55a275ffbc50 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f649d378740 1 -- 172.21.15.98:0/4282479172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6484013470 con 0x55a275ffbc50 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64737fe640 1 -- 172.21.15.98:0/4282479172 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f64880cf320 con 0x55a275ffbc50 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f649d378740 10 client.18963 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f649d378740 1 -- 172.21.15.98:0/4282479172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a276019740 con 0x55a275ffbc50 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64737fe640 1 -- 172.21.15.98:0/4282479172 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6484013470 con 0x55a275ffbc50 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f64737fe640 1 -- 172.21.15.98:0/4282479172 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f648809aa30 con 0x55a275ffbc50 2024-05-09T09:02:00.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.145+0000 7f649d378740 10 client.18963 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.146+0000 7f649d378740 10 client.18963 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.146+0000 7f649d378740 20 client.18963 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.146+0000 7f649d378740 1 --2- 172.21.15.98:0/4282479172 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a27601ac50 0x55a27601d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.146+0000 7f649d378740 4 client.18963 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.146+0000 7f649d378740 1 -- 172.21.15.98:0/4282479172 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a275f6e370 con 0x55a27601ac50 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.146+0000 7f6493fff640 1 --2- 172.21.15.98:0/4282479172 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a27601ac50 0x55a27601d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.146+0000 7f64737fe640 10 client.18963 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.146+0000 7f6493fff640 1 --2- 172.21.15.98:0/4282479172 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a27601ac50 0x55a27601d090 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=841baa8fd8490f08 server_cookie=271314410ee1c92b in_seq=0 out_seq=0 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f64737fe640 1 -- 172.21.15.98:0/4282479172 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a275f6e370 con 0x55a27601ac50 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f64737fe640 10 client.18963 handle_command_reply: tid=0 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f649d378740 1 client.18963 shutdown 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f649d378740 1 -- 172.21.15.98:0/4282479172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 msgr2=0x55a276003770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f649d378740 1 --2- 172.21.15.98:0/4282479172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a276003770 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f6488005fd0 tx=0x7f648809b3b0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f649d378740 1 -- 172.21.15.98:0/4282479172 shutdown_connections 2024-05-09T09:02:00.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f649d378740 1 --2- 172.21.15.98:0/4282479172 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a27601ac50 0x55a27601d090 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f649d378740 1 --2- 172.21.15.98:0/4282479172 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a276006320 0x55a276003cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f649d378740 1 --2- 172.21.15.98:0/4282479172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a276003770 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f649d378740 1 --2- 172.21.15.98:0/4282479172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a275ffb330 0x55a276003230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f64737fe640 0 client.18963 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T09:02:00.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f649d378740 1 -- 172.21.15.98:0/4282479172 >> 172.21.15.98:0/4282479172 conn(0x55a275ff7650 msgr2=0x55a275ff67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f649d378740 1 -- 172.21.15.98:0/4282479172 shutdown_connections 2024-05-09T09:02:00.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f649d378740 1 -- 172.21.15.98:0/4282479172 wait complete. 2024-05-09T09:02:00.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.147+0000 7f649d378740 20 client.18963 trim_cache size 0 max 16384 2024-05-09T09:02:00.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.160+0000 7f649d378740 1 Processor -- start 2024-05-09T09:02:00.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.160+0000 7f649d378740 1 -- start start 2024-05-09T09:02:00.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.160+0000 7f649d378740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffb330 0x55a276016410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.160+0000 7f649d378740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a275ffbc50 0x55a276002390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.160+0000 7f649d378740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a276006320 0x55a275ffda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.160+0000 7f649d378740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f64941acc60 con 0x55a275ffb330 2024-05-09T09:02:00.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.160+0000 7f649d378740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f64941ae7a0 con 0x55a275ffbc50 2024-05-09T09:02:00.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.160+0000 7f649d378740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f64941ad330 con 0x55a276006320 2024-05-09T09:02:00.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.161+0000 7f6493fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffb330 0x55a276016410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.161+0000 7f6493fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffb330 0x55a276016410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37806/0 (socket says 172.21.15.98:37806) 2024-05-09T09:02:00.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.161+0000 7f6493fff640 1 -- 172.21.15.98:0/3177107499 learned_addr learned my addr 172.21.15.98:0/3177107499 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.161+0000 7f6493fff640 1 -- 172.21.15.98:0/3177107499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a276006320 msgr2=0x55a275ffda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.161+0000 7f6493fff640 1 --2- 172.21.15.98:0/3177107499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a276006320 0x55a275ffda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.161+0000 7f6493fff640 1 -- 172.21.15.98:0/3177107499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a275ffbc50 msgr2=0x55a276002390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.161+0000 7f6493fff640 1 --2- 172.21.15.98:0/3177107499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a275ffbc50 0x55a276002390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.161+0000 7f6493fff640 1 -- 172.21.15.98:0/3177107499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a275ff8880 con 0x55a275ffb330 2024-05-09T09:02:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.161+0000 7f6493fff640 1 --2- 172.21.15.98:0/3177107499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffb330 0x55a276016410 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f6484009a70 tx=0x7f6484009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da0633fa328ddbb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.161+0000 7f6479ffb640 1 -- 172.21.15.98:0/3177107499 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6484017040 con 0x55a275ffb330 2024-05-09T09:02:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.161+0000 7f6479ffb640 1 -- 172.21.15.98:0/3177107499 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f648402b070 con 0x55a275ffb330 2024-05-09T09:02:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.161+0000 7f6479ffb640 1 -- 172.21.15.98:0/3177107499 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6484013890 con 0x55a275ffb330 2024-05-09T09:02:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 -- 172.21.15.98:0/3177107499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffb330 msgr2=0x55a276016410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 --2- 172.21.15.98:0/3177107499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffb330 0x55a276016410 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f6484009a70 tx=0x7f6484009110 comp rx=0 tx=0).stop 2024-05-09T09:02:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 -- 172.21.15.98:0/3177107499 shutdown_connections 2024-05-09T09:02:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 --2- 172.21.15.98:0/3177107499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a276006320 0x55a275ffda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 --2- 172.21.15.98:0/3177107499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a275ffbc50 0x55a276002390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 --2- 172.21.15.98:0/3177107499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffb330 0x55a276016410 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 -- 172.21.15.98:0/3177107499 >> 172.21.15.98:0/3177107499 conn(0x55a275ff7650 msgr2=0x55a275ff5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 -- 172.21.15.98:0/3177107499 shutdown_connections 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 -- 172.21.15.98:0/3177107499 wait complete. 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 -- start start 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 Processor -- start 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a275ffb330 0x55a276012d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a275fffc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a276006320 0x55a2760001f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f64941ae0d0 con 0x55a275ffbc50 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f64941ada00 con 0x55a275ffb330 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f64941aac60 con 0x55a276006320 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.162+0000 7f649d378740 10 client.0 _ll_register_callbacks cb 0x7f64880d2e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a275fffc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a275fffc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37812/0 (socket says 172.21.15.98:37812) 2024-05-09T09:02:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64937fe640 1 -- 172.21.15.98:0/2498243415 learned_addr learned my addr 172.21.15.98:0/2498243415 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64937fe640 1 -- 172.21.15.98:0/2498243415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a276006320 msgr2=0x55a2760001f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f6498b2d640 1 --2- 172.21.15.98:0/2498243415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a276006320 0x55a2760001f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64937fe640 1 --2- 172.21.15.98:0/2498243415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a276006320 0x55a2760001f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64937fe640 1 -- 172.21.15.98:0/2498243415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a275ffb330 msgr2=0x55a276012d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64937fe640 1 --2- 172.21.15.98:0/2498243415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a275ffb330 0x55a276012d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64937fe640 1 -- 172.21.15.98:0/2498243415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a276000a20 con 0x55a275ffbc50 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f6498b2d640 1 --2- 172.21.15.98:0/2498243415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a276006320 0x55a2760001f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64937fe640 1 --2- 172.21.15.98:0/2498243415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a275fffc90 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f6488009610 tx=0x7f648809b4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e5630e6002288d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64737fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64737fe640 1 -- 172.21.15.98:0/2498243415 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64880cf330 con 0x55a275ffbc50 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64737fe640 1 -- 172.21.15.98:0/2498243415 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f6488084d20 con 0x55a275ffbc50 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f649d378740 1 -- 172.21.15.98:0/2498243415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f648402b070 con 0x55a275ffbc50 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64737fe640 1 -- 172.21.15.98:0/2498243415 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f648809bce0 con 0x55a275ffbc50 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f649d378740 10 client.18969 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:00.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f649d378740 1 -- 172.21.15.98:0/2498243415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a276081260 con 0x55a275ffbc50 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64737fe640 1 -- 172.21.15.98:0/2498243415 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f648402b070 con 0x55a275ffbc50 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.163+0000 7f64737fe640 1 -- 172.21.15.98:0/2498243415 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f64880b4230 con 0x55a275ffbc50 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.164+0000 7f649d378740 10 client.18969 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.164+0000 7f649d378740 10 client.18969 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.164+0000 7f649d378740 20 client.18969 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.164+0000 7f649d378740 1 --2- 172.21.15.98:0/2498243415 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a276081950 0x55a276083d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.164+0000 7f649d378740 4 client.18969 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.164+0000 7f649d378740 1 -- 172.21.15.98:0/2498243415 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a275f686c0 con 0x55a276081950 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.164+0000 7f6493fff640 1 --2- 172.21.15.98:0/2498243415 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a276081950 0x55a276083d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.164+0000 7f64737fe640 10 client.18969 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.164+0000 7f6493fff640 1 --2- 172.21.15.98:0/2498243415 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a276081950 0x55a276083d70 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1c0078d2d5f7445e server_cookie=c97ce528276114ba in_seq=0 out_seq=0 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.164+0000 7f64737fe640 1 -- 172.21.15.98:0/2498243415 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a275f686c0 con 0x55a276081950 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.164+0000 7f64737fe640 10 client.18969 handle_command_reply: tid=0 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.164+0000 7f649d378740 1 client.18969 shutdown 2024-05-09T09:02:00.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649d378740 1 -- 172.21.15.98:0/2498243415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 msgr2=0x55a275fffc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649d378740 1 --2- 172.21.15.98:0/2498243415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a275fffc90 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f6488009610 tx=0x7f648809b4e0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649d378740 1 -- 172.21.15.98:0/2498243415 shutdown_connections 2024-05-09T09:02:00.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649d378740 1 --2- 172.21.15.98:0/2498243415 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x55a276081950 0x55a276083d70 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649d378740 1 --2- 172.21.15.98:0/2498243415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a276006320 0x55a2760001f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649d378740 1 --2- 172.21.15.98:0/2498243415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a275ffbc50 0x55a275fffc90 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f64737fe640 0 client.18969 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T09:02:00.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649d378740 1 --2- 172.21.15.98:0/2498243415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a275ffb330 0x55a276012d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649d378740 1 -- 172.21.15.98:0/2498243415 >> 172.21.15.98:0/2498243415 conn(0x55a275ff7650 msgr2=0x55a275ff67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649d378740 1 -- 172.21.15.98:0/2498243415 shutdown_connections 2024-05-09T09:02:00.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649d378740 1 -- 172.21.15.98:0/2498243415 wait complete. 2024-05-09T09:02:00.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649d378740 20 client.18969 trim_cache size 0 max 16384 2024-05-09T09:02:00.171 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:02:00.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649a590640 1 -- 172.21.15.98:0/2258385566 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f647c0a2d00 msgr2=0x7f647c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649a590640 1 --2- 172.21.15.98:0/2258385566 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f647c0a2d00 0x7f647c0a51c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f648c009a60 tx=0x7f648c005ef0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649a590640 1 -- 172.21.15.98:0/2258385566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64941021c0 msgr2=0x7f64941986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649a590640 1 --2- 172.21.15.98:0/2258385566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64941021c0 0x7f64941986a0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f648800d950 tx=0x7f648800dd50 comp rx=0 tx=0).stop 2024-05-09T09:02:00.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649a590640 1 -- 172.21.15.98:0/2258385566 shutdown_connections 2024-05-09T09:02:00.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649a590640 1 --2- 172.21.15.98:0/2258385566 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f647c0a2d00 0x7f647c0a51c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649a590640 1 --2- 172.21.15.98:0/2258385566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6494107810 0x7f649419ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649a590640 1 --2- 172.21.15.98:0/2258385566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64941021c0 0x7f64941986a0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649a590640 1 --2- 172.21.15.98:0/2258385566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6494101800 0x7f6494198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649a590640 1 -- 172.21.15.98:0/2258385566 >> 172.21.15.98:0/2258385566 conn(0x7f6494078c80 msgr2=0x7f64940ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649a590640 1 -- 172.21.15.98:0/2258385566 shutdown_connections 2024-05-09T09:02:00.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.165+0000 7f649a590640 1 -- 172.21.15.98:0/2258385566 wait complete. 2024-05-09T09:02:00.176 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T09:02:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.279+0000 7fe1bce9a640 1 -- 172.21.15.98:0/2839454830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b806a7f0 msgr2=0x7fe1b810e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.279+0000 7fe1bce9a640 1 --2- 172.21.15.98:0/2839454830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b806a7f0 0x7fe1b810e710 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009ca0 tx=0x7fe1b002a950 comp rx=0 tx=0).stop 2024-05-09T09:02:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.279+0000 7fe1bce9a640 1 -- 172.21.15.98:0/2839454830 shutdown_connections 2024-05-09T09:02:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.279+0000 7fe1bce9a640 1 --2- 172.21.15.98:0/2839454830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b806a7f0 0x7fe1b810e710 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.279+0000 7fe1bce9a640 1 --2- 172.21.15.98:0/2839454830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1b8069e50 0x7fe1b806a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.279+0000 7fe1bce9a640 1 --2- 172.21.15.98:0/2839454830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1b8107910 0x7fe1b8107cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.279+0000 7fe1bce9a640 1 -- 172.21.15.98:0/2839454830 >> 172.21.15.98:0/2839454830 conn(0x7fe1b8077710 msgr2=0x7fe1b8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.279+0000 7fe1bce9a640 1 -- 172.21.15.98:0/2839454830 shutdown_connections 2024-05-09T09:02:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.279+0000 7fe1bce9a640 1 -- 172.21.15.98:0/2839454830 wait complete. 2024-05-09T09:02:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.280+0000 7fe1bce9a640 1 Processor -- start 2024-05-09T09:02:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.280+0000 7fe1bce9a640 1 -- start start 2024-05-09T09:02:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.280+0000 7fe1bce9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b8069e50 0x7fe1b8198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.280+0000 7fe1bce9a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1b806a7f0 0x7fe1b81985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.280+0000 7fe1bce9a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1b8107910 0x7fe1b819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.280+0000 7fe1bce9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b8111c90 con 0x7fe1b8069e50 2024-05-09T09:02:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.280+0000 7fe1bce9a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b8111b10 con 0x7fe1b806a7f0 2024-05-09T09:02:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.280+0000 7fe1bce9a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b8111e10 con 0x7fe1b8107910 2024-05-09T09:02:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.280+0000 7fe1b659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b8069e50 0x7fe1b8198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1b659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b8069e50 0x7fe1b8198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37826/0 (socket says 172.21.15.98:37826) 2024-05-09T09:02:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1b659c640 1 -- 172.21.15.98:0/218868937 learned_addr learned my addr 172.21.15.98:0/218868937 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1b5d9b640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1b806a7f0 0x7fe1b81985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1b6d9d640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1b8107910 0x7fe1b819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1b659c640 1 -- 172.21.15.98:0/218868937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1b8107910 msgr2=0x7fe1b819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1b659c640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1b8107910 0x7fe1b819c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1b659c640 1 -- 172.21.15.98:0/218868937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1b806a7f0 msgr2=0x7fe1b81985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1b659c640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1b806a7f0 0x7fe1b81985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1b659c640 1 -- 172.21.15.98:0/218868937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b819cfc0 con 0x7fe1b8069e50 2024-05-09T09:02:00.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1b5d9b640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1b806a7f0 0x7fe1b81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1b659c640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b8069e50 0x7fe1b8198070 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800e9f0 tx=0x7fe1a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1a77fe640 1 -- 172.21.15.98:0/218868937 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a800cd80 con 0x7fe1b8069e50 2024-05-09T09:02:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1a77fe640 1 -- 172.21.15.98:0/218868937 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1a8005500 con 0x7fe1b8069e50 2024-05-09T09:02:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1bce9a640 1 -- 172.21.15.98:0/218868937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b819d2b0 con 0x7fe1b8069e50 2024-05-09T09:02:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1a77fe640 1 -- 172.21.15.98:0/218868937 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8010690 con 0x7fe1b8069e50 2024-05-09T09:02:00.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.281+0000 7fe1bce9a640 1 -- 172.21.15.98:0/218868937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b81a4b40 con 0x7fe1b8069e50 2024-05-09T09:02:00.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.283+0000 7fe1a77fe640 1 -- 172.21.15.98:0/218868937 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe1a80046c0 con 0x7fe1b8069e50 2024-05-09T09:02:00.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.283+0000 7fe1bfba2740 1 Processor -- start 2024-05-09T09:02:00.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1bfba2740 1 -- start start 2024-05-09T09:02:00.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1a77fe640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe1a00a2d00 0x7fe1a00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1bfba2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b75e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1bfba2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 0x55e85b76d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1bfba2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b76d730 0x55e85b76fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1bfba2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e85b6c3c40 con 0x55e85b75e230 2024-05-09T09:02:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1bfba2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e85b66b580 con 0x55e85b76d730 2024-05-09T09:02:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1bfba2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e85b6bfc00 con 0x55e85b75eb50 2024-05-09T09:02:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1b6d9d640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe1a00a2d00 0x7fe1a00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1a77fe640 1 -- 172.21.15.98:0/218868937 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fe1a80cfda0 con 0x7fe1b8069e50 2024-05-09T09:02:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1b659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b75e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1b659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b75e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37834/0 (socket says 172.21.15.98:37834) 2024-05-09T09:02:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1b659c640 1 -- 172.21.15.98:0/1840068279 learned_addr learned my addr 172.21.15.98:0/1840068279 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1b5d9b640 1 --2- 172.21.15.98:0/1840068279 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 0x55e85b76d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1b6d9d640 1 --2- 172.21.15.98:0/1840068279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b76d730 0x55e85b76fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1b659c640 1 -- 172.21.15.98:0/1840068279 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 msgr2=0x55e85b76d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1b659c640 1 --2- 172.21.15.98:0/1840068279 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 0x55e85b76d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1b659c640 1 -- 172.21.15.98:0/1840068279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b76d730 msgr2=0x55e85b76fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1b659c640 1 --2- 172.21.15.98:0/1840068279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b76d730 0x55e85b76fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.284+0000 7fe1b659c640 1 -- 172.21.15.98:0/1840068279 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e85b3d73f0 con 0x55e85b75e230 2024-05-09T09:02:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.285+0000 7fe1b659c640 1 --2- 172.21.15.98:0/1840068279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b75e610 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fe1a80a45f0 tx=0x7fe1a809b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6a8d0dc8fe70fd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.285+0000 7fe1b6d9d640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe1a00a2d00 0x7fe1a00a51c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009430 tx=0x7fe1b002f930 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.285+0000 7fe1a4ff9640 1 -- 172.21.15.98:0/1840068279 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a809beb0 con 0x55e85b75e230 2024-05-09T09:02:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.285+0000 7fe1a4ff9640 1 -- 172.21.15.98:0/1840068279 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1a80a0260 con 0x55e85b75e230 2024-05-09T09:02:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.285+0000 7fe1a4ff9640 1 -- 172.21.15.98:0/1840068279 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8098520 con 0x55e85b75e230 2024-05-09T09:02:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.285+0000 7fe1bfba2740 1 -- 172.21.15.98:0/1840068279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 msgr2=0x55e85b75e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.285+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/1840068279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b75e610 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fe1a80a45f0 tx=0x7fe1a809b9f0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.285+0000 7fe1bfba2740 1 -- 172.21.15.98:0/1840068279 shutdown_connections 2024-05-09T09:02:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.285+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/1840068279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b76d730 0x55e85b76fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.285+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/1840068279 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 0x55e85b76d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.285+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/1840068279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b75e610 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.285+0000 7fe1bfba2740 1 -- 172.21.15.98:0/1840068279 >> 172.21.15.98:0/1840068279 conn(0x55e85b75a650 msgr2=0x55e85b75aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1bfba2740 1 -- 172.21.15.98:0/1840068279 shutdown_connections 2024-05-09T09:02:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1bfba2740 1 -- 172.21.15.98:0/1840068279 wait complete. 2024-05-09T09:02:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1bfba2740 1 -- start start 2024-05-09T09:02:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1bfba2740 1 Processor -- start 2024-05-09T09:02:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1bfba2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b76a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1bfba2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 0x55e85b76ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1bfba2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b76d730 0x55e85b76b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1bfba2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e85b7704f0 con 0x55e85b75e230 2024-05-09T09:02:00.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1bfba2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e85b67ec40 con 0x55e85b76d730 2024-05-09T09:02:00.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1bfba2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e85b770670 con 0x55e85b75eb50 2024-05-09T09:02:00.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1bfba2740 10 client.0 _ll_register_callbacks cb 0x55e85b3d6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:00.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1b659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b76a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1b659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b76a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37840/0 (socket says 172.21.15.98:37840) 2024-05-09T09:02:00.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1b659c640 1 -- 172.21.15.98:0/3275810301 learned_addr learned my addr 172.21.15.98:0/3275810301 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1b6d9d640 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b76d730 0x55e85b76b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1b5d9b640 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 0x55e85b76ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1b659c640 1 -- 172.21.15.98:0/3275810301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 msgr2=0x55e85b76ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1b659c640 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 0x55e85b76ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1b659c640 1 -- 172.21.15.98:0/3275810301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b76d730 msgr2=0x55e85b76b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1b659c640 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b76d730 0x55e85b76b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1b659c640 1 -- 172.21.15.98:0/3275810301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e85b4a17d0 con 0x55e85b75e230 2024-05-09T09:02:00.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1b5d9b640 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 0x55e85b76ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:00.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.286+0000 7fe1b6d9d640 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b76d730 0x55e85b76b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:00.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe1b659c640 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b76a6c0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8098310 tx=0x7fe1a809c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87cfaf8c88dffaf7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe19dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:00.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe19dffb640 1 -- 172.21.15.98:0/3275810301 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a809beb0 con 0x55e85b75e230 2024-05-09T09:02:00.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe19dffb640 1 -- 172.21.15.98:0/3275810301 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fe1a8098520 con 0x55e85b75e230 2024-05-09T09:02:00.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3275810301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1a80a0260 con 0x55e85b75e230 2024-05-09T09:02:00.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe19dffb640 1 -- 172.21.15.98:0/3275810301 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe1a80a4690 con 0x55e85b75e230 2024-05-09T09:02:00.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe1bfba2740 10 client.18981 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:00.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3275810301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e85b77c730 con 0x55e85b75e230 2024-05-09T09:02:00.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe19dffb640 1 -- 172.21.15.98:0/3275810301 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1a80b8080 con 0x55e85b75e230 2024-05-09T09:02:00.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe19dffb640 1 -- 172.21.15.98:0/3275810301 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe1a809b300 con 0x55e85b75e230 2024-05-09T09:02:00.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe1bfba2740 10 client.18981 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:00.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe1bfba2740 10 client.18981 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T09:02:00.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe1bfba2740 20 client.18981 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:00.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e85b77da80 0x55e85b77fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe1bfba2740 4 client.18981 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:02:00.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.287+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3275810301 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e85b6d1370 con 0x55e85b77da80 2024-05-09T09:02:00.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.288+0000 7fe1b5d9b640 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e85b77da80 0x55e85b77fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.288+0000 7fe19dffb640 10 client.18981 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T09:02:00.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.288+0000 7fe1b5d9b640 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e85b77da80 0x55e85b77fe70 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=813f74e9552d5678 server_cookie=f0c515a4f30f79cd in_seq=0 out_seq=0 2024-05-09T09:02:00.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe19dffb640 1 -- 172.21.15.98:0/3275810301 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e85b6d1370 con 0x55e85b77da80 2024-05-09T09:02:00.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe19dffb640 10 client.18981 handle_command_reply: tid=0 2024-05-09T09:02:00.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe1bfba2740 1 client.18981 shutdown 2024-05-09T09:02:00.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3275810301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 msgr2=0x55e85b76a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b76a6c0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8098310 tx=0x7fe1a809c6f0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3275810301 shutdown_connections 2024-05-09T09:02:00.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e85b77da80 0x55e85b77fe70 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b76d730 0x55e85b76b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 0x55e85b76ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/3275810301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b76a6c0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe19dffb640 0 client.18981 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T09:02:00.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3275810301 >> 172.21.15.98:0/3275810301 conn(0x55e85b75a650 msgr2=0x55e85b760480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3275810301 shutdown_connections 2024-05-09T09:02:00.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3275810301 wait complete. 2024-05-09T09:02:00.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.289+0000 7fe1bfba2740 20 client.18981 trim_cache size 0 max 16384 2024-05-09T09:02:00.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.302+0000 7fe1bfba2740 1 Processor -- start 2024-05-09T09:02:00.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.302+0000 7fe1bfba2740 1 -- start start 2024-05-09T09:02:00.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.302+0000 7fe1bfba2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b760600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.302+0000 7fe1bfba2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b75eb50 0x55e85b759630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.302+0000 7fe1bfba2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b76d730 0x55e85b767960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.302+0000 7fe1bfba2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b81acbb0 con 0x55e85b75e230 2024-05-09T09:02:00.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.302+0000 7fe1bfba2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b81ae6f0 con 0x55e85b75eb50 2024-05-09T09:02:00.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.302+0000 7fe1bfba2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b81ad280 con 0x55e85b76d730 2024-05-09T09:02:00.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.303+0000 7fe1b659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b760600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.303+0000 7fe1b659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b760600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37854/0 (socket says 172.21.15.98:37854) 2024-05-09T09:02:00.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.303+0000 7fe1b659c640 1 -- 172.21.15.98:0/1342725393 learned_addr learned my addr 172.21.15.98:0/1342725393 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.303+0000 7fe1b659c640 1 -- 172.21.15.98:0/1342725393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b76d730 msgr2=0x55e85b767960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.303+0000 7fe1b659c640 1 --2- 172.21.15.98:0/1342725393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b76d730 0x55e85b767960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.303+0000 7fe1b659c640 1 -- 172.21.15.98:0/1342725393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b75eb50 msgr2=0x55e85b759630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.303+0000 7fe1b659c640 1 --2- 172.21.15.98:0/1342725393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b75eb50 0x55e85b759630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.303+0000 7fe1b659c640 1 -- 172.21.15.98:0/1342725393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e85b75e980 con 0x55e85b75e230 2024-05-09T09:02:00.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.303+0000 7fe1b659c640 1 --2- 172.21.15.98:0/1342725393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b760600 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800e9c0 tx=0x7fe1a809b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953ecefac814aa30 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.303+0000 7fe19ffff640 1 -- 172.21.15.98:0/1342725393 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a809beb0 con 0x55e85b75e230 2024-05-09T09:02:00.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.303+0000 7fe19ffff640 1 -- 172.21.15.98:0/1342725393 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1a80a0260 con 0x55e85b75e230 2024-05-09T09:02:00.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.303+0000 7fe19ffff640 1 -- 172.21.15.98:0/1342725393 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a809b810 con 0x55e85b75e230 2024-05-09T09:02:00.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 -- 172.21.15.98:0/1342725393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 msgr2=0x55e85b760600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/1342725393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b760600 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800e9c0 tx=0x7fe1a809b9f0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 -- 172.21.15.98:0/1342725393 shutdown_connections 2024-05-09T09:02:00.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/1342725393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b76d730 0x55e85b767960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/1342725393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b75eb50 0x55e85b759630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/1342725393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b75e230 0x55e85b760600 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 -- 172.21.15.98:0/1342725393 >> 172.21.15.98:0/1342725393 conn(0x55e85b75a650 msgr2=0x55e85b758f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 -- 172.21.15.98:0/1342725393 shutdown_connections 2024-05-09T09:02:00.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 -- 172.21.15.98:0/1342725393 wait complete. 2024-05-09T09:02:00.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 -- start start 2024-05-09T09:02:00.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 Processor -- start 2024-05-09T09:02:00.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b75e230 0x55e85b783280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 0x55e85b763920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b76d730 0x55e85b763e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b81ae020 con 0x55e85b76d730 2024-05-09T09:02:00.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b81ad950 con 0x55e85b75e230 2024-05-09T09:02:00.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b81aabb0 con 0x55e85b75eb50 2024-05-09T09:02:00.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1bfba2740 10 client.0 _ll_register_callbacks cb 0x7fe1a809bc80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:00.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1b6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b76d730 0x55e85b763e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1b6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b76d730 0x55e85b763e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37864/0 (socket says 172.21.15.98:37864) 2024-05-09T09:02:00.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.304+0000 7fe1b6d9d640 1 -- 172.21.15.98:0/3572523053 learned_addr learned my addr 172.21.15.98:0/3572523053 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1b6d9d640 1 -- 172.21.15.98:0/3572523053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 msgr2=0x55e85b763920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1b6d9d640 1 --2- 172.21.15.98:0/3572523053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 0x55e85b763920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1b6d9d640 1 -- 172.21.15.98:0/3572523053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b75e230 msgr2=0x55e85b783280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1b659c640 1 --2- 172.21.15.98:0/3572523053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b75e230 0x55e85b783280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1b6d9d640 1 --2- 172.21.15.98:0/3572523053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b75e230 0x55e85b783280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1b6d9d640 1 -- 172.21.15.98:0/3572523053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e85b764690 con 0x55e85b76d730 2024-05-09T09:02:00.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1b659c640 1 --2- 172.21.15.98:0/3572523053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b75e230 0x55e85b783280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:00.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1b6d9d640 1 --2- 172.21.15.98:0/3572523053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b76d730 0x55e85b763e60 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009ca0 tx=0x7fe1b002a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f6e4751248ad95 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe19dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:00.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe19dffb640 1 -- 172.21.15.98:0/3572523053 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b004d070 con 0x55e85b76d730 2024-05-09T09:02:00.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe19dffb640 1 -- 172.21.15.98:0/3572523053 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fe1b0004230 con 0x55e85b76d730 2024-05-09T09:02:00.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3572523053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1a80a0260 con 0x55e85b76d730 2024-05-09T09:02:00.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe19dffb640 1 -- 172.21.15.98:0/3572523053 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe1b0059050 con 0x55e85b76d730 2024-05-09T09:02:00.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1bfba2740 10 client.18987 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:00.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3572523053 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e85b7e8440 con 0x55e85b76d730 2024-05-09T09:02:00.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe19dffb640 1 -- 172.21.15.98:0/3572523053 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1a80a0260 con 0x55e85b76d730 2024-05-09T09:02:00.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe19dffb640 1 -- 172.21.15.98:0/3572523053 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe1b003c9a0 con 0x55e85b76d730 2024-05-09T09:02:00.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1bfba2740 10 client.18987 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:00.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1bfba2740 10 client.18987 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T09:02:00.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1bfba2740 20 client.18987 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:00.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/3572523053 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e85b7e8b60 0x55e85b7eaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1bfba2740 4 client.18987 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:02:00.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.305+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3572523053 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e85b6cb6c0 con 0x55e85b7e8b60 2024-05-09T09:02:00.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe1b659c640 1 --2- 172.21.15.98:0/3572523053 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e85b7e8b60 0x55e85b7eaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe1b659c640 1 --2- 172.21.15.98:0/3572523053 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e85b7e8b60 0x55e85b7eaf80 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8ccdbe6cfbed2e7b server_cookie=967a9a343c5bd693 in_seq=0 out_seq=0 2024-05-09T09:02:00.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe19dffb640 10 client.18987 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T09:02:00.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe19dffb640 1 -- 172.21.15.98:0/3572523053 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e85b6cb6c0 con 0x55e85b7e8b60 2024-05-09T09:02:00.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe19dffb640 10 client.18987 handle_command_reply: tid=0 2024-05-09T09:02:00.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe1bfba2740 1 client.18987 shutdown 2024-05-09T09:02:00.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3572523053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b76d730 msgr2=0x55e85b763e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/3572523053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b76d730 0x55e85b763e60 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009ca0 tx=0x7fe1b002a950 comp rx=0 tx=0).stop 2024-05-09T09:02:00.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3572523053 shutdown_connections 2024-05-09T09:02:00.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/3572523053 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55e85b7e8b60 0x55e85b7eaf80 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/3572523053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e85b76d730 0x55e85b763e60 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/3572523053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e85b75eb50 0x55e85b763920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe1bfba2740 1 --2- 172.21.15.98:0/3572523053 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e85b75e230 0x55e85b783280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe19dffb640 0 client.18987 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T09:02:00.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3572523053 >> 172.21.15.98:0/3572523053 conn(0x55e85b75a650 msgr2=0x55e85b7597b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.306+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3572523053 shutdown_connections 2024-05-09T09:02:00.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.307+0000 7fe1bfba2740 1 -- 172.21.15.98:0/3572523053 wait complete. 2024-05-09T09:02:00.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.307+0000 7fe1bfba2740 20 client.18987 trim_cache size 0 max 16384 2024-05-09T09:02:00.314 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:02:00.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.307+0000 7fe1bce9a640 1 -- 172.21.15.98:0/218868937 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe1a00a2d00 msgr2=0x7fe1a00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.307+0000 7fe1bce9a640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe1a00a2d00 0x7fe1a00a51c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009430 tx=0x7fe1b002f930 comp rx=0 tx=0).stop 2024-05-09T09:02:00.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.307+0000 7fe1bce9a640 1 -- 172.21.15.98:0/218868937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b8069e50 msgr2=0x7fe1b8198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.307+0000 7fe1bce9a640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b8069e50 0x7fe1b8198070 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800e9f0 tx=0x7fe1a800edf0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.307+0000 7fe1bce9a640 1 -- 172.21.15.98:0/218868937 shutdown_connections 2024-05-09T09:02:00.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.307+0000 7fe1bce9a640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe1a00a2d00 0x7fe1a00a51c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.307+0000 7fe1bce9a640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1b8107910 0x7fe1b819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.307+0000 7fe1bce9a640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1b806a7f0 0x7fe1b81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.307+0000 7fe1bce9a640 1 --2- 172.21.15.98:0/218868937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1b8069e50 0x7fe1b8198070 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.307+0000 7fe1bce9a640 1 -- 172.21.15.98:0/218868937 >> 172.21.15.98:0/218868937 conn(0x7fe1b8077710 msgr2=0x7fe1b80ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.307+0000 7fe1bce9a640 1 -- 172.21.15.98:0/218868937 shutdown_connections 2024-05-09T09:02:00.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.307+0000 7fe1bce9a640 1 -- 172.21.15.98:0/218868937 wait complete. 2024-05-09T09:02:00.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T09:02:00.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.421+0000 7f6f3071b640 1 -- 172.21.15.98:0/4248745810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f2806a8b0 msgr2=0x7f6f2810e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.421+0000 7f6f3071b640 1 --2- 172.21.15.98:0/4248745810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f2806a8b0 0x7f6f2810e6e0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009cd0 tx=0x7f6f2002a8e0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.421+0000 7f6f3071b640 1 -- 172.21.15.98:0/4248745810 shutdown_connections 2024-05-09T09:02:00.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.421+0000 7f6f3071b640 1 --2- 172.21.15.98:0/4248745810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f2806a8b0 0x7f6f2810e6e0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.421+0000 7f6f3071b640 1 --2- 172.21.15.98:0/4248745810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6f28069ef0 0x7f6f2806a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.421+0000 7f6f3071b640 1 --2- 172.21.15.98:0/4248745810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f28107820 0x7f6f28107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.421+0000 7f6f3071b640 1 -- 172.21.15.98:0/4248745810 >> 172.21.15.98:0/4248745810 conn(0x7f6f28077790 msgr2=0x7f6f28077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.421+0000 7f6f3071b640 1 -- 172.21.15.98:0/4248745810 shutdown_connections 2024-05-09T09:02:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.421+0000 7f6f3071b640 1 -- 172.21.15.98:0/4248745810 wait complete. 2024-05-09T09:02:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f3071b640 1 Processor -- start 2024-05-09T09:02:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f3071b640 1 -- start start 2024-05-09T09:02:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f3071b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6f28069ef0 0x7f6f28197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f3071b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f2806a8b0 0x7f6f28198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f3071b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f28107820 0x7f6f2819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f3071b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6f28111bf0 con 0x7f6f2806a8b0 2024-05-09T09:02:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f3071b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6f28111a70 con 0x7f6f28107820 2024-05-09T09:02:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f3071b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6f28111d70 con 0x7f6f28069ef0 2024-05-09T09:02:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f2dcb6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f2806a8b0 0x7f6f28198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f2dcb6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f2806a8b0 0x7f6f28198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37878/0 (socket says 172.21.15.98:37878) 2024-05-09T09:02:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f2dcb6640 1 -- 172.21.15.98:0/2959638100 learned_addr learned my addr 172.21.15.98:0/2959638100 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f2ecb8640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f28107820 0x7f6f2819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f2e4b7640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6f28069ef0 0x7f6f28197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f2dcb6640 1 -- 172.21.15.98:0/2959638100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6f28069ef0 msgr2=0x7f6f28197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f2dcb6640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6f28069ef0 0x7f6f28197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f2dcb6640 1 -- 172.21.15.98:0/2959638100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f28107820 msgr2=0x7f6f2819c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f2dcb6640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f28107820 0x7f6f2819c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f2dcb6640 1 -- 172.21.15.98:0/2959638100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2819cf40 con 0x7f6f2806a8b0 2024-05-09T09:02:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.422+0000 7f6f2ecb8640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f28107820 0x7f6f2819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:00.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.423+0000 7f6f2dcb6640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f2806a8b0 0x7f6f28198490 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400c920 tx=0x7f6f2400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.423+0000 7f6f1f7fe640 1 -- 172.21.15.98:0/2959638100 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f24007bb0 con 0x7f6f2806a8b0 2024-05-09T09:02:00.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.423+0000 7f6f1f7fe640 1 -- 172.21.15.98:0/2959638100 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f24007d50 con 0x7f6f2806a8b0 2024-05-09T09:02:00.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.423+0000 7f6f3071b640 1 -- 172.21.15.98:0/2959638100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f2819d230 con 0x7f6f2806a8b0 2024-05-09T09:02:00.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.423+0000 7f6f1f7fe640 1 -- 172.21.15.98:0/2959638100 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f240187d0 con 0x7f6f2806a8b0 2024-05-09T09:02:00.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.423+0000 7f6f3071b640 1 -- 172.21.15.98:0/2959638100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f281a4ac0 con 0x7f6f2806a8b0 2024-05-09T09:02:00.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.425+0000 7f6f1f7fe640 1 -- 172.21.15.98:0/2959638100 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6f2400f450 con 0x7f6f2806a8b0 2024-05-09T09:02:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.425+0000 7f6f3342b740 1 Processor -- start 2024-05-09T09:02:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.425+0000 7f6f3342b740 1 -- start start 2024-05-09T09:02:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.425+0000 7f6f3342b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773330 0x55cf42773710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.425+0000 7f6f3342b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 0x55cf4277dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.425+0000 7f6f3342b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf4277e320 0x55cf42780710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f3342b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cf426d8c40 con 0x55cf4277e320 2024-05-09T09:02:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f3342b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cf42680580 con 0x55cf42773c50 2024-05-09T09:02:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f3342b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cf426d4c00 con 0x55cf42773330 2024-05-09T09:02:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f2dcb6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf4277e320 0x55cf42780710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f2dcb6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf4277e320 0x55cf42780710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37886/0 (socket says 172.21.15.98:37886) 2024-05-09T09:02:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f2dcb6640 1 -- 172.21.15.98:0/2588039077 learned_addr learned my addr 172.21.15.98:0/2588039077 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f2e4b7640 1 --2- 172.21.15.98:0/2588039077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 0x55cf4277dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f2ecb8640 1 --2- 172.21.15.98:0/2588039077 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773330 0x55cf42773710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f1f7fe640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6f140a2d80 0x7f6f140a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f2dcb6640 1 -- 172.21.15.98:0/2588039077 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773330 msgr2=0x55cf42773710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f2dcb6640 1 --2- 172.21.15.98:0/2588039077 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773330 0x55cf42773710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f2dcb6640 1 -- 172.21.15.98:0/2588039077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 msgr2=0x55cf4277dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f2dcb6640 1 --2- 172.21.15.98:0/2588039077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 0x55cf4277dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f2dcb6640 1 -- 172.21.15.98:0/2588039077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf423ec3f0 con 0x55cf4277e320 2024-05-09T09:02:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f2e4b7640 1 --2- 172.21.15.98:0/2588039077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 0x55cf4277dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:02:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f2ecb8640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6f140a2d80 0x7f6f140a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f1f7fe640 1 -- 172.21.15.98:0/2959638100 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f6f240cf940 con 0x7f6f2806a8b0 2024-05-09T09:02:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f2dcb6640 1 --2- 172.21.15.98:0/2588039077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf4277e320 0x55cf42780710 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f6f240ccbf0 tx=0x7f6f24090ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3837e0a2d8558cf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f1cff9640 1 -- 172.21.15.98:0/2588039077 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2409b3b0 con 0x55cf4277e320 2024-05-09T09:02:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f2ecb8640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6f140a2d80 0x7f6f140a5240 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009430 tx=0x7f6f2003e040 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f1cff9640 1 -- 172.21.15.98:0/2588039077 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f24066ae0 con 0x55cf4277e320 2024-05-09T09:02:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.426+0000 7f6f1cff9640 1 -- 172.21.15.98:0/2588039077 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2406a790 con 0x55cf4277e320 2024-05-09T09:02:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.427+0000 7f6f3342b740 1 -- 172.21.15.98:0/2588039077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf4277e320 msgr2=0x55cf42780710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.427+0000 7f6f3342b740 1 --2- 172.21.15.98:0/2588039077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf4277e320 0x55cf42780710 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f6f240ccbf0 tx=0x7f6f24090ee0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.427+0000 7f6f3342b740 1 -- 172.21.15.98:0/2588039077 shutdown_connections 2024-05-09T09:02:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.427+0000 7f6f3342b740 1 --2- 172.21.15.98:0/2588039077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf4277e320 0x55cf42780710 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.427+0000 7f6f3342b740 1 --2- 172.21.15.98:0/2588039077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 0x55cf4277dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.427+0000 7f6f3342b740 1 --2- 172.21.15.98:0/2588039077 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773330 0x55cf42773710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.427+0000 7f6f3342b740 1 -- 172.21.15.98:0/2588039077 >> 172.21.15.98:0/2588039077 conn(0x55cf4276f650 msgr2=0x55cf4276fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.427+0000 7f6f3342b740 1 -- 172.21.15.98:0/2588039077 shutdown_connections 2024-05-09T09:02:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.427+0000 7f6f3342b740 1 -- 172.21.15.98:0/2588039077 wait complete. 2024-05-09T09:02:00.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f3342b740 1 -- start start 2024-05-09T09:02:00.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f3342b740 1 Processor -- start 2024-05-09T09:02:00.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f3342b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773330 0x55cf4277b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f3342b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 0x55cf4277b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f3342b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf4277e320 0x55cf4277bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f3342b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cf427810b0 con 0x55cf4277e320 2024-05-09T09:02:00.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f3342b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cf42693c40 con 0x55cf42773c50 2024-05-09T09:02:00.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f3342b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cf42781230 con 0x55cf42773330 2024-05-09T09:02:00.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f3342b740 10 client.0 _ll_register_callbacks cb 0x55cf423ebbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:00.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f2ecb8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf4277e320 0x55cf4277bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f2ecb8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf4277e320 0x55cf4277bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37894/0 (socket says 172.21.15.98:37894) 2024-05-09T09:02:00.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f2ecb8640 1 -- 172.21.15.98:0/2830947952 learned_addr learned my addr 172.21.15.98:0/2830947952 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f2e4b7640 1 --2- 172.21.15.98:0/2830947952 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773330 0x55cf4277b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f2dcb6640 1 --2- 172.21.15.98:0/2830947952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 0x55cf4277b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f2ecb8640 1 -- 172.21.15.98:0/2830947952 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773330 msgr2=0x55cf4277b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f2ecb8640 1 --2- 172.21.15.98:0/2830947952 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773330 0x55cf4277b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f2ecb8640 1 -- 172.21.15.98:0/2830947952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 msgr2=0x55cf4277b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f2ecb8640 1 --2- 172.21.15.98:0/2830947952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 0x55cf4277b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f2ecb8640 1 -- 172.21.15.98:0/2830947952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf424b67d0 con 0x55cf4277e320 2024-05-09T09:02:00.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f2ecb8640 1 --2- 172.21.15.98:0/2830947952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf4277e320 0x55cf4277bce0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f6f20003bf0 tx=0x7f6f20009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7bc63daa3910a07 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:00.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.428+0000 7f6f11ffb640 1 -- 172.21.15.98:0/2830947952 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f20004240 con 0x55cf4277e320 2024-05-09T09:02:00.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.429+0000 7f6f11ffb640 1 -- 172.21.15.98:0/2830947952 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f6f200043e0 con 0x55cf4277e320 2024-05-09T09:02:00.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.429+0000 7f6f3342b740 1 -- 172.21.15.98:0/2830947952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f24066ae0 con 0x55cf4277e320 2024-05-09T09:02:00.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.429+0000 7f6f11ffb640 1 -- 172.21.15.98:0/2830947952 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6f20031430 con 0x55cf4277e320 2024-05-09T09:02:00.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.429+0000 7f6f3342b740 10 client.18999 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:00.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.429+0000 7f6f3342b740 1 -- 172.21.15.98:0/2830947952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cf4278d2a0 con 0x55cf4277e320 2024-05-09T09:02:00.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.429+0000 7f6f11ffb640 1 -- 172.21.15.98:0/2830947952 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f24066ae0 con 0x55cf4277e320 2024-05-09T09:02:00.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.429+0000 7f6f11ffb640 1 -- 172.21.15.98:0/2830947952 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f6f20030070 con 0x55cf4277e320 2024-05-09T09:02:00.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.429+0000 7f6f3342b740 10 client.18999 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:00.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.429+0000 7f6f3342b740 10 client.18999 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:02:00.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.429+0000 7f6f3342b740 20 client.18999 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:00.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.429+0000 7f6f3342b740 1 --2- 172.21.15.98:0/2830947952 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55cf4278e7b0 0x55cf42790bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.429+0000 7f6f3342b740 4 client.18999 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:02:00.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.429+0000 7f6f3342b740 1 -- 172.21.15.98:0/2830947952 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf426e6370 con 0x55cf4278e7b0 2024-05-09T09:02:00.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.429+0000 7f6f2e4b7640 1 --2- 172.21.15.98:0/2830947952 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55cf4278e7b0 0x55cf42790bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.430+0000 7f6f11ffb640 10 client.18999 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T09:02:00.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.430+0000 7f6f2e4b7640 1 --2- 172.21.15.98:0/2830947952 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55cf4278e7b0 0x55cf42790bc0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4327a170311245db server_cookie=8a7be8b16fe67167 in_seq=0 out_seq=0 2024-05-09T09:02:00.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.430+0000 7f6f11ffb640 1 -- 172.21.15.98:0/2830947952 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf426e6370 con 0x55cf4278e7b0 2024-05-09T09:02:00.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.430+0000 7f6f11ffb640 10 client.18999 handle_command_reply: tid=0 2024-05-09T09:02:00.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.430+0000 7f6f3342b740 1 client.18999 shutdown 2024-05-09T09:02:00.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.431+0000 7f6f3342b740 1 -- 172.21.15.98:0/2830947952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf4277e320 msgr2=0x55cf4277bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.431+0000 7f6f3342b740 1 --2- 172.21.15.98:0/2830947952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf4277e320 0x55cf4277bce0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f6f20003bf0 tx=0x7f6f20009360 comp rx=0 tx=0).stop 2024-05-09T09:02:00.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.431+0000 7f6f3342b740 1 -- 172.21.15.98:0/2830947952 shutdown_connections 2024-05-09T09:02:00.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.431+0000 7f6f3342b740 1 --2- 172.21.15.98:0/2830947952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf4277e320 0x55cf4277bce0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.431+0000 7f6f3342b740 1 --2- 172.21.15.98:0/2830947952 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55cf4278e7b0 0x55cf42790bc0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.431+0000 7f6f3342b740 1 --2- 172.21.15.98:0/2830947952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 0x55cf4277b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.431+0000 7f6f11ffb640 0 client.18999 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T09:02:00.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.431+0000 7f6f3342b740 1 --2- 172.21.15.98:0/2830947952 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773330 0x55cf4277b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.431+0000 7f6f3342b740 1 -- 172.21.15.98:0/2830947952 >> 172.21.15.98:0/2830947952 conn(0x55cf4276f650 msgr2=0x55cf42775450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.431+0000 7f6f3342b740 1 -- 172.21.15.98:0/2830947952 shutdown_connections 2024-05-09T09:02:00.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.431+0000 7f6f3342b740 1 -- 172.21.15.98:0/2830947952 wait complete. 2024-05-09T09:02:00.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.431+0000 7f6f3342b740 20 client.18999 trim_cache size 0 max 16384 2024-05-09T09:02:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.444+0000 7f6f3342b740 1 Processor -- start 2024-05-09T09:02:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.444+0000 7f6f3342b740 1 -- start start 2024-05-09T09:02:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.444+0000 7f6f3342b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf42773330 0x55cf42789fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.444+0000 7f6f3342b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773c50 0x55cf4277a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.444+0000 7f6f3342b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf4277e320 0x55cf42775ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.444+0000 7f6f3342b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6f281aed60 con 0x55cf42773330 2024-05-09T09:02:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.444+0000 7f6f3342b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6f281b08a0 con 0x55cf4277e320 2024-05-09T09:02:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.444+0000 7f6f3342b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6f281af430 con 0x55cf42773c50 2024-05-09T09:02:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.444+0000 7f6f2e4b7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf42773330 0x55cf42789fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.444+0000 7f6f2e4b7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf42773330 0x55cf42789fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37898/0 (socket says 172.21.15.98:37898) 2024-05-09T09:02:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.444+0000 7f6f2e4b7640 1 -- 172.21.15.98:0/2312133189 learned_addr learned my addr 172.21.15.98:0/2312133189 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f2dcb6640 1 --2- 172.21.15.98:0/2312133189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773c50 0x55cf4277a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f2ecb8640 1 --2- 172.21.15.98:0/2312133189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf4277e320 0x55cf42775ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f2e4b7640 1 -- 172.21.15.98:0/2312133189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773c50 msgr2=0x55cf4277a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f2e4b7640 1 --2- 172.21.15.98:0/2312133189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773c50 0x55cf4277a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f2e4b7640 1 -- 172.21.15.98:0/2312133189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf4277e320 msgr2=0x55cf42775ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f2e4b7640 1 --2- 172.21.15.98:0/2312133189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf4277e320 0x55cf42775ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f2e4b7640 1 -- 172.21.15.98:0/2312133189 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf42760d20 con 0x55cf42773330 2024-05-09T09:02:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f2dcb6640 1 --2- 172.21.15.98:0/2312133189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773c50 0x55cf4277a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f2e4b7640 1 --2- 172.21.15.98:0/2312133189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf42773330 0x55cf42789fa0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800a5c0 tx=0x7f6f1800a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d17904df7a906b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f13fff640 1 -- 172.21.15.98:0/2312133189 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18020170 con 0x55cf42773330 2024-05-09T09:02:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f13fff640 1 -- 172.21.15.98:0/2312133189 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f18024020 con 0x55cf42773330 2024-05-09T09:02:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f13fff640 1 -- 172.21.15.98:0/2312133189 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1801dc90 con 0x55cf42773330 2024-05-09T09:02:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f3342b740 1 -- 172.21.15.98:0/2312133189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf42773330 msgr2=0x55cf42789fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f3342b740 1 --2- 172.21.15.98:0/2312133189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf42773330 0x55cf42789fa0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800a5c0 tx=0x7f6f1800a2d0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f3342b740 1 -- 172.21.15.98:0/2312133189 shutdown_connections 2024-05-09T09:02:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f3342b740 1 --2- 172.21.15.98:0/2312133189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf4277e320 0x55cf42775ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f3342b740 1 --2- 172.21.15.98:0/2312133189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf42773c50 0x55cf4277a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f3342b740 1 --2- 172.21.15.98:0/2312133189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf42773330 0x55cf42789fa0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f3342b740 1 -- 172.21.15.98:0/2312133189 >> 172.21.15.98:0/2312133189 conn(0x55cf4276f650 msgr2=0x55cf4276a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f3342b740 1 -- 172.21.15.98:0/2312133189 shutdown_connections 2024-05-09T09:02:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.445+0000 7f6f3342b740 1 -- 172.21.15.98:0/2312133189 wait complete. 2024-05-09T09:02:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f3342b740 1 -- start start 2024-05-09T09:02:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f3342b740 1 Processor -- start 2024-05-09T09:02:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f3342b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf42773330 0x55cf427866a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f3342b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 0x55cf42777ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f3342b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf4277e320 0x55cf42778010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f3342b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6f281b01d0 con 0x55cf42773330 2024-05-09T09:02:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f3342b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6f281afb00 con 0x55cf42773c50 2024-05-09T09:02:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f3342b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6f281acd60 con 0x55cf4277e320 2024-05-09T09:02:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f3342b740 10 client.0 _ll_register_callbacks cb 0x7f6f20009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f2e4b7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf42773330 0x55cf427866a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f2e4b7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf42773330 0x55cf427866a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37908/0 (socket says 172.21.15.98:37908) 2024-05-09T09:02:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f2e4b7640 1 -- 172.21.15.98:0/1500109079 learned_addr learned my addr 172.21.15.98:0/1500109079 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f2ecb8640 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf4277e320 0x55cf42778010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f2dcb6640 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 0x55cf42777ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f2e4b7640 1 -- 172.21.15.98:0/1500109079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf4277e320 msgr2=0x55cf42778010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f2e4b7640 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf4277e320 0x55cf42778010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f2e4b7640 1 -- 172.21.15.98:0/1500109079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 msgr2=0x55cf42777ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f2e4b7640 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 0x55cf42777ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f2e4b7640 1 -- 172.21.15.98:0/1500109079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf42778580 con 0x55cf42773330 2024-05-09T09:02:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f2dcb6640 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 0x55cf42777ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.446+0000 7f6f2ecb8640 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf4277e320 0x55cf42778010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f2e4b7640 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf42773330 0x55cf427866a0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f6f18005940 tx=0x7f6f1801eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d018f01dea18289e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f11ffb640 1 -- 172.21.15.98:0/1500109079 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1801eec0 con 0x55cf42773330 2024-05-09T09:02:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f11ffb640 1 -- 172.21.15.98:0/1500109079 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f6f18005040 con 0x55cf42773330 2024-05-09T09:02:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f3342b740 1 -- 172.21.15.98:0/1500109079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f18024020 con 0x55cf42773330 2024-05-09T09:02:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f11ffb640 1 -- 172.21.15.98:0/1500109079 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6f1803c050 con 0x55cf42773330 2024-05-09T09:02:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f3342b740 10 client.19005 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f3342b740 1 -- 172.21.15.98:0/1500109079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cf427f49f0 con 0x55cf42773330 2024-05-09T09:02:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f11ffb640 1 -- 172.21.15.98:0/1500109079 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f18024020 con 0x55cf42773330 2024-05-09T09:02:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f11ffb640 1 -- 172.21.15.98:0/1500109079 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f6f18020170 con 0x55cf42773330 2024-05-09T09:02:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f3342b740 10 client.19005 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f3342b740 10 client.19005 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:02:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f3342b740 20 client.19005 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f3342b740 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55cf427f50a0 0x55cf427f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f3342b740 4 client.19005 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:02:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.447+0000 7f6f3342b740 1 -- 172.21.15.98:0/1500109079 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf426e06c0 con 0x55cf427f50a0 2024-05-09T09:02:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f2dcb6640 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55cf427f50a0 0x55cf427f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f2dcb6640 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55cf427f50a0 0x55cf427f74c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=283cd283414e8f70 server_cookie=856884190bd14e9d in_seq=0 out_seq=0 2024-05-09T09:02:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f11ffb640 10 client.19005 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T09:02:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f11ffb640 1 -- 172.21.15.98:0/1500109079 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cf426e06c0 con 0x55cf427f50a0 2024-05-09T09:02:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f11ffb640 10 client.19005 handle_command_reply: tid=0 2024-05-09T09:02:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f3342b740 1 client.19005 shutdown 2024-05-09T09:02:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f3342b740 1 -- 172.21.15.98:0/1500109079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf42773330 msgr2=0x55cf427866a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f3342b740 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf42773330 0x55cf427866a0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f6f18005940 tx=0x7f6f1801eb10 comp rx=0 tx=0).stop 2024-05-09T09:02:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f3342b740 1 -- 172.21.15.98:0/1500109079 shutdown_connections 2024-05-09T09:02:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f3342b740 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cf4277e320 0x55cf42778010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f3342b740 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x55cf427f50a0 0x55cf427f74c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f3342b740 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf42773c50 0x55cf42777ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f3342b740 1 --2- 172.21.15.98:0/1500109079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cf42773330 0x55cf427866a0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f11ffb640 0 client.19005 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T09:02:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f3342b740 1 -- 172.21.15.98:0/1500109079 >> 172.21.15.98:0/1500109079 conn(0x55cf4276f650 msgr2=0x55cf4276df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.448+0000 7f6f3342b740 1 -- 172.21.15.98:0/1500109079 shutdown_connections 2024-05-09T09:02:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.449+0000 7f6f3342b740 1 -- 172.21.15.98:0/1500109079 wait complete. 2024-05-09T09:02:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.449+0000 7f6f3342b740 20 client.19005 trim_cache size 0 max 16384 2024-05-09T09:02:00.457 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:02:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.449+0000 7f6f3071b640 1 -- 172.21.15.98:0/2959638100 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6f140a2d80 msgr2=0x7f6f140a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.449+0000 7f6f3071b640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6f140a2d80 0x7f6f140a5240 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009430 tx=0x7f6f2003e040 comp rx=0 tx=0).stop 2024-05-09T09:02:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.449+0000 7f6f3071b640 1 -- 172.21.15.98:0/2959638100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f2806a8b0 msgr2=0x7f6f28198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.449+0000 7f6f3071b640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f2806a8b0 0x7f6f28198490 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400c920 tx=0x7f6f2400cd20 comp rx=0 tx=0).stop 2024-05-09T09:02:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.449+0000 7f6f3071b640 1 -- 172.21.15.98:0/2959638100 shutdown_connections 2024-05-09T09:02:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.449+0000 7f6f3071b640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6f140a2d80 0x7f6f140a5240 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.449+0000 7f6f3071b640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f28107820 0x7f6f2819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.449+0000 7f6f3071b640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f2806a8b0 0x7f6f28198490 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.449+0000 7f6f3071b640 1 --2- 172.21.15.98:0/2959638100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6f28069ef0 0x7f6f28197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.449+0000 7f6f3071b640 1 -- 172.21.15.98:0/2959638100 >> 172.21.15.98:0/2959638100 conn(0x7f6f28077790 msgr2=0x7f6f280fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.449+0000 7f6f3071b640 1 -- 172.21.15.98:0/2959638100 shutdown_connections 2024-05-09T09:02:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.449+0000 7f6f3071b640 1 -- 172.21.15.98:0/2959638100 wait complete. 2024-05-09T09:02:00.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T09:02:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.578+0000 7fda5fd09640 1 -- 172.21.15.98:0/1223338152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda58101420 msgr2=0x7fda5810d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.578+0000 7fda5fd09640 1 --2- 172.21.15.98:0/1223338152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda58101420 0x7fda5810d550 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fda50009ca0 tx=0x7fda5002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.578+0000 7fda5fd09640 1 -- 172.21.15.98:0/1223338152 shutdown_connections 2024-05-09T09:02:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.578+0000 7fda5fd09640 1 --2- 172.21.15.98:0/1223338152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda58101420 0x7fda5810d550 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.578+0000 7fda5fd09640 1 --2- 172.21.15.98:0/1223338152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda58100a80 0x7fda58100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.578+0000 7fda5fd09640 1 --2- 172.21.15.98:0/1223338152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda58106780 0x7fda58106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.578+0000 7fda5fd09640 1 -- 172.21.15.98:0/1223338152 >> 172.21.15.98:0/1223338152 conn(0x7fda58076530 msgr2=0x7fda58076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.578+0000 7fda5fd09640 1 -- 172.21.15.98:0/1223338152 shutdown_connections 2024-05-09T09:02:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.578+0000 7fda5fd09640 1 -- 172.21.15.98:0/1223338152 wait complete. 2024-05-09T09:02:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5fd09640 1 Processor -- start 2024-05-09T09:02:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5fd09640 1 -- start start 2024-05-09T09:02:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5fd09640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda58100a80 0x7fda58198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5fd09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda58101420 0x7fda58198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5fd09640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda58106780 0x7fda5819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5fd09640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fda58110ad0 con 0x7fda58101420 2024-05-09T09:02:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5fd09640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fda58110950 con 0x7fda58100a80 2024-05-09T09:02:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5fd09640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fda58110c50 con 0x7fda58106780 2024-05-09T09:02:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5d2a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda58101420 0x7fda58198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5d2a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda58101420 0x7fda58198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37932/0 (socket says 172.21.15.98:37932) 2024-05-09T09:02:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5daa5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda58100a80 0x7fda58198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5d2a4640 1 -- 172.21.15.98:0/2638442579 learned_addr learned my addr 172.21.15.98:0/2638442579 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5e2a6640 1 --2- 172.21.15.98:0/2638442579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda58106780 0x7fda5819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5d2a4640 1 -- 172.21.15.98:0/2638442579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda58106780 msgr2=0x7fda5819c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5d2a4640 1 --2- 172.21.15.98:0/2638442579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda58106780 0x7fda5819c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5d2a4640 1 -- 172.21.15.98:0/2638442579 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda58100a80 msgr2=0x7fda58198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5d2a4640 1 --2- 172.21.15.98:0/2638442579 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda58100a80 0x7fda58198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5d2a4640 1 -- 172.21.15.98:0/2638442579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5819d090 con 0x7fda58101420 2024-05-09T09:02:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.579+0000 7fda5d2a4640 1 --2- 172.21.15.98:0/2638442579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda58101420 0x7fda58198680 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fda4800d950 tx=0x7fda4800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.580+0000 7fda4effd640 1 -- 172.21.15.98:0/2638442579 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48017070 con 0x7fda58101420 2024-05-09T09:02:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.580+0000 7fda4effd640 1 -- 172.21.15.98:0/2638442579 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda48009c30 con 0x7fda58101420 2024-05-09T09:02:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.580+0000 7fda5fd09640 1 -- 172.21.15.98:0/2638442579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda5819d380 con 0x7fda58101420 2024-05-09T09:02:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.580+0000 7fda4effd640 1 -- 172.21.15.98:0/2638442579 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4800f6b0 con 0x7fda58101420 2024-05-09T09:02:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.580+0000 7fda5fd09640 1 -- 172.21.15.98:0/2638442579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda581a4c10 con 0x7fda58101420 2024-05-09T09:02:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.582+0000 7fda4effd640 1 -- 172.21.15.98:0/2638442579 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fda480109d0 con 0x7fda58101420 2024-05-09T09:02:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.583+0000 7fda62a15740 1 Processor -- start 2024-05-09T09:02:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.583+0000 7fda62a15740 1 -- start start 2024-05-09T09:02:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.583+0000 7fda62a15740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf73710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.583+0000 7fda62a15740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73c50 0x55a10bf7dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.583+0000 7fda62a15740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 0x55a10bf80710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.583+0000 7fda62a15740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a10bed8c40 con 0x55a10bf73330 2024-05-09T09:02:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.583+0000 7fda62a15740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a10be80580 con 0x55a10bf7e320 2024-05-09T09:02:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.583+0000 7fda62a15740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a10bed4c00 con 0x55a10bf73c50 2024-05-09T09:02:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.583+0000 7fda5e2a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf73710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.583+0000 7fda5e2a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf73710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37934/0 (socket says 172.21.15.98:37934) 2024-05-09T09:02:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.583+0000 7fda5e2a6640 1 -- 172.21.15.98:0/3588241758 learned_addr learned my addr 172.21.15.98:0/3588241758 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda5d2a4640 1 --2- 172.21.15.98:0/3588241758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 0x55a10bf80710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda4effd640 1 --2- 172.21.15.98:0/2638442579 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fda440a2d00 0x7fda440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda5daa5640 1 --2- 172.21.15.98:0/3588241758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73c50 0x55a10bf7dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda5e2a6640 1 -- 172.21.15.98:0/3588241758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73c50 msgr2=0x55a10bf7dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda5e2a6640 1 --2- 172.21.15.98:0/3588241758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73c50 0x55a10bf7dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda5e2a6640 1 -- 172.21.15.98:0/3588241758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 msgr2=0x55a10bf80710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda5e2a6640 1 --2- 172.21.15.98:0/3588241758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 0x55a10bf80710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda5e2a6640 1 -- 172.21.15.98:0/3588241758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a10bbec3f0 con 0x55a10bf73330 2024-05-09T09:02:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda5d2a4640 1 --2- 172.21.15.98:0/3588241758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 0x55a10bf80710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda5e2a6640 1 --2- 172.21.15.98:0/2638442579 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fda440a2d00 0x7fda440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda4effd640 1 -- 172.21.15.98:0/2638442579 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fda480cf400 con 0x7fda58101420 2024-05-09T09:02:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda5e2a6640 1 --2- 172.21.15.98:0/3588241758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf73710 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fda50004af0 tx=0x7fda500099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4f620d5a21fc4a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda43fff640 1 -- 172.21.15.98:0/3588241758 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50042040 con 0x55a10bf73330 2024-05-09T09:02:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda43fff640 1 -- 172.21.15.98:0/3588241758 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda50036040 con 0x55a10bf73330 2024-05-09T09:02:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda43fff640 1 -- 172.21.15.98:0/3588241758 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5002f990 con 0x55a10bf73330 2024-05-09T09:02:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.584+0000 7fda5e2a6640 1 --2- 172.21.15.98:0/2638442579 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fda440a2d00 0x7fda440a51c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fda5004d000 tx=0x7fda5004e000 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.585+0000 7fda62a15740 1 -- 172.21.15.98:0/3588241758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 msgr2=0x55a10bf73710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.585+0000 7fda62a15740 1 --2- 172.21.15.98:0/3588241758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf73710 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fda50004af0 tx=0x7fda500099d0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.585+0000 7fda62a15740 1 -- 172.21.15.98:0/3588241758 shutdown_connections 2024-05-09T09:02:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.585+0000 7fda62a15740 1 --2- 172.21.15.98:0/3588241758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 0x55a10bf80710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.585+0000 7fda62a15740 1 --2- 172.21.15.98:0/3588241758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73c50 0x55a10bf7dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.585+0000 7fda62a15740 1 --2- 172.21.15.98:0/3588241758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf73710 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.585+0000 7fda62a15740 1 -- 172.21.15.98:0/3588241758 >> 172.21.15.98:0/3588241758 conn(0x55a10bf6f650 msgr2=0x55a10bf6fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.585+0000 7fda62a15740 1 -- 172.21.15.98:0/3588241758 shutdown_connections 2024-05-09T09:02:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.585+0000 7fda62a15740 1 -- 172.21.15.98:0/3588241758 wait complete. 2024-05-09T09:02:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.585+0000 7fda62a15740 1 -- start start 2024-05-09T09:02:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.585+0000 7fda62a15740 1 Processor -- start 2024-05-09T09:02:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda62a15740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf7b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda62a15740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73c50 0x55a10bf7b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda62a15740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 0x55a10bf7bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda62a15740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a10bf81110 con 0x55a10bf73330 2024-05-09T09:02:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda62a15740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a10be93c40 con 0x55a10bf7e320 2024-05-09T09:02:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda62a15740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a10bf81290 con 0x55a10bf73c50 2024-05-09T09:02:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda62a15740 10 client.0 _ll_register_callbacks cb 0x55a10bbebbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda5daa5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf7b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda5daa5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf7b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37948/0 (socket says 172.21.15.98:37948) 2024-05-09T09:02:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda5daa5640 1 -- 172.21.15.98:0/1336636496 learned_addr learned my addr 172.21.15.98:0/1336636496 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda5d2a4640 1 --2- 172.21.15.98:0/1336636496 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73c50 0x55a10bf7b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda5e2a6640 1 --2- 172.21.15.98:0/1336636496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 0x55a10bf7bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda5daa5640 1 -- 172.21.15.98:0/1336636496 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73c50 msgr2=0x55a10bf7b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda5daa5640 1 --2- 172.21.15.98:0/1336636496 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73c50 0x55a10bf7b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda5daa5640 1 -- 172.21.15.98:0/1336636496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 msgr2=0x55a10bf7bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda5daa5640 1 --2- 172.21.15.98:0/1336636496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 0x55a10bf7bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda5daa5640 1 -- 172.21.15.98:0/1336636496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a10bcb67d0 con 0x55a10bf73330 2024-05-09T09:02:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda5daa5640 1 --2- 172.21.15.98:0/1336636496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf7b210 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fda5400b940 tx=0x7fda5400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c2f7aaad1804b2d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda417fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.586+0000 7fda417fa640 1 -- 172.21.15.98:0/1336636496 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5401b440 con 0x55a10bf73330 2024-05-09T09:02:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.587+0000 7fda417fa640 1 -- 172.21.15.98:0/1336636496 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fda5401b5e0 con 0x55a10bf73330 2024-05-09T09:02:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.587+0000 7fda62a15740 1 -- 172.21.15.98:0/1336636496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda50036040 con 0x55a10bf73330 2024-05-09T09:02:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.587+0000 7fda417fa640 1 -- 172.21.15.98:0/1336636496 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fda54024050 con 0x55a10bf73330 2024-05-09T09:02:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.587+0000 7fda62a15740 10 client.19017 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.587+0000 7fda62a15740 1 -- 172.21.15.98:0/1336636496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a10bf8d370 con 0x55a10bf73330 2024-05-09T09:02:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.587+0000 7fda417fa640 1 -- 172.21.15.98:0/1336636496 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda50036040 con 0x55a10bf73330 2024-05-09T09:02:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.587+0000 7fda417fa640 1 -- 172.21.15.98:0/1336636496 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fda54019020 con 0x55a10bf73330 2024-05-09T09:02:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.587+0000 7fda62a15740 10 client.19017 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.587+0000 7fda62a15740 10 client.19017 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:02:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.587+0000 7fda62a15740 20 client.19017 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.587+0000 7fda62a15740 1 --2- 172.21.15.98:0/1336636496 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a10bf8e830 0x55a10bf90cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.587+0000 7fda62a15740 4 client.19017 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:02:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.587+0000 7fda62a15740 1 -- 172.21.15.98:0/1336636496 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a10bee6370 con 0x55a10bf8e830 2024-05-09T09:02:00.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.587+0000 7fda5d2a4640 1 --2- 172.21.15.98:0/1336636496 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a10bf8e830 0x55a10bf90cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.588+0000 7fda417fa640 10 client.19017 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:00.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.588+0000 7fda5d2a4640 1 --2- 172.21.15.98:0/1336636496 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a10bf8e830 0x55a10bf90cb0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2b3ca60488af7f4 server_cookie=be75367908c4bc2d in_seq=0 out_seq=0 2024-05-09T09:02:00.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.588+0000 7fda417fa640 1 -- 172.21.15.98:0/1336636496 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a10bee6370 con 0x55a10bf8e830 2024-05-09T09:02:00.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.588+0000 7fda417fa640 10 client.19017 handle_command_reply: tid=0 2024-05-09T09:02:00.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.588+0000 7fda62a15740 1 client.19017 shutdown 2024-05-09T09:02:00.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.588+0000 7fda62a15740 1 -- 172.21.15.98:0/1336636496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 msgr2=0x55a10bf7b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.588+0000 7fda62a15740 1 --2- 172.21.15.98:0/1336636496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf7b210 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fda5400b940 tx=0x7fda5400be00 comp rx=0 tx=0).stop 2024-05-09T09:02:00.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.589+0000 7fda62a15740 1 -- 172.21.15.98:0/1336636496 shutdown_connections 2024-05-09T09:02:00.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.589+0000 7fda62a15740 1 --2- 172.21.15.98:0/1336636496 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a10bf8e830 0x55a10bf90cb0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.589+0000 7fda62a15740 1 --2- 172.21.15.98:0/1336636496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 0x55a10bf7bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.589+0000 7fda62a15740 1 --2- 172.21.15.98:0/1336636496 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73c50 0x55a10bf7b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.589+0000 7fda62a15740 1 --2- 172.21.15.98:0/1336636496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf7b210 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.589+0000 7fda417fa640 0 client.19017 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:00.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.589+0000 7fda62a15740 1 -- 172.21.15.98:0/1336636496 >> 172.21.15.98:0/1336636496 conn(0x55a10bf6f650 msgr2=0x55a10bf75350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.589+0000 7fda62a15740 1 -- 172.21.15.98:0/1336636496 shutdown_connections 2024-05-09T09:02:00.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.589+0000 7fda62a15740 1 -- 172.21.15.98:0/1336636496 wait complete. 2024-05-09T09:02:00.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.589+0000 7fda62a15740 20 client.19017 trim_cache size 0 max 16384 2024-05-09T09:02:00.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda62a15740 1 Processor -- start 2024-05-09T09:02:00.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda62a15740 1 -- start start 2024-05-09T09:02:00.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda62a15740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf8a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda62a15740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73c50 0x55a10bf7a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda62a15740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 0x55a10bf7d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda62a15740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fda581aedd0 con 0x55a10bf73330 2024-05-09T09:02:00.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda62a15740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fda581b0910 con 0x55a10bf7e320 2024-05-09T09:02:00.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda62a15740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fda581af4a0 con 0x55a10bf73c50 2024-05-09T09:02:00.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda5daa5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf8a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda5daa5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf8a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37964/0 (socket says 172.21.15.98:37964) 2024-05-09T09:02:00.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda5daa5640 1 -- 172.21.15.98:0/4141385687 learned_addr learned my addr 172.21.15.98:0/4141385687 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda5daa5640 1 -- 172.21.15.98:0/4141385687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73c50 msgr2=0x55a10bf7a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda5daa5640 1 --2- 172.21.15.98:0/4141385687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73c50 0x55a10bf7a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda5daa5640 1 -- 172.21.15.98:0/4141385687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 msgr2=0x55a10bf7d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda5daa5640 1 --2- 172.21.15.98:0/4141385687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 0x55a10bf7d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda5daa5640 1 -- 172.21.15.98:0/4141385687 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a10bf6f480 con 0x55a10bf73330 2024-05-09T09:02:00.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.602+0000 7fda5daa5640 1 --2- 172.21.15.98:0/4141385687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf8a030 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fda54011af0 tx=0x7fda5400bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e505f41440ee1b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.603+0000 7fda437fe640 1 -- 172.21.15.98:0/4141385687 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54003de0 con 0x55a10bf73330 2024-05-09T09:02:00.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.603+0000 7fda437fe640 1 -- 172.21.15.98:0/4141385687 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda5402d070 con 0x55a10bf73330 2024-05-09T09:02:00.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.603+0000 7fda437fe640 1 -- 172.21.15.98:0/4141385687 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54004630 con 0x55a10bf73330 2024-05-09T09:02:00.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.603+0000 7fda62a15740 1 -- 172.21.15.98:0/4141385687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 msgr2=0x55a10bf8a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.603+0000 7fda62a15740 1 --2- 172.21.15.98:0/4141385687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf8a030 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fda54011af0 tx=0x7fda5400bfb0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.603+0000 7fda62a15740 1 -- 172.21.15.98:0/4141385687 shutdown_connections 2024-05-09T09:02:00.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.603+0000 7fda62a15740 1 --2- 172.21.15.98:0/4141385687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf7e320 0x55a10bf7d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.603+0000 7fda62a15740 1 --2- 172.21.15.98:0/4141385687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73c50 0x55a10bf7a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.603+0000 7fda62a15740 1 --2- 172.21.15.98:0/4141385687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf73330 0x55a10bf8a030 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.603+0000 7fda62a15740 1 -- 172.21.15.98:0/4141385687 >> 172.21.15.98:0/4141385687 conn(0x55a10bf6f650 msgr2=0x55a10bf7c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.603+0000 7fda62a15740 1 -- 172.21.15.98:0/4141385687 shutdown_connections 2024-05-09T09:02:00.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.603+0000 7fda62a15740 1 -- 172.21.15.98:0/4141385687 wait complete. 2024-05-09T09:02:00.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.603+0000 7fda62a15740 1 -- start start 2024-05-09T09:02:00.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda62a15740 1 Processor -- start 2024-05-09T09:02:00.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda62a15740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73330 0x55a10bf78080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda62a15740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf73c50 0x55a10bf785c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda62a15740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf7e320 0x55a10bf78b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda62a15740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fda581b0240 con 0x55a10bf7e320 2024-05-09T09:02:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda62a15740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fda581afb70 con 0x55a10bf73c50 2024-05-09T09:02:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda62a15740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fda581acdd0 con 0x55a10bf73330 2024-05-09T09:02:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda62a15740 10 client.0 _ll_register_callbacks cb 0x7fda54004370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda5e2a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf7e320 0x55a10bf78b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda5e2a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf7e320 0x55a10bf78b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37970/0 (socket says 172.21.15.98:37970) 2024-05-09T09:02:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda5e2a6640 1 -- 172.21.15.98:0/102811832 learned_addr learned my addr 172.21.15.98:0/102811832 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda5d2a4640 1 --2- 172.21.15.98:0/102811832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf73c50 0x55a10bf785c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda5e2a6640 1 -- 172.21.15.98:0/102811832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73330 msgr2=0x55a10bf78080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda5e2a6640 1 --2- 172.21.15.98:0/102811832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73330 0x55a10bf78080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda5e2a6640 1 -- 172.21.15.98:0/102811832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf73c50 msgr2=0x55a10bf785c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda5e2a6640 1 --2- 172.21.15.98:0/102811832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf73c50 0x55a10bf785c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda5e2a6640 1 -- 172.21.15.98:0/102811832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a10bff3fc0 con 0x55a10bf7e320 2024-05-09T09:02:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda5e2a6640 1 --2- 172.21.15.98:0/102811832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf7e320 0x55a10bf78b00 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fda50009a60 tx=0x7fda500093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537b800fd43c90db server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.604+0000 7fda417fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda417fa640 1 -- 172.21.15.98:0/102811832 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5005f060 con 0x55a10bf7e320 2024-05-09T09:02:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda417fa640 1 -- 172.21.15.98:0/102811832 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7fda5002f610 con 0x55a10bf7e320 2024-05-09T09:02:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda62a15740 1 -- 172.21.15.98:0/102811832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda5402d070 con 0x55a10bf7e320 2024-05-09T09:02:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda417fa640 1 -- 172.21.15.98:0/102811832 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fda50042040 con 0x55a10bf7e320 2024-05-09T09:02:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda62a15740 10 client.19023 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda62a15740 1 -- 172.21.15.98:0/102811832 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a10bff5280 con 0x55a10bf7e320 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda417fa640 1 -- 172.21.15.98:0/102811832 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda5402d070 con 0x55a10bf7e320 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda417fa640 1 -- 172.21.15.98:0/102811832 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fda500305c0 con 0x55a10bf7e320 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda62a15740 10 client.19023 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda62a15740 10 client.19023 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda62a15740 20 client.19023 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda62a15740 1 --2- 172.21.15.98:0/102811832 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a10bff55b0 0x55a10bff79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda62a15740 4 client.19023 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda62a15740 1 -- 172.21.15.98:0/102811832 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a10bee06c0 con 0x55a10bff55b0 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.605+0000 7fda5daa5640 1 --2- 172.21.15.98:0/102811832 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a10bff55b0 0x55a10bff79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda417fa640 10 client.19023 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda5daa5640 1 --2- 172.21.15.98:0/102811832 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a10bff55b0 0x55a10bff79a0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f21d2ea2eb50ae9a server_cookie=459738027f60c634 in_seq=0 out_seq=0 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda417fa640 1 -- 172.21.15.98:0/102811832 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a10bee06c0 con 0x55a10bff55b0 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda417fa640 10 client.19023 handle_command_reply: tid=0 2024-05-09T09:02:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda62a15740 1 client.19023 shutdown 2024-05-09T09:02:00.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda62a15740 1 -- 172.21.15.98:0/102811832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf7e320 msgr2=0x55a10bf78b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda62a15740 1 --2- 172.21.15.98:0/102811832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf7e320 0x55a10bf78b00 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fda50009a60 tx=0x7fda500093a0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda62a15740 1 -- 172.21.15.98:0/102811832 shutdown_connections 2024-05-09T09:02:00.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda62a15740 1 --2- 172.21.15.98:0/102811832 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55a10bff55b0 0x55a10bff79a0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda62a15740 1 --2- 172.21.15.98:0/102811832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a10bf7e320 0x55a10bf78b00 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda62a15740 1 --2- 172.21.15.98:0/102811832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a10bf73c50 0x55a10bf785c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda417fa640 0 client.19023 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:00.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda62a15740 1 --2- 172.21.15.98:0/102811832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a10bf73330 0x55a10bf78080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda62a15740 1 -- 172.21.15.98:0/102811832 >> 172.21.15.98:0/102811832 conn(0x55a10bf6f650 msgr2=0x55a10bf7ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda62a15740 1 -- 172.21.15.98:0/102811832 shutdown_connections 2024-05-09T09:02:00.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda62a15740 1 -- 172.21.15.98:0/102811832 wait complete. 2024-05-09T09:02:00.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.606+0000 7fda62a15740 20 client.19023 trim_cache size 0 max 16384 2024-05-09T09:02:00.622 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:02:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.607+0000 7fda5fd09640 1 -- 172.21.15.98:0/2638442579 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fda440a2d00 msgr2=0x7fda440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.607+0000 7fda5fd09640 1 --2- 172.21.15.98:0/2638442579 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fda440a2d00 0x7fda440a51c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fda5004d000 tx=0x7fda5004e000 comp rx=0 tx=0).stop 2024-05-09T09:02:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.607+0000 7fda5fd09640 1 -- 172.21.15.98:0/2638442579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda58101420 msgr2=0x7fda58198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.607+0000 7fda5fd09640 1 --2- 172.21.15.98:0/2638442579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda58101420 0x7fda58198680 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fda4800d950 tx=0x7fda4800de10 comp rx=0 tx=0).stop 2024-05-09T09:02:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.607+0000 7fda5fd09640 1 -- 172.21.15.98:0/2638442579 shutdown_connections 2024-05-09T09:02:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.607+0000 7fda5fd09640 1 --2- 172.21.15.98:0/2638442579 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fda440a2d00 0x7fda440a51c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.607+0000 7fda5fd09640 1 --2- 172.21.15.98:0/2638442579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda58106780 0x7fda5819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.607+0000 7fda5fd09640 1 --2- 172.21.15.98:0/2638442579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda58101420 0x7fda58198680 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.607+0000 7fda5fd09640 1 --2- 172.21.15.98:0/2638442579 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda58100a80 0x7fda58198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:00.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.607+0000 7fda5fd09640 1 -- 172.21.15.98:0/2638442579 >> 172.21.15.98:0/2638442579 conn(0x7fda58076530 msgr2=0x7fda58104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:00.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.607+0000 7fda5fd09640 1 -- 172.21.15.98:0/2638442579 shutdown_connections 2024-05-09T09:02:00.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:00.607+0000 7fda5fd09640 1 -- 172.21.15.98:0/2638442579 wait complete. 2024-05-09T09:02:00.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:00 smithi105 ceph-mon[34498]: pgmap v355: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:02:00.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:00 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3520996217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:02:00.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:00 smithi098 ceph-mon[30092]: pgmap v355: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:02:00.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:00 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3520996217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:02:00.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:00 smithi142 ceph-mon[34625]: pgmap v355: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T09:02:00.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:00 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3520996217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:02:01.625 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T09:02:01.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:02:01.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T09:02:01.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.745+0000 7f3c97815640 1 -- 172.21.15.98:0/2297235078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c90100700 msgr2=0x7f3c90100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.745+0000 7f3c97815640 1 --2- 172.21.15.98:0/2297235078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c90100700 0x7f3c90100b80 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009930 tx=0x7f3c8002a420 comp rx=0 tx=0).stop 2024-05-09T09:02:01.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.745+0000 7f3c97815640 1 -- 172.21.15.98:0/2297235078 shutdown_connections 2024-05-09T09:02:01.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.745+0000 7f3c97815640 1 --2- 172.21.15.98:0/2297235078 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c901010c0 0x7f3c9010d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.745+0000 7f3c97815640 1 --2- 172.21.15.98:0/2297235078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c90100700 0x7f3c90100b80 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.745+0000 7f3c97815640 1 --2- 172.21.15.98:0/2297235078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c90106710 0x7f3c90106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.745+0000 7f3c97815640 1 -- 172.21.15.98:0/2297235078 >> 172.21.15.98:0/2297235078 conn(0x7f3c90077830 msgr2=0x7f3c90077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:01.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.745+0000 7f3c97815640 1 -- 172.21.15.98:0/2297235078 shutdown_connections 2024-05-09T09:02:01.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.745+0000 7f3c97815640 1 -- 172.21.15.98:0/2297235078 wait complete. 2024-05-09T09:02:01.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.745+0000 7f3c97815640 1 Processor -- start 2024-05-09T09:02:01.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c97815640 1 -- start start 2024-05-09T09:02:01.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c97815640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c90100700 0x7f3c901990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c97815640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c901010c0 0x7f3c90199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c97815640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c90106710 0x7f3c901a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c97815640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c90110ba0 con 0x7f3c901010c0 2024-05-09T09:02:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c97815640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c90110a20 con 0x7f3c90106710 2024-05-09T09:02:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c97815640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3c90110d20 con 0x7f3c90100700 2024-05-09T09:02:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c94db0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c901010c0 0x7f3c90199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c94db0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c901010c0 0x7f3c90199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37974/0 (socket says 172.21.15.98:37974) 2024-05-09T09:02:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c94db0640 1 -- 172.21.15.98:0/2106005082 learned_addr learned my addr 172.21.15.98:0/2106005082 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c955b1640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c90100700 0x7f3c901990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c95db2640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c90106710 0x7f3c901a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c94db0640 1 -- 172.21.15.98:0/2106005082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c90100700 msgr2=0x7f3c901990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c94db0640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c90100700 0x7f3c901990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c94db0640 1 -- 172.21.15.98:0/2106005082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c90106710 msgr2=0x7f3c901a4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c94db0640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c90106710 0x7f3c901a4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c94db0640 1 -- 172.21.15.98:0/2106005082 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c901a46a0 con 0x7f3c901010c0 2024-05-09T09:02:01.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c95db2640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c90106710 0x7f3c901a4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:01.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.746+0000 7f3c94db0640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c901010c0 0x7f3c90199620 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009a50 tx=0x7f3c80004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:01.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.747+0000 7f3c867fc640 1 -- 172.21.15.98:0/2106005082 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8003f030 con 0x7f3c901010c0 2024-05-09T09:02:01.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.747+0000 7f3c97815640 1 -- 172.21.15.98:0/2106005082 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c901a4870 con 0x7f3c901010c0 2024-05-09T09:02:01.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.747+0000 7f3c867fc640 1 -- 172.21.15.98:0/2106005082 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c80003ad0 con 0x7f3c901010c0 2024-05-09T09:02:01.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.747+0000 7f3c867fc640 1 -- 172.21.15.98:0/2106005082 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80035070 con 0x7f3c901010c0 2024-05-09T09:02:01.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.747+0000 7f3c97815640 1 -- 172.21.15.98:0/2106005082 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c901a4d30 con 0x7f3c901010c0 2024-05-09T09:02:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.749+0000 7f3c867fc640 1 -- 172.21.15.98:0/2106005082 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3c80042030 con 0x7f3c901010c0 2024-05-09T09:02:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.749+0000 7f3c9a600740 1 Processor -- start 2024-05-09T09:02:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.749+0000 7f3c9a600740 1 -- start start 2024-05-09T09:02:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.749+0000 7f3c9a600740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 0x5578c374e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.749+0000 7f3c9a600740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 0x5578c374ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c9a600740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c3759740 0x5578c375bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c9a600740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5578c36b3990 con 0x5578c3759740 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c9a600740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5578c365b5b0 con 0x5578c374e9c0 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c9a600740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5578c36af950 con 0x5578c37500b0 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c94db0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c3759740 0x5578c375bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c94db0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c3759740 0x5578c375bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37976/0 (socket says 172.21.15.98:37976) 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c94db0640 1 -- 172.21.15.98:0/1836849210 learned_addr learned my addr 172.21.15.98:0/1836849210 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c955b1640 1 --2- 172.21.15.98:0/1836849210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 0x5578c374ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c95db2640 1 --2- 172.21.15.98:0/1836849210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 0x5578c374e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c94db0640 1 -- 172.21.15.98:0/1836849210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 msgr2=0x5578c374e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c94db0640 1 --2- 172.21.15.98:0/1836849210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 0x5578c374e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c94db0640 1 -- 172.21.15.98:0/1836849210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 msgr2=0x5578c374ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c94db0640 1 --2- 172.21.15.98:0/1836849210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 0x5578c374ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c94db0640 1 -- 172.21.15.98:0/1836849210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578c348f7e0 con 0x5578c3759740 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c867fc640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3c7c0a2d00 0x7f3c7c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c95db2640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3c7c0a2d00 0x7f3c7c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c94db0640 1 --2- 172.21.15.98:0/1836849210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c3759740 0x5578c375bb30 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f3c800ed2a0 tx=0x7f3c800f8320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb9a69e83eeaa00 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c867fc640 1 -- 172.21.15.98:0/2106005082 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f3c800efff0 con 0x7f3c901010c0 2024-05-09T09:02:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c7b7fe640 1 -- 172.21.15.98:0/1836849210 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80049020 con 0x5578c3759740 2024-05-09T09:02:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c7b7fe640 1 -- 172.21.15.98:0/1836849210 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c800bc800 con 0x5578c3759740 2024-05-09T09:02:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.750+0000 7f3c7b7fe640 1 -- 172.21.15.98:0/1836849210 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c800bbc40 con 0x5578c3759740 2024-05-09T09:02:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.751+0000 7f3c9a600740 1 -- 172.21.15.98:0/1836849210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c3759740 msgr2=0x5578c375bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.751+0000 7f3c9a600740 1 --2- 172.21.15.98:0/1836849210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c3759740 0x5578c375bb30 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f3c800ed2a0 tx=0x7f3c800f8320 comp rx=0 tx=0).stop 2024-05-09T09:02:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.751+0000 7f3c95db2640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3c7c0a2d00 0x7f3c7c0a51c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3c88007f90 tx=0x7f3c88007520 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.751+0000 7f3c9a600740 1 -- 172.21.15.98:0/1836849210 shutdown_connections 2024-05-09T09:02:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.751+0000 7f3c9a600740 1 --2- 172.21.15.98:0/1836849210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c3759740 0x5578c375bb30 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.751+0000 7f3c9a600740 1 --2- 172.21.15.98:0/1836849210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 0x5578c374ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.751+0000 7f3c9a600740 1 --2- 172.21.15.98:0/1836849210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 0x5578c374e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.751+0000 7f3c9a600740 1 -- 172.21.15.98:0/1836849210 >> 172.21.15.98:0/1836849210 conn(0x5578c37170a0 msgr2=0x5578c374af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.751+0000 7f3c9a600740 1 -- 172.21.15.98:0/1836849210 shutdown_connections 2024-05-09T09:02:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.751+0000 7f3c9a600740 1 -- 172.21.15.98:0/1836849210 wait complete. 2024-05-09T09:02:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.752+0000 7f3c9a600740 1 -- start start 2024-05-09T09:02:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.752+0000 7f3c9a600740 1 Processor -- start 2024-05-09T09:02:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.752+0000 7f3c9a600740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 0x5578c3756690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.752+0000 7f3c9a600740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 0x5578c3756bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.752+0000 7f3c9a600740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c3759740 0x5578c3757110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.752+0000 7f3c9a600740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5578c375c390 con 0x5578c3759740 2024-05-09T09:02:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.752+0000 7f3c9a600740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5578c366eb40 con 0x5578c374e9c0 2024-05-09T09:02:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.752+0000 7f3c9a600740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5578c375c510 con 0x5578c37500b0 2024-05-09T09:02:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.752+0000 7f3c9a600740 10 client.0 _ll_register_callbacks cb 0x5578c36630f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.752+0000 7f3c95db2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c3759740 0x5578c3757110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.752+0000 7f3c95db2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c3759740 0x5578c3757110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37990/0 (socket says 172.21.15.98:37990) 2024-05-09T09:02:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.752+0000 7f3c955b1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 0x5578c3756690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.752+0000 7f3c95db2640 1 -- 172.21.15.98:0/1791181120 learned_addr learned my addr 172.21.15.98:0/1791181120 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.752+0000 7f3c94db0640 1 --2- 172.21.15.98:0/1791181120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 0x5578c3756bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c95db2640 1 -- 172.21.15.98:0/1791181120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 msgr2=0x5578c3756bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c95db2640 1 --2- 172.21.15.98:0/1791181120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 0x5578c3756bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c95db2640 1 -- 172.21.15.98:0/1791181120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 msgr2=0x5578c3756690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c95db2640 1 --2- 172.21.15.98:0/1791181120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 0x5578c3756690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c95db2640 1 -- 172.21.15.98:0/1791181120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578c34eff70 con 0x5578c3759740 2024-05-09T09:02:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c955b1640 1 --2- 172.21.15.98:0/1791181120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 0x5578c3756690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:02:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c94db0640 1 --2- 172.21.15.98:0/1791181120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 0x5578c3756bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c95db2640 1 --2- 172.21.15.98:0/1791181120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c3759740 0x5578c3757110 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f3c88010e10 tx=0x7f3c88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c59099841e16c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c78ff9640 1 -- 172.21.15.98:0/1791181120 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c88021070 con 0x5578c3759740 2024-05-09T09:02:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c78ff9640 1 -- 172.21.15.98:0/1791181120 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f3c8801d450 con 0x5578c3759740 2024-05-09T09:02:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c9a600740 1 -- 172.21.15.98:0/1791181120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c800bc800 con 0x5578c3759740 2024-05-09T09:02:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c78ff9640 1 -- 172.21.15.98:0/1791181120 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3c8802d050 con 0x5578c3759740 2024-05-09T09:02:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c9a600740 10 client.19035 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c9a600740 1 -- 172.21.15.98:0/1791181120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5578c3387170 con 0x5578c3759740 2024-05-09T09:02:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c78ff9640 1 -- 172.21.15.98:0/1791181120 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c800bc800 con 0x5578c3759740 2024-05-09T09:02:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.753+0000 7f3c78ff9640 1 -- 172.21.15.98:0/1791181120 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3c88025ae0 con 0x5578c3759740 2024-05-09T09:02:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.754+0000 7f3c9a600740 10 client.19035 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.754+0000 7f3c9a600740 10 client.19035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:02:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.754+0000 7f3c9a600740 20 client.19035 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.754+0000 7f3c9a600740 1 --2- 172.21.15.98:0/1791181120 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5578c3769640 0x5578c376ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.754+0000 7f3c9a600740 4 client.19035 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:02:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.754+0000 7f3c9a600740 1 -- 172.21.15.98:0/1791181120 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578c36c10c0 con 0x5578c3769640 2024-05-09T09:02:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.754+0000 7f3c955b1640 1 --2- 172.21.15.98:0/1791181120 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5578c3769640 0x5578c376ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.754+0000 7f3c955b1640 1 --2- 172.21.15.98:0/1791181120 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5578c3769640 0x5578c376ba30 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d88dc2c2a3a783fd server_cookie=c69e47f8e8989533 in_seq=0 out_seq=0 2024-05-09T09:02:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.754+0000 7f3c78ff9640 10 client.19035 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.755+0000 7f3c78ff9640 1 -- 172.21.15.98:0/1791181120 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5578c36c10c0 con 0x5578c3769640 2024-05-09T09:02:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.755+0000 7f3c78ff9640 10 client.19035 handle_command_reply: tid=0 2024-05-09T09:02:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.755+0000 7f3c9a600740 1 client.19035 shutdown 2024-05-09T09:02:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.755+0000 7f3c9a600740 1 -- 172.21.15.98:0/1791181120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c3759740 msgr2=0x5578c3757110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.755+0000 7f3c9a600740 1 --2- 172.21.15.98:0/1791181120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c3759740 0x5578c3757110 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f3c88010e10 tx=0x7f3c88004a40 comp rx=0 tx=0).stop 2024-05-09T09:02:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.755+0000 7f3c9a600740 1 -- 172.21.15.98:0/1791181120 shutdown_connections 2024-05-09T09:02:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.755+0000 7f3c9a600740 1 --2- 172.21.15.98:0/1791181120 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5578c3769640 0x5578c376ba30 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.755+0000 7f3c9a600740 1 --2- 172.21.15.98:0/1791181120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c3759740 0x5578c3757110 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.755+0000 7f3c9a600740 1 --2- 172.21.15.98:0/1791181120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 0x5578c3756bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.755+0000 7f3c9a600740 1 --2- 172.21.15.98:0/1791181120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 0x5578c3756690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.755+0000 7f3c78ff9640 0 client.19035 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.755+0000 7f3c9a600740 1 -- 172.21.15.98:0/1791181120 >> 172.21.15.98:0/1791181120 conn(0x5578c37170a0 msgr2=0x5578c3750d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.755+0000 7f3c9a600740 1 -- 172.21.15.98:0/1791181120 shutdown_connections 2024-05-09T09:02:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.755+0000 7f3c9a600740 1 -- 172.21.15.98:0/1791181120 wait complete. 2024-05-09T09:02:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.756+0000 7f3c9a600740 20 client.19035 trim_cache size 0 max 16384 2024-05-09T09:02:01.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c9a600740 1 Processor -- start 2024-05-09T09:02:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c9a600740 1 -- start start 2024-05-09T09:02:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c9a600740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c374e9c0 0x5578c3749e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c9a600740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 0x5578c3765250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c9a600740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c3759740 0x5578c3750ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c9a600740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c901aef50 con 0x5578c374e9c0 2024-05-09T09:02:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c9a600740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c901b0a90 con 0x5578c3759740 2024-05-09T09:02:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c9a600740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3c901af620 con 0x5578c37500b0 2024-05-09T09:02:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c955b1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c374e9c0 0x5578c3749e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c955b1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c374e9c0 0x5578c3749e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37996/0 (socket says 172.21.15.98:37996) 2024-05-09T09:02:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c955b1640 1 -- 172.21.15.98:0/43904247 learned_addr learned my addr 172.21.15.98:0/43904247 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c955b1640 1 -- 172.21.15.98:0/43904247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 msgr2=0x5578c3765250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c955b1640 1 --2- 172.21.15.98:0/43904247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 0x5578c3765250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c955b1640 1 -- 172.21.15.98:0/43904247 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c3759740 msgr2=0x5578c3750ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c955b1640 1 --2- 172.21.15.98:0/43904247 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c3759740 0x5578c3750ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.773+0000 7f3c955b1640 1 -- 172.21.15.98:0/43904247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578c3749470 con 0x5578c374e9c0 2024-05-09T09:02:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.774+0000 7f3c955b1640 1 --2- 172.21.15.98:0/43904247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c374e9c0 0x5578c3749e60 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c009fd0 tx=0x7f3c8c0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b9179ea89a4ba5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.774+0000 7f3c7affd640 1 -- 172.21.15.98:0/43904247 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c023070 con 0x5578c374e9c0 2024-05-09T09:02:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.774+0000 7f3c7affd640 1 -- 172.21.15.98:0/43904247 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c8c01f8b0 con 0x5578c374e9c0 2024-05-09T09:02:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.774+0000 7f3c7affd640 1 -- 172.21.15.98:0/43904247 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c015b70 con 0x5578c374e9c0 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.774+0000 7f3c9a600740 1 -- 172.21.15.98:0/43904247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c374e9c0 msgr2=0x5578c3749e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.774+0000 7f3c9a600740 1 --2- 172.21.15.98:0/43904247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c374e9c0 0x5578c3749e60 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c009fd0 tx=0x7f3c8c0047d0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.774+0000 7f3c9a600740 1 -- 172.21.15.98:0/43904247 shutdown_connections 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.774+0000 7f3c9a600740 1 --2- 172.21.15.98:0/43904247 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c3759740 0x5578c3750ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.774+0000 7f3c9a600740 1 --2- 172.21.15.98:0/43904247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c37500b0 0x5578c3765250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.774+0000 7f3c9a600740 1 --2- 172.21.15.98:0/43904247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c374e9c0 0x5578c3749e60 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.774+0000 7f3c9a600740 1 -- 172.21.15.98:0/43904247 >> 172.21.15.98:0/43904247 conn(0x5578c37170a0 msgr2=0x5578c3750490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.774+0000 7f3c9a600740 1 -- 172.21.15.98:0/43904247 shutdown_connections 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.774+0000 7f3c9a600740 1 -- 172.21.15.98:0/43904247 wait complete. 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.775+0000 7f3c9a600740 1 -- start start 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.775+0000 7f3c9a600740 1 Processor -- start 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.775+0000 7f3c9a600740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 0x5578c3761bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.775+0000 7f3c9a600740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c37500b0 0x5578c375e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.775+0000 7f3c9a600740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c3759740 0x5578c375ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.775+0000 7f3c9a600740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c901b03c0 con 0x5578c37500b0 2024-05-09T09:02:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.775+0000 7f3c9a600740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c901afcf0 con 0x5578c374e9c0 2024-05-09T09:02:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.775+0000 7f3c9a600740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3c901acf50 con 0x5578c3759740 2024-05-09T09:02:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.775+0000 7f3c9a600740 10 client.0 _ll_register_callbacks cb 0x7f3c88014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.775+0000 7f3c94db0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c37500b0 0x5578c375e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.775+0000 7f3c94db0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c37500b0 0x5578c375e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38008/0 (socket says 172.21.15.98:38008) 2024-05-09T09:02:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.775+0000 7f3c94db0640 1 -- 172.21.15.98:0/2028427158 learned_addr learned my addr 172.21.15.98:0/2028427158 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.775+0000 7f3c955b1640 1 --2- 172.21.15.98:0/2028427158 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 0x5578c3761bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c94db0640 1 -- 172.21.15.98:0/2028427158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c3759740 msgr2=0x5578c375ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c94db0640 1 --2- 172.21.15.98:0/2028427158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c3759740 0x5578c375ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c94db0640 1 -- 172.21.15.98:0/2028427158 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 msgr2=0x5578c3761bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c94db0640 1 --2- 172.21.15.98:0/2028427158 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 0x5578c3761bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c94db0640 1 -- 172.21.15.98:0/2028427158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578c375f4c0 con 0x5578c37500b0 2024-05-09T09:02:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c955b1640 1 --2- 172.21.15.98:0/2028427158 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 0x5578c3761bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c94db0640 1 --2- 172.21.15.98:0/2028427158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c37500b0 0x5578c375e730 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f3c800bbe00 tx=0x7f3c800314b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7605223ec1a69 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c78ff9640 1 -- 172.21.15.98:0/2028427158 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80049040 con 0x5578c37500b0 2024-05-09T09:02:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c78ff9640 1 -- 172.21.15.98:0/2028427158 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f3c800f3910 con 0x5578c37500b0 2024-05-09T09:02:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c9a600740 1 -- 172.21.15.98:0/2028427158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c8c01f8b0 con 0x5578c37500b0 2024-05-09T09:02:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c78ff9640 1 -- 172.21.15.98:0/2028427158 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3c800bba60 con 0x5578c37500b0 2024-05-09T09:02:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c9a600740 10 client.19041 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c9a600740 1 -- 172.21.15.98:0/2028427158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5578c37d0c50 con 0x5578c37500b0 2024-05-09T09:02:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c78ff9640 1 -- 172.21.15.98:0/2028427158 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c8c01f8b0 con 0x5578c37500b0 2024-05-09T09:02:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c78ff9640 1 -- 172.21.15.98:0/2028427158 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3c800ed5b0 con 0x5578c37500b0 2024-05-09T09:02:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c9a600740 10 client.19041 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c9a600740 10 client.19041 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:02:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c9a600740 20 client.19041 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c9a600740 1 --2- 172.21.15.98:0/2028427158 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5578c37d11b0 0x5578c37d35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c9a600740 4 client.19041 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T09:02:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.776+0000 7f3c9a600740 1 -- 172.21.15.98:0/2028427158 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5578c36bb410 con 0x5578c37d11b0 2024-05-09T09:02:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.777+0000 7f3c955b1640 1 --2- 172.21.15.98:0/2028427158 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5578c37d11b0 0x5578c37d35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.777+0000 7f3c955b1640 1 --2- 172.21.15.98:0/2028427158 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5578c37d11b0 0x5578c37d35d0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5acf5e49dd328323 server_cookie=6b465c9003b627e8 in_seq=0 out_seq=0 2024-05-09T09:02:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.777+0000 7f3c78ff9640 10 client.19041 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.778+0000 7f3c78ff9640 1 -- 172.21.15.98:0/2028427158 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5578c36bb410 con 0x5578c37d11b0 2024-05-09T09:02:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.778+0000 7f3c78ff9640 10 client.19041 handle_command_reply: tid=0 2024-05-09T09:02:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.778+0000 7f3c9a600740 1 client.19041 shutdown 2024-05-09T09:02:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.778+0000 7f3c9a600740 1 -- 172.21.15.98:0/2028427158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c37500b0 msgr2=0x5578c375e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.778+0000 7f3c9a600740 1 --2- 172.21.15.98:0/2028427158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c37500b0 0x5578c375e730 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f3c800bbe00 tx=0x7f3c800314b0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.778+0000 7f3c9a600740 1 -- 172.21.15.98:0/2028427158 shutdown_connections 2024-05-09T09:02:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.778+0000 7f3c9a600740 1 --2- 172.21.15.98:0/2028427158 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5578c37d11b0 0x5578c37d35d0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.778+0000 7f3c9a600740 1 --2- 172.21.15.98:0/2028427158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5578c3759740 0x5578c375ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.778+0000 7f3c78ff9640 0 client.19041 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.778+0000 7f3c9a600740 1 --2- 172.21.15.98:0/2028427158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c37500b0 0x5578c375e730 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.778+0000 7f3c9a600740 1 --2- 172.21.15.98:0/2028427158 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5578c374e9c0 0x5578c3761bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.778+0000 7f3c9a600740 1 -- 172.21.15.98:0/2028427158 >> 172.21.15.98:0/2028427158 conn(0x5578c37170a0 msgr2=0x5578c3756ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.778+0000 7f3c9a600740 1 -- 172.21.15.98:0/2028427158 shutdown_connections 2024-05-09T09:02:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.778+0000 7f3c9a600740 1 -- 172.21.15.98:0/2028427158 wait complete. 2024-05-09T09:02:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.779+0000 7f3c9a600740 20 client.19041 trim_cache size 0 max 16384 2024-05-09T09:02:01.790 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:02:01.790 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-05-09T09:02:01.790 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "8a689514-dbed-47cb-8ca4-8c4499d8c421", 2024-05-09T09:02:01.790 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-05-09T09:02:01.790 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:02:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.779+0000 7f3c97815640 1 -- 172.21.15.98:0/2106005082 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3c7c0a2d00 msgr2=0x7f3c7c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.779+0000 7f3c97815640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3c7c0a2d00 0x7f3c7c0a51c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3c88007f90 tx=0x7f3c88007520 comp rx=0 tx=0).stop 2024-05-09T09:02:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.779+0000 7f3c97815640 1 -- 172.21.15.98:0/2106005082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c901010c0 msgr2=0x7f3c90199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.779+0000 7f3c97815640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c901010c0 0x7f3c90199620 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009a50 tx=0x7f3c80004270 comp rx=0 tx=0).stop 2024-05-09T09:02:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.779+0000 7f3c97815640 1 -- 172.21.15.98:0/2106005082 shutdown_connections 2024-05-09T09:02:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.779+0000 7f3c97815640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f3c7c0a2d00 0x7f3c7c0a51c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.779+0000 7f3c97815640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c90106710 0x7f3c901a4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.779+0000 7f3c97815640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c901010c0 0x7f3c90199620 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.779+0000 7f3c97815640 1 --2- 172.21.15.98:0/2106005082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c90100700 0x7f3c901990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.779+0000 7f3c97815640 1 -- 172.21.15.98:0/2106005082 >> 172.21.15.98:0/2106005082 conn(0x7f3c90077830 msgr2=0x7f3c90103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.779+0000 7f3c97815640 1 -- 172.21.15.98:0/2106005082 shutdown_connections 2024-05-09T09:02:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.779+0000 7f3c97815640 1 -- 172.21.15.98:0/2106005082 wait complete. 2024-05-09T09:02:01.793 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:02:01.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:02:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.898+0000 7f688dd14640 1 -- 172.21.15.98:0/2893220370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6888107810 msgr2=0x7f6888107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.898+0000 7f688dd14640 1 --2- 172.21.15.98:0/2893220370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6888107810 0x7f6888107bf0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f6878009930 tx=0x7f687802a420 comp rx=0 tx=0).stop 2024-05-09T09:02:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.898+0000 7f688dd14640 1 -- 172.21.15.98:0/2893220370 shutdown_connections 2024-05-09T09:02:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.898+0000 7f688dd14640 1 --2- 172.21.15.98:0/2893220370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68881021c0 0x7f688810e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.898+0000 7f688dd14640 1 --2- 172.21.15.98:0/2893220370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6888101800 0x7f6888101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.898+0000 7f688dd14640 1 --2- 172.21.15.98:0/2893220370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6888107810 0x7f6888107bf0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.898+0000 7f688dd14640 1 -- 172.21.15.98:0/2893220370 >> 172.21.15.98:0/2893220370 conn(0x7f6888078c80 msgr2=0x7f6888079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.898+0000 7f688dd14640 1 -- 172.21.15.98:0/2893220370 shutdown_connections 2024-05-09T09:02:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.898+0000 7f688dd14640 1 -- 172.21.15.98:0/2893220370 wait complete. 2024-05-09T09:02:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f688dd14640 1 Processor -- start 2024-05-09T09:02:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f688dd14640 1 -- start start 2024-05-09T09:02:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f688dd14640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6888101800 0x7f6888198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f688dd14640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68881021c0 0x7f68881986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f688dd14640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6888107810 0x7f688819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f688dd14640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6888111c20 con 0x7f6888101800 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f688dd14640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6888111aa0 con 0x7f68881021c0 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f688dd14640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6888111da0 con 0x7f6888107810 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f68877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6888101800 0x7f6888198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f68877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6888101800 0x7f6888198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38026/0 (socket says 172.21.15.98:38026) 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f68877fe640 1 -- 172.21.15.98:0/3266329963 learned_addr learned my addr 172.21.15.98:0/3266329963 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f6886ffd640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68881021c0 0x7f68881986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f6887fff640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6888107810 0x7f688819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f68877fe640 1 -- 172.21.15.98:0/3266329963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6888107810 msgr2=0x7f688819ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f68877fe640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6888107810 0x7f688819ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f68877fe640 1 -- 172.21.15.98:0/3266329963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68881021c0 msgr2=0x7f68881986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f68877fe640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68881021c0 0x7f68881986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f68877fe640 1 -- 172.21.15.98:0/3266329963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f688819d170 con 0x7f6888101800 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.899+0000 7f6886ffd640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68881021c0 0x7f68881986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:02:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.900+0000 7f68877fe640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6888101800 0x7f6888198180 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f6878009230 tx=0x7f6878004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.900+0000 7f6884ff9640 1 -- 172.21.15.98:0/3266329963 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687803f030 con 0x7f6888101800 2024-05-09T09:02:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.900+0000 7f688dd14640 1 -- 172.21.15.98:0/3266329963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f688819d400 con 0x7f6888101800 2024-05-09T09:02:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.900+0000 7f6884ff9640 1 -- 172.21.15.98:0/3266329963 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6878003ad0 con 0x7f6888101800 2024-05-09T09:02:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.900+0000 7f6884ff9640 1 -- 172.21.15.98:0/3266329963 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6878035070 con 0x7f6888101800 2024-05-09T09:02:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.900+0000 7f688dd14640 1 -- 172.21.15.98:0/3266329963 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68881a4cf0 con 0x7f6888101800 2024-05-09T09:02:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.902+0000 7f6884ff9640 1 -- 172.21.15.98:0/3266329963 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6878042030 con 0x7f6888101800 2024-05-09T09:02:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.902+0000 7f6890a25740 1 Processor -- start 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.902+0000 7f6890a25740 1 -- start start 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6884ff9640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f68700a2d00 0x7f68700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6890a25740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1322fb0 0x5574f1323390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6890a25740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1323960 0x5574f1332140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6890a25740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1332680 0x5574f1334a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6890a25740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5574f1288d00 con 0x5574f1323960 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6890a25740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5574f12304c0 con 0x5574f1332680 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6890a25740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5574f1284cc0 con 0x5574f1322fb0 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6887fff640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f68700a2d00 0x7f68700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6886ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1323960 0x5574f1332140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6886ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1323960 0x5574f1332140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38036/0 (socket says 172.21.15.98:38036) 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6884ff9640 1 -- 172.21.15.98:0/3266329963 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f68780efc80 con 0x7f6888101800 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6886ffd640 1 -- 172.21.15.98:0/1270386405 learned_addr learned my addr 172.21.15.98:0/1270386405 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f68877fe640 1 --2- 172.21.15.98:0/1270386405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1322fb0 0x5574f1323390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6886ffd640 1 -- 172.21.15.98:0/1270386405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1322fb0 msgr2=0x5574f1323390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6886ffd640 1 --2- 172.21.15.98:0/1270386405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1322fb0 0x5574f1323390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6886ffd640 1 -- 172.21.15.98:0/1270386405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1332680 msgr2=0x5574f1334a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6886ffd640 1 --2- 172.21.15.98:0/1270386405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1332680 0x5574f1334a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6886ffd640 1 -- 172.21.15.98:0/1270386405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574f10667d0 con 0x5574f1323960 2024-05-09T09:02:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6887fff640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f68700a2d00 0x7f68700a51c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f68800092f0 tx=0x7f68800097f0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f6886ffd640 1 --2- 172.21.15.98:0/1270386405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1323960 0x5574f1332140 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f687c0114f0 tx=0x7f687c0119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ae66ac98f74d99 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f686dffb640 1 -- 172.21.15.98:0/1270386405 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c01e070 con 0x5574f1323960 2024-05-09T09:02:01.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f686dffb640 1 -- 172.21.15.98:0/1270386405 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f687c0035f0 con 0x5574f1323960 2024-05-09T09:02:01.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.903+0000 7f686dffb640 1 -- 172.21.15.98:0/1270386405 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c00f760 con 0x5574f1323960 2024-05-09T09:02:01.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.904+0000 7f6890a25740 1 -- 172.21.15.98:0/1270386405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1323960 msgr2=0x5574f1332140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.904+0000 7f6890a25740 1 --2- 172.21.15.98:0/1270386405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1323960 0x5574f1332140 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f687c0114f0 tx=0x7f687c0119b0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.904+0000 7f6890a25740 1 -- 172.21.15.98:0/1270386405 shutdown_connections 2024-05-09T09:02:01.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.904+0000 7f6890a25740 1 --2- 172.21.15.98:0/1270386405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1332680 0x5574f1334a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.904+0000 7f6890a25740 1 --2- 172.21.15.98:0/1270386405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1323960 0x5574f1332140 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.904+0000 7f6890a25740 1 --2- 172.21.15.98:0/1270386405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1322fb0 0x5574f1323390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.904+0000 7f6890a25740 1 -- 172.21.15.98:0/1270386405 >> 172.21.15.98:0/1270386405 conn(0x5574f131f5a0 msgr2=0x5574f131f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:01.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.904+0000 7f6890a25740 1 -- 172.21.15.98:0/1270386405 shutdown_connections 2024-05-09T09:02:01.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.904+0000 7f6890a25740 1 -- 172.21.15.98:0/1270386405 wait complete. 2024-05-09T09:02:01.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.904+0000 7f6890a25740 1 -- start start 2024-05-09T09:02:01.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f6890a25740 1 Processor -- start 2024-05-09T09:02:01.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f6890a25740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1322fb0 0x5574f132f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f6890a25740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1323960 0x5574f132fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f6890a25740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1332680 0x5574f1330120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f6890a25740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5574f1335440 con 0x5574f1322fb0 2024-05-09T09:02:01.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f6890a25740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5574f1243b90 con 0x5574f1323960 2024-05-09T09:02:01.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f6890a25740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5574f13355c0 con 0x5574f1332680 2024-05-09T09:02:01.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f6890a25740 10 client.0 _ll_register_callbacks cb 0x5574f127fda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:01.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f68877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1322fb0 0x5574f132f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f68877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1322fb0 0x5574f132f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38050/0 (socket says 172.21.15.98:38050) 2024-05-09T09:02:01.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f68877fe640 1 -- 172.21.15.98:0/1230460274 learned_addr learned my addr 172.21.15.98:0/1230460274 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:01.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f6887fff640 1 --2- 172.21.15.98:0/1230460274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1332680 0x5574f1330120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f6886ffd640 1 --2- 172.21.15.98:0/1230460274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1323960 0x5574f132fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f68877fe640 1 -- 172.21.15.98:0/1230460274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1332680 msgr2=0x5574f1330120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f68877fe640 1 --2- 172.21.15.98:0/1230460274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1332680 0x5574f1330120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f68877fe640 1 -- 172.21.15.98:0/1230460274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1323960 msgr2=0x5574f132fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f68877fe640 1 --2- 172.21.15.98:0/1230460274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1323960 0x5574f132fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f68877fe640 1 -- 172.21.15.98:0/1230460274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574f0f9c3f0 con 0x5574f1322fb0 2024-05-09T09:02:01.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f68877fe640 1 --2- 172.21.15.98:0/1230460274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1322fb0 0x5574f132f6a0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f68780b8760 tx=0x7f68780bb990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f354167a5dd2fab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.905+0000 7f6866ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6866ffd640 1 -- 172.21.15.98:0/1230460274 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6878049050 con 0x5574f1322fb0 2024-05-09T09:02:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6866ffd640 1 -- 172.21.15.98:0/1230460274 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f68780dd1c0 con 0x5574f1322fb0 2024-05-09T09:02:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6890a25740 1 -- 172.21.15.98:0/1230460274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f687c0035f0 con 0x5574f1322fb0 2024-05-09T09:02:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6866ffd640 1 -- 172.21.15.98:0/1230460274 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f68780c0140 con 0x5574f1322fb0 2024-05-09T09:02:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6890a25740 10 client.19053 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6890a25740 1 -- 172.21.15.98:0/1230460274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5574f0f5c0f0 con 0x5574f1322fb0 2024-05-09T09:02:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6866ffd640 1 -- 172.21.15.98:0/1230460274 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f687c0035f0 con 0x5574f1322fb0 2024-05-09T09:02:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6866ffd640 1 -- 172.21.15.98:0/1230460274 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f68780e1080 con 0x5574f1322fb0 2024-05-09T09:02:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6890a25740 10 client.19053 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6890a25740 10 client.19053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:02:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6890a25740 20 client.19053 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6890a25740 1 --2- 172.21.15.98:0/1230460274 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5574f1342aa0 0x5574f1344f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6890a25740 4 client.19053 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:02:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6890a25740 1 -- 172.21.15.98:0/1230460274 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574f1296430 con 0x5574f1342aa0 2024-05-09T09:02:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.906+0000 7f6886ffd640 1 --2- 172.21.15.98:0/1230460274 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5574f1342aa0 0x5574f1344f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.907+0000 7f6866ffd640 10 client.19053 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.907+0000 7f6886ffd640 1 --2- 172.21.15.98:0/1230460274 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5574f1342aa0 0x5574f1344f30 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89f2bdf71762a771 server_cookie=229f1b10ecd7eb14 in_seq=0 out_seq=0 2024-05-09T09:02:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.907+0000 7f6866ffd640 1 -- 172.21.15.98:0/1230460274 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5574f1296430 con 0x5574f1342aa0 2024-05-09T09:02:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.907+0000 7f6866ffd640 10 client.19053 handle_command_reply: tid=0 2024-05-09T09:02:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.907+0000 7f6890a25740 1 client.19053 shutdown 2024-05-09T09:02:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.908+0000 7f6890a25740 1 -- 172.21.15.98:0/1230460274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1322fb0 msgr2=0x5574f132f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.908+0000 7f6890a25740 1 --2- 172.21.15.98:0/1230460274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1322fb0 0x5574f132f6a0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f68780b8760 tx=0x7f68780bb990 comp rx=0 tx=0).stop 2024-05-09T09:02:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.908+0000 7f6890a25740 1 -- 172.21.15.98:0/1230460274 shutdown_connections 2024-05-09T09:02:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.908+0000 7f6890a25740 1 --2- 172.21.15.98:0/1230460274 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5574f1342aa0 0x5574f1344f30 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.908+0000 7f6890a25740 1 --2- 172.21.15.98:0/1230460274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1332680 0x5574f1330120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.908+0000 7f6890a25740 1 --2- 172.21.15.98:0/1230460274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1323960 0x5574f132fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.908+0000 7f6866ffd640 0 client.19053 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.908+0000 7f6890a25740 1 --2- 172.21.15.98:0/1230460274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1322fb0 0x5574f132f6a0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.908+0000 7f6890a25740 1 -- 172.21.15.98:0/1230460274 >> 172.21.15.98:0/1230460274 conn(0x5574f131f5a0 msgr2=0x5574f1325410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.908+0000 7f6890a25740 1 -- 172.21.15.98:0/1230460274 shutdown_connections 2024-05-09T09:02:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.908+0000 7f6890a25740 1 -- 172.21.15.98:0/1230460274 wait complete. 2024-05-09T09:02:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.908+0000 7f6890a25740 20 client.19053 trim_cache size 0 max 16384 2024-05-09T09:02:01.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.925+0000 7f6890a25740 1 Processor -- start 2024-05-09T09:02:01.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.925+0000 7f6890a25740 1 -- start start 2024-05-09T09:02:01.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.925+0000 7f6890a25740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1322fb0 0x5574f133e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.925+0000 7f6890a25740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1323960 0x5574f131e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.925+0000 7f6890a25740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1332680 0x5574f134a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.925+0000 7f6890a25740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68881acda0 con 0x5574f1322fb0 2024-05-09T09:02:01.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.925+0000 7f6890a25740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f68881ae8e0 con 0x5574f1332680 2024-05-09T09:02:01.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.925+0000 7f6890a25740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f68881ad470 con 0x5574f1323960 2024-05-09T09:02:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.925+0000 7f68877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1322fb0 0x5574f133e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.926+0000 7f68877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1322fb0 0x5574f133e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38058/0 (socket says 172.21.15.98:38058) 2024-05-09T09:02:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.926+0000 7f68877fe640 1 -- 172.21.15.98:0/60888785 learned_addr learned my addr 172.21.15.98:0/60888785 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.926+0000 7f68877fe640 1 -- 172.21.15.98:0/60888785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1323960 msgr2=0x5574f131e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.926+0000 7f68877fe640 1 --2- 172.21.15.98:0/60888785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1323960 0x5574f131e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.926+0000 7f68877fe640 1 -- 172.21.15.98:0/60888785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1332680 msgr2=0x5574f134a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.926+0000 7f68877fe640 1 --2- 172.21.15.98:0/60888785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1332680 0x5574f134a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.926+0000 7f68877fe640 1 -- 172.21.15.98:0/60888785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574f131f3d0 con 0x5574f1322fb0 2024-05-09T09:02:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.926+0000 7f68877fe640 1 --2- 172.21.15.98:0/60888785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1322fb0 0x5574f133e370 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f68780e9890 tx=0x7f68780bba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=727e8b955a039767 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.926+0000 7f686d7fa640 1 -- 172.21.15.98:0/60888785 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6878049070 con 0x5574f1322fb0 2024-05-09T09:02:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.926+0000 7f686d7fa640 1 -- 172.21.15.98:0/60888785 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68780e9b90 con 0x5574f1322fb0 2024-05-09T09:02:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.926+0000 7f686d7fa640 1 -- 172.21.15.98:0/60888785 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6878003c80 con 0x5574f1322fb0 2024-05-09T09:02:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 -- 172.21.15.98:0/60888785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1322fb0 msgr2=0x5574f133e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 --2- 172.21.15.98:0/60888785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1322fb0 0x5574f133e370 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f68780e9890 tx=0x7f68780bba00 comp rx=0 tx=0).stop 2024-05-09T09:02:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 -- 172.21.15.98:0/60888785 shutdown_connections 2024-05-09T09:02:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 --2- 172.21.15.98:0/60888785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1332680 0x5574f134a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 --2- 172.21.15.98:0/60888785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1323960 0x5574f131e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 --2- 172.21.15.98:0/60888785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1322fb0 0x5574f133e370 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 -- 172.21.15.98:0/60888785 >> 172.21.15.98:0/60888785 conn(0x5574f131f5a0 msgr2=0x5574f131df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 -- 172.21.15.98:0/60888785 shutdown_connections 2024-05-09T09:02:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 -- 172.21.15.98:0/60888785 wait complete. 2024-05-09T09:02:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 -- start start 2024-05-09T09:02:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 Processor -- start 2024-05-09T09:02:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1322fb0 0x5574f13369b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1323960 0x5574f1336ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1332680 0x5574f13a8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68881ae210 con 0x5574f1332680 2024-05-09T09:02:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f68881adb40 con 0x5574f1323960 2024-05-09T09:02:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f68881aada0 con 0x5574f1322fb0 2024-05-09T09:02:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.927+0000 7f6890a25740 10 client.0 _ll_register_callbacks cb 0x7f6878009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6887fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1332680 0x5574f13a8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6887fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1332680 0x5574f13a8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38070/0 (socket says 172.21.15.98:38070) 2024-05-09T09:02:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6887fff640 1 -- 172.21.15.98:0/2545025658 learned_addr learned my addr 172.21.15.98:0/2545025658 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f68877fe640 1 --2- 172.21.15.98:0/2545025658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1322fb0 0x5574f13369b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6887fff640 1 -- 172.21.15.98:0/2545025658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1322fb0 msgr2=0x5574f13369b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6887fff640 1 --2- 172.21.15.98:0/2545025658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1322fb0 0x5574f13369b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6886ffd640 1 --2- 172.21.15.98:0/2545025658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1323960 0x5574f1336ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6887fff640 1 -- 172.21.15.98:0/2545025658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1323960 msgr2=0x5574f1336ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6887fff640 1 --2- 172.21.15.98:0/2545025658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1323960 0x5574f1336ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6887fff640 1 -- 172.21.15.98:0/2545025658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574f13a85a0 con 0x5574f1332680 2024-05-09T09:02:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6887fff640 1 --2- 172.21.15.98:0/2545025658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1332680 0x5574f13a8060 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x5574f1320730 tx=0x7f6880004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bbb1fda8565cbd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6866ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6866ffd640 1 -- 172.21.15.98:0/2545025658 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6880025070 con 0x5574f1332680 2024-05-09T09:02:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6890a25740 1 -- 172.21.15.98:0/2545025658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68780e9b90 con 0x5574f1332680 2024-05-09T09:02:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6866ffd640 1 -- 172.21.15.98:0/2545025658 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f6880004d20 con 0x5574f1332680 2024-05-09T09:02:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6866ffd640 1 -- 172.21.15.98:0/2545025658 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6880030050 con 0x5574f1332680 2024-05-09T09:02:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6890a25740 10 client.19059 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6890a25740 1 -- 172.21.15.98:0/2545025658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5574f13a98e0 con 0x5574f1332680 2024-05-09T09:02:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.928+0000 7f6866ffd640 1 -- 172.21.15.98:0/2545025658 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68780e9b90 con 0x5574f1332680 2024-05-09T09:02:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.929+0000 7f6866ffd640 1 -- 172.21.15.98:0/2545025658 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f688000f040 con 0x5574f1332680 2024-05-09T09:02:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.929+0000 7f6890a25740 10 client.19059 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.929+0000 7f6890a25740 10 client.19059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:02:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.929+0000 7f6890a25740 20 client.19059 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.929+0000 7f6890a25740 1 --2- 172.21.15.98:0/2545025658 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5574f13aa090 0x5574f13ac4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.929+0000 7f6890a25740 4 client.19059 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.929+0000 7f6890a25740 1 -- 172.21.15.98:0/2545025658 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5574f1290780 con 0x5574f13aa090 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.929+0000 7f68877fe640 1 --2- 172.21.15.98:0/2545025658 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5574f13aa090 0x5574f13ac4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.929+0000 7f6866ffd640 10 client.19059 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.929+0000 7f68877fe640 1 --2- 172.21.15.98:0/2545025658 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5574f13aa090 0x5574f13ac4b0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8928bdebfef90274 server_cookie=82b93657905acd6d in_seq=0 out_seq=0 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6866ffd640 1 -- 172.21.15.98:0/2545025658 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5574f1290780 con 0x5574f13aa090 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6866ffd640 10 client.19059 handle_command_reply: tid=0 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6890a25740 1 client.19059 shutdown 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6890a25740 1 -- 172.21.15.98:0/2545025658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1332680 msgr2=0x5574f13a8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6890a25740 1 --2- 172.21.15.98:0/2545025658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1332680 0x5574f13a8060 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x5574f1320730 tx=0x7f6880004a40 comp rx=0 tx=0).stop 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6890a25740 1 -- 172.21.15.98:0/2545025658 shutdown_connections 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6890a25740 1 --2- 172.21.15.98:0/2545025658 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5574f13aa090 0x5574f13ac4b0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6890a25740 1 --2- 172.21.15.98:0/2545025658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5574f1332680 0x5574f13a8060 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6866ffd640 0 client.19059 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6890a25740 1 --2- 172.21.15.98:0/2545025658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574f1323960 0x5574f1336ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6890a25740 1 --2- 172.21.15.98:0/2545025658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574f1322fb0 0x5574f13369b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6890a25740 1 -- 172.21.15.98:0/2545025658 >> 172.21.15.98:0/2545025658 conn(0x5574f131f5a0 msgr2=0x5574f132fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:01.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6890a25740 1 -- 172.21.15.98:0/2545025658 shutdown_connections 2024-05-09T09:02:01.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6890a25740 1 -- 172.21.15.98:0/2545025658 wait complete. 2024-05-09T09:02:01.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f6890a25740 20 client.19059 trim_cache size 0 max 16384 2024-05-09T09:02:01.937 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:02:01.937 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T09:02:01.937 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-05-09T09:02:01.937 INFO:teuthology.orchestra.run.smithi098.stdout: "8a689514-dbed-47cb-8ca4-8c4499d8c421": { 2024-05-09T09:02:01.937 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-05-09T09:02:01.937 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "8a689514-dbed-47cb-8ca4-8c4499d8c421", 2024-05-09T09:02:01.938 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-05-09T09:02:01.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:01.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:01.938 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:02:01.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f688dd14640 1 -- 172.21.15.98:0/3266329963 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f68700a2d00 msgr2=0x7f68700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f688dd14640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f68700a2d00 0x7f68700a51c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f68800092f0 tx=0x7f68800097f0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f688dd14640 1 -- 172.21.15.98:0/3266329963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6888101800 msgr2=0x7f6888198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:01.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f688dd14640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6888101800 0x7f6888198180 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f6878009230 tx=0x7f6878004270 comp rx=0 tx=0).stop 2024-05-09T09:02:01.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f688dd14640 1 -- 172.21.15.98:0/3266329963 shutdown_connections 2024-05-09T09:02:01.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f688dd14640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f68700a2d00 0x7f68700a51c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f688dd14640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6888107810 0x7f688819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f688dd14640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68881021c0 0x7f68881986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f688dd14640 1 --2- 172.21.15.98:0/3266329963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6888101800 0x7f6888198180 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:01.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.930+0000 7f688dd14640 1 -- 172.21.15.98:0/3266329963 >> 172.21.15.98:0/3266329963 conn(0x7f6888078c80 msgr2=0x7f68880ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:01.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.931+0000 7f688dd14640 1 -- 172.21.15.98:0/3266329963 shutdown_connections 2024-05-09T09:02:01.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:01.931+0000 7f688dd14640 1 -- 172.21.15.98:0/3266329963 wait complete. 2024-05-09T09:02:01.941 INFO:tasks.cephfs.filesystem:scrub status for tag:8a689514-dbed-47cb-8ca4-8c4499d8c421 - {'path': '/', 'tag': '8a689514-dbed-47cb-8ca4-8c4499d8c421', 'options': 'recursive,force'} 2024-05-09T09:02:02.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:02 smithi142 ceph-mon[34625]: pgmap v356: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:02:02.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:02 smithi105 ceph-mon[34498]: pgmap v356: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:02:02.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:02 smithi098 ceph-mon[30092]: pgmap v356: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T09:02:03.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:03 smithi105 ceph-mon[34498]: scrub queued for path: / 2024-05-09T09:02:03.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:03 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T09:02:03.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:03 smithi105 ceph-mon[34498]: scrub summary: active paths [/] 2024-05-09T09:02:03.737 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:03 smithi105 ceph-mon[34498]: scrub summary: idle+waiting paths [/] 2024-05-09T09:02:03.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:03 smithi098 ceph-mon[30092]: scrub queued for path: / 2024-05-09T09:02:03.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:03 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T09:02:03.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:03 smithi098 ceph-mon[30092]: scrub summary: active paths [/] 2024-05-09T09:02:03.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:03 smithi098 ceph-mon[30092]: scrub summary: idle+waiting paths [/] 2024-05-09T09:02:03.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:03 smithi142 ceph-mon[34625]: scrub queued for path: / 2024-05-09T09:02:03.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:03 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T09:02:03.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:03 smithi142 ceph-mon[34625]: scrub summary: active paths [/] 2024-05-09T09:02:03.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:03 smithi142 ceph-mon[34625]: scrub summary: idle+waiting paths [/] 2024-05-09T09:02:04.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:04 smithi105 ceph-mon[34498]: pgmap v357: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1023 B/s wr, 10 op/s 2024-05-09T09:02:04.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:04 smithi098 ceph-mon[30092]: pgmap v357: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1023 B/s wr, 10 op/s 2024-05-09T09:02:04.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:04 smithi142 ceph-mon[34625]: pgmap v357: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1023 B/s wr, 10 op/s 2024-05-09T09:02:05.283 INFO:tasks.workunit.client.0.smithi098.stdout:stopping iogen 2024-05-09T09:02:05.283 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 'stopping iogen' 2024-05-09T09:02:05.283 INFO:tasks.workunit.client.0.smithi098.stderr:+ ./iogen -k 2024-05-09T09:02:05.284 INFO:tasks.workunit.client.0.smithi098.stdout:OK 2024-05-09T09:02:05.284 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo OK 2024-05-09T09:02:05.738 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:05 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:02:05.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:05 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:02:05.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:05 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:02:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.096+0000 7f9fe05f9640 1 -- 172.21.15.98:0/1015211306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fd81033e0 msgr2=0x7f9fd810b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.096+0000 7f9fe05f9640 1 --2- 172.21.15.98:0/1015211306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fd81033e0 0x7f9fd810b8c0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009900 tx=0x7f9fc802a510 comp rx=0 tx=0).stop 2024-05-09T09:02:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.096+0000 7f9fe05f9640 1 -- 172.21.15.98:0/1015211306 shutdown_connections 2024-05-09T09:02:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.096+0000 7f9fe05f9640 1 --2- 172.21.15.98:0/1015211306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fd810be00 0x7f9fd810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.096+0000 7f9fe05f9640 1 --2- 172.21.15.98:0/1015211306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fd81033e0 0x7f9fd810b8c0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.096+0000 7f9fe05f9640 1 --2- 172.21.15.98:0/1015211306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fd8102ac0 0x7f9fd8102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.096+0000 7f9fe05f9640 1 -- 172.21.15.98:0/1015211306 >> 172.21.15.98:0/1015211306 conn(0x7f9fd80778b0 msgr2=0x7f9fd8077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.097+0000 7f9fe05f9640 1 -- 172.21.15.98:0/1015211306 shutdown_connections 2024-05-09T09:02:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.097+0000 7f9fe05f9640 1 -- 172.21.15.98:0/1015211306 wait complete. 2024-05-09T09:02:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.097+0000 7f9fe05f9640 1 Processor -- start 2024-05-09T09:02:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.097+0000 7f9fe05f9640 1 -- start start 2024-05-09T09:02:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.097+0000 7f9fe05f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fd8102ac0 0x7f9fd8198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.097+0000 7f9fe05f9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fd81033e0 0x7f9fd81986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.097+0000 7f9fe05f9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fd810be00 0x7f9fd819ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.097+0000 7f9fe05f9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd8111c40 con 0x7f9fd8102ac0 2024-05-09T09:02:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.097+0000 7f9fe05f9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd8111ac0 con 0x7f9fd810be00 2024-05-09T09:02:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.097+0000 7f9fe05f9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd8111dc0 con 0x7f9fd81033e0 2024-05-09T09:02:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.097+0000 7f9fde395640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fd8102ac0 0x7f9fd8198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.097+0000 7f9fde395640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fd8102ac0 0x7f9fd8198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38280/0 (socket says 172.21.15.98:38280) 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.097+0000 7f9fde395640 1 -- 172.21.15.98:0/404035868 learned_addr learned my addr 172.21.15.98:0/404035868 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.098+0000 7f9fdeb96640 1 --2- 172.21.15.98:0/404035868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fd810be00 0x7f9fd819ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.098+0000 7f9fddb94640 1 --2- 172.21.15.98:0/404035868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fd81033e0 0x7f9fd81986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.098+0000 7f9fde395640 1 -- 172.21.15.98:0/404035868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fd81033e0 msgr2=0x7f9fd81986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.098+0000 7f9fde395640 1 --2- 172.21.15.98:0/404035868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fd81033e0 0x7f9fd81986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.098+0000 7f9fde395640 1 -- 172.21.15.98:0/404035868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fd810be00 msgr2=0x7f9fd819ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.098+0000 7f9fde395640 1 --2- 172.21.15.98:0/404035868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fd810be00 0x7f9fd819ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.098+0000 7f9fde395640 1 -- 172.21.15.98:0/404035868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd819d150 con 0x7f9fd8102ac0 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.098+0000 7f9fde395640 1 --2- 172.21.15.98:0/404035868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fd8102ac0 0x7f9fd8198160 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400b4a0 tx=0x7f9fd400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.098+0000 7f9fcf7fe640 1 -- 172.21.15.98:0/404035868 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4014070 con 0x7f9fd8102ac0 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.098+0000 7f9fe05f9640 1 -- 172.21.15.98:0/404035868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd819d440 con 0x7f9fd8102ac0 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.098+0000 7f9fcf7fe640 1 -- 172.21.15.98:0/404035868 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd4010af0 con 0x7f9fd8102ac0 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.098+0000 7f9fcf7fe640 1 -- 172.21.15.98:0/404035868 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4017d10 con 0x7f9fd8102ac0 2024-05-09T09:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.098+0000 7f9fe05f9640 1 -- 172.21.15.98:0/404035868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fd81a4cd0 con 0x7f9fd8102ac0 2024-05-09T09:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.100+0000 7f9fcf7fe640 1 -- 172.21.15.98:0/404035868 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9fd4017070 con 0x7f9fd8102ac0 2024-05-09T09:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.101+0000 7f9fe33e5740 1 Processor -- start 2024-05-09T09:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.101+0000 7f9fe33e5740 1 -- start start 2024-05-09T09:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.101+0000 7f9fe33e5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7dd450 0x55d7cb7dd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.101+0000 7f9fe33e5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.101+0000 7f9fe33e5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7e8630 0x55d7cb7eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fe33e5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d7cb742340 con 0x55d7cb7ddd70 2024-05-09T09:02:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fe33e5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d7cb6ea290 con 0x55d7cb7e8630 2024-05-09T09:02:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fe33e5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d7cb73e300 con 0x55d7cb7dd450 2024-05-09T09:02:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fcf7fe640 1 --2- 172.21.15.98:0/404035868 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9fc40a2da0 0x7f9fc40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fde395640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fddb94640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7e8630 0x55d7cb7eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fde395640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38290/0 (socket says 172.21.15.98:38290) 2024-05-09T09:02:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fde395640 1 -- 172.21.15.98:0/4030883762 learned_addr learned my addr 172.21.15.98:0/4030883762 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fdeb96640 1 --2- 172.21.15.98:0/4030883762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7dd450 0x55d7cb7dd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fdeb96640 1 --2- 172.21.15.98:0/404035868 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9fc40a2da0 0x7f9fc40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fddb94640 1 -- 172.21.15.98:0/4030883762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7dd450 msgr2=0x55d7cb7dd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fddb94640 1 --2- 172.21.15.98:0/4030883762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7dd450 0x55d7cb7dd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fddb94640 1 -- 172.21.15.98:0/4030883762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7ddd70 msgr2=0x55d7cb7e80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fddb94640 1 --2- 172.21.15.98:0/4030883762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fcf7fe640 1 -- 172.21.15.98:0/404035868 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f9fd40ceb20 con 0x7f9fd8102ac0 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fddb94640 1 -- 172.21.15.98:0/4030883762 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7cb70e8e0 con 0x55d7cb7e8630 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fddb94640 1 --2- 172.21.15.98:0/4030883762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7e8630 0x55d7cb7eaa20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009230 tx=0x7f9fc8032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f807dd20b8719f18 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fdeb96640 1 --2- 172.21.15.98:0/404035868 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9fc40a2da0 0x7f9fc40a5260 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0009990 tx=0x7f9fd0009b20 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fccbfe640 1 -- 172.21.15.98:0/4030883762 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc803f070 con 0x55d7cb7e8630 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.102+0000 7f9fccbfe640 1 -- 172.21.15.98:0/4030883762 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fc80317b0 con 0x55d7cb7e8630 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.103+0000 7f9fccbfe640 1 -- 172.21.15.98:0/4030883762 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8035070 con 0x55d7cb7e8630 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.103+0000 7f9fe33e5740 1 -- 172.21.15.98:0/4030883762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7e8630 msgr2=0x55d7cb7eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.103+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/4030883762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7e8630 0x55d7cb7eaa20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009230 tx=0x7f9fc8032ce0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.103+0000 7f9fe33e5740 1 -- 172.21.15.98:0/4030883762 shutdown_connections 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.103+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/4030883762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7e8630 0x55d7cb7eaa20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.103+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/4030883762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.103+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/4030883762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7dd450 0x55d7cb7dd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.103+0000 7f9fe33e5740 1 -- 172.21.15.98:0/4030883762 >> 172.21.15.98:0/4030883762 conn(0x55d7cb7d9960 msgr2=0x55d7cb7d9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.103+0000 7f9fe33e5740 1 -- 172.21.15.98:0/4030883762 shutdown_connections 2024-05-09T09:02:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.103+0000 7f9fe33e5740 1 -- 172.21.15.98:0/4030883762 wait complete. 2024-05-09T09:02:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fe33e5740 1 -- start start 2024-05-09T09:02:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fe33e5740 1 Processor -- start 2024-05-09T09:02:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fe33e5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7dd450 0x55d7cb7e55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fe33e5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fe33e5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7e8630 0x55d7cb7e6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fe33e5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d7cb7eb200 con 0x55d7cb7ddd70 2024-05-09T09:02:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fe33e5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d7cb6fd790 con 0x55d7cb7dd450 2024-05-09T09:02:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fe33e5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d7cb7eb380 con 0x55d7cb7e8630 2024-05-09T09:02:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fe33e5740 10 client.0 _ll_register_callbacks cb 0x55d7cb738a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fddb94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fddb94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38296/0 (socket says 172.21.15.98:38296) 2024-05-09T09:02:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fde395640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7dd450 0x55d7cb7e55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fddb94640 1 -- 172.21.15.98:0/3494068599 learned_addr learned my addr 172.21.15.98:0/3494068599 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fdeb96640 1 --2- 172.21.15.98:0/3494068599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7e8630 0x55d7cb7e6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fddb94640 1 -- 172.21.15.98:0/3494068599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7e8630 msgr2=0x55d7cb7e6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fddb94640 1 --2- 172.21.15.98:0/3494068599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7e8630 0x55d7cb7e6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fddb94640 1 -- 172.21.15.98:0/3494068599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7dd450 msgr2=0x55d7cb7e55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fddb94640 1 --2- 172.21.15.98:0/3494068599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7dd450 0x55d7cb7e55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fddb94640 1 -- 172.21.15.98:0/3494068599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7cb456210 con 0x55d7cb7ddd70 2024-05-09T09:02:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fdeb96640 1 --2- 172.21.15.98:0/3494068599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7e8630 0x55d7cb7e6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.104+0000 7f9fde395640 1 --2- 172.21.15.98:0/3494068599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7dd450 0x55d7cb7e55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:02:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fddb94640 1 --2- 172.21.15.98:0/3494068599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e5b10 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f9fc802aa30 tx=0x7f9fc802ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93ad8fbf80bceee1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fc1ffb640 1 -- 172.21.15.98:0/3494068599 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8035070 con 0x55d7cb7ddd70 2024-05-09T09:02:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fc1ffb640 1 -- 172.21.15.98:0/3494068599 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f9fc8004240 con 0x55d7cb7ddd70 2024-05-09T09:02:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fe33e5740 1 -- 172.21.15.98:0/3494068599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fc80317b0 con 0x55d7cb7ddd70 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fc1ffb640 1 -- 172.21.15.98:0/3494068599 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9fc8053050 con 0x55d7cb7ddd70 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fe33e5740 10 client.19071 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fe33e5740 1 -- 172.21.15.98:0/3494068599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d7cb522620 con 0x55d7cb7ddd70 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fc1ffb640 1 -- 172.21.15.98:0/3494068599 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fc80317b0 con 0x55d7cb7ddd70 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fc1ffb640 1 -- 172.21.15.98:0/3494068599 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f9fc80047c0 con 0x55d7cb7ddd70 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fe33e5740 10 client.19071 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fe33e5740 10 client.19071 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fe33e5740 20 client.19071 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/3494068599 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d7cb7f81b0 0x55d7cb7fa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fe33e5740 4 client.19071 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.105+0000 7f9fe33e5740 1 -- 172.21.15.98:0/3494068599 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7cb74fa70 con 0x55d7cb7f81b0 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.106+0000 7f9fde395640 1 --2- 172.21.15.98:0/3494068599 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d7cb7f81b0 0x55d7cb7fa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.106+0000 7f9fde395640 1 --2- 172.21.15.98:0/3494068599 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d7cb7f81b0 0x55d7cb7fa5a0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=830142cd038e882e server_cookie=e0f8d2605b97a8a6 in_seq=0 out_seq=0 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.106+0000 7f9fc1ffb640 10 client.19071 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.106+0000 7f9fc1ffb640 1 -- 172.21.15.98:0/3494068599 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7cb74fa70 con 0x55d7cb7f81b0 2024-05-09T09:02:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.106+0000 7f9fc1ffb640 10 client.19071 handle_command_reply: tid=0 2024-05-09T09:02:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.106+0000 7f9fe33e5740 1 client.19071 shutdown 2024-05-09T09:02:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.107+0000 7f9fe33e5740 1 -- 172.21.15.98:0/3494068599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7ddd70 msgr2=0x55d7cb7e5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.107+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/3494068599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e5b10 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f9fc802aa30 tx=0x7f9fc802ab90 comp rx=0 tx=0).stop 2024-05-09T09:02:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.107+0000 7f9fe33e5740 1 -- 172.21.15.98:0/3494068599 shutdown_connections 2024-05-09T09:02:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.107+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/3494068599 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d7cb7f81b0 0x55d7cb7fa5a0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.107+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/3494068599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7e8630 0x55d7cb7e6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.107+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/3494068599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e5b10 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.107+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/3494068599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7dd450 0x55d7cb7e55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.107+0000 7f9fc1ffb640 0 client.19071 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.107+0000 7f9fe33e5740 1 -- 172.21.15.98:0/3494068599 >> 172.21.15.98:0/3494068599 conn(0x55d7cb7d9960 msgr2=0x55d7cb7df6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.107+0000 7f9fe33e5740 1 -- 172.21.15.98:0/3494068599 shutdown_connections 2024-05-09T09:02:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.107+0000 7f9fe33e5740 1 -- 172.21.15.98:0/3494068599 wait complete. 2024-05-09T09:02:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.107+0000 7f9fe33e5740 20 client.19071 trim_cache size 0 max 16384 2024-05-09T09:02:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fe33e5740 1 Processor -- start 2024-05-09T09:02:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fe33e5740 1 -- start start 2024-05-09T09:02:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fe33e5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7dd450 0x55d7cb7e7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fe33e5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fe33e5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7e8630 0x55d7cb7e2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fe33e5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd81aef50 con 0x55d7cb7e8630 2024-05-09T09:02:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fe33e5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd81b0a90 con 0x55d7cb7dd450 2024-05-09T09:02:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fe33e5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd81af620 con 0x55d7cb7ddd70 2024-05-09T09:02:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fdeb96640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7e8630 0x55d7cb7e2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fdeb96640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7e8630 0x55d7cb7e2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38298/0 (socket says 172.21.15.98:38298) 2024-05-09T09:02:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fde395640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7dd450 0x55d7cb7e7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fdeb96640 1 -- 172.21.15.98:0/2592624387 learned_addr learned my addr 172.21.15.98:0/2592624387 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fddb94640 1 --2- 172.21.15.98:0/2592624387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fddb94640 1 -- 172.21.15.98:0/2592624387 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7dd450 msgr2=0x55d7cb7e7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fddb94640 1 --2- 172.21.15.98:0/2592624387 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7dd450 0x55d7cb7e7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fddb94640 1 -- 172.21.15.98:0/2592624387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7e8630 msgr2=0x55d7cb7e2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fddb94640 1 --2- 172.21.15.98:0/2592624387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7e8630 0x55d7cb7e2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fddb94640 1 -- 172.21.15.98:0/2592624387 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7cb7d7f00 con 0x55d7cb7ddd70 2024-05-09T09:02:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.122+0000 7f9fdeb96640 1 --2- 172.21.15.98:0/2592624387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7e8630 0x55d7cb7e2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:02:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.123+0000 7f9fddb94640 1 --2- 172.21.15.98:0/2592624387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e2830 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8001eb0 tx=0x7f9fc8009630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feec3e5014310c65 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.123+0000 7f9fc3fff640 1 -- 172.21.15.98:0/2592624387 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8035040 con 0x55d7cb7ddd70 2024-05-09T09:02:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.123+0000 7f9fc3fff640 1 -- 172.21.15.98:0/2592624387 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fc8004d80 con 0x55d7cb7ddd70 2024-05-09T09:02:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.123+0000 7f9fc3fff640 1 -- 172.21.15.98:0/2592624387 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc803f040 con 0x55d7cb7ddd70 2024-05-09T09:02:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.123+0000 7f9fe33e5740 1 -- 172.21.15.98:0/2592624387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7ddd70 msgr2=0x55d7cb7e2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.123+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/2592624387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e2830 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8001eb0 tx=0x7f9fc8009630 comp rx=0 tx=0).stop 2024-05-09T09:02:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.123+0000 7f9fe33e5740 1 -- 172.21.15.98:0/2592624387 shutdown_connections 2024-05-09T09:02:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.123+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/2592624387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7e8630 0x55d7cb7e2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.123+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/2592624387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7e2830 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.123+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/2592624387 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7dd450 0x55d7cb7e7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fe33e5740 1 -- 172.21.15.98:0/2592624387 >> 172.21.15.98:0/2592624387 conn(0x55d7cb7d9960 msgr2=0x55d7cb7d4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fe33e5740 1 -- 172.21.15.98:0/2592624387 shutdown_connections 2024-05-09T09:02:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fe33e5740 1 -- 172.21.15.98:0/2592624387 wait complete. 2024-05-09T09:02:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fe33e5740 1 -- start start 2024-05-09T09:02:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fe33e5740 1 Processor -- start 2024-05-09T09:02:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fe33e5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7dd450 0x55d7cb7fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fe33e5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7fb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fe33e5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7e8630 0x55d7cb7fba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fe33e5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd81b03c0 con 0x55d7cb7e8630 2024-05-09T09:02:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fe33e5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd81afcf0 con 0x55d7cb7ddd70 2024-05-09T09:02:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fe33e5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd81acf50 con 0x55d7cb7dd450 2024-05-09T09:02:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fe33e5740 10 client.0 _ll_register_callbacks cb 0x7f9fd409a820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fddb94640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7fb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fdeb96640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7e8630 0x55d7cb7fba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fddb94640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7fb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:42588/0 (socket says 172.21.15.98:42588) 2024-05-09T09:02:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fddb94640 1 -- 172.21.15.98:0/2153968092 learned_addr learned my addr 172.21.15.98:0/2153968092 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fdeb96640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7e8630 0x55d7cb7fba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38306/0 (socket says 172.21.15.98:38306) 2024-05-09T09:02:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.124+0000 7f9fde395640 1 --2- 172.21.15.98:0/2153968092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7dd450 0x55d7cb7fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fddb94640 1 -- 172.21.15.98:0/2153968092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7dd450 msgr2=0x55d7cb7fafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fddb94640 1 --2- 172.21.15.98:0/2153968092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7dd450 0x55d7cb7fafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fddb94640 1 -- 172.21.15.98:0/2153968092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7e8630 msgr2=0x55d7cb7fba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fddb94640 1 --2- 172.21.15.98:0/2153968092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7e8630 0x55d7cb7fba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fddb94640 1 -- 172.21.15.98:0/2153968092 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7cb85d390 con 0x55d7cb7ddd70 2024-05-09T09:02:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fde395640 1 --2- 172.21.15.98:0/2153968092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7dd450 0x55d7cb7fafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fddb94640 1 --2- 172.21.15.98:0/2153968092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7fb4e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8000c00 tx=0x7f9fc8031510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab7b4399a53bd3f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-09T09:02:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fc1ffb640 1 -- 172.21.15.98:0/2153968092 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8035070 con 0x55d7cb7ddd70 2024-05-09T09:02:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fe33e5740 1 -- 172.21.15.98:0/2153968092 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fc8004d80 con 0x55d7cb7ddd70 2024-05-09T09:02:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fc1ffb640 1 -- 172.21.15.98:0/2153968092 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9562+0+0 (secure 0 0 0) 0x7f9fc80046c0 con 0x55d7cb7ddd70 2024-05-09T09:02:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fc1ffb640 1 -- 172.21.15.98:0/2153968092 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9fc8068050 con 0x55d7cb7ddd70 2024-05-09T09:02:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fe33e5740 10 client.27683 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fe33e5740 1 -- 172.21.15.98:0/2153968092 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d7cb85e5e0 con 0x55d7cb7ddd70 2024-05-09T09:02:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fc1ffb640 1 -- 172.21.15.98:0/2153968092 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fc8004d80 con 0x55d7cb7ddd70 2024-05-09T09:02:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fc1ffb640 1 -- 172.21.15.98:0/2153968092 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f9fc8058440 con 0x55d7cb7ddd70 2024-05-09T09:02:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fe33e5740 10 client.27683 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fe33e5740 10 client.27683 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.h 2024-05-09T09:02:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fe33e5740 20 client.27683 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.125+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/2153968092 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d7cb85ebc0 0x55d7cb860fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.126+0000 7f9fe33e5740 4 client.27683 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:02:06.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.126+0000 7f9fe33e5740 1 -- 172.21.15.98:0/2153968092 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d7cb749dc0 con 0x55d7cb85ebc0 2024-05-09T09:02:06.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.126+0000 7f9fde395640 1 --2- 172.21.15.98:0/2153968092 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d7cb85ebc0 0x55d7cb860fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:06.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.126+0000 7f9fc1ffb640 10 client.27683 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:06.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.126+0000 7f9fde395640 1 --2- 172.21.15.98:0/2153968092 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d7cb85ebc0 0x55d7cb860fe0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7740a76c8e862b5 server_cookie=d27b5f71f988f4 in_seq=0 out_seq=0 2024-05-09T09:02:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fc1ffb640 1 -- 172.21.15.98:0/2153968092 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d7cb749dc0 con 0x55d7cb85ebc0 2024-05-09T09:02:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fc1ffb640 10 client.27683 handle_command_reply: tid=0 2024-05-09T09:02:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fe33e5740 1 client.27683 shutdown 2024-05-09T09:02:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fe33e5740 1 -- 172.21.15.98:0/2153968092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7ddd70 msgr2=0x55d7cb7fb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/2153968092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7fb4e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8000c00 tx=0x7f9fc8031510 comp rx=0 tx=0).stop 2024-05-09T09:02:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fe33e5740 1 -- 172.21.15.98:0/2153968092 shutdown_connections 2024-05-09T09:02:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/2153968092 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x55d7cb85ebc0 0x55d7cb860fe0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/2153968092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d7cb7e8630 0x55d7cb7fba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fc1ffb640 0 client.27683 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:06.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/2153968092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7cb7ddd70 0x55d7cb7fb4e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fe33e5740 1 --2- 172.21.15.98:0/2153968092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d7cb7dd450 0x55d7cb7fafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fe33e5740 1 -- 172.21.15.98:0/2153968092 >> 172.21.15.98:0/2153968092 conn(0x55d7cb7d9960 msgr2=0x55d7cb7d8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:06.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fe33e5740 1 -- 172.21.15.98:0/2153968092 shutdown_connections 2024-05-09T09:02:06.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fe33e5740 1 -- 172.21.15.98:0/2153968092 wait complete. 2024-05-09T09:02:06.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.132+0000 7f9fe33e5740 20 client.27683 trim_cache size 0 max 16384 2024-05-09T09:02:06.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-05-09T09:02:06.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-05-09T09:02:06.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-05-09T09:02:06.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-05-09T09:02:06.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.133+0000 7f9fe05f9640 1 -- 172.21.15.98:0/404035868 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9fc40a2da0 msgr2=0x7f9fc40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:06.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.133+0000 7f9fe05f9640 1 --2- 172.21.15.98:0/404035868 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9fc40a2da0 0x7f9fc40a5260 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0009990 tx=0x7f9fd0009b20 comp rx=0 tx=0).stop 2024-05-09T09:02:06.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.133+0000 7f9fe05f9640 1 -- 172.21.15.98:0/404035868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fd8102ac0 msgr2=0x7f9fd8198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.133+0000 7f9fe05f9640 1 --2- 172.21.15.98:0/404035868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fd8102ac0 0x7f9fd8198160 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400b4a0 tx=0x7f9fd400b960 comp rx=0 tx=0).stop 2024-05-09T09:02:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.133+0000 7f9fe05f9640 1 -- 172.21.15.98:0/404035868 shutdown_connections 2024-05-09T09:02:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.133+0000 7f9fe05f9640 1 --2- 172.21.15.98:0/404035868 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f9fc40a2da0 0x7f9fc40a5260 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.133+0000 7f9fe05f9640 1 --2- 172.21.15.98:0/404035868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fd810be00 0x7f9fd819ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.133+0000 7f9fe05f9640 1 --2- 172.21.15.98:0/404035868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fd81033e0 0x7f9fd81986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.133+0000 7f9fe05f9640 1 --2- 172.21.15.98:0/404035868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fd8102ac0 0x7f9fd8198160 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.133+0000 7f9fe05f9640 1 -- 172.21.15.98:0/404035868 >> 172.21.15.98:0/404035868 conn(0x7f9fd80778b0 msgr2=0x7f9fd80fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.133+0000 7f9fe05f9640 1 -- 172.21.15.98:0/404035868 shutdown_connections 2024-05-09T09:02:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-05-09T09:02:06.133+0000 7f9fe05f9640 1 -- 172.21.15.98:0/404035868 wait complete. 2024-05-09T09:02:06.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:06 smithi105 ceph-mon[34498]: pgmap v358: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-05-09T09:02:06.986 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:06 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:06.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:06 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:06.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:06 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:06.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:06 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:06.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:06 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:02:06.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:06 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:02:06.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:06 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:07.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:06 smithi098 ceph-mon[30092]: pgmap v358: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-05-09T09:02:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:06 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:06 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:06 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:06 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:06 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:02:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:06 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:02:07.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:06 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:07.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:06 smithi142 ceph-mon[34625]: pgmap v358: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-05-09T09:02:07.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:06 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:06 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:06 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:06 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:06 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:02:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:06 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:02:07.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:06 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:02:07.303 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-09T09:02:07.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-09T09:02:07.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:07 smithi105 ceph-mon[34498]: scrub summary: idle 2024-05-09T09:02:07.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:07 smithi105 ceph-mon[34498]: osdmap e108: 12 total, 12 up, 12 in 2024-05-09T09:02:07.987 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:07 smithi105 ceph-mon[34498]: pgmap v360: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 56 KiB/s wr, 34 op/s 2024-05-09T09:02:08.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:07 smithi098 ceph-mon[30092]: scrub summary: idle 2024-05-09T09:02:08.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:07 smithi098 ceph-mon[30092]: osdmap e108: 12 total, 12 up, 12 in 2024-05-09T09:02:08.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:07 smithi098 ceph-mon[30092]: pgmap v360: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 56 KiB/s wr, 34 op/s 2024-05-09T09:02:08.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:07 smithi142 ceph-mon[34625]: scrub summary: idle 2024-05-09T09:02:08.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:07 smithi142 ceph-mon[34625]: osdmap e108: 12 total, 12 up, 12 in 2024-05-09T09:02:08.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:07 smithi142 ceph-mon[34625]: pgmap v360: 129 pgs: 129 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 56 KiB/s wr, 34 op/s 2024-05-09T09:02:08.443 INFO:tasks.workunit:Stopping ['suites/iogen.sh'] on client.0... 2024-05-09T09:02:08.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-09T09:02:08.918 DEBUG:teuthology.parallel:result is None 2024-05-09T09:02:08.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-09T09:02:08.987 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-09T09:02:08.988 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-05-09T09:02:09.000 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi098.front.sepia.ceph.com 2024-05-09T09:02:09.012 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-05-09T09:02:09.024 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-05-09T09:02:09.024 INFO:teuthology.task.exec:Executing custom commands... 2024-05-09T09:02:09.024 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-05-09T09:02:09.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-05-09T09:02:09.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-05-09T09:02:09.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-09T09:02:09.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.385+0000 7fd681fdc640 1 -- 172.21.15.98:0/2516767509 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd67c100790 msgr2=0x7fd67c100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:09.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.385+0000 7fd681fdc640 1 --2- 172.21.15.98:0/2516767509 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd67c100790 0x7fd67c100c10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009930 tx=0x7fd66c02a480 comp rx=0 tx=0).stop 2024-05-09T09:02:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.389+0000 7fd681fdc640 1 -- 172.21.15.98:0/2516767509 shutdown_connections 2024-05-09T09:02:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.389+0000 7fd681fdc640 1 --2- 172.21.15.98:0/2516767509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd67c101150 0x7fd67c10d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.389+0000 7fd681fdc640 1 --2- 172.21.15.98:0/2516767509 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd67c100790 0x7fd67c100c10 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.389+0000 7fd681fdc640 1 --2- 172.21.15.98:0/2516767509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd67c106750 0x7fd67c106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.389+0000 7fd681fdc640 1 -- 172.21.15.98:0/2516767509 >> 172.21.15.98:0/2516767509 conn(0x7fd67c077880 msgr2=0x7fd67c077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.389+0000 7fd681fdc640 1 -- 172.21.15.98:0/2516767509 shutdown_connections 2024-05-09T09:02:09.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.390+0000 7fd681fdc640 1 -- 172.21.15.98:0/2516767509 wait complete. 2024-05-09T09:02:09.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.391+0000 7fd681fdc640 1 Processor -- start 2024-05-09T09:02:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.392+0000 7fd681fdc640 1 -- start start 2024-05-09T09:02:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.392+0000 7fd681fdc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd67c100790 0x7fd67c072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.392+0000 7fd681fdc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd67c101150 0x7fd67c073250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.392+0000 7fd681fdc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd67c106750 0x7fd67c06bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.392+0000 7fd681fdc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c10faa0 con 0x7fd67c106750 2024-05-09T09:02:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.392+0000 7fd681fdc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c10f920 con 0x7fd67c100790 2024-05-09T09:02:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.392+0000 7fd681fdc640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c10fc20 con 0x7fd67c101150 2024-05-09T09:02:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.393+0000 7fd67bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd67c106750 0x7fd67c06bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.393+0000 7fd67bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd67c106750 0x7fd67c06bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38320/0 (socket says 172.21.15.98:38320) 2024-05-09T09:02:09.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.393+0000 7fd67bfff640 1 -- 172.21.15.98:0/3088620580 learned_addr learned my addr 172.21.15.98:0/3088620580 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.393+0000 7fd67bfff640 1 -- 172.21.15.98:0/3088620580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd67c101150 msgr2=0x7fd67c073250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.393+0000 7fd67bfff640 1 --2- 172.21.15.98:0/3088620580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd67c101150 0x7fd67c073250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.394+0000 7fd67bfff640 1 -- 172.21.15.98:0/3088620580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd67c100790 msgr2=0x7fd67c072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.394+0000 7fd67bfff640 1 --2- 172.21.15.98:0/3088620580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd67c100790 0x7fd67c072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.394+0000 7fd67bfff640 1 -- 172.21.15.98:0/3088620580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67c06c5b0 con 0x7fd67c106750 2024-05-09T09:02:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.394+0000 7fd67bfff640 1 --2- 172.21.15.98:0/3088620580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd67c106750 0x7fd67c06bde0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fd67400d750 tx=0x7fd67400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.395+0000 7fd678ff9640 1 -- 172.21.15.98:0/3088620580 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd674017070 con 0x7fd67c106750 2024-05-09T09:02:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.395+0000 7fd678ff9640 1 -- 172.21.15.98:0/3088620580 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd674005500 con 0x7fd67c106750 2024-05-09T09:02:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.395+0000 7fd678ff9640 1 -- 172.21.15.98:0/3088620580 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd674004d90 con 0x7fd67c106750 2024-05-09T09:02:09.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.397+0000 7fd681fdc640 1 -- 172.21.15.98:0/3088620580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd67c06c8a0 con 0x7fd67c106750 2024-05-09T09:02:09.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.397+0000 7fd681fdc640 1 -- 172.21.15.98:0/3088620580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd67c070830 con 0x7fd67c106750 2024-05-09T09:02:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.399+0000 7fd681fdc640 1 -- 172.21.15.98:0/3088620580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd660005290 con 0x7fd67c106750 2024-05-09T09:02:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.403+0000 7fd678ff9640 1 -- 172.21.15.98:0/3088620580 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd674009bd0 con 0x7fd67c106750 2024-05-09T09:02:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.404+0000 7fd678ff9640 1 --2- 172.21.15.98:0/3088620580 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd65c0a2fd0 0x7fd65c0a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.404+0000 7fd678ff9640 1 -- 172.21.15.98:0/3088620580 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9650+0+0 (secure 0 0 0) 0x7fd6740cfa60 con 0x7fd67c106750 2024-05-09T09:02:09.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.406+0000 7fd67b7fe640 1 --2- 172.21.15.98:0/3088620580 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd65c0a2fd0 0x7fd65c0a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:09.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.407+0000 7fd67b7fe640 1 --2- 172.21.15.98:0/3088620580 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd65c0a2fd0 0x7fd65c0a5490 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd66400ab90 tx=0x7fd664009250 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.410+0000 7fd678ff9640 1 -- 172.21.15.98:0/3088620580 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd674020020 con 0x7fd67c106750 2024-05-09T09:02:09.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.809+0000 7fd681fdc640 1 -- 172.21.15.98:0/3088620580 --> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fd660000bc0 con 0x7fd65c0a2fd0 2024-05-09T09:02:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.836+0000 7fd678ff9640 1 -- 172.21.15.98:0/3088620580 <== mgr.15555 v2:172.21.15.98:6800/4136638623 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fd660000bc0 con 0x7fd65c0a2fd0 2024-05-09T09:02:09.836 INFO:teuthology.orchestra.run.smithi098.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-09T00:00:00", "created": "2024-05-09T08:50:46", "first": "2024-05-09T08:51:00", "last": "2024-05-09T09:02:00", "last_pruned": "2024-05-09T09:02:00", "created_count": 12, "pruned_count": 6, "active": true} 2024-05-09T09:02:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.840+0000 7fd681fdc640 1 -- 172.21.15.98:0/3088620580 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd65c0a2fd0 msgr2=0x7fd65c0a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.840+0000 7fd681fdc640 1 --2- 172.21.15.98:0/3088620580 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd65c0a2fd0 0x7fd65c0a5490 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd66400ab90 tx=0x7fd664009250 comp rx=0 tx=0).stop 2024-05-09T09:02:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.840+0000 7fd681fdc640 1 -- 172.21.15.98:0/3088620580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd67c106750 msgr2=0x7fd67c06bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.840+0000 7fd681fdc640 1 --2- 172.21.15.98:0/3088620580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd67c106750 0x7fd67c06bde0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fd67400d750 tx=0x7fd67400dc10 comp rx=0 tx=0).stop 2024-05-09T09:02:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.840+0000 7fd681fdc640 1 -- 172.21.15.98:0/3088620580 shutdown_connections 2024-05-09T09:02:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.840+0000 7fd681fdc640 1 --2- 172.21.15.98:0/3088620580 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd65c0a2fd0 0x7fd65c0a5490 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.840+0000 7fd681fdc640 1 --2- 172.21.15.98:0/3088620580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd67c106750 0x7fd67c06bde0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.840+0000 7fd681fdc640 1 --2- 172.21.15.98:0/3088620580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd67c101150 0x7fd67c073250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.840+0000 7fd681fdc640 1 --2- 172.21.15.98:0/3088620580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd67c100790 0x7fd67c072d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:09.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.840+0000 7fd681fdc640 1 -- 172.21.15.98:0/3088620580 >> 172.21.15.98:0/3088620580 conn(0x7fd67c077880 msgr2=0x7fd67c1030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:09.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.840+0000 7fd681fdc640 1 -- 172.21.15.98:0/3088620580 shutdown_connections 2024-05-09T09:02:09.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:09.840+0000 7fd681fdc640 1 -- 172.21.15.98:0/3088620580 wait complete. 2024-05-09T09:02:09.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-09T09:02:10.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.034+0000 7ff644a11640 1 -- 172.21.15.98:0/1112287485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6380a4dc0 msgr2=0x7ff6380a51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:10.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.034+0000 7ff644a11640 1 --2- 172.21.15.98:0/1112287485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6380a4dc0 0x7ff6380a51a0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7ff63001a910 tx=0x7ff63003b500 comp rx=0 tx=0).stop 2024-05-09T09:02:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.035+0000 7ff644a11640 1 -- 172.21.15.98:0/1112287485 shutdown_connections 2024-05-09T09:02:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.035+0000 7ff644a11640 1 --2- 172.21.15.98:0/1112287485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6380b7a50 0x7ff6380b9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.035+0000 7ff644a11640 1 --2- 172.21.15.98:0/1112287485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6380a56e0 0x7ff6380b7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.035+0000 7ff644a11640 1 --2- 172.21.15.98:0/1112287485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6380a4dc0 0x7ff6380a51a0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.035+0000 7ff644a11640 1 -- 172.21.15.98:0/1112287485 >> 172.21.15.98:0/1112287485 conn(0x7ff63801a620 msgr2=0x7ff63801aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.035+0000 7ff644a11640 1 -- 172.21.15.98:0/1112287485 shutdown_connections 2024-05-09T09:02:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.035+0000 7ff644a11640 1 -- 172.21.15.98:0/1112287485 wait complete. 2024-05-09T09:02:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.036+0000 7ff644a11640 1 Processor -- start 2024-05-09T09:02:10.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.037+0000 7ff644a11640 1 -- start start 2024-05-09T09:02:10.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.037+0000 7ff644a11640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6380a4dc0 0x7ff6380b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:10.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.037+0000 7ff644a11640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6380a56e0 0x7ff6380b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:10.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.037+0000 7ff644a11640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6380b7a50 0x7ff6380b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:10.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.037+0000 7ff644a11640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6380af680 con 0x7ff6380a56e0 2024-05-09T09:02:10.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.037+0000 7ff644a11640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff6380af500 con 0x7ff6380b7a50 2024-05-09T09:02:10.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.037+0000 7ff644a11640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff6380af800 con 0x7ff6380a4dc0 2024-05-09T09:02:10.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.037+0000 7ff63e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6380a56e0 0x7ff6380b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:10.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.037+0000 7ff63e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6380a56e0 0x7ff6380b31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38332/0 (socket says 172.21.15.98:38332) 2024-05-09T09:02:10.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.037+0000 7ff63e59c640 1 -- 172.21.15.98:0/4221346286 learned_addr learned my addr 172.21.15.98:0/4221346286 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:10.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.038+0000 7ff63f59e640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6380b7a50 0x7ff6380b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:10.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.038+0000 7ff63ed9d640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6380a4dc0 0x7ff6380b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:10.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.038+0000 7ff63e59c640 1 -- 172.21.15.98:0/4221346286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6380a4dc0 msgr2=0x7ff6380b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:10.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.038+0000 7ff63e59c640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6380a4dc0 0x7ff6380b7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:10.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.038+0000 7ff63e59c640 1 -- 172.21.15.98:0/4221346286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6380b7a50 msgr2=0x7ff6380b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:10.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.038+0000 7ff63e59c640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6380b7a50 0x7ff6380b36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:10.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.038+0000 7ff63e59c640 1 -- 172.21.15.98:0/4221346286 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6380b3cb0 con 0x7ff6380a56e0 2024-05-09T09:02:10.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.038+0000 7ff63e59c640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6380a56e0 0x7ff6380b31b0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7ff63400d980 tx=0x7ff63400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:10.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.039+0000 7ff627fff640 1 -- 172.21.15.98:0/4221346286 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff634017070 con 0x7ff6380a56e0 2024-05-09T09:02:10.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.039+0000 7ff644a11640 1 -- 172.21.15.98:0/4221346286 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6380b3fa0 con 0x7ff6380a56e0 2024-05-09T09:02:10.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.039+0000 7ff63ed9d640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6380a4dc0 0x7ff6380b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:10.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.039+0000 7ff644a11640 1 -- 172.21.15.98:0/4221346286 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff638147e20 con 0x7ff6380a56e0 2024-05-09T09:02:10.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.039+0000 7ff627fff640 1 -- 172.21.15.98:0/4221346286 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff634009c30 con 0x7ff6380a56e0 2024-05-09T09:02:10.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.039+0000 7ff627fff640 1 -- 172.21.15.98:0/4221346286 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63400f620 con 0x7ff6380a56e0 2024-05-09T09:02:10.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.041+0000 7ff644a11640 1 -- 172.21.15.98:0/4221346286 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff62c0052c0 con 0x7ff6380a56e0 2024-05-09T09:02:10.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.042+0000 7ff627fff640 1 -- 172.21.15.98:0/4221346286 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff634010460 con 0x7ff6380a56e0 2024-05-09T09:02:10.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.043+0000 7ff627fff640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff6280a2eb0 0x7ff6280a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:10.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.044+0000 7ff627fff640 1 -- 172.21.15.98:0/4221346286 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9650+0+0 (secure 0 0 0) 0x7ff6340cf3b0 con 0x7ff6380a56e0 2024-05-09T09:02:10.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.044+0000 7ff63ed9d640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff6280a2eb0 0x7ff6280a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:10.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.044+0000 7ff63ed9d640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff6280a2eb0 0x7ff6280a5370 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff630000c00 tx=0x7ff630005a20 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:10.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.052+0000 7ff627fff640 1 -- 172.21.15.98:0/4221346286 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff63409bb40 con 0x7ff6380a56e0 2024-05-09T09:02:10.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:09 smithi105 ceph-mon[34498]: pgmap v361: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 56 KiB/s wr, 34 op/s 2024-05-09T09:02:10.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:09 smithi098 ceph-mon[30092]: pgmap v361: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 56 KiB/s wr, 34 op/s 2024-05-09T09:02:10.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:09 smithi142 ceph-mon[34625]: pgmap v361: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 20 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 56 KiB/s wr, 34 op/s 2024-05-09T09:02:10.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.447+0000 7ff644a11640 1 -- 172.21.15.98:0/4221346286 --> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7ff62c000bf0 con 0x7ff6280a2eb0 2024-05-09T09:02:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.481+0000 7ff627fff640 1 -- 172.21.15.98:0/4221346286 <== mgr.15555 v2:172.21.15.98:6800/4136638623 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7ff62c000bf0 con 0x7ff6280a2eb0 2024-05-09T09:02:10.480 INFO:teuthology.orchestra.run.smithi098.stdout:/ 1m 6m3h 2024-05-09T09:02:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.485+0000 7ff644a11640 1 -- 172.21.15.98:0/4221346286 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff6280a2eb0 msgr2=0x7ff6280a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.485+0000 7ff644a11640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff6280a2eb0 0x7ff6280a5370 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff630000c00 tx=0x7ff630005a20 comp rx=0 tx=0).stop 2024-05-09T09:02:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.485+0000 7ff644a11640 1 -- 172.21.15.98:0/4221346286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6380a56e0 msgr2=0x7ff6380b31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.485+0000 7ff644a11640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6380a56e0 0x7ff6380b31b0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7ff63400d980 tx=0x7ff63400de40 comp rx=0 tx=0).stop 2024-05-09T09:02:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.485+0000 7ff644a11640 1 -- 172.21.15.98:0/4221346286 shutdown_connections 2024-05-09T09:02:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.485+0000 7ff644a11640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff6280a2eb0 0x7ff6280a5370 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.485+0000 7ff644a11640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6380b7a50 0x7ff6380b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.485+0000 7ff644a11640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6380a56e0 0x7ff6380b31b0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.485+0000 7ff644a11640 1 --2- 172.21.15.98:0/4221346286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6380a4dc0 0x7ff6380b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:10.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.485+0000 7ff644a11640 1 -- 172.21.15.98:0/4221346286 >> 172.21.15.98:0/4221346286 conn(0x7ff63801a620 msgr2=0x7ff6380b7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:10.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.487+0000 7ff644a11640 1 -- 172.21.15.98:0/4221346286 shutdown_connections 2024-05-09T09:02:10.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:10.487+0000 7ff644a11640 1 -- 172.21.15.98:0/4221346286 wait complete. 2024-05-09T09:02:10.505 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-05-09T09:02:10.516 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-05-09T09:02:11.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:10 smithi105 ceph-mon[34498]: from='client.19080 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:02:11.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:10 smithi105 ceph-mon[34498]: from='client.19086 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:02:11.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:10 smithi098 ceph-mon[30092]: from='client.19080 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:02:11.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:10 smithi098 ceph-mon[30092]: from='client.19086 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:02:11.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:10 smithi142 ceph-mon[34625]: from='client.19080 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:02:11.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:10 smithi142 ceph-mon[34625]: from='client.19086 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:02:12.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:11 smithi105 ceph-mon[34498]: pgmap v362: 129 pgs: 5 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 112 active+clean; 14 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-05-09T09:02:12.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:11 smithi098 ceph-mon[30092]: pgmap v362: 129 pgs: 5 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 112 active+clean; 14 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-05-09T09:02:12.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:11 smithi142 ceph-mon[34625]: pgmap v362: 129 pgs: 5 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 112 active+clean; 14 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-05-09T09:02:14.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:13 smithi105 ceph-mon[34498]: pgmap v363: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 80 KiB/s wr, 73 op/s 2024-05-09T09:02:14.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:13 smithi098 ceph-mon[30092]: pgmap v363: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 80 KiB/s wr, 73 op/s 2024-05-09T09:02:14.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:13 smithi142 ceph-mon[34625]: pgmap v363: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 80 KiB/s wr, 73 op/s 2024-05-09T09:02:16.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:15 smithi105 ceph-mon[34498]: pgmap v364: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 11 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 57 KiB/s wr, 144 op/s 2024-05-09T09:02:16.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:15 smithi098 ceph-mon[30092]: pgmap v364: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 11 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 57 KiB/s wr, 144 op/s 2024-05-09T09:02:16.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:15 smithi142 ceph-mon[34625]: pgmap v364: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 11 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 57 KiB/s wr, 144 op/s 2024-05-09T09:02:18.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:17 smithi105 ceph-mon[34498]: pgmap v365: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 27 KiB/s wr, 255 op/s 2024-05-09T09:02:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:17 smithi098 ceph-mon[30092]: pgmap v365: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 27 KiB/s wr, 255 op/s 2024-05-09T09:02:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:17 smithi142 ceph-mon[34625]: pgmap v365: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 27 KiB/s wr, 255 op/s 2024-05-09T09:02:19.236 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:18 smithi105 ceph-mon[34498]: osdmap e109: 12 total, 12 up, 12 in 2024-05-09T09:02:19.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:18 smithi098 ceph-mon[30092]: osdmap e109: 12 total, 12 up, 12 in 2024-05-09T09:02:19.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:18 smithi142 ceph-mon[34625]: osdmap e109: 12 total, 12 up, 12 in 2024-05-09T09:02:20.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:19 smithi098 ceph-mon[30092]: pgmap v367: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 27 KiB/s wr, 276 op/s 2024-05-09T09:02:20.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:19 smithi142 ceph-mon[34625]: pgmap v367: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 27 KiB/s wr, 276 op/s 2024-05-09T09:02:20.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:19 smithi105 ceph-mon[34498]: pgmap v367: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 27 KiB/s wr, 276 op/s 2024-05-09T09:02:22.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:21 smithi098 ceph-mon[30092]: pgmap v368: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.6 KiB/s wr, 261 op/s 2024-05-09T09:02:22.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:21 smithi142 ceph-mon[34625]: pgmap v368: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.6 KiB/s wr, 261 op/s 2024-05-09T09:02:22.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:21 smithi105 ceph-mon[34498]: pgmap v368: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.6 KiB/s wr, 261 op/s 2024-05-09T09:02:24.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:23 smithi098 ceph-mon[30092]: pgmap v369: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.7 KiB/s wr, 236 op/s 2024-05-09T09:02:24.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:23 smithi142 ceph-mon[34625]: pgmap v369: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.7 KiB/s wr, 236 op/s 2024-05-09T09:02:24.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:23 smithi105 ceph-mon[34498]: pgmap v369: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.7 KiB/s wr, 236 op/s 2024-05-09T09:02:26.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:25 smithi098 ceph-mon[30092]: pgmap v370: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 4.2 KiB/s wr, 148 op/s 2024-05-09T09:02:26.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:25 smithi142 ceph-mon[34625]: pgmap v370: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 4.2 KiB/s wr, 148 op/s 2024-05-09T09:02:26.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:25 smithi105 ceph-mon[34498]: pgmap v370: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 4.2 KiB/s wr, 148 op/s 2024-05-09T09:02:28.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:28 smithi098 ceph-mon[30092]: pgmap v371: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 4.1 KiB/s wr, 29 op/s 2024-05-09T09:02:28.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:28 smithi142 ceph-mon[34625]: pgmap v371: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 4.1 KiB/s wr, 29 op/s 2024-05-09T09:02:28.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:28 smithi105 ceph-mon[34498]: pgmap v371: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 4.1 KiB/s wr, 29 op/s 2024-05-09T09:02:30.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:30 smithi098 ceph-mon[30092]: pgmap v372: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 3.8 KiB/s wr, 18 op/s 2024-05-09T09:02:30.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:30 smithi142 ceph-mon[34625]: pgmap v372: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 3.8 KiB/s wr, 18 op/s 2024-05-09T09:02:30.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:30 smithi105 ceph-mon[34498]: pgmap v372: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 3.8 KiB/s wr, 18 op/s 2024-05-09T09:02:31.942 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:02:31.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:02:32.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.063+0000 7f542dfd5640 1 -- 172.21.15.98:0/378426931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5428100700 msgr2=0x7f5428100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.063+0000 7f542dfd5640 1 --2- 172.21.15.98:0/378426931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5428100700 0x7f5428100b80 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f5410009930 tx=0x7f541002a420 comp rx=0 tx=0).stop 2024-05-09T09:02:32.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.063+0000 7f542dfd5640 1 -- 172.21.15.98:0/378426931 shutdown_connections 2024-05-09T09:02:32.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.063+0000 7f542dfd5640 1 --2- 172.21.15.98:0/378426931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54281010c0 0x7f542810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.063+0000 7f542dfd5640 1 --2- 172.21.15.98:0/378426931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5428100700 0x7f5428100b80 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.063+0000 7f542dfd5640 1 --2- 172.21.15.98:0/378426931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5428106710 0x7f5428106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.063+0000 7f542dfd5640 1 -- 172.21.15.98:0/378426931 >> 172.21.15.98:0/378426931 conn(0x7f5428077830 msgr2=0x7f5428077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.063+0000 7f542dfd5640 1 -- 172.21.15.98:0/378426931 shutdown_connections 2024-05-09T09:02:32.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.063+0000 7f542dfd5640 1 -- 172.21.15.98:0/378426931 wait complete. 2024-05-09T09:02:32.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.064+0000 7f542dfd5640 1 Processor -- start 2024-05-09T09:02:32.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.064+0000 7f542dfd5640 1 -- start start 2024-05-09T09:02:32.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.064+0000 7f542dfd5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5428100700 0x7f5428198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.064+0000 7f542dfd5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54281010c0 0x7f5428198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.064+0000 7f542dfd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5428106710 0x7f542819ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.064+0000 7f542dfd5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5428111c20 con 0x7f5428106710 2024-05-09T09:02:32.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.064+0000 7f542dfd5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5428111aa0 con 0x7f5428100700 2024-05-09T09:02:32.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.064+0000 7f542dfd5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5428111da0 con 0x7f54281010c0 2024-05-09T09:02:32.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.064+0000 7f5427fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5428106710 0x7f542819ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.064+0000 7f5427fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5428106710 0x7f542819ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51000/0 (socket says 172.21.15.98:51000) 2024-05-09T09:02:32.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.064+0000 7f5427fff640 1 -- 172.21.15.98:0/649964756 learned_addr learned my addr 172.21.15.98:0/649964756 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.064+0000 7f541ffff640 1 --2- 172.21.15.98:0/649964756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54281010c0 0x7f5428198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.064+0000 7f54277fe640 1 --2- 172.21.15.98:0/649964756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5428100700 0x7f5428198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.065+0000 7f5427fff640 1 -- 172.21.15.98:0/649964756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54281010c0 msgr2=0x7f5428198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.065+0000 7f5427fff640 1 --2- 172.21.15.98:0/649964756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54281010c0 0x7f5428198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.065+0000 7f5427fff640 1 -- 172.21.15.98:0/649964756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5428100700 msgr2=0x7f5428198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.065+0000 7f5427fff640 1 --2- 172.21.15.98:0/649964756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5428100700 0x7f5428198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.065+0000 7f5427fff640 1 -- 172.21.15.98:0/649964756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542819d130 con 0x7f5428106710 2024-05-09T09:02:32.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.065+0000 7f5427fff640 1 --2- 172.21.15.98:0/649964756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5428106710 0x7f542819ca00 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f542000eb10 tx=0x7f542000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.065+0000 7f54257fa640 1 -- 172.21.15.98:0/649964756 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420004b20 con 0x7f5428106710 2024-05-09T09:02:32.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.065+0000 7f54257fa640 1 -- 172.21.15.98:0/649964756 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5420004cc0 con 0x7f5428106710 2024-05-09T09:02:32.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.065+0000 7f542dfd5640 1 -- 172.21.15.98:0/649964756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f542819d420 con 0x7f5428106710 2024-05-09T09:02:32.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.065+0000 7f54257fa640 1 -- 172.21.15.98:0/649964756 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420010760 con 0x7f5428106710 2024-05-09T09:02:32.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.065+0000 7f542dfd5640 1 -- 172.21.15.98:0/649964756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54281a4cb0 con 0x7f5428106710 2024-05-09T09:02:32.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.067+0000 7f54257fa640 1 -- 172.21.15.98:0/649964756 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5420005020 con 0x7f5428106710 2024-05-09T09:02:32.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f5430db6740 1 Processor -- start 2024-05-09T09:02:32.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f5430db6740 1 -- start start 2024-05-09T09:02:32.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f5430db6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2280 0x563dcf7f2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f5430db6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7f2ba0 0x563dcf7fcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f5430db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7fd270 0x563dcf7ff660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f5430db6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563dcf757d00 con 0x563dcf7fd270 2024-05-09T09:02:32.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f5430db6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563dcf6ff4c0 con 0x563dcf7f2ba0 2024-05-09T09:02:32.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f5430db6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563dcf753cc0 con 0x563dcf7f2280 2024-05-09T09:02:32.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f54257fa640 1 --2- 172.21.15.98:0/649964756 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54140a2d80 0x7f54140a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f541ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7fd270 0x563dcf7ff660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f5427fff640 1 --2- 172.21.15.98:0/649964756 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54140a2d80 0x7f54140a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f541ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7fd270 0x563dcf7ff660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51002/0 (socket says 172.21.15.98:51002) 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f541ffff640 1 -- 172.21.15.98:0/197515724 learned_addr learned my addr 172.21.15.98:0/197515724 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f541ffff640 1 --2- 172.21.15.98:0/197515724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2280 0x563dcf7f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f54277fe640 1 --2- 172.21.15.98:0/197515724 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7f2ba0 0x563dcf7fcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.068+0000 7f54257fa640 1 -- 172.21.15.98:0/649964756 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f54200cfd90 con 0x7f5428106710 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.069+0000 7f541ffff640 1 -- 172.21.15.98:0/197515724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2280 msgr2=0x563dcf7f2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.069+0000 7f541ffff640 1 --2- 172.21.15.98:0/197515724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2280 0x563dcf7f2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.069+0000 7f541ffff640 1 -- 172.21.15.98:0/197515724 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7f2ba0 msgr2=0x563dcf7fcd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.069+0000 7f541ffff640 1 --2- 172.21.15.98:0/197515724 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7f2ba0 0x563dcf7fcd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.069+0000 7f541ffff640 1 -- 172.21.15.98:0/197515724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dcf5357d0 con 0x563dcf7fd270 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.069+0000 7f5427fff640 1 --2- 172.21.15.98:0/649964756 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54140a2d80 0x7f54140a5240 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f542009bba0 tx=0x7f542000efd0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.069+0000 7f541ffff640 1 --2- 172.21.15.98:0/197515724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7fd270 0x563dcf7ff660 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f541002a940 tx=0x7f541002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57be3019af98961e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.069+0000 7f541dffb640 1 -- 172.21.15.98:0/197515724 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410035070 con 0x563dcf7fd270 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.069+0000 7f541dffb640 1 -- 172.21.15.98:0/197515724 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5410005020 con 0x563dcf7fd270 2024-05-09T09:02:32.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.069+0000 7f541dffb640 1 -- 172.21.15.98:0/197515724 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410004a00 con 0x563dcf7fd270 2024-05-09T09:02:32.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.070+0000 7f5430db6740 1 -- 172.21.15.98:0/197515724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7fd270 msgr2=0x563dcf7ff660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.070+0000 7f5430db6740 1 --2- 172.21.15.98:0/197515724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7fd270 0x563dcf7ff660 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f541002a940 tx=0x7f541002a420 comp rx=0 tx=0).stop 2024-05-09T09:02:32.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.070+0000 7f5430db6740 1 -- 172.21.15.98:0/197515724 shutdown_connections 2024-05-09T09:02:32.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.070+0000 7f5430db6740 1 --2- 172.21.15.98:0/197515724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7fd270 0x563dcf7ff660 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.070+0000 7f5430db6740 1 --2- 172.21.15.98:0/197515724 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7f2ba0 0x563dcf7fcd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.070+0000 7f5430db6740 1 --2- 172.21.15.98:0/197515724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2280 0x563dcf7f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.070+0000 7f5430db6740 1 -- 172.21.15.98:0/197515724 >> 172.21.15.98:0/197515724 conn(0x563dcf7ee5a0 msgr2=0x563dcf7ee980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.070+0000 7f5430db6740 1 -- 172.21.15.98:0/197515724 shutdown_connections 2024-05-09T09:02:32.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.070+0000 7f5430db6740 1 -- 172.21.15.98:0/197515724 wait complete. 2024-05-09T09:02:32.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.070+0000 7f5430db6740 1 -- start start 2024-05-09T09:02:32.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.070+0000 7f5430db6740 1 Processor -- start 2024-05-09T09:02:32.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.070+0000 7f5430db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7f2280 0x563dcf7fa220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.070+0000 7f5430db6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 0x563dcf7fa760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.070+0000 7f5430db6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7fd270 0x563dcf7faca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f5430db6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563dcf800000 con 0x563dcf7f2280 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f5430db6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563dcf712b90 con 0x563dcf7fd270 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f5430db6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563dcf800180 con 0x563dcf7f2ba0 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f5430db6740 10 client.0 _ll_register_callbacks cb 0x563dcf74eda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f541ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7f2280 0x563dcf7fa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f541ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7f2280 0x563dcf7fa220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51008/0 (socket says 172.21.15.98:51008) 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f541ffff640 1 -- 172.21.15.98:0/1536636296 learned_addr learned my addr 172.21.15.98:0/1536636296 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f541ffff640 1 --2- 172.21.15.98:0/1536636296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7fd270 0x563dcf7faca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f541ffff640 1 -- 172.21.15.98:0/1536636296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 msgr2=0x563dcf7fa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f541ffff640 1 --2- 172.21.15.98:0/1536636296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 0x563dcf7fa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f541ffff640 1 -- 172.21.15.98:0/1536636296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7fd270 msgr2=0x563dcf7faca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f541ffff640 1 --2- 172.21.15.98:0/1536636296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7fd270 0x563dcf7faca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f541ffff640 1 -- 172.21.15.98:0/1536636296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dcf46b3f0 con 0x563dcf7f2280 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f541ffff640 1 --2- 172.21.15.98:0/1536636296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7f2280 0x563dcf7fa220 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f5410001eb0 tx=0x7f5410005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab018131b38e6968 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f5406ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f5406ffd640 1 -- 172.21.15.98:0/1536636296 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410055060 con 0x563dcf7f2280 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f5406ffd640 1 -- 172.21.15.98:0/1536636296 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f54100053d0 con 0x563dcf7f2280 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f5430db6740 1 -- 172.21.15.98:0/1536636296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5410005020 con 0x563dcf7f2280 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f5406ffd640 1 -- 172.21.15.98:0/1536636296 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5410035070 con 0x563dcf7f2280 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f5430db6740 10 client.19098 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.071+0000 7f5430db6740 1 -- 172.21.15.98:0/1536636296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563dcf42b0f0 con 0x563dcf7f2280 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.072+0000 7f5406ffd640 1 -- 172.21.15.98:0/1536636296 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5410005020 con 0x563dcf7f2280 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.072+0000 7f5406ffd640 1 -- 172.21.15.98:0/1536636296 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f541003d680 con 0x563dcf7f2280 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.072+0000 7f5430db6740 10 client.19098 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.072+0000 7f5430db6740 10 client.19098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.072+0000 7f5430db6740 20 client.19098 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.072+0000 7f5430db6740 1 --2- 172.21.15.98:0/1536636296 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563dcf80d4a0 0x563dcf80f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.072+0000 7f5430db6740 4 client.19098 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.072+0000 7f5430db6740 1 -- 172.21.15.98:0/1536636296 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563dcf765430 con 0x563dcf80d4a0 2024-05-09T09:02:32.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.072+0000 7f54277fe640 1 --2- 172.21.15.98:0/1536636296 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563dcf80d4a0 0x563dcf80f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.072+0000 7f5406ffd640 10 client.19098 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.072+0000 7f54277fe640 1 --2- 172.21.15.98:0/1536636296 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563dcf80d4a0 0x563dcf80f890 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e23d493d42351cd5 server_cookie=5f324880bfa46089 in_seq=0 out_seq=0 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.073+0000 7f5406ffd640 1 -- 172.21.15.98:0/1536636296 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563dcf765430 con 0x563dcf80d4a0 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.073+0000 7f5406ffd640 10 client.19098 handle_command_reply: tid=0 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.073+0000 7f5430db6740 1 client.19098 shutdown 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.073+0000 7f5430db6740 1 -- 172.21.15.98:0/1536636296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7f2280 msgr2=0x563dcf7fa220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.073+0000 7f5430db6740 1 --2- 172.21.15.98:0/1536636296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7f2280 0x563dcf7fa220 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f5410001eb0 tx=0x7f5410005d50 comp rx=0 tx=0).stop 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.073+0000 7f5430db6740 1 -- 172.21.15.98:0/1536636296 shutdown_connections 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.073+0000 7f5430db6740 1 --2- 172.21.15.98:0/1536636296 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563dcf80d4a0 0x563dcf80f890 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.073+0000 7f5430db6740 1 --2- 172.21.15.98:0/1536636296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7fd270 0x563dcf7faca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.073+0000 7f5430db6740 1 --2- 172.21.15.98:0/1536636296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 0x563dcf7fa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.073+0000 7f5430db6740 1 --2- 172.21.15.98:0/1536636296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7f2280 0x563dcf7fa220 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.073+0000 7f5406ffd640 0 client.19098 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.073+0000 7f5430db6740 1 -- 172.21.15.98:0/1536636296 >> 172.21.15.98:0/1536636296 conn(0x563dcf7ee5a0 msgr2=0x563dcf7f43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.073+0000 7f5430db6740 1 -- 172.21.15.98:0/1536636296 shutdown_connections 2024-05-09T09:02:32.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.073+0000 7f5430db6740 1 -- 172.21.15.98:0/1536636296 wait complete. 2024-05-09T09:02:32.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.074+0000 7f5430db6740 20 client.19098 trim_cache size 0 max 16384 2024-05-09T09:02:32.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f5430db6740 1 Processor -- start 2024-05-09T09:02:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f5430db6740 1 -- start start 2024-05-09T09:02:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f5430db6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7f2280 0x563dcf7f9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f5430db6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 0x563dcf7f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f5430db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7fd270 0x563dcf7fa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f5430db6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f54281aef50 con 0x563dcf7fd270 2024-05-09T09:02:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f5430db6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f54281b0a90 con 0x563dcf7f2280 2024-05-09T09:02:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f5430db6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f54281af620 con 0x563dcf7f2ba0 2024-05-09T09:02:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f541ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7fd270 0x563dcf7fa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f541ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7fd270 0x563dcf7fa8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51022/0 (socket says 172.21.15.98:51022) 2024-05-09T09:02:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f541ffff640 1 -- 172.21.15.98:0/2199993505 learned_addr learned my addr 172.21.15.98:0/2199993505 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f54277fe640 1 --2- 172.21.15.98:0/2199993505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 0x563dcf7f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f541ffff640 1 -- 172.21.15.98:0/2199993505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 msgr2=0x563dcf7f4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f541ffff640 1 --2- 172.21.15.98:0/2199993505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 0x563dcf7f4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f541ffff640 1 -- 172.21.15.98:0/2199993505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7f2280 msgr2=0x563dcf7f9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f541ffff640 1 --2- 172.21.15.98:0/2199993505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7f2280 0x563dcf7f9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f541ffff640 1 -- 172.21.15.98:0/2199993505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dcf7ffa20 con 0x563dcf7fd270 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f54277fe640 1 --2- 172.21.15.98:0/2199993505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 0x563dcf7f4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f541ffff640 1 --2- 172.21.15.98:0/2199993505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7fd270 0x563dcf7fa8f0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f541002abd0 tx=0x7f5410005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafb9568ca31beef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.091+0000 7f541d7fa640 1 -- 172.21.15.98:0/2199993505 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410065070 con 0x563dcf7fd270 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.092+0000 7f541d7fa640 1 -- 172.21.15.98:0/2199993505 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f541003d650 con 0x563dcf7fd270 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.092+0000 7f541d7fa640 1 -- 172.21.15.98:0/2199993505 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410055040 con 0x563dcf7fd270 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.092+0000 7f5430db6740 1 -- 172.21.15.98:0/2199993505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7fd270 msgr2=0x563dcf7fa8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.092+0000 7f5430db6740 1 --2- 172.21.15.98:0/2199993505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7fd270 0x563dcf7fa8f0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f541002abd0 tx=0x7f5410005fd0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.092+0000 7f5430db6740 1 -- 172.21.15.98:0/2199993505 shutdown_connections 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.092+0000 7f5430db6740 1 --2- 172.21.15.98:0/2199993505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7fd270 0x563dcf7fa8f0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.092+0000 7f5430db6740 1 --2- 172.21.15.98:0/2199993505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 0x563dcf7f4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.092+0000 7f5430db6740 1 --2- 172.21.15.98:0/2199993505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7f2280 0x563dcf7f9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.092+0000 7f5430db6740 1 -- 172.21.15.98:0/2199993505 >> 172.21.15.98:0/2199993505 conn(0x563dcf7ee5a0 msgr2=0x563dcf7fa1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.092+0000 7f5430db6740 1 -- 172.21.15.98:0/2199993505 shutdown_connections 2024-05-09T09:02:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.092+0000 7f5430db6740 1 -- 172.21.15.98:0/2199993505 wait complete. 2024-05-09T09:02:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.092+0000 7f5430db6740 1 -- start start 2024-05-09T09:02:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f5430db6740 1 Processor -- start 2024-05-09T09:02:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f5430db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7f2280 0x563dcf810350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f5430db6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 0x563dcf810890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f5430db6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7fd270 0x563dcf810dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f5430db6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f54281b03c0 con 0x563dcf7f2280 2024-05-09T09:02:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f5430db6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f54281afcf0 con 0x563dcf7fd270 2024-05-09T09:02:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f5430db6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f54281acf50 con 0x563dcf7f2ba0 2024-05-09T09:02:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f5430db6740 10 client.0 _ll_register_callbacks cb 0x7f5418009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f541ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7f2280 0x563dcf810350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f541ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7f2280 0x563dcf810350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51032/0 (socket says 172.21.15.98:51032) 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f541ffff640 1 -- 172.21.15.98:0/1323821071 learned_addr learned my addr 172.21.15.98:0/1323821071 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f54277fe640 1 --2- 172.21.15.98:0/1323821071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 0x563dcf810890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f541ffff640 1 --2- 172.21.15.98:0/1323821071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7fd270 0x563dcf810dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f541ffff640 1 -- 172.21.15.98:0/1323821071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 msgr2=0x563dcf810890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f541ffff640 1 --2- 172.21.15.98:0/1323821071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 0x563dcf810890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f541ffff640 1 -- 172.21.15.98:0/1323821071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7fd270 msgr2=0x563dcf810dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f541ffff640 1 --2- 172.21.15.98:0/1323821071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7fd270 0x563dcf810dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.093+0000 7f541ffff640 1 -- 172.21.15.98:0/1323821071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dcf8726d0 con 0x563dcf7f2280 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f541ffff640 1 --2- 172.21.15.98:0/1323821071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7f2280 0x563dcf810350 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f5410033000 tx=0x7f5410005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9319c836b6ea6755 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5406ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5406ffd640 1 -- 172.21.15.98:0/1323821071 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410065070 con 0x563dcf7f2280 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5406ffd640 1 -- 172.21.15.98:0/1323821071 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f541004ab40 con 0x563dcf7f2280 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5430db6740 1 -- 172.21.15.98:0/1323821071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f541003d650 con 0x563dcf7f2280 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5406ffd640 1 -- 172.21.15.98:0/1323821071 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5410055040 con 0x563dcf7f2280 2024-05-09T09:02:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5430db6740 10 client.19104 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5430db6740 1 -- 172.21.15.98:0/1323821071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563dcf873950 con 0x563dcf7f2280 2024-05-09T09:02:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5406ffd640 1 -- 172.21.15.98:0/1323821071 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f541003d650 con 0x563dcf7f2280 2024-05-09T09:02:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5406ffd640 1 -- 172.21.15.98:0/1323821071 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f541003c4b0 con 0x563dcf7f2280 2024-05-09T09:02:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5430db6740 10 client.19104 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5430db6740 10 client.19104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:02:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5430db6740 20 client.19104 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5430db6740 1 --2- 172.21.15.98:0/1323821071 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563dcf873e70 0x563dcf876260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5430db6740 4 client.19104 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:02:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.094+0000 7f5430db6740 1 -- 172.21.15.98:0/1323821071 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563dcf7f98d0 con 0x563dcf873e70 2024-05-09T09:02:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f54277fe640 1 --2- 172.21.15.98:0/1323821071 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563dcf873e70 0x563dcf876260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f54277fe640 1 --2- 172.21.15.98:0/1323821071 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563dcf873e70 0x563dcf876260 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f5c6287572240ad server_cookie=b7eacbd8d031c192 in_seq=0 out_seq=0 2024-05-09T09:02:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f5406ffd640 10 client.19104 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f5406ffd640 1 -- 172.21.15.98:0/1323821071 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563dcf7f98d0 con 0x563dcf873e70 2024-05-09T09:02:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f5406ffd640 10 client.19104 handle_command_reply: tid=0 2024-05-09T09:02:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f5430db6740 1 client.19104 shutdown 2024-05-09T09:02:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f5430db6740 1 -- 172.21.15.98:0/1323821071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7f2280 msgr2=0x563dcf810350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f5430db6740 1 --2- 172.21.15.98:0/1323821071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7f2280 0x563dcf810350 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f5410033000 tx=0x7f5410005780 comp rx=0 tx=0).stop 2024-05-09T09:02:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f5430db6740 1 -- 172.21.15.98:0/1323821071 shutdown_connections 2024-05-09T09:02:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f5430db6740 1 --2- 172.21.15.98:0/1323821071 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x563dcf873e70 0x563dcf876260 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f5430db6740 1 --2- 172.21.15.98:0/1323821071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563dcf7fd270 0x563dcf810dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f5430db6740 1 --2- 172.21.15.98:0/1323821071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563dcf7f2ba0 0x563dcf810890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f5406ffd640 0 client.19104 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f5430db6740 1 --2- 172.21.15.98:0/1323821071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563dcf7f2280 0x563dcf810350 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.095+0000 7f5430db6740 1 -- 172.21.15.98:0/1323821071 >> 172.21.15.98:0/1323821071 conn(0x563dcf7ee5a0 msgr2=0x563dcf7f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f5430db6740 1 -- 172.21.15.98:0/1323821071 shutdown_connections 2024-05-09T09:02:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f5430db6740 1 -- 172.21.15.98:0/1323821071 wait complete. 2024-05-09T09:02:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f5430db6740 20 client.19104 trim_cache size 0 max 16384 2024-05-09T09:02:32.102 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:02:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-05-09T09:02:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-05-09T09:02:32.102 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:02:32.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f542dfd5640 1 -- 172.21.15.98:0/649964756 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54140a2d80 msgr2=0x7f54140a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f542dfd5640 1 --2- 172.21.15.98:0/649964756 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54140a2d80 0x7f54140a5240 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f542009bba0 tx=0x7f542000efd0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f542dfd5640 1 -- 172.21.15.98:0/649964756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5428106710 msgr2=0x7f542819ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f542dfd5640 1 --2- 172.21.15.98:0/649964756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5428106710 0x7f542819ca00 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f542000eb10 tx=0x7f542000c2d0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f542dfd5640 1 -- 172.21.15.98:0/649964756 shutdown_connections 2024-05-09T09:02:32.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f542dfd5640 1 --2- 172.21.15.98:0/649964756 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54140a2d80 0x7f54140a5240 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f542dfd5640 1 --2- 172.21.15.98:0/649964756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5428106710 0x7f542819ca00 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f542dfd5640 1 --2- 172.21.15.98:0/649964756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54281010c0 0x7f5428198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f542dfd5640 1 --2- 172.21.15.98:0/649964756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5428100700 0x7f5428198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f542dfd5640 1 -- 172.21.15.98:0/649964756 >> 172.21.15.98:0/649964756 conn(0x7f5428077830 msgr2=0x7f5428102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f542dfd5640 1 -- 172.21.15.98:0/649964756 shutdown_connections 2024-05-09T09:02:32.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.096+0000 7f542dfd5640 1 -- 172.21.15.98:0/649964756 wait complete. 2024-05-09T09:02:32.107 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T09:02:32.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:02:32.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.209+0000 7f19414ee640 1 -- 172.21.15.98:0/1362156769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f193c101c50 msgr2=0x7f193c10d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.209+0000 7f19414ee640 1 --2- 172.21.15.98:0/1362156769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f193c101c50 0x7f193c10d5e0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f1934009cd0 tx=0x7f193402a970 comp rx=0 tx=0).stop 2024-05-09T09:02:32.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.209+0000 7f19414ee640 1 -- 172.21.15.98:0/1362156769 shutdown_connections 2024-05-09T09:02:32.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.209+0000 7f19414ee640 1 --2- 172.21.15.98:0/1362156769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f193c101c50 0x7f193c10d5e0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.209+0000 7f19414ee640 1 --2- 172.21.15.98:0/1362156769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c101290 0x7f193c101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.209+0000 7f19414ee640 1 --2- 172.21.15.98:0/1362156769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c105770 0x7f193c105b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.209+0000 7f19414ee640 1 -- 172.21.15.98:0/1362156769 >> 172.21.15.98:0/1362156769 conn(0x7f193c075a10 msgr2=0x7f193c075e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.209+0000 7f19414ee640 1 -- 172.21.15.98:0/1362156769 shutdown_connections 2024-05-09T09:02:32.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.209+0000 7f19414ee640 1 -- 172.21.15.98:0/1362156769 wait complete. 2024-05-09T09:02:32.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f19414ee640 1 Processor -- start 2024-05-09T09:02:32.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f19414ee640 1 -- start start 2024-05-09T09:02:32.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f19414ee640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c101290 0x7f193c197010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f19414ee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f193c101c50 0x7f193c197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f19414ee640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c105770 0x7f193c19b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f19414ee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f193c110b30 con 0x7f193c101c50 2024-05-09T09:02:32.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f19414ee640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f193c1109b0 con 0x7f193c105770 2024-05-09T09:02:32.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f19414ee640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f193c110cb0 con 0x7f193c101290 2024-05-09T09:02:32.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f1933fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f193c101c50 0x7f193c197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f1933fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f193c101c50 0x7f193c197550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51044/0 (socket says 172.21.15.98:51044) 2024-05-09T09:02:32.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f1933fff640 1 -- 172.21.15.98:0/3264051774 learned_addr learned my addr 172.21.15.98:0/3264051774 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f193affd640 1 --2- 172.21.15.98:0/3264051774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c101290 0x7f193c197010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f193b7fe640 1 --2- 172.21.15.98:0/3264051774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c105770 0x7f193c19b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f1933fff640 1 -- 172.21.15.98:0/3264051774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c101290 msgr2=0x7f193c197010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f1933fff640 1 --2- 172.21.15.98:0/3264051774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c101290 0x7f193c197010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f1933fff640 1 -- 172.21.15.98:0/3264051774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c105770 msgr2=0x7f193c19b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.210+0000 7f1933fff640 1 --2- 172.21.15.98:0/3264051774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c105770 0x7f193c19b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.211+0000 7f1933fff640 1 -- 172.21.15.98:0/3264051774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193c19bfb0 con 0x7f193c101c50 2024-05-09T09:02:32.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.211+0000 7f1933fff640 1 --2- 172.21.15.98:0/3264051774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f193c101c50 0x7f193c197550 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f1924009900 tx=0x7f1924009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.211+0000 7f1938ff9640 1 -- 172.21.15.98:0/3264051774 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192400d030 con 0x7f193c101c50 2024-05-09T09:02:32.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.211+0000 7f19414ee640 1 -- 172.21.15.98:0/3264051774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f193c19c2a0 con 0x7f193c101c50 2024-05-09T09:02:32.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.211+0000 7f1938ff9640 1 -- 172.21.15.98:0/3264051774 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f192400d1d0 con 0x7f193c101c50 2024-05-09T09:02:32.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.211+0000 7f1938ff9640 1 -- 172.21.15.98:0/3264051774 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1924005280 con 0x7f193c101c50 2024-05-09T09:02:32.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.211+0000 7f19414ee640 1 -- 172.21.15.98:0/3264051774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f193c100180 con 0x7f193c101c50 2024-05-09T09:02:32.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.212+0000 7f19414ee640 1 -- 172.21.15.98:0/3264051774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f193c068aa0 con 0x7f193c101c50 2024-05-09T09:02:32.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.213+0000 7f1938ff9640 1 -- 172.21.15.98:0/3264051774 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f19240054e0 con 0x7f193c101c50 2024-05-09T09:02:32.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.214+0000 7f1938ff9640 1 --2- 172.21.15.98:0/3264051774 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f19280a2d00 0x7f19280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.214+0000 7f1938ff9640 1 -- 172.21.15.98:0/3264051774 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f19240d0b60 con 0x7f193c101c50 2024-05-09T09:02:32.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.214+0000 7f193affd640 1 --2- 172.21.15.98:0/3264051774 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f19280a2d00 0x7f19280a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.214+0000 7f193affd640 1 --2- 172.21.15.98:0/3264051774 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f19280a2d00 0x7f19280a51c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f193c076f20 tx=0x7f192c005f90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.218+0000 7f1938ff9640 1 -- 172.21.15.98:0/3264051774 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f192409cd20 con 0x7f193c101c50 2024-05-09T09:02:32.289 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:32 smithi105 ceph-mon[34498]: pgmap v373: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 3.7 KiB/s wr, 13 op/s 2024-05-09T09:02:32.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:32 smithi098 ceph-mon[30092]: pgmap v373: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 3.7 KiB/s wr, 13 op/s 2024-05-09T09:02:32.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:32 smithi142 ceph-mon[34625]: pgmap v373: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 3.7 KiB/s wr, 13 op/s 2024-05-09T09:02:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.433+0000 7f19414ee640 1 -- 172.21.15.98:0/3264051774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f193c100c60 con 0x7f193c101c50 2024-05-09T09:02:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.434+0000 7f1938ff9640 1 -- 172.21.15.98:0/3264051774 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11941 (secure 0 0 0) 0x7f19240a1020 con 0x7f193c101c50 2024-05-09T09:02:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T09:02:32.434 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":26581,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/561027115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":561027115},{"type":"v1","addr":"172.21.15.105:6835","nonce":561027115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:56:52.891273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16641":{"gid":16641,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T09:02:32.434 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 37 2024-05-09T09:02:32.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.436+0000 7f19414ee640 1 -- 172.21.15.98:0/3264051774 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f19280a2d00 msgr2=0x7f19280a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.436+0000 7f19414ee640 1 --2- 172.21.15.98:0/3264051774 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f19280a2d00 0x7f19280a51c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f193c076f20 tx=0x7f192c005f90 comp rx=0 tx=0).stop 2024-05-09T09:02:32.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.436+0000 7f19414ee640 1 -- 172.21.15.98:0/3264051774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f193c101c50 msgr2=0x7f193c197550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.436+0000 7f19414ee640 1 --2- 172.21.15.98:0/3264051774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f193c101c50 0x7f193c197550 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f1924009900 tx=0x7f1924009dc0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.436+0000 7f19414ee640 1 -- 172.21.15.98:0/3264051774 shutdown_connections 2024-05-09T09:02:32.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.436+0000 7f19414ee640 1 --2- 172.21.15.98:0/3264051774 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f19280a2d00 0x7f19280a51c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.436+0000 7f19414ee640 1 --2- 172.21.15.98:0/3264051774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c105770 0x7f193c19b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.436+0000 7f19414ee640 1 --2- 172.21.15.98:0/3264051774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f193c101c50 0x7f193c197550 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.436+0000 7f19414ee640 1 --2- 172.21.15.98:0/3264051774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c101290 0x7f193c197010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.436+0000 7f19414ee640 1 -- 172.21.15.98:0/3264051774 >> 172.21.15.98:0/3264051774 conn(0x7f193c075a10 msgr2=0x7f193c0fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.436+0000 7f19414ee640 1 -- 172.21.15.98:0/3264051774 shutdown_connections 2024-05-09T09:02:32.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.436+0000 7f19414ee640 1 -- 172.21.15.98:0/3264051774 wait complete. 2024-05-09T09:02:32.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T09:02:32.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.551+0000 7fb19f7f9640 1 -- 172.21.15.98:0/1939257750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb198105c80 msgr2=0x7fb198106130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.551+0000 7fb19f7f9640 1 --2- 172.21.15.98:0/1939257750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb198105c80 0x7fb198106130 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fb190009cd0 tx=0x7fb19002a970 comp rx=0 tx=0).stop 2024-05-09T09:02:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.551+0000 7fb19f7f9640 1 -- 172.21.15.98:0/1939257750 shutdown_connections 2024-05-09T09:02:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.551+0000 7fb19f7f9640 1 --2- 172.21.15.98:0/1939257750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb198105c80 0x7fb198106130 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.551+0000 7fb19f7f9640 1 --2- 172.21.15.98:0/1939257750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1981011a0 0x7fb198105660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.551+0000 7fb19f7f9640 1 --2- 172.21.15.98:0/1939257750 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb198100880 0x7fb198100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.551+0000 7fb19f7f9640 1 -- 172.21.15.98:0/1939257750 >> 172.21.15.98:0/1939257750 conn(0x7fb198076ab0 msgr2=0x7fb198076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.551+0000 7fb19f7f9640 1 -- 172.21.15.98:0/1939257750 shutdown_connections 2024-05-09T09:02:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.551+0000 7fb19f7f9640 1 -- 172.21.15.98:0/1939257750 wait complete. 2024-05-09T09:02:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.551+0000 7fb19f7f9640 1 Processor -- start 2024-05-09T09:02:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.551+0000 7fb19f7f9640 1 -- start start 2024-05-09T09:02:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19f7f9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb198100880 0x7fb19819e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19f7f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1981011a0 0x7fb19819ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19f7f9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb198105c80 0x7fb1981a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19f7f9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1981181d0 con 0x7fb1981011a0 2024-05-09T09:02:32.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19f7f9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb198118050 con 0x7fb198105c80 2024-05-09T09:02:32.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19f7f9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb198118350 con 0x7fb198100880 2024-05-09T09:02:32.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19cd94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1981011a0 0x7fb19819ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19cd94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1981011a0 0x7fb19819ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51070/0 (socket says 172.21.15.98:51070) 2024-05-09T09:02:32.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19cd94640 1 -- 172.21.15.98:0/1447157802 learned_addr learned my addr 172.21.15.98:0/1447157802 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19d595640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb198100880 0x7fb19819e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19dd96640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb198105c80 0x7fb1981a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19cd94640 1 -- 172.21.15.98:0/1447157802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb198100880 msgr2=0x7fb19819e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19cd94640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb198100880 0x7fb19819e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19cd94640 1 -- 172.21.15.98:0/1447157802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb198105c80 msgr2=0x7fb1981a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19cd94640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb198105c80 0x7fb1981a2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19cd94640 1 -- 172.21.15.98:0/1447157802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1981a3700 con 0x7fb1981011a0 2024-05-09T09:02:32.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19d595640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb198100880 0x7fb19819e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:32.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19cd94640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1981011a0 0x7fb19819ec50 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fb18800d950 tx=0x7fb18800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb18e7fc640 1 -- 172.21.15.98:0/1447157802 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb188017070 con 0x7fb1981011a0 2024-05-09T09:02:32.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb18e7fc640 1 -- 172.21.15.98:0/1447157802 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb188009c30 con 0x7fb1981011a0 2024-05-09T09:02:32.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb18e7fc640 1 -- 172.21.15.98:0/1447157802 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18800f6b0 con 0x7fb1981011a0 2024-05-09T09:02:32.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19f7f9640 1 -- 172.21.15.98:0/1447157802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1981a39f0 con 0x7fb1981011a0 2024-05-09T09:02:32.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.552+0000 7fb19f7f9640 1 -- 172.21.15.98:0/1447157802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1981ab280 con 0x7fb1981011a0 2024-05-09T09:02:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.554+0000 7fb18e7fc640 1 -- 172.21.15.98:0/1447157802 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb1880109d0 con 0x7fb1981011a0 2024-05-09T09:02:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.555+0000 7fb1a25e9740 1 Processor -- start 2024-05-09T09:02:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.555+0000 7fb1a25e9740 1 -- start start 2024-05-09T09:02:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.555+0000 7fb18e7fc640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb1840a2d80 0x7fb1840a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.555+0000 7fb1a25e9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457f330 0x55bfe457f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.555+0000 7fb1a25e9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457fc50 0x55bfe4589de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.555+0000 7fb1a25e9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe458a320 0x55bfe458c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.555+0000 7fb1a25e9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bfe44e4c40 con 0x55bfe457f330 2024-05-09T09:02:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.555+0000 7fb1a25e9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bfe448c580 con 0x55bfe458a320 2024-05-09T09:02:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.555+0000 7fb1a25e9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55bfe44e0c00 con 0x55bfe457fc50 2024-05-09T09:02:32.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.555+0000 7fb19dd96640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb1840a2d80 0x7fb1840a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.555+0000 7fb19d595640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457f330 0x55bfe457f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.555+0000 7fb18e7fc640 1 -- 172.21.15.98:0/1447157802 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7fb1880cf430 con 0x7fb1981011a0 2024-05-09T09:02:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.555+0000 7fb19d595640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457f330 0x55bfe457f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51078/0 (socket says 172.21.15.98:51078) 2024-05-09T09:02:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.555+0000 7fb19d595640 1 -- 172.21.15.98:0/2444185885 learned_addr learned my addr 172.21.15.98:0/2444185885 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.556+0000 7fb19cd94640 1 --2- 172.21.15.98:0/2444185885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457fc50 0x55bfe4589de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.556+0000 7fb19d595640 1 -- 172.21.15.98:0/2444185885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457fc50 msgr2=0x55bfe4589de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.556+0000 7fb19dd96640 1 --2- 172.21.15.98:0/2444185885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe458a320 0x55bfe458c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.556+0000 7fb19d595640 1 --2- 172.21.15.98:0/2444185885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457fc50 0x55bfe4589de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.556+0000 7fb19d595640 1 -- 172.21.15.98:0/2444185885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe458a320 msgr2=0x55bfe458c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.556+0000 7fb19d595640 1 --2- 172.21.15.98:0/2444185885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe458a320 0x55bfe458c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.556+0000 7fb19d595640 1 -- 172.21.15.98:0/2444185885 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfe41f83f0 con 0x55bfe457f330 2024-05-09T09:02:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.556+0000 7fb19cd94640 1 --2- 172.21.15.98:0/2444185885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457fc50 0x55bfe4589de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.556+0000 7fb19d595640 1 --2- 172.21.15.98:0/2444185885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457f330 0x55bfe457f710 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fb194009d40 tx=0x7fb1940117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5edaeb7cf1ca7107 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.556+0000 7fb19dd96640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb1840a2d80 0x7fb1840a5240 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb190005e30 tx=0x7fb190009860 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.556+0000 7fb1837fe640 1 -- 172.21.15.98:0/2444185885 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194012650 con 0x55bfe457f330 2024-05-09T09:02:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.556+0000 7fb1837fe640 1 -- 172.21.15.98:0/2444185885 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb194009e30 con 0x55bfe457f330 2024-05-09T09:02:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.556+0000 7fb1837fe640 1 -- 172.21.15.98:0/2444185885 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19400fc30 con 0x55bfe457f330 2024-05-09T09:02:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.557+0000 7fb1a25e9740 1 -- 172.21.15.98:0/2444185885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457f330 msgr2=0x55bfe457f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.557+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/2444185885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457f330 0x55bfe457f710 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fb194009d40 tx=0x7fb1940117b0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.557+0000 7fb1a25e9740 1 -- 172.21.15.98:0/2444185885 shutdown_connections 2024-05-09T09:02:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.557+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/2444185885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe458a320 0x55bfe458c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.557+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/2444185885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457fc50 0x55bfe4589de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.557+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/2444185885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457f330 0x55bfe457f710 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.557+0000 7fb1a25e9740 1 -- 172.21.15.98:0/2444185885 >> 172.21.15.98:0/2444185885 conn(0x55bfe457b650 msgr2=0x55bfe457ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.557+0000 7fb1a25e9740 1 -- 172.21.15.98:0/2444185885 shutdown_connections 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.557+0000 7fb1a25e9740 1 -- 172.21.15.98:0/2444185885 wait complete. 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.557+0000 7fb1a25e9740 1 -- start start 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.557+0000 7fb1a25e9740 1 Processor -- start 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb1a25e9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 0x55bfe4587260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb1a25e9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457fc50 0x55bfe45877a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb1a25e9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe458a320 0x55bfe4587ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb1a25e9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bfe458d110 con 0x55bfe457fc50 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb1a25e9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bfe449fc40 con 0x55bfe458a320 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb1a25e9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55bfe458d290 con 0x55bfe457f330 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb1a25e9740 10 client.0 _ll_register_callbacks cb 0x55bfe41f7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb19cd94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457fc50 0x55bfe45877a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb19cd94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457fc50 0x55bfe45877a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51088/0 (socket says 172.21.15.98:51088) 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb19cd94640 1 -- 172.21.15.98:0/1730651920 learned_addr learned my addr 172.21.15.98:0/1730651920 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb19d595640 1 --2- 172.21.15.98:0/1730651920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 0x55bfe4587260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb19cd94640 1 -- 172.21.15.98:0/1730651920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 msgr2=0x55bfe4587260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb19cd94640 1 --2- 172.21.15.98:0/1730651920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 0x55bfe4587260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb19cd94640 1 -- 172.21.15.98:0/1730651920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe458a320 msgr2=0x55bfe4587ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb19cd94640 1 --2- 172.21.15.98:0/1730651920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe458a320 0x55bfe4587ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb19cd94640 1 -- 172.21.15.98:0/1730651920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfe42c27d0 con 0x55bfe457fc50 2024-05-09T09:02:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb19d595640 1 --2- 172.21.15.98:0/1730651920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 0x55bfe4587260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb19cd94640 1 --2- 172.21.15.98:0/1730651920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457fc50 0x55bfe45877a0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fb18809b4d0 tx=0x7fb18809b180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150ec28123cc938a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb180ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb180ff9640 1 -- 172.21.15.98:0/1730651920 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb188020050 con 0x55bfe457fc50 2024-05-09T09:02:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb1a25e9740 1 -- 172.21.15.98:0/1730651920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb194009e30 con 0x55bfe457fc50 2024-05-09T09:02:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb180ff9640 1 -- 172.21.15.98:0/1730651920 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7fb18809f5d0 con 0x55bfe457fc50 2024-05-09T09:02:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb180ff9640 1 -- 172.21.15.98:0/1730651920 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb188093170 con 0x55bfe457fc50 2024-05-09T09:02:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb1a25e9740 10 client.19122 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:32.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb1a25e9740 1 -- 172.21.15.98:0/1730651920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bfe4599330 con 0x55bfe457fc50 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.558+0000 7fb180ff9640 1 -- 172.21.15.98:0/1730651920 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb194009e30 con 0x55bfe457fc50 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.559+0000 7fb180ff9640 1 -- 172.21.15.98:0/1730651920 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb188084ce0 con 0x55bfe457fc50 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.559+0000 7fb1a25e9740 10 client.19122 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.559+0000 7fb1a25e9740 10 client.19122 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.559+0000 7fb1a25e9740 20 client.19122 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.559+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/1730651920 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55bfe459a840 0x55bfe459ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.559+0000 7fb1a25e9740 4 client.19122 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.559+0000 7fb1a25e9740 1 -- 172.21.15.98:0/1730651920 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfe44f2370 con 0x55bfe459a840 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.559+0000 7fb19d595640 1 --2- 172.21.15.98:0/1730651920 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55bfe459a840 0x55bfe459ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.559+0000 7fb19d595640 1 --2- 172.21.15.98:0/1730651920 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55bfe459a840 0x55bfe459ccd0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2b5b8463dd3420ae server_cookie=89fee8bed4cb1ba8 in_seq=0 out_seq=0 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.559+0000 7fb180ff9640 10 client.19122 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.560+0000 7fb180ff9640 1 -- 172.21.15.98:0/1730651920 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bfe44f2370 con 0x55bfe459a840 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.560+0000 7fb180ff9640 10 client.19122 handle_command_reply: tid=0 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.560+0000 7fb1a25e9740 1 client.19122 shutdown 2024-05-09T09:02:32.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.560+0000 7fb1a25e9740 1 -- 172.21.15.98:0/1730651920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457fc50 msgr2=0x55bfe45877a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.560+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/1730651920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457fc50 0x55bfe45877a0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fb18809b4d0 tx=0x7fb18809b180 comp rx=0 tx=0).stop 2024-05-09T09:02:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.560+0000 7fb1a25e9740 1 -- 172.21.15.98:0/1730651920 shutdown_connections 2024-05-09T09:02:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.560+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/1730651920 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55bfe459a840 0x55bfe459ccd0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.560+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/1730651920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe458a320 0x55bfe4587ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.560+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/1730651920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457fc50 0x55bfe45877a0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.560+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/1730651920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 0x55bfe4587260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.560+0000 7fb180ff9640 0 client.19122 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T09:02:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.560+0000 7fb1a25e9740 1 -- 172.21.15.98:0/1730651920 >> 172.21.15.98:0/1730651920 conn(0x55bfe457b650 msgr2=0x55bfe4581450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.560+0000 7fb1a25e9740 1 -- 172.21.15.98:0/1730651920 shutdown_connections 2024-05-09T09:02:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.560+0000 7fb1a25e9740 1 -- 172.21.15.98:0/1730651920 wait complete. 2024-05-09T09:02:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.561+0000 7fb1a25e9740 20 client.19122 trim_cache size 0 max 16384 2024-05-09T09:02:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.573+0000 7fb1a25e9740 1 Processor -- start 2024-05-09T09:02:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.573+0000 7fb1a25e9740 1 -- start start 2024-05-09T09:02:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb1a25e9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 0x55bfe4595fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb1a25e9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe457fc50 0x55bfe4581a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb1a25e9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe458a320 0x55bfe4586110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb1a25e9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1981b5470 con 0x55bfe458a320 2024-05-09T09:02:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb1a25e9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb1981b6fb0 con 0x55bfe457fc50 2024-05-09T09:02:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb1a25e9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb1981b5b40 con 0x55bfe457f330 2024-05-09T09:02:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb19dd96640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe458a320 0x55bfe4586110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb19dd96640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe458a320 0x55bfe4586110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51102/0 (socket says 172.21.15.98:51102) 2024-05-09T09:02:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb19dd96640 1 -- 172.21.15.98:0/3459801966 learned_addr learned my addr 172.21.15.98:0/3459801966 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb19cd94640 1 --2- 172.21.15.98:0/3459801966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe457fc50 0x55bfe4581a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb19dd96640 1 -- 172.21.15.98:0/3459801966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 msgr2=0x55bfe4595fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb19dd96640 1 --2- 172.21.15.98:0/3459801966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 0x55bfe4595fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb19dd96640 1 -- 172.21.15.98:0/3459801966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe457fc50 msgr2=0x55bfe4581a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb19dd96640 1 --2- 172.21.15.98:0/3459801966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe457fc50 0x55bfe4581a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb19dd96640 1 -- 172.21.15.98:0/3459801966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfe4580030 con 0x55bfe458a320 2024-05-09T09:02:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb19cd94640 1 --2- 172.21.15.98:0/3459801966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe457fc50 0x55bfe4581a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb19dd96640 1 --2- 172.21.15.98:0/3459801966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe458a320 0x55bfe4586110 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fb190035000 tx=0x7fb190043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2d748ac5ca4698 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb182ffd640 1 -- 172.21.15.98:0/3459801966 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb190034070 con 0x55bfe458a320 2024-05-09T09:02:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb182ffd640 1 -- 172.21.15.98:0/3459801966 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb19002f880 con 0x55bfe458a320 2024-05-09T09:02:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.574+0000 7fb182ffd640 1 -- 172.21.15.98:0/3459801966 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb190004390 con 0x55bfe458a320 2024-05-09T09:02:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.575+0000 7fb1a25e9740 1 -- 172.21.15.98:0/3459801966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe458a320 msgr2=0x55bfe4586110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.575+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/3459801966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe458a320 0x55bfe4586110 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fb190035000 tx=0x7fb190043000 comp rx=0 tx=0).stop 2024-05-09T09:02:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.575+0000 7fb1a25e9740 1 -- 172.21.15.98:0/3459801966 shutdown_connections 2024-05-09T09:02:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.575+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/3459801966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe458a320 0x55bfe4586110 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.575+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/3459801966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe457fc50 0x55bfe4581a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.575+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/3459801966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 0x55bfe4595fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.575+0000 7fb1a25e9740 1 -- 172.21.15.98:0/3459801966 >> 172.21.15.98:0/3459801966 conn(0x55bfe457b650 msgr2=0x55bfe4585a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.575+0000 7fb1a25e9740 1 -- 172.21.15.98:0/3459801966 shutdown_connections 2024-05-09T09:02:32.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.575+0000 7fb1a25e9740 1 -- 172.21.15.98:0/3459801966 wait complete. 2024-05-09T09:02:32.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.575+0000 7fb1a25e9740 1 -- start start 2024-05-09T09:02:32.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.575+0000 7fb1a25e9740 1 Processor -- start 2024-05-09T09:02:32.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb1a25e9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 0x55bfe4583e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb1a25e9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457fc50 0x55bfe45843d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb1a25e9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe458a320 0x55bfe4584910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb1a25e9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1981b68e0 con 0x55bfe457fc50 2024-05-09T09:02:32.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb1a25e9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb1981b6210 con 0x55bfe458a320 2024-05-09T09:02:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb1a25e9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb1981b3470 con 0x55bfe457f330 2024-05-09T09:02:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb1a25e9740 10 client.0 _ll_register_callbacks cb 0x7fb18800df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb19cd94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457fc50 0x55bfe45843d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb19cd94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457fc50 0x55bfe45843d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51114/0 (socket says 172.21.15.98:51114) 2024-05-09T09:02:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb19cd94640 1 -- 172.21.15.98:0/4115350824 learned_addr learned my addr 172.21.15.98:0/4115350824 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb19d595640 1 --2- 172.21.15.98:0/4115350824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 0x55bfe4583e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb19cd94640 1 -- 172.21.15.98:0/4115350824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 msgr2=0x55bfe4583e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb19cd94640 1 --2- 172.21.15.98:0/4115350824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 0x55bfe4583e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb19cd94640 1 -- 172.21.15.98:0/4115350824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe458a320 msgr2=0x55bfe4584910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb19cd94640 1 --2- 172.21.15.98:0/4115350824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe458a320 0x55bfe4584910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb19cd94640 1 -- 172.21.15.98:0/4115350824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfe45ffe10 con 0x55bfe457fc50 2024-05-09T09:02:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb19cd94640 1 --2- 172.21.15.98:0/4115350824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457fc50 0x55bfe45843d0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fb18809b1f0 tx=0x7fb188099a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2cf97610a7bbd5b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb180ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb180ff9640 1 -- 172.21.15.98:0/4115350824 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb188020070 con 0x55bfe457fc50 2024-05-09T09:02:32.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb180ff9640 1 -- 172.21.15.98:0/4115350824 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7fb18809ab90 con 0x55bfe457fc50 2024-05-09T09:02:32.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.576+0000 7fb1a25e9740 1 -- 172.21.15.98:0/4115350824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb19002f880 con 0x55bfe457fc50 2024-05-09T09:02:32.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.577+0000 7fb180ff9640 1 -- 172.21.15.98:0/4115350824 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb18809b310 con 0x55bfe457fc50 2024-05-09T09:02:32.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.577+0000 7fb1a25e9740 10 client.19128 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:32.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.577+0000 7fb1a25e9740 1 -- 172.21.15.98:0/4115350824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bfe4600fe0 con 0x55bfe457fc50 2024-05-09T09:02:32.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.577+0000 7fb180ff9640 1 -- 172.21.15.98:0/4115350824 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb19002f880 con 0x55bfe457fc50 2024-05-09T09:02:32.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.577+0000 7fb180ff9640 1 -- 172.21.15.98:0/4115350824 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb1880b4330 con 0x55bfe457fc50 2024-05-09T09:02:32.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.577+0000 7fb1a25e9740 10 client.19128 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:32.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.577+0000 7fb1a25e9740 10 client.19128 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.b 2024-05-09T09:02:32.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.577+0000 7fb1a25e9740 20 client.19128 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:32.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.577+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/4115350824 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55bfe4601650 0x55bfe4603a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.577+0000 7fb1a25e9740 4 client.19128 mds_command: new command op to 14667 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.577+0000 7fb1a25e9740 1 -- 172.21.15.98:0/4115350824 --> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bfe44ec6c0 con 0x55bfe4601650 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.577+0000 7fb19d595640 1 --2- 172.21.15.98:0/4115350824 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55bfe4601650 0x55bfe4603a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.577+0000 7fb180ff9640 10 client.19128 ms_handle_connect on v2:172.21.15.105:6838/1933710170 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.577+0000 7fb19d595640 1 --2- 172.21.15.98:0/4115350824 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55bfe4601650 0x55bfe4603a40 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fe19c83c24eeadc5 server_cookie=c868d7c714bc9295 in_seq=0 out_seq=0 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb180ff9640 1 -- 172.21.15.98:0/4115350824 <== mds.4 v2:172.21.15.105:6838/1933710170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bfe44ec6c0 con 0x55bfe4601650 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb180ff9640 10 client.19128 handle_command_reply: tid=0 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb1a25e9740 1 client.19128 shutdown 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb1a25e9740 1 -- 172.21.15.98:0/4115350824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457fc50 msgr2=0x55bfe45843d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/4115350824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457fc50 0x55bfe45843d0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fb18809b1f0 tx=0x7fb188099a50 comp rx=0 tx=0).stop 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb1a25e9740 1 -- 172.21.15.98:0/4115350824 shutdown_connections 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/4115350824 >> [v2:172.21.15.105:6838/1933710170,v1:172.21.15.105:6839/1933710170] conn(0x55bfe4601650 0x55bfe4603a40 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/4115350824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bfe458a320 0x55bfe4584910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/4115350824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bfe457fc50 0x55bfe45843d0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb180ff9640 0 client.19128 ms_handle_reset on v2:172.21.15.105:6838/1933710170 2024-05-09T09:02:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb1a25e9740 1 --2- 172.21.15.98:0/4115350824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bfe457f330 0x55bfe4583e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb1a25e9740 1 -- 172.21.15.98:0/4115350824 >> 172.21.15.98:0/4115350824 conn(0x55bfe457b650 msgr2=0x55bfe4587ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb1a25e9740 1 -- 172.21.15.98:0/4115350824 shutdown_connections 2024-05-09T09:02:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb1a25e9740 1 -- 172.21.15.98:0/4115350824 wait complete. 2024-05-09T09:02:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.578+0000 7fb1a25e9740 20 client.19128 trim_cache size 0 max 16384 2024-05-09T09:02:32.585 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:02:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.579+0000 7fb19f7f9640 1 -- 172.21.15.98:0/1447157802 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb1840a2d80 msgr2=0x7fb1840a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.579+0000 7fb19f7f9640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb1840a2d80 0x7fb1840a5240 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb190005e30 tx=0x7fb190009860 comp rx=0 tx=0).stop 2024-05-09T09:02:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.579+0000 7fb19f7f9640 1 -- 172.21.15.98:0/1447157802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1981011a0 msgr2=0x7fb19819ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.579+0000 7fb19f7f9640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1981011a0 0x7fb19819ec50 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fb18800d950 tx=0x7fb18800de10 comp rx=0 tx=0).stop 2024-05-09T09:02:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.579+0000 7fb19f7f9640 1 -- 172.21.15.98:0/1447157802 shutdown_connections 2024-05-09T09:02:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.579+0000 7fb19f7f9640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb1840a2d80 0x7fb1840a5240 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.579+0000 7fb19f7f9640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb198105c80 0x7fb1981a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.579+0000 7fb19f7f9640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1981011a0 0x7fb19819ec50 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.579+0000 7fb19f7f9640 1 --2- 172.21.15.98:0/1447157802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb198100880 0x7fb19819e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.579+0000 7fb19f7f9640 1 -- 172.21.15.98:0/1447157802 >> 172.21.15.98:0/1447157802 conn(0x7fb198076ab0 msgr2=0x7fb198103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.579+0000 7fb19f7f9640 1 -- 172.21.15.98:0/1447157802 shutdown_connections 2024-05-09T09:02:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.579+0000 7fb19f7f9640 1 -- 172.21.15.98:0/1447157802 wait complete. 2024-05-09T09:02:32.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T09:02:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.692+0000 7f37a56bd640 1 -- 172.21.15.98:0/3456284029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a0106810 msgr2=0x7f37a0106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.692+0000 7f37a56bd640 1 --2- 172.21.15.98:0/3456284029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a0106810 0x7f37a0106bf0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f3790009930 tx=0x7f379002a420 comp rx=0 tx=0).stop 2024-05-09T09:02:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.692+0000 7f37a56bd640 1 -- 172.21.15.98:0/3456284029 shutdown_connections 2024-05-09T09:02:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.692+0000 7f37a56bd640 1 --2- 172.21.15.98:0/3456284029 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37a01006f0 0x7f37a010e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.692+0000 7f37a56bd640 1 --2- 172.21.15.98:0/3456284029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37a00ffd30 0x7f37a01001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.692+0000 7f37a56bd640 1 --2- 172.21.15.98:0/3456284029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a0106810 0x7f37a0106bf0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.692+0000 7f37a56bd640 1 -- 172.21.15.98:0/3456284029 >> 172.21.15.98:0/3456284029 conn(0x7f37a0076cf0 msgr2=0x7f37a00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.692+0000 7f37a56bd640 1 -- 172.21.15.98:0/3456284029 shutdown_connections 2024-05-09T09:02:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.692+0000 7f37a56bd640 1 -- 172.21.15.98:0/3456284029 wait complete. 2024-05-09T09:02:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f37a56bd640 1 Processor -- start 2024-05-09T09:02:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f37a56bd640 1 -- start start 2024-05-09T09:02:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f37a56bd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a00ffd30 0x7f37a01981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f37a56bd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37a01006f0 0x7f37a0198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f37a56bd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37a0106810 0x7f37a019ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f37a56bd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37a0111cf0 con 0x7f37a00ffd30 2024-05-09T09:02:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f37a56bd640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f37a0111b70 con 0x7f37a0106810 2024-05-09T09:02:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f37a56bd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f37a0111e70 con 0x7f37a01006f0 2024-05-09T09:02:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f379effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a00ffd30 0x7f37a01981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f379effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a00ffd30 0x7f37a01981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51132/0 (socket says 172.21.15.98:51132) 2024-05-09T09:02:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f379effd640 1 -- 172.21.15.98:0/2663754634 learned_addr learned my addr 172.21.15.98:0/2663754634 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f379f7fe640 1 --2- 172.21.15.98:0/2663754634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37a0106810 0x7f37a019ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f379effd640 1 -- 172.21.15.98:0/2663754634 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37a01006f0 msgr2=0x7f37a0198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f379effd640 1 --2- 172.21.15.98:0/2663754634 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37a01006f0 0x7f37a0198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f379effd640 1 -- 172.21.15.98:0/2663754634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37a0106810 msgr2=0x7f37a019ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f379effd640 1 --2- 172.21.15.98:0/2663754634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37a0106810 0x7f37a019ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.693+0000 7f379effd640 1 -- 172.21.15.98:0/2663754634 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a019d1c0 con 0x7f37a00ffd30 2024-05-09T09:02:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.694+0000 7f379effd640 1 --2- 172.21.15.98:0/2663754634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a00ffd30 0x7f37a01981d0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f3790009230 tx=0x7f3790004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.694+0000 7f379cff9640 1 -- 172.21.15.98:0/2663754634 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379003f030 con 0x7f37a00ffd30 2024-05-09T09:02:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.694+0000 7f379cff9640 1 -- 172.21.15.98:0/2663754634 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3790003ad0 con 0x7f37a00ffd30 2024-05-09T09:02:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.694+0000 7f37a56bd640 1 -- 172.21.15.98:0/2663754634 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37a019d450 con 0x7f37a00ffd30 2024-05-09T09:02:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.694+0000 7f379cff9640 1 -- 172.21.15.98:0/2663754634 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3790035070 con 0x7f37a00ffd30 2024-05-09T09:02:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.694+0000 7f37a56bd640 1 -- 172.21.15.98:0/2663754634 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37a01a4d40 con 0x7f37a00ffd30 2024-05-09T09:02:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.697+0000 7f37a83d0740 1 Processor -- start 2024-05-09T09:02:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.697+0000 7f379cff9640 1 -- 172.21.15.98:0/2663754634 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3790042030 con 0x7f37a00ffd30 2024-05-09T09:02:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.697+0000 7f37a83d0740 1 -- start start 2024-05-09T09:02:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.697+0000 7f37a83d0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5f330 0x5632ced5f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.697+0000 7f37a83d0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced5fc50 0x5632ced69de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.697+0000 7f37a83d0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 0x5632ced6c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.697+0000 7f37a83d0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5632cecc4c40 con 0x5632ced5f330 2024-05-09T09:02:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.697+0000 7f37a83d0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5632cec6c580 con 0x5632ced5fc50 2024-05-09T09:02:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.697+0000 7f37a83d0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5632cecc0c00 con 0x5632ced6a320 2024-05-09T09:02:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.697+0000 7f379f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5f330 0x5632ced5f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.697+0000 7f379f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5f330 0x5632ced5f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51140/0 (socket says 172.21.15.98:51140) 2024-05-09T09:02:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.697+0000 7f379f7fe640 1 -- 172.21.15.98:0/1494852240 learned_addr learned my addr 172.21.15.98:0/1494852240 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.697+0000 7f379effd640 1 --2- 172.21.15.98:0/1494852240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced5fc50 0x5632ced69de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.697+0000 7f3797fff640 1 --2- 172.21.15.98:0/1494852240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 0x5632ced6c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.698+0000 7f379f7fe640 1 -- 172.21.15.98:0/1494852240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 msgr2=0x5632ced6c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.698+0000 7f379f7fe640 1 --2- 172.21.15.98:0/1494852240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 0x5632ced6c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.698+0000 7f379f7fe640 1 -- 172.21.15.98:0/1494852240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced5fc50 msgr2=0x5632ced69de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.698+0000 7f379f7fe640 1 --2- 172.21.15.98:0/1494852240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced5fc50 0x5632ced69de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.698+0000 7f379f7fe640 1 -- 172.21.15.98:0/1494852240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632ce9d83f0 con 0x5632ced5f330 2024-05-09T09:02:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.698+0000 7f3797fff640 1 --2- 172.21.15.98:0/1494852240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 0x5632ced6c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.698+0000 7f379cff9640 1 --2- 172.21.15.98:0/2663754634 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f378c0a2d00 0x7f378c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.698+0000 7f379f7fe640 1 --2- 172.21.15.98:0/1494852240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5f330 0x5632ced5f710 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f3798009eb0 tx=0x7f379800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0294dad8eb40522 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.698+0000 7f379effd640 1 --2- 172.21.15.98:0/2663754634 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f378c0a2d00 0x7f378c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.698+0000 7f37957fa640 1 -- 172.21.15.98:0/1494852240 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3798011640 con 0x5632ced5f330 2024-05-09T09:02:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.698+0000 7f37957fa640 1 -- 172.21.15.98:0/1494852240 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3798011cc0 con 0x5632ced5f330 2024-05-09T09:02:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.698+0000 7f379cff9640 1 -- 172.21.15.98:0/2663754634 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f37900eff50 con 0x7f37a00ffd30 2024-05-09T09:02:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.698+0000 7f37957fa640 1 -- 172.21.15.98:0/1494852240 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379801bd10 con 0x5632ced5f330 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 -- 172.21.15.98:0/1494852240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5f330 msgr2=0x5632ced5f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f379effd640 1 --2- 172.21.15.98:0/2663754634 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f378c0a2d00 0x7f378c0a51c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f37900ed340 tx=0x7f37900f3b80 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 --2- 172.21.15.98:0/1494852240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5f330 0x5632ced5f710 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f3798009eb0 tx=0x7f379800f730 comp rx=0 tx=0).stop 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 -- 172.21.15.98:0/1494852240 shutdown_connections 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 --2- 172.21.15.98:0/1494852240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 0x5632ced6c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 --2- 172.21.15.98:0/1494852240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced5fc50 0x5632ced69de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 --2- 172.21.15.98:0/1494852240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5f330 0x5632ced5f710 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 -- 172.21.15.98:0/1494852240 >> 172.21.15.98:0/1494852240 conn(0x5632ced5b650 msgr2=0x5632ced5ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 -- 172.21.15.98:0/1494852240 shutdown_connections 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 -- 172.21.15.98:0/1494852240 wait complete. 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 -- start start 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 Processor -- start 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced5f330 0x5632ced67250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5fc50 0x5632ced67790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced6a320 0x5632ced67cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5632ced6d110 con 0x5632ced5fc50 2024-05-09T09:02:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5632cec7fc40 con 0x5632ced6a320 2024-05-09T09:02:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.699+0000 7f37a83d0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5632ced6d290 con 0x5632ced5f330 2024-05-09T09:02:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f37a83d0740 10 client.0 _ll_register_callbacks cb 0x5632ce9d7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f3797fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5fc50 0x5632ced67790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f3797fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5fc50 0x5632ced67790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51146/0 (socket says 172.21.15.98:51146) 2024-05-09T09:02:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f3797fff640 1 -- 172.21.15.98:0/2373156928 learned_addr learned my addr 172.21.15.98:0/2373156928 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f379f7fe640 1 --2- 172.21.15.98:0/2373156928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced5f330 0x5632ced67250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f379f7fe640 1 --2- 172.21.15.98:0/2373156928 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced6a320 0x5632ced67cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f3797fff640 1 -- 172.21.15.98:0/2373156928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced5f330 msgr2=0x5632ced67250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f3797fff640 1 --2- 172.21.15.98:0/2373156928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced5f330 0x5632ced67250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f3797fff640 1 -- 172.21.15.98:0/2373156928 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced6a320 msgr2=0x5632ced67cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f3797fff640 1 --2- 172.21.15.98:0/2373156928 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced6a320 0x5632ced67cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f3797fff640 1 -- 172.21.15.98:0/2373156928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632ceaa27d0 con 0x5632ced5fc50 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f379f7fe640 1 --2- 172.21.15.98:0/2373156928 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced6a320 0x5632ced67cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f3797fff640 1 --2- 172.21.15.98:0/2373156928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5fc50 0x5632ced67790 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f378800e650 tx=0x7f378800eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d4914db3cda824 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f37867fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f37867fc640 1 -- 172.21.15.98:0/2373156928 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378800aa30 con 0x5632ced5fc50 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f37867fc640 1 -- 172.21.15.98:0/2373156928 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f378800abd0 con 0x5632ced5fc50 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f37a83d0740 1 -- 172.21.15.98:0/2373156928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3798011cc0 con 0x5632ced5fc50 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f37867fc640 1 -- 172.21.15.98:0/2373156928 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f378800add0 con 0x5632ced5fc50 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f37a83d0740 10 client.19140 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f37a83d0740 1 -- 172.21.15.98:0/2373156928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5632ced79340 con 0x5632ced5fc50 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.700+0000 7f37867fc640 1 -- 172.21.15.98:0/2373156928 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3798011cc0 con 0x5632ced5fc50 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.701+0000 7f37867fc640 1 -- 172.21.15.98:0/2373156928 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3788010730 con 0x5632ced5fc50 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.701+0000 7f37a83d0740 10 client.19140 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.701+0000 7f37a83d0740 10 client.19140 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.701+0000 7f37a83d0740 20 client.19140 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.701+0000 7f37a83d0740 1 --2- 172.21.15.98:0/2373156928 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5632ced7a800 0x5632ced7cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.701+0000 7f37a83d0740 4 client.19140 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.701+0000 7f37a83d0740 1 -- 172.21.15.98:0/2373156928 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632cecd2370 con 0x5632ced7a800 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.701+0000 7f379f7fe640 1 --2- 172.21.15.98:0/2373156928 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5632ced7a800 0x5632ced7cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.701+0000 7f37867fc640 10 client.19140 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.701+0000 7f379f7fe640 1 --2- 172.21.15.98:0/2373156928 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5632ced7a800 0x5632ced7cc20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=26baa34bb3838159 server_cookie=163528471efa51f4 in_seq=0 out_seq=0 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37867fc640 1 -- 172.21.15.98:0/2373156928 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632cecd2370 con 0x5632ced7a800 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37867fc640 10 client.19140 handle_command_reply: tid=0 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37a83d0740 1 client.19140 shutdown 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37a83d0740 1 -- 172.21.15.98:0/2373156928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5fc50 msgr2=0x5632ced67790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37a83d0740 1 --2- 172.21.15.98:0/2373156928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5fc50 0x5632ced67790 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f378800e650 tx=0x7f378800eb10 comp rx=0 tx=0).stop 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37a83d0740 1 -- 172.21.15.98:0/2373156928 shutdown_connections 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37a83d0740 1 --2- 172.21.15.98:0/2373156928 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5632ced7a800 0x5632ced7cc20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37a83d0740 1 --2- 172.21.15.98:0/2373156928 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced6a320 0x5632ced67cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37a83d0740 1 --2- 172.21.15.98:0/2373156928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5fc50 0x5632ced67790 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37867fc640 0 client.19140 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T09:02:32.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37a83d0740 1 --2- 172.21.15.98:0/2373156928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced5f330 0x5632ced67250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37a83d0740 1 -- 172.21.15.98:0/2373156928 >> 172.21.15.98:0/2373156928 conn(0x5632ced5b650 msgr2=0x5632ced61350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37a83d0740 1 -- 172.21.15.98:0/2373156928 shutdown_connections 2024-05-09T09:02:32.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37a83d0740 1 -- 172.21.15.98:0/2373156928 wait complete. 2024-05-09T09:02:32.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.702+0000 7f37a83d0740 20 client.19140 trim_cache size 0 max 16384 2024-05-09T09:02:32.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.715+0000 7f37a83d0740 1 Processor -- start 2024-05-09T09:02:32.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f37a83d0740 1 -- start start 2024-05-09T09:02:32.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f37a83d0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced5f330 0x5632ced76000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f37a83d0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5fc50 0x5632ced663b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f37a83d0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 0x5632ced67af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f37a83d0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37a01aee10 con 0x5632ced5fc50 2024-05-09T09:02:32.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f37a83d0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f37a01b0950 con 0x5632ced5f330 2024-05-09T09:02:32.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f37a83d0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f37a01af4e0 con 0x5632ced6a320 2024-05-09T09:02:32.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f3797fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5fc50 0x5632ced663b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f3797fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5fc50 0x5632ced663b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51150/0 (socket says 172.21.15.98:51150) 2024-05-09T09:02:32.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f3797fff640 1 -- 172.21.15.98:0/4008783833 learned_addr learned my addr 172.21.15.98:0/4008783833 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f379f7fe640 1 --2- 172.21.15.98:0/4008783833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 0x5632ced67af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f3797fff640 1 -- 172.21.15.98:0/4008783833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 msgr2=0x5632ced67af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f3797fff640 1 --2- 172.21.15.98:0/4008783833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 0x5632ced67af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f3797fff640 1 -- 172.21.15.98:0/4008783833 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced5f330 msgr2=0x5632ced76000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f3797fff640 1 --2- 172.21.15.98:0/4008783833 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced5f330 0x5632ced76000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f3797fff640 1 -- 172.21.15.98:0/4008783833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632ced4cd20 con 0x5632ced5fc50 2024-05-09T09:02:32.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f379f7fe640 1 --2- 172.21.15.98:0/4008783833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 0x5632ced67af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:32.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.716+0000 7f3797fff640 1 --2- 172.21.15.98:0/4008783833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5fc50 0x5632ced663b0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f3788000c00 tx=0x7f378800afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484375bc68c0ceb6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.717+0000 7f3794ff9640 1 -- 172.21.15.98:0/4008783833 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378802e070 con 0x5632ced5fc50 2024-05-09T09:02:32.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.717+0000 7f3794ff9640 1 -- 172.21.15.98:0/4008783833 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3788010910 con 0x5632ced5fc50 2024-05-09T09:02:32.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.717+0000 7f3794ff9640 1 -- 172.21.15.98:0/4008783833 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378801a860 con 0x5632ced5fc50 2024-05-09T09:02:32.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.717+0000 7f37a83d0740 1 -- 172.21.15.98:0/4008783833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5fc50 msgr2=0x5632ced663b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.717+0000 7f37a83d0740 1 --2- 172.21.15.98:0/4008783833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5fc50 0x5632ced663b0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f3788000c00 tx=0x7f378800afd0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.717+0000 7f37a83d0740 1 -- 172.21.15.98:0/4008783833 shutdown_connections 2024-05-09T09:02:32.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.717+0000 7f37a83d0740 1 --2- 172.21.15.98:0/4008783833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 0x5632ced67af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.717+0000 7f37a83d0740 1 --2- 172.21.15.98:0/4008783833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5fc50 0x5632ced663b0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.717+0000 7f37a83d0740 1 --2- 172.21.15.98:0/4008783833 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced5f330 0x5632ced76000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.717+0000 7f37a83d0740 1 -- 172.21.15.98:0/4008783833 >> 172.21.15.98:0/4008783833 conn(0x5632ced5b650 msgr2=0x5632ced59f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.717+0000 7f37a83d0740 1 -- 172.21.15.98:0/4008783833 shutdown_connections 2024-05-09T09:02:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.717+0000 7f37a83d0740 1 -- 172.21.15.98:0/4008783833 wait complete. 2024-05-09T09:02:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f37a83d0740 1 -- start start 2024-05-09T09:02:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f37a83d0740 1 Processor -- start 2024-05-09T09:02:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f37a83d0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5f330 0x5632ced72980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f37a83d0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced5fc50 0x5632ced6dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f37a83d0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 0x5632ced6e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f37a83d0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37a01b0280 con 0x5632ced5f330 2024-05-09T09:02:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f37a83d0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f37a01afbb0 con 0x5632ced5fc50 2024-05-09T09:02:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f37a83d0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f37a01ace10 con 0x5632ced6a320 2024-05-09T09:02:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f37a83d0740 10 client.0 _ll_register_callbacks cb 0x7f378800c310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:32.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f379f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5f330 0x5632ced72980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f379f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5f330 0x5632ced72980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51154/0 (socket says 172.21.15.98:51154) 2024-05-09T09:02:32.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f379f7fe640 1 -- 172.21.15.98:0/1501869959 learned_addr learned my addr 172.21.15.98:0/1501869959 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f3797fff640 1 --2- 172.21.15.98:0/1501869959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced5fc50 0x5632ced6dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f379f7fe640 1 -- 172.21.15.98:0/1501869959 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 msgr2=0x5632ced6e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f379f7fe640 1 --2- 172.21.15.98:0/1501869959 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 0x5632ced6e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f379f7fe640 1 -- 172.21.15.98:0/1501869959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced5fc50 msgr2=0x5632ced6dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f379f7fe640 1 --2- 172.21.15.98:0/1501869959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced5fc50 0x5632ced6dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f379f7fe640 1 -- 172.21.15.98:0/1501869959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632ced6eb10 con 0x5632ced5f330 2024-05-09T09:02:32.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f379f7fe640 1 --2- 172.21.15.98:0/1501869959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5f330 0x5632ced72980 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f3798009e80 tx=0x7f3798005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda4e905f4ece193 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.718+0000 7f37867fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:32.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f37867fc640 1 -- 172.21.15.98:0/1501869959 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3798009950 con 0x5632ced5f330 2024-05-09T09:02:32.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f37867fc640 1 -- 172.21.15.98:0/1501869959 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f3798009af0 con 0x5632ced5f330 2024-05-09T09:02:32.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f37a83d0740 1 -- 172.21.15.98:0/1501869959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3788010910 con 0x5632ced5f330 2024-05-09T09:02:32.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f37867fc640 1 -- 172.21.15.98:0/1501869959 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3798032050 con 0x5632ced5f330 2024-05-09T09:02:32.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f37a83d0740 10 client.19146 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:32.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f37a83d0740 1 -- 172.21.15.98:0/1501869959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5632cede0e00 con 0x5632ced5f330 2024-05-09T09:02:32.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f37867fc640 1 -- 172.21.15.98:0/1501869959 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3788010910 con 0x5632ced5f330 2024-05-09T09:02:32.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f37867fc640 1 -- 172.21.15.98:0/1501869959 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3798003a90 con 0x5632ced5f330 2024-05-09T09:02:32.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f37a83d0740 10 client.19146 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:32.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f37a83d0740 10 client.19146 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-05-09T09:02:32.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f37a83d0740 20 client.19146 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:32.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f37a83d0740 1 --2- 172.21.15.98:0/1501869959 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5632cede12a0 0x5632cede3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f37a83d0740 4 client.19146 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:02:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f37a83d0740 1 -- 172.21.15.98:0/1501869959 --> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5632ceccc6c0 con 0x5632cede12a0 2024-05-09T09:02:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.719+0000 7f3797fff640 1 --2- 172.21.15.98:0/1501869959 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5632cede12a0 0x5632cede3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37867fc640 10 client.19146 ms_handle_connect on v2:172.21.15.142:6832/606414638 2024-05-09T09:02:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f3797fff640 1 --2- 172.21.15.98:0/1501869959 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5632cede12a0 0x5632cede3690 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e61296ca048cb286 server_cookie=b6036ac85ca43f59 in_seq=0 out_seq=0 2024-05-09T09:02:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37867fc640 1 -- 172.21.15.98:0/1501869959 <== mds.3 v2:172.21.15.142:6832/606414638 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5632ceccc6c0 con 0x5632cede12a0 2024-05-09T09:02:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37867fc640 10 client.19146 handle_command_reply: tid=0 2024-05-09T09:02:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37a83d0740 1 client.19146 shutdown 2024-05-09T09:02:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37a83d0740 1 -- 172.21.15.98:0/1501869959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5f330 msgr2=0x5632ced72980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37a83d0740 1 --2- 172.21.15.98:0/1501869959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5f330 0x5632ced72980 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f3798009e80 tx=0x7f3798005c50 comp rx=0 tx=0).stop 2024-05-09T09:02:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37a83d0740 1 -- 172.21.15.98:0/1501869959 shutdown_connections 2024-05-09T09:02:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37a83d0740 1 --2- 172.21.15.98:0/1501869959 >> [v2:172.21.15.142:6832/606414638,v1:172.21.15.142:6833/606414638] conn(0x5632cede12a0 0x5632cede3690 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37a83d0740 1 --2- 172.21.15.98:0/1501869959 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ced6a320 0x5632ced6e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37867fc640 0 client.19146 ms_handle_reset on v2:172.21.15.142:6832/606414638 2024-05-09T09:02:32.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37a83d0740 1 --2- 172.21.15.98:0/1501869959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632ced5fc50 0x5632ced6dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37a83d0740 1 --2- 172.21.15.98:0/1501869959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632ced5f330 0x5632ced72980 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37a83d0740 1 -- 172.21.15.98:0/1501869959 >> 172.21.15.98:0/1501869959 conn(0x5632ced5b650 msgr2=0x5632ced60a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37a83d0740 1 -- 172.21.15.98:0/1501869959 shutdown_connections 2024-05-09T09:02:32.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37a83d0740 1 -- 172.21.15.98:0/1501869959 wait complete. 2024-05-09T09:02:32.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.720+0000 7f37a83d0740 20 client.19146 trim_cache size 0 max 16384 2024-05-09T09:02:32.727 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:02:32.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.721+0000 7f37a56bd640 1 -- 172.21.15.98:0/2663754634 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f378c0a2d00 msgr2=0x7f378c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.721+0000 7f37a56bd640 1 --2- 172.21.15.98:0/2663754634 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f378c0a2d00 0x7f378c0a51c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f37900ed340 tx=0x7f37900f3b80 comp rx=0 tx=0).stop 2024-05-09T09:02:32.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.721+0000 7f37a56bd640 1 -- 172.21.15.98:0/2663754634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a00ffd30 msgr2=0x7f37a01981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.721+0000 7f37a56bd640 1 --2- 172.21.15.98:0/2663754634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a00ffd30 0x7f37a01981d0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f3790009230 tx=0x7f3790004270 comp rx=0 tx=0).stop 2024-05-09T09:02:32.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.721+0000 7f37a56bd640 1 -- 172.21.15.98:0/2663754634 shutdown_connections 2024-05-09T09:02:32.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.721+0000 7f37a56bd640 1 --2- 172.21.15.98:0/2663754634 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f378c0a2d00 0x7f378c0a51c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.721+0000 7f37a56bd640 1 --2- 172.21.15.98:0/2663754634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37a0106810 0x7f37a019ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.721+0000 7f37a56bd640 1 --2- 172.21.15.98:0/2663754634 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37a01006f0 0x7f37a0198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.721+0000 7f37a56bd640 1 --2- 172.21.15.98:0/2663754634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a00ffd30 0x7f37a01981d0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.721+0000 7f37a56bd640 1 -- 172.21.15.98:0/2663754634 >> 172.21.15.98:0/2663754634 conn(0x7f37a0076cf0 msgr2=0x7f37a01049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.721+0000 7f37a56bd640 1 -- 172.21.15.98:0/2663754634 shutdown_connections 2024-05-09T09:02:32.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.721+0000 7f37a56bd640 1 -- 172.21.15.98:0/2663754634 wait complete. 2024-05-09T09:02:32.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T09:02:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.836+0000 7f6af4170640 1 -- 172.21.15.98:0/1242768131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec1017f0 msgr2=0x7f6aec101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.836+0000 7f6af4170640 1 --2- 172.21.15.98:0/1242768131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec1017f0 0x7f6aec101c70 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0009cd0 tx=0x7f6ae002a8e0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.836+0000 7f6af4170640 1 -- 172.21.15.98:0/1242768131 shutdown_connections 2024-05-09T09:02:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.836+0000 7f6af4170640 1 --2- 172.21.15.98:0/1242768131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6aec1021b0 0x7f6aec10e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.836+0000 7f6af4170640 1 --2- 172.21.15.98:0/1242768131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec1017f0 0x7f6aec101c70 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.836+0000 7f6af4170640 1 --2- 172.21.15.98:0/1242768131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6aec107800 0x7f6aec107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.836+0000 7f6af4170640 1 -- 172.21.15.98:0/1242768131 >> 172.21.15.98:0/1242768131 conn(0x7f6aec0fd2a0 msgr2=0x7f6aec0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.836+0000 7f6af4170640 1 -- 172.21.15.98:0/1242768131 shutdown_connections 2024-05-09T09:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.836+0000 7f6af4170640 1 -- 172.21.15.98:0/1242768131 wait complete. 2024-05-09T09:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.836+0000 7f6af4170640 1 Processor -- start 2024-05-09T09:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af4170640 1 -- start start 2024-05-09T09:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af4170640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6aec1017f0 0x7f6aec1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af4170640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec1021b0 0x7f6aec198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af4170640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6aec107800 0x7f6aec19ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af4170640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec111c10 con 0x7f6aec1021b0 2024-05-09T09:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af4170640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec111a90 con 0x7f6aec107800 2024-05-09T09:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af4170640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec111d90 con 0x7f6aec1017f0 2024-05-09T09:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af170b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec1021b0 0x7f6aec198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af170b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec1021b0 0x7f6aec198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51170/0 (socket says 172.21.15.98:51170) 2024-05-09T09:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af170b640 1 -- 172.21.15.98:0/2609418289 learned_addr learned my addr 172.21.15.98:0/2609418289 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af1f0c640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6aec1017f0 0x7f6aec1981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af270d640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6aec107800 0x7f6aec19ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af170b640 1 -- 172.21.15.98:0/2609418289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6aec1017f0 msgr2=0x7f6aec1981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af170b640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6aec1017f0 0x7f6aec1981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af170b640 1 -- 172.21.15.98:0/2609418289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6aec107800 msgr2=0x7f6aec19ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af170b640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6aec107800 0x7f6aec19ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af170b640 1 -- 172.21.15.98:0/2609418289 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aec19d1c0 con 0x7f6aec1021b0 2024-05-09T09:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af270d640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6aec107800 0x7f6aec19ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.837+0000 7f6af170b640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec1021b0 0x7f6aec198710 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f6ae002ae00 tx=0x7f6ae0003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.838+0000 7f6aeaffd640 1 -- 172.21.15.98:0/2609418289 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0004040 con 0x7f6aec1021b0 2024-05-09T09:02:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.838+0000 7f6aeaffd640 1 -- 172.21.15.98:0/2609418289 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ae00046c0 con 0x7f6aec1021b0 2024-05-09T09:02:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.838+0000 7f6af4170640 1 -- 172.21.15.98:0/2609418289 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aec19d450 con 0x7f6aec1021b0 2024-05-09T09:02:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.838+0000 7f6aeaffd640 1 -- 172.21.15.98:0/2609418289 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0030e70 con 0x7f6aec1021b0 2024-05-09T09:02:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.838+0000 7f6af4170640 1 -- 172.21.15.98:0/2609418289 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aec1a4d40 con 0x7f6aec1021b0 2024-05-09T09:02:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.840+0000 7f6aeaffd640 1 -- 172.21.15.98:0/2609418289 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6ae00305a0 con 0x7f6aec1021b0 2024-05-09T09:02:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.840+0000 7f6af6e7d740 1 Processor -- start 2024-05-09T09:02:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.841+0000 7f6af6e7d740 1 -- start start 2024-05-09T09:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.841+0000 7f6aeaffd640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6ad40a2d00 0x7f6ad40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.841+0000 7f6af6e7d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f8a330 0x556573f8a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.841+0000 7f6af6e7d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8ac50 0x556573f94de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.841+0000 7f6af6e7d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f95320 0x556573f97710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.841+0000 7f6af6e7d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556573eefc40 con 0x556573f95320 2024-05-09T09:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.841+0000 7f6af6e7d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556573e97580 con 0x556573f8a330 2024-05-09T09:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.841+0000 7f6af6e7d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556573eebc00 con 0x556573f8ac50 2024-05-09T09:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.841+0000 7f6aeaffd640 1 -- 172.21.15.98:0/2609418289 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f6ae002e030 con 0x7f6aec1021b0 2024-05-09T09:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.841+0000 7f6af270d640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6ad40a2d00 0x7f6ad40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.841+0000 7f6af270d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f95320 0x556573f97710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.841+0000 7f6af270d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f95320 0x556573f97710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51184/0 (socket says 172.21.15.98:51184) 2024-05-09T09:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.841+0000 7f6af270d640 1 -- 172.21.15.98:0/1543241752 learned_addr learned my addr 172.21.15.98:0/1543241752 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.841+0000 7f6af1f0c640 1 --2- 172.21.15.98:0/1543241752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f8a330 0x556573f8a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.842+0000 7f6af170b640 1 --2- 172.21.15.98:0/1543241752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8ac50 0x556573f94de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.842+0000 7f6af270d640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6ad40a2d00 0x7f6ad40a51c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6adc015470 tx=0x7f6adc015870 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.842+0000 7f6af270d640 1 -- 172.21.15.98:0/1543241752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8ac50 msgr2=0x556573f94de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.842+0000 7f6af270d640 1 --2- 172.21.15.98:0/1543241752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8ac50 0x556573f94de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.842+0000 7f6af270d640 1 -- 172.21.15.98:0/1543241752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f8a330 msgr2=0x556573f8a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.842+0000 7f6af270d640 1 --2- 172.21.15.98:0/1543241752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f8a330 0x556573f8a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.842+0000 7f6af270d640 1 -- 172.21.15.98:0/1543241752 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556573c033f0 con 0x556573f95320 2024-05-09T09:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.842+0000 7f6af270d640 1 --2- 172.21.15.98:0/1543241752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f95320 0x556573f97710 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f6adc012650 tx=0x7f6adc01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1237061cec08efb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.842+0000 7f6ad3fff640 1 -- 172.21.15.98:0/1543241752 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc024070 con 0x556573f95320 2024-05-09T09:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.842+0000 7f6ad3fff640 1 -- 172.21.15.98:0/1543241752 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6adc01f930 con 0x556573f95320 2024-05-09T09:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.842+0000 7f6ad3fff640 1 -- 172.21.15.98:0/1543241752 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc013940 con 0x556573f95320 2024-05-09T09:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.843+0000 7f6af6e7d740 1 -- 172.21.15.98:0/1543241752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f95320 msgr2=0x556573f97710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.843+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/1543241752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f95320 0x556573f97710 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f6adc012650 tx=0x7f6adc01c000 comp rx=0 tx=0).stop 2024-05-09T09:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.843+0000 7f6af6e7d740 1 -- 172.21.15.98:0/1543241752 shutdown_connections 2024-05-09T09:02:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.843+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/1543241752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f95320 0x556573f97710 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.843+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/1543241752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8ac50 0x556573f94de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.843+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/1543241752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f8a330 0x556573f8a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.843+0000 7f6af6e7d740 1 -- 172.21.15.98:0/1543241752 >> 172.21.15.98:0/1543241752 conn(0x556573f86650 msgr2=0x556573f86a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.843+0000 7f6af6e7d740 1 -- 172.21.15.98:0/1543241752 shutdown_connections 2024-05-09T09:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.843+0000 7f6af6e7d740 1 -- 172.21.15.98:0/1543241752 wait complete. 2024-05-09T09:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.844+0000 7f6af6e7d740 1 -- start start 2024-05-09T09:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.844+0000 7f6af6e7d740 1 Processor -- start 2024-05-09T09:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.844+0000 7f6af6e7d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8a330 0x556573f92280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.844+0000 7f6af6e7d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f8ac50 0x556573f927c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.844+0000 7f6af6e7d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f95320 0x556573f92d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.844+0000 7f6af6e7d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556573f980b0 con 0x556573f95320 2024-05-09T09:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.844+0000 7f6af6e7d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556573eaac40 con 0x556573f8ac50 2024-05-09T09:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.844+0000 7f6af6e7d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556573f98230 con 0x556573f8a330 2024-05-09T09:02:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.844+0000 7f6af6e7d740 10 client.0 _ll_register_callbacks cb 0x556573c02be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.844+0000 7f6af270d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f95320 0x556573f92d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.844+0000 7f6af270d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f95320 0x556573f92d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51198/0 (socket says 172.21.15.98:51198) 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6af270d640 1 -- 172.21.15.98:0/1955729961 learned_addr learned my addr 172.21.15.98:0/1955729961 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6af1f0c640 1 --2- 172.21.15.98:0/1955729961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8a330 0x556573f92280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6af170b640 1 --2- 172.21.15.98:0/1955729961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f8ac50 0x556573f927c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6af270d640 1 -- 172.21.15.98:0/1955729961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8a330 msgr2=0x556573f92280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6af270d640 1 --2- 172.21.15.98:0/1955729961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8a330 0x556573f92280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6af270d640 1 -- 172.21.15.98:0/1955729961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f8ac50 msgr2=0x556573f927c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6af270d640 1 --2- 172.21.15.98:0/1955729961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f8ac50 0x556573f927c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6af270d640 1 -- 172.21.15.98:0/1955729961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556573ccd7d0 con 0x556573f95320 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6af270d640 1 --2- 172.21.15.98:0/1955729961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f95320 0x556573f92d00 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f6adc022040 tx=0x7f6adc005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8dfe913850124f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6ad17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6ad17fa640 1 -- 172.21.15.98:0/1955729961 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc02d060 con 0x556573f95320 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6ad17fa640 1 -- 172.21.15.98:0/1955729961 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f6adc01f660 con 0x556573f95320 2024-05-09T09:02:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6af6e7d740 1 -- 172.21.15.98:0/1955729961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6adc01f930 con 0x556573f95320 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6ad17fa640 1 -- 172.21.15.98:0/1955729961 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6adc034050 con 0x556573f95320 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6af6e7d740 10 client.19158 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.845+0000 7f6af6e7d740 1 -- 172.21.15.98:0/1955729961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556573fa42d0 con 0x556573f95320 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.846+0000 7f6ad17fa640 1 -- 172.21.15.98:0/1955729961 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6adc01f930 con 0x556573f95320 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.846+0000 7f6ad17fa640 1 -- 172.21.15.98:0/1955729961 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f6adc013700 con 0x556573f95320 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.846+0000 7f6af6e7d740 10 client.19158 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.846+0000 7f6af6e7d740 10 client.19158 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.846+0000 7f6af6e7d740 20 client.19158 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.846+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/1955729961 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x556573fa5790 0x556573fa7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.846+0000 7f6af6e7d740 4 client.19158 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.846+0000 7f6af6e7d740 1 -- 172.21.15.98:0/1955729961 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556573efd370 con 0x556573fa5790 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.846+0000 7f6af1f0c640 1 --2- 172.21.15.98:0/1955729961 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x556573fa5790 0x556573fa7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.846+0000 7f6ad17fa640 10 client.19158 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.846+0000 7f6af1f0c640 1 --2- 172.21.15.98:0/1955729961 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x556573fa5790 0x556573fa7b80 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fffc19b4800aa463 server_cookie=dd463b9866cfb70b in_seq=0 out_seq=0 2024-05-09T09:02:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.847+0000 7f6ad17fa640 1 -- 172.21.15.98:0/1955729961 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556573efd370 con 0x556573fa5790 2024-05-09T09:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.847+0000 7f6ad17fa640 10 client.19158 handle_command_reply: tid=0 2024-05-09T09:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.847+0000 7f6af6e7d740 1 client.19158 shutdown 2024-05-09T09:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.847+0000 7f6af6e7d740 1 -- 172.21.15.98:0/1955729961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f95320 msgr2=0x556573f92d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.847+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/1955729961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f95320 0x556573f92d00 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f6adc022040 tx=0x7f6adc005f80 comp rx=0 tx=0).stop 2024-05-09T09:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.847+0000 7f6af6e7d740 1 -- 172.21.15.98:0/1955729961 shutdown_connections 2024-05-09T09:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.847+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/1955729961 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x556573fa5790 0x556573fa7b80 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.847+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/1955729961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f95320 0x556573f92d00 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.847+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/1955729961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f8ac50 0x556573f927c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.847+0000 7f6ad17fa640 0 client.19158 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T09:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.847+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/1955729961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8a330 0x556573f92280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.847+0000 7f6af6e7d740 1 -- 172.21.15.98:0/1955729961 >> 172.21.15.98:0/1955729961 conn(0x556573f86650 msgr2=0x556573f8c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.847+0000 7f6af6e7d740 1 -- 172.21.15.98:0/1955729961 shutdown_connections 2024-05-09T09:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.847+0000 7f6af6e7d740 1 -- 172.21.15.98:0/1955729961 wait complete. 2024-05-09T09:02:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.848+0000 7f6af6e7d740 20 client.19158 trim_cache size 0 max 16384 2024-05-09T09:02:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.861+0000 7f6af6e7d740 1 Processor -- start 2024-05-09T09:02:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.861+0000 7f6af6e7d740 1 -- start start 2024-05-09T09:02:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.861+0000 7f6af6e7d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f8a330 0x556573fa0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.861+0000 7f6af6e7d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8ac50 0x556573f8cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.861+0000 7f6af6e7d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f95320 0x556573fad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.861+0000 7f6af6e7d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1aee80 con 0x556573f8a330 2024-05-09T09:02:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.861+0000 7f6af6e7d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1b09c0 con 0x556573f95320 2024-05-09T09:02:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.861+0000 7f6af6e7d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1af550 con 0x556573f8ac50 2024-05-09T09:02:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.861+0000 7f6af1f0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f8a330 0x556573fa0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.861+0000 7f6af1f0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f8a330 0x556573fa0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51206/0 (socket says 172.21.15.98:51206) 2024-05-09T09:02:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.861+0000 7f6af1f0c640 1 -- 172.21.15.98:0/2235678237 learned_addr learned my addr 172.21.15.98:0/2235678237 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.862+0000 7f6af170b640 1 --2- 172.21.15.98:0/2235678237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8ac50 0x556573f8cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.862+0000 7f6af1f0c640 1 -- 172.21.15.98:0/2235678237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8ac50 msgr2=0x556573f8cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.862+0000 7f6af1f0c640 1 --2- 172.21.15.98:0/2235678237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8ac50 0x556573f8cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.862+0000 7f6af1f0c640 1 -- 172.21.15.98:0/2235678237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f95320 msgr2=0x556573fad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.862+0000 7f6af270d640 1 --2- 172.21.15.98:0/2235678237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f95320 0x556573fad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.862+0000 7f6af1f0c640 1 --2- 172.21.15.98:0/2235678237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f95320 0x556573fad1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.862+0000 7f6af1f0c640 1 -- 172.21.15.98:0/2235678237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556573fa7ef0 con 0x556573f8a330 2024-05-09T09:02:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.862+0000 7f6af1f0c640 1 --2- 172.21.15.98:0/2235678237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f8a330 0x556573fa0f70 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x556573f98380 tx=0x7f6ae4009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55455849223a3e2a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.862+0000 7f6ad37fe640 1 -- 172.21.15.98:0/2235678237 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae40158f0 con 0x556573f8a330 2024-05-09T09:02:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.862+0000 7f6ad37fe640 1 -- 172.21.15.98:0/2235678237 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ae4024070 con 0x556573f8a330 2024-05-09T09:02:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.862+0000 7f6ad37fe640 1 -- 172.21.15.98:0/2235678237 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae4014790 con 0x556573f8a330 2024-05-09T09:02:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.863+0000 7f6af6e7d740 1 -- 172.21.15.98:0/2235678237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f8a330 msgr2=0x556573fa0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.863+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/2235678237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f8a330 0x556573fa0f70 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x556573f98380 tx=0x7f6ae4009110 comp rx=0 tx=0).stop 2024-05-09T09:02:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.863+0000 7f6af6e7d740 1 -- 172.21.15.98:0/2235678237 shutdown_connections 2024-05-09T09:02:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.863+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/2235678237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f95320 0x556573fad1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.863+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/2235678237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8ac50 0x556573f8cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.863+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/2235678237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f8a330 0x556573fa0f70 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.863+0000 7f6af6e7d740 1 -- 172.21.15.98:0/2235678237 >> 172.21.15.98:0/2235678237 conn(0x556573f86650 msgr2=0x556573f857b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.863+0000 7f6af6e7d740 1 -- 172.21.15.98:0/2235678237 shutdown_connections 2024-05-09T09:02:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.863+0000 7f6af6e7d740 1 -- 172.21.15.98:0/2235678237 wait complete. 2024-05-09T09:02:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.863+0000 7f6af6e7d740 1 -- start start 2024-05-09T09:02:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af6e7d740 1 Processor -- start 2024-05-09T09:02:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af6e7d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8a330 0x556573f9d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af6e7d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f8ac50 0x556573f9d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af6e7d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f95320 0x556573f9a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af6e7d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1b02f0 con 0x556573f8ac50 2024-05-09T09:02:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af6e7d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1afc20 con 0x556573f95320 2024-05-09T09:02:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af6e7d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1ace80 con 0x556573f8a330 2024-05-09T09:02:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af6e7d740 10 client.0 _ll_register_callbacks cb 0x7f6adc022070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af170b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f8ac50 0x556573f9d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af170b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f8ac50 0x556573f9d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51212/0 (socket says 172.21.15.98:51212) 2024-05-09T09:02:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af170b640 1 -- 172.21.15.98:0/603765697 learned_addr learned my addr 172.21.15.98:0/603765697 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af270d640 1 --2- 172.21.15.98:0/603765697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f95320 0x556573f9a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af170b640 1 -- 172.21.15.98:0/603765697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8a330 msgr2=0x556573f9d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af170b640 1 --2- 172.21.15.98:0/603765697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8a330 0x556573f9d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af170b640 1 -- 172.21.15.98:0/603765697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f95320 msgr2=0x556573f9a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af170b640 1 --2- 172.21.15.98:0/603765697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f95320 0x556573f9a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.864+0000 7f6af170b640 1 -- 172.21.15.98:0/603765697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556573f9ad70 con 0x556573f8ac50 2024-05-09T09:02:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6af170b640 1 --2- 172.21.15.98:0/603765697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f8ac50 0x556573f9d9c0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00b9af0 tx=0x7f6ae00dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c23f09e4684bdfa4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6ad17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6ad17fa640 1 -- 172.21.15.98:0/603765697 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae00bd120 con 0x556573f8ac50 2024-05-09T09:02:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6ad17fa640 1 -- 172.21.15.98:0/603765697 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f6ae00b9790 con 0x556573f8ac50 2024-05-09T09:02:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6af6e7d740 1 -- 172.21.15.98:0/603765697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ae4024070 con 0x556573f8ac50 2024-05-09T09:02:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6ad17fa640 1 -- 172.21.15.98:0/603765697 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6ae00c14d0 con 0x556573f8ac50 2024-05-09T09:02:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6af6e7d740 10 client.19164 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6af6e7d740 1 -- 172.21.15.98:0/603765697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55657400c6a0 con 0x556573f8ac50 2024-05-09T09:02:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6ad17fa640 1 -- 172.21.15.98:0/603765697 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ae4024070 con 0x556573f8ac50 2024-05-09T09:02:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6ad17fa640 1 -- 172.21.15.98:0/603765697 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f6ae004e070 con 0x556573f8ac50 2024-05-09T09:02:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6af6e7d740 10 client.19164 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6af6e7d740 10 client.19164 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T09:02:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6af6e7d740 20 client.19164 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/603765697 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55657400ce20 0x55657400f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6af6e7d740 4 client.19164 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:02:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6af6e7d740 1 -- 172.21.15.98:0/603765697 --> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556573ef76c0 con 0x55657400ce20 2024-05-09T09:02:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.865+0000 7f6af1f0c640 1 --2- 172.21.15.98:0/603765697 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55657400ce20 0x55657400f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6ad17fa640 10 client.19164 ms_handle_connect on v2:172.21.15.142:6836/1837318616 2024-05-09T09:02:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6af1f0c640 1 --2- 172.21.15.98:0/603765697 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55657400ce20 0x55657400f240 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1d259a14aab7a89f server_cookie=155f0c6ed0d6bca9 in_seq=0 out_seq=0 2024-05-09T09:02:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6ad17fa640 1 -- 172.21.15.98:0/603765697 <== mds.2 v2:172.21.15.142:6836/1837318616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556573ef76c0 con 0x55657400ce20 2024-05-09T09:02:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6ad17fa640 10 client.19164 handle_command_reply: tid=0 2024-05-09T09:02:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6af6e7d740 1 client.19164 shutdown 2024-05-09T09:02:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6af6e7d740 1 -- 172.21.15.98:0/603765697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f8ac50 msgr2=0x556573f9d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/603765697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f8ac50 0x556573f9d9c0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00b9af0 tx=0x7f6ae00dd040 comp rx=0 tx=0).stop 2024-05-09T09:02:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6af6e7d740 1 -- 172.21.15.98:0/603765697 shutdown_connections 2024-05-09T09:02:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/603765697 >> [v2:172.21.15.142:6836/1837318616,v1:172.21.15.142:6837/1837318616] conn(0x55657400ce20 0x55657400f240 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/603765697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556573f95320 0x556573f9a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6ad17fa640 0 client.19164 ms_handle_reset on v2:172.21.15.142:6836/1837318616 2024-05-09T09:02:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/603765697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556573f8ac50 0x556573f9d9c0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6af6e7d740 1 --2- 172.21.15.98:0/603765697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556573f8a330 0x556573f9d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6af6e7d740 1 -- 172.21.15.98:0/603765697 >> 172.21.15.98:0/603765697 conn(0x556573f86650 msgr2=0x556573f93930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6af6e7d740 1 -- 172.21.15.98:0/603765697 shutdown_connections 2024-05-09T09:02:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6af6e7d740 1 -- 172.21.15.98:0/603765697 wait complete. 2024-05-09T09:02:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.866+0000 7f6af6e7d740 20 client.19164 trim_cache size 0 max 16384 2024-05-09T09:02:32.875 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:02:32.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.867+0000 7f6af4170640 1 -- 172.21.15.98:0/2609418289 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6ad40a2d00 msgr2=0x7f6ad40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.867+0000 7f6af4170640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6ad40a2d00 0x7f6ad40a51c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6adc015470 tx=0x7f6adc015870 comp rx=0 tx=0).stop 2024-05-09T09:02:32.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.867+0000 7f6af4170640 1 -- 172.21.15.98:0/2609418289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec1021b0 msgr2=0x7f6aec198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.867+0000 7f6af4170640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec1021b0 0x7f6aec198710 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f6ae002ae00 tx=0x7f6ae0003970 comp rx=0 tx=0).stop 2024-05-09T09:02:32.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.867+0000 7f6af4170640 1 -- 172.21.15.98:0/2609418289 shutdown_connections 2024-05-09T09:02:32.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.867+0000 7f6af4170640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f6ad40a2d00 0x7f6ad40a51c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.867+0000 7f6af4170640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6aec107800 0x7f6aec19ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.867+0000 7f6af4170640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec1021b0 0x7f6aec198710 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.867+0000 7f6af4170640 1 --2- 172.21.15.98:0/2609418289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6aec1017f0 0x7f6aec1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.867+0000 7f6af4170640 1 -- 172.21.15.98:0/2609418289 >> 172.21.15.98:0/2609418289 conn(0x7f6aec0fd2a0 msgr2=0x7f6aec0fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.867+0000 7f6af4170640 1 -- 172.21.15.98:0/2609418289 shutdown_connections 2024-05-09T09:02:32.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.867+0000 7f6af4170640 1 -- 172.21.15.98:0/2609418289 wait complete. 2024-05-09T09:02:32.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T09:02:32.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.981+0000 7ff710110640 1 -- 172.21.15.98:0/242144727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff708106780 msgr2=0x7ff708106b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.981+0000 7ff710110640 1 --2- 172.21.15.98:0/242144727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff708106780 0x7ff708106b60 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7ff704009900 tx=0x7ff70402a410 comp rx=0 tx=0).stop 2024-05-09T09:02:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.981+0000 7ff710110640 1 -- 172.21.15.98:0/242144727 shutdown_connections 2024-05-09T09:02:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.981+0000 7ff710110640 1 --2- 172.21.15.98:0/242144727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff708101420 0x7ff70810d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.981+0000 7ff710110640 1 --2- 172.21.15.98:0/242144727 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff708100a80 0x7ff708100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.981+0000 7ff710110640 1 --2- 172.21.15.98:0/242144727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff708106780 0x7ff708106b60 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.981+0000 7ff710110640 1 -- 172.21.15.98:0/242144727 >> 172.21.15.98:0/242144727 conn(0x7ff708076530 msgr2=0x7ff708076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.982+0000 7ff710110640 1 -- 172.21.15.98:0/242144727 shutdown_connections 2024-05-09T09:02:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.982+0000 7ff710110640 1 -- 172.21.15.98:0/242144727 wait complete. 2024-05-09T09:02:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.982+0000 7ff710110640 1 Processor -- start 2024-05-09T09:02:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.982+0000 7ff710110640 1 -- start start 2024-05-09T09:02:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.982+0000 7ff710110640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff708100a80 0x7ff708196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.982+0000 7ff710110640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff708101420 0x7ff708197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.982+0000 7ff710110640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff708106780 0x7ff70819b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.982+0000 7ff710110640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff708110ae0 con 0x7ff708106780 2024-05-09T09:02:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.982+0000 7ff710110640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff708110960 con 0x7ff708101420 2024-05-09T09:02:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.982+0000 7ff710110640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff708110c60 con 0x7ff708100a80 2024-05-09T09:02:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.982+0000 7ff70e6ad640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff708106780 0x7ff70819b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.982+0000 7ff70e6ad640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff708106780 0x7ff70819b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51242/0 (socket says 172.21.15.98:51242) 2024-05-09T09:02:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.982+0000 7ff70e6ad640 1 -- 172.21.15.98:0/3521431182 learned_addr learned my addr 172.21.15.98:0/3521431182 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff708101420 0x7ff708197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff70deac640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff708100a80 0x7ff708196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff70e6ad640 1 -- 172.21.15.98:0/3521431182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff708100a80 msgr2=0x7ff708196f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff70e6ad640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff708100a80 0x7ff708196f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff70e6ad640 1 -- 172.21.15.98:0/3521431182 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff708101420 msgr2=0x7ff708197460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff70e6ad640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff708101420 0x7ff708197460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff70e6ad640 1 -- 172.21.15.98:0/3521431182 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff70819bf10 con 0x7ff708106780 2024-05-09T09:02:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff708101420 0x7ff708197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff70deac640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff708100a80 0x7ff708196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff70e6ad640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff708106780 0x7ff70819b7e0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7ff70000eb10 tx=0x7ff70000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff6feffd640 1 -- 172.21.15.98:0/3521431182 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff700004b20 con 0x7ff708106780 2024-05-09T09:02:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff6feffd640 1 -- 172.21.15.98:0/3521431182 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff700004cc0 con 0x7ff708106780 2024-05-09T09:02:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff710110640 1 -- 172.21.15.98:0/3521431182 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff70819c200 con 0x7ff708106780 2024-05-09T09:02:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff6feffd640 1 -- 172.21.15.98:0/3521431182 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff700010760 con 0x7ff708106780 2024-05-09T09:02:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.983+0000 7ff710110640 1 -- 172.21.15.98:0/3521431182 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7081a3a90 con 0x7ff708106780 2024-05-09T09:02:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.985+0000 7ff6feffd640 1 -- 172.21.15.98:0/3521431182 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff700005020 con 0x7ff708106780 2024-05-09T09:02:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.986+0000 7ff712e1d740 1 Processor -- start 2024-05-09T09:02:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.986+0000 7ff712e1d740 1 -- start start 2024-05-09T09:02:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.986+0000 7ff712e1d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596109fe000 0x5596109fe3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.986+0000 7ff712e1d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe920 0x559610a0aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.986+0000 7ff712e1d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a0d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.986+0000 7ff712e1d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559610963c40 con 0x559610a0b530 2024-05-09T09:02:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.986+0000 7ff712e1d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55961090b580 con 0x5596109fe000 2024-05-09T09:02:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.986+0000 7ff712e1d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55961095fc00 con 0x5596109fe920 2024-05-09T09:02:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.986+0000 7ff6feffd640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff6f40a2d80 0x7ff6f40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.986+0000 7ff70d6ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a0d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.986+0000 7ff70d6ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a0d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51256/0 (socket says 172.21.15.98:51256) 2024-05-09T09:02:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.986+0000 7ff70d6ab640 1 -- 172.21.15.98:0/4169681858 learned_addr learned my addr 172.21.15.98:0/4169681858 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.986+0000 7ff70deac640 1 --2- 172.21.15.98:0/4169681858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe920 0x559610a0aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.986+0000 7ff70e6ad640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff6f40a2d80 0x7ff6f40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.987+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/4169681858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596109fe000 0x5596109fe3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.987+0000 7ff6feffd640 1 -- 172.21.15.98:0/3521431182 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7ff7000cfed0 con 0x7ff708106780 2024-05-09T09:02:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.987+0000 7ff70d6ab640 1 -- 172.21.15.98:0/4169681858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe920 msgr2=0x559610a0aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.987+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/4169681858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe920 0x559610a0aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.987+0000 7ff70d6ab640 1 -- 172.21.15.98:0/4169681858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596109fe000 msgr2=0x5596109fe3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.987+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/4169681858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596109fe000 0x5596109fe3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.987+0000 7ff70d6ab640 1 -- 172.21.15.98:0/4169681858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596106773f0 con 0x559610a0b530 2024-05-09T09:02:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.987+0000 7ff70e6ad640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff6f40a2d80 0x7ff6f40a5240 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff70009bf10 tx=0x7ff70006eff0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.987+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/4169681858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a0d920 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800be80 tx=0x7ff6f800b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7997b276462dfeb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.987+0000 7ff6f3fff640 1 -- 172.21.15.98:0/4169681858 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8019070 con 0x559610a0b530 2024-05-09T09:02:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.987+0000 7ff6f3fff640 1 -- 172.21.15.98:0/4169681858 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6f8011dc0 con 0x559610a0b530 2024-05-09T09:02:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.987+0000 7ff6f3fff640 1 -- 172.21.15.98:0/4169681858 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f80049c0 con 0x559610a0b530 2024-05-09T09:02:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.988+0000 7ff712e1d740 1 -- 172.21.15.98:0/4169681858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 msgr2=0x559610a0d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.988+0000 7ff712e1d740 1 --2- 172.21.15.98:0/4169681858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a0d920 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800be80 tx=0x7ff6f800b790 comp rx=0 tx=0).stop 2024-05-09T09:02:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.988+0000 7ff712e1d740 1 -- 172.21.15.98:0/4169681858 shutdown_connections 2024-05-09T09:02:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.988+0000 7ff712e1d740 1 --2- 172.21.15.98:0/4169681858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a0d920 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.988+0000 7ff712e1d740 1 --2- 172.21.15.98:0/4169681858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe920 0x559610a0aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.988+0000 7ff712e1d740 1 --2- 172.21.15.98:0/4169681858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596109fe000 0x5596109fe3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.988+0000 7ff712e1d740 1 -- 172.21.15.98:0/4169681858 >> 172.21.15.98:0/4169681858 conn(0x5596109fa650 msgr2=0x5596109faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.988+0000 7ff712e1d740 1 -- 172.21.15.98:0/4169681858 shutdown_connections 2024-05-09T09:02:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.988+0000 7ff712e1d740 1 -- 172.21.15.98:0/4169681858 wait complete. 2024-05-09T09:02:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.988+0000 7ff712e1d740 1 -- start start 2024-05-09T09:02:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.988+0000 7ff712e1d740 1 Processor -- start 2024-05-09T09:02:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff712e1d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596109fe000 0x559610a084b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff712e1d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe920 0x559610a089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff712e1d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a08f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff712e1d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559610a0e2f0 con 0x559610a0b530 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff712e1d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55961091ec40 con 0x5596109fe000 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff712e1d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559610a0e470 con 0x5596109fe920 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff712e1d740 10 client.0 _ll_register_callbacks cb 0x559610676be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff70d6ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a08f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff70d6ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a08f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51264/0 (socket says 172.21.15.98:51264) 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff70d6ab640 1 -- 172.21.15.98:0/3813929146 learned_addr learned my addr 172.21.15.98:0/3813929146 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff70deac640 1 --2- 172.21.15.98:0/3813929146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe920 0x559610a089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff70d6ab640 1 -- 172.21.15.98:0/3813929146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe920 msgr2=0x559610a089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/3813929146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe920 0x559610a089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff70d6ab640 1 -- 172.21.15.98:0/3813929146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596109fe000 msgr2=0x559610a084b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/3813929146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596109fe000 0x559610a084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff70d6ab640 1 -- 172.21.15.98:0/3813929146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596107417d0 con 0x559610a0b530 2024-05-09T09:02:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/3813929146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a08f30 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8013fa0 tx=0x7ff6f8011fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5891d3d839906788 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff6f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff6f17fa640 1 -- 172.21.15.98:0/3813929146 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8019040 con 0x559610a0b530 2024-05-09T09:02:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff712e1d740 1 -- 172.21.15.98:0/3813929146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6f8011dc0 con 0x559610a0b530 2024-05-09T09:02:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.989+0000 7ff6f17fa640 1 -- 172.21.15.98:0/3813929146 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7ff6f8013050 con 0x559610a0b530 2024-05-09T09:02:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.990+0000 7ff6f17fa640 1 -- 172.21.15.98:0/3813929146 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff6f802f050 con 0x559610a0b530 2024-05-09T09:02:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.990+0000 7ff712e1d740 10 client.19176 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.990+0000 7ff712e1d740 1 -- 172.21.15.98:0/3813929146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559610a1a540 con 0x559610a0b530 2024-05-09T09:02:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.990+0000 7ff6f17fa640 1 -- 172.21.15.98:0/3813929146 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6f8011dc0 con 0x559610a0b530 2024-05-09T09:02:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.990+0000 7ff6f17fa640 1 -- 172.21.15.98:0/3813929146 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7ff6f8004a50 con 0x559610a0b530 2024-05-09T09:02:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.990+0000 7ff712e1d740 10 client.19176 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.990+0000 7ff712e1d740 10 client.19176 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:02:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.990+0000 7ff712e1d740 20 client.19176 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.990+0000 7ff712e1d740 1 --2- 172.21.15.98:0/3813929146 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559610a1b8f0 0x559610a1dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.990+0000 7ff712e1d740 4 client.19176 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:02:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.990+0000 7ff712e1d740 1 -- 172.21.15.98:0/3813929146 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559610971370 con 0x559610a1b8f0 2024-05-09T09:02:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.990+0000 7ff70deac640 1 --2- 172.21.15.98:0/3813929146 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559610a1b8f0 0x559610a1dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.991+0000 7ff6f17fa640 10 client.19176 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T09:02:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.991+0000 7ff70deac640 1 --2- 172.21.15.98:0/3813929146 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559610a1b8f0 0x559610a1dce0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=353f2d393868390a server_cookie=538874cf79394ec3 in_seq=0 out_seq=0 2024-05-09T09:02:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.991+0000 7ff6f17fa640 1 -- 172.21.15.98:0/3813929146 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559610971370 con 0x559610a1b8f0 2024-05-09T09:02:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.991+0000 7ff6f17fa640 10 client.19176 handle_command_reply: tid=0 2024-05-09T09:02:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.991+0000 7ff712e1d740 1 client.19176 shutdown 2024-05-09T09:02:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.991+0000 7ff712e1d740 1 -- 172.21.15.98:0/3813929146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 msgr2=0x559610a08f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.991+0000 7ff712e1d740 1 --2- 172.21.15.98:0/3813929146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a08f30 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8013fa0 tx=0x7ff6f8011fd0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.991+0000 7ff712e1d740 1 -- 172.21.15.98:0/3813929146 shutdown_connections 2024-05-09T09:02:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.991+0000 7ff712e1d740 1 --2- 172.21.15.98:0/3813929146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a08f30 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.991+0000 7ff712e1d740 1 --2- 172.21.15.98:0/3813929146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe920 0x559610a089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.991+0000 7ff712e1d740 1 --2- 172.21.15.98:0/3813929146 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559610a1b8f0 0x559610a1dce0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.991+0000 7ff712e1d740 1 --2- 172.21.15.98:0/3813929146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596109fe000 0x559610a084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.991+0000 7ff712e1d740 1 -- 172.21.15.98:0/3813929146 >> 172.21.15.98:0/3813929146 conn(0x5596109fa650 msgr2=0x559610a004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.991+0000 7ff6f17fa640 0 client.19176 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T09:02:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.992+0000 7ff712e1d740 1 -- 172.21.15.98:0/3813929146 shutdown_connections 2024-05-09T09:02:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.992+0000 7ff712e1d740 1 -- 172.21.15.98:0/3813929146 wait complete. 2024-05-09T09:02:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:32.992+0000 7ff712e1d740 20 client.19176 trim_cache size 0 max 16384 2024-05-09T09:02:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff712e1d740 1 Processor -- start 2024-05-09T09:02:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff712e1d740 1 -- start start 2024-05-09T09:02:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff712e1d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5596109fe000 0x559610a17220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff712e1d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe920 0x559610a07ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff712e1d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559610a0b530 0x559610a232b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff712e1d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081acc70 con 0x5596109fe000 2024-05-09T09:02:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff712e1d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081ae7b0 con 0x559610a0b530 2024-05-09T09:02:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff712e1d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081ad340 con 0x5596109fe920 2024-05-09T09:02:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff70d6ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5596109fe000 0x559610a17220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff70d6ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5596109fe000 0x559610a17220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51274/0 (socket says 172.21.15.98:51274) 2024-05-09T09:02:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff70d6ab640 1 -- 172.21.15.98:0/4135069874 learned_addr learned my addr 172.21.15.98:0/4135069874 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff70d6ab640 1 -- 172.21.15.98:0/4135069874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe920 msgr2=0x559610a07ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/4135069874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe920 0x559610a07ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff70d6ab640 1 -- 172.21.15.98:0/4135069874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559610a0b530 msgr2=0x559610a232b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/4135069874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559610a0b530 0x559610a232b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff70d6ab640 1 -- 172.21.15.98:0/4135069874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559610a0d920 con 0x5596109fe000 2024-05-09T09:02:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/4135069874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5596109fe000 0x559610a17220 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80059f0 tx=0x7ff6f8005320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b7d76c4463416b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.005+0000 7ff6f37fe640 1 -- 172.21.15.98:0/4135069874 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f802c070 con 0x5596109fe000 2024-05-09T09:02:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.006+0000 7ff6f37fe640 1 -- 172.21.15.98:0/4135069874 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6f8021070 con 0x5596109fe000 2024-05-09T09:02:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.006+0000 7ff6f37fe640 1 -- 172.21.15.98:0/4135069874 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8019070 con 0x5596109fe000 2024-05-09T09:02:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.006+0000 7ff712e1d740 1 -- 172.21.15.98:0/4135069874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5596109fe000 msgr2=0x559610a17220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.006+0000 7ff712e1d740 1 --2- 172.21.15.98:0/4135069874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5596109fe000 0x559610a17220 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80059f0 tx=0x7ff6f8005320 comp rx=0 tx=0).stop 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.006+0000 7ff712e1d740 1 -- 172.21.15.98:0/4135069874 shutdown_connections 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.006+0000 7ff712e1d740 1 --2- 172.21.15.98:0/4135069874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559610a0b530 0x559610a232b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.006+0000 7ff712e1d740 1 --2- 172.21.15.98:0/4135069874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe920 0x559610a07ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.006+0000 7ff712e1d740 1 --2- 172.21.15.98:0/4135069874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5596109fe000 0x559610a17220 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.006+0000 7ff712e1d740 1 -- 172.21.15.98:0/4135069874 >> 172.21.15.98:0/4135069874 conn(0x5596109fa650 msgr2=0x5596109f97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.006+0000 7ff712e1d740 1 -- 172.21.15.98:0/4135069874 shutdown_connections 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.006+0000 7ff712e1d740 1 -- 172.21.15.98:0/4135069874 wait complete. 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.006+0000 7ff712e1d740 1 -- start start 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff712e1d740 1 Processor -- start 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff712e1d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe000 0x559610a13920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff712e1d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596109fe920 0x559610a04dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff712e1d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a05300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff712e1d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081ae0e0 con 0x559610a0b530 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff712e1d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081ada10 con 0x5596109fe920 2024-05-09T09:02:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff712e1d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081aac70 con 0x5596109fe000 2024-05-09T09:02:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff712e1d740 10 client.0 _ll_register_callbacks cb 0x7ff7000093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff70d6ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a05300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff70d6ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a05300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51286/0 (socket says 172.21.15.98:51286) 2024-05-09T09:02:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff70d6ab640 1 -- 172.21.15.98:0/1856007719 learned_addr learned my addr 172.21.15.98:0/1856007719 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/1856007719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe000 0x559610a13920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff70deac640 1 --2- 172.21.15.98:0/1856007719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596109fe920 0x559610a04dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff70d6ab640 1 -- 172.21.15.98:0/1856007719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe000 msgr2=0x559610a13920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/1856007719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe000 0x559610a13920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff70d6ab640 1 -- 172.21.15.98:0/1856007719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596109fe920 msgr2=0x559610a04dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/1856007719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596109fe920 0x559610a04dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.007+0000 7ff70d6ab640 1 -- 172.21.15.98:0/1856007719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559610a058f0 con 0x559610a0b530 2024-05-09T09:02:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.008+0000 7ff70d6ab640 1 --2- 172.21.15.98:0/1856007719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a05300 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800b870 tx=0x7ff6f8014f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33b359a9b8f59ca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.008+0000 7ff6f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.009+0000 7ff6f17fa640 1 -- 172.21.15.98:0/1856007719 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f802c070 con 0x559610a0b530 2024-05-09T09:02:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.009+0000 7ff6f17fa640 1 -- 172.21.15.98:0/1856007719 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7ff6f8011dc0 con 0x559610a0b530 2024-05-09T09:02:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.010+0000 7ff712e1d740 1 -- 172.21.15.98:0/1856007719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6f8021070 con 0x559610a0b530 2024-05-09T09:02:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.010+0000 7ff6f17fa640 1 -- 172.21.15.98:0/1856007719 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff6f8019040 con 0x559610a0b530 2024-05-09T09:02:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.010+0000 7ff712e1d740 10 client.19182 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.010+0000 7ff712e1d740 1 -- 172.21.15.98:0/1856007719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559610a81900 con 0x559610a0b530 2024-05-09T09:02:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.010+0000 7ff6f17fa640 1 -- 172.21.15.98:0/1856007719 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6f8021070 con 0x559610a0b530 2024-05-09T09:02:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.010+0000 7ff6f17fa640 1 -- 172.21.15.98:0/1856007719 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7ff6f8028460 con 0x559610a0b530 2024-05-09T09:02:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.010+0000 7ff712e1d740 10 client.19182 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.010+0000 7ff712e1d740 10 client.19182 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.010+0000 7ff712e1d740 20 client.19182 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.010+0000 7ff712e1d740 1 --2- 172.21.15.98:0/1856007719 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559610a81e60 0x559610a84250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.010+0000 7ff712e1d740 4 client.19182 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.010+0000 7ff712e1d740 1 -- 172.21.15.98:0/1856007719 --> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55961096b6c0 con 0x559610a81e60 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.010+0000 7ff70deac640 1 --2- 172.21.15.98:0/1856007719 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559610a81e60 0x559610a84250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff6f17fa640 10 client.19182 ms_handle_connect on v2:172.21.15.142:6838/3036634959 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff70deac640 1 --2- 172.21.15.98:0/1856007719 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559610a81e60 0x559610a84250 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=26e6d290dc797a78 server_cookie=4e087d146676fc11 in_seq=0 out_seq=0 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff6f17fa640 1 -- 172.21.15.98:0/1856007719 <== mds.1 v2:172.21.15.142:6838/3036634959 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55961096b6c0 con 0x559610a81e60 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff6f17fa640 10 client.19182 handle_command_reply: tid=0 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff712e1d740 1 client.19182 shutdown 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff712e1d740 1 -- 172.21.15.98:0/1856007719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 msgr2=0x559610a05300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff712e1d740 1 --2- 172.21.15.98:0/1856007719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a05300 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800b870 tx=0x7ff6f8014f90 comp rx=0 tx=0).stop 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff712e1d740 1 -- 172.21.15.98:0/1856007719 shutdown_connections 2024-05-09T09:02:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff712e1d740 1 --2- 172.21.15.98:0/1856007719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559610a0b530 0x559610a05300 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff712e1d740 1 --2- 172.21.15.98:0/1856007719 >> [v2:172.21.15.142:6838/3036634959,v1:172.21.15.142:6839/3036634959] conn(0x559610a81e60 0x559610a84250 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff712e1d740 1 --2- 172.21.15.98:0/1856007719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596109fe920 0x559610a04dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff712e1d740 1 --2- 172.21.15.98:0/1856007719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5596109fe000 0x559610a13920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff6f17fa640 0 client.19182 ms_handle_reset on v2:172.21.15.142:6838/3036634959 2024-05-09T09:02:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff712e1d740 1 -- 172.21.15.98:0/1856007719 >> 172.21.15.98:0/1856007719 conn(0x5596109fa650 msgr2=0x559610a09b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff712e1d740 1 -- 172.21.15.98:0/1856007719 shutdown_connections 2024-05-09T09:02:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.011+0000 7ff712e1d740 1 -- 172.21.15.98:0/1856007719 wait complete. 2024-05-09T09:02:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.012+0000 7ff712e1d740 20 client.19182 trim_cache size 0 max 16384 2024-05-09T09:02:33.015 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:02:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.012+0000 7ff710110640 1 -- 172.21.15.98:0/3521431182 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff6f40a2d80 msgr2=0x7ff6f40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.012+0000 7ff710110640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff6f40a2d80 0x7ff6f40a5240 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff70009bf10 tx=0x7ff70006eff0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.012+0000 7ff710110640 1 -- 172.21.15.98:0/3521431182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff708106780 msgr2=0x7ff70819b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.012+0000 7ff710110640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff708106780 0x7ff70819b7e0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7ff70000eb10 tx=0x7ff70000c2d0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.012+0000 7ff710110640 1 -- 172.21.15.98:0/3521431182 shutdown_connections 2024-05-09T09:02:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.012+0000 7ff710110640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff6f40a2d80 0x7ff6f40a5240 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.012+0000 7ff710110640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff708106780 0x7ff70819b7e0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.012+0000 7ff710110640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff708101420 0x7ff708197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.012+0000 7ff710110640 1 --2- 172.21.15.98:0/3521431182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff708100a80 0x7ff708196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.012+0000 7ff710110640 1 -- 172.21.15.98:0/3521431182 >> 172.21.15.98:0/3521431182 conn(0x7ff708076530 msgr2=0x7ff7080ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.012+0000 7ff710110640 1 -- 172.21.15.98:0/3521431182 shutdown_connections 2024-05-09T09:02:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.012+0000 7ff710110640 1 -- 172.21.15.98:0/3521431182 wait complete. 2024-05-09T09:02:33.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T09:02:33.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.126+0000 7f54c72a5640 1 -- 172.21.15.98:0/3017930182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54c0106b90 msgr2=0x7f54c010e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:33.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.126+0000 7f54c72a5640 1 --2- 172.21.15.98:0/3017930182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54c0106b90 0x7f54c010e7a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009ca0 tx=0x7f54b802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.126+0000 7f54c72a5640 1 -- 172.21.15.98:0/3017930182 shutdown_connections 2024-05-09T09:02:33.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.126+0000 7f54c72a5640 1 --2- 172.21.15.98:0/3017930182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54c0106b90 0x7f54c010e7a0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.126+0000 7f54c72a5640 1 --2- 172.21.15.98:0/3017930182 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54c0100d80 0x7f54c01064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.126+0000 7f54c72a5640 1 --2- 172.21.15.98:0/3017930182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54c01003d0 0x7f54c01007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.126+0000 7f54c72a5640 1 -- 172.21.15.98:0/3017930182 >> 172.21.15.98:0/3017930182 conn(0x7f54c00759c0 msgr2=0x7f54c0075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:33.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.126+0000 7f54c72a5640 1 -- 172.21.15.98:0/3017930182 shutdown_connections 2024-05-09T09:02:33.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.126+0000 7f54c72a5640 1 -- 172.21.15.98:0/3017930182 wait complete. 2024-05-09T09:02:33.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c72a5640 1 Processor -- start 2024-05-09T09:02:33.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c72a5640 1 -- start start 2024-05-09T09:02:33.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c72a5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54c01003d0 0x7f54c0198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c72a5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54c0106b90 0x7f54c01985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c72a5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54c019c940 0x7f54c019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c72a5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f54c0111c80 con 0x7f54c0106b90 2024-05-09T09:02:33.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c72a5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f54c0111b00 con 0x7f54c01003d0 2024-05-09T09:02:33.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c72a5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f54c0111e00 con 0x7f54c019c940 2024-05-09T09:02:33.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c4840640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54c0106b90 0x7f54c01985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c4840640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54c0106b90 0x7f54c01985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51304/0 (socket says 172.21.15.98:51304) 2024-05-09T09:02:33.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c4840640 1 -- 172.21.15.98:0/569962146 learned_addr learned my addr 172.21.15.98:0/569962146 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:33.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c5842640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54c019c940 0x7f54c019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c5041640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54c01003d0 0x7f54c0198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c4840640 1 -- 172.21.15.98:0/569962146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54c019c940 msgr2=0x7f54c019cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:33.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c4840640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54c019c940 0x7f54c019cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c4840640 1 -- 172.21.15.98:0/569962146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54c01003d0 msgr2=0x7f54c0198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:33.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c4840640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54c01003d0 0x7f54c0198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c4840640 1 -- 172.21.15.98:0/569962146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c019d430 con 0x7f54c0106b90 2024-05-09T09:02:33.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c5842640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54c019c940 0x7f54c019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:33.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c5041640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54c01003d0 0x7f54c0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:33.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.127+0000 7f54c4840640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54c0106b90 0x7f54c01985c0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f54b400e9c0 tx=0x7f54b400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:33.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.128+0000 7f54b27fc640 1 -- 172.21.15.98:0/569962146 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b400cd80 con 0x7f54c0106b90 2024-05-09T09:02:33.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.128+0000 7f54b27fc640 1 -- 172.21.15.98:0/569962146 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54b4005500 con 0x7f54c0106b90 2024-05-09T09:02:33.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.128+0000 7f54c72a5640 1 -- 172.21.15.98:0/569962146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54c019d720 con 0x7f54c0106b90 2024-05-09T09:02:33.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.128+0000 7f54b27fc640 1 -- 172.21.15.98:0/569962146 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b4010690 con 0x7f54c0106b90 2024-05-09T09:02:33.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.128+0000 7f54c72a5640 1 -- 172.21.15.98:0/569962146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54c01a4f10 con 0x7f54c0106b90 2024-05-09T09:02:33.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.130+0000 7f54b27fc640 1 -- 172.21.15.98:0/569962146 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f54b40046c0 con 0x7f54c0106b90 2024-05-09T09:02:33.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.130+0000 7f54c9fb8740 1 Processor -- start 2024-05-09T09:02:33.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.130+0000 7f54c9fb8740 1 -- start start 2024-05-09T09:02:33.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c9fb8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 0x5611f8623710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c9fb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f8623c50 0x5611f862dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c9fb8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f862e320 0x5611f8630710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c9fb8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5611f8588c40 con 0x5611f8623c50 2024-05-09T09:02:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c9fb8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5611f8530580 con 0x5611f862e320 2024-05-09T09:02:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c9fb8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5611f8584c00 con 0x5611f8623330 2024-05-09T09:02:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c5041640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f8623c50 0x5611f862dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c5041640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f8623c50 0x5611f862dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51306/0 (socket says 172.21.15.98:51306) 2024-05-09T09:02:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c5041640 1 -- 172.21.15.98:0/878734445 learned_addr learned my addr 172.21.15.98:0/878734445 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54b27fc640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54ac0a2d00 0x7f54ac0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c4840640 1 --2- 172.21.15.98:0/878734445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f862e320 0x5611f8630710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c5842640 1 --2- 172.21.15.98:0/878734445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 0x5611f8623710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c5842640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54ac0a2d00 0x7f54ac0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c5041640 1 -- 172.21.15.98:0/878734445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 msgr2=0x5611f8623710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:33.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c5041640 1 --2- 172.21.15.98:0/878734445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 0x5611f8623710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c5041640 1 -- 172.21.15.98:0/878734445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f862e320 msgr2=0x5611f8630710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:33.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c5041640 1 --2- 172.21.15.98:0/878734445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f862e320 0x5611f8630710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c5041640 1 -- 172.21.15.98:0/878734445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611f829c3f0 con 0x5611f8623c50 2024-05-09T09:02:33.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c4840640 1 --2- 172.21.15.98:0/878734445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f862e320 0x5611f8630710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:33.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54b27fc640 1 -- 172.21.15.98:0/569962146 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f54b40cfce0 con 0x7f54c0106b90 2024-05-09T09:02:33.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c5041640 1 --2- 172.21.15.98:0/878734445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f8623c50 0x5611f862dde0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00bc20 tx=0x7f54bc00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87b238bb6cc696d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:33.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54ab7fe640 1 -- 172.21.15.98:0/878734445 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc017620 con 0x5611f8623c50 2024-05-09T09:02:33.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54c5842640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54ac0a2d00 0x7f54ac0a51c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f54b8005fe0 tx=0x7f54b80095c0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:33.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54ab7fe640 1 -- 172.21.15.98:0/878734445 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54bc017ca0 con 0x5611f8623c50 2024-05-09T09:02:33.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.131+0000 7f54ab7fe640 1 -- 172.21.15.98:0/878734445 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc00fd00 con 0x5611f8623c50 2024-05-09T09:02:33.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.132+0000 7f54c9fb8740 1 -- 172.21.15.98:0/878734445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f8623c50 msgr2=0x5611f862dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:33.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.132+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/878734445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f8623c50 0x5611f862dde0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00bc20 tx=0x7f54bc00d710 comp rx=0 tx=0).stop 2024-05-09T09:02:33.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.132+0000 7f54c9fb8740 1 -- 172.21.15.98:0/878734445 shutdown_connections 2024-05-09T09:02:33.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.132+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/878734445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f862e320 0x5611f8630710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.132+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/878734445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f8623c50 0x5611f862dde0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.132+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/878734445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 0x5611f8623710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.132+0000 7f54c9fb8740 1 -- 172.21.15.98:0/878734445 >> 172.21.15.98:0/878734445 conn(0x5611f861f650 msgr2=0x5611f861fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:33.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.132+0000 7f54c9fb8740 1 -- 172.21.15.98:0/878734445 shutdown_connections 2024-05-09T09:02:33.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.132+0000 7f54c9fb8740 1 -- 172.21.15.98:0/878734445 wait complete. 2024-05-09T09:02:33.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.132+0000 7f54c9fb8740 1 -- start start 2024-05-09T09:02:33.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.132+0000 7f54c9fb8740 1 Processor -- start 2024-05-09T09:02:33.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c9fb8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 0x5611f862b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c9fb8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f8623c50 0x5611f862b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c9fb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f862e320 0x5611f862bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c9fb8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5611f8631110 con 0x5611f862e320 2024-05-09T09:02:33.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c9fb8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5611f8543c40 con 0x5611f8623c50 2024-05-09T09:02:33.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c9fb8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5611f8631290 con 0x5611f8623330 2024-05-09T09:02:33.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c9fb8740 10 client.0 _ll_register_callbacks cb 0x5611f829bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:33.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c5842640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f862e320 0x5611f862bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c5842640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f862e320 0x5611f862bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51310/0 (socket says 172.21.15.98:51310) 2024-05-09T09:02:33.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c5842640 1 -- 172.21.15.98:0/3516828334 learned_addr learned my addr 172.21.15.98:0/3516828334 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:33.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c5041640 1 --2- 172.21.15.98:0/3516828334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 0x5611f862b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c4840640 1 --2- 172.21.15.98:0/3516828334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f8623c50 0x5611f862b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c5842640 1 -- 172.21.15.98:0/3516828334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 msgr2=0x5611f862b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:33.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c5842640 1 --2- 172.21.15.98:0/3516828334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 0x5611f862b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c5842640 1 -- 172.21.15.98:0/3516828334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f8623c50 msgr2=0x5611f862b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:33.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c5842640 1 --2- 172.21.15.98:0/3516828334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f8623c50 0x5611f862b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c5842640 1 -- 172.21.15.98:0/3516828334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611f83667d0 con 0x5611f862e320 2024-05-09T09:02:33.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54c5842640 1 --2- 172.21.15.98:0/3516828334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f862e320 0x5611f862bca0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f54b802f430 tx=0x7f54b802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ea5e5b780ad261 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:33.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.133+0000 7f54a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:33.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54a8ff9640 1 -- 172.21.15.98:0/3516828334 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b804d070 con 0x5611f862e320 2024-05-09T09:02:33.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54a8ff9640 1 -- 172.21.15.98:0/3516828334 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f54b8004450 con 0x5611f862e320 2024-05-09T09:02:33.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54c9fb8740 1 -- 172.21.15.98:0/3516828334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54bc017ca0 con 0x5611f862e320 2024-05-09T09:02:33.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54a8ff9640 1 -- 172.21.15.98:0/3516828334 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f54b8043070 con 0x5611f862e320 2024-05-09T09:02:33.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54c9fb8740 10 client.19194 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:33.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54c9fb8740 1 -- 172.21.15.98:0/3516828334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5611f863d2d0 con 0x5611f862e320 2024-05-09T09:02:33.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54a8ff9640 1 -- 172.21.15.98:0/3516828334 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54bc017ca0 con 0x5611f862e320 2024-05-09T09:02:33.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54a8ff9640 1 -- 172.21.15.98:0/3516828334 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f54b80305c0 con 0x5611f862e320 2024-05-09T09:02:33.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54c9fb8740 10 client.19194 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:33.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54c9fb8740 10 client.19194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:02:33.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54c9fb8740 20 client.19194 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:33.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/3516828334 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5611f863e790 0x5611f8640bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54c9fb8740 4 client.19194 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:02:33.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54c9fb8740 1 -- 172.21.15.98:0/3516828334 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611f8596370 con 0x5611f863e790 2024-05-09T09:02:33.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.134+0000 7f54c5041640 1 --2- 172.21.15.98:0/3516828334 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5611f863e790 0x5611f8640bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.135+0000 7f54c5041640 1 --2- 172.21.15.98:0/3516828334 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5611f863e790 0x5611f8640bc0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0caa62dada003c6 server_cookie=ccf24d403b957736 in_seq=0 out_seq=0 2024-05-09T09:02:33.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.135+0000 7f54a8ff9640 10 client.19194 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:33.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54a8ff9640 1 -- 172.21.15.98:0/3516828334 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5611f8596370 con 0x5611f863e790 2024-05-09T09:02:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54a8ff9640 10 client.19194 handle_command_reply: tid=0 2024-05-09T09:02:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54c9fb8740 1 client.19194 shutdown 2024-05-09T09:02:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54c9fb8740 1 -- 172.21.15.98:0/3516828334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f862e320 msgr2=0x5611f862bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/3516828334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f862e320 0x5611f862bca0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f54b802f430 tx=0x7f54b802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54c9fb8740 1 -- 172.21.15.98:0/3516828334 shutdown_connections 2024-05-09T09:02:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/3516828334 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5611f863e790 0x5611f8640bc0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/3516828334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f862e320 0x5611f862bca0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54a8ff9640 0 client.19194 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/3516828334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f8623c50 0x5611f862b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/3516828334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 0x5611f862b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54c9fb8740 1 -- 172.21.15.98:0/3516828334 >> 172.21.15.98:0/3516828334 conn(0x5611f861f650 msgr2=0x5611f8625450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:33.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54c9fb8740 1 -- 172.21.15.98:0/3516828334 shutdown_connections 2024-05-09T09:02:33.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54c9fb8740 1 -- 172.21.15.98:0/3516828334 wait complete. 2024-05-09T09:02:33.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.136+0000 7f54c9fb8740 20 client.19194 trim_cache size 0 max 16384 2024-05-09T09:02:33.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.149+0000 7f54c9fb8740 1 Processor -- start 2024-05-09T09:02:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.149+0000 7f54c9fb8740 1 -- start start 2024-05-09T09:02:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.149+0000 7f54c9fb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f8623330 0x5611f8639fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.149+0000 7f54c9fb8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623c50 0x5611f862a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.149+0000 7f54c9fb8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f862e320 0x5611f8625a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.149+0000 7f54c9fb8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f54c01aee20 con 0x5611f8623330 2024-05-09T09:02:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.149+0000 7f54c9fb8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f54c01b0960 con 0x5611f862e320 2024-05-09T09:02:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.149+0000 7f54c9fb8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f54c01af4f0 con 0x5611f8623c50 2024-05-09T09:02:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.149+0000 7f54c5041640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f8623330 0x5611f8639fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.149+0000 7f54c5041640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f8623330 0x5611f8639fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51312/0 (socket says 172.21.15.98:51312) 2024-05-09T09:02:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.149+0000 7f54c5041640 1 -- 172.21.15.98:0/2373041122 learned_addr learned my addr 172.21.15.98:0/2373041122 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.150+0000 7f54c4840640 1 --2- 172.21.15.98:0/2373041122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623c50 0x5611f862a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.150+0000 7f54c5842640 1 --2- 172.21.15.98:0/2373041122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f862e320 0x5611f8625a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.150+0000 7f54c5041640 1 -- 172.21.15.98:0/2373041122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623c50 msgr2=0x5611f862a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.150+0000 7f54c5041640 1 --2- 172.21.15.98:0/2373041122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623c50 0x5611f862a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.150+0000 7f54c5041640 1 -- 172.21.15.98:0/2373041122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f862e320 msgr2=0x5611f8625a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.150+0000 7f54c5041640 1 --2- 172.21.15.98:0/2373041122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f862e320 0x5611f8625a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.150+0000 7f54c5041640 1 -- 172.21.15.98:0/2373041122 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611f8623a80 con 0x5611f8623330 2024-05-09T09:02:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.150+0000 7f54c5842640 1 --2- 172.21.15.98:0/2373041122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f862e320 0x5611f8625a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.150+0000 7f54c5041640 1 --2- 172.21.15.98:0/2373041122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f8623330 0x5611f8639fd0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00bbf0 tx=0x7f54bc00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6235ee57fc8f9a6c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.150+0000 7f54aaffd640 1 -- 172.21.15.98:0/2373041122 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc02d070 con 0x5611f8623330 2024-05-09T09:02:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.150+0000 7f54aaffd640 1 -- 172.21.15.98:0/2373041122 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54bc00fe50 con 0x5611f8623330 2024-05-09T09:02:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.150+0000 7f54aaffd640 1 -- 172.21.15.98:0/2373041122 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc029350 con 0x5611f8623330 2024-05-09T09:02:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.151+0000 7f54c9fb8740 1 -- 172.21.15.98:0/2373041122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f8623330 msgr2=0x5611f8639fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.151+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/2373041122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f8623330 0x5611f8639fd0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00bbf0 tx=0x7f54bc00d710 comp rx=0 tx=0).stop 2024-05-09T09:02:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.151+0000 7f54c9fb8740 1 -- 172.21.15.98:0/2373041122 shutdown_connections 2024-05-09T09:02:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.151+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/2373041122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f862e320 0x5611f8625a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.151+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/2373041122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623c50 0x5611f862a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.151+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/2373041122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f8623330 0x5611f8639fd0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.151+0000 7f54c9fb8740 1 -- 172.21.15.98:0/2373041122 >> 172.21.15.98:0/2373041122 conn(0x5611f861f650 msgr2=0x5611f861fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.151+0000 7f54c9fb8740 1 -- 172.21.15.98:0/2373041122 shutdown_connections 2024-05-09T09:02:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.151+0000 7f54c9fb8740 1 -- 172.21.15.98:0/2373041122 wait complete. 2024-05-09T09:02:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.151+0000 7f54c9fb8740 1 -- start start 2024-05-09T09:02:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.151+0000 7f54c9fb8740 1 Processor -- start 2024-05-09T09:02:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c9fb8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 0x5611f86364e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c9fb8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f8623c50 0x5611f8636a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c9fb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f862e320 0x5611f8627630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c9fb8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f54c01b0290 con 0x5611f862e320 2024-05-09T09:02:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c9fb8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f54c01afbc0 con 0x5611f8623c50 2024-05-09T09:02:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c9fb8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f54c01ace20 con 0x5611f8623330 2024-05-09T09:02:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c9fb8740 10 client.0 _ll_register_callbacks cb 0x7f54b4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:02:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c5842640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f862e320 0x5611f8627630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c5842640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f862e320 0x5611f8627630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51326/0 (socket says 172.21.15.98:51326) 2024-05-09T09:02:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c5842640 1 -- 172.21.15.98:0/128124269 learned_addr learned my addr 172.21.15.98:0/128124269 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c4840640 1 --2- 172.21.15.98:0/128124269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f8623c50 0x5611f8636a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c5041640 1 --2- 172.21.15.98:0/128124269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 0x5611f86364e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c5842640 1 -- 172.21.15.98:0/128124269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 msgr2=0x5611f86364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c5842640 1 --2- 172.21.15.98:0/128124269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 0x5611f86364e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c5842640 1 -- 172.21.15.98:0/128124269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f8623c50 msgr2=0x5611f8636a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:02:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c5842640 1 --2- 172.21.15.98:0/128124269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f8623c50 0x5611f8636a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c5842640 1 -- 172.21.15.98:0/128124269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611f8627e30 con 0x5611f862e320 2024-05-09T09:02:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c5842640 1 --2- 172.21.15.98:0/128124269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f862e320 0x5611f8627630 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f54b8031040 tx=0x7f54b8030870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5078503c53d78021 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-09T09:02:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54a8ff9640 1 -- 172.21.15.98:0/128124269 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b8063020 con 0x5611f862e320 2024-05-09T09:02:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54a8ff9640 1 -- 172.21.15.98:0/128124269 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f54b8004660 con 0x5611f862e320 2024-05-09T09:02:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c9fb8740 1 -- 172.21.15.98:0/128124269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54bc00fe50 con 0x5611f862e320 2024-05-09T09:02:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54a8ff9640 1 -- 172.21.15.98:0/128124269 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f54b806b050 con 0x5611f862e320 2024-05-09T09:02:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c9fb8740 10 client.19200 fetch_fsmap learned FSMap version 37 2024-05-09T09:02:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.152+0000 7f54c9fb8740 1 -- 172.21.15.98:0/128124269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5611f86a4750 con 0x5611f862e320 2024-05-09T09:02:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.153+0000 7f54a8ff9640 1 -- 172.21.15.98:0/128124269 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54bc00fe50 con 0x5611f862e320 2024-05-09T09:02:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.153+0000 7f54a8ff9640 1 -- 172.21.15.98:0/128124269 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f54b8052460 con 0x5611f862e320 2024-05-09T09:02:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.153+0000 7f54c9fb8740 10 client.19200 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:02:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.153+0000 7f54c9fb8740 10 client.19200 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-05-09T09:02:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.153+0000 7f54c9fb8740 20 client.19200 populate_metadata read hostname 'smithi098' 2024-05-09T09:02:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.153+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/128124269 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5611f863e790 0x5611f86a6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.153+0000 7f54c9fb8740 4 client.19200 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:02:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.153+0000 7f54c9fb8740 1 -- 172.21.15.98:0/128124269 --> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5611f85906c0 con 0x5611f863e790 2024-05-09T09:02:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.153+0000 7f54c5041640 1 --2- 172.21.15.98:0/128124269 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5611f863e790 0x5611f86a6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54a8ff9640 10 client.19200 ms_handle_connect on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54c5041640 1 --2- 172.21.15.98:0/128124269 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5611f863e790 0x5611f86a6f30 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38bb563ac347db8c server_cookie=7c7cac99c85e7a74 in_seq=0 out_seq=0 2024-05-09T09:02:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54a8ff9640 1 -- 172.21.15.98:0/128124269 <== mds.0 v2:172.21.15.105:6836/992088454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5611f85906c0 con 0x5611f863e790 2024-05-09T09:02:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54a8ff9640 10 client.19200 handle_command_reply: tid=0 2024-05-09T09:02:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54c9fb8740 1 client.19200 shutdown 2024-05-09T09:02:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54c9fb8740 1 -- 172.21.15.98:0/128124269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f862e320 msgr2=0x5611f8627630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/128124269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f862e320 0x5611f8627630 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f54b8031040 tx=0x7f54b8030870 comp rx=0 tx=0).stop 2024-05-09T09:02:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54c9fb8740 1 -- 172.21.15.98:0/128124269 shutdown_connections 2024-05-09T09:02:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/128124269 >> [v2:172.21.15.105:6836/992088454,v1:172.21.15.105:6837/992088454] conn(0x5611f863e790 0x5611f86a6f30 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/128124269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611f862e320 0x5611f8627630 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/128124269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5611f8623c50 0x5611f8636a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54a8ff9640 0 client.19200 ms_handle_reset on v2:172.21.15.105:6836/992088454 2024-05-09T09:02:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54c9fb8740 1 --2- 172.21.15.98:0/128124269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5611f8623330 0x5611f86364e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54c9fb8740 1 -- 172.21.15.98:0/128124269 >> 172.21.15.98:0/128124269 conn(0x5611f861f650 msgr2=0x5611f861df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54c9fb8740 1 -- 172.21.15.98:0/128124269 shutdown_connections 2024-05-09T09:02:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54c9fb8740 1 -- 172.21.15.98:0/128124269 wait complete. 2024-05-09T09:02:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.154+0000 7f54c9fb8740 20 client.19200 trim_cache size 0 max 16384 2024-05-09T09:02:33.161 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-05-09T09:02:33.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.155+0000 7f54c72a5640 1 -- 172.21.15.98:0/569962146 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54ac0a2d00 msgr2=0x7f54ac0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.155+0000 7f54c72a5640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54ac0a2d00 0x7f54ac0a51c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f54b8005fe0 tx=0x7f54b80095c0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.155+0000 7f54c72a5640 1 -- 172.21.15.98:0/569962146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54c0106b90 msgr2=0x7f54c01985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.155+0000 7f54c72a5640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54c0106b90 0x7f54c01985c0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f54b400e9c0 tx=0x7f54b400ee80 comp rx=0 tx=0).stop 2024-05-09T09:02:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.155+0000 7f54c72a5640 1 -- 172.21.15.98:0/569962146 shutdown_connections 2024-05-09T09:02:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.155+0000 7f54c72a5640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54ac0a2d00 0x7f54ac0a51c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.155+0000 7f54c72a5640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54c019c940 0x7f54c019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.155+0000 7f54c72a5640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54c0106b90 0x7f54c01985c0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.155+0000 7f54c72a5640 1 --2- 172.21.15.98:0/569962146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54c01003d0 0x7f54c0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.155+0000 7f54c72a5640 1 -- 172.21.15.98:0/569962146 >> 172.21.15.98:0/569962146 conn(0x7f54c00759c0 msgr2=0x7f54c00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.155+0000 7f54c72a5640 1 -- 172.21.15.98:0/569962146 shutdown_connections 2024-05-09T09:02:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:33.155+0000 7f54c72a5640 1 -- 172.21.15.98:0/569962146 wait complete. 2024-05-09T09:02:33.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:33 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3264051774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:02:33.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:33 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3264051774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:02:33.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:33 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3264051774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:02:34.165 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-05-09T09:02:34.165 INFO:tasks.fwd_scrub:done joining 2024-05-09T09:02:34.165 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-05-09T09:02:34.176 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:02:34.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.289+0000 7f1bc81d5640 1 -- 172.21.15.98:0/3279570282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc0101220 msgr2=0x7f1bc01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:34.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.289+0000 7f1bc81d5640 1 --2- 172.21.15.98:0/3279570282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc0101220 0x7f1bc01056e0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4009900 tx=0x7f1bb402a480 comp rx=0 tx=0).stop 2024-05-09T09:02:34.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.290+0000 7f1bc81d5640 1 -- 172.21.15.98:0/3279570282 shutdown_connections 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.290+0000 7f1bc81d5640 1 --2- 172.21.15.98:0/3279570282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bc0105c20 0x7f1bc01060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.290+0000 7f1bc81d5640 1 --2- 172.21.15.98:0/3279570282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc0101220 0x7f1bc01056e0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.290+0000 7f1bc81d5640 1 --2- 172.21.15.98:0/3279570282 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bc0100900 0x7f1bc0100ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.290+0000 7f1bc81d5640 1 -- 172.21.15.98:0/3279570282 >> 172.21.15.98:0/3279570282 conn(0x7f1bc0077890 msgr2=0x7f1bc0077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.290+0000 7f1bc81d5640 1 -- 172.21.15.98:0/3279570282 shutdown_connections 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.290+0000 7f1bc81d5640 1 -- 172.21.15.98:0/3279570282 wait complete. 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.290+0000 7f1bc81d5640 1 Processor -- start 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.290+0000 7f1bc81d5640 1 -- start start 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.290+0000 7f1bc81d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc0100900 0x7f1bc019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.290+0000 7f1bc81d5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bc0101220 0x7f1bc019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc81d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bc0105c20 0x7f1bc01a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc81d5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc01181e0 con 0x7f1bc0100900 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc81d5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc0118060 con 0x7f1bc0105c20 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc81d5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc0118360 con 0x7f1bc0101220 2024-05-09T09:02:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc5f71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc0100900 0x7f1bc019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc5f71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc0100900 0x7f1bc019e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51344/0 (socket says 172.21.15.98:51344) 2024-05-09T09:02:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc5f71640 1 -- 172.21.15.98:0/2749224544 learned_addr learned my addr 172.21.15.98:0/2749224544 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc5770640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bc0101220 0x7f1bc019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc5f71640 1 -- 172.21.15.98:0/2749224544 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bc0101220 msgr2=0x7f1bc019ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc5f71640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bc0101220 0x7f1bc019ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc5f71640 1 -- 172.21.15.98:0/2749224544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bc0105c20 msgr2=0x7f1bc01a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc6772640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bc0105c20 0x7f1bc01a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc5f71640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bc0105c20 0x7f1bc01a2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc5f71640 1 -- 172.21.15.98:0/2749224544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bc01a3680 con 0x7f1bc0100900 2024-05-09T09:02:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc5770640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bc0101220 0x7f1bc019ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc6772640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bc0105c20 0x7f1bc01a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc5f71640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc0100900 0x7f1bc019e6e0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000cce0 tx=0x7f1bb000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bbeffd640 1 -- 172.21.15.98:0/2749224544 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0004b20 con 0x7f1bc0100900 2024-05-09T09:02:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bbeffd640 1 -- 172.21.15.98:0/2749224544 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bb0004cc0 con 0x7f1bc0100900 2024-05-09T09:02:34.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bbeffd640 1 -- 172.21.15.98:0/2749224544 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb000f7d0 con 0x7f1bc0100900 2024-05-09T09:02:34.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc81d5640 1 -- 172.21.15.98:0/2749224544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bc01a3970 con 0x7f1bc0100900 2024-05-09T09:02:34.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.291+0000 7f1bc81d5640 1 -- 172.21.15.98:0/2749224544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bc01ab200 con 0x7f1bc0100900 2024-05-09T09:02:34.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.292+0000 7f1bc81d5640 1 -- 172.21.15.98:0/2749224544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bc006a660 con 0x7f1bc0100900 2024-05-09T09:02:34.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.293+0000 7f1bbeffd640 1 -- 172.21.15.98:0/2749224544 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1bb000f970 con 0x7f1bc0100900 2024-05-09T09:02:34.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.294+0000 7f1bbeffd640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1ba80a2d00 0x7f1ba80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:34.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.294+0000 7f1bc5770640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1ba80a2d00 0x7f1ba80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:34.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.294+0000 7f1bbeffd640 1 -- 172.21.15.98:0/2749224544 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f1bb00cf9d0 con 0x7f1bc0100900 2024-05-09T09:02:34.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.295+0000 7f1bc5770640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1ba80a2d00 0x7f1ba80a51c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4000c00 tx=0x7f1bb4032d50 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.299+0000 7f1bbeffd640 1 -- 172.21.15.98:0/2749224544 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1bb0014030 con 0x7f1bc0100900 2024-05-09T09:02:34.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:34 smithi098 ceph-mon[30092]: pgmap v374: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-05-09T09:02:34.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:34 smithi142 ceph-mon[34625]: pgmap v374: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-05-09T09:02:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.479+0000 7f1bc81d5640 1 -- 172.21.15.98:0/2749224544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1bc01ab510 con 0x7f1bc0100900 2024-05-09T09:02:34.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.480+0000 7f1bbeffd640 1 -- 172.21.15.98:0/2749224544 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) ==== 74+0+25392 (secure 0 0 0) 0x7f1bb009c1e0 con 0x7f1bc0100900 2024-05-09T09:02:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T09:02:34.480 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":109,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T09:02:17.976388+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":108,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":108,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":1}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1013153207":"2024-05-10T08:50:38.492493+0000","172.21.15.98:6800/1228297308":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/808614683":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/1734294041":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/2113007677":"2024-05-10T08:50:38.492493+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1228297308":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]},{"pool":4,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:02:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.483+0000 7f1bc81d5640 1 -- 172.21.15.98:0/2749224544 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1ba80a2d00 msgr2=0x7f1ba80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.483+0000 7f1bc81d5640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1ba80a2d00 0x7f1ba80a51c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4000c00 tx=0x7f1bb4032d50 comp rx=0 tx=0).stop 2024-05-09T09:02:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.483+0000 7f1bc81d5640 1 -- 172.21.15.98:0/2749224544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc0100900 msgr2=0x7f1bc019e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.483+0000 7f1bc81d5640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc0100900 0x7f1bc019e6e0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000cce0 tx=0x7f1bb000a620 comp rx=0 tx=0).stop 2024-05-09T09:02:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.483+0000 7f1bc81d5640 1 -- 172.21.15.98:0/2749224544 shutdown_connections 2024-05-09T09:02:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.483+0000 7f1bc81d5640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1ba80a2d00 0x7f1ba80a51c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.483+0000 7f1bc81d5640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bc0105c20 0x7f1bc01a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.483+0000 7f1bc81d5640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1bc0101220 0x7f1bc019ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.483+0000 7f1bc81d5640 1 --2- 172.21.15.98:0/2749224544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc0100900 0x7f1bc019e6e0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.483+0000 7f1bc81d5640 1 -- 172.21.15.98:0/2749224544 >> 172.21.15.98:0/2749224544 conn(0x7f1bc0077890 msgr2=0x7f1bc01036b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.483+0000 7f1bc81d5640 1 -- 172.21.15.98:0/2749224544 shutdown_connections 2024-05-09T09:02:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.483+0000 7f1bc81d5640 1 -- 172.21.15.98:0/2749224544 wait complete. 2024-05-09T09:02:34.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:34 smithi105 ceph-mon[34498]: pgmap v374: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-05-09T09:02:34.493 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:43:39.424443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:48:15.866450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:48:17.122162+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 108, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:48:20.591475+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 108, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T09:02:34.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:02:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.599+0000 7fd8dd934640 1 -- 172.21.15.98:0/2966218066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d81013d0 msgr2=0x7fd8d810d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.599+0000 7fd8dd934640 1 --2- 172.21.15.98:0/2966218066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d81013d0 0x7fd8d810d5a0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009930 tx=0x7fd8d002a420 comp rx=0 tx=0).stop 2024-05-09T09:02:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.599+0000 7fd8dd934640 1 -- 172.21.15.98:0/2966218066 shutdown_connections 2024-05-09T09:02:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.599+0000 7fd8dd934640 1 --2- 172.21.15.98:0/2966218066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d81013d0 0x7fd8d810d5a0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.599+0000 7fd8dd934640 1 --2- 172.21.15.98:0/2966218066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8d8100a10 0x7fd8d8100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.599+0000 7fd8dd934640 1 --2- 172.21.15.98:0/2966218066 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d8106900 0x7fd8d8106ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.599+0000 7fd8dd934640 1 -- 172.21.15.98:0/2966218066 >> 172.21.15.98:0/2966218066 conn(0x7fd8d8077710 msgr2=0x7fd8d8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.599+0000 7fd8dd934640 1 -- 172.21.15.98:0/2966218066 shutdown_connections 2024-05-09T09:02:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.599+0000 7fd8dd934640 1 -- 172.21.15.98:0/2966218066 wait complete. 2024-05-09T09:02:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.600+0000 7fd8dd934640 1 Processor -- start 2024-05-09T09:02:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.600+0000 7fd8dd934640 1 -- start start 2024-05-09T09:02:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.600+0000 7fd8dd934640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d8100a10 0x7fd8d81981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.600+0000 7fd8dd934640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d81013d0 0x7fd8d81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.600+0000 7fd8dd934640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8d8106900 0x7fd8d819ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.600+0000 7fd8dd934640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d8111cf0 con 0x7fd8d81013d0 2024-05-09T09:02:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.600+0000 7fd8dd934640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d8111b70 con 0x7fd8d8106900 2024-05-09T09:02:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.600+0000 7fd8dd934640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d8111e70 con 0x7fd8d8100a10 2024-05-09T09:02:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.600+0000 7fd8d67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d81013d0 0x7fd8d81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.600+0000 7fd8d67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d81013d0 0x7fd8d81986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51352/0 (socket says 172.21.15.98:51352) 2024-05-09T09:02:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.600+0000 7fd8d67fc640 1 -- 172.21.15.98:0/2992291835 learned_addr learned my addr 172.21.15.98:0/2992291835 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.600+0000 7fd8d6ffd640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d8100a10 0x7fd8d81981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.601+0000 7fd8d77fe640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8d8106900 0x7fd8d819ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.601+0000 7fd8d67fc640 1 -- 172.21.15.98:0/2992291835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d8100a10 msgr2=0x7fd8d81981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.601+0000 7fd8d67fc640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d8100a10 0x7fd8d81981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.601+0000 7fd8d67fc640 1 -- 172.21.15.98:0/2992291835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8d8106900 msgr2=0x7fd8d819ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.601+0000 7fd8d67fc640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8d8106900 0x7fd8d819ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.601+0000 7fd8d67fc640 1 -- 172.21.15.98:0/2992291835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d819d150 con 0x7fd8d81013d0 2024-05-09T09:02:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.601+0000 7fd8d6ffd640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d8100a10 0x7fd8d81981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.601+0000 7fd8d67fc640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d81013d0 0x7fd8d81986f0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fd8c000d950 tx=0x7fd8c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.601+0000 7fd8dc932640 1 -- 172.21.15.98:0/2992291835 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c0017030 con 0x7fd8d81013d0 2024-05-09T09:02:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.601+0000 7fd8dc932640 1 -- 172.21.15.98:0/2992291835 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8c0004020 con 0x7fd8d81013d0 2024-05-09T09:02:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.601+0000 7fd8dd934640 1 -- 172.21.15.98:0/2992291835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8d819d440 con 0x7fd8d81013d0 2024-05-09T09:02:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.601+0000 7fd8dc932640 1 -- 172.21.15.98:0/2992291835 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c00047a0 con 0x7fd8d81013d0 2024-05-09T09:02:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.601+0000 7fd8dd934640 1 -- 172.21.15.98:0/2992291835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8d81a4cd0 con 0x7fd8d81013d0 2024-05-09T09:02:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.602+0000 7fd8dd934640 1 -- 172.21.15.98:0/2992291835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8d8068aa0 con 0x7fd8d81013d0 2024-05-09T09:02:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.603+0000 7fd8dc932640 1 -- 172.21.15.98:0/2992291835 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd8c00041d0 con 0x7fd8d81013d0 2024-05-09T09:02:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.604+0000 7fd8dc932640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8c40a2d00 0x7fd8c40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.604+0000 7fd8d6ffd640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8c40a2d00 0x7fd8c40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.604+0000 7fd8dc932640 1 -- 172.21.15.98:0/2992291835 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7fd8c00cf350 con 0x7fd8d81013d0 2024-05-09T09:02:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.604+0000 7fd8d6ffd640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8c40a2d00 0x7fd8c40a51c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8005cc0 tx=0x7fd8c8005c30 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:34.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.608+0000 7fd8dc932640 1 -- 172.21.15.98:0/2992291835 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd8c009bb60 con 0x7fd8d81013d0 2024-05-09T09:02:34.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.786+0000 7fd8dd934640 1 -- 172.21.15.98:0/2992291835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd8d8108bf0 con 0x7fd8d81013d0 2024-05-09T09:02:34.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.787+0000 7fd8dc932640 1 -- 172.21.15.98:0/2992291835 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) ==== 93+0+10 (secure 0 0 0) 0x7fd8c009b460 con 0x7fd8d81013d0 2024-05-09T09:02:34.786 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-09T09:02:34.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.788+0000 7fd8dd934640 1 -- 172.21.15.98:0/2992291835 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8c40a2d00 msgr2=0x7fd8c40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:34.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.788+0000 7fd8dd934640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8c40a2d00 0x7fd8c40a51c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8005cc0 tx=0x7fd8c8005c30 comp rx=0 tx=0).stop 2024-05-09T09:02:34.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.788+0000 7fd8dd934640 1 -- 172.21.15.98:0/2992291835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d81013d0 msgr2=0x7fd8d81986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:34.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.788+0000 7fd8dd934640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d81013d0 0x7fd8d81986f0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fd8c000d950 tx=0x7fd8c000de10 comp rx=0 tx=0).stop 2024-05-09T09:02:34.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.789+0000 7fd8dd934640 1 -- 172.21.15.98:0/2992291835 shutdown_connections 2024-05-09T09:02:34.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.789+0000 7fd8dd934640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fd8c40a2d00 0x7fd8c40a51c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.789+0000 7fd8dd934640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8d8106900 0x7fd8d819ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.789+0000 7fd8dd934640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8d81013d0 0x7fd8d81986f0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.789+0000 7fd8dd934640 1 --2- 172.21.15.98:0/2992291835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d8100a10 0x7fd8d81981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.789+0000 7fd8dd934640 1 -- 172.21.15.98:0/2992291835 >> 172.21.15.98:0/2992291835 conn(0x7fd8d8077710 msgr2=0x7fd8d80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:34.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.789+0000 7fd8dd934640 1 -- 172.21.15.98:0/2992291835 shutdown_connections 2024-05-09T09:02:34.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.789+0000 7fd8dd934640 1 -- 172.21.15.98:0/2992291835 wait complete. 2024-05-09T09:02:34.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T09:02:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.905+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/1931858804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b81011c0 msgr2=0x7fb7b8105680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.905+0000 7fb7bfe2a640 1 --2- 172.21.15.98:0/1931858804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b81011c0 0x7fb7b8105680 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8009900 tx=0x7fb7a802a510 comp rx=0 tx=0).stop 2024-05-09T09:02:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.905+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/1931858804 shutdown_connections 2024-05-09T09:02:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.905+0000 7fb7bfe2a640 1 --2- 172.21.15.98:0/1931858804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7b8105bc0 0x7fb7b8106050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.905+0000 7fb7bfe2a640 1 --2- 172.21.15.98:0/1931858804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b81011c0 0x7fb7b8105680 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.905+0000 7fb7bfe2a640 1 --2- 172.21.15.98:0/1931858804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b81008a0 0x7fb7b8100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.905+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/1931858804 >> 172.21.15.98:0/1931858804 conn(0x7fb7b8077830 msgr2=0x7fb7b8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.905+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/1931858804 shutdown_connections 2024-05-09T09:02:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.905+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/1931858804 wait complete. 2024-05-09T09:02:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7bfe2a640 1 Processor -- start 2024-05-09T09:02:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7bfe2a640 1 -- start start 2024-05-09T09:02:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7bfe2a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7b81008a0 0x7fb7b819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7bfe2a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b81011c0 0x7fb7b819ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7bfe2a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b8105bc0 0x7fb7b81a2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7bfe2a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b81181e0 con 0x7fb7b8105bc0 2024-05-09T09:02:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7bfe2a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b8118060 con 0x7fb7b81008a0 2024-05-09T09:02:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7bfe2a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b8118360 con 0x7fb7b81011c0 2024-05-09T09:02:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7be3c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b8105bc0 0x7fb7b81a2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7be3c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b8105bc0 0x7fb7b81a2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51370/0 (socket says 172.21.15.98:51370) 2024-05-09T09:02:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7be3c7640 1 -- 172.21.15.98:0/2894993991 learned_addr learned my addr 172.21.15.98:0/2894993991 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7bd3c5640 1 --2- 172.21.15.98:0/2894993991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b81011c0 0x7fb7b819ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7bdbc6640 1 --2- 172.21.15.98:0/2894993991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7b81008a0 0x7fb7b819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7be3c7640 1 -- 172.21.15.98:0/2894993991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b81011c0 msgr2=0x7fb7b819ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7be3c7640 1 --2- 172.21.15.98:0/2894993991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b81011c0 0x7fb7b819ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7be3c7640 1 -- 172.21.15.98:0/2894993991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7b81008a0 msgr2=0x7fb7b819e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7be3c7640 1 --2- 172.21.15.98:0/2894993991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7b81008a0 0x7fb7b819e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.906+0000 7fb7be3c7640 1 -- 172.21.15.98:0/2894993991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b81a3720 con 0x7fb7b8105bc0 2024-05-09T09:02:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.907+0000 7fb7be3c7640 1 --2- 172.21.15.98:0/2894993991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b8105bc0 0x7fb7b81a2ff0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009fd0 tx=0x7fb7b000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.907+0000 7fb7aeffd640 1 -- 172.21.15.98:0/2894993991 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0017070 con 0x7fb7b8105bc0 2024-05-09T09:02:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.907+0000 7fb7aeffd640 1 -- 172.21.15.98:0/2894993991 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b0003e50 con 0x7fb7b8105bc0 2024-05-09T09:02:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.907+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/2894993991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b81a3a10 con 0x7fb7b8105bc0 2024-05-09T09:02:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.907+0000 7fb7aeffd640 1 -- 172.21.15.98:0/2894993991 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b00106c0 con 0x7fb7b8105bc0 2024-05-09T09:02:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.907+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/2894993991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b81ab2a0 con 0x7fb7b8105bc0 2024-05-09T09:02:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.908+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/2894993991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7b8068650 con 0x7fb7b8105bc0 2024-05-09T09:02:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.909+0000 7fb7aeffd640 1 -- 172.21.15.98:0/2894993991 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb7b0003970 con 0x7fb7b8105bc0 2024-05-09T09:02:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.910+0000 7fb7aeffd640 1 --2- 172.21.15.98:0/2894993991 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb7a40a2d00 0x7fb7a40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.910+0000 7fb7bdbc6640 1 --2- 172.21.15.98:0/2894993991 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb7a40a2d00 0x7fb7a40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.910+0000 7fb7aeffd640 1 -- 172.21.15.98:0/2894993991 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7fb7b00cf640 con 0x7fb7b8105bc0 2024-05-09T09:02:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.910+0000 7fb7bdbc6640 1 --2- 172.21.15.98:0/2894993991 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb7a40a2d00 0x7fb7a40a51c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4005fd0 tx=0x7fb7b4005850 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:34.914+0000 7fb7aeffd640 1 -- 172.21.15.98:0/2894993991 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb7b009be50 con 0x7fb7b8105bc0 2024-05-09T09:02:35.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:35 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2749224544' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:02:35.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:35 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2992291835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:02:35.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.096+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/2894993991 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb7b81ab4a0 con 0x7fb7b8105bc0 2024-05-09T09:02:35.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.096+0000 7fb7aeffd640 1 -- 172.21.15.98:0/2894993991 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) ==== 104+0+11 (secure 0 0 0) 0x7fb7b009b750 con 0x7fb7b8105bc0 2024-05-09T09:02:35.096 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T09:02:35.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.098+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/2894993991 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb7a40a2d00 msgr2=0x7fb7a40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.098+0000 7fb7bfe2a640 1 --2- 172.21.15.98:0/2894993991 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb7a40a2d00 0x7fb7a40a51c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4005fd0 tx=0x7fb7b4005850 comp rx=0 tx=0).stop 2024-05-09T09:02:35.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.098+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/2894993991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b8105bc0 msgr2=0x7fb7b81a2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.098+0000 7fb7bfe2a640 1 --2- 172.21.15.98:0/2894993991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b8105bc0 0x7fb7b81a2ff0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009fd0 tx=0x7fb7b000b260 comp rx=0 tx=0).stop 2024-05-09T09:02:35.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.098+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/2894993991 shutdown_connections 2024-05-09T09:02:35.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.098+0000 7fb7bfe2a640 1 --2- 172.21.15.98:0/2894993991 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb7a40a2d00 0x7fb7a40a51c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.098+0000 7fb7bfe2a640 1 --2- 172.21.15.98:0/2894993991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7b8105bc0 0x7fb7b81a2ff0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.098+0000 7fb7bfe2a640 1 --2- 172.21.15.98:0/2894993991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b81011c0 0x7fb7b819ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.098+0000 7fb7bfe2a640 1 --2- 172.21.15.98:0/2894993991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7b81008a0 0x7fb7b819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.098+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/2894993991 >> 172.21.15.98:0/2894993991 conn(0x7fb7b8077830 msgr2=0x7fb7b8103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:35.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.099+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/2894993991 shutdown_connections 2024-05-09T09:02:35.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.099+0000 7fb7bfe2a640 1 -- 172.21.15.98:0/2894993991 wait complete. 2024-05-09T09:02:35.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T09:02:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.213+0000 7f5a699d5640 1 -- 172.21.15.98:0/2947736627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a64106b90 msgr2=0x7f5a6410e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.213+0000 7f5a699d5640 1 --2- 172.21.15.98:0/2947736627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a64106b90 0x7f5a6410e7a0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009ca0 tx=0x7f5a5c02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.213+0000 7f5a699d5640 1 -- 172.21.15.98:0/2947736627 shutdown_connections 2024-05-09T09:02:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.213+0000 7f5a699d5640 1 --2- 172.21.15.98:0/2947736627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a64106b90 0x7f5a6410e7a0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.213+0000 7f5a699d5640 1 --2- 172.21.15.98:0/2947736627 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a64100d80 0x7f5a641064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.213+0000 7f5a699d5640 1 --2- 172.21.15.98:0/2947736627 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5a641003d0 0x7f5a641007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.213+0000 7f5a699d5640 1 -- 172.21.15.98:0/2947736627 >> 172.21.15.98:0/2947736627 conn(0x7f5a640759c0 msgr2=0x7f5a64075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.213+0000 7f5a699d5640 1 -- 172.21.15.98:0/2947736627 shutdown_connections 2024-05-09T09:02:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.213+0000 7f5a699d5640 1 -- 172.21.15.98:0/2947736627 wait complete. 2024-05-09T09:02:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a699d5640 1 Processor -- start 2024-05-09T09:02:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a699d5640 1 -- start start 2024-05-09T09:02:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a699d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5a641003d0 0x7f5a64198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a699d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a64106b90 0x7f5a64198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a699d5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a6419c9c0 0x7f5a6419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a699d5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a64111c80 con 0x7f5a64106b90 2024-05-09T09:02:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a699d5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5a64111b00 con 0x7f5a641003d0 2024-05-09T09:02:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a699d5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5a64111e00 con 0x7f5a6419c9c0 2024-05-09T09:02:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a5bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a64106b90 0x7f5a64198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a5bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a64106b90 0x7f5a64198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51386/0 (socket says 172.21.15.98:51386) 2024-05-09T09:02:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a5bfff640 1 -- 172.21.15.98:0/3700482880 learned_addr learned my addr 172.21.15.98:0/3700482880 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a62ffd640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5a641003d0 0x7f5a64198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a637fe640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a6419c9c0 0x7f5a6419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a5bfff640 1 -- 172.21.15.98:0/3700482880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a6419c9c0 msgr2=0x7f5a6419ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a5bfff640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a6419c9c0 0x7f5a6419ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a5bfff640 1 -- 172.21.15.98:0/3700482880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5a641003d0 msgr2=0x7f5a64198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a5bfff640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5a641003d0 0x7f5a64198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a5bfff640 1 -- 172.21.15.98:0/3700482880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6419d4b0 con 0x7f5a64106b90 2024-05-09T09:02:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.214+0000 7f5a62ffd640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5a641003d0 0x7f5a64198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:02:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.215+0000 7f5a5bfff640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a64106b90 0x7f5a64198640 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c00d770 tx=0x7f5a4c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.215+0000 7f5a689d3640 1 -- 172.21.15.98:0/3700482880 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c017070 con 0x7f5a64106b90 2024-05-09T09:02:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.215+0000 7f5a689d3640 1 -- 172.21.15.98:0/3700482880 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a4c009c30 con 0x7f5a64106b90 2024-05-09T09:02:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.215+0000 7f5a699d5640 1 -- 172.21.15.98:0/3700482880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a6419d7a0 con 0x7f5a64106b90 2024-05-09T09:02:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.215+0000 7f5a689d3640 1 -- 172.21.15.98:0/3700482880 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c00f620 con 0x7f5a64106b90 2024-05-09T09:02:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.215+0000 7f5a699d5640 1 -- 172.21.15.98:0/3700482880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a641a4f90 con 0x7f5a64106b90 2024-05-09T09:02:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.216+0000 7f5a699d5640 1 -- 172.21.15.98:0/3700482880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a64068aa0 con 0x7f5a64106b90 2024-05-09T09:02:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.217+0000 7f5a689d3640 1 -- 172.21.15.98:0/3700482880 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5a4c01a070 con 0x7f5a64106b90 2024-05-09T09:02:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.218+0000 7f5a689d3640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5a500a2d00 0x7f5a500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.218+0000 7f5a62ffd640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5a500a2d00 0x7f5a500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.218+0000 7f5a689d3640 1 -- 172.21.15.98:0/3700482880 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7f5a4c0cf3a0 con 0x7f5a64106b90 2024-05-09T09:02:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.218+0000 7f5a62ffd640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5a500a2d00 0x7f5a500a51c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5a54005fd0 tx=0x7f5a5400b5f0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:35.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.221+0000 7f5a689d3640 1 -- 172.21.15.98:0/3700482880 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5a4c009de0 con 0x7f5a64106b90 2024-05-09T09:02:35.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.398+0000 7f5a699d5640 1 -- 172.21.15.98:0/3700482880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5a64100d80 con 0x7f5a64106b90 2024-05-09T09:02:35.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.398+0000 7f5a689d3640 1 -- 172.21.15.98:0/3700482880 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) ==== 100+0+11 (secure 0 0 0) 0x7f5a4c09bbb0 con 0x7f5a64106b90 2024-05-09T09:02:35.397 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-09T09:02:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.400+0000 7f5a699d5640 1 -- 172.21.15.98:0/3700482880 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5a500a2d00 msgr2=0x7f5a500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.400+0000 7f5a699d5640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5a500a2d00 0x7f5a500a51c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5a54005fd0 tx=0x7f5a5400b5f0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.400+0000 7f5a699d5640 1 -- 172.21.15.98:0/3700482880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a64106b90 msgr2=0x7f5a64198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.400+0000 7f5a699d5640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a64106b90 0x7f5a64198640 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c00d770 tx=0x7f5a4c00dc30 comp rx=0 tx=0).stop 2024-05-09T09:02:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.400+0000 7f5a699d5640 1 -- 172.21.15.98:0/3700482880 shutdown_connections 2024-05-09T09:02:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.400+0000 7f5a699d5640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f5a500a2d00 0x7f5a500a51c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.400+0000 7f5a699d5640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a6419c9c0 0x7f5a6419ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.400+0000 7f5a699d5640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a64106b90 0x7f5a64198640 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.400+0000 7f5a699d5640 1 --2- 172.21.15.98:0/3700482880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5a641003d0 0x7f5a64198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.400+0000 7f5a699d5640 1 -- 172.21.15.98:0/3700482880 >> 172.21.15.98:0/3700482880 conn(0x7f5a640759c0 msgr2=0x7f5a640fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.400+0000 7f5a699d5640 1 -- 172.21.15.98:0/3700482880 shutdown_connections 2024-05-09T09:02:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.400+0000 7f5a699d5640 1 -- 172.21.15.98:0/3700482880 wait complete. 2024-05-09T09:02:35.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:35 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2749224544' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:02:35.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:35 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2992291835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:02:35.410 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T09:02:35.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:35 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2749224544' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:02:35.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:35 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2992291835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:02:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.515+0000 7fb23d0df640 1 -- 172.21.15.98:0/2985424004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb238106710 msgr2=0x7fb238106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.515+0000 7fb23d0df640 1 --2- 172.21.15.98:0/2985424004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb238106710 0x7fb238106af0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fb228009cd0 tx=0x7fb22802a8e0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.515+0000 7fb23d0df640 1 -- 172.21.15.98:0/2985424004 shutdown_connections 2024-05-09T09:02:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.515+0000 7fb23d0df640 1 --2- 172.21.15.98:0/2985424004 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb238101110 0x7fb23810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.515+0000 7fb23d0df640 1 --2- 172.21.15.98:0/2985424004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb238100750 0x7fb238100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.515+0000 7fb23d0df640 1 --2- 172.21.15.98:0/2985424004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb238106710 0x7fb238106af0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.515+0000 7fb23d0df640 1 -- 172.21.15.98:0/2985424004 >> 172.21.15.98:0/2985424004 conn(0x7fb238077890 msgr2=0x7fb238077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.515+0000 7fb23d0df640 1 -- 172.21.15.98:0/2985424004 shutdown_connections 2024-05-09T09:02:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.515+0000 7fb23d0df640 1 -- 172.21.15.98:0/2985424004 wait complete. 2024-05-09T09:02:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.516+0000 7fb23d0df640 1 Processor -- start 2024-05-09T09:02:35.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.516+0000 7fb23d0df640 1 -- start start 2024-05-09T09:02:35.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.516+0000 7fb23d0df640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb238100750 0x7fb2380fd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:35.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.516+0000 7fb23d0df640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb238101110 0x7fb2380fdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:35.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.516+0000 7fb23d0df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb238106710 0x7fb2380fe170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.516+0000 7fb23d0df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb238110ae0 con 0x7fb238106710 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.516+0000 7fb23d0df640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb238110960 con 0x7fb238100750 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.516+0000 7fb23d0df640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb238110c60 con 0x7fb238101110 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.516+0000 7fb23759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb238106710 0x7fb2380fe170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.516+0000 7fb23759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb238106710 0x7fb2380fe170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51404/0 (socket says 172.21.15.98:51404) 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.517+0000 7fb23759e640 1 -- 172.21.15.98:0/129864463 learned_addr learned my addr 172.21.15.98:0/129864463 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.517+0000 7fb236d9d640 1 --2- 172.21.15.98:0/129864463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb238100750 0x7fb2380fd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.517+0000 7fb23659c640 1 --2- 172.21.15.98:0/129864463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb238101110 0x7fb2380fdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.517+0000 7fb23759e640 1 -- 172.21.15.98:0/129864463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb238101110 msgr2=0x7fb2380fdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.517+0000 7fb23759e640 1 --2- 172.21.15.98:0/129864463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb238101110 0x7fb2380fdc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.517+0000 7fb23759e640 1 -- 172.21.15.98:0/129864463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb238100750 msgr2=0x7fb2380fd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.517+0000 7fb23759e640 1 --2- 172.21.15.98:0/129864463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb238100750 0x7fb2380fd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.517+0000 7fb23759e640 1 -- 172.21.15.98:0/129864463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2381a6530 con 0x7fb238106710 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.517+0000 7fb23759e640 1 --2- 172.21.15.98:0/129864463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb238106710 0x7fb2380fe170 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fb23000c970 tx=0x7fb23000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.517+0000 7fb21ffff640 1 -- 172.21.15.98:0/129864463 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb230007bb0 con 0x7fb238106710 2024-05-09T09:02:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.517+0000 7fb23d0df640 1 -- 172.21.15.98:0/129864463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2381a6820 con 0x7fb238106710 2024-05-09T09:02:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.517+0000 7fb23d0df640 1 -- 172.21.15.98:0/129864463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2381a6d70 con 0x7fb238106710 2024-05-09T09:02:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.517+0000 7fb21ffff640 1 -- 172.21.15.98:0/129864463 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb230007d50 con 0x7fb238106710 2024-05-09T09:02:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.517+0000 7fb21ffff640 1 -- 172.21.15.98:0/129864463 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23000f450 con 0x7fb238106710 2024-05-09T09:02:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.519+0000 7fb23d0df640 1 -- 172.21.15.98:0/129864463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb224005290 con 0x7fb238106710 2024-05-09T09:02:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.520+0000 7fb21ffff640 1 -- 172.21.15.98:0/129864463 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb2300046c0 con 0x7fb238106710 2024-05-09T09:02:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.521+0000 7fb21ffff640 1 --2- 172.21.15.98:0/129864463 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb2200a2db0 0x7fb2200a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.521+0000 7fb236d9d640 1 --2- 172.21.15.98:0/129864463 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb2200a2db0 0x7fb2200a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.521+0000 7fb21ffff640 1 -- 172.21.15.98:0/129864463 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7fb2300cf2f0 con 0x7fb238106710 2024-05-09T09:02:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.521+0000 7fb236d9d640 1 --2- 172.21.15.98:0/129864463 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb2200a2db0 0x7fb2200a5270 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb228006010 tx=0x7fb228009900 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:35.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.525+0000 7fb21ffff640 1 -- 172.21.15.98:0/129864463 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb230014030 con 0x7fb238106710 2024-05-09T09:02:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.702+0000 7fb23d0df640 1 -- 172.21.15.98:0/129864463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb224005850 con 0x7fb238106710 2024-05-09T09:02:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.703+0000 7fb21ffff640 1 -- 172.21.15.98:0/129864463 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v109) ==== 103+0+11 (secure 0 0 0) 0x7fb23009bb00 con 0x7fb238106710 2024-05-09T09:02:35.702 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T09:02:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.705+0000 7fb23d0df640 1 -- 172.21.15.98:0/129864463 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb2200a2db0 msgr2=0x7fb2200a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.705+0000 7fb23d0df640 1 --2- 172.21.15.98:0/129864463 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb2200a2db0 0x7fb2200a5270 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb228006010 tx=0x7fb228009900 comp rx=0 tx=0).stop 2024-05-09T09:02:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.705+0000 7fb23d0df640 1 -- 172.21.15.98:0/129864463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb238106710 msgr2=0x7fb2380fe170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.705+0000 7fb23d0df640 1 --2- 172.21.15.98:0/129864463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb238106710 0x7fb2380fe170 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fb23000c970 tx=0x7fb23000ce30 comp rx=0 tx=0).stop 2024-05-09T09:02:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.705+0000 7fb23d0df640 1 -- 172.21.15.98:0/129864463 shutdown_connections 2024-05-09T09:02:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.705+0000 7fb23d0df640 1 --2- 172.21.15.98:0/129864463 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fb2200a2db0 0x7fb2200a5270 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.705+0000 7fb23d0df640 1 --2- 172.21.15.98:0/129864463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb238106710 0x7fb2380fe170 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.705+0000 7fb23d0df640 1 --2- 172.21.15.98:0/129864463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb238101110 0x7fb2380fdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.705+0000 7fb23d0df640 1 --2- 172.21.15.98:0/129864463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb238100750 0x7fb2380fd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.705+0000 7fb23d0df640 1 -- 172.21.15.98:0/129864463 >> 172.21.15.98:0/129864463 conn(0x7fb238077890 msgr2=0x7fb2380ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.705+0000 7fb23d0df640 1 -- 172.21.15.98:0/129864463 shutdown_connections 2024-05-09T09:02:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.705+0000 7fb23d0df640 1 -- 172.21.15.98:0/129864463 wait complete. 2024-05-09T09:02:35.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-05-09T09:02:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.817+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2502739471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf80100970 msgr2=0x7fbf8010e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.817+0000 7fbf7f59e640 1 --2- 172.21.15.98:0/2502739471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf80100970 0x7fbf8010e640 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fbf78009cd0 tx=0x7fbf7802a8e0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.817+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2502739471 shutdown_connections 2024-05-09T09:02:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.817+0000 7fbf7f59e640 1 --2- 172.21.15.98:0/2502739471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf80100970 0x7fbf8010e640 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.817+0000 7fbf7f59e640 1 --2- 172.21.15.98:0/2502739471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf800fffb0 0x7fbf80100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.817+0000 7fbf7f59e640 1 --2- 172.21.15.98:0/2502739471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf801035f0 0x7fbf801039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.817+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2502739471 >> 172.21.15.98:0/2502739471 conn(0x7fbf80075810 msgr2=0x7fbf80075c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.817+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2502739471 shutdown_connections 2024-05-09T09:02:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.817+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2502739471 wait complete. 2024-05-09T09:02:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7f59e640 1 Processor -- start 2024-05-09T09:02:35.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7f59e640 1 -- start start 2024-05-09T09:02:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7f59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf800fffb0 0x7fbf80198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7f59e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf80100970 0x7fbf80198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7f59e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf801035f0 0x7fbf8019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7f59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf80111be0 con 0x7fbf800fffb0 2024-05-09T09:02:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7f59e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbf80111a60 con 0x7fbf80100970 2024-05-09T09:02:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7f59e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbf80111d60 con 0x7fbf801035f0 2024-05-09T09:02:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf800fffb0 0x7fbf80198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf800fffb0 0x7fbf80198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57692/0 (socket says 172.21.15.98:57692) 2024-05-09T09:02:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7e59c640 1 -- 172.21.15.98:0/2289098956 learned_addr learned my addr 172.21.15.98:0/2289098956 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:02:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7dd9b640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf80100970 0x7fbf80198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7e59c640 1 -- 172.21.15.98:0/2289098956 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf801035f0 msgr2=0x7fbf8019c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7ed9d640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf801035f0 0x7fbf8019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7e59c640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf801035f0 0x7fbf8019c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7e59c640 1 -- 172.21.15.98:0/2289098956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf80100970 msgr2=0x7fbf80198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7e59c640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf80100970 0x7fbf80198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7e59c640 1 -- 172.21.15.98:0/2289098956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8019cfe0 con 0x7fbf800fffb0 2024-05-09T09:02:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7dd9b640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf80100970 0x7fbf80198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.818+0000 7fbf7ed9d640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf801035f0 0x7fbf8019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:02:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.819+0000 7fbf7e59c640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf800fffb0 0x7fbf80198040 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fbf7000c920 tx=0x7fbf7000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.819+0000 7fbf6f7fe640 1 -- 172.21.15.98:0/2289098956 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf70007c70 con 0x7fbf800fffb0 2024-05-09T09:02:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.819+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2289098956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8019d2d0 con 0x7fbf800fffb0 2024-05-09T09:02:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.819+0000 7fbf6f7fe640 1 -- 172.21.15.98:0/2289098956 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf70007e10 con 0x7fbf800fffb0 2024-05-09T09:02:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.819+0000 7fbf6f7fe640 1 -- 172.21.15.98:0/2289098956 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf70018890 con 0x7fbf800fffb0 2024-05-09T09:02:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.819+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2289098956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf801a4b60 con 0x7fbf800fffb0 2024-05-09T09:02:35.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.820+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2289098956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf80069870 con 0x7fbf800fffb0 2024-05-09T09:02:35.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.821+0000 7fbf6f7fe640 1 -- 172.21.15.98:0/2289098956 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fbf70018a70 con 0x7fbf800fffb0 2024-05-09T09:02:35.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.822+0000 7fbf6f7fe640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbf680a2d00 0x7fbf680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:02:35.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.822+0000 7fbf7dd9b640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbf680a2d00 0x7fbf680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:02:35.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.822+0000 7fbf6f7fe640 1 -- 172.21.15.98:0/2289098956 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9562+0+0 (secure 0 0 0) 0x7fbf700cf9e0 con 0x7fbf800fffb0 2024-05-09T09:02:35.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.822+0000 7fbf7dd9b640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbf680a2d00 0x7fbf680a51c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbf800fd220 tx=0x7fbf74005d30 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:02:35.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:35.826+0000 7fbf6f7fe640 1 -- 172.21.15.98:0/2289098956 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbf70014030 con 0x7fbf800fffb0 2024-05-09T09:02:36.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:36.052+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2289098956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fbf80109c90 con 0x7fbf800fffb0 2024-05-09T09:02:36.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:36.060+0000 7fbf6f7fe640 1 -- 172.21.15.98:0/2289098956 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+740900 (secure 0 0 0) 0x7fbf7009c1f0 con 0x7fbf800fffb0 2024-05-09T09:02:36.069 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T09:02:36.069 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:02:36.070 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 26, 2024-05-09T09:02:36.070 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.070 INFO:teuthology.orchestra.run.smithi098.stdout: "active_gid": 15555, 2024-05-09T09:02:36.070 INFO:teuthology.orchestra.run.smithi098.stdout: "active_name": "x", 2024-05-09T09:02:36.070 INFO:teuthology.orchestra.run.smithi098.stdout: "active_addrs": { 2024-05-09T09:02:36.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-05-09T09:02:36.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-05-09T09:02:36.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6800", 2024-05-09T09:02:36.070 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 4136638623 2024-05-09T09:02:36.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-05-09T09:02:36.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6801", 2024-05-09T09:02:36.071 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 4136638623 2024-05-09T09:02:36.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.071 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-05-09T09:02:36.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.071 INFO:teuthology.orchestra.run.smithi098.stdout: "active_addr": "172.21.15.98:6801/4136638623", 2024-05-09T09:02:36.071 INFO:teuthology.orchestra.run.smithi098.stdout: "active_change": "2024-05-09T08:50:38.492729+0000", 2024-05-09T09:02:36.071 INFO:teuthology.orchestra.run.smithi098.stdout: "active_mgr_features": 4540701547738038271, 2024-05-09T09:02:36.071 INFO:teuthology.orchestra.run.smithi098.stdout: "available": true, 2024-05-09T09:02:36.071 INFO:teuthology.orchestra.run.smithi098.stdout: "standbys": [ 2024-05-09T09:02:36.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.071 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 24991, 2024-05-09T09:02:36.071 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "z", 2024-05-09T09:02:36.071 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_features": 4540701547738038271, 2024-05-09T09:02:36.071 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.072 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-05-09T09:02:36.073 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.073 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.073 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.073 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.073 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.073 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.074 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.074 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.074 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.074 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.074 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.074 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.074 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.074 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.074 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.074 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.074 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.074 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.075 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.075 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.075 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.075 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.075 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.075 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.075 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.075 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.075 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.075 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.075 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.075 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.076 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.077 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.077 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.077 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.077 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.077 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.077 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.077 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.077 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.077 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.077 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.077 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.077 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.078 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.078 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.078 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.078 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.078 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.078 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.078 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.078 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.078 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.078 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-05-09T09:02:36.078 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-05-09T09:02:36.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.079 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.080 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.081 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.082 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.082 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-05-09T09:02:36.082 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.082 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.082 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.082 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-05-09T09:02:36.082 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-05-09T09:02:36.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.082 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.082 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.082 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-05-09T09:02:36.082 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.082 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.083 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.084 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-05-09T09:02:36.084 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.084 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.084 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.084 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-05-09T09:02:36.084 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-05-09T09:02:36.084 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.084 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.084 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.084 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.084 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.084 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.085 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-05-09T09:02:36.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.087 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.087 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.087 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.087 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.087 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.087 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.087 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-05-09T09:02:36.087 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.087 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.087 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.087 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-05-09T09:02:36.087 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-05-09T09:02:36.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.088 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.088 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.088 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-05-09T09:02:36.088 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.088 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.088 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.088 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-05-09T09:02:36.088 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T09:02:36.088 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.088 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.088 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-05-09T09:02:36.088 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-05-09T09:02:36.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:36.089 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.089 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.089 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:36.089 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-05-09T09:02:36.089 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-05-09T09:02:36.089 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.089 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-09T09:02:36.089 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T09:02:36.089 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.089 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.089 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-05-09T09:02:36.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-05-09T09:02:36.089 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.090 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.091 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.091 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-05-09T09:02:36.091 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.091 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.091 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.091 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-09T09:02:36.091 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-09T09:02:36.091 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.091 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.091 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-05-09T09:02:36.091 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-05-09T09:02:36.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.091 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.092 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.093 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-05-09T09:02:36.093 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.093 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.093 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.093 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-05-09T09:02:36.093 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T09:02:36.093 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.093 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.093 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-05-09T09:02:36.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-05-09T09:02:36.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:36.093 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.093 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.094 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:36.094 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-05-09T09:02:36.094 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-05-09T09:02:36.094 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.094 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-09T09:02:36.094 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T09:02:36.094 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.094 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.094 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.094 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.094 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.094 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.095 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.095 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.095 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.095 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.095 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.095 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.095 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.095 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.095 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.095 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.095 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.095 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.095 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.096 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.096 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.096 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.096 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.096 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.096 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.096 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.096 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.096 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.096 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.096 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.097 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.097 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.097 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.097 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.097 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.097 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.097 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.097 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.097 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.097 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.097 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.098 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.098 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.098 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.098 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.098 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.098 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.098 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.098 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.098 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.098 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.098 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.098 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.099 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.099 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.099 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.099 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.099 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.099 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.099 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.099 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.099 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.099 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.099 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.099 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-05-09T09:02:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-05-09T09:02:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-09T09:02:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-05-09T09:02:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-05-09T09:02:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-05-09T09:02:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-09T09:02:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-05-09T09:02:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-05-09T09:02:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-05-09T09:02:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-05-09T09:02:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-09T09:02:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-05-09T09:02:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-09T09:02:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-05-09T09:02:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-05-09T09:02:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-05-09T09:02:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-09T09:02:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-05-09T09:02:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-05-09T09:02:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-05-09T09:02:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-09T09:02:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-05-09T09:02:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-05-09T09:02:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-09T09:02:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-05-09T09:02:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-05-09T09:02:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-09T09:02:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-05-09T09:02:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-05-09T09:02:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-05-09T09:02:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-09T09:02:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-05-09T09:02:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-05-09T09:02:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-05-09T09:02:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-09T09:02:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-05-09T09:02:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-05-09T09:02:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-05-09T09:02:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-09T09:02:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-05-09T09:02:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-05-09T09:02:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-05-09T09:02:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-09T09:02:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-05-09T09:02:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.129 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-05-09T09:02:36.129 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-05-09T09:02:36.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.129 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.129 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.129 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-09T09:02:36.129 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.129 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.129 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.129 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-05-09T09:02:36.129 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.129 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.129 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_samba": { 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_samba", 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Samba/SMB container image", 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.130 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-05-09T09:02:36.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.132 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-05-09T09:02:36.133 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.134 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.135 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.136 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.137 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.138 INFO:teuthology.orchestra.run.smithi098.stdout: "grafana_dashboards_path": { 2024-05-09T09:02:36.138 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "grafana_dashboards_path", 2024-05-09T09:02:36.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.138 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.138 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.138 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-09T09:02:36.138 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.138 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.138 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.138 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-09T09:02:36.138 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.138 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.138 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.140 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-05-09T09:02:36.140 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-05-09T09:02:36.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.140 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.140 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.140 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.140 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.140 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.140 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.140 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-09T09:02:36.140 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.140 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.140 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.141 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-05-09T09:02:36.141 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-05-09T09:02:36.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.141 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.141 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.141 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.141 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.141 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.141 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.141 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-09T09:02:36.141 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.141 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.141 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.142 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.143 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.143 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.143 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.143 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.143 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.143 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-05-09T09:02:36.143 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-05-09T09:02:36.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.143 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.143 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.143 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.143 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.144 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.145 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.146 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.146 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.146 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.146 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.146 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.146 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.146 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.146 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.146 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.146 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.146 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.146 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.146 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.147 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.148 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.149 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.150 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.151 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.151 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.151 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-09T09:02:36.151 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.151 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.151 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.151 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-05-09T09:02:36.151 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-05-09T09:02:36.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.151 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.151 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.151 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.151 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.152 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.152 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.152 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-05-09T09:02:36.152 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.152 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.152 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-05-09T09:02:36.152 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-05-09T09:02:36.152 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.152 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.152 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.152 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-05-09T09:02:36.152 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.153 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.154 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-05-09T09:02:36.154 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.154 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.154 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.154 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-09T09:02:36.154 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.154 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.154 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.154 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-05-09T09:02:36.154 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-05-09T09:02:36.154 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.154 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.154 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.155 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-09T09:02:36.155 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.155 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.155 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.155 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-09T09:02:36.155 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.155 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.155 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.155 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-05-09T09:02:36.155 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-05-09T09:02:36.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.155 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.155 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.156 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.157 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.157 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.157 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.157 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-09T09:02:36.157 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.157 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.157 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.157 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-05-09T09:02:36.157 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-05-09T09:02:36.157 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.157 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.157 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.157 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.158 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.158 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.158 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.158 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-09T09:02:36.158 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.158 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.158 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.158 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-05-09T09:02:36.158 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-05-09T09:02:36.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.158 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.158 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.158 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.159 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.160 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-09T09:02:36.161 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.162 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.162 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.162 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-05-09T09:02:36.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-05-09T09:02:36.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.162 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.162 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.162 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.162 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.162 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.162 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.162 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-09T09:02:36.162 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.163 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.164 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-05-09T09:02:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-05-09T09:02:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-09T09:02:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-09T09:02:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-05-09T09:02:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-09T09:02:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-09T09:02:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-09T09:02:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-09T09:02:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-09T09:02:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-09T09:02:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-09T09:02:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-09T09:02:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-09T09:02:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-09T09:02:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-09T09:02:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-09T09:02:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-09T09:02:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-05-09T09:02:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-09T09:02:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-09T09:02:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.194 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-09T09:02:36.194 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-09T09:02:36.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.194 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.194 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.194 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.194 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.194 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.194 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.194 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.194 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.194 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.194 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.196 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-09T09:02:36.197 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.198 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.198 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.198 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.198 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.198 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.198 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.198 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.198 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.198 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.198 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.198 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-09T09:02:36.198 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-09T09:02:36.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.199 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.200 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.201 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.202 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.203 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.204 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.206 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.206 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.206 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.206 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-09T09:02:36.206 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-09T09:02:36.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.206 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.206 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.206 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-09T09:02:36.206 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.206 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.206 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.207 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.208 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.209 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.210 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-05-09T09:02:36.210 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-09T09:02:36.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.210 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.210 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.210 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.210 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.210 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.210 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.210 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.210 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.210 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.210 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.211 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-09T09:02:36.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.213 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.213 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.213 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.213 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.213 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.213 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.213 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.213 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.213 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.213 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.213 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-09T09:02:36.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-09T09:02:36.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.214 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.214 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.214 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:36.214 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.214 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.214 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.214 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.214 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.214 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.214 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.214 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-09T09:02:36.214 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-09T09:02:36.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.215 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.217 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.218 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.218 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.218 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.218 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.218 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.218 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.218 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.218 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-05-09T09:02:36.218 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-05-09T09:02:36.218 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.218 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.219 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.220 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.221 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.222 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.222 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.222 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.222 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.222 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.222 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.222 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.222 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.222 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.222 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.222 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.222 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.222 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.223 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.223 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.223 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.223 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.223 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.223 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.223 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.223 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.223 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.223 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.223 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.223 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.226 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.227 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-05-09T09:02:36.227 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.227 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.227 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.227 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-05-09T09:02:36.227 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-05-09T09:02:36.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.227 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.227 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.227 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.227 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.227 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.227 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.228 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-05-09T09:02:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-05-09T09:02:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-05-09T09:02:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.230 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.231 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.232 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.232 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-05-09T09:02:36.232 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-05-09T09:02:36.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.232 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.232 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.232 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-05-09T09:02:36.232 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.232 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.232 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.232 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.232 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-05-09T09:02:36.232 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.234 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.234 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.234 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-05-09T09:02:36.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-05-09T09:02:36.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.234 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.234 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.234 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.234 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.234 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.234 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.234 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.234 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.235 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.236 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.237 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.238 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.239 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.240 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.241 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.243 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-05-09T09:02:36.244 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-05-09T09:02:36.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.244 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.244 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.244 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-05-09T09:02:36.244 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.244 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.244 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.244 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-09T09:02:36.244 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.244 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.244 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:36.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:36.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.246 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.246 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.246 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:36.246 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.246 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.246 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.246 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-05-09T09:02:36.246 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.246 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.246 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.246 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-05-09T09:02:36.246 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-05-09T09:02:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-05-09T09:02:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-09T09:02:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.248 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.249 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.250 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.250 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.250 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.250 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.250 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.250 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.250 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.250 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.250 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.250 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.250 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.251 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.252 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.253 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-05-09T09:02:36.253 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-05-09T09:02:36.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.253 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.253 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.253 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-05-09T09:02:36.253 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.253 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.253 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.253 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.253 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.253 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.253 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.254 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-05-09T09:02:36.254 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-05-09T09:02:36.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.254 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.254 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.254 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-05-09T09:02:36.254 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.254 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.254 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.254 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.254 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.254 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.254 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.255 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.256 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.257 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.257 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-09T09:02:36.257 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.257 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.257 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.257 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-05-09T09:02:36.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-05-09T09:02:36.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.257 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.257 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.257 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-05-09T09:02:36.257 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.258 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.258 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.258 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-09T09:02:36.258 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.258 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.258 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.258 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-05-09T09:02:36.258 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-05-09T09:02:36.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.258 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.258 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.258 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.258 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.259 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.260 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.261 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.264 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-05-09T09:02:36.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.266 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.267 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-05-09T09:02:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-05-09T09:02:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-05-09T09:02:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-05-09T09:02:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T09:02:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-05-09T09:02:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-05-09T09:02:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-09T09:02:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-05-09T09:02:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-05-09T09:02:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-05-09T09:02:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-09T09:02:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-05-09T09:02:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-05-09T09:02:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-09T09:02:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-09T09:02:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-05-09T09:02:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-05-09T09:02:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-05-09T09:02:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-05-09T09:02:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-05-09T09:02:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-05-09T09:02:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-05-09T09:02:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-05-09T09:02:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-05-09T09:02:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-05-09T09:02:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-09T09:02:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-09T09:02:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-05-09T09:02:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.311 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.311 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.311 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.311 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.311 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.311 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.311 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.311 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.311 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.311 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.311 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.311 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.312 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.313 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.314 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.314 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.314 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.314 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.314 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.314 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.314 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.314 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.314 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.315 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-05-09T09:02:36.315 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.315 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.315 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.315 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-05-09T09:02:36.315 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-05-09T09:02:36.315 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.315 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.315 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.315 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.315 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.315 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.315 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.315 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.316 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.317 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.318 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.319 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.319 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.319 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.319 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.319 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.319 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.319 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.319 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.319 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.319 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.319 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.319 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.319 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.319 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.321 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-05-09T09:02:36.321 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-05-09T09:02:36.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.321 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.321 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.321 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.321 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.321 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.321 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.321 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-05-09T09:02:36.321 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-05-09T09:02:36.321 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-05-09T09:02:36.321 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.321 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-05-09T09:02:36.322 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.322 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.322 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.322 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.322 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-05-09T09:02:36.322 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.322 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.322 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.322 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.323 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.324 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.324 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.324 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.324 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.324 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.324 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.324 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.324 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.324 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.324 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.324 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.325 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.325 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.325 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.325 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.325 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.325 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.325 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.325 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.325 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.325 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.325 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.325 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.326 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.326 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.326 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.326 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.326 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.326 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.326 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.326 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.326 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.326 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.326 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.326 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.326 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.327 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.327 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.327 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.327 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.327 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.327 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.327 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.327 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.327 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.327 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.327 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.328 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-05-09T09:02:36.328 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.328 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.328 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.328 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.328 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.328 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.328 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.328 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.328 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.328 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.329 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.330 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.332 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.332 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.332 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.332 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.332 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.332 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.332 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.332 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.332 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.332 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.332 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.332 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.333 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.335 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.335 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.335 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.335 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.335 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.335 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.335 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.335 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.335 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.335 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.335 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.336 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.337 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.338 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.338 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.338 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:36.338 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:36.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.338 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.338 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.338 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:36.338 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.338 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.338 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.338 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.338 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-09T09:02:36.339 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.340 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.340 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.340 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-05-09T09:02:36.340 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.340 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.340 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.340 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-05-09T09:02:36.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-05-09T09:02:36.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.340 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.340 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.341 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.342 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.342 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.342 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.342 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.342 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.342 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.342 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.342 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.342 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.342 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.342 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.342 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.344 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.345 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.346 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.347 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.348 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.349 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.349 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.349 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-05-09T09:02:36.349 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.349 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.349 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.349 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-05-09T09:02:36.349 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-05-09T09:02:36.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.349 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.349 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.349 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.350 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.350 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.350 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.350 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.350 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.350 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.350 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.350 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-05-09T09:02:36.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-05-09T09:02:36.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.350 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.350 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.350 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.351 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.351 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.351 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.351 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-09T09:02:36.351 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-09T09:02:36.351 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.351 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.351 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.351 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.351 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.351 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-05-09T09:02:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-05-09T09:02:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-05-09T09:02:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-05-09T09:02:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-05-09T09:02:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-05-09T09:02:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-05-09T09:02:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-05-09T09:02:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-09T09:02:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-05-09T09:02:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-05-09T09:02:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-05-09T09:02:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-09T09:02:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-05-09T09:02:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-05-09T09:02:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-05-09T09:02:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-05-09T09:02:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-05-09T09:02:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-05-09T09:02:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-05-09T09:02:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-05-09T09:02:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-05-09T09:02:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-05-09T09:02:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-05-09T09:02:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-05-09T09:02:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-05-09T09:02:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.376 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.377 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.377 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.377 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.377 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.377 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.377 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.377 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.377 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.377 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.377 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.377 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.377 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.378 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.378 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.378 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.378 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.378 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.378 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.378 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.378 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.378 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.378 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.378 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.378 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.378 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-05-09T09:02:36.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-05-09T09:02:36.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.381 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.382 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.383 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.384 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.385 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.387 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.388 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.388 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.388 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.388 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.388 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.388 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.388 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.388 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.388 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.388 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.388 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.389 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.390 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.391 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.393 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-05-09T09:02:36.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-05-09T09:02:36.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.394 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.394 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.394 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-05-09T09:02:36.394 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.394 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.394 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.394 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-09T09:02:36.394 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.394 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.394 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-05-09T09:02:36.395 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.395 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.395 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.395 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.395 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.395 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.395 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.395 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.395 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.395 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.396 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.397 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.398 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.398 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.398 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.398 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.398 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.398 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.398 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.398 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.398 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.398 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.398 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.398 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.398 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.398 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.399 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.399 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.399 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.399 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.399 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.399 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.399 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.399 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.399 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.399 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.399 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.399 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.400 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.401 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-05-09T09:02:36.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-05-09T09:02:36.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.402 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.402 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.402 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.402 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.402 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.402 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.402 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.402 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.402 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.402 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.402 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-05-09T09:02:36.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-05-09T09:02:36.403 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.403 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.403 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.403 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.403 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.403 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.403 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.403 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.403 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.403 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.403 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.405 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-05-09T09:02:36.406 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-05-09T09:02:36.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.406 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.406 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.406 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.406 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.406 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.406 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.406 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.406 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.406 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.406 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.406 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-05-09T09:02:36.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-05-09T09:02:36.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.408 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.408 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.408 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.408 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-05-09T09:02:36.408 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-05-09T09:02:36.408 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.408 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.408 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.408 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.408 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.408 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-05-09T09:02:36.408 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.409 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.410 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.411 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.413 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.414 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.414 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.414 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.414 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.414 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.414 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.414 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.414 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.414 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.414 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.414 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.414 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.414 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.415 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.416 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-05-09T09:02:36.418 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.419 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.419 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.419 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.419 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.419 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.419 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.419 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.419 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.419 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.419 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.419 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.419 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.420 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.421 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.422 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.424 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.425 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.426 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.427 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.428 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-05-09T09:02:36.429 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.429 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.429 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.429 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-05-09T09:02:36.429 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-05-09T09:02:36.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.429 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.429 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.429 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-09T09:02:36.429 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.429 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.429 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.429 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.429 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.430 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.431 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-09T09:02:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-05-09T09:02:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-05-09T09:02:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-09T09:02:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-05-09T09:02:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-05-09T09:02:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-09T09:02:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-05-09T09:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-05-09T09:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-05-09T09:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-05-09T09:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-09T09:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-05-09T09:02:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-05-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-05-09T09:02:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.460 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.461 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.461 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.461 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-05-09T09:02:36.461 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.461 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.461 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.461 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.463 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.464 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.466 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.466 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.466 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.466 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.466 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.466 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.466 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.466 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.466 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-05-09T09:02:36.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-05-09T09:02:36.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.466 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.466 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.467 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-05-09T09:02:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-05-09T09:02:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-05-09T09:02:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.470 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.470 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.470 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-09T09:02:36.470 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.470 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.470 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-05-09T09:02:36.470 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.470 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.470 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.471 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:36.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.474 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.475 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.476 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.478 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.478 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.478 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.478 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.478 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.478 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.478 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.478 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-05-09T09:02:36.478 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-05-09T09:02:36.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.478 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.478 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-05-09T09:02:36.479 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.480 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.481 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.481 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.481 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.481 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.481 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.481 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-05-09T09:02:36.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.481 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 25070, 2024-05-09T09:02:36.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "y", 2024-05-09T09:02:36.481 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_features": 4540701547738038271, 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.482 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.483 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-05-09T09:02:36.483 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.483 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.483 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.483 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.483 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.483 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.483 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.483 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.483 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.483 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.483 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.484 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.484 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.484 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.484 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.484 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.484 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.484 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.484 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.484 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.484 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.484 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.484 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.484 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.485 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.486 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.486 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.486 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.486 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.486 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.486 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.486 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.486 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.486 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.486 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.486 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.486 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.486 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.487 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-05-09T09:02:36.490 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-05-09T09:02:36.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.490 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.490 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.490 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.490 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.490 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.490 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.490 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-05-09T09:02:36.490 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.490 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.490 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.490 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-05-09T09:02:36.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-05-09T09:02:36.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.493 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.494 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.494 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.494 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.494 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.494 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-09T09:02:36.494 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.494 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.494 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.494 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-05-09T09:02:36.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-05-09T09:02:36.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.496 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.498 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-05-09T09:02:36.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-05-09T09:02:36.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:36.498 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.498 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.498 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:36.498 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-05-09T09:02:36.498 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-05-09T09:02:36.498 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.498 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-09T09:02:36.498 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T09:02:36.498 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.498 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.499 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-05-09T09:02:36.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-05-09T09:02:36.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.502 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.502 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.502 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-05-09T09:02:36.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.502 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.502 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.502 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-05-09T09:02:36.502 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T09:02:36.502 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.502 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.502 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-05-09T09:02:36.502 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-05-09T09:02:36.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-09T09:02:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-05-09T09:02:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-05-09T09:02:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-09T09:02:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-09T09:02:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-05-09T09:02:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-05-09T09:02:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-05-09T09:02:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-05-09T09:02:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-05-09T09:02:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-09T09:02:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-05-09T09:02:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-05-09T09:02:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-09T09:02:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-05-09T09:02:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-05-09T09:02:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-09T09:02:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-05-09T09:02:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-05-09T09:02:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-05-09T09:02:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-09T09:02:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-05-09T09:02:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-05-09T09:02:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-05-09T09:02:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-09T09:02:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-05-09T09:02:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-05-09T09:02:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-05-09T09:02:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-09T09:02:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-05-09T09:02:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-05-09T09:02:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-05-09T09:02:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-09T09:02:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-05-09T09:02:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-05-09T09:02:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-05-09T09:02:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-09T09:02:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-05-09T09:02:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-05-09T09:02:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-05-09T09:02:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-09T09:02:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-05-09T09:02:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-05-09T09:02:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-05-09T09:02:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-05-09T09:02:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_samba": { 2024-05-09T09:02:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_samba", 2024-05-09T09:02:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-09T09:02:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Samba/SMB container image", 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-05-09T09:02:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-05-09T09:02:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-05-09T09:02:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-09T09:02:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-05-09T09:02:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-05-09T09:02:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-05-09T09:02:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "grafana_dashboards_path": { 2024-05-09T09:02:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "grafana_dashboards_path", 2024-05-09T09:02:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-09T09:02:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-09T09:02:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-05-09T09:02:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-05-09T09:02:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-05-09T09:02:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-05-09T09:02:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-05-09T09:02:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-09T09:02:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-09T09:02:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-09T09:02:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-09T09:02:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-05-09T09:02:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-05-09T09:02:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-05-09T09:02:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-05-09T09:02:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-05-09T09:02:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-05-09T09:02:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-05-09T09:02:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-05-09T09:02:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-09T09:02:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-05-09T09:02:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-05-09T09:02:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-09T09:02:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-05-09T09:02:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-05-09T09:02:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-05-09T09:02:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-05-09T09:02:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-05-09T09:02:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-09T09:02:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-05-09T09:02:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-05-09T09:02:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-09T09:02:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-05-09T09:02:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-05-09T09:02:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-09T09:02:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-05-09T09:02:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-05-09T09:02:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-09T09:02:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-05-09T09:02:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-05-09T09:02:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-05-09T09:02:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-05-09T09:02:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-09T09:02:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-05-09T09:02:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-05-09T09:02:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-09T09:02:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-09T09:02:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-09T09:02:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-09T09:02:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-09T09:02:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-09T09:02:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-09T09:02:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-09T09:02:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-09T09:02:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-09T09:02:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-09T09:02:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-09T09:02:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-09T09:02:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-09T09:02:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-05-09T09:02:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-05-09T09:02:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-09T09:02:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-09T09:02:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-09T09:02:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-09T09:02:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-09T09:02:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-09T09:02:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-09T09:02:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-09T09:02:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-09T09:02:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-09T09:02:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-09T09:02:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-09T09:02:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-09T09:02:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-09T09:02:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-09T09:02:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-09T09:02:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-09T09:02:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-09T09:02:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-05-09T09:02:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-05-09T09:02:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-09T09:02:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-09T09:02:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-09T09:02:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-05-09T09:02:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-05-09T09:02:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-05-09T09:02:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-05-09T09:02:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-05-09T09:02:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-05-09T09:02:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-05-09T09:02:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-05-09T09:02:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-05-09T09:02:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-05-09T09:02:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-05-09T09:02:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-05-09T09:02:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-05-09T09:02:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-05-09T09:02:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-05-09T09:02:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-05-09T09:02:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-05-09T09:02:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-05-09T09:02:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-05-09T09:02:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-09T09:02:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-05-09T09:02:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-05-09T09:02:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-05-09T09:02:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-05-09T09:02:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-05-09T09:02:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-05-09T09:02:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-05-09T09:02:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-09T09:02:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-05-09T09:02:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-05-09T09:02:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-09T09:02:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-05-09T09:02:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-05-09T09:02:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-05-09T09:02:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-05-09T09:02:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-09T09:02:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-05-09T09:02:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-05-09T09:02:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-05-09T09:02:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-05-09T09:02:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-05-09T09:02:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-05-09T09:02:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-05-09T09:02:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-05-09T09:02:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-05-09T09:02:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-05-09T09:02:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-05-09T09:02:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-05-09T09:02:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-05-09T09:02:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-05-09T09:02:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-05-09T09:02:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-05-09T09:02:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-05-09T09:02:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-09T09:02:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-05-09T09:02:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-05-09T09:02:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.707 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.707 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.707 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.707 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.707 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-05-09T09:02:36.707 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-09T09:02:36.707 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.707 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.708 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-05-09T09:02:36.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-05-09T09:02:36.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.708 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.708 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.708 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-05-09T09:02:36.708 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-05-09T09:02:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-05-09T09:02:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-05-09T09:02:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-05-09T09:02:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-05-09T09:02:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-05-09T09:02:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-05-09T09:02:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-05-09T09:02:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-05-09T09:02:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-05-09T09:02:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-05-09T09:02:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-09T09:02:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-09T09:02:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-05-09T09:02:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-05-09T09:02:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.724 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.724 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.724 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.724 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.724 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.724 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.724 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.724 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.724 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.724 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.724 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.724 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.724 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.725 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.725 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.725 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.725 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.725 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.725 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.725 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.725 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.725 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.725 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.725 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.725 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-05-09T09:02:36.726 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.726 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.726 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.726 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.726 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.726 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.726 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.726 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.726 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.726 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.726 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.726 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.726 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.727 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.727 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.727 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.727 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.727 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.727 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.727 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.727 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.727 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.727 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.727 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.727 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-05-09T09:02:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-05-09T09:02:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-05-09T09:02:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-05-09T09:02:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-05-09T09:02:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-05-09T09:02:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.755 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.756 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.756 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.756 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.756 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.756 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.756 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.756 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.756 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.756 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.756 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.757 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.757 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.757 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.757 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.757 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.757 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.757 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.757 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.757 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:36.757 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:36.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.757 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.757 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.758 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:36.758 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.758 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.758 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.758 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.758 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.758 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.758 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.758 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-05-09T09:02:36.758 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-05-09T09:02:36.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.759 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.759 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.759 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-05-09T09:02:36.759 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-05-09T09:02:36.759 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.759 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.759 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-05-09T09:02:36.759 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-09T09:02:36.759 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.759 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.759 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.759 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.760 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-05-09T09:02:36.760 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.760 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.760 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.760 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-05-09T09:02:36.760 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-05-09T09:02:36.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.760 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.760 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.760 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.760 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.760 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.760 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.760 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-09T09:02:36.761 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.761 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.761 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.761 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-05-09T09:02:36.761 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-05-09T09:02:36.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.761 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.761 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.761 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.761 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.761 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.761 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.761 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.762 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.762 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.762 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.762 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.762 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.762 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.762 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.762 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.762 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.762 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.762 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.764 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.765 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.766 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.766 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.766 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.766 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.766 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.766 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.766 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.766 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.766 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.766 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.766 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.766 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.767 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.768 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.769 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.769 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-05-09T09:02:36.769 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.769 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.769 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.769 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-05-09T09:02:36.769 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-05-09T09:02:36.769 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.769 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.769 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.769 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.770 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.771 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.772 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.773 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.773 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.773 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.773 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.773 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.773 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.773 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.773 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.773 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.773 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.773 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.773 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.774 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.774 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.774 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.774 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.774 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.774 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.774 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.774 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.774 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.774 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.774 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.774 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.774 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.774 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.775 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.775 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.775 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.775 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.775 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.775 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.775 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.775 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.775 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.775 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.775 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.775 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.775 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.776 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.776 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.776 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.776 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.776 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-05-09T09:02:36.776 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-05-09T09:02:36.776 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.776 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.776 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.776 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.776 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.776 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.777 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.778 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.778 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.778 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.778 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.778 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-05-09T09:02:36.778 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-05-09T09:02:36.778 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.778 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.778 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.778 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-05-09T09:02:36.778 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.778 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.780 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-09T09:02:36.781 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.781 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.781 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.781 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-05-09T09:02:36.781 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-05-09T09:02:36.781 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.781 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.781 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.781 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-05-09T09:02:36.781 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.781 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.781 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-05-09T09:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-05-09T09:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-05-09T09:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.784 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-05-09T09:02:36.784 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-05-09T09:02:36.784 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-05-09T09:02:36.784 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.784 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.784 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.784 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.784 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.784 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-05-09T09:02:36.784 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.785 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.785 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.785 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.785 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.785 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.785 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.785 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.785 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.785 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.785 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.785 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.785 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.785 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.786 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.786 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.786 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.786 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.786 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.786 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.786 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.786 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.786 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.786 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.786 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.787 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.788 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-05-09T09:02:36.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.790 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.790 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.790 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.790 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.790 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.790 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.790 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.790 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.790 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.790 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.790 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-05-09T09:02:36.790 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-05-09T09:02:36.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.791 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.791 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.791 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.791 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.791 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.791 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.791 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.791 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.791 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.791 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.791 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-05-09T09:02:36.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-05-09T09:02:36.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.792 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-05-09T09:02:36.793 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.793 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.793 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.793 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-05-09T09:02:36.793 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-05-09T09:02:36.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.793 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.793 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.793 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.793 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.793 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.793 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.793 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.793 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-05-09T09:02:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-05-09T09:02:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.794 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.795 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.799 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.799 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.799 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.799 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.799 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.799 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.799 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.799 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.799 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.799 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.799 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.799 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.799 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.800 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.802 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.803 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.804 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.804 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.804 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.804 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.804 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.804 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.804 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.804 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.804 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.804 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.804 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.805 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.806 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.806 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.806 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.806 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.806 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.806 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.806 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.806 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.806 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.806 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.806 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.806 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.807 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.808 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.808 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.808 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.808 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.808 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.808 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.808 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.808 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.808 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.808 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.808 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.808 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.808 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.808 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.809 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.809 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.809 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.809 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.809 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.809 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.809 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.809 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.809 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.809 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.809 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.809 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.811 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.812 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.813 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.813 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-09T09:02:36.813 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.813 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.813 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.813 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-05-09T09:02:36.813 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.813 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.813 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.813 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.814 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.814 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.814 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.814 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.814 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.814 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.814 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.814 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.814 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.814 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.814 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.814 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.815 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.816 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.818 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.818 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.818 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.818 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.818 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.818 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.818 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.818 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.818 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.818 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.818 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-05-09T09:02:36.818 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-05-09T09:02:36.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.819 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.820 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.821 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.822 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.822 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.822 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.822 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.822 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.822 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.822 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.822 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-05-09T09:02:36.822 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-05-09T09:02:36.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.822 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.822 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.822 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.823 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.823 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.823 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.823 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.823 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.823 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.823 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.823 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-05-09T09:02:36.823 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-05-09T09:02:36.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.823 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.823 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.823 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.824 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-05-09T09:02:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-05-09T09:02:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-05-09T09:02:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-05-09T09:02:36.826 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.827 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.828 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.829 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.830 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.830 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-05-09T09:02:36.830 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.830 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.830 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.830 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-05-09T09:02:36.830 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-05-09T09:02:36.830 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.830 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.830 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.831 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.831 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.831 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.831 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.831 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-05-09T09:02:36.831 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.831 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.831 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.831 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-05-09T09:02:36.831 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-05-09T09:02:36.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.831 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.831 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.832 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.832 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.832 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.832 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.832 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-05-09T09:02:36.832 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.832 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.832 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.832 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.832 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.832 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.832 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.833 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.835 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-05-09T09:02:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-05-09T09:02:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-05-09T09:02:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-09T09:02:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-05-09T09:02:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-05-09T09:02:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-05-09T09:02:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-05-09T09:02:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-09T09:02:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-05-09T09:02:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-05-09T09:02:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-09T09:02:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-05-09T09:02:36.859 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.860 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.861 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.861 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.861 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.861 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.861 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.861 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.861 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.861 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.861 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.861 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-05-09T09:02:36.861 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-05-09T09:02:36.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.861 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.862 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.862 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.862 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.862 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.862 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.862 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.862 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.862 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.862 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.862 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-05-09T09:02:36.862 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-05-09T09:02:36.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.862 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-09T09:02:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-05-09T09:02:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-05-09T09:02:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.866 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-05-09T09:02:36.866 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.866 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.866 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.866 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.866 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.866 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.866 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.866 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-05-09T09:02:36.866 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-05-09T09:02:36.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.866 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.866 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.867 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.868 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.869 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.870 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.870 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.870 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.870 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.870 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.870 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.870 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.870 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.870 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.870 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.870 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.870 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.871 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.872 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.874 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-05-09T09:02:36.874 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-05-09T09:02:36.874 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.874 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.874 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.874 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.874 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.874 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.874 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.874 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.874 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.874 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.874 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.875 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.876 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.877 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.878 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.878 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.878 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.878 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.878 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.878 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.878 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.878 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.878 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.878 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.878 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.878 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.878 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.879 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.880 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.880 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.880 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.880 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.880 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.880 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.880 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.880 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.880 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.880 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.880 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.880 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.880 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-05-09T09:02:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.881 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.882 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-05-09T09:02:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-05-09T09:02:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-09T09:02:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-05-09T09:02:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-05-09T09:02:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-09T09:02:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-05-09T09:02:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-09T09:02:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "modules": [ 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-05-09T09:02:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "dashboard", 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "iostat", 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "nfs", 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "restful", 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_schedule" 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-05-09T09:02:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-05-09T09:02:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.909 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.911 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-05-09T09:02:36.912 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.913 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.914 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-05-09T09:02:36.914 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-05-09T09:02:36.914 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.914 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.914 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.914 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.914 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.914 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.914 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.914 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-09T09:02:36.914 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.914 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.914 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.915 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.916 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.917 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-05-09T09:02:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T09:02:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-05-09T09:02:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-05-09T09:02:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-05-09T09:02:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-05-09T09:02:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-09T09:02:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-09T09:02:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-05-09T09:02:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-05-09T09:02:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-05-09T09:02:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-05-09T09:02:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-05-09T09:02:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-05-09T09:02:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-09T09:02:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-05-09T09:02:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-05-09T09:02:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-05-09T09:02:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-05-09T09:02:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-05-09T09:02:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-05-09T09:02:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-05-09T09:02:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-05-09T09:02:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-05-09T09:02:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-05-09T09:02:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-05-09T09:02:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-09T09:02:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-05-09T09:02:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-05-09T09:02:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-05-09T09:02:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-05-09T09:02:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-09T09:02:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-09T09:02:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-05-09T09:02:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-05-09T09:02:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-09T09:02:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-05-09T09:02:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-05-09T09:02:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-05-09T09:02:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-05-09T09:02:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-09T09:02:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-05-09T09:02:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-05-09T09:02:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-05-09T09:02:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-09T09:02:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-05-09T09:02:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-05-09T09:02:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-05-09T09:02:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-09T09:02:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-05-09T09:02:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-05-09T09:02:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-09T09:02:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-09T09:02:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-05-09T09:02:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-05-09T09:02:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-09T09:02:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-05-09T09:02:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-05-09T09:02:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-05-09T09:02:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-09T09:02:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-05-09T09:02:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-05-09T09:02:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.943 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.943 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.943 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.943 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.943 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.943 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-05-09T09:02:36.943 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-05-09T09:02:36.943 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-05-09T09:02:36.943 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.943 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-09T09:02:36.943 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.943 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.943 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-05-09T09:02:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-05-09T09:02:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-09T09:02:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.945 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-05-09T09:02:36.945 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-05-09T09:02:36.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.945 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.945 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.945 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.945 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.945 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.945 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.945 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-09T09:02:36.945 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.945 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-05-09T09:02:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-05-09T09:02:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-09T09:02:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.947 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-05-09T09:02:36.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-05-09T09:02:36.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.947 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.947 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.947 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-09T09:02:36.947 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.947 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.947 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.947 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-05-09T09:02:36.947 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.947 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.947 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.948 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-05-09T09:02:36.948 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-05-09T09:02:36.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.948 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.948 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.948 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-09T09:02:36.948 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.948 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.948 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.948 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-05-09T09:02:36.948 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.948 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.948 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.949 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-05-09T09:02:36.949 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-05-09T09:02:36.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.949 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.949 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.949 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-09T09:02:36.949 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.949 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.949 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.949 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-05-09T09:02:36.949 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.949 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.949 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.950 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-05-09T09:02:36.950 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-05-09T09:02:36.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.950 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.950 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.950 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-09T09:02:36.950 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.950 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.950 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.950 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-05-09T09:02:36.950 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.950 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.950 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.952 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-05-09T09:02:36.952 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-05-09T09:02:36.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.952 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.952 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.952 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-09T09:02:36.952 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.952 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.952 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.952 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-05-09T09:02:36.952 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.952 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.952 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.953 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-05-09T09:02:36.953 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-05-09T09:02:36.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.953 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.953 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.953 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-09T09:02:36.953 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.953 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.953 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.953 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-05-09T09:02:36.953 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.953 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.953 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.954 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-05-09T09:02:36.954 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-05-09T09:02:36.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.954 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.954 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.954 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-09T09:02:36.954 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.954 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.954 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.954 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-05-09T09:02:36.954 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.954 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.954 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-05-09T09:02:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-05-09T09:02:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-09T09:02:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-05-09T09:02:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.957 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-05-09T09:02:36.957 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-05-09T09:02:36.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.957 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.957 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.957 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-09T09:02:36.957 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.957 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.957 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.957 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-05-09T09:02:36.957 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.957 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.957 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.959 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_samba": { 2024-05-09T09:02:36.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_samba", 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Samba/SMB container image", 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-05-09T09:02:36.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.962 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.962 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.962 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-09T09:02:36.962 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.962 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.962 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.962 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-05-09T09:02:36.962 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.962 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.962 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.962 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-05-09T09:02:36.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-05-09T09:02:36.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.963 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-09T09:02:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-05-09T09:02:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-05-09T09:02:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.965 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.965 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-05-09T09:02:36.965 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.965 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.965 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.965 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-09T09:02:36.965 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.965 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.965 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.965 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-05-09T09:02:36.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-05-09T09:02:36.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.965 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.966 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.966 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-05-09T09:02:36.966 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.966 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.966 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.966 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-09T09:02:36.966 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.966 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.966 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.966 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-05-09T09:02:36.967 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-05-09T09:02:36.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.967 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.967 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.967 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-05-09T09:02:36.967 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.967 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.967 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.967 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-05-09T09:02:36.967 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.967 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.967 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.968 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-05-09T09:02:36.968 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-05-09T09:02:36.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.968 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.968 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.968 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.968 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.968 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.968 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.968 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-09T09:02:36.968 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.968 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.968 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "grafana_dashboards_path": { 2024-05-09T09:02:36.969 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "grafana_dashboards_path", 2024-05-09T09:02:36.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.970 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.970 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.970 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-09T09:02:36.970 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.970 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.970 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.970 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-09T09:02:36.970 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.970 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.970 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.970 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-05-09T09:02:36.970 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-05-09T09:02:36.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:36.971 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.971 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.971 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:36.971 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.971 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.971 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.971 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-05-09T09:02:36.971 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.971 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.971 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.971 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-05-09T09:02:36.972 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-05-09T09:02:36.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.972 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.972 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.972 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.972 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.972 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.972 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.972 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-09T09:02:36.972 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.972 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.972 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.972 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-05-09T09:02:36.973 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-05-09T09:02:36.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.973 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.973 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.973 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.973 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.973 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.973 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.973 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-09T09:02:36.973 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.973 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.973 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.973 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:36.974 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:36.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.974 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.974 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.974 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.974 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.974 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.974 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.974 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.974 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.974 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.974 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.974 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.975 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.976 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-09T09:02:36.976 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.976 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.976 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.976 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:36.976 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:36.976 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.976 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.976 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.976 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.976 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.976 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.976 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-09T09:02:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.977 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.978 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:36.978 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:36.978 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:36.978 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:36.978 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:36.978 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:36.978 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.978 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.978 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.978 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.978 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.978 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:36.978 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-09T09:02:36.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.980 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.981 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.981 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-05-09T09:02:36.981 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.981 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.981 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.981 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-09T09:02:36.981 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.981 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.981 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.981 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-05-09T09:02:36.981 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-05-09T09:02:36.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.981 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.982 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.982 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.982 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.982 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.982 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.982 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-05-09T09:02:36.982 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.982 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.982 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.982 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-05-09T09:02:36.982 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-05-09T09:02:36.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.982 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-05-09T09:02:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-05-09T09:02:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-05-09T09:02:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.985 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.986 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.986 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-05-09T09:02:36.986 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.986 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.986 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.986 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-09T09:02:36.986 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.986 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.986 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.986 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-05-09T09:02:36.986 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-05-09T09:02:36.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.987 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.987 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.987 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-09T09:02:36.987 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.987 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.987 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.987 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-09T09:02:36.987 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.987 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.987 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.987 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-05-09T09:02:36.987 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-05-09T09:02:36.987 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.988 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.988 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.988 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.988 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.988 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.988 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.988 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-09T09:02:36.988 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.988 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.988 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.988 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-05-09T09:02:36.988 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-05-09T09:02:36.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.989 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.990 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.992 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.992 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.992 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.992 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-09T09:02:36.992 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.992 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.992 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.992 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-05-09T09:02:36.992 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-05-09T09:02:36.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:36.992 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.993 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.994 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:36.994 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.994 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.994 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.994 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-09T09:02:36.994 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.994 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.994 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-05-09T09:02:36.994 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-05-09T09:02:36.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.994 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.995 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.995 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:36.995 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.995 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.995 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.995 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-09T09:02:36.995 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.995 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.995 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.995 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-05-09T09:02:36.995 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-05-09T09:02:36.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.996 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-05-09T09:02:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-05-09T09:02:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-05-09T09:02:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.997 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:36.999 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:36.999 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:36.999 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:36.999 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:36.999 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-09T09:02:36.999 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:36.999 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:36.999 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:36.999 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.999 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:36.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:36.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:36.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-05-09T09:02:36.999 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.000 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.000 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.000 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.000 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.000 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.000 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.000 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.000 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.000 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.000 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.000 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.000 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.002 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.002 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.002 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.002 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.002 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.002 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.002 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.002 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.002 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.002 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.002 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.002 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.003 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.003 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.003 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.003 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.003 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.003 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.003 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.003 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.003 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.003 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.003 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.003 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.003 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.003 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.005 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-05-09T09:02:37.005 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-05-09T09:02:37.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:37.005 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.005 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.005 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-05-09T09:02:37.005 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.005 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.005 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.005 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-09T09:02:37.005 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.005 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.005 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.006 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-05-09T09:02:37.006 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-05-09T09:02:37.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:37.006 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.006 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.006 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-05-09T09:02:37.006 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.006 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.006 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.006 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-09T09:02:37.006 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.006 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.006 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.007 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.007 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.007 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-05-09T09:02:37.007 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.007 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.007 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.007 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-09T09:02:37.007 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-09T09:02:37.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.007 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.007 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.007 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:37.008 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.008 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.008 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.008 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.008 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.008 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.008 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.008 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-05-09T09:02:37.008 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-09T09:02:37.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.008 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.008 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.008 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.009 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.009 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.009 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.009 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.009 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.009 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.009 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.009 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-09T09:02:37.009 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-09T09:02:37.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.009 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.009 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.009 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.010 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.010 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.010 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.010 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.010 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.010 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.010 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.010 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-05-09T09:02:37.010 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-05-09T09:02:37.010 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.010 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.010 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.010 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.011 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.011 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.011 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.011 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.011 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.011 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.011 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.011 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-09T09:02:37.011 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-09T09:02:37.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.011 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.011 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.011 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-05-09T09:02:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-09T09:02:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.013 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.013 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.013 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.013 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.013 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.013 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.013 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.013 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-09T09:02:37.013 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-09T09:02:37.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.013 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.013 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.013 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.014 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.014 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.014 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.014 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.014 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.014 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.014 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.014 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-09T09:02:37.014 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-09T09:02:37.014 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.014 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.014 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.014 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.015 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.015 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.015 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.015 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.015 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.015 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.015 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.015 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-09T09:02:37.015 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-09T09:02:37.015 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.015 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.015 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.015 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.016 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.016 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.016 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.016 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.016 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.016 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.016 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.016 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-09T09:02:37.016 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-09T09:02:37.016 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.016 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.016 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.016 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.017 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.017 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.017 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.017 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.017 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.017 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.017 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.017 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-09T09:02:37.017 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-09T09:02:37.017 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.017 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.017 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.018 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.019 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.020 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.020 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.020 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.020 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.020 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.020 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.020 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-09T09:02:37.020 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-09T09:02:37.020 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.020 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.020 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.020 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.020 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.021 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-05-09T09:02:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-05-09T09:02:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-05-09T09:02:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-09T09:02:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-05-09T09:02:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-09T09:02:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-09T09:02:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-09T09:02:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-09T09:02:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-09T09:02:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-09T09:02:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-09T09:02:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-09T09:02:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-09T09:02:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-09T09:02:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-09T09:02:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-09T09:02:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-09T09:02:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-09T09:02:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-09T09:02:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-09T09:02:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-09T09:02:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-09T09:02:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-09T09:02:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-09T09:02:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-09T09:02:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-05-09T09:02:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-05-09T09:02:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-09T09:02:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-05-09T09:02:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-05-09T09:02:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-05-09T09:02:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-05-09T09:02:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-05-09T09:02:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-05-09T09:02:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-05-09T09:02:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-05-09T09:02:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-05-09T09:02:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-05-09T09:02:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-05-09T09:02:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-05-09T09:02:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-05-09T09:02:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-05-09T09:02:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-05-09T09:02:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-05-09T09:02:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-05-09T09:02:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-05-09T09:02:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-05-09T09:02:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-05-09T09:02:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-05-09T09:02:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-05-09T09:02:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-05-09T09:02:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-05-09T09:02:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-09T09:02:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-05-09T09:02:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-05-09T09:02:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-05-09T09:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-09T09:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-05-09T09:02:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-05-09T09:02:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-05-09T09:02:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-05-09T09:02:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-05-09T09:02:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-05-09T09:02:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-05-09T09:02:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-09T09:02:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-05-09T09:02:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-05-09T09:02:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-09T09:02:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-05-09T09:02:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-05-09T09:02:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-05-09T09:02:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-05-09T09:02:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-09T09:02:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-05-09T09:02:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-05-09T09:02:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-05-09T09:02:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-05-09T09:02:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-05-09T09:02:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-05-09T09:02:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-05-09T09:02:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-05-09T09:02:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-05-09T09:02:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-05-09T09:02:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-05-09T09:02:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-05-09T09:02:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-05-09T09:02:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-09T09:02:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-05-09T09:02:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-05-09T09:02:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-05-09T09:02:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-09T09:02:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-05-09T09:02:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-05-09T09:02:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-05-09T09:02:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-05-09T09:02:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-05-09T09:02:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-05-09T09:02:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-05-09T09:02:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-05-09T09:02:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-05-09T09:02:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-05-09T09:02:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-05-09T09:02:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-05-09T09:02:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-05-09T09:02:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-05-09T09:02:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-09T09:02:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-05-09T09:02:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-05-09T09:02:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-05-09T09:02:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-05-09T09:02:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-05-09T09:02:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-05-09T09:02:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-05-09T09:02:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-05-09T09:02:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-09T09:02:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-05-09T09:02:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-05-09T09:02:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-05-09T09:02:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-09T09:02:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-09T09:02:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.130 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.132 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-05-09T09:02:37.133 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-05-09T09:02:37.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.133 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.133 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.133 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.133 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.133 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.133 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.133 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-05-09T09:02:37.133 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-09T09:02:37.133 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.133 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.133 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-05-09T09:02:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-05-09T09:02:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-05-09T09:02:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-05-09T09:02:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-05-09T09:02:37.135 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-05-09T09:02:37.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.136 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.136 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.136 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.136 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.136 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.136 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.136 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-05-09T09:02:37.136 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.136 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.136 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.137 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-05-09T09:02:37.137 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-05-09T09:02:37.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.137 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.137 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.137 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-05-09T09:02:37.137 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.137 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.137 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.137 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-09T09:02:37.137 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-09T09:02:37.137 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.137 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.138 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-05-09T09:02:37.138 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.138 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.138 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.138 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.138 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.138 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.138 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.139 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.139 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.139 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.139 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.139 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.139 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.139 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.139 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.139 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.139 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.139 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.139 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.139 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.139 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.141 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.142 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.142 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.142 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.142 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.142 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.142 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.142 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.142 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.142 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.142 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.142 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.143 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.144 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.145 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.146 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.146 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.146 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.146 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.146 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.146 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.146 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.146 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.146 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.146 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.146 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.147 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-05-09T09:02:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.149 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.150 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.150 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.150 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.150 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.150 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.150 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.150 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.150 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.150 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.150 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.150 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.152 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.152 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.152 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.152 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.152 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.152 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.152 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.152 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.152 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.152 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.152 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.152 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.153 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.153 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.153 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.153 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.153 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.153 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.153 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.153 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.153 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.153 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.153 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.154 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.154 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.154 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.154 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.154 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.154 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.154 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-05-09T09:02:37.154 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.154 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.154 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.156 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.157 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.157 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.157 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.157 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.157 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.157 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.157 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.157 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.157 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.157 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.157 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.157 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.157 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.158 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.158 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.158 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.158 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.158 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.158 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.158 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.158 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.158 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.158 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.158 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.158 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.159 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.159 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.159 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.159 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.159 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.159 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.159 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.159 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.159 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.159 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.159 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.159 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.159 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-05-09T09:02:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.161 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-05-09T09:02:37.162 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.162 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.162 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.162 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.162 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.162 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.162 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.164 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.164 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.164 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.164 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.164 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.164 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.164 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.164 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.164 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.164 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.164 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.165 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.165 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.165 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.165 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.165 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.165 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.165 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.165 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.165 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.165 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.165 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.165 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.166 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.166 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.166 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.166 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.166 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.166 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.166 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.166 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.166 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.166 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-05-09T09:02:37.167 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.168 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.169 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.170 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.170 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.170 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.170 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.170 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.170 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.170 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.170 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.170 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.170 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.170 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.171 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.171 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.171 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.171 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.171 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.171 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.171 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.171 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.171 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.171 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.171 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.171 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.171 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.172 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.173 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.175 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.175 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.175 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.175 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.175 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.175 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.175 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.175 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.175 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.175 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.175 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.175 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.176 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.176 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.176 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.176 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.176 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.176 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.176 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.176 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.176 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.176 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.176 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.176 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.176 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.176 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.177 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.177 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.177 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.177 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.177 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.177 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.177 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.177 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.177 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.177 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.177 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.178 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.178 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.178 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.178 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.178 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:37.178 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:37.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:37.178 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.178 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.178 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:37.178 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.178 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.179 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.180 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-05-09T09:02:37.180 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-09T09:02:37.180 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.180 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.180 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.180 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.180 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-05-09T09:02:37.180 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.180 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.180 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.180 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-05-09T09:02:37.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-05-09T09:02:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-05-09T09:02:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-05-09T09:02:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-05-09T09:02:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-05-09T09:02:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-05-09T09:02:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-05-09T09:02:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-05-09T09:02:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-05-09T09:02:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-05-09T09:02:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-05-09T09:02:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-05-09T09:02:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-05-09T09:02:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-05-09T09:02:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-05-09T09:02:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.209 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.209 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.210 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.210 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.210 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.210 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.210 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.210 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.210 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.210 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.210 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.210 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.210 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.210 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.211 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.211 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.211 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.211 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.211 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.211 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.211 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.211 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.211 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.211 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.211 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.211 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.211 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.212 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.212 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.212 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.212 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.212 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.212 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.212 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.212 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.212 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.212 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.212 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.212 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-05-09T09:02:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-05-09T09:02:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-05-09T09:02:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-05-09T09:02:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-05-09T09:02:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-05-09T09:02:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-05-09T09:02:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-05-09T09:02:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-05-09T09:02:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-05-09T09:02:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-05-09T09:02:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-05-09T09:02:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.225 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.225 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.225 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.225 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.225 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.225 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.225 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-05-09T09:02:37.225 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.225 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.225 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.225 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.226 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.226 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.226 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.226 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.226 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.226 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.226 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.226 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.226 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.226 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.226 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.226 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.227 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.230 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.230 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.230 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.230 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.230 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.230 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.230 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.230 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.230 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.230 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.230 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.230 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.231 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-05-09T09:02:37.231 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.231 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.231 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.231 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.231 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.231 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.231 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.231 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.231 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.231 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.232 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.232 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.232 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.232 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.232 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.232 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.232 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.232 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.232 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.232 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.232 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.232 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.232 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.234 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.234 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.234 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.234 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.234 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.234 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.234 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.234 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.234 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.234 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.234 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.234 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.234 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.235 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.236 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.236 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.236 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.236 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-05-09T09:02:37.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-05-09T09:02:37.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.236 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.236 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.236 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-05-09T09:02:37.236 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.236 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.236 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.236 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-09T09:02:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-05-09T09:02:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.239 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.240 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.240 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.240 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.240 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.240 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.240 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.240 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.240 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.240 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.240 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.240 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.241 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.241 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.241 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.241 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.241 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.241 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.241 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.241 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.241 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.241 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.241 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.242 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.242 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.242 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.242 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.242 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.242 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.242 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.242 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.242 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.242 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-05-09T09:02:37.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-05-09T09:02:37.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.242 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-05-09T09:02:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-05-09T09:02:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.243 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.244 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.245 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.245 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.245 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.245 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.245 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.245 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.245 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-05-09T09:02:37.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-05-09T09:02:37.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.245 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-05-09T09:02:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-05-09T09:02:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-05-09T09:02:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.247 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.248 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.249 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.249 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.249 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.249 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.249 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.249 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-05-09T09:02:37.249 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-05-09T09:02:37.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.249 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.249 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.249 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.249 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.250 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.250 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.250 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.250 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.250 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.250 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-05-09T09:02:37.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-05-09T09:02:37.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.250 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.250 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.250 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.250 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-05-09T09:02:37.251 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-05-09T09:02:37.251 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.251 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.251 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.251 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.251 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.251 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-05-09T09:02:37.251 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-05-09T09:02:37.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.251 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.251 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.251 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.251 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.252 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.252 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.252 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.252 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.252 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.252 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.252 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-05-09T09:02:37.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-05-09T09:02:37.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.252 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.252 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.252 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.253 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.253 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.253 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.253 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.253 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.253 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.253 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.253 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-05-09T09:02:37.253 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-05-09T09:02:37.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.253 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.253 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.253 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.254 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.254 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.254 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.254 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.254 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.254 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.254 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.254 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-05-09T09:02:37.254 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.254 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.256 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-05-09T09:02:37.256 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-05-09T09:02:37.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.256 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.256 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.256 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.256 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.256 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.256 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.256 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-05-09T09:02:37.256 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.256 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.256 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.258 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.258 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.258 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.258 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.258 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.258 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.258 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.258 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.258 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.258 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.258 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.258 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.259 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.259 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.259 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.259 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.259 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.259 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.259 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.259 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.259 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.259 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.259 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.259 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.260 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.260 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.260 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.260 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.260 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.260 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.260 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.260 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.260 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.260 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.260 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.260 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.260 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.261 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.261 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.261 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.261 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.261 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.261 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.261 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.261 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.261 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.261 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.261 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.261 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.261 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.261 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.262 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-05-09T09:02:37.262 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.262 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.262 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.262 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.262 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.262 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.262 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.262 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.263 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.265 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.265 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.265 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.265 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.265 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.265 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.265 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.265 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.265 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.265 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.265 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.265 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.266 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.266 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.266 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.266 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.266 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.266 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.266 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.266 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.266 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.266 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.266 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.268 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.269 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.269 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.269 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.269 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.269 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.269 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.269 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.269 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.269 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.269 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.269 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.269 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.270 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.270 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.270 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.270 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.270 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.270 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.270 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.270 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.270 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.270 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.270 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.270 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.271 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-05-09T09:02:37.272 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.273 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.274 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:37.274 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:37.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:37.274 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.274 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.274 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-05-09T09:02:37.274 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.274 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.274 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.274 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.274 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.274 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.274 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.275 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.278 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.278 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.278 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.278 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.278 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.278 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.278 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.278 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.278 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.278 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.278 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.278 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.279 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.279 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.279 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.279 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.279 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.279 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.279 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.279 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.279 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.279 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.279 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.279 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-05-09T09:02:37.280 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.280 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.280 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.280 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-05-09T09:02:37.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-05-09T09:02:37.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.280 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.280 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.280 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.280 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.281 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.281 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.281 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-09T09:02:37.281 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.281 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.281 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.281 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-05-09T09:02:37.281 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-05-09T09:02:37.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.281 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.281 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.281 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-09T09:02:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-05-09T09:02:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-05-09T09:02:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.282 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.283 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.283 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.283 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.283 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-09T09:02:37.283 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.283 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.283 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.283 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-05-09T09:02:37.283 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-05-09T09:02:37.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.283 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.283 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.283 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.284 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.284 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.284 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.284 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-09T09:02:37.284 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.284 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.284 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.284 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-05-09T09:02:37.284 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-05-09T09:02:37.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.284 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.284 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.284 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-05-09T09:02:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-05-09T09:02:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-05-09T09:02:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-05-09T09:02:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-09T09:02:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-05-09T09:02:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-05-09T09:02:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-05-09T09:02:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-05-09T09:02:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-05-09T09:02:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-05-09T09:02:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-05-09T09:02:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-05-09T09:02:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-05-09T09:02:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-09T09:02:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.309 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.309 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.309 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.309 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.309 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.309 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.309 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.309 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.309 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.309 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.309 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.310 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.310 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.310 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.310 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.310 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.310 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.310 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.310 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.310 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.310 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.310 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.310 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.310 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.311 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.311 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.311 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.311 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.311 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.311 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.311 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.311 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.311 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.311 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.311 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.311 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.312 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.312 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.312 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-05-09T09:02:37.312 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-05-09T09:02:37.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.312 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.312 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.312 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-05-09T09:02:37.312 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.312 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.312 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.312 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-09T09:02:37.313 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.313 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.313 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.313 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-05-09T09:02:37.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-05-09T09:02:37.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.313 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.313 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.313 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.313 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.313 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.313 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.313 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-05-09T09:02:37.314 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.314 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.314 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.314 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-05-09T09:02:37.314 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-05-09T09:02:37.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.314 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.314 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.314 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-05-09T09:02:37.314 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.314 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.314 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.314 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-09T09:02:37.315 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.315 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.315 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.315 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-05-09T09:02:37.315 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-05-09T09:02:37.315 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.315 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.315 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.315 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-05-09T09:02:37.315 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.315 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.315 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.315 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-09T09:02:37.316 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.316 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.316 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-05-09T09:02:37.316 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-05-09T09:02:37.316 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-05-09T09:02:37.316 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-05-09T09:02:37.316 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-05-09T09:02:37.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-05-09T09:02:37.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-05-09T09:02:37.317 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.317 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.317 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-05-09T09:02:37.317 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.317 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.317 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.317 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.317 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.317 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.317 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-05-09T09:02:37.317 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-05-09T09:02:37.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.318 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.318 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.318 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.318 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.318 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.318 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.318 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.318 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.318 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.318 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.318 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-05-09T09:02:37.318 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-05-09T09:02:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.320 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.321 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.322 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.322 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-05-09T09:02:37.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-05-09T09:02:37.322 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.322 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.322 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.322 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-05-09T09:02:37.322 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.322 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.322 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-05-09T09:02:37.322 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-05-09T09:02:37.322 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-05-09T09:02:37.323 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-05-09T09:02:37.323 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-05-09T09:02:37.323 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-05-09T09:02:37.323 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-05-09T09:02:37.323 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.323 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.323 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.323 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.323 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.323 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-05-09T09:02:37.323 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-05-09T09:02:37.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-05-09T09:02:37.323 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-05-09T09:02:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-05-09T09:02:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-05-09T09:02:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-05-09T09:02:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-05-09T09:02:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-05-09T09:02:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-05-09T09:02:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-05-09T09:02:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-05-09T09:02:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-05-09T09:02:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-05-09T09:02:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-05-09T09:02:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-05-09T09:02:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-05-09T09:02:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-09T09:02:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-05-09T09:02:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-05-09T09:02:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-05-09T09:02:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-05-09T09:02:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-05-09T09:02:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-05-09T09:02:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-05-09T09:02:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "services": { 2024-05-09T09:02:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "dashboard": "https://172.21.15.98:8443/" 2024-05-09T09:02:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "always_on_modules": { 2024-05-09T09:02:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus": [ 2024-05-09T09:02:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-05-09T09:02:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-05-09T09:02:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-05-09T09:02:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-05-09T09:02:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-05-09T09:02:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-05-09T09:02:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-05-09T09:02:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-05-09T09:02:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-05-09T09:02:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific": [ 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy": [ 2024-05-09T09:02:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-05-09T09:02:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-05-09T09:02:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-05-09T09:02:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-05-09T09:02:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-05-09T09:02:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-05-09T09:02:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-05-09T09:02:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-05-09T09:02:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-05-09T09:02:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-05-09T09:02:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "reef": [ 2024-05-09T09:02:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-05-09T09:02:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-05-09T09:02:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-05-09T09:02:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-05-09T09:02:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-05-09T09:02:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-05-09T09:02:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-05-09T09:02:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-05-09T09:02:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-05-09T09:02:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-05-09T09:02:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-05-09T09:02:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "squid": [ 2024-05-09T09:02:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-05-09T09:02:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-05-09T09:02:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-05-09T09:02:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-05-09T09:02:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "last_failure_osd_epoch": 95, 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "active_clients": [ 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-05-09T09:02:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3222875421 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "libcephsqlite", 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1324741939 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-05-09T09:02:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-05-09T09:02:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-05-09T09:02:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-05-09T09:02:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3009181090 2024-05-09T09:02:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-05-09T09:02:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-05-09T09:02:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-05-09T09:02:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 694492892 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 78049063 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-05-09T09:02:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-05-09T09:02:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-05-09T09:02:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-09T09:02:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-05-09T09:02:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-05-09T09:02:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2441496877 2024-05-09T09:02:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-05-09T09:02:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-05-09T09:02:37.336 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:02:37.338 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:36 smithi105 ceph-mon[34498]: pgmap v375: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 11 op/s 2024-05-09T09:02:37.338 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:36 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2894993991' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:02:37.338 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:36 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3700482880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:02:37.338 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:36 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/129864463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:02:37.338 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:36 smithi142 ceph-mon[34625]: pgmap v375: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 11 op/s 2024-05-09T09:02:37.338 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:36 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2894993991' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:02:37.338 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:36 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3700482880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:02:37.338 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:36 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/129864463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:02:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:36.065+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2289098956 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbf680a2d00 msgr2=0x7fbf680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:36.065+0000 7fbf7f59e640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbf680a2d00 0x7fbf680a51c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbf800fd220 tx=0x7fbf74005d30 comp rx=0 tx=0).stop 2024-05-09T09:02:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:36.065+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2289098956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf800fffb0 msgr2=0x7fbf80198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:02:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:36.065+0000 7fbf7f59e640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf800fffb0 0x7fbf80198040 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fbf7000c920 tx=0x7fbf7000cde0 comp rx=0 tx=0).stop 2024-05-09T09:02:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:36.065+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2289098956 shutdown_connections 2024-05-09T09:02:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:36.065+0000 7fbf7f59e640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fbf680a2d00 0x7fbf680a51c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:36.065+0000 7fbf7f59e640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf801035f0 0x7fbf8019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:36.065+0000 7fbf7f59e640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf80100970 0x7fbf80198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:36.065+0000 7fbf7f59e640 1 --2- 172.21.15.98:0/2289098956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf800fffb0 0x7fbf80198040 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:02:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:36.065+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2289098956 >> 172.21.15.98:0/2289098956 conn(0x7fbf80075810 msgr2=0x7fbf800fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:02:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:36.065+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2289098956 shutdown_connections 2024-05-09T09:02:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:02:36.065+0000 7fbf7f59e640 1 -- 172.21.15.98:0/2289098956 wait complete. 2024-05-09T09:02:37.346 DEBUG:tasks.check_counter:Getting stats from a 2024-05-09T09:02:37.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph daemon mds.a perf dump 2024-05-09T09:02:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:36 smithi098 ceph-mon[30092]: pgmap v375: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 11 op/s 2024-05-09T09:02:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:36 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2894993991' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:02:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:36 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3700482880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:02:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:36 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/129864463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:02:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:37 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/2289098956' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-09T09:02:37.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:37 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/2289098956' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-09T09:02:37.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:37 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/2289098956' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-09T09:02:37.599 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:02:38.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:38 smithi098 ceph-mon[30092]: pgmap v376: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-05-09T09:02:38.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:38 smithi142 ceph-mon[34625]: pgmap v376: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-05-09T09:02:38.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:38 smithi105 ceph-mon[34498]: pgmap v376: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-05-09T09:02:39.498 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:02:39.498 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:02:39.498 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1677, 2024-05-09T09:02:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 1679, 2024-05-09T09:02:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 502845, 2024-05-09T09:02:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 533287, 2024-05-09T09:02:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 6, 2024-05-09T09:02:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-05-09T09:02:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.274567301, 2024-05-09T09:02:39.500 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.051259456, 2024-05-09T09:02:39.500 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.050433415, 2024-05-09T09:02:39.500 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.153876963, 2024-05-09T09:02:39.500 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:39.500 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1677, 2024-05-09T09:02:39.500 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.128580732, 2024-05-09T09:02:39.500 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000076673 2024-05-09T09:02:39.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.501 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:39.501 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.501 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.501 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.502 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T09:02:39.502 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T09:02:39.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.502 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:02:39.502 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 999, 2024-05-09T09:02:39.502 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 979, 2024-05-09T09:02:39.503 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1844588, 2024-05-09T09:02:39.503 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 377076, 2024-05-09T09:02:39.503 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-05-09T09:02:39.503 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-05-09T09:02:39.503 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.161161963, 2024-05-09T09:02:39.503 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.039228910, 2024-05-09T09:02:39.504 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.061458627, 2024-05-09T09:02:39.504 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.064026975, 2024-05-09T09:02:39.504 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:39.504 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 979, 2024-05-09T09:02:39.504 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.090733996, 2024-05-09T09:02:39.504 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000092680 2024-05-09T09:02:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1666816, 2024-05-09T09:02:39.506 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 148048 2024-05-09T09:02:39.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.506 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:02:39.506 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 3, 2024-05-09T09:02:39.506 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 179, 2024-05-09T09:02:39.506 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 588, 2024-05-09T09:02:39.507 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 473966, 2024-05-09T09:02:39.507 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 6, 2024-05-09T09:02:39.507 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 4, 2024-05-09T09:02:39.507 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.014820213, 2024-05-09T09:02:39.507 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.010463146, 2024-05-09T09:02:39.507 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.002080275, 2024-05-09T09:02:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000016106, 2024-05-09T09:02:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 176, 2024-05-09T09:02:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.019117607, 2024-05-09T09:02:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000108622 2024-05-09T09:02:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-09T09:02:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 473648 2024-05-09T09:02:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-05-09T09:02:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-05-09T09:02:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-05-09T09:02:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-05-09T09:02:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-05-09T09:02:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-05-09T09:02:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 3923, 2024-05-09T09:02:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.127683946, 2024-05-09T09:02:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000032547 2024-05-09T09:02:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-05-09T09:02:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-05-09T09:02:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-05-09T09:02:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-05-09T09:02:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 0, 2024-05-09T09:02:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 0, 2024-05-09T09:02:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-05-09T09:02:39.514 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.514 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.514 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.514 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-05-09T09:02:39.514 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 0, 2024-05-09T09:02:39.515 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 0, 2024-05-09T09:02:39.515 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-05-09T09:02:39.515 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 0, 2024-05-09T09:02:39.515 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 0, 2024-05-09T09:02:39.515 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-05-09T09:02:39.515 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 0, 2024-05-09T09:02:39.515 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 0, 2024-05-09T09:02:39.516 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 0, 2024-05-09T09:02:39.516 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 0, 2024-05-09T09:02:39.516 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 0, 2024-05-09T09:02:39.516 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 0, 2024-05-09T09:02:39.516 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 0, 2024-05-09T09:02:39.516 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 0, 2024-05-09T09:02:39.517 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 1, 2024-05-09T09:02:39.517 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 0, 2024-05-09T09:02:39.517 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 0, 2024-05-09T09:02:39.517 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 0, 2024-05-09T09:02:39.517 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 0, 2024-05-09T09:02:39.517 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:02:39.518 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-05-09T09:02:39.518 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 0, 2024-05-09T09:02:39.518 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-05-09T09:02:39.518 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-05-09T09:02:39.518 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 0, 2024-05-09T09:02:39.518 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 0, 2024-05-09T09:02:39.519 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 0, 2024-05-09T09:02:39.519 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 0, 2024-05-09T09:02:39.519 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-05-09T09:02:39.519 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 0, 2024-05-09T09:02:39.519 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-05-09T09:02:39.519 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 0, 2024-05-09T09:02:39.519 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 0, 2024-05-09T09:02:39.520 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 0, 2024-05-09T09:02:39.520 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T09:02:39.520 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-05-09T09:02:39.520 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:02:39.520 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:02:39.520 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 0, 2024-05-09T09:02:39.520 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T09:02:39.521 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T09:02:39.521 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 0, 2024-05-09T09:02:39.521 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 0, 2024-05-09T09:02:39.521 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T09:02:39.521 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 0, 2024-05-09T09:02:39.521 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:02:39.521 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:02:39.522 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:02:39.522 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 0, 2024-05-09T09:02:39.522 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 0, 2024-05-09T09:02:39.522 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 0, 2024-05-09T09:02:39.522 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 0 2024-05-09T09:02:39.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.523 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-05-09T09:02:39.523 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 0, 2024-05-09T09:02:39.523 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 0, 2024-05-09T09:02:39.523 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 0, 2024-05-09T09:02:39.523 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 0, 2024-05-09T09:02:39.523 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 0, 2024-05-09T09:02:39.524 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 0, 2024-05-09T09:02:39.524 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-05-09T09:02:39.524 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:02:39.524 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 0, 2024-05-09T09:02:39.524 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 0, 2024-05-09T09:02:39.524 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-05-09T09:02:39.524 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-05-09T09:02:39.525 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-05-09T09:02:39.525 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:02:39.525 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:02:39.525 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-05-09T09:02:39.526 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-05-09T09:02:39.526 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:02:39.526 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:02:39.526 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T09:02:39.526 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 0, 2024-05-09T09:02:39.526 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-05-09T09:02:39.527 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 0, 2024-05-09T09:02:39.527 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-05-09T09:02:39.527 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-05-09T09:02:39.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.527 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-05-09T09:02:39.527 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-05-09T09:02:39.527 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 0, 2024-05-09T09:02:39.528 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-05-09T09:02:39.528 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-05-09T09:02:39.528 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 6, 2024-05-09T09:02:39.528 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-05-09T09:02:39.528 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-05-09T09:02:39.528 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 0, 2024-05-09T09:02:39.529 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-05-09T09:02:39.529 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-05-09T09:02:39.529 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 2, 2024-05-09T09:02:39.529 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 2, 2024-05-09T09:02:39.529 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-05-09T09:02:39.529 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-05-09T09:02:39.529 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 4194304, 2024-05-09T09:02:39.530 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 4195956, 2024-05-09T09:02:39.530 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 4195815, 2024-05-09T09:02:39.530 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-05-09T09:02:39.530 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.530 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.530 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.531 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 6 2024-05-09T09:02:39.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.531 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-05-09T09:02:39.531 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 3, 2024-05-09T09:02:39.532 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 3, 2024-05-09T09:02:39.532 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 0, 2024-05-09T09:02:39.532 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 1, 2024-05-09T09:02:39.532 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 1, 2024-05-09T09:02:39.532 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 0, 2024-05-09T09:02:39.533 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 0, 2024-05-09T09:02:39.533 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 0, 2024-05-09T09:02:39.533 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 0, 2024-05-09T09:02:39.533 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 0, 2024-05-09T09:02:39.533 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 0, 2024-05-09T09:02:39.534 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 0, 2024-05-09T09:02:39.534 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 49876, 2024-05-09T09:02:39.534 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 215420 2024-05-09T09:02:39.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.534 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-05-09T09:02:39.535 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 0, 2024-05-09T09:02:39.535 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 0, 2024-05-09T09:02:39.535 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 0, 2024-05-09T09:02:39.535 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 0, 2024-05-09T09:02:39.535 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 0, 2024-05-09T09:02:39.536 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-05-09T09:02:39.536 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.536 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.536 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.537 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-05-09T09:02:39.537 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.537 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.537 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.538 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-05-09T09:02:39.538 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.538 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.538 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.539 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-05-09T09:02:39.539 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.539 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.539 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.540 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-05-09T09:02:39.540 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.540 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.540 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.541 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-05-09T09:02:39.541 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.541 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.541 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.542 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-05-09T09:02:39.542 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.542 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.542 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.543 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-05-09T09:02:39.543 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.543 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.543 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.544 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-05-09T09:02:39.544 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.544 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.544 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.545 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-05-09T09:02:39.545 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.545 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.545 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.546 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-05-09T09:02:39.546 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.546 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.546 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.547 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-05-09T09:02:39.547 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.548 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.548 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.549 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-05-09T09:02:39.549 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.549 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.549 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.550 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-05-09T09:02:39.550 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.550 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.550 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.550 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-05-09T09:02:39.551 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.551 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.551 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.551 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-05-09T09:02:39.551 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.552 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.552 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.552 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-05-09T09:02:39.552 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.552 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.552 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.553 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-05-09T09:02:39.553 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.553 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.553 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.554 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-05-09T09:02:39.554 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.554 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.554 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.554 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-05-09T09:02:39.554 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.555 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.555 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.555 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-05-09T09:02:39.555 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.555 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.556 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.556 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-05-09T09:02:39.556 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.556 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.556 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.557 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-05-09T09:02:39.557 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.557 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.557 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.557 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-05-09T09:02:39.557 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.557 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.557 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.558 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-05-09T09:02:39.558 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.558 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.558 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.558 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-05-09T09:02:39.558 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.558 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.558 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.558 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-05-09T09:02:39.558 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.558 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.558 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.559 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-05-09T09:02:39.559 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.559 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.559 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.559 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-05-09T09:02:39.559 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.559 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.559 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.559 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:02:39.559 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:02:39.559 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 0, 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 2, 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 2, 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 2, 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 0, 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 0, 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 1191, 2024-05-09T09:02:39.560 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:02:39.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:02:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:02:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:02:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-05-09T09:02:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:02:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-05-09T09:02:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:02:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:02:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:02:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:02:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:02:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:02:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:02:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-05-09T09:02:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 189393, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 51, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-05-09T09:02:39.563 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 9712, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 187, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 8464, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 13, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-05-09T09:02:39.564 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 2398, 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 21, 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 3, 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 2398, 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2398, 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 3.840951942, 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.001601731 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-05-09T09:02:39.565 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-05-09T09:02:39.566 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2398, 2024-05-09T09:02:39.566 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 2399 2024-05-09T09:02:39.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.566 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 2398, 2024-05-09T09:02:39.566 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 2398, 2024-05-09T09:02:39.566 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 0, 2024-05-09T09:02:39.566 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-05-09T09:02:39.566 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-05-09T09:02:39.566 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 1553, 2024-05-09T09:02:39.566 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 0, 2024-05-09T09:02:39.566 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 843, 2024-05-09T09:02:39.566 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 0, 2024-05-09T09:02:39.566 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 0, 2024-05-09T09:02:39.566 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 0, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 0, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 1, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 0, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-05-09T09:02:39.567 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-05-09T09:02:39.568 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 109, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 19, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 5, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 5, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 0, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 0, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 2, 2024-05-09T09:02:39.569 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 0 2024-05-09T09:02:39.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.570 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-05-09T09:02:39.570 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 0, 2024-05-09T09:02:39.570 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:02:39.570 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-05-09T09:02:39.570 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-05-09T09:02:39.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.570 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-05-09T09:02:39.570 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 0 2024-05-09T09:02:39.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.570 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-05-09T09:02:39.570 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-05-09T09:02:39.570 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T09:02:39.570 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 0, 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 0, 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 0, 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 2675, 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 2090261, 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 2675, 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-05-09T09:02:39.571 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 2675, 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 2090261, 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-05-09T09:02:39.572 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:39.573 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:39.573 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 2398, 2024-05-09T09:02:39.573 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 1657, 2024-05-09T09:02:39.573 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 2398, 2024-05-09T09:02:39.573 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1657, 2024-05-09T09:02:39.573 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:39.573 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.573 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.573 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.573 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:39.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.573 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-05-09T09:02:39.573 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:39.573 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-05-09T09:02:39.574 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:39.574 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-05-09T09:02:39.574 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-05-09T09:02:39.574 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:39.574 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:39.574 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 2398, 2024-05-09T09:02:39.574 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 2398, 2024-05-09T09:02:39.574 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 2398, 2024-05-09T09:02:39.574 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 2398, 2024-05-09T09:02:39.574 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:39.574 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.574 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.574 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.574 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:39.575 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x55dfcec046e0": { 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-05-09T09:02:39.576 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-05-09T09:02:39.577 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-05-09T09:02:39.577 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:39.577 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:39.577 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:39.577 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:39.577 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:39.577 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:39.577 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:02:40.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:40 smithi098 ceph-mon[30092]: pgmap v377: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 8 op/s 2024-05-09T09:02:40.389 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-05-09T09:02:40.389 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-05-09T09:02:40.389 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-05-09T09:02:40.389 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-05-09T09:02:40.390 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T09:02:40.390 DEBUG:tasks.check_counter:Getting stats from d 2024-05-09T09:02:40.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph daemon mds.d perf dump 2024-05-09T09:02:40.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:40 smithi142 ceph-mon[34625]: pgmap v377: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 8 op/s 2024-05-09T09:02:40.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:40 smithi105 ceph-mon[34498]: pgmap v377: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 8 op/s 2024-05-09T09:02:40.639 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:02:42.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:42 smithi098 ceph-mon[30092]: pgmap v378: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 11 op/s 2024-05-09T09:02:42.373 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:02:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:02:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 752, 2024-05-09T09:02:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 753, 2024-05-09T09:02:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 42173319, 2024-05-09T09:02:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 238649, 2024-05-09T09:02:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 8, 2024-05-09T09:02:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 4, 2024-05-09T09:02:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.120253042, 2024-05-09T09:02:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.026473044, 2024-05-09T09:02:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.054033604, 2024-05-09T09:02:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.027391773, 2024-05-09T09:02:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 752, 2024-05-09T09:02:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.146417331, 2024-05-09T09:02:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000194703 2024-05-09T09:02:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 2688, 2024-05-09T09:02:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 448 2024-05-09T09:02:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:02:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 417, 2024-05-09T09:02:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 523, 2024-05-09T09:02:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 71226432, 2024-05-09T09:02:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 351241, 2024-05-09T09:02:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 6, 2024-05-09T09:02:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 5, 2024-05-09T09:02:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.222758360, 2024-05-09T09:02:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.018043551, 2024-05-09T09:02:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.133274006, 2024-05-09T09:02:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.066773155, 2024-05-09T09:02:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 521, 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.120046543, 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000230415 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 160, 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 218896 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 177, 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 176, 2024-05-09T09:02:42.376 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 25381604, 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 94623, 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 4, 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.061334705, 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.013922836, 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.039224675, 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.007616058, 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 175, 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.042921266, 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000245264 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.377 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 208576, 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 89216 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1868, 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.060113739, 2024-05-09T09:02:42.378 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000032180 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 0, 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 0, 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-05-09T09:02:42.379 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 0, 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 0, 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 0, 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 0, 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 84, 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 9, 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 75, 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 0, 2024-05-09T09:02:42.380 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 72, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 0, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 0, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 0, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 2, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 0, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 0, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 0, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 0, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 0, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 0, 2024-05-09T09:02:42.381 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 0, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 0, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 0, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 0, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 4, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 234, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 6, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 0, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T09:02:42.382 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 0, 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 0, 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 0, 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 0, 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 0, 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 0, 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 0 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 0, 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 0, 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 0, 2024-05-09T09:02:42.383 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 0, 2024-05-09T09:02:42.384 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 0, 2024-05-09T09:02:42.384 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 0, 2024-05-09T09:02:42.384 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-05-09T09:02:42.384 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:02:42.384 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 0, 2024-05-09T09:02:42.384 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 0, 2024-05-09T09:02:42.384 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-05-09T09:02:42.384 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-05-09T09:02:42.384 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-05-09T09:02:42.384 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:02:42.384 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:02:42.384 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-05-09T09:02:42.384 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-05-09T09:02:42.384 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 0, 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 0, 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 0, 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 34271, 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-05-09T09:02:42.385 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 0, 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 2, 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 26, 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 754944865, 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 755249119, 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 755246514, 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 34271 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 43, 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 61, 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 18, 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 19, 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 20, 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 1, 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 84, 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 84, 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 0, 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 0, 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 0, 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 0, 2024-05-09T09:02:42.387 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 51456, 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 215396 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 0, 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 0, 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 0, 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 0, 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 0, 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-05-09T09:02:42.388 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.389 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.389 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.389 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-05-09T09:02:42.389 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.389 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.389 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.389 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-05-09T09:02:42.389 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-05-09T09:02:42.390 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-05-09T09:02:42.391 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.392 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.392 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.392 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-05-09T09:02:42.392 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.392 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.392 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.392 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-05-09T09:02:42.393 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.393 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.393 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.393 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-05-09T09:02:42.393 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.393 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.393 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.393 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-05-09T09:02:42.393 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.393 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.394 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.395 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.396 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.397 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.398 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.399 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:02:42.399 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:02:42.399 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-05-09T09:02:42.399 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.399 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.399 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:42.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.399 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-05-09T09:02:42.399 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 2, 2024-05-09T09:02:42.399 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 2, 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 0, 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 0, 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 0, 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 0, 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 1828, 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:02:42.400 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:02:42.401 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:02:42.402 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 231532, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 117, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 9712, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 187, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-05-09T09:02:42.403 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 223266, 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 1914, 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 1182, 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 26, 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 4, 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 1182, 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-05-09T09:02:42.404 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1182, 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 12.697845296, 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.010742677 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1182, 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 1183 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 1182, 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 1182, 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 0, 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 747, 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 0, 2024-05-09T09:02:42.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 432, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 0, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 0, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 0, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 0, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 2, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 0, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-05-09T09:02:42.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-05-09T09:02:42.407 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 109, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 12, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 10, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 10, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 0, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-05-09T09:02:42.408 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 0, 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 2, 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 0 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 0, 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 0 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-05-09T09:02:42.409 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 0, 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 0, 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 0, 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 1343, 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 138649611, 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 1343, 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-05-09T09:02:42.410 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 1343, 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 138649611, 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-05-09T09:02:42.411 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 1182, 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 138179736, 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 1182, 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 138179736, 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-05-09T09:02:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 1182, 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 1182, 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 1182, 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1182, 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.413 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.414 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x55d201be86e0": { 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-05-09T09:02:42.415 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:42.416 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:42.416 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:42.416 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:42.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:42.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:42.416 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:02:42.416 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:42 smithi142 ceph-mon[34625]: pgmap v378: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 11 op/s 2024-05-09T09:02:42.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:42 smithi105 ceph-mon[34498]: pgmap v378: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 11 op/s 2024-05-09T09:02:43.253 INFO:tasks.check_counter:Daemon mds.d mds.exported=0 2024-05-09T09:02:43.254 INFO:tasks.check_counter:Daemon mds.d mds.imported=0 2024-05-09T09:02:43.254 INFO:tasks.check_counter:Daemon mds.d mds.root_rsnaps=6 2024-05-09T09:02:43.254 INFO:tasks.check_counter:Daemon mds.d mds_server.req_mksnap_latency.avgcount=0 2024-05-09T09:02:43.254 INFO:tasks.check_counter:Daemon mds.d mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T09:02:43.254 DEBUG:tasks.check_counter:Getting stats from g 2024-05-09T09:02:43.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph daemon mds.g perf dump 2024-05-09T09:02:43.499 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:02:44.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:44 smithi098 ceph-mon[30092]: pgmap v379: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T09:02:44.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:44 smithi142 ceph-mon[34625]: pgmap v379: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T09:02:44.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:44 smithi105 ceph-mon[34498]: pgmap v379: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T09:02:45.311 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:02:45.311 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:02:45.311 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 907, 2024-05-09T09:02:45.311 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 893, 2024-05-09T09:02:45.312 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1876188, 2024-05-09T09:02:45.312 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 349046, 2024-05-09T09:02:45.312 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-05-09T09:02:45.312 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-05-09T09:02:45.312 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.140853903, 2024-05-09T09:02:45.312 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.048615698, 2024-05-09T09:02:45.312 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.059479483, 2024-05-09T09:02:45.312 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.036479796, 2024-05-09T09:02:45.312 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:45.312 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 892, 2024-05-09T09:02:45.312 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.083834669, 2024-05-09T09:02:45.312 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000093985 2024-05-09T09:02:45.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.312 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1664336, 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 148880 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1150, 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 1326, 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 297231, 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 841529, 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 7, 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 5, 2024-05-09T09:02:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.106094318, 2024-05-09T09:02:45.314 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.034058141, 2024-05-09T09:02:45.314 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.023710334, 2024-05-09T09:02:45.314 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.040707996, 2024-05-09T09:02:45.314 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:45.314 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1324, 2024-05-09T09:02:45.314 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.157545118, 2024-05-09T09:02:45.314 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000118991 2024-05-09T09:02:45.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.314 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:45.314 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.314 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.314 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 477680 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 4, 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 6, 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1629, 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 1284, 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 4, 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001248846, 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000063750, 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001027901, 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000068165, 2024-05-09T09:02:45.315 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 4, 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.001968979, 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000492244 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-05-09T09:02:45.316 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-05-09T09:02:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-05-09T09:02:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-05-09T09:02:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-05-09T09:02:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 3021, 2024-05-09T09:02:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.076560812, 2024-05-09T09:02:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000025342 2024-05-09T09:02:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-05-09T09:02:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-05-09T09:02:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-05-09T09:02:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.318 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.318 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:45.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.318 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-05-09T09:02:45.318 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 0, 2024-05-09T09:02:45.318 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 0, 2024-05-09T09:02:45.318 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-05-09T09:02:45.318 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.318 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.318 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.318 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-05-09T09:02:45.318 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 0, 2024-05-09T09:02:45.318 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 0, 2024-05-09T09:02:45.319 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-05-09T09:02:45.319 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 0, 2024-05-09T09:02:45.319 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 0, 2024-05-09T09:02:45.319 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-05-09T09:02:45.319 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 0, 2024-05-09T09:02:45.319 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 0, 2024-05-09T09:02:45.319 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 0, 2024-05-09T09:02:45.319 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 0, 2024-05-09T09:02:45.319 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 0, 2024-05-09T09:02:45.319 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 0, 2024-05-09T09:02:45.319 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 0, 2024-05-09T09:02:45.319 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 0, 2024-05-09T09:02:45.319 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 1, 2024-05-09T09:02:45.319 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 0, 2024-05-09T09:02:45.320 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 0, 2024-05-09T09:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 0, 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 0, 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 0, 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 0 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 0, 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 0, 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 0, 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 0, 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 0, 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 0, 2024-05-09T09:02:45.322 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 0, 2024-05-09T09:02:45.323 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-05-09T09:02:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 0, 2024-05-09T09:02:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-05-09T09:02:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-05-09T09:02:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-05-09T09:02:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-05-09T09:02:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 0, 2024-05-09T09:02:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-05-09T09:02:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-05-09T09:02:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 8, 2024-05-09T09:02:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-05-09T09:02:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-05-09T09:02:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 0, 2024-05-09T09:02:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-05-09T09:02:45.325 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-05-09T09:02:45.325 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 2, 2024-05-09T09:02:45.325 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 2, 2024-05-09T09:02:45.325 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-05-09T09:02:45.325 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-05-09T09:02:45.325 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 4194304, 2024-05-09T09:02:45.325 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 4196655, 2024-05-09T09:02:45.325 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 4196514, 2024-05-09T09:02:45.325 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-05-09T09:02:45.325 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.325 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.325 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.325 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 8 2024-05-09T09:02:45.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.326 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-05-09T09:02:45.326 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 3, 2024-05-09T09:02:45.326 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 3, 2024-05-09T09:02:45.326 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 0, 2024-05-09T09:02:45.326 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 1, 2024-05-09T09:02:45.326 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 1, 2024-05-09T09:02:45.326 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 0, 2024-05-09T09:02:45.326 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 0, 2024-05-09T09:02:45.326 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 0, 2024-05-09T09:02:45.326 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 0, 2024-05-09T09:02:45.326 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 0, 2024-05-09T09:02:45.326 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 0, 2024-05-09T09:02:45.326 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 0, 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 49512, 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 215420 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 0, 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 0, 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 0, 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 0, 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 0, 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.327 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.328 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.329 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-05-09T09:02:45.330 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.330 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.330 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.330 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-05-09T09:02:45.330 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.330 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.330 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.330 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-05-09T09:02:45.330 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.330 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.330 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.331 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-05-09T09:02:45.331 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.331 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.331 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.331 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-05-09T09:02:45.331 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.332 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.332 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.332 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-05-09T09:02:45.332 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.332 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.332 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.332 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-05-09T09:02:45.332 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.332 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.332 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.333 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-05-09T09:02:45.333 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.333 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.333 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.333 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-05-09T09:02:45.333 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.333 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.333 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.333 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-05-09T09:02:45.333 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.333 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.333 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-05-09T09:02:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-05-09T09:02:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-05-09T09:02:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.335 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-05-09T09:02:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-05-09T09:02:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-05-09T09:02:45.336 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.337 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.337 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.337 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-05-09T09:02:45.337 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.337 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.337 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.337 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-05-09T09:02:45.337 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.337 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.337 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.337 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-05-09T09:02:45.338 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.338 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.338 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.338 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:02:45.338 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:02:45.338 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-05-09T09:02:45.338 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.338 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.338 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.338 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:45.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.338 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-05-09T09:02:45.338 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 2, 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 4, 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 2, 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 4, 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 0, 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 0, 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 2935, 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:02:45.339 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:02:45.340 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:02:45.341 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:02:45.342 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:02:45.342 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 197490, 2024-05-09T09:02:45.342 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 56, 2024-05-09T09:02:45.342 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:02:45.342 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-05-09T09:02:45.342 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-05-09T09:02:45.342 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-05-09T09:02:45.342 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:02:45.342 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-05-09T09:02:45.342 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:02:45.342 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-05-09T09:02:45.342 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 9712, 2024-05-09T09:02:45.342 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 187, 2024-05-09T09:02:45.342 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 8464, 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 13, 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 1776, 2024-05-09T09:02:45.343 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 21, 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 3, 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 1776, 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1776, 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 218.454242090, 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.123003514 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1776, 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 1777 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 1776, 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 1776, 2024-05-09T09:02:45.344 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 0, 2024-05-09T09:02:45.345 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-05-09T09:02:45.345 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-05-09T09:02:45.345 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 1137, 2024-05-09T09:02:45.345 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 0, 2024-05-09T09:02:45.345 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 637, 2024-05-09T09:02:45.345 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 0, 2024-05-09T09:02:45.345 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 0, 2024-05-09T09:02:45.345 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-05-09T09:02:45.345 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-05-09T09:02:45.345 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-05-09T09:02:45.345 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 0, 2024-05-09T09:02:45.345 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 0, 2024-05-09T09:02:45.345 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-05-09T09:02:45.345 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 1, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 0, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-05-09T09:02:45.346 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-05-09T09:02:45.347 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-05-09T09:02:45.347 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-05-09T09:02:45.347 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-05-09T09:02:45.347 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-05-09T09:02:45.347 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-05-09T09:02:45.347 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-05-09T09:02:45.347 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-05-09T09:02:45.347 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-05-09T09:02:45.347 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-05-09T09:02:45.347 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-05-09T09:02:45.347 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-05-09T09:02:45.347 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-05-09T09:02:45.347 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 109, 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 19, 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 6, 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 6, 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 0, 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 0, 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 2, 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 0 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 0, 2024-05-09T09:02:45.348 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 0 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 0, 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 0, 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 0, 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-05-09T09:02:45.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.350 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:02:45.350 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:45.350 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-05-09T09:02:45.350 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:45.350 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 2057, 2024-05-09T09:02:45.350 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1972804, 2024-05-09T09:02:45.350 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:45.350 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 2057, 2024-05-09T09:02:45.350 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-05-09T09:02:45.350 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-05-09T09:02:45.350 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 2057, 2024-05-09T09:02:45.350 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1972804, 2024-05-09T09:02:45.350 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 1776, 2024-05-09T09:02:45.351 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 2356, 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 1776, 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 2356, 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:45.352 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:45.353 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 1776, 2024-05-09T09:02:45.353 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 1776, 2024-05-09T09:02:45.353 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 1776, 2024-05-09T09:02:45.353 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1776, 2024-05-09T09:02:45.353 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:45.353 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.353 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.353 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:45.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.353 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:02:45.353 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:45.353 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-05-09T09:02:45.353 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:45.354 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x55778f2946e0": { 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:45.355 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:45.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:45.356 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:02:46.126 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-05-09T09:02:46.126 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-05-09T09:02:46.126 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-05-09T09:02:46.126 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-05-09T09:02:46.126 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T09:02:46.126 DEBUG:tasks.check_counter:Getting stats from j 2024-05-09T09:02:46.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph daemon mds.j perf dump 2024-05-09T09:02:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:46 smithi098 ceph-mon[30092]: pgmap v380: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-09T09:02:46.370 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:02:46.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:46 smithi142 ceph-mon[34625]: pgmap v380: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-09T09:02:46.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:46 smithi105 ceph-mon[34498]: pgmap v380: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-09T09:02:48.179 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:48 smithi098 ceph-mon[30092]: pgmap v381: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-09T09:02:48.204 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-09T09:02:48.204 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:02:48.204 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 4, 2024-05-09T09:02:48.204 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 4, 2024-05-09T09:02:48.204 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 3024, 2024-05-09T09:02:48.204 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 448, 2024-05-09T09:02:48.204 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-05-09T09:02:48.204 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-05-09T09:02:48.204 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001284748, 2024-05-09T09:02:48.204 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000034441, 2024-05-09T09:02:48.204 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000493496, 2024-05-09T09:02:48.204 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:02:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 3, 2024-05-09T09:02:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000655198, 2024-05-09T09:02:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000218399 2024-05-09T09:02:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-05-09T09:02:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 448 2024-05-09T09:02:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:02:48.206 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 253, 2024-05-09T09:02:48.206 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 242, 2024-05-09T09:02:48.206 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1590048, 2024-05-09T09:02:48.206 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 148880, 2024-05-09T09:02:48.206 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-05-09T09:02:48.206 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-05-09T09:02:48.206 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.055700987, 2024-05-09T09:02:48.206 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.016396168, 2024-05-09T09:02:48.206 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.040564092, 2024-05-09T09:02:48.206 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.005894770, 2024-05-09T09:02:48.206 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:48.206 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 241, 2024-05-09T09:02:48.206 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.033836104, 2024-05-09T09:02:48.206 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000140398 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1590048, 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 148880 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 2, 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 184, 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 320, 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 187888, 2024-05-09T09:02:48.207 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.016788604, 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.011705235, 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001937754, 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 181, 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.017720096, 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000097901 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 187888 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:02:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:02:48.210 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:02:48.210 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:02:48.210 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:02:48.210 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:02:48.210 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:02:48.210 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:02:48.210 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:02:48.210 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:02:48.210 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-05-09T09:02:48.210 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:02:48.210 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-05-09T09:02:48.210 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:02:48.210 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:02:48.210 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:02:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:02:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:02:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:02:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:02:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-05-09T09:02:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:02:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:02:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:02:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-05-09T09:02:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-05-09T09:02:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-05-09T09:02:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:02:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:02:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148977, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 30, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:02:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 255, 2024-05-09T09:02:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1554387, 2024-05-09T09:02:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 255, 2024-05-09T09:02:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-05-09T09:02:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-05-09T09:02:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 255, 2024-05-09T09:02:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1554387, 2024-05-09T09:02:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-05-09T09:02:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-05-09T09:02:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-05-09T09:02:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-05-09T09:02:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-09T09:02:48.214 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-09T09:02:48.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:48 smithi142 ceph-mon[34625]: pgmap v381: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-09T09:02:48.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:48 smithi105 ceph-mon[34498]: pgmap v381: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-09T09:02:48.970 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-05-09T09:02:48.970 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-05-09T09:02:48.970 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-05-09T09:02:48.970 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-05-09T09:02:48.970 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-05-09T09:02:48.971 DEBUG:tasks.check_counter:Getting stats from b 2024-05-09T09:02:48.971 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph daemon mds.b perf dump 2024-05-09T09:02:49.260 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:02:50.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:50 smithi098 ceph-mon[30092]: pgmap v382: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-05-09T09:02:50.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:50 smithi142 ceph-mon[34625]: pgmap v382: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-05-09T09:02:50.436 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:50 smithi105 ceph-mon[34498]: pgmap v382: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-05-09T09:02:51.257 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-05-09T09:02:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:02:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 693, 2024-05-09T09:02:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 726, 2024-05-09T09:02:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 201483, 2024-05-09T09:02:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 421010, 2024-05-09T09:02:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 9, 2024-05-09T09:02:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 4, 2024-05-09T09:02:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.090407170, 2024-05-09T09:02:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.037159763, 2024-05-09T09:02:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.049354064, 2024-05-09T09:02:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.004435859, 2024-05-09T09:02:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 724, 2024-05-09T09:02:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.052804372, 2024-05-09T09:02:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000072934 2024-05-09T09:02:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 1321, 2024-05-09T09:02:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000410267, 2024-05-09T09:02:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000310 2024-05-09T09:02:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T09:02:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T09:02:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:02:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 341, 2024-05-09T09:02:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 502, 2024-05-09T09:02:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 1690153, 2024-05-09T09:02:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 576120, 2024-05-09T09:02:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 11, 2024-05-09T09:02:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 6, 2024-05-09T09:02:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.059728242, 2024-05-09T09:02:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.018613770, 2024-05-09T09:02:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.039154649, 2024-05-09T09:02:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.008402298, 2024-05-09T09:02:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 502, 2024-05-09T09:02:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.037044584, 2024-05-09T09:02:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000073793 2024-05-09T09:02:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 66, 2024-05-09T09:02:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000015728, 2024-05-09T09:02:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000238 2024-05-09T09:02:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 1687072, 2024-05-09T09:02:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 518800 2024-05-09T09:02:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:02:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 75, 2024-05-09T09:02:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 114, 2024-05-09T09:02:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 15217, 2024-05-09T09:02:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 613360, 2024-05-09T09:02:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 49, 2024-05-09T09:02:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 7, 2024-05-09T09:02:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.021881352, 2024-05-09T09:02:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.003211574, 2024-05-09T09:02:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.016727195, 2024-05-09T09:02:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000330437, 2024-05-09T09:02:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 111, 2024-05-09T09:02:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.013523658, 2024-05-09T09:02:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000121834 2024-05-09T09:02:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 78, 2024-05-09T09:02:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000010186, 2024-05-09T09:02:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000130 2024-05-09T09:02:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 4480, 2024-05-09T09:02:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 135744 2024-05-09T09:02:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-05-09T09:02:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 1, 2024-05-09T09:02:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-05-09T09:02:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "finisher-MDSRank": { 2024-05-09T09:02:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "queue_len": 0, 2024-05-09T09:02:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "complete_latency": { 2024-05-09T09:02:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 50, 2024-05-09T09:02:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.001873174, 2024-05-09T09:02:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000037463 2024-05-09T09:02:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "finisher-PurgeQueue": { 2024-05-09T09:02:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "queue_len": 0, 2024-05-09T09:02:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "complete_latency": { 2024-05-09T09:02:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 2, 2024-05-09T09:02:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000005805, 2024-05-09T09:02:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000002902 2024-05-09T09:02:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "mds": { 2024-05-09T09:02:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "request": 15, 2024-05-09T09:02:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "reply": 9, 2024-05-09T09:02:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "reply_latency": { 2024-05-09T09:02:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 9, 2024-05-09T09:02:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.003429858, 2024-05-09T09:02:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000381095 2024-05-09T09:02:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "slow_reply": 0, 2024-05-09T09:02:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "forward": 6, 2024-05-09T09:02:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_fetch_complete": 0, 2024-05-09T09:02:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_fetch_keys": 0, 2024-05-09T09:02:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_commit": 11, 2024-05-09T09:02:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_split": 0, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_merge": 0, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes": 12, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_top": 0, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_bottom": 12, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_pin_tail": 0, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_pinned": 11, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_expired": 1, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_with_caps": 1, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "caps": 1, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "subtrees": 2, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse": 23, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_hit": 16, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_forward": 1, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_discover": 3, 2024-05-09T09:02:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:02:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_remote_ino": 0, 2024-05-09T09:02:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_lock": 0, 2024-05-09T09:02:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "load_cent": 0, 2024-05-09T09:02:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "q": 0, 2024-05-09T09:02:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "exported": 0, 2024-05-09T09:02:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "exported_inodes": 0, 2024-05-09T09:02:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "imported": 0, 2024-05-09T09:02:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "imported_inodes": 0, 2024-05-09T09:02:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_dir_fetch": 0, 2024-05-09T09:02:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_backtrace_fetch": 0, 2024-05-09T09:02:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_peer_discover": 1, 2024-05-09T09:02:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rfiles": 22, 2024-05-09T09:02:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rbytes": 10738142493, 2024-05-09T09:02:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rsnaps": 6, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_set_tag": 0, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dir_inodes": 0, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_file_inodes": 0, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_inode_file_caps": 0, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_grant": 0, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:02:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_caps": 0, 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_caps_dirty": 0, 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_cap_release": 0, 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "process_request_cap_release": 0 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cache": { 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_update": 0, 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_update_receipt": 0, 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_try_discover": 0, 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_send_discover": 4, 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_handle_discover": 0, 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays": 0, 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays_delayed": 0, 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_created": 0, 2024-05-09T09:02:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_enqueued": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_reintegrated": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_migrated": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_processing": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_started": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_completed": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_exportdir": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_flush": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_fragmentdir": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_fragstats": 0, 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_inodestats": 0 2024-05-09T09:02:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_log": { 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "evlrg": 0, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "evadd": 8, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "evex": 0, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "evtrm": 0, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "ev": 8, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "evexg": 0, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "evexd": 0, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "segadd": 2, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "segex": 0, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "segtrm": 0, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "seg": 2, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "segmjr": 2, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "segexg": 0, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "segexd": 0, 2024-05-09T09:02:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "expos": 4194304, 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "wrpos": 4196655, 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "rdpos": 0, 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "jlat": { 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 7, 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.014949898, 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.002135699 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "replayed": 0 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_mem": { 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "ino": 15, 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "ino+": 15, 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "ino-": 0, 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "dir": 13, 2024-05-09T09:02:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "dir+": 14, 2024-05-09T09:02:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "dir-": 1, 2024-05-09T09:02:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "dn": 12, 2024-05-09T09:02:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "dn+": 13, 2024-05-09T09:02:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "dn-": 1, 2024-05-09T09:02:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "cap": 1, 2024-05-09T09:02:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "cap+": 6, 2024-05-09T09:02:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "cap-": 5, 2024-05-09T09:02:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "rss": 55300, 2024-05-09T09:02:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "heap": 223612 2024-05-09T09:02:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_server": { 2024-05-09T09:02:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "dispatch_client_request": 20, 2024-05-09T09:02:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "dispatch_server_request": 0, 2024-05-09T09:02:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_request": 15, 2024-05-09T09:02:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_session": 28, 2024-05-09T09:02:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_peer_request": 1, 2024-05-09T09:02:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: "req_create_latency": { 2024-05-09T09:02:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getattr_latency": { 2024-05-09T09:02:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 5, 2024-05-09T09:02:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.001927095, 2024-05-09T09:02:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000385419 2024-05-09T09:02:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getfilelock_latency": { 2024-05-09T09:02:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.276 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.276 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.276 INFO:teuthology.orchestra.run.smithi105.stdout: "req_link_latency": { 2024-05-09T09:02:51.276 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.276 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.276 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.276 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookup_latency": { 2024-05-09T09:02:51.276 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 4, 2024-05-09T09:02:51.276 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.001502763, 2024-05-09T09:02:51.276 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000375690 2024-05-09T09:02:51.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.276 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookuphash_latency": { 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupino_latency": { 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupname_latency": { 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.277 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupparent_latency": { 2024-05-09T09:02:51.278 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.278 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.278 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.278 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupsnap_latency": { 2024-05-09T09:02:51.278 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.278 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.278 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.278 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lssnap_latency": { 2024-05-09T09:02:51.278 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.278 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.278 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.279 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mkdir_latency": { 2024-05-09T09:02:51.279 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.279 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.279 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.279 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mknod_latency": { 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mksnap_latency": { 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: "req_open_latency": { 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.280 INFO:teuthology.orchestra.run.smithi105.stdout: "req_readdir_latency": { 2024-05-09T09:02:51.281 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.281 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.281 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.281 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rename_latency": { 2024-05-09T09:02:51.281 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.281 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.281 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.281 INFO:teuthology.orchestra.run.smithi105.stdout: "req_renamesnap_latency": { 2024-05-09T09:02:51.281 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.281 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.281 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.282 INFO:teuthology.orchestra.run.smithi105.stdout: "req_snapdiff_latency": { 2024-05-09T09:02:51.282 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.282 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.282 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.282 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmdir_latency": { 2024-05-09T09:02:51.282 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.282 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.282 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.282 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmsnap_latency": { 2024-05-09T09:02:51.282 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.282 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.283 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.283 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmxattr_latency": { 2024-05-09T09:02:51.283 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.283 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.283 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.283 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setattr_latency": { 2024-05-09T09:02:51.283 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.283 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.283 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.283 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setdirlayout_latency": { 2024-05-09T09:02:51.283 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setfilelock_latency": { 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setlayout_latency": { 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setxattr_latency": { 2024-05-09T09:02:51.284 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.285 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.285 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.285 INFO:teuthology.orchestra.run.smithi105.stdout: "req_symlink_latency": { 2024-05-09T09:02:51.285 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.285 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.285 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.285 INFO:teuthology.orchestra.run.smithi105.stdout: "req_unlink_latency": { 2024-05-09T09:02:51.285 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.285 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.285 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.285 INFO:teuthology.orchestra.run.smithi105.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getvxattr_latency": { 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_sessions": { 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: "session_count": 2, 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: "session_add": 4, 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: "session_remove": 2, 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: "sessions_open": 2, 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: "sessions_stale": 0, 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: "total_load": 7, 2024-05-09T09:02:51.286 INFO:teuthology.orchestra.run.smithi105.stdout: "average_load": 1, 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: "avg_session_uptime": 712, 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: "mdthresh_evicted": 0 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:02:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:02:51.288 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 258049, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 111, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 264, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 3, 2024-05-09T09:02:51.289 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-05-09T09:02:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-05-09T09:02:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:02:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-05-09T09:02:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:02:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-05-09T09:02:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 9712, 2024-05-09T09:02:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 187, 2024-05-09T09:02:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:02:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:02:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-05-09T09:02:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-05-09T09:02:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 59488, 2024-05-09T09:02:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 239, 2024-05-09T09:02:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-05-09T09:02:51.291 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-05-09T09:02:51.291 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-05-09T09:02:51.291 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-05-09T09:02:51.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.291 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter": { 2024-05-09T09:02:51.291 INFO:teuthology.orchestra.run.smithi105.stdout: "op_active": 0, 2024-05-09T09:02:51.291 INFO:teuthology.orchestra.run.smithi105.stdout: "op_laggy": 0, 2024-05-09T09:02:51.291 INFO:teuthology.orchestra.run.smithi105.stdout: "op_send": 47, 2024-05-09T09:02:51.291 INFO:teuthology.orchestra.run.smithi105.stdout: "op_send_bytes": 13014, 2024-05-09T09:02:51.291 INFO:teuthology.orchestra.run.smithi105.stdout: "op_resend": 30, 2024-05-09T09:02:51.291 INFO:teuthology.orchestra.run.smithi105.stdout: "op_reply": 47, 2024-05-09T09:02:51.291 INFO:teuthology.orchestra.run.smithi105.stdout: "op_latency": { 2024-05-09T09:02:51.291 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 47, 2024-05-09T09:02:51.291 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.335993368, 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.007148795 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: "op_inflight": 0, 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: "oplen_avg": { 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 47, 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 72 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: "op": 47, 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: "op_r": 0, 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: "op_w": 47, 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: "op_rmw": 0, 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: "op_pg": 0, 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_stat": 0, 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_create": 10, 2024-05-09T09:02:51.292 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_read": 0, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_write": 7, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_writefull": 8, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_writesame": 0, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_append": 0, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_zero": 0, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_truncate": 0, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_delete": 6, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_mapext": 0, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_sparse_read": 0, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_clonerange": 0, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_getxattr": 0, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_setxattr": 20, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_rmxattr": 0, 2024-05-09T09:02:51.293 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:02:51.294 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_call": 0, 2024-05-09T09:02:51.294 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_watch": 0, 2024-05-09T09:02:51.294 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_notify": 0, 2024-05-09T09:02:51.294 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:02:51.294 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_pgls": 0, 2024-05-09T09:02:51.294 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:02:51.294 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_other": 0, 2024-05-09T09:02:51.294 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_active": 0, 2024-05-09T09:02:51.294 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_send": 0, 2024-05-09T09:02:51.294 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_resend": 0, 2024-05-09T09:02:51.294 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_ping": 0, 2024-05-09T09:02:51.294 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_active": 0, 2024-05-09T09:02:51.294 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_send": 0, 2024-05-09T09:02:51.294 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_resend": 0, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_active": 0, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_send": 0, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_resend": 0, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_active": 0, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_send": 0, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_resend": 0, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "command_active": 0, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "command_send": 0, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "command_resend": 0, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "map_epoch": 109, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "map_full": 0, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "map_inc": 19, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_sessions": 9, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_session_open": 9, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_session_close": 0, 2024-05-09T09:02:51.295 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_laggy": 0, 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_wr": 19, 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_rd": 0, 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_del": 2 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: "oft": { 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_objs": 1, 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_updates": 0, 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_removes": 0 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: "osd-slow-ops": { 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: "slow_ops_count": 0 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: "purge_queue": { 2024-05-09T09:02:51.296 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_ops": 0, 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing": 0, 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_high_water": 0, 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executed_ops": 0, 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executed": 0, 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_item_in_journal": 0 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 1105, 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 1787671, 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 1105, 2024-05-09T09:02:51.297 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-09T09:02:51.298 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-09T09:02:51.298 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 1105, 2024-05-09T09:02:51.298 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 1787671, 2024-05-09T09:02:51.298 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:51.298 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.298 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.298 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.298 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:51.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.298 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-objecter_bytes": { 2024-05-09T09:02:51.298 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:51.298 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-05-09T09:02:51.298 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:51.298 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-09T09:02:51.299 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-09T09:02:51.299 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:51.299 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:51.299 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 47, 2024-05-09T09:02:51.299 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 13014, 2024-05-09T09:02:51.299 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 47, 2024-05-09T09:02:51.299 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 13014, 2024-05-09T09:02:51.299 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:51.299 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.299 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.299 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.299 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:51.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.299 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-objecter_ops": { 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 1024, 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 47, 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 47, 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 47, 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 47, 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.300 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 3758096384, 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 0, 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 0, 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:51.301 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.302 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.302 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.302 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:51.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:51.302 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-write_buf_throttle-0x559c60342500": { 2024-05-09T09:02:51.302 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:51.302 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 3758096384, 2024-05-09T09:02:51.302 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:51.302 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 8, 2024-05-09T09:02:51.302 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 2351, 2024-05-09T09:02:51.302 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:51.303 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 8, 2024-05-09T09:02:51.303 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-09T09:02:51.303 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-09T09:02:51.303 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 7, 2024-05-09T09:02:51.303 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 2351, 2024-05-09T09:02:51.303 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:51.303 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:51.303 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:51.303 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:51.303 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:51.303 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:51.303 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-05-09T09:02:52.124 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-05-09T09:02:52.124 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-05-09T09:02:52.124 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-05-09T09:02:52.124 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=0 2024-05-09T09:02:52.124 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T09:02:52.125 DEBUG:tasks.check_counter:Getting stats from e 2024-05-09T09:02:52.125 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph daemon mds.e perf dump 2024-05-09T09:02:52.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:52 smithi098 ceph-mon[30092]: pgmap v383: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-05-09T09:02:52.373 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:02:52.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:52 smithi142 ceph-mon[34625]: pgmap v383: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-05-09T09:02:52.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:52 smithi105 ceph-mon[34498]: pgmap v383: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-05-09T09:02:54.106 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:54 smithi105 ceph-mon[34498]: pgmap v384: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-05-09T09:02:54.231 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-05-09T09:02:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:02:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 99, 2024-05-09T09:02:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 98, 2024-05-09T09:02:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 168160, 2024-05-09T09:02:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 59712, 2024-05-09T09:02:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 4, 2024-05-09T09:02:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 0, 2024-05-09T09:02:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.017018248, 2024-05-09T09:02:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.004944121, 2024-05-09T09:02:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.006535400, 2024-05-09T09:02:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.003470216, 2024-05-09T09:02:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 97, 2024-05-09T09:02:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.006209041, 2024-05-09T09:02:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000064010 2024-05-09T09:02:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 168160, 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 59712 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 5, 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 77, 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 3184, 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 91312, 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 3, 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 2, 2024-05-09T09:02:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.004956927, 2024-05-09T09:02:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.003203464, 2024-05-09T09:02:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.000919848, 2024-05-09T09:02:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:02:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 76, 2024-05-09T09:02:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.006286802, 2024-05-09T09:02:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000082721 2024-05-09T09:02:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-05-09T09:02:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 91312 2024-05-09T09:02:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:02:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 0, 2024-05-09T09:02:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 2, 2024-05-09T09:02:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 0, 2024-05-09T09:02:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 0, 2024-05-09T09:02:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 2, 2024-05-09T09:02:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 1, 2024-05-09T09:02:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.000497072, 2024-05-09T09:02:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.000000000, 2024-05-09T09:02:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.000423797, 2024-05-09T09:02:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:02:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T09:02:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T09:02:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-05-09T09:02:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 0, 2024-05-09T09:02:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-05-09T09:02:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-05-09T09:02:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:02:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-05-09T09:02:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:02:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:02:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:02:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:02:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:02:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:02:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:02:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:02:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:02:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:02:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:02:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:02:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:02:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-05-09T09:02:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:02:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-05-09T09:02:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:02:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:02:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:02:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:02:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:02:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:02:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:02:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-05-09T09:02:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:02:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:02:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:02:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-05-09T09:02:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-05-09T09:02:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-05-09T09:02:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:02:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:02:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:02:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:02:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 149854, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 28, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 0, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 0, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 0, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-05-09T09:02:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-05-09T09:02:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 928, 2024-05-09T09:02:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 2, 2024-05-09T09:02:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-05-09T09:02:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-05-09T09:02:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-05-09T09:02:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-05-09T09:02:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:02:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-05-09T09:02:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 100, 2024-05-09T09:02:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 154203, 2024-05-09T09:02:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 100, 2024-05-09T09:02:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-09T09:02:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-09T09:02:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 100, 2024-05-09T09:02:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 154203, 2024-05-09T09:02:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:54.243 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-05-09T09:02:54.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:54 smithi098 ceph-mon[30092]: pgmap v384: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-05-09T09:02:54.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:54 smithi142 ceph-mon[34625]: pgmap v384: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-05-09T09:02:54.913 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.e 2024-05-09T09:02:54.914 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.e 2024-05-09T09:02:54.914 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-05-09T09:02:54.914 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-05-09T09:02:54.914 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-05-09T09:02:54.914 DEBUG:tasks.check_counter:Getting stats from h 2024-05-09T09:02:54.914 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph daemon mds.h perf dump 2024-05-09T09:02:55.155 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:02:56.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:56 smithi098 ceph-mon[30092]: pgmap v385: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-09T09:02:56.345 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:56 smithi105 ceph-mon[34498]: pgmap v385: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-09T09:02:56.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:56 smithi142 ceph-mon[34625]: pgmap v385: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-09T09:02:56.930 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-05-09T09:02:56.930 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:02:56.930 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 5651, 2024-05-09T09:02:56.930 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 5694, 2024-05-09T09:02:56.930 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 1846576, 2024-05-09T09:02:56.930 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 202460421, 2024-05-09T09:02:56.930 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 224, 2024-05-09T09:02:56.930 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 5, 2024-05-09T09:02:56.930 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 2.674796850, 2024-05-09T09:02:56.930 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.911658022, 2024-05-09T09:02:56.931 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.219997206, 2024-05-09T09:02:56.931 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 1.204953961, 2024-05-09T09:02:56.931 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:56.931 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 5696, 2024-05-09T09:02:56.931 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 1.532941766, 2024-05-09T09:02:56.931 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000269126 2024-05-09T09:02:56.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.931 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:56.931 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 3159, 2024-05-09T09:02:56.931 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000690787, 2024-05-09T09:02:56.931 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000218 2024-05-09T09:02:56.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.931 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 45584, 2024-05-09T09:02:56.932 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 2864 2024-05-09T09:02:56.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.932 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:02:56.932 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 275595, 2024-05-09T09:02:56.932 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 277444, 2024-05-09T09:02:56.932 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 92012670, 2024-05-09T09:02:56.932 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 325241824, 2024-05-09T09:02:56.932 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 28, 2024-05-09T09:02:56.932 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 7, 2024-05-09T09:02:56.932 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 14.246822751, 2024-05-09T09:02:56.932 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 5.616990878, 2024-05-09T09:02:56.932 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 16.574006265, 2024-05-09T09:02:56.932 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 2.448135573, 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 277438, 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 65.338991005, 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000235508 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 184220, 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.091311104, 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000495 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 468672, 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 279120 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 4069, 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 3887, 2024-05-09T09:02:56.933 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 2821405, 2024-05-09T09:02:56.934 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 187465527, 2024-05-09T09:02:56.934 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 40, 2024-05-09T09:02:56.934 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 7, 2024-05-09T09:02:56.934 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 2.140264628, 2024-05-09T09:02:56.934 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.649592594, 2024-05-09T09:02:56.934 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.154163837, 2024-05-09T09:02:56.934 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 1.329918093, 2024-05-09T09:02:56.934 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:56.934 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 3886, 2024-05-09T09:02:56.934 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.772254741, 2024-05-09T09:02:56.934 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000198727 2024-05-09T09:02:56.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.934 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:56.934 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 2165, 2024-05-09T09:02:56.935 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000499566, 2024-05-09T09:02:56.935 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000230 2024-05-09T09:02:56.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.935 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 1723488, 2024-05-09T09:02:56.935 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 541568 2024-05-09T09:02:56.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.935 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-05-09T09:02:56.935 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 1, 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: "finisher-MDSRank": { 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: "queue_len": 0, 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: "complete_latency": { 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 3055, 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 203.097069169, 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.066480219 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: "finisher-PurgeQueue": { 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: "queue_len": 0, 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: "complete_latency": { 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 2475, 2024-05-09T09:02:56.936 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.091829475, 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000037102 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: "mds": { 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: "request": 259, 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: "reply": 259, 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: "reply_latency": { 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 259, 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 20.806452099, 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.080333791 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: "slow_reply": 0, 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: "forward": 0, 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_fetch_complete": 0, 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_fetch_keys": 0, 2024-05-09T09:02:56.937 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_commit": 148, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_split": 1, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_merge": 0, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes": 121, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_top": 40, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_bottom": 81, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_pin_tail": 0, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_pinned": 63, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_expired": 0, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_with_caps": 3, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "caps": 3, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "subtrees": 2, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse": 346, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_hit": 335, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_forward": 0, 2024-05-09T09:02:56.938 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_discover": 0, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_remote_ino": 0, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_lock": 0, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "load_cent": 0, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "q": 0, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "exported": 0, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "exported_inodes": 0, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "imported": 0, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "imported_inodes": 0, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_dir_fetch": 0, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_backtrace_fetch": 596, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_peer_discover": 0, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rfiles": 4, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rbytes": 234, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rsnaps": 6, 2024-05-09T09:02:56.939 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_backtrace_fetch": 596, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_set_tag": 0, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dir_inodes": 210, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dir_base_inodes": 20, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dirfrag_rstats": 190, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_file_inodes": 386, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_inode_file_caps": 1, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_revoke": 52, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_grant": 3328, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_caps": 282710, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_caps_dirty": 282628, 2024-05-09T09:02:56.940 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_cap_release": 2, 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "process_request_cap_release": 56 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cache": { 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_update": 0, 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_update_receipt": 0, 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_try_discover": 0, 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_send_discover": 0, 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_handle_discover": 9, 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays": 4, 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays_delayed": 0, 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_created": 22, 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_enqueued": 18, 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_reintegrated": 0, 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_migrated": 0, 2024-05-09T09:02:56.941 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_processing": 0, 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_started": 0, 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_completed": 0, 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_enqueue_scrub": 20, 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_exportdir": 0, 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_flush": 0, 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_fragmentdir": 1, 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_fragstats": 0, 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_inodestats": 0 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_log": { 2024-05-09T09:02:56.942 INFO:teuthology.orchestra.run.smithi105.stdout: "evlrg": 0, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "evadd": 271376, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "evex": 271317, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "evtrm": 271317, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "ev": 59, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "evexg": 0, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "evexd": 10, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "segadd": 388, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "segex": 386, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "segtrm": 386, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "seg": 2, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "segmjr": 1, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "segexg": 0, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "segexd": 1, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "expos": 754944865, 2024-05-09T09:02:56.943 INFO:teuthology.orchestra.run.smithi105.stdout: "wrpos": 755249119, 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: "rdpos": 0, 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: "jlat": { 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 4730, 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 1280.646448707, 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.270749777 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: "replayed": 0 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_mem": { 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: "ino": 93, 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: "ino+": 131, 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: "ino-": 38, 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: "dir": 29, 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: "dir+": 30, 2024-05-09T09:02:56.944 INFO:teuthology.orchestra.run.smithi105.stdout: "dir-": 1, 2024-05-09T09:02:56.945 INFO:teuthology.orchestra.run.smithi105.stdout: "dn": 121, 2024-05-09T09:02:56.945 INFO:teuthology.orchestra.run.smithi105.stdout: "dn+": 139, 2024-05-09T09:02:56.945 INFO:teuthology.orchestra.run.smithi105.stdout: "dn-": 18, 2024-05-09T09:02:56.945 INFO:teuthology.orchestra.run.smithi105.stdout: "cap": 3, 2024-05-09T09:02:56.945 INFO:teuthology.orchestra.run.smithi105.stdout: "cap+": 40, 2024-05-09T09:02:56.945 INFO:teuthology.orchestra.run.smithi105.stdout: "cap-": 37, 2024-05-09T09:02:56.945 INFO:teuthology.orchestra.run.smithi105.stdout: "rss": 88292, 2024-05-09T09:02:56.945 INFO:teuthology.orchestra.run.smithi105.stdout: "heap": 231804 2024-05-09T09:02:56.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.946 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_server": { 2024-05-09T09:02:56.946 INFO:teuthology.orchestra.run.smithi105.stdout: "dispatch_client_request": 331, 2024-05-09T09:02:56.946 INFO:teuthology.orchestra.run.smithi105.stdout: "dispatch_server_request": 1, 2024-05-09T09:02:56.946 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_request": 259, 2024-05-09T09:02:56.946 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_session": 38, 2024-05-09T09:02:56.946 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_peer_request": 2, 2024-05-09T09:02:56.946 INFO:teuthology.orchestra.run.smithi105.stdout: "req_create_latency": { 2024-05-09T09:02:56.946 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 22, 2024-05-09T09:02:56.946 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.025215857, 2024-05-09T09:02:56.946 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.001146175 2024-05-09T09:02:56.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.946 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getattr_latency": { 2024-05-09T09:02:56.946 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 131, 2024-05-09T09:02:56.947 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.075166815, 2024-05-09T09:02:56.947 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000573792 2024-05-09T09:02:56.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.947 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getfilelock_latency": { 2024-05-09T09:02:56.947 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.947 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.947 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.947 INFO:teuthology.orchestra.run.smithi105.stdout: "req_link_latency": { 2024-05-09T09:02:56.947 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.947 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.947 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.947 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookup_latency": { 2024-05-09T09:02:56.948 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 17, 2024-05-09T09:02:56.948 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.012460274, 2024-05-09T09:02:56.948 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000732957 2024-05-09T09:02:56.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.948 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookuphash_latency": { 2024-05-09T09:02:56.948 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.948 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.948 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.948 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupino_latency": { 2024-05-09T09:02:56.948 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 1, 2024-05-09T09:02:56.949 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000138186, 2024-05-09T09:02:56.949 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000138186 2024-05-09T09:02:56.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.949 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupname_latency": { 2024-05-09T09:02:56.949 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.949 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.949 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.949 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupparent_latency": { 2024-05-09T09:02:56.949 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.949 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.949 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.949 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupsnap_latency": { 2024-05-09T09:02:56.950 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.950 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.950 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.950 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lssnap_latency": { 2024-05-09T09:02:56.950 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 12, 2024-05-09T09:02:56.951 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.005140725, 2024-05-09T09:02:56.951 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000428393 2024-05-09T09:02:56.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.951 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mkdir_latency": { 2024-05-09T09:02:56.951 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 10, 2024-05-09T09:02:56.951 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.016711775, 2024-05-09T09:02:56.951 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.001671177 2024-05-09T09:02:56.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.951 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mknod_latency": { 2024-05-09T09:02:56.951 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mksnap_latency": { 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 12, 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 17.560936710, 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 1.463411392 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: "req_open_latency": { 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 1, 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000244381, 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000244381 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: "req_readdir_latency": { 2024-05-09T09:02:56.952 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 12, 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.040486636, 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.003373886 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rename_latency": { 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 2, 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.002111575, 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.001055787 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: "req_renamesnap_latency": { 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: "req_snapdiff_latency": { 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.953 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.954 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.954 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmdir_latency": { 2024-05-09T09:02:56.954 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 4, 2024-05-09T09:02:56.954 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.994527365, 2024-05-09T09:02:56.954 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.248631841 2024-05-09T09:02:56.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.954 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmsnap_latency": { 2024-05-09T09:02:56.954 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 6, 2024-05-09T09:02:56.954 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 2.002978030, 2024-05-09T09:02:56.954 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.333829671 2024-05-09T09:02:56.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.954 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmxattr_latency": { 2024-05-09T09:02:56.954 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.955 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.955 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.955 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setattr_latency": { 2024-05-09T09:02:56.955 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 2, 2024-05-09T09:02:56.955 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.001469520, 2024-05-09T09:02:56.955 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000734760 2024-05-09T09:02:56.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.955 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setdirlayout_latency": { 2024-05-09T09:02:56.955 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.955 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.955 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.955 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setfilelock_latency": { 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setlayout_latency": { 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setxattr_latency": { 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 9, 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.020379429, 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.002264381 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.956 INFO:teuthology.orchestra.run.smithi105.stdout: "req_symlink_latency": { 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: "req_unlink_latency": { 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 18, 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.048484821, 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.002693601 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getvxattr_latency": { 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.957 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_sessions": { 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: "session_count": 2, 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: "session_add": 6, 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: "session_remove": 4, 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: "sessions_open": 2, 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: "sessions_stale": 0, 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: "total_load": 72, 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: "average_load": 36, 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: "avg_session_uptime": 727, 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: "mdthresh_evicted": 0 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-05-09T09:02:56.958 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:02:56.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-05-09T09:02:56.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:02:56.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:02:56.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:02:56.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:02:56.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:02:56.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:02:56.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:02:56.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:02:56.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:02:56.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:02:56.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:02:56.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:02:56.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:02:56.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-05-09T09:02:56.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:02:56.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-05-09T09:02:56.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:02:56.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:02:56.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:02:56.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:02:56.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:02:56.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:02:56.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 354491, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 157, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 440, 2024-05-09T09:02:56.961 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 5, 2024-05-09T09:02:56.962 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-05-09T09:02:56.962 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-05-09T09:02:56.962 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:02:56.962 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-05-09T09:02:56.962 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:02:56.962 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-05-09T09:02:56.962 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 9712, 2024-05-09T09:02:56.962 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 187, 2024-05-09T09:02:56.962 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:02:56.962 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:02:56.962 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-05-09T09:02:56.962 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-05-09T09:02:56.962 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 369598, 2024-05-09T09:02:56.962 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 2846, 2024-05-09T09:02:56.963 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-05-09T09:02:56.963 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-05-09T09:02:56.963 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-05-09T09:02:56.963 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-05-09T09:02:56.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.963 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter": { 2024-05-09T09:02:56.963 INFO:teuthology.orchestra.run.smithi105.stdout: "op_active": 0, 2024-05-09T09:02:56.963 INFO:teuthology.orchestra.run.smithi105.stdout: "op_laggy": 0, 2024-05-09T09:02:56.963 INFO:teuthology.orchestra.run.smithi105.stdout: "op_send": 9548, 2024-05-09T09:02:56.963 INFO:teuthology.orchestra.run.smithi105.stdout: "op_send_bytes": 751773691, 2024-05-09T09:02:56.963 INFO:teuthology.orchestra.run.smithi105.stdout: "op_resend": 37, 2024-05-09T09:02:56.963 INFO:teuthology.orchestra.run.smithi105.stdout: "op_reply": 9548, 2024-05-09T09:02:56.963 INFO:teuthology.orchestra.run.smithi105.stdout: "op_latency": { 2024-05-09T09:02:56.963 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 9548, 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 595.583256324, 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.062377802 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: "op_inflight": 0, 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: "oplen_avg": { 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 9548, 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 9945 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: "op": 9548, 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: "op_r": 637, 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: "op_w": 8911, 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: "op_rmw": 0, 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: "op_pg": 0, 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_stat": 118, 2024-05-09T09:02:56.964 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_create": 66, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_read": 21, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_write": 4738, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_writefull": 171, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_writesame": 0, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_append": 0, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_zero": 0, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_truncate": 1, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_delete": 3465, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_mapext": 0, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_sparse_read": 0, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_clonerange": 0, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_getxattr": 616, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_setxattr": 110, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:02:56.965 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_rmxattr": 0, 2024-05-09T09:02:56.966 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:02:56.966 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_call": 0, 2024-05-09T09:02:56.966 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_watch": 0, 2024-05-09T09:02:56.966 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_notify": 0, 2024-05-09T09:02:56.966 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:02:56.966 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_pgls": 0, 2024-05-09T09:02:56.966 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:02:56.966 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_other": 0, 2024-05-09T09:02:56.966 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_active": 0, 2024-05-09T09:02:56.966 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_send": 0, 2024-05-09T09:02:56.966 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_resend": 0, 2024-05-09T09:02:56.966 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_ping": 0, 2024-05-09T09:02:56.966 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_active": 0, 2024-05-09T09:02:56.966 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_send": 0, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_resend": 0, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_active": 0, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_send": 0, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_resend": 0, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_active": 0, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_send": 0, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_resend": 0, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "command_active": 0, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "command_send": 0, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "command_resend": 0, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "map_epoch": 109, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "map_full": 0, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "map_inc": 24, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_sessions": 12, 2024-05-09T09:02:56.967 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_session_open": 12, 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_session_close": 0, 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_laggy": 0, 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_wr": 622, 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_rd": 0, 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_del": 17 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: "oft": { 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_objs": 1, 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_updates": 0, 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_removes": 0 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: "osd-slow-ops": { 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: "slow_ops_count": 2 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: "purge_queue": { 2024-05-09T09:02:56.968 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_ops": 0, 2024-05-09T09:02:56.969 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_ops_high_water": 522, 2024-05-09T09:02:56.969 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing": 0, 2024-05-09T09:02:56.969 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_high_water": 5, 2024-05-09T09:02:56.969 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executed_ops": 2596, 2024-05-09T09:02:56.969 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executed": 18, 2024-05-09T09:02:56.969 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_item_in_journal": 0 2024-05-09T09:02:56.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.969 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:02:56.969 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:56.969 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-05-09T09:02:56.969 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:56.969 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 285311, 2024-05-09T09:02:56.969 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 75267026, 2024-05-09T09:02:56.969 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:56.970 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 285311, 2024-05-09T09:02:56.970 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-09T09:02:56.970 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-09T09:02:56.970 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 285311, 2024-05-09T09:02:56.970 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 75267026, 2024-05-09T09:02:56.970 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:56.970 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.970 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.970 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.970 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:56.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-objecter_bytes": { 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 9548, 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 751776517, 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 9548, 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 751776517, 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.971 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-objecter_ops": { 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 1024, 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 9548, 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 9548, 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 9548, 2024-05-09T09:02:56.972 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 9548, 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 3758096384, 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 18, 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 2826, 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 18, 2024-05-09T09:02:56.973 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 1, 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 2826, 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-write_buf_throttle-0x5622589b0500": { 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 3758096384, 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 271376, 2024-05-09T09:02:56.974 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 751054815, 2024-05-09T09:02:56.975 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:56.975 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 271376, 2024-05-09T09:02:56.975 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-09T09:02:56.975 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-09T09:02:56.975 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 4730, 2024-05-09T09:02:56.975 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 751054815, 2024-05-09T09:02:56.975 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:56.975 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:56.975 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:56.975 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:56.975 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:56.975 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:56.975 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-05-09T09:02:57.718 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-05-09T09:02:57.718 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-05-09T09:02:57.718 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-05-09T09:02:57.718 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=12 2024-05-09T09:02:57.718 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=6 2024-05-09T09:02:57.718 DEBUG:tasks.check_counter:Getting stats from k 2024-05-09T09:02:57.719 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph daemon mds.k perf dump 2024-05-09T09:02:57.957 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:02:58.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:02:58 smithi098 ceph-mon[30092]: pgmap v386: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T09:02:58.344 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:02:58 smithi105 ceph-mon[34498]: pgmap v386: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T09:02:58.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:02:58 smithi142 ceph-mon[34625]: pgmap v386: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T09:02:59.687 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-05-09T09:02:59.687 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:02:59.687 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 850, 2024-05-09T09:02:59.687 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 852, 2024-05-09T09:02:59.687 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 225802, 2024-05-09T09:02:59.687 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 270364, 2024-05-09T09:02:59.687 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 6, 2024-05-09T09:02:59.687 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 2, 2024-05-09T09:02:59.687 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.078002978, 2024-05-09T09:02:59.687 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.028872261, 2024-05-09T09:02:59.688 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.018987069, 2024-05-09T09:02:59.688 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.022684460, 2024-05-09T09:02:59.688 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:59.688 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 850, 2024-05-09T09:02:59.688 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.039861920, 2024-05-09T09:02:59.688 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000046896 2024-05-09T09:02:59.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.688 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:59.688 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.688 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.688 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.688 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T09:02:59.688 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T09:02:59.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.689 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:02:59.689 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 1069, 2024-05-09T09:02:59.689 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 1044, 2024-05-09T09:02:59.689 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 1886334, 2024-05-09T09:02:59.689 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 401565, 2024-05-09T09:02:59.689 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 4, 2024-05-09T09:02:59.689 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 3, 2024-05-09T09:02:59.689 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.126619017, 2024-05-09T09:02:59.689 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.034852158, 2024-05-09T09:02:59.689 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.057210072, 2024-05-09T09:02:59.689 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.038640387, 2024-05-09T09:02:59.689 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:59.689 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 1044, 2024-05-09T09:02:59.690 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.055400922, 2024-05-09T09:02:59.690 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000053066 2024-05-09T09:02:59.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.690 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:59.690 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.690 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.690 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.690 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 1696400, 2024-05-09T09:02:59.690 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 156064 2024-05-09T09:02:59.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.690 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:02:59.690 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 842, 2024-05-09T09:02:59.690 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 1028, 2024-05-09T09:02:59.691 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 282560, 2024-05-09T09:02:59.691 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 752224, 2024-05-09T09:02:59.691 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 6, 2024-05-09T09:02:59.691 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 4, 2024-05-09T09:02:59.691 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.107398912, 2024-05-09T09:02:59.691 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.036011650, 2024-05-09T09:02:59.691 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.027282626, 2024-05-09T09:02:59.691 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.033671192, 2024-05-09T09:02:59.691 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:02:59.691 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 1025, 2024-05-09T09:02:59.691 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.067091589, 2024-05-09T09:02:59.691 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000065455 2024-05-09T09:02:59.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.691 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 485104 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 1, 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: "finisher-MDSRank": { 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: "queue_len": 0, 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: "complete_latency": { 2024-05-09T09:02:59.692 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 4079, 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.124977480, 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000030639 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: "finisher-PurgeQueue": { 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: "queue_len": 0, 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: "complete_latency": { 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: "mds": { 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: "request": 0, 2024-05-09T09:02:59.693 INFO:teuthology.orchestra.run.smithi105.stdout: "reply": 0, 2024-05-09T09:02:59.694 INFO:teuthology.orchestra.run.smithi105.stdout: "reply_latency": { 2024-05-09T09:02:59.694 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.694 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.694 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.694 INFO:teuthology.orchestra.run.smithi105.stdout: "slow_reply": 0, 2024-05-09T09:02:59.694 INFO:teuthology.orchestra.run.smithi105.stdout: "forward": 0, 2024-05-09T09:02:59.694 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_fetch_complete": 0, 2024-05-09T09:02:59.694 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_fetch_keys": 0, 2024-05-09T09:02:59.694 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_commit": 0, 2024-05-09T09:02:59.694 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_split": 0, 2024-05-09T09:02:59.694 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_merge": 0, 2024-05-09T09:02:59.694 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes": 0, 2024-05-09T09:02:59.694 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_top": 0, 2024-05-09T09:02:59.695 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_bottom": 0, 2024-05-09T09:02:59.695 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_pin_tail": 0, 2024-05-09T09:02:59.695 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_pinned": 0, 2024-05-09T09:02:59.695 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_expired": 0, 2024-05-09T09:02:59.695 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_with_caps": 0, 2024-05-09T09:02:59.695 INFO:teuthology.orchestra.run.smithi105.stdout: "caps": 0, 2024-05-09T09:02:59.695 INFO:teuthology.orchestra.run.smithi105.stdout: "subtrees": 1, 2024-05-09T09:02:59.695 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse": 0, 2024-05-09T09:02:59.695 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_hit": 0, 2024-05-09T09:02:59.695 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_forward": 0, 2024-05-09T09:02:59.695 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_discover": 0, 2024-05-09T09:02:59.695 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:02:59.695 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_remote_ino": 0, 2024-05-09T09:02:59.695 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_lock": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "load_cent": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "q": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "exported": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "exported_inodes": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "imported": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "imported_inodes": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_dir_fetch": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_backtrace_fetch": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_peer_discover": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rfiles": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rbytes": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rsnaps": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_set_tag": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:02:59.696 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dir_inodes": 0, 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_file_inodes": 0, 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_inode_file_caps": 0, 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_grant": 0, 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_caps": 0, 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_caps_dirty": 0, 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_cap_release": 0, 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "process_request_cap_release": 0 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cache": { 2024-05-09T09:02:59.697 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_update": 0, 2024-05-09T09:02:59.698 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_update_receipt": 0, 2024-05-09T09:02:59.698 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_try_discover": 0, 2024-05-09T09:02:59.698 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_send_discover": 0, 2024-05-09T09:02:59.698 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_handle_discover": 0, 2024-05-09T09:02:59.698 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays": 0, 2024-05-09T09:02:59.698 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays_delayed": 0, 2024-05-09T09:02:59.698 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:02:59.698 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_created": 0, 2024-05-09T09:02:59.698 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_enqueued": 0, 2024-05-09T09:02:59.698 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_reintegrated": 0, 2024-05-09T09:02:59.698 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_migrated": 0, 2024-05-09T09:02:59.698 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_processing": 0, 2024-05-09T09:02:59.699 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:02:59.699 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:02:59.699 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_started": 0, 2024-05-09T09:02:59.699 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_completed": 0, 2024-05-09T09:02:59.699 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:02:59.699 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:02:59.699 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T09:02:59.699 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_exportdir": 0, 2024-05-09T09:02:59.699 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_flush": 0, 2024-05-09T09:02:59.699 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_fragmentdir": 0, 2024-05-09T09:02:59.699 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_fragstats": 0, 2024-05-09T09:02:59.699 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_inodestats": 0 2024-05-09T09:02:59.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.699 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_log": { 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "evlrg": 0, 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "evadd": 0, 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "evex": 0, 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "evtrm": 0, 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "ev": 9, 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "evexg": 0, 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "evexd": 0, 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "segadd": 0, 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "segex": 0, 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "segtrm": 0, 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "seg": 2, 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "segmjr": 2, 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "segexg": 0, 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "segexd": 0, 2024-05-09T09:02:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: "expos": 4194304, 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: "wrpos": 4196879, 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: "rdpos": 4196738, 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: "jlat": { 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: "replayed": 9 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_mem": { 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: "ino": 3, 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: "ino+": 3, 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: "ino-": 0, 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: "dir": 1, 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: "dir+": 1, 2024-05-09T09:02:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: "dir-": 0, 2024-05-09T09:02:59.702 INFO:teuthology.orchestra.run.smithi105.stdout: "dn": 0, 2024-05-09T09:02:59.702 INFO:teuthology.orchestra.run.smithi105.stdout: "dn+": 0, 2024-05-09T09:02:59.702 INFO:teuthology.orchestra.run.smithi105.stdout: "dn-": 0, 2024-05-09T09:02:59.702 INFO:teuthology.orchestra.run.smithi105.stdout: "cap": 0, 2024-05-09T09:02:59.702 INFO:teuthology.orchestra.run.smithi105.stdout: "cap+": 0, 2024-05-09T09:02:59.702 INFO:teuthology.orchestra.run.smithi105.stdout: "cap-": 0, 2024-05-09T09:02:59.702 INFO:teuthology.orchestra.run.smithi105.stdout: "rss": 52928, 2024-05-09T09:02:59.702 INFO:teuthology.orchestra.run.smithi105.stdout: "heap": 215420 2024-05-09T09:02:59.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.702 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_server": { 2024-05-09T09:02:59.702 INFO:teuthology.orchestra.run.smithi105.stdout: "dispatch_client_request": 0, 2024-05-09T09:02:59.702 INFO:teuthology.orchestra.run.smithi105.stdout: "dispatch_server_request": 0, 2024-05-09T09:02:59.702 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_request": 0, 2024-05-09T09:02:59.703 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_session": 0, 2024-05-09T09:02:59.703 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_peer_request": 0, 2024-05-09T09:02:59.703 INFO:teuthology.orchestra.run.smithi105.stdout: "req_create_latency": { 2024-05-09T09:02:59.703 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.703 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.703 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.703 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getattr_latency": { 2024-05-09T09:02:59.703 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.703 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.703 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.703 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getfilelock_latency": { 2024-05-09T09:02:59.703 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.704 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.704 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.704 INFO:teuthology.orchestra.run.smithi105.stdout: "req_link_latency": { 2024-05-09T09:02:59.704 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.704 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.704 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.704 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookup_latency": { 2024-05-09T09:02:59.704 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.704 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.704 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.705 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookuphash_latency": { 2024-05-09T09:02:59.705 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.705 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.705 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.705 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupino_latency": { 2024-05-09T09:02:59.705 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.705 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.705 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.705 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupname_latency": { 2024-05-09T09:02:59.705 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.705 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.705 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupparent_latency": { 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupsnap_latency": { 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lssnap_latency": { 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.706 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.707 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mkdir_latency": { 2024-05-09T09:02:59.707 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.707 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.707 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.707 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mknod_latency": { 2024-05-09T09:02:59.708 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.708 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.708 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.708 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mksnap_latency": { 2024-05-09T09:02:59.708 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.708 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.708 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.708 INFO:teuthology.orchestra.run.smithi105.stdout: "req_open_latency": { 2024-05-09T09:02:59.708 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.708 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.708 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.709 INFO:teuthology.orchestra.run.smithi105.stdout: "req_readdir_latency": { 2024-05-09T09:02:59.709 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.709 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.709 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.709 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rename_latency": { 2024-05-09T09:02:59.709 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.709 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.709 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.709 INFO:teuthology.orchestra.run.smithi105.stdout: "req_renamesnap_latency": { 2024-05-09T09:02:59.709 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.709 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.709 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.710 INFO:teuthology.orchestra.run.smithi105.stdout: "req_snapdiff_latency": { 2024-05-09T09:02:59.710 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.710 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.710 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.710 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmdir_latency": { 2024-05-09T09:02:59.710 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.710 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.710 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.714 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmsnap_latency": { 2024-05-09T09:02:59.714 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.714 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.714 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.714 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmxattr_latency": { 2024-05-09T09:02:59.714 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.714 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.714 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.714 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setattr_latency": { 2024-05-09T09:02:59.714 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.714 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.715 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.715 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setdirlayout_latency": { 2024-05-09T09:02:59.715 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.715 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.715 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.715 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setfilelock_latency": { 2024-05-09T09:02:59.715 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.715 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.715 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.715 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setlayout_latency": { 2024-05-09T09:02:59.715 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setxattr_latency": { 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: "req_symlink_latency": { 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: "req_unlink_latency": { 2024-05-09T09:02:59.716 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getvxattr_latency": { 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_sessions": { 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: "session_count": 1, 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: "session_add": 4, 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: "session_remove": 3, 2024-05-09T09:02:59.717 INFO:teuthology.orchestra.run.smithi105.stdout: "sessions_open": 3, 2024-05-09T09:02:59.718 INFO:teuthology.orchestra.run.smithi105.stdout: "sessions_stale": 0, 2024-05-09T09:02:59.718 INFO:teuthology.orchestra.run.smithi105.stdout: "total_load": 0, 2024-05-09T09:02:59.718 INFO:teuthology.orchestra.run.smithi105.stdout: "average_load": 0, 2024-05-09T09:02:59.718 INFO:teuthology.orchestra.run.smithi105.stdout: "avg_session_uptime": 4706, 2024-05-09T09:02:59.718 INFO:teuthology.orchestra.run.smithi105.stdout: "mdthresh_evicted": 0 2024-05-09T09:02:59.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.718 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-05-09T09:02:59.718 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:02:59.718 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-05-09T09:02:59.718 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:02:59.718 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:02:59.718 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:02:59.718 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:02:59.719 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:02:59.720 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:02:59.720 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:02:59.720 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:02:59.720 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:02:59.720 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-05-09T09:02:59.720 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:02:59.720 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:02:59.720 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:02:59.720 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-05-09T09:02:59.720 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-05-09T09:02:59.720 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-05-09T09:02:59.720 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:02:59.720 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:02:59.720 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:02:59.721 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:02:59.721 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 189473, 2024-05-09T09:02:59.721 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 53, 2024-05-09T09:02:59.721 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:02:59.721 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 0, 2024-05-09T09:02:59.721 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-05-09T09:02:59.721 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-05-09T09:02:59.721 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:02:59.721 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-05-09T09:02:59.721 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:02:59.721 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-05-09T09:02:59.721 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 9712, 2024-05-09T09:02:59.721 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 187, 2024-05-09T09:02:59.721 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 8464, 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 13, 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter": { 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: "op_active": 0, 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: "op_laggy": 0, 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: "op_send": 2462, 2024-05-09T09:02:59.722 INFO:teuthology.orchestra.run.smithi105.stdout: "op_send_bytes": 21, 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: "op_resend": 3, 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: "op_reply": 2462, 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: "op_latency": { 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 2462, 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 22.656552720, 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.009202499 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: "op_inflight": 0, 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: "oplen_avg": { 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 2462, 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 2463 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: "op": 2462, 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: "op_r": 2462, 2024-05-09T09:02:59.723 INFO:teuthology.orchestra.run.smithi105.stdout: "op_w": 0, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "op_rmw": 0, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "op_pg": 0, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_stat": 1610, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_create": 0, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_read": 850, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_write": 0, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_writefull": 0, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_writesame": 0, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_append": 0, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_zero": 0, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_truncate": 0, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_delete": 0, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_mapext": 0, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_sparse_read": 0, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_clonerange": 0, 2024-05-09T09:02:59.724 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_getxattr": 1, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_setxattr": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_rmxattr": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_call": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_watch": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_notify": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_pgls": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_other": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_active": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_send": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_resend": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_ping": 0, 2024-05-09T09:02:59.725 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_active": 0, 2024-05-09T09:02:59.726 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_send": 0, 2024-05-09T09:02:59.726 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_resend": 0, 2024-05-09T09:02:59.726 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_active": 0, 2024-05-09T09:02:59.726 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_send": 0, 2024-05-09T09:02:59.726 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_resend": 0, 2024-05-09T09:02:59.726 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_active": 0, 2024-05-09T09:02:59.726 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_send": 0, 2024-05-09T09:02:59.726 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_resend": 0, 2024-05-09T09:02:59.726 INFO:teuthology.orchestra.run.smithi105.stdout: "command_active": 0, 2024-05-09T09:02:59.726 INFO:teuthology.orchestra.run.smithi105.stdout: "command_send": 0, 2024-05-09T09:02:59.726 INFO:teuthology.orchestra.run.smithi105.stdout: "command_resend": 0, 2024-05-09T09:02:59.726 INFO:teuthology.orchestra.run.smithi105.stdout: "map_epoch": 109, 2024-05-09T09:02:59.726 INFO:teuthology.orchestra.run.smithi105.stdout: "map_full": 0, 2024-05-09T09:02:59.726 INFO:teuthology.orchestra.run.smithi105.stdout: "map_inc": 19, 2024-05-09T09:02:59.727 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_sessions": 5, 2024-05-09T09:02:59.727 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_session_open": 5, 2024-05-09T09:02:59.727 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_session_close": 0, 2024-05-09T09:02:59.727 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_laggy": 0, 2024-05-09T09:02:59.727 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_wr": 0, 2024-05-09T09:02:59.727 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_rd": 2, 2024-05-09T09:02:59.727 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_del": 0 2024-05-09T09:02:59.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.727 INFO:teuthology.orchestra.run.smithi105.stdout: "oft": { 2024-05-09T09:02:59.727 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_objs": 0, 2024-05-09T09:02:59.727 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:02:59.727 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_updates": 0, 2024-05-09T09:02:59.727 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_removes": 0 2024-05-09T09:02:59.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: "osd-slow-ops": { 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: "slow_ops_count": 0 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: "purge_queue": { 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_ops": 0, 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing": 0, 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_high_water": 0, 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executed_ops": 0, 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executed": 0, 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_item_in_journal": 0 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-05-09T09:02:59.728 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:59.729 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 2757, 2024-05-09T09:02:59.729 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 2128666, 2024-05-09T09:02:59.729 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:59.729 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 2757, 2024-05-09T09:02:59.729 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-09T09:02:59.729 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-09T09:02:59.729 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 2757, 2024-05-09T09:02:59.729 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 2128666, 2024-05-09T09:02:59.729 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:59.729 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.729 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.729 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.729 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:59.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.730 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-objecter_bytes": { 2024-05-09T09:02:59.730 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:59.730 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-05-09T09:02:59.730 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:59.730 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-09T09:02:59.730 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-09T09:02:59.730 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:59.730 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:59.730 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 2462, 2024-05-09T09:02:59.730 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 2580, 2024-05-09T09:02:59.730 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 2462, 2024-05-09T09:02:59.730 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 2580, 2024-05-09T09:02:59.730 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:59.730 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-objecter_ops": { 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 1024, 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 2462, 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 2462, 2024-05-09T09:02:59.731 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 2462, 2024-05-09T09:02:59.732 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 2462, 2024-05-09T09:02:59.732 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:59.732 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.732 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.732 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.732 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:59.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.732 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:02:59.732 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:59.732 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 3758096384, 2024-05-09T09:02:59.732 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:59.732 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-09T09:02:59.732 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-09T09:02:59.732 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:59.733 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:59.733 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-09T09:02:59.733 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-09T09:02:59.733 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 0, 2024-05-09T09:02:59.733 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 0, 2024-05-09T09:02:59.733 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:59.733 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.733 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.733 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.733 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:59.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-09T09:02:59.733 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-write_buf_throttle-0x55c38832e6e0": { 2024-05-09T09:02:59.733 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-09T09:02:59.733 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 3758096384, 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 0, 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 0, 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:59.734 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-09T09:02:59.735 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-05-09T09:03:00.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:00 smithi098 ceph-mon[30092]: pgmap v387: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-05-09T09:03:00.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:00 smithi142 ceph-mon[34625]: pgmap v387: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-05-09T09:03:00.461 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:00 smithi105 ceph-mon[34498]: pgmap v387: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-05-09T09:03:00.489 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-05-09T09:03:00.489 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-05-09T09:03:00.489 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-05-09T09:03:00.489 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-05-09T09:03:00.489 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T09:03:00.490 DEBUG:tasks.check_counter:Getting stats from c 2024-05-09T09:03:00.490 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph daemon mds.c perf dump 2024-05-09T09:03:00.765 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:03:02.275 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:02 smithi142 ceph-mon[34625]: pgmap v388: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T09:03:02.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:02 smithi098 ceph-mon[30092]: pgmap v388: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T09:03:02.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:02 smithi105 ceph-mon[34498]: pgmap v388: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T09:03:02.683 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-05-09T09:03:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:03:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 40, 2024-05-09T09:03:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 195, 2024-05-09T09:03:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 9304, 2024-05-09T09:03:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 463577, 2024-05-09T09:03:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 18, 2024-05-09T09:03:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 4, 2024-05-09T09:03:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.018316238, 2024-05-09T09:03:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.006819924, 2024-05-09T09:03:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.004989230, 2024-05-09T09:03:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.000648815, 2024-05-09T09:03:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:03:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 189, 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.024592543, 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000130119 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 15, 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000002111, 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000140 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 355216 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 63, 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 99, 2024-05-09T09:03:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 14032, 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 567090, 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 45, 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 4, 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.021227197, 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.002879986, 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.016299169, 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.000320380, 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 97, 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.017442463, 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000179819 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 54, 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000007069, 2024-05-09T09:03:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000130 2024-05-09T09:03:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 5040, 2024-05-09T09:03:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 132976 2024-05-09T09:03:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:03:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 807, 2024-05-09T09:03:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 951, 2024-05-09T09:03:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 1862606, 2024-05-09T09:03:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 461634, 2024-05-09T09:03:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 6, 2024-05-09T09:03:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 4, 2024-05-09T09:03:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.125776091, 2024-05-09T09:03:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.043021334, 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.080024647, 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.013356298, 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 954, 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.050992117, 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000053450 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 1117, 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000355985, 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000318 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 1701856, 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 165648 2024-05-09T09:03:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "cct": { 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "total_workers": 1, 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "unhealthy_workers": 0 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-MDSRank": { 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 57, 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.002123657, 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000037257 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-PurgeQueue": { 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2024-05-09T09:03:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 2, 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000008438, 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000004219 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "mds": { 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "request": 0, 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "reply": 0, 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "reply_latency": { 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "slow_reply": 0, 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "forward": 0, 2024-05-09T09:03:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch_complete": 0, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch_keys": 0, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_commit": 11, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_split": 0, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_merge": 0, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes": 10, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_top": 0, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_bottom": 10, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pin_tail": 0, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pinned": 10, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_expired": 0, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_with_caps": 0, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "caps": 0, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "subtrees": 1, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse": 0, 2024-05-09T09:03:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_hit": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_forward": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_discover": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_remote_ino": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_lock": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "load_cent": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "q": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "exported": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "exported_inodes": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "imported": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "imported_inodes": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_dir_fetch": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_backtrace_fetch": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_peer_discover": 0, 2024-05-09T09:03:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rfiles": 22, 2024-05-09T09:03:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rbytes": 10738142493, 2024-05-09T09:03:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rsnaps": 6, 2024-05-09T09:03:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T09:03:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_set_tag": 0, 2024-05-09T09:03:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:03:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:03:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dir_inodes": 0, 2024-05-09T09:03:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T09:03:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T09:03:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_file_inodes": 0, 2024-05-09T09:03:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_inode_file_caps": 0, 2024-05-09T09:03:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T09:03:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_grant": 0, 2024-05-09T09:03:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:03:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:03:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:03:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_caps": 0, 2024-05-09T09:03:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_caps_dirty": 0, 2024-05-09T09:03:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_cap_release": 0, 2024-05-09T09:03:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "process_request_cap_release": 0 2024-05-09T09:03:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_cache": { 2024-05-09T09:03:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_update": 0, 2024-05-09T09:03:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_update_receipt": 0, 2024-05-09T09:03:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_try_discover": 0, 2024-05-09T09:03:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_send_discover": 1, 2024-05-09T09:03:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_handle_discover": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_delayed": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_created": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_enqueued": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_reintegrated": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_migrated": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_processing": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_started": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_completed": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:03:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_exportdir": 0, 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_flush": 0, 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragmentdir": 0, 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragstats": 0, 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_inodestats": 0 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_log": { 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "evlrg": 0, 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "evadd": 9, 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "evex": 0, 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "evtrm": 0, 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "ev": 9, 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "evexg": 0, 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "evexd": 0, 2024-05-09T09:03:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "segadd": 2, 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "segex": 0, 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "segtrm": 0, 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "seg": 2, 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "segmjr": 2, 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "segexg": 0, 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "segexd": 0, 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "expos": 4194304, 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "wrpos": 4196879, 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "rdpos": 0, 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "jlat": { 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 8, 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.022560617, 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.002820077 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "replayed": 0 2024-05-09T09:03:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_mem": { 2024-05-09T09:03:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "ino": 13, 2024-05-09T09:03:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "ino+": 13, 2024-05-09T09:03:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "ino-": 0, 2024-05-09T09:03:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "dir": 11, 2024-05-09T09:03:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "dir+": 12, 2024-05-09T09:03:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "dir-": 1, 2024-05-09T09:03:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "dn": 10, 2024-05-09T09:03:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "dn+": 10, 2024-05-09T09:03:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "dn-": 0, 2024-05-09T09:03:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "cap": 0, 2024-05-09T09:03:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "cap+": 0, 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "cap-": 0, 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "rss": 52412, 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "heap": 223612 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_server": { 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_client_request": 0, 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_server_request": 0, 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_request": 0, 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_session": 22, 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_peer_request": 0, 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "req_create_latency": { 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getattr_latency": { 2024-05-09T09:03:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getfilelock_latency": { 2024-05-09T09:03:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "req_link_latency": { 2024-05-09T09:03:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookup_latency": { 2024-05-09T09:03:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookuphash_latency": { 2024-05-09T09:03:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupino_latency": { 2024-05-09T09:03:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupname_latency": { 2024-05-09T09:03:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupparent_latency": { 2024-05-09T09:03:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupsnap_latency": { 2024-05-09T09:03:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.703 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.703 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.703 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lssnap_latency": { 2024-05-09T09:03:02.703 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.703 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.703 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.703 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mkdir_latency": { 2024-05-09T09:03:02.703 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.704 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.704 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.704 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mknod_latency": { 2024-05-09T09:03:02.704 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.704 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.704 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.704 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mksnap_latency": { 2024-05-09T09:03:02.704 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.704 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "req_open_latency": { 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "req_readdir_latency": { 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rename_latency": { 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: "req_renamesnap_latency": { 2024-05-09T09:03:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: "req_snapdiff_latency": { 2024-05-09T09:03:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmdir_latency": { 2024-05-09T09:03:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmsnap_latency": { 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmxattr_latency": { 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setattr_latency": { 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setdirlayout_latency": { 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setfilelock_latency": { 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setlayout_latency": { 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setxattr_latency": { 2024-05-09T09:03:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: "req_symlink_latency": { 2024-05-09T09:03:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: "req_unlink_latency": { 2024-05-09T09:03:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:03:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:03:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getvxattr_latency": { 2024-05-09T09:03:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_sessions": { 2024-05-09T09:03:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: "session_count": 1, 2024-05-09T09:03:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: "session_add": 4, 2024-05-09T09:03:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "session_remove": 3, 2024-05-09T09:03:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_open": 1, 2024-05-09T09:03:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_stale": 0, 2024-05-09T09:03:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "total_load": 0, 2024-05-09T09:03:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "average_load": 0, 2024-05-09T09:03:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "avg_session_uptime": 717, 2024-05-09T09:03:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "mdthresh_evicted": 0 2024-05-09T09:03:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "mempool": { 2024-05-09T09:03:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:03:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_items": 0, 2024-05-09T09:03:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:03:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:03:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_items": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_items": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:03:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:03:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:03:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:03:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:03:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:03:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:03:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_items": 0, 2024-05-09T09:03:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:03:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:03:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:03:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_items": 0, 2024-05-09T09:03:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_bytes": 0, 2024-05-09T09:03:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_items": 0, 2024-05-09T09:03:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:03:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:03:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:03:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:03:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_bytes": 229991, 2024-05-09T09:03:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_items": 70, 2024-05-09T09:03:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:03:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_items": 0, 2024-05-09T09:03:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_bytes": 0, 2024-05-09T09:03:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_items": 0, 2024-05-09T09:03:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:03:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_items": 0, 2024-05-09T09:03:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:03:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_items": 0, 2024-05-09T09:03:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_bytes": 9712, 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_items": 187, 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_bytes": 0, 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_items": 0, 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_bytes": 50664, 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_items": 198, 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_bytes": 0, 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_items": 0, 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_bytes": 0, 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_items": 0 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "objecter": { 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "op_active": 0, 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "op_laggy": 0, 2024-05-09T09:03:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send": 46, 2024-05-09T09:03:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send_bytes": 13057, 2024-05-09T09:03:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: "op_resend": 33, 2024-05-09T09:03:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: "op_reply": 46, 2024-05-09T09:03:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: "op_latency": { 2024-05-09T09:03:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 46, 2024-05-09T09:03:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.236995648, 2024-05-09T09:03:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.005152079 2024-05-09T09:03:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: "op_inflight": 0, 2024-05-09T09:03:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: "oplen_avg": { 2024-05-09T09:03:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 46, 2024-05-09T09:03:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 68 2024-05-09T09:03:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: "op": 46, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "op_r": 0, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "op_w": 46, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "op_rmw": 0, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "op_pg": 0, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_stat": 0, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_create": 10, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_read": 0, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_write": 8, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writefull": 9, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writesame": 0, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_append": 0, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_zero": 0, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_truncate": 0, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_delete": 6, 2024-05-09T09:03:02.717 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_mapext": 0, 2024-05-09T09:03:02.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_sparse_read": 0, 2024-05-09T09:03:02.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_clonerange": 0, 2024-05-09T09:03:02.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_getxattr": 0, 2024-05-09T09:03:02.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_setxattr": 20, 2024-05-09T09:03:02.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:03:02.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_rmxattr": 0, 2024-05-09T09:03:02.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:03:02.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_call": 0, 2024-05-09T09:03:02.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_watch": 0, 2024-05-09T09:03:02.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_notify": 0, 2024-05-09T09:03:02.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:03:02.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls": 0, 2024-05-09T09:03:02.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:03:02.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_other": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_active": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_send": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_resend": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_ping": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_active": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_send": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_resend": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_active": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_send": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_resend": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_active": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_send": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_resend": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "command_active": 0, 2024-05-09T09:03:02.719 INFO:teuthology.orchestra.run.smithi142.stdout: "command_send": 0, 2024-05-09T09:03:02.720 INFO:teuthology.orchestra.run.smithi142.stdout: "command_resend": 0, 2024-05-09T09:03:02.720 INFO:teuthology.orchestra.run.smithi142.stdout: "map_epoch": 109, 2024-05-09T09:03:02.720 INFO:teuthology.orchestra.run.smithi142.stdout: "map_full": 0, 2024-05-09T09:03:02.720 INFO:teuthology.orchestra.run.smithi142.stdout: "map_inc": 19, 2024-05-09T09:03:02.720 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_sessions": 8, 2024-05-09T09:03:02.720 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_open": 8, 2024-05-09T09:03:02.720 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_close": 0, 2024-05-09T09:03:02.720 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_laggy": 0, 2024-05-09T09:03:02.720 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_wr": 14, 2024-05-09T09:03:02.720 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_rd": 0, 2024-05-09T09:03:02.720 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_del": 1 2024-05-09T09:03:02.720 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.720 INFO:teuthology.orchestra.run.smithi142.stdout: "oft": { 2024-05-09T09:03:02.720 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_objs": 1, 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_updates": 0, 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_removes": 0 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: "osd-slow-ops": { 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: "slow_ops_count": 0 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: "purge_queue": { 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops": 0, 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing": 0, 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_high_water": 0, 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed_ops": 0, 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed": 0, 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_item_in_journal": 0 2024-05-09T09:03:02.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.722 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:03:02.722 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:02.722 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-05-09T09:03:02.722 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:02.722 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 906, 2024-05-09T09:03:02.722 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 1784286, 2024-05-09T09:03:02.722 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:02.722 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 906, 2024-05-09T09:03:02.722 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-05-09T09:03:02.722 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-05-09T09:03:02.722 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 906, 2024-05-09T09:03:02.722 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 1784286, 2024-05-09T09:03:02.722 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:02.722 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.723 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.723 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.723 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:02.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.723 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_bytes": { 2024-05-09T09:03:02.723 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:02.723 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-05-09T09:03:02.723 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:02.723 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-05-09T09:03:02.723 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-05-09T09:03:02.723 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:02.723 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-05-09T09:03:02.723 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 46, 2024-05-09T09:03:02.723 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 13057, 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 46, 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 13057, 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_ops": { 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 1024, 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:02.724 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 46, 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 46, 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 46, 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 46, 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:02.725 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-05-09T09:03:02.726 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-05-09T09:03:02.726 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:02.726 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-05-09T09:03:02.726 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-05-09T09:03:02.726 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-05-09T09:03:02.726 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 0, 2024-05-09T09:03:02.726 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 0, 2024-05-09T09:03:02.726 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:02.726 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.726 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.726 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.726 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:02.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:02.726 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle-0x55e8b4b5e500": { 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 9, 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 2575, 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 9, 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 8, 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 2575, 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:02.727 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:02.728 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-05-09T09:03:03.510 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-05-09T09:03:03.510 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-05-09T09:03:03.510 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-05-09T09:03:03.510 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-05-09T09:03:03.510 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T09:03:03.510 DEBUG:tasks.check_counter:Getting stats from f 2024-05-09T09:03:03.510 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph daemon mds.f perf dump 2024-05-09T09:03:03.761 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:03:04.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:04 smithi142 ceph-mon[34625]: pgmap v389: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T09:03:04.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:04 smithi098 ceph-mon[30092]: pgmap v389: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T09:03:04.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:04 smithi105 ceph-mon[34498]: pgmap v389: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T09:03:05.667 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-05-09T09:03:05.668 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:03:05.668 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 2528, 2024-05-09T09:03:05.668 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 2530, 2024-05-09T09:03:05.668 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 715523, 2024-05-09T09:03:05.668 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 803911, 2024-05-09T09:03:05.668 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 6, 2024-05-09T09:03:05.668 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 2, 2024-05-09T09:03:05.668 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.223042479, 2024-05-09T09:03:05.668 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.069988579, 2024-05-09T09:03:05.668 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.056402035, 2024-05-09T09:03:05.668 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.081050038, 2024-05-09T09:03:05.668 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:03:05.669 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 2528, 2024-05-09T09:03:05.669 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.098479125, 2024-05-09T09:03:05.669 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000038955 2024-05-09T09:03:05.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.669 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:03:05.669 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.669 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.669 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.669 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T09:03:05.669 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T09:03:05.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.669 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:03:05.670 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 293, 2024-05-09T09:03:05.670 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 270, 2024-05-09T09:03:05.670 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 1693388, 2024-05-09T09:03:05.670 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 155034, 2024-05-09T09:03:05.670 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 3, 2024-05-09T09:03:05.670 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 2, 2024-05-09T09:03:05.670 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.054207547, 2024-05-09T09:03:05.670 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.016108930, 2024-05-09T09:03:05.670 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.040247529, 2024-05-09T09:03:05.670 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.007434187, 2024-05-09T09:03:05.670 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:03:05.670 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 269, 2024-05-09T09:03:05.670 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.015170541, 2024-05-09T09:03:05.671 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000056396 2024-05-09T09:03:05.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.671 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:03:05.671 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.671 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.671 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.671 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 1692832, 2024-05-09T09:03:05.671 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 154336 2024-05-09T09:03:05.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.671 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:03:05.671 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 7, 2024-05-09T09:03:05.671 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 190, 2024-05-09T09:03:05.671 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 3612, 2024-05-09T09:03:05.672 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 487310, 2024-05-09T09:03:05.672 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 6, 2024-05-09T09:03:05.672 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 4, 2024-05-09T09:03:05.672 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.011825118, 2024-05-09T09:03:05.672 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.007642559, 2024-05-09T09:03:05.672 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.002298037, 2024-05-09T09:03:05.672 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.000026885, 2024-05-09T09:03:05.672 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:03:05.672 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 188, 2024-05-09T09:03:05.672 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.015320636, 2024-05-09T09:03:05.672 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000081492 2024-05-09T09:03:05.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.672 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:03:05.672 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.673 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.673 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.673 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-05-09T09:03:05.673 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 486992 2024-05-09T09:03:05.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.673 INFO:teuthology.orchestra.run.smithi142.stdout: "cct": { 2024-05-09T09:03:05.673 INFO:teuthology.orchestra.run.smithi142.stdout: "total_workers": 1, 2024-05-09T09:03:05.673 INFO:teuthology.orchestra.run.smithi142.stdout: "unhealthy_workers": 0 2024-05-09T09:03:05.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.673 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-MDSRank": { 2024-05-09T09:03:05.673 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2024-05-09T09:03:05.673 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2024-05-09T09:03:05.673 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 4191, 2024-05-09T09:03:05.674 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.122908644, 2024-05-09T09:03:05.674 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000029326 2024-05-09T09:03:05.674 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:05.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.674 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-PurgeQueue": { 2024-05-09T09:03:05.674 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2024-05-09T09:03:05.674 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2024-05-09T09:03:05.674 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.674 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.674 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.674 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:05.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.674 INFO:teuthology.orchestra.run.smithi142.stdout: "mds": { 2024-05-09T09:03:05.675 INFO:teuthology.orchestra.run.smithi142.stdout: "request": 0, 2024-05-09T09:03:05.675 INFO:teuthology.orchestra.run.smithi142.stdout: "reply": 0, 2024-05-09T09:03:05.675 INFO:teuthology.orchestra.run.smithi142.stdout: "reply_latency": { 2024-05-09T09:03:05.675 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.675 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.675 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.675 INFO:teuthology.orchestra.run.smithi142.stdout: "slow_reply": 0, 2024-05-09T09:03:05.676 INFO:teuthology.orchestra.run.smithi142.stdout: "forward": 0, 2024-05-09T09:03:05.676 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch_complete": 0, 2024-05-09T09:03:05.676 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch_keys": 0, 2024-05-09T09:03:05.676 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_commit": 0, 2024-05-09T09:03:05.676 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_split": 0, 2024-05-09T09:03:05.676 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_merge": 0, 2024-05-09T09:03:05.676 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes": 0, 2024-05-09T09:03:05.676 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_top": 0, 2024-05-09T09:03:05.676 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_bottom": 0, 2024-05-09T09:03:05.676 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pin_tail": 0, 2024-05-09T09:03:05.676 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pinned": 0, 2024-05-09T09:03:05.676 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_expired": 0, 2024-05-09T09:03:05.676 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_with_caps": 0, 2024-05-09T09:03:05.676 INFO:teuthology.orchestra.run.smithi142.stdout: "caps": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "subtrees": 1, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_hit": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_forward": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_discover": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_remote_ino": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_lock": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "load_cent": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "q": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "exported": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "exported_inodes": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "imported": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "imported_inodes": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_dir_fetch": 0, 2024-05-09T09:03:05.677 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_backtrace_fetch": 0, 2024-05-09T09:03:05.678 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_peer_discover": 0, 2024-05-09T09:03:05.678 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rfiles": 0, 2024-05-09T09:03:05.678 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rbytes": 0, 2024-05-09T09:03:05.678 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rsnaps": 0, 2024-05-09T09:03:05.678 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T09:03:05.678 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_set_tag": 0, 2024-05-09T09:03:05.678 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:03:05.678 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:03:05.678 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dir_inodes": 0, 2024-05-09T09:03:05.678 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T09:03:05.678 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T09:03:05.678 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_file_inodes": 0, 2024-05-09T09:03:05.678 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_inode_file_caps": 0, 2024-05-09T09:03:05.678 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T09:03:05.679 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_grant": 0, 2024-05-09T09:03:05.679 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:03:05.679 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:03:05.679 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:03:05.679 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_caps": 0, 2024-05-09T09:03:05.679 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_caps_dirty": 0, 2024-05-09T09:03:05.679 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_cap_release": 0, 2024-05-09T09:03:05.679 INFO:teuthology.orchestra.run.smithi142.stdout: "process_request_cap_release": 0 2024-05-09T09:03:05.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.679 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_cache": { 2024-05-09T09:03:05.679 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_update": 0, 2024-05-09T09:03:05.679 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_update_receipt": 0, 2024-05-09T09:03:05.679 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_try_discover": 0, 2024-05-09T09:03:05.679 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_send_discover": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_handle_discover": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_delayed": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_created": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_enqueued": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_reintegrated": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_migrated": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_processing": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_started": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_completed": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:03:05.680 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:03:05.681 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T09:03:05.681 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_exportdir": 0, 2024-05-09T09:03:05.681 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_flush": 0, 2024-05-09T09:03:05.681 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragmentdir": 0, 2024-05-09T09:03:05.681 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragstats": 0, 2024-05-09T09:03:05.681 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_inodestats": 0 2024-05-09T09:03:05.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.681 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_log": { 2024-05-09T09:03:05.681 INFO:teuthology.orchestra.run.smithi142.stdout: "evlrg": 0, 2024-05-09T09:03:05.681 INFO:teuthology.orchestra.run.smithi142.stdout: "evadd": 0, 2024-05-09T09:03:05.681 INFO:teuthology.orchestra.run.smithi142.stdout: "evex": 0, 2024-05-09T09:03:05.681 INFO:teuthology.orchestra.run.smithi142.stdout: "evtrm": 0, 2024-05-09T09:03:05.681 INFO:teuthology.orchestra.run.smithi142.stdout: "ev": 5, 2024-05-09T09:03:05.681 INFO:teuthology.orchestra.run.smithi142.stdout: "evexg": 0, 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "evexd": 0, 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "segadd": 0, 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "segex": 0, 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "segtrm": 0, 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "seg": 2, 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "segmjr": 2, 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "segexg": 0, 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "segexd": 0, 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "expos": 4194304, 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "wrpos": 4195732, 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "rdpos": 4195424, 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "jlat": { 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.683 INFO:teuthology.orchestra.run.smithi142.stdout: "replayed": 5 2024-05-09T09:03:05.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.683 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_mem": { 2024-05-09T09:03:05.683 INFO:teuthology.orchestra.run.smithi142.stdout: "ino": 3, 2024-05-09T09:03:05.683 INFO:teuthology.orchestra.run.smithi142.stdout: "ino+": 3, 2024-05-09T09:03:05.683 INFO:teuthology.orchestra.run.smithi142.stdout: "ino-": 0, 2024-05-09T09:03:05.683 INFO:teuthology.orchestra.run.smithi142.stdout: "dir": 1, 2024-05-09T09:03:05.683 INFO:teuthology.orchestra.run.smithi142.stdout: "dir+": 1, 2024-05-09T09:03:05.683 INFO:teuthology.orchestra.run.smithi142.stdout: "dir-": 0, 2024-05-09T09:03:05.683 INFO:teuthology.orchestra.run.smithi142.stdout: "dn": 0, 2024-05-09T09:03:05.683 INFO:teuthology.orchestra.run.smithi142.stdout: "dn+": 0, 2024-05-09T09:03:05.683 INFO:teuthology.orchestra.run.smithi142.stdout: "dn-": 0, 2024-05-09T09:03:05.683 INFO:teuthology.orchestra.run.smithi142.stdout: "cap": 0, 2024-05-09T09:03:05.684 INFO:teuthology.orchestra.run.smithi142.stdout: "cap+": 0, 2024-05-09T09:03:05.684 INFO:teuthology.orchestra.run.smithi142.stdout: "cap-": 0, 2024-05-09T09:03:05.684 INFO:teuthology.orchestra.run.smithi142.stdout: "rss": 50348, 2024-05-09T09:03:05.684 INFO:teuthology.orchestra.run.smithi142.stdout: "heap": 215420 2024-05-09T09:03:05.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.684 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_server": { 2024-05-09T09:03:05.684 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_client_request": 0, 2024-05-09T09:03:05.684 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_server_request": 0, 2024-05-09T09:03:05.684 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_request": 0, 2024-05-09T09:03:05.684 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_session": 0, 2024-05-09T09:03:05.684 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_peer_request": 0, 2024-05-09T09:03:05.684 INFO:teuthology.orchestra.run.smithi142.stdout: "req_create_latency": { 2024-05-09T09:03:05.684 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.684 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getattr_latency": { 2024-05-09T09:03:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getfilelock_latency": { 2024-05-09T09:03:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: "req_link_latency": { 2024-05-09T09:03:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookup_latency": { 2024-05-09T09:03:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookuphash_latency": { 2024-05-09T09:03:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupino_latency": { 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupname_latency": { 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupparent_latency": { 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupsnap_latency": { 2024-05-09T09:03:05.688 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.688 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.688 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.688 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lssnap_latency": { 2024-05-09T09:03:05.688 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.688 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.688 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.688 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mkdir_latency": { 2024-05-09T09:03:05.688 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.689 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.689 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.689 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mknod_latency": { 2024-05-09T09:03:05.689 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.689 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.689 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.689 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mksnap_latency": { 2024-05-09T09:03:05.689 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.690 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.690 INFO:teuthology.orchestra.run.smithi142.stdout: "req_open_latency": { 2024-05-09T09:03:05.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.690 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.690 INFO:teuthology.orchestra.run.smithi142.stdout: "req_readdir_latency": { 2024-05-09T09:03:05.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.690 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.690 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rename_latency": { 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: "req_renamesnap_latency": { 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: "req_snapdiff_latency": { 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.691 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmdir_latency": { 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmsnap_latency": { 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmxattr_latency": { 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.692 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setattr_latency": { 2024-05-09T09:03:05.693 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.693 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.693 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.693 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setdirlayout_latency": { 2024-05-09T09:03:05.693 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.693 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.693 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.693 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setfilelock_latency": { 2024-05-09T09:03:05.693 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.693 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.693 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.694 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setlayout_latency": { 2024-05-09T09:03:05.694 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.694 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.694 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.694 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setxattr_latency": { 2024-05-09T09:03:05.694 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.694 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.694 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.694 INFO:teuthology.orchestra.run.smithi142.stdout: "req_symlink_latency": { 2024-05-09T09:03:05.694 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.694 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.695 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.695 INFO:teuthology.orchestra.run.smithi142.stdout: "req_unlink_latency": { 2024-05-09T09:03:05.695 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.695 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.695 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.695 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:03:05.695 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:03:05.695 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getvxattr_latency": { 2024-05-09T09:03:05.695 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.695 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.695 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.695 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:05.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.696 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_sessions": { 2024-05-09T09:03:05.696 INFO:teuthology.orchestra.run.smithi142.stdout: "session_count": 1, 2024-05-09T09:03:05.696 INFO:teuthology.orchestra.run.smithi142.stdout: "session_add": 2, 2024-05-09T09:03:05.696 INFO:teuthology.orchestra.run.smithi142.stdout: "session_remove": 1, 2024-05-09T09:03:05.696 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_open": 1, 2024-05-09T09:03:05.696 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_stale": 0, 2024-05-09T09:03:05.696 INFO:teuthology.orchestra.run.smithi142.stdout: "total_load": 0, 2024-05-09T09:03:05.696 INFO:teuthology.orchestra.run.smithi142.stdout: "average_load": 0, 2024-05-09T09:03:05.696 INFO:teuthology.orchestra.run.smithi142.stdout: "avg_session_uptime": 2071, 2024-05-09T09:03:05.696 INFO:teuthology.orchestra.run.smithi142.stdout: "mdthresh_evicted": 0 2024-05-09T09:03:05.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.696 INFO:teuthology.orchestra.run.smithi142.stdout: "mempool": { 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_items": 0, 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:03:05.697 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:03:05.698 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_items": 0, 2024-05-09T09:03:05.698 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:03:05.698 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_items": 0, 2024-05-09T09:03:05.698 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:03:05.698 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:03:05.698 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:03:05.698 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:03:05.698 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:03:05.698 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:03:05.698 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:03:05.698 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_items": 0, 2024-05-09T09:03:05.698 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:03:05.698 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:03:05.699 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:03:05.699 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_items": 0, 2024-05-09T09:03:05.699 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_bytes": 0, 2024-05-09T09:03:05.699 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_items": 0, 2024-05-09T09:03:05.699 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:03:05.699 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:03:05.699 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:03:05.699 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:03:05.699 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_bytes": 181393, 2024-05-09T09:03:05.699 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_items": 49, 2024-05-09T09:03:05.699 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:03:05.699 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_items": 0, 2024-05-09T09:03:05.699 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_bytes": 0, 2024-05-09T09:03:05.699 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_items": 0, 2024-05-09T09:03:05.700 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:03:05.700 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_items": 0, 2024-05-09T09:03:05.700 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:03:05.700 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_items": 0, 2024-05-09T09:03:05.700 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_bytes": 9712, 2024-05-09T09:03:05.700 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_items": 187, 2024-05-09T09:03:05.700 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:03:05.700 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:03:05.700 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_bytes": 0, 2024-05-09T09:03:05.700 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_items": 0, 2024-05-09T09:03:05.700 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_bytes": 8464, 2024-05-09T09:03:05.700 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_items": 13, 2024-05-09T09:03:05.700 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_bytes": 0, 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_items": 0, 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_bytes": 0, 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_items": 0 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: "objecter": { 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: "op_active": 0, 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: "op_laggy": 0, 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send": 2531, 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send_bytes": 21, 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: "op_resend": 3, 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: "op_reply": 2531, 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: "op_latency": { 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 2531, 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 5.639895718, 2024-05-09T09:03:05.701 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.002228327 2024-05-09T09:03:05.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.702 INFO:teuthology.orchestra.run.smithi142.stdout: "op_inflight": 0, 2024-05-09T09:03:05.702 INFO:teuthology.orchestra.run.smithi142.stdout: "oplen_avg": { 2024-05-09T09:03:05.702 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 2531, 2024-05-09T09:03:05.702 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 2532 2024-05-09T09:03:05.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.702 INFO:teuthology.orchestra.run.smithi142.stdout: "op": 2531, 2024-05-09T09:03:05.702 INFO:teuthology.orchestra.run.smithi142.stdout: "op_r": 2531, 2024-05-09T09:03:05.702 INFO:teuthology.orchestra.run.smithi142.stdout: "op_w": 0, 2024-05-09T09:03:05.702 INFO:teuthology.orchestra.run.smithi142.stdout: "op_rmw": 0, 2024-05-09T09:03:05.702 INFO:teuthology.orchestra.run.smithi142.stdout: "op_pg": 0, 2024-05-09T09:03:05.702 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_stat": 1658, 2024-05-09T09:03:05.702 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_create": 0, 2024-05-09T09:03:05.702 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_read": 871, 2024-05-09T09:03:05.703 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_write": 0, 2024-05-09T09:03:05.703 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writefull": 0, 2024-05-09T09:03:05.703 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writesame": 0, 2024-05-09T09:03:05.703 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_append": 0, 2024-05-09T09:03:05.703 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_zero": 0, 2024-05-09T09:03:05.703 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_truncate": 0, 2024-05-09T09:03:05.703 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_delete": 0, 2024-05-09T09:03:05.703 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_mapext": 0, 2024-05-09T09:03:05.703 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_sparse_read": 0, 2024-05-09T09:03:05.703 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_clonerange": 0, 2024-05-09T09:03:05.703 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_getxattr": 1, 2024-05-09T09:03:05.703 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_setxattr": 0, 2024-05-09T09:03:05.703 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:03:05.703 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_rmxattr": 0, 2024-05-09T09:03:05.704 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:03:05.704 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_call": 0, 2024-05-09T09:03:05.704 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_watch": 0, 2024-05-09T09:03:05.704 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_notify": 0, 2024-05-09T09:03:05.704 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:03:05.704 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls": 0, 2024-05-09T09:03:05.704 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:03:05.704 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_other": 0, 2024-05-09T09:03:05.704 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_active": 0, 2024-05-09T09:03:05.704 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_send": 0, 2024-05-09T09:03:05.704 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_resend": 0, 2024-05-09T09:03:05.704 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_ping": 0, 2024-05-09T09:03:05.704 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_active": 0, 2024-05-09T09:03:05.704 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_send": 0, 2024-05-09T09:03:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_resend": 0, 2024-05-09T09:03:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_active": 0, 2024-05-09T09:03:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_send": 0, 2024-05-09T09:03:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_resend": 0, 2024-05-09T09:03:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_active": 0, 2024-05-09T09:03:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_send": 0, 2024-05-09T09:03:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_resend": 0, 2024-05-09T09:03:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: "command_active": 0, 2024-05-09T09:03:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: "command_send": 0, 2024-05-09T09:03:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: "command_resend": 0, 2024-05-09T09:03:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: "map_epoch": 109, 2024-05-09T09:03:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: "map_full": 0, 2024-05-09T09:03:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: "map_inc": 19, 2024-05-09T09:03:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_sessions": 4, 2024-05-09T09:03:05.706 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_open": 4, 2024-05-09T09:03:05.706 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_close": 0, 2024-05-09T09:03:05.706 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_laggy": 0, 2024-05-09T09:03:05.706 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_wr": 0, 2024-05-09T09:03:05.706 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_rd": 2, 2024-05-09T09:03:05.706 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_del": 0 2024-05-09T09:03:05.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.706 INFO:teuthology.orchestra.run.smithi142.stdout: "oft": { 2024-05-09T09:03:05.706 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_objs": 0, 2024-05-09T09:03:05.706 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:03:05.706 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_updates": 0, 2024-05-09T09:03:05.706 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_removes": 0 2024-05-09T09:03:05.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.706 INFO:teuthology.orchestra.run.smithi142.stdout: "osd-slow-ops": { 2024-05-09T09:03:05.707 INFO:teuthology.orchestra.run.smithi142.stdout: "slow_ops_count": 0 2024-05-09T09:03:05.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.707 INFO:teuthology.orchestra.run.smithi142.stdout: "purge_queue": { 2024-05-09T09:03:05.707 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops": 0, 2024-05-09T09:03:05.707 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T09:03:05.707 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing": 0, 2024-05-09T09:03:05.707 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_high_water": 0, 2024-05-09T09:03:05.707 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed_ops": 0, 2024-05-09T09:03:05.707 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed": 0, 2024-05-09T09:03:05.707 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_item_in_journal": 0 2024-05-09T09:03:05.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.707 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:03:05.707 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:05.707 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 2824, 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 2140567, 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 2824, 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 2824, 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 2140567, 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:05.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.709 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_bytes": { 2024-05-09T09:03:05.709 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:05.709 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-05-09T09:03:05.709 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:05.709 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-05-09T09:03:05.709 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-05-09T09:03:05.709 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:05.709 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-05-09T09:03:05.709 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 2531, 2024-05-09T09:03:05.709 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 1433, 2024-05-09T09:03:05.709 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 2531, 2024-05-09T09:03:05.709 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 1433, 2024-05-09T09:03:05.709 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:05.709 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_ops": { 2024-05-09T09:03:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 1024, 2024-05-09T09:03:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-05-09T09:03:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-05-09T09:03:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-05-09T09:03:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 2531, 2024-05-09T09:03:05.711 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 2531, 2024-05-09T09:03:05.711 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 2531, 2024-05-09T09:03:05.711 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 2531, 2024-05-09T09:03:05.711 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:05.711 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.711 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.711 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.711 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:05.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.711 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:03:05.711 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:05.711 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2024-05-09T09:03:05.711 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:05.711 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-05-09T09:03:05.712 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-05-09T09:03:05.712 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:05.712 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-05-09T09:03:05.712 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-05-09T09:03:05.712 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-05-09T09:03:05.712 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 0, 2024-05-09T09:03:05.712 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 0, 2024-05-09T09:03:05.712 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:05.712 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.712 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.712 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.712 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:05.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:05.712 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle-0x556d27b7c6e0": { 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 0, 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 0, 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:05.713 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:05.714 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:05.714 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:05.714 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-05-09T09:03:06.261 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:06 smithi142 ceph-mon[34625]: pgmap v390: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-05-09T09:03:06.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:06 smithi098 ceph-mon[30092]: pgmap v390: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-05-09T09:03:06.476 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-05-09T09:03:06.476 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-05-09T09:03:06.476 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-05-09T09:03:06.476 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-05-09T09:03:06.476 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T09:03:06.476 DEBUG:tasks.check_counter:Getting stats from i 2024-05-09T09:03:06.477 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph daemon mds.i perf dump 2024-05-09T09:03:06.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:06 smithi105 ceph-mon[34498]: pgmap v390: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-05-09T09:03:06.951 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:03:07.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:07 smithi142 ceph-mon[34625]: osdmap e110: 12 total, 12 up, 12 in 2024-05-09T09:03:07.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:07 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:03:07.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:07 smithi105 ceph-mon[34498]: osdmap e110: 12 total, 12 up, 12 in 2024-05-09T09:03:07.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:07 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:03:07.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:07 smithi098 ceph-mon[30092]: osdmap e110: 12 total, 12 up, 12 in 2024-05-09T09:03:07.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:07 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:03:08.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:08 smithi142 ceph-mon[34625]: pgmap v392: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:03:08.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:08 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:03:08.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:08 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:03:08.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:08 smithi105 ceph-mon[34498]: pgmap v392: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:03:08.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:08 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:03:08.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:08 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:03:08.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:08 smithi098 ceph-mon[30092]: pgmap v392: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:03:08.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:08 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:03:08.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:08 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:03:09.109 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-05-09T09:03:09.109 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:03:09.109 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 994, 2024-05-09T09:03:09.109 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 851, 2024-05-09T09:03:09.110 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 283020, 2024-05-09T09:03:09.110 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 785431, 2024-05-09T09:03:09.110 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 48, 2024-05-09T09:03:09.110 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 7, 2024-05-09T09:03:09.110 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.103518490, 2024-05-09T09:03:09.110 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.032755208, 2024-05-09T09:03:09.110 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.072323731, 2024-05-09T09:03:09.110 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.005818220, 2024-05-09T09:03:09.110 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:03:09.110 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 849, 2024-05-09T09:03:09.110 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.078078114, 2024-05-09T09:03:09.110 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000091964 2024-05-09T09:03:09.110 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.110 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:03:09.111 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 1669, 2024-05-09T09:03:09.111 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000469024, 2024-05-09T09:03:09.111 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000281 2024-05-09T09:03:09.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.111 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T09:03:09.111 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T09:03:09.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.111 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:03:09.111 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 42, 2024-05-09T09:03:09.111 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 266, 2024-05-09T09:03:09.111 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 11307, 2024-05-09T09:03:09.111 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 552346, 2024-05-09T09:03:09.111 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 15, 2024-05-09T09:03:09.111 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 4, 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.023999259, 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.011434043, 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.009753910, 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.000244849, 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 264, 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.027286569, 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000103358 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 73, 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000035531, 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000486 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 5280, 2024-05-09T09:03:09.112 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 488224 2024-05-09T09:03:09.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.113 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:03:09.113 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 335, 2024-05-09T09:03:09.113 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 309, 2024-05-09T09:03:09.113 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 1708347, 2024-05-09T09:03:09.113 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 169496, 2024-05-09T09:03:09.113 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 6, 2024-05-09T09:03:09.113 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 5, 2024-05-09T09:03:09.113 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.055141748, 2024-05-09T09:03:09.113 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.016969324, 2024-05-09T09:03:09.113 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.039683177, 2024-05-09T09:03:09.113 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.007906269, 2024-05-09T09:03:09.113 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:03:09.113 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 308, 2024-05-09T09:03:09.114 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.027722572, 2024-05-09T09:03:09.114 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000090008 2024-05-09T09:03:09.114 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.114 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:03:09.114 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.114 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.114 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.114 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.114 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 1705536, 2024-05-09T09:03:09.114 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 159312 2024-05-09T09:03:09.114 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.114 INFO:teuthology.orchestra.run.smithi142.stdout: "cct": { 2024-05-09T09:03:09.114 INFO:teuthology.orchestra.run.smithi142.stdout: "total_workers": 1, 2024-05-09T09:03:09.114 INFO:teuthology.orchestra.run.smithi142.stdout: "unhealthy_workers": 0 2024-05-09T09:03:09.115 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.115 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-MDSRank": { 2024-05-09T09:03:09.115 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2024-05-09T09:03:09.115 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2024-05-09T09:03:09.115 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 53, 2024-05-09T09:03:09.115 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.001651553, 2024-05-09T09:03:09.115 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000031161 2024-05-09T09:03:09.115 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:09.115 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.115 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-PurgeQueue": { 2024-05-09T09:03:09.115 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2024-05-09T09:03:09.115 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2024-05-09T09:03:09.115 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 2, 2024-05-09T09:03:09.115 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000009851, 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000004925 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: "mds": { 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: "request": 6, 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: "reply": 3, 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: "reply_latency": { 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 3, 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.001223309, 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000407769 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: "slow_reply": 0, 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: "forward": 3, 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch_complete": 0, 2024-05-09T09:03:09.116 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch_keys": 0, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_commit": 11, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_split": 0, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_merge": 0, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes": 14, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_top": 0, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_bottom": 14, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pin_tail": 0, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pinned": 14, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_expired": 0, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_with_caps": 2, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "caps": 2, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "subtrees": 2, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse": 8, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_hit": 3, 2024-05-09T09:03:09.117 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_forward": 0, 2024-05-09T09:03:09.118 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_discover": 2, 2024-05-09T09:03:09.118 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:03:09.118 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_remote_ino": 0, 2024-05-09T09:03:09.118 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_lock": 0, 2024-05-09T09:03:09.118 INFO:teuthology.orchestra.run.smithi142.stdout: "load_cent": 0, 2024-05-09T09:03:09.118 INFO:teuthology.orchestra.run.smithi142.stdout: "q": 0, 2024-05-09T09:03:09.118 INFO:teuthology.orchestra.run.smithi142.stdout: "exported": 0, 2024-05-09T09:03:09.118 INFO:teuthology.orchestra.run.smithi142.stdout: "exported_inodes": 0, 2024-05-09T09:03:09.118 INFO:teuthology.orchestra.run.smithi142.stdout: "imported": 0, 2024-05-09T09:03:09.118 INFO:teuthology.orchestra.run.smithi142.stdout: "imported_inodes": 0, 2024-05-09T09:03:09.118 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_dir_fetch": 0, 2024-05-09T09:03:09.118 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_backtrace_fetch": 0, 2024-05-09T09:03:09.118 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_peer_discover": 0, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rfiles": 22, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rbytes": 10738142493, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rsnaps": 6, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_set_tag": 0, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dir_inodes": 0, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_file_inodes": 0, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_inode_file_caps": 0, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_revoke": 2, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_grant": 3, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:03:09.119 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_caps": 2, 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_caps_dirty": 0, 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_cap_release": 0, 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "process_request_cap_release": 3 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_cache": { 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_update": 0, 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_update_receipt": 0, 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_try_discover": 0, 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_send_discover": 3, 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_handle_discover": 0, 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays": 0, 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_delayed": 0, 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:03:09.120 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_created": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_enqueued": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_reintegrated": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_migrated": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_processing": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_started": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_completed": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_exportdir": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_flush": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragmentdir": 0, 2024-05-09T09:03:09.121 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragstats": 0, 2024-05-09T09:03:09.122 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_inodestats": 0 2024-05-09T09:03:09.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.122 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_log": { 2024-05-09T09:03:09.122 INFO:teuthology.orchestra.run.smithi142.stdout: "evlrg": 0, 2024-05-09T09:03:09.122 INFO:teuthology.orchestra.run.smithi142.stdout: "evadd": 5, 2024-05-09T09:03:09.122 INFO:teuthology.orchestra.run.smithi142.stdout: "evex": 0, 2024-05-09T09:03:09.122 INFO:teuthology.orchestra.run.smithi142.stdout: "evtrm": 0, 2024-05-09T09:03:09.122 INFO:teuthology.orchestra.run.smithi142.stdout: "ev": 5, 2024-05-09T09:03:09.122 INFO:teuthology.orchestra.run.smithi142.stdout: "evexg": 0, 2024-05-09T09:03:09.122 INFO:teuthology.orchestra.run.smithi142.stdout: "evexd": 0, 2024-05-09T09:03:09.122 INFO:teuthology.orchestra.run.smithi142.stdout: "segadd": 2, 2024-05-09T09:03:09.122 INFO:teuthology.orchestra.run.smithi142.stdout: "segex": 0, 2024-05-09T09:03:09.122 INFO:teuthology.orchestra.run.smithi142.stdout: "segtrm": 0, 2024-05-09T09:03:09.123 INFO:teuthology.orchestra.run.smithi142.stdout: "seg": 2, 2024-05-09T09:03:09.123 INFO:teuthology.orchestra.run.smithi142.stdout: "segmjr": 2, 2024-05-09T09:03:09.123 INFO:teuthology.orchestra.run.smithi142.stdout: "segexg": 0, 2024-05-09T09:03:09.123 INFO:teuthology.orchestra.run.smithi142.stdout: "segexd": 0, 2024-05-09T09:03:09.123 INFO:teuthology.orchestra.run.smithi142.stdout: "expos": 4194304, 2024-05-09T09:03:09.123 INFO:teuthology.orchestra.run.smithi142.stdout: "wrpos": 4195732, 2024-05-09T09:03:09.123 INFO:teuthology.orchestra.run.smithi142.stdout: "rdpos": 0, 2024-05-09T09:03:09.123 INFO:teuthology.orchestra.run.smithi142.stdout: "jlat": { 2024-05-09T09:03:09.123 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 4, 2024-05-09T09:03:09.123 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.007862390, 2024-05-09T09:03:09.123 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001965597 2024-05-09T09:03:09.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: "replayed": 0 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_mem": { 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: "ino": 17, 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: "ino+": 17, 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: "ino-": 0, 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: "dir": 15, 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: "dir+": 16, 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: "dir-": 1, 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: "dn": 14, 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: "dn+": 14, 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: "dn-": 0, 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: "cap": 2, 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: "cap+": 3, 2024-05-09T09:03:09.124 INFO:teuthology.orchestra.run.smithi142.stdout: "cap-": 1, 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: "rss": 55612, 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: "heap": 223612 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_server": { 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_client_request": 8, 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_server_request": 0, 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_request": 6, 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_session": 14, 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_peer_request": 0, 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: "req_create_latency": { 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getattr_latency": { 2024-05-09T09:03:09.125 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 2, 2024-05-09T09:03:09.126 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.001059208, 2024-05-09T09:03:09.126 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000529604 2024-05-09T09:03:09.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.126 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getfilelock_latency": { 2024-05-09T09:03:09.126 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.126 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.126 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.126 INFO:teuthology.orchestra.run.smithi142.stdout: "req_link_latency": { 2024-05-09T09:03:09.126 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.126 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.126 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.126 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookup_latency": { 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookuphash_latency": { 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupino_latency": { 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 1, 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000164101, 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000164101 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupname_latency": { 2024-05-09T09:03:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupparent_latency": { 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupsnap_latency": { 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lssnap_latency": { 2024-05-09T09:03:09.128 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mkdir_latency": { 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mknod_latency": { 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mksnap_latency": { 2024-05-09T09:03:09.129 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.130 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.130 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.130 INFO:teuthology.orchestra.run.smithi142.stdout: "req_open_latency": { 2024-05-09T09:03:09.130 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.130 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.130 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.130 INFO:teuthology.orchestra.run.smithi142.stdout: "req_readdir_latency": { 2024-05-09T09:03:09.130 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.130 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.130 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.130 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rename_latency": { 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: "req_renamesnap_latency": { 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: "req_snapdiff_latency": { 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.131 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmdir_latency": { 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmsnap_latency": { 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmxattr_latency": { 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.132 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setattr_latency": { 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setdirlayout_latency": { 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setfilelock_latency": { 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.133 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setlayout_latency": { 2024-05-09T09:03:09.134 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.134 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.134 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.134 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setxattr_latency": { 2024-05-09T09:03:09.134 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.134 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.134 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.134 INFO:teuthology.orchestra.run.smithi142.stdout: "req_symlink_latency": { 2024-05-09T09:03:09.134 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.134 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.134 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: "req_unlink_latency": { 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getvxattr_latency": { 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_sessions": { 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: "session_count": 1, 2024-05-09T09:03:09.135 INFO:teuthology.orchestra.run.smithi142.stdout: "session_add": 2, 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "session_remove": 1, 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_open": 1, 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_stale": 0, 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "total_load": 2, 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "average_load": 2, 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "avg_session_uptime": 751, 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "mdthresh_evicted": 0 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "mempool": { 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_items": 0, 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:03:09.136 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_items": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_items": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:03:09.137 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:03:09.138 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:03:09.138 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:03:09.138 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_items": 0, 2024-05-09T09:03:09.138 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:03:09.138 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:03:09.138 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:03:09.138 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_items": 0, 2024-05-09T09:03:09.138 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_bytes": 0, 2024-05-09T09:03:09.138 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_items": 0, 2024-05-09T09:03:09.138 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:03:09.138 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:03:09.138 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:03:09.138 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:03:09.138 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_bytes": 280169, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_items": 102, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_items": 0, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_bytes": 0, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_items": 0, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_items": 0, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_items": 0, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_bytes": 9952, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_items": 193, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_bytes": 0, 2024-05-09T09:03:09.139 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_items": 0, 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_bytes": 68157, 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_items": 293, 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_bytes": 0, 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_items": 0, 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_bytes": 0, 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_items": 0 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "objecter": { 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "op_active": 0, 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "op_laggy": 0, 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send": 42, 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send_bytes": 12135, 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "op_resend": 33, 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "op_reply": 42, 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "op_latency": { 2024-05-09T09:03:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 42, 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.214996055, 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.005118953 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: "op_inflight": 0, 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: "oplen_avg": { 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 42, 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 66 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: "op": 42, 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: "op_r": 0, 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: "op_w": 42, 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: "op_rmw": 0, 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: "op_pg": 0, 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_stat": 0, 2024-05-09T09:03:09.141 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_create": 10, 2024-05-09T09:03:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_read": 0, 2024-05-09T09:03:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_write": 4, 2024-05-09T09:03:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writefull": 7, 2024-05-09T09:03:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writesame": 0, 2024-05-09T09:03:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_append": 0, 2024-05-09T09:03:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_zero": 0, 2024-05-09T09:03:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_truncate": 0, 2024-05-09T09:03:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_delete": 6, 2024-05-09T09:03:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_mapext": 0, 2024-05-09T09:03:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_sparse_read": 0, 2024-05-09T09:03:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_clonerange": 0, 2024-05-09T09:03:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_getxattr": 0, 2024-05-09T09:03:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_setxattr": 20, 2024-05-09T09:03:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_rmxattr": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_call": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_watch": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_notify": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_other": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_active": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_send": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_resend": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_ping": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_active": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_send": 0, 2024-05-09T09:03:09.143 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_resend": 0, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_active": 0, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_send": 0, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_resend": 0, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_active": 0, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_send": 0, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_resend": 0, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "command_active": 0, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "command_send": 0, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "command_resend": 0, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "map_epoch": 110, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "map_full": 0, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "map_inc": 20, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_sessions": 10, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_open": 10, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_close": 0, 2024-05-09T09:03:09.144 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_laggy": 0, 2024-05-09T09:03:09.145 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_wr": 18, 2024-05-09T09:03:09.145 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_rd": 0, 2024-05-09T09:03:09.145 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_del": 1 2024-05-09T09:03:09.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.145 INFO:teuthology.orchestra.run.smithi142.stdout: "oft": { 2024-05-09T09:03:09.145 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_objs": 1, 2024-05-09T09:03:09.145 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:03:09.145 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_updates": 0, 2024-05-09T09:03:09.145 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_removes": 0 2024-05-09T09:03:09.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.145 INFO:teuthology.orchestra.run.smithi142.stdout: "osd-slow-ops": { 2024-05-09T09:03:09.145 INFO:teuthology.orchestra.run.smithi142.stdout: "slow_ops_count": 0 2024-05-09T09:03:09.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.145 INFO:teuthology.orchestra.run.smithi142.stdout: "purge_queue": { 2024-05-09T09:03:09.146 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops": 0, 2024-05-09T09:03:09.146 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T09:03:09.146 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing": 0, 2024-05-09T09:03:09.146 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_high_water": 0, 2024-05-09T09:03:09.146 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed_ops": 0, 2024-05-09T09:03:09.146 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed": 0, 2024-05-09T09:03:09.146 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_item_in_journal": 0 2024-05-09T09:03:09.146 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.146 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:03:09.146 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:09.146 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-05-09T09:03:09.146 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:09.146 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 1367, 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 1860212, 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 1367, 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 1367, 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 1860212, 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_bytes": { 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:09.147 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 42, 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 12135, 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 42, 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 12135, 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.148 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_ops": { 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 1024, 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 42, 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 42, 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 42, 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 42, 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.149 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.150 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:09.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.150 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:03:09.150 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:09.150 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2024-05-09T09:03:09.150 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:09.150 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-05-09T09:03:09.150 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-05-09T09:03:09.150 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:09.150 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-05-09T09:03:09.150 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-05-09T09:03:09.150 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-05-09T09:03:09.150 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 0, 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 0, 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle-0x55d164d2c500": { 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 5, 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 1428, 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:09.151 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 5, 2024-05-09T09:03:09.152 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-05-09T09:03:09.152 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-05-09T09:03:09.152 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 4, 2024-05-09T09:03:09.152 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 1428, 2024-05-09T09:03:09.152 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:09.152 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:09.152 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:09.152 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:09.152 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:09.152 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:09.152 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-05-09T09:03:09.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:09 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:03:09.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:09 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:03:09.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:09 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:03:09.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:09 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:03:09.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:09 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:03:09.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:09 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:03:09.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:09 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:03:09.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:09 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:03:09.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:09 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' 2024-05-09T09:03:09.853 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-05-09T09:03:09.853 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-05-09T09:03:09.853 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-05-09T09:03:09.853 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-05-09T09:03:09.854 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T09:03:09.854 DEBUG:tasks.check_counter:Getting stats from l 2024-05-09T09:03:09.854 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 -- ceph daemon mds.l perf dump 2024-05-09T09:03:10.092 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:03:10.097 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:10 smithi142 ceph-mon[34625]: pgmap v393: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:03:10.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:10 smithi105 ceph-mon[34498]: pgmap v393: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:03:10.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:10 smithi098 ceph-mon[30092]: pgmap v393: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:03:12.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:12 smithi142 ceph-mon[34625]: pgmap v394: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:03:12.483 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-05-09T09:03:12.483 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:03:12.483 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 814, 2024-05-09T09:03:12.483 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 944, 2024-05-09T09:03:12.483 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 1860340, 2024-05-09T09:03:12.483 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 329207, 2024-05-09T09:03:12.484 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 7, 2024-05-09T09:03:12.484 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 3, 2024-05-09T09:03:12.484 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.127155849, 2024-05-09T09:03:12.484 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.043577025, 2024-05-09T09:03:12.484 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.077153151, 2024-05-09T09:03:12.484 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.013382406, 2024-05-09T09:03:12.484 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:03:12.484 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 943, 2024-05-09T09:03:12.484 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.061323115, 2024-05-09T09:03:12.484 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000065029 2024-05-09T09:03:12.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.484 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:03:12.484 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 1074, 2024-05-09T09:03:12.485 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000333015, 2024-05-09T09:03:12.485 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000310 2024-05-09T09:03:12.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.485 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 1696224, 2024-05-09T09:03:12.485 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 163152 2024-05-09T09:03:12.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.485 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:03:12.485 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 61, 2024-05-09T09:03:12.485 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 101, 2024-05-09T09:03:12.485 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 13533, 2024-05-09T09:03:12.485 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 561361, 2024-05-09T09:03:12.485 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 45, 2024-05-09T09:03:12.486 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 3, 2024-05-09T09:03:12.486 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.022817040, 2024-05-09T09:03:12.486 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.003184646, 2024-05-09T09:03:12.486 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.017498786, 2024-05-09T09:03:12.486 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.000379102, 2024-05-09T09:03:12.486 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:03:12.486 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 99, 2024-05-09T09:03:12.486 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.016441659, 2024-05-09T09:03:12.486 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000166077 2024-05-09T09:03:12.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.486 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:03:12.486 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 54, 2024-05-09T09:03:12.486 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000007801, 2024-05-09T09:03:12.487 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000144 2024-05-09T09:03:12.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.487 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 5040, 2024-05-09T09:03:12.487 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 133680 2024-05-09T09:03:12.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.487 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:03:12.487 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 22, 2024-05-09T09:03:12.487 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 175, 2024-05-09T09:03:12.487 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 4808, 2024-05-09T09:03:12.487 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 464442, 2024-05-09T09:03:12.487 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 15, 2024-05-09T09:03:12.487 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 3, 2024-05-09T09:03:12.488 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.012854638, 2024-05-09T09:03:12.488 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.006277490, 2024-05-09T09:03:12.488 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.004700784, 2024-05-09T09:03:12.488 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.000340333, 2024-05-09T09:03:12.488 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:03:12.488 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 173, 2024-05-09T09:03:12.488 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.025545664, 2024-05-09T09:03:12.488 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000147662 2024-05-09T09:03:12.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.488 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:03:12.488 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 8, 2024-05-09T09:03:12.488 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000641, 2024-05-09T09:03:12.489 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000080 2024-05-09T09:03:12.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.489 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 160, 2024-05-09T09:03:12.489 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 358064 2024-05-09T09:03:12.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.489 INFO:teuthology.orchestra.run.smithi142.stdout: "cct": { 2024-05-09T09:03:12.489 INFO:teuthology.orchestra.run.smithi142.stdout: "total_workers": 1, 2024-05-09T09:03:12.489 INFO:teuthology.orchestra.run.smithi142.stdout: "unhealthy_workers": 0 2024-05-09T09:03:12.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.489 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-MDSRank": { 2024-05-09T09:03:12.489 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2024-05-09T09:03:12.489 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2024-05-09T09:03:12.489 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 42, 2024-05-09T09:03:12.490 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.002038838, 2024-05-09T09:03:12.490 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000048543 2024-05-09T09:03:12.490 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:12.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.490 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-PurgeQueue": { 2024-05-09T09:03:12.490 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2024-05-09T09:03:12.490 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2024-05-09T09:03:12.490 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 2, 2024-05-09T09:03:12.490 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000011188, 2024-05-09T09:03:12.490 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000005594 2024-05-09T09:03:12.490 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:12.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.490 INFO:teuthology.orchestra.run.smithi142.stdout: "mds": { 2024-05-09T09:03:12.491 INFO:teuthology.orchestra.run.smithi142.stdout: "request": 0, 2024-05-09T09:03:12.491 INFO:teuthology.orchestra.run.smithi142.stdout: "reply": 0, 2024-05-09T09:03:12.491 INFO:teuthology.orchestra.run.smithi142.stdout: "reply_latency": { 2024-05-09T09:03:12.491 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.491 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.491 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.491 INFO:teuthology.orchestra.run.smithi142.stdout: "slow_reply": 0, 2024-05-09T09:03:12.491 INFO:teuthology.orchestra.run.smithi142.stdout: "forward": 0, 2024-05-09T09:03:12.491 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch_complete": 0, 2024-05-09T09:03:12.491 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch_keys": 0, 2024-05-09T09:03:12.491 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_commit": 11, 2024-05-09T09:03:12.492 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_split": 0, 2024-05-09T09:03:12.492 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_merge": 0, 2024-05-09T09:03:12.492 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes": 10, 2024-05-09T09:03:12.492 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_top": 0, 2024-05-09T09:03:12.492 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_bottom": 10, 2024-05-09T09:03:12.492 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pin_tail": 0, 2024-05-09T09:03:12.492 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pinned": 10, 2024-05-09T09:03:12.492 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_expired": 0, 2024-05-09T09:03:12.492 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_with_caps": 0, 2024-05-09T09:03:12.492 INFO:teuthology.orchestra.run.smithi142.stdout: "caps": 0, 2024-05-09T09:03:12.492 INFO:teuthology.orchestra.run.smithi142.stdout: "subtrees": 1, 2024-05-09T09:03:12.492 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse": 0, 2024-05-09T09:03:12.492 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_hit": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_forward": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_discover": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_remote_ino": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_lock": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "load_cent": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "q": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "exported": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "exported_inodes": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "imported": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "imported_inodes": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_dir_fetch": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_backtrace_fetch": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_peer_discover": 0, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rfiles": 22, 2024-05-09T09:03:12.493 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rbytes": 10738142493, 2024-05-09T09:03:12.494 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rsnaps": 6, 2024-05-09T09:03:12.494 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T09:03:12.494 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_set_tag": 0, 2024-05-09T09:03:12.494 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:03:12.494 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:03:12.494 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dir_inodes": 0, 2024-05-09T09:03:12.494 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T09:03:12.494 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T09:03:12.494 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_file_inodes": 0, 2024-05-09T09:03:12.494 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_inode_file_caps": 0, 2024-05-09T09:03:12.494 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T09:03:12.494 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_grant": 0, 2024-05-09T09:03:12.494 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:03:12.495 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:03:12.495 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:03:12.495 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_caps": 0, 2024-05-09T09:03:12.495 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_caps_dirty": 0, 2024-05-09T09:03:12.495 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_cap_release": 0, 2024-05-09T09:03:12.495 INFO:teuthology.orchestra.run.smithi142.stdout: "process_request_cap_release": 0 2024-05-09T09:03:12.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.495 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_cache": { 2024-05-09T09:03:12.495 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_update": 0, 2024-05-09T09:03:12.495 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_update_receipt": 0, 2024-05-09T09:03:12.495 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_try_discover": 0, 2024-05-09T09:03:12.495 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_send_discover": 1, 2024-05-09T09:03:12.495 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_handle_discover": 0, 2024-05-09T09:03:12.495 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_delayed": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_created": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_enqueued": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_reintegrated": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_migrated": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_processing": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_started": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_completed": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_exportdir": 0, 2024-05-09T09:03:12.496 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_flush": 0, 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragmentdir": 0, 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragstats": 0, 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_inodestats": 0 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_log": { 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "evlrg": 0, 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "evadd": 6, 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "evex": 0, 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "evtrm": 0, 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "ev": 6, 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "evexg": 0, 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "evexd": 0, 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "segadd": 2, 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "segex": 0, 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "segtrm": 0, 2024-05-09T09:03:12.497 INFO:teuthology.orchestra.run.smithi142.stdout: "seg": 2, 2024-05-09T09:03:12.498 INFO:teuthology.orchestra.run.smithi142.stdout: "segmjr": 2, 2024-05-09T09:03:12.498 INFO:teuthology.orchestra.run.smithi142.stdout: "segexg": 0, 2024-05-09T09:03:12.498 INFO:teuthology.orchestra.run.smithi142.stdout: "segexd": 0, 2024-05-09T09:03:12.498 INFO:teuthology.orchestra.run.smithi142.stdout: "expos": 4194304, 2024-05-09T09:03:12.498 INFO:teuthology.orchestra.run.smithi142.stdout: "wrpos": 4195956, 2024-05-09T09:03:12.498 INFO:teuthology.orchestra.run.smithi142.stdout: "rdpos": 0, 2024-05-09T09:03:12.498 INFO:teuthology.orchestra.run.smithi142.stdout: "jlat": { 2024-05-09T09:03:12.498 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 5, 2024-05-09T09:03:12.498 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.010987404, 2024-05-09T09:03:12.498 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.002197480 2024-05-09T09:03:12.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.498 INFO:teuthology.orchestra.run.smithi142.stdout: "replayed": 0 2024-05-09T09:03:12.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.498 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_mem": { 2024-05-09T09:03:12.499 INFO:teuthology.orchestra.run.smithi142.stdout: "ino": 13, 2024-05-09T09:03:12.499 INFO:teuthology.orchestra.run.smithi142.stdout: "ino+": 13, 2024-05-09T09:03:12.499 INFO:teuthology.orchestra.run.smithi142.stdout: "ino-": 0, 2024-05-09T09:03:12.499 INFO:teuthology.orchestra.run.smithi142.stdout: "dir": 11, 2024-05-09T09:03:12.499 INFO:teuthology.orchestra.run.smithi142.stdout: "dir+": 12, 2024-05-09T09:03:12.499 INFO:teuthology.orchestra.run.smithi142.stdout: "dir-": 1, 2024-05-09T09:03:12.499 INFO:teuthology.orchestra.run.smithi142.stdout: "dn": 10, 2024-05-09T09:03:12.499 INFO:teuthology.orchestra.run.smithi142.stdout: "dn+": 10, 2024-05-09T09:03:12.499 INFO:teuthology.orchestra.run.smithi142.stdout: "dn-": 0, 2024-05-09T09:03:12.499 INFO:teuthology.orchestra.run.smithi142.stdout: "cap": 0, 2024-05-09T09:03:12.499 INFO:teuthology.orchestra.run.smithi142.stdout: "cap+": 0, 2024-05-09T09:03:12.499 INFO:teuthology.orchestra.run.smithi142.stdout: "cap-": 0, 2024-05-09T09:03:12.499 INFO:teuthology.orchestra.run.smithi142.stdout: "rss": 54972, 2024-05-09T09:03:12.499 INFO:teuthology.orchestra.run.smithi142.stdout: "heap": 223612 2024-05-09T09:03:12.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.500 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_server": { 2024-05-09T09:03:12.500 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_client_request": 0, 2024-05-09T09:03:12.500 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_server_request": 0, 2024-05-09T09:03:12.500 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_request": 0, 2024-05-09T09:03:12.500 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_session": 8, 2024-05-09T09:03:12.500 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_peer_request": 0, 2024-05-09T09:03:12.500 INFO:teuthology.orchestra.run.smithi142.stdout: "req_create_latency": { 2024-05-09T09:03:12.500 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.500 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.500 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.500 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getattr_latency": { 2024-05-09T09:03:12.501 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.501 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.501 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.501 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getfilelock_latency": { 2024-05-09T09:03:12.501 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.501 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.501 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.501 INFO:teuthology.orchestra.run.smithi142.stdout: "req_link_latency": { 2024-05-09T09:03:12.501 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.501 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.501 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.502 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookup_latency": { 2024-05-09T09:03:12.502 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.502 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.502 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.502 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookuphash_latency": { 2024-05-09T09:03:12.502 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.502 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.502 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.502 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupino_latency": { 2024-05-09T09:03:12.502 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.503 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.503 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.503 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupname_latency": { 2024-05-09T09:03:12.503 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.503 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.503 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.503 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupparent_latency": { 2024-05-09T09:03:12.503 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.503 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.503 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.503 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupsnap_latency": { 2024-05-09T09:03:12.504 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.504 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.504 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.504 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lssnap_latency": { 2024-05-09T09:03:12.504 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.504 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.504 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.504 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mkdir_latency": { 2024-05-09T09:03:12.504 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.505 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:12 smithi105 ceph-mon[34498]: pgmap v394: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:03:12.505 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.505 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.506 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mknod_latency": { 2024-05-09T09:03:12.506 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.506 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.506 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.506 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mksnap_latency": { 2024-05-09T09:03:12.506 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.506 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.506 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.506 INFO:teuthology.orchestra.run.smithi142.stdout: "req_open_latency": { 2024-05-09T09:03:12.506 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.506 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.506 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: "req_readdir_latency": { 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rename_latency": { 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: "req_renamesnap_latency": { 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: "req_snapdiff_latency": { 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmdir_latency": { 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmsnap_latency": { 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.508 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmxattr_latency": { 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setattr_latency": { 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setdirlayout_latency": { 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.510 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setfilelock_latency": { 2024-05-09T09:03:12.510 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.510 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.510 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.510 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setlayout_latency": { 2024-05-09T09:03:12.510 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.510 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.510 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.510 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setxattr_latency": { 2024-05-09T09:03:12.510 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.511 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.511 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.511 INFO:teuthology.orchestra.run.smithi142.stdout: "req_symlink_latency": { 2024-05-09T09:03:12.511 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.511 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.511 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.511 INFO:teuthology.orchestra.run.smithi142.stdout: "req_unlink_latency": { 2024-05-09T09:03:12.511 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.511 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.511 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.511 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:03:12.512 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:03:12.512 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getvxattr_latency": { 2024-05-09T09:03:12.512 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.512 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.512 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.512 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:12.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.512 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_sessions": { 2024-05-09T09:03:12.512 INFO:teuthology.orchestra.run.smithi142.stdout: "session_count": 0, 2024-05-09T09:03:12.512 INFO:teuthology.orchestra.run.smithi142.stdout: "session_add": 2, 2024-05-09T09:03:12.512 INFO:teuthology.orchestra.run.smithi142.stdout: "session_remove": 2, 2024-05-09T09:03:12.512 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_open": 0, 2024-05-09T09:03:12.512 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_stale": 0, 2024-05-09T09:03:12.512 INFO:teuthology.orchestra.run.smithi142.stdout: "total_load": 0, 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: "average_load": 0, 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: "avg_session_uptime": 86, 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: "mdthresh_evicted": 0 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: "mempool": { 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_items": 0, 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:03:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:03:12.514 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:03:12.514 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:03:12.514 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:03:12.514 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:03:12.514 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:03:12.514 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_items": 0, 2024-05-09T09:03:12.514 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:03:12.514 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_items": 0, 2024-05-09T09:03:12.514 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:03:12.514 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:03:12.514 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:03:12.514 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:03:12.514 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:03:12.515 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:03:12.515 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:03:12.515 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_items": 0, 2024-05-09T09:03:12.515 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:03:12.515 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:03:12.515 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:03:12.515 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_items": 0, 2024-05-09T09:03:12.515 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_bytes": 0, 2024-05-09T09:03:12.515 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_items": 0, 2024-05-09T09:03:12.515 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:03:12.515 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:03:12.515 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:03:12.515 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:03:12.516 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_bytes": 237687, 2024-05-09T09:03:12.516 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_items": 69, 2024-05-09T09:03:12.516 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:03:12.516 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_items": 0, 2024-05-09T09:03:12.516 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_bytes": 0, 2024-05-09T09:03:12.516 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_items": 0, 2024-05-09T09:03:12.516 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:03:12.516 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_items": 0, 2024-05-09T09:03:12.516 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:03:12.516 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_items": 0, 2024-05-09T09:03:12.516 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_bytes": 9952, 2024-05-09T09:03:12.516 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_items": 193, 2024-05-09T09:03:12.516 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:03:12.517 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:03:12.517 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_bytes": 0, 2024-05-09T09:03:12.517 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_items": 0, 2024-05-09T09:03:12.517 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_bytes": 50664, 2024-05-09T09:03:12.517 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_items": 198, 2024-05-09T09:03:12.517 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_bytes": 0, 2024-05-09T09:03:12.517 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_items": 0, 2024-05-09T09:03:12.517 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_bytes": 0, 2024-05-09T09:03:12.517 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_items": 0 2024-05-09T09:03:12.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.517 INFO:teuthology.orchestra.run.smithi142.stdout: "objecter": { 2024-05-09T09:03:12.517 INFO:teuthology.orchestra.run.smithi142.stdout: "op_active": 0, 2024-05-09T09:03:12.517 INFO:teuthology.orchestra.run.smithi142.stdout: "op_laggy": 0, 2024-05-09T09:03:12.517 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send": 41, 2024-05-09T09:03:12.518 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send_bytes": 11954, 2024-05-09T09:03:12.518 INFO:teuthology.orchestra.run.smithi142.stdout: "op_resend": 33, 2024-05-09T09:03:12.518 INFO:teuthology.orchestra.run.smithi142.stdout: "op_reply": 41, 2024-05-09T09:03:12.518 INFO:teuthology.orchestra.run.smithi142.stdout: "op_latency": { 2024-05-09T09:03:12.518 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 41, 2024-05-09T09:03:12.518 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.198996328, 2024-05-09T09:03:12.518 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.004853568 2024-05-09T09:03:12.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.518 INFO:teuthology.orchestra.run.smithi142.stdout: "op_inflight": 0, 2024-05-09T09:03:12.518 INFO:teuthology.orchestra.run.smithi142.stdout: "oplen_avg": { 2024-05-09T09:03:12.518 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 41, 2024-05-09T09:03:12.518 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 63 2024-05-09T09:03:12.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "op": 41, 2024-05-09T09:03:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "op_r": 0, 2024-05-09T09:03:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "op_w": 41, 2024-05-09T09:03:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "op_rmw": 0, 2024-05-09T09:03:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "op_pg": 0, 2024-05-09T09:03:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_stat": 0, 2024-05-09T09:03:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_create": 10, 2024-05-09T09:03:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_read": 0, 2024-05-09T09:03:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_write": 5, 2024-05-09T09:03:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writefull": 7, 2024-05-09T09:03:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writesame": 0, 2024-05-09T09:03:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_append": 0, 2024-05-09T09:03:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_zero": 0, 2024-05-09T09:03:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_truncate": 0, 2024-05-09T09:03:12.520 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_delete": 6, 2024-05-09T09:03:12.520 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_mapext": 0, 2024-05-09T09:03:12.520 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_sparse_read": 0, 2024-05-09T09:03:12.520 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_clonerange": 0, 2024-05-09T09:03:12.520 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_getxattr": 0, 2024-05-09T09:03:12.520 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_setxattr": 20, 2024-05-09T09:03:12.520 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:03:12.520 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_rmxattr": 0, 2024-05-09T09:03:12.520 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:03:12.520 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_call": 0, 2024-05-09T09:03:12.520 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_watch": 0, 2024-05-09T09:03:12.520 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_notify": 0, 2024-05-09T09:03:12.520 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:03:12.521 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls": 0, 2024-05-09T09:03:12.521 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:03:12.521 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_other": 0, 2024-05-09T09:03:12.521 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_active": 0, 2024-05-09T09:03:12.521 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_send": 0, 2024-05-09T09:03:12.521 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_resend": 0, 2024-05-09T09:03:12.521 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_ping": 0, 2024-05-09T09:03:12.521 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_active": 0, 2024-05-09T09:03:12.521 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_send": 0, 2024-05-09T09:03:12.521 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_resend": 0, 2024-05-09T09:03:12.521 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_active": 0, 2024-05-09T09:03:12.521 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_send": 0, 2024-05-09T09:03:12.522 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_resend": 0, 2024-05-09T09:03:12.522 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_active": 0, 2024-05-09T09:03:12.522 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_send": 0, 2024-05-09T09:03:12.522 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_resend": 0, 2024-05-09T09:03:12.522 INFO:teuthology.orchestra.run.smithi142.stdout: "command_active": 0, 2024-05-09T09:03:12.522 INFO:teuthology.orchestra.run.smithi142.stdout: "command_send": 0, 2024-05-09T09:03:12.522 INFO:teuthology.orchestra.run.smithi142.stdout: "command_resend": 0, 2024-05-09T09:03:12.522 INFO:teuthology.orchestra.run.smithi142.stdout: "map_epoch": 110, 2024-05-09T09:03:12.522 INFO:teuthology.orchestra.run.smithi142.stdout: "map_full": 0, 2024-05-09T09:03:12.522 INFO:teuthology.orchestra.run.smithi142.stdout: "map_inc": 20, 2024-05-09T09:03:12.522 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_sessions": 8, 2024-05-09T09:03:12.522 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_open": 8, 2024-05-09T09:03:12.522 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_close": 0, 2024-05-09T09:03:12.522 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_laggy": 0, 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_wr": 14, 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_rd": 0, 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_del": 1 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: "oft": { 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_objs": 1, 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_updates": 0, 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_removes": 0 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: "osd-slow-ops": { 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: "slow_ops_count": 0 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: "purge_queue": { 2024-05-09T09:03:12.523 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops": 0, 2024-05-09T09:03:12.524 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T09:03:12.524 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing": 0, 2024-05-09T09:03:12.524 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_high_water": 0, 2024-05-09T09:03:12.524 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed_ops": 0, 2024-05-09T09:03:12.524 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed": 0, 2024-05-09T09:03:12.524 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_item_in_journal": 0 2024-05-09T09:03:12.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.524 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:03:12.524 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:12.524 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-05-09T09:03:12.524 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:12.524 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 893, 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 1778474, 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 893, 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 893, 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 1778474, 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_bytes": { 2024-05-09T09:03:12.525 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:12.526 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-05-09T09:03:12.526 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:12.526 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-05-09T09:03:12.526 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-05-09T09:03:12.526 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:12.526 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-05-09T09:03:12.526 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 41, 2024-05-09T09:03:12.526 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 11954, 2024-05-09T09:03:12.526 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 41, 2024-05-09T09:03:12.526 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 11954, 2024-05-09T09:03:12.526 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:12.526 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.526 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.526 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.527 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:12.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.527 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_ops": { 2024-05-09T09:03:12.527 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:12.527 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 1024, 2024-05-09T09:03:12.527 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:12.527 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-05-09T09:03:12.527 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-05-09T09:03:12.527 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:12.527 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-05-09T09:03:12.527 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 41, 2024-05-09T09:03:12.527 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 41, 2024-05-09T09:03:12.527 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 41, 2024-05-09T09:03:12.527 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 41, 2024-05-09T09:03:12.528 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:12.528 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.528 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.528 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.528 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:12.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.528 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:03:12.528 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:12.528 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2024-05-09T09:03:12.528 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:12.528 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-05-09T09:03:12.528 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-05-09T09:03:12.528 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:12.528 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 0, 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 0, 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle-0x55610436a500": { 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-05-09T09:03:12.529 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 6, 2024-05-09T09:03:12.530 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 1652, 2024-05-09T09:03:12.530 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-05-09T09:03:12.530 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 6, 2024-05-09T09:03:12.530 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-05-09T09:03:12.530 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-05-09T09:03:12.530 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 5, 2024-05-09T09:03:12.530 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 1652, 2024-05-09T09:03:12.530 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-05-09T09:03:12.530 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-05-09T09:03:12.530 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-05-09T09:03:12.530 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-05-09T09:03:12.530 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:12.530 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-05-09T09:03:12.531 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-05-09T09:03:12.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:12 smithi098 ceph-mon[30092]: pgmap v394: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:03:13.256 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-05-09T09:03:13.257 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-05-09T09:03:13.257 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=6 2024-05-09T09:03:13.257 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-05-09T09:03:13.257 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T09:03:13.257 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-09T09:03:13.273 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-09T09:03:13.273 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T09:03:13.273 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T09:03:13.304 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-09T09:03:13.304 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T09:03:13.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:03:13.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.519+0000 7f5130bfb640 1 -- 172.21.15.98:0/3190146311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f512c1021c0 msgr2=0x7f512c10e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:13.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.519+0000 7f5130bfb640 1 --2- 172.21.15.98:0/3190146311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f512c1021c0 0x7f512c10e6d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f5124009cd0 tx=0x7f512402a8e0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.519+0000 7f5130bfb640 1 -- 172.21.15.98:0/3190146311 shutdown_connections 2024-05-09T09:03:13.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.519+0000 7f5130bfb640 1 --2- 172.21.15.98:0/3190146311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f512c1021c0 0x7f512c10e6d0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.519+0000 7f5130bfb640 1 --2- 172.21.15.98:0/3190146311 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f512c101800 0x7f512c101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.519+0000 7f5130bfb640 1 --2- 172.21.15.98:0/3190146311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f512c107810 0x7f512c107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.519+0000 7f5130bfb640 1 -- 172.21.15.98:0/3190146311 >> 172.21.15.98:0/3190146311 conn(0x7f512c078bf0 msgr2=0x7f512c079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.519+0000 7f5130bfb640 1 -- 172.21.15.98:0/3190146311 shutdown_connections 2024-05-09T09:03:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.519+0000 7f5130bfb640 1 -- 172.21.15.98:0/3190146311 wait complete. 2024-05-09T09:03:13.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f5130bfb640 1 Processor -- start 2024-05-09T09:03:13.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f5130bfb640 1 -- start start 2024-05-09T09:03:13.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f5130bfb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f512c101800 0x7f512c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f5130bfb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f512c1021c0 0x7f512c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f5130bfb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f512c107810 0x7f512c19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f5130bfb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f512c111be0 con 0x7f512c101800 2024-05-09T09:03:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f5130bfb640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f512c111a60 con 0x7f512c107810 2024-05-09T09:03:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f5130bfb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f512c111d60 con 0x7f512c1021c0 2024-05-09T09:03:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f512a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f512c101800 0x7f512c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f512a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f512c101800 0x7f512c1980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54058/0 (socket says 172.21.15.98:54058) 2024-05-09T09:03:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f512a59c640 1 -- 172.21.15.98:0/1058058270 learned_addr learned my addr 172.21.15.98:0/1058058270 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:03:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f5129d9b640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f512c1021c0 0x7f512c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f512ad9d640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f512c107810 0x7f512c19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f512a59c640 1 -- 172.21.15.98:0/1058058270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f512c1021c0 msgr2=0x7f512c198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f512a59c640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f512c1021c0 0x7f512c198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f512a59c640 1 -- 172.21.15.98:0/1058058270 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f512c107810 msgr2=0x7f512c19c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f512a59c640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f512c107810 0x7f512c19c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f512a59c640 1 -- 172.21.15.98:0/1058058270 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f512c19d060 con 0x7f512c101800 2024-05-09T09:03:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.520+0000 7f5129d9b640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f512c1021c0 0x7f512c198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:03:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.521+0000 7f512a59c640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f512c101800 0x7f512c1980c0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f511c00ea60 tx=0x7f511c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.521+0000 7f51237fe640 1 -- 172.21.15.98:0/1058058270 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c00cde0 con 0x7f512c101800 2024-05-09T09:03:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.521+0000 7f51237fe640 1 -- 172.21.15.98:0/1058058270 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f511c0035f0 con 0x7f512c101800 2024-05-09T09:03:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.521+0000 7f5130bfb640 1 -- 172.21.15.98:0/1058058270 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f512c19d350 con 0x7f512c101800 2024-05-09T09:03:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.521+0000 7f51237fe640 1 -- 172.21.15.98:0/1058058270 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c010670 con 0x7f512c101800 2024-05-09T09:03:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.521+0000 7f5130bfb640 1 -- 172.21.15.98:0/1058058270 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f512c1a4be0 con 0x7f512c101800 2024-05-09T09:03:13.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.522+0000 7f5130bfb640 1 -- 172.21.15.98:0/1058058270 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f512c068aa0 con 0x7f512c101800 2024-05-09T09:03:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.523+0000 7f51237fe640 1 -- 172.21.15.98:0/1058058270 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f511c0037a0 con 0x7f512c101800 2024-05-09T09:03:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.524+0000 7f51237fe640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f51180a2d00 0x7f51180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.524+0000 7f5129d9b640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f51180a2d00 0x7f51180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.524+0000 7f51237fe640 1 -- 172.21.15.98:0/1058058270 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9650+0+0 (secure 0 0 0) 0x7f511c0cfd40 con 0x7f512c101800 2024-05-09T09:03:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.524+0000 7f5129d9b640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f51180a2d00 0x7f51180a51c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f512c105e80 tx=0x7f511400b5f0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:13.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.527+0000 7f51237fe640 1 -- 172.21.15.98:0/1058058270 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f511c09c580 con 0x7f512c101800 2024-05-09T09:03:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.707+0000 7f5130bfb640 1 -- 172.21.15.98:0/1058058270 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f512c109800 con 0x7f512c101800 2024-05-09T09:03:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.709+0000 7f51237fe640 1 -- 172.21.15.98:0/1058058270 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+25527 (secure 0 0 0) 0x7f511c09be80 con 0x7f512c101800 2024-05-09T09:03:13.709 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T09:03:13.709 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":110,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T09:03:06.075022+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":1}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1013153207":"2024-05-10T08:50:38.492493+0000","172.21.15.98:6800/1228297308":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/808614683":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/1734294041":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/2113007677":"2024-05-10T08:50:38.492493+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1228297308":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]},{"pool":4,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]},{"pool":4,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:03:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.711+0000 7f5130bfb640 1 -- 172.21.15.98:0/1058058270 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f51180a2d00 msgr2=0x7f51180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.711+0000 7f5130bfb640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f51180a2d00 0x7f51180a51c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f512c105e80 tx=0x7f511400b5f0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.711+0000 7f5130bfb640 1 -- 172.21.15.98:0/1058058270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f512c101800 msgr2=0x7f512c1980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.711+0000 7f5130bfb640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f512c101800 0x7f512c1980c0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f511c00ea60 tx=0x7f511c00ef20 comp rx=0 tx=0).stop 2024-05-09T09:03:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.711+0000 7f5130bfb640 1 -- 172.21.15.98:0/1058058270 shutdown_connections 2024-05-09T09:03:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.711+0000 7f5130bfb640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f51180a2d00 0x7f51180a51c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.711+0000 7f5130bfb640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f512c107810 0x7f512c19c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.711+0000 7f5130bfb640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f512c1021c0 0x7f512c198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.711+0000 7f5130bfb640 1 --2- 172.21.15.98:0/1058058270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f512c101800 0x7f512c1980c0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.711+0000 7f5130bfb640 1 -- 172.21.15.98:0/1058058270 >> 172.21.15.98:0/1058058270 conn(0x7f512c078bf0 msgr2=0x7f512c104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.711+0000 7f5130bfb640 1 -- 172.21.15.98:0/1058058270 shutdown_connections 2024-05-09T09:03:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.712+0000 7f5130bfb640 1 -- 172.21.15.98:0/1058058270 wait complete. 2024-05-09T09:03:13.723 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:43:39.424443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:48:15.866450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:48:17.122162+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 110, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:48:20.591475+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 110, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T09:03:13.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.829+0000 7f79c08ba640 1 -- 172.21.15.98:0/131308540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79bc1021c0 msgr2=0x7f79bc10e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.829+0000 7f79c08ba640 1 --2- 172.21.15.98:0/131308540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79bc1021c0 0x7f79bc10e6d0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009cd0 tx=0x7f79b402a8e0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.829+0000 7f79c08ba640 1 -- 172.21.15.98:0/131308540 shutdown_connections 2024-05-09T09:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.829+0000 7f79c08ba640 1 --2- 172.21.15.98:0/131308540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79bc1021c0 0x7f79bc10e6d0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.829+0000 7f79c08ba640 1 --2- 172.21.15.98:0/131308540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79bc101800 0x7f79bc101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.829+0000 7f79c08ba640 1 --2- 172.21.15.98:0/131308540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79bc107810 0x7f79bc107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.829+0000 7f79c08ba640 1 -- 172.21.15.98:0/131308540 >> 172.21.15.98:0/131308540 conn(0x7f79bc078c80 msgr2=0x7f79bc079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:13.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.829+0000 7f79c08ba640 1 -- 172.21.15.98:0/131308540 shutdown_connections 2024-05-09T09:03:13.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.829+0000 7f79c08ba640 1 -- 172.21.15.98:0/131308540 wait complete. 2024-05-09T09:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79c08ba640 1 Processor -- start 2024-05-09T09:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79c08ba640 1 -- start start 2024-05-09T09:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79c08ba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79bc101800 0x7f79bc1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79c08ba640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79bc1021c0 0x7f79bc198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79c08ba640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79bc107810 0x7f79bc19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79c08ba640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc111be0 con 0x7f79bc101800 2024-05-09T09:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79c08ba640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc111a60 con 0x7f79bc1021c0 2024-05-09T09:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79c08ba640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc111d60 con 0x7f79bc107810 2024-05-09T09:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79ba59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79bc101800 0x7f79bc1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79ba59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79bc101800 0x7f79bc1980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54076/0 (socket says 172.21.15.98:54076) 2024-05-09T09:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79ba59c640 1 -- 172.21.15.98:0/348856066 learned_addr learned my addr 172.21.15.98:0/348856066 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79b9d9b640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79bc1021c0 0x7f79bc198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79bad9d640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79bc107810 0x7f79bc19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79ba59c640 1 -- 172.21.15.98:0/348856066 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79bc107810 msgr2=0x7f79bc19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79ba59c640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79bc107810 0x7f79bc19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79ba59c640 1 -- 172.21.15.98:0/348856066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79bc1021c0 msgr2=0x7f79bc198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79ba59c640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79bc1021c0 0x7f79bc198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79ba59c640 1 -- 172.21.15.98:0/348856066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79bc19d0b0 con 0x7f79bc101800 2024-05-09T09:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79b9d9b640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79bc1021c0 0x7f79bc198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.830+0000 7f79ba59c640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79bc101800 0x7f79bc1980c0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f79ac00d950 tx=0x7f79ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.831+0000 7f79ab7fe640 1 -- 172.21.15.98:0/348856066 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ac017070 con 0x7f79bc101800 2024-05-09T09:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.831+0000 7f79ab7fe640 1 -- 172.21.15.98:0/348856066 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79ac009c30 con 0x7f79bc101800 2024-05-09T09:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.831+0000 7f79c08ba640 1 -- 172.21.15.98:0/348856066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79bc19d3a0 con 0x7f79bc101800 2024-05-09T09:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.831+0000 7f79ab7fe640 1 -- 172.21.15.98:0/348856066 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ac00f6b0 con 0x7f79bc101800 2024-05-09T09:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.831+0000 7f79c08ba640 1 -- 172.21.15.98:0/348856066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79bc1a4c30 con 0x7f79bc101800 2024-05-09T09:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.832+0000 7f79c08ba640 1 -- 172.21.15.98:0/348856066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79bc068aa0 con 0x7f79bc101800 2024-05-09T09:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.833+0000 7f79ab7fe640 1 -- 172.21.15.98:0/348856066 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f79ac0109d0 con 0x7f79bc101800 2024-05-09T09:03:13.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.833+0000 7f79ab7fe640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f79a40a2d00 0x7f79a40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:13.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.833+0000 7f79b9d9b640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f79a40a2d00 0x7f79a40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:13.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.833+0000 7f79ab7fe640 1 -- 172.21.15.98:0/348856066 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9650+0+0 (secure 0 0 0) 0x7f79ac0cf510 con 0x7f79bc101800 2024-05-09T09:03:13.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.834+0000 7f79b9d9b640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f79a40a2d00 0x7f79a40a51c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f79b00058c0 tx=0x7f79b0005850 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:13.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:13.838+0000 7f79ab7fe640 1 -- 172.21.15.98:0/348856066 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f79ac009de0 con 0x7f79bc101800 2024-05-09T09:03:14.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.015+0000 7f79c08ba640 1 -- 172.21.15.98:0/348856066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f79bc02d1c0 con 0x7f79bc101800 2024-05-09T09:03:14.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.016+0000 7f79ab7fe640 1 -- 172.21.15.98:0/348856066 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) ==== 93+0+10 (secure 0 0 0) 0x7f79ac09bcc0 con 0x7f79bc101800 2024-05-09T09:03:14.015 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-09T09:03:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.017+0000 7f79c08ba640 1 -- 172.21.15.98:0/348856066 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f79a40a2d00 msgr2=0x7f79a40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.017+0000 7f79c08ba640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f79a40a2d00 0x7f79a40a51c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f79b00058c0 tx=0x7f79b0005850 comp rx=0 tx=0).stop 2024-05-09T09:03:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.017+0000 7f79c08ba640 1 -- 172.21.15.98:0/348856066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79bc101800 msgr2=0x7f79bc1980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.017+0000 7f79c08ba640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79bc101800 0x7f79bc1980c0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f79ac00d950 tx=0x7f79ac00de10 comp rx=0 tx=0).stop 2024-05-09T09:03:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.017+0000 7f79c08ba640 1 -- 172.21.15.98:0/348856066 shutdown_connections 2024-05-09T09:03:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.017+0000 7f79c08ba640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f79a40a2d00 0x7f79a40a51c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.017+0000 7f79c08ba640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79bc107810 0x7f79bc19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.017+0000 7f79c08ba640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79bc1021c0 0x7f79bc198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.017+0000 7f79c08ba640 1 --2- 172.21.15.98:0/348856066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79bc101800 0x7f79bc1980c0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.018+0000 7f79c08ba640 1 -- 172.21.15.98:0/348856066 >> 172.21.15.98:0/348856066 conn(0x7f79bc078c80 msgr2=0x7f79bc0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.018+0000 7f79c08ba640 1 -- 172.21.15.98:0/348856066 shutdown_connections 2024-05-09T09:03:14.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.018+0000 7f79c08ba640 1 -- 172.21.15.98:0/348856066 wait complete. 2024-05-09T09:03:14.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T09:03:14.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.135+0000 7f1c36c29640 1 -- 172.21.15.98:0/680283106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c30101800 msgr2=0x7f1c30101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.135+0000 7f1c36c29640 1 --2- 172.21.15.98:0/680283106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c30101800 0x7f1c30101c80 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c009930 tx=0x7f1c1c02a420 comp rx=0 tx=0).stop 2024-05-09T09:03:14.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.135+0000 7f1c36c29640 1 -- 172.21.15.98:0/680283106 shutdown_connections 2024-05-09T09:03:14.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.135+0000 7f1c36c29640 1 --2- 172.21.15.98:0/680283106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c301021c0 0x7f1c3010e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.135+0000 7f1c36c29640 1 --2- 172.21.15.98:0/680283106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c30101800 0x7f1c30101c80 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.135+0000 7f1c36c29640 1 --2- 172.21.15.98:0/680283106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c30107810 0x7f1c30107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.135+0000 7f1c36c29640 1 -- 172.21.15.98:0/680283106 >> 172.21.15.98:0/680283106 conn(0x7f1c30078c80 msgr2=0x7f1c30079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.135+0000 7f1c36c29640 1 -- 172.21.15.98:0/680283106 shutdown_connections 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.135+0000 7f1c36c29640 1 -- 172.21.15.98:0/680283106 wait complete. 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.135+0000 7f1c36c29640 1 Processor -- start 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.135+0000 7f1c36c29640 1 -- start start 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c36c29640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c30101800 0x7f1c30198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c36c29640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c301021c0 0x7f1c301986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c36c29640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c30107810 0x7f1c3019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c36c29640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1c30111c20 con 0x7f1c30101800 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c36c29640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1c30111aa0 con 0x7f1c301021c0 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c36c29640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1c30111da0 con 0x7f1c30107810 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c349c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c30101800 0x7f1c30198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:14.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c349c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c30101800 0x7f1c30198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54094/0 (socket says 172.21.15.98:54094) 2024-05-09T09:03:14.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c349c5640 1 -- 172.21.15.98:0/4255791391 learned_addr learned my addr 172.21.15.98:0/4255791391 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:03:14.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c2ffff640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c301021c0 0x7f1c301986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:14.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c351c6640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c30107810 0x7f1c3019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:14.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c349c5640 1 -- 172.21.15.98:0/4255791391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c30107810 msgr2=0x7f1c3019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c349c5640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c30107810 0x7f1c3019ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c349c5640 1 -- 172.21.15.98:0/4255791391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c301021c0 msgr2=0x7f1c301986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c349c5640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c301021c0 0x7f1c301986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c349c5640 1 -- 172.21.15.98:0/4255791391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c3019d170 con 0x7f1c30101800 2024-05-09T09:03:14.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c2ffff640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c301021c0 0x7f1c301986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:03:14.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c351c6640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c30107810 0x7f1c3019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:03:14.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c349c5640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c30101800 0x7f1c30198180 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400e9f0 tx=0x7f1c2400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:14.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c2dffb640 1 -- 172.21.15.98:0/4255791391 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400cd80 con 0x7f1c30101800 2024-05-09T09:03:14.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c2dffb640 1 -- 172.21.15.98:0/4255791391 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c24005500 con 0x7f1c30101800 2024-05-09T09:03:14.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c36c29640 1 -- 172.21.15.98:0/4255791391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c3019d460 con 0x7f1c30101800 2024-05-09T09:03:14.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.136+0000 7f1c2dffb640 1 -- 172.21.15.98:0/4255791391 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c24010690 con 0x7f1c30101800 2024-05-09T09:03:14.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.137+0000 7f1c36c29640 1 -- 172.21.15.98:0/4255791391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c301a4cf0 con 0x7f1c30101800 2024-05-09T09:03:14.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.138+0000 7f1c36c29640 1 -- 172.21.15.98:0/4255791391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c30068aa0 con 0x7f1c30101800 2024-05-09T09:03:14.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.139+0000 7f1c2dffb640 1 -- 172.21.15.98:0/4255791391 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1c240046c0 con 0x7f1c30101800 2024-05-09T09:03:14.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.139+0000 7f1c2dffb640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1c200a2d00 0x7f1c200a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:14.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.139+0000 7f1c2ffff640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1c200a2d00 0x7f1c200a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:14.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.139+0000 7f1c2dffb640 1 -- 172.21.15.98:0/4255791391 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9650+0+0 (secure 0 0 0) 0x7f1c240cfda0 con 0x7f1c30101800 2024-05-09T09:03:14.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.140+0000 7f1c2ffff640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1c200a2d00 0x7f1c200a51c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c000c00 tx=0x7f1c1c005a20 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:14.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.144+0000 7f1c2dffb640 1 -- 172.21.15.98:0/4255791391 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1c2409c550 con 0x7f1c30101800 2024-05-09T09:03:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.320+0000 7f1c36c29640 1 -- 172.21.15.98:0/4255791391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1c3002d1c0 con 0x7f1c30101800 2024-05-09T09:03:14.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.321+0000 7f1c2dffb640 1 -- 172.21.15.98:0/4255791391 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) ==== 104+0+11 (secure 0 0 0) 0x7f1c2409be50 con 0x7f1c30101800 2024-05-09T09:03:14.320 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T09:03:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.322+0000 7f1c36c29640 1 -- 172.21.15.98:0/4255791391 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1c200a2d00 msgr2=0x7f1c200a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.322+0000 7f1c36c29640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1c200a2d00 0x7f1c200a51c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c000c00 tx=0x7f1c1c005a20 comp rx=0 tx=0).stop 2024-05-09T09:03:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.322+0000 7f1c36c29640 1 -- 172.21.15.98:0/4255791391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c30101800 msgr2=0x7f1c30198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.323+0000 7f1c36c29640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c30101800 0x7f1c30198180 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400e9f0 tx=0x7f1c2400edf0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.323+0000 7f1c36c29640 1 -- 172.21.15.98:0/4255791391 shutdown_connections 2024-05-09T09:03:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.323+0000 7f1c36c29640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f1c200a2d00 0x7f1c200a51c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.323+0000 7f1c36c29640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c30107810 0x7f1c3019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.323+0000 7f1c36c29640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c301021c0 0x7f1c301986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.323+0000 7f1c36c29640 1 --2- 172.21.15.98:0/4255791391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c30101800 0x7f1c30198180 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.323+0000 7f1c36c29640 1 -- 172.21.15.98:0/4255791391 >> 172.21.15.98:0/4255791391 conn(0x7f1c30078c80 msgr2=0x7f1c300ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.323+0000 7f1c36c29640 1 -- 172.21.15.98:0/4255791391 shutdown_connections 2024-05-09T09:03:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.323+0000 7f1c36c29640 1 -- 172.21.15.98:0/4255791391 wait complete. 2024-05-09T09:03:14.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:14 smithi098 ceph-mon[30092]: pgmap v395: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:03:14.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:14 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1058058270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:03:14.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:14 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/348856066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:03:14.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T09:03:14.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:14 smithi142 ceph-mon[34625]: pgmap v395: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:03:14.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:14 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1058058270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:03:14.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:14 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/348856066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:03:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.440+0000 7fc2d169e640 1 -- 172.21.15.98:0/4243769498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2cc100960 msgr2=0x7fc2cc106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.440+0000 7fc2d169e640 1 --2- 172.21.15.98:0/4243769498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2cc100960 0x7fc2cc106490 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009900 tx=0x7fc2b402a410 comp rx=0 tx=0).stop 2024-05-09T09:03:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.440+0000 7fc2d169e640 1 -- 172.21.15.98:0/4243769498 shutdown_connections 2024-05-09T09:03:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.440+0000 7fc2d169e640 1 --2- 172.21.15.98:0/4243769498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2cc106bc0 0x7fc2cc10e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.440+0000 7fc2d169e640 1 --2- 172.21.15.98:0/4243769498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2cc100960 0x7fc2cc106490 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.440+0000 7fc2d169e640 1 --2- 172.21.15.98:0/4243769498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2cc0fffb0 0x7fc2cc100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.440+0000 7fc2d169e640 1 -- 172.21.15.98:0/4243769498 >> 172.21.15.98:0/4243769498 conn(0x7fc2cc076cd0 msgr2=0x7fc2cc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.440+0000 7fc2d169e640 1 -- 172.21.15.98:0/4243769498 shutdown_connections 2024-05-09T09:03:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.440+0000 7fc2d169e640 1 -- 172.21.15.98:0/4243769498 wait complete. 2024-05-09T09:03:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.440+0000 7fc2d169e640 1 Processor -- start 2024-05-09T09:03:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.440+0000 7fc2d169e640 1 -- start start 2024-05-09T09:03:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.440+0000 7fc2d169e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2cc0fffb0 0x7fc2cc198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.440+0000 7fc2d169e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2cc106bc0 0x7fc2cc198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.440+0000 7fc2d169e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2cc19c9e0 0x7fc2cc19ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2d169e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2cc111c40 con 0x7fc2cc106bc0 2024-05-09T09:03:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2d169e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc2cc111ac0 con 0x7fc2cc0fffb0 2024-05-09T09:03:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2d169e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc2cc111dc0 con 0x7fc2cc19c9e0 2024-05-09T09:03:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2c3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2cc106bc0 0x7fc2cc198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2c3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2cc106bc0 0x7fc2cc198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54104/0 (socket says 172.21.15.98:54104) 2024-05-09T09:03:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2c3fff640 1 -- 172.21.15.98:0/3320194132 learned_addr learned my addr 172.21.15.98:0/3320194132 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:03:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2cb7fe640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2cc19c9e0 0x7fc2cc19ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2caffd640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2cc0fffb0 0x7fc2cc198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2c3fff640 1 -- 172.21.15.98:0/3320194132 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2cc19c9e0 msgr2=0x7fc2cc19ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2c3fff640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2cc19c9e0 0x7fc2cc19ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2c3fff640 1 -- 172.21.15.98:0/3320194132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2cc0fffb0 msgr2=0x7fc2cc198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2c3fff640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2cc0fffb0 0x7fc2cc198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2c3fff640 1 -- 172.21.15.98:0/3320194132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2cc19d4d0 con 0x7fc2cc106bc0 2024-05-09T09:03:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2caffd640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2cc0fffb0 0x7fc2cc198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:03:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2c3fff640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2cc106bc0 0x7fc2cc198660 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4006010 tx=0x7fc2b40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2c8ff9640 1 -- 172.21.15.98:0/3320194132 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b403f030 con 0x7fc2cc106bc0 2024-05-09T09:03:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2c8ff9640 1 -- 172.21.15.98:0/3320194132 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2b4003ce0 con 0x7fc2cc106bc0 2024-05-09T09:03:14.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2d169e640 1 -- 172.21.15.98:0/3320194132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2cc19d760 con 0x7fc2cc106bc0 2024-05-09T09:03:14.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2c8ff9640 1 -- 172.21.15.98:0/3320194132 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b4035070 con 0x7fc2cc106bc0 2024-05-09T09:03:14.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.441+0000 7fc2d169e640 1 -- 172.21.15.98:0/3320194132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2cc1a4f20 con 0x7fc2cc106bc0 2024-05-09T09:03:14.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.442+0000 7fc2d169e640 1 -- 172.21.15.98:0/3320194132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2cc067dc0 con 0x7fc2cc106bc0 2024-05-09T09:03:14.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.444+0000 7fc2c8ff9640 1 -- 172.21.15.98:0/3320194132 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc2b4042030 con 0x7fc2cc106bc0 2024-05-09T09:03:14.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.444+0000 7fc2c8ff9640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc2b80a2d00 0x7fc2b80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:14.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.444+0000 7fc2caffd640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc2b80a2d00 0x7fc2b80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:14.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.444+0000 7fc2c8ff9640 1 -- 172.21.15.98:0/3320194132 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9650+0+0 (secure 0 0 0) 0x7fc2b40efe40 con 0x7fc2cc106bc0 2024-05-09T09:03:14.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.445+0000 7fc2caffd640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc2b80a2d00 0x7fc2b80a51c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00baa0 tx=0x7fc2bc005d90 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:14.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.448+0000 7fc2c8ff9640 1 -- 172.21.15.98:0/3320194132 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc2b40bc5f0 con 0x7fc2cc106bc0 2024-05-09T09:03:14.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:14 smithi105 ceph-mon[34498]: pgmap v395: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-05-09T09:03:14.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:14 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1058058270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:03:14.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:14 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/348856066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:03:14.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.625+0000 7fc2d169e640 1 -- 172.21.15.98:0/3320194132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc2cc100960 con 0x7fc2cc106bc0 2024-05-09T09:03:14.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.625+0000 7fc2c8ff9640 1 -- 172.21.15.98:0/3320194132 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) ==== 100+0+11 (secure 0 0 0) 0x7fc2b40bbef0 con 0x7fc2cc106bc0 2024-05-09T09:03:14.624 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-09T09:03:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.627+0000 7fc2d169e640 1 -- 172.21.15.98:0/3320194132 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc2b80a2d00 msgr2=0x7fc2b80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.627+0000 7fc2d169e640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc2b80a2d00 0x7fc2b80a51c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00baa0 tx=0x7fc2bc005d90 comp rx=0 tx=0).stop 2024-05-09T09:03:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.627+0000 7fc2d169e640 1 -- 172.21.15.98:0/3320194132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2cc106bc0 msgr2=0x7fc2cc198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.627+0000 7fc2d169e640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2cc106bc0 0x7fc2cc198660 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4006010 tx=0x7fc2b40042f0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.627+0000 7fc2d169e640 1 -- 172.21.15.98:0/3320194132 shutdown_connections 2024-05-09T09:03:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.627+0000 7fc2d169e640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc2b80a2d00 0x7fc2b80a51c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.627+0000 7fc2d169e640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2cc19c9e0 0x7fc2cc19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.627+0000 7fc2d169e640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2cc106bc0 0x7fc2cc198660 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.627+0000 7fc2d169e640 1 --2- 172.21.15.98:0/3320194132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2cc0fffb0 0x7fc2cc198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.627+0000 7fc2d169e640 1 -- 172.21.15.98:0/3320194132 >> 172.21.15.98:0/3320194132 conn(0x7fc2cc076cd0 msgr2=0x7fc2cc104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.627+0000 7fc2d169e640 1 -- 172.21.15.98:0/3320194132 shutdown_connections 2024-05-09T09:03:14.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.627+0000 7fc2d169e640 1 -- 172.21.15.98:0/3320194132 wait complete. 2024-05-09T09:03:14.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T09:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.742+0000 7f82525b0640 1 -- 172.21.15.98:0/385806039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824c100600 msgr2=0x7f824c1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.742+0000 7f82525b0640 1 --2- 172.21.15.98:0/385806039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824c100600 0x7f824c1009e0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f823c009900 tx=0x7f823c02a410 comp rx=0 tx=0).stop 2024-05-09T09:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.742+0000 7f82525b0640 1 -- 172.21.15.98:0/385806039 shutdown_connections 2024-05-09T09:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.742+0000 7f82525b0640 1 --2- 172.21.15.98:0/385806039 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f824c10bdd0 0x7f824c10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.742+0000 7f82525b0640 1 --2- 172.21.15.98:0/385806039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f824c100fb0 0x7f824c10b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.742+0000 7f82525b0640 1 --2- 172.21.15.98:0/385806039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824c100600 0x7f824c1009e0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.742+0000 7f82525b0640 1 -- 172.21.15.98:0/385806039 >> 172.21.15.98:0/385806039 conn(0x7f824c075a30 msgr2=0x7f824c075e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.742+0000 7f82525b0640 1 -- 172.21.15.98:0/385806039 shutdown_connections 2024-05-09T09:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.742+0000 7f82525b0640 1 -- 172.21.15.98:0/385806039 wait complete. 2024-05-09T09:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.743+0000 7f82525b0640 1 Processor -- start 2024-05-09T09:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.743+0000 7f82525b0640 1 -- start start 2024-05-09T09:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.743+0000 7f82525b0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824c100600 0x7f824c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.743+0000 7f82525b0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f824c10bdd0 0x7f824c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.743+0000 7f82525b0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f824c19ca20 0x7f824c19ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.743+0000 7f82525b0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f824c111c10 con 0x7f824c100600 2024-05-09T09:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.743+0000 7f82525b0640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f824c111a90 con 0x7f824c19ca20 2024-05-09T09:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.743+0000 7f82525b0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f824c111d90 con 0x7f824c10bdd0 2024-05-09T09:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f824bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824c100600 0x7f824c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f824bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824c100600 0x7f824c1981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54130/0 (socket says 172.21.15.98:54130) 2024-05-09T09:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f824bfff640 1 -- 172.21.15.98:0/148830771 learned_addr learned my addr 172.21.15.98:0/148830771 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f8250b4d640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f824c19ca20 0x7f824c19ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f824b7fe640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f824c10bdd0 0x7f824c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f824bfff640 1 -- 172.21.15.98:0/148830771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f824c10bdd0 msgr2=0x7f824c1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f824bfff640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f824c10bdd0 0x7f824c1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f824bfff640 1 -- 172.21.15.98:0/148830771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f824c19ca20 msgr2=0x7f824c19ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f824bfff640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f824c19ca20 0x7f824c19ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f824bfff640 1 -- 172.21.15.98:0/148830771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f824c19d510 con 0x7f824c100600 2024-05-09T09:03:14.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f824b7fe640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f824c10bdd0 0x7f824c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:03:14.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f824bfff640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824c100600 0x7f824c1981b0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f823c006010 tx=0x7f823c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:14.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f82497fa640 1 -- 172.21.15.98:0/148830771 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c03f030 con 0x7f824c100600 2024-05-09T09:03:14.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f82497fa640 1 -- 172.21.15.98:0/148830771 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f823c003ce0 con 0x7f824c100600 2024-05-09T09:03:14.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f82525b0640 1 -- 172.21.15.98:0/148830771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f824c19d7a0 con 0x7f824c100600 2024-05-09T09:03:14.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f82497fa640 1 -- 172.21.15.98:0/148830771 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c035070 con 0x7f824c100600 2024-05-09T09:03:14.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.744+0000 7f82525b0640 1 -- 172.21.15.98:0/148830771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f824c1a4f60 con 0x7f824c100600 2024-05-09T09:03:14.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.745+0000 7f82525b0640 1 -- 172.21.15.98:0/148830771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f824c068aa0 con 0x7f824c100600 2024-05-09T09:03:14.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.747+0000 7f82497fa640 1 -- 172.21.15.98:0/148830771 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f823c042030 con 0x7f824c100600 2024-05-09T09:03:14.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.748+0000 7f82497fa640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f82340a2d00 0x7f82340a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:14.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.748+0000 7f824b7fe640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f82340a2d00 0x7f82340a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:14.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.748+0000 7f82497fa640 1 -- 172.21.15.98:0/148830771 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9650+0+0 (secure 0 0 0) 0x7f823c0f0220 con 0x7f824c100600 2024-05-09T09:03:14.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.748+0000 7f824b7fe640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f82340a2d00 0x7f82340a51c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f824c10db80 tx=0x7f8240005e50 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:14.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.751+0000 7f82497fa640 1 -- 172.21.15.98:0/148830771 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f823c0bca90 con 0x7f824c100600 2024-05-09T09:03:14.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.930+0000 7f82525b0640 1 -- 172.21.15.98:0/148830771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f824c100fb0 con 0x7f824c100600 2024-05-09T09:03:14.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.930+0000 7f82497fa640 1 -- 172.21.15.98:0/148830771 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v110) ==== 103+0+11 (secure 0 0 0) 0x7f823c0bc390 con 0x7f824c100600 2024-05-09T09:03:14.930 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-09T09:03:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.932+0000 7f82525b0640 1 -- 172.21.15.98:0/148830771 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f82340a2d00 msgr2=0x7f82340a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.932+0000 7f82525b0640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f82340a2d00 0x7f82340a51c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f824c10db80 tx=0x7f8240005e50 comp rx=0 tx=0).stop 2024-05-09T09:03:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.932+0000 7f82525b0640 1 -- 172.21.15.98:0/148830771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824c100600 msgr2=0x7f824c1981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.932+0000 7f82525b0640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824c100600 0x7f824c1981b0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f823c006010 tx=0x7f823c0042f0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.933+0000 7f82525b0640 1 -- 172.21.15.98:0/148830771 shutdown_connections 2024-05-09T09:03:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.933+0000 7f82525b0640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f82340a2d00 0x7f82340a51c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.933+0000 7f82525b0640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f824c19ca20 0x7f824c19ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.933+0000 7f82525b0640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f824c10bdd0 0x7f824c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.933+0000 7f82525b0640 1 --2- 172.21.15.98:0/148830771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824c100600 0x7f824c1981b0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.933+0000 7f82525b0640 1 -- 172.21.15.98:0/148830771 >> 172.21.15.98:0/148830771 conn(0x7f824c075a30 msgr2=0x7f824c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.933+0000 7f82525b0640 1 -- 172.21.15.98:0/148830771 shutdown_connections 2024-05-09T09:03:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:14.933+0000 7f82525b0640 1 -- 172.21.15.98:0/148830771 wait complete. 2024-05-09T09:03:14.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T09:03:15.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.046+0000 7fe0ce215640 1 -- 172.21.15.98:0/3185994675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0c8101800 msgr2=0x7fe0c8101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:15.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.046+0000 7fe0ce215640 1 --2- 172.21.15.98:0/3185994675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0c8101800 0x7fe0c8101c80 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc009930 tx=0x7fe0bc02a420 comp rx=0 tx=0).stop 2024-05-09T09:03:15.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.046+0000 7fe0ce215640 1 -- 172.21.15.98:0/3185994675 shutdown_connections 2024-05-09T09:03:15.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.046+0000 7fe0ce215640 1 --2- 172.21.15.98:0/3185994675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0c81021c0 0x7fe0c810e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.046+0000 7fe0ce215640 1 --2- 172.21.15.98:0/3185994675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0c8101800 0x7fe0c8101c80 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.046+0000 7fe0ce215640 1 --2- 172.21.15.98:0/3185994675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0c8107810 0x7fe0c8107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.046+0000 7fe0ce215640 1 -- 172.21.15.98:0/3185994675 >> 172.21.15.98:0/3185994675 conn(0x7fe0c8078c80 msgr2=0x7fe0c8079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:15.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.046+0000 7fe0ce215640 1 -- 172.21.15.98:0/3185994675 shutdown_connections 2024-05-09T09:03:15.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.046+0000 7fe0ce215640 1 -- 172.21.15.98:0/3185994675 wait complete. 2024-05-09T09:03:15.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.047+0000 7fe0ce215640 1 Processor -- start 2024-05-09T09:03:15.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.047+0000 7fe0ce215640 1 -- start start 2024-05-09T09:03:15.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.047+0000 7fe0ce215640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0c8101800 0x7fe0c8073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:15.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.047+0000 7fe0ce215640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0c81021c0 0x7fe0c8074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:15.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.047+0000 7fe0ce215640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0c8107810 0x7fe0c80745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:15.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.047+0000 7fe0ce215640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c8111be0 con 0x7fe0c8101800 2024-05-09T09:03:15.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.047+0000 7fe0ce215640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c8111a60 con 0x7fe0c8107810 2024-05-09T09:03:15.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.047+0000 7fe0ce215640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c8111d60 con 0x7fe0c81021c0 2024-05-09T09:03:15.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.047+0000 7fe0c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0c8101800 0x7fe0c8073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:15.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.047+0000 7fe0c77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0c8101800 0x7fe0c8073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54150/0 (socket says 172.21.15.98:54150) 2024-05-09T09:03:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.047+0000 7fe0c77fe640 1 -- 172.21.15.98:0/4064357370 learned_addr learned my addr 172.21.15.98:0/4064357370 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:03:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.047+0000 7fe0c6ffd640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0c81021c0 0x7fe0c8074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.047+0000 7fe0c7fff640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0c8107810 0x7fe0c80745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.048+0000 7fe0c77fe640 1 -- 172.21.15.98:0/4064357370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0c81021c0 msgr2=0x7fe0c8074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.048+0000 7fe0c77fe640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0c81021c0 0x7fe0c8074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.048+0000 7fe0c77fe640 1 -- 172.21.15.98:0/4064357370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0c8107810 msgr2=0x7fe0c80745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.048+0000 7fe0c77fe640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0c8107810 0x7fe0c80745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.048+0000 7fe0c77fe640 1 -- 172.21.15.98:0/4064357370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c81a65e0 con 0x7fe0c8101800 2024-05-09T09:03:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.048+0000 7fe0c6ffd640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0c81021c0 0x7fe0c8074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:03:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.048+0000 7fe0c7fff640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0c8107810 0x7fe0c80745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:03:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.048+0000 7fe0c77fe640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0c8101800 0x7fe0c8073b30 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800c920 tx=0x7fe0b800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.048+0000 7fe0c4ff9640 1 -- 172.21.15.98:0/4064357370 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8007c70 con 0x7fe0c8101800 2024-05-09T09:03:15.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.048+0000 7fe0c4ff9640 1 -- 172.21.15.98:0/4064357370 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0b8007e10 con 0x7fe0c8101800 2024-05-09T09:03:15.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.048+0000 7fe0ce215640 1 -- 172.21.15.98:0/4064357370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c81a68d0 con 0x7fe0c8101800 2024-05-09T09:03:15.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.048+0000 7fe0c4ff9640 1 -- 172.21.15.98:0/4064357370 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8018890 con 0x7fe0c8101800 2024-05-09T09:03:15.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.048+0000 7fe0ce215640 1 -- 172.21.15.98:0/4064357370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c81a6df0 con 0x7fe0c8101800 2024-05-09T09:03:15.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.049+0000 7fe0ce215640 1 -- 172.21.15.98:0/4064357370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0c8068aa0 con 0x7fe0c8101800 2024-05-09T09:03:15.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.050+0000 7fe0c4ff9640 1 -- 172.21.15.98:0/4064357370 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe0b8018a50 con 0x7fe0c8101800 2024-05-09T09:03:15.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.051+0000 7fe0c4ff9640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe0b00a2d00 0x7fe0b00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:15.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.051+0000 7fe0c6ffd640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe0b00a2d00 0x7fe0b00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:15.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.051+0000 7fe0c4ff9640 1 -- 172.21.15.98:0/4064357370 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9650+0+0 (secure 0 0 0) 0x7fe0b80cfce0 con 0x7fe0c8101800 2024-05-09T09:03:15.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.051+0000 7fe0c6ffd640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe0b00a2d00 0x7fe0b00a51c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc000c00 tx=0x7fe0bc005a20 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:15.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.055+0000 7fe0c4ff9640 1 -- 172.21.15.98:0/4064357370 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe0b8014030 con 0x7fe0c8101800 2024-05-09T09:03:15.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.248+0000 7fe0ce215640 1 -- 172.21.15.98:0/4064357370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe0c8075060 con 0x7fe0c8101800 2024-05-09T09:03:15.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.248+0000 7fe0c4ff9640 1 -- 172.21.15.98:0/4064357370 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v110) ==== 81+0+941 (secure 0 0 0) 0x7fe0c8075060 con 0x7fe0c8101800 2024-05-09T09:03:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1013153207 2024-05-10T08:50:38.492493+0000 2024-05-09T09:03:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/1228297308 2024-05-10T08:50:38.492493+0000 2024-05-09T09:03:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/808614683 2024-05-10T08:50:38.492493+0000 2024-05-09T09:03:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1734294041 2024-05-10T08:50:38.492493+0000 2024-05-09T09:03:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2163305674 2024-05-10T08:41:17.098271+0000 2024-05-09T09:03:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1385598801 2024-05-10T08:41:17.098271+0000 2024-05-09T09:03:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/498605085 2024-05-10T08:41:17.098271+0000 2024-05-09T09:03:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2113007677 2024-05-10T08:50:38.492493+0000 2024-05-09T09:03:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/1313924889 2024-05-10T08:40:50.793131+0000 2024-05-09T09:03:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/498605085 2024-05-10T08:41:17.098271+0000 2024-05-09T09:03:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/812399881 2024-05-10T08:40:50.793131+0000 2024-05-09T09:03:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2540720935 2024-05-10T08:40:50.793131+0000 2024-05-09T09:03:15.249 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/1313924889 2024-05-10T08:40:50.793131+0000 2024-05-09T09:03:15.249 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/1228297308 2024-05-10T08:50:38.492493+0000 2024-05-09T09:03:15.249 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3487867282 2024-05-10T08:41:17.098271+0000 2024-05-09T09:03:15.249 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/645511714 2024-05-10T08:40:50.793131+0000 2024-05-09T09:03:15.249 INFO:teuthology.orchestra.run.smithi098.stderr:listed 16 entries 2024-05-09T09:03:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.250+0000 7fe0ce215640 1 -- 172.21.15.98:0/4064357370 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe0b00a2d00 msgr2=0x7fe0b00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.250+0000 7fe0ce215640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe0b00a2d00 0x7fe0b00a51c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc000c00 tx=0x7fe0bc005a20 comp rx=0 tx=0).stop 2024-05-09T09:03:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.250+0000 7fe0ce215640 1 -- 172.21.15.98:0/4064357370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0c8101800 msgr2=0x7fe0c8073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.250+0000 7fe0ce215640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0c8101800 0x7fe0c8073b30 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800c920 tx=0x7fe0b800cde0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.251+0000 7fe0ce215640 1 -- 172.21.15.98:0/4064357370 shutdown_connections 2024-05-09T09:03:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.251+0000 7fe0ce215640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fe0b00a2d00 0x7fe0b00a51c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.251+0000 7fe0ce215640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0c8107810 0x7fe0c80745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.251+0000 7fe0ce215640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0c81021c0 0x7fe0c8074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.251+0000 7fe0ce215640 1 --2- 172.21.15.98:0/4064357370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0c8101800 0x7fe0c8073b30 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.251+0000 7fe0ce215640 1 -- 172.21.15.98:0/4064357370 >> 172.21.15.98:0/4064357370 conn(0x7fe0c8078c80 msgr2=0x7fe0c80ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:15.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.251+0000 7fe0ce215640 1 -- 172.21.15.98:0/4064357370 shutdown_connections 2024-05-09T09:03:15.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.251+0000 7fe0ce215640 1 -- 172.21.15.98:0/4064357370 wait complete. 2024-05-09T09:03:15.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:15 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4255791391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:03:15.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:15 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3320194132' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:03:15.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:15 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/148830771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:03:15.261 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-09T09:03:15.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:03:15.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-09T09:03:15.320 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-05-09T09:03:15.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:03:15.321 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T09:03:15.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:03:15.361 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-05-09T09:03:15.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:15 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/4255791391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:03:15.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:15 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3320194132' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:03:15.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:15 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/148830771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:03:15.438 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T09:03:15.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:03:15.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T09:03:15.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:15 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/4255791391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:03:15.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:15 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3320194132' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:03:15.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:15 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/148830771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:03:15.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:03:15.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-05-09T09:03:15.614 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-09T09:03:15.627 INFO:tasks.cephadm:Teardown begin 2024-05-09T09:03:15.627 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T09:03:15.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:03:15.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.732+0000 7f965607b640 1 -- 172.21.15.98:0/2624896195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9650106740 msgr2=0x7f9650106b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:15.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.732+0000 7f965607b640 1 --2- 172.21.15.98:0/2624896195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9650106740 0x7f9650106b20 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f9640009900 tx=0x7f964002a480 comp rx=0 tx=0).stop 2024-05-09T09:03:15.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.732+0000 7f965607b640 1 -- 172.21.15.98:0/2624896195 shutdown_connections 2024-05-09T09:03:15.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.732+0000 7f965607b640 1 --2- 172.21.15.98:0/2624896195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9650101810 0x7f965010d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.732+0000 7f965607b640 1 --2- 172.21.15.98:0/2624896195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650100e50 0x7f96501012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.732+0000 7f965607b640 1 --2- 172.21.15.98:0/2624896195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9650106740 0x7f9650106b20 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.732+0000 7f965607b640 1 -- 172.21.15.98:0/2624896195 >> 172.21.15.98:0/2624896195 conn(0x7f9650076cd0 msgr2=0x7f96500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:15.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.732+0000 7f965607b640 1 -- 172.21.15.98:0/2624896195 shutdown_connections 2024-05-09T09:03:15.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.732+0000 7f965607b640 1 -- 172.21.15.98:0/2624896195 wait complete. 2024-05-09T09:03:15.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.732+0000 7f965607b640 1 Processor -- start 2024-05-09T09:03:15.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f965607b640 1 -- start start 2024-05-09T09:03:15.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f965607b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650100e50 0x7f9650198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:15.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f965607b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9650101810 0x7f9650198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:15.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f965607b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9650106740 0x7f965019c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:15.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f965607b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9650110b60 con 0x7f9650101810 2024-05-09T09:03:15.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f965607b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f96501109e0 con 0x7f9650106740 2024-05-09T09:03:15.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f965607b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9650110ce0 con 0x7f9650100e50 2024-05-09T09:03:15.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f964effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9650101810 0x7f9650198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:15.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f964effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9650101810 0x7f9650198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35554/0 (socket says 172.21.15.98:35554) 2024-05-09T09:03:15.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f964effd640 1 -- 172.21.15.98:0/3378028843 learned_addr learned my addr 172.21.15.98:0/3378028843 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:03:15.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f964f7fe640 1 --2- 172.21.15.98:0/3378028843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650100e50 0x7f9650198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:15.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f964ffff640 1 --2- 172.21.15.98:0/3378028843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9650106740 0x7f965019c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:15.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f964effd640 1 -- 172.21.15.98:0/3378028843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650100e50 msgr2=0x7f9650198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:15.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f964effd640 1 --2- 172.21.15.98:0/3378028843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650100e50 0x7f9650198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f964effd640 1 -- 172.21.15.98:0/3378028843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9650106740 msgr2=0x7f965019c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:15.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f964effd640 1 --2- 172.21.15.98:0/3378028843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9650106740 0x7f965019c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f964effd640 1 -- 172.21.15.98:0/3378028843 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965019d0d0 con 0x7f9650101810 2024-05-09T09:03:15.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.733+0000 7f964effd640 1 --2- 172.21.15.98:0/3378028843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9650101810 0x7f9650198670 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f964400e9f0 tx=0x7f964400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:15.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.734+0000 7f964cff9640 1 -- 172.21.15.98:0/3378028843 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964400cd80 con 0x7f9650101810 2024-05-09T09:03:15.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.734+0000 7f965607b640 1 -- 172.21.15.98:0/3378028843 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965019d3c0 con 0x7f9650101810 2024-05-09T09:03:15.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.734+0000 7f965607b640 1 -- 172.21.15.98:0/3378028843 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96501a4c50 con 0x7f9650101810 2024-05-09T09:03:15.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.734+0000 7f964cff9640 1 -- 172.21.15.98:0/3378028843 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9644005500 con 0x7f9650101810 2024-05-09T09:03:15.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.734+0000 7f964cff9640 1 -- 172.21.15.98:0/3378028843 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9644010690 con 0x7f9650101810 2024-05-09T09:03:15.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.735+0000 7f965607b640 1 -- 172.21.15.98:0/3378028843 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9650067dc0 con 0x7f9650101810 2024-05-09T09:03:15.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.736+0000 7f964cff9640 1 -- 172.21.15.98:0/3378028843 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f96440046c0 con 0x7f9650101810 2024-05-09T09:03:15.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.736+0000 7f964cff9640 1 --2- 172.21.15.98:0/3378028843 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f96380a2e00 0x7f96380a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:15.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.737+0000 7f964cff9640 1 -- 172.21.15.98:0/3378028843 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9650+0+0 (secure 0 0 0) 0x7f96440cfd50 con 0x7f9650101810 2024-05-09T09:03:15.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.737+0000 7f964f7fe640 1 --2- 172.21.15.98:0/3378028843 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f96380a2e00 0x7f96380a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:15.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.737+0000 7f964f7fe640 1 --2- 172.21.15.98:0/3378028843 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f96380a2e00 0x7f96380a52c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9640000c00 tx=0x7f9640005b50 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.741+0000 7f964cff9640 1 -- 172.21.15.98:0/3378028843 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f964409c4a0 con 0x7f9650101810 2024-05-09T09:03:15.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.965+0000 7f965607b640 1 -- 172.21.15.98:0/3378028843 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96501a4f40 con 0x7f9650101810 2024-05-09T09:03:15.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.966+0000 7f964cff9640 1 -- 172.21.15.98:0/3378028843 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11941 (secure 0 0 0) 0x7f964409bda0 con 0x7f9650101810 2024-05-09T09:03:15.965 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T09:03:15.965 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":26581,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/561027115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":561027115},{"type":"v1","addr":"172.21.15.105:6835","nonce":561027115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:56:52.891273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16641":{"gid":16641,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T09:03:15.965 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 37 2024-05-09T09:03:15.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.968+0000 7f965607b640 1 -- 172.21.15.98:0/3378028843 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f96380a2e00 msgr2=0x7f96380a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:15.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.968+0000 7f965607b640 1 --2- 172.21.15.98:0/3378028843 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f96380a2e00 0x7f96380a52c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9640000c00 tx=0x7f9640005b50 comp rx=0 tx=0).stop 2024-05-09T09:03:15.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.968+0000 7f965607b640 1 -- 172.21.15.98:0/3378028843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9650101810 msgr2=0x7f9650198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:15.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.968+0000 7f965607b640 1 --2- 172.21.15.98:0/3378028843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9650101810 0x7f9650198670 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f964400e9f0 tx=0x7f964400edf0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.968+0000 7f965607b640 1 -- 172.21.15.98:0/3378028843 shutdown_connections 2024-05-09T09:03:15.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.968+0000 7f965607b640 1 --2- 172.21.15.98:0/3378028843 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f96380a2e00 0x7f96380a52c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.968+0000 7f965607b640 1 --2- 172.21.15.98:0/3378028843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9650106740 0x7f965019c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.968+0000 7f965607b640 1 --2- 172.21.15.98:0/3378028843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9650101810 0x7f9650198670 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.968+0000 7f965607b640 1 --2- 172.21.15.98:0/3378028843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650100e50 0x7f9650198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:15.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.968+0000 7f965607b640 1 -- 172.21.15.98:0/3378028843 >> 172.21.15.98:0/3378028843 conn(0x7f9650076cd0 msgr2=0x7f9650104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:15.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.968+0000 7f965607b640 1 -- 172.21.15.98:0/3378028843 shutdown_connections 2024-05-09T09:03:15.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:15.968+0000 7f965607b640 1 -- 172.21.15.98:0/3378028843 wait complete. 2024-05-09T09:03:15.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:03:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.085+0000 7f969aa2a640 1 -- 172.21.15.98:0/1335788527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694100640 msgr2=0x7f9694100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.085+0000 7f969aa2a640 1 --2- 172.21.15.98:0/1335788527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694100640 0x7f9694100a20 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f9684009900 tx=0x7f968402a410 comp rx=0 tx=0).stop 2024-05-09T09:03:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.085+0000 7f969aa2a640 1 -- 172.21.15.98:0/1335788527 shutdown_connections 2024-05-09T09:03:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.085+0000 7f969aa2a640 1 --2- 172.21.15.98:0/1335788527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f969410e1f0 0x7f96941105e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.085+0000 7f969aa2a640 1 --2- 172.21.15.98:0/1335788527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9694100f60 0x7f969410dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.085+0000 7f969aa2a640 1 --2- 172.21.15.98:0/1335788527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694100640 0x7f9694100a20 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.085+0000 7f969aa2a640 1 -- 172.21.15.98:0/1335788527 >> 172.21.15.98:0/1335788527 conn(0x7f9694077830 msgr2=0x7f9694077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.085+0000 7f969aa2a640 1 -- 172.21.15.98:0/1335788527 shutdown_connections 2024-05-09T09:03:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.085+0000 7f969aa2a640 1 -- 172.21.15.98:0/1335788527 wait complete. 2024-05-09T09:03:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.086+0000 7f969aa2a640 1 Processor -- start 2024-05-09T09:03:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.086+0000 7f969aa2a640 1 -- start start 2024-05-09T09:03:16.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.086+0000 7f969aa2a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694100640 0x7f969419a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:16.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.086+0000 7f969aa2a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9694100f60 0x7f969419a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.086+0000 7f969aa2a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f969410e1f0 0x7f969419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.086+0000 7f969aa2a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9694113e20 con 0x7f9694100640 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.086+0000 7f969aa2a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9694113ca0 con 0x7f969410e1f0 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.086+0000 7f969aa2a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9694113fa0 con 0x7f9694100f60 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.086+0000 7f9693fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694100640 0x7f969419a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.086+0000 7f9693fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694100640 0x7f969419a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35568/0 (socket says 172.21.15.98:35568) 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.086+0000 7f9693fff640 1 -- 172.21.15.98:0/1229774807 learned_addr learned my addr 172.21.15.98:0/1229774807 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.086+0000 7f9698fc7640 1 --2- 172.21.15.98:0/1229774807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f969410e1f0 0x7f969419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.086+0000 7f96937fe640 1 --2- 172.21.15.98:0/1229774807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9694100f60 0x7f969419a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.087+0000 7f9693fff640 1 -- 172.21.15.98:0/1229774807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9694100f60 msgr2=0x7f969419a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.087+0000 7f9693fff640 1 --2- 172.21.15.98:0/1229774807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9694100f60 0x7f969419a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.087+0000 7f9693fff640 1 -- 172.21.15.98:0/1229774807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f969410e1f0 msgr2=0x7f969419ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.087+0000 7f9693fff640 1 --2- 172.21.15.98:0/1229774807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f969410e1f0 0x7f969419ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.087+0000 7f9693fff640 1 -- 172.21.15.98:0/1229774807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f969419f330 con 0x7f9694100640 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.087+0000 7f9693fff640 1 --2- 172.21.15.98:0/1229774807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694100640 0x7f969419a340 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f9684006010 tx=0x7f96840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.087+0000 7f96917fa640 1 -- 172.21.15.98:0/1229774807 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968403f030 con 0x7f9694100640 2024-05-09T09:03:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.087+0000 7f96917fa640 1 -- 172.21.15.98:0/1229774807 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9684003ce0 con 0x7f9694100640 2024-05-09T09:03:16.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.087+0000 7f969aa2a640 1 -- 172.21.15.98:0/1229774807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f969419f5c0 con 0x7f9694100640 2024-05-09T09:03:16.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.087+0000 7f96917fa640 1 -- 172.21.15.98:0/1229774807 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9684035070 con 0x7f9694100640 2024-05-09T09:03:16.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.087+0000 7f969aa2a640 1 -- 172.21.15.98:0/1229774807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96941a6eb0 con 0x7f9694100640 2024-05-09T09:03:16.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.088+0000 7f969aa2a640 1 -- 172.21.15.98:0/1229774807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9694068650 con 0x7f9694100640 2024-05-09T09:03:16.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.089+0000 7f96917fa640 1 -- 172.21.15.98:0/1229774807 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9684042030 con 0x7f9694100640 2024-05-09T09:03:16.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.090+0000 7f96917fa640 1 --2- 172.21.15.98:0/1229774807 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f967c0a2d00 0x7f967c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:16.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.090+0000 7f96937fe640 1 --2- 172.21.15.98:0/1229774807 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f967c0a2d00 0x7f967c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:16.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.090+0000 7f96917fa640 1 -- 172.21.15.98:0/1229774807 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9650+0+0 (secure 0 0 0) 0x7f96840f0030 con 0x7f9694100640 2024-05-09T09:03:16.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.090+0000 7f96937fe640 1 --2- 172.21.15.98:0/1229774807 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f967c0a2d00 0x7f967c0a51c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9688005fd0 tx=0x7f9688005850 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:16.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.093+0000 7f96917fa640 1 -- 172.21.15.98:0/1229774807 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f96840bc8a0 con 0x7f9694100640 2024-05-09T09:03:16.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.316+0000 7f969aa2a640 1 -- 172.21.15.98:0/1229774807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9694109380 con 0x7f9694100640 2024-05-09T09:03:16.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.317+0000 7f96917fa640 1 -- 172.21.15.98:0/1229774807 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11941 (secure 0 0 0) 0x7f96840bc1a0 con 0x7f9694100640 2024-05-09T09:03:16.316 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T09:03:16.317 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3172071169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3172071169},{"type":"v1","addr":"172.21.15.98:6835","nonce":3172071169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":26581,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/561027115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":561027115},{"type":"v1","addr":"172.21.15.105:6835","nonce":561027115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:48:17.933763+0000","modified":"2024-05-09T08:56:52.891273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24461,"mds_1":24457,"mds_2":24439,"mds_3":24388,"mds_4":14667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3981776050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3981776050},{"type":"v1","addr":"172.21.15.98:6839","nonce":3981776050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/2714016730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2714016730},{"type":"v1","addr":"172.21.15.98:6841","nonce":2714016730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14667":{"gid":14667,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":8,"addr":"172.21.15.105:6839/1933710170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1933710170},{"type":"v1","addr":"172.21.15.105:6839","nonce":1933710170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16641":{"gid":16641,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6837/3349671892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3349671892},{"type":"v1","addr":"172.21.15.98:6837","nonce":3349671892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.142:6833/606414638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":606414638},{"type":"v1","addr":"172.21.15.142:6833","nonce":606414638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1472919356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1472919356},{"type":"v1","addr":"172.21.15.105:6833","nonce":1472919356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/2791041858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":2791041858},{"type":"v1","addr":"172.21.15.142:6835","nonce":2791041858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.142:6837/1837318616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1837318616},{"type":"v1","addr":"172.21.15.142:6837","nonce":1837318616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6839/3036634959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3036634959},{"type":"v1","addr":"172.21.15.142:6839","nonce":3036634959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6837/992088454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":992088454},{"type":"v1","addr":"172.21.15.105:6837","nonce":992088454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[14667,24439,24388,24457,24461]},"id":1}]} 2024-05-09T09:03:16.317 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 37 2024-05-09T09:03:16.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.319+0000 7f969aa2a640 1 -- 172.21.15.98:0/1229774807 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f967c0a2d00 msgr2=0x7f967c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.319+0000 7f969aa2a640 1 --2- 172.21.15.98:0/1229774807 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f967c0a2d00 0x7f967c0a51c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9688005fd0 tx=0x7f9688005850 comp rx=0 tx=0).stop 2024-05-09T09:03:16.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.319+0000 7f969aa2a640 1 -- 172.21.15.98:0/1229774807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694100640 msgr2=0x7f969419a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.319+0000 7f969aa2a640 1 --2- 172.21.15.98:0/1229774807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694100640 0x7f969419a340 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f9684006010 tx=0x7f96840042f0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.319+0000 7f969aa2a640 1 -- 172.21.15.98:0/1229774807 shutdown_connections 2024-05-09T09:03:16.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.319+0000 7f969aa2a640 1 --2- 172.21.15.98:0/1229774807 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f967c0a2d00 0x7f967c0a51c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.319+0000 7f969aa2a640 1 --2- 172.21.15.98:0/1229774807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f969410e1f0 0x7f969419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.319+0000 7f969aa2a640 1 --2- 172.21.15.98:0/1229774807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9694100f60 0x7f969419a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.319+0000 7f969aa2a640 1 --2- 172.21.15.98:0/1229774807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694100640 0x7f969419a340 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.319+0000 7f969aa2a640 1 -- 172.21.15.98:0/1229774807 >> 172.21.15.98:0/1229774807 conn(0x7f9694077830 msgr2=0x7f969410f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:16.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.319+0000 7f969aa2a640 1 -- 172.21.15.98:0/1229774807 shutdown_connections 2024-05-09T09:03:16.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.319+0000 7f969aa2a640 1 -- 172.21.15.98:0/1229774807 wait complete. 2024-05-09T09:03:16.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:03:16.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:16 smithi142 ceph-mon[34625]: pgmap v396: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-05-09T09:03:16.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:16 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/4064357370' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T09:03:16.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:16 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3378028843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:03:16.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 -- 172.21.15.98:0/2472620305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2bc10be60 msgr2=0x7ff2bc10e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 --2- 172.21.15.98:0/2472620305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2bc10be60 0x7ff2bc10e290 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009ca0 tx=0x7ff2b402a950 comp rx=0 tx=0).stop 2024-05-09T09:03:16.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 -- 172.21.15.98:0/2472620305 shutdown_connections 2024-05-09T09:03:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 --2- 172.21.15.98:0/2472620305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2bc10be60 0x7ff2bc10e290 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 --2- 172.21.15.98:0/2472620305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2bc1033b0 0x7ff2bc10b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 --2- 172.21.15.98:0/2472620305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2bc102a90 0x7ff2bc102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 -- 172.21.15.98:0/2472620305 >> 172.21.15.98:0/2472620305 conn(0x7ff2bc076ab0 msgr2=0x7ff2bc076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 -- 172.21.15.98:0/2472620305 shutdown_connections 2024-05-09T09:03:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 -- 172.21.15.98:0/2472620305 wait complete. 2024-05-09T09:03:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 Processor -- start 2024-05-09T09:03:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 -- start start 2024-05-09T09:03:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2bc102a90 0x7ff2bc19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2bc1033b0 0x7ff2bc19a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2bc10be60 0x7ff2bc19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc111bf0 con 0x7ff2bc10be60 2024-05-09T09:03:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc111a70 con 0x7ff2bc102a90 2024-05-09T09:03:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.434+0000 7ff2c3129640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc111d70 con 0x7ff2bc1033b0 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2c16c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2bc10be60 0x7ff2bc19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2c16c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2bc10be60 0x7ff2bc19eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35592/0 (socket says 172.21.15.98:35592) 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2c16c6640 1 -- 172.21.15.98:0/1581434651 learned_addr learned my addr 172.21.15.98:0/1581434651 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2bbfff640 1 --2- 172.21.15.98:0/1581434651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2bc1033b0 0x7ff2bc19a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2c0ec5640 1 --2- 172.21.15.98:0/1581434651 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2bc102a90 0x7ff2bc19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2c16c6640 1 -- 172.21.15.98:0/1581434651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2bc1033b0 msgr2=0x7ff2bc19a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2c16c6640 1 --2- 172.21.15.98:0/1581434651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2bc1033b0 0x7ff2bc19a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2c16c6640 1 -- 172.21.15.98:0/1581434651 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2bc102a90 msgr2=0x7ff2bc19a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2c16c6640 1 --2- 172.21.15.98:0/1581434651 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2bc102a90 0x7ff2bc19a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2c16c6640 1 -- 172.21.15.98:0/1581434651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2bc19f2a0 con 0x7ff2bc10be60 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2c16c6640 1 --2- 172.21.15.98:0/1581434651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2bc10be60 0x7ff2bc19eb70 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7ff2b402ae70 tx=0x7ff2b402f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2b9ffb640 1 -- 172.21.15.98:0/1581434651 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4034040 con 0x7ff2bc10be60 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2b9ffb640 1 -- 172.21.15.98:0/1581434651 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2b4030920 con 0x7ff2bc10be60 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2c3129640 1 -- 172.21.15.98:0/1581434651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2bc19f530 con 0x7ff2bc10be60 2024-05-09T09:03:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2b9ffb640 1 -- 172.21.15.98:0/1581434651 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4045480 con 0x7ff2bc10be60 2024-05-09T09:03:16.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.435+0000 7ff2c3129640 1 -- 172.21.15.98:0/1581434651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2bc1a6e20 con 0x7ff2bc10be60 2024-05-09T09:03:16.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.436+0000 7ff2c3129640 1 -- 172.21.15.98:0/1581434651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2bc06a650 con 0x7ff2bc10be60 2024-05-09T09:03:16.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.438+0000 7ff2b9ffb640 1 -- 172.21.15.98:0/1581434651 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff2b4037020 con 0x7ff2bc10be60 2024-05-09T09:03:16.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.438+0000 7ff2b9ffb640 1 --2- 172.21.15.98:0/1581434651 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff2ac0a2d00 0x7ff2ac0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:16.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.439+0000 7ff2c0ec5640 1 --2- 172.21.15.98:0/1581434651 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff2ac0a2d00 0x7ff2ac0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:16.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.439+0000 7ff2b9ffb640 1 -- 172.21.15.98:0/1581434651 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9650+0+0 (secure 0 0 0) 0x7ff2b40f0550 con 0x7ff2bc10be60 2024-05-09T09:03:16.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.439+0000 7ff2c0ec5640 1 --2- 172.21.15.98:0/1581434651 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff2ac0a2d00 0x7ff2ac0a51c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0005d00 tx=0x7ff2b000b410 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:16.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.443+0000 7ff2b9ffb640 1 -- 172.21.15.98:0/1581434651 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff2b40bceb0 con 0x7ff2bc10be60 2024-05-09T09:03:16.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:16 smithi105 ceph-mon[34498]: pgmap v396: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-05-09T09:03:16.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:16 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/4064357370' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T09:03:16.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:16 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3378028843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:03:16.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:16 smithi098 ceph-mon[30092]: pgmap v396: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-05-09T09:03:16.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:16 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/4064357370' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T09:03:16.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:16 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3378028843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:03:16.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.625+0000 7ff2c3129640 1 -- 172.21.15.98:0/1581434651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff2bc1a6ff0 con 0x7ff2bc10be60 2024-05-09T09:03:16.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.627+0000 7ff2b9ffb640 1 -- 172.21.15.98:0/1581434651 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+25527 (secure 0 0 0) 0x7ff2b40bc7b0 con 0x7ff2bc10be60 2024-05-09T09:03:16.626 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-09T09:03:16.626 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":110,"fsid":"a57d9712-0ddf-11ef-bc98-c7b262605968","created":"2024-05-09T08:40:17.456061+0000","modified":"2024-05-09T09:03:06.075022+0000","last_up_change":"2024-05-09T08:47:27.633435+0000","last_in_change":"2024-05-09T08:47:07.930998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:43:39.424443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:48:15.866450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:48:17.122162+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:48:20.591475+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9aaf45c6-e99a-4a52-88f0-b24f52befeef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6803","nonce":2307851214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6805","nonce":2307851214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6809","nonce":2307851214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2307851214},{"type":"v1","addr":"172.21.15.98:6807","nonce":2307851214}]},"public_addr":"172.21.15.98:6803/2307851214","cluster_addr":"172.21.15.98:6805/2307851214","heartbeat_back_addr":"172.21.15.98:6809/2307851214","heartbeat_front_addr":"172.21.15.98:6807/2307851214","state":["exists","up"]},{"osd":1,"uuid":"fec79393-4941-4307-8146-dfbf97c37be3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6801","nonce":2333305222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6803","nonce":2333305222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6807","nonce":2333305222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2333305222},{"type":"v1","addr":"172.21.15.105:6805","nonce":2333305222}]},"public_addr":"172.21.15.105:6801/2333305222","cluster_addr":"172.21.15.105:6803/2333305222","heartbeat_back_addr":"172.21.15.105:6807/2333305222","heartbeat_front_addr":"172.21.15.105:6805/2333305222","state":["exists","up"]},{"osd":2,"uuid":"4b659273-578f-462c-8089-3cad73e6f8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6801","nonce":1020582557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6803","nonce":1020582557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6807","nonce":1020582557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1020582557},{"type":"v1","addr":"172.21.15.142:6805","nonce":1020582557}]},"public_addr":"172.21.15.142:6801/1020582557","cluster_addr":"172.21.15.142:6803/1020582557","heartbeat_back_addr":"172.21.15.142:6807/1020582557","heartbeat_front_addr":"172.21.15.142:6805/1020582557","state":["exists","up"]},{"osd":3,"uuid":"4d229212-2643-4c3a-93c1-25adad98e37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6811","nonce":2745237287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6813","nonce":2745237287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6817","nonce":2745237287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2745237287},{"type":"v1","addr":"172.21.15.98:6815","nonce":2745237287}]},"public_addr":"172.21.15.98:6811/2745237287","cluster_addr":"172.21.15.98:6813/2745237287","heartbeat_back_addr":"172.21.15.98:6817/2745237287","heartbeat_front_addr":"172.21.15.98:6815/2745237287","state":["exists","up"]},{"osd":4,"uuid":"3cf860bd-1c29-45e5-ba31-5dd49f76913a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6809","nonce":3797460474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6811","nonce":3797460474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6815","nonce":3797460474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3797460474},{"type":"v1","addr":"172.21.15.105:6813","nonce":3797460474}]},"public_addr":"172.21.15.105:6809/3797460474","cluster_addr":"172.21.15.105:6811/3797460474","heartbeat_back_addr":"172.21.15.105:6815/3797460474","heartbeat_front_addr":"172.21.15.105:6813/3797460474","state":["exists","up"]},{"osd":5,"uuid":"9d2c5554-188a-45eb-b33b-cf929f0b40fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6809","nonce":1058034789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6811","nonce":1058034789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6815","nonce":1058034789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1058034789},{"type":"v1","addr":"172.21.15.142:6813","nonce":1058034789}]},"public_addr":"172.21.15.142:6809/1058034789","cluster_addr":"172.21.15.142:6811/1058034789","heartbeat_back_addr":"172.21.15.142:6815/1058034789","heartbeat_front_addr":"172.21.15.142:6813/1058034789","state":["exists","up"]},{"osd":6,"uuid":"2aa58eef-6903-41c6-9cb2-5b80475736ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6819","nonce":1723462836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6821","nonce":1723462836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6825","nonce":1723462836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1723462836},{"type":"v1","addr":"172.21.15.98:6823","nonce":1723462836}]},"public_addr":"172.21.15.98:6819/1723462836","cluster_addr":"172.21.15.98:6821/1723462836","heartbeat_back_addr":"172.21.15.98:6825/1723462836","heartbeat_front_addr":"172.21.15.98:6823/1723462836","state":["exists","up"]},{"osd":7,"uuid":"72436c50-ea38-4783-a40f-2312deff10e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6817","nonce":352668764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6819","nonce":352668764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6823","nonce":352668764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":352668764},{"type":"v1","addr":"172.21.15.105:6821","nonce":352668764}]},"public_addr":"172.21.15.105:6817/352668764","cluster_addr":"172.21.15.105:6819/352668764","heartbeat_back_addr":"172.21.15.105:6823/352668764","heartbeat_front_addr":"172.21.15.105:6821/352668764","state":["exists","up"]},{"osd":8,"uuid":"a0716bad-82d8-4713-b945-f640664a0783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6817","nonce":1607245901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6819","nonce":1607245901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6823","nonce":1607245901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1607245901},{"type":"v1","addr":"172.21.15.142:6821","nonce":1607245901}]},"public_addr":"172.21.15.142:6817/1607245901","cluster_addr":"172.21.15.142:6819/1607245901","heartbeat_back_addr":"172.21.15.142:6823/1607245901","heartbeat_front_addr":"172.21.15.142:6821/1607245901","state":["exists","up"]},{"osd":9,"uuid":"ae9746f9-c5b6-4574-8627-8cc50a41fa9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6827","nonce":1742411414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6829","nonce":1742411414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6833","nonce":1742411414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1742411414},{"type":"v1","addr":"172.21.15.98:6831","nonce":1742411414}]},"public_addr":"172.21.15.98:6827/1742411414","cluster_addr":"172.21.15.98:6829/1742411414","heartbeat_back_addr":"172.21.15.98:6833/1742411414","heartbeat_front_addr":"172.21.15.98:6831/1742411414","state":["exists","up"]},{"osd":10,"uuid":"1963b555-2424-4d7e-b769-95433251224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6825","nonce":1999822218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6827","nonce":1999822218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6831","nonce":1999822218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1999822218},{"type":"v1","addr":"172.21.15.105:6829","nonce":1999822218}]},"public_addr":"172.21.15.105:6825/1999822218","cluster_addr":"172.21.15.105:6827/1999822218","heartbeat_back_addr":"172.21.15.105:6831/1999822218","heartbeat_front_addr":"172.21.15.105:6829/1999822218","state":["exists","up"]},{"osd":11,"uuid":"0d685640-23a3-4c93-8a66-1db5b01c3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6825","nonce":4047948207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6827","nonce":4047948207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6831","nonce":4047948207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":4047948207},{"type":"v1","addr":"172.21.15.142:6829","nonce":4047948207}]},"public_addr":"172.21.15.142:6825/4047948207","cluster_addr":"172.21.15.142:6827/4047948207","heartbeat_back_addr":"172.21.15.142:6831/4047948207","heartbeat_front_addr":"172.21.15.142:6829/4047948207","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:42:49.804632+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:13.033122+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:43:36.504973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:01.659556+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:26.357114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:44:50.896579+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:17.328777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:45:42.640848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:07.792882+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:46:34.756926+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:00.601562+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:47:26.337305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":1}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1013153207":"2024-05-10T08:50:38.492493+0000","172.21.15.98:6800/1228297308":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/808614683":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/1734294041":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/2163305674":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/1385598801":"2024-05-10T08:41:17.098271+0000","172.21.15.98:6801/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/2113007677":"2024-05-10T08:50:38.492493+0000","172.21.15.98:6800/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6800/498605085":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/812399881":"2024-05-10T08:40:50.793131+0000","172.21.15.98:0/2540720935":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1313924889":"2024-05-10T08:40:50.793131+0000","172.21.15.98:6801/1228297308":"2024-05-10T08:50:38.492493+0000","172.21.15.98:0/3487867282":"2024-05-10T08:41:17.098271+0000","172.21.15.98:0/645511714":"2024-05-10T08:40:50.793131+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]},{"pool":4,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]},{"pool":4,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:03:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.629+0000 7ff2c3129640 1 -- 172.21.15.98:0/1581434651 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff2ac0a2d00 msgr2=0x7ff2ac0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.629+0000 7ff2c3129640 1 --2- 172.21.15.98:0/1581434651 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff2ac0a2d00 0x7ff2ac0a51c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0005d00 tx=0x7ff2b000b410 comp rx=0 tx=0).stop 2024-05-09T09:03:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.629+0000 7ff2c3129640 1 -- 172.21.15.98:0/1581434651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2bc10be60 msgr2=0x7ff2bc19eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.629+0000 7ff2c3129640 1 --2- 172.21.15.98:0/1581434651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2bc10be60 0x7ff2bc19eb70 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7ff2b402ae70 tx=0x7ff2b402f5b0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.629+0000 7ff2c3129640 1 -- 172.21.15.98:0/1581434651 shutdown_connections 2024-05-09T09:03:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.629+0000 7ff2c3129640 1 --2- 172.21.15.98:0/1581434651 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7ff2ac0a2d00 0x7ff2ac0a51c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.629+0000 7ff2c3129640 1 --2- 172.21.15.98:0/1581434651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2bc10be60 0x7ff2bc19eb70 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.629+0000 7ff2c3129640 1 --2- 172.21.15.98:0/1581434651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2bc1033b0 0x7ff2bc19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.629+0000 7ff2c3129640 1 --2- 172.21.15.98:0/1581434651 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2bc102a90 0x7ff2bc19a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.629+0000 7ff2c3129640 1 -- 172.21.15.98:0/1581434651 >> 172.21.15.98:0/1581434651 conn(0x7ff2bc076ab0 msgr2=0x7ff2bc0ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.629+0000 7ff2c3129640 1 -- 172.21.15.98:0/1581434651 shutdown_connections 2024-05-09T09:03:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.629+0000 7ff2c3129640 1 -- 172.21.15.98:0/1581434651 wait complete. 2024-05-09T09:03:16.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T09:03:16.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.744+0000 7f476c2d4640 1 -- 172.21.15.98:0/3398391476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47641010c0 msgr2=0x7f476410d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.744+0000 7f476c2d4640 1 --2- 172.21.15.98:0/3398391476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47641010c0 0x7f476410d5d0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f475c009cd0 tx=0x7f475c02a8e0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.744+0000 7f476c2d4640 1 -- 172.21.15.98:0/3398391476 shutdown_connections 2024-05-09T09:03:16.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.744+0000 7f476c2d4640 1 --2- 172.21.15.98:0/3398391476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47641010c0 0x7f476410d5d0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.744+0000 7f476c2d4640 1 --2- 172.21.15.98:0/3398391476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4764100700 0x7f4764100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.744+0000 7f476c2d4640 1 --2- 172.21.15.98:0/3398391476 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4764106710 0x7f4764106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.744+0000 7f476c2d4640 1 -- 172.21.15.98:0/3398391476 >> 172.21.15.98:0/3398391476 conn(0x7f4764077830 msgr2=0x7f4764077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:16.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.744+0000 7f476c2d4640 1 -- 172.21.15.98:0/3398391476 shutdown_connections 2024-05-09T09:03:16.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.744+0000 7f476c2d4640 1 -- 172.21.15.98:0/3398391476 wait complete. 2024-05-09T09:03:16.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.744+0000 7f476c2d4640 1 Processor -- start 2024-05-09T09:03:16.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.744+0000 7f476c2d4640 1 -- start start 2024-05-09T09:03:16.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476c2d4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4764100700 0x7f4764199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:16.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476c2d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47641010c0 0x7f4764199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:16.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476c2d4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4764106710 0x7f47641a4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:16.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476c2d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4764110b20 con 0x7f47641010c0 2024-05-09T09:03:16.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476c2d4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f47641109a0 con 0x7f4764100700 2024-05-09T09:03:16.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476c2d4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4764110ca0 con 0x7f4764106710 2024-05-09T09:03:16.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476986f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47641010c0 0x7f4764199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:16.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476a070640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4764100700 0x7f4764199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:16.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476986f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47641010c0 0x7f4764199540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35608/0 (socket says 172.21.15.98:35608) 2024-05-09T09:03:16.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476a070640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4764100700 0x7f4764199000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.98:59902/0 (socket says 172.21.15.98:59902) 2024-05-09T09:03:16.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476986f640 1 -- 172.21.15.98:0/1695026847 learned_addr learned my addr 172.21.15.98:0/1695026847 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:03:16.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476a871640 1 --2- 172.21.15.98:0/1695026847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4764106710 0x7f47641a4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:16.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476986f640 1 -- 172.21.15.98:0/1695026847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4764106710 msgr2=0x7f47641a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476986f640 1 --2- 172.21.15.98:0/1695026847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4764106710 0x7f47641a4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476986f640 1 -- 172.21.15.98:0/1695026847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4764100700 msgr2=0x7f4764199000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476986f640 1 --2- 172.21.15.98:0/1695026847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4764100700 0x7f4764199000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476986f640 1 -- 172.21.15.98:0/1695026847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47641a45c0 con 0x7f47641010c0 2024-05-09T09:03:16.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476986f640 1 --2- 172.21.15.98:0/1695026847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47641010c0 0x7f4764199540 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f475800d950 tx=0x7f475800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:16.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f47577fe640 1 -- 172.21.15.98:0/1695026847 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4758017030 con 0x7f47641010c0 2024-05-09T09:03:16.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f47577fe640 1 -- 172.21.15.98:0/1695026847 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4758004020 con 0x7f47641010c0 2024-05-09T09:03:16.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476c2d4640 1 -- 172.21.15.98:0/1695026847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47641a4790 con 0x7f47641010c0 2024-05-09T09:03:16.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f47577fe640 1 -- 172.21.15.98:0/1695026847 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47580047a0 con 0x7f47641010c0 2024-05-09T09:03:16.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.745+0000 7f476c2d4640 1 -- 172.21.15.98:0/1695026847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47641a4cb0 con 0x7f47641010c0 2024-05-09T09:03:16.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.747+0000 7f476c2d4640 1 -- 172.21.15.98:0/1695026847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4764068650 con 0x7f47641010c0 2024-05-09T09:03:16.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.748+0000 7f47577fe640 1 -- 172.21.15.98:0/1695026847 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f47580041d0 con 0x7f47641010c0 2024-05-09T09:03:16.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.748+0000 7f47577fe640 1 --2- 172.21.15.98:0/1695026847 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f47500a2d00 0x7f47500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:16.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.749+0000 7f476a070640 1 --2- 172.21.15.98:0/1695026847 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f47500a2d00 0x7f47500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:16.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.749+0000 7f47577fe640 1 -- 172.21.15.98:0/1695026847 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9650+0+0 (secure 0 0 0) 0x7f47580cf860 con 0x7f47641010c0 2024-05-09T09:03:16.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.749+0000 7f476a070640 1 --2- 172.21.15.98:0/1695026847 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f47500a2d00 0x7f47500a51c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f476000bb60 tx=0x7f476000b390 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:16.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.752+0000 7f47577fe640 1 -- 172.21.15.98:0/1695026847 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f475809c010 con 0x7f47641010c0 2024-05-09T09:03:16.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.969+0000 7f476c2d4640 1 -- 172.21.15.98:0/1695026847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4764068fb0 con 0x7f47641010c0 2024-05-09T09:03:16.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.972+0000 7f47577fe640 1 -- 172.21.15.98:0/1695026847 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v38) ==== 161+0+0 (secure 0 0 0) 0x7f475809b910 con 0x7f47641010c0 2024-05-09T09:03:16.971 INFO:teuthology.orchestra.run.smithi098.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T09:03:16.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.974+0000 7f476c2d4640 1 -- 172.21.15.98:0/1695026847 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f47500a2d00 msgr2=0x7f47500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.974+0000 7f476c2d4640 1 --2- 172.21.15.98:0/1695026847 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f47500a2d00 0x7f47500a51c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f476000bb60 tx=0x7f476000b390 comp rx=0 tx=0).stop 2024-05-09T09:03:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.974+0000 7f476c2d4640 1 -- 172.21.15.98:0/1695026847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47641010c0 msgr2=0x7f4764199540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.974+0000 7f476c2d4640 1 --2- 172.21.15.98:0/1695026847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47641010c0 0x7f4764199540 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f475800d950 tx=0x7f475800de10 comp rx=0 tx=0).stop 2024-05-09T09:03:16.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.977+0000 7f476c2d4640 1 -- 172.21.15.98:0/1695026847 shutdown_connections 2024-05-09T09:03:16.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.977+0000 7f476c2d4640 1 --2- 172.21.15.98:0/1695026847 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f47500a2d00 0x7f47500a51c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.977+0000 7f476c2d4640 1 --2- 172.21.15.98:0/1695026847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4764106710 0x7f47641a4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.977+0000 7f476c2d4640 1 --2- 172.21.15.98:0/1695026847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47641010c0 0x7f4764199540 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.977+0000 7f476c2d4640 1 --2- 172.21.15.98:0/1695026847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4764100700 0x7f4764199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:16.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.977+0000 7f476c2d4640 1 -- 172.21.15.98:0/1695026847 >> 172.21.15.98:0/1695026847 conn(0x7f4764077830 msgr2=0x7f4764102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:16.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.977+0000 7f476c2d4640 1 -- 172.21.15.98:0/1695026847 shutdown_connections 2024-05-09T09:03:16.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:16.977+0000 7f476c2d4640 1 -- 172.21.15.98:0/1695026847 wait complete. 2024-05-09T09:03:16.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T09:03:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.099+0000 7f948108c640 1 -- 172.21.15.98:0/596941934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c101120 msgr2=0x7f947c10d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.099+0000 7f948108c640 1 --2- 172.21.15.98:0/596941934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c101120 0x7f947c10d5e0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f9474009cd0 tx=0x7f947402a8e0 comp rx=0 tx=0).stop 2024-05-09T09:03:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.099+0000 7f948108c640 1 -- 172.21.15.98:0/596941934 shutdown_connections 2024-05-09T09:03:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.099+0000 7f948108c640 1 --2- 172.21.15.98:0/596941934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c101120 0x7f947c10d5e0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.099+0000 7f948108c640 1 --2- 172.21.15.98:0/596941934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f947c100760 0x7f947c100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.099+0000 7f948108c640 1 --2- 172.21.15.98:0/596941934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f947c106720 0x7f947c106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.099+0000 7f948108c640 1 -- 172.21.15.98:0/596941934 >> 172.21.15.98:0/596941934 conn(0x7f947c077890 msgr2=0x7f947c077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.099+0000 7f948108c640 1 -- 172.21.15.98:0/596941934 shutdown_connections 2024-05-09T09:03:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.099+0000 7f948108c640 1 -- 172.21.15.98:0/596941934 wait complete. 2024-05-09T09:03:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.100+0000 7f948108c640 1 Processor -- start 2024-05-09T09:03:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.100+0000 7f948108c640 1 -- start start 2024-05-09T09:03:17.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.100+0000 7f948108c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f947c100760 0x7f947c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:17.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.100+0000 7f948108c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f947c101120 0x7f947c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:17.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.100+0000 7f948108c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c106720 0x7f947c19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:17.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.100+0000 7f948108c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f947c111c80 con 0x7f947c106720 2024-05-09T09:03:17.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.100+0000 7f948108c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f947c111b00 con 0x7f947c101120 2024-05-09T09:03:17.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.100+0000 7f948108c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f947c111e00 con 0x7f947c100760 2024-05-09T09:03:17.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.100+0000 7f947b59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c106720 0x7f947c19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:17.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.100+0000 7f947b59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c106720 0x7f947c19c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35692/0 (socket says 172.21.15.98:35692) 2024-05-09T09:03:17.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.100+0000 7f947b59e640 1 -- 172.21.15.98:0/3374444923 learned_addr learned my addr 172.21.15.98:0/3374444923 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:03:17.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.101+0000 7f947ad9d640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f947c100760 0x7f947c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:17.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.101+0000 7f947a59c640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f947c101120 0x7f947c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:17.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.101+0000 7f947b59e640 1 -- 172.21.15.98:0/3374444923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f947c100760 msgr2=0x7f947c1980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:17.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.101+0000 7f947b59e640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f947c100760 0x7f947c1980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.101+0000 7f947b59e640 1 -- 172.21.15.98:0/3374444923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f947c101120 msgr2=0x7f947c1985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.101+0000 7f947b59e640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f947c101120 0x7f947c1985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.101+0000 7f947b59e640 1 -- 172.21.15.98:0/3374444923 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947c19d040 con 0x7f947c106720 2024-05-09T09:03:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.101+0000 7f947a59c640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f947c101120 0x7f947c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:03:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.101+0000 7f947b59e640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c106720 0x7f947c19c910 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f9474033040 tx=0x7f9474031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.101+0000 7f9463fff640 1 -- 172.21.15.98:0/3374444923 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9474004040 con 0x7f947c106720 2024-05-09T09:03:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.101+0000 7f9463fff640 1 -- 172.21.15.98:0/3374444923 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94740046c0 con 0x7f947c106720 2024-05-09T09:03:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.101+0000 7f948108c640 1 -- 172.21.15.98:0/3374444923 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f947c19d2d0 con 0x7f947c106720 2024-05-09T09:03:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.101+0000 7f9463fff640 1 -- 172.21.15.98:0/3374444923 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9474046480 con 0x7f947c106720 2024-05-09T09:03:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.101+0000 7f948108c640 1 -- 172.21.15.98:0/3374444923 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f947c1a4bc0 con 0x7f947c106720 2024-05-09T09:03:17.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.102+0000 7f948108c640 1 -- 172.21.15.98:0/3374444923 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f947c067dc0 con 0x7f947c106720 2024-05-09T09:03:17.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.103+0000 7f9463fff640 1 -- 172.21.15.98:0/3374444923 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f94740305a0 con 0x7f947c106720 2024-05-09T09:03:17.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.104+0000 7f9463fff640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f94640a2d00 0x7f94640a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:17.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.104+0000 7f947ad9d640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f94640a2d00 0x7f94640a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:17.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.104+0000 7f9463fff640 1 -- 172.21.15.98:0/3374444923 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9992+0+0 (secure 0 0 0) 0x7f947402e030 con 0x7f947c106720 2024-05-09T09:03:17.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.104+0000 7f947ad9d640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f94640a2d00 0x7f94640a51c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f946c00bb60 tx=0x7f946c00b390 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:17.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.108+0000 7f9463fff640 1 -- 172.21.15.98:0/3374444923 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f947404b060 con 0x7f947c106720 2024-05-09T09:03:17.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.327+0000 7f948108c640 1 -- 172.21.15.98:0/3374444923 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f947c02d1c0 con 0x7f947c106720 2024-05-09T09:03:17.338 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 09:03:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d[71020]: ignoring --setuser ceph since I am not root 2024-05-09T09:03:17.338 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 09:03:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d[71020]: ignoring --setgroup ceph since I am not root 2024-05-09T09:03:17.338 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 09:03:17 smithi098 ceph-mds[71024]: -- 172.21.15.98:0/461677240 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562409575a40 con 0x562409549000 2024-05-09T09:03:17.338 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 09:03:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d[71020]: starting mds.d at 2024-05-09T09:03:17.338 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 09:03:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-g[71625]: ignoring --setuser ceph since I am not root 2024-05-09T09:03:17.338 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 09:03:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-g[71625]: ignoring --setgroup ceph since I am not root 2024-05-09T09:03:17.338 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 09:03:17 smithi098 ceph-mds[71629]: -- 172.21.15.98:0/518134880 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5602ef3c5a40 con 0x5602ef398c00 2024-05-09T09:03:17.339 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 09:03:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-g[71625]: starting mds.g at 2024-05-09T09:03:17.339 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 09:03:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-a[71988]: ignoring --setuser ceph since I am not root 2024-05-09T09:03:17.339 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 09:03:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-a[71988]: ignoring --setgroup ceph since I am not root 2024-05-09T09:03:17.339 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 09:03:17 smithi098 ceph-mds[71992]: -- 172.21.15.98:0/4286327811 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563844767a40 con 0x56384473a800 2024-05-09T09:03:17.339 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 09:03:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-a[71988]: starting mds.a at 2024-05-09T09:03:17.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:16 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a[30088]: 2024-05-09T09:03:16.970+0000 7f871a42a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T09:03:17.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:17 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1229774807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:03:17.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:17 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1581434651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:03:17.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:17 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1695026847' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T09:03:17.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:17 smithi098 ceph-mon[30092]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-09T09:03:17.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:17 smithi098 ceph-mon[30092]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T09:03:17.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:17 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/1695026847' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-09T09:03:17.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:17 smithi098 ceph-mon[30092]: osdmap e111: 12 total, 12 up, 12 in 2024-05-09T09:03:17.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:17 smithi098 ceph-mon[30092]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-05-09T09:03:17.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:17 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1229774807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:03:17.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:17 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1581434651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:03:17.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:17 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1695026847' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T09:03:17.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:17 smithi142 ceph-mon[34625]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-09T09:03:17.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:17 smithi142 ceph-mon[34625]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T09:03:17.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:17 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/1695026847' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-09T09:03:17.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:17 smithi142 ceph-mon[34625]: osdmap e111: 12 total, 12 up, 12 in 2024-05-09T09:03:17.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:17 smithi142 ceph-mon[34625]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-05-09T09:03:17.409 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 09:03:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-c[64588]: ignoring --setuser ceph since I am not root 2024-05-09T09:03:17.409 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 09:03:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-c[64588]: ignoring --setgroup ceph since I am not root 2024-05-09T09:03:17.409 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 09:03:17 smithi142 ceph-mds[64597]: -- 172.21.15.142:0/3802335427 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b956909a40 con 0x55b9568dd000 2024-05-09T09:03:17.409 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 09:03:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-c[64588]: starting mds.c at 2024-05-09T09:03:17.410 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 09:03:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-f[64279]: ignoring --setuser ceph since I am not root 2024-05-09T09:03:17.410 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 09:03:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-f[64279]: ignoring --setgroup ceph since I am not root 2024-05-09T09:03:17.410 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 09:03:17 smithi142 ceph-mds[64283]: -- 172.21.15.142:0/931144282 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563a9b067a40 con 0x563a9b03ac00 2024-05-09T09:03:17.410 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 09:03:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-f[64279]: starting mds.f at 2024-05-09T09:03:17.410 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 09:03:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-i[64906]: ignoring --setuser ceph since I am not root 2024-05-09T09:03:17.410 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 09:03:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-i[64906]: ignoring --setgroup ceph since I am not root 2024-05-09T09:03:17.410 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 09:03:17 smithi142 ceph-mds[64910]: -- 172.21.15.142:0/3901674489 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e4ee677a40 con 0x55e4ee64ac00 2024-05-09T09:03:17.411 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 09:03:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-i[64906]: starting mds.i at 2024-05-09T09:03:17.411 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 09:03:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-l[63968]: ignoring --setuser ceph since I am not root 2024-05-09T09:03:17.411 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 09:03:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-l[63968]: ignoring --setgroup ceph since I am not root 2024-05-09T09:03:17.411 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 09:03:17 smithi142 ceph-mds[63972]: -- 172.21.15.142:0/3128188096 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560ffcaa5a40 con 0x560ffca79000 2024-05-09T09:03:17.411 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 09:03:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-l[63968]: starting mds.l at 2024-05-09T09:03:17.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:17 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1229774807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:03:17.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:17 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1581434651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:03:17.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:17 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1695026847' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T09:03:17.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:17 smithi105 ceph-mon[34498]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-09T09:03:17.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:17 smithi105 ceph-mon[34498]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T09:03:17.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:17 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/1695026847' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-09T09:03:17.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:17 smithi105 ceph-mon[34498]: osdmap e111: 12 total, 12 up, 12 in 2024-05-09T09:03:17.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:17 smithi105 ceph-mon[34498]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-05-09T09:03:17.487 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 09:03:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-b[64843]: ignoring --setuser ceph since I am not root 2024-05-09T09:03:17.487 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 09:03:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-b[64843]: ignoring --setgroup ceph since I am not root 2024-05-09T09:03:17.488 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 09:03:17 smithi105 ceph-mds[64847]: -- 172.21.15.105:0/282511785 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5609009eda40 con 0x5609009c0800 2024-05-09T09:03:17.488 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 09:03:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-b[64843]: starting mds.b at 2024-05-09T09:03:17.488 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 09:03:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-k[63896]: ignoring --setuser ceph since I am not root 2024-05-09T09:03:17.488 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 09:03:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-k[63896]: ignoring --setgroup ceph since I am not root 2024-05-09T09:03:17.488 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 09:03:17 smithi105 ceph-mds[63900]: -- 172.21.15.105:0/2739449356 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d15b6a3a40 con 0x55d15b676800 2024-05-09T09:03:17.488 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 09:03:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-k[63896]: starting mds.k at 2024-05-09T09:03:17.488 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 09:03:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-k[63896]: 2024-05-09T09:03:17.061+0000 7f9363eba700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6833/0: (98) Address already in use 2024-05-09T09:03:17.488 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 09:03:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h[64536]: ignoring --setuser ceph since I am not root 2024-05-09T09:03:17.489 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 09:03:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h[64536]: ignoring --setgroup ceph since I am not root 2024-05-09T09:03:17.489 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 09:03:17 smithi105 ceph-mds[64540]: -- 172.21.15.105:0/235446863 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e74aceba40 con 0x55e74acbec00 2024-05-09T09:03:17.489 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 09:03:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h[64536]: starting mds.h at 2024-05-09T09:03:17.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.974+0000 7f9463fff640 1 -- 172.21.15.98:0/3374444923 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v39) ==== 104+0+0 (secure 0 0 0) 0x7f94740bd500 con 0x7f947c106720 2024-05-09T09:03:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.976+0000 7f948108c640 1 -- 172.21.15.98:0/3374444923 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f94640a2d00 msgr2=0x7f94640a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.976+0000 7f948108c640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f94640a2d00 0x7f94640a51c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f946c00bb60 tx=0x7f946c00b390 comp rx=0 tx=0).stop 2024-05-09T09:03:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.976+0000 7f948108c640 1 -- 172.21.15.98:0/3374444923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c106720 msgr2=0x7f947c19c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.976+0000 7f948108c640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c106720 0x7f947c19c910 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f9474033040 tx=0x7f9474031e60 comp rx=0 tx=0).stop 2024-05-09T09:03:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.976+0000 7f948108c640 1 -- 172.21.15.98:0/3374444923 shutdown_connections 2024-05-09T09:03:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.976+0000 7f948108c640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f94640a2d00 0x7f94640a51c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.976+0000 7f948108c640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c106720 0x7f947c19c910 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.976+0000 7f948108c640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f947c101120 0x7f947c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.976+0000 7f948108c640 1 --2- 172.21.15.98:0/3374444923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f947c100760 0x7f947c1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.976+0000 7f948108c640 1 -- 172.21.15.98:0/3374444923 >> 172.21.15.98:0/3374444923 conn(0x7f947c077890 msgr2=0x7f947c103280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.977+0000 7f948108c640 1 -- 172.21.15.98:0/3374444923 shutdown_connections 2024-05-09T09:03:17.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:17.977+0000 7f948108c640 1 -- 172.21.15.98:0/3374444923 wait complete. 2024-05-09T09:03:17.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T09:03:18.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.104+0000 7fca0b15a640 1 -- 172.21.15.98:0/2186578226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca0406a870 msgr2=0x7fca0410e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:18.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.104+0000 7fca0b15a640 1 --2- 172.21.15.98:0/2186578226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca0406a870 0x7fca0410e6e0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc009cd0 tx=0x7fc9fc02a8e0 comp rx=0 tx=0).stop 2024-05-09T09:03:18.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.104+0000 7fca0b15a640 1 -- 172.21.15.98:0/2186578226 shutdown_connections 2024-05-09T09:03:18.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.104+0000 7fca0b15a640 1 --2- 172.21.15.98:0/2186578226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca0406a870 0x7fca0410e6e0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:18.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.104+0000 7fca0b15a640 1 --2- 172.21.15.98:0/2186578226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca04069eb0 0x7fca0406a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:18.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.104+0000 7fca0b15a640 1 --2- 172.21.15.98:0/2186578226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca04107820 0x7fca04107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:18.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.104+0000 7fca0b15a640 1 -- 172.21.15.98:0/2186578226 >> 172.21.15.98:0/2186578226 conn(0x7fca04077790 msgr2=0x7fca04077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:18.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.104+0000 7fca0b15a640 1 -- 172.21.15.98:0/2186578226 shutdown_connections 2024-05-09T09:03:18.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.104+0000 7fca0b15a640 1 -- 172.21.15.98:0/2186578226 wait complete. 2024-05-09T09:03:18.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fca0b15a640 1 Processor -- start 2024-05-09T09:03:18.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fca0b15a640 1 -- start start 2024-05-09T09:03:18.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fca0b15a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca04069eb0 0x7fca04198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:18.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fca0b15a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca0406a870 0x7fca041985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:18.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fca0b15a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca04107820 0x7fca0419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fca0b15a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca04111bf0 con 0x7fca0406a870 2024-05-09T09:03:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fca0b15a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fca04111a70 con 0x7fca04107820 2024-05-09T09:03:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fca0b15a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fca04111d70 con 0x7fca04069eb0 2024-05-09T09:03:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fc9fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca0406a870 0x7fca041985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fc9fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca0406a870 0x7fca041985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35720/0 (socket says 172.21.15.98:35720) 2024-05-09T09:03:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fc9fbfff640 1 -- 172.21.15.98:0/918569482 learned_addr learned my addr 172.21.15.98:0/918569482 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:03:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fca096f7640 1 --2- 172.21.15.98:0/918569482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca04107820 0x7fca0419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fca08ef6640 1 --2- 172.21.15.98:0/918569482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca04069eb0 0x7fca04198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fc9fbfff640 1 -- 172.21.15.98:0/918569482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca04069eb0 msgr2=0x7fca04198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fc9fbfff640 1 --2- 172.21.15.98:0/918569482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca04069eb0 0x7fca04198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fc9fbfff640 1 -- 172.21.15.98:0/918569482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca04107820 msgr2=0x7fca0419c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fc9fbfff640 1 --2- 172.21.15.98:0/918569482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca04107820 0x7fca0419c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fc9fbfff640 1 -- 172.21.15.98:0/918569482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0419d080 con 0x7fca0406a870 2024-05-09T09:03:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.105+0000 7fc9fbfff640 1 --2- 172.21.15.98:0/918569482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca0406a870 0x7fca041985d0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400d950 tx=0x7fc9f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.106+0000 7fc9f9ffb640 1 -- 172.21.15.98:0/918569482 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4017030 con 0x7fca0406a870 2024-05-09T09:03:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.106+0000 7fca0b15a640 1 -- 172.21.15.98:0/918569482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca0419d370 con 0x7fca0406a870 2024-05-09T09:03:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.106+0000 7fc9f9ffb640 1 -- 172.21.15.98:0/918569482 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9f4004020 con 0x7fca0406a870 2024-05-09T09:03:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.106+0000 7fc9f9ffb640 1 -- 172.21.15.98:0/918569482 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f40047a0 con 0x7fca0406a870 2024-05-09T09:03:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.106+0000 7fca0b15a640 1 -- 172.21.15.98:0/918569482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca041a4c00 con 0x7fca0406a870 2024-05-09T09:03:18.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.107+0000 7fca0b15a640 1 -- 172.21.15.98:0/918569482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca04068aa0 con 0x7fca0406a870 2024-05-09T09:03:18.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.108+0000 7fc9f9ffb640 1 -- 172.21.15.98:0/918569482 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc9f40041d0 con 0x7fca0406a870 2024-05-09T09:03:18.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.108+0000 7fc9f9ffb640 1 --2- 172.21.15.98:0/918569482 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc9ec0a2d00 0x7fc9ec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:18.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.109+0000 7fca08ef6640 1 --2- 172.21.15.98:0/918569482 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc9ec0a2d00 0x7fc9ec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:18.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.109+0000 7fc9f9ffb640 1 -- 172.21.15.98:0/918569482 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9992+0+0 (secure 0 0 0) 0x7fc9f40cf3f0 con 0x7fca0406a870 2024-05-09T09:03:18.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.109+0000 7fca08ef6640 1 --2- 172.21.15.98:0/918569482 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc9ec0a2d00 0x7fc9ec0a51c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fca0000cac0 tx=0x7fca0000c4a0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:18.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.113+0000 7fc9f9ffb640 1 -- 172.21.15.98:0/918569482 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc9f409ba50 con 0x7fca0406a870 2024-05-09T09:03:18.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:18.299+0000 7fca0b15a640 1 -- 172.21.15.98:0/918569482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fca04109dd0 con 0x7fca0406a870 2024-05-09T09:03:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: pgmap v397: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 189 B/s wr, 9 op/s 2024-05-09T09:03:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3374444923' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T09:03:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-09T09:03:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-09T09:03:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: Cluster is now healthy 2024-05-09T09:03:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3374444923' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-09T09:03:18.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.142:6832/2314653247,v1:172.21.15.142:6833/2314653247] up:boot 2024-05-09T09:03:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.142:6834/1978767939,v1:172.21.15.142:6835/1978767939] up:boot 2024-05-09T09:03:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.142:6836/3924214332,v1:172.21.15.142:6837/3924214332] up:boot 2024-05-09T09:03:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.142:6838/3550943432,v1:172.21.15.142:6839/3550943432] up:boot 2024-05-09T09:03:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.98:6836/218697956,v1:172.21.15.98:6837/218697956] up:boot 2024-05-09T09:03:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.105:6832/2718100564,v1:172.21.15.105:6837/2718100564] up:boot 2024-05-09T09:03:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.105:6833/3234002513,v1:172.21.15.105:6838/3234002513] up:boot 2024-05-09T09:03:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.105:6836/3792768876,v1:172.21.15.105:6839/3792768876] up:boot 2024-05-09T09:03:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.98:6838/632801433,v1:172.21.15.98:6840/632801433] up:boot 2024-05-09T09:03:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: mds.? [v2:172.21.15.98:6839/4104096566,v1:172.21.15.98:6841/4104096566] up:boot 2024-05-09T09:03:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: fsmap 12 up:standby 2024-05-09T09:03:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T09:03:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T09:03:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:03:18.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T09:03:18.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T09:03:18.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T09:03:18.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T09:03:18.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T09:03:18.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T09:03:18.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T09:03:18.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T09:03:18.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:18 smithi098 ceph-mon[30092]: osdmap e112: 12 total, 12 up, 12 in 2024-05-09T09:03:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: pgmap v397: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 189 B/s wr, 9 op/s 2024-05-09T09:03:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3374444923' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T09:03:18.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: Cluster is now healthy 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3374444923' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.142:6832/2314653247,v1:172.21.15.142:6833/2314653247] up:boot 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.142:6834/1978767939,v1:172.21.15.142:6835/1978767939] up:boot 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.142:6836/3924214332,v1:172.21.15.142:6837/3924214332] up:boot 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.142:6838/3550943432,v1:172.21.15.142:6839/3550943432] up:boot 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.98:6836/218697956,v1:172.21.15.98:6837/218697956] up:boot 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.105:6832/2718100564,v1:172.21.15.105:6837/2718100564] up:boot 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.105:6833/3234002513,v1:172.21.15.105:6838/3234002513] up:boot 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.105:6836/3792768876,v1:172.21.15.105:6839/3792768876] up:boot 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.98:6838/632801433,v1:172.21.15.98:6840/632801433] up:boot 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: mds.? [v2:172.21.15.98:6839/4104096566,v1:172.21.15.98:6841/4104096566] up:boot 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: fsmap 12 up:standby 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T09:03:18.409 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T09:03:18.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:03:18.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T09:03:18.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T09:03:18.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T09:03:18.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T09:03:18.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T09:03:18.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T09:03:18.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T09:03:18.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T09:03:18.410 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:18 smithi142 ceph-mon[34625]: osdmap e112: 12 total, 12 up, 12 in 2024-05-09T09:03:18.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: pgmap v397: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 189 B/s wr, 9 op/s 2024-05-09T09:03:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3374444923' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T09:03:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-09T09:03:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-09T09:03:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: Cluster is now healthy 2024-05-09T09:03:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3374444923' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-09T09:03:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.142:6832/2314653247,v1:172.21.15.142:6833/2314653247] up:boot 2024-05-09T09:03:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.142:6834/1978767939,v1:172.21.15.142:6835/1978767939] up:boot 2024-05-09T09:03:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.142:6836/3924214332,v1:172.21.15.142:6837/3924214332] up:boot 2024-05-09T09:03:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.142:6838/3550943432,v1:172.21.15.142:6839/3550943432] up:boot 2024-05-09T09:03:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.98:6836/218697956,v1:172.21.15.98:6837/218697956] up:boot 2024-05-09T09:03:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.105:6832/2718100564,v1:172.21.15.105:6837/2718100564] up:boot 2024-05-09T09:03:18.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.105:6833/3234002513,v1:172.21.15.105:6838/3234002513] up:boot 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.105:6836/3792768876,v1:172.21.15.105:6839/3792768876] up:boot 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.98:6838/632801433,v1:172.21.15.98:6840/632801433] up:boot 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: mds.? [v2:172.21.15.98:6839/4104096566,v1:172.21.15.98:6841/4104096566] up:boot 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: fsmap 12 up:standby 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T09:03:18.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: from='mgr.15555 172.21.15.98:0/56705201' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T09:03:18.489 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:18 smithi105 ceph-mon[34498]: osdmap e112: 12 total, 12 up, 12 in 2024-05-09T09:03:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.129+0000 7fc9f9ffb640 1 -- 172.21.15.98:0/918569482 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v113) ==== 181+0+0 (secure 0 0 0) 0x7fc9f409b350 con 0x7fca0406a870 2024-05-09T09:03:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' removed 2024-05-09T09:03:19.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.133+0000 7fca0b15a640 1 -- 172.21.15.98:0/918569482 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc9ec0a2d00 msgr2=0x7fc9ec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:19.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.133+0000 7fca0b15a640 1 --2- 172.21.15.98:0/918569482 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc9ec0a2d00 0x7fc9ec0a51c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fca0000cac0 tx=0x7fca0000c4a0 comp rx=0 tx=0).stop 2024-05-09T09:03:19.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.133+0000 7fca0b15a640 1 -- 172.21.15.98:0/918569482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca0406a870 msgr2=0x7fca041985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:19.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.133+0000 7fca0b15a640 1 --2- 172.21.15.98:0/918569482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca0406a870 0x7fca041985d0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400d950 tx=0x7fc9f400de10 comp rx=0 tx=0).stop 2024-05-09T09:03:19.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.133+0000 7fca0b15a640 1 -- 172.21.15.98:0/918569482 shutdown_connections 2024-05-09T09:03:19.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.133+0000 7fca0b15a640 1 --2- 172.21.15.98:0/918569482 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7fc9ec0a2d00 0x7fc9ec0a51c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.133+0000 7fca0b15a640 1 --2- 172.21.15.98:0/918569482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca04107820 0x7fca0419c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.133+0000 7fca0b15a640 1 --2- 172.21.15.98:0/918569482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca0406a870 0x7fca041985d0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.133+0000 7fca0b15a640 1 --2- 172.21.15.98:0/918569482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca04069eb0 0x7fca04198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.133+0000 7fca0b15a640 1 -- 172.21.15.98:0/918569482 >> 172.21.15.98:0/918569482 conn(0x7fca04077790 msgr2=0x7fca040fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.133+0000 7fca0b15a640 1 -- 172.21.15.98:0/918569482 shutdown_connections 2024-05-09T09:03:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.134+0000 7fca0b15a640 1 -- 172.21.15.98:0/918569482 wait complete. 2024-05-09T09:03:19.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T09:03:19.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.260+0000 7f549f93b640 1 -- 172.21.15.98:0/1726408953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5498107820 msgr2=0x7f5498107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:19.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.260+0000 7f549f93b640 1 --2- 172.21.15.98:0/1726408953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5498107820 0x7f5498107c00 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f5494009930 tx=0x7f549402a420 comp rx=0 tx=0).stop 2024-05-09T09:03:19.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.260+0000 7f549f93b640 1 -- 172.21.15.98:0/1726408953 shutdown_connections 2024-05-09T09:03:19.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.260+0000 7f549f93b640 1 --2- 172.21.15.98:0/1726408953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f549806a870 0x7f549810e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:19.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.260+0000 7f549f93b640 1 --2- 172.21.15.98:0/1726408953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5498069eb0 0x7f549806a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:19.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.260+0000 7f549f93b640 1 --2- 172.21.15.98:0/1726408953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5498107820 0x7f5498107c00 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:19.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.260+0000 7f549f93b640 1 -- 172.21.15.98:0/1726408953 >> 172.21.15.98:0/1726408953 conn(0x7f5498077790 msgr2=0x7f5498077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:19.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.260+0000 7f549f93b640 1 -- 172.21.15.98:0/1726408953 shutdown_connections 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.260+0000 7f549f93b640 1 -- 172.21.15.98:0/1726408953 wait complete. 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.261+0000 7f549f93b640 1 Processor -- start 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.261+0000 7f549f93b640 1 -- start start 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.261+0000 7f549f93b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5498069eb0 0x7f5498198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.261+0000 7f549f93b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f549806a870 0x7f54981985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.261+0000 7f549f93b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5498107820 0x7f549819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.261+0000 7f549f93b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5498111bf0 con 0x7f5498107820 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.261+0000 7f549f93b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5498111a70 con 0x7f549806a870 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.261+0000 7f549f93b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5498111d70 con 0x7f5498069eb0 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.261+0000 7f549ded8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5498107820 0x7f549819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.261+0000 7f549ded8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5498107820 0x7f549819c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35748/0 (socket says 172.21.15.98:35748) 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.261+0000 7f549ded8640 1 -- 172.21.15.98:0/3874221213 learned_addr learned my addr 172.21.15.98:0/3874221213 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.261+0000 7f549ced6640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f549806a870 0x7f54981985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.261+0000 7f549d6d7640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5498069eb0 0x7f5498198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:19.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.262+0000 7f549ded8640 1 -- 172.21.15.98:0/3874221213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5498069eb0 msgr2=0x7f5498198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:19.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.262+0000 7f549ded8640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5498069eb0 0x7f5498198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:19.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.262+0000 7f549ded8640 1 -- 172.21.15.98:0/3874221213 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f549806a870 msgr2=0x7f54981985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:19.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.262+0000 7f549ded8640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f549806a870 0x7f54981985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:19.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.262+0000 7f549ded8640 1 -- 172.21.15.98:0/3874221213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549819d030 con 0x7f5498107820 2024-05-09T09:03:19.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.262+0000 7f549ced6640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f549806a870 0x7f54981985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:03:19.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.262+0000 7f549ded8640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5498107820 0x7f549819c900 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f549000cad0 tx=0x7f549000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:19.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.262+0000 7f548e7fc640 1 -- 172.21.15.98:0/3874221213 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549000ae20 con 0x7f5498107820 2024-05-09T09:03:19.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.262+0000 7f548e7fc640 1 -- 172.21.15.98:0/3874221213 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5490005020 con 0x7f5498107820 2024-05-09T09:03:19.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.262+0000 7f548e7fc640 1 -- 172.21.15.98:0/3874221213 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549000f6f0 con 0x7f5498107820 2024-05-09T09:03:19.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.262+0000 7f549f93b640 1 -- 172.21.15.98:0/3874221213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f549819d320 con 0x7f5498107820 2024-05-09T09:03:19.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.262+0000 7f549f93b640 1 -- 172.21.15.98:0/3874221213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54981a4bb0 con 0x7f5498107820 2024-05-09T09:03:19.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.263+0000 7f549f93b640 1 -- 172.21.15.98:0/3874221213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5498068aa0 con 0x7f5498107820 2024-05-09T09:03:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.265+0000 7f548e7fc640 1 -- 172.21.15.98:0/3874221213 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f54900051d0 con 0x7f5498107820 2024-05-09T09:03:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.265+0000 7f548e7fc640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54840a2d00 0x7f54840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.266+0000 7f549d6d7640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54840a2d00 0x7f54840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.266+0000 7f548e7fc640 1 -- 172.21.15.98:0/3874221213 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9400+0+0 (secure 0 0 0) 0x7f54900cfc80 con 0x7f5498107820 2024-05-09T09:03:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.266+0000 7f549d6d7640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54840a2d00 0x7f54840a51c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5494000c00 tx=0x7f5494031490 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.270+0000 7f548e7fc640 1 -- 172.21.15.98:0/3874221213 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5490014030 con 0x7f5498107820 2024-05-09T09:03:19.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:19 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/918569482' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:03:19.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:19.451+0000 7f549f93b640 1 -- 172.21.15.98:0/3874221213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f549802d1c0 con 0x7f5498107820 2024-05-09T09:03:19.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:19 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/918569482' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:03:19.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:19 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/918569482' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:03:20.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.131+0000 7f548e7fc640 1 -- 172.21.15.98:0/3874221213 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v114) ==== 169+0+0 (secure 0 0 0) 0x7f549009c4d0 con 0x7f5498107820 2024-05-09T09:03:20.130 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' removed 2024-05-09T09:03:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.134+0000 7f549f93b640 1 -- 172.21.15.98:0/3874221213 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54840a2d00 msgr2=0x7f54840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.134+0000 7f549f93b640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54840a2d00 0x7f54840a51c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5494000c00 tx=0x7f5494031490 comp rx=0 tx=0).stop 2024-05-09T09:03:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.134+0000 7f549f93b640 1 -- 172.21.15.98:0/3874221213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5498107820 msgr2=0x7f549819c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.134+0000 7f549f93b640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5498107820 0x7f549819c900 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f549000cad0 tx=0x7f549000cf90 comp rx=0 tx=0).stop 2024-05-09T09:03:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.134+0000 7f549f93b640 1 -- 172.21.15.98:0/3874221213 shutdown_connections 2024-05-09T09:03:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.134+0000 7f549f93b640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f54840a2d00 0x7f54840a51c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.134+0000 7f549f93b640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5498107820 0x7f549819c900 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.134+0000 7f549f93b640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f549806a870 0x7f54981985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.134+0000 7f549f93b640 1 --2- 172.21.15.98:0/3874221213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5498069eb0 0x7f5498198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:20.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.134+0000 7f549f93b640 1 -- 172.21.15.98:0/3874221213 >> 172.21.15.98:0/3874221213 conn(0x7f5498077790 msgr2=0x7f54980fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:20.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.134+0000 7f549f93b640 1 -- 172.21.15.98:0/3874221213 shutdown_connections 2024-05-09T09:03:20.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.134+0000 7f549f93b640 1 -- 172.21.15.98:0/3874221213 wait complete. 2024-05-09T09:03:20.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-05-09T09:03:20.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.257+0000 7f4c7661c640 1 -- 172.21.15.98:0/2069933964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c701003d0 msgr2=0x7f4c701007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:20.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.257+0000 7f4c7661c640 1 --2- 172.21.15.98:0/2069933964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c701003d0 0x7f4c701007b0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009900 tx=0x7f4c6002a410 comp rx=0 tx=0).stop 2024-05-09T09:03:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.257+0000 7f4c7661c640 1 -- 172.21.15.98:0/2069933964 shutdown_connections 2024-05-09T09:03:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.257+0000 7f4c7661c640 1 --2- 172.21.15.98:0/2069933964 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c70106b90 0x7f4c7010e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.257+0000 7f4c7661c640 1 --2- 172.21.15.98:0/2069933964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c70100d80 0x7f4c701064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.257+0000 7f4c7661c640 1 --2- 172.21.15.98:0/2069933964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c701003d0 0x7f4c701007b0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.257+0000 7f4c7661c640 1 -- 172.21.15.98:0/2069933964 >> 172.21.15.98:0/2069933964 conn(0x7f4c700759c0 msgr2=0x7f4c70075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.257+0000 7f4c7661c640 1 -- 172.21.15.98:0/2069933964 shutdown_connections 2024-05-09T09:03:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.257+0000 7f4c7661c640 1 -- 172.21.15.98:0/2069933964 wait complete. 2024-05-09T09:03:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.257+0000 7f4c7661c640 1 Processor -- start 2024-05-09T09:03:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c7661c640 1 -- start start 2024-05-09T09:03:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c7661c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c701003d0 0x7f4c701980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c7661c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c70106b90 0x7f4c70198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c7661c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c7019c9a0 0x7f4c7019ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c7661c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4c70111c80 con 0x7f4c7019c9a0 2024-05-09T09:03:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c7661c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4c70111b00 con 0x7f4c70106b90 2024-05-09T09:03:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c7661c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4c70111e00 con 0x7f4c701003d0 2024-05-09T09:03:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c74bb9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c7019c9a0 0x7f4c7019ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c74bb9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c7019c9a0 0x7f4c7019ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35762/0 (socket says 172.21.15.98:35762) 2024-05-09T09:03:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c74bb9640 1 -- 172.21.15.98:0/551968538 learned_addr learned my addr 172.21.15.98:0/551968538 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-09T09:03:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c6ffff640 1 --2- 172.21.15.98:0/551968538 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c701003d0 0x7f4c701980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c6f7fe640 1 --2- 172.21.15.98:0/551968538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c70106b90 0x7f4c70198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c74bb9640 1 -- 172.21.15.98:0/551968538 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c701003d0 msgr2=0x7f4c701980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c74bb9640 1 --2- 172.21.15.98:0/551968538 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c701003d0 0x7f4c701980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c74bb9640 1 -- 172.21.15.98:0/551968538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c70106b90 msgr2=0x7f4c70198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c74bb9640 1 --2- 172.21.15.98:0/551968538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c70106b90 0x7f4c70198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c74bb9640 1 -- 172.21.15.98:0/551968538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7019d490 con 0x7f4c7019c9a0 2024-05-09T09:03:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.258+0000 7f4c74bb9640 1 --2- 172.21.15.98:0/551968538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c7019c9a0 0x7f4c7019ce50 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f4c6800cce0 tx=0x7f4c6800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.259+0000 7f4c6d7fa640 1 -- 172.21.15.98:0/551968538 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6800f450 con 0x7f4c7019c9a0 2024-05-09T09:03:20.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.259+0000 7f4c7661c640 1 -- 172.21.15.98:0/551968538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c7019d780 con 0x7f4c7019c9a0 2024-05-09T09:03:20.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.259+0000 7f4c6d7fa640 1 -- 172.21.15.98:0/551968538 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c6800fad0 con 0x7f4c7019c9a0 2024-05-09T09:03:20.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.259+0000 7f4c6d7fa640 1 -- 172.21.15.98:0/551968538 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c68018c10 con 0x7f4c7019c9a0 2024-05-09T09:03:20.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.259+0000 7f4c7661c640 1 -- 172.21.15.98:0/551968538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c701a4f20 con 0x7f4c7019c9a0 2024-05-09T09:03:20.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.260+0000 7f4c7661c640 1 -- 172.21.15.98:0/551968538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c70068aa0 con 0x7f4c7019c9a0 2024-05-09T09:03:20.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.261+0000 7f4c6d7fa640 1 -- 172.21.15.98:0/551968538 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4c6800f5f0 con 0x7f4c7019c9a0 2024-05-09T09:03:20.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.262+0000 7f4c6d7fa640 1 --2- 172.21.15.98:0/551968538 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4c580a2d00 0x7f4c580a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:03:20.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.262+0000 7f4c6ffff640 1 --2- 172.21.15.98:0/551968538 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4c580a2d00 0x7f4c580a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:03:20.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.262+0000 7f4c6d7fa640 1 -- 172.21.15.98:0/551968538 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8754+0+0 (secure 0 0 0) 0x7f4c680cfac0 con 0x7f4c7019c9a0 2024-05-09T09:03:20.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.262+0000 7f4c6ffff640 1 --2- 172.21.15.98:0/551968538 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4c580a2d00 0x7f4c580a51c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4c60000c00 tx=0x7f4c600058d0 comp rx=0 tx=0).ready entity=mgr.15555 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:03:20.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.267+0000 7f4c6d7fa640 1 -- 172.21.15.98:0/551968538 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4c68014030 con 0x7f4c7019c9a0 2024-05-09T09:03:20.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:20 smithi142 ceph-mon[34625]: pgmap v400: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T09:03:20.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:20 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/918569482' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T09:03:20.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:20 smithi142 ceph-mon[34625]: osdmap e113: 12 total, 12 up, 12 in 2024-05-09T09:03:20.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:20 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3874221213' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:03:20.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:20.449+0000 7f4c7661c640 1 -- 172.21.15.98:0/551968538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4c70109df0 con 0x7f4c7019c9a0 2024-05-09T09:03:20.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:20 smithi105 ceph-mon[34498]: pgmap v400: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T09:03:20.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:20 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/918569482' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T09:03:20.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:20 smithi105 ceph-mon[34498]: osdmap e113: 12 total, 12 up, 12 in 2024-05-09T09:03:20.487 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:20 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3874221213' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:03:20.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:20 smithi098 ceph-mon[30092]: pgmap v400: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T09:03:20.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:20 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/918569482' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T09:03:20.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:20 smithi098 ceph-mon[30092]: osdmap e113: 12 total, 12 up, 12 in 2024-05-09T09:03:20.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:20 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3874221213' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:03:21.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:21.142+0000 7f4c6d7fa640 1 -- 172.21.15.98:0/551968538 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v115) ==== 178+0+0 (secure 0 0 0) 0x7f4c6809c5f0 con 0x7f4c7019c9a0 2024-05-09T09:03:21.142 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data_ec' removed 2024-05-09T09:03:21.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:21.146+0000 7f4c7661c640 1 -- 172.21.15.98:0/551968538 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4c580a2d00 msgr2=0x7f4c580a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:21.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:21.146+0000 7f4c7661c640 1 --2- 172.21.15.98:0/551968538 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4c580a2d00 0x7f4c580a51c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4c60000c00 tx=0x7f4c600058d0 comp rx=0 tx=0).stop 2024-05-09T09:03:21.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:21.146+0000 7f4c7661c640 1 -- 172.21.15.98:0/551968538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c7019c9a0 msgr2=0x7f4c7019ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:03:21.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:21.146+0000 7f4c7661c640 1 --2- 172.21.15.98:0/551968538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c7019c9a0 0x7f4c7019ce50 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f4c6800cce0 tx=0x7f4c6800a620 comp rx=0 tx=0).stop 2024-05-09T09:03:21.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:21.146+0000 7f4c7661c640 1 -- 172.21.15.98:0/551968538 shutdown_connections 2024-05-09T09:03:21.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:21.146+0000 7f4c7661c640 1 --2- 172.21.15.98:0/551968538 >> [v2:172.21.15.98:6800/4136638623,v1:172.21.15.98:6801/4136638623] conn(0x7f4c580a2d00 0x7f4c580a51c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:21.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:21.146+0000 7f4c7661c640 1 --2- 172.21.15.98:0/551968538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c7019c9a0 0x7f4c7019ce50 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:21.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:21.146+0000 7f4c7661c640 1 --2- 172.21.15.98:0/551968538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c70106b90 0x7f4c70198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:21.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:21.146+0000 7f4c7661c640 1 --2- 172.21.15.98:0/551968538 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c701003d0 0x7f4c701980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:03:21.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:21.146+0000 7f4c7661c640 1 -- 172.21.15.98:0/551968538 >> 172.21.15.98:0/551968538 conn(0x7f4c700759c0 msgr2=0x7f4c700fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:03:21.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:21.146+0000 7f4c7661c640 1 -- 172.21.15.98:0/551968538 shutdown_connections 2024-05-09T09:03:21.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-09T09:03:21.147+0000 7f4c7661c640 1 -- 172.21.15.98:0/551968538 wait complete. 2024-05-09T09:03:21.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:03:21.216 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:03:21.245 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:03:21.281 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-09T09:03:21.281 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-09T09:03:21.324 DEBUG:teuthology.orchestra.run.smithi105:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-09T09:03:21.340 DEBUG:teuthology.orchestra.run.smithi142:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-09T09:03:21.365 INFO:tasks.cephadm:Stopping all daemons... 2024-05-09T09:03:21.365 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-09T09:03:21.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.a 2024-05-09T09:03:21.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:21 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/3874221213' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T09:03:21.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:21 smithi142 ceph-mon[34625]: osdmap e114: 12 total, 12 up, 12 in 2024-05-09T09:03:21.408 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:21 smithi142 ceph-mon[34625]: from='client.? 172.21.15.98:0/551968538' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:03:21.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:21 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/3874221213' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T09:03:21.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:21 smithi098 ceph-mon[30092]: osdmap e114: 12 total, 12 up, 12 in 2024-05-09T09:03:21.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:21 smithi098 ceph-mon[30092]: from='client.? 172.21.15.98:0/551968538' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:03:21.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:21 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/3874221213' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T09:03:21.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:21 smithi105 ceph-mon[34498]: osdmap e114: 12 total, 12 up, 12 in 2024-05-09T09:03:21.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:21 smithi105 ceph-mon[34498]: from='client.? 172.21.15.98:0/551968538' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:03:21.761 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:21 smithi098 systemd[1]: Stopping Ceph mon.a for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:03:22.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:21 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a[30088]: 2024-05-09T09:03:21.761+0000 7f871fc35700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:03:22.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:21 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a[30088]: 2024-05-09T09:03:21.761+0000 7f871fc35700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-09T09:03:22.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:21 smithi098 podman[105132]: 2024-05-09 09:03:21.955646179 +0000 UTC m=+0.332351941 container died 3ed83f274f2c63abef2cc7f6b6cf066f924ff174464da0af85b24ae57aedf276 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-09T09:03:22.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:22 smithi098 podman[105132]: 2024-05-09 09:03:22.503898515 +0000 UTC m=+0.880604275 container cleanup 3ed83f274f2c63abef2cc7f6b6cf066f924ff174464da0af85b24ae57aedf276 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-09T09:03:22.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:May 09 09:03:22 smithi098 bash[105132]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-a 2024-05-09T09:03:22.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.a.service' 2024-05-09T09:03:23.060 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:03:23.060 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-09T09:03:23.060 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-09T09:03:23.060 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.b 2024-05-09T09:03:23.407 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:23 smithi105 systemd[1]: Stopping Ceph mon.b for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:03:23.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:23 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-b[34494]: 2024-05-09T09:03:23.407+0000 7f2ce4bf8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:03:23.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:23 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-b[34494]: 2024-05-09T09:03:23.407+0000 7f2ce4bf8700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-09T09:03:23.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:23 smithi105 podman[70979]: 2024-05-09 09:03:23.500433247 +0000 UTC m=+0.214497986 container died 085dc4d67b8c2e9bd519efca9aad2a29ac53f7798167dd857e419d4d236cf23e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:03:24.145 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:23 smithi105 podman[70979]: 2024-05-09 09:03:23.910564857 +0000 UTC m=+0.624629613 container cleanup 085dc4d67b8c2e9bd519efca9aad2a29ac53f7798167dd857e419d4d236cf23e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-96805dc, ceph=True, io.buildah.version=1.33.5) 2024-05-09T09:03:24.145 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:23 smithi105 bash[70979]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-b 2024-05-09T09:03:24.384 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.b.service' 2024-05-09T09:03:24.405 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:24 smithi105 podman[70993]: 2024-05-09 09:03:24.146192873 +0000 UTC m=+0.648542936 container remove 085dc4d67b8c2e9bd519efca9aad2a29ac53f7798167dd857e419d4d236cf23e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True) 2024-05-09T09:03:24.405 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:24 smithi105 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.b.service: Deactivated successfully. 2024-05-09T09:03:24.405 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:24 smithi105 systemd[1]: Stopped Ceph mon.b for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T09:03:24.405 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 09 09:03:24 smithi105 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.b.service: Consumed 13.635s CPU time. 2024-05-09T09:03:24.458 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:03:24.459 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-09T09:03:24.459 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-09T09:03:24.459 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.c 2024-05-09T09:03:24.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:24 smithi142 systemd[1]: Stopping Ceph mon.c for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:03:25.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:24 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-c[34608]: 2024-05-09T09:03:24.790+0000 7f562378c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:03:25.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:24 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-c[34608]: 2024-05-09T09:03:24.790+0000 7f562378c700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-09T09:03:25.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:24 smithi142 podman[70612]: 2024-05-09 09:03:24.92439658 +0000 UTC m=+0.255307615 container died 50e9afbdc9d32bc6a96c2d04d493f1cf3fb4d076091d6f887023d26ba5244eed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-c, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240507, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-96805dc) 2024-05-09T09:03:25.567 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:25 smithi142 podman[70612]: 2024-05-09 09:03:25.317188197 +0000 UTC m=+0.648099231 container cleanup 50e9afbdc9d32bc6a96c2d04d493f1cf3fb4d076091d6f887023d26ba5244eed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux ) 2024-05-09T09:03:25.567 INFO:journalctl@ceph.mon.c.smithi142.stdout:May 09 09:03:25 smithi142 bash[70612]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mon-c 2024-05-09T09:03:25.714 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mon.c.service' 2024-05-09T09:03:25.762 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:03:25.763 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-09T09:03:25.763 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-09T09:03:25.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mgr.x 2024-05-09T09:03:26.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 09:03:25 smithi098 systemd[1]: Stopping Ceph mgr.x for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:03:26.409 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 09:03:26 smithi098 podman[105250]: 2024-05-09 09:03:26.106939992 +0000 UTC m=+0.158899934 container died 2da87a81222bd30e0a2e92a0c51263e07943e91030291513edd046d1301bde19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-09T09:03:26.778 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 09:03:26 smithi098 podman[105250]: 2024-05-09 09:03:26.518335798 +0000 UTC m=+0.570295732 container cleanup 2da87a81222bd30e0a2e92a0c51263e07943e91030291513edd046d1301bde19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-09T09:03:26.778 INFO:journalctl@ceph.mgr.x.smithi098.stdout:May 09 09:03:26 smithi098 bash[105250]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-x 2024-05-09T09:03:26.900 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mgr.x.service' 2024-05-09T09:03:26.939 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:03:26.939 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-09T09:03:26.939 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-09T09:03:26.939 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mgr.y 2024-05-09T09:03:27.236 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 09:03:26 smithi105 systemd[1]: Stopping Ceph mgr.y for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:03:27.558 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 09:03:27 smithi105 podman[71099]: 2024-05-09 09:03:27.289837027 +0000 UTC m=+0.159152493 container died 0ee24e7f39c9e532db2eac014de2805232f334fa689b914d68ac559653178a6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True) 2024-05-09T09:03:27.986 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 09:03:27 smithi105 podman[71099]: 2024-05-09 09:03:27.651446162 +0000 UTC m=+0.520761635 container cleanup 0ee24e7f39c9e532db2eac014de2805232f334fa689b914d68ac559653178a6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507) 2024-05-09T09:03:27.986 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 09 09:03:27 smithi105 bash[71099]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-y 2024-05-09T09:03:28.811 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mgr.y.service' 2024-05-09T09:03:28.844 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:03:28.844 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-09T09:03:28.844 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-09T09:03:28.844 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mgr.z 2024-05-09T09:03:29.158 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 09:03:28 smithi142 systemd[1]: Stopping Ceph mgr.z for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:03:29.463 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 09:03:29 smithi142 podman[70733]: 2024-05-09 09:03:29.159543942 +0000 UTC m=+0.135406729 container died 725df350de80a739fcf38f4d6e7868bd7b5077ceb192783090cd71717b18f50d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, io.buildah.version=1.33.5) 2024-05-09T09:03:29.831 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 09:03:29 smithi142 podman[70733]: 2024-05-09 09:03:29.564412676 +0000 UTC m=+0.540275465 container cleanup 725df350de80a739fcf38f4d6e7868bd7b5077ceb192783090cd71717b18f50d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc) 2024-05-09T09:03:29.831 INFO:journalctl@ceph.mgr.z.smithi142.stdout:May 09 09:03:29 smithi142 bash[70733]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mgr-z 2024-05-09T09:03:29.954 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mgr.z.service' 2024-05-09T09:03:29.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:03:29.988 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-09T09:03:29.988 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-09T09:03:29.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.0 2024-05-09T09:03:30.297 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 09:03:30 smithi098 systemd[1]: Stopping Ceph osd.0 for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:03:30.587 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 09:03:30 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0[44312]: 2024-05-09T09:03:30.298+0000 7fdb4f342700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:03:30.587 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 09:03:30 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0[44312]: 2024-05-09T09:03:30.298+0000 7fdb4f342700 -1 osd.0 115 *** Got signal Terminated *** 2024-05-09T09:03:30.588 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 09:03:30 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0[44312]: 2024-05-09T09:03:30.298+0000 7fdb4f342700 -1 osd.0 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:03:36.149 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 09:03:35 smithi098 podman[105372]: 2024-05-09 09:03:35.842403001 +0000 UTC m=+5.666036962 container died 62481a5cd8001a84db7665ec2ff9e464e0372ec47d047113f9aad10c2af35a60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_BRANCH=HEAD) 2024-05-09T09:03:36.575 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 09:03:36 smithi098 podman[105372]: 2024-05-09 09:03:36.291178939 +0000 UTC m=+6.114812900 container cleanup 62481a5cd8001a84db7665ec2ff9e464e0372ec47d047113f9aad10c2af35a60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-09T09:03:36.575 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 09:03:36 smithi098 bash[105372]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0 2024-05-09T09:03:36.837 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 09:03:36 smithi098 podman[105389]: 2024-05-09 09:03:36.576201162 +0000 UTC m=+0.722105293 container remove 62481a5cd8001a84db7665ec2ff9e464e0372ec47d047113f9aad10c2af35a60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS) 2024-05-09T09:03:37.262 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 09:03:36 smithi098 podman[105480]: 2024-05-09 09:03:36.892502644 +0000 UTC m=+0.017632932 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:03:37.262 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 09:03:37 smithi098 podman[105480]: 2024-05-09 09:03:37.027693775 +0000 UTC m=+0.152824051 container create 907685a30e22f9c968d962752ba50a021fb1e84c9a53d791ec6fbae9f7e42192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-09T09:03:37.588 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 09:03:37 smithi098 podman[105480]: 2024-05-09 09:03:37.263847646 +0000 UTC m=+0.388977923 container init 907685a30e22f9c968d962752ba50a021fb1e84c9a53d791ec6fbae9f7e42192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507) 2024-05-09T09:03:37.588 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 09:03:37 smithi098 podman[105480]: 2024-05-09 09:03:37.267121959 +0000 UTC m=+0.392252236 container start 907685a30e22f9c968d962752ba50a021fb1e84c9a53d791ec6fbae9f7e42192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:03:37.588 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 09:03:37 smithi098 podman[105480]: 2024-05-09 09:03:37.389119804 +0000 UTC m=+0.514250090 container attach 907685a30e22f9c968d962752ba50a021fb1e84c9a53d791ec6fbae9f7e42192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0-deactivate, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:03:38.087 INFO:journalctl@ceph.osd.0.smithi098.stdout:May 09 09:03:37 smithi098 podman[105480]: 2024-05-09 09:03:37.825728992 +0000 UTC m=+0.950859270 container died 907685a30e22f9c968d962752ba50a021fb1e84c9a53d791ec6fbae9f7e42192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-05-09T09:03:38.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.0.service' 2024-05-09T09:03:38.476 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:03:38.476 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-09T09:03:38.477 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-09T09:03:38.477 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.1 2024-05-09T09:03:38.790 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 09:03:38 smithi105 systemd[1]: Stopping Ceph osd.1 for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:03:39.236 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 09:03:38 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-1[39079]: 2024-05-09T09:03:38.790+0000 7f97e6ebd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:03:39.237 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 09:03:38 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-1[39079]: 2024-05-09T09:03:38.790+0000 7f97e6ebd700 -1 osd.1 115 *** Got signal Terminated *** 2024-05-09T09:03:39.237 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 09:03:38 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-1[39079]: 2024-05-09T09:03:38.790+0000 7f97e6ebd700 -1 osd.1 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:03:44.723 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 09:03:44 smithi105 podman[71234]: 2024-05-09 09:03:44.429469068 +0000 UTC m=+5.767642389 container died e326c4105a1d52f214018a3ceff16a5892801ca7688e0e8b67873dd1062cf14d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-09T09:03:44.987 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 09:03:44 smithi105 podman[71234]: 2024-05-09 09:03:44.816259725 +0000 UTC m=+6.154433048 container cleanup e326c4105a1d52f214018a3ceff16a5892801ca7688e0e8b67873dd1062cf14d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:03:44.987 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 09:03:44 smithi105 bash[71234]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-1 2024-05-09T09:03:45.430 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 09:03:45 smithi105 podman[71254]: 2024-05-09 09:03:45.10136856 +0000 UTC m=+0.661060357 container remove e326c4105a1d52f214018a3ceff16a5892801ca7688e0e8b67873dd1062cf14d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-1, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc) 2024-05-09T09:03:45.704 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 09:03:45 smithi105 podman[71346]: 2024-05-09 09:03:45.335134873 +0000 UTC m=+0.017987192 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:03:45.704 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 09:03:45 smithi105 podman[71346]: 2024-05-09 09:03:45.460854958 +0000 UTC m=+0.143707276 container create b24c70d50fe914c486cc0e507f289cc3f7f791c28e27bd009f7aca19552a637e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-1-deactivate, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507) 2024-05-09T09:03:45.704 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 09:03:45 smithi105 podman[71346]: 2024-05-09 09:03:45.705670202 +0000 UTC m=+0.388522536 container init b24c70d50fe914c486cc0e507f289cc3f7f791c28e27bd009f7aca19552a637e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-1-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-96805dc) 2024-05-09T09:03:45.986 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 09:03:45 smithi105 podman[71346]: 2024-05-09 09:03:45.708749366 +0000 UTC m=+0.391601689 container start b24c70d50fe914c486cc0e507f289cc3f7f791c28e27bd009f7aca19552a637e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240507) 2024-05-09T09:03:45.987 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 09:03:45 smithi105 podman[71346]: 2024-05-09 09:03:45.839025488 +0000 UTC m=+0.521877824 container attach b24c70d50fe914c486cc0e507f289cc3f7f791c28e27bd009f7aca19552a637e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2) 2024-05-09T09:03:46.486 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 09 09:03:46 smithi105 podman[71346]: 2024-05-09 09:03:46.209198072 +0000 UTC m=+0.892050383 container died b24c70d50fe914c486cc0e507f289cc3f7f791c28e27bd009f7aca19552a637e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, RELEASE=squid-96805dc) 2024-05-09T09:03:47.114 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.1.service' 2024-05-09T09:03:47.155 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:03:47.155 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-09T09:03:47.155 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-09T09:03:47.155 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.2 2024-05-09T09:03:47.467 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 09:03:47 smithi142 systemd[1]: Stopping Ceph osd.2 for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:03:47.908 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 09:03:47 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-2[39778]: 2024-05-09T09:03:47.467+0000 7f6d1e40f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:03:47.908 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 09:03:47 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-2[39778]: 2024-05-09T09:03:47.467+0000 7f6d1e40f700 -1 osd.2 115 *** Got signal Terminated *** 2024-05-09T09:03:47.908 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 09:03:47 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-2[39778]: 2024-05-09T09:03:47.467+0000 7f6d1e40f700 -1 osd.2 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:03:53.359 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 09:03:53 smithi142 podman[70870]: 2024-05-09 09:03:53.054173085 +0000 UTC m=+5.700306165 container died e7ad21352328a94b8b608c04113a304e7af4f2bae51c43850f17b4a907451e17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-2, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-09T09:03:53.658 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 09:03:53 smithi142 podman[70870]: 2024-05-09 09:03:53.460126853 +0000 UTC m=+6.106259886 container cleanup e7ad21352328a94b8b608c04113a304e7af4f2bae51c43850f17b4a907451e17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:03:53.658 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 09:03:53 smithi142 bash[70870]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-2 2024-05-09T09:03:54.029 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 09:03:53 smithi142 podman[70887]: 2024-05-09 09:03:53.728343619 +0000 UTC m=+0.657009468 container remove e7ad21352328a94b8b608c04113a304e7af4f2bae51c43850f17b4a907451e17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-2, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240507) 2024-05-09T09:03:54.331 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 09:03:54 smithi142 podman[70975]: 2024-05-09 09:03:53.933912223 +0000 UTC m=+0.017757976 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:03:54.331 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 09:03:54 smithi142 podman[70975]: 2024-05-09 09:03:54.065859014 +0000 UTC m=+0.149704754 container create e38140136e5ff2f3254b474affb991b80d853aafb7327ecf6e9def8127ccb61a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True) 2024-05-09T09:03:54.636 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 09:03:54 smithi142 podman[70975]: 2024-05-09 09:03:54.332616943 +0000 UTC m=+0.416462691 container init e38140136e5ff2f3254b474affb991b80d853aafb7327ecf6e9def8127ccb61a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-96805dc, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:03:54.636 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 09:03:54 smithi142 podman[70975]: 2024-05-09 09:03:54.336162864 +0000 UTC m=+0.420008613 container start e38140136e5ff2f3254b474affb991b80d853aafb7327ecf6e9def8127ccb61a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-09T09:03:54.636 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 09:03:54 smithi142 podman[70975]: 2024-05-09 09:03:54.465847104 +0000 UTC m=+0.549692848 container attach e38140136e5ff2f3254b474affb991b80d853aafb7327ecf6e9def8127ccb61a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:03:54.908 INFO:journalctl@ceph.osd.2.smithi142.stdout:May 09 09:03:54 smithi142 podman[70975]: 2024-05-09 09:03:54.637644422 +0000 UTC m=+0.721490163 container died e38140136e5ff2f3254b474affb991b80d853aafb7327ecf6e9def8127ccb61a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-09T09:03:55.338 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.2.service' 2024-05-09T09:03:55.377 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:03:55.377 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-09T09:03:55.378 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-09T09:03:55.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.3 2024-05-09T09:03:55.695 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:03:55 smithi098 systemd[1]: Stopping Ceph osd.3 for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:03:55.957 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:03:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3[51245]: 2024-05-09T09:03:55.696+0000 7f1aa498b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:03:55.957 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:03:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3[51245]: 2024-05-09T09:03:55.696+0000 7f1aa498b700 -1 osd.3 115 *** Got signal Terminated *** 2024-05-09T09:03:55.957 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:03:55 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3[51245]: 2024-05-09T09:03:55.696+0000 7f1aa498b700 -1 osd.3 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:03:57.337 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:03:56 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:03:56.915+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:03:57.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:03:57 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:03:57.493+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:03:58.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:03:57 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:03:57.932+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:03:58.337 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:03:57 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:03:57.942+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:03:58.837 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:03:58 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3[51245]: 2024-05-09T09:03:58.521+0000 7f1a9cf77700 -1 osd.3 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.619240+0000 front 2024-05-09T09:03:32.619211+0000 (oldest deadline 2024-05-09T09:03:57.919303+0000) 2024-05-09T09:03:58.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:03:58 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:03:58.460+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:03:58.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:03:58 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:03:58.437+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:03:59.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:03:58 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:03:58.951+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:03:59.338 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:03:58 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:03:58.926+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:03:59.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:03:59 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:03:59.115+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:03:59.837 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:03:59 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3[51245]: 2024-05-09T09:03:59.530+0000 7f1a9cf77700 -1 osd.3 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.619240+0000 front 2024-05-09T09:03:32.619211+0000 (oldest deadline 2024-05-09T09:03:57.919303+0000) 2024-05-09T09:03:59.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:03:59 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:03:59.469+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:03:59.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:03:59 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:03:59.454+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:00.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:03:59 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:03:59.916+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:00.337 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:03:59 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:03:59.893+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:00.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:00 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:00.140+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:00.837 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:04:00 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3[51245]: 2024-05-09T09:04:00.514+0000 7f1a9cf77700 -1 osd.3 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.619240+0000 front 2024-05-09T09:03:32.619211+0000 (oldest deadline 2024-05-09T09:03:57.919303+0000) 2024-05-09T09:04:00.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:00 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:00.477+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:00.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:00 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:00.465+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:01.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:00 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:00.928+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:01.263 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:00 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:00.912+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:01.264 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:01 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:01.128+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:01.555 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:04:01 smithi098 podman[105585]: 2024-05-09 09:04:01.265222743 +0000 UTC m=+5.687035649 container died b1e2bbd93c77b578b7bbf5b00639dc53f5e2b4388c0a30779f37185277129d9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507) 2024-05-09T09:04:01.838 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:04:01 smithi098 podman[105585]: 2024-05-09 09:04:01.655539233 +0000 UTC m=+6.077352138 container cleanup b1e2bbd93c77b578b7bbf5b00639dc53f5e2b4388c0a30779f37185277129d9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:04:01.838 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:04:01 smithi098 bash[105585]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3 2024-05-09T09:04:01.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:01 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:01.447+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:01.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:01 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:01.454+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:02.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:01 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:01.906+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:02.268 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:02 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:02.166+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:02.268 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:04:01 smithi098 podman[105614]: 2024-05-09 09:04:01.936782331 +0000 UTC m=+0.667114636 container remove b1e2bbd93c77b578b7bbf5b00639dc53f5e2b4388c0a30779f37185277129d9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-09T09:04:02.269 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:01 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:01.941+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:02.588 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:04:02 smithi098 podman[105708]: 2024-05-09 09:04:02.172940605 +0000 UTC m=+0.018004375 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:04:02.588 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:04:02 smithi098 podman[105708]: 2024-05-09 09:04:02.3103804 +0000 UTC m=+0.155444157 container create 0be6b6ed07e5947b622be42a241d543142411a6e94858e105e90f2ae6a67c0f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-09T09:04:02.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:02 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:02.427+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:02.658 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:02 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:02.407+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:02.926 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:04:02 smithi098 podman[105708]: 2024-05-09 09:04:02.638139543 +0000 UTC m=+0.483203310 container init 0be6b6ed07e5947b622be42a241d543142411a6e94858e105e90f2ae6a67c0f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-09T09:04:02.926 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:04:02 smithi098 podman[105708]: 2024-05-09 09:04:02.641556829 +0000 UTC m=+0.486620591 container start 0be6b6ed07e5947b622be42a241d543142411a6e94858e105e90f2ae6a67c0f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.build-date=20240507) 2024-05-09T09:04:02.926 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:04:02 smithi098 podman[105708]: 2024-05-09 09:04:02.730016585 +0000 UTC m=+0.575080354 container attach 0be6b6ed07e5947b622be42a241d543142411a6e94858e105e90f2ae6a67c0f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3-deactivate, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-09T09:04:03.183 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:02 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:02.991+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:03.183 INFO:journalctl@ceph.osd.3.smithi098.stdout:May 09 09:04:02 smithi098 podman[105708]: 2024-05-09 09:04:02.928155172 +0000 UTC m=+0.773218930 container died 0be6b6ed07e5947b622be42a241d543142411a6e94858e105e90f2ae6a67c0f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-09T09:04:03.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:02 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:02.888+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:03.557 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:03 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:03.183+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:03.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.3.service' 2024-05-09T09:04:03.620 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:04:03.621 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-09T09:04:03.621 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-09T09:04:03.621 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.4 2024-05-09T09:04:03.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:03 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:03.434+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:03.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:03 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:03.428+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:03.928 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 09:04:03 smithi105 systemd[1]: Stopping Ceph osd.4 for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:04:03.928 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:03 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:03.885+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:04.236 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 09:04:03 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-4[46017]: 2024-05-09T09:04:03.928+0000 7f03abea8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:04:04.236 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 09:04:03 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-4[46017]: 2024-05-09T09:04:03.928+0000 7f03abea8700 -1 osd.4 115 *** Got signal Terminated *** 2024-05-09T09:04:04.236 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 09:04:03 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-4[46017]: 2024-05-09T09:04:03.928+0000 7f03abea8700 -1 osd.4 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:04:04.337 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:04 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:04.006+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:04.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:04 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:04.213+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:04.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:04 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:04.472+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:04.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:04 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:04.467+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:05.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:04 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:04.840+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:05.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:05 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:05.177+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:05.338 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:05 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:05.002+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:05.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:05 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:05.453+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:05.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:05 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:05.484+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:05.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:05 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:05.484+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.604294+0000 front 2024-05-09T09:03:42.604144+0000 (oldest deadline 2024-05-09T09:04:04.904035+0000) 2024-05-09T09:04:06.138 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:05 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:05.870+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:06.138 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:05 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:05.803+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:06.337 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:05 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:05.962+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:06.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:06 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:06.191+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:06.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:06 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:06.191+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:06.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:06 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:06.485+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:06.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:06 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:06.522+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:06.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:06 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:06.522+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.604294+0000 front 2024-05-09T09:03:42.604144+0000 (oldest deadline 2024-05-09T09:04:04.904035+0000) 2024-05-09T09:04:07.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:06 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:06.914+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:07.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:06 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:06.806+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:07.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:06 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:06.806+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:07.337 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:06 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:06.948+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:07.338 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:06 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:06.948+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:07.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:07 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:07.175+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:07.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:07 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:07.175+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:07.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:07 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:07.523+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:07.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:07 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:07.552+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:07.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:07 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:07.552+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.604294+0000 front 2024-05-09T09:03:42.604144+0000 (oldest deadline 2024-05-09T09:04:04.904035+0000) 2024-05-09T09:04:07.909 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:07 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:07.462+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:08.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:07 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:07.888+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:08.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:07 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:07.838+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:08.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:07 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:07.838+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:08.337 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:07 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:07.957+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:08.338 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:07 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:07.957+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:08.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:08 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:08.177+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:08.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:08 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:08.177+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:08.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:08 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:08.532+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:08.908 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:08 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:08.442+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:08.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:08 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:08.505+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:08.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:08 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:08.505+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.604294+0000 front 2024-05-09T09:03:42.604144+0000 (oldest deadline 2024-05-09T09:04:04.904035+0000) 2024-05-09T09:04:09.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:08 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:08.865+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:09.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:08 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:08.812+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:09.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:08 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:08.812+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:09.338 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:08 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:08.965+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:09.338 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:08 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:08.965+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:09.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:09 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:09.194+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:09.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:09 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:09.194+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:09.864 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 09:04:09 smithi105 podman[71451]: 2024-05-09 09:04:09.604001689 +0000 UTC m=+5.788052865 container died 6a089ee8b9973f5389010fee4132833f203994891c119142e974e60d1b8aefaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-4, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-05-09T09:04:09.864 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:09 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:09.779+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:09.864 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:09 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:09.779+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:09.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:09 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:09.504+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:09.908 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:09 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:09.481+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:09.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:09 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:09.551+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:09.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:09 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:09.551+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.604294+0000 front 2024-05-09T09:03:42.604144+0000 (oldest deadline 2024-05-09T09:04:04.904035+0000) 2024-05-09T09:04:10.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:09 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:09.864+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:10.236 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 09:04:10 smithi105 podman[71451]: 2024-05-09 09:04:10.005484348 +0000 UTC m=+6.189535520 container cleanup 6a089ee8b9973f5389010fee4132833f203994891c119142e974e60d1b8aefaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-4, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:04:10.236 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 09:04:10 smithi105 bash[71451]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-4 2024-05-09T09:04:10.337 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:09 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:09.953+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:10.338 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:09 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:09.953+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:10.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:10 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:10.150+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:10.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:10 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:10.150+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:10.624 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 09:04:10 smithi105 podman[71480]: 2024-05-09 09:04:10.281874633 +0000 UTC m=+0.673452806 container remove 6a089ee8b9973f5389010fee4132833f203994891c119142e974e60d1b8aefaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-4, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-09T09:04:10.908 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:10 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:10.463+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:10.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:10 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:10.493+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:10.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:10 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:10.534+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:10.909 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:10 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:10.534+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.604294+0000 front 2024-05-09T09:03:42.604144+0000 (oldest deadline 2024-05-09T09:04:04.904035+0000) 2024-05-09T09:04:10.909 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 09:04:10 smithi105 podman[71571]: 2024-05-09 09:04:10.528445167 +0000 UTC m=+0.017657862 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:04:10.909 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 09:04:10 smithi105 podman[71571]: 2024-05-09 09:04:10.657944396 +0000 UTC m=+0.147157126 container create 2f8495039239ca9889b4e9261c1ae6605a5aa77d6a2809e555864d961c40bcb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:04:10.909 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:10 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:10.758+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:10.909 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:10 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:10.758+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:11.178 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:10 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:10.904+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:11.178 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:10 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:10.904+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:11.199 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:10 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:10.912+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:11.199 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 09:04:10 smithi105 podman[71571]: 2024-05-09 09:04:10.910436296 +0000 UTC m=+0.399648984 container init 2f8495039239ca9889b4e9261c1ae6605a5aa77d6a2809e555864d961c40bcb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, ceph=True, org.label-schema.schema-version=1.0) 2024-05-09T09:04:11.199 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 09:04:10 smithi105 podman[71571]: 2024-05-09 09:04:10.913606328 +0000 UTC m=+0.402819020 container start 2f8495039239ca9889b4e9261c1ae6605a5aa77d6a2809e555864d961c40bcb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:04:11.199 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 09:04:11 smithi105 podman[71571]: 2024-05-09 09:04:11.010485142 +0000 UTC m=+0.499697847 container attach 2f8495039239ca9889b4e9261c1ae6605a5aa77d6a2809e555864d961c40bcb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux ) 2024-05-09T09:04:11.486 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 09 09:04:11 smithi105 podman[71571]: 2024-05-09 09:04:11.201148276 +0000 UTC m=+0.690360957 container died 2f8495039239ca9889b4e9261c1ae6605a5aa77d6a2809e555864d961c40bcb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-09T09:04:11.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:11 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:11.179+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:11.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:11 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:11.179+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:11.817 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:11 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:11.722+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:11.817 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:11 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:11.722+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:11.836 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.4.service' 2024-05-09T09:04:11.874 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:04:11.875 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-09T09:04:11.875 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-09T09:04:11.875 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.5 2024-05-09T09:04:11.901 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:11 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:11.447+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:11.901 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:11 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:11.561+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:11.901 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:11 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:11.561+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.604294+0000 front 2024-05-09T09:03:42.604144+0000 (oldest deadline 2024-05-09T09:04:04.904035+0000) 2024-05-09T09:04:11.902 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:11 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:11.455+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:12.148 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:11 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:11.862+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:12.148 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:11 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:11.862+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:12.158 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:11 smithi142 systemd[1]: Stopping Ceph osd.5 for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:04:12.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:11 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:11.894+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:12.421 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:12 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:12.162+0000 7fcdf708d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:04:12.421 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:12 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:12.162+0000 7fcdf708d700 -1 osd.5 115 *** Got signal Terminated *** 2024-05-09T09:04:12.422 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:12 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:12.162+0000 7fcdf708d700 -1 osd.5 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:04:12.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:12 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:12.149+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:12.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:12 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:12.149+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:12.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:12 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:12.422+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:12.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:12 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:12.602+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:12.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:12 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:12.602+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.604294+0000 front 2024-05-09T09:03:42.604144+0000 (oldest deadline 2024-05-09T09:04:04.904035+0000) 2024-05-09T09:04:12.908 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:12 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:12.466+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:13.141 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:12 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:12.857+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:13.141 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:12 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:12.857+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:13.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:12 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:12.915+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:13.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:12 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:12.756+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:13.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:12 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:12.756+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:13.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:13 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:13.142+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:13.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:13 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:13.142+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:13.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:13 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:13.374+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:13.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:13 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:13.418+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:13.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:13 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:13.418+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:13.658 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:13 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:13.583+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:13.659 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:13 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:13.583+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.604294+0000 front 2024-05-09T09:03:42.604144+0000 (oldest deadline 2024-05-09T09:04:04.904035+0000) 2024-05-09T09:04:13.986 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:13 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:13.933+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:13.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:13 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:13.712+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:13.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:13 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:13.712+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:14.188 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:13 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:13.842+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:14.188 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:13 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:13.842+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:14.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:14 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:14.189+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:14.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:14 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:14.189+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:14.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:14 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:14.189+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:14.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:14 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:14.375+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:14.658 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:14 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:14.573+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:14.658 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:14 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:14.573+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.604294+0000 front 2024-05-09T09:03:42.604144+0000 (oldest deadline 2024-05-09T09:04:04.904035+0000) 2024-05-09T09:04:14.659 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:14 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:14.374+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:14.659 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:14 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:14.374+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:15.227 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:14 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:14.843+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:15.227 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:14 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:14.843+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:15.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:14 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:14.758+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:15.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:14 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:14.758+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:15.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:14 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:14.910+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:15.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:14 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:14.910+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:15.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:15 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:15.228+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:15.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:15 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:15.228+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:15.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:15 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:15.228+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:15.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:15 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:15.382+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:15.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:15 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:15.382+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:15.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:15 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:15.372+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:15.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:15 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:15.372+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:15.659 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:15 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:15.555+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:15.659 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:15 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:15.555+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.604294+0000 front 2024-05-09T09:03:42.604144+0000 (oldest deadline 2024-05-09T09:04:04.904035+0000) 2024-05-09T09:04:15.659 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:15 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:15.555+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.206298+0000 front 2024-05-09T09:03:51.206274+0000 (oldest deadline 2024-05-09T09:04:15.306370+0000) 2024-05-09T09:04:16.087 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:15 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:15.812+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:16.087 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:15 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:15.812+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:16.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:15 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:15.914+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:16.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:15 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:15.914+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:16.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:15 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:15.761+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:16.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:15 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:15.761+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:16.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:16 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:16.247+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:16.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:16 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:16.247+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:16.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:16 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:16.247+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:16.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:16 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:16.386+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:16.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:16 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:16.386+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:16.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:16 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:16.407+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:16.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:16 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:16.407+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:16.659 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:16 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:16.529+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:32.102858+0000 front 2024-05-09T09:03:32.102736+0000 (oldest deadline 2024-05-09T09:03:58.002540+0000) 2024-05-09T09:04:16.659 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:16 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:16.529+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.604294+0000 front 2024-05-09T09:03:42.604144+0000 (oldest deadline 2024-05-09T09:04:04.904035+0000) 2024-05-09T09:04:16.659 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:16 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5[46279]: 2024-05-09T09:04:16.529+0000 7fcdef679700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.206298+0000 front 2024-05-09T09:03:51.206274+0000 (oldest deadline 2024-05-09T09:04:15.306370+0000) 2024-05-09T09:04:17.087 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:16 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:16.794+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:17.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:16 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:16.794+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:17.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:16 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:16.947+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:17.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:16 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:16.947+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:17.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:16 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:16.811+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:17.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:16 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:16.811+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:17.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:16 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:16.811+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:17.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:17.234+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:17.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:17.234+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:17.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:17.234+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:17.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:17.440+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:17.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:17.440+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:17.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:17.354+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:17.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:17 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:17.354+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:18.087 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:17.823+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:18.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:17.823+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:18.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:17 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:17.823+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:52.134515+0000 front 2024-05-09T09:03:52.134537+0000 (oldest deadline 2024-05-09T09:04:17.434609+0000) 2024-05-09T09:04:18.113 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:17 smithi142 podman[71092]: 2024-05-09 09:04:17.811506388 +0000 UTC m=+5.743828647 container died 0d7e1188335878625579d155f8000a7914cf426cdbbba81b29a2b6e07a8d6f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-09T09:04:18.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:17.839+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:18.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:17.839+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:18.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:17.839+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:18.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:17.995+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:18.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:17 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:17.995+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:18.408 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:18 smithi142 podman[71092]: 2024-05-09 09:04:18.222391574 +0000 UTC m=+6.154713831 container cleanup 0d7e1188335878625579d155f8000a7914cf426cdbbba81b29a2b6e07a8d6f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, GIT_CLEAN=True) 2024-05-09T09:04:18.408 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:18 smithi142 bash[71092]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5 2024-05-09T09:04:18.408 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:18 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:18.337+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:18.408 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:18 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:18.337+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:18.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:18 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:18.278+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:18.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:18 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:18.278+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:18.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:18 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:18.278+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:18.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:18 smithi142 podman[71110]: 2024-05-09 09:04:18.623612225 +0000 UTC m=+0.809967731 container remove 0d7e1188335878625579d155f8000a7914cf426cdbbba81b29a2b6e07a8d6f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:04:18.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:18 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:18.475+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:18.909 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:18 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:18.475+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:19.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:19 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:19.002+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:19.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:19 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:19.002+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:19.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:18 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:18.808+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:19.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:18 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:18.808+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:19.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:18 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:18.808+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:19.256 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:18 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:18.847+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:19.256 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:18 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:18.847+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:19.256 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:18 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:18.847+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:52.134515+0000 front 2024-05-09T09:03:52.134537+0000 (oldest deadline 2024-05-09T09:04:17.434609+0000) 2024-05-09T09:04:19.285 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:18 smithi142 podman[71200]: 2024-05-09 09:04:18.881322767 +0000 UTC m=+0.017852443 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:04:19.286 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:19 smithi142 podman[71200]: 2024-05-09 09:04:19.007912579 +0000 UTC m=+0.144442247 container create bff6fffce0571ec1a003f329e470f9b6d9751cfc7a3fc1bef2147d9622284715 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux ) 2024-05-09T09:04:19.572 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:19 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:19.373+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:19.573 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:19 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:19.373+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:19.573 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:19 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:19.506+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:19.573 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:19 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:19.506+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:19.573 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:19 smithi142 podman[71200]: 2024-05-09 09:04:19.285248481 +0000 UTC m=+0.421778152 container init bff6fffce0571ec1a003f329e470f9b6d9751cfc7a3fc1bef2147d9622284715 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-09T09:04:19.573 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:19 smithi142 podman[71200]: 2024-05-09 09:04:19.288812633 +0000 UTC m=+0.425342302 container start bff6fffce0571ec1a003f329e470f9b6d9751cfc7a3fc1bef2147d9622284715 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507) 2024-05-09T09:04:19.573 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:19 smithi142 podman[71200]: 2024-05-09 09:04:19.377043842 +0000 UTC m=+0.513573547 container attach bff6fffce0571ec1a003f329e470f9b6d9751cfc7a3fc1bef2147d9622284715 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc) 2024-05-09T09:04:19.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:19 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:19.257+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:19.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:19 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:19.257+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:19.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:19 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:19.257+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:19.908 INFO:journalctl@ceph.osd.5.smithi142.stdout:May 09 09:04:19 smithi142 podman[71200]: 2024-05-09 09:04:19.57477921 +0000 UTC m=+0.711308882 container died bff6fffce0571ec1a003f329e470f9b6d9751cfc7a3fc1bef2147d9622284715 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-5-deactivate, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-09T09:04:20.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:20 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:20.041+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:20.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:20 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:20.041+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:20.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:19 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:19.811+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:20.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:19 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:19.811+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:20.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:19 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:19.811+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:20.242 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:19 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:19.858+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:20.242 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:19 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:19.858+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:20.242 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:19 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:19.858+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:52.134515+0000 front 2024-05-09T09:03:52.134537+0000 (oldest deadline 2024-05-09T09:04:17.434609+0000) 2024-05-09T09:04:20.262 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.5.service' 2024-05-09T09:04:20.344 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:04:20.344 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-09T09:04:20.344 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-09T09:04:20.344 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.6 2024-05-09T09:04:20.538 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:20 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:20.374+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:20.538 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:20 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:20.374+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:20.587 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:20 smithi098 systemd[1]: Stopping Ceph osd.6 for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:04:20.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:20 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:20.243+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:20.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:20 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:20.243+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:20.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:20 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:20.243+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:20.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:20 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:20.539+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:20.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:20 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:20.539+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:21.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:20 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:20.670+0000 7fa84bc99700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:04:21.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:20 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:20.670+0000 7fa84bc99700 -1 osd.6 115 *** Got signal Terminated *** 2024-05-09T09:04:21.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:20 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:20.670+0000 7fa84bc99700 -1 osd.6 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:04:21.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:20 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:20.888+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:21.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:20 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:20.888+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:21.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:20 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:20.888+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:52.134515+0000 front 2024-05-09T09:03:52.134537+0000 (oldest deadline 2024-05-09T09:04:17.434609+0000) 2024-05-09T09:04:21.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:21 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:21.053+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:21.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:21 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:21.053+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:21.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:20 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:20.840+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:21.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:20 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:20.840+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:21.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:20 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:20.840+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:21.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:21 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:21.262+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:21.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:21 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:21.262+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:21.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:21 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:21.262+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:21.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:21 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:21.344+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:21.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:21 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:21.344+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:21.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:21 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:21.557+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:21.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:21 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:21.557+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:22.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:22 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:22.021+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:22.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:22 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:22.021+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:22.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:21 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:21.854+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:22.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:21 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:21.854+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:22.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:21 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:21.854+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:22.246 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:21 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:21.855+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:22.246 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:21 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:21.855+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:22.246 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:21 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:21.855+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:52.134515+0000 front 2024-05-09T09:03:52.134537+0000 (oldest deadline 2024-05-09T09:04:17.434609+0000) 2024-05-09T09:04:22.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:22 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:22.247+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:22.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:22 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:22.247+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:22.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:22 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:22.247+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:22.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:22 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:22.570+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:22.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:22 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:22.570+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:22.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:22 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:22.323+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:22.659 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:22 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:22.323+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:23.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:22 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:22.821+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:23.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:22 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:22.821+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:23.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:22 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:22.821+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:52.134515+0000 front 2024-05-09T09:03:52.134537+0000 (oldest deadline 2024-05-09T09:04:17.434609+0000) 2024-05-09T09:04:23.238 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:22 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:22.997+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:23.238 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:22 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:22.997+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:23.239 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:22 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:22.888+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:23.239 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:22 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:22.889+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:23.239 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:22 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:22.889+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:23.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:23 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:23.242+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:23.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:23 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:23.242+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:23.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:23 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:23.242+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:23.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:23 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:23.324+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:23.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:23 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:23.324+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:23.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:23 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:23.532+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:23.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:23 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:23.532+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:23.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:23 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:23.532+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:24.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:23 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:23.836+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:24.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:23 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:23.836+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:24.088 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:23 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:23.836+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:52.134515+0000 front 2024-05-09T09:03:52.134537+0000 (oldest deadline 2024-05-09T09:04:17.434609+0000) 2024-05-09T09:04:24.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:23 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:23.993+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:24.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:23 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:23.993+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:24.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:23 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:23.915+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:24.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:23 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:23.915+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:24.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:23 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:23.915+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:24.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:24 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:24.286+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:24.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:24 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:24.286+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:24.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:24 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:24.286+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:24.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:24 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:24.365+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:24.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:24 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:24.365+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:24.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:24 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:24.568+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:24.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:24 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:24.568+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:24.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:24 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:24.568+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:25.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:24 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:24.990+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:25.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:24 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:24.990+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:25.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:24 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:24.990+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.912823+0000 front 2024-05-09T09:03:59.912929+0000 (oldest deadline 2024-05-09T09:04:24.612821+0000) 2024-05-09T09:04:25.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:24 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:24.879+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:25.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:24 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:24.879+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:25.238 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:24 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:24.879+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:25.261 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:24 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:24.848+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.431968+0000 front 2024-05-09T09:03:33.432094+0000 (oldest deadline 2024-05-09T09:03:56.332000+0000) 2024-05-09T09:04:25.261 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:24 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:24.848+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:39.232859+0000 front 2024-05-09T09:03:39.232839+0000 (oldest deadline 2024-05-09T09:04:06.833485+0000) 2024-05-09T09:04:25.262 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:24 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6[58171]: 2024-05-09T09:04:24.848+0000 7fa84828d700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:52.134515+0000 front 2024-05-09T09:03:52.134537+0000 (oldest deadline 2024-05-09T09:04:17.434609+0000) 2024-05-09T09:04:25.538 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:25 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:25.263+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:25.539 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:25 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:25.263+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:25.539 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:25 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:25.263+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:25.539 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:25 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:25.263+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:25.598 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:25 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:25.328+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:25.598 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:25 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:25.328+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:25.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:25 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:25.599+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:25.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:25 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:25.599+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:25.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:25 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:25.599+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:26.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:26 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:26.003+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:26.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:26 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:26.003+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:26.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:26 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:26.003+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.912823+0000 front 2024-05-09T09:03:59.912929+0000 (oldest deadline 2024-05-09T09:04:24.612821+0000) 2024-05-09T09:04:26.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:25 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:25.880+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:26.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:25 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:25.880+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:26.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:25 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:25.880+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:26.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:25 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:25.880+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:26.421 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:26 smithi098 podman[105813]: 2024-05-09 09:04:26.153997401 +0000 UTC m=+5.609611440 container died 8fe42d775f1cf6fd994f1be9cb0c6bf76f5149431cbf6c3ed0e620d4df75c31d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-05-09T09:04:26.421 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:26 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:26.305+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:26.421 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:26 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:26.305+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:26.421 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:26 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:26.305+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:26.421 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:26 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:26.305+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:26.577 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:26 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:26.285+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:26.577 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:26 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:26.285+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:26.577 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:26 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:26.285+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:26.837 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:26 smithi098 podman[105813]: 2024-05-09 09:04:26.521378251 +0000 UTC m=+5.976992288 container cleanup 8fe42d775f1cf6fd994f1be9cb0c6bf76f5149431cbf6c3ed0e620d4df75c31d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_BRANCH=HEAD) 2024-05-09T09:04:26.837 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:26 smithi098 bash[105813]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6 2024-05-09T09:04:26.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:26 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:26.578+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:26.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:26 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:26.578+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:26.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:26 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:26.578+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:27.181 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:26 smithi098 podman[105843]: 2024-05-09 09:04:26.839430941 +0000 UTC m=+0.685961155 container remove 8fe42d775f1cf6fd994f1be9cb0c6bf76f5149431cbf6c3ed0e620d4df75c31d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux ) 2024-05-09T09:04:27.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:26 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:26.996+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:27.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:26 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:26.996+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:27.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:26 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:26.996+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.912823+0000 front 2024-05-09T09:03:59.912929+0000 (oldest deadline 2024-05-09T09:04:24.612821+0000) 2024-05-09T09:04:27.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:26 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:26.869+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:27.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:26 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:26.869+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:27.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:26 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:26.869+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:27.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:26 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:26.869+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:27.534 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:27 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:27.346+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:27.535 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:27 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:27.346+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:27.535 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:27 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:27.346+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:27.535 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:27 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:27.346+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:27.535 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:27 smithi098 podman[105935]: 2024-05-09 09:04:27.085662481 +0000 UTC m=+0.017934148 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:04:27.535 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:27 smithi098 podman[105935]: 2024-05-09 09:04:27.224041612 +0000 UTC m=+0.156313262 container create 25cb5ded06ba44e00c0fff8efc40bc7fbadda9dc079f8fe45b0d890912d19a57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-09T09:04:27.535 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:27 smithi098 podman[105935]: 2024-05-09 09:04:27.535620853 +0000 UTC m=+0.467892505 container init 25cb5ded06ba44e00c0fff8efc40bc7fbadda9dc079f8fe45b0d890912d19a57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-96805dc, io.buildah.version=1.33.5) 2024-05-09T09:04:27.551 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:27 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:27.260+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:27.551 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:27 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:27.260+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:27.551 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:27 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:27.260+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:27.828 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:27 smithi098 podman[105935]: 2024-05-09 09:04:27.539265284 +0000 UTC m=+0.471536936 container start 25cb5ded06ba44e00c0fff8efc40bc7fbadda9dc079f8fe45b0d890912d19a57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-09T09:04:27.828 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:27 smithi098 podman[105935]: 2024-05-09 09:04:27.618372207 +0000 UTC m=+0.550643853 container attach 25cb5ded06ba44e00c0fff8efc40bc7fbadda9dc079f8fe45b0d890912d19a57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6-deactivate, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240507) 2024-05-09T09:04:27.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:27 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:27.551+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:27.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:27 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:27.551+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:27.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:27 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:27.551+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:28.087 INFO:journalctl@ceph.osd.6.smithi098.stdout:May 09 09:04:27 smithi098 podman[105935]: 2024-05-09 09:04:27.829955301 +0000 UTC m=+0.762226939 container died 25cb5ded06ba44e00c0fff8efc40bc7fbadda9dc079f8fe45b0d890912d19a57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc) 2024-05-09T09:04:28.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:28 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:28.026+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:28.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:28 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:28.026+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:28.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:28 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:28.026+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.912823+0000 front 2024-05-09T09:03:59.912929+0000 (oldest deadline 2024-05-09T09:04:24.612821+0000) 2024-05-09T09:04:28.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:27 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:27.862+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:28.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:27 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:27.862+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:28.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:27 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:27.862+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:28.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:27 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:27.862+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:28.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:28 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:28.311+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:28.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:28 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:28.311+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:28.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:28 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:28.311+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:28.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:28 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:28.311+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:28.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.6.service' 2024-05-09T09:04:28.527 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:04:28.527 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-09T09:04:28.528 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-09T09:04:28.528 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.7 2024-05-09T09:04:28.570 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:28 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:28.210+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:28.570 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:28 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:28.210+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:28.570 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:28 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:28.210+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:28.830 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:28 smithi105 systemd[1]: Stopping Ceph osd.7 for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:04:28.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:28 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:28.571+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:28.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:28 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:28.571+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:28.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:28 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:28.571+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:29.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:28 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:28.831+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:29.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:28 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:28.831+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:29.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:28 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:28.831+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:29.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:28 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:28.831+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:29.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:28 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:28.842+0000 7fcc90d92700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:04:29.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:28 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:28.842+0000 7fcc90d92700 -1 osd.7 115 *** Got signal Terminated *** 2024-05-09T09:04:29.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:28 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:28.842+0000 7fcc90d92700 -1 osd.7 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:04:29.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:29 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:29.035+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:29.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:29 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:29.035+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:29.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:29 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:29.035+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.912823+0000 front 2024-05-09T09:03:59.912929+0000 (oldest deadline 2024-05-09T09:04:24.612821+0000) 2024-05-09T09:04:29.538 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:29 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:29.225+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:29.538 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:29 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:29.225+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:29.538 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:29 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:29.225+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:29.837 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:29 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:29.346+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:29.837 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:29 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:29.346+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:29.838 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:29 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:29.346+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:29.838 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:29 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:29.346+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:29.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:29 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:29.539+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:29.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:29 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:29.539+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:29.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:29 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:29.539+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:30.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:30 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:30.067+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:30.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:30 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:30.067+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:30.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:30 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:30.067+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.912823+0000 front 2024-05-09T09:03:59.912929+0000 (oldest deadline 2024-05-09T09:04:24.612821+0000) 2024-05-09T09:04:30.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:29 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:29.866+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:30.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:29 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:29.866+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:30.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:29 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:29.866+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:30.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:29 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:29.866+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:30.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:30 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:30.301+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:30.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:30 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:30.301+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:30.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:30 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:30.301+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:30.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:30 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:30.301+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:30.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:30 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:30.301+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:30.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:30 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:30.268+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:30.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:30 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:30.268+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:30.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:30 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:30.268+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:30.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:30 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:30.501+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:30.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:30 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:30.501+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:30.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:30 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:30.501+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:30.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:30 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:30.501+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:31.084 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:30 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:30.821+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:31.084 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:30 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:30.821+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:31.084 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:30 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:30.821+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:31.084 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:30 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:30.821+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:31.084 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:30 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:30.821+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:31.486 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:31.085+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:31.486 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:31.085+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:31.487 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:31.085+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.912823+0000 front 2024-05-09T09:03:59.912929+0000 (oldest deadline 2024-05-09T09:04:24.612821+0000) 2024-05-09T09:04:31.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:31.515+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:31.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:31.515+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:31.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:31.515+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:31.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:31.515+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:31.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:31.269+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:31.659 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:31.269+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:31.659 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:31.269+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:31.659 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:31 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:31.269+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:08.464406+0000 front 2024-05-09T09:04:08.464306+0000 (oldest deadline 2024-05-09T09:04:30.764249+0000) 2024-05-09T09:04:31.837 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:31 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:31.345+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:31.838 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:31 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:31.345+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:31.838 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:31 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:31.345+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:31.838 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:31 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:31.345+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:31.838 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:31 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:31.345+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:32.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:32 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:32.052+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:32.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:32 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:32.052+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:32.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:32 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:32.052+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.912823+0000 front 2024-05-09T09:03:59.912929+0000 (oldest deadline 2024-05-09T09:04:24.612821+0000) 2024-05-09T09:04:32.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:31.818+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:32.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:31.818+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:32.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:31.818+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:32.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:31.818+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:32.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:31 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:31.818+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:32.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:32 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:32.301+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:32.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:32 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:32.301+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:32.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:32 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:32.301+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:32.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:32 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:32.301+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:32.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:32 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:32.301+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:32.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:32 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:32.261+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:32.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:32 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:32.261+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:32.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:32 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:32.261+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:32.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:32 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:32.261+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:08.464406+0000 front 2024-05-09T09:04:08.464306+0000 (oldest deadline 2024-05-09T09:04:30.764249+0000) 2024-05-09T09:04:32.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:32 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:32.480+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:32.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:32 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:32.480+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:32.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:32 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:32.480+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:32.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:32 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:32.480+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:33.051 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:32 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:32.778+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:33.052 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:32 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:32.778+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:33.052 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:32 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:32.778+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:33.052 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:32 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:32.778+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:33.052 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:32 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:32.778+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:33.486 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:33.053+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:43.407785+0000 front 2024-05-09T09:03:43.407815+0000 (oldest deadline 2024-05-09T09:04:05.107956+0000) 2024-05-09T09:04:33.486 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:33.053+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.509545+0000 front 2024-05-09T09:03:51.509316+0000 (oldest deadline 2024-05-09T09:04:14.409538+0000) 2024-05-09T09:04:33.487 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:33.053+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.912823+0000 front 2024-05-09T09:03:59.912929+0000 (oldest deadline 2024-05-09T09:04:24.612821+0000) 2024-05-09T09:04:33.487 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7[52596]: 2024-05-09T09:04:33.053+0000 7fcc8d386700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:08.614044+0000 front 2024-05-09T09:04:08.614195+0000 (oldest deadline 2024-05-09T09:04:32.114273+0000) 2024-05-09T09:04:33.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:33 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:33.289+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:33.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:33 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:33.289+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:33.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:33 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:33.289+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:33.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:33 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:33.289+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:33.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:33 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:33.289+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:33.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:33 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:33.223+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:33.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:33 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:33.223+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:33.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:33 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:33.223+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:33.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:33 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:33.223+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:08.464406+0000 front 2024-05-09T09:04:08.464306+0000 (oldest deadline 2024-05-09T09:04:30.764249+0000) 2024-05-09T09:04:33.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:33 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:33.448+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:33.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:33 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:33.448+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:33.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:33 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:33.448+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:33.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:33 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:33.448+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:33.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:33.733+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:33.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:33.733+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:33.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:33.733+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:33.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:33.733+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:33.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:33.733+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:34.459 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:34 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:34.206+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:34.459 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:34 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:34.206+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:34.459 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:34 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:34.206+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:34.459 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:34 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:34.206+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:08.464406+0000 front 2024-05-09T09:04:08.464306+0000 (oldest deadline 2024-05-09T09:04:30.764249+0000) 2024-05-09T09:04:34.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:34 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:34.303+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:34.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:34 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:34.303+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:34.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:34 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:34.303+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:34.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:34 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:34.303+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:34.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:34 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:34.303+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:34.624 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:34 smithi105 podman[71678]: 2024-05-09 09:04:34.33652061 +0000 UTC m=+5.609946712 container died 8be93cb580d56b0d9eeb446379c62d7db3365445396772a567371df123413942 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507) 2024-05-09T09:04:34.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:34 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:34.459+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:34.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:34 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:34.459+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:34.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:34 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:34.459+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:34.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:34 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:34.459+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:34.986 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:34 smithi105 podman[71678]: 2024-05-09 09:04:34.726133103 +0000 UTC m=+5.999559202 container cleanup 8be93cb580d56b0d9eeb446379c62d7db3365445396772a567371df123413942 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-96805dc, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:04:34.986 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:34 smithi105 bash[71678]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7 2024-05-09T09:04:34.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:34 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:34.704+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:34.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:34 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:34.704+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:34.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:34 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:34.704+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:34.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:34 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:34.704+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:34.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:34 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:34.704+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:35.333 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:34 smithi105 podman[71696]: 2024-05-09 09:04:34.994218435 +0000 UTC m=+0.654725318 container remove 8be93cb580d56b0d9eeb446379c62d7db3365445396772a567371df123413942 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-09T09:04:35.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:35 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:35.323+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:35.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:35 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:35.323+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:35.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:35 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:35.323+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:35.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:35 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:35.323+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:35.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:35 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:35.323+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:35.596 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:35 smithi105 podman[71789]: 2024-05-09 09:04:35.237902601 +0000 UTC m=+0.017922723 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:04:35.596 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:35 smithi105 podman[71789]: 2024-05-09 09:04:35.377872646 +0000 UTC m=+0.157892810 container create 4f1241e16b4ca1aff2c97da050a9edb1db2daf017e013347148210d36c9a1a87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-09T09:04:35.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:35 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:35.197+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:35.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:35 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:35.197+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:35.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:35 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:35.197+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:35.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:35 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:35.197+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:08.464406+0000 front 2024-05-09T09:04:08.464306+0000 (oldest deadline 2024-05-09T09:04:30.764249+0000) 2024-05-09T09:04:35.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:35 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:35.437+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:35.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:35 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:35.437+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:35.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:35 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:35.437+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:35.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:35 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:35.437+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:35.889 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:35 smithi105 podman[71789]: 2024-05-09 09:04:35.597815339 +0000 UTC m=+0.377835461 container init 4f1241e16b4ca1aff2c97da050a9edb1db2daf017e013347148210d36c9a1a87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-09T09:04:35.890 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:35 smithi105 podman[71789]: 2024-05-09 09:04:35.601044465 +0000 UTC m=+0.381064587 container start 4f1241e16b4ca1aff2c97da050a9edb1db2daf017e013347148210d36c9a1a87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-96805dc) 2024-05-09T09:04:35.890 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:35 smithi105 podman[71789]: 2024-05-09 09:04:35.697750747 +0000 UTC m=+0.477770878 container attach 4f1241e16b4ca1aff2c97da050a9edb1db2daf017e013347148210d36c9a1a87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-05-09T09:04:35.890 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:35 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:35.714+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:35.890 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:35 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:35.714+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:35.890 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:35 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:35.714+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:35.890 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:35 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:35.714+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:35.890 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:35 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:35.714+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:36.236 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 09 09:04:35 smithi105 podman[71789]: 2024-05-09 09:04:35.891407347 +0000 UTC m=+0.671427464 container died 4f1241e16b4ca1aff2c97da050a9edb1db2daf017e013347148210d36c9a1a87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-7-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:04:36.560 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:36.231+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:36.560 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:36.231+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:36.560 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:36.231+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:36.561 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:36.231+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:08.464406+0000 front 2024-05-09T09:04:08.464306+0000 (oldest deadline 2024-05-09T09:04:30.764249+0000) 2024-05-09T09:04:36.561 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:36.398+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:36.561 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:36.398+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:36.561 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:36.398+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:36.561 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:36.398+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:36.561 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:36 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:36.294+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:36.561 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:36 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:36.294+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:36.561 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:36 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:36.294+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:36.562 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:36 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:36.294+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:36.562 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:36 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:36.294+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:36.603 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.7.service' 2024-05-09T09:04:36.643 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:04:36.643 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-09T09:04:36.643 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-09T09:04:36.644 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.8 2024-05-09T09:04:36.862 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:36 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:36.728+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:36.862 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:36 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:36.728+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:36.862 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:36 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:36.728+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:36.862 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:36 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:36.728+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:36.862 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:36 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:36.728+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:36.908 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:36 smithi142 systemd[1]: Stopping Ceph osd.8 for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:04:37.189 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:36.949+0000 7fb2fe810700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:04:37.189 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:36.949+0000 7fb2fe810700 -1 osd.8 115 *** Got signal Terminated *** 2024-05-09T09:04:37.189 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:36 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:36.949+0000 7fb2fe810700 -1 osd.8 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:04:37.189 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:37 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:37.189+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:37.189 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:37 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:37.189+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:37.189 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:37 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:37.189+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:37.189 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:37 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:37.189+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:08.464406+0000 front 2024-05-09T09:04:08.464306+0000 (oldest deadline 2024-05-09T09:04:30.764249+0000) 2024-05-09T09:04:37.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:37 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:37.305+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:37.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:37 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:37.305+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:37.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:37 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:37.305+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:37.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:37 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:37.305+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:37.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:37 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:37.305+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:37.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:37 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:37.370+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:37.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:37 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:37.370+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:37.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:37 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:37.370+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:37.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:37 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:37.370+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:37.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:37 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:37.688+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:37.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:37 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:37.688+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:37.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:37 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:37.688+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:37.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:37 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:37.688+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:37.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:37 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:37.688+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:38.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:38.331+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:38.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:38.331+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:38.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:38.331+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:38.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:38.331+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:38.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:38 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:38.331+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:38.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:38 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:38.202+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:38.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:38 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:38.202+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:38.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:38 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:38.202+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:38.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:38 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:38.202+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:08.464406+0000 front 2024-05-09T09:04:08.464306+0000 (oldest deadline 2024-05-09T09:04:30.764249+0000) 2024-05-09T09:04:38.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:38 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:38.403+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:38.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:38 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:38.403+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:38.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:38 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:38.403+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:38.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:38 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:38.403+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:38.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:38 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:38.403+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:38.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:38 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:38.675+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:38.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:38 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:38.675+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:38.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:38 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:38.675+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:38.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:38 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:38.675+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:38.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:38 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:38.675+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:39.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:39 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:39.298+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:39.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:39 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:39.298+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:39.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:39 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:39.298+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:39.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:39 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:39.298+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:39.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:39 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:39.298+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:39.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:39 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:39.205+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:39.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:39 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:39.205+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:39.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:39 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:39.205+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:39.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:39 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:39.205+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:08.464406+0000 front 2024-05-09T09:04:08.464306+0000 (oldest deadline 2024-05-09T09:04:30.764249+0000) 2024-05-09T09:04:39.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:39 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:39.442+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:39.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:39 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:39.442+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:39.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:39 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:39.442+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:39.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:39 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:39.442+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:39.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:39 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:39.442+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:39.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:39 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:39.717+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:39.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:39 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:39.717+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:39.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:39 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:39.717+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:39.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:39 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:39.717+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:39.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:39 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:39.717+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:39.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:39 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:39.717+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:40.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:40 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:40.251+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:40.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:40 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:40.251+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:40.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:40 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:40.251+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:40.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:40 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:40.251+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:40.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:40 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:40.251+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:40.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:40 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:40.403+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:40.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:40 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:40.403+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:40.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:40 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:40.403+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:40.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:40 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:40.403+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:40.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:40 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:40.403+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:40.659 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:40 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:40.163+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:40.659 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:40 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:40.163+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:40.659 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:40 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:40.163+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:40.659 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:40 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:40.163+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:08.464406+0000 front 2024-05-09T09:04:08.464306+0000 (oldest deadline 2024-05-09T09:04:30.764249+0000) 2024-05-09T09:04:40.659 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:40 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:40.163+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.165802+0000 front 2024-05-09T09:04:14.165772+0000 (oldest deadline 2024-05-09T09:04:40.065938+0000) 2024-05-09T09:04:41.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:40 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:40.750+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:41.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:40 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:40.750+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:41.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:40 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:40.750+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:41.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:40 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:40.750+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:41.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:40 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:40.750+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:41.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:40 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:40.750+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:41.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:41 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:41.253+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:41.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:41 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:41.253+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:41.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:41 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:41.253+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:41.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:41 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:41.253+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:41.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:41 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:41.253+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:41.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:41 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:41.253+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.147738+0000 front 2024-05-09T09:04:16.147684+0000 (oldest deadline 2024-05-09T09:04:40.847456+0000) 2024-05-09T09:04:41.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:41 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:41.179+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:42.061041+0000 front 2024-05-09T09:03:42.061161+0000 (oldest deadline 2024-05-09T09:04:07.361029+0000) 2024-05-09T09:04:41.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:41 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:41.179+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:47.361369+0000 front 2024-05-09T09:03:47.361428+0000 (oldest deadline 2024-05-09T09:04:13.261473+0000) 2024-05-09T09:04:41.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:41 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:41.179+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:56.762363+0000 front 2024-05-09T09:03:56.762337+0000 (oldest deadline 2024-05-09T09:04:25.562986+0000) 2024-05-09T09:04:41.659 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:41 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:41.179+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:08.464406+0000 front 2024-05-09T09:04:08.464306+0000 (oldest deadline 2024-05-09T09:04:30.764249+0000) 2024-05-09T09:04:41.659 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:41 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8[52842]: 2024-05-09T09:04:41.179+0000 7fb2fae04700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.165802+0000 front 2024-05-09T09:04:14.165772+0000 (oldest deadline 2024-05-09T09:04:40.065938+0000) 2024-05-09T09:04:41.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:41 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:41.380+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:41.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:41 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:41.380+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:41.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:41 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:41.380+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:41.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:41 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:41.380+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:41.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:41 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:41.380+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:42.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:41 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:41.768+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:42.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:41 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:41.768+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:42.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:41 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:41.768+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:42.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:41 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:41.768+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:42.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:41 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:41.768+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:42.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:41 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:41.768+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:42.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:42 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:42.203+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:42.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:42 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:42.203+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:42.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:42 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:42.203+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:42.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:42 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:42.203+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:42.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:42 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:42.203+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:42.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:42 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:42.203+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.147738+0000 front 2024-05-09T09:04:16.147684+0000 (oldest deadline 2024-05-09T09:04:40.847456+0000) 2024-05-09T09:04:42.658 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:42 smithi142 podman[71306]: 2024-05-09 09:04:42.612917554 +0000 UTC m=+5.777044749 container died 9916b511a5680035b4de35b5ff476c35980eec7a3ec5e35bcef8d0a4e972df95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc) 2024-05-09T09:04:42.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:42 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:42.406+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:42.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:42 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:42.406+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:42.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:42 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:42.406+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:42.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:42 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:42.406+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:42.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:42 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:42.406+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:43.158 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:42 smithi142 podman[71306]: 2024-05-09 09:04:42.95012073 +0000 UTC m=+6.114247931 container cleanup 9916b511a5680035b4de35b5ff476c35980eec7a3ec5e35bcef8d0a4e972df95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:04:43.158 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:42 smithi142 bash[71306]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8 2024-05-09T09:04:43.236 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:42 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:42.767+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:43.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:42 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:42.767+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:43.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:42 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:42.767+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:43.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:42 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:42.767+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:43.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:42 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:42.767+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:43.237 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:42 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:42.767+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:43.448 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:43 smithi142 podman[71335]: 2024-05-09 09:04:43.168200399 +0000 UTC m=+0.548866000 container remove 9916b511a5680035b4de35b5ff476c35980eec7a3ec5e35bcef8d0a4e972df95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-09T09:04:43.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:43 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:43.215+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:43.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:43 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:43.215+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:43.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:43 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:43.215+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:43.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:43 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:43.215+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:43.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:43 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:43.215+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:43.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:43 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:43.215+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.147738+0000 front 2024-05-09T09:04:16.147684+0000 (oldest deadline 2024-05-09T09:04:40.847456+0000) 2024-05-09T09:04:43.754 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:43 smithi142 podman[71420]: 2024-05-09 09:04:43.510294547 +0000 UTC m=+0.018162694 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:04:43.754 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:43 smithi142 podman[71420]: 2024-05-09 09:04:43.627374874 +0000 UTC m=+0.135243013 container create 9bfe0f71f6dd26f888903b35313cefb5b2137191dde3fc3a8e7aa5455f98a072 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:04:43.754 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:43 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:43.449+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:43.754 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:43 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:43.449+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:43.754 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:43 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:43.449+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:43.755 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:43 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:43.449+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:43.755 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:43 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:43.449+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:43.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:43 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:43.719+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:43.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:43 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:43.719+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:43.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:43 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:43.719+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:43.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:43 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:43.719+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:43.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:43 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:43.719+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:43.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:43 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:43.719+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:44.158 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:43 smithi142 podman[71420]: 2024-05-09 09:04:43.896673147 +0000 UTC m=+0.404541294 container init 9bfe0f71f6dd26f888903b35313cefb5b2137191dde3fc3a8e7aa5455f98a072 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-96805dc) 2024-05-09T09:04:44.158 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:43 smithi142 podman[71420]: 2024-05-09 09:04:43.900474822 +0000 UTC m=+0.408342961 container start 9bfe0f71f6dd26f888903b35313cefb5b2137191dde3fc3a8e7aa5455f98a072 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-96805dc) 2024-05-09T09:04:44.158 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:43 smithi142 podman[71420]: 2024-05-09 09:04:43.9798268 +0000 UTC m=+0.487694938 container attach 9bfe0f71f6dd26f888903b35313cefb5b2137191dde3fc3a8e7aa5455f98a072 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-96805dc, GIT_BRANCH=HEAD) 2024-05-09T09:04:44.458 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:44 smithi142 podman[71420]: 2024-05-09 09:04:44.193092957 +0000 UTC m=+0.700961087 container died 9bfe0f71f6dd26f888903b35313cefb5b2137191dde3fc3a8e7aa5455f98a072 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-09T09:04:44.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:44 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:44.234+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:44.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:44 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:44.234+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:44.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:44 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:44.234+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:44.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:44 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:44.234+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:44.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:44 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:44.234+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:44.589 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:44 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:44.234+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.147738+0000 front 2024-05-09T09:04:16.147684+0000 (oldest deadline 2024-05-09T09:04:40.847456+0000) 2024-05-09T09:04:44.709 INFO:journalctl@ceph.osd.8.smithi142.stdout:May 09 09:04:44 smithi142 podman[71445]: 2024-05-09 09:04:44.708172695 +0000 UTC m=+0.507776445 container remove 9bfe0f71f6dd26f888903b35313cefb5b2137191dde3fc3a8e7aa5455f98a072 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc) 2024-05-09T09:04:44.709 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:44 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:44.459+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:44.709 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:44 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:44.459+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:44.709 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:44 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:44.459+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:44.710 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:44 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:44.459+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:44.710 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:44 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:44.459+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:44.727 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.8.service' 2024-05-09T09:04:44.766 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:04:44.766 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-09T09:04:44.766 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-09T09:04:44.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.9 2024-05-09T09:04:44.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:44 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:44.729+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:44.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:44 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:44.729+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:44.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:44 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:44.729+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:44.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:44 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:44.729+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:44.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:44 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:44.729+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:44.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:44 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:44.729+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:45.082 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:44 smithi098 systemd[1]: Stopping Ceph osd.9 for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:04:45.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:45 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:45.083+0000 7f346f8ac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:04:45.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:45 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:45.083+0000 7f346f8ac700 -1 osd.9 115 *** Got signal Terminated *** 2024-05-09T09:04:45.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:45 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:45.083+0000 7f346f8ac700 -1 osd.9 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:04:45.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:45 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:45.190+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:45.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:45 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:45.190+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:45.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:45 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:45.190+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:45.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:45 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:45.190+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:45.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:45 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:45.190+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:45.338 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:45 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:45.190+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.147738+0000 front 2024-05-09T09:04:16.147684+0000 (oldest deadline 2024-05-09T09:04:40.847456+0000) 2024-05-09T09:04:45.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:45 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:45.431+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:45.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:45 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:45.431+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:45.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:45 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:45.431+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:45.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:45 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:45.431+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:45.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:45 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:45.431+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:45.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:45 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:45.694+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:45.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:45 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:45.694+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:45.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:45 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:45.694+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:45.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:45 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:45.694+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:45.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:45 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:45.694+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:45.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:45 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:45.694+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:46.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:46 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:46.207+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:46.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:46 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:46.207+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:46.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:46 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:46.207+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:46.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:46 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:46.207+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:46.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:46 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:46.207+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:46.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:46 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:46.207+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.147738+0000 front 2024-05-09T09:04:16.147684+0000 (oldest deadline 2024-05-09T09:04:40.847456+0000) 2024-05-09T09:04:46.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:46 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:46.418+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:46.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:46 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:46.418+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:46.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:46 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:46.418+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:46.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:46 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:46.418+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:46.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:46 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:46.418+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:46.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:46 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:46.693+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:46.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:46 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:46.693+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:46.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:46 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:46.693+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:46.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:46 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:46.693+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:46.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:46 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:46.693+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:46.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:46 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:46.693+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:46.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:46 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:46.693+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:21.675820+0000 front 2024-05-09T09:04:21.675780+0000 (oldest deadline 2024-05-09T09:04:46.375698+0000) 2024-05-09T09:04:47.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:47 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:47.191+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:47.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:47 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:47.191+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:47.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:47 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:47.191+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:47.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:47 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:47.191+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:47.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:47 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:47.191+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:47.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:47 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:47.191+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.147738+0000 front 2024-05-09T09:04:16.147684+0000 (oldest deadline 2024-05-09T09:04:40.847456+0000) 2024-05-09T09:04:47.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:47 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:47.394+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:47.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:47 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:47.394+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:47.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:47 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:47.394+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:47.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:47 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:47.394+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:47.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:47 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:47.394+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:47.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:47 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:47.658+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:47.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:47 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:47.658+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:47.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:47 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:47.658+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:47.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:47 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:47.658+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:47.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:47 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:47.658+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:47.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:47 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:47.658+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:47.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:47 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:47.658+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:21.675820+0000 front 2024-05-09T09:04:21.675780+0000 (oldest deadline 2024-05-09T09:04:46.375698+0000) 2024-05-09T09:04:48.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:48 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:48.143+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:48.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:48 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:48.143+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:48.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:48 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:48.143+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:48.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:48 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:48.143+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:48.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:48 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:48.143+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:48.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:48 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:48.143+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.147738+0000 front 2024-05-09T09:04:16.147684+0000 (oldest deadline 2024-05-09T09:04:40.847456+0000) 2024-05-09T09:04:48.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:48 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:48.373+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:48.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:48 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:48.373+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:48.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:48 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:48.373+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:48.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:48 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:48.373+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:48.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:48 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:48.373+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:48.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:48 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:48.689+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:49.036 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:48 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:48.689+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:49.036 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:48 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:48.689+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:49.037 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:48 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:48.689+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:49.037 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:48 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:48.689+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:49.037 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:48 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:48.689+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:49.037 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:48 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:48.689+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:21.675820+0000 front 2024-05-09T09:04:21.675780+0000 (oldest deadline 2024-05-09T09:04:46.375698+0000) 2024-05-09T09:04:49.587 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:49 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:49.151+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.840592+0000 front 2024-05-09T09:03:34.840716+0000 (oldest deadline 2024-05-09T09:03:58.940978+0000) 2024-05-09T09:04:49.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:49 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:49.151+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:40.542902+0000 front 2024-05-09T09:03:40.542840+0000 (oldest deadline 2024-05-09T09:04:05.743124+0000) 2024-05-09T09:04:49.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:49 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:49.151+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.044284+0000 front 2024-05-09T09:03:51.044202+0000 (oldest deadline 2024-05-09T09:04:13.944338+0000) 2024-05-09T09:04:49.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:49 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:49.151+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.245142+0000 front 2024-05-09T09:03:59.245202+0000 (oldest deadline 2024-05-09T09:04:24.545384+0000) 2024-05-09T09:04:49.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:49 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:49.151+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:07.446864+0000 front 2024-05-09T09:04:07.447034+0000 (oldest deadline 2024-05-09T09:04:29.746377+0000) 2024-05-09T09:04:49.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:49 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9[65178]: 2024-05-09T09:04:49.151+0000 7f346bea0700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.147738+0000 front 2024-05-09T09:04:16.147684+0000 (oldest deadline 2024-05-09T09:04:40.847456+0000) 2024-05-09T09:04:49.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:49 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:49.388+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:49.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:49 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:49.388+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:49.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:49 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:49.388+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:49.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:49 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:49.388+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:49.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:49 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:49.388+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:49.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:49 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:49.388+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:04:49.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:49 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:49.685+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:49.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:49 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:49.685+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:49.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:49 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:49.685+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:49.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:49 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:49.685+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:49.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:49 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:49.685+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:49.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:49 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:49.685+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:49.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:49 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:49.685+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:21.675820+0000 front 2024-05-09T09:04:21.675780+0000 (oldest deadline 2024-05-09T09:04:46.375698+0000) 2024-05-09T09:04:50.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:50 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:50.405+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:50.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:50 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:50.405+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:50.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:50 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:50.405+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:50.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:50 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:50.405+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:50.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:50 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:50.405+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:50.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:50 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:50.405+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:04:50.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:50 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:50.660+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:50.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:50 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:50.660+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:50.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:50 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:50.660+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:50.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:50 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:50.660+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:50.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:50 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:50.660+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:50.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:50 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:50.660+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:50.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:50 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:50.660+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:21.675820+0000 front 2024-05-09T09:04:21.675780+0000 (oldest deadline 2024-05-09T09:04:46.375698+0000) 2024-05-09T09:04:51.115 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:50 smithi098 podman[106039]: 2024-05-09 09:04:50.84656434 +0000 UTC m=+5.882748679 container died d7410066843682ab0292bb0cdbebcfde7453b4c47c066ad13a8786e86e2f6c07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:04:51.424 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:51 smithi098 podman[106039]: 2024-05-09 09:04:51.191748139 +0000 UTC m=+6.227932474 container cleanup d7410066843682ab0292bb0cdbebcfde7453b4c47c066ad13a8786e86e2f6c07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-05-09T09:04:51.425 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:51 smithi098 bash[106039]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9 2024-05-09T09:04:51.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:51 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:51.396+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:51.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:51 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:51.396+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:51.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:51 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:51.396+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:51.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:51 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:51.396+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:51.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:51 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:51.396+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:51.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:51 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:51.396+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:04:51.837 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:51 smithi098 podman[106056]: 2024-05-09 09:04:51.426334139 +0000 UTC m=+0.574652591 container remove d7410066843682ab0292bb0cdbebcfde7453b4c47c066ad13a8786e86e2f6c07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-05-09T09:04:51.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:51 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:51.635+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:51.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:51 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:51.635+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:51.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:51 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:51.635+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:51.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:51 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:51.635+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:51.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:51 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:51.635+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:51.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:51 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:51.635+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:51.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:51 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:51.635+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:21.675820+0000 front 2024-05-09T09:04:21.675780+0000 (oldest deadline 2024-05-09T09:04:46.375698+0000) 2024-05-09T09:04:52.153 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:51 smithi098 podman[106142]: 2024-05-09 09:04:51.746313107 +0000 UTC m=+0.018712291 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:04:52.153 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:51 smithi098 podman[106142]: 2024-05-09 09:04:51.868900178 +0000 UTC m=+0.141299390 container create 029baa109af40f085ecf7bcec1f057cf05e15f3dab21a071e4b5b50fda9e4361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-96805dc) 2024-05-09T09:04:52.153 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:52 smithi098 podman[106142]: 2024-05-09 09:04:52.071822501 +0000 UTC m=+0.344221681 container init 029baa109af40f085ecf7bcec1f057cf05e15f3dab21a071e4b5b50fda9e4361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-96805dc) 2024-05-09T09:04:52.153 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:52 smithi098 podman[106142]: 2024-05-09 09:04:52.07533656 +0000 UTC m=+0.347735730 container start 029baa109af40f085ecf7bcec1f057cf05e15f3dab21a071e4b5b50fda9e4361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-09T09:04:52.154 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:52 smithi098 podman[106142]: 2024-05-09 09:04:52.154780519 +0000 UTC m=+0.427179702 container attach 029baa109af40f085ecf7bcec1f057cf05e15f3dab21a071e4b5b50fda9e4361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:04:52.588 INFO:journalctl@ceph.osd.9.smithi098.stdout:May 09 09:04:52 smithi098 podman[106142]: 2024-05-09 09:04:52.367470399 +0000 UTC m=+0.639869569 container died 029baa109af40f085ecf7bcec1f057cf05e15f3dab21a071e4b5b50fda9e4361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-96805dc) 2024-05-09T09:04:52.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:52 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:52.367+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:52.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:52 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:52.367+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:52.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:52 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:52.367+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:52.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:52 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:52.367+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:52.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:52 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:52.367+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:52.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:52 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:52.367+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:04:52.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.9.service' 2024-05-09T09:04:52.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:52 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:52.662+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:52.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:52 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:52.662+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:52.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:52 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:52.662+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:52.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:52 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:52.662+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:52.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:52 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:52.662+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:52.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:52 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:52.662+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:52.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:52 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:52.662+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:21.675820+0000 front 2024-05-09T09:04:21.675780+0000 (oldest deadline 2024-05-09T09:04:46.375698+0000) 2024-05-09T09:04:52.991 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:04:52.992 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-09T09:04:52.992 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-09T09:04:52.992 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.10 2024-05-09T09:04:53.303 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:53 smithi105 systemd[1]: Stopping Ceph osd.10 for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:04:53.630 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:53 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:53.303+0000 7efcc4375700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:04:53.630 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:53 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:53.303+0000 7efcc4375700 -1 osd.10 115 *** Got signal Terminated *** 2024-05-09T09:04:53.630 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:53 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:53.303+0000 7efcc4375700 -1 osd.10 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:04:53.630 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:53 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:53.630+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:53.631 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:53 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:53.630+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:53.631 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:53 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:53.630+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:53.631 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:53 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:53.630+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:53.631 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:53 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:53.630+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:53.631 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:53 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:53.630+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:53.631 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:53 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:53.630+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:21.675820+0000 front 2024-05-09T09:04:21.675780+0000 (oldest deadline 2024-05-09T09:04:46.375698+0000) 2024-05-09T09:04:53.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:53 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:53.324+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:53.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:53 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:53.324+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:53.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:53 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:53.324+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:53.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:53 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:53.324+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:53.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:53 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:53.324+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:53.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:53 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:53.324+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:04:54.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:54 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:54.292+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:54.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:54 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:54.292+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:54.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:54 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:54.292+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:54.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:54 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:54.292+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:54.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:54 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:54.292+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:54.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:54 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:54.292+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:04:54.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:54 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:54.587+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:54.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:54 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:54.587+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:54.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:54 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:54.587+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:54.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:54 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:54.587+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:54.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:54 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:54.587+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:54.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:54 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:54.587+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:54.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:54 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:54.587+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:21.675820+0000 front 2024-05-09T09:04:21.675780+0000 (oldest deadline 2024-05-09T09:04:46.375698+0000) 2024-05-09T09:04:55.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:55 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:55.270+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:55.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:55 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:55.270+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:55.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:55 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:55.270+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:55.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:55 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:55.270+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:55.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:55 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:55.270+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:55.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:55 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:55.270+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:04:55.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:55 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:55.270+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-09T09:04:33.787453+0000 front 2024-05-09T09:04:33.787519+0000 (oldest deadline 2024-05-09T09:04:54.887392+0000) 2024-05-09T09:04:55.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:55 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:55.562+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:55.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:55 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:55.562+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:55.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:55 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:55.562+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:55.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:55 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:55.562+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:55.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:55 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:55.562+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:55.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:55 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:55.562+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:55.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:55 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:55.562+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:21.675820+0000 front 2024-05-09T09:04:21.675780+0000 (oldest deadline 2024-05-09T09:04:46.375698+0000) 2024-05-09T09:04:56.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:56 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:56.291+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:56.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:56 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:56.291+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:56.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:56 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:56.291+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:56.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:56 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:56.291+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:56.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:56 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:56.291+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:56.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:56 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:56.291+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:04:56.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:56 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:56.291+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-09T09:04:33.787453+0000 front 2024-05-09T09:04:33.787519+0000 (oldest deadline 2024-05-09T09:04:54.887392+0000) 2024-05-09T09:04:56.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:56 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:56.556+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:56.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:56 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:56.556+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:56.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:56 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:56.556+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:56.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:56 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:56.556+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:56.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:56 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:56.556+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:56.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:56 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:56.556+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:56.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:56 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:56.556+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:21.675820+0000 front 2024-05-09T09:04:21.675780+0000 (oldest deadline 2024-05-09T09:04:46.375698+0000) 2024-05-09T09:04:57.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:57 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:57.330+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:57.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:57 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:57.330+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:57.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:57 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:57.330+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:57.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:57 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:57.330+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:57.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:57 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:57.330+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:57.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:57 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:57.330+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:04:57.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:57 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:57.330+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-09T09:04:33.787453+0000 front 2024-05-09T09:04:33.787519+0000 (oldest deadline 2024-05-09T09:04:54.887392+0000) 2024-05-09T09:04:57.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:57 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:57.570+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:33.567023+0000 front 2024-05-09T09:03:33.566928+0000 (oldest deadline 2024-05-09T09:03:57.666722+0000) 2024-05-09T09:04:57.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:57 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:57.570+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-09T09:03:41.068475+0000 front 2024-05-09T09:03:41.068498+0000 (oldest deadline 2024-05-09T09:04:06.368616+0000) 2024-05-09T09:04:57.986 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:57 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:57.571+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:50.870793+0000 front 2024-05-09T09:03:50.870931+0000 (oldest deadline 2024-05-09T09:04:16.071208+0000) 2024-05-09T09:04:57.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:57 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:57.571+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:59.973318+0000 front 2024-05-09T09:03:59.973430+0000 (oldest deadline 2024-05-09T09:04:25.273182+0000) 2024-05-09T09:04:57.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:57 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:57.571+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.273547+0000 front 2024-05-09T09:04:05.273567+0000 (oldest deadline 2024-05-09T09:04:30.573530+0000) 2024-05-09T09:04:57.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:57 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:57.571+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:14.674776+0000 front 2024-05-09T09:04:14.674831+0000 (oldest deadline 2024-05-09T09:04:39.374682+0000) 2024-05-09T09:04:57.987 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:57 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10[59445]: 2024-05-09T09:04:57.571+0000 7efcc0969700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:21.675820+0000 front 2024-05-09T09:04:21.675780+0000 (oldest deadline 2024-05-09T09:04:46.375698+0000) 2024-05-09T09:04:58.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:58 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:58.354+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:58.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:58 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:58.354+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:58.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:58 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:58.354+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:58.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:58 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:58.354+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:58.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:58 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:58.354+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:58.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:58 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:58.354+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:04:58.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:58 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:58.354+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-09T09:04:33.787453+0000 front 2024-05-09T09:04:33.787519+0000 (oldest deadline 2024-05-09T09:04:54.887392+0000) 2024-05-09T09:04:59.161 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:58 smithi105 podman[71909]: 2024-05-09 09:04:58.875899974 +0000 UTC m=+5.693744597 container died d46657d58b9656cd16aa8e2707bee2eaf8cc333642f6ae335fa9a6b7ed7da1d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:04:59.486 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:59 smithi105 podman[71909]: 2024-05-09 09:04:59.23736006 +0000 UTC m=+6.055204680 container cleanup d46657d58b9656cd16aa8e2707bee2eaf8cc333642f6ae335fa9a6b7ed7da1d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5) 2024-05-09T09:04:59.486 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:59 smithi105 bash[71909]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10 2024-05-09T09:04:59.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:59 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:59.327+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:04:59.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:59 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:59.327+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:04:59.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:59 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:59.327+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:04:59.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:59 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:59.327+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:04:59.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:59 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:59.327+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:04:59.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:59 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:59.327+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:04:59.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:04:59 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:04:59.327+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-09T09:04:33.787453+0000 front 2024-05-09T09:04:33.787519+0000 (oldest deadline 2024-05-09T09:04:54.887392+0000) 2024-05-09T09:04:59.840 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:59 smithi105 podman[71927]: 2024-05-09 09:04:59.513396103 +0000 UTC m=+0.630890787 container remove d46657d58b9656cd16aa8e2707bee2eaf8cc333642f6ae335fa9a6b7ed7da1d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc) 2024-05-09T09:05:00.140 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:59 smithi105 podman[72020]: 2024-05-09 09:04:59.745044409 +0000 UTC m=+0.017337554 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:05:00.140 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:04:59 smithi105 podman[72020]: 2024-05-09 09:04:59.84765566 +0000 UTC m=+0.119948843 container create f5666e4ce528f4c80352ef571343bdcd7bb76f923c49643894fa27a22ed5e276 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:05:00.140 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:05:00 smithi105 podman[72020]: 2024-05-09 09:05:00.067391656 +0000 UTC m=+0.339684790 container init f5666e4ce528f4c80352ef571343bdcd7bb76f923c49643894fa27a22ed5e276 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-09T09:05:00.140 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:05:00 smithi105 podman[72020]: 2024-05-09 09:05:00.070896385 +0000 UTC m=+0.343189516 container start f5666e4ce528f4c80352ef571343bdcd7bb76f923c49643894fa27a22ed5e276 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:05:00.141 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:05:00 smithi105 podman[72020]: 2024-05-09 09:05:00.141775477 +0000 UTC m=+0.414068614 container attach f5666e4ce528f4c80352ef571343bdcd7bb76f923c49643894fa27a22ed5e276 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:05:00.486 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 09 09:05:00 smithi105 podman[72020]: 2024-05-09 09:05:00.371225099 +0000 UTC m=+0.643518240 container died f5666e4ce528f4c80352ef571343bdcd7bb76f923c49643894fa27a22ed5e276 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-10-deactivate, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-09T09:05:00.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:00 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:00.342+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:05:00.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:00 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:00.342+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:05:00.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:00 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:00.342+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:05:00.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:00 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:00.342+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:05:00.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:00 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:00.342+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:05:00.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:00 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:00.342+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:05:00.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:00 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:00.342+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-09T09:04:33.787453+0000 front 2024-05-09T09:04:33.787519+0000 (oldest deadline 2024-05-09T09:04:54.887392+0000) 2024-05-09T09:05:01.014 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.10.service' 2024-05-09T09:05:01.052 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:05:01.052 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-09T09:05:01.053 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-09T09:05:01.053 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.11 2024-05-09T09:05:01.341 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:01 smithi142 systemd[1]: Stopping Ceph osd.11 for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:05:01.342 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:01 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:01.342+0000 7fa71d171700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:05:01.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:01 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:01.342+0000 7fa71d171700 -1 osd.11 115 *** Got signal Terminated *** 2024-05-09T09:05:01.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:01 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:01.342+0000 7fa71d171700 -1 osd.11 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:05:01.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:01 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:01.386+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:05:01.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:01 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:01.386+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:05:01.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:01 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:01.386+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:05:01.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:01 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:01.386+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:05:01.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:01 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:01.386+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:05:01.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:01 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:01.386+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:05:01.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:01 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:01.386+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-09T09:04:33.787453+0000 front 2024-05-09T09:04:33.787519+0000 (oldest deadline 2024-05-09T09:04:54.887392+0000) 2024-05-09T09:05:02.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:02 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:02.379+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:05:02.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:02 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:02.379+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:05:02.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:02 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:02.379+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:05:02.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:02 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:02.379+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:05:02.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:02 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:02.379+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:05:02.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:02 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:02.379+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:05:02.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:02 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:02.379+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-09T09:04:33.787453+0000 front 2024-05-09T09:04:33.787519+0000 (oldest deadline 2024-05-09T09:04:54.887392+0000) 2024-05-09T09:05:03.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:03 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:03.377+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:05:03.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:03 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:03.377+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:05:03.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:03 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:03.377+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:05:03.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:03 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:03.377+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:05:03.658 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:03 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:03.377+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:05:03.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:03 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:03.377+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:05:03.659 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:03 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:03.377+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-09T09:04:33.787453+0000 front 2024-05-09T09:04:33.787519+0000 (oldest deadline 2024-05-09T09:04:54.887392+0000) 2024-05-09T09:05:04.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:04 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:04.411+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:05:04.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:04 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:04.411+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:05:04.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:04 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:04.411+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:05:04.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:04 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:04.411+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:05:04.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:04 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:04.411+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:05:04.909 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:04 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:04.411+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:05:04.909 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:04 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:04.411+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-09T09:04:33.787453+0000 front 2024-05-09T09:04:33.787519+0000 (oldest deadline 2024-05-09T09:04:54.887392+0000) 2024-05-09T09:05:04.909 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:04 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:04.411+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6820 osd.8 since back 2024-05-09T09:04:41.888913+0000 front 2024-05-09T09:04:41.888953+0000 (oldest deadline 2024-05-09T09:05:03.588962+0000) 2024-05-09T09:05:05.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:05 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:05.440+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-05-09T09:03:34.977650+0000 front 2024-05-09T09:03:34.977663+0000 (oldest deadline 2024-05-09T09:03:57.277436+0000) 2024-05-09T09:05:05.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:05 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:05.440+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-05-09T09:03:51.780923+0000 front 2024-05-09T09:03:51.780815+0000 (oldest deadline 2024-05-09T09:04:14.680978+0000) 2024-05-09T09:05:05.908 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:05 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:05.440+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-05-09T09:03:58.182145+0000 front 2024-05-09T09:03:58.182168+0000 (oldest deadline 2024-05-09T09:04:22.882028+0000) 2024-05-09T09:05:05.909 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:05 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:05.440+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-09T09:04:05.783431+0000 front 2024-05-09T09:04:05.783491+0000 (oldest deadline 2024-05-09T09:04:30.483464+0000) 2024-05-09T09:05:05.909 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:05 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:05.440+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-05-09T09:04:16.384364+0000 front 2024-05-09T09:04:16.384390+0000 (oldest deadline 2024-05-09T09:04:38.084599+0000) 2024-05-09T09:05:05.909 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:05 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:05.440+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-05-09T09:04:23.886339+0000 front 2024-05-09T09:04:23.886305+0000 (oldest deadline 2024-05-09T09:04:48.586071+0000) 2024-05-09T09:05:05.909 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:05 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:05.440+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-09T09:04:33.787453+0000 front 2024-05-09T09:04:33.787519+0000 (oldest deadline 2024-05-09T09:04:54.887392+0000) 2024-05-09T09:05:05.909 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:05 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11[59436]: 2024-05-09T09:05:05.440+0000 7fa719765700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.142:6820 osd.8 since back 2024-05-09T09:04:41.888913+0000 front 2024-05-09T09:04:41.888953+0000 (oldest deadline 2024-05-09T09:05:03.588962+0000) 2024-05-09T09:05:07.100 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:06 smithi142 podman[71524]: 2024-05-09 09:05:06.847624324 +0000 UTC m=+5.604231088 container died 71ca3b94aed152f0de97c6be826742b37e329c6edd976126275308e38ef20c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-09T09:05:07.408 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:07 smithi142 podman[71524]: 2024-05-09 09:05:07.192704451 +0000 UTC m=+5.949311260 container cleanup 71ca3b94aed152f0de97c6be826742b37e329c6edd976126275308e38ef20c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-09T09:05:07.408 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:07 smithi142 bash[71524]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11 2024-05-09T09:05:07.851 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:07 smithi142 podman[71542]: 2024-05-09 09:05:07.510833974 +0000 UTC m=+0.666862041 container remove 71ca3b94aed152f0de97c6be826742b37e329c6edd976126275308e38ef20c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-09T09:05:08.145 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:07 smithi142 podman[71628]: 2024-05-09 09:05:07.85295281 +0000 UTC m=+0.111786817 container create 2f2d14a7d679fa70d8e1b1af14e1622b581f9dfd164ba4a6cf0ca8527a042123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-96805dc, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:05:08.145 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:07 smithi142 podman[71628]: 2024-05-09 09:05:07.759335377 +0000 UTC m=+0.018169393 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:05:08.145 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:08 smithi142 podman[71628]: 2024-05-09 09:05:08.08026329 +0000 UTC m=+0.339097331 container init 2f2d14a7d679fa70d8e1b1af14e1622b581f9dfd164ba4a6cf0ca8527a042123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:05:08.145 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:08 smithi142 podman[71628]: 2024-05-09 09:05:08.083459725 +0000 UTC m=+0.342293734 container start 2f2d14a7d679fa70d8e1b1af14e1622b581f9dfd164ba4a6cf0ca8527a042123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.schema-version=1.0) 2024-05-09T09:05:08.146 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:08 smithi142 podman[71628]: 2024-05-09 09:05:08.146866512 +0000 UTC m=+0.405700526 container attach 2f2d14a7d679fa70d8e1b1af14e1622b581f9dfd164ba4a6cf0ca8527a042123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc) 2024-05-09T09:05:08.408 INFO:journalctl@ceph.osd.11.smithi142.stdout:May 09 09:05:08 smithi142 podman[71628]: 2024-05-09 09:05:08.373998925 +0000 UTC m=+0.632832936 container died 2f2d14a7d679fa70d8e1b1af14e1622b581f9dfd164ba4a6cf0ca8527a042123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:05:09.004 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@osd.11.service' 2024-05-09T09:05:09.042 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:05:09.042 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-09T09:05:09.042 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-09T09:05:09.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.a 2024-05-09T09:05:09.337 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 09:05:09 smithi098 systemd[1]: Stopping Ceph mds.a for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:05:09.837 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 09:05:09 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-a[71988]: 2024-05-09T09:05:09.343+0000 7fd51ec18700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:05:09.837 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 09:05:09 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-a[71988]: 2024-05-09T09:05:09.343+0000 7fd51ec18700 -1 mds.a *** got signal Terminated *** 2024-05-09T09:05:13.359 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 09:05:13 smithi098 podman[106259]: 2024-05-09 09:05:13.096844382 +0000 UTC m=+3.858619117 container died b659e2da800b915f5d8046e515147a8b6527b2367f9522604c863502873ea1fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-09T09:05:13.734 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 09:05:13 smithi098 podman[106259]: 2024-05-09 09:05:13.468603878 +0000 UTC m=+4.230378616 container cleanup b659e2da800b915f5d8046e515147a8b6527b2367f9522604c863502873ea1fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-a, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-09T09:05:13.734 INFO:journalctl@ceph.mds.a.smithi098.stdout:May 09 09:05:13 smithi098 bash[106259]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-a 2024-05-09T09:05:13.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.a.service' 2024-05-09T09:05:13.919 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:05:13.919 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-09T09:05:13.919 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-09T09:05:13.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.d 2024-05-09T09:05:14.255 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 09:05:14 smithi098 systemd[1]: Stopping Ceph mds.d for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:05:14.587 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 09:05:14 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d[71020]: 2024-05-09T09:05:14.255+0000 7fe645638700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:05:14.587 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 09:05:14 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d[71020]: 2024-05-09T09:05:14.255+0000 7fe645638700 -1 mds.d *** got signal Terminated *** 2024-05-09T09:05:17.337 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 09:05:17 smithi098 podman[106380]: 2024-05-09 09:05:17.075639297 +0000 UTC m=+2.930684084 container died c7c51b9ae468a49a9f13d1b19d9d753d2a5c61739d4f9bab59428e0213a53352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-09T09:05:17.786 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 09:05:17 smithi098 podman[106380]: 2024-05-09 09:05:17.431050909 +0000 UTC m=+3.286095702 container cleanup c7c51b9ae468a49a9f13d1b19d9d753d2a5c61739d4f9bab59428e0213a53352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-09T09:05:17.787 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 09:05:17 smithi098 bash[106380]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d 2024-05-09T09:05:17.787 INFO:journalctl@ceph.mds.d.smithi098.stdout:May 09 09:05:17 smithi098 podman[106394]: 2024-05-09 09:05:17.590452318 +0000 UTC m=+0.506009608 container remove c7c51b9ae468a49a9f13d1b19d9d753d2a5c61739d4f9bab59428e0213a53352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-d, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_BRANCH=HEAD) 2024-05-09T09:05:17.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.d.service' 2024-05-09T09:05:17.845 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:05:17.845 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-09T09:05:17.845 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-09T09:05:17.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.g 2024-05-09T09:05:18.088 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 09:05:17 smithi098 systemd[1]: Stopping Ceph mds.g for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:05:18.587 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 09:05:18 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-g[71625]: 2024-05-09T09:05:18.201+0000 7f46aca49700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:05:18.587 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 09:05:18 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-g[71625]: 2024-05-09T09:05:18.201+0000 7f46aca49700 -1 mds.g *** got signal Terminated *** 2024-05-09T09:05:21.376 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 09:05:21 smithi098 podman[106502]: 2024-05-09 09:05:21.090528542 +0000 UTC m=+3.028954918 container died a66220490dfca0510895d4bdf85a7867bc74cc1d1928d278abcb78ab3ec9265a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-g, org.label-schema.build-date=20240507, GIT_CLEAN=True, RELEASE=squid-96805dc, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-09T09:05:21.709 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 09:05:21 smithi098 podman[106502]: 2024-05-09 09:05:21.46023375 +0000 UTC m=+3.398660135 container cleanup a66220490dfca0510895d4bdf85a7867bc74cc1d1928d278abcb78ab3ec9265a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-g, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:05:21.709 INFO:journalctl@ceph.mds.g.smithi098.stdout:May 09 09:05:21 smithi098 bash[106502]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-g 2024-05-09T09:05:21.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.g.service' 2024-05-09T09:05:21.871 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:05:21.871 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-09T09:05:21.871 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-09T09:05:21.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.j 2024-05-09T09:05:22.087 INFO:journalctl@ceph.mds.j.smithi098.stdout:May 09 09:05:21 smithi098 systemd[1]: Stopping Ceph mds.j for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:05:22.587 INFO:journalctl@ceph.mds.j.smithi098.stdout:May 09 09:05:22 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-j[70671]: 2024-05-09T09:05:22.188+0000 7f4beba8f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:05:22.587 INFO:journalctl@ceph.mds.j.smithi098.stdout:May 09 09:05:22 smithi098 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-j[70671]: 2024-05-09T09:05:22.188+0000 7f4beba8f700 -1 mds.j *** got signal Terminated *** 2024-05-09T09:05:26.465 INFO:journalctl@ceph.mds.j.smithi098.stdout:May 09 09:05:26 smithi098 podman[106624]: 2024-05-09 09:05:26.197196274 +0000 UTC m=+4.108721340 container died ecf3da05d375be0db8c2e9a74c98864bd3bbb51812e4eb0c938edbd1e43b8f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-j, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-09T09:05:26.821 INFO:journalctl@ceph.mds.j.smithi098.stdout:May 09 09:05:26 smithi098 podman[106624]: 2024-05-09 09:05:26.572615942 +0000 UTC m=+4.484141007 container cleanup ecf3da05d375be0db8c2e9a74c98864bd3bbb51812e4eb0c938edbd1e43b8f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-j, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:05:26.821 INFO:journalctl@ceph.mds.j.smithi098.stdout:May 09 09:05:26 smithi098 bash[106624]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-j 2024-05-09T09:05:26.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.j.service' 2024-05-09T09:05:26.977 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:05:26.977 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-09T09:05:26.977 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-09T09:05:26.977 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.b 2024-05-09T09:05:27.364 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 09:05:27 smithi105 systemd[1]: Stopping Ceph mds.b for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:05:27.623 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 09:05:27 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-b[64843]: 2024-05-09T09:05:27.454+0000 7f57c7837700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:05:27.623 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 09:05:27 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-b[64843]: 2024-05-09T09:05:27.454+0000 7f57c7837700 -1 mds.b *** got signal Terminated *** 2024-05-09T09:05:29.394 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 09:05:29 smithi105 podman[72125]: 2024-05-09 09:05:29.075643359 +0000 UTC m=+1.903432953 container died 1a83673e3f3356270df53c9ac846e49a5ef15ca4fa84cfd71aaf73537586d233 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-b, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2) 2024-05-09T09:05:29.394 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 09:05:29 smithi105 podman[72125]: 2024-05-09 09:05:29.396233551 +0000 UTC m=+2.224023142 container cleanup 1a83673e3f3356270df53c9ac846e49a5ef15ca4fa84cfd71aaf73537586d233 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-05-09T09:05:29.713 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 09:05:29 smithi105 bash[72125]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-b 2024-05-09T09:05:29.713 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 09 09:05:29 smithi105 podman[72151]: 2024-05-09 09:05:29.588878838 +0000 UTC m=+0.508971107 container remove 1a83673e3f3356270df53c9ac846e49a5ef15ca4fa84cfd71aaf73537586d233 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-b, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-09T09:05:29.744 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.b.service' 2024-05-09T09:05:29.778 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:05:29.779 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-09T09:05:29.779 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-09T09:05:29.779 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.e 2024-05-09T09:05:30.115 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 09:05:29 smithi105 systemd[1]: Stopping Ceph mds.e for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:05:30.486 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 09:05:30 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68883]: 2024-05-09T09:05:30.116+0000 7fd9db8af700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:05:30.486 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 09:05:30 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e[68883]: 2024-05-09T09:05:30.116+0000 7fd9db8af700 -1 mds.e *** got signal Terminated *** 2024-05-09T09:05:33.049 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 09:05:32 smithi105 podman[72260]: 2024-05-09 09:05:32.78446225 +0000 UTC m=+2.778362842 container died 50591709f9973a4f7ec52cdb079accfa23e9c5457e76f8d6d978e73ed788bdc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.build-date=20240507, GIT_CLEAN=True) 2024-05-09T09:05:33.424 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 09:05:33 smithi105 podman[72260]: 2024-05-09 09:05:33.158244653 +0000 UTC m=+3.152145250 container cleanup 50591709f9973a4f7ec52cdb079accfa23e9c5457e76f8d6d978e73ed788bdc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:05:33.424 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 09 09:05:33 smithi105 bash[72260]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-e 2024-05-09T09:05:33.592 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.e.service' 2024-05-09T09:05:33.623 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:05:33.624 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-09T09:05:33.624 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-09T09:05:33.624 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.h 2024-05-09T09:05:33.943 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 09:05:33 smithi105 systemd[1]: Stopping Ceph mds.h for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:05:34.236 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 09:05:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h[64536]: 2024-05-09T09:05:33.945+0000 7f720afff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:05:34.236 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 09:05:33 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h[64536]: 2024-05-09T09:05:33.945+0000 7f720afff700 -1 mds.h *** got signal Terminated *** 2024-05-09T09:05:37.460 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 09:05:37 smithi105 podman[72381]: 2024-05-09 09:05:37.152582929 +0000 UTC m=+3.312370733 container died d2eef274aff1aa3f88ce3d8b1294e846edd32ee2c37658d560c0e94db276c1ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, ceph=True, org.label-schema.license=GPLv2) 2024-05-09T09:05:37.460 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 09:05:37 smithi105 podman[72381]: 2024-05-09 09:05:37.461916563 +0000 UTC m=+3.621704415 container cleanup d2eef274aff1aa3f88ce3d8b1294e846edd32ee2c37658d560c0e94db276c1ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-09T09:05:37.736 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 09:05:37 smithi105 bash[72381]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h 2024-05-09T09:05:37.736 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 09 09:05:37 smithi105 podman[72395]: 2024-05-09 09:05:37.671412546 +0000 UTC m=+0.519284588 container remove d2eef274aff1aa3f88ce3d8b1294e846edd32ee2c37658d560c0e94db276c1ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-h, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, ceph=True) 2024-05-09T09:05:37.803 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.h.service' 2024-05-09T09:05:37.833 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:05:37.834 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-09T09:05:37.834 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-09T09:05:37.834 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.k 2024-05-09T09:05:38.056 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 09:05:37 smithi105 systemd[1]: Stopping Ceph mds.k for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:05:38.486 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 09:05:38 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-k[63896]: 2024-05-09T09:05:38.164+0000 7f9361eb6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:05:38.486 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 09:05:38 smithi105 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-k[63896]: 2024-05-09T09:05:38.164+0000 7f9361eb6700 -1 mds.k *** got signal Terminated *** 2024-05-09T09:05:41.407 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 09:05:41 smithi105 podman[72502]: 2024-05-09 09:05:41.084061286 +0000 UTC m=+3.024324039 container died 39d1fef78d2833387f20f9d7a11b9f7c77b33c68f6d3274c7d303167b83489b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-k, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-09T09:05:41.731 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 09:05:41 smithi105 podman[72502]: 2024-05-09 09:05:41.507135636 +0000 UTC m=+3.447398388 container cleanup 39d1fef78d2833387f20f9d7a11b9f7c77b33c68f6d3274c7d303167b83489b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-k, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-09T09:05:41.731 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 09 09:05:41 smithi105 bash[72502]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-k 2024-05-09T09:05:41.953 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.k.service' 2024-05-09T09:05:41.984 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:05:41.984 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-09T09:05:41.984 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-09T09:05:41.984 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.c 2024-05-09T09:05:42.278 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 09:05:42 smithi142 systemd[1]: Stopping Ceph mds.c for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:05:42.907 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 09:05:42 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-c[64588]: 2024-05-09T09:05:42.541+0000 7f3be97e6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:05:42.908 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 09:05:42 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-c[64588]: 2024-05-09T09:05:42.541+0000 7f3be97e6700 -1 mds.c *** got signal Terminated *** 2024-05-09T09:05:45.339 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 09:05:45 smithi142 podman[71746]: 2024-05-09 09:05:45.041508208 +0000 UTC m=+2.602295057 container died a83c33cdbf6a3a0741786ea6c5356f2009592b745537d2a91fd5d0e239f286df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-c, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:05:45.658 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 09:05:45 smithi142 podman[71746]: 2024-05-09 09:05:45.448922808 +0000 UTC m=+3.009709664 container cleanup a83c33cdbf6a3a0741786ea6c5356f2009592b745537d2a91fd5d0e239f286df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-c, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-09T09:05:45.658 INFO:journalctl@ceph.mds.c.smithi142.stdout:May 09 09:05:45 smithi142 bash[71746]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-c 2024-05-09T09:05:45.785 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.c.service' 2024-05-09T09:05:45.820 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:05:45.820 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-09T09:05:45.820 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-09T09:05:45.820 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.f 2024-05-09T09:05:46.143 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 09:05:45 smithi142 systemd[1]: Stopping Ceph mds.f for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:05:46.408 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 09:05:46 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-f[64279]: 2024-05-09T09:05:46.144+0000 7f2ce2e8d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:05:46.408 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 09:05:46 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-f[64279]: 2024-05-09T09:05:46.144+0000 7f2ce2e8d700 -1 mds.f *** got signal Terminated *** 2024-05-09T09:05:49.326 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 09:05:49 smithi142 podman[71877]: 2024-05-09 09:05:49.043850902 +0000 UTC m=+3.002951912 container died ea75d301ead0b9caa8bad2fccafe84480eb678f203fd2f7e976b03152bc12dfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-f, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:05:49.658 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 09:05:49 smithi142 podman[71877]: 2024-05-09 09:05:49.410744592 +0000 UTC m=+3.369845608 container cleanup ea75d301ead0b9caa8bad2fccafe84480eb678f203fd2f7e976b03152bc12dfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-f, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:05:49.658 INFO:journalctl@ceph.mds.f.smithi142.stdout:May 09 09:05:49 smithi142 bash[71877]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-f 2024-05-09T09:05:49.792 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.f.service' 2024-05-09T09:05:49.824 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:05:49.824 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-09T09:05:49.824 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-09T09:05:49.824 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.i 2024-05-09T09:05:50.154 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 09:05:49 smithi142 systemd[1]: Stopping Ceph mds.i for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:05:50.408 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 09:05:50 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-i[64906]: 2024-05-09T09:05:50.156+0000 7f4640ffd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:05:50.408 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 09:05:50 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-i[64906]: 2024-05-09T09:05:50.156+0000 7f4640ffd700 -1 mds.i *** got signal Terminated *** 2024-05-09T09:05:53.337 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 09:05:53 smithi142 podman[71996]: 2024-05-09 09:05:53.053291894 +0000 UTC m=+3.012757014 container died 8d37654b0fb09f1bbdce58a9804c3af29b9b8a25172378bf87fcc5b58093e68c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-i, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-09T09:05:53.658 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 09:05:53 smithi142 podman[71996]: 2024-05-09 09:05:53.430746796 +0000 UTC m=+3.390211912 container cleanup 8d37654b0fb09f1bbdce58a9804c3af29b9b8a25172378bf87fcc5b58093e68c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-i, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:05:53.658 INFO:journalctl@ceph.mds.i.smithi142.stdout:May 09 09:05:53 smithi142 bash[71996]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-i 2024-05-09T09:05:53.830 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.i.service' 2024-05-09T09:05:53.861 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:05:53.861 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-09T09:05:53.861 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-09T09:05:53.861 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.l 2024-05-09T09:05:54.158 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 09:05:53 smithi142 systemd[1]: Stopping Ceph mds.l for a57d9712-0ddf-11ef-bc98-c7b262605968... 2024-05-09T09:05:54.658 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 09:05:54 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-l[63968]: 2024-05-09T09:05:54.193+0000 7f4c6b6a3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:05:54.658 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 09:05:54 smithi142 ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-l[63968]: 2024-05-09T09:05:54.193+0000 7f4c6b6a3700 -1 mds.l *** got signal Terminated *** 2024-05-09T09:05:57.408 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 09:05:57 smithi142 podman[72115]: 2024-05-09 09:05:57.055954256 +0000 UTC m=+2.972315722 container died 7a77746dd9b0e33991462528211ed3fbb9e329e638d1fb05669721c25ca0ae89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-l, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:05:57.867 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 09:05:57 smithi142 podman[72115]: 2024-05-09 09:05:57.676034441 +0000 UTC m=+3.592395913 container cleanup 7a77746dd9b0e33991462528211ed3fbb9e329e638d1fb05669721c25ca0ae89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-09T09:05:57.867 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 09:05:57 smithi142 bash[72115]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-l 2024-05-09T09:05:58.096 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.l.service' 2024-05-09T09:05:58.117 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 09:05:57 smithi142 podman[72129]: 2024-05-09 09:05:57.868541174 +0000 UTC m=+0.534545641 container remove 7a77746dd9b0e33991462528211ed3fbb9e329e638d1fb05669721c25ca0ae89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-a57d9712-0ddf-11ef-bc98-c7b262605968-mds-l, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-09T09:05:58.117 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 09:05:58 smithi142 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.l.service: Deactivated successfully. 2024-05-09T09:05:58.117 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 09:05:58 smithi142 systemd[1]: Stopped Ceph mds.l for a57d9712-0ddf-11ef-bc98-c7b262605968. 2024-05-09T09:05:58.117 INFO:journalctl@ceph.mds.l.smithi142.stdout:May 09 09:05:58 smithi142 systemd[1]: ceph-a57d9712-0ddf-11ef-bc98-c7b262605968@mds.l.service: Consumed 1.991s CPU time. 2024-05-09T09:05:58.126 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:05:58.126 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-09T09:05:58.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-09T09:05:58.317 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T09:06:00.867 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-09T09:06:01.056 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T09:06:03.550 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-09T09:06:03.738 INFO:teuthology.orchestra.run.smithi142.stdout:Deleting cluster with fsid: a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T09:06:06.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:06:06.146 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:06:06.177 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:06:06.202 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-09T09:06:06.204 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693/remote/smithi098/crash 2024-05-09T09:06:06.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/crash -- . 2024-05-09T09:06:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-09T09:06:06.243 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-05-09T09:06:06.245 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693/remote/smithi105/crash 2024-05-09T09:06:06.246 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/crash -- . 2024-05-09T09:06:06.359 INFO:teuthology.orchestra.run.smithi105.stderr:tar: /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-09T09:06:06.359 INFO:teuthology.orchestra.run.smithi105.stderr:tar: Error is not recoverable: exiting now 2024-05-09T09:06:06.361 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693/remote/smithi142/crash 2024-05-09T09:06:06.362 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/crash -- . 2024-05-09T09:06:06.400 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-09T09:06:06.400 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now 2024-05-09T09:06:06.401 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-09T09:06:06.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-09T09:06:06.500 INFO:tasks.cephadm:Compressing logs... 2024-05-09T09:06:06.500 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T09:06:06.503 DEBUG:teuthology.orchestra.run.smithi105:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T09:06:06.505 DEBUG:teuthology.orchestra.run.smithi142:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T09:06:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-09T09:06:06.529 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-09T09:06:06.529 INFO:teuthology.orchestra.run.smithi142.stderr:: No such file or directory 2024-05-09T09:06:06.529 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.log 2024-05-09T09:06:06.530 INFO:teuthology.orchestra.run.smithi142.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.2.log 2024-05-09T09:06:06.530 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.f.log 2024-05-09T09:06:06.530 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-09T09:06:06.532 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.5.log 2024-05-09T09:06:06.532 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.c.log 2024-05-09T09:06:06.532 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mon.c.log 2024-05-09T09:06:06.532 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mgr.z.log 2024-05-09T09:06:06.533 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.i.log 2024-05-09T09:06:06.533 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mon.c.log: /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-09T09:06:06.533 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.11.log 2024-05-09T09:06:06.533 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.cephadm.log: -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.8.log 2024-05-09T09:06:06.533 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-09T09:06:06.533 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.l.log 2024-05-09T09:06:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-09T09:06:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.log 2024-05-09T09:06:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-09T09:06:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.g.log 2024-05-09T09:06:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.log: /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.j.log 2024-05-09T09:06:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.3.log 2024-05-09T09:06:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.0.log 2024-05-09T09:06:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mgr.x.log 2024-05-09T09:06:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mon.a.log 2024-05-09T09:06:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.6.log 2024-05-09T09:06:06.536 INFO:teuthology.orchestra.run.smithi098.stderr: 86.6% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-09T09:06:06.537 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-09T09:06:06.538 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.a.log 2024-05-09T09:06:06.538 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-09T09:06:06.539 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.d.log 2024-05-09T09:06:06.539 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.9.log 2024-05-09T09:06:06.539 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-05-09T09:06:06.570 INFO:teuthology.orchestra.run.smithi105.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-09T09:06:06.571 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-09T09:06:06.571 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.log 2024-05-09T09:06:06.572 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-09T09:06:06.573 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.7.log 2024-05-09T09:06:06.573 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.log: /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.b.log 2024-05-09T09:06:06.573 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.4.log 2024-05-09T09:06:06.576 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.1.log 2024-05-09T09:06:06.576 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mgr.y.log 2024-05-09T09:06:06.576 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.h.log 2024-05-09T09:06:06.576 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.10.log 2024-05-09T09:06:06.576 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-09T09:06:06.576 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.e.log 2024-05-09T09:06:06.576 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-09T09:06:06.576 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mon.b.log 2024-05-09T09:06:06.576 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.k.log 2024-05-09T09:06:06.577 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-volume.log: /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.l.log: 93.1% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.l.log.gz 2024-05-09T09:06:06.604 INFO:teuthology.orchestra.run.smithi142.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-09T09:06:06.619 INFO:teuthology.orchestra.run.smithi142.stderr: 83.8% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-09T09:06:06.630 INFO:teuthology.orchestra.run.smithi142.stderr: 86.6% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-09T09:06:06.670 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.9.log: /var/log/ceph/ceph-client.admin.log: 94.4% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.g.log.gz 2024-05-09T09:06:06.671 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mon.b.log: /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.k.log: 92.0% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mgr.y.log.gz 2024-05-09T09:06:06.701 INFO:teuthology.orchestra.run.smithi105.stderr: 83.9% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-09T09:06:06.703 INFO:teuthology.orchestra.run.smithi105.stderr: 86.6% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-09T09:06:06.714 INFO:teuthology.orchestra.run.smithi142.stderr: 90.2% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-09T09:06:06.715 INFO:teuthology.orchestra.run.smithi098.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-09T09:06:06.716 INFO:teuthology.orchestra.run.smithi105.stderr: 90.1% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-09T09:06:06.728 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-09T09:06:06.730 INFO:teuthology.orchestra.run.smithi105.stderr: 92.9% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.b.log.gz 2024-05-09T09:06:06.747 INFO:teuthology.orchestra.run.smithi098.stderr: 91.1% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-09T09:06:06.751 INFO:teuthology.orchestra.run.smithi105.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-09T09:06:06.752 INFO:teuthology.orchestra.run.smithi142.stderr: 91.0% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-09T09:06:06.792 INFO:teuthology.orchestra.run.smithi098.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-05-09T09:06:06.813 INFO:teuthology.orchestra.run.smithi142.stderr: 92.1% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mgr.z.log.gz 2024-05-09T09:06:06.824 INFO:teuthology.orchestra.run.smithi098.stderr: 84.1% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-09T09:06:06.916 INFO:teuthology.orchestra.run.smithi105.stderr: 91.0% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-09T09:06:06.925 INFO:teuthology.orchestra.run.smithi098.stderr: 94.6% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.j.log.gz 2024-05-09T09:06:06.968 INFO:teuthology.orchestra.run.smithi142.stderr: 92.7% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.i.log.gz 2024-05-09T09:06:07.590 INFO:teuthology.orchestra.run.smithi105.stderr: 94.5% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.k.log.gz 2024-05-09T09:06:07.759 INFO:teuthology.orchestra.run.smithi142.stderr: 92.9% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.c.log.gz 2024-05-09T09:06:07.885 INFO:teuthology.orchestra.run.smithi142.stderr: 91.9% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mon.c.log.gz 2024-05-09T09:06:08.026 INFO:teuthology.orchestra.run.smithi098.stderr: 94.4% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.a.log.gz 2024-05-09T09:06:08.424 INFO:teuthology.orchestra.run.smithi142.stderr: 94.6% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.f.log.gz 2024-05-09T09:06:10.094 INFO:teuthology.orchestra.run.smithi105.stderr: 91.8% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mon.b.log.gz 2024-05-09T09:06:10.522 INFO:teuthology.orchestra.run.smithi098.stderr: 90.2% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mgr.x.log.gz 2024-05-09T09:06:11.275 INFO:teuthology.orchestra.run.smithi098.stderr: 90.5% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mon.a.log.gz 2024-05-09T09:06:52.107 INFO:teuthology.orchestra.run.smithi098.stderr: 97.2% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.d.log.gz 2024-05-09T09:08:25.040 INFO:teuthology.orchestra.run.smithi142.stderr: 92.4% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.5.log.gz 2024-05-09T09:09:26.162 INFO:teuthology.orchestra.run.smithi142.stderr: 92.7% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.8.log.gz 2024-05-09T09:09:37.319 INFO:teuthology.orchestra.run.smithi105.stderr: 97.3% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.e.log.gz 2024-05-09T09:09:51.547 INFO:teuthology.orchestra.run.smithi142.stderr: 92.6% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.11.log.gz 2024-05-09T09:10:11.076 INFO:teuthology.orchestra.run.smithi142.stderr: 92.6% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.2.log.gz 2024-05-09T09:10:11.077 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-05-09T09:10:11.077 INFO:teuthology.orchestra.run.smithi142.stderr:real 4m4.558s 2024-05-09T09:10:11.077 INFO:teuthology.orchestra.run.smithi142.stderr:user 4m32.527s 2024-05-09T09:10:11.077 INFO:teuthology.orchestra.run.smithi142.stderr:sys 0m10.509s 2024-05-09T09:10:33.627 INFO:teuthology.orchestra.run.smithi105.stderr: 92.7% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.1.log.gz 2024-05-09T09:11:50.474 INFO:teuthology.orchestra.run.smithi098.stderr: 92.5% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.3.log.gz 2024-05-09T09:12:35.745 INFO:teuthology.orchestra.run.smithi105.stderr: 92.5% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.4.log.gz 2024-05-09T09:13:47.585 INFO:teuthology.orchestra.run.smithi098.stderr: 92.6% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.0.log.gz 2024-05-09T09:15:10.155 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.6.log.gz 2024-05-09T09:15:19.406 INFO:teuthology.orchestra.run.smithi105.stderr: 92.8% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.10.log.gz 2024-05-09T09:15:46.859 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.9.log.gz 2024-05-09T09:15:46.861 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-05-09T09:15:46.861 INFO:teuthology.orchestra.run.smithi098.stderr:real 9m40.344s 2024-05-09T09:15:46.861 INFO:teuthology.orchestra.run.smithi098.stderr:user 5m29.247s 2024-05-09T09:15:46.861 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m14.753s 2024-05-09T09:16:03.216 INFO:teuthology.orchestra.run.smithi105.stderr: 92.5% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-osd.7.log.gz 2024-05-09T09:16:32.236 INFO:teuthology.orchestra.run.smithi105.stderr: 94.3% -- replaced with /var/log/ceph/a57d9712-0ddf-11ef-bc98-c7b262605968/ceph-mds.h.log.gz 2024-05-09T09:16:32.238 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-05-09T09:16:32.238 INFO:teuthology.orchestra.run.smithi105.stderr:real 10m25.718s 2024-05-09T09:16:32.238 INFO:teuthology.orchestra.run.smithi105.stderr:user 5m57.925s 2024-05-09T09:16:32.239 INFO:teuthology.orchestra.run.smithi105.stderr:sys 0m16.308s 2024-05-09T09:16:32.239 INFO:tasks.cephadm:Archiving logs... 2024-05-09T09:16:32.240 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693/remote/smithi098/log 2024-05-09T09:16:32.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T09:17:28.821 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693/remote/smithi105/log 2024-05-09T09:17:28.822 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T09:18:29.256 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693/remote/smithi142/log 2024-05-09T09:18:29.623 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T09:19:18.224 INFO:tasks.cephadm:Removing cluster... 2024-05-09T09:19:18.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 --force 2024-05-09T09:19:18.432 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T09:19:19.076 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 --force 2024-05-09T09:19:19.330 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T09:19:19.989 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a57d9712-0ddf-11ef-bc98-c7b262605968 --force 2024-05-09T09:19:20.275 INFO:teuthology.orchestra.run.smithi142.stdout:Deleting cluster with fsid: a57d9712-0ddf-11ef-bc98-c7b262605968 2024-05-09T09:19:20.876 INFO:tasks.cephadm:Removing cephadm ... 2024-05-09T09:19:20.876 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-09T09:19:20.897 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-09T09:19:20.916 DEBUG:teuthology.orchestra.run.smithi142:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-09T09:19:20.931 INFO:tasks.cephadm:Teardown complete 2024-05-09T09:19:20.931 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-09T09:19:20.943 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-09T09:19:20.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T09:19:20.947 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T09:19:20.958 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T09:19:21.093 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-09T09:19:21.104 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-09T09:19:21.104 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T09:19:21.107 DEBUG:teuthology.orchestra.run.smithi105:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T09:19:21.110 DEBUG:teuthology.orchestra.run.smithi142:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T09:19:21.122 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-05-09T09:19:21.123 INFO:teuthology.orchestra.run.smithi142.stderr:bash: line 1: ntpq: command not found 2024-05-09T09:19:21.124 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-05-09T09:19:21.127 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T09:19:21.127 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-05-09T09:19:21.127 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 9 +508us[ +508us] +/- 67ms 2024-05-09T09:19:21.127 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -725us[ -719us] +/- 58ms 2024-05-09T09:19:21.127 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +428us[ +428us] +/- 80ms 2024-05-09T09:19:21.127 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:19:21.142 INFO:teuthology.orchestra.run.smithi142.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T09:19:21.142 INFO:teuthology.orchestra.run.smithi142.stdout:=============================================================================== 2024-05-09T09:19:21.142 INFO:teuthology.orchestra.run.smithi142.stdout:^* hv01.front.sepia.ceph.com 3 6 377 10 +505us[ +525us] +/- 67ms 2024-05-09T09:19:21.142 INFO:teuthology.orchestra.run.smithi142.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 14 -726us[ -706us] +/- 58ms 2024-05-09T09:19:21.142 INFO:teuthology.orchestra.run.smithi142.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 +418us[ +437us] +/- 80ms 2024-05-09T09:19:21.142 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:19:21.163 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T09:19:21.163 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-05-09T09:19:21.163 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 11 +497us[ +497us] +/- 67ms 2024-05-09T09:19:21.163 INFO:teuthology.orchestra.run.smithi105.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -737us[ -722us] +/- 58ms 2024-05-09T09:19:21.163 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 +413us[ +428us] +/- 80ms 2024-05-09T09:19:21.163 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:19:21.164 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-09T09:19:21.175 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-09T09:19:21.176 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-09T09:19:21.187 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T09:19:21.321 DEBUG:teuthology.orchestra.run.smithi105:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T09:19:21.421 DEBUG:teuthology.orchestra.run.smithi142:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T09:19:21.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T09:19:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:19:21.538 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T09:19:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:19:21.568 DEBUG:teuthology.orchestra.run.smithi142:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T09:19:21.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:19:21.595 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-09T09:19:21.620 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-09T09:19:21.633 INFO:teuthology.task.internal:Duration was 2803.426800 seconds 2024-05-09T09:19:21.633 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-09T09:19:21.645 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-09T09:19:21.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T09:19:21.649 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T09:19:21.652 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T09:19:21.700 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T09:19:21.709 INFO:teuthology.orchestra.run.smithi142.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T09:19:21.712 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T09:19:22.036 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-09T09:19:22.036 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-05-09T09:19:22.037 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T09:19:22.063 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi105.front.sepia.ceph.com 2024-05-09T09:19:22.063 DEBUG:teuthology.orchestra.run.smithi105:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T09:19:22.097 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi142.front.sepia.ceph.com 2024-05-09T09:19:22.097 DEBUG:teuthology.orchestra.run.smithi142:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T09:19:22.126 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-09T09:19:22.126 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T09:19:22.129 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T09:19:22.139 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T09:19:22.191 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-09T09:19:22.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T09:19:22.197 DEBUG:teuthology.orchestra.run.smithi105:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T09:19:22.211 DEBUG:teuthology.orchestra.run.smithi142:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T09:19:22.618 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-09T09:19:22.630 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-09T09:19:22.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T09:19:22.662 DEBUG:teuthology.orchestra.run.smithi105:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T09:19:22.711 DEBUG:teuthology.orchestra.run.smithi142:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T09:19:22.747 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-09T09:19:22.759 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:19:22.762 DEBUG:teuthology.orchestra.run.smithi105:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:19:22.765 DEBUG:teuthology.orchestra.run.smithi142:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:19:22.792 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-05-09T09:19:22.793 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = core 2024-05-09T09:19:22.818 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = core 2024-05-09T09:19:22.836 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:19:22.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:19:22.868 DEBUG:teuthology.orchestra.run.smithi105:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:19:22.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:19:22.883 DEBUG:teuthology.orchestra.run.smithi142:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:19:22.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:19:22.897 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-09T09:19:22.910 INFO:teuthology.task.internal:Transferring archived files... 2024-05-09T09:19:22.911 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693/remote/smithi098 2024-05-09T09:19:22.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T09:19:22.955 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693/remote/smithi105 2024-05-09T09:19:22.956 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T09:19:22.991 DEBUG:teuthology.misc:Transferring archived files from smithi142:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698693/remote/smithi142 2024-05-09T09:19:22.992 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T09:19:23.028 INFO:teuthology.task.internal:Removing archive directory... 2024-05-09T09:19:23.028 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T09:19:23.031 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T09:19:23.033 DEBUG:teuthology.orchestra.run.smithi142:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T09:19:23.075 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-09T09:19:23.091 INFO:teuthology.task.internal:Not uploading archives. 2024-05-09T09:19:23.091 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-09T09:19:23.107 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-09T09:19:23.107 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T09:19:23.110 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T09:19:23.112 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T09:19:23.122 INFO:teuthology.orchestra.run.smithi098.stdout: 265585 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 09:19 /home/ubuntu/cephtest 2024-05-09T09:19:23.124 INFO:teuthology.orchestra.run.smithi105.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 09:19 /home/ubuntu/cephtest 2024-05-09T09:19:23.129 INFO:teuthology.orchestra.run.smithi142.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 09:19 /home/ubuntu/cephtest 2024-05-09T09:19:23.130 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-09T09:19:23.143 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-09T09:19:23.170 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} duration: 2803.42679977417 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-05-09T09:19:23.171 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T09:19:23.253 INFO:teuthology.run:pass